Feature #12280: IO.copy_stream(IO, IO) fails with "pread() not implemented"

• Status: Open
• Assignee: -
• Target version: -
• Tags: core, win

[Eregon (Benoit Daloze)]

IO.copy_stream on Windows with IO having an offset fails with:
NotImplementedError: pread() not implemented

This seems suboptimal, should a clearer be reported?
Is it a platform limitation?

[normalperson (Eric Wong)]

[e-mail] wrote:

IO.copy_stream on Windows with IO having an offset fails with:
NotImplementedError: pread() not implemented

This seems suboptimal, should a clearer be reported?

Not sure. Maybe include a bug report address to get pread
implemented in Windows? :)

Is it a platform limitation?

Yes, it's a platform limitation.

lseek+read can be used to workaround the limitation in
single-threaded programs only (and only as long as the FD is not
shared across fork; maybe FD sharing across fork isn't possible
in Windows...)

But I guess an error message is better than a racy workaround
which can corrupt data with concurrent access.

[Eregon (Benoit Daloze)]

Eric Wong wrote:

[e-mail] wrote:

IO.copy_stream on Windows with IO having an offset fails with:
NotImplementedError: pread() not implemented

This seems suboptimal, should a clearer be reported?

Not sure. Maybe include a bug report address to get pread
implemented in Windows? :)

Is it a platform limitation?

Yes, it's a platform limitation.

lseek+read can be used to workaround the limitation in
single-threaded programs only (and only as long as the FD is not
shared across fork; maybe FD sharing across fork isn't possible
in Windows...)

But I guess an error message is better than a racy workaround
which can corrupt data with concurrent access.

Thanks for the quick reply!

For me, having a message like:
"NotImplementedError: IO.copy_stream with offset is not supported on Windows since it is not atomic"
would be a good enough fix for this.

[naruse (Yui NARUSE)]

Benoit Daloze wrote:

For me, having a message like:
"NotImplementedError: IO.copy_stream with offset is not supported on Windows since it is not atomic"
would be a good enough fix for this.

IO.copy_stream is a utility method.
It works as expected and fast if there's a suitable API like sendfile.

On this precondition, IO.copy_stream should work even on Windows.
You can simply acquire GVL or fine grained lock to achieve multithread safety.

Anyway you can provide pread with ReadFile.
You may know, on Windows read() is our own function and the real name is rb_w32_read.
You can implement rb_w32_pread with ReadFile in win32/win32.c.

[shyouhei (Shyouhei Urabe)]

+1 to add a fallback implementation when pread is unavailable.

[normalperson (Eric Wong)]

[e-mail] wrote:

Benoit Daloze wrote:

For me, having a message like:
"NotImplementedError: IO.copy_stream with offset is not supported on Windows since it is not atomic"
would be a good enough fix for this.

IO.copy_stream is a utility method.
It works as expected and fast if there's a suitable API like sendfile.

On this precondition, IO.copy_stream should work even on Windows.
You can simply acquire GVL or fine grained lock to achieve multithread safety.

Can open files be shared across processes in Windows?
If so, thread-safety is not enough; pread provides atomicity across
forked processes (and also send_io/recv_io) in *nix; not just threads.

Anyway you can provide pread with ReadFile.
You may know, on Windows read() is our own function and the real name is rb_w32_read.
You can implement rb_w32_pread with ReadFile in win32/win32.c.

Anyways, I'll leave Windows devs to implement if it is possible
and safe :)

[jeremyevans0 (Jeremy Evans)]

• Tracker changed from Bug to Feature
• Backport deleted (2.1: UNKNOWN, 2.2: UNKNOWN, 2.3: UNKNOWN)