Feature #12280
openIO.copy_stream(IO, IO) fails with "pread() not implemented"
Added by Eregon (Benoit Daloze) over 8 years ago. Updated 12 months ago.
Description
IO.copy_stream on Windows with IO having an offset fails with:
NotImplementedError: pread() not implemented
This seems suboptimal, should a clearer be reported?
Is it a platform limitation?
Updated by normalperson (Eric Wong) over 8 years ago
eregontp@gmail.com wrote:
IO.copy_stream on Windows with IO having an offset fails with:
NotImplementedError: pread() not implementedThis seems suboptimal, should a clearer be reported?
Not sure. Maybe include a bug report address to get pread
implemented in Windows? :)
Is it a platform limitation?
Yes, it's a platform limitation.
lseek+read can be used to workaround the limitation in
single-threaded programs only (and only as long as the FD is not
shared across fork; maybe FD sharing across fork isn't possible
in Windows...)
But I guess an error message is better than a racy workaround
which can corrupt data with concurrent access.
Updated by Eregon (Benoit Daloze) over 8 years ago
Eric Wong wrote:
eregontp@gmail.com wrote:
IO.copy_stream on Windows with IO having an offset fails with:
NotImplementedError: pread() not implementedThis seems suboptimal, should a clearer be reported?
Not sure. Maybe include a bug report address to get pread
implemented in Windows? :)Is it a platform limitation?
Yes, it's a platform limitation.
lseek+read can be used to workaround the limitation in
single-threaded programs only (and only as long as the FD is not
shared across fork; maybe FD sharing across fork isn't possible
in Windows...)But I guess an error message is better than a racy workaround
which can corrupt data with concurrent access.
Thanks for the quick reply!
For me, having a message like:
"NotImplementedError: IO.copy_stream with offset is not supported on Windows since it is not atomic"
would be a good enough fix for this.
Updated by naruse (Yui NARUSE) over 8 years ago
Benoit Daloze wrote:
For me, having a message like:
"NotImplementedError: IO.copy_stream with offset is not supported on Windows since it is not atomic"
would be a good enough fix for this.
IO.copy_stream is a utility method.
It works as expected and fast if there's a suitable API like sendfile.
On this precondition, IO.copy_stream should work even on Windows.
You can simply acquire GVL or fine grained lock to achieve multithread safety.
Anyway you can provide pread with ReadFile.
You may know, on Windows read() is our own function and the real name is rb_w32_read.
You can implement rb_w32_pread with ReadFile in win32/win32.c.
Updated by shyouhei (Shyouhei Urabe) over 8 years ago
+1 to add a fallback implementation when pread is unavailable.
Updated by normalperson (Eric Wong) over 8 years ago
naruse@airemix.jp wrote:
Benoit Daloze wrote:
For me, having a message like:
"NotImplementedError: IO.copy_stream with offset is not supported on Windows since it is not atomic"
would be a good enough fix for this.IO.copy_stream is a utility method.
It works as expected and fast if there's a suitable API like sendfile.On this precondition, IO.copy_stream should work even on Windows.
You can simply acquire GVL or fine grained lock to achieve multithread safety.
Can open files be shared across processes in Windows?
If so, thread-safety is not enough; pread provides atomicity across
forked processes (and also send_io/recv_io) in *nix; not just threads.
Anyway you can provide pread with ReadFile.
You may know, on Windows read() is our own function and the real name is rb_w32_read.
You can implement rb_w32_pread with ReadFile in win32/win32.c.
Anyways, I'll leave Windows devs to implement if it is possible
and safe :)
Updated by jeremyevans0 (Jeremy Evans) 12 months ago
- Tracker changed from Bug to Feature
- Backport deleted (
2.1: UNKNOWN, 2.2: UNKNOWN, 2.3: UNKNOWN)