Bug #21634: Combining read(1) with eof? causes dropout of results unexpectedly on Windows.

• Status: Open
• Assignee: -
• Target version: -
• ruby -v: ruby 3.5.0dev (2025-10-03T08:59:54Z master 5b2ec0eb1b) +PRISM [x64-mingw-ucrt]
• Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN

[YO4 (Yoshinao Muramatsu)]

On Windows, when reading a file containing EOF(\x1A), using read(1) with IO#eof? causes unexpected dropout of results.

irb(main):001> IO.binwrite("txt", "abcd\x1A")
=> 5
irb(main):002> open("txt", "r") { p _1.read(1) until _1.eof? }; # works fine
"a"
"b"
"c"
"d"
"\x1A"
irb(main):003> open("txt", "rt") { p _1.read(1) until _1.eof? }; # has failure
"b"
"d"
irb(main):004>

The problem disappeared when I commented out one of the following lines (though this will break other things).

• previous_mode = set_binary_mode_with_seek_cur(fptr); in io_read()
• flush_before_seek(fptr, false); in set_binary_mode_with_seek_cur(()
• io_unread(fptr, discard_rbuf); in flush_before_seek()

Within io_unread(), rbuf.len should have changed as 5, 4, 3,... but instead changed as 4, 2,(end).
Since inconsistencies already exist at this point, the problem appears to originate elsewhere.

I found this in ruby master but the same issue was found at least in ruby-1.9.3-p551.

[YO4 (Yoshinao Muramatsu)]

The IO that has mode_enc "rt" will read with O_BINARY but opend with O_TEXT.
This leads fill_cbuf using O_TEXT at rb_io_eof unexpectedly.

I made PR #18410.

[nobu (Nobuyoshi Nakada)]

YO4 (Yoshinao Muramatsu) wrote in #note-1:

The IO that has mode_enc "rt" will read with O_BINARY but opend with O_TEXT.
This leads fill_cbuf using O_TEXT at rb_io_eof unexpectedly.

I made PR #18410.

Thank you for the patch.

IO#eof? behavior seems changing.

With "txt" file that its content is "abcd\x1A\r\n",
the current IO#eof? returns true at "\x1A", and further more read stops there.

> .\miniruby.exe -v -e "open('txt', 'rt') {|f| p f.read(4); p f.eof?; p f.read}"    
ruby 3.5.0dev (2025-10-11T06:00:21Z master e8f0e1423b) +PRISM [arm64-mswin64_140]
"abcd"
true
""

However, with your PR, it seems simply "\x1A" is not considered EOF.

> .\miniruby-new.exe -v -e "open('txt', 'rt') {|f| p f.read(4); p f.eof?; p f.read}"
ruby 3.5.0dev (2025-10-11T06:05:13Z eof-and-fpos 6e568e9cb2) +PRISM [arm64-mswin64_140]
last_commit=Set O_BINARY correctly at rb_io_eof()
"abcd"
false
"\u001A\n"