Feature #22400: Use rb_long_t for lengths so that String and Array can exceed 2GiB on mswin

• Status: Open
• Assignee: -
• Target version: -

[hsbt (Hiroshi SHIBATA)]

On x64-mswin64, long is 32 bits while pointers are 64 bits. String, Array and MatchData keep their lengths in long, so none of them can reach 2GiB.

"a" * 2**31                              # RangeError: bignum too big to convert into 'long'
File.open(path, "rb") { it.read(2**31) } # RangeError
Array.new(2**31)                         # RangeError

I propose giving these lengths and indices a type of their own, rb_long_t, in two steps.

The first step is https://github.com/ruby/ruby/pull/18862. It defines typedef long rb_long_t everywhere and uses it for the lengths and indices of RString, RArray and RMatch, the C API that carries them, and the core, adding LONGT2NUM, NUM2LONGT and PRIdLONGT. A static assert keeps it long, so nothing changes, and NUM2LONG and FIX2LONG still return long.

The second step makes rb_long_t 64 bits on mswin, as rb_off_t already does for off_t there. mingw keeps long, and Fixnum (#14378) stays as is. This breaks extensions that pass a long * to an out-parameter such as rb_range_beg_len, because Ruby then writes 8 bytes into a 4-byte variable. -we4133 makes that a compile error, and I will send fixes to the gems it hits: nokogiri, google-protobuf, nokolexbor, string_view, readline-ext and iconv. An extension supporting older Ruby can typedef long rb_long_t when RB_LONGT_MAX is undefined.

No existing type fits. ssize_t and intptr_t are int on ILP32 and would change long * parameters there. SIGNED_VALUE is already 64 bits on mswin, so it could not land as a no-op first.

To catch new code that adds a plain long for a length, I would like a CI job that builds with rb_long_t set to long long on Linux, where %ld and long * mismatches fail.

First, do you agree with this direction and with carrying it out? If so, is rb_long_t a good name? And since few C extensions are distributed for mswin, I think we could make the second step in 4.1 as well, without a transition period. What do you think?

[hsbt (Hiroshi SHIBATA)]

• Related to Feature #14378: Increase Fixnum range on Windows from 31 bits to 63 bits added

[hsbt (Hiroshi SHIBATA)]

• Related to Bug #20614: Integer#size returns incorrect values on 64-bit Windows added

[hsbt (Hiroshi SHIBATA)]

• Related to Bug #11235: [BUG] Segmentation fault added

[byroot (Jean Boussier)]

What do you think?

I think I'd be worth trying for a preview (there was no 4.1.0preview this year?) as to see what the blast radius really is.

Other than that, I wonder if one avenue could be to have this be a build flag first, allowing gem maintainers 1 year to test and handle the change (bonus point if it's available in ruby/setup-ruby.