Misc #20488
closedDocument source file size restrictions
Description
I was hoping we might be able to decide on an official maximum size for a source file in terms of bytes/lines/columns/etc.
Ruby uses fixed integer sizes to represent line numbers, column numbers, offsets, etc. These can overflow with files that are too big. Sometimes they will fail with cryptic messages like:
ruby: negative string size (or size too big) (ArgumentError)
(That's for 2^31 "a"s in a row.) Note that for python for the same file you will get:
OverflowError: Parser column offset overflow - source line is too big
and for perl you will get:
Identifier too long at test.pl line 1.
For files with 2^32 newlines, Ruby just crashes on my machine with the current parser (prism finishes, but its newline counter overflows so it gets all the offsets wrong).
Would it be okay to say:
- maximum line: 31 bits
- maximum column: 32 bits
- maximum file byte size: 32 bits
This would also help with memory savings — I would like to only use 32 bits for offsets in the file, as opposed to the current 64 bits I'm using (which seems unnecessarily large).
Updated by matz (Yukihiro Matsumoto) 5 months ago
The limit will be declared (for CRuby):
- maximum line: signed 32 bits (
eval
uses negative line numbers) - maximum column: unsigned 32 bits
- maximum file byte size: unsigned 32 bits
Matz.