Bug #22542
open
ASCII-incompatible default_internal mislabels ENV and Dir strings without converting them
Description
Encoding.default_internal accepts ASCII-incompatible encodings like UTF-16LE. IO transcodes into it correctly, but ENV and Dir return the original UTF-8 bytes relabeled as UTF-16LE.
$ LC_ALL=C.UTF-8 FOO=hello ruby -E UTF-8:UTF-16LE --disable-gems -e 'str = ENV["FOO"]; p str.bytes; p str.encoding; p str.valid_encoding?'
[104, 101, 108, 108, 111]
#<Encoding:UTF-16LE>
false
The cause is the shortcut in rb_external_str_new_with_enc for ASCII-only input:
/* ASCII compatible, and ASCII only string, no conversion in
* default_internal */
if ((eidx == rb_ascii8bit_encindex()) ||
(eidx == rb_usascii_encindex()) ||
(rb_enc_asciicompat(eenc) && !search_nonascii(ptr, ptr + len))) {
return rb_enc_str_new(ptr, len, ienc);
}
This skips the conversion and tags the bytes with ienc, which is only valid if ienc is also ASCII-compatible. IO uses its own transcoding path, so it isn't affected.
Related to #18338, which is the same kind of mislabeling caused by an ASCII-incompatible default_external (via the filesystem encoding).
Should we even support non-ascii-compatible internal and external encodings? It doesn't seem like they've ever worked and so I can't imagine a demand for using them.
No data to display