Bug #9715


ENV data yield ASCII-8BIT encoded strings under Windows with unicode username

Added by thomthom (Thomas Thomassen) over 7 years ago. Updated 7 months ago.

Target version:
ruby -v:
ruby 2.2.0dev (2014-04-07 trunk 45530) [i386-mswin32_100]


My testing scenario:
English Windows, Unicode username: てすと

Home directory: C:\Users\てすと\

The values returned from ENV have different encoding depending on their content. It appear to be OEM encoding label to most value, except when they contain characters not included in the OEM codepage. When they are not, for instance ENV['HOME'] when the username is "てすと" will have ASCII-8BIT.

(I find the "ASCII-8BIT" name for an encoding confusing, as ASCII is 7bit - byte range 0-127)
But it appear that "ASCII-8BIT" is also aliased as "binary"? So Ruby is here returning a binary string when ENV contain byte characters not included in the OEM code page?

Reading the docs for Encoding:

Returns default internal encoding. Strings will be transcoded to the default internal encoding in the following places if the default internal encoding is not nil:
::default_internal is initialized by the source file's internal_encoding or -E option.

This includes ENV - but, even when I run ruby with the -E flag the ENV encoding doesn't change. It's still using the OEM code page - or ASCII-8BIT.
However, regardless of having set -E or not, ENV do appear to return UTF-8 bytes in the strings that contain the Unicode username.

This is one of several areas where I have found -E to have no effect on Ruby's string handling. I understand that some of Ruby's file handling is for backwards compatibility reasons, but I'm finding it difficult to set up a system which can properly handle Unicode files under Windows. Is this deliberate due to backwards compatibility decisions? Or have I simply not found the correct configuration flags for it? To me it appear bugged - inconsistent with what the documentation says. But please enlighten me if I am incorrect. My ideal situation would be for all strings to default to UTF-8.


C:\ruby-220\usr\bin>ruby -E UTF-8:UTF-8 -e "p ENV['ProgramFiles'].encoding"

C:\ruby-220\usr\bin>ruby -E UTF-8:UTF-8 -e "p ENV['ProgramFiles'].bytes"
[67, 58, 92, 80, 114, 111, 103, 114, 97, 109, 32, 70, 105, 108, 101, 115, 32, 40, 120, 56, 54, 41]
C:\ruby-220\usr\bin>ruby -e "p ENV['HOME']"

C:\ruby-220\usr\bin>ruby -e "p ENV['HOME'].encoding"

C:\ruby-220\usr\bin>ruby -e "p ENV['HOME'].bytes"
[67, 58, 47, 85, 115, 101, 114, 115, 47, 227, 129, 166, 227, 129, 153, 227, 129, 168]

C:\ruby-220\usr\bin>ruby -e "p __ENCODING__"

C:\ruby-220\usr\bin>ruby -e "p Encoding.default_internal"

C:\ruby-220\usr\bin>ruby -e "p Encoding.default_external"

C:\ruby-220\usr\bin>ruby -e "p Encoding.find('filesystem')"

C:\ruby-220\usr\bin>ruby -E UTF-8:UTF-8 -e "p ENV['HOME'].encoding"

C:\ruby-220\usr\bin>ruby -E UTF-8:UTF-8 -e "p ENV['HOME'].bytes"
[67, 58, 47, 85, 115, 101, 114, 115, 47, 227, 129, 166, 227, 129, 153, 227, 129, 168]

Related issues

Related to Ruby master - Feature #12650: Use UTF-8 encoding for ENV on WindowsClosedcruby-windowsActions

Also available in: Atom PDF