Bug #6266: encoding related exception with recent integrated psych

• Status: Closed
• Assignee: tenderlovemaking (Aaron Patterson)
• Target version: 1.9.3
• ruby -v: ruby 1.9.3p176 (2012-04-04 revision 35238) [i386-mingw32]

[jonforums (Jon Forums)]

Newer versions of Psych integrated into both ruby_1_9_3 and trunk create exceptions if the yaml file isn't perceived to be correctly encoded.

While I discovered this in my code that uses the 'logging' gem on Win7 32bit, this test snippet shows the same failure on both ruby_1_9_3@r35238 and trunk@r35229:

hello.yml uses LF-only line endings

C:\temp>type hello.yml
:greeting: hello yaml

yaml_test.rb uses LF-only

C:\temp>type yaml_test.rb
require 'yaml'
puts "YAML engine: #{YAML::ENGINE.yamler} #{'v'+Psych::VERSION if defined?(Psych)}"

fix: 'rb'

fix: 'r:utf-8'

fix: Encoding.default_external = Encoding::UTF_8 if defined?(Encoding)

fix: chcp 65001 in cmd.exe before running script

fix: run script with ruby -Eutf-8

File.open('hello.yml', 'r') do |f|
puts YAML.load_stream(f)
end

IBM437 codepage is a typical default for the cmd.exe shell

C:\temp>chcp
Active code page: 437

C:\temp>ruby -v yaml_test.rb
ruby 1.9.3p176 (2012-04-04 revision 35238) [i386-mingw32]
YAML engine: psych v1.3.1
C:/ruby193/lib/ruby/1.9.1/psych.rb:206:in parse': YAML file must be UTF-8, UTF-16LE, or UTF-16BE, not IBM437 (ArgumentError) from C:/ruby193/lib/ruby/1.9.1/psych.rb:206:in parse_stream'
from C:/ruby193/lib/ruby/1.9.1/psych.rb:289:in load_stream' from yaml_test.rb:10:in block in '
from yaml_test.rb:9:in open' from yaml_test.rb:9:in '

older versions of integrated psych do not cause exceptions

C:\temp>chcp
Active code page: 437

C:\temp>ruby -v yaml_test.rb
tcs-ruby 1.9.3p134 (2012-02-19, TCS patched 2012-02-25) [i386-mingw32]
YAML engine: psych v1.2.2
{:greeting=>"hello yaml"}

I've not tried on Arch with bash.

From the test_yaml.rb comments you can see some of the workarounds. None of these are good options because they require changes to either a users code/yaml, gem code that uses the psych library, or the ruby invocation command line.

For both trunk and ruby_1_9_3, I'd prefer psych implement a fallback strategy that Just Works combined with an informative warning rather than a hard exception.

[jonforums (Jon Forums)]

On Arch (3.2.14-1-ARCH) the sample also exceptions when I change my locale to en_US.iso88591 from en_US.utf8.

But on trunk it doesn't matter anymore :)

$ ruby -v yaml_test.rb
ruby 2.0.0dev (2012-04-07 trunk 35247) [i686-linux]
YAML engine: psych v1.3.1
{:greeting=>"hello yaml"}

...and trunk looks good on Win7 32bit

C:\temp>chcp
Active code page: 437

C:\temp>ruby -v yaml_test.rb
ruby 2.0.0dev (2012-04-07 trunk 35247) [i386-mingw32]
YAML engine: psych v1.3.1
{:greeting=>"hello yaml"}

Speedy...thanks! I'll fetch ruby_1_9_3 later and double check.

• File noname noname added

On Sat, Apr 07, 2012 at 06:31:25AM +0900, jonforums (Jon Forums) wrote:

Issue #6266 has been updated by jonforums (Jon Forums).

On Arch (3.2.14-1-ARCH) the sample also exceptions when I change my locale to en_US.iso88591 from en_US.utf8.

But on trunk it doesn't matter anymore :)

$ ruby -v yaml_test.rb
ruby 2.0.0dev (2012-04-07 trunk 35247) [i686-linux]
YAML engine: psych v1.3.1
{:greeting=>"hello yaml"}

...and trunk looks good on Win7 32bit

C:\temp>chcp
Active code page: 437

C:\temp>ruby -v yaml_test.rb
ruby 2.0.0dev (2012-04-07 trunk 35247) [i386-mingw32]
YAML engine: psych v1.3.1
{:greeting=>"hello yaml"}

Speedy...thanks! I'll fetch ruby_1_9_3 later and double check.

I need to backport the fix to ruby_1_9_3 first. :-)

I'll put together the patches for the 1.9.3 branch and open a ticket to
backport. Thanks for testing!

--
Aaron Patterson
http://tenderlovemaking.com/

[mame (Yusuke Endoh)]

• Status changed from Open to Assigned

[jonforums (Jon Forums)]

r35245 fixed this issue, and r35251 backported to ruby_1_9_3

[mame (Yusuke Endoh)]

• Status changed from Assigned to Closed

Okay, closing. Thanks!

--
Yusuke Endoh [e-mail]