Bug #9766: Add force_encoding option to csv

• Status: Closed
• Assignee: JEG2 (James Gray)
• Target version: 2.2.0
• ruby -v: all
• Backport: 2.0.0: DONE, 2.1: DONE

[dtaniwaki (DAISUKE TANIWAKI)]

Hi there,

I have a trouble when I use csv#generate with encoding 'Shift-JIS' option. I investigated it for a long time and found it is caused by compatibility within "UTF-8" and "Shift-JIS". Since "Shift-JIS" can be converted to UTF-8, a row with UTF-8 strings added to the csv instance makes the encoding of whole rows UTF-8.

Take a look at the code below.
https://github.com/dtaniwaki/ruby/blob/trunk/lib/csv.rb#L1658

Here's the code example.

irb(main):002:0> s = generate(encoding: 'SJIS') do |csv|
    csv << ['あ']
  end
=> ["あ"]

irb(main):003:0> s
=> "あ\n"

irb(main):004:0> s.encoding
=> #<Encoding:UTF-8>

I was intended to make SJIS encoded csv, but the result was UTF-8 csv. I think everyone think it should generate Shift-JIS encoded csv string, so could you consider to merge the change attached to this issue?

The expected result is here.

irb(main):002:0> s = generate(encoding: 'SJIS', force_encoding: true) do |csv|
    csv << ['あ']
  end
=> ["あ"]

irb(main):003:0> s
=> "\x{E381}\x82\n"

irb(main):004:0> s.encoding
=> #<Encoding:Windows-31J>

[dtaniwaki (DAISUKE TANIWAKI)]

The method to reproduce this should be CSV.generate in the code blocks.

I wish it could be merged in all the versions of ruby.

[nobu (Nobuyoshi Nakada)]

• File 0001-csv.rb-honor-encoding-option.patch 0001-csv.rb-honor-encoding-option.patch added
• Tracker changed from Feature to Bug
• Description updated (diff)
• Status changed from Open to Assigned
• Assignee set to JEG2 (James Gray)
• ruby -v set to all
• Backport set to 2.0.0: REQUIRED, 2.1: REQUIRED

I'd rather think it a bug.

[nobu (Nobuyoshi Nakada)]

• Status changed from Assigned to Closed
• % Done changed from 0 to 100

Applied in changeset r46391.

---

csv.rb: honor encoding option

• lib/csv.rb (CSV#<<): honor explicity given encoding. based on
  the patch by DAISUKE TANIWAKI at
  [ruby-core:62113]. [Bug #9766]

[nagachika (Tomoyuki Chikanaga)]

• Backport changed from 2.0.0: REQUIRED, 2.1: REQUIRED to 2.0.0: REQUIRED, 2.1: DONE

r46391 and r46395 were backported into ruby_2_1 branch at r47586.

[usa (Usaku NAKAMURA)]

• Backport changed from 2.0.0: REQUIRED, 2.1: DONE to 2.0.0: DONE, 2.1: DONE

backported into ruby_2_0_0 at r47608.