Bug #22493: Case-insensitive matching is inconsistent in some single-byte encodings

• Status: Open
• Assignee: -
• Target version: -
• Backport: 3.3: REQUIRED, 3.4: REQUIRED, 4.0: REQUIRED

[mame (Yusuke Endoh)]

Each single-byte encoding has two case tables: the lowercase table (used for matching and String#downcase) and the list of case pairs (used for the search optimization and character classes). They disagree in some encodings, so case-insensitive matching gives different results depending on the direction or the search method, and some characters that are not case pairs are matched with each other.

e = Encoding::Windows_1251
big = "\x80".force_encoding(e)    # "Ђ"
small = "\x90".force_encoding(e)  # "ђ"
Regexp.new(big, Regexp::IGNORECASE) =~ small             #=> 0
Regexp.new(small, Regexp::IGNORECASE) =~ big             #=> nil (expected: 0)
big.start_with?(Regexp.new(small, Regexp::IGNORECASE))   #=> true

e = Encoding::Windows_1252
Regexp.new("\xD7".force_encoding(e), Regexp::IGNORECASE) =~ "\xF7".force_encoding(e) #=> 0 (expected: nil; "×" and "÷")

@make_now_just (Hiroya Fujinami) found this and wrote the fix. The fix makes the two tables agree, following the Unicode case mapping of each character. All the changes are:

• ISO-8859-16
  • lowercase table: 0xA4 € no longer maps to 0xA5 „
• Windows-1250
  • lowercase table: 0xD7 × no longer maps to 0xF7 ÷
• Windows-1251
  • case pairs added: Ђ/ђ (0x80/0x90), Ѓ/ѓ (0x81/0x83), Љ/љ (0x8A/0x9A), Њ/њ (0x8C/0x9C), Ќ/ќ (0x8D/0x9D), Ћ/ћ (0x8E/0x9E), Џ/џ (0x8F/0x9F), Ў/ў (0xA1/0xA2), Ј/ј (0xA3/0xBC), Ґ/ґ (0xA5/0xB4), Є/є (0xAA/0xBA), Ї/ї (0xAF/0xBF), І/і (0xB2/0xB3), Ѕ/ѕ (0xBD/0xBE) (they were already in the lowercase table)
• Windows-1252
  • lowercase table: 0xAF ¯ no longer maps to 0xBF ¿, and 0xD7 × no longer maps to 0xF7 ÷
• Windows-1253
  • lowercase table: 0xB6 ¶ no longer maps to 0xDC ά
  • case pairs: ¶/ά (0xB6/0xDC) replaced with Ά/ά (0xA2/0xDC)
• Windows-1254
  • case pairs added: Š/š (0x8A/0x9A), Œ/œ (0x8C/0x9C), Ÿ/ÿ (0x9F/0xFF) (they were already in the lowercase table)

The wrong entries seem to come from copying the tables of another encoding: windows_1252.c was made from windows_1251.c (0xAF/0xBF and 0xD7/0xF7 are case pairs in Windows-1251), Windows-1253 from ISO-8859-7 (where 0xB6 is Ά), and Windows-1254 from ISO-8859-9 (which has no characters in 0x80-0x9F).

This does not change the Greek final sigma ς and µ, or the Turkish İ, which also differ from Unicode case folding.

PR: https://github.com/ruby/ruby/pull/19644

@duerst (Martin Dürst) @ima1zumi (Mari Imaizumi) Could you review these changes?

[mame (Yusuke Endoh)]

• Description updated (diff)
• Backport changed from 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: UNKNOWN to 3.3: REQUIRED, 3.4: REQUIRED, 4.0: REQUIRED