Project

General

Profile

Actions

Bug #19455

closed

Ruby 3.2: wrong Regexp encoding with non-ASCII comments

Added by janosch-x (Janosch Müller) over 1 year ago. Updated over 1 year ago.

Status:
Closed
Assignee:
-
Target version:
-
ruby -v:
ruby 3.2.0 (2022-12-25 revision a528908271)
[ruby-core:112533]

Description

comments and comment groups don't trigger the correct Regexp#encoding on Ruby 3.2 anymore:

# ruby 3.1
/#a/x.encoding          # => #<Encoding:US-ASCII> # OK
/(?#a)/.encoding        # => #<Encoding:US-ASCII> # OK
/#ü/x.encoding          # => #<Encoding:UTF-8>    # OK
/(?#ü)/.encoding        # => #<Encoding:UTF-8>    # OK

# ruby 3.2
/#a/x.encoding          # => #<Encoding:US-ASCII> # OK
/(?#a)/.encoding        # => #<Encoding:US-ASCII> # OK
/#ü/x.encoding          # => #<Encoding:US-ASCII> # BUG
/(?#ü)/.encoding        # => #<Encoding:US-ASCII> # BUG

/#ü/x.inspect           # => "/#\\xC3\\xBC/x"
/(?#ü)/.inspect         # => "/(?#\\xC3\\xBC)/"

# bug is hidden if there are non-ascii chars outside comments
/ü#ü/x.encoding         # => #<Encoding:UTF-8>
/ü(?#ü)/.encoding       # => #<Encoding:UTF-8>

i think these changes might be the cause: https://github.com/ruby/ruby/commit/ec3542229b29ec93062e9d90e877ea29d3c19472#diff-c3675fa319803b2f5a775defa40694edb9a761baa3a54fa78e1fdef8f918cc7cR2837-R2890

@jeremyevans0 (Jeremy Evans) JFYI

Updated by jeremyevans0 (Jeremy Evans) over 1 year ago

I'm not sure that this a bug. If all non-comment characters considered in the regexp are in the US-ASCII range, it seems reasonable for US-ASCII to be used as the regexp encoding. I'll add this ticket to the next developer meeting and see what other committers think.

Updated by mame (Yusuke Endoh) over 1 year ago

@janosch-x (Janosch Müller) Do you have any specific problem with this change? For example, a string that used to match no longer matches, or vice versa.

Updated by janosch-x (Janosch Müller) over 1 year ago

i don't have a problem with this myself and the matching behavior is not affected as far as i can tell.

notable behavioral differences are:

  • /#ü/x.source == '#ü' used to be true but is now false
    • this might break some tests or metaprogramming (not very likely IMO)
  • /#{/#ü/x.source}/ now raises ArgumentError (invalid multibyte character)

Updated by mame (Yusuke Endoh) over 1 year ago

Discussed at the dev meeting. @matz (Yukihiro Matsumoto) said he would prefer 3.1 behavior if possible (but not high priority). @nobu (Nobuyoshi Nakada) said he would take a look.

Actions #6

Updated by jeremyevans (Jeremy Evans) over 1 year ago

  • Status changed from Open to Closed

Applied in changeset git|a8ba1ddd78544b4bda749051d44f7b2a8a0ec5ff.


Use UTF-8 encoding for literal extended regexps with UTF-8 characters in comments

Fixes [Bug #19455]

Actions #7

Updated by nagachika (Tomoyuki Chikanaga) over 1 year ago

  • Backport changed from 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN to 3.0: DONTNEED, 3.1: DONTNEED, 3.2: REQUIRED

Updated by nagachika (Tomoyuki Chikanaga) over 1 year ago

  • Backport changed from 3.0: DONTNEED, 3.1: DONTNEED, 3.2: REQUIRED to 3.0: DONTNEED, 3.1: DONTNEED, 3.2: DONE

ruby_3_2 be09d77b966c7bcc77957927f16cefe66b365495 merged revision(s) a8ba1ddd78544b4bda749051d44f7b2a8a0ec5ff.

Actions

Also available in: Atom PDF

Like1
Like0Like0Like0Like0Like0Like0Like0Like0