Bug #19924: Character literal escaped \xFF stops parsing

• Status: Closed
• Assignee: -
• Target version: -
• Backport: 3.0: REQUIRED, 3.1: REQUIRED, 3.2: DONE

[kddnewton (Kevin Newton)]

When you have a character literal, you can have an escaped character. For things that don't actually need escaping, this works out to just returning the character itself. For example:

?\d # => "d"

This works for every codepoint in ASCII-8bit except 0xFF. When that byte is hit, the entire parser will stop parsing and just return nil.

eval(<<~RUBY)
  # encoding: ascii-8bit

  p ?\\\xFF

  puts "this will never be parsed"
RUBY

If you try to parse it with RubyVM::AbstractSyntaxTree, it's clear the parser just stops:

RubyVM::AbstractSyntaxTree.parse(<<~RUBY)
  # encoding: ascii-8bit

  p ?\\\xFF

  puts "this will never be parsed"
RUBY

# => (SCOPE@1:0-2:1 tbl: [] args: nil body: (VCALL@2:0-2:1 :p))

[nobu (Nobuyoshi Nakada)]

• Status changed from Open to Closed

Applied in changeset git|17b0643392749f45b7aacb64fc1c1bd704d42b4c.

---

[Bug #19924] Source code should be unsigned char stream

Use peekc or nextc to fetch the next character, instead of reading
from lex.pcur directly, for compilers that plain char is signed.

[nobu (Nobuyoshi Nakada)]

• Backport changed from 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN to 3.0: REQUIRED, 3.1: REQUIRED, 3.2: REQUIRED

[nagachika (Tomoyuki Chikanaga)]

• Backport changed from 3.0: REQUIRED, 3.1: REQUIRED, 3.2: REQUIRED to 3.0: REQUIRED, 3.1: REQUIRED, 3.2: DONE

ruby_3_2 482e1f573ee45ef562cc6d218ba26d44660f8e0a merged revision(s) 17b0643392749f45b7aacb64fc1c1bd704d42b4c.