Bug #21756: Ripper fails to parse pathological heredoc

• Status: Open
• Assignee: -
• Target version: -
• ruby -v: ruby 4.0.0dev (2025-11-28T10:49:46Z master dcb9e17f46) +PRISM [x86_64-linux]
• Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN

[Earlopain (Earlopain _)]

# test.rb
pp <<-A, %w[j\
i
A
j]

$ RBENV_VERSION=ruby-dev ruby -rripper -ve "Ripper.lex(File.read('test.rb'), raise_errors: true)"
ruby 4.0.0dev (2025-11-28T10:49:46Z master dcb9e17f46) +PRISM [x86_64-linux]
/home/user/.rbenv/versions/ruby-dev/lib/ruby/4.0.0+0/ripper/lexer.rb:180:in 'Ripper::Lexer#parse': syntax error, unexpected literal content, expecting ' ' (SyntaxError)
        from /home/user/.rbenv/versions/ruby-dev/lib/ruby/4.0.0+0/ripper/lexer.rb:155:in 'Ripper::Lexer#lex'
        from /home/user/.rbenv/versions/ruby-dev/lib/ruby/4.0.0+0/ripper/lexer.rb:52:in 'Ripper.lex'
        from -e:1:in '<main>'

Both prism and parse.y are able to interpret this code:

$ RBENV_VERSION=ruby-dev ruby test.rb
"i\n"
["j\n", "j"]
$ RBENV_VERSION=ruby-dev ruby --parser=parse.y test.rb
"i\n"
["j\n" + "j"]

[tompng (tomoya ishida)]

Looks like there is one more separate issue here. Prism and parse.y execution result differs.

p <<-A, %w[a\
A
b c]
# Prism:   ["a\n", "b", "c"]
# parse.y: ["a\nb", "c"]

Changing %w to %W, Prism and parse.y are the same

p <<-A, %W[a\
A
b c]
# => Both ["a\nb", "c"]

[Earlopain (Earlopain _)]

Ah, yeah. It's not expecting disjointed strings in arrays where interpolation is not allowed. That doesn't hold with line continuations and heredocs. I'm working on a fix, should not be so difficult.

Edit: Sorry, I totally missed the discrepancy in the output even in my original report. , vs + didn't register for me.

[Earlopain (Earlopain _)]

For the prism part of the issue https://github.com/ruby/prism/pull/3778