Bug #19563
closedRipper.tokenize(code).join != code when heredoc and multiline %w[] literal is on the same line
Description
Ripper.tokenize "<<EOF || %w[hello\nEOF\n\n\n\nworld]"
# actual result
["<<EOF", " ", "||", " ", "%w[", "hello", "\n\n\n\n", "EOF\n", "world", "]"]
# expected result
["<<EOF", " ", "||", " ", "%w[", "hello", "\n", "EOF\n", "\n\n\n", "world", "]"]
same for %i[]
literal.
Files
Updated by nobu (Nobuyoshi Nakada) over 1 year ago
Similar for Unicode codepoints.
p Ripper.tokenize("<<EOS || %[\\u{4a\n""EOS\n\n\n\n""5a}]").join("")
# "<<EOS || %[EOS\n\n\n\n5a}]"
Updated by nobu (Nobuyoshi Nakada) over 1 year ago
- File 0001-Bug-19563-Yield-words-separators-per-lines.patch 0001-Bug-19563-Yield-words-separators-per-lines.patch added
The attached patch fails in an IRB test.
I'm not sure about this prompt transition, should this be fixed as expected?
As the line 004 ends the heredoc C
but is inside %W[
, isn't ]
ok here?
[25/39] TestIRB::TestRubyLex#test_heredoc_with_embexpr = 0.00 s
1) Failure:
TestIRB::TestRubyLex#test_heredoc_with_embexpr [/Users/nobu/src/ruby/master/src/test/irb/test_ruby_lex.rb:246]:
Expected dynamic prompt:
001:0:":*
002:0:":*
003:0:":*
004:0:":*
005:0:]:*
006:0:":*
007:0:":*
008:0:":*
009:0:]:*
010:0:]:*
011:0: :>
012:0: :*
Actual dynamic prompt:
001:0:":*
002:0:":*
003:0:":*
004:0:]:*
005:0:]:*
006:0:":*
007:0:":*
008:0:":*
009:0:]:*
010:0:]:*
011:0: :>
012:0: :*
.
<["001:0:\":* ",
"002:0:\":* ",
"003:0:\":* ",
"004:0:\":* ",
"005:0:]:* ",
"006:0:\":* ",
"007:0:\":* ",
"008:0:\":* ",
"009:0:]:* ",
"010:0:]:* ",
"011:0: :> ",
"012:0: :* "]> expected but was
<["001:0:\":* ",
"002:0:\":* ",
"003:0:\":* ",
"004:0:]:* ",
"005:0:]:* ",
"006:0:\":* ",
"007:0:\":* ",
"008:0:\":* ",
"009:0:]:* ",
"010:0:]:* ",
"011:0: :> ",
"012:0: :* "]>.
Updated by tompng (tomoya ishida) over 1 year ago
I've opend a pull request to fix irb's test. Changes expected prompt and target code to avoid this bug in ruby 2.7~3.2.
https://github.com/ruby/irb/pull/558
Updated by tompng (tomoya ishida) over 1 year ago
I merged irb's pull request. I think the failing test is fixed now.
Updated by nobu (Nobuyoshi Nakada) over 1 year ago
- Status changed from Open to Closed
Applied in changeset git|ac8a16237c727ae2a1446ef6dc810d0e750971fb.
[Bug #19563] Yield words separators per lines
So that newlines across a here-doc terminator will be separated
tokens.
Updated by nagachika (Tomoyuki Chikanaga) over 1 year ago
MEMO: 4af9bd52cbb8cff7d149a8565012ab1153a4b5b1 is the follow-up commit for ac8a16237c727ae2a1446ef6dc810d0e750971fb.
Updated by nobu (Nobuyoshi Nakada) over 1 year ago
- Related to Bug #19544: Custom quotes inconsistency added
Updated by nobu (Nobuyoshi Nakada) over 1 year ago
- Backport changed from 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN to 3.0: REQUIRED, 3.1: REQUIRED, 3.2: REQUIRED