Project

General

Profile

Actions

Bug #19563

closed

Ripper.tokenize(code).join != code when heredoc and multiline %w[] literal is on the same line

Added by tompng (tomoya ishida) over 1 year ago. Updated over 1 year ago.

Status:
Closed
Assignee:
-
Target version:
-
ruby -v:
ruby 3.3.0dev (2023-03-29T21:57:52Z master 1b06422767) [x86_64-linux]
[ruby-core:113059]

Description

Ripper.tokenize "<<EOF || %w[hello\nEOF\n\n\n\nworld]"
# actual result
["<<EOF", " ", "||", " ", "%w[", "hello", "\n\n\n\n", "EOF\n", "world", "]"]
# expected result
["<<EOF", " ", "||", " ", "%w[", "hello", "\n", "EOF\n", "\n\n\n", "world", "]"]

same for %i[] literal.


Files


Related issues 1 (0 open1 closed)

Related to Ruby master - Bug #19544: Custom quotes inconsistencyClosedActions

Updated by nobu (Nobuyoshi Nakada) over 1 year ago

Similar for Unicode codepoints.

p Ripper.tokenize("<<EOS || %[\\u{4a\n""EOS\n\n\n\n""5a}]").join("")
# "<<EOS || %[EOS\n\n\n\n5a}]"

Updated by nobu (Nobuyoshi Nakada) over 1 year ago

The attached patch fails in an IRB test.
I'm not sure about this prompt transition, should this be fixed as expected?
As the line 004 ends the heredoc C but is inside %W[, isn't ] ok here?

[25/39] TestIRB::TestRubyLex#test_heredoc_with_embexpr = 0.00 s                                       
  1) Failure:
TestIRB::TestRubyLex#test_heredoc_with_embexpr [/Users/nobu/src/ruby/master/src/test/irb/test_ruby_lex.rb:246]:
Expected dynamic prompt:
001:0:":* 
002:0:":* 
003:0:":* 
004:0:":* 
005:0:]:* 
006:0:":* 
007:0:":* 
008:0:":* 
009:0:]:* 
010:0:]:* 
011:0: :> 
012:0: :* 

Actual dynamic prompt:
001:0:":* 
002:0:":* 
003:0:":* 
004:0:]:* 
005:0:]:* 
006:0:":* 
007:0:":* 
008:0:":* 
009:0:]:* 
010:0:]:* 
011:0: :> 
012:0: :* 
.
<["001:0:\":* ",
 "002:0:\":* ",
 "003:0:\":* ",
 "004:0:\":* ",
 "005:0:]:* ",
 "006:0:\":* ",
 "007:0:\":* ",
 "008:0:\":* ",
 "009:0:]:* ",
 "010:0:]:* ",
 "011:0: :> ",
 "012:0: :* "]> expected but was
<["001:0:\":* ",
 "002:0:\":* ",
 "003:0:\":* ",
 "004:0:]:* ",
 "005:0:]:* ",
 "006:0:\":* ",
 "007:0:\":* ",
 "008:0:\":* ",
 "009:0:]:* ",
 "010:0:]:* ",
 "011:0: :> ",
 "012:0: :* "]>.

Updated by tompng (tomoya ishida) over 1 year ago

I've opend a pull request to fix irb's test. Changes expected prompt and target code to avoid this bug in ruby 2.7~3.2.
https://github.com/ruby/irb/pull/558

Updated by tompng (tomoya ishida) over 1 year ago

I merged irb's pull request. I think the failing test is fixed now.

Actions #5

Updated by nobu (Nobuyoshi Nakada) over 1 year ago

  • Status changed from Open to Closed

Applied in changeset git|ac8a16237c727ae2a1446ef6dc810d0e750971fb.


[Bug #19563] Yield words separators per lines

So that newlines across a here-doc terminator will be separated
tokens.

Cf. https://github.com/ruby/irb/pull/558

Actions #7

Updated by nobu (Nobuyoshi Nakada) over 1 year ago

  • Related to Bug #19544: Custom quotes inconsistency added
Actions #8

Updated by nobu (Nobuyoshi Nakada) over 1 year ago

  • Backport changed from 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN to 3.0: REQUIRED, 3.1: REQUIRED, 3.2: REQUIRED
Actions

Also available in: Atom PDF

Like0
Like0Like0Like0Like0Like0Like0Like0Like0