Bug #11485
closedRipper parser generates nil tokens for ignored_nl event during parse errors
Description
This seems to be a regression in the Ripper parser where the on_ignored_nl
event is passed a nil value for the token parameter in the event when a syntax error occurs. I'm not entirely sure of the details, but this is the most minimal reproduction example I could produce:
require 'ripper'
class Parser < Ripper
def on_ignored_nl(*args) p args end
def on_parse_error(msg) $stderr.puts "ERROR! #{msg}" end
end
Parser.new(DATA, 'stdin').parse
__END__
something # comment
...
Note the "..." which causes the parse error, but the comment on the line before causes the ignored_nl event to get called with nil
as the argument. I'm not sure if it's specific to the ...
token or if there are other ways to trigger this parse error. I tried with other invalid syntaxes and could not reproduce with those.
Running the above produces the following output (in 2.0.0p481 and 2.3.0-dev respective):
~$ ruby -v
ruby 2.0.0p481 (2014-05-08 revision 45883) [universal.x86_64-darwin14]
~$ ruby ripper-bug-2.2.0.rb
ERROR! syntax error, unexpected ..., expecting end-of-input
~$ ruby -v
ruby 2.3.0dev (2015-08-24 trunk 51672) [x86_64-darwin14]
~$ ruby ripper-bug-2.2.0.rb
[nil]
ERROR! syntax error, unexpected ..., expecting end-of-input
I would not expect the token parameter to ever be nil
, since a token event should always have a token to pass in if it is triggered. Note that this event typically produces a "\n" token value (and still does in normal cases). The expectation should be to either provide a valid token or not trigger the event at all.
(This behavior is present in 2.2.0p95 as well.)
Files
Updated by jeremyevans0 (Jeremy Evans) over 5 years ago
This bug is still present on master. Attached is a patch that fixes it, by not calling dispatch_delayed_token
when ripper is used if the dispatched value is nil
.
Updated by nobu (Nobuyoshi Nakada) over 5 years ago
- Description updated (diff)
- Backport changed from 2.0.0: DONTNEED, 2.1: UNKNOWN, 2.2: UNKNOWN to 2.4: REQUIRED, 2.5: REQUIRED, 2.6: REQUIRED
Thank you for the patch, somehow I've missed this issue at all.
dispatch_delayed_token
is an empty expression unless ripper, so it is useless inside #ifndef RIPPER
.
And delayed tokens should not be nil
generally.
Updated by nobu (Nobuyoshi Nakada) over 5 years ago
[ruby-core:<unknown>]
This seems not having sent to the ML, and it happens sometimes, but I'm not sure why.
Updated by nobu (Nobuyoshi Nakada) over 5 years ago
- Status changed from Open to Closed
Applied in changeset git|f19e048d244c709e2990ddbb0f986e8b51357bd2.
Do not dispatch a nil token in ripper
As a comment token includes the newline, so delayed newline token
just follows it should not be dispatched. [Bug #11485]
Co-Authored-By: Jeremy Evans code@jeremyevans.net