https://redmine.ruby-lang.org/https://redmine.ruby-lang.org/favicon.ico?17113305112009-03-04T04:30:17ZRuby Issue Tracking SystemRuby 1.8 - Bug #1240: parser bug in 1.8.7 and 1.9.1p0https://redmine.ruby-lang.org/issues/1240?journal_id=34252009-03-04T04:30:17Zzenspider (Ryan Davis)
<ul></ul><p>=begin</p>
<p>On Mar 2, 2009, at 22:30 , Thomer Gil wrote:</p>
<blockquote>
<p>Bug <a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: parser bug in 1.8.7 and 1.9.1p0 (Closed)" href="https://redmine.ruby-lang.org/issues/1240">#1240</a>: parser bug in 1.8.7 and 1.9.1p0<br>
<a href="http://redmine.ruby-lang.org/issues/show/1240" class="external">http://redmine.ruby-lang.org/issues/show/1240</a></p>
<p>Author: Thomer Gil<br>
Status: Open, Priority: Normal<br>
Category: core<br>
ruby -v: ruby 1.9.1p0 (2009-01-30 revision 21907) [i686-linux]</p>
<a name="ruby-parser-accepts-first-line-not-the-second"></a>
<h1 >ruby parser accepts first line, not the second.<a href="#ruby-parser-accepts-first-line-not-the-second" class="wiki-anchor">¶</a></h1>
<p>x y { "#{}".z { } }<br>
x y { "#{}".z do end }</p>
</blockquote>
<p>1.8.6 as well... this also appears to be the minimimal reproduction.<br>
Remove anything and it parses. How confounding.</p>
<p>If/When this gets fixed, please respond with the patch or commit # so<br>
I can track it. Thanks.</p>
<p>=end</p> Ruby 1.8 - Bug #1240: parser bug in 1.8.7 and 1.9.1p0https://redmine.ruby-lang.org/issues/1240?journal_id=34262009-03-04T04:35:16Zshyouhei (Shyouhei Urabe)shyouhei@ruby-lang.org
<ul><li><strong>Assignee</strong> set to <i>matz (Yukihiro Matsumoto)</i></li></ul><p>=begin</p>
<p>=end</p> Ruby 1.8 - Bug #1240: parser bug in 1.8.7 and 1.9.1p0https://redmine.ruby-lang.org/issues/1240?journal_id=34272009-03-04T06:44:01Zdaz (Dave B)
<ul></ul><p>=begin<br>
As there's no context for this "bug", I've tried to provide some.</p>
<p>The parser takes different routes depending on what has been seen,<br>
so context is important.</p>
<p>def x(args=nil)<br>
p [:x_args, args]<br>
block_given? and (puts '--> x blk'; yield)<br>
end</p>
<p>def y(args=nil)<br>
p [:y_args, args]<br>
block_given? and (puts '--> y blk'; yield)<br>
end</p>
<p>class String<br>
def z(args=nil)<br>
p [:z_args, args]<br>
block_given? and (puts '--> z blk'; yield)<br>
end<br>
end</p>
<p>x y { "#{}".z { "GIGO\n".display } }</p>
<a name="x-y-z-do-GIGOndisplay-end-parser-fail"></a>
<h1 >x y { "#{}".z do "GIGO\n".display end } # parser fail<a href="#x-y-z-do-GIGOndisplay-end-parser-fail" class="wiki-anchor">¶</a></h1>
<a name="Contrasting-with-"></a>
<h1 >Contrasting with ...<a href="#Contrasting-with-" class="wiki-anchor">¶</a></h1>
<p>p "#{}".z { "GIGO\n".display }<br>
puts '---'<br>
p "#{}".z do "GIGO\n".display end</p>
<a name="-what-was-do-end-in-the-original-report"></a>
<h1 >... what was "do end" in the original report<a href="#-what-was-do-end-in-the-original-report" class="wiki-anchor">¶</a></h1>
<a name="expected-to-bind-to"></a>
<h1 >expected to bind to?<a href="#expected-to-bind-to" class="wiki-anchor">¶</a></h1>
<p>daz</p>
<p>=end</p> Ruby 1.8 - Bug #1240: parser bug in 1.8.7 and 1.9.1p0https://redmine.ruby-lang.org/issues/1240?journal_id=34362009-03-04T19:29:21Zzenspider (Ryan Davis)
<ul></ul><p>=begin</p>
<p>On Mar 3, 2009, at 13:42 , Dave B wrote:</p>
<blockquote>
<p>As there's no context for this "bug", I've tried to provide some.</p>
</blockquote>
<p>This isn't a "bug", it is a bug, as I think I've shown below.</p>
<blockquote>
<a name="Contrasting-with-"></a>
<h1 >Contrasting with ...<a href="#Contrasting-with-" class="wiki-anchor">¶</a></h1>
<p>p "#{}".z { "GIGO\n".display }<br>
puts '---'<br>
p "#{}".z do "GIGO\n".display end</p>
<a name="-what-was-do-end-in-the-original-report"></a>
<h1 >... what was "do end" in the original report<a href="#-what-was-do-end-in-the-original-report" class="wiki-anchor">¶</a></h1>
<a name="expected-to-bind-to"></a>
<h1 >expected to bind to?<a href="#expected-to-bind-to" class="wiki-anchor">¶</a></h1>
</blockquote>
<p>I find this view more clear:</p>
<blockquote>
<p>% echo 'p "#{}".z do "GIGO\n".display end' | parse_tree_show<br>
s(:iter,<br>
s(:call,<br>
nil,<br>
:p,<br>
s(:arglist, s(:call, s(:dstr, "", s(:evstr)), :z, s(:arglist)))),<br>
nil,<br>
s(:call, s(:str, "GIGO\n"), :display, s(:arglist)))</p>
</blockquote>
<p>so the do/end (aka iter) is bound to the call to #p.</p>
<p>From the original report, minus the empty interpolation:</p>
<blockquote>
<p>% echo 'x y { "".z do end }' | parse_tree_show<br>
s(:call,<br>
nil,<br>
:x,<br>
s(:arglist,<br>
s(:iter,<br>
s(:call, nil, :y, s(:arglist)),<br>
nil,<br>
s(:iter, s(:call, s(:str, ""), :z, s(:arglist)), nil)))</p>
</blockquote>
<p>the interpolation would replace the s(:str, "") with:</p>
<blockquote>
<p>% echo '"#{}"' | parse_tree_show<br>
s(:dstr, "", s(:evstr))</p>
</blockquote>
<p>So really there isn't any reason why this shouldn't be parseable. If<br>
this isn't solved by the weekend I'll take a whack at it with<br>
ruby_parser and translate/backport from there. I <em>hate</em> these<br>
codepaths tho and I haven't gotten around to rewriting the string<br>
stack yet, so it isn't fun.</p>
<p>=end</p> Ruby 1.8 - Bug #1240: parser bug in 1.8.7 and 1.9.1p0https://redmine.ruby-lang.org/issues/1240?journal_id=99382010-04-08T00:31:21Zmame (Yusuke Endoh)mame@ruby-lang.org
<ul><li><strong>Assignee</strong> changed from <i>matz (Yukihiro Matsumoto)</i> to <i>mame (Yusuke Endoh)</i></li></ul><p>=begin<br>
Hi,</p>
<p>2009/3/3 Thomer Gil <a href="mailto:redmine@ruby-lang.org" class="email">redmine@ruby-lang.org</a>:</p>
<blockquote>
<a name="ruby-parser-accepts-first-line-not-the-second"></a>
<h1 >ruby parser accepts first line, not the second.<a href="#ruby-parser-accepts-first-line-not-the-second" class="wiki-anchor">¶</a></h1>
<p>x y { "#{}".z { } }<br>
x y { "#{}".z do end }</p>
</blockquote>
<p>Interesting. I found similar bugs:</p>
<p>while ("#{}".foo { } ); end # ok<br>
while ("#{}".foo do end); end # parse error</p>
<p>while ->{ 1.times { } }.call; done # ok<br>
while ->{ 1.times do end }.call; done # parse error</p>
<p>The cause is mishandling of cond_stack and cmdarg_stack.<br>
When "#{}" is parsed, the parser pushes a value into these stacks once<br>
(at #{ (tSTRING_DBEG)), and pops twice (at '}' and action of the rule<br>
`string_content'.)</p>
<p>In addition, the parser forgets to push at ->{ (tLAMBEG).</p>
<p>I think it is better for not parser but lexer to push into the stacks<br>
at tSTRING_DBEG. See my patch.</p>
<p>BTW, I also noticed that block call with <code>do' keyword does not work in </code>until' condition:</p>
<p>until begin 1.times { } end do end # ok<br>
until begin 1.times do end end do end # parse error<br>
~~<br>
until if true then 1.times { } end do end # ok<br>
until if true then 1.times do end end do end # parse error<br>
~~<br>
until until true do 1.times { } end do end # ok<br>
until until true do 1.times do end end do end # parse error<br>
~~<br>
until class Foo; 1.times { } ; end do end # ok<br>
until class Foo; 1.times do end; end do end # parse error<br>
~~<br>
until case; when true; 1.times { } ; end do end # ok<br>
until case; when true; 1.times do end; end do end # parse error<br>
~~</p>
<p>This is because the underlined <code>do's are not considered as block call but beginning of </code>until' body.</p>
<p>Although this is confusing a little and can be actually fixed, the fix<br>
needs many COND_PUSH(0)/COND_POP(), which may decrease performance and<br>
code maintenability. In addition, writing such a long and complex<br>
condition directly is absolutely bad (even insane) style.<br>
So, we should accept the above behaviors as spec, I think.</p>
<p>Here is a patch to fix the issue Thomer reported.<br>
I'll commit it unless there is objection.</p>
<p>diff --git a/parse.y b/parse.y<br>
index 340a825..f234ae3 100644<br>
--- a/parse.y<br>
+++ b/parse.y<br>
@@ -4033,14 +4033,10 @@ string_content : tSTRING_CONTENT<br>
$$ = lex_strterm;<br>
lex_strterm = 0;<br>
lex_state = EXPR_BEG;</p>
<ul>
<li>
<pre><code> COND_PUSH(0);
</code></pre>
</li>
<li>
<pre><code> CMDARG_PUSH(0);
}
compstmt '}'
{
lex_strterm = $<node>2;
</code></pre>
</li>
<li>
<pre><code> COND_LEXPOP();
</code></pre>
</li>
<li>
<pre><code> CMDARG_LEXPOP();
/*%%%*/
if ($3) $3->flags &= ~NODE_FL_NEWLINE;
$$ = new_evstr($3);
</code></pre>
</li>
</ul>
<p>@@ -5873,6 +5869,8 @@ parser_parse_string(struct parser_params *parser, NODE *quote)<br>
pushback(c);<br>
return tSTRING_DVAR;<br>
case '{':</p>
<ul>
<li>
<pre><code> COND_PUSH(0);
</code></pre>
</li>
<li>
<pre><code> CMDARG_PUSH(0);
return tSTRING_DBEG;
</code></pre>
}<br>
tokadd('#');<br>
@@ -6070,6 +6068,8 @@ parser_here_document(struct parser_params *parser, NODE *here)<br>
pushback(c);<br>
return tSTRING_DVAR;<br>
case '{':</li>
<li>
<pre><code> COND_PUSH(0);
</code></pre>
</li>
<li>
<pre><code> CMDARG_PUSH(0);
return tSTRING_DBEG;
}
tokadd('#');
</code></pre>
</li>
</ul>
<p>@@ -7314,6 +7314,8 @@ parser_yylex(struct parser_params *parser)<br>
lex_state = EXPR_BEG;<br>
lpar_beg = 0;<br>
--paren_nest;</p>
<ul>
<li>
<pre><code> COND_PUSH(0);
</code></pre>
</li>
<li>
<pre><code> CMDARG_PUSH(0);
return tLAMBEG;
</code></pre>
}<br>
if (IS_ARG() || lex_state == EXPR_END)</li>
</ul>
<p>--<br>
Yusuke ENDOH <a href="mailto:mame@tsg.ne.jp" class="email">mame@tsg.ne.jp</a><br>
=end</p> Ruby 1.8 - Bug #1240: parser bug in 1.8.7 and 1.9.1p0https://redmine.ruby-lang.org/issues/1240?journal_id=99522010-04-08T20:11:15Zmame (Yusuke Endoh)mame@ruby-lang.org
<ul></ul><p>=begin<br>
Hi,</p>
<p>2010/4/8 Caleb Clausen <a href="mailto:vikkous@gmail.com" class="email">vikkous@gmail.com</a>:</p>
<blockquote>
<p>Yusuke Endoh wrote:</p>
<blockquote>
<p>BTW, I also noticed that block call with <code>do' keyword does not work in </code>until' condition:</p>
<p>?until begin 1.times { } ? ?end do end # ok<br>
?until begin 1.times do end end do end # parse error<br>
? ? ? ? ? ? ? ? ? ? ?~~<br>
[snip... more like that]</p>
<p>This is because the underlined <code>do's are not considered as block call but beginning of </code>until' body.</p>
<p>Although this is confusing a little and can be actually fixed, the fix<br>
needs many COND_PUSH(0)/COND_POP(), which may decrease performance and<br>
code maintenability. ?In addition, writing such a long and complex<br>
condition directly is absolutely bad (even insane) style.<br>
So, we should accept the above behaviors as spec, I think.</p>
</blockquote>
<p>Wait.... please don't harden any bugs into the specification of the<br>
language. If you want to say that it's too obscure/too hard to fix,<br>
that's fine. WONTFIX is ok, under the circumstances. If you want to<br>
say it's implementation dependent behavior, that's at least<br>
acceptable. But let's not pretend that what's clearly a bug is<br>
intended behavior. (Best of all would be to fix the bug, but I'm not<br>
really asking for that.)</p>
</blockquote>
<p>You are right. I guess it be considered as WONTFIX.</p>
<blockquote>
<p>My own lexer & parser were able to parse all the examples in this bug<br>
report correctly and I did not need to change anything.</p>
</blockquote>
<p>Great.</p>
<p>BTW, how about the following? :-)</p>
<a name="ok"></a>
<h1 >ok<a href="#ok" class="wiki-anchor">¶</a></h1>
<p>while (((((((((((((((((((((((((((((((false)))))))))))))))))))))))))))))))<br>
do; end</p>
<a name="parse-error"></a>
<h1 >parse error<a href="#parse-error" class="wiki-anchor">¶</a></h1>
<p>while ((((((((((((((((((((((((((((((((false))))))))))))))))))))))))))))))))<br>
do; end</p>
<p>If your CPU is 64 bit, please run after 32 '(' and ')' are added :-p</p>
<p>IMHO, do_cond (omittable `do' keyword in the part of while/until/for<br>
constructs) brings confusion but little benefit to the Ruby syntax.<br>
How about removal of do_cond in 2.0?</p>
<p>--<br>
Yusuke ENDOH <a href="mailto:mame@tsg.ne.jp" class="email">mame@tsg.ne.jp</a></p>
<p>=end</p> Ruby 1.8 - Bug #1240: parser bug in 1.8.7 and 1.9.1p0https://redmine.ruby-lang.org/issues/1240?journal_id=99932010-04-09T02:10:44Zcoatl (caleb clausen)
<ul></ul><p>=begin<br>
On 4/8/10, Yusuke ENDOH <a href="mailto:mame@tsg.ne.jp" class="email">mame@tsg.ne.jp</a> wrote:</p>
<blockquote>
<p>2010/4/8 Caleb Clausen <a href="mailto:vikkous@gmail.com" class="email">vikkous@gmail.com</a>:</p>
<blockquote>
<p>Wait.... please don't harden any bugs into the specification of the<br>
language. If you want to say that it's too obscure/too hard to fix,<br>
that's fine. WONTFIX is ok, under the circumstances. If you want to<br>
say it's implementation dependent behavior, that's at least<br>
acceptable. But let's not pretend that what's clearly a bug is<br>
intended behavior. (Best of all would be to fix the bug, but I'm not<br>
really asking for that.)</p>
</blockquote>
<p>You are right. I guess it be considered as WONTFIX.</p>
</blockquote>
<p>Thank you.</p>
<blockquote>
<p>BTW, how about the following? :-)</p>
<a name="ok"></a>
<h1 >ok<a href="#ok" class="wiki-anchor">¶</a></h1>
<p>while (((((((((((((((((((((((((((((((false)))))))))))))))))))))))))))))))<br>
do; end</p>
<a name="parse-error"></a>
<h1 >parse error<a href="#parse-error" class="wiki-anchor">¶</a></h1>
<p>while ((((((((((((((((((((((((((((((((false))))))))))))))))))))))))))))))))<br>
do; end</p>
</blockquote>
<p>Ooooo! Both of those don't parse right in redparse. But not for the reason you seem to expect; it's objecting to the do on its own line. That is to say, this also doesn't parse:</p>
<p>while false<br>
do; end</p>
<p>Thanks for the bug. I'll fix it right away.</p>
<p>Taking out the newline before do makes them parse right again. (For MRI too, tho.)</p>
<p>Incidentally, all the ruby versions I have (1.8.6, 1.9.0, 1.9.1) also object to do after a newline. I guess this was added (relatively) recently?</p>
<blockquote>
<p>If your CPU is 64 bit, please run after 32 '(' and ')' are added :-p</p>
</blockquote>
<p>This was on a 32 bit cpu. I wouldn't anticipate any problems with any number of nested parentheses. (Until, that is, you run out of memory.)</p>
<blockquote>
<p>IMHO, do_cond (omittable `do' keyword in the part of while/until/for<br>
constructs) brings confusion but little benefit to the Ruby syntax.<br>
How about removal of do_cond in 2.0?</p>
</blockquote>
<p>Hear, hear! I agree wholeheartedly. (Unless there's someone out there who actually likes this? Speak now or forever hold your peace.)<br>
=end</p> Ruby 1.8 - Bug #1240: parser bug in 1.8.7 and 1.9.1p0https://redmine.ruby-lang.org/issues/1240?journal_id=133782010-09-14T16:37:39Zshyouhei (Shyouhei Urabe)shyouhei@ruby-lang.org
<ul><li><strong>Status</strong> changed from <i>Open</i> to <i>Assigned</i></li></ul><p>=begin</p>
<p>=end</p> Ruby 1.8 - Bug #1240: parser bug in 1.8.7 and 1.9.1p0https://redmine.ruby-lang.org/issues/1240?journal_id=188292011-07-05T13:01:50Zmame (Yusuke Endoh)mame@ruby-lang.org
<ul><li><strong>Status</strong> changed from <i>Assigned</i> to <i>Closed</i></li></ul><p>The original issue that OP reported was already fixed at r27359.<br>
The commit caused a regression <a href="https://blade.ruby-lang.org/ruby-core/29579">[ruby-core:29579]</a>, and nobu fixed<br>
it at r27387.</p>
<p>So, closed. Thanks.</p>
<p>--<br>
Yusuke Endoh <a href="mailto:mame@tsg.ne.jp" class="email">mame@tsg.ne.jp</a></p>