https://redmine.ruby-lang.org/https://redmine.ruby-lang.org/favicon.ico?17113305112012-04-03T07:53:11ZRuby Issue Tracking SystemRuby master - Bug #6246: 1.9.3-p125 intermittent segfaulthttps://redmine.ruby-lang.org/issues/6246?journal_id=256192012-04-03T07:53:11ZAnonymous
<ul><li><strong>File</strong> <a href="/attachments/2566">noname</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/2566/noname">noname</a> added</li></ul><p>On Tue, Apr 03, 2012 at 04:22:04AM +0900, jshow (Jodi Showers) wrote:</p>
<blockquote>
<p>Issue <a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: 1.9.3-p125 intermittent segfault (Closed)" href="https://redmine.ruby-lang.org/issues/6246">#6246</a> has been reported by jshow (Jodi Showers).</p>
<hr>
<p>Bug <a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: 1.9.3-p125 intermittent segfault (Closed)" href="https://redmine.ruby-lang.org/issues/6246">#6246</a>: 1.9.3-p125 intermittent segfault<br>
<a href="https://bugs.ruby-lang.org/issues/6246" class="external">https://bugs.ruby-lang.org/issues/6246</a></p>
<p>Author: jshow (Jodi Showers)<br>
Status: Open<br>
Priority: Normal<br>
Assignee:<br>
Category:<br>
Target version:<br>
ruby -v: ruby 1.9.3p125 (2012-02-16 revision 34643) [x86_64-linux]</p>
<p>I've seen this behavior in 1.9.3p0 and 1.9.3p125</p>
<p>Intermittently, a rails 3.2 rake task will break while compiling assets - about 1 in 10 runs will segfault. Our code is 10K's of lines of proprietary code. I am working on a simplified version you can use.</p>
<p>RAILS_ENV=new_staging bundle exec rake assets:precompile --trace</p>
<p>command: <a href="https://gist.github.com/2254457" class="external">https://gist.github.com/2254457</a> (stacktrace and gem list)</p>
<p>backtrace : <a href="https://gist.github.com/2286260" class="external">https://gist.github.com/2286260</a></p>
<p>Operating System : Debian Squeeze 6.0.4<br>
Ruby installed through RVM 1.11.3</p>
</blockquote>
<p>Can you either provide the application so that we can try to reproduce<br>
the issue, somehow reduce the list of moving parts, or run valgrind against<br>
your app?</p>
<p>The size of your gem list is daunting, it could be anything in there.</p>
<p>If you need, I can give you the valgrind command to use. :-)</p>
<p>--<br>
Aaron Patterson<br>
<a href="http://tenderlovemaking.com/" class="external">http://tenderlovemaking.com/</a></p> Ruby master - Bug #6246: 1.9.3-p125 intermittent segfaulthttps://redmine.ruby-lang.org/issues/6246?journal_id=256232012-04-03T10:29:59Zjshow (Jodi Showers)jodi@homestars.com
<ul></ul><p>yes please - never used valgrind but will learn.</p>
<p>thanks</p> Ruby master - Bug #6246: 1.9.3-p125 intermittent segfaulthttps://redmine.ruby-lang.org/issues/6246?journal_id=256242012-04-03T11:15:13Zjshow (Jodi Showers)jodi@homestars.com
<ul></ul><p>Aaron, I ran a valgrind as follows:</p>
<p>valgrind /home/deploy/.rvm/rubies/ruby-1.9.3-p125/bin/ruby -Iext:test /home/deploy/.rvm/gems/ruby-1.9.3-p125@boogala/bin/bundle exec rake RAILS_ENV=new_staging assets:precompile</p>
<p>the results are here: <a href="https://gist.github.com/2288812" class="external">https://gist.github.com/2288812</a></p>
<p>Long before the actual startup of the rake task (rails boot) there were a great number of valgrind outputs -</p>
<p>==11285== Use of uninitialised value of size 8<br>
==11285== at 0x4EA3203: gc_mark_children (gc.c:1867)<br>
==11285== by 0x4EA491C: mark_current_machine_context (gc.c:1650)<br>
==11285== by 0x4EA4A3D: gc_marks (gc.c:2474)<br>
==11285== by 0x4EA5DBF: rb_newobj (gc.c:2195)<br>
==11285== by 0x4F46115: str_alloc (string.c:373)<br>
==11285== by 0x4FA0132: vm_call0 (vm_eval.c:79)<br>
==11285== by 0x4FA0CC9: rb_funcall (vm_eval.c:236)<br>
==11285== by 0x4ED926E: rb_obj_alloc (object.c:1603)<br>
==11285== by 0x4EDA835: rb_obj_dup (object.c:315)<br>
==11285== by 0xDE7034D: path_initialize (pathname.c:41)<br>
==11285== by 0x4FA0132: vm_call0 (vm_eval.c:79)<br>
==11285== by 0x4ED931F: rb_class_new_instance (object.c:1636)<br>
==11285==</p>
<p>I can send all of these if you feel they'll help. None of these were outputted after the actual rake task began.</p> Ruby master - Bug #6246: 1.9.3-p125 intermittent segfaulthttps://redmine.ruby-lang.org/issues/6246?journal_id=256422012-04-04T02:23:17ZAnonymous
<ul><li><strong>File</strong> <a href="/attachments/2576">noname</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/2576/noname">noname</a> added</li></ul><p>On Tue, Apr 03, 2012 at 11:15:14AM +0900, jshow (Jodi Showers) wrote:</p>
<blockquote>
<p>Issue <a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: 1.9.3-p125 intermittent segfault (Closed)" href="https://redmine.ruby-lang.org/issues/6246">#6246</a> has been updated by jshow (Jodi Showers).</p>
<p>Aaron, I ran a valgrind as follows:</p>
<p>valgrind /home/deploy/.rvm/rubies/ruby-1.9.3-p125/bin/ruby -Iext:test /home/deploy/.rvm/gems/ruby-1.9.3-p125@boogala/bin/bundle exec rake RAILS_ENV=new_staging assets:precompile</p>
</blockquote>
<p>Try running valgrind with these flags:</p>
<p>--num-callers=50 --error-limit=no --partial-loads-ok=yes --undef-value-errors=no</p>
<p>Post all of the errors in a gist if you don't mind.</p>
<p>Can you also try your code against edge 1.9.3? Nari-san has fixed a<br>
segv error in that branch. If we're lucky it's the same issue, and we<br>
don't have to debug anymore. ;)</p>
<p>--<br>
Aaron Patterson<br>
<a href="http://tenderlovemaking.com/" class="external">http://tenderlovemaking.com/</a></p> Ruby master - Bug #6246: 1.9.3-p125 intermittent segfaulthttps://redmine.ruby-lang.org/issues/6246?journal_id=256442012-04-04T10:18:25Zjshow (Jodi Showers)jodi@homestars.com
<ul></ul><p>Aaron,</p>
<p>attached is the full output from valgrind using your options - <a href="https://gist.github.com/2296847" class="external">https://gist.github.com/2296847</a></p>
<p>I'm presently looking for the edge branch - not seeing it. Do you mean head?</p>
<p>thanks</p> Ruby master - Bug #6246: 1.9.3-p125 intermittent segfaulthttps://redmine.ruby-lang.org/issues/6246?journal_id=256472012-04-04T11:53:14ZAnonymous
<ul><li><strong>File</strong> <a href="/attachments/2577">noname</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/2577/noname">noname</a> added</li></ul><p>On Wed, Apr 04, 2012 at 10:18:26AM +0900, jshow (Jodi Showers) wrote:</p>
<blockquote>
<p>Issue <a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: 1.9.3-p125 intermittent segfault (Closed)" href="https://redmine.ruby-lang.org/issues/6246">#6246</a> has been updated by jshow (Jodi Showers).</p>
<p>Aaron,</p>
<p>attached is the full output from valgrind using your options - <a href="https://gist.github.com/2296847" class="external">https://gist.github.com/2296847</a></p>
<p>I'm presently looking for the edge branch - not seeing it. Do you mean head?</p>
</blockquote>
<p>I guess? I mean the ruby_1_9_3 branch:</p>
<p><a href="https://github.com/ruby/ruby/tree/ruby_1_9_3" class="external">https://github.com/ruby/ruby/tree/ruby_1_9_3</a></p>
<p>:-)</p>
<p>--<br>
Aaron Patterson<br>
<a href="http://tenderlovemaking.com/" class="external">http://tenderlovemaking.com/</a></p> Ruby master - Bug #6246: 1.9.3-p125 intermittent segfaulthttps://redmine.ruby-lang.org/issues/6246?journal_id=256482012-04-04T11:53:14ZAnonymous
<ul><li><strong>File</strong> <a href="/attachments/2578">noname</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/2578/noname">noname</a> added</li></ul><p>On Wed, Apr 04, 2012 at 10:18:26AM +0900, jshow (Jodi Showers) wrote:</p>
<blockquote>
<p>Issue <a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: 1.9.3-p125 intermittent segfault (Closed)" href="https://redmine.ruby-lang.org/issues/6246">#6246</a> has been updated by jshow (Jodi Showers).</p>
<p>Aaron,</p>
<p>attached is the full output from valgrind using your options - <a href="https://gist.github.com/2296847" class="external">https://gist.github.com/2296847</a></p>
</blockquote>
<p>I'm thinking that your segv is happening in a subshell (or not the<br>
process on which you're running valgrind).</p>
<p>Are there hours when I can meet you on IRC? It might be faster than<br>
mail exchange here. :-)</p>
<p>--<br>
Aaron Patterson<br>
<a href="http://tenderlovemaking.com/" class="external">http://tenderlovemaking.com/</a></p> Ruby master - Bug #6246: 1.9.3-p125 intermittent segfaulthttps://redmine.ruby-lang.org/issues/6246?journal_id=256552012-04-04T22:45:49Zjshow (Jodi Showers)jodi@homestars.com
<ul></ul><p>I'm in the north american eastern time zone</p>
<p>I can meet today if you can</p>
<p>11am eastern is doable for me - just over an hour away</p>
<p>In the meantime I'll start trying ruby-head</p>
<p>much thanks Aaron</p> Ruby master - Bug #6246: 1.9.3-p125 intermittent segfaulthttps://redmine.ruby-lang.org/issues/6246?journal_id=256562012-04-05T00:08:27Zjshow (Jodi Showers)jodi@homestars.com
<ul></ul><p>Aaron, I've updated this gist with a run that uses a more direct command line - which I think was hiding some of the valgrind output</p>
<p>valgrind --num-callers=50 --error-limit=no --partial-loads-ok=yes --undef-value-errors=no rake RAILS_ENV=new_staging assets:precompile</p>
<p><a href="https://gist.github.com/2296847" class="external">https://gist.github.com/2296847</a></p>
<p>still it seems we're not getting much value from valgrind - it's reporting 0 errors from 0 contexts</p>
<p>I'm re-running now with -v and --leak-check=full for more</p> Ruby master - Bug #6246: 1.9.3-p125 intermittent segfaulthttps://redmine.ruby-lang.org/issues/6246?journal_id=256582012-04-05T00:36:52Zjshow (Jodi Showers)jodi@homestars.com
<ul><li><strong>File</strong> <a href="/attachments/2580">valgrind_full.txt.zip</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/2580/valgrind_full.txt.zip">valgrind_full.txt.zip</a> added</li></ul><p>attached is a more verbose valgrind</p>
<p>$ valgrind --num-callers=50 --error-limit=no --partial-loads-ok=yes --undef-value-errors=no -v --leak-check=full rake RAILS_ENV=new_staging assets:precompile</p> Ruby master - Bug #6246: 1.9.3-p125 intermittent segfaulthttps://redmine.ruby-lang.org/issues/6246?journal_id=257052012-04-08T01:52:32Zhectormalot (Dennis de Reus)dennis@densoft.nl
<ul></ul><p>I have a similar problem when creating a new rails (3.2.3) app. It segfaults on the bundle install step. I think this is related, so I won't create a new bugreport.</p>
<p>Ruby -v: ruby 1.9.3p125 (2012-02-16 revision 34643) [x86_64-darwin11.3.0]</p>
<p>I found this when I created a new rails (3.2.3) project. Ruby gave a segfault when running bundle install. The details are in this gist: <a href="https://gist.github.com/2330262" class="external">https://gist.github.com/2330262</a>. I'm not sure when the issue arises exactly, but I've also had trouble running the tests from active_admin using ruby 1.9.3-p125, going back to 1.9.2 didn't gave any segmentation faults.</p>
<p>I'm sorry for not giving a more detailed bugreport, but I'm not that well versed in ruby yet. I hope that other with the same problem can add to this report.</p> Ruby master - Bug #6246: 1.9.3-p125 intermittent segfaulthttps://redmine.ruby-lang.org/issues/6246?journal_id=257482012-04-09T22:49:11Zjshow (Jodi Showers)jodi@homestars.com
<ul></ul><p>Dennis, looking at your ruby trace, it looks like you've run into something else</p>
<p>It appears that you are on osx, and there are some documented (try google) about ssl and ruby - seems that the ssl libs shipped with osx are not compatible with the latest builds of ruby</p> Ruby master - Bug #6246: 1.9.3-p125 intermittent segfaulthttps://redmine.ruby-lang.org/issues/6246?journal_id=259562012-04-17T21:00:35Zmame (Yusuke Endoh)mame@ruby-lang.org
<ul><li><strong>Status</strong> changed from <i>Open</i> to <i>Assigned</i></li><li><strong>Assignee</strong> set to <i>authorNari (Narihiro Nakamura)</i></li></ul><p>Hello,</p>
<p>Perhaps I could reproduce this issue with small example:</p>
<p>class Foo<br>
def Foo.shutdown(*args)<br>
p Bar.inspect.size<br>
end<br>
Bar = (0..300).map do |x|<br>
x = x.to_s<br>
ObjectSpace.define_finalizer(x, Foo.method(:shutdown))<br>
x<br>
end<br>
end</p>
<p>This code causes segfault in my system.<br>
The number (300) may vary depending on a system.</p>
<p>This looks like a bug of GC finalizer.<br>
A finalizer invoked at process termination, seems to refer<br>
GC'ed object, which lead to segfault. I guess.</p>
<p>In fact, the "logging" gem, which is included in OP's trace,<br>
seems to use define_finalizer as a substitute for at_exit.</p>
<p><a href="https://github.com/TwP/logging/blob/master/lib/logging.rb#L552" class="external">https://github.com/TwP/logging/blob/master/lib/logging.rb#L552</a></p>
<p>Nari, please check it.</p>
<p>--<br>
Yusuke Endoh <a href="mailto:mame@tsg.ne.jp" class="email">mame@tsg.ne.jp</a></p> Ruby master - Bug #6246: 1.9.3-p125 intermittent segfaulthttps://redmine.ruby-lang.org/issues/6246?journal_id=261222012-04-24T02:14:00Zjshow (Jodi Showers)jodi@homestars.com
<ul></ul><p>Yusuke Endoh,</p>
<p>I have tried your example, but unfortunately couldn't cause a crash. I experimented with various values of as high as 3000.</p>
<p>I would appreciate whatever effort can be applied - I am literally stuck as 125 crashes so often it's unusable.</p>
<p>If I can provide more details I would be happy to do so</p>
<p>OT, but I don't receive any emails from redmine - I'm watching this issue, and have the general mail settings turned on. Hard to follow - I presently come back to visit regularly.</p>
<p>thanks.</p> Ruby master - Bug #6246: 1.9.3-p125 intermittent segfaulthttps://redmine.ruby-lang.org/issues/6246?journal_id=261282012-04-24T03:53:21Znormalperson (Eric Wong)normalperson@yhbt.net
<ul></ul><p>"jshow (Jodi Showers)" <a href="mailto:jodi@homestars.com" class="email">jodi@homestars.com</a> wrote:</p>
<blockquote>
<p>I would appreciate whatever effort can be applied - I am literally<br>
stuck as 125 crashes so often it's unusable.</p>
</blockquote>
<p>As Aaron mentioned, did you get a chance to try the ruby_1_9_3 branch?<br>
Now that Ruby 1.9.3-p194 is out, maybe it's easier to give that a try.<br>
Again, if we're lucky we it's the same issue we encountered with<br>
Nokogiri and fixed in 1.9.3-p194.</p>
<blockquote>
<p>OT, but I don't receive any emails from redmine - I'm watching this<br>
issue, and have the general mail settings turned on. Hard to follow -<br>
I presently come back to visit regularly.</p>
</blockquote>
<p>I'm not sure if this redmine is configured emails users directly, but<br>
subscribing to the ruby-core list works.</p> Ruby master - Bug #6246: 1.9.3-p125 intermittent segfaulthttps://redmine.ruby-lang.org/issues/6246?journal_id=261592012-04-24T20:23:21Zmame (Yusuke Endoh)mame@ruby-lang.org
<ul></ul><p>Hello,</p>
<p>2012/4/24, jshow (Jodi Showers) <a href="mailto:jodi@homestars.com" class="email">jodi@homestars.com</a>:</p>
<blockquote>
<p>I have tried your example, but unfortunately couldn't cause a crash. I<br>
experimented with various values of as high as 3000.</p>
</blockquote>
<p>It is difficult to reproduce a problem about GC with a small code.<br>
Indeed, my example does not cause SEGV on my another system.</p>
<p>Can you try the following patch?<br>
This is an ad-hoc patch that stops finalizer call at exit.<br>
SEGV should disappear if my guess is right.<br>
If it still causes SEGV, my example is another issue.</p>
<p>diff --git a/eval.c b/eval.c<br>
index 4dddcaa..b2358f8 100644<br>
--- a/eval.c<br>
+++ b/eval.c<br>
@@ -98,7 +98,6 @@ ruby_finalize_1(void)<br>
{<br>
ruby_sig_finalize();<br>
GET_THREAD()->errinfo = Qnil;</p>
<ul>
<li>rb_gc_call_finalizer_at_exit();<br>
}</li>
</ul>
<p>void</p>
<p>--<br>
Yusuke Endoh <a href="mailto:mame@tsg.ne.jp" class="email">mame@tsg.ne.jp</a></p> Ruby master - Bug #6246: 1.9.3-p125 intermittent segfaulthttps://redmine.ruby-lang.org/issues/6246?journal_id=261722012-04-25T04:31:40Zjshow (Jodi Showers)jodi@homestars.com
<ul></ul><p>I've been using p194 for about a day now - no crashes yet</p>
<p>I'll notify you here over the next while</p>
<p>if the crashing resumes, I'll try the patch Yusuke.</p>
<p>much thanks to you both.</p> Ruby master - Bug #6246: 1.9.3-p125 intermittent segfaulthttps://redmine.ruby-lang.org/issues/6246?journal_id=261752012-04-25T05:53:26Zcout (Paul Brannan)curlypaul924@gmail.com
<ul></ul><blockquote>
</blockquote>
<p>/home/deploy/.rvm/rubies/ruby-1.9.3-p125/lib/libruby.so.1.9(rb_gc_call_finalizer_at_exit+0x219) [0x7f9f37246359] gc.c:3044</p>
<blockquote>
<p>/home/deploy/.rvm/rubies/ruby-1.9.3-p125/lib/libruby.so.1.9(ruby_cleanup+0x1ec) [0x7f9f37233fac] eval.c:101<br>
/home/deploy/.rvm/rubies/ruby-1.9.3-p125/lib/libruby.so.1.9(ruby_run_node+0x25) [0x7f9f37234205] eval.c:244</p>
</blockquote>
<p>So we're cleaning up, and:</p>
<blockquote>
<p>#0 st_lookup (table=0x0, key=792, value=0x7fffffffc858) at st.c:330<br>
#1 0x00007ffff7b1629f in search_method (recv=,<br>
mid=792) at vm_method.c:374<br>
#2 rb_method_entry_get_without_cache (recv=,<br>
mid=792) at vm_method.c:393</p>
</blockquote>
<p>Someone called a method on an object that had already been GC'd.</p>
<p>That someone is in the logging gem:</p>
<p>ObjectSpace.define_finalizer self, Logging.method(:shutdown)</p>
<p>Ruby makes no guarantees about the order in which finalizers are called<br>
at exit. This can be problematic, because the finalizer may need to<br>
perform operations on objects that have already been GC'd.</p>
<p>I have avoided this before in C code by explicitly checking for a GC'd<br>
object. But since the logging gem is ruby (not C), I don't know what it<br>
should do.</p>
<p>IMO, it should have one finalizer per appender, rather than a single<br>
finalizer that cleans up all the appenders. But I'm not sure that would<br>
fix the crash.</p>
<p>Paul</p> Ruby master - Bug #6246: 1.9.3-p125 intermittent segfaulthttps://redmine.ruby-lang.org/issues/6246?journal_id=264302012-05-04T04:14:51Zjshow (Jodi Showers)jodi@homestars.com
<ul></ul><p>Since my last post, I've continued to see the crash</p>
<p>But today I came across this bug report for the logging gem</p>
<p><a href="https://github.com/TwP/logging/issues/30" class="external">https://github.com/TwP/logging/issues/30</a></p>
<p>which may very well be the source</p>
<p>I'll update my experiences here</p> Ruby master - Bug #6246: 1.9.3-p125 intermittent segfaulthttps://redmine.ruby-lang.org/issues/6246?journal_id=295702012-09-20T14:33:14ZauthorNari (Narihiro Nakamura)authorNari@gmail.com
<ul><li><strong>Status</strong> changed from <i>Assigned</i> to <i>Closed</i></li></ul><p>Hi!</p>
<p>I think ruby-1.9.3-p194 fixed this bug.<br>
if you see the crash with ruby-1.9.3-p194, please reopen this ticket.</p>
<p>Thanks.</p>