Bug #4272: rb_enc_str_new() causes segmentfault when using threads in parallel

• Status: Closed
• Assignee: -
• Target version: -
• ruby -v: 1.9.2-p136

[ibc (Iñaki Baz Castillo)]

=begin
Hi, I attach a custom C extension which shows how rb_enc_str_new() causes segmentfault when running multiple threads. The bug is reproducible in 1.9.2-p0, 1.9.2-p136 and 1.9 today's snapshot (2011-01-12).

In order to test the bug compile the extension and use the included test_bug.rb script passing a long number (iterations) as argument (1000, 10000...).

~# ruby extconf.rb && make && ruby test_bug.rb 5000

The provided C extension is very simple and just contains two methods:

TestEncoding::Tester#test_rb_str_new => returns a string encoded in ASCII-8BIT (generated with "rb_str_new").
TestEncoding::Tester#test_rb_enc_str_new => returns a string encoded in UTF-8 (generated with "rb_enc_str_new").

When running threads in parallel with the second method, after some iterations Ruby crashses (as the provided test_bug.rb shows).
=end

[ibc (Iñaki Baz Castillo)]

=begin
I've checked the provided test script under ruby-1.9.1-p376 and ruby-1.9.1-p430 and it DOES NOT crash.
=end

[ibc (Iñaki Baz Castillo)]

• File test_bug.rb test_bug.rb added

=begin

=end

[ibc (Iñaki Baz Castillo)]

=begin
(please use the latest test_bug.rb attachment, it seems is not possible to remove an attachment in this Redmine)
=end

[ibc (Iñaki Baz Castillo)]

=begin
It's interesting the fact that (using my attached C extension) the following code doesn't crash (I mean under ruby 1.9.2-xxxx as under 1.9.1 it never crashes):

times.times do |num|
tester.test_rb_str_new
threads << Thread.new { }
end

but the following codes DO crash:

times.times do |num|
tester.test_rb_str_new.inspect
threads << Thread.new { }
end

times.times do |num|
a = tester.test_rb_str_new
threads << Thread.new { }
end

=end

[ibc (Iñaki Baz Castillo)]

=begin
The segmentfault just depends on the version of Ruby running the ruby code, and not on the version of Ruby compiling the C extension. This is:

a) C extension compiled with Ruby 1.9.1 and test_bug.rb executed with Ruby 1.9.2 => CRASH

b) C extension compiled with Ruby 1.9.2 and test_bug.rb executed with Ruby 1.9.1 => doesn't crash
=end

[ibc (Iñaki Baz Castillo)]

=begin
The bug also occurs if I use rb_ascii8bit_encoding(), rb_usascii_encoding() or rb_locale_encoding() so it must be a bug in rb_enc_str_new() function.
=end

[nagachika (Tomoyuki Chikanaga)]

=begin
Hi,

In my environment script just like the following cause "[BUG] rb_register_sigaltstack. malloc error".
It's because all Thread object is stored in `threads' variable and cannot be GC'ed.
Could you show us your error messages?

threads = []
300000.times do |i|
puts i
threads << Thread.new{}
end
threads.each {|t| t.join}
=end

[matz (Yukihiro Matsumoto)]

=begin
Hi,

I could not reproduce your problem, but your script allocates
thousands of threads by default. Each thread consumes at least 1M of
memory, so that allocating thousands of threads could cause memory
starvation, which may be the source of your problem.

						matz.

In message "Re: [ruby-core:34399] [Ruby 1.9-Bug#4272][Open] rb_enc_str_new() causes segmentfault when using threads in parallel"
on Wed, 12 Jan 2011 11:04:21 +0900, Iñaki Baz Castillo [e-mail] writes:

|Hi, I attach a custom C extension which shows how rb_enc_str_new() causes segmentfault when running multiple threads. The bug is reproducible in 1.9.2-p0, 1.9.2-p136 and 1.9 today's snapshot (2011-01-12).
|
|In order to test the bug compile the extension and use the included test_bug.rb script passing a long number (iterations) as argument (1000, 10000...).
|
| ~# ruby extconf.rb && make && ruby test_bug.rb 5000
|
|The provided C extension is very simple and just contains two methods:
|
| TestEncoding::Tester#test_rb_str_new => returns a string encoded in ASCII-8BIT (generated with "rb_str_new").
| TestEncoding::Tester#test_rb_enc_str_new => returns a string encoded in UTF-8 (generated with "rb_enc_str_new").
|
|When running threads in parallel with the second method, after some iterations Ruby crashses (as the provided test_bug.rb shows).

=end

[naruse (Yui NARUSE)]

=begin
I can reproduce this on ruby 1.9.3dev (2011-01-12 trunk 30517) [x86_64-freebsd8.2].
Anyway what your environment? (why you didn't copy ruby -v ?)
=end

[nagachika (Tomoyuki Chikanaga)]

=begin
Hi,
I can reproduce SEGV on Mac OS X 10.6.6.
Please add the following line in test_encoding.c

#include <ruby/encoding.h>

Without that header, rb_utf8_encoding() has no prototype declaration and its return value is estimated to be int.

=end

[ibc (Iñaki Baz Castillo)]

• File test_bug2.rb test_bug2.rb added

=begin
Yukihiro:

I could not reproduce your problem

Do you mean under Ruby 1.9.2 64 bits? If so please use a bigger argument (ruby test_bug.rb 100000). I've realized that depending on the machine it crashes sooner or later.

but your script allocates thousands of threads by default. Each thread consumes at least 1M of memory, so
that allocating thousands of threads could cause memory starvation, which may be the source of your problem.

The segmentfault just occurs when using rb_enc_str_new(). If you edit test_bug.rb and remove/comment the first iteration with "tester.test_rb_str_new" the segmentfault also occurs. And it never occurs if you just leave the iteration with "tester.test_rb_str_new".

And more important, I also reproduce always the segmentfault using a fixed pool of 20 threads (in EventMachine). And also in this modified test script (max 20 threads are created) you can run by loading my attached C extension (I attach now test_bug2.rb with this code):

 require "./test_encoding"


 times = ( ( ARGV[0] and ARGV[0].to_i > 0 ) ? ARGV[0].to_i : 5000 )
 threads = []
 num_threads = 20
 tester = TestEncoding::Tester.new

 times.times do |num|
   puts "iteration [#{num}]"
   str = tester.test_rb_enc_str_new
   threads << Thread.new { }  unless num_threads == 0
   num_threads -= 1
 end

Using Ruby 1.9.2- produces different segmentfault (randomly, but it crashes always in my machines):

1. Crash on Thread.initialize:

iteration [1981]
iteration [1982]
iteration [1983]
iteration [1984]
test_bug.rb:12: [BUG] Segmentation fault
ruby 1.9.2p0 (2010-08-18 revision 29036) [x86_64-linux]

-- control frame ----------
c:0008 p:---- s:0025 b:0025 l:000024 d:000024 CFUNC :initialize
c:0007 p:---- s:0023 b:0023 l:000022 d:000022 CFUNC :new
c:0006 p:0063 s:0020 b:0019 l:001f08 d:000018 BLOCK test_bug.rb:12
c:0005 p:---- s:0015 b:0015 l:000014 d:000014 FINISH
c:0004 p:---- s:0013 b:0013 l:000012 d:000012 CFUNC :times
c:0003 p:0121 s:0010 b:0010 l:001f08 d:000968 EVAL test_bug.rb:9
c:0002 p:---- s:0004 b:0004 l:000003 d:000003 FINISH
c:0001 p:0000 s:0002 b:0002 l:001f08 d:001f08 TOP

-- Ruby level backtrace information ----------------------------------------
test_bug.rb:9:in <main>' test_bug.rb:9:in times'
test_bug.rb:12:in block in <main>' test_bug.rb:12:in new'
test_bug.rb:12:in `initialize'

-- C level backtrace information -------------------------------------------
/usr/lib/libruby-1.9.1.so.1.9(rb_vm_bugreport+0x5f) [0x7f10c71c658f]
/usr/lib/libruby-1.9.1.so.1.9(+0x5a7be) [0x7f10c70cd7be]
/usr/lib/libruby-1.9.1.so.1.9(rb_bug+0xa6) [0x7f10c70ce136]
/usr/lib/libruby-1.9.1.so.1.9(+0xf4a75) [0x7f10c7167a75]
/lib/libpthread.so.0(+0xfb40) [0x7f10c6e65b40]
/usr/lib/libruby-1.9.1.so.1.9(+0x6edc8) [0x7f10c70e1dc8]
/usr/lib/libruby-1.9.1.so.1.9(rb_thread_mark+0x4c) [0x7f10c71b1b8c]
/usr/lib/libruby-1.9.1.so.1.9(+0x6e87a) [0x7f10c70e187a]
/usr/lib/libruby-1.9.1.so.1.9(+0x13ea89) [0x7f10c71b1a89]
/usr/lib/libruby-1.9.1.so.1.9(st_foreach+0xb4) [0x7f10c7170284]
/usr/lib/libruby-1.9.1.so.1.9(rb_vm_mark+0x29) [0x7f10c71b1f09]
/usr/lib/libruby-1.9.1.so.1.9(+0x6e87a) [0x7f10c70e187a]
/usr/lib/libruby-1.9.1.so.1.9(+0x6fd98) [0x7f10c70e2d98]
/usr/lib/libruby-1.9.1.so.1.9(rb_newobj+0x47) [0x7f10c70e37c7]
/usr/lib/libruby-1.9.1.so.1.9(rb_data_typed_object_alloc+0x26) [0x7f10c70e3886]
/usr/lib/libruby-1.9.1.so.1.9(+0x13f97a) [0x7f10c71b297a]
/usr/lib/libruby-1.9.1.so.1.9(rb_vm_make_proc+0x56) [0x7f10c71b44e6]
/usr/lib/libruby-1.9.1.so.1.9(+0x64be3) [0x7f10c70d7be3]
/usr/lib/libruby-1.9.1.so.1.9(+0x154a92) [0x7f10c71c7a92]
/usr/lib/libruby-1.9.1.so.1.9(+0x149ca5) [0x7f10c71bcca5]
/usr/lib/libruby-1.9.1.so.1.9(+0x154cc8) [0x7f10c71c7cc8]
/usr/lib/libruby-1.9.1.so.1.9(+0x14c93e) [0x7f10c71bf93e]
/usr/lib/libruby-1.9.1.so.1.9(+0x143f61) [0x7f10c71b6f61]
/usr/lib/libruby-1.9.1.so.1.9(+0x148b44) [0x7f10c71bbb44]
/usr/lib/libruby-1.9.1.so.1.9(rb_yield+0x238) [0x7f10c71c2408]
/usr/lib/libruby-1.9.1.so.1.9(+0x96571) [0x7f10c7109571]
/usr/lib/libruby-1.9.1.so.1.9(+0x14c93e) [0x7f10c71bf93e]
/usr/lib/libruby-1.9.1.so.1.9(+0x143f61) [0x7f10c71b6f61]
/usr/lib/libruby-1.9.1.so.1.9(+0x148b44) [0x7f10c71bbb44]
/usr/lib/libruby-1.9.1.so.1.9(rb_iseq_eval_main+0xb1) [0x7f10c71c2c81]
/usr/lib/libruby-1.9.1.so.1.9(+0x5ecca) [0x7f10c70d1cca]
/usr/lib/libruby-1.9.1.so.1.9(ruby_exec_node+0x1d) [0x7f10c70d22ed]
/usr/lib/libruby-1.9.1.so.1.9(ruby_run_node+0x1e) [0x7f10c70d3e3e]
ruby(main+0x4b) [0x40092b]
/lib/libc.so.6(__libc_start_main+0xfe) [0x7f10c6229d8e]
ruby() [0x400819]

2. Crash on Thread#new:

iteration [1780]
iteration [1781]
iteration [1782]
iteration [1783]
test_bug.rb:12: [BUG] Segmentation fault
ruby 1.9.2p0 (2010-08-18 revision 29036) [x86_64-linux]

-- control frame ----------
c:0007 p:---- s:0023 b:0023 l:000022 d:000022 CFUNC :new
c:0006 p:0063 s:0020 b:0019 l:000048 d:000018 BLOCK test_bug.rb:12
c:0005 p:---- s:0015 b:0015 l:000014 d:000014 FINISH
c:0004 p:---- s:0013 b:0013 l:000012 d:000012 CFUNC :times
c:0003 p:0121 s:0010 b:0010 l:000048 d:0011b8 EVAL test_bug.rb:9
c:0002 p:---- s:0004 b:0004 l:000003 d:000003 FINISH
c:0001 p:0000 s:0002 b:0002 l:000048 d:000048 TOP

-- Ruby level backtrace information ----------------------------------------
test_bug.rb:9:in <main>' test_bug.rb:9:in times'
test_bug.rb:12:in block in <main>' test_bug.rb:12:in new'

-- C level backtrace information -------------------------------------------
/usr/lib/libruby-1.9.1.so.1.9(rb_vm_bugreport+0x5f) [0x7f53d34ec58f]
/usr/lib/libruby-1.9.1.so.1.9(+0x5a7be) [0x7f53d33f37be]
/usr/lib/libruby-1.9.1.so.1.9(rb_bug+0xa6) [0x7f53d33f4136]
/usr/lib/libruby-1.9.1.so.1.9(+0xf4a75) [0x7f53d348da75]
/lib/libpthread.so.0(+0xfb40) [0x7f53d318bb40]
/usr/lib/libruby-1.9.1.so.1.9(+0x6edc8) [0x7f53d3407dc8]
/usr/lib/libruby-1.9.1.so.1.9(rb_thread_mark+0x4c) [0x7f53d34d7b8c]
/usr/lib/libruby-1.9.1.so.1.9(+0x6e87a) [0x7f53d340787a]
/usr/lib/libruby-1.9.1.so.1.9(+0x13ea89) [0x7f53d34d7a89]
/usr/lib/libruby-1.9.1.so.1.9(st_foreach+0xb4) [0x7f53d3496284]
/usr/lib/libruby-1.9.1.so.1.9(rb_vm_mark+0x29) [0x7f53d34d7f09]
/usr/lib/libruby-1.9.1.so.1.9(+0x6e87a) [0x7f53d340787a]
/usr/lib/libruby-1.9.1.so.1.9(+0x6fd98) [0x7f53d3408d98]
/usr/lib/libruby-1.9.1.so.1.9(+0x70b4c) [0x7f53d3409b4c]
/usr/lib/libruby-1.9.1.so.1.9(+0x1403af) [0x7f53d34d93af]
/usr/lib/libruby-1.9.1.so.1.9(rb_thread_alloc+0x66) [0x7f53d34e9286]
/usr/lib/libruby-1.9.1.so.1.9(+0x154cb8) [0x7f53d34edcb8]
/usr/lib/libruby-1.9.1.so.1.9(+0x14c93e) [0x7f53d34e593e]
/usr/lib/libruby-1.9.1.so.1.9(+0x143f61) [0x7f53d34dcf61]
/usr/lib/libruby-1.9.1.so.1.9(+0x148b44) [0x7f53d34e1b44]
/usr/lib/libruby-1.9.1.so.1.9(rb_yield+0x238) [0x7f53d34e8408]
/usr/lib/libruby-1.9.1.so.1.9(+0x96571) [0x7f53d342f571]
/usr/lib/libruby-1.9.1.so.1.9(+0x14c93e) [0x7f53d34e593e]
/usr/lib/libruby-1.9.1.so.1.9(+0x143f61) [0x7f53d34dcf61]
/usr/lib/libruby-1.9.1.so.1.9(+0x148b44) [0x7f53d34e1b44]
/usr/lib/libruby-1.9.1.so.1.9(rb_iseq_eval_main+0xb1) [0x7f53d34e8c81]
/usr/lib/libruby-1.9.1.so.1.9(+0x5ecca) [0x7f53d33f7cca]
/usr/lib/libruby-1.9.1.so.1.9(ruby_exec_node+0x1d) [0x7f53d33f82ed]
/usr/lib/libruby-1.9.1.so.1.9(ruby_run_node+0x1e) [0x7f53d33f9e3e]
ruby(main+0x4b) [0x40092b]
/lib/libc.so.6(__libc_start_main+0xfe) [0x7f53d254fd8e]
ruby() [0x400819]

3. Sometimes it crashes on "times.times do".

However as I said, in same machine using Ruby 1.9.1 it doesn't crash.

My system data:

• AMD Phenom(tm) II X4 965 Processor
• Ubuntu 10.10 64 bits
• gcc-4.4
• Ruby 1.9.2-p0, 1.9.2-p136, and snapshot version (2011-01-12)
• 4GB RAM
  =end

[ibc (Iñaki Baz Castillo)]

=begin
Yui:

I can reproduce this on ruby 1.9.3dev (2011-01-12 trunk 30517) [x86_64-freebsd8.2].
Anyway what your environment? (why you didn't copy ruby -v ?)

Hi, I've tested with these versions of Ruby in my Linux Ubuntu 64 bits:

• CRASH in:
  • ruby 1.9.2p0 (2010-08-18 revision 29036) [x86_64-linux]
  • ruby 1.9.2p136 (2010-12-25 revision 30365) [x86_64-linux]
  • ruby 1.9.3dev (2011-01-11 trunk 30513) [x86_64-linux]
• All ok in:
  • ruby 1.9.1p430 (2010-08-16 revision 28998) [x86_64-linux]
    =end

[ibc (Iñaki Baz Castillo)]

=begin
Tomoyuki:

Please add the following line in test_encoding.c

#include <ruby/encoding.h>

Without that header, rb_utf8_encoding() has no prototype declaration and its return value is estimated to be int.

Yes!! that is! no more crashes after adding it! Thanks a lot.

Now just a two questions remain:

• Why doesn't it crash on Ruby-1.9.1-xxx?
• Should "ruby.h" include "ruby/encoding.h"?

=end

[naruse (Yui NARUSE)]

• Status changed from Open to Closed

=begin
"ruby/encoding.h" fixes this on my environment too.
=end