Backport #744: memory leak in callcc?

• Status: Closed
• Assignee: nobu (Nobuyoshi Nakada)

[rogerdpack (Roger Pack)]

=begin
from http://rubyforge.org/tracker/?func=detail&atid=1698&aid=7896&group_id=426
this code
require 'generator'
loop { g = Generator.new {|x| (1..3).each {|i| x.yield i}} }

seems to leak for me--I'm not sure if this is expected or not.

Thanks.
=end

[rogerdpack (Roger Pack)]

=begin

If we knew what wrote these, we might be able to explicitly clear them
once
they go out of scope. This should result in better significantly GC
performance all around.

Yeah I've wondered that too. Maybe we can have a hackfest for it some saturday :)
http://redmine.ruby-lang.org/issues/show/649
is related [and somewhat frustrating to be honest]. My thought is that maybe there's a way to "clear the stack" of data that isn't currently "useful" and thus clear it of old references [I realize this may be hard].

Thoughts?
-=R
=end

[duerst (Martin Dürst)]

=begin
At 12:54 08/11/17, Brent Roman wrote:

One could hack the gcc to force it to initialize automatic variables to zero
even though this violates the 'C' langauage spec.

I haven't read the spec, but my guess (having worked on other specs)
is that the only thing that the 'C' language spec says is that the
value is undefined. A value that happens to be zero would still be
undefined, as far as I understand.

Regards, Martin.

#-#-# Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-# http://www.sw.it.aoyama.ac.jp mailto:[e-mail]

=end

[rogerdpack (Roger Pack)]

=begin

After a couple weeks of long nights and false starts, I feel I may have come
up with
a fix for a large class of Ruby memory leak. The basic technique is a
refinement of the
one Kurt Stephens suggested. It not only eliminates the leaks in this one
liner:

Wow thanks for doing that. I'd say please create a redmine bug for it
[or attach it to an existing]. A patch to 1.8.7 would be sweet :)
A patch for 1.9 would be great too :)

I'd imagine that your system is "better" than just blindly doing a
garbage_collect()
{
clear_stack();
....do normal gc
}
void clear_stack()
{
a = char[10000];
memclear(a);
}
?

Thanks!
-=R
Note that I use gcc 3.4.5 I assume that won't be a problem though.

=end

[candlerb (Brian Candler)]

=begin
The problem can be demonstrated with a very simple program (attached), and
it looks to me like a bug in gcc - surely it should overlap stack
assignments for automatic variables which aren't in scope simultaneously?

One solution to rb_eval() might be an ugly union at the top of the function
(second attachment). But it seems wrong to have to do this just to code
around an implementation problem with one particular compiler, albeit a
ubiquitous one.

Regards,

Brian.

Attachment: (unnamed)
Attachment: (unnamed)
=end

[candlerb (Brian Candler)]

=begin

What I did come up with was not ugly at all. Factor the unwieldy switch
statement of rb_eval() into separate functions to handle each node
type

Did you replace the whole switch statement with a dispatch table? That
sounds like a sensible thing to do anyway.

OTOH, if this is for ruby 1.8.x, I'm afraid you may not find much interest
in such changes while the focus is all on 1.9.

Perhaps worth checking how 1.9's bytecode interpreter stacks up under the
same conditions?

OTOH, 1.9 doesn't have callcc anyway, so maybe your application code would
need a lot of restructuring to use Fiber instead. I don't know if it's
possible to implement callcc in terms of Fiber.

Regards,

Brian.

=end

[cout (Paul Brannan)]

=begin
On Mon, Dec 01, 2008 at 06:29:00PM +0900, Brian Candler wrote:

OTOH, 1.9 doesn't have callcc anyway, so maybe your application code would
need a lot of restructuring to use Fiber instead. I don't know if it's
possible to implement callcc in terms of Fiber.

1.9 does have callcc (require 'continuation'). It's probably not good
to use it, though.

Paul

=end

[rogerdpack (Roger Pack)]

=begin

I just bang on Ruby 1.6.8 for our robotics application.

I was wondering why the older version :)

You seem to already be doing a lot of excellent Ruby testing with current
versions.
If I spent a couple days developing these two patches for Ruby 1.8.7,
would you be willing to run
regression tests against them and to report the results here?

Absolutely. I'll test them against some trivial stuff and a small
rails app and see if they help memory wise and check for speed :)

P.P.S. The way GC is currently invoked causes it to occur when that stack
is already near its maximum depth. This patch tries to make GC normally
occur is part of CHECK_INTS, when the stack tends to be shallower.
At that point, clearing the stack can be much more effective.

I wonder if there are less intrusive ways, like changing [from a previous post]

       VALUE l = rb_eval(self,node->nd_recv);
        VALUE r = rb_eval(self,node->nd_value);
        result = rb_reg_match(l, r);
    }
    break;

    /* nodes for speed-up(literal match) */
  case NODE_MATCH3:
    {
        VALUE r = rb_eval(self,node->nd_recv);
        VALUE l = rb_eval(self,node->nd_value);

....
to
...

VALUE l = NULL;
VALUE r = NULL;

        l = rb_eval(self,node->nd_recv);
        r = rb_eval(self,node->nd_value);
        result = rb_reg_match(l, r);
    }
    break;

    /* nodes for speed-up(literal match) */
  case NODE_MATCH3:
    {
        r = rb_eval(self,node->nd_recv);
        l = rb_eval(self,node->nd_value);

[reuse same variable].

Also re: size --doesn't 1.9 have rubygems pre-installed so that it
isn't as large of a standard library? [just pointing out that maybe it
could use some minimizing love still?] :)
Thanks!
-=R

=end

[rogerdpack (Roger Pack)]

=begin

I implemented a scheme for recording the maximum depth of the C stack in
xmalloc and during garbage collection itself. However, I realized that
there was no point in clearing the stack when it is near its maximum depth.
Instead, stack clearing is deferred until CHECK_INTS, as this tends to
happen
between evaluation of nodes, when the stack is likely to be shallower.

At this point
a tight loop quickly zeros the region between the current top of stack, as
returned by alloca(0), and the maximum recorded stack extent. It also
updates
the stack extent so no memory is cleared repeatedly if the stack contracts
further.

This is sweet. I liked the idea so much I coded my own [perhaps much
smaller, definitely less effective] version. It only includes the
stack clearing you referred to, and doesn't even monitor "exactly" the
stack size, but approximates it by metering it once every CHECK_INTS.
Ruby seems to run "as fast as normal" with it, and collect better.

In principle, you'd only have to clear the stack once "between each
GC" so if you kept track of which portions of it you'd been able to
clear, you could avoid a few stack clearings :)
I'm not sure exactly how much cpu that would save, though.

This patch also doesn't fix the
loop {@x=callcc{|c|c}}
aspect [presumably because ruby's green threads copy chunks of the
stack to heap, so they aren't cleaned]--so I'd imagine it's less
effective in multi-threaded codes [but hopefully still helpful].

Look forward to the real patch when it comes in :)

Note that as it is currently, if you run GC.start it also calls
clean_stack, so if you run GC.start when your program is at it "inner
depth [most nested call]" it will notice exactly how deep it is, and
hopefully clean up the stack "all the way" when you ascend out of deep
calls. I suppose creating a new call "GC.clear_stack" would be
useful.

i.e. GC.start -> GC.start + "clean stack/make a note of how deep the
stack is currently"

With [1] it successfully prevents the string 'a' from not being
garbage collected:
With [2] it successfully collects a few more objects than the unpatched does.
I'm not positive how well it works but I think it does.

Enjoy.

-=R

[0] patch: http://wilkboardonline.com/roger/clear_stack_only2.diff

[1] file.rb:

def does_nothing
end
def deep(how, gc = false)
if(how == 175)
'a'*1000
end
if how == 300
print "222222deepest"
GC.start
print "222222deepest"
return
end
deep(how+1)
20.times {does_nothing}
end
puts
deep(0)
GC.start
deep(0, true)
count = 0
ObjectSpace.each_object(String) do |s| print s, ' '; count = count+1; end
print count

[2] file2.rb:

count = 0
ObjectSpace.each_object{|o| count += 1 }
print count
GC.disable
def go depth
if depth == 50
GC.enable
GC.start
return
end
if(rand(10) == 3)
a = 'abcd'
go(depth+1)
go(depth+1)
end
if(rand(10) == 3)
b = 'abcd'
end
go(depth+1)
end

go 0
count = 0
ObjectSpace.each_object{|o| count += 1 }
print count

=end

[rogerdpack (Roger Pack)]

=begin
First thanks for doing all that hard work. I'm sure it's not pleasant
to try and figure this all out, and you seem to have done a very
thorough job :)

A few questions.

Process Size Inital/Final User's CPU time (from the time command)
Unpatched 1.8.7-p72: 30MB/97MB 92 seconds
MBARI 6 atop 1.8.7-p2: 30MB/57MB 100 seconds

Is this the time to complete test-all?

I wonder why it uses more total time... :) [the RAM usage looks nice
though]. Makes me wish we had similar patches for 1.9, too [running
make test-all on 1.9 for me typically uses like 400MB RSS for some
reason...].

The patched version reports one additional failure:
2) Failure:
test_should_propagate_signaled(TestBeginEndBlock)
[./ruby/test_beginendblock.rb:81]:
<""> expected to be =~
</Interrupt$/>.

Does it report this consistently?

Interestingly, with 1.8.6 HEAD on mingw currently I get this:

3. Failure:
   test_should_propagate_signaled(TestBeginEndBlock)
   [../ruby_1_8/test/ruby/test_beginendblock.rb:83]:
   expected but was
   <3>.

And, the drb test segfaults with the patched version.
(so I removed it for both the patched and unpatched for comparason)

Maybe you could post a gdb backtrace [in case someone can figure out
what's going on...]

Question--The install instructions mention using
-mpreferred-stack-boundary=2, though in the writeup it says it helps
only slightly--but you recommend it because it still helps?

re MBARI2: gc sometimes segfaults: do you have any examples of how it
does this? So these old frames are collected but not really--is that
what happens?

re: MBARI3 is it possible to use memzero to forcefully overwrite local
variables [though as you pointed out, it would still leave
temporaries]. Are there any other culprits besides rb_eval [and
doesn't eval get called fairly rarely so this isn't a help for most
progs?]

You mention that after this the callcc stuff should work--do you think
that only applying this one patch should be sufficient for that to
happen?

why remove the dynamic malloc_limit?
One thing you might want to try would be the ruby benchmark suite with
and without [1].

MBARI5 : ruby extends the stack when it needs to thread shift from a
"smaller stack" thread to a larger stack thread, is that right? After
shifting to a smaller stack might be a good time to clean the stack...

re: MBARI6 question: why are these included with 5 other gc patches?
[besides that they're cool and useful]? Might be convenient to just
include the 1.9 style syntax by default [I might could come up with a
patch for it].:)

re: sourceref--it might be convenient to tie in with SCRIPT_LINES__
stuff, perhaps [thanks to nobu for pointing out its existence recently
to me].

I suppose my only wish list for these would be that it didn't clear
the stack but once per thread per GC. I might could help out sometime
with it.

Thanks much for your work on these. I'll give them a shot on windows
mingw/linux by next weekend.
Cheers.
-=r
[1] http://github.com/acangiano/ruby-benchmark-suite/tree/master

=end

[gnufied (hemant kumar)]

=begin
Hey Brent,

Thanks for patches man. I am yet to dig deeper, but I benchmarked
rails against them:

Here is the Average request/response for patched version:

Requests per second: 234.77 [#/sec] (mean)
Time per request: 42.594 [ms] (mean)
Time per request: 4.259 [ms] (mean, across all concurrent requests)
Transfer rate: 108.82 [Kbytes/sec] received

Memory usage stayed around 30MB

For Stock Ruby version:

Requests per second: 138.48 [#/sec] (mean)
Time per request: 72.214 [ms] (mean)
Time per request: 7.221 [ms] (mean, across all concurrent requests)
Transfer rate: 64.21 [Kbytes/sec] received

Memory usage stayed around 53 MB

I compiled both ruby versions without "--disable-pthread" and was
wondering if your patches modify anything there.

On Mon, Dec 22, 2008 at 3:29 PM, Brent Roman [e-mail] wrote:

Roger,

I just updated the patches at:

http://sites.google.com/site/brentsrubypatches

to fix the bug that was causing the drb test suite to segfault.

All the test suites now run to completion.

Responses to your questions:

R: Is this the time to complete test-all?, What patches for about 1.9?,
Why slower?

B: This is the time to complete the command:
ruby runner.rb
in the test subdir of the 1.8.7p72 directory.

I suspect that the unpatched interpreter is leaking throughout the execution
of the tests.
Process size just keep increasing. With these patches is stabilizes about
1/3 the way through.
These techniques may work with v1.9 as my understanding is that the GC is
largely unchanged.

Apps that don't swap context much will be a few percent slower. Those that
do should be faster. There certainly is more that can be down to optimize
the stack clearing. My initial goal was to plug the memory leaks so that
Ruby apps could run for long periods without swapping (or worse). In
practice, once a Ruby process starts swapping to virtual memory, its
performance degrades much more than a few percent.

R:

The patched version reports one additional failure:
2) Failure:
test_should_propagate_signaled(TestBeginEndBlock)
[./ruby/test_beginendblock.rb:81]:
<""> expected to be =~
</Interrupt$/>.

Does it report this consistently?

B: Funny you should ask that...
No, it does not fail consistently. Any ideas what's happening here?
It does feel like the same problem you see with or mingw port.

R: The install instructions mention using
-mpreferred-stack-boundary=2, though in the writeup it says it helps
only slightly--but you recommend it because it still helps?

B: Yes, stack-boundary=2 helps keep the frames a little smaller.
For a multi-threaded app, this is probably worth the little performance hit.
For a single threaded app, it may be better to leave out the
-mpreferred-stack-boundary=2
We need more benchmarking to tell.
Ruby should no longer leak memory regardless.

R: re: MBARI2: gc sometimes segfaults: do you have any examples of how it
does this? So these old frames are collected but not really--is that
what happens?

B: Have a look at this post of mine dated 12/03/07
http://markmail.org/message/jjmqzsxenp7oaojm

R: re: MBARI3 is it possible to use memzero to forcefully overwrite local
variables [though as you pointed out, it would still leave
temporaries]. Are there any other culprits besides rb_eval [and
doesn't eval get called fairly rarely so this isn't a help for most
progs?]

B:
I suspect memzero would be slower than the tight loop I have zeroing the
stack now.
In any case, the temporaries are critically important. rb_eval is the 800
pound gorrilla :-)

R:
You mention that after this the callcc stuff should work--do you think
that only applying this one patch should be sufficient for that to
happen?

B:
I think so. However, I'd recommend installing at least MBARI2 as well to
improve performance.

R:
why remove the dynamic malloc_limit?

B:
Because I believe the malloc_limit should be tuned for your target
environment.
In a target with 32MB DRAM, malloc_limit should not be 8MB and I certainly
don't want it to increase on its own. Remember, once Ruby starts swapping,
performance goes into the toilet.
I probably won't be motivated enough to benchmark it. A few percent run
time change does not matter much to me. I want my app to run for months at
a time and to play nice with others.

R:
MBARI5 : ruby extends the stack when it needs to thread shift from a
"smaller stack" thread to a larger stack thread, is that right? After
shifting to a smaller stack might be a good time to clean the stack...

B:
The MBARI3 patch updates the stack extent at a number of points, including
on every context switch, but it defers clearing it until the next
CHECKINTS(), when the stack is likely to be smaller still. Even so,
optimizing this further is definitely possible. I've considered only
clearing the stack after GC.increase rises to 75% of GC.limit, for instance.

R:
re: MBARI6 question: why are these included with 5 other gc patches?
[besides that they're cool and useful]? Might be convenient to just
include the 1.9 style syntax by default [I might could come up with a
patch for it].:)

B:
MBARI6 probably should have been packaged separately.
My line and file patches predate the 1.9 stuff by about 5 years.
See:
http://markmail.org/message/ybrbhvvzlhyv552y
I did think of redoing them in the 1.9 style, but I don't particularly like
the idea
of returning an array in this context, where numbered indices replace named
attributes.
In any case, I can emulate the 1.9 style methods with a tiny bit of Ruby
glue.

R:
I suppose my only wish list for these would be that it didn't clear
the stack but once per thread per GC. I might could help out sometime
with it.

B:
That's on my wish list too. I'd be very grateful for any help, even just
discussing ideas.

• brent

--
View this message in context: http://www.nabble.com/-ruby-core%3A19846---Bug--744--memory-leak-in-callcc--tp20447794p21125317.html
Sent from the ruby-core mailing list archive at Nabble.com.

--
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

=end

[gnufied (hemant kumar)]

=begin
Hi

On Tue, Dec 23, 2008 at 9:57 AM, Brent Roman [e-mail] wrote:

My patches don't mess with any of the pthread stuff.

I'm a pleasantly surprised by your rails benchmark results.
I would have expected this memory savings, but I can't think of why a single
threaded application like Rails (that doesn't use Continuations), would see
the sort of speed up you observed. I'd expect it to be 2 to 10 percent
slower unless it was doing a lot of context switches.

I did get an off list response from a chinese website that confirms the
rails memory savings, but
they said there was no change in speed.

Are you talking about (http://www.javaeye.com)?

Does your Rails application use threads or continuations?

No, I was just benchmarking a hello world rails application.

Are you comparing ruby built from the same source tarball with the same
compiler options before and after patching?

Yes. Essentially before patching and after patching.

=end

[rogerdpack (Roger Pack)]

=begin
Seems to overall be a tidge slower for "micro" stuff--5 or 10%.
viz:
lloyd gc bench:
187 unpatched:

arrays_read.rb time 0.072516
arrays_read_yaml.rb time 0.671292
classes_read.rb time 0.040723
classes_read_yaml.rb time 0.736394
create_arrays.rb time 0.165607
create_arrays_yaml.rb time 7.638495
create_hashes.rb time 0.136778
create_hashes_yaml.rb time 20.888187
create_ostructs.rb time 2.028835
create_ostructs_yaml.rb time 10.707594
create_weak_hashes.rb time 0.946386
create_weak_hashes2.rb time 0.389543
growarray.rb time 1.788691
hashes_read.rb time 0.037333
hashes_read_yaml.rb time 1.687161
ostruct_read.rb time 1.691467
ostruct_read_yaml.rb time 1.05634
plist.rb time 4.27333
shrinkarray.rb time 1.751121
weak_hashes_read.rb time 0.293751

187 patched:

arrays_read.rb time 0.060988
arrays_read_yaml.rb time 0.706926
classes_read.rb time 0.041115
classes_read_yaml.rb time 0.736123
create_arrays.rb time 0.171677
create_arrays_yaml.rb time 7.715646
create_hashes.rb time 0.121288
create_hashes_yaml.rb time 21.457203
create_ostructs.rb time 2.020391
create_ostructs_yaml.rb time 11.011948
create_weak_hashes.rb time 1.035461
create_weak_hashes2.rb time 0.381697
growarray.rb time 1.865321
hashes_read.rb time 0.0376
hashes_read_yaml.rb time 1.802083
ostruct_read.rb time 1.705456
ostruct_read_yaml.rb time 1.108687
plist.rb time 4.64743
shrinkarray.rb time 1.833105
weak_hashes_read.rb time 0.293376

But that's for micro-benchmarks.
I think the reason we see people's performance increase is that since
the GC is suddenly more effective, it doesn't get called as often. A
big win for larger apps.

Overall I'd call it a large win for Ruby in terms of being much more
stable size-wise in a multi-threaded environment and suggest their
incorporation verbatim. All 6 :)

raw ruby-benchmark-suite comparison is in the footnote.
Note a few things:
one test erred with 187 normal but succeeded with MBARI patches
(core-library/bm_so_concatenate.rb)
the threaded tests do indeed run faster with MBARI.

normal:
core-library/bm_vm3_thread_create_join.rb,0.20678186416626
patched:
core-library/bm_vm3_thread_create_join.rb,0.0140390396118164

Some other thoughts I've had are that theoretically you only need to
clear the stack once between GC's, so you may be able to just keep a
"range already cleared" per thread or what not, and reset it after
each GC. This would especially work if rb_thread_alone is true.

You might be able to get away with only checking for stack depth once
every CHECK_INT [instead of with xmalloc].

Maybe even clear the stack only at ruby_stack_check [though this is
probably too infrequent].

I did a small experiment with memset versus tight loop and [somehow] a
tight loop seems to win.

I think there is some potential for optimization if you were to use
fixed 2K heap chunks and binary search for is_pointer_to_heap [with
cacheing of the most recently found heap chunk to help save on speed].
Theoretically it might bring RAM usage down even further [1.9 does
this].

I know that at least for me I will definitely use these for my own
apps so that they have more control for memory.

Re: javaeye.com speed "almost the same" with railsbench GC patch +
these versus just railsbench GC patch--I think that what is happening
in this case is that GC is being called only when the freelist is used
up, since the malloc_limit is so large. Tough to know how to speed it
up in that case [except for running GC in a different process and
earlier].

Thanks for your hard work. I think it was something a few of us had
thought necessary but never got up the gumption to do :)

-=r

Some raw data [to me this means little compared to the rails stuffs
reported earlier].

ruby-benchmark-suite
with patch:
Benchmark Name,Time #1,Time #2,Average Time,Standard Deviation,Input Size
Startup,0.00860691070556641,0.00712394714355469,0.007865428924561,0.000741481781006,n/a
real-world/bm_hilbert_matrix.rb,0.0715880393981934,0.0691721439361572,0.070380091667175,0.001207947731018,10
real-world/bm_hilbert_matrix.rb,0.705732822418213,0.707005977630615,0.706369400024414,0.000636577606201,20
real-world/bm_hilbert_matrix.rb,2.71448302268982,2.73366808891296,2.724075555801392,0.009592533111572,30
real-world/bm_hilbert_matrix.rb,7.9450159072876,8.08562898635864,8.015322446823120,0.070306539535522,40
standard-library/bm_app_mandelbrot.rb,3.50128412246704,3.50250101089478,3.501892566680908,0.000608444213867,n/a
micro-benchmarks/bm_meteor_contest.rb,47.9955010414124,48.7175140380859,48.356507539749146,0.361006498336792,n/a
micro-benchmarks/bm_app_pentomino.rb,149.979510068893,150.394422769547,150.186966419219971,0.207456350326538,n/a
micro-benchmarks/bm_fasta.rb,63.8084781169891,54.929356098175,59.368917107582092,4.439561009407043,n/a
micro-benchmarks/bm_fannkuch.rb,0.0105619430541992,0.0125219821929932,0.011541962623596,0.000980019569397,6
micro-benchmarks/bm_fannkuch.rb,0.777202129364014,0.779121875762939,0.778162002563477,0.000959873199463,8
micro-benchmarks/bm_fannkuch.rb,85.7568709850311,85.9479658603668,85.852418422698975,0.095547437667847,10
micro-benchmarks/bm_nbody.rb,15.366986989975,15.376590013504,15.371788501739502,0.004801511764526,n/a
micro-benchmarks/bm_reverse_compliment.rb,8.94560790061951,8.99379897117615,8.969703435897827,0.024095535278320,n/a
micro-benchmarks/bm_quicksort.rb,7.39512896537781,7.40803289413452,7.401580929756165,0.006451964378357,n/a
micro-benchmarks/bm_mergesort.rb,4.32359004020691,4.32240605354309,4.322998046875000,0.000591993331909,n/a
micro-benchmarks/bm_nsieve_bits.rb,36.2198901176453,36.2111361026764,36.215513110160828,0.004377007484436,n/a
micro-benchmarks/bm_mandelbrot.rb,115.146002054214,115.128952026367,115.137477040290833,0.008525013923645,n/a
micro-benchmarks/bm_lucas_lehmer.rb,20.1205780506134,20.1091129779816,20.114845514297485,0.005732536315918,9689
micro-benchmarks/bm_lucas_lehmer.rb,21.7712378501892,21.766205072403,21.768721461296082,0.002516388893127,9941
micro-benchmarks/bm_lucas_lehmer.rb,31.5163369178772,31.5209879875183,31.518662452697754,0.002325534820557,11213
micro-benchmarks/bm_lucas_lehmer.rb,Timeout: 150.00 seconds,,,,19937
micro-benchmarks/bm_fractal.rb,50.184531211853,50.1739339828491,50.179232597351074,0.005298614501953,n/a
micro-benchmarks/bm_knucleotide.rb,2.19779801368713,2.3165180683136,2.257158041000366,0.059360027313232,n/a
micro-benchmarks/bm_monte_carlo_pi.rb,27.104642868042,27.153263092041,27.128952980041504,0.024310111999512,n/a
micro-benchmarks/bm_word_anagrams.rb,13.1331388950348,12.0968029499054,12.614970922470093,0.518167972564697,n/a
micro-benchmarks/bm_binary_trees.rb,101.144680023193,102.439230918884,101.791955471038818,0.647275447845459,n/a
micro-benchmarks/bm_spectral_norm.rb,1.51863193511963,1.51901316642761,1.518822550773621,0.000190615653992,n/a
micro-benchmarks/bm_nsieve.rb,33.2746829986572,33.2826149463654,33.278648972511292,0.003965973854065,n/a
micro-benchmarks/bm_regex_dna.rb,5.48113203048706,6.14786696434021,5.814499497413635,0.333367466926575,n/a
micro-benchmarks/bm_sum_file.rb,14.6941390037537,14.5948259830475,14.644482493400574,0.049656510353088,n/a
micro-benchmarks/bm_partial_sums.rb,37.6713261604309,37.645623922348,37.658475041389465,0.012851119041443,n/a
micro-benchmarks/bm_so_sieve.rb,116.000460863113,116.059950828552,116.030205845832825,0.029744982719421,n/a
core-features/bm_vm1_rescue.rb,0.296025037765503,0.280431985855103,0.288228511810303,0.007796525955200,n/a
core-features/bm_vm1_length.rb,20.3501679897308,20.2971909046173,20.323679447174072,0.026488542556763,10
core-features/bm_vm1_length.rb,20.3735370635986,20.3696429729462,20.371590018272400,0.001947045326233,100
core-features/bm_vm1_length.rb,20.3027820587158,20.3756489753723,20.339215517044067,0.036433458328247,1000
core-features/bm_vm1_length.rb,20.3291130065918,20.3042759895325,20.316694498062134,0.012418508529663,10000
core-features/bm_so_ackermann.rb,0.0445468425750732,0.0442321300506592,0.044389486312866,0.000157356262207,5
core-features/bm_so_ackermann.rb,0.727099895477295,0.724959135055542,0.726029515266418,0.001070380210876,7
core-features/bm_so_ackermann.rb,12.0094769001007,11.994206905365,12.001841902732849,0.007634997367859,9
core-features/bm_vm2_poly_method.rb,4.42595386505127,4.39683985710144,4.411396861076355,0.014557003974915,1000000
core-features/bm_vm2_poly_method.rb,8.86316585540771,8.91431498527527,8.888740420341492,0.025574564933777,2000000
core-features/bm_vm2_poly_method.rb,18.3531260490417,18.3768548965454,18.364990472793579,0.011864423751831,4000000
core-features/bm_vm2_poly_method.rb,35.1964910030365,35.3743059635162,35.285398483276367,0.088907480239868,8000000
core-features/bm_app_tak.rb,0.170708179473877,0.170413970947266,0.170561075210571,0.000147104263306,5
core-features/bm_app_tak.rb,0.616703987121582,0.613183975219727,0.614943981170654,0.001760005950928,6
core-features/bm_app_tak.rb,1.96985197067261,1.96115493774414,1.965503454208374,0.004348516464233,7
core-features/bm_so_random.rb,0.248342990875244,0.251052856445312,0.249697923660278,0.001354932785034,100000
core-features/bm_so_random.rb,1.25759100914001,1.2419810295105,1.249786019325256,0.007804989814758,500000
core-features/bm_so_random.rb,2.50672101974487,2.487135887146,2.496928453445435,0.009792566299438,1000000
core-features/bm_vm1_swap.rb,9.50407981872559,9.48183703422546,9.492958426475525,0.011121392250061,10000000
core-features/bm_vm1_swap.rb,19.0615699291229,19.1599650382996,19.110767483711243,0.049197554588318,20000000
core-features/bm_vm1_swap.rb,37.5512471199036,37.8161060810089,37.683676600456238,0.132429480552673,40000000
core-features/bm_app_fib.rb,0.02223801612854,0.0222301483154297,0.022234082221985,0.000003933906555,20
core-features/bm_app_fib.rb,2.72851300239563,2.7235860824585,2.726049542427063,0.002463459968567,30
core-features/bm_app_fib.rb,30.3188951015472,30.1245629787445,30.221729040145874,0.097166061401367,35
core-features/bm_vm2_zsuper.rb,0.935019969940186,0.979220867156982,0.957120418548584,0.022100448608398,1000000
core-features/bm_vm2_zsuper.rb,1.99105310440063,1.85130095481873,1.921177029609680,0.069876074790955,2000000
core-features/bm_vm2_zsuper.rb,3.85666489601135,3.85760307312012,3.857133984565735,0.000469088554382,4000000
core-features/bm_vm2_zsuper.rb,7.69519710540771,7.70335698127747,7.699277043342590,0.004079937934875,8000000
core-features/bm_app_factorial.rb,0.00926995277404785,0.00803399085998535,0.008651971817017,0.000617980957031,1000
core-features/bm_app_factorial.rb,0.0369241237640381,0.0322320461273193,0.034578084945679,0.002346038818359,2000
core-features/bm_app_factorial.rb,0.245321989059448,0.241642951965332,0.243482470512390,0.001839518547058,5000
core-features/bm_app_factorial.rb,Error: stack level too deep,,,,10000
core-features/bm_app_tarai.rb,6.74241399765015,6.74293112754822,6.742672562599182,0.000258564949036,3
core-features/bm_app_tarai.rb,8.1383171081543,8.13438200950623,8.136349558830261,0.001967549324036,4
core-features/bm_app_tarai.rb,9.85991907119751,9.85500311851501,9.857461094856262,0.002457976341248,5
core-features/bm_vm1_const.rb,9.00359511375427,18.2473177909851,13.625456452369690,4.621861338615417,n/a
core-features/bm_so_nested_loop.rb,0.00854110717773438,0.00856804847717285,0.008554577827454,0.000013470649719,5
core-features/bm_so_nested_loop.rb,0.471377849578857,0.481393098831177,0.476385474205017,0.005007624626160,10
core-features/bm_so_nested_loop.rb,5.18516802787781,5.36552095413208,5.275344491004944,0.090176463127136,15
core-features/bm_vm1_ensure.rb,0.0761599540710449,0.0757908821105957,0.075975418090820,0.000184535980225,100000
core-features/bm_vm1_ensure.rb,0.761255979537964,0.760828971862793,0.761042475700378,0.000213503837585,1000000
core-features/bm_vm1_ensure.rb,7.46813201904297,7.48478889465332,7.476460456848145,0.008328437805176,10000000
core-features/bm_vm2_proc.rb,1.35628509521484,1.35869193077087,1.357488512992859,0.001203417778015,1000000
core-features/bm_vm2_proc.rb,2.70739102363586,2.70443820953369,2.705914616584778,0.001476407051086,2000000
core-features/bm_vm2_proc.rb,5.41131019592285,5.4192328453064,5.415271520614624,0.003961324691772,4000000
core-features/bm_vm2_proc.rb,10.8037610054016,10.8294909000397,10.816625952720642,0.012864947319031,8000000
core-features/bm_loop_times.rb,5.33636403083801,5.50595808029175,5.421161055564880,0.084797024726868,10000000
core-features/bm_loop_times.rb,10.5615899562836,10.6464350223541,10.604012489318848,0.042422533035278,20000000
core-features/bm_loop_times.rb,16.3860912322998,15.2040379047394,15.795064568519592,0.591026663780212,30000000
core-features/bm_vm2_unif1.rb,0.562043190002441,0.56634783744812,0.564195513725281,0.002152323722839,1000000
core-features/bm_vm2_unif1.rb,1.14902997016907,1.12535285949707,1.137191414833069,0.011838555335999,2000000
core-features/bm_vm2_unif1.rb,2.30900192260742,2.34629487991333,2.327648401260376,0.018646478652954,4000000
core-features/bm_vm2_unif1.rb,4.63608813285828,4.79328298568726,4.714685559272766,0.078597426414490,8000000
core-features/bm_vm1_simplereturn.rb,6.01986694335938,6.00216197967529,6.011014461517334,0.008852481842041,10000000
core-features/bm_vm1_simplereturn.rb,12.1239230632782,12.0794620513916,12.101692557334900,0.022230505943298,20000000
core-features/bm_vm1_simplereturn.rb,17.8927519321442,17.8568298816681,17.874790906906128,0.017961025238037,30000000
core-features/bm_loop_whileloop.rb,0.0550401210784912,0.0549750328063965,0.055007576942444,0.000032544136047,100000
core-features/bm_loop_whileloop.rb,0.55099892616272,0.550863981246948,0.550931453704834,0.000067472457886,1000000
core-features/bm_loop_whileloop.rb,5.51106715202332,5.50824809074402,5.509657621383667,0.001409530639648,10000000
core-features/bm_vm2_send.rb,0.680418014526367,0.68721079826355,0.683814406394958,0.003396391868591,1000000
core-features/bm_vm2_send.rb,1.44064593315125,1.3845579624176,1.412601947784424,0.028043985366821,2000000
core-features/bm_vm2_send.rb,2.74571800231934,2.76275110244751,2.754234552383423,0.008516550064087,4000000
core-features/bm_vm2_send.rb,5.66572713851929,5.52676200866699,5.596244573593140,0.069482564926147,8000000
core-features/bm_vm1_block.rb,0.0927438735961914,0.0923471450805664,0.092545509338379,0.000198364257812,100000
core-features/bm_vm1_block.rb,0.925873994827271,0.922999858856201,0.924436926841736,0.001437067985535,1000000
core-features/bm_vm1_block.rb,9.22466993331909,9.25748586654663,9.241077899932861,0.016407966613770,10000000
core-features/bm_vm2_super.rb,0.868482828140259,0.879498958587646,0.873990893363953,0.005508065223694,1000000
core-features/bm_vm2_super.rb,1.7648811340332,1.7305908203125,1.747735977172852,0.017145156860352,2000000
core-features/bm_vm2_super.rb,3.53582000732422,3.50967812538147,3.522749066352844,0.013070940971375,4000000
core-features/bm_vm2_super.rb,7.03860211372375,7.0927209854126,7.065661549568176,0.027059435844421,8000000
core-features/bm_so_object.rb,2.1716628074646,2.17931509017944,2.175488948822021,0.003826141357422,500000
core-features/bm_so_object.rb,4.34661197662354,4.35301184654236,4.349811911582947,0.003199934959412,1000000
core-features/bm_so_object.rb,6.54321002960205,6.57184290885925,6.557526469230652,0.014316439628601,1500000
core-features/bm_app_raise.rb,6.15209579467773,6.16153502464294,6.156815409660339,0.004719614982605,n/a
core-library/bm_so_exception.rb,15.2897758483887,15.3124811649323,15.301128506660461,0.011352658271790,n/a
core-library/bm_so_concatenate.rb,94.1705470085144,86.1923739910126,90.181460499763489,3.989086508750916,5000
core-library/bm_so_concatenate.rb,Error: failed to allocate memory,,,,10000
core-library/bm_so_concatenate.rb,Error: failed to allocate memory,,,,15000
core-library/bm_so_count_words.rb,12.6775200366974,12.6623919010162,12.669955968856812,0.007564067840576,n/a
core-library/bm_vm2_array.rb,0.747035026550293,0.747730016708374,0.747382521629333,0.000347495079041,1000000
core-library/bm_vm2_array.rb,1.49641180038452,1.49419784545898,1.495304822921753,0.001106977462769,2000000
core-library/bm_vm2_array.rb,2.99150490760803,2.99072194099426,2.991113424301147,0.000391483306885,4000000
core-library/bm_vm2_array.rb,5.98674607276917,5.98161792755127,5.984182000160217,0.002564072608948,8000000
core-library/bm_vm2_regexp.rb,0.963823080062866,0.955650806427002,0.959736943244934,0.004086136817932,10
core-library/bm_vm2_regexp.rb,1.10640692710876,1.1020519733429,1.104229450225830,0.002177476882935,100
core-library/bm_vm2_regexp.rb,2.06975698471069,2.06661009788513,2.068183541297913,0.001573443412781,1000
core-library/bm_vm2_regexp.rb,12.8606810569763,12.8909890651703,12.875835061073303,0.015154004096985,10000
core-library/bm_vm3_thread_create_join.rb,0.0140390396118164,0.0140399932861328,0.014039516448975,0.000000476837158,1000
core-library/bm_vm3_thread_create_join.rb,0.14684009552002,0.142390966415405,0.144615530967712,0.002224564552307,10000
core-library/bm_vm3_thread_create_join.rb,1.42791819572449,1.42644500732422,1.427181601524353,0.000736594200134,100000
core-library/bm_app_strconcat.rb,5.00886416435242,4.99663209915161,5.002748131752014,0.006116032600403,n/a
core-library/bm_so_lists.rb,17.1406989097595,17.1364989280701,17.138598918914795,0.002099990844727,n/a
core-library/bm_so_matrix.rb,2.97365689277649,2.97791600227356,2.975786447525024,0.002129554748535,n/a
core-library/bm_pathname.rb,9.67186689376831,9.60481905937195,9.638342976570129,0.033523917198181,n/a
core-library/bm_so_array.rb,9.70830893516541,9.71373295783997,9.711020946502686,0.002712011337280,n/a

without patch:

Benchmark Name,Time #1,Time #2,Average Time,Standard Deviation,Input Size
Startup,0.00891494750976562,0.0071098804473877,0.008012413978577,0.000902533531189,n/a
real-world/bm_hilbert_matrix.rb,0.0650041103363037,0.0641639232635498,0.064584016799927,0.000420093536377,10
real-world/bm_hilbert_matrix.rb,0.648383140563965,0.593698024749756,0.621040582656860,0.027342557907104,20
real-world/bm_hilbert_matrix.rb,2.48112893104553,2.53910279273987,2.510115861892700,0.028986930847168,30
real-world/bm_hilbert_matrix.rb,7.38045883178711,7.66462898254395,7.522543907165527,0.142085075378418,40
standard-library/bm_app_mandelbrot.rb,3.16141700744629,3.16698312759399,3.164200067520142,0.002783060073853,n/a
micro-benchmarks/bm_meteor_contest.rb,44.2824368476868,45.2044949531555,44.743465900421143,0.461029052734375,n/a
micro-benchmarks/bm_app_pentomino.rb,123.641210079193,124.70353603363,124.172373056411743,0.531162977218628,n/a
micro-benchmarks/bm_fasta.rb,53.9337210655212,46.0915009975433,50.012611031532288,3.921110033988953,n/a
micro-benchmarks/bm_fannkuch.rb,0.00894379615783691,0.0106801986694336,0.009811997413635,0.000868201255798,6
micro-benchmarks/bm_fannkuch.rb,0.672353982925415,0.676352024078369,0.674353003501892,0.001999020576477,8
micro-benchmarks/bm_fannkuch.rb,74.7806649208069,75.063551902771,74.922108411788940,0.141443490982056,10
micro-benchmarks/bm_nbody.rb,13.873694896698,13.8695569038391,13.871625900268555,0.002068996429443,n/a
micro-benchmarks/bm_reverse_compliment.rb,8.97242403030396,8.94234395027161,8.957383990287781,0.015040040016174,n/a
micro-benchmarks/bm_quicksort.rb,7.04030704498291,7.07406520843506,7.057186126708984,0.016879081726074,n/a
micro-benchmarks/bm_mergesort.rb,3.57215404510498,3.57273411750793,3.572444081306458,0.000290036201477,n/a
micro-benchmarks/bm_nsieve_bits.rb,30.060455083847,30.0687861442566,30.064620614051819,0.004165530204773,n/a
micro-benchmarks/bm_mandelbrot.rb,97.2698559761047,97.3307840824127,97.300320029258728,0.030464053153992,n/a
micro-benchmarks/bm_lucas_lehmer.rb,20.239284992218,20.231260061264,20.235272526741028,0.004012465476990,9689
micro-benchmarks/bm_lucas_lehmer.rb,22.0810799598694,22.0774850845337,22.079282522201538,0.001797437667847,9941
micro-benchmarks/bm_lucas_lehmer.rb,31.978404045105,31.9865000247955,31.982452034950256,0.004047989845276,11213
micro-benchmarks/bm_lucas_lehmer.rb,Timeout: 150.00 seconds,,,,19937
micro-benchmarks/bm_fractal.rb,42.6185228824615,42.6280272006989,42.623275041580200,0.004752159118652,n/a
micro-benchmarks/bm_knucleotide.rb,2.08846092224121,2.21108198165894,2.149771451950073,0.061310529708862,n/a
micro-benchmarks/bm_monte_carlo_pi.rb,23.9794390201569,23.9089260101318,23.944182515144348,0.035256505012512,n/a
micro-benchmarks/bm_word_anagrams.rb,12.3941428661346,12.7155430316925,12.554842948913574,0.160700082778931,n/a
micro-benchmarks/bm_binary_trees.rb,82.171679019928,81.3044950962067,81.738087058067322,0.433591961860657,n/a
micro-benchmarks/bm_spectral_norm.rb,1.35397505760193,1.35372304916382,1.353849053382874,0.000126004219055,n/a
micro-benchmarks/bm_nsieve.rb,23.6701729297638,23.6756160259247,23.672894477844238,0.002721548080444,n/a
micro-benchmarks/bm_regex_dna.rb,5.42796611785889,6.06900095939636,5.748483538627625,0.320517420768738,n/a
micro-benchmarks/bm_sum_file.rb,15.2133920192719,15.1973860263824,15.205389022827148,0.008002996444702,n/a
micro-benchmarks/bm_partial_sums.rb,33.0055561065674,32.8080351352692,32.906795620918274,0.098760485649109,n/a
micro-benchmarks/bm_so_sieve.rb,84.2921900749207,83.7600059509277,84.026098012924194,0.266092061996460,n/a
core-features/bm_vm1_rescue.rb,0.289474964141846,0.276180028915405,0.282827496528625,0.006647467613220,n/a
core-features/bm_vm1_length.rb,16.6632568836212,17.2531778812408,16.958217382431030,0.294960498809814,10
core-features/bm_vm1_length.rb,17.1057379245758,16.5685300827026,16.837134003639221,0.268603920936584,100
core-features/bm_vm1_length.rb,16.5657980442047,17.0870249271393,16.826411485671997,0.260613441467285,1000
core-features/bm_vm1_length.rb,17.1546268463135,16.5853810310364,16.870003938674927,0.284622907638550,10000
core-features/bm_so_ackermann.rb,0.0397160053253174,0.0391659736633301,0.039440989494324,0.000275015830994,5
core-features/bm_so_ackermann.rb,0.659607887268066,0.658763885498047,0.659185886383057,0.000422000885010,7
core-features/bm_so_ackermann.rb,Error: stack level too deep,,,,9
core-features/bm_vm2_poly_method.rb,3.11452198028564,3.17220616340637,3.143364071846008,0.028842091560364,1000000
core-features/bm_vm2_poly_method.rb,6.2718460559845,6.33952903747559,6.305687546730042,0.033841490745544,2000000
core-features/bm_vm2_poly_method.rb,12.6923749446869,13.1363949775696,12.914384961128235,0.222010016441345,4000000
core-features/bm_vm2_poly_method.rb,26.4767730236053,26.1098349094391,26.293303966522217,0.183469057083130,8000000
core-features/bm_app_tak.rb,0.135724067687988,0.135273933410645,0.135499000549316,0.000225067138672,5
core-features/bm_app_tak.rb,0.493720054626465,0.491595029830933,0.492657542228699,0.001062512397766,6
core-features/bm_app_tak.rb,1.57262206077576,1.55252599716187,1.562574028968811,0.010048031806946,7
core-features/bm_so_random.rb,0.209523916244507,0.209694147109985,0.209609031677246,0.000085115432739,100000
core-features/bm_so_random.rb,1.04600214958191,1.04535102844238,1.045676589012146,0.000325560569763,500000
core-features/bm_so_random.rb,2.10444784164429,2.09432411193848,2.099385976791382,0.005061864852905,1000000
core-features/bm_vm1_swap.rb,8.73163294792175,8.67143487930298,8.701533913612366,0.030099034309387,10000000
core-features/bm_vm1_swap.rb,17.5122091770172,17.5956919193268,17.553950548171997,0.041741371154785,20000000
core-features/bm_vm1_swap.rb,34.7529518604279,34.8331568241119,34.793054342269897,0.040102481842041,40000000
core-features/bm_app_fib.rb,0.0184860229492188,0.0182771682739258,0.018381595611572,0.000104427337646,20
core-features/bm_app_fib.rb,2.23634505271912,2.2517249584198,2.244035005569458,0.007689952850342,30
core-features/bm_app_fib.rb,24.7462511062622,24.8339931964874,24.790122151374817,0.043871045112610,35
core-features/bm_vm2_zsuper.rb,0.919327974319458,0.896940946578979,0.908134460449219,0.011193513870239,1000000
core-features/bm_vm2_zsuper.rb,1.82282018661499,1.81718993186951,1.820005059242249,0.002815127372742,2000000
core-features/bm_vm2_zsuper.rb,3.54206895828247,3.70834898948669,3.625208973884583,0.083140015602112,4000000
core-features/bm_vm2_zsuper.rb,7.16689205169678,7.22776818275452,7.197330117225647,0.030438065528870,8000000
core-features/bm_app_factorial.rb,0.0101971626281738,0.00816202163696289,0.009179592132568,0.001017570495605,1000
core-features/bm_app_factorial.rb,0.0399060249328613,0.0334930419921875,0.036699533462524,0.003206491470337,2000
core-features/bm_app_factorial.rb,Error: stack level too deep,,,,5000
core-features/bm_app_factorial.rb,Error: stack level too deep,,,,10000
core-features/bm_app_tarai.rb,5.36701798439026,5.35589003562927,5.361454010009766,0.005563974380493,3
core-features/bm_app_tarai.rb,6.47303104400635,6.47742319107056,6.475227117538452,0.002196073532104,4
core-features/bm_app_tarai.rb,7.84382104873657,7.862135887146,7.852978467941284,0.009157419204712,5
core-features/bm_vm1_const.rb,8.81164598464966,18.2617099285126,13.536677956581116,4.725031971931458,n/a
core-features/bm_so_nested_loop.rb,0.0085291862487793,0.00852203369140625,0.008525609970093,0.000003576278687,5
core-features/bm_so_nested_loop.rb,0.47477388381958,0.482151031494141,0.478462457656860,0.003688573837280,10
core-features/bm_so_nested_loop.rb,5.32685899734497,5.50593280792236,5.416395902633667,0.089536905288696,15
core-features/bm_vm1_ensure.rb,0.069011926651001,0.0690209865570068,0.069016456604004,0.000004529953003,100000
core-features/bm_vm1_ensure.rb,0.69483208656311,0.688596963882446,0.691714525222778,0.003117561340332,1000000
core-features/bm_vm1_ensure.rb,6.77466106414795,6.78414702415466,6.779404044151306,0.004742980003357,10000000
core-features/bm_vm2_proc.rb,1.20864987373352,1.21058702468872,1.209618449211121,0.000968575477600,1000000
core-features/bm_vm2_proc.rb,2.42319822311401,2.42020487785339,2.421701550483704,0.001496672630310,2000000
core-features/bm_vm2_proc.rb,4.83699607849121,4.83324503898621,4.835120558738708,0.001875519752502,4000000
core-features/bm_vm2_proc.rb,9.69077706336975,9.66815495491028,9.679466009140015,0.011311054229736,8000000
core-features/bm_loop_times.rb,4.91027808189392,4.85836100578308,4.884319543838501,0.025958538055420,10000000
core-features/bm_loop_times.rb,9.6381299495697,9.79347586631775,9.715802907943726,0.077672958374023,20000000
core-features/bm_loop_times.rb,14.0774569511414,14.2317838668823,14.154620409011841,0.077163457870483,30000000
core-features/bm_vm2_unif1.rb,0.5470130443573,0.575318098068237,0.561165571212769,0.014152526855469,1000000
core-features/bm_vm2_unif1.rb,1.15082097053528,1.09408688545227,1.122453927993774,0.028367042541504,2000000
core-features/bm_vm2_unif1.rb,2.34767317771912,2.39744305610657,2.372558116912842,0.024884939193726,4000000
core-features/bm_vm2_unif1.rb,4.54664993286133,4.72602391242981,4.636336922645569,0.089686989784241,8000000
core-features/bm_vm1_simplereturn.rb,5.74362897872925,5.69164609909058,5.717637538909912,0.025991439819336,10000000
core-features/bm_vm1_simplereturn.rb,11.5744888782501,11.6653461456299,11.619917511940002,0.045428633689880,20000000
core-features/bm_vm1_simplereturn.rb,16.4999470710754,17.2785489559174,16.889248013496399,0.389300942420959,30000000
core-features/bm_loop_whileloop.rb,0.04463791847229,0.0446460247039795,0.044641971588135,0.000004053115845,100000
core-features/bm_loop_whileloop.rb,0.445381879806519,0.445470094680786,0.445425987243652,0.000044107437134,1000000
core-features/bm_loop_whileloop.rb,4.45436692237854,4.45519089698792,4.454778909683228,0.000411987304688,10000000
core-features/bm_vm2_send.rb,0.666916847229004,0.647531032562256,0.657223939895630,0.009692907333374,1000000
core-features/bm_vm2_send.rb,1.31860494613647,1.33532500267029,1.326964974403381,0.008360028266907,2000000
core-features/bm_vm2_send.rb,2.61160898208618,2.61185622215271,2.611732602119446,0.000123620033264,4000000
core-features/bm_vm2_send.rb,5.21212983131409,5.2205491065979,5.216339468955994,0.004209637641907,8000000
core-features/bm_vm1_block.rb,0.0858049392700195,0.0851829051971436,0.085493922233582,0.000311017036438,100000
core-features/bm_vm1_block.rb,0.851619005203247,0.855067014694214,0.853343009948730,0.001724004745483,1000000
core-features/bm_vm1_block.rb,8.54761481285095,8.54824185371399,8.547928333282471,0.000313520431519,10000000
core-features/bm_vm2_super.rb,0.820611000061035,0.826474905014038,0.823542952537537,0.002931952476501,1000000
core-features/bm_vm2_super.rb,1.66156506538391,1.7179229259491,1.689743995666504,0.028178930282593,2000000
core-features/bm_vm2_super.rb,3.43505811691284,3.11910891532898,3.277083516120911,0.157974600791931,4000000
core-features/bm_vm2_super.rb,6.57702493667603,6.63278102874756,6.604902982711792,0.027878046035767,8000000
core-features/bm_so_object.rb,2.12350010871887,2.12192320823669,2.122711658477783,0.000788450241089,500000
core-features/bm_so_object.rb,4.24955415725708,4.26192998886108,4.255742073059082,0.006187915802002,1000000
core-features/bm_so_object.rb,6.3975510597229,6.39479994773865,6.396175503730774,0.001375555992126,1500000
core-features/bm_app_raise.rb,6.0956289768219,6.09303498268127,6.094331979751587,0.001296997070312,n/a
core-library/bm_so_exception.rb,14.7507960796356,14.7960770130157,14.773436546325684,0.022640466690063,n/a
core-library/bm_so_concatenate.rb,Error: failed to allocate memory,,,,5000
core-library/bm_so_concatenate.rb,Error: failed to allocate memory,,,,10000
core-library/bm_so_concatenate.rb,Error: string sizes too big,,,,15000
core-library/bm_so_count_words.rb,12.6491630077362,12.6552991867065,12.652231097221375,0.003068089485168,n/a
core-library/bm_vm2_array.rb,0.745777130126953,0.743591070175171,0.744684100151062,0.001093029975891,1000000
core-library/bm_vm2_array.rb,1.49140596389771,1.49024105072021,1.490823507308960,0.000582456588745,2000000
core-library/bm_vm2_array.rb,2.98199105262756,2.97910499572754,2.980548024177551,0.001443028450012,4000000
core-library/bm_vm2_array.rb,5.96128010749817,5.95619893074036,5.958739519119263,0.002540588378906,8000000
core-library/bm_vm2_regexp.rb,0.969479084014893,0.94690990447998,0.958194494247437,0.011284589767456,10
core-library/bm_vm2_regexp.rb,1.14880299568176,1.13046097755432,1.139631986618042,0.009171009063721,100
core-library/bm_vm2_regexp.rb,2.10414791107178,2.11249113082886,2.108319520950317,0.004171609878540,1000
core-library/bm_vm2_regexp.rb,12.8499979972839,12.8127069473267,12.831352472305298,0.018645524978638,10000
core-library/bm_vm3_thread_create_join.rb,0.0200490951538086,0.0200908184051514,0.020069956779480,0.000020861625671,1000
core-library/bm_vm3_thread_create_join.rb,0.20678186416626,0.203513860702515,0.205147862434387,0.001634001731873,10000
core-library/bm_vm3_thread_create_join.rb,2.04601502418518,2.04799294471741,2.047003984451294,0.000988960266113,100000
core-library/bm_app_strconcat.rb,5.07870984077454,5.0784158706665,5.078562855720520,0.000146985054016,n/a
core-library/bm_so_lists.rb,14.3261790275574,14.3377418518066,14.331960439682007,0.005781412124634,n/a
core-library/bm_so_matrix.rb,2.74682712554932,2.76162505149841,2.754226088523865,0.007398962974548,n/a
core-library/bm_pathname.rb,9.27168798446655,9.26259899139404,9.267143487930298,0.004544496536255,n/a
core-library/bm_so_array.rb,8.83219408988953,8.83007097244263,8.831132531166077,0.001061558723450,n/a

=end

[rogerdpack (Roger Pack)]

=begin

You ran this benchmark suite, correct?

http://github.com/acangiano/ruby-benchmark-suite/tree/master

Yeah, that and http://lloydforge.org/projects/misc/
the latter taking considerably less time to run :)

I don't believe that these patches cause GC to run any less frequently by
default.
GC is still run (by default) after allocating 8MB of objects. Nothing I'm
doing causes Ruby to allocate fewer or smaller objects. I do believe we are
seeing that applications with large stack space(s) spend a lot of time
during GC scanning each and every word on those stacks. These patches make
those stacks much smaller and zero out most ghost object pointers so they no
longer need to be marked.

It would be interesting to see if the GC is being caused by malloc
versus running out of free list. If it's the latter then the patches
could indeed cause GC to be less frequent. If not then maybe it's as
you said--GC just takes less time as there's less to traversal during
the mark phase.

Brent: A >14x speed up. Whoopie! :-0
Yeah I think multi-thread apps will definitely like this.
Unfortunately most benchmarks are single threaded and micro-y so won't
show the "real" speedup [Antonio's included].

Brent: I don't think that clearing the stack once would be sufficient.
And, "clearing the stack" is a bit misleading. The unused memory area
between the current stack pointer and its deepest recorded extent is being
filled with zeros. It's not really part of the stack when it is cleared.
This memory will become stack as new frames are pushed, thus approximating
the effect of an imaginary compiler option to initialize all local
variables and temporaries to zero. So, I've got to clear the stack when
it is shallow and record its depth when it is deep. Recording the depth
is very quick -- only about five machine instructions. For single
threaded apps, I could perhaps figure out, when the stack was shallow,
that a GC was about to occur, and get away with zeroing the stack at that
one point. However, recall that the collector scans each thread's stack
in multithreaded apps (and those using Continuations). So, I'd need to
know when a GC or a context switch was going to occur while the stack was
still shallow. I haven't figured out how to implement that oracle
function (and I doubt it is possible).

Hmm so the biggest speed hit is probably in the clearing of the stack
[over and over] right? [judging from your comment that measurement is
cheap].

I was just suggesting that once a thread has [reached a very shallow
spot and cleaned the stack in its entirety] it only needs to repeat
that after the next GC--left over references from this round will be
cleared [once] after the subsequent GC (when the thread reaches a
shallow point again). So if you're willing to wait a couple of GC's,
you only have to clear once per GC, per thread. So the oracle is "do
it once after each GC."
Sorry it's hard to explain.

Anyway imagine a single threaded app. As long as that app clears the
stack "once and well" [say the first time it gets very high it cleans
off the whole thing--or accomplish this piece-wise as it grows high
the first time] then in a staggered way, every reference to garbage
will eventually be zeroed out and the item collected.
Not that it really matters I'm just trying to make sure that my
thought has been explained well.
thoughts?
-=r

=end

[rogerdpack (Roger Pack)]

=begin
Hmm interesting.
So I was looking at it from the single threaded perspective so
obviously missed some subtle implications.

If I understand correctly, the problem is that

1. If you have a large stacked thread "full of garbage" then this
   garbage will be copied into the stack of a small stack after context
   switch if it grows.
2. If a single thread creates a very "dirty" stack then goes into a
   deep nested loop [ex: going to sleep forever within a very nested
   call], it will not free the invalid references until it comes out of
   that deep stack later.

I suppose we can operate under the assumption that when the program
starts, the extent of the stack is "clear" of bad references.

A few tricks up our sleeve:
We can do a stack cleaning around the time of a context switch:
We can clear the difference in size between the stacks after each
context switch.
We could clear that difference PLUS re-clear the "cleared once" area
below the stack, after each context switch.

Or perhaps do the "clear at most once" trick only if rb_thread_alone,
though I think the above would already do that.

So anyway we could basically reset the "already cleared" markers once
per context switch, instead of once per GC, and re-clear that stacks
damage. Would that help?
In reality I'm not sure if these would be necessary. How can we tell
how much is necessary?

Old notes:

So let's then keep two values, per thread. One being the top of a
"clean section" the other the bottom of the "clean section" [already
swept section].

Make this "clean section" grow as possible [check it every CHECK_INT,
if you're above it, grow it, if you're below it, reset it to start
below you, etc.]. So we have track of, per thread, a growing cleaned
area.

Now when you context switch, if you switch from a large stack to a
shorter stack, clean the difference, plus the "dirty but clean now"
section--clean it again. Reset the pointers.

I guess just try it out :) Or I might get around to it eventually.

Comments inline:

My bogus2 benchmark switches between one thread having a very deep stack and
another with a shallow stack. It's the worst conceivable case of stack
thrashing. It runs about 15% faster if I disable only the clearing of the
stack.

I wonder if that's what causes the micro-benchmark slowdowns [what are
they like 5%?] What about disabling the depth checker, too? What's
its impact?

When whatever transient ghost references remain, change value, GC will
eventually collect the objects to which they referred. Correct?

Yeah

2. GC is not triggered by any thread's particular activities. It may be
   that a given thread, whose stack has become full of ghost references due to
   deferred stack clearing, stops running for long periods of time. Or, that a
   such a thread just never happens to be running when a GC is triggered.

True if a thread "doesn't run at all" between GC's then it won't clear
its stack until...it runs again at some point :)
A thread basically gets a window of 1 GC to create as much trash as it
wants, and, if it ceases running, retains that much trash.

-=r

=end

[headius (Charles Nutter)]

=begin
Stephen Sykes wrote:

Brent,

A report from the field...

We have been using your patches in a production Rails environment
since you released them, and this is on x86_64-linux.

We notice no problems, ruby works well and is significantly faster.

The patches also appear to help method-call performance a bit:

BEFORE:
$ ./ruby -I lib ../jruby/bench/language/bench_method_dispatch_only.rb
Test ruby method: 100k loops calling self's foo 100 times
1.580000 0.010000 1.590000 ( 1.619531)
1.570000 0.000000 1.570000 ( 1.609721)
1.610000 0.010000 1.620000 ( 1.627628)
1.570000 0.010000 1.580000 ( 1.600705)
1.580000 0.000000 1.580000 ( 1.601550)
1.570000 0.010000 1.580000 ( 1.597049)
1.570000 0.010000 1.580000 ( 1.608728)
1.570000 0.010000 1.580000 ( 1.594988)
1.570000 0.000000 1.570000 ( 1.601885)
1.570000 0.010000 1.580000 ( 1.630782)
[headius @ 247:~/projects/ruby-1.8.7-p72]
$ ./ruby -I lib ../jruby/bench/bench_tak.rb 10
user system total real
13.510000 0.060000 13.570000 ( 13.768144)
13.530000 0.070000 13.600000 ( 13.773649)

AFTER:
$ ./ruby -I lib ../jruby/bench/language/bench_method_dispatch_only.rb
Test ruby method: 100k loops calling self's foo 100 times
1.360000 0.010000 1.370000 ( 1.416073)
1.350000 0.000000 1.350000 ( 1.381519)
1.360000 0.010000 1.370000 ( 1.376705)
1.350000 0.000000 1.350000 ( 1.380676)
1.350000 0.010000 1.360000 ( 1.377904)
1.360000 0.010000 1.370000 ( 1.465818)
1.350000 0.000000 1.350000 ( 1.379431)
1.350000 0.010000 1.360000 ( 1.372702)
1.340000 0.010000 1.350000 ( 1.374763)
1.350000 0.010000 1.360000 ( 1.376614)
$ ./ruby -I lib ../jruby/bench/bench_tak.rb 10
user system total real
12.860000 0.060000 12.920000 ( 13.091790)
12.950000 0.060000 13.010000 ( 13.241058)

For comparison, Ruby 1.9.1RC1 numbers:

$ ruby1.9 ../jruby/bench/language/bench_method_dispatch_only.rb
Test ruby method: 100k loops calling self's foo 100 times
0.650000 0.000000 0.650000 ( 0.665707)
0.650000 0.010000 0.660000 ( 0.657540)
0.650000 0.000000 0.650000 ( 0.662093)
0.650000 0.010000 0.660000 ( 0.667457)
0.650000 0.000000 0.650000 ( 0.670909)
0.650000 0.000000 0.650000 ( 0.665737)
0.650000 0.010000 0.660000 ( 0.664140)
0.650000 0.000000 0.650000 ( 0.667239)
0.650000 0.000000 0.650000 ( 0.662808)
0.650000 0.010000 0.660000 ( 0.661229)
$ ruby1.9 ../jruby/bench/bench_tak.rb 10
user system total real
2.900000 0.010000 2.910000 ( 2.951113)
2.890000 0.020000 2.910000 ( 2.958036)

• Charlie

=end

[hramrach (Michal Suchanek)]

=begin
2009/1/14 Brent Roman [e-mail]:

Could you post some (brief) PPC OSx benchmark results
comparing runtime and peak process size before and after patching, taking
care to build ruby with the same compiler options each time?

• brent

How do you measure peak process size?

I have an application that takes about an hour to run and requires
about 2G RSS with ruby 1.8, and about half with JRuby.

I would be interested in comparing the performance with and without the patch.

Thanks

Michal

=end

[rogerdpack (Roger Pack)]

=begin
Here's my field report.
I have a small rails app on a linode slice. After running it awhile I noticed that the system stopped responding--it was running out of RAM.

For some reason my rails app was growing by 8MB of RSS per request. If anybody wants to look into this in more depth I'd be happy to give them access.

Updating to 187 trunk: same result.
Updated to 187 + MBARI patches. Problem gone.
Also the total RSS now starts at 59MB and [4 days later] has appeared to stabilize at 62MB. Without patches it starts at 78MB, so a 25% RAM use reduction, which is very nice for those on slices.

I'd encourage the inclusion of these patches into trunk for the next patch release.

A few thoughts on compiler differences: would using the SET_STACK_END macros help? Maybe it could revert to a method call [so force go down on stack] as a way to check the stack end? Or just always add to 20 to what alloca returns or what not?

re: measure peak process size: sys-proctable might help.

Thanks much for your work. It spared me hours of debugging and has improved my opinion of Ruby. Three cheers :)
Where to send donation?

-=r
=end

[nobu (Nobuyoshi Nakada)]

=begin
Hi,

At Mon, 19 Jan 2009 17:15:02 +0900,
Brent Roman wrote in [ruby-core:21429]:

I have just posted yet another update to the MBARI7 patch at:
http://sites.google.com/site/brentsrubypatches/

Can't you make patches against the head of stable branch?

Current status:

MBARI1: already merged except for a new method.

MBARI2: backported stack-rewind at thread creation from old
1.9, so I think this patch is no longer needed.

MBARI4: your patch makes Emacs c-mode.el confused.
http://www.atdot.net/sp/readonly/rb_eval_split is
more c-mode.el friendly.

MBARI5: already merged.

And could you separate new features from bug fixes?

--
Nobu Nakada

=end

[hramrach (Michal Suchanek)]

=begin
2009/1/19 Nobuyoshi Nakada [e-mail]:

Hi,

At Mon, 19 Jan 2009 17:15:02 +0900,
Brent Roman wrote in [ruby-core:21429]:

I have just posted yet another update to the MBARI7 patch at:
http://sites.google.com/site/brentsrubypatches/

Can't you make patches against the head of stable branch?

Current status:

MBARI1: already merged except for a new method.

MBARI2: backported stack-rewind at thread creation from old
1.9, so I think this patch is no longer needed.

MBARI4: your patch makes Emacs c-mode.el confused.
http://www.atdot.net/sp/readonly/rb_eval_split is
more c-mode.el friendly.

Perhaps it should be the other way around?

That is the Emacs c-mode should be fixed to work with any code rather
than code modified to work around Emacs quirks.

Thanks

Michal

=end

[nobu (Nobuyoshi Nakada)]

=begin
Hi,

At Tue, 20 Jan 2009 19:33:50 +0900,
Michal Suchanek wrote in [ruby-core:21457]:

That is the Emacs c-mode should be fixed to work with any code rather
than code modified to work around Emacs quirks.

By implementing C preprocessor in emacs lisp?
Nice challenge. :)

--
Nobu Nakada

=end

[nobu (Nobuyoshi Nakada)]

=begin
Hi,

At Wed, 21 Jan 2009 18:21:19 +0900,
Brent Roman wrote in [ruby-core:21483]:

Regarding the patches already applied to HEAD:

MBARI1:
Can you explain why the Continuation#thread method is not acceptable?
It does seem to be an intrinsic property of every Continuation and
without this method, one must often maintain a separate (weak) reference
to the thread on which each continuation operates.

I don't say it's not acceptable. It's not a part of the bug
fix, so should be another request.

MBARI2:
I like push/pop_thread_anchor() better than my hack to hide other threads' stacks.
However, I don't see code in rb_thread_save_context() to copy only the active
stack for each thread. This is a very important optimization.
Are you doing this optimization some other way that I am overlooking?
(To see how important it can be, try running my bogus1.rb and bogus2.rb benchmarks)

What do you mean by "active stack"? The stack region which is
actually used by thread? The current code reduces those erea
by rewinding the stack.

MBARI4:
I'll be happy to incorporate your clever eval_body() #define.
It cleans the static inline function decls up nicely. Does it also restore the Emacs c-mode.el
compatibility? If this isn't what bothers emacs, please explain, and I'll try to code around it.
(Please understand that I haven't used emacs in any serious way for 15 years,
when I discovered nedit :-)

Compatibility against older Emacs? I can't test it now.

--
Nobu Nakada

=end

[hramrach (Michal Suchanek)]

=begin
2009/1/22 Brent Roman [e-mail]:

Michal,

I've got no immediate plans to port these patches to 1.8.6.
Why is this important for you? I (perhaps naively) thought 1.8.7
would run just about anything that 1.8.6 does.

It's far from that simple.

1.8.7 backports a few 1.9 features that were "easy enough" to backport
breaking quite a bit of valid 1.8 code.

Sure the code can be updated easily in most cases but there is large
portion of code that hits the differences and cannot just run on 1.8.7
untouched.

Thanks

Michal

=end

[nobu (Nobuyoshi Nakada)]

=begin
Hi,

At Fri, 23 Jan 2009 05:36:05 +0900,
Brent Roman wrote in [ruby-core:21530]:

Hi Nobu,

MBARI2:
It was late when I compared the patches. If you are actually
rewinding the stack, that's even better than this patch's technique
of linking directly to the base frame to skip around the parent
threads stacks while leaving them in place.

MBARI4:
I meant that I don't use emacs anymore, so I won't test against it.
Even so, I wish you would explain what about this patch confused
Emacs c-mode.el so I can avoid such constructs in future.
I'm guessing it had something to do with the NOINLINE function
declarations, but I'm still not sure.

I seemed missing something. It indents like:

NOINLINE(static VALUE
eval_match2(self, node))
VALUE self;
NODE *node;

This isn't bad too much, but c-beginning-of-defun jumps to the
beginning of the line `NODE *node;' line, not eval_match2.

Also, since VC8 needs prototype declaration or definition
for noinline, your patch causes compile error with it.

I won't object you even if you were propose to drop the

support for VC8 or later :)

Do you think you will merge these changes into the 1.8.6 release?

We'll have to merge them into the 1.8 head first.

--
Nobu Nakada

=end

[brent (Brent Roman)]

=begin

I cannot seem to build a working Ruby 1.8.7 for the PPC64 under Leopard 10.5
Has anyone else managed it?

It runs simple test scripts, but hangs on the test suite.
This is 1.8.7-p72 without any patches.

Here's how I'm building:

$ export ARCHFLAG="-arch ppc64"
$ CFLAGS="-O2 -m64 -fno-stack-protector" configure --prefix=$HOME
$ make
$ sudo make install
$ ruby -v
ruby 1.8.7 (2008-08-11 patchlevel 72) [powerpc-darwin9.6.0]

$ uname -a
Darwin G5-Client.shore.mbari.org 9.6.0 Darwin Kernel Version 9.6.0: Thu Nov
6 19:35:49 PST 2008; root:xnu-1228.9.57~1/RELEASE_PPC Power Macintosh

$ gcc -v
Using built-in specs.
Target: powerpc-apple-darwin9
Configured with: /var/tmp/gcc/gcc-5465~16/src/configure --disable-checking
-enable-werror --prefix=/usr --mandir=/share/man
--enable-languages=c,objc,c++,obj-c++
--program-transform-name=/^[cg][^.-]*$/s/$/-4.0/
--with-gxx-include-dir=/include/c++/4.0.0 --with-slibdir=/usr/lib
--build=i686-apple-darwin9 --program-prefix= --host=powerpc-apple-darwin9
--target=powerpc-apple-darwin9
Thread model: posix
gcc version 4.0.1 (Apple Inc. build 5465)

$ cd test
$ time ruby runner.rb
HANGS here.....

Any ideas?

• brent

--
View this message in context: http://www.nabble.com/-ruby-core%3A19846---Bug--744--memory-leak-in-callcc--tp20447794p21742003.html
Sent from the ruby-core mailing list archive at Nabble.com.

=end

[brent (Brent Roman)]

=begin

Michal,

I finally managed to get a working Ruby 1.8.7 on ppc64 with and without
MBARI patches
under OSx 10.5 as follows:

export ARCH_FLAG="-arch ppc64"
CFLAGS="-O2 -g -m64 -fno-stack-protector" configure

There may yet be a problem with my build configuration, but I don't think
the MBARI patches have anything to do with these failures.
Both patched and unpatched versions fail the same 6 tests.
Does anyone have a ppc64 (64-bit code) Ruby that does not fail these tests?

Note that I am using the PowerPC patch of 1/23/09. For now, it must be
applied manually
after the MBARI7 patch. I will integrate it after it has been tested on
x86_64.
(It should be called the 64-bit patch, as it is intended to fix x86_64 as
well as ppc64)

http://sites.google.com/site/brentsrubypatches/Home/ruby-1.8.7-p72-mbariPPC.patch?attredirects=0

The PPC patch changes the meaning of the STACK_WIPE_SITES #define.
See rubysig.h for details.

One interesting observation is that my unpatched ppc64 ruby did not leak
when executing:

ruby -e "loop{@x=callcc{|c|c}}"

This could be because the ppc versions put ruby call arguments on the heap
rather than the 'C' stack.

• brent

results:

The patched ppc64 Ruby runs the test suite about 30 seconds quicker.
341 vs. 312 seconds
It used a bit less RAM, but the difference wasn't large:
114Mb vs. 106Mb peak VSIZE

I've included the details of the run below.
Can anyone verify whether or not these failures occur with unpatched
1.8.7-p72?

---

$ uname -a
Darwin G5-Client.shore.mbari.org 9.6.0 Darwin Kernel Version 9.6.0: Thu Nov
6 19:35:49 PST 2008; root:xnu-1228.9.57~1/RELEASE_PPC Power Mac

$ ruby -v
ruby 1.8.7 (2009-1-23 MBARI 7/0x5770 on patchlevel 72) [powerpc-darwin9.6.0]

$ file ~/bin/ruby
/u/brent/bin/ruby: Mach-O 64-bit executable ppc64

$ time ruby runner.rb
Loaded suite .
Started
........................................................................................................................................................................................................................................................................................................................................................F..........................Warning:
OpenSSL::PKCS7::PKCS7 is deprecated after Ruby 1.9; use OpenSSL::PKCS7
instead
.Warning: OpenSSL::PKCS7::PKCS7 is deprecated after Ruby 1.9; use
OpenSSL::PKCS7 instead
.Warning: OpenSSL::PKCS7::PKCS7 is deprecated after Ruby 1.9; use
OpenSSL::PKCS7 instead
Warning: OpenSSL::PKCS7::PKCS7 is deprecated after Ruby 1.9; use
OpenSSL::PKCS7 instead
Warning: OpenSSL::PKCS7::PKCS7 is deprecated after Ruby 1.9; use
OpenSSL::PKCS7 instead
..............................FF....F..................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................../ruby/test_array.rb:536:
warning: given block not used
........................................................................F.........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................E........................................................................................................................................................................................................................................................................................................................................................................................
Finished in 312.740792 seconds.

1. Failure:
   test_decode(OpenSSL::TestASN1) [./openssl/test_asn1.rb:195]:
   <"\217\a\362Q38\262\332\212H6N\244\022n\267\343I8\233\000\017|\361\265\024\335\353\202\237h\016\201\032bxV\300\343N\252\227w\320\263\241%\035s\366P\2147>dy\306\004\023\367\267\v\214\272\fY\331\326\016\346\216\003\310\323\ek+Y}is\361\263\034\313\f\006e\200V\274\302\222\201\314\260\350\210\321<G\317\024\260H\371+\002\350\210\216cHk\375\246\301\324c\363\324\203\225\330\221
   \036"> expected but was
   <"\246\317\022M\337\207
   \202\022\374\221\214\375\365\307\231\030\375t\027\306Y.\022\302\207\377\224\234\370l\a\211\r\241\225\003\220d\323k\346[>\351\004M\v\347\336\240\365\265\242\226\324?\214eR\300p\003!m#\217\e6\250\306G\324#\004\273\240\376\357\265\367\3658\275t?\342\274\335.\370\261\227\325)V\376\240Z\276\206\2056b\305\022s\tY%\025r\207\267\323\226\315\243L\203\023\306K">.

2. Failure:
   test_create_by_factory(OpenSSL::TestX509Extension)
   [./openssl/test_x509ext.rb:41]:
   <"0\022\006\003U\035\023\001\001\000\004\b0\006\001\001\000\002\001\002">
   expected but was
   <"0\022\006\003U\035\023\001\001\377\004\b0\006\001\001\377\002\001\002">.

3. Failure:
   test_new(OpenSSL::TestX509Extension) [./openssl/test_x509ext.rb:29]:
   expected but was
   .

4. Failure:
   test_attr(OpenSSL::TestX509Request) [./openssl/test_x509req.rb:94]:
   <[["keyUsage", "Digital Signature, Key Encipherment", true],
   ["subjectAltName", "email:[e-mail]", false]]> expected but
   was
   <[["keyUsage", "Digital Signature, Key Encipherment", false],
   ["subjectAltName", "email:[e-mail]", false]]>.

5. Failure:
   test_should_propagate_signaled(TestBeginEndBlock)
   [./ruby/test_beginendblock.rb:81]:
   <""> expected to be =~
   </Interrupt$/>.

6. Error:
   test_fd_passing(TestUNIXSocket):
   SocketError: file descriptor was not passed (msg_controllen=20, 24 expected)
   ./socket/test_unix.rb:19:in recv_io' ./socket/test_unix.rb:19:in test_fd_passing'

1976 tests, 1668917 assertions, 5 failures, 1 errors

real 5m23.872s
user 3m54.124s
sys 0m27.542s

Bugzilla from [e-mail] wrote:

I've tried that myself but it didn't work very well
(ruby test/runner.rb fails 3 tests on 0x2770, and segfaults when i use
0x4770,
on x86_64 machine)

I also tried building on ppc64, with 0x4770 it wont even build, segfaults
on
launching miniruby:

gcc -O2 -g -DRUBY_EXPORT -D_GNU_SOURCE=1 -L. -rdynamic -Wl,-export-
dynamic main.o libruby-static.a -ldl -lcrypt -lm -o miniruby
./ext/purelib.rb:2: [BUG] Segmentation fault
ruby 1.8.7 (2009-1-18 MBARI 7/0x4770 on patchlevel 72) [powerpc64-linux]
make: *** [.rbconfig.time] Aborted

With 0x2770 it builds & runs the same test suite with 6 failures & 1
error.
(Although i'm not sure how much they are actually ruby's fault)

Regards,
-- mb

--
View this message in context: http://www.nabble.com/-ruby-core%3A19846---Bug--744--memory-leak-in-callcc--tp20447794p21773950.html
Sent from the ruby-core mailing list archive at Nabble.com.

=end

[yugui (Yuki Sonoda)]

• Assignee set to matz (Yukihiro Matsumoto)
• ruby -v set to -

=begin

=end

[brent (Brent Roman)]

=begin

Roger,

With native threading, each thread gets its own private stack managed by the
OS.
So, yes, in Ruby 1.9, there should not be any ghost references from one
thread's stack creeping onto another's. However, there is still the
potential for ghost object references within any given thread's stack.

GC.limit= determines that number of bytes that will be allocated (or
reallocated)
before a garbage collection pass is automatically triggered. It defaults to
8e6 bytes. I set it to 2e6 bytes on memory limited embedded targets.
The process size will "breathe" by this amount of bytes while Ruby runs.
Some might want to breathe deeper (and less often) if they've got bigger
lungs.
GC.limit is documented in ri as such.
It is the primary GC tunable. If someone introduces a free list limit, they
can call it GC.freelist_limit. I'm would not be confused by that.

Nonetheless, if a couple more folks complain, I'll change GC.limit to
something longer.
I expect that it will get renamed in any case if it makes it into the
thrunk.

• brent

Roger Pack wrote:

On Mon, Jan 19, 2009 at 1:15 AM, Brent Roman [e-mail] wrote:

Yuki and Roger,

I'm glad to hear these patches are working out well for you.

I assume that with 1.9 this style patch isn't as necessary as threads
don't
"share garbage" between each other--is that right? [each thread could
still
clean itself, but at least they don't share garbage between threads--is
that
right?]

Also I might recommend renaming GC#limit to GC#malloc_limit or
GC#alloc_limit since "limit" is somewhat ambiguous--is it a limit to the
number of free pointers it will use? malloc size? [that type of thing].
Thanks so much!
-=r

--
View this message in context: http://www.nabble.com/-ruby-core%3A19846---Bug--744--memory-leak-in-callcc--tp20447794p21679891.html
Sent from the ruby-core mailing list archive at Nabble.com.

=end

[brent (Brent Roman)]

=begin

I just updated the MBARI7 patch for Ruby 1.8.7-p72 at:

http://sites.google.com/site/brentsrubypatches/

I've tested this February 9, 2009 compiling with GNUC targeting the
following CPU types:

ppc, ppc64, arm, i386, and x86_64

For each CPU, no more errors test suite errors occurred patched than
unpatched.

I'd welcome any feedback on ppc or x86_64 in particular.
[If you run into trouble, please include the output of gcc -v and uname -a]

I'm working on github release next, including patches for 1.8.6
Is 1.8.6-p287 (patchlevel 287) the specific version I should target?

• brent

--
View this message in context: http://www.nabble.com/-ruby-core%3A19846---Bug--744--memory-leak-in-callcc--tp20447794p21928710.html
Sent from the ruby-core mailing list archive at Nabble.com.

=end

[brent (Brent Roman)]

=begin

Michael,

I have just posted the MBARI patches on GitHub at:

http://github.com/brentr/matzruby/tree/v1_8_7_72-mbari

I believe you can pull from it via this git URL:

git://github.com/brentr/matzruby.git

A few points regarding your difficulties with the porting the MBARI patches
to 1.8.6:

1. Your report helped me identify the cause of the test failure in
   TestBeginEndBlock.
   There was always a bit of a race condition in handling the
   ruby/suicide.rb test case
   (Would CHECK_INTS get called before the interpreter terminated?)
   The big rb_eval() refactoring of MBARI4 moved the point at which
   CHECK_INTS is invoked
   and made that race much more likely. Even so, it always worked
   sometimes :-)
   My fix is to invoke CHECK_INTS just after sending a signal to any
   process
   It's in the MBARI7 patch dated 2/13/09 on github. (Not yet on my
   website)

2. I never see the YAML failure here. That may be a problem unique to
   1.8.6
   or it may be an error in porting the patches.

3. This is the only failure I see and you don't list it:

4. Failure:
   test_client_session(OpenSSL::TestSSL)
   [./openssl/test_ssl.rb:426:in test_client_session' ./openssl/test_ssl.rb:417:in times'
   ./openssl/test_ssl.rb:417:in test_client_session' ./openssl/test_ssl.rb:129:in call'
   ./openssl/test_ssl.rb:129:in start_server' ./openssl/test_ssl.rb:416:in test_client_session']:
   is not true.

Any clues? I'm guessing that I'm missing a supporting library.

3. I'm working on a version of these patches for 1.8.6-p287 right now.
   Stay tuned...

Git seems to be behaving. If it's laughing at me, it is doing so behind my
back.
Please do try to build from my git repo and let me know how that goes.

• brent

Michael King-2 wrote:

I was attempting to backport your MBARI patches to 1.8.6 p287, which is
what
my company is currently using in production.

When I was backporting all 7 patches I was seeing these errors:

1. Failure:
   test_should_propagate_signaled(TestBeginEndBlock)
   [./test/ruby/test_beginendblock.rb:82]:
   <""> expected to be =~
   </Interrupt$/>.

2. Error:
   test_object_id_collision(YAML_Unit_Tests):
   RuntimeError: id collision in ordered map
   ./test/yaml/test_yaml.rb:1281:in `test_object_id_collision'

--
View this message in context: http://www.nabble.com/-ruby-core%3A19846---Bug--744--memory-leak-in-callcc--tp20447794p22010674.html
Sent from the ruby-core mailing list archive at Nabble.com.

=end

[brent (Brent Roman)]

=begin

Roger,

Ummm... Moving binaries between different CPUs doesn't always work.
How, exactly, did the host and target machines differ?

Anyway...
The version I just pushed to github at:

http://github.com/brentr/matzruby/tree/v1_8_7_72-mbari

passed the Ruby test suite earlier this week on ppc, ppc64, arm, i386, and
x86_64 CPUs.

As mentioned in my previous post, this version should fix the failing
TestBeginEndBlock test.
Could you try building from my github repo to verify this and
let me know if your bittorent test still fails?

I'm still new to the git stuff.
Please let me know whether I've set up my repository correctly.

• brent

Roger Pack wrote:

Here's an interesting one.
I built 1.8.7p72 with the mbari patches. Works fine on the computer
where it was built. If I run it on another computer on the same
network, same OS [slightly different cpu], it sometimes [depending on
the moon phase] results in:

[09:1721][rdp@ilab2:~/tmp_src]$ ruby driver.rb -pbitTorrent
--name=yanc_and_bittorrent_100_take2
/home/rdp/i386/lib/ruby/site_ruby/1.8/rubygems/specification.rb:48:
[BUG] terminated node (0xb7c3505c)
ruby 1.8.7 (2009-1-18 MBARI 7/0x4770 on patchlevel 72) [i686-linux]

Aborted

[13:2228][rdp@ilab1:]$ gcc -v
uReading specs from
/home/rdp/installs/lib/gcc/i686-pc-linux-gnu/3.4.6/specs
Configured with: ./configure --prefix=/home/rdp/installs
Thread model: posix
gcc version 3.4.6
[13:2228][rdp@ilab1:]$ uname -a
Linux ilab1 2.6.24-23-generic #1 SMP Mon Jan 26 00:13:11 UTC 2009 i686
GNU/Linux

make test-all clears except a few zlib errors [it isn't installed] and

4. Failure:
   test_should_propagate_signaled(TestBeginEndBlock)
   [./test/ruby/test_beginendblock.rb:81]:
   <""> expected to be =~
   </Interrupt$/>.

any thoughts?
Thanks!
-=r

--
View this message in context: http://www.nabble.com/-ruby-core%3A19846---Bug--744--memory-leak-in-callcc--tp20447794p22010810.html
Sent from the ruby-core mailing list archive at Nabble.com.

=end

[stepheneb (Stephen Bannasch)]

=begin
At 6:18 PM +0900 2/14/09, Brent Roman wrote:

Michael,

I have just posted the MBARI patches on GitHub at:

http://github.com/brentr/matzruby/tree/v1_8_7_72-mbari

I believe you can pull from it via this git URL:

git://github.com/brentr/matzruby.git

Please do try to build from my git repo and let me know how that goes.

Brent, thanks for putting them on github. This makes it very easy to
follow your work now.

Building it worked fine.

When I built the latest v1_8_7_72-mbari branch:

commit 6b169f9546ad52cb0edb9a19d48110e08f86a296
Author: Brent Roman [e-mail]
Date: Fri Feb 13 23:06:56 2009 -0800

and ran the latest full suite of rubyspecs on it I got 8 failures and
14 errors.

See the full output from mspec here: http://gist.github.com/64442

It doesn't look like your patches have much to do with those errors
... but I'm not sure. I haven't worked with 1.8.7 much. The trunk
version of 1.8.7 doesn't build and install correctly on my system.

Here's how I built and tested your branch:

I already have the matzruby git repo cloned so I added the mbari repo
as another remote, fetched and checked out the remote branch
v1_8_7_72-mbari into my working dir.

$ cd ruby/src/matzruby.git/
$ git remote add mbari git://github.com/brentr/matzruby.git
$ git remote -v
mbari git://github.com/brentr/matzruby.git
origin git://github.com/rubyspec/matzruby.git

$ git pull
$ git fetch mbari
$ git co -b v1_8_7_72-mbari mbari/v1_8_7_72-mbari

Built it and made sure it can print it's version:

$ autoconf && ./configure
--prefix=/Users/stephen/dev/ruby/builds/mri/v1_8_7_72-mbari
$ make clean && make && make install
$ /Users/stephen/dev/ruby/builds/mri/v1_8_7_72-mbari/bin/ruby -v
ruby 1.8.7 (2009-2-13 MBARI 7/0x8770 on patchlevel 72) [i686-darwin9.6.0]

Here's a summary of the files that have changed between Brent's mbari
branch and the tag v1_8_7_72:

$ git diff --stat v1_8_7_72
ChangeLog | 197 +++++
common.mk | 2 +-
eval.c | 2354 ++++++++++++++++++++++++++++++++----------------------
gc.c | 589 ++++++++-------
intern.h | 2 +-
missing/alloca.c | 8 +-
node.h | 6 +-
rubysig.h | 212 +++++-
signal.c | 3 +-
version.h | 17 +-
10 files changed, 2123 insertions(+), 1267 deletions(-)

A closer look at the changes in my favorite diff viewer (GitX):

$ git diff v1_8_7_72 | gitx

Run the latest rubyspecs against it

$ cd /Users/stephen/dev/ruby/src/rubyspec.git
$ which mspec
/Users/stephen/dev/ruby/src/mspec.git/bin/mspec

$ git pull
Already up-to-date.

Running just the core rubyspec tests:

$ mspec -t /Users/stephen/dev/ruby/builds/mri/v1_8_7_72-mbari/bin/ruby core
ruby 1.8.7 (2009-2-13 MBARI 7/0x8770 on patchlevel 72) [i686-darwin9.6.0]
..EE..EE.....................................................................E........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................F.......................................................................................................................................................................................................................................................
.........................................................................................................................................

ARGF.bytes returns an Enumerable::Enumerator when passed no block ERROR
NoMethodError: undefined method be_an_instance_of' for #<Object:0x3409b8> ./core/argf/shared/each_byte.rb:41 ./core/argf/shared/each_byte.rb:39 ./core/argf/bytes_spec.rb:2:in all?'
./core/argf/bytes_spec.rb:5
./core/argf/bytes_spec.rb:4

ARGF.chars returns an Enumerable::Enumerator when passed no block ERROR
NoMethodError: undefined method be_an_instance_of' for #<Object:0x33acd4> ./core/argf/shared/each_char.rb:32 ./core/argf/shared/each_char.rb:30 ./core/argf/chars_spec.rb:2:in all?'
./core/argf/chars_spec.rb:5
./core/argf/chars_spec.rb:4

ARGF.each_byte returns an Enumerable::Enumerator when passed no block ERROR
NoMethodError: undefined method be_an_instance_of' for #<Object:0x330d60> ./core/argf/shared/each_byte.rb:41 ./core/argf/shared/each_byte.rb:39 ./core/argf/each_byte_spec.rb:2:in all?'
./core/argf/each_byte_spec.rb:4

ARGF.each_char returns an Enumerable::Enumerator when passed no block ERROR
NoMethodError: undefined method be_an_instance_of' for #<Object:0x32ddcc> ./core/argf/shared/each_char.rb:32 ./core/argf/shared/each_char.rb:30 ./core/argf/each_char_spec.rb:2:in all?'
./core/argf/each_char_spec.rb:5
./core/argf/each_char_spec.rb:4

An exception occurred during: before :all ERROR
LoadError:
dlopen(/Users/stephen/dev/ruby/builds/mri/v1_8_7_72-mbari/lib/ruby/1.8/i686-darwin9.6.0/dl.bundle,
9): Symbol not found: _rb_DLStdcallCallbackProcs
Referenced from:
/Users/stephen/dev/ruby/builds/mri/v1_8_7_72-mbari/lib/ruby/1.8/i686-darwin9.6.0/dl.bundle
Expected in: flat namespace

/Users/stephen/dev/ruby/builds/mri/v1_8_7_72-mbari/lib/ruby/1.8/i686-darwin9.6.0/dl.bundle
/Users/stephen/dev/ruby/builds/mri/v1_8_7_72-mbari/lib/ruby/1.8/i686-darwin9.6.0/dl.bundle
./core/array/pack_spec.rb:2363
./core/array/pack_spec.rb:2272:in `all?'
./core/array/pack_spec.rb:2426

An exception occurred during: before :all ERROR
LoadError:
dlopen(/Users/stephen/dev/ruby/builds/mri/v1_8_7_72-mbari/lib/ruby/1.8/i686-darwin9.6.0/dl.bundle,
9): Symbol not found: _rb_DLStdcallCallbackProcs
Referenced from:
/Users/stephen/dev/ruby/builds/mri/v1_8_7_72-mbari/lib/ruby/1.8/i686-darwin9.6.0/dl.bundle
Expected in: flat namespace

/Users/stephen/dev/ruby/builds/mri/v1_8_7_72-mbari/lib/ruby/1.8/i686-darwin9.6.0/dl.bundle
/Users/stephen/dev/ruby/builds/mri/v1_8_7_72-mbari/lib/ruby/1.8/i686-darwin9.6.0/dl.bundle
./core/array/pack_spec.rb:2363
./core/array/pack_spec.rb:2363:in `all?'
./core/array/pack_spec.rb:2475

Module#autoload shares the autoload request across dup'ed copies of
modules FAILED
Expected NameError
but got TypeError (wrong autoload table:
#Proc:0x005358a4@./core/module/autoload_spec.rb:252)
./core/module/autoload_spec.rb:252
./core/module/autoload_spec.rb:238:in `all?'
./core/module/autoload_spec.rb:15

Finished in 13.107626 seconds

1127 files, 5697 examples, 19595 expectations, 1 failure, 6 errors

=end

[brent (Brent Roman)]

=begin

Roger,

Yes, I ran into that too.
I just ran the tests from the 1.8.2p72 release.
Or, you can replace test/runner.rb with this:

require 'test/unit'

rcsid = %w$Id$
if rcsid[3]
Version = rcsid[2].scan(/\d+/).collect!(&method(:Integer)).freeze
Release = rcsid[3].freeze
end

exit Test::Unit::AutoRunner.run(true, File.dirname($0))

Git does not do CVS style keyword substitution.
runner.rb was relying on this.

• brent

Roger Pack wrote:

Brent, thanks for putting them on github. This makes it very easy to
follow
your work now.

I was able to fork and build it quite easily, too. I have a few
changes to the README in my fork if you'd like them :)
Unfortunately I can't seem to run tests for some reason:

[14:1409][rdp@ilab1:~/dev/mbari_patches_my_fork]$ make test-all
./miniruby -I./lib -I.ext/common -I./- -r./ext/purelib.rb
./runruby.rb --extout=.ext -- "./test/runner.rb" --basedir="./test"
--runner=console
./test/runner.rb:4: private method `scan' called for nil:NilClass
(NoMethodError)
make: *** [test-all] Error 1

But that doesn't matter since they weren't able to recreate the
collected bug anyway. Now that I think of it, there wasn't any real
difference I know of between the two machines. My hunch is that the
bug would be reproducible [in time] on both, though I've never seen it
on one.
Thanks!
-=r

--
View this message in context: http://www.nabble.com/-ruby-core%3A19846---Bug--744--memory-leak-in-callcc--tp20447794p22017266.html
Sent from the ruby-core mailing list archive at Nabble.com.

=end

[brent (Brent Roman)]

=begin

Stephan,

My acceptance test is that the test suite delivered with ruby produce no new
failures running patched vs. unpatched.

I really don't want to get into mspec. However,
just a cursory glance at the errors it output leads
me to believe it was testing against ruby 1.8.6 specs.
Have you tried this same mspec against unpatched 1.8.7-p72?

I built and tested with the following:

$ cd ruby
$ git clone git://github.com/brentr/matzruby.git mri.git
$ cd mri.git
$ git checkout -b v1_8_7_72-mbari origin/v1_8_7_72-mbari
$ autoconf
$ CFLAGS="-O2 -fno-stack-protector" configure --prefix=$HOME/ruby/stage
$ make -j3
$ make install
$ cd test
$ time ~/ruby/stage/bin/ruby runner.rb

Output:

1. Failure:
   test_client_session(OpenSSL::TestSSL)
   [./openssl/test_ssl.rb:426:in test_client_session' ./openssl/test_ssl.rb:417:in times'
   ./openssl/test_ssl.rb:417:in test_client_session' ./openssl/test_ssl.rb:129:in call'
   ./openssl/test_ssl.rb:129:in start_server' ./openssl/test_ssl.rb:416:in test_client_session']:
   is not true.

1985 tests, 1345472 assertions, 1 failures, 0 errors

real 4m10.124s
user 1m38.430s
sys 0m4.160s

This is the same single failure I've always seen with every 1.8.7-p72 Ruby
on my machine. I'm still hoping someone might tell me what
might cause this.

• brent

Stephen Bannasch-3 wrote:

At 6:18 PM +0900 2/14/09, Brent Roman wrote:

Michael,

I have just posted the MBARI patches on GitHub at:

http://github.com/brentr/matzruby/tree/v1_8_7_72-mbari

I believe you can pull from it via this git URL:

git://github.com/brentr/matzruby.git

Please do try to build from my git repo and let me know how that goes.

Brent, thanks for putting them on github. This makes it very easy to
follow your work now.

Building it worked fine.

When I built the latest v1_8_7_72-mbari branch:

commit 6b169f9546ad52cb0edb9a19d48110e08f86a296
Author: Brent Roman [e-mail]
Date: Fri Feb 13 23:06:56 2009 -0800

and ran the latest full suite of rubyspecs on it I got 8 failures and
14 errors.

--
View this message in context: http://www.nabble.com/-ruby-core%3A19846---Bug--744--memory-leak-in-callcc--tp20447794p22019733.html
Sent from the ruby-core mailing list archive at Nabble.com.

=end

[brent (Brent Roman)]

=begin

I just pushed out a version of the MBARI patches for Ruby 1.8.6-p287 onto:

git://github.com/brentr/matzruby.git

in the branch v1_8_6_287-mbari

I'm down to just one test failure:

1. Error:
   test_object_id_collision(YAML_Unit_Tests):
   RuntimeError: id collision in ordered map
   ./test/yaml/test_yaml.rb:1281:in `test_object_id_collision'

However, I'm not motivated to investigate much further because this
test fails on every version of 1.8.6-p287 I build from source on
four different linux boxes with varying versions of gcc including those
built directly from the archive:

ftp://ruby-lang.org/pub/ruby/ruby-1.8.6-p287.tar.bz2

On the other hand, I recall Michael King claimed to have gotten 1.8.6-p287
to complete
the test suite without any errors whatsoever.

I'm building like this:
$ CFLAGS="-O2 -fno-stack-protector" configure --prefix=$HOME/ruby/test
$ make -j3 && make install

(I've tried all sorts of CFLAGS, so please no comments about those)
and running the yaml test with:

$ cd test
$ ~/ruby/test/bin/ruby runner.rb yaml
Loaded suite yaml
Started
.........E................................................
Finished in 0.409985 seconds.

1. Error:
   test_object_id_collision(YAML_Unit_Tests):
   RuntimeError: id collision in ordered map
   ./yaml/test_yaml.rb:1281:in `test_object_id_collision'

58 tests, 206 assertions, 0 failures, 1 errors

If you try that on your Ruby 1.8.6-p287 built from source,
do you see the error?

Is there another way to build it from source that avoids the error?

Please respond with details on your build procedure, environment etc.
only if you've built Ruby 1.8.6-p287 from source and do not see the above
error.
If you've got it working, I'd sure like to how exactly how!

• brent

P.S. Note that this issue was supposedly fixed by a patch applied on
6/15/08.
That patch appears to be present in 1.8.6-p287.
See http://redmine.ruby-lang.org/issues/show/411
If no one responds, I'll add this report to redmine, but for now, I'm
assuming
I've got a problem with my build procedure.

--
View this message in context: http://www.nabble.com/-ruby-core%3A19846---Bug--744--memory-leak-in-callcc--tp20447794p22065464.html
Sent from the ruby-core mailing list archive at Nabble.com.

=end

[brent (Brent Roman)]

=begin

I used to think it was more, but in fact, -fno-stack-protector probably saves
less than 1% of execution time. The stack clearing of the MBARI patches
invokes alloca often, so any extra overhead there will be felt more than
without stack clearing.

Note also that the stack-protector stuff was added to gcc to
detect malicious attempts to hack the stack in 'C' code that processes
networking data. Ruby's stack cannot be hacked that way as all
array indecies are checked explicitly. So, in Ruby, gcc's
stack-protector is sort of like wearing a belt and suspenders.

• brent

Roger Pack wrote:

I'm building like this:
$ CFLAGS="-O2 -fno-stack-protector" configure --prefix=$HOME/ruby/test
$ make -j3 && make install

Question:
does the -fno-stack-protector stuff make much of a speed difference?
Thanks!
-=r

--
View this message in context: http://www.nabble.com/-ruby-core%3A19846---Bug--744--memory-leak-in-callcc--tp20447794p22086518.html
Sent from the ruby-core mailing list archive at Nabble.com.

=end

[mbabej (Michal Babej)]

=begin
On Saturday 14 of February 2009 08:17:22 Roger Pack wrote:

Here's an interesting one.
I built 1.8.7p72 with the mbari patches. Works fine on the computer
where it was built. If I run it on another computer on the same
network, same OS [slightly different cpu], it sometimes [depending on
the moon phase] results in:

[09:1721][rdp@ilab2:~/tmp_src]$ ruby driver.rb -pbitTorrent
--name=yanc_and_bittorrent_100_take2
/home/rdp/i386/lib/ruby/site_ruby/1.8/rubygems/specification.rb:48:
[BUG] terminated node (0xb7c3505c)
ruby 1.8.7 (2009-1-18 MBARI 7/0x4770 on patchlevel 72) [i686-linux]

Aborted

The moon has shifted phases since January :) Seriously though, I've also found
Jan 18 version to segfault/abort randomly on my x86_64, however latest from
git (Feb 15) is working very nice so far - only 2 failures
test_client_session(OpenSSL) and test_readline. Could you try the latest and
report the results ?

-- Michal

=end

[mbabej (Michal Babej)]

=begin
Hi,

On Friday 20 of February 2009 23:14:13 Roger Pack wrote:

The moon is in a good phase. LOL.
It does seem more stable using the latest version. I will report back
if the errors occur more.
Turns out, good moon phases end right after writing a positive feedback emails
:) Feb 15 ruby-mbari runs the full test suite with same errors as unpatched
ruby on my machine, but it still segfaults on some certain tests. E.g.
running "test/runner.rb net" in row quickly results in segfault.

P.S. i wrote a small script to see how ruby works with fork's copy-on-write
mechanism. It allocates an array of 1 mil float, then forks, and in the child
starts rewriting the array in batches (batch size is ARGV[0]). It gives
completely different results for mbari ruby, and i'd be glad if someone could
explain why :)

-- Michal

Attachment: array_test.rb
=end

[brent (Brent Roman)]

=begin

Roger,

I was unaware of the interaction between YYSTACK_USE_ALLOCA and the MBARI
patches.
Does anyone have a test case I can debug?

• brent

Roger Pack wrote:

... it has something to do with the same reason that GC
refuses if the yy_parse stack is on the stack [?] whatever that means,
anyway.

Thanks!
-=r

--
View this message in context: http://www.nabble.com/-ruby-core%3A19846---Bug--744--memory-leak-in-callcc--tp20447794p22138256.html
Sent from the ruby-core mailing list archive at Nabble.com.

=end

[brent (Brent Roman)]

=begin

Michal,

What you are seeing in unpatched ruby is memory leaking between your
"passes" in array_test.rb.
This is just another manifestation of the same leak that occurs with
unpatched ruby and the script:

loop do
@x=callcc{|c|c}
end

(see leakcheck.rb in the MBARIpatches tarball and the innocent redmine
entry at the top of this endless thread)

The uninitialized stack for iteration n+1 contains old (dead) object
references from
iteration n. The GC strings them all together into a linked list of object
references.
It therfore cannot collect any of them until the whole loop terminates.

The stack clearing patches break this bogus chain of stale object reference
links and
thus allow the GC to properly identify refs from previous iterations of the
loop as
being "dead".

I pushed an update to the patches onto github last night that seems to
improve
stability of the MBARI patches on the x86_64 platform. Others platforms
seem to be working
great, but the x86_64 still has exhibits vexing, very occasional segfaults.

I'll be working on it through this rainy weekend. If I can see it, I'm
confident I can (eventually)
fix it.

• brent

Bugzilla from [e-mail] wrote:

Hi,

On Friday 20 of February 2009 23:14:13 Roger Pack wrote:

The moon is in a good phase. LOL.
It does seem more stable using the latest version. I will report back
if the errors occur more.
Turns out, good moon phases end right after writing a positive feedback
emails
:) Feb 15 ruby-mbari runs the full test suite with same errors as
unpatched
ruby on my machine, but it still segfaults on some certain tests. E.g.
running "test/runner.rb net" in row quickly results in segfault.

P.S. i wrote a small script to see how ruby works with fork's
copy-on-write
mechanism. It allocates an array of 1 mil float, then forks, and in the
child
starts rewriting the array in batches (batch size is ARGV[0]). It gives
completely different results for mbari ruby, and i'd be glad if someone
could
explain why :)

-- Michal

--
View this message in context: http://www.nabble.com/-ruby-core%3A19846---Bug--744--memory-leak-in-callcc--tp20447794p22139637.html
Sent from the ruby-core mailing list archive at Nabble.com.

=end

[brent (Brent Roman)]

=begin

Aman,

When I merge the MBARI patches with 1.8 HEAD, I also plan to replace the
stack optimization introduced in the MBARI2 patch with the (better) thread
anchors already in HEAD (which, I think, were originally backported from
1.9). This should happen in the next week or so. In the meantime, you
might want to try this patch against the current (MBARI 8B) patches on 1.8.6
or 1.8.7:

http://www.nabble.com/file/p22385077/rmMBARI2.patch

It just disables the MBARI2 patch and leaves the rest intact.
It would be very helpful to find out whether or not that alone eliminates
God's segfaults.

Will you give this a try?
If it works, I'll do an 8C patch that to replace the stack splicing of
MBARI2 with stack anchors on 1.8.7-p72 and perhaps 1.8.6-p287 as well.

• brent

Aman Gupta-6 wrote:

I am continuing to see random segfaults on x86_64, especially with god
(http://god.rubyforge.org/), which makes liberal use of threads and
forking.

...

So far I've been unable to come up with a reproducible test case, but
I've managed to narrow the problem down to mbari2. Vanilla ruby 1.8.7
does not have this issue, whereas 1.8.7+mbari2 will segfault randomly
every few days.

Perhaps it is worth backporting thread anchors from ruby 1.8 HEAD?

Aman

--
View this message in context: http://www.nabble.com/-ruby-core%3A19846---Bug--744--memory-leak-in-callcc--tp20447794p22448384.html
Sent from the ruby-core mailing list archive at Nabble.com.

=end

[brent (Brent Roman)]

=begin

Aman,

It's quite possible that the double-frees are occurring both with and
without the MBARI2 patch, but they are not causing segfaults unless MBARI2
is applied. You may want to try using valgrind or some similar tool to
catch the double frees. (valgrind is really very good at this)

A few days ago, I pushed a branch to my github repo with the MBARI patches
applied to ruby_1_8 head. These patches use the "thread anchors" backported
(by Nobu, I believe) from 1.9 . It seems to be a bit slower than my
approach, but it may well be more robust. The branch is called
ruby_1_8-mbari:

git://github.com/brentr/matzruby.git

http://github.com/brentr/matzruby/commits/ruby_1_8-mbari/

It is a dev version, but this snapshot did pass the bundled ruby test suite
and all my tests as well.
It would give you the benefits of the MBARI2 patch via the thread anchors.

I'd really be must interested in finding out whether there are still
double-frees happening. Let me know what you find. If the double-frees
only happen with MBARI2 applied, I'll consider replacing MBARI2 with the
thread anchors from 1.8.8-dev

• brent

Aman Gupta-6 wrote:

I can confirm that removing mbari2 fixes the issue. I was able to get
a better stack trace, but am still unsure about the root cause and
unable to reproduce it consistently. It seems like a double free is
occurring for some reason and that eventually causes the segfault.

*** glibc detected *** free(): invalid pointer: 0x0000000002312734 ***
*** glibc detected *** free(): invalid pointer: 0x0000000002312734 ***

Core was generated by `ruby gems/local/gems/god-0.7.8/bin/god'.
Program terminated with signal 6, Aborted.
#0 0x00007fc3d0cbb07b in raise () from /lib/libc.so.6
(gdb) bt
#0 0x00007fc3d0cbb07b in raise () from /lib/libc.so.6
...

God uses a double-fork to spawn processes, and it looks like the
double free usually occurs when the first forked process (in
process.rb:215) dies. God also uses a C extension
(http://github.com/mojombo/god/blob/master/ext/god/netlink_handler.c)
which could be causing issues across the fork.

Aman

On Tue, Mar 10, 2009 at 8:46 PM, Brent Roman [e-mail] wrote:

Aman,

When I merge the MBARI patches with 1.8 HEAD, I also plan to replace the
stack optimization introduced in the MBARI2 patch with the (better)
thread
anchors already in HEAD (which, I think, were originally backported from
1.9). This should happen in the next week or so. In the meantime, you
might want to try this patch against the current (MBARI 8B) patches on
1.8.6
or 1.8.7:

http://www.nabble.com/file/p22385077/rmMBARI2.patch

It just disables the MBARI2 patch and leaves the rest intact.
It would be very helpful to find out whether or not that alone eliminates
God's segfaults.

Will you give this a try?
If it works, I'll do an 8C patch that to replace the stack splicing of
MBARI2 with stack anchors on 1.8.7-p72 and perhaps 1.8.6-p287 as well.

• brent

--
View this message in context: http://www.nabble.com/-ruby-core%3A19846---Bug--744--memory-leak-in-callcc--tp20447794p22614822.html
Sent from the ruby-core mailing list archive at Nabble.com.

=end

[brent (Brent Roman)]

=begin

Aman,

Could you reduce this repeatable failure to a script I could easily run to
reproduce it here?

My main machine is a mac mini running linux, but I can always reboot it into
OS/x
I've got access to PPC macs too, but they only run OS/x.

• brent

Aman Gupta-6 wrote:

I've had no more issues since reverting mbari2. I'm able to reproduce
the segfault on my mac:

ruby(83833) malloc: *** error for object 0x152e6d4: Non-aligned
pointer being freed
*** set a breakpoint in malloc_error_break to debug

--
View this message in context: http://www.nabble.com/-ruby-core%3A19846---Bug--744--memory-leak-in-callcc--tp20447794p22795862.html
Sent from the ruby-core mailing list archive at Nabble.com.

=end

[rogerdpack (Roger Pack)]

=begin
is anybody still getting segfaults with the latest MBARI patches?
They are working well for me, at least I haven't run into the segfaults of last Dec./Jan. for quite awhile.
Thanks.
-=r
=end

[brent (Brent Roman)]

=begin

Rogar,

I have no outstanding problem reports aside from Aman's issues with God on
x86_64 reported
here over a month ago. I'm still hoping he can distill this failure into
something I can replicate
and fix.

I merged the full patch set into the 1.8 trunk in mid-March, but I've had no
feedback from
the core developers since then.

• brent

Nobuyoshi Nakada-3 wrote:

Issue #744 has been updated by Roger Pack.

is anybody still getting segfaults with the latest MBARI patches?
They are working well for me, at least I haven't run into the segfaults of
last Dec./Jan. for quite awhile.
Thanks.
-=r

http://redmine.ruby-lang.org/issues/show/744

---

http://redmine.ruby-lang.org

--
View this message in context: http://www.nabble.com/-ruby-core%3A19846---Bug--744--memory-leak-in-callcc--tp20447794p23399399.html
Sent from the ruby-core mailing list archive at Nabble.com.

=end

[nobu (Nobuyoshi Nakada)]

=begin
Hi,

At Wed, 6 May 2009 12:25:12 +0900,
Brent Roman wrote in [ruby-core:23365]:

I merged the full patch set into the 1.8 trunk in mid-March, but I've had no
feedback from
the core developers since then.

Sorry to be late, but I have to resolve conflicts after it and
split directly irrelevant changes.

--
Nobu Nakada

=end

[shyouhei (Shyouhei Urabe)]

• Status changed from Open to Assigned
• Assignee changed from matz (Yukihiro Matsumoto) to nobu (Nobuyoshi Nakada)

=begin

=end

[jeremyevans0 (Jeremy Evans)]

• Tracker changed from Bug to Backport
• Project changed from 2 to 5
• Description updated (diff)
• Status changed from Assigned to Closed
• ruby -v deleted (-)