Project

General

Profile

Actions

Bug #5985

closed

miniruby skews "make benchmark" results

Added by normalperson (Eric Wong) about 12 years ago. Updated about 10 years ago.

Status:
Closed
Target version:
ruby -v:
-
Backport:
[ruby-core:42443]

Description

miniruby has fewer objects and a smaller heap than the normal "ruby",
benchmarks that are affected by GC performance is skewed heavily.
This is most noticeable with the vm3_gc benchmark using the same
Ruby revision/build but different executables (miniruby vs ruby):

$ make
$ make install
$ make benchmark-each ITEM=vm3_gc
ruby ../benchmark/driver.rb -v
--executables="ruby; ./miniruby -I../lib -I. -I.ext/common ../tool/runruby.rb --extout=.ext -- --disable-gems"
--pattern=vm3_gc --directory=../benchmark
total: 1 trial(s) (1 trial(s) for 1 benchmark(s))
2012-02-08 18:57:12 +0000
target 0: ruby 2.0.0dev (2012-02-08 trunk 34493) [x86_64-linux]
target 1: ruby 2.0.0dev (2012-02-08 trunk 34493) [x86_64-linux]


vm3_gc

#! /usr/bin/ruby
5000.times do
100.times do
{"xxxx"=>"yyyy"}
end
GC.start
end

ruby 2.0.0dev (2012-02-08 trunk 34493) [x86_64-linux] .2.752270221710205
ruby 2.0.0dev (2012-02-08 trunk 34493) [x86_64-linux] .1.857623815536499


raw data:

[["vm3_gc", [[2.752270221710205], [1.857623815536499]]]]

Elapesed time: 4.611220872 (sec)

benchmark results:
name ruby 2.0.0dev (2012-02-08 trunk 34493) [x86_64-linux] ruby 2.0.0dev (2012-02-08 trunk 34493) [x86_64-linux] average difference
vm3_gc 2.752 1.858 -0.895

average total difference is -0.894646406173706

Actions #1

Updated by ko1 (Koichi Sasada) about 12 years ago

  • Assignee set to ko1 (Koichi Sasada)

I'll change it.

Actions #2

Updated by shyouhei (Shyouhei Urabe) about 12 years ago

  • Status changed from Open to Assigned

Updated by ko1 (Koichi Sasada) over 11 years ago

  • Status changed from Assigned to Closed

I changed to accept multiple -e' like "-e path1 -e path2". or "-e label1::path1 -e label2::path2". You don't need to use ;' character.

Updated by ko1 (Koichi Sasada) over 11 years ago

Sorry, the previous comment is for [Bug #7380].

BTW, the current benchmark using ruby' instead of miniruby'.

The difference is `--disable-gems':

$ make benchmark COMPARE_RUBY='installed::~/tmp/trunk/bin/ruby --disable-gems' OPTS='-p vm3_gc'
...

benchmark results:
Execution time (sec)
name installed built-ruby
vm3_gc 1.806 1.823

I think this ticket can be closed.

(or remove --disable-gems ?)

Updated by normalperson (Eric Wong) over 11 years ago

"ko1 (Koichi Sasada)" wrote:

Issue #5985 has been updated by ko1 (Koichi Sasada).

Sorry, the previous comment is for [Bug #7380].

BTW, the current benchmark using ruby' instead of miniruby'.

The difference is `--disable-gems':

$ make benchmark COMPARE_RUBY='installed::~/tmp/trunk/bin/ruby --disable-gems' OPTS='-p vm3_gc'
...

benchmark results:
Execution time (sec)
name installed built-ruby
vm3_gc 1.806 1.823

I think this ticket can be closed.

(or remove --disable-gems ?)

I favor removing --disable-gems from the benchmark command.

I think it's unrealistic to use/benchmark Ruby without gems nowadays.

Actions #6

Updated by normalperson (Eric Wong) about 10 years ago

Btw, I'm still having this problem and it's very noticeable with:
make benchmark-each ITEM=bm_so_reverse_complement

This is running the same COMPARE_RUBY as what was just installed via
"make install" Strangely, miniruby is the one that is slow, here.

 $ make benchmark-each ITEM=bm_so_reverse_complement
 ruby 2.2.0dev (2014-01-14 trunk 44595) [x86_64-linux]   1.530159944
 built-ruby      2.93252692

 -----------------------------------------------------------
 raw data:

 [["so_reverse_complement", [[1.530159944], [2.93252692]]]]

 Elapsed time: 4.464871542 (sec)
 -----------------------------------------------------------
 benchmark results:
 Execution time (sec)
 name    ruby 2.2.0dev (2014-01-14 trunk 44595) [x86_64-linux]
 built-ruby
 so_reverse_complement   1.530   2.933

 Speedup ratio: compare with the result of `ruby 2.2.0dev (2014-01-14
 trunk 44595) [x86_64-linux]' (greater is better)
 name    built-ruby
 so_reverse_complement   0.522

Compare with the following, the numbers only differ in noise:

 $ ruby --disable=gems ./benchmark/driver.rb -v  --executables="a::ruby \
 --disable=gems; b::ruby --disable=gems" \
 --pattern=bm_so_reverse_complement --directory=./benchmark

 a       1.547410982
 b       1.537930862

 -----------------------------------------------------------
 raw data:

 [["so_reverse_complement", [[1.547410982], [1.537930862]]]]

 Elapsed time: 3.087557723 (sec)
 -----------------------------------------------------------
 benchmark results:
 Execution time (sec)
 name    a       b
 so_reverse_complement   1.547   1.538

 Speedup ratio: compare with the result of `a' (greater is better)
 name    b
 so_reverse_complement   1.006
Actions #7

Updated by Eregon (Benoit Daloze) about 10 years ago

Eric Wong wrote:

ko1 (Koichi Sasada) wrote:

(or remove --disable-gems ?)

I favor removing --disable-gems from the benchmark command.
I think it's unrealistic to use/benchmark Ruby without gems nowadays.

Unfortunately that would make the comparison between different ruby versions more brittle as the gems might change, particularly how much allocations they do, no?

I have found as well some benchmarks in benchmark/ are very sensitive to the heap usage.

Actions #8

Updated by Eregon (Benoit Daloze) about 10 years ago

  • ruby -v changed from ruby 2.0.0dev (2012-02-08 trunk 34493) [x86_64-linux] to -

Issue #5985 has been updated by Benoit Daloze.

Eric Wong wrote:

ko1 (Koichi Sasada) wrote:

(or remove --disable-gems ?)

I favor removing --disable-gems from the benchmark command.
I think it's unrealistic to use/benchmark Ruby without gems nowadays.

Unfortunately that would make the comparison between different ruby versions more brittle as the gems might change, particularly how much allocations they do, no?

I have found as well some benchmarks in benchmark/ are very sensitive to the heap usage.


Bug #5985: miniruby skews "make benchmark" results
https://bugs.ruby-lang.org/issues/5985#change-44341

  • Author: Eric Wong
  • Status: Closed
  • Priority: Low
  • Assignee: Koichi Sasada
  • Category: build
  • Target version: 2.0.0
  • ruby -v: ruby 2.0.0dev (2012-02-08 trunk 34493) [x86_64-linux]
  • Backport:

miniruby has fewer objects and a smaller heap than the normal "ruby",
benchmarks that are affected by GC performance is skewed heavily.
This is most noticeable with the vm3_gc benchmark using the same
Ruby revision/build but different executables (miniruby vs ruby):

$ make
$ make install
$ make benchmark-each ITEM=vm3_gc
ruby ../benchmark/driver.rb -v
--executables="ruby; ./miniruby -I../lib -I. -I.ext/common ../tool/runruby.rb --extout=.ext -- --disable-gems"
--pattern=vm3_gc --directory=../benchmark
total: 1 trial(s) (1 trial(s) for 1 benchmark(s))
2012-02-08 18:57:12 +0000
target 0: ruby 2.0.0dev (2012-02-08 trunk 34493) [x86_64-linux]
target 1: ruby 2.0.0dev (2012-02-08 trunk 34493) [x86_64-linux]


vm3_gc

#! /usr/bin/ruby
5000.times do
100.times do
{"xxxx"=>"yyyy"}
end
GC.start
end

ruby 2.0.0dev (2012-02-08 trunk 34493) [x86_64-linux] .2.752270221710205
ruby 2.0.0dev (2012-02-08 trunk 34493) [x86_64-linux] .1.857623815536499


raw data:

[["vm3_gc", [[2.752270221710205], [1.857623815536499]]]]

Elapesed time: 4.611220872 (sec)

benchmark results:
name ruby 2.0.0dev (2012-02-08 trunk 34493) [x86_64-linux] ruby 2.0.0dev (2012-02-08 trunk 34493) [x86_64-linux] average difference
vm3_gc 2.752 1.858 -0.895

average total difference is -0.894646406173706

--
http://bugs.ruby-lang.org/

Actions #9

Updated by normalperson (Eric Wong) about 10 years ago

wrote:

Eric Wong wrote:

ko1 (Koichi Sasada) wrote:

(or remove --disable-gems ?)

I favor removing --disable-gems from the benchmark command.
I think it's unrealistic to use/benchmark Ruby without gems nowadays.

Unfortunately that would make the comparison between different ruby
versions more brittle as the gems might change, particularly how much
allocations they do, no?

Good point. Perhaps keeping --disable-gems globally is better.

I have found as well some benchmarks in benchmark/ are very sensitive
to the heap usage.

Exactly.

Updated by normalperson (Eric Wong) about 10 years ago

Eric Wong wrote:

Btw, I'm still having this problem and it's very noticeable with:
make benchmark-each ITEM=bm_so_reverse_complement

Actually, this benchmark is all over the place because
benchmark/driver.rb slurps the gigantic output of this benchmark.

Patch at https://bugs.ruby-lang.org/issues/9430

Updated by normalperson (Eric Wong) about 10 years ago

Btw, I'm still getting more consistent results with normal ruby instead
of miniruby, probably because the encodings still affect heap size and
memory layout.

I suggest defining BENCHRUBY in my patch:
http://bogomips.org/ruby.git/patch?id=1ec0da52b7ce2e0e16f17

The following changes since commit 0558b8a5c40b93c01f5724fe8a3ab409d4374490:

  • ext/psych/lib/psych.rb: fix typo by @jwworth [fix GH-500] * lib/rake/file_list.rb: ditto (2014-01-19 05:48:33 +0000)

are available in the git repository at:

git://80x24.org/ruby.git benchmark

for you to fetch changes up to 1ec0da52b7ce2e0e16f1729933ea795b12fea7ae:

common.mk: define BENCHRUBY to avoid miniruby (2014-01-19 07:02:16 +0000)


Eric Wong (1):
common.mk: define BENCHRUBY to avoid miniruby

common.mk | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)

Updated by normalperson (Eric Wong) about 10 years ago

I would like to commit this soon:

 * common.mk: define BENCHRUBY to avoid miniruby heap size difference
   [ruby-core:59887] [Bug #5985]

Updated by ko1 (Koichi Sasada) about 10 years ago

+BENCHRUBY = ./$(PROGRAM) -I$(srcdir)/lib -I. -I$(EXTOUT)/common $(RUN_OPTS)

I'm not sure BENCHRUBY works with libruby in builddir (don't use installed libruby?).

Updated by normalperson (Eric Wong) about 10 years ago

wrote:

+BENCHRUBY = ./$(PROGRAM) -I$(srcdir)/lib -I. -I$(EXTOUT)/common
$(RUN_OPTS)

I'm not sure BENCHRUBY works with libruby in builddir (don't use
installed libruby?).

Good catch, I tested out-of-tree build, but not --enable-shared.
I can force the use of STATIC_RUBY, it looks like.
I have this working both with and without --enable-shared.

http://bogomips.org/ruby.git/patch/?id=7495b0fdc7

--- a/common.mk
+++ b/common.mk
@@ -1001,14 +1001,14 @@ COMPARE_RUBY = $(BASERUBY)
ITEM =
OPTS =

-BENCHRUBY = ./$(PROGRAM) -I$(srcdir)/lib -I. -I$(EXTOUT)/common $(RUN_OPTS)
+BENCHRUBY = ./$(STATIC_RUBY) -I$(srcdir)/lib -I. -I$(EXTOUT)/common $(RUN_OPTS)

-benchmark: $(PROGRAM) PHONY
+benchmark: $(PROGRAM) PHONY $(STATIC_RUBY)
$(BASERUBY) $(srcdir)/benchmark/driver.rb -v
--executables="$(COMPARE_RUBY); built-ruby::$(BENCHRUBY)"
--pattern='bm_' --directory=$(srcdir)/benchmark $(OPTS)

-benchmark-each: $(PROGRAM) PHONY
+benchmark-each: $(PROGRAM) PHONY $(STATIC_RUBY)
$(BASERUBY) $(srcdir)/benchmark/driver.rb -v
--executables="$(COMPARE_RUBY); built-ruby::$(BENCHRUBY)"
--pattern=$(ITEM) --directory=$(srcdir)/benchmark $(OPTS)

Updated by ko1 (Koichi Sasada) about 10 years ago

+BENCHRUBY = ./$(STATIC_RUBY) -I$(srcdir)/lib -I. -I$(EXTOUT)/common $(RUN_OPT

We need to discuss which version we want to compare. static_ruby and dynamic (using libruby) ruby show different performance (basically, dynamic ruby is slower).

To avoid such difference, recently I use benchmark/driver.rb directly to compare installed rubies. To do so, I install non-modified trunk and modified trunk before benchmark.

What should happen on "make benchmark"? This rule I wrote is to compare installed "ruby 1.8" and built miniruby. it is not fair comparison but it is enough to check the rough performance.

Updated by normalperson (Eric Wong) about 10 years ago

wrote:

+BENCHRUBY = ./$(STATIC_RUBY) -I$(srcdir)/lib -I. -I$(EXTOUT)/common $(RUN_OPT

We need to discuss which version we want to compare. static_ruby and
dynamic (using libruby) ruby show different performance (basically,
dynamic ruby is slower).

To avoid such difference, recently I use benchmark/driver.rb directly
to compare installed rubies. To do so, I install non-modified trunk
and modified trunk before benchmark.

What should happen on "make benchmark"? This rule I wrote is to
compare installed "ruby 1.8" and built miniruby. it is not fair
comparison but it is enough to check the rough performance.

I can count 3 options right now for "make benchmark":

a) "make benchmark" becomes a help message to tell people
to only benchmark with installed Rubies.

b) use LD_PRELOAD for shared Rubies (not portable?)

c) ignore dynamic for benchmark, it will always
be slower than static (neither is nearly as fast as miniruby)

I think c) is the easiest path right now, static-ruby is closer to
dynamic ruby. miniruby is too far different and unrealistic.

In all cases, we should probably add a warning if any benchmarked
ruby is dynamic.

Updated by normalperson (Eric Wong) about 10 years ago

May I commit my current STATIC_RUBY version?

It is not perfect for dynamic ruby, but it is better than current
"make benchmark". We can make it perfect later.

Actions

Also available in: Atom PDF

Like0
Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0