Bug #13861: Performance regressoion in Hash literal in Ruby 2.5.0-dev

• Status: Third Party's Issue
• Assignee: -
• Target version: -
• ruby -v: ruby 2.5.0dev (2017-09-02 trunk 59626) [x86_64-linux]
• Backport: 2.2: UNKNOWN, 2.3: UNKNOWN, 2.4: UNKNOWN

[watson1978 (Shizuo Fujita)]

Seems Ruby 2.5.0-dev has performance regressoion in Hash literal.
I've confirmed it with below benchmark script

Hash literal performance -> 35.7 % slow down since Ruby 2.4.1

Environment

• Ubuntu 17.04
• gcc version 7.0.1

Ruby 2.5.0-dev

$ ./miniruby -v -Ilib -I../benchmark-ips/lib  ~/tmp/bench.rb
ruby 2.5.0dev (2017-09-02 trunk 59626) [x86_64-linux]
/home/watson/tmp/bench.rb:7: warning: assigned but unused variable - hash
/home/watson/tmp/bench.rb:15: warning: assigned but unused variable - string
Warming up --------------------------------------
                Hash    93.650k i/100ms
              String   226.366k i/100ms
Calculating -------------------------------------
                Hash      1.713M (± 0.2%) i/s -      8.616M in   5.030997s
              String     16.477M (± 0.1%) i/s -     82.397M in   5.000892s

Ruby 2.4.1

$ ruby -v -Ilib -I../benchmark-ips/lib  ~/tmp/bench.rb
ruby 2.4.1p111 (2017-03-22 revision 58053) [x86_64-linux]
/home/watson/tmp/bench.rb:7: warning: assigned but unused variable - hash
/home/watson/tmp/bench.rb:15: warning: assigned but unused variable - string
Warming up --------------------------------------
                Hash   117.135k i/100ms
              String   259.608k i/100ms
Calculating -------------------------------------
                Hash      2.663M (± 0.3%) i/s -     13.353M in   5.014087s
              String     17.259M (± 0.2%) i/s -     86.449M in   5.009036s

Benchmark code

require 'benchmark/ips'

Benchmark.ips do |x|
  x.report "Hash" do |loop|
    count = 0
    while count < loop
      hash = {foo: 12, bar: 34, baz: 56}
      count += 1
    end
  end

  x.report "String" do |loop|
    count = 0
    while count < loop
      string = "hello world"
      count += 1
    end
  end
end

[MSP-Greg (Greg L)]

watson1978 (Shizuo Fujita) wrote:

Seems Ruby 2.5.0-dev has performance regression in Hash literal.

Similar results on MinGW Win7 gcc 7.1.0 using ruby for both

ruby 2.5.0dev (2017-09-02 trunk 59723) [x64-mingw32]
ruby 2.4.1p111 (2017-03-22 revision 58053) [x64-mingw32]

[shyouhei (Shyouhei Urabe)]

• File tmp2.png tmp2.png added

watson1978 (Shizuo Fujita) wrote:

Hash literal performance -> 35.7 % slow down since Ruby 2.4.1

I have to disagree.

require 'benchmark/ips'
n = 32

Benchmark.ips do |x|
  # precalc literals
  lit = []
  (0..n).each{|i|lit[i]="x.report(#{i}){{".dup}
  (0..n).each{|i|(0...i).each{|j|lit[n-j]<< "i#{i}: #{i}, "}}
  (0..n).each{|i|lit[i]<< "}}"}
  bench = lit.join("\n")

  # let's go
  eval bench
end

This benchmark yields attached result on my machine.

It is clear that trunk is overall much faster than 2.4. Hash literals of 3 and 4 elements are the only cases that needs improvements.

[MSP-Greg (Greg L)]

shyouhei (Shyouhei Urabe) wrote:

I have to disagree.

Thank you for the thorough analysis and the reminder to always vary the length when testing anything with one...

[shyouhei (Shyouhei Urabe)]

• Status changed from Open to Third Party's Issue

After looking at this more closely, I now suspect it's benchmark/ips who's doing something nasty. Without it, trunk runs order of magnitude faster than 2.4.

zsh % time ruby@trunk --disable-gems -ve 'i=0; while i<100_000_000 do { foo: 1, bar: 2, baz: 3 }; i+=1 end'
ruby 2.5.0dev (2017-09-03 trunk 59738) [x86_64-darwin15]
ruby --disable-gems -ve   1.77s user 0.03s system 98% cpu 1.833 total
zsh % time ruby@ruby_2_4 --disable-gems -ve 'i=0; while i<100_000_000 do { foo: 1, bar: 2, baz: 3 }; i+=1 end'
ruby 2.4.2p181 (2017-08-05 revision 59520) [x86_64-darwin15]
ruby --disable-gems -ve   25.55s user 0.08s system 99% cpu 25.708 total

[watson1978 (Shizuo Fujita)]

I think this is not 3rd party issue.
Because this regression appears since 2017-04-28

$ ./miniruby -v -Ilib -I../benchmark-ips/lib  ~/tmp/bench.rb
ruby 2.5.0dev (2017-04-27 trunk 58469) [x86_64-linux]
/home/watson/tmp/bench.rb:7: warning: assigned but unused variable - hash
/home/watson/tmp/bench.rb:15: warning: assigned but unused variable - string
Warming up --------------------------------------
                Hash   119.163k i/100ms
              String   194.413k i/100ms
Calculating -------------------------------------
                Hash      2.587M (± 0.5%) i/s -     12.989M in   5.021489s
              String     13.874M (±20.8%) i/s -     66.489M in   5.005903s

$ ./miniruby -v -Ilib -I../benchmark-ips/lib  ~/tmp/bench.rb
ruby 2.5.0dev (2017-04-28 trunk 58489) [x86_64-linux]
/home/watson/tmp/bench.rb:7: warning: assigned but unused variable - hash
/home/watson/tmp/bench.rb:15: warning: assigned but unused variable - string
Warming up --------------------------------------
                Hash   102.010k i/100ms
              String   227.168k i/100ms
Calculating -------------------------------------
                Hash      1.674M (± 2.4%) i/s -      8.365M in   5.000468s
              String     12.829M (± 0.6%) i/s -     64.289M in   5.011156s

[watson1978 (Shizuo Fujita)]

Seems this regression was fixed by https://svn.ruby-lang.org/cgi-bin/viewvc.cgi?revision=59744&view=revision

Thank you for fixing!!!

before

$ ./miniruby -v -Ilib -I../benchmark-ips/lib  ~/tmp/bench.rb
ruby 2.5.0dev (2017-09-05 trunk 59753) [x86_64-linux]
/home/watson/tmp/bench.rb:7: warning: assigned but unused variable - hash
/home/watson/tmp/bench.rb:15: warning: assigned but unused variable - string
Warming up --------------------------------------
                Hash    96.137k i/100ms
              String   219.522k i/100ms
Calculating -------------------------------------
                Hash      1.723M (± 0.6%) i/s -      8.652M in   5.021705s
              String     16.975M (± 0.3%) i/s -     84.955M in   5.004690s

after

$ ./miniruby -v -Ilib -I../benchmark-ips/lib  ~/tmp/bench.rb
ruby 2.5.0dev (2017-09-06 trunk 59810) [x86_64-linux]
/home/watson/tmp/bench.rb:7: warning: assigned but unused variable - hash
/home/watson/tmp/bench.rb:15: warning: assigned but unused variable - string
Warming up --------------------------------------
                Hash   135.385k i/100ms
              String   234.168k i/100ms
Calculating -------------------------------------
                Hash      2.863M (± 0.1%) i/s -     14.351M in   5.012065s
              String     16.869M (± 7.1%) i/s -     83.832M in   5.010604s