Project

General

Profile

Feature #11625

Unlock GVL for SHA1 calculations

Added by tenderlovemaking (Aaron Patterson) almost 5 years ago. Updated almost 2 years ago.

Status:
Assigned
Priority:
Normal
Assignee:
-
Target version:
-
[ruby-core:71228]

Description

I'm trying to calculate many sha1 checksums, but the current sha1 implementation doesn't unlock the GVL, so I can't do it in parallel. I've attached a patch that unlocks the GVL when calculating sha1sums so that I can do them in parallel.

The good point about this patch is that I can calculate sha1's in parallel. Here is the test code I'm using:

require 'digest/sha1'
require 'thread'

Thread.abort_on_exception = true

THREADS = (ENV['THREADS'] || 1).to_i

store = 'x' * (ENV['SIZE'] || 1024).to_i

queue = Queue.new

600000.times do
  queue << store
end

THREADS.times { queue << nil }

ts = THREADS.times.map {
  Thread.new {
    while work = queue.pop
      Digest::SHA1.hexdigest(work)
    end
  }
}

ts.each(&:join)

Here is what the output looks like after I've applied the patch:

[aaron@TC ruby (trunk)]$ THREADS=1 SIZE=4096 time ./ruby test.rb 
       22.62 real        21.78 user         0.66 sys
[aaron@TC ruby (trunk)]$ THREADS=4 SIZE=4096 time ./ruby test.rb 
       15.87 real        34.53 user         8.27 sys
[aaron@TC ruby (trunk)]$ 

The digests that I'm calculating are for fairly large strings, so this patch works well for me. The downsides are that it seems slightly slower (though I'm not sure that it's significant) with a single thread:

Test code:

require 'benchmark/ips'
require 'digest/sha1'

Benchmark.ips do |x|
  x.report('sha1') { Digest::SHA1.hexdigest('x' * 4096) }
end

Before my patch (higher numbers are better):

[aaron@TC ruby (trunk)]$ ./ruby shaips.rb 
Calculating -------------------------------------
                sha1     2.604k i/100ms
-------------------------------------------------
                sha1     27.441k (± 3.9%) i/s -    138.012k

After my patch:

[aaron@TC ruby (trunk)]$ ./ruby shaips.rb 
Calculating -------------------------------------
                sha1     2.419k i/100ms
-------------------------------------------------
                sha1     25.848k (± 2.8%) i/s -    130.626k

Other downside is that I changed the update method to dup strings so that the GVL can be safely released.

This patch pays off for me because of the size of the strings I'm working with, but I'm not sure if it's fine for the general case.


Files

sha1gvl.diff (3.79 KB) sha1gvl.diff tenderlovemaking (Aaron Patterson), 10/27/2015 09:41 PM
digest.patch (7.96 KB) digest.patch steved (Steve Dierker), 09/25/2018 11:24 AM

Also available in: Atom PDF