Bug #17864
closed[BUG] try to mark T_NONE object (in Ractors)
Description
Hello,
I tried to make an experiment with Ractors, computing SHA1 using Ractors (I was hopping to divide the computation time because SHA1 hashing does not release GVL and I'm running a "GUI" that has time constraint).
When I test with StringIO everything works fine, I can compute 400 StringIO that are like 100MB without issues. But when I use files (what I need to do in real environment) I face a internal bug (that doesn't not spawn in all attempt, it's very random).
And here's all the information about the bug:
ruby -v = ruby 3.0.0p0 (2020-12-25 revision 95aff21468) [i386-mingw32]
Error log:
<OBJ_INFO:gc_mark_ptr@../ruby-3.0.0/gc.c:6106> 0x053fbf68 [0 M ] T_NONE
D:/nuriy/persoWork/game_launcher/src/IODigester.rb:105: [BUG] try to mark T_NONE object
ruby 3.0.0p0 (2020-12-25 revision 95aff21468) [i386-mingw32]
-- Control frame information -----------------------------------------------
c:0003 p:---- s:0017 e:000016 CFUNC :read
c:0002 p:0056 s:0012 e:000011 BLOCK D:/nuriy/persoWork/game_launcher/src/IODigester.rb:105 [FINISH]
c:0001 p:---- s:0003 e:000002 (none) [FINISH]
-- Ruby level backtrace information ----------------------------------------
D:/nuriy/persoWork/game_launcher/src/IODigester.rb:105:in `block in make_ractor'
D:/nuriy/persoWork/game_launcher/src/IODigester.rb:105:in `read'
-- C level backtrace information -------------------------------------------
C:\WINDOWS\SYSTEM32\ntdll.dll(ZwWaitForSingleObject+0xc) [0x775729ac]
C:\WINDOWS\System32\KERNELBASE.dll(WaitForSingleObject+0x12) [0x767d0452]
D:\nuriy\persoWork\game_launcher\msvcrt-ruby300.dll(rb_vm_bugreport+0x669) [0x68134f29]
D:\nuriy\persoWork\game_launcher\msvcrt-ruby300.dll(rb_source_location+0x53) [0x68118d83]
C:\WINDOWS\System32\KERNELBASE.dll(CreateEventA+0x24) [0x767e1914]
D:\nuriy\persoWork\game_launcher\msvcrt-ruby300.dll(rb_vm_proc_local_ep+0xd81) [0x680d8741]
C:\WINDOWS\SYSTEM32\ntdll.dll(RtlGetAppContainerNamedObjectPath+0x11e) [0x77567a4e]
-- Other runtime information -----------------------------------------------
* Loaded script: src/IODigester.ytest copy.rb
* Loaded features:
0 enumerator.so
1 thread.rb
2 rational.so
3 complex.so
4 ruby2_keywords.rb
5 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/i386-mingw32/enc/encdb.so
6 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/i386-mingw32/enc/trans/transdb.so
7 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/i386-mingw32/rbconfig.rb
8 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/compatibility.rb
9 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/defaults.rb
10 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/deprecate.rb
11 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/errors.rb
12 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/exceptions.rb
13 D:/nuriy/persoWork/game_launcher/lib/ruby/site_ruby/3.0.0/ruby_installer/runtime/singleton.rb
14 D:/nuriy/persoWork/game_launcher/lib/ruby/site_ruby/3.0.0/ruby_installer/runtime.rb
15 D:/nuriy/persoWork/game_launcher/lib/ruby/site_ruby/3.0.0/ruby_installer/runtime/msys2_installation.rb
16 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/i386-mingw32/fiddle.so
17 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/fiddle/closure.rb
18 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/fiddle/function.rb
19 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/fiddle/version.rb
20 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/fiddle.rb
21 D:/nuriy/persoWork/game_launcher/lib/ruby/site_ruby/3.0.0/ruby_installer/runtime/dll_directory.rb
22 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/fiddle/value.rb
23 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/fiddle/pack.rb
24 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/fiddle/struct.rb
25 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/fiddle/cparser.rb
26 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/fiddle/import.rb
27 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/i386-mingw32/enc/utf_16le.so
28 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/i386-mingw32/enc/trans/utf_16_32.so
29 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/win32/registry.rb
30 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/i386-mingw32/enc/trans/single_byte.so
31 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/defaults/operating_system.rb
32 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/basic_specification.rb
33 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/stub_specification.rb
34 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/text.rb
35 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/user_interaction.rb
36 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/specification_policy.rb
37 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/util/list.rb
38 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/platform.rb
39 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/version.rb
40 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/requirement.rb
41 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/specification.rb
42 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/util.rb
43 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/dependency.rb
44 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/core_ext/kernel_gem.rb
45 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/i386-mingw32/monitor.so
46 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/monitor.rb
47 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/core_ext/kernel_require.rb
48 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/core_ext/kernel_warn.rb
49 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems.rb
50 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/rubygems/path_support.rb
51 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean/version.rb
52 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean/core_ext/name_error.rb
53 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean/levenshtein.rb
54 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean/jaro_winkler.rb
55 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean/spell_checker.rb
56 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean/spell_checkers/name_error_checkers/class_name_checker.rb
57 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean/spell_checkers/name_error_checkers/variable_name_checker.rb
58 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean/spell_checkers/name_error_checkers.rb
59 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean/spell_checkers/method_name_checker.rb
60 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean/spell_checkers/key_error_checker.rb
61 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean/spell_checkers/null_checker.rb
62 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean/tree_spell_checker.rb
63 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean/spell_checkers/require_path_checker.rb
64 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean/formatters/plain_formatter.rb
65 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/did_you_mean.rb
66 D:/nuriy/persoWork/game_launcher/src/IODigester.rb
67 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/i386-mingw32/digest.so
68 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/digest.rb
69 D:/nuriy/persoWork/game_launcher/lib/ruby/3.0.0/i386-mingw32/digest/sha1.so
Note: I noticed while writing the thread that my ruby version in this project was not updated, I tried with ruby 3.0.1p64 (2021-04-05 revision 0fb782ee38) [i386-mingw32]
and I have nothing but Idk if that's luck or not. If someone can confirm that this bug is solved that would be nice because I really hope to use that in production even if Ractors are still experimental.
Additional info
Here's the class I wrote to handle the SHA1 computation over ractors:
# Class responsive of digesting data (hashing) over several CPU core using Ractors
#
# How to use ?
# 1. First of all encapsulate all your data into IO object (File or StringIO)
# 2. Then create a new IODigester object by passing the Digest class you need and all the IO to digest in a Hash
# 3. Call the start function to let the system instanciate everything
# * You can check the overall progress using #progess, it's a number between 0 & 1
# 4. If failed? respond true, you can grab the failure info using #failures (key = IO key, value = StandardError object)
#
# @example Demo
# digester = IODigester.new(Digest::SHA1, { 'test' => io1, 'test2' => io2 })
# digester.start { |d| print format("\r%<progress>.2f%%", progress: d.progress * 100) }
# digester.hashes # => { 'test' => '<SHA1 of test>', 'test2' => '<SHA1 of test2>' }
#
# @note IODigester requires the IO to respond to size and give accurate value for size (in bytes)!
class IODigester
# Get the hashes
# @return [Hash{ String => String }]
attr_reader :hashes
# Get the failures
# @return [Hash{ String => StandardError }]
attr_reader :failures
# Create a new IODigester
# @param klass [Class<Digest::Base>] digest class to use to digest data
# @param ios [Hash{ String => IO }] io to digest (key can be symbol if you want)
def initialize(klass, ios)
@klass = klass
@ios = ios.clone
@progresses = @ios.keys.map { |k| [k, 0] }.to_h
@hashes = {}
@failures = {}
end
# Start the process
# @return [self]
# @yieldparam d [self] current digester
def start(&block)
raise 'Already started' if @ractors
@ractors = @ios.map { |key, value| make_ractor(key, value) }
process_ractors(block)
return self
end
# Test if the process has failures
# @return [Boolean]
def failed?
return !@failures.empty?
end
# Get the progress
# @return [Float]
def progress
@progresses.reduce(0.0) { |prev, curr| prev + curr.last } / @progresses.size
end
private
# Function that processes all the ractors (in a Thread in order to not lock the GVL)
# @param block [Proc]
def process_ractors(block)
until @ractors.empty?
begin
ractor, object = Ractor.select(*@ractors)
parse_ractor_response(ractor, object)
block&.call(self)
rescue Ractor::RemoteError => e
@failures[e.ractor.name] = e.cause
@ractors.delete(e.ractor)
end
end
end
# Function that analyzes the payload of the received object from Ractor.select
# @param ractor [Ractor] ractor object that we're talking about
# @param object [Array] response from the ractor
def parse_ractor_response(ractor, object)
message, body = object
case message
when :progress
@progresses[ractor.name] = body
when :result
@progresses[ractor.name] = 1
@hashes[ractor.name] = body
@ractors.delete(ractor)
end
end
# Function that creates the ractor
# @param name [String] name of the io (hash key)
# @param io [IO] io to process
# @return [Ractor]
def make_ractor(name, io)
ractor = Ractor.new(@klass, io, name: name) do |digest_klass, io_to_process|
close_incoming
# @type [Integer]
io_size = io_to_process.size
io_read = 0
# @type [Digest::Base]
digest = digest_klass.new
while io_size > io_read
Ractor.yield [:progress, io_read.to_f / io_size]
# @type [String]
data = io_to_process.read(20_971_520)
io_read += data.bytesize
digest << data
end
Ractor.yield [:result, digest.hexdigest]
end
return ractor
end
end
Here's the script I use to try to reproduce the bug:
require_relative 'IODigester'
require 'digest'
ios = (
Dir['D:/nuriy/Work/pokemonsdk/Release/Data/*'] +
Dir['D:/nuriy/Work/pokemonsdk/Release/pokemonsdk/**/*']
).reject { |f| File.directory?(f) }.map { |f| [f, File.new(f, 'rb')] }.to_h
digester = IODigester.new(Digest::SHA1, ios)
digester.start { |d| print format("\r%<progress>.2f%%", progress: d.progress * 100) }
p digester.hashes
(I'm not closing the files because I don't actually care in that test since the bug appears in "start")
Here's the files in case you need it: https://www.mediafire.com/file/wt0eqzthxpb1nc8/Release.7z/file (it's over 2MB)