Bug #19436
closedCall Cache for singleton methods can lead to "memory leaks"
Description
Using "memory leaks" with quotes, because strictly speaking the memory isn't leaked, but it can nonetheless lead to large memory overheads.
Minimal Reproduction¶
module Foo
def bar
end
end
def call_bar(obj)
# Here the call cache we'll keep a ref on the method_entry
# which then keep a ref on the singleton_class, making that
# instance immortal until the method is called again with
# another instance.
# The reference chain is IMEMO(callcache) -> IMEMO(ment) -> ICLASS -> CLASS(singleton) -> OBJECT
obj.bar
end
obj = Object.new
obj.extend(Foo)
call_bar(obj)
id = obj.object_id
obj = nil
4.times { GC.start }
p ObjectSpace._id2ref(id)
Explanation¶
Call caches keep a strong reference onto the "callable method entry" (CME), which itself keeps a strong reference on the called object class
and in the cache of a singleton class, it keeps a strong reference onto the attached_object
(instance).
This means that any call site that calls a singleton method, will effectively keep a strong reference onto the last receiver.
If the method is frequently called it's not too bad, but if it's infrequently called, it's effectively a (bounded) memory leak.
And if the attached_object
is big, the wasted memory can be very substantial.
Practical Implications¶
Once relative common API impacted by this is Rails' extending
API. This API allow to extend a "query result set" with a module.
These query results set can sometimes be very big, especially since they keep references to the instantiated ActiveRecord::Base
instances etc.
Possible Solutions¶
Only keep a weak reference to the CME¶
The fairly "obvious" solution is to keep a weak reference to the CME, that's what I explored in https://github.com/ruby/ruby/pull/7272, and it seems to work.
However in debug mode It does fail on an assertion during compaction, but it's isn't quite clear to me what the impact is.
Additionally, something that makes me think this would be the right solution, is that call caches already try to avoid marking the class:
# vm_callinfo.h:275
struct rb_callcache {
const VALUE flags;
/* inline cache: key */
const VALUE klass; // should not mark it because klass can not be free'd
// because of this marking. When klass is collected,
// cc will be cleared (cc->klass = 0) at vm_ccs_free().
So it appears that the class being also marked through the CME is some kind of oversight?
Don't cache based on some heuristics¶
If the above isn't possible or too complicated, an alternative would be to not cache CMEs found in singleton classes, except if it's the the singleton class of a Class
or Module
.
It would make repeated calls to such methods slower, but the assumption is that it's unlikely that these CME would live very long.
Make Class#attached_object
a weak reference¶
Alternatively we could make the attached_object
a weak reference, which would drastically limit the amount of memory that may be leaked in such scenario.
The downside is that Class#attached_object
was very recently exposed in Ruby 3.2.0, so it means changing its semantic a bit.
Updated by Eregon (Benoit Daloze) over 1 year ago
I think making Class#attached_object
a weak reference is a better and easier solution.
Keeping a weak reference to the CME would actually be an unacceptable memory overhead on JVM, as it would mean one extra WeakReference object for each of these call caches.
I suppose there could be workarounds like storing the WeakReference on the singleton class, and use a direct reference otherwise, but then it'd make method lookup more complex and slower in interpreter or when polymorphic (need to handle cached method lookup on regular and singleton class differently).
FWIW not caching in calls to objects with a singleton class is not an option, it's important to cache for e.g. calling method on classes.
That approach is probably also quite difficult to get right on CRuby, also considering ractors and multithreading when the class GCs.
As a general note, creating a singleton class is not cheap, this should only be used for class objects (which always have one) and for a few rare global objects where it's convenient.
Using Object#extend
objects often/on the fast path is just "making programs slow and uncached".
Updated by byroot (Jean Boussier) over 1 year ago
FWIW not caching in calls to objects with a singleton class is not an option, it's important to cache for e.g. calling method on classes.
I specifically tried to say that the heuristic would be to not cache calls on singleton methods unless it's the singleton_class of a Module
or Class
.
Using Object#extend objects often/on the fast path is just "making programs slow and uncached".
Yeah I agree. I tried to avoid this in Rails by caching these dynamic classes, unfortunately that's not possible because of the API: https://github.com/rails/rails/pull/47314. So I'll try to see if this API can be replaced by a better one and deprecated.
Updated by byroot (Jean Boussier) over 1 year ago
Keeping a weak reference to the CME would actually be an unacceptable memory overhead on JVM
Does that mean TruffleRuby suffer from the same "leak"? I was actually planning to ask you how it does deal with this case for inspiration.
Updated by Eregon (Benoit Daloze) over 1 year ago
byroot (Jean Boussier) wrote in #note-2:
I specifically tried to say that the heuristic would be to not cache calls on singleton methods unless it's the singleton_class of a
Module
orClass
.
I missed that sorry.
I think there are quite a few cases where a global object (which is not a module/class) with a singleton class is used, it'd be unfortunate and slow to not inline cache lookup in those cases.
I think it makes sense to only try caching one (or a few) singleton class(es), and give up when seeing a second one or more (already happens on TruffleRuby, but for a higher limit, 8 currently).
That means we still need to make Class#attached_object
a weak reference.
Updated by Eregon (Benoit Daloze) over 1 year ago
byroot (Jean Boussier) wrote in #note-3:
Does that mean TruffleRuby suffer from the same "leak"? I was actually planning to ask you how it does deal with this case for inspiration.
Yes, since TruffleRuby caches the the Class object in method lookup (LookupMethodNode), and currently the Class object has a strong reference to the attached_object
.
Updated by byroot (Jean Boussier) over 1 year ago
That means we still need to make Class#attached_object a weak reference.
I agree that conceptually, it's the simplest solution. That being said, in MRI weak references generally require a finalizer, which is annoying. But I'll see if I can implement that.
Updated by ko1 (Koichi Sasada) over 1 year ago
Matz asked me to solve this memory leaking issue, so I'll consider about it before Ruby 3.3.
One idea is re-introduce class serial.
Updated by byroot (Jean Boussier) over 1 year ago
One idea is re-introduce class serial.
Does that mean the method cache could be global again?
Updated by Eregon (Benoit Daloze) over 1 year ago
@ko1 (Koichi Sasada) I think the real solution to this category of problems, not just this specific one is to make Class#attached_object
(internally) a weak reference.
So I would strongly advise against changing the method lookup for this, it's an independent problem.
For instance, if someone does some caching in a Hash based on the Class, they don't expect holding onto a Class also holds onto the singleton object forever.
So Class#attached_object
(the internal field used for it) is the leak, not the inline cache itself which needs to hold onto the class.
Updated by Eregon (Benoit Daloze) over 1 year ago
Also as mentioned in https://bugs.ruby-lang.org/issues/19436#note-1 I believe TruffleRuby and JRuby (and probably more) would never use a weak reference in the inline cache, it's too much of a footprint overhead and also an extra indirection at least in the interpreter.
So then the only other solution I see is Class#attached_object
(internally) being weak.
I think CRuby should follow the same idea, otherwise CRuby would be knowingly hurting compatibility between Ruby implementations here.
A per-class serial as far as I understand it is no good for a JIT, if e.g. methods are defined in some class late or repeatedly, it would invalidate all inline cache of method lookup on that class and all related JITed code, needlessly.
https://medium.com/graalvm/precise-method-and-constant-invalidation-in-truffleruby-4dd56c6bac1a has more details on that.
Updated by ko1 (Koichi Sasada) over 1 year ago
byroot (Jean Boussier) wrote in #note-8:
One idea is re-introduce class serial.
Does that mean the method cache could be global again?
No. Current key is a class value (pointer) but the idea is making class serial (each class has a serial) as an inline cache key.
Updated by ko1 (Koichi Sasada) over 1 year ago
Eregon (Benoit Daloze) wrote in #note-9:
@ko1 (Koichi Sasada) I think the real solution to this category of problems, not just this specific one is to make
Class#attached_object
(internally) a weak reference.
o = Object.new
c = o.singleton_class
o = nil
# do some task
p c.attached_object #=> ???
making weak reference, attached_objec
may return nil for the collected object?
I'm not sure it is acceptable or not.
Updated by Eregon (Benoit Daloze) over 1 year ago
ko1 (Koichi Sasada) wrote in #note-12:
making weak reference,
attached_objec
may return nil for the collected object?I'm not sure it is acceptable or not.
Yes, this is indeed a consequence.
I think it's fine, Class#attached_object
is very recent and this is necessary to solve a fundamental leak.
Updated by Eregon (Benoit Daloze) over 1 year ago
ko1 (Koichi Sasada) wrote in #note-11:
No. Current key is a class value (pointer) but the idea is making class serial (each class has a serial) as an inline cache key.
That would mean an extra indirection/memory load on every cached method lookup, i.e. obj->klass == CACHED_CLASS
(current) vs obj->klass->serial == CACHED_SERIAL
.
Also it would require these serials to be unique, otherwise that's not a correct check (so maybe more of a "unique ID" than serial).
Updated by ufuk (Ufuk Kayserilioglu) over 1 year ago
Let me add a few points in this conversation:
making weak reference, attached_objec may return nil for the collected object?
I'm not sure it is acceptable or not.
I agree with @Eregon (Benoit Daloze) on this point that this is totally acceptable. When I was proposing Class#attached_object
the argument was that it would be helpful in cases where the objects are still supposed to be in memory for introspection purposes. There is very little value in trying to introspect an attached object that was already garbage collected, and it would not be great to retain objects just because they had a singleton class defined for them.
So, I am +1 for slightly changing the semantics of Class#attached_object
before it becomes more widely used.
I should also say, though, that the purist in me still considers that the cache should be the one holding weak references, since, after all it is a cache and a miss should be relatively inconsequential, as opposed to the link between a singleton class and its attached object. I do understand, however, that it might not be the desired implementation and pragmatism over purism might be more warranted here.
Updated by byroot (Jean Boussier) over 1 year ago
- Assignee set to ko1 (Koichi Sasada)
Updated by byroot (Jean Boussier) over 1 year ago
- Related to Feature #19783: Weak References in the GC added
Updated by Ruby (Elena Mangano) over 1 year ago
- Status changed from Open to Closed
Applied in changeset git|c330037c1a38cc257dbe21022fcc7b13159c2821.
cc->cme
should not be marked.
cc is callcache.
cc->klass (klass) should not be marked because if the klass is
free'ed, the cc->klass will be cleared by vm_cc_invalidate()
.
cc->cme (cme) should not be marked because if cc is invalidated
when cme is free'ed.
- klass marks cme if klass uses cme.
- caller classe's ccs->cme marks cc->cme.
- if cc is invalidated (klass doesn't refer the cc),
cc is invalidated byvm_cc_invalidate()
and cc->cme is
not be accessed. - On the multi-Ractors, cme will be collected with global GC
so that it is safe if GC is not interleaving while accessing
cc and cme.
fix [Bug #19436]
10_000.times{|i|
# p i if (i%1_000) == 0
str = "x" * 1_000_000
def str.foo = nil
eval "def call#{i}(s) = s.foo"
send "call#{i}", str
}
Without this patch:
real 1m5.639s
user 0m6.637s
sys 0m58.292s
and with this patch:
real 0m2.045s
user 0m1.627s
sys 0m0.164s
Updated by byroot (Jean Boussier) over 1 year ago
@ko1 (Koichi Sasada) thanks for the patch, however we're running into some assertion failures on our CI:
Assertion Failed: vm_callinfo.h:339:vm_cc_class_check:cc->klass == 0 || RB_TYPE_P(cc->klass, T_CLASS) || RB_TYPE_P(cc->klass, T_ICLASS)
/usr/local/ruby/bin/ruby(rb_vm_bugreport) /tmp/ruby-build.20230728100454.392.WYrSwe/ruby-3.3.0-6391132c03ac08da0483adb986ff9a54e41f9e14/vm_dump.c:1088
/usr/local/ruby/bin/ruby(rb_assert_failure+0x77) [0x563d185aeb55] /tmp/ruby-build.20230728100454.392.WYrSwe/ruby-3.3.0-6391132c03ac08da0483adb986ff9a54e41f9e14/error.c:879
/usr/local/ruby/bin/ruby(vm_cc_class_check+0x1f) [0x563d1859d954] /tmp/ruby-build.20230728100454.392.WYrSwe/ruby-3.3.0-6391132c03ac08da0483adb986ff9a54e41f9e14/vm_callinfo.h:339
/usr/local/ruby/bin/ruby(vm_search_method_fastpath+0x5) [0x563d1877a9b8] /tmp/ruby-build.20230728100454.392.WYrSwe/ruby-3.3.0-6391132c03ac08da0483adb986ff9a54e41f9e14/vm_insnhelper.c:2181
Which I suspect are related?
Updated by ko1 (Koichi Sasada) over 1 year ago
Thanks.
Could you give some more lines of C backtrace?
Updated by byroot (Jean Boussier) over 1 year ago
@ko1 (Koichi Sasada) of course, here you go:
Assertion Failed: vm_callinfo.h:339:vm_cc_class_check:cc->klass == 0 || RB_TYPE_P(cc->klass, T_CLASS) || RB_TYPE_P(cc->klass, T_ICLASS)
ruby 3.3.0dev (2023-07-30T08:05:58Z shopify b5c74d5488) [x86_64-linux]
-- Threading information ---------------------------------------------------
Total ractor count: 1
Ruby thread count for this ractor: 5
-- C level backtrace information -------------------------------------------
/usr/local/ruby/bin/ruby(rb_print_backtrace+0xd) [0x55dcf6947c3a] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_dump.c:772
/usr/local/ruby/bin/ruby(rb_vm_bugreport) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_dump.c:1088
/usr/local/ruby/bin/ruby(rb_assert_failure+0x77) [0x55dcf6751c85] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/error.c:879
/usr/local/ruby/bin/ruby(vm_cc_class_check+0x1f) [0x55dcf6740995] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_callinfo.h:339
/usr/local/ruby/bin/ruby(vm_search_method_fastpath+0x5) [0x55dcf691db28] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:2181
/usr/local/ruby/bin/ruby(vm_search_super_method) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:4637
/usr/local/ruby/bin/ruby(vm_sendish+0xac) [0x55dcf6938e40] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5520
/usr/local/ruby/bin/ruby(vm_exec_core) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:939
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(vm_call0_cc+0x136) [0x55dcf692f236] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:107
/usr/local/ruby/bin/ruby(send_internal+0x124) [0x55dcf6930834] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:1273
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0xa5) [0x55dcf6936fa5] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:835
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(rb_yield+0xa3) [0x55dcf692e033] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520
/usr/local/ruby/bin/ruby(rb_ary_each+0x7a) [0x55dcf6a0a8ba] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/array.c:2528
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(vm_call0_cc+0x136) [0x55dcf692f236] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:107
/usr/local/ruby/bin/ruby(send_internal+0x124) [0x55dcf6930834] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:1273
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0xa5) [0x55dcf6936fa5] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:835
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(catch_i+0xa3) [0x55dcf692da33] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520
/usr/local/ruby/bin/ruby(vm_catch_protect+0xfd) [0x55dcf69266bd] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:2360
/usr/local/ruby/bin/ruby(rb_catch_obj+0x28) [0x55dcf692697d] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:2386
/usr/local/ruby/bin/ruby(rb_f_catch) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:2336
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(rb_yield+0xa3) [0x55dcf692e033] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520
/usr/local/ruby/bin/ruby(rb_ary_each+0x7a) [0x55dcf6a0a8ba] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/array.c:2528
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(load_iseq_eval+0x47) [0x55dcf67bfc55] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:728
/usr/local/ruby/bin/ruby(require_internal) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:1236
/usr/local/ruby/bin/ruby(rb_require_string+0x38) [0x55dcf67bff03] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:1329
/usr/local/ruby/bin/ruby(rb_f_require) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:969
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_call_alias+0x9d) [0x55dcf692b8dd] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3828
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0xa5) [0x55dcf6936fa5] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:835
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(vm_call0_cc+0x136) [0x55dcf692f236] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:107
/usr/local/ruby/bin/ruby(rb_funcallv_scope+0x6f) [0x55dcf6932fac] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:1062
/usr/local/ruby/bin/ruby(rb_funcallv) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:1077
/usr/local/ruby/bin/ruby(autoload_feature_require+0x70) [0x55dcf690bee6] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/variable.c:2739
/usr/local/ruby/bin/ruby(autoload_try_load) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/variable.c:2753
/usr/local/ruby/bin/ruby(rb_ensure+0x121) [0x55dcf67594e1] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/eval.c:1009
/usr/local/ruby/bin/ruby(rb_autoload_load+0x12f) [0x55dcf690a94f] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/variable.c:2814
/usr/local/ruby/bin/ruby(rb_const_lookup+0x0) [0x55dcf690ab9b] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/variable.c:2916
/usr/local/ruby/bin/ruby(rb_const_search_from) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/variable.c:2903
/usr/local/ruby/bin/ruby(rb_const_search+0x7) [0x55dcf690b097] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/variable.c:2938
/usr/local/ruby/bin/ruby(rb_const_get_0) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/variable.c:2865
/usr/local/ruby/bin/ruby(rb_mod_const_get+0x1f5) [0x55dcf67ec8e5] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/object.c:2401
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0xa5) [0x55dcf6936fa5] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:835
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(rb_yield+0xa3) [0x55dcf692e033] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520
/usr/local/ruby/bin/ruby(rb_ary_each+0x7a) [0x55dcf6a0a8ba] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/array.c:2528
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(rb_yield_values2+0xa0) [0x55dcf692dae0] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520
/usr/local/ruby/bin/ruby(each_pair_i_fast+0x37) [0x55dcf6780cf7] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/hash.c:3046
/usr/local/ruby/bin/ruby(RHASH_ST_TABLE+0x0) [0x55dcf678111f] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/hash.c:1253
/usr/local/ruby/bin/ruby(hash_foreach_iter) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/hash.c:1255
/usr/local/ruby/bin/ruby(st_general_foreach+0x3b) [0x55dcf68ab030] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/st.c:1516
/usr/local/ruby/bin/ruby(rb_st_foreach_check) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/st.c:1621
/usr/local/ruby/bin/ruby(hash_foreach_call+0x3d) [0x55dcf678431d] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/hash.c:1382
/usr/local/ruby/bin/ruby(rb_ensure+0x121) [0x55dcf67594e1] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/eval.c:1009
/usr/local/ruby/bin/ruby(rb_hash_foreach+0x91) [0x55dcf6784519] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/hash.c:1406
/usr/local/ruby/bin/ruby(rb_hash_foreach) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/hash.c:1392
/usr/local/ruby/bin/ruby(rb_hash_each_pair+0x47) [0x55dcf67845b7] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/hash.c:3081
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(rb_yield+0x11d) [0x55dcf692e0ad] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520
/usr/local/ruby/bin/ruby(rb_ensure+0x121) [0x55dcf67594e1] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/eval.c:1009
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(rb_yield+0xa3) [0x55dcf692e033] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520
/usr/local/ruby/bin/ruby(rb_ary_each+0x7a) [0x55dcf6a0a8ba] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/array.c:2528
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(yield_under+0x234) [0x55dcf692ceb4] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(rb_yield+0xa3) [0x55dcf692e033] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520
/usr/local/ruby/bin/ruby(rb_ary_each+0x7a) [0x55dcf6a0a8ba] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/array.c:2528
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(vm_call0_cc+0x136) [0x55dcf692f236] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:107
/usr/local/ruby/bin/ruby(rb_vm_call_kw+0x4e) [0x55dcf692f9fe] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:70
/usr/local/ruby/bin/ruby(rb_method_call_pass_called_kw+0x9f) [0x55dcf6834b7f] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/proc.c:2482
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(rb_yield+0xa3) [0x55dcf692e033] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520
/usr/local/ruby/bin/ruby(rb_ary_each+0x7a) [0x55dcf6a0a8ba] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/array.c:2528
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(vm_call0_cc+0x136) [0x55dcf692f236] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:107
/usr/local/ruby/bin/ruby(rb_vm_call_kw+0x4e) [0x55dcf692f9fe] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:70
/usr/local/ruby/bin/ruby(rb_method_call_pass_called_kw+0x9f) [0x55dcf6834b7f] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/proc.c:2482
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(rb_yield+0xa3) [0x55dcf692e033] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520
/usr/local/ruby/bin/ruby(rb_ary_each+0x7a) [0x55dcf6a0a8ba] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/array.c:2528
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(vm_call0_cc+0x136) [0x55dcf692f236] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:107
/usr/local/ruby/bin/ruby(rb_vm_call_kw+0x4e) [0x55dcf692f9fe] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:70
/usr/local/ruby/bin/ruby(rb_method_call_pass_called_kw+0x9f) [0x55dcf6834b7f] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/proc.c:2482
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(rb_yield+0xa3) [0x55dcf692e033] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520
/usr/local/ruby/bin/ruby(rb_ary_each+0x7a) [0x55dcf6a0a8ba] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/array.c:2528
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(vm_call0_cc+0x136) [0x55dcf692f236] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:107
/usr/local/ruby/bin/ruby(rb_vm_call_kw+0x4e) [0x55dcf692f9fe] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:70
/usr/local/ruby/bin/ruby(rb_method_call_pass_called_kw+0x9f) [0x55dcf6834b7f] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/proc.c:2482
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(rb_yield+0xa3) [0x55dcf692e033] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520
/usr/local/ruby/bin/ruby(rb_ary_each+0x7a) [0x55dcf6a0a8ba] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/array.c:2528
/usr/local/ruby/bin/ruby(vm_call0_cfunc_with_frame+0xe8) [0x55dcf692e7d0] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:170
/usr/local/ruby/bin/ruby(vm_call0_cfunc) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:184
/usr/local/ruby/bin/ruby(vm_call0_body) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:230
/usr/local/ruby/bin/ruby(vm_call0_cc+0x136) [0x55dcf692f236] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:107
/usr/local/ruby/bin/ruby(rb_vm_call_kw+0x4e) [0x55dcf692f9fe] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:70
/usr/local/ruby/bin/ruby(rb_method_call_pass_called_kw+0x9f) [0x55dcf6834b7f] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/proc.c:2482
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(load_iseq_eval+0x47) [0x55dcf67bfc55] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:728
/usr/local/ruby/bin/ruby(require_internal) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:1236
/usr/local/ruby/bin/ruby(rb_require_string+0x35) [0x55dcf67bff94] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:1329
/usr/local/ruby/bin/ruby(rb_f_require_relative) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:989
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0xa5) [0x55dcf6936fa5] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:835
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(load_iseq_eval+0x47) [0x55dcf67bfc55] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:728
/usr/local/ruby/bin/ruby(require_internal) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:1236
/usr/local/ruby/bin/ruby(rb_require_string+0x38) [0x55dcf67bff03] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:1329
/usr/local/ruby/bin/ruby(rb_f_require) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:969
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_call_alias+0x9d) [0x55dcf692b8dd] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3828
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0xa5) [0x55dcf6936fa5] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:835
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(load_iseq_eval+0x47) [0x55dcf67bfc55] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:728
/usr/local/ruby/bin/ruby(require_internal) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:1236
/usr/local/ruby/bin/ruby(rb_require_string+0x38) [0x55dcf67bff03] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:1329
/usr/local/ruby/bin/ruby(rb_f_require) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:969
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_call_alias+0x9d) [0x55dcf692b8dd] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3828
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0xa5) [0x55dcf6936fa5] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:835
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(rb_yield+0xa3) [0x55dcf692e033] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:1520
/usr/local/ruby/bin/ruby(rb_ary_each+0x7a) [0x55dcf6a0a8ba] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/array.c:2528
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0x135) [0x55dcf6937035] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:815
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(vm_call0_cc+0x136) [0x55dcf692f236] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:107
/usr/local/ruby/bin/ruby(send_internal+0x124) [0x55dcf6930834] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_eval.c:1273
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0xa5) [0x55dcf6936fa5] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:835
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(raise_load_if_failed+0x0) [0x55dcf67bd63d] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:728
/usr/local/ruby/bin/ruby(rb_load_internal) /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:791
/usr/local/ruby/bin/ruby(rb_f_load+0x89) [0x55dcf67bdef9] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/load.c:864
/usr/local/ruby/bin/ruby(vm_call_cfunc_with_frame_+0x108) [0x55dcf6924b18] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:3458
/usr/local/ruby/bin/ruby(vm_sendish+0x178) [0x55dcf691d828] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm_insnhelper.c:5517
/usr/local/ruby/bin/ruby(vm_exec_core+0xa5) [0x55dcf6936fa5] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/insns.def:835
/usr/local/ruby/bin/ruby(rb_vm_exec+0x12c) [0x55dcf692840c] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/vm.c:2399
/usr/local/ruby/bin/ruby(rb_ec_exec_node+0xc1) [0x55dcf67530b1] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/eval.c:287
/usr/local/ruby/bin/ruby(ruby_run_node+0x56) [0x55dcf6758406] /tmp/ruby-build.20230730080618.391.qAleew/ruby-3.3.0-b5c74d548872388921402ff2db36be15e924a89b/eval.c:328
/usr/local/ruby/bin/ruby(rb_main+0x21) [0x55dcf6752637] ./main.c:39
/usr/local/ruby/bin/ruby(main) ./main.c:58
/lib/x86_64-linux-gnu/libc.so.6(0x7f686651ad90) [0x7f686651ad90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7f686651ae40]
[0x55dcf6752685]
Updated by ko1 (Koichi Sasada) over 1 year ago
Thanks,
Could you try current master (I hope it was fixed).
Updated by byroot (Jean Boussier) over 1 year ago
@ko1 (Koichi Sasada) I don't see any more failures, thank you very much.
So overall the patch is all these commits?
c330037c1a38cc257dbe21022fcc7b13159c2821 - cc->cme should not be marked.
6391132c03ac08da0483adb986ff9a54e41f9e14 - fix typo (CACH_ -> CACHE_)
6dc15cc8895b28800d5c187929d846ac4eb7cd3f - do not clear cme but invalidate cc
7a7aba755d8da34555445510dc568380d0d96791 - check liveness of cc->klass and cc->cme_
087a2deccfb9d99961f1ce8526b80c5f72ee9a5d - check cc->* liveness strictly
36023d5cb751d62fca0c27901c07527b20170f4d - mark cc->cme_ if it is for super
280419d0e0ba3e96e19551c70cba789fbedd80e1 - calling->cd instead of calling->ci
I'm asking because I may test it more extensively in production which involve backporting it to our custom 3.2.
Updated by ko1 (Koichi Sasada) over 1 year ago
36023d5cb751d62fca0c27901c07527b20170f4d - mark cc->cme_ if it is for super
for this SEGV. But other patches can be required.
Updated by cedricm (Cédric Maïon) about 1 year ago
Hi,
Will this be backported to 3.2?
I'm currently seeing this in production on a large Rails app: when YJIT is enabled, after usual initial warmup, unicorn workers memory usage continue to grows linearly until reaching a significant amount of RAM -- at which point they are either killed by a monitoring tool due to unreasonable memory usage (>650MB), or some cache gets flushed and a big chunk of memory gets released (and then it goes back for another round).
Memory usage is perfectly stable without YJIT (~420MB).
I believe that this is linked to this bug.
Cheers
Updated by byroot (Jean Boussier) about 1 year ago
This bug is unrelated to YJIT, it happens with or without YJIT enabled. It thus cannot explain what you are seeing.
As for memory usage with YJIT, what you are describing isn't out of the ordinary. The default --yjit-exec-mem-size
is 64MB
, and the rule of thumb is that if you had the metadata YJIT need to track, you will find an overhead of ~4x yjit-exec-mem-size
, so roughly 250MB.
So you probably want to raise your Unicorn memory limit a bit more, or reduce the --yjit-exec-mem-size
.
Updated by byroot (Jean Boussier) 6 months ago
- Related to Bug #20485: Simple use of Fiber makes GC leak objects with singleton method added