Bug #22476
open
rb_profile_frames() called from a signal handler segfaults while vm_make_env_each() is escaping a chain of frames
Description
Sampling profilers (like StackProf, Vernier, Datadog) call rb_profile_frames() from a signal handler.
Datadog uses a slight variant of rb_profile_frames() but it does not matter, it reproduces either way.
rb_profile_frames() segfaults with a NULL dereference if the signal arrives while vm_make_env_each() is moving a chain of frames to the heap, e.g. during proc {}, define_method(:foo) {} or Thread.new {} called from a block.
This was first seen in production with the Datadog profiler on Ruby 3.4 with YJIT (background: https://github.com/DataDog/dd-trace-rb/pull/6173):
ddtrace_rb_profile_frames (private_vm_api_access.c:528)
prepare_sample_thread (collectors_stack.c:636)
handle_sampling_signal (collectors_cpu_and_wall_time_worker.c:686)
__kernel_rt_sigreturn
rb_yjit_invalidate_ep_is_bp (yjit/src/invariants.rs:579)
vm_make_env_each (vm.c:1044)
vm_make_env_each (vm.c:969)
vm_make_env_each (vm.c:969)
rb_vm_make_proc_lambda (vm.c:1429)
rb_mod_define_method_with_visibility (proc.c:2260)
rb_mod_define_method (proc.c:2368)
vm_call_cfunc_with_frame (vm_insnhelper.c:3840)
We hadn't seen it before with the Datadog profiler because we were not retrieving the CME for Ruby frames (only the ISEQ). We were getting the CME only for native frames.
The attached script below reproduces it on master using plain rb_profile_frames(), with no gems. It also crashes without YJIT.
Reproduction¶
ruby rb_profile_frames_signal_crash.rb (MODE=define_method and MODE=thread also crash). The script compiles a small C extension that installs a SIGPROF handler calling rb_profile_frames(), and starts a pthread that sends SIGPROF to the main thread with pthread_kill().
On ruby 4.1.0dev (2026-10-27T12:50:14Z master 32f06ce98c) +PRISM [arm64-darwin25] it crashes within ~0.03-0.25s with back-to-back signals, and within ~1s with INTERVAL_US=1000 (1 kHz sampling):
rb_profile_frames_signal_crash.rb:112: [BUG] Segmentation fault at 0x0000000000000000
ruby 4.1.0dev (2026-10-27T12:50:14Z master 32f06ce98c) +PRISM [arm64-darwin25]
Native backtrace of the crashing thread (from macOS sample):
rb_vm_make_proc_lambda vm.c:1733
vm_make_env_each vm.c:1141
_sigtramp
sigprof_handler (the SIGPROF handler of the reproducer)
thread_profile_frames vm_backtrace.c:2144
_sigtramp
rb_bug_for_fatal_signal error.c:1167
rb_vm_bugreport vm_dump.c:1433
rb_vmdebug_stack_dump_raw vm_dump.c:503
Side note: after printing the [BUG] header, the crash reporter hangs in rb_vmdebug_stack_dump_raw() (vm_dump.c:503) and the process never exits, so it needs timeout -s KILL. Probably it trips over the same inconsistent frame.
Cause¶
Let B be a block frame whose prev EP is a local frame A that has not escaped yet. vm_make_env_each(B):
- recurses into
vm_make_env_each(A)(vm.c:1141). At the end, this overwrites the flags slot of A's on-stack EP with the env object:VM_STACK_ENV_WRITE(ep, 0, (VALUE)env)(vm.c:1213). - Only then repoints B's
ep[VM_ENV_DATA_INDEX_SPECVAL]to A's heap EP (vm.c:1142).
Between 1 and 2, B's prev EP still points to A's old stack EP, whose ep[0] is now an object pointer instead of flags. If rb_profile_frames() runs at that point, rb_vm_frame_method_entry_unchecked() (called from thread_profile_frames(), vm_backtrace.c:2143) walks from B:
- B's ME_CREF is not a method entry, so it moves on to A's stale stack EP.
VM_ENV_LOCAL_P_UNCHECKED()reads the env pointer as flags. The pointer is aligned, soVM_ENV_FLAG_LOCAL(0x2) is clear and A is treated as non-local.- If A's ME_CREF is not a method entry either (e.g. A is a class body or the top-level frame, which have a cref), it follows
VM_ENV_PREV_EP_UNCHECKED(), i.e.ep[-1]. For a local frame with no block that isVM_BLOCK_HANDLER_NONE, so the next EP is NULL. - It dereferences NULL → SEGV.
On 3.4, rb_yjit_invalidate_ep_is_bp() runs inside vm_make_env_each(A) after step 1, which widens the window. That explains the YJIT frame in the production backtrace. On master that call happens before the env is created, so the window is only the function return, but it is still easy to hit.
Possible fix directions¶
- Keep A's stack EP walkable until every child EP that points to it has been updated. For example, update B's SPECVAL before A's stack
ep[0]is overwritten, or keep the env reference that the stack slot needs for GC marking somewhere other than the flags slot. - Or make
rb_vm_frame_method_entry_unchecked()stop walking when the prev EP is NULL. That avoids this crash, but on stale data it could still return a wrong method entry.
Affected versions¶
I could verify it locally and have it crash within 1 second with ruby rb_profile_frames_signal_crash.rb on:
- master
- 4.0
- 3.4
- 3.3
- 3.2
- Likely older versions
Files
Updated by 
Updated by