Feature #21539: Export symbols for external profiling of stripped shared Ruby builds

• Status: Open
• Assignee: -
• Target version: -

[dalehamel (Dale Hamel)]

Motivation

External profilers such as OpenTelemetry's eBPF profiler collect Ruby stacks from outside the Ruby process.

Distribution packages commonly ship shared libruby but strip the full symbol table (.symtab) and debug information. Some of the symbols needed to discover Ruby's execution state are currently local symbols, so they
disappear when stripped. Public dynamic symbols in .dynsym survive normal stripping.

I'd like to narrow this proposal to making the necessary discovery symbols available in these stripped shared builds.

Proposed exports

Please export these three existing symbols from shared libruby:

Symbol	Purpose
ruby_current_ec	Locate the sampled native thread's execution context through TLS, providing access to its Ruby stack.
rb_vm_exec	Identify the function's address range, where native stack unwinding should hand off to Ruby stack unwinding. The profiler does not call this function.
ruby_global_symbols	Resolve Ruby method IDs to names for methods implemented in C. Stack walking can work without this, but those method names become <unknown cfunc>.

The profiler also needs ruby_version, which is already exported and requires no change.

The requested outcome is that these symbols are available through .dynsym in installed shared builds, without requiring distributions to retain the full symbol table or users to install debug information just to locate them.

Scope

This is a symbol-visibility request, not a request for a general out-of-process profiling API:

• No requirement to stabilize internal struct layouts or make these internals a supported C-extension API.
• No additional profiling stack, per-frame bookkeeping, or in-process profiling machinery.
• The profiler remains responsible for supporting version-specific layouts and interpreter/JIT behavior.

This would address the symbol-discovery obstacle for stripped distribution builds without taking on the broader API design discussed in #19119.

Related profiler discussion: https://github.com/open-telemetry/opentelemetry-ebpf-profiler/issues/202

[mame (Yusuke Endoh)]

• Related to Feature #19119: Add an interface for out-of-process profiling tools to access Ruby information added

[ivoanjo (Ivo Anjo)]

+1 this would be really cool as well for the Datadog profiler as we currently rely on access to internal headers and thus are together with rbspy/rbperf on the group of "needs to be updated whenever Ruby internals change".

[dalehamel (Dale Hamel)]

• Subject changed from Facilitate walking native and interpreter (and jit?) stacks from outside of the ruby process to Export symbols for external profiling of stripped shared Ruby builds
• Description updated (diff)

[dalehamel (Dale Hamel)]

Updated the description and subject to reduce the scope of this substantially to just exporting a few symbols

[Eregon (Benoit Daloze)]

+1, this would make external profilers more reliable (there are several important advantages of external profilers, e.g. reduce a lot the observer effect, no need to change anything in the already-running program).
Datadog is shipping the OpenTelemetry's eBPF profiler and it needs this change to work for the common case of Ruby installations with symbols stripped (e.g. by the Linux distribution).

[Eregon (Benoit Daloze)]

For clarity, it's also fine if these symbols get renamed/changed/etc, exporting them doesn't imply any future compatibility, it's just best-effort.
Without the export it's very difficult and hacky to get to them, which translates to cases that simply don't work based on how Ruby was installed (e.g. stripped or not).

[dalehamel (Dale Hamel)]

• File protected.patch protected.patch added
• File exported.patch exported.patch added

Summary

Exporting the three symbols had no measurable cost on ruby-bench's lobsters, with or without ZJIT, on x86-64 or arm64. With default visibility, though, GCC compiles vm.c differently, which can move interpreter microbenchmarks on x86-64 by roughly ±5–9%. ELF protected visibility keeps the symbols in .dynsym while leaving the interpreter's generated code unchanged, so that's what I'd suggest.

Performance evaluation

Setup

• Ruby master b709bdb3e37451bb1103b50fb280968dbf038103, ruby-bench 13e09f746431f2159db478f15b80a2c7cd95aadb.
• Ubuntu 24.04, GCC 13.3, default flags (-O3). Configured with --enable-shared --enable-yjit --enable-zjit.
• Two dedicated cloud VMs: x86-64 (Intel Xeon Platinum 8581C, SMT disabled) and arm64 (Google Axion, Neoverse-V2). Benchmark processes were pinned to an isolated core.
• Three builds: unmodified, exported with default visibility, and exported with ELF protected visibility.
• Each block ran every build once, each in a fresh process, with build order balanced across blocks. lobsters used 18 blocks (15 warmup + 20 measured iterations); the microbenchmarks used 12.
• Results are paired ratios of mean iteration time with 95% Student-t intervals. They are not adjusted for multiple comparisons.

lobsters (negative = faster than unmodified)

CPU	Mode	default visibility	protected
x86-64	interpreter	-0.58% [-1.04, -0.11]	-0.77% [-1.25, -0.28]
x86-64	ZJIT	-0.37% [-0.62, -0.12]	-0.11% [-0.28, +0.06]
arm64	interpreter	-1.06% [-2.19, +0.07]	-0.90% [-1.70, -0.09]
arm64	ZJIT	+0.40% [-0.86, +1.67]	+0.35% [-1.12, +1.84]

I wouldn't count the small interpreter speedups as a real benefit. They also appear with protected, whose vm_exec_core compiles identically to the unmodified build (only addresses differ), so they are most likely code layout effects.

With the patch, --disable-shared builds export these symbols too, because ruby is linked with -rdynamic. Their compiler output is essentially unchanged, and lobsters showed no difference (all point estimates within ±0.2%).

Microbenchmarks (interpreter)

With default visibility, GCC has to assume rb_vm_exec may be interposed, so it optimizes vm.c differently. The 14 internal calls and tail calls to rb_vm_exec also go through the PLT. On x86-64, vm_exec_core grows from 27,387 to 27,456 bytes and its stack frame grows by 32 bytes. On arm64 it grows from 26,184 to 26,512 bytes.

This affects some interpreter microbenchmarks on x86-64:

CPU	Benchmark	default visibility	protected
x86-64	fib	+5.29% [+4.47, +6.12]	+0.87% [-0.47, +2.23]
x86-64	cfunc_itself	-8.51% [-12.81, -4.00]	-2.75% [-7.55, +2.30]

The x86-64 fib regression executes only 0.49% more instructions. Hardware counters show it as front-end stalls: the front-end-bound share of pipeline slots rises from 3.2% to 12.5%, versus 4.4% with protected (one profiling run per build). That points to a layout change in the dispatch loop, not PLT or TLS overhead; fib essentially never reaches the PLT calls. On arm64, both microbenchmarks stay within about 1% for both variants, apart from one slow outlier process.

An independent x86-64 rebuild with GCC 13.4 and GCC 14.3 (nixpkgs, with different hardening flags) confirmed the same code generation pattern; timing wasn't repeated there:

• The default-visibility vm_exec_core grows by 23 instructions with GCC 13.4 and 69 with GCC 14.3. The size of the change depends on the compiler.
• Both compilers show the same 32-byte frame growth and the same 14 PLT sites.

Suggested implementation

protected visibility avoids the code generation change. On ELF the symbols still appear in .dynsym after strip --strip-all, listed as GLOBAL PROTECTED, but references inside libruby stay bound locally:

• Calls to rb_vm_exec stay direct.
• vm_exec_core compiles identically to the unmodified build, instruction for instruction. This was checked with GCC 13.3 on both architectures, and with GCC 13.4 and 14.3 on x86-64.
• The remaining code differences are small:
  • In symbol.c, each access to ruby_global_symbols gains one instruction: a GOT load on arm64, a lea after linker relaxation on x86-64.
  • On x86-64, the ruby_current_ec accesses in vm.c switch from local-dynamic to general-dynamic TLS, with the same instruction count and the same __tls_get_addr call.
• The linker still emits a TLS relocation that names ruby_current_ec, which is how profilers locate it in stripped builds.

The tested change (abbreviated):

--- a/vm.c
+++ b/vm.c
-VALUE vm_exec(rb_execution_context_t *);
+__attribute__((visibility("protected"))) VALUE vm_exec(rb_execution_context_t *);
@@
 #ifdef RB_THREAD_LOCAL_SPECIFIER
-RB_THREAD_LOCAL_SPECIFIER rb_execution_context_t *ruby_current_ec;
+RUBY_SYMBOL_EXPORT_BEGIN
+__attribute__((visibility("protected"))) RB_THREAD_LOCAL_SPECIFIER rb_execution_context_t *ruby_current_ec;
+RUBY_SYMBOL_EXPORT_END
--- a/thread_pthread.h
+++ b/thread_pthread.h
-    RUBY_EXTERN RB_THREAD_LOCAL_SPECIFIER struct rb_execution_context_struct *ruby_current_ec;
+    RUBY_SYMBOL_EXPORT_BEGIN
+    RUBY_EXTERN __attribute__((visibility("protected"))) RB_THREAD_LOCAL_SPECIFIER struct rb_execution_context_struct *ruby_current_ec;
+    RUBY_SYMBOL_EXPORT_END
--- a/symbol.c
+++ b/symbol.c
-rb_symbols_t ruby_global_symbols = {
+RUBY_SYMBOL_EXPORT_BEGIN
+__attribute__((visibility("protected"))) rb_symbols_t ruby_global_symbols = {
     .next_id = tNEXT_ID,
 };
+RUBY_SYMBOL_EXPORT_END

The default-visibility variant I compared uses RUBY_FUNC_EXPORTED and RUBY_SYMBOL_EXPORT_BEGIN/END at the same four places.

Notes:

• An upstream version would need a macro that falls back to the existing default-visibility export where protected isn't supported. Mach-O compilers warn about it and fall back to default visibility.
• The vm.c definition must be annotated as well as the header. On arm64 and ppc64, RB_THREAD_CURRENT_EC_NOINLINE skips the thread_pthread.h declaration.
• Protected data is only a problem if an executable links directly against ruby_global_symbols and gets a copy relocation. Nothing outside symbol.c references it in C; Ruby's own lldb helpers already find it by name. Profilers that look symbols up through ELF aren't affected, and TLS variables aren't copy-relocated.

All of this was measured with GCC; Clang wasn't tested. Full patches attached

[alanwu (Alan Wu)]

• Related to Bug #22396: Using rb_postponed_job_trigger causes YJIT/ZJIT compiled methods side exits added

[nayefg (Nayef Ghattas)]

• File debug-aliases.patch debug-aliases.patch added

Following up on protected.patch: in our tests, exporting the original symbols had two effects, even with protected visibility.

• With GCC, exporting ruby_global_symbols changes how rb_id2str addresses the table. On x86-64 the linker relaxes the GOT load into an extra LEA; on arm64 the GOT load remains. Against a build exporting only ruby_current_ec, a loop calling rb_id2name was about 2% slower on a Xeon and 6% slower by median on a Neoverse N1.
• The longer code also moved other functions on x86-64. rb_backref_get moved by 80 bytes without changing its instructions, and a loop calling it was about 4% slower on the Xeon against the same reference build. Changing its alignment shifted timings by 3–5%, showing that code placement matters too.

I'd suggest a small variation: leave the originals unchanged and add a protected alias next to each definition. These aliases would be best-effort debugging symbols, without a stable-layout guarantee. Ruby's own code keeps using the originals, with their existing declarations and visibility.

extern __typeof__(ruby_global_symbols) ruby_debug_global_symbols
    __attribute__((alias("ruby_global_symbols"), visibility("protected")));

Proposed names, open to discussion:

• ruby_debug_current_ec: the same TLS slot as ruby_current_ec;
• ruby_debug_vm_exec: the actual rb_vm_exec function, including its full symbol size;
• ruby_debug_global_symbols: the original symbol table.

We tested the alias approach on Ruby b709bdb3 with GCC 13.3.0 and Clang 14.0.6 on x86-64 and arm64 Linux. The aliases survived strip --strip-all and were still found by the OpenTelemetry profiler's ELF/TLS lookup code. Internal access sequences were unchanged, and the x86-64 builds also avoided the code displacement seen with direct exports. The benchmarks showed no consistent slowdown.

The full patch is attached; the investigation repository has the assembly and benchmark details.

[nobu (Nobuyoshi Nakada)]

I think I could understand the necessity about the former 2 symbols, but not for ruby_global_symbols.
The profiler has to access it during the VM is locked?

The third patch, debug-aliases.patch looks fine, but it doesn't look nice that repeating the complex conditions and revealing aliasing.

[dalehamel (Dale Hamel)]

• File debug-export.patch debug-export.patch added

The profiler has to access it during the VM is locked?

No. The profiler only reads ruby_global_symbols, and it doesn't need the VM lock (which an external process couldn't take anyway).

It is used to name methods implemented in C. A Ruby method's frame has its name as strings in the iseq, but a C method's frame only has the method entry's def->original_id, and ruby_global_symbols is the only place to turn that ID into a name. Without it, C methods such as Array#each and Kernel#require, and the methods of C extensions, lose their names (they appear as e.g. Array#<unknown cfunc>). The C function pointer doesn't help: over 90% of the functions behind core methods have no symbol in a stripped libruby, and their C names aren't the Ruby method names anyway.

The lookup happens later, in the profiler's own process. At sample time it only records the method entry pointer. Afterwards it reads the ID and its name from the Ruby process's memory while Ruby keeps running, following the same path as rb_id2str(): next_id, ids, the bucket, then the entry. Ruby itself already reads the table this way without the VM lock: only the writers, set_id_entry() and id_entry_dir_grow(), hold it, and they publish with release stores that get_id_serial_entry() pairs with acquire loads. An entry never changes once published, so a name read once can be cached. If a read races with a resize of the directory, that one lookup can fail; the Ruby process is never affected.

it doesn't look nice that repeating the complex conditions and revealing aliasing.

Good callout and agreed.

I've attached a revised version of debug-aliases.patch, debug-export.patch, that defines the condition once. A RUBY_DEBUG_EXPORT() macro in internal/compilers.h wraps the alias and visibility attributes, and reuses configure's existing HAVE_ATTRIBUTE_FUNCTION_ALIAS check. RUBY_ALIAS_FUNCTION itself doesn't fit here: without alias support it falls back to a wrapper function, and it has no form for variables. Each site is now one line:

RUBY_DEBUG_EXPORT_TLS(ruby_debug_current_ec, ruby_current_ec)
RUBY_DEBUG_EXPORT(ruby_debug_vm_exec, vm_exec)
RUBY_DEBUG_EXPORT(ruby_debug_global_symbols, ruby_global_symbols)

It is enabled only on Linux, where these profilers run. The ruby_debug_ prefix follows the debugger helpers libruby already exports, such as ruby_debug_print_value.

The machine output should be nearly identical to Nayef's patch and performance should be the same, just the code is cleaner in this one.

I tested it on b709bdb3 with GCC 13.3 on x86-64 and arm64, and with Clang 18 on x86-64; it also applies cleanly to current master.

• make test-basic passes, with no compiler warnings in the changed files.
• After strip --strip-all, .dynsym gains only the three ruby_debug_* names, each with the same value, size and type as its original.
• vm.o and symbol.o compile to the same instructions as the unmodified sources, except two assertion line numbers. With GCC on x86-64, no function in the linked libruby moves either.

If you'd prefer the check in configure, or other names, I can change that.

[Eregon (Benoit Daloze)]

@dalehamel (Dale Hamel) Could you update the dev-meeting summary at https://bugs.ruby-lang.org/issues/22417#note-2 to reflect the latest patch?

I think opening a PR to ruby/ruby would also be helpful to help review.

[dalehamel (Dale Hamel)]

Eregon (Benoit Daloze) wrote in #note-12:

@dalehamel (Dale Hamel) Could you update the dev-meeting summary at https://bugs.ruby-lang.org/issues/22417#note-2 to reflect the latest patch?

I think opening a PR to ruby/ruby would also be helpful to help review.

Thanks for the guidance, I updated the comment and opened the PR https://github.com/ruby/ruby/pull/19613