Bug #22188: addr2line doesn't find symbols when compiled with GCC LTO

• Status: Closed
• Assignee: luke-gru (Luke Gruber)
• Target version: -
• Backport: 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: DONE

[rwstauner (Randy Stauner)]

When compiling with LTO (the XCFLAGS and XLDFLAGS below) RUBY_CRASH_REPORT does
not resolve symbols.

(
  export optflags="-O3" cflags="-fno-omit-frame-pointer"
  export XCFLAGS="-flto=auto -ffat-lto-objects -Werror=lto-type-mismatch"
  export XLDFLAGS="-flto=auto -ffat-lto-objects"

  ./configure --disable-install-doc --disable-shared
  make -j miniruby
)

Simple crash simulation:

./miniruby -e 'Process.kill("SEGV", $$)'

On master the output includes:

-- C level backtrace information -------------------------------------------
/src/miniruby(0x5608c71bafdd) [0x5608c71bafdd]
/src/miniruby(0x5608c71c6f46) [0x5608c71c6f46]
/src/miniruby(0x5608c6f31370) [0x5608c6f31370]
/src/miniruby(0x5608c70e3150) [0x5608c70e3150]
/usr/lib/libc.so.6(0x7f62122ac2d0) [0x7f62122ac2d0]
/usr/lib/libc.so.6(kill+0xb) [0x7f62122ac4db]
/src/miniruby(0x5608c70e1a65) [0x5608c70e1a65]
/src/miniruby(0x5608c7189d1e) [0x5608c7189d1e]
/src/miniruby(0x5608c718dea7) [0x5608c718dea7]

/src/miniruby(0x5608c719417e) [0x5608c719417e]

/src/miniruby(0x5608c71b793c) [0x5608c71b793c]
/src/miniruby(0x5608c6f3d13c) [0x5608c6f3d13c]
/src/miniruby(0x5608c6f42321) [0x5608c6f42321]
/src/miniruby(0x5608c6e4f9b3) [0x5608c6e4f9b3]
/usr/lib/libc.so.6(0x7f62122956c1) [0x7f62122956c1]
/usr/lib/libc.so.6(__libc_start_main+0x89) [0x7f62122957f9]
[0x5608c6e4f9f5]

With a tiny change to fall back to symtab (even after parsing DWARF) we can get this
(empty lines added for comparison with next block):

-- C level backtrace information -------------------------------------------
/src/miniruby(rb_print_backtrace+0x1d) [0x55f6397aafdd] /src/vm_dump.c:1111

/src/miniruby(rb_vm_bugreport+0xa26) [0x55f6397b6f46] /src/vm_dump.c:1473

/src/miniruby(rb_bug_for_fatal_signal+0x110) [0x55f639521370] /src/error.c:1140
/src/miniruby(sigsegv+0x50) [0x55f6396d3150] /src/signal.c:948
/usr/lib/libc.so.6(0x7f991555d2d0) [0x7f991555d2d0]
/usr/lib/libc.so.6(kill+0xb) [0x7f991555d4db]
/src/miniruby(rb_f_kill+0x205) [0x55f6396d1a65] /src/signal.c:487
/src/miniruby(vm_call_cfunc_with_frame_+0xfe) [0x55f639779d1e] /src/vm_insnhelper.c:3821

/src/miniruby(vm_sendish.constprop.0+0xb7) [0x55f63977dea7] /src/vm_insnhelper.c:6056
/src/miniruby(vm_exec_core.lto_priv.0+0x6e) [0x55f63978417e] /src/insns.def:909
/src/miniruby(rb_vm_exec+0xec) [0x55f6397a793c] /src/vm.c:2825

/src/miniruby(rb_ec_exec_node+0xac) [0x55f63952d13c] /src/eval.c:284
/src/miniruby(ruby_run_node+0x61) [0x55f639532321] /src/eval.c:322
/src/miniruby(main+0x63) [0x55f63943f9b3] ./main.c:42

PR: https://github.com/ruby/ruby/pull/17743

Currently the code in addr2line.c has a stub:

case DW_FORM_ref_addr:
  goto finish; /* not supported yet */

If we add support for that we can get better names and inlined frames
(empty lines added for comparison with previous block):

-- C level backtrace information -------------------------------------------
/src/miniruby(rb_print_backtrace+0x1d) [0x55cd045f81dd] /src/vm_dump.c:1111

/src/miniruby(fprintf+0x0) [0x55cd04604146] /src/vm_dump.c:1473
/src/miniruby(rb_vm_bugreport) /src/vm_dump.c:1476

/src/miniruby(rb_bug_for_fatal_signal+0x110) [0x55cd0436e370] /src/error.c:1140
/src/miniruby(sigsegv+0x50) [0x55cd04520150] /src/signal.c:948
/usr/lib/libc.so.6(0x7f9ef85522d0) [0x7f9ef85522d0]
/usr/lib/libc.so.6(kill+0xb) [0x7f9ef85524db]
/src/miniruby(rb_f_kill+0x205) [0x55cd0451ea65] /src/signal.c:487
/src/miniruby(vm_call_cfunc_with_frame_+0xfe) [0x55cd045c6f1e] /src/vm_insnhelper.c:3821

/src/miniruby(vm_sendish+0xb7) [0x55cd045cb0a7] /src/vm_insnhelper.c:6056
/src/miniruby(vm_exec_core+0x6e) [0x55cd045d137e] /src/insns.def:909
/src/miniruby(vm_exec_loop+0x16) [0x55cd045f4b3c] /src/vm.c:2825
/src/miniruby(rb_vm_exec) /src/vm.c:2801

/src/miniruby(rb_ec_exec_node+0xac) [0x55cd0437a13c] /src/eval.c:284
/src/miniruby(ruby_run_node+0x61) [0x55cd0437f321] /src/eval.c:322
/src/miniruby(rb_main+0x22) [0x55cd0428c9b3] ./main.c:42

/src/miniruby(main) ./main.c:62
/usr/lib/libc.so.6(0x7f9ef853b6c1) [0x7f9ef853b6c1]
/usr/lib/libc.so.6(__libc_start_main+0x89) [0x7f9ef853b7f9]
/src/miniruby(_start+0x25) [0x55cd0428c9f5] ./main.c:63

PR: https://github.com/ruby/ruby/pull/17744

[rwstauner (Randy Stauner)]

• Description updated (diff)

[rwstauner (Randy Stauner)]

• Subject changed from Debug symbols missing when compiled with GCC LTO to addr2line doesn't find symbols when compiled with GCC LTO

[luke-gru (Luke Gruber)]

• Assignee set to luke-gru (Luke Gruber)
• Backport changed from 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: UNKNOWN to 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: REQUIRED

In my opinion this deserves a backport. We are using LTO in production in some of our apps (Ruby 4.0.x) and not having backtraces makes debugging considerably harder. PR here.

• Status changed from Open to Closed

Applied in changeset git|fcb4bceedb3949f8d6c40ceed29cb61f1ab37632.

---

Support cross-CU DW_FORM_ref_addr in addr2line's abstract_origin

Under GCC LTO the C-level backtrace in a crash report printed only bare
hex addresses (e.g. "ruby(0x...) [0x...]") for every ruby frame, even
though gdb resolved the same binary fine.

The concrete out-of-line function DIE carries DW_AT_low_pc, so the crash
PC matches a DIE and the DWARF parse "succeeds"; but its name is reached
only through a DW_AT_abstract_origin encoded as DW_FORM_ref_addr, a
section-relative reference into another CU (LTO emits the concrete and
abstract instances in separate CUs). addr2line.c handled the CU-relative
ref forms but had DW_FORM_ref_addr as "not supported yet".

[Bug #22188]

[rwstauner (Randy Stauner)]

backport for 4.0: https://github.com/ruby/ruby/pull/18011

[alanwu (Alan Wu)]

• Backport changed from 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: REQUIRED to 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: DONE

Backport PR merged.

[rwstauner (Randy Stauner)]

It might be worth noting for anyone that wants to try this, if you do LTO with fat LTO objects and a static build (as described at the top) this will make native extension compilation much slower.

You can speed this up by stripping the LTO IR from the object that the gems will link against:

objcopy --remove-section=".gnu.lto_*" "${INSTALL_DIR}/lib/libruby-static.a"