Misc #19146: Failed to activate YJIT for 3.2.0-dev on macOS Ventura (M1: Apple Silicon)

• Status: Closed
• Assignee: -

[hachi8833 (Shozo Hatta)]

(hsbt kindly suggested to open the issue here. Thank you!)

I found that I'm unable to activate YJIT for the latest 3.2.0-dev in rbenv with ruby_build.
Then I tried the build from the freshest 3.2.0-dev source, but the symptom remains.

I checked the setup with https://github.com/ruby/ruby/blob/master/doc/contributing/building_ruby.md

• M1 Macbook Pro 2021
• macOS Ventura 13.0.1
• rbenv 1.2.0-16-gc4395e5
• CommandLine Tools: 14.1.0.0.1.1666437224
• rustc 1.62.1 (e092d0b6b 2022-07-16)
• Ruby: 3.1.2p20 (2022-04-12 revision 4491bb740a) [arm64-darwin21]
• Homebrew: 3.6.12
  • openssl@3: stable 3.0.7 (bottled) [keg-only]
  • readline: stable 8.2.1 (bottled) [keg-only]
  • libyaml: stable 0.2.5 (bottled)
  • bison: stable 3.8.2 (bottled) [keg-only]
  • gperf: stable 3.1 (bottled)
  • zlib: stable 1.2.13 (bottled), HEAD [keg-only]
  • libffi: stable 3.4.4 (bottled), HEAD [keg-only]
• source: 3.2.0-dev(master, 66e5200)

As far as I tried, if and only if specifying --disable-yjit, the build is successful:

# success with --disable-yjit
$ ./autogen.sh;mkdir build && cd build;mkdir ~/.rubies;../configure --prefix="${HOME}/.rubies/ruby-master" --disable-yjit --with-opt-dir="$(brew --prefix openssl)" --disable-install-doc --disable-install-rdoc;make install

(Note that addding (or not adding) --disable-shared or --enable-shared to ../configure does not affect the success)

# success with --disable-yjit
Configuration summary for ruby version 3.2.0

   * Installation prefix: /Users/hachi8833/.rubies/ruby-master
   * exec prefix:         ${prefix}
   * arch:                arm64-darwin22
   * site arch:           ${arch}
   * RUBY_BASE_NAME:      ruby
   * ruby lib prefix:     ${libdir}/${RUBY_BASE_NAME}
   * site libraries path: ${rubylibprefix}/${sitearch}
   * vendor path:         ${rubylibprefix}/vendor_ruby
   * target OS:           darwin22
   * compiler:            clang
   * with thread:         pthread
   * with coroutine:      arm64
   * enable shared libs:  no
   * dynamic library ext: bundle
   * CFLAGS:              -fdeclspec ${optflags} ${debugflags} ${warnflags}
   * LDFLAGS:             -L. -L/opt/homebrew/opt/openssl@3/lib \
                          -fstack-protector-strong \
                          -L/opt/homebrew/opt/openssl@3/lib
   * DLDFLAGS:            -L/opt/homebrew/opt/openssl@3/lib \
                          -L/opt/homebrew/opt/openssl@3/lib \
                          -Wl,-multiply_defined,suppress
   * optflags:            -O3 -fno-fast-math
   * debugflags:          -ggdb3
   * warnflags:           -Wall -Wextra -Wextra-tokens \
                          -Wdeprecated-declarations -Wdivision-by-zero \
                          -Wdiv-by-zero -Wimplicit-function-declaration \
                          -Wimplicit-int -Wmisleading-indentation \
                          -Wpointer-arith -Wshorten-64-to-32 \
                          -Wwrite-strings -Wold-style-definition \
                          -Wmissing-noreturn -Wno-cast-function-type \
                          -Wno-constant-logical-operand -Wno-long-long \
                          -Wno-missing-field-initializers \
                          -Wno-overlength-strings -Wno-parentheses-equality \
                          -Wno-self-assign -Wno-tautological-compare \
                          -Wno-unused-parameter -Wno-unused-value \
                          -Wunused-variable -Wundef
   * strip command:       strip -A -n
   * install doc:         no
   * MJIT support:        yes
   * YJIT support:        no
   * man page type:       doc
   * BASERUBY -v:         ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) \
                          [arm64-darwin21]

But without specifying --disable-yjit, the build fails regardless any other options such as --disable-shared or --enable-shared:

# fails if --disable-yjit is missing:
$ ./autogen.sh;mkdir build && cd build;mkdir ~/.rubies;../configure --prefix="${HOME}/.rubies/ruby-master" --with-opt-dir="$(brew --prefix openssl)" --disable-install-doc --disable-install-rdoc;make install

# fails if --disable-yjit is missing:
Configuration summary for ruby version 3.2.0

   * Installation prefix: /Users/hachi8833/.rubies/ruby-master
   * exec prefix:         ${prefix}
   * arch:                arm64-darwin22
   * site arch:           ${arch}
   * RUBY_BASE_NAME:      ruby
   * ruby lib prefix:     ${libdir}/${RUBY_BASE_NAME}
   * site libraries path: ${rubylibprefix}/${sitearch}
   * vendor path:         ${rubylibprefix}/vendor_ruby
   * target OS:           darwin22
   * compiler:            clang
   * with thread:         pthread
   * with coroutine:      arm64
   * enable shared libs:  no
   * dynamic library ext: bundle
   * CFLAGS:              -fdeclspec ${optflags} ${debugflags} ${warnflags}
   * LDFLAGS:             -L. -L/opt/homebrew/opt/openssl@3/lib \
                          -fstack-protector-strong \
                          -L/opt/homebrew/opt/openssl@3/lib
   * DLDFLAGS:            -L/opt/homebrew/opt/openssl@3/lib \
                          -L/opt/homebrew/opt/openssl@3/lib \
                          -Wl,-multiply_defined,suppress
   * optflags:            -O3 -fno-fast-math
   * debugflags:          -ggdb3
   * warnflags:           -Wall -Wextra -Wextra-tokens \
                          -Wdeprecated-declarations -Wdivision-by-zero \
                          -Wdiv-by-zero -Wimplicit-function-declaration \
                          -Wimplicit-int -Wmisleading-indentation \
                          -Wpointer-arith -Wshorten-64-to-32 \
                          -Wwrite-strings -Wold-style-definition \
                          -Wmissing-noreturn -Wno-cast-function-type \
                          -Wno-constant-logical-operand -Wno-long-long \
                          -Wno-missing-field-initializers \
                          -Wno-overlength-strings -Wno-parentheses-equality \
                          -Wno-self-assign -Wno-tautological-compare \
                          -Wno-unused-parameter -Wno-unused-value \
                          -Wunused-variable -Wundef
   * strip command:       strip -A -n
   * install doc:         no
   * MJIT support:        yes
   * YJIT support:        yes
   * man page type:       doc
   * BASERUBY -v:         ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) \
                          [arm64-darwin21]

# fails if --disable-yjit is missing:
(...)
building Rust YJIT (release mode)
touch yjit/target/release/libyjit.a
linking miniruby
ld: warning: ignoring file yjit/target/release/libyjit.a, building for macOS-arm64 but attempting to link with file built for macOS-x86_64
Undefined symbols for architecture arm64:
  "_rb_yjit_before_ractor_spawn", referenced from:
      _ractor_create in ractor.o
  "_rb_yjit_bop_redefined", referenced from:
      _rb_vm_check_redefinition_opt_method in vm.o
  "_rb_yjit_call_threshold", referenced from:
      _rb_vm_exec in vm.o
      _vm_sendish in vm.o
  "_rb_yjit_cme_invalidate", referenced from:
      _clear_method_cache_by_id_in_class in vm.o
  "_rb_yjit_code_gc", referenced from:
      _mjit_compile_invokebuiltin_for_rb_yjit_code_gc in yjit.o
      _Init_builtin_yjit.yjit_table in yjit.o
  "_rb_yjit_constant_ic_update", referenced from:
      _vm_exec_core in vm.o
  "_rb_yjit_constant_state_changed", referenced from:
      _rb_clear_constant_cache_for_id in vm.o
  "_rb_yjit_disasm_iseq", referenced from:
      _mjit_compile_invokebuiltin_for_rb_yjit_disasm_iseq in yjit.o
      _Init_builtin_yjit.yjit_table in yjit.o
  "_rb_yjit_enabled_p", referenced from:
      _rb_jit_cont_finish in cont.o
      _rb_jit_cont_init in cont.o
      _fiber_initialize in cont.o
      _rb_threadptr_root_fiber_setup in cont.o
      _cont_free in cont.o
      _cont_capture in cont.o
      _clear_method_cache_by_id_in_class in vm.o
      ...
  "_rb_yjit_get_exit_locations", referenced from:
      _mjit_compile_invokebuiltin_for_rb_yjit_get_exit_locations in yjit.o
      _Init_builtin_yjit.yjit_table in yjit.o
  "_rb_yjit_get_stats", referenced from:
      _mjit_compile_invokebuiltin_for_rb_yjit_get_stats in yjit.o
      _Init_builtin_yjit.yjit_table in yjit.o
  "_rb_yjit_init_rust", referenced from:
      _rb_yjit_init in yjit.o
  "_rb_yjit_insns_compiled", referenced from:
      _mjit_compile_invokebuiltin_for_rb_yjit_insns_compiled in yjit.o
      _Init_builtin_yjit.yjit_table in yjit.o
  "_rb_yjit_iseq_free", referenced from:
      _rb_iseq_free in iseq.o
  "_rb_yjit_iseq_gen_entry_point", referenced from:
      _rb_yjit_compile_iseq in yjit.o
  "_rb_yjit_iseq_mark", referenced from:
      _rb_iseq_mark in iseq.o
  "_rb_yjit_iseq_update_references", referenced from:
      _rb_iseq_update_references in iseq.o
  "_rb_yjit_parse_option", referenced from:
      _proc_options in ruby.o
  "_rb_yjit_reset_stats_bang", referenced from:
      _mjit_compile_invokebuiltin_for_rb_yjit_reset_stats_bang in yjit.o
      _Init_builtin_yjit.yjit_table in yjit.o
  "_rb_yjit_root_mark", referenced from:
      _yjit_root_type in yjit.o
  "_rb_yjit_simulate_oom_bang", referenced from:
      _mjit_compile_invokebuiltin_for_rb_yjit_simulate_oom_bang in yjit.o
      _Init_builtin_yjit.yjit_table in yjit.o
  "_rb_yjit_stats_enabled_p", referenced from:
      _mjit_compile_invokebuiltin_for_rb_yjit_stats_enabled_p in yjit.o
      _Init_builtin_yjit.yjit_table in yjit.o
  "_rb_yjit_trace_exit_locations_enabled_p", referenced from:
      _mjit_compile_invokebuiltin_for_rb_yjit_trace_exit_locations_enabled_p in yjit.o
      _Init_builtin_yjit.yjit_table in yjit.o
  "_rb_yjit_tracing_invalidate_all", referenced from:
      _tracepoint_enable_m in vm_trace.o
      _update_global_event_hook in vm_trace.o
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [miniruby] Error 1

In short, I cannot build 3.2.0-dev unless specifying --disable-yjit. Could you please look into it?

[maximecb (Maxime Chevalier-Boisvert)]

Hello @hachi8833 (Shozo Hatta).

It seems YJIT is failing to link because you have an x86-64 rustc toolchain installed on a Mac M1 arm64. We've made it so YJIT auto-enables when rustc is present, but we're not detecting that the right toolchain is installed. We're looking into a fix.

In the mean time, if you do want to compile YJIT, you could try installing/activating the aarch64 darwin toolchain on your machine, e.g.:

# List toolchains
rustup toolchain list

# Install Apple aarch64 toolchain
rustup toolchain install stable-aarch64-apple-darwin

# Use aarch64 toolchain
rustup override set stable-aarch64-apple-darwin

[alanwu (Alan Wu)]

• Status changed from Open to Closed

Applied in changeset git|a81c89b7c877cc0ddf40a8d4672ca90090e116c6.

---

YJIT: Make sure rustc's target matches before enabling (#6804)

For people using Rosetta 2 on ARM Macs, it can happen that the
rustc in the PATH compiles for x86_64 while clang is targeting
ARM. We were enabling YJIT in these situations because the test
program compiled fine, but caused linking failure later due to
the architecture mismatch.

Adjust the test program to fail when rustc's target arch is different
from ruby's target arch.

[Bug #19146]