Misc #22275: Ruby::Box support plan for RubyGems and Bundler

• Status: Open
• Assignee: -

[hsbt (Hiroshi SHIBATA)]

This is a tracking issue for making RubyGems and Bundler work with Ruby::Box, so that reviewers can see the whole picture and what to look at next in one place.

Goal

The goal is to realize Feature #13847 with Ruby::Box, which means loading a specific version of a gem, including default gems and C extensions, isolated from the copy that RubyGems and Bundler themselves use. Vendoring with namespace rewriting has been our workaround for pure-Ruby gems for years. Box removes that limitation.

As a further step, we want RubyGems and Bundler to activate different gem versions per box. That enables the use cases already proposed around Box, such as gradual dependency upgrades, plugin systems with conflicting dependencies, and multiple applications in one process.

First milestone

Before designing such APIs, Box needs to leave experimental status. Our first milestone is that Rails and RubyGems/Bundler themselves work under RUBY_BOX=1, and that the ruby/rubygems repository runs its test suites with Box enabled continuously so we do not regress.

---

Current status

ruby/rubygems now runs both suites against a ruby-core master build with and without RUBY_BOX=1 on every pull request and push (ruby/rubygems#9826), and all four jobs pass. That covers the continuous part of the first milestone for ruby/rubygems.

With the stdio, $? and defined? fixes merged, ruby/rubygems#9892 removed 28 pending marks from the rubygems suite and 3 skips from the bundler suite. The rubygems suite now runs 3619 tests with 0 failures in both modes, with 10 pendings under Box and 8 without. The two box-only pendings wait on $VERBOSE, and the remaining bundler skip waits on -r, both below.

ruby/ruby's own make check passes under RUBY_BOX=1 on ruby/ruby#18705, which collects every open fix below and adds a CI lane for it. The branch is for measuring the integrated state, not for merging as it is. Decisions 3 to 7 came out of that work.

Critical problems

• ruby/ruby#18579 (Bug #22282) is the last fix the rubygems suite waits on: under Box, $VERBOSE = nil does not reach the interpreter. @tagomoris (Satoshi Tagomori) asked on the pull request whether $VERBOSE can stay per box instead, which is decision 3.
• Bug #22295 (decision 2) has no fix yet. The bundler install specs skip it, and the cargo builder test works around it by requiring inside -e.

Decisions needed

These block the rest and need a direction rather than more review.

1. Which box a builtin written in Ruby resolves its calls in. It is the master box by default, which matches not doing local rebinding (Bug #21362#note-3). ruby/ruby#18544 settled Marshal.load by opting it into the caller's box with Primitive.attr! :caller_user_box. What remains is which other builtins opt in the same way, such as Ractor.new (ruby/ruby#18821), Kernel#clone, Kernel#warn and prelude (Bug #22332), and whether core-class stubs should reach builtins (Misc #22296). Opting every builtin in by default brings the inline cache leak of Bug #21362 back, because all boxes share a builtin's call sites. On the ruby/ruby#18705 branch, which tries that, redefining Integer#succ in one box changes 10.times.to_a in the main box.
2. Whether command-line -r activates gems, and which boxes it reaches (Bug #22295). Under Box, -r and RUBYOPT=-r go through Ruby::Box#require without the RubyGems override, so a regular gem cannot be loaded that way, and a test harness that injects a gem with RUBYOPT=-r fails. The feature also stays in the main box, so a box created later does not see it. Item 3 of Bug #21760 reports the same gap in Ruby::Box#require itself.
3. What C-backed global variables mean in an optional box (Bug #22307). Assignments to $/, $stdout and the like stay in a box-local table that C never reads. ruby/ruby#18711 makes the main box behave like Ruby without boxes and leaves optional boxes open. $VERBOSE in ruby/ruby#18579 is the same question for one variable.
4. How prism ships (Bug #22305). The statically linked prism loads in only one box, so a gem that requires it, such as error_highlight, fails in every box but the first.
5. How JIT hooks reach boxes (Bug #22306). Enabling YJIT switches Array#each and a few other methods to Ruby only in the master box, so the root and main boxes never get them.
6. What a box's top self should be (Bug #22331). Only the master box's top self has include, using, private and the other top-level definition methods, so Ruby::Box#load and load(file, true) fail.
7. Where prelude is evaluated (Bug #22332). It runs once in the master box, so pp and binding.irb require into it.

---

Open pull requests

ruby/ruby

• ruby/ruby#18579: assignments to $VERBOSE and $DEBUG have no effect (Bug #22282). Waiting on decision 3.
• ruby/ruby#18711: C-backed global variables in the main box (Bug #22307). It depends on ruby/ruby#18704, which fixes alias $new $old in a box.
• ruby/ruby#18708: trace_var hooks do not run for assignments in a box.
• ruby/ruby#18707: the top self of a box lacks the top-level definition methods (Bug #22331).
• ruby/ruby#18713: pp and binding.irb require into the master box (Bug #22332). Feature #21881 proposed loading prelude.rb per box, and this does it.
• ruby/ruby#18821: Ractor.new resolves the classes of its arguments in the master box, so passing an instance of a class required in the main box raises undefined class/module. It touches the Ractor and Box integration that Feature #22226#note-6 left to be decided with the Box author.
• ruby/ruby#18842: a box classext is attached to its class only after it is filled in, so a GC in between frees what it holds.
• ruby/ruby#18710: RUBY_FREE_AT_EXIT=1 hangs at exit on macOS and crashes on Linux.
• ruby/ruby#18701: a frozen $LOADED_FEATURES raises FrozenError while the feature index is rebuilt. This is not specific to Box, but every box hits it on its first require.

Fixed

ruby/ruby

• ruby/ruby#18544: Marshal.load resolved classes outside the caller's box (Bug #22090). This was the last item on the Bundler critical path.
• ruby/ruby#18574: reassigning $stdout and $stderr was invisible to builtin writers (Bug #21867), which broke output-capturing test helpers everywhere.
• ruby/ruby#18577: $? was uninitialized after Kernel#system and IO.popen (Bug #22280).
• ruby/ruby#18586: defined? did not see global variables assigned in a box (Bug #22283), so mkmf have_devel? recursed until the stack was exhausted.
• ruby/ruby#18218 / ruby/ruby#18219: BUNDLER_SETUP was consumed outside the main box (Bug #22123). Shipped in 4.0.7.
• ruby/ruby#18509: box-local extension DLLs on Windows were unloaded too early. They are now deferred.
• ruby/ruby#18534: autoload-triggered require bypassed the box's Kernel#require (Bug #21830). This alone took Rails from 500 on every request to fully working.
• ruby/ruby#18535: with --disable=gems, modules prepended to Kernel in a user box ended up behind Kernel in the ancestry (Bug #22270).
• ruby/ruby#18536: the box extension copy embedded the full path in the temporary filename, exceeding NAME_MAX on deep paths, and the name was predictable (Bug #22110, Bug #22271).
• ruby/ruby#18578: the process-private directory added by the previous fix did not survive fork. A forked child removed it at exit (Bug #22271).
• ruby/ruby#18575: box resolution crashed on an IFUNC frame (Bug #21977). Without this the bundler suite died with [BUG] and the dead workers cascaded into unrelated failures.
• ruby/ruby#18546: Symbol#to_proc ignored box-local method definitions (Bug #22015, revived ruby/ruby#16865). Without this Bundler's spec harness died in rspec-core's &:shellsplit before running a single spec.
• ruby/ruby#18475: an isolated Proc made in a class or module body lost its box and crashed (Bug #22260).
• ruby/ruby#18698, ruby/ruby#18702, ruby/ruby#18703, ruby/ruby#18706 and ruby/ruby#18712: test and harness fixes so that make check can run under RUBY_BOX=1.

ruby/rubygems

• ruby/rubygems#9892: re-enabled the stdio capture tests under Box once Bug #21867 was fixed.
• ruby/rubygems#9826: made the RubyGems and Bundler suites green under RUBY_BOX=1 and added the Box lanes to CI.
• ruby/rubygems#9809: Bundler evaluated gemspecs through TOPLEVEL_BINDING, which always belongs to the main box. It now uses a binding in the box Bundler is loaded in, so gemspecs resolve the right Gem::Specification.
• ruby/rubygems#9810: the gem CLI died under RUBY_BOX=1. Marshal-based deep copies could not resolve Gem:: constants across boxes and were replaced with a plain deep dup, and RUBY_BOX is now stripped from extension build subprocesses, where mkmf have_devel? recurses until SystemStackError (Bug #22283). It includes a CLI canary test that runs under RUBY_BOX=1.

[hsbt (Hiroshi SHIBATA)]

• Description updated (diff)

[hsbt (Hiroshi SHIBATA)]

• Related to Feature #13847: Gem activated problem for default gems added

[hsbt (Hiroshi SHIBATA)]

• Related to Bug #22123: Ruby::Box + `BUNDLER_SETUP` can evaluate gemspecs before main-box RubyGems initialization added

[hsbt (Hiroshi SHIBATA)]

• Related to Bug #21830: Ruby::Box and Kernel#require added

[hsbt (Hiroshi SHIBATA)]

• Related to Bug #22015: Success without RUBY_BOX=1, Failure with RUBY_BOX=1 added

[hsbt (Hiroshi SHIBATA)]

• Related to Bug #22090: Enabling Ruby::Box breaks Marshal.load added

[hsbt (Hiroshi SHIBATA)]

• Related to Bug #22110: Ruby::Box C-extension loading can be denied by predictable temporary filename added

[hsbt (Hiroshi SHIBATA)]

• Related to Bug #21867: enabling Ruby::Box changes puts/warn to no longer use $stdout/$stderr added

[hsbt (Hiroshi SHIBATA)]

• Related to Bug #22271: Ruby::Box fails to load C extensions from deep paths (temporary file name exceeds NAME_MAX) added

[Eregon (Benoit Daloze)]

(for some reason this didn't get sent to the ruby-core mailing list. Other recent Misc tickets didn't have the issue, so unclear what happened)

[hsbt (Hiroshi SHIBATA)]

• Description updated (diff)

[hsbt (Hiroshi SHIBATA)]

Status update. ruby/ruby#18534, ruby/ruby#18535, ruby/ruby#18536 and ruby/ruby#18578 have been merged since this issue was filed, and three more core bugs were found and reported along the way (Bug #22271, Bug #22280, Bug #22282).

The Bundler spec suite now runs under RUBY_BOX=1 at parity with a non-box run: 3957 examples, with the same single failure in both modes, which is an unrelated upstream problem in a Rust extension test. The rubygems suite stays at 0 failures. Six core pull requests are waiting for review, and ruby/ruby#18546, ruby/ruby#18544 and ruby/ruby#18575 are the ones that gate the suite.

The next milestone is to land those three, add a RUBY_BOX=1 lane to the ruby/rubygems CI so we stop regressing, and drop the pending marks as the rest land. After that we can discuss what is left before Box can lose the experimental label.

[hsbt (Hiroshi SHIBATA)]

• Related to Bug #22282: $VERBOSE and $DEBUG assignments have no effect under RUBY_BOX=1 added

[hsbt (Hiroshi SHIBATA)]

• Related to Bug #21977: Ruby::Box crash with `RUBY_BOX=1` via `Binding` refinement, `UnboundMethod#bind_call`, and `Symbol#to_proc` added

[hsbt (Hiroshi SHIBATA)]

• Related to Bug #22280: Ruby::Box breaks $? after Kernel#system / IO.popen added

[hsbt (Hiroshi SHIBATA)]

• Related to Bug #21760: Ruby::Box: a couple of require-related problems added

[hsbt (Hiroshi SHIBATA)]

• Description updated (diff)

[hsbt (Hiroshi SHIBATA)]

• Related to Bug #22283: `defined?` does not see global variables assigned in a box added

[hsbt (Hiroshi SHIBATA)]

• Related to Bug #22295: `ruby -r<gem>` does not activate gems under RUBY_BOX=1 added

[hsbt (Hiroshi SHIBATA)]

• Description updated (diff)

[hsbt (Hiroshi SHIBATA)]

• Related to Misc #22296: Ruby::Box: method stubs on core classes are invisible to internal builtins added

[hsbt (Hiroshi SHIBATA)]

• Description updated (diff)

[hsbt (Hiroshi SHIBATA)]

• Related to Bug #22305: Ruby::Box: statically linked prism loads in only one box added

[hsbt (Hiroshi SHIBATA)]

• Related to Bug #22306: Ruby::Box: with_jit hooks do not reach existing boxes added

[hsbt (Hiroshi SHIBATA)]

@Eregon (Benoit Daloze) @shugo (Shugo Maeda) reported the same problem, and it turned out to be a bug in the MCP server on this Redmine. Issues and notes written through it were never sent to ruby-core. It is fixed now, and both the web UI and MCP send mail to the list again.

[hsbt (Hiroshi SHIBATA)]

• Related to Bug #22307: Ruby::Box: assignments to C-backed global variables do not reach C added

[hsbt (Hiroshi SHIBATA)]

• Description updated (diff)

[hsbt (Hiroshi SHIBATA)]

• Related to Bug #22331: Ruby::Box: a box's top self lacks the top-level definition methods added

[hsbt (Hiroshi SHIBATA)]

• Related to Bug #22332: Ruby::Box: prelude is evaluated only in the master box, so pp and binding.irb load into it added

[hsbt (Hiroshi SHIBATA)]

• Description updated (diff)

[hsbt (Hiroshi SHIBATA)]

• Related to Bug #22380: Ruby::Box: a GC while copying a class into a box crashes added

[hsbt (Hiroshi SHIBATA)]

I merged the following four, since they fix plain bugs such as crashes and need no design decision.

• https://github.com/ruby/ruby/pull/18704: alias $new $old in a box created a separate variable.
• https://github.com/ruby/ruby/pull/18708: trace_var hooks did not run for assignments in a box.
• https://github.com/ruby/ruby/pull/18710: RUBY_FREE_AT_EXIT=1 hung or crashed at exit (#22339, 4.0 backport requested).
• https://github.com/ruby/ruby/pull/18842: a GC while copying a class into a box crashed (#22380, 4.0 backport requested).

[vo.x (Vit Ondruch)]

I think that RubyGems / Bundler should not be loaded into Ruby::Box by default. I have shared a few of my thoughts with @tagomoris (Satoshi Tagomori) and based on that he implemented #22329.

But I'd like to go further. I think that it would be nice to have RubyGems / Bundler loaded only into the root Ruby::Box (and sorry if I get the naming wrong). Now if I simplify things, bundle exec just resolves gems, populates RUBYLIB with the gem paths and actually removes RubyGems from further action. So to emulate this, it would be nice to do something like Ruby::Box.new(load_path: Bundle.resolve(path_to_gemfile)).

Alternatively, the RubyGems / Bunlder could be loaded into some isolated Ruby::Box.

All I want to say that I think that loading RubyGems / Bundler by default into Ruby::Box would be missed opportunity IMHO.