Misc #22275
open
Ruby::Box support plan for RubyGems and Bundler
Description
This is a tracking issue for making RubyGems and Bundler work with Ruby::Box, so that reviewers can see the whole picture and what to look at next in one place.
Goal¶
The goal is to realize Feature #13847 with Ruby::Box: loading a specific version of a gem, including default gems and C extensions, isolated from the copy that RubyGems/Bundler themselves use. Vendoring with namespace rewriting has been our workaround for pure-Ruby gems for years; Box removes that limitation.
As a further step, we want RubyGems/Bundler to activate different gem versions per box, which enables the use cases already proposed around Box: gradual dependency upgrades, plugin systems with conflicting dependencies, and multiple applications in one process.
First milestone¶
Before designing such APIs, Box needs to leave experimental status. Our first milestone is that Rails and RubyGems/Bundler themselves work under RUBY_BOX=1, and that the ruby/rubygems repository runs its test suites with Box enabled continuously so we do not regress.
Current status¶
Both suites now run under RUBY_BOX=1 with the pull requests below applied. The rubygems suite is at 0 failures (ruby/rubygems#9826, open). The bundler suite runs 3957 examples with box and non-box at parity; the single remaining failure is an unrelated upstream problem in a Rust extension test. Pending marks reference the unfixed core bugs below.
Fixed¶
ruby/ruby¶
- ruby/ruby#18218 / ruby/ruby#18219 —
BUNDLER_SETUPwas consumed outside the main box (Bug #22123). Shipped in 4.0.7. - ruby/ruby#18509 — box-local extension DLLs on Windows were unloaded too early; now deferred.
- ruby/ruby#18534 — autoload-triggered require bypassed the box's
Kernel#require(Bug #21830). This alone took Rails from 500 on every request to fully working. - ruby/ruby#18535 — with
--disable=gems, modules prepended toKernelin a user box ended up behindKernelin the ancestry (Bug #22270). - ruby/ruby#18536 — the box extension copy embedded the full path in the temporary filename, exceeding NAME_MAX on deep paths, and the name was predictable (Bug #22110, Bug #22271).
- ruby/ruby#18578 — the process-private directory added by the previous fix did not survive
fork; a forked child removed it at exit (Bug #22271).
ruby/rubygems¶
- ruby/rubygems#9809 — Bundler evaluated gemspecs through
TOPLEVEL_BINDING, which always belongs to the main box. It now uses a binding in the box Bundler is loaded in, so gemspecs resolve the rightGem::Specification. - ruby/rubygems#9810 — the
gemCLI died underRUBY_BOX=1.Marshal-based deep copies could not resolveGem::constants across boxes and were replaced with a plain deep dup, andRUBY_BOXis now stripped from extension build subprocesses, where mkmfhave_devel?recurses untilSystemStackError(Bug #22283). Includes a CLI canary test that runs underRUBY_BOX=1.
Open pull requests¶
All review-requested to @tagomoris (Satoshi Tagomori):
- ruby/ruby#18546 —
Symbol#to_procignores box-local method definitions (Bug #22015, revives ruby/ruby#16865). This alone unblocks Bundler's spec harness, which dies in rspec-core's&:shellsplitbefore running a single spec. - ruby/ruby#18544 —
Marshal.loadresolves classes in the root box (Bug #22090); an in-process round-trip ofGem::Versionfails today. - ruby/ruby#18575 — box resolution crashes on an IFUNC frame (Bug #21977).
- ruby/ruby#18574 — reassigning
$stdout/$stderris invisible to builtin writers (Bug #21867), which breaks output-capturing test helpers everywhere. - ruby/ruby#18577 —
$?is uninitialized afterKernel#systemandIO.popen(Bug #22280). - ruby/ruby#18579 — assignments to
$VERBOSEand$DEBUGhave no effect (Bug #22282). - ruby/ruby#18586 —
defined?does not see global variables assigned in a box (Bug #22283). This is why mkmfhave_devel?never memoizes and recurses until the stack is exhausted, which is the "stack level too deep in extconf.rb" entry under Known issues indoc/language/box.md.
Remaining problems without a fix¶
ruby -r<gem>andRUBYOPT=-r<gem>do not activate gems under Box.require_libraries_in_main_box()callsrb_require_string()directly, bypassing the RubyGemsKernel#requireoverride. The closest existing report is item 3 of Bug #21760.- Stubbing a core class does not reach code already loaded in the root box, so existing test suites change behaviour under Box without failing loudly.
allow(File).to receive(:expand_path)in the main box leavesPathname#expand_pathcalling the real method, and the suite then takes a different path than it does without Box. We hit this in the Bundler suite and scoped the stub as a workaround, but existing test code cannot be expected to know the rule. If this is intended box semantics, it should at least be documented as a known limitation, because rspec-mocks style stubbing of core classes is everywhere.
Critical path¶
The most critical items are ruby/ruby#18546, ruby/ruby#18544 and ruby/ruby#18575. The first gates Bundler's entire spec suite, the second makes the artifice-based install specs fail wholesale, and without the third the suite dies with [BUG] and the dead workers cascade into unrelated failures. With those three the bundler suite runs end to end and the rest can be marked pending. ruby/ruby#18574 and ruby/ruby#18577 then let us drop most of the pending marks.
Updated by 
Updated by