Feature #22300: `Ractor.check_isolation`: report Ractor isolation violations as warnings instead of raising

• Status: Open
• Assignee: -
• Target version: 4.1

[ufuk (Ufuk Kayserilioglu)]

Summary

We are proposing a Ractor.check_isolation { ... } method which runs the given block in a special non-main Ractor.

Inside that Ractor, supported operations that would fail because of Ractor isolation report a :ractor_isolation category warning, and then continue, instead of raising an isolation error.

The special Ractor that is used to run the Ractor.check_isolation block is a real non-main Ractor. For example, Ractor.main? returns false and Ractor.current returns a genuine Ractor with its own default port. This is possible by ensuring that no other Ractor executes Ruby concurrently with this special Ractor. That exclusivity makes the "warn and continue" contract safe.

Examples

A trivial code example:

class Config
  @settings = {} # not shareable
  @values = [] # also not shareable

  def self.settings = @settings
  def self.values = @values
end

# Today: the first violation ends the run.
Ractor.new { Config.settings }.value
#=> Ractor::RemoteError
# whose cause is Ractor::IsolationError: can not
# get unshareable values from instance variables of
# classes/modules from non-main Ractors


# With `check_isolation`: the violation is a warning, and the run continues.
Ractor.check_isolation do
  Config.settings # warning, not exception
  Config.values # ...and the next one is also reported
  Ractor.main? #=> false
end

The next example is a test suite that can collect the supported isolation violations in a single run:

# test_helper.rb
ISOLATION_WARNINGS = Thread::Queue.new

Warning.singleton_class.prepend(Module.new do
  def warn(msg, category: nil)
    if category == :ractor_isolation &&
        !Thread.current[:capturing_isolation_warning]
      Thread.current[:capturing_isolation_warning] = true
      begin
        ISOLATION_WARNINGS << Ractor.make_shareable(msg)
      ensure
        Thread.current[:capturing_isolation_warning] = false
      end
      return
    end
    super
  end
end)

class ActiveSupport::TestCase
  def run(...)
    Ractor.check_isolation { super }
  end
end

1. Motivation

1.1 An isolation error stops the run at the first violation

Migrating a large codebase to Ractor-safety is difficult, since running it inside a real Ractor gives only one violation per run, which needs to be made Ractor-safe before you can observe the next violation. A large codebase has thousands, if not tens of thousands of these Ractor isolation violations that need to be identified.

Our team at Shopify started an effort to make Rails Ractor-safe earlier this year, and as part of that work we needed the full list of isolation problems and not just the first item. To tackle this problem, we used an agent assisted loop against a scaffolded Rails application which served 165 endpoints inside a Ractor. We directed the agent loop to identify and fix the source of the Ractor::IsolationError, and run the loop again, until all endpoints could successfully run.

While this approach did end up getting us a list of isolation violations, it proved to be an cumbersome task even with agent assistance. Instead, when we built the original prototype of Ractor.check_isolation, our initial run reported 50,995 warnings, of which 353 were unique in a matter of minutes with no agent assistance at all. A later run against Rails main reported 58,315 warnings, of which 419 were unique. The same work needs 419 build/fix/rebuild cycles (performed by agents or humans) to collect the same information.

1.2 You cannot put an existing test suite inside a real Ractor

The obvious alternative to the above is to put a Ractor.new { ... } around each test in the test suite of the codebase. This approach, while attractive, does not work on real codebases, for three main reasons:

1. The test code must itself be Ractor-safe. Making the tests safe is often harder than making the application safe. Most test helpers hold mutable global state on purpose.
2. The block of Ractor.new must be isolated. It cannot close over the test case, the fixtures or the assertion state.
3. Non-shareable arguments and return values are copied. However, copying is not possible for some objects, for example a Proc, and the run aborts.

Ractor.check_isolation removes all three obstacles. The block is not isolated. Arguments, the return value and outgoing message payloads pass by reference. Moreover, any isolation violation warning reported from the test file sources can easily be discarded as irrelevant.

1.3 There is no way to keep a codebase Ractor-safe

Given the above point, this is the argument that matters most in the long term, in our opinion.

A codebase that is Ractor-safe today will not stay Ractor-safe in the future. As the codebase evolves, one new lazy memoization, or one new unfrozen constant in a dependency, will reintroduce a violation. A normal test suite cannot detect the change, because the test suite does not run in a Ractor (as explain in the previous point). A gem can therefore break its downstream users without any signal, and the failure would only appear in production.

Ractor.check_isolation turns Ractor safety into a property that CI can measure. A gem or an application can wrap its tests in the block and fails the build when the count of unique warnings increases.

In essence, our argument is that this proposal gives the ecosystem two things it does not have today:

• a burn-down list, to make a codebase Ractor-safe;
• a ratchet, to keep it Ractor-safe.

2. Proposal

Ractor.check_isolation(*args, name: nil) { |*args| ... } # -> the value of the block

Ractor.check_isolation method:

• runs the block in a new, real, and special non-main Ractor;
• does not isolate the block, and passes arguments by reference;
• reports each supported isolation violation as a :ractor_isolation category warning, and continues without raising;
• unlike an ordinary Ractor, returns a non-shareable block value without copying it, preserving object identity.

The reported violations match the constructs that would have raised a Ractor::IsolationError under a normal non-main Ractor. They include, but are not limited to:

• instance variables and class variables of classes and modules;
• constants that hold non-shareable objects;
• global variables;
• Ractor.make_shareable, Ractor.shareable_proc and
  Ractor.shareable_lambda failures;
• calls to Ractor-unsafe C methods;
• fork from a non-main Ractor;
• defining or undefining finalizers on objects owned by another Ractor;
• a call to a method that define_method defined with a non-shareable Proc;
• isolation of the check block when it captures outer variables or uses yield.

The user suppresses isolation-violation warnings with

Warning[:ractor_isolation] = false or with -W:no-ractor_isolation.

The exclusivity of the special Ractor that run the block is a precondition of the method, not a mode that the user should select. The current patch relies on RUBY_RACTOR_EXCLUSIVE=1 and warns when it is unavailable. However, for the final solution we propose that Ractor.check_isolation instead establish exclusivity itself and raise when it cannot. Section 4.3 discusses the implementation constraints regarding this.

The exclusive mode mentioned here is not a stop-the-world barrier. A blocking operation still hands the run slot over, so that, for example, work that the block correctly dispatches to the main Ractor completes, and the program does not deadlock.

3. Implementation

Our current patch is on this branch: https://github.com/Shopify/ruby/compare/master...Shopify:ruby:ractor-check-isolation

The branch exposes the design described here under the name Ractor.check_isolation.

Main points:

• A new warning category, RB_WARN_CATEGORY_RACTOR_ISOLATION, registered as :ractor_isolation.
• A flag on the special rb_ractor_t. A helper, rb_ractor_isolation_violation(), replaces the raise at many of the covered violation sites. The helper warns when the current Ractor has the flag set, and raises otherwise.
• ractor_create0() creates and marks the special Ractor. When rb_thread_create_ractor() starts it, thread_create_core() skips rb_proc_isolate_bang() and retains the argument array by reference. rb_proc_check_isolation_warn() reports the Proc-isolation violations that Ractor.new would raise, and leaves the closure intact.
• ractor_prepare_payload() passes a non-shareable message payload by reference, with a warning, instead of copying it.
• The bmethod call paths warn instead of raising for a cross-Ractor call to a method defined with a non-shareable Proc.
• On builds with M:N scheduling, ruby_mn_threads_params() enables it and sets max_cpu = 1 under RUBY_RACTOR_EXCLUSIVE=1.
• The first call switches the VM into multi-ractor mode. This switch is one-way today.

Two costs with the current patch are worth stating explicitly:

1. The VM keeps the multi-ractor overhead for the rest of the process, and
2. the checks sit on paths that a production process executes.

We consider these costs acceptable since this feature is a development and CI tool, and should not be enabled in production mode.

4. Open questions

We have used this feature for four months on Rails and on two large production applications at Shopify. The following questions come out of that use. We do not have a strong position on most of them.

1. Is warn the right surface?: Warning.warn was the cheapest thing to build, but it may be the weakest part of the design, for the following reasons:

   • The volume is high. In one Shopify application, the run produced more warning text than the 6 GB memory limit of the CI worker, so it was killed. The consumers want a deduplication by site, not a stream of strings.
   • A string throws the structure away. The consumers want the class, the variable name, the reference chain and the location as data, rather than parsing whatever information is available from the message.
   • Re-entrancy is a real hazard. Any code in the handler can itself trigger a violation. We hit an infinite recursion this way, and we also hit a stack overflow because formatting the message called to_s on a proxy module.
   • Warning.warn is global process state. A warning that the special Ractor reports is awkward to collect in the caller.
     The alternatives are a callback, or a reporter object:

   Ractor.check_isolation(on_violation: ->(violation) { ... }) { ... }
   Ractor.check_isolation(reporter: MyReporter.new) { ... }
   

   A violation object can carry the class, the name, the kind, the reference chain and the backtrace. The warning category can stay as the default when the caller gives no callback.

2. Should the checks be a build-time option? The checks add work to hot paths. An alternative is to compile them only into a build that configures them, in the same way as RGENGC_CHECK_MODE. This trades convenience for speed. We do not know which side upstream prefers, but considering that so few Ruby developers know how to build a special version of Ruby, we think this will stop the widespread adoption of the feature.

3. How does the method establish exclusivity? The current patch reads a boot flag, RUBY_RACTOR_EXCLUSIVE=1, and warns once when the flag is absent. We do not propose that as the interface for the final state.

   The constraint for why we have the boot flag is technical and we welcome suggestions to removing it. Two mechanisms in thread_sched.c decide the outcome:

   • vm->ractor.sched.max_cpu caps the number of shared native threads. The VM reads it when an NT rejoins the pool, so lowering it at runtime does not retire the NTs that already exist. They retire lazily, and SNT_KEEP_MINIMUM keeps some of them alive.
   • enable_mn_threads decides how each thread gets a native thread, and the VM reads it when the thread is created. A thread created while the flag was off holds a dedicated native thread. A dedicated NT is its own OS thread and ignores max_cpu completely.

   Together these mean that a switch at the time of the call cannot cover the threads that the process already has, but it can cover a process with no other living threads.

   We therefore propose:

   • Ractor.check_isolation switches the VM to a single run slot when it is the only living thread in the process, and runs the block;
   • otherwise it raises, and names the threads that prevent the guarantee.

   This removes the flag from the common case, and it turns a silent unsound result into an error.

   Ideally, we would like to find a solution where we can create a special exclusive Ractor regardless of the state of the VM, but need more guidance on how to achieve that.

4. How does this interact with Feature #22226? Class and module ownership changes which accesses are violations. The checker must follow the same rule, and we are prepared to improve the proposal to obey this invariant as well. Moreove, the special Ractor should also behave like how a real Ractor would with respect to the ownership of the classes/modules that the block creates. Otherwise, it would report fewer violations than a production worker would hit.

5. Is passing by reference acceptable in a public API? The block, the arguments, the return value and outgoing message payloads all bypass the isolation rules. This is sound because the special Ractor runs exclusively, so it is not a safety concern.

6. What are the semantics for nested calls and for threads? In our current patch, a nested call creates another special Ractor and resumes the calling Ractor when it returns. A thread that the block creates belongs to the special Ractor and therefore observes the check and only raises warnings. On the other hand, a Ractor created with Ractor.new does not inherit the check, and raises Ractor::IsolationError for isolation errors. We think this behaviour is right, but it should be specified.

7. Should the VM deduplicate? Most consumers want unique violations per source location, and they build that themselves. A Set of sites in the VM would cut the volume by two orders of magnitude. On the other hand, it might also hide a violation that occurs on a second path through the same line, and would make it hard to collect how many times each unique violation is seen.

5. Related issues

• #22226: Ractor: class/module ownership. Changes which accesses are violations.
• #22252: improve the "defined with an un-shareable Proc in a different Ractor" error. This feature downgrades the same check to a warning.
• ruby/ruby#15983: the reference chain in Ractor::IsolationError#detailed_message. Adding the same chain to the warnings would make a violation more actionable.

6. Acknowledgements

Different parts of this patch were built by Ufuk Kayserilioglu, Edouard Chin and Hartley McGuire, using LLM assistance. @ko1 (Koichi Sasada) provided guidance on our original approach which was using a special Thread instead of a special Ractor.