Feature #21550: Ractor.sharable_proc/sharable_lambda to make sharable Proc object

• Status: Open
• Assignee: ko1 (Koichi Sasada)
• Target version: 3.5

[ko1 (Koichi Sasada)]

Let's introduce a way to make a sharable Proc.

• Ractor.shareable_proc(self: nil, &block) makes proc.
• Ractor.shareable_lambda(self: nil, &block) makes lambda.

See also: https://bugs.ruby-lang.org/issues/21039

Background

Motivation

Being able to create a shareable Proc is important for Ractors. For example, we often want to send a task to another Ractor:

worker = Ractor.new do
  while task = Ractor.receive
    task.call(...)
  end
end

task = (sharable_proc)
worker << task

task = (sharable_proc)
worker << task

task = (sharable_proc)
worker << task

There are various ways to represent a task, but using a Proc is straightforward.

However, to make a Proc shareable today, self must also be shareable, which leads to patterns like:

  nil.instance_eval{ Proc.new{ ... } }

This is noisy and cryptic. We propose dedicated methods to create shareable Proc objects directly.

Specification

• Ractor.shareable_proc(self: nil, &block) makes a proc.
• Ractor.shareable_lambda(self: nil, &block) makes a lambda.

Both methods create the Proc/lambda with the given self and make the resulting object shareable.

Captured outer variables follow the current Ractor.make_shareable semantics:

• If a captured outer local variable refers to a shareable object, a shareable Proc may read it.
• If any captured outer variable refers to a non‑shareable object, creating the shareable Proc raises an error.

a = 42
b = "str"
Ractor.sharalbe_proc{
  p a #=> 42
}

Ractor.sharalbe_proc{ # error when making a sharealbe proc
  p b #=> 42
}

• The captured outer local variables are copied by value when the shareable Proc is created. Subsequent modifications of those variables in the creator scope do not affect the Proc.

a = 42
shpr = Ractor.sharable_proc{
  p a
}
a = 0
shpr.call #=> 42

• Assigning to outer local variables from within the shareable Proc is not allowed (error at creation).

a = 42
Ractor.shareable_proc{ # error when making a sharealbe proc
  a = 43
}

More about outer-variable handling are discussed below.

In other words, from the perspective of a shareable Proc, captured outer locals are read‑only constants.

This proposal does not change the semantics of Ractor.make_shareable() itself.

Discussion about outer local variables

[Feature #21039] discusses how captured variables should be handled.
I propose two options.

1. No problem to change the outer-variable semantics

@Eregon (Benoit Daloze) noted that the current behavior of Ractor.make_shareable(proc_obj) can surprise users. While that is understandable, Ruby already has similar surprises.

For instance:

RSpec.describe 'foo' do
  p self #=> RSpec::ExampleGroups::Foo
end

Here, self is implicitly replaced, likely via instance_exec.
This can be surprising if one does not know self can change, yet it is accepted in Ruby.
We view the current situation as a similar kind of surprise.

2. Enforce a strict rule for non‑lexical usage

The difficulty is that it is hard to know which block will become shareable unless it is lexically usage.

# (1) On this code, it is clear that the block will be shareable block:

a = 42
Ractor.sharable_proc{
  p a
}

# (2) On this code, it is not clear that the block becomes sharable or not
get path do
  p a
end

# (3) On this code, it has no problem because
get '/hello' do
  "world"
end

The idea is to allow accessing captured outer variables only for lexically explicit uses of Ractor.shareable_proc as in (1), and to raise an error for non‑lexical cases as in (2).
So the example (3) is allowed if the block becomes sharable or not.

The strict rule is same as Ractor.new block rule.

[ko1 (Koichi Sasada)]

• Description updated (diff)

[ko1 (Koichi Sasada)]

• Description updated (diff)

[ko1 (Koichi Sasada)]

• Related to Feature #21039: Ractor.make_shareable breaks block semantics (seeing updated captured variables) of existing blocks added

[ko1 (Koichi Sasada)]

• Description updated (diff)

[matz (Yukihiro Matsumoto)]

I understand the concern for future confusion, but it's a trade-off. I'd accept confusion here (option 1) to avoid complex semantics and implementation.

Matz.

[Eregon (Benoit Daloze)]

Summary: I think option 2 is great it's both flexible, clear and safe, and we should implement it.
It addresses the concerns in #21039.
It should be easy to implement, because if given a Proc it's the same semantics as Ractor.new(proc_object), and when given a literal block it's the desired more flexible semantics which anyway both options want.

What follow is some more thoughts, which might lead to some improvements later but anyway I believe it's fine to implement option 2 as-is.

---

I wonder if the ability to capture outer local variables for shareable procs is even needed at all.
Because that only works if the value of the captured variables is shareable, which seems not so likely (only numbers/true/false/nil/frozen-string-literals/modules/regexps are frozen/shareable without an explicit freeze/make_shareable call).
I would like to see some concrete examples of blocks to be made shareable which capture outer local variables, and how they handle the case of captured variable values not already being deeply frozen (cc @tenderlovemaking (Aaron Patterson) you said there were some in Rails IIRC).

Related to that, maybe it would be useful for

o = Object.new
Ractor.shareable_proc {
  o
}

to call Ractor.make_shareable(o)?
Or maybe Ractor.make_shareable(o, copy: true) on Ractor.shareable_proc(copy: true) { ... }?
Though not sure if reasonable or too surprising. It might be surprising for cases where it freezes many objects, e.g. if o is Foo.new([]) the Array also gets frozen (which might break things).
I think that case makes it clear this might maybe be acceptable with a literal block, but would never be acceptable with a Proc object (it would freeze things far away and be hard to debug).

In some way we can see a block as an object with an extra @ivar being the "captured environment" which contains the captured variables.
From that POV, making the Proc shareable then would rather naturally make that "captured environment" and objects inside shareable as well.

---

I find it interesting that we all seem to agree on Assigning to outer local variables from within the shareable Proc is not allowed (error at creation) but not on Assigning to outer local variables from outside the shareable Proc is not allowed through an exception when trying to create a shareable Proc in such a case.
IMO they are just both sides of the same issue.
But I understand the second one is more tricky because e.g. it's impossible to detect assignments within eval at (file) parse time (IMO so rare of an edge case to not worry about that too much).
I also agree the second one is more rarely used, though that doesn't mean it doesn't matter.

---

With option 1 it would break existing blocks, so I am against it, e.g.:

visits = 0
get '/' do # assuming this block is not made shareable (because that is forbidden, see `Assigning to outer local variables from within the shareable Proc is not allowed (error at creation)`)
  visits += 1
  "Hello"
end
get '/visits' do # if this block is made shareable and called, it will behave incorrectly and use a snapshot of `visits`
  visits.to_s
end

Concretely, assuming the second block is made shareable at load time, if the block is used only in non-main Ractors but not in main Ractors:

• if the request hits the main Ractor it would be correct (the number of visits of /)
• if it hits a non-main Ractor it would be incorrect (0).

That's pretty bad for obvious reasons.
If the shared block is instead used by all Ractors, including the main one then it would always be incorrect (0). Still clearly breaking the program/intended semantics.

So standard Ruby code, specifically blocks (and their intended logic) would be broken "just because a Proc has been made Ractor-shareable".

This is worse than just instance_exec, because a block called with instance_exec is typically always called with the same kind of receiver.
And the worse case with instance_exec is a NoMethodError or calling the wrong method (very unlikely).
The worse case with option 1 is reading a stale/outdated/inconsistent value, very much like a stale read which is usually a race condition/multithreading problem, but here Ractor would actually introduce this issue even though Ractor is meant to prevent such issues.