Feature #18275: Add an option to define_method to not capture the surrounding environment

• Status: Assigned
• Assignee: ko1 (Koichi Sasada)
• Target version: -

[vinistock (Vinicius Stock)]

Invoking define_method will capture the surrounding environment, making sure we have access to anything defined in that surrounding scope. However, that’s not always necessary. There are uses for define_method where the surrounding environment is not needed.

Always capturing the surrounding environment slows down even the methods that don’t need access to it. Additionally, it prevents methods created using define_method to exist in all Ractors in a program.

If we could add an option to disable capturing the surrounding environment for define_method, we could make it so that it creates the dynamic method in all Ractors.

There could also be some performance benefits for the usages that do not need the surrounding environment. By not having to keep references to the surrounding scope, the GC could let go of locals from that environment, which might benefit GC as well.

Another option could be to accept the list of locals that the define_method invocation will need, as a way of letting go of references that are no longer needed.

Examples:

# Current behavior
#
# All of the surrounding environment is captured and references are kept for the locals
# The method created only exists in the current Ractor, due to possible references to the captured variables

some_random_thing = "a" * 10000
some_captured_block = -> { ... }

define_method(:my_method, &some_captured_block)

# Enable/disable all option
#
# Add an option that allows disabling capturing the surrounding environment completely
# The method created exists in all Ractors and none of the references are kept

some_random_thing = "a" * 10000
some_captured_block = -> { ... }

define_method(:my_method, capture_environment: false, &some_captured_block)

# Choose variables option
#
# Add an option that allows indicating which locals are needed for a define_method invocation
# The method created exists in all Ractors if no locals are needed
# The method is created only in the current Ractor if at least one local is needed
# All “unneeded” locals are let go

some_random_thing = "a" * 10000 # kept because `my_method` needs it
another_random_thing = "b" * 10000 # not kept
some_captured_block = -> { ... }

define_method(:my_method, needs: [:some_random_thing], &some_captured_block)

[shyouhei (Shyouhei Urabe)]

Can you tell us why you cannot define your methods using normal def statements? As far as I read your proposal there seems no need of define_method to creep in.

[Eregon (Benoit Daloze)]

In general if define_method takes an existing Proc, it should IMHO behave exactly like calling that Proc.
If we do anything special, every call like that to define_method would become very expensive as we'd need to e.g. change bytecodes of that Proc, and suddenly we'd have different behavior (e.g., regarding assignment, eval inside that Proc, etc).

There is already a way to do this via define_method(:name, &Ractor.make_shareable(proc)), isn't it? (if captured variables are needed, otherwise def is better of course).

[vinistock (Vinicius Stock)]

Regarding the examples, they are over simplified for the sake of understanding. Sure, you wouldn't need define_method in those cases. I'm talking more generally when someone needs to capture a block or surrounding locals to use in define_method.

Maybe here's a better example

def self.test(name, &block)
  define_method("test_#{name}", &block)
end

Indeed I haven't thought about using Ractor.make_shareable in the proc. But that still does not create the method in all Ractors, only inside the one where the invocation happens.

Could we make invocations to define_method with shareable procs create the method in all Ractors?

Additionally, is there a way to mark an inline block as shareable or would we always need to assign it to a variable? For example

define_method(:blah) do
  # How do I make this shareable without first assigning it to a local?
end

[nobu (Nobuyoshi Nakada)]

• Description updated (diff)

vinistock (Vinicius Stock) wrote in #note-3:

Indeed I haven't thought about using Ractor.make_shareable in the proc. But that still does not create the method in all Ractors, only inside the one where the invocation happens.

Could we make invocations to define_method with shareable procs create the method in all Ractors?

I can't get your point.
Methods are unrelated to Ractor.

Additionally, is there a way to mark an inline block as shareable or would we always need to assign it to a variable? For example

define_method(:blah) do
  # How do I make this shareable without first assigning it to a local?
end

define_method(:blah, &Ractor.make_shareable(proc do
end))

[byroot (Jean Boussier)]

Maybe it's orthogonal, but what if define_method accepted any object responding to call?

e.g.

class SomeCallback
  def initialize(foo, bar)
     @foo = foo
     @bar = bar
  end

  def call
    @foo.call(@bar)
  end
end

define_method(:my_method, SomeCallback.new(some_random_thing, 42))

This would allow tighter control on what's captured, and also be helpful for many other use cases such as delegation etc.

[vinistock (Vinicius Stock)]

I can't get your point.
Methods are unrelated to Ractor.

If we create a method using define_method, it only exists in the Ractor that made the invocation. Trying to invoke that method from a different Ractor throws an error (something like defined in another Ractor). This can be limiting for certain types of Ractor applications.

Let's continue with the previous example of a test framework.

class Test
  def self.test(name, &block)
    define_method("test_#{name}", &block)
  end
end

class MyTest < Test
  test "that it works" do
    # ...
  end
end

In a scenario like that, all methods defined by the test singleton method will be created when loading the test classes, such as MyTest. This typically means that the methods will end up being defined in the main Ractor while requiring files.

If we try to create a worker pool of Ractors to run the tests, none of the Ractors have access to the methods that were defined invoking test.

I imagined that the reason for that is because define_method captures the surrounding environment. That's why I thought, maybe if we don't capture it we could have the dynamic be accessible to all Ractors.

Is there a different reason behind why other Ractors can't invoke methods dynamically defined by another Ractor?

[Eregon (Benoit Daloze)]

• Status changed from Open to Rejected

vinistock (Vinicius Stock) wrote in #note-3:

Indeed I haven't thought about using Ractor.make_shareable in the proc. But that still does not create the method in all Ractors, only inside the one where the invocation happens.

That's not true, methods are the same for all Ractors, always.
They might throw an exception on non-main Ractor, and the way to fix that is to use Ractor.make_shareable.

[ko1 (Koichi Sasada)]

• Status changed from Rejected to Open
• Assignee set to ko1 (Koichi Sasada)

Eregon's comment #7 is true. This is why Ractor.make_shareable(proc_object) is supported.

So

class C
  a = 10
  define_method(:show_a, &Ractor.make_shareable(pr))
end

Ractor.new{ C.new.show_a }.take
#=> 10

works completely.

On the other hands, writing Ractor.make_shareable() is long, so define_method with keyword argument makes easier to write.

class C
  a = 10
  pr = Proc.new{p a}
  define_method(:show_a, shareable: true){p a}
end

for example.

[Eregon (Benoit Daloze)]

+1 for define_method(:show_a, shareable: true){p a}.
That nicely avoids the issue of whether Ractor.make_shareable(some_Proc) should make the receiver shareable, which is in the unique case of define_method does not matter.

[Dan0042 (Daniel DeLorme)]

Does Ractor.make_shareable(proc{}) actually do anything? It seem to only check if the proc is (can be?) shareable and raise an error otherwise, but it doesn't freeze any objects like Ractor.make_shareable(obj) usually does.

So in that case, why is there a need to explicitly specify shareable or not for define_method ? If the proc can be shareable, why not automatically define the method as shareable?

edit: sorry, Ractor.make_shareable(proc{}) of course makes a copy of the context. My brain must have been elsewhere.

[hsbt (Hiroshi SHIBATA)]

• Status changed from Open to Assigned