Feature #8693: lambda invoked by yield acts as a proc with respect to return

• Status: Closed
• Assignee: ktsj (Kazuki Tsujimoto)
• Target version: 2.1.0

[rits (First Last)]

irb(main):004:0> def m1; yield end; def m2; m1 &->{return 0}; 1 end; m2
=> 0

[nobu (Nobuyoshi Nakada)]

• Status changed from Open to Rejected

It's spec.

[rits (First Last)]

what is the reason for this?

why should yield and block.call behave differently?

why should lambdas sometimes not have lambda semantics? When the code is written it has to assume lambda or proc semantics, it can't handle both, so why should the semantics vary?

PS email notification on bugs.ruby-lang.org is not working

[rits (First Last)]

rits (First Last) wrote:

what is the reason for this?

why should yield and block.call behave differently?

why should lambdas sometimes not have lambda semantics? When the code is written it has to assume lambda or proc semantics, it can't handle both, so why should the semantics vary?

PS email notification on bugs.ruby-lang.org is not working

could you please explain

[alexeymuranov (Alexey Muranov)]

rits (First Last) wrote:

what is the reason for this?

why should yield and block.call behave differently?

My guess: the ampersand operator has converted the "lambda" to a "block" (so the lambda object is lost).

[rits (First Last)]

alexeymuranov (Alexey Muranov) wrote:

rits (First Last) wrote:

what is the reason for this?

why should yield and block.call behave differently?

My guess: the ampersand operator has converted the "lambda" to a "block" (so the lambda object is lost).

it's not lost:

irb(main):011:0> def m1; p Proc.new; yield end; def m2; l = ->{return 0}; p l; m1 &l; 1 end; m2
#<Proc:0x00000002b76748@(irb):11 (lambda)>
#<Proc:0x00000002b76748@(irb):11 (lambda)>
=> 0

which makes sense, being in the block slot keeps the lambda alive

[rits (First Last)]

rits (First Last) wrote:

what is the reason for this?

why should yield and block.call behave differently?

why should lambdas sometimes not have lambda semantics? When the code is written it has to assume lambda or proc semantics, it can't handle both, so why should the semantics vary?

PS email notification on bugs.ruby-lang.org is not working

@nobu (Nobuyoshi Nakada) - please explain

[nobu (Nobuyoshi Nakada)]

return always returns from the enclosing method.
This is not affected by if it is in a block.
Only exception is the case nowhere to return from the block as the method already has exited, but your code is not in such case.

[rits (First Last)]

nobu (Nobuyoshi Nakada) wrote:

`return' always returns from the enclosing method.

That is not true, and so by design, from "The ruby programming language":

As we said earlier, lambdas work more like methods than blocks. A return statement in a lambda, therefore, returns from the lambda itself, not from the method that surrounds the creation site of the lambda.

---

This bug is an exception to this rule, one that does not make sense. If there is a good reason for it, you have not given it. If this behavior is not a bug (i.e. by design) then please answer:

what is the reason for this exception to the general lambda rule?

why should yield and block.call behave differently?

why should lambdas sometimes not have lambda semantics? When the code is written it has to assume lambda or proc semantics, it can't handle both, so why should the semantics vary?

---

Can anything be done about broken email alerts on bugs.ruby-lang.org?

[alexeymuranov (Alexey Muranov)]

This looks strange to me too. Also this:

def y; yield end
L = lambda{return 0}

def f; y &lambda{return 0} end
def g; y &L end

f # => 0
g # LocalJumpError: unexpected return

[rits (First Last)]

alexeymuranov (Alexey Muranov) wrote:

This looks strange to me too. Also this:

def y; yield end
L = lambda{return 0}

def f; y &lambda{return 0} end
def g; y &L end

f # => 0
g # LocalJumpError: unexpected return

That a return in a lambda invoked by yield attempts to return from the method enclosing the lambda definition, has already been demonstrated. The LocalJumpError is just a consequence of that, L is not defined in a method, so the attempted return fails. I don't think your example reveals anything new, does it?

[alexeymuranov (Alexey Muranov)]

I don't know. I could imagine that yielding to the lambda "converted to block" could try to return from the method from which the yielding method was called.

+1 to nobu

[alexeymuranov (Alexey Muranov)]

boris_stitnicky (Boris Stitnicky) wrote:

+1 to nobu

Boris, can you explain, please?

@Alexey: It's spec. As soon as you unpack the block with the pretzel, it forgets about its lambda-ness. If you wish to pass lambdas inside the method, you have to do it through the argument list:

def m1 f
  f.call
  return 1
end

m1 -> { return 0 }
#=> 1

[alexeymuranov (Alexey Muranov)]

@boris (Boris Ruf), this is what i thought first, but rits demonstrated that lambda is not forgotten. So it looks like some optimization or cheating to me: instead of properly forgetting about the lambda, it hangs around just in case (and can unexpectedly show up in Proc.new).

Can you maybe give a link to a place where this behavior is documented, anyway?

[nobu (Nobuyoshi Nakada)]

Then it's a bug in that book.

[deivid (David Rodríguez)]

Maybe this is related to #8622?

[rits (First Last)]

boris_stitnicky (Boris Stitnicky) wrote:

@Alexey: It's spec. As soon as you unpack the block with the pretzel, it forgets about its lambda-ness. If you wish to pass lambdas inside the method, you have to do it through the argument list:

This notion that & somehow extracts the block from the lambda "shell", throwing the lambda shell away is incorrect. & associates the passed proc/lambda with the block slot of the method, the proc/lambda never goes away, and yield does not invoke a naked block extracted and separated from the lambda but the lambda in the block slot. In the following example you can see that the original lambda is very much alive and when this lambda is invoked with yield it properly complains about argument count.

irb(main):006:0> def m1; p Proc.new; yield end; def m2; p l = ->_{return 0}; m1 &l; 1 end; m2
#<Proc:0x00000002a1e0d0@(irb):6 (lambda)>
#<Proc:0x00000002a1e0d0@(irb):6 (lambda)>
ArgumentError: wrong number of arguments (0 for 1)

The bottom line is, yield invokes the lambda, and the lambda runs with lambda semantics, with one exception, the return, which must be a bug, ruby is not supposed to have proc/lambda chimeras.

[rits (First Last)]

nobu (Nobuyoshi Nakada) wrote:

Then it's a bug in that book.

What exactly are you claiming is a bug in the book? The general rule about lambda semantics, that return in a lambda is supposed to return from the lambda? Are you suggesting the following is a bug?

irb(main):007:0> def m1 l; l.() end; def m2; m1 ->{return 0}; 1 end; m2
=> 1

Since matz is the author of both the language and the book, perhaps he can clarify where the bug is. In case he has not seen this thread, please ask him to weigh in.

[alexeymuranov (Alexey Muranov)]

I agree it would be nice if Matz clarified the issue.

rits (First Last) wrote:

This notion that & somehow extracts the block from the lambda "shell", throwing the lambda shell away is incorrect.

If you mean that this is not how things work, apparently it is true. But for me it would be the most natural if "&" was converting a lambda/proc into a block, and "&p" in the end of the argument list was wrapping the block into a proc called "p". For me in fact it would be the only non-surprizing behavior.

Consider this:

  def m(*a); a end
  a = []
  a.object_id     # => 2155500640
  m(*a).object_id # => 2155435920 (different)

This looks completely normal to me.

However, i am surprised with this:

  def m(&p); p end
  p = proc{}
  p.object_id     # => 2154923560
  m(&p).object_id # => 2154923560 (same)

[rits (First Last)]

alexeymuranov (Alexey Muranov) wrote:

I agree it would be nice if Matz clarified the issue.

rits (First Last) wrote:

This notion that & somehow extracts the block from the lambda "shell", throwing the lambda shell away is incorrect.

If you mean that this is not how things work, apparently it is true. But for me it would be the most natural if "&" was converting a lambda/proc into a block, and "&p" in the end of the argument list was wrapping the block into a proc called "p". For me in fact it would be the only non-surprizing behavior.

Consider this:

  def m *a
    a.object_id
  end

  a = []

  a.object_id # => 2155500640
  m *a        # => 2155435920 (different)

This looks completely normal to me.

• extracts individual elements and passes them individually, in fact the receiving method can then regroup the individual args in many different ways:

irb(main):010:0> def m1(a, *args, b); [a, args, b] end; m1 *[1, 2, 3, 4]
=> [1, [2, 3], 4]

the original array, by definition, is discarded, only the elements are passed, so this can't work any other way, whereas method(&proc_or_lamda) can theoretically works in both modes (keep and invoke the proc/lambda or "extract" and keep just the block). At a minimum it should be consistent. Currently it keeps and invokes the proc, but with hybrid proc/lambda semantics (argument checking of a lambda, return of a proc). The current behavior is bug regardless of which conceptual model you pick.

As to which model is preferable, I suppose that's debatable. I like the current model of keeping and invoking the proc/lambda (with this bug fixed). More on that below.

However, i am surprised with this:

  def m &p
    p.object_id
  end

  p = proc{}

  p.object_id # => 2154923560
  m &p        # => 2154923560 (same)

blocks are not first class citizens in ruby, there's no block class or block objects, so conceptually it makes sense to think of blocks as a role (or slot). So the block role/slot can be played/assumed/taken by a syntactic ruby block or by a proc. If you think of it as a role/slot then the proc does not need to be unwrapped into a block "object" and then re-wrapped, but merely assumes the block role or is put into the block slot, and is then looked up in the block slot. This model has implication for this bug, yield would check the block slot and if there's a proc there, simply invoke it. I think this is very close to what actually happens (given the evidence I provided: lambda is retained, lambda arity is enforced). The only quirk is the return

@Alexey: The OP complaint is a spec. But in the discussion (which I thoroughly read only upon
your reminder), different issues appear. That of block "hanging around", accessible by Proc.new:

l = -> {}
#<Proc:0xb85c3df8@(irb):38 (lambda)>
def x; Proc.new end
x &l
#<Proc:0xb85c3df8@(irb):38 (lambda)>

With its lambda-ness being (counterintuitively) remembered. Also, that of LocalJumpError:

def y &b; yield end
y &-> { return 42 }

Like you said, it is not clear, to what degree these are Ruby bugs, documentation bugs,
bugs in our heads, or bugs at all. But their (interesting and enlightening) discussion
here seems to me beyond the scope of the OP.

[alexeymuranov (Alexey Muranov)]

@rits (First Last): i plan to stop commenting on this thread and hope that Matz or someone else will provide and authoritative explanation some time.

I want to mention however how i view blocks in Ruby. For me a block "{ |x| x + 1 }" is somewhat like "1, 2, 3": "1, 2, 3" is not an object, but can appear in square brackets, like in "[0, 1, 2, 3]", or after a method name in a method call, like in "f(1, 2, 3)". There is a syntax to unwrap an array into "1, 2, 3" ("[0, *[1, 2, 3]] == [0, 1, 2, 3]"), which is also appropriately used to wrap "1, 2, 3" into an array in a method call. I would think that blocks and their wrapping procs would behave the same.

[mame (Yusuke Endoh)]

My guess:

• yield invokes any block as a plain block
• lambda block always checks the arguments

Anyway, I don't recommend you to write a code that depends on this behavior because this is considered an implementation-detail.

If you want to use the behavior as a spec, or change the behavior, it would be good to make a feature request with actual use case.
From my experience, "just making it consistent" is not an effective reason to do it.

--
Yusuke Endoh [e-mail]

[rits (First Last)]

mame (Yusuke Endoh) wrote:

My guess:

• yield invokes any block as a plain block
• lambda block always checks the arguments

A "plain block" that "checks arguments" is an oxymoron. Obviously such a thing should not exist.

If it behaved truly as a plain block that would be saner, however, as long as the lambda is alive (and it is) it still does not make sense for it to run as anything but a lambda.

Incidentally, here's another indication that under the current model, procs passed as blocks are not intended to be decomposed/unpacked into blocks (losing their proc shell in the process), but merely take the block slot as themselves (procs):

irb(main):001:0> l = ->{}
=> #<Proc:0xb9471e90@(irb):1 (lambda)>
irb(main):002:0> proc &l
=> #<Proc:0xb9471e90@(irb):1 (lambda)>

Anyway, I don't recommend you to write a code that depends on this behavior because this is considered an implementation-detail.

If you want to use the behavior as a spec, or change the behavior, it would be good to make a feature request with actual use case.
From my experience, "just making it consistent" is not an effective reason to do it.

The reason for consistency is avoiding surprise and bugs. If you write a lambda, you expect it to work as a lambda always, as the closest thing to a "spec" we have (ruby programming lang), declares it should. If it doesn't, that's a guaranteed bug in your code.

[mame (Yusuke Endoh)]

rits (First Last) wrote:

A "plain block" that "checks arguments" is an oxymoron. Obviously such a thing should not exist.

I do not agree nor disagree with your opinion.

Incidentally, here's another indication that under the current model, procs passed as blocks are not intended to be decomposed/unpacked into blocks (losing their proc shell in the process), but merely take the block slot as themselves (procs):

Imagine a natural Ruby implementation of proc:

def proc(&blk)
  blk
end

It does not use "yield".
It looks reasonable to me to return a lambda when lambda is given.

The reason for consistency is avoiding surprise and bugs.

In general, compatibility beats consistency.
A better reason is often required to change a spec and/or behavior.
Matz sometimes does so according to his mood, though.

--
Yusuke Endoh [e-mail]

[nobu (Nobuyoshi Nakada)]

• Tracker changed from Bug to Feature
• Status changed from Rejected to Assigned
• Assignee set to matz (Yukihiro Matsumoto)

[rits (First Last)]

mame (Yusuke Endoh) wrote:

A better reason is often required to change a spec and/or behavior.

You and others keep referring to some "spec" without linking or quoting it. Since you seem to respect the idea of a spec, allow me to quote again the closest thing there is to a spec, matz' "the ruby programming language" book:

"A return statement in a lambda, therefore, returns from the lambda itself, not from the method that surrounds the creation site of the lambda"

"The fact that return in a lambda only returns from the lambda itself means that we never have to worry about LocalJumpError"

yield is invoking a lambda, as already demonstrated, it is not a naked block somehow extracted and separated from the discarded lambda, the original lambda is still alive and still enforcing arity. Therefore lambda semantics should hold.

Having yield invoke lambdas with consistent (as opposed to current schizophrenic behavior) block semantics, I would argue is still a violation of the spec and common sense. The definition of a lambda is its semantics, so they should not vary.

[matz (Yukihiro Matsumoto)]

We discussed about the issue, and concluded the OP's intuition is natural.
So we seriously consider changing the behavior of return in lambda in 2.1.
But since it's incompatible change, we may give up the feature if some major incompatibility issue arise.

Matz.

[ktsj (Kazuki Tsujimoto)]

• File lambda-yield.patch lambda-yield.patch added

I agree with the proposal, so I've attached a patch.
With this patch:

• yield invokes lambda as a lambda block
• lambda block always checks the arguments

But note that it may introduce incompatibility with JIS X 3017, ISO/IEC 30170.

IPA Ruby Standardization WG Draft(http://www.ipa.go.jp/osc/english/ruby/ruby_draft_specification_agreement.html) specifies
Kernel.lambda as follows. (Sorry, I don't have the official standard document.)

15.3.1.2.6 Kernel.lambda

Behavior: The method creates an instance of the class Proc as Proc.new does (see 15.2.17.3.1).
However, the way in which block is evaluated differs from the one described in 11.3.3 except
when block is called by a yield-expression.

[nobu (Nobuyoshi Nakada)]

• Category set to core
• Assignee changed from matz (Yukihiro Matsumoto) to ktsj (Kazuki Tsujimoto)
• Target version set to 2.1.0

Seems fine.
And now matz considers it a bug in the ISO spec.

Attached a proposal for the NEWS and the test.

[ktsj (Kazuki Tsujimoto)]

Thanks a lot, but I think you forgot to attach the patch.

[nobu (Nobuyoshi Nakada)]

• File 0001-returning-from-lambda.patch 0001-returning-from-lambda.patch added

I've pasted the path at the description text...

[ktsj (Kazuki Tsujimoto)]

• Status changed from Assigned to Closed
• % Done changed from 0 to 100

This issue was solved with changeset r42455.
First, thank you for reporting this issue.
Your contribution to Ruby is greatly appreciated.
May Ruby be with you.

---

• vm_insnhelper.c (vm_invoke_block): returning from lambda proc
  now always exits from the Proc. [ruby-core:56193] [Feature #8693]

• NEWS, test/ruby/test_lambda.rb: ditto. Patch by nobu.

[akr (Akira Tanaka)]

• Related to Feature #15973: Let Kernel#lambda always return a lambda added