Feature #10344: [PATCH] Implement Fiber#raise

• Status: Closed
• Assignee: ioquatix (Samuel Williams)
• Target version: 2.7

[nome (Knut Franke)]

While it is possible to implement this in pure Ruby (by wrapping Fiber.yield and Fiber#resume), this feels like a low-level feature that ought to be provided out of the box. Also, the C implementation is more straight-forward, and more efficient. Unfortunately, it is not quite possible to implement this as a C extension module (without resorting to wrappers again); cf. the change to make_passing_arg().

Example usage:

fib = Fiber.new do
  counter = 0
  loop { counter += Fiber.yield }
  counter
end
fib.resume
fib.resume 10
fib.resume 100
fib.raise StopIteration # => 110

[ko1 (Koichi Sasada)]

Do you have good example to use it?

[funny_falcon (Yura Sokolov)]

Yes: it is missing piece to make "greenlet" port, in other words - green threads that mimics real threads.
To accomplish that there is a need to raise exception in a fiber waiting on event if event fails.
As Knut said it could be implemented in a Ruby, but it feels to be dirty way. If implemented in C it will lead to cleaner implementation.

[nome (Knut Franke)]

For some more sophisticated examples, see https://github.com/nome/coroutines. The library does work with vanilla Ruby, but the patch improves performance.

Also, similar code can be simplified by using Fiber#raise. Compare e.g. the two implementation of Consumer::Yielder#await at
https://github.com/nome/coroutines/blob/master/lib/coroutines/base.rb

[ko1 (Koichi Sasada)]

On 2014/10/12 1:28, [e-mail] wrote:

For some more sophisticated examples, see https://github.com/nome/coroutines. The library does work with vanilla Ruby, but the patch improves performance.

Also, similar code can be simplified by using Fiber#raise. Compare e.g. the two implementation of Consumer::Yielder#await at
https://github.com/nome/coroutines/blob/master/lib/coroutines/base.rb

I understand this feature helps some libraries. But I can't understand
why it is important.

I'm afraid that introducing such feature increases complexity of Fiber.
Basically, I want to recommend strongly that using Fiber as
semi-croutine, ristricted feature.

At least, such feature should be located at ext/fiber.

--
// SASADA Koichi at atdot dot net

[funny_falcon (Yura Sokolov)]

At least, such feature should be located at ext/fiber.

even if ext/fiber it is still should be implemented in C .

I'm afraid that introducing such feature increases complexity of Fiber.

Not too much. Look at the patch: it uses same hook as rb_fiber_terminate.

By the way, why Thread has this feature?
http://www.ruby-doc.org/core-2.1.3/Thread.html#method-i-raise
And why Fiber hasn't? In fact, it is strange that this feature still not
presented, cause it is expected to exists.

[nome (Knut Franke)]

• File 0001-Implement-Fiber-raise.patch 0001-Implement-Fiber-raise.patch added
• File 0001-Implement-Fiber-raise-in-ext-fiber.patch 0001-Implement-Fiber-raise-in-ext-fiber.patch added

I understand this feature helps some libraries. But I can't understand why it is important.

Without Fiber#raise, libraries need to use some rather inelegant and inefficient workarounds (wrapping or tagging exceptions, wrapping Fiber.yield etc). Also, having Thread#raise arguably leads one to expect the same functionality in Fiber. I'm not sure whether the feature qualifies as "important", but I still think it is worth including.

Basically, I want to recommend strongly that using Fiber as semi-croutine, ristricted feature.

I think Consumer is an example of a semi-coroutine (in the sense that it uses Fiber.yield, not Fiber#transfer) that benefits from having Fiber#raise.

At least, such feature should be located at ext/fiber.

I disagree, because the feature is applicable to the restricted (semi-coroutine) fibers available without requiring 'fiber', and indeed implementable on top of them. Nevertheless, I've attached a variant of the patch which adds the method only in ext/fiber (as a compromise solution).

[ko1 (Koichi Sasada)]

Yura Sokolov wrote:

At least, such feature should be located at ext/fiber.

even if ext/fiber it is still should be implemented in C .

I agree.

I'm afraid that introducing such feature increases complexity of Fiber.

Not too much. Look at the patch: it uses same hook as rb_fiber_terminate.

I don't mention about implementation, but specification.

By the way, why Thread has this feature?
http://www.ruby-doc.org/core-2.1.3/Thread.html#method-i-raise
And why Fiber hasn't? In fact, it is strange that this feature still not
presented, cause it is expected to exists.

Thread#raise is one of inter communication method (but asynchronouse, difficult, unrecommended way).

Fibers can be under control without such communication.

[ko1 (Koichi Sasada)]

Knut Franke wrote:

I understand this feature helps some libraries. But I can't understand why it is important.

Without Fiber#raise, libraries need to use some rather inelegant and inefficient workarounds (wrapping or tagging exceptions, wrapping Fiber.yield etc).

I understand.

Also, having Thread#raise arguably leads one to expect the same functionality in Fiber. I'm not sure whether the feature qualifies as "important", but I still think it is worth including.

I don't expect there is Fiber#raise. Fiber and Thread are different.

Basically, I want to recommend strongly that using Fiber as semi-croutine, ristricted feature.

I think Consumer is an example of a semi-coroutine (in the sense that it uses Fiber.yield, not Fiber#transfer) that benefits from having Fiber#raise.

Interesting. I don't know details of this library. Could you explain why it is important in this library?

[nome (Knut Franke)]

Koichi Sasada wrote:

I think Consumer is an example of a semi-coroutine (in the sense that it uses Fiber.yield, not Fiber#transfer) that benefits from having Fiber#raise.

Interesting. I don't know details of this library. Could you explain why it is important in this library?

Consumer provides an abstraction for (semi-)coroutines that accept (consume) values; analogous to an Enumerator, which produces (enumerates) values. Since a consumer may need to do resource cleanup and/or produce a final result, we need some way to signal end of values. I think the most natural way to do this is to raise StopIteration in the consumer: The situation is analogous to calling Enumerator#next when no more values are available, and StopIteration terminates Kernel#loop, which often allows writing consumers very concisely.

In general, I think Fiber is a powerful primitive that can be used by libraries to build abstractions like Enumerator, Consumer and others. While Enumerator does not require the #raise feature, other abstractions can very well benefit from it.

[yugui (Yuki Sonoda)]

• Assignee set to ko1 (Koichi Sasada)
• Target version set to 2.5

Let me add another use case which I discussed with naruse and akr in RubyKaigi.

Fiber#raise is helpful to improve compatibility of methods with blocks (internal iterators) and Enumerator (external iterators).
This happened for me when I tried to write an adapter between two independently-developed APIs.

Here's a simplified example below. You can also find the real example in https://github.com/supership-jp/activerecord-spanner-adapter/blob/master/lib/active_record/connection_adapters/spanner/client.rb.

Example

Suppose that we have a third-paty API, which we cannot control.

def with_transaction
  tx = Transaction.new
  begin
    yield tx
  rescue
    tx.rollback
  else
    tx.commit
  end
end

And now I need to begin a transaction in a method and then need to commit or rollback the transaction in other methods.

class TransactionManager
  def begin_transaction
     @iter = Transaction.enum_for(:begin_transaction)
     @tx = @iter.next # skip error handlings for simplicity
  end

  def commit_transaction
    loop { @iter.next }
  ensure
    @tx = nil
  end

  def abort_transaction(reason = RuntimeError.new)
    # ??? How can I let the iterator know the error raised
    @iter.raise(reason)
  end
  
  attr_reader :tx
end

In other words, internal iterators in Ruby can manage resources in it but external iterators can't even if the user tried to take responsibility of calling cleanup mechanism.
In the real usecase, with_transaction was an API given by google-cloud-spanner gem and the interface of the TransactionManager was the one I had to implement for ActiveRecord.

Proposal

Although I could certainly implement the adapter without the native Fiber#raise, it would be still helpful if internal iterators and external iterators in Ruby were more consistent.

Can we have Fiber#raise and Enumerator#raise on top of it to improve the consistency? ko1?

FYI: I've confirmed that the patch by nome still works fine for ruby-trunk. https://github.com/yugui/ruby/commit/5a04a8b49b5086686fd75c6635c95c12ccc6caa8

[naruse (Yui NARUSE)]

• Target version deleted (2.5)

[ioquatix (Samuel Williams)]

I think this is a good idea. I already ended up implementing it by hand, but it's messy.

https://github.com/socketry/async/blob/e169aac7bc47f251ae7c6ebe94820b41bc2d063f/lib/async/task.rb#L43-L55

[ioquatix (Samuel Williams)]

@ko1 (Koichi Sasada) do you think we can aim for 2.6? I don't mind reviewing this patch and merging it if you are happy with it.

[ioquatix (Samuel Williams)]

• Status changed from Open to Assigned
• Assignee changed from ko1 (Koichi Sasada) to ioquatix (Samuel Williams)
• Target version set to 2.7

We will experiment and aim to merge this into 2.7

[ioquatix (Samuel Williams)]

• Related to Bug #595: Fiber ignores ensure clause added

• Status changed from Assigned to Closed

Applied in changeset trunk|r66610.

---

Implement Fiber#raise. Fixes #10344.

This allows raising exceptions in another fiber, similarly to
Thread#raise.

[ioquatix (Samuel Williams)]

@nome (Knut Franke) can you review the relevant commits and give me feedback?

[naruse (Yui NARUSE)]

@ioquatix (Samuel Williams) Is this feature accepted by matz?
A change which is visible from Ruby and non trivial bugs requires matz's approval.

And also you need to add description to NEWS if you add new feature.

[ko1 (Koichi Sasada)]

naruse (Yui NARUSE) wrote:

@ioquatix (Samuel Williams) Is this feature accepted by matz?
A change which is visible from Ruby and non trivial bugs requires matz's approval.

this is a support comment.

Matz agreed this feature several years ago (maybe I'm only person against, and I also agree to introduce this feature now, except naming issue. but nobody except me has no objection :p)
I'm not sure about naming issue, but he doesn't say anything last time.

Anyway, it is safe to discuss at next dev meeting (I will speak for Samuel).

Thanks,
Koichi

[ioquatix (Samuel Williams)]

Thanks I didn’t know approval procedure.

I will update NEWS one I’m sure the feature has stabilised.

[naruse (Yui NARUSE)]

ko1 (Koichi Sasada) wrote:

naruse (Yui NARUSE) wrote:

@ioquatix (Samuel Williams) Is this feature accepted by matz?
A change which is visible from Ruby and non trivial bugs requires matz's approval.

this is a support comment.

Matz agreed this feature several years ago (maybe I'm only person against, and I also agree to introduce this feature now, except naming issue. but nobody except me has no objection :p)
I'm not sure about naming issue, but he doesn't say anything last time.

Anyway, it is safe to discuss at next dev meeting (I will speak for Samuel).

Sure, thank you for comment.

ioquatix (Samuel Williams) wrote:

Thanks I didn’t know approval procedure.

I will update NEWS one I’m sure the feature has stabilised.

Anyway please update NEWS now to avoid forgetting adding the description.

[decuplet (Nikita Shilnikov)]

Shortly after I started to work on a library implementing algebraic effects (https://github.com/dry-rb/dry-effects) I stumbled upon lack of Fiber#raise. From the user POV it is important to signal improper use of effects with a meaningful stacktrace so they can quickly detect what's wrong. It's possible to emulate #raise with checks but it's quite inefficient and seems redundant. Thank you all folks for discussing and working on this, I hope it'll land in 2.7. If you're interested I can show a clear example but the task is almost identical to @ioquatix's one.

[ioquatix (Samuel Williams)]

NEWS is updated.