Feature #15567
closedAllow ensure to match specific situations
Description
There are some situations where rescue Exception
or ensure
are not sufficient to correctly, efficiently and easily handle abnormal flow control.
Take the following program for example:
def doot
yield
ensure
# Did the function run to completion?
return "abnormal" if $!
end
puts doot{throw :foo}
puts doot{raise "Boom"}
puts doot{"Hello World"}
catch(:foo) do
puts doot{throw :foo}
end
Using rescue Exception
is not sufficient as it is not invoked by throw
.
Using ensure
is inefficient because it's triggered every time, even though exceptional case might never happen or happen very infrequently.
I propose some way to limit the scope of the ensure block:
def doot
yield
ensure when raise, throw
return "abnormal"
end
The scope should be one (or more) of raise
, throw
, return
, next
, break
, redo
, retry
(everything in enum ruby_tag_type
except all except for RUBY_TAG_FATAL
).
Additionally, it might be nice to support the inverted pattern, i.e.
def doot
yield
ensure when not return
return "abnormal"
end
Inverted patterns allow user to specify the behaviour without having problems if future scopes are introduced.
return
in this case matches both explicit and implicit.
Updated by Eregon (Benoit Daloze) almost 6 years ago
I think the workaround using a variable set after yield
(success = true
) is not too bad, and clear enough. And the performance would be fine in this case (i.e. there would be no overhead if the JIT profiles the branch and only one branch is taken, like TruffleRuby).
Ensure
should always be executed no matter the circumstances IMHO, so I don't like to make it conditional with extra syntax.
Not executing ensure
when using break
, etc sounds like a bug in the vast majority of cases.
Updated by ioquatix (Samuel Williams) almost 6 years ago
@Eregon (Benoit Daloze) - thanks for your response.
Ensure
should always be executed no matter the circumstances IMHO, so I don't like to make it conditional with extra syntax.
This isn't about not executing ensure
, but allowing user to handle situations like "returned normally" vs "non-standard flow control".
I think the workaround using a variable set after
yield
(success = true
) is not too bad, and clear enough. And the performance would be fine in this case (i.e. there would be no overhead if the JIT profiles the branch and only one branch is taken, like TruffleRuby).
Personally, I think it's ugly, and I also think it is inefficient. I also don't think it's clearly conveying what the user is trying to do.
The problem is, people write code like this:
begin
..
ensure
abnormal_path if $!
end
This actually wrong and fails in the case of throw
wrapped in catch
as given in the original examples.
It's actually not clear from the code if this is what the user wanted or not. Because you can't express clearly what you are actually interested in.
There should be no overhead when exiting normally in the following situation:
begin
yield
ensure when not return
return :abnormal
end
As soon as you write code like:
begin
yield
success = true
ensure
return :abnormal unless success
end
you guarantee that the code must pass through the ensure
block. But your intention is, only execute this code if the code didn't return normally (or some other flow control).
So, I don't think the argument about always executing ensure
holds up - this isn't about changing the semantics of ensure
but extending it to handle explicitly what people are already doing, albeit probably incorrectly and inefficiently.
Updated by ioquatix (Samuel Williams) almost 6 years ago
- Target version deleted (
2.7)
Updated by shevegen (Robert A. Heiler) almost 6 years ago
I was about to write a long comment, but then I noticed that my reply would be too long, so I shortened
this. So just a few key statements:
I don't like the proposed syntax suggestion here in particular:
ensure when not return
The reason why I dislike it is partially due to the syntax verbosity; but also because, more significantly,
because it appears to do conditional checks in very different means than have existed before in ruby.
I would expect "when
" usually in a case-structure.
There was this other recent suggestion to also add "when
", to allow for chaining after the addition of
the alias "then
" to "yield_self
", leading to chains of .when .then .when .then
; and I also am not a huge fan
of using "when
" that way. Of course there is more than one way to do something, but my own personal
preference would be to keep "when
" reserved for case-when layouts.
As for catch
/throw
: I think it is a lot more common to see begin
/rescue
than catch
/throw
. Even if this
may be slower (e. g. some people doing control flow logic with begin
/rescue
), I think the intent and
clarity of intent is much clearer with control flows based on begin
/rescue
(and ensure) style layout
of code. So to conclude and finish my comment, I think it would be better to not introduce a more
complicated ensure conditional clause as proposed in the suggestion here. IMO you can re-structure
your code to keep ensure
simple, so it should not be necessary to add additional conditionals into
ensure
, even more so when these are unusual in regards to syntax - it would be the first time that I
would see "ensure when
" and I don't quite like that syntax suggestion in this context.
Updated by ioquatix (Samuel Williams) almost 6 years ago
@shevegen (Robert A. Heiler) - thanks for your feedback.
The reason why I dislike it is partially due to the syntax verbosity
With regards to syntax, it should use existing keywords, otherwise it can cause more problems. The reason for not
is already explained, because otherwise user may have to enumerate all other tags which is inconvenient and may break if new tags are introduced in the future. If you can propose better syntax, please do.
but also because, more significantly, because it appears to do conditional checks in very different means than have existed before in ruby.
This isn't really a conditional check like if
, and it's not even a big change internally, it simply exposes a bit more of the exception handling mechanisms. It should be more efficient, because ensure when not return
can be handled as part of exceptional flow control.
It's more similar to rescue
than ensure
in it's overhead. In MRI, rescue
has some overhead, but it's possible to implement zero-cost exception handling.
The reason why I think this is a good idea is because Ruby provides a lot of flow control statements, but handling these abnormal flow control is very tricky and easy to get wrong. Statements like ensure ... unless $!
is actually incorrect as well as inefficient. rescue Exception
doesn't handle all ways a function can exit.
I think the key things to think about would be:
- Is there a better syntax for this proposal?
- Does introducing such feature have any practical downside?
- Implementation cost.
- Maintenance cost.
- Does it introduce technical debt?
- Does it solve a problem that can't be solved some other way?
-
ensure when not return
can be partly solved some other way but with some overhead. -
ensure when raise
is similar torescue Exception
but doesn't requireraise
to resume default exception handling. -
ensure when retry
and other formulations (throw
,return
,next
,break
,redo
,retry
) have no equivalent right now. -
ensure when ..., ..., ...
is practically impossible, each case needs to be handled independently if possible.
-
- Even if there are existing solutions, does this proposed solution communicate more clearly the intention of the programmer?
- Do we want to expose this level of behaviour to user code?
- Could we limit what tags you can match? I suspect the most important one would be
ensure when not return
, but I can also imagine it could be pretty useful to catch things likeretry
andredo
.
Updated by ioquatix (Samuel Williams) almost 6 years ago
One more thing to bear in mind is that you can avoid this feature if you don't like it. Because it's for some very specific situations. It doesn't affect existing code in any way. But for situations like mine where I have a handful of use cases, and efficiency is a concern (to an extent) as well as readability and correctness, I would welcome such a feature. If you don't need it you won't be affected by it, but if you need it, there is actually no other practical solution in the language right now.
Updated by jeremyevans0 (Jeremy Evans) almost 6 years ago
ioquatix (Samuel Williams) wrote:
One more thing to bear in mind is that you can avoid this feature if you don't like it. Because it's for some very specific situations. It doesn't affect existing code in any way. But for situations like mine where I have a handful of use cases, and efficiency is a concern (to an extent) as well as readability and correctness, I would welcome such a feature. If you don't need it you won't be affected by it, but if you need it, there is actually no other practical solution in the language right now.
In the very rare cases where you want to treat a non-local, non-exception exit differently from a normal exit, as Eregon mentioned, there is a simple existing practical solution that is unlikely to be significantly less efficient:
def doot
ret = yield
normal_exit = true
ret
ensure
# Did the block return normally
return "abnormal" if $! || !normal_exit
end
I think we should definitely not add syntax in an attempt to make it easier to treat a non-local, non-exception exit differently than a normal exit, as doing so is usually a mistake.
You may not consider the above approach a simple and practical solution. However, considering you have already conceded that your proposal is for "very specific situations" and you only have "handful of use cases", even if you consider it neither simple nor practical, it's still easy and possible. New syntax is not necessary to support what you want, and I do not think new syntax should be added for "very specific situations" with only a "handful of use cases" when it is already possible to get the behavior you want with existing syntax.
Updated by ioquatix (Samuel Williams) almost 6 years ago
So, @jeremyevans0 (Jeremy Evans), I agree with everything you say.
That being said, this proposal goes beyond just that single case. For example, there is no easy way to capture retry
or redo
exiting a block.
Looking at your example, e.g.
def doot
ret = yield
normal_exit = true
ret
ensure
# Did the block return normally
return "abnormal" if $! || !normal_exit
end
I certainly prefer
def doot
yield
ensure when not return
return "abnormal"
end
It's clearer what's going on (especially if function is more complex than 1 line of actual code), it should be more efficiently implemented by VM since it only affects non-normal flow control and there is no conditional checks required.
Updated by Eregon (Benoit Daloze) almost 6 years ago
ioquatix (Samuel Williams) wrote:
For example, there is no easy way to capture
retry
orredo
exiting a block.
Which is intended, it would break the semantics of those if they were caught, or it would be very surprising if some ensure blocks are not run in these cases even though they exit the scope.
I think there is < 1% case where it's sensible to not run ensure in this kind of conditions, and the workaround seems good enough for these rare cases.
it should be more efficiently implemented by VM since it only affects non-normal flow control and there is no conditional checks required.
Please don't assume that. A check will be needed too in the general case, and it can already optimize very well with the local variable (e.g., zero overhead on TruffleRuby if only using the normal path, or only the exception path).
Updated by Eregon (Benoit Daloze) almost 6 years ago
@ioquatix (Samuel Williams) Could you give one or two real-world examples where this would be useful?
Updated by matz (Yukihiro Matsumoto) over 5 years ago
- Status changed from Open to Rejected
The language feature similar to ensure
can be observed in many languages (unwind-protocol
in Lisp, defer
in Go, finally
in Java and others), but neither of them considers the situation. That indicates there's no real-world use-case for it. So currently I believe there's no benefit enough the implementation cost and making the language more complex.
Matz.
Updated by ioquatix (Samuel Williams) over 5 years ago
Okay, thanks for your time and input. That makes sense.
Updated by ioquatix (Samuel Williams) over 3 years ago
I revisited this situation again.
This behaviour is very confusing:
def transaction
begin
puts "Begin Transaction"
yield
rescue
puts "Abort Transaction"
else
puts "Commit Transaction"
end
end
catch(:ball) do
transaction do
throw :ball
end
end
In particular, trying to handle this by using ensure ... unless $!
can fail in some situations due to scope of $!
def transaction
begin
puts "Begin Transaction"
yield
rescue
puts "Abort Transaction"
ensure
puts "Commit Transaction" unless $!
end
end
catch(:ball) do
begin
raise "Problem"
rescue
transaction do
throw :ball
end
end
end
It seems like we must involve some elaborate state tracking in order to get the correct behaviour (calling either abort or commit predictably). So, I believe we should reconsider this feature, or some evolution of it.
Updated by ioquatix (Samuel Williams) over 3 years ago
Here is working example:
def transaction
failed = false
begin
puts "Begin Transaction"
yield
rescue Exception
puts "Abort Transaction"
failed = true
raise
ensure
puts "Commit Transaction" unless failed
end
end
catch(:ball) do
begin
raise "Problem"
rescue
transaction do
throw :ball
end
end
end
This seems overly complex to me personally.
Updated by jeremyevans0 (Jeremy Evans) over 3 years ago
ioquatix (Samuel Williams) wrote in #note-14:
Here is working example:
def transaction failed = false begin puts "Begin Transaction" yield rescue Exception puts "Abort Transaction" failed = true raise ensure puts "Commit Transaction" unless failed end end catch(:ball) do begin raise "Problem" rescue transaction do throw :ball end end end
This seems overly complex to me personally.
You can simplify this slightly using a local variable:
def transaction
failed = false
begin
puts "Begin Transaction"
yield
rescue Exception => exc
puts "Abort Transaction"
raise
ensure
puts "Commit Transaction" unless exc
end
end
catch(:ball) do
begin
raise "Problem"
rescue
transaction do
throw :ball
end
end
end
I think such code makes sense. Exceptions abort the transaction, but should be reraised and not swallowed, so the raise
doesn't feel out of place.
Looking at my earlier example:
def doot
ret = yield
normal_exit = true
ret
ensure
# Did the block return normally
return "abnormal" if $! || !normal_exit
end
This is a case that wouldn't work correctly in the transaction scenario, where it is called inside an existing rescue
block. So you would currently have to use:
def doot
ret = yield
normal_exit = true
ret
rescue Exception => exc
raise
ensure
# Did the block return normally
return "abnormal" if exc || !normal_exit
end
About the only way I can think to make that easier would be for ensure
to support =>
:
def doot
ret = yield
normal_exit = true
ret
ensure => exc
# Did the block return normally
return "abnormal" if exc || !normal_exit
end
In this case exc
would be nil
if the code before ensure did not raise an exception, and the exception instance if it did.
However, I think the cases where this actually matters (ensure
without rescue
but that cares about whether an exception has been raised) are so few that we shouldn't consider language modifications to support them.
Updated by Eregon (Benoit Daloze) over 3 years ago
What's wrong with this, which works fine?
def transaction
begin
puts "Begin Transaction"
result = yield
success = true
result
ensure
if success
puts "Commit Transaction"
else
puts "Abort Transaction"
end
end
end
catch(:ball) do
transaction do
throw :ball
end
end
IMHO it's clear and explicit.
What we want to check here is ultimately if the block was successful (no error of any sort) and the local variable is pretty clear.
In fact, you might even want to allow the block to return :abort
or so, and then it would simply be if success && result != :abort
.
FWIW I think there are very few use cases where throw
/catch
are a good idea, but this applies to break
and others.
Short answer: use ensure
if you want to be sure something runs, rescue
only captures Exception
, as expected.
Updated by ioquatix (Samuel Williams) over 3 years ago
I suspect that there is existing buggy code which has ensure blocks and checks $!
which fails in the case I mentioned.
I think the ensure block should be able to capture the current exception without using a global.
I like the proposed syntax:
begin
...
ensure => exception
if exception
abort
else
commit
end
end
It allows existing code that uses if $!
to be retrofitted to do the correct thing.
Updated by ioquatix (Samuel Williams) over 3 years ago
@Eregon (Benoit Daloze) throw
should not abort the transaction, it should be committed.
Updated by Eregon (Benoit Daloze) over 3 years ago
I suspect that there is existing buggy code which has ensure blocks and checks $! which fails in the case I mentioned.
Typically that's an anti-pattern.
In 99% cases, the ensure code should do its logic regardless whether there is an ongoing exception/unwind or not.
Seems Jeremy agrees in comment 7:
I think we should definitely not add syntax in an attempt to make it easier to treat a non-local, non-exception exit differently than a normal exit, as doing so is usually a mistake.
Eregon (Benoit Daloze) throw should not abort the transaction, it should be committed.
That is surprising to me.
If throw
is performed in the middle of the transaction block then maybe only half the operations are done, committing in that case seems wrong.
transaction do
update1
some_call_that_ends_up_in_throw
update2
end
Updated by jeremyevans0 (Jeremy Evans) over 3 years ago
Eregon (Benoit Daloze) wrote in #note-19:
Eregon (Benoit Daloze) throw should not abort the transaction, it should be committed.
That is surprising to me.
Ifthrow
is performed in the middle of the transaction block then maybe only half the operations are done, committing in that case seems wrong.transaction do update1 some_call_that_ends_up_in_throw update2 end
Rails 7 is going to start treating non-local, non-exception exits as rolling back the transaction, the same as exception exits. I believe this is to work around the fact that Timeout uses throw (https://github.com/rails/rails/pull/29333).
Sequel won't make the same change. I think that non-local, non-exception exits should commit the transaction, and only exceptions should roll the transaction back. Changing the behavior just because of Timeout's design (even if Timeout is not used) seems to be a mistake to me. I certainly have a lot of code that uses throw to exit transactions and expects the transaction will be committed. Sinatra and Roda both use throw for returning responses to web requests, for example:
DB.transaction do
DB[:table].insert
unless update_related_table?
redirect '/some-path' # throw
end
DB[:related_table].update
end
Updated by Eregon (Benoit Daloze) over 3 years ago
Thanks for the link.
Is throw
used for other things than redirect
?
If redirect
would use a regular exception, and for instance pass []
as the backtrace, it would probably be as efficient, but then gives the possibility to differentiate for example with rescue => e; e&.should_commit?
or so.
throw
/catch
has AFAIK no way to communicate any information, so there is no way to know if the throw
is meant as a convenience return like for redirect
or as some kind of bailout which should not commit the transaction.
It can't even be differentiated from return/break so it seems a bad idea to use throw
in the first place.
It feels weird that Timeout uses throw
and not raise
, and it seems counter-intuitive (everyone knows Timeout.timeout {}
raises Timeout::Error, and exception, throw
is unexpected)
It also adds quite some complexity: https://github.com/ruby/timeout/blob/4893cde0eda321448a1a86487ac9b571f6c35727/lib/timeout.rb#L29-L50
Does anyone know what's the point of that?
What I could find is https://github.com/ruby/timeout/commit/238c003c921e6e555760f8e96968562a622a99c4
and https://bugs.ruby-lang.org/issues/8730
IMHO rescue Exception
without re-raise is always the bug.
Updated by jeremyevans0 (Jeremy Evans) over 3 years ago
Eregon (Benoit Daloze) wrote in #note-21:
Thanks for the link.
Is
throw
used for other things thanredirect
?
Yes, it's used anytime you want to return a response immediately. I just used redirect as an example.
If
redirect
would use a regular exception, and for instance pass[]
as the backtrace, it would probably be as efficient, but then gives the possibility to differentiate for example withrescue => e; e&.should_commit?
or so.
throw
/catch
has AFAIK no way to communicate any information, so there is no way to know if thethrow
is meant as a convenience return like forredirect
or as some kind of bailout which should not commit the transaction.
It can't even be differentiated from return/break so it seems a bad idea to usethrow
in the first place.
Changing a throw (which doesn't indicate error) to an exception (which indicates error) for a case that is not an error makes no sense to me. The problem is that Timeout uses throw instead of an exception for error cases (timeouts), not code that uses throw for non-error cases.
It feels weird that Timeout uses
throw
and notraise
, and it seems counter-intuitive (everyone knowsTimeout.timeout {}
raises Timeout::Error, and exception,throw
is unexpected)
It also adds quite some complexity: https://github.com/ruby/timeout/blob/4893cde0eda321448a1a86487ac9b571f6c35727/lib/timeout.rb#L29-L50
Does anyone know what's the point of that?
I believe this is because exception handling inside the Timeout.timeout block doesn't interfere with timeout handling.
What I could find is https://github.com/ruby/timeout/commit/238c003c921e6e555760f8e96968562a622a99c4
and https://bugs.ruby-lang.org/issues/8730
Yep, that pretty much confirms it. I don't agree with the tradeoff. Using throw for error cases is wrong, IMO. It would be better to have cases where Timeout didn't work due to improper rescuing, than not being able to use throw correctly because of Timeout's implementation. There are already places where Timeout doesn't work correctly, after all. A timeout error in my mind is more similar to a TERM signal, and Ruby handles that using an exception.
I should point out that this is not just a throw issue, it occurs for any non local exit. Here's an example using return. We have to use a transaction around the initial retrieval of the record due to the use of FOR UPDATE, because in certain conditions we want to update the record later, and don't want any modifications to the record in between. We need the transaction to commit and not rollback on exit, otherwise we lose the update to the associated object.
def foo
record = nil
transaction do
return :a unless record = dataset.for_update.first
record.associated_object.update()
return :b if record.some_condition?
record.update() if record.some_other_condition?
end
record
end
While you can generally rewrite code to avoid non-local exits like this, often the non-local exit approach is simplest.
Updated by Eregon (Benoit Daloze) over 3 years ago
A timeout error in my mind is more similar to a TERM signal, and Ruby handles that using an exception.
I think that's a good point.
I agree for similar things like Ctrl+C/SIGINT which is an Interrupt
exception (< SignalException < Exception), and Kernel#exit
which is a SystemExit
exception (< Exception).
Nothing can guarantee to bubble through user code without stopping in between (it's always possible to hang/loop via ensure
),
except a fatal error that would immediately end the process or skip even ensure
blocks and always reach the C main().
So proper exception handling (e.g. no rescue Exception; # ignore
, ensure
code must be bounded in time/steps) is always needed for correctness, and if Ctrl+C/exit work with an exception, so should Timeout.timeout.
Maybe we should open a new issue so that Timeout.timeout uses an Exception and not throw
?
I'm not sure if throw
is always used as non-local control flow, and not as some sort of exception (e.g., to indicate some 403).
BTW I guess we could replace throw with a non-local return
using handle_request(proc { return })
instead of catch
, but it wouldn't change anything related to this issue, both are considered non-local exits and non-distinguishable.
That's an argument for treating throw
as control flow much like non-local return
and not as exceptional though.
Updated by ioquatix (Samuel Williams) over 3 years ago
Thanks for the discussion. I'll try to summarise my position briefly.
Maybe we should open a new issue so that Timeout.timeout uses an Exception and not throw?
I agree Timeout should raise an exception internally and we should change this. Happy to work on a PR with you all. I am very strongly in favour of this. If we decide to do this, we should let Rails team know about our intended change so they don't end up working themselves into a corner.
Yes, it's used anytime you want to return a response immediately. I just used redirect as an example.
Yes, I also do this and agree with Jeremy's position here. While I do respect everyone's input, to me, exception = abort transaction, throw/break/next = commit.
Therefore, I do feel strongly that we need some precise way to capture the exception as we go through the ensure block. The previous suggestion makes sense to me, as in:
begin
raise "Boom"
ensure => exception
end
This actually makes it very easy to do the correct thing. The alternatives are tricky to get right:
failed = false
begin
rescue Exception # "Exception" usually missing... so buggy.
# error
failed = true
raise
ensure
if failed
abort
else
commit
end
end
I'm pretty sure I've got bugs in my code due to this, due to mis-understanding the flow control. So, I think we should make this easier for developers to get right.
Updated by ioquatix (Samuel Williams) over 3 years ago
- Status changed from Rejected to Open
Updated by ioquatix (Samuel Williams) over 3 years ago
Here are some examples of situations where $!
is used inside ensure
block. In every case, there is a potential buggy behaviour.
I can find more, but I think it's sufficient to show the problem.
Updated by ioquatix (Samuel Williams) over 3 years ago
Also there is a problem with the proposed work around:
begin
rescue Exception => exception
raise
ensure
if exception
abort
else
commit
end
end
This only works provided you don't have any other rescue
blocks, which you can see in some of the above examples may be an issue, or at least require careful implementation.
Updated by ioquatix (Samuel Williams) about 3 years ago
- Status changed from Open to Rejected
I will make a more limited proposal.
Updated by Eregon (Benoit Daloze) about 3 years ago
- Related to Feature #18083: Capture error in ensure block. added