Bug #15362: [PATCH] Avoid GCing dead stack after switching away from a fiber

• Status: Closed
• Assignee: ioquatix (Samuel Williams)
• Target version: 2.6
• Backport: 2.5: DONE

[alanwu (Alan Wu)]

Hello! I have a patch that fixes Bug #14561. It's not a platform specific issue but
it affects the default build configuration for MacOS and is causing segfaults on 2.5.x.
I've put the test for this in a separate patch because I'm not sure if we want to have
a 5 second test that only matters for non-default build configs and doesn't catch things reliably on Linux.
I tested this on both trunk and ruby_2_5, on MacOS and on Linux, on various build configs.

Please let me know if anything in my understanding is wrong. I've pasted my commit message below.

---

Fibers save execution contextes, and execution contexts include a native
stack pointer. It may happen that a Fiber outlive the native thread
it executed on. Consider the following code adapted from Bug #14561:

enum = Enumerator.new { |y| y << 1 }
thread = Thread.new { enum.peek }  # fiber constructed inside the
                                   # block and saved inside `enum`
thread.join
sleep 5      # thread finishes and thread cache wait time runs out.
             # Native thread exits, possibly freeing its stack.
GC.start     # segfault because GC tires to mark the dangling stack pointer
             # inside `enum`'s fiber

The problem is masked by FIBER_USE_COROUTINE and FIBER_USE_NATIVE,
as those implementations already do what this commit does.
Generally on Linux systems, FIBER_USE_NATIVE is 1 even when
one uses ./configure --disable-fiber-coroutine, since most
Linux systems have getcontext() and setcontext() which
turns on FIBER_USE_NATIVE. (compile with `make
DEFS="-DFIBER_USE_NATIVE=0" to explicitly disable it)

Furthermore, when both FIBER_USE_COROUTINE and FIBER_USE_NATIVE
are off, and the GC reads from the stack of a dead native
thread, MRI does not segfault on Linux. This is probably due to
libpthread not marking the page where the dead stack lives as
unreadable. Nevertheless, this use-after-free is visible through
Valgrind.

On ruby_2_5, this is an acute problem, since it doesn't have FIBER_USE_COROUTINE.
Thread cache is also unavailable for 2.5.x, triggering this issue
more often. (thread cache gives this bug a grace period since
it makes native threads wait a little before exiting)

This issue is very visible on MacOS on 2.5.x since libpthread marks
the dead stack as unreadable, consistently turning this use-after-free
into a segfault.

Fixes Bug #14561

• cont.c: Set saved_ec.machine.stack_end to NULL when switching away from a
  fiber to keep the GC marking it. saved_ec gets rehydrated with a
  stack pointer if/when the fiber runs again.

[alanwu (Alan Wu)]

• Description updated (diff)

[aselder (Andrew Selder)]

Alan,

You're amazing... We're completely blocked on upgrading by this bug. You're a savior

Thanks,

Andrew

[ioquatix (Samuel Williams)]

• Assignee set to ioquatix (Samuel Williams)
• Target version set to 2.6
• Backport changed from 2.4: UNKNOWN, 2.5: UNKNOWN to 2.5: REQUIRED

I will take look at it.

[ioquatix (Samuel Williams)]

I looked at the patch it seemed okay.

I committed it to trunk r66111

Once we've confirmed it solves the issue, I will backport 2.5 branch, hopefully it can be included 2.5.4 or next patch release.

[ioquatix (Samuel Williams)]

After I merged this patch, https://travis-ci.org/ruby/ruby/jobs/462071967 fails. I am testing locally in more detail.

[ioquatix (Samuel Williams)]

Okay, it was random failure. That concerns me too. But, it passed now. I'll merge to ruby_2_5 too.

[normalperson (Eric Wong)]

[e-mail] wrote:

After I merged this patch,
https://travis-ci.org/ruby/ruby/jobs/462071967 fails. I am
testing locally in more detail.

Alan + Sam: thanks for investigating this.

Perhaps an alternative fix would be to avoid calling
rb_gc_mark_machine_stack() if the owner thread is dead.

I couldn't reproduce the original problem on FreeBSD 11.2
with both FIBER_USE_NATIVE and FIBER_USE_COROUTINE disabled;
so I can't verify my hypothesis
(Didn't bother testing on Linux)

https://bugs.ruby-lang.org/issues/15362#change-75323

[ioquatix (Samuel Williams)]

@normalperson (Eric Wong) I am open to ideas. I don't know a lot about how thread and GC interact within Ruby, unfortunately it's not an area I spent much time on. So, if you have better proposal, I suggest you implement it. This change included a spec, so I was happy to merge it even thought I didn't fully understand what is going wrong. Confirmed green build matrix so I am happy to close. If you want to make a change, let me know, otherwise I will close this and it would be backported to 2.5.

[ioquatix (Samuel Williams)]

@normalperson (Eric Wong) I thought about it a bit more, perhaps both ideas should be implemented for maximum protection. We could add some notes in the source code too about the issue/invariants.

[normalperson (Eric Wong)]

[e-mail] wrote:

@normalperson (Eric Wong) I thought about it a bit more, perhaps both
ideas should be implemented for maximum protection. We could
add some notes in the source code too about the
issue/invariants.

NAK. I'm against belt-and-suspenders fixes because it favors
lack-of-understanding and makes Ruby slower with more overhead.

Looking at Alan's original fix (which you committed), it seems
to make sense. Anyways, my hypothesis is here:

https://80x24.org/spew/[e-mail]/raw

It may allow Alan's simple one-line fix to be reverted.
However, I prefer Alan's one-liner fix since it's smaller
with instructions with no extra branches.

[ioquatix (Samuel Williams)]

• Status changed from Open to Closed

Okay, I think we are in agreement then.

[alanwu (Alan Wu)]

A comment for future maintainers:
Since &fib->cont.saved_ec == ruby_current_execution_context_ptr in the place my patch makes the change, if GC starts below where I set stack_end to NULL, the stack doesn't get marked.
This is not a problem right now since there is no call to any function that could trigger GC (correct me if I'm wrong), but future changes should be careful to not introduce such calls.
I wish there was some way to codify this constraint and have the compiler statically check this for us.

[dazuma (Daniel Azuma)]

Did this get backported to ruby_2_5? I don't see a corresponding commit in the github mirror https://github.com/ruby/ruby/commits/ruby_2_5

[aselder (Andrew Selder)]

This still looks like it's waiting on a backport to Ruby 2.5. Also, does anyone know when the next release of the Ruby 2.5 branch will be done?

Thanks!

[nagachika (Tomoyuki Chikanaga)]

• Has duplicate Bug #14561: Consistent 2.5.0 seg fault in GC, related to accessing an enumerator in a thread added

[nagachika (Tomoyuki Chikanaga)]

• Backport changed from 2.5: REQUIRED to 2.5: DONE

ruby_2_5 r66777 merged revision(s) 66111.

[wanabe (_ wanabe)]

• Related to Bug #14714: Ruby 2.5.1 Segmentation Fault in GC added

[wanabe (_ wanabe)]

• Related to Bug #15375: Crash report for Ruby 2.5.3p105 added