Your patch uses RB_VM_LOCK_ENTER_NO_BARRIER but it should block normal use of rb_bug() (using rb_bug() is irregular case though).
So I think it should use simpler mechanism to synchronize rb_bug() calling. For example, introducing a global variable to avoid multiple rb_bug() calls.
(btw VM_ASSERT() calls rb_bug() if RUBY_DEBUG (or other macros) is defined, so rb_bug() is suitable for the example)
Thanks for your comment. I can make it simpler, but I am a bit confused as to what I should do instead. If the first thread gets to the global variable first and enters rb_vm_bugreport, then other threads that also try to enter this function should be blocked, but there are only a few ways to block them:
use a mutex, like the VM lock in my patch
use a blocking system call like sleep (which is also safe in a signal handler in case someone uses VM_ASSERT or rb_bug in a signal handler
busy-wait until the process ends, which is going to be soon anyway
Would you prefer #2?
Also when you say use a global variable, do you mean an atomic global? I'm open to doing whatever you want, because maybe I'm overthinking it for just a debug case anyway.