Project

General

Profile

Actions

Bug #14009

closed

macOS High Sierra and “fork” compatibility

Added by ticky (Jessica Stokes) over 6 years ago. Updated over 5 years ago.

Status:
Closed
Assignee:
-
Target version:
-
ruby -v:
ruby 2.4.2p198 (2017-09-14 revision 59899) [x86_64-darwin17]
[ruby-core:83239]

Description

This was originally discussed on the issue tracker for Puma (https://github.com/puma/puma/issues/1421), however, it is possible that it would make more sense for inclusion in the Ruby implementation itself.

macOS High Sierra has changed the behaviour of the fork syscall such that initialising Objective-C APIs in forked processes are treated as errors. (see http://sealiesoftware.com/blog/archive/2017/6/5/Objective-C_and_fork_in_macOS_1013.html for more details)

This means that many applications which use forking to process concurrently will forcibly crash if the forked process calls out to any Objective-C library when Objective-C was not already initialised in the host process. This includes Puma, Unicorn, iodine and Passenger.

A workaround I proposed for Puma was to implicitly load the Objective-C runtime before performing any forks (https://github.com/puma/puma/issues/1421#issuecomment-332650703). This causes forked processes using other Objective-C APIs to not crash.

The workaround (specific to Puma’s DSL) was:

# Work around macOS 10.13 and later being very picky about
# `fork` usage and interactions with Objective-C code
# see: <https://github.com/puma/puma/issues/1421>
if /darwin/ =~ RUBY_PLATFORM
  before_fork do
    require 'fiddle'
    # Dynamically load Foundation.framework, ~implicitly~ initialising
    # the Objective-C runtime before any forking happens in Puma
    Fiddle.dlopen '/System/Library/Frameworks/Foundation.framework/Foundation'
  end
end

A similar fix has now been included in Passenger (https://github.com/phusion/passenger/blob/2a55a84e5de721d8bd806a8fea0bcedf27583c29/src/ruby_supportlib/phusion_passenger/loader_shared_helpers.rb#L84-L105).

It was, however, proposed that it might make more sense for Ruby on macOS High Sierra and onward to implicitly initialise the Objective-C framework itself, so that forked processes work roughly as expected even if they intend to use Objective-C APIs.

I understand that this is a heavy-handed move, but it seems to me that this relatively common technique will remain broken in Ruby unless everyone deploys a workaround (iodine has already expressed disinterest in doing so) or Ruby adopts one at the higher level.

This issue is also applicable to all Ruby versions which support fork and run on macOS High Sierra.

Thank you for your time. :)


Related issues 1 (0 open1 closed)

Related to Ruby master - Feature #5446: at_fork callback APIClosedkosaki (Motohiro KOSAKI)Actions

Updated by hongli (Hongli Lai) over 6 years ago

Perhaps it helps if I further clarify this issue for those who are unfamiliar with it.

As you probably already know, forking (but without exec'ing) in a multithreaded environment is inherently dangerous and the environment must be carefully written to support such a thing. Apple's Objective-C libraries have traditionally not supported being called in a forked (but not exec'd) child process at all, but since High Sierra 10.13 they've tried to add limited support for this. However in doing so they've also defined rules on what is not allowed after forking. One of the rules state that it is not allowed to call the initialize function of certain Objective-C classes after forking; that may only happen before forking.

Makes sense so far. The problem occurs because of a combination of three things:

  1. Ruby itself is not linked to any Objective-C libraries, and so does not initialize Objective-C classes by itself.
  2. The user may use gems that do link to Objective-C libraries. Due to how these gems are used, it can occur that these gems end up calling Objective-C initializers after the app server has forked.
  3. The new Apple-enforced rule checks then abort the process with a warning like this:
objc[81924]: +[__NSPlaceholderDictionary initialize] may have been in progress in another thread when fork() was called.
objc[81924]: +[__NSPlaceholderDictionary initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.

By itself, Apple's error check makes sense. Forking is dangerous. But all these factors combined make less sense. Adding a workaround in Ruby (in the form of ensuring that Objective-C initializers are called before forking) will at least ensure that we return to pre-High Sierra behavior.

Adding a workaround inside Ruby would only be a part of the whole solution. In order to fully fix the problem, cooperation from the wider Ruby community is required: all the gem authors will also have to ensure that native libraries don't spawn any threads until the app server has forked. But I do believe that having a workaround inside Ruby is an essential part of the entire fix, because updating all gems takes a lot of time and effort.

Actions #2

Updated by shyouhei (Shyouhei Urabe) over 6 years ago

Updated by myst (Boaz Segev) over 6 years ago

I should point out that "Foundation.framework/Foundation" seems to be enough.

This is the C code I'm currently considering for the iodine Ruby server.

#ifdef __APPLE__
  void *obj_c_runtime = dlopen("Foundation.framework/Foundation", RTLD_LAZY);
  /* iodine runs here and `fork` might be called */
  dlclose(obj_c_runtime);
#else
  /* iodine runs here and `fork` might be called */
#endif

Comments:

  • The __APPLE__ directive will only work on gcc (gnu compiler), clang and intel compilers (other compilers might be supported, but I don't know).

  • The code doesn't check for errors, because a quite failure doesn't effect behavior. If Objective-C isn't available, than forking will not be effected.

  • The code doesn't check for High Sierra specifically and will load Objective-C on all macOS versions (anything over 10.9).

The memory increase was from 4388Kb (Ruby VM) to ~4560Kb (compared using Fiddle, not a direct C call), which seems insignificant on the larger scale of things.

For comparison:

  • Baseline: loading the Objective-C using Fiddle caused a 3.9% increase in memory consumption relative to the basic the VM core.

  • Loading the Socket Standard library (require 'socket') caused a 16% increase in memory consumption relative to the basic the VM core.

  • Loading Sinatra caused more than a 196% increase in memory consumption relative to the basic the VM core.

  • Loading Rails caused more than a 264% increase in memory consumption relative to the basic the VM core.

It's my understanding that a simple patch will load the Objective-C library within the ruby_init() function and ignore any errors in loading the library.

A more complex approach will lazy load the library only when Process#fork is called... but this might effect C extensions that might call fork directly and it might also effect the use of Ruby as a library (when ruby_init() is called within a C application that uses Ruby internally).

Personally, I'm both lazy and paranoid and I would assume that the simple approach would both work better and fail better (expose issues earlier rather than later).

Actions #4

Updated by nobu (Nobuyoshi Nakada) over 6 years ago

  • Status changed from Open to Feedback

Does linking Foundation framework instead of CoreFoundation framework work?

For 2.4:

diff --git a/configure.in b/configure.in
index bb7cc4daa5..4a2243afa1 100644
--- a/configure.in
+++ b/configure.in
@@ -4144,8 +4144,8 @@ AS_CASE(["$target_os"],
 	],
     [darwin*], [
 	RUBY_APPEND_OPTION(CFLAGS, -pipe)
-	RUBY_APPEND_OPTION(XLDFLAGS, [-framework CoreFoundation])
-	RUBY_APPEND_OPTION(LIBRUBYARG_STATIC, [-framework CoreFoundation])
+	RUBY_APPEND_OPTION(XLDFLAGS, [-framework Foundation])
+	RUBY_APPEND_OPTION(LIBRUBYARG_STATIC, [-framework Foundation])
 	],
     [osf*], [
 	if test "$GCC" != "yes" ; then

For trunk:

diff --git a/configure.ac b/configure.ac
index e6cd95cc9d..6cd99e6ef2 100644
--- a/configure.ac
+++ b/configure.ac
@@ -4116,8 +4116,8 @@ AS_CASE(["$target_os"],
 	],
     [darwin*], [
 	RUBY_APPEND_OPTION(CFLAGS, -pipe)
-	RUBY_APPEND_OPTION(XLDFLAGS, [-framework CoreFoundation])
-	RUBY_APPEND_OPTION(LIBRUBYARG_STATIC, [-framework CoreFoundation])
+	RUBY_APPEND_OPTION(XLDFLAGS, [-framework Foundation])
+	RUBY_APPEND_OPTION(LIBRUBYARG_STATIC, [-framework Foundation])
 	],
     [osf*], [
 	AS_IF([test "$GCC" != "yes" ], [

Updated by myst (Boaz Segev) over 6 years ago

I can't test because I can't install High Sierra on my machine (I'm also a musician and my professional audio applications don't support High Sierra).... @ticky (Jessica Stokes)?

Updated by hongli (Hongli Lai) over 6 years ago

nobu (Nobuyoshi Nakada) wrote:

Does linking Foundation framework instead of CoreFoundation framework work?

According to my tests on High Sierra, linking to Foundation helps, but linking to CoreFoundation does not. CoreFoundation does not (at load time) initialize any Objective-C classes. Here are test scripts which simulate the issue (and a working fix and a non-working fix) on High Sierra:

broken.rb

# This script simulates a child process that initializes ObjC classes,
# and should crash.
require 'fiddle'

def init_objc_classes
  Fiddle.dlopen('/System/Library/Frameworks/Foundation.framework/Foundation')
end

pid = fork do
  init_objc_classes
end
Process.waitpid(pid)

Test run:

$ ruby broken.rb
objc[768]: +[__NSPlaceholderDictionary initialize] may have been in progress in another thread when fork() was called.
objc[768]: +[__NSPlaceholderDictionary initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.

fixed.rb

# This script simulates a process that initializes ObjC classes before,
# forking, and should NOT crash.
require 'fiddle'

def init_objc_classes
  Fiddle.dlopen('/System/Library/Frameworks/Foundation.framework/Foundation')
end

init_objc_classes
pid = fork do
  # The following does nothing because ObjC classes
  # are already initialized before forking.
  init_objc_classes
end
Process.waitpid(pid)
$ ruby fixed.rb
(no crash)

nobu-fix.rb

# This script simulates a process that tries to initialize ObjC classes before,
# forking, and still crashes because the method doesn't actually work. This
# simulates nobu's proposed fix in https://bugs.ruby-lang.org/issues/14009#note-4
require 'fiddle'

def invoke_nobu_fix
  Fiddle.dlopen('/System/Library/Frameworks/CoreFoundation.framework/CoreFoundation')
end

def init_objc_classes
  Fiddle.dlopen('/System/Library/Frameworks/Foundation.framework/Foundation')
end

invoke_nobu_fix
pid = fork do
  init_objc_classes
end
Process.waitpid(pid)

Test run:

$ ruby nobu-fix.rb
objc[1066]: +[__NSPlaceholderDictionary initialize] may have been in progress in another thread when fork() was called.
objc[1066]: +[__NSPlaceholderDictionary initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.

And here is a sanity check to verify that loading CoreFoundation in fact does not initialize prohibited ObjC classes:

# This script should not crash in the child. It should verify
# that CoreFoundation does not initialize any Objective-C
# classes.
require 'fiddle'

def load_core_foundation
  Fiddle.dlopen('/System/Library/Frameworks/CoreFoundation.framework/CoreFoundation')
end

load_core_foundation
pid = fork do
  load_core_foundation
end
Process.waitpid(pid)

Test run:

$ ruby corefoundation-sanity-check.rb
(no crash)

Updated by nobu (Nobuyoshi Nakada) over 6 years ago

"linking Foundation framework" means linking ruby and libruby.dylib with -framework Foundation compiler option.
You need to apply my patch to the source then rebuild it.

Updated by myst (Boaz Segev) over 6 years ago

EDIT:

Please ignore the original comment.

I tested linkage in Sierra (not High Sierra), the -framework instruction is honored even when no symbols are used.

The linker approach seems safe to use in macOS. I wish I could test it on High Sierra.


Original:

@nobu (Nobuyoshi Nakada) , I'm wondering if some linkers might optimize this instruction away...?

My question is whether or not an optimized linker might ignore the instruction after noticing that no symbols from the shared library are used.

If this is the case, than the linker approach will work on some linkers and not on others.

I think the documentation I am referencing (the documentation for ld) is the same for both static and dynamic libraries (but maybe it's only how static libraries are resolved):

-larchive
--library=archive
Add archive file archive to the list of files to link. This option may be used any number of times. ld will search its path-list for occurrences of libarchive.a for every archive specified. On systems which support shared libraries, ld may also search for libraries with extensions other than .a. Specifically, on ELF and SunOS systems, ld will search a directory for a library with an extension of .so before searching for one with an extension of .a. By convention, a .so extension indicates a shared library. The linker will search an archive only once, at the location where it is specified on the command line. If the archive defines a symbol which was undefined in some object which appeared before the archive on the command line, the linker will include the appropriate file(s) from the archive. However, an undefined symbol in an object appearing later on the command line will not cause the linker to search the archive again. See the -( option for a way to force the linker to search archives multiple times. You may list the same archive multiple times on the command line. This type of archive searching is standard for Unix linkers. However, if you are using ld on AIX, note that it is different from the behaviour of the AIX linker.

Because Ruby doesn't have any undefined symbols that are defined in the Foundation library, this library might be ignored and the patch will fail.

Updated by ticky (Jessica Stokes) over 6 years ago

Thank you all for your input! :)

I’ll look at getting 2.4.2 built with nobu’s patch and check how it looks from High Sierra.

Updated by ticky (Jessica Stokes) over 6 years ago

Alright, I’ve checked it out, and can confirm that the patch @nobu (Nobuyoshi Nakada) suggested does indeed work around the original issue I encountered in Puma, with my mitigations removed from our application code.

I can also confirm that with that patch applied, none of the code snippets supplied by @hongli (Hongli Lai) fail on High Sierra, which means this definitely works around the problem. :)

Updated by mperham (Mike Perham) over 6 years ago

The original cause was lazy loading the pg gem in the forked child process. Does this fix allow the same pg gem loading to work as before?

Updated by hongli (Hongli Lai) over 6 years ago

Regarding nobu's comment on whether linkers can ignore "useless" library links (comment #8): yes they can, but both on GNU/Linux systems as well as on macOS, this behavior is at present not the default.

On GNU systems, --as-needed lets the GNU linker skip linking unused libraries. There is one blog article out there that claims that --as-needed has been made the default on newer GNU systems, but I've not personally verified this, nor have I found any other sources so far.

macOS's linker has -dead_strip_dylibs which does the same.

So we don't have to worry about this issue.

Updated by ticky (Jessica Stokes) over 6 years ago

@mperham (Mike Perham) loading pg within a fork works just fine with this patch :)

Updated by nobu (Nobuyoshi Nakada) over 6 years ago

Thank you for the confirmation.
As for "useless" library link, -u linker option could force to link it.
You can see which libraries are linked dynamically, by otool -L command.

Actions #15

Updated by nobu (Nobuyoshi Nakada) over 6 years ago

  • Status changed from Feedback to Closed

Applied in changeset trunk|r60182.


configure.ac: link Foundation framework

  • configure.ac (XLDFLAGS): link against Foundation framework and
    let __NSPlaceholderDictionary initialize, to get rid of crash
    after fork on macOS High Sierra. [ruby-core:83239] [Bug #14009]
Actions #16

Updated by nagachika (Tomoyuki Chikanaga) over 6 years ago

  • Backport changed from 2.3: UNKNOWN, 2.4: UNKNOWN to 2.3: UNKNOWN, 2.4: REQUIRED

Updated by nobu (Nobuyoshi Nakada) over 6 years ago

Just an idea, environment variable DYLD_INSERT_LIBRARIES=/System/Library/Frameworks/Foundation.framework/Versions/Current/Foundation doesn't work?

Updated by myst (Boaz Segev) over 6 years ago

I'm happy this was merged to the Ruby trunk.

Just an idea, environment variable DYLD_INSERT_LIBRARIES=/System/Library/Frameworks/Foundation.framework/Versions/Current/Foundation doesn't work?

Interesting idea :)

It would probably work exactly like the patch, except it will be user controlled instead of persistent.

Updated by nobu (Nobuyoshi Nakada) over 6 years ago

myst (Boaz Segev) wrote:

It would probably work exactly like the patch, except it will be user controlled instead of persistent.

It's a temporary repair until the next release, of course.

Updated by ticky (Jessica Stokes) over 6 years ago

Just for reference, people seem confused about the fact this patch hasn’t made it into Ruby 2.4.3; as far as I can tell it will only appear in version 2.5. Is a backport forthcoming or is the idea for people who need High Sierra compatibility to simply move to 2.5?

https://github.com/puma/puma/issues/1421#issuecomment-352083160

Updated by nagachika (Tomoyuki Chikanaga) over 6 years ago

Thank you for your notice.
It's just due to my laziness that the backport has been delayed.
I will backport this soon.

Updated by ticky (Jessica Stokes) over 6 years ago

@nagachika (Tomoyuki Chikanaga) thank you for the update! I am glad it hasn’t been forgotten. :)

Updated by nagachika (Tomoyuki Chikanaga) over 6 years ago

  • Backport changed from 2.3: UNKNOWN, 2.4: REQUIRED to 2.3: UNKNOWN, 2.4: DONE

ruby_2_4 r61358 merged revision(s) 60182.

Updated by usa (Usaku NAKAMURA) about 6 years ago

  • Backport changed from 2.3: UNKNOWN, 2.4: DONE to 2.3: DONE, 2.4: DONE

ruby_2_3 r62144 merged revision(s) 60182.

Updated by quaeler (loki der quaeler) over 5 years ago

nagachika (Tomoyuki Chikanaga) wrote:

ruby_2_4 r61358 merged revision(s) 60182.

With 2.4.4 (ruby 2.4.4p296 (2018-03-28 revision 63013) [x86_64-darwin17]), I am still seeing
09:27:49 processor.1 | objc[44872]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called.
09:27:49 processor.1 | objc[44872]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.

Shouldn't this be fixed since the backport happened?

Actions

Also available in: Atom PDF

Like0
Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0