Bug #19115
closedRubyGems fails to detect OpenSSL in --with-static-linked-ext builds
Description
Related discussion: https://bugs.ruby-lang.org/issues/18876
We are seeing OpenSSL failing to autoload with the Ruby build we've been using. (We first observed this when using Gem.install
that relies on autoload of OpenSSL). Our last working build was Ruby 2.7.2 with similar config.
This causes RubyGems' HAVE_OPENSSL
constant to be incorrectly initialized, and subsequent calls to Gem.install
fails. There might be other scenarios that fail as a result of that, but we've not managed to identified that.
module Gem
HAVE_OPENSSL = defined? OpenSSL::SSL # :nodoc:
Our Ruby 3.1.2 config:
'--prefix=/Users/user/.conan/data/ruby/3.1.2/sketchup/stable/build/bd95825b99dcb274d0e577fbdc953653a9ea0fb9'
'--with-openssl-dir=/Users/user/.conan/data/openssl/1.1.1q/sketchup/stable/package/93ae0b0e7eebe0611c04d3e0d9bbf49fbce92332'
'--with-libyaml-dir=/Users/user/.conan/data/libyaml/0.2.5/sketchup/stable/package/a56a950abed2e10dbdc26845400f0a034d97c454'
'--disable-install-doc'
'--disable-install-rdoc'
'--enable-shared'
'--enable-load-relative'
'--with-static-linked-ext'
'--without-debug'
'--without-gdbm'
'--without-gettext'
'--without-irb'
'--without-mkmf'
'--without-rdoc'
'--without-readline'
'--without-tk'
'--bindir=${prefix}/bin'
'--sbindir=${prefix}/bin'
'--libexecdir=${prefix}/bin'
'--libdir=${prefix}/lib'
'--includedir=${prefix}/include'
'--oldincludedir=${prefix}/include'
'--datarootdir=${prefix}/share' 'cflags=-mmacosx-version-min=10.14 -fdeclspec' 'cxxflags=-mmacosx-version-min=10.14 -fdeclspec' 'LDFLAGS=-mmacosx-version-min=10.14 -fdeclspec'
We also tested with latest build from master (November 8th 2022):
./ruby -ve 'p RbConfig::CONFIG["configure_args"]'
ruby 3.2.0dev (2022-11-07T19:35:21Z master b14f133054) [x86_64-darwin20]
" '--prefix=/Users/vmehta/ruby/ruby-master/' '--with-openssl-dir=/Users/vmehta/.conan/data/openssl/1.1.1q/sketchup/stable/package/f2d937af1fa19d5fc4095849a65d1927e9e75ae7/' '--with-libyaml-dir=/Users/vmehta/.conan/data/libyaml/0.2.5/sketchup/stable/package/3fc084e254210603a5c5aece184b2d45e2509b30' '--disable-install-doc' '--disable-install-rdoc' '--enable-shared' '--enable-load-relative' '--with-static-linked-ext' '--without-debug' '--without-gdbm' '--without-gettext' '--without-irb' '--without-mkmf' '--without-rdoc' '--without-readline' '--without-tk'"
Using an RVM of Ruby 3.1.2 this appear to work as expected. But using out configuration, that we've used for the Ruby 2.x versions are now failing. We haven't been able to figure out the reason for this.
Updated by thomthom (Thomas Thomassen) almost 2 years ago
After looking into this closer, autoload
has been broken with this config at least back to Ruby 2.5 (Did test anything older).
What caused us to detect this in Ruby 3.1 was that RubyGems was using autoload
for OpenSSL where it previously use require
. We've avoided noticing this bug by happenstance and now it's finally bitten us.
Strangely, I've not been able to reproduce this with any autoload..
# foo.rb
puts 'foo.rb loaded'
module Foo; end
autoload(:Foo, 'C:/Users/tthomas2/Desktop/foo')
=> nil
p defined?(Foo)
"constant"
=> "constant"
Foo
foo.rb loaded
=> Foo
That works.
But not OpenSSL
, like Gem
is using it. Similarly, the example in https://bugs.ruby-lang.org/issues/18876#note-4 that was using Ripper:
autoload(:Ripper, "ripper"); p defined?(Ripper)
nil
=> nil
I'm not sure why some autoload
fails, while some doesn't.
Updated by alanwu (Alan Wu) almost 2 years ago
The issue is specific to extensions that are statically linked into the build in --with-static-linked-ext
.
autoload?
also doesn't work correctly with these extensions:
$ ruby --disable-all -ve 'autoload(:Ripper, "ripper"); p autoload?(:Ripper)'
ruby 3.2.0dev (2022-11-15T17:26:51Z autoload 9751b54971) [arm64-darwin22]
nil
This is coming from the special treatment for native extensions.
They are inserted into vm->loading_table
and so are treated as
being in the middle of being required on boot even when they're not actually
required yet. This weird loading state causes autoload?
and defined?
to
return nil
because it looks like querying the state of the constant while
it is autoloading.
I haven't tried making a patch, but it seems wise to stop using vm->loading_table
for this purpose.
Updated by thomthom (Thomas Thomassen) almost 2 years ago
So, this isn't necessarily limited to macOS? It just happens to be the configuration we're using on that platform?
Updated by alanwu (Alan Wu) almost 2 years ago
thomthom (Thomas Thomassen) wrote in #note-3:
So, this isn't necessarily limited to macOS? It just happens to be the configuration we're using on that platform?
Right.
Updated by alanwu (Alan Wu) over 1 year ago
I've submitted a PR to fix this issue and I thought I'd also post some updates.
Here is a script to summarize the mechanics of the bug:
puts RUBY_DESCRIPTION # ruby 3.2.0dev (2022-01-20) [wasm32-wasi]
autoload :Ripper, 'ripper' # can resolve to ripper.so, which is builtin
p [autoload?(:Ripper), defined?(Ripper), Object.const_defined?(:Ripper)]
# => [nil, nil, false]
# Compare with a library that's an .rb file
autoload :Abbrev, 'abbrev'
p [autoload?(:Abbrev), defined?(Abbrev), Object.const_defined?(:Abbrev)]
# => ["abbrev", "constant", true]
It's specific to builds configured with --with-static-linked-ext
.
Since WASM builds use this option you can reproduce the issue in a browser!
For native builds, the issue impacts RubyGems can make it think OpenSSL
is unavailable even though it is. The error is the same as mentioned in [Bug #18876]:
ERROR: While executing gem ... (Gem::Exception)
OpenSSL is not available. Install OpenSSL and rebuild Ruby (preferred) or use non-HTTPS sources
Also, some corrections for my mistakes in [Bug #18876].
I thought defined? AutoloadConst
would trigger loading but it's supposed
to behave more like checking whether loading is necessary.
I also thought the issue was fixed on master but it was actually due to me
not including Ripper in my test build which made ripper
resolve to ripper.rb
when the issue could only reproduce with .so
features like ripper.so
that are statically linked.
Updated by alanwu (Alan Wu) over 1 year ago
- Subject changed from OpenSSL fails to autoload (macOS) to RubyGems fails to detect OpenSSL in --with-static-linked-ext builds
Updated by alanwu (Alan Wu) over 1 year ago
- Status changed from Open to Closed
Applied in changeset git|790cf4b6d0475614afb127b416e87cfa39044d67.
Fix autoload status of statically linked extensions
Previously, for statically-linked extensions, we used
vm->loading_table
to delay calling the init function until the
extensions are required. This caused the extensions to look like they
are in the middle of being loaded even before they're required.
(rb_feature_p()
returned true with a loading path output.) Combined
with autoload, queries like defined?(CONST)
and Module#autoload?
were confused by this and returned nil incorrectly. RubyGems uses
defined?
to detect if OpenSSL is available and failed when OpenSSL was
available in builds using --with-static-linked-ext
.
Use a dedicated table for the init functions instead of adding them to
the loading table. This lets us remove some logic from non-EXTSTATIC
builds.
[Bug #19115]
Updated by thomthom (Thomas Thomassen) over 1 year ago
Will this patch be pack-ported to the 3.1 branch?
Updated by alanwu (Alan Wu) over 1 year ago
thomthom (Thomas Thomassen) wrote in #note-8:
Will this patch be pack-ported to the 3.1 branch?
It's up to the branch maintainer, @nagachika (Tomoyuki Chikanaga), to decide.
Updated by thomthom (Thomas Thomassen) over 1 year ago
I built Ruby 3.1.2 that we use in our product with this patch applied, and it worked on our systems. Would be nice to see it backported.
Updated by hsbt (Hiroshi SHIBATA) over 1 year ago
- Backport changed from 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN to 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: REQUIRED
Updated by nagachika (Tomoyuki Chikanaga) over 1 year ago
- Backport changed from 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: REQUIRED to 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: DONE
ruby_3_1 5c5a1135b2e688eca7dbc7ce085f37a4d9fa3fd1 merged revision(s) 790cf4b6d0475614afb127b416e87cfa39044d67.