Bug #19371: Having Psych 5 installed raises an error during another gem's C-extension installation when parsing YAML

• Status: Third Party's Issue
• Assignee: hsbt (Hiroshi SHIBATA)
• Target version: -
• ruby -v: ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [aarch64-linux]
• Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN

[tombruijn (Tom de Bruijn)]

Summary

There's an issue on Ruby versions with Psych 4 installed by default (Ruby 2.6 through 3.1) after installing the Psych gem version 5. This problem occurs when a Ruby gem has a C-extension installation script that parses a YAML string.

I'm reporting it here and not with on the Psych gem repo, because it looks more like an issue with which Ruby C-extension is load during other gem's C-extension installation.

Background

I have a gem that parses a YAML string in the C-extension installation script, or it calls Gem.configuration[:http_proxy], which parses the .gemrc file as YAML. This triggers the error mentioned below.

This YAML parsing is done in the gem's ext/extconf.rb file. An example gem can be found in this repository: https://github.com/tombruijn/yaml-dummy-gem, see the ext/extconf.rb file.

The problem

On Ruby 3.1.3 Psych version 4 is installed by default. When it parses the YAML file, it will use Psych 4.

When Psych 5 is also installed on Ruby 3.1.3, it is no longer be able to parse the YAML file. The following error is raised:

$ bundle install
Fetching https://github.com/tombruijn/yaml-dummy-gem.git
Resolving dependencies...
Using bundler 2.3.7
Using yaml-dummy-gem 1.0.0 from https://github.com/tombruijn/yaml-dummy-gem.git (at main@a48852d)
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /usr/local/bundle/bundler/gems/yaml-dummy-gem-a48852dac33d/ext
/usr/local/bin/ruby -I /usr/local/lib/ruby/3.1.0 -r ./siteconf20230123-730-rmbnnl.rb extconf.rb
/usr/local/lib/ruby/3.1.0/psych.rb:459:in `parse_stream': undefined method `parse' for #<Psych::Parser:0x0000ffff8078c7f8
@handler=#<Psych::Handlers::DocumentStream:0x0000ffff8078c910 @stack=[], @last=nil, @root=nil, @start_line=nil, @start_column=nil, @end_line=nil, @end_column=nil,
@block=#<Proc:0x0000ffff8078c848 /usr/local/lib/ruby/3.1.0/psych.rb:399>>, @external_encoding=0> (NoMethodError)

      parser.parse yaml, filename
            ^^^^^^
	from /usr/local/lib/ruby/3.1.0/psych.rb:399:in `parse'
	from extconf.rb:3:in `<main>'

extconf failed, exit code 1

Gem files will remain installed in /usr/local/bundle/bundler/gems/yaml-dummy-gem-a48852dac33d for inspection.
Results logged to /usr/local/bundle/bundler/gems/extensions/aarch64-linux/3.1.0/yaml-dummy-gem-a48852dac33d/gem_make.out

  /usr/local/lib/ruby/3.1.0/rubygems/ext/builder.rb:95:in `run'
  /usr/local/lib/ruby/3.1.0/rubygems/ext/ext_conf_builder.rb:47:in `block in build'
  /usr/local/lib/ruby/3.1.0/tempfile.rb:317:in `open'
  /usr/local/lib/ruby/3.1.0/rubygems/ext/ext_conf_builder.rb:26:in `build'
  /usr/local/lib/ruby/3.1.0/rubygems/ext/builder.rb:161:in `build_extension'
  /usr/local/lib/ruby/3.1.0/rubygems/ext/builder.rb:195:in `block in build_extensions'
  /usr/local/lib/ruby/3.1.0/rubygems/ext/builder.rb:192:in `each'
  /usr/local/lib/ruby/3.1.0/rubygems/ext/builder.rb:192:in `build_extensions'
  /usr/local/lib/ruby/3.1.0/rubygems/installer.rb:853:in `build_extensions'
  /usr/local/lib/ruby/3.1.0/bundler/rubygems_gem_installer.rb:71:in `build_extensions'
  /usr/local/lib/ruby/3.1.0/bundler/source/path/installer.rb:34:in `post_install'
  /usr/local/lib/ruby/3.1.0/bundler/source/path.rb:244:in `generate_bin'
  /usr/local/lib/ruby/3.1.0/bundler/source/git.rb:194:in `install'
  /usr/local/lib/ruby/3.1.0/bundler/installer/gem_installer.rb:54:in `install'
  /usr/local/lib/ruby/3.1.0/bundler/installer/gem_installer.rb:16:in `install_from_spec'
  /usr/local/lib/ruby/3.1.0/bundler/installer/parallel_installer.rb:186:in `do_install'
  /usr/local/lib/ruby/3.1.0/bundler/installer/parallel_installer.rb:177:in `block in worker_pool'
  /usr/local/lib/ruby/3.1.0/bundler/worker.rb:62:in `apply_func'
  /usr/local/lib/ruby/3.1.0/bundler/worker.rb:57:in `block in process_queue'
  /usr/local/lib/ruby/3.1.0/bundler/worker.rb:54:in `loop'
  /usr/local/lib/ruby/3.1.0/bundler/worker.rb:54:in `process_queue'
  /usr/local/lib/ruby/3.1.0/bundler/worker.rb:91:in `block (2 levels) in create_threads'

An error occurred while installing yaml-dummy-gem (1.0.0), and Bundler cannot continue.

In Gemfile:
  yaml-dummy-gem

Debugging results

The error is raised because the Psych::Parser#parse method cannot be found. In Psych version 4, this method is defined by the Psych C-extension.

In Psych version 5 the parse method is defined in the gem's Ruby code. This method calls a C function registered as the private _native_parse method, which is the renamed version of the parse C-function in Psych version 4.

From what I can tell, the Psych version 4 C-extension is no longer loaded when Psych version 5 is installed in this scenario. There is a mix up in which Psych gem version's C-extension is loaded during my dummy gem's C-extension installation. It load the Psych 4 Ruby code, with the Psych 5 C-extension∂.

I confirmed this by modifying the standard installed Psych gem's code on Ruby 3.1 (Docker image ruby:3.1), with the following the change, which prints true on error. This means the Psych 4 gem has the Psych 5 C-extension loaded where _native_parse is defined.

diff --git lib/psych.rb lib/psych.rb
index 42d79ef..1a690d2 100644
--- lib/psych.rb
+++ lib/psych.rb
@@ -452,6 +452,9 @@ def self.parser
   def self.parse_stream yaml, filename: nil, &block
     if block_given?
       parser = Psych::Parser.new(Handlers::DocumentStream.new(&block))
+      # This returns `true`, but it should be `false`. The `_native_parse`
+      # method is defined in the Psych 5 C-extension, not Psych 4.
+      puts parser.respond_to? :_native_parse, true # => true
       parser.parse yaml, filename

This error only occurs during the gem's extension installation in ext/extconf.rb. If the gem parses YAML when a Ruby app is running, it will not produce the same error with Psych version 5 installed.

This issue does not occur on Ruby 3.2, where Psych version 5 is installed by default.

I have confirmed this error occurs on the latest patch releases of the following Ruby versions: 3.1, 3.0, 2.7 and 2.6.

$ gem list psych
psych (5.0.2, default: 4.0.3)

Code to reproduce

Here is a basic Ruby gem that only parses a YAML file during extension installation: https://github.com/tombruijn/yaml-dummy-gem

Here is a small project that triggers the error: https://github.com/tombruijn/yaml-dummy-ruby-app

A GitHub actions workflow shows the results for all affected Ruby versions: https://github.com/tombruijn/yaml-dummy-ruby-app/actions/runs/3969088933

The example app repo also has instructions to run the example app locally. Please follow the instructions in the README to see the error.

[jeremyevans0 (Jeremy Evans)]

I can confirm this issue. We recently experienced it in rack's CI (https://github.com/rack/rack/actions/runs/3955055993/jobs/6776667247#step:3:65). In our case, we were able to easily work around the issue because the gem that was pulling in psych (rdoc) wasn't actually needed in CI, so we just removed it as a dependency.

[hsbt (Hiroshi SHIBATA)]

• Status changed from Open to Assigned
• Assignee set to hsbt (Hiroshi SHIBATA)

Thanks for your investigation. I got same report at https://github.com/ruby/psych/discussions/607.

[hsbt (Hiroshi SHIBATA)]

• Status changed from Assigned to Third Party's Issue

This error caused Psych 4 was already activated by Gem::ConfigFile via Gem.configuration. After that, Psych 5 try to call _native_parse from Psych 4 binary in your code.

I have no idea to fix this yet. We mitigate this C extension conflict with pure ruby yaml parser without gemspec parsing.

@tombruijn (Tom de Bruijn) I'll do work this issue at https://github.com/rubygems/rubygems. Can you move this issue to rubygems repo? Thanks.

[tombruijn (Tom de Bruijn)]

hsbt (Hiroshi SHIBATA) wrote in #note-3:

@tombruijn (Tom de Bruijn) I'll do work this issue at https://github.com/rubygems/rubygems. Can you move this issue to rubygems repo? Thanks.

I have created an issue here: https://github.com/rubygems/rubygems/issues/6611

Sorry for the late response on transferring the issue to this issue tracker. The notification from the Ruby issue tracker was marked as spam for me because of some domain verification issue.