Bug #16904: rubygems: psych: superclass mismatch for class Mark (TypeError)

• Status: Closed
• Assignee: hsbt (Hiroshi SHIBATA)
• Target version: -
• Backport: 2.5: UNKNOWN, 2.6: UNKNOWN, 2.7: DONE

[jaruga (Jun Aruga)]

Is it possible to backport the following commits into Ruby?

Remove explicit psych activation
https://github.com/rubygems/rubygems/commit/1ccf0912a161d20e0c4a7b139fd76e8739a411ba

bundler: Remove explicit psych activation.
https://github.com/rubygems/rubygems/commit/88478bb56945ffa9868491cacd0bc74d9d73c848

It causes the following kind of error when psych is installed as not default gem but regular gem. It's a Fedora Ruby specific case. This is not the case of Ruby built from source normally.

$ cat test.rb 
require 'psych'
Gem.load_yaml # <= The error happens.

$ ruby test.rb
/usr/lib64/gems/ruby/psych-3.1.0/psych.so: warning: already initialized constant Psych::Parser::ANY
...
/usr/share/gems/gems/psych-3.1.0/lib/psych/parser.rb:34:in `<class:Parser>': superclass mismatch for class Mark (TypeError)

It is possibly also related to https://github.com/ruby/psych/issues/386 .

Thank you.

[jaruga (Jun Aruga)]

The error does not happen in Ruby 2.6.
So, it's great if it is backported to Ruby 2.7.

[nobu (Nobuyoshi Nakada)]

I can't see that error in 2.7 too.

$ for ruby in ruby2.{0..7}; do $ruby -rpsych -e 'p [RUBY_VERSION, Gem.load_yaml, Psych::VERSION]'; done
["2.0.0", true, "2.0.0"]
["2.1.9", true, "2.0.5"]
["2.2.10", true, "2.0.8"]
["2.3.8", true, "2.1.0"]
["2.4.10", true, "2.2.2"]
["2.5.8", true, "3.0.2"]
["2.6.6", true, "3.1.0"]
["2.7.1", true, "3.1.0"]

[nobu (Nobuyoshi Nakada)]

After installing psych 3.1.0, no error found.
What's wrong in Fedora?

$ for ruby in ruby2.{2..7}; do $ruby -rpsych -e 'p [RUBY_VERSION, Gem.load_yaml, Psych::VERSION]'; done
["2.2.10", true, "3.1.0"]
["2.3.8", true, "3.1.0"]
["2.4.10", true, "3.1.0"]
["2.5.8", true, "3.1.0"]
["2.6.6", true, "3.1.0"]
["2.7.1", true, "3.1.0"]

[vo.x (Vit Ondruch)]

What is wrong with Ruby?

$ ruby -v
ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-linux]

$ gem env
RubyGems Environment:
  - RUBYGEMS VERSION: 3.1.2
  - RUBY VERSION: 2.7.1 (2020-03-31 patchlevel 83) [x86_64-linux]
  - INSTALLATION DIRECTORY: /usr/local/lib/ruby/gems/2.7.0
  - USER INSTALLATION DIRECTORY: /builddir/.gem/ruby/2.7.0
  - RUBY EXECUTABLE: /usr/local/bin/ruby
  - GIT EXECUTABLE: /usr/bin/git
  - EXECUTABLE DIRECTORY: /usr/local/bin
  - SPEC CACHE DIRECTORY: /builddir/.gem/specs
  - SYSTEM CONFIGURATION DIRECTORY: /usr/local/etc
  - RUBYGEMS PLATFORMS:
    - ruby
    - x86_64-linux
  - GEM PATHS:
     - /usr/local/lib/ruby/gems/2.7.0
     - /builddir/.gem/ruby/2.7.0
  - GEM CONFIGURATION:
     - :update_sources => true
     - :verbose => true
     - :backtrace => false
     - :bulk_threshold => 1000
  - REMOTE SOURCES:
     - https://rubygems.org/
  - SHELL PATH:
     - /usr/local/bin/
     - /usr/bin
     - /bin
     - /usr/sbin
     - /sbin
     - /usr/local/sbin

$ gem list | grep psych
psych (default: 3.1.0)

$ gem install psych --user-install
Fetching psych-3.1.0.gem
WARNING:  You don't have /builddir/.gem/ruby/2.7.0/bin in your PATH,
	  gem executables will not run.
Building native extensions. This could take a while...
Successfully installed psych-3.1.0
Parsing documentation for psych-3.1.0
Installing ri documentation for psych-3.1.0
Done installing documentation for psych after 0 seconds
1 gem installed

$ gem list | grep psych
psych (3.1.0)

$ RUBYOPT="--disable-gems" ruby -r psych -e 'require "rubygems"; Gem.load_yaml'
/builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych.so: warning: already initialized constant Psych::Parser::ANY
/builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych.so: warning: already initialized constant Psych::Parser::UTF8
/builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych.so: warning: already initialized constant Psych::Parser::UTF16LE
/builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych.so: warning: already initialized constant Psych::Parser::UTF16BE
/builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych/class_loader.rb:7: warning: already initialized constant Psych::ClassLoader::BIG_DECIMAL
/usr/local/lib/ruby/2.7.0/psych/class_loader.rb:7: warning: previous definition of BIG_DECIMAL was here
/builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych/class_loader.rb:8: warning: already initialized constant Psych::ClassLoader::COMPLEX
/usr/local/lib/ruby/2.7.0/psych/class_loader.rb:8: warning: previous definition of COMPLEX was here
/builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych/class_loader.rb:9: warning: already initialized constant Psych::ClassLoader::DATE
/usr/local/lib/ruby/2.7.0/psych/class_loader.rb:9: warning: previous definition of DATE was here
/builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych/class_loader.rb:10: warning: already initialized constant Psych::ClassLoader::DATE_TIME
/usr/local/lib/ruby/2.7.0/psych/class_loader.rb:10: warning: previous definition of DATE_TIME was here
/builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych/class_loader.rb:11: warning: already initialized constant Psych::ClassLoader::EXCEPTION
/usr/local/lib/ruby/2.7.0/psych/class_loader.rb:11: warning: previous definition of EXCEPTION was here
/builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych/class_loader.rb:12: warning: already initialized constant Psych::ClassLoader::OBJECT
/usr/local/lib/ruby/2.7.0/psych/class_loader.rb:12: warning: previous definition of OBJECT was here
/builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych/class_loader.rb:13: warning: already initialized constant Psych::ClassLoader::PSYCH_OMAP
/usr/local/lib/ruby/2.7.0/psych/class_loader.rb:13: warning: previous definition of PSYCH_OMAP was here
/builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych/class_loader.rb:14: warning: already initialized constant Psych::ClassLoader::PSYCH_SET
/usr/local/lib/ruby/2.7.0/psych/class_loader.rb:14: warning: previous definition of PSYCH_SET was here
/builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych/class_loader.rb:15: warning: already initialized constant Psych::ClassLoader::RANGE
/usr/local/lib/ruby/2.7.0/psych/class_loader.rb:15: warning: previous definition of RANGE was here
/builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych/class_loader.rb:16: warning: already initialized constant Psych::ClassLoader::RATIONAL
/usr/local/lib/ruby/2.7.0/psych/class_loader.rb:16: warning: previous definition of RATIONAL was here
/builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych/class_loader.rb:17: warning: already initialized constant Psych::ClassLoader::REGEXP
/usr/local/lib/ruby/2.7.0/psych/class_loader.rb:17: warning: previous definition of REGEXP was here
/builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych/class_loader.rb:18: warning: already initialized constant Psych::ClassLoader::STRUCT
/usr/local/lib/ruby/2.7.0/psych/class_loader.rb:18: warning: previous definition of STRUCT was here
/builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych/class_loader.rb:19: warning: already initialized constant Psych::ClassLoader::SYMBOL
/usr/local/lib/ruby/2.7.0/psych/class_loader.rb:19: warning: previous definition of SYMBOL was here
/builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych/class_loader.rb:65: warning: already initialized constant Psych::ClassLoader::CACHE
/usr/local/lib/ruby/2.7.0/psych/class_loader.rb:65: warning: previous definition of CACHE was here
/builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych/scalar_scanner.rb:9: warning: already initialized constant Psych::ScalarScanner::TIME
/usr/local/lib/ruby/2.7.0/psych/scalar_scanner.rb:9: warning: previous definition of TIME was here
/builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych/scalar_scanner.rb:12: warning: already initialized constant Psych::ScalarScanner::FLOAT
/usr/local/lib/ruby/2.7.0/psych/scalar_scanner.rb:12: warning: previous definition of FLOAT was here
/builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych/scalar_scanner.rb:17: warning: already initialized constant Psych::ScalarScanner::INTEGER
/usr/local/lib/ruby/2.7.0/psych/scalar_scanner.rb:17: warning: previous definition of INTEGER was here
/builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych/nodes/stream.rb:13: warning: already initialized constant Psych::Nodes::Stream::ANY
/usr/local/lib/ruby/2.7.0/psych/nodes/stream.rb:13: warning: previous definition of ANY was here
/builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych/nodes/stream.rb:16: warning: already initialized constant Psych::Nodes::Stream::UTF8
/usr/local/lib/ruby/2.7.0/psych/nodes/stream.rb:16: warning: previous definition of UTF8 was here
/builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych/nodes/stream.rb:19: warning: already initialized constant Psych::Nodes::Stream::UTF16LE
/usr/local/lib/ruby/2.7.0/psych/nodes/stream.rb:19: warning: previous definition of UTF16LE was here
/builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych/nodes/stream.rb:22: warning: already initialized constant Psych::Nodes::Stream::UTF16BE
/usr/local/lib/ruby/2.7.0/psych/nodes/stream.rb:22: warning: previous definition of UTF16BE was here
/builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych/nodes/sequence.rb:43: warning: already initialized constant Psych::Nodes::Sequence::ANY
/usr/local/lib/ruby/2.7.0/psych/nodes/sequence.rb:43: warning: previous definition of ANY was here
/builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych/nodes/sequence.rb:46: warning: already initialized constant Psych::Nodes::Sequence::BLOCK
/usr/local/lib/ruby/2.7.0/psych/nodes/sequence.rb:46: warning: previous definition of BLOCK was here
/builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych/nodes/sequence.rb:49: warning: already initialized constant Psych::Nodes::Sequence::FLOW
/usr/local/lib/ruby/2.7.0/psych/nodes/sequence.rb:49: warning: previous definition of FLOW was here
/builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych/nodes/scalar.rb:10: warning: already initialized constant Psych::Nodes::Scalar::ANY
/usr/local/lib/ruby/2.7.0/psych/nodes/scalar.rb:10: warning: previous definition of ANY was here
/builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych/nodes/scalar.rb:13: warning: already initialized constant Psych::Nodes::Scalar::PLAIN
/usr/local/lib/ruby/2.7.0/psych/nodes/scalar.rb:13: warning: previous definition of PLAIN was here
/builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych/nodes/scalar.rb:16: warning: already initialized constant Psych::Nodes::Scalar::SINGLE_QUOTED
/usr/local/lib/ruby/2.7.0/psych/nodes/scalar.rb:16: warning: previous definition of SINGLE_QUOTED was here
/builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych/nodes/scalar.rb:19: warning: already initialized constant Psych::Nodes::Scalar::DOUBLE_QUOTED
/usr/local/lib/ruby/2.7.0/psych/nodes/scalar.rb:19: warning: previous definition of DOUBLE_QUOTED was here
/builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych/nodes/scalar.rb:22: warning: already initialized constant Psych::Nodes::Scalar::LITERAL
/usr/local/lib/ruby/2.7.0/psych/nodes/scalar.rb:22: warning: previous definition of LITERAL was here
/builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych/nodes/scalar.rb:25: warning: already initialized constant Psych::Nodes::Scalar::FOLDED
/usr/local/lib/ruby/2.7.0/psych/nodes/scalar.rb:25: warning: previous definition of FOLDED was here
/builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych/nodes/mapping.rb:17: warning: already initialized constant Psych::Nodes::Mapping::ANY
/usr/local/lib/ruby/2.7.0/psych/nodes/mapping.rb:17: warning: previous definition of ANY was here
/builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych/nodes/mapping.rb:20: warning: already initialized constant Psych::Nodes::Mapping::BLOCK
/usr/local/lib/ruby/2.7.0/psych/nodes/mapping.rb:20: warning: previous definition of BLOCK was here
/builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych/nodes/mapping.rb:23: warning: already initialized constant Psych::Nodes::Mapping::FLOW
/usr/local/lib/ruby/2.7.0/psych/nodes/mapping.rb:23: warning: previous definition of FLOW was here
/builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych/visitors/visitor.rb:11: warning: already initialized constant Psych::Visitors::Visitor::DISPATCH
/usr/local/lib/ruby/2.7.0/psych/visitors/visitor.rb:11: warning: previous definition of DISPATCH was here
/builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych/visitors/to_ruby.rb:334: warning: already initialized constant Psych::Visitors::ToRuby::SHOVEL
/usr/local/lib/ruby/2.7.0/psych/visitors/to_ruby.rb:336: warning: previous definition of SHOVEL was here
/builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych/handler.rb:27: warning: already initialized constant Psych::Handler::OPTIONS
/usr/local/lib/ruby/2.7.0/psych/handler.rb:27: warning: previous definition of OPTIONS was here
/builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych/handler.rb:30: warning: already initialized constant Psych::Handler::EVENTS
/usr/local/lib/ruby/2.7.0/psych/handler.rb:30: warning: previous definition of EVENTS was here
Traceback (most recent call last):
	9: from -e:1:in `<main>'
	8: from /usr/local/lib/ruby/2.7.0/rubygems.rb:696:in `load_yaml'
	7: from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:72:in `require'
	6: from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:72:in `require'
	5: from /builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych.rb:24:in `<top (required)>'
	4: from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:72:in `require'
	3: from /usr/local/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:72:in `require'
	2: from /builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych/parser.rb:2:in `<top (required)>'
	1: from /builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych/parser.rb:33:in `<module:Psych>'
/builddir/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych/parser.rb:34:in `<class:Parser>': superclass mismatch for class Mark (TypeError)

Just FTR, the patch just hides this issue, it does not resolve it.

The hacks with default gems are hiding a lot of issues. It would be nice if somebody helped to make them less exceptional 1. Or the --disable-gems option should possibly be removed if it is not properly supported anymore.

[vo.x (Vit Ondruch)]

Also, I believe that call to require 'psych' should result to subsequent calls of require 'psych' to be no-op. This is apparently not the case.

[vo.x (Vit Ondruch)]

vo.x (Vit Ondruch) wrote in #note-4:

Just FTR, the patch just hides this issue, it does not resolve it.

BTW this 1 is my original RubyGems ticket which you can use to get pointers to various related issues such as 2.

[jaruga (Jun Aruga)]

What's wrong in Fedora?

Thank you for checking the issue on your environment.
In Fedora Ruby, there is no default gem psych gemspec: there is no file such as /usr/share/gems/specifications/default/psych-*.gemspec .
You can see https://github.com/rubygems/rubygems/issues/3629#issuecomment-630288731 for detail.

This is the wrong point. That's why I wrote "It's a Fedora Ruby specific case. This is not the case of Ruby built from source normally.".

Though this issue can happen in Upstream Ruby too, according to https://github.com/ruby/psych/issues/386 , I has not tried it on my environment.

And I was able to reproduce the error by Vit's way on upstream Ruby (Ruby built from the source).

$ which ruby
/usr/local/ruby-2.7.1/bin/ruby

$ ruby -v
ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-linux]

$ gem install psych --user-install

$ gem list psych

*** LOCAL GEMS ***

psych (3.1.0)


$ RUBYOPT="--disable-gems" ruby -r psych -e 'require "rubygems"; Gem.load_yaml'
...
/home/jaruga/.gem/ruby/2.7.0/gems/psych-3.1.0/lib/psych/parser.rb:34:in `<class:Parser>': superclass mismatch for class Mark (TypeError)

[vo.x (Vit Ondruch)]

BTW I don't understand how this could work:

$ for psych in psych{1..3}; do mkdir $psych; echo "puts '$psych'" > $psych/psych.rb; done

$ find .
.
./psych2
./psych2/psych.rb
./psych1
./psych1/psych.rb
./psych3
./psych3/psych.rb

$ ruby -e '(1..3).each {|psych| $:.unshift("psych#{psych}"); require "psych" }'
psych1
psych2
psych3

In my opinion, the call to require "psych" should succeed only once, no matter on the state of $LOAD_PATH. I can report this as separate ticket ...

[vo.x (Vit Ondruch)]

vo.x (Vit Ondruch) wrote in #note-8:

Actually my original intention was to try this with disabled gems and it seems Ruby itself behaves as I would expect:

$ ruby --disable-gems -e '(1..3).each {|psych| $:.unshift("psych#{psych}"); require "psych" }'
psych1

[vo.x (Vit Ondruch)]

vo.x (Vit Ondruch) wrote in #note-8:

Upstream ticket: https://github.com/rubygems/rubygems/issues/3647

[vo.x (Vit Ondruch)]

There was merged this 1 PR, which should resolve the issue even without removal of the gem call. I have not verified it myself, but the test case was modeled after my test case in comment #note-10, so I am positive about its functionality.

[jaruga (Jun Aruga)]

Is it possible to do backport the fixes to Ruby 2.7?

For Fedora Ruby, we applied the following 4 patches (merged pull-requests) to fix this issue in Fedora.

https://github.com/rubygems/rubygems/pull/3124
https://github.com/rubygems/rubygems/pull/3133
https://github.com/rubygems/rubygems/pull/3153
https://github.com/rubygems/rubygems/pull/3639

[vo.x (Vit Ondruch)]

• Backport changed from 2.5: UNKNOWN, 2.6: UNKNOWN, 2.7: UNKNOWN to 2.5: UNKNOWN, 2.6: UNKNOWN, 2.7: REQUIRED

This was fixed in master:

https://github.com/ruby/ruby/commit/a18e81d797135de6e143a600e4f4d2b00ab23bf9
https://github.com/ruby/ruby/commit/5aa5aad48aa3effce60de3559d9bd107bfc9a91d
https://github.com/ruby/ruby/commit/07dca5c02c95322a39833705d3acd87b3adb00ac
https://github.com/ruby/ruby/commit/f0f138aa5d5857fb435bfaef95201d530f6d35da
https://github.com/ruby/ruby/commit/b9d431a7b129635356d03bb0bbdb9b5060a35729

Closing this to trigger backport request.

[vo.x (Vit Ondruch)]

• Status changed from Open to Closed

[hsbt (Hiroshi SHIBATA)]

jaruga (Jun Aruga) wrote in #note-12:

Is it possible to do backport the fixes to Ruby 2.7?

For Fedora Ruby, we applied the following 4 patches (merged pull-requests) to fix this issue in Fedora.

https://github.com/rubygems/rubygems/pull/3124
https://github.com/rubygems/rubygems/pull/3133
https://github.com/rubygems/rubygems/pull/3153
https://github.com/rubygems/rubygems/pull/3639

After releasing RG 3.1.5 contains the above patches, We will request to merge it to ruby_2_7 branch.

[hsbt (Hiroshi SHIBATA)]

• Assignee set to hsbt (Hiroshi SHIBATA)

[hsbt (Hiroshi SHIBATA)]

I released RubyGems 3.1.6 https://blog.rubygems.org/2021/01/26/3.1.6-released.html and create the backport request for this into ruby_2_7 branch.

[nagachika (Tomoyuki Chikanaga)]

• Backport changed from 2.5: UNKNOWN, 2.6: UNKNOWN, 2.7: REQUIRED to 2.5: UNKNOWN, 2.6: UNKNOWN, 2.7: DONE

I have backported "Merge RubyGems-3.1.6" at https://github.com/ruby/ruby/commit/99b4f20fd65075d2d9ee02f8c7a6b5d6a914ce31. I think the changeset may be related with this issue too.