Feature #17873: Update of default gems in Ruby 3.1

• Status: Closed
• Assignee: hsbt (Hiroshi SHIBATA)
• Target version: -

[hsbt (Hiroshi SHIBATA)]

I promoted many standard libraries to the default gems. You can see them at https://stdgems.org/#default-gems-ruby-301

I propose the following libraries to promote the bundled gems from the default gems.

matrix
prime
net/ftp
net/smtp
net/pop
net/imap

They have primary maintainer in the canonical repository in github. So, they can release the new version with their convenience.

@shugo (Shugo Maeda) @marcandre (Marc-Andre Lafortune) Any thought?

And I also propose to extract the following libraries from the default gems.

tracer
dbm
gdbm

They are no longer actively maintained and not widely used today.

[hsbt (Hiroshi SHIBATA)]

• Tracker changed from Bug to Feature
• Backport deleted (2.6: UNKNOWN, 2.7: UNKNOWN, 3.0: UNKNOWN)

[Eregon (Benoit Daloze)]

And I also propose to extract the following libraries from the default gems.

Those are already default gems.
So I guess you mean no longer including them with Ruby, and gem install NAME would be needed to use them?

Seems fine in general.

I'm hesitant about fcntl since it's so small (just a bunch of constants) and I think it doesn't make so much sense to have that as a separate gem (it's already a default gem, though I don't see the point of that).

[hsbt (Hiroshi SHIBATA)]

So I guess you mean no longer including them with Ruby, and gem install NAME would be needed to use them?

Yes. I meant to remove them ruby repository and release.

[byroot (Jean Boussier)]

They are no longer actively maintained and not widely used today.

mutex_m

In my opinion mutex_m is widely used. Even Rails uses it: https://github.com/rails/rails/search?q=mutex_m

[hsbt (Hiroshi SHIBATA)]

• Description updated (diff)

In my opinion mutex_m is widely used. Even Rails uses it: https://github.com/rails/rails/search?q=mutex_m

Thanks for your information. I agreed your opinion and removed mutex_m from this proposal.

[universato (Yoshimine Sato)]

I would like prime library to remain in the default gems or the bundled gems.
I do competitive programming as a hobby, and I sometimes use it and find it useful.
Since external libraries are usually not available for competitive programming, I would like prime library to remain in internal gems.

Some Japanese competitors have also mentioned prime library as one of the advantages of Ruby.
Rubyで競プロするときのTips - ARMERIA
【AtCoder】Rubyで競プロに入門する - Qiita
【不安解消】Rubyでも競技プログラミングで問題が解ける→茶色に行けました! | RyuCoding
Rubyで競プロをしていた時に使っていたライブラリを供養する - Qiita

[hsbt (Hiroshi SHIBATA)]

• Description updated (diff)
• Status changed from Open to Assigned

I would like prime library to remain in the default gems or the bundled gems.

Thanks. I moved prime to the bundled gems list.

And I removed fcntl from this proposal. It's also widely used in other gems.

[marcandre (Marc-Andre Lafortune)]

Thanks, I agree about prime.

I imagine matrix users are limited and will be ok. I'm not sure about ostruct.

Could we consider to bundle them in next major version but with some warning? If you "update" the gem then the conversion warning would not be there. This could ease conversion.

[shugo (Shugo Maeda)]

I agree about net/ftp and net/imap.

[hsbt (Hiroshi SHIBATA)]

• Description updated (diff)

[hsbt (Hiroshi SHIBATA)]

• Description updated (diff)

I'm not sure about ostruct.

Thanks, I'm also aware this. So, I removed ostruct from this proposal.

[hsbt (Hiroshi SHIBATA)]

• Description updated (diff)

I added dbm and gdbm to removal list with other commiter's opinion.

[knu (Akinori MUSHA)]

I don't see a good reason to remove the syslog library from ruby.
It is so small, simple, and almost maintenance-free because the syslog facility is unlikely to change anymore.

Syslog has long been a standard logging facility in Unix/Linux and I believe it is good for ruby to have access to it out of the box.
I can imagine it is used by many admins in their system scripts.

Also, there seem to be many gems that load syslog.

knu@aluminium:~$ gem-codesearch "^ *require ['\"]syslog[/\"']"  | ruby -ne 'puts $_.split("/")[3]' | sort -u | wc -l
259

[hsbt (Hiroshi SHIBATA)]

• Description updated (diff)

@knu (Akinori MUSHA) I have no strong opinion to remove syslog in Ruby 3.1. I withdraws it from this proposal.

[hsbt (Hiroshi SHIBATA)]

I applied this proposal at https://github.com/ruby/ruby/pull/4530

But I keep to open this until releasing preview or rc version of Ruby 3.1.

[byroot (Jean Boussier)]

Just for the record:

• The matrix removal broke the capybara gem: https://github.com/teamcapybara/capybara/pull/2468
• The net-smtp remove broke the mail gem: https://github.com/mikel/mail/pull/1439

Both are Rails dependencies.

Doesn't mean it should be reverted, I personally think the change make sense, but I figured it should be reported here.

[byroot (Jean Boussier)]

Actually, there's a bit of a problem. If you add a default gem to the gemspec it cause double loading issues, e.g. with mail, net-smtp pulls digest:

/opt/rubies/2.7.2/lib/ruby/2.7.0/digest.rb:6: warning: previous definition of REQUIRE_MUTEX was here
~/.gem/ruby/2.7.2/gems/digest-3.0.0/lib/digest.rb:8: warning: method redefined; discarding old const_missing

So I'm not sure what the path forward is for these gems.

[deivid (David Rodríguez)]

This sometimes happens when rubygems loads a default gem too early before bundler has been setup. Then when user code loads the actual gem it uses the $LOAD_PATH setup by bundler instead of the default one, causing the double load. Up until now we always found ways to fix these issues by making usages of default gems inside rubygems as lazy as possible, or by vendoring copies of the default gems that we need under our namespace so that we don't interfere with user's choice, but it seems like we're still loading digest too early.

I'll have a look at this.

[byroot (Jean Boussier)]

I'll have a look at this.

Thank you!

I used digest as an example, but there is also net/protocol: https://buildkite.com/rails/rails/builds/77709#8f4e417a-153d-403c-9ad0-6579b3c04adf/999-1010

[deivid (David Rodríguez)]

The net-protocol one seems similar to: https://github.com/ruby/net-imap/issues/16.

[marcandre (Marc-Andre Lafortune)]

byroot (Jean Boussier) wrote in #note-16:

Just for the record:

• The matrix removal broke the capybara gem: https://github.com/teamcapybara/capybara/pull/2468
• The net-smtp remove broke the mail gem: https://github.com/mikel/mail/pull/1439

Both are Rails dependencies.

Doesn't mean it should be reverted, I personally think the change make sense, but I figured it should be reported here.

Thanks.

I'll repeat: could we consider to still bundle these gems in next major version but with some warning? If you "update" the gem then the conversion warning would not be there. This could ease conversion.

[Eregon (Benoit Daloze)]

@byroot (Jean Boussier) @deivid (David Rodríguez) I think it would best to track this double inclusion problem as an issue on https://github.com/rubygems/rubygems/issues, could you file one?

[deivid (David Rodríguez)]

Thanks for the remainder, I forgot. I did try to repro this at the time by simulating locally what Rails CI does, but I couldn't trigger the double load. Best to open an issue at the rubygems repository with repro steps.

[byroot (Jean Boussier)]

but I couldn't trigger the double load. Best to open an issue at the rubygems repository with repro steps.

So I was able to repro, but I can confirm that it is fixed on latest rubygems/bundler.

The problem is that lots of users don't keep both of these up to date, so that will cause lots of confusion.

[deivid (David Rodríguez)]

Great to hear that it's fixed!

Regarding your concern, this is a change for ruby 3.1, so as long as ruby 3.1 provides a fixed version of bundler & rubygems, it should be ok, right?

[byroot (Jean Boussier)]

so as long as ruby 3.1 provides a fixed version of bundler & rubygems, it should be ok, right?

No, the problem is that to be compatible with 3.1, these gems will have to specify these dependencies. But anyone using gems defining these dependencies but still using older rubygems/bundler will have the double loading issue.

So there's really nothing we can do :/, people will have to upgrade their rubygems/bundler.

[deivid (David Rodríguez)]

I see. It doesn't seem like a big deal to me.

[Eregon (Benoit Daloze)]

Doesn't it mean that in addition to the double-loading warnings (e.g., when using Rails) it might also break on current Ruby releases with the version of RubyGems they ship? (due to the two loaded versions creating some incompatibility)
It's also likely hard to guess that double loading means "update to latest RubyGems and Bundler" (they can be other causes for it).

In a Gemfile, it would be possible to specify those gems are only needed for Ruby 3.1+, but in a gemspec that does not seem possible unfortunately.

@byroot (Jean Boussier) Which Ruby versions did you see was affected? If it's only unsupported Ruby versions at least it'd be a smaller concern.

FWIW Rails main currently does this, the same can be done for generated applications's Gemfile.
https://github.com/rails/rails/blob/acee501edd0274acb1b073aac1ac41a3181aafdd/Gemfile#L172-L182
It is not pretty, but if it avoids double-loading maybe it is better than specifying the dependencies in the gem and potentially breaking running Rails on many Ruby releases.

[deivid (David Rodríguez)]

Yeah, technically it could cause issues, that's why ruby prints a warning. But I've never seen an actual issue in real life, just users bothered by the warning. Concerned users usually look for the root cause, and eventually figure out they need to upgrade rubygems/bundler or report the issue to the affected repository. That's how I've been normally finding out about and fixing these issues in the past years since the gemification of the stdlib started.

[byroot (Jean Boussier)]

Hum, actually I used a docker image to try to pinpoint when it was fixed, and I'm no longer sure it is fixed, or at least another issue cropped up:

# frozen_string_literal: true

require "bundler/inline"

gemfile(true) do
  source "https://rubygems.org"
  gem "mail", github: "Shopify/mail", branch: "net-smtp-dependency"
end

require "mail"

$ docker run -v /tmp:/tmp -it ruby:3.0.1 bash -c 'gem update --system --no-document && gem install --no-document bundler && ruby /tmp/repro-double-loading.rb'
....

Installing net-imap 0.2.2
Using mail 2.8.0.edge from https://github.com/Shopify/mail.git (at net-smtp-dependency@58704be)
/usr/local/bundle/gems/bundler-2.2.22/lib/bundler/runtime.rb:302:in `check_for_activated_spec!': You have already activated net-protocol 0.1.0, but your Gemfile requires net-protocol 0.1.1. Since net-protocol is a default gem, you can either remove your dependency on it or try updating to a newer version of bundler that supports net-protocol as a default gem. (Gem::LoadError)

So now I get a hard failure. Not sure why it works on my host.

[deivid (David Rodríguez)]

Alright, thanks. I'll have a look at that, then.

[deivid (David Rodríguez)]

At first glance, it looks specific to bundler/inline, when installing gems and using gems happens in the same process. I'll fix it by vendoring net/http and its dependencies under the Bundler:: namespace like we do with the other pure ruby libraries that bundler depends on.

[byroot (Jean Boussier)]

I figured the digest issue. It only happens if you have a git source gem as well in the Gemfile, because the git source does a MD5 of the repository URL.

I submitted a PR that could fix the problem: https://github.com/rubygems/rubygems/pull/4989

[hsbt (Hiroshi SHIBATA)]

I prepared versions for Ruby 3.1.0-preview1. But I keep the version numbers of the following libraries.

• RubyGems/Bundler: I coordinate the version numbering with @deivid (David Rodríguez)
• Reline, Irb: @aycabta (aycabta .)
• BIgDecimal: @mrkn (Kenta Murata)
• Fiddle, CSV, StringScanner: @kou (Kouhei Sutou)
• IpAddr, Set: @knu (Akinori MUSHA)
• OpenSSL: @rhenium (Kazuki Yamaguchi)
• DidYouMean: @yuki24 (Yuki Nishijima)
• Erb: @k0kubun (Takashi Kokubun)

Can maintainers update the version number of them before releasing Ruby 3.1.0preview1?

[kou (Kouhei Sutou)]

OK. I'll do.

[kou (Kouhei Sutou)]

Done: Fiddle, CSV, StringScanner

[hsbt (Hiroshi SHIBATA)]

• Status changed from Assigned to Closed

Ruby 3.1 released. Thanks to develop and release for this.

[Eregon (Benoit Daloze)]

Note: Whenever a gem is extracted from stdlib, we should be careful the gem can be installed on older Ruby versions, otherwise it's not possible for a library to depend on this newer gem via RubyGems (and it has to, otherwise require fails when using Bundler).
See https://github.com/ruby/io-wait/pull/9#issuecomment-1026883889

[retro (Josef Šimánek)]

Eregon (Benoit Daloze) wrote in #note-38:

Note: Whenever a gem is extracted from stdlib, we should be careful the gem can be installed on older Ruby versions, otherwise it's not possible for a library to depend on this newer gem via RubyGems (and it has to, otherwise require fails when using Bundler).
See https://github.com/ruby/io-wait/pull/9#issuecomment-1026883889

In your comment, you're specifically asking that gem can be installed on EOL Ruby. I think that's worth discussion.