Project

General

Profile

Actions

Feature #17873

open

Update of default gems in Ruby 3.1

Added by hsbt (Hiroshi SHIBATA) 5 months ago. Updated 4 days ago.

Status:
Assigned
Priority:
Normal
Target version:
-
[ruby-core:103912]

Description

I promoted many standard libraries to the default gems. You can see them at https://stdgems.org/#default-gems-ruby-301

I propose the following libraries to promote the bundled gems from the default gems.

matrix
prime
net/ftp
net/smtp
net/pop
net/imap

They have primary maintainer in the canonical repository in github. So, they can release the new version with their convenience.

shugo (Shugo Maeda) marcandre (Marc-Andre Lafortune) Any thought?

And I also propose to extract the following libraries from the default gems.

tracer
dbm
gdbm

They are no longer actively maintained and not widely used today.

Actions #1

Updated by hsbt (Hiroshi SHIBATA) 5 months ago

  • Backport deleted (2.6: UNKNOWN, 2.7: UNKNOWN, 3.0: UNKNOWN)
  • Tracker changed from Bug to Feature

Updated by Eregon (Benoit Daloze) 5 months ago

And I also propose to extract the following libraries from the default gems.

Those are already default gems.
So I guess you mean no longer including them with Ruby, and gem install NAME would be needed to use them?

Seems fine in general.

I'm hesitant about fcntl since it's so small (just a bunch of constants) and I think it doesn't make so much sense to have that as a separate gem (it's already a default gem, though I don't see the point of that).

Updated by hsbt (Hiroshi SHIBATA) 5 months ago

So I guess you mean no longer including them with Ruby, and gem install NAME would be needed to use them?

Yes. I meant to remove them ruby repository and release.

Updated by byroot (Jean Boussier) 5 months ago

They are no longer actively maintained and not widely used today.

mutex_m

In my opinion mutex_m is widely used. Even Rails uses it: https://github.com/rails/rails/search?q=mutex_m

Updated by hsbt (Hiroshi SHIBATA) 5 months ago

  • Description updated (diff)

In my opinion mutex_m is widely used. Even Rails uses it: https://github.com/rails/rails/search?q=mutex_m

Thanks for your information. I agreed your opinion and removed mutex_m from this proposal.

Updated by universato (Yoshimine Sato) 5 months ago

I would like prime library to remain in the default gems or the bundled gems.
I do competitive programming as a hobby, and I sometimes use it and find it useful.
Since external libraries are usually not available for competitive programming, I would like prime library to remain in internal gems.

Some Japanese competitors have also mentioned prime library as one of the advantages of Ruby.
Rubyで競プロするときのTips - ARMERIA
【AtCoder】Rubyで競プロに入門する - Qiita
【不安解消】Rubyでも競技プログラミングで問題が解ける→茶色に行けました! | RyuCoding
Rubyで競プロをしていた時に使っていたライブラリを供養する - Qiita

Updated by hsbt (Hiroshi SHIBATA) 5 months ago

  • Status changed from Open to Assigned
  • Description updated (diff)

I would like prime library to remain in the default gems or the bundled gems.

Thanks. I moved prime to the bundled gems list.

And I removed fcntl from this proposal. It's also widely used in other gems.

Updated by marcandre (Marc-Andre Lafortune) 5 months ago

Thanks, I agree about prime.

I imagine matrix users are limited and will be ok. I'm not sure about ostruct.

Could we consider to bundle them in next major version but with some warning? If you "update" the gem then the conversion warning would not be there. This could ease conversion.

Updated by shugo (Shugo Maeda) 5 months ago

I agree about net/ftp and net/imap.

Actions #10

Updated by hsbt (Hiroshi SHIBATA) 5 months ago

  • Description updated (diff)

Updated by hsbt (Hiroshi SHIBATA) 5 months ago

  • Description updated (diff)

I'm not sure about ostruct.

Thanks, I'm also aware this. So, I removed ostruct from this proposal.

Updated by hsbt (Hiroshi SHIBATA) 5 months ago

  • Description updated (diff)

I added dbm and gdbm to removal list with other commiter's opinion.

Updated by knu (Akinori MUSHA) 5 months ago

I don't see a good reason to remove the syslog library from ruby.
It is so small, simple, and almost maintenance-free because the syslog facility is unlikely to change anymore.

Syslog has long been a standard logging facility in Unix/Linux and I believe it is good for ruby to have access to it out of the box.
I can imagine it is used by many admins in their system scripts.

Also, there seem to be many gems that load syslog.

knu@aluminium:~$ gem-codesearch "^ *require ['\"]syslog[/\"']"  | ruby -ne 'puts $_.split("/")[3]' | sort -u | wc -l
259

Updated by hsbt (Hiroshi SHIBATA) 5 months ago

  • Description updated (diff)

knu (Akinori MUSHA) I have no strong opinion to remove syslog in Ruby 3.1. I withdraws it from this proposal.

Updated by hsbt (Hiroshi SHIBATA) 5 months ago

I applied this proposal at https://github.com/ruby/ruby/pull/4530

But I keep to open this until releasing preview or rc version of Ruby 3.1.

Updated by byroot (Jean Boussier) 5 months ago

Just for the record:

Both are Rails dependencies.

Doesn't mean it should be reverted, I personally think the change make sense, but I figured it should be reported here.

Updated by byroot (Jean Boussier) 5 months ago

Actually, there's a bit of a problem. If you add a default gem to the gemspec it cause double loading issues, e.g. with mail, net-smtp pulls digest:

/opt/rubies/2.7.2/lib/ruby/2.7.0/digest.rb:6: warning: previous definition of REQUIRE_MUTEX was here
~/.gem/ruby/2.7.2/gems/digest-3.0.0/lib/digest.rb:8: warning: method redefined; discarding old const_missing

So I'm not sure what the path forward is for these gems.

Updated by deivid (David Rodríguez) 5 months ago

This sometimes happens when rubygems loads a default gem too early before bundler has been setup. Then when user code loads the actual gem it uses the $LOAD_PATH setup by bundler instead of the default one, causing the double load. Up until now we always found ways to fix these issues by making usages of default gems inside rubygems as lazy as possible, or by vendoring copies of the default gems that we need under our namespace so that we don't interfere with user's choice, but it seems like we're still loading digest too early.

I'll have a look at this.

Updated by byroot (Jean Boussier) 5 months ago

I'll have a look at this.

Thank you!

I used digest as an example, but there is also net/protocol: https://buildkite.com/rails/rails/builds/77709#8f4e417a-153d-403c-9ad0-6579b3c04adf/999-1010

Updated by marcandre (Marc-Andre Lafortune) 5 months ago

byroot (Jean Boussier) wrote in #note-16:

Just for the record:

Both are Rails dependencies.

Doesn't mean it should be reverted, I personally think the change make sense, but I figured it should be reported here.

Thanks.

I'll repeat: could we consider to still bundle these gems in next major version but with some warning? If you "update" the gem then the conversion warning would not be there. This could ease conversion.

Updated by Eregon (Benoit Daloze) 4 months ago

byroot (Jean Boussier) deivid (David Rodríguez) I think it would best to track this double inclusion problem as an issue on https://github.com/rubygems/rubygems/issues, could you file one?

Updated by deivid (David Rodríguez) 3 months ago

Thanks for the remainder, I forgot. I did try to repro this at the time by simulating locally what Rails CI does, but I couldn't trigger the double load. Best to open an issue at the rubygems repository with repro steps.

Updated by byroot (Jean Boussier) 3 months ago

but I couldn't trigger the double load. Best to open an issue at the rubygems repository with repro steps.

So I was able to repro, but I can confirm that it is fixed on latest rubygems/bundler.

The problem is that lots of users don't keep both of these up to date, so that will cause lots of confusion.

Updated by deivid (David Rodríguez) 3 months ago

Great to hear that it's fixed!

Regarding your concern, this is a change for ruby 3.1, so as long as ruby 3.1 provides a fixed version of bundler & rubygems, it should be ok, right?

Updated by byroot (Jean Boussier) 3 months ago

so as long as ruby 3.1 provides a fixed version of bundler & rubygems, it should be ok, right?

No, the problem is that to be compatible with 3.1, these gems will have to specify these dependencies. But anyone using gems defining these dependencies but still using older rubygems/bundler will have the double loading issue.

So there's really nothing we can do :/, people will have to upgrade their rubygems/bundler.

Actions #27

Updated by deivid (David Rodríguez) 3 months ago

I see. It doesn't seem like a big deal to me.

Updated by Eregon (Benoit Daloze) 3 months ago

Doesn't it mean that in addition to the double-loading warnings (e.g., when using Rails) it might also break on current Ruby releases with the version of RubyGems they ship? (due to the two loaded versions creating some incompatibility)
It's also likely hard to guess that double loading means "update to latest RubyGems and Bundler" (they can be other causes for it).

In a Gemfile, it would be possible to specify those gems are only needed for Ruby 3.1+, but in a gemspec that does not seem possible unfortunately.

byroot (Jean Boussier) Which Ruby versions did you see was affected? If it's only unsupported Ruby versions at least it'd be a smaller concern.

FWIW Rails main currently does this, the same can be done for generated applications's Gemfile.
https://github.com/rails/rails/blob/acee501edd0274acb1b073aac1ac41a3181aafdd/Gemfile#L172-L182
It is not pretty, but if it avoids double-loading maybe it is better than specifying the dependencies in the gem and potentially breaking running Rails on many Ruby releases.

Updated by deivid (David Rodríguez) 3 months ago

Yeah, technically it could cause issues, that's why ruby prints a warning. But I've never seen an actual issue in real life, just users bothered by the warning. Concerned users usually look for the root cause, and eventually figure out they need to upgrade rubygems/bundler or report the issue to the affected repository. That's how I've been normally finding out about and fixing these issues in the past years since the gemification of the stdlib started.

Updated by byroot (Jean Boussier) 3 months ago

Hum, actually I used a docker image to try to pinpoint when it was fixed, and I'm no longer sure it is fixed, or at least another issue cropped up:

# frozen_string_literal: true

require "bundler/inline"

gemfile(true) do
  source "https://rubygems.org"
  gem "mail", github: "Shopify/mail", branch: "net-smtp-dependency"
end

require "mail"
$ docker run -v /tmp:/tmp -it ruby:3.0.1 bash -c 'gem update --system --no-document && gem install --no-document bundler && ruby /tmp/repro-double-loading.rb'
....

Installing net-imap 0.2.2
Using mail 2.8.0.edge from https://github.com/Shopify/mail.git (at net-smtp-dependency@58704be)
/usr/local/bundle/gems/bundler-2.2.22/lib/bundler/runtime.rb:302:in `check_for_activated_spec!': You have already activated net-protocol 0.1.0, but your Gemfile requires net-protocol 0.1.1. Since net-protocol is a default gem, you can either remove your dependency on it or try updating to a newer version of bundler that supports net-protocol as a default gem. (Gem::LoadError)

So now I get a hard failure. Not sure why it works on my host.

Updated by deivid (David Rodríguez) 3 months ago

Alright, thanks. I'll have a look at that, then.

Updated by deivid (David Rodríguez) 3 months ago

At first glance, it looks specific to bundler/inline, when installing gems and using gems happens in the same process. I'll fix it by vendoring net/http and its dependencies under the Bundler:: namespace like we do with the other pure ruby libraries that bundler depends on.

Updated by byroot (Jean Boussier) 4 days ago

I figured the digest issue. It only happens if you have a git source gem as well in the Gemfile, because the git source does a MD5 of the repository URL.

I submitted a PR that could fix the problem: https://github.com/rubygems/rubygems/pull/4989

Actions

Also available in: Atom PDF