Feature #19351: Promote bundled gems at Ruby 3.3

• Status: Closed
• Assignee: hsbt (Hiroshi SHIBATA)
• Target version: -

[hsbt (Hiroshi SHIBATA)]

In Ruby 3.2, the default gems and bundled gems are changed only adding syntax_suggest. I and some committers are considering promote default gems to bundled gems again for Ruby 3.3+.

We hope to keep the current developer experience with dependency resolution and ignore the additional work like "Put gem "xxx" into your Gemfile" for developers.

Proposal for Ruby 3.3.0-preview2

I propose the following libraries will promote default gems to bundled gems at Ruby 3.3.0-preview2

abbrev
getoptlong
observable
resolv-replace
rinda
nkf (C-ext)
syslog (C-ext)
base64
drb
mutex_m
csv

Other default gems depends on make test-all or other standard libraries. It's hard to promote in this time. And I submit a PoC of bypass feature for Bundler and bundled gems: https://github.com/rubygems/rubygems/pull/6811

This feature add load_paths defined by Gem.bundled_gems that is pair list of Gem name and version under the Bundler environment. This mean user can require bundled gems like csv without gem "csv" on Gemfile.

And we need to warn like "'csv' is loaded without Gemfile, add "gem 'csv'" in your Gemfile" in require or other place. I have no idea how notice this yet.

---

Proposal

I propose the following libraries will promote default gems to bundled gems at Ruby 3.3. They are not the dependencies of Rails and RubyGems/Bundler.

abbrev
getoptlong
observable
resolv-replace
rinda
nkf (C-ext)
syslog (C-ext)

Update: I removed optparse and un from above list. Because they are used by Ruby build process.

optparse
un

Update 2: I also removed the following libraries. resolv and fcntl are used by test of Ruby internal like test_io.rb. And we don't have built process of C extension at Windows platform. I gave up to extract win32ole in this time.

resolv
fcntl (C-ext)
win32ole (C-ext)

Additional works

I also propose to promote rails dependencies without rubygems/bundler deps:

base64
drb
mutex_m

Update: delegate is used by tempfile. We need to keep delegate as default gems for build process. and I added reline into above list.

Update 2: The following libraries used by tests of ruby/ruby and other libraries like psych. I need to remove their dependency from ruby/ruby.

benchmark
forwardable
ipaddr
irb
reline
ostruct
rdoc
singleton
tsort
weakref
bigdecimal (C-ext)
date(datetime) (C-ext)

And gems maintained by @kou (Kouhei Sutou)

csv

Following gems also maintained by @kou (Kouhei Sutou), but they are used on RubyGems/Bundler or MJIT. Maybe, We couldn't promote them because RubyGems/Bundler couldn't bundle C-ext gems.

fiddle (C-ext): used by MJIT
stringio (C-ext) used by RubyGems/Bundler
strscan (C-ext) used by RubyGems/Bundler

Follow-up feature

But if we promote them to bundled gems, many of users need to add like gem "csv" into their Gemfile. I'm considering avoiding this situation.

Can we introduce the specific feature of bundled gems to RubyGems or Bundler? Example, bundler have allowed list for bundled gems. So, listed gems could be require without Gemfile under the bundle exec.

[vo.x (Vit Ondruch)]

But if we promote them to bundled gems, many of users need to add gem "csv" into their Gemfile. I'm considering to avoid this situation.

Please let gems to be gems. Please let everybody fix their Gemfiles. Don't make exceptions. There is nothing wrong with specifying dependencies.

[hsbt (Hiroshi SHIBATA)]

• Description updated (diff)

[vo.x (Vit Ondruch)]

vo.x (Vit Ondruch) wrote in #note-1:

But if we promote them to bundled gems, many of users need to add gem "csv" into their Gemfile. I'm considering to avoid this situation.

Please let gems to be gems. Please let everybody fix their Gemfiles. Don't make exceptions. There is nothing wrong with specifying dependencies.

BTW I don't think there would be that much fixing needed after all. E.g. if Rails console depends on IRB and Rails specified the dependency on IRB, Rails users would not need to do anything out of ordinary. But of course, the could be some exceptional handling for IRB. The only differences would be, that they would get faster IRB updates, which they won't notice ATM.

[hsbt (Hiroshi SHIBATA)]

• Description updated (diff)

[hsbt (Hiroshi SHIBATA)]

I update rails dependencies.

@vo.x (Vit Ondruch) Thanks for your comment. Maybe we should remove irb from this list.

I welcome feedback for these list and additional gem proposal like cgi, erb and others.

[byroot (Jean Boussier)]

Yeah, a few of these will need to be handled by rubygems/bundler.

I believe that for instance strscan is used to parse the Gemfile.lock. Historically Bundler simply vendored the gems it needs, but not sure how it could do that since it has a C-ext part.

I assume you notified bundler people? What do they think?

[Eregon (Benoit Daloze)]

vo.x (Vit Ondruch) wrote in #note-3:

BTW I don't think there would be that much fixing needed after all.

As we have seen with the mail gem, it can be a huge pain to move from default gem to bundled gem (in that case for net-* gems).

What is the advantage to move from default gem to bundled gem?
In both cases it's shipped with Ruby and they can be updated.
So it seems the only difference is the disadvantage to break every script using one of these libraries and Bundler by forcing to add them in the Gemfile (which typically doesn't work for older Ruby versions so adds many complications).

Please also consider the concerns in #18567.
Quoting them here:

There are however multiple unwanted side effects of this:

1. Removing gems from stdlib (e.g., #17873) is a breaking change, which makes upgrading to a new Ruby version more difficult.
   I think this should only be done if there is a clear gain.
   Being a default gem is already enough to fix a security issue without a CRuby release.
2. When any gem depends on a default gem, it tends to break on all Ruby implementations except CRuby, and for older Ruby versions.

1. is the same as I just said, but makes it clear this change hurts adopting new Ruby versions.

2. is a very important requirement if we do this, to ensure it works on JRuby/TruffleRuby before publishing the gem.
   Replacing the implementation of a default gem (on JRuby/TruffleRuby) is only an issue if people depend on it explicitly. But for bundled gems it doesn't work at all.
   So we need to ensure those gems work on JRuby/TruffleRuby, or we need time so JRuby/TruffleRuby devs can fix that before the gem is published/people add a dependency on it.

[Eregon (Benoit Daloze)]

• Related to Bug #18567: Depending on default gems in stdlib gems when not needed considered harmful added

[kou (Kouhei Sutou)]

I'm OK with gems maintained by me.

[hsbt (Hiroshi SHIBATA)]

• Status changed from Open to Assigned
• Assignee set to hsbt (Hiroshi SHIBATA)

I believe that for instance strscan is used to parse the Gemfile.lock. Historically Bundler simply vendored the gems it needs, but not sure how it could do that since it has a C-ext part.

Thanks. I also confirm that RubyGems/Bundler uses stringio that is C-ext. I may remove them from this proposal.

[hsbt (Hiroshi SHIBATA)]

• Description updated (diff)

[hsbt (Hiroshi SHIBATA)]

• Description updated (diff)

[hsbt (Hiroshi SHIBATA)]

• Related to Feature #16963: Remove English.rb from Ruby 2.8/3.0 added

[hsbt (Hiroshi SHIBATA)]

• Tracker changed from Bug to Feature
• Backport deleted (2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN)

[hsbt (Hiroshi SHIBATA)]

• Related to Bug #18169: Local copies of gemified libraries are being released out of sync with their gems added

[hsbt (Hiroshi SHIBATA)]

What is the advantage to move from default gem to bundled gem?

It makes to leave out-of-sync status with published gem like #18169. I'm checking ALL diffs and commits of ruby/* repos every day for resolving this issue after #18169 requests. Example for https://github.com/ruby/ruby/pull/7025.

And we can add new maintainer easily different with ruby/ruby repo. In fact, net-smtp, net-imap, rexml and etc are maintained and released by maintainer's convenience.

[hsbt (Hiroshi SHIBATA)]

• Description updated (diff)

[Eregon (Benoit Daloze)]

hsbt (Hiroshi SHIBATA) wrote in #note-16:

It makes to leave out-of-sync status with published gem like #18169. I'm checking ALL diffs and commits of ruby/* repos every day for resolving this issue after #18169 requests. Example for https://github.com/ruby/ruby/pull/7025.

My understanding is this is fully automated now, i.e., a commit to the ruby/some_default_gem repo is immediately applied to ruby/ruby and vice-versa.
So it seems imopossible to get out of sync, isn't it?

And we can add new maintainer easily different with ruby/ruby repo. In fact, net-smtp, net-imap, rexml and etc are maintained and released by maintainer's convenience.

The only constraint there is to have a new version number at ruby/ruby release time for those gems, right?
That can be done by just bumping the patch version or so of such gems in ruby/ruby, isn't it?

I think the pain of everyone using Ruby to change these gems from default to bundled gems is not worth those advantages.
At least things which don't use Bundler should not be impacted, e.g. ruby -rbenchmark -e 'p Benchmark.realtime { ... }'.

Also it seems the main motivation is #18169.
I think #18567 is far more problematic in practice for other Ruby implementations.
So like I said in https://bugs.ruby-lang.org/issues/18567#note-30, if we move anything to default/bundled gems, please make sure it works on JRuby/TruffleRuby by adding them in CI and defaulting to stdlib unless it works as-is (details described there).

[hsbt (Hiroshi SHIBATA)]

So it seems imopossible to get out of sync, isn't it?

No. some of commits conflicts usually. and some of people push ruby/ruby directly for fixing CI failure, not ruby/* repo. So, we are always out-of-sync status without my work.

I think the pain of everyone using Ruby to change these gems from default to bundled gems is not worth those advantages.

Because you didn't work release work of CRuby. Release team easily released new version of CGI without Ruby releases. Release team always spent over 6 hours with 5-6 people for security release of Ruby. We really hope to leave this hard work.

if we move anything to default/bundled gems, please make sure it works on JRuby/TruffleRuby by adding them in CI

I already work to merge JRuby code with JRuby team. I welcome your patch for TruffleRuby.

[Eregon (Benoit Daloze)]

hsbt (Hiroshi SHIBATA) wrote in #note-19:

No. some of commits conflicts usually. and some of people push ruby/ruby directly for fixing CI failure, not ruby/* repo. So, we are always out-of-sync status without my work.

I remember some commit I made, maybe in ruby/some_gem being immediately committed to ruby/ruby too.
So maybe that automatic cherry-picking is only in one direction, not the other, and maybe only for some gems and not all?

Because you didn't work release work of CRuby. Release team easily released new version of CGI without Ruby releases. Release team always spent over 6 hours with 5-6 people for security release of Ruby. We really hope to leave this hard work.

Correct me if I'm wrong, but to fix a security issue in a bundled or default gem it's the same, there is no need to release CRuby, only to release the gem, isn't it?

Maybe a CRuby release is considered for some default gems/big security issue because it's harder for users to find out about the security issue (e.g., dependabot won't tell them)?
But then they would also need to find about the new CRuby release, given it's the same channel (https://www.ruby-lang.org/ announcements) it doesn't seem to help much, except for ruby installed by package manager maybe (rare for Ruby applications).

[hsbt (Hiroshi SHIBATA)]

So maybe that automatic cherry-picking is only in one direction, not the other, and maybe only for some gems and not all?

I prepared auto-sync for all of default gems. But it's sometimes failed and ruby/ruby to ruby/* is not available for it. I always pick them and manually push all of repos.

[hsbt (Hiroshi SHIBATA)]

• Description updated (diff)

[nobu (Nobuyoshi Nakada)]

• Description updated (diff)

[hsbt (Hiroshi SHIBATA)]

• Description updated (diff)

We discussed about this at #19357

@mame (Yusuke Endoh) suggested It's hard to research affected gems that are promoted bundled gems from default gems. Ex. resolve-replace, nkf are used by prod application in the real world. I'll work to introduce PoC for bypassing clean_env of bundler before triaging default gems.

[hsbt (Hiroshi SHIBATA)]

• Description updated (diff)

[hsbt (Hiroshi SHIBATA)]

• Description updated (diff)

[hsbt (Hiroshi SHIBATA)]

• Related to Feature #19702: Promote racc as bundled gems added

[hsbt (Hiroshi SHIBATA)]

• Description updated (diff)

[hsbt (Hiroshi SHIBATA)]

• Description updated (diff)

[hsbt (Hiroshi SHIBATA)]

• Description updated (diff)

[hsbt (Hiroshi SHIBATA)]

• Description updated (diff)

https://github.com/ruby/ruby/pull/7436 is working fine to load bundled gems without Gemfile declaration.

[hsbt (Hiroshi SHIBATA)]

• Description updated (diff)

[hsbt (Hiroshi SHIBATA)]

• Description updated (diff)

[hsbt (Hiroshi SHIBATA)]

I update proposal for next preview release.

And I'm considering to promote them after warnings about bundled gems. This plan is:

1. Ruby 3.3: warn for adding bundled gems to Gemfile when user load bundled gems without gem 'foo' in their Gemfile. We can add this warning logic into kernel_require.rb or other place.
2. Ruby 3.4 or later: Raise LoadError same as current behavior and promote bundled gems.

This plan is unnecessary with $LOAD_PATH hack of bundled gems.

[hsbt (Hiroshi SHIBATA)]

• Description updated (diff)

[hsbt (Hiroshi SHIBATA)]

I discussed this proposal in Misc #19722: DevMeeting-2023-07-13

• Ruby 3.3:
  • Warn for adding bundled gems to be addressed Ruby 3.4 to Gemfile when user load its gem without gem 'foo' in their Gemfile. We can add this warning logic into kernel_require.rb or LoadError or other place.
  • Also warn existing bundled gems was loaded without gem 'foo' of Gemfile. Ex. net-smtp, rexml etc.
• Ruby 3.4:
  • Promote bundled gems.
  • Raise LoadError same as current behavior with warnings of Ruby 3.3.

I withdraw a proposal of bypass $LOAD_PATH hack. I will proceed with above plan instead.

[vo.x (Vit Ondruch)]

hsbt (Hiroshi SHIBATA) wrote in #note-36:

I withdraw a proposal of bypass $LOAD_PATH hack.

I appreciate that. Thx.

[deivid (David Rodríguez)]

Nice.

The way I see it, this warning would ideally detect whether the gem was loaded by library code or application code. If loaded by application code, it would directly recommend to add gem "foo" to the Gemfile as the proper solution. If loaded by library code, it would recommend to bug library author to add the dependency to their gemspec, and adding gem "foo" to the Gemfile as a temporary workaround to remove the warning.

[hsbt (Hiroshi SHIBATA)]

If loaded by library code, it would recommend to bug library author to add the dependency to their gemspec, and adding gem "foo" to the Gemfile as a temporary workaround to remove the warning.

Yes, I prefer it. I'm considering how notice these cases for users.

[deivid (David Rodríguez)]

Tricky, yes! I guess we could detect whether the caller lives inside a GEM_HOME or not to decide whether the caller is a library.

[vo.x (Vit Ondruch)]

Honestly, if there should be some exception, it would be better to have some explicit list then trying to detect something.

[deivid (David Rodríguez)]

I guess we can try to predict via codesearch which libraries will be affected by this and fix them in advance?

[Eregon (Benoit Daloze)]

I think it's best to just mention both possibilities in the warning message.
Detecting this is inherently brittle.

[deivid (David Rodríguez)]

Yeah, it probably makes sense to mention both possibilities and let the user figure out which one applies to her.

[hsbt (Hiroshi SHIBATA)]

• Related to Feature #19776: Warn bundled gems when it called from `require` added

[hsbt (Hiroshi SHIBATA)]

• Status changed from Assigned to Closed

I filed new issue for warning feature: https://bugs.ruby-lang.org/issues/19776

[hsbt (Hiroshi SHIBATA)]

• Related to Feature #19843: Promote bigdecimal as bundled gems at Ruby 3.4 added

[hsbt (Hiroshi SHIBATA)]

• Related to Feature #20187: Bundled gems at Ruby 3.4 added

[Eregon (Benoit Daloze)]

• Related to Feature #20309: Bundled gems for Ruby 3.5 added