Project

General

Profile

Gemifying Ruby standard library

Please post comments to https://bugs.ruby-lang.org/issues/5481. We are going to update this page as a summary.

Motivation

  • ruby's release cycle is slow for some standard libraries;
    • ex. security fix for WEBrick, xmlrpc and Zlib.
    • ex. API iteration for net/http, OpenSSL, json, psych, RDoc, minitest and rake.
  • There's already the feature called 'default gems' in ruby and some stdlibs are already using it:
    • rake
    • rdoc
    • minitest
    • json
    • io-console
    • bigdecimal
    • And some gems are already doing out-of-band releases.
  • When releasing we should give independence equally to all stdlibs, but in a consistent and controllable way.

Proposal

  • Allow out-of-band stdlib releases.
    • We are not proposing changes to ruby's release management, the release manager would decide when they release ruby and stdlib.
  • Allow more stdlibs to be installed as a 'default gem'
  • Register these gems on RubyGems.org
    • Introduce a new mechanism: controlling supported ruby version so that we can avoid installing unexpected version of stdlib gems. For example, a WEBrick gem for ruby 2.0.1 (released from ruby_2_0_1 branch) should not be installed for ruby 2.0.0 (released from ruby_2_0_0 branch) unless we know it works for both 2.0.0 and 2.0.1.

Note:

  • Moving stdlibs repository location is not a target of this proposal. The implementation details of stdlib gems should hide this from ruby committers.
  • ruby_1_9_3 is not a target of this proposal. The change should be introduced from 2.0.0 release.

Implementation

Install stdlibs as 'default gems'

  • Newly created stdlib gems version scheme: ruby's version + '.n'(dot plus a number)
    • ex. WEBrick 'default gem': 2.0.0.0, 2.0.0.1, ...
    • Gems which supports multiple ruby versions (ex. rake, rdoc, etc.) keeps their version.
  • Controlling supported ruby version by specifying "platform". In RubyGems, currently the ruby platform only allows a major and minor version (1.8 or 1.9)
  • Uninstalling 'default gems' should be blocked to avoid confusion. RubyGems bundled with 1.9.3 allows users to uninstall 'default gems' now. 
    % gem uninstall webrick -v 2.0.0.0 #=> should raise an error.
  • An updated version of an stdlib gem should be treated as regular gems. With the current 'default gems' implementation, a newer gem will not automatically override the default gem with 'require': 
    % ruby -rwebrick -e 'p WEBrick::VERSION' #=> 2.0.0.0
% gem update webrick
Updating installed gems
Updating webrick
Fetching: webrick-2.0.1.5.gem (100%)
Successfully installed webrick-2.0.1.5
Gems updated: webrick
Installing ri documentation for webrick-2.0.1.5...
Installing RDoc documentation for webrick-2.0.1.5...
% ruby -rwebrick -e 'p WEBrick::VERSION'                    #=> 2.0.0.0 << should be 2.0.1.5
  • Make autoload for stdlib gems work as long as autoload feature exists in 2.0. 
    % ruby -e 'autoload :JSON, "json"; p JSON' #=> "JSON"

For existing 'default gems' see tool/rbinstall.rb. It installs stdlibs as 'default gems'.

  • It scans {lib,ext}/**/*.gemspec and installs it as a spec-only gems via rubygems.
  • Then installs gemdir/bin/* as executables (it would be better to use rubygems instead.)
  • It also reads defs/default_gems, but those are obsolete.

Register 'default gems' on RubyGems.org

  • Improve 'default gems' for creating gem from stdlib file/directory layout.
  • stdlib gems maintainer registers the updated gems on RubyGems.org.
    • Maintainer must have an account at RubyGems.org

ToDo

  • After installing updated stdlib gems, those should be treated as regular gems. How?
    1. Installing 'default gems' as a real gem to /usr/local/lib/ruby/default_gems/, not to /usr/local/lib/ruby/site_ruby/
      • advantage: simple implementation
      • drawback: stdlib gems cannot be loaded with --disable-gems.
    2. Implement it as a new feature of RubyGems.
      • advantage: stdlib can be loaded without --disable-gems
      • drawback: may penalize ruby startup time
      • drawback: may be complex to implement
        The RubyGems team tried to implement a feature similar to this for ruby 1.9.1 and ruby 1.9.2, but it did not work out very well...
    3. ?
  • To avoid installing an incompatible version of stdlib gems, update RubyGems to allow three digits (1.8/1.9 -> 1.9.X) on a "ruby" platform in the gem spec.
  • Uninstalling 'default gems' should be blocked. Set default path to /usr/local/lib/ruby/default_gems/
  • Improve 'default gems' for creating gem from stdlib file/directory layout.
  • Decide which stdlibs should be gemified. Let's ask maintainers. We would need to find maintainers of some stdlibs if needed.
  • Decide tagging/branching policy for stdlib gems. Up to the release manager?
  • Find a way to allow autoloading stdlib gems.
    • It's because autoload does not respect require overwriting now. When we install 'default gems' as a real gem, it needs some care to work.

What stdlibs should be gemified?

Principle

  • The lib must have a maintainer.
  • The lib should be highly independent from ruby's code base.
    • Feature dependencies are OK since RubyGems can resolve them. ex. net/http -> openssl

Stdlib status

  • lib/benchmark
  • lib/cgi
  • lib/csv
  • lib/drb
  • lib/erb
  • lib/irb
  • lib/logger: NaHi will maintain this as a 'default gem'
  • lib/optparse
  • lib/pstore
  • lib/racc
  • lib/rexml
  • lib/rinda
  • lib/rss
  • lib/webrick: NaHi will maintain this as a 'default gem'
  • lib/xmlrpc
  • lib/yaml
  • lib/rake
  • lib/net: Do we split this to core, ftp, http (and https), mail (imap, pop and smtp) and telnet?
  • ext/curses
  • ext/date
  • ext/digest
  • ext/iconv: Deprecated. New products and libraries should not use this
  • ext/openssl
  • ext/sych

Already a 'default gem':

  • lib/minitest
  • lib/rake
  • lib/rdoc
  • lib/rubygems
  • ext/bigdecima
  • ext/io-console
  • ext/json
  • ext/psych

Background

There was an awesome keynote by Aaron at RubyKaigi2011.

  • RubyKaigi 2011 keynote by Aaron Patterson (tenderlove) / AT&T Interactive: http://vimeo.com/26507951
  • Most relevant parts for this topic
    • 14'27" -- 25'55": Ruby's release cycle and gem inconsistencies
    • 24'05" -- 27'25": Converting stdlib into gems and release them along with the ruby tarball ("That's one thing I'd like to see changed")

At the next session, Matz and Japanese ruby-core committers discussed the next ruby release. Matz said 'there will be no 1.9.4 because it becomes 2.0'. (Note: See the discussion at https://bugs.ruby-lang.org/issues/5056.)

Any detailed changes for 2.0 were not known but I (NaHi) thought that gemifying stdlib should happen before the changes because it would include incompatible behaviors. The work for gemifying would be postponed after 2.0 release (we must be busy for stabilizing 2.0 release.) I was thinking that gemifying stdlib for faster iteration (shorter release cycle) which had been explained in Aaron's keynote was a must and I wanted to see that happen soon. I talked to Aaron(tenderlove) and Eric(drbrain, a maintainer of RubyGems) first about how we could make it happen and to make this proposal.

I also asked Matz about it.

Then I got the reponse 'Talk to Yuki Sonoda(yugui) and Aaron. I say yes'.

So I talked with many ruby committers including Yuki and Aaron at RubyKaigi2011. Lots of discussion about the benefits and what must not happen. Not a unanimous accord, but almost all committers there agreed/understood what we're planning to do by gemifying stdlib.

Now, at the time 1.9.3 is being released, I want to start the work.

Like0