Bug #20641: `lib/bundled_gems.rb` makes `Kernel.require` over 100x slower

• Status: Closed
• Assignee: -
• Target version: -
• Backport: 3.1: DONTNEED, 3.2: DONTNEED, 3.3: REQUIRED

[byroot (Jean Boussier)]

I just discovered this while profiling Active Record's test suite, and I noticed 40% of the runtime was in $LOAD_PATH.resolve_feature_path, so much I thought it was a profiler bug.

But it turns out it's real. Various APIs do call require late, for instance Psych calls require 'date' every single time it parses a date: https://github.com/ruby/psych/blob/be0ba74e5613c20f213403e15914d24944c2652d/lib/psych/scalar_scanner.rb#L64

I've put together a quick benchmark:

# frozen_string_literal: true

require "bundler/inline"

gemfile(true) do
  source "https://rubygems.org"

  gem "benchmark-ips"
end

Benchmark.ips do |x|
  x.report(RUBY_VERSION) { require "erb" }
  x.save! "/tmp/require.bench"
  x.compare!
end

And the difference is massive:

Calculating -------------------------------------
               3.2.2      6.450M (± 1.2%) i/s -     32.822M in   5.089538s

Comparison:
               3.2.2:  6449899.4 i/s
               3.3.3:    46996.8 i/s - 137.24x  slower

And that's with a small $LOAD_PATH, the bigger the application, the worse it is, if I add:

100.times do |i|
  $LOAD_PATH.unshift("/tmp/empty-#{i}")
end

Calculating -------------------------------------
               3.3.3      6.198k (± 6.1%) i/s -     30.968k in   5.018955s

Comparison:
               3.2.2:  3380939.6 i/s
               3.3.3:     6198.4 i/s - 545.46x  slower

I'm looking at a way to speed this up, but this is big enough that I believe we should backport the fix.

cc @hsbt (Hiroshi SHIBATA)

[byroot (Jean Boussier)]

Some extra context. This codepath is only active if using Bundler >= 2.5.0.

[byroot (Jean Boussier)]

• Status changed from Open to Closed

Applied in changeset git|82aee1a9467c0f1bd33eb0247c5a0a8b8b9a5049.

---

bundled_gems.rb: Add a fast path

[Bug #20641] Gem::BUNDLED_GEMS.warning? adds a lot of extra
work on top of require. When the call end up atually loading code
the overhead is somewhat marginal.

However it's not uncommon for code to go some late require in some
paths, so it's expected that calling require with something already
required is somewhat fast, and bundled_gems.rb breaks this assumption.

To avoid this, we can have a fast path that in most case allow to
short-circuit all the heavy computations. If we extract the feature
basename and it doesn't match any of the bundled gems we care about
we can return very early.

With this change require 'date' is now only 1.33x slower on Ruby
3.3.3, than it was on Ruby 3.2.2, whereas before this change it
was at least 100x slower.