Backport #2723: $: length affects re-require time of already loaded files

• Status: Rejected
• Assignee: knu (Akinori MUSHA)

[ghazel (Greg Hazel)]

=begin
This occurs on 1.8.6, 1.8.7 and 1.9.1.

The Kernel#require docs say: 'A feature will not be loaded if it‘s name already appears in $"."'. However, re-requiring the same file still scans all the load paths in $: even if the file will not be loaded.

This is exceedingly problematic on Windows, where stat()ing the list of all paths becomes very slow very quickly:

$:.length: 9, 0.005000 seconds
$:.length: 12, 0.005690 seconds
$:.length: 21, 0.010520 seconds
$:.length: 48, 0.022430 seconds
$:.length: 129, 0.062840 seconds
$:.length: 372, 0.179510 seconds
$:.length: 1101, 0.517070 seconds

Since it is not possible to create a new file called "set.rb" (or even "set.so", "set.o" and "set.dll" etc) somewhere else in the load path and have "require 'set'" pickup that new file, is it really necessary to scan the paths?

Specifying "require 'set.rb'" by hand works quickly.

$:.length: 9, 0.000030 seconds
$:.length: 12, 0.000030 seconds
$:.length: 21, 0.000030 seconds
$:.length: 48, 0.000030 seconds
$:.length: 129, 0.000030 seconds
$:.length: 372, 0.000030 seconds
$:.length: 1101, 0.000030 seconds

But it is very unlikely everyone will switch their require lines.

There are many solutions to this problem, depending on what the intended behavior is. The current solution is a significant source of wasted time when requiring a large code base. My application takes 20 full seconds of require time which are eliminated by hacking this extra scan step out. ($:.length == 169, $".length == 993)
=end

[ghazel (Greg Hazel)]

=begin
As a side note, winnt_stat can get a 27% speed up in the stat miss case by using GetFileAttributesEx. That is the fastest stat replacement I have found, and is actually faster than issuing a single system call using CreateFile. However, that improvement is not enough to make this ticket a non-issue.
=end

[naruse (Yui NARUSE)]

• Priority changed from Normal to 3

=begin

=end

[ghazel (Greg Hazel)]

=begin
Why Low priority? This is the most significant speed problem with Ruby on Windows. It is why starting the Rails environment takes 30 seconds.

If I submitted a patch, would it be accepted?

=end

[shyouhei (Shyouhei Urabe)]

• Priority changed from 3 to 5

=begin

If I submitted a patch, would it be accepted?

That depends, but worth trying at least.
=end

[ghazel (Greg Hazel)]

=begin
Depends on what? I'd like to make the patch work in a way that would be accepted.

What exactly is the intended behavior here?

Should re-requiring a module by name "foo.rb" ever load a different file?
Should re-requiring a module by name "foo" which first found and loaded "foo.rb" ever load a different file like "foo.so"?
=end

[shugo (Shugo Maeda)]

=begin
I guess the speed of require has got faster in the SVN trunk.
Could you try it?

However, if a relative path is included in $:, requrie gets slower because paths in $" are absolute paths.

If paths in $" are relative, the following code loads foo.rb twice.

require "/path/to/foo"
require "foo"

Paths in $" are therefore absolute paths now.

=end

[ghazel (Greg Hazel)]

=begin
The speed in SVN trunk is much better:

$:.length: 9, 0.000090 seconds
$:.length: 12, 0.000150 seconds
$:.length: 21, 0.000250 seconds
$:.length: 48, 0.000490 seconds
$:.length: 129, 0.001160 seconds
$:.length: 372, 0.003140 seconds
$:.length: 1101, 0.008981 seconds

Still not constant, which I think it probably could be if the absolute paths of both "/path/to/foo" and "/path/to/foo.rb" were written to $" when "foo" is loaded. Much better, though.

Any chance this speed improvement will be backported to 1.8.x?

=end

[naruse (Yui NARUSE)]

• Status changed from Open to Closed

=begin
fixed on r26902 of trunk.
=end

[ghazel (Greg Hazel)]

=begin
Could we keep this ticket open for the 1.8.x line until the fix is backported?
=end

[shyouhei (Shyouhei Urabe)]

• Status changed from Closed to Assigned
• Assignee set to knu (Akinori MUSHA)
• Priority changed from 5 to Normal

=begin
OK, ticket moved to backport 1.8.
=end

[shugo (Shugo Maeda)]

=begin
Hi,

2010/3/13 Greg Hazel [e-mail]:

Still not constant, which I think it probably could be if the absolute paths of both "/path/to/foo" and "/path/to/foo.rb" were written to $" when "foo" is loaded.

It may not be enough to improve the speed.

I guess that the speed of require could be faster if Ruby had two
lists for $", where the one list stores arguments of require as-is,
and the other list stores expanded absolute paths.
In that case, if a feature name, which is an argument of require, is
included in the former list, there's no need to lookup the latter
list. But this is not compatible with the current behavior when a
relative path is included in $: and the current directory is changed.
For example, in the SVN trunk, the following code loads foo/x.rb
first, then loads bar/x.rb:

$:.push(".")
Dir.chdir("foo")
require "x"
Dir.chdir("../bar")
require "x"

I doubt, however, that this is intended behavior.

Anyway, if the speed in the SVN trunk is not enough for real world
applications, I will try some hacks.

--
Shugo Maeda

=end

[shugo (Shugo Maeda)]

=begin
Hi,

2010/3/16 Roger Pack [e-mail]:

Any chance this speed improvement will be backported to 1.8.x?

Maybe file a new ticket under 1.8 branch?  Or maybe someone can move
this ticket there for you.

Shyouhei has already reopened the ticket, and has assigned it to knu.

http://redmine.ruby-lang.org/issues/show/2723#note-10

--
Shugo Maeda

=end

[ghazel (Greg Hazel)]

=begin
Shugo,

Not sure if you were looking for a response, but yes I do think an improvement should be made over what is in SVN trunk. $: and $" are ever growing as the number of gems used increases, and while is understandable that loading a new file would be affected by this, it's unexpected that already loaded files would see a performance hit too. While SVN trunk improves on this, time spent searching for already loaded files it is still one of the largest factors in my application startup.

=end

[knu (Akinori MUSHA)]

• Project changed from Ruby 1.8 to Backport187
• Description updated (diff)
• Status changed from Assigned to Rejected

Closing this, Ruby 1.8.7 had gone EOL over three years ago.