https://redmine.ruby-lang.org/https://redmine.ruby-lang.org/favicon.ico?17097754782011-12-15T19:23:13ZRuby Issue Tracking SystemRuby master - Feature #5767: Cache expanded_load_path to reduce startup timehttps://redmine.ruby-lang.org/issues/5767?journal_id=228372011-12-15T19:23:13Znow (Nikolai Weibull)now@disu.se
<ul></ul><p>On Thu, Dec 15, 2011 at 10:18, Yura Sokolov <a href="mailto:funny.falcon@gmail.com" class="email">funny.falcon@gmail.com</a> wrote:</p>
<blockquote>
<p>Issue <a class="issue tracker-2 status-5 priority-4 priority-default closed" title="Feature: Cache expanded_load_path to reduce startup time (Closed)" href="https://redmine.ruby-lang.org/issues/5767">#5767</a> has been reported by Yura Sokolov.</p>
<hr>
<p>Feature <a class="issue tracker-2 status-5 priority-4 priority-default closed" title="Feature: Cache expanded_load_path to reduce startup time (Closed)" href="https://redmine.ruby-lang.org/issues/5767">#5767</a>: Cache expanded_load_path to reduce startup time<br>
<a href="http://redmine.ruby-lang.org/issues/5767" class="external">http://redmine.ruby-lang.org/issues/5767</a></p>
<p>Author: Yura Sokolov<br>
Status: Open<br>
Priority: Normal<br>
Assignee:<br>
Category:<br>
Target version:</p>
<p>This patch add caching of expanded load path.<br>
It reduces rails startup time by 33%<br>
(patch against 1.9.3-p0 and simple performance test is here <a href="https://gist.github.com/1480404" class="external">https://gist.github.com/1480404</a> )</p>
</blockquote>
<p>Is it only me, or is treating $LOAD_PATH like an array hurting<br>
possible optimizations? I mean, how sane is it to allow<br>
$LOAD_PATH.map!{ … }?</p> Ruby master - Feature #5767: Cache expanded_load_path to reduce startup timehttps://redmine.ruby-lang.org/issues/5767?journal_id=228382011-12-15T19:53:16Zshyouhei (Shyouhei Urabe)shyouhei@ruby-lang.org
<ul></ul><p>Nikolai Weibull wrote:</p>
<blockquote>
<p>On Thu, Dec 15, 2011 at 10:18, Yura Sokolov <a href="mailto:funny.falcon@gmail.com" class="email">funny.falcon@gmail.com</a> wrote:</p>
<blockquote>
<p>Issue <a class="issue tracker-2 status-5 priority-4 priority-default closed" title="Feature: Cache expanded_load_path to reduce startup time (Closed)" href="https://redmine.ruby-lang.org/issues/5767">#5767</a> has been reported by Yura Sokolov.</p>
<hr>
<p>Feature <a class="issue tracker-2 status-5 priority-4 priority-default closed" title="Feature: Cache expanded_load_path to reduce startup time (Closed)" href="https://redmine.ruby-lang.org/issues/5767">#5767</a>: Cache expanded_load_path to reduce startup time<br>
<a href="http://redmine.ruby-lang.org/issues/5767" class="external">http://redmine.ruby-lang.org/issues/5767</a></p>
<p>Author: Yura Sokolov<br>
Status: Open<br>
Priority: Normal<br>
Assignee:<br>
Category:<br>
Target version:</p>
<p>This patch add caching of expanded load path.<br>
It reduces rails startup time by 33%<br>
(patch against 1.9.3-p0 and simple performance test is here <a href="https://gist.github.com/1480404" class="external">https://gist.github.com/1480404</a> )</p>
</blockquote>
<p>Is it only me, or is treating $LOAD_PATH like an array hurting<br>
possible optimizations? I mean, how sane is it to allow<br>
$LOAD_PATH.map!{ … }?</p>
</blockquote>
<p>$LOAD_PATH determines the order to search required files. At least it must be ordered linearly.</p> Ruby master - Feature #5767: Cache expanded_load_path to reduce startup timehttps://redmine.ruby-lang.org/issues/5767?journal_id=228392011-12-15T20:47:52Zfunny_falcon (Yura Sokolov)funny.falcon@gmail.com
<ul></ul><p>Nikolai Weibull wrote:</p>
<blockquote>
<p>Is it only me, or is treating $LOAD_PATH like an array hurting<br>
possible optimizations? I mean, how sane is it to allow<br>
$LOAD_PATH.map!{ … }?</p>
</blockquote>
<p>I think, you right. $LOAD_PATH and $LOADED_FEATURES should be restricted.<br>
But currently they not.<br>
When I trying to improve $LOADED_FEATURES at <a class="issue tracker-2 status-5 priority-4 priority-default closed" title="Feature: Not complex patch to improve `require` time (load.c) (Closed)" href="https://redmine.ruby-lang.org/issues/5427">#5427</a>, I remove dangerous methods.</p>
<p>In this patch, it just invalidates cache when complex self modifying method is called,<br>
and keeps cache valid in case of simple methods, like <code>push</code>, <code>pop</code>, <code><<</code></p> Ruby master - Feature #5767: Cache expanded_load_path to reduce startup timehttps://redmine.ruby-lang.org/issues/5767?journal_id=228472011-12-16T16:10:30Zfunny_falcon (Yura Sokolov)funny.falcon@gmail.com
<ul><li><strong>File</strong> <a href="/attachments/2321">cache_expanded_load_path.patch</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/2321/cache_expanded_load_path.patch">cache_expanded_load_path.patch</a> added</li></ul><p>Cosmetic update to patch: more atomic cache access.</p> Ruby master - Feature #5767: Cache expanded_load_path to reduce startup timehttps://redmine.ruby-lang.org/issues/5767?journal_id=228522011-12-17T21:07:09Zfunny_falcon (Yura Sokolov)funny.falcon@gmail.com
<ul><li><strong>File</strong> <a href="/attachments/2323">find_file_safe_and_cache.patch</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/2323/find_file_safe_and_cache.patch">find_file_safe_and_cache.patch</a> added</li></ul> Ruby master - Feature #5767: Cache expanded_load_path to reduce startup timehttps://redmine.ruby-lang.org/issues/5767?journal_id=228532011-12-17T21:12:34Zfunny_falcon (Yura Sokolov)funny.falcon@gmail.com
<ul></ul><p>There is report about 47% startup time reduction <a href="https://gist.github.com/1480404#gistcomment-69769" class="external">https://gist.github.com/1480404#gistcomment-69769</a></p> Ruby master - Feature #5767: Cache expanded_load_path to reduce startup timehttps://redmine.ruby-lang.org/issues/5767?journal_id=228552011-12-18T01:03:20Zluislavena (Luis Lavena)luislavena@gmail.com
<ul></ul><p>For those interested here are some raw performance numbers and comparison between default (no patch) and other patches combined:</p>
<p><a href="https://gist.github.com/1490555" class="external">https://gist.github.com/1490555</a></p> Ruby master - Feature #5767: Cache expanded_load_path to reduce startup timehttps://redmine.ruby-lang.org/issues/5767?journal_id=228672011-12-19T04:07:34Zfunny_falcon (Yura Sokolov)funny.falcon@gmail.com
<ul><li><strong>File</strong> <a href="/attachments/2327">cache_expanded_load_path.patch</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/2327/cache_expanded_load_path.patch">cache_expanded_load_path.patch</a> added</li></ul> Ruby master - Feature #5767: Cache expanded_load_path to reduce startup timehttps://redmine.ruby-lang.org/issues/5767?journal_id=228742011-12-19T15:57:08Zfunny_falcon (Yura Sokolov)funny.falcon@gmail.com
<ul><li><strong>File</strong> <a href="/attachments/2329">cache_expanded_load_path.patch</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/2329/cache_expanded_load_path.patch">cache_expanded_load_path.patch</a> added</li></ul> Ruby master - Feature #5767: Cache expanded_load_path to reduce startup timehttps://redmine.ruby-lang.org/issues/5767?journal_id=233622012-01-18T19:14:07Zfunny_falcon (Yura Sokolov)funny.falcon@gmail.com
<ul></ul><p>I removed messing with file.c.<br>
Instead rb_get_load_path returns expanded paths when optimization is enabled.</p>
<p>I have updated pull request, so that it contains one commit.</p>
<p><a href="https://github.com/ruby/ruby/pull/68.patch" class="external">https://github.com/ruby/ruby/pull/68.patch</a><br>
<a href="https://github.com/ruby/ruby/pull/68" class="external">https://github.com/ruby/ruby/pull/68</a></p> Ruby master - Feature #5767: Cache expanded_load_path to reduce startup timehttps://redmine.ruby-lang.org/issues/5767?journal_id=235262012-02-04T14:54:55Zfunny_falcon (Yura Sokolov)funny.falcon@gmail.com
<ul></ul><p><a href="https://gist.github.com/1688857" class="external">https://gist.github.com/1688857</a></p>
<p>Most of success of that gist is cause of this issue.<br>
Cites from comments:</p>
<p>Thanks for sharing that. Indeed the speed increase is quite impressive! I've tested the time it take to run some rspec test before and after. Here are the results:</p>
<p>Before: Finished in 402.79 seconds (3521 examples, 3 failures, 39 pending)<br>
After: Finished in 222,82 seconds (3521 examples, 3 failures, 39 pending). WOOW!</p>
<hr>
<p>Obligatory benchmarks:</p>
<p>Test Suite on 1.9.3-p0 proper : 2m06s<br>
Rails boot time : 9.1s</p>
<p>Suite on 1.9.3-p0-perf : 1m32s<br>
Rails boot time : 6.5s</p>
<p>Not bad.</p>
<hr>
<p>$ time rake</p>
<p>before:<br>
real 0m37.919s<br>
user 0m6.474s<br>
sys 0m0.919s</p>
<p>after:<br>
real 0m34.236s<br>
user 0m3.367s<br>
sys 0m0.848s</p>
<p>$ time rails runner "Time.now"</p>
<p>before:<br>
real 0m4.589s<br>
user 0m3.871s<br>
sys 0m0.629s</p>
<p>after:<br>
real 0m2.437s<br>
user 0m1.889s<br>
sys 0m0.534s</p>
<hr>
<p>5.5 secs instead of 8.8 secs for rails runner "puts "omfg""<br>
rspec . 15.1 seconds instead of 18.2 seconds</p>
<hr>
<p>One of my Rails apps, boot time (measured with ./bin/rails runner 'puts 3'</p>
<pre><code>1.9.3-perf: 5.1 seconds
1.9.3-p0: 8.1 seconds
1.9.2-p290: 18.2 seconds
</code></pre>
<hr>
<p>My rake parallel:spec went down from 68 to 38sec!!!!!!!!!! That's 44% cut off! Fantastic! THANK YOU!</p>
<hr>
<p>API spec tests ~30% faster!</p>
<hr>
<p>time bundle exec rake environment</p>
<p>Before:<br>
14.60s user 1.26s system 100% cpu 15.861 total</p>
<p>After:<br>
5.11s user 1.10s system 99% cpu 6.212 total</p>
<p>60.8% speedup!!!!!!</p>
<hr> Ruby master - Feature #5767: Cache expanded_load_path to reduce startup timehttps://redmine.ruby-lang.org/issues/5767?journal_id=254942012-03-31T10:28:58Zmame (Yusuke Endoh)mame@ruby-lang.org
<ul><li><strong>Status</strong> changed from <i>Open</i> to <i>Assigned</i></li><li><strong>Assignee</strong> set to <i>nobu (Nobuyoshi Nakada)</i></li></ul><p>Sorry I don't catch up the discussion.</p>
<p>What's the status?<br>
The proposal is COMPLETELY compatible?<br>
It may be helpful for me to create a short summary of the proposal and discussion.</p>
<p>I tentatively assign this to nobu.<br>
Any other committer is interested in this issue?</p>
<p>--<br>
Yusuke Endoh <a href="mailto:mame@tsg.ne.jp" class="email">mame@tsg.ne.jp</a></p> Ruby master - Feature #5767: Cache expanded_load_path to reduce startup timehttps://redmine.ruby-lang.org/issues/5767?journal_id=255232012-03-31T19:03:12Zfunny_falcon (Yura Sokolov)funny.falcon@gmail.com
<ul></ul><blockquote>
<p>The proposal is COMPLETELY compatible?</p>
</blockquote>
<p>This proposal is completely compatible, since LOAD_PATH remains the same array, methods are overridden only to keep cache in sync.</p>
<p>Proposal is about keeping result of expanding LOAD_PATH entries in a 'invisible' cache.<br>
This cache is lazy invalidated on changing directory and filesystem encoding, so that<br>
the whole semantic of LOAD_PATH remains the same.<br>
Methods of LOAD_PATH overridden to keep cache in sync, so that there is no need to expand<br>
all paths again if entries were only appended to any side of LOAD_PATH, or removed from any side.</p>
<p>It saves up to 35% of big application's startup time and still brings slight improvement<br>
for small scripts.</p>
<p>(Only Jeremy Evan's sequel use self crafted super fast load system which does not gain improvement from this patch)</p>
<p>I didn't check it against current trunk, but believe there could not be big problems.</p>
<p>There is still small room for improvement this patch: make chached expanded paths class to be<br>
subclass of String, and then test against it in a file_expand_path, so that to not expand it<br>
twice in calls from rb_find_file_ext_safe and rb_find_file_safe.</p>
<p>Yura aka funny_falcon</p> Ruby master - Feature #5767: Cache expanded_load_path to reduce startup timehttps://redmine.ruby-lang.org/issues/5767?journal_id=255242012-03-31T19:23:37Ztrans (Thomas Sawyer)
<ul></ul><p>Does this have any effect on <a href="https://bugs.ruby-lang.org/issues/4969" class="external">https://bugs.ruby-lang.org/issues/4969</a> (scroll down to post #7).</p> Ruby master - Feature #5767: Cache expanded_load_path to reduce startup timehttps://redmine.ruby-lang.org/issues/5767?journal_id=255342012-04-01T00:04:57Zfunny_falcon (Yura Sokolov)funny.falcon@gmail.com
<ul></ul><blockquote>
<p>Does this have any effect on <a href="https://bugs.ruby-lang.org/issues/4969" class="external">https://bugs.ruby-lang.org/issues/4969</a> (scroll down to post #7).</p>
</blockquote>
<p>Thomas Sawyer, this proposal has no any effect on <a class="issue tracker-2 status-6 priority-4 priority-default closed" title="Feature: Subtle issue with require (Rejected)" href="https://redmine.ruby-lang.org/issues/4969">#4969</a> because it doesn't change require semantic<br>
in any way. It only improve performance.</p>
<p>I leave my comment at <a class="issue tracker-2 status-6 priority-4 priority-default closed" title="Feature: Subtle issue with require (Rejected)" href="https://redmine.ruby-lang.org/issues/4969">#4969</a> and I still take that point: <a class="issue tracker-2 status-6 priority-4 priority-default closed" title="Feature: Subtle issue with require (Rejected)" href="https://redmine.ruby-lang.org/issues/4969">#4969</a> is about making game more tangled but easier to win instead of playing by the rules, imho.</p> Ruby master - Feature #5767: Cache expanded_load_path to reduce startup timehttps://redmine.ruby-lang.org/issues/5767?journal_id=331392012-11-20T09:34:52Zh.shirosaki (Hiroshi Shirosaki)h.shirosaki@gmail.com
<ul><li><strong>Status</strong> changed from <i>Assigned</i> to <i>Closed</i></li></ul><p>Fixed by <a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: require is slow in its bookkeeping; can make Rails startup 2.2x faster (Closed)" href="https://redmine.ruby-lang.org/issues/7158">#7158</a>.</p>