https://redmine.ruby-lang.org/https://redmine.ruby-lang.org/favicon.ico?17113305112021-11-16T13:38:57ZRuby Issue Tracking SystemRuby master - Feature #18339: GVL instrumentation APIhttps://redmine.ruby-lang.org/issues/18339?journal_id=946802021-11-16T13:38:57Zmame (Yusuke Endoh)mame@ruby-lang.org
<ul></ul><p>Looks interesting.</p>
<p>I understood that the basic usage is to measure elapsed time between RUBY_INTERNAL_EVENT_GVL_ACQUIRE_ENTER and RUBY_INTERNAL_EVENT_GVL_ACQUIRE_EXIT, to increase the number of threads when the elapsed time is short, and to decrease the number when the time is long. Is my understanding right? I have no idea how to use RUBY_INTERNAL_EVENT_GVL_RELEASE.</p>
<p>I have one concern. I think the hooks are called without GVL, but I wonder if the current hook implementation is robust enough without GVL.</p>
<p>Another idea is to provide a method <code>Thread.gvl_waiting_thread_count</code>. Puma or an application monitor can call it periodically (every one second, every request, or something), reduce thread pool size when the count is high, and add a new thread when the count is low. This is less flexible, but maybe more robust.</p>
<blockquote>
<p>require to saturate one process with fake workload</p>
</blockquote>
<p>I couldn't understand this. Could you explain this?</p> Ruby master - Feature #18339: GVL instrumentation APIhttps://redmine.ruby-lang.org/issues/18339?journal_id=946812021-11-16T13:50:11Zbyroot (Jean Boussier)byroot@ruby-lang.org
<ul></ul><blockquote>
<p>to increase the number of threads when the elapsed time is short, and to decrease the number when the time is long. Is my understanding right?</p>
</blockquote>
<p>That is correct.</p>
<p>Another potential use case is to simply instrument the number of waiting thread, less precise, but faster.</p>
<blockquote>
<p>I have no idea how to use RUBY_INTERNAL_EVENT_GVL_RELEASE.</p>
</blockquote>
<p>I mostly suggest it for completness. I don't have an use case for it, but maybe you might want to instrument how long threads hold the GVL etc.</p>
<blockquote>
<p>I think the hooks are called without GVL</p>
</blockquote>
<p>Yes, that is indeed a concern. I looked at the GC hooks and I was under the impression that it was already the case there. I might be wrong.</p>
<blockquote>
<p>Another idea is to provide a method Thread.gvl_waiting_thread_count.</p>
</blockquote>
<p>Yes, it would be less flexible and less precise but would help. I'd still prefer the C hooks though.</p>
<p>How I see it <code>gvl_waiting_thread_count</code> is useful for low overhead monitoring, it will tell you you might have a problem. Once you see there's one, you might want to temporarily enable actual time measurement to finely tune your service.</p>
<blockquote>
<p>require to saturate one process with fake workload / I couldn't understand this. Could you explain this?</p>
</blockquote>
<p>Yes, to measure the effects of your number of threads, you must saturate your server. Meaning having all threads always working, otherwise your setting might look fine one day, but once you have a spike of traffic the performance tanks.</p>
<p>So usually you do this with some synthetic (fake) traffic.</p>
<p>The problem is that not all work loads are equal. Some endpoints will be CPU intensive, some other IO intensive, it's very hard to create synthetic traffic that perfectly reflect production. And even if you can, in a few weeks or months the pattern may have changed.</p>
<p>That is why I think constant monitoring of the situation in production is preferable to a "test bench" method using fake load. Does it make more sense?</p> Ruby master - Feature #18339: GVL instrumentation APIhttps://redmine.ruby-lang.org/issues/18339?journal_id=946822021-11-16T14:20:43Zmame (Yusuke Endoh)mame@ruby-lang.org
<ul></ul><p>I see! You are creating an artificially high load condition to adjust the best settings for high loads, aren't you?<br>
And you think this feature will allow to automatically adjust settings according to the current actual load.<br>
I don't know how difficult it is to implement such an automatic adjustment, but it is super cool if possible.<br>
Thank you for the clarification!</p> Ruby master - Feature #18339: GVL instrumentation APIhttps://redmine.ruby-lang.org/issues/18339?journal_id=946852021-11-16T15:02:37Zbyroot (Jean Boussier)byroot@ruby-lang.org
<ul></ul><blockquote>
<p>You are creating an artificially high load condition to adjust the best settings for high loads, aren't you?</p>
</blockquote>
<p>Yes.</p>
<blockquote>
<p>And you think this feature will allow to automatically adjust settings according to the current actual load.</p>
</blockquote>
<p>That's a possibility, I'm not certain of that. The main objective short term is monitoring for humans, but yes a nice feature would be puma/sidekiq/etc being able to apply backpressure by reducing their concurrency dynamically.</p> Ruby master - Feature #18339: GVL instrumentation APIhttps://redmine.ruby-lang.org/issues/18339?journal_id=947382021-11-18T14:11:12Zmame (Yusuke Endoh)mame@ruby-lang.org
<ul></ul><p><a class="user active user-mention" href="https://redmine.ruby-lang.org/users/13">@matz (Yukihiro Matsumoto)</a> left this to <a class="user active user-mention" href="https://redmine.ruby-lang.org/users/17">@ko1 (Koichi Sasada)</a>, and ko1 was basically positive for this proposal.</p>
<p>byroot (Jean Boussier) wrote in <a href="#note-2">#note-2</a>:</p>
<blockquote>
<blockquote>
<p>I think the hooks are called without GVL<br>
Yes, that is indeed a concern. I looked at the GC hooks and I was under the impression that it was already the case there. I might be wrong.</p>
</blockquote>
</blockquote>
<p>According to ko1, GC hooks are called with GVL, so this is the first time to call hooks in parallel. The current hook implementation is not robust at all for such a usage. It will break if one thread adds/removes a new hook and another thread iterates the hook list simultaneously. You need to lock the data structure by using a mutex, or copy GVL-related hook set to thread-local storage before releasing the GVL.</p> Ruby master - Feature #18339: GVL instrumentation APIhttps://redmine.ruby-lang.org/issues/18339?journal_id=947392021-11-18T14:15:12Zbyroot (Jean Boussier)byroot@ruby-lang.org
<ul></ul><blockquote>
<p>ko1 was basically positive for this proposal.</p>
</blockquote>
<p>That's great to hear!</p>
<blockquote>
<p>The current hook implementation is not robust at all for such a usage.</p>
</blockquote>
<p>Understood. So since <a class="user active user-mention" href="https://redmine.ruby-lang.org/users/17">@ko1 (Koichi Sasada)</a> is mostly positive on the idea, how should we proceed? Is it realistic to try to harden the existing <code>rb_tracepoint_new</code> API, or should this GVL instrumentation API just not rely on it?</p> Ruby master - Feature #18339: GVL instrumentation APIhttps://redmine.ruby-lang.org/issues/18339?journal_id=950862021-12-03T04:09:30Zko1 (Koichi Sasada)
<ul></ul><p>ah, it not easy to implement...<br>
could you try?</p> Ruby master - Feature #18339: GVL instrumentation APIhttps://redmine.ruby-lang.org/issues/18339?journal_id=950992021-12-03T09:04:50Zbyroot (Jean Boussier)byroot@ruby-lang.org
<ul></ul><blockquote>
<p>could you try?</p>
</blockquote>
<p>Sure. Would likely be a next year thing anyway.</p>
<p>However Is a new dedicated hook API ok? Since apparently making the tracepoint API usable outside the GVL would be very hard.</p> Ruby master - Feature #18339: GVL instrumentation APIhttps://redmine.ruby-lang.org/issues/18339?journal_id=962012022-01-27T16:22:39Zbyroot (Jean Boussier)byroot@ruby-lang.org
<ul></ul><p>I opened a draft for it: <a href="https://github.com/ruby/ruby/pull/5500" class="external">https://github.com/ruby/ruby/pull/5500</a></p>
<p><a class="user active user-mention" href="https://redmine.ruby-lang.org/users/17">@ko1 (Koichi Sasada)</a> if you'd like to have a look.</p> Ruby master - Feature #18339: GVL instrumentation APIhttps://redmine.ruby-lang.org/issues/18339?journal_id=964272022-02-08T18:52:13Zbyroot (Jean Boussier)byroot@ruby-lang.org
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/96427/diff?detail_id=62016">diff</a>)</li></ul> Ruby master - Feature #18339: GVL instrumentation APIhttps://redmine.ruby-lang.org/issues/18339?journal_id=964972022-02-15T05:52:01Zko1 (Koichi Sasada)
<ul></ul><ul>
<li>Could you write a picture (or flow) what happens on GVL hooks with GVL acquire/release?</li>
<li>It seems it doesn't use TracePoint data structures, so I think we don't need to use <code>RUBY_INTERNAL_EVENT_*</code> naming.</li>
<li>I'm planning to introduce M:N threads so I'm not sure what you want to monitor on it.</li>
</ul> Ruby master - Feature #18339: GVL instrumentation APIhttps://redmine.ruby-lang.org/issues/18339?journal_id=964992022-02-15T08:11:52Zbyroot (Jean Boussier)byroot@ruby-lang.org
<ul></ul><blockquote>
<p>Could you write a picture (or flow) what happens on GVL hooks with GVL acquire/release?</p>
</blockquote>
<p>I'm sorry, I don't understand the question.</p>
<blockquote>
<p>so I think we don't need to use RUBY_INTERNAL_EVENT_* naming.</p>
</blockquote>
<p>Sure. It was just for consistency, but I can rename it to anything.</p>
<blockquote>
<p>I'm planning to introduce M:N threads so I'm not sure what you want to monitor on it.</p>
</blockquote>
<p>I'm not very familiar with M:N threads, but from my limited understanding yes it might make what I'm trying to do difficult.</p>
<p>What I want to do with these hooks is to be able to tell users wether they spawned too many threads. So measure how much time threads spent waiting to be scheduled when they could have been executing.</p>
<p>Maybe there would be a better solution for this?</p> Ruby master - Feature #18339: GVL instrumentation APIhttps://redmine.ruby-lang.org/issues/18339?journal_id=972502022-04-14T07:28:05Zbyroot (Jean Boussier)byroot@ruby-lang.org
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/97250/diff?detail_id=62384">diff</a>)</li></ul> Ruby master - Feature #18339: GVL instrumentation APIhttps://redmine.ruby-lang.org/issues/18339?journal_id=978322022-06-03T13:14:48Zbyroot (Jean Boussier)byroot@ruby-lang.org
<ul><li><strong>Status</strong> changed from <i>Open</i> to <i>Closed</i></li></ul><p>Merged as <a href="https://github.com/ruby/ruby/commit/9125374726fbf68c05ee7585d4a374ffc5efc5db" class="external">https://github.com/ruby/ruby/commit/9125374726fbf68c05ee7585d4a374ffc5efc5db</a> after several rounds of review with <a class="user active user-mention" href="https://redmine.ruby-lang.org/users/17">@ko1 (Koichi Sasada)</a></p>