https://redmine.ruby-lang.org/https://redmine.ruby-lang.org/favicon.ico?17113305112020-07-29T15:34:46ZRuby Issue Tracking SystemRuby master - Feature #17059: epoll as the backend of IO.select on Linuxhttps://redmine.ruby-lang.org/issues/17059?journal_id=867972020-07-29T15:34:46Zdsh0416 (Delton Ding)
<ul><li><strong>Subject</strong> changed from <i>epoll as IO.select</i> to <i>epoll as IO.select's backend</i></li></ul> Ruby master - Feature #17059: epoll as the backend of IO.select on Linuxhttps://redmine.ruby-lang.org/issues/17059?journal_id=867992020-07-29T15:41:42Zdsh0416 (Delton Ding)
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/86799/diff?detail_id=57602">diff</a>)</li></ul> Ruby master - Feature #17059: epoll as the backend of IO.select on Linuxhttps://redmine.ruby-lang.org/issues/17059?journal_id=868002020-07-29T15:42:26Zdsh0416 (Delton Ding)
<ul><li><strong>Subject</strong> changed from <i>epoll as IO.select's backend</i> to <i>epoll as the backend of IO.select</i></li></ul> Ruby master - Feature #17059: epoll as the backend of IO.select on Linuxhttps://redmine.ruby-lang.org/issues/17059?journal_id=868012020-07-29T15:46:19Zdsh0416 (Delton Ding)
<ul><li><strong>Subject</strong> changed from <i>epoll as the backend of IO.select</i> to <i>epoll as the backend of IO.select on Linux</i></li></ul> Ruby master - Feature #17059: epoll as the backend of IO.select on Linuxhttps://redmine.ruby-lang.org/issues/17059?journal_id=868022020-07-29T15:55:54Zdsh0416 (Delton Ding)
<ul><li><strong>File</strong> <a href="/attachments/8503">epoll.h</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/8503/epoll.h">epoll.h</a> added</li></ul> Ruby master - Feature #17059: epoll as the backend of IO.select on Linuxhttps://redmine.ruby-lang.org/issues/17059?journal_id=870342020-08-12T07:38:32Zko1 (Koichi Sasada)
<ul></ul><p>does it improve the performance?<br>
My understanding is the advantage of <code>epoll</code> is advanced registration.<br>
However, <code>select</code> wrapping interface can not use it.</p>
<p>I have no experience to use <code>epoll</code>, so correct me it is wrong.</p>
<p>Thanks,<br>
Koichi</p> Ruby master - Feature #17059: epoll as the backend of IO.select on Linuxhttps://redmine.ruby-lang.org/issues/17059?journal_id=870792020-08-16T01:12:32Zdsh0416 (Delton Ding)
<ul></ul><p>It should greatly improve the performance.<br>
Advanced registration is a feature of <code>epoll</code>,<br>
but the performance is also an important part for it.<br>
The benchmark from libevent shows the performance of epoll and poll or select,<br>
are on a totally different stage.<br>
So this may be important to performance.</p> Ruby master - Feature #17059: epoll as the backend of IO.select on Linuxhttps://redmine.ruby-lang.org/issues/17059?journal_id=870802020-08-16T09:02:08Zshyouhei (Shyouhei Urabe)shyouhei@ruby-lang.org
<ul></ul><p><a class="user active user-mention" href="https://redmine.ruby-lang.org/users/17">@ko1 (Koichi Sasada)</a> is not talking about efficiency of epoll in general, but questioning that of your patch.</p> Ruby master - Feature #17059: epoll as the backend of IO.select on Linuxhttps://redmine.ruby-lang.org/issues/17059?journal_id=870812020-08-16T10:24:09Zdsh0416 (Delton Ding)
<ul></ul><p>In general, event handling gems like nio4r could provide a similar <code>select</code> interface with multiple backends including select, kqueue and epoll support.</p>
<p>On the side of Ruby meta-programming, this part is easy to be implemented, and could provide a much better performance comparing to the default IO.select.</p>
<p>Since Ruby merged the Fiber scheduler recently, the <code>IO.select</code> injection from the core library may be important to provide better native I/O performance.</p>
<p>But Ruby's implementation of the <code>IO.select</code> has a lot of things coupled with the POSIX <code>select</code>, like the <code>fdset_t</code>.</p>
<p>From the patch, Ruby does use the POSIX <code>fdset_t</code> in some platforms, but Ruby also defines its own structs on some other platforms for non-standard select implementation. this gives the opportunity to inject the <code>epoll</code> API here.</p>
<p>This is my current idea, and I'm working on implementing this.</p>
<p>I'm wondering if changing the <code>rb_f_select</code> directly could be a better idea or not. But there's no such macro control yet in this method, and all other implementations are with the <code>select_internal</code>.</p> Ruby master - Feature #17059: epoll as the backend of IO.select on Linuxhttps://redmine.ruby-lang.org/issues/17059?journal_id=870822020-08-16T21:08:06Zdsh0416 (Delton Ding)
<ul><li><strong>File</strong> <a href="/attachments/8539">epoll.h</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/8539/epoll.h">epoll.h</a> added</li></ul><p>Update the WIP implementation</p> Ruby master - Feature #17059: epoll as the backend of IO.select on Linuxhttps://redmine.ruby-lang.org/issues/17059?journal_id=870832020-08-16T23:49:23Zioquatix (Samuel Williams)samuel@oriontransfer.net
<ul></ul><p>This is a nice idea, and I've considered exposing <code>wait_select</code> on the scheduler interface.</p>
<p>I'd suggest we try to go down this route.</p>
<p>The scheduler interface can cache the I/O registration... so in theory it addresses @ko1's concerns.</p> Ruby master - Feature #17059: epoll as the backend of IO.select on Linuxhttps://redmine.ruby-lang.org/issues/17059?journal_id=870872020-08-17T07:37:59Zdsh0416 (Delton Ding)
<ul></ul><p>Thanks for advice.</p>
<p>To separate the process of registration and wait is a good idea for performance.</p>
<p>Since even the <code>select</code> itself could also take advantages from this,</p>
<p>and simplify the whole I/O multiplexing process.</p> Ruby master - Feature #17059: epoll as the backend of IO.select on Linuxhttps://redmine.ruby-lang.org/issues/17059?journal_id=870892020-08-17T09:40:59ZEregon (Benoit Daloze)
<ul></ul><p>I'm unclear how using <code>epoll</code> can help for the user calling <code>IO.select</code>.<br>
With the API of select(), I'd expect <code>epoll</code> is no faster than <code>select()</code>.</p>
<p>Regarding the Fiber scheduler, then I think a new event would be needed, calling epoll() instead of select() wouldn't solve anything, right?</p> Ruby master - Feature #17059: epoll as the backend of IO.select on Linuxhttps://redmine.ruby-lang.org/issues/17059?journal_id=870902020-08-17T09:49:09Zko1 (Koichi Sasada)
<ul></ul><p>I want to know the general idea how to use <code>epoll</code> for <code>IO.select</code> backend.</p>
<pre><code class="C syntaxhl" data-language="C"><span class="cp">#include</span> <span class="cpf"><stdlib.h></span><span class="cp">
#include</span> <span class="cpf"><stdio.h></span><span class="cp">
</span>
<span class="cp">#define _GNU_SOURCE
#include</span> <span class="cpf"><unistd.h></span><span class="cp">
</span>
<span class="cp">#include</span> <span class="cpf"><sys/resource.h></span><span class="cp">
#include</span> <span class="cpf"><poll.h></span><span class="cp">
#define N 2000 // 10k
</span>
<span class="k">static</span> <span class="kt">void</span>
<span class="nf">task_poll</span><span class="p">(</span><span class="kt">int</span> <span class="n">fds</span><span class="p">[])</span>
<span class="p">{</span>
<span class="k">struct</span> <span class="n">pollfd</span> <span class="n">pfd</span><span class="p">[</span><span class="n">N</span><span class="p">];</span>
<span class="k">for</span> <span class="p">(</span><span class="kt">int</span> <span class="n">i</span><span class="o">=</span><span class="mi">0</span><span class="p">;</span> <span class="n">i</span><span class="o"><</span><span class="n">N</span><span class="p">;</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
<span class="k">struct</span> <span class="n">pollfd</span> <span class="o">*</span><span class="n">p</span><span class="p">;</span>
<span class="c1">// in</span>
<span class="n">p</span> <span class="o">=</span> <span class="o">&</span><span class="n">pfd</span><span class="p">[</span><span class="n">i</span><span class="p">];</span>
<span class="n">p</span><span class="o">-></span><span class="n">fd</span> <span class="o">=</span> <span class="n">fds</span><span class="p">[</span><span class="n">i</span><span class="o">*</span><span class="mi">2</span><span class="p">];</span>
<span class="n">p</span><span class="o">-></span><span class="n">events</span> <span class="o">=</span> <span class="n">POLLIN</span><span class="p">;</span>
<span class="p">}</span>
<span class="kt">int</span> <span class="n">r</span> <span class="o">=</span> <span class="n">poll</span><span class="p">(</span><span class="o">&</span><span class="n">pfd</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">N</span><span class="p">,</span> <span class="mi">0</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="n">r</span><span class="o">==</span><span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// timeout</span>
<span class="p">}</span>
<span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="n">r</span><span class="o">></span><span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
<span class="k">for</span> <span class="p">(</span><span class="kt">int</span> <span class="n">i</span><span class="o">=</span><span class="mi">0</span><span class="p">;</span> <span class="n">i</span><span class="o"><</span><span class="n">N</span><span class="o">*</span><span class="mi">2</span><span class="p">;</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
<span class="n">fprintf</span><span class="p">(</span><span class="n">stderr</span><span class="p">,</span> <span class="s">"%d %d (%d)</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">i</span><span class="p">,</span> <span class="n">pfd</span><span class="p">[</span><span class="n">i</span><span class="p">].</span><span class="n">fd</span><span class="p">,</span> <span class="p">(</span><span class="kt">int</span><span class="p">)</span><span class="n">pfd</span><span class="p">[</span><span class="n">i</span><span class="p">].</span><span class="n">revents</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">else</span> <span class="p">{</span>
<span class="n">fprintf</span><span class="p">(</span><span class="n">stderr</span><span class="p">,</span> <span class="s">"poll (RLIMIT_NOFILE:%d and N:%d)</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="p">(</span><span class="kt">int</span><span class="p">)</span><span class="n">RLIMIT_NOFILE</span><span class="p">,</span> <span class="n">N</span><span class="o">*</span><span class="mi">2</span><span class="p">);</span>
<span class="n">exit</span><span class="p">(</span><span class="mi">1</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="cp">#include</span> <span class="cpf"><sys/epoll.h></span><span class="cp">
</span>
<span class="k">static</span> <span class="kt">void</span>
<span class="nf">task_epoll</span><span class="p">(</span><span class="kt">int</span> <span class="n">fds</span><span class="p">[])</span>
<span class="p">{</span>
<span class="k">struct</span> <span class="n">epoll_event</span> <span class="n">events</span><span class="p">[</span><span class="n">N</span><span class="p">];</span>
<span class="kt">int</span> <span class="n">efd</span> <span class="o">=</span> <span class="n">epoll_create</span><span class="p">(</span><span class="n">N</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="n">efd</span> <span class="o"><</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
<span class="n">perror</span><span class="p">(</span><span class="s">"epoll"</span><span class="p">);</span>
<span class="n">exit</span><span class="p">(</span><span class="mi">1</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">for</span> <span class="p">(</span><span class="kt">int</span> <span class="n">i</span><span class="o">=</span><span class="mi">0</span><span class="p">;</span> <span class="n">i</span><span class="o"><</span><span class="n">N</span><span class="p">;</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
<span class="k">struct</span> <span class="n">epoll_event</span> <span class="o">*</span><span class="n">e</span> <span class="o">=</span> <span class="o">&</span><span class="n">events</span><span class="p">[</span><span class="n">i</span><span class="p">];</span>
<span class="n">e</span><span class="o">-></span><span class="n">events</span> <span class="o">=</span> <span class="n">EPOLLIN</span><span class="p">;</span>
<span class="n">e</span><span class="o">-></span><span class="n">data</span><span class="p">.</span><span class="n">fd</span> <span class="o">=</span> <span class="n">fds</span><span class="p">[</span><span class="n">i</span><span class="o">*</span><span class="mi">2</span><span class="p">];</span>
<span class="k">if</span> <span class="p">(</span><span class="n">epoll_ctl</span><span class="p">(</span><span class="n">efd</span><span class="p">,</span> <span class="n">EPOLL_CTL_ADD</span><span class="p">,</span> <span class="n">fds</span><span class="p">[</span><span class="n">i</span><span class="o">*</span><span class="mi">2</span><span class="p">],</span> <span class="n">e</span><span class="p">)</span> <span class="o"><</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
<span class="n">perror</span><span class="p">(</span><span class="s">"epoll_ctl"</span><span class="p">);</span>
<span class="n">exit</span><span class="p">(</span><span class="mi">1</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="kt">int</span> <span class="n">r</span> <span class="o">=</span> <span class="n">epoll_wait</span><span class="p">(</span><span class="n">efd</span><span class="p">,</span> <span class="n">events</span><span class="p">,</span> <span class="n">N</span><span class="p">,</span> <span class="mi">0</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="n">r</span> <span class="o">==</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// timeout</span>
<span class="p">}</span>
<span class="k">else</span> <span class="k">if</span> <span class="p">(</span><span class="n">r</span> <span class="o">></span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
<span class="k">for</span> <span class="p">(</span><span class="kt">int</span> <span class="n">i</span><span class="o">=</span><span class="mi">0</span><span class="p">;</span> <span class="n">i</span><span class="o"><</span><span class="n">r</span><span class="p">;</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
<span class="n">fprintf</span><span class="p">(</span><span class="n">stderr</span><span class="p">,</span> <span class="s">"%d fd:%d</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">i</span><span class="p">,</span> <span class="n">events</span><span class="p">[</span><span class="n">i</span><span class="p">].</span><span class="n">data</span><span class="p">.</span><span class="n">fd</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">else</span> <span class="p">{</span>
<span class="n">perror</span><span class="p">(</span><span class="s">"epoll_wait"</span><span class="p">);</span>
<span class="n">exit</span><span class="p">(</span><span class="mi">1</span><span class="p">);</span>
<span class="p">}</span>
<span class="c1">// clear</span>
<span class="n">close</span><span class="p">(</span><span class="n">efd</span><span class="p">);</span>
<span class="p">}</span>
<span class="kt">int</span> <span class="nf">main</span><span class="p">(</span><span class="kt">void</span><span class="p">)</span>
<span class="p">{</span>
<span class="kt">int</span> <span class="n">fds</span><span class="p">[</span><span class="n">N</span> <span class="o">*</span> <span class="mi">2</span><span class="p">];</span>
<span class="kt">int</span> <span class="n">i</span><span class="p">;</span>
<span class="k">for</span> <span class="p">(</span><span class="n">i</span><span class="o">=</span><span class="mi">0</span><span class="p">;</span> <span class="n">i</span><span class="o"><</span><span class="n">N</span><span class="p">;</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">pipe</span><span class="p">(</span><span class="o">&</span><span class="n">fds</span><span class="p">[</span><span class="n">i</span><span class="o">*</span><span class="mi">2</span><span class="p">])</span> <span class="o"><</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
<span class="n">perror</span><span class="p">(</span><span class="s">"pipe"</span><span class="p">);</span>
<span class="n">fprintf</span><span class="p">(</span><span class="n">stderr</span><span class="p">,</span> <span class="s">"i:%d</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">i</span><span class="p">);</span>
<span class="n">exit</span><span class="p">(</span><span class="mi">1</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">for</span> <span class="p">(</span><span class="n">i</span><span class="o">=</span><span class="mi">0</span><span class="p">;</span> <span class="n">i</span><span class="o"><</span><span class="mi">1000</span> <span class="o">*</span> <span class="mi">10</span><span class="p">;</span> <span class="n">i</span><span class="o">++</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// task_xxx emulates IO.select</span>
<span class="c1">// task_poll(fds); // real 0m0.537s</span>
<span class="c1">// task_epoll(fds); // real 0m11.191s</span>
<span class="p">}</span>
<span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
</code></pre>
<p><code>epoll</code> version is x20 slower on my machine.<br>
any misunderstanding?</p>
<p>(<code>efd</code> can be reusable, but I'm not sure how to clear all registered fds)</p> Ruby master - Feature #17059: epoll as the backend of IO.select on Linuxhttps://redmine.ruby-lang.org/issues/17059?journal_id=870912020-08-17T09:53:37Zko1 (Koichi Sasada)
<ul></ul><p>I understand <code>epoll</code> will help by introducing new event handling APIs.<br>
But not sure why <code>IO.select</code> can improve the performance with <code>epoll</code>.<br>
This is my question by first comment and maybe <a href="https://bugs.ruby-lang.org/issues/17059#note-13" class="external">https://bugs.ruby-lang.org/issues/17059#note-13</a> is same.</p> Ruby master - Feature #17059: epoll as the backend of IO.select on Linuxhttps://redmine.ruby-lang.org/issues/17059?journal_id=870952020-08-17T12:00:26Zdsh0416 (Delton Ding)
<ul></ul><p>The benchmark looks good. I've tested with similar code, and it's 46x slower on my machine.<br>
It looks like <code>epoll</code> is highly depended on the time that <code>epoll_ctl</code> engaged.</p>
<p>Since the scheduler now have other registration control including <code>rb_io_wait_readable</code> and <code>rb_io_wait_writable</code> are introduced in the current <code>Scheduler</code>.<br>
I would try to use these methods to deal with the registration then, and replace the <code>IO.select</code> in the <code>Scheduler#run</code> for performance.</p>
<p>Is this a proper way to implement then?</p> Ruby master - Feature #17059: epoll as the backend of IO.select on Linuxhttps://redmine.ruby-lang.org/issues/17059?journal_id=870972020-08-17T17:37:25Zko1 (Koichi Sasada)
<ul><li><strong>Status</strong> changed from <i>Open</i> to <i>Rejected</i></li></ul><blockquote>
<p>Since the scheduler now have other registration control including rb_io_wait_readable and rb_io_wait_writable are introduced in the current Scheduler.<br>
I would try to use these methods to deal with the registration then, and replace the IO.select in the Scheduler#run for performance.</p>
</blockquote>
<p>I'm not sure what is your idea, but at least I reject this ticket because <code>IO.select</code> is not good place to use <code>epoll</code>.</p>
<p>Maybe design Ruby-level event handling API and use it by scheduler is the best.</p>
<pre><code class="ruby syntaxhl" data-language="ruby"><span class="n">s</span> <span class="o">=</span> <span class="no">IO</span><span class="o">::</span><span class="no">Selector</span>
<span class="n">s</span><span class="p">.</span><span class="nf">add</span><span class="p">(</span><span class="n">io1</span><span class="p">,</span> <span class="s1">'r'</span><span class="p">)</span>
<span class="n">s</span><span class="p">.</span><span class="nf">add</span><span class="p">(</span><span class="n">io2</span><span class="p">,</span> <span class="s1">'w'</span><span class="p">)</span>
<span class="n">s</span><span class="p">.</span><span class="nf">add</span><span class="p">(</span><span class="n">io3</span><span class="p">,</span> <span class="s1">'rw'</span><span class="p">)</span>
<span class="n">s</span><span class="p">.</span><span class="nf">wait</span> <span class="c1">#=> ready io array</span>
</code></pre>
<p>for example.</p> Ruby master - Feature #17059: epoll as the backend of IO.select on Linuxhttps://redmine.ruby-lang.org/issues/17059?journal_id=871052020-08-18T09:07:28ZEregon (Benoit Daloze)
<ul></ul><p>dsh0416 (Delton Ding) wrote in <a href="#note-16">#note-16</a>:</p>
<blockquote>
<p>I would try to use these methods to deal with the registration then, and replace the <code>IO.select</code> in the <code>Scheduler#run</code> for performance.</p>
</blockquote>
<p>Where do you see <code>IO.select</code> in the <code>Scheduler</code>?<br>
Here? <a href="https://github.com/ruby/ruby/blob/701217572f865375b137d2830d4da0c3e78de046/test/fiber/scheduler.rb#L44" class="external">https://github.com/ruby/ruby/blob/701217572f865375b137d2830d4da0c3e78de046/test/fiber/scheduler.rb#L44</a><br>
That's a test scheduler, and using <code>IO.select()</code> is good enough for prototyping but basically nothing else.</p>
<p><a class="user active user-mention" href="https://redmine.ruby-lang.org/users/3344">@ioquatix (Samuel Williams)</a> it seems worth clarifying that with a comment there.<br>
Might also be worth linking to the async scheduler as a real-world example there or in <code>doc/fiber.rdoc</code>.</p>
<p>Real schedulers like the one for <code>async</code> use epoll/kqueue/libev/etc internally, for instance by using <code>nio4r</code>.</p> Ruby master - Feature #17059: epoll as the backend of IO.select on Linuxhttps://redmine.ruby-lang.org/issues/17059?journal_id=871062020-08-18T09:09:15ZEregon (Benoit Daloze)
<ul></ul><p>In other words I don't think we need to have selectors in Ruby core.<br>
That's part of the beauty of this new scheduler API: it lets you implement it in various way, including how do you want to wait for IO.</p> Ruby master - Feature #17059: epoll as the backend of IO.select on Linuxhttps://redmine.ruby-lang.org/issues/17059?journal_id=871092020-08-18T16:05:57Zdsh0416 (Delton Ding)
<ul></ul><p>Yes. I was just figured out that the scheduler is an example in the tests, where the real scheduler is designed to be separated from the ruby-core.</p>