https://redmine.ruby-lang.org/https://redmine.ruby-lang.org/favicon.ico?17113305112012-04-14T10:29:25ZRuby Issue Tracking SystemRuby master - Feature #6293: new queue / blocking queueshttps://redmine.ruby-lang.org/issues/6293?journal_id=258952012-04-14T10:29:25Znormalperson (Eric Wong)normalperson@yhbt.net
<ul></ul><p>"tenderlovemaking (Aaron Patterson)" <a href="mailto:aaron@tenderlovemaking.com" class="email">aaron@tenderlovemaking.com</a> wrote:</p>
<blockquote>
<p>Whenever I use queues, I either use them in a blocking or non-blocking<br>
manner only, so I have separated them in to two classes Thread::Queue,<br>
and Thread::BlockingQueue.</p>
</blockquote>
<p>I don't think queues should be limited to strictly<br>
blocking/non-blocking.</p>
<p>Outside of Ruby, but I've occassionally needed queues that could<br>
toggle between blocking/non-blocking (even in the same thread)<br>
to enforce some sort of fairness. Something like:</p>
<p>while task = queue.take<br>
while task.run_one_timeslice</p>
<pre><code> if pending_task = queue.trytake # non-blocking
# reschedule task if another task shows up
queue.push(task)
task = pending_task
end
end
</code></pre>
<p>end</p>
<p>Also, I very often use queues (often Unix pipes) where one end is<br>
blocking and the other end is not (though you don't implement<br>
non-blocking push at all)</p> Ruby master - Feature #6293: new queue / blocking queueshttps://redmine.ruby-lang.org/issues/6293?journal_id=258962012-04-14T10:57:48Zmame (Yusuke Endoh)mame@ruby-lang.org
<ul><li><strong>Status</strong> changed from <i>Open</i> to <i>Assigned</i></li><li><strong>Assignee</strong> set to <i>matz (Yukihiro Matsumoto)</i></li></ul><p>Hello,</p>
<p>2012/4/14 tenderlovemaking (Aaron Patterson) <a href="mailto:aaron@tenderlovemaking.com" class="email">aaron@tenderlovemaking.com</a>:</p>
<blockquote>
<p>I'd like to add new Queue objects to ruby.</p>
</blockquote>
<p>I bet matz does not accept it because there is no good reason.<br>
Why don't you extend the traditional Queue class?</p>
<blockquote>
<ul>
<li>include Enumerable</li>
</ul>
</blockquote>
<p>The semantics is not trivial. See the discussion in <a class="issue tracker-2 status-5 priority-4 priority-default closed" title="Feature: add Queue#each() method and include Enumerable (Closed)" href="https://redmine.ruby-lang.org/issues/4589">#4589</a>.</p>
<p>--<br>
Yusuke Endoh <a href="mailto:mame@tsg.ne.jp" class="email">mame@tsg.ne.jp</a></p> Ruby master - Feature #6293: new queue / blocking queueshttps://redmine.ruby-lang.org/issues/6293?journal_id=258982012-04-14T11:23:15ZAnonymous
<ul><li><strong>File</strong> <a href="/attachments/2598">noname</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/2598/noname">noname</a> added</li></ul><p>On Sat, Apr 14, 2012 at 10:23:56AM +0900, Eric Wong wrote:</p>
<blockquote>
<p>"tenderlovemaking (Aaron Patterson)" <a href="mailto:aaron@tenderlovemaking.com" class="email">aaron@tenderlovemaking.com</a> wrote:</p>
<blockquote>
<p>Whenever I use queues, I either use them in a blocking or non-blocking<br>
manner only, so I have separated them in to two classes Thread::Queue,<br>
and Thread::BlockingQueue.</p>
</blockquote>
<p>I don't think queues should be limited to strictly<br>
blocking/non-blocking.</p>
<p>Outside of Ruby, but I've occassionally needed queues that could<br>
toggle between blocking/non-blocking (even in the same thread)<br>
to enforce some sort of fairness. Something like:</p>
<p>while task = queue.take<br>
while task.run_one_timeslice</p>
<pre><code> if pending_task = queue.trytake # non-blocking
# reschedule task if another task shows up
queue.push(task)
task = pending_task
end
end
</code></pre>
<p>end</p>
</blockquote>
<p>The patch I submitted includes a method <code>poll</code> which will not block if a<br>
timeout isn't supplied. In your case, I would rewrite as:</p>
<pre><code> queue = Thread::BlockingQueue.new
while task = queue.shift
while task.run_one_timeslice
if pending_task = queue.poll # non-blocking
# reschedule task if another task shows up
queue.push(task)
task = pending_task
end
end
end
</code></pre>
<blockquote>
<p>Also, I very often use queues (often Unix pipes) where one end is<br>
blocking and the other end is not (though you don't implement<br>
non-blocking push at all)</p>
</blockquote>
<p>Both of these queues should have non-blocking push since neither is<br>
bounded. If this patch is applied, I'll add sized queues which can<br>
block. :-)</p>
<p>--<br>
Aaron Patterson<br>
<a href="http://tenderlovemaking.com/" class="external">http://tenderlovemaking.com/</a></p> Ruby master - Feature #6293: new queue / blocking queueshttps://redmine.ruby-lang.org/issues/6293?journal_id=258992012-04-14T11:29:14ZAnonymous
<ul><li><strong>File</strong> <a href="/attachments/2599">noname</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/2599/noname">noname</a> added</li></ul><p>On Sat, Apr 14, 2012 at 10:58:12AM +0900, mame (Yusuke Endoh) wrote:</p>
<blockquote>
<p>Issue <a class="issue tracker-2 status-2 priority-4 priority-default" title="Feature: new queue / blocking queues (Assigned)" href="https://redmine.ruby-lang.org/issues/6293">#6293</a> has been updated by mame (Yusuke Endoh).</p>
<p>Status changed from Open to Assigned<br>
Assignee set to matz (Yukihiro Matsumoto)</p>
<p>Hello,</p>
<p>2012/4/14 tenderlovemaking (Aaron Patterson) <a href="mailto:aaron@tenderlovemaking.com" class="email">aaron@tenderlovemaking.com</a>:</p>
<blockquote>
<p>I'd like to add new Queue objects to ruby.</p>
</blockquote>
<p>I bet matz does not accept it because there is no good reason.<br>
Why don't you extend the traditional Queue class?</p>
</blockquote>
<p>I could, but I think the changes I want would add too much complexity<br>
to the traditional Queue class. We have to add branching code for every<br>
method that could block / not-block. Not to mention users must pass the<br>
magical true | false to indicate if they want blocking or non-blocking<br>
behavior. For example:</p>
<p>queue.pop(true) # does "true" mean I enable blocking?<br>
queue.pop(false) # does "false" mean I disable blocking?</p>
<p>It seems confusing to me, especially given the <em>actual</em> api with twists<br>
your brain with a double negative (from thread.rb):</p>
<p>def pop(non_block=false)<br>
...<br>
end</p>
<p>So calling pop() means we're doing a not not blocking call. :(</p>
<blockquote>
<blockquote>
<ul>
<li>include Enumerable</li>
</ul>
</blockquote>
<p>The semantics is not trivial. See the discussion in <a class="issue tracker-2 status-5 priority-4 priority-default closed" title="Feature: add Queue#each() method and include Enumerable (Closed)" href="https://redmine.ruby-lang.org/issues/4589">#4589</a>.</p>
</blockquote>
<p>I've read <a class="issue tracker-2 status-5 priority-4 priority-default closed" title="Feature: add Queue#each() method and include Enumerable (Closed)" href="https://redmine.ruby-lang.org/issues/4589">#4589</a>. I don't think it's that much of a problem. Other<br>
languages[1] include enumerable type behavior in their queues, and<br>
consistency is not guaranteed.</p>
<ol>
<li><a href="http://docs.oracle.com/javase/6/docs/api/java/util/Queue.html#methods_inherited_from_class_java.util.Collection" class="external">http://docs.oracle.com/javase/6/docs/api/java/util/Queue.html#methods_inherited_from_class_java.util.Collection</a></li>
</ol>
<p>--<br>
Aaron Patterson<br>
<a href="http://tenderlovemaking.com/" class="external">http://tenderlovemaking.com/</a></p> Ruby master - Feature #6293: new queue / blocking queueshttps://redmine.ruby-lang.org/issues/6293?journal_id=259272012-04-16T18:29:37Zko1 (Koichi Sasada)
<ul></ul><p>Hi,</p>
<p>Just an idea.</p>
<p>(2012/04/14 11:25), Aaron Patterson wrote:</p>
<blockquote>
<p>I could, but I think the changes I want would add too much<br>
complexity to the traditional Queue class. We have to add<br>
branching code for every method that could block / not-block. Not<br>
to mention users must pass the magical true | false to indicate if<br>
they want blocking or non-blocking behavior. For example:</p>
<p>queue.pop(true) # does "true" mean I enable blocking?<br>
queue.pop(false) # does "false" mean I disable blocking?</p>
<p>It seems confusing to me, especially given the <em>actual</em> api with<br>
twists your brain with a double negative (from thread.rb):</p>
<p>def pop(non_block=false) ... end</p>
<p>So calling pop() means we're doing a not not blocking call. :(</p>
</blockquote>
<p>How about to add try_pop?</p>
<blockquote>
<blockquote>
<blockquote>
<ul>
<li>include Enumerable</li>
</ul>
</blockquote>
<p>The semantics is not trivial. See the discussion in <a class="issue tracker-2 status-5 priority-4 priority-default closed" title="Feature: add Queue#each() method and include Enumerable (Closed)" href="https://redmine.ruby-lang.org/issues/4589">#4589</a>.</p>
</blockquote>
<p>I've read <a class="issue tracker-2 status-5 priority-4 priority-default closed" title="Feature: add Queue#each() method and include Enumerable (Closed)" href="https://redmine.ruby-lang.org/issues/4589">#4589</a>. I don't think it's that much of a problem.<br>
Other languages[1] include enumerable type behavior in their<br>
queues, and consistency is not guaranteed.</p>
<ol>
<li>
</ol>
<p><a href="http://docs.oracle.com/javase/6/docs/api/java/util/Queue.html#methods_inherited_from_class_java.util.Collection" class="external">http://docs.oracle.com/javase/6/docs/api/java/util/Queue.html#methods_inherited_from_class_java.util.Collection</a></p>
</blockquote>
<p>How</p>
<blockquote>
</blockquote>
<p>about to implement Queue#to_a method that generate array which<br>
contains queues containing objects?</p>
<a name="I-agree-the-name-ThreadQueue-because-it-is-clear"></a>
<h1 >I agree the name Thread::Queue because it is clear<a href="#I-agree-the-name-ThreadQueue-because-it-is-clear" class="wiki-anchor">¶</a></h1>
<a name="that-this-object-should-be-for-synchronization"></a>
<h1 >that this object should be for synchronization.<a href="#that-this-object-should-be-for-synchronization" class="wiki-anchor">¶</a></h1>
<a name="Simple-Queue-class-is-ambiguous-that-we-can-use-it"></a>
<h1 >Simple Queue class is ambiguous that we can use it<a href="#Simple-Queue-class-is-ambiguous-that-we-can-use-it" class="wiki-anchor">¶</a></h1>
<a name="for-synchronization-or-not-However-in-the-Ruby-world"></a>
<h1 >for synchronization or not. However, in the Ruby world,<a href="#for-synchronization-or-not-However-in-the-Ruby-world" class="wiki-anchor">¶</a></h1>
<a name="Queue-is-familiar-as-a-data-for-synchronization"></a>
<h1 >::Queue is familiar as a data for synchronization.<a href="#Queue-is-familiar-as-a-data-for-synchronization" class="wiki-anchor">¶</a></h1>
<p>--<br>
// SASADA Koichi at atdot dot net</p> Ruby master - Feature #6293: new queue / blocking queueshttps://redmine.ruby-lang.org/issues/6293?journal_id=259392012-04-17T00:29:13ZAnonymous
<ul><li><strong>File</strong> <a href="/attachments/2612">noname</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/2612/noname">noname</a> added</li></ul><p>On Mon, Apr 16, 2012 at 06:25:59PM +0900, SASADA Koichi wrote:</p>
<blockquote>
<p>Hi,</p>
<p>Just an idea.</p>
<p>(2012/04/14 11:25), Aaron Patterson wrote:</p>
<blockquote>
<p>I could, but I think the changes I want would add too much<br>
complexity to the traditional Queue class. We have to add<br>
branching code for every method that could block / not-block. Not<br>
to mention users must pass the magical true | false to indicate if<br>
they want blocking or non-blocking behavior. For example:</p>
<p>queue.pop(true) # does "true" mean I enable blocking?<br>
queue.pop(false) # does "false" mean I disable blocking?</p>
<p>It seems confusing to me, especially given the <em>actual</em> api with<br>
twists your brain with a double negative (from thread.rb):</p>
<p>def pop(non_block=false) ... end</p>
<p>So calling pop() means we're doing a not not blocking call. :(</p>
</blockquote>
<p>How about to add try_pop?</p>
</blockquote>
<p>try_pop seems fine, but it still seems strange to combine blocking and<br>
non-blocking queues (but maybe <em>I</em> am the one who is strange).</p>
<p>In the case of BlockingQueue#pop in the patch I submitted, it allows a<br>
timeout. I don't think it's a feature that should be abandoned.</p>
<blockquote>
<blockquote>
<blockquote>
<blockquote>
<ul>
<li>include Enumerable</li>
</ul>
</blockquote>
<p>The semantics is not trivial. See the discussion in <a class="issue tracker-2 status-5 priority-4 priority-default closed" title="Feature: add Queue#each() method and include Enumerable (Closed)" href="https://redmine.ruby-lang.org/issues/4589">#4589</a>.</p>
</blockquote>
<p>I've read <a class="issue tracker-2 status-5 priority-4 priority-default closed" title="Feature: add Queue#each() method and include Enumerable (Closed)" href="https://redmine.ruby-lang.org/issues/4589">#4589</a>. I don't think it's that much of a problem.<br>
Other languages[1] include enumerable type behavior in their<br>
queues, and consistency is not guaranteed.</p>
<ol>
<li>
</ol>
<p><a href="http://docs.oracle.com/javase/6/docs/api/java/util/Queue.html#methods_inherited_from_class_java.util.Collection" class="external">http://docs.oracle.com/javase/6/docs/api/java/util/Queue.html#methods_inherited_from_class_java.util.Collection</a></p>
</blockquote>
<p>How</p>
<blockquote>
</blockquote>
<p>about to implement Queue#to_a method that generate array which<br>
contains queues containing objects?</p>
</blockquote>
<p>That seems fine! Then we can eliminate Enumerable mixed in. :-)</p>
<blockquote>
<a name="I-agree-the-name-ThreadQueue-because-it-is-clear"></a>
<h1 >I agree the name Thread::Queue because it is clear<a href="#I-agree-the-name-ThreadQueue-because-it-is-clear" class="wiki-anchor">¶</a></h1>
<a name="that-this-object-should-be-for-synchronization"></a>
<h1 >that this object should be for synchronization.<a href="#that-this-object-should-be-for-synchronization" class="wiki-anchor">¶</a></h1>
<a name="Simple-Queue-class-is-ambiguous-that-we-can-use-it"></a>
<h1 >Simple Queue class is ambiguous that we can use it<a href="#Simple-Queue-class-is-ambiguous-that-we-can-use-it" class="wiki-anchor">¶</a></h1>
<a name="for-synchronization-or-not-However-in-the-Ruby-world"></a>
<h1 >for synchronization or not. However, in the Ruby world,<a href="#for-synchronization-or-not-However-in-the-Ruby-world" class="wiki-anchor">¶</a></h1>
<a name="Queue-is-familiar-as-a-data-for-synchronization"></a>
<h1 >::Queue is familiar as a data for synchronization.<a href="#Queue-is-familiar-as-a-data-for-synchronization" class="wiki-anchor">¶</a></h1>
</blockquote>
<p>Yes. I think keeping ::Queue is good. I think ::Queue can be<br>
implemented in terms of Thread::BlockingQueue. I would have refactored<br>
::Queue to be a subclass of Thread::BlockingQueue, but I didn't want my<br>
patch to be ignored because the diff size. :-)</p>
<p>--<br>
Aaron Patterson<br>
<a href="http://tenderlovemaking.com/" class="external">http://tenderlovemaking.com/</a></p> Ruby master - Feature #6293: new queue / blocking queueshttps://redmine.ruby-lang.org/issues/6293?journal_id=260032012-04-19T15:23:18Zko1 (Koichi Sasada)
<ul></ul><p>Hi,</p>
<p>(2012/04/17 0:24), Aaron Patterson wrote:</p>
<blockquote>
<blockquote>
<blockquote>
<p>So calling pop() means we're doing a not not blocking call. :(</p>
</blockquote>
<p>How about to add try_pop?</p>
</blockquote>
<p>try_pop seems fine, but it still seems strange to combine blocking<br>
and non-blocking queues (but maybe <em>I</em> am the one who is strange).</p>
<p>In the case of BlockingQueue#pop in the patch I submitted, it<br>
allows a timeout. I don't think it's a feature that should be<br>
abandoned.</p>
</blockquote>
<p>I understand that ::Queue#pop should receive timeout extra parameter.</p>
<p>However, I'm not sure we need to separate (Blocking|Unblocking)Queue yet.</p>
<blockquote>
<blockquote>
<p>How</p>
<blockquote>
</blockquote>
<p>about to implement Queue#to_a method that generate array which<br>
contains queues containing objects?</p>
</blockquote>
<p>That seems fine! Then we can eliminate Enumerable mixed in. :-)</p>
</blockquote>
<p>Yes. And it is clear semantics.</p>
<p>::Queue#each can be implemented on at least two semantics:</p>
<ol>
<li>block the end of queue. (like IO)</li>
<li>return when reach end of queue. (like Array)</li>
</ol>
<p>Against IO, Queue#each with semantics (1) can't stop. It is similar<br>
to the cycle object (generated by Enumerator#cycle).</p>
<p>--<br>
// SASADA Koichi at atdot dot net</p> Ruby master - Feature #6293: new queue / blocking queueshttps://redmine.ruby-lang.org/issues/6293?journal_id=260182012-04-20T03:53:29Zcout (Paul Brannan)curlypaul924@gmail.com
<ul></ul><p>On Mon, 2012-04-16 at 18:25 +0900, SASADA Koichi wrote:</p>
<blockquote>
<p>How about to add try_pop?</p>
</blockquote>
<p>I like it! It's consistent with Mutex.</p>
<p>Paul</p> Ruby master - Feature #6293: new queue / blocking queueshttps://redmine.ruby-lang.org/issues/6293?journal_id=260602012-04-21T07:59:16ZAnonymous
<ul><li><strong>File</strong> <a href="/attachments/2625">queue.patch</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/2625/queue.patch">queue.patch</a> added</li><li><strong>File</strong> <a href="/attachments/2626">noname</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/2626/noname">noname</a> added</li></ul><p>On Thu, Apr 19, 2012 at 03:20:50PM +0900, SASADA Koichi wrote:</p>
<blockquote>
<p>Hi,</p>
<p>(2012/04/17 0:24), Aaron Patterson wrote:</p>
<blockquote>
<blockquote>
<blockquote>
<p>So calling pop() means we're doing a not not blocking call. :(</p>
</blockquote>
<p>How about to add try_pop?</p>
</blockquote>
<p>try_pop seems fine, but it still seems strange to combine blocking<br>
and non-blocking queues (but maybe <em>I</em> am the one who is strange).</p>
<p>In the case of BlockingQueue#pop in the patch I submitted, it<br>
allows a timeout. I don't think it's a feature that should be<br>
abandoned.</p>
</blockquote>
<p>I understand that ::Queue#pop should receive timeout extra parameter.</p>
</blockquote>
<p>So the interface would be:</p>
<p>queue.pop(true, 10)</p>
<a name="or"></a>
<h1 >or<a href="#or" class="wiki-anchor">¶</a></h1>
<p>queue.pop(false, 10)</p>
<p>It seems confusing. I do not think it is as clear as:</p>
<p>queue = BlockingQueue.new<br>
queue.pop 10</p>
<p>I'm not sure the clarity of ::Queue API really matters though. The<br>
queues I've submitted do not allow <code>nil</code>. In this way, we can have<br>
non-blocking reads that do not throw exceptions.</p>
<p>I think the queues I've submitted have different (but better) semantics.</p>
<blockquote>
<p>However, I'm not sure we need to separate (Blocking|Unblocking)Queue yet.</p>
</blockquote>
<p>I would like them so that I can change queue types without changing the<br>
code that pops off the queue. I simply change the queue I instantiate,<br>
and the rest of my code should not have to change.</p>
<blockquote>
<blockquote>
<blockquote>
<p>How</p>
<blockquote>
</blockquote>
<p>about to implement Queue#to_a method that generate array which<br>
contains queues containing objects?</p>
</blockquote>
<p>That seems fine! Then we can eliminate Enumerable mixed in. :-)</p>
</blockquote>
<p>Yes. And it is clear semantics.</p>
<p>::Queue#each can be implemented on at least two semantics:</p>
<ol>
<li>block the end of queue. (like IO)</li>
<li>return when reach end of queue. (like Array)</li>
</ol>
<p>Against IO, Queue#each with semantics (1) can't stop. It is similar<br>
to the cycle object (generated by Enumerator#cycle).</p>
</blockquote>
<p>I agree. I've attached an updated patch that uses to_a and removes Enumerable.</p>
<p>--<br>
Aaron Patterson<br>
<a href="http://tenderlovemaking.com/" class="external">http://tenderlovemaking.com/</a></p> Ruby master - Feature #6293: new queue / blocking queueshttps://redmine.ruby-lang.org/issues/6293?journal_id=332912012-11-20T22:36:01Zmame (Yusuke Endoh)mame@ruby-lang.org
<ul><li><strong>Target version</strong> set to <i>2.6</i></li></ul> Ruby master - Feature #6293: new queue / blocking queueshttps://redmine.ruby-lang.org/issues/6293?journal_id=687662017-12-25T18:15:04Znaruse (Yui NARUSE)naruse@airemix.jp
<ul><li><strong>Target version</strong> deleted (<del><i>2.6</i></del>)</li></ul>