Project

General

Profile

Actions

Bug #9082

closed

popen3 hangs when stderr gets lots of output

Added by rosenfeld (Rodrigo Rosenfeld Rosas) almost 11 years ago. Updated over 8 years ago.

Status:
Rejected
Assignee:
-
Target version:
-
ruby -v:
ruby 2.0.0p247 (2013-06-27 revision 41674) [x86_64-linux]
Backport:
[ruby-core:58176]

Description

Create this program test.rb:

STDERR.puts "some error line\n" * 10_000

Then, try this:

ruby -r open3 -e "Open3.popen3('ruby test.rb'){|i,o,e,t|i.close;o.read}"

For this particular case, if I do "e.read" before "o.read" it works. But for my real case involving the "tidy" command with lots of warnings in the stderr it will freeze even if I call "e.read". Since I don't really need this output, I'm using popen2 and redirecting stderr to /dev/null, so this is not urgent, but I thought you'd like to know about this.

Also, if you replace 10_000 with 1_000 above it also works here.

Updated by jeremyevans0 (Jeremy Evans) almost 11 years ago

I think this is expected behavior, not a bug. Your test.rb program blocks outputting to stderr because the pipe buffer gets filled. The reason it works for less output to stderr is that the program exits before the stderr pipe buffer is full. The reason it works if you read stderr before stdout is that you drain the stderr pipe buffer, allowing the program to exit.

Updated by Anonymous almost 11 years ago

  • Status changed from Open to Rejected

Jeremy is correct, this is not a bug in popen3.

To properly handle things like this, look into IO.select.

Updated by rosenfeld (Rodrigo Rosenfeld Rosas) almost 11 years ago

Will it also block on STDOUT? If so, I suppose people shouldn't use the open3 library for interacting with commands like 'tidy' and several others that will process files (even less than 1MB)

I don't see this limitation in the docs nor any API for dealing with such cases...

Updated by Anonymous almost 11 years ago

Rodrigo: Yes, this is how UNIX works.

Updated by rosenfeld (Rodrigo Rosenfeld Rosas) almost 11 years ago

At least with the block form. I think this should be documented if this is the case.

Updated by rosenfeld (Rodrigo Rosenfeld Rosas) almost 11 years ago

Also, is it expected that even if e.read is called inside the block it will hang up?

Updated by Anonymous almost 11 years ago

If you call e.read then test.rb is able to write the entire message to STDERR then exit. When it exits, both stderr and stdout are closed so e.read returns.

Updated by rosenfeld (Rodrigo Rosenfeld Rosas) almost 11 years ago

I see, but usually a process will write both to STDOUT and STDERR so it may happen that both buffers will be full. What should one do in such case?

Updated by Anonymous almost 11 years ago

You need to use IO.select to handle that.

Updated by rosenfeld (Rodrigo Rosenfeld Rosas) almost 11 years ago

I'm assuming that if I use popen2 or redirect all STDOUT to /dev/null I don't need IO.select, right?

Updated by akr (Akira Tanaka) almost 11 years ago

rosenfeld (Rodrigo Rosenfeld Rosas) wrote:

At least with the block form. I think this should be documented if this is the case.

The document already has following description:

% ri Open3.popen3
...
You should be careful to avoid deadlocks. Since pipes are fixed length buffer,
Open3.popen3("prog") {|i, o, e, t| o.read } deadlocks if the program generates
many output on stderr. You should be read stdout and stderr simultaneously
(using thread or IO.select). However if you don't need stderr output,
Open3.popen2 can be used. If merged stdout and stderr output is not a problem,
you can use Open3.popen2e. If you really needs stdout and stderr output as
separate strings, you can consider Open3.capture3.

Updated by rosenfeld (Rodrigo Rosenfeld Rosas) almost 11 years ago

Also, I'm not sure how I should use IO.select. Looking at the documentation it's not clear to me how I should use it in this case...

Updated by rosenfeld (Rodrigo Rosenfeld Rosas) almost 11 years ago

Sorry, haven't seen your message before posting, @akr (Akira Tanaka). Thanks, I surely overlooked the documentation, sorry!

Updated by bitsweat (Jeremy Daer) over 8 years ago

Note you can use Open3.capture3 to safely read stdout and stderr without doing popen3 + IO.select yourself.

Internally, this uses separate threads to read stderr & stdout and to wait for the child process to finish: https://github.com/ruby/ruby/blob/1ec544695fa02d714180ef9c34e755027b6a2103/lib/open3.rb#L257-L273

Actions

Also available in: Atom PDF

Like0
Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0