Bug #9082: popen3 hangs when stderr gets lots of output

• Status: Rejected
• Assignee: -
• Target version: -
• ruby -v: ruby 2.0.0p247 (2013-06-27 revision 41674) [x86_64-linux]

[rosenfeld (Rodrigo Rosenfeld Rosas)]

Create this program test.rb:

STDERR.puts "some error line\n" * 10_000

Then, try this:

ruby -r open3 -e "Open3.popen3('ruby test.rb'){|i,o,e,t|i.close;o.read}"

For this particular case, if I do "e.read" before "o.read" it works. But for my real case involving the "tidy" command with lots of warnings in the stderr it will freeze even if I call "e.read". Since I don't really need this output, I'm using popen2 and redirecting stderr to /dev/null, so this is not urgent, but I thought you'd like to know about this.

Also, if you replace 10_000 with 1_000 above it also works here.

[jeremyevans0 (Jeremy Evans)]

I think this is expected behavior, not a bug. Your test.rb program blocks outputting to stderr because the pipe buffer gets filled. The reason it works for less output to stderr is that the program exits before the stderr pipe buffer is full. The reason it works if you read stderr before stdout is that you drain the stderr pipe buffer, allowing the program to exit.

• Status changed from Open to Rejected

Jeremy is correct, this is not a bug in popen3.

To properly handle things like this, look into IO.select.

[rosenfeld (Rodrigo Rosenfeld Rosas)]

Will it also block on STDOUT? If so, I suppose people shouldn't use the open3 library for interacting with commands like 'tidy' and several others that will process files (even less than 1MB)

I don't see this limitation in the docs nor any API for dealing with such cases...

Rodrigo: Yes, this is how UNIX works.

[rosenfeld (Rodrigo Rosenfeld Rosas)]

At least with the block form. I think this should be documented if this is the case.

[rosenfeld (Rodrigo Rosenfeld Rosas)]

Also, is it expected that even if e.read is called inside the block it will hang up?

If you call e.read then test.rb is able to write the entire message to STDERR then exit. When it exits, both stderr and stdout are closed so e.read returns.

[rosenfeld (Rodrigo Rosenfeld Rosas)]

I see, but usually a process will write both to STDOUT and STDERR so it may happen that both buffers will be full. What should one do in such case?

You need to use IO.select to handle that.

[rosenfeld (Rodrigo Rosenfeld Rosas)]

I'm assuming that if I use popen2 or redirect all STDOUT to /dev/null I don't need IO.select, right?

[akr (Akira Tanaka)]

rosenfeld (Rodrigo Rosenfeld Rosas) wrote:

At least with the block form. I think this should be documented if this is the case.

The document already has following description:

% ri Open3.popen3
...
You should be careful to avoid deadlocks. Since pipes are fixed length buffer,
Open3.popen3("prog") {|i, o, e, t| o.read } deadlocks if the program generates
many output on stderr. You should be read stdout and stderr simultaneously
(using thread or IO.select). However if you don't need stderr output,
Open3.popen2 can be used. If merged stdout and stderr output is not a problem,
you can use Open3.popen2e. If you really needs stdout and stderr output as
separate strings, you can consider Open3.capture3.

[rosenfeld (Rodrigo Rosenfeld Rosas)]

Also, I'm not sure how I should use IO.select. Looking at the documentation it's not clear to me how I should use it in this case...

[rosenfeld (Rodrigo Rosenfeld Rosas)]

Sorry, haven't seen your message before posting, @akr (Akira Tanaka). Thanks, I surely overlooked the documentation, sorry!

[bitsweat (Jeremy Daer)]

Note you can use Open3.capture3 to safely read stdout and stderr without doing popen3 + IO.select yourself.

Internally, this uses separate threads to read stderr & stdout and to wait for the child process to finish: https://github.com/ruby/ruby/blob/1ec544695fa02d714180ef9c34e755027b6a2103/lib/open3.rb#L257-L273