Project

General

Profile

Bug #9082

popen3 hangs when stderr gets lots of output

Added by rosenfeld (Rodrigo Rosenfeld Rosas) almost 6 years ago. Updated over 3 years ago.

Status:
Rejected
Priority:
Normal
Assignee:
-
Target version:
-
ruby -v:
ruby 2.0.0p247 (2013-06-27 revision 41674) [x86_64-linux]
Backport:
[ruby-core:58176]

Description

Create this program test.rb:

STDERR.puts "some error line\n" * 10_000

Then, try this:

ruby -r open3 -e "Open3.popen3('ruby test.rb'){|i,o,e,t|i.close;o.read}"

For this particular case, if I do "e.read" before "o.read" it works. But for my real case involving the "tidy" command with lots of warnings in the stderr it will freeze even if I call "e.read". Since I don't really need this output, I'm using popen2 and redirecting stderr to /dev/null, so this is not urgent, but I thought you'd like to know about this.

Also, if you replace 10_000 with 1_000 above it also works here.

History

Updated by jeremyevans0 (Jeremy Evans) almost 6 years ago

I think this is expected behavior, not a bug. Your test.rb program blocks outputting to stderr because the pipe buffer gets filled. The reason it works for less output to stderr is that the program exits before the stderr pipe buffer is full. The reason it works if you read stderr before stdout is that you drain the stderr pipe buffer, allowing the program to exit.

Updated by charliesome (Charlie Somerville) almost 6 years ago

  • Status changed from Open to Rejected

Jeremy is correct, this is not a bug in popen3.

To properly handle things like this, look into IO.select.

Updated by rosenfeld (Rodrigo Rosenfeld Rosas) almost 6 years ago

Will it also block on STDOUT? If so, I suppose people shouldn't use the open3 library for interacting with commands like 'tidy' and several others that will process files (even less than 1MB)

I don't see this limitation in the docs nor any API for dealing with such cases...

Updated by charliesome (Charlie Somerville) almost 6 years ago

Rodrigo: Yes, this is how UNIX works.

Updated by rosenfeld (Rodrigo Rosenfeld Rosas) almost 6 years ago

At least with the block form. I think this should be documented if this is the case.

Updated by rosenfeld (Rodrigo Rosenfeld Rosas) almost 6 years ago

Also, is it expected that even if e.read is called inside the block it will hang up?

Updated by charliesome (Charlie Somerville) almost 6 years ago

If you call e.read then test.rb is able to write the entire message to STDERR then exit. When it exits, both stderr and stdout are closed so e.read returns.

Updated by rosenfeld (Rodrigo Rosenfeld Rosas) almost 6 years ago

I see, but usually a process will write both to STDOUT and STDERR so it may happen that both buffers will be full. What should one do in such case?

Updated by charliesome (Charlie Somerville) almost 6 years ago

You need to use IO.select to handle that.

Updated by rosenfeld (Rodrigo Rosenfeld Rosas) almost 6 years ago

I'm assuming that if I use popen2 or redirect all STDOUT to /dev/null I don't need IO.select, right?

Updated by akr (Akira Tanaka) almost 6 years ago

rosenfeld (Rodrigo Rosenfeld Rosas) wrote:

At least with the block form. I think this should be documented if this is the case.

The document already has following description:

% ri Open3.popen3
...
You should be careful to avoid deadlocks. Since pipes are fixed length buffer,
Open3.popen3("prog") {|i, o, e, t| o.read } deadlocks if the program generates
many output on stderr. You should be read stdout and stderr simultaneously
(using thread or IO.select). However if you don't need stderr output,
Open3.popen2 can be used. If merged stdout and stderr output is not a problem,
you can use Open3.popen2e. If you really needs stdout and stderr output as
separate strings, you can consider Open3.capture3.

Updated by rosenfeld (Rodrigo Rosenfeld Rosas) almost 6 years ago

Also, I'm not sure how I should use IO.select. Looking at the documentation it's not clear to me how I should use it in this case...

Updated by rosenfeld (Rodrigo Rosenfeld Rosas) almost 6 years ago

Sorry, haven't seen your message before posting, @akr. Thanks, I surely overlooked the documentation, sorry!

Updated by bitsweat (Jeremy Daer) over 3 years ago

Note you can use Open3.capture3 to safely read stdout and stderr without doing popen3 + IO.select yourself.

Internally, this uses separate threads to read stderr & stdout and to wait for the child process to finish: https://github.com/ruby/ruby/blob/1ec544695fa02d714180ef9c34e755027b6a2103/lib/open3.rb#L257-L273

Also available in: Atom PDF