Feature #6612
closed
Add streaming inflate and deflate to Zlib
Description
=begin
Currently there is no way to control the inflate output size of a deflate stream from ruby. For example, 50MB of "0"s compress to just under 50KB:
ruby -rzlib -e 'p Zlib.deflate("0" * 50_000_000, Zlib::BEST_COMPRESSION).length' #=> 48611
When inflating this, 50MB are allocated which is undesirable.
The attached patch allows Zlib::Inflate#inflate, Zlib::Deflate#deflate, Zlib::ZStream#finish and other methods that end up calling zstream_expand_buffer or zstream_detach_buffer to be called with a block which gives the user more control over the amount of memory allocated in their process. (A fixed maximum chunk size of 16384 bytes is used in this patch.)
The new API looks like:
z = Zlib::Inflate.new
z.inflate deflate_string do |chunk|
# write chunk to output stream
# nil is returned from inflate
end
footer = z.finish
flush buffer to output stream¶
Here's a comparison of resource usage:
$ dd if=/dev/zero of=/dev/stdout bs=1m count=1024 | gzip -c > 1G.gz
$ cat test.rb
require 'zlib'
gzipped = File.read '1G.gz'
z = Zlib::Inflate.new Zlib::MAX_WBITS + 32
z.inflate gzipped do |chunk|
# do nothing with chunk
# current ruby will ignore this block
end
z.finish
With existing ruby (trunk 35758) 1GB of memory is allocated:
$ /usr/bin/time -l ruby20 test.rb
3.86 real 3.24 user 0.61 sys
1080475648 maximum resident set size
0 average shared memory size
0 average unshared data size
0 average unshared stack size
263860 page reclaims
0 page faults
0 swaps
0 block input operations
0 block output operations
0 messages sent
0 messages received
0 signals received
1 voluntary context switches
34 involuntary context switches
With the patch only 37MB are allocated:
$ /usr/bin/time -l ./ruby20 -I .ext/x86_64-darwin11.4.0 test.rb
3.47 real 3.43 user 0.03 sys
36724736 maximum resident set size
0 average shared memory size
0 average unshared data size
0 average unshared stack size
8981 page reclaims
0 page faults
0 swaps
0 block input operations
9 block output operations
0 messages sent
0 messages received
0 signals received
1 voluntary context switches
90 involuntary context switches
Some notes about this patch:
zstream_expand_buffer() yields each block of inflate (or deflate) output instead of expanding the buffer (as in non-block output). zstream_expand_buffer_into() is used to reduce duplication as well.
zstream_detach_buffer() yields the buffer and returns nil if a block was given. OBJ_INFECT was moved into zstream_detach_buffer() as well.
A new stream flag ZSTREAM_FLAG_GZFILE is added to prevent a block given to a gzip method that invokes zstream_expand_buffer() or zstream_detach_buffer() from yielding.
To ensure proper handling of ((|z->buf|)) such as resetting the buffer after yielding a chunk, rb_protect is used in zstream_expand_buffer and zstream_run. In zstream_run, the for loop has been extracted to zstream_run_loop.
In zstream_run_loop, Zlib::BufError is no longer raised when flushing (finishing) the stream. This allows the stream to be flushed into one string when the inflate block is interrupted.
=end
Files
Updated by 
Updated by