Project

General

Profile

Feature #5605

[PATCH] net/http: use IO.copy_stream for requests using body_stream

Added by normalperson (Eric Wong) over 8 years ago. Updated over 8 years ago.

Status:
Closed
Priority:
Normal
Target version:
[ruby-core:40898]

Description

This significantly reduces both user and system CPU usage in the
client while making uploads. When using plain HTTP with a
known Content-Length, IO.copy_stream may also use sendfile() to
further reduce user CPU time.

I tested using the following script to upload a 1 gigabyte file.

------------------- net-http-upload.rb -------------------------
require "net/http"
require "benchmark"
require "tempfile"
tmp = Tempfile.new("sparse")
tmp.sysseek(1024 * 1024 * 1024) # 1 gigabyte
tmp.syswrite(".")
host = "127.0.0.1"
port = 1234
res = nil

Net::HTTP.start(host, port) do |http|
put = Net::HTTP::Put.new("/")
tmp.sysseek(0)
put.body_stream = tmp
put.content_length = tmp.size
puts "with Content-Length: #{tmp.size}"
bm = Benchmark.measure { res = http.request(put) }
p [ res, res.body ]
printf("utime=%0.3f stime=%0.3f\n", bm.utime, bm.stime)
end

Net::HTTP.start(host, port) do |http|
put = Net::HTTP::Put.new("/")
tmp.sysseek(0)
put.body_stream = tmp
put["Transfer-Encoding"] = "chunked"
puts "with Transfer-Encoding: chunked"
bm = Benchmark.measure { res = http.request(put) }
p [ res, res.body ]
printf("utime=%0.3f stime=%0.3f\n", bm.utime, bm.stime)
end

------------------------ Results -------------------------------

before

with Content-Length: 1073741825
[#,
"dcdd67a8f07b73796c0485890d48fa697966d09f\n"]
utime=2.660 stime=1.680
with Transfer-Encoding: chunked
[#,
"dcdd67a8f07b73796c0485890d48fa697966d09f\n"]
utime=6.230 stime=1.900

after

with Content-Length: 1073741825
[#,
"dcdd67a8f07b73796c0485890d48fa697966d09f\n"]
utime=0.010 stime=0.410
with Transfer-Encoding: chunked
[#,
"dcdd67a8f07b73796c0485890d48fa697966d09f\n"]
utime=0.320 stime=0.620

----------------------- Server setup --------------------------
Any server that'll accept chunked uploads works, I'm most
familiar with unicorn. I used unicorn 4.1.1 with
rewindable_input disabled, running t/sha1.ru from the Rainbows!
source tree. t/sha1.ru returns the SHA1 of the uploaded body to
verify the upload is sending correct data.

$ git clone git://bogomips.org/rainbows
$ unicorn -E none -p 1234 -c unicorn.conf rainbows/t/sha1.ru

unicorn.conf.rb only had one line to disable rewindable input:

rewindable_input false

This was to prevent disk I/O from slowing the overall runtime
of the test, it doesn't have a significant effect on measured
CPU time in the client process even on the same box.


Files

Also available in: Atom PDF