https://redmine.ruby-lang.org/https://redmine.ruby-lang.org/favicon.ico?17113305112018-08-07T07:52:06ZRuby Issue Tracking SystemRuby master - Bug #14972: Net::HTTP inconsistently raises EOFError when peer closes the connectionhttps://redmine.ruby-lang.org/issues/14972?journal_id=733522018-08-07T07:52:06Zjoshc (Josh C)josh.nw@gmail.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/73352/diff?detail_id=49611">diff</a>)</li></ul> Ruby master - Bug #14972: Net::HTTP inconsistently raises EOFError when peer closes the connectionhttps://redmine.ruby-lang.org/issues/14972?journal_id=735702018-08-17T09:00:45Znaruse (Yui NARUSE)naruse@airemix.jp
<ul></ul><p>What is the ideal behavior you think? Just below?</p>
<pre><code>diff --git a/lib/net/http/response.rb b/lib/net/http/response.rb
index 66132985d9..7c744d02f4 100644
--- a/lib/net/http/response.rb
+++ b/lib/net/http/response.rb
@@ -290,7 +290,7 @@ def read_body_0(dest)
clen = content_length()
if clen
- @socket.read clen, dest, true # ignore EOF
+ @socket.read clen, dest
return
end
clen = range_length()
</code></pre> Ruby master - Bug #14972: Net::HTTP inconsistently raises EOFError when peer closes the connectionhttps://redmine.ruby-lang.org/issues/14972?journal_id=741292018-09-20T22:40:15Zjoshc (Josh C)josh.nw@gmail.com
<ul></ul><p>If <code>@socket.read clen, dest</code> reads fully <code>clen</code> bytes then that seems ok. But if it can read fewer than <code>clen</code> bytes, then we should keep reading until we read <code>clen</code> bytes or reach EOF.</p> Ruby master - Bug #14972: Net::HTTP inconsistently raises EOFError when peer closes the connectionhttps://redmine.ruby-lang.org/issues/14972?journal_id=764572019-01-22T23:37:30Zjoshc (Josh C)josh.nw@gmail.com
<ul></ul><p>I submitted a PR against trunk: <a href="https://github.com/ruby/ruby/pull/2074" class="external">https://github.com/ruby/ruby/pull/2074</a></p> Ruby master - Bug #14972: Net::HTTP inconsistently raises EOFError when peer closes the connectionhttps://redmine.ruby-lang.org/issues/14972?journal_id=765702019-01-29T13:57:23Znaruse (Yui NARUSE)naruse@airemix.jp
<ul><li><strong>Target version</strong> set to <i>2.7</i></li><li><strong>Backport</strong> changed from <i>2.3: UNKNOWN, 2.4: UNKNOWN, 2.5: UNKNOWN</i> to <i>2.3: DONTNEED, 2.4: DONTNEED, 2.5: DONTNEED</i></li></ul><p>I checked the code again and I noticed I wrote a code which depends current behavior before.<br>
It is to resume with the partially downloaded result.</p>
<p>I consider something like this code with a option or changing the behavior with migration period...</p> Ruby master - Bug #14972: Net::HTTP inconsistently raises EOFError when peer closes the connectionhttps://redmine.ruby-lang.org/issues/14972?journal_id=767702019-02-11T18:32:29Zjoshc (Josh C)josh.nw@gmail.com
<ul></ul><blockquote>
<p>It is to resume with the partially downloaded result.</p>
</blockquote>
<p>Doesn't <code>Net::HTTPResponse#read_body</code> raise if called more than once? How can the caller resume?</p> Ruby master - Bug #14972: Net::HTTP inconsistently raises EOFError when peer closes the connectionhttps://redmine.ruby-lang.org/issues/14972?journal_id=767772019-02-12T05:03:06Znaruse (Yui NARUSE)naruse@airemix.jp
<ul></ul><p>joshc (Josh C) wrote:</p>
<blockquote>
<blockquote>
<p>It is to resume with the partially downloaded result.</p>
</blockquote>
<p>Doesn't <code>Net::HTTPResponse#read_body</code> raise if called more than once? How can the caller resume?</p>
</blockquote>
<p>Save first response body, set range HTTP Header and concat the 2nd response body.</p> Ruby master - Bug #14972: Net::HTTP inconsistently raises EOFError when peer closes the connectionhttps://redmine.ruby-lang.org/issues/14972?journal_id=805062019-08-08T23:18:25Zjoshc (Josh C)josh.nw@gmail.com
<ul></ul><p>When a range is requested, the content-length of the response is the number of bytes in the partial response, so I would still expect an exception to be raised if the partial response is truncated:</p>
<pre><code>$ curl -s -v -r 0-100 -O https://cache.ruby-lang.org/pub/ruby/2.6/ruby-2.6.3.tar.gz
* Trying 151.101.65.178...
...
> GET /pub/ruby/2.6/ruby-2.6.3.tar.gz HTTP/2
> Host: cache.ruby-lang.org
> Range: bytes=0-100
> User-Agent: curl/7.54.0
> Accept: */*
>
...
< content-type: application/x-tar
< server: AmazonS3
< accept-ranges: bytes
< age: 1533346
< content-range: bytes 0-100/16784748
< accept-ranges: bytes
< date: Thu, 08 Aug 2019 23:12:49 GMT
< via: 1.1 varnish
< x-served-by: cache-sea1040-SEA
< x-cache: HIT
< x-cache-hits: 0
< x-timer: S1565305970.902169,VS0,VE0
< content-length: 101
<
{ [101 bytes data]
</code></pre> Ruby master - Bug #14972: Net::HTTP inconsistently raises EOFError when peer closes the connectionhttps://redmine.ruby-lang.org/issues/14972?journal_id=806632019-08-12T21:13:02Zjoshc (Josh C)josh.nw@gmail.com
<ul></ul><p>naruse (Yui NARUSE) wrote:</p>
<blockquote>
<p>joshc (Josh C) wrote:</p>
<blockquote>
<blockquote>
<p>It is to resume with the partially downloaded result.</p>
</blockquote>
<p>Doesn't <code>Net::HTTPResponse#read_body</code> raise if called more than once? How can the caller resume?</p>
</blockquote>
<p>Save first response body, set range HTTP Header and concat the 2nd response body.</p>
</blockquote>
<p>If you save the first response body, and make a new request with <code>Range: bytes=X-Y</code>, then the <code>Content-Length</code> header in the second response should specify the number of bytes to expect, or the <code>Content-Length</code> header should be omitted in the case of chunked encoding. For example, given:</p>
<pre><code>require 'net/http'
require 'uri'
require 'openssl'
uri = URI("http://cache.ruby-lang.org/pub/ruby/2.6/ruby-2.6.3.tar.gz")
http = Net::HTTP.new(uri.host, uri.port)
#http.set_debug_output($stderr)
http.start
begin
pos = 0
req = Net::HTTP::Get.new(uri.path)
req['Accept'] = '*/*'
req['Range'] = "bytes=#{pos}-9"
http.request(req) do |response|
clen = response['Content-Length'].to_i
puts "Content-Length #{clen}"
puts "Content-Range #{response['Content-Range']}"
pos += clen
end
req = Net::HTTP::Get.new(uri.path)
req['Accept'] = '*/*'
req['Range'] = "bytes=#{pos}-#{pos+9}"
http.request(req) do |response|
clen = response['Content-Length'].to_i
puts "Content-Length #{clen}"
puts "Content-Range #{response['Content-Range']}"
pos += clen
end
puts "Downloaded #{pos} bytes"
ensure
http.finish
end
</code></pre>
<p>Produces:</p>
<pre><code>$ ruby range.rb
Content-Length 10
Content-Range bytes 0-9/16784748
Content-Length 10
Content-Range bytes 10-19/16784748
Downloaded 20 bytes
</code></pre>
<p>In other words, if the <code>Content-Length</code> is specified, it should always specify the number of bytes to read (or drain) from the socket. <a href="https://tools.ietf.org/html/rfc7230#section-3.4" class="external">https://tools.ietf.org/html/rfc7230#section-3.4</a> specifically says:</p>
<blockquote>
<p>A client that receives an incomplete response message, which can<br>
occur when a connection is closed prematurely or when decoding a<br>
supposedly chunked transfer coding fails, MUST record the message as<br>
incomplete.<br>
...<br>
A message that uses a valid Content-Length is incomplete<br>
if the size of the message body received (in octets) is less than the<br>
value given by Content-Length.</p>
</blockquote>
<p>So it should never be ok to silently ignore EOF.</p> Ruby master - Bug #14972: Net::HTTP inconsistently raises EOFError when peer closes the connectionhttps://redmine.ruby-lang.org/issues/14972?journal_id=832862019-12-20T08:00:27Znaruse (Yui NARUSE)naruse@airemix.jp
<ul><li><strong>Target version</strong> deleted (<del><i>2.7</i></del>)</li></ul> Ruby master - Bug #14972: Net::HTTP inconsistently raises EOFError when peer closes the connectionhttps://redmine.ruby-lang.org/issues/14972?journal_id=906712021-03-01T21:20:40Zjeremyevans0 (Jeremy Evans)merch-redmine@jeremyevans.net
<ul><li><strong>Status</strong> changed from <i>Open</i> to <i>Assigned</i></li><li><strong>Assignee</strong> set to <i>naruse (Yui NARUSE)</i></li></ul><p>I added a pull request that offers a way to support this in a backwards compatible manner: <a href="https://github.com/ruby/net-http/pull/15" class="external">https://github.com/ruby/net-http/pull/15</a>.</p> Ruby master - Bug #14972: Net::HTTP inconsistently raises EOFError when peer closes the connectionhttps://redmine.ruby-lang.org/issues/14972?journal_id=973302022-04-20T06:44:51Zjeremyevans (Jeremy Evans)code@jeremyevans.net
<ul><li><strong>Status</strong> changed from <i>Assigned</i> to <i>Closed</i></li></ul><p>Applied in changeset <a class="changeset" title="[ruby/net-http] Add ignore_eof access to HTTP and HTTPResponse The ignore_eof setting on HTTPRes..." href="https://redmine.ruby-lang.org/projects/ruby-master/repository/git/revisions/90ccc5674affb64036ca86f566c8650f654fc0f6">git|90ccc5674affb64036ca86f566c8650f654fc0f6</a>.</p>
<hr>
<p>[ruby/net-http] Add ignore_eof access to HTTP and HTTPResponse</p>
<p>The ignore_eof setting on HTTPResponse makes it so an EOFError is<br>
raised when reading bodies with a defined Content-Length, if the<br>
body read was truncated due to the socket be closed.</p>
<p>The ignore_eof setting on HTTP sets the values used in responses<br>
that are created by the object.</p>
<p>For backwards compatibility, the default is for both settings is<br>
true. However, unless you are specifically tested for and handling<br>
truncated responses, it's a good idea to set ignore_eof to false so<br>
that errors are raised for truncated responses, instead of those<br>
errors silently being ignored.</p>
<p>Fixes [Bug <a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: Net::HTTP inconsistently raises EOFError when peer closes the connection (Closed)" href="https://redmine.ruby-lang.org/issues/14972">#14972</a>]</p>
<p><a href="https://github.com/ruby/net-http/commit/4d47e34995" class="external">https://github.com/ruby/net-http/commit/4d47e34995</a></p>