Bug #11526

Streaming HTTP requests are not idempotent and should not be retried

Added by tdg5 (Danny Guinther) almost 5 years ago. Updated about 1 year ago.

Target version:



A colleague of mine ran into an issue today where he discovered that streaming HTTP requests made with Net::HTTP would retry some types of errors without giving any indication that an error had occurred and the request stream had been rewound. This ultimately resulted in a response body that contained an incomplete response followed by the complete response body.

I think when the request is being streamed it cannot be considered idempotent because there's no telling what sort of side effects might occur in the block that the response chunk is yielded to.

I've attached a diff and will create a GitHub PR shortly that aims to deal with this behavior by only retrying the request if a block is not given.

That said, I suspect this issue warrants further discussion.

We are not the first to run into this issue. One of the developers of the aws-sdk gem ran into this same issue in the last 6 months and chose to handle the issue by clearing out the IDEMPOTENT_METHODS_ collection such that no requests would automatically retry. This seems like overkill to me, but makes sense for a minimally evasive monkeypatch.

The author of that patch, Trevor Rowe, suggested that he would create an issue here, but I have been unable to find such an issue.

The related aws-sdk GitHub issue:
The related aws-sdk GitHub commit:

If I can provide any further information on this matter, please let me know.

Thanks in advance for any and all help!


no_retry_http_streams.diff (2.3 KB) no_retry_http_streams.diff Patch that changes Net::HTTP behavior to only retry the request if a block is not given tdg5 (Danny Guinther), 09/14/2015 07:12 PM
trowe-net-http-idempotent-retry-fix.diff (6.52 KB) trowe-net-http-idempotent-retry-fix.diff tdg5 (Danny Guinther), 09/15/2015 05:40 PM
trowe-net-http-idempotent-retry-fix-with-test-fixes.diff (7.42 KB) trowe-net-http-idempotent-retry-fix-with-test-fixes.diff tdg5 (Danny Guinther), 09/15/2015 08:00 PM

Also available in: Atom PDF