Bug #15468
closedNet::Protocol::BufferedIO#write raises NoMethodError when sending large multi-byte string
Description
ruby-2.6.0で大きなJSON文字列をNet::HTTPでPOSTすると以下のように NoMothodError が発生するようになりました。
$ ruby -rnet/http -rjson -v -e "Net::HTTP.post(URI('http://httpbin.org/post'), { text: 'あ'*100_000 }.to_json, 'Content-Type' => 'application/json')"
ruby 2.6.0p0 (2018-12-25 revision 66547) [x86_64-linux]
Traceback (most recent call last):
19: from -e:1:in `<main>'
18: from lib/ruby/2.6.0/net/http.rb:500:in `post'
17: from lib/ruby/2.6.0/net/http.rb:605:in `start'
16: from lib/ruby/2.6.0/net/http.rb:920:in `start'
15: from lib/ruby/2.6.0/net/http.rb:502:in `block in post'
14: from lib/ruby/2.6.0/net/http.rb:1281:in `post'
13: from lib/ruby/2.6.0/net/http.rb:1493:in `send_entity'
12: from lib/ruby/2.6.0/net/http.rb:1479:in `request'
11: from lib/ruby/2.6.0/net/http.rb:1506:in `transport_request'
10: from lib/ruby/2.6.0/net/http.rb:1506:in `catch'
9: from lib/ruby/2.6.0/net/http.rb:1507:in `block in transport_request'
8: from lib/ruby/2.6.0/net/http/generic_request.rb:123:in `exec'
7: from lib/ruby/2.6.0/net/http/generic_request.rb:189:in `send_request_with_body'
6: from lib/ruby/2.6.0/net/protocol.rb:247:in `write'
5: from lib/ruby/2.6.0/net/protocol.rb:265:in `writing'
4: from lib/ruby/2.6.0/net/protocol.rb:248:in `block in write'
3: from lib/ruby/2.6.0/net/protocol.rb:275:in `write0'
2: from lib/ruby/2.6.0/net/protocol.rb:275:in `each_with_index'
1: from lib/ruby/2.6.0/net/protocol.rb:275:in `each'
lib/ruby/2.6.0/net/protocol.rb:280:in `block in write0': undefined method `bytesize' for nil:NilClass (NoMethodError)
理由は、Net::Protocol::BufferedIO#write0で、String#[]で送信されていない文字列を求める際に文字列にマルチバイト文字が含まれているとIO#write_nonblockで返ってくる送信されたバイト数が文字数を超えるため、nilになるためだと考えています。https://github.com/ruby/ruby/pull/2058 を送ってみました。
Updated by rafaelfranca (Rafael França) almost 6 years ago
Can we get this issue closed and the with right backport field values?
Updated by nobu (Nobuyoshi Nakada) almost 6 years ago
- Status changed from Open to Closed
- Backport changed from 2.4: UNKNOWN, 2.5: UNKNOWN, 2.6: UNKNOWN to 2.4: UNKNOWN, 2.5: UNKNOWN, 2.6: REQUIRED
Updated by hsbt (Hiroshi SHIBATA) almost 6 years ago
- Backport changed from 2.4: UNKNOWN, 2.5: UNKNOWN, 2.6: REQUIRED to 2.4: DONTNEED, 2.5: DONTNEED, 2.6: REQUIRED
Updated by naruse (Yui NARUSE) almost 6 years ago
- Backport changed from 2.4: DONTNEED, 2.5: DONTNEED, 2.6: REQUIRED to 2.4: DONTNEED, 2.5: DONTNEED, 2.6: DONE
ruby_2_6 r66799 merged revision(s) 66582.
Updated by ioquatix (Samuel Williams) almost 6 years ago
Can we please fast track the release of this? It's breaking a ton of stuff.
Stepping back a little bit from this issue, I think we see a picture of Ruby where:
- Handling binary data can cause bugs like this (mismatch between binary and encoded string).
- Reading and writing strings is tricky, e.g. https://bugs.ruby-lang.org/issues/13626
- There are some odd edge cases which can cause problems, e.g. https://bugs.ruby-lang.org/issues/14975
- Reading and writing can put pressure on GC even though no additional strings should be allocated, see https://bugs.ruby-lang.org/issues/15560 for ideas to fix this.
I feel like some improvements to handling binary data could be a very useful addition to Ruby.
I welcome further discussion here and on the above issues so that we can avoid issues like this in the future. Ideally, we provide APIs such that this kind of bug is impossible.