Project

General

Profile

Feature #4142

multipart/form-data for net/http

Added by naruse (Yui NARUSE) almost 9 years ago. Updated about 8 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Target version:
-
[ruby-dev:42729]

Description

=begin
multipart/form-data 対応を net/http に入れませんか。
追加される API は Net::HTTPRequest#set_form になります。

akr さんからは multipart/form-data 用のデータを出力する API 案も示唆されたのですが、
chunked encoding を考慮に入れるとうまくまとまらなかったので見送っています。

diff --git a/lib/net/http.rb b/lib/net/http.rb
index 4d475b1..2751f77 100644
--- a/lib/net/http.rb
+++ b/lib/net/http.rb
@@ -22,6 +22,7 @@
require 'net/protocol'
autoload :OpenSSL, 'openssl'
require 'uri'
+autoload :SecureRandom, 'securerandom'

module Net #:nodoc:

@@ -1772,7 +1773,8 @@ module Net #:nodoc:
alias content_type= set_content_type

  # Set header fields and a body from HTML form data.
  • # +params+ should be a Hash containing HTML form data.
  • # +params+ should be an Array of Arrays or
  • # a Hash containing HTML form data.
    # Optional argument +sep+ means data record separator.
    #
    # Values are URL encoded as necessary and the content-type is set to
    @@ -1792,6 +1794,48 @@ module Net #:nodoc:

    alias form_data= set_form_data

  • # Set a HTML form data set.

  • # +params+ is the form data set; it is an Array of Arrays or a Hash

  • # +enctype is the type to encode the form data set.

  • # It is application/x-www-form-urlencoded or multipart/form-data.

  • # +formpot+ is an optional hash to specify the detail.

  • #

  • # boundary:: the boundary of the multipart message

  • # charset:: the charset of the message. All names and the values of

  • # non-file fields are encoded as the charset.

  • #

  • # Each item of params is an array and contains following items:

  • # +name+:: the name of the field

  • # +value+:: the value of the field, it should be a String or a File

  • # +opt+:: an optional hash to specify additional information

  • #

  • # Each item is a file field or a normal field.

  • # If +value+ is a File object or the +opt+ have a filename key,

  • # the item is treated as a file field.

  • #

  • # If Transfer-Encoding is set as chunked, this send the request in

  • # chunked encoding. Because chunked encoding is HTTP/1.1 feature,

  • # you must confirm the server to support HTTP/1.1 before sending it.

  • #

  • # Example:

  • # http.set_form("q", "ruby"], ["lang", "en")

  • #

  • # See also RFC 2388, RFC 2616, HTML 4.01, and HTML5

  • #

  • def set_form(params, enctype='application/x-www-form-urlencoded', formopt={})

  •  @body_data = params
    
  •  @body = nil
    
  •  @body_stream = nil
    
  •  @form_option = formopt
    
  •  case enctype
    
  •  when /\Aapplication\/x-www-form-urlencoded\z/i,
    
  •    /\Amultipart\/form-data\z/i
    
  •    self.content_type = enctype
    
  •  else
    
  •    raise ArgumentError, "invalid enctype: #{enctype}"
    
  •  end
    
  • end
    +
    # Set the Authorization: header for "Basic" authorization.
    def basic_auth(account, password)
    @header['authorization'] = [basic_encode(account, password)]
    @@ -1849,6 +1893,7 @@ module Net #:nodoc:
    self['User-Agent'] ||= 'Ruby'
    @body = nil
    @body_stream = nil

  •  @body_data = nil
    

    end

    attr_reader :method
    @@ -1876,6 +1921,7 @@ module Net #:nodoc:
    def body=(str)
    @body = str
    @body_stream = nil

  •  @body_data = nil
    str
    

    end

@@ -1884,6 +1930,7 @@ module Net #:nodoc:
def body_stream=(input)
@body = nil
@body_stream = input

  • @body_data = nil input end

@@ -1901,6 +1948,8 @@ module Net #:nodoc:
send_request_with_body sock, ver, path, @body
elsif @body_stream
send_request_with_body_stream sock, ver, path, @body_stream

  • elsif @body_data
  •    send_request_with_body_data sock, ver, path, @body_data
    else
      write_header sock, ver, path
    end
    

    @@ -1935,6 +1984,92 @@ module Net #:nodoc:
    end
    end

  • def send_request_with_body_data(sock, ver, path, params)

  •  if /\Amultipart\/form-data\z/i !~ self.content_type
    
  •    self.content_type = 'application/x-www-form-urlencoded'
    
  •    return send_request_with_body(sock, ver, path, URI.encode_www_form(params))
    
  •  end
    

    +

  •  opt = @form_option.dup
    
  •  opt[:boundary] ||= SecureRandom.urlsafe_base64(40)
    
  •  self.set_content_type(self.content_type, boundary: opt[:boundary])
    
  •  if chunked?
    
  •    write_header sock, ver, path
    
  •    encode_multipart_form_data(sock, params, opt)
    
  •  else
    
  •    require 'tempfile'
    
  •    file = Tempfile.new('multipart')
    
  •    encode_multipart_form_data(file, params, opt)
    
  •    file.rewind
    
  •    self.content_length = file.size
    
  •    write_header sock, ver, path
    
  •    IO.copy_stream(file, sock)
    
  •  end
    
  • end
    +

  • def encode_multipart_form_data(out, params, opt)

  •  charset = opt[:charset]
    
  •  boundary = opt[:boundary]
    
  •  boundary ||= SecureRandom.urlsafe_base64(40)
    
  •  chunked_p = chunked?
    

    +

  •  buf = ''
    
  •  params.each do |key, value, h={}|
    
  •    key = quote_string(key, charset)
    
  •    filename =
    
  •      h.key?(:filename) ? h[:filename] :
    
  •      value.respond_to?(:to_path) ? File.basename(value.to_path) :
    
  •      nil
    

    +

  •    buf << "--#{boundary}\r\n"
    
  •    if filename
    
  •      filename = quote_string(filename, charset)
    
  •      type = h[:content_type] || 'application/octet-stream'
    
  •      buf << "Content-Disposition: form-data; " \
    
  •        "name=\"#{key}\"; filename=\"#{filename}\"\r\n" \
    
  •        "Content-Type: #{type}\r\n\r\n"
    
  •      if !out.respond_to?(:write) || !value.respond_to?(:read)
    
  •        # if +out+ is not an IO or +value+ is not an IO
    
  •        buf << (value.respond_to?(:read) ? value.read : value)
    
  •      elsif value.respond_to?(:size) && chunked_p
    
  •        # if +out+ is an IO and +value+ is a File, use IO.copy_stream
    
  •        flush_buffer(out, buf, chunked_p)
    
  •        out << "%x\r\n" % value.size if chunked_p
    
  •        IO.copy_stream(value, out)
    
  •        out << "\r\n" if chunked_p
    
  •      else
    
  •        # +out+ is an IO, and +value+ is not a File but an IO
    
  •        flush_buffer(out, buf, chunked_p)
    
  •        1 while flush_buffer(out, value.read(4096), chunked_p)
    
  •      end
    
  •    else
    
  •      # non-file field:
    
  •      #   HTML5 says, "The parts of the generated multipart/form-data
    
  •      #   resource that correspond to non-file fields must not have a
    
  •      #   Content-Type header specified."
    
  •      buf << "Content-Disposition: form-data; name=\"#{key}\"\r\n\r\n"
    
  •      buf << (value.respond_to?(:read) ? value.read : value)
    
  •    end
    
  •    buf << "\r\n"
    
  •  end
    
  •  buf << "--#{boundary}--\r\n"
    
  •  flush_buffer(out, buf, chunked_p)
    
  •  out << "0\r\n\r\n" if chunked_p
    
  • end
    +

  • def quote_string(str, charset)

  •  str = str.encode(charset, fallback:->(c){'&#%d;'%c.encode("UTF-8").ord}) if charset
    
  •  str = str.gsub(/[\\"]/, '\\\\\&')
    
  • end
    +

  • def flush_buffer(out, buf, chunked_p)

  •  return unless buf
    
  •  out << "%x\r\n"%buf.bytesize if chunked_p
    
  •  out << buf
    
  •  out << "\r\n" if chunked_p
    
  •  buf.clear
    
  • end
    +
    def supply_default_content_type
    return if content_type()
    warn 'net/http: warning: Content-Type did not set; using application/x-www-form-urlencoded' if $VERBOSE
    diff --git a/lib/net/protocol.rb b/lib/net/protocol.rb
    index 2a6cfb4..a3ffa71 100644
    --- a/lib/net/protocol.rb
    +++ b/lib/net/protocol.rb
    @@ -168,6 +168,8 @@ module Net # :nodoc:
    }
    end

  • alias << write
    +
    def writeline(str)
    writing {
    write0 str + "\r\n"
    diff --git a/test/net/http/test_http.rb b/test/net/http/test_http.rb
    index 76280ad..12c03a4 100644
    --- a/test/net/http/test_http.rb
    +++ b/test/net/http/test_http.rb
    @@ -303,6 +303,102 @@ module TestNetHTTP_version_1_2_methods
    assert_equal data.size, res.body.size
    assert_equal data, res.body
    end
    +

  • def test_set_form

  • require 'tempfile'

  • file = Tempfile.new('ruby-test')

  • file << "\u{30c7}\u{30fc}\u{30bf}"

  • data = [

  •  ['name', 'Gonbei Nanashi'],
    
  •  ['name', "\u{540d}\u{7121}\u{3057}\u{306e}\u{6a29}\u{5175}\u{885b}"],
    
  •  ['s"i\o', StringIO.new("\u{3042 3044 4e9c 925b}")],
    
  •  ["file", file, filename: "ruby-test"]
    
  • ]

  • expected = <<"EOM".gsub(/\n/, "\r\n")
    +--
    +Content-Disposition: form-data; name="name"
    +
    +Gonbei Nanashi
    +--
    +Content-Disposition: form-data; name="name"
    +
    +\xE5\x90\x8D\xE7\x84\xA1\xE3\x81\x97\xE3\x81\xAE\xE6\xA8\xA9\xE5\x85\xB5\xE8\xA1\x9B
    +--
    +Content-Disposition: form-data; name="s\"i\\o"
    +
    +\xE3\x81\x82\xE3\x81\x84\xE4\xBA\x9C\xE9\x89\x9B
    +--
    +Content-Disposition: form-data; name="file"; filename="ruby-test"
    +Content-Type: application/octet-stream
    +
    +\xE3\x83\x87\xE3\x83\xBC\xE3\x82\xBF
    +----
    +EOM

  • start {|http|

  •  _test_set_form_urlencoded(http, data.reject{|k,v|!v.is_a?(String)})
    
  •  _test_set_form_multipart(http, false, data, expected)
    
  •  _test_set_form_multipart(http, true, data, expected)
    
  • }

  • end
    +

  • def _test_set_form_urlencoded(http, data)

  • req = Net::HTTP::Post.new('/')

  • req.set_form(data)

  • res = http.request req

  • assert_equal "name=Gonbei+Nanashi&name=%E5%90%8D%E7%84%A1%E3%81%97%E3%81%AE%E6%A8%A9%E5%85%B5%E8%A1%9B", res.body

  • end
    +

  • def _test_set_form_multipart(http, chunked_p, data, expected)

  • data.each{|k,v|v.rewind rescue nil}

  • req = Net::HTTP::Post.new('/')

  • req.set_form(data, 'multipart/form-data')

  • req['Transfer-Encoding'] = 'chunked' if chunked_p

  • res = http.request req

  • body = res.body

  • assert_match(/\A--(?\S+)/, body)

  • /\A--(?\S+)/ =~ body

  • expected = expected.gsub(//, boundary)

  • assert_equal(expected, body)

  • end
    +

  • def test_set_form_with_file

  • require 'tempfile'

  • file = Tempfile.new('ruby-test')

  • file << $test_net_http_data

  • filename = File.basename(file.to_path)

  • data = 'file', file

  • expected = <<"EOM".gsub(/\n/, "\r\n")
    +--
    +Content-Disposition: form-data; name="file"; filename=""
    +Content-Type: application/octet-stream
    +
    +
    +----
    +EOM

  • expected.sub!(//, filename)

  • expected.sub!(//, $test_net_http_data)

  • start {|http|

  •  data.each{|k,v|v.rewind rescue nil}
    
  •  req = Net::HTTP::Post.new('/')
    
  •  req.set_form(data, 'multipart/form-data')
    
  •  res = http.request req
    
  •  body = res.body
    
  •  header, _ = body.split(/\r\n\r\n/, 2)
    
  •  assert_match(/\A--(?<boundary>\S+)/, body)
    
  •  /\A--(?<boundary>\S+)/ =~ body
    
  •  expected = expected.gsub(/<boundary>/, boundary)
    
  •  assert_match(/^--(?<boundary>\S+)\r\n/, header)
    
  •  assert_match(
    
  •    /^Content-Disposition: form-data; name="file"; filename="#{filename}"\r\n/,
    
  •    header)
    
  •  assert_equal(expected, body)
    

    +

  •  data.each{|k,v|v.rewind rescue nil}
    
  •  req['Transfer-Encoding'] = 'chunked'
    
  •  res = http.request req
    
  •  #assert_equal(expected, res.body)
    
  • }

  • end
    end

class TestNetHTTP_version_1_1 < Test::Unit::TestCase
=end

History

#1

Updated by naruse (Yui NARUSE) almost 9 years ago

  • Status changed from Open to Closed
  • % Done changed from 0 to 100

=begin
This issue was solved with changeset r30188.
Yui, thank you for reporting this issue.
Your contribution to Ruby is greatly appreciated.
May Ruby be with you.

=end

Updated by nahi (Hiroshi Nakamura) about 8 years ago

  • Status changed from Closed to Open

Naruse-san, would you please add an explanation of this feature to NEWS file?

Updated by nahi (Hiroshi Nakamura) about 8 years ago

すいませんruby-devだった。orz

成瀬さん、NEWSになんか書きませんか。

Updated by naruse (Yui NARUSE) about 8 years ago

  • Status changed from Open to Closed

r32241 で書きました。

Also available in: Atom PDF