Bug #20972: OpenSSL Memory Usage

• Status: Closed
• Assignee: -
• Target version: -
• ruby -v: 3.3.6
• Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN

[mrkgrandjean (Mark Grandjean)]

While testing large file uploads I noticed the OpenSSL Buffering is allocating a lot of memory for file uploads.
Similar to the issue in this ticket from a few years ago https://bugs.ruby-lang.org/issues/14426

Using the same test code from that ticket

require "http"
require "memory_profiler"
require "stringio"

body = StringIO.new("a" * 5*1024*1024)

MemoryProfiler.report do
  HTTP.post("https://example.com", body: body)
end.pretty_print

in ruby 2.6.10

allocated memory by gem
-----------------------------------
   1913287  unicode_normalize
   1053463  llhttp-ffi-0.5.0
    187849  http-5.2.0
     50168  openssl
     36180  addressable-2.8.7
      8904  ipaddr
       272  other

While testing in Ruby 3.3.6

allocated memory by gem
-----------------------------------
  16104976  openssl
   2393456  unicode_normalize
   1053880  llhttp-ffi-0.5.0
    151633  http-5.2.0
      9813  addressable-2.8.7
      7872  ipaddr
       240  other
        40  forwardable

It looks like the memory allocation issue has returned.

[byroot (Jean Boussier)]

• Related to Feature #14426: [PATCH] openssl: reduce memory allocation in OpenSSL::Buffering#do_write added

[luke-gru (Luke Gruber)]

I took a look at this and this is due to 2 things that I could notice:

The call to string.b if it's not encoded as binary already, in the Buffer class in openssl/buffering.rb. The other one is a call to
rb_str_new_frozen in ossl_ssl_write_internal.

Getting rid of the rb_str_new_frozen call brings the allocated memory down to ~ 5639000 and removing the other one gets it down to
42840.

I'll leave it to someone with more familiarity with the openssl codebase to provide a good fix, if it's possible, possibly using locktmp or something for
the frozen case. Maybe it's also possible to just force the encoding as binary for the provided string instead of duplicating it.

[byroot (Jean Boussier)]

I think you are mostly correct, what exactly cause the memory growth is hard to point precisely (especially with memory_profiler because this codepath depends a lot on shared strings, so it's not fully clear which call exactly cause the unsharing.

But you are right that using rb_str_locktmp would avoid some allocations and probably some copying, and there are a few other small changes that can be made to rely less on string sharing, and reduce allocations a bit: https://github.com/ruby/openssl/pull/831

[byroot (Jean Boussier)]

• Status changed from Open to Closed

Applied in changeset git|2f5d31d38ad6918410da1c41936e043f47725d4f.

---

[ruby/openssl] Reduce OpenSSL::Buffering#do_write overhead

[Bug #20972]

The rb_str_new_freeze was added in https://github.com/ruby/openssl/issues/452
to better handle concurrent use of a Socket, but SSL sockets can't be used
concurrently AFAIK, so we might as well just error cleanly.

By using rb_str_locktmp we can ensure attempts at concurrent write
will raise an error, be we avoid causing a copy of the bytes.

We also use the newer String#append_as_bytes method when available
to save on some more copies.

https://github.com/ruby/openssl/commit/0d8c17aa85

Co-Authored-By: [e-mail]