Project

General

Profile

Actions

Bug #16284

closed

Net/HTTP consuming multiple times more memory compared to other libraries

Added by zogash (Milovan Zogovic) over 4 years ago. Updated over 4 years ago.

Status:
Closed
Assignee:
-
Target version:
-
ruby -v:
ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-darwin18]
[ruby-core:95588]

Description

I have an issue where Net/HTTP library is causing very high memory usage when requesting for medium to large sized content.

Here is the script to reproduce the issue:

gem 'excon'

require 'uri'
require 'net/http'
require 'excon'

uri = URI('https://file-examples.com/wp-content/uploads/2017/04/file_example_MP4_1920_18MG.mp4')

def print_memory
  mem = `ps -o rss -p #{Process::pid}`.chomp.split("\n").last.strip.to_i
  puts "Memory: #{mem/1024} MB"
end

25.times do
  case ARGV[0]
  when 'EXCON' then Excon.get(uri.to_s)
  when 'NET' then Net::HTTP.get_response(uri)
  end
  print_memory
end

The script downloads 18MB size video file 25 times in a row and prints the process memory. After 25 iterations, Net/HTTP ends up almost 300MB of ram, while Excon only 60MB. Here is the complete result of RAM size the process used after each iteration:

> Excon
Memory: 41 MB
Memory: 60 MB
Memory: 95 MB
Memory: 58 MB
Memory: 95 MB
Memory: 49 MB
Memory: 55 MB
Memory: 51 MB
Memory: 71 MB
Memory: 66 MB
Memory: 58 MB
Memory: 65 MB
Memory: 89 MB
Memory: 76 MB
Memory: 83 MB
Memory: 64 MB
Memory: 75 MB
Memory: 98 MB
Memory: 64 MB
Memory: 58 MB
Memory: 63 MB
Memory: 69 MB
Memory: 75 MB
Memory: 70 MB
Memory: 54 MB

> Net::HTTP
Memory: 51 MB
Memory: 81 MB
Memory: 104 MB
Memory: 129 MB
Memory: 146 MB
Memory: 166 MB
Memory: 163 MB
Memory: 144 MB
Memory: 163 MB
Memory: 161 MB
Memory: 169 MB
Memory: 187 MB
Memory: 204 MB
Memory: 221 MB
Memory: 238 MB
Memory: 255 MB
Memory: 272 MB
Memory: 289 MB
Memory: 306 MB
Memory: 306 MB
Memory: 306 MB
Memory: 306 MB
Memory: 317 MB
Memory: 299 MB
Memory: 294 MB

I have tried with ruby 2.7.0-preview2 and the issue is present there as well.

Actions

Also available in: Atom PDF

Like0
Like0Like0Like0Like0