Bug #22455
openPer-Ractor GC: an idle Ractor costs ~100 KB of memory (17 KB on 4.0)
Description
Description¶
With the per-Ractor GC, an idle Ractor (blocked in Ractor.receive)
costs ~101 KB of RSS against ~17 KB on 4.0.7. In our server, a
connection's Ractor, with a few threads and a few hundred live objects,
went from ~55–80 KB on 4.0 to ~590 KB on 4.1. 25,000 connections need
~15 GB instead of ~1.4 GB.
The per-Ractor GC removes our main 4.0 problem: every GC used to stop
and mark all Ractors, so its pauses grew with the number of
connections. A Ractor per connection is the shape it should favour.
At this memory cost, that shape is hard to use for many connections.
Is the floor per Ractor (heap pages per size class?) expected, and can
it be lowered, for example by allocating a Ractor's pages lazily or
sharing partly-used pages?
Reproduction¶
# 4,000 Ractors blocked in Ractor.receive: memory each, and the cost of a full GC.
Warning[:experimental] = false
def rss_mb = File.read("/proc/self/status")[/VmRSS:\s+(\d+)/, 1].to_i / 1024.0
def now_ms = Process.clock_gettime(Process::CLOCK_MONOTONIC) * 1000
GC.start
base = rss_mb
ractors = Array.new(4000) { Ractor.new { Ractor.receive } }
sleep 1
puts format("rss per idle Ractor: %.0f KB", (rss_mb - base) * 1024 / ractors.size)
3.times do
t = now_ms
GC.start
puts format("GC.start: %.1f ms", now_ms - t)
end
Results¶
Same machine as above.
| 4.0.7 | 4.1.0-preview1 | |
|---|---|---|
| RSS per idle Ractor | 17 KB | 101 KB |
GC.start (global) with 4,000 idle Ractors |
10–21 ms | 43–141 ms |
The global GC's cost matters together with Bug #22454: if global GCs are
what reclaims exited Ractors, a server that holds thousands of Ractors
pays this pause each time. We saw single global GCs of 0.8–3.9 s, most
of it sweeping, in a server holding 2,000–5,000 connection Ractors.
Updated by peterzhu2118 (Peter Zhu) 16 days ago
- Backport changed from 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: UNKNOWN to 3.3: DONTNEED, 3.4: DONTNEED, 4.0: DONTNEED
Thank you for this bug report. I have a patch here that should reduce the memory used by Ractors that don't allocate many objects. Please give it a try. https://github.com/ruby/ruby/pull/19707
Updated by matteo.folin@gmail.com (Matteo Folin) 4 days ago
I tested PR #19707 (https://github.com/ruby/ruby/pull/19707) against its parent commit on Linux.
Setup
- Patch:
4fa541456c. Baseline:13cebf685b, its parent on master. - Both built from clean source with default
configureoptions, in adebian:trixiecontainer on aarch64-linux (8 CPUs). - The two builds were run alternately, 5 rounds each.
- RSS comes from
VmRSSand peak RSS fromVmHWMin/proc/self/status.
Idle Ractors (the reproduction script from this issue, 4,000 Ractors blocked in Ractor.receive)
| Baseline | Patch | |
|---|---|---|
| RSS per idle Ractor | 101.5 KB (all 5 rounds) | 41.5 KB (all 5 rounds) |
First GC.start |
77–1004 ms, median 179 | 46–182 ms, median 71 |
Following GC.starts |
34–56 ms | 24–33 ms |
The baseline gives the same 101 KB as in the original report. The patch removes about 60% of the per-Ractor cost, but it's still about 2.4× the 17 KB I measured on 4.0.7.
Allocation-heavy Ractors (an attempt at the case raised in the PR review)
8 Ractors, each keeping 1M strings while also producing garbage:
rs = Array.new(8) do
Ractor.new(1_000_000) do |keep|
kept = []
keep.times do |i|
kept << "kept-string-#{i}"
5.times { "garbage-#{i}" * 4 }
end
kept.size
end
end
rs.sum(&:value)
| Baseline | Patch | |
|---|---|---|
Peak RSS (VmHWM) |
728–774 MB, median 738 | 714–773 MB, median 741 |
| RSS at end | 680–718 MB | 668–722 MB |
In this workload I see no difference in peak RSS between an approximation of the benchmark in the review, though,and it ran on aarch64 rather than x86_64. If that script can be shared, I'm happy to run it on the same setup.
On macOS (arm64), the patch also improves idle Ractors and shows no increase in the allocation-heavy case. RSS readings there were too noisy to give exact numbers.