Bug #22511
open
Please backport ee90e8e380 (concurrent set size overcount) to 4.0: fstring table keeps doubling in long-running apps
Description
Commit ee90e8e380 ("Concurrent set fix when encountering garbage obj in find_or_insert", GitHub PR #16701) is in master but not in any 4.0.x release, including the latest 4.0.9.
Without it, the fstring table in a long-running web process keeps doubling its capacity under steady load.
We'd like to ask for it to be backported to 4.0.
What we see¶
The process handles a steady stream of requests.
Each request uses a few dozen unfrozen Strings as Hash keys, drawn from a few thousand distinct values.
rb_hash_key_str interns each one, and each interned copy dies shortly after the request.
After upgrading from 3.4 to 4.0, memory grows in steps.
Each step is a single anonymous mapping that doubles in size: +8, +16, +32, +64 MB, with the interval between steps doubling as well.
After about 10 hours a process holds a single 256 MB block, i.e. 16M concurrent_set entries, while the live fstring count is well under 1% of that.
On 3.4 the same workload shows no step growth.
Cause¶
This matches the overcount described in ee90e8e380.
When rb_concurrent_set_find_or_insert finds a garbage key with the same hash during lazy sweep, it CASes the key to EMPTY and re-inserts with size++, without the matching size--.
When the dead string is swept later, rb_concurrent_set_delete_by_identity no longer finds it, so the count is never corrected.
With the same strings re-interned constantly, size drifts upward.
At each 75% load-factor resize, expected_size = size - deleted_entries is inflated and the capacity doubles.
Verification¶
Same application and same load, side by side for 25 minutes.
The only difference is that the second Ruby is 4.0.9 built from the release tarball with ee90e8e380 applied (it applies cleanly).
| minute | 4.0.9 RSS | 4.0.9 + ee90e8e380 RSS |
|---|---|---|
| 6 | 273.2 MB | 275.3 MB |
| 7 | 281.2 MB (+8) | 276.6 MB |
| 18 | 283.3 MB | 279.3 MB |
| 19 | 299.3 MB (+16) | 279.4 MB |
| 25 | 301.2 MB | 285.5 MB |
Freezing the strings before using them as Hash keys also removes the steps on unpatched 4.0, which confirms the fstring table as the source.
We could not reproduce this with a small standalone script.
In synthetic loops only a few percent of interns land during lazy sweep, far fewer than in a large application heap.
We can rerun our test against a 4.0 backport branch if that helps.
Workaround: freeze strings before using them as Hash keys.
Related: Feature #21268 (concurrent fstring table), GitHub PR #16701.
No data to display