Bug #9035: [proposal] new RUBY_GC_HEAP_GROWTH_MAX_OBJ tuning parameter

• Status: Closed
• Assignee: ko1 (Koichi Sasada)
• Target version: 2.1.0
• ruby -v: ruby 2.1.0dev
• Backport: 1.9.3: UNKNOWN, 2.0.0: UNKNOWN

[tmm1 (Aman Karmani)]

=begin
= Background

Currently, whenever the Ruby GC runs out of object slots the heap is grown by 1.8x (((%GC_HEAP_GROWTH_FACTOR%))).
This works well for small programs, but given a large application it will cause rapid increase in memory.
For example, if a ruby program is using 1 million objects and runs out of slots, it will grow to 1.8 million objects.
This is a huge increase.

= Proposal

Introduce ((%RUBY_GC_HEAP_GROWTH_MAX_OBJ%)) to avoid exponential growth in larger ruby applications.
After this limit is reached, heap growth will happen linearly instead of exponentially.
For example, if ((%growth_max=100k%)) then the application above will grow from 1mm objects to 1.1mm objects.

= Discussion

((%RUBY_GC_HEAP_GROWTH_MAX%)) can be represented either in a) bytesize, or b) number of objects.

Arguments for (a)

• more user friendly, since users are exposed primarily to RSS and want to control final RSS value
• a bytesize based formula can include more varibles, for example the size of bitmaps associated with each (({RVALUE}))

Arguments for (b)

• consistent with existing tuning parameters (all object based, i.e. RUBY_HEAP_MIN_SLOTS)
• GC tuning already requires measuring "startup objects", "objects per request", so object based is more friendly
• bytesize is confusing because ((%GROWTH_MAX%)) only controls growth of ruby heap, not total RSS
• this is an advanced feature meant for use by experts (default value will be 0 = no change from behavior)
  since experts will be using it, we can assume they have already measured the number of objects in their ruby app first
  =end

[nobu (Nobuyoshi Nakada)]

• Description updated (diff)

[tmm1 (Aman Karmani)]

I would prefer (b) above. Exposing a bytesize based limit is just going to confuse users, who will expect the variable to limit their overall RSS. Since the limit only affects the ruby heap, the impact on overall RSS is quite minimal. For example, a large rails application can be 400MB RSS, but only 30MB of that will be allocated by the ruby heap.

A user might set RUBY_GC_HEAP_GROWTH_MAX_BYTESIZE=1MB to control RSS growth, but this will only limit the number of new slots added to the ruby heap. If the first new slot is occupied by a 100MB String object, then RSS will still grow by 101MB. Using RUBY_GC_HEAP_GROWTH_MAX_OBJS instead makes it more clear what the intention is, and forces the user to tune based on the number of objects their application typically uses.

I have prepared a patch here: https://github.com/tmm1/ruby/compare/heap-slots-growth-max.diff. If there are no objections, I will merge it.

[tmm1 (Aman Karmani)]

• Status changed from Open to Closed
• % Done changed from 0 to 100

This issue was solved with changeset r43409.
Aman, thank you for reporting this issue.
Your contribution to Ruby is greatly appreciated.
May Ruby be with you.

---

• gc.c: add new initial_growth_max tuning parameter. [ruby-core:57928] [Bug #9035]