Feature #22236: New API for adjustable JIT warmup

• Status: Open
• Assignee: jit
• Target version: -

[byroot (Jean Boussier)]

Context

YJIT, and probably ZJIT in the near future, do provide a significant latency reduction once warmed up,
however, during the warmup phase they tend to cause significant overhead, which temporary degrade the service latency, because of the compilation overhead.

Thankfully, ZJIT warmup is fast, but still, for project using continuous delivery, this warmup phase can be restarted several times and hour,
and is a bit disruptive.

Prior Art

Currently, the only way to smooth the compilation overhead over time is to adjust --yjit-call-threshold which defines how many time a method need to be called before it is considered for compilation.

This is useful, but hard to reason about, and can easily be invalidated by code changes. It's also not quite granular enough, because many mehtods are called once per cycle, hence will all become eligible during the same cycle.

Proposal

Since both JITs keep track of how long they spent in the compiler, I propose that they accept an "adjustable compilation time budget".
Concretely something like RubyVM::YJIT.max_compile_time_ns = 1_000_000_000.

• Whenever the JIT is done compiling a block and it increments RubyVM::YJIT.runtime_stats(:compile_time_ns), if compile_time_ns > max_compile_time_ns then the JIT disable itself.
• Whenever max_compile_time_ns is re-assigned, if it's larger than compile_time_ns the JIT re-enable itself.

Expected Use Cases

Such API would allow services using one process per request to allocate a compilation budget at the start of a cycle (request, job, etc):

def call(env)
  RubyVM::YJIT.max_compile_time_ns = RubyVM::YJIT.runtime_stats(:compile_time_ns) + 20_000_000 # 20ms budget
  @app.call(env)
end

For services handling work concurrently, a time based approach can be taken:

Thread.new do
  loop do
     # 20ms budget every 100ms
    RubyVM::YJIT.max_compile_time_ns = RubyVM::YJIT.runtime_stats(:compile_time_ns) + 20_000_000
    sleep 0.1
  end
end

These two example are way easier to reason about than configuring call-threshold and don't need frequent re-tuning.

Considerations

This API is meant as best effort. I'm aware that at least YJIT isn't able to full disable compilation, as it leave compilation stubs and have little choice but to compile them when hit.

But the "disabled" state doesn't have to be absolute, it simply is a good signal for the JIT to know to delay as much work as possible for later.

I also know that JITs don't know how long they'll spend in compilation when they enter it, it's absolutely fine if they go over the set limit slightly, as long as they disable themselves right after.

[k0kubun (Takashi Kokubun)]

• Assignee set to jit