Feature #18559
closedAllocation tracing: Objects created by the parser are attributed to Kernel.require
Description
Marking this as a feature, because I think it should be improved but can hardly be considered a bug.
Repro¶
Consider the following script:
# /tmp/allocation-source.rb
require 'objspace'
require 'tmpdir'
source = File.join(Dir.tmpdir, "foo.rb")
File.write(source, <<~RUBY)
# frozen_string_literal: true
class Foo
def plop
"fizz"
end
end
RUBY
ObjectSpace.trace_object_allocations_start
GC.start
gen = GC.count
require(source)
ObjectSpace.dump_all(output: $stdout, since: gen)
Expected behavior¶
I'd expect the ObjectSpace.dump_all
output to attribute all new objects, including T_IMEMO
etc, to foo.rb
Actual behavior¶
They are attributed to the source file that called Kernel.require
(so with --disable-gems
):
{"address":"0x11acaec78", "type":"CLASS", "class":"0x11acaebb0", "superclass":"0x10fa4a848", "name":"Foo", "references":["0x10fa4a848", "0x11acaea98", "0x11acaf790"], "file":"/var/folders/vy/srfpq1vn6hv5r6bzkvcw13y80000gn/T/foo.rb", "line":2, "generation":1, "memsize":544, "flags":{"wb_protected":true}}
{"address":"0x11acaeca0", "type":"IMEMO", "class":"0x8", "imemo_type":"cref", "references":["0x10fa4a848"], "file":"/tmp/allocation-source.rb", "line":19, "method":"require", "generation":1, "memsize":40, "flags":{"wb_protected":true}}
{"address":"0x11acaecc8", "type":"STRING", "class":"0x10fa42418", "frozen":true, "embedded":true, "fstring":true, "bytesize":4, "value":"fizz", "encoding":"UTF-8", "file":"/tmp/allocation-source.rb", "line":19, "method":"require", "generation":1, "memsize":40, "flags":{"wb_protected":true}}
{"address":"0x11acaecf0", "type":"ARRAY", "class":"0x10fa28f68", "frozen":true, "length":2, "embedded":true, "references":["0x11acaff88", "0x11acaf240"], "file":"/tmp/allocation-source.rb", "line":19, "method":"require", "generation":1, "memsize":40, "flags":{"wb_protected":true}}
{"address":"0x11acaed18", "type":"IMEMO", "imemo_type":"iseq", "references":["0x11acaecc8", "0x11acaf600", "0x11acaf600", "0x11acaecf0"], "file":"/tmp/allocation-source.rb", "line":19, "method":"require", "generation":1, "memsize":416, "flags":{"wb_protected":true}}
{"address":"0x11acaf1a0", "type":"ARRAY", "class":"0x10fa28f68", "frozen":true, "length":2, "embedded":true, "references":["0x11acaff88", "0x11acaf240"], "file":"/tmp/allocation-source.rb", "line":19, "method":"require", "generation":1, "memsize":40, "flags":{"wb_protected":true}}
{"address":"0x11acaf1c8", "type":"IMEMO", "imemo_type":"iseq", "references":["0x11acaed18", "0x11acaf1f0", "0x11acaf1f0", "0x11acaf1a0", "0x11acaf290"], "file":"/tmp/allocation-source.rb", "line":19, "method":"require", "generation":1, "memsize":456, "flags":{"wb_protected":true}}
{"address":"0x11acaf1f0", "type":"STRING", "class":"0x10fa42418", "frozen":true, "embedded":true, "fstring":true, "bytesize":11, "value":"<class:Foo>", "file":"/tmp/allocation-source.rb", "line":19, "method":"require", "generation":1, "memsize":40, "flags":{"wb_protected":true}}
{"address":"0x11acaf218", "type":"ARRAY", "class":"0x10fa28f68", "frozen":true, "length":2, "embedded":true, "references":["0x11acaff88", "0x11acaf240"], "file":"/tmp/allocation-source.rb", "line":19, "method":"require", "generation":1, "memsize":40, "flags":{"wb_protected":true}}
{"address":"0x11acaf240", "type":"STRING", "class":"0x10fa42418", "frozen":true, "fstring":true, "bytesize":63, "value":"/private/var/folders/vy/srfpq1vn6hv5r6bzkvcw13y80000gn/T/foo.rb", "encoding":"UTF-8", "file":"/tmp/allocation-source.rb", "line":19, "method":"require", "generation":1, "memsize":104, "flags":{"wb_protected":true}}
....
Why is it a problem?¶
This behavior makes it impossible to properly analyze which part of an application use the most memory. For instance when using heap-profiler
on an app using Bootsnap
, all objects created as a result of loading source file are attributed to bootsnap:
retained memory by gem
-----------------------------------
351.64 MB bootsnap-1.10.2
If this behaved as I expect, heap-profiler
would be able to report how much each gem contribute to the app RAM usage.
Updated by byroot (Jean Boussier) over 2 years ago
So I tried adding a frame
before the ISeq is compiled: https://github.com/ruby/ruby/pull/5998
It somewhat works for regular require
but:
- It's awkward to put an uncompiled ISeq in the stack, there is even an
assert
to prevent this. - It doesn't work when loading the ISeq with
load_from_binary
because the source location is inside the ISeq. We could re-order the iseq fields so it's the first loaded element, but still very awkward.
So I tried a much simpler, ad hoc solution which is to store an override on the execution context before starting the compilation: https://github.com/ruby/ruby/pull/6057. If present TracePoint expose that. It's not very clean conceptually, but seem much simpler and safer.
@ko1 (Koichi Sasada) expressed some reservations about it, but I don't see any other solution. Opinions welcome.
Updated by byroot (Jean Boussier) over 2 years ago
@ko1 (Koichi Sasada): Because the precise implementation for it is hard, I like foo.rb:0. I will ask the original poster if the lineno is really important
The lineno
isn't essential, if you think it's too much, I'll still be happy with lineno=0
for objects created by the parser/compiler.
However my current PR seem to be able to provide the correct line number just fine, but maybe as you suggest it adds lots of overhead? Up to you to decide.
Updated by byroot (Jean Boussier) about 2 years ago
I had a quick chat with @ko1 (Koichi Sasada), he plans to try a better implementation after RubyKaigi (mid-september).
I'll hold on this.
Updated by byroot (Jean Boussier) about 2 years ago
- Status changed from Open to Closed
This was implemented by @ko1 (Koichi Sasada) in e35c528d721d209ed8531b10b46c2ac725ea7bf5
I tested it on our system it has the desired effect. Thank you!