Project

General

Profile

Actions

Feature #18559

closed

Allocation tracing: Objects created by the parser are attributed to Kernel.require

Added by byroot (Jean Boussier) almost 3 years ago. Updated about 2 years ago.

Status:
Closed
Assignee:
-
Target version:
-
[ruby-core:107389]

Description

Marking this as a feature, because I think it should be improved but can hardly be considered a bug.

Repro

Consider the following script:

# /tmp/allocation-source.rb
require 'objspace'
require 'tmpdir'

source = File.join(Dir.tmpdir, "foo.rb")
File.write(source, <<~RUBY)
  # frozen_string_literal: true
  class Foo
    def plop
      "fizz"
    end
  end
RUBY

ObjectSpace.trace_object_allocations_start

GC.start
gen = GC.count
require(source)
ObjectSpace.dump_all(output: $stdout, since: gen)

Expected behavior

I'd expect the ObjectSpace.dump_all output to attribute all new objects, including T_IMEMO etc, to foo.rb

Actual behavior

They are attributed to the source file that called Kernel.require (so with --disable-gems):

{"address":"0x11acaec78", "type":"CLASS", "class":"0x11acaebb0", "superclass":"0x10fa4a848", "name":"Foo", "references":["0x10fa4a848", "0x11acaea98", "0x11acaf790"], "file":"/var/folders/vy/srfpq1vn6hv5r6bzkvcw13y80000gn/T/foo.rb", "line":2, "generation":1, "memsize":544, "flags":{"wb_protected":true}}
{"address":"0x11acaeca0", "type":"IMEMO", "class":"0x8", "imemo_type":"cref", "references":["0x10fa4a848"], "file":"/tmp/allocation-source.rb", "line":19, "method":"require", "generation":1, "memsize":40, "flags":{"wb_protected":true}}
{"address":"0x11acaecc8", "type":"STRING", "class":"0x10fa42418", "frozen":true, "embedded":true, "fstring":true, "bytesize":4, "value":"fizz", "encoding":"UTF-8", "file":"/tmp/allocation-source.rb", "line":19, "method":"require", "generation":1, "memsize":40, "flags":{"wb_protected":true}}
{"address":"0x11acaecf0", "type":"ARRAY", "class":"0x10fa28f68", "frozen":true, "length":2, "embedded":true, "references":["0x11acaff88", "0x11acaf240"], "file":"/tmp/allocation-source.rb", "line":19, "method":"require", "generation":1, "memsize":40, "flags":{"wb_protected":true}}
{"address":"0x11acaed18", "type":"IMEMO", "imemo_type":"iseq", "references":["0x11acaecc8", "0x11acaf600", "0x11acaf600", "0x11acaecf0"], "file":"/tmp/allocation-source.rb", "line":19, "method":"require", "generation":1, "memsize":416, "flags":{"wb_protected":true}}
{"address":"0x11acaf1a0", "type":"ARRAY", "class":"0x10fa28f68", "frozen":true, "length":2, "embedded":true, "references":["0x11acaff88", "0x11acaf240"], "file":"/tmp/allocation-source.rb", "line":19, "method":"require", "generation":1, "memsize":40, "flags":{"wb_protected":true}}
{"address":"0x11acaf1c8", "type":"IMEMO", "imemo_type":"iseq", "references":["0x11acaed18", "0x11acaf1f0", "0x11acaf1f0", "0x11acaf1a0", "0x11acaf290"], "file":"/tmp/allocation-source.rb", "line":19, "method":"require", "generation":1, "memsize":456, "flags":{"wb_protected":true}}
{"address":"0x11acaf1f0", "type":"STRING", "class":"0x10fa42418", "frozen":true, "embedded":true, "fstring":true, "bytesize":11, "value":"<class:Foo>", "file":"/tmp/allocation-source.rb", "line":19, "method":"require", "generation":1, "memsize":40, "flags":{"wb_protected":true}}
{"address":"0x11acaf218", "type":"ARRAY", "class":"0x10fa28f68", "frozen":true, "length":2, "embedded":true, "references":["0x11acaff88", "0x11acaf240"], "file":"/tmp/allocation-source.rb", "line":19, "method":"require", "generation":1, "memsize":40, "flags":{"wb_protected":true}}
{"address":"0x11acaf240", "type":"STRING", "class":"0x10fa42418", "frozen":true, "fstring":true, "bytesize":63, "value":"/private/var/folders/vy/srfpq1vn6hv5r6bzkvcw13y80000gn/T/foo.rb", "encoding":"UTF-8", "file":"/tmp/allocation-source.rb", "line":19, "method":"require", "generation":1, "memsize":104, "flags":{"wb_protected":true}}
....

Why is it a problem?

This behavior makes it impossible to properly analyze which part of an application use the most memory. For instance when using heap-profiler on an app using Bootsnap, all objects created as a result of loading source file are attributed to bootsnap:

retained memory by gem
-----------------------------------
 351.64 MB  bootsnap-1.10.2

If this behaved as I expect, heap-profiler would be able to report how much each gem contribute to the app RAM usage.

Actions #1

Updated by byroot (Jean Boussier) over 2 years ago

  • Description updated (diff)

Updated by byroot (Jean Boussier) over 2 years ago

So I tried adding a frame before the ISeq is compiled: https://github.com/ruby/ruby/pull/5998

It somewhat works for regular require but:

  • It's awkward to put an uncompiled ISeq in the stack, there is even an assert to prevent this.
  • It doesn't work when loading the ISeq with load_from_binary because the source location is inside the ISeq. We could re-order the iseq fields so it's the first loaded element, but still very awkward.

So I tried a much simpler, ad hoc solution which is to store an override on the execution context before starting the compilation: https://github.com/ruby/ruby/pull/6057. If present TracePoint expose that. It's not very clean conceptually, but seem much simpler and safer.

@ko1 (Koichi Sasada) expressed some reservations about it, but I don't see any other solution. Opinions welcome.

Updated by byroot (Jean Boussier) over 2 years ago

@ko1 (Koichi Sasada): Because the precise implementation for it is hard, I like foo.rb:0. I will ask the original poster if the lineno is really important

The lineno isn't essential, if you think it's too much, I'll still be happy with lineno=0 for objects created by the parser/compiler.

However my current PR seem to be able to provide the correct line number just fine, but maybe as you suggest it adds lots of overhead? Up to you to decide.

Updated by byroot (Jean Boussier) about 2 years ago

I had a quick chat with @ko1 (Koichi Sasada), he plans to try a better implementation after RubyKaigi (mid-september).

I'll hold on this.

Updated by byroot (Jean Boussier) about 2 years ago

  • Status changed from Open to Closed

This was implemented by @ko1 (Koichi Sasada) in e35c528d721d209ed8531b10b46c2ac725ea7bf5

I tested it on our system it has the desired effect. Thank you!

Actions

Also available in: Atom PDF

Like0
Like0Like0Like0Like0Like0