Project

General

Profile

Misc #17050

profiler gem

Added by ioquatix (Samuel Williams) 12 days ago. Updated 10 days ago.

Status:
Open
Priority:
Normal
[ruby-core:99319]

Description

I would like to reuse profiler gem namespace:

https://rubygems.org/gems/profiler/versions

It currently conflicts with profile/lib/profiler.rb however this is no longer part of stdlib.

cc ko1 (Koichi Sasada)

Updated by ioquatix (Samuel Williams) 12 days ago

Also I'd be happy to receive/resuse the profile namespace too but either one is fine and the profile gem actually has an interesting implementation.

Updated by duerst (Martin Dürst) 12 days ago

Can you tell us shortly what you plan to use that gem/namespace for? If it's something that can (in some reasonable way) replace the (former?) profile/profiler functionality, then please go for it. At one point, I was planning to work on this, but I didn't find the time.

Updated by ioquatix (Samuel Williams) 12 days ago

My initial goal is to expose some general data structures for efficiently capturing large memory profiles on top of ObjectSpace and GC modules, similar to how memory_profiler gem works. I don't mind making separate gem, but it seems like the profile or profiler gem would be ideal and have not been updated for a long time. Initially I was thinking of making separate gems and trying to provide a unified interface, e.g.

  • memory_profiler -> profiler-memory
  • stackprof -> profiler-stack
  • ruby-prof -> profiler-sample

etc.

I think that's too ambitious initially, but yeah, that's a longer term idea, with an initial focus on profiler-memory of which I've already got working code and just looking how to structure it best for longer term integration.

A sideways step would be to provide rspec-memory hooks for using these profilers either for general summary or assertions around blocks of code (of which I already maintain the rspec-memory gem which does some parts of this).

ko1 (Koichi Sasada) said he wants to use profile gem, but I'm not sure if he also intended to use profiler gem. So, ko1 (Koichi Sasada), can I use one of them, and you can use the other?

Updated by Eregon (Benoit Daloze) 12 days ago

I'd recommend starting with something specific like profiler-memory.

profiler is very generic but therefore also unclear and will make people think it's a stdlib-like profiler, but I guess it will not be.

I also think it's too early to tell if your gem will be "the profiler everyone uses".

BTW, stackprof is a CPU sampling profiler, and ruby-prof seems a tracing CPU profiler.

As a side note it'd be nice if profiler-like gems work on more than 1 implementation.
I think most them only work on CRuby, because they use C APIs that are efficient on CRuby but basically inefficient on other implementations (see https://github.com/oracle/truffleruby/issues/2044#issuecomment-654848324 for details regarding stackprof & TruffleRuby).

Updated by ioquatix (Samuel Williams) 12 days ago

I agree all your points. I already have working profiler-memory to a certain extent.

Updated by Eregon (Benoit Daloze) 12 days ago

In the profile stdlib, both files are used: -rprofile starts the profiler immediately, while profiler.rb is like a library:
https://github.com/ruby/ruby/blob/ruby_2_6/lib/profile.rb

So it seems to me not advisable to reuse those "require names" for something not strictly compatible since profile was only removed in 2.7.

Updated by byroot (Jean Boussier) 12 days ago

@ioquatrix I’m curious. What’s the difference between your memory-profiler and the existing memory_profiler?

For context I’ve been working on https://github.com/shopify/heap-profiler for the last week, and I’m also very interested in improving ObjectSpace and GC APIs

Updated by ioquatix (Samuel Williams) 11 days ago

I was able to profile 200+ million* allocations with my fork which was impossible with the current implementations. Right now my current plan is to upstream these changes to memory_profiler. (*sorry, incorrectly said billions previously :)

#9

Updated by jeremyevans0 (Jeremy Evans) 10 days ago

  • Backport deleted (2.5: UNKNOWN, 2.6: UNKNOWN, 2.7: UNKNOWN)
  • Tracker changed from Bug to Misc

Also available in: Atom PDF