Misc #17050

profiler gem

Added by ioquatix (Samuel Williams) 6 months ago. Updated 5 months ago.



I would like to reuse profiler gem namespace:

It currently conflicts with profile/lib/profiler.rb however this is no longer part of stdlib.

cc ko1 (Koichi Sasada)

Updated by ioquatix (Samuel Williams) 6 months ago

Also I'd be happy to receive/resuse the profile namespace too but either one is fine and the profile gem actually has an interesting implementation.

Updated by duerst (Martin Dürst) 6 months ago

Can you tell us shortly what you plan to use that gem/namespace for? If it's something that can (in some reasonable way) replace the (former?) profile/profiler functionality, then please go for it. At one point, I was planning to work on this, but I didn't find the time.

Updated by ioquatix (Samuel Williams) 6 months ago

My initial goal is to expose some general data structures for efficiently capturing large memory profiles on top of ObjectSpace and GC modules, similar to how memory_profiler gem works. I don't mind making separate gem, but it seems like the profile or profiler gem would be ideal and have not been updated for a long time. Initially I was thinking of making separate gems and trying to provide a unified interface, e.g.

  • memory_profiler -> profiler-memory
  • stackprof -> profiler-stack
  • ruby-prof -> profiler-sample


I think that's too ambitious initially, but yeah, that's a longer term idea, with an initial focus on profiler-memory of which I've already got working code and just looking how to structure it best for longer term integration.

A sideways step would be to provide rspec-memory hooks for using these profilers either for general summary or assertions around blocks of code (of which I already maintain the rspec-memory gem which does some parts of this).

ko1 (Koichi Sasada) said he wants to use profile gem, but I'm not sure if he also intended to use profiler gem. So, ko1 (Koichi Sasada), can I use one of them, and you can use the other?

Updated by Eregon (Benoit Daloze) 6 months ago

I'd recommend starting with something specific like profiler-memory.

profiler is very generic but therefore also unclear and will make people think it's a stdlib-like profiler, but I guess it will not be.

I also think it's too early to tell if your gem will be "the profiler everyone uses".

BTW, stackprof is a CPU sampling profiler, and ruby-prof seems a tracing CPU profiler.

As a side note it'd be nice if profiler-like gems work on more than 1 implementation.
I think most them only work on CRuby, because they use C APIs that are efficient on CRuby but basically inefficient on other implementations (see for details regarding stackprof & TruffleRuby).

Updated by ioquatix (Samuel Williams) 6 months ago

I agree all your points. I already have working profiler-memory to a certain extent.

Updated by Eregon (Benoit Daloze) 6 months ago

In the profile stdlib, both files are used: -rprofile starts the profiler immediately, while profiler.rb is like a library:

So it seems to me not advisable to reuse those "require names" for something not strictly compatible since profile was only removed in 2.7.

Updated by byroot (Jean Boussier) 6 months ago

@ioquatrix I’m curious. What’s the difference between your memory-profiler and the existing memory_profiler?

For context I’ve been working on for the last week, and I’m also very interested in improving ObjectSpace and GC APIs

Updated by ioquatix (Samuel Williams) 6 months ago

I was able to profile 200+ million* allocations with my fork which was impossible with the current implementations. Right now my current plan is to upstream these changes to memory_profiler. (*sorry, incorrectly said billions previously :)


Updated by jeremyevans0 (Jeremy Evans) 6 months ago

  • Backport deleted (2.5: UNKNOWN, 2.6: UNKNOWN, 2.7: UNKNOWN)
  • Tracker changed from Bug to Misc

Updated by ko1 (Koichi Sasada) 5 months ago

  • Status changed from Open to Feedback

Sorry for late response.

I have a plan to remake current profile library with recent APIs.

Updated by ioquatix (Samuel Williams) 5 months ago

  • Status changed from Feedback to Closed

Okay, I look forward to your profiler.

Also available in: Atom PDF