Misc #17050: profiler gem

• Status: Closed
• Assignee: hsbt (Hiroshi SHIBATA)

[ioquatix (Samuel Williams)]

I would like to reuse profiler gem namespace:

https://rubygems.org/gems/profiler/versions

It currently conflicts with profile/lib/profiler.rb however this is no longer part of stdlib.

cc @ko1 (Koichi Sasada)

[ioquatix (Samuel Williams)]

Also I'd be happy to receive/resuse the profile namespace too but either one is fine and the profile gem actually has an interesting implementation.

[duerst (Martin Dürst)]

Can you tell us shortly what you plan to use that gem/namespace for? If it's something that can (in some reasonable way) replace the (former?) profile/profiler functionality, then please go for it. At one point, I was planning to work on this, but I didn't find the time.

[ioquatix (Samuel Williams)]

My initial goal is to expose some general data structures for efficiently capturing large memory profiles on top of ObjectSpace and GC modules, similar to how memory_profiler gem works. I don't mind making separate gem, but it seems like the profile or profiler gem would be ideal and have not been updated for a long time. Initially I was thinking of making separate gems and trying to provide a unified interface, e.g.

• memory_profiler -> profiler-memory
• stackprof -> profiler-stack
• ruby-prof -> profiler-sample

etc.

I think that's too ambitious initially, but yeah, that's a longer term idea, with an initial focus on profiler-memory of which I've already got working code and just looking how to structure it best for longer term integration.

A sideways step would be to provide rspec-memory hooks for using these profilers either for general summary or assertions around blocks of code (of which I already maintain the rspec-memory gem which does some parts of this).

@ko1 (Koichi Sasada) said he wants to use profile gem, but I'm not sure if he also intended to use profiler gem. So, @ko1 (Koichi Sasada), can I use one of them, and you can use the other?

[Eregon (Benoit Daloze)]

I'd recommend starting with something specific like profiler-memory.

profiler is very generic but therefore also unclear and will make people think it's a stdlib-like profiler, but I guess it will not be.

I also think it's too early to tell if your gem will be "the profiler everyone uses".

BTW, stackprof is a CPU sampling profiler, and ruby-prof seems a tracing CPU profiler.

As a side note it'd be nice if profiler-like gems work on more than 1 implementation.
I think most them only work on CRuby, because they use C APIs that are efficient on CRuby but basically inefficient on other implementations (see https://github.com/oracle/truffleruby/issues/2044#issuecomment-654848324 for details regarding stackprof & TruffleRuby).

[ioquatix (Samuel Williams)]

I agree all your points. I already have working profiler-memory to a certain extent.

[Eregon (Benoit Daloze)]

In the profile stdlib, both files are used: -rprofile starts the profiler immediately, while profiler.rb is like a library:
https://github.com/ruby/ruby/blob/ruby_2_6/lib/profile.rb

So it seems to me not advisable to reuse those "require names" for something not strictly compatible since profile was only removed in 2.7.

[byroot (Jean Boussier)]

@ioquatrix I’m curious. What’s the difference between your memory-profiler and the existing memory_profiler?

For context I’ve been working on https://github.com/shopify/heap-profiler for the last week, and I’m also very interested in improving ObjectSpace and GC APIs

[ioquatix (Samuel Williams)]

I was able to profile 200+ million* allocations with my fork which was impossible with the current implementations. Right now my current plan is to upstream these changes to memory_profiler. (*sorry, incorrectly said billions previously :)

[jeremyevans0 (Jeremy Evans)]

• Tracker changed from Bug to Misc
• Backport deleted (2.5: UNKNOWN, 2.6: UNKNOWN, 2.7: UNKNOWN)

[ko1 (Koichi Sasada)]

• Status changed from Open to Feedback

Sorry for late response.

I have a plan to remake current profile library with recent APIs.

[ioquatix (Samuel Williams)]

• Status changed from Feedback to Closed

Okay, I look forward to your profiler.