Feature #16027: Update Ruby's dtrace / USDT API to match what is exposed via the TracePoint API

• Status: Assigned
• Assignee: ko1 (Koichi Sasada)
• Target version: -

[dalehamel (Dale Hamel)]

Abstract

I propose that Ruby's "dtrace" support be extended to match what is available in the TracePoint API, as was the case until feature [Feature #15289] landed.

Background

I will refer to Ruby's "dtrace" bindings as USDT bindings for simplicity, as this is the typo of dtrace probe that they support.

Prior to [Feature #15289] being merged, Ruby's tracepoint API was able to trace only 'all' instances of a type of event.

Ruby added support for tracing ruby with dtrace, and so Ruby's USDT Ruby TracePoint API were "in sync".

Once the Ruby TracePoint API recently added the ability to do filtered tracing in [Feature #15289], it added new functionality but brought the TracePoint and USDT API out of sync.

Currently the TracePoint API is ahead of the USDT API, which presents the problem. There is valuable debug information available, but we do not have
a way to access it with dtrace instrumentation.

Additionally, the recent release of bpftrace adds support for USDT tracing on linux, which makes this a valuable opportunity to be able to use Ruby's TracePoint API in an efficient and targeted way for production tracing. To achieve this, we must synchronize the features of the USDT and TracePoint API.

What is currently lacking is the ability to do filtered, selective tracing as the TracePoint#enable call now supports as per prelude.rb#L141

Proposal

When enabling a TracePoint, users can specify a flag: usdt: [LIST_OF_SIMPLE_TYPES], which will trigger Ruby to also enable the USDT API for when it enables TracePoints.

Within the TracePoint block, users can call tp.fire to send USDT data. So the new default API is:

trace.enable(target: nil, target_line: nil, target_thread: nil: usdt: nil)

And the usage might look like:

trace.enable(target: method(:foo), target_line: 5, usdt: [Integer, String]) do |tp|
  tp.fire(tp.lineno, "Any String I want to send")
end

The types specified must be simple types such as Integer or String, given by their names as constants. When data to the tracepoint, the types must match. If they don't, the tracer won't be able to interpret them properly, but nothing should crash.

Details

I propose that Ruby optionally generate ELF (Linux) or DOF (Darwin) annotations for TracePoint targets when they are enabled.

As ruby is a dynamic language, it cannot do this natively (yet) though Ruby JIT may make this easier, but for now it is not suitable for production use.

To get around this, Ruby can either generate the DOF or ELF stub shared library itself, for example it may do one per class, treating the class as the "provider" for the USDT API, and the methods as tracepoints. This is the approach used by libusdt, which generates DOF usable on Darwin, BSD, and other platforms, and libstapsdt, which generates ELF stubs for use on linux.

When a tracepoint is triggered, the user may be able to call a new API TracePoint#fire, to send data to the Kernel via the USDT API, using the generated ELF stub as a bridge, giving the kernel an address to target in order to receive this data.

Upon enabling a tracepoint, we can either generate these stubs internally, or by linking to an external library that must be enabled at configure time (without this, USDT tracing wouldn't be enabled at all).

It may be possible to use the existing bridge that is used by ruby jit, or have an experimental flag such as --usdt that enables support for generating these stubs.

It may be more consistent with the future Ruby JIT to do this, or else Ruby can generate these stubs by its own native code, but this will require a sort of merging of libusdt and libstapsdt. This would add a dependency to the libelf development header, but that is probably not a problem on Linux platforms.

I would suggest the first approach, if this feature is accepted, would be to try and implement the ELF / DOF generation directly in Ruby. What libstapsdt and libusdt do isn't that complex and could be done in its own C file that probably wouldn't be too large.

Failing that approach, it may be worth investigating the Ruby JIT code to see if a compiler can generate these stubs for us easily. This approach would be to have ruby generate C code that results in the necessary DOF/ELF annotations, and have the compiler pipeline used by ruby JIT to generate the file. This couples the feature to ruby jit though.

Usecase

This feature would be used by dtrace / bpftrace users to debug ruby applications. It may be possible for other platforms to benefit from this too, but I think the main use case is for Linux system administrators and developers to use external debuggers (dtrace/bpftrace) to introspect Ruby's behavior.

Discussion

Pros:

• Syncs the Ruby TracePoint and USDT API
• Allows for much more dynamic and targeted USDT tracing
• Can help to find problems in both development and production
• Can be used for performance and error analysis
• Is better than printing, as emitting/collecting data is only done while a "debugger is attached"

Cons:

• Complexity introduced, in order to generate the ELF/DOF stub files
• Not easily ported to other platforms
• Isn't fully consistent with the current dtrace functionality of Ruby, which is built-in to the VM

Limitation

This will only work on *Nix platforms, and probably just on Linux to start, as that is where most of the benefits are.

If the Ruby JIT approach is preferred or much simpler, then that functionality will be tied to the Ruby JIT functionality.

See also

• https://bpf.sh/usdt-report-doc/index.html a document describing my experimental gem ruby-static-tracing, which prototypes this functionality outside of the RubyVM
• https://bpf.sh/production-breakpoints-doc/index.html a work-in-progress on adding more dynamic method and line based USDT tracing to ruby, built atop ruby-static-tracing now using the ruby tracepoint API.

[shevegen (Robert A. Heiler)]

You refer to koichi in #15289, and to kokobun here, so perhaps they may comment
here in due time. :)

As for running only on *nix - I think there is at the least a ~loose policy that
ruby should run on as many operating systems as possible. I am not sure if I refer
to any existing policy or not (we would have to ask matz for it), but there are examples
of implementations or features that were rejected in the past if they'd only work e. g.
on linux and not on windows. Not sure if this applies to the case here (I don't even
know whether dtrace works on windows), but I thought I should add it to the part about
limitation - then it may have to be a gem perhaps.

[dalehamel (Dale Hamel)]

Hi Robert,

Thank you for your reply and feedback.

To clarify, I don't think that this feature not working on, for instance, Windows is much of a big deal, and perhaps it could be extended to Windows or any other unsupported platforms if there is an equivalent tracer to bpftrace/dtrace that uses a similar format to ELF/DOF.

As it stands now, the existing dtrace/usdt probes are platform specific anyways, and already only work on *nix, so my opinion is that this extends existing platform specific functionality more than it introduces new platform specific functionality. These *nix platforms are already able to use these probes as documented here https://github.com/ruby/ruby/blob/master/doc/dtrace_probes.rdoc and it looks like it was introduced by Aaron Patterson so perhaps he may be interested to comment or give his perspective.

[nobu (Nobuyoshi Nakada)]

• Description updated (diff)

dalehamel (Dale Hamel) wrote:

Once the Ruby TracePoint API recently added the ability to do filtered tracing in [Feature #15289], it added new functionality but brought the TracePoint and USDT API out of sync.

As far as I understand, target provides just an internal filtering, so I wonder how it involves external USDT API.

When enabling a TracePoint, users can specify a flag: usdt: [LIST_OF_SIMPLE_TYPES], which will trigger Ruby to also enable the USDT API for when it enables TracePoints.

Within the TracePoint block, users can call tp.fire to send USDT data. So the new default API is:

What data will the "list of simple types" match against, the arguments
to TracePoint#fire?

[k0kubun (Takashi Kokubun)]

Ruby JIT may make this easier
It may be possible to use the existing bridge that is used by ruby jit
It may be more consistent with the future Ruby JIT to do this
it may be worth investigating the Ruby JIT code to see if a compiler can generate these stubs for us easily
If the Ruby JIT approach is preferred or much simpler

You seem to hope JIT can hook your feature easily, but I believe that's not true. For example, JIT often fallbacks to VM interpretation when some unexpected things like a method redefinition or TracePoint enablement happen. If we implemented your feature using JIT, the introspection would be randomly disabled and sporadic. We should use right tools for the right place. JIT is for optimization, not for debugging.

[dalehamel (Dale Hamel)]

As far as I understand, target provides just an internal filtering, so I wonder how it involves external USDT API.

The external USDT API is "always on" or "always off", it must be enabled for all events of a type or not used at all. I don't believe that the newly added filtering will affect it in any way.

What data will the "list of simple types" match against, the arguments to TracePoint#fire?

Yes, this would just be used if the user specifies a block handler for the Tracepoint, so it must be the user that decides if data will be sent to the Tracepoint (as I cannot think if there is any way to do this dynamically). Unlike the built-in USDT tracepoints, these enabled tracepoints I think would need to be more dynamic. They could also be standardized, so that they use the same types by default as the dtrace events of the type being traced.

For example, for the call event type, the USDT api via dtrace (https://github.com/ruby/ruby/blob/master/doc/dtrace_probes.rdoc) is:

ruby:::method-entry(classname, methodname, filename, lineno);
This probe is fired just before a method is entered.

classname name of the class (a string)
methodname name of the method about to be executed (a string)
filename the file name where the method is being called (a string)
lineno the line number where the method is being called (an int)

So then perhaps for tracepoints enabled on the :call event type the default value for this should be [String, String, String, Integer], and without a block handler, tp.fire would send (classname, methodname, filename, lineno). If a user wants to change what they fire instead, they can override this with a different list of basic types. This keeps it the same as the existing USDT API, while offering the possibility for extension.

You seem to hope JIT can hook your feature easily, but I believe that's not true. For example, JIT often fallbacks to VM interpretation when some unexpected things like a method redefinition or TracePoint enablement happen. If we implemented your feature using JIT, the introspection would be randomly disabled and sporadic. We should use right tools for the right place. JIT is for optimization, not for debugging.

Thank you for pointing this out. Yes, I don't have a lot of confidence in this idea either, but I thought I would still mention it as a potential implementation possibility. If this feature is accepted, I think it would make more sense to do ELF/DOF generation by Ruby instead. I mentioned this mostly because I thought there may be a chance the ruby JIT faculties could simplify implementation, but I would not count on this.

Thank you for your replies and feedback nobu and k0kubun, I hope my response addresses your concerns and clarify the description for you.

[duerst (Martin Dürst)]

@dalehamel (Dale Hamel): Do you have any parts implemented already, or any plans for implementation? Would you be ready to do significant implementation work if this feature got accepted? What parts do you think could be done as a Gem, and what parts would need to be part of the Ruby implementation core? How could this be made more OS-independent (besides Linux, Darvin, and Windows, there are also various BSD flavors,...).

[dalehamel (Dale Hamel)]

Hi Martin,

Do you have any parts implemented already

Yes,I have a prototype gem that adds StaticTracing.tracepoint as a way to define
a stub library that can be used for a debugger (dtrace/bpftrace) to attach
to, similar to the existing TracePoint API. I've been experimenting with using
this, in combination with the TracePoint api, to execute static tracepoints
from within the existing TracePoint handler context.

I've hit a wall, where in order to improve functionality I need more access to context,
which is only available from the RubyVM, and not exposed via the TracePoint API.

I don't yet have a prototype using internal VM context, as I thought it would be
pragmatic to see if there is interest in such functionality before investing the time in
further prototyping.

any plans for implementation?

Yes, but it's still a work in progress as I learn more through prototyping.
Here's what I've got so far, feedback is welcome (and appreciated):

What is implemented needs to be in line with the existing setup of Ruby, and
be minimally intrusive. Adding this functionality should enhance Ruby, not
bog it down and make it more complex.

For this reason, I feel that it makes sense to tie static tracepoints to Ruby's
existing, bulit-in TracePoint API. They appear to operate in a similar way,
and are analogous types of breakpoint. What a TracePoint event is to the RubyVM,
a USDT event is to Kernel.

As Ruby already has USDT tracing built in, I believe it makes sense to target
the same libraries already used. On BSD, these headers are provided by the
system.

On linux, the SDT headers are provided by SystemTap, and do a similar thing.
These headers allow for the RubyVM to build static tracepoints into its own
source code, because Ruby is written in C.

On both platforms, the approach is analagous - the source code has some notes
added to it at a particular location outside of code space, indicating relocation
points for particular addresses in the code from which to read data.

I believe it makes sense to folow on this approach, as it is consistent with
the libraries and toolchain already supported by these built-in VM tracepoints.
This feature is an extension of that, allowing for ruby processes to provide
access to this same debugging metadata that system kernel-based handlers can
read and handle the event.

I think the process would probably go like this:

• Prove out as much as I can with a Ruby Gem prototype
• Contrast this with the smallest possible patches to ruby that I can do to
  achieve or enhance this functionality.
• Build for this feature to be accepted behind a ./configure flag so that it
  can be toggled and optional, as the existing dtrace support is (and perhaps
  update --enable-dtrace to --enable-usdt).

I also want to clarify about JIT: I believe that this would be extremely easy to
enable for Ruby JIT'd objects. As the dtrace macros are headers, it seems like
they should be injected in the C generated by Ruby for a particular instruction
sequence. If that instruction sequence contains a tracepoint, with USDT,
attributes, then the code that prepares the ruby source only needs to add a
single line, and the macro will inject the probe. It looks like dtrace headers
were stubbed in https://github.com/ruby/ruby/commit/49f52937bd9461a677123a16a011c7bc261900a4
but could probably be replaced with actual dtrace probes.

This is why I mention JIT - JIT negates the need for an elf stub, as there is
already a shared object with the executable code - it can use the traditional
header macro approach for injecting USDT probes. By simply ispecting a ruby
process, these JIT probse should already be transparently available.

For full support in this JIT future, we would need to consider how we enable
these tracepoints on both ruby instruction sequences, for non-JIT code, and
their native counterparts.

To support dynamic object, we can use ELF stub libraries or DOF debug annotations.

Would you be ready to do significant implementation work if this feature got accepted?

Yes, that's certainly my intention

What parts do you think could be done as a Gem

I don't know yet, but I'm tring to figure this out. As I said, I've hit a wall
as to what I can get from the existing TracePoint API, so I need to peak inside
and see what I can do with VM context to enhance it.

and what parts would need to be part of the Ruby implementation core?

I don't know where the line is, and it depends mostly on if support for USDT
tracepoints on ruby code (instead of just the VM) is a desirable feature for
ruby Core.

I'd aim to make the changes as small as possible, as it looks like the existing
dtrace integration is pretty minimally intrusive. It would come down to whether
it is better to have the stub generation libraries in Ruby Core, or linked to
externally. I think the latter is probably better.

If that's the case, then some small amount of glue code might be sufficient to
expose the needed context to build a userspace version of this functionality.

How could this be made more OS-independent (besides Linux, Darvin, and Windows, there are also various BSD flavors,...).

Good news, BSD and Solaris already work with these dtrace APIS. Darwin, Solaris,
BSD, Linux would all be pretty easy to support, it's just Windows that would be
tricky.

To support Linux, ELF stubs are generated. To support BSD, the dtrace DOF format
is generated.

[ko1 (Koichi Sasada)]

• Status changed from Open to Assigned
• Assignee set to ko1 (Koichi Sasada)

I need to study USDT...

• I think dtrace feature is useful because they can operate outside of process (do not need to modify observed process). However, your proposal seems to introduce new Ruby APIs to enable them. What do you think about it?
• I can't understand how to use usdt: keyword. Could you show some example scenario with dtrace command line?

[dalehamel (Dale Hamel)]

Thanks for the reply and your work on TracePoints, Koichi.

I need to study USDT...

I can recommend some resources, I might suggest this excerpt of my own writing on this,
as I've been researching this topic a great deal lately. That document is from
a larger 'work in progress' of my findings, though I wouldn't recommend reading
all of it just yet as it needs more editing.

Another good spot is https://awesome-dtrace.com/, which details more
traditional and conventional uses of dtrace for USDT.

I think dtrace feature is useful because they can operate outside of process

Yes, this is the key advantage of USDT tracing versus other approaches. A
standardized debugging approach can be used for different programming languages
(V8 and Python have built-in dtrace support for instance), allowing for the
same tools to be used by experts who work on different languages. This is less
true of Java, which has its own debugging ecosystem as I understand it, but
even Java has USDT support to a degree.

do not need to modify observed process

This is not exactly true, the process has hooks built-into it, which indicate
addresses of statically defined tracepoints.

The kernel can inject a debugging instruction (int3) at this address. When the
code is executed, it will trigger a handler, uprobes in the case of linux,
which can read data that the program emits to the breakpoint.

Unless the program is under observation, there will be no breakpoint
instruction at this address. It depends on how the program handles this
behavior, but usually it will noop and the program executes unmodified.

In the case of the RubyVM, these tracepoints are embedded in the code using
the sys/sdt.h header macros for defining USDT probes for dtrace.

For the case of dynamic code, we must prepare a stub executable, which can
contain the address for the kernel to insert its debugging instruction.

So the process isn't exactly unmodified, but the modifications should be minimal.
At the time when a TracePoint is enabled, a static tracepoint would be generated
and loaded into memory, which acts as the bridge between ruby code and the kernel.

However, your proposal seems to introduce new Ruby APIs to enable them. What do you think about it?

I want to try and keep the API change as clean as possible, so that static
tracepoints are complementary to the builtin tracepoints, as an extension to
use them as a way to attach an external debugger like dtrace or bpftrace to
the process.

I hope my explanation above is what you're looking for, but I'd be happy to
clarify if not.

Could you show some example scenario with dtrace command line?

For basic usage of a static tracepoint:

require 'ruby-static-tracing'

t = StaticTracing::Tracepoint.new('global', 'hello_test', String)
puts t.provider.enable

Signal.trap('USR2') do
  puts "TRAP #{t.enabled?}"
  t.fire('Hello world') if t.enabled?
  sleep 2
end

loop { puts t.enabled?; sleep 1 }

We can then attach to the process by specifying the PID, and using the following dtrace script:

global*:::hello_test
{
  printf("%s\n", copyinstr(arg0))
}

Or the following bpftrace script (linux):

usdt::global:hello_test
{
  printf("%s\n", str(arg0));
}

This is from one of the integration tests. It will do the following:

• Create an ELF (or DOF) stub matching the namespace specified, above I used
  'global'. This corresponds to the name of the shared library stub, which will
  be like global-stub.so.
• The stub will have notes that can be read with readelf --notes indicating
  the address of the tracepoint function and the tracepoint arguments
• A debugger can be attached by specifying "global:hello_test" to find the
  probe information
• Once attached, the loop above will indicate the tracepoint is enabled, as
  the kernel will have overwritten the first byte (safely) with an int3/ 0xCC
  debug instruction (on x86, other platforms support different instructions).
• To simulate the tracepoint being hit, we can send kill -USR2 $(pidof ruby)
  to enter the trap handler, which will fire the probe
• Note that it only fires the probe data if it is enabled, meaning if no
  debugger is attached it doesn't fire the probe. This can be used to guard
  against expensive logic
• The debugger (dtrace/bpftrace) will then output "Hello world", showing the
  transmission of debugging data from ruby to the debugger through the Kernel.

The call to fire is done like a method call, in the case of ruby like a
method call to a native extension. The arguments are thus put on the stack,
and the kernel can grab them in a predictable way. This is why we must indicate
the type when a tracepoint is fired, so the kernel has a way to read off the
arguments and give them to the debugger program.

I can't understand how to use usdt: keyword.

I have rethought the API to try and simplify and clarify.

Maybe a better name for it would by "tracepoint_types" or "arg_types".
Currently on linux USDT probes are limited to 6 arguments, but I think some
platforms support more than 6. The static tracepoint must emit a type that
matches what was declared, and where to read the data for these arguments from.

So, the way this might look if builtin to ruby could be:

trace.enable(target: nil, target_line: nil, target_thread: nil: static_tracepoints: nil)

If static_tracepoints (above i had said just usdt, but maybe static_tracepoints
is more descriptive?) is set, then if we enable a tracepoint without a block,
such as:

trace.enable(target: method(:foo), target_line: 5)

Then we could default to emitting the same tracepoint arguments as are
already emitted,
so a tracepoint could be created and enabled as a member of the TracePoint instance:

@static_tracepoint = StaticTracing::TracePoint.new(:method_foo, :line_5, String, String, String, Integer)

When the tracepoint is hit, it would by default fire off the data that matches
this signature for the default handler:

...
if @static_tracepoint.enabled?
  @static_tracepoint.fire(tp.classname, tp.methodname, tp.filename, tp.lineno)
end
...

Note that @static_tracepoint.enabled? is checking if an address in memory is a
noop (0x90) or breakpoint (0xCC), and guarding actually firing the tracepoint on this.

If a block is specified when creating the tracepoint, then we must specify the
signature of the tracepoint.

This is more powerful, as it allows for any data in the tracepoint debugging
context to be emitted to the user.

For instance, enable with a block may look like:

trace.enable(target: method(:foo), target_line: 5, arg_types: [Integer, String]) do |tp|
  tp.fire(tp.lineno, "Any String I want!")
end

Which would result the dtrace/bpftrace getting output like:

5 Any String I want!

I hope this helps to clarify and improve on my original suggestions, I am sorry
for yet another wall of text, and appreciate the time you've taken in
considering this feature.

[ko1 (Koichi Sasada)]

Thank you for your detailed comment.

dalehamel (Dale Hamel) wrote:

I can recommend some resources, I might suggest this excerpt of my own writing on this,
as I've been researching this topic a great deal lately. That document is from
a larger 'work in progress' of my findings, though I wouldn't recommend reading
all of it just yet as it needs more editing.

Another good spot is https://awesome-dtrace.com/, which details more
traditional and conventional uses of dtrace for USDT.

Thank you. I'll check them later (...when?)

I think dtrace feature is useful because they can operate outside of process

Yes, this is the key advantage of USDT tracing versus other approaches. A
standardized debugging approach can be used for different programming languages
(V8 and Python have built-in dtrace support for instance), allowing for the
same tools to be used by experts who work on different languages. This is less
true of Java, which has its own debugging ecosystem as I understand it, but
even Java has USDT support to a degree.

Could you show us how V8 and Python provide USDT features?

do not need to modify observed process

This is not exactly true, the process has hooks built-into it, which indicate
addresses of statically defined tracepoints.

The kernel can inject a debugging instruction (int3) at this address. When the
code is executed, it will trigger a handler, uprobes in the case of linux,
which can read data that the program emits to the breakpoint.

Unless the program is under observation, there will be no breakpoint
instruction at this address. It depends on how the program handles this
behavior, but usually it will noop and the program executes unmodified.

In the casy of the RubyVM, these tracepoints are embedded in the code using
the sys/sdt.h header macros for defining USDT probes for dtrace.

For the case of dynamic code, we must prepare a stub executable, which can
contain the address for the kernel to insert its debugging instruction.

So the process isn't exactly unmodified, but the modifications should be minimal.
At the time when a TracePoint is enabled, a static tracepoint would be generated
and loaded into memory, which acts as the bridge between ruby code and the kernel.

However, your proposal seems to introduce new Ruby APIs to enable them. What do you think about it?

I want to try and keep the API change as clean as possible, so that static
tracepoints are complementary to the builtin tracepoints, as an extension to
use them as a way to attach an external debugger like dtrace or bpftrace to
the process.

I hope my explanation above is what you're looking for, but I'd be happy to
clarify if not.

If my understanding is correct, dtrace does not need to modify original source code.
I think your extension requires to modify ruby (.rb) source code.
(people need to write TracePoint code)

Is it acceptable?
I don't against your proposal, but I'm afraid how they are useful.

Could you show some example scenario with dtrace command line?

For basic usage of a static tracepoint:

require 'ruby-static-tracing'

t = StaticTracing::Tracepoint.new('global', 'hello_test', String)
puts t.provider.enable

Signal.trap('USR2') do
  puts "TRAP #{t.enabled?}"
  t.fire('Hello world') if t.enabled?
  sleep 2
end

loop { puts t.enabled?; sleep 1 }

This example is easy to understand.
I think it is some kind of logging system, like syslog.
For this purpose, it should be good but maybe TracePoint is not good name...

I can't understand how to use usdt: keyword.

I have rethought the API to try and simplify and clarify.

Maybe a better name for it would by "tracepoint_types" or "arg_types".
Currently on linux USDT probes are limited to 6 arguments, but I think some
platforms support more than 6. The static tracepoint must emit a type that
matches what was declared, and where to read the data for these arguments from.

So, the way this might look if builtin to ruby could be:

trace.enable(target: nil, target_line: nil, target_thread: nil: static_tracepoints: nil)

If static_tracepoints (above i had said just usdt, but maybe static_tracepoints
is more descriptive?) is set, then if we enable a tracepoint without a block,
such as:

trace.enable(target: method(:foo), target_line: 5)

Then we could default to emitting the same tracepoint arguments as are
already emitted,
so a tracepoint could be created and enabled as a member of the TracePoint instance:

@static_tracepoint = StaticTracing::TracePoint.new(:method_foo, :line_5, String, String, String, Integer)

When the tracepoint is hit, it would by default fire off the data that matches
this signature for the default handler:

...
if @static_tracepoint.enabled?
  @static_tracepoint.fire(tp.classname, tp.methodname, tp.filename, tp.lineno)
end
...

Maybe this is bad design.
How about to introduce fire method in some class?

TracePoint.new(...){|tp|
  USDT.fire(info with tp) # or usdt_instance.fire(...)
}

is simpler.

Note that @static_tracepoint.enabled? is checking if an address in memory is a
noop (0x90) or breakpoint (0xCC), and guarding actually firing the tracepoint on this.

BTW, on the Linux system when I checked implementation, they used global variable to detect enable/disable and the checking code was slow.
This is why I don't introduce them aggressively (or I don't introduce them).
Is that changed?

[dalehamel (Dale Hamel)]

Could you show us how V8 and Python provide USDT features?

V8 has ustack helpers (which I'd love to try and figure out how to do for MRI) https://www.joyent.com/blog/understanding-dtrace-ustack-helpers which make the type of probing i'm proposing here a little moot. It is still possible though, here's an example https://www.npmjs.com/package/usdt

Python also had some work on a ustack helper, I'm not sure if it landed though as i can only find an old repo for it. It has other dtrace probes available too https://docs.python.org/3/howto/instrumentation.html but these seem more analagous to what's already in ruby. It is however possible to do this type of dynamic probing in python, like in node, with https://pypi.org/project/stapsdt/ which also uses the ELF stubbing approach.

I think what makes ruby different is that it seems like TracePoint is a natural hook-point for debugging, and I'm not aware if either of python or V8 has something like this exactly.

If my understanding is correct, dtrace does not need to modify original source code.
I think your extension requires to modify ruby (.rb) source code.
(people need to write TracePoint code)
Is it acceptable?
I don't against your proposal, but I'm afraid how they are useful.

I think it's desirable, but I think it might need some further extensions of what is exposed within the tracepoint context (eg, some way to get at more VM internals like what the value the previous statement evaluated to, look at the stack, etc).

I would anticipate that the default block handler for a static tracepoint, extending a similar API to the RubyVM dtrace handlers, would provide some decent value on their own.

I suspect that the real utility of this feature will be to have a userspace libraries of tracepoint helpers, to build up a variety of debugging tools for the automatic handling of a tracepoint. For example, a "recipe" to inspect the value at a particular line, have "entry" and "exit" tracepoints that can be used to measure latency between the execution of lines within the same thread, getting bits of the stack trace, and a general "hook point" for extremely targeted debugging. Some probes may be more expensive to run and comprehensive, and others may be very fast to gather simple data from local variables and fire it off to be measured.

For this purpose, it should be good but maybe TracePoint is not good name...

Can you elaborate?

Maybe this is bad design.

Which aspect of it do you think should change?

How about to introduce fire method in some class?

Update: I updated this section to add a rough idea with references to vm_trace.c

Do You mean that the static tracepoint would not be a private member of TracePoint, but instead should be a different class? I was thinking the usdt hanlder be a member so that it when a tracepoint is enabled, the stub library can be loaded. If this is done inside the tracepoint block, i'm not sure where the stub would be initiated.

I had been thinking a way to try this might be to add a new member, maybe "usdt" to rb_tp_t, then (conditionally) initialize it from rb_tracepoint_enable_for_target.

This would allow for firing static tracepoints whenever ruby's built-in tracepoints are hit, as we can run the handler from tp_call_trace. This means that static tracepoints being fired are an optional side-effect of creating a TracePoint. If a new member "usdt" is added to the tracepoint object, the signature for fire can be specified during userspace 'enable' call of the tracepoint, or else a default proc can be built to handle events and fire only if a debugger is attached. Or users can use with the existing proc handler, and make calls to tp.usdt object for "tp.usdt.fire", and perhaps other methods specific to usdt tracing so that the existing tracepoint API doesn't need to be complicated.

BTW, on the Linux system when I checked implementation, they used global variable to detect enable/disable and the checking code was slow.
This is why I don't introduce them aggressively (or I don't introduce them).
Is that changed?

I'm not sure I know what you are referring to with the global variable / slow check. A reference implementation is here https://github.com/sthima/libstapsdt/blob/master/src/libstapsdt.c#L185-L193

You can see that it is comparing the address of the function pointer to a mask, to ensure that the expected 0x90 is there, as the definition of _fire is done in assembly. dtrace probes can also use a semaphore instead of this check, which should also be a fast operation.

This speed also matters only in so much as 'returning fast' while tracepoints are enabled, but not attached. This enabled check will also of course only be in effect when the ruby TracePoint that they are associated with has been enabled.