Feature #20282: Enhancing Ruby's Coverage with Per-Test Coverage Reports

• Status: Open
• Assignee: -
• Target version: -

[ioquatix (Samuel Williams)]

As Ruby applications grow in complexity, the need for more sophisticated testing and coverage analysis tools becomes paramount. Current coverage tools in Ruby offer a good starting point but fall short in delivering the granularity and flexibility required by modern development practices. Specifically, there is a significant gap in "per-test coverage" reporting, which limits developers' ability to pinpoint exactly which tests exercise which lines of code. This proposal seeks to initiate a discussion around improving Ruby's coverage module to address this gap.

Objectives

The primary goal of this initiative is to introduce support for per-test coverage reports within Ruby, focusing on three key areas:

1. Scoped Coverage Data Capture: Implementing the capability to capture coverage data within user-defined scopes, such as global, thread, or fiber scopes. This would allow for more granular control over the coverage analysis process.

2. Efficient Data Capture Controls: Developing mechanisms to efficiently control the capture of coverage data. This includes the ability to exclude specific files, include/ignore/merge eval'd code, to ensure that the coverage data is both relevant and manageable.

3. Compatibility and Consistency: Ensuring that the coverage data is exposed in a manner that is consistent with existing coverage tools and standards. This compatibility is crucial for integrating with a wide array of tooling and for facilitating a seamless developer experience.

Proposed Solutions

The heart of this proposal lies in the introduction of a new subclassable component within the Coverage module, tentatively named Coverage::Capture. This component would allow users to define custom coverage capture behaviors tailored to their specific needs. Below is a hypothetical interface for such a mechanism:

class Coverage::Capture
  def self.start
    self.new.tap(&:start)
  end
  
  # Start receiving coverage callbacks.
  def start
  end

  # Stop receiving coverage callbacks.
  def stop
  end

  # User-overridable statement coverage callback.
  def statement(iseq, location)
    fetch(iseq)&.statement_coverage.increment(location)
  end

  # Additional methods for branch/declaration coverage would follow a similar pattern.
end

class MyCoverageCapture < Coverage::Capture
  # Provides efficient data capture controls - can return nil if skipping coverage for this iseq, or can store coverage data per-thread, per-fiber, etc.
  def fetch(iseq)
    @coverage[iseq] ||= Coverage.default_coverage(iseq)
  end  
end

# Usage example:
my_coverage_capture = MyCoverageCapture.start
# Execute test suite or specific tests
my_coverage_capture.stop
# Access detailed coverage data
puts my_coverage_capture.coverage.statement_coverage

In addition, we'd need a well defined interface for Coverage.default_coverage, which includes line, branch and declaration coverage statistics. I suggest we take inspiration from the proposed interface defined by the vscode text editor: https://github.com/microsoft/vscode/blob/b44593a612337289c079425a5b2cc7010216eef4/src/vscode-dts/vscode.proposed.testCoverage.d.ts - this interface was designed to be compatible with a wide range of coverage libraries, so represents the intersection of that functionality.

# Hypothetical interface (mostly copied from vscode's proposed interface):
module Coverage
  # Contains coverage metadata for a file
  class Target
    attr_reader :instruction_sequence
    attr_accessor :statement_coverage, :branch_coverage, :declaration_coverage, :detailed_coverage

    # @param statement_coverage [Hash(Location, StatementCoverage)] A hash table of statement coverage instances keyed on location.
    # Similar structures for other coverage data.
    def initialize(instruction_sequence, statement_coverage, branch_coverage=nil, declaration_coverage=nil)
      @instruction_sequence = instruction_sequence
      @statement_coverage = statement_coverage
      @branch_coverage = branch_coverage
      @declaration_coverage = declaration_coverage
    end
  end

  # Coverage information for a single statement or line.
  class StatementCoverage
    # The number of times this statement was executed, or a boolean indicating
    # whether it was executed if the exact count is unknown. If zero or false,
    # the statement will be marked as un-covered.
    attr_accessor :executed
  
    # Statement location (line number? or range? or position? AST?)
    attr_accessor :location
  
    # Coverage from branches of this line or statement. If it's not a
    # conditional, this will be empty.
    attr_accessor :branches
  
    # Initializes a new instance of the StatementCoverage class.
    #
    # @parameter executed [Number, Boolean] The number of times this statement was executed, or a
    # boolean indicating whether it was executed if the exact count is unknown. If zero or false,
    # the statement will be marked as un-covered.
    #
    # @parameter location [Position, Range] The statement position.
    #
    # @parameter branches [Array(BranchCoverage)] Coverage from branches of this line.
    # If it's not a conditional, this should be omitted.
    def initialize(executed, location, branches=[])
      @executed = executed
      @location = location
      @branches = branches
    end
  end

  # Coverage information for a branch
  class BranchCoverage
    # The number of times this branch was executed, or a boolean indicating
    # whether it was executed if the exact count is unknown. If zero or false,
    # the branch will be marked as un-covered.
    attr_accessor :executed
  
    # Branch location.
    attr_accessor :location
  
    # Label for the branch, used in the context of "the ${label} branch was
    # not taken," for example.
    attr_accessor :label
  
    # Initializes a new instance of the BranchCoverage class.
    #
    # @param executed [Number, Boolean] The number of times this branch was executed, or a
    # boolean indicating whether it was executed if the exact count is unknown. If zero or false,
    # the branch will be marked as un-covered.
    #
    # @param location [Position, Range] (optional) The branch position.
    #
    # @param label [String] (optional) Label for the branch, used in the context of
    # "the ${label} branch was not taken," for example.
    def initialize(executed, location=nil, label=nil)
      @executed = executed
      @location = location
      @label = label
    end
  end

  # Coverage information for a declaration
  class DeclarationCoverage
    # Name of the declaration. Depending on the reporter and language, this
    # may be types such as functions, methods, or namespaces.
    attr_accessor :name
  
    # The number of times this declaration was executed, or a boolean
    # indicating whether it was executed if the exact count is unknown. If
    # zero or false, the declaration will be marked as un-covered.
    attr_accessor :executed
  
    # Declaration location.
    attr_accessor :location
  
    # Initializes a new instance of the DeclarationCoverage class.
    #
    # @param name [String] Name of the declaration.
    #
    # @param executed [Number, Boolean] The number of times this declaration was executed, or a
    # boolean indicating whether it was executed if the exact count is unknown. If zero or false,
    # the declaration will be marked as un-covered.
    #
    # @param location [Position, Range] The declaration position.
    def initialize(name, executed, location)
      @name = name
      @executed = executed
      @location = location
    end
  end
end

By following this format, we will be compatible with a wide range of external tools.

[ioquatix (Samuel Williams)]

It also occurred to me, that Coverage.default_coverage(iseq) might need to mutate the iseq. In the sense that we need to "turn on coverage trace points/callbacks for the given iseq". In that case, we might want to rename such a method, e.g. Coverage.enable_coverage_for(iseq) or something like that.

[ioquatix (Samuel Williams)]

• Description updated (diff)

[mame (Yusuke Endoh)]

Coverage.resume and Coverage.suspend were introduced for per-test coverage. You can use them as follows.

# target.rb
def test1
  p 1
end

def test2
  p 2
end

require "coverage"

Coverage.setup

load "target.rb"

Coverage.resume
test1
Coverage.suspend
pp Coverage.result(stop: false, clear: true)["test.rb"] #=> [0, 1, nil, nil, 0, 0, nil]

Coverage.resume
test2
Coverage.suspend
pp Coverage.result(stop: false, clear: true)["test.rb"] #=> [0, 0, nil, nil, 0, 1, nil]

BTW, in https://bugs.ruby-lang.org/issues/19857#change-106238, I think I clearly said that I do no longer intend to make Coverage library complicated for the time being. I was honestly even disappointed for you to create this proposal. Why don't you understand? Is this because of my bad English? Frankly, I am very tired of communicating with you.

[ioquatix (Samuel Williams)]

Coverage.suspend and Coverage.resume does not work for multi-thread test runner unfortunately, unless there is something I'm missing. In my integration with vscode and sus, this can be a problem, as the test host can run multiple tests at the same time in order to handle the requests from vscode.

Why don't you understand? Is this because of my bad English? Frankly, I am very tired of communicating with you.

I'm sorry to hear that. When we discussed it last time, you suggested introducing some kind of "coverage2" library:

I don't have time right now, but I would like to rebuild the library as a pure external gem in future, i.e. coverage2 or something, with dynamically generated code, etc. in mind from the beginning.

This proposal does not change the existing interface, but introduce new interface which conceivably could be in a gem. However, such a coverage2 cannot exist without some support from Ruby, e.g. instrumentation of instruction sequence, and so on. That is outlined in the proposal, e.g. Coverage.enable_coverage(iseq) etc. It might be possible to use RubyVM internals to achieve this, but it won't be compatible with TruffleRuby or JRuby etc. So, I think it's good to have a central proposal, no?

[mame (Yusuke Endoh)]

Coverage.suspend and Coverage.resume does not work for multi-thread test runner unfortunately

Yes, I mentioned the issue when I propose Coverage.resume/suspend in https://bugs.ruby-lang.org/issues/18176#note-1. Per-thread coverage is a far more difficult problem than per-test one. This would significantly conflict with the original API design and the current implementation. The format of Coverage.result will change again, and the internal data structure will need to be also changed.

The coverage library has been repeatedly extended so that both the API and the implementation have become rather complicated. This was something I regretted for a long time, but "coverage for eval" has led me to believe that we should not repeat it any further. I hope you understand.

That is outlined in the proposal, e.g. Coverage.enable_coverage(iseq) etc. It might be possible to use RubyVM internals to achieve this, but it won't be compatible with TruffleRuby or JRuby etc.

Dealing with iseq directly is not compatible with other Ruby implementation, no? And a low-level method to instrument an existing iseq is clearly beyond the scope of responsibility of Coverage library. It clearly looks like a messed up proposal to me. I hope you understand that there is no way such a proposal is accepted.

[anmarchenko (Andrey Marchenko)]

ioquatix (Samuel Williams) wrote:

As Ruby applications grow in complexity, the need for more sophisticated testing and coverage analysis tools becomes paramount. Current coverage tools in Ruby offer a good starting point but fall short in delivering the granularity and flexibility required by modern development practices. Specifically, there is a significant gap in "per-test coverage" reporting, which limits developers' ability to pinpoint exactly which tests exercise which lines of code. This proposal seeks to initiate a discussion around improving Ruby's coverage module to address this gap.

Hi! I think this would be a great addition to Ruby, it would help me to solve an important business case at Datadog.

I work as a library developer at Datadog and we are working on developer tools for test visibility and smart test execution in Ruby. One product that we are working on right now for Ruby is an intelligent test runner https://docs.datadoghq.com/intelligent_test_runner that saves time for devs by running only relevant tests for a feature branch. We use test impact analysis technique for that https://martinfowler.com/articles/rise-test-impact-analysis.html that requires us to collect per test code coverage for users of our library when they run their test suite using minitest/rspec/cucumber/etc.

We need to create a solution that:

• tracks code coverage per test while being invisible for users, do not interferes with existing simplecov/covered setup
• has low performance overhead
• works with parallel and sequential test runners

I am evaluating several possible solutions for this case, one being using Coverage.suspend/resume with Coverage.result(stop: false, clear: true). The most important problem with this solution for me is not only that it does not support threaded parallel executors, but it also skews existing code coverage that our users might have. All major code coverage libraries in Ruby (simplecov/covered/single_cov) use Coverage under the hood and Coverage has only one global state. So if we would use suspend/resume and clear results after each test, this would break all existing code coverage libraries. I would be delighted to see some API to spy on current code coverage collection and receive what was covered between specific points of time (start/stop of each test) and I think it would benefit many developer tools out there.

[mame (Yusuke Endoh)]

@anmarchenko (Andrey Marchenko) I think you want to look into a C API called rb_thread_add_event_hook. You can use it to register your C hook function to be called for each specified event, e.g., line execution (RUBY_EVENT_LINE), method call (RUBY_EVENT_CALL), etc. You can register a hook per thread. It also allows multiple hooks registered simultaneously. It is orthogonal with Coverage, i.e., does not interfere with an existing Coverage setup. You need to write C, but you can create something specific to your use case, which could be more efficient than general-purpose Coverage library.