Feature #20282
Updated by ioquatix (Samuel Williams) 9 months ago
As Ruby applications grow in complexity, the need for more sophisticated testing and coverage analysis tools becomes paramount. Current coverage tools in Ruby offer a good starting point but fall short in delivering the granularity and flexibility required by modern development practices. Specifically, there is a significant gap in "per-test coverage" reporting, which limits developers' ability to pinpoint exactly which tests exercise which lines of code. This proposal seeks to initiate a discussion around improving Ruby's coverage module to address this gap.
## Objectives
The primary goal of this initiative is to introduce support for per-test coverage reports within Ruby, focusing on three key areas:
1. Scoped Coverage Data Capture: Implementing the capability to capture coverage data within user-defined scopes, such as global, thread, or fiber scopes. This would allow for more granular control over the coverage analysis process.
2. Efficient Data Capture Controls: Developing mechanisms to efficiently control the capture of coverage data. This includes the ability to exclude specific files, include/ignore/merge eval'd code, to ensure that the coverage data is both relevant and manageable.
3. Compatibility and Consistency: Ensuring that the coverage data is exposed in a manner that is consistent with existing coverage tools and standards. This compatibility is crucial for integrating with a wide array of tooling and for facilitating a seamless developer experience.
## Proposed Solutions
The heart of this proposal lies in the introduction of a new subclassable component within the Coverage module, tentatively named `Coverage::Capture`. This component would allow users to define custom coverage capture behaviors tailored to their specific needs. Below is a hypothetical interface for such a mechanism:
```ruby
class Coverage::Capture
def self.start
self.new.tap(&:start)
end
# Start receiving coverage callbacks.
def start
end
# Stop receiving coverage callbacks.
def stop
end
# User-overridable statement coverage callback.
def statement(iseq, location)
fetch(iseq)&.statement_coverage.increment(location)
end
# Additional methods for branch/declaration coverage would follow a similar pattern.
end
class MyCoverageCapture < Coverage::Capture
# Provides efficient data capture controls - can return nil if skipping coverage for this iseq, or can store coverage data per-thread, per-fiber, etc.
def fetch(iseq)
@coverage[iseq] ||= Coverage.default_coverage(iseq)
end
end
# Usage example:
my_coverage_capture = MyCoverageCapture.start
# Execute test suite or specific tests
my_coverage_capture.stop
# Access detailed coverage data
puts my_coverage_capture.coverage.statement_coverage
```
In addition, we'd need a well defined interface for `Coverage.default_coverage`, which includes line, branch and declaration coverage statistics. I suggest we take inspiration from the proposed interface defined by the vscode text editor: https://github.com/microsoft/vscode/blob/b44593a612337289c079425a5b2cc7010216eef4/src/vscode-dts/vscode.proposed.testCoverage.d.ts - this interface was designed to be compatible with a wide range of coverage libraries, so represents the intersection of that functionality.
```ruby
# Hypothetical interface (mostly copied from vscode's proposed interface):
module Coverage
# Contains coverage metadata for a file
class Target
attr_reader :instruction_sequence
attr_accessor :statement_coverage, :branch_coverage, :declaration_coverage, :detailed_coverage
# @param statement_coverage [Hash(Location, StatementCoverage)] A hash table of statement coverage instances keyed on location.
# Similar structures for other coverage data.
def initialize(instruction_sequence, statement_coverage, branch_coverage=nil, declaration_coverage=nil)
@instruction_sequence = instruction_sequence
@statement_coverage = statement_coverage
@branch_coverage = branch_coverage
@declaration_coverage = declaration_coverage
end
end
# Coverage information for a single statement or line.
class StatementCoverage
# The number of times this statement was executed, or a boolean indicating
# whether it was executed if the exact count is unknown. If zero or false,
# the statement will be marked as un-covered.
attr_accessor :executed
# Statement location (line number? or range? or position? AST?)
attr_accessor :location
# Coverage from branches of this line or statement. If it's not a
# conditional, this will be empty.
attr_accessor :branches
# Initializes a new instance of the StatementCoverage class.
#
# @parameter executed [Number, Boolean] The number of times this statement was executed, or a
# boolean indicating whether it was executed if the exact count is unknown. If zero or false,
# the statement will be marked as un-covered.
#
# @parameter location [Position, Range] The statement position.
#
# @parameter branches [Array(BranchCoverage)] Coverage from branches of this line.
# If it's not a conditional, this should be omitted.
def initialize(executed, location, branches=[])
@executed = executed
@location = location
@branches = branches
end
end
# Coverage information for a branch
class BranchCoverage
# The number of times this branch was executed, or a boolean indicating
# whether it was executed if the exact count is unknown. If zero or false,
# the branch will be marked as un-covered.
attr_accessor :executed
# Branch location.
attr_accessor :location
# Label for the branch, used in the context of "the ${label} branch was
# not taken," for example.
attr_accessor :label
# Initializes a new instance of the BranchCoverage class.
#
# @param executed [Number, Boolean] The number of times this branch was executed, or a
# boolean indicating whether it was executed if the exact count is unknown. If zero or false,
# the branch will be marked as un-covered.
#
# @param location [Position, Range] (optional) The branch position.
#
# @param label [String] (optional) Label for the branch, used in the context of
# "the ${label} branch was not taken," for example.
def initialize(executed, location=nil, label=nil)
@executed = executed
@location = location
@label = label
end
end
# Coverage information for a declaration
class DeclarationCoverage
# Name of the declaration. Depending on the reporter and language, this
# may be types such as functions, methods, or namespaces.
attr_accessor :name
# The number of times this declaration was executed, or a boolean
# indicating whether it was executed if the exact count is unknown. If
# zero or false, the declaration will be marked as un-covered.
attr_accessor :executed
# Declaration location.
attr_accessor :location
# Initializes a new instance of the DeclarationCoverage class.
#
# @param name [String] Name of the declaration.
#
# @param executed [Number, Boolean] The number of times this declaration was executed, or a
# boolean indicating whether it was executed if the exact count is unknown. If zero or false,
# the declaration will be marked as un-covered.
#
# @param location [Position, Range] The declaration position.
def initialize(name, executed, location)
@name = name
@executed = executed
@location = location
end
end
end
```
By following this format, we will be compatible with a wide range of external tools.