Misc #20739
openTest suite does not carry over CLI options
Description
I'm not sure if anything needs to change here, but I wanted to report this in case anyone had any particular ideas. When running the ruby/ruby test suite, lots of tests call EnvUtil.invoke_ruby
directly or indirectly through assert_in_out_err
. That spawns a subprocess that runs Ruby and reports its status back through status, stdout, and stderr. The issue is, the subprocess does not always use the same options, which means there are lots of different ways it can be run.
For RJIT/YJIT, we have -DRJIT_FORCE_ENABLE
and -DYJIT_FORCE_ENABLE
, which forces the subprocess to have it enabled. Still, this isn't always used, so when running the test suite for RJIT/YJIT, a lot of tests are not actually running with those compilers.
Prism and parse.y have a similar problem. When running in the same process, it will respect the --parser
option. But when running in a subprocess it will fall back to whatever the default parser is. This presents a problem for some tests that assert against RUBY_DESCRIPTION in a subprocess, as it may have +PRISM
in the description.
We have tried to fix it before by adding RUN_OPTS/RUBYOPT to the subprocess options, but this causes lots of other problems. I am not sure of the correct solution, or if we should even bother fixing this. Please let me know what you think.
Updated by Eregon (Benoit Daloze) about 2 months ago ยท Edited
I think it's a real issue and sometimes valuable to fix (it hurts the coverage of tests * flags).
#6648 could be helpful to automatically inherit passed flags.
Alternatively, we could ensure we only pass options for tests through RUBYOPT
, and that RUBYOPT is never cleared/overridden without considering the previous value.
MSpec currently handles this by having an extra subprocess, so all test processes always have all the -T-flag
options consistently.
(#6648 would be nicer)
Updated by Eregon (Benoit Daloze) about 2 months ago
- Related to Feature #6648: Provide a standard API for retrieving all command-line flags passed to Ruby added