Misc #20661
openStop retrying tests in `make test-all` command by default
Description
Summary¶
Currently, tests are retried once if they fail when tests are executed as parallel tests in the make test-all
command by default. This is to prevent test failures caused by parallel testing. That's why failed tests are executed serially, not in parallel.
Source code: https://github.com/ruby/ruby/blob/master/tool/lib/test/unit.rb#L728-L751
However, there is a problem, which might hide a real issue like https://bugs.ruby-lang.org/issues/20314. To prevent missing this "real" problem, stopping the retry of tests in the make test-all
command by default would be better.
Concerns when stopping the retry of tests in the make test-all
command¶
Here are some concerns about stopping the retry mechanism in make test-all
command.
1. There are some flaky tests in the test suite. Are we okay with that?¶
Yes, all flaky tests are tracked by Launchable, and they're mainly monitored by @mame (Yusuke Endoh). Thus, they'll be fixed in the future.
2. When a test that is not related to my change fails, what should we do?¶
Just click the "Re-run jobs" button via the GitHub UI.