Bug #12597: Produce test failure if some tests cannot be executed due to lacking data

• Status: Closed
• Assignee: nobu (Nobuyoshi Nakada)
• Target version: -
• ruby -v: ruby 2.4.0dev (2016-07-19 trunk 55704) [x86_64-cygwin]
• Backport: 2.1: UNKNOWN, 2.2: UNKNOWN, 2.3: UNKNOWN
• Tags: encoding, test

[duerst (Martin Dürst)]

In some cases, some files (such as Unicode data files) may be missing and cannot be downloaded.

In these cases, tests should not fail silently. It is not good to produce a long list of test failures when there is only a single problem. Therefore, e.g. test/ruby/enc/test_case_comprehensive.rb up to r55520 produced a single explicit failure when tests were missing.

In revision r55521, this test was removed, but it should be reinstated to avoid silent failure.

[jeremyevans0 (Jeremy Evans)]

• Status changed from Open to Closed

Currently, there is a skip instead of a failure if the unicode data files are not available. I tried changing this from a skip to a failure, but it resulted in the CI jobs that run tests failing: https://github.com/jeremyevans/ruby/runs/2859985432#step:15:120

Since CI does not run these tests due to the absence of the unicode data, I don't think it is reasonable that the tests fail if the unicode data is not available. So I'm going to close this. @duerst (Martin Dürst) If you think this is still a bug, please reopen and explain how you think it should work. One possible solution is skipping if we detect that we are running on CI, and failing otherwise, but I'm against that. Another solution is making all CI test jobs download the unicode data files, but I'm guessing that is disabled due to bandwidth issues it would introduce.