Bug #16361: TestEnv#test_fetch failure

• Status: Closed
• Assignee: -
• Target version: -
• ruby -v: ruby 2.7.0dev (2019-11-22 master f9d20a1bf1) [x86_64-linux]
• Backport: 2.5: UNKNOWN, 2.6: UNKNOWN
• Tags: maybenotbug, test

[vo.x (Vit Ondruch)]

I am facing following test suite error:

  1) Failure:
TestEnv#test_fetch [/builddir/build/BUILD/ruby-2.7.0-f9d20a1bf1/test/ruby/test_env.rb:123]:
[ruby-core:56062] [Feature #8649].
Expected Exception(KeyError) was raised, but the message doesn't match.
<"key not found: \"test\""> expected but was
<"key not found: \"test\"\n" + "Did you mean?  \"TESTS\"">.

Trying to execute this test independently, it passes just fine. So it seems that the did_you_mean gem is loaded unexpectedly on this place. I just tried:

mv test/optparse/test_did_you_mean.rb{,.bak}

since this is the latest addition IMO and the test suite passes just fine.

I am not really sure why it should fail on my setup and it probably does not fail in any other CI. Neither I am sure what would be the best option to fix this. The assertion could be updated to accept this message by simple sed:

sed -i "/'key not found: \"test\"'/ s/'/\//g" test/ruby/test_env.rb

[mame (Yusuke Endoh)]

How did you run the test? Currently, bundled gems should be unavailable in make test-all, and some tests don't assume that bundled gems are loaded.

Currently, did_you_mean is a bundled gem, so should not be loaded in make test-all. test/optparse/test_did_you_mean.rb is disabled in effect: require "did_you_mean" rescue return. So this test file should not be loaded in the whole test suite run.

We plan to convert did_you_mean to a default gem. Until the conversion, the test file will be disabled.

[vo.x (Vit Ondruch)]

mame (Yusuke Endoh) wrote:

How did you run the test?

I am using the snapshot to execute the test suite. Therefore I assume that did_you_mean gem is available and the test suite should pass in this configuration.

[mame (Yusuke Endoh)]

I cannot reproduce the issue with ruby 2.7.0-preview3.

$ make test-all TESTS=test/ruby/test_env.rb
fatal: not a git repository (or any of the parent directories): .git
./revision.h unchanged
config.status: creating ruby-runner.h
making mjit_build_dir.so
Run options: "--ruby=./miniruby -I./lib -I. -I.ext/common  ./tool/runruby.rb --extout=.ext  -- --disable-gems" --excludes-dir=./test/excludes --name=!/memory_leak/

# Running tests:

Finished tests in 0.026560s, 1618.9994 tests/s, 40889.1485 assertions/s.
43 tests, 1086 assertions, 0 failures, 0 errors, 0 skips

ruby -v: ruby 2.7.0preview3 (2019-11-22 master 35608760ff) [x86_64-linux]

Could you elaborate on what you did?

[vo.x (Vit Ondruch)]

mame (Yusuke Endoh) wrote:

I cannot reproduce the issue with ruby 2.7.0-preview3.

$ make test-all TESTS=test/ruby/test_env.rb

... snip ...

That is what I more or less said, quoting myself:

Trying to execute this test independently, it passes just fine.

You have to run the whole test suite. Take the snapshot (mine was from 2019-11-22) and run the whole test suite:

$ make test-all

[mame (Yusuke Endoh)]

So are you running just make test-all normally? I misguessed that you reconstructed the tarball for rpm package or something. Sorry.

I have never seen the issue in CIs (I may overlook of course), and we have no such a report against preview3. I've run the following but cannot reproduce the issue.

$ wget https://cache.ruby-lang.org/pub/ruby/snapshot.tar.gz
$ tar xf snapshot.tar.gz
$ cd snapshot
$ ./configure && make -j && make test-all

[vo.x (Vit Ondruch)]

Ah, finally I understand where does the issue comes from. I execute the test suite via:

$ ./configure && make -j && make test-all TESTS="-v "

(actually this is the specific command 1) and apparently the TESTS comes from this place.

[vo.x (Vit Ondruch)]

This is shorter reproducer:

$ make test-all TESTS="test/optparse/test_did_you_mean.rb test/ruby/test_env.rb"
fatal: not a git repository (or any of the parent directories): .git
./revision.h unchanged
Run options: "--ruby=./miniruby -I./lib -I. -I.ext/common  ./tool/runruby.rb --extout=.ext  -- --disable-gems" --excludes-dir=./test/excludes --name=!/memory_leak/

# Running tests:

[11/55] TestEnv#test_fetch = 0.00 s               
  1) Failure:
TestEnv#test_fetch [/builddir/ruby-2.7.0-053f78e139/test/ruby/test_env.rb:123]:
[ruby-core:56062] [Feature #8649].
Expected Exception(KeyError) was raised, but the message doesn't match.
<"key not found: \"test\""> expected but was
<"key not found: \"test\"\n" + "Did you mean?  \"TESTS\"">.

Finished tests in 0.024122s, 2280.0665 tests/s, 32957.3255 assertions/s.  
55 tests, 795 assertions, 1 failures, 0 errors, 0 skips

ruby -v: ruby 2.7.0dev (2019-11-22 master 053f78e139) [x86_64-linux]
make: *** [uncommon.mk:780: yes-test-all] Error 1

[vo.x (Vit Ondruch)]

And I can trigger different errors:

$ make test-all TUST="" TESTS="test/optparse/test_did_you_mean.rb test/ruby/test_env.rb"
fatal: not a git repository (or any of the parent directories): .git
./revision.h unchanged
Run options: "--ruby=./miniruby -I./lib -I. -I.ext/common  ./tool/runruby.rb --extout=.ext  -- --disable-gems" --excludes-dir=./test/excludes --name=!/memory_leak/

# Running tests:

[11/55] TestEnv#test_fetch = 0.00 s               
  1) Failure:
TestEnv#test_fetch [/builddir/ruby-2.7.0-053f78e139/test/ruby/test_env.rb:123]:
[ruby-core:56062] [Feature #8649].
Expected Exception(KeyError) was raised, but the message doesn't match.
<"key not found: \"test\""> expected but was
<"key not found: \"test\"\n" +
"Did you mean?  \"TUST\"\n" +
"               \"TESTS\"">.

Finished tests in 0.025079s, 2193.0325 tests/s, 32177.7674 assertions/s.  
55 tests, 807 assertions, 1 failures, 0 errors, 0 skips

ruby -v: ruby 2.7.0dev (2019-11-22 master 053f78e139) [x86_64-linux]
make: *** [uncommon.mk:780: yes-test-all] Error 1

[vo.x (Vit Ondruch)]

The test case was relaxed by 1 similarly to my initial proposal. If this is desired fix and not just workaround, then this could be closed.

[mame (Yusuke Endoh)]

• Status changed from Open to Closed

Okay, thanks!