Bug #19297: Don't download content from internet to execute Ruby test suite

• Status: Open
• Assignee: -
• Target version: -
• ruby -v: ruby 3.2.0 (2022-12-25 revision a528908271) [x86_64-linux]
• Backport: 2.7: DONTNEED, 3.0: DONTNEED, 3.1: DONTNEED, 3.2: REQUIRED

[vo.x (Vit Ondruch)]

Trying to build Ruby 3.2.0 for Fedora and execute its test suite via make check as we always did 1, the test suite suddenly fails (while it was working with git|c5eefb7f37):

... snip ...

C-API Util function ruby_strtod
- converts a string to a double and returns the remaining string
- returns 0 and the full string if there's no numerical value


Finished in 45.737677 seconds

3827 files, 31635 examples, 177877 expectations, 0 failures, 0 errors, 0 tagged
./miniruby -I/builddir/build/BUILD/ruby-3.2.0/lib -I. -I.ext/common  /builddir/build/BUILD/ruby-3.2.0/tool/runruby.rb --extout=.ext  -- --disable-gems -C "/builddir/build/BUILD/ruby-3.2.0" bin/gem install --no-document \
	--install-dir .bundle --conservative "bundler" "rake" "rspec:~> 3" #"ruby-prof"
ERROR:  Could not find a valid gem 'bundler' (>= 0), here is why:
          Unable to download data from https://rubygems.org/ - SocketError: Failed to open TCP connection to rubygems.org:443 (getaddrinfo: Temporary failure in name resolution) (https://rubygems.org/specs.4.8.gz)
ERROR:  Could not find a valid gem 'rspec' (~> 3), here is why:
          Unable to download data from https://rubygems.org/ - SocketError: Failed to open TCP connection to rubygems.org:443 (getaddrinfo: Temporary failure in name resolution) (https://rubygems.org/specs.4.8.gz)
make: Leaving directory '/builddir/build/BUILD/ruby-3.2.0/redhat-linux-build'
make: *** [uncommon.mk:1464: yes-test-syntax-suggest-prepare] Error 2

This is obviously due to the test suite trying to download rspec from the internet, while Fedora builders does not have internet access (and won't ever have for security reasons). If I am not mistaken, this is caused by git|cae53842735237ccf71a13873fd0d1ae7f165582. Now

1. Can this be fixed?
2. Can the tarball be always self contained?

[vo.x (Vit Ondruch)]

• Backport changed from 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN to 2.7: DONTNEED, 3.0: DONTNEED, 3.1: DONTNEED, 3.2: REQUIRED

[nobu (Nobuyoshi Nakada)]

MSpec claims that “it is syntax-compatible with RSpec 2”, but the test of syntax-suggest doesn’t work with MSpec as-is.
Any idea?

I’m ok to revert that commit.

[matsuda (Akira Matsuda)]

MSpec claims that “it is syntax-compatible with RSpec 2”

...and syntax-suggest's specs are written in RSpec 3 which is thoroughly backward incompatible with RSpec 2 that we loved.

[Eregon (Benoit Daloze)]

MSpec is a specialized framework that is syntax-compatible with RSpec 2 for basic things like describe, it blocks and before, after actions.

From https://github.com/ruby/mspec/#readme
i.e., not for the rest.
One cannot run a RSpec spec suite with MSpec (AFAIK).
Running ruby/spec with RSpec 2 doesn't work either.
So it's just similarities in syntax but one cannot replace the other.

This seems the same situation as Bundler specs which uses RSpec 3 as well.

[hsbt (Hiroshi SHIBATA)]

How test bundler in Fedora builder? It needs to download RSpec from rubygems.org since Ruby 2.5.

[vo.x (Vit Ondruch)]

hsbt (Hiroshi SHIBATA) wrote in #note-5:

How test bundler in Fedora builder?

We don't test Bundler unfortunately (in any automated way). But the Bundler test suite is not executed as part of Ruby make check

It needs to download RSpec from rubygems.org since Ruby 2.5.

If this was just about RSpec, we could cope with that, at least for the independent rubygem-bundler package we have in Fedora. Unfortunately, the Bundler test suite relies on network connectivity in many other ways, if I am not mistaken.

Just FTR, we can execute the Bundler test suite locally or in Copr where we can enable network connectivity (but that is manual process). Long term, we are considering to leverage Fedora CI in some way (probably not via upstream test suite, because that is designed to run in the Git repo clone, not to test the installed Bundler). However the tests enabled during package build are the typically the most convenient way (and also closest to what upstreams do) to know our package meets some standards.

[schneems (Richard Schneeman)]

For a fix on our side. We could:

Switch to test-unit

Update syntax_suggest to use test unit instead of rspec (since it seems that works fine on other libraries).

This would be straight-forward, though tedious.

Rescue bundler and exit

Rescue this bundler require and exit(0) with a warning https://github.com/ruby/syntax_suggest/blob/e0c53bba3455d294692f721b4d6ca9b89cf58f99/spec/spec_helper.rb#L3.

This would be the easiest thing to do, but might introduce the case in the future where tests aren't being run on CI accidentally and we don't notice the warning in the output.

Vendor Rspec in Ruby core

This would likely be harder than re-writing the test suite to use test-unit. Also Rspec is much larger than test-unit, it is split into many gems. This would allow future default gems to use rspec but the cost seems

Anything else

Any other ideas? If not, I think the best path forward is to switch syntax suggest over to test-unit.

[Eregon (Benoit Daloze)]

How about adding make test-offline and that only runs test suites which don't need any internet connection?
I think that's a better fix, those who want to test offline must accept they only test a subset, I'm confident Bundler will not rewrite their test suite just to please people testing in offline-only scenarios.

[hsbt (Hiroshi SHIBATA)]

We can add the option of skipping test-syntax-suggest to make check.

And Fedora can use make test test-tool test-all test-spec instead of make check for past behavior.

[deivid (David Rodríguez)]

For what it's worth, Bundler's main test suite should pass now without an internet connection since https://github.com/rubygems/rubygems/pull/6243.

[rubyFeedback (robert heiler)]

I would like to add to Vit Ondruch's suggestion, from another point
of view, so +1 from me here.

I currently have some issues with openssl/gnutls, on my linux machine;
I am in the process of trying to resolve it, but it requires of me
to slowly read up on how this all works and what not. Even the current
LFS/BLFS (Linux from scratch) instructions don't work for me; they
did work for me in the past, on a slackware base system (my current
base system is manjaro, so I am not as familar with it as I was with
slackware). I can not use ruby + gem to upload gems right now. But I
do not know on my other machine with slackware this all works fine -
anyway, something is wrong with my setup.

To my surprise, when I tried to compile the latest ruby, it tried to
download something near the end of the process - I think it was some
gem, or something. This then failed because I could not use ruby for
any https-connection. The result was that I could not get a working
ruby binary.

I am not sure how I resolved it (ruby as-is works), but to me what
was surprising was that there was any (!) step that included
downloading ANYTHING from the world wide web, and the second surprise
was then that this step could fail. I never had that with older ruby
variants, so something seems to have changed in the setup upstream,
and I assume that the core team may not be aware of such "niche
problems" (like no internet connection, or only a faulty one).

So, from this point of view, I highly recommend and encourage what
Vit suggested - decreasing failure points makes sense, and not
having to depend on any external downloads should be the default,
in my opinion. (People should then be able to fix their problems
at a later time too, for instance, if it is in ext/ openssl, then
they could re-compile only these parts. Mandating a working
internet connection in order for ruby to work, appears awkward
to me - ruby should ideally work even without a working internet
connection. Perhaps we could one day define or re-define ruby
in a super-modular way starting via mruby, and then build "on
top" of that as a platform with addons/gems, as-is, customized
to any system, be it small, or large.)

There are also secondary aspects to consider, such as reproducible
builds, so I really want to +1 this suggestion in general.
(Evidently my use case is different from Vit's, but it is still
somewhat related to "require a working internet connection for
ruby to work". It is not always up to individuals to fix NOT
having a working internet connection).

PS: I do not recall the problem precisely, but if someone could
perhaps test compiling ruby from source, without a working
internet connection, I think this may be a way to reproduce
the problem (and perhaps without a base ruby either, or with
a base ruby; perhaps that leads to different behaviour. But I
am 100% convinced that there was a problem in regards to not
having a working internet connection, and I recall that in older
ruby releases this was not the case.)

[vo.x (Vit Ondruch)]

This is still issue, testing with ruby 3.3.0dev (2023-09-05 master 7c8932365f) [x86_64-linux]