Project

General

Profile

Actions

Feature #16012

open

Add a (small) test-install suite?

Added by MSP-Greg (Greg L) over 4 years ago. Updated about 1 year ago.

Status:
Assigned
Target version:
-
[ruby-core:93845]

Description

At various times there has been discussion about whether testing should require make install. Although I prefer to do testing using install (vs build), I see it as a choice, and not a requirement.

From time to time various issues have arisen that cannot be found with 'build' testing. Often, these issues cause CI test failure with master/trunk/ruby-head in external repos. Sometimes people blame 'Core', other times Travis, or rvm. Regardless, it doesn't look good.

So, might a small set of tests that check install functionality be added? It may need to be two separate (but equivalent) scripts. One for *nix, one for Windows.

In ruby-loco, I'm using a ps1 script to check that CLI bin files work. As soon as the update is pushed here, I'll add a test for nested bundler commands...


Related issues 2 (0 open2 closed)

Related to Ruby master - Misc #16013: Bundler...Feedbackhsbt (Hiroshi SHIBATA)Actions
Related to Ruby master - Feature #13534: Checking installation results of default gemsClosedhsbt (Hiroshi SHIBATA)Actions

Updated by nobu (Nobuyoshi Nakada) over 4 years ago

What do you mean by to "check install functionality" concretely?
Something other than make install with DESTDIR?

Updated by MSP-Greg (Greg L) over 4 years ago

@nobu (Nobuyoshi Nakada),

What do you mean by to "check install functionality" concretely?

Anything that cannot be checked in the build folder.

Sometime recently, at least some of the bin stubs broke. So checking that all bin stubs run correctly would be one set of tests.

See https://ci.appveyor.com/project/MSP-Greg/ruby-loco/builds/26107569#L2961 for what I added to ruby-loco for cli. The build/job will fail if they don't pass.

Also, Bundler can be installed as a gem, it's also tested in it's own repo. But, some of the logic used when it is a default gem (as in trunk) was incorrect, and repos that had nested 'bundle' commands had CI failures.

This test suite would be small, as there aren't a lot of things to check, and it would only be usable/informative when used after make install.

One thing in particular it would help with is verifying that 'ruby-head' functions corrrectly when used in external repo/gem's CI testing...

I remember a while ago I was helping a popular gem with Appveyor, and I suggested that they test on 'ruby-head'. The response was essentially 'no, it fails too often, we don't want to bother". Not a common opinion, and I've also seen failures blamed on Travis, rvm, etc.

Personally, I want the Ruby community to think that 'ruby-head' is solid. The more testing it's exposed to, the better...

Updated by MSP-Greg (Greg L) over 4 years ago

An example test (no assert) that's currently failing, run from a trunk install

require 'tmpdir'

Dir.mktmpdir 'test_bundler' do |dir|
  File.write 'Gemfile', '', mode:'wb'
  str = `bundle exec "bundle exec 'ruby -v'"`
  puts str
  puts str.strip == RUBY_DESCRIPTION
end

Updated by hsbt (Hiroshi SHIBATA) over 4 years ago

Recently, I added test-tool target for the common library under the tool directory. We can test rbinstall.rb with it with the installation specification.

I know your concern. But your ruby-head only points Travis environment especially RVM. The some CI uses docker image provided the docker inc or the original images. The traditional CI like Travis or Azure Pipelines uses their internal environment or original distribution etc. We couldn't support all of the environments.

But I would like to add the system test for the ruby version managers with Docker like https://github.com/oracle/truffleruby/blob/master/doc/contributor/docker.md.

The system test was:

  • install ruby master used by ruby managers or make install.
  • assert installation files with the expected structures.
  • assert binstub behavior with ruby managers or not.

The additional works, We can test them with --program-suffix, --prefix and DESTDIR etc.

Actions #5

Updated by hsbt (Hiroshi SHIBATA) over 4 years ago

Updated by MSP-Greg (Greg L) over 4 years ago

I understand the issue with several packagers/builders, etc.

But, getting things checked in the build here is a start, and we would know that the internal make install worked correctly. We don't know that now.

The tool could also be used as a check to be used by the other packagers/builders...

Should it be part of the make system, or a bash/shell/ps1 script, or a ruby script?

Lastly, you make a good point about '--program-suffix, --prefix and DESTDIR etc.' I'm suggesting testing things that are difficult to test without the 'struture' of a true install.

You're also suggesting adding checking for the install itself, which I've never had issue with. But I've never used '--program-suffix' or '--prefix'. Nor have I seen the folder structure or contents fail. But they could...

Updated by hsbt (Hiroshi SHIBATA) over 4 years ago

  • Status changed from Open to Assigned
  • Assignee set to hsbt (Hiroshi SHIBATA)
Actions #8

Updated by hsbt (Hiroshi SHIBATA) about 1 year ago

  • Related to Feature #13534: Checking installation results of default gems added

Updated by hsbt (Hiroshi SHIBATA) about 1 year ago

I'm considering this request again. I'll add small check into https://github.com/ruby/chkbuild.

Updated by Eregon (Benoit Daloze) about 1 year ago

FYI there is this script to check bin/ files, which could be reused: https://github.com/ruby/ruby-dev-builder/blob/master/cli_test.rb

Updated by Eregon (Benoit Daloze) about 1 year ago

I think this could become part of ruby/spec, and then we'd run ruby/spec on both built and on installed CRuby with chkbuild.

BTW there are similar specs in https://github.com/oracle/truffleruby/blob/master/spec/truffle/launcher_spec.rb

Actions

Also available in: Atom PDF

Like0
Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0Like0