Project

General

Profile

Feature #16655

Each test on test-all should run `srand(seed)` at setup

Added by ko1 (Koichi Sasada) 8 months ago. Updated 29 days ago.

Status:
Closed
Priority:
Normal
Target version:
[ruby-core:97272]

Description

Points

  • call srand($seed) before each test. $seed is given seed number by --seed N

Background

Some tests use rand() to produce random number to generate random test examples.

Example:

  def test_local_variable_set_wb
    assert_ruby_status([], <<-'end;', '[Bug #13605]', timeout: 30)
      b = binding
      n = 20_000

      n.times do |i|
        v = rand(2_000)
        name = "n#{v}"
        value = Object.new
        b.local_variable_set name, value
      end
    end;
  end

This test generates 2,000 randomly named ivar which point random numbers.

However, several tests calls srand(0) and this random sequence is affected by test methods order.

Two problems:

  • Test authors of this kind of tests should assume that every test processes uses very different random sequence to increase the test coverage.
  • We can not get reproducible results on such tests which use random sequence with --seed because of srand(0) using tests.

Proposal

To solve the above two problems, I'll insert srand($seed) at the beginning of each test ($seed is given by --seed option).
This proposal respects --seed option more, so we can expect more deterministic results.

I measured performance overhead and I can't get meaningful slow down.

Additional note

Surprisingly, --seed option is ignored just now on test-all, so I'll fix this problem too.

#1

Updated by ko1 (Koichi Sasada) 8 months ago

  • Status changed from Open to Closed

Applied in changeset git|116c6320956ce80efcb1db849a190ebd7f8c8ad3.


don't ignore --seed option

--seed N option is just ignored so respect this option.
[Feature #16655]

Also making "Run options" display pretty.

#2

Updated by ko1 (Koichi Sasada) 8 months ago

  • Description updated (diff)
#3

Updated by hsbt (Hiroshi SHIBATA) 29 days ago

  • Target version changed from 36 to 3.0

Also available in: Atom PDF