WCT BATS Tests
(top) (pkg)

WCT provides a copy of BATS as the version coming with many operating systems is not up to date. It will be used by the build system. When running BATS tests directly the user should assure:

$ PATH=/path/to/my/wire-cell-toolkit/test/bats/bin:$PATH
$ export BATS_LIB_PATH=/path/to/my/wire-cell-toolkit/test

bats_load_library wct-bats.sh

@test "check wire-cell version" {
    wct --version

    # Will fail one day when WCT v1 is released!
    [[ -n $(echo $output | grep '^0' ) ]] 
}

@test "check wcsonnet help" {
    local wcs=$(wcb_env_value WCSONNET)
    run $wcs --help
    echo $output    # shown only on error
    [[ $status -eq 0 ]]
}

The command wct is actually a function from wct-bats.sh that runs wire-cell and instruments some default checking. The wcsonnet is not so wrapped but is available as a wcb variable. The run command is a BATS helper to fill $output and $status, which we check.

A BATS test should run from anywhere:

bats test/test/test_my_first_bats.bats

As in this example, all @test functions in a BATS file will be executed sequentially. A subset of tests with names that match a filter can be selected for running. Using above example:

bats -f "wire-cell" test/test/test_my_first_bats.bats

The* Creating a BATS test

Here we give some detailed guidance on making a BATS test.

See section on Framework and Writing tests for general guidance on writing WCT tests. Start by creating a BATS file. Here, we pick the “test” aka “atomic” category:

<pkg>/test/<category><sep><name>.bats

As with all tests, the category and name should be unique. If more than one test shares the same file name, though perhaps different packages or file name extensions, it is the duty of the developers to assure these tests are mutually compatible in their execution and any file production.

If the test relates to an issue on GitHub (and ideally there is a test for every issue) then this is an excellent pattern:

<pkg>/test/test_issueNNNN.bats

BATS tests should be placed in a <pkg> which best provides the run-time dependencies. For example, if PyTorch is required they should be placed in the <pytorch> sub package.

Edit the .bats file, add the @test function:

#!/usr/bin/env bats

load ../../test/wct-bats.sh

# bats test_tags=tag1,tag2
@test "assure the frob correctly kerplunks" {
    echo "nothing yet"
}

Pick a test description that describes a positive test outcome. Check that the test works:

$ bats <pkg>/test/test_issueNNNN.bats

So far, it can’t fail. Below we progressively add more tests constraints. It is recomended to build up tests in this way as you hunt a bug or develop a new feature. That is, don’t fix the bug or make the feature first. Rather, write the tests and fix the bug / make the feature so that the initially failing tests then succeed.

Typically an @test will consist of one or more stanzas with the following four lines:

run some_command             # (1)
echo "$output"               # (2)
[[ "$status" -eq 0 ]]        # (3)
                             # (4)
[[ -n "$(echo "$output" | grep 'required thing') ]]
[[ -z "$(echo "$output" | grep 'verboten thing') ]]

We explain each:

1. Use Bats run to run some command under test.
2. The run will stuff command output to $output which we echo. We will only see this output on the terminal if the overall test fails. (see logging below).
3. Assert that the command exited with a success status code (0).
4. Perform some checks on the stdout in $output and/or on any files produced by some_command.

In addition to the special @test "" {} function forms, BATS supports two functions that are called once per file. The first is called prior to any @test and the second called after all @test.

function setup_file () {
  # startup code
}
function teardown_file () {
  # shutdown code
}

One example for using setup_file is to run any long-running programs that produce output required by more than one @test.

BATS has a concept of a context-dependent temporary working directory. The contexts are:

test

a single @test function.

file

a .bats test file, such as in setup_file() or teardown() functions.

run

an invocation of the bats command.

Typically, run is not used. The wct-bats.sh library provides some helpers to work with temporary areas:

cd_tmp      (1)
cd_tmp file (2)

Where:

1. The shell will change to the temporary directory for the current context. In setup_file() this is the file context.
2. Explicitly change to the file context. This is typical to use in a @test function to utilize files produced in this scope.

By default bats will delete all temporary directories after completion of the test run. When tests fail it can be useful to examine what was placed in the temporary directories. To allow this run the test like:

$ bats --no-tempdir-cleanup path/to/test_foo.bats

The temporary directory will be printed to the terminal.

Alternatively, wct-bats.sh overrules default temporary directories, combines them and does not delete them when WCTEST_TMPDIR is defined. This can be useful while developing and debugging tests, particularly in combination with writing long running tests in an idempotent fashion. Do not define this variable in any test but instead in your interactive shell session:

$ WCTEST_TMPDIR=$HOME/my-wct-tmp-dir bats [...]

Some BATS tests may use or create files that persist beyond the temporary context via the WCT test data repository (see Data repository). The wct-bats.org library provides some functions to help work with such files.

For a test that produces historical files, they may be saved to the “history” category of the repo with:

# bats test_tags=history
@test "make history" {
  # ...
  saveout -c history my-file-for-history.npz
}

A known input file may be resolved as:

local myinput=$(input_file relative/path/data.ext)

A file from a version of a category is resolved:

# from current version of history category
local myfile=$(category_path relative/path/data.ext)
# from specific version of plots category
local plot20=$(category_path -c plots -v 0.20.0 relative/path/data.png)

All released versions of a the history category directory and all versions of the plots category:

local myhistpaths_released=( category_version_paths )
local myhistpaths_plus_dirty=( category_version_paths -c plots --dirty )

Likewise, but just the version strings

local myhistvers_released=( category_versions )

The wct-bats.sh BATS library provides a helper function to run a particular test command in an idempotent manner. The function is called like:

run_idempotently [sources] [targets] -- <command line>

Where one or more sources are specified with -s|--source <filename> and one or more targets with -t|--target <filename> options. The <command line> will only be executed if:

• No source or no target given.
• Any target files are missing.
• Any target file is older than any source file.

When any of these conditions are not met, the run_idempotently will simply announce (yell) that it is not running the command line and immediately return.

Otherwise, the command line is run and the $status code is checked before returning.

Thus, when the developer runs and re-runs the BATS test with WCTEST_TMPDIR set to a fixed directory the <command line> will only be re-run when needed.

While this will not speed up normal testing, it can dramatically speed up re-running the test by a developer. This can help during development of the test itself, developing code that is being tested and investigating test failures. This development pattern is also helped with bats -f <filter> and use of setup_file as described next.

Another method to run tests in an idempotent manner is to place common, perhaps long running, tasks in the setup_file function, run the entire test with WCTEST_TMPDIR set and then re-run specific tests with bats -f <filter>. When a specific test is exercising some issue, this lets the developer focus on just that issue and reuse prior results. Consider the example:

function setup_file () {
  cd_tmp file
  run my_slow_command -o output1.txt 
  [[ "$status" -eq 0 ]]
}

@test "Some test for number one" {
  cd_tmp file
  run test_some_test1 output1.txt
}

@test "Some test for number two" {
  cd_tmp file
  run test_some_test2 output1.txt
}

Then the developer may do something like:

$ WCTEST_TEMPDIR=/tmp/my-test bats my-test.bats
$ WCTEST_TEMPDIR=/tmp/my-test bats -f one my-test.bats  

To force a full re-run simply remove the /tmp/my-test and perhaps run after unseting WCTEST_TMPDIR.

As shown in the 5 one can assert test tags above a @test. One can also have file-level tags.

# bats file_tags=issue:202

# bats test_tags=topic:noise
@test "test noise spectra for issue 202" {
  ...
}
# bats test_tags=history
@test "make history" {
  ...
  saveout -c history somefile.npz
}

Tag name conventions are defined here:

implicit

The test only performs implicit tests (“it ran and didn’t crash”) and side effects (report, history).

report

The test produces a “report” of files saved to output.

history

The test produces results relevant to multiple released versions (see Historical tests). Only place this tag on tests that produce history files

issue:<number>

The test is relevant to GitHub issue of the given number.

pkg:<name>

The test is part of package named <name> (gen, util, etc)

topic:<name>

The test relates to topic named <name> (wires, response, etc)

time:N

The test requires on order \(10^N\) seconds to run, limited to \(N \in [1, 2, 3]\).

By default, all tests are run. The user may explicitly include or exclude tests. For example, to run tests tagged as being related to wires and that take a few minutes or less to run and explicitly those in the util/ sub package:

bats --filter-tags 'topic:wires,!time:3' util/test/test*.bats

See also the wcb --test-duration=<seconds> options described in section Framework.

BATS uses the “test anything protocol” to combine multiple tests in a coherent way. We need not be overly concerned with the details but it does mean that BATS captures stdout and stderr from the individual tests. When the user wishes to see diagnostic messages directly this causes annoyance. But, no worry as there are three mechanisms to emit and view such user diagonstics.