WCT Testing with C++
(top) (pkg)

Edit a file named to match the <pkg>/test/doctest*.cxx pattern:

emacs util/test/doctest-my-first-test.cxx

Include the single doctest.h header, and any others your test code may require and provide at least one TEST_CASE("...").

#include "WireCellUtil/doctest.h"
#include "WireCellUtil/Logging.h"
TEST_CASE("my first test") {
    bool ok=true;
    CHECK(1 + 1 == 2);          // does not halt testing
    REQUIRE(ok == true);        // halts testing
    SUBCASE("a test using above as existing context") {
        ok = false;
        CHECK(!ok);
    }
    SUBCASE("another test using copy of the context") {
        // subcase above does not change our 'ok' variable
        CHECK(ok);
    }
}

TEST_CASE("my second test") {
    spdlog::debug("this test is very trivial");
    CHECK(true);
}

Compile and run just one test:

waf --target=wcdoctest-test
./build/test/wcdoctest-test --test-case='my first test'
./build/test/wcdoctest-test   # runs all test cases

The doctest runner has many options

./build/util/wcdoctest-util --help

Developers are encouraged not to use std::cout or std::cerr in doctest tests. Instead, as shown in the above example, we should use logging at debug level (or trace).

#include "WireCellUtil/Logging.h"
TEST_CASE("...") {
    spdlog::debug("some message");
    // ...
}

By default, these messages will not be seen. But they can be turned on:

$ SPDLOG_LEVEL=DEBUG ./build/test/wcdoctest-test

An “atomic” C++ test source file matches:

<pkg>/test/test*.cxx
<pkg>/test/atomic*.cxx

Each atomic source file must provide a main() function and results in a similarly named executable found at:

build/<pkg>/test*
build/<pkg>/atomic*

Some reasons to write atomic tests (compared to doctest tests) include:

• The developer wishes the test to accept optional command line arguments to perform variant tests.
• The test is long-running (more than about 1 second) and so benefits from task-level parallelism provided by waf.

A trivial atomic test is shown:

int main(int argc, char* argv[])
{
    bool ok = true;
    if (!ok) {
        return 1;
    }
    return 0;
}

Compile and run with:

$ waf --tests --target=atomic-simple
$ ./build/test/atomic-simple

Like in section 2, developers should use debug level logging instead of std::cout or std::cerr. For this to work, the code requires some boilerplate:

#include "spdlog/spdlog.h"
#include "spdlog/cfg/env.h"

int main(int argc, char* argv[])
{
    // required for SPDLOG_LEVEL env var
    spdlog::cfg::load_env_levels(); 

    spdlog::debug("all messages should be at debug or trace");
    spdlog::info("avoid use of info() despite this example");

    // Now some "tests"
    bool ok = true;
    if (!ok) return 1;

    return 0;
}

$ ./build/test/atomic-simple-logging
[2023-04-25 11:56:26.348] [info] avoid use of info() despite this example

$ SPDLOG_LEVEL=debug ./build/test/atomic-simple-logging
[2023-04-25 11:59:47.884] [debug] all messages should be at debug or trace
[2023-04-25 11:59:47.884] [info] avoid use of info() despite this example

It is possible make an atomic test use doctest. It will still be processed as an atomic test by WCT build system but it will gain the facilities of doctest. Along with logging, it requires a bit more boilerplate:

#define DOCTEST_CONFIG_IMPLEMENT
#include "WireCellUtil/doctest.h"
#include "spdlog/spdlog.h"
#include "spdlog/cfg/env.h"
int main(int argc, char** argv) {
    spdlog::cfg::load_env_levels();
    doctest::Context context;
    context.applyCommandLine(argc, argv);
    int rc = context.run();
    if (rc) { return rc; }

    bool ok = true;             // start my tests
    if (!ok) return 1;
    return 0;
}

$ waf --tests --target=atomic-doctest
$ ./build/test/atomic-doctest
$ ./build/test/atomic-doctest --help

An atomic test must run with no command line arguments. However, we may allow optional arguments. One example:

aux/test/test_idft.cxx

This tests various aspects of the IDFT interface implementations. It can be run as an atomic test with the default IDFT implementation:

$ ./build/aux/test_idft

It can also be run in a variant form by giving optional command line argumetns:

$ ./build/aux/test_idft FftwDFT WireCellAux
$ ./build/aux/test_idft TorchDFT WireCellPytorch
$ ./build/aux/test_idft cuFftDFT WireCellCuda

The first variant is actually identical to the atomic call. The latter two require that WCT is build with support for PyTorch and CUDA, respectively. An atomic test for each of the latter two variants can be found in their respective packages.

Tests tend to grow. Developers are strongly urged to grow tests in a way that defines separate test cases separately. When a developer writes a doctest test this is easily done by add more TEST_CASE() and/or SUBCASE() instances to the source file. When writing an atomic test, the developer must invent their own “mini unit test framework”. One common pattern is “bag of test_* functions. Functions are distinquished by name and/or templates:

static
void test_2d_threads(IDFT::pointer dft, int nthreads, int nloops, int size = 1024)
{
    // ...
}
template<typename ValueType>
void test_2d_transpose(IDFT::pointer dft, int nrows, int ncols)
{
   // ...
}

int main(int argc, char* argv[])
{
    // ...
    test_2d_transpose<IDFT::scalar_t>(idft, 2, 8);
    // ...
    return 0;
}

A test is successful if it completes with a return status code of zero. A failed test can be indicated in a number of ways:

• return non-zero status code from main().
• throw an exception.
• call assert() or abort().
• call WCT’s Assert() or AssertMsg().
• apply doctest assertion macros.

The test developer is free to use any or a mix of these methods and is strongly urged to use them pervasively throughout the test code.

As introduced above, WCT provides some support for testing. The first are simple wrappers around assert() and one that will print a message if the assertion fails:

#include "WireCellUtil/Testing.h"

int main()
{
    int x = 42;
    Assert(x == 42);
    AssertMsg(x == 0, "Not the right answer");
    return 0;
}

In addition, WCT provides facilities for reporting simple performance statistics, specifically CPU time and memory usage.

#include "WireCellUtil/TimeKeeper.h"
#include "WireCellUtil/MemUsage.h"
#include "WireCellUtil/ExecMon.h"

TimeKeeper

a “stopwatch” to record time along with a message for various steps in a test

MemUsage

similar but to record memory usage

ExecMon

combine the two.

See test_timekeeper.cxx, test_memusage.cxx and test_execmon.cxx, respectively, in util/test/.

Tests may produce files, even atomic tests that may have no files governing waf task dependencies. These files can be useful to persist beyond the test job. The ideal location for these files is the build/ directory and as sibling to the C++ test executable. C++ has a simple pattern to achieve this:

int main(int argc, char* argv[])
{
    std::string name = argv[0];
    std::string outname = name + ".ext";
    std::string outname2 = name + "_other.ext";
    // open and write to outname and outname 2....
    return 0;
}

As the C++ test executable is found build/<pkg>/<prefix><sep><name>, these output files will be found there as siblings.

Likewise, an atomic test must not expect any input files specified by the caller. However, it may load files that can be found from the environment. A common example is to find a WCT “wires” file or others provided by wire-cell-data. Here is a C++ pattern do that in a way that naturally allows an atomic test to also be called in a variant manner.

int main(int argc, char* argv[])
{
    const char* filename = "microboone-celltree-wires-v2.1.json.bz2";
    if (argc > 1) {
        filename = argv[1];
    }
    // use filename...
    return 0;
}

See util/test/test_wireschema.cxx for an example.

For this kind of file to be found the user must define WIRECELL_PATH to include a directory holding the contents of wire-cell-data.

In principle the path in argv[0] may also be used to locate the top of the wire-cell-toolkit source in order to locate files provided by the source and use them as input.