Testing

A test is a block you write in the file it is about, and here's the catch: every invocation of echoc except echoc test drops it before it is parsed.

test adds_up
{
    $a = 22;
    $b = 20;
    assert($a + $b == 42);
}

echoc test

No framework to install, no separate test directory, no main to wire up. And nothing to strip out later: not compiled and discarded, never seen. A test costs your release build exactly nothing, and it can't end up in a binary by accident.

A test is a function

test <name> { ... } is a function of no arguments that returns nothing. Structurally that's all it is, so it gets the same type checking, the same ownership rules, the same destructor calls and the same diagnostics as any other body, because it is one:

struct Point
{
    int32 $x;
    int32 $y;
}

test a_point_holds_what_it_was_given
{
    Point $p = Point(3, 4);

    assert($p->x == 3);
    assert($p->y == 4);
}

It sits at file scope, beside your functions and types. You can't write one inside a function, and you can't mark it public or private: nothing in the language can name a test, so there is nobody for a visibility modifier to describe.

The body is its own scope in both directions. What you declare inside stays inside:

test uses_a_fixture
{
    struct Fixture
    {
        int32 $seed;
    }

    Fixture $f = Fixture(7);
    assert($f->seed == 7);
}

And nothing outside leaks in, in the one way that might catch you out. A variable at file scope belongs to the program's own frame, so a test can't read it, exactly as no other function can:

$outer = 1;

function reads_it() : int32
{
    // error: a function body cannot reach it
    return $outer;
}

Put shared setup in a function and call it from each test. That's the pattern, and there is no setUp hook coming.

assert

There is no expect, no should, no matcher library, and no way to catch a failure.

test overflows
{
    $a = 1;
    assert($a == 2, 'one is not two');
}

FAIL 1/1  main/overflows.eco::overflows  (exited 1)
1 test, 1 failed

x main/overflows.eco::overflows  exited 1
    assertion failed: one is not two
      at overflows.eco:5

That's not minimalism for its own sake. Echo has no exceptions and nothing in it can unwind, so a failed assert ends the process it is in. So does die, and so does a segfault in whatever you are testing. A process boundary is the only thing that can survive any of those. See Errors and panics for the wider picture.

Which is exactly what you get: each test runs in a process of its own. echoc test starts a fresh process for every one, so a test that aborts, exits or crashes takes nothing with it and the run carries on:

test fails_first
{
    assert(1 == 2, 'nope');
}

test still_runs
{
    assert(true);
}

FAIL 1/2  main/two.eco::fails_first  (exited 1)
ok   2/2  main/two.eco::still_runs
2 tests, 1 failed

Once you have that, you don't need anything else. A matcher library would be a nicer sentence in the failure output, and that's all it would be.

Sometimes the point is that it dies. A die is a failed test unless you say otherwise:

#[tests: expects death]
test an_unloaded_call_dies
{
    die('not loaded');
}

ok   1/1  main/an_unloaded_call_dies.eco::an_unloaded_call_dies
1 test passed

The child has to exit non-zero. A hang is still a timeout, not a death. A failed assert counts, because it is the same __eco_abort a die is. Matching the message is not built yet.

Note: assert is a debug-build thing. echoc test defaults to --debug for exactly this reason, and echoc test --release runs your suite with every assertion elided, which is a run that cannot fail. Try not to do that by accident.

Tags

The file and the module come for free. The group is yours:

#[group: "arithmetic"]
test adds_up
{
    assert(1 + 1 == 2);
}

#[group: "arithmetic"]
test subtracts
{
    assert(2 - 1 == 1);
}

test ungrouped
{
    assert(true);
}

The compiler reads nothing out of a group. It exists so that you can run some of your tests instead of all of them, which is the next section.

Filters pick which tests run

--filter, repeatable:

echoc test --filter adds_up              # by name
echoc test --filter group:arithmetic     # by group
echoc test --filter file:src/math.eco    # by file
echoc test --filter module:mylib         # by module

A bare word is a test's name, because that's the case you want shortest. Everything else is tagged, and a tag you misspell is refused rather than quietly read as a name:

error: 'grup' is not a test filter, expected one of: name, group, file, module.

Written more than once, filters add up. Two group: words run both groups rather than intersecting to nothing.

--timeout <ms> is the other valve: a test that is still running when the clock fires is killed and reported as timed out, rather than hanging the run until something outside echoc stops it. Zero, and the default of not passing the flag at all, wait forever.

A filter that matches no test at all is an error, not a green run of nothing:

[error] No Tests Selected

  nothing this run compiled matches those filters. It declares 3 tests: ...

A file: matches any path ending in what you wrote, at a path boundary, so file:math.eco finds src/math.eco and leaves src/mymath.eco alone. Write as much of the path as you need to be unambiguous and no more.

Names only have to be unique within a file. Two files may each declare a test adds_up, which is why the runner reports module/file::name rather than just the name.

--verbose

A plain run reports a line per test and then its failures, which is what you want from a suite you expect to pass. When you want the transcript instead, ask for it:

echoc test --verbose

mylib/io.eco
  + reads_a_line                                        0 ms
  1 test, 0 ms

mylib/math.eco
  + adds_up                                             0 ms  (unit)
  + subtracts                                           0 ms
  x divides                                             1 ms  exited 1
      assertion failed: nope
        at math.eco:16
  3 tests, 1 ms

4 tests, 1 failed

x mylib/math.eco::divides  exited 1
    assertion failed: nope
      at math.eco:16

Every test that ran, under the file it is written in, with its group, what it cost, and a total per file. The listing replaces the per-test lines rather than decorating them, so what you read on a terminal is what a pipe and a CI log record.

The other thing it buys you is a passing test's own output. Every test's streams are captured whichever way you run, and normally a green one's are thrown away, so a dprint you left in a test that works has no way of reaching you. Under --verbose it is quoted under that test's line.

Coverage

echoc test --coverage counts which lines and branches those processes actually ran, including the ones that assert and die. See Coverage.

--report

--report <file> writes the same run as JSON Lines. The listing you already know does not change.

Each test is one object. A die or a failed assert also carries failure, parsed from the abort text Echo already prints:

{"v":2,"type":"test","id":"main/math.eco::divides","status":"fail",...,"failure":{"headline":"assertion failed","message":"nope","file":"math.eco","line":16}}

A pass, a timeout, and a signal omit it. A #[tests: expects death] test that did die still carries it: the status is pass, and the site is still there. See Crash reports.

What a test knows about itself

Every test process starts with five variables naming the test it runs:

Variable	Holds
ECHO_TEST_ID	module/file::name, the same id the listing and --report print
ECHO_TEST_NAME	the test's name
ECHO_TEST_FILE	the path of the file it is written in
ECHO_TEST_MODULE	the module
ECHO_TEST_GROUP	its #[group:], or empty

#[group: "identity"]
test knows_its_group
{
    assert(std::env::var('ECHO_TEST_GROUP', '') == 'identity');
}

Libraries that sit on test blocks use these: a snapshot named after the test, a report joined on ECHO_TEST_ID. A test that only asserts never needs them.

Test targets

If you find yourself typing the same filters, name them in the manifest:

#[module: "mylib"]
#[version: "0.1.0"]

#[sources: "src/*.eco"]

#[target: test]
#[target: test { name: "quick", groups: ["unit"] }]
#[target: test { name: "io", files: ["src/io.eco"] }]

echoc test --target quick

groups: and files: say exactly what --filter group: and --filter file: say. What makes a test target unlike #[target: exe] rather than another flavour of it is that it produces no binary and names no entry file, so echoc build never builds one and echoc build --target quick is refused.

Here's the bit people miss: a bare echoc test runs every test the module has, whatever targets it declares. A target narrows only when you name it. So #[target: test] on its own buys you the word tests to pass to --target and nothing else.

Modules covers the rest of the manifest.

Which modules

echoc test              # this project's tests
echoc test -m lib       # that library's tests
echoc test math.eco     # the tests of a loose file, the same way `run` takes one

A library you depend on keeps its own tests to itself. They are not parsed, not type checked and not run, which is also why adding a dependency doesn't slow your test runs down. Running them is a matter of pointing at it instead.

Test-only code

Sometimes a test needs a helper that has no business existing in your release build. #[if: tests] is a condition like any other, true exactly when the current invocation is compiling tests:

#[if: tests]
function fixture() : int32
{
    return 42;
}
#[end]

test uses_the_fixture
{
    assert(fixture() == 42);
}

Outside echoc test that function does not exist. Conditional compilation is the rest of that mechanism.

You can't set tests yourself with --define. Whatever compiles your test blocks has to be the same thing that runs them, or you would have a build carrying tests nothing ever calls.

A whole directory of it

Once the helpers outnumber the #[if: tests] lines around them, give the test target a scope and put them in a directory of their own:

#[module: "mylib"]

#[sources: "src/*.eco"]

#[target: test] {
    #[sources: "tests/*.eco"]
}

tests/ is now part of mylib when echoc test builds it, so its files see everything in src/ without a public anywhere, the default visibility being the module's own. Any other invocation does not compile the directory at all, and there is no catch below either: a file a build never reads does not have to lex.

Modules covers what else a scope may say, including a #[depends:] only your tests need.

The catch

A dropped test body still has to be lexable. The filter drops tokens, so your body has to have produced some. A body that doesn't parse is invisible to a normal build:

test never_compiled_here
{
    $a = no_such_function(no_such_argument);
}

echo 'compiles fine';

An unterminated string or a character the lexer doesn't know is a different story, and still fails every build, because lexing happens first.

In practice this means a test body is only checked when you run echoc test, which is the same deal an #[if:] region for another platform gives you. So run your tests.

Two more limits, both on the list: a test is not run under --track-allocations unless you ask, so a leak inside one does not fail it, and running a suite needs echoc, because there is no standalone test binary yet. A test that hangs is what --timeout is for.

Next

• The echoc CLI for how test relates to run and build.
• Modules for giving the test target sources of its own.
• Errors and panics for why a failure ends a process rather than unwinding.