notmuch-test will now call aggregate-results.sh with file list
that it compiles based on the test ran, and aggregate-results
will report failure is any of the test files are missing.
With this notmuch-test no longer has to exit in non-parallel
run if some test fail to write its report file -- so it works
as parallel tests in this sense.
Changed test_done() in test-lib.sh write report file in one write(2),
so there is (even) less chance it being partially written. Also,
now it writes 'total' last and aggregate-results.sh expects this
line to exist in all report files for reporting to be successful.
Added 'set -eu' to notmuch-test and modified code to work with
these settings. That makes it harder to get mistakes slipped
into committed code.
Previously, when all tests were skipped on a test file, there were
no indication of this in the final results aggregate-results.sh
printed.
Now count of the files where all tests were skipped is printed.
- all variables in $((...)) without leading $
- all comparisons use -gt, -eq or -ne
- no -a nor -o inside [ ... ] expressions
- all indentation levels using one tab
Dropped unnecessary empty string check when reading results files.
Replaced pluralize() which was executed in subshell with
pluralize_s(). pluralize_s sets $s to 's' or '' based on value of
$1. Calls to pluralize_s are done in context of current shell, so
no forks to subshells executed.
It isn't completely clear what we want to do here, but
1) We currently don't fail if we skip a whole test file (mainly because
we neglect to count those skipped tests properly). This change at least
makes the two kinds of skipping consistent.
2) Automated build environments may have good reasons for building with
a minimal set of prereqs, and we don't want to discourage running our
test suite by breaking builds.
If any of the tests in our test system is not passing the execution
of the test suite completes with nonzero exit value.
It is better to rely on the exit value of the test system instead
of some arbitrary strings in test output (or use both).
Change #!/bin/bash at start of tests to "#!/usr/bin/env bash". That way
systems running on bash < 4 can prepend bash >= 4 to path before
running the tests.
A bug in the results-aggregation code was causing the test suite to report
"all tests passed" even when there were failures, (as long as there were
also no "broken" tests). Fix this.
This makes the new, git-derived test suite report results in a manner
similar to the original notmuch test suite.
Notable changes include:
* No more initial '*' on every line
* Only colorize a single word
* Don't print useless test numbers
* Use "PASS" in place of "ok"
* Begin sentences with a capital letter
* Print test descriptions for each block
* Separate each block of tests with a blank line
* Don't summarize counts between each block
Git uses a simple and yet powerful test framework, written in shell.
The framework is easy to use for both users and developers so I think
it would help if it is used in notmuch as well.
This is a copy of Git's test framework from commit
b6b0afdc30e066788592ca07c9a6c6936c68cc11 in git repository.
Signed-off-by: Michal Sojka <sojkam1@fel.cvut.cz>
