test suite: don't consider skipped individual tests as failing

It isn't completely clear what we want to do here, but

1) We currently don't fail if we skip a whole test file (mainly because
we neglect to count those skipped tests properly). This change at least
makes the two kinds of skipping consistent.

2) Automated build environments may have good reasons for building with
a minimal set of prereqs, and we don't want to discourage running our
test suite by breaking builds.
This commit is contained in:
David Bremner 2015-09-24 08:13:30 -03:00
parent 3c46a9e6df
commit 3cf7ed26c0
2 changed files with 11 additions and 1 deletions

View file

@ -117,6 +117,13 @@ Note that some tests in the existing test suite rely on previous test
items, so you cannot arbitrarily skip any test and expect the items, so you cannot arbitrarily skip any test and expect the
remaining tests to be unaffected. remaining tests to be unaffected.
Currently we do not consider skipped tests as build failures. For
maximum robustness, when setting up automated build processes, you
should explicitely skip tests, rather than relying on notmuch's
detection of missing prerequisites. In the future we may treat tests
unable to run because of missing prerequisites, but not explicitely
skipped by the user, as failures.
Writing Tests Writing Tests
------------- -------------
The test script is written as a shell script. It should start with The test script is written as a shell script. It should start with

View file

@ -82,7 +82,10 @@ if [ "$skipped" != "0" ]; then
echo "$skipped $tests skipped." echo "$skipped $tests skipped."
fi fi
if [ $success -gt 0 -a $fixed -eq 0 -a $failed -eq 0 -a $skipped -eq 0 ] # Note that we currently do not consider skipped tests as failing the
# build.
if [ $success -gt 0 -a $fixed -eq 0 -a $failed -eq 0 ]
then then
exit 0 exit 0
else else