From 3cf7ed26c06fb3fa7145948fd9a9f2973037a5fd Mon Sep 17 00:00:00 2001 From: David Bremner Date: Thu, 24 Sep 2015 08:13:30 -0300 Subject: [PATCH] test suite: don't consider skipped individual tests as failing It isn't completely clear what we want to do here, but 1) We currently don't fail if we skip a whole test file (mainly because we neglect to count those skipped tests properly). This change at least makes the two kinds of skipping consistent. 2) Automated build environments may have good reasons for building with a minimal set of prereqs, and we don't want to discourage running our test suite by breaking builds. --- test/README | 7 +++++++ test/aggregate-results.sh | 5 ++++- 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/test/README b/test/README index ce403cef..e54e36b7 100644 --- a/test/README +++ b/test/README @@ -117,6 +117,13 @@ Note that some tests in the existing test suite rely on previous test items, so you cannot arbitrarily skip any test and expect the remaining tests to be unaffected. +Currently we do not consider skipped tests as build failures. For +maximum robustness, when setting up automated build processes, you +should explicitely skip tests, rather than relying on notmuch's +detection of missing prerequisites. In the future we may treat tests +unable to run because of missing prerequisites, but not explicitely +skipped by the user, as failures. + Writing Tests ------------- The test script is written as a shell script. It should start with diff --git a/test/aggregate-results.sh b/test/aggregate-results.sh index b016edb9..63228546 100755 --- a/test/aggregate-results.sh +++ b/test/aggregate-results.sh @@ -82,7 +82,10 @@ if [ "$skipped" != "0" ]; then echo "$skipped $tests skipped." fi -if [ $success -gt 0 -a $fixed -eq 0 -a $failed -eq 0 -a $skipped -eq 0 ] +# Note that we currently do not consider skipped tests as failing the +# build. + +if [ $success -gt 0 -a $fixed -eq 0 -a $failed -eq 0 ] then exit 0 else