In certain cases of test suite failure, the summary report was not
being printed. In particular, any failure on the parallel test suite,
and any aborted test in the serialized test suite would end up hiding
the summary.
It's better to always show the summary where we can (while preserving
the return code). If we do abort due to this high-level failure,
though, we should also announce to the user that we're doing so as
close to the end of the process as possible, to make it easier to find
the problem.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
To aid in diagnosing test suite tooling that interacts poorly with
coreutils' timeout, it's handy to be able to bypass it entirely.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Done via $COLORS_WITHOUT_TTY environment variable as passing options
to commands through parallel(1) does not look trivial.
Reorganized color checking in test-lib.sh a bit for this (perhaps
were not fully necessary but rest still an improvement):
- color checking commands in subshell are not run before arg parsing
(args may disable colors with --no-color)
- [ -t 1 ] is checked before forking subshell
When a parallel build fails (or when it times out, if timeout is
present), the test suite should not blithely succeed. Catch these
failures and at least report them.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
The current 2 minute timeout is reasonable, but to exercise the test
suite or induce timeout failures, we might want to make it shorter.
This makes it configurable so you can run (for example):
make check NOTMUCH_TEST_TIMEOUT=10s
We stick with the default of 2m.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
If either the moreutils or GNU parallel utility are available, run all
tests in parallel. On my eight core machine this makes for a ~x7
speed-up in the full test suite (1m24s -> 12s).
The design of the test suite makes this parallelization trivial.
The add_email_corpus test utility includes logic that tries to re-use
an index of the corpus if available. This was seemingly done as an
optimization, so that every test that uses the corpus didn't have to
create it's own index of the corpus. However, this has the perverse
side effect of entangling tests together, and breaks parallelization.
Forcing each test to do it's own index does increase the overall time
of the test slightly (~6%), but this will be more than made up for in
the next patch that introduces paraellization.
971cdc72cd renamed corpus.mail to
corpora.mail. Although 971cdc72cd
updated some of the remaining corpus.mail references, two remained,
causing the test suite to leave behind an unignored corpora.mail
directory.
A few simple things that applies well to test/notmuch-test.sh
- Shell does pathname expansion also without doing `echo ...` in subshell.
- Redirections >/dev/null 2>/dev/null hide (improbable but) potential
serious errors; adding -f to rm instead.
- Inter-file capitalization consistency in comments.
- Unnecesary space removal.
Files in test directories had only copyright of a single individual,
of which code was adapted here as a base of the test system.
Since then many Notmuch Developers have contributed to the test
system, which is now acknowledged with a constant string in some
of the test files.
The README file in test directory instructed new files contain a
copyright notice, but that has never been done (and it is also not
needed). To simplify things a bit (and lessen confusion) this
instruction is now removed.
As a side enchangement, all of the 3 entries in the whole source
tree cd'ing to `dirname` of "$0" now uses syntax cd "$(dirname "$0")".
This makes these particular lines work when current working directory
is e.g. /c/Program Files/notmuch/test/.
(Probably it would fail elsewhere, though.)
All test scripts to be executed are now named as T\d\d\d-name.sh,
numers in increments of 10.
This eases adding new tests and developers to see which are test scripts
that are executed by test suite and in which order.
In preparation for quiet mode print empty line before writing the
test description. This is done now in function designed for it --
it will also be called when test fails.
If any of the tests in our test system is not passing the execution
of the test suite completes with nonzero exit value.
It is better to rely on the exit value of the test system instead
of some arbitrary strings in test output (or use both).
Previously, if a test script aborted (e.g., because it passed too few
arguments to a test function), the test driver loop would simply
continue on to the next test script and the final results would
declare that everything passed (except that the test count would look
suspiciously low, but maybe you just misremembered how many tests
there were).
Now, if a test script exits with a non-zero status and did not produce
a final results file, we propagate that failure out of the driver loop
immediately.
To keep this simple, this patch removes the PID from the test-results
file name. This PID was inherited from the git test system and seems
unnecessary, since the file name already includes the name of the test
script and the test-results directory is created anew for each run.
Test the date/time parser module directly, independent of notmuch,
using the parse-time test tool.
Credits to Michal Sojka <sojkam1@fel.cvut.cz> for writing most of the
tests.
There didn't seem to be these basic tests for --format=text,
as there are for --format=json. These are just the tests from
the `json' script, with adjusted expected outputs.
Systematically test the exclude options for search. Also move the
search existing exclude tests into the new test. There is some overlap
between the two sets of tests but many of the existing ones are there
because they triggered bugs in the past so I have kept them to ensure
coverage.
The test is currently broken and will be fixed by a subsequent patch.
The patch adds a new file for tests of Emacs notmuch-show view.
Based on patch by David Edmondson [1].
[1] id:"1327562380-12894-4-git-send-email-dme@dme.org"
Add a new test function to allow simpler testing of emacs
functionality.
`test_emacs_expect_t' takes one argument - a lisp expression to
evaluate. The test passes if the expression returns `t', otherwise it
fails and the output is reported to the tester.
We start modestly, with a (slightly modified) test case from Kazuo
Teramoto. Originally it just made sure the bindings didn't crash; here
we check that by comparing the output with that of notmuch search.
Some tests don't break when HUP signal is sent tho those (by
pressing ctrl-c on the terminal). Therefore, the top-level
test script catches the HUP and sends TERM signal to the
started test script.
This tests notmuch new's ability to recover from arbitrary stopping
failures. It interrupts notmuch new after every database commit and,
on every resulting database snapshot, re-runs notmuch new to
completion and checks that the final database state is invariant.
Test for bug. Current stemming support for notmuch adds extra terms
to the DB which aren't removed when the file renames are detected.
When folder tags are added to a message, Xapian terms for both XFOLDER
and ZXFOLDER are generated. When one of the filenames are
renamed/removed, only the XFOLDER tags are removed, leaving it possible
for a match on a folder: tag that was previously but is no longer a
match in the maildir.
The worry here is that a binary linking with libnotmuch might lose
access to Xapian::Error symbols because libnotmuch hides them.
We are careful here to create ./fakedb/.notmuch in order to trigger a
Xapian exception, and not just a missing file check.
Thanks to jrollins and amddragon for suggestions.
(cherry picked from commit 66f37f5f6864a988f94ddb893e3a176af57f6c8e)
Each top level test (basic, corpus, etc...) is run with a fixed
timeout of 2 minutes.
The goal here is to treat a hung test as a failure. The emacs test for
sending mail is known to be problematic on the Debian
autobuilders. This is both a bandaid fix for that, and a sensible long
term feature.
(cherry picked from commit 5f99c80e02736c90495558d9b88008a768876b29)
