mirror of
https://git.notmuchmail.org/git/notmuch
synced 2024-11-21 18:38:08 +01:00
209e756cd4
This should be quite handy for doing automated testing of the emacs-based functionality in notmuch. This function invokes emacs with the necessary command-line arguments, (to run in batch mode with no local initialization, to load the notmuch code from the source directory, and to ensure an 80-column width).
195 lines
6.6 KiB
Text
195 lines
6.6 KiB
Text
Notmuch test suite
|
|
==================
|
|
This directory contains the test suite for notmuch.
|
|
|
|
When fixing bugs or enhancing notmuch, you are strongly encouraged to
|
|
add tests in this directory to cover what you are trying to fix or
|
|
enhance.
|
|
|
|
Running Tests
|
|
-------------
|
|
The easiest way to run tests is to say "make test", (or simply run the
|
|
notmuch-test script). Either command will run all available tests.
|
|
|
|
Alternately, you can run a specific subset of tests by simply invoking
|
|
one of the executable scripts in this directory, (such as ./search,
|
|
./reply, etc.)
|
|
|
|
The following command-line options are available when running tests:
|
|
|
|
--debug::
|
|
This may help the person who is developing a new test.
|
|
It causes the command defined with test_debug to run.
|
|
|
|
--immediate::
|
|
This causes the test to immediately exit upon the first
|
|
failed test.
|
|
|
|
--valgrind::
|
|
Execute notmuch with valgrind and exit with status
|
|
126 on errors (just like regular tests, this will only stop
|
|
the test script when running under -i). Valgrind errors
|
|
go to stderr, so you might want to pass the -v option, too.
|
|
|
|
Since it makes no sense to run the tests with --valgrind and
|
|
not see any output, this option implies --verbose. For
|
|
convenience, it also implies --tee.
|
|
|
|
--tee::
|
|
In addition to printing the test output to the terminal,
|
|
write it to files named 't/test-results/$TEST_NAME.out'.
|
|
As the names depend on the tests' file names, it is safe to
|
|
run the tests with this option in parallel.
|
|
|
|
When invoking the test suite via "make test" any of the above options
|
|
can be specified as follows:
|
|
|
|
make test OPTIONS="--verbose"
|
|
|
|
Skipping Tests
|
|
--------------
|
|
If, for any reason, you need to skip one or more tests, you can do so
|
|
by setting the NOTMUCH_SKIP_TESTS variable to the name of one or more
|
|
sections of tests.
|
|
|
|
For example:
|
|
|
|
$ NOTMUCH_SKIP_TESTS="search reply" make test
|
|
|
|
Even more fine-grained skipping is possible by appending a test number
|
|
(or glob pattern) after the section name. For example, the first
|
|
search test and the second reply test could be skipped with:
|
|
|
|
$ NOTMUCH_SKIP_TESTS="search.1 reply.2" make test
|
|
|
|
Note that some tests in the existing test suite rely on previous test
|
|
items, so you cannot arbitrarily skip any test and expect the
|
|
remaining tests to be unaffected.
|
|
|
|
Writing Tests
|
|
-------------
|
|
The test script is written as a shell script. It should start
|
|
with the standard "#!/bin/bash" with copyright notices, and an
|
|
assignment to variable 'test_description', like this:
|
|
|
|
#!/bin/bash
|
|
#
|
|
# Copyright (c) 2005 Junio C Hamano
|
|
#
|
|
|
|
test_description='xxx test (option --frotz)
|
|
|
|
This test exercises the "notmuch xxx" command when
|
|
given the option --frotz.'
|
|
|
|
Source 'test-lib.sh'
|
|
--------------------
|
|
After assigning test_description, the test script should source
|
|
test-lib.sh like this:
|
|
|
|
. ./test-lib.sh
|
|
|
|
This test harness library does the following things:
|
|
|
|
- If the script is invoked with command line argument --help
|
|
(or -h), it shows the test_description and exits.
|
|
|
|
- Creates a temporary directory with default notmuch-config and a
|
|
mail store with a corpus of mail, (initially, 50 early messages
|
|
sent to the notmuch list). This directory is
|
|
test/tmp.<test-basename>. The path to notmuch-config is exported in
|
|
NOTMUCH_CONFIG environment variable and mail store path is stored
|
|
in MAIL_DIR variable.
|
|
|
|
- Defines standard test helper functions for your scripts to
|
|
use. These functions are designed to make all scripts behave
|
|
consistently when command line arguments --verbose (or -v),
|
|
--debug (or -d), and --immediate (or -i) is given.
|
|
|
|
End with test_done
|
|
------------------
|
|
Your script will be a sequence of tests, using helper functions
|
|
from the test harness library. At the end of the script, call
|
|
'test_done'.
|
|
|
|
Test harness library
|
|
--------------------
|
|
There are a handful helper functions defined in the test harness
|
|
library for your script to use.
|
|
|
|
test_expect_success <message> <script>
|
|
|
|
This takes two strings as parameter, and evaluates the
|
|
<script>. If it yields success, test is considered
|
|
successful. <message> should state what it is testing.
|
|
|
|
test_expect_failure <message> <script>
|
|
|
|
This is NOT the opposite of test_expect_success, but is used
|
|
to mark a test that demonstrates a known breakage. Unlike
|
|
the usual test_expect_success tests, which say "ok" on
|
|
success and "FAIL" on failure, this will say "FIXED" on
|
|
success and "still broken" on failure. Failures from these
|
|
tests won't cause -i (immediate) to stop.
|
|
|
|
test_begin_subtest <message>
|
|
|
|
Set the test description message for a subsequent test_expect_equal
|
|
invocation (see below).
|
|
|
|
test_expect_equal <output> <expected>
|
|
|
|
This is an often-used convenience function built on top of
|
|
test_expect_success. It uses the message from the last
|
|
test_begin_subtest call, so call before calling
|
|
test_expect_equal. This function generates a successful test if
|
|
both the <output> and <expected> strings are identical. If not, it
|
|
will generate a failure and print the difference of the two
|
|
strings.
|
|
|
|
test_debug <script>
|
|
|
|
This takes a single argument, <script>, and evaluates it only
|
|
when the test script is started with --debug command line
|
|
argument. This is primarily meant for use during the
|
|
development of a new test script.
|
|
|
|
test_emacs <emacs-lisp-expressions>
|
|
|
|
This function executes the provided emacs lisp script within
|
|
emacs. The script can be a sequence of emacs lisp expressions,
|
|
(that is, they will be evaluated within a progn form). The lisp
|
|
expressions can call `message' to generate output on stdout to be
|
|
examined by the calling test script.
|
|
|
|
test_done
|
|
|
|
Your test script must have test_done at the end. Its purpose
|
|
is to summarize successes and failures in the test script and
|
|
exit with an appropriate error code.
|
|
|
|
There are also a number of mail-specific functions which are useful in
|
|
writing tests:
|
|
|
|
generate_message
|
|
|
|
Generates a message with an optional template. Most tests will
|
|
actually prefere to call add_message. See below.
|
|
|
|
add_message
|
|
|
|
Generate a message and add it to the database (by calling "notmuch
|
|
new"). It is sufficient to simply call add_message with no
|
|
arguments if you don't care about the content of the message. If
|
|
more control is needed, arguments can be provide to specify many
|
|
different header values for the new message. See the documentation
|
|
within test-lib.sh or refer to many example calls within existing
|
|
tests.
|
|
|
|
add_email_corpus
|
|
|
|
This function should be called at the beginning of a test file
|
|
when a test needs to operate on a non-empty body of messages. It
|
|
will intialize the mail database to a known state of 50 sample
|
|
messages, (culled from the early history of the notmuch mailing
|
|
list).
|