mirror of
https://git.notmuchmail.org/git/notmuch
synced 2024-11-22 02:48:08 +01:00
3977b250cd
"Seperate: The second most common misspelling on the Internet..." (from http://www.re-vision.com/spelling/separate.html). s/seperate/separate/ in debian/NEWS.Debian & test/README done.
255 lines
9 KiB
Text
255 lines
9 KiB
Text
Notmuch test suite
|
|
==================
|
|
This directory contains the test suite for notmuch.
|
|
|
|
When fixing bugs or enhancing notmuch, you are strongly encouraged to
|
|
add tests in this directory to cover what you are trying to fix or
|
|
enhance.
|
|
|
|
Prerequisites
|
|
-------------
|
|
Some tests require external dependencies to run. Without them, they
|
|
will be skipped, or (rarely) marked failed. Please install these, so
|
|
that you know if you break anything.
|
|
|
|
- dtach(1)
|
|
- emacs(1)
|
|
- emacsclient(1)
|
|
- gdb(1)
|
|
- gpg(1)
|
|
- python(1)
|
|
|
|
Running Tests
|
|
-------------
|
|
The easiest way to run tests is to say "make test", (or simply run the
|
|
notmuch-test script). Either command will run all available tests.
|
|
|
|
Alternately, you can run a specific subset of tests by simply invoking
|
|
one of the executable scripts in this directory, (such as ./search,
|
|
./reply, etc). Note that you will probably want "make test-binaries"
|
|
before running individual tests.
|
|
|
|
The following command-line options are available when running tests:
|
|
|
|
--debug::
|
|
This may help the person who is developing a new test.
|
|
It causes the command defined with test_debug to run.
|
|
|
|
--immediate::
|
|
This causes the test to immediately exit upon the first
|
|
failed test.
|
|
|
|
--valgrind::
|
|
Execute notmuch with valgrind and exit with status
|
|
126 on errors (just like regular tests, this will only stop
|
|
the test script when running under -i). Valgrind errors
|
|
go to stderr, so you might want to pass the -v option, too.
|
|
|
|
Since it makes no sense to run the tests with --valgrind and
|
|
not see any output, this option implies --verbose. For
|
|
convenience, it also implies --tee.
|
|
|
|
--tee::
|
|
In addition to printing the test output to the terminal,
|
|
write it to files named 't/test-results/$TEST_NAME.out'.
|
|
As the names depend on the tests' file names, it is safe to
|
|
run the tests with this option in parallel.
|
|
|
|
--root=<dir>::
|
|
This runs the testsuites specified under a separate directory.
|
|
However, caution is advised, as not all tests are maintained
|
|
with this relocation in mind, so some tests may behave
|
|
differently.
|
|
|
|
Pointing this argument at a tmpfs filesystem can improve the
|
|
speed of the test suite for some users.
|
|
|
|
When invoking the test suite via "make test" any of the above options
|
|
can be specified as follows:
|
|
|
|
make test OPTIONS="--verbose"
|
|
|
|
You can choose an emacs binary to run the tests in one of the
|
|
following ways.
|
|
|
|
TEST_EMACS=my-special-emacs make test
|
|
TEST_EMACS=my-special-emacs ./emacs
|
|
make test TEST_EMACS=my-special-emacs
|
|
|
|
Skipping Tests
|
|
--------------
|
|
If, for any reason, you need to skip one or more tests, you can do so
|
|
by setting the NOTMUCH_SKIP_TESTS variable to the name of one or more
|
|
sections of tests.
|
|
|
|
For example:
|
|
|
|
$ NOTMUCH_SKIP_TESTS="search reply" make test
|
|
|
|
Even more fine-grained skipping is possible by appending a test number
|
|
(or glob pattern) after the section name. For example, the first
|
|
search test and the second reply test could be skipped with:
|
|
|
|
$ NOTMUCH_SKIP_TESTS="search.1 reply.2" make test
|
|
|
|
Note that some tests in the existing test suite rely on previous test
|
|
items, so you cannot arbitrarily skip any test and expect the
|
|
remaining tests to be unaffected.
|
|
|
|
Writing Tests
|
|
-------------
|
|
The test script is written as a shell script. It should start with
|
|
the standard "#!/usr/bin/env bash" with copyright notices, and an
|
|
assignment to variable 'test_description', like this:
|
|
|
|
#!/usr/bin/env bash
|
|
#
|
|
# Copyright (c) 2005 Junio C Hamano
|
|
#
|
|
|
|
test_description='xxx test (option --frotz)
|
|
|
|
This test exercises the "notmuch xxx" command when
|
|
given the option --frotz.'
|
|
|
|
Source 'test-lib.sh'
|
|
--------------------
|
|
After assigning test_description, the test script should source
|
|
test-lib.sh like this:
|
|
|
|
. ./test-lib.sh
|
|
|
|
This test harness library does the following things:
|
|
|
|
- If the script is invoked with command line argument --help
|
|
(or -h), it shows the test_description and exits.
|
|
|
|
- Creates a temporary directory with default notmuch-config and a
|
|
mail store with a corpus of mail, (initially, 50 early messages
|
|
sent to the notmuch list). This directory is
|
|
test/tmp.<test-basename>. The path to notmuch-config is exported in
|
|
NOTMUCH_CONFIG environment variable and mail store path is stored
|
|
in MAIL_DIR variable.
|
|
|
|
- Defines standard test helper functions for your scripts to
|
|
use. These functions are designed to make all scripts behave
|
|
consistently when command line arguments --verbose (or -v),
|
|
--debug (or -d), and --immediate (or -i) is given.
|
|
|
|
End with test_done
|
|
------------------
|
|
Your script will be a sequence of tests, using helper functions
|
|
from the test harness library. At the end of the script, call
|
|
'test_done'.
|
|
|
|
Test harness library
|
|
--------------------
|
|
There are a handful helper functions defined in the test harness
|
|
library for your script to use.
|
|
|
|
test_expect_success <message> <script>
|
|
|
|
This takes two strings as parameter, and evaluates the
|
|
<script>. If it yields success, test is considered
|
|
successful. <message> should state what it is testing.
|
|
|
|
test_begin_subtest <message>
|
|
|
|
Set the test description message for a subsequent test_expect_equal
|
|
invocation (see below).
|
|
|
|
test_subtest_known_broken
|
|
|
|
Mark the current test as broken. Such tests are expected to fail.
|
|
Unlike the normal tests, which say "PASS" on success and "FAIL" on
|
|
failure, these will say "FIXED" on success and "BROKEN" on failure.
|
|
Failures from these tests won't cause -i (immediate) to stop. A
|
|
test must call this before any test_expect_* function.
|
|
|
|
test_expect_equal <output> <expected>
|
|
|
|
This is an often-used convenience function built on top of
|
|
test_expect_success. It uses the message from the last
|
|
test_begin_subtest call, so call before calling
|
|
test_expect_equal. This function generates a successful test if
|
|
both the <output> and <expected> strings are identical. If not, it
|
|
will generate a failure and print the difference of the two
|
|
strings.
|
|
|
|
test_expect_equal_file <file1> <file2>
|
|
|
|
Identical to test_exepect_equal, except that <file1> and <file2>
|
|
are files instead of strings. This is a much more robust method to
|
|
compare formatted textual information, since it also notices
|
|
whitespace and closing newline differences.
|
|
|
|
test_debug <script>
|
|
|
|
This takes a single argument, <script>, and evaluates it only
|
|
when the test script is started with --debug command line
|
|
argument. This is primarily meant for use during the
|
|
development of a new test script.
|
|
|
|
test_emacs <emacs-lisp-expressions>
|
|
|
|
This function executes the provided emacs lisp script within
|
|
emacs. The script can be a sequence of emacs lisp expressions,
|
|
(that is, they will be evaluated within a progn form). Emacs
|
|
stdout and stderr is not available, the common way to get output
|
|
is to save it to a file. There are some auxiliary functions
|
|
useful in emacs tests provided in test-lib.el. Do not use `setq'
|
|
for setting variables in Emacs tests because it affects other
|
|
tests that may run in the same Emacs instance. Use `let' instead
|
|
so the scope of the changed variables is limited to a single test.
|
|
|
|
test_emacs_expect_t <emacs-lisp-expressions>
|
|
|
|
This function executes the provided emacs lisp script within
|
|
emacs in a manner similar to 'test_emacs'. The expressions should
|
|
return the value `t' to indicate that the test has passed. If the
|
|
test does not return `t' then it is considered failed and all data
|
|
returned by the test is reported to the tester.
|
|
|
|
test_done
|
|
|
|
Your test script must have test_done at the end. Its purpose
|
|
is to summarize successes and failures in the test script and
|
|
exit with an appropriate error code.
|
|
|
|
There are also a number of notmuch-specific auxiliary functions and
|
|
variables which are useful in writing tests:
|
|
|
|
generate_message
|
|
|
|
Generates a message with an optional template. Most tests will
|
|
actually prefer to call add_message. See below.
|
|
|
|
add_message
|
|
|
|
Generate a message and add it to the database (by calling "notmuch
|
|
new"). It is sufficient to simply call add_message with no
|
|
arguments if you don't care about the content of the message. If
|
|
more control is needed, arguments can be provide to specify many
|
|
different header values for the new message. See the documentation
|
|
within test-lib.sh or refer to many example calls within existing
|
|
tests.
|
|
|
|
add_email_corpus
|
|
|
|
This function should be called at the beginning of a test file
|
|
when a test needs to operate on a non-empty body of messages. It
|
|
will initialize the mail database to a known state of 50 sample
|
|
messages, (culled from the early history of the notmuch mailing
|
|
list).
|
|
|
|
notmuch_counter_reset
|
|
$notmuch_counter_command
|
|
notmuch_counter_value
|
|
|
|
These allow to count how many times notmuch binary is called.
|
|
notmuch_counter_reset() function generates a script that counts
|
|
how many times it is called and resets the counter to zero. The
|
|
function sets $notmuch_counter_command variable to the path to the
|
|
generated script that should be called instead of notmuch to do
|
|
the counting. The notmuch_counter_value() function prints the
|
|
current counter value.
|