debian/notmuch
1
0
Fork 0
mirror of https://git.notmuchmail.org/git/notmuch synced 2024-11-25 04:18:08 +01:00
Code Issues Projects Releases Packages Wiki Activity Actions
notmuch/test/notmuch-test

109 lines
2 KiB
Text
Raw Normal View History

test: change "#!/bin/bash" to "#!/usr/bin/env bash" enhances portability Change #!/bin/bash at start of tests to "#!/usr/bin/env bash". That way systems running on bash < 4 can prepend bash >= 4 to path before running the tests.
2010-12-01 21:27:52 +01:00
#!/usr/bin/env bash
Add a very rough beginning of a test suite. This notmuch-test script simply runs a few different notmuch operations, (things that I found were useful while testing the rename-support code). It's not useful as a test suite yet, since it doesn't actually check the results of any operation, (the user of the suite has to know what the results should be and must manually verify them. So there's no integration with the build system yet, (no "make test" target). But I didn't want to lose what I had so far, so here it is.
2010-01-23 19:36:39 +01:00
test: Fix test suite to integrate with our non-recursive Makefile system. This avoids "make test" emitting messages from three (3!) recursive invocations of make. We change the invocations of the tests themselves to occur directly from the shell script rather than having the shell script invoke make again and using wildcards in the Makefile.
2010-09-17 21:10:54 +02:00
# Run tests
#
# Copyright (c) 2005 Junio C Hamano
#
# Adapted from a Makefile to a shell script by Carl Worth (2010)
test: Emit a friendly error message if run with bash < 4.0. The bash code in the test suite is using associative arrays which were only added to bash as of release 4.0. If the test suite is run with an older bash, we now immediately error out and explain the situation, (instead of emitting confusing error messages and failing dozens of tests, which is what happened before this change).
2010-10-28 21:07:42 +02:00
if [ ${BASH_VERSINFO[0]} -lt 4 ]; then
echo "Error: The notmuch test suite requires a bash version >= 4.0"
echo "due to use of associative arrays within the test suite."
echo "Please try again with a newer bash (or help us fix the"
echo "test suite to be more portable). Thanks."
exit 1
fi
test: Fix test suite to integrate with our non-recursive Makefile system. This avoids "make test" emitting messages from three (3!) recursive invocations of make. We change the invocations of the tests themselves to occur directly from the shell script rather than having the shell script invoke make again and using wildcards in the Makefile.
2010-09-17 21:10:54 +02:00
cd $(dirname "$0")
test: Make it easier to resolve conflicts when adding new tests Currently, there are two places in the test framework that contain very long list on a single line. Whenever a test is added (or changed) in several branches and these branches are merged, it results in conflict which is hard to resolve because one has to go through the whole long line to find where the conflict is. This patch splits these long lists to several lines so that the conflicts are easier to resolve.
2011-01-23 12:12:24 +01:00
TESTS="
basic
test: add simple tests for online help Nothing fancy, but we can at least detect segmentation faults.
2011-11-25 18:28:11 +01:00
help-test
test: Add compact test Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-31 05:10:08 +01:00
compact
test: Add tests for 'config' command Start a new test script.
2012-04-14 03:41:03 +02:00
config
test: add basic test for notmuch setup And annotate with test_subtest_known_broken. Hooray.
2013-05-05 21:18:19 +02:00
setup
test: Make it easier to resolve conflicts when adding new tests Currently, there are two places in the test framework that contain very long list on a single line. Whenever a test is added (or changed) in several branches and these branches are merged, it results in conflict which is hard to resolve because one has to go through the whole long line to find where the conflict is. This patch splits these long lists to several lines so that the conflicts are easier to resolve.
2011-01-23 12:12:24 +01:00
new
test: add tests for notmuch count Signed-off-by: Jani Nikula <jani@nikula.org>
2011-11-15 21:08:52 +01:00
count
test: add tests for insert Add tests for new 'insert' command.
2013-06-23 06:23:59 +02:00
insert
test: Make it easier to resolve conflicts when adding new tests Currently, there are two places in the test framework that contain very long list on a single line. Whenever a test is added (or changed) in several branches and these branches are merged, it results in conflict which is hard to resolve because one has to go through the whole long line to find where the conflict is. This patch splits these long lists to several lines so that the conflicts are easier to resolve.
2011-01-23 12:12:24 +01:00
search
search-output
search-by-folder
search-position-overlap-bug
test: Rename and clarify the search-lwn test Now that we understand the bug here, we rename this test to search-insufficient-from-quoting to clarify the bug being exercised, (which occurs when the From: line contains an unquoted '.' character). We also mark these tests as expected failures until the bug gets fixed.
2011-01-28 21:21:25 +01:00
search-insufficient-from-quoting
test: add tests for notmuch search --offset and --limit Signed-off-by: Jani Nikula <jani@nikula.org>
2011-11-15 21:08:53 +01:00
search-limiting
test: add some exclude tests Systematically test the exclude options for search. Also move the search existing exclude tests into the new test. There is some overlap between the two sets of tests but many of the existing ones are there because they triggered bugs in the past so I have kept them to ensure coverage.
2012-04-07 18:10:07 +02:00
excludes
test: Add a test script for "notmuch tag"
2011-11-08 04:57:11 +01:00
tagging
test: Make it easier to resolve conflicts when adding new tests Currently, there are two places in the test framework that contain very long list on a single line. Whenever a test is added (or changed) in several branches and these branches are merged, it results in conflict which is hard to resolve because one has to go through the whole long line to find where the conflict is. This patch splits these long lists to several lines so that the conflicts are easier to resolve.
2011-01-23 12:12:24 +01:00
json
Adding tests for --format=sexp. Add basic tests, the same as for json, for the S-Expression output format.
2012-12-06 22:12:14 +01:00
sexp
test: add basic show, search --format=text tests There didn't seem to be these basic tests for --format=text, as there are for --format=json. These are just the tests from the `json' script, with adjusted expected outputs.
2012-06-24 05:27:48 +02:00
text
test: Add a test of "notmuch show" with a multipart message This tests "notmuch show" with both --format=text and --format=json on a message with some non-trivial MIME multipart nesting, (multiple parts within a multipart/mixed part which is within a multipart/signed part). The test captures the current behavior (where only the leaf nodes of the MIME structure are emitted as a flat list---the multipart parts are effectively ignored). We plan to soon change the json output at least to emit an actual hierarchy matching the MIME structure, (at which point we will update this test).
2011-05-17 00:25:25 +02:00
multipart
test: Make it easier to resolve conflicts when adding new tests Currently, there are two places in the test framework that contain very long list on a single line. Whenever a test is added (or changed) in several branches and these branches are merged, it results in conflict which is hard to resolve because one has to go through the whole long line to find where the conflict is. This patch splits these long lists to several lines so that the conflicts are easier to resolve.
2011-01-23 12:12:24 +01:00
thread-naming
raw
reply
test: add tests for "notmuch reply" --reply-to=sender
2012-01-14 15:46:19 +01:00
reply-to-sender
test: Make it easier to resolve conflicts when adding new tests Currently, there are two places in the test framework that contain very long list on a single line. Whenever a test is added (or changed) in several branches and these branches are merged, it results in conflict which is hard to resolve because one has to go through the whole long line to find where the conflict is. This patch splits these long lists to several lines so that the conflicts are easier to resolve.
2011-01-23 12:12:24 +01:00
dump-restore
uuencode
thread-order
author-order
from-guessing
long-id
encoding
emacs
Rename/rewrite the new emacs-forgetfulness test (to emacs-large-search-buffer) The new name is more descriptive of the bug being tested. Also, the test is rewritten slightly so that it's much more plain to see how the bug manifests itself, (that messages are droped from the emacs result at regular intervals). Primarily, this is by collapsing the large blobs used to inflate the message subjects.
2011-03-10 22:22:04 +01:00
emacs-large-search-buffer
test: emacs: test notmuch-wash-subject-to-* functions Signed-off-by: Jani Nikula <jani@nikula.org>
2011-12-27 17:04:40 +01:00
emacs-subject-to-filename
test: Make it easier to resolve conflicts when adding new tests Currently, there are two places in the test framework that contain very long list on a single line. Whenever a test is added (or changed) in several branches and these branches are merged, it results in conflict which is hard to resolve because one has to go through the whole long line to find where the conflict is. This patch splits these long lists to several lines so that the conflicts are easier to resolve.
2011-01-23 12:12:24 +01:00
maildir-sync
test: add crypto tests for signature verification and decryption This adds a new "crypto" test script to the test suite to test PGP/MIME signature verification and message decryption. Included here is a test GNUPGHOME with a test secret key (passwordless), and test for: * signing/verification * signing/verification with full owner trust * verification with signer key unavailable * encryption/decryption * decryption failure with missing key * encryption/decryption + signing/verfifying * reply to encrypted message * verification of signature from revoked key These tests are not expected to pass now, but will as crypto functionality is included.
2011-05-26 03:01:16 +02:00
crypto
tests: add a test for symbol hiding side effects The worry here is that a binary linking with libnotmuch might lose access to Xapian::Error symbols because libnotmuch hides them. We are careful here to create ./fakedb/.notmuch in order to trigger a Xapian exception, and not just a missing file check. Thanks to jrollins and amddragon for suggestions. (cherry picked from commit 66f37f5f6864a988f94ddb893e3a176af57f6c8e)
2011-06-22 13:58:01 +02:00
symbol-hiding
test:Folder tags shouldn't match after removal of file in given folder Test for bug. Current stemming support for notmuch adds extra terms to the DB which aren't removed when the file renames are detected. When folder tags are added to a message, Xapian terms for both XFOLDER and ZXFOLDER are generated. When one of the filenames are renamed/removed, only the XFOLDER tags are removed, leaving it possible for a match on a folder: tag that was previously but is no longer a match in the maildir.
2011-06-27 19:12:24 +02:00
search-folder-coherence
test: Test atomicity of notmuch new. This tests notmuch new's ability to recover from arbitrary stopping failures. It interrupts notmuch new after every database commit and, on every resulting database snapshot, re-runs notmuch new to completion and checks that the final database state is invariant.
2011-01-28 19:03:24 +01:00
atomicity
test: add tests for python bindings We start modestly, with a (slightly modified) test case from Kazuo Teramoto. Originally it just made sure the bindings didn't crash; here we check that by comparing the output with that of notmuch search.
2011-12-05 02:50:08 +01:00
python
test: add tests for hooks Signed-off-by: Jani Nikula <jani@nikula.org>
2011-12-08 23:48:31 +01:00
hooks
test: tests for command-line-arguments.c This was needed because no current notmuch code exercises the NOTMUCH_OPT_STRING style arguments.
2011-12-07 19:37:13 +01:00
argument-parsing
test: remove ".sh" extension from the recently added Emacs tests All test files, except for the recently added Emacs tests, do not have ".sh" extension. So remove it from the new test files for consistency.
2012-01-28 05:47:39 +01:00
emacs-test-functions
emacs-address-cleaning
emacs: Tests for user-defined sections A new file was added for notmuch-hello tests.
2012-02-17 15:48:08 +01:00
emacs-hello
test: add test for hiding Original Message region at beginning of a message The test is currently broken and will be fixed by a subsequent patch. The patch adds a new file for tests of Emacs notmuch-show view. Based on patch by David Edmondson [1]. [1] id:"1327562380-12894-4-git-send-email-dme@dme.org"
2012-02-04 08:36:36 +01:00
emacs-show
test: move emacs-tree test into mainline We move the emacs-tree test and associated files into the main test directory and add the test to the list in notmuch-test.
2013-10-29 23:55:32 +01:00
emacs-tree
test: Add test for messages with missing headers Currently the JSON tests for search and show are broken because notmuch attempts to dereference a NULL pointer.
2012-08-08 03:28:33 +02:00
missing-headers
test/hex-escaping: new test for hex escaping routines These are more like unit tests, to (try to) make sure the library functionality is working before building more complicated things on top of it.
2012-06-15 00:08:38 +02:00
hex-escaping
test: add smoke tests for the date/time parser module Test the date/time parser module directly, independent of notmuch, using the parse-time test tool. Credits to Michal Sojka <sojkam1@fel.cvut.cz> for writing most of the tests.
2012-10-30 21:32:35 +01:00
parse-time-string
test: add tests for date:since..until range queries A brief initial test set.
2012-10-30 21:32:38 +01:00
search-date
test: add tests for the handling of References and In-Reply-To headers These tests are known_broken, the following commit fixes them. amended per id:87txmi1zq3.fsf@nikula.org id:87vc6yalo7.fsf@zancas.localnet
2013-03-06 04:31:48 +01:00
thread-replies
test: Make it easier to resolve conflicts when adding new tests Currently, there are two places in the test framework that contain very long list on a single line. Whenever a test is added (or changed) in several branches and these branches are merged, it results in conflict which is hard to resolve because one has to go through the whole long line to find where the conflict is. This patch splits these long lists to several lines so that the conflicts are easier to resolve.
2011-01-23 12:12:24 +01:00
"
test: allow specifying tests to run with NOTMUCH_TESTS env var This is useful for just running a specific subset of tests, ie: NOTMUCH_TESTS=crypto make test
2011-05-07 21:08:34 +02:00
TESTS=${NOTMUCH_TESTS:=$TESTS}
test: Rename all tests to get rid of the ugly numbers in file names. The numbers were meaningless, and they made it hard to find a file of interest. Instead, we get the ordering we want by adding an explicit list of tests to run to the notmuch-test script.
2010-09-18 00:58:39 +02:00
test: Fix test suite to integrate with our non-recursive Makefile system. This avoids "make test" emitting messages from three (3!) recursive invocations of make. We change the invocations of the tests themselves to occur directly from the shell script rather than having the shell script invoke make again and using wildcards in the Makefile.
2010-09-17 21:10:54 +02:00
# Clean up any results from a previous run
Revert "test: Break on test script (or other) error" This reverts commit f22a7ec1e28d1264cf9d67d78796b8ab22e09a35. Interrupting the test suite due to an actual bug in a test script would be just fine, but interrupting the run of the entire test suite at the first test failure is unacceptable.
2010-12-08 00:28:44 +01:00
rm -r test-results >/dev/null 2>/dev/null
test: Fix test suite to integrate with our non-recursive Makefile system. This avoids "make test" emitting messages from three (3!) recursive invocations of make. We change the invocations of the tests themselves to occur directly from the shell script rather than having the shell script invoke make again and using wildcards in the Makefile.
2010-09-17 21:10:54 +02:00
tests: Add optional use of timeout utility, if present. Each top level test (basic, corpus, etc...) is run with a fixed timeout of 2 minutes. The goal here is to treat a hung test as a failure. The emacs test for sending mail is known to be problematic on the Debian autobuilders. This is both a bandaid fix for that, and a sensible long term feature. (cherry picked from commit 5f99c80e02736c90495558d9b88008a768876b29)
2011-05-28 11:50:11 +02:00
# test for timeout utility
if command -v timeout >/dev/null; then
TEST_TIMEOUT_CMD="timeout 2m "
echo "INFO: using 2 minute timeout for tests"
else
TEST_TIMEOUT_CMD=""
fi
test: make all tests terminable with Ctrl-c Some tests don't break when HUP signal is sent tho those (by pressing ctrl-c on the terminal). Therefore, the top-level test script catches the HUP and sends TERM signal to the started test script.
2011-11-08 17:02:25 +01:00
trap 'e=$?; kill $!; exit $e' HUP INT TERM
test: Fix test suite to integrate with our non-recursive Makefile system. This avoids "make test" emitting messages from three (3!) recursive invocations of make. We change the invocations of the tests themselves to occur directly from the shell script rather than having the shell script invoke make again and using wildcards in the Makefile.
2010-09-17 21:10:54 +02:00
# Run the tests
test: Rename all tests to get rid of the ugly numbers in file names. The numbers were meaningless, and they made it hard to find a file of interest. Instead, we get the ordering we want by adding an explicit list of tests to run to the notmuch-test script.
2010-09-18 00:58:39 +02:00
for test in $TESTS; do
test: make all tests terminable with Ctrl-c Some tests don't break when HUP signal is sent tho those (by pressing ctrl-c on the terminal). Therefore, the top-level test script catches the HUP and sends TERM signal to the started test script.
2011-11-08 17:02:25 +01:00
$TEST_TIMEOUT_CMD ./$test "$@" &
wait $!
test: Abort driver if a test script aborts Previously, if a test script aborted (e.g., because it passed too few arguments to a test function), the test driver loop would simply continue on to the next test script and the final results would declare that everything passed (except that the test count would look suspiciously low, but maybe you just misremembered how many tests there were). Now, if a test script exits with a non-zero status and did not produce a final results file, we propagate that failure out of the driver loop immediately. To keep this simple, this patch removes the PID from the test-results file name. This PID was inherited from the git test system and seems unnecessary, since the file name already includes the name of the test script and the test-results directory is created anew for each run.
2012-11-28 05:13:16 +01:00
# If the test failed without producing results, then it aborted,
# so we should abort, too.
RES=$?
if [[ $RES != 0 && ! -e "test-results/${test%.sh}" ]]; then
exit $RES
fi
test: Fix test suite to integrate with our non-recursive Makefile system. This avoids "make test" emitting messages from three (3!) recursive invocations of make. We change the invocations of the tests themselves to occur directly from the shell script rather than having the shell script invoke make again and using wildcards in the Makefile.
2010-09-17 21:10:54 +02:00
done
test: make all tests terminable with Ctrl-c Some tests don't break when HUP signal is sent tho those (by pressing ctrl-c on the terminal). Therefore, the top-level test script catches the HUP and sends TERM signal to the started test script.
2011-11-08 17:02:25 +01:00
trap - HUP INT TERM
test: Fix test suite to integrate with our non-recursive Makefile system. This avoids "make test" emitting messages from three (3!) recursive invocations of make. We change the invocations of the tests themselves to occur directly from the shell script rather than having the shell script invoke make again and using wildcards in the Makefile.
2010-09-17 21:10:54 +02:00
# Report results
test: print empty line at the beginning of test script, not at end In preparation for quiet mode print empty line before writing the test description. This is done now in function designed for it -- it will also be called when test fails.
2013-11-25 18:08:18 +01:00
echo
test: Rename all tests to get rid of the ugly numbers in file names. The numbers were meaningless, and they made it hard to find a file of interest. Instead, we get the ordering we want by adding an explicit list of tests to run to the notmuch-test script.
2010-09-18 00:58:39 +02:00
./aggregate-results.sh test-results/*
test: exit with nonzero value when not all tests completed successfully If any of the tests in our test system is not passing the execution of the test suite completes with nonzero exit value. It is better to rely on the exit value of the test system instead of some arbitrary strings in test output (or use both).
2013-09-08 17:53:30 +02:00
ev=$?
test: Fix test suite to integrate with our non-recursive Makefile system. This avoids "make test" emitting messages from three (3!) recursive invocations of make. We change the invocations of the tests themselves to occur directly from the shell script rather than having the shell script invoke make again and using wildcards in the Makefile.
2010-09-17 21:10:54 +02:00
# Clean up
test: force deletion of test remnants This keeps the test from failing if only a subset of the remnants were available for deletion, because e.g. only a subset of the tests were run.
2011-05-07 21:10:04 +02:00
rm -rf test-results corpus.mail
test: exit with nonzero value when not all tests completed successfully If any of the tests in our test system is not passing the execution of the test suite completes with nonzero exit value. It is better to rely on the exit value of the test system instead of some arbitrary strings in test output (or use both).
2013-09-08 17:53:30 +02:00
exit $ev
Reference in a new issue Copy permalink