We put some effort into testing the built copy rather than some
installed copy. On the other hand for people like packagers, testing
the installed copy is also of interest.
When NOTMUCH_TEST_INSTALLED is set to a nonempty value, tests do not
require a built notmuch tree or running configure.
Some of the tests marked as broken when running against installed
notmuch are probably fixable.
test_require_external prereq has to move to test-lib-common.sh, and
the new shell functions print_emacs_header and time_emacs are provided.
The somewhat indirect way of printing the output is to avoid the extra
"" present on string values from emacsclient.
In 8e7fb88237 Jani replaced the use of $(pwd -P) to find the
TEST_DIRECTORY in order to better support out of tree
builds. Unfortunately the performance-tests need a different value for
the variable and were thus broken.
This commit splits out the setting of this variable for the two sets
of tests. Performance tests still don't work out of tree, because
the handling of the downloaded corpus needs to be updated.
backup_database() and restore_database() used to store the backups in
the test specific temporary directory, through the current working
directory being there. Commit 8e7fb88237 ("test: use source and
build paths in test-lib-common.sh") started using a test specific
backup directories under the build tree test directory. This was in
error. Switch back to the old location, but using paths to the
location instead of relying on current working directory.
Reported by Daniel Kahn Gillmor <dkg@fifthhorseman.net>.
The primary motivation here is to fix TMP_DIRECTORY cleanup prior to
running each test when the current working directory is not the test
subdirectory. Tests with failures would leave their TMP_DIRECTORY
directory behind for debugging, and repeated out-of-tree test runs
would have old temp directories. (This lead to e.g. T310-emacs.sh
hanging because emacs would prompt for overwriting files.)
We remove the likely anyway defunct --root test option while at it,
just to be on the safe side when doing 'rm -rf' on the TMP_DIRECTORY.
Make test-lib-common.sh load test-lib-<$PLATFORM>.sh to create
additional shim for platform specifics.
Use test-lib-FREEBSD.sh to call GNU utilities instead of native ones.
- amended by db following Tomi's suggestions
We already use this directory for dtach sockets, so it makes sense to
put gnupg sockets there as well. There doesn't seem to be a clean way
to put a fully functional socket in a different location than
GNUPGHOME.
Previously LD_LIBRARY_PATH was exported (and environment changed)
in the middle of test case execution, when a function setting it
was called.
Previously the old contents of LD_LIBRARY_PATH was lost (if any)
when it was re-set and exported. In some systems the old contents of
LD_LIBRARY_PATH was needed to e.g. locate suitable gmime library.
In scripts that include test-lib-common.sh but not test-lib.sh
the die() implementation needs to be a bit different due to
fd redirection differences. test-lib-common.sh implements die()
only if it was not implemented already.
Many of the external links found in the notmuch source can be resolved
using https instead of http. This changeset addresses as many as i
could find, without touching the e-mail corpus or expected outputs
found in tests.
Files in test directories had only copyright of a single individual,
of which code was adapted here as a base of the test system.
Since then many Notmuch Developers have contributed to the test
system, which is now acknowledged with a constant string in some
of the test files.
The README file in test directory instructed new files contain a
copyright notice, but that has never been done (and it is also not
needed). To simplify things a bit (and lessen confusion) this
instruction is now removed.
As a side enchangement, all of the 3 entries in the whole source
tree cd'ing to `dirname` of "$0" now uses syntax cd "$(dirname "$0")".
This makes these particular lines work when current working directory
is e.g. /c/Program Files/notmuch/test/.
(Probably it would fail elsewhere, though.)
When notmuch sources are at a symlinked path, some tests fail because
one part of the test uses physical path and another uses logical
path (with symlinks). For example the following test output is
produced when the test is started from /home/src/symlink-to-notmuch,
which is a symlink to /home/src/notmuch.
FAIL notmuch-fcc-dirs set to a string
--- T310-emacs.26.OUTPUT 2015-12-29 08:54:29.055878637 +0000
+++ T310-emacs.26.EXPECTED 2015-12-29 08:54:29.055878637 +0000
@@ -1,5 +1,5 @@
From: Notmuch Test Suite <test_suite@notmuchmail.org>
To:
Subject:
-Fcc: /home/src/notmuch/test/tmp.T310-emacs/mail/sent-string
+Fcc: /home/src/symlink-to-notmuch/test/tmp.T310-emacs/mail/sent-string
--text follows this line--
nil
This commit makes all paths in test scripts physical. With it, all
tests pass even when run from a symlinked directory.
The files (test) scripts source (with builtin command `.`) provides
information which the scripts depend, and without the `source` to
succeed allowing script to continue may lead to dangerous situations
(e.g. rm -rf "${undefined_variable}"/*).
At the end of all source (.) lines construct ' || exit 1' was added;
In our case the script script will exit if it cannot find (or read) the
file to be sourced. Additionally script would also exits if the last
command of the sourced file exited nonzero.
When something in tests fails one possibility to test is to run
the test script as `bash -x TXXX-testname.sh`. As stderr (fd 2) was
redirected to separate file during test execution also this set -x
(xtrace) output would also go there.
test-lib.sh saves the stderr to fd 7 from where it can be restored,
and bash has BASH_XTRACEFD variable, which is now given the same value
7, making bash to output all xtrade information (consistently) there.
This lib file used to save fd's 1 & 2 to 6 & 7 (respectively) in
test_begin_subtest(), but as those needs to be set *before* XTRACEFD
variable is set those are now saved at the beginning of the lib (once).
This is safe and simple thing to do.
To make xtrace output more verbose PS4 variable was set to contain the
source file, line number and if execution is in function, that function
name. Setting this variable has no effect when not xtracing.
As it is known that fd 6 is redirected stdout, printing status can now
use that fd, instead of saving stdout to fd 5 and use it.
It turns out there was a reason the old man pages were stored in a man
compatible hierarchy, namely so that we could run man on them before
installing.
Hardcode doc build location into test suite. This isn't ideal, but
let's unbreak the test suite for now.
