Several people have observed that this is surprisingly slow, and we
have a proposal to add tagging into this code path, so we want to make
sure it doesn't imply too much of a performance hit.
In 8e7fb88237 Jani replaced the use of $(pwd -P) to find the
TEST_DIRECTORY in order to better support out of tree
builds. Unfortunately the performance-tests need a different value for
the variable and were thus broken.
This commit splits out the setting of this variable for the two sets
of tests. Performance tests still don't work out of tree, because
the handling of the downloaded corpus needs to be updated.
A leading / in paths in a .gitignore file matches the beginning of the
path, meaning that for patterns without slashes, git will match files
only in the current directory as opposed to in any subdirectory.
Prefix relevant paths with / in .gitignore files, to prevent
accidentally ignoring files in subdirectories and possibly slightly
improve the performance of "git status".
Many of the external links found in the notmuch source can be resolved
using https instead of http. This changeset addresses as many as i
could find, without touching the e-mail corpus or expected outputs
found in tests.
Files in test directories had only copyright of a single individual,
of which code was adapted here as a base of the test system.
Since then many Notmuch Developers have contributed to the test
system, which is now acknowledged with a constant string in some
of the test files.
The README file in test directory instructed new files contain a
copyright notice, but that has never been done (and it is also not
needed). To simplify things a bit (and lessen confusion) this
instruction is now removed.
As a side enchangement, all of the 3 entries in the whole source
tree cd'ing to `dirname` of "$0" now uses syntax cd "$(dirname "$0")".
This makes these particular lines work when current working directory
is e.g. /c/Program Files/notmuch/test/.
(Probably it would fail elsewhere, though.)
The files (test) scripts source (with builtin command `.`) provides
information which the scripts depend, and without the `source` to
succeed allowing script to continue may lead to dangerous situations
(e.g. rm -rf "${undefined_variable}"/*).
At the end of all source (.) lines construct ' || exit 1' was added;
In our case the script script will exit if it cannot find (or read) the
file to be sourced. Additionally script would also exits if the last
command of the sourced file exited nonzero.
It turns out to be inconvenient to delete the downloaded datafiles with
distclean, so I propose a new target which does that instead.
The closest conventional target is 'maintainer-clean'; the difference
here is that having the original source tarball is not enough to
reconstruct these files.
It seems that between version 1.26 and 1.27 of gnu tar, directories to
be extracted read with --files-from are no longer recursively extacted.
This patch puts them on the command line instead.
Previously, we cleaned the downloaded performance corpus and the
cached indexes on 'make clean'. This seems heavy-handed, since these
take a long time to download, unpack, and index. They also aren't
make targets to begin with. Move cleaning these to 'make distclean'.
This isn't exactly the right meaning of "distclean", but it's closer.
Support for out-of-tree builds was added in
commit 3e4a9d60a9
Author: Carl Worth <cworth@cworth.org>
Date: Wed Mar 9 15:02:42 2011 -0800
build: Add support for non-source-directory builds.
and broken in
commit 7beeb8c88a
Author: David Bremner <bremner@debian.org>
Date: Sat Nov 17 12:28:15 2012 -0400
test: initial performance testing infrastructure
Fix the build breakage.
Out-of-tree 'make test' has been broken since earlier than the above,
and remains broken, as does out-of-tree perf test, but at least the
build now works.
Somehow this file was not added to the patch set which split the tests
into time and memory tests. Take advantage of the the "new" way of
running tests to avoid listing the explicitly.
In id:87vcc2q5n2.fsf@nikula.org, Jani points out a memory leak in the
current version of the sup restore code. Among other things, this test
is intended to verify a fix for that leak.
The idea is run some code under valgrind --leak-check=full and report
a summary, leaving the user to peruse the log file if they want.
We go to some lengths to preserve the log files from accidental
overwriting; the full corpus takes about 3 hours to run under valgrind
on my machine.
The naming of the log directories may be slightly controversial; in
the unlikely event of two runs in less than a second, the log will be
overwritten. A previous version with mktemp+timestamp was dismissed as
overkill; just mktemp alone does not sort nicely.
One new test is included, to check notmuch new for memory leaks.
This is almost entirely renaming files, except for updating a few
references to those file names, and changing the makefile target.
A new set of memory tests will be run separately because they take
much longer.
This makes the tag set a bit less trivial.
Note that if you use the small corpus, this is not so interesting (and
is also a bit noisy) since the messages will not be found. In the
future this could be checked for.
Conflicts:
performance-test/01-dump-restore
Austin suggested a while ago that the corpus size be printed in the
header. In the end it seems the corpus will be fixed per test script,
so this suggestion indeed makes sense.
The tabbing was wrapping on my usual 80 column terminal, so I joined
the input and output columns together.
Unlike in the correctness tests, the most common cause of non-zero
return seems to be the user interrupting, so killing the run seems
like the friendly thing to do.
This is not near as fancy as as the unit tests, on the theory that
the code should typically be crashing when performance tuning.
Nonetheless, there is plenty of room for improvement. Several more of
the pieces of the test infrastructure (e.g. the option parsing) could
be factored out into test/test-lib-common.sh
