Previously, we cleaned the downloaded performance corpus and the
cached indexes on 'make clean'. This seems heavy-handed, since these
take a long time to download, unpack, and index. They also aren't
make targets to begin with. Move cleaning these to 'make distclean'.
This isn't exactly the right meaning of "distclean", but it's closer.
Support for out-of-tree builds was added in
commit 3e4a9d60a9
Author: Carl Worth <cworth@cworth.org>
Date: Wed Mar 9 15:02:42 2011 -0800
build: Add support for non-source-directory builds.
and broken in
commit 7beeb8c88a
Author: David Bremner <bremner@debian.org>
Date: Sat Nov 17 12:28:15 2012 -0400
test: initial performance testing infrastructure
Fix the build breakage.
Out-of-tree 'make test' has been broken since earlier than the above,
and remains broken, as does out-of-tree perf test, but at least the
build now works.
Somehow this file was not added to the patch set which split the tests
into time and memory tests. Take advantage of the the "new" way of
running tests to avoid listing the explicitly.
In id:87vcc2q5n2.fsf@nikula.org, Jani points out a memory leak in the
current version of the sup restore code. Among other things, this test
is intended to verify a fix for that leak.
The idea is run some code under valgrind --leak-check=full and report
a summary, leaving the user to peruse the log file if they want.
We go to some lengths to preserve the log files from accidental
overwriting; the full corpus takes about 3 hours to run under valgrind
on my machine.
The naming of the log directories may be slightly controversial; in
the unlikely event of two runs in less than a second, the log will be
overwritten. A previous version with mktemp+timestamp was dismissed as
overkill; just mktemp alone does not sort nicely.
One new test is included, to check notmuch new for memory leaks.
This is almost entirely renaming files, except for updating a few
references to those file names, and changing the makefile target.
A new set of memory tests will be run separately because they take
much longer.
This makes the tag set a bit less trivial.
Note that if you use the small corpus, this is not so interesting (and
is also a bit noisy) since the messages will not be found. In the
future this could be checked for.
Conflicts:
performance-test/01-dump-restore
Austin suggested a while ago that the corpus size be printed in the
header. In the end it seems the corpus will be fixed per test script,
so this suggestion indeed makes sense.
The tabbing was wrapping on my usual 80 column terminal, so I joined
the input and output columns together.
Unlike in the correctness tests, the most common cause of non-zero
return seems to be the user interrupting, so killing the run seems
like the friendly thing to do.
This is not near as fancy as as the unit tests, on the theory that
the code should typically be crashing when performance tuning.
Nonetheless, there is plenty of room for improvement. Several more of
the pieces of the test infrastructure (e.g. the option parsing) could
be factored out into test/test-lib-common.sh
