debian/notmuch
1
0
Fork 0
mirror of https://git.notmuchmail.org/git/notmuch synced 2024-11-22 10:58:10 +01:00
Code Issues Projects Releases Packages Wiki Activity Actions
notmuch/performance-test/README

105 lines
3.4 KiB
Text
Raw Normal View History

perf-test: initial version of memory test infrastructure. The idea is run some code under valgrind --leak-check=full and report a summary, leaving the user to peruse the log file if they want. We go to some lengths to preserve the log files from accidental overwriting; the full corpus takes about 3 hours to run under valgrind on my machine. The naming of the log directories may be slightly controversial; in the unlikely event of two runs in less than a second, the log will be overwritten. A previous version with mktemp+timestamp was dismissed as overkill; just mktemp alone does not sort nicely. One new test is included, to check notmuch new for memory leaks.
2012-12-16 13:33:17 +01:00
Performance Tests
-----------------
This directory contains two kinds of performance tests: time tests,
and memory tests. The former use gnu time, and the latter use
valgrind.
test: initial performance testing infrastructure This is not near as fancy as as the unit tests, on the theory that the code should typically be crashing when performance tuning. Nonetheless, there is plenty of room for improvement. Several more of the pieces of the test infrastructure (e.g. the option parsing) could be factored out into test/test-lib-common.sh
2012-11-17 17:28:15 +01:00
Pre-requisites
--------------
In addition to having notmuch, you need:
- gpg
- gnu tar
perf-test: initial version of memory test infrastructure. The idea is run some code under valgrind --leak-check=full and report a summary, leaving the user to peruse the log file if they want. We go to some lengths to preserve the log files from accidental overwriting; the full corpus takes about 3 hours to run under valgrind on my machine. The naming of the log directories may be slightly controversial; in the unlikely event of two runs in less than a second, the log will be overwritten. A previous version with mktemp+timestamp was dismissed as overkill; just mktemp alone does not sort nicely. One new test is included, to check notmuch new for memory leaks.
2012-12-16 13:33:17 +01:00
- gnu time (for the time tests)
test: initial performance testing infrastructure This is not near as fancy as as the unit tests, on the theory that the code should typically be crashing when performance tuning. Nonetheless, there is plenty of room for improvement. Several more of the pieces of the test infrastructure (e.g. the option parsing) could be factored out into test/test-lib-common.sh
2012-11-17 17:28:15 +01:00
- xz. Some speedup can be gotten by installing "pixz", but this is
probably only worthwhile if you are debugging the tests.
perf-test: initial version of memory test infrastructure. The idea is run some code under valgrind --leak-check=full and report a summary, leaving the user to peruse the log file if they want. We go to some lengths to preserve the log files from accidental overwriting; the full corpus takes about 3 hours to run under valgrind on my machine. The naming of the log directories may be slightly controversial; in the unlikely event of two runs in less than a second, the log will be overwritten. A previous version with mktemp+timestamp was dismissed as overkill; just mktemp alone does not sort nicely. One new test is included, to check notmuch new for memory leaks.
2012-12-16 13:33:17 +01:00
- valgrind (for the memory tests)
perf-test: add option to run perf Although the generation of perf data is not as slow as valgrind, it seems simplest to re-use the machinery already there to save the logs in a timestamped subdirectory.
2021-04-25 17:23:52 +02:00
- perf (optional, for more fine-grained timing)
test: initial performance testing infrastructure This is not near as fancy as as the unit tests, on the theory that the code should typically be crashing when performance tuning. Nonetheless, there is plenty of room for improvement. Several more of the pieces of the test infrastructure (e.g. the option parsing) could be factored out into test/test-lib-common.sh
2012-11-17 17:28:15 +01:00
Getting set up to run tests:
----------------------------
performance-test/README: update instructions for fetching corpus. It turns out not everybody on the planet has my gpg key.
2012-11-28 22:27:00 +01:00
First, you need to get the corpus. If you don't already have the gpg
key for David Bremner, run
test: initial performance testing infrastructure This is not near as fancy as as the unit tests, on the theory that the code should typically be crashing when performance tuning. Nonetheless, there is plenty of room for improvement. Several more of the pieces of the test infrastructure (e.g. the option parsing) could be factored out into test/test-lib-common.sh
2012-11-17 17:28:15 +01:00
perf-test: update corpus signature In the decade (!) since this corpus was last updated, the keyserver network is essentially dead, and I have migrated gpg keys. Bump the version number as a clean way of switching signatures. Also update the instructions to suggest using "--locate-external-key" to download the public key. By default this uses WKD, which is now supported for my UID.
2023-05-31 13:19:18 +02:00
% gpg --locate-external-key 'david@tethera.net'
performance-test/README: update instructions for fetching corpus. It turns out not everybody on the planet has my gpg key.
2012-11-28 22:27:00 +01:00
This should get you a key with fingerprint
perf-test: update corpus signature In the decade (!) since this corpus was last updated, the keyserver network is essentially dead, and I have migrated gpg keys. Bump the version number as a clean way of switching signatures. Also update the instructions to suggest using "--locate-external-key" to download the public key. By default this uses WKD, which is now supported for my UID.
2023-05-31 13:19:18 +02:00
7A18 807F 100A 4570 C596 8420 7E4E 65C8 720B 706B
performance-test/README: update instructions for fetching corpus. It turns out not everybody on the planet has my gpg key.
2012-11-28 22:27:00 +01:00
(the last 8 digits are printed as the "key id").
To fetch the actual corpus it should work to run
test: initial performance testing infrastructure This is not near as fancy as as the unit tests, on the theory that the code should typically be crashing when performance tuning. Nonetheless, there is plenty of room for improvement. Several more of the pieces of the test infrastructure (e.g. the option parsing) could be factored out into test/test-lib-common.sh
2012-11-17 17:28:15 +01:00
% make download-corpus
In case that fails or is too slow, check
Use https instead of http where possible Many of the external links found in the notmuch source can be resolved using https instead of http. This changeset addresses as many as i could find, without touching the e-mail corpus or expected outputs found in tests.
2016-06-02 18:26:14 +02:00
https://notmuchmail.org/corpus
test: initial performance testing infrastructure This is not near as fancy as as the unit tests, on the theory that the code should typically be crashing when performance tuning. Nonetheless, there is plenty of room for improvement. Several more of the pieces of the test infrastructure (e.g. the option parsing) could be factored out into test/test-lib-common.sh
2012-11-17 17:28:15 +01:00
for a list of mirrors.
Running tests
-------------
perf-test: initial version of memory test infrastructure. The idea is run some code under valgrind --leak-check=full and report a summary, leaving the user to peruse the log file if they want. We go to some lengths to preserve the log files from accidental overwriting; the full corpus takes about 3 hours to run under valgrind on my machine. The naming of the log directories may be slightly controversial; in the unlikely event of two runs in less than a second, the log will be overwritten. A previous version with mktemp+timestamp was dismissed as overkill; just mktemp alone does not sort nicely. One new test is included, to check notmuch new for memory leaks.
2012-12-16 13:33:17 +01:00
The easiest way to run performance tests is to say "make perf-test".
This will run all time and memory tests. Be aware that the memory
tests are quite time consuming when run on the full corpus, and that
depending on your interests it may be more sensible to run "make
time-test" or "make memory-test". You can also invoke one of the
scripts notmuch-time-test or notmuch-memory-test or run a more
specific subset of tests by simply invoking one of the executable
scripts in this directory, (such as ./T00-new). Each test script
supports the following arguments
perf-test: update README Describe new argument parsing and mention cache handling routines.
2012-12-05 13:48:50 +01:00
--small / --medium / --large Choose corpus size.
--debug Enable debugging. In particular don't delete
perf-test: add option to run perf Although the generation of perf data is not as slow as valgrind, it seems simplest to re-use the machinery already there to save the logs in a timestamped subdirectory.
2021-04-25 17:23:52 +02:00
temporary directories.
--perf Run perf record in place of /usr/bin/time. Perf output can be
found in a log directory.
--call-graph {fp,lbr,dwarf} Call graph option for perf record. Default is 'lbr'.
test: initial performance testing infrastructure This is not near as fancy as as the unit tests, on the theory that the code should typically be crashing when performance tuning. Nonetheless, there is plenty of room for improvement. Several more of the pieces of the test infrastructure (e.g. the option parsing) could be factored out into test/test-lib-common.sh
2012-11-17 17:28:15 +01:00
perf-test: initial version of memory test infrastructure. The idea is run some code under valgrind --leak-check=full and report a summary, leaving the user to peruse the log file if they want. We go to some lengths to preserve the log files from accidental overwriting; the full corpus takes about 3 hours to run under valgrind on my machine. The naming of the log directories may be slightly controversial; in the unlikely event of two runs in less than a second, the log will be overwritten. A previous version with mktemp+timestamp was dismissed as overkill; just mktemp alone does not sort nicely. One new test is included, to check notmuch new for memory leaks.
2012-12-16 13:33:17 +01:00
When using the make targets, you can pass arguments to all test
scripts by defining the make variable OPTIONS.
perf-test: add option to run perf Although the generation of perf data is not as slow as valgrind, it seems simplest to re-use the machinery already there to save the logs in a timestamped subdirectory.
2021-04-25 17:23:52 +02:00
Log Directory
-------------
The memory tests, and the time tests when option '--perf' is given
save their output in a directory named as follows
log.$test_name-$corpus_size-$timestamp
These directories are removed by "make clean".
test: initial performance testing infrastructure This is not near as fancy as as the unit tests, on the theory that the code should typically be crashing when performance tuning. Nonetheless, there is plenty of room for improvement. Several more of the pieces of the test infrastructure (e.g. the option parsing) could be factored out into test/test-lib-common.sh
2012-11-17 17:28:15 +01:00
Writing tests
-------------
perf-test: initial version of memory test infrastructure. The idea is run some code under valgrind --leak-check=full and report a summary, leaving the user to peruse the log file if they want. We go to some lengths to preserve the log files from accidental overwriting; the full corpus takes about 3 hours to run under valgrind on my machine. The naming of the log directories may be slightly controversial; in the unlikely event of two runs in less than a second, the log will be overwritten. A previous version with mktemp+timestamp was dismissed as overkill; just mktemp alone does not sort nicely. One new test is included, to check notmuch new for memory leaks.
2012-12-16 13:33:17 +01:00
Have a look at "T01-dump-restore" for an example time test and
"M00-new" for an example memory test. In both cases sourcing
"perf-test-lib.sh" is mandatory.
perf-test: update README Describe new argument parsing and mention cache handling routines.
2012-12-05 13:48:50 +01:00
perf-test: initial version of memory test infrastructure. The idea is run some code under valgrind --leak-check=full and report a summary, leaving the user to peruse the log file if they want. We go to some lengths to preserve the log files from accidental overwriting; the full corpus takes about 3 hours to run under valgrind on my machine. The naming of the log directories may be slightly controversial; in the unlikely event of two runs in less than a second, the log will be overwritten. A previous version with mktemp+timestamp was dismissed as overkill; just mktemp alone does not sort nicely. One new test is included, to check notmuch new for memory leaks.
2012-12-16 13:33:17 +01:00
Basics:
- '(time|memory)_start' unpacks the mail corpus and calls notmuch new if it
perf-test: update README Describe new argument parsing and mention cache handling routines.
2012-12-05 13:48:50 +01:00
cannot find a cache of the appropriate corpus.
perf-test: initial version of memory test infrastructure. The idea is run some code under valgrind --leak-check=full and report a summary, leaving the user to peruse the log file if they want. We go to some lengths to preserve the log files from accidental overwriting; the full corpus takes about 3 hours to run under valgrind on my machine. The naming of the log directories may be slightly controversial; in the unlikely event of two runs in less than a second, the log will be overwritten. A previous version with mktemp+timestamp was dismissed as overkill; just mktemp alone does not sort nicely. One new test is included, to check notmuch new for memory leaks.
2012-12-16 13:33:17 +01:00
- '(time|memory)_run' runs the command under time or valgrind. Currently
"memory_run" does not support i/o redirection in the command.
- '(time|memory)_done' does the cleanup; comment it out or pass --debug to the
perf-test: update README Describe new argument parsing and mention cache handling routines.
2012-12-05 13:48:50 +01:00
script to leave the temporary files around.
perf-test: split basic into 00-new, 01-dump-restore, and 02-tag We use the new "time_start" function to restore the database from cache if possible.
2012-12-06 02:34:30 +01:00
perf-test: initial version of memory test infrastructure. The idea is run some code under valgrind --leak-check=full and report a summary, leaving the user to peruse the log file if they want. We go to some lengths to preserve the log files from accidental overwriting; the full corpus takes about 3 hours to run under valgrind on my machine. The naming of the log directories may be slightly controversial; in the unlikely event of two runs in less than a second, the log will be overwritten. A previous version with mktemp+timestamp was dismissed as overkill; just mktemp alone does not sort nicely. One new test is included, to check notmuch new for memory leaks.
2012-12-16 13:33:17 +01:00
Utility functions include
- 'add_email_corpus' unpacks a set of messages and tags
- 'cache_database': makes a snapshot of the current database
- 'uncache_database': forces the next '(time|memory)_start' to rebuild the
database.
perf-test: split basic into 00-new, 01-dump-restore, and 02-tag We use the new "time_start" function to restore the database from cache if possible.
2012-12-06 02:34:30 +01:00
Scripts are run in the order specified in notmuch-perf-test. In the
future this order might be chosen automatically so please follow the
perf-test: initial version of memory test infrastructure. The idea is run some code under valgrind --leak-check=full and report a summary, leaving the user to peruse the log file if they want. We go to some lengths to preserve the log files from accidental overwriting; the full corpus takes about 3 hours to run under valgrind on my machine. The naming of the log directories may be slightly controversial; in the unlikely event of two runs in less than a second, the log will be overwritten. A previous version with mktemp+timestamp was dismissed as overkill; just mktemp alone does not sort nicely. One new test is included, to check notmuch new for memory leaks.
2012-12-16 13:33:17 +01:00
convention of starting the name with 'T' or 'M' followed by two digits
to specify the order.
Reference in a new issue Copy permalink