When NOTMUCH_TEST_QUIET environment variable is set to non-null value
messages when new test script starts and when test PASSes are disabled.
This eases picking the cases when tests FAIL (as those are still printed).
And require that if TEST_EMACS is specified, so is TEST_EMACSCLIENT.
Previously, the test framework always used "emacsclient", even if the
Emacs in use was overridden by TEST_EMACS. This causes problems if
both Emacs 23 and Emacs 24 are installed, the Emacs 23 emacsclient is
the system default, but TEST_EMACS is set to emacs24. Specifically,
with an Emacs 24 server and an Emacs 23 client, emacs tests that run
very quickly may produce no output from emacsclient, causing the test
to fail.
The Emacs server uses a very simple line-oriented protocol in which
the client sends a request to evaluate an expression and the server
sends a request to print the result of evaluation. Prior to Emacs bzr
commit 107565 on March 11th, 2012 (released in Emacs 24.1), if
multiple commands were sent to the emacsclient between when it sent
the evaluation command and when it entered its receive loop, it would
only process the first response command, ignoring the rest of the
received buffer. This wasn't a problem with the Emacs 23 server
because it sent only the command to print the evaluation result.
However, the Emacs 24 server first sends an unprompted command
specifying the PID of the Emacs server, then processes the evaluation
request, then sends the command to print the result. If the
evaluation is fast enough, it can send both of these commands before
emacsclient enters the receive loop. Hence, if an Emacs 24 server is
used with an Emacs 23 emacsclient, it may miss the response printing
command, ultimately causing intermittent notmuch test failures.
Before the change, test_expect_equal_file() function treated the first
argument as "actual output file" and the second argument as "expected
output file". When the test fails, the files are copied for later
inspection. The first files was copied to "$testname.output" and the
second file to "$testname.expected". The argument order for
test_expect_equal_file() is often wrong which results in confusing
diff output and incorrectly named files.
The patch solves the issue by changing test_expect_equal_file() to
treat arguments just as two files, without any special properties
(like "actual" and "expected"). The file names for copying is now
based on the given file name: "$testname.$file1" and
"$testname.$file2". E.g. if test_expect_equal_file() is called with
"OUTPUT" and "EXPECTED", the copied files can be named
"emacs.1.OUTPUT" and "emacs.1.EXPECTED".
The down side of this approach is that diff argument order depends on
test_expect_equal_file() argument order. So sometimes we get diff
from expected to actual results, and sometimes the other way around.
But the files are always named correctly.
Add a new test function to allow simpler testing of emacs
functionality.
`test_emacs_expect_t' takes one argument - a lisp expression to
evaluate. The test passes if the expression returns `t', otherwise it
fails and the output is reported to the tester.
Add an explicit note to the README explaining what programs are
necessary and the perhaps-surprising behavior of skipping tests if
they aren't present.
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
As we start to pay more attention to emacs24, it helps to be able to
select a different version of emacs to run the tests with to verify
version specific bugs.
A separate variable TEST_EMACS is needed to avoid being overwritten by the
make variable EMACS in Makefile.config
For what it's worth, the value of emacs is chosen at the time
tmp.emacs/run_emacs is created, so is fixed for all subtests.
The patch adds two auxiliary functions and a variable:
notmuch_counter_reset
$notmuch_counter_command
notmuch_counter_value
They allow to count how many times notmuch binary is called.
notmuch_counter_reset() function generates a script that counts how
many times it is called and resets the counter to zero. The function
sets $notmuch_counter_command variable to the path to the generated
script that should be called instead of notmuch to do the counting.
The notmuch_counter_value() function returns the current counter
value.
There is existing support for broken tests. But it is not convenient
to use. The primary issue is that we have to maintain a set of
test_expect_*_failure functions which are equivalent to the normal
test_expect_* counterparts except for what functions are called for
result reporting. The patch adds test_subtest_known_broken function
which marks a subset as broken, making the normal test_expect_*
functions behave as test_expect_*_failure. All test_expect_*_failure
functions are removed. Test_known_broken_failure_ is changed to
format details the same way as test_failure_ does.
Another benefit of this change is that the diff when a broken test is
fixed would be small and nice.
Documentation is updated accordingly.
Update test_emacs documentation in test/README according to the latest
changes in emacs tests. Move the note regarding setting variables
from test/emacs to test/README.
Change add_email_corpus, emacs_deliver_message and tests to use
$TEST_DIRECTORY instead of '..'.
This improves the behavior of the usage of --root=<dir>, as the
assumption of what '..' means will usually be incorrect.
Document -root option in README and update valgrind to work with
-root.
Various typo fixes in auxiliary text files included with the source,
(README, TODO, etc.).
Signed-off-by: Pieter Praet <pieter@praet.org>
Edited-by: Carl Worth <cworth@cworth.org> Restricted to just text files.
Change #!/bin/bash at start of tests to "#!/usr/bin/env bash". That way
systems running on bash < 4 can prepend bash >= 4 to path before
running the tests.
This should be quite handy for doing automated testing of the
emacs-based functionality in notmuch. This function invokes emacs with
the necessary command-line arguments, (to run in batch mode with no
local initialization, to load the notmuch code from the source
directory, and to ensure an 80-column width).
While adding the documentation here for add_email_corpus I noticed
that the other email-adding functions in test-lib.sh were not yet
documented here, so add all of that documentation.
In order for --valgrind to be useful, we drop noisy additional output of
all of the commands being executed in verbose mode. This makes --verbose
alone quite useless, so we don't document it any more.
Also, add a zlib valgrind suppression that was showing up frequently in the
test suite.
This file was obviously describing the git test suite previously, and
would have been very hard to understand in the context of the notmuch
test suite. HOpefully it's easier to follow now.
This removes Git specific things from the test-lib.sh and adds helper
functions for notmuch taken from Carl's notmuch-test script. README is
also slightly modified to reflect the current state.
Signed-off-by: Michal Sojka <sojkam1@fel.cvut.cz>
Git uses a simple and yet powerful test framework, written in shell.
The framework is easy to use for both users and developers so I think
it would help if it is used in notmuch as well.
This is a copy of Git's test framework from commit
b6b0afdc30e066788592ca07c9a6c6936c68cc11 in git repository.
Signed-off-by: Michal Sojka <sojkam1@fel.cvut.cz>
