Before the change, every Emacs test ran in a separate Emacs
instance. Starting Emacs many times wastes considerable time and
it gets worse as the test suite grows. The patch solves this by
using a single Emacs server and emacsclient(1) to run multiple
tests. Emacs server is started on the first test_emacs call and
stopped when test_done is called. We take care not to leave
orphan Emacs processes behind when test is terminated by whatever
reason: Emacs server runs a watchdog that periodically checks
that the test is still running.
Some tests need to provide user input. Before the change, this
was done using echo(1) to Emacs stdin. This no longer works and
instead `standard-input' variable is set accordingly to make
`read' return the appropriate string.
Without this, mail messages delivered by emacs_deliver_message might
not be seen by the next invocation of "notmuch new", (which can lead
to test-suite failures if emacs_deliver_message is fast enough).
Change add_email_corpus, emacs_deliver_message and tests to use
$TEST_DIRECTORY instead of '..'.
This improves the behavior of the usage of --root=<dir>, as the
assumption of what '..' means will usually be incorrect.
Document -root option in README and update valgrind to work with
-root.
Instead of generating auxiliary run_emacs script every time
test_emacs is run, do it once in the beginning of the test.
Also, use absolute paths in the script to make it more robust.
Using `setq' for setting variables in Emacs tests affect other
tests that may run in the same Emacs environment. Currently it
works because each test is run in a separate Emacs instance. But
in the future multiple tests will run in a single Emacs instance.
The patch changes all variables to use `let', so the scope of the
change is limited to a single test.
Few Emacs tests used sed(1) to remove unexpected output in the
beginning to avoid getting confused by messages such as "Parsing
/home/cworth/.mailrc... done". This is no longer needed since
tests are run in a temporary home directory instead of the user's
one. So remove these sed(1) calls.
Before the change, the common Emacs test scheme was to print
buffer content to stdout and redirect it to a file or capture it
in a shell variable. This does not work if we switch to using
emacsclient(1) for running the tests, because you can not print
to the stdout in this case. (Actually, you can print to stdout
from Emacs server, but you can not capture the output on
emacsclient(1)).
The patch introduces new Emacs test auxiliary functions:
`test-output' and `test-visible-output'. These functions are
used to save buffer content to a file directly from Emacs. For
most tests the changes are trivial, because Emacs stdout output
was redirected to a file anyway. But some tests captured the
output in a shell variable and compare it with the expected
output using test_expect_equal. These tests are changed to use
files and test_expect_equal_file instead.
Note: even if we do not switch Emacs tests to emacsclient(1), the
patch makes tests cleaner and is an improvement.
Most test_emacs calls have long arguments that consist of many
expressions. Putting them on a single line makes it hard to read
and produces poor diff when they are changed. The patch puts
every expression in test_emacs calls on a separate line.
Few Emacs tests had test_expect_equal_file arguments in the wrong
order: the first argument should be the test output and the
second one should be the expected.
Update the test mail corpus to have two files with the same content to
expose the bug where a single message with multiple filenames only
reports a single filename.
Update expected results for search --output=files to match new
behavior for multiple files corresponding to a single message
Signed-off-by: Mark Anderson <ma.skies@gmail.com>
Various typo fixes in some test-case data.
Signed-off-by: Pieter Praet <pieter@praet.org>
Edited-by: Carl Worth <cworth@cworth.org> Restricted to just
test-case data.
Various typo fixes in documentation within the code that can be made
available to the user, (emacs function help strings, "notmuch help"
output, notmuch man page, etc.).
Signed-off-by: Pieter Praet <pieter@praet.org>
Edited-by: Carl Worth <cworth@cworth.org> Restricted to just
documentation and fixed fix of "comman" to "common" rather than
"command".
Various typo fixes in comments within the source code.
Signed-off-by: Pieter Praet <pieter@praet.org>
Edited-by: Carl Worth <cworth@cworth.org> Restricted to just
source-code comments, (and fixed fix of "descriptios" to "descriptors"
rather than "descriptions").
Various typo fixes in comments within the Makefile and other build scripts.
Signed-off-by: Pieter Praet <pieter@praet.org>
Edited-by: Carl Worth <cworth@cworth.org> Restricted to just build files.
Various typo fixes in auxiliary text files included with the source,
(README, TODO, etc.).
Signed-off-by: Pieter Praet <pieter@praet.org>
Edited-by: Carl Worth <cworth@cworth.org> Restricted to just text files.
We exercise each of the documented values (nil, a string, and a
list). For the list, we test matching a specific entry, matching a
catch-all regular expression, and no match at all (in which case there
is no FCC set).
The worry here is that a binary linking with libnotmuch might lose
access to Xapian::Error symbols because libnotmuch hides them.
We are careful here to create ./fakedb/.notmuch in order to trigger a
Xapian exception, and not just a missing file check.
Thanks to jrollins and amddragon for suggestions.
(cherry picked from commit 66f37f5f6864a988f94ddb893e3a176af57f6c8e)
The main() function should be written as just another function with a
return value. This allows for more reliable code reuse. Imagine that
main() grows too large and needs to be factored into multiple
functions. At that point, exit() is probably the wrong thing, yet can
also be hard to notice as it's in less-frequently-tested exceptional
cases.
Each top level test (basic, corpus, etc...) is run with a fixed
timeout of 2 minutes.
The goal here is to treat a hung test as a failure. The emacs test for
sending mail is known to be problematic on the Debian
autobuilders. This is both a bandaid fix for that, and a sensible long
term feature.
(cherry picked from commit 5f99c80e02736c90495558d9b88008a768876b29)
The intent was always to make these Received headers span multiple
lines. But the escapes were causing the shell to ignore the newlines,
so that the result instead was long Received headers on a single line
each.
Fixing the intent here doesn't actually change the test-suite results
at all.
This is much more realistic, as most messages in the wild will have multiple
Received headers. Also, this demonstrates a current bug in the Received
header parsing, (multiple Received headers are not properly concatenated
depending on the order in which headers are parsed in a message).
This tests the recently-added detection/hiding of top-posted quotations and
in the testing, it takes advantage of the less-recently-added
visible-buffer-string function for emitting only the visible text
from a buffer.
Again, this is a much cleaner and more thorough test, and in fact
exposes a bug in the format=text output, that will be fixed the next
commit. Because of this, some of the multipart tests currently fail.
This is a much cleaner way to do the emacs tests, since we're actually
comparing output against existing files with expected output. We also
won't miss any trailing newlines this way.
And speaking of which, one of the expected output files was actually
missing a trailing blank line that was actually in one of the original
messages, so this was fixed.
This feature was recently added, so it of course needs a test now.
Signed-off-by: Jameson Graef Rollins <jrollins@finestructure.net>
Edited-by: Carl Worth <cworth@cworth.org> Fixed test to use
notmuch_search_sanitize in order to be robust against unpredictable
thread ID numbers, (due to unpredictable order in which the filesystem
presents files).
In the master branch in test/emacs two tests access the build users home
directory, so does emacs_deliver_message in the crypto branch.
The tests should not touch the build user's home directory. The patch
creates a directory in the temporary test directory and sets home
accordingly.
In case of a non-existent home directory, the tests are failing without
this patch.
Signed-off-by: Jameson Graef Rollins <jrollins@finestructure.net>
This test doesn't have anything to do with json, and has everything to
do with testing search capability, so I'm not sure why it was in the
wrong place.
The "Search for non-existent message prints nothing" test fits better
with the existing tests in search-output, so move it there. Also add a
similar test for the --format=json case.
These tests also use the new test_expect_equal_file function, (to ensure
that the presence of a trailing newline is correctly tested).
These test now properly test for the presence of a newline at the end
of all output. Right now some of these test will fail because the
search output is currently broken to *not* produce proper newlines in
some cases.
Since commit 2f8871df6e notmuch has been
using a function (show_part_content) originally written only for text
parts to save all MIME parts. The problem with this is that this
function converts CRLF pairs to LF only and optionally converts to
UTF-8 encoding. These two conversions have the potential to corrupt
binary data when passed through the function.
This test demonstrates that corruption, and so fails currently, until
we fix the bug.
Not that it affects the correctness of the test, but it's nice to use
proper spelling. This kind of change could invalidate a signature on the
test message, but I think that would have happened previously when the
HTML part was added in the first place.
Use .gz filenames for saved attachments in the tests to check
that Emacs does not re-compress the file.
Use test_expect_equal_file instead of test_expect_equal to avoid
binary output on the console.
Before the change, test_expect_equal_file moved files it compared
in case of failure. The patch changes it to copy the files
instead. This allows testing non-temporary files which are
stored in git.
Note: the change should not result in new temporary files left
after the tests. Test_expect_equal_file used to move files only
on failure, so callers had to cleanup them anyway.
The primary goal here is to keep the decrypted output as similarly
structured as undecrypted output as possible. Now, when decrypting
parts, only the original encrypted part is replaced by the it's
decrypted content. If this part isn't itself a multipart, then all
part numbering should remain consistent during decryption.
The only draw back here is that the useless application/pgp-encrypted
sub-part of the multipart/encrypted part is also emitted. But this
part can be easily ignored by clients.
Some folks have complained about the part renumbering that occurs when
the entire multipart/signed part is replaced with the part contents
after verification. This is primarily because it incurs an additional
computational cost to retrieve individual parts, since verification
has to be performed again to ensure that part numbering is consistent.
This patch simply leaves the full multipart/signed part as is.
The emacs crypto test is also updated to reflect this change.
This patch adds the tag "signed" to messages with any multipart/signed
parts, and the tag "encrypted" to messages with any
multipart/encrypted parts. This only occurs when messages are indexed
during notmuch new, so a database rebuild is required to have old
messages tagged.
This adds a new "crypto" test script to the test suite to test
PGP/MIME signature verification and message decryption. Included here
is a test GNUPGHOME with a test secret key (passwordless), and test
for:
* signing/verification
* signing/verification with full owner trust
* verification with signer key unavailable
* encryption/decryption
* decryption failure with missing key
* encryption/decryption + signing/verfifying
* reply to encrypted message
* verification of signature from revoked key
These tests are not expected to pass now, but will as crypto
functionality is included.
We need to be able to test for the presence of a newline at the end of
output. There's no good way to capture trailing newlines in bash, so
redirecting output to a file is the next best thing. This new
function should be used when testing for output that is expected to
have trailing newlines.
The next commit will demonstrate the use of this.
The patch replaces all (message (buffer-string)) calls in emacs
tests with (princ (buffer-string)). This avoids accidentally
interpreting '%' as format specifiers and makes code simpler
because we do not need to capture stderr.
Also, the patch works around an Emacs (23.3+1-1 on current Debian
Unstable) segfault in "Ensure that emacs doesn't drop results"
test. Note: the segfault does not happen on every test run.
Though, it seems to be consistently reproducible if the test uses
300 messages instead of 30. Hopefully, it is the crash described
in Emacs bug #8545 [1] which is already fixed.
[1] http://debbugs.gnu.org/cgi/bugreport.cgi?bug=8545
