Update test_emacs documentation in test/README according to the latest
changes in emacs tests. Move the note regarding setting variables
from test/emacs to test/README.
eb4cf465 introduces changes which weren't part of the submitted
patch (id:"87liwlip2j.fsf@gmail.com"), presumably made during
resolving merge conflicts.
The first one causes the title of a test to be printed a second time,
in place of the correct title:
PASS Message with .. in Message-Id:
PASS Message with .. in Message-Id:
instead of:
PASS Message with .. in Message-Id:
PASS Sending a message via (fake) SMTP
The second one is simply the insertion of a line break, so no harm there.
This commit reverts both changes, as they were clearly accidental.
Signed-off-by: Pieter Praet <pieter@praet.org>
Before the change, every Emacs test ran in a separate Emacs
instance. Starting Emacs many times wastes considerable time and
it gets worse as the test suite grows. The patch solves this by
using a single Emacs server and emacsclient(1) to run multiple
tests. Emacs server is started on the first test_emacs call and
stopped when test_done is called. We take care not to leave
orphan Emacs processes behind when test is terminated by whatever
reason: Emacs server runs a watchdog that periodically checks
that the test is still running.
Some tests need to provide user input. Before the change, this
was done using echo(1) to Emacs stdin. This no longer works and
instead `standard-input' variable is set accordingly to make
`read' return the appropriate string.
Change add_email_corpus, emacs_deliver_message and tests to use
$TEST_DIRECTORY instead of '..'.
This improves the behavior of the usage of --root=<dir>, as the
assumption of what '..' means will usually be incorrect.
Document -root option in README and update valgrind to work with
-root.
Using `setq' for setting variables in Emacs tests affect other
tests that may run in the same Emacs environment. Currently it
works because each test is run in a separate Emacs instance. But
in the future multiple tests will run in a single Emacs instance.
The patch changes all variables to use `let', so the scope of the
change is limited to a single test.
Few Emacs tests used sed(1) to remove unexpected output in the
beginning to avoid getting confused by messages such as "Parsing
/home/cworth/.mailrc... done". This is no longer needed since
tests are run in a temporary home directory instead of the user's
one. So remove these sed(1) calls.
Before the change, the common Emacs test scheme was to print
buffer content to stdout and redirect it to a file or capture it
in a shell variable. This does not work if we switch to using
emacsclient(1) for running the tests, because you can not print
to the stdout in this case. (Actually, you can print to stdout
from Emacs server, but you can not capture the output on
emacsclient(1)).
The patch introduces new Emacs test auxiliary functions:
`test-output' and `test-visible-output'. These functions are
used to save buffer content to a file directly from Emacs. For
most tests the changes are trivial, because Emacs stdout output
was redirected to a file anyway. But some tests captured the
output in a shell variable and compare it with the expected
output using test_expect_equal. These tests are changed to use
files and test_expect_equal_file instead.
Note: even if we do not switch Emacs tests to emacsclient(1), the
patch makes tests cleaner and is an improvement.
Most test_emacs calls have long arguments that consist of many
expressions. Putting them on a single line makes it hard to read
and produces poor diff when they are changed. The patch puts
every expression in test_emacs calls on a separate line.
Few Emacs tests had test_expect_equal_file arguments in the wrong
order: the first argument should be the test output and the
second one should be the expected.
We exercise each of the documented values (nil, a string, and a
list). For the list, we test matching a specific entry, matching a
catch-all regular expression, and no match at all (in which case there
is no FCC set).
This tests the recently-added detection/hiding of top-posted quotations and
in the testing, it takes advantage of the less-recently-added
visible-buffer-string function for emitting only the visible text
from a buffer.
This is a much cleaner way to do the emacs tests, since we're actually
comparing output against existing files with expected output. We also
won't miss any trailing newlines this way.
And speaking of which, one of the expected output files was actually
missing a trailing blank line that was actually in one of the original
messages, so this was fixed.
Use .gz filenames for saved attachments in the tests to check
that Emacs does not re-compress the file.
Use test_expect_equal_file instead of test_expect_equal to avoid
binary output on the console.
The patch replaces all (message (buffer-string)) calls in emacs
tests with (princ (buffer-string)). This avoids accidentally
interpreting '%' as format specifiers and makes code simpler
because we do not need to capture stderr.
Also, the patch works around an Emacs (23.3+1-1 on current Debian
Unstable) segfault in "Ensure that emacs doesn't drop results"
test. Note: the segfault does not happen on every test run.
Though, it seems to be consistently reproducible if the test uses
300 messages instead of 30. Hopefully, it is the crash described
in Emacs bug #8545 [1] which is already fixed.
[1] http://debbugs.gnu.org/cgi/bugreport.cgi?bug=8545
Change #!/bin/bash at start of tests to "#!/usr/bin/env bash". That way
systems running on bash < 4 can prepend bash >= 4 to path before
running the tests.
Use varying dates in the test messages to test the order authors are
listed in. Add tests with repeated author names and unusual date
ordering. Most of these are broken at the moment, but will be fixed
shortly.
Edited-by: Carl Worth <cworth@cworth.org>: Also update the expected
results for existing emacs tests that currently codify the incorrect
author ordering, (and similarly note them as broken in the current
test suite).
The FCC code saves a message in maildir format, and sets the S flag by
default, so now, automatically, FCC messages will not show up as
"unread", (which seems natural enough).
This was too rude of a thing to do and could easily introduce
problems, (as reported by Rob Browning whose environment required some
HOME-specific things for shell startup).
Instead, implement more focused changes to ensure that particular file
in $HOME don't cause problems. Specifically, we fix known problems
with ~/.signature and ~/.mailrc here.
I still don't know everything about how I want search order to be
customizable, but I do like the current defaults, (namely, performing
a new search gives results newest first, but performing a saved search
like "tag:inbox" gives results as oldest first).
Until we come up with a better plan for people to select what *they*
want, (rather than just getting what I want), let's codify the current
results in the test suite.
Some recently-added tests used hard-coded thread ID values in search
specifications. This is unreliable since the thread IDs depend on the
order in which "notmuch new" encounters new files, (which in turn can
depend on inode ordering within the filesystem).
Fix these by using the new "notmuch search --output=threads" to find the
correct thread IDs given a hard-coded (but reliable) message ID.
The reply is primarily taken care of by "notmuch reply" which is already
thoroughly tested. But a recent bug is inserting a duplicate From header
in the emacs-based reply. So exercise that bug here.
We test that the message we sent via (fake) SMTP is included in the mail
index after a "notmuch new". This verifies that the FCC setting indeed
successfully saved the sent message within the notmuch mail store.
Simply setting an explicit date is cleaner than letting the current,
(arbitrary), date get generated for the email message and then constantly
filtering that date out of search results.
Now that the FCC code is fixed to use the notmuch database path, we can
actually enable this by default, which should be highly useful for all
new users of notmuch.
Rather than *reall* sending mail here, we instead have a new test
program, smtp-dummy which implements (a small piece of) the
server-side SMTP protocol and saves a mail message to the filename
provided. This gives us reasonable test coverage of a large chunk of
the notmuch+emacs code base (down to talking to an SMTP server with
the final mail contents).
If Xapian sees unquoted ".." as in id:123..456 then it thinks that's a
range specification. We avoid this problem by instead passing
id:"123..456" to Xapian.
We simulate the act of selecting the "inbox" saved search from
notmuch-hello and the act of selecting a desired thread from the
notmuch-search results.
The test for the navigation of notmuch-hello is currently marked as
BROKEN since its output is in the opposite order compared to the
'(notmuch-search "tag:inbox")' test. This question of ordering is a
currently open issue on the notmuch mailing list, so we'll let the
test suite reflect that for now.
Finally, this commit also abstracts some common emacs lisp code,
(waiting for the current buffer's process to complete), into a new
notmuch-test-wait function that is made available to anything calling
test_emacs.
