We need to be able to test for the presence of a newline at the end of
output. There's no good way to capture trailing newlines in bash, so
redirecting output to a file is the next best thing. This new
function should be used when testing for output that is expected to
have trailing newlines.
The next commit will demonstrate the use of this.
The patch replaces all (message (buffer-string)) calls in emacs
tests with (princ (buffer-string)). This avoids accidentally
interpreting '%' as format specifiers and makes code simpler
because we do not need to capture stderr.
Also, the patch works around an Emacs (23.3+1-1 on current Debian
Unstable) segfault in "Ensure that emacs doesn't drop results"
test. Note: the segfault does not happen on every test run.
Though, it seems to be consistently reproducible if the test uses
300 messages instead of 30. Hopefully, it is the crash described
in Emacs bug #8545 [1] which is already fixed.
[1] http://debbugs.gnu.org/cgi/bugreport.cgi?bug=8545
Change #!/bin/bash at start of tests to "#!/usr/bin/env bash". That way
systems running on bash < 4 can prepend bash >= 4 to path before
running the tests.
The patch adds test-lib.el file for Emacs tests auxiliary stuff.
Currently, it implements two functions: `visible-buffer-string'
and `visible-buffer-substring'. These are similar to standard
counterparts without "visible-" prefix but exclude invisible
text. The functions are not used anywhere at the moment but
should be useful for testing hiding/showing in the Emacs
interface.
Edited-by: Carl Worth <cworth@cworth.org> Fixed "basic" test to ignore
new test-lib.el file.
The example multipart message is made a bit more complicated by adding
a message/rfc822 message, and the all parts are output and tested in
all output formats.
Remove double quotes and flatten "foo@bar.com <foo@bar.com>" to
"foo@bar.com".
Edited-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net> (clean up
expected output for emacs tests).
Signed-off-by: Jameson Rollins <jrollins@finestructure.net>
The compilation of the smtp-dummy program would fail if a build was
attempted on a system without getline. Fix this by simply including
the existing notmuch_compat_srcs variable when constructing the list
of source files for compiling smtp-dummy.
Previously, notmuch show flattened all output, losing information
about the nesting of the MIME hierarchy. Now, the output is properly
nested, (both in the --format=text and --format=json output), so that
clients can analyze the original MIME structure.
Internally, this required splitting the final closing delimiter out of
the various show_part functions and putting it into a new
show_part_end function instead. Also, the show_part function now
accepts a new "first" argument that is set not only for the first MIME
part of a message, but also for each first MIME part within a series
of multipart parts. This "first" argument controls the omission of a
preceding comma when printing a part (for json).
Many thanks to David Edmondson <dme@dme.org> for originally
identifying the lack of nesting in the json output and submitting an
early implementation of this feature. Thanks as well to Jameson Graef
Rollins <jrollins@finestructure.net> for carefully shepherding David's
patches through a remarkably long review process, patiently explaining
them, and providing a cleaned up series that led to this final
implementation. Jameson also provided the new emacs code here.
Previously, the outer multipart part of any multipart/mixed,
multipart/signed, etc. MIME message was silently omitted from the
"notmuch show" output. This prevented any client from correctly
determining to which parts a signature applies, for example.
Now, we actually emit these parts as their own parts. The output is
still flattened---the contained parts are not yet included "within"
the multipart part---so it's still not possible to determine to which
parts a signature applies, but this is one step along the path.
The test suite is updated to reflect this change, (though we'll
eventually want to fix the emacs interface to not display buttons for
the multipart enclosure parts as there's nothing useful for the user
to actually do with them).
This tests "notmuch show" with both --format=text and --format=json on
a message with some non-trivial MIME multipart nesting, (multiple parts
within a multipart/mixed part which is within a multipart/signed part).
The test captures the current behavior (where only the leaf nodes of
the MIME structure are emitted as a flat list---the multipart parts
are effectively ignored). We plan to soon change the json output at
least to emit an actual hierarchy matching the MIME structure, (at
which point we will update this test).
Theses were expected failures only due to a bug in GMime (with
versions of GMime before 2.4.18). As of GMime version 2.4.18 this bug
is fixed and these tests now pass.
With the previous commit, unexpected output before or between search results
would be displayed. However, trailing junk from the "notmuch search" output
would still be silently swallowed.
The most common case for an error message from "notmuch search" would be
an invalid command-line, and in that case, there would be no search results
and the trailing error message would get swallowed.
We fix the process sentinel to check for leftover data and add it to the
final buffer. We also add a test case to ensure this works.
Rather than silently swallowing unexpected output, the emacs interface will now
display it. This will allow error messages to actually arrive at the emacs
interface (though not in an especially pretty way). This also allows for easier
investigation of the inadvertent swallowing of search results that span page
boundaries (as demonstrated by the recent added emacs-large-search-buffer test).
The page-boundary bug has been present since a commit from 2009-11-24:
93af7b5745
Many thanks to Thomas Schwinge for tracking that bug down and
contributing the test for it.
The new name is more descriptive of the bug being tested. Also, the test
is rewritten slightly so that it's much more plain to see how the bug
manifests itself, (that messages are droped from the emacs result at
regular intervals). Primarily, this is by collapsing the large blobs
used to inflate the message subjects.
In the original json code, search matching nothing would return a
valid, empty json array (that is, "[]"). I broke this in commit
6dcb7592e3 when adding support for
--output=threads|messages|tags. This time, while fixing the bug also
add a test to the test suite to help avoid future regressions.
Now that we understand the bug here, we rename this test to
search-insufficient-from-quoting to clarify the bug being exercised,
(which occurs when the From: line contains an unquoted '.' character).
We also mark these tests as expected failures until the bug gets fixed.
Currently, there are two places in the test framework that contain very
long list on a single line. Whenever a test is added (or changed) in
several branches and these branches are merged, it results in conflict
which is hard to resolve because one has to go through the whole long
line to find where the conflict is.
This patch splits these long lists to several lines so that the
conflicts are easier to resolve.
Currently, whenever we call index_terms multiple times for a single
field, the term generator is being reset to position 0 each time. This
means that with text such as:
To: a@b.c, x@y.z
one can get a bogus match by searching for:
To: a@y.c
Thanks to Mark Anderson for reporting the bug, (and providing a nice,
minimal test case that inspired what is used here).
This is a new feature which is not implemente yet, so these tests mostly
fail currently. A subsequent commit will add the feature and cause these
tests to start passing.
These tests verify that we can search for containing folders of mail files
by word or by phrase and that the search terms are updated correctly when
directories are renamed.
This reverts commit f22a7ec1e2.
Interrupting the test suite due to an actual bug in a test script
would be just fine, but interrupting the run of the entire test suite
at the first test failure is unacceptable.
Previously, this directory was only preserved for failing tests. But
it's important to be able to easily debug known-broken tests, so
preserve the actual vs. expected output for those as well.
Use varying dates in the test messages to test the order authors are
listed in. Add tests with repeated author names and unusual date
ordering. Most of these are broken at the moment, but will be fixed
shortly.
Edited-by: Carl Worth <cworth@cworth.org>: Also update the expected
results for existing emacs tests that currently codify the incorrect
author ordering, (and similarly note them as broken in the current
test suite).
Incremental search does not match strings that span a
visible/invisible boundary. This results in failure to correctly
isearch for authors in `notmuch-search' mode if the name of the author
is split between the visible and invisible components of the authors
string. To avoid this, attempt to truncate the visible component of
the authors string on a boundary between authors, such that the
entirety of an author's name is either visible or invisible.
When a test fails, a tmp.<testname> file is left behind. These files
are useful for the person debugging the test failure, but are never
anything we want to commit.
Edited-by: Carl Worth <cworth@cworth.org>: Changed from tmp.emacs to
tmp.* and added explanation in the commit message.
This testing *does* capture the bug of missing '[' and ']' characters
int "notmuch search --output=tags" case. This is another manifestation
of the same bug causing the missing final newline (as mentioned in the
previous commit).
This code simply wasn't being exercised by the test suite before, so
this will be useful.
Meanwhile, there's currently a bug in "notmuch search --output=tags"
in that it doesn't print a final newline. But the current test suite
isn't able to catch this bug since the $() construct of the shell
doesn't preserve the distinction of whether the final newline is
present or not.
This test script does some initial test setup (generating a few
messages), which is all well and good, but we don't need to print that
as a test result---particularly since the test result was effectively
hard-coded to always pass.
When test_begin_subtest is not followed by corresponding test_expect_equal,
the output of the rest of the test script is errornously suppressed. Add
code to detect these bugs in test scripts.
Break notmuch-test whenever a test script returns non-zero status.
This happens either when some test from the script fails or when there
is an error in the script.
This is especially useful in the latter case since the error may not
appear in the final aggregated results.
The newline was removed from say_color in commit 222926ab to allow
printing test status in the beginning of the line. Error messages are
never followed by other text so we add the newline to error function.
Git-style tests (test_expect_success etc.) suppress stdout and stderr
unless -v is given. Notmuch-style tests (created by test_begin_subtest
and test_expect_equal) do not have this behavior so implement it the
same.
Additionally, for both test styles, the test-lib.sh is changed so that
the content of suppressed stdout and stderr is shown in case of failed
test.
Finally a test for this functionality is added to basic tests.
This is to prevent notmuch from destroying any information the user
has encoded as flags in the maildir filename. Tests are also added to
the test suite to verify the documented behavior.
Some people use notmuch with non-maildir files, (for example, email
messages in MH format, or else cool things like using sluk[*] to suck
down feeds into a format that notmuch can index).
To better support uses like that, don't do any renaming for files that
are not in a directory named either "new" or "cur".
[*] https://github.com/krl/sluk/
The FCC code saves a message in maildir format, and sets the S flag by
default, so now, automatically, FCC messages will not show up as
"unread", (which seems natural enough).
There's nothing in the current API documentation that would suggest
the behavior being tested here. Attempt to implement this could have
some nasty side effects, (such as notmuch_message_maildir_flags_to_tags
implicitly calling notmuch_message_tags_to_maildir_flags and maybe
even opening up some bad looping possibilities).
Much better to stick with what we have documented, which we believe will
actually be useful, (and easy enough to comprehend).
These needed to be changed to be brought up to the current state of
the maildir-sync tests. This includes style changes, but also the
elimination of any assumption about pre-existing message filenames,
(such as msg-003) which actually don't exist anymore.
Also, the known broken tests are changed to emit FAIL rather than
BROKEN simply to make them easier to fix, (so that they print the
current problems rather than hiding them).
Finally, an additional test is added to ensure that when a duplicate
file is added without flags, it doesn't invalidate flags from other
duplicates, (instead the flags are effectively merged).
Add maildir synchronization tests for multiple messages with the same
message-id. As this is not yet implemented in notmuch, some of these
teste are marked as BROKEN.
I use $(< ) operator to avoid fiddling with stripped trailing newlines
from test results which happens when output+=$(command) is used.
This change reworks these tests in several ways:
1. Bring tests into "new" test style preferring test_expect_equal over
test_expect_success in almost all cases.
2. Don't emit test results for intermediate items not actually being
tested, (things like "no new messages", "search for message",
etc.). Those things are already covered by existing tests such as
"basic" or "search" and only serve to obscure what's actually being
tested.
3. Change sense of the test showing failure to rename a file from
"new" to "cur" when "cur" doesn't exist.
In this case, notmuch should detect that this is not a maildir and
should not attempt to do any renaming of the file.
4. Extend dump/restore test to also exercise addition of tag, not just
removal.
Both items #3 and #4 above show shortcomings in the current
implementation. These are currently resulting in test results of FAIL
and indicate bugs that need to be fixed.
We have test names like maildir-sync now, so it's cleaner if the
temporary files created are named things like maildir-sync-10.out
rather than maildir-10.out. Presumably the extra stripping here came
from naming conventions in git's test suite.
This is part of an effort to avoid proliferation of excessive
top-level notmuch commands. Also, "raw" better captures the
functionality here, (as opposed to "cat" which is a fairly oblique
reference to a bad Unix abbreviation whose metaphor doesn't work here
since "notmuch cat" operates only on a single message and hence cannot
"con'cat'enate" anything).
This command outputs a raw message matched by search term to the
standard output. It allows MUAs to access the messages for piping,
attachment manipulation, etc. by running notmuch cat rather then
directly access the file. This will simplify the MUAs when they need
to operate on a remote database.
Edited-by: Carl Worth <cworth@cworth.org>: Remove trailing whitespace,
add missing "test_done" to new test script to avoid "Unexpected exit"
error.
This was too rude of a thing to do and could easily introduce
problems, (as reported by Rob Browning whose environment required some
HOME-specific things for shell startup).
Instead, implement more focused changes to ensure that particular file
in $HOME don't cause problems. Specifically, we fix known problems
with ~/.signature and ~/.mailrc here.
The original mails used to pupulate the mail corpus had had their
attachments (obnoxiously) scrubbed by the pipermail mail archiver.
Since we actually want to test the handling of attachments, this is
less than useful. Restore these files from my own collection, (with
some Received and similar headers pruned).
I still don't know everything about how I want search order to be
customizable, but I do like the current defaults, (namely, performing
a new search gives results newest first, but performing a saved search
like "tag:inbox" gives results as oldest first).
Until we come up with a better plan for people to select what *they*
want, (rather than just getting what I want), let's codify the current
results in the test suite.
After any emacs test failure, the tmp.emacs directory will have this
run_emacs script in it which the user can use to run emacs within the
test suite environment, (pointing at the test suite's notmuch
database, using the local notmuch command-line program, and the local
notmuch emacs lisp code).
My scripts expect that empty search result is actually empty. Since
commit 6dcb7592, even empty search prints a newline character and this
breaks my scripts.
This patch adds a test for this bug. In the test I cannot use
test_expect_equal function as $() operator suppresses the final
newline and this kind of difference is not detected.
test/search | 5 +++++
1 files changed, 5 insertions(+), 0 deletions(-)
The bash code in the test suite is using associative arrays which were
only added to bash as of release 4.0.
If the test suite is run with an older bash, we now immediately error
out and explain the situation, (instead of emitting confusing error
messages and failing dozens of tests, which is what happened before
this change).
Some recently-added tests used hard-coded thread ID values in search
specifications. This is unreliable since the thread IDs depend on the
order in which "notmuch new" encounters new files, (which in turn can
depend on inode ordering within the filesystem).
Fix these by using the new "notmuch search --output=threads" to find the
correct thread IDs given a hard-coded (but reliable) message ID.
The reply is primarily taken care of by "notmuch reply" which is already
thoroughly tested. But a recent bug is inserting a duplicate From header
in the emacs-based reply. So exercise that bug here.
Update the tests so that they no longer expect the Bcc header in the
output of "notmuch reply" now that it has been removed.
Edited-by Carl Worth: Simply applying the change to our newly
modularized test suite.
We test that the message we sent via (fake) SMTP is included in the mail
index after a "notmuch new". This verifies that the FCC setting indeed
successfully saved the sent message within the notmuch mail store.
Simply setting an explicit date is cleaner than letting the current,
(arbitrary), date get generated for the email message and then constantly
filtering that date out of search results.
Now that the FCC code is fixed to use the notmuch database path, we can
actually enable this by default, which should be highly useful for all
new users of notmuch.
Rather than *reall* sending mail here, we instead have a new test
program, smtp-dummy which implements (a small piece of) the
server-side SMTP protocol and saves a mail message to the filename
provided. This gives us reasonable test coverage of a large chunk of
the notmuch+emacs code base (down to talking to an SMTP server with
the final mail contents).
We set the HOME environment variable to the test directory to avoid
the tests relying on any configuration files from the test author's
own home directory, (such as ${HOME}/.emacs or similar).
If Xapian sees unquoted ".." as in id:123..456 then it thinks that's a
range specification. We avoid this problem by instead passing
id:"123..456" to Xapian.
We simulate the act of selecting the "inbox" saved search from
notmuch-hello and the act of selecting a desired thread from the
notmuch-search results.
The test for the navigation of notmuch-hello is currently marked as
BROKEN since its output is in the opposite order compared to the
'(notmuch-search "tag:inbox")' test. This question of ordering is a
currently open issue on the notmuch mailing list, so we'll let the
test suite reflect that for now.
Finally, this commit also abstracts some common emacs lisp code,
(waiting for the current buffer's process to complete), into a new
notmuch-test-wait function that is made available to anything calling
test_emacs.
This should be quite handy for doing automated testing of the
emacs-based functionality in notmuch. This function invokes emacs with
the necessary command-line arguments, (to run in batch mode with no
local initialization, to load the notmuch code from the source
directory, and to ensure an 80-column width).
While adding the documentation here for add_email_corpus I noticed
that the other email-adding functions in test-lib.sh were not yet
documented here, so add all of that documentation.
When the NOTMUCH variable was originally invented it was used as an
explicit path to the notmuch binary being tested. Today, the test
suite sets the PATH variable instead, so the NOTMUCH variable always
has a value of simply "notmuch".
We simplifying that by using the constant value rather than the
continual variable reference.
