We're starting to get test output that's fairly long, so it's much
kinder to just show a diff rather than displaying the complete
expected and actual output. To allow the user to investigate things
after the fact, we save the expected and actual output to files named
test-${test_number}.expected and test-${test_number}.output .
We recently added a feature to name threads based on the messages that
actually matched the search, (as opposed to simply the oldest or
newest message in the thread whether it matched or not). So add tests
for that, and (surprise, surprise!) the feature does not entirely
work.
Hurrah---no more manual verification of that PASS column.
This means that "make test" can actually be a useful part of the
release process now, (since it will exit with non-zero status if there
are any failures).
Previously some tests (dump/restore) were doing ad-hoc verification of
values and their own printing of PASS/FAIL, etc. This made it
impossible to count test pass/fail rates in a single place.
The only reason these tests were written that way was because the old
execute_expecting function only worked if one could directly test the
stdout output of a notmuch command. The recent switch to pass_if_equal
means that all tests can use it.
This feature was added recently and should have gotten a new test at
the time.
As this test demonstrates, the code is broken, ("notmuch search '*'
returns bogus dates of the Unix epoch for any threads where the
term "and" does not appear in any messages).
Using a date in the current year makes the test suite fragile since
the search output will include a date of "January 05" for now, but
will start doing "2010-01-05" in the future.
The filenames aren't predictable (including the current directory) nor
stable from one run to the next (including the PID). This makes it
hard to predict the output from a search command that returns such a
message (such as "*").
The original goal was simply to ensure that each generated message was
distinguishable somehow. So just use the message counter instead.
The old execute_expecting function was doing far too much for its own
good. One of the worst aspects of this was that it introduced
shell-quoting challengers where the caller could not easily control
the precise invocation of the command to be executed.
I personally couldn't find a way to test "notmuch search '*'" without
the shell expanding * against files in the current directory, or
having bogus quotation marks appearing in the search string,
(defeating the recognition of "*" as a special search term).
Hopefully this aspect of the test suite will be much easier to maintain now.
The recent fix to properly decode encoded headers made the expected
output of "notmuch reply" differ by a single space, (previously, there
were two spaces before the References: value and now there is just
one).
Fix the test suite so that these are all noted as correct results
again.
These new tests demonstrate a bug as follows:
Multiple messages are added to the database
All of these message references a common parent
The parent message does not exist in the databas
In this scenario, the messages will not be recognized as belonging to
the same thread. We consider this a bug, and the new tests treat this
as a failure.
Edited by Carl Worth <cworth@cworth.org>: Split these tests into their
own commit (before the fix of the bug). This lets me see the actual
failure in the test suite, before the fix is applied. Also fix the
alignment of new messages from test suite, (so that the PASS portions
all line up---which is important while we're still manually verifying
test-suite results).
These results have all the same terms as the target phrase, but
not in the expected order. They are designed to ensure that we
actually test phrase searches.
And as it turns out, we're not currently quoting the search terms
properly, so the phrase-search tests now fail with this commit.
The sequential identifiers have the advantage of being guaranteed to
be unique (until we overflow a 64-bit unsigned integer), and also take
up half as much space in the "notmuch search" output (16 columns
rather than 32).
This change also has the side effect of fixing a bug where notmuch
could block on /dev/random at startup (waiting for some entropy to
appear). This bug was hit hard by the test suite, (which could easily
exhaust the available entropy on common systems---resulting in large
delays of the test suite).
These tests were surprisingly simple to write---not much code at all
and most of them worked the first time even with hand-prepared
versions of the expected output.
The previous generate_message function is what's needed when testing
"notmuch new". But after that, we never want to generate a message
without also adding it to the index. So create a new add_message
function with this convenience.
This is a test for the recently added feature where we detect that the
reply-to address already exists in the To: or Cc: header so will
already be replied to. In this case we want to include the From:
address in our reply, (where, otherwise we would use the Reply-To
address *instead* of the address in the From header).
The feature tested here is that we reply to both the sender and to
others addresses on the To: line of the original message, but that we
don't reply to our own address.
This is the standard support of reply-to, (replying to that address
rather than the from address). It has nothing to do with the proposed
feature for extra-clever handling of a mail from a mailing-list that
has munged the reply-to header.
When reply to a message addresses to an address configured in the
other_email setting in the configuration file, the reply should use
that address in the From header. Test this.
We were sleeping merely to ensure that our updates to the mail store
would result in the mtime of the appropriate directories being
updated. We make the test suite run faster by not sleeping, but
instead explicitly updating the mtime of the directory to a future
time with touch.
We're careful to ensure that the time is not merely in the future
compared to the current time, but also later than any previous update
to the same directory mtime.
This makes the test suite bash-specific, but that's not much of
an issue for me, (if somebody else would prefer some other language
then they can rewrite the test suite and maintain it).
The advantage here is that we'll now be able to easily generate
custom messages for testing operations that depend on the message
content, (such as "notmuch reply", etc.).
This notmuch-test script simply runs a few different notmuch operations,
(things that I found were useful while testing the rename-support code).
It's not useful as a test suite yet, since it doesn't actually check
the results of any operation, (the user of the suite has to know what
the results should be and must manually verify them. So there's no
integration with the build system yet, (no "make test" target).
But I didn't want to lose what I had so far, so here it is.
