Commit graph

226 commits

Author SHA1 Message Date
Dmitry Kurochkin
4cc6727688 test: improve known broken tests support
There is existing support for broken tests.  But it is not convenient
to use.  The primary issue is that we have to maintain a set of
test_expect_*_failure functions which are equivalent to the normal
test_expect_* counterparts except for what functions are called for
result reporting.  The patch adds test_subtest_known_broken function
which marks a subset as broken, making the normal test_expect_*
functions behave as test_expect_*_failure.  All test_expect_*_failure
functions are removed.  Test_known_broken_failure_ is changed to
format details the same way as test_failure_ does.

Another benefit of this change is that the diff when a broken test is
fixed would be small and nice.

Documentation is updated accordingly.
2011-09-10 10:13:27 -03:00
Dmitry Kurochkin
0db3a4d5be test: update documentation for test_emacs in test/README
Update test_emacs documentation in test/README according to the latest
changes in emacs tests.  Move the note regarding setting variables
from test/emacs to test/README.
2011-09-10 10:12:03 -03:00
Jameson Graef Rollins
c8598d9a99 test: overhaul multipart test to test for improved message/rfc822 handling
The main goal of this overhaul is to define how message/rfc822 parts
should be handled.  message/rfc822 parts should be output in a similar
fashion to the outer message, including some subset of the rfc822
headers.  The following decisions about formatting of message/rfc822
parts were made:

The format and content of message/rfc822 parts shall be as similar as
possible to that of full messages.  In particular, for formatted
outputs, the "content" of rfc822 part output should include "headers"
and "body" fields).

The "body" field shall include the body of the message.

The "headers" field shall include the following headers, since these
are the ones available from the GMimeMessage:

  "From"
  "To"
  "Cc"
  "Subject"
  "Date"

However, for the case of --format=raw the raw rfc822 should be output,
including all headers.

A subset of relevant headers shall be output in reply.

The test embedded rfc822 message is also modified to be itself
multipart, so we can more fully test how all sub parts of the message
part are output.

Note added by Committer:

Currently, expect one test (--format=raw --part=3, rfc822 part) to fail.
2011-09-05 22:57:39 -03:00
Jameson Graef Rollins
ec2b0a98cc test: Fix date in test message in multipart test.
The test message date, "Tue, 05 Jan 2001 15:43:57 -0000", is not
actually a real date.  05 Jan 2001 was in fact a Friday, not a
Tuesday.  Date parsers (such as "date" in coreutils) will return "Fri"
as the day for this string, even if "Tue" is specified.

Also, the time zone "-0000" is actually always returned as "+0000", so
we change that here was well.

This will be relevant for later patches when we begin parsing rfc822
part headers, where gmime returns a parsed date string.

If we do want to test date parsing, we should do that in a separate
test.
2011-09-05 08:54:56 -03:00
Jameson Graef Rollins
ea3a26f590 test: some small fixes to multipart test
There were two "--format=text --part=0" tests.  One of them was
supposed to be a test for "--format=text --part=1".

There were also two errant "test_expect_equal_file OUTPUT EXPECTED"
lines, that are removed here.
2011-09-05 08:54:56 -03:00
Carl Worth
580de27177 emacs: Fix to unconditionally display subject changes in collapsed thread view
The feature to show subject changes in the collapsed thread view was
originally added (8ab433607) with an option
(notmuch-show-always-show-subject) to display the subject
for all messages, even when there was no change.

The subsequent commit (4f04d273) changed the sense of the test (or to
and) and the name of the controlling variable
(notmuch-show-elide-same-subject).

But this commit is broken in a few ways:

  1. The original definition of notmuch-show-always-show-subject was
     left around

     But the variable isn't actually used in the code at all, so it
     just adds clutter and confusion to the customization interface.

  2. The name and description of the controlling variable doesn't
     match the implementation

     The name suggests that setting the variable to t will cause
     repeated subjects to be elided, (suggesting that when it is nil
     all subjects will be shown).

     However, when the variable is nil, no subjects are shown. So a
     correct name for the variable in this sense would be
     notmuch-show-subject-changes.

Showing subject changes is a useful feature, and should be on by
default. (We don't want to bury generally useful features behind
customizations that users have to find).

Rather than fixing the name of the variable and changing its default
value, here we remove the condition entirely, such that the feature is
enabled unconditionally.

So both the currently-used variable and the stale definition of the
formerly-used are removed.

Also, the one relevant test-suite result is updated, (showing the
intial subject of a collapsed thread, and no subject display for later
messages that do not change the subject).
2011-07-01 02:00:25 -07:00
Jameson Graef Rollins
a31ad0bd01 test: document test_expect_equal_file
This test was not properly documented when it was originally added (my
bad).
2011-06-29 15:36:43 -07:00
Austin Clements
47afbdfb79 test: Nix increment_mtime.
With the fix for the mtime race, this workaround is no longer
necessary.
2011-06-29 15:26:45 -07:00
Pieter Praet
ec1b568929 test: revert non-intentional changes introduced in eb4cf465
eb4cf465 introduces changes which weren't part of the submitted
patch (id:"87liwlip2j.fsf@gmail.com"), presumably made during
resolving merge conflicts.

The first one causes the title of a test to be printed a second time,
in place of the correct title:

  PASS   Message with .. in Message-Id:
  PASS   Message with .. in Message-Id:

instead of:

  PASS   Message with .. in Message-Id:
  PASS   Sending a message via (fake) SMTP

The second one is simply the insertion of a line break, so no harm there.

This commit reverts both changes, as they were clearly accidental.

Signed-off-by: Pieter Praet <pieter@praet.org>
2011-06-29 14:20:57 -07:00
Carl Worth
55a78d5dbd test: Use increment_mtime rather than sleep
The sleep was to force the directory's mtime to advance between the
previous notmuch new and the subsequent rm;notmuch new.

The current convention is to use the existing increment_mtime function
for this purpose, (which avoids the test suite being slowed down by
calls to sleep).

Thanks to Austin Clements for noticing this undesired sleep.
2011-06-29 14:11:38 -07:00
Mark Anderson
86e0baeb6d test:Folder tags shouldn't match after removal of file in given folder
Test for bug.  Current stemming support for notmuch adds extra terms
to the DB which aren't removed when the file renames are detected.

When folder tags are added to a message, Xapian terms for both XFOLDER
and ZXFOLDER are generated.  When one of the filenames are
renamed/removed, only the XFOLDER tags are removed, leaving it possible
for a match on a folder: tag that was previously but is no longer a
match in the maildir.
2011-06-29 14:10:05 -07:00
Dmitry Kurochkin
f74c4bc2f3 test: fix tests after notmuch show output changes related to filenames
Adding the filename propery alongside ID and Content-type.

This makes the test suite pass again after the recent change.
2011-06-28 20:06:20 -07:00
Dmitry Kurochkin
a854d06e92 test: use emacsclient(1) for Emacs tests
Before the change, every Emacs test ran in a separate Emacs
instance.  Starting Emacs many times wastes considerable time and
it gets worse as the test suite grows.  The patch solves this by
using a single Emacs server and emacsclient(1) to run multiple
tests.  Emacs server is started on the first test_emacs call and
stopped when test_done is called.  We take care not to leave
orphan Emacs processes behind when test is terminated by whatever
reason: Emacs server runs a watchdog that periodically checks
that the test is still running.

Some tests need to provide user input.  Before the change, this
was done using echo(1) to Emacs stdin.  This no longer works and
instead `standard-input' variable is set accordingly to make
`read' return the appropriate string.
2011-06-28 17:10:55 -07:00
Carl Worth
6ea26cfb81 test: Increment mtime when delivering a message with emacs_deliver_message
Without this, mail messages delivered by emacs_deliver_message might
not be seen by the next invocation of "notmuch new", (which can lead
to test-suite failures if emacs_deliver_message is fast enough).
2011-06-28 17:10:55 -07:00
Mark Anderson
eb4cf465a5 test:Improve test behaviors when --root is used
Change add_email_corpus, emacs_deliver_message and tests to use
$TEST_DIRECTORY instead of '..'.

This improves the behavior of the usage of --root=<dir>, as the
assumption of what '..' means will usually be incorrect.

Document -root option in README and update valgrind to work with
-root.
2011-06-28 16:01:56 -07:00
Dmitry Kurochkin
273d896097 test: generate run_emacs script once on test startup
Instead of generating auxiliary run_emacs script every time
test_emacs is run, do it once in the beginning of the test.

Also, use absolute paths in the script to make it more robust.
2011-06-28 15:06:47 -07:00
Dmitry Kurochkin
0cc5483a9c test: set variables using let' instead of setq' in Emacs tests
Using `setq' for setting variables in Emacs tests affect other
tests that may run in the same Emacs environment.  Currently it
works because each test is run in a separate Emacs instance.  But
in the future multiple tests will run in a single Emacs instance.
The patch changes all variables to use `let', so the scope of the
change is limited to a single test.
2011-06-28 15:06:47 -07:00
Dmitry Kurochkin
0417c1fad6 test: use emacs_deliver_message in Emacs SMTP send test
Minor changes to expected results of other Emacs tests were
needed because the message Date header changed.
2011-06-28 15:06:47 -07:00
Dmitry Kurochkin
5eb6b2767a test: remove some sed(1) calls in Emacs tests
Few Emacs tests used sed(1) to remove unexpected output in the
beginning to avoid getting confused by messages such as "Parsing
/home/cworth/.mailrc... done".  This is no longer needed since
tests are run in a temporary home directory instead of the user's
one.  So remove these sed(1) calls.
2011-06-28 15:06:47 -07:00
Dmitry Kurochkin
3b24b396c4 test: save buffer content to file instead of printing it in Emacs tests
Before the change, the common Emacs test scheme was to print
buffer content to stdout and redirect it to a file or capture it
in a shell variable.  This does not work if we switch to using
emacsclient(1) for running the tests, because you can not print
to the stdout in this case. (Actually, you can print to stdout
from Emacs server, but you can not capture the output on
emacsclient(1)).

The patch introduces new Emacs test auxiliary functions:
`test-output' and `test-visible-output'.  These functions are
used to save buffer content to a file directly from Emacs.  For
most tests the changes are trivial, because Emacs stdout output
was redirected to a file anyway.  But some tests captured the
output in a shell variable and compare it with the expected
output using test_expect_equal.  These tests are changed to use
files and test_expect_equal_file instead.

Note: even if we do not switch Emacs tests to emacsclient(1), the
patch makes tests cleaner and is an improvement.
2011-06-28 15:06:47 -07:00
Dmitry Kurochkin
caeb05493d test: wrap and indent test_emacs calls
Most test_emacs calls have long arguments that consist of many
expressions.  Putting them on a single line makes it hard to read
and produces poor diff when they are changed.  The patch puts
every expression in test_emacs calls on a separate line.
2011-06-28 15:06:47 -07:00
Dmitry Kurochkin
dd75723921 test: cleanup test_emacs
Move auxiliary function definition and configuration from command
line to test-lib.el.
2011-06-28 15:06:47 -07:00
Dmitry Kurochkin
5806c471de test: do not set `message-signature' in test_emacs
It is no longer needed since tests are run in a temporary home
directory instead of the user's one.
2011-06-28 15:06:47 -07:00
Dmitry Kurochkin
e4fc21e88b test: fix argument order of test_expect_equal_file in few tests
Few Emacs tests had test_expect_equal_file arguments in the wrong
order: the first argument should be the test output and the
second one should be the expected.
2011-06-28 15:06:47 -07:00
Mark Anderson
c87da6961d test:Expect multiple filenames for message with multiple files
Update the test mail corpus to have two files with the same content to
expose the bug where a single message with multiple filenames only
reports a single filename.

Update expected results for search --output=files to match new
behavior for multiple files corresponding to a single message

Signed-off-by: Mark Anderson <ma.skies@gmail.com>
2011-06-28 12:05:50 -07:00
Dmitry Kurochkin
8bf0c1c3de test: remove useless test_emacs call from an emacs FCC test
This was inadvertently left over when debugging an early version of
this commit. -Carl Worth <cworth@cworth.org>
2011-06-23 17:11:04 -07:00
Pieter Praet
b4c9ee2809 fix sum moar typos [test-case data]
Various typo fixes in some test-case data.

Signed-off-by: Pieter Praet <pieter@praet.org>

Edited-by: Carl Worth <cworth@cworth.org> Restricted to just
test-case data.
2011-06-23 15:59:11 -07:00
Pieter Praet
432e091924 fix sum moar typos [user-visible documentation in code]
Various typo fixes in documentation within the code that can be made
available to the user, (emacs function help strings, "notmuch help"
output, notmuch man page, etc.).

Signed-off-by: Pieter Praet <pieter@praet.org>

Edited-by: Carl Worth <cworth@cworth.org> Restricted to just
documentation and fixed fix of "comman" to "common" rather than
"command".
2011-06-23 15:58:50 -07:00
Pieter Praet
8bb6f7869c fix sum moar typos [comments in source code]
Various typo fixes in comments within the source code.

Signed-off-by: Pieter Praet <pieter@praet.org>

Edited-by: Carl Worth <cworth@cworth.org> Restricted to just
source-code comments, (and fixed fix of "descriptios" to "descriptors"
rather than "descriptions").
2011-06-23 15:58:39 -07:00
Pieter Praet
730acd4764 fix sum moar typos [build scripts, Makefiles]
Various typo fixes in comments within the Makefile and other build scripts.

Signed-off-by: Pieter Praet <pieter@praet.org>

Edited-by: Carl Worth <cworth@cworth.org> Restricted to just build files.
2011-06-23 15:44:59 -07:00
Pieter Praet
18bf91def9 fix sum moar typos [text files]
Various typo fixes in auxiliary text files included with the source,
(README, TODO, etc.).

Signed-off-by: Pieter Praet <pieter@praet.org>

Edited-by: Carl Worth <cworth@cworth.org> Restricted to just text files.
2011-06-23 15:40:50 -07:00
Carl Worth
76b54f1898 test: Test emacs message composing with various values for notmuch-fcc-dirs
We exercise each of the documented values (nil, a string, and a
list). For the list, we test matching a specific entry, matching a
catch-all regular expression, and no match at all (in which case there
is no FCC set).
2011-06-23 15:19:06 -07:00
David Bremner
fba968dbfa tests: add a test for symbol hiding side effects
The worry here is that a binary linking with libnotmuch might lose
access to Xapian::Error symbols because libnotmuch hides them.

We are careful here to create ./fakedb/.notmuch in order to trigger a
Xapian exception, and not just a missing file check.

Thanks to jrollins and amddragon for suggestions.
(cherry picked from commit 66f37f5f6864a988f94ddb893e3a176af57f6c8e)
2011-06-23 07:05:25 -03:00
Carl Worth
b0ba84f9e7 smtp-dummy: Prefer return rather than exit() in main.
The main() function should be written as just another function with a
return value. This allows for more reliable code reuse. Imagine that
main() grows too large and needs to be factored into multiple
functions. At that point, exit() is probably the wrong thing, yet can
also be hard to notice as it's in less-frequently-tested exceptional
cases.
2011-06-22 06:38:33 -07:00
Dmitry Kurochkin
41a094624a Fix compilation warnings in test/smtp-dummy.c.
* Remove unused variables in main(): buf, bytes and greeting.
* Replace return with no value in main() with exit(3).
2011-06-22 06:38:33 -07:00
David Bremner
114eb1c520 tests: Add optional use of timeout utility, if present.
Each top level test (basic, corpus, etc...) is run with a fixed
timeout of 2 minutes.

The goal here is to treat a hung test as a failure. The emacs test for
sending mail is known to be problematic on the Debian
autobuilders. This is both a bandaid fix for that, and a sensible long
term feature.
(cherry picked from commit 5f99c80e02736c90495558d9b88008a768876b29)
2011-06-22 08:11:32 -03:00
Dmitry Kurochkin
59b251ef94 test emacs: Add tests for hiding messages in notmuch-show view
This test is expected to fail as it exposes a current bug, (which we
hope to fix soon).
2011-06-15 07:07:32 -07:00
Carl Worth
b7db7ea420 test: Fix from-guessing to actually span Received headers over multiple lines
The intent was always to make these Received headers span multiple
lines. But the escapes were causing the shell to ignore the newlines,
so that the result instead was long Received headers on a single line
each.

Fixing the intent here doesn't actually change the test-suite results
at all.
2011-06-10 17:20:22 -07:00
Carl Worth
ac43a96988 test: Extend from-guessing test with a test with multiple Received headers
This is much more realistic, as most messages in the wild will have multiple
Received headers. Also, this demonstrates a current bug in the Received
header parsing, (multiple Received headers are not properly concatenated
depending on the order in which headers are parsed in a message).
2011-06-10 17:03:14 -07:00
Carl Worth
d5edb1122d test: Add test that emacs detects and hides top-post quotes of original messages
This tests the recently-added detection/hiding of top-posted quotations and
in the testing, it takes advantage of the less-recently-added
visible-buffer-string function for emitting only the visible text
from a buffer.
2011-06-10 16:28:26 -07:00
Jameson Graef Rollins
61ff61cc2c test: modify multipart test to use test_expect_equal_file
Again, this is a much cleaner and more thorough test, and in fact
exposes a bug in the format=text output, that will be fixed the next
commit.  Because of this, some of the multipart tests currently fail.
2011-06-03 15:24:35 -07:00
Jameson Graef Rollins
5b0bf70faa test: update emacs test to use test_expect_equal_file
This is a much cleaner way to do the emacs tests, since we're actually
comparing output against existing files with expected output.  We also
won't miss any trailing newlines this way.

And speaking of which, one of the expected output files was actually
missing a trailing blank line that was actually in one of the original
messages, so this was fixed.
2011-06-03 14:29:05 -07:00
Dmitry Kurochkin
79a587d963 test: add "notmuch-show for message with invalid From" test 2011-06-03 14:07:19 -07:00
Austin Clements
c62a0d016f Fix misspelling in search output sanitization test.
A simple spelling fix in the text description.

Signed-off-by: Jameson Graef Rollins <jrollins@finestructure.net>
2011-06-03 12:30:55 -07:00
Jameson Graef Rollins
c4e6d3291c test: cleanup search-output test names (no functional change)
Signed-off-by: Jameson Graef Rollins <jrollins@finestructure.net>
2011-06-01 21:59:42 -07:00
Florian Friesdorf
8dcd8b6987 test: add test for sanitized notmuch-search output
This feature was recently added, so it of course needs a test now.

Signed-off-by: Jameson Graef Rollins <jrollins@finestructure.net>

Edited-by: Carl Worth <cworth@cworth.org> Fixed test to use
notmuch_search_sanitize in order to be robust against unpredictable
thread ID numbers, (due to unpredictable order in which the filesystem
presents files).
2011-06-01 21:58:16 -07:00
Florian Friesdorf
e584c64c5b test: Create and set temporary home directory
In the master branch in test/emacs two tests access the build users home
directory, so does emacs_deliver_message in the crypto branch.

The tests should not touch the build user's home directory. The patch
creates a directory in the temporary test directory and sets home
accordingly.

In case of a non-existent home directory, the tests are failing without
this patch.

Signed-off-by: Jameson Graef Rollins <jrollins@finestructure.net>
2011-06-01 17:46:51 -07:00
Jameson Graef Rollins
afdfa5674c test: remove json test for search null result, since it's being more properly tested in search-output
The test in json was inferior to the one in search-output, since it
wasn't properly testing for the presence of a trailing newline.
2011-06-01 16:35:11 -07:00
Jameson Graef Rollins
7fdeb0fb63 test: move utf-8 subject search test from json to search test script
This test doesn't have anything to do with json, and has everything to
do with testing search capability, so I'm not sure why it was in the
wrong place.
2011-06-01 16:35:11 -07:00
Jameson Graef Rollins
7a453716c2 test: Move a test from search to search-output, and add a similar json test
The "Search for non-existent message prints nothing" test fits better
with the existing tests in search-output, so move it there. Also add a
similar test for the --format=json case.

These tests also use the new test_expect_equal_file function, (to ensure
that the presence of a trailing newline is correctly tested).
2011-06-01 16:31:52 -07:00