We start modestly, with a (slightly modified) test case from Kazuo
Teramoto. Originally it just made sure the bindings didn't crash; here
we check that by comparing the output with that of notmuch search.
The fake missing binary functions check if the binary has already be
added to the diagnostic message to avoid duplicates. Unfortunately,
this check was buggy because the message string does not have the
trailing space.
test_missing_external_prereq_${binary}_ variable indicates that the
binary is missing. It must be set in test_declare_external_prereq()
outside of the fake $binary() function.
If emacs is not available, test_expect_equal would be called with only
one argument. The patch fixes this by quoting the (possibly empty)
$(cat OUTPUT) argument.
Some tests (e.g. crypto) do a common initialization required for all
subtests. The patch adds a check for missing external dependencies
during this initialization. If any prerequisites are missing, all
subtests are skipped.
The check is run on the first call of test_reset_state_ function, so
no changes for the tests are needed.
There is existing support for general prerequisites in the test suite.
But it is not very convenient to use: every test case has to keep
track for it's dependencies and they have to be explicitly listed.
The patch aims to add better support for a particular type of external
dependencies: external executables. The main idea is to replace
missing external binaries with shell functions that have the same
name. These functions always fail and keep track of missing
dependencies for a subtest. The result reporting functions later can
check that an external binaries are missing and correctly report SKIP
result instead of FAIL. The primary benefit is that the test cases do
not need to declare their dependencies or be changed in any way.
If symbol-test is built in symbol-hiding with hardcoded g++ invokation,
it's not so easy to pass $(srcdir) which is required to find notmuch.h
when srcdir and builddir are separate directories.
Basic test 'Ensure that all available tests will be run by
notmuch-test' compares all tests that are run with listing of test/
directory. There is a growing list of exceptions for files and
directories which located in the test/ directory but are not tests.
Moreover some (probably buggy) tests do create files in the the test/
directory which may be left behind in case of failure. This makes the
basic test fail.
The patch changes the test to look only for regular executable files.
This makes the exception list much smaller. And since no tests should
create executables in the test/ directory (if there are, they should
be fixed), the basic test should not be affected by failed or
interrupted tests.
The error is easy to miss, because the test passes and stderr is not
printed. But if you run basic tests in verbose mode (./basic
--verbose), you get:
sed: can't read notmuch-test: No such file or directory
The issue is that sed command is given two files: notmuch-test and
$TEST_DIRECTORY/notmuch-test. And there is no notmuch-test file in
the current directory (test/tmp.basic/). The patch just removes the
non-existing file from the sed command.
Previous behaviour was to indent messages in a thread according
to depth by one space per level. This commit tests if setting
notmuch-indent-messages-width to `4' provides a message thread
with four spaces of indentation thread depth.
Previous behaviour was to indent messages in a thread according
to depth by one space per level. In
id:1311028119-50637-1-git-send-email-fgeller@gmail.com Felix
Geller proposed a patch in order to turn indentation off. This
commit tests if instead setting notmuch-indent-messages-width to
`0' does turn off indentation.
Previous behaviour was to indent messages in a thread according
to depth by one space per level. This is still the case with
notmuch-indent-messages-width default value `1'. This test
succeeds if output with default value is same as in "Basic
notmuch-show view in emacs".
Some tests don't break when HUP signal is sent tho those (by
pressing ctrl-c on the terminal). Therefore, the top-level
test script catches the HUP and sends TERM signal to the
started test script.
If mail sending from emacs fails before it has chance to connect
to the smtp-dummy mail server, the opportunistic QUIT message
sending makes smtp-dummy to exit.
Due to 108-character limit in unix domain socket path this change
is required; it is more probable that length of ${TMPDIR:-/tmp} is
shorter than length of path to the current directory of notmuch test
source directory. One can expect to create reasonable-length unix
domain sockets wherever $TMPDIR points to.
The TEST_TMPDIR if first needed to hold dtach's socket (due
to 108-character limit in socket file names). Later it can be
used to hold other temporary files; directory deleted at exit.
Do not redirect test_emacs stderr to /dev/null. Test_emacs uses
emacsclient(1) now and it does not print unwanted messages (like
those from `message') to stderr. But it does print useful
errors, e.g. when emacs server connection fails, given expression
is not valid or undefined function is called.
It is very convenient when C-e (bound to `widget-end-of-line') ignores
trailing spaces inside the search widget. But it only does so if a
widget is not followed by a newline (that is why it works in the saved
search widgets). The patch just adds an invisible space after the
search widget to get the desirable behavior of `widget-end-of-line'.
The extra space is also added to expected results of emacs tests.
Use `previous-single-char-property-change' instead of going
through each character by hand and testing it's visibility. This
fixes `notmuch-show-advance-and-archive' to work for the last
message in thread with hidden signature.
Set SCREENRC and SYSSCREENRC environment variables to "/dev/null"
as suggested by Jim Paris to avoid potential problems with
screen(1) configuration files.
Before the change, emacs run in daemon mode without any visible
buffers. Turns out that this affects emacs behavior in some
cases. In particular, `window-end' function returns `point-max'
instead of the last visible position. That makes it hard or
impossible to implement some tests. The patch runs emacs in a
detached screen(1) session. So that it works exactly as if it
has a visible window.
Note: screen terminates when emacs exits. So the patch does not
introduce new "running processes left behind" issues.
Modify command line argument handling to take a --accumulate flag.
Test for extra arguments beyond the input file.
The --accumulate switch causes the union of the existing and new tags to be
applied, instead of replacing each message's tags as they are read in from the
dump file.
Based on a patch by Thomas Schwinge:
id:"1317317857-29636-1-git-send-email-thomas@schwinge.name"
Flesh out what ``notmuch restore --accumulate'' is supposed to do.
Its tests are currently XFAILed; the functionality will be added in
future patch(es).
Based on a patch by Thomas Schwinge:
id:"1317317811-29540-1-git-send-email-thomas@schwinge.name"
Thanks to Thomas Schwinge for noticing yet another place where quoting
matters. Since the shell translates \. to ., the regex passed to grep
is too generous without the quotes.
The use of [.] is the suggestion of Tomi Ollila.
Several new tests are added, and existing use of test_begin_subtest is
replaced by test_expect_success to catch failing commands in cases where
we execute more than one command.
Based on changes in
id:"1317317811-29540-1-git-send-email-thomas@schwinge.name"
- explain test_expect_equal_file
- remove mention of test_expect_failure, since that function was removed.
Based on id:"1317317811-29540-1-git-send-email-thomas@schwinge.name"
We print an intentionally non-specific message on stderr, since it
isn't clear if there will be some global output file argument to
replace.
We update the test suite atomically, since it relies on having the
same text in two files.
The main motivation here is allow the fast dumping of tag data for
messages having certain tags. In practice it seems too slow to pipe
dump to grep.
All dump-restore tests should be working now, so we update test/dump-restore
accordingly
The idea here is that we want to deprecate the use of arguments to
dump and restore to specify paths, since in particular we want to use
the non-option arguments to dump to form a query.
The first test tests that the notmuch-show-refresh-view function
produces the exact same output for an unmodified show buffer. This
test should pass since the relevant functionality has already been
applied.
The second test tests show refresh for a show buffer that has been
modified by navigation and message visibility toggling. Ideally
refresh-view should preserve this state of the notmuch-show buffer.
Unfortunately it currently does not, so this test is know to be broken
and is marked as such.
There's no reason to output "Non-text part:" lines for parts that are
not leaf nodes, eg. multipart/* and message/rfc822. We fix the text
here to test for their absence. The next patch will fix reply
accordingly.
The setup is useless if gdb is not present, so it doesn't hurt to skip
it. The diff here is huge, but the commit is really just moving most
of the script inside the initial if, and adding an else block to print
a warning.
This addresses atomicity of tag synchronization, the last atomicity
problems in notmuch new. Each message add or remove is wrapped in its
own atomic section, so interrupting notmuch new doesn't lose progress.
This tests notmuch new's ability to recover from arbitrary stopping
failures. It interrupts notmuch new after every database commit and,
on every resulting database snapshot, re-runs notmuch new to
completion and checks that the final database state is invariant.
This means that test_subtest_known_broken needs to be called before
every known broken subtest, which is no different than what is
documented for the test_begin_subtest case.
The assumption is that every test ends up calling either skipping,
calling test_ok_ or test_failure_ and and the latter in turn delegate
to the known_broken versions in the case where
test_subtest_known_broken_ is set.
Human-friendly scenario:
* open a thread where a message which ends with an HTML part is
followed by another message
* make the first message visible
* goto the beginning of the second message (first line, first colon)
* hit "RET"
Result: nothing happens except for "No URL at point" message
Expected result: the second message is shown/hidden
The root cause is that the HTML part has `keymap' text property set.
In particular, "RET" is bound to visit a URL at point. The problem is
that `keymap' property affects the next character following the region
it is set to (see elisp manual [1]). Hence, the first character of
the next message has a keymap inherited from the HTML part.
[1] http://www.gnu.org/software/emacs/elisp/html_node/Special-Properties.html
There is existing support for broken tests. But it is not convenient
to use. The primary issue is that we have to maintain a set of
test_expect_*_failure functions which are equivalent to the normal
test_expect_* counterparts except for what functions are called for
result reporting. The patch adds test_subtest_known_broken function
which marks a subset as broken, making the normal test_expect_*
functions behave as test_expect_*_failure. All test_expect_*_failure
functions are removed. Test_known_broken_failure_ is changed to
format details the same way as test_failure_ does.
Another benefit of this change is that the diff when a broken test is
fixed would be small and nice.
Documentation is updated accordingly.
Update test_emacs documentation in test/README according to the latest
changes in emacs tests. Move the note regarding setting variables
from test/emacs to test/README.
The main goal of this overhaul is to define how message/rfc822 parts
should be handled. message/rfc822 parts should be output in a similar
fashion to the outer message, including some subset of the rfc822
headers. The following decisions about formatting of message/rfc822
parts were made:
The format and content of message/rfc822 parts shall be as similar as
possible to that of full messages. In particular, for formatted
outputs, the "content" of rfc822 part output should include "headers"
and "body" fields).
The "body" field shall include the body of the message.
The "headers" field shall include the following headers, since these
are the ones available from the GMimeMessage:
"From"
"To"
"Cc"
"Subject"
"Date"
However, for the case of --format=raw the raw rfc822 should be output,
including all headers.
A subset of relevant headers shall be output in reply.
The test embedded rfc822 message is also modified to be itself
multipart, so we can more fully test how all sub parts of the message
part are output.
Note added by Committer:
Currently, expect one test (--format=raw --part=3, rfc822 part) to fail.
The test message date, "Tue, 05 Jan 2001 15:43:57 -0000", is not
actually a real date. 05 Jan 2001 was in fact a Friday, not a
Tuesday. Date parsers (such as "date" in coreutils) will return "Fri"
as the day for this string, even if "Tue" is specified.
Also, the time zone "-0000" is actually always returned as "+0000", so
we change that here was well.
This will be relevant for later patches when we begin parsing rfc822
part headers, where gmime returns a parsed date string.
If we do want to test date parsing, we should do that in a separate
test.
There were two "--format=text --part=0" tests. One of them was
supposed to be a test for "--format=text --part=1".
There were also two errant "test_expect_equal_file OUTPUT EXPECTED"
lines, that are removed here.
The feature to show subject changes in the collapsed thread view was
originally added (8ab433607) with an option
(notmuch-show-always-show-subject) to display the subject
for all messages, even when there was no change.
The subsequent commit (4f04d273) changed the sense of the test (or to
and) and the name of the controlling variable
(notmuch-show-elide-same-subject).
But this commit is broken in a few ways:
1. The original definition of notmuch-show-always-show-subject was
left around
But the variable isn't actually used in the code at all, so it
just adds clutter and confusion to the customization interface.
2. The name and description of the controlling variable doesn't
match the implementation
The name suggests that setting the variable to t will cause
repeated subjects to be elided, (suggesting that when it is nil
all subjects will be shown).
However, when the variable is nil, no subjects are shown. So a
correct name for the variable in this sense would be
notmuch-show-subject-changes.
Showing subject changes is a useful feature, and should be on by
default. (We don't want to bury generally useful features behind
customizations that users have to find).
Rather than fixing the name of the variable and changing its default
value, here we remove the condition entirely, such that the feature is
enabled unconditionally.
So both the currently-used variable and the stale definition of the
formerly-used are removed.
Also, the one relevant test-suite result is updated, (showing the
intial subject of a collapsed thread, and no subject display for later
messages that do not change the subject).
eb4cf465 introduces changes which weren't part of the submitted
patch (id:"87liwlip2j.fsf@gmail.com"), presumably made during
resolving merge conflicts.
The first one causes the title of a test to be printed a second time,
in place of the correct title:
PASS Message with .. in Message-Id:
PASS Message with .. in Message-Id:
instead of:
PASS Message with .. in Message-Id:
PASS Sending a message via (fake) SMTP
The second one is simply the insertion of a line break, so no harm there.
This commit reverts both changes, as they were clearly accidental.
Signed-off-by: Pieter Praet <pieter@praet.org>
The sleep was to force the directory's mtime to advance between the
previous notmuch new and the subsequent rm;notmuch new.
The current convention is to use the existing increment_mtime function
for this purpose, (which avoids the test suite being slowed down by
calls to sleep).
Thanks to Austin Clements for noticing this undesired sleep.
Test for bug. Current stemming support for notmuch adds extra terms
to the DB which aren't removed when the file renames are detected.
When folder tags are added to a message, Xapian terms for both XFOLDER
and ZXFOLDER are generated. When one of the filenames are
renamed/removed, only the XFOLDER tags are removed, leaving it possible
for a match on a folder: tag that was previously but is no longer a
match in the maildir.
Before the change, every Emacs test ran in a separate Emacs
instance. Starting Emacs many times wastes considerable time and
it gets worse as the test suite grows. The patch solves this by
using a single Emacs server and emacsclient(1) to run multiple
tests. Emacs server is started on the first test_emacs call and
stopped when test_done is called. We take care not to leave
orphan Emacs processes behind when test is terminated by whatever
reason: Emacs server runs a watchdog that periodically checks
that the test is still running.
Some tests need to provide user input. Before the change, this
was done using echo(1) to Emacs stdin. This no longer works and
instead `standard-input' variable is set accordingly to make
`read' return the appropriate string.
Without this, mail messages delivered by emacs_deliver_message might
not be seen by the next invocation of "notmuch new", (which can lead
to test-suite failures if emacs_deliver_message is fast enough).
Change add_email_corpus, emacs_deliver_message and tests to use
$TEST_DIRECTORY instead of '..'.
This improves the behavior of the usage of --root=<dir>, as the
assumption of what '..' means will usually be incorrect.
Document -root option in README and update valgrind to work with
-root.
Instead of generating auxiliary run_emacs script every time
test_emacs is run, do it once in the beginning of the test.
Also, use absolute paths in the script to make it more robust.
Using `setq' for setting variables in Emacs tests affect other
tests that may run in the same Emacs environment. Currently it
works because each test is run in a separate Emacs instance. But
in the future multiple tests will run in a single Emacs instance.
The patch changes all variables to use `let', so the scope of the
change is limited to a single test.
Few Emacs tests used sed(1) to remove unexpected output in the
beginning to avoid getting confused by messages such as "Parsing
/home/cworth/.mailrc... done". This is no longer needed since
tests are run in a temporary home directory instead of the user's
one. So remove these sed(1) calls.
Before the change, the common Emacs test scheme was to print
buffer content to stdout and redirect it to a file or capture it
in a shell variable. This does not work if we switch to using
emacsclient(1) for running the tests, because you can not print
to the stdout in this case. (Actually, you can print to stdout
from Emacs server, but you can not capture the output on
emacsclient(1)).
The patch introduces new Emacs test auxiliary functions:
`test-output' and `test-visible-output'. These functions are
used to save buffer content to a file directly from Emacs. For
most tests the changes are trivial, because Emacs stdout output
was redirected to a file anyway. But some tests captured the
output in a shell variable and compare it with the expected
output using test_expect_equal. These tests are changed to use
files and test_expect_equal_file instead.
Note: even if we do not switch Emacs tests to emacsclient(1), the
patch makes tests cleaner and is an improvement.
Most test_emacs calls have long arguments that consist of many
expressions. Putting them on a single line makes it hard to read
and produces poor diff when they are changed. The patch puts
every expression in test_emacs calls on a separate line.
