Previously, search started the async notmuch process directly. Now,
it uses `notmuch-start-notmuch'. This simplifies the process sentinel
a bit and means that we no longer have to worry about errors
interleaved with the JSON output.
We also update the tests of Emacs error handling, since the error
output is now separated from the search results buffer.
We now check error handling more carefully in the last test in
test/emacs and we're about to add more error handling tests. (This
was also a strange place for this test, since it had nothing to do
with large search buffers.)
This unifies the part button actions and the underlying part action
functions into single interactive command that simply applies to the
part containing point using the just-added part p-list text property
instead of button properties. Since all part actions can be performed
by applying the appropriate mm function to an mm-handle, this patch
abstracts out the creation of mm handles, making the implementations
of the part commands trivial. This also eliminates our special
handling for part save in favor of using the appropriate mm function.
This necessarily modifies the way we handle the default part button
action, but in a way that does not change the meaning of the
notmuch-show-part-button-default-action defcustom.
Since these commands are no longer specific to buttons, this patch
eliminates the extra metadata stored with each button. This also
eliminates one rather special-purpose macro for a collection of
general purpose part handling utilities.
--stderr=FILE tests were added to test/help-test as it is the one
doing most global option testing. Also, it was simplest to test
this new option using `notmuch help` command.
Presently, the code which finds the parent of a message as it is being
added to the database assumes that the first Message-ID-like substring
of the In-Reply-To header is the parent Message ID. Some mail clients,
however, put stuff other than the Message-ID of the parent in the
In-Reply-To header, such as the email address of the sender of the
parent. This can fool notmuch.
The updated algorithm prefers the last Message ID in the References
header. The References header lists messages oldest-first, so the last
Message ID is the parent (RFC2822, p. 24). The References header is
also less likely to be in a non-standard
syntax (http://cr.yp.to/immhf/thread.html,
http://www.jwz.org/doc/threading.html). In case the References header
is not to be found, fall back to the old behavior.
V2 of this patch, incorporating feedback from Jani and (indirectly)
Austin.
The use of realpath(3) in
commit 58ed67992d
Author: Jani Nikula <jani@nikula.org>
Date: Sun Apr 7 20:15:03 2013 +0300
cli: config: do not overwrite symlinks when saving config file
broke config file save when the file does not exist, which results in
'notmuch setup' always failing to create a new config file.
Fix by checking ENOENT from realpath(3).
Use realpath to canonicalize the config path before writing.
Previously 'notmuch setup' and 'notmuch config set' overwrote the
config file even if it was a symbolic link.
We now have a notmuch_config_is_new() function to query whether a
config was created or not. Change the notmuch_config_open() is_new
parameter into boolean create_new to determine whether the function
should create a new config if one doesn't exist. This reduces the
complexity of the API.
When execution of tests is interrupted by signal coming outside of the
test system itself, output just one line "interrupted by signal <num>"
message to standard output. This distinguishes the case from internal
exit and reduces noise.
Set the variable '$test_subtest_name' in all functions which starts
a new test and use that variable in all functions that output
test results.
Additionally output the latest '$test_subtest_name' in case of
abnormal exit, to avoid confusion.
Instead of checking immediately for the watched process, delay a
minute, or in the case that process-attributes returns nil, for two
minutes. This is intended to cope with the case that
process-attributes is unimplimented, and returns always returns nil.
In this case, the watchdog check is the same as the two minute limit
imposed by timeout.
The TERM environment variable is set to 'dumb' when running tests, but
the original value of it is stored for echoing colors and running emacs
(somewhat interactively) in detached session. Emacs requires some
terminal control sequences to be available for interactive operation.
In case original TERM is (also) 'dumb' (or unset/empty) emacs cannot
run interactively. To fix this problem dtach (and emacs as it's child
process) is run with TERM=vt100 in case original TERM was unset, empty
or 'dumb'. This way there is a chance to run emacs tests with different
user terminals and potentially find problems there.
This test also serves as documentation of the quoting
requirements. The comment lines are so that it exactly matches the man
page. Nothing more embarrassing than having an example in the man page
fail.
The (now fixed) bug that this test revealed is that unquoted
message-ids with whitespace or other control characters in them are
split into several tokens by the Xapian query parser.
This is based on the similar test for notmuch restore, but the parser
in batch tagging mode is less tolerant of a few cases, in particular
those tested by illegal_tag.
We recently switched to popping up a buffer to report CLI errors, but
this was too intrusive, especially for transient errors and especially
since we made fewer things ignore errors. This patch changes this to
display a basic error message in the minibuffer (using Emacs' usual
error handling path) and, if there are additional details, to log
these to a separate error buffer and reference the error buffer from
the minibuffer message. This is more in line with how Emacs typically
handles errors, but makes the details available to the user without
flooding them with the details.
Given this split, we pare down the basic message and make it more
user-friendly, and also make the verbose message even more detailed
(and more debugging-oriented).
This switches the new batch-tag format away from using a home-grown
hex-encoding scheme for message IDs in the dump to simply using Xapian
queries with Xapian quoting syntax.
This has a variety of advantages beyond presenting a cleaner and more
consistent interface. Foremost is that it will dramatically simplify
the quoting for batch tagging, which shares the same input format.
While the hex-encoding is no better or worse for the simple ID queries
used by dump/restore, it becomes onerous for general-purpose queries
used in batch tagging. It also better handles strange cases like
"id:foo and bar", since this is no longer syntactically valid.
When we switch to using regular Xapian queries in the dump format, \n
will cause problems, so we disallow it. Specially, while Xapian can
quote and parse queries containing \n without difficultly, quoted
queries containing \n still span multiple lines, which breaks the
line-orientedness of the dump format. Strictly speaking, we could
still round-trip these, but it would significantly complicate restore
as well as scripts that deal with tag dumps. This complexity would
come at absolutely no benefit: because of the RFC 2822 unfolding
rules, no amount of standards negligence can produce a message with a
message ID containing a line break (not even Outlook can do it!).
Hence, we simply disallow it.
This patch corrects several undesirable behaviours:
1) Empty files were not detected, leading to buffer read overrun.
2) An initial blank line cause restore to silently abort
3) Initial comment line caused format detection to fail
notmuch_json_show_sanitize replaced "filename" field values even in part
structures, where the value is predictable. Make it only normalize the
filename value if it is an absolute path (begins with slash), which is
true of the Maildir filenames that were intended to be normalized away.
This slightly changes the output of an existing test since we now
report non-zero exits with a pop-up buffer instead of at the end of
the search results.
* test/emacs:
- Rename subtests "{Add,Remove} tag from notmuch-show view" to
"notmuch-show: {add,remove} single tag {to,from} single message"
to be consistent with the following tests.
- New subtest "notmuch-show: add multiple tags to single message":
`notmuch-show-add-tag' ("+") can add multiple tags to a message.
- New subtest "notmuch-show: remove multiple tags from single message":
`notmuch-show-remove-tag' ("-") can remove multiple tags from a message.
We want to test both that error/warning messages are generated when
they should be, and not generated when they should not be. This varies
between restore and batch tagging.
These one need the completed functionality in notmuch-restore. Fairly
exotic tags are tested, but no weird message id's.
We test each possible input to autodetection, both explicit (with
--format=auto) and implicit (without --format).
The quoting for ${SEARCH} is broken when it's supposed to be '*', and
it seems tricky to get it right. Just drop the variable and use '*'
directly. Before this, none of the messages ever matched, and the test
was comparing zeros.
test_expect_equal_json uses json.tool from the system Python. While
Python 2 wasn't picky about the encoding of stdin, Python 3 decodes
stdin strictly according to the environment. Since we set LC_ALL=C
for the tests, Python 3's json.tool was assuming stdin would be in
ASCII and aborting when it couldn't decode the UTF-8 characters from
some of the JSON tests. This patch sets the PYTHONIOENCODING
environment variable to utf-8 when invoking json.tool to override
Python's default encoding choice.
Without this change, GCC complains as follows:
gcc test/random-corpus.o test/database-test.o notmuch-config.o command-line-arguments.o lib/libnotmuch.a util/libutil.a parse-time-string/libparse-time-string.a -o test/random-corpus -lgmime-2.6 -lgio-2.0 -lgobject-2.0 -lglib-2.0 -Wl,-rpath,/usr/lib -ltalloc -lxapian
/usr/bin/ld: lib/libnotmuch.a(database.o): undefined reference to symbol '_ZNSs4_Rep10_M_destroyERKSaIcE@@GLIBCXX_3.4'
/usr/bin/ld: note: '_ZNSs4_Rep10_M_destroyERKSaIcE@@GLIBCXX_3.4' is defined in DSO /usr/lib/libstdc++.so.6 so try adding it to the linker command line
/usr/lib/libstdc++.so.6: could not read symbols: Invalid operation
collect2: error: ld returned 1 exit status
make: *** [test/random-corpus] Error 1
We demonstrate the current notmuch restore parser being confused by
message-id's and tags containing non alpha numeric characters
(particularly space and parentheses are problematic because they are
not escaped by notmuch dump).
We save the files as hex escaped on disk so that terminal emulators
will not get confused if the test fails (as we mostly expect it to do).
Initial use case is testing dump and restore, so we only have
message-ids and tags.
The message ID's are nothing like RFC compliant, but it doesn't seem
any harder to roundtrip random UTF-8 strings than RFC-compliant ones.
Tags are UTF-8, even though notmuch is in principle more generous than
that.
updated for id:m2wr04ocro.fsf@guru.guru-group.fi
- talk about Unicode value rather some specific encoding
- call talloc_realloc less times
Initially, provide a way to create "stub" messages in the notmuch
database without corresponding files. This is essentially cut and
paste from lib/database.cc. This is a seperate file since we don't
want to export these symbols from libnotmuch or bloat the library with
non-exported code.
Previously, the test framework generated a variable name for each
external prereq as a poor man's associative array. Unfortunately,
prereqs names may not be legal variable names, leading to
unintelligible bash errors like
test_missing_external_prereq_emacsclient.emacs24_=t: command not found
Using proper associative arrays to track prereqs, in addition to being
much cleaner than generating variable names and using grep to
carefully construct unique string lists, removes restrictions on
prereq names.
Previously, if a test script aborted (e.g., because it passed too few
arguments to a test function), the test driver loop would simply
continue on to the next test script and the final results would
declare that everything passed (except that the test count would look
suspiciously low, but maybe you just misremembered how many tests
there were).
Now, if a test script exits with a non-zero status and did not produce
a final results file, we propagate that failure out of the driver loop
immediately.
To keep this simple, this patch removes the PID from the test-results
file name. This PID was inherited from the git test system and seems
unnecessary, since the file name already includes the name of the test
script and the test-results directory is created anew for each run.
And require that if TEST_EMACS is specified, so is TEST_EMACSCLIENT.
Previously, the test framework always used "emacsclient", even if the
Emacs in use was overridden by TEST_EMACS. This causes problems if
both Emacs 23 and Emacs 24 are installed, the Emacs 23 emacsclient is
the system default, but TEST_EMACS is set to emacs24. Specifically,
with an Emacs 24 server and an Emacs 23 client, emacs tests that run
very quickly may produce no output from emacsclient, causing the test
to fail.
The Emacs server uses a very simple line-oriented protocol in which
the client sends a request to evaluate an expression and the server
sends a request to print the result of evaluation. Prior to Emacs bzr
commit 107565 on March 11th, 2012 (released in Emacs 24.1), if
multiple commands were sent to the emacsclient between when it sent
the evaluation command and when it entered its receive loop, it would
only process the first response command, ignoring the rest of the
received buffer. This wasn't a problem with the Emacs 23 server
because it sent only the command to print the evaluation result.
However, the Emacs 24 server first sends an unprompted command
specifying the PID of the Emacs server, then processes the evaluation
request, then sends the command to print the result. If the
evaluation is fast enough, it can send both of these commands before
emacsclient enters the receive loop. Hence, if an Emacs 24 server is
used with an Emacs 23 emacsclient, it may miss the response printing
command, ultimately causing intermittent notmuch test failures.
Previously, many tests in emacs-subject-to-filename didn't quote the
$output argument to test_expect_equal. As a result, if $output was
empty, test_expect_equal would be passed only one argument and would
abort the entire test script. By quoting the argument, we ensure
test_expect_equal will always receive two arguments.
We now test for user ignore patterns before attempting to determine if
a directory entry is itself a directory. As a result, we no longer
abort for broken symlinks if the user has explicitly ignored them.
This fixes the test added in the previous patch. It also slightly
changes the debug output checked by another test of ignores.
The macro with-current-notmuch-show-message executes command
`notmuch show --format=raw id:...` which just outputs the contents
of the mail file verbatim (into temporary buffer). In case e.g. utf-8
locale is used the temporary buffer has buffer-file-coding-system as
utf-8. In this case Emacs converts the data to multibyte format, guessing
that input is in utf-8.
However, the "raw" (MIME) message may contain octet data in any other
8bit format, and as no (MIME-)content spesific handling to the message
is done at this point, conversion to other formats may lose information.
By setting coding-system-for-read 'no-conversion drops the conversion part
and makes this handle input as notmuch-get-bodypart-internal() does.
This marks the broken test in previous change fixed.
Previously, we would treat multi-message mboxes as one giant email,
which, besides the obvious incorrect indexing, often led to
out-of-memory errors for archival mboxes. Now we explicitly reject
multi-message mboxes. For historical reasons, we retain support for
single-message mboxes, but official deprecate this behavior.
This test is currently broken. Note that its brokenness cascades and
causes the next test to fail as well (because notmuch incorrectly
indexes the mbox file).
There are currently 45 TESTS scripts. 36 of those load
test-lib.sh using '. ./test-lib.sh' and 9 '. test-lib.sh'.
In latter case test-lib.sh is first searched from directories
in PATH (posix) and then from current directory (bash feature).
Changed the 9 files to execute '. ./test-lib.sh'. The test-lib.sh
should never be loaded from directory in PATH.
Previously, this would simply indicate that the grep failed without
any indication of the Emacs output it failed on. Now we take
advantage of the test framework's handling of stdout to display the
incorrect Emacs output if the test fails.
It seems we have never tested the case that restore --accumulate
actually adds tags. I noticed this when I started optimizing and no
tests failed.
The bracketing with "restore --input=dump.expected" are to make sure
we start in a known state, and we leave the database in a known state
for the next test.
The test designed to exercise Emacs' rendering of HTML emails
containing images inadvertently assumed w3m was available under Emacs
23. The real point of this test was to check that Emacs 24's shr
renderer didn't crash when given img tags, so use shr if it's
available, html2text otherwise (which is built in), and do only a
simple sanity check of the result.
This regexp agrees with Xapian query syntax much more closely, though
we specifically disallow various cases that would be confusing in the
context of an email body (e.g., punctuation at the end of an id: link
is not considered part of the id: link because it's probably part of
the surrounding text).
In particular, this handles id: links that are not surrounded by
quotes much better, which stash is much more likely to generate now
that we don't quote id's that don't need to be quoted. It also
handles quoted id: links better.
We update the buttonization test to reflect the new pattern.
This matches the current behavior of the buttonizer, so it passes, but
many of these cases are not what you'd want (and some of them aren't
even valid Xapian queries). The next patch will fix the handling of
these cases and update the test.
Over time, maintaining this very long regex has become irritating,
especially when resolving conflicts.
This patch replaces the call to sed with multiple extra arguments to
find. Since each test binary is now on it's own line, this should
make resolving conflicts easier.
Test the date/time parser module directly, independent of notmuch,
using the parse-time test tool.
Credits to Michal Sojka <sojkam1@fel.cvut.cz> for writing most of the
tests.
Add a smoke testing tool to support testing the date/time parser
module directly and independent of the rest of notmuch.
Credits to Michal Sojka <sojkam1@fel.cvut.cz> for the stdin parsing
idea and consequent massive improvement in testability.
Currently, we only properly escape stashed id queries, but there are
other places where the Emacs UI constructs queries for boolean terms.
Since this escaping function is meant to be used in other places, it
avoids escaping strings that don't need escaping.
This disallows adding empty tags, since nothing but confusion follows
in their wake, and disallows adding tags that begin with "-" because
they are also confusing, the tag "-" is impossible to remove using the
CLI, and because the syntax for removing such tags conflicts with long
argument syntax.
This does not place any restrictions on what tags can be removed, as
that would make it difficult for people who have the misfortune of
already having malformed tags to remove these tags.
Although messages are created in a particular order, it seems that
when they are created on a tmpfs, they do not always come back in the
same order, leading to the same files being ignored but being output
in a different order. This causes the test to fail because the outputs
being compared are the same.
Fix the failures by sorting the output of notmuch --debug and
comparing this to a hand-sorted version of its output.
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
The use of --background option (instead of shell '&') ensures that
smtp-dummy is listening its server socket until execution of shell
script can continue, thus the client will always have socket where
to connect.
smtp-dummy outputs smtp_dummy_pid variable in shell assignment format;
eval'ing that output makes that variable available for the shell.
As the smtp-dummy instance is no longer child process of the script
the SIGKILL signal sent to it will ensure it is going away in case
the mail sender fails to connect to smtp-dummy.
When shell executes background process using '&' the scheduling of
that new process is arbitrary. It could be that smtp-dummy doesn't
get execution time to listen() it's server socket until some other
process attempts to connect() to it. The --background option in
smtp-dummy makes it to go background *after* it started to listen
its server socket.
When --background option is used, the line "smtp_dummy_pid='<pid>'"
is printed to stdout from where shell can eval it.
Demonstrates that *every* file/directory which matches one of the values
in 'new.ignore' will be ignored, independent of its depth/location in
the mail store.
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Obviates the need to create a 'NOTMUCH_NEW' clone which runs
'notmuch new --debug'. This will be used in a later patch.
Doesn't cause any issues for other tests.
* test/emacs:
- New subtest "notmuch-show: collapse all messages in thread":
`notmuch-show-open-or-close-all' with prefix arg ("C-u M-RET")
collapses all messages in thread.
- New subtest "notmuch-show: uncollapse all messages in thread":
`notmuch-show-open-or-close-all' without prefix arg ("M-RET")
uncollapses all messages in thread.
* test/emacs:
- New subtest "notmuch-show: show message headers":
Setting `notmuch-message-headers-visible' to t causes all headers
defined in `notmuch-message-headers' to be shown.
- New subtest "notmuch-show: hide message headers":
Setting `notmuch-message-headers-visible' to nil causes all headers
defined in `notmuch-message-headers' to be hidden.
("Subject:" may be an exception; See the use of `headers-start' in
`notmuch-show-insert-msg')
- New subtest "notmuch-show: hide message headers (w/ notmuch-show-toggle-headers)":
Setting `notmuch-message-headers-visible' to t causes all headers
defined in `notmuch-message-headers' to be shown, but they can be
hidden for the current message by running `notmuch-show-toggle-headers'.
This requires changing the contents of the crypto tests, as one thread
that was marked read by the earlier tests in test/emacs is no longer
marked read.
This moves tests for:
- 09d19ac "test: emacs: toggle eliding of non-matching messages in
`notmuch-show'", which should have actually read: "test: emacs:
toggle processing of cryptographic MIME parts in `notmuch-show'".
See commit 19ec74c5.
- 5ea1dbe "test: emacs: toggle eliding of non-matching messages in
`notmuch-show'"
- 345faab "test: emacs: toggle thread content indentation in
`notmuch-show'"
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Since $TEST_DIRECTORY is an absolute path, any filenames generated
with it will be complete paths. Only use the basename to generate
suffixes for filenames.
Signed-off-by: Ethan Glasser-Camp <ethan@betacantrips.com>
Most Emacs tests end with a call to (test-output), which saves the
buffer to a filed called OUTPUT. Previously, if the test code failed
with an exception before this call, the test framework would then
compare against the OUTPUT file from the last Emacs test, resulting in
confusing diffs.
This requires one tweak to an emacs test that made two calls to
test_emacs and expected an OUTPUT file from the first call. We simply
reverse the order of the test_emacs calls.
On FreeBSD, and probably anywhere else someone installed xapian to
some other prefix, we need to use XAPIAN_LDFLAGS to make the linker can
actually find libxapian.
Before the change, test_expect_equal_file() function treated the first
argument as "actual output file" and the second argument as "expected
output file". When the test fails, the files are copied for later
inspection. The first files was copied to "$testname.output" and the
second file to "$testname.expected". The argument order for
test_expect_equal_file() is often wrong which results in confusing
diff output and incorrectly named files.
The patch solves the issue by changing test_expect_equal_file() to
treat arguments just as two files, without any special properties
(like "actual" and "expected"). The file names for copying is now
based on the given file name: "$testname.$file1" and
"$testname.$file2". E.g. if test_expect_equal_file() is called with
"OUTPUT" and "EXPECTED", the copied files can be named
"emacs.1.OUTPUT" and "emacs.1.EXPECTED".
The down side of this approach is that diff argument order depends on
test_expect_equal_file() argument order. So sometimes we get diff
from expected to actual results, and sometimes the other way around.
But the files are always named correctly.
The behaviour of "emacsclient --eval nil" changed from emacs23 to
emacs24, and in emacs24 it prints 'nil' rather than an empty string.
(format "%S" foo) produces a sexpr form of foo, and is consistent
between the two versions.
The version of message.el in emacs24 omits the charset=us-ascii,
causing the current version of this test to fail. With this patch, we
accept either option. According to RFC 2046, they are semantically
equivalent.
When running emacs tests using emacs 23.1.1 the tests block (until timeout)
when emacs function (notmuch-test-wait) is called.
There is an emacs bug #2930 titled:
23.0.92; `accept-process-output' and `sleep-for' do not run sentinel
It seems this is present in emacs 23.1.
Calling list-processes after accept-process-output seems work around
this problem; in case Emacs version is 23.1 a defadvice is activated
to do just that.
notmuch-test-wait called sleep-for in a loop to wait unconditionally 0.1
seconds while waiting for process to exit.
accept-process-output returns as soon as there is any data available
from process, so using it avoids unnecessary fixed delays.
Both of these functions run process sentinels.
The string function in a sprinter may be called with a NULL string
pointer (eg if a header is absent). This causes a segfault. We fix
this by checking for a null pointer in the string functions and update
the sprinter documentation.
At the moment some output when format=text is done directly rather than
via an sprinter: in that case a null pointer is passed to printf or
similar and a "(null)" appears in the output. That behaviour is not
changed in this patch.
The syntax --output=filename is a smaller change than deleting the
output argument completely, and conceivably useful e.g. when running
notmuch under a debugger.
Format canonicalization of JSON output is no longer necessary, so
remove it. Value canonicalization (e.g., normalizing thread IDs) is
still necessary, so all of the sanitization functions remain.
Previously, we used a variety of ad-hoc canonicalizations for JSON
output in the test suite, but were ultimately very sensitive to JSON
irrelevancies such as whitespace. This introduces a new test
comparison function, test_expect_equal_json, that first pretty-prints
*both* the actual and expected JSON and the compares the result.
The current implementation of this simply uses Python's json.tool to
perform pretty-printing (with a fallback to the identity function if
parsing fails). However, since the interface it introduces is
semantically high-level, we could swap in other mechanisms in the
future, such as another pretty-printer or something that does not
re-order object keys (if we decide that we care about that).
In general, this patch does not remove the existing ad-hoc
canonicalization because it does no harm. We do have to remove the
newline-after-comma rule from notmuch_json_show_sanitize and
filter_show_json because it results in invalid JSON that cannot be
pretty-printed.
Most of this patch simply replaces test_expect_equal and
test_expect_equal_file with test_expect_equal_json. It changes the
expected JSON in a few places where sanitizers had placed newlines
after commas inside strings.
These extra directories cause problems for building on Debian
twice in a row.
In order to remove directories, we need to us "rm -rf" instead of
"rm -f". So now we should be extra careful what we add to the
variable CLEAN.
This patch switches from the current ad-hoc printer to the structured
formatters in sprinter.h, sprinter-text.c and sprinter-json.c.
The JSON tests are changed slightly in order to make them PASS for the
new structured output formatter.
The text tests pass without adaptation.
The JSON format eliminates the complex escaping issues that have
plagued the text search format. This uses the incremental JSON parser
so that, like the text parser, it can output search results
incrementally.
This slows down the parser by about ~4X, but puts us in a good
position to optimize either by improving the JSON parser (evidence
suggests this can reduce the overhead to ~40% over the text format) or
by switching to S-expressions (evidence suggests this will more than
double performance over the text parser). [1]
This also fixes the incremental search parsing test.
This has one minor side-effect on search result formatting.
Previously, the date field was always padded to a fixed width of 12
characters because of how the text parser's regexp was written. The
JSON format doesn't do this. We could pad it out in Emacs before
formatting it, but, since all of the other fields are variable width,
we instead fix notmuch-search-result-format to take the variable-width
field and pad it out. For users who have customized this variable,
we'll mention in the NEWS how to fix this slight format change.
[1] id:"20110720205007.GB21316@mit.edu"
This advises the search process filter to make it process one
character at a time in order to test the pessimal case for incremental
search output parsing.
The text parser fails this test because it gets tricked into thinking
a parenthetical remark in a subject is the tag list.
There didn't seem to be these basic tests for --format=text,
as there are for --format=json. These are just the tests from
the `json' script, with adjusted expected outputs.
Previously, notmuch new only synchronized maildir flags to tags for
files with a maildir "info" part. Since messages in new/ don't have
an info part, notmuch would ignore them for flag-to-tag
synchronization.
This patch makes notmuch consider messages in new/ to be legitimate
maildir messages that simply have no maildir flags set. The most
visible effect of this is that such messages now automatically get the
unread tag.
Currently, notmuch new only synchronizes maildir flags to tags for
files that have an "info" part. However, in maildir, new mail doesn't
gain the info part until it moves from new/ to cur/. Hence, even
though mail in new/ doesn't have an info part, it is still a maildir
message and thus has maildir flags (though none of them set).
The most visible effect of not synchronizing maildir flags for
messages in new/ is that newly delivered messages don't get the unread
tag (unless it is assigned by some other mechanism, like new.tags).
This patch does *not* modify the test for messages in cur/ that do not
have an "info" part. Unlike a message in new/, a message in cur/
without an info part is no longer a maildir message, and thus
shouldn't be subject to maildir flag synchronization.
Add tests for picking up user's From address from fallback headers
Envelope-To, X-Original-To, and Delivered-To.
Signed-off-by: Jani Nikula <jani@nikula.org>
This moves our logic to get a file's type into one function. This has
several benefits: we can support OSes and file systems that do not
provide dirent.d_type or always return DT_UNKNOWN, complex
symlink-handling logic has been replaced by a simple stat fall-through
in one place, and the error message for un-stat-able file is more
accurate (previously, the error always mentioned directories, even
though a broken symlink is not a directory).
Now that notmuch_database_find_message_by_filename works on read-only
databases, remove the workaround that disabled it on read-write
databases.
This also adds a regression test for find_message_by_filename.
The customization widget referred to a non-existing function
`notmuch-hello-insert-query-list'. The patch changes it to the
correct one - `notmuch-hello-insert-searches'. The relevant test is
fixed now.
The tests use default values from customization widgets to make sure
that these customization widgets work (at least on basic level).
The custom queries section test is currently broken.
The output of the HTML reply test in the emacs suite can vary
depending on which HTML renderers are installed on the machine running
the tests. The renderer that is always available is emacs's builtin
html2text function. In order to get consistency, force the test to use
html2text even if other renderers are available.
Quote non-text parts nicely by displaying them with mm-display-part
before calling message-cite-original to quote them. HTML-only emails
can now be quoted correctly. We re-use some code from notmuch-show
(notmuch-show-mm-display-part-inline), which has been moved to
notmuch-lib.el.
Mark the test for this feature as not broken.
With the latest reply infrastructure, we should be able to nicely
quote HTML-only emails. But currently emacs quotes the raw HTML
instead of parsing it first. This commit adds a test for this case.
This test currently marked as broken.
It has been a long-standing issue that notmuch_database_open doesn't
return any indication of why it failed. This patch changes its
prototype to return a notmuch_status_t and set an out-argument to the
database itself, like other functions that return both a status and an
object.
In the interest of atomicity, this also updates every use in the CLI
so that notmuch still compiles. Since this patch does not update the
bindings, the Python bindings test fails.
Add a command to list all configuration items with their associated
values.
One use is as follows: a MUA may prefer to store data in a central
notmuch configuration file so that the data is accessible across
different machines, e.g. an addressbook. The list command helps
to implement features such as tab completion on the keys.
This patch removes trailing spaces in notmuch-hello view.
A side effect of this change is that tag/query buttons no longer
include a space at the end. This means that pressing RET when the
point is at the first character after the tag/query button no longer
works (note that this is the standard behavior for buttons). We may
change this behavior in the future (without adding trailing spaces
back) if people would find this change inconvenient.
Systematically test the exclude options for search. Also move the
search existing exclude tests into the new test. There is some overlap
between the two sets of tests but many of the existing ones are there
because they triggered bugs in the past so I have kept them to ensure
coverage.
Move the option --no-exclude to the --exclude= scheme. Since there is
no way to flag messages only true and false are implemented. Note
that, for consistency with other commands, this is implemented as a
keyword option rather than a boolean option.
In the new reply code, the References header gets inserted by
message.el using a function called message-shorten-references. Unlike
all the other header-inserting functions, it doesn't put a newline
after the header, causing the next header to end up on the same
line. In our case, this header happened to be User-Agent, so it's hard
to notice. This is probably a bug in message.el, but we need to work
around it.
This fixes the problem by wrapping message-shorten-references in a
function that inserts a newline after if necessary. This should
protect against the message.el bug being fixed in the future.
By default, emacs hides the User-Agent and References headers when
composing mail. This is a good thing for users, but a bad thing for
testing, since we can create ugly or invalid headers and not have it
show up in the tests.
By setting message-hidden-headers to an empty list, we force emacs to
show all the headers, so we can check that they're correct. Users
won't see this, but it will let us catch future bugs.
As a side-effect, this breaks all the reply tests, since there is a
bug with the References and User-Agent headers, fixed in the next commit.
Bug 1: Replying from alternate addresses
----------------------------------------
The reply code was inconsistent in its use of symbols and strings for
header names being passed to message.el functions. This caused the
From header to be lookup up incorrectly, causing an additional From
header to be added with the user's primary address instead of the
correct alternate address.
This is fixed by using symbols everywhere, i.e. never using strings
for header names when interacting with message.el.
This change also removes our use of `mail-header`, since we don't use
it anywhere else, and using assq makes it clear how the header lists
are expected to work.
Bug 2: Duplicate headers in emacs 23.2
--------------------------------------
The message.el code in emacs 23.2 assumes that header names will
always be passed as symbols, so our use of strings caused
problems. The symptom was that on 23.2 (and presumably on earlier
versions) the reply message would end up with two of some headers.
Converting everything to symbols also fixes this issue.
Since the recent reply changes were pushed, there has been a bug that
causes emacs to always reply from the primary address, even if the
JSON or default CLI reply output uses an alternate address.
This adds two tests to the emacs test library based on the two "Reply
form..." tests in the reply test library. One is currently marked
broken.
This adds a lib function to turn a message ID into a properly escaped
message ID query and uses this function wherever we previously
hand-constructed ID queries. Wherever this new function is used,
documentation has been clarified to refer to "id: queries" instead of
"message IDs".
This fixes the broken test introduced by the previous patch.
To simplify code, keep all tagging operations in a single array
instead of separate add and remove arrays. Apply tag changes in the
order specified on the command line, instead of first removing and
then adding the tags.
This results in a minor functional change: If a tag is both added and
removed, the last specified operation is now used. Previously the tag
was always added. Change the relevant test to reflect the new
behaviour.
Signed-off-by: Jani Nikula <jani@nikula.org>
The current behaviour is that regardless of the order in which the
addition and removal of a tag are specified, the tag is added.
Signed-off-by: Jani Nikula <jani@nikula.org>
Use the new JSON reply format to create replies in emacs. Quote HTML
parts nicely by using mm-display-part to turn them into displayable
text, then quoting them with message-cite-original. This is very
useful for users who regularly receive HTML-only email.
Use message-mode's message-cite-original function to create the
quoted body for reply messages. In order to make this act like the
existing notmuch defaults, you will need to set the following in
your emacs configuration:
message-citation-line-format "On %a, %d %b %Y, %f wrote:"
message-citation-line-function 'message-insert-formatted-citation-line
The tests have been updated to reflect the (ugly) emacs default.
This new JSON format for replies includes headers generated for a
reply message as well as the headers of the original message. Using
this data, a client can intelligently create a reply. For example, the
emacs client will be able to create replies with quoted HTML parts by
parsing the HTML parts.
This adds three tests for --output=messages searches. One test is for
the case when one exclude tag does not occur in the Xapian
database. This triggers a Xapian bug in some cases and causes the
whole exclusion to fail. The next commit avoids this bug.
The tests for the exclude code in search and count use the line
notmuch config set search.exclude_tags = deleted
which actually sets the exclude tags to be "=" and "deleted". Remove
the "=" from this line.
Before the change, messages generated by generate_message() used "Test
message #N" for default subject where N is the generated messages
counter. Since message subject is commonly present in expected
results, there is a chance of breaking other tests when a new
generate_message() call is added. The patch changes default subject
value for generated messages to subtest name if it is available. If
subtest name is not available (i.e. message is generated during test
initialization), the old default value is used (in this case it is
fine to have the counter in the subject).
Another benefit of this change is a sane default value for subject in
generated messages, which would allow to simplify code like:
test_begin_subtest "test for a cool feature"
add_message [subject]="message for test for a cool feature"
Before the change, the first subtest in raw format tests just
generated messages and checked that they are added successfully. This
is not really a raw format test, it is creating of environment
required for other subtests to run. The patch removes the first
subtest from raw and replaces it with bare add_message calls, similar
to how it is done in other tests.
TODO: we should check that test environment was created successfully.
Currently, many tests do add_message(), notmuch new and other calls
without checking the results. We should come up with a general
solution for this, i.e. if any command during test initialization
fails, all tests should be skipped with appropriate error message.
This is fully compatible for root and leaf parts, but now has proper
support for interior parts. This requires some design decisions that
were guided by what I would want if I were to save a part.
Specifically:
- Leaf parts are printed without headers and with transfer decoding.
This is what makes sense for saving attachments. (Furthermore, the
transfer decoding is necessary since, without the headers, the
caller would not be able to interpret non-transfer-decoded output.)
- Message parts are printed with their message headers, but without
enclosing part headers. This is what makes sense for saving a
message as a whole (which is a message part) and for saving attached
messages. This is symmetric for whole messages and for attached
messages, though we special-case the whole message for performance
reasons (and corner-case correctness reasons: given malformed input,
GMime may not be able to reproduce it from the parsed
representation).
- Multipart parts are printed with their headers and all child parts.
It's not clear what the best thing to do for multipart is, but this
was the most natural to implement and can be justified because such
parts can't be interpreted without their headers.
As an added benefit, we can move the special-case code for part 0 into
the raw formatter.
Previously, there was only one CRLF between the terminating boundary
of the embedded multipart/alternative and the boundary of the
containing multipart. However, according the RFC 1341, 7.2.1:
The boundary must be followed immediately either by another CRLF and
the header fields for the next part, or by two CRLFs, in which case
there are no header fields for the next part
and
The CRLF preceding the encapsulation line is considered part of the
boundary so that it is possible to have a part that does not end
with a CRLF (line break).
Thus, there must be *two* CRLFs between these boundaries: one that
ends the terminating boundary and one that begins the enclosing
boundary.
While GMime accepted the message we had before, it could not produce
such a message.
gmime-2.6 had a bug [1] which made it impossible to tell why a signature
verification failed when the signer key was unavailable (empty "sigstatus" field
in the JSON output). Since 00b5623d the corresponding test is marked as broken
when using gmime-2.6 (2.4 is fine).
This bug has been fixed in gmime 2.6.5, which is now the minimal gmime-2.6
version required for building notmuch (gmime-2.4 is still available). As a
consequence the version check in test/crypto can be removed.
[Added by db]
Although less unambigously a bug, Gmime 2.6 prior to 2.6.7 also was
more strict about parsing, and rejected messages with initial "From "
headers. This restriction is relaxed in [2]. For reasons explained in [3],
we want to keep this more relaxed parsing for now.
[1] https://bugzilla.gnome.org/show_bug.cgi?id=668085
[2] http://git.gnome.org/browse/gmime/commit/?id=d311f576baf750476e06e9a1367a2dc1793ea7eb
[3] id:"1331385931-1610-1-git-send-email-david@tethera.net"
notmuch show outputs the exclude flag so many tests using notmuch
show failed. This commit adds "excluded:0" or "excluded: false" to
the expected outputs. After this commit there should be no failing
tests.
notmuch-search.c now returns all matching threads even if it the
match is a search.tag_excluded message (but with a mark indicating
this). Update the test to reflect this.
This has three ramifications:
- Blank To and Cc headers are no longer output for messages.
- Dates are now canonicalized for messages, which means they always
have a day of the week and GMT is printed +0000 (never -0000)
- Invalid From message headers are handled slightly differently, since
they get parsed by GMime now instead of notmuch.
Previously, top-level message headers were printed as Subject, From,
To, Date, while embedded message headers were printed From, To,
Subject, Date. This makes both cases use the former order and updates
the tests accordingly.
Emails that are encoded differently than as ASCII or UTF-8 are not
indexed properly by notmuch. It is not possible to search for non-ASCII
words within those messages.
Consensus seems to be that people prefer that refreshing show buffers
retains state by default, rather than resetting it by default. This
turns out to be the case in the code, as well. In fact, there's even
a test for this that's been marked broken for several months, which
this patch finally gets to mark as fixed.
* emacs/notmuch-show.el
(notmuch-show-stash-mlarchive-link-alist):
New defcustom of type `alist' (key = name, value = URI),
containing Mailing List Archive URI's for searching by Message-Id.
(notmuch-show-stash-mlarchive-link-default):
New defcustom, default MLA to use when `notmuch-show-stash-mlarchive-link'
received no user input whatsoever. Available choices are generated using
the contents of `notmuch-show-stash-mlarchive-link-alist'.
(notmuch-show-stash-map):
Added keybinds "l" and "L" for `notmuch-show-stash-mlarchive-link'
respectively `notmuch-show-stash-mlarchive-link-and-go'.
(notmuch-show-stash-mlarchive-link):
New function, stashes a URI pointing to the current message at one
of the MLAs configured in `notmuch-show-stash-mlarchive-link-alist'.
Prompts user with `completing-read' if not provided with an MLA key.
(notmuch-show-stash-mlarchive-link-and-go):
New function, uses `notmuch-show-stash-mlarchive-link' to
stash a URI, and then visits it using the browser configured
in `browse-url-browser-function'.
Based on original work [1] by David Edmondson <dme@dme.org>.
[1] id:"1327397873-20596-1-git-send-email-dme@dme.org"
`notmuch-show-stash-mlarchive-link' stashes a URI pointing to the current message
at one of the MLAs configured in `notmuch-show-stash-mlarchive-link-alist'.
Marked as "broken": fixed in next commit.
When tests are skipped due to missing prereqs, those prereqs are only
displayed when running with the `--verbose' option. This is essential
information when troubleshooting, so always send to stdout.
A new configuration key 'new.ignore' is used to determine which
files and directories user wants not to be scanned as new mails.
Mark the corresponding test as no longer broken.
This work merges my previous attempts and Andreas Amann's work
in id:"ylp7hi23mw8.fsf@tyndall.ie"
Files and directories which are specified in 'new.ignore' in the
config file shouldn't be indexed nor reported by `notmuch new'.
This is basically Pieter's work with Austin's comments addressed.
This makes the text formatter take advantage of the new code
structure. The previously duplicated header logic is now unified,
several things that we used to compute repeatedly across different
callbacks are now computed once, and the code is simpler overall and
32% shorter.
Unifying the header logic causes this to format some dates slightly
differently, so the two affected test cases are updated.
After the recent tagging operations changes, functions bound to "+"
and "-" in notmuch-search and notmuch-show views always read input
from the minibuffer. Use kbd macros instead of calling them directly.
`Notmuch-wash-region-to-button' is the function that creates hidden
regions with buttons for signatures, citations and original messages.
Before the change, it did not work correctly if the to-be-hidden
region started at the beginning of a message: the visibility toggle
button was hidden as well. The patch fixes this. There are two parts
in the fix:
* Use `insert-before-markers' instead of `insert' for creating the
button, so that it does not get added to the hidden overlay.
* Stop using PREFIX argument for adding a newline before the button.
The newline should not be added before a button at the beginning of
buffer.
The corresponding test is fixed now.
The test is currently broken and will be fixed by a subsequent patch.
The patch adds a new file for tests of Emacs notmuch-show view.
Based on patch by David Edmondson [1].
[1] id:"1327562380-12894-4-git-send-email-dme@dme.org"
Emacs message-mode uses certain text strings to indicate how to attach
files to outgoing mail. If these are present in the text of an email,
and a user is tricked into replying to the message, the user’s files
could be exposed.
Edited-by: Pieter Praet <pieter@praet.org>: Rebased to release branch.
The test is broken at this time; the next commit will introduce a fix.
Edited-by: Pieter Praet <pieter@praet.org>:
Rebased to release branch, moved expected output into the actual test,
and fixed "Fcc:" line.
`mail-header-parse-address' expects un-decoded mailbox parts, which is
not what we have at this point. Replace it with simple string
deconstruction.
Mark the corresponding test as no longer broken.
Minor whitespace cleanup.
