Previously, this was implemented using a horrible GDB script (because
there is no such thing as a non-horrible GDB script). This GDB script
often broke with newer versions of GDB for mysterious reasons. Port
the test script to GDB's Python API, which makes the code much cleaner
and, hopefully, more stable.
In the future, tests may rely on debug symbols being present in
notmuch, so we plan to switch the default flags.
The main purpose of this test is to help explain the perhaps
mysterious failures of other tests which rely on symbols being
present.
The test of viewing 8bit messages is known-broken. The rest pass, but
for very fragile reasons. The next several commits will fix the
known-broken test and make our charset handling robust.
Previously we did this for a single test, but some other proposed
tests ( id:1398105468-14317-3-git-send-email-amdragon@mit.edu ) show
similar breakage when switching renderers.
After yet another variation in objdump output caused this test to fail
(on a Debian port, no less), I decided whatever putative benefit we
get from looking at the object files instead of the library isn't
worth the maintenence headache.
This version uses nm -P. nm -P should be portable, and fixed format.
It purposely doesn't use the -D argument, since that is non-POSIX and
nm on GNU/Linux seems do the right thing without it.
It still won't work out of the box on e.g. Mac OS/X. I think the right
thing to do there is to move some more configuration information into
sh.config.
Previously, if the user ran any subcommand that required a
configuration (e.g., notmuch new) but didn't have a configuration,
notmuch would give the rather un-friendly and un-actionable message
Error reading configuration file .notmuch-config: No such file or directory
Since this condition is expected for new users, this patch adds
specific handling for the file-not-found case to give a message that
is friendly and actionable.
The version number has always been pretty meaningless to the user and
it's about to become even more meaningless with the introduction of
"features". Hopefully, the database will remain on version 3 for some
time to come; however, the introduction of new features over time in
version 3 will necessitate upgrades within version 3. It would be
confusing if we always tell the user they've been "upgraded to version
3". If the user wants to know what's new, they should read the news.
According to RFC2822 References and In-Reply-To headers are supposed
to contain one or more Message-IDs, however older RFC822 allowed
almost any content. When both References and In-Reply-To headers ends
with something else that a Message-ID (see e.g. [1]), the thread
structure presented by notmuch is incorrect. The reason is that
notmuch treats this case as if the email contained no "replyto"
information (see _notmuch_database_link_message_to_parents).
This patch changes the parse_references() function to return the last
valid Message-ID encountered rather than NULL resulting from the last
hunk of text not being the Message-ID.
[1] https://lkml.org/lkml/headers/2014/5/19/864
Previously the build system was generating automatic header
dependencies for test sources, but only smtp-dummy was in SRCS, so
only its dependencies were being included. Add all of the test
sources to SRCS so that the root Makefile.local includes their
dependencies.
This should help new users off to a better start with the addition of
more sensible saved searches and default shortcut keys. Most existing
users have probably customized this variable and won't be affected.
Dump currently only takes the read lock. Xapian can cope with some
changes while maintaining a read snapshot but with more changes it
fails. Currently notmuch just gives a xapian error.
To avoid this we take the write lock when dumping. This prevents other
notmuch processes from modifying the xapian database preventing this
error.
Discussion with Olly on irc indicates that this is currently the best
solution: in xapian trunk there may be better possibilities using
snapshots but they need to make it to a release and propogate out to
users before we can switch approach.
Finally, this breaks one use case: pipelines of the form
notmuch dump | ... | notmuch restore
According to Olly this is already very fragile: it will only work on
small databases. One of the tests relies on this behaviour so fix that
to store the dump rather than use a pipe.
These tests deliver all possible (single-root) four-message threads in
all possible orders and check that notmuch successfully links them
into threads. These tests supersede and replace the previous and much
less thorough "T260-thread-order" tests.
There are two variants of the test: one delivers messages that
reference only their immediate parent and the other delivers messages
that reference all of their parents. The latter test is currently
known-broken.
The unread/read changes will use the post-command-hook. test_emacs
does not call the post-command-hook. This adds a notmuch-test-progn
which takes a list of commands as argument and executes them in turn
but runs the post-command-hook after each one.
The caller can batch operations (ie to stop post-command-hook from
being interleaved) by wrapping the batch of operations inside a progn.
We also explicitly run the post-command-hook before getting the output
from a test; this makes sense as this will be a place the user would
be seeing the information.
This adds a 100 termpos gap between all phrases indexed by
_notmuch_message_gen_terms. This fixes a bug where terms from the end
of one header and the beginning of another header could match together
in a single phrase and a separate bug where term positions of
un-prefixed terms overlapped.
This fix only affects newly indexed messages. Messages that are
already indexed won't benefit from this fix without re-indexing, but
the fix won't make things any worse for existing messages.
This adds two known-broken tests and one working test related to the
term positions assigned to terms from different headers or MIME parts.
The first test fails because we don't create a termpos gap between
different headers. The second test fails because we don't adjust
termpos at all when indexing multiple parts.
Previously, we indexed the name and address parts of from/to headers
with two calls to _notmuch_message_gen_terms. In general, this
indicates that these parts are separate phrases. However, because of
an implementation quirk, the two calls to _notmuch_message_gen_terms
generated adjacent term positions for the prefixed terms, which
happens to be the right thing to do in this case, but the wrong thing
to do for all other calls. Furthermore, _notmuch_message_gen_terms
produced potentially overlapping term positions for the un-prefixed
copies of the terms, which is simply wrong.
This change indexes both the name and address in a single call to
_notmuch_message_gen_terms, indicating that they should be part of a
single phrase. This masks the problem with the un-prefixed terms
(fixing the two known-broken tests) and puts us in a position to fix
the unintentionally phrases generated by other calls to
_notmuch_message_gen_terms.
Two of these are currently known-broken. We index the name and
address parts in two separate calls to _notmuch_message_gen_terms.
Currently this has the effect of placing the term positions of the
prefixed terms from the second call right after those of the first
call, but screws up the term positions of the non-prefixed terms.
Two of the search tests for "from" and "to" queries were clearly
trying to search for prefixed phrases, but forgot to shell quote the
phrases. Fix this by quoting them correctly.
This is effectively a revert of
commit 6812136bf5
Author: Jani Nikula <jani@nikula.org>
Date: Mon Mar 31 00:21:48 2014 +0300
lib: drop support for single-message mbox files
The intention was to drop support for indexing new single-message mbox
files (and whether that was a good idea in the first place is
arguable). However this inadvertently broke support for reading
headers from previously indexed single-message mbox files, which is
far worse.
Distinguishing between the two cases would require more code than
simply bringing back support for single-message mbox files.
At least in emacs24, this removes the "site-lisp" directories from the
load path in addition to enforcing --no-site-lisp --no-init-file.
This works around a slightly mysterious bug on Debian that causes
test-lib.el not to load when there is cl-lib.el(c) in some site-lisp
directory. It should be harmless in general since we really don't
want to load any files from addon packages to emacs.
It turns out to be inconvenient to delete the downloaded datafiles with
distclean, so I propose a new target which does that instead.
The closest conventional target is 'maintainer-clean'; the difference
here is that having the original source tarball is not enough to
reconstruct these files.
The linking to talloc is hard-coded in the testing Makefile. This patch
causes the linking to talloc to be done according to how TALLOC_LDFLAGS
was configured.
Signed-off-by: Charles Celerier <cceleri@cs.stanford.edu>
- The old test was quite impossible to debug; the new one shows the difference
between the two directories, if any.
- "repository" doesn't make sense for out of tree builds. Or tarball
builds, for that matter.
All we do here is calculate the backup filename, and call the existing
dump routine.
Also take the opportunity to add a message about being safe to
interrupt.
The main goal is to support gzipped output for future internal
calls (e.g. from notmuch-new) to notmuch_database_dump.
The additional dependency is not very heavy since xapian already pulls
in zlib.
We want the dump to be "atomic", in the sense that after running the
dump file is either present and complete, or not present. This avoids
certain classes of mishaps involving overwriting a good backup with a
bad or partial one.
We've supported mbox files containing a single message for historical
reasons, but the support has been deprecated, with a warning message
while indexing, since Notmuch 0.15. Finally drop the support, and
consider all mbox files non-email.
Previously, the term escaper used a blacklist of characters that
needed escaping. This blacklist turned out to be somewhat incomplete;
for example, it did not contain non-whitespace ASCII control
characters or Unicode "fancy quotes", both of which do require the
term to be escaped.
Switch to a whitelist of characters that are definitely safe to leave
unquoted. This fixes the broken test introduced by the previous
patch.
The current term escaper gets most of these right, but fails to escape
things containing Unicode "fancy quotes" or things containing
non-whitespace control characters.
I still have one machine with old enough Xapian to not have compaction
support. Make the tests check for unsupported compact operation when
compact is not available.
This allows (and requires) the original-tags to be passed along with
the current-tags to be passed to notmuch-tag-format-tags. This allows
the tag formatting to show added and deleted tags.By default a removed
tag is displayed with strike-through in red (if strike-through is not
available, eg on a terminal, inverse video is used instead) and an
added tag is displayed underlined in green.
If the caller does not wish to use the new feature it can pass
current-tags for both arguments and, at this point, we do exactly that
in the three callers of this function.
Note, we cannot tidily allow original-tags to be optional because we would
need to distinguish nil meaning "we are not specifying original-tags"
from nil meaning there were no original-tags (an empty list).
We use this in subsequent patches to make it clear when a message was
unread when you first loaded a show buffer (previously the unread tag
could be removed before a user realised that it had been unread).
The code adds into the existing tag formatting code. The user can
specify exactly how a tag should be displayed normally, when deleted,
or when added.
Since the formatting code matches regexps a user can match all deleted
tags with a ".*" in notmuch-tag-deleted-formats. For example setting
notmuch-tag-deleted-formats to '((".*" nil)) tells notmuch not to show
deleted tags at all.
All the variables are customizable; however, more complicated cases
like changing the face depending on the type of display will require
custom lisp.
Currently this overrides notmuch-tag-deleted-formats for the tests
setting it to '((".*" nil)) so that they get removed from the display
and, thus, all tests still pass.
It turns out there was a reason the old man pages were stored in a man
compatible hierarchy, namely so that we could run man on them before
installing.
Hardcode doc build location into test suite. This isn't ideal, but
let's unbreak the test suite for now.
The checksum file is used by the test infrastructure to verify the downloaded
test database is the one we had in mind. Note that this test is
rather strict, and the the checksum file needs to be recommitted when
the database is regenerated.
add a pattern .gitignore to ignore the actual databases
Test the upgrade from probabilistic to boolean folder: terms, and
addition of path: terms.
The test depends on the pre-built test corpus and database tarball and
checksum file being in place. If it's not, the test is skipped. The
mechanism to fetch the test database will be added later.
At the time of writing, a working test database and checksum file is
available at
http://notmuchmail.org/releases/test-databases/
It has been noted that some non-GNU environments make lack
sha256sum. We leave this portability issue for a followup patch.
In xapian terms, convert folder: prefix from probabilistic to boolean
prefix, matching the paths, relative from the maildir root, of the
message files, ignoring the maildir new and cur leaf directories.
folder:foo matches all message files in foo, foo/new, and foo/cur.
folder:foo/new does *not* match message files in foo/new.
folder:"" matches all message files in the top level maildir and its
new and cur subdirectories.
This change constitutes a database change: bump the database version
and add database upgrade support for folder: terms. The upgrade also
adds path: terms.
Finally, fix the folder search test for literal folder: search, as
some of the folder: matching capabilities are lost in the
probabilistic to boolean prefix change.
We will need this for improved folder search tests, but having some
folders should exercise our code paths better anyway.
Modify the relevant test accordingly to make it pass.
This reorganization triggers a bug in the test suite, namely that it
expects the output of --output=files to be in a certain order. So we
add the fix for that into the same commit.
This mainly involves sorting, although the case --duplicate=$n
requires more subtlety.
Sanitize tabs and newlines to spaces rather than question marks in
--output=summary --format=text output.
This will also hide any difference in unfolding a header that has been
folded with a tab. Our own header parser replaces tabs with spaces,
while gmime would retain the tab.
The printf builtin "%(fmt)T" specifier (which allows time values
to use strftime-like formatting) is introduced in bash 4.2.
Trying to execute this in pre-4.2 bash will fail -- and if this
happens execute the fallback piece of perl code to do the same thing.
The test names assigned to NOTMUCH_SKIP_TESTS variable can now be given
with or without the Tddd- prefix for tester convenience:
The test name without Tddd -prefix stays constant even when test filenames
are renumbered.
The test name with Tddd -prefix is printed out when tests run.
According the semantics of make, the expansion of $(dir) in recipes
uses dynamic scope, i.e. the value at the time the recipe is run. This
means if test/Makefile.local is not the last sub-makefile included,
all heck breaks loose.
Previously, we stripped the "Tnnn-" part from the test name when
printing its description at the beginning of each test. However, this
makes it difficult to find the source script for a test (e.g., when a
test fails). Put this prefix back.
Searching by Message-Id no longer works via the old mail-archive.com
API, though I have contacted them in hopes that they restore it to
prevent dead links. Anyway, the new API is cleaner.
Acked-by: Austin Clements <amdragon@MIT.EDU>
Script `notmuch-test` expects the results file have T\d\d\d- part
intact so the results files (and some test output files) are now
name as such.
Without this change `notmuch-test` will exit in case the test
script it was executing exited with nonzero value.
The T\d\d\d- part is dropped in new variable $this_test_bare which is
used in progress informational messages and when loading .el files in
emacs tests (whenever $this_test_bare.el exists).
If there is a syntax error in the emacs test library, it causes other
tests to hang or crash without a useful error message.
This test could be eliminated if the error reporting for emacs tests
was somehow improved.
All test scripts to be executed are now named as T\d\d\d-name.sh,
numers in increments of 10.
This eases adding new tests and developers to see which are test scripts
that are executed by test suite and in which order.
When naming test scripts in format 'T\d\d\d-name.sh' the list of
tests to run are created dynamically. This makes test
'Ensure that all available tests will be run by notmuch-test'
in test/basic obsolete.
Previously the tags on each line in tree view were separarted by ", "
not just " ". This is different from show and search views.
This patch removes this comma. This is a large patch as essentially
every line of each of the expected outputs in the tree tests needs
updating.
Apart from aesthetic reasons this simplifies the switch to
notmuch-tag-format-tags in the next patch.
In some environments (at least Hurd), process-attributes is
unimplimented and always returns nil. This ends up causing test
failures (see e.g. id:87a9ffofsc.fsf@zancas.localnet).
Historically and according to POSIX 1003.1-2001, a signal of 0 can be
used to check the validity of a pid. This seems less heinous than
parsing the output of ps(1).
This fixes a non-deterministic failure in "Ignore files and
directories specified in new.ignore (multiple occurrences)". The test
assumed that all directories would be scanned, even though nothing
updated the mtime of ${MAIL_DIR}. It *usually* worked nevertheless
because the tests run quickly enough that the directory mtime is
usually the same as the current time, so notmuch new does not update
the mtime in the database (because more changes could occur in the
same second). However, when it occasionally did update the mtime in
the database, the notmuch new call in this test would (correctly) skip
"pass 2" of scanning ${MAIL_DIR}, causing it to skip the following
expected lines:
(D) add_files_recursive, pass 2: explicitly ignoring ${MAIL_DIR}/.git
(D) add_files_recursive, pass 2: explicitly ignoring ${MAIL_DIR}/.ignored_hidden_file
(D) add_files_recursive, pass 2: explicitly ignoring ${MAIL_DIR}/ignored_file
This patch fixes this problem by touching ${MAIL_DIR} to ensure it
gets scanned and by rearranging the test to ensure the directories are
touched immediately before the main notmuch new call in the test.
There is an obscure bug in notmuch-hello that very occasionally causes
emacs_deliver_message to fail. Since it it doesn't serve any actual
purpose in the function we delete it, and leave tracking down the the
bug for another day.
Most of the tests previously using emacs_deliver_message do not use
the actual transmitted message, so we replace it with a simpler (and
presumably more reliable function) that only saves (and indexes) an
fcc copy of the message.
When NOTMUCH_TEST_QUIET environment variable is set to non-null value
messages when new test script starts and when test PASSes are disabled.
This eases picking the cases when tests FAIL (as those are still printed).
In preparation for quiet mode print empty line before writing the
test description. This is done now in function designed for it --
it will also be called when test fails.
test-lib.sh sometimes did equivalent of `basename "$0" .sh`, sometimes
skipping the basename part and sometimes .sh part. This worked as
we never had path components in $0 (more than ./) nor .sh ending.
Now the equivalent of `basename "$0" .sh` is done once and used
everywhere. In the future we may have .sh suffix in test names
-- removing those is a good idea.
The choice of decreasing timestamps is a hack which reduces the number
of existing tests which fail. This can be changed to increasing
if/when somebody wants update another 47 tests.
add a new function notmuch_date_sanitize for rfc822-ish things. Add
date sanitization to notmuch_show_sanitize_all and use it more places.
This is all in aid of a transition to unique timestamps on messages.
Eventually we want test messages to have distinct dates to avoid
reproducability problems. This sanitization will prevent some test
failures when that change is made.
Replace the use of a local function in maildir-sync with
notmuch_json_show_sanitize
This was causing test failures because version strings varied in
length between GNU/Linux and GNU/KFreeBSD. One can also imagine
different versions of gnupg causing the same failure.
When executed command line is written to *Notmuch errors* buffer,
shell-quote-argument will backslash-escape any char that is not in
"POSIX filename characters" (i.e. matching "[^-0-9a-zA-Z_./\n]").
Currently in two emacs tests shell has expanded $PWD as part of
emacs variable, which will later be fed to #'shell-quote-argument
and finally written to ERROR file. If $PWD contained non-POSIX
filename characters, data in ERROR file will not match $PWD when
later comparing in shell. Therefore, in these two particular cases
the escaped $PWD is replaced with YYY in ERROR file and expected
content is adjusted accordingly.
This fixes races in thread-local and global tagging in notmuch-search
(e.g., "+", "-", "a", "*", etc.). Previously, these would modify tags
of new messages that arrived after the search. Now they only operate
on the messages that were in the threads when the search was
performed. This prevents surprises like archiving messages that
arrived in a thread after the search results were shown.
This eliminates `notmuch-search-find-thread-id-region(-search)'
because these functions strongly encouraged racy usage.
This fixes the two broken tests added by the previous patch.
These tests check that both thread-local and global search tagging
operations are race-free. They are currently known-broken because
they aren't race-free.
These queries will match exactly the set of messages currently in the
thread, even if more messages later arrive. Two queries are provided:
one for matched messages and one for unmatched messages.
This can be used to fix race conditions with tagging threads from
search results. While tagging based on a thread: query can affect
messages that arrived after the search, tagging based on stable
queries affects only the messages the user was shown in the search UI.
Since we want clients to be able to depend on the presence of these
queries, this ushers in schema version 2.
(Unfortunately, it's difficult to first demonstrate this problem with
a known-broken test because modern Linux kernels have argument length
limits in the megabytes, which makes Emacs really slow!)
It was looking in completely the wrong place for the backup and the
(test) xapian database. Unfortunately test_begin_subtest hides the
relevant errors.
Currently notmuch-show looks at the prefix-arg directly via
current-prefix-arg. This changes it to use the interactive
specification.
One test (for elide-toggle functionality) set the prefix arg
directly. Update this test to set the new argument directly.
One test (reply to encrypted message in the crypto test) recently
started failing on some systems. The failure I saw were two extra
lines of the form
<87d2nbc5xg.fsf@host.i-did-not-set--mail-host-address--so-tickle-me>
The test pipes the output through
grep -v -e '^In-Reply-To:' -e '^References:'
which would normally these two ids but it does not, in this case,
because they are so long they get put on a separate line in the output.
To fix this we set mail-host-address for emacs deliver. example.com
seems a sensible address to use. This is short enough that we don't
get the line breaks above and the tests then all pass.
As explained by Jeffrey Stedfast, the author of GMime, quoted in [1]:
> Passing the GMIME_ENABLE_RFC2047_WORKAROUNDS flag to g_mime_init()
> *should* solve the decoding problem mentioned in the thread. This
> flag should be safe to pass into g_mime_init() without any bad side
> effects and my unit tests do test that code-path.
The thread being referred to is [2].
[1] id:87bo56viyo.fsf@nikula.org
[2] id:08cb1dcd-c5db-4e33-8b09-7730cb3d59a2@gmail.com
Some common broken RFC 2047 encodings that we currently let gmime
parse strictly. We could tell gmime to be forgiving in what it accepts
as RFC 2047 encoding, making these tests pass.
When 'xpg_echo' bash shell option is unset (usually the default)
echo builtin does not expand backslash-escape sequences by default
(i.e. '\n' is echoed as '\n' instead of newline). Not all bash
installations have this feature we depend on activated by default.
Note that the feature is bash (and GNU /bin/echo) specific. It is used
as it is convenient. If portability is needed (elsewhere) use printf(1)
(also often available as a shell builtin).
If any of the tests in our test system is not passing the execution
of the test suite completes with nonzero exit value.
It is better to rely on the exit value of the test system instead
of some arbitrary strings in test output (or use both).
notmuch_message_tags_to_maildir_flags() unconditionally moves messages from
maildir directory "new/" to maildir directory "cur/", which makes messages lose
their "new" status in the MUA. However some users want to keep this "new"
status after, for instance, an auto-tagging of new messages.
However, as Austin mentioned and according to the maildir specification,
messages living in "new/" are not allowed to have flags, even if mutt allows it
to happen. For this reason, this patch prevents moving messages from "new/" to
"cur/", only if no flags have to be changed. It's hopefully enough to satisfy
mutt (and maybe other MUAs showing the "new" status) users checking the "new"
status.
Changelog:
* v2: Fix bool type as well as NULL returned despite having no errors (Austin
Clements)
* v4: Tag the related test (contributed by Michal Sojka) as working
Signed-off-by: Louis Rilling <l.rilling@av7.net>
[Condition for keeping messages in new/ was extended to satisfy all
tests from the previous patch. -Michal Sojka]
[Added by David Bremner, to keep the tests passing at each commit]
update insert tests for new maildir synchronization rules
As of id:1355952747-27350-4-git-send-email-sojkam1@fel.cvut.cz
we are more conservative about moving messages from ./new to ./cur.
This updates the insert tests to match
As mentioned by Jani Nikula in id:87vcccp4y3.fsf@nikula.org, some cases
of maildir synchronization are not covered by our tests. Let's add the
missing tests.
Some MUA's like mutt show the difference between "new" emails living in maildir
directory new/, and "old" emails living in maildir directory cur/. However
notmuch tag unconditionally moves selected messages from new/ to cur/, even if
no maildir synchronized tag is changed.
While maildir specification forbids messages with tags living in new/, there is
no need to move messages to cur/ when no maildir synchronized tag is changed.
Thus notmuch can remain transparent with respect to other MUA's.
[ Edited commit log to better describe the intended changes, and tag the
test as broken until the actual changes are implemented -- Louis Rilling ]
Signed-off-by: Louis Rilling <l.rilling@av7.net>
[ Converted to use test_subtest_known_broken, David Bremner ]
RFC 2047 states that the encoding and charset in an encoded word are
case-insensitive, so force them to lower case in the reply test. This
fixes an issue caused by GMime versions (somewhere between 2.6.10 and
2.6.16), which changed the capitalization of the encoding.
Previously, reply's default text format used an odd mix of RFC 2045
MIME encoding for the reply template's body and some made-up RFC
2822-like UTF-8 format for the headers. The intent was to present the
headers to the user in a nice, un-encoded format, but this assumed
that whatever ultimately sent the email would RFC 2047-encode the
headers, while at the same time the body was already RFC 2045 encoded,
so it assumed that whatever sent the email would *not* re-encode the
body.
This can be fixed by either producing a fully decoded UTF-8 reply
template, or a fully encoded MIME-compliant RFC 2822 message. This
patch does the latter because it is
a) Well-defined by RFC 2822 and MIME (while any UTF-8 format would be
ad hoc).
b) Ready to be piped to sendmail. The point of the text format is to
be minimal, so a user should be able to pop up the template in
whatever editor they want, edit it, and push it to sendmail.
c) Consistent with frontend capabilities. If a frontend has the
smarts to RFC 2047 encode the headers before sending the mail, it
probably has the smarts to RFC 2047 decode them before presenting
the template to a user for editing.
Also, as far as I know, nothing automated consumes the reply text
format, so changing this should not cause serious problems. (And if
anything does still consume this format, it probably gets these
encoding issues wrong anyway.)
Previously, the References header code seemed to assume
notmuch_message_get_header would return NULL if the header was not
present, but it actually returns "". As a result of this, it was
inserting an unnecessary space when concatenating an empty or missing
original references header with the new reference.
This shows up in only two tests because the text reply format later
passes the whole reply template through g_mime_filter_headers, which
has the side effect of stripping out this extra space.
This switches `notmuch-mua-reply' and `notmuch-query-get-threads' to
the S-exp format. These were the last two uses of the JSON format in
the Emacs frontend.
While looked good on paper, its attempted use caused confusion, complexity,
and potential for information leak when passed through wrapper scripts.
For slimmer code and to lessen demand for maintenance/support the set of
commits which added top level --stderr= option is now reverted.
The find option syntax `-perm +111` is deprecated gnu find feature.
The replacement `( -perm -100 -o -perm -10 -o -perm 1 )` should also
work outside of the GNU domain.
In addition to being the Right Thing to do, this noticeably improves
the time taken to display the first page of search results, since it's
roughly an order of magnitude faster than the JSON parser.
Interestingly, it does *not* significantly improve the time to
completely fill a large search buffer because for large search
buffers, the cost of creating author invisibility overlays and
inserting text (which slows down with more overlays) dominates.
However, the time required to display the first page of results is
generally more important to the user experience.
Previously, search started the async notmuch process directly. Now,
it uses `notmuch-start-notmuch'. This simplifies the process sentinel
a bit and means that we no longer have to worry about errors
interleaved with the JSON output.
We also update the tests of Emacs error handling, since the error
output is now separated from the search results buffer.
We now check error handling more carefully in the last test in
test/emacs and we're about to add more error handling tests. (This
was also a strange place for this test, since it had nothing to do
with large search buffers.)
This unifies the part button actions and the underlying part action
functions into single interactive command that simply applies to the
part containing point using the just-added part p-list text property
instead of button properties. Since all part actions can be performed
by applying the appropriate mm function to an mm-handle, this patch
abstracts out the creation of mm handles, making the implementations
of the part commands trivial. This also eliminates our special
handling for part save in favor of using the appropriate mm function.
This necessarily modifies the way we handle the default part button
action, but in a way that does not change the meaning of the
notmuch-show-part-button-default-action defcustom.
Since these commands are no longer specific to buttons, this patch
eliminates the extra metadata stored with each button. This also
eliminates one rather special-purpose macro for a collection of
general purpose part handling utilities.
--stderr=FILE tests were added to test/help-test as it is the one
doing most global option testing. Also, it was simplest to test
this new option using `notmuch help` command.
Presently, the code which finds the parent of a message as it is being
added to the database assumes that the first Message-ID-like substring
of the In-Reply-To header is the parent Message ID. Some mail clients,
however, put stuff other than the Message-ID of the parent in the
In-Reply-To header, such as the email address of the sender of the
parent. This can fool notmuch.
The updated algorithm prefers the last Message ID in the References
header. The References header lists messages oldest-first, so the last
Message ID is the parent (RFC2822, p. 24). The References header is
also less likely to be in a non-standard
syntax (http://cr.yp.to/immhf/thread.html,
http://www.jwz.org/doc/threading.html). In case the References header
is not to be found, fall back to the old behavior.
V2 of this patch, incorporating feedback from Jani and (indirectly)
Austin.
The use of realpath(3) in
commit 58ed67992d
Author: Jani Nikula <jani@nikula.org>
Date: Sun Apr 7 20:15:03 2013 +0300
cli: config: do not overwrite symlinks when saving config file
broke config file save when the file does not exist, which results in
'notmuch setup' always failing to create a new config file.
Fix by checking ENOENT from realpath(3).
Use realpath to canonicalize the config path before writing.
Previously 'notmuch setup' and 'notmuch config set' overwrote the
config file even if it was a symbolic link.
We now have a notmuch_config_is_new() function to query whether a
config was created or not. Change the notmuch_config_open() is_new
parameter into boolean create_new to determine whether the function
should create a new config if one doesn't exist. This reduces the
complexity of the API.
When execution of tests is interrupted by signal coming outside of the
test system itself, output just one line "interrupted by signal <num>"
message to standard output. This distinguishes the case from internal
exit and reduces noise.
Set the variable '$test_subtest_name' in all functions which starts
a new test and use that variable in all functions that output
test results.
Additionally output the latest '$test_subtest_name' in case of
abnormal exit, to avoid confusion.
Instead of checking immediately for the watched process, delay a
minute, or in the case that process-attributes returns nil, for two
minutes. This is intended to cope with the case that
process-attributes is unimplimented, and returns always returns nil.
In this case, the watchdog check is the same as the two minute limit
imposed by timeout.
The TERM environment variable is set to 'dumb' when running tests, but
the original value of it is stored for echoing colors and running emacs
(somewhat interactively) in detached session. Emacs requires some
terminal control sequences to be available for interactive operation.
In case original TERM is (also) 'dumb' (or unset/empty) emacs cannot
run interactively. To fix this problem dtach (and emacs as it's child
process) is run with TERM=vt100 in case original TERM was unset, empty
or 'dumb'. This way there is a chance to run emacs tests with different
user terminals and potentially find problems there.
This test also serves as documentation of the quoting
requirements. The comment lines are so that it exactly matches the man
page. Nothing more embarrassing than having an example in the man page
fail.
The (now fixed) bug that this test revealed is that unquoted
message-ids with whitespace or other control characters in them are
split into several tokens by the Xapian query parser.
This is based on the similar test for notmuch restore, but the parser
in batch tagging mode is less tolerant of a few cases, in particular
those tested by illegal_tag.
We recently switched to popping up a buffer to report CLI errors, but
this was too intrusive, especially for transient errors and especially
since we made fewer things ignore errors. This patch changes this to
display a basic error message in the minibuffer (using Emacs' usual
error handling path) and, if there are additional details, to log
these to a separate error buffer and reference the error buffer from
the minibuffer message. This is more in line with how Emacs typically
handles errors, but makes the details available to the user without
flooding them with the details.
Given this split, we pare down the basic message and make it more
user-friendly, and also make the verbose message even more detailed
(and more debugging-oriented).
This switches the new batch-tag format away from using a home-grown
hex-encoding scheme for message IDs in the dump to simply using Xapian
queries with Xapian quoting syntax.
This has a variety of advantages beyond presenting a cleaner and more
consistent interface. Foremost is that it will dramatically simplify
the quoting for batch tagging, which shares the same input format.
While the hex-encoding is no better or worse for the simple ID queries
used by dump/restore, it becomes onerous for general-purpose queries
used in batch tagging. It also better handles strange cases like
"id:foo and bar", since this is no longer syntactically valid.
When we switch to using regular Xapian queries in the dump format, \n
will cause problems, so we disallow it. Specially, while Xapian can
quote and parse queries containing \n without difficultly, quoted
queries containing \n still span multiple lines, which breaks the
line-orientedness of the dump format. Strictly speaking, we could
still round-trip these, but it would significantly complicate restore
as well as scripts that deal with tag dumps. This complexity would
come at absolutely no benefit: because of the RFC 2822 unfolding
rules, no amount of standards negligence can produce a message with a
message ID containing a line break (not even Outlook can do it!).
Hence, we simply disallow it.
This patch corrects several undesirable behaviours:
1) Empty files were not detected, leading to buffer read overrun.
2) An initial blank line cause restore to silently abort
3) Initial comment line caused format detection to fail
notmuch_json_show_sanitize replaced "filename" field values even in part
structures, where the value is predictable. Make it only normalize the
filename value if it is an absolute path (begins with slash), which is
true of the Maildir filenames that were intended to be normalized away.
This slightly changes the output of an existing test since we now
report non-zero exits with a pop-up buffer instead of at the end of
the search results.
* test/emacs:
- Rename subtests "{Add,Remove} tag from notmuch-show view" to
"notmuch-show: {add,remove} single tag {to,from} single message"
to be consistent with the following tests.
- New subtest "notmuch-show: add multiple tags to single message":
`notmuch-show-add-tag' ("+") can add multiple tags to a message.
- New subtest "notmuch-show: remove multiple tags from single message":
`notmuch-show-remove-tag' ("-") can remove multiple tags from a message.
We want to test both that error/warning messages are generated when
they should be, and not generated when they should not be. This varies
between restore and batch tagging.
These one need the completed functionality in notmuch-restore. Fairly
exotic tags are tested, but no weird message id's.
We test each possible input to autodetection, both explicit (with
--format=auto) and implicit (without --format).
The quoting for ${SEARCH} is broken when it's supposed to be '*', and
it seems tricky to get it right. Just drop the variable and use '*'
directly. Before this, none of the messages ever matched, and the test
was comparing zeros.
test_expect_equal_json uses json.tool from the system Python. While
Python 2 wasn't picky about the encoding of stdin, Python 3 decodes
stdin strictly according to the environment. Since we set LC_ALL=C
for the tests, Python 3's json.tool was assuming stdin would be in
ASCII and aborting when it couldn't decode the UTF-8 characters from
some of the JSON tests. This patch sets the PYTHONIOENCODING
environment variable to utf-8 when invoking json.tool to override
Python's default encoding choice.
Without this change, GCC complains as follows:
gcc test/random-corpus.o test/database-test.o notmuch-config.o command-line-arguments.o lib/libnotmuch.a util/libutil.a parse-time-string/libparse-time-string.a -o test/random-corpus -lgmime-2.6 -lgio-2.0 -lgobject-2.0 -lglib-2.0 -Wl,-rpath,/usr/lib -ltalloc -lxapian
/usr/bin/ld: lib/libnotmuch.a(database.o): undefined reference to symbol '_ZNSs4_Rep10_M_destroyERKSaIcE@@GLIBCXX_3.4'
/usr/bin/ld: note: '_ZNSs4_Rep10_M_destroyERKSaIcE@@GLIBCXX_3.4' is defined in DSO /usr/lib/libstdc++.so.6 so try adding it to the linker command line
/usr/lib/libstdc++.so.6: could not read symbols: Invalid operation
collect2: error: ld returned 1 exit status
make: *** [test/random-corpus] Error 1
We demonstrate the current notmuch restore parser being confused by
message-id's and tags containing non alpha numeric characters
(particularly space and parentheses are problematic because they are
not escaped by notmuch dump).
We save the files as hex escaped on disk so that terminal emulators
will not get confused if the test fails (as we mostly expect it to do).
Initial use case is testing dump and restore, so we only have
message-ids and tags.
The message ID's are nothing like RFC compliant, but it doesn't seem
any harder to roundtrip random UTF-8 strings than RFC-compliant ones.
Tags are UTF-8, even though notmuch is in principle more generous than
that.
updated for id:m2wr04ocro.fsf@guru.guru-group.fi
- talk about Unicode value rather some specific encoding
- call talloc_realloc less times
Initially, provide a way to create "stub" messages in the notmuch
database without corresponding files. This is essentially cut and
paste from lib/database.cc. This is a seperate file since we don't
want to export these symbols from libnotmuch or bloat the library with
non-exported code.
Previously, the test framework generated a variable name for each
external prereq as a poor man's associative array. Unfortunately,
prereqs names may not be legal variable names, leading to
unintelligible bash errors like
test_missing_external_prereq_emacsclient.emacs24_=t: command not found
Using proper associative arrays to track prereqs, in addition to being
much cleaner than generating variable names and using grep to
carefully construct unique string lists, removes restrictions on
prereq names.
Previously, if a test script aborted (e.g., because it passed too few
arguments to a test function), the test driver loop would simply
continue on to the next test script and the final results would
declare that everything passed (except that the test count would look
suspiciously low, but maybe you just misremembered how many tests
there were).
Now, if a test script exits with a non-zero status and did not produce
a final results file, we propagate that failure out of the driver loop
immediately.
To keep this simple, this patch removes the PID from the test-results
file name. This PID was inherited from the git test system and seems
unnecessary, since the file name already includes the name of the test
script and the test-results directory is created anew for each run.
And require that if TEST_EMACS is specified, so is TEST_EMACSCLIENT.
Previously, the test framework always used "emacsclient", even if the
Emacs in use was overridden by TEST_EMACS. This causes problems if
both Emacs 23 and Emacs 24 are installed, the Emacs 23 emacsclient is
the system default, but TEST_EMACS is set to emacs24. Specifically,
with an Emacs 24 server and an Emacs 23 client, emacs tests that run
very quickly may produce no output from emacsclient, causing the test
to fail.
The Emacs server uses a very simple line-oriented protocol in which
the client sends a request to evaluate an expression and the server
sends a request to print the result of evaluation. Prior to Emacs bzr
commit 107565 on March 11th, 2012 (released in Emacs 24.1), if
multiple commands were sent to the emacsclient between when it sent
the evaluation command and when it entered its receive loop, it would
only process the first response command, ignoring the rest of the
received buffer. This wasn't a problem with the Emacs 23 server
because it sent only the command to print the evaluation result.
However, the Emacs 24 server first sends an unprompted command
specifying the PID of the Emacs server, then processes the evaluation
request, then sends the command to print the result. If the
evaluation is fast enough, it can send both of these commands before
emacsclient enters the receive loop. Hence, if an Emacs 24 server is
used with an Emacs 23 emacsclient, it may miss the response printing
command, ultimately causing intermittent notmuch test failures.
Previously, many tests in emacs-subject-to-filename didn't quote the
$output argument to test_expect_equal. As a result, if $output was
empty, test_expect_equal would be passed only one argument and would
abort the entire test script. By quoting the argument, we ensure
test_expect_equal will always receive two arguments.
We now test for user ignore patterns before attempting to determine if
a directory entry is itself a directory. As a result, we no longer
abort for broken symlinks if the user has explicitly ignored them.
This fixes the test added in the previous patch. It also slightly
changes the debug output checked by another test of ignores.
The macro with-current-notmuch-show-message executes command
`notmuch show --format=raw id:...` which just outputs the contents
of the mail file verbatim (into temporary buffer). In case e.g. utf-8
locale is used the temporary buffer has buffer-file-coding-system as
utf-8. In this case Emacs converts the data to multibyte format, guessing
that input is in utf-8.
However, the "raw" (MIME) message may contain octet data in any other
8bit format, and as no (MIME-)content spesific handling to the message
is done at this point, conversion to other formats may lose information.
By setting coding-system-for-read 'no-conversion drops the conversion part
and makes this handle input as notmuch-get-bodypart-internal() does.
This marks the broken test in previous change fixed.
Previously, we would treat multi-message mboxes as one giant email,
which, besides the obvious incorrect indexing, often led to
out-of-memory errors for archival mboxes. Now we explicitly reject
multi-message mboxes. For historical reasons, we retain support for
single-message mboxes, but official deprecate this behavior.
This test is currently broken. Note that its brokenness cascades and
causes the next test to fail as well (because notmuch incorrectly
indexes the mbox file).
There are currently 45 TESTS scripts. 36 of those load
test-lib.sh using '. ./test-lib.sh' and 9 '. test-lib.sh'.
In latter case test-lib.sh is first searched from directories
in PATH (posix) and then from current directory (bash feature).
Changed the 9 files to execute '. ./test-lib.sh'. The test-lib.sh
should never be loaded from directory in PATH.
Previously, this would simply indicate that the grep failed without
any indication of the Emacs output it failed on. Now we take
advantage of the test framework's handling of stdout to display the
incorrect Emacs output if the test fails.
