The exact error messages returned by regerror() aren't standardized;
relying on them isn't portable. Thus, add a a prefix to make clear that
the subsequent message is a regexp parsing error, and only look for this
prefix in the test suite, ignoring the rest of the message.
POSIX doesn't specify the flushing behaviour of the STDOUT stream, so
it's invalid to assume a particular order between the stdout and stderr
output. The current test breaks on musl due to this.
I can't figure out how checking the sign of a bool ever worked. The
following program demonstrates the problem (i.e. for me it prints 1).
#include <stdio.h>
#include <stdbool.h>
int main(int argc, char **argv) {
bool x;
x = -1;
printf("x = %d\n", x);
}
This seems to be mandated by the C99 standard 6.3.1.2.
When generating a reply message, if the user was the originator and
only recipient of the original message, include the user as a
recipient of the reply.
b31e44c678 introduced message-id-parse
as a new binary created by the test suite. It shows up as something
additional to git, but git ought to know to ignore it.
As reported by Sean Whitton, there are mailers (in particular the
Debian Bug Tracking System) that have sensible In-Reply-To headers,
but un-useful-for-notmuch References (in particular with the BTS, the
oldest reference is last). I looked at a sample of about 200K
messages, and only about 0.5% these had something other than a single
message-id in In-Reply-To. On this basis, if we see a single
message-id in In-Reply-To, consider that as authoritative.
The idea is that if a message-id parses with this function, the MUA
generating it was probably sane, and in particular it's probably safe
to use the result as a parent from In-Reply-to.
In a future commit, we will start trusting In-Reply-To's when they
look sane (i.e. a single message-id). Modify these tests so they will
keep passing (i.e. keep choosing References) when that happens.
The current scheme of choosing the replyto (i.e. the default parent
for threading purposes) does not work well for mailers that put
the oldest Reference last.
We (finally) implement the XXX comment. It requires a bit of care not
to reparent all of the possible toplevel messages.
_notmuch_messages_has_next is not ready to be a public function yet,
since it punts on the mset case. We know in the one case it is called,
the notmuch_messages_t is just a regular list / iterator.
This is mainly to lay out the structure of the final code. The problem
isn't really solved yet, although some very simple cases are
better (hence the fixed test). We need two passes through the messages
because we need to be careful not to re-parent too many messages and
end up without any toplevel messages.
For non-root messages, this should not should anything currently, as
the messages are already added in date order. In the future we will
add some non-root messages in a second pass out of order and the
sorting will be useful. It does fix the order of multiple
root-messages (although it is overkill for that).
This documents the bug discussed at
id:87efgmmysi.fsf@len.workgroup
The underlying issue is that the reply to a ghost (missing) message is
falsely classified as a root message in _resolve_thread_relationships.
There are two pairs of tests; in each case the the first test is
simpler / more robust, but also easier to fool.
There are 3 threads here, two synthetic, and one anonymized one using
data from Gregor. They test various aspects of thread
ordering/construction in the presence of replies to ghost messages.
This clarifies that the breakage seen with Xapian 1.4.6 does not have
to do with "funny" tags.
This test is "known broken", but only with xapian 1.4.6, and there's
curently no convenient way to mark that.
Messages that contain Windows-1252 are frequently mislabeled as ISO
8859-1, which may result in non-printable characters when displaying
the message. The test asserts that such characters (in this case
curved quotes) are displayed correctly.
Add fancy new feature, which makes "notmuch show" capable of actually
indexing messages that it just decrypted.
This enables a workflow where messages can come in in the background
and be indexed using "--decrypt=auto". But when showing an encrypted
message for the first time, it gets automatically indexed.
This is something of a departure for "notmuch show" -- in particular,
because it requires read/write access to the database. However, this
might be a common use case -- people get mail delivered and indexed in
the background, but only want access to their secret key to happen
when they're directly interacting with notmuch itself.
In such a scenario, they couldn't search newly-delivered, encrypted
messages, but they could search for them once they've read them.
Documentation of this new feature also uses a table form, similar to
that found in the description of index.decrypt in notmuch-config(1).
A notmuch UI that wants to facilitate this workflow while also
offering an interactive search interface might instead make use of
these additional commands while the user is at the console:
Count received encrypted messages (if > 0, there are some things we
haven't yet tried to index, and therefore can't yet search):
notmuch count tag:encrypted and \
not property:index.decryption=success and \
not property:index.decryption=failure
Reindex those messages:
notmuch reindex --try-decrypt=true tag:encrypted and \
not property:index.decryption=success and \
not property:index.decryption=failure
Wherever the test relies on directories being scanned, this option
should be used to avoid skipping them due to mtimes on directories
matching the database.
Most of these just check that adding the flag does not break existing
functionality. The one test that does check the full-scan
functionality had to be rewritten to output debugging info.
This way, one can build for a different Ruby than $PATH/ruby
(e. g. different versions, or Ruby in other paths).
Signed-off-by: Thomas Schneider <qsx@chaotikum.eu>
This change allows queries of the form
thread:{from:me} and thread:{from:jian} and not thread:{from:dave}
This is still somewhat brute-force, but it's a big improvement over
both the shell script solution and the previous proposal [1], because it
does not build the whole thread structure just generate a
query. A further potential optimization is to replace the calls to
notmuch with more specialized Xapian code; in particular it's not
likely that reading all of the message metadata is a win here.
[1]: id:20170820213240.20526-1-david@tethera.net
Correct URLs that have crept into the notmuch codebase with http://
when https:// is possible.
As part of this conversion, this changeset also indicates the current
preferred upstream URLs for both gmime and sup. the new URLs are
https-enabled, the old ones are not.
This also fixes T310-emacs.sh, thanks to Bremner for catching it.
We expect this to give the same answer as the non-regexp subject
search. It does not because the regexp search relies on the value
slot, which currently contains only one subject.
These roughly replicate the equivalent C tests, although they rely on
the database state created by the former tests, since the python
bindings currently provide read-only access to properties.
The previous loop handling code chooses the last message in the
message list, which turns out to be the last in date order.
See the comment in _notmuch_thread_create.
In some cases (e.g. when building a publicly-visible e-mail archive)
it doesn't make any sense to restrict visibility of the message to the
current user account.
This adds a --world-readable boolean option for "notmuch insert", so
that those who want to archive their mail publicly can feed their
archiver with:
notmuch insert --world-readable
Other local delivery agents (postfix's local, and dovecot's lda) all
default to delivery in mode 0600 rather than relying on the user's
umask, so this fix doesn't change the default.
Also, this does not override the user's umask. if the umask is
already set tight, it will not become looser as the result of passing
--world-readable.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
This brings the --decrypt argument to "notmuch reply" into line with
the other --decrypt arguments (in "show", "new", "insert", and
"reindex"). This patch is really just about bringing consistency to
the user interface.
We also use the recommended form in the emacs MUA when replying, and
update test T350 to match.
We also expand tab completion for it, update the emacs bindings, and
update T350, T357, and T450 to match.
Make use of the bool-to-keyword backward-compatibility feature.
We might change some notmuch command line tools that used to be
booleans into keyword arguments.
In that case, there are some legacy tools that will expect to be able
to do "notmuch foo --bar" instead of "notmuch foo --bar=baz".
This patch makes it possible to support that older API, while
providing a warning and an encouragement to upgrade.
We adopt a pythonic idiom here with an optional argument, rather than
exposing the user to the C indexopts object directly.
This now includes a simple test to ensure that the decrypt_policy
argument works as expected.
This test passes with older versions of Xapian as well, because
neither query returns any results.
This should resolve the travis build failure at
https://travis-ci.org/notmuch/notmuch/builds/318571658
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
The current behaviour is at best under-documented. The modified test in
T470-missing-headers.sh previously relied on printf doing the right
thing with NULL, which seems icky.
The use of talloc_strdup here is probably overkill, but it avoids
having to enforce that thread->authors is never mutated outside
_resolve_thread_authors_string.
If we're building against a version of Xapian that doesn't offer
retrying the lock, we should be honest and describe the tests as
broken, rather than marking them as missing a test prerequisite.
missing test prerequisites should be for specific components of the
test harness that are missing, not for the backend library notmuch
uses.
Named queries don't work without Xapian FieldProcessor. Rather than
silently skipping them, we should explictly mark them as broken when
building against an older version of Xapian.
Previously, the test suite had simply silently skipped the absolute
date test if we're using an archaic version of Xapian. For
correctness, we should instead mark the test as broken.
This also changes from string to numeric comparison when checking
NOMTUCH_HAVE_XAPIAN_FIELD_PROCESSOR for consistency with other tests.
The old name has a bit of a feeling of hungarian notation. Also many
generators in the core are named with the suffix "s" to indicate
iterables: dict.items, dict.keys for example.
This test will pass if either the notmuch show mitigation code is
working correctly, or upstream emacs mime handling code has it's own
fix for https://bugs.gnu.org/28350.
When i'm trying to understand a message signature, i care that i know
who it came from (the "validity" of the identity associated with the
key), *not* whether i'm willing to accept the keyholder's other
identity assertions (the "trust" associated with the certificate).
We've been reporting User ID information based on the "trust"
associated with the certificate, because GMime didn't clearly expose
the validity of the User IDs.
This change relies on fixes made in GMime 3.0.3 and later which
include https://github.com/jstedfast/gmime/pull/18.
Here's the configuration choice for people who want a cleartext index,
but don't want stashed session keys.
Interestingly, this "nostash" decryption policy is actually the same
policy that should be used by "notmuch show" and "notmuch reply",
since they never modify the index or database when they are invoked
with --decrypt.
We take advantage of this parallel to tune the behavior of those
programs so that we're not requesting session keys from GnuPG during
"show" and "reply" that we would then otherwise just throw away.
If you're going to store the cleartext index of an encrypted message,
in most situations you might just as well store the session key.
Doing this storage has efficiency and recoverability advantages.
Combined with a schedule of regular OpenPGP subkey rotation and
destruction, this can also offer security benefits, like "deletable
e-mail", which is the store-and-forward analog to "forward secrecy".
But wait, i hear you saying, i have a special need to store cleartext
indexes but it's really bad for me to store session keys! Maybe
(let's imagine) i get lots of e-mails with incriminating photos
attached, and i want to be able to search for them by the text in the
e-mail, but i don't want someone with access to the index to be
actually able to see the photos themselves.
Fret not, the next patch in this series will support your wacky
uncommon use case.
There are some situations where the user wants to get rid of the
cleartext index of a message. For example, if they're indexing
encrypted messages normally, but suddenly they run across a message
that they really don't want any trace of in their index.
In that case, the natural thing to do is:
notmuch reindex --decrypt=false id:whatever@example.biz
But of course, clearing the cleartext index without clearing the
stashed session key is just silly. So we do the expected thing and
also destroy any stashed session keys while we're destroying the index
of the cleartext.
Note that stashed session keys are stored in the xapian database, but
xapian does not currently allow safe deletion (see
https://trac.xapian.org/ticket/742).
As a workaround, after removing session keys and cleartext material
from the database, the user probably should do something like "notmuch
compact" to try to purge whatever recoverable data is left in the
xapian freelist. This problem really needs to be addressed within
xapian, though, if we want it fixed right.
The new "auto" decryption policy is not only good for "notmuch show"
and "notmuch reindex". It's also useful for indexing messages --
there's no good reason to not try to go ahead and index the cleartext
of a message that we have a stashed session key for.
This change updates the defaults and tunes the test suite to make sure
that they have taken effect.
When showing a message, if the user doesn't specify --decrypt= at all,
but a stashed session key is known to notmuch, notmuch should just go
ahead and try to decrypt the message with the session key (without
bothering the user for access to their asymmetric secret key).
The user can disable this at the command line with --decrypt=false if
they really don't want to look at the e-mail that they've asked
notmuch to show them.
and of course, "notmuch show --decrypt" still works for accessing the
user's secret keys if necessary.
If the user doesn't specify --decrypt= at all, but a stashed session
key is known to notmuch, when replying to an encrypted message,
notmuch should just go ahead and decrypt.
The user can disable this at the command line with --decrypt=false,
though it's not clear why they would ever want to do that.
This new automatic decryption policy should make it possible to
decrypt messages that we have stashed session keys for, without
incurring a call to the user's asymmetric keys.
Future patches in this series will introduce new policies; this merely
readies the way for them.
We also convert --try-decrypt to a keyword argument instead of a boolean.
the command-line interface for indexing (reindex, new, insert) used
--try-decrypt; and the configuration records used index.try_decrypt.
But by comparison with "show" and "reply", there doesn't seem to be
any reason for the "try" prefix.
This changeset adjusts the command-line interface and the
configuration interface.
For the moment, i've left indexopts_{set,get}_try_decrypt alone. The
subsequent changeset will address those.
If the version of GMime we're building against doesn't support session
key extraction or re-use, mark the tests that rely on session key
capabilities as known-broken.
This should resolve test suite failures on ubuntu trusty and debian
jessie and earlier, which have GMime 2.6.20 -- session key support was
introduced in GMime 2.6.21.
If you've got a notmuch dump that includes stashed session keys for
every decrypted message, and you've got your message archive, you
should be able to get back to the same index that you had before.
Here we add a simple test that give some flavor of how that works.
In 8e7fb88237 Jani replaced the use of $(pwd -P) to find the
TEST_DIRECTORY in order to better support out of tree
builds. Unfortunately the performance-tests need a different value for
the variable and were thus broken.
This commit splits out the setting of this variable for the two sets
of tests. Performance tests still don't work out of tree, because
the handling of the downloaded corpus needs to be updated.
backup_database() and restore_database() used to store the backups in
the test specific temporary directory, through the current working
directory being there. Commit 8e7fb88237 ("test: use source and
build paths in test-lib-common.sh") started using a test specific
backup directories under the build tree test directory. This was in
error. Switch back to the old location, but using paths to the
location instead of relying on current working directory.
Reported by Daniel Kahn Gillmor <dkg@fifthhorseman.net>.
Enable override of the index.try_decrypt setting on a per-run basis
when invoking "notmuch reindex". This allows the possibility of (for
example) an emacs keybinding that adds the cleartext of the currently
shown decrypted message to the index, making it searchable in the
future.
It also enables one-time indexing of all messages matching some query,
like so:
notmuch reindex tag:encrypted and\
not property:index.decryption=success and\
from:alice@example.org
We also update the documentation and tab completion, and add a few
more tests.
Enable override of the index.try_decrypt setting on a per-message
basis when invoking "notmuch insert".
We also update the documentation and tab completion, and add more tests.
Enable override of the index.try_decrypt setting during "notmuch new"
on a per-invocation basis.
We update the documentation and tab completion, and also add a test.
Subsequent patches may want to send GNU-style --long-arguments to
notmuch new in the test suite, in particular when invoking
emacs_fcc_message. This changeset makes that possible.
The primary motivation here is to fix TMP_DIRECTORY cleanup prior to
running each test when the current working directory is not the test
subdirectory. Tests with failures would leave their TMP_DIRECTORY
directory behind for debugging, and repeated out-of-tree test runs
would have old temp directories. (This lead to e.g. T310-emacs.sh
hanging because emacs would prompt for overwriting files.)
We remove the likely anyway defunct --root test option while at it,
just to be on the safe side when doing 'rm -rf' on the TMP_DIRECTORY.
Turns out round trip tests didn't really round trip anything. Broken
by yours truly in 971cdc72cd ("test: make it possible to have
multiple corpora"). Ooops.
C99 stdbool turned 18 this year. There really is no reason to use our
own, except in the library interface for backward
compatibility. Convert the cli and test binaries to stdbool.
Several changes at once, just to not have to change the same lines
several times over:
- Use designated initializers to initialize opt desc arrays.
- Only initialize the needed fields.
- Remove arg_id (short options) as unused.
- Replace opt_type and output_var with several type safe output
variables, where the output variable being non-NULL determines the
type. Introduce checks to ensure only one is set. The downside is
some waste of const space per argument; this could be saved by
retaining opt_type and using a union, but that's still pretty
verbose.
- Fix some variables due to the type safety. Mostly a good thing, but
leads to some enums being changed to ints. This is pedantically
correct, but somewhat annoying. We could also cast, but that defeats
the purpose a bit.
- Terminate the opt desc arrays using {}.
The output variable type safety and the ability to add new fields for
just some output types or arguments are the big wins. For example, if
we wanted to add a variable to set when the argument is present, we
could do so for just the arguments that need it.
Beauty is in the eye of the beholder, but I think this looks nice when
defining the arguments, and reduces some of the verbosity we have
there.
This imports a message with ISO-8859-2 encoded characters, then opens
the database using the python bindings. We peek through all mesage
parts, afterwards print the message id.
Signed-off-by: Florian Klink <flokli@flokli.de>
Signed-off-by: Andreas Rammhold <andreas@rammhold.de>
Commit a7964c86d1 ("emacs: Sanitize authors and subjects in search
and show") added sanitization of header information for display. Do
the same for reply subjects.
This fixes the long-standing annoying artefact of certain versions of
mailman using tab as folding whitespace, leading to tabs in reply
subjects.
This is a logical followup to "lib: index the content type of
signature parts", which will make it easier to record the message
structure of all messages.
It's useful (*) to be able to easily find messages with certain types
of signatures. Having the mimetype: prefix searches fail for some
content types is also genuinely surprising (*). Index the content type
of signature parts.
While at it, switch to the gmime convenience constants for content and
signature part indexes.
*) At least for developers of email software!
Add known broken subtests for searching signed and encrypted messages
using mimetype: prefix search for the content-types of signed and
encrypted parts.
In [1] Mark showed that the the current code (d7a49e81) is not
consistent in it's handling of subjects of messages with duplicate
message-ids (or in notmuch-speak, of messages with multiple files).
notmuch-search uses indexing order and explicitedly preserves the
first. notmuch-show (apparently) uses alphabetical (or at least xapian
term order) of filenames. In a perfect world we would probably report
all subjects in the json output; at the very least we should be
consistent.
[1]: id:87378dny3d.fsf@qmul.ac.uk
In [1], Mark gave a test that was behaving strangly. This turns out to
be specific to reindexing. I suppose one could argue that picking the
lexicographically last file name is a defensible choice, but it's
almost as easy to take the first, which seems more intuitive. So mark
the current situation as broken.
[1]: id:1503859703-2973-1-git-send-email-markwalters1009@gmail.com
The existing test for notmuch search had the first in filename order
the same as the first indexed, which made it harder to understand what
the underlying behaviour is. Add a file with a lexicographically
smaller name, but later index time to clarify this.
The original intent of this test was to verify that notmuch show was
not crashing when the first file (where headers are being read from)
was deleted. Run the output through some sanitization so that as we
add and delete copies we don't have to update this test.
Changed "" quotes to '' as we're not supposed to dynamically
alter python program (via shell $variable expansion).
Added space to python program to match general python style.
Replaced $* with 'idiomatic' "$@" to serve as better example.
In [1], Vladimir Panteleev observed that the In-Reply-To and
References headers could be wrapped in the 'default' output format of
notmuch-reply, depending on the version of Emacs creating the
message. In my own experiments notmuch-reply sometimes wraps headers
with only one message-id if that message-id is long enough. However it
happens, this causes the previous approach using grep to fail.
Since I found the proposed unwrapping shell fragment in [1] a bit hard
to follow, I decided to write a little python script instead. Then
Tomi suggested a slight generalization of my script, and here we are.
[1] id:20170817175145.3204-7-notmuch@thecybershadow.net
On some system configurations, setting a breakpoint on the "add_file"
function then issuing "continue" in gdb causes the debugger to
seemingly jump over the add_file invocation. This results in a test
failure, as the "Handle files vanishing between scandir and add_file"
subtest expects add_file to be called and fail due to the vanishing
file. The compiler optimization level also plays a role - the problem
can be reproduced with CFLAGS having -O2 but not -Og.
This problem was observed manifesting as a test failure on Travis CI
configured with "dist: trusty" and "sudo: false". It was not
reproducible on a local Docker image of Travis' runtime environment,
so Travis' virtualization infrastructure likely plays a role as well.
* T050-new.sh: Breakpoint notmuch_database_add_message instead of
add_file to the same effect, and avoid bad gdb behaviour on Travis
CI.
Amended by db:
s/notmuch_database_add_message/notmuch_database_index_file/
Somehow the wrapper function doesn't work as a breakpoint; perhaps due
to inlining.
We reorder reading maildir flags to avoid overwriting 'new.tags'. The
inverted status of 'unread' means the maildir flag needs to be checked
a second time.
I backpedalled here on the idea of supporting 'new.tags' without
'unread' in the presence of maildir syncing. For files in 'new/', it
seems quite natural to tag them as 'unread'.
json.tool does not sort or otherwise normalize the order of JSON keys
in its output, which can result in test failures on some test systems.
Instead, use a one-line Python script passed to the interpreter
directly on its command line. Use sort_keys=True for json.dump to
ensure the key order is normalized. The script works with both Python
2 and 3.
* test/test-lib.sh: Update test_expect_equal_json.
We need a way to pass parameters to the indexing functionality on the
first index, not just on reindexing. The obvious place is in
notmuch_database_add_message. But since modifying the argument list
would break both API and ABI, we needed a new name.
I considered notmuch_database_add_message_with_params(), but the
functionality we're talking about doesn't always add a message. It
tries to index a specific file, possibly adding a message, but
possibly doing other things, like adding terms to an existing message,
or failing to deal with message objects entirely (e.g. because the
file didn't contain a message).
So i chose the function name notmuch_database_index_file.
I confess i'm a little concerned about confusing future notmuch
developers with the new name, since we already have a private
_notmuch_message_index_file function, and the two do rather different
things. But i think the added clarity for people linking against the
future libnotmuch and the capacity for using index parameters makes
this a worthwhile tradeoff. (that said, if anyone has another name
that they strongly prefer, i'd be happy to go with it)
This changeset also adjusts the tests so that we test whether the new,
preferred function returns bad values (since the deprecated function
just calls the new one).
We can keep the deprecated n_d_add_message function around as long as
we like, but at the next place where we're forced to break API or ABI
we can probably choose to drop the name relatively safely.
NOTE: there is probably more cleanup to do in the ruby and go bindings
to complete the deprecation directly. I don't know those languages
well enough to attempt a fix; i don't know how to test them; and i
don't know the culture around those languages about API additions or
deprecations.
This patch fixes the 'Deleted first duplicate file does not stop
notmuch show from working' test.
If a message to be shown has several duplicated files, and for some
reason the first file in the list is not available anymore, notmuch
will exit with an error.
This is clearly a problem in the database, but we are not going to let
this problem be a show-stopper. Let's walk through the list, and show
the first existing file.
Signed-off-by: Yuri Volchkov <yuri.volchkov@gmail.com>
This patch fixes the "Insert message into folder with trailing /"
test. The problem was insufficient path canonization.
From database's point of view, "Sent" and "Sent/" are different
folders. If user runs (note the last '/'):
notmuch insert --folder=maildir/Sent/ < test.msg
notmuch will create an extra XDIRECTORY record for the folder
'Sent/'. This means that database will have _TWO_ records for _ONE_
physical folder: 'Sent' and 'Sent/'. However, the 'notmuch new'
command will update only records related to the first one (the correct
one).
Now, if user moved the email file (e.g. from 'Sent/new' to
'Sent/cur'), 'notmuch new' will add a record about the new file, but
will not delete the old record.
Signed-off-by: Yuri Volchkov <yuri.volchkov@gmail.com>
From database's point of view, "Drafts" and "Drafts/" are different
folders
Signed-off-by: Yuri Volchkov <yuri.volchkov@gmail.com>
Amended: add test_subtest_known_broken (db)
In the commit 51cd69feb1 the part of the
test "test runs if prerequisite is satisfied" has been
removed. However, there was a remainder of that test - variable
'haveit'.
Kill it, to not to confuse people.
Signed-off-by: Yuri Volchkov <yuri.volchkov@gmail.com>
A leading / in paths in a .gitignore file matches the beginning of the
path, meaning that for patterns without slashes, git will match files
only in the current directory as opposed to in any subdirectory.
Prefix relevant paths with / in .gitignore files, to prevent
accidentally ignoring files in subdirectories and possibly slightly
improve the performance of "git status".
971cdc72cd renamed corpus.mail to
corpora.mail. Although 971cdc72cd
updated some of the remaining corpus.mail references, two remained,
causing the test suite to leave behind an unignored corpora.mail
directory.
The corresponding xapian document just gets more terms added to it,
but this doesn't seem to break anything. Values on the other hand get
overwritten, which is a bit annoying, but arguably it is not worse to
take the values (from, subject, date) from the last file indexed
rather than the first.
There are many other problems that could be tested, but these ones we
have some hope of fixing because it doesn't require UI changes, just
indexing changes.
There are some cases like remote usage where this might cause
problems, but those users can easily customize the variable. The
inconvenience seems to be outweighed by the security benefit for most
users.
gmime 3.0 no longer offers a means to set the path for gpg.
Users can set $PATH anyway if they want to pick a
differently-installed gpg (e.g. /usr/local/bin/gpg), so this isn't
much of a reduction in functionality.
The one main difference is for people who have tried to use "gpg2" to
make use of gpg 2.1, but that isn't usefully co-installable anyway.
The reply-to munging code might behave differently whether there's an
exact match on the strings or not, or whether the string is a raw
addr-spec instead of an name-addr. These tests cover those variations
(i also had to tweak json output further below when this new test was
added).
Since the error field is unused by the emacs front end, no changes are
needed other than bumping the format version number.
As it is, this is a bit overengineered, but it will reduce duplication
when we support gmime 3.0
This is another case where the behaviour of gmime-2.6 and gmime-3.0
seems to differ. It may be that we prefer the more lax parsing of the
previous version, but that should be tested separately.
Since gnupg 2.1.20, gpg-agent no longer shut itself down when
$GNUPGHOME directory is removed.
Add exit hooks to the test modules which execute `gpgconf --kill all`
Add exit hooks to execute `gpgconf --kill all` in the modules that
create $GNUPGHOME for gpg to work with.
New function at_exit_function registers given function to be called
at script termination.
Functions so registered are called in the reverse order of their
registration; no arguments are passed.
Function is called only once; re-adding with function name already
registered will remove previous registration.
New function rm_exit_function can be used to remove registration.
Modules (and possibly test-lib.sh functions) in future commits will
register such functions.
The dynamic generation of the linker version script for libnotmuch
exports has grown rather complicated.
Reverse the visibility control by hiding symbols by default using
-fvisibility=hidden, and explicitly exporting symbols in notmuch.h
using #pragma GCC visibility. (We could also use __attribute__
((visibility ("default"))) for each exported function, but the pragma
is more convenient.)
The above is not quite enough alone, as it would "leak" a number of
weak symbols from Xapian and C++ standard library. Combine it with a
small static version script that filters out everything except the
notmuch_* symbols that we explicitly exposed, and the C++ RTTI
typeinfo symbols for exception handling.
Finally, as the symbol hiding test can no longer look at the generated
symbol table, switch the test to parse the functions from notmuch.h.
From a UI perspective this looks similar to what was already provided
for from, subject, and mid, but the implementation is quite
different. It uses the database's list of terms to construct a term
based query equivalent to the passed regular expression.
'quite' on IRC reported that notmuch new was grinding to a halt during
initial indexing, and we eventually narrowed the problem down to some
html parts with large embedded images. These cause the number of terms
added to the Xapian database to explode (the first 400 messages
generated 4.6M unique terms), and of course the resulting terms are
not much use for searching.
The second test is sanity check for any "improved" indexing of HTML.
These 210 messages are in several long threads, which is good for
testing our threading code, and may be useful just as a larger test
corpus in the future.
The non-field processor behaviour is is convert the corresponding
queries into a search for the unprefixed terms. This yields pretty
surprising results so I decided to generate a query that would match
the terms (i.e. none with that prefix) generated for an empty header.
Make test-lib-common.sh load test-lib-<$PLATFORM>.sh to create
additional shim for platform specifics.
Use test-lib-FREEBSD.sh to call GNU utilities instead of native ones.
- amended by db following Tomi's suggestions
The argument is that if the string passed to the field processor has
no spaces, then the added quotes won't have any benefit except for
disabling wildcards. But disabling wildcards doesn't seem very useful
in the normal Xapian query parser, since they're stripped before
generating terms anyway. It does mean that the query 'from:"foo*"' will
not be precisely equivalent to 'from:foo' as it is for the non
field-processor version.
This was broken by the addition of regexp searching. The detection of
wildcards is not currently done in the recursive call to parse_query,
because of quoting issues.
This function was deprecated in notmuch 0.21. We re-use the name for
a status returning version, and deprecate the _st name. One or two
remaining uses of the (removed) non-status returning version fixed at
the same time
Apparently some systems (MacOS?) have a system library called libutil
and the name conflict causes problems. Since this library is quite
notmuch specific, rename it to something less generic.
Often Linux systems are shipped with wget(1) by default (and no curl).
Many BSDs, macOS, and e.g. some Linux minimal/container images
comes with curl(1) (and no wget).
Attempting to download with curl if wget is not available increases
the likelihood for this to succeed.
For reasons not completely understood at this time, gmime (as of
2.6.22) is returning a date before 1900 on bad date input. Since this
confuses some other software, we clamp such dates to 0,
i.e. 1970-01-01.
Some gdb python exceptions on some os environments (e.g. macOS
Sierra, non-codesigned gdb) do not make gdb exit (but to drop down
to nonexistent command line?).
Mitigate this chance by explict SystemExit on all exceptions.
The contents of output file 'gdb.out' is unchanged.
Unify the subtests by requiring test_begin_subtest before
test_expect_code. (Similar change for test_expect_success has already
been done.)
This increases clarity in the test scripts by having a separate line
for the start of the subtest with the heading, and makes it possible
to simplify the test infrastructure by making all subtests similar.
Unify the subtests by requiring test_begin_subtest before
test_expect_success. (Similar change for test_expect_code will
follow.)
This increases clarity in the test scripts by having a separate line
for the start of the subtest with the heading, and makes it possible
to simplify the test infrastructure by making all subtests similar.
The only place where we use the implicit prereq check is T000-basic.sh
where we check that it works. It's an added complication that we don't
use. Remove it.
The test_have_prereq function can still be used for the same effect in
subtests that use test_begin_subtest. For now, this will make it
impossible to have prereqs in one-line subtests that don't require
test_begin_subtest. This will be fixed in follow-up work.
Known broken tests are, well, known broken. Do not print the result
diff for them unless V=1 is specified. Now that the test description
is printed also when known broken tests fail, the user can also skip
to running the individual failing tests.
Headers of more than 998 characters should be folded when sending.
However, until recently, emacs did not do this.
This adds a (known broken) test for this when sending messages in
emacs. We will backport the fix to notmuch-emacs in the next
changeset.
We filter added exclude at add time, rather than modifying the query by
count search. As noted in the comments, there are several ignored
conditions here.
The main goal is to prepare the way for non-destructive (or at least
less destructive) exclude tag handling. It does this by having a
pre-parsed query available for further processing. This also allows us
to provide slightly more precise error messages.
mid: is the url scheme suggested by URL 2392. We also plan to
introduce more flexible searches for mid: than are possible with
id: (in order not to break assumptions about the special behaviour of
id:, e.g. identifying at most one message).
the idea is that you can run
% notmuch search subject:/<your-favourite-regexp>/
% notmuch search from:/<your-favourite-regexp>/
or
% notmuch search subject:"your usual phrase search"
% notmuch search from:"usual phrase search"
This feature is only available with recent Xapian, specifically
support for field processors is needed.
It should work with bindings, since it extends the query parser.
This is easy to extend for other value slots, but currently the only
value slots are date, message_id, from, subject, and last_mod. Date is
already searchable; message_id is left for a followup commit.
This was originally written by Austin Clements, and ported to Xapian
field processors (from Austin's custom query parser) by yours truly.
We already use this directory for dtach sockets, so it makes sense to
put gnupg sockets there as well. There doesn't seem to be a clean way
to put a fully functional socket in a different location than
GNUPGHOME.
This reverts commit e7b88e8b0a.
It turns out that this does not work well in environments without a
running systemd (or some other provider of /run/user)
Apparently our test system does not use the same flags for compiling
tests as it does for compiling notmuch. Make the test compatible with
C89. Also remove one unused loop index.
Instead of just having the first filename for the message, list all
duplicate filenames of the message as a list in the formatted
outputs. This bumps the format version to 3.
The retries are hardcoded to a small number, and error handling aborts
than propagating errors from notmuch_database_reopen. These are both
somewhat justified by the assumption that most things that can go
wrong in Xapian::Database::reopen are rare and fatal. Here's the brief
discussion with Xapian upstream:
24-02-2017 08:12:57 < bremner> any intuition about how likely
Xapian::Database::reopen is to fail? I'm catching a
DatabaseModifiedError somewhere where handling any further errors is
tricky, and wondering about treating a failed reopen as as "the
impossible happened, stopping"
24-02-2017 16:22:34 < olly> bremner: there should not be much scope for
failure - stuff like out of memory or disk errors, which are probably a
good enough excuse to stop
$NOTMUCH_PYTHON is sourced from sh.config, configured by
./configure and stated to be used as:
"Name of python command to use in configure and the test suite."
This enables the shortened socket pathes in /run or equivalent. The
explicit call to gpgconf is needed for nonstandard GNUPGHOME settings.
(amended according to id:m2fujatr4k.fsf@guru.guru-group.fi)
Attempt to distinguish between errors indicating misconfiguration or
programmer error, which we consider "permanent", in the sense that
automatic retries are unlikely to be useful, and those indicating
transient error conditions. We consider XAPIAN_EXCEPTION transient
because it covers the important special case of locking failure.
Running `gdb command < input` is not as reliable way to give input
to the command (some installations of gdb consume it). Use "set args"
gdb command to have input redirected at gdb 'run' time.
GnuPG 2.1.16 is now injecting the full issuer fingerprint in its
signatures, which makes them about 32 octets larger when
ascii-armored.
This change in size means that the size of the MIME parts will vary
depending on the version of gpg that the user has installed. at any
rate, the signature part should be non-zero (this is true for
basically any MIME part), so we just test for that instead of an exact
size.
emacs24 and emacs23 have different secure tag defaults: in particular,
mml-secure-message-sign only signs the part on emacs23 but the whole
message on emacs24. This difference makes one of the draft tests fail
(which causes a cascade of later failures) on emacs23. It seems that
travis uses emacs23 so it is useful to fix this.
We do this by forcing the whole message to be signed in either case --
the code snippet is extracted from mml-secure-message-sign on emacs24.
Provide functionality to resume editing a message previously saved with
notmuch-draft-save, including decoding the X-Notmuch-Emacs-Secure
header.
Resume gets the raw file from notmuch and using the emacs function
mime-to-mml reconstructs the message (including attachments).
'e' is bound to resume a draft from show or tree mode.
This provides initial support for postponing in the emacs frontend;
resuming will follow in a later commit. On saving/postponing it uses
notmuch insert to put the message in the notmuch database
Current bindings are C-x C-s to save a draft, C-c C-p to postpone a
draft (save and exit compose buffer).
Previous drafts get tagged deleted on subsequent saves, or on the
message being sent.
Each draft gets its own message-id, and we use the namespace
draft-.... for draft message ids (so, at least for most people, drafts
are easily distinguisable).
Moved the 2 basename(1) executions to the test failure branch in
test_expect_equal_file ().
The output of basename(1) executions in function test_expect_equal_file ()
are only used when tests fails -- when all tests pass these 2 basename(1)
executions are no longer done at all.
Thanks to Lucas (id:147263183913.27784.12274024193186585889@mbp) for the
bug report and the test case.
I decided to use the python version because the python bindings could
use more exercise.
In case of the test script is to be relaunced under valgrind, or --tee
is requested, use the $BASH shell variable to locate the command
interpreter. The $SHELL variable is re-set by non-interactive shells
so in case the shell uses some other shell (e.g. zsh) for interactive
use these bash scripts continue to work.
This is a strange corner case where the removing of the user's address
from the To: header does the wrong thing. If we think it is
worth (eventually) fixing, this test can serve as a reminder.
We want to be able to query the properties directly, like:
notmuch count property:foo=bar
which should return a count of messages where the property with key
"foo" has value equal to "bar".
This stops the (usually incorrect) sigstatus and encstatus buttons
appearing when replying in emacs, and updates the test suite to match.
Overriding the status button functions is a little unusual but much
less intrusive than passing an argument all the way down the call
chain. It also makes it clear exactly what it does.
We also hide the application/pgp-encrypted part as it can only contain
"Version: 1". We do this in notmuch show, which means it also happens
when replying.
Pass in GMimeMessage to simplify To/Cc/Bcc headers. We'll eventually
remove the notmuch message passing altogether, but keep both for now
to not make too big changes at once.
Getting the headers from GMimeMessage using GMime functions fixes the
error on duplicate Cc headers reported by Daniel Kahn Gillmor
<dkg@fifthhorseman.net> in id:87d1ngv95p.fsf@alice.fifthhorseman.net.
Get rid of an intermediate function.
The small annoyance is the ownership differences in the address lists.
As Daniel Kahn Gillmor <dkg@fifthhorseman.net> reports in
id:87d1ngv95p.fsf@alice.fifthhorseman.net, notmuch show combines
multiple Cc: fields into one, while notmuch reply does not. While such
messages are in violation of RFC 5322, it would be reasonable to
expect notmuch to be consistent. Add a known broken test to document
this expectation.
This also starts a new "broken" corpus for messages which are broken.
Details:
The original message is formatted using the message printing in
notmuch-show.c. For Cc:, it uses g_mime_message_get_recipients(),
which apparently combines all Cc: fields into one internally.
The addresses in the reply headers, OTOH, are based on headers queried
through libnotmuch. It boils down to g_mime_object_get_header() in
lib/message-file.c, which returns only the first occurence of header.
We largely use the corpus under test/corpus for
testing. Unfortunately, many of our tests have grown to depend on
having exactly this set of messages, making it hard to add new message
files for testing specific cases.
We do use a lot of add_message from within the tests, but it's not
possible to use that for adding broken messages, and adding several
messages at once can get unwieldy.
Move the basic corpus under tests/corpora/default, and make it
possible to add new, independent corpora along its side. This means
tons of renames with a few tweaks to add_email_corpus function in
test-lib.sh to let tests specify which corpus to use.
Originally the intent was to make the test more robust against changing
test keys. It turns out that (unscientifically) gpg --with-colons output
changes more often than our test key. Rather than making the script more
complex, just hard code the fingerprint.
This fixes Debian bug #847013; I expect similar test failures as other
distros adopt gnupg 2.1.15
With all the preparation it is now simple to add the actual insert
code. Since insert can fail for many reasons we let the user decide
interactively deal with it.
We modify test-lib.el to set file fcc, so that all the old tests and
emacs_fcc_message from test-lib.sh still work
gdb sometimes writes warnings to stdout, which we don't need/want, and
for some reason --batch-silent isn't enough to hide. So in this commit
we write them to a log file, which is probably better for debugging
anyway. To see an illustrative test failure before this change, run
% make
% touch notmuch-count.c
% cd test && ./T060-count.sh
(cherry picked from commit f45fa5bdd3)
The User-Agent: header can be fun and interesting, but it also leaks
quite a bit of information about the user and their software stack.
This represents a potential security risk (attackers can target the
particular stack) and also an anonymity risk (a user trying to
preserve their anonymity by sending mail from a non-associated account
might reveal quite a lot of information if their choice of mail user
agent is exposed).
This change also avoids hiding the User-Agent header by default, so
that people who decide they want to send it will at least see it (and
can edit it if they want to) before sending.
It makes sense to have safer defaults.
Previously notmuch-wash made the width of the text (approximately) the
window-width minus the depth in thread. This is correct for the
default indentation of 1 per message depth, but is incorrect for any
other setting of notmuch-show-indent-messages-width.
As notmuch-show-indent-messages-width is customisable, and notmuch-tree
sets it to zero to avoid indenting messages in the message pane, this
bug can show up in real use.
Two of the tests had to be updated: when
notmuch-show-indent-messages-width is 0, then the new (correct) word
wrapping happens later, when notmuch-show-indent-messages-width is 4,
then the new word wrapping happens sooner.
gdb sometimes writes warnings to stdout, which we don't need/want, and
for some reason --batch-silent isn't enough to hide. So in this commit
we write them to a log file, which is probably better for debugging
anyway. To see an illustrative test failure before this change, run
% make
% touch notmuch-count.c
% cd test && ./T060-count.sh
This has been wrong since bbbdf0478e, but the race condition was not
previously been (often?) triggered in the tests. With the DB_RETRY_LOCK
patches, it manifests itself as a deadlock.
This support will be present only if the appropriate version of xapian
is available _and_ the user did not disable the feature when
building. So there really needs to be some way for the user to check.
This has been wrong since bbbdf0478e, but the race condition was not
previously been (often?) triggered in the tests. With the DB_RETRY_LOCK
patches, it manifests itself as a deadlock.
A few simple things that applies well to test/notmuch-test.sh
- Shell does pathname expansion also without doing `echo ...` in subshell.
- Redirections >/dev/null 2>/dev/null hide (improbable but) potential
serious errors; adding -f to rm instead.
- Inter-file capitalization consistency in comments.
- Unnecesary space removal.
The trick of having a common header file doesn't work to share between
test scripts, so make an include file in the test directory.
The use of #include <notmuch-test.h> looks slightly pretentious, but
the include file is not actually in the current (temporary) directory.
Place PYTHONPATH to the environment when python is executed in a way
that current shell environment is not affected. This also allows adding
the old value of PYTHONPATH to the end of the new value (otherwise it
would have been appended again and again when test_python is called).
At the same time, use -B option to avoid writing .pyc files to
bindings/python/* (which are not cleared out by distclean).
Drop the (unused) prefix code which preserved the original stdout of the
python program and opened sys.stdout to OUTPUT. In place of that there
is now note how (debug) information can be printed to original stdout.
Previously LD_LIBRARY_PATH was exported (and environment changed)
in the middle of test case execution, when a function setting it
was called.
Previously the old contents of LD_LIBRARY_PATH was lost (if any)
when it was re-set and exported. In some systems the old contents of
LD_LIBRARY_PATH was needed to e.g. locate suitable gmime library.
In scripts that include test-lib-common.sh but not test-lib.sh
the die() implementation needs to be a bit different due to
fd redirection differences. test-lib-common.sh implements die()
only if it was not implemented already.
Added die() function to test-lib.sh with the following first use of it:
If notmuch new fails during email corpus addition the database is
most probably inexistent or broken and the added corpus would be
unusable while running single tests, giving misleading failures
("only" full 'make test' cleans out old corpus).
Many of the external links found in the notmuch source can be resolved
using https instead of http. This changeset addresses as many as i
could find, without touching the e-mail corpus or expected outputs
found in tests.
Most of the infrastructure here is general, only the validation/dispatch
is hardcoded to a particular prefix.
A notable change in behaviour is that notmuch-config now opens the
database e.g. on every call to list, which fails with an error message
if the database doesn't exit yet.
Since xapian provides the ability to restrict the iterator to a given
prefix, we expose this ability to the user. Otherwise we mimic the other
iterator interfances in notmuch (e.g. tags.c).
This is a thin wrapper around the Xapian metadata API. The job of this
layer is to keep the config key value pairs from colliding with other
metadata by transparently prefixing the keys, along with the usual glue
to provide a C interface.
The split of _get_config into two functions is to allow returning of the
return value with different memory ownership semantics.
The Ruby bindings were missing a way to get all the tags of the
database. Now you should be able to access this with the public
instance method `all_tags` of your database object.
Example of use:
notmuchdb = Notmuch::Database.new path, { :create => false,
:mode => Notmuch::MODE_READ_ONLY }
my_tags = notmuchdb.all_tags
my_tags.each { |tag|
print tag
}
my_tags.destroy!
Amended by db: improve error reporting, add test
Files in test directories had only copyright of a single individual,
of which code was adapted here as a base of the test system.
Since then many Notmuch Developers have contributed to the test
system, which is now acknowledged with a constant string in some
of the test files.
The README file in test directory instructed new files contain a
copyright notice, but that has never been done (and it is also not
needed). To simplify things a bit (and lessen confusion) this
instruction is now removed.
As a side enchangement, all of the 3 entries in the whole source
tree cd'ing to `dirname` of "$0" now uses syntax cd "$(dirname "$0")".
This makes these particular lines work when current working directory
is e.g. /c/Program Files/notmuch/test/.
(Probably it would fail elsewhere, though.)
In case of notmuch reply and notmuch show --part=N it is required that
search terms match to one message. If match count was != 1, error
message "Error: search term did not match precisely one message"
was too vague to explain what happened.
By appending (matched <num> messages) to the error message it
makes the problem more understandable (e.g when <num> is '0'
user reckons the query had a typo in it).
To fully complete the ghost-on-removal-when-shared-thread-exists
proposal, we need to clear all ghost messages when the last active
message is removed from a thread.
Amended by db: Remove the last test of T530, as it no longer makes sense
if we are garbage collecting ghost messages.
implement ghost-on-removal, the solution to T590-thread-breakage.sh
that just adds a ghost message after removing each message.
It leaks information about whether we've ever seen a given message id,
but it's a fairly simple implementation.
Note that _resolve_message_id_to_thread_id already introduces new
message_ids to the database, so i think just searching for a given
message ID may introduce the same metadata leakage.
This test (T590-thread-breakage.sh) has known-broken subtests.
If you have a two-message thread where message "B" is in-reply-to "A",
notmuch rightly sees this as a single thread.
But if you:
* remove "A" from the message store
* run "notmuch new"
* add "A" back into the message store
* re-run "notmuch new"
Then notmuch sees the messages as distinct threads.
This happens because if you insert "B" initially (before anything is
known about "A"), then a "ghost message" gets added to the database in
reference to "A" that is in the same thread, which "A" takes over when
it appears.
But if "A" is subsequently removed, no ghost message is retained, so
when "A" appears, it is treated as a new thread.
I see a few options to fix this:
ghost-on-removal
----------------
We could unilaterally add a ghost upon message removal. This has a
few disadvantages: the message index would leak information about what
messages the user has ever been exposed to, and we also create a
perpetually-growing dataset -- the ghosts can never be removed.
ghost-on-removal-when-shared-thread-exists
------------------------------------------
We could add a ghost upon message removal iff there are other
non-ghost messages with the same thread ID.
We'd also need to remove all ghost messages that share a thread when
the last non-ghost message in that thread is removed.
This still has a bit of information leakage, though: the message index
would reveal that i've seen a newer message in a thread, even if i had
deleted it from my message store
track-dependencies
------------------
rather than a simple "ghost-message" we could store all the (A,B)
message-reference pairs internally, showing which messages A reference
which other messages B.
Then removal of message X would require deleting all message-reference
pairs (X,B), and only deleting a ghost message if no (A,X) reference
pair exists.
This requires modifying the database by adding a new and fairly weird
table that would need to be indexed by both columns. I don't know
whether xapian has nice ways to do that.
scan-dependencies
-----------------
Without modifying the database, we could do something less efficient.
Upon removal of message X, we could scan the headers of all non-ghost
messages that share a thread with X. If any of those messages refers
to X, we would add a ghost message. If none of them do, then we would
just drop X entirely from the table.
---------------------
One risk of attempted fixes to this problem is that we could fail to
remove the search term indexes entirely. This test contains
additional subtests to guard against that.
This test also ensures that the right number of ghost messages exist
in each situation; this will help us ensure we don't accumulate ghosts
indefinitely or leak too much information about what messages we've
seen or not seen, while still making it easy to reassemble threads
when messages come in out-of-order.
The code to skip multiple slashes in _notmuch_database_split_path()
skips back one character too much. This is compensated by a +1 in the
length parameter to the strndup() call. Mostly this works fine, but if
the path is to a file under a top level directory with one character
long name, the directory part is mistaken to be part of the file name
(slash == path in code). The returned directory name will be the empty
string and the basename will be the full path, breaking the indexing
logic in notmuch new.
Fix the multiple slash skipping to keep the slash variable pointing at
the last slash, and adjust strndup() accordingly.
The bug was introduced in
commit e890b0cf40
Author: Carl Worth <cworth@cworth.org>
Date: Sat Dec 19 13:20:26 2009 -0800
database: Store the parent ID for each directory document.
just a little over two months after the initial commit in the Notmuch
code history, making this the longest living bug in Notmuch to date.
In several places in the test suite we intentionally corrupt the Xapian
database in order to test error handling. This corruption is specific to
the on-disk organization of the database, and that changed with the
glass backend. We use the previously computed default backend to make
the tests adapt to changing names.
This is mainly for the test suite. We already expect the tests to be
run in the same environment as configure was run, at least to get the
name of the python interpreter. So we are not really imposing a new
restriction.
