Commit graph

1187 commits

Author SHA1 Message Date
Daniel Kahn Gillmor
afb0b11dcd crypto: Avoid pretending to verify signatures on unsigned encrypted mail
Unsigned encrypted mail shows up with a weird empty signature list.
If we successfully decrypted and there was no signature in it, we
should just not show a sigstatus at all.

The documentation for g_mime_decrypt_result_get_signatures says:

    a GMimeSignatureList or NULL if the stream was not signed.
2019-04-26 08:03:15 -03:00
Daniel Kahn Gillmor
22ec4a36e6 tests: move FINGERPRINT definition to add_gnupg_home
If a test has added a GnuPG homedir, it may well want to know the
fingerprint.  This saves us from having to redefine this magic string
in multiple places when more tests eventually use the GnuPG homedir.
2019-04-24 07:16:46 -03:00
David Bremner
319dd95ebb lib: add 'body:' field, stop indexing headers twice.
The new `body:` field (in Xapian terms) or prefix (in slightly
sloppier notmuch) terms allows matching terms that occur only in the
body.

Unprefixed query terms should continue to match anywhere (header or
body) in the message.

This follows a suggestion of Olly Betts to use the facility (since
Xapian 1.0.4) to add the same field with multiple prefixes. The double
indexing of previous versions is thus replaced with a query time
expension of unprefixed query terms to the various prefixed
equivalent.

Reindexing will be needed for 'body:' searches to work correctly;
otherwise they will also match messages where the term occur in
headers (demonstrated by the new tests in T530-upgrade.sh)
2019-04-17 08:48:16 -03:00
Örjan Ekeberg
bfe0f8dcc4 test: add test for checking forwarded messages
Add test of forwarding messages from within emacs.
The first test checks that a references header is properly
added to the new message.  The second test checks that the
send-hook of the forwarding message adds a forwarded-tag
to the original message.
2019-04-14 08:01:30 -03:00
David Edmondson
639d21d5b9 test: Add emacs attachment check tests. 2019-03-28 14:37:21 -03:00
David Bremner
d25dcc589c lib: use phrase search for anything not ending in '*'
Anything that does not look like a wildcard should be safe to
quote. This should fix the problem searching for xapian keywords.
2019-03-28 14:34:37 -03:00
David Bremner
168211c563 test: add two known broken tests searching for xapian keywords
Thanks to plujon for pointing out this problem on IRC. The underlying
issue is that the quotes are stripped before the field processors get
the query string, and the heuristic for putting them back is not quite
right.
2019-03-28 14:32:11 -03:00
Luis Ressel
9f7e851263 Prepend regerror() messages with "regexp error: "
The exact error messages returned by regerror() aren't standardized;
relying on them isn't portable. Thus, add a a prefix to make clear that
the subsequent message is a regexp parsing error, and only look for this
prefix in the test suite, ignoring the rest of the message.
2019-03-11 22:24:55 -03:00
Luis Ressel
a1e649276e test/T030-config: Separate stdout and stderr output
POSIX doesn't specify the flushing behaviour of the STDOUT stream, so
it's invalid to assume a particular order between the stdout and stderr
output. The current test breaks on musl due to this.
2019-03-11 22:13:54 -03:00
David Bremner
71eaa19350 Merge branch 'release'
Changes from 0.28.3
2019-03-06 08:53:26 -04:00
David Bremner
e88297c072 lib/string_map: fix return type of string_cmp
I can't figure out how checking the sign of a bool ever worked. The
following program demonstrates the problem (i.e. for me it prints 1).

 #include <stdio.h>
 #include <stdbool.h>
 int main(int argc, char **argv) {
    bool x;
    x = -1;
    printf("x = %d\n", x);
 }

This seems to be mandated by the C99 standard 6.3.1.2.
2019-03-05 21:46:41 -04:00
David Bremner
09595cf577 lib: Add known broken test for string_map binary search.
Because the string_map functions are not exported, we test via message
properties.
2019-03-05 21:46:41 -04:00
rhn
b350c688e1 test: Check for replies obeying lifetime guarantees
The test attempts to check that a message coming from a thread outlives its messages list and gets destroyed together with the thread.
2019-01-25 20:37:26 -04:00
David Edmondson
576cff7654 reply: Include sender as recipient if they were the original recipient
When generating a reply message, if the user was the originator and
only recipient of the original message, include the user as a
recipient of the reply.
2018-12-07 08:03:50 -04:00
Maxime Coste
baa4185c30 cli: notmuch show support for --include-html with --format=text 2018-11-24 09:12:30 -04:00
Maxime Coste
37e5bc00ae cli: notmuch show support for --body=false with --format=text 2018-11-24 09:11:42 -04:00
Daniel Kahn Gillmor
311f530013 test: git should ignore message-id-parse
b31e44c678 introduced message-id-parse
as a new binary created by the test suite.  It shows up as something
additional to git, but git ought to know to ignore it.
2018-10-08 12:50:03 -03:00
David Edmondson
e8cb6b2cd6 emacs: Call `notmuch-mua-send-hook' hooks when sending a message
Previously any hook functions attached to `notmuch-mua-send-hook' were
ignored.
2018-09-28 20:22:42 -03:00
David Edmondson
4e213fe9b4 test: Check that `notmuch-mua-send-hook' is called on sending a message 2018-09-28 20:22:33 -03:00
David Bremner
c846e15ffe test: make regexp test conditional on field processors
Normally we'd mark it broken, but perversely missing regexp support
actually makes the test pass.
2018-09-14 08:54:20 -03:00
David Bremner
c6ba5522a5 test: mark thread subqueries broken without field processors
Currently these tests just fail when notmuch is built against Xapian
1.2.x
2018-09-14 08:54:09 -03:00
David Edmondson
4cd5a0a3d5 test: Absolute and relative directory paths. 2018-09-08 20:19:42 -03:00
David Bremner
87934c432c lib: change parent strategy to use In-Reply-To if it looks sane
As reported by Sean Whitton, there are mailers (in particular the
Debian Bug Tracking System) that have sensible In-Reply-To headers,
but un-useful-for-notmuch References (in particular with the BTS, the
oldest reference is last). I looked at a sample of about 200K
messages, and only about 0.5% these had something other than a single
message-id in In-Reply-To. On this basis, if we see a single
message-id in In-Reply-To, consider that as authoritative.
2018-09-06 08:07:13 -03:00
David Bremner
b31e44c678 lib: add _notmuch_message_id_parse_strict
The idea is that if a message-id parses with this function, the MUA
generating it was probably sane, and in particular it's probably safe
to use the result as a parent from In-Reply-to.
2018-09-06 08:07:13 -03:00
David Bremner
35053c2b9a test/thread-replies: mangle In-Reply-To's
In a future commit, we will start trusting In-Reply-To's when they
look sane (i.e. a single message-id). Modify these tests so they will
keep passing (i.e. keep choosing References) when that happens.
2018-09-06 08:07:13 -03:00
David Bremner
ea08032ae4 test: add known broken test for good In-Reply-To / bad References
The current scheme of choosing the replyto (i.e. the default parent
for threading purposes) does not work well for mailers that put
the oldest Reference last.
2018-09-06 08:07:13 -03:00
David Bremner
46dce33abc lib/thread: change _resolve_thread_relationships to use depths
We (finally) implement the XXX comment. It requires a bit of care not
to reparent all of the possible toplevel messages.

_notmuch_messages_has_next is not ready to be a public function yet,
since it punts on the mset case. We know in the one case it is called,
the notmuch_messages_t is just a regular list / iterator.
2018-09-06 08:07:13 -03:00
David Bremner
a330858284 lib/thread: initial use of references as for fallback parenting
This is mainly to lay out the structure of the final code. The problem
isn't really solved yet, although some very simple cases are
better (hence the fixed test). We need two passes through the messages
because we need to be careful not to re-parent too many messages and
end up without any toplevel messages.
2018-09-06 08:07:13 -03:00
David Bremner
9b568e73e1 lib/thread: sort sibling messages by date
For non-root messages, this should not should anything currently, as
the messages are already added in date order. In the future we will
add some non-root messages in a second pass out of order and the
sorting will be useful. It does fix the order of multiple
root-messages (although it is overkill for that).
2018-09-06 08:07:12 -03:00
David Bremner
ccb52edb4c test: add known broken tests for "ghost roots"
This documents the bug discussed at

     id:87efgmmysi.fsf@len.workgroup

The underlying issue is that the reply to a ghost (missing) message is
falsely classified as a root message in _resolve_thread_relationships.

There are two pairs of tests; in each case the the first test is
simpler / more robust, but also easier to fool.
2018-09-06 08:07:12 -03:00
David Bremner
ebd131ac07 test: start threading test corpus
There are 3 threads here, two synthetic, and one anonymized one using
data from Gregor. They test various aspects of thread
ordering/construction in the presence of replies to ghost messages.
2018-09-06 08:07:12 -03:00
David Bremner
dfda1745bf test/tag: add test for updating during batch tag operations
This clarifies that the breakage seen with Xapian 1.4.6 does not have
to do with "funny" tags.

This test is "known broken", but only with xapian 1.4.6, and there's
curently no convenient way to mark that.
2018-08-29 06:37:48 -03:00
Sebastian Poeplau
0f08bf7166 lib: detect mislabeled Windows-1252 parts
Use GMime functionality to detect mislabeled messages and apply the
correct (Windows) encoding instead.
2018-08-29 06:34:39 -03:00
Sebastian Poeplau
7074bb8f80 test: add known broken test for mislabeled Windows-1252 encoding
Messages that contain Windows-1252 are frequently mislabeled as ISO
8859-1, which may result in non-printable characters when displaying
the message. The test asserts that such characters (in this case
curved quotes) are displayed correctly.
2018-08-29 06:28:13 -03:00
David Bremner
46cf1a98f3 test: fix hardcoded paths in T050-new.sh
The previous paths will fail almost everywhere.
2018-05-26 16:46:44 -07:00
Daniel Kahn Gillmor
aa605f7e8a cli/show: enable --decrypt=stash
Add fancy new feature, which makes "notmuch show" capable of actually
indexing messages that it just decrypted.

This enables a workflow where messages can come in in the background
and be indexed using "--decrypt=auto".  But when showing an encrypted
message for the first time, it gets automatically indexed.

This is something of a departure for "notmuch show" -- in particular,
because it requires read/write access to the database.  However, this
might be a common use case -- people get mail delivered and indexed in
the background, but only want access to their secret key to happen
when they're directly interacting with notmuch itself.

In such a scenario, they couldn't search newly-delivered, encrypted
messages, but they could search for them once they've read them.

Documentation of this new feature also uses a table form, similar to
that found in the description of index.decrypt in notmuch-config(1).

A notmuch UI that wants to facilitate this workflow while also
offering an interactive search interface might instead make use of
these additional commands while the user is at the console:

Count received encrypted messages (if > 0, there are some things we
haven't yet tried to index, and therefore can't yet search):

     notmuch count tag:encrypted and \
         not property:index.decryption=success and \
         not property:index.decryption=failure

Reindex those messages:

     notmuch reindex --try-decrypt=true tag:encrypted and \
         not property:index.decryption=success and \
         not property:index.decryption=failure
2018-05-26 07:43:30 -07:00
Daniel Kahn Gillmor
9d114a8552 test-lib: add notmuch_show_part for "notmuch show --format=text"
Thanks to David Bremner for this improved readability!
2018-05-26 07:42:28 -07:00
David Bremner
84b2963726 test: use --full-scan in T050-new.sh
Wherever the test relies on directories being scanned, this option
should be used to avoid skipping them due to mtimes on directories
matching the database.
2018-05-22 09:29:59 -07:00
David Bremner
d1acff6153 test: add tests for notmuch new --full-scan
Most of these just check that adding the flag does not break existing
functionality. The one test that does check the full-scan
functionality had to be rewritten to output debugging info.
2018-05-22 09:29:48 -07:00
Thomas Schneider
b2e4778ea4 build: Allow user to specify ruby executable
This way, one can build for a different Ruby than $PATH/ruby
(e. g. different versions, or Ruby in other paths).

Signed-off-by: Thomas Schneider <qsx@chaotikum.eu>
2018-05-10 21:01:06 -03:00
David Bremner
5ad39ebf75 lib: add thread subqueries.
This change allows queries of the form

 thread:{from:me} and thread:{from:jian} and not thread:{from:dave}

This is still somewhat brute-force, but it's a big improvement over
both the shell script solution and the previous proposal [1], because it
does not build the whole thread structure just generate a
query. A further potential optimization is to replace the calls to
notmuch with more specialized Xapian code; in particular it's not
likely that reading all of the message metadata is a win here.

[1]: id:20170820213240.20526-1-david@tethera.net
2018-05-07 08:42:53 -03:00
Florian Klink
140e8fb118 T460-emacs-tree: sync $EXPECTED with test name
This syncs the $EXPECTED directory name with the tests filename.

Signed-off-by: Florian Klink <flokli@flokli.de>
2018-05-03 21:03:33 -03:00
Daniel Kahn Gillmor
c20a5eb805 move more http -> https
Correct URLs that have crept into the notmuch codebase with http://
when https:// is possible.

As part of this conversion, this changeset also indicates the current
preferred upstream URLs for both gmime and sup.  the new URLs are
https-enabled, the old ones are not.

This also fixes T310-emacs.sh, thanks to Bremner for catching it.
2018-05-03 20:59:20 -03:00
David Bremner
4ac23c5978 test: add known broken test for regexp search of second subject
We expect this to give the same answer as the non-regexp subject
search. It does not because the regexp search relies on the value
slot, which currently contains only one subject.
2018-05-03 07:44:49 -03:00
David Bremner
f3f4297c7f test: tests for python bindings get_property / get_properties
These roughly replicate the equivalent C tests, although they rely on
the database state created by the former tests, since the python
bindings currently provide read-only access to properties.
2018-05-02 20:30:51 -03:00
David Bremner
963ccabe93 Merge branch 'release'
reference loop fixes to be included in 0.26.2
2018-04-25 17:36:32 -03:00
David Bremner
e8ac065a50 test: re-enable disabled test in T700-reindex.sh
The extra test_done looks like a typo
2018-04-24 22:59:21 -03:00
David Bremner
491b1f4b40 lib: choose oldest message when breaking reference loops
This preserves a sensible thread order
2018-04-23 23:00:20 -03:00
David Bremner
4e085b6d92 test: add known broken test for thread ordering from a loop
The previous loop handling code chooses the last message in the
message list, which turns out to be the last in date order.
See the comment in _notmuch_thread_create.
2018-04-23 23:00:20 -03:00
David Bremner
9293d6da27 lib: break reference loop by choosing arbitrary top level msg
Other parts of notmuch (e.g. notmuch show) expect each thread to
contain at least one top level message, and crash if this expectation
is not met.
2018-04-20 11:23:31 -03:00
David Bremner
ab55ca8e0a test: add known broken test for indexing an In-Reply-To loop.
This documents the bug discussed in

     id:87d10042pu.fsf@curie.anarc.at
2018-04-20 11:23:31 -03:00
David Bremner
044cbd920c test: two new messages for the 'broken' corpus
These have an 'In-Reply-To' loop, which currently confuses "notmuch
new".
2018-04-20 11:23:31 -03:00
Georg Faerber
c117306f2d Fix typos as found by codespell
Signed-off-by: Georg Faerber <georg@riseup.net>
2018-03-24 20:09:54 -03:00
Daniel Kahn Gillmor
b6e3efde05 cli/insert: add --world-readable flag
In some cases (e.g. when building a publicly-visible e-mail archive)
it doesn't make any sense to restrict visibility of the message to the
current user account.

This adds a --world-readable boolean option for "notmuch insert", so
that those who want to archive their mail publicly can feed their
archiver with:

    notmuch insert --world-readable

Other local delivery agents (postfix's local, and dovecot's lda) all
default to delivery in mode 0600 rather than relying on the user's
umask, so this fix doesn't change the default.

Also, this does not override the user's umask.  if the umask is
already set tight, it will not become looser as the result of passing
--world-readable.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2018-03-24 20:08:11 -03:00
Daniel Kahn Gillmor
54982e520c fix typos 2018-01-04 20:35:58 -04:00
Daniel Kahn Gillmor
af8255fb71 cli/reply: make --decrypt take a keyword
This brings the --decrypt argument to "notmuch reply" into line with
the other --decrypt arguments (in "show", "new", "insert", and
"reindex").  This patch is really just about bringing consistency to
the user interface.

We also use the recommended form in the emacs MUA when replying, and
update test T350 to match.
2017-12-29 16:45:55 -04:00
Daniel Kahn Gillmor
8ea4a99d74 cli/show: make --decrypt take a keyword.
We also expand tab completion for it, update the emacs bindings, and
update T350, T357, and T450 to match.

Make use of the bool-to-keyword backward-compatibility feature.
2017-12-29 16:45:46 -04:00
Daniel Kahn Gillmor
0ada2a05c9 cli: some keyword options can be supplied with no argument
We might change some notmuch command line tools that used to be
booleans into keyword arguments.

In that case, there are some legacy tools that will expect to be able
to do "notmuch foo --bar" instead of "notmuch foo --bar=baz".

This patch makes it possible to support that older API, while
providing a warning and an encouragement to upgrade.
2017-12-29 16:45:35 -04:00
Daniel Kahn Gillmor
6aec7a76b9 python: add decrypt_policy argument to Database.index_file()
We adopt a pythonic idiom here with an optional argument, rather than
exposing the user to the C indexopts object directly.

This now includes a simple test to ensure that the decrypt_policy
argument works as expected.
2017-12-24 09:47:35 -04:00
Daniel Kahn Gillmor
b1de8e885d test: final named query test works regardless of Xapian FieldProcessor
This test passes with older versions of Xapian as well, because
neither query returns any results.

This should resolve the travis build failure at
https://travis-ci.org/notmuch/notmuch/builds/318571658

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2017-12-21 09:35:20 -04:00
David Bremner
7cfa1c6961 lib: return "" rather than NULL from notmuch_thread_get_authors
The current behaviour is at best under-documented. The modified test in
T470-missing-headers.sh previously relied on printf doing the right
thing with NULL, which seems icky.

The use of talloc_strdup here is probably overkill, but it avoids
having to enforce that thread->authors is never mutated outside
_resolve_thread_authors_string.
2017-12-21 09:22:30 -04:00
Daniel Kahn Gillmor
8e1c3fa5bb test: retrying lock tests are broken when built against older Xapian
If we're building against a version of Xapian that doesn't offer
retrying the lock, we should be honest and describe the tests as
broken, rather than marking them as missing a test prerequisite.

missing test prerequisites should be for specific components of the
test harness that are missing, not for the backend library notmuch
uses.
2017-12-19 07:14:13 -04:00
Daniel Kahn Gillmor
bace15c275 test: named query tests are broken when missing Xapian FieldProcessor
Named queries don't work without Xapian FieldProcessor.  Rather than
silently skipping them, we should explictly mark them as broken when
building against an older version of Xapian.
2017-12-19 07:14:07 -04:00
Daniel Kahn Gillmor
776a2982ae test: searching by absolute date is broken without Xapian FieldProcessor
Previously, the test suite had simply silently skipped the absolute
date test if we're using an archaic version of Xapian.  For
correctness, we should instead mark the test as broken.

This also changes from string to numeric comparison when checking
NOMTUCH_HAVE_XAPIAN_FIELD_PROCESSOR for consistency with other tests.
2017-12-19 07:14:03 -04:00
l-m-h@web.de
660f1a5a33 test: Add test to unset config items with the python bindings 2017-12-19 06:42:50 -04:00
l-m-h@web.de
3444c731d2 python: Rename get_config_list to get_configs
The old name has a bit of a feeling of hungarian notation.  Also many
generators in the core are named with the suffix "s" to indicate
iterables: dict.items, dict.keys for example.
2017-12-19 06:42:11 -04:00
l-m-h@web.de
7a07fd8625 test: Add tests for new python bindings
The tests where adopted from the tests for the corresponding C functions
in test/T590-libconfig.sh.
2017-12-19 06:41:23 -04:00
Jani Nikula
7354d6d8f6 test: test regexp based new.ignore
Just some basics.
2017-12-15 07:55:02 -04:00
Jani Nikula
f3fc97c000 cli: add support for only printing the addresses in notmuch address
The notmuch address output is much more useful for scripts with just
the addresses printed. Support this using the --output=address option.
2017-12-14 21:28:50 -04:00
Jani Nikula
ab4b033c70 test: expand argument parsing sanity checks
Test the various boolean formats and --no- prefixed boolean and
keyword flag arguments.
2017-12-13 08:31:03 -04:00
David Bremner
040c3236af Merge branch 'release'
Conflicts:
        NEWS

Add in NEWS from point release
2017-12-08 22:19:06 -04:00
David Bremner
151b2c3747 test/emacs: add exploit mitigation test
This test will pass if either the notmuch show mitigation code is
working correctly, or upstream emacs mime handling code has it's own
fix for https://bugs.gnu.org/28350.
2017-12-08 20:36:20 -04:00
Daniel Kahn Gillmor
cb855d8a9d crypto: signature verification reports valid User IDs
When i'm trying to understand a message signature, i care that i know
who it came from (the "validity" of the identity associated with the
key), *not* whether i'm willing to accept the keyholder's other
identity assertions (the "trust" associated with the certificate).

We've been reporting User ID information based on the "trust"
associated with the certificate, because GMime didn't clearly expose
the validity of the User IDs.

This change relies on fixes made in GMime 3.0.3 and later which
include https://github.com/jstedfast/gmime/pull/18.
2017-12-08 20:35:18 -04:00
Daniel Kahn Gillmor
fccebbaeef crypto: add --decrypt=nostash to avoid stashing session keys
Here's the configuration choice for people who want a cleartext index,
but don't want stashed session keys.

Interestingly, this "nostash" decryption policy is actually the same
policy that should be used by "notmuch show" and "notmuch reply",
since they never modify the index or database when they are invoked
with --decrypt.

We take advantage of this parallel to tune the behavior of those
programs so that we're not requesting session keys from GnuPG during
"show" and "reply" that we would then otherwise just throw away.
2017-12-08 08:08:47 -04:00
Daniel Kahn Gillmor
29648a137c crypto: actually stash session keys when decrypt=true
If you're going to store the cleartext index of an encrypted message,
in most situations you might just as well store the session key.
Doing this storage has efficiency and recoverability advantages.

Combined with a schedule of regular OpenPGP subkey rotation and
destruction, this can also offer security benefits, like "deletable
e-mail", which is the store-and-forward analog to "forward secrecy".

But wait, i hear you saying, i have a special need to store cleartext
indexes but it's really bad for me to store session keys!  Maybe
(let's imagine) i get lots of e-mails with incriminating photos
attached, and i want to be able to search for them by the text in the
e-mail, but i don't want someone with access to the index to be
actually able to see the photos themselves.

Fret not, the next patch in this series will support your wacky
uncommon use case.
2017-12-08 08:08:47 -04:00
Daniel Kahn Gillmor
6a9626a2fd cli/reindex: destroy stashed session keys when --decrypt=false
There are some situations where the user wants to get rid of the
cleartext index of a message.  For example, if they're indexing
encrypted messages normally, but suddenly they run across a message
that they really don't want any trace of in their index.

In that case, the natural thing to do is:

   notmuch reindex --decrypt=false id:whatever@example.biz

But of course, clearing the cleartext index without clearing the
stashed session key is just silly.  So we do the expected thing and
also destroy any stashed session keys while we're destroying the index
of the cleartext.

Note that stashed session keys are stored in the xapian database, but
xapian does not currently allow safe deletion (see
https://trac.xapian.org/ticket/742).

As a workaround, after removing session keys and cleartext material
from the database, the user probably should do something like "notmuch
compact" to try to purge whatever recoverable data is left in the
xapian freelist.  This problem really needs to be addressed within
xapian, though, if we want it fixed right.
2017-12-08 08:08:47 -04:00
Daniel Kahn Gillmor
076f86025d cli/new, insert, reindex: change index.decrypt to "auto" by default
The new "auto" decryption policy is not only good for "notmuch show"
and "notmuch reindex".  It's also useful for indexing messages --
there's no good reason to not try to go ahead and index the cleartext
of a message that we have a stashed session key for.

This change updates the defaults and tunes the test suite to make sure
that they have taken effect.
2017-12-08 08:08:46 -04:00
Daniel Kahn Gillmor
a1260896f6 cli/show: use decryption policy "auto" by default.
When showing a message, if the user doesn't specify --decrypt= at all,
but a stashed session key is known to notmuch, notmuch should just go
ahead and try to decrypt the message with the session key (without
bothering the user for access to their asymmetric secret key).

The user can disable this at the command line with --decrypt=false if
they really don't want to look at the e-mail that they've asked
notmuch to show them.

and of course, "notmuch show --decrypt" still works for accessing the
user's secret keys if necessary.
2017-12-08 08:08:46 -04:00
Daniel Kahn Gillmor
6802b333eb cli/reply: use decryption policy "auto" by default.
If the user doesn't specify --decrypt= at all, but a stashed session
key is known to notmuch, when replying to an encrypted message,
notmuch should just go ahead and decrypt.

The user can disable this at the command line with --decrypt=false,
though it's not clear why they would ever want to do that.
2017-12-08 08:08:46 -04:00
Daniel Kahn Gillmor
e4890b5bf9 crypto: new decryption policy "auto"
This new automatic decryption policy should make it possible to
decrypt messages that we have stashed session keys for, without
incurring a call to the user's asymmetric keys.
2017-12-08 08:07:53 -04:00
Daniel Kahn Gillmor
798aa789b5 lib: convert notmuch decryption policy to an enum
Future patches in this series will introduce new policies; this merely
readies the way for them.

We also convert --try-decrypt to a keyword argument instead of a boolean.
2017-12-08 08:07:02 -04:00
Daniel Kahn Gillmor
d3964e81ac indexing: Change from try_decrypt to decrypt
the command-line interface for indexing (reindex, new, insert) used
--try-decrypt; and the configuration records used index.try_decrypt.
But by comparison with "show" and "reply", there doesn't seem to be
any reason for the "try" prefix.

This changeset adjusts the command-line interface and the
configuration interface.

For the moment, i've left indexopts_{set,get}_try_decrypt alone.  The
subsequent changeset will address those.
2017-12-08 08:05:53 -04:00
Daniel Kahn Gillmor
9d23ebfcc3 test: session keys are known broken without session key support
If the version of GMime we're building against doesn't support session
key extraction or re-use, mark the tests that rely on session key
capabilities as known-broken.

This should resolve test suite failures on ubuntu trusty and debian
jessie and earlier, which have GMime 2.6.20 -- session key support was
introduced in GMime 2.6.21.
2017-12-05 20:09:25 -04:00
Daniel Kahn Gillmor
6d7249fec9 crypto: Test restore of cleartext index from stashed session keys
If you've got a notmuch dump that includes stashed session keys for
every decrypted message, and you've got your message archive, you
should be able to get back to the same index that you had before.

Here we add a simple test that give some flavor of how that works.
2017-12-04 21:53:05 -04:00
Daniel Kahn Gillmor
836ec85b0c test/corpora: add an encrypted message for index decryption tests 2017-12-04 21:53:05 -04:00
David Bremner
1e4e8e3bf5 test: unbreak performance tests
In 8e7fb88237 Jani replaced the use of $(pwd -P) to find the
TEST_DIRECTORY in order to better support out of tree
builds. Unfortunately the performance-tests need a different value for
the variable and were thus broken.

This commit splits out the setting of this variable for the two sets
of tests.  Performance tests still don't work out of tree, because
the handling of the downloaded corpus needs to be updated.
2017-12-04 21:04:38 -04:00
Jani Nikula
cf08295c50 test: test notmuch insert --folder=""
Test insert into top level folder.
2017-11-08 09:47:14 -05:00
Jani Nikula
1b91884296 test: fix test database backup/restore location
backup_database() and restore_database() used to store the backups in
the test specific temporary directory, through the current working
directory being there. Commit 8e7fb88237 ("test: use source and
build paths in test-lib-common.sh") started using a test specific
backup directories under the build tree test directory. This was in
error. Switch back to the old location, but using paths to the
location instead of relying on current working directory.

Reported by Daniel Kahn Gillmor <dkg@fifthhorseman.net>.
2017-10-29 09:30:30 -07:00
Daniel Kahn Gillmor
8116891744 cli/reindex: add --try-decrypt=(true|false)
Enable override of the index.try_decrypt setting on a per-run basis
when invoking "notmuch reindex".  This allows the possibility of (for
example) an emacs keybinding that adds the cleartext of the currently
shown decrypted message to the index, making it searchable in the
future.

It also enables one-time indexing of all messages matching some query,
like so:

    notmuch reindex tag:encrypted and\
       not property:index.decryption=success and\
       from:alice@example.org

We also update the documentation and tab completion, and add a few
more tests.
2017-10-21 19:58:52 -03:00
Daniel Kahn Gillmor
c5356b9ed5 cli/insert: add --try-decrypt=(true|false)
Enable override of the index.try_decrypt setting on a per-message
basis when invoking "notmuch insert".

We also update the documentation and tab completion, and add more tests.
2017-10-21 19:58:43 -03:00
Daniel Kahn Gillmor
35456d4b0c cli/new: add --try-decrypt=(true|false)
Enable override of the index.try_decrypt setting during "notmuch new"
on a per-invocation basis.

We update the documentation and tab completion, and also add a test.
2017-10-21 19:58:32 -03:00
Daniel Kahn Gillmor
92f318abe4 tests: emacs_fcc_message: allow passing --arguments to notmuch new
Subsequent patches may want to send GNU-style --long-arguments to
notmuch new in the test suite, in particular when invoking
emacs_fcc_message.  This changeset makes that possible.
2017-10-21 19:58:23 -03:00
Jani Nikula
d7929d61a2 test: use source path in add_gnupg_home
Make a distinction between source and build directories.
2017-10-21 16:34:04 -03:00
Jani Nikula
d455c8b74a test: remove --root option and fix TMP_DIRECTORY cleanup
The primary motivation here is to fix TMP_DIRECTORY cleanup prior to
running each test when the current working directory is not the test
subdirectory. Tests with failures would leave their TMP_DIRECTORY
directory behind for debugging, and repeated out-of-tree test runs
would have old temp directories. (This lead to e.g. T310-emacs.sh
hanging because emacs would prompt for overwriting files.)

We remove the likely anyway defunct --root test option while at it,
just to be on the safe side when doing 'rm -rf' on the TMP_DIRECTORY.
2017-10-21 11:32:20 -03:00
Jani Nikula
409d877364 test: use source and build paths in T590-thread-breakage.sh
Make a distinction between source and build directories.
2017-10-20 20:50:17 -03:00
Jani Nikula
3c23780396 test: use source and build paths in T480-hex-escaping.sh
Make a distinction between source and build directories.
2017-10-20 20:50:03 -03:00
Jani Nikula
1ab72b13cd test: use source and build paths in T4?0-emacs-*.sh
Make a distinction between source and build directories.
2017-10-20 20:49:50 -03:00
Jani Nikula
e216d9b7ab test: use source and build paths in T380-atomicity.sh
Make a distinction between source and build directories.
2017-10-20 20:49:37 -03:00
Jani Nikula
0364a6d74b test: use source and build paths in T360-symbol-hiding.sh
Make a distinction between source and build directories.
2017-10-20 20:49:26 -03:00
Jani Nikula
bc349ad316 test: use source and build paths in T355-smime.sh
Make a distinction between source and build directories.
2017-10-20 20:49:18 -03:00
Jani Nikula
f6b098b1a6 test: use source and build paths in T310-emacs.sh
Make a distinction between source and build directories.
2017-10-20 19:56:13 -03:00
Jani Nikula
b079133ca8 test: use source and build paths in T260-thread-order.sh
Make a distinction between source and build directories.
2017-10-20 19:55:57 -03:00
Jani Nikula
32c088b524 test: use source and build paths in T160-json.sh and T170-sexp.sh
Make a distinction between source and build directories.
2017-10-20 19:55:49 -03:00
Jani Nikula
fc12f6f07b test: use source and build paths in T000-basic.sh
Make a distinction between source and build directories.
2017-10-20 19:55:41 -03:00
Jani Nikula
a8cd1ec26b test: use source and build paths in notmuch-test
Make a distinction between source and build directories.
2017-10-20 19:55:31 -03:00
Jani Nikula
8dfa116010 test: mkdir and cd to $TMP_DIRECTORY instead of $test for testing
$TMP_DIRECTORY is a full path, while $test is not.
2017-10-20 19:55:10 -03:00
Jani Nikula
8e7fb88237 test: use source and build paths in test-lib-common.sh
Make a distinction between source and build directories.

The expectation is that anyone sourcing test-lib-common.sh has sourced
export-dirs.sh.
2017-10-20 19:54:43 -03:00
Jani Nikula
a9894fce06 test: use source and build paths in test-lib.sh
Make a distinction between source and build directories.
2017-10-20 19:54:08 -03:00
Jani Nikula
1f3c114f08 test: add a script to figure out source and build directories
Expect the tests to be run somewhere in the build directory
hierarchy. Allow for source and build directories to be passed through
the environment.
2017-10-20 19:53:55 -03:00
Jani Nikula
e1bd4f2363 test: check for notmuch binary in test-lib.sh
Move notmuch executable check into common code. Redundant for
notmuch-test runs, but works also for individual tests.
2017-10-20 19:53:31 -03:00
Jani Nikula
a863de1e43 test: use $(dirname "$0") for sourcing test-lib.sh
Don't assume the tests are always run from within the source tree.
2017-10-20 19:52:49 -03:00
Jani Nikula
0dd58ea7e7 build: use NOTMUCH_SRCDIR for referencing notmuch-test
Don't assume the tests are always run from within the source tree.
2017-10-20 19:52:24 -03:00
Jani Nikula
1c2152ce73 test: use corpora/default not corpus in T480-hex-escaping.sh
Turns out round trip tests didn't really round trip anything. Broken
by yours truly in 971cdc72cd ("test: make it possible to have
multiple corpora"). Ooops.
2017-10-20 19:52:14 -03:00
Daniel Kahn Gillmor
ee5df7d7db tests: prepare for more crypto tests (using add_gnupg_home)
Move add_gnupg_home to test-lib.sh to prepare it for reuse.
2017-10-14 10:53:42 -03:00
Jani Nikula
0f314c0c99 cli: convert notmuch_bool_t to stdbool
C99 stdbool turned 18 this year. There really is no reason to use our
own, except in the library interface for backward
compatibility. Convert the cli and test binaries to stdbool.
2017-10-09 22:24:02 -03:00
Jani Nikula
816097f05b hex-xcode: use notmuch_bool_t for boolean arguments
Pedantically correct, although they're the same underlying type.
2017-10-04 22:02:45 -03:00
Jani Nikula
9165df2135 test: expand argument parsing tests
Test and use the new .present field, only output the parameters
given. Test space between parameter name and value.
2017-10-04 22:01:50 -03:00
Jani Nikula
25960b5ecd test: add opt_inherit to arg-test
Just split the arguments to two opt desc arrays.
2017-10-04 22:01:15 -03:00
Jani Nikula
e8cba9da18 test: add boolean argument to arg-test
Surprisingly it's not there.
2017-10-04 22:00:54 -03:00
Jani Nikula
4a6721970a cli: use designated initializers for opt desc
Several changes at once, just to not have to change the same lines
several times over:

- Use designated initializers to initialize opt desc arrays.

- Only initialize the needed fields.

- Remove arg_id (short options) as unused.

- Replace opt_type and output_var with several type safe output
  variables, where the output variable being non-NULL determines the
  type. Introduce checks to ensure only one is set. The downside is
  some waste of const space per argument; this could be saved by
  retaining opt_type and using a union, but that's still pretty
  verbose.

- Fix some variables due to the type safety. Mostly a good thing, but
  leads to some enums being changed to ints. This is pedantically
  correct, but somewhat annoying. We could also cast, but that defeats
  the purpose a bit.

- Terminate the opt desc arrays using {}.

The output variable type safety and the ability to add new fields for
just some output types or arguments are the big wins. For example, if
we wanted to add a variable to set when the argument is present, we
could do so for just the arguments that need it.

Beauty is in the eye of the beholder, but I think this looks nice when
defining the arguments, and reduces some of the verbosity we have
there.
2017-10-04 22:00:42 -03:00
Florian Klink
073d05eec3 T390-python: add test for get_message_parts and special characters
This imports a message with ISO-8859-2 encoded characters, then opens
the database using the python bindings. We peek through all mesage
parts, afterwards print the message id.

Signed-off-by: Florian Klink <flokli@flokli.de>
Signed-off-by: Andreas Rammhold <andreas@rammhold.de>
2017-10-02 07:21:41 -03:00
Jani Nikula
b2b65b4e6d emacs: sanitize subject in replies
Commit a7964c86d1 ("emacs: Sanitize authors and subjects in search
and show") added sanitization of header information for display. Do
the same for reply subjects.

This fixes the long-standing annoying artefact of certain versions of
mailman using tab as folding whitespace, leading to tabs in reply
subjects.
2017-09-27 08:15:02 -03:00
Jani Nikula
77eb76d9df test: add emacs reply test for subjects with TAB
Expect TABs to be sanitized from the subject line. Known broken.
2017-09-27 08:10:31 -03:00
Daniel Kahn Gillmor
f4ac5ecd5c lib: index the content-type of the parts of encrypted messages
This is a logical followup to "lib: index the content type of
signature parts", which will make it easier to record the message
structure of all messages.
2017-09-17 20:01:19 -03:00
Jani Nikula
55c047ee0b lib: index the content type of signature parts
It's useful (*) to be able to easily find messages with certain types
of signatures. Having the mimetype: prefix searches fail for some
content types is also genuinely surprising (*). Index the content type
of signature parts.

While at it, switch to the gmime convenience constants for content and
signature part indexes.

*) At least for developers of email software!
2017-09-17 20:01:00 -03:00
Jani Nikula
30c2761465 test: signed and encrypted part content type indexing
Add known broken subtests for searching signed and encrypted messages
using mimetype: prefix search for the content-types of signed and
encrypted parts.
2017-09-17 20:00:43 -03:00
David Bremner
64e30aeb24 test/duplicate-mid: check for subject with notmuch-show
In [1] Mark showed that the the current code (d7a49e81) is not
consistent in it's handling of subjects of messages with duplicate
message-ids (or in notmuch-speak, of messages with multiple files).
notmuch-search uses indexing order and explicitedly preserves the
first. notmuch-show (apparently) uses alphabetical (or at least xapian
term order) of filenames. In a perfect world we would probably report
all subjects in the json output; at the very least we should be
consistent.

[1]: id:87378dny3d.fsf@qmul.ac.uk
2017-09-05 21:53:06 -03:00
David Bremner
debfae20db lib: enforce that n_message_reindex takes headers from first file
This is still a bit stopgap to be only choosing one set of headers,
but this seems like a more defensible set of headers to choose.
2017-09-05 21:51:57 -03:00
David Bremner
36a3d65034 test: known broken test for subject after reindexing
In [1], Mark gave a test that was behaving strangly. This turns out to
be specific to reindexing.  I suppose one could argue that picking the
lexicographically last file name is a defensible choice, but it's
almost as easy to take the first, which seems more intuitive. So mark
the current situation as broken.

[1]: id:1503859703-2973-1-git-send-email-markwalters1009@gmail.com
2017-09-05 21:51:21 -03:00
David Bremner
17aebb6e7d test/duplicate-mid: clarify index order vs filename order
The existing test for notmuch search had the first in filename order
the same as the first indexed, which made it harder to understand what
the underlying behaviour is. Add a file with a lexicographically
smaller name, but later index time to clarify this.
2017-09-05 21:51:07 -03:00
David Bremner
44914f1c3b test: make fallback to duplicate test more robust.
The original intent of this test was to verify that notmuch show was
not crashing when the first file (where headers are being read from)
was deleted. Run the output through some sanitization so that as we
add and delete copies we don't have to update this test.
2017-09-05 21:50:54 -03:00
Tomi Ollila
7526538808 test: notmuch_drop_mail_headers() style update
Changed "" quotes to '' as we're not supposed to dynamically
alter python program (via shell $variable expansion).

Added space to python program to match general python style.

Replaced $* with 'idiomatic' "$@" to serve as better example.
2017-09-04 08:03:52 -03:00
David Bremner
ec37900c5f test/crypto: remove headers more robustly
In [1], Vladimir Panteleev observed that the In-Reply-To and
References headers could be wrapped in the 'default' output format of
notmuch-reply, depending on the version of Emacs creating the
message. In my own experiments notmuch-reply sometimes wraps headers
with only one message-id if that message-id is long enough. However it
happens, this causes the previous approach using grep to fail.

Since I found the proposed unwrapping shell fragment in [1] a bit hard
to follow, I decided to write a little python script instead. Then
Tomi suggested a slight generalization of my script, and here we are.

[1] id:20170817175145.3204-7-notmuch@thecybershadow.net
2017-09-03 08:06:15 -03:00
Vladimir Panteleev
95b82bb326 test: Fix T050-new.sh on some Travis CI machines
On some system configurations, setting a breakpoint on the "add_file"
function then issuing "continue" in gdb causes the debugger to
seemingly jump over the add_file invocation. This results in a test
failure, as the "Handle files vanishing between scandir and add_file"
subtest expects add_file to be called and fail due to the vanishing
file. The compiler optimization level also plays a role - the problem
can be reproduced with CFLAGS having -O2 but not -Og.

This problem was observed manifesting as a test failure on Travis CI
configured with "dist: trusty" and "sudo: false". It was not
reproducible on a local Docker image of Travis' runtime environment,
so Travis' virtualization infrastructure likely plays a role as well.

* T050-new.sh: Breakpoint notmuch_database_add_message instead of
  add_file to the same effect, and avoid bad gdb behaviour on Travis
  CI.

Amended by db:

	s/notmuch_database_add_message/notmuch_database_index_file/

Somehow the wrapper function doesn't work as a breakpoint; perhaps due
to inlining.
2017-08-30 07:24:44 -03:00
David Bremner
536d695a31 test/smtp-dummy: convert to 'goto DONE' style
Clean up several cppcheck warnings of the form

      - test/smtp-dummy.c:170: error: Resource leak: output

Conform to overall notmuch code style.
2017-08-30 07:16:24 -03:00
David Bremner
1fa5bd282e test/smtp-dummy: uncrustify
For some reason lost in the mists of time this code was indented 8 spaces.
2017-08-30 07:14:18 -03:00
David Bremner
0082a55785 CLI/new: support maildir synced tags in new.tags
We reorder reading maildir flags to avoid overwriting 'new.tags'. The
inverted status of 'unread' means the maildir flag needs to be checked
a second time.

I backpedalled here on the idea of supporting 'new.tags' without
'unread' in the presence of maildir syncing.  For files in 'new/', it
seems quite natural to tag them as 'unread'.
2017-08-29 21:57:38 -03:00
David Edmondson
ca4852649e test: Perform T170 tests that don't require dtach before any that do.
This avoids the later tests seeing different versions of the database
depending on whether dtach is available.
2017-08-28 08:02:02 -03:00
Vladimir Panteleev
8c772eaa0d test: Use small Python script for JSON normalization instead of json.tool
json.tool does not sort or otherwise normalize the order of JSON keys
in its output, which can result in test failures on some test systems.

Instead, use a one-line Python script passed to the interpreter
directly on its command line. Use sort_keys=True for json.dump to
ensure the key order is normalized. The script works with both Python
2 and 3.

* test/test-lib.sh: Update test_expect_equal_json.
2017-08-23 08:07:45 -03:00
Daniel Kahn Gillmor
b10ce6bc23 database: add n_d_index_file (deprecates n_d_add_message)
We need a way to pass parameters to the indexing functionality on the
first index, not just on reindexing.  The obvious place is in
notmuch_database_add_message.  But since modifying the argument list
would break both API and ABI, we needed a new name.

I considered notmuch_database_add_message_with_params(), but the
functionality we're talking about doesn't always add a message.  It
tries to index a specific file, possibly adding a message, but
possibly doing other things, like adding terms to an existing message,
or failing to deal with message objects entirely (e.g. because the
file didn't contain a message).

So i chose the function name notmuch_database_index_file.

I confess i'm a little concerned about confusing future notmuch
developers with the new name, since we already have a private
_notmuch_message_index_file function, and the two do rather different
things.  But i think the added clarity for people linking against the
future libnotmuch and the capacity for using index parameters makes
this a worthwhile tradeoff.  (that said, if anyone has another name
that they strongly prefer, i'd be happy to go with it)

This changeset also adjusts the tests so that we test whether the new,
preferred function returns bad values (since the deprecated function
just calls the new one).

We can keep the deprecated n_d_add_message function around as long as
we like, but at the next place where we're forced to break API or ABI
we can probably choose to drop the name relatively safely.

NOTE: there is probably more cleanup to do in the ruby and go bindings
to complete the deprecation directly.  I don't know those languages
well enough to attempt a fix; i don't know how to test them; and i
don't know the culture around those languages about API additions or
deprecations.
2017-08-23 07:38:37 -03:00
Yuri Volchkov
09fa51303c show: workaround for the missing file problem
This patch fixes the 'Deleted first duplicate file does not stop
notmuch show from working' test.

If a message to be shown has several duplicated files, and for some
reason the first file in the list is not available anymore, notmuch
will exit with an error.

This is clearly a problem in the database, but we are not going to let
this problem be a show-stopper. Let's walk through the list, and show
the first existing file.

Signed-off-by: Yuri Volchkov <yuri.volchkov@gmail.com>
2017-08-22 18:48:29 -03:00
Yuri Volchkov
39cc84e9fc test: show id:<> works even if the first duplicate is deleted
Signed-off-by: Yuri Volchkov <yuri.volchkov@gmail.com>
2017-08-22 18:48:19 -03:00
Yuri Volchkov
33a170e116 insert: strip trailing / in folder path
This patch fixes the "Insert message into folder with trailing /"
test. The problem was insufficient path canonization.

From database's point of view, "Sent" and "Sent/" are different
folders. If user runs (note the last '/'):

    notmuch insert --folder=maildir/Sent/ < test.msg

notmuch will create an extra XDIRECTORY record for the folder
'Sent/'. This means that database will have _TWO_ records for _ONE_
physical folder: 'Sent' and 'Sent/'. However, the 'notmuch new'
command will update only records related to the first one (the correct
one).

Now, if user moved the email file (e.g. from 'Sent/new' to
'Sent/cur'), 'notmuch new' will add a record about the new file, but
will not delete the old record.

Signed-off-by: Yuri Volchkov <yuri.volchkov@gmail.com>
2017-08-22 18:48:09 -03:00
David Bremner
1092c747ef test: move generate_message, add_message into test-lib-common.sh
The plan is to use at least the former in the perf test suite.
2017-08-20 08:34:03 -03:00
Yuri Volchkov
cd8551d53a test: insert into the folder with trailing /
From database's point of view, "Drafts" and "Drafts/" are different
folders

Signed-off-by: Yuri Volchkov <yuri.volchkov@gmail.com>
Amended: add test_subtest_known_broken (db)
2017-08-20 08:28:55 -03:00
Yuri Volchkov
02761b2e5c test: remove remainder of previously killed basic test
In the commit 51cd69feb1 the part of the
test "test runs if prerequisite is satisfied" has been
removed. However, there was a remainder of that test - variable
'haveit'.

Kill it, to not to confuse people.

Signed-off-by: Yuri Volchkov <yuri.volchkov@gmail.com>
2017-08-18 19:45:17 -03:00
Vladimir Panteleev
ca4688e103 Use rooted paths in .gitignore files
A leading / in paths in a .gitignore file matches the beginning of the
path, meaning that for patterns without slashes, git will match files
only in the current directory as opposed to in any subdirectory.

Prefix relevant paths with / in .gitignore files, to prevent
accidentally ignoring files in subdirectories and possibly slightly
improve the performance of "git status".
2017-08-18 19:42:35 -03:00
Vladimir Panteleev
4d44976135 test: Update extant references to corpus.mail
971cdc72cd renamed corpus.mail to
corpora.mail. Although 971cdc72cd
updated some of the remaining corpus.mail references, two remained,
causing the test suite to leave behind an unignored corpora.mail
directory.
2017-08-18 19:42:19 -03:00
David Bremner
f385055528 test: add test for ,S message without 'unread' in new.tags
This is arguably overkill, but it helps to understand the complicated
interactions here between maildir tags and configuration.
2017-08-16 21:53:43 -03:00
David Bremner
e1f9ab4849 test: add missing quotes in maildir-sync test.
Oops. ';' is significant in the shell. Who knew.
2017-08-16 21:48:51 -03:00