Commit graph

5761 commits

Author SHA1 Message Date
Daniel Kahn Gillmor
d3964e81ac indexing: Change from try_decrypt to decrypt
the command-line interface for indexing (reindex, new, insert) used
--try-decrypt; and the configuration records used index.try_decrypt.
But by comparison with "show" and "reply", there doesn't seem to be
any reason for the "try" prefix.

This changeset adjusts the command-line interface and the
configuration interface.

For the moment, i've left indexopts_{set,get}_try_decrypt alone.  The
subsequent changeset will address those.
2017-12-08 08:05:53 -04:00
Daniel Kahn Gillmor
de80ede3df cli/help: give a hint about notmuch-emacs-mua
"notmuch help" doesn't mention "notmuch-emacs-mua" even though we
support it through the try_external_command() mechanism.

In addition, "notmuch help emacs-mua" doesn't work, even though we
ship the appropriate manpage.

This changeset fixes both of these problems.
2017-12-07 08:20:00 -04:00
Daniel Kahn Gillmor
03f4f75124 debian/control: build-depend on python3-sphinx instead of python-sphinx
python2 is going to be deprecated, and python3-sphinx is available all
the way back to oldoldstable.  let's use the more modern version.

To make this work and still ship the manpages, tell ./configure to
prefer python3 over python, if it exists.
2017-12-07 08:18:29 -04:00
Daniel Kahn Gillmor
9d23ebfcc3 test: session keys are known broken without session key support
If the version of GMime we're building against doesn't support session
key extraction or re-use, mark the tests that rely on session key
capabilities as known-broken.

This should resolve test suite failures on ubuntu trusty and debian
jessie and earlier, which have GMime 2.6.20 -- session key support was
introduced in GMime 2.6.21.
2017-12-05 20:09:25 -04:00
Daniel Kahn Gillmor
6d7249fec9 crypto: Test restore of cleartext index from stashed session keys
If you've got a notmuch dump that includes stashed session keys for
every decrypted message, and you've got your message archive, you
should be able to get back to the same index that you had before.

Here we add a simple test that give some flavor of how that works.
2017-12-04 21:53:05 -04:00
Daniel Kahn Gillmor
836ec85b0c test/corpora: add an encrypted message for index decryption tests 2017-12-04 21:53:05 -04:00
Daniel Kahn Gillmor
a990585408 crypto: use stashed session-key properties for decryption, if available
When doing any decryption, if the notmuch database knows of any
session keys associated with the message in question, try them before
defaulting to using default symmetric crypto.

This changeset does the primary work in _notmuch_crypto_decrypt, which
grows some new parameters to handle it.

The primary advantage this patch offers is a significant speedup when
rendering large encrypted threads ("notmuch show") if session keys
happen to be cached.

Additionally, it permits message composition without access to
asymmetric secret keys ("notmuch reply"); and it permits recovering a
cleartext index when reindexing after a "notmuch restore" for those
messages that already have a session key stored.

Note that we may try multiple decryptions here (e.g. if there are
multiple session keys in the database), but we will ignore and throw
away all the GMime errors except for those that come from last
decryption attempt.  Since we don't necessarily know at the time of
the decryption that this *is* the last decryption attempt, we'll ask
for the errors each time anyway.

This does nothing if no session keys are stashed in the database,
which is fine.  Actually stashing session keys in the database will
come as a subsequent patch.
2017-12-04 21:48:31 -04:00
Daniel Kahn Gillmor
0ff13f862c configure: session key handling in gmime maps to built_with("session_key")
This flag should make it easier to write the code for session-key
handling.

Note that this only works for GMime 2.6.21 and later (the session key
interface wasn't available before then).  It should be fine to build
the rest of notmuch if this functionality isn't available.

Note that this also adds the "session_key" built_with() aspect to
libnotmuch.
2017-12-04 21:39:50 -04:00
Daniel Kahn Gillmor
5f2832ae21 crypto: add _notmuch_crypto_decrypt wrapper function
We will use this centralized function to consolidate the awkward
behavior around different gmime versions.

It's only invoked from two places: mime-node.c's
node_decrypt_and_verify() and lib/index.cc's
_index_encrypted_mime_part().

However, those two places have some markedly distinct logic, so the
interface for this _notmuch_crypto_decrypt function is going to get a
little bit clunky.  It's worthwhile, though, for the sake of keeping
these #if directives reasonably well-contained.
2017-12-04 21:39:24 -04:00
Daniel Kahn Gillmor
9beda4b981 mime-node: handle decrypt_result more safely
If (for whatever reason) we don't get a decrypt_result back, or it's
not structured the way we expect it to be, we shouldn't choke on it.
2017-12-04 21:39:13 -04:00
Daniel Kahn Gillmor
842d7dffcb debian: bump Standards-Version to 4.1.1
No minor changes were necessary to become compliant with Debian policy
version 4.1.1, so this is basically a freebie.
2017-12-04 21:24:33 -04:00
Daniel Kahn Gillmor
550f88f917 debian/control: Priority: extra is deprecated
Move transitional package to to "oldlibs/optional"

This resolves two lintian warnings:

W: notmuch-emacs: transitional-package-should-be-oldlibs-optional oldlibs/extra
W: notmuch-emacs: priority-extra-is-replaced-by-priority-optional
2017-12-04 21:23:59 -04:00
Daniel Kahn Gillmor
7cb5292614 debian/changelog: remove trailing whitespace 2017-12-04 21:23:52 -04:00
David Bremner
1e4e8e3bf5 test: unbreak performance tests
In 8e7fb88237 Jani replaced the use of $(pwd -P) to find the
TEST_DIRECTORY in order to better support out of tree
builds. Unfortunately the performance-tests need a different value for
the variable and were thus broken.

This commit splits out the setting of this variable for the two sets
of tests.  Performance tests still don't work out of tree, because
the handling of the downloaded corpus needs to be updated.
2017-12-04 21:04:38 -04:00
Jani Nikula
cf08295c50 test: test notmuch insert --folder=""
Test insert into top level folder.
2017-11-08 09:47:14 -05:00
Jani Nikula
733ccfabca cli: allow empty strings for notmuch insert --folder argument
Now that it's easy to add argument specific modifiers in opt
descriptions, add a new .allow_empty field to allow empty strings for
individual string arguments while retaining strict checks
elsewhere. Use this for notmuch insert --folder, where the empty
string means top level folder.
2017-11-08 09:46:44 -05:00
David Bremner
7ac96b149f Merge branch 'release'
Changes from 0.25.2 release
2017-11-05 20:25:54 -04:00
David Bremner
83f2661363 debian: changelog for 0.25.2-1 2017-11-05 20:06:23 -04:00
David Bremner
42871517c6 version: bump to 0.25.2 2017-11-05 19:59:30 -04:00
David Bremner
58bf820b78 NEWS for 0.25.2 2017-11-05 19:54:40 -04:00
Jani Nikula
cd3f5e1a93 cli/crypto: fix segfault on failed gmime2 crypto context creation
Commit 1fdc08d0ff ("cli/crypto: treat failure to create a crypto
context as fatal.") started treating crypto context creation failures
"as fatal", returning NULL from _mime_node_create().

Unfortunately, we do not have NULL checks for _mime_node_create()
failures. The only caller, mime_node_child(), could check and return
NULL (as it's documented to do on errors) but none of the several call
sites have NULL checks either. And none of them really have a trivial
but feasible and graceful way of recovery.

So while the right thing to do would be to handle NULL returns
properly all over the place, and we have other scenarios that do
return NULL from above mentioned functions, the crypto context
creation failure is something that does seem to show up regularly in
some scenarios, revert back to the functionality before commit
1fdc08d0ff as an interim fix.
2017-11-05 15:41:13 -04:00
Jani Nikula
1b91884296 test: fix test database backup/restore location
backup_database() and restore_database() used to store the backups in
the test specific temporary directory, through the current working
directory being there. Commit 8e7fb88237 ("test: use source and
build paths in test-lib-common.sh") started using a test specific
backup directories under the build tree test directory. This was in
error. Switch back to the old location, but using paths to the
location instead of relying on current working directory.

Reported by Daniel Kahn Gillmor <dkg@fifthhorseman.net>.
2017-10-29 09:30:30 -07:00
Daniel Kahn Gillmor
9479baf6f1 NEWS: cleartext indexing
give a short summary of cleartext indexing of encrypted e-mail
messages.
2017-10-22 14:02:00 -03:00
Jani Nikula
61c346c286 NEWS: test suite out-of-tree builds
Out-of-tree builds now work and supersede --root option.
2017-10-22 11:03:21 -03:00
Daniel Kahn Gillmor
8116891744 cli/reindex: add --try-decrypt=(true|false)
Enable override of the index.try_decrypt setting on a per-run basis
when invoking "notmuch reindex".  This allows the possibility of (for
example) an emacs keybinding that adds the cleartext of the currently
shown decrypted message to the index, making it searchable in the
future.

It also enables one-time indexing of all messages matching some query,
like so:

    notmuch reindex tag:encrypted and\
       not property:index.decryption=success and\
       from:alice@example.org

We also update the documentation and tab completion, and add a few
more tests.
2017-10-21 19:58:52 -03:00
Daniel Kahn Gillmor
c5356b9ed5 cli/insert: add --try-decrypt=(true|false)
Enable override of the index.try_decrypt setting on a per-message
basis when invoking "notmuch insert".

We also update the documentation and tab completion, and add more tests.
2017-10-21 19:58:43 -03:00
Daniel Kahn Gillmor
35456d4b0c cli/new: add --try-decrypt=(true|false)
Enable override of the index.try_decrypt setting during "notmuch new"
on a per-invocation basis.

We update the documentation and tab completion, and also add a test.
2017-10-21 19:58:32 -03:00
Daniel Kahn Gillmor
92f318abe4 tests: emacs_fcc_message: allow passing --arguments to notmuch new
Subsequent patches may want to send GNU-style --long-arguments to
notmuch new in the test suite, in particular when invoking
emacs_fcc_message.  This changeset makes that possible.
2017-10-21 19:58:23 -03:00
Daniel Kahn Gillmor
5bdb8549f6 cli: set up shared command-line arguments for indexing
We have an indexopts structure for manipulating indexing in different
ways, but we also have three command-line invocations that can trigger
indexing: new, insert, and reindex.

This changeset prepares a common parser that these subcommands can
share.

Note: if the deprecated crypto.gpg_path configuration option is set to
anything other than "gpg", we ignore it (and print a warning on
stderr, if built against gmime < 3.0).

At the moment, it's just --try-decrypt, but others will likely follow.
2017-10-21 19:58:11 -03:00
Daniel Kahn Gillmor
d0da7a0a1c config: define new option index.try_decrypt
By default, notmuch won't try to decrypt on indexing.  With this
patch, we make it possible to indicate a per-database preference using
the config variable "index.try_decrypt", which by default will be
false.

At indexing time, the database needs some way to know its internal
defaults for how to index encrypted parts.  It shouldn't be contingent
on an external config file (since that can't be retrieved from the
database object itself), so we store it in the database.

This behaves similarly to the query.* configurations, which are also
stored in the database itself, so we're not introducing any new
dependencies by requiring that it be stored in the database.
2017-10-21 19:54:33 -03:00
Daniel Kahn Gillmor
886c0b1666 config: test whether an item is stored in the database by name
QUERY_STRING was only used in two places, both to test whether a
variable should be stored in (or retrieved from) the database.

Since other configuration variables might be stored in the database in
the future, consolidate that test into a single function.

We also document that these configuration options should not be placed
in the config file.
2017-10-21 19:54:24 -03:00
Daniel Kahn Gillmor
4dfcc8c9b2 crypto: index encrypted parts when indexopts try_decrypt is set.
If we see index options that ask us to decrypt when indexing a
message, and we encounter an encrypted part, we'll try to descend into
it.

If we can decrypt, we add the property index.decryption=success.

If we can't decrypt (or recognize the encrypted type of mail), we add
the property index.decryption=failure.

Note that a single message may have both values of the
"index.decryption" property: "success" and "failure".  For example,
consider a message that includes multiple layers of encryption.  If we
manage to decrypt the outer layer ("index.decryption=success"), but
fail on the inner layer ("index.decryption=failure").

Because of the property name, this will be automatically cleared (and
possibly re-set) during re-indexing.  This means it will subsequently
correspond to the actual semantics of the stored index.
2017-10-21 19:53:19 -03:00
Daniel Kahn Gillmor
0bb05ff693 reindex: drop all properties named with prefix "index."
This allows us to create new properties that will be automatically set
during indexing, and cleared during re-indexing, just by choice of
property name.
2017-10-21 19:53:08 -03:00
Daniel Kahn Gillmor
6575b7eb31 doc: add notmuch-properties(7)
We will want a user-facing place to record details about the use of
notmuch properties shortly.  This establishes a new manual page for
that purpose.
2017-10-21 19:52:55 -03:00
Daniel Kahn Gillmor
20ff9de24d index: implement notmuch_indexopts_t with try_decrypt
This is currently mostly a wrapper around _notmuch_crypto_t that keeps
its internals private and doesn't expose any of the GMime API.
However, non-crypto indexing options might also be added later
(e.g. filters or other transformations).
2017-10-21 19:52:47 -03:00
Jani Nikula
d6929040a4 build: don't copy the test directory for out-of-tree builds
It should now work without.
2017-10-21 16:34:55 -03:00
Jani Nikula
d7929d61a2 test: use source path in add_gnupg_home
Make a distinction between source and build directories.
2017-10-21 16:34:04 -03:00
Jani Nikula
d455c8b74a test: remove --root option and fix TMP_DIRECTORY cleanup
The primary motivation here is to fix TMP_DIRECTORY cleanup prior to
running each test when the current working directory is not the test
subdirectory. Tests with failures would leave their TMP_DIRECTORY
directory behind for debugging, and repeated out-of-tree test runs
would have old temp directories. (This lead to e.g. T310-emacs.sh
hanging because emacs would prompt for overwriting files.)

We remove the likely anyway defunct --root test option while at it,
just to be on the safe side when doing 'rm -rf' on the TMP_DIRECTORY.
2017-10-21 11:32:20 -03:00
Jani Nikula
409d877364 test: use source and build paths in T590-thread-breakage.sh
Make a distinction between source and build directories.
2017-10-20 20:50:17 -03:00
Jani Nikula
3c23780396 test: use source and build paths in T480-hex-escaping.sh
Make a distinction between source and build directories.
2017-10-20 20:50:03 -03:00
Jani Nikula
1ab72b13cd test: use source and build paths in T4?0-emacs-*.sh
Make a distinction between source and build directories.
2017-10-20 20:49:50 -03:00
Jani Nikula
e216d9b7ab test: use source and build paths in T380-atomicity.sh
Make a distinction between source and build directories.
2017-10-20 20:49:37 -03:00
Jani Nikula
0364a6d74b test: use source and build paths in T360-symbol-hiding.sh
Make a distinction between source and build directories.
2017-10-20 20:49:26 -03:00
Jani Nikula
bc349ad316 test: use source and build paths in T355-smime.sh
Make a distinction between source and build directories.
2017-10-20 20:49:18 -03:00
Jani Nikula
f6b098b1a6 test: use source and build paths in T310-emacs.sh
Make a distinction between source and build directories.
2017-10-20 19:56:13 -03:00
Jani Nikula
b079133ca8 test: use source and build paths in T260-thread-order.sh
Make a distinction between source and build directories.
2017-10-20 19:55:57 -03:00
Jani Nikula
32c088b524 test: use source and build paths in T160-json.sh and T170-sexp.sh
Make a distinction between source and build directories.
2017-10-20 19:55:49 -03:00
Jani Nikula
fc12f6f07b test: use source and build paths in T000-basic.sh
Make a distinction between source and build directories.
2017-10-20 19:55:41 -03:00
Jani Nikula
a8cd1ec26b test: use source and build paths in notmuch-test
Make a distinction between source and build directories.
2017-10-20 19:55:31 -03:00
Jani Nikula
8dfa116010 test: mkdir and cd to $TMP_DIRECTORY instead of $test for testing
$TMP_DIRECTORY is a full path, while $test is not.
2017-10-20 19:55:10 -03:00