In certain conditions the parallel calls to sphinx-build could
collide, yielding a crash like
Exception occurred:
File "/usr/lib/python3/dist-packages/sphinx/environment.py", line 1261, in get_doctree
doctree = pickle.load(f)
EOFError: Ran out of input
Many of the manpages didn't treat literal text as literal text. I've
tried to normalize some of the restructured text to make it a bit more
regular.
several of the synopsis lines are still untouched by this cleanup, but
i'm not sure what the right way to represent those is in .rst,
actually.
In particular find that if i rebuild the manpages, sometimes i end up
with some of the synopsis lines showing – (U+2013 EN DASH) where they
should have -- (2 × U+002D HYPHEN-MINUS) in the generated nroff
output, though i have not tracked down the source of this error yet.
This is mainly to improve discoverability. It seems that doing
variable cross-references is not easy without using some sphinx
extension/customization.
Add fancy new feature, which makes "notmuch show" capable of actually
indexing messages that it just decrypted.
This enables a workflow where messages can come in in the background
and be indexed using "--decrypt=auto". But when showing an encrypted
message for the first time, it gets automatically indexed.
This is something of a departure for "notmuch show" -- in particular,
because it requires read/write access to the database. However, this
might be a common use case -- people get mail delivered and indexed in
the background, but only want access to their secret key to happen
when they're directly interacting with notmuch itself.
In such a scenario, they couldn't search newly-delivered, encrypted
messages, but they could search for them once they've read them.
Documentation of this new feature also uses a table form, similar to
that found in the description of index.decrypt in notmuch-config(1).
A notmuch UI that wants to facilitate this workflow while also
offering an interactive search interface might instead make use of
these additional commands while the user is at the console:
Count received encrypted messages (if > 0, there are some things we
haven't yet tried to index, and therefore can't yet search):
notmuch count tag:encrypted and \
not property:index.decryption=success and \
not property:index.decryption=failure
Reindex those messages:
notmuch reindex --try-decrypt=true tag:encrypted and \
not property:index.decryption=success and \
not property:index.decryption=failure
I think we've diverged enough from the Xapian query parser
that we can't rely on that syntax description [1]. As far as I can
tell, [1] also only discusses quotes in the context of phrases.
[1]: https://xapian.org/docs/queryparser.html
Currently, notmuch has the levers needed to set coherent crypto policy
around how cleartext is indexed, which also has an impact on how
messages are rendered. But we don't have a lot of documentation about
how to do sensible things. This is an initial attempt to address
that.
The first example shows a way to selectively index specific messages.
The next two examples are about aligning the existing database with
crypto indexing policy
The default crypto policy is to not index cleartext, and to only
decrypt messages on display when explicitly requested.
The other sensible crypto policy is to index cleartext while stashing
session keys. messages indexed in this way will be searchable, and
will be decrypted on display automatically unless the user explicitly
asks for it to *not* be decrypted.
The policy for indexing *new* messages is stored in the database as
the config variable index.decrypt.
But setting policy for new messages doesn't retroactively affect
already indexed messages.
This patch attempts to document ways that someone can efficiently
align their pre-existing database with their new policy.
I'm not sure this is the right place to document these examples, but i
do want them to be user-facing and relatively easy to find. I'm happy
to entertain suggestions for where else we should put them.
In some cases (e.g. when building a publicly-visible e-mail archive)
it doesn't make any sense to restrict visibility of the message to the
current user account.
This adds a --world-readable boolean option for "notmuch insert", so
that those who want to archive their mail publicly can feed their
archiver with:
notmuch insert --world-readable
Other local delivery agents (postfix's local, and dovecot's lda) all
default to delivery in mode 0600 rather than relying on the user's
umask, so this fix doesn't change the default.
Also, this does not override the user's umask. if the umask is
already set tight, it will not become looser as the result of passing
--world-readable.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Need to be clearer about specifying time ranges using timestamps.
Legacy syntax which predates the date prefix is still supported, but
timestamps used in conjunction with the date prefix require additional
syntax.
Make all parameter descriptions etc. use reStructuredText definition
lists with uniform style and indentation. Remove redundant indentation
from around the lists. Remove blank lines between term lines and
definition blocks. Use four spaces for indentation.
This is almost completely whitespace and paragraph reflow changes.
This brings the --decrypt argument to "notmuch reply" into line with
the other --decrypt arguments (in "show", "new", "insert", and
"reindex"). This patch is really just about bringing consistency to
the user interface.
We also use the recommended form in the emacs MUA when replying, and
update test T350 to match.
We also expand tab completion for it, update the emacs bindings, and
update T350, T357, and T450 to match.
Make use of the bool-to-keyword backward-compatibility feature.
Add support for using /<regex>/ style regular expressions in
new.ignore, mixed with the old style verbatim file and directory
basenames. The regex is matched against the relative path from the
database path.
Having first a list of prefixes followed by detailed descriptions was
viable when we didn't have all that many prefixes. Now, arranging the
prefix descriptions in a definition list makes more sense.
While at it, include all the supported prefix forms, especially some
missing regex ones.
Now that the range of sensible decryption policies has come into full
view, we take a bit of space to document the distinctions.
Most people will use either "auto" or "true" -- but we provide "false"
and "nostash" to handle use cases that might reasonably be requested.
Note also that these can be combined in sensible ways. Like, if your
mail comes in regularly to a service that doesn't have access to your
secret keys, but does have access to your index, and you feel
comfortable adding selected encrypted messages to the index after
you've read them, you could stay in "auto" normally, and then when you
find yourself reading an indexable message (e.g. one you want to be
able to search for in the future, and that you don't mind exposing to
whatever entities have access to your inde), you can do:
notmuch reindex --decrypt=true id:whatever@example.biz
That leaves your default the same (still "auto") but you get the
cleartext index and stashed session key benefits for that particular
message.
Here's the configuration choice for people who want a cleartext index,
but don't want stashed session keys.
Interestingly, this "nostash" decryption policy is actually the same
policy that should be used by "notmuch show" and "notmuch reply",
since they never modify the index or database when they are invoked
with --decrypt.
We take advantage of this parallel to tune the behavior of those
programs so that we're not requesting session keys from GnuPG during
"show" and "reply" that we would then otherwise just throw away.
If you're going to store the cleartext index of an encrypted message,
in most situations you might just as well store the session key.
Doing this storage has efficiency and recoverability advantages.
Combined with a schedule of regular OpenPGP subkey rotation and
destruction, this can also offer security benefits, like "deletable
e-mail", which is the store-and-forward analog to "forward secrecy".
But wait, i hear you saying, i have a special need to store cleartext
indexes but it's really bad for me to store session keys! Maybe
(let's imagine) i get lots of e-mails with incriminating photos
attached, and i want to be able to search for them by the text in the
e-mail, but i don't want someone with access to the index to be
actually able to see the photos themselves.
Fret not, the next patch in this series will support your wacky
uncommon use case.
There are some situations where the user wants to get rid of the
cleartext index of a message. For example, if they're indexing
encrypted messages normally, but suddenly they run across a message
that they really don't want any trace of in their index.
In that case, the natural thing to do is:
notmuch reindex --decrypt=false id:whatever@example.biz
But of course, clearing the cleartext index without clearing the
stashed session key is just silly. So we do the expected thing and
also destroy any stashed session keys while we're destroying the index
of the cleartext.
Note that stashed session keys are stored in the xapian database, but
xapian does not currently allow safe deletion (see
https://trac.xapian.org/ticket/742).
As a workaround, after removing session keys and cleartext material
from the database, the user probably should do something like "notmuch
compact" to try to purge whatever recoverable data is left in the
xapian freelist. This problem really needs to be addressed within
xapian, though, if we want it fixed right.
The new "auto" decryption policy is not only good for "notmuch show"
and "notmuch reindex". It's also useful for indexing messages --
there's no good reason to not try to go ahead and index the cleartext
of a message that we have a stashed session key for.
This change updates the defaults and tunes the test suite to make sure
that they have taken effect.
The stashed session keys are stored internally as notmuch properties.
So a user or developer who is reading about those properties might
want to understand how they fit into the bigger picture.
Note here that decrypting with a stored session key no longer needs
-decrypt for "notmuch show" and "notmuch reply".
When showing a message, if the user doesn't specify --decrypt= at all,
but a stashed session key is known to notmuch, notmuch should just go
ahead and try to decrypt the message with the session key (without
bothering the user for access to their asymmetric secret key).
The user can disable this at the command line with --decrypt=false if
they really don't want to look at the e-mail that they've asked
notmuch to show them.
and of course, "notmuch show --decrypt" still works for accessing the
user's secret keys if necessary.
If the user doesn't specify --decrypt= at all, but a stashed session
key is known to notmuch, when replying to an encrypted message,
notmuch should just go ahead and decrypt.
The user can disable this at the command line with --decrypt=false,
though it's not clear why they would ever want to do that.
This new automatic decryption policy should make it possible to
decrypt messages that we have stashed session keys for, without
incurring a call to the user's asymmetric keys.
