Since release 0.32, libnotmuch provides searching for database and
configuration paths. This commit changes the python module notmuch2 to
use those facilities.
This fixes the bug reported in [1], along with a couple of the
deprecation warnings in the python bindings.
Database.default_path is deprecated, since it no longer faithfully
reflects what libnotmuch is doing, and it is also no longer used in
the bindings themselves.
This commit choose the default of config=CONFIG.EMPTY (equivalent to
passing "" to notmuch_database_open_with_config). This makes the
change upward compatible API-wise (at least as far as the test suite
verifies), but changing the default to CONFIG.SEARCH would probably be
more convenient for bindings users.
[1]: id:87h7d4wp6b.fsf@tethera.net
This commit does not enable using saved s-expression queries, only
saving and retrieving them from the config file or the database. Use
in queries will be enabled in a following commit.
New --sort CLI option documented in notmuch-show's man page, and
notmuch-search-toggle-order mentioned in doc/notmuch-emacs.rst and
devel/emacs-keybindings.org (in the latter, there's also some
whitespace changes in a table introduced by org-mode).
Example reference to a command-line option using the option role
reference. This creates a hyperlink in html, and the usual boldface
style in man page. This could be used throughout the man pages.
Use the program and option directives to document the subcommand
options. This unifies a lot of option documentation throughout.
This also makes it possible to reference options with :option:`--foo`
(within .. program::) or :option:`subcommand --foo` (globally). This
is left for later work.
See https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#directive-program
Note: There is a lot of indentation change, but intentionally there is
no reflow. Using 'git diff -w' or 'git show -w' to ignore white space
changes makes this a very easy change to review.
Make man1/notmuch.rst the single point of truth for describing notmuch
environment variables. Use the envvar directive for that, and
reference them with the envvar role.
Drive-by cleanup configuration file and hook directory search order
documentation.
Add internal hyperlink targets for man pages and cross-reference them
using the any role reference. There are a number of alternatives to
accomplish this, but this seems like the combination that retains the
man page section number and the same boldface style in the man pages.
As a bonus, we get sanity checking on the links; for example
notmuch-search-terms.rst had a reference to notmuch-properties(1)
i.e. the wrong section.
The obvious semantic follow-up change would be to only have meaningful
"see also" references instead of having them all everywhere.
Using manpage role references generates helpful links in html
documentation, while retaining the same boldface style in the man
pages.
The external man page site is configurable. The Debian manpage site
seems like a good fit for Notmuch.
Previously the fact that some configuration options were only stored
in the database (and thus editing the config file had no effect) was a
source of user confusion. This was fixed with the series ending at
d9af0af164.
On the other hand, the underlying partition of config options into
those stored by default in the database and those stored in the config
file remained. This is also confusing, since now some invocations of
"notmuch config set" modify the config file, and others silently
modify the database instead.
With this commit, it is up to the user to decide which configuration
to modify. A new "--database" option is provided for notmuch config to
enable modifying the configuration information in the database;
otherwise the default is to update an external config file.
Introduce a new configuration value for the mail root, and use it to
locate mail messages in preference to the database.path (which
previously implied the mail messages were also in this location.
Initially only a subset of the CLI is tested in a split
configuration. Further changes will be needed for the remainder of the
CLI to work in split configurations.
Remove STORED IN DATABASE discussion, describe search rules.
Currently profiles seem a bit pointless. They will make more sense
when they apply to databases as well.
The features that require field processor support, are now just
documented w/o mentioning **Xapian Field Processors**' is needed
for those.
Replaced "compact" and "field_processor" with "retry_lock" in
build_with config option, as it is currently the only one that
is optionally excluded. The former 2 are now documented as
features always included.
Dropped one 'we' "passive" in notmuch-search-terms.rst. It was the
only one, and inconsistent with rest of the documentation in that
file.
Dropped message about conditional open-ended ranges support, as
those are now always supported.
id:CA+Tk8fzRiqxWpd=r8=DRvEewNZXUZgD7MKyRLB1A=R-LxxGEZw@mail.gmail.com
started a thread of discussion that showed that the cli's current
idiosyncrasies around dealing with boolean options were not
understandable.
This attempts to improve the documentation at least (actual changes to
the API might be better, but have not reached consensus).
Note that no one in the discussion thread identified any other
(non-boolean) command-line options that cannot use space as a
separator. If such an option is identified (or introduced in the
future), it should be added explicitly to this part of the manual.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
crypto.gpg_path was only used when we built against gmime versions
before 3.0. Since we now depend on gmime 3.0.3 or later, it is
meaningless.
The removal of the field from the _notmuch_config struct would be an
ABI change if that struct were externally exposed, but it is not, so
it's safe to unilaterally remove it.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
The escaping in the description of the output of "notmuch-config list"
appears to have been inherited from some previous attempts at
documentation. It leaked out in the actual generated manpage
documentation, where it looks like this:
list Every configuration item is printed to stdout, each on a
separate line of the form:
*section*.\ *item*\ =\ *value*
This simplification cleans up the overescaping.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Many of the manpages didn't treat literal text as literal text. I've
tried to normalize some of the restructured text to make it a bit more
regular.
several of the synopsis lines are still untouched by this cleanup, but
i'm not sure what the right way to represent those is in .rst,
actually.
In particular find that if i rebuild the manpages, sometimes i end up
with some of the synopsis lines showing – (U+2013 EN DASH) where they
should have -- (2 × U+002D HYPHEN-MINUS) in the generated nroff
output, though i have not tracked down the source of this error yet.
Add fancy new feature, which makes "notmuch show" capable of actually
indexing messages that it just decrypted.
This enables a workflow where messages can come in in the background
and be indexed using "--decrypt=auto". But when showing an encrypted
message for the first time, it gets automatically indexed.
This is something of a departure for "notmuch show" -- in particular,
because it requires read/write access to the database. However, this
might be a common use case -- people get mail delivered and indexed in
the background, but only want access to their secret key to happen
when they're directly interacting with notmuch itself.
In such a scenario, they couldn't search newly-delivered, encrypted
messages, but they could search for them once they've read them.
Documentation of this new feature also uses a table form, similar to
that found in the description of index.decrypt in notmuch-config(1).
A notmuch UI that wants to facilitate this workflow while also
offering an interactive search interface might instead make use of
these additional commands while the user is at the console:
Count received encrypted messages (if > 0, there are some things we
haven't yet tried to index, and therefore can't yet search):
notmuch count tag:encrypted and \
not property:index.decryption=success and \
not property:index.decryption=failure
Reindex those messages:
notmuch reindex --try-decrypt=true tag:encrypted and \
not property:index.decryption=success and \
not property:index.decryption=failure
Currently, notmuch has the levers needed to set coherent crypto policy
around how cleartext is indexed, which also has an impact on how
messages are rendered. But we don't have a lot of documentation about
how to do sensible things. This is an initial attempt to address
that.
The first example shows a way to selectively index specific messages.
The next two examples are about aligning the existing database with
crypto indexing policy
The default crypto policy is to not index cleartext, and to only
decrypt messages on display when explicitly requested.
The other sensible crypto policy is to index cleartext while stashing
session keys. messages indexed in this way will be searchable, and
will be decrypted on display automatically unless the user explicitly
asks for it to *not* be decrypted.
The policy for indexing *new* messages is stored in the database as
the config variable index.decrypt.
But setting policy for new messages doesn't retroactively affect
already indexed messages.
This patch attempts to document ways that someone can efficiently
align their pre-existing database with their new policy.
I'm not sure this is the right place to document these examples, but i
do want them to be user-facing and relatively easy to find. I'm happy
to entertain suggestions for where else we should put them.
In some cases (e.g. when building a publicly-visible e-mail archive)
it doesn't make any sense to restrict visibility of the message to the
current user account.
This adds a --world-readable boolean option for "notmuch insert", so
that those who want to archive their mail publicly can feed their
archiver with:
notmuch insert --world-readable
Other local delivery agents (postfix's local, and dovecot's lda) all
default to delivery in mode 0600 rather than relying on the user's
umask, so this fix doesn't change the default.
Also, this does not override the user's umask. if the umask is
already set tight, it will not become looser as the result of passing
--world-readable.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Make all parameter descriptions etc. use reStructuredText definition
lists with uniform style and indentation. Remove redundant indentation
from around the lists. Remove blank lines between term lines and
definition blocks. Use four spaces for indentation.
This is almost completely whitespace and paragraph reflow changes.
This brings the --decrypt argument to "notmuch reply" into line with
the other --decrypt arguments (in "show", "new", "insert", and
"reindex"). This patch is really just about bringing consistency to
the user interface.
We also use the recommended form in the emacs MUA when replying, and
update test T350 to match.
We also expand tab completion for it, update the emacs bindings, and
update T350, T357, and T450 to match.
Make use of the bool-to-keyword backward-compatibility feature.
Add support for using /<regex>/ style regular expressions in
new.ignore, mixed with the old style verbatim file and directory
basenames. The regex is matched against the relative path from the
database path.
Now that the range of sensible decryption policies has come into full
view, we take a bit of space to document the distinctions.
Most people will use either "auto" or "true" -- but we provide "false"
and "nostash" to handle use cases that might reasonably be requested.
Note also that these can be combined in sensible ways. Like, if your
mail comes in regularly to a service that doesn't have access to your
secret keys, but does have access to your index, and you feel
comfortable adding selected encrypted messages to the index after
you've read them, you could stay in "auto" normally, and then when you
find yourself reading an indexable message (e.g. one you want to be
able to search for in the future, and that you don't mind exposing to
whatever entities have access to your inde), you can do:
notmuch reindex --decrypt=true id:whatever@example.biz
That leaves your default the same (still "auto") but you get the
cleartext index and stashed session key benefits for that particular
message.
Here's the configuration choice for people who want a cleartext index,
but don't want stashed session keys.
Interestingly, this "nostash" decryption policy is actually the same
policy that should be used by "notmuch show" and "notmuch reply",
since they never modify the index or database when they are invoked
with --decrypt.
We take advantage of this parallel to tune the behavior of those
programs so that we're not requesting session keys from GnuPG during
"show" and "reply" that we would then otherwise just throw away.
If you're going to store the cleartext index of an encrypted message,
in most situations you might just as well store the session key.
Doing this storage has efficiency and recoverability advantages.
Combined with a schedule of regular OpenPGP subkey rotation and
destruction, this can also offer security benefits, like "deletable
e-mail", which is the store-and-forward analog to "forward secrecy".
But wait, i hear you saying, i have a special need to store cleartext
indexes but it's really bad for me to store session keys! Maybe
(let's imagine) i get lots of e-mails with incriminating photos
attached, and i want to be able to search for them by the text in the
e-mail, but i don't want someone with access to the index to be
actually able to see the photos themselves.
Fret not, the next patch in this series will support your wacky
uncommon use case.
There are some situations where the user wants to get rid of the
cleartext index of a message. For example, if they're indexing
encrypted messages normally, but suddenly they run across a message
that they really don't want any trace of in their index.
In that case, the natural thing to do is:
notmuch reindex --decrypt=false id:whatever@example.biz
But of course, clearing the cleartext index without clearing the
stashed session key is just silly. So we do the expected thing and
also destroy any stashed session keys while we're destroying the index
of the cleartext.
Note that stashed session keys are stored in the xapian database, but
xapian does not currently allow safe deletion (see
https://trac.xapian.org/ticket/742).
As a workaround, after removing session keys and cleartext material
from the database, the user probably should do something like "notmuch
compact" to try to purge whatever recoverable data is left in the
xapian freelist. This problem really needs to be addressed within
xapian, though, if we want it fixed right.
The new "auto" decryption policy is not only good for "notmuch show"
and "notmuch reindex". It's also useful for indexing messages --
there's no good reason to not try to go ahead and index the cleartext
of a message that we have a stashed session key for.
This change updates the defaults and tunes the test suite to make sure
that they have taken effect.
When showing a message, if the user doesn't specify --decrypt= at all,
but a stashed session key is known to notmuch, notmuch should just go
ahead and try to decrypt the message with the session key (without
bothering the user for access to their asymmetric secret key).
The user can disable this at the command line with --decrypt=false if
they really don't want to look at the e-mail that they've asked
notmuch to show them.
and of course, "notmuch show --decrypt" still works for accessing the
user's secret keys if necessary.
If the user doesn't specify --decrypt= at all, but a stashed session
key is known to notmuch, when replying to an encrypted message,
notmuch should just go ahead and decrypt.
The user can disable this at the command line with --decrypt=false,
though it's not clear why they would ever want to do that.
This new automatic decryption policy should make it possible to
decrypt messages that we have stashed session keys for, without
incurring a call to the user's asymmetric keys.
