Add support for using /<regex>/ style regular expressions in
new.ignore, mixed with the old style verbatim file and directory
basenames. The regex is matched against the relative path from the
database path.
Having first a list of prefixes followed by detailed descriptions was
viable when we didn't have all that many prefixes. Now, arranging the
prefix descriptions in a definition list makes more sense.
While at it, include all the supported prefix forms, especially some
missing regex ones.
Now that the range of sensible decryption policies has come into full
view, we take a bit of space to document the distinctions.
Most people will use either "auto" or "true" -- but we provide "false"
and "nostash" to handle use cases that might reasonably be requested.
Note also that these can be combined in sensible ways. Like, if your
mail comes in regularly to a service that doesn't have access to your
secret keys, but does have access to your index, and you feel
comfortable adding selected encrypted messages to the index after
you've read them, you could stay in "auto" normally, and then when you
find yourself reading an indexable message (e.g. one you want to be
able to search for in the future, and that you don't mind exposing to
whatever entities have access to your inde), you can do:
notmuch reindex --decrypt=true id:whatever@example.biz
That leaves your default the same (still "auto") but you get the
cleartext index and stashed session key benefits for that particular
message.
Here's the configuration choice for people who want a cleartext index,
but don't want stashed session keys.
Interestingly, this "nostash" decryption policy is actually the same
policy that should be used by "notmuch show" and "notmuch reply",
since they never modify the index or database when they are invoked
with --decrypt.
We take advantage of this parallel to tune the behavior of those
programs so that we're not requesting session keys from GnuPG during
"show" and "reply" that we would then otherwise just throw away.
If you're going to store the cleartext index of an encrypted message,
in most situations you might just as well store the session key.
Doing this storage has efficiency and recoverability advantages.
Combined with a schedule of regular OpenPGP subkey rotation and
destruction, this can also offer security benefits, like "deletable
e-mail", which is the store-and-forward analog to "forward secrecy".
But wait, i hear you saying, i have a special need to store cleartext
indexes but it's really bad for me to store session keys! Maybe
(let's imagine) i get lots of e-mails with incriminating photos
attached, and i want to be able to search for them by the text in the
e-mail, but i don't want someone with access to the index to be
actually able to see the photos themselves.
Fret not, the next patch in this series will support your wacky
uncommon use case.
There are some situations where the user wants to get rid of the
cleartext index of a message. For example, if they're indexing
encrypted messages normally, but suddenly they run across a message
that they really don't want any trace of in their index.
In that case, the natural thing to do is:
notmuch reindex --decrypt=false id:whatever@example.biz
But of course, clearing the cleartext index without clearing the
stashed session key is just silly. So we do the expected thing and
also destroy any stashed session keys while we're destroying the index
of the cleartext.
Note that stashed session keys are stored in the xapian database, but
xapian does not currently allow safe deletion (see
https://trac.xapian.org/ticket/742).
As a workaround, after removing session keys and cleartext material
from the database, the user probably should do something like "notmuch
compact" to try to purge whatever recoverable data is left in the
xapian freelist. This problem really needs to be addressed within
xapian, though, if we want it fixed right.
The new "auto" decryption policy is not only good for "notmuch show"
and "notmuch reindex". It's also useful for indexing messages --
there's no good reason to not try to go ahead and index the cleartext
of a message that we have a stashed session key for.
This change updates the defaults and tunes the test suite to make sure
that they have taken effect.
The stashed session keys are stored internally as notmuch properties.
So a user or developer who is reading about those properties might
want to understand how they fit into the bigger picture.
Note here that decrypting with a stored session key no longer needs
-decrypt for "notmuch show" and "notmuch reply".
When showing a message, if the user doesn't specify --decrypt= at all,
but a stashed session key is known to notmuch, notmuch should just go
ahead and try to decrypt the message with the session key (without
bothering the user for access to their asymmetric secret key).
The user can disable this at the command line with --decrypt=false if
they really don't want to look at the e-mail that they've asked
notmuch to show them.
and of course, "notmuch show --decrypt" still works for accessing the
user's secret keys if necessary.
If the user doesn't specify --decrypt= at all, but a stashed session
key is known to notmuch, when replying to an encrypted message,
notmuch should just go ahead and decrypt.
The user can disable this at the command line with --decrypt=false,
though it's not clear why they would ever want to do that.
This new automatic decryption policy should make it possible to
decrypt messages that we have stashed session keys for, without
incurring a call to the user's asymmetric keys.
the command-line interface for indexing (reindex, new, insert) used
--try-decrypt; and the configuration records used index.try_decrypt.
But by comparison with "show" and "reply", there doesn't seem to be
any reason for the "try" prefix.
This changeset adjusts the command-line interface and the
configuration interface.
For the moment, i've left indexopts_{set,get}_try_decrypt alone. The
subsequent changeset will address those.
When doing any decryption, if the notmuch database knows of any
session keys associated with the message in question, try them before
defaulting to using default symmetric crypto.
This changeset does the primary work in _notmuch_crypto_decrypt, which
grows some new parameters to handle it.
The primary advantage this patch offers is a significant speedup when
rendering large encrypted threads ("notmuch show") if session keys
happen to be cached.
Additionally, it permits message composition without access to
asymmetric secret keys ("notmuch reply"); and it permits recovering a
cleartext index when reindexing after a "notmuch restore" for those
messages that already have a session key stored.
Note that we may try multiple decryptions here (e.g. if there are
multiple session keys in the database), but we will ignore and throw
away all the GMime errors except for those that come from last
decryption attempt. Since we don't necessarily know at the time of
the decryption that this *is* the last decryption attempt, we'll ask
for the errors each time anyway.
This does nothing if no session keys are stashed in the database,
which is fine. Actually stashing session keys in the database will
come as a subsequent patch.
Now that it's easy to add argument specific modifiers in opt
descriptions, add a new .allow_empty field to allow empty strings for
individual string arguments while retaining strict checks
elsewhere. Use this for notmuch insert --folder, where the empty
string means top level folder.
Enable override of the index.try_decrypt setting on a per-run basis
when invoking "notmuch reindex". This allows the possibility of (for
example) an emacs keybinding that adds the cleartext of the currently
shown decrypted message to the index, making it searchable in the
future.
It also enables one-time indexing of all messages matching some query,
like so:
notmuch reindex tag:encrypted and\
not property:index.decryption=success and\
from:alice@example.org
We also update the documentation and tab completion, and add a few
more tests.
Enable override of the index.try_decrypt setting on a per-message
basis when invoking "notmuch insert".
We also update the documentation and tab completion, and add more tests.
Enable override of the index.try_decrypt setting during "notmuch new"
on a per-invocation basis.
We update the documentation and tab completion, and also add a test.
By default, notmuch won't try to decrypt on indexing. With this
patch, we make it possible to indicate a per-database preference using
the config variable "index.try_decrypt", which by default will be
false.
At indexing time, the database needs some way to know its internal
defaults for how to index encrypted parts. It shouldn't be contingent
on an external config file (since that can't be retrieved from the
database object itself), so we store it in the database.
This behaves similarly to the query.* configurations, which are also
stored in the database itself, so we're not introducing any new
dependencies by requiring that it be stored in the database.
QUERY_STRING was only used in two places, both to test whether a
variable should be stored in (or retrieved from) the database.
Since other configuration variables might be stored in the database in
the future, consolidate that test into a single function.
We also document that these configuration options should not be placed
in the config file.
If we see index options that ask us to decrypt when indexing a
message, and we encounter an encrypted part, we'll try to descend into
it.
If we can decrypt, we add the property index.decryption=success.
If we can't decrypt (or recognize the encrypted type of mail), we add
the property index.decryption=failure.
Note that a single message may have both values of the
"index.decryption" property: "success" and "failure". For example,
consider a message that includes multiple layers of encryption. If we
manage to decrypt the outer layer ("index.decryption=success"), but
fail on the inner layer ("index.decryption=failure").
Because of the property name, this will be automatically cleared (and
possibly re-set) during re-indexing. This means it will subsequently
correspond to the actual semantics of the stored index.
This allows us to create new properties that will be automatically set
during indexing, and cleared during re-indexing, just by choice of
property name.
By default, Sphinx tries to pre-process text through SmartyPants,
which attempts to convert ASCII quotes and dashes to Unicode
characters. Unfortunately, this mangles technical text such as command
lines. For instance, this excerpt from notmuch-tag.rst:
**notmuch** **tag** **--batch** [--input=<*filename*>]
got turned into:
notmuch tag –batch [–input=<filename>]
That's an en-dash and an em-dash respectively.
Not only are these characters visually confusing and could easily be
mistaken for a single dash, copying and pasting such command lines
into a terminal is doomed to result in incomprehensible error
messages.
* doc/conf.py: Disable SmartyPants.
A leading / in paths in a .gitignore file matches the beginning of the
path, meaning that for patterns without slashes, git will match files
only in the current directory as opposed to in any subdirectory.
Prefix relevant paths with / in .gitignore files, to prevent
accidentally ignoring files in subdirectories and possibly slightly
improve the performance of "git status".
gmime 3.0 no longer offers a means to set the path for gpg.
Users can set $PATH anyway if they want to pick a
differently-installed gpg (e.g. /usr/local/bin/gpg), so this isn't
much of a reduction in functionality.
The one main difference is for people who have tried to use "gpg2" to
make use of gpg 2.1, but that isn't usefully co-installable anyway.
the idea is that you can run
% notmuch search subject:/<your-favourite-regexp>/
% notmuch search from:/<your-favourite-regexp>/
or
% notmuch search subject:"your usual phrase search"
% notmuch search from:"usual phrase search"
This feature is only available with recent Xapian, specifically
support for field processors is needed.
It should work with bindings, since it extends the query parser.
This is easy to extend for other value slots, but currently the only
value slots are date, message_id, from, subject, and last_mod. Date is
already searchable; message_id is left for a followup commit.
This was originally written by Austin Clements, and ported to Xapian
field processors (from Austin's custom query parser) by yours truly.
In most part, our .rst documents are indented with 8 spaces instead
of tabs. Bring the rest of the lines to the same format.
Also, on one (supposedly empty) line, trailing spaces were removed.
If the --hello parameter is given, display the notmuch hello buffer
instead of the message composition buffer if no message composition
parameters are given.
Signed-off-by: Jani Nikula <jani@nikula.org>
Install man pages based on $(MAN_GZIP_FILES), which directly
corresponds to the man page source rst files. This way we can filter
the man pages to be installed as needed.
Use $(wildcard ...) to generate the list of man pages based on the rst
source files present in the man page directories, instead of reading
conf.py. This has three main benefits:
1) This makes the man page build slightly less complicated and easier
to understand. At least there are fewer moving parts.
2) This makes the build fail if we add a man page rst file, but fail
to add it to conf.py.
3) We can use Sphinx constructs in conf.py that are not available when
importing the file into a normal python program such as
mkdocdeps.py.
If Sphinx fails to create any of the roff files, having touch create
them hides the errors until someone realizes, possibly much later,
that the resulting files are empty. (Note that gzip doesn't fail on
empty input files.) Sphinx will change the timestamps of any files it
has written anyway.
Fix make sphinx-texinfo warnings:
WARNING: undefined label: notmuch-jump (if the link has no caption the
label must precede a section header)
WARNING: undefined label: notmuch-saved-searches (if the link has no
caption the label must precede a section header)
No need to repeat mostly the same information twice in conf.py. We
probably want to have a corresponding texinfo document for all the man
pages. Python list comprehension to the rescue. (The reverse is not
true; we have a texinfo document for notmuch-emacs we don't want as a
man page.)
There should be no user visible changes.
Add support for composing an email in the Notmuch Emacs UI using a
mailto: URL. The mailto: URL mode is mutually exclusive with
specifying other message modifying parameters and positional
arguments.
We want to be able to query the properties directly, like:
notmuch count property:foo=bar
which should return a count of messages where the property with key
"foo" has value equal to "bar".
sphinx-build emits a minor warning:
[...]doc/man7/notmuch-search-terms.rst:223: WARNING: Block quote ends without a blank line; unexpected unindent.
And the tabular representation of boolean or probabilistic prefixes
currently renders like this when i view it in man:
┌───────────────────────────┬────────────────────────────┐
│Boolean │ Probabilistic │
└───────────────────────────┴────────────────────────────┘
│ tag: id: │ from: to: │
│ │ │
│ thread: folder: │ subject: attach‐ │
│ path: │ ment: mimetype: │
└───────────────────────────┴────────────────────────────┘
This isn't just ugly: it's confusing, because it seems to imply that
some of the prefixes in the left-hand column are somehow related to
specific other prefixes in the right-hand column.
The Definition List representation introduced by this patch should be
simpler for readers to understand, and doesn't have the warning.
Many of the external links found in the notmuch source can be resolved
using https instead of http. This changeset addresses as many as i
could find, without touching the e-mail corpus or expected outputs
found in tests.
Most of the infrastructure here is general, only the validation/dispatch
is hardcoded to a particular prefix.
A notable change in behaviour is that notmuch-config now opens the
database e.g. on every call to list, which fails with an error message
if the database doesn't exit yet.
Current documentation and comments in the code do not correspond to
the actual code and tests in the test suite ("Un-munging Reply-To" in
T230-reply-to-sender.sh). Fix it.
<since> and <until> for the lastmod: prefix right below the date:
prefix description give the impression one could use last modified
dates to lastmod: which is not at all the case. Use
<initial-revision>..<final-revision> instead.
Before this change with --auto-daemon but without --create-frame
emacs server was started but no clients stay connected to it
(in both graphical and terminal displays).
Note that this changes how --client --auto-daemon works on
graphical display; New emacs frame is now created for the
message (and message-exit-actions hook appended).
Make the default behaviour for --client the same as emacsclient
default: do not create a new frame. Add a new option --create-frame,
passing the same option to emacsclient to create a frame.
Automatically starting Emacs in daemon mode if the Emacs server is not
running is a matter of preference better not hard coded in
notmuch-emacs-mua. Add an option to control the behaviour.
The order of the results with --output=count and --deduplicate=address
are unspecified as they're based on a hash table traversal. This being
the case, optimize the query by explicitly requesting unsorted
results. Clarify the documentation accordingly.
It doesn't seem likely we can support simple date:<expr> expanding to
date:<expr>..<expr> any time soon. (This can be done with a future
version of Xapian, or with a custom query query parser.) In the mean
time, provide shorthand date:<expr>..! to mean the same. This is
useful, as the expansion takes place before interpetation, and we can
use, for example, date:yesterday..! to match from beginning of
yesterday to end of yesterday.
Idea from Mark Walters <markwalters1009@gmail.com>.
The function notmuch_exit_if_unmatched_db_uuid is split from
notmuch_process_shared_options because it needs an open notmuch
database.
There are two exceptional cases in uuid handling.
1) notmuch config and notmuch setup don't currently open the database,
so it doesn't make sense to check the UUID.
2) notmuch compact opens the database inside the library, so we either
need to open the database just to check uuid, or change the API.
In the short term we need a way to get lastmod information e.g. for
the test suite. In the long term we probably want to add lastmod
information to at least the structured output for several other
clients (e.g. show, search).
Thanks to FreeBSD port maintainer Mikhail for report and the original
the original patch.
This is the right thing (TM) and also apparently fixes the build on
FreeBSD.
Previously we set up a way for the top level notmuch command to choose
which gpg binary was invoked by libgmime. In this commit we add the
(mostly boilerplate) code to allow the notmuch-config command to read
and write this path, and use it in the appropriate struct.
Update tests for new default variable
Notmuch uses long options exclusively all around. The short options in
notmuch-emacs-mua are intentionally just a compatible subset of
mutt(1). Keep it this way, if only to make documenting the fact easy!
The Notmuch style --client long option remains, of course.
Questions related to the way that probabilistic prefixes and phrases
are handled come up quite often and it is nicer to have the documentation self contained. Hopefully putting it in subsections prevents it from being overwhelming.
Adds new entry to the NEWS file, and updates the search terms section
of the man page. The search terms section needs to be updated again
once the new section in the documentation covering probablistic terms
has been committed.
As discussed in
id:8cc9dd580ad672527e12f43706f9803b2c8e99d8.1405220724.git.wking@tremily.us,
execfile is unavailable in python3.
The approach of this commit avoids modifying the python module path,
which is arguably preferable since it avoids potentially accidentally
importing a module from the wrong place.
When no --output option were given, change default to display senders
only.
When only --count option is given, display senders (in contrary to not
displaying anything).
Document that --count with --sort=**order** is not supported option
combination.
Clean up some whitespace in the documentation.
One test updated to have --output=count without sender nor recipient
output option.
Changed one tab in doc/man1/notmuch-show.rst to 8 spaces.
Removed one leading tab in doc/man5/notmuch-hooks.rst to make
the SYNOPSIS style consistent with other pages.
This filters out duplicate addresses from address command output.
It also also adds tests for the address command.
The code here is an extended version of a patch from Jani Nikula.
Now, when address related outputs are in a separate command, it makes
no sense to combine multiple --output options in search command line.
Using switch statement to handle different outputs is more readable
than a series of if statements.
This moves address-related functionality from search command to the
new address command. The implementation shares almost all code and
some command line options.
Options --offset and --limit were intentionally not included in the
address command, because they refer to messages numbers, which users
do not see in the output. This could confuse users because, for
example, they could see more addresses in the output that what was
specified with --limit. This functionality can be correctly
reimplemented for address subcommand later.
Also useless values of --exclude flag were not included in the address
command.
This was inspired by a patch from Jani Nikula.
The new outputs allow printing senders, recipients or both of matching
messages. To print both, the user can use --output=sender and
--output=recipients simultaneously.
Currently, the same address can appear multiple times in the output.
The next commit will change this. For this reason, tests are
introduced there.
We use mailbox_t rather than InternetAddressMailbox because we will
need to extend it in a following commit.
This code is based on a patch from Jani Nikula.
The post-new hook might no longer be needed or run very often if
notmuch insert is being used. Therefore a post-insert hook is needed
(arguably pre-insert not so much, so don't add one). Also add the
--no-hooks option to skip hooks.
Add --keep option to keep any remaining stuff in index or file. We
could distinguish between failures to index and failures to apply tags
or maildir sync, but for simplicity just have one.
Although this manual is far from complete, it may be helpful for
someone. In particular building it as part of the standard build
process makes it easier to find problems when editing the
notmuch-emacs-manual.
There is a doxygen bug about these odd files,
https://bugzilla.gnome.org/show_bug.cgi?id=727796
But it isn't clear if / when a fix will be provided, so just delete it
to avoid e.g. confusing man-to-wiki.pl
Avoid:
$ make HAVE_SPHINX=1 sphinx-html
python ./doc/mkdocdeps.py ./doc doc/_build doc/docdeps.mk
sphinx-build -b html -d doc/_build/doctrees -q ./doc doc/_build/html
Making output directory...
WARNING: html_static_path entry '/home/wking/src/notmuch/notmuch/doc/_static' does not exist
because we have no static source in doc/_static.
Previously, this was a verbatim copy of the --format=json text.
Change it to instead reference the JSON text and actually describe how
the S-expression format works.
Avoid:
$ make HAVE_SPHINX=0 HAVE_RST2MAN=1 build-man
python ./doc/prerst2man.py ./doc doc/_build/man
Traceback (most recent call last):
File "./doc/prerst2man.py", line 65, in <module>
os.system('set -x; rst2man {0} {1}/{2}.{3}'
NameError: name 'os' is not defined
make: *** [doc/_build/man/man1/notmuch.1] Error 1
by using system directly. We don't need the 'os.' namespacing,
because the function was imported with:
from os import makedirs, system
Before this patch, the open was unnecessarily early and relied on the
process cleanup to close. Neither one of these was a real problem,
but PEP 343's context managers (which landed in Python 2.5) make
proper cleanup very easy.
[1]: http://legacy.python.org/dev/peps/pep-0343/
The roff build rule builds all of the roff files in a single command.
Previously, this was expressed as a multi-target rule, but since this
is equivalent to specifying a copy of the rule for each target, make
-jN could start up to N parallel instances of this command. Fix this
by bottlenecking this rule through a single stamp file.
This also removes the unused man.stamp from CLEAN.
The main goal is to support gzipped output for future internal
calls (e.g. from notmuch-new) to notmuch_database_dump.
The additional dependency is not very heavy since xapian already pulls
in zlib.
We want the dump to be "atomic", in the sense that after running the
dump file is either present and complete, or not present. This avoids
certain classes of mishaps involving overwriting a good backup with a
bad or partial one.
While adding that fixed (also other) typos noticed by aspell(1) run,
and capitalized Emacs and (most) Notmuch terms to match how emacs
Info documentation seems to look in general.
The subtle part is adding .rst and .py files to vpath so they can be
used as dependencies without prefixing with $(srcdir)
We also change the interface to mkbuildeps.py: rather than getting the
containing directory from the conf file path, we go the other way.
Because sphinx-build does not provide a convenient way of listing
which builders exist, and some people actually have pre 1.0 sphinx, we
try loading a relevant python module.
Currently the assumption is that no python in path -> no sphinx-build
in path.
It turns out there was a reason the old man pages were stored in a man
compatible hierarchy, namely so that we could run man on them before
installing.
Hardcode doc build location into test suite. This isn't ideal, but
let's unbreak the test suite for now.
This helps avoid build artifacts (namely, nroff and gzipped-nroff man
pages) owned by root.
The variables allow choosing which generator to use for the man page.
These will be hooked to configure in a following commit.
The python script mkdocdeps.py is used to import the list of man pages
from the sphinx configuration to make.
This will delete the (release only) target update-man-versions. This
will be replaced in a followup commit.
This is the output from sphinx-quickstart, massaged a bit, along with
our existing man pages converted to rst.
A skeleton notmuch-emacs manual is also included. It is not suitable
for end user use yet.
