Attempt to distinguish between errors indicating misconfiguration or
programmer error, which we consider "permanent", in the sense that
automatic retries are unlikely to be useful, and those indicating
transient error conditions. We consider XAPIAN_EXCEPTION transient
because it covers the important special case of locking failure.
If some software other than notmuch new renames or removes files
during the notmuch new scan (specifically after scandir but before
indexing the file), keep going instead of bailing out. Failing to
index the file is just a race condition between notmuch and the other
software; the rename could happen after the notmuch new scan
anyway. It's not fatal, and we'll catch the renamed files on the next
scan.
Add a new exit code for when files vanished, so the caller has a
chance to detect the race and re-run notmuch new to recover.
Reported by Paul Wise <pabs@debian.org> at
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=843127
Many of the external links found in the notmuch source can be resolved
using https instead of http. This changeset addresses as many as i
could find, without touching the e-mail corpus or expected outputs
found in tests.
notmuch-show --verify will now also process S/MIME multiparts if
encountered. Requires gmime-2.6 and gpgsm.
Based on work by Jameson Graef Rollins <jrollins@finestructure.net>.
Currently we key the address hash table with the case sensitive "name
<address>". Switch to case insensitive keying with just address, and
store the case sensitive name and address in linked lists. This will
be helpful in adding support for different deduplication schemes in
the future.
There will be a slight performance penalty for the current full case
sensitive name + address deduplication, but this is simpler as a whole
when other deduplication schemes are added, and I expect the schemes
to be added to become more popular than the current default.
Aparet from the possible performance penalty, the only user visible
change should be the change in the output ordering for
--output=count. The order is not guaranteed (and is based on hash
table traversal) currently anyway, so this should be of no
consequence.
It's becoming a maintenance burden to do anything things with the
crypto glue code twice, once for 2.4 and once for 2.6. I don't have
any 2.4 version available to test on my development machine anymore,
so the 2.4 specific code paths are likely not very well tested.
The function notmuch_exit_if_unmatched_db_uuid is split from
notmuch_process_shared_options because it needs an open notmuch
database.
There are two exceptional cases in uuid handling.
1) notmuch config and notmuch setup don't currently open the database,
so it doesn't make sense to check the UUID.
2) notmuch compact opens the database inside the library, so we either
need to open the database just to check uuid, or change the API.
In particular this fixes a recently encountered bug where the
"--config" argument to "notmuch setup" is silently ignored, which the
unpleasant consequence of overwriting the users config file.
Unfortunately it seems trickier to support --config globally
The non-trivial changes are in notmuch.c; most of the other changes
consists of blindly inserting two lines into every subcommand.
Previously we set up a way for the top level notmuch command to choose
which gpg binary was invoked by libgmime. In this commit we add the
(mostly boilerplate) code to allow the notmuch-config command to read
and write this path, and use it in the appropriate struct.
Update tests for new default variable
GMIME takes a path to gpg, but we hardcode that path. In this commit
we set up argument passing and option storage to allow this path to
specified in the top level notmuch command.
This moves address-related functionality from search command to the
new address command. The implementation shares almost all code and
some command line options.
Options --offset and --limit were intentionally not included in the
address command, because they refer to messages numbers, which users
do not see in the output. This could confuse users because, for
example, they could see more addresses in the output that what was
specified with --limit. This functionality can be correctly
reimplemented for address subcommand later.
Also useless values of --exclude flag were not included in the address
command.
This was inspired by a patch from Jani Nikula.
The main goal is to support gzipped output for future internal
calls (e.g. from notmuch-new) to notmuch_database_dump.
The additional dependency is not very heavy since xapian already pulls
in zlib.
We want the dump to be "atomic", in the sense that after running the
dump file is either present and complete, or not present. This avoids
certain classes of mishaps involving overwriting a good backup with a
bad or partial one.
These queries will match exactly the set of messages currently in the
thread, even if more messages later arrive. Two queries are provided:
one for matched messages and one for unmatched messages.
This can be used to fix race conditions with tagging threads from
search results. While tagging based on a thread: query can affect
messages that arrived after the search, tagging based on stable
queries affects only the messages the user was shown in the search UI.
Since we want clients to be able to depend on the presence of these
queries, this ushers in schema version 2.
Previously, the CLI would print a deprecation warning if a client
requested any format version other than the current one. However, if
we add fields that are backwards-compatible, but want clients to be
able to depend on, we need to bump the version, but that doesn't make
the older version deprecated.
Hence, separate out the "minimum active" version and only print a
warning for requests below this version number.
For my client, the largest bottleneck for displaying large threads is
exporting each html part individually since by default notmuch will not
show the json parts. For large threads there can be quite a few parts and
each must be exported and decoded one by one. Also, I then have to deal
with all the crazy charsets which I can do through a library but is a
pain.
Therefore, this patch adds an --include-html option that causes the
text/html parts to be included as part of the output of show.
diff man/man1/notmuch-show.1
The notmuch insert command reads a message from standard input,
writes it to a Maildir folder, and then incorporates the message into
the notmuch database. Essentially it moves the functionality of
notmuch-deliver into notmuch.
Though it could be used as an alternative to notmuch new, the reason
I want this is to allow my notmuch frontend to add postponed or sent
messages to the mail store and notmuch database, without resorting to
another tool (e.g. notmuch-deliver) nor directly modifying the maildir.
While looked good on paper, its attempted use caused confusion, complexity,
and potential for information leak when passed through wrapper scripts.
For slimmer code and to lessen demand for maintenance/support the set of
commits which added top level --stderr= option is now reverted.
With this option all writes to stderr are redirected to the specified
FILE (or to stdout on case FILE is '-'). This is immediately useful
in emacs interface as some of its exec intefaces do not provide
separation of stdout and stderr.
This allows specifying config file as a top level argument to notmuch,
and generally makes it possible to override config file options in
main(), without having to touch the subcommands.
If the config file does not exist, one will be created for the notmuch
main command and setup and help subcommands. Help is special in this
regard; the config is created just to avoid errors about missing
config, but it will not be saved.
This also makes notmuch config the talloc context for subcommands.
We now have a notmuch_config_is_new() function to query whether a
config was created or not. Change the notmuch_config_open() is_new
parameter into boolean create_new to determine whether the function
should create a new config if one doesn't exist. This reduces the
complexity of the API.
Keep track of whether the config is newly created, and add
notmuch_config_is_new() accessor function to query this.
This is to support anyone with a config handle to check this, instead
of just whoever called notmuch_config_open().
Currently there is a period of pain whenever we make
backward-incompatible changes to the structured output format, which
discourages not only backward-incompatible improvements to the format,
but also backwards-compatible additions that may not be "perfect". In
the end, these problems limit experimentation and innovation.
This series of patches introduces a way for CLI callers to request a
specific format version on the command line and to determine if the
CLI does not supported the requested version (and perhaps present a
useful diagnostic to the user). Since the caller requests a format
version, it's also possible for the CLI to support multiple
incompatible versions simultaneously, unlike the alternate approach of
including version information in the output.
This patch lays the groundwork by introducing a versioning convention,
standard exit codes, and a utility function to check the requested
version and produce standardized diagnostic messages and exit
statuses.
All the structured output functions in notmuch-reply and notmuch-show
are renamed to a generic name (as they do not contain any json-specific
code anyway). This patch is a preparation to actually using the new
S-Expression sprinter in notmuch-reply and notmuch-show.
There are several levels of function calls between where we create the
sprinter and the call to the part formatter in show_message. This
feeds the sprinter through all of them and into the part formatters.
This associates an sprinter constructor with each show format and uses
this to construct the appropriate sprinter. Currently nothing is done
with this sprinter, but the following patches will weave it through
the layers of notmuch show.
This option allows the caller to suppress the output of the bodies of
the messages. Currently this is only implemented for format=json.
This is used by notmuch-pick.el (although not needed) because it gives
a speed-up of at least a factor of a two (and in some cases a speed up
of more than a factor of 8); moreover it reduces the memory usage in
emacs hugely.
All formats except Json can output empty messages for non
entire-thread, but in Json format we output "null" to keep the other
elements (e.g. the replies to the omitted message) in the correct
place.
This has the affect of lazily creating the crypto contexts only when
needed. This removes code duplication from notmuch-show and
notmuch-reply, and should speed up these functions considerably if the
crypto flags are provided but the messages don't have any
cryptographic parts.
Use this flag rather than depend on the existence of an initialized
gpgctx, to determine whether we should verify a multipart/signed. We
will be moving to create the ctx lazily, so we don't want to depend on
it being previously initialized if it's not needed.
This new structure, notmuch_crypto_t, keeps all relevant crypto
contexts and parameters together, and will make it easier to pass the
stuff around and clean it up. The name of the crypto context inside
this new struct will change, to reflect that it is actually a GPG
context, which is a sub type of Crypto context. There are other types
of Crypto contexts (Pkcs7 in particular, which we hope to support) so
we want to be clear.
The new crypto.c contains functions to return the proper context from
the struct for a given protocol (and initialize it if needed), and to
cleanup a struct by releasing the crypto contexts.
gmime 2.4 defines GMimeCipherContext, while 2.6 defines
GMimeCryptoContext. typedef them both to notmuch_crypto_context_t to
cover this discrepancy and remove a bunch of #ifdefs.
This moves notmuch show to the --exclude=(true|false) naming
scheme. When exclude=false show returns all threads that match
including those that only match in an excluded message. The excluded
messages are flagged.
When exclude=true the behaviour depends on whether --entire-thread is
set. If it is not set then show only returns the messages which match
and are not excluded. If it is set then show returns all messages in
the threads that match in a non-excluded message, flagging the excluded
messages in these threads. The rationale is that it is awkward to use
a thread with some missing messages.
Previously, show and reply had separate implementations of decoding
and printing text parts. Now both use show's implementation, which
was more complete. Show's implementation has been extended with an
option to add reply quoting to the extracted part (this is implemented
as a named flag to avoid naked booleans, even though it's the only
flag it can take).
This new JSON format for replies includes headers generated for a
reply message as well as the headers of the original message. Using
this data, a client can intelligently create a reply. For example, the
emacs client will be able to create replies with quoted HTML parts by
parsing the HTML parts.
Formatter errors are propagated to the exit status of notmuch show.
This isn't used by the JSON or text formatters, but it will be useful
for the raw format, which is pickier.
A new configuration key 'new.ignore' is used to determine which
files and directories user wants not to be scanned as new mails.
Mark the corresponding test as no longer broken.
This work merges my previous attempts and Andreas Amann's work
in id:"ylp7hi23mw8.fsf@tyndall.ie"
Use notmuch_bool_t instead of int for entire_thread, raw, and decrypt
boolean fields in notmuch_show_params_t. No functional changes.
Signed-off-by: Jani Nikula <jani@nikula.org>
This callback is the gateway to the new mime_node_t-based formatters.
This maintains backwards compatibility so the formatters can be
transitioned one at a time. Once all formatters are converted, the
formatter structure can be reduced to only message_set_{start,sep,end}
and part, most of show_message can be deleted, and all of
show-message.c can be deleted.
This makes the part numbers readily accessible to formatters.
Hierarchical part numbering would be a more natural and efficient fit
for MIME and may be the way to go in the future, but depth-first
numbering maintains compatibility with what we currently do.
All other config-related functions and args include the section
title in their name, so for the sake of consistency, mirror that.
Also, the "auto"matic part is a given, so that was dropped.
There are lots of API changes in gmime 2.6 crypto handling. By adding
preprocessor directives, it is however possible to add gmime 2.6 compatibility
while preserving compatibility with gmime 2.4 too.
This is mostly based on id:"8762i8hrb9.fsf@bookbinder.fernseed.info".
This was tested against both gmime 2.6.4 and 2.4.31. With gmime 2.4.31, the
crypto tests all work fine (as expected). With gmime 2.6.4, one crypto test is
currently broken (signature verification with signer key unavailable), most
likely because of a bug in gmime which will hopefully be fixed in a future
version.
This adds a "search" section to the config file and an
"auto_tag_exclusions" setting in that section. The search and count
commands pass tag tags from the configuration to the library.
This function matches how we number parts for the --part argument to
show. It will allow us to jump directly to the desired part, rather
than traversing the entire tree and carefully tracking whether or not
we're "in the zone".
This wraps all of the complex MIME part handling in a single, simple
function that gets part N from *any* MIME object, so traversing a MIME
part tree becomes a two-line for loop. Furthermore, the MIME node
structure provides easy access to envelopes for message parts as well
as cryptographic information.
This code is directly derived from the current show_message_body code
(much of it is identical), but the control relation is inverted:
instead of show_message_body controlling the traversal of the MIME
structure and invoking callbacks, the caller controls the traversal of
the MIME structure.
Add mechanism for running user defined hooks. Hooks are executables or
symlinks to executables stored under the new notmuch hooks directory,
<database-path>/.notmuch/hooks.
No hooks are introduced here, but adding support for a hook is now a simple
matter of calling the new notmuch_run_hook() function at an appropriate
location with the hook name.
Signed-off-by: Jani Nikula <jani@nikula.org>
As we noticed when Jani kindly converted things to getopt_long, much
of the work in argument parsing in notmuch is due to the the key-value
style arguments like --format=(raw|json|text).
The framework here provides positional arguments, simple switches,
and --key=value style arguments that can take a value being an integer,
a string, or one of a set of keywords.
This new function takes a GMimeMessage as input, and outputs the
formatted headers. This allows for message/rfc822 parts to be
formatted on output in a similar way to full messages (see previous
patch that overhauls the multipart test for more info).
Our use of GMimeSession was unneeded boilerplate, and we weren't doing
anything with it. This simplifies and clarifies that assumption.
If we want to do anything fancier later, the examples in the gmime
source are a reasonable source to work from in defining a new
GMimeSession derivative.
Since GMimeSession is going away in GMime 2.6, though, i don't
recommend using it.
This adds support for decrypting PGP/MIME-encrypted parts to
notmuch-show and notmuch-reply. The --decrypt option implies
--verify. Once decryption (and possibly signature verification) is
done, a new part_encstatus formatter is emitted, the part_sigstatus
formatter is emitted, and the entire multipart/encrypted part is
replaced by the contents of the encrypted part.
At the moment only a json part_encstatus formatting function is
available, even though decryption is done for all formats. Emacs
support to follow.
This is primarily for notmuch-show, although the functionality is
added to show-message. Once signatures are processed a new
part_sigstatus formatter is emitted, and the entire multipart/signed
part is replaced with the contents of the signed part.
At the moment only a json part_sigstatus formatting function is
available. Emacs support to follow.
The original work for this patch was done by
Daniel Kahn Gillmor <dkg@fifthhorseman.net>
whose help with this functionality I greatly appreciate.
Future improvements (eg. crypto support) will require adding new part
header. By breaking up the output of part headers from the output of
part content, we can easily out new part headers with new formatting
functions.
Outputting of single MIME parts is moved to an option of notmuch show,
instead of being handled in it's own sub-command. The recent rework
of multipart mime allowed for this change but consolidating part
handling into a single recursive function (show_message_part) that
includes formatting. This allows for far simpler handling single
output of a single part, including formatting.
We rename here in order to make do_show_single into a generic function
for handling output of just a single message, or which format=raw is a
special case. The raw case is handled by setting a new parameter,
params.raw, which is used to tell do_show_single to output a single
message as a raw file.
This is mostly in preparation for much improved part handling to
follow imminently.
A new field "part_sep" is added to the notmuch_show_format structure,
to be used for part separation. This is cleaner than the "first"
argument that was being passed around to the part arguments, and
allows the function that handles overall part output formatting
(show_message_part) to directly handle when outputting the separator.
This simplifies the passing of arguments to the show functions. This
will be very useful as we accumulate more parameters that will need to
be passed. Currently only the entire_thread parameter is passed this
way.
Various show_message* functions require formatting functions, which
were previously being passed individually as arguments. Since we will
need to be needing to passing in more formatting function in the
future (ie. for crypto support), we here modify things so that we just
pass in the entire format structure. This will make things much
simpler down the line as we need to pass in new format functions.
We move the show_format structure into notmuch-client.c as
notmuch_show_format. This also affects notmuch-reply.c, so we create
a mostly-empty format_reply to pass the reply_part function to
show_message_body.
Previously, notmuch show flattened all output, losing information
about the nesting of the MIME hierarchy. Now, the output is properly
nested, (both in the --format=text and --format=json output), so that
clients can analyze the original MIME structure.
Internally, this required splitting the final closing delimiter out of
the various show_part functions and putting it into a new
show_part_end function instead. Also, the show_part function now
accepts a new "first" argument that is set not only for the first MIME
part of a message, but also for each first MIME part within a series
of multipart parts. This "first" argument controls the omission of a
preceding comma when printing a part (for json).
Many thanks to David Edmondson <dme@dme.org> for originally
identifying the lack of nesting in the json output and submitting an
early implementation of this feature. Thanks as well to Jameson Graef
Rollins <jrollins@finestructure.net> for carefully shepherding David's
patches through a remarkably long review process, patiently explaining
them, and providing a cleaned up series that led to this final
implementation. Jameson also provided the new emacs code here.
Since the name of the configuration parameter here is:
maildir.synchronize_flags
the convention is that the functions to get and set this parameter
should match it in name. Hence:
notmuch_config_get_maildir_synchronize_flags
etc. (as opposed to notmuch_config_get_maildir_sync).
This adds group [maildir] and key 'synchronize_flags' to the
configuration file. Its value enables (true) or diables (false) the
synchronization between notmuch tags and maildir flags. By default,
the synchronization is disabled.
This command outputs a raw message matched by search term to the
standard output. It allows MUAs to access the messages for piping,
attachment manipulation, etc. by running notmuch cat rather then
directly access the file. This will simplify the MUAs when they need
to operate on a remote database.
Edited-by: Carl Worth <cworth@cworth.org>: Remove trailing whitespace,
add missing "test_done" to new test script to avoid "Unexpected exit"
error.
Add a new_tags option in the [messages] section of the configuration
file to allow the user to specify which tags should be added to new
messages by notmuch new.
A new 'part' subcommand allows the user to extract a single part from
a MIME message. Usage:
notmuch part --part=<n> <search terms>
The search terms must match only a single message
(e.g. id:foo@bar.com). The part number specified refers to the part
identifiers output by `notmuch show'. The content of the part is
written the stdout with no formatting or identification marks. It is
not JSON formatted.
In the case of notmuch-show, "--format=json" also implies
"--entire-thread" as the thread structure is implicit in the emitted
document tree.
As a coincidence to the implementation, multipart message ID numbers are
now incremented with each part printed. This changes the previous
semantics, which were unclear and not necessary related to the actual
ordering of the message parts.
This was really the last thing keeping the initial run of "notmuch
new" being different from all other runs. And I'm taking a fresh
look at the performance of "notmuch new" anyway, so I think we can
safely drop this optimization.
This is a new notmuch command that can be used to search for all tags
found in the database. The resulting list is alphabetically sorted.
The primary use-case for this new command is to provide the tag
completion feature in Emacs (and other interfaces).
Signed-off-by: Jan Janak <jan@ryngle.com>
Getting the count of matching threads or messages is a fairly
expensive operation. Xapian provides a very efficient mechanism that
returns an approximate value, so use that for this new command.
This returns the number of matching messages, not threads, as that is
cheap to compute.
Signed-off-by: Keith Packard <keithp@keithp.com>
When running "notmuch new --verbose", ANSI escapes are used. This may not be
desirable when the output of the command is *not* being sent to a terminal
(e.g. when piping output into another command). In that case each file
processed is printed in a new line and ANSI escapes are not used at all.
For very large mail boxes, it is desirable to know which files are being
processed e.g. when a crash occurs to know which one was the cause. Also,
it may be interesting to have a better idea of how the operation is
progressing when processing mailboxes with big messages.
This patch adds support for printing messages as they are processed by
"notmuch new":
* The "new" command now supports a "--verbose" flag.
* When running in verbose mode, the file path of the message about to be
processed is printed in the following format:
current/total: /path/to/message/file
Where "current" is the number of messages processed so far and "total" is
the total count of files to be processed.
The status line is erased using an ANSI sequence "\033[K" (erase current
line from the cursor to the end of line) each time it is refreshed. This
should not pose a problem because nearly every terminal supports it.
* The signal handler for SIGALRM and the timer are not enabled when running
in verbose mode, because we are already printing progress with each file,
periodical reports are not neccessary.
I felt sorry for Carl trying to step through an exception from xapian
and suffering from the SIGALARMs..
We can detect if the user launched notmuch under a debugger by either
checking our cmdline for the presence of the gdb string or querying if
valgrind is controlling our process. For the latter we need to add a
compile time check for the valgrind development library, and so add the
initial support to build Makefile.config from configure.
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: Carl Worth <cworth@cworth.org>
[ickle: And do not install the timer when under the debugger]
We take the recently created text from the notmuch manual page and
update the "notmuch help" command to use similar text. In particular,
we add a new "notmuch help search-terms" for documenting the search
syntax that is common to several commands.
This time, things are actually tested. The current results aren't
exactly the same as previous results since the incremental search
doesn't necessarily see all the new messages that pertain to the
thread. This means that some author names are missing.
I plan to fix this by doing an additional database search for all
messages in each thread. Of course, this will also be different than
before since now the result will display *all* authors in the thread
(rather than only those that matched the search) but that's probably
what we really want to display anyway.
It's quite possible for someone to read the documentation and run
"notmuch setup" rather than just "notmuch". In that case, we don't
want to be any less welcoming.
This will allow for things like the database path to be specified
without any cheesy NOTMUCH_BASE environment variable. It also will
allow "notmuch reply" to recognize the user's email address when
constructing a reply in order to do the right thing, (that is, to use
the user's address to which mail was sent as From:, and not to reply
to the user's own addresses).
With this change, the "notmuch setup" command is now strictly for
changing the configuration of notmuch. It no longer creates the
database, but instead instructs the user to call "notmuch new" to do
that.
Reviewed-by: Carl Worth <cworth@cworth.org>
Keith wrote all the code here against notmuch before notmuch.c was
split up into multiple files. So I've pushed the code around in
various ways to match the new code structure, but have generally tried
to avoid making any changes to the behavior of the code.
I did fix one bug---a missing call to g_mime_stream_file_set_owner in
show_part which would cause "notmuch show" to go off into the weeds
when trying to show multiple messages, (since the first stream would
fclose stdout).
Now that the client sources are alone here in their own directory,
(with all the library sources down inside the lib directory), we can
break the client up into multiple files without mixing the files up.
The hope is that these smaller files will be easier to manage and
maintain.
