All the structured output functions in notmuch-reply and notmuch-show
are renamed to a generic name (as they do not contain any json-specific
code anyway). This patch is a preparation to actually using the new
S-Expression sprinter in notmuch-reply and notmuch-show.
Previously, the only mention of devel/schemata was a comment at the
top of format_part_json, but the JSON output code is spread across
several functions that are distributed across notmuch-show.c. Add
references from the other three key JSON output functions.
Output the Reply-To header field if present in a message.
I want to be able to see what the sender intended in my mail client,
before hitting the reply key. Only json output is changed,
like the recently added Bcc field.
Unlike the previous patches, this function is used for all formats.
However, for formats other than the JSON format, the sprinter methods
used by show_message are all no-ops, so this code continues to
function correctly for all of the formats.
Converting show_message eliminates show_null_message in the process,
since this maps directly to an sprinter method.
There are several levels of function calls between where we create the
sprinter and the call to the part formatter in show_message. This
feeds the sprinter through all of them and into the part formatters.
This associates an sprinter constructor with each show format and uses
this to construct the appropriate sprinter. Currently nothing is done
with this sprinter, but the following patches will weave it through
the layers of notmuch show.
This option allows the caller to suppress the output of the bodies of
the messages. Currently this is only implemented for format=json.
This is used by notmuch-pick.el (although not needed) because it gives
a speed-up of at least a factor of a two (and in some cases a speed up
of more than a factor of 8); moreover it reduces the memory usage in
emacs hugely.
The --entire-thread option in notmuch-show.c defaults to true when
format=json. Previously there was no way to turn this off. This patch
makes it respect --entire-thread=false.
To do this the patch moves the --entire-thread option to be a keyword
option using the new command line parsing to allow the existing
--entire-thread to keep working.
All formats except Json can output empty messages for non
entire-thread, but in Json format we output "null" to keep the other
elements (e.g. the replies to the omitted message) in the correct
place.
This has the affect of lazily creating the crypto contexts only when
needed. This removes code duplication from notmuch-show and
notmuch-reply, and should speed up these functions considerably if the
crypto flags are provided but the messages don't have any
cryptographic parts.
Use this flag rather than depend on the existence of an initialized
gpgctx, to determine whether we should verify a multipart/signed. We
will be moving to create the ctx lazily, so we don't want to depend on
it being previously initialized if it's not needed.
It has been a long-standing issue that notmuch_database_open doesn't
return any indication of why it failed. This patch changes its
prototype to return a notmuch_status_t and set an out-argument to the
database itself, like other functions that return both a status and an
object.
In the interest of atomicity, this also updates every use in the CLI
so that notmuch still compiles. Since this patch does not update the
bindings, the Python bindings test fails.
This moves notmuch show to the --exclude=(true|false) naming
scheme. When exclude=false show returns all threads that match
including those that only match in an excluded message. The excluded
messages are flagged.
When exclude=true the behaviour depends on whether --entire-thread is
set. If it is not set then show only returns the messages which match
and are not excluded. If it is set then show returns all messages in
the threads that match in a non-excluded message, flagging the excluded
messages in these threads. The rationale is that it is awkward to use
a thread with some missing messages.
Previously, show and reply had separate implementations of decoding
and printing text parts. Now both use show's implementation, which
was more complete. Show's implementation has been extended with an
option to add reply quoting to the extracted part (this is implemented
as a named flag to avoid naked booleans, even though it's the only
flag it can take).
This new JSON format for replies includes headers generated for a
reply message as well as the headers of the original message. Using
this data, a client can intelligently create a reply. For example, the
emacs client will be able to create replies with quoted HTML parts by
parsing the HTML parts.
This is fully compatible for root and leaf parts, but now has proper
support for interior parts. This requires some design decisions that
were guided by what I would want if I were to save a part.
Specifically:
- Leaf parts are printed without headers and with transfer decoding.
This is what makes sense for saving attachments. (Furthermore, the
transfer decoding is necessary since, without the headers, the
caller would not be able to interpret non-transfer-decoded output.)
- Message parts are printed with their message headers, but without
enclosing part headers. This is what makes sense for saving a
message as a whole (which is a message part) and for saving attached
messages. This is symmetric for whole messages and for attached
messages, though we special-case the whole message for performance
reasons (and corner-case correctness reasons: given malformed input,
GMime may not be able to reproduce it from the parsed
representation).
- Multipart parts are printed with their headers and all child parts.
It's not clear what the best thing to do for multipart is, but this
was the most natural to implement and can be justified because such
parts can't be interpreted without their headers.
As an added benefit, we can move the special-case code for part 0 into
the raw formatter.
Formatter errors are propagated to the exit status of notmuch show.
This isn't used by the JSON or text formatters, but it will be useful
for the raw format, which is pickier.
In all cases of notmuch count/search/show where the results returned
cannot reflect the exclude flag return just the matched not-excluded
results. If the caller wishes to have all the matched results (i.e.,
including the excluded ones) they should call with the
--no-exclude option.
The relevant cases are
count: both threads and messages
search: all cases except the summary view
show: mbox format
This adds the excludes to notmuch-show.c. We do not exclude when only
a single message (or part) is requested. notmuch-show will output the
exclude information when either text or json format is requested. As
this changes the output from notmuch-show it breaks many tests (in a
trivial and expected fashion).
This has three ramifications:
- Blank To and Cc headers are no longer output for messages.
- Dates are now canonicalized for messages, which means they always
have a day of the week and GMT is printed +0000 (never -0000)
- Invalid From message headers are handled slightly differently, since
they get parsed by GMime now instead of notmuch.
Previously, top-level message headers were printed as Subject, From,
To, Date, while embedded message headers were printed From, To,
Subject, Date. This makes both cases use the former order and updates
the tests accordingly.
As before, this is all code movement and a smidgen of glue. This
moves the existing JSON formatter code into one self-recursive
function, but doesn't change any of the logic to take advantage of the
new structure.
In general, "leafs" of the JSON structure are left in helper functions
(most of them untouched), so that it's easy to see the overall
structure of the format from the main recursive function.
The last lines of notmuch_show_command() function were
unreachable. Fix it by using a variable for return value.
Signed-off-by: Jani Nikula <jani@nikula.org>
Use the new notmuch argument parser to handle arguments in "notmuch
show". There are three minor functional changes:
1) Also set params.raw = TRUE when defaulting to raw format when part
is requested but format is not specified. This was a bug, and
--part=0 without --format=raw did not work previously.
2) Set params.decrypt = FALSE if crypto context creation fails.
3) Only use the parameters for the last --format if specified multiple
times. Previously this could have resulted in a non-working mixture
of parameters.
Signed-off-by: Jani Nikula <jani@nikula.org>
Use notmuch_bool_t instead of int for entire_thread, raw, and decrypt
boolean fields in notmuch_show_params_t. No functional changes.
Signed-off-by: Jani Nikula <jani@nikula.org>
This makes the text formatter take advantage of the new code
structure. The previously duplicated header logic is now unified,
several things that we used to compute repeatedly across different
callbacks are now computed once, and the code is simpler overall and
32% shorter.
Unifying the header logic causes this to format some dates slightly
differently, so the two affected test cases are updated.
This is all code movement and a smidgen of glue. This moves the
existing text formatter code into one self-recursive function, but
doesn't change any of the logic. The next patch will actually take
advantage of what the new structure has to offer.
Note that this patch retains format_headers_message_part_text because
it is also used by the raw format.
This callback is the gateway to the new mime_node_t-based formatters.
This maintains backwards compatibility so the formatters can be
transitioned one at a time. Once all formatters are converted, the
formatter structure can be reduced to only message_set_{start,sep,end}
and part, most of show_message can be deleted, and all of
show-message.c can be deleted.
Previously, top-level message headers were printed as Subject, From,
To, Date, while embedded message headers were printed From, To,
Subject, Date. This makes both cases use the former order and updates
the tests accordingly.
Strangely, the raw format also uses this function, so this also fixes
the two raw format tests affected by this change.
JSON does not support hex literals (0x..) so numbers must be formatted
as %d instead of %x.
Currently, the possible values for the gmime error code are 1 (expired
signature), 2 (no public key), 4 (expired key) and 8 (revoked key).
The other possible value is 16 (unsupported algorithm) but obviously
it is much more rare. If this happens, the current code will add
'"errors": 10'. This is valid JSON (it looks like a decimal number)
but it is incorrect (should be 16, not 10).
Since this is just an issue in the JSON encoder, no changes are needed
on the Emacs side (or in other UIs using the JSON output).
There are lots of API changes in gmime 2.6 crypto handling. By adding
preprocessor directives, it is however possible to add gmime 2.6 compatibility
while preserving compatibility with gmime 2.4 too.
This is mostly based on id:"8762i8hrb9.fsf@bookbinder.fernseed.info".
This was tested against both gmime 2.6.4 and 2.4.31. With gmime 2.4.31, the
crypto tests all work fine (as expected). With gmime 2.6.4, one crypto test is
currently broken (signature verification with signer key unavailable), most
likely because of a bug in gmime which will hopefully be fixed in a future
version.
For showing a message in raw format, rather than silently succeeding
when a read or a write fails (or, probably, looping if a read fails),
try to print an error message and exit with a non-zero status.
This silences one of the buildbot warnings about unused results. While
my libc lacks the declarations that trigger these warnings, this can
be tested by adding the following to notmuch.h:
__attribute__((warn_unused_result))
size_t fwrite(const void *ptr, size_t size, size_t nmemb, FILE *stream);
In my test case added g_object_unref(charset_filter) reduces memory
consumption over 90% when 'notmuch show --format=text "*"' is
executed (~11000 messages, RES ~330M -> ~25M).
g_object_unref() releases the memory of the InternetAddressList object
returned by internet_address_list_parse_string() -- when last (only)
reference is released, internet_address_list_finalize() will do cleanup.
previously we deleted the subcommand name from argv before passing to
the subcommand. In this version, the deletion is done in the actual
subcommands. Although this causes some duplication of code, it allows
us to be more flexible about how we parse command line arguments in
the subcommand, including possibly using off-the-shelf routines like
getopt_long that expect the name of the command in argv[0].
This was a minor oversite in checking of part type when outputing
content raw. This was causing gmime was to throw an exception to
stderr.
Unfortunately the gmime exception was not being caught by notmuch, or
the test suite. I'm not sure if notmuch should have done anything in
this case, but certainly the test suite should be capable of detecting
that something unexpected was output to stderr.
This new function takes a GMimeMessage as input, and outputs the
formatted headers. This allows for message/rfc822 parts to be
formatted on output in a similar way to full messages (see previous
patch that overhauls the multipart test for more info).
Before the change, notmuch show output had filename only for
parts with "Content-Disposition: attachment". But parts with
inline disposition may have filename as well.
The patch makes notmuch show always output filename if available,
independent of Content-Disposition. Both JSON and text output
formats are changed.
Also, the patch adds Content-id to text output format of notmuch
show.
The main goal of these changes is to have filenames on Emacs
buttons for inline attachments. In particular, this is very
helpful for inline patches.
Note: text format changes may require updates in clients that use
it. The changes are:
* text part header format changed from:
^Lpart{ ID: 2, Content-type: text/x-diff
to:
^Lpart{ ID: 2, Filename: cool-feature.patch, Content-type: text/x-diff
* attachment format changed from:
^Lattachment{ ID: 4, Content-type: application/octet-stream
Attachment: data.tar.bz2 (application/octet-stream)
Non-text part: application/octet-stream
^Lattachment}
to:
^Lattachment{ ID: 4, Filename: data.tar.bz2, Content-type: application/octet-stream
Non-text part: application/octet-stream
^Lattachment}
Our use of GMimeSession was unneeded boilerplate, and we weren't doing
anything with it. This simplifies and clarifies that assumption.
If we want to do anything fancier later, the examples in the gmime
source are a reasonable source to work from in defining a new
GMimeSession derivative.
Since GMimeSession is going away in GMime 2.6, though, i don't
recommend using it.
GMime has a nasty habit of taking ownership by default of any FILE*
handed to it va g_mime_stream_file_new. Specifically it will close the
FILE* when the stream is destroyed---even though GMime didn't open the
file itself.
To avoid this bad behavior, we have to carefully set_owner(FALSE)
after calling g_mime_stream_file_new. In the format_part_content_text
function, since commit d92146d3a6 we've
been calling g_mime_stream_file_new unconditionally, but only calling
g_mime_stream_file_set_owner(FALSE) conditionally.
This led to the FILE* being closed early when notmuch show output was
redirected to a file.
Fixing this fixes the test-suite cases that broke with the previous
commit, (which added redirected "notmuch show" calls to the test suite
to expose this bug).
Edited-by: Carl Worth <cworth@cworth.org> with a new commit message to
explain the bug and fix.
Now that this function registers an internal error if called with a
non-text part, we can remove the conditions for multipart and
message-part content types.
This fixes the recently-added test case (in test/multipart)
demonstrating corruption of binary parts that happen to contain CRLF
pairs.
We restore the original code from show_one_part_content to
format_part_content_raw. Then, for good measure, we rename
show_part_content to the more descriptive show_text_part_content and
add an internal error if it is ever called with a non-text part.
This adds support for decrypting PGP/MIME-encrypted parts to
notmuch-show and notmuch-reply. The --decrypt option implies
--verify. Once decryption (and possibly signature verification) is
done, a new part_encstatus formatter is emitted, the part_sigstatus
formatter is emitted, and the entire multipart/encrypted part is
replaced by the contents of the encrypted part.
At the moment only a json part_encstatus formatting function is
available, even though decryption is done for all formats. Emacs
support to follow.
This is primarily for notmuch-show, although the functionality is
added to show-message. Once signatures are processed a new
part_sigstatus formatter is emitted, and the entire multipart/signed
part is replaced with the contents of the signed part.
At the moment only a json part_sigstatus formatting function is
available. Emacs support to follow.
The original work for this patch was done by
Daniel Kahn Gillmor <dkg@fifthhorseman.net>
whose help with this functionality I greatly appreciate.
Future improvements (eg. crypto support) will require adding new part
header. By breaking up the output of part headers from the output of
part content, we can easily out new part headers with new formatting
functions.
Since message/rfc822 parts are really just a special kind of
multipart, we here normalize the handling of the two. This will
provide access to sub-parts of message/rfc822 parts, which was
previously unavailable.
Outputting of single MIME parts is moved to an option of notmuch show,
instead of being handled in it's own sub-command. The recent rework
of multipart mime allowed for this change but consolidating part
handling into a single recursive function (show_message_part) that
includes formatting. This allows for far simpler handling single
output of a single part, including formatting.
We rename here in order to make do_show_single into a generic function
for handling output of just a single message, or which format=raw is a
special case. The raw case is handled by setting a new parameter,
params.raw, which is used to tell do_show_single to output a single
message as a raw file.
This is mostly in preparation for much improved part handling to
follow imminently.
A new field "part_sep" is added to the notmuch_show_format structure,
to be used for part separation. This is cleaner than the "first"
argument that was being passed around to the part arguments, and
allows the function that handles overall part output formatting
(show_message_part) to directly handle when outputting the separator.
This simplifies the passing of arguments to the show functions. This
will be very useful as we accumulate more parameters that will need to
be passed. Currently only the entire_thread parameter is passed this
way.
Various show_message* functions require formatting functions, which
were previously being passed individually as arguments. Since we will
need to be needing to passing in more formatting function in the
future (ie. for crypto support), we here modify things so that we just
pass in the entire format structure. This will make things much
simpler down the line as we need to pass in new format functions.
We move the show_format structure into notmuch-client.c as
notmuch_show_format. This also affects notmuch-reply.c, so we create
a mostly-empty format_reply to pass the reply_part function to
show_message_body.
It's simply one more property of a MIME part that might be useful, and
json makes it so easy to add additional properties.
Signed-off-by: Jameson Rollins <jrollins@finestructure.net>
Previously, notmuch show flattened all output, losing information
about the nesting of the MIME hierarchy. Now, the output is properly
nested, (both in the --format=text and --format=json output), so that
clients can analyze the original MIME structure.
Internally, this required splitting the final closing delimiter out of
the various show_part functions and putting it into a new
show_part_end function instead. Also, the show_part function now
accepts a new "first" argument that is set not only for the first MIME
part of a message, but also for each first MIME part within a series
of multipart parts. This "first" argument controls the omission of a
preceding comma when printing a part (for json).
Many thanks to David Edmondson <dme@dme.org> for originally
identifying the lack of nesting in the json output and submitting an
early implementation of this feature. Thanks as well to Jameson Graef
Rollins <jrollins@finestructure.net> for carefully shepherding David's
patches through a remarkably long review process, patiently explaining
them, and providing a cleaned up series that led to this final
implementation. Jameson also provided the new emacs code here.
Previously, the outer multipart part of any multipart/mixed,
multipart/signed, etc. MIME message was silently omitted from the
"notmuch show" output. This prevented any client from correctly
determining to which parts a signature applies, for example.
Now, we actually emit these parts as their own parts. The output is
still flattened---the contained parts are not yet included "within"
the multipart part---so it's still not possible to determine to which
parts a signature applies, but this is one step along the path.
The test suite is updated to reflect this change, (though we'll
eventually want to fix the emacs interface to not display buttons for
the multipart enclosure parts as there's nothing useful for the user
to actually do with them).
This is part of an effort to avoid proliferation of excessive
top-level notmuch commands. Also, "raw" better captures the
functionality here, (as opposed to "cat" which is a fairly oblique
reference to a bad Unix abbreviation whose metaphor doesn't work here
since "notmuch cat" operates only on a single message and hence cannot
"con'cat'enate" anything).
This command outputs a raw message matched by search term to the
standard output. It allows MUAs to access the messages for piping,
attachment manipulation, etc. by running notmuch cat rather then
directly access the file. This will simplify the MUAs when they need
to operate on a remote database.
Edited-by: Carl Worth <cworth@cworth.org>: Remove trailing whitespace,
add missing "test_done" to new test script to avoid "Unexpected exit"
error.
We don't love the mbox format, but it's still sometimes the most
practical way to share a collection of messages as a single file.
Here we implement the "mboxrd" variant of the mbox file format. This
variant applies reversible escaping by prefixing a '>' character to
all lines in the email messages matching the regular expression:
"^>*From "
This allows the escaping to be reliably removed. A reader should remove
a '>' from any line matching the regular expression:
"^>>*From "
More details on the mboxrd formats (and others as well) can be found
here:
http://homepage.ntlworld.com/jonathan.deboynepollard/FGA/mail-mbox-formats.html
Micah Anderson reported an issue where a message failed to display in
the emacs interface, (it instead gave an error, "json-read-string: Bad
string format").
Micah tracked this down to the json output from "notmuch show" being
interrupted by a GMime error message:
gmime-CRITICAL **: g_mime_stream_filter_add: assertion
`GMIME_IS_FILTER (filter)
I tracked this down further to notmuch passing a NULL value to
g_mime_stream_filter_add. And this was due to calling
g_mime_filter_charset_new with a value of "unknown-8bit".
So we add a test message withe a Conten-Type of "text/plain;
charset=unknown-8bit" from Micah's message. Then we fix "notmuch show"
to test for NULL before calling g_mime_stream_filter_add. Bug fixed.
A new 'part' subcommand allows the user to extract a single part from
a MIME message. Usage:
notmuch part --part=<n> <search terms>
The search terms must match only a single message
(e.g. id:foo@bar.com). The part number specified refers to the part
identifiers output by `notmuch show'. The content of the part is
written the stdout with no formatting or identification marks. It is
not JSON formatted.
Include a 'date_unix' and 'date_relative' field in the JSON output for
each message. 'date_relative' can be used by a UI implementation,
whereas 'date_unix' is useful when scripting.
I ran into this while looking at the vim plugin. Vim's system() call
redirects output to a file and it was missing many of the part{ lines.
If stream_stdout is setup too early, it will overwrite the part start
when notmuch is redirected to a file.
Reviewed-by Carl Worth <cworth@cworth.org>: GMime is calling fseek
before every write to reset the FILE* to the position it believes is
correct based on the writes it has seen. Our code was getting
incorrect results because our GMime writes were interleaved with
non-GMime writes via printf.
The bug appears when writing to a file because it's seekable, but not
when writing to a pipe which is not.
The previous json patches forgot to add the notmuch tags to the json
output. This is inconsistent to the text output so here they are. We
just output a 'tags' field that sends notmuch tags as a json array.
Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
This is a fairly old regression. There has always been code to avoid
printing empty headers (such as Cc or Bcc with no values), but it has
been broken since notmuch_message_get_header was changed to return
an empty string rather than a NULL pointer for these fields.
We rename 'has_more' to 'valid' so that it can function whether
iterating in a forward or reverse direction. We also rename
'advance' to 'move_to_next' to setup parallel naming with
the proposed functions 'move_to_first', 'move_to_last', and
'move_to_previous'.
In the case of notmuch-show, "--format=json" also implies
"--entire-thread" as the thread structure is implicit in the emitted
document tree.
As a coincidence to the implementation, multipart message ID numbers are
now incremented with each part printed. This changes the previous
semantics, which were unclear and not necessary related to the actual
ordering of the message parts.
