This new JSON format for replies includes headers generated for a
reply message as well as the headers of the original message. Using
this data, a client can intelligently create a reply. For example, the
emacs client will be able to create replies with quoted HTML parts by
parsing the HTML parts.
This is fully compatible for root and leaf parts, but now has proper
support for interior parts. This requires some design decisions that
were guided by what I would want if I were to save a part.
Specifically:
- Leaf parts are printed without headers and with transfer decoding.
This is what makes sense for saving attachments. (Furthermore, the
transfer decoding is necessary since, without the headers, the
caller would not be able to interpret non-transfer-decoded output.)
- Message parts are printed with their message headers, but without
enclosing part headers. This is what makes sense for saving a
message as a whole (which is a message part) and for saving attached
messages. This is symmetric for whole messages and for attached
messages, though we special-case the whole message for performance
reasons (and corner-case correctness reasons: given malformed input,
GMime may not be able to reproduce it from the parsed
representation).
- Multipart parts are printed with their headers and all child parts.
It's not clear what the best thing to do for multipart is, but this
was the most natural to implement and can be justified because such
parts can't be interpreted without their headers.
As an added benefit, we can move the special-case code for part 0 into
the raw formatter.
Formatter errors are propagated to the exit status of notmuch show.
This isn't used by the JSON or text formatters, but it will be useful
for the raw format, which is pickier.
In all cases of notmuch count/search/show where the results returned
cannot reflect the exclude flag return just the matched not-excluded
results. If the caller wishes to have all the matched results (i.e.,
including the excluded ones) they should call with the
--no-exclude option.
The relevant cases are
count: both threads and messages
search: all cases except the summary view
show: mbox format
This adds the excludes to notmuch-show.c. We do not exclude when only
a single message (or part) is requested. notmuch-show will output the
exclude information when either text or json format is requested. As
this changes the output from notmuch-show it breaks many tests (in a
trivial and expected fashion).
This has three ramifications:
- Blank To and Cc headers are no longer output for messages.
- Dates are now canonicalized for messages, which means they always
have a day of the week and GMT is printed +0000 (never -0000)
- Invalid From message headers are handled slightly differently, since
they get parsed by GMime now instead of notmuch.
Previously, top-level message headers were printed as Subject, From,
To, Date, while embedded message headers were printed From, To,
Subject, Date. This makes both cases use the former order and updates
the tests accordingly.
As before, this is all code movement and a smidgen of glue. This
moves the existing JSON formatter code into one self-recursive
function, but doesn't change any of the logic to take advantage of the
new structure.
In general, "leafs" of the JSON structure are left in helper functions
(most of them untouched), so that it's easy to see the overall
structure of the format from the main recursive function.
The last lines of notmuch_show_command() function were
unreachable. Fix it by using a variable for return value.
Signed-off-by: Jani Nikula <jani@nikula.org>
Use the new notmuch argument parser to handle arguments in "notmuch
show". There are three minor functional changes:
1) Also set params.raw = TRUE when defaulting to raw format when part
is requested but format is not specified. This was a bug, and
--part=0 without --format=raw did not work previously.
2) Set params.decrypt = FALSE if crypto context creation fails.
3) Only use the parameters for the last --format if specified multiple
times. Previously this could have resulted in a non-working mixture
of parameters.
Signed-off-by: Jani Nikula <jani@nikula.org>
Use notmuch_bool_t instead of int for entire_thread, raw, and decrypt
boolean fields in notmuch_show_params_t. No functional changes.
Signed-off-by: Jani Nikula <jani@nikula.org>
This makes the text formatter take advantage of the new code
structure. The previously duplicated header logic is now unified,
several things that we used to compute repeatedly across different
callbacks are now computed once, and the code is simpler overall and
32% shorter.
Unifying the header logic causes this to format some dates slightly
differently, so the two affected test cases are updated.
This is all code movement and a smidgen of glue. This moves the
existing text formatter code into one self-recursive function, but
doesn't change any of the logic. The next patch will actually take
advantage of what the new structure has to offer.
Note that this patch retains format_headers_message_part_text because
it is also used by the raw format.
This callback is the gateway to the new mime_node_t-based formatters.
This maintains backwards compatibility so the formatters can be
transitioned one at a time. Once all formatters are converted, the
formatter structure can be reduced to only message_set_{start,sep,end}
and part, most of show_message can be deleted, and all of
show-message.c can be deleted.
Previously, top-level message headers were printed as Subject, From,
To, Date, while embedded message headers were printed From, To,
Subject, Date. This makes both cases use the former order and updates
the tests accordingly.
Strangely, the raw format also uses this function, so this also fixes
the two raw format tests affected by this change.
JSON does not support hex literals (0x..) so numbers must be formatted
as %d instead of %x.
Currently, the possible values for the gmime error code are 1 (expired
signature), 2 (no public key), 4 (expired key) and 8 (revoked key).
The other possible value is 16 (unsupported algorithm) but obviously
it is much more rare. If this happens, the current code will add
'"errors": 10'. This is valid JSON (it looks like a decimal number)
but it is incorrect (should be 16, not 10).
Since this is just an issue in the JSON encoder, no changes are needed
on the Emacs side (or in other UIs using the JSON output).
There are lots of API changes in gmime 2.6 crypto handling. By adding
preprocessor directives, it is however possible to add gmime 2.6 compatibility
while preserving compatibility with gmime 2.4 too.
This is mostly based on id:"8762i8hrb9.fsf@bookbinder.fernseed.info".
This was tested against both gmime 2.6.4 and 2.4.31. With gmime 2.4.31, the
crypto tests all work fine (as expected). With gmime 2.6.4, one crypto test is
currently broken (signature verification with signer key unavailable), most
likely because of a bug in gmime which will hopefully be fixed in a future
version.
For showing a message in raw format, rather than silently succeeding
when a read or a write fails (or, probably, looping if a read fails),
try to print an error message and exit with a non-zero status.
This silences one of the buildbot warnings about unused results. While
my libc lacks the declarations that trigger these warnings, this can
be tested by adding the following to notmuch.h:
__attribute__((warn_unused_result))
size_t fwrite(const void *ptr, size_t size, size_t nmemb, FILE *stream);
In my test case added g_object_unref(charset_filter) reduces memory
consumption over 90% when 'notmuch show --format=text "*"' is
executed (~11000 messages, RES ~330M -> ~25M).
g_object_unref() releases the memory of the InternetAddressList object
returned by internet_address_list_parse_string() -- when last (only)
reference is released, internet_address_list_finalize() will do cleanup.
previously we deleted the subcommand name from argv before passing to
the subcommand. In this version, the deletion is done in the actual
subcommands. Although this causes some duplication of code, it allows
us to be more flexible about how we parse command line arguments in
the subcommand, including possibly using off-the-shelf routines like
getopt_long that expect the name of the command in argv[0].
This was a minor oversite in checking of part type when outputing
content raw. This was causing gmime was to throw an exception to
stderr.
Unfortunately the gmime exception was not being caught by notmuch, or
the test suite. I'm not sure if notmuch should have done anything in
this case, but certainly the test suite should be capable of detecting
that something unexpected was output to stderr.
This new function takes a GMimeMessage as input, and outputs the
formatted headers. This allows for message/rfc822 parts to be
formatted on output in a similar way to full messages (see previous
patch that overhauls the multipart test for more info).
Before the change, notmuch show output had filename only for
parts with "Content-Disposition: attachment". But parts with
inline disposition may have filename as well.
The patch makes notmuch show always output filename if available,
independent of Content-Disposition. Both JSON and text output
formats are changed.
Also, the patch adds Content-id to text output format of notmuch
show.
The main goal of these changes is to have filenames on Emacs
buttons for inline attachments. In particular, this is very
helpful for inline patches.
Note: text format changes may require updates in clients that use
it. The changes are:
* text part header format changed from:
^Lpart{ ID: 2, Content-type: text/x-diff
to:
^Lpart{ ID: 2, Filename: cool-feature.patch, Content-type: text/x-diff
* attachment format changed from:
^Lattachment{ ID: 4, Content-type: application/octet-stream
Attachment: data.tar.bz2 (application/octet-stream)
Non-text part: application/octet-stream
^Lattachment}
to:
^Lattachment{ ID: 4, Filename: data.tar.bz2, Content-type: application/octet-stream
Non-text part: application/octet-stream
^Lattachment}
Our use of GMimeSession was unneeded boilerplate, and we weren't doing
anything with it. This simplifies and clarifies that assumption.
If we want to do anything fancier later, the examples in the gmime
source are a reasonable source to work from in defining a new
GMimeSession derivative.
Since GMimeSession is going away in GMime 2.6, though, i don't
recommend using it.
GMime has a nasty habit of taking ownership by default of any FILE*
handed to it va g_mime_stream_file_new. Specifically it will close the
FILE* when the stream is destroyed---even though GMime didn't open the
file itself.
To avoid this bad behavior, we have to carefully set_owner(FALSE)
after calling g_mime_stream_file_new. In the format_part_content_text
function, since commit d92146d3a6 we've
been calling g_mime_stream_file_new unconditionally, but only calling
g_mime_stream_file_set_owner(FALSE) conditionally.
This led to the FILE* being closed early when notmuch show output was
redirected to a file.
Fixing this fixes the test-suite cases that broke with the previous
commit, (which added redirected "notmuch show" calls to the test suite
to expose this bug).
Edited-by: Carl Worth <cworth@cworth.org> with a new commit message to
explain the bug and fix.
Now that this function registers an internal error if called with a
non-text part, we can remove the conditions for multipart and
message-part content types.
This fixes the recently-added test case (in test/multipart)
demonstrating corruption of binary parts that happen to contain CRLF
pairs.
We restore the original code from show_one_part_content to
format_part_content_raw. Then, for good measure, we rename
show_part_content to the more descriptive show_text_part_content and
add an internal error if it is ever called with a non-text part.
This adds support for decrypting PGP/MIME-encrypted parts to
notmuch-show and notmuch-reply. The --decrypt option implies
--verify. Once decryption (and possibly signature verification) is
done, a new part_encstatus formatter is emitted, the part_sigstatus
formatter is emitted, and the entire multipart/encrypted part is
replaced by the contents of the encrypted part.
At the moment only a json part_encstatus formatting function is
available, even though decryption is done for all formats. Emacs
support to follow.
This is primarily for notmuch-show, although the functionality is
added to show-message. Once signatures are processed a new
part_sigstatus formatter is emitted, and the entire multipart/signed
part is replaced with the contents of the signed part.
At the moment only a json part_sigstatus formatting function is
available. Emacs support to follow.
The original work for this patch was done by
Daniel Kahn Gillmor <dkg@fifthhorseman.net>
whose help with this functionality I greatly appreciate.
Future improvements (eg. crypto support) will require adding new part
header. By breaking up the output of part headers from the output of
part content, we can easily out new part headers with new formatting
functions.
Since message/rfc822 parts are really just a special kind of
multipart, we here normalize the handling of the two. This will
provide access to sub-parts of message/rfc822 parts, which was
previously unavailable.
Outputting of single MIME parts is moved to an option of notmuch show,
instead of being handled in it's own sub-command. The recent rework
of multipart mime allowed for this change but consolidating part
handling into a single recursive function (show_message_part) that
includes formatting. This allows for far simpler handling single
output of a single part, including formatting.
We rename here in order to make do_show_single into a generic function
for handling output of just a single message, or which format=raw is a
special case. The raw case is handled by setting a new parameter,
params.raw, which is used to tell do_show_single to output a single
message as a raw file.
This is mostly in preparation for much improved part handling to
follow imminently.
