As before, this is all code movement and a smidgen of glue. This
moves the existing JSON formatter code into one self-recursive
function, but doesn't change any of the logic to take advantage of the
new structure.
In general, "leafs" of the JSON structure are left in helper functions
(most of them untouched), so that it's easy to see the overall
structure of the format from the main recursive function.
The last lines of notmuch_show_command() function were
unreachable. Fix it by using a variable for return value.
Signed-off-by: Jani Nikula <jani@nikula.org>
Use the new notmuch argument parser to handle arguments in "notmuch
show". There are three minor functional changes:
1) Also set params.raw = TRUE when defaulting to raw format when part
is requested but format is not specified. This was a bug, and
--part=0 without --format=raw did not work previously.
2) Set params.decrypt = FALSE if crypto context creation fails.
3) Only use the parameters for the last --format if specified multiple
times. Previously this could have resulted in a non-working mixture
of parameters.
Signed-off-by: Jani Nikula <jani@nikula.org>
Use notmuch_bool_t instead of int for entire_thread, raw, and decrypt
boolean fields in notmuch_show_params_t. No functional changes.
Signed-off-by: Jani Nikula <jani@nikula.org>
This makes the text formatter take advantage of the new code
structure. The previously duplicated header logic is now unified,
several things that we used to compute repeatedly across different
callbacks are now computed once, and the code is simpler overall and
32% shorter.
Unifying the header logic causes this to format some dates slightly
differently, so the two affected test cases are updated.
This is all code movement and a smidgen of glue. This moves the
existing text formatter code into one self-recursive function, but
doesn't change any of the logic. The next patch will actually take
advantage of what the new structure has to offer.
Note that this patch retains format_headers_message_part_text because
it is also used by the raw format.
This callback is the gateway to the new mime_node_t-based formatters.
This maintains backwards compatibility so the formatters can be
transitioned one at a time. Once all formatters are converted, the
formatter structure can be reduced to only message_set_{start,sep,end}
and part, most of show_message can be deleted, and all of
show-message.c can be deleted.
Previously, top-level message headers were printed as Subject, From,
To, Date, while embedded message headers were printed From, To,
Subject, Date. This makes both cases use the former order and updates
the tests accordingly.
Strangely, the raw format also uses this function, so this also fixes
the two raw format tests affected by this change.
JSON does not support hex literals (0x..) so numbers must be formatted
as %d instead of %x.
Currently, the possible values for the gmime error code are 1 (expired
signature), 2 (no public key), 4 (expired key) and 8 (revoked key).
The other possible value is 16 (unsupported algorithm) but obviously
it is much more rare. If this happens, the current code will add
'"errors": 10'. This is valid JSON (it looks like a decimal number)
but it is incorrect (should be 16, not 10).
Since this is just an issue in the JSON encoder, no changes are needed
on the Emacs side (or in other UIs using the JSON output).
There are lots of API changes in gmime 2.6 crypto handling. By adding
preprocessor directives, it is however possible to add gmime 2.6 compatibility
while preserving compatibility with gmime 2.4 too.
This is mostly based on id:"8762i8hrb9.fsf@bookbinder.fernseed.info".
This was tested against both gmime 2.6.4 and 2.4.31. With gmime 2.4.31, the
crypto tests all work fine (as expected). With gmime 2.6.4, one crypto test is
currently broken (signature verification with signer key unavailable), most
likely because of a bug in gmime which will hopefully be fixed in a future
version.
For showing a message in raw format, rather than silently succeeding
when a read or a write fails (or, probably, looping if a read fails),
try to print an error message and exit with a non-zero status.
This silences one of the buildbot warnings about unused results. While
my libc lacks the declarations that trigger these warnings, this can
be tested by adding the following to notmuch.h:
__attribute__((warn_unused_result))
size_t fwrite(const void *ptr, size_t size, size_t nmemb, FILE *stream);
In my test case added g_object_unref(charset_filter) reduces memory
consumption over 90% when 'notmuch show --format=text "*"' is
executed (~11000 messages, RES ~330M -> ~25M).
g_object_unref() releases the memory of the InternetAddressList object
returned by internet_address_list_parse_string() -- when last (only)
reference is released, internet_address_list_finalize() will do cleanup.
previously we deleted the subcommand name from argv before passing to
the subcommand. In this version, the deletion is done in the actual
subcommands. Although this causes some duplication of code, it allows
us to be more flexible about how we parse command line arguments in
the subcommand, including possibly using off-the-shelf routines like
getopt_long that expect the name of the command in argv[0].
This was a minor oversite in checking of part type when outputing
content raw. This was causing gmime was to throw an exception to
stderr.
Unfortunately the gmime exception was not being caught by notmuch, or
the test suite. I'm not sure if notmuch should have done anything in
this case, but certainly the test suite should be capable of detecting
that something unexpected was output to stderr.
This new function takes a GMimeMessage as input, and outputs the
formatted headers. This allows for message/rfc822 parts to be
formatted on output in a similar way to full messages (see previous
patch that overhauls the multipart test for more info).
Before the change, notmuch show output had filename only for
parts with "Content-Disposition: attachment". But parts with
inline disposition may have filename as well.
The patch makes notmuch show always output filename if available,
independent of Content-Disposition. Both JSON and text output
formats are changed.
Also, the patch adds Content-id to text output format of notmuch
show.
The main goal of these changes is to have filenames on Emacs
buttons for inline attachments. In particular, this is very
helpful for inline patches.
Note: text format changes may require updates in clients that use
it. The changes are:
* text part header format changed from:
^Lpart{ ID: 2, Content-type: text/x-diff
to:
^Lpart{ ID: 2, Filename: cool-feature.patch, Content-type: text/x-diff
* attachment format changed from:
^Lattachment{ ID: 4, Content-type: application/octet-stream
Attachment: data.tar.bz2 (application/octet-stream)
Non-text part: application/octet-stream
^Lattachment}
to:
^Lattachment{ ID: 4, Filename: data.tar.bz2, Content-type: application/octet-stream
Non-text part: application/octet-stream
^Lattachment}
Our use of GMimeSession was unneeded boilerplate, and we weren't doing
anything with it. This simplifies and clarifies that assumption.
If we want to do anything fancier later, the examples in the gmime
source are a reasonable source to work from in defining a new
GMimeSession derivative.
Since GMimeSession is going away in GMime 2.6, though, i don't
recommend using it.
GMime has a nasty habit of taking ownership by default of any FILE*
handed to it va g_mime_stream_file_new. Specifically it will close the
FILE* when the stream is destroyed---even though GMime didn't open the
file itself.
To avoid this bad behavior, we have to carefully set_owner(FALSE)
after calling g_mime_stream_file_new. In the format_part_content_text
function, since commit d92146d3a6 we've
been calling g_mime_stream_file_new unconditionally, but only calling
g_mime_stream_file_set_owner(FALSE) conditionally.
This led to the FILE* being closed early when notmuch show output was
redirected to a file.
Fixing this fixes the test-suite cases that broke with the previous
commit, (which added redirected "notmuch show" calls to the test suite
to expose this bug).
Edited-by: Carl Worth <cworth@cworth.org> with a new commit message to
explain the bug and fix.
Now that this function registers an internal error if called with a
non-text part, we can remove the conditions for multipart and
message-part content types.
This fixes the recently-added test case (in test/multipart)
demonstrating corruption of binary parts that happen to contain CRLF
pairs.
We restore the original code from show_one_part_content to
format_part_content_raw. Then, for good measure, we rename
show_part_content to the more descriptive show_text_part_content and
add an internal error if it is ever called with a non-text part.
This adds support for decrypting PGP/MIME-encrypted parts to
notmuch-show and notmuch-reply. The --decrypt option implies
--verify. Once decryption (and possibly signature verification) is
done, a new part_encstatus formatter is emitted, the part_sigstatus
formatter is emitted, and the entire multipart/encrypted part is
replaced by the contents of the encrypted part.
At the moment only a json part_encstatus formatting function is
available, even though decryption is done for all formats. Emacs
support to follow.
This is primarily for notmuch-show, although the functionality is
added to show-message. Once signatures are processed a new
part_sigstatus formatter is emitted, and the entire multipart/signed
part is replaced with the contents of the signed part.
At the moment only a json part_sigstatus formatting function is
available. Emacs support to follow.
The original work for this patch was done by
Daniel Kahn Gillmor <dkg@fifthhorseman.net>
whose help with this functionality I greatly appreciate.
Future improvements (eg. crypto support) will require adding new part
header. By breaking up the output of part headers from the output of
part content, we can easily out new part headers with new formatting
functions.
Since message/rfc822 parts are really just a special kind of
multipart, we here normalize the handling of the two. This will
provide access to sub-parts of message/rfc822 parts, which was
previously unavailable.
Outputting of single MIME parts is moved to an option of notmuch show,
instead of being handled in it's own sub-command. The recent rework
of multipart mime allowed for this change but consolidating part
handling into a single recursive function (show_message_part) that
includes formatting. This allows for far simpler handling single
output of a single part, including formatting.
We rename here in order to make do_show_single into a generic function
for handling output of just a single message, or which format=raw is a
special case. The raw case is handled by setting a new parameter,
params.raw, which is used to tell do_show_single to output a single
message as a raw file.
This is mostly in preparation for much improved part handling to
follow imminently.
A new field "part_sep" is added to the notmuch_show_format structure,
to be used for part separation. This is cleaner than the "first"
argument that was being passed around to the part arguments, and
allows the function that handles overall part output formatting
(show_message_part) to directly handle when outputting the separator.
This simplifies the passing of arguments to the show functions. This
will be very useful as we accumulate more parameters that will need to
be passed. Currently only the entire_thread parameter is passed this
way.
Various show_message* functions require formatting functions, which
were previously being passed individually as arguments. Since we will
need to be needing to passing in more formatting function in the
future (ie. for crypto support), we here modify things so that we just
pass in the entire format structure. This will make things much
simpler down the line as we need to pass in new format functions.
We move the show_format structure into notmuch-client.c as
notmuch_show_format. This also affects notmuch-reply.c, so we create
a mostly-empty format_reply to pass the reply_part function to
show_message_body.
It's simply one more property of a MIME part that might be useful, and
json makes it so easy to add additional properties.
Signed-off-by: Jameson Rollins <jrollins@finestructure.net>
Previously, notmuch show flattened all output, losing information
about the nesting of the MIME hierarchy. Now, the output is properly
nested, (both in the --format=text and --format=json output), so that
clients can analyze the original MIME structure.
Internally, this required splitting the final closing delimiter out of
the various show_part functions and putting it into a new
show_part_end function instead. Also, the show_part function now
accepts a new "first" argument that is set not only for the first MIME
part of a message, but also for each first MIME part within a series
of multipart parts. This "first" argument controls the omission of a
preceding comma when printing a part (for json).
Many thanks to David Edmondson <dme@dme.org> for originally
identifying the lack of nesting in the json output and submitting an
early implementation of this feature. Thanks as well to Jameson Graef
Rollins <jrollins@finestructure.net> for carefully shepherding David's
patches through a remarkably long review process, patiently explaining
them, and providing a cleaned up series that led to this final
implementation. Jameson also provided the new emacs code here.
Previously, the outer multipart part of any multipart/mixed,
multipart/signed, etc. MIME message was silently omitted from the
"notmuch show" output. This prevented any client from correctly
determining to which parts a signature applies, for example.
Now, we actually emit these parts as their own parts. The output is
still flattened---the contained parts are not yet included "within"
the multipart part---so it's still not possible to determine to which
parts a signature applies, but this is one step along the path.
The test suite is updated to reflect this change, (though we'll
eventually want to fix the emacs interface to not display buttons for
the multipart enclosure parts as there's nothing useful for the user
to actually do with them).
This is part of an effort to avoid proliferation of excessive
top-level notmuch commands. Also, "raw" better captures the
functionality here, (as opposed to "cat" which is a fairly oblique
reference to a bad Unix abbreviation whose metaphor doesn't work here
since "notmuch cat" operates only on a single message and hence cannot
"con'cat'enate" anything).
This command outputs a raw message matched by search term to the
standard output. It allows MUAs to access the messages for piping,
attachment manipulation, etc. by running notmuch cat rather then
directly access the file. This will simplify the MUAs when they need
to operate on a remote database.
Edited-by: Carl Worth <cworth@cworth.org>: Remove trailing whitespace,
add missing "test_done" to new test script to avoid "Unexpected exit"
error.
We don't love the mbox format, but it's still sometimes the most
practical way to share a collection of messages as a single file.
Here we implement the "mboxrd" variant of the mbox file format. This
variant applies reversible escaping by prefixing a '>' character to
all lines in the email messages matching the regular expression:
"^>*From "
This allows the escaping to be reliably removed. A reader should remove
a '>' from any line matching the regular expression:
"^>>*From "
More details on the mboxrd formats (and others as well) can be found
here:
http://homepage.ntlworld.com/jonathan.deboynepollard/FGA/mail-mbox-formats.html
Micah Anderson reported an issue where a message failed to display in
the emacs interface, (it instead gave an error, "json-read-string: Bad
string format").
Micah tracked this down to the json output from "notmuch show" being
interrupted by a GMime error message:
gmime-CRITICAL **: g_mime_stream_filter_add: assertion
`GMIME_IS_FILTER (filter)
I tracked this down further to notmuch passing a NULL value to
g_mime_stream_filter_add. And this was due to calling
g_mime_filter_charset_new with a value of "unknown-8bit".
So we add a test message withe a Conten-Type of "text/plain;
charset=unknown-8bit" from Micah's message. Then we fix "notmuch show"
to test for NULL before calling g_mime_stream_filter_add. Bug fixed.
A new 'part' subcommand allows the user to extract a single part from
a MIME message. Usage:
notmuch part --part=<n> <search terms>
The search terms must match only a single message
(e.g. id:foo@bar.com). The part number specified refers to the part
identifiers output by `notmuch show'. The content of the part is
written the stdout with no formatting or identification marks. It is
not JSON formatted.
