The recent change to use json for notmuch-search.el introduced a bug
in the code for keeping position on refresh. The problem is a
comparison between (plist-get result :thread) and a thread-id returned
by notmuch-search-find-thread-id: the latter is prefixed with
"thread:"
We fix this by adding an option to notmuch-search-find-thread-id to
return the bare thread-id. It appears that notmuch-search-refresh-view
is the only caller of notmuch-search that supplies a thread-id so this
change should be safe (but could theoretically break users .emacs
functions).
The string function in a sprinter may be called with a NULL string
pointer (eg if a header is absent). This causes a segfault. We fix
this by checking for a null pointer in the string functions and update
the sprinter documentation.
At the moment some output when format=text is done directly rather than
via an sprinter: in that case a null pointer is passed to printf or
similar and a "(null)" appears in the output. That behaviour is not
changed in this patch.
Previously, the Emacs byte compiler produced the warning
the function `remove-if-not' might not be defined at runtime.
because we only required cl at compile-time (not runtime). This fixes
this warning by requiring cl at runtime, ensuring that the definition
of remove-if-not is available.
The syntax --output=filename is a smaller change than deleting the
output argument completely, and conceivably useful e.g. when running
notmuch under a debugger.
This should allow users to install notmuch-emacs with only emacs24
installed on their system. For good measure, allow building with
emacs24 as a 4th choice.
Unlike the previous patches, this function is used for all formats.
However, for formats other than the JSON format, the sprinter methods
used by show_message are all no-ops, so this code continues to
function correctly for all of the formats.
Converting show_message eliminates show_null_message in the process,
since this maps directly to an sprinter method.
There are several levels of function calls between where we create the
sprinter and the call to the part formatter in show_message. This
feeds the sprinter through all of them and into the part formatters.
This associates an sprinter constructor with each show format and uses
this to construct the appropriate sprinter. Currently nothing is done
with this sprinter, but the following patches will weave it through
the layers of notmuch show.
This method allows callers to output strings with specific lengths.
It's useful both for strings with embedded NULs (which JSON can
represent, though parser support is apparently spotty), and
non-terminated strings.
Format canonicalization of JSON output is no longer necessary, so
remove it. Value canonicalization (e.g., normalizing thread IDs) is
still necessary, so all of the sanitization functions remain.
Previously, we used a variety of ad-hoc canonicalizations for JSON
output in the test suite, but were ultimately very sensitive to JSON
irrelevancies such as whitespace. This introduces a new test
comparison function, test_expect_equal_json, that first pretty-prints
*both* the actual and expected JSON and the compares the result.
The current implementation of this simply uses Python's json.tool to
perform pretty-printing (with a fallback to the identity function if
parsing fails). However, since the interface it introduces is
semantically high-level, we could swap in other mechanisms in the
future, such as another pretty-printer or something that does not
re-order object keys (if we decide that we care about that).
In general, this patch does not remove the existing ad-hoc
canonicalization because it does no harm. We do have to remove the
newline-after-comma rule from notmuch_json_show_sanitize and
filter_show_json because it results in invalid JSON that cannot be
pretty-printed.
Most of this patch simply replaces test_expect_equal and
test_expect_equal_file with test_expect_equal_json. It changes the
expected JSON in a few places where sanitizers had placed newlines
after commas inside strings.
The pipe message function (when used with a prefix) uses a search of
the form "id:<id1> or id:<id2>" etc. Since the user says precisely
which messages are wanted by opening them it should not use excludes.
Add a --remove-dups flag which removes duplicate files from search and
thread results. Uses fdupes if installed. Otherwise it runs a size and
Digest::SHA scan on each file to detect duplicates.
Signed-off-by: Stefano Zacchiroli <zack@upsilon.cc>
In commit 5d0883e the function notmuch-search-next-thread was changed.
In particular it only goes to the next message if there is a next
message. This breaks notmuch-show-archive-thread-then-next. Fix this
by going to the "next" message whenever we are on a current message.
Create a variable DISTCLEAN which contains a list of things to
clean in the distclean target (in addition to running the clean
target).
The deleted comment seems to be false these days, since we do
create files during configuration.
Use "rm -rf" here as well in case we want to add directories to
DISTCLEAN.
These extra directories cause problems for building on Debian
twice in a row.
In order to remove directories, we need to us "rm -rf" instead of
"rm -f". So now we should be extra careful what we add to the
variable CLEAN.
'configure' script uses parameter substring extensively. It is Posix shell
feature. Original Bourne shell does not have such features. Some systems
still ships such shells as /bin/sh (for compatibility reasons -- shell
scripts written on those platforms are expected to work on 1990's systems).
Just testing whether parameter substring processing works will make the
shell exit due to syntax error if it is not compatible. Therefore the test
is executed in a subshell -- subshell exits with nonzero value when the
operation in question fails.
As 'if ! ...' does not work in Bourne shell, Short-circuiting construct
'||' is used to print information message and exit when expected.
Previously in notmuch show --format=json implied --entire-thread. This
is still the default but it is now possible to disable this. Update
the manpage to reflect this.
Previously body: was a compulsory field in a message. The new
--body=false option causes notmuch show to omit this field so update
schemata to reflect this.
This option allows the caller to suppress the output of the bodies of
the messages. Currently this is only implemented for format=json.
This is used by notmuch-pick.el (although not needed) because it gives
a speed-up of at least a factor of a two (and in some cases a speed up
of more than a factor of 8); moreover it reduces the memory usage in
emacs hugely.
This patch switches from the current ad-hoc printer to the structured
formatters in sprinter.h, sprinter-text.c and sprinter-json.c.
The JSON tests are changed slightly in order to make them PASS for the
new structured output formatter.
The text tests pass without adaptation.
Using the new structured printer support in sprinter.h, implement
sprinter_json_create, which returns a new JSON structured output
formatter. The formatter prints output similar to the existing JSON, but
with differences in whitespace (mostly newlines, --output=summary prints
the entire message summary on one line, not split across multiple lines).
Also implement a "structured" formatter for plain text that prints
prefixed strings, to be used with notmuch-search.c plain text output.
This patch adds a new struct type sprinter_t, which is used for
structured formatting, e.g. JSON or S-Expressions. The structure printer
is heavily based on code from Austin Clements
(id:87d34hsdx8.fsf@awakening.csail.mit.edu).
It includes the following functions:
/* Start a new map/dictionary structure. This should be followed by
* a sequence of alternating calls to map_key and one of the
* value-printing functions until the map is ended by end.
*/
void (*begin_map) (struct sprinter *);
/* Start a new list/array structure.
*/
void (*begin_list) (struct sprinter *);
/* End the last opened list or map structure.
*/
void (*end) (struct sprinter *);
/* Print one string/integer/boolean/null element (possibly inside a
* list or map, followed or preceded by separators).
* For string, the char * must be UTF-8 encoded.
*/
void (*string) (struct sprinter *, const char *);
void (*integer) (struct sprinter *, int);
void (*boolean) (struct sprinter *, notmuch_bool_t);
void (*null) (struct sprinter *);
/* Print the key of a map's key/value pair. The char * must be UTF-8
* encoded.
*/
void (*map_key) (struct sprinter *, const char *);
/* Insert a separator (usually extra whitespace) for improved
* readability without affecting the abstract syntax of the
* structure being printed.
* For JSON, this could simply be a line break.
*/
void (*separator) (struct sprinter *);
/* Set the current string prefix. This only affects the text
* printer, which will print this string, followed by a colon,
* before any string. For other printers, this does nothing.
*/
void (*set_prefix) (struct sprinter *, const char *);
To support the plain text format properly, the following additional
function must also be implemented:
/* Set the current string prefix. This only affects the text
* printer, which will print this string, followed by a colon,
* before any string. For other printers, this does nothing.
*/
void (*set_prefix) (struct sprinter *, const char *);
The structure also contains a flag that should be set to FALSE in all
custom printers and to TRUE in the plain text formatter.
/* True if this is the special-cased plain text printer.
*/
notmuch_bool_t is_text_printer;
The printer can (and should) use internal state to insert delimiters
and syntax at the correct places.
Example:
format->begin_map(format);
format->map_key(format, "foo");
format->begin_list(format);
format->integer(format, 1);
format->integer(format, 2);
format->integer(format, 3);
format->end(format);
format->map_key(format, "bar");
format->begin_map(format);
format->map_key(format, "baaz");
format->string(format, "hello world");
format->end(format);
format->end(format);
would output JSON as follows:
{"foo": [1, 2, 3], "bar": { "baaz": "hello world"}}
At this point, the only remaining functions that don't support
multi-line search result formats are the thread navigation functions.
This patch fixes that by rewriting them in terms of
notmuch-search-result-{beginning,end}.
This changes the behavior of notmuch-search-previous-thread slightly
so that if point isn't at the beginning of a result, it first moves
point to the beginning of the result.
Previously we ignored any notmuch-search-result-format customizations
for tag formatting because we needed to be able to parse back in the
result line and update the tags in place. We no longer do either of
these things, so we can allow customization of this format.
(Coincidentally, previously we still allowed too much customization of
the tags format, since moving it earlier on the line or removing it
from the line would interfere with the tagging mechanism. There is
now no problem with doing such things.)
Since the result object contains everything that the other text
properties recorded, we can remove the other text properties and
simply look in the plist of the appropriate result object.
This simplifies the traversal of regions of results and eliminates the
need for save-excursions (which tend to get in the way of maintaining
point when we make changes to the buffer). It also fixes some strange
corner cases in the old line-based code where results that bordered
the region but were not included in it could be affected by region
commands. Coincidentally, this also essentially enables multi-line
search result formats; the only remaining non-multi-line-capable
functions are notmuch-search-{next,previous}-thread, which are only
used for interactive navigation.
Now that we keep the full thread result object, we can refresh a
result after any changes by simply deleting and reconstructing the
result line from scratch.
A convenient side-effect of this wholesale replacement is that search
now re-applies notmuch-search-line-faces when tags change.
Previously, tag-based search result highlighting was done by creating
an overlay over each search result. However, overlays have annoying
front- and rear-advancement semantics that make it difficult to
manipulate text at their boundaries, which the next patch will do.
They also have performance problems (creating an overlay is linear in
the number of overlays between point and the new overlay, making
highlighting a search buffer quadratic in the number of results).
Text properties have neither problem. However, text properties make
it more difficult to apply multiple faces since, unlike with overlays,
a given character can only have a single 'face text property. Hence,
we introduce a utility function that combines faces into any existing
'face text properties.
Using this utility function, it's straightforward to apply all of the
appropriate tag faces in notmuch-search-color-line.
notmuch-message-mark-replied used "apply" to change message tags
according to notmuch-message-replied-tags after sending a reply. This
works if the latter is a single-element list. But with the recently
changed format of tag changes, it breaks for multiple-element lists.
Use "funcall" to properly pass the list of tag changes as a single
argument.
The JSON format eliminates the complex escaping issues that have
plagued the text search format. This uses the incremental JSON parser
so that, like the text parser, it can output search results
incrementally.
This slows down the parser by about ~4X, but puts us in a good
position to optimize either by improving the JSON parser (evidence
suggests this can reduce the overhead to ~40% over the text format) or
by switching to S-expressions (evidence suggests this will more than
double performance over the text parser). [1]
This also fixes the incremental search parsing test.
This has one minor side-effect on search result formatting.
Previously, the date field was always padded to a fixed width of 12
characters because of how the text parser's regexp was written. The
JSON format doesn't do this. We could pad it out in Emacs before
formatting it, but, since all of the other fields are variable width,
we instead fix notmuch-search-result-format to take the variable-width
field and pad it out. For users who have customized this variable,
we'll mention in the NEWS how to fix this slight format change.
[1] id:"20110720205007.GB21316@mit.edu"
