Presently, the code which finds the parent of a message as it is being
added to the database assumes that the first Message-ID-like substring
of the In-Reply-To header is the parent Message ID. Some mail clients,
however, put stuff other than the Message-ID of the parent in the
In-Reply-To header, such as the email address of the sender of the
parent. This can fool notmuch.
The updated algorithm prefers the last Message ID in the References
header. The References header lists messages oldest-first, so the last
Message ID is the parent (RFC2822, p. 24). The References header is
also less likely to be in a non-standard
syntax (http://cr.yp.to/immhf/thread.html,
http://www.jwz.org/doc/threading.html). In case the References header
is not to be found, fall back to the old behavior.
V2 of this patch, incorporating feedback from Jani and (indirectly)
Austin.
The use of realpath(3) in
commit 58ed67992d
Author: Jani Nikula <jani@nikula.org>
Date: Sun Apr 7 20:15:03 2013 +0300
cli: config: do not overwrite symlinks when saving config file
broke config file save when the file does not exist, which results in
'notmuch setup' always failing to create a new config file.
Fix by checking ENOENT from realpath(3).
Use realpath to canonicalize the config path before writing.
Previously 'notmuch setup' and 'notmuch config set' overwrote the
config file even if it was a symbolic link.
We now have a notmuch_config_is_new() function to query whether a
config was created or not. Change the notmuch_config_open() is_new
parameter into boolean create_new to determine whether the function
should create a new config if one doesn't exist. This reduces the
complexity of the API.
When execution of tests is interrupted by signal coming outside of the
test system itself, output just one line "interrupted by signal <num>"
message to standard output. This distinguishes the case from internal
exit and reduces noise.
Set the variable '$test_subtest_name' in all functions which starts
a new test and use that variable in all functions that output
test results.
Additionally output the latest '$test_subtest_name' in case of
abnormal exit, to avoid confusion.
Instead of checking immediately for the watched process, delay a
minute, or in the case that process-attributes returns nil, for two
minutes. This is intended to cope with the case that
process-attributes is unimplimented, and returns always returns nil.
In this case, the watchdog check is the same as the two minute limit
imposed by timeout.
The TERM environment variable is set to 'dumb' when running tests, but
the original value of it is stored for echoing colors and running emacs
(somewhat interactively) in detached session. Emacs requires some
terminal control sequences to be available for interactive operation.
In case original TERM is (also) 'dumb' (or unset/empty) emacs cannot
run interactively. To fix this problem dtach (and emacs as it's child
process) is run with TERM=vt100 in case original TERM was unset, empty
or 'dumb'. This way there is a chance to run emacs tests with different
user terminals and potentially find problems there.
This test also serves as documentation of the quoting
requirements. The comment lines are so that it exactly matches the man
page. Nothing more embarrassing than having an example in the man page
fail.
The (now fixed) bug that this test revealed is that unquoted
message-ids with whitespace or other control characters in them are
split into several tokens by the Xapian query parser.
This is based on the similar test for notmuch restore, but the parser
in batch tagging mode is less tolerant of a few cases, in particular
those tested by illegal_tag.
We recently switched to popping up a buffer to report CLI errors, but
this was too intrusive, especially for transient errors and especially
since we made fewer things ignore errors. This patch changes this to
display a basic error message in the minibuffer (using Emacs' usual
error handling path) and, if there are additional details, to log
these to a separate error buffer and reference the error buffer from
the minibuffer message. This is more in line with how Emacs typically
handles errors, but makes the details available to the user without
flooding them with the details.
Given this split, we pare down the basic message and make it more
user-friendly, and also make the verbose message even more detailed
(and more debugging-oriented).
This switches the new batch-tag format away from using a home-grown
hex-encoding scheme for message IDs in the dump to simply using Xapian
queries with Xapian quoting syntax.
This has a variety of advantages beyond presenting a cleaner and more
consistent interface. Foremost is that it will dramatically simplify
the quoting for batch tagging, which shares the same input format.
While the hex-encoding is no better or worse for the simple ID queries
used by dump/restore, it becomes onerous for general-purpose queries
used in batch tagging. It also better handles strange cases like
"id:foo and bar", since this is no longer syntactically valid.
When we switch to using regular Xapian queries in the dump format, \n
will cause problems, so we disallow it. Specially, while Xapian can
quote and parse queries containing \n without difficultly, quoted
queries containing \n still span multiple lines, which breaks the
line-orientedness of the dump format. Strictly speaking, we could
still round-trip these, but it would significantly complicate restore
as well as scripts that deal with tag dumps. This complexity would
come at absolutely no benefit: because of the RFC 2822 unfolding
rules, no amount of standards negligence can produce a message with a
message ID containing a line break (not even Outlook can do it!).
Hence, we simply disallow it.
This patch corrects several undesirable behaviours:
1) Empty files were not detected, leading to buffer read overrun.
2) An initial blank line cause restore to silently abort
3) Initial comment line caused format detection to fail
notmuch_json_show_sanitize replaced "filename" field values even in part
structures, where the value is predictable. Make it only normalize the
filename value if it is an absolute path (begins with slash), which is
true of the Maildir filenames that were intended to be normalized away.
This slightly changes the output of an existing test since we now
report non-zero exits with a pop-up buffer instead of at the end of
the search results.
* test/emacs:
- Rename subtests "{Add,Remove} tag from notmuch-show view" to
"notmuch-show: {add,remove} single tag {to,from} single message"
to be consistent with the following tests.
- New subtest "notmuch-show: add multiple tags to single message":
`notmuch-show-add-tag' ("+") can add multiple tags to a message.
- New subtest "notmuch-show: remove multiple tags from single message":
`notmuch-show-remove-tag' ("-") can remove multiple tags from a message.
We want to test both that error/warning messages are generated when
they should be, and not generated when they should not be. This varies
between restore and batch tagging.
These one need the completed functionality in notmuch-restore. Fairly
exotic tags are tested, but no weird message id's.
We test each possible input to autodetection, both explicit (with
--format=auto) and implicit (without --format).
The quoting for ${SEARCH} is broken when it's supposed to be '*', and
it seems tricky to get it right. Just drop the variable and use '*'
directly. Before this, none of the messages ever matched, and the test
was comparing zeros.
test_expect_equal_json uses json.tool from the system Python. While
Python 2 wasn't picky about the encoding of stdin, Python 3 decodes
stdin strictly according to the environment. Since we set LC_ALL=C
for the tests, Python 3's json.tool was assuming stdin would be in
ASCII and aborting when it couldn't decode the UTF-8 characters from
some of the JSON tests. This patch sets the PYTHONIOENCODING
environment variable to utf-8 when invoking json.tool to override
Python's default encoding choice.
Without this change, GCC complains as follows:
gcc test/random-corpus.o test/database-test.o notmuch-config.o command-line-arguments.o lib/libnotmuch.a util/libutil.a parse-time-string/libparse-time-string.a -o test/random-corpus -lgmime-2.6 -lgio-2.0 -lgobject-2.0 -lglib-2.0 -Wl,-rpath,/usr/lib -ltalloc -lxapian
/usr/bin/ld: lib/libnotmuch.a(database.o): undefined reference to symbol '_ZNSs4_Rep10_M_destroyERKSaIcE@@GLIBCXX_3.4'
/usr/bin/ld: note: '_ZNSs4_Rep10_M_destroyERKSaIcE@@GLIBCXX_3.4' is defined in DSO /usr/lib/libstdc++.so.6 so try adding it to the linker command line
/usr/lib/libstdc++.so.6: could not read symbols: Invalid operation
collect2: error: ld returned 1 exit status
make: *** [test/random-corpus] Error 1
We demonstrate the current notmuch restore parser being confused by
message-id's and tags containing non alpha numeric characters
(particularly space and parentheses are problematic because they are
not escaped by notmuch dump).
We save the files as hex escaped on disk so that terminal emulators
will not get confused if the test fails (as we mostly expect it to do).
Initial use case is testing dump and restore, so we only have
message-ids and tags.
The message ID's are nothing like RFC compliant, but it doesn't seem
any harder to roundtrip random UTF-8 strings than RFC-compliant ones.
Tags are UTF-8, even though notmuch is in principle more generous than
that.
updated for id:m2wr04ocro.fsf@guru.guru-group.fi
- talk about Unicode value rather some specific encoding
- call talloc_realloc less times
Initially, provide a way to create "stub" messages in the notmuch
database without corresponding files. This is essentially cut and
paste from lib/database.cc. This is a seperate file since we don't
want to export these symbols from libnotmuch or bloat the library with
non-exported code.
