The big change here is the support for maildir-flag synchronization.
But there are a number of other thigns as well---library support for
multiple filenames, new ruby bindings, improvements to the vim
interface, and a few tweaks to the emacs interface.
This add a "stash-map" for search-mode, just like in show-mode, and
adds one function, bound to "i" to stash the thread-id of the current
selected thread.
Couldn't think of the correct way to stash other thread info, so I
didn't add any other stash functions for now.
Here we move the notmuch-show/notmuch-show-do-stash function to
notmuch-lib/notmuch-common-do-stash. Nothing in this function is
notmuch-show mode specific, so this move will make it cleaner to be
used by other modes (such as notmuch-search).
Re-work the declaration and definition of `notmuch-fcc-dirs'. The
variable now allows three types of values:
- nil: no Fcc header is added,
- a string: the value of `notmuch-fcc-dirs' is the name of the
folder to use,
- a list: the folder is chosen based on the From address of the
current message using a list of regular expressions and
corresponding folders:
((\"Sebastian@SSpaeth.de\" . \"privat\")
(\"spaetz@sspaeth.de\" . \"OUTBOX.OSS\")
(\".*\" . \"defaultinbox\"))
If none of the regular expressions match the From address, no
Fcc header will be added.
Remove 're: ' or 'Re: ' from anywhere within a subject line rather
than just at the beginning. This is to accommodate threads where a
mailing list sometimes inserts a subject prefix.
For example, if a thread has the subjects:
[Orgmode] org-indent, org-inlinetask: patches on github
Re: [Orgmode] org-indent, org-inlinetask: patches on github
[Orgmode] Re: org-indent, org-inlinetask: patches on github
the last of these would not have been considered the same and would
therefore have been shown.
This is to prevent notmuch from destroying any information the user
has encoded as flags in the maildir filename. Tests are also added to
the test suite to verify the documented behavior.
Some people use notmuch with non-maildir files, (for example, email
messages in MH format, or else cool things like using sluk[*] to suck
down feeds into a format that notmuch can index).
To better support uses like that, don't do any renaming for files that
are not in a directory named either "new" or "cur".
[*] https://github.com/krl/sluk/
I had originally hoped for better semantics, such as doing nothing in
non-maildir directories, and preserving unknown maildir flags that
happen to be present.
We could still do those things, of course, but for now, remove them
from the documentation since the implementation does not do these
things yet.
The intent of "notmuch setup" is that it adds new, documented sections
to configuration files that were created before such sections were
defined. But to make this work, we have to explicitly set an option
in the maildir group if it didn't exist previously.
The FCC code saves a message in maildir format, and sets the S flag by
default, so now, automatically, FCC messages will not show up as
"unread", (which seems natural enough).
This synchronization is one of those features that should just happen
automatically. We allow for customization in case someone *really*
wants to turn it off, but we don't need to prompt for this
interactively.
People with special needs can find the configuration file on their
own.
It is totally legitimate for a non-maildir directory to be named "new"
(and not have a directory next to it named "cur"). To support this
case at least, be silent about any rename failure.
There's nothing in the current API documentation that would suggest
the behavior being tested here. Attempt to implement this could have
some nasty side effects, (such as notmuch_message_maildir_flags_to_tags
implicitly calling notmuch_message_tags_to_maildir_flags and maybe
even opening up some bad looping possibilities).
Much better to stick with what we have documented, which we believe will
actually be useful, (and easy enough to comprehend).
If a filename has no maildir info at all, (that is, it does not
contain the sequence ":2,"), we consider this distinct from a filename
with an empty maildir info, (the ":2," separator is present, but no
flags characters follow).
Specifically, we regard a missing info field as providing no
information, so tags will remain unchanged. On the other hand, an info
field that is present but has no flags set will cause various tags to
be cleared, (or in the case of "unread", added).
This fixes the "remove info" case of the maildir-sync tests in the
test suite.
When a file in the mailstore is renamed, this appears to "notmuch new"
as both an added file and a removed file (for the same message). We
want the synchronization of the maildir_flags to reflect the final
state, (after the rename is complete). Therefore, it's incorrect to
perform the synchronization immediately after adding a new
file. Instead we queue up these synchronizations (by message ID[*])
and perform them after the removals are complete.
With this change, the "dump/restore" case of the maildir-sync tests,
as well as the recent "remove 'S'" case both now pass where they were
failing before.
Interestingly, the "remove info" test was passing before, but now
fails. This is actually due to a separate bug, (and the bug just fixed
was masking it, by preventing the test from performing as desired).
[*] It's important to queue by message ID---queueing actual message
objects does not work since the message objects will retain stale data
such as the old filenames.
Previously the documentation of notmuch_message_maildir_flags_to_tags
suggested that the presence of a flag would cause tags to be added,
(or in the case of "unread", removed). But the case of absent maildir
flags was not explicitly described.
What we actually want, is that for supported flags, the absence of the
flag in all messages causes the corresponding tag to be removed,
(or in the case of "unread", added). So document that explicitly.
This is the case recently added to the test suite as a failing test,
(so we'll need to do bug fixing before the documentation is honest
here).
We have tests to ensure that when the notmuch library renames a file
that that rename takes place immediately in the database, (without
requiring something like "notmuch new" to notice the change).
This was working when the code was first added, but recently broke in
the reworking of the maildir-synchronization interface since the
tags_to_maildir_flags function can no longer assume that it is being
called as part of _notmuch_message_sync.
Fortunately, the fix is as simple as adding an explicit call to
_notmuch_message_sync.
As documented, this function now iterates over all filenames for the
message, computing a logical OR of the flags set on the filenames,
then uses the final result to set tags on the message.
This change fixes 3 of the 10 maildir-sync tests that have been
failing since being added.
This augments the existing notmuch_message_get_filename by allowing
the caller access to all filenames in the case of multiple files for a
single message.
To support this, we split the iterator (notmuch_filenames_t) away from
the list storage (notmuch_filename_list_t) where previously these were
a single object (notmuch_filenames_t). Then, whenever the user asks
for a file or filename, the message object lazily creates a complete
notmuch_filename_list_t and then:
For notmuch_message_get_filename, returns the first filename
in the list.
For notmuch_message_get_filenames, creates and returns a new
iterator for the filename list.
The new implementation is simply a talloc-based list of strings. The
former support (a list of database terms with a common prefix) is
implemented by simply pre-iterating over the terms and populating the
list. This should provide no performance disadvantage as callers of
thigns like notmuch_directory_get_child_files are very likely to
always iterate over all filenames anyway.
This new implementation of notmuch_filenames_t is in preparation for
adding API to query all of the filenames for a single message.
This rather ugly hack was recently obviated by the removal of the
notmuch_database_set_maildir_sync function. Now, clients must make
explicit calls to do any syncrhonization between maildir flags and
tags. So the library no longer needs to worry about doing inconsistent
synchronization while a message is only partially added.
Instead of having an API for setting a library-wide flag for
synchronization (notmuch_database_set_maildir_sync) we instead
implement maildir synchronization with two new library functions:
notmuch_message_maildir_flags_to_tags
and notmuch_message_tags_to_maildir_flags
These functions are nicely documented here, (though the implementation
does not quite match the documentation yet---as plainly evidenced by
the current results of the test suite).
Since the name of the configuration parameter here is:
maildir.synchronize_flags
the convention is that the functions to get and set this parameter
should match it in name. Hence:
notmuch_config_get_maildir_synchronize_flags
etc. (as opposed to notmuch_config_get_maildir_sync).
These needed to be changed to be brought up to the current state of
the maildir-sync tests. This includes style changes, but also the
elimination of any assumption about pre-existing message filenames,
(such as msg-003) which actually don't exist anymore.
Also, the known broken tests are changed to emit FAIL rather than
BROKEN simply to make them easier to fix, (so that they print the
current problems rather than hiding them).
Finally, an additional test is added to ensure that when a duplicate
file is added without flags, it doesn't invalidate flags from other
duplicates, (instead the flags are effectively merged).
Add maildir synchronization tests for multiple messages with the same
message-id. As this is not yet implemented in notmuch, some of these
teste are marked as BROKEN.
I use $(< ) operator to avoid fiddling with stripped trailing newlines
from test results which happens when output+=$(command) is used.
Tags in a notmuch database affect all messages with the identical
message-ID. But maildir tags affect individual files. And since
multiple files can contain the identical message-ID, there is not a
one-to-one correspondence between messages affected by tags and flags.
This is particularly dangerous with the 'T' (== "trashed") maildir
flag and the corresponding "deleted" tag in the notmuch
database. Since these flags/tags are often used to trigger
irreversible deletion operations, the lack of one-to-one
correspondence can be potentially dangerous.
For example, consider the following sequence:
1. A third-party application is used to identify duplicate messages
in the mail store, and mark all-but-one of each duplicate with
the 'T' flag for subsequent deletion.
2. A "notmuch new" operation reads that 'T' flag, adding the
"deleted" flag to the corresponding messages within the notmuch
database.
3. A subsequent notmuch operation, (such as a "notmuch dump; notmuch
restore" cycle) synchronized the "deleted" tag back to the mail
store, applying the 'T' flag to all(!) filenames with duplicate
message IDs.
4. A third-party application reads the 'T' flags and irreversibly
deletes all mail messages which had any duplicates(!).
In order to avoid this scenario, we simply refuse to synchronize the
'T' flag with the "deleted" tag. Instead, applications can set 'T' and
act on it to delete files, or can set "deleted" and act on it to
delete files. But in either case the semantics are clear and there is
never dangerous propagation through the one-to-many mapping of notmuch
message objects to files.
This change reworks these tests in several ways:
1. Bring tests into "new" test style preferring test_expect_equal over
test_expect_success in almost all cases.
2. Don't emit test results for intermediate items not actually being
tested, (things like "no new messages", "search for message",
etc.). Those things are already covered by existing tests such as
"basic" or "search" and only serve to obscure what's actually being
tested.
3. Change sense of the test showing failure to rename a file from
"new" to "cur" when "cur" doesn't exist.
In this case, notmuch should detect that this is not a maildir and
should not attempt to do any renaming of the file.
4. Extend dump/restore test to also exercise addition of tag, not just
removal.
Both items #3 and #4 above show shortcomings in the current
implementation. These are currently resulting in test results of FAIL
and indicate bugs that need to be fixed.
This adds group [maildir] and key 'synchronize_flags' to the
configuration file. Its value enables (true) or diables (false) the
synchronization between notmuch tags and maildir flags. By default,
the synchronization is disabled.
This patch allows bi-directional synchronization between maildir
flags and certain tags. The flag-to-tag mapping is defined by flag2tag
array.
The synchronization works this way:
1) Whenever notmuch new is executed, the following happens:
o New messages are tagged with configured new_tags.
o For new or renamed messages with maildir info present in the file
name, the tags defined in flag2tag are either added or removed
depending on the flags from the file name.
2) Whenever notmuch tag (or notmuch restore) is executed, a new set of
flags based on the tags is constructed for every message and a new
file name is prepared based on the old file name but with the new
flags. If the flags differs and the old message was in 'new'
directory then this is replaced with 'cur' in the new file name. If
the new and old file names differ, the file is renamed and notmuch
database is updated accordingly.
The rename happens before the database is updated. In case of crash
between rename and database update, the next run of notmuch new
brings the database in sync with the mail store again.
We have test names like maildir-sync now, so it's cleaner if the
temporary files created are named things like maildir-sync-10.out
rather than maildir-10.out. Presumably the extra stripping here came
from naming conventions in git's test suite.
Possilby used by more systems, and besides the code wasn't really
working properly anyway.
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
