Unfortunately old versions of GCC and clang do not provide byte order
macros, so we re-invent them.
If UTIL_BYTE_ORDER is not defined or defined to 0, we fall back to
macros supported by recent versions of GCC and clang
Previously PLATFORM_BYTE_ORDER and IS_LITTLE_ENDIAN were not defined,
so the little endian code was always compiled in.
This will have the effect that the "SHA1s" on big endian architectures
will change (i.e. become actual sha1s). So someone re-indexing their
database could conceivable lose tags on messages without a message-id
header.
In case previous notmuch compact has been interrupted there is old
work-in-progress database compact directory partially filled. Remove
it just before starting to fill the directory with new files.
It is less error prone and window of failure opportunity is smaller
if the old (backup) database is always renamed (instead of sometimes
rmtree'd) before new (compacted) database is put into its place.
Finally rmtree() old database in case old database backup is not kept.
catch Xapian::Error in compact code in lib/database.cc to be consistent
with other code in addition to not making software crash on uncaught
other Xapian error.
There have been some plans for making build incompatible changes to
the library API. This is inconvenient, but it is much more so without
a way to easily conditional build against multiple versions of
notmuch.
The macro has been lifted from glib.
The extra path component added by the lib is a magic value that the
caller just has to know. This is demonstrated by the current code,
which indeed has "xapian.old" both sides of the interface. Use the
backup path provided by the lib caller verbatim, without adding
anything to it.
The queries don't really work after a database is closed, and we would
like them to be freed if the database is destroyed.
Acknowledged-by: David Bremner <david@tethera.net>
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
This function uses Xapian's Compactor machinery to compact the notmuch
database. The compacted database is built in a temporary directory and
later moved into place while the original uncompacted database is
preserved.
Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
As explained by Jeffrey Stedfast, the author of GMime, quoted in [1]:
> Passing the GMIME_ENABLE_RFC2047_WORKAROUNDS flag to g_mime_init()
> *should* solve the decoding problem mentioned in the thread. This
> flag should be safe to pass into g_mime_init() without any bad side
> effects and my unit tests do test that code-path.
The thread being referred to is [2].
[1] id:87bo56viyo.fsf@nikula.org
[2] id:08cb1dcd-c5db-4e33-8b09-7730cb3d59a2@gmail.com
notmuch_message_tags_to_maildir_flags() unconditionally moves messages from
maildir directory "new/" to maildir directory "cur/", which makes messages lose
their "new" status in the MUA. However some users want to keep this "new"
status after, for instance, an auto-tagging of new messages.
However, as Austin mentioned and according to the maildir specification,
messages living in "new/" are not allowed to have flags, even if mutt allows it
to happen. For this reason, this patch prevents moving messages from "new/" to
"cur/", only if no flags have to be changed. It's hopefully enough to satisfy
mutt (and maybe other MUAs showing the "new" status) users checking the "new"
status.
Changelog:
* v2: Fix bool type as well as NULL returned despite having no errors (Austin
Clements)
* v4: Tag the related test (contributed by Michal Sojka) as working
Signed-off-by: Louis Rilling <l.rilling@av7.net>
[Condition for keeping messages in new/ was extended to satisfy all
tests from the previous patch. -Michal Sojka]
[Added by David Bremner, to keep the tests passing at each commit]
update insert tests for new maildir synchronization rules
As of id:1355952747-27350-4-git-send-email-sojkam1@fel.cvut.cz
we are more conservative about moving messages from ./new to ./cur.
This updates the insert tests to match
Long story short, fix build on recent (3.2+) clang.
The long story for posterity follows.
gcc 4.6 added new warnings about structs with greater visibility than
their fields. The warnings were silenced by adjusting visibility in
commit d5523ead90
Author: Carl Worth <cworth@cworth.org>
Date: Wed May 11 13:23:13 2011 -0700
Mark some structures in the library interface with visibility=default attribute.
Later on,
commit 3b76adf9e2
Author: Austin Clements <amdragon@MIT.EDU>
Date: Sat Jan 14 19:17:33 2012 -0500
lib: Add support for automatically excluding tags from queries
changed visibility of struct _notmuch_string_list for the same reason, and
commit 1a53f9f116
Author: Mark Walters <markwalters1009@gmail.com>
Date: Thu Mar 1 22:30:38 2012 +0000
lib: Add the exclude flag to notmuch_query_search_threads
split the struct _notmuch_string_list and its typedef
notmuch_string_list_t as a way to make a forward declaration for
_notmuch_thread_create().
The subtle difference was that the struct definition now had 'visible'
in it, while the typedef didn't, and it was within the #pragma GCC
visibility push(hidden) block. This went unnoticed, as the then common
versions of clang didn't care about this.
A later change in clang (I did not dig into when this change was
introduced) caused the following error:
CXX -O2 lib/database.o
In file included from lib/database.cc:21:
In file included from ./lib/database-private.h:33:
./lib/notmuch-private.h:479:8: error: visibility does not match previous declaration
struct visible _notmuch_string_list {
^
./lib/notmuch-private.h:67:33: note: expanded from macro 'visible'
^
./lib/notmuch-private.h:52:13: note: previous attribute is here
^
1 error generated.
make: *** [lib/database.o] Error 1
This is slightly misleading due to the reference to the #pragma. The
real culprit is the typedef within the #pragma.
We could just add 'visible' to the typedef, or move the typedef
outside of the #pragma, and be done with it, but juggle the
declarations a bit to accommodate moving the typedef back with the
struct, and keep the visibility attribute in one place.
The problem was originally reported by Simonas Kazlauskas
<s@kazlauskas.me> in id:20130418102507.GA23688@godbox but I was only
able to reproduce and investigate now that I upgraded clang.
notmuch_message_get_header started returning some headers straight
from the database in 567bcbc, but this comment explicitly claimed all
headers were read from the message file.
Add NOTMUCH_EXCLUDE_FLAG to notmuch_exclude_t so that it can
cover all four values of search --exclude in the cli.
Previously the way to avoid any message being marked excluded was to
pass in an empty list of excluded tags: since we now have an explicit
option we might as well honour it.
The enum is in a slightly strange order as the existing FALSE/TRUE
options correspond to the new
NOTMUCH_EXCLUDE_FLAG/NOTMUCH_EXCLUDE_TRUE options so this means we do
not need to bump the version number.
Indeed, an example of this is that the cli count and show still use
FALSE/TRUE and still work.
Presently, the code which finds the parent of a message as it is being
added to the database assumes that the first Message-ID-like substring
of the In-Reply-To header is the parent Message ID. Some mail clients,
however, put stuff other than the Message-ID of the parent in the
In-Reply-To header, such as the email address of the sender of the
parent. This can fool notmuch.
The updated algorithm prefers the last Message ID in the References
header. The References header lists messages oldest-first, so the last
Message ID is the parent (RFC2822, p. 24). The References header is
also less likely to be in a non-standard
syntax (http://cr.yp.to/immhf/thread.html,
http://www.jwz.org/doc/threading.html). In case the References header
is not to be found, fall back to the old behavior.
V2 of this patch, incorporating feedback from Jani and (indirectly)
Austin.
Xapian::TermIterator::operator* returns std::string which is destroyed
as soon as (*i).c_str() finishes. The remembered pointer 'term' then
references invalid memory.
Signed-off-by: Vladimir Marek <vlmarek@volny.cz>
Notmuch automatically re-orders names of the format "Last, First" to
"First Last" when the associated email address is
First.Last@example.com. But, if a name is of the format "Last,First"
then notmuch will format the name as "irst Last". Handle any number of
spaces after the comma, including none.
Previously, getting the list of all messages in a thread required
recursively traversing the thread's message hierarchy, which was both
difficult and resulted in messages being out of order. This adds a
public function to retrieve an iterator over all of the messages in a
thread in oldest-first order.
Previously, thread.cc built up a list of all messages, then
proceeded to tear it apart to transform it into a list of
top-level messages. Now we simply build a new list of top-level
messages.
This simplifies the interface to _notmuch_message_add_reply,
eliminates the pointer acrobatics from
_resolve_thread_relationships, and will enable us to do things
with the list of all messages in the following patches.
Previously, there were various opportunities for memory leaks in the
error-handling paths of this function. Use a local talloc context and
some reparenting to make eliminate these leaks, while keeping the
control flow simple.
Using char instead of int allows for simpler definitions of the
DOCIDSET macros so the code is easier to understand and consistent with
respect to memory-usage. Estimated reduction of memory-usage for
bitmap about 8 times.
Apparently as of GMime 2.4, you don't need to call
internet_address_list_destroy anymore, but you still need to call
g_object_unref (from the GMime Changelog).
On the medium performance corpus, valgrind shows "possibly lost"
leakage in "notmuch new" dropping from 7M to 300k.
The message->headers hash table values get data returned by
g_mime_utils_header_decode_text ().
The pointer returned by g_mime_utils_header_decode_text is from the
following line in rfc2047_decode_tokens
return g_string_free (decoded, FALSE);
The docs for g_string_free say
Frees the memory allocated for the GString. If free_segment is TRUE
it also frees the character data. If it's FALSE, the caller gains
ownership of the buffer and must free it after use with g_free().
The remaining frees and allocations referencing to message->headers hash
values have been changed to use g_free and g_malloc functions.
This combines and completes the changes started by David Bremner.
Previously, we would treat multi-message mboxes as one giant email,
which, besides the obvious incorrect indexing, often led to
out-of-memory errors for archival mboxes. Now we explicitly reject
multi-message mboxes. For historical reasons, we retain support for
single-message mboxes, but official deprecate this behavior.
Add a custom value range processor to enable date and time searches of
the form date:since..until, where "since" and "until" are expressions
understood by the previously added date/time parser, to restrict the
results to messages within a particular time range (based on the Date:
header).
If "since" or "until" describes date/time at an accuracy of days or
less, the values are rounded according to the accuracy, towards past
for "since" and towards future for "until". For example,
date:november..yesterday would match from the beginning of November
until the end of yesterday. Expressions such as date:today..today
means since the beginning of today until the end of today.
Open-ended ranges are supported (since Xapian 1.2.1), i.e. you can
specify date:..until or date:since.. to not limit the start or end
date, respectively.
CAVEATS:
Xapian does not support spaces in range expressions. You can replace
the spaces with '_', or (in most cases) '-', or (in some cases) leave
the spaces out altogether.
Entering date:expr without ".." (for example date:yesterday) will not
work as you might expect. You can achieve the expected result by
duplicating the expr both sides of ".." (for example
date:yesterday..yesterday).
Open-ended ranges won't work with pre-1.2.1 Xapian, but they don't
produce an error either.
Signed-off-by: Jani Nikula <jani@nikula.org>
OpenBSD's build flags are identical to FreeBSD, except that libraries
need to be explicitly linked against libc. No code changes are
necessary.
From: Cody Cutler <ccutler@csail.mit.edu>
Fix the COERCE_STATUS macro to handle _internal_error being declared
as void function.
Note that the function _internal_error does not return. Evaluating to
NOTMUCH_STATUS_SUCCESS is done purely to appease the compiler.
Signed-off-by: Justus Winter <4winter@informatik.uni-hamburg.de>
The API documentation (notmuch.h) states that the parameter may be NULL,
but the implementation only checked the current element, potentially
dereferencing a NULL pointer in the process.
Signed-off-by: Sascha Silbe <sascha-pgp@silbe.org>
Previously, notmuch new only synchronized maildir flags to tags for
files with a maildir "info" part. Since messages in new/ don't have
an info part, notmuch would ignore them for flag-to-tag
synchronization.
This patch makes notmuch consider messages in new/ to be legitimate
maildir messages that simply have no maildir flags set. The most
visible effect of this is that such messages now automatically get the
unread tag.
Previously, we synchronized flags to tags for any message that looked
like it had maildir flags in its file name, regardless of whether it
was in a maildir-like directory structure. This was asymmetric with
tag-to-flag synchronization, which only applied to messages in
directories named new/ and cur/ (introduced by 95dd5fe5).
This change makes our interpretation stricter and addresses this
asymmetry by only synchronizing flags to tags for messages in
directories named new/ or cur/. It also prepares us to treat messages
in new/ as maildir messages, even though they lack maildir flags.
This way notmuch_message_maildir_flags_to_tags can call it. It makes
more sense for this to be just above all of the maildir
synchronization code rather than mixed in the middle.
Previously, if passed a filename with a directory that did not exist
in the database, _notmuch_message_remove_filename would needlessly
create that directory document. Fix it so that doesn't happen.
Previously, _notmuch_database_filename_to_direntry would abort with an
internal error when called on a read-only database. Now that creating
the directory document is optional,
notmuch_database_find_message_by_filename can disable directory
document creation (as it should) and, as a result, not abort on
read-only databases.
Using the new support from _notmuch_directory_create, this makes
notmuch_database_get_directory a read-only operation that simply
returns the directory object if it exists or NULL otherwise. This
also means that notmuch_database_get_directory can work on read-only
databases.
This change breaks the directory mtime workaround in notmuch-new.c by
fixing the exact issue it was working around. This permits mtime
update races to prevent scans of changed directories, which
non-deterministically breaks a few tests. The next patch fixes this.
