Commit graph

385 commits

Author SHA1 Message Date
Austin Clements
4a38588488 lib: Simplify upgrade code using a transaction
Previously, the upgrade was organized as two passes -- an upgrade
pass, and a separate cleanup pass -- so the database was always in a
valid state.  This change substantially simplifies this code by
performing the upgrade in a transaction and combining both passes in
to one.  This 1) eliminates a lot of duplicate code between the
passes, 2) speeds up the upgrade process, 3) makes progress reporting
more accurate, 4) eliminates the potential for stale data if the
upgrade is interrupted during the cleanup pass, and 5) makes it easier
to reason about the safety of the upgrade code.
2014-08-30 10:45:36 -07:00
Austin Clements
8363c90531 lib: Database version 3: Introduce fine-grained "features"
Previously, our database schema was versioned by a single number.
Each database schema change had to occur "atomically" in Notmuch's
development history: before some commit, Notmuch used version N, after
that commit, it used version N+1.  Hence, each new schema version
could introduce only one change, the task of developing a schema
change fell on a single person, and it all had to happen and be
perfect in a single commit series.  This made introducing a new schema
version hard.  We've seen only two schema changes in the history of
Notmuch.

This commit introduces database schema version 3; hopefully the last
schema version we'll need for a while.  With this version, we switch
from a single version number to "features": a set of named,
independent aspects of the database schema.

Features should make backwards compatibility easier.  For many things,
it should be easy to support databases both with and without a
feature, which will allow us to make upgrades optional and will enable
"unstable" features that can be developed and tested over time.

Features also make forwards compatibility easier.  The features
recorded in a database include "compatibility flags," which can
indicate to an older version of Notmuch when it must support a given
feature to open the database for read or for write.  This lets us
replace the old vague "I don't recognize this version, so something
might go wrong, but I promise to try my best" warnings upon opening a
database with an unknown version with precise errors.  If a database
is safe to open for read/write despite unknown features, an older
version will know that and issue no message at all.  If the database
is not safe to open for read/write because of unknown features, an
older version will know that, too, and can tell the user exactly which
required features it lacks support for.
2014-08-30 10:42:08 -07:00
Michal Sojka
028c56061e Make parsing of References and In-Reply-To header less error prone
According to RFC2822 References and In-Reply-To headers are supposed
to contain one or more Message-IDs, however older RFC822 allowed
almost any content. When both References and In-Reply-To headers ends
with something else that a Message-ID (see e.g. [1]), the thread
structure presented by notmuch is incorrect. The reason is that
notmuch treats this case as if the email contained no "replyto"
information (see _notmuch_database_link_message_to_parents).

This patch changes the parse_references() function to return the last
valid Message-ID encountered rather than NULL resulting from the last
hunk of text not being the Message-ID.

[1] https://lkml.org/lkml/headers/2014/5/19/864
2014-08-16 17:45:16 -07:00
Austin Clements
0c1292051e lib: Improve documentation of _notmuch_message_create_for_message_id
Clarify the state of the returned message when
_notmuch_message_create_for_message_id returns
NOTMUCH_PRIVATE_STATUS_NO_DOCUMENT_FOUND.
2014-08-05 08:14:15 -03:00
Austin Clements
1d652c8719 lib: Fix slight misinformation in the database schema doc
The database schema documentation made it sound like each mail
document had exactly one on-disk message file, which hasn't been true
for a long time.
2014-08-04 18:58:11 -03:00
Austin Clements
30de720ba0 lib: Invalidate message metadata in _notmuch_message_gen_terms
Previously, we invalidated stored message metadata in
_notmuch_message_add_term and _notmuch_message_remove_term, but not in
_notmuch_message_gen_terms.  This doesn't currently result in any bugs
because of our limited uses of _notmuch_message_gen_terms, but it may
could cause trouble in the future.
2014-08-04 18:57:55 -03:00
Charles Celerier
df8885f62c lib: Start all function names in notmuch-private.h with
As noted in devel/STYLE, every private library function should start
with _notmuch. This patch corrects function naming that did not adhere
to this style in lib/notmuch-private.h. In particular, the old function
names that now begin with _notmuch are

    notmuch_sha1_of_file
    notmuch_sha1_of_string
    notmuch_message_file_close
    notmuch_message_file_get_header
    notmuch_message_file_open
    notmuch_message_get_author
    notmuch_message_set_author

Signed-off-by: Charles Celerier <cceleri@cs.stanford.edu>
2014-07-13 12:25:29 -03:00
Jani Nikula
ab24e883b0 lib: add return status to database close and destroy
notmuch_database_close may fail in Xapian ->flush() or ->close(), so
report the status. Similarly for notmuch_database_destroy which calls
close.

This is required for notmuch insert to report error status if message
indexing failed.
2014-07-09 20:29:36 -03:00
Austin Clements
dc64ab6720 lib: Separate all phrases indexed by _notmuch_message_gen_terms
This adds a 100 termpos gap between all phrases indexed by
_notmuch_message_gen_terms.  This fixes a bug where terms from the end
of one header and the beginning of another header could match together
in a single phrase and a separate bug where term positions of
un-prefixed terms overlapped.

This fix only affects newly indexed messages.  Messages that are
already indexed won't benefit from this fix without re-indexing, but
the fix won't make things any worse for existing messages.
2014-06-18 18:03:18 -03:00
Austin Clements
44327ca86d lib: Index name and address of from/to headers as a phrase
Previously, we indexed the name and address parts of from/to headers
with two calls to _notmuch_message_gen_terms.  In general, this
indicates that these parts are separate phrases.  However, because of
an implementation quirk, the two calls to _notmuch_message_gen_terms
generated adjacent term positions for the prefixed terms, which
happens to be the right thing to do in this case, but the wrong thing
to do for all other calls.  Furthermore, _notmuch_message_gen_terms
produced potentially overlapping term positions for the un-prefixed
copies of the terms, which is simply wrong.

This change indexes both the name and address in a single call to
_notmuch_message_gen_terms, indicating that they should be part of a
single phrase.  This masks the problem with the un-prefixed terms
(fixing the two known-broken tests) and puts us in a position to fix
the unintentionally phrases generated by other calls to
_notmuch_message_gen_terms.
2014-06-18 17:55:14 -03:00
Jani Nikula
0cc0144875 lib: resurrect support for single-message mbox files
This is effectively a revert of

commit 6812136bf5
Author: Jani Nikula <jani@nikula.org>
Date:   Mon Mar 31 00:21:48 2014 +0300

    lib: drop support for single-message mbox files

The intention was to drop support for indexing new single-message mbox
files (and whether that was a good idea in the first place is
arguable). However this inadvertently broke support for reading
headers from previously indexed single-message mbox files, which is
far worse.

Distinguishing between the two cases would require more code than
simply bringing back support for single-message mbox files.
2014-06-13 22:59:04 -03:00
David Bremner
3242e29e57 build: add canonicalize_file_name to symbols exported from libnotmuch.so
This is needed for our compat version of canonicalize_file_name to be used.
2014-04-19 05:58:34 +09:00
Jani Nikula
473930bb6f lib: replace the header parser with gmime
The notmuch library includes a full blown message header parser. Yet
the same message headers are parsed by gmime during indexing. Switch
to gmime parsing completely.

These are the main changes:

* Gmime stops header parsing at the first invalid header, and presumes
  the message body starts from there. The current parser is quite
  liberal in accepting broken headers. The change means we will be
  much pickier about accepting invalid messages.

* The current parser converts tabs used in header folding to
  spaces. Gmime preserve the tabs. Due to a broken python library used
  in mailman, there are plenty of mailing lists that produce headers
  with tabs in header folding, and we'll see plenty of tabs. (This
  change has been mitigated in preparatory patches.)

* For pure header parsing, the current parser is likely faster than
  gmime, which parses the whole message rather than just the
  headers. Since we parse the message and its headers using gmime for
  indexing anyway, this avoids and extra header parsing round when
  adding new messages. In case of duplicate messages, we'll end up
  parsing the full message although just headers would be
  sufficient. All in all this should still speed up 'notmuch new'.

* Calls to notmuch_message_get_header() may be slightly slower than
  previously for headers that are not indexed in the database, due to
  parsing of the whole message. Within the notmuch code base, notmuch
  reply is the only such user.
2014-04-05 12:53:04 -03:00
Jani Nikula
6812136bf5 lib: drop support for single-message mbox files
We've supported mbox files containing a single message for historical
reasons, but the support has been deprecated, with a warning message
while indexing, since Notmuch 0.15. Finally drop the support, and
consider all mbox files non-email.
2014-04-05 12:52:42 -03:00
Jani Nikula
1fa8e40561 lib: make folder: prefix literal
In xapian terms, convert folder: prefix from probabilistic to boolean
prefix, matching the paths, relative from the maildir root, of the
message files, ignoring the maildir new and cur leaf directories.

folder:foo matches all message files in foo, foo/new, and foo/cur.

folder:foo/new does *not* match message files in foo/new.

folder:"" matches all message files in the top level maildir and its
new and cur subdirectories.

This change constitutes a database change: bump the database version
and add database upgrade support for folder: terms. The upgrade also
adds path: terms.

Finally, fix the folder search test for literal folder: search, as
some of the folder: matching capabilities are lost in the
probabilistic to boolean prefix change.
2014-03-11 19:51:22 -03:00
Jani Nikula
59823f9642 lib: add support for path: prefix searches
The path: prefix is a literal boolean prefix matching the paths,
relative from the maildir root, of the message files.

path:foo matches all message files in foo (but not in foo/new or
foo/cur).

path:foo/new matches all message files in foo/new.

path:"" matches all message files in the top level maildir.

path:foo/** matches all message files in foo and recursively in all
subdirectories of foo.

path:** matches all message files recursively, i.e. all messages.
2014-03-11 19:51:22 -03:00
Jani Nikula
4d150eba67 lib: refactor folder term update after filename removal
Abstract some blocks of code for reuse. No functional changes.
2014-03-11 19:51:22 -03:00
Gaute Hope
91aede05a3 doc: notmuch_result_move_to_next -> notmuch_tags_move_to_next
Fix typo in docs.
2014-02-13 08:43:34 -04:00
David Bremner
3b74537f49 lib: update documentation for notmuch_database_get_directory
Clarify that using the directory after destroying the corresponding
database is not permitted.

This is implicit in the description of notmuch_database_destroy, but
it doesn't hurt to be explicit, and we do express similar "ownership"
relationships at other places in the docs.
2014-01-26 09:36:38 -04:00
David Bremner
29f1252126 lib: make notmuch_threads_valid return FALSE when passed NULL
Without this patch, the example code in the header docs crashes for certain
invalid queries (see id:871u00oimv.fsf@approx.mit.edu)
2014-01-24 20:20:54 -04:00
Tomi Valkeinen
075d53dde5 lib: fix error handling
Currently if a Xapian exception happens in notmuch_message_get_header,
the exception is not caught leading to crash. In
notmuch_message_get_date the exception is caught, but an internal error
is raised, again leading to crash.

This patch fixes the error handling by making both functions catch the
Xapian exceptions, print an error and return NULL or 0.

The 'notmuch->exception_reported' is also set, as is done elsewhere,
even if I don't really get the idea of that field.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@iki.fi>
2014-01-18 14:47:35 -04:00
Jani Nikula
f94834407b lib: fix clang compiler warning
With some combination of clang and talloc, not using the return value
of talloc_steal() produces a warning. Ignore it, as talloc_steal() has
no failure modes per documentation.
2014-01-18 14:39:51 -04:00
Jani Nikula
b65ca8e0ba lib: modify notmuch.h for automatic document generation
Minimal changes to produce a sensible result.
2014-01-05 09:05:00 -04:00
Austin Clements
3d4b0930bd lib: Bump library version from 3.0.0 to 3.1.0
This version of the library introduces LIBNOTMUCH_CHECK_VERSION and
the *_VERSION macros.  Bumping the version number is also necessary to
make the comment on LIBNOTMUCH_CHECK_VERSION no longer a lie.
2013-12-07 10:31:19 +08:00
Austin Clements
8fb16e277e lib: Replace NOTMUCH_*_VERSION with LIBNOTMUCH_*_VERSION
This makes it clear that these macros refer to the *library* version,
and not to the notmuch application-level release.  Since there are no
consumers of these macros yet, this is now or never.
2013-12-07 10:31:11 +08:00
Austin Clements
4b14ccf0d6 lib: Make VERSION macros agree with soname version
We have two distinct "library version" numbers: the soname version and
the version macros.  We need both for different reasons: the version
macros enable easy compile-time version detection (and conditional
compilation), while the soname version enables runtime version
detection (which includes the version checking done by things like the
Python bindings).

However, currently, these two version numbers are different, which is
unnecessary and can lead to confusion (especially in things like
Debian, which include the soname version in the package name).  This
patch makes them the same by bumping the version macros up to agree
with the soname version.

(We should probably keep the version number in just one place so they
can't get out of sync, but that can be done in another patch.)
2013-12-07 10:31:04 +08:00
David Bremner
b9f0e6923d util: detect byte order
Unfortunately old versions of GCC and clang do not provide byte order
macros, so we re-invent them.

If UTIL_BYTE_ORDER is not defined or defined to 0, we fall back to
macros supported by recent versions of GCC and clang
2013-11-27 07:43:29 -04:00
David Bremner
20adfe39d7 lib: fix byte order test in libsha1.c
Previously PLATFORM_BYTE_ORDER and IS_LITTLE_ENDIAN were not defined,
so the little endian code was always compiled in.

This will have the effect that the "SHA1s" on big endian architectures
will change (i.e. become actual sha1s). So someone re-indexing their
database could conceivable lose tags on messages without a message-id
header.
2013-11-25 07:55:54 -04:00
Tomi Ollila
2fd7ef64ba compact: improve error messages on failures after compaction
The error messages written during the steps replacing old
database with new now includes relevant paths and strerror.
2013-11-19 20:15:02 -04:00
Tomi Ollila
6452ae0fcb compact: unconditionally remove old wip database compact directory
In case previous notmuch compact has been interrupted there is old
work-in-progress database compact directory partially filled. Remove
it just before starting to fill the directory with new files.
2013-11-19 20:14:28 -04:00
Tomi Ollila
cb6cc296e2 compact: preserve backup database until compacted database is in place
It is less error prone and window of failure opportunity is smaller
if the old (backup) database is always renamed (instead of sometimes
rmtree'd) before new (compacted) database is put into its place.
Finally rmtree() old database in case old database backup is not kept.
2013-11-19 20:13:25 -04:00
Tomi Ollila
19a89753ca compact: catch Xapian::Error consistently
catch Xapian::Error in compact code in lib/database.cc to be consistent
with other code in addition to not making software crash on uncaught
other Xapian error.
2013-11-17 20:25:43 -04:00
Tomi Ollila
4d5986e8ad compact: tidy formatting
Notmuch compact code whitespace changes to match devel/STYLE.
2013-11-17 20:25:25 -04:00
Austin Clements
215a8dc29c lib: Document extent of some return values
This documents the extent of the notmuch_messages_t* pointers returned
by notmuch_thread_get_toplevel_messages and
notmuch_thread_get_messages.
2013-11-08 20:14:08 -04:00
Jani Nikula
060990db14 lib: add library version check macro
There have been some plans for making build incompatible changes to
the library API. This is inconvenient, but it is much more so without
a way to easily conditional build against multiple versions of
notmuch.

The macro has been lifted from glib.
2013-11-07 19:15:06 -04:00
Jani Nikula
00d2ac2b41 lib: use the compaction backup path provided by the caller
The extra path component added by the lib is a magic value that the
caller just has to know. This is demonstrated by the current code,
which indeed has "xapian.old" both sides of the interface. Use the
backup path provided by the lib caller verbatim, without adding
anything to it.
2013-11-07 06:51:16 -04:00
David Bremner
20abbe89a3 lib: update documentation of callback functions for database_compact and database_upgrade.
Compact was missing callback documentation entirely, and upgrade did not discuss the
closure parameter.
2013-11-07 06:50:50 -04:00
Jani Nikula
180dba66e4 lib: add closure parameter to compact status update callback
This provides much more flexibility for the caller.
2013-11-07 06:46:42 -04:00
Jani Nikula
35ca5feb28 lib: do not leak the database in compaction
Destroy instead of close the database after compaction, and also on
error path, to not leak the database.
2013-11-07 06:46:25 -04:00
Jani Nikula
a95dbba156 lib: check talloc success in compact
In line with the allocation checks all around.
2013-11-06 17:49:46 -04:00
Jani Nikula
8e4e537cee lib: construct compactor within try block to catch any exceptions
Constructors may also throw exceptions. Catch them.
2013-11-06 17:49:36 -04:00
Felipe Contreras
fd733a4f5c query: bind queries to database objects
The queries don't really work after a database is closed, and we would
like them to be freed if the database is destroyed.

Acknowledged-by: David Bremner <david@tethera.net>
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2013-11-02 07:20:01 -06:00
Jani Nikula
8e4900b8a7 lib: fix build on !HAVE_XAPIAN_COMPACT
Minimal change to build notmuch against xapian that doesn't have
compaction support.
2013-10-30 21:16:22 -03:00
Ben Gamari
0bd11b654e database: Add notmuch_database_compact_close
This function uses Xapian's Compactor machinery to compact the notmuch
database. The compacted database is built in a temporary directory and
later moved into place while the original uncompacted database is
preserved.

Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-09 21:46:49 -03:00
Jani Nikula
71521f06b0 lib/cli: pass GMIME_ENABLE_RFC2047_WORKAROUNDS to g_mime_init()
As explained by Jeffrey Stedfast, the author of GMime, quoted in [1]:

> Passing the GMIME_ENABLE_RFC2047_WORKAROUNDS flag to g_mime_init()
> *should* solve the decoding problem mentioned in the thread. This
> flag should be safe to pass into g_mime_init() without any bad side
> effects and my unit tests do test that code-path.

The thread being referred to is [2].

[1] id:87bo56viyo.fsf@nikula.org
[2] id:08cb1dcd-c5db-4e33-8b09-7730cb3d59a2@gmail.com
2013-09-14 14:13:43 -03:00
Louis Rilling
a9b2135c75 tags_to_maildir_flags: Don't rename if no flags change
notmuch_message_tags_to_maildir_flags() unconditionally moves messages from
maildir directory "new/" to maildir directory "cur/", which makes messages lose
their "new" status in the MUA. However some users want to keep this "new"
status after, for instance, an auto-tagging of new messages.

However, as Austin mentioned and according to the maildir specification,
messages living in "new/" are not allowed to have flags, even if mutt allows it
to happen. For this reason, this patch prevents moving messages from "new/" to
"cur/", only if no flags have to be changed. It's hopefully enough to satisfy
mutt (and maybe other MUAs showing the "new" status) users checking the "new"
status.

Changelog:
* v2: Fix bool type as well as NULL returned despite having no errors (Austin
      Clements)
* v4: Tag the related test (contributed by Michal Sojka) as working

Signed-off-by: Louis Rilling <l.rilling@av7.net>

[Condition for keeping messages in new/ was extended to satisfy all
 tests from the previous patch. -Michal Sojka]

[Added by David Bremner, to keep the tests passing at each commit]

update insert tests for new maildir synchronization rules

As of id:1355952747-27350-4-git-send-email-sojkam1@fel.cvut.cz
we are more conservative about moving messages from ./new to ./cur.
This updates the insert tests to match
2013-09-03 20:41:51 -03:00
Jani Nikula
db465e443f lib: fix clang build
Long story short, fix build on recent (3.2+) clang.

The long story for posterity follows.

gcc 4.6 added new warnings about structs with greater visibility than
their fields. The warnings were silenced by adjusting visibility in

commit d5523ead90
Author: Carl Worth <cworth@cworth.org>
Date:   Wed May 11 13:23:13 2011 -0700

    Mark some structures in the library interface with visibility=default attribute.

Later on,

commit 3b76adf9e2
Author: Austin Clements <amdragon@MIT.EDU>
Date:   Sat Jan 14 19:17:33 2012 -0500

    lib: Add support for automatically excluding tags from queries

changed visibility of struct _notmuch_string_list for the same reason, and

commit 1a53f9f116
Author: Mark Walters <markwalters1009@gmail.com>
Date:   Thu Mar 1 22:30:38 2012 +0000

    lib: Add the exclude flag to notmuch_query_search_threads

split the struct _notmuch_string_list and its typedef
notmuch_string_list_t as a way to make a forward declaration for
_notmuch_thread_create().

The subtle difference was that the struct definition now had 'visible'
in it, while the typedef didn't, and it was within the #pragma GCC
visibility push(hidden) block. This went unnoticed, as the then common
versions of clang didn't care about this.

A later change in clang (I did not dig into when this change was
introduced) caused the following error:

CXX  -O2 lib/database.o
In file included from lib/database.cc:21:
In file included from ./lib/database-private.h:33:
./lib/notmuch-private.h:479:8: error: visibility does not match previous declaration
struct visible _notmuch_string_list {
       ^
./lib/notmuch-private.h:67:33: note: expanded from macro 'visible'
                                ^
./lib/notmuch-private.h:52:13: note: previous attribute is here
            ^
1 error generated.
make: *** [lib/database.o] Error 1

This is slightly misleading due to the reference to the #pragma. The
real culprit is the typedef within the #pragma.

We could just add 'visible' to the typedef, or move the typedef
outside of the #pragma, and be done with it, but juggle the
declarations a bit to accommodate moving the typedef back with the
struct, and keep the visibility attribute in one place.

The problem was originally reported by Simonas Kazlauskas
<s@kazlauskas.me> in id:20130418102507.GA23688@godbox but I was only
able to reproduce and investigate now that I upgraded clang.
2013-09-01 07:06:54 -03:00
Austin Clements
3fbb518335 lib: Document which strings are returned in UTF-8
Any string that ultimately comes from notmuch_message_file_get_header
is in UTF-8.
2013-08-13 17:43:34 +02:00
Austin Clements
a7bf045fb1 lib: Correct out-of-date doc comment
notmuch_message_get_header started returning some headers straight
from the database in 567bcbc, but this comment explicitly claimed all
headers were read from the message file.
2013-08-13 17:43:07 +02:00
Mark Walters
d29d7e1ee2 lib: add NOTMUCH_EXCLUDE_FLAG to notmuch_exclude_t
Add NOTMUCH_EXCLUDE_FLAG to notmuch_exclude_t so that it can
cover all four values of search --exclude in the cli.

Previously the way to avoid any message being marked excluded was to
pass in an empty list of excluded tags: since we now have an explicit
option we might as well honour it.

The enum is in a slightly strange order as the existing FALSE/TRUE
options correspond to the new
NOTMUCH_EXCLUDE_FLAG/NOTMUCH_EXCLUDE_TRUE options so this means we do
not need to bump the version number.

Indeed, an example of this is that the cli count and show still use
FALSE/TRUE and still work.
2013-06-24 22:53:16 -07:00