Commit graph

219 commits

Author SHA1 Message Date
Jani Nikula
a95dbba156 lib: check talloc success in compact
In line with the allocation checks all around.
2013-11-06 17:49:46 -04:00
Jani Nikula
8e4e537cee lib: construct compactor within try block to catch any exceptions
Constructors may also throw exceptions. Catch them.
2013-11-06 17:49:36 -04:00
Jani Nikula
8e4900b8a7 lib: fix build on !HAVE_XAPIAN_COMPACT
Minimal change to build notmuch against xapian that doesn't have
compaction support.
2013-10-30 21:16:22 -03:00
Ben Gamari
0bd11b654e database: Add notmuch_database_compact_close
This function uses Xapian's Compactor machinery to compact the notmuch
database. The compacted database is built in a temporary directory and
later moved into place while the original uncompacted database is
preserved.

Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-09 21:46:49 -03:00
Jani Nikula
71521f06b0 lib/cli: pass GMIME_ENABLE_RFC2047_WORKAROUNDS to g_mime_init()
As explained by Jeffrey Stedfast, the author of GMime, quoted in [1]:

> Passing the GMIME_ENABLE_RFC2047_WORKAROUNDS flag to g_mime_init()
> *should* solve the decoding problem mentioned in the thread. This
> flag should be safe to pass into g_mime_init() without any bad side
> effects and my unit tests do test that code-path.

The thread being referred to is [2].

[1] id:87bo56viyo.fsf@nikula.org
[2] id:08cb1dcd-c5db-4e33-8b09-7730cb3d59a2@gmail.com
2013-09-14 14:13:43 -03:00
Tomi Ollila
8d6aa603ef cli: Guard deprecated g_type_init calls
g_type_init was deprecated in GLib 2.35.1.  In order to compile
cleanly, guard these with a suitable #if.

(commit msg from https://bugs.freedesktop.org/attachment.cgi?id=73774 )
2013-06-08 20:42:33 -03:00
Aaron Ecay
cf8aaafbad lib/database.cc: change how the parent of a message is calculated
Presently, the code which finds the parent of a message as it is being
added to the database assumes that the first Message-ID-like substring
of the In-Reply-To header is the parent Message ID.  Some mail clients,
however, put stuff other than the Message-ID of the parent in the
In-Reply-To header, such as the email address of the sender of the
parent.  This can fool notmuch.

The updated algorithm prefers the last Message ID in the References
header.  The References header lists messages oldest-first, so the last
Message ID is the parent (RFC2822, p. 24).  The References header is
also less likely to be in a non-standard
syntax (http://cr.yp.to/immhf/thread.html,
http://www.jwz.org/doc/threading.html).  In case the References header
is not to be found, fall back to the old behavior.

V2 of this patch, incorporating feedback from Jani and (indirectly)
Austin.
2013-05-13 21:29:13 -03:00
Austin Clements
610f0e0992 lib: Reject multi-message mboxes and deprecate single-message mbox
Previously, we would treat multi-message mboxes as one giant email,
which, besides the obvious incorrect indexing, often led to
out-of-memory errors for archival mboxes.  Now we explicitly reject
multi-message mboxes.  For historical reasons, we retain support for
single-message mboxes, but official deprecate this behavior.
2012-11-26 21:12:10 -04:00
Jani Nikula
90cd1bac4e lib: add date range query support
Add a custom value range processor to enable date and time searches of
the form date:since..until, where "since" and "until" are expressions
understood by the previously added date/time parser, to restrict the
results to messages within a particular time range (based on the Date:
header).

If "since" or "until" describes date/time at an accuracy of days or
less, the values are rounded according to the accuracy, towards past
for "since" and towards future for "until". For example,
date:november..yesterday would match from the beginning of November
until the end of yesterday. Expressions such as date:today..today
means since the beginning of today until the end of today.

Open-ended ranges are supported (since Xapian 1.2.1), i.e. you can
specify date:..until or date:since.. to not limit the start or end
date, respectively.

CAVEATS:

Xapian does not support spaces in range expressions. You can replace
the spaces with '_', or (in most cases) '-', or (in some cases) leave
the spaces out altogether.

Entering date:expr without ".." (for example date:yesterday) will not
work as you might expect. You can achieve the expected result by
duplicating the expr both sides of ".." (for example
date:yesterday..yesterday).

Open-ended ranges won't work with pre-1.2.1 Xapian, but they don't
produce an error either.

Signed-off-by: Jani Nikula <jani@nikula.org>
2012-10-31 16:55:32 -03:00
Austin Clements
cdd698f969 lib: Make notmuch_database_find_message_by_filename not crash on read-only databases
Previously, _notmuch_database_filename_to_direntry would abort with an
internal error when called on a read-only database.  Now that creating
the directory document is optional,
notmuch_database_find_message_by_filename can disable directory
document creation (as it should) and, as a result, not abort on
read-only databases.
2012-05-23 22:31:47 -03:00
Austin Clements
fe1ca14104 lib: Make notmuch_database_get_directory return NULL if the directory is not found
Using the new support from _notmuch_directory_create, this makes
notmuch_database_get_directory a read-only operation that simply
returns the directory object if it exists or NULL otherwise.  This
also means that notmuch_database_get_directory can work on read-only
databases.

This change breaks the directory mtime workaround in notmuch-new.c by
fixing the exact issue it was working around.  This permits mtime
update races to prevent scans of changed directories, which
non-deterministically breaks a few tests.  The next patch fixes this.
2012-05-23 22:30:55 -03:00
Austin Clements
67ae2377a9 lib: Perform the same transformation to _notmuch_database_filename_to_direntry
Now _notmuch_database_filename_to_direntry takes a flags argument and
can indicate if the necessary directory documents do not exist.
Again, callers have been updated, but retain their original behavior.
2012-05-23 22:30:43 -03:00
Austin Clements
0c950146a1 lib: Perform the same transformation to _notmuch_database_find_directory_id
Now _notmuch_database_find_directory_id takes a flags argument, which
it passes through to _notmuch_directory_create and can indicate if the
directory does not exist.  Again, callers have been updated, but
retain their original behavior.
2012-05-23 22:30:32 -03:00
Austin Clements
f69314fbd3 lib: Make directory document creation optional for _notmuch_directory_create
Previously this function would create directory documents if they
didn't exist.  As a result, it could only be used on writable
databases.  This adds an argument to make creation optional and to
make this function work on read-only databases.  We use a flag
argument to avoid a bare boolean and to permit future expansion.

Both callers have been updated, but currently retain the old behavior.
We'll take advantage of the new argument in the following patches.
2012-05-23 22:30:20 -03:00
Austin Clements
7199d22f43 lib/cli: Make notmuch_database_get_directory return a status code
Previously, notmuch_database_get_directory had no way to indicate how
it had failed.  This changes its prototype to return a status code and
set an out-argument to the retrieved directory, like similar functions
in the library API.  This does *not* change its currently broken
behavior of creating directory objects when they don't exist, but it
does document it and paves the way for fixing this.  Also, it can now
check for a read-only database and return
NOTMUCH_STATUS_READ_ONLY_DATABASE instead of crashing.

In the interest of atomicity, this also updates calls from the CLI so
that notmuch still compiles.
2012-05-15 08:56:33 -03:00
Austin Clements
ba57294218 lib/cli: Make notmuch_database_create return a status code
This is the notmuch_database_create equivalent of the previous change.

In this case, there were places where errors were not being propagated
correctly in notmuch_database_create or in calls to it.  These have
been fixed, using the new status value.
2012-05-05 10:12:26 -03:00
Austin Clements
5fddc07dc3 lib/cli: Make notmuch_database_open return a status code
It has been a long-standing issue that notmuch_database_open doesn't
return any indication of why it failed.  This patch changes its
prototype to return a notmuch_status_t and set an out-argument to the
database itself, like other functions that return both a status and an
object.

In the interest of atomicity, this also updates every use in the CLI
so that notmuch still compiles.  Since this patch does not update the
bindings, the Python bindings test fails.
2012-05-05 10:11:57 -03:00
Justus Winter
7864350c93 Split notmuch_database_close into two functions
Formerly notmuch_database_close closed the xapian database and
destroyed the talloc structure associated with the notmuch database
object. Split notmuch_database_close into notmuch_database_close and
notmuch_database_destroy.

This makes it possible for long running programs to close the xapian
database and thus release the lock associated with it without
destroying the data structures obtained from it.

This also makes the api more consistent since every other data
structure has a destructor function.

The comments in notmuch.h are a courtesy of Austin Clements.

Signed-off-by: Justus Winter <4winter@informatik.uni-hamburg.de>
2012-04-28 09:21:13 -03:00
Justus Winter
ea54c4fdc7 Fix error reporting in notmuch_database_find_message_by_filename
Formerly it was possible for *message_ret to be left
uninitialized. The documentation however clearly states that "[o]n any
failure or when the message is not found, this function initializes
'*message' to NULL".

Signed-off-by: Justus Winter <4winter@informatik.uni-hamburg.de>
2012-03-18 07:58:35 -03:00
Justus Winter
cfc5f1059a Actually close the xapian database in notmuch_database_close
Formerly the xapian database object was deleted and closed in its
destructor once the object was garbage collected. Explicitly call
close() so that the database and the associated lock is released
immediately.

The comment is a courtesy of Austin Clements.

Signed-off-by: Justus Winter <4winter@informatik.uni-hamburg.de>
2012-03-03 11:30:07 -04:00
Justus Winter
e2e95caa51 Prevent segmentation fault in notmuch_database_close
Previously opening a notmuch database in read write mode that has been
locked resulted in the notmuch_database_open function executing
notmuch_database_close as a cleanup function. notmuch_database_close
failed to check whether the xapian database has in fact been created.

Add a check whether the xapian database object has actually been
created before trying to call its flush method.

Signed-off-by: Justus Winter <4winter@informatik.uni-hamburg.de>
2012-02-20 23:03:25 -04:00
Austin Clements
c32116d048 lib: Use talloc to simplify cleanup in notmuch_database_open
Previously, we manually "free"d various pointers in
notmuch_database_open.  Use a local talloc context instead to simplify
cleanup and eliminate various NULL pointer initializations and
conditionals.
2012-02-03 21:15:45 -04:00
Austin Clements
6c0adab23e lib: Release resources if notmuch_database_open fails
Previously, if a Xapian exception occurred in notmuch_database_open,
we failed to clean up the allocated notmuch_database_t object.
2012-02-03 21:15:26 -04:00
Austin Clements
a8ee1c75c3 lib: Don't delete uninitialized pointers
In the error-handling paths of notmuch_database_open, we call
notmuch_database_close, which "delete"s several objects referenced by
the notmuch_database_t object.  However, some of these pointers may be
uninitialized, resulting in undefined behavior.  Hence, allocate the
notmuch_database_t with talloc_zero to make sure these pointers are
NULL so that "delete"ing them is harmless.
2012-02-03 21:14:59 -04:00
Kazuo Teramoto
442d405ad3 lib: call g_mime_init() from notmuch_database_open()
As reported in
id:"CAEbOPGyuHnz4BPtDutnTPUHcP3eYcRCRkXhYoJR43RUMw671+g@mail.gmail.com"
sometimes gmime tries to access a NULL pointer, e.g. g_mime_iconv_open()
tries to access iconv_cache that is NULL if g_mime_init() is not called.
This causes notmuch to segfault when calling gmime functions.

Calling g_mime_init() initializes iconv_cache and others variables needed
by gmime, making sure they are initialized when notmuch calls gmime
functions.

Test marked fix by db.
2011-12-31 23:08:15 -04:00
Thomas Jost
824dad76b6 Fix comments about what is stored in the database
Commit 567bcbc2 introduced two new values for each message (content of the
"From" and "Subject" headers), but the comments about the database schema had
not been updated accordingly.
2011-12-23 15:08:34 -04:00
David Edmondson
77ec8108a1 notmuch: Quiet buildbot warnings.
Cast away the result of various *write functions. Provide a default
value for some variables to avoid "use before set" warnings.
2011-12-21 07:32:16 -04:00
David Bremner
69dc421ab3 lib: call g_type_init from notmuch_database_open
We want to make sure g_type_init is called before any GObject
functionality is used.
2011-12-04 22:00:25 -04:00
Austin Clements
567bcbc294 Store "from" and "subject" headers in the database.
This is a rebase and cleanup of Istvan Marko's patch from
id:m3pqnj2j7a.fsf@zsu.kismala.com

Search retrieves these headers for every message in the search
results.  Previously, this required opening and parsing every message
file.  Storing them directly in the database significantly reduces IO
and computation, speeding up search by between 50% and 10X.

Taking full advantage of this requires a database rebuild, but it will
fall back to the old behavior for messages that do not have headers
stored in the database.
2011-11-14 17:10:58 -04:00
David Bremner
606e4461f8 lib/database.cc: use flush Xapian method instead of commit
Apparently the method was renamed in Xapian 1.1.0 but the old method
name will stay around for a while. It seems better to stick with the
old name to make notmuch compile with older versions of Xapian, at
least for now.
2011-11-02 20:38:22 -03:00
David Bremner
1dedfc90f6 xutil.c: remove duplicate copies, create new library libutil.a to contain xutil.
We keep the lib/xutil.c version. As a consequence, also factor out
_internal_error and associated macros.  It might be overkill to make a
new file error_util.c for this, but _internal_error does not really
belong in database.cc.
2011-10-30 23:09:49 -03:00
Ali Polatel
02a3076711 lib: make find_message{,by_filename) report errors
Previously, the functions notmuch_database_find_message() and
notmuch_database_find_message_by_filename() functions did not properly
report error condition to the library user.

For more information, read the thread on the notmuch mailing list
starting with my mail "id:871uv2unfd.fsf@gmail.com"

Make these functions accept a pointer to 'notmuch_message_t' as argument
and return notmuch_status_t which may be used to check for any error
condition.

restore: Modify for the new notmuch_database_find_message()
new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 07:55:29 +03:00
Ali Polatel
9b3f16ce71 lib: destroy message object after message removal
notmuch_database_remove_message() must call notmuch_message_destroy()
once it is done handling message removal.
2011-10-04 07:39:57 +03:00
Austin Clements
51c3c0b2d2 lib: Wrap notmuch_database_add_message in an atomic section.
Adding a message may involve changes to multiple database documents,
and thus needs to be done in a transaction.  This makes add_message
(and, I think, the whole library) atomicity-safe: library callers only
needs to use atomic sections if they needs atomicity across multiple
library calls.
2011-09-24 20:00:28 -03:00
Austin Clements
7a8046ced8 lib: Add an API to find a message by filename.
notmuch_database_find_message_by_filename is mostly stolen from
notmuch_database_remove_message, so this patch also vastly simplfies
the latter using the former.

This API is also useful in its own right and will be used in a later
patch for eager maildir flag synchronization.
2011-09-24 20:00:28 -03:00
Austin Clements
bfe4555325 lib: Remove message document directly after removing the last file name.
Previously, notmuch_database_remove_message would remove the message
file name, sync the change to the message document, re-find the
message document, and then delete it if there were no more file names.
An interruption after sync'ing would result in a file-name-less,
permanently un-removable zombie message that would produce errors and
odd results in searches.  We could wrap this in an atomic section, but
it's much simpler to eliminate the round-about approach and just
delete the message document instead of sync'ing it if we removed the
last filename.
2011-09-23 21:50:39 -04:00
Austin Clements
e59cc0031f lib: Add support for nested atomic sections.
notmuch_database_t now keeps a nesting count and we only start a
transaction or commit for the outermost atomic section.

Introduces a new error, NOTMUCH_STATUS_UNBALANCED_ATOMIC.
2011-09-23 21:50:38 -04:00
Austin Clements
957f1ba3fc lib: Add notmuch_database_{begin,end}_atomic.
These operations translate into non-flushed Xapian transactions,
allowing arbitrary groups of database operations to be performed
atomically.
2011-09-23 21:50:38 -04:00
Pieter Praet
8bb6f7869c fix sum moar typos [comments in source code]
Various typo fixes in comments within the source code.

Signed-off-by: Pieter Praet <pieter@praet.org>

Edited-by: Carl Worth <cworth@cworth.org> Restricted to just
source-code comments, (and fixed fix of "descriptios" to "descriptors"
rather than "descriptions").
2011-06-23 15:58:39 -07:00
Austin Clements
206938ec9b Add a generic function to get a list of terms with some prefix.
Replace _notmuch_convert_tags with this and simplify
_create_filenames_for_terms_with_prefix.  This will also come in handy
shortly to get the message file name list.
2011-03-21 02:45:18 -04:00
Austin Clements
f3c1eebfaf Implement an internal generic string list and use it.
This replaces the guts of the filename list and tag list, making those
interfaces simple iterators over the generic string list.  The
directory, message filename, and tags-related code now build generic
string lists and then wraps them in specific iterators.  The real wins
come in later patches, when we use these for even more generic
functionality.

As a nice side-effect, this also eliminates the annoying dependency on
GList in the tag list.
2011-03-21 02:45:18 -04:00
Carl Worth
99cfa27030 Add support for folder-based searching.
A new "folder:" prefix in the query string can now be used to match
the directories in which mail files are stored.

The addition of this feature causes the recently added
search-by-folder tests to now pass.
2011-01-15 15:37:43 -08:00
Carl Worth
8bef9ba922 Tighten up a memory allocation.
Using the local talloc context ensures that the memory we are using
here will be freed shortly, (rather than hanging on for a long time
with the notmuch database object).
2011-01-15 15:37:43 -08:00
Carl Worth
d422dcf0a2 lib: Remove the notion of TAGS_INVALID
This rather ugly hack was recently obviated by the removal of the
notmuch_database_set_maildir_sync function. Now, clients must make
explicit calls to do any syncrhonization between maildir flags and
tags. So the library no longer needs to worry about doing inconsistent
synchronization while a message is only partially added.
2010-11-11 03:40:19 -08:00
Carl Worth
bb74e9dff8 lib: Rework interface for maildir_flags synchronization
Instead of having an API for setting a library-wide flag for
synchronization (notmuch_database_set_maildir_sync) we instead
implement maildir synchronization with two new library functions:

	notmuch_message_maildir_flags_to_tags
  and   notmuch_message_tags_to_maildir_flags

These functions are nicely documented here, (though the implementation
does not quite match the documentation yet---as plainly evidenced by
the current results of the test suite).
2010-11-11 03:40:19 -08:00
Michal Sojka
d9d3d3e6f0 Make maildir synchronization configurable
This adds group [maildir] and key 'synchronize_flags' to the
configuration file. Its value enables (true) or diables (false) the
synchronization between notmuch tags and maildir flags. By default,
the synchronization is disabled.
2010-11-10 13:09:32 -08:00
Michal Sojka
088801a14a Maildir synchronization
This patch allows bi-directional synchronization between maildir
flags and certain tags. The flag-to-tag mapping is defined by flag2tag
array.

The synchronization works this way:

1) Whenever notmuch new is executed, the following happens:
   o New messages are tagged with configured new_tags.
   o For new or renamed messages with maildir info present in the file
     name, the tags defined in flag2tag are either added or removed
     depending on the flags from the file name.

2) Whenever notmuch tag (or notmuch restore) is executed, a new set of
   flags based on the tags is constructed for every message and a new
   file name is prepared based on the old file name but with the new
   flags. If the flags differs and the old message was in 'new'
   directory then this is replaced with 'cur' in the new file name. If
   the new and old file names differ, the file is renamed and notmuch
   database is updated accordingly.

   The rename happens before the database is updated. In case of crash
   between rename and database update, the next run of notmuch new
   brings the database in sync with the mail store again.
2010-11-10 13:09:31 -08:00
Michal Sojka
b65a783195 lib: Return added message even if it already was in the database 2010-11-10 13:09:31 -08:00
Carl Worth
d064bd696c lib: Eliminate some redundant includes of xapian.h
Most files including this already include database-private.h which
includes xapian.h already.
2010-11-01 23:24:40 -07:00
Carl Worth
67c3bc9db4 lib: Add some missing static qualifiers.
These various functions and data are all used only locally, so should
be marked static. Ensuring we get these right will avoid us accidentally
leaking unintended symbols through the library interface.
2010-11-01 21:58:43 -07:00
Carl Worth
029830c1f1 lib: Fix use-after-free bug.
Thanks to the new git-based test suite, it's easy to run the whole
test suite in valgrind, (simply "make test OPTIONS="--valgrind"), and
doing so showed this obvious use-after-free bug, (triggered by the
thread-order tests).
2010-09-20 22:01:52 +00:00
Carl Worth
7b78eb4af6 Add support (and tests) for messages with really long message IDs.
Scott Henson reported an internal error that occurred when he tried to
add a message that referenced another message with a message ID well
over 300 characters in length. The bug here was running into a Xapian
limit for the length of metadata key names, (which is even more
restrictive than the Xapian limit for the length of terms).

We fix this by noticing long message ID values and instead using a
message ID of the form "notmuch-sha1-<sha1_sum_of_message_id>". That
is, we use SHA1 to generate a compressed, (but still unique), version
of the message ID.

We add support to the test suite to exercise this fix. The tests add a
message referencing the long message ID, then add the message with the
long message ID, then finally add another message referencing the long
ID. Each of these tests exercise different code paths where the
special handling is implemented.

A final test ensures that all three messages are stitched together
into a single thread---guaranteeing that the three code paths all act
consistently.
2010-06-04 13:35:07 -07:00
Carl Worth
98845fdbb2 Avoid database corruption by not adding partially-constructed mail documents.
Previously we were using Xapian's add_document to allocate document ID
values for notmuch_message_t objects.  This had the drawback of adding
a partially constructed mail document to the database. If notmuch was
subsequently interrupted before fully populating this document, then
later runs would be quite confused when seeing the partial documents.

There are reports from the wild of people hitting internal errors of
the form "Message ... has no thread ID" for example, (which is
currently an unrecoverable error).

We fix this by manually allocating document IDs without adding
documents. With this change, we never call Xapian's add_document
method, but only replace_document with either the current document ID
of a message or a new one that we have allocated.
2010-06-04 10:16:53 -07:00
Carl Worth
9ef68f1444 lib: Audit all notmuch_database call for Xapian exception handling.
Our current approach is for top-level entry poitns in the library
to have try/catch blocks that catch any Xapian exception and print
a message. Add a few missing blocks and fix up the documentation.
2010-04-24 07:27:50 -07:00
Carl Worth
14073b8851 lib: Remove condition regarding a NULL parent_thread_id.
A recent change guaranteed that a message ID can never be resolved
to a NULL thread ID, so we don't need this extra case.
2010-04-12 15:54:03 -07:00
Carl Worth
071022c253 lib: Always add reference terms to the database.
Previously, we were only adding the reference terms for cases where
the referenced message did not yet exist in the database. For thread
presentation, it's useful to have the connection information provided
by the references, even when the messages are present. So add this
term unconditionally.
2010-04-12 15:45:40 -07:00
Carl Worth
328626d0fd lib: Document the metadata stored within the Xapian database.
We are currently storing "version", "last_thread_id", and
"thread_id_*" values so document how each of these are used.
2010-04-12 15:15:14 -07:00
Carl Worth
af49741228 lib: Fix line-wrapping in _notmuch_database_link_message.
This function had some excessively long lines due to nested
expressions. It's simple enough to un-nest these and have readable
line lengths.
2010-04-12 14:41:34 -07:00
Carl Worth
f8dc5c08e4 lib: Fix internal documentation of _notmuch_database_link_message
This function was recently modified, (to include a metadata lookup for
a message's thread ID before looking for parent/child thread IDs), but
the documentation wasn't updated. Fix that.
2010-04-12 14:35:25 -07:00
Carl Worth
5c20bdf035 lib: Simplify code flow in _resolve_message_id_to_thread_id
There are two primary cases in this function, (the message exists in
the database or it does not). Previously the code for these two cases
was split and intermingled with goto-spaghetti connections.
2010-04-12 14:29:36 -07:00
Carl Worth
e9bb90ba2c lib: Fix internal documentation of _resolve_message_id_to_thread_id
We no longer return NULL, but instead generate a new thread ID for
messages that we haven't seen yet.
2010-04-12 14:19:15 -07:00
James Westby
40ea73cf05 Store thread ids for messages that we haven't seen yet
This allows us to thread messages even when we receive them out of
order, or never receive the root.

The thread ids for messages that aren't present but are referred to are
stored as metadata in the database and then retrieved if we ever get
that message.

When determining the thread id for a message we also check for this
metadata so that we can thread descendants of a message together before
we receive it.

Edited by Carl Worth <cworth@cworth.org>: Split this portion of the
commit from the earlier-applied portion adding test cases.
2010-04-12 14:11:57 -07:00
Carl Worth
e3046c688b Add is:<tag> as a synonym for tag:<tag> in search terms.
I like the readability of this, it provides compatibility with people
trained in this syntax by sup, and it even saves one character.
2010-03-09 16:03:58 -08:00
Carl Worth
4e5d2f22db lib: Rename iterator functions to prepare for reverse iteration.
We rename 'has_more' to 'valid' so that it can function whether
iterating in a forward or reverse direction. We also rename
'advance' to 'move_to_next' to setup parallel naming with
the proposed functions 'move_to_first', 'move_to_last', and
'move_to_previous'.
2010-03-09 09:22:29 -08:00
Carl Worth
e0a8dee8bc Fix printf for when uint64_t != unsigned long long int
Thanks to Michal Sojka <sojkam1@fel.cvut.cz> for pointing out the
correct fix, which I verified in the freely-available WG14/N1124 draft
(from the C99 working group) which is available here:

http://www.open-std.org/JTC1/SC22/wg14/www/docs/n1124.pdf
2010-02-09 11:14:16 -08:00
Carl Worth
9439b217c3 Switch from random to sequential thread identifiers.
The sequential identifiers have the advantage of being guaranteed to
be unique (until we overflow a 64-bit unsigned integer), and also take
up half as much space in the "notmuch search" output (16 columns
rather than 32).

This change also has the side effect of fixing a bug where notmuch
could block on /dev/random at startup (waiting for some entropy to
appear). This bug was hit hard by the test suite, (which could easily
exhaust the available entropy on common systems---resulting in large
delays of the test suite).
2010-02-09 11:14:11 -08:00
Carl Worth
c340c1bd11 notmuch new: Print upgrade progress report as a percentage.
Previously we were printing a number of messages upgraded so far. The
original motivation for this was to accurately reflect the fact that
there are two passes, (so each message is processed twice and it's not
accurate to represent with a single count). But as it turns out, the
second pass takes zero time (relatively speaking) so we're still not
accounting for it.

If nothing else, the percentage-based reporting makes for a cleaner
API for the progress_notify function.
2010-01-09 17:38:23 -08:00
Carl Worth
d12801c8b4 lib: Split the database upgrade into two phases for safer operation.
The first phase copies data from the old format to the new format
without deleting anything. This allows an old notmuch to still use the
database if the upgrade process gets interrupted. The second phase
performs the deletion (after updating the database version number). If
the second phase is interrupted, there will be some unused data in the
database, but it shouldn't cause any actual harm.
2010-01-09 11:13:12 -08:00
Carl Worth
5fe5e802ab lib: Delete stale timestamp documents during database upgrade.
Once we move the timestamp to the new directory document, we don't
need the old one anymore.
2010-01-08 09:57:09 -08:00
Carl Worth
1c86b48329 notmuch new: Fix progress notification on database upgrade.
This was firing continuously rather than just once per second as
intended.
2010-01-07 21:24:44 -08:00
Carl Worth
909f52bd8c lib: Implement versioning in the database and provide upgrade function.
The recent support for renames in the database is our first time
(since notmuch has had more than a single user) that we have a
database format change. To support smooth upgrades we now encode a
database format version number in the Xapian metadata.

Going forward notmuch will emit a warning if used to read from a
database with a newer version than it natively supports, and will
refuse to write to a database with a newer version.

The library also provides functions to query the database format
version:

	notmuch_database_get_version

to ask if notmuch wants a newer version than that:

	notmuch_database_needs_upgrade

and a function to actually perform that upgrade:

	notmuch_database_upgrade
2010-01-07 18:26:31 -08:00
Carl Worth
807aef93d3 Prefer READ_ONLY consistently over READONLY.
Previously we had NOTMUCH_DATABASE_MODE_READ_ONLY but
NOTMUCH_STATUS_READONLY_DATABASE which was ugly and confusing. Rename
the latter to NOTMUCH_STATUS_READ_ONLY_DATABASE for consistency.
2010-01-07 10:29:05 -08:00
Carl Worth
f93b7218c3 lib: Consolidate checks for read-only database.
Previously, many checks were deep in the library just before a cast
operation. These have now been replaced with internal errors and new
checks have instead been added at the beginning of all top-levelentry
points requiring a read-write database.

The new checks now also use a single function for checking and
printing the error message. This will give us a convenient location to
extend the check, (such as based on database version as well).
2010-01-07 10:19:44 -08:00
Carl Worth
6ed606c19e lib: Clarify internal documentation of _notmuch_database_filename_to_direntry
The original wording made it sound like this function was just doing
some string manipulation. But this function actually creates new
directory documents as a side effect. So make that explicit in its
documentation.
2010-01-07 09:31:58 -08:00
Carl Worth
4b418343f6 lib: Indicate whether notmuch_database_remove_message removed anything.
Similar to the return value of notmuch_database_add_message, we now
enhance the return value of notmuch_database_remove_message to
indicate whether the message document was entirely removed (SUCCESS)
or whether only this filename was removed and the document exists
under other filenamed (DUPLICATE_MESSAGE_ID).
2010-01-06 10:32:06 -08:00
Carl Worth
3f32fd8a1c Add missing comment for NOTMUCH_STATUS_READONLY_DATABASE.
And adjust the string representation of the same to match.
2010-01-06 10:32:06 -08:00
Carl Worth
d807e28f43 lib: Implement new notmuch_directory_t API.
This new directory ojbect provides all the infrastructure needed to
detect when files or directories are deleted or renamed. There's still
code needed on top of this (within "notmuch new") to actually do that
detection.
2010-01-06 10:32:06 -08:00
Carl Worth
f11aaa3678 database: Add new, public notmuch_database_remove_message
This will allow applications to support the removal of messages, (such
as when a file is deleted from the mail store). No removal support is
provided yet in commands such as "notmuch new".
2010-01-06 10:32:06 -08:00
Carl Worth
44a74912c7 database: Add new find_doc_ids_for_term interface.
The existing find_doc_ids function is convenient when the caller
doesn't want to be bothered constructing a term. But when the caller
*does* have the term already, that interface is just wasteful. So we
export a lower-level interface that maps a pre-constructed term to a
document-ID iterators.
2010-01-06 10:32:06 -08:00
Carl Worth
d7e5f5827e database: Make find_unique_doc_id enforce uniqueness (for a debug build)
Catching any violation of this unique-ness constraint is very much in
line with similar, existing INTERNAL_ERROR cases.
2010-01-06 10:32:06 -08:00
Carl Worth
498edff503 database: Abstract _filename_to_direntry from _add_message
The code to map a filename to a direntry is something that we're going
to want in a future _remove_message function, so put it in a new
function _notmuch_database_filename_to_direntry .
2010-01-06 10:32:05 -08:00
Carl Worth
1376a90db6 database: Allowing storing multiple filenames for a single message ID.
The library interface is unchanged so far, (still just
notmuch_database_add_message), but internally, the old
_set_filename function is now _add_filename instead.
2010-01-06 10:32:05 -08:00
Carl Worth
6ca6c089e9 database: Store mail filename as a new 'direntry' term, not as 'data'.
Instead of storing the complete message filename in the data portion
of a mail document we now store a 'direntry' term that contains the
document ID of a directory document and also the basename of the
message filename within that directory. This will allow us to easily
store multple filenames for a single message, and will also allow us
to find mail documents for files that previously existed in a
directory but that have since been deleted.
2010-01-06 10:32:05 -08:00
Carl Worth
84742d86ab database: Split _find_parent_id into _split_path and _find_directory_id
Some pending commits want the _split_path functionality separate from
mapping a directory to a document ID. The split_path function now
returns the basename as well as the directory name.
2010-01-06 10:32:05 -08:00
Carl Worth
4c1cca888f database: Store directory path in 'data' of directory documents.
We're planning to have mail documents refer to directory documents for
the path of the containing directory. To support this, we need the
path in the data, (since the path in the 'directory' term can be
irretrievable as it will be the SHA1 sum of the path for a very long
path).
2010-01-06 10:32:05 -08:00
Carl Worth
406ec4b15d database: Export _notmuch_database_find_parent_id for internal use.
We'll soon have mail documents referring to their parent directory's
directory documents, so we'll need access to _find_parent_id in files
such as message.cc.
2010-01-06 10:32:05 -08:00
Carl Worth
e890b0cf40 database: Store the parent ID for each directory document.
Storing the document ID of the parent of each directory document will
allow us to find all child-directory documents for a given directory
document. We will need this in order to detect directories that have
been removed from the mail store, (though we aren't yet doing this).
2010-01-06 10:32:05 -08:00
Carl Worth
851c97aed7 database: Rename internal directory value from XTIMESTAMP to XDIRECTORY.
The recent change from storing absolute paths to relative paths means
that new directory documents will already be created, (and the old
ones will just linger stale in the database). Given that, we might as
well put a clean name on the term in the new documents, (and no real
flag day is needed).
2010-01-06 10:32:05 -08:00
Carl Worth
154bf7ac67 database: Store directory paths as relative, not absolute.
We were already storing relative mail filenames, so this is consistent
with that. Additionally, it means that directory documents remain
valid even if the database is relocated within its containing
filesystem.
2010-01-06 10:32:05 -08:00
Carl Worth
9257622da8 lib: Document that the filename is stored in the 'data' of a mail document
Our database schema documentation previously didn't give any
indication of where this most essential piece of information is
stored.
2010-01-06 10:32:05 -08:00
Carl Worth
50ae83a17f lib: Rename set/get_timestamp to set/get_directory_mtime.
I've been suitably scolded by Keith for doing a premature
generalization that ended up just making the documentation more
convoluted. Fix that.
2010-01-06 10:32:05 -08:00
Carl Worth
ba12bf1f26 lib: Abstract the extraction of a relative path from set_filename
We'll soon be having multiple entry points that accept a filename
path, so we want common code for getting a relative path from a
potentially absolute path.
2010-01-06 10:32:05 -08:00
Fernando Carrijo
db68eea013 Nuke the remainings of _notmuch_message_add_thread_id.
The function _notmuch_message_add_thread_id has been removed
from the private interface of notmuch. There's no reason for
one to keep a declaration of its prototype in the code base.
Also, lets update a commentary that referenced that function
and escaped from previous scrutiny.

Signed-off-by: Fernando Carrijo <fcarrijo@yahoo.com.br>
2009-12-09 12:09:55 -08:00
Jan Janak
c3c52e464b notmuch: New function to retrieve all tags from the database.
This patch adds a new function called notmuch_database_get_all_tags
which can be used to obtain a list of all tags from the database
(in other words, the list contains all tags from all messages). The
function produces an alphabetically sorted list.

To add support for the new function, we rip the guts off of
notmuch_message_get_tags and put them in a new generic function
called _notmuch_convert_tags. The generic function takes a
Xapian::TermIterator as argument and uses the iterator to find tags.
This makes the function usable with different Xapian objects.

Function notmuch_message_get_tags is then reimplemented to call the
generic function with message->doc.termlist_begin() as argument.

Similarly, we implement notmuch_message_database_get_all_tags, the
function calls the generic function with db->xapian_db->allterms_begin()
as argument.

Finally, notmuch_database_get_all_tags is exported through
lib/notmuch.h

Signed-off-by: Jan Janak <jan@ryngle.com>
2009-11-26 07:01:52 -08:00
Carl Worth
793cbf8049 Add rudimentary date-based search.
The rudimentary aspect here is that the date ranges are specified with
UNIX timestamp values (number of seconds since 1970-01-01 UTC). One
thing that can help here is using the date program to determins
timestamps, such as:

	$(date +%s -d 2009-10-01)..$(date +%s)

Long-term, we'll probably need to do our own query parsing to be able
to support directly-specified dates and also relative expressions like
"since:'2 months ago'".
2009-11-23 17:17:08 +01:00
Chris Wilson
3e4ab913db lib/database.cc: coding style
Carl claims he must have been distracted when he wrote this...

Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
2009-11-22 05:26:59 +01:00
Carl Worth
5d56e931b9 add_message: Use sha-1 in place of overly long message ID.
Since Xapian has a limit on the maximum length of a term, we have
to check for that before trying to add the message ID as a term.

This fixes the bug reported by Mike Hommey here:

	<20091120132625.GA19246@glandium.org>

I've also constructed 20 files with a range of message ID lengths
centered around the Xapian term-length limit which I'll use to seed a
new test suite soon.
2009-11-22 04:03:49 +01:00
Carl Worth
f336ee034b get_timestamp: Ensure that return value is 0 in case of exception.
Just to be on the safe side of things.
2009-11-22 03:55:39 +01:00
Carl Worth
e2341cbc09 Catch and optionally print about exception at database->flush.
If an earlier exception occurred, then it's not unexpected for the
flush to fail as well. So in that case, we'll silently catch the
exception. Otherwise, make some noise about things going wrong at the
time of flush.
2009-11-22 03:54:20 +01:00
Carl Worth
591f901241 Print information about where Xapian exception occurred.
Previously, our Xapian exception reports where identical so they
were hard to track down.
2009-11-22 03:51:35 +01:00
Carl Worth
637f99d8f3 Rename NOTMUCH_DATABASE_MODE_WRITABLE to NOTMUCH_DATABASE_MODE_READ_WRITE
And correspondingly, READONLY to READ_ONLY.
2009-11-21 22:10:18 +01:00
Chris Wilson
f379aa5284 Permit opening the notmuch database in read-only mode.
We only rarely need to actually open the database for writing, but we
always create a Xapian::WritableDatabase. This has the effect of
preventing searches and like whilst updating the index.

Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Acked-by: Carl Worth <cworth@cworth.org>
2009-11-21 22:04:49 +01:00
Carl Worth
3ae12b1e28 add_message: Re-fix handling of non-mail files.
More fallout from _get_header now returning "" for missing headers.

The bug here is that we would no longer detect that a file is not an
email message and give up on it like we should.

And this time, I actually audited all callers to
notmuch_message_get_header, so hopefully we're done fixing this
bug over and over.
2009-11-20 21:46:37 +01:00
Carl Worth
656e4c413d notmuch_database_add_message: Add missing error-value propagation.
Thanks to Mike Hommey for doing the analysis that led to noticing that
this was missing.
2009-11-20 21:02:11 +01:00
Carl Worth
52292c5485 add_message: Properly handle missing Message-ID once again.
There's been a fair amount of fallout from when we changed
message_file_get_header from returning NULL to returning "" for
missing headers. This is yet more fallout from that, (where we were
accepting an empty message-ID rather than generating one like we want
to).
2009-11-20 19:36:01 +01:00
Ingmar Vanhassel
2ce25b93a7 Typsos 2009-11-18 03:21:36 -08:00
Carl Worth
fc3a3be337 linke_message: Avoid segfault when In-Reply-to header is empty.
This was recently introduced in commit:

	64c03ae97f

which was adding extra checks to avoid adding a self-referencing
message.

How many times am I going to fix a dumb regression like this and say
"we really need a test suite" before I actually sit down and write the
test suite?
2009-11-18 01:36:30 -08:00
Carl Worth
0da0131096 database: Make _parse_message_id static once again.
We had exposed this to the internal implementation for a short time,
(only while we had the silly code fetching In-Reply-To values from
message files instead of from the database). Make this private again
as it should be.
2009-11-17 18:50:13 -08:00
Carl Worth
c50891f449 database: Add "replyto" to the database schema documentation.
Maybe ths lack of this documentation is why I forgot we were actually
storing this and wrote the ugly code to fetch In-Reply-To from message
files rather than from the database.
2009-11-17 18:48:38 -08:00
Carl Worth
6e9fdf0abf database: Rename "ref" prefix name to "reference"
Which is more consistent with the XREFERENCE prefix used in the terms
in the database. Also remove some stale documentation describing the
removal of resolved references from the database (we no longer do
this).
2009-11-17 18:44:02 -08:00
Carl Worth
64c03ae97f add_message: Don't add any self-references to the database.
In our scheme it's illegal for any message to refer to itself, (nor
would it be useful for anything anyway). Cut these self-references off
at the source, before they trip up any internal errors.
2009-11-17 17:55:37 -08:00
Carl Worth
24a25ffba9 Remove the talloc_owner argument from create_for_message_id.
This function has only one caller, and that one caller was passing the
same value for both talloc_owner and the notmuch database. Dropping
the redundant argument simplifies the documentation of this function
considerably.
2009-11-17 17:42:32 -08:00
Carl Worth
12d3014d88 Fix broken commit.
Oops. I should have actually compiled before pushing.
2009-11-17 09:04:14 -08:00
Carl Worth
0dab6a2c1e add_message: Avoid a memory leak when user holds on to message return.
When this function was originally written, the 'message' object was
always destroyed locally, so I thought it would be good to use a NULL
talloc context to make it more obvious if there was any leak.

Since then, however, this function has been changed to optionally
return the added message, and in that case we *don't* free the message
locally, so let's let the database be the talloc context.
2009-11-17 08:50:14 -08:00
Carl Worth
9034e396b6 database: Fix a typo in a commit.
Nothing significant here, but we might as well not keep things
misspelled when we notice.
2009-11-15 20:23:27 -08:00
Carl Worth
9b1c6c250b Export _parse_message_id to the library implementation.
Not exported through the public interface, but the thread code is
going to want to be able to parse In-Reply-To headers so needs access
to this code.
2009-11-15 20:21:43 -08:00
Carl Worth
4d35c3544d Don't create "contact" terms in the database.
We never did export any interface to get at these, and when I went to
use these, I found them inadequate, (because I wanted to distinguish
address found in from: from those found in To:). Meanwhile, it was
easy enough to extract addresses with a search like:

	notmuch show tag:sent | grep ^To:

so the storage of contact terms was just wasting space. Stop that.
2009-11-12 09:38:24 -08:00
Carl Worth
305e76bc0a notmuch: Add a configuration system.
This will allow for things like the database path to be specified
without any cheesy NOTMUCH_BASE environment variable. It also will
allow "notmuch reply" to recognize the user's email address when
constructing a reply in order to do the right thing, (that is, to use
the user's address to which mail was sent as From:, and not to reply
to the user's own addresses).

With this change, the "notmuch setup" command is now strictly for
changing the configuration of notmuch. It no longer creates the
database, but instead instructs the user to call "notmuch new" to do
that.
2009-11-11 17:01:55 -08:00
Carl Worth
1465493210 libify: Move library sources down into lib directory.
A "make" invocation still works from the top-level, but not from
down inside the lib directory yet.
2009-11-09 16:24:03 -08:00
Renamed from database.cc (Browse further)