Commit graph

235 commits

Author SHA1 Message Date
Carl Worth
029830c1f1 lib: Fix use-after-free bug.
Thanks to the new git-based test suite, it's easy to run the whole
test suite in valgrind, (simply "make test OPTIONS="--valgrind"), and
doing so showed this obvious use-after-free bug, (triggered by the
thread-order tests).
2010-09-20 22:01:52 +00:00
Carl Worth
d64d0cc8d9 make install: Run ldconfig or install a DT_RUNPATH in binary as appropriate.
Various users were confused as to why they couldn't run notmuch
immediately after "make install", (with linker errors saying that
libnotmuch.so could not be found). The errors came from two different
causes:

1. The user had installed to a system library directory, but had not
   yet run ldconfig.

2. The user had installed to some non-system directory, and had not
   set the LD_LIBRARY_PATH variable.

With this change we fix both problems (on Linux) without the user
having to do anything additional. We first use ldconfig to find the
system library directories. If the user is installing to one of these,
then we run ldconfig as part of "make install".

For case (2) we use the -rpath and --enable-new-dtags linker options
to install a DT_RUNPATH entry in the binary. This entry tells the
dynamic linker where to find libnotmuch. Without the
--enable-new-dtags option only a DT_RPATH option would be installed,
(which has the drawback of not allowing any override with the
LD_LIBRARY_PATH variable).

Distributions (such as Debian and Fedora) don't want to see binaries
packaged with a DT_RPATH or DT_RUNPATH entry. This should be avoided
automatically as long as the packages install to standard locations,
(such as /usr/lib).
2010-06-04 16:52:56 -07:00
Carl Worth
7b78eb4af6 Add support (and tests) for messages with really long message IDs.
Scott Henson reported an internal error that occurred when he tried to
add a message that referenced another message with a message ID well
over 300 characters in length. The bug here was running into a Xapian
limit for the length of metadata key names, (which is even more
restrictive than the Xapian limit for the length of terms).

We fix this by noticing long message ID values and instead using a
message ID of the form "notmuch-sha1-<sha1_sum_of_message_id>". That
is, we use SHA1 to generate a compressed, (but still unique), version
of the message ID.

We add support to the test suite to exercise this fix. The tests add a
message referencing the long message ID, then add the message with the
long message ID, then finally add another message referencing the long
ID. Each of these tests exercise different code paths where the
special handling is implemented.

A final test ensures that all three messages are stitched together
into a single thread---guaranteeing that the three code paths all act
consistently.
2010-06-04 13:35:07 -07:00
Carl Worth
98845fdbb2 Avoid database corruption by not adding partially-constructed mail documents.
Previously we were using Xapian's add_document to allocate document ID
values for notmuch_message_t objects.  This had the drawback of adding
a partially constructed mail document to the database. If notmuch was
subsequently interrupted before fully populating this document, then
later runs would be quite confused when seeing the partial documents.

There are reports from the wild of people hitting internal errors of
the form "Message ... has no thread ID" for example, (which is
currently an unrecoverable error).

We fix this by manually allocating document IDs without adding
documents. With this change, we never call Xapian's add_document
method, but only replace_document with either the current document ID
of a message or a new one that we have allocated.
2010-06-04 10:16:53 -07:00
Carl Worth
361b9d4bd9 Fix misnamed function in internal documentation.
The documentation for several functions mentioned
_notmuch_message_set_sync which doesn't exist. Fix these to reference
_notmuch_message_sync instead.
2010-06-04 09:54:46 -07:00
Tomas Carnecky
a54cecfc8e Add support for the Solaris platform
Like on Mac OS X, the linker doesn't automatically resolve dependencies.

Signed-off-by: Tomas Carnecky <tom@dbservice.com>
2010-06-03 18:17:03 -07:00
Dirk Hohndel
a258cb32b3 Fix SEGV in _thread_cleanup_author if author ends with ', '
Admittedly, an author name ending in ',' guarantees this is spam, and
indeed this was triggered by a spam email, but that doesn't mean we
shouldn't handle this case correctly.
We now check that there is actually a component of the name (presumably
the first name) after the comma in the author name.

Signed-off-by: Dirk Hohndel <hohndel@infradead.org>
2010-04-27 16:34:27 -07:00
Carl Worth
0fb28c65f2 lib: Increment library version to 1.1.0
For the addition of the new NOTMUCH_SORT_UNSORTED value.
2010-04-27 02:02:14 -07:00
Carl Worth
c210d5632e lib: Re-implement moving of thread authors.
Just before releasing 0.3 we received reports of crashes that were
bisected to the commit adding thread-author moving. Sure enough,
valgrind pointed to buffer overruns in _thread_move_matched_author.

Rather than trying to make sense of all the by strncpy, strchr, +1,
and +2 of that code, I reimplemented thread-author ordering with a
pair of hash tables and an array.

Valgrind is at least happy now on the test cases it was complaining
about previously.
2010-04-27 01:48:03 -07:00
Dirk Hohndel
5b8b0377cb Make Received: header special in notmuch_message_file_get_header
With this patch the Received: header becomes special in the way
we treat headers - this is the only header for which we concatenate
all the instances we find (instead of just returning the first one).

This will be used in the From guessing code for replies as we need to
be able to walk ALL of the Received: headers in a message to have a
good chance to guess which mailbox this email was delivered to.

Signed-off-by: Dirk Hohndel <hohndel@infradead.org>
2010-04-26 14:44:06 -07:00
Dirk Hohndel
cd19671f51 Simple attempt to display author names in a friendlier way
This patch only addresses the typical Outlook/Exchange case
where we have "Last, First" <first.last@company.com> or
"Last, First MI" <first.mi.last@company.com>.

In the future we should be more fexible as to the formats
we recognize, but for now we address this one as it is the
Exchange default setting and therefore the most common one.

Signed-off-by: Dirk Hohndel <hohndel@infradead.org>
2010-04-26 11:45:29 -07:00
Dirk Hohndel
26d8d960ee Reorder displayed names of thread authors
When displaying threads as result of a search it makes sense to list those
authors first who match the search. The matching authors are separated from the
non-matching ones with a '|' instead of a ','

Imagine the default "+inbox" query. Those mails in the thread that
match the query are actually "new" (whatever that means). And some
people seem to think that it would be much better to see those author
names first. For example, imagine a long and drawn out thread that once
was started by me; you have long read the older part of the thread and
removed the inbox tag. Whenever a new email comes in on this thread,
prior to this patch the author column in the search display will first show
"Dirk Hohndel" - I think it should first show the actual author(s) of the new
mail(s).

Signed-off-by: Dirk Hohndel <hohndel@infradead.org>
2010-04-26 11:45:00 -07:00
Dirk Hohndel
57561414d7 Add authors member to message
message->authors contains the author's name (as we want to print it)
get / set methods are declared in notmuch-private.h

Signed-off-by: Dirk Hohndel <hohndel@infradead.org>
2010-04-26 11:44:49 -07:00
Carl Worth
138fd38afe lib: Ensure notmuch_query_search_messages returns NULL on an exception.
Previously, this function may have segfaulted immediately after
reporting the exception.
2010-04-24 07:27:50 -07:00
Carl Worth
e3e0e26806 lib: Document that notmuch_query_count_messages may return 0 if an exception occurs
This isn't a behavioral change---just a calrification in the documentation.
2010-04-24 07:27:50 -07:00
Carl Worth
9ef68f1444 lib: Audit all notmuch_database call for Xapian exception handling.
Our current approach is for top-level entry poitns in the library
to have try/catch blocks that catch any Xapian exception and print
a message. Add a few missing blocks and fix up the documentation.
2010-04-24 07:27:50 -07:00
Carl Worth
3dbef312fb lib: Audit calls to notmuch_message_get_header to handle NULL return
Sebastian Spaeth reported [*] a segfault within libnotmuch when
running notmuch operations while an asyncronous offlineimap job had
removed some files from the mail store. Avoid this by handling all
cases where notmuch_message_get_header could return NULL.

[*] See message id:87d3xqti3o.fsf@SSpaeth.de on notmuch@notmuchmail.org
2010-04-24 06:50:04 -07:00
Carl Worth
7c421b87b0 lib: Simplify code to set subject from matched message.
Simply moving the code from _add_matched_message to a new
_set_subject_from_message function.
2010-04-24 06:50:04 -07:00
Carl Worth
21965718a5 Revert "thread: Simplify code for assigning the subject."
This reverts commit 36e4459a32.

With the two previous reverts, this fixes the recent message-sorting
regression, so the test suite now passes again.
2010-04-22 14:01:41 -07:00
Carl Worth
a109966080 Revert "thread: Fix sort of search when constructing threads."
This reverts commit f43990ce13.
2010-04-22 14:00:33 -07:00
Carl Worth
6a0cba4ae0 Revert "thread: Removed unsed sort argument from _thread_add_matched_message"
This reverts commit 7fb56f9dc5.
2010-04-22 14:00:17 -07:00
Carl Worth
7fb56f9dc5 thread: Removed unsed sort argument from _thread_add_matched_message
The reworked solution for naming a thread based on the subject of
oldest/newest matching message no longer needs this argument.
2010-04-21 17:05:16 -07:00
Sebastian Spaeth
aadb15a002 query.cc: allow to return query results unsorted
Previously, we always sorted the returned results by some string value,
(newest-to-oldest by default), however in some cases (as when applying
tags to a search result) we are not interested in any special order.

This introduces a NOTMUCH_SORT_UNSORTED value that does just that. It is
not used at the moment anywhere in the code.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2010-04-21 16:06:05 -07:00
Carl Worth
f43990ce13 thread: Fix sort of search when constructing threads.
The thread-naming feature depends on the matched messages being passed
down in a precise order, (the order of the top-level search). We fix
the feature by passing that sort order down.
2010-04-21 15:52:28 -07:00
Carl Worth
36e4459a32 thread: Simplify code for assigning the subject.
We know that matched messages are always added in order, so we can
always just grab the subject from the first message. This is the same
approach that was used previously in _thread_add_message. That is, the
recent feature of renaming a thread based on the subject of the
"first" matched message is as simple as moving the subject assignment
from _thread_add_message to _thread_add_matched_message.
2010-04-21 15:06:02 -07:00
Jesse Rosenthal
4971b85641 Name thread based on matching msgs instead of first msg.
At the moment all threads are named based on the name of the first message
in the thread. However, this can cause problems if people either start
new threads by replying-all (as unfortunately, many out there do) or
change the subject of their mails to reflect a shift in a thread on a
list.

This patch names threads based on (a) matches for the query, and (b) the
search order. If the search order is oldest-first (as in the default
inbox) it chooses the oldest matching message as the subject. If the
search order is newest-first it chooses the newest one.

Reply prefixes ("Re: ", "Aw: ", "Sv: ", "Vs: ") are ignored
(case-insensitively) so a Re: won't change the subject.

Note that this adds a "sort" argument to _notmuch_thread_create and
_thread_add_matched_message, so that when constructing the thread we can
be aware of the sort order.

Signed-off-by: Jesse Rosenthal <jrosenthal@jhu.edu>
2010-04-21 14:56:53 -07:00
Carl Worth
c48dcc302c lib: search_threads: Fix nested search to handle original search of "*"
When constructing a thread, we usually run a nested query to find all
messages in the thread that match the original search string. However,
we need to have special-case handling of an original search string of
"*" now that that is a supported means of specifying all messages.

The special-case ends up bein quite simple---we do less work, (just
skipping the nested search since we know that all messages must
match). I had been wanting to write this identical code to more
efficiently handle "notmuch search thread:<foo>" which was previously
running two identical searches. So that case is now more efficient as
well.
2010-04-15 14:54:40 -07:00
Carl Worth
72ea1b71c6 Makefile: Add library version information on OS X.
This encodes the library version into the library, where the linking
binary can pick it up, and the linker can even enforce mismatches in
the minor release, (such as linking a binary against version 1.2 and
then attempting to run it against version 1.1).
2010-04-14 16:18:19 -07:00
Carl Worth
1036867897 Makefile: Fix library linking command for OS X
I'm not sure which system Aaron used, but on the machine I have access
to, (Darwin 8.11.0), the -shared and -dylib_install_name options are
not recognized. Instead I use -dynamic_lib and -install_name as
documented here:

http://www.finkproject.org/doc/porting/shared.php
2010-04-14 16:16:05 -07:00
Aaron Ecay
8c8079a8b1 Add infrastructure for building shared library on OS X.
This patch adds a configure check for OS X (actually Darwin),
and sets up the Makefiles to build a proper shared library on
that platform.

Signed-off-by: Aaron Ecay <aaronecay@gmail.com>
2010-04-14 16:10:27 -07:00
Carl Worth
f206408358 Makefile: Move compat sources from the client code to the library.
Since the library code needs these as well.
2010-04-14 16:03:18 -07:00
martin f. krafft
449a418c65 Do not segfault on empty mime parts
notmuch previously unconditionally checked mime parts for various
properties, but not for NULL, which is the case if libgmime encounters
an empty mime part.

Upon encounter of an empty mime part, the following is printed to
stderr (the second line due to my patch):

  (process:17197): gmime-CRITICAL **: g_mime_message_get_mime_part: assertion `GMIME_IS_MESSAGE (message)' failed
  Warning: Not indexing empty mime part.

This is probably a bug that should get addressed in libgmime, but for
not, my patch is an acceptable workaround.

Signed-off-by: martin f. krafft <madduck@madduck.net>
2010-04-13 08:49:06 -07:00
Michael Forney
9ddde6eb14 Fix typo in notmuch.h documentation regarding database open modes
Reviewed-by: Carl Worth <cworth@cworth.org>:

The original proposal for having different open modes used the name
WRITABLE. I didn't like that name, (easy to misspell as WRITEABLE even
for native English speakers). So we renamed it to READ_WRITE
immediately, but apparently some of the documentation held the old
name for a while.
2010-04-13 08:39:10 -07:00
Carl Worth
14073b8851 lib: Remove condition regarding a NULL parent_thread_id.
A recent change guaranteed that a message ID can never be resolved
to a NULL thread ID, so we don't need this extra case.
2010-04-12 15:54:03 -07:00
Carl Worth
071022c253 lib: Always add reference terms to the database.
Previously, we were only adding the reference terms for cases where
the referenced message did not yet exist in the database. For thread
presentation, it's useful to have the connection information provided
by the references, even when the messages are present. So add this
term unconditionally.
2010-04-12 15:45:40 -07:00
Carl Worth
328626d0fd lib: Document the metadata stored within the Xapian database.
We are currently storing "version", "last_thread_id", and
"thread_id_*" values so document how each of these are used.
2010-04-12 15:15:14 -07:00
Carl Worth
af49741228 lib: Fix line-wrapping in _notmuch_database_link_message.
This function had some excessively long lines due to nested
expressions. It's simple enough to un-nest these and have readable
line lengths.
2010-04-12 14:41:34 -07:00
Carl Worth
f8dc5c08e4 lib: Fix internal documentation of _notmuch_database_link_message
This function was recently modified, (to include a metadata lookup for
a message's thread ID before looking for parent/child thread IDs), but
the documentation wasn't updated. Fix that.
2010-04-12 14:35:25 -07:00
Carl Worth
5c20bdf035 lib: Simplify code flow in _resolve_message_id_to_thread_id
There are two primary cases in this function, (the message exists in
the database or it does not). Previously the code for these two cases
was split and intermingled with goto-spaghetti connections.
2010-04-12 14:29:36 -07:00
Carl Worth
e9bb90ba2c lib: Fix internal documentation of _resolve_message_id_to_thread_id
We no longer return NULL, but instead generate a new thread ID for
messages that we haven't seen yet.
2010-04-12 14:19:15 -07:00
James Westby
40ea73cf05 Store thread ids for messages that we haven't seen yet
This allows us to thread messages even when we receive them out of
order, or never receive the root.

The thread ids for messages that aren't present but are referred to are
stored as metadata in the database and then retrieved if we ever get
that message.

When determining the thread id for a message we also check for this
metadata so that we can thread descendants of a message together before
we receive it.

Edited by Carl Worth <cworth@cworth.org>: Split this portion of the
commit from the earlier-applied portion adding test cases.
2010-04-12 14:11:57 -07:00
Carl Worth
e100871981 lib: Handle "*" as a query string to match all messages.
This seems like a generally useful thing to support, (but the previous
support through an empty string was not convenient for some users,
(such as the command-line client).
2010-04-09 17:43:58 -07:00
Dirk Hohndel
4563f669ca fix obvious cut and paste error
the wrong variable is checked for success of an allocation

Signed-off-by: Dirk Hohndel <hohndel@infradead.org>
2010-04-06 18:55:56 -07:00
Dirk Hohndel
a48f368778 fix notmuch_message_file_get_header
fix notmuch_message_file_get_header to always return the first instance
 of the header you are looking for

Signed-off-by: Dirk Hohndel <hohndel@infradead.org>
2010-04-06 18:47:28 -07:00
Carl Worth
ae9d67fd81 Avoid needlessly linking final notmuch binary against libXapian.
The libnotmuch.so library already does, so we don't need to do
it again. (Thanks to a Debian debhelper warning for pointing this
out.)
2010-04-06 18:30:43 -07:00
Carl Worth
14e98e454e configure: Add support for a --includedir option
Very similar to the existing --libdir option.
2010-04-06 14:42:09 -07:00
Carl Worth
f89b3d16db Makefiles: Eliminate the useless quiet_* functions.
With the original quiet function, there's an actual purpose (hiding
excessively long compiler command lines so that warnings and errors
from the compiler can be seen).

But with things like quiet_symlink there's nothing quieter. In fact
"SYMLINK" is longer than "ln -sf". So all this is doing is hiding the
actual command from the user for no real benefit.

The only actual reason we implemented the quiet_* functions was to be
able to neatly right-align the command name and left-align the arguments.

Let's give up on that, and just left-align everything, simplifying the
Makefiles considerably. Now, the only instances of a captialized command
name in the output is if there's some actually shortening of the command
itself.
2010-04-06 14:36:31 -07:00
Carl Worth
da2403c310 RELEASING: Add this file describing the steps to make a release.
These steps might be changing a bit as we work on making the initial
0.1 release.
2010-04-05 15:29:54 -07:00
David Edmondson
d3884a5984 Makefile.local: Automatically use makefile mode
We add a magic line to the beginning of each Makefile.local file to
help the editor know that it should use makefile mode for editing the
file, (even though the filename isn't exactly "Makefile").

Edited-by: Carl Worth <cworth@cworth.org>: Expand treatment from
emacs/Makefile.local to each instance of Makefile.local.
2010-04-03 12:31:49 -07:00
Carl Worth
f689c83af4 Compile a static notmuch binary (but only install the shared version)
The idea here is to allow a new user of notmuch to be able to run
notmuch immediately after compiling, (without having to install
the shared library first). This also ensures that the test suite
tests the locally compiled library, and not whatever installled
version of the library the dynamic linker happens to find.
2010-04-01 15:03:40 -07:00
Michal Sojka
b884ab2ef1 Makefile: Create include directory when installing headers
When I wanted to create a debian package from the current master, make
install failed because of non-existent include directory. This patch
fixes this minor issue.
2010-04-01 05:13:21 -07:00
Carl Worth
c0961e6a82 lib: Switch to a 3-part version number for the library interface.
With a carefully documented description of how to increment the
various version components.
2010-04-01 00:41:25 -07:00
Carl Worth
33d5cc415e Makefiles: Make the install rules quiet like the compilation rules.
The output from make is looking better all the time, (though the
columns still aren't lined up).
2010-03-31 23:54:21 -07:00
Carl Worth
7b52b2c318 Move installation of library from top-level to lib/Makefile.local
We had a fairly ugly violation of modularity with the top-level
Makefile.local isntalling everything, (even when the build commands
for the library were down in lib/Makefile.local).
2010-03-31 22:54:15 -07:00
Saleem Abdulrasool
07378d0d14 Fix target dependencies for multiple jobs
Signed-off-by: Ingmar Vanhassel <ingmar@exherbo.org>
2010-03-31 17:41:28 -07:00
Ben Gamari
266ab595a2 Build and link against notmuch shared library, install notmuch.h
Signed-off-by: Ingmar Vanhassel <ingmar@exherbo.org>
2010-03-31 17:38:27 -07:00
Carl Worth
b957a1b029 emacs: Fix the notmuch-search-authors-width variable.
This variable existed previously, but wasn't actually used for anything.
2010-03-31 13:32:00 -07:00
Carl Worth
e002fe8a7a Clarify documentation of notmuch_database_add_message.
For the case of adding a file that already exist, (with the same
filename). In this case, nothing will happen to the database, but
that wasn't clear before.
2010-03-31 13:31:10 -07:00
Carl Worth
86232e62ab Makefile: Fix Makefiles to depend on all child Makefile fragments.
We were previously maintaining two lists of the child Makefile
fragments---one for the includes and another for the dependencies. So,
of course, they drifted and the dependency list wasn't up to date.

We fix this by adding a single subdirs variable, and then using GNU
Makefile substitution to generate both the include and the dependency
lists.

Some side effect of this change caused the '=' assignment of the dir
variable to not work anymore. I'm not sure why that is, but using ':='
makes sense here and fixes the problem.
2010-03-10 10:59:57 -08:00
Carl Worth
e3046c688b Add is:<tag> as a synonym for tag:<tag> in search terms.
I like the readability of this, it provides compatibility with people
trained in this syntax by sup, and it even saves one character.
2010-03-09 16:03:58 -08:00
Carl Worth
c446f22dee lib: Silence a compiler warning.
The original code was harmless, but apparently some compilers aren't
able to think deep enough to catch that.
2010-03-09 12:07:26 -08:00
Fernando Carrijo
7f2629520c Fix a few documentation typos in notmuch.h
Fix a few documentation typos in notmuch.h

Signed-off-by: Fernando Carrijo <fcarrijo@yahoo.com.br>
2010-03-09 10:32:58 -08:00
Fernando Carrijo
bc69bf09cb Update documentation of notmuch_query_create
Commit cd467caf renamed notmuch_query_search to notmuch_query_search_messages.
Commit 1ba3d46f created notmuch_query_search_threads. We better keep the docs
of notmuch_query_create consistent with those changes.

Signed-off-by: Fernando Carrijo <fcarrijo@yahoo.com.br>

Edited-by: Carl Worth to explicitly list the full name of each
function being referenced.
2010-03-09 10:29:38 -08:00
Carl Worth
64646841f7 lib: Document what move_to_next does at the end of the list.
Explicitly mention that there's an invalid position after the last
item in the list.
2010-03-09 09:24:45 -08:00
Carl Worth
4e5d2f22db lib: Rename iterator functions to prepare for reverse iteration.
We rename 'has_more' to 'valid' so that it can function whether
iterating in a forward or reverse direction. We also rename
'advance' to 'move_to_next' to setup parallel naming with
the proposed functions 'move_to_first', 'move_to_last', and
'move_to_previous'.
2010-03-09 09:22:29 -08:00
Carl Worth
e0a8dee8bc Fix printf for when uint64_t != unsigned long long int
Thanks to Michal Sojka <sojkam1@fel.cvut.cz> for pointing out the
correct fix, which I verified in the freely-available WG14/N1124 draft
(from the C99 working group) which is available here:

http://www.open-std.org/JTC1/SC22/wg14/www/docs/n1124.pdf
2010-02-09 11:14:16 -08:00
Carl Worth
9439b217c3 Switch from random to sequential thread identifiers.
The sequential identifiers have the advantage of being guaranteed to
be unique (until we overflow a 64-bit unsigned integer), and also take
up half as much space in the "notmuch search" output (16 columns
rather than 32).

This change also has the side effect of fixing a bug where notmuch
could block on /dev/random at startup (waiting for some entropy to
appear). This bug was hit hard by the test suite, (which could easily
exhaust the available entropy on common systems---resulting in large
delays of the test suite).
2010-02-09 11:14:11 -08:00
Carl Worth
7a9bacac67 notmuch.h: Fix a couple of typos in the documentation.
Obviously, the spell-checker isn't able to catch every mistake
I make.
2010-02-05 17:31:40 -08:00
Carl Worth
2bc0af15aa Eliminate some useless gobject boilerplate.
If we had external users of this filter then they might expect some of
these macros to exist. But since this is just internal, that's just
unneeded noise.
2010-02-04 17:26:00 -08:00
Carl Worth
3767c6f9f9 notmuch new: Don't index uuencoded data.
With modern MIME attachments, we're already avoiding indexing the
attachments. But for old-school uuencoded data in the mail, we have
been directly indexing the encoded data as terms, (which is not useful
at all---nobody will ever ytry to search based on the seemingly random
uuencoded data).

Additionally, indexing a modestly large uuencoded file seems to make
Xapian go insane, (consuming *lots* of memory).

We fix both problems by detecting uuencoded content and not performing
any indexing of it.
2010-02-04 17:08:11 -08:00
Carl Worth
c340c1bd11 notmuch new: Print upgrade progress report as a percentage.
Previously we were printing a number of messages upgraded so far. The
original motivation for this was to accurately reflect the fact that
there are two passes, (so each message is processed twice and it's not
accurate to represent with a single count). But as it turns out, the
second pass takes zero time (relatively speaking) so we're still not
accounting for it.

If nothing else, the percentage-based reporting makes for a cleaner
API for the progress_notify function.
2010-01-09 17:38:23 -08:00
Carl Worth
ccf2e0cc42 lib: Add non-content terms with a WDF value of 0.
The WDF is the "within-document frequency" value for a particular
term. It's intended to provide an indication of how frequent a term is
within a document, (for use in computing relevance). Xapian's term
generator already computes WDF values when we use that, (which we do
for indexing all mail content).

We don't use the term generator when adding single terms for things
that don't actually appear in the mail document, (such as tags, the
filename, etc.). In this case, the WDF value for these terms doesn't
matter much.

But Xapian's flint backend can be more efficient with changes to terms
that don't affect the document "length". So there's a performance
advantage for manipulating tags (with the flint backend) if the WDF of
these terms is 0.
2010-01-09 11:18:27 -08:00
Carl Worth
45b1856782 lib: Explicitly set BoolWeight when searching.
All notmuch searches currently sort by value (either date or message
ID) so it's just wasted effort for Xapian to compute relevance values
for each result. We now explicitly tell Xapian that we're uninterested
in the relevance values.
2010-01-09 11:16:40 -08:00
Carl Worth
d12801c8b4 lib: Split the database upgrade into two phases for safer operation.
The first phase copies data from the old format to the new format
without deleting anything. This allows an old notmuch to still use the
database if the upgrade process gets interrupted. The second phase
performs the deletion (after updating the database version number). If
the second phase is interrupted, there will be some unused data in the
database, but it shouldn't cause any actual harm.
2010-01-09 11:13:12 -08:00
Carl Worth
5fe5e802ab lib: Delete stale timestamp documents during database upgrade.
Once we move the timestamp to the new directory document, we don't
need the old one anymore.
2010-01-08 09:57:09 -08:00
Carl Worth
1c86b48329 notmuch new: Fix progress notification on database upgrade.
This was firing continuously rather than just once per second as
intended.
2010-01-07 21:24:44 -08:00
Carl Worth
909f52bd8c lib: Implement versioning in the database and provide upgrade function.
The recent support for renames in the database is our first time
(since notmuch has had more than a single user) that we have a
database format change. To support smooth upgrades we now encode a
database format version number in the Xapian metadata.

Going forward notmuch will emit a warning if used to read from a
database with a newer version than it natively supports, and will
refuse to write to a database with a newer version.

The library also provides functions to query the database format
version:

	notmuch_database_get_version

to ask if notmuch wants a newer version than that:

	notmuch_database_needs_upgrade

and a function to actually perform that upgrade:

	notmuch_database_upgrade
2010-01-07 18:26:31 -08:00
Carl Worth
807aef93d3 Prefer READ_ONLY consistently over READONLY.
Previously we had NOTMUCH_DATABASE_MODE_READ_ONLY but
NOTMUCH_STATUS_READONLY_DATABASE which was ugly and confusing. Rename
the latter to NOTMUCH_STATUS_READ_ONLY_DATABASE for consistency.
2010-01-07 10:29:05 -08:00
Carl Worth
f93b7218c3 lib: Consolidate checks for read-only database.
Previously, many checks were deep in the library just before a cast
operation. These have now been replaced with internal errors and new
checks have instead been added at the beginning of all top-levelentry
points requiring a read-write database.

The new checks now also use a single function for checking and
printing the error message. This will give us a convenient location to
extend the check, (such as based on database version as well).
2010-01-07 10:19:44 -08:00
Carl Worth
6ed606c19e lib: Clarify internal documentation of _notmuch_database_filename_to_direntry
The original wording made it sound like this function was just doing
some string manipulation. But this function actually creates new
directory documents as a side effect. So make that explicit in its
documentation.
2010-01-07 09:31:58 -08:00
Carl Worth
a274848f95 notmuch_message_get_filename: Support old-style filename storage.
When a notmuch database is upgraded to the new database format, (to
support file rename and deletion), any message documents corresponding
to deleted files will not currently be upgraded. This means that a
search matching these documents will find no filenames in the expected
place.

Go ahead and return the filename as originally stored, (rather than
aborting with an internal error), in this case.
2010-01-07 09:22:34 -08:00
Carl Worth
957ae198e7 lib: Treat NULL as a valid (and empty) notmuch_filenames_t iterator.
This will be convenient to avoid some special-casing in higher-level
code.
2010-01-06 14:35:11 -08:00
Carl Worth
4b418343f6 lib: Indicate whether notmuch_database_remove_message removed anything.
Similar to the return value of notmuch_database_add_message, we now
enhance the return value of notmuch_database_remove_message to
indicate whether the message document was entirely removed (SUCCESS)
or whether only this filename was removed and the document exists
under other filenamed (DUPLICATE_MESSAGE_ID).
2010-01-06 10:32:06 -08:00
Carl Worth
777cd23d9d lib: Update documentation of notmuch_database_add_message.
Previously, adding a filename with the same message ID as an existing
message would do nothing. But we recently fixed this to instead add
the new filename to the existing message document. So update the
documentation to match now.
2010-01-06 10:32:06 -08:00
Carl Worth
6ef6ddba80 Index content from citations and signatures.
In the presentation we often omit citations and signatures, but this
is not content that should be omitted from the index, (especially
when the citation detection is wrong---see cases where a line
beginning with "From" is corrupted to ">From" by mail processing
tools).
2010-01-06 10:32:06 -08:00
Carl Worth
341d49b061 Makefiles: Use .DEFAULT to support arbitrary targets from sub directories.
Taking advantage of the .DEFAULT construct means that we won't need to
explicitly list targets such as "clean", etc. in each sub-Makefile.
2010-01-06 10:32:06 -08:00
Carl Worth
3f32fd8a1c Add missing comment for NOTMUCH_STATUS_READONLY_DATABASE.
And adjust the string representation of the same to match.
2010-01-06 10:32:06 -08:00
Carl Worth
d807e28f43 lib: Implement new notmuch_directory_t API.
This new directory ojbect provides all the infrastructure needed to
detect when files or directories are deleted or renamed. There's still
code needed on top of this (within "notmuch new") to actually do that
detection.
2010-01-06 10:32:06 -08:00
Carl Worth
ba07fe1819 Revamp the proposed directory-tracking API slightly.
This commit contains my changes to the API proposed by Keith. Nothing
is dramatically different. There are minor things like changing
notmuch_files_t to notmuch_filenames_t and then various things needed
for completeness as noticed while implementing this, (such as
notmuch_directory_destroy and notmuch_directory_set_mtime).
2010-01-06 10:32:06 -08:00
Keith Packard
95deec1b27 Prototypes for directory tracking
There's no functionality here yet---just a sketch of what the
interface could look like.
2010-01-06 10:32:06 -08:00
Carl Worth
f11aaa3678 database: Add new, public notmuch_database_remove_message
This will allow applications to support the removal of messages, (such
as when a file is deleted from the mail store). No removal support is
provided yet in commands such as "notmuch new".
2010-01-06 10:32:06 -08:00
Carl Worth
44a74912c7 database: Add new find_doc_ids_for_term interface.
The existing find_doc_ids function is convenient when the caller
doesn't want to be bothered constructing a term. But when the caller
*does* have the term already, that interface is just wasteful. So we
export a lower-level interface that maps a pre-constructed term to a
document-ID iterators.
2010-01-06 10:32:06 -08:00
Carl Worth
d7e5f5827e database: Make find_unique_doc_id enforce uniqueness (for a debug build)
Catching any violation of this unique-ness constraint is very much in
line with similar, existing INTERNAL_ERROR cases.
2010-01-06 10:32:06 -08:00
Carl Worth
498edff503 database: Abstract _filename_to_direntry from _add_message
The code to map a filename to a direntry is something that we're going
to want in a future _remove_message function, so put it in a new
function _notmuch_database_filename_to_direntry .
2010-01-06 10:32:05 -08:00
Carl Worth
1376a90db6 database: Allowing storing multiple filenames for a single message ID.
The library interface is unchanged so far, (still just
notmuch_database_add_message), but internally, the old
_set_filename function is now _add_filename instead.
2010-01-06 10:32:05 -08:00
Carl Worth
6ca6c089e9 database: Store mail filename as a new 'direntry' term, not as 'data'.
Instead of storing the complete message filename in the data portion
of a mail document we now store a 'direntry' term that contains the
document ID of a directory document and also the basename of the
message filename within that directory. This will allow us to easily
store multple filenames for a single message, and will also allow us
to find mail documents for files that previously existed in a
directory but that have since been deleted.
2010-01-06 10:32:05 -08:00
Carl Worth
84742d86ab database: Split _find_parent_id into _split_path and _find_directory_id
Some pending commits want the _split_path functionality separate from
mapping a directory to a document ID. The split_path function now
returns the basename as well as the directory name.
2010-01-06 10:32:05 -08:00
Carl Worth
4c1cca888f database: Store directory path in 'data' of directory documents.
We're planning to have mail documents refer to directory documents for
the path of the containing directory. To support this, we need the
path in the data, (since the path in the 'directory' term can be
irretrievable as it will be the SHA1 sum of the path for a very long
path).
2010-01-06 10:32:05 -08:00
Carl Worth
406ec4b15d database: Export _notmuch_database_find_parent_id for internal use.
We'll soon have mail documents referring to their parent directory's
directory documents, so we'll need access to _find_parent_id in files
such as message.cc.
2010-01-06 10:32:05 -08:00
Carl Worth
e890b0cf40 database: Store the parent ID for each directory document.
Storing the document ID of the parent of each directory document will
allow us to find all child-directory documents for a given directory
document. We will need this in order to detect directories that have
been removed from the mail store, (though we aren't yet doing this).
2010-01-06 10:32:05 -08:00