Commit graph

1032 commits

Author SHA1 Message Date
Carl Worth
28dd86af05 Implement notmuch_tags_t on top of new notmuch_terms_t
The generic notmuch_terms_t iterator should provide support for
notmuch_thread_ids_t when we switch as well, (And it would be
interesting to see if we could reasonably make this support a
PostingIterator too. Time will tell.)
2009-10-25 00:31:13 -07:00
Carl Worth
9ec68aa9c4 Shuffle the value numbers around in the database.
First, it's nice that for now we don't have any users yet, so we
can make incompatible changes to the database layout like this
without causing trouble. ;-)

There are a few reasons for this change. First, we now use value 0
uniformly as a timestamp for both mail and timestamp documents, (which
lets us cleanup an ugly and fragile bare 0 in the add_value and
get_value calls in the timestamp code).

Second, I want to drop the thread value entirely, so putting it at the
end of the list means we can drop it as compatible change in the
future. (I almost want to drop the message-ID value too, but it's nice
to be able to sort on it to get diff-able output from "notmuch dump".)

But the thread value we never use as a value, (we would never sort on
it, for example). And it's totally redundant with the thread terms we
store already. So expect it to disappear soon.
2009-10-24 23:05:08 -07:00
Carl Worth
65a272832e Invent our own prefix values.
We're now dropping all pretense of keeping the database directly
compatible with sup's current xapian backend. (But perhaps someone
might write a new nothmuch backend for sup in the future.)

In coming up with the prefix values here, I tried to follow the
conventions of http://xapian.org/docs/omega/termprefixes.html as
closely as makes sense, (with some domain translation from "web"
to "email archive").
2009-10-24 22:57:47 -07:00
Carl Worth
0aa355cc8f Split BOOLEAN_PREFIX into INTERNAL and EXTERNAL subsets.
The idea here is that only some of the prefix names (such as "id" and
"tag") actually make sense in external user-supplied query
strings. Other things like "type" are internal implementation details
of how we store things in the database. So internal machinery will add
those terms to the database and we don't need to support them in the
string itself.

With this, we can now simply loop over the external prefix values to
let the quiery parser know about them. So as we add prefixes in the
future, we'll only need to add them to this list.
2009-10-24 22:38:43 -07:00
Carl Worth
2a9b4fce7c Change all occurrences of "msgid" to "id".
What's good for the user is good for the internals.
2009-10-24 22:29:49 -07:00
Carl Worth
36809b5dfb Add bash-completion script for notmuch.
It's not much of a script, (we don't have that many commands after
all), but it's the kind of thing that's nice to have and gives the
tool a slightly more polished feel.
2009-10-24 22:28:22 -07:00
Carl Worth
aa46a683a8 Add the magic to allow searches such as "tag:inbox".
The key for this is call add_boolean_prefix on the QueryParser
object. That tells the query parser to take something like "tag:inbox"
and transform it into the "Linbox" term and do what it needs to do to
make this term a requirement of the search. We're starting to have a
real system here.

Also, I didn't want to expose the ugly name of "msgid" to the user, so
we add a prefix name of simply "id" instead.
2009-10-24 22:23:58 -07:00
Carl Worth
17f9c6a0ef Use _find_prefix instead of hard-coded term in notmuch_query_search
I'm planning to change prefix values soon, which would break code
like this. So eliminate the fragility by going through our existing
_find_prefix function.
2009-10-24 22:21:57 -07:00
Carl Worth
15d949b740 Fix bit-twiddling brain damage in notmuch_query_search
Here's the big bug that was preventing any searches from working at
all like desired. I did the work to carefully pick out exactly the
flags that I wanted, and then I threw it away by trying to combine
them with & instead of | (so just passing 0 for flags instead).

Much better now.
2009-10-24 22:20:13 -07:00
Carl Worth
526b7144f7 Add debugging code for examining query strings.
It's nice that Xapian provides a little function to print a textual
representation of the entire query tree. So now, if you compile
like so:

	make CFLAGS=-DDEBUG_QUERY

then you get a nice output of the query string received by the query
module, and the final query actually being sent to Xapian.
2009-10-24 22:18:20 -07:00
Carl Worth
144b2cbc55 Add a preliminary "notmuch search" command.
This isn't behaving at all like it's documented yet, (for example,
it's returning message IDs not thread IDs[*]). In fact, the output
code is just a copy of the body of "notmuch dump", so all you
get for now is message ID and tags.

But this should at least be enough to start exercising the query
functionality, (which is currently very buggy).

[*] I'll want to convert the databse to store thread documents
before fixing that.
2009-10-24 22:16:10 -07:00
Carl Worth
0e72d6a9ec notmuch_database_create: Document idea to (optionally) return a status
The current problem is that when this function fails the caller
doesn't get any information about what the particular failure
was, (something in the filesystem? or in Xapian?). We should fix
that.
2009-10-24 22:14:31 -07:00
Carl Worth
309c0e1cc5 notmuch setup/new: Propagate failure from notmuch_database_set_timestamp
With some recent testing, the timestamp was failing, (overflowing
the term limit), and reporting an error, but the top-level notmuch
command was still returning a success return value.

I think it's high time to add a test suite, (and the code base is
small enough that if we add it now it shouldn't be *too* hard to
shoot for a very high coverage percentage).
2009-10-24 22:11:38 -07:00
Carl Worth
0bc73af96c Fix timestamp generation to avoid overflowing the term limit
The previous code was only correct as long as the timestamp prefix
was only a single character. But with the recent change to a
multi-character prefix, this broke. So fix it now.
2009-10-24 22:10:03 -07:00
Carl Worth
f281f4b677 Trim down prefix list to things we are actually using.
I've decided not to try for sup compatibility at the leve of the
xapian datbase. There's just too much about sup's usage of the
database that I don't like, (beyond the embedded ruby data structures
there is redundant storage of message IDs, thread IDs, and dates (in
both terms and values)).

I'm going to fix that up in the database of notmuch, with some other
changes as well. (I plan to drop "reference" terms once linkage to a
thread ID through the reference is established.  I also plan to add
actual documents to represent threads.)

So with all that incompatibility, I might as well make my own prefix
values. And while doing that, I should try to be as compatible as
possible with the conventions described here:

http://xapian.org/docs/omega/termprefixes.html
2009-10-24 22:04:59 -07:00
Carl Worth
e37b7cc2da Move the prefix-string arrays back into database.cc from message.cc
Yes, I'm being wishy-washy here, moving code back and forth. But
this is where these really do belong.
2009-10-24 21:52:48 -07:00
Carl Worth
e1e1f0cb3b Revert "Remove some unneeded initializers."
This reverts commit fb1bae0700.

These initializers were totally necessary. I clearly wasn't
thinking straight when I removed them.
2009-10-24 08:06:23 -07:00
Carl Worth
1f193675c4 Cut the enthusiasm a bit.
It gets annoying pretty quick.
2009-10-23 17:25:23 -07:00
Carl Worth
c037dc5e53 Make "notmuch new" ignore directories that are read-only.
With this, "notmuch new" is now plenty fast even with large archives
spanning many sub-directories. Document this both in "notmuch help"
and also in the output of notmuch setup.
2009-10-23 17:25:03 -07:00
Carl Worth
ef944693c3 add_files: Pull one stat out of the recrusive function.
There's no need to stat each directory both before and after each
recursive call.
2009-10-23 16:23:18 -07:00
Carl Worth
e58d85dd75 More fixing of plurals.
It definitely doesn't help that we have the same messages in both
"setup" and "new". Should combine those really.
2009-10-23 16:00:24 -07:00
Carl Worth
6cccc10bf0 More care in final status reporting.
Printing "Added 1 new messages" just looks like lack of attention
to detail, (but yes plurals can be annoying this way).
2009-10-23 15:57:39 -07:00
Carl Worth
cbb2675ec5 Print a better message than "0s" for zero seconds.
It's nice to have a tool that at least construct actual sentences.
2009-10-23 15:55:36 -07:00
Carl Worth
491ece966f Add new "notmuch new" command.
Finally, I can get new messages into my notmuch database without
having to run a complete "notmuch setup" again. This takes
advantage of the recent timestamp capabilities in the database
to avoid looking into directories that haven't changed since the
last time "notmuch new" was run.
2009-10-23 15:48:05 -07:00
Carl Worth
9577e8a105 add_files: Change to return a status value instead of void
Also change to use goto rather than early returns. And once again,
there were lots of bugs in the error cases previously.
2009-10-23 15:39:11 -07:00
Carl Worth
a7a897b883 notmuch setup: Clean up the progress printing a bit.
Get rid of a useless leading 0 on the seconds value, and make a
distinction between "files" and "messages", (we process many
files, but not all of them are recongized as messages). Finally,
add a summary line at the end saying how many unique messages
were added to the database. Since this comes right after the
total number of files, it gives the user at least a hint as
to how many messages were encountered with duplicate message IDs.
2009-10-23 15:29:06 -07:00
Carl Worth
fd11c8e150 Re-order documentation a bit.
The notmuch_database_get_default_path function is unique in not
accepting a notmuch_database_t* (nor creating one). So list it
outside the other notmuch_database functions.
2009-10-23 15:17:16 -07:00
Carl Worth
03c40c2bb0 notmuch_message_get_filename: Improve documentation.
Fix a typo, and add clarifications about the lifetime and readonly
nature of the return value.
2009-10-23 15:12:03 -07:00
Carl Worth
fb1bae0700 Remove some unneeded initializers.
Some people might argue for more initializers to be "safer",
but I actually prefer to leave things this way. It saves
typing, but the real benefit is that the things that do
require initialization stand out so we know to watch them
carefully. And with valgrind, we actually get to catch
errors earlier if we *don't* initialize them. So that can
be "safer" ironically enough.
2009-10-23 14:55:50 -07:00
Carl Worth
336deb279e notmuch setup: Fix a couple of error paths.
We had early returns instead of goto statments, and sure enough,
they were leaking. Much cleaner this way.
2009-10-23 14:55:02 -07:00
Carl Worth
a5e619f11f _find_prefix: Exit when given an invalid prefix name.
This will be a nice safety check for internal sanity.
2009-10-23 14:45:33 -07:00
Carl Worth
b3cbcea8fd Add NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID
And document that notmuch_database_add_message can return this
value. This pushes the hard decision of what to do with duplicate
messages out to the user, but that's OK. (We weren't really doing
anything with these ourselves, and this way the user is at least
informed of the issue, rather than it just getting papered over
internally.)
2009-10-23 14:40:33 -07:00
Carl Worth
edd37e68a9 Clean up comments to not include spaces before tabs.
This were just unclean, (an invisble sort of uncleanliness, but still
there are liable to make for ugly diffs). Oh, wait, like this one!
But at least it's not sprinkled among code changes.
2009-10-23 14:37:09 -07:00
Carl Worth
5ebb21600e Clarify documentation and error string for NOTMUCH_STATUS_TAG_TOO_LONG
It's helpful to point out NOTMUCH_STATUS_TAG_MAX for users.
2009-10-23 14:36:38 -07:00
Carl Worth
68a10091d6 Add notmuch_database_set_timestamp and notmuch_database_get_timestamp
These will be very helpful to implement an efficient "notmuch new"
command which imports new mail messages that have appeared.
2009-10-23 14:31:01 -07:00
Carl Worth
668f20bdfb database: Add private find_unique_doc_id and find_unique_document functions
These are a generalization of the unique-ness testing of
notmuch_database_find_message. More preparation for
firectory timestamps.
2009-10-23 14:24:07 -07:00
Carl Worth
edbf7f645c database: Similarly rename find_message_by_docid to find_document_for_doc_id
Again preferring notmuch_database_t* over Xapian::Database*.

Also, we're standardizing on "doc_id" rather than "docid" locally, (as
an analoge to "message_id"), in spite of the "Xapian::docid" name,
(which, fortunately, we can ignore and just us "unsigned int" instead).
2009-10-23 14:12:06 -07:00
Carl Worth
9fc4a365d6 database: Rename internal find_messages_by_term to find_doc_ids
This name is a more accurate description of what it does, and
the more general naming will make sense as we start storing
non-message documents in the database (such as directory
timestamps).

Also, don't pass around a Xapian::Database where it's more our
style to pass a notmuch_database_t*.
2009-10-23 14:06:24 -07:00
Carl Worth
6b228e4509 sha1: Add new notmuch_sha1_of_string function
We'll be using this for storing really long terms in the database
and when we just need to look them up, (and never read back the
original data directly from the database). For example, storing
arbitrarily long directory paths in the database along with
mtime timestamps.

Note that if we did want to store arbitrarily long terms and also
be able to read them back, the Xapian folks recommending splitting
the term off with multiple prefixes. See the note near the end
of this page:

http://trac.xapian.org/wiki/FAQ/UniqueIds
2009-10-23 13:54:53 -07:00
Carl Worth
c9fbe6b58b notmuch restore: Print names of tags that cannot be applied
This helps the user gauge the severity of the error.

For example, when restoring my sup tags I see a bunch of tags missing
for message IDs of the form "sup-faked-...". That's not surprising
since I know that sup generates these with the md5sum of the message
header while notmuch uses the sha-1 of the entire message. But how
much will this hurt?

Well, now that I can see that most of the missing tags are just
"attachment", then I'm not concerned, (I'll be automatically creating
that tag in the future based on the message contents). But if a
missing tag is "inbox" then that's more concerning because that's data
that I can't easily regenerate outside of sup.
2009-10-23 06:08:22 -07:00
Carl Worth
db93109cfe notmuch_tags_has_more: Fix to use string.empty rather than string.size
I'm really interested in the length of the data here, not the size
of the storage.
2009-10-23 06:06:20 -07:00
Carl Worth
ce5d782962 Fix notmuch_message_get_message_id to never return NULL.
With the recent improvements to the handling of message IDs we
"know" that a NULL message ID is impossible, (so we simply
abort if the impossible happens).
2009-10-23 06:04:57 -07:00
Carl Worth
6ccdffcd87 add_message: Fix to not add multiple documents with the same message ID
Here's the second big fix to message-ID handling, (the first was to
generate message IDs when an email contained none). Now, with no
document missing a message ID, and no two documents having the same
message ID, we have a nice consistent database where the message ID
can be used as a unique key.
2009-10-23 06:00:10 -07:00
Carl Worth
1b5d8984c6 Add _notmuch_message_create_for_message_id
This is the last piece needed for add_message to be able to properly
support a message with a duplicate message ID.  This function creates
a new notmuch_message_t object but one that may reference an existing
document in the database.
2009-10-23 05:53:52 -07:00
Carl Worth
69b25a75ec Fix _notmuch_message_create to catch Xapian DocNotFoundError.
This function is only supposed to be called with a doc_id that
was queried from the database already. So there's an internal
error if no document with that doc_id can be found in the database.

In that case, return NULL.
2009-10-23 05:48:52 -07:00
Carl Worth
17548e314a Add internal functions for manipulating a new notmuch_message_t
This will support the add_message function in incrementally creating
state in a new notmuch_message_t. The new functions are

      _notmuch_message_set_filename
      _notmuch_message_add_thread_id
      _notmuch_message_ensure_thread_id
      _notmuch_message_set_date
      _notmuch_message_sync
2009-10-23 05:48:52 -07:00
Carl Worth
868d3b3068 Add notmuch_message_get_filename
This is a new public function to find the filename of the original
email message for a message-object that was found in the database.

We may change this function in the future to support returning a
list of filenames, (for messages with duplicate message IDs).
2009-10-23 05:48:46 -07:00
Carl Worth
31044d10ed add_message: Re-order the code a bit (find message-id first).
We're preparing for being able to deal with files with duplicate
message IDs here. The plan is to create a notmuch_message_t object in
add_message that may or may not reference a document that exists in
the database. So to do this, we have to find the message ID before we
do any manipulation of the doc.
2009-10-23 05:30:37 -07:00
Carl Worth
c78358fa8a Move thread_id generation code from database.cc to message.cc
It's really up to the message to decide how to generate these.
2009-10-23 05:25:58 -07:00
Carl Worth
97775ef438 Move the _notmuch_message_sync from private to public interfaces
The idea here is to allow internal users to see a non-synced message
object, (for example, while parsing a message file and incrementally
adding terms, etc.). We're willing to take the care to get the
improved performance.

But for the public interface, keeping everything synced will be much
less confusing, (reference lots of sup bugs that happen due to
message state being altered by the user but not synced to the database).
2009-10-23 05:20:03 -07:00