debian/notmuch
1
0
Fork 0
mirror of https://git.notmuchmail.org/git/notmuch synced 2024-12-23 01:44:52 +01:00
Code Issues Projects Releases Packages Wiki Activity Actions
notmuch/lib/database.cc

2717 lines
79 KiB
C++
Raw Normal View History

notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
/* database.cc - The database interfaces of the notmuch mail library
*
* Copyright © 2009 Carl Worth
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
Use https instead of http where possible Many of the external links found in the notmuch source can be resolved using https instead of http. This changeset addresses as many as i could find, without touching the e-mail corpus or expected outputs found in tests.
2016-06-02 18:26:14 +02:00
* along with this program. If not, see https://www.gnu.org/licenses/ .
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
*
* Author: Carl Worth <cworth@cworth.org>
*/
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
#include "database-private.h"
lib: add date range query support Add a custom value range processor to enable date and time searches of the form date:since..until, where "since" and "until" are expressions understood by the previously added date/time parser, to restrict the results to messages within a particular time range (based on the Date: header). If "since" or "until" describes date/time at an accuracy of days or less, the values are rounded according to the accuracy, towards past for "since" and towards future for "until". For example, date:november..yesterday would match from the beginning of November until the end of yesterday. Expressions such as date:today..today means since the beginning of today until the end of today. Open-ended ranges are supported (since Xapian 1.2.1), i.e. you can specify date:..until or date:since.. to not limit the start or end date, respectively. CAVEATS: Xapian does not support spaces in range expressions. You can replace the spaces with '_', or (in most cases) '-', or (in some cases) leave the spaces out altogether. Entering date:expr without ".." (for example date:yesterday) will not work as you might expect. You can achieve the expected result by duplicating the expr both sides of ".." (for example date:yesterday..yesterday). Open-ended ranges won't work with pre-1.2.1 Xapian, but they don't produce an error either. Signed-off-by: Jani Nikula <jani@nikula.org>
2012-10-30 21:32:37 +01:00
#include "parse-time-vrp.h"
lib: add support for named queries This relies on the optional presense of xapian field processors, and the library config API.
2016-03-22 11:54:54 +01:00
#include "query-fp.h"
lib: Database version 3: Introduce fine-grained "features" Previously, our database schema was versioned by a single number. Each database schema change had to occur "atomically" in Notmuch's development history: before some commit, Notmuch used version N, after that commit, it used version N+1. Hence, each new schema version could introduce only one change, the task of developing a schema change fell on a single person, and it all had to happen and be perfect in a single commit series. This made introducing a new schema version hard. We've seen only two schema changes in the history of Notmuch. This commit introduces database schema version 3; hopefully the last schema version we'll need for a while. With this version, we switch from a single version number to "features": a set of named, independent aspects of the database schema. Features should make backwards compatibility easier. For many things, it should be easy to support databases both with and without a feature, which will allow us to make upgrades optional and will enable "unstable" features that can be developed and tested over time. Features also make forwards compatibility easier. The features recorded in a database include "compatibility flags," which can indicate to an older version of Notmuch when it must support a given feature to open the database for read or for write. This lets us replace the old vague "I don't recognize this version, so something might go wrong, but I promise to try my best" warnings upon opening a database with an unknown version with precise errors. If a database is safe to open for read/write despite unknown features, an older version will know that and issue no message at all. If the database is not safe to open for read/write because of unknown features, an older version will know that, too, and can tell the user exactly which required features it lacks support for.
2014-08-25 19:26:00 +02:00
#include "string-util.h"
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
#include <iostream>
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
#include <sys/time.h>
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
#include <sys/stat.h>
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
#include <signal.h>
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
#include <ftw.h>
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
Remove all calls to g_strdup_printf Replacing them with calls to talloc_asprintf if possible, otherwise to asprintf (with it's painful error-handling leaving the pointer undefined).
2009-10-26 23:17:10 +01:00
#include <glib.h> /* g_free, GPtrArray, GHashTable */
lib: call g_type_init from notmuch_database_open We want to make sure g_type_init is called before any GObject functionality is used.
2011-12-04 20:20:39 +01:00
#include <glib-object.h> /* g_type_init */
notmuch: Switch from gmime to custom, ad-hoc parsing of headers. Since we're currently just trying to stitch together In-Reply-To and References headers we don't need that much sophistication. It's when we later add full-text searching that GMime will be useful. So for now, even though my own code here is surely very buggy compared to GMime it's also a lot faster. And speed is what we're after for the initial index creation.
2009-10-19 21:54:40 +02:00
lib: call g_mime_init() from notmuch_database_open() As reported in id:"CAEbOPGyuHnz4BPtDutnTPUHcP3eYcRCRkXhYoJR43RUMw671+g@mail.gmail.com" sometimes gmime tries to access a NULL pointer, e.g. g_mime_iconv_open() tries to access iconv_cache that is NULL if g_mime_init() is not called. This causes notmuch to segfault when calling gmime functions. Calling g_mime_init() initializes iconv_cache and others variables needed by gmime, making sure they are initialized when notmuch calls gmime functions. Test marked fix by db.
2011-12-31 05:37:41 +01:00
#include <gmime/gmime.h> /* g_mime_init */
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
using namespace std;
Move the prefix-string arrays back into database.cc from message.cc Yes, I'm being wishy-washy here, moving code back and forth. But this is where these really do belong.
2009-10-25 05:52:48 +01:00
#define ARRAY_SIZE(arr) (sizeof (arr) / sizeof (arr[0]))
typedef struct {
const char *name;
const char *prefix;
lib: merge internal prefix tables Replace multiple tables with some flags in a single table. This makes the code in notmuch_database_open_verbose a bit shorter, and it should also make it easier to add other options to fields, e.g. regexp searching.
2017-02-17 04:07:49 +01:00
notmuch_field_flag_t flags;
Move the prefix-string arrays back into database.cc from message.cc Yes, I'm being wishy-washy here, moving code back and forth. But this is where these really do belong.
2009-10-25 05:52:48 +01:00
} prefix_t;
lib: Database version 3: Introduce fine-grained "features" Previously, our database schema was versioned by a single number. Each database schema change had to occur "atomically" in Notmuch's development history: before some commit, Notmuch used version N, after that commit, it used version N+1. Hence, each new schema version could introduce only one change, the task of developing a schema change fell on a single person, and it all had to happen and be perfect in a single commit series. This made introducing a new schema version hard. We've seen only two schema changes in the history of Notmuch. This commit introduces database schema version 3; hopefully the last schema version we'll need for a while. With this version, we switch from a single version number to "features": a set of named, independent aspects of the database schema. Features should make backwards compatibility easier. For many things, it should be easy to support databases both with and without a feature, which will allow us to make upgrades optional and will enable "unstable" features that can be developed and tested over time. Features also make forwards compatibility easier. The features recorded in a database include "compatibility flags," which can indicate to an older version of Notmuch when it must support a given feature to open the database for read or for write. This lets us replace the old vague "I don't recognize this version, so something might go wrong, but I promise to try my best" warnings upon opening a database with an unknown version with precise errors. If a database is safe to open for read/write despite unknown features, an older version will know that and issue no message at all. If the database is not safe to open for read/write because of unknown features, an older version will know that, too, and can tell the user exactly which required features it lacks support for.
2014-08-25 19:26:00 +02:00
#define NOTMUCH_DATABASE_VERSION 3
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
#define STRINGIFY(s) _SUB_STRINGIFY(s)
#define _SUB_STRINGIFY(s) #s
Use the Xapian::DB_RETRY_LOCK flag when available Xapian 1.3 has introduced the DB_RETRY_LOCK flag (Xapian bug 275). Detect it in configure and optionally use it. With this flag commands that need the write lock will wait for their turn instead of aborting when it's not immediately available. Amended by db: allow disabling in configure
2016-06-26 17:29:43 +02:00
#if HAVE_XAPIAN_DB_RETRY_LOCK
#define DB_ACTION (Xapian::DB_CREATE_OR_OPEN | Xapian::DB_RETRY_LOCK)
#else
#define DB_ACTION Xapian::DB_CREATE_OR_OPEN
#endif
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
/* Here's the current schema for our database (for NOTMUCH_DATABASE_VERSION):
Add comment documenting our current database schema. I've got schemes to change this schema somewhat dramatically, so I want a place to be able to record and review those changes.
2009-10-25 16:57:09 +01:00
*
lib: Update database schema doc for ghost messages This describes the structure of ghost mail documents. Ghost messages are not yet implemented.
2014-10-23 14:30:34 +02:00
* We currently have three different types of documents (mail, ghost,
* and directory) and also some metadata.
Add comment documenting our current database schema. I've got schemes to change this schema somewhat dramatically, so I want a place to be able to record and review those changes.
2009-10-25 16:57:09 +01:00
*
* Mail document
* -------------
lib: Fix slight misinformation in the database schema doc The database schema documentation made it sound like each mail document had exactly one on-disk message file, which hasn't been true for a long time.
2014-08-01 04:59:13 +02:00
* A mail document is associated with a particular email message. It
* is stored in one or more files on disk (though only one has its
* content indexed) and is uniquely identified by its "id" field
* (which is generally the message ID). It is indexed with the
* following prefixed terms which the database uses to construct
* threads, etc.:
Add comment documenting our current database schema. I've got schemes to change this schema somewhat dramatically, so I want a place to be able to record and review those changes.
2009-10-25 16:57:09 +01:00
*
* Single terms of given prefix:
*
* type: mail
*
Add support (and tests) for messages with really long message IDs. Scott Henson reported an internal error that occurred when he tried to add a message that referenced another message with a message ID well over 300 characters in length. The bug here was running into a Xapian limit for the length of metadata key names, (which is even more restrictive than the Xapian limit for the length of terms). We fix this by noticing long message ID values and instead using a message ID of the form "notmuch-sha1-<sha1_sum_of_message_id>". That is, we use SHA1 to generate a compressed, (but still unique), version of the message ID. We add support to the test suite to exercise this fix. The tests add a message referencing the long message ID, then add the message with the long message ID, then finally add another message referencing the long ID. Each of these tests exercise different code paths where the special handling is implemented. A final test ensures that all three messages are stitched together into a single thread---guaranteeing that the three code paths all act consistently.
2010-06-04 21:39:36 +02:00
* id: Unique ID of mail. This is from the Message-ID header
* if present and not too long (see NOTMUCH_MESSAGE_ID_MAX).
* If it's present and too long, then we use
* "notmuch-sha1-<sha1_sum_of_message_id>".
* If this header is not present, we use
* "notmuch-sha1-<sha1_sum_of_entire_file>".
Add comment documenting our current database schema. I've got schemes to change this schema somewhat dramatically, so I want a place to be able to record and review those changes.
2009-10-25 16:57:09 +01:00
*
Change database to store only a single thread ID per message. Instead of supporting multiple thread IDs, we now merge together thread IDs if one message is ever found to belong to more than one thread. This allows for constructing complete threads when, for example, a child message doesn't include a complete list of References headers back to the beginning of the thread. It also simplifies dealing with mapping a message ID to a thread ID which is now a simple get_thread_id just like get_message_id, (and no longer an iterator-based thing like get_tags).
2009-10-25 22:54:13 +01:00
* thread: The ID of the thread to which the mail belongs
*
database: Add "replyto" to the database schema documentation. Maybe ths lack of this documentation is why I forgot we were actually storing this and wrote the ugly code to fetch In-Reply-To from message files rather than from the database.
2009-11-18 03:48:38 +01:00
* replyto: The ID from the In-Reply-To header of the mail (if any).
*
Add comment documenting our current database schema. I've got schemes to change this schema somewhat dramatically, so I want a place to be able to record and review those changes.
2009-10-25 16:57:09 +01:00
* Multiple terms of given prefix:
*
fix sum moar typos [comments in source code] Various typo fixes in comments within the source code. Signed-off-by: Pieter Praet <pieter@praet.org> Edited-by: Carl Worth <cworth@cworth.org> Restricted to just source-code comments, (and fixed fix of "descriptios" to "descriptors" rather than "descriptions").
2011-06-20 22:14:21 +02:00
* reference: All message IDs from In-Reply-To and References
database: Rename "ref" prefix name to "reference" Which is more consistent with the XREFERENCE prefix used in the terms in the database. Also remove some stale documentation describing the removal of resolved references from the database (we no longer do this).
2009-11-18 03:44:02 +01:00
* headers in the message.
Add comment documenting our current database schema. I've got schemes to change this schema somewhat dramatically, so I want a place to be able to record and review those changes.
2009-10-25 16:57:09 +01:00
*
database: Rename "ref" prefix name to "reference" Which is more consistent with the XREFERENCE prefix used in the terms in the database. Also remove some stale documentation describing the removal of resolved references from the database (we no longer do this).
2009-11-18 03:44:02 +01:00
* tag: Any tags associated with this message by the user.
Add comment documenting our current database schema. I've got schemes to change this schema somewhat dramatically, so I want a place to be able to record and review those changes.
2009-10-25 16:57:09 +01:00
*
lib: Implement new notmuch_directory_t API. This new directory ojbect provides all the infrastructure needed to detect when files or directories are deleted or renamed. There's still code needed on top of this (within "notmuch new") to actually do that detection.
2010-01-05 22:29:23 +01:00
* file-direntry: A colon-separated pair of values
* (INTEGER:STRING), where INTEGER is the
* document ID of a directory document, and
* STRING is the name of a file within that
* directory for this mail message.
database: Store mail filename as a new 'direntry' term, not as 'data'. Instead of storing the complete message filename in the data portion of a mail document we now store a 'direntry' term that contains the document ID of a directory document and also the basename of the message filename within that directory. This will allow us to easily store multple filenames for a single message, and will also allow us to find mail documents for files that previously existed in a directory but that have since been deleted.
2009-12-21 17:23:26 +01:00
*
add property: query prefix to search for specific properties We want to be able to query the properties directly, like: notmuch count property:foo=bar which should return a count of messages where the property with key "foo" has value equal to "bar".
2016-07-08 11:15:36 +02:00
* property: Has a property with key=value
* FIXME: if no = is present, should match on any value
*
Fix comments about what is stored in the database Commit 567bcbc2 introduced two new values for each message (content of the "From" and "Subject" headers), but the comments about the database schema had not been updated accordingly.
2011-12-13 18:11:41 +01:00
* A mail document also has four values:
Add comment documenting our current database schema. I've got schemes to change this schema somewhat dramatically, so I want a place to be able to record and review those changes.
2009-10-25 16:57:09 +01:00
*
* TIMESTAMP: The time_t value corresponding to the message's
* Date header.
*
* MESSAGE_ID: The unique ID of the mail mess (see "id" above)
*
Fix comments about what is stored in the database Commit 567bcbc2 introduced two new values for each message (content of the "From" and "Subject" headers), but the comments about the database schema had not been updated accordingly.
2011-12-13 18:11:41 +01:00
* FROM: The value of the "From" header
*
* SUBJECT: The value of the "Subject" header
*
lib: Add per-message last modification tracking This adds a new document value that stores the revision of the last modification to message metadata, where the revision number increases monotonically with each database commit. An alternative would be to store the wall-clock time of the last modification of each message. In principle this is simpler and has the advantage that any process can determine the current timestamp without support from libnotmuch. However, even assuming a computer's clock never goes backward and ignoring clock skew in networked environments, this has a fatal flaw. Xapian uses (optimistic) snapshot isolation, which means reads can be concurrent with writes. Given this, consider the following time line with a write and two read transactions: write |-X-A--------------| read 1 |---B---| read 2 |---| The write transaction modifies message X and records the wall-clock time of the modification at A. The writer hangs around for a while and later commits its change. Read 1 is concurrent with the write, so it doesn't see the change to X. It does some query and records the wall-clock time of its results at B. Transaction read 2 later starts after the write commits and queries for changes since wall-clock time B (say the reads are performing an incremental backup). Even though read 1 could not see the change to X, read 2 is told (correctly) that X has not changed since B, the time of the last read. In fact, X changed before wall-clock time A, but the change was not visible until *after* wall-clock time B, so read 2 misses the change to X. This is tricky to solve in full-blown snapshot isolation, but because Xapian serializes writes, we can use a simple, monotonically increasing database revision number. Furthermore, maintaining this revision number requires no more IO than a wall-clock time solution because Xapian already maintains statistics on the upper (and lower) bound of each value stream.
2014-10-13 08:20:01 +02:00
* LAST_MOD: The revision number as of the last tag or
* filename change.
*
database: Add "replyto" to the database schema documentation. Maybe ths lack of this documentation is why I forgot we were actually storing this and wrote the ugly code to fetch In-Reply-To from message files rather than from the database.
2009-11-18 03:48:38 +01:00
* In addition, terms from the content of the message are added with
* "from", "to", "attachment", and "subject" prefixes for use by the
Add support for folder-based searching. A new "folder:" prefix in the query string can now be used to match the directories in which mail files are stored. The addition of this feature causes the recently added search-by-folder tests to now pass.
2011-01-15 23:09:04 +01:00
* user in searching. Similarly, terms from the path of the mail
lib: add support for path: prefix searches The path: prefix is a literal boolean prefix matching the paths, relative from the maildir root, of the message files. path:foo matches all message files in foo (but not in foo/new or foo/cur). path:foo/new matches all message files in foo/new. path:"" matches all message files in the top level maildir. path:foo/** matches all message files in foo and recursively in all subdirectories of foo. path:** matches all message files recursively, i.e. all messages.
2014-02-08 19:50:44 +01:00
* message are added with "folder" and "path" prefixes. But the
* database doesn't really care itself about any of these.
database: Add "replyto" to the database schema documentation. Maybe ths lack of this documentation is why I forgot we were actually storing this and wrote the ugly code to fetch In-Reply-To from message files rather than from the database.
2009-11-18 03:48:38 +01:00
*
database: Store mail filename as a new 'direntry' term, not as 'data'. Instead of storing the complete message filename in the data portion of a mail document we now store a 'direntry' term that contains the document ID of a directory document and also the basename of the message filename within that directory. This will allow us to easily store multple filenames for a single message, and will also allow us to find mail documents for files that previously existed in a directory but that have since been deleted.
2009-12-21 17:23:26 +01:00
* The data portion of a mail document is empty.
lib: Document that the filename is stored in the 'data' of a mail document Our database schema documentation previously didn't give any indication of where this most essential piece of information is stored.
2009-12-19 22:05:06 +01:00
*
lib: Update database schema doc for ghost messages This describes the structure of ghost mail documents. Ghost messages are not yet implemented.
2014-10-23 14:30:34 +02:00
* Ghost mail document [if NOTMUCH_FEATURE_GHOSTS]
* -----------------------------------------------
* A ghost mail document is like a mail document, but where we don't
* have the message content. These are used to track thread reference
* information for messages we haven't received.
*
* A ghost mail document has type: ghost; id and thread fields that
* are identical to the mail document fields; and a MESSAGE_ID value.
*
lib: Rename set/get_timestamp to set/get_directory_mtime. I've been suitably scolded by Keith for doing a premature generalization that ended up just making the documentation more convoluted. Fix that.
2009-12-17 23:33:34 +01:00
* Directory document
Add comment documenting our current database schema. I've got schemes to change this schema somewhat dramatically, so I want a place to be able to record and review those changes.
2009-10-25 16:57:09 +01:00
* ------------------
lib: Rename set/get_timestamp to set/get_directory_mtime. I've been suitably scolded by Keith for doing a premature generalization that ended up just making the documentation more convoluted. Fix that.
2009-12-17 23:33:34 +01:00
* A directory document is used by a client of the notmuch library to
Add comment documenting our current database schema. I've got schemes to change this schema somewhat dramatically, so I want a place to be able to record and review those changes.
2009-10-25 16:57:09 +01:00
* maintain data necessary to allow for efficient polling of mail
lib: Rename set/get_timestamp to set/get_directory_mtime. I've been suitably scolded by Keith for doing a premature generalization that ended up just making the documentation more convoluted. Fix that.
2009-12-17 23:33:34 +01:00
* directories.
Add comment documenting our current database schema. I've got schemes to change this schema somewhat dramatically, so I want a place to be able to record and review those changes.
2009-10-25 16:57:09 +01:00
*
lib: Implement new notmuch_directory_t API. This new directory ojbect provides all the infrastructure needed to detect when files or directories are deleted or renamed. There's still code needed on top of this (within "notmuch new") to actually do that detection.
2010-01-05 22:29:23 +01:00
* All directory documents contain one term:
Add comment documenting our current database schema. I've got schemes to change this schema somewhat dramatically, so I want a place to be able to record and review those changes.
2009-10-25 16:57:09 +01:00
*
database: Store directory paths as relative, not absolute. We were already storing relative mail filenames, so this is consistent with that. Additionally, it means that directory documents remain valid even if the database is relocated within its containing filesystem.
2009-12-19 22:11:00 +01:00
* directory: The directory path (relative to the database path)
database: Store directory path in 'data' of directory documents. We're planning to have mail documents refer to directory documents for the path of the containing directory. To support this, we need the path in the data, (since the path in the 'directory' term can be irretrievable as it will be the SHA1 sum of the path for a very long path).
2009-12-21 00:46:41 +01:00
* Or the SHA1 sum of the directory path (if the
* path itself is too long to fit in a Xapian
* term).
*
lib: Implement new notmuch_directory_t API. This new directory ojbect provides all the infrastructure needed to detect when files or directories are deleted or renamed. There's still code needed on top of this (within "notmuch new") to actually do that detection.
2010-01-05 22:29:23 +01:00
* And all directory documents for directories other than top-level
* directories also contain the following term:
Add comment documenting our current database schema. I've got schemes to change this schema somewhat dramatically, so I want a place to be able to record and review those changes.
2009-10-25 16:57:09 +01:00
*
lib: Implement new notmuch_directory_t API. This new directory ojbect provides all the infrastructure needed to detect when files or directories are deleted or renamed. There's still code needed on top of this (within "notmuch new") to actually do that detection.
2010-01-05 22:29:23 +01:00
* directory-direntry: A colon-separated pair of values
* (INTEGER:STRING), where INTEGER is the
* document ID of the parent directory
* document, and STRING is the name of this
* directory within that parent.
*
* All directory documents have a single value:
Add comment documenting our current database schema. I've got schemes to change this schema somewhat dramatically, so I want a place to be able to record and review those changes.
2009-10-25 16:57:09 +01:00
*
lib: Rename set/get_timestamp to set/get_directory_mtime. I've been suitably scolded by Keith for doing a premature generalization that ended up just making the documentation more convoluted. Fix that.
2009-12-17 23:33:34 +01:00
* TIMESTAMP: The mtime of the directory (at last scan)
database: Store directory path in 'data' of directory documents. We're planning to have mail documents refer to directory documents for the path of the containing directory. To support this, we need the path in the data, (since the path in the 'directory' term can be irretrievable as it will be the SHA1 sum of the path for a very long path).
2009-12-21 00:46:41 +01:00
*
* The data portion of a directory document contains the path of the
lib: Implement new notmuch_directory_t API. This new directory ojbect provides all the infrastructure needed to detect when files or directories are deleted or renamed. There's still code needed on top of this (within "notmuch new") to actually do that detection.
2010-01-05 22:29:23 +01:00
* directory (relative to the database path).
lib: Document the metadata stored within the Xapian database. We are currently storing "version", "last_thread_id", and "thread_id_*" values so document how each of these are used.
2010-04-13 00:15:14 +02:00
*
* Database metadata
* -----------------
* Xapian allows us to store arbitrary name-value pairs as
* "metadata". We currently use the following metadata names with the
* given meanings:
*
* version The database schema version, (which is distinct
* from both the notmuch package version (see
* notmuch --version) and the libnotmuch library
* version. The version is stored as an base-10
* ASCII integer. The initial database version
* was 1, (though a schema existed before that
* were no "version" database value existed at
fix sum moar typos [comments in source code] Various typo fixes in comments within the source code. Signed-off-by: Pieter Praet <pieter@praet.org> Edited-by: Carl Worth <cworth@cworth.org> Restricted to just source-code comments, (and fixed fix of "descriptios" to "descriptors" rather than "descriptions").
2011-06-20 22:14:21 +02:00
* all). Successive versions are allocated as
lib: Document the metadata stored within the Xapian database. We are currently storing "version", "last_thread_id", and "thread_id_*" values so document how each of these are used.
2010-04-13 00:15:14 +02:00
* changes are made to the database (such as by
* indexing new fields).
*
lib: Database version 3: Introduce fine-grained "features" Previously, our database schema was versioned by a single number. Each database schema change had to occur "atomically" in Notmuch's development history: before some commit, Notmuch used version N, after that commit, it used version N+1. Hence, each new schema version could introduce only one change, the task of developing a schema change fell on a single person, and it all had to happen and be perfect in a single commit series. This made introducing a new schema version hard. We've seen only two schema changes in the history of Notmuch. This commit introduces database schema version 3; hopefully the last schema version we'll need for a while. With this version, we switch from a single version number to "features": a set of named, independent aspects of the database schema. Features should make backwards compatibility easier. For many things, it should be easy to support databases both with and without a feature, which will allow us to make upgrades optional and will enable "unstable" features that can be developed and tested over time. Features also make forwards compatibility easier. The features recorded in a database include "compatibility flags," which can indicate to an older version of Notmuch when it must support a given feature to open the database for read or for write. This lets us replace the old vague "I don't recognize this version, so something might go wrong, but I promise to try my best" warnings upon opening a database with an unknown version with precise errors. If a database is safe to open for read/write despite unknown features, an older version will know that and issue no message at all. If the database is not safe to open for read/write because of unknown features, an older version will know that, too, and can tell the user exactly which required features it lacks support for.
2014-08-25 19:26:00 +02:00
* features The set of features supported by this
* database. This consists of a set of
* '\n'-separated lines, where each is a feature
* name, a '\t', and compatibility flags. If the
* compatibility flags contain 'w', then the
* opener must support this feature to safely
* write this database. If the compatibility
* flags contain 'r', then the opener must
* support this feature to read this database.
* Introduced in database version 3.
*
lib: Document the metadata stored within the Xapian database. We are currently storing "version", "last_thread_id", and "thread_id_*" values so document how each of these are used.
2010-04-13 00:15:14 +02:00
* last_thread_id The last thread ID generated. This is stored
* as a 16-byte hexadecimal ASCII representation
* of a 64-bit unsigned integer. The first ID
* generated is 1 and the value will be
* incremented for each thread ID.
*
lib: document config metadata This probably should have been part of 3458e3c89c, but I missed it.
2016-06-05 00:17:41 +02:00
* C* metadata keys starting with C indicate
* configuration data. It can be managed with the
* n_database_*config* API. There is a convention
* of hierarchical keys separated by '.' (e.g.
* query.notmuch stores the value for the named
* query 'notmuch'), but it is not enforced by the
* API.
*
lib: Update database schema doc for ghost messages This describes the structure of ghost mail documents. Ghost messages are not yet implemented.
2014-10-23 14:30:34 +02:00
* Obsolete metadata
* -----------------
*
* If ! NOTMUCH_FEATURE_GHOSTS, there are no ghost mail documents.
* Instead, the database has the following additional database
* metadata:
*
lib: Document the metadata stored within the Xapian database. We are currently storing "version", "last_thread_id", and "thread_id_*" values so document how each of these are used.
2010-04-13 00:15:14 +02:00
* thread_id_* A pre-allocated thread ID for a particular
fix sum moar typos [comments in source code] Various typo fixes in comments within the source code. Signed-off-by: Pieter Praet <pieter@praet.org> Edited-by: Carl Worth <cworth@cworth.org> Restricted to just source-code comments, (and fixed fix of "descriptios" to "descriptors" rather than "descriptions").
2011-06-20 22:14:21 +02:00
* message. This is actually an arbitrarily large
Add support (and tests) for messages with really long message IDs. Scott Henson reported an internal error that occurred when he tried to add a message that referenced another message with a message ID well over 300 characters in length. The bug here was running into a Xapian limit for the length of metadata key names, (which is even more restrictive than the Xapian limit for the length of terms). We fix this by noticing long message ID values and instead using a message ID of the form "notmuch-sha1-<sha1_sum_of_message_id>". That is, we use SHA1 to generate a compressed, (but still unique), version of the message ID. We add support to the test suite to exercise this fix. The tests add a message referencing the long message ID, then add the message with the long message ID, then finally add another message referencing the long ID. Each of these tests exercise different code paths where the special handling is implemented. A final test ensures that all three messages are stitched together into a single thread---guaranteeing that the three code paths all act consistently.
2010-06-04 21:39:36 +02:00
* family of metadata name. Any particular name is
* formed by concatenating "thread_id_" with a message
* ID (or the SHA1 sum of a message ID if it is very
* long---see description of 'id' in the mail
* document). The value stored is a thread ID.
lib: Document the metadata stored within the Xapian database. We are currently storing "version", "last_thread_id", and "thread_id_*" values so document how each of these are used.
2010-04-13 00:15:14 +02:00
*
* These thread ID metadata values are stored
* whenever a message references a parent message
* that does not yet exist in the database. A
* thread ID will be allocated and stored, and if
* the message is later added, the stored thread
* ID will be used (and the metadata value will
* be cleared).
*
* Even before a message is added, it's
* pre-allocated thread ID is useful so that all
* descendant messages that reference this common
* parent can be recognized as belonging to the
* same thread.
Add comment documenting our current database schema. I've got schemes to change this schema somewhat dramatically, so I want a place to be able to record and review those changes.
2009-10-25 16:57:09 +01:00
*/
Invent our own prefix values. We're now dropping all pretense of keeping the database directly compatible with sup's current xapian backend. (But perhaps someone might write a new nothmuch backend for sup in the future.) In coming up with the prefix values here, I tried to follow the conventions of http://xapian.org/docs/omega/termprefixes.html as closely as makes sense, (with some domain translation from "web" to "email archive").
2009-10-25 06:49:35 +01:00
/* With these prefix values we follow the conventions published here:
*
Use https instead of http where possible Many of the external links found in the notmuch source can be resolved using https instead of http. This changeset addresses as many as i could find, without touching the e-mail corpus or expected outputs found in tests.
2016-06-02 18:26:14 +02:00
* https://xapian.org/docs/omega/termprefixes.html
Invent our own prefix values. We're now dropping all pretense of keeping the database directly compatible with sup's current xapian backend. (But perhaps someone might write a new nothmuch backend for sup in the future.) In coming up with the prefix values here, I tried to follow the conventions of http://xapian.org/docs/omega/termprefixes.html as closely as makes sense, (with some domain translation from "web" to "email archive").
2009-10-25 06:49:35 +01:00
*
* as much as makes sense. Note that I took some liberty in matching
* the reserved prefix values to notmuch concepts, (for example, 'G'
* is documented as "newsGroup (or similar entity - e.g. a web forum
* name)", for which I think the thread is the closest analogue in
* notmuch. This in spite of the fact that we will eventually be
* storing mailing-list messages where 'G' for "mailing list name"
* might be even a closer analogue. I'm treating the single-character
* prefixes preferentially for core notmuch concepts (which will be
* nearly universal to all mail messages).
*/
lib: merge internal prefix tables Replace multiple tables with some flags in a single table. This makes the code in notmuch_database_open_verbose a bit shorter, and it should also make it easier to add other options to fields, e.g. regexp searching.
2017-02-17 04:07:49 +01:00
static const
prefix_t prefix_table[] = {
/* name term prefix flags */
{ "type", "T", NOTMUCH_FIELD_NO_FLAGS },
{ "reference", "XREFERENCE", NOTMUCH_FIELD_NO_FLAGS },
{ "replyto", "XREPLYTO", NOTMUCH_FIELD_NO_FLAGS },
{ "directory", "XDIRECTORY", NOTMUCH_FIELD_NO_FLAGS },
{ "file-direntry", "XFDIRENTRY", NOTMUCH_FIELD_NO_FLAGS },
{ "directory-direntry", "XDDIRENTRY", NOTMUCH_FIELD_NO_FLAGS },
{ "thread", "G", NOTMUCH_FIELD_EXTERNAL },
{ "tag", "K", NOTMUCH_FIELD_EXTERNAL },
{ "is", "K", NOTMUCH_FIELD_EXTERNAL },
{ "id", "Q", NOTMUCH_FIELD_EXTERNAL },
{ "path", "P", NOTMUCH_FIELD_EXTERNAL },
{ "property", "XPROPERTY", NOTMUCH_FIELD_EXTERNAL },
lib: make folder: prefix literal In xapian terms, convert folder: prefix from probabilistic to boolean prefix, matching the paths, relative from the maildir root, of the message files, ignoring the maildir new and cur leaf directories. folder:foo matches all message files in foo, foo/new, and foo/cur. folder:foo/new does *not* match message files in foo/new. folder:"" matches all message files in the top level maildir and its new and cur subdirectories. This change constitutes a database change: bump the database version and add database upgrade support for folder: terms. The upgrade also adds path: terms. Finally, fix the folder search test for literal folder: search, as some of the folder: matching capabilities are lost in the probabilistic to boolean prefix change.
2014-02-08 20:20:42 +01:00
/*
lib: reword comment about XFOLDER: prefix I believe the current one is misleading, because in my experiments Xapian did not add : when prefix and term were both upper case. Indeed, it's hard to see how it could, because prefixes are added at a layer above Xapian in our code. See _notmuch_message_add_term for an example. Also try to explain why this is a good idea. As far as I can ascertain, this is more of an issue for a system trying to work with an unknown set of prefixes. Since notmuch has a fixed set of prefixes, and we can hopefully be trusted not to add XGOLD and XGOLDEN as prefixes, it is harder for problems to arise.
2016-08-18 10:06:54 +02:00
* Unconditionally add ':' to reduce potential ambiguity with
* overlapping prefixes and/or terms that start with capital
* letters. See Xapian document termprefixes.html for related
* discussion.
lib: make folder: prefix literal In xapian terms, convert folder: prefix from probabilistic to boolean prefix, matching the paths, relative from the maildir root, of the message files, ignoring the maildir new and cur leaf directories. folder:foo matches all message files in foo, foo/new, and foo/cur. folder:foo/new does *not* match message files in foo/new. folder:"" matches all message files in the top level maildir and its new and cur subdirectories. This change constitutes a database change: bump the database version and add database upgrade support for folder: terms. The upgrade also adds path: terms. Finally, fix the folder search test for literal folder: search, as some of the folder: matching capabilities are lost in the probabilistic to boolean prefix change.
2014-02-08 20:20:42 +01:00
*/
lib: merge internal prefix tables Replace multiple tables with some flags in a single table. This makes the code in notmuch_database_open_verbose a bit shorter, and it should also make it easier to add other options to fields, e.g. regexp searching.
2017-02-17 04:07:49 +01:00
{ "folder", "XFOLDER:", NOTMUCH_FIELD_EXTERNAL },
{ "from", "XFROM", NOTMUCH_FIELD_EXTERNAL |
NOTMUCH_FIELD_PROBABILISTIC },
{ "to", "XTO", NOTMUCH_FIELD_EXTERNAL |
NOTMUCH_FIELD_PROBABILISTIC },
{ "attachment", "XATTACHMENT", NOTMUCH_FIELD_EXTERNAL |
NOTMUCH_FIELD_PROBABILISTIC },
{ "mimetype", "XMIMETYPE", NOTMUCH_FIELD_EXTERNAL |
NOTMUCH_FIELD_PROBABILISTIC },
{ "subject", "XSUBJECT", NOTMUCH_FIELD_EXTERNAL |
NOTMUCH_FIELD_PROBABILISTIC },
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
};
Move the prefix-string arrays back into database.cc from message.cc Yes, I'm being wishy-washy here, moving code back and forth. But this is where these really do belong.
2009-10-25 05:52:48 +01:00
const char *
_find_prefix (const char *name)
{
unsigned int i;
lib: merge internal prefix tables Replace multiple tables with some flags in a single table. This makes the code in notmuch_database_open_verbose a bit shorter, and it should also make it easier to add other options to fields, e.g. regexp searching.
2017-02-17 04:07:49 +01:00
for (i = 0; i < ARRAY_SIZE (prefix_table); i++) {
if (strcmp (name, prefix_table[i].name) == 0)
return prefix_table[i].prefix;
lib/database.cc: coding style Carl claims he must have been distracted when he wrote this... Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
2009-11-22 00:14:39 +01:00
}
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
Add an INTERNAL_ERROR macro and use it for all internal errors. We were previously just doing fprintf;exit at each point, but I wanted to add file and line-number details to all messages, so it makes sense to use a single macro for that.
2009-10-25 18:22:41 +01:00
INTERNAL_ERROR ("No prefix exists for '%s'\n", name);
Move the prefix-string arrays back into database.cc from message.cc Yes, I'm being wishy-washy here, moving code back and forth. But this is where these really do belong.
2009-10-25 05:52:48 +01:00
return "";
}
lib: Database version 3: Introduce fine-grained "features" Previously, our database schema was versioned by a single number. Each database schema change had to occur "atomically" in Notmuch's development history: before some commit, Notmuch used version N, after that commit, it used version N+1. Hence, each new schema version could introduce only one change, the task of developing a schema change fell on a single person, and it all had to happen and be perfect in a single commit series. This made introducing a new schema version hard. We've seen only two schema changes in the history of Notmuch. This commit introduces database schema version 3; hopefully the last schema version we'll need for a while. With this version, we switch from a single version number to "features": a set of named, independent aspects of the database schema. Features should make backwards compatibility easier. For many things, it should be easy to support databases both with and without a feature, which will allow us to make upgrades optional and will enable "unstable" features that can be developed and tested over time. Features also make forwards compatibility easier. The features recorded in a database include "compatibility flags," which can indicate to an older version of Notmuch when it must support a given feature to open the database for read or for write. This lets us replace the old vague "I don't recognize this version, so something might go wrong, but I promise to try my best" warnings upon opening a database with an unknown version with precise errors. If a database is safe to open for read/write despite unknown features, an older version will know that and issue no message at all. If the database is not safe to open for read/write because of unknown features, an older version will know that, too, and can tell the user exactly which required features it lacks support for.
2014-08-25 19:26:00 +02:00
static const struct {
/* NOTMUCH_FEATURE_* value. */
_notmuch_features value;
/* Feature name as it appears in the database. This name should
* be appropriate for displaying to the user if an older version
* of notmuch doesn't support this feature. */
const char *name;
/* Compatibility flags when this feature is declared. */
const char *flags;
} feature_names[] = {
{ NOTMUCH_FEATURE_FILE_TERMS,
"multiple paths per message", "rw" },
{ NOTMUCH_FEATURE_DIRECTORY_DOCS,
"relative directory paths", "rw" },
/* Header values are not required for reading a database because a
* reader can just refer to the message file. */
{ NOTMUCH_FEATURE_FROM_SUBJECT_ID_VALUES,
"from/subject/message-ID in database", "w" },
{ NOTMUCH_FEATURE_BOOL_FOLDER,
"exact folder:/path: search", "rw" },
lib: Add a ghost messages database feature This will be implemented over the next several patches. The feature is not yet "enabled" (this does not add it to NOTMUCH_FEATURES_CURRENT).
2014-10-23 14:30:33 +02:00
{ NOTMUCH_FEATURE_GHOSTS,
"mail documents for missing messages", "w"},
Add the NOTMUCH_FEATURE_INDEXED_MIMETYPES database feature This feature will exist in all newly created databases, but there is no upgrade provided for it. If this flag exists, it indicates that the database was created after the indexed MIME-types feature was added.
2015-01-23 00:43:37 +01:00
/* Knowledge of the index mime-types are not required for reading
* a database because a reader will just be unable to query
* them. */
{ NOTMUCH_FEATURE_INDEXED_MIMETYPES,
"indexed MIME types", "w"},
lib: Add per-message last modification tracking This adds a new document value that stores the revision of the last modification to message metadata, where the revision number increases monotonically with each database commit. An alternative would be to store the wall-clock time of the last modification of each message. In principle this is simpler and has the advantage that any process can determine the current timestamp without support from libnotmuch. However, even assuming a computer's clock never goes backward and ignoring clock skew in networked environments, this has a fatal flaw. Xapian uses (optimistic) snapshot isolation, which means reads can be concurrent with writes. Given this, consider the following time line with a write and two read transactions: write |-X-A--------------| read 1 |---B---| read 2 |---| The write transaction modifies message X and records the wall-clock time of the modification at A. The writer hangs around for a while and later commits its change. Read 1 is concurrent with the write, so it doesn't see the change to X. It does some query and records the wall-clock time of its results at B. Transaction read 2 later starts after the write commits and queries for changes since wall-clock time B (say the reads are performing an incremental backup). Even though read 1 could not see the change to X, read 2 is told (correctly) that X has not changed since B, the time of the last read. In fact, X changed before wall-clock time A, but the change was not visible until *after* wall-clock time B, so read 2 misses the change to X. This is tricky to solve in full-blown snapshot isolation, but because Xapian serializes writes, we can use a simple, monotonically increasing database revision number. Furthermore, maintaining this revision number requires no more IO than a wall-clock time solution because Xapian already maintains statistics on the upper (and lower) bound of each value stream.
2014-10-13 08:20:01 +02:00
{ NOTMUCH_FEATURE_LAST_MOD,
"modification tracking", "w"},
lib: Database version 3: Introduce fine-grained "features" Previously, our database schema was versioned by a single number. Each database schema change had to occur "atomically" in Notmuch's development history: before some commit, Notmuch used version N, after that commit, it used version N+1. Hence, each new schema version could introduce only one change, the task of developing a schema change fell on a single person, and it all had to happen and be perfect in a single commit series. This made introducing a new schema version hard. We've seen only two schema changes in the history of Notmuch. This commit introduces database schema version 3; hopefully the last schema version we'll need for a while. With this version, we switch from a single version number to "features": a set of named, independent aspects of the database schema. Features should make backwards compatibility easier. For many things, it should be easy to support databases both with and without a feature, which will allow us to make upgrades optional and will enable "unstable" features that can be developed and tested over time. Features also make forwards compatibility easier. The features recorded in a database include "compatibility flags," which can indicate to an older version of Notmuch when it must support a given feature to open the database for read or for write. This lets us replace the old vague "I don't recognize this version, so something might go wrong, but I promise to try my best" warnings upon opening a database with an unknown version with precise errors. If a database is safe to open for read/write despite unknown features, an older version will know that and issue no message at all. If the database is not safe to open for read/write because of unknown features, an older version will know that, too, and can tell the user exactly which required features it lacks support for.
2014-08-25 19:26:00 +02:00
};
Add notmuch_status_to_string function. Be kind and let the user print error messages, not just error codes.
2009-10-22 01:12:53 +02:00
const char *
notmuch_status_to_string (notmuch_status_t status)
{
switch (status) {
case NOTMUCH_STATUS_SUCCESS:
return "No error occurred";
Add -Wswitch-enum and fix warnings. Having to enumerate all the enum values at every switch is annoying, but this warning actually found a bug, (missing support for NOTMUCH_STATUS_OUT_OF_MEMORY in notmuch_status_to_string).
2009-10-26 00:03:45 +01:00
case NOTMUCH_STATUS_OUT_OF_MEMORY:
return "Out of memory";
Prefer READ_ONLY consistently over READONLY. Previously we had NOTMUCH_DATABASE_MODE_READ_ONLY but NOTMUCH_STATUS_READONLY_DATABASE which was ugly and confusing. Rename the latter to NOTMUCH_STATUS_READ_ONLY_DATABASE for consistency.
2010-01-07 19:29:05 +01:00
case NOTMUCH_STATUS_READ_ONLY_DATABASE:
Add missing comment for NOTMUCH_STATUS_READONLY_DATABASE. And adjust the string representation of the same to match.
2010-01-06 00:01:58 +01:00
return "Attempt to write to a read-only database";
Add notmuch_status_to_string function. Be kind and let the user print error messages, not just error codes.
2009-10-22 01:12:53 +02:00
case NOTMUCH_STATUS_XAPIAN_EXCEPTION:
return "A Xapian exception occurred";
Generate message ID (using SHA1) when a mail message contains none. This is important as we're using the message ID as the unique key in our database. So previously, all messages with no message ID would be treated as the same message---not good at all.
2009-10-23 00:31:56 +02:00
case NOTMUCH_STATUS_FILE_ERROR:
return "Something went wrong trying to read or write a file";
Add notmuch_status_to_string function. Be kind and let the user print error messages, not just error codes.
2009-10-22 01:12:53 +02:00
case NOTMUCH_STATUS_FILE_NOT_EMAIL:
return "File is not an email";
Add NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID And document that notmuch_database_add_message can return this value. This pushes the hard decision of what to do with duplicate messages out to the user, but that's OK. (We weren't really doing anything with these ourselves, and this way the user is at least informed of the issue, rather than it just getting papered over internally.)
2009-10-23 23:40:33 +02:00
case NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:
return "Message ID is identical to a message in database";
Add notmuch_status_to_string function. Be kind and let the user print error messages, not just error codes.
2009-10-22 01:12:53 +02:00
case NOTMUCH_STATUS_NULL_POINTER:
return "Erroneous NULL pointer";
case NOTMUCH_STATUS_TAG_TOO_LONG:
Clarify documentation and error string for NOTMUCH_STATUS_TAG_TOO_LONG It's helpful to point out NOTMUCH_STATUS_TAG_MAX for users.
2009-10-23 23:34:21 +02:00
return "Tag value is too long (exceeds NOTMUCH_TAG_MAX)";
notmuch restore: Fix to remove all tags before adding tags. This means that the restore operation will now properly pick up the removal of tags indicated by the tag just not being present in the dump file. We added a few new public functions in order to support this: notmuch_message_freeze notmuch_message_remove_all_tags notmuch_message_thaw
2009-10-27 06:25:45 +01:00
case NOTMUCH_STATUS_UNBALANCED_FREEZE_THAW:
Typsos
2009-11-18 00:23:42 +01:00
return "Unbalanced number of calls to notmuch_message_freeze/thaw";
lib: Add support for nested atomic sections. notmuch_database_t now keeps a nesting count and we only start a transaction or commit for the outermost atomic section. Introduces a new error, NOTMUCH_STATUS_UNBALANCED_ATOMIC.
2011-06-11 05:35:06 +02:00
case NOTMUCH_STATUS_UNBALANCED_ATOMIC:
return "Unbalanced number of calls to notmuch_database_begin_atomic/end_atomic";
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
case NOTMUCH_STATUS_UNSUPPORTED_OPERATION:
return "Unsupported operation";
lib: Return an error from operations that require an upgrade Previously, there was no protection against a caller invoking an operation on an old database version that would effectively corrupt the database by treating it like a newer version. According to notmuch.h, any caller that opens the database in read/write mode is supposed to check if the database needs upgrading and perform an upgrade if it does. This would protect against this, but nobody (even the CLI) actually does this. However, with features, it's easy to protect against incompatible operations on a fine-grained basis. This lightweight change allows callers to safely operate on old database versions, while preventing specific operations that would corrupt the database with an informative error message.
2014-08-25 19:26:08 +02:00
case NOTMUCH_STATUS_UPGRADE_REQUIRED:
return "Operation requires a database upgrade";
lib: add NOTMUCH_STATUS_PATH_ERROR The difference with FILE_ERROR is that this is for things that are wrong with the path before looking at the disk. Add some 3 tests; two broken as a reminder to actually use this new code.
2015-06-10 07:58:44 +02:00
case NOTMUCH_STATUS_PATH_ERROR:
return "Path supplied is illegal for this function";
Add notmuch_status_to_string function. Be kind and let the user print error messages, not just error codes.
2009-10-22 01:12:53 +02:00
default:
case NOTMUCH_STATUS_LAST_STATUS:
return "Unknown error status value";
}
}
lib: add a log function with output to a string in notmuch_database_t In principle in the future this could do something fancier than asprintf.
2014-12-26 09:01:01 +01:00
void
_notmuch_database_log (notmuch_database_t *notmuch,
const char *format,
...)
{
va_list va_args;
va_start (va_args, format);
if (notmuch->status_string)
talloc_free (notmuch->status_string);
notmuch->status_string = talloc_vasprintf (notmuch, format, va_args);
lib: provide _notmuch_database_log_append _notmuch_database_log clears the log buffer each time. Rather than introducing more complicated semantics about for this function, provide a second function that does not clear the buffer. This is mainly a convenience function for callers constructing complex or multi-line log messages. The changes to query.cc are to make sure that the common code path of the new function is tested.
2016-07-15 12:25:41 +02:00
va_end (va_args);
}
void
_notmuch_database_log_append (notmuch_database_t *notmuch,
const char *format,
...)
{
va_list va_args;
va_start (va_args, format);
if (notmuch->status_string)
notmuch->status_string = talloc_vasprintf_append (notmuch->status_string, format, va_args);
else
notmuch->status_string = talloc_vasprintf (notmuch, format, va_args);
lib: add a log function with output to a string in notmuch_database_t In principle in the future this could do something fancier than asprintf.
2014-12-26 09:01:01 +01:00
va_end (va_args);
}
database: Add new find_doc_ids_for_term interface. The existing find_doc_ids function is convenient when the caller doesn't want to be bothered constructing a term. But when the caller *does* have the term already, that interface is just wasteful. So we export a lower-level interface that maps a pre-constructed term to a document-ID iterators.
2009-12-22 00:12:52 +01:00
static void
find_doc_ids_for_term (notmuch_database_t *notmuch,
const char *term,
Xapian::PostingIterator *begin,
Xapian::PostingIterator *end)
{
*begin = notmuch->xapian_db->postlist_begin (term);
*end = notmuch->xapian_db->postlist_end (term);
}
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
static void
database: Rename internal find_messages_by_term to find_doc_ids This name is a more accurate description of what it does, and the more general naming will make sense as we start storing non-message documents in the database (such as directory timestamps). Also, don't pass around a Xapian::Database where it's more our style to pass a notmuch_database_t*.
2009-10-23 23:06:24 +02:00
find_doc_ids (notmuch_database_t *notmuch,
const char *prefix_name,
const char *value,
Xapian::PostingIterator *begin,
Xapian::PostingIterator *end)
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
{
char *term;
Remove all calls to g_strdup_printf Replacing them with calls to talloc_asprintf if possible, otherwise to asprintf (with it's painful error-handling leaving the pointer undefined).
2009-10-26 23:17:10 +01:00
term = talloc_asprintf (notmuch, "%s%s",
_find_prefix (prefix_name), value);
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
database: Add new find_doc_ids_for_term interface. The existing find_doc_ids function is convenient when the caller doesn't want to be bothered constructing a term. But when the caller *does* have the term already, that interface is just wasteful. So we export a lower-level interface that maps a pre-constructed term to a document-ID iterators.
2009-12-22 00:12:52 +01:00
find_doc_ids_for_term (notmuch, term, begin, end);
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
Remove all calls to g_strdup_printf Replacing them with calls to talloc_asprintf if possible, otherwise to asprintf (with it's painful error-handling leaving the pointer undefined).
2009-10-26 23:17:10 +01:00
talloc_free (term);
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
}
lib: Implement new notmuch_directory_t API. This new directory ojbect provides all the infrastructure needed to detect when files or directories are deleted or renamed. There's still code needed on top of this (within "notmuch new") to actually do that detection.
2010-01-05 22:29:23 +01:00
notmuch_private_status_t
_notmuch_database_find_unique_doc_id (notmuch_database_t *notmuch,
const char *prefix_name,
const char *value,
unsigned int *doc_id)
database: Add private find_unique_doc_id and find_unique_document functions These are a generalization of the unique-ness testing of notmuch_database_find_message. More preparation for firectory timestamps.
2009-10-23 23:24:07 +02:00
{
Xapian::PostingIterator i, end;
find_doc_ids (notmuch, prefix_name, value, &i, &end);
if (i == end) {
*doc_id = 0;
return NOTMUCH_PRIVATE_STATUS_NO_DOCUMENT_FOUND;
}
database: Make find_unique_doc_id enforce uniqueness (for a debug build) Catching any violation of this unique-ness constraint is very much in line with similar, existing INTERNAL_ERROR cases.
2009-12-22 00:11:32 +01:00
*doc_id = *i;
#if DEBUG_DATABASE_SANITY
i++;
if (i != end)
INTERNAL_ERROR ("Term %s:%s is not unique as expected.\n",
prefix_name, value);
#endif
return NOTMUCH_PRIVATE_STATUS_SUCCESS;
database: Add private find_unique_doc_id and find_unique_document functions These are a generalization of the unique-ness testing of notmuch_database_find_message. More preparation for firectory timestamps.
2009-10-23 23:24:07 +02:00
}
database: Similarly rename find_message_by_docid to find_document_for_doc_id Again preferring notmuch_database_t* over Xapian::Database*. Also, we're standardizing on "doc_id" rather than "docid" locally, (as an analoge to "message_id"), in spite of the "Xapian::docid" name, (which, fortunately, we can ignore and just us "unsigned int" instead).
2009-10-23 23:12:06 +02:00
static Xapian::Document
find_document_for_doc_id (notmuch_database_t *notmuch, unsigned doc_id)
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
{
database: Similarly rename find_message_by_docid to find_document_for_doc_id Again preferring notmuch_database_t* over Xapian::Database*. Also, we're standardizing on "doc_id" rather than "docid" locally, (as an analoge to "message_id"), in spite of the "Xapian::docid" name, (which, fortunately, we can ignore and just us "unsigned int" instead).
2009-10-23 23:12:06 +02:00
return notmuch->xapian_db->get_document (doc_id);
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
}
Add support (and tests) for messages with really long message IDs. Scott Henson reported an internal error that occurred when he tried to add a message that referenced another message with a message ID well over 300 characters in length. The bug here was running into a Xapian limit for the length of metadata key names, (which is even more restrictive than the Xapian limit for the length of terms). We fix this by noticing long message ID values and instead using a message ID of the form "notmuch-sha1-<sha1_sum_of_message_id>". That is, we use SHA1 to generate a compressed, (but still unique), version of the message ID. We add support to the test suite to exercise this fix. The tests add a message referencing the long message ID, then add the message with the long message ID, then finally add another message referencing the long ID. Each of these tests exercise different code paths where the special handling is implemented. A final test ensures that all three messages are stitched together into a single thread---guaranteeing that the three code paths all act consistently.
2010-06-04 21:39:36 +02:00
/* Generate a compressed version of 'message_id' of the form:
*
* notmuch-sha1-<sha1_sum_of_message_id>
*/
lib: Move message ID compression to _notmuch_message_create_for_message_id Previously, this was performed by notmuch_database_add_message. This happens to be the only caller currently (which is why this was safe), but we're about to introduce more callers, and it makes more sense to put responsibility for ID compression in the lower-level function rather than requiring each caller to handle it.
2014-10-07 01:17:07 +02:00
char *
_notmuch_message_id_compressed (void *ctx, const char *message_id)
Add support (and tests) for messages with really long message IDs. Scott Henson reported an internal error that occurred when he tried to add a message that referenced another message with a message ID well over 300 characters in length. The bug here was running into a Xapian limit for the length of metadata key names, (which is even more restrictive than the Xapian limit for the length of terms). We fix this by noticing long message ID values and instead using a message ID of the form "notmuch-sha1-<sha1_sum_of_message_id>". That is, we use SHA1 to generate a compressed, (but still unique), version of the message ID. We add support to the test suite to exercise this fix. The tests add a message referencing the long message ID, then add the message with the long message ID, then finally add another message referencing the long ID. Each of these tests exercise different code paths where the special handling is implemented. A final test ensures that all three messages are stitched together into a single thread---guaranteeing that the three code paths all act consistently.
2010-06-04 21:39:36 +02:00
{
char *sha1, *compressed;
lib: Start all function names in notmuch-private.h with As noted in devel/STYLE, every private library function should start with _notmuch. This patch corrects function naming that did not adhere to this style in lib/notmuch-private.h. In particular, the old function names that now begin with _notmuch are notmuch_sha1_of_file notmuch_sha1_of_string notmuch_message_file_close notmuch_message_file_get_header notmuch_message_file_open notmuch_message_get_author notmuch_message_set_author Signed-off-by: Charles Celerier <cceleri@cs.stanford.edu>
2014-05-13 11:44:05 +02:00
sha1 = _notmuch_sha1_of_string (message_id);
Add support (and tests) for messages with really long message IDs. Scott Henson reported an internal error that occurred when he tried to add a message that referenced another message with a message ID well over 300 characters in length. The bug here was running into a Xapian limit for the length of metadata key names, (which is even more restrictive than the Xapian limit for the length of terms). We fix this by noticing long message ID values and instead using a message ID of the form "notmuch-sha1-<sha1_sum_of_message_id>". That is, we use SHA1 to generate a compressed, (but still unique), version of the message ID. We add support to the test suite to exercise this fix. The tests add a message referencing the long message ID, then add the message with the long message ID, then finally add another message referencing the long ID. Each of these tests exercise different code paths where the special handling is implemented. A final test ensures that all three messages are stitched together into a single thread---guaranteeing that the three code paths all act consistently.
2010-06-04 21:39:36 +02:00
compressed = talloc_asprintf (ctx, "notmuch-sha1-%s", sha1);
free (sha1);
return compressed;
}
lib: make find_message{,by_filename) report errors Previously, the functions notmuch_database_find_message() and notmuch_database_find_message_by_filename() functions did not properly report error condition to the library user. For more information, read the thread on the notmuch mailing list starting with my mail "id:871uv2unfd.fsf@gmail.com" Make these functions accept a pointer to 'notmuch_message_t' as argument and return notmuch_status_t which may be used to check for any error condition. restore: Modify for the new notmuch_database_find_message() new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 06:55:29 +02:00
notmuch_status_t
database: Add new notmuch_database_find_message With this function, and the recently added support for notmuch_message_get_thread_ids, we now recode the find_thread_ids function to work just the way we expect a user of the public notmuch API to work. Not too bad really.
2009-10-22 00:37:51 +02:00
notmuch_database_find_message (notmuch_database_t *notmuch,
lib: make find_message{,by_filename) report errors Previously, the functions notmuch_database_find_message() and notmuch_database_find_message_by_filename() functions did not properly report error condition to the library user. For more information, read the thread on the notmuch mailing list starting with my mail "id:871uv2unfd.fsf@gmail.com" Make these functions accept a pointer to 'notmuch_message_t' as argument and return notmuch_status_t which may be used to check for any error condition. restore: Modify for the new notmuch_database_find_message() new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 06:55:29 +02:00
const char *message_id,
notmuch_message_t **message_ret)
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
{
database: Add private find_unique_doc_id and find_unique_document functions These are a generalization of the unique-ness testing of notmuch_database_find_message. More preparation for firectory timestamps.
2009-10-23 23:24:07 +02:00
notmuch_private_status_t status;
unsigned int doc_id;
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
lib: make find_message{,by_filename) report errors Previously, the functions notmuch_database_find_message() and notmuch_database_find_message_by_filename() functions did not properly report error condition to the library user. For more information, read the thread on the notmuch mailing list starting with my mail "id:871uv2unfd.fsf@gmail.com" Make these functions accept a pointer to 'notmuch_message_t' as argument and return notmuch_status_t which may be used to check for any error condition. restore: Modify for the new notmuch_database_find_message() new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 06:55:29 +02:00
if (message_ret == NULL)
return NOTMUCH_STATUS_NULL_POINTER;
Add support (and tests) for messages with really long message IDs. Scott Henson reported an internal error that occurred when he tried to add a message that referenced another message with a message ID well over 300 characters in length. The bug here was running into a Xapian limit for the length of metadata key names, (which is even more restrictive than the Xapian limit for the length of terms). We fix this by noticing long message ID values and instead using a message ID of the form "notmuch-sha1-<sha1_sum_of_message_id>". That is, we use SHA1 to generate a compressed, (but still unique), version of the message ID. We add support to the test suite to exercise this fix. The tests add a message referencing the long message ID, then add the message with the long message ID, then finally add another message referencing the long ID. Each of these tests exercise different code paths where the special handling is implemented. A final test ensures that all three messages are stitched together into a single thread---guaranteeing that the three code paths all act consistently.
2010-06-04 21:39:36 +02:00
if (strlen (message_id) > NOTMUCH_MESSAGE_ID_MAX)
lib: Move message ID compression to _notmuch_message_create_for_message_id Previously, this was performed by notmuch_database_add_message. This happens to be the only caller currently (which is why this was safe), but we're about to introduce more callers, and it makes more sense to put responsibility for ID compression in the lower-level function rather than requiring each caller to handle it.
2014-10-07 01:17:07 +02:00
message_id = _notmuch_message_id_compressed (notmuch, message_id);
Add support (and tests) for messages with really long message IDs. Scott Henson reported an internal error that occurred when he tried to add a message that referenced another message with a message ID well over 300 characters in length. The bug here was running into a Xapian limit for the length of metadata key names, (which is even more restrictive than the Xapian limit for the length of terms). We fix this by noticing long message ID values and instead using a message ID of the form "notmuch-sha1-<sha1_sum_of_message_id>". That is, we use SHA1 to generate a compressed, (but still unique), version of the message ID. We add support to the test suite to exercise this fix. The tests add a message referencing the long message ID, then add the message with the long message ID, then finally add another message referencing the long ID. Each of these tests exercise different code paths where the special handling is implemented. A final test ensures that all three messages are stitched together into a single thread---guaranteeing that the three code paths all act consistently.
2010-06-04 21:39:36 +02:00
lib: Audit all notmuch_database call for Xapian exception handling. Our current approach is for top-level entry poitns in the library to have try/catch blocks that catch any Xapian exception and print a message. Add a few missing blocks and fix up the documentation.
2010-04-24 16:22:34 +02:00
try {
status = _notmuch_database_find_unique_doc_id (notmuch, "id",
message_id, &doc_id);
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
lib: Audit all notmuch_database call for Xapian exception handling. Our current approach is for top-level entry poitns in the library to have try/catch blocks that catch any Xapian exception and print a message. Add a few missing blocks and fix up the documentation.
2010-04-24 16:22:34 +02:00
if (status == NOTMUCH_PRIVATE_STATUS_NO_DOCUMENT_FOUND)
lib: make find_message{,by_filename) report errors Previously, the functions notmuch_database_find_message() and notmuch_database_find_message_by_filename() functions did not properly report error condition to the library user. For more information, read the thread on the notmuch mailing list starting with my mail "id:871uv2unfd.fsf@gmail.com" Make these functions accept a pointer to 'notmuch_message_t' as argument and return notmuch_status_t which may be used to check for any error condition. restore: Modify for the new notmuch_database_find_message() new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 06:55:29 +02:00
*message_ret = NULL;
else {
*message_ret = _notmuch_message_create (notmuch, notmuch, doc_id,
NULL);
if (*message_ret == NULL)
return NOTMUCH_STATUS_OUT_OF_MEMORY;
}
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
lib: make find_message{,by_filename) report errors Previously, the functions notmuch_database_find_message() and notmuch_database_find_message_by_filename() functions did not properly report error condition to the library user. For more information, read the thread on the notmuch mailing list starting with my mail "id:871uv2unfd.fsf@gmail.com" Make these functions accept a pointer to 'notmuch_message_t' as argument and return notmuch_status_t which may be used to check for any error condition. restore: Modify for the new notmuch_database_find_message() new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 06:55:29 +02:00
return NOTMUCH_STATUS_SUCCESS;
lib: Audit all notmuch_database call for Xapian exception handling. Our current approach is for top-level entry poitns in the library to have try/catch blocks that catch any Xapian exception and print a message. Add a few missing blocks and fix up the documentation.
2010-04-24 16:22:34 +02:00
} catch (const Xapian::Error &error) {
lib: replace almost all fprintfs in library with _n_d_log This is not supposed to change any functionality from an end user point of view. Note that it will eliminate some output to stderr. The query debugging output is left as is; it doesn't really fit with the current primitive logging model. The remaining "bad" fprintf will need an internal API change.
2014-12-26 17:25:35 +01:00
_notmuch_database_log (notmuch, "A Xapian exception occurred finding message: %s.\n",
lib: Audit all notmuch_database call for Xapian exception handling. Our current approach is for top-level entry poitns in the library to have try/catch blocks that catch any Xapian exception and print a message. Add a few missing blocks and fix up the documentation.
2010-04-24 16:22:34 +02:00
error.get_msg().c_str());
notmuch->exception_reported = TRUE;
lib: make find_message{,by_filename) report errors Previously, the functions notmuch_database_find_message() and notmuch_database_find_message_by_filename() functions did not properly report error condition to the library user. For more information, read the thread on the notmuch mailing list starting with my mail "id:871uv2unfd.fsf@gmail.com" Make these functions accept a pointer to 'notmuch_message_t' as argument and return notmuch_status_t which may be used to check for any error condition. restore: Modify for the new notmuch_database_find_message() new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 06:55:29 +02:00
*message_ret = NULL;
return NOTMUCH_STATUS_XAPIAN_EXCEPTION;
lib: Audit all notmuch_database call for Xapian exception handling. Our current approach is for top-level entry poitns in the library to have try/catch blocks that catch any Xapian exception and print a message. Add a few missing blocks and fix up the documentation.
2010-04-24 16:22:34 +02:00
}
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
}
notmuch: Switch from gmime to custom, ad-hoc parsing of headers. Since we're currently just trying to stitch together In-Reply-To and References headers we don't need that much sophistication. It's when we later add full-text searching that GMime will be useful. So for now, even though my own code here is surely very buggy compared to GMime it's also a lot faster. And speed is what we're after for the initial index creation.
2009-10-19 21:54:40 +02:00
/* Advance 'str' past any whitespace or RFC 822 comments. A comment is
* a (potentially nested) parenthesized sequence with '\' used to
* escape any character (including parentheses).
*
* If the sequence to be skipped continues to the end of the string,
* then 'str' will be left pointing at the final terminating '\0'
* character.
*/
static void
skip_space_and_comments (const char **str)
{
const char *s;
s = *str;
while (*s && (isspace (*s) || *s == '(')) {
while (*s && isspace (*s))
s++;
if (*s == '(') {
int nesting = 1;
s++;
while (*s && nesting) {
lib/database.cc: coding style Carl claims he must have been distracted when he wrote this... Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
2009-11-22 00:14:39 +01:00
if (*s == '(') {
notmuch: Switch from gmime to custom, ad-hoc parsing of headers. Since we're currently just trying to stitch together In-Reply-To and References headers we don't need that much sophistication. It's when we later add full-text searching that GMime will be useful. So for now, even though my own code here is surely very buggy compared to GMime it's also a lot faster. And speed is what we're after for the initial index creation.
2009-10-19 21:54:40 +02:00
nesting++;
lib/database.cc: coding style Carl claims he must have been distracted when he wrote this... Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
2009-11-22 00:14:39 +01:00
} else if (*s == ')') {
notmuch: Switch from gmime to custom, ad-hoc parsing of headers. Since we're currently just trying to stitch together In-Reply-To and References headers we don't need that much sophistication. It's when we later add full-text searching that GMime will be useful. So for now, even though my own code here is surely very buggy compared to GMime it's also a lot faster. And speed is what we're after for the initial index creation.
2009-10-19 21:54:40 +02:00
nesting--;
lib/database.cc: coding style Carl claims he must have been distracted when he wrote this... Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
2009-11-22 00:14:39 +01:00
} else if (*s == '\\') {
notmuch: Switch from gmime to custom, ad-hoc parsing of headers. Since we're currently just trying to stitch together In-Reply-To and References headers we don't need that much sophistication. It's when we later add full-text searching that GMime will be useful. So for now, even though my own code here is surely very buggy compared to GMime it's also a lot faster. And speed is what we're after for the initial index creation.
2009-10-19 21:54:40 +02:00
if (*(s+1))
s++;
lib/database.cc: coding style Carl claims he must have been distracted when he wrote this... Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
2009-11-22 00:14:39 +01:00
}
notmuch: Switch from gmime to custom, ad-hoc parsing of headers. Since we're currently just trying to stitch together In-Reply-To and References headers we don't need that much sophistication. It's when we later add full-text searching that GMime will be useful. So for now, even though my own code here is surely very buggy compared to GMime it's also a lot faster. And speed is what we're after for the initial index creation.
2009-10-19 21:54:40 +02:00
s++;
}
}
}
*str = s;
}
/* Parse an RFC 822 message-id, discarding whitespace, any RFC 822
fix sum moar typos [comments in source code] Various typo fixes in comments within the source code. Signed-off-by: Pieter Praet <pieter@praet.org> Edited-by: Carl Worth <cworth@cworth.org> Restricted to just source-code comments, (and fixed fix of "descriptios" to "descriptors" rather than "descriptions").
2011-06-20 22:14:21 +02:00
* comments, and the '<' and '>' delimiters.
notmuch: Switch from gmime to custom, ad-hoc parsing of headers. Since we're currently just trying to stitch together In-Reply-To and References headers we don't need that much sophistication. It's when we later add full-text searching that GMime will be useful. So for now, even though my own code here is surely very buggy compared to GMime it's also a lot faster. And speed is what we're after for the initial index creation.
2009-10-19 21:54:40 +02:00
*
* If not NULL, then *next will be made to point to the first character
* not parsed, (possibly pointing to the final '\0' terminator.
*
Remove all calls to g_strdup_printf Replacing them with calls to talloc_asprintf if possible, otherwise to asprintf (with it's painful error-handling leaving the pointer undefined).
2009-10-26 23:17:10 +01:00
* Returns a newly talloc'ed string belonging to 'ctx'.
notmuch: Switch from gmime to custom, ad-hoc parsing of headers. Since we're currently just trying to stitch together In-Reply-To and References headers we don't need that much sophistication. It's when we later add full-text searching that GMime will be useful. So for now, even though my own code here is surely very buggy compared to GMime it's also a lot faster. And speed is what we're after for the initial index creation.
2009-10-19 21:54:40 +02:00
*
* Returns NULL if there is any error parsing the message-id. */
database: Make _parse_message_id static once again. We had exposed this to the internal implementation for a short time, (only while we had the silly code fetching In-Reply-To values from message files instead of from the database). Make this private again as it should be.
2009-11-18 03:50:13 +01:00
static char *
Export _parse_message_id to the library implementation. Not exported through the public interface, but the thread code is going to want to be able to parse In-Reply-To headers so needs access to this code.
2009-11-16 05:21:43 +01:00
_parse_message_id (void *ctx, const char *message_id, const char **next)
notmuch: Switch from gmime to custom, ad-hoc parsing of headers. Since we're currently just trying to stitch together In-Reply-To and References headers we don't need that much sophistication. It's when we later add full-text searching that GMime will be useful. So for now, even though my own code here is surely very buggy compared to GMime it's also a lot faster. And speed is what we're after for the initial index creation.
2009-10-19 21:54:40 +02:00
{
const char *s, *end;
notmuch setup: Collapse internal whitespace within message-id I'm too lazy to see what the RFC says, but I know that having whitespace inside a message-ID is sure to confuse things. And besides, this makes things more compatible with sup so that I have some hope of importing sup labels.
2009-10-21 19:07:34 +02:00
char *result;
notmuch: Switch from gmime to custom, ad-hoc parsing of headers. Since we're currently just trying to stitch together In-Reply-To and References headers we don't need that much sophistication. It's when we later add full-text searching that GMime will be useful. So for now, even though my own code here is surely very buggy compared to GMime it's also a lot faster. And speed is what we're after for the initial index creation.
2009-10-19 21:54:40 +02:00
add_message: Re-fix handling of non-mail files. More fallout from _get_header now returning "" for missing headers. The bug here is that we would no longer detect that a file is not an email message and give up on it like we should. And this time, I actually audited all callers to notmuch_message_get_header, so hopefully we're done fixing this bug over and over.
2009-11-20 21:46:37 +01:00
if (message_id == NULL || *message_id == '\0')
notmuch: Switch from gmime to custom, ad-hoc parsing of headers. Since we're currently just trying to stitch together In-Reply-To and References headers we don't need that much sophistication. It's when we later add full-text searching that GMime will be useful. So for now, even though my own code here is surely very buggy compared to GMime it's also a lot faster. And speed is what we're after for the initial index creation.
2009-10-19 21:54:40 +02:00
return NULL;
s = message_id;
skip_space_and_comments (&s);
/* Skip any unstructured text as well. */
while (*s && *s != '<')
s++;
if (*s == '<') {
s++;
} else {
if (next)
*next = s;
return NULL;
}
skip_space_and_comments (&s);
end = s;
while (*end && *end != '>')
end++;
if (next) {
if (*end)
*next = end + 1;
else
*next = end;
}
if (end > s && *end == '>')
end--;
notmuch setup: Collapse internal whitespace within message-id I'm too lazy to see what the RFC says, but I know that having whitespace inside a message-ID is sure to confuse things. And besides, this makes things more compatible with sup so that I have some hope of importing sup labels.
2009-10-21 19:07:34 +02:00
if (end <= s)
notmuch: Switch from gmime to custom, ad-hoc parsing of headers. Since we're currently just trying to stitch together In-Reply-To and References headers we don't need that much sophistication. It's when we later add full-text searching that GMime will be useful. So for now, even though my own code here is surely very buggy compared to GMime it's also a lot faster. And speed is what we're after for the initial index creation.
2009-10-19 21:54:40 +02:00
return NULL;
notmuch setup: Collapse internal whitespace within message-id I'm too lazy to see what the RFC says, but I know that having whitespace inside a message-ID is sure to confuse things. And besides, this makes things more compatible with sup so that I have some hope of importing sup labels.
2009-10-21 19:07:34 +02:00
Remove all calls to g_strdup_printf Replacing them with calls to talloc_asprintf if possible, otherwise to asprintf (with it's painful error-handling leaving the pointer undefined).
2009-10-26 23:17:10 +01:00
result = talloc_strndup (ctx, s, end - s + 1);
notmuch setup: Collapse internal whitespace within message-id I'm too lazy to see what the RFC says, but I know that having whitespace inside a message-ID is sure to confuse things. And besides, this makes things more compatible with sup so that I have some hope of importing sup labels.
2009-10-21 19:07:34 +02:00
/* Finally, collapse any whitespace that is within the message-id
* itself. */
{
char *r;
int len;
for (r = result, len = strlen (r); *r; r++, len--)
if (*r == ' ' || *r == '\t')
memmove (r, r+1, len);
}
return result;
notmuch: Switch from gmime to custom, ad-hoc parsing of headers. Since we're currently just trying to stitch together In-Reply-To and References headers we don't need that much sophistication. It's when we later add full-text searching that GMime will be useful. So for now, even though my own code here is surely very buggy compared to GMime it's also a lot faster. And speed is what we're after for the initial index creation.
2009-10-19 21:54:40 +02:00
}
Remove all calls to g_strdup_printf Replacing them with calls to talloc_asprintf if possible, otherwise to asprintf (with it's painful error-handling leaving the pointer undefined).
2009-10-26 23:17:10 +01:00
/* Parse a References header value, putting a (talloc'ed under 'ctx')
add_message: Don't add any self-references to the database. In our scheme it's illegal for any message to refer to itself, (nor would it be useful for anything anyway). Cut these self-references off at the source, before they trip up any internal errors.
2009-11-18 02:20:32 +01:00
* copy of each referenced message-id into 'hash'.
*
* We explicitly avoid including any reference identical to
* 'message_id' in the result (to avoid mass confusion when a single
* message references itself cyclically---and yes, mail messages are
* not infrequent in the wild that do this---don't ask me why).
lib/database.cc: change how the parent of a message is calculated Presently, the code which finds the parent of a message as it is being added to the database assumes that the first Message-ID-like substring of the In-Reply-To header is the parent Message ID. Some mail clients, however, put stuff other than the Message-ID of the parent in the In-Reply-To header, such as the email address of the sender of the parent. This can fool notmuch. The updated algorithm prefers the last Message ID in the References header. The References header lists messages oldest-first, so the last Message ID is the parent (RFC2822, p. 24). The References header is also less likely to be in a non-standard syntax (http://cr.yp.to/immhf/thread.html, http://www.jwz.org/doc/threading.html). In case the References header is not to be found, fall back to the old behavior. V2 of this patch, incorporating feedback from Jani and (indirectly) Austin.
2013-03-06 04:31:49 +01:00
*
* Return the last reference parsed, if it is not equal to message_id.
*/
static char *
Remove all calls to g_strdup_printf Replacing them with calls to talloc_asprintf if possible, otherwise to asprintf (with it's painful error-handling leaving the pointer undefined).
2009-10-26 23:17:10 +01:00
parse_references (void *ctx,
add_message: Don't add any self-references to the database. In our scheme it's illegal for any message to refer to itself, (nor would it be useful for anything anyway). Cut these self-references off at the source, before they trip up any internal errors.
2009-11-18 02:20:32 +01:00
const char *message_id,
Remove all calls to g_strdup_printf Replacing them with calls to talloc_asprintf if possible, otherwise to asprintf (with it's painful error-handling leaving the pointer undefined).
2009-10-26 23:17:10 +01:00
GHashTable *hash,
notmuch: Switch from gmime to custom, ad-hoc parsing of headers. Since we're currently just trying to stitch together In-Reply-To and References headers we don't need that much sophistication. It's when we later add full-text searching that GMime will be useful. So for now, even though my own code here is surely very buggy compared to GMime it's also a lot faster. And speed is what we're after for the initial index creation.
2009-10-19 21:54:40 +02:00
const char *refs)
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
{
Make parsing of References and In-Reply-To header less error prone According to RFC2822 References and In-Reply-To headers are supposed to contain one or more Message-IDs, however older RFC822 allowed almost any content. When both References and In-Reply-To headers ends with something else that a Message-ID (see e.g. [1]), the thread structure presented by notmuch is incorrect. The reason is that notmuch treats this case as if the email contained no "replyto" information (see _notmuch_database_link_message_to_parents). This patch changes the parse_references() function to return the last valid Message-ID encountered rather than NULL resulting from the last hunk of text not being the Message-ID. [1] https://lkml.org/lkml/headers/2014/5/19/864
2014-05-26 16:05:57 +02:00
char *ref, *last_ref = NULL;
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
add_message: Re-fix handling of non-mail files. More fallout from _get_header now returning "" for missing headers. The bug here is that we would no longer detect that a file is not an email message and give up on it like we should. And this time, I actually audited all callers to notmuch_message_get_header, so hopefully we're done fixing this bug over and over.
2009-11-20 21:46:37 +01:00
if (refs == NULL || *refs == '\0')
lib/database.cc: change how the parent of a message is calculated Presently, the code which finds the parent of a message as it is being added to the database assumes that the first Message-ID-like substring of the In-Reply-To header is the parent Message ID. Some mail clients, however, put stuff other than the Message-ID of the parent in the In-Reply-To header, such as the email address of the sender of the parent. This can fool notmuch. The updated algorithm prefers the last Message ID in the References header. The References header lists messages oldest-first, so the last Message ID is the parent (RFC2822, p. 24). The References header is also less likely to be in a non-standard syntax (http://cr.yp.to/immhf/thread.html, http://www.jwz.org/doc/threading.html). In case the References header is not to be found, fall back to the old behavior. V2 of this patch, incorporating feedback from Jani and (indirectly) Austin.
2013-03-06 04:31:49 +01:00
return NULL;
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
notmuch: Switch from gmime to custom, ad-hoc parsing of headers. Since we're currently just trying to stitch together In-Reply-To and References headers we don't need that much sophistication. It's when we later add full-text searching that GMime will be useful. So for now, even though my own code here is surely very buggy compared to GMime it's also a lot faster. And speed is what we're after for the initial index creation.
2009-10-19 21:54:40 +02:00
while (*refs) {
Export _parse_message_id to the library implementation. Not exported through the public interface, but the thread code is going to want to be able to parse In-Reply-To headers so needs access to this code.
2009-11-16 05:21:43 +01:00
ref = _parse_message_id (ctx, refs, &refs);
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
Make parsing of References and In-Reply-To header less error prone According to RFC2822 References and In-Reply-To headers are supposed to contain one or more Message-IDs, however older RFC822 allowed almost any content. When both References and In-Reply-To headers ends with something else that a Message-ID (see e.g. [1]), the thread structure presented by notmuch is incorrect. The reason is that notmuch treats this case as if the email contained no "replyto" information (see _notmuch_database_link_message_to_parents). This patch changes the parse_references() function to return the last valid Message-ID encountered rather than NULL resulting from the last hunk of text not being the Message-ID. [1] https://lkml.org/lkml/headers/2014/5/19/864
2014-05-26 16:05:57 +02:00
if (ref && strcmp (ref, message_id)) {
lib: fix g_hash_table related read-after-free bug The two g_hash_table functions (insert, add) have different behaviour with respect to existing keys. g_hash_table_insert frees the new key, while g_hash_table_add (which is really g_hash_table_replace in disguise) frees the existing key. With this change 'ref' is live until the end of the function (assuming single-threaded access to 'hash'). We can't guarantee it will continue to be live in the future (i.e. there may be a future key duplication) so we copy it with the allocation context passed to parse_references (in practice this is the notmuch_message_t object whose parents we are finding). Thanks to Tomi for the simpler approach to the problem based on reading the fine glib manual.
2017-02-22 11:17:47 +01:00
g_hash_table_add (hash, ref);
Make parsing of References and In-Reply-To header less error prone According to RFC2822 References and In-Reply-To headers are supposed to contain one or more Message-IDs, however older RFC822 allowed almost any content. When both References and In-Reply-To headers ends with something else that a Message-ID (see e.g. [1]), the thread structure presented by notmuch is incorrect. The reason is that notmuch treats this case as if the email contained no "replyto" information (see _notmuch_database_link_message_to_parents). This patch changes the parse_references() function to return the last valid Message-ID encountered rather than NULL resulting from the last hunk of text not being the Message-ID. [1] https://lkml.org/lkml/headers/2014/5/19/864
2014-05-26 16:05:57 +02:00
last_ref = ref;
}
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
}
lib/database.cc: change how the parent of a message is calculated Presently, the code which finds the parent of a message as it is being added to the database assumes that the first Message-ID-like substring of the In-Reply-To header is the parent Message ID. Some mail clients, however, put stuff other than the Message-ID of the parent in the In-Reply-To header, such as the email address of the sender of the parent. This can fool notmuch. The updated algorithm prefers the last Message ID in the References header. The References header lists messages oldest-first, so the last Message ID is the parent (RFC2822, p. 24). The References header is also less likely to be in a non-standard syntax (http://cr.yp.to/immhf/thread.html, http://www.jwz.org/doc/threading.html). In case the References header is not to be found, fall back to the old behavior. V2 of this patch, incorporating feedback from Jani and (indirectly) Austin.
2013-03-06 04:31:49 +01:00
/* The return value of this function is used to add a parent
* reference to the database. We should avoid making a message
Make parsing of References and In-Reply-To header less error prone According to RFC2822 References and In-Reply-To headers are supposed to contain one or more Message-IDs, however older RFC822 allowed almost any content. When both References and In-Reply-To headers ends with something else that a Message-ID (see e.g. [1]), the thread structure presented by notmuch is incorrect. The reason is that notmuch treats this case as if the email contained no "replyto" information (see _notmuch_database_link_message_to_parents). This patch changes the parse_references() function to return the last valid Message-ID encountered rather than NULL resulting from the last hunk of text not being the Message-ID. [1] https://lkml.org/lkml/headers/2014/5/19/864
2014-05-26 16:05:57 +02:00
* its own parent, thus the above check.
lib/database.cc: change how the parent of a message is calculated Presently, the code which finds the parent of a message as it is being added to the database assumes that the first Message-ID-like substring of the In-Reply-To header is the parent Message ID. Some mail clients, however, put stuff other than the Message-ID of the parent in the In-Reply-To header, such as the email address of the sender of the parent. This can fool notmuch. The updated algorithm prefers the last Message ID in the References header. The References header lists messages oldest-first, so the last Message ID is the parent (RFC2822, p. 24). The References header is also less likely to be in a non-standard syntax (http://cr.yp.to/immhf/thread.html, http://www.jwz.org/doc/threading.html). In case the References header is not to be found, fall back to the old behavior. V2 of this patch, incorporating feedback from Jani and (indirectly) Austin.
2013-03-06 04:31:49 +01:00
*/
lib: fix g_hash_table related read-after-free bug The two g_hash_table functions (insert, add) have different behaviour with respect to existing keys. g_hash_table_insert frees the new key, while g_hash_table_add (which is really g_hash_table_replace in disguise) frees the existing key. With this change 'ref' is live until the end of the function (assuming single-threaded access to 'hash'). We can't guarantee it will continue to be live in the future (i.e. there may be a future key duplication) so we copy it with the allocation context passed to parse_references (in practice this is the notmuch_message_t object whose parents we are finding). Thanks to Tomi for the simpler approach to the problem based on reading the fine glib manual.
2017-02-22 11:17:47 +01:00
return talloc_strdup(ctx, last_ref);
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
}
lib/cli: Make notmuch_database_create return a status code This is the notmuch_database_create equivalent of the previous change. In this case, there were places where errors were not being propagated correctly in notmuch_database_create or in calls to it. These have been fixed, using the new status value.
2012-04-30 18:25:34 +02:00
notmuch_status_t
notmuch_database_create (const char *path, notmuch_database_t **database)
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
{
char *status_string = NULL;
notmuch_status_t status;
status = notmuch_database_create_verbose (path, database,
&status_string);
if (status_string) {
fputs (status_string, stderr);
free (status_string);
}
return status;
}
notmuch_status_t
notmuch_database_create_verbose (const char *path,
notmuch_database_t **database,
char **status_string)
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
{
lib/cli: Make notmuch_database_create return a status code This is the notmuch_database_create equivalent of the previous change. In this case, there were places where errors were not being propagated correctly in notmuch_database_create or in calls to it. These have been fixed, using the new status value.
2012-04-30 18:25:34 +02:00
notmuch_status_t status = NOTMUCH_STATUS_SUCCESS;
notmuch_database_create/open: Fix to handle NULL as documented. When documenting these functions I described support for a NOTMUCH_BASE environment variable to be consulted in the case of a NULL path. Only, I had forgotten to actually write the code. This code exists now, with a new, exported function: notmuch_database_default_path
2009-10-20 18:56:25 +02:00
notmuch_database_t *notmuch = NULL;
char *notmuch_path = NULL;
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
char *message = NULL;
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
struct stat st;
int err;
notmuch_database_create/open: Fix to handle NULL as documented. When documenting these functions I described support for a NOTMUCH_BASE environment variable to be consulted in the case of a NULL path. Only, I had forgotten to actually write the code. This code exists now, with a new, exported function: notmuch_database_default_path
2009-10-20 18:56:25 +02:00
notmuch: Add a configuration system. This will allow for things like the database path to be specified without any cheesy NOTMUCH_BASE environment variable. It also will allow "notmuch reply" to recognize the user's email address when constructing a reply in order to do the right thing, (that is, to use the user's address to which mail was sent as From:, and not to reply to the user's own addresses). With this change, the "notmuch setup" command is now strictly for changing the configuration of notmuch. It no longer creates the database, but instead instructs the user to call "notmuch new" to do that.
2009-11-12 02:01:55 +01:00
if (path == NULL) {
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
message = strdup ("Error: Cannot create a database for a NULL path.\n");
lib/cli: Make notmuch_database_create return a status code This is the notmuch_database_create equivalent of the previous change. In this case, there were places where errors were not being propagated correctly in notmuch_database_create or in calls to it. These have been fixed, using the new status value.
2012-04-30 18:25:34 +02:00
status = NOTMUCH_STATUS_NULL_POINTER;
notmuch: Add a configuration system. This will allow for things like the database path to be specified without any cheesy NOTMUCH_BASE environment variable. It also will allow "notmuch reply" to recognize the user's email address when constructing a reply in order to do the right thing, (that is, to use the user's address to which mail was sent as From:, and not to reply to the user's own addresses). With this change, the "notmuch setup" command is now strictly for changing the configuration of notmuch. It no longer creates the database, but instead instructs the user to call "notmuch new" to do that.
2009-11-12 02:01:55 +01:00
goto DONE;
}
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
lib: reject relative paths in n_d_{create,open}_verbose There are many places in the notmuch code where the path is assumed to be absolute. If someone (TM) wants a project, one could remove these assumptions. In the mean time, prevent users from shooting themselves in the foot. Update test suite mark tests for this error as no longer broken, and also convert some tests that used relative paths for nonexistent directories.
2015-06-08 08:02:22 +02:00
if (path[0] != '/') {
message = strdup ("Error: Database path must be absolute.\n");
status = NOTMUCH_STATUS_PATH_ERROR;
goto DONE;
}
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
err = stat (path, &st);
if (err) {
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
IGNORE_RESULT (asprintf (&message, "Error: Cannot create database at %s: %s.\n",
path, strerror (errno)));
lib/cli: Make notmuch_database_create return a status code This is the notmuch_database_create equivalent of the previous change. In this case, there were places where errors were not being propagated correctly in notmuch_database_create or in calls to it. These have been fixed, using the new status value.
2012-04-30 18:25:34 +02:00
status = NOTMUCH_STATUS_FILE_ERROR;
notmuch_database_create/open: Fix to handle NULL as documented. When documenting these functions I described support for a NOTMUCH_BASE environment variable to be consulted in the case of a NULL path. Only, I had forgotten to actually write the code. This code exists now, with a new, exported function: notmuch_database_default_path
2009-10-20 18:56:25 +02:00
goto DONE;
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
}
if (! S_ISDIR (st.st_mode)) {
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
IGNORE_RESULT (asprintf (&message, "Error: Cannot create database at %s: "
"Not a directory.\n",
path));
lib/cli: Make notmuch_database_create return a status code This is the notmuch_database_create equivalent of the previous change. In this case, there were places where errors were not being propagated correctly in notmuch_database_create or in calls to it. These have been fixed, using the new status value.
2012-04-30 18:25:34 +02:00
status = NOTMUCH_STATUS_FILE_ERROR;
notmuch_database_create/open: Fix to handle NULL as documented. When documenting these functions I described support for a NOTMUCH_BASE environment variable to be consulted in the case of a NULL path. Only, I had forgotten to actually write the code. This code exists now, with a new, exported function: notmuch_database_default_path
2009-10-20 18:56:25 +02:00
goto DONE;
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
}
Remove all calls to g_strdup_printf Replacing them with calls to talloc_asprintf if possible, otherwise to asprintf (with it's painful error-handling leaving the pointer undefined).
2009-10-26 23:17:10 +01:00
notmuch_path = talloc_asprintf (NULL, "%s/%s", path, ".notmuch");
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
err = mkdir (notmuch_path, 0755);
if (err) {
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
IGNORE_RESULT (asprintf (&message, "Error: Cannot create directory %s: %s.\n",
notmuch_path, strerror (errno)));
lib/cli: Make notmuch_database_create return a status code This is the notmuch_database_create equivalent of the previous change. In this case, there were places where errors were not being propagated correctly in notmuch_database_create or in calls to it. These have been fixed, using the new status value.
2012-04-30 18:25:34 +02:00
status = NOTMUCH_STATUS_FILE_ERROR;
notmuch_database_create/open: Fix to handle NULL as documented. When documenting these functions I described support for a NOTMUCH_BASE environment variable to be consulted in the case of a NULL path. Only, I had forgotten to actually write the code. This code exists now, with a new, exported function: notmuch_database_default_path
2009-10-20 18:56:25 +02:00
goto DONE;
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
}
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
status = notmuch_database_open_verbose (path,
NOTMUCH_DATABASE_MODE_READ_WRITE,
&notmuch, &message);
lib/cli: Make notmuch_database_create return a status code This is the notmuch_database_create equivalent of the previous change. In this case, there were places where errors were not being propagated correctly in notmuch_database_create or in calls to it. These have been fixed, using the new status value.
2012-04-30 18:25:34 +02:00
if (status)
goto DONE;
lib: Database version 3: Introduce fine-grained "features" Previously, our database schema was versioned by a single number. Each database schema change had to occur "atomically" in Notmuch's development history: before some commit, Notmuch used version N, after that commit, it used version N+1. Hence, each new schema version could introduce only one change, the task of developing a schema change fell on a single person, and it all had to happen and be perfect in a single commit series. This made introducing a new schema version hard. We've seen only two schema changes in the history of Notmuch. This commit introduces database schema version 3; hopefully the last schema version we'll need for a while. With this version, we switch from a single version number to "features": a set of named, independent aspects of the database schema. Features should make backwards compatibility easier. For many things, it should be easy to support databases both with and without a feature, which will allow us to make upgrades optional and will enable "unstable" features that can be developed and tested over time. Features also make forwards compatibility easier. The features recorded in a database include "compatibility flags," which can indicate to an older version of Notmuch when it must support a given feature to open the database for read or for write. This lets us replace the old vague "I don't recognize this version, so something might go wrong, but I promise to try my best" warnings upon opening a database with an unknown version with precise errors. If a database is safe to open for read/write despite unknown features, an older version will know that and issue no message at all. If the database is not safe to open for read/write because of unknown features, an older version will know that, too, and can tell the user exactly which required features it lacks support for.
2014-08-25 19:26:00 +02:00
Add the NOTMUCH_FEATURE_INDEXED_MIMETYPES database feature This feature will exist in all newly created databases, but there is no upgrade provided for it. If this flag exists, it indicates that the database was created after the indexed MIME-types feature was added.
2015-01-23 00:43:37 +01:00
/* Upgrade doesn't add these feature to existing databases, but
* new databases have them. */
lib: Database version 3: Introduce fine-grained "features" Previously, our database schema was versioned by a single number. Each database schema change had to occur "atomically" in Notmuch's development history: before some commit, Notmuch used version N, after that commit, it used version N+1. Hence, each new schema version could introduce only one change, the task of developing a schema change fell on a single person, and it all had to happen and be perfect in a single commit series. This made introducing a new schema version hard. We've seen only two schema changes in the history of Notmuch. This commit introduces database schema version 3; hopefully the last schema version we'll need for a while. With this version, we switch from a single version number to "features": a set of named, independent aspects of the database schema. Features should make backwards compatibility easier. For many things, it should be easy to support databases both with and without a feature, which will allow us to make upgrades optional and will enable "unstable" features that can be developed and tested over time. Features also make forwards compatibility easier. The features recorded in a database include "compatibility flags," which can indicate to an older version of Notmuch when it must support a given feature to open the database for read or for write. This lets us replace the old vague "I don't recognize this version, so something might go wrong, but I promise to try my best" warnings upon opening a database with an unknown version with precise errors. If a database is safe to open for read/write despite unknown features, an older version will know that and issue no message at all. If the database is not safe to open for read/write because of unknown features, an older version will know that, too, and can tell the user exactly which required features it lacks support for.
2014-08-25 19:26:00 +02:00
notmuch->features |= NOTMUCH_FEATURE_FROM_SUBJECT_ID_VALUES;
Add the NOTMUCH_FEATURE_INDEXED_MIMETYPES database feature This feature will exist in all newly created databases, but there is no upgrade provided for it. If this flag exists, it indicates that the database was created after the indexed MIME-types feature was added.
2015-01-23 00:43:37 +01:00
notmuch->features |= NOTMUCH_FEATURE_INDEXED_MIMETYPES;
lib: Database version 3: Introduce fine-grained "features" Previously, our database schema was versioned by a single number. Each database schema change had to occur "atomically" in Notmuch's development history: before some commit, Notmuch used version N, after that commit, it used version N+1. Hence, each new schema version could introduce only one change, the task of developing a schema change fell on a single person, and it all had to happen and be perfect in a single commit series. This made introducing a new schema version hard. We've seen only two schema changes in the history of Notmuch. This commit introduces database schema version 3; hopefully the last schema version we'll need for a while. With this version, we switch from a single version number to "features": a set of named, independent aspects of the database schema. Features should make backwards compatibility easier. For many things, it should be easy to support databases both with and without a feature, which will allow us to make upgrades optional and will enable "unstable" features that can be developed and tested over time. Features also make forwards compatibility easier. The features recorded in a database include "compatibility flags," which can indicate to an older version of Notmuch when it must support a given feature to open the database for read or for write. This lets us replace the old vague "I don't recognize this version, so something might go wrong, but I promise to try my best" warnings upon opening a database with an unknown version with precise errors. If a database is safe to open for read/write despite unknown features, an older version will know that and issue no message at all. If the database is not safe to open for read/write because of unknown features, an older version will know that, too, and can tell the user exactly which required features it lacks support for.
2014-08-25 19:26:00 +02:00
lib/cli: Make notmuch_database_create return a status code This is the notmuch_database_create equivalent of the previous change. In this case, there were places where errors were not being propagated correctly in notmuch_database_create or in calls to it. These have been fixed, using the new status value.
2012-04-30 18:25:34 +02:00
status = notmuch_database_upgrade (notmuch, NULL, NULL);
if (status) {
notmuch_database_close(notmuch);
notmuch = NULL;
}
notmuch_database_create/open: Fix to handle NULL as documented. When documenting these functions I described support for a NOTMUCH_BASE environment variable to be consulted in the case of a NULL path. Only, I had forgotten to actually write the code. This code exists now, with a new, exported function: notmuch_database_default_path
2009-10-20 18:56:25 +02:00
DONE:
if (notmuch_path)
Remove all calls to g_strdup_printf Replacing them with calls to talloc_asprintf if possible, otherwise to asprintf (with it's painful error-handling leaving the pointer undefined).
2009-10-26 23:17:10 +01:00
talloc_free (notmuch_path);
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
if (message) {
if (status_string)
*status_string = message;
else
free (message);
}
lib/cli: Make notmuch_database_create return a status code This is the notmuch_database_create equivalent of the previous change. In this case, there were places where errors were not being propagated correctly in notmuch_database_create or in calls to it. These have been fixed, using the new status value.
2012-04-30 18:25:34 +02:00
if (database)
*database = notmuch;
else
talloc_free (notmuch);
return status;
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
}
lib: Consolidate checks for read-only database. Previously, many checks were deep in the library just before a cast operation. These have now been replaced with internal errors and new checks have instead been added at the beginning of all top-levelentry points requiring a read-write database. The new checks now also use a single function for checking and printing the error message. This will give us a convenient location to extend the check, (such as based on database version as well).
2010-01-07 19:19:44 +01:00
notmuch_status_t
_notmuch_database_ensure_writable (notmuch_database_t *notmuch)
{
if (notmuch->mode == NOTMUCH_DATABASE_MODE_READ_ONLY) {
lib: replace almost all fprintfs in library with _n_d_log This is not supposed to change any functionality from an end user point of view. Note that it will eliminate some output to stderr. The query debugging output is left as is; it doesn't really fit with the current primitive logging model. The remaining "bad" fprintf will need an internal API change.
2014-12-26 17:25:35 +01:00
_notmuch_database_log (notmuch, "Cannot write to a read-only database.\n");
Prefer READ_ONLY consistently over READONLY. Previously we had NOTMUCH_DATABASE_MODE_READ_ONLY but NOTMUCH_STATUS_READONLY_DATABASE which was ugly and confusing. Rename the latter to NOTMUCH_STATUS_READ_ONLY_DATABASE for consistency.
2010-01-07 19:29:05 +01:00
return NOTMUCH_STATUS_READ_ONLY_DATABASE;
lib: Consolidate checks for read-only database. Previously, many checks were deep in the library just before a cast operation. These have now been replaced with internal errors and new checks have instead been added at the beginning of all top-levelentry points requiring a read-write database. The new checks now also use a single function for checking and printing the error message. This will give us a convenient location to extend the check, (such as based on database version as well).
2010-01-07 19:19:44 +01:00
}
return NOTMUCH_STATUS_SUCCESS;
}
lib: Add per-message last modification tracking This adds a new document value that stores the revision of the last modification to message metadata, where the revision number increases monotonically with each database commit. An alternative would be to store the wall-clock time of the last modification of each message. In principle this is simpler and has the advantage that any process can determine the current timestamp without support from libnotmuch. However, even assuming a computer's clock never goes backward and ignoring clock skew in networked environments, this has a fatal flaw. Xapian uses (optimistic) snapshot isolation, which means reads can be concurrent with writes. Given this, consider the following time line with a write and two read transactions: write |-X-A--------------| read 1 |---B---| read 2 |---| The write transaction modifies message X and records the wall-clock time of the modification at A. The writer hangs around for a while and later commits its change. Read 1 is concurrent with the write, so it doesn't see the change to X. It does some query and records the wall-clock time of its results at B. Transaction read 2 later starts after the write commits and queries for changes since wall-clock time B (say the reads are performing an incremental backup). Even though read 1 could not see the change to X, read 2 is told (correctly) that X has not changed since B, the time of the last read. In fact, X changed before wall-clock time A, but the change was not visible until *after* wall-clock time B, so read 2 misses the change to X. This is tricky to solve in full-blown snapshot isolation, but because Xapian serializes writes, we can use a simple, monotonically increasing database revision number. Furthermore, maintaining this revision number requires no more IO than a wall-clock time solution because Xapian already maintains statistics on the upper (and lower) bound of each value stream.
2014-10-13 08:20:01 +02:00
/* Allocate a revision number for the next change. */
unsigned long
_notmuch_database_new_revision (notmuch_database_t *notmuch)
{
unsigned long new_revision = notmuch->revision + 1;
/* If we're in an atomic section, hold off on updating the
* committed revision number until we commit the atomic section.
*/
if (notmuch->atomic_nesting)
notmuch->atomic_dirty = TRUE;
else
notmuch->revision = new_revision;
return new_revision;
}
lib: Database version 3: Introduce fine-grained "features" Previously, our database schema was versioned by a single number. Each database schema change had to occur "atomically" in Notmuch's development history: before some commit, Notmuch used version N, after that commit, it used version N+1. Hence, each new schema version could introduce only one change, the task of developing a schema change fell on a single person, and it all had to happen and be perfect in a single commit series. This made introducing a new schema version hard. We've seen only two schema changes in the history of Notmuch. This commit introduces database schema version 3; hopefully the last schema version we'll need for a while. With this version, we switch from a single version number to "features": a set of named, independent aspects of the database schema. Features should make backwards compatibility easier. For many things, it should be easy to support databases both with and without a feature, which will allow us to make upgrades optional and will enable "unstable" features that can be developed and tested over time. Features also make forwards compatibility easier. The features recorded in a database include "compatibility flags," which can indicate to an older version of Notmuch when it must support a given feature to open the database for read or for write. This lets us replace the old vague "I don't recognize this version, so something might go wrong, but I promise to try my best" warnings upon opening a database with an unknown version with precise errors. If a database is safe to open for read/write despite unknown features, an older version will know that and issue no message at all. If the database is not safe to open for read/write because of unknown features, an older version will know that, too, and can tell the user exactly which required features it lacks support for.
2014-08-25 19:26:00 +02:00
/* Parse a database features string from the given database version.
* Returns the feature bit set.
*
* For version < 3, this ignores the features string and returns a
* hard-coded set of features.
*
* If there are unrecognized features that are required to open the
* database in mode (which should be 'r' or 'w'), return a
* comma-separated list of unrecognized but required features in
* *incompat_out suitable for presenting to the user. *incompat_out
* will be allocated from ctx.
*/
static _notmuch_features
_parse_features (const void *ctx, const char *features, unsigned int version,
char mode, char **incompat_out)
{
_notmuch_features res = static_cast<_notmuch_features>(0);
unsigned int namelen, i;
size_t llen = 0;
const char *flags;
/* Prior to database version 3, features were implied by the
* version number. */
if (version == 0)
return NOTMUCH_FEATURES_V0;
else if (version == 1)
return NOTMUCH_FEATURES_V1;
else if (version == 2)
return NOTMUCH_FEATURES_V2;
/* Parse the features string */
while ((features = strtok_len_c (features + llen, "\n", &llen)) != NULL) {
flags = strchr (features, '\t');
if (! flags || flags > features + llen)
continue;
namelen = flags - features;
for (i = 0; i < ARRAY_SIZE (feature_names); ++i) {
if (strlen (feature_names[i].name) == namelen &&
strncmp (feature_names[i].name, features, namelen) == 0) {
res |= feature_names[i].value;
break;
}
}
if (i == ARRAY_SIZE (feature_names) && incompat_out) {
/* Unrecognized feature */
const char *have = strchr (flags, mode);
if (have && have < features + llen) {
/* This feature is required to access this database in
* 'mode', but we don't understand it. */
if (! *incompat_out)
*incompat_out = talloc_strdup (ctx, "");
*incompat_out = talloc_asprintf_append_buffer (
*incompat_out, "%s%.*s", **incompat_out ? ", " : "",
namelen, features);
}
}
}
return res;
}
static char *
_print_features (const void *ctx, unsigned int features)
{
unsigned int i;
char *res = talloc_strdup (ctx, "");
for (i = 0; i < ARRAY_SIZE (feature_names); ++i)
if (features & feature_names[i].value)
res = talloc_asprintf_append_buffer (
res, "%s\t%s\n", feature_names[i].name, feature_names[i].flags);
return res;
}
lib/cli: Make notmuch_database_open return a status code It has been a long-standing issue that notmuch_database_open doesn't return any indication of why it failed. This patch changes its prototype to return a notmuch_status_t and set an out-argument to the database itself, like other functions that return both a status and an object. In the interest of atomicity, this also updates every use in the CLI so that notmuch still compiles. Since this patch does not update the bindings, the Python bindings test fails.
2012-04-30 18:25:33 +02:00
notmuch_status_t
Permit opening the notmuch database in read-only mode. We only rarely need to actually open the database for writing, but we always create a Xapian::WritableDatabase. This has the effect of preventing searches and like whilst updating the index. Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Acked-by: Carl Worth <cworth@cworth.org>
2009-11-21 20:54:25 +01:00
notmuch_database_open (const char *path,
lib/cli: Make notmuch_database_open return a status code It has been a long-standing issue that notmuch_database_open doesn't return any indication of why it failed. This patch changes its prototype to return a notmuch_status_t and set an out-argument to the database itself, like other functions that return both a status and an object. In the interest of atomicity, this also updates every use in the CLI so that notmuch still compiles. Since this patch does not update the bindings, the Python bindings test fails.
2012-04-30 18:25:33 +02:00
notmuch_database_mode_t mode,
notmuch_database_t **database)
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
{
char *status_string = NULL;
notmuch_status_t status;
status = notmuch_database_open_verbose (path, mode, database,
&status_string);
if (status_string) {
fputs (status_string, stderr);
free (status_string);
}
return status;
}
notmuch_status_t
notmuch_database_open_verbose (const char *path,
notmuch_database_mode_t mode,
notmuch_database_t **database,
char **status_string)
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
{
lib/cli: Make notmuch_database_open return a status code It has been a long-standing issue that notmuch_database_open doesn't return any indication of why it failed. This patch changes its prototype to return a notmuch_status_t and set an out-argument to the database itself, like other functions that return both a status and an object. In the interest of atomicity, this also updates every use in the CLI so that notmuch still compiles. Since this patch does not update the bindings, the Python bindings test fails.
2012-04-30 18:25:33 +02:00
notmuch_status_t status = NOTMUCH_STATUS_SUCCESS;
lib: Use talloc to simplify cleanup in notmuch_database_open Previously, we manually "free"d various pointers in notmuch_database_open. Use a local talloc context instead to simplify cleanup and eliminate various NULL pointer initializations and conditionals.
2012-01-29 06:50:10 +01:00
void *local = talloc_new (NULL);
notmuch_database_create/open: Fix to handle NULL as documented. When documenting these functions I described support for a NOTMUCH_BASE environment variable to be consulted in the case of a NULL path. Only, I had forgotten to actually write the code. This code exists now, with a new, exported function: notmuch_database_default_path
2009-10-20 18:56:25 +02:00
notmuch_database_t *notmuch = NULL;
lib: Database version 3: Introduce fine-grained "features" Previously, our database schema was versioned by a single number. Each database schema change had to occur "atomically" in Notmuch's development history: before some commit, Notmuch used version N, after that commit, it used version N+1. Hence, each new schema version could introduce only one change, the task of developing a schema change fell on a single person, and it all had to happen and be perfect in a single commit series. This made introducing a new schema version hard. We've seen only two schema changes in the history of Notmuch. This commit introduces database schema version 3; hopefully the last schema version we'll need for a while. With this version, we switch from a single version number to "features": a set of named, independent aspects of the database schema. Features should make backwards compatibility easier. For many things, it should be easy to support databases both with and without a feature, which will allow us to make upgrades optional and will enable "unstable" features that can be developed and tested over time. Features also make forwards compatibility easier. The features recorded in a database include "compatibility flags," which can indicate to an older version of Notmuch when it must support a given feature to open the database for read or for write. This lets us replace the old vague "I don't recognize this version, so something might go wrong, but I promise to try my best" warnings upon opening a database with an unknown version with precise errors. If a database is safe to open for read/write despite unknown features, an older version will know that and issue no message at all. If the database is not safe to open for read/write because of unknown features, an older version will know that, too, and can tell the user exactly which required features it lacks support for.
2014-08-25 19:26:00 +02:00
char *notmuch_path, *xapian_path, *incompat_features;
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
char *message = NULL;
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
struct stat st;
int err;
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
unsigned int i, version;
lib: call g_mime_init() from notmuch_database_open() As reported in id:"CAEbOPGyuHnz4BPtDutnTPUHcP3eYcRCRkXhYoJR43RUMw671+g@mail.gmail.com" sometimes gmime tries to access a NULL pointer, e.g. g_mime_iconv_open() tries to access iconv_cache that is NULL if g_mime_init() is not called. This causes notmuch to segfault when calling gmime functions. Calling g_mime_init() initializes iconv_cache and others variables needed by gmime, making sure they are initialized when notmuch calls gmime functions. Test marked fix by db.
2011-12-31 05:37:41 +01:00
static int initialized = 0;
notmuch_database_create/open: Fix to handle NULL as documented. When documenting these functions I described support for a NOTMUCH_BASE environment variable to be consulted in the case of a NULL path. Only, I had forgotten to actually write the code. This code exists now, with a new, exported function: notmuch_database_default_path
2009-10-20 18:56:25 +02:00
lib/cli: Make notmuch_database_open return a status code It has been a long-standing issue that notmuch_database_open doesn't return any indication of why it failed. This patch changes its prototype to return a notmuch_status_t and set an out-argument to the database itself, like other functions that return both a status and an object. In the interest of atomicity, this also updates every use in the CLI so that notmuch still compiles. Since this patch does not update the bindings, the Python bindings test fails.
2012-04-30 18:25:33 +02:00
if (path == NULL) {
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
message = strdup ("Error: Cannot open a database for a NULL path.\n");
lib/cli: Make notmuch_database_open return a status code It has been a long-standing issue that notmuch_database_open doesn't return any indication of why it failed. This patch changes its prototype to return a notmuch_status_t and set an out-argument to the database itself, like other functions that return both a status and an object. In the interest of atomicity, this also updates every use in the CLI so that notmuch still compiles. Since this patch does not update the bindings, the Python bindings test fails.
2012-04-30 18:25:33 +02:00
status = NOTMUCH_STATUS_NULL_POINTER;
goto DONE;
}
lib: reject relative paths in n_d_{create,open}_verbose There are many places in the notmuch code where the path is assumed to be absolute. If someone (TM) wants a project, one could remove these assumptions. In the mean time, prevent users from shooting themselves in the foot. Update test suite mark tests for this error as no longer broken, and also convert some tests that used relative paths for nonexistent directories.
2015-06-08 08:02:22 +02:00
if (path[0] != '/') {
message = strdup ("Error: Database path must be absolute.\n");
status = NOTMUCH_STATUS_PATH_ERROR;
goto DONE;
}
lib: Use talloc to simplify cleanup in notmuch_database_open Previously, we manually "free"d various pointers in notmuch_database_open. Use a local talloc context instead to simplify cleanup and eliminate various NULL pointer initializations and conditionals.
2012-01-29 06:50:10 +01:00
if (! (notmuch_path = talloc_asprintf (local, "%s/%s", path, ".notmuch"))) {
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
message = strdup ("Out of memory\n");
lib/cli: Make notmuch_database_open return a status code It has been a long-standing issue that notmuch_database_open doesn't return any indication of why it failed. This patch changes its prototype to return a notmuch_status_t and set an out-argument to the database itself, like other functions that return both a status and an object. In the interest of atomicity, this also updates every use in the CLI so that notmuch still compiles. Since this patch does not update the bindings, the Python bindings test fails.
2012-04-30 18:25:33 +02:00
status = NOTMUCH_STATUS_OUT_OF_MEMORY;
Remove all calls to g_strdup_printf Replacing them with calls to talloc_asprintf if possible, otherwise to asprintf (with it's painful error-handling leaving the pointer undefined).
2009-10-26 23:17:10 +01:00
goto DONE;
}
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
err = stat (notmuch_path, &st);
if (err) {
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
IGNORE_RESULT (asprintf (&message, "Error opening database at %s: %s\n",
notmuch_path, strerror (errno)));
lib/cli: Make notmuch_database_open return a status code It has been a long-standing issue that notmuch_database_open doesn't return any indication of why it failed. This patch changes its prototype to return a notmuch_status_t and set an out-argument to the database itself, like other functions that return both a status and an object. In the interest of atomicity, this also updates every use in the CLI so that notmuch still compiles. Since this patch does not update the bindings, the Python bindings test fails.
2012-04-30 18:25:33 +02:00
status = NOTMUCH_STATUS_FILE_ERROR;
notmuch_database_create/open: Fix to handle NULL as documented. When documenting these functions I described support for a NOTMUCH_BASE environment variable to be consulted in the case of a NULL path. Only, I had forgotten to actually write the code. This code exists now, with a new, exported function: notmuch_database_default_path
2009-10-20 18:56:25 +02:00
goto DONE;
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
}
lib: Use talloc to simplify cleanup in notmuch_database_open Previously, we manually "free"d various pointers in notmuch_database_open. Use a local talloc context instead to simplify cleanup and eliminate various NULL pointer initializations and conditionals.
2012-01-29 06:50:10 +01:00
if (! (xapian_path = talloc_asprintf (local, "%s/%s", notmuch_path, "xapian"))) {
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
message = strdup ("Out of memory\n");
lib/cli: Make notmuch_database_open return a status code It has been a long-standing issue that notmuch_database_open doesn't return any indication of why it failed. This patch changes its prototype to return a notmuch_status_t and set an out-argument to the database itself, like other functions that return both a status and an object. In the interest of atomicity, this also updates every use in the CLI so that notmuch still compiles. Since this patch does not update the bindings, the Python bindings test fails.
2012-04-30 18:25:33 +02:00
status = NOTMUCH_STATUS_OUT_OF_MEMORY;
Remove all calls to g_strdup_printf Replacing them with calls to talloc_asprintf if possible, otherwise to asprintf (with it's painful error-handling leaving the pointer undefined).
2009-10-26 23:17:10 +01:00
goto DONE;
}
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
lib: call g_type_init from notmuch_database_open We want to make sure g_type_init is called before any GObject functionality is used.
2011-12-04 20:20:39 +01:00
/* Initialize the GLib type system and threads */
cli: Guard deprecated g_type_init calls g_type_init was deprecated in GLib 2.35.1. In order to compile cleanly, guard these with a suitable #if. (commit msg from https://bugs.freedesktop.org/attachment.cgi?id=73774 )
2013-06-07 20:41:13 +02:00
#if !GLIB_CHECK_VERSION(2, 35, 1)
lib: call g_type_init from notmuch_database_open We want to make sure g_type_init is called before any GObject functionality is used.
2011-12-04 20:20:39 +01:00
g_type_init ();
cli: Guard deprecated g_type_init calls g_type_init was deprecated in GLib 2.35.1. In order to compile cleanly, guard these with a suitable #if. (commit msg from https://bugs.freedesktop.org/attachment.cgi?id=73774 )
2013-06-07 20:41:13 +02:00
#endif
lib: call g_type_init from notmuch_database_open We want to make sure g_type_init is called before any GObject functionality is used.
2011-12-04 20:20:39 +01:00
lib: call g_mime_init() from notmuch_database_open() As reported in id:"CAEbOPGyuHnz4BPtDutnTPUHcP3eYcRCRkXhYoJR43RUMw671+g@mail.gmail.com" sometimes gmime tries to access a NULL pointer, e.g. g_mime_iconv_open() tries to access iconv_cache that is NULL if g_mime_init() is not called. This causes notmuch to segfault when calling gmime functions. Calling g_mime_init() initializes iconv_cache and others variables needed by gmime, making sure they are initialized when notmuch calls gmime functions. Test marked fix by db.
2011-12-31 05:37:41 +01:00
/* Initialize gmime */
if (! initialized) {
lib/cli: pass GMIME_ENABLE_RFC2047_WORKAROUNDS to g_mime_init() As explained by Jeffrey Stedfast, the author of GMime, quoted in [1]: > Passing the GMIME_ENABLE_RFC2047_WORKAROUNDS flag to g_mime_init() > *should* solve the decoding problem mentioned in the thread. This > flag should be safe to pass into g_mime_init() without any bad side > effects and my unit tests do test that code-path. The thread being referred to is [2]. [1] id:87bo56viyo.fsf@nikula.org [2] id:08cb1dcd-c5db-4e33-8b09-7730cb3d59a2@gmail.com
2013-09-11 19:36:43 +02:00
g_mime_init (GMIME_ENABLE_RFC2047_WORKAROUNDS);
lib: call g_mime_init() from notmuch_database_open() As reported in id:"CAEbOPGyuHnz4BPtDutnTPUHcP3eYcRCRkXhYoJR43RUMw671+g@mail.gmail.com" sometimes gmime tries to access a NULL pointer, e.g. g_mime_iconv_open() tries to access iconv_cache that is NULL if g_mime_init() is not called. This causes notmuch to segfault when calling gmime functions. Calling g_mime_init() initializes iconv_cache and others variables needed by gmime, making sure they are initialized when notmuch calls gmime functions. Test marked fix by db.
2011-12-31 05:37:41 +01:00
initialized = 1;
}
lib: Don't delete uninitialized pointers In the error-handling paths of notmuch_database_open, we call notmuch_database_close, which "delete"s several objects referenced by the notmuch_database_t object. However, some of these pointers may be uninitialized, resulting in undefined behavior. Hence, allocate the notmuch_database_t with talloc_zero to make sure these pointers are NULL so that "delete"ing them is harmless.
2012-01-29 06:50:08 +01:00
notmuch = talloc_zero (NULL, notmuch_database_t);
Catch and optionally print about exception at database->flush. If an earlier exception occurred, then it's not unexpected for the flush to fail as well. So in that case, we'll silently catch the exception. Otherwise, make some noise about things going wrong at the time of flush.
2009-11-22 03:54:20 +01:00
notmuch->exception_reported = FALSE;
lib: add a log function with output to a string in notmuch_database_t In principle in the future this could do something fancier than asprintf.
2014-12-26 09:01:01 +01:00
notmuch->status_string = NULL;
Convert notmuch_database_t to start using talloc. This will be handy as we can hang future talloc allocations off of the datbase now.
2009-10-21 23:00:37 +02:00
notmuch->path = talloc_strdup (notmuch, path);
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
Fix add_message and get_filename to strip/re-add the database path. We now store only a relative path inside the database so the database is not nicely relocatable.
2009-10-29 00:50:14 +01:00
if (notmuch->path[strlen (notmuch->path) - 1] == '/')
notmuch->path[strlen (notmuch->path) - 1] = '\0';
Permit opening the notmuch database in read-only mode. We only rarely need to actually open the database for writing, but we always create a Xapian::WritableDatabase. This has the effect of preventing searches and like whilst updating the index. Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Acked-by: Carl Worth <cworth@cworth.org>
2009-11-21 20:54:25 +01:00
notmuch->mode = mode;
lib: Add support for nested atomic sections. notmuch_database_t now keeps a nesting count and we only start a transaction or commit for the outermost atomic section. Introduces a new error, NOTMUCH_STATUS_UNBALANCED_ATOMIC.
2011-06-11 05:35:06 +02:00
notmuch->atomic_nesting = 0;
lib: add _notmuch_database_reopen The main expected use is to recover from a Xapian::DatabaseChanged exception.
2017-02-24 02:38:24 +01:00
notmuch->view = 1;
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
try {
Switch from random to sequential thread identifiers. The sequential identifiers have the advantage of being guaranteed to be unique (until we overflow a 64-bit unsigned integer), and also take up half as much space in the "notmuch search" output (16 columns rather than 32). This change also has the side effect of fixing a bug where notmuch could block on /dev/random at startup (waiting for some entropy to appear). This bug was hit hard by the test suite, (which could easily exhaust the available entropy on common systems---resulting in large delays of the test suite).
2010-02-08 20:33:33 +01:00
string last_thread_id;
lib: Add per-message last modification tracking This adds a new document value that stores the revision of the last modification to message metadata, where the revision number increases monotonically with each database commit. An alternative would be to store the wall-clock time of the last modification of each message. In principle this is simpler and has the advantage that any process can determine the current timestamp without support from libnotmuch. However, even assuming a computer's clock never goes backward and ignoring clock skew in networked environments, this has a fatal flaw. Xapian uses (optimistic) snapshot isolation, which means reads can be concurrent with writes. Given this, consider the following time line with a write and two read transactions: write |-X-A--------------| read 1 |---B---| read 2 |---| The write transaction modifies message X and records the wall-clock time of the modification at A. The writer hangs around for a while and later commits its change. Read 1 is concurrent with the write, so it doesn't see the change to X. It does some query and records the wall-clock time of its results at B. Transaction read 2 later starts after the write commits and queries for changes since wall-clock time B (say the reads are performing an incremental backup). Even though read 1 could not see the change to X, read 2 is told (correctly) that X has not changed since B, the time of the last read. In fact, X changed before wall-clock time A, but the change was not visible until *after* wall-clock time B, so read 2 misses the change to X. This is tricky to solve in full-blown snapshot isolation, but because Xapian serializes writes, we can use a simple, monotonically increasing database revision number. Furthermore, maintaining this revision number requires no more IO than a wall-clock time solution because Xapian already maintains statistics on the upper (and lower) bound of each value stream.
2014-10-13 08:20:01 +02:00
string last_mod;
Switch from random to sequential thread identifiers. The sequential identifiers have the advantage of being guaranteed to be unique (until we overflow a 64-bit unsigned integer), and also take up half as much space in the "notmuch search" output (16 columns rather than 32). This change also has the side effect of fixing a bug where notmuch could block on /dev/random at startup (waiting for some entropy to appear). This bug was hit hard by the test suite, (which could easily exhaust the available entropy on common systems---resulting in large delays of the test suite).
2010-02-08 20:33:33 +01:00
Rename NOTMUCH_DATABASE_MODE_WRITABLE to NOTMUCH_DATABASE_MODE_READ_WRITE And correspondingly, READONLY to READ_ONLY.
2009-11-21 22:10:18 +01:00
if (mode == NOTMUCH_DATABASE_MODE_READ_WRITE) {
Permit opening the notmuch database in read-only mode. We only rarely need to actually open the database for writing, but we always create a Xapian::WritableDatabase. This has the effect of preventing searches and like whilst updating the index. Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Acked-by: Carl Worth <cworth@cworth.org>
2009-11-21 20:54:25 +01:00
notmuch->xapian_db = new Xapian::WritableDatabase (xapian_path,
Use the Xapian::DB_RETRY_LOCK flag when available Xapian 1.3 has introduced the DB_RETRY_LOCK flag (Xapian bug 275). Detect it in configure and optionally use it. With this flag commands that need the write lock will wait for their turn instead of aborting when it's not immediately available. Amended by db: allow disabling in configure
2016-06-26 17:29:43 +02:00
DB_ACTION);
Permit opening the notmuch database in read-only mode. We only rarely need to actually open the database for writing, but we always create a Xapian::WritableDatabase. This has the effect of preventing searches and like whilst updating the index. Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Acked-by: Carl Worth <cworth@cworth.org>
2009-11-21 20:54:25 +01:00
} else {
notmuch->xapian_db = new Xapian::Database (xapian_path);
lib: Database version 3: Introduce fine-grained "features" Previously, our database schema was versioned by a single number. Each database schema change had to occur "atomically" in Notmuch's development history: before some commit, Notmuch used version N, after that commit, it used version N+1. Hence, each new schema version could introduce only one change, the task of developing a schema change fell on a single person, and it all had to happen and be perfect in a single commit series. This made introducing a new schema version hard. We've seen only two schema changes in the history of Notmuch. This commit introduces database schema version 3; hopefully the last schema version we'll need for a while. With this version, we switch from a single version number to "features": a set of named, independent aspects of the database schema. Features should make backwards compatibility easier. For many things, it should be easy to support databases both with and without a feature, which will allow us to make upgrades optional and will enable "unstable" features that can be developed and tested over time. Features also make forwards compatibility easier. The features recorded in a database include "compatibility flags," which can indicate to an older version of Notmuch when it must support a given feature to open the database for read or for write. This lets us replace the old vague "I don't recognize this version, so something might go wrong, but I promise to try my best" warnings upon opening a database with an unknown version with precise errors. If a database is safe to open for read/write despite unknown features, an older version will know that and issue no message at all. If the database is not safe to open for read/write because of unknown features, an older version will know that, too, and can tell the user exactly which required features it lacks support for.
2014-08-25 19:26:00 +02:00
}
/* Check version. As of database version 3, we represent
* changes in terms of features, so assume a version bump
* means a dramatically incompatible change. */
version = notmuch_database_get_version (notmuch);
if (version > NOTMUCH_DATABASE_VERSION) {
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
IGNORE_RESULT (asprintf (&message,
"Error: Notmuch database at %s\n"
" has a newer database format version (%u) than supported by this\n"
" version of notmuch (%u).\n",
notmuch_path, version, NOTMUCH_DATABASE_VERSION));
lib: Database version 3: Introduce fine-grained "features" Previously, our database schema was versioned by a single number. Each database schema change had to occur "atomically" in Notmuch's development history: before some commit, Notmuch used version N, after that commit, it used version N+1. Hence, each new schema version could introduce only one change, the task of developing a schema change fell on a single person, and it all had to happen and be perfect in a single commit series. This made introducing a new schema version hard. We've seen only two schema changes in the history of Notmuch. This commit introduces database schema version 3; hopefully the last schema version we'll need for a while. With this version, we switch from a single version number to "features": a set of named, independent aspects of the database schema. Features should make backwards compatibility easier. For many things, it should be easy to support databases both with and without a feature, which will allow us to make upgrades optional and will enable "unstable" features that can be developed and tested over time. Features also make forwards compatibility easier. The features recorded in a database include "compatibility flags," which can indicate to an older version of Notmuch when it must support a given feature to open the database for read or for write. This lets us replace the old vague "I don't recognize this version, so something might go wrong, but I promise to try my best" warnings upon opening a database with an unknown version with precise errors. If a database is safe to open for read/write despite unknown features, an older version will know that and issue no message at all. If the database is not safe to open for read/write because of unknown features, an older version will know that, too, and can tell the user exactly which required features it lacks support for.
2014-08-25 19:26:00 +02:00
notmuch->mode = NOTMUCH_DATABASE_MODE_READ_ONLY;
notmuch_database_destroy (notmuch);
notmuch = NULL;
status = NOTMUCH_STATUS_FILE_ERROR;
goto DONE;
}
/* Check features. */
incompat_features = NULL;
notmuch->features = _parse_features (
local, notmuch->xapian_db->get_metadata ("features").c_str (),
version, mode == NOTMUCH_DATABASE_MODE_READ_WRITE ? 'w' : 'r',
&incompat_features);
if (incompat_features) {
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
IGNORE_RESULT (asprintf (&message,
"Error: Notmuch database at %s\n"
" requires features (%s)\n"
" not supported by this version of notmuch.\n",
notmuch_path, incompat_features));
lib: Database version 3: Introduce fine-grained "features" Previously, our database schema was versioned by a single number. Each database schema change had to occur "atomically" in Notmuch's development history: before some commit, Notmuch used version N, after that commit, it used version N+1. Hence, each new schema version could introduce only one change, the task of developing a schema change fell on a single person, and it all had to happen and be perfect in a single commit series. This made introducing a new schema version hard. We've seen only two schema changes in the history of Notmuch. This commit introduces database schema version 3; hopefully the last schema version we'll need for a while. With this version, we switch from a single version number to "features": a set of named, independent aspects of the database schema. Features should make backwards compatibility easier. For many things, it should be easy to support databases both with and without a feature, which will allow us to make upgrades optional and will enable "unstable" features that can be developed and tested over time. Features also make forwards compatibility easier. The features recorded in a database include "compatibility flags," which can indicate to an older version of Notmuch when it must support a given feature to open the database for read or for write. This lets us replace the old vague "I don't recognize this version, so something might go wrong, but I promise to try my best" warnings upon opening a database with an unknown version with precise errors. If a database is safe to open for read/write despite unknown features, an older version will know that and issue no message at all. If the database is not safe to open for read/write because of unknown features, an older version will know that, too, and can tell the user exactly which required features it lacks support for.
2014-08-25 19:26:00 +02:00
notmuch->mode = NOTMUCH_DATABASE_MODE_READ_ONLY;
notmuch_database_destroy (notmuch);
notmuch = NULL;
status = NOTMUCH_STATUS_FILE_ERROR;
goto DONE;
Permit opening the notmuch database in read-only mode. We only rarely need to actually open the database for writing, but we always create a Xapian::WritableDatabase. This has the effect of preventing searches and like whilst updating the index. Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Acked-by: Carl Worth <cworth@cworth.org>
2009-11-21 20:54:25 +01:00
}
Switch from random to sequential thread identifiers. The sequential identifiers have the advantage of being guaranteed to be unique (until we overflow a 64-bit unsigned integer), and also take up half as much space in the "notmuch search" output (16 columns rather than 32). This change also has the side effect of fixing a bug where notmuch could block on /dev/random at startup (waiting for some entropy to appear). This bug was hit hard by the test suite, (which could easily exhaust the available entropy on common systems---resulting in large delays of the test suite).
2010-02-08 20:33:33 +01:00
Avoid database corruption by not adding partially-constructed mail documents. Previously we were using Xapian's add_document to allocate document ID values for notmuch_message_t objects. This had the drawback of adding a partially constructed mail document to the database. If notmuch was subsequently interrupted before fully populating this document, then later runs would be quite confused when seeing the partial documents. There are reports from the wild of people hitting internal errors of the form "Message ... has no thread ID" for example, (which is currently an unrecoverable error). We fix this by manually allocating document IDs without adding documents. With this change, we never call Xapian's add_document method, but only replace_document with either the current document ID of a message or a new one that we have allocated.
2010-06-04 19:16:53 +02:00
notmuch->last_doc_id = notmuch->xapian_db->get_lastdocid ();
Switch from random to sequential thread identifiers. The sequential identifiers have the advantage of being guaranteed to be unique (until we overflow a 64-bit unsigned integer), and also take up half as much space in the "notmuch search" output (16 columns rather than 32). This change also has the side effect of fixing a bug where notmuch could block on /dev/random at startup (waiting for some entropy to appear). This bug was hit hard by the test suite, (which could easily exhaust the available entropy on common systems---resulting in large delays of the test suite).
2010-02-08 20:33:33 +01:00
last_thread_id = notmuch->xapian_db->get_metadata ("last_thread_id");
if (last_thread_id.empty ()) {
notmuch->last_thread_id = 0;
} else {
const char *str;
char *end;
str = last_thread_id.c_str ();
notmuch->last_thread_id = strtoull (str, &end, 16);
if (*end != '\0')
INTERNAL_ERROR ("Malformed database last_thread_id: %s", str);
}
lib: Add per-message last modification tracking This adds a new document value that stores the revision of the last modification to message metadata, where the revision number increases monotonically with each database commit. An alternative would be to store the wall-clock time of the last modification of each message. In principle this is simpler and has the advantage that any process can determine the current timestamp without support from libnotmuch. However, even assuming a computer's clock never goes backward and ignoring clock skew in networked environments, this has a fatal flaw. Xapian uses (optimistic) snapshot isolation, which means reads can be concurrent with writes. Given this, consider the following time line with a write and two read transactions: write |-X-A--------------| read 1 |---B---| read 2 |---| The write transaction modifies message X and records the wall-clock time of the modification at A. The writer hangs around for a while and later commits its change. Read 1 is concurrent with the write, so it doesn't see the change to X. It does some query and records the wall-clock time of its results at B. Transaction read 2 later starts after the write commits and queries for changes since wall-clock time B (say the reads are performing an incremental backup). Even though read 1 could not see the change to X, read 2 is told (correctly) that X has not changed since B, the time of the last read. In fact, X changed before wall-clock time A, but the change was not visible until *after* wall-clock time B, so read 2 misses the change to X. This is tricky to solve in full-blown snapshot isolation, but because Xapian serializes writes, we can use a simple, monotonically increasing database revision number. Furthermore, maintaining this revision number requires no more IO than a wall-clock time solution because Xapian already maintains statistics on the upper (and lower) bound of each value stream.
2014-10-13 08:20:01 +02:00
/* Get current highest revision number. */
last_mod = notmuch->xapian_db->get_value_upper_bound (
NOTMUCH_VALUE_LAST_MOD);
if (last_mod.empty ())
notmuch->revision = 0;
else
notmuch->revision = Xapian::sortable_unserialise (last_mod);
lib: API to retrieve database revision and UUID This exposes the committed database revision to library users along with a UUID that can be used to detect when revision numbers are no longer comparable (e.g., because the database has been replaced).
2014-10-13 08:20:02 +02:00
notmuch->uuid = talloc_strdup (
notmuch, notmuch->xapian_db->get_uuid ().c_str ());
lib: Add per-message last modification tracking This adds a new document value that stores the revision of the last modification to message metadata, where the revision number increases monotonically with each database commit. An alternative would be to store the wall-clock time of the last modification of each message. In principle this is simpler and has the advantage that any process can determine the current timestamp without support from libnotmuch. However, even assuming a computer's clock never goes backward and ignoring clock skew in networked environments, this has a fatal flaw. Xapian uses (optimistic) snapshot isolation, which means reads can be concurrent with writes. Given this, consider the following time line with a write and two read transactions: write |-X-A--------------| read 1 |---B---| read 2 |---| The write transaction modifies message X and records the wall-clock time of the modification at A. The writer hangs around for a while and later commits its change. Read 1 is concurrent with the write, so it doesn't see the change to X. It does some query and records the wall-clock time of its results at B. Transaction read 2 later starts after the write commits and queries for changes since wall-clock time B (say the reads are performing an incremental backup). Even though read 1 could not see the change to X, read 2 is told (correctly) that X has not changed since B, the time of the last read. In fact, X changed before wall-clock time A, but the change was not visible until *after* wall-clock time B, so read 2 misses the change to X. This is tricky to solve in full-blown snapshot isolation, but because Xapian serializes writes, we can use a simple, monotonically increasing database revision number. Furthermore, maintaining this revision number requires no more IO than a wall-clock time solution because Xapian already maintains statistics on the upper (and lower) bound of each value stream.
2014-10-13 08:20:01 +02:00
notmuch dump: Fix the sorting of results. To properly support sorting in notmuch_query we know use an Enquire object. We also throw in a QueryParser too, so we're really close to being able to support arbitrary full-text searches. I took a look at the supported QueryParser syntax and chose a set of flags for everything I like, (such as supporting Boolean operators in either case ("AND" or "and"), supporting phrase searching, supporting + and - to include/preclude terms, and supporting a trailing * on any term as a wildcard).
2009-10-21 09:35:56 +02:00
notmuch->query_parser = new Xapian::QueryParser;
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
notmuch->term_gen = new Xapian::TermGenerator;
notmuch->term_gen->set_stemmer (Xapian::Stem ("english"));
Add rudimentary date-based search. The rudimentary aspect here is that the date ranges are specified with UNIX timestamp values (number of seconds since 1970-01-01 UTC). One thing that can help here is using the date program to determins timestamps, such as: $(date +%s -d 2009-10-01)..$(date +%s) Long-term, we'll probably need to do our own query parsing to be able to support directly-specified dates and also relative expressions like "since:'2 months ago'".
2009-11-23 16:58:35 +01:00
notmuch->value_range_processor = new Xapian::NumberValueRangeProcessor (NOTMUCH_VALUE_TIMESTAMP);
lib: add date range query support Add a custom value range processor to enable date and time searches of the form date:since..until, where "since" and "until" are expressions understood by the previously added date/time parser, to restrict the results to messages within a particular time range (based on the Date: header). If "since" or "until" describes date/time at an accuracy of days or less, the values are rounded according to the accuracy, towards past for "since" and towards future for "until". For example, date:november..yesterday would match from the beginning of November until the end of yesterday. Expressions such as date:today..today means since the beginning of today until the end of today. Open-ended ranges are supported (since Xapian 1.2.1), i.e. you can specify date:..until or date:since.. to not limit the start or end date, respectively. CAVEATS: Xapian does not support spaces in range expressions. You can replace the spaces with '_', or (in most cases) '-', or (in some cases) leave the spaces out altogether. Entering date:expr without ".." (for example date:yesterday) will not work as you might expect. You can achieve the expected result by duplicating the expr both sides of ".." (for example date:yesterday..yesterday). Open-ended ranges won't work with pre-1.2.1 Xapian, but they don't produce an error either. Signed-off-by: Jani Nikula <jani@nikula.org>
2012-10-30 21:32:37 +01:00
notmuch->date_range_processor = new ParseTimeValueRangeProcessor (NOTMUCH_VALUE_TIMESTAMP);
lib: optionally support single argument date: queries This relies on the FieldProcessor API, which is only present in xapian >= 1.3.
2016-03-22 11:54:44 +01:00
#if HAVE_XAPIAN_FIELD_PROCESSOR
/* This currently relies on the query parser to pass anything
* with a .. to the range processor */
lib: Let Xapian manage the memory for FieldProcessors It turns out this is exactly what release() is for; Xapian will deallocate the objects when it's done with them.
2017-02-17 04:07:50 +01:00
{
Xapian::FieldProcessor * date_fp = new DateFieldProcessor();
Xapian::FieldProcessor * query_fp =
new QueryFieldProcessor (*notmuch->query_parser, notmuch);
notmuch->query_parser->add_boolean_prefix("date", date_fp->release ());
notmuch->query_parser->add_boolean_prefix("query", query_fp->release ());
}
lib: optionally support single argument date: queries This relies on the FieldProcessor API, which is only present in xapian >= 1.3.
2016-03-22 11:54:44 +01:00
#endif
lib: Add "lastmod:" queries for filtering by last modification The implementation is essentially the same as the date range search prior to Jani's fancy date parser.
2014-10-13 08:20:03 +02:00
notmuch->last_mod_range_processor = new Xapian::NumberValueRangeProcessor (NOTMUCH_VALUE_LAST_MOD, "lastmod:");
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
notmuch dump: Fix the sorting of results. To properly support sorting in notmuch_query we know use an Enquire object. We also throw in a QueryParser too, so we're really close to being able to support arbitrary full-text searches. I took a look at the supported QueryParser syntax and chose a set of flags for everything I like, (such as supporting Boolean operators in either case ("AND" or "and"), supporting phrase searching, supporting + and - to include/preclude terms, and supporting a trailing * on any term as a wildcard).
2009-10-21 09:35:56 +02:00
notmuch->query_parser->set_default_op (Xapian::Query::OP_AND);
notmuch->query_parser->set_database (*notmuch->xapian_db);
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
notmuch->query_parser->set_stemmer (Xapian::Stem ("english"));
notmuch->query_parser->set_stemming_strategy (Xapian::QueryParser::STEM_SOME);
Add rudimentary date-based search. The rudimentary aspect here is that the date ranges are specified with UNIX timestamp values (number of seconds since 1970-01-01 UTC). One thing that can help here is using the date program to determins timestamps, such as: $(date +%s -d 2009-10-01)..$(date +%s) Long-term, we'll probably need to do our own query parsing to be able to support directly-specified dates and also relative expressions like "since:'2 months ago'".
2009-11-23 16:58:35 +01:00
notmuch->query_parser->add_valuerangeprocessor (notmuch->value_range_processor);
lib: add date range query support Add a custom value range processor to enable date and time searches of the form date:since..until, where "since" and "until" are expressions understood by the previously added date/time parser, to restrict the results to messages within a particular time range (based on the Date: header). If "since" or "until" describes date/time at an accuracy of days or less, the values are rounded according to the accuracy, towards past for "since" and towards future for "until". For example, date:november..yesterday would match from the beginning of November until the end of yesterday. Expressions such as date:today..today means since the beginning of today until the end of today. Open-ended ranges are supported (since Xapian 1.2.1), i.e. you can specify date:..until or date:since.. to not limit the start or end date, respectively. CAVEATS: Xapian does not support spaces in range expressions. You can replace the spaces with '_', or (in most cases) '-', or (in some cases) leave the spaces out altogether. Entering date:expr without ".." (for example date:yesterday) will not work as you might expect. You can achieve the expected result by duplicating the expr both sides of ".." (for example date:yesterday..yesterday). Open-ended ranges won't work with pre-1.2.1 Xapian, but they don't produce an error either. Signed-off-by: Jani Nikula <jani@nikula.org>
2012-10-30 21:32:37 +01:00
notmuch->query_parser->add_valuerangeprocessor (notmuch->date_range_processor);
lib: Add "lastmod:" queries for filtering by last modification The implementation is essentially the same as the date range search prior to Jani's fancy date parser.
2014-10-13 08:20:03 +02:00
notmuch->query_parser->add_valuerangeprocessor (notmuch->last_mod_range_processor);
Split BOOLEAN_PREFIX into INTERNAL and EXTERNAL subsets. The idea here is that only some of the prefix names (such as "id" and "tag") actually make sense in external user-supplied query strings. Other things like "type" are internal implementation details of how we store things in the database. So internal machinery will add those terms to the database and we don't need to support them in the string itself. With this, we can now simply loop over the external prefix values to let the quiery parser know about them. So as we add prefixes in the future, we'll only need to add them to this list.
2009-10-25 06:38:43 +01:00
lib: merge internal prefix tables Replace multiple tables with some flags in a single table. This makes the code in notmuch_database_open_verbose a bit shorter, and it should also make it easier to add other options to fields, e.g. regexp searching.
2017-02-17 04:07:49 +01:00
for (i = 0; i < ARRAY_SIZE (prefix_table); i++) {
const prefix_t *prefix = &prefix_table[i];
if (prefix->flags & NOTMUCH_FIELD_EXTERNAL) {
if (prefix->flags & NOTMUCH_FIELD_PROBABILISTIC) {
notmuch->query_parser->add_prefix (prefix->name, prefix->prefix);
} else {
notmuch->query_parser->add_boolean_prefix (prefix->name,
prefix->prefix);
}
}
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
}
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
} catch (const Xapian::Error &error) {
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
IGNORE_RESULT (asprintf (&message, "A Xapian exception occurred opening database: %s\n",
error.get_msg().c_str()));
Split notmuch_database_close into two functions Formerly notmuch_database_close closed the xapian database and destroyed the talloc structure associated with the notmuch database object. Split notmuch_database_close into notmuch_database_close and notmuch_database_destroy. This makes it possible for long running programs to close the xapian database and thus release the lock associated with it without destroying the data structures obtained from it. This also makes the api more consistent since every other data structure has a destructor function. The comments in notmuch.h are a courtesy of Austin Clements. Signed-off-by: Justus Winter <4winter@informatik.uni-hamburg.de>
2012-04-25 15:20:16 +02:00
notmuch_database_destroy (notmuch);
Fix segfault in case of the database lock not being available. We were nicely reporting the lock-aquisition failure, but then marching along trying to use the database object and just crashing badly. So don't do that.
2009-10-28 07:57:37 +01:00
notmuch = NULL;
lib/cli: Make notmuch_database_open return a status code It has been a long-standing issue that notmuch_database_open doesn't return any indication of why it failed. This patch changes its prototype to return a notmuch_status_t and set an out-argument to the database itself, like other functions that return both a status and an object. In the interest of atomicity, this also updates every use in the CLI so that notmuch still compiles. Since this patch does not update the bindings, the Python bindings test fails.
2012-04-30 18:25:33 +02:00
status = NOTMUCH_STATUS_XAPIAN_EXCEPTION;
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
}
lib/database.cc: coding style Carl claims he must have been distracted when he wrote this... Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
2009-11-22 00:14:39 +01:00
notmuch_database_create/open: Fix to handle NULL as documented. When documenting these functions I described support for a NOTMUCH_BASE environment variable to be consulted in the case of a NULL path. Only, I had forgotten to actually write the code. This code exists now, with a new, exported function: notmuch_database_default_path
2009-10-20 18:56:25 +02:00
DONE:
lib: Use talloc to simplify cleanup in notmuch_database_open Previously, we manually "free"d various pointers in notmuch_database_open. Use a local talloc context instead to simplify cleanup and eliminate various NULL pointer initializations and conditionals.
2012-01-29 06:50:10 +01:00
talloc_free (local);
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
if (message) {
if (status_string)
*status_string = message;
else
free (message);
}
lib/cli: Make notmuch_database_open return a status code It has been a long-standing issue that notmuch_database_open doesn't return any indication of why it failed. This patch changes its prototype to return a notmuch_status_t and set an out-argument to the database itself, like other functions that return both a status and an object. In the interest of atomicity, this also updates every use in the CLI so that notmuch still compiles. Since this patch does not update the bindings, the Python bindings test fails.
2012-04-30 18:25:33 +02:00
if (database)
*database = notmuch;
else
talloc_free (notmuch);
return status;
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
}
lib: add return status to database close and destroy notmuch_database_close may fail in Xapian ->flush() or ->close(), so report the status. Similarly for notmuch_database_destroy which calls close. This is required for notmuch insert to report error status if message indexing failed.
2014-04-16 14:59:16 +02:00
notmuch_status_t
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
notmuch_database_close (notmuch_database_t *notmuch)
{
lib: add return status to database close and destroy notmuch_database_close may fail in Xapian ->flush() or ->close(), so report the status. Similarly for notmuch_database_destroy which calls close. This is required for notmuch insert to report error status if message indexing failed.
2014-04-16 14:59:16 +02:00
notmuch_status_t status = NOTMUCH_STATUS_SUCCESS;
Actually close the xapian database in notmuch_database_close Formerly the xapian database object was deleted and closed in its destructor once the object was garbage collected. Explicitly call close() so that the database and the associated lock is released immediately. The comment is a courtesy of Austin Clements. Signed-off-by: Justus Winter <4winter@informatik.uni-hamburg.de>
2012-03-02 15:58:39 +01:00
/* Many Xapian objects (and thus notmuch objects) hold references to
* the database, so merely deleting the database may not suffice to
* close it. Thus, we explicitly close it here. */
if (notmuch->xapian_db != NULL) {
try {
lib: Simplify close and codify aborting atomic section In Xapian, closing a database implicitly aborts any outstanding transaction and commits changes. For historical reasons, notmuch_database_close had grown to almost, but not quite duplicate this behavior. Before closing the database, it would explicitly (and unnecessarily) commit it. However, if there was an outstanding transaction (ie atomic section), commit would throw a Xapian exception, which notmuch_database_close would unnecessarily print to stderr, even though notmuch_database_close would ultimately abort the transaction anyway when it called close. This patch simplifies notmuch_database_close to explicitly abort any outstanding transaction and then just call Database::close. This works for both read-only and read/write databases, takes care of committing changes, unifies the exception handling path, and codifies aborting outstanding transactions. This is currently the only way to abort an atomic section (and may remain so, since it would be difficult to roll back things we may have cached from rolled-back modifications).
2014-10-02 21:19:08 +02:00
/* If there's an outstanding transaction, it's unclear if
* closing the Xapian database commits everything up to
* that transaction, or may discard committed (but
* unflushed) transactions. To be certain, explicitly
* cancel any outstanding transaction before closing. */
if (notmuch->mode == NOTMUCH_DATABASE_MODE_READ_WRITE &&
notmuch->atomic_nesting)
(static_cast <Xapian::WritableDatabase *> (notmuch->xapian_db))
->cancel_transaction ();
/* Close the database. This implicitly flushes
* outstanding changes. */
Actually close the xapian database in notmuch_database_close Formerly the xapian database object was deleted and closed in its destructor once the object was garbage collected. Explicitly call close() so that the database and the associated lock is released immediately. The comment is a courtesy of Austin Clements. Signed-off-by: Justus Winter <4winter@informatik.uni-hamburg.de>
2012-03-02 15:58:39 +01:00
notmuch->xapian_db->close();
} catch (const Xapian::Error &error) {
lib: Simplify close and codify aborting atomic section In Xapian, closing a database implicitly aborts any outstanding transaction and commits changes. For historical reasons, notmuch_database_close had grown to almost, but not quite duplicate this behavior. Before closing the database, it would explicitly (and unnecessarily) commit it. However, if there was an outstanding transaction (ie atomic section), commit would throw a Xapian exception, which notmuch_database_close would unnecessarily print to stderr, even though notmuch_database_close would ultimately abort the transaction anyway when it called close. This patch simplifies notmuch_database_close to explicitly abort any outstanding transaction and then just call Database::close. This works for both read-only and read/write databases, takes care of committing changes, unifies the exception handling path, and codifies aborting outstanding transactions. This is currently the only way to abort an atomic section (and may remain so, since it would be difficult to roll back things we may have cached from rolled-back modifications).
2014-10-02 21:19:08 +02:00
status = NOTMUCH_STATUS_XAPIAN_EXCEPTION;
if (! notmuch->exception_reported) {
lib: replace almost all fprintfs in library with _n_d_log This is not supposed to change any functionality from an end user point of view. Note that it will eliminate some output to stderr. The query debugging output is left as is; it doesn't really fit with the current primitive logging model. The remaining "bad" fprintf will need an internal API change.
2014-12-26 17:25:35 +01:00
_notmuch_database_log (notmuch, "Error: A Xapian exception occurred closing database: %s\n",
lib: Simplify close and codify aborting atomic section In Xapian, closing a database implicitly aborts any outstanding transaction and commits changes. For historical reasons, notmuch_database_close had grown to almost, but not quite duplicate this behavior. Before closing the database, it would explicitly (and unnecessarily) commit it. However, if there was an outstanding transaction (ie atomic section), commit would throw a Xapian exception, which notmuch_database_close would unnecessarily print to stderr, even though notmuch_database_close would ultimately abort the transaction anyway when it called close. This patch simplifies notmuch_database_close to explicitly abort any outstanding transaction and then just call Database::close. This works for both read-only and read/write databases, takes care of committing changes, unifies the exception handling path, and codifies aborting outstanding transactions. This is currently the only way to abort an atomic section (and may remain so, since it would be difficult to roll back things we may have cached from rolled-back modifications).
2014-10-02 21:19:08 +02:00
error.get_msg().c_str());
}
Actually close the xapian database in notmuch_database_close Formerly the xapian database object was deleted and closed in its destructor once the object was garbage collected. Explicitly call close() so that the database and the associated lock is released immediately. The comment is a courtesy of Austin Clements. Signed-off-by: Justus Winter <4winter@informatik.uni-hamburg.de>
2012-03-02 15:58:39 +01:00
}
}
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
delete notmuch->term_gen;
Split notmuch_database_close into two functions Formerly notmuch_database_close closed the xapian database and destroyed the talloc structure associated with the notmuch database object. Split notmuch_database_close into notmuch_database_close and notmuch_database_destroy. This makes it possible for long running programs to close the xapian database and thus release the lock associated with it without destroying the data structures obtained from it. This also makes the api more consistent since every other data structure has a destructor function. The comments in notmuch.h are a courtesy of Austin Clements. Signed-off-by: Justus Winter <4winter@informatik.uni-hamburg.de>
2012-04-25 15:20:16 +02:00
notmuch->term_gen = NULL;
notmuch dump: Fix the sorting of results. To properly support sorting in notmuch_query we know use an Enquire object. We also throw in a QueryParser too, so we're really close to being able to support arbitrary full-text searches. I took a look at the supported QueryParser syntax and chose a set of flags for everything I like, (such as supporting Boolean operators in either case ("AND" or "and"), supporting phrase searching, supporting + and - to include/preclude terms, and supporting a trailing * on any term as a wildcard).
2009-10-21 09:35:56 +02:00
delete notmuch->query_parser;
Split notmuch_database_close into two functions Formerly notmuch_database_close closed the xapian database and destroyed the talloc structure associated with the notmuch database object. Split notmuch_database_close into notmuch_database_close and notmuch_database_destroy. This makes it possible for long running programs to close the xapian database and thus release the lock associated with it without destroying the data structures obtained from it. This also makes the api more consistent since every other data structure has a destructor function. The comments in notmuch.h are a courtesy of Austin Clements. Signed-off-by: Justus Winter <4winter@informatik.uni-hamburg.de>
2012-04-25 15:20:16 +02:00
notmuch->query_parser = NULL;
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
delete notmuch->xapian_db;
Split notmuch_database_close into two functions Formerly notmuch_database_close closed the xapian database and destroyed the talloc structure associated with the notmuch database object. Split notmuch_database_close into notmuch_database_close and notmuch_database_destroy. This makes it possible for long running programs to close the xapian database and thus release the lock associated with it without destroying the data structures obtained from it. This also makes the api more consistent since every other data structure has a destructor function. The comments in notmuch.h are a courtesy of Austin Clements. Signed-off-by: Justus Winter <4winter@informatik.uni-hamburg.de>
2012-04-25 15:20:16 +02:00
notmuch->xapian_db = NULL;
Add rudimentary date-based search. The rudimentary aspect here is that the date ranges are specified with UNIX timestamp values (number of seconds since 1970-01-01 UTC). One thing that can help here is using the date program to determins timestamps, such as: $(date +%s -d 2009-10-01)..$(date +%s) Long-term, we'll probably need to do our own query parsing to be able to support directly-specified dates and also relative expressions like "since:'2 months ago'".
2009-11-23 16:58:35 +01:00
delete notmuch->value_range_processor;
Split notmuch_database_close into two functions Formerly notmuch_database_close closed the xapian database and destroyed the talloc structure associated with the notmuch database object. Split notmuch_database_close into notmuch_database_close and notmuch_database_destroy. This makes it possible for long running programs to close the xapian database and thus release the lock associated with it without destroying the data structures obtained from it. This also makes the api more consistent since every other data structure has a destructor function. The comments in notmuch.h are a courtesy of Austin Clements. Signed-off-by: Justus Winter <4winter@informatik.uni-hamburg.de>
2012-04-25 15:20:16 +02:00
notmuch->value_range_processor = NULL;
lib: add date range query support Add a custom value range processor to enable date and time searches of the form date:since..until, where "since" and "until" are expressions understood by the previously added date/time parser, to restrict the results to messages within a particular time range (based on the Date: header). If "since" or "until" describes date/time at an accuracy of days or less, the values are rounded according to the accuracy, towards past for "since" and towards future for "until". For example, date:november..yesterday would match from the beginning of November until the end of yesterday. Expressions such as date:today..today means since the beginning of today until the end of today. Open-ended ranges are supported (since Xapian 1.2.1), i.e. you can specify date:..until or date:since.. to not limit the start or end date, respectively. CAVEATS: Xapian does not support spaces in range expressions. You can replace the spaces with '_', or (in most cases) '-', or (in some cases) leave the spaces out altogether. Entering date:expr without ".." (for example date:yesterday) will not work as you might expect. You can achieve the expected result by duplicating the expr both sides of ".." (for example date:yesterday..yesterday). Open-ended ranges won't work with pre-1.2.1 Xapian, but they don't produce an error either. Signed-off-by: Jani Nikula <jani@nikula.org>
2012-10-30 21:32:37 +01:00
delete notmuch->date_range_processor;
notmuch->date_range_processor = NULL;
lib: Add "lastmod:" queries for filtering by last modification The implementation is essentially the same as the date range search prior to Jani's fancy date parser.
2014-10-13 08:20:03 +02:00
delete notmuch->last_mod_range_processor;
notmuch->last_mod_range_processor = NULL;
lib: add return status to database close and destroy notmuch_database_close may fail in Xapian ->flush() or ->close(), so report the status. Similarly for notmuch_database_destroy which calls close. This is required for notmuch insert to report error status if message indexing failed.
2014-04-16 14:59:16 +02:00
return status;
Split notmuch_database_close into two functions Formerly notmuch_database_close closed the xapian database and destroyed the talloc structure associated with the notmuch database object. Split notmuch_database_close into notmuch_database_close and notmuch_database_destroy. This makes it possible for long running programs to close the xapian database and thus release the lock associated with it without destroying the data structures obtained from it. This also makes the api more consistent since every other data structure has a destructor function. The comments in notmuch.h are a courtesy of Austin Clements. Signed-off-by: Justus Winter <4winter@informatik.uni-hamburg.de>
2012-04-25 15:20:16 +02:00
}
lib: add _notmuch_database_reopen The main expected use is to recover from a Xapian::DatabaseChanged exception.
2017-02-24 02:38:24 +01:00
notmuch_status_t
_notmuch_database_reopen (notmuch_database_t *notmuch)
{
if (notmuch->mode != NOTMUCH_DATABASE_MODE_READ_ONLY)
return NOTMUCH_STATUS_UNSUPPORTED_OPERATION;
try {
notmuch->xapian_db->reopen ();
} catch (const Xapian::Error &error) {
if (! notmuch->exception_reported) {
_notmuch_database_log (notmuch, "Error: A Xapian exception reopening database: %s\n",
error.get_msg ().c_str ());
notmuch->exception_reported = TRUE;
}
return NOTMUCH_STATUS_XAPIAN_EXCEPTION;
}
notmuch->view++;
return NOTMUCH_STATUS_SUCCESS;
}
compact: tidy formatting Notmuch compact code whitespace changes to match devel/STYLE.
2013-11-13 18:02:43 +01:00
static int
unlink_cb (const char *path,
unused (const struct stat *sb),
unused (int type),
unused (struct FTW *ftw))
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
{
compact: tidy formatting Notmuch compact code whitespace changes to match devel/STYLE.
2013-11-13 18:02:43 +01:00
return remove (path);
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
}
compact: tidy formatting Notmuch compact code whitespace changes to match devel/STYLE.
2013-11-13 18:02:43 +01:00
static int
rmtree (const char *path)
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
{
compact: tidy formatting Notmuch compact code whitespace changes to match devel/STYLE.
2013-11-13 18:02:43 +01:00
return nftw (path, unlink_cb, 64, FTW_DEPTH | FTW_PHYS);
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
}
class NotmuchCompactor : public Xapian::Compactor
{
notmuch_compact_status_cb_t status_cb;
lib: add closure parameter to compact status update callback This provides much more flexibility for the caller.
2013-11-03 13:24:45 +01:00
void *status_closure;
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
public:
lib: add closure parameter to compact status update callback This provides much more flexibility for the caller.
2013-11-03 13:24:45 +01:00
NotmuchCompactor(notmuch_compact_status_cb_t cb, void *closure) :
compact: tidy formatting Notmuch compact code whitespace changes to match devel/STYLE.
2013-11-13 18:02:43 +01:00
status_cb (cb), status_closure (closure) { }
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
virtual void
set_status (const std::string &table, const std::string &status)
{
compact: tidy formatting Notmuch compact code whitespace changes to match devel/STYLE.
2013-11-13 18:02:43 +01:00
char *msg;
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
if (status_cb == NULL)
return;
compact: tidy formatting Notmuch compact code whitespace changes to match devel/STYLE.
2013-11-13 18:02:43 +01:00
if (status.length () == 0)
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
msg = talloc_asprintf (NULL, "compacting table %s", table.c_str());
else
msg = talloc_asprintf (NULL, " %s", status.c_str());
if (msg == NULL) {
return;
}
compact: tidy formatting Notmuch compact code whitespace changes to match devel/STYLE.
2013-11-13 18:02:43 +01:00
status_cb (msg, status_closure);
talloc_free (msg);
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
}
};
/* Compacts the given database, optionally saving the original database
* in backup_path. Additionally, a callback function can be provided to
* give the user feedback on the progress of the (likely long-lived)
* compaction process.
*
* The backup path must point to a directory on the same volume as the
* original database. Passing a NULL backup_path will result in the
* uncompacted database being deleted after compaction has finished.
* Note that the database write lock will be held during the
* compaction process to protect data integrity.
*/
notmuch_status_t
compact: tidy formatting Notmuch compact code whitespace changes to match devel/STYLE.
2013-11-13 18:02:43 +01:00
notmuch_database_compact (const char *path,
const char *backup_path,
lib: add closure parameter to compact status update callback This provides much more flexibility for the caller.
2013-11-03 13:24:45 +01:00
notmuch_compact_status_cb_t status_cb,
void *closure)
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
{
lib: check talloc success in compact In line with the allocation checks all around.
2013-11-03 13:24:43 +01:00
void *local;
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
char *notmuch_path, *xapian_path, *compact_xapian_path;
notmuch_status_t ret = NOTMUCH_STATUS_SUCCESS;
notmuch_database_t *notmuch = NULL;
struct stat statbuf;
compact: preserve backup database until compacted database is in place It is less error prone and window of failure opportunity is smaller if the old (backup) database is always renamed (instead of sometimes rmtree'd) before new (compacted) database is put into its place. Finally rmtree() old database in case old database backup is not kept.
2013-11-14 23:03:25 +01:00
notmuch_bool_t keep_backup;
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
char *message = NULL;
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
lib: check talloc success in compact In line with the allocation checks all around.
2013-11-03 13:24:43 +01:00
local = talloc_new (NULL);
if (! local)
return NOTMUCH_STATUS_OUT_OF_MEMORY;
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
ret = notmuch_database_open_verbose (path,
NOTMUCH_DATABASE_MODE_READ_WRITE,
&notmuch,
&message);
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
if (ret) {
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
if (status_cb) status_cb (message, closure);
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
goto DONE;
}
if (! (notmuch_path = talloc_asprintf (local, "%s/%s", path, ".notmuch"))) {
ret = NOTMUCH_STATUS_OUT_OF_MEMORY;
goto DONE;
}
if (! (xapian_path = talloc_asprintf (local, "%s/%s", notmuch_path, "xapian"))) {
ret = NOTMUCH_STATUS_OUT_OF_MEMORY;
goto DONE;
}
if (! (compact_xapian_path = talloc_asprintf (local, "%s.compact", xapian_path))) {
ret = NOTMUCH_STATUS_OUT_OF_MEMORY;
goto DONE;
}
compact: preserve backup database until compacted database is in place It is less error prone and window of failure opportunity is smaller if the old (backup) database is always renamed (instead of sometimes rmtree'd) before new (compacted) database is put into its place. Finally rmtree() old database in case old database backup is not kept.
2013-11-14 23:03:25 +01:00
if (backup_path == NULL) {
if (! (backup_path = talloc_asprintf (local, "%s.old", xapian_path))) {
ret = NOTMUCH_STATUS_OUT_OF_MEMORY;
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
goto DONE;
}
compact: preserve backup database until compacted database is in place It is less error prone and window of failure opportunity is smaller if the old (backup) database is always renamed (instead of sometimes rmtree'd) before new (compacted) database is put into its place. Finally rmtree() old database in case old database backup is not kept.
2013-11-14 23:03:25 +01:00
keep_backup = FALSE;
}
else {
keep_backup = TRUE;
}
if (stat (backup_path, &statbuf) != -1) {
lib: replace almost all fprintfs in library with _n_d_log This is not supposed to change any functionality from an end user point of view. Note that it will eliminate some output to stderr. The query debugging output is left as is; it doesn't really fit with the current primitive logging model. The remaining "bad" fprintf will need an internal API change.
2014-12-26 17:25:35 +01:00
_notmuch_database_log (notmuch, "Path already exists: %s\n", backup_path);
compact: preserve backup database until compacted database is in place It is less error prone and window of failure opportunity is smaller if the old (backup) database is always renamed (instead of sometimes rmtree'd) before new (compacted) database is put into its place. Finally rmtree() old database in case old database backup is not kept.
2013-11-14 23:03:25 +01:00
ret = NOTMUCH_STATUS_FILE_ERROR;
goto DONE;
}
if (errno != ENOENT) {
lib: replace almost all fprintfs in library with _n_d_log This is not supposed to change any functionality from an end user point of view. Note that it will eliminate some output to stderr. The query debugging output is left as is; it doesn't really fit with the current primitive logging model. The remaining "bad" fprintf will need an internal API change.
2014-12-26 17:25:35 +01:00
_notmuch_database_log (notmuch, "Unknown error while stat()ing path: %s\n",
compact: preserve backup database until compacted database is in place It is less error prone and window of failure opportunity is smaller if the old (backup) database is always renamed (instead of sometimes rmtree'd) before new (compacted) database is put into its place. Finally rmtree() old database in case old database backup is not kept.
2013-11-14 23:03:25 +01:00
strerror (errno));
ret = NOTMUCH_STATUS_FILE_ERROR;
goto DONE;
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
}
compact: unconditionally remove old wip database compact directory In case previous notmuch compact has been interrupted there is old work-in-progress database compact directory partially filled. Remove it just before starting to fill the directory with new files.
2013-11-14 23:03:26 +01:00
/* Unconditionally attempt to remove old work-in-progress database (if
* any). This is "protected" by database lock. If this fails due to write
* errors (etc), the following code will fail and provide error message.
*/
(void) rmtree (compact_xapian_path);
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
try {
compact: tidy formatting Notmuch compact code whitespace changes to match devel/STYLE.
2013-11-13 18:02:43 +01:00
NotmuchCompactor compactor (status_cb, closure);
lib: construct compactor within try block to catch any exceptions Constructors may also throw exceptions. Catch them.
2013-11-03 13:24:42 +01:00
compact: tidy formatting Notmuch compact code whitespace changes to match devel/STYLE.
2013-11-13 18:02:43 +01:00
compactor.set_renumber (false);
compactor.add_source (xapian_path);
compactor.set_destdir (compact_xapian_path);
compactor.compact ();
compact: catch Xapian::Error consistently catch Xapian::Error in compact code in lib/database.cc to be consistent with other code in addition to not making software crash on uncaught other Xapian error.
2013-11-13 18:02:44 +01:00
} catch (const Xapian::Error &error) {
lib: replace almost all fprintfs in library with _n_d_log This is not supposed to change any functionality from an end user point of view. Note that it will eliminate some output to stderr. The query debugging output is left as is; it doesn't really fit with the current primitive logging model. The remaining "bad" fprintf will need an internal API change.
2014-12-26 17:25:35 +01:00
_notmuch_database_log (notmuch, "Error while compacting: %s\n", error.get_msg().c_str());
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
ret = NOTMUCH_STATUS_XAPIAN_EXCEPTION;
goto DONE;
}
compact: preserve backup database until compacted database is in place It is less error prone and window of failure opportunity is smaller if the old (backup) database is always renamed (instead of sometimes rmtree'd) before new (compacted) database is put into its place. Finally rmtree() old database in case old database backup is not kept.
2013-11-14 23:03:25 +01:00
if (rename (xapian_path, backup_path)) {
lib: replace almost all fprintfs in library with _n_d_log This is not supposed to change any functionality from an end user point of view. Note that it will eliminate some output to stderr. The query debugging output is left as is; it doesn't really fit with the current primitive logging model. The remaining "bad" fprintf will need an internal API change.
2014-12-26 17:25:35 +01:00
_notmuch_database_log (notmuch, "Error moving %s to %s: %s\n",
compact: improve error messages on failures after compaction The error messages written during the steps replacing old database with new now includes relevant paths and strerror.
2013-11-14 23:03:27 +01:00
xapian_path, backup_path, strerror (errno));
compact: preserve backup database until compacted database is in place It is less error prone and window of failure opportunity is smaller if the old (backup) database is always renamed (instead of sometimes rmtree'd) before new (compacted) database is put into its place. Finally rmtree() old database in case old database backup is not kept.
2013-11-14 23:03:25 +01:00
ret = NOTMUCH_STATUS_FILE_ERROR;
goto DONE;
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
}
compact: tidy formatting Notmuch compact code whitespace changes to match devel/STYLE.
2013-11-13 18:02:43 +01:00
if (rename (compact_xapian_path, xapian_path)) {
lib: replace almost all fprintfs in library with _n_d_log This is not supposed to change any functionality from an end user point of view. Note that it will eliminate some output to stderr. The query debugging output is left as is; it doesn't really fit with the current primitive logging model. The remaining "bad" fprintf will need an internal API change.
2014-12-26 17:25:35 +01:00
_notmuch_database_log (notmuch, "Error moving %s to %s: %s\n",
compact: improve error messages on failures after compaction The error messages written during the steps replacing old database with new now includes relevant paths and strerror.
2013-11-14 23:03:27 +01:00
compact_xapian_path, xapian_path, strerror (errno));
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
ret = NOTMUCH_STATUS_FILE_ERROR;
goto DONE;
}
compact: improve error messages on failures after compaction The error messages written during the steps replacing old database with new now includes relevant paths and strerror.
2013-11-14 23:03:27 +01:00
if (! keep_backup) {
if (rmtree (backup_path)) {
lib: replace almost all fprintfs in library with _n_d_log This is not supposed to change any functionality from an end user point of view. Note that it will eliminate some output to stderr. The query debugging output is left as is; it doesn't really fit with the current primitive logging model. The remaining "bad" fprintf will need an internal API change.
2014-12-26 17:25:35 +01:00
_notmuch_database_log (notmuch, "Error removing old database %s: %s\n",
compact: improve error messages on failures after compaction The error messages written during the steps replacing old database with new now includes relevant paths and strerror.
2013-11-14 23:03:27 +01:00
backup_path, strerror (errno));
ret = NOTMUCH_STATUS_FILE_ERROR;
goto DONE;
}
}
compact: preserve backup database until compacted database is in place It is less error prone and window of failure opportunity is smaller if the old (backup) database is always renamed (instead of sometimes rmtree'd) before new (compacted) database is put into its place. Finally rmtree() old database in case old database backup is not kept.
2013-11-14 23:03:25 +01:00
compact: tidy formatting Notmuch compact code whitespace changes to match devel/STYLE.
2013-11-13 18:02:43 +01:00
DONE:
lib: add return status to database close and destroy notmuch_database_close may fail in Xapian ->flush() or ->close(), so report the status. Similarly for notmuch_database_destroy which calls close. This is required for notmuch insert to report error status if message indexing failed.
2014-04-16 14:59:16 +02:00
if (notmuch) {
notmuch_status_t ret2;
lib: replace almost all fprintfs in library with _n_d_log This is not supposed to change any functionality from an end user point of view. Note that it will eliminate some output to stderr. The query debugging output is left as is; it doesn't really fit with the current primitive logging model. The remaining "bad" fprintf will need an internal API change.
2014-12-26 17:25:35 +01:00
const char *str = notmuch_database_status_string (notmuch);
if (status_cb && str)
status_cb (str, closure);
lib: add return status to database close and destroy notmuch_database_close may fail in Xapian ->flush() or ->close(), so report the status. Similarly for notmuch_database_destroy which calls close. This is required for notmuch insert to report error status if message indexing failed.
2014-04-16 14:59:16 +02:00
ret2 = notmuch_database_destroy (notmuch);
/* don't clobber previous error status */
if (ret == NOTMUCH_STATUS_SUCCESS && ret2 != NOTMUCH_STATUS_SUCCESS)
ret = ret2;
}
lib: do not leak the database in compaction Destroy instead of close the database after compaction, and also on error path, to not leak the database.
2013-11-03 13:24:44 +01:00
compact: tidy formatting Notmuch compact code whitespace changes to match devel/STYLE.
2013-11-13 18:02:43 +01:00
talloc_free (local);
lib: do not leak the database in compaction Destroy instead of close the database after compaction, and also on error path, to not leak the database.
2013-11-03 13:24:44 +01:00
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
return ret;
}
lib: add return status to database close and destroy notmuch_database_close may fail in Xapian ->flush() or ->close(), so report the status. Similarly for notmuch_database_destroy which calls close. This is required for notmuch insert to report error status if message indexing failed.
2014-04-16 14:59:16 +02:00
notmuch_status_t
Split notmuch_database_close into two functions Formerly notmuch_database_close closed the xapian database and destroyed the talloc structure associated with the notmuch database object. Split notmuch_database_close into notmuch_database_close and notmuch_database_destroy. This makes it possible for long running programs to close the xapian database and thus release the lock associated with it without destroying the data structures obtained from it. This also makes the api more consistent since every other data structure has a destructor function. The comments in notmuch.h are a courtesy of Austin Clements. Signed-off-by: Justus Winter <4winter@informatik.uni-hamburg.de>
2012-04-25 15:20:16 +02:00
notmuch_database_destroy (notmuch_database_t *notmuch)
{
lib: add return status to database close and destroy notmuch_database_close may fail in Xapian ->flush() or ->close(), so report the status. Similarly for notmuch_database_destroy which calls close. This is required for notmuch insert to report error status if message indexing failed.
2014-04-16 14:59:16 +02:00
notmuch_status_t status;
status = notmuch_database_close (notmuch);
Convert notmuch_database_t to start using talloc. This will be handy as we can hang future talloc allocations off of the datbase now.
2009-10-21 23:00:37 +02:00
talloc_free (notmuch);
lib: add return status to database close and destroy notmuch_database_close may fail in Xapian ->flush() or ->close(), so report the status. Similarly for notmuch_database_destroy which calls close. This is required for notmuch insert to report error status if message indexing failed.
2014-04-16 14:59:16 +02:00
return status;
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
}
const char *
notmuch_database_get_path (notmuch_database_t *notmuch)
{
return notmuch->path;
}
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
unsigned int
notmuch_database_get_version (notmuch_database_t *notmuch)
{
unsigned int version;
string version_string;
const char *str;
char *end;
version_string = notmuch->xapian_db->get_metadata ("version");
if (version_string.empty ())
return 0;
str = version_string.c_str ();
if (str == NULL || *str == '\0')
return 0;
version = strtoul (str, &end, 10);
if (*end != '\0')
INTERNAL_ERROR ("Malformed database version: %s", str);
return version;
}
notmuch_bool_t
notmuch_database_needs_upgrade (notmuch_database_t *notmuch)
{
lib: Database version 3: Introduce fine-grained "features" Previously, our database schema was versioned by a single number. Each database schema change had to occur "atomically" in Notmuch's development history: before some commit, Notmuch used version N, after that commit, it used version N+1. Hence, each new schema version could introduce only one change, the task of developing a schema change fell on a single person, and it all had to happen and be perfect in a single commit series. This made introducing a new schema version hard. We've seen only two schema changes in the history of Notmuch. This commit introduces database schema version 3; hopefully the last schema version we'll need for a while. With this version, we switch from a single version number to "features": a set of named, independent aspects of the database schema. Features should make backwards compatibility easier. For many things, it should be easy to support databases both with and without a feature, which will allow us to make upgrades optional and will enable "unstable" features that can be developed and tested over time. Features also make forwards compatibility easier. The features recorded in a database include "compatibility flags," which can indicate to an older version of Notmuch when it must support a given feature to open the database for read or for write. This lets us replace the old vague "I don't recognize this version, so something might go wrong, but I promise to try my best" warnings upon opening a database with an unknown version with precise errors. If a database is safe to open for read/write despite unknown features, an older version will know that and issue no message at all. If the database is not safe to open for read/write because of unknown features, an older version will know that, too, and can tell the user exactly which required features it lacks support for.
2014-08-25 19:26:00 +02:00
return notmuch->mode == NOTMUCH_DATABASE_MODE_READ_WRITE &&
((NOTMUCH_FEATURES_CURRENT & ~notmuch->features) ||
(notmuch_database_get_version (notmuch) < NOTMUCH_DATABASE_VERSION));
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
}
static volatile sig_atomic_t do_progress_notify = 0;
static void
handle_sigalrm (unused (int signal))
{
do_progress_notify = 1;
}
/* Upgrade the current database.
*
* After opening a database in read-write mode, the client should
* check if an upgrade is needed (notmuch_database_needs_upgrade) and
* if so, upgrade with this function before making any modifications.
*
* The optional progress_notify callback can be used by the caller to
* provide progress indication to the user. If non-NULL it will be
* called periodically with 'count' as the number of messages upgraded
* so far and 'total' the overall number of messages that will be
* converted.
*/
notmuch_status_t
notmuch_database_upgrade (notmuch_database_t *notmuch,
void (*progress_notify) (void *closure,
notmuch new: Print upgrade progress report as a percentage. Previously we were printing a number of messages upgraded so far. The original motivation for this was to accurately reflect the fact that there are two passes, (so each message is processed twice and it's not accurate to represent with a single count). But as it turns out, the second pass takes zero time (relatively speaking) so we're still not accounting for it. If nothing else, the percentage-based reporting makes for a cleaner API for the progress_notify function.
2010-01-10 02:38:23 +01:00
double progress),
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
void *closure)
{
lib: Database version 3: Introduce fine-grained "features" Previously, our database schema was versioned by a single number. Each database schema change had to occur "atomically" in Notmuch's development history: before some commit, Notmuch used version N, after that commit, it used version N+1. Hence, each new schema version could introduce only one change, the task of developing a schema change fell on a single person, and it all had to happen and be perfect in a single commit series. This made introducing a new schema version hard. We've seen only two schema changes in the history of Notmuch. This commit introduces database schema version 3; hopefully the last schema version we'll need for a while. With this version, we switch from a single version number to "features": a set of named, independent aspects of the database schema. Features should make backwards compatibility easier. For many things, it should be easy to support databases both with and without a feature, which will allow us to make upgrades optional and will enable "unstable" features that can be developed and tested over time. Features also make forwards compatibility easier. The features recorded in a database include "compatibility flags," which can indicate to an older version of Notmuch when it must support a given feature to open the database for read or for write. This lets us replace the old vague "I don't recognize this version, so something might go wrong, but I promise to try my best" warnings upon opening a database with an unknown version with precise errors. If a database is safe to open for read/write despite unknown features, an older version will know that and issue no message at all. If the database is not safe to open for read/write because of unknown features, an older version will know that, too, and can tell the user exactly which required features it lacks support for.
2014-08-25 19:26:00 +02:00
void *local = talloc_new (NULL);
lib: Use database features to drive upgrade Previously, we had database version information hard-coded in the upgrade code. Slightly re-organize the upgrade process around the set of new database features to be enabled by the upgrade.
2014-08-25 19:26:04 +02:00
Xapian::TermIterator t, t_end;
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
Xapian::WritableDatabase *db;
struct sigaction action;
struct itimerval timerval;
notmuch_bool_t timer_is_active = FALSE;
lib: Use database features to drive upgrade Previously, we had database version information hard-coded in the upgrade code. Slightly re-organize the upgrade process around the set of new database features to be enabled by the upgrade.
2014-08-25 19:26:04 +02:00
enum _notmuch_features target_features, new_features;
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
notmuch_status_t status;
lib: Implement upgrade to ghost messages feature Somehow this is the first upgrade pass that actually does *any* error checking, so this also adds the bit of necessary infrastructure to handle that.
2014-10-23 14:30:38 +02:00
notmuch_private_status_t private_status;
lib: move query variable to function scope This is a prelude to deallocating it (if necessary) on the error path.
2015-09-27 17:31:55 +02:00
notmuch_query_t *query = NULL;
notmuch new: Print upgrade progress report as a percentage. Previously we were printing a number of messages upgraded so far. The original motivation for this was to accurately reflect the fact that there are two passes, (so each message is processed twice and it's not accurate to represent with a single count). But as it turns out, the second pass takes zero time (relatively speaking) so we're still not accounting for it. If nothing else, the percentage-based reporting makes for a cleaner API for the progress_notify function.
2010-01-10 02:38:23 +01:00
unsigned int count = 0, total = 0;
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
status = _notmuch_database_ensure_writable (notmuch);
if (status)
return status;
db = static_cast <Xapian::WritableDatabase *> (notmuch->xapian_db);
lib: Use database features to drive upgrade Previously, we had database version information hard-coded in the upgrade code. Slightly re-organize the upgrade process around the set of new database features to be enabled by the upgrade.
2014-08-25 19:26:04 +02:00
target_features = notmuch->features | NOTMUCH_FEATURES_CURRENT;
new_features = NOTMUCH_FEATURES_CURRENT & ~notmuch->features;
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
lib: Fix endless upgrade problem 48db8c8 introduced a disagreement between when notmuch_database_needs_upgrade returned TRUE and when notmuch_database_upgrade actually performed an upgrade. As a result, if a database had a version less than 3, but no new features were required, notmuch new would call notmuch_database_upgrade to perform an upgrade, but notmuch_database_upgrade would return immediately without updating the database version. Hence, the next notmuch new would do the same, and so on. Fix this by ensuring that the upgrade-required logic is identical between the two.
2014-09-02 00:49:07 +02:00
if (! notmuch_database_needs_upgrade (notmuch))
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
return NOTMUCH_STATUS_SUCCESS;
if (progress_notify) {
cli: change "setup" to "set up" where used as a verb The word "setup" is a noun, not a verb. Change occurrences of "setup" where used as a verb to "set up".
2015-05-27 19:53:52 +02:00
/* Set up our handler for SIGALRM */
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
memset (&action, 0, sizeof (struct sigaction));
action.sa_handler = handle_sigalrm;
sigemptyset (&action.sa_mask);
action.sa_flags = SA_RESTART;
sigaction (SIGALRM, &action, NULL);
/* Then start a timer to send SIGALRM once per second. */
timerval.it_interval.tv_sec = 1;
timerval.it_interval.tv_usec = 0;
timerval.it_value.tv_sec = 1;
timerval.it_value.tv_usec = 0;
setitimer (ITIMER_REAL, &timerval, NULL);
timer_is_active = TRUE;
}
lib: Report progress for combined upgrade operation Previously, some parts of upgrade didn't report progress and for others it was possible for the progress meter to restart at 0 part way through the upgrade because each stage was reported separately. Fix this by computing the total amount of work that needs to be done up-front and updating completed work monotonically.
2014-08-25 19:26:06 +02:00
/* Figure out how much total work we need to do. */
if (new_features &
lib: Add per-message last modification tracking This adds a new document value that stores the revision of the last modification to message metadata, where the revision number increases monotonically with each database commit. An alternative would be to store the wall-clock time of the last modification of each message. In principle this is simpler and has the advantage that any process can determine the current timestamp without support from libnotmuch. However, even assuming a computer's clock never goes backward and ignoring clock skew in networked environments, this has a fatal flaw. Xapian uses (optimistic) snapshot isolation, which means reads can be concurrent with writes. Given this, consider the following time line with a write and two read transactions: write |-X-A--------------| read 1 |---B---| read 2 |---| The write transaction modifies message X and records the wall-clock time of the modification at A. The writer hangs around for a while and later commits its change. Read 1 is concurrent with the write, so it doesn't see the change to X. It does some query and records the wall-clock time of its results at B. Transaction read 2 later starts after the write commits and queries for changes since wall-clock time B (say the reads are performing an incremental backup). Even though read 1 could not see the change to X, read 2 is told (correctly) that X has not changed since B, the time of the last read. In fact, X changed before wall-clock time A, but the change was not visible until *after* wall-clock time B, so read 2 misses the change to X. This is tricky to solve in full-blown snapshot isolation, but because Xapian serializes writes, we can use a simple, monotonically increasing database revision number. Furthermore, maintaining this revision number requires no more IO than a wall-clock time solution because Xapian already maintains statistics on the upper (and lower) bound of each value stream.
2014-10-13 08:20:01 +02:00
(NOTMUCH_FEATURE_FILE_TERMS | NOTMUCH_FEATURE_BOOL_FOLDER |
NOTMUCH_FEATURE_LAST_MOD)) {
lib: move query variable to function scope This is a prelude to deallocating it (if necessary) on the error path.
2015-09-27 17:31:55 +02:00
query = notmuch_query_create (notmuch, "");
lib: add versions of n_q_count_{message,threads} with status return Although I think it's a pretty bad idea to continue using the old API, this allows both a more gentle transition for clients of the library, and allows us to break one monolithic change into a series
2015-09-27 17:31:57 +02:00
unsigned msg_count;
status = notmuch_query_count_messages_st (query, &msg_count);
if (status)
goto DONE;
total += msg_count;
lib: Report progress for combined upgrade operation Previously, some parts of upgrade didn't report progress and for others it was possible for the progress meter to restart at 0 part way through the upgrade because each stage was reported separately. Fix this by computing the total amount of work that needs to be done up-front and updating completed work monotonically.
2014-08-25 19:26:06 +02:00
notmuch_query_destroy (query);
lib: add versions of n_q_count_{message,threads} with status return Although I think it's a pretty bad idea to continue using the old API, this allows both a more gentle transition for clients of the library, and allows us to break one monolithic change into a series
2015-09-27 17:31:57 +02:00
query = NULL;
lib: Report progress for combined upgrade operation Previously, some parts of upgrade didn't report progress and for others it was possible for the progress meter to restart at 0 part way through the upgrade because each stage was reported separately. Fix this by computing the total amount of work that needs to be done up-front and updating completed work monotonically.
2014-08-25 19:26:06 +02:00
}
if (new_features & NOTMUCH_FEATURE_DIRECTORY_DOCS) {
t_end = db->allterms_end ("XTIMESTAMP");
for (t = db->allterms_begin ("XTIMESTAMP"); t != t_end; t++)
++total;
}
lib: Implement upgrade to ghost messages feature Somehow this is the first upgrade pass that actually does *any* error checking, so this also adds the bit of necessary infrastructure to handle that.
2014-10-23 14:30:38 +02:00
if (new_features & NOTMUCH_FEATURE_GHOSTS) {
/* The ghost message upgrade converts all thread_id_*
* metadata values into ghost message documents. */
t_end = db->metadata_keys_end ("thread_id_");
for (t = db->metadata_keys_begin ("thread_id_"); t != t_end; ++t)
++total;
}
lib: Report progress for combined upgrade operation Previously, some parts of upgrade didn't report progress and for others it was possible for the progress meter to restart at 0 part way through the upgrade because each stage was reported separately. Fix this by computing the total amount of work that needs to be done up-front and updating completed work monotonically.
2014-08-25 19:26:06 +02:00
lib: Simplify upgrade code using a transaction Previously, the upgrade was organized as two passes -- an upgrade pass, and a separate cleanup pass -- so the database was always in a valid state. This change substantially simplifies this code by performing the upgrade in a transaction and combining both passes in to one. This 1) eliminates a lot of duplicate code between the passes, 2) speeds up the upgrade process, 3) makes progress reporting more accurate, 4) eliminates the potential for stale data if the upgrade is interrupted during the cleanup pass, and 5) makes it easier to reason about the safety of the upgrade code.
2014-08-25 19:26:03 +02:00
/* Perform the upgrade in a transaction. */
db->begin_transaction (true);
lib: Database version 3: Introduce fine-grained "features" Previously, our database schema was versioned by a single number. Each database schema change had to occur "atomically" in Notmuch's development history: before some commit, Notmuch used version N, after that commit, it used version N+1. Hence, each new schema version could introduce only one change, the task of developing a schema change fell on a single person, and it all had to happen and be perfect in a single commit series. This made introducing a new schema version hard. We've seen only two schema changes in the history of Notmuch. This commit introduces database schema version 3; hopefully the last schema version we'll need for a while. With this version, we switch from a single version number to "features": a set of named, independent aspects of the database schema. Features should make backwards compatibility easier. For many things, it should be easy to support databases both with and without a feature, which will allow us to make upgrades optional and will enable "unstable" features that can be developed and tested over time. Features also make forwards compatibility easier. The features recorded in a database include "compatibility flags," which can indicate to an older version of Notmuch when it must support a given feature to open the database for read or for write. This lets us replace the old vague "I don't recognize this version, so something might go wrong, but I promise to try my best" warnings upon opening a database with an unknown version with precise errors. If a database is safe to open for read/write despite unknown features, an older version will know that and issue no message at all. If the database is not safe to open for read/write because of unknown features, an older version will know that, too, and can tell the user exactly which required features it lacks support for.
2014-08-25 19:26:00 +02:00
/* Set the target features so we write out changes in the desired
* format. */
lib: Use database features to drive upgrade Previously, we had database version information hard-coded in the upgrade code. Slightly re-organize the upgrade process around the set of new database features to be enabled by the upgrade.
2014-08-25 19:26:04 +02:00
notmuch->features = target_features;
lib: Database version 3: Introduce fine-grained "features" Previously, our database schema was versioned by a single number. Each database schema change had to occur "atomically" in Notmuch's development history: before some commit, Notmuch used version N, after that commit, it used version N+1. Hence, each new schema version could introduce only one change, the task of developing a schema change fell on a single person, and it all had to happen and be perfect in a single commit series. This made introducing a new schema version hard. We've seen only two schema changes in the history of Notmuch. This commit introduces database schema version 3; hopefully the last schema version we'll need for a while. With this version, we switch from a single version number to "features": a set of named, independent aspects of the database schema. Features should make backwards compatibility easier. For many things, it should be easy to support databases both with and without a feature, which will allow us to make upgrades optional and will enable "unstable" features that can be developed and tested over time. Features also make forwards compatibility easier. The features recorded in a database include "compatibility flags," which can indicate to an older version of Notmuch when it must support a given feature to open the database for read or for write. This lets us replace the old vague "I don't recognize this version, so something might go wrong, but I promise to try my best" warnings upon opening a database with an unknown version with precise errors. If a database is safe to open for read/write despite unknown features, an older version will know that and issue no message at all. If the database is not safe to open for read/write because of unknown features, an older version will know that, too, and can tell the user exactly which required features it lacks support for.
2014-08-25 19:26:00 +02:00
lib: Reorganize upgrade around document types Rather than potentially making multiple passes over the same type of data in the database, reorganize upgrade around each type of data that may be upgraded. This eliminates code duplication, will make multi-version upgrades faster, and will let us improve progress reporting.
2014-08-25 19:26:05 +02:00
/* Perform per-message upgrades. */
if (new_features &
lib: Add per-message last modification tracking This adds a new document value that stores the revision of the last modification to message metadata, where the revision number increases monotonically with each database commit. An alternative would be to store the wall-clock time of the last modification of each message. In principle this is simpler and has the advantage that any process can determine the current timestamp without support from libnotmuch. However, even assuming a computer's clock never goes backward and ignoring clock skew in networked environments, this has a fatal flaw. Xapian uses (optimistic) snapshot isolation, which means reads can be concurrent with writes. Given this, consider the following time line with a write and two read transactions: write |-X-A--------------| read 1 |---B---| read 2 |---| The write transaction modifies message X and records the wall-clock time of the modification at A. The writer hangs around for a while and later commits its change. Read 1 is concurrent with the write, so it doesn't see the change to X. It does some query and records the wall-clock time of its results at B. Transaction read 2 later starts after the write commits and queries for changes since wall-clock time B (say the reads are performing an incremental backup). Even though read 1 could not see the change to X, read 2 is told (correctly) that X has not changed since B, the time of the last read. In fact, X changed before wall-clock time A, but the change was not visible until *after* wall-clock time B, so read 2 misses the change to X. This is tricky to solve in full-blown snapshot isolation, but because Xapian serializes writes, we can use a simple, monotonically increasing database revision number. Furthermore, maintaining this revision number requires no more IO than a wall-clock time solution because Xapian already maintains statistics on the upper (and lower) bound of each value stream.
2014-10-13 08:20:01 +02:00
(NOTMUCH_FEATURE_FILE_TERMS | NOTMUCH_FEATURE_BOOL_FOLDER |
NOTMUCH_FEATURE_LAST_MOD)) {
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
notmuch_messages_t *messages;
notmuch_message_t *message;
lib: Split the database upgrade into two phases for safer operation. The first phase copies data from the old format to the new format without deleting anything. This allows an old notmuch to still use the database if the upgrade process gets interrupted. The second phase performs the deletion (after updating the database version number). If the second phase is interrupted, there will be some unused data in the database, but it shouldn't cause any actual harm.
2010-01-09 20:13:12 +01:00
char *filename;
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
lib: move query variable to function scope This is a prelude to deallocating it (if necessary) on the error path.
2015-09-27 17:31:55 +02:00
query = notmuch_query_create (notmuch, "");
lib: migrate notmuch_database_upgrade to new query_search API Here we depend on the error path cleaning up query
2015-09-27 17:32:01 +02:00
status = notmuch_query_search_messages_st (query, &messages);
if (status)
goto DONE;
for (;
lib: Rename iterator functions to prepare for reverse iteration. We rename 'has_more' to 'valid' so that it can function whether iterating in a forward or reverse direction. We also rename 'advance' to 'move_to_next' to setup parallel naming with the proposed functions 'move_to_first', 'move_to_last', and 'move_to_previous'.
2010-03-09 18:22:29 +01:00
notmuch_messages_valid (messages);
notmuch_messages_move_to_next (messages))
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
{
notmuch new: Fix progress notification on database upgrade. This was firing continuously rather than just once per second as intended.
2010-01-08 06:24:44 +01:00
if (do_progress_notify) {
notmuch new: Print upgrade progress report as a percentage. Previously we were printing a number of messages upgraded so far. The original motivation for this was to accurately reflect the fact that there are two passes, (so each message is processed twice and it's not accurate to represent with a single count). But as it turns out, the second pass takes zero time (relatively speaking) so we're still not accounting for it. If nothing else, the percentage-based reporting makes for a cleaner API for the progress_notify function.
2010-01-10 02:38:23 +01:00
progress_notify (closure, (double) count / total);
notmuch new: Fix progress notification on database upgrade. This was firing continuously rather than just once per second as intended.
2010-01-08 06:24:44 +01:00
do_progress_notify = 0;
}
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
message = notmuch_messages_get (messages);
lib: Reorganize upgrade around document types Rather than potentially making multiple passes over the same type of data in the database, reorganize upgrade around each type of data that may be upgraded. This eliminates code duplication, will make multi-version upgrades faster, and will let us improve progress reporting.
2014-08-25 19:26:05 +02:00
/* Before version 1, each message document had its
* filename in the data field. Copy that into the new
* format by calling notmuch_message_add_filename.
*/
if (new_features & NOTMUCH_FEATURE_FILE_TERMS) {
filename = _notmuch_message_talloc_copy_data (message);
if (filename && *filename != '\0') {
_notmuch_message_add_filename (message, filename);
_notmuch_message_clear_data (message);
}
talloc_free (filename);
lib: Split the database upgrade into two phases for safer operation. The first phase copies data from the old format to the new format without deleting anything. This allows an old notmuch to still use the database if the upgrade process gets interrupted. The second phase performs the deletion (after updating the database version number). If the second phase is interrupted, there will be some unused data in the database, but it shouldn't cause any actual harm.
2010-01-09 20:13:12 +01:00
}
lib: Reorganize upgrade around document types Rather than potentially making multiple passes over the same type of data in the database, reorganize upgrade around each type of data that may be upgraded. This eliminates code duplication, will make multi-version upgrades faster, and will let us improve progress reporting.
2014-08-25 19:26:05 +02:00
/* Prior to version 2, the "folder:" prefix was
* probabilistic and stemmed. Change it to the current
* boolean prefix. Add "path:" prefixes while at it.
*/
if (new_features & NOTMUCH_FEATURE_BOOL_FOLDER)
_notmuch_message_upgrade_folder (message);
lib: Add per-message last modification tracking This adds a new document value that stores the revision of the last modification to message metadata, where the revision number increases monotonically with each database commit. An alternative would be to store the wall-clock time of the last modification of each message. In principle this is simpler and has the advantage that any process can determine the current timestamp without support from libnotmuch. However, even assuming a computer's clock never goes backward and ignoring clock skew in networked environments, this has a fatal flaw. Xapian uses (optimistic) snapshot isolation, which means reads can be concurrent with writes. Given this, consider the following time line with a write and two read transactions: write |-X-A--------------| read 1 |---B---| read 2 |---| The write transaction modifies message X and records the wall-clock time of the modification at A. The writer hangs around for a while and later commits its change. Read 1 is concurrent with the write, so it doesn't see the change to X. It does some query and records the wall-clock time of its results at B. Transaction read 2 later starts after the write commits and queries for changes since wall-clock time B (say the reads are performing an incremental backup). Even though read 1 could not see the change to X, read 2 is told (correctly) that X has not changed since B, the time of the last read. In fact, X changed before wall-clock time A, but the change was not visible until *after* wall-clock time B, so read 2 misses the change to X. This is tricky to solve in full-blown snapshot isolation, but because Xapian serializes writes, we can use a simple, monotonically increasing database revision number. Furthermore, maintaining this revision number requires no more IO than a wall-clock time solution because Xapian already maintains statistics on the upper (and lower) bound of each value stream.
2014-10-13 08:20:01 +02:00
/* Prior to NOTMUCH_FEATURE_LAST_MOD, messages did not
* track modification revisions. Give all messages the
* next available revision; since we just started tracking
* revisions for this database, that will be 1.
*/
if (new_features & NOTMUCH_FEATURE_LAST_MOD)
_notmuch_message_upgrade_last_mod (message);
lib: Reorganize upgrade around document types Rather than potentially making multiple passes over the same type of data in the database, reorganize upgrade around each type of data that may be upgraded. This eliminates code duplication, will make multi-version upgrades faster, and will let us improve progress reporting.
2014-08-25 19:26:05 +02:00
_notmuch_message_sync (message);
lib: Split the database upgrade into two phases for safer operation. The first phase copies data from the old format to the new format without deleting anything. This allows an old notmuch to still use the database if the upgrade process gets interrupted. The second phase performs the deletion (after updating the database version number). If the second phase is interrupted, there will be some unused data in the database, but it shouldn't cause any actual harm.
2010-01-09 20:13:12 +01:00
notmuch_message_destroy (message);
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
count++;
}
lib: Split the database upgrade into two phases for safer operation. The first phase copies data from the old format to the new format without deleting anything. This allows an old notmuch to still use the database if the upgrade process gets interrupted. The second phase performs the deletion (after updating the database version number). If the second phase is interrupted, there will be some unused data in the database, but it shouldn't cause any actual harm.
2010-01-09 20:13:12 +01:00
notmuch_query_destroy (query);
lib: add versions of n_q_count_{message,threads} with status return Although I think it's a pretty bad idea to continue using the old API, this allows both a more gentle transition for clients of the library, and allows us to break one monolithic change into a series
2015-09-27 17:31:57 +02:00
query = NULL;
lib: Use database features to drive upgrade Previously, we had database version information hard-coded in the upgrade code. Slightly re-organize the upgrade process around the set of new database features to be enabled by the upgrade.
2014-08-25 19:26:04 +02:00
}
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
lib: Reorganize upgrade around document types Rather than potentially making multiple passes over the same type of data in the database, reorganize upgrade around each type of data that may be upgraded. This eliminates code duplication, will make multi-version upgrades faster, and will let us improve progress reporting.
2014-08-25 19:26:05 +02:00
/* Perform per-directory upgrades. */
/* Before version 1 we stored directory timestamps in
lib: Use database features to drive upgrade Previously, we had database version information hard-coded in the upgrade code. Slightly re-organize the upgrade process around the set of new database features to be enabled by the upgrade.
2014-08-25 19:26:04 +02:00
* XTIMESTAMP documents instead of the current XDIRECTORY
* documents. So copy those as well. */
if (new_features & NOTMUCH_FEATURE_DIRECTORY_DOCS) {
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
t_end = notmuch->xapian_db->allterms_end ("XTIMESTAMP");
for (t = notmuch->xapian_db->allterms_begin ("XTIMESTAMP");
t != t_end;
t++)
{
Xapian::PostingIterator p, p_end;
std::string term = *t;
p_end = notmuch->xapian_db->postlist_end (term);
for (p = notmuch->xapian_db->postlist_begin (term);
p != p_end;
p++)
{
Xapian::Document document;
time_t mtime;
notmuch_directory_t *directory;
notmuch new: Print upgrade progress report as a percentage. Previously we were printing a number of messages upgraded so far. The original motivation for this was to accurately reflect the fact that there are two passes, (so each message is processed twice and it's not accurate to represent with a single count). But as it turns out, the second pass takes zero time (relatively speaking) so we're still not accounting for it. If nothing else, the percentage-based reporting makes for a cleaner API for the progress_notify function.
2010-01-10 02:38:23 +01:00
if (do_progress_notify) {
progress_notify (closure, (double) count / total);
do_progress_notify = 0;
}
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
document = find_document_for_doc_id (notmuch, *p);
mtime = Xapian::sortable_unserialise (
document.get_value (NOTMUCH_VALUE_TIMESTAMP));
lib/cli: Make notmuch_database_get_directory return a status code Previously, notmuch_database_get_directory had no way to indicate how it had failed. This changes its prototype to return a status code and set an out-argument to the retrieved directory, like similar functions in the library API. This does *not* change its currently broken behavior of creating directory objects when they don't exist, but it does document it and paves the way for fixing this. Also, it can now check for a read-only database and return NOTMUCH_STATUS_READ_ONLY_DATABASE instead of crashing. In the interest of atomicity, this also updates calls from the CLI so that notmuch still compiles.
2012-05-14 01:36:09 +02:00
directory = _notmuch_directory_create (notmuch, term.c_str() + 10,
lib: Make directory document creation optional for _notmuch_directory_create Previously this function would create directory documents if they didn't exist. As a result, it could only be used on writable databases. This adds an argument to make creation optional and to make this function work on read-only databases. We use a flag argument to avoid a bare boolean and to permit future expansion. Both callers have been updated, but currently retain the old behavior. We'll take advantage of the new argument in the following patches.
2012-05-18 06:13:34 +02:00
NOTMUCH_FIND_CREATE, &status);
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
notmuch_directory_set_mtime (directory, mtime);
notmuch_directory_destroy (directory);
lib: Simplify upgrade code using a transaction Previously, the upgrade was organized as two passes -- an upgrade pass, and a separate cleanup pass -- so the database was always in a valid state. This change substantially simplifies this code by performing the upgrade in a transaction and combining both passes in to one. This 1) eliminates a lot of duplicate code between the passes, 2) speeds up the upgrade process, 3) makes progress reporting more accurate, 4) eliminates the potential for stale data if the upgrade is interrupted during the cleanup pass, and 5) makes it easier to reason about the safety of the upgrade code.
2014-08-25 19:26:03 +02:00
db->delete_document (*p);
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
}
lib: Report progress for combined upgrade operation Previously, some parts of upgrade didn't report progress and for others it was possible for the progress meter to restart at 0 part way through the upgrade because each stage was reported separately. Fix this by computing the total amount of work that needs to be done up-front and updating completed work monotonically.
2014-08-25 19:26:06 +02:00
++count;
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
}
}
lib: Implement upgrade to ghost messages feature Somehow this is the first upgrade pass that actually does *any* error checking, so this also adds the bit of necessary infrastructure to handle that.
2014-10-23 14:30:38 +02:00
/* Perform metadata upgrades. */
/* Prior to NOTMUCH_FEATURE_GHOSTS, thread IDs for missing
* messages were stored as database metadata. Change these to
* ghost messages.
*/
if (new_features & NOTMUCH_FEATURE_GHOSTS) {
notmuch_message_t *message;
std::string message_id, thread_id;
t_end = db->metadata_keys_end (NOTMUCH_METADATA_THREAD_ID_PREFIX);
for (t = db->metadata_keys_begin (NOTMUCH_METADATA_THREAD_ID_PREFIX);
t != t_end; ++t) {
if (do_progress_notify) {
progress_notify (closure, (double) count / total);
do_progress_notify = 0;
}
message_id = (*t).substr (
strlen (NOTMUCH_METADATA_THREAD_ID_PREFIX));
thread_id = db->get_metadata (*t);
/* Create ghost message */
message = _notmuch_message_create_for_message_id (
notmuch, message_id.c_str (), &private_status);
if (private_status == NOTMUCH_PRIVATE_STATUS_SUCCESS) {
/* Document already exists; ignore the stored thread ID */
} else if (private_status ==
NOTMUCH_PRIVATE_STATUS_NO_DOCUMENT_FOUND) {
private_status = _notmuch_message_initialize_ghost (
message, thread_id.c_str ());
if (! private_status)
_notmuch_message_sync (message);
}
if (private_status) {
lib: replace almost all fprintfs in library with _n_d_log This is not supposed to change any functionality from an end user point of view. Note that it will eliminate some output to stderr. The query debugging output is left as is; it doesn't really fit with the current primitive logging model. The remaining "bad" fprintf will need an internal API change.
2014-12-26 17:25:35 +01:00
_notmuch_database_log (notmuch,
lib: Implement upgrade to ghost messages feature Somehow this is the first upgrade pass that actually does *any* error checking, so this also adds the bit of necessary infrastructure to handle that.
2014-10-23 14:30:38 +02:00
"Upgrade failed while creating ghost messages.\n");
status = COERCE_STATUS (private_status, "Unexpected status from _notmuch_message_initialize_ghost");
goto DONE;
}
/* Clear saved metadata thread ID */
db->set_metadata (*t, "");
++count;
}
}
status = NOTMUCH_STATUS_SUCCESS;
lib: Database version 3: Introduce fine-grained "features" Previously, our database schema was versioned by a single number. Each database schema change had to occur "atomically" in Notmuch's development history: before some commit, Notmuch used version N, after that commit, it used version N+1. Hence, each new schema version could introduce only one change, the task of developing a schema change fell on a single person, and it all had to happen and be perfect in a single commit series. This made introducing a new schema version hard. We've seen only two schema changes in the history of Notmuch. This commit introduces database schema version 3; hopefully the last schema version we'll need for a while. With this version, we switch from a single version number to "features": a set of named, independent aspects of the database schema. Features should make backwards compatibility easier. For many things, it should be easy to support databases both with and without a feature, which will allow us to make upgrades optional and will enable "unstable" features that can be developed and tested over time. Features also make forwards compatibility easier. The features recorded in a database include "compatibility flags," which can indicate to an older version of Notmuch when it must support a given feature to open the database for read or for write. This lets us replace the old vague "I don't recognize this version, so something might go wrong, but I promise to try my best" warnings upon opening a database with an unknown version with precise errors. If a database is safe to open for read/write despite unknown features, an older version will know that and issue no message at all. If the database is not safe to open for read/write because of unknown features, an older version will know that, too, and can tell the user exactly which required features it lacks support for.
2014-08-25 19:26:00 +02:00
db->set_metadata ("features", _print_features (local, notmuch->features));
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
db->set_metadata ("version", STRINGIFY (NOTMUCH_DATABASE_VERSION));
lib: Split the database upgrade into two phases for safer operation. The first phase copies data from the old format to the new format without deleting anything. This allows an old notmuch to still use the database if the upgrade process gets interrupted. The second phase performs the deletion (after updating the database version number). If the second phase is interrupted, there will be some unused data in the database, but it shouldn't cause any actual harm.
2010-01-09 20:13:12 +01:00
lib: Implement upgrade to ghost messages feature Somehow this is the first upgrade pass that actually does *any* error checking, so this also adds the bit of necessary infrastructure to handle that.
2014-10-23 14:30:38 +02:00
DONE:
if (status == NOTMUCH_STATUS_SUCCESS)
db->commit_transaction ();
else
db->cancel_transaction ();
lib: Split the database upgrade into two phases for safer operation. The first phase copies data from the old format to the new format without deleting anything. This allows an old notmuch to still use the database if the upgrade process gets interrupted. The second phase performs the deletion (after updating the database version number). If the second phase is interrupted, there will be some unused data in the database, but it shouldn't cause any actual harm.
2010-01-09 20:13:12 +01:00
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
if (timer_is_active) {
/* Now stop the timer. */
timerval.it_interval.tv_sec = 0;
timerval.it_interval.tv_usec = 0;
timerval.it_value.tv_sec = 0;
timerval.it_value.tv_usec = 0;
setitimer (ITIMER_REAL, &timerval, NULL);
/* And disable the signal handler. */
action.sa_handler = SIG_IGN;
sigaction (SIGALRM, &action, NULL);
}
lib: add versions of n_q_count_{message,threads} with status return Although I think it's a pretty bad idea to continue using the old API, this allows both a more gentle transition for clients of the library, and allows us to break one monolithic change into a series
2015-09-27 17:31:57 +02:00
if (query)
notmuch_query_destroy (query);
lib: Database version 3: Introduce fine-grained "features" Previously, our database schema was versioned by a single number. Each database schema change had to occur "atomically" in Notmuch's development history: before some commit, Notmuch used version N, after that commit, it used version N+1. Hence, each new schema version could introduce only one change, the task of developing a schema change fell on a single person, and it all had to happen and be perfect in a single commit series. This made introducing a new schema version hard. We've seen only two schema changes in the history of Notmuch. This commit introduces database schema version 3; hopefully the last schema version we'll need for a while. With this version, we switch from a single version number to "features": a set of named, independent aspects of the database schema. Features should make backwards compatibility easier. For many things, it should be easy to support databases both with and without a feature, which will allow us to make upgrades optional and will enable "unstable" features that can be developed and tested over time. Features also make forwards compatibility easier. The features recorded in a database include "compatibility flags," which can indicate to an older version of Notmuch when it must support a given feature to open the database for read or for write. This lets us replace the old vague "I don't recognize this version, so something might go wrong, but I promise to try my best" warnings upon opening a database with an unknown version with precise errors. If a database is safe to open for read/write despite unknown features, an older version will know that and issue no message at all. If the database is not safe to open for read/write because of unknown features, an older version will know that, too, and can tell the user exactly which required features it lacks support for.
2014-08-25 19:26:00 +02:00
talloc_free (local);
lib: Implement upgrade to ghost messages feature Somehow this is the first upgrade pass that actually does *any* error checking, so this also adds the bit of necessary infrastructure to handle that.
2014-10-23 14:30:38 +02:00
return status;
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
}
lib: Add notmuch_database_{begin,end}_atomic. These operations translate into non-flushed Xapian transactions, allowing arbitrary groups of database operations to be performed atomically.
2011-01-29 17:25:24 +01:00
notmuch_status_t
notmuch_database_begin_atomic (notmuch_database_t *notmuch)
{
lib: Add support for nested atomic sections. notmuch_database_t now keeps a nesting count and we only start a transaction or commit for the outermost atomic section. Introduces a new error, NOTMUCH_STATUS_UNBALANCED_ATOMIC.
2011-06-11 05:35:06 +02:00
if (notmuch->mode == NOTMUCH_DATABASE_MODE_READ_ONLY ||
notmuch->atomic_nesting > 0)
goto DONE;
lib: Add notmuch_database_{begin,end}_atomic. These operations translate into non-flushed Xapian transactions, allowing arbitrary groups of database operations to be performed atomically.
2011-01-29 17:25:24 +01:00
lib/database.cc: fix misleading indentation Found by gcc 6.1.1 -Wmisleading-indentation option (set by -Wall).
2016-09-27 17:06:52 +02:00
if (notmuch_database_needs_upgrade (notmuch))
return NOTMUCH_STATUS_UPGRADE_REQUIRED;
forbid atomic transactions on writable, upgradable databases We can't (but currently do) allow upgrades within transactions because upgrades need their own transactions. We don't want to re-use the current transaction because bailing out of an upgrade would mean loosing all previous changes (because our "atomic" transactions don't commit before hand). This gives us two options: 1. Fail at the beginning of upgrade (tell the user to end the transaction, upgrade, and start over). 2. Don't allow the user to start the transaction. I went with the latter because: 1. There is no reason to call `begin_atomic` unless you intend to to write to the database and anyone intending to write to the database should upgrade it first. 2. This means that nothing inside an atomic transaction can ever fail with NOTMUCH_STATUS_UPGRADE_REQUIRED.
2015-10-25 22:30:39 +01:00
lib: Add notmuch_database_{begin,end}_atomic. These operations translate into non-flushed Xapian transactions, allowing arbitrary groups of database operations to be performed atomically.
2011-01-29 17:25:24 +01:00
try {
(static_cast <Xapian::WritableDatabase *> (notmuch->xapian_db))->begin_transaction (false);
} catch (const Xapian::Error &error) {
lib: replace almost all fprintfs in library with _n_d_log This is not supposed to change any functionality from an end user point of view. Note that it will eliminate some output to stderr. The query debugging output is left as is; it doesn't really fit with the current primitive logging model. The remaining "bad" fprintf will need an internal API change.
2014-12-26 17:25:35 +01:00
_notmuch_database_log (notmuch, "A Xapian exception occurred beginning transaction: %s.\n",
lib: Add notmuch_database_{begin,end}_atomic. These operations translate into non-flushed Xapian transactions, allowing arbitrary groups of database operations to be performed atomically.
2011-01-29 17:25:24 +01:00
error.get_msg().c_str());
notmuch->exception_reported = TRUE;
return NOTMUCH_STATUS_XAPIAN_EXCEPTION;
}
lib: Add support for nested atomic sections. notmuch_database_t now keeps a nesting count and we only start a transaction or commit for the outermost atomic section. Introduces a new error, NOTMUCH_STATUS_UNBALANCED_ATOMIC.
2011-06-11 05:35:06 +02:00
DONE:
notmuch->atomic_nesting++;
lib: Add notmuch_database_{begin,end}_atomic. These operations translate into non-flushed Xapian transactions, allowing arbitrary groups of database operations to be performed atomically.
2011-01-29 17:25:24 +01:00
return NOTMUCH_STATUS_SUCCESS;
}
notmuch_status_t
notmuch_database_end_atomic (notmuch_database_t *notmuch)
{
Xapian::WritableDatabase *db;
lib: Add support for nested atomic sections. notmuch_database_t now keeps a nesting count and we only start a transaction or commit for the outermost atomic section. Introduces a new error, NOTMUCH_STATUS_UNBALANCED_ATOMIC.
2011-06-11 05:35:06 +02:00
if (notmuch->atomic_nesting == 0)
return NOTMUCH_STATUS_UNBALANCED_ATOMIC;
if (notmuch->mode == NOTMUCH_DATABASE_MODE_READ_ONLY ||
notmuch->atomic_nesting > 1)
goto DONE;
lib: Add notmuch_database_{begin,end}_atomic. These operations translate into non-flushed Xapian transactions, allowing arbitrary groups of database operations to be performed atomically.
2011-01-29 17:25:24 +01:00
db = static_cast <Xapian::WritableDatabase *> (notmuch->xapian_db);
try {
db->commit_transaction ();
/* This is a hack for testing. Xapian never flushes on a
* non-flushed commit, even if the flush threshold is 1.
* However, we rely on flushing to test atomicity. */
const char *thresh = getenv ("XAPIAN_FLUSH_THRESHOLD");
if (thresh && atoi (thresh) == 1)
lib: replace deprecated xapian call 'flush()' with 'commit()' This will make notmuch incompatible with Xapian before 1.1.0, which is more than 6 years old this point.
2016-10-05 13:34:33 +02:00
db->commit ();
lib: Add notmuch_database_{begin,end}_atomic. These operations translate into non-flushed Xapian transactions, allowing arbitrary groups of database operations to be performed atomically.
2011-01-29 17:25:24 +01:00
} catch (const Xapian::Error &error) {
lib: replace almost all fprintfs in library with _n_d_log This is not supposed to change any functionality from an end user point of view. Note that it will eliminate some output to stderr. The query debugging output is left as is; it doesn't really fit with the current primitive logging model. The remaining "bad" fprintf will need an internal API change.
2014-12-26 17:25:35 +01:00
_notmuch_database_log (notmuch, "A Xapian exception occurred committing transaction: %s.\n",
lib: Add notmuch_database_{begin,end}_atomic. These operations translate into non-flushed Xapian transactions, allowing arbitrary groups of database operations to be performed atomically.
2011-01-29 17:25:24 +01:00
error.get_msg().c_str());
notmuch->exception_reported = TRUE;
return NOTMUCH_STATUS_XAPIAN_EXCEPTION;
}
lib: Add support for nested atomic sections. notmuch_database_t now keeps a nesting count and we only start a transaction or commit for the outermost atomic section. Introduces a new error, NOTMUCH_STATUS_UNBALANCED_ATOMIC.
2011-06-11 05:35:06 +02:00
lib: Add per-message last modification tracking This adds a new document value that stores the revision of the last modification to message metadata, where the revision number increases monotonically with each database commit. An alternative would be to store the wall-clock time of the last modification of each message. In principle this is simpler and has the advantage that any process can determine the current timestamp without support from libnotmuch. However, even assuming a computer's clock never goes backward and ignoring clock skew in networked environments, this has a fatal flaw. Xapian uses (optimistic) snapshot isolation, which means reads can be concurrent with writes. Given this, consider the following time line with a write and two read transactions: write |-X-A--------------| read 1 |---B---| read 2 |---| The write transaction modifies message X and records the wall-clock time of the modification at A. The writer hangs around for a while and later commits its change. Read 1 is concurrent with the write, so it doesn't see the change to X. It does some query and records the wall-clock time of its results at B. Transaction read 2 later starts after the write commits and queries for changes since wall-clock time B (say the reads are performing an incremental backup). Even though read 1 could not see the change to X, read 2 is told (correctly) that X has not changed since B, the time of the last read. In fact, X changed before wall-clock time A, but the change was not visible until *after* wall-clock time B, so read 2 misses the change to X. This is tricky to solve in full-blown snapshot isolation, but because Xapian serializes writes, we can use a simple, monotonically increasing database revision number. Furthermore, maintaining this revision number requires no more IO than a wall-clock time solution because Xapian already maintains statistics on the upper (and lower) bound of each value stream.
2014-10-13 08:20:01 +02:00
if (notmuch->atomic_dirty) {
++notmuch->revision;
notmuch->atomic_dirty = FALSE;
}
lib: Add support for nested atomic sections. notmuch_database_t now keeps a nesting count and we only start a transaction or commit for the outermost atomic section. Introduces a new error, NOTMUCH_STATUS_UNBALANCED_ATOMIC.
2011-06-11 05:35:06 +02:00
DONE:
notmuch->atomic_nesting--;
lib: Add notmuch_database_{begin,end}_atomic. These operations translate into non-flushed Xapian transactions, allowing arbitrary groups of database operations to be performed atomically.
2011-01-29 17:25:24 +01:00
return NOTMUCH_STATUS_SUCCESS;
}
lib: API to retrieve database revision and UUID This exposes the committed database revision to library users along with a UUID that can be used to detect when revision numbers are no longer comparable (e.g., because the database has been replaced).
2014-10-13 08:20:02 +02:00
unsigned long
notmuch_database_get_revision (notmuch_database_t *notmuch,
const char **uuid)
{
if (uuid)
*uuid = notmuch->uuid;
return notmuch->revision;
}
lib: Rename set/get_timestamp to set/get_directory_mtime. I've been suitably scolded by Keith for doing a premature generalization that ended up just making the documentation more convoluted. Fix that.
2009-12-17 23:33:34 +01:00
/* We allow the user to use arbitrarily long paths for directories. But
* we have a term-length limit. So if we exceed that, we'll use the
* SHA-1 of the path for the database term.
Add notmuch_database_set_timestamp and notmuch_database_get_timestamp These will be very helpful to implement an efficient "notmuch new" command which imports new mail messages that have appeared.
2009-10-23 23:31:01 +02:00
*
lib: Rename set/get_timestamp to set/get_directory_mtime. I've been suitably scolded by Keith for doing a premature generalization that ended up just making the documentation more convoluted. Fix that.
2009-12-17 23:33:34 +01:00
* Note: This function may return the original value of 'path'. If it
* does not, then the caller is responsible to free() the returned
* value.
Add notmuch_database_set_timestamp and notmuch_database_get_timestamp These will be very helpful to implement an efficient "notmuch new" command which imports new mail messages that have appeared.
2009-10-23 23:31:01 +02:00
*/
lib: Implement new notmuch_directory_t API. This new directory ojbect provides all the infrastructure needed to detect when files or directories are deleted or renamed. There's still code needed on top of this (within "notmuch new") to actually do that detection.
2010-01-05 22:29:23 +01:00
const char *
_notmuch_database_get_directory_db_path (const char *path)
Add notmuch_database_set_timestamp and notmuch_database_get_timestamp These will be very helpful to implement an efficient "notmuch new" command which imports new mail messages that have appeared.
2009-10-23 23:31:01 +02:00
{
lib: Rename set/get_timestamp to set/get_directory_mtime. I've been suitably scolded by Keith for doing a premature generalization that ended up just making the documentation more convoluted. Fix that.
2009-12-17 23:33:34 +01:00
int term_len = strlen (_find_prefix ("directory")) + strlen (path);
Fix timestamp generation to avoid overflowing the term limit The previous code was only correct as long as the timestamp prefix was only a single character. But with the recent change to a multi-character prefix, this broke. So fix it now.
2009-10-25 06:10:03 +01:00
if (term_len > NOTMUCH_TERM_MAX)
lib: Start all function names in notmuch-private.h with As noted in devel/STYLE, every private library function should start with _notmuch. This patch corrects function naming that did not adhere to this style in lib/notmuch-private.h. In particular, the old function names that now begin with _notmuch are notmuch_sha1_of_file notmuch_sha1_of_string notmuch_message_file_close notmuch_message_file_get_header notmuch_message_file_open notmuch_message_get_author notmuch_message_set_author Signed-off-by: Charles Celerier <cceleri@cs.stanford.edu>
2014-05-13 11:44:05 +02:00
return _notmuch_sha1_of_string (path);
Fix timestamp generation to avoid overflowing the term limit The previous code was only correct as long as the timestamp prefix was only a single character. But with the recent change to a multi-character prefix, this broke. So fix it now.
2009-10-25 06:10:03 +01:00
else
lib: Rename set/get_timestamp to set/get_directory_mtime. I've been suitably scolded by Keith for doing a premature generalization that ended up just making the documentation more convoluted. Fix that.
2009-12-17 23:33:34 +01:00
return path;
Add notmuch_database_set_timestamp and notmuch_database_get_timestamp These will be very helpful to implement an efficient "notmuch new" command which imports new mail messages that have appeared.
2009-10-23 23:31:01 +02:00
}
database: Split _find_parent_id into _split_path and _find_directory_id Some pending commits want the _split_path functionality separate from mapping a directory to a document ID. The split_path function now returns the basename as well as the directory name.
2009-12-21 17:14:52 +01:00
/* Given a path, split it into two parts: the directory part is all
* components except for the last, and the basename is that last
* component. Getting the return-value for either part is optional
* (the caller can pass NULL).
database: Export _notmuch_database_find_parent_id for internal use. We'll soon have mail documents referring to their parent directory's directory documents, so we'll need access to _find_parent_id in files such as message.cc.
2009-12-20 00:11:55 +01:00
*
* The original 'path' can represent either a regular file or a
database: Split _find_parent_id into _split_path and _find_directory_id Some pending commits want the _split_path functionality separate from mapping a directory to a document ID. The split_path function now returns the basename as well as the directory name.
2009-12-21 17:14:52 +01:00
* directory---the splitting will be carried out in the same way in
* either case. Trailing slashes on 'path' will be ignored, and any
database: Export _notmuch_database_find_parent_id for internal use. We'll soon have mail documents referring to their parent directory's directory documents, so we'll need access to _find_parent_id in files such as message.cc.
2009-12-20 00:11:55 +01:00
* cases of multiple '/' characters appearing in series will be
* treated as a single '/'.
*
database: Split _find_parent_id into _split_path and _find_directory_id Some pending commits want the _split_path functionality separate from mapping a directory to a document ID. The split_path function now returns the basename as well as the directory name.
2009-12-21 17:14:52 +01:00
* Allocation (if any) will have 'ctx' as the talloc owner. But
* pointers will be returned within the original path string whenever
* possible.
database: Export _notmuch_database_find_parent_id for internal use. We'll soon have mail documents referring to their parent directory's directory documents, so we'll need access to _find_parent_id in files such as message.cc.
2009-12-20 00:11:55 +01:00
*
database: Split _find_parent_id into _split_path and _find_directory_id Some pending commits want the _split_path functionality separate from mapping a directory to a document ID. The split_path function now returns the basename as well as the directory name.
2009-12-21 17:14:52 +01:00
* Note: If 'path' is non-empty and contains no non-trailing slash,
* (that is, consists of a filename with no parent directory), then
* the directory returned will be an empty string. However, if 'path'
* is an empty string, then both directory and basename will be
* returned as NULL.
database: Export _notmuch_database_find_parent_id for internal use. We'll soon have mail documents referring to their parent directory's directory documents, so we'll need access to _find_parent_id in files such as message.cc.
2009-12-20 00:11:55 +01:00
*/
database: Store the parent ID for each directory document. Storing the document ID of the parent of each directory document will allow us to find all child-directory documents for a given directory document. We will need this in order to detect directories that have been removed from the mail store, (though we aren't yet doing this).
2009-12-19 22:20:26 +01:00
notmuch_status_t
database: Split _find_parent_id into _split_path and _find_directory_id Some pending commits want the _split_path functionality separate from mapping a directory to a document ID. The split_path function now returns the basename as well as the directory name.
2009-12-21 17:14:52 +01:00
_notmuch_database_split_path (void *ctx,
const char *path,
const char **directory,
const char **basename)
database: Store the parent ID for each directory document. Storing the document ID of the parent of each directory document will allow us to find all child-directory documents for a given directory document. We will need this in order to detect directories that have been removed from the mail store, (though we aren't yet doing this).
2009-12-19 22:20:26 +01:00
{
database: Split _find_parent_id into _split_path and _find_directory_id Some pending commits want the _split_path functionality separate from mapping a directory to a document ID. The split_path function now returns the basename as well as the directory name.
2009-12-21 17:14:52 +01:00
const char *slash;
database: Store the parent ID for each directory document. Storing the document ID of the parent of each directory document will allow us to find all child-directory documents for a given directory document. We will need this in order to detect directories that have been removed from the mail store, (though we aren't yet doing this).
2009-12-19 22:20:26 +01:00
if (path == NULL || *path == '\0') {
database: Split _find_parent_id into _split_path and _find_directory_id Some pending commits want the _split_path functionality separate from mapping a directory to a document ID. The split_path function now returns the basename as well as the directory name.
2009-12-21 17:14:52 +01:00
if (directory)
*directory = NULL;
if (basename)
*basename = NULL;
database: Store the parent ID for each directory document. Storing the document ID of the parent of each directory document will allow us to find all child-directory documents for a given directory document. We will need this in order to detect directories that have been removed from the mail store, (though we aren't yet doing this).
2009-12-19 22:20:26 +01:00
return NOTMUCH_STATUS_SUCCESS;
}
/* Find the last slash (not counting a trailing slash), if any. */
slash = path + strlen (path) - 1;
/* First, skip trailing slashes. */
lib: clean up _notmuch_database_split_path Make the logic it a bit easier to read. No functional changes.
2016-04-10 21:43:23 +02:00
while (slash != path && *slash == '/')
database: Store the parent ID for each directory document. Storing the document ID of the parent of each directory document will allow us to find all child-directory documents for a given directory document. We will need this in order to detect directories that have been removed from the mail store, (though we aren't yet doing this).
2009-12-19 22:20:26 +01:00
--slash;
/* Then, find a slash. */
lib: clean up _notmuch_database_split_path Make the logic it a bit easier to read. No functional changes.
2016-04-10 21:43:23 +02:00
while (slash != path && *slash != '/') {
database: Split _find_parent_id into _split_path and _find_directory_id Some pending commits want the _split_path functionality separate from mapping a directory to a document ID. The split_path function now returns the basename as well as the directory name.
2009-12-21 17:14:52 +01:00
if (basename)
*basename = slash;
database: Store the parent ID for each directory document. Storing the document ID of the parent of each directory document will allow us to find all child-directory documents for a given directory document. We will need this in order to detect directories that have been removed from the mail store, (though we aren't yet doing this).
2009-12-19 22:20:26 +01:00
--slash;
}
/* Finally, skip multiple slashes. */
lib: clean up _notmuch_database_split_path Make the logic it a bit easier to read. No functional changes.
2016-04-10 21:43:23 +02:00
while (slash != path && *(slash - 1) == '/')
database: Store the parent ID for each directory document. Storing the document ID of the parent of each directory document will allow us to find all child-directory documents for a given directory document. We will need this in order to detect directories that have been removed from the mail store, (though we aren't yet doing this).
2009-12-19 22:20:26 +01:00
--slash;
database: Split _find_parent_id into _split_path and _find_directory_id Some pending commits want the _split_path functionality separate from mapping a directory to a document ID. The split_path function now returns the basename as well as the directory name.
2009-12-21 17:14:52 +01:00
if (slash == path) {
if (directory)
*directory = talloc_strdup (ctx, "");
if (basename)
*basename = path;
} else {
if (directory)
lib: fix handling of one character long directory names at top level The code to skip multiple slashes in _notmuch_database_split_path() skips back one character too much. This is compensated by a +1 in the length parameter to the strndup() call. Mostly this works fine, but if the path is to a file under a top level directory with one character long name, the directory part is mistaken to be part of the file name (slash == path in code). The returned directory name will be the empty string and the basename will be the full path, breaking the indexing logic in notmuch new. Fix the multiple slash skipping to keep the slash variable pointing at the last slash, and adjust strndup() accordingly. The bug was introduced in commit e890b0cf4011fd9fd77ebd87343379e4a778888b Author: Carl Worth <cworth@cworth.org> Date: Sat Dec 19 13:20:26 2009 -0800 database: Store the parent ID for each directory document. just a little over two months after the initial commit in the Notmuch code history, making this the longest living bug in Notmuch to date.
2016-04-10 21:43:22 +02:00
*directory = talloc_strndup (ctx, path, slash - path);
database: Split _find_parent_id into _split_path and _find_directory_id Some pending commits want the _split_path functionality separate from mapping a directory to a document ID. The split_path function now returns the basename as well as the directory name.
2009-12-21 17:14:52 +01:00
}
database: Store the parent ID for each directory document. Storing the document ID of the parent of each directory document will allow us to find all child-directory documents for a given directory document. We will need this in order to detect directories that have been removed from the mail store, (though we aren't yet doing this).
2009-12-19 22:20:26 +01:00
database: Split _find_parent_id into _split_path and _find_directory_id Some pending commits want the _split_path functionality separate from mapping a directory to a document ID. The split_path function now returns the basename as well as the directory name.
2009-12-21 17:14:52 +01:00
return NOTMUCH_STATUS_SUCCESS;
}
lib: Perform the same transformation to _notmuch_database_find_directory_id Now _notmuch_database_find_directory_id takes a flags argument, which it passes through to _notmuch_directory_create and can indicate if the directory does not exist. Again, callers have been updated, but retain their original behavior.
2012-05-18 06:13:35 +02:00
/* Find the document ID of the specified directory.
*
* If (flags & NOTMUCH_FIND_CREATE), a new directory document will be
* created if one does not exist for 'path'. Otherwise, if the
* directory document does not exist, this sets *directory_id to
* ((unsigned int)-1) and returns NOTMUCH_STATUS_SUCCESS.
*/
database: Split _find_parent_id into _split_path and _find_directory_id Some pending commits want the _split_path functionality separate from mapping a directory to a document ID. The split_path function now returns the basename as well as the directory name.
2009-12-21 17:14:52 +01:00
notmuch_status_t
_notmuch_database_find_directory_id (notmuch_database_t *notmuch,
const char *path,
lib: Perform the same transformation to _notmuch_database_find_directory_id Now _notmuch_database_find_directory_id takes a flags argument, which it passes through to _notmuch_directory_create and can indicate if the directory does not exist. Again, callers have been updated, but retain their original behavior.
2012-05-18 06:13:35 +02:00
notmuch_find_flags_t flags,
database: Split _find_parent_id into _split_path and _find_directory_id Some pending commits want the _split_path functionality separate from mapping a directory to a document ID. The split_path function now returns the basename as well as the directory name.
2009-12-21 17:14:52 +01:00
unsigned int *directory_id)
{
lib: Implement new notmuch_directory_t API. This new directory ojbect provides all the infrastructure needed to detect when files or directories are deleted or renamed. There's still code needed on top of this (within "notmuch new") to actually do that detection.
2010-01-05 22:29:23 +01:00
notmuch_directory_t *directory;
notmuch_status_t status;
database: Split _find_parent_id into _split_path and _find_directory_id Some pending commits want the _split_path functionality separate from mapping a directory to a document ID. The split_path function now returns the basename as well as the directory name.
2009-12-21 17:14:52 +01:00
if (path == NULL) {
*directory_id = 0;
return NOTMUCH_STATUS_SUCCESS;
}
lib: Perform the same transformation to _notmuch_database_find_directory_id Now _notmuch_database_find_directory_id takes a flags argument, which it passes through to _notmuch_directory_create and can indicate if the directory does not exist. Again, callers have been updated, but retain their original behavior.
2012-05-18 06:13:35 +02:00
directory = _notmuch_directory_create (notmuch, path, flags, &status);
if (status || !directory) {
lib: Implement new notmuch_directory_t API. This new directory ojbect provides all the infrastructure needed to detect when files or directories are deleted or renamed. There's still code needed on top of this (within "notmuch new") to actually do that detection.
2010-01-05 22:29:23 +01:00
*directory_id = -1;
return status;
database: Store the parent ID for each directory document. Storing the document ID of the parent of each directory document will allow us to find all child-directory documents for a given directory document. We will need this in order to detect directories that have been removed from the mail store, (though we aren't yet doing this).
2009-12-19 22:20:26 +01:00
}
lib: Implement new notmuch_directory_t API. This new directory ojbect provides all the infrastructure needed to detect when files or directories are deleted or renamed. There's still code needed on top of this (within "notmuch new") to actually do that detection.
2010-01-05 22:29:23 +01:00
*directory_id = _notmuch_directory_get_document_id (directory);
database: Store the parent ID for each directory document. Storing the document ID of the parent of each directory document will allow us to find all child-directory documents for a given directory document. We will need this in order to detect directories that have been removed from the mail store, (though we aren't yet doing this).
2009-12-19 22:20:26 +01:00
lib: Implement new notmuch_directory_t API. This new directory ojbect provides all the infrastructure needed to detect when files or directories are deleted or renamed. There's still code needed on top of this (within "notmuch new") to actually do that detection.
2010-01-05 22:29:23 +01:00
notmuch_directory_destroy (directory);
database: Export _notmuch_database_find_parent_id for internal use. We'll soon have mail documents referring to their parent directory's directory documents, so we'll need access to _find_parent_id in files such as message.cc.
2009-12-20 00:11:55 +01:00
lib: Implement new notmuch_directory_t API. This new directory ojbect provides all the infrastructure needed to detect when files or directories are deleted or renamed. There's still code needed on top of this (within "notmuch new") to actually do that detection.
2010-01-05 22:29:23 +01:00
return NOTMUCH_STATUS_SUCCESS;
database: Store the parent ID for each directory document. Storing the document ID of the parent of each directory document will allow us to find all child-directory documents for a given directory document. We will need this in order to detect directories that have been removed from the mail store, (though we aren't yet doing this).
2009-12-19 22:20:26 +01:00
}
database: Store mail filename as a new 'direntry' term, not as 'data'. Instead of storing the complete message filename in the data portion of a mail document we now store a 'direntry' term that contains the document ID of a directory document and also the basename of the message filename within that directory. This will allow us to easily store multple filenames for a single message, and will also allow us to find mail documents for files that previously existed in a directory but that have since been deleted.
2009-12-21 17:23:26 +01:00
const char *
_notmuch_database_get_directory_path (void *ctx,
notmuch_database_t *notmuch,
unsigned int doc_id)
{
Xapian::Document document;
document = find_document_for_doc_id (notmuch, doc_id);
return talloc_strdup (ctx, document.get_data ().c_str ());
}
database: Abstract _filename_to_direntry from _add_message The code to map a filename to a direntry is something that we're going to want in a future _remove_message function, so put it in a new function _notmuch_database_filename_to_direntry .
2009-12-22 00:09:56 +01:00
/* Given a legal 'filename' for the database, (either relative to
* database path or absolute with initial components identical to
* database path), return a new string (with 'ctx' as the talloc
* owner) suitable for use as a direntry term value.
lib: Clarify internal documentation of _notmuch_database_filename_to_direntry The original wording made it sound like this function was just doing some string manipulation. But this function actually creates new directory documents as a side effect. So make that explicit in its documentation.
2010-01-07 18:31:58 +01:00
*
lib: Perform the same transformation to _notmuch_database_filename_to_direntry Now _notmuch_database_filename_to_direntry takes a flags argument and can indicate if the necessary directory documents do not exist. Again, callers have been updated, but retain their original behavior.
2012-05-18 06:13:36 +02:00
* If (flags & NOTMUCH_FIND_CREATE), the necessary directory documents
* will be created in the database as needed. Otherwise, if the
* necessary directory documents do not exist, this sets
* *direntry to NULL and returns NOTMUCH_STATUS_SUCCESS.
database: Abstract _filename_to_direntry from _add_message The code to map a filename to a direntry is something that we're going to want in a future _remove_message function, so put it in a new function _notmuch_database_filename_to_direntry .
2009-12-22 00:09:56 +01:00
*/
notmuch_status_t
_notmuch_database_filename_to_direntry (void *ctx,
notmuch_database_t *notmuch,
const char *filename,
lib: Perform the same transformation to _notmuch_database_filename_to_direntry Now _notmuch_database_filename_to_direntry takes a flags argument and can indicate if the necessary directory documents do not exist. Again, callers have been updated, but retain their original behavior.
2012-05-18 06:13:36 +02:00
notmuch_find_flags_t flags,
database: Abstract _filename_to_direntry from _add_message The code to map a filename to a direntry is something that we're going to want in a future _remove_message function, so put it in a new function _notmuch_database_filename_to_direntry .
2009-12-22 00:09:56 +01:00
char **direntry)
{
const char *relative, *directory, *basename;
Xapian::docid directory_id;
notmuch_status_t status;
relative = _notmuch_database_relative_path (notmuch, filename);
status = _notmuch_database_split_path (ctx, relative,
&directory, &basename);
if (status)
return status;
lib: Perform the same transformation to _notmuch_database_filename_to_direntry Now _notmuch_database_filename_to_direntry takes a flags argument and can indicate if the necessary directory documents do not exist. Again, callers have been updated, but retain their original behavior.
2012-05-18 06:13:36 +02:00
status = _notmuch_database_find_directory_id (notmuch, directory, flags,
database: Abstract _filename_to_direntry from _add_message The code to map a filename to a direntry is something that we're going to want in a future _remove_message function, so put it in a new function _notmuch_database_filename_to_direntry .
2009-12-22 00:09:56 +01:00
&directory_id);
lib: Perform the same transformation to _notmuch_database_filename_to_direntry Now _notmuch_database_filename_to_direntry takes a flags argument and can indicate if the necessary directory documents do not exist. Again, callers have been updated, but retain their original behavior.
2012-05-18 06:13:36 +02:00
if (status || directory_id == (unsigned int)-1) {
*direntry = NULL;
database: Abstract _filename_to_direntry from _add_message The code to map a filename to a direntry is something that we're going to want in a future _remove_message function, so put it in a new function _notmuch_database_filename_to_direntry .
2009-12-22 00:09:56 +01:00
return status;
lib: Perform the same transformation to _notmuch_database_filename_to_direntry Now _notmuch_database_filename_to_direntry takes a flags argument and can indicate if the necessary directory documents do not exist. Again, callers have been updated, but retain their original behavior.
2012-05-18 06:13:36 +02:00
}
database: Abstract _filename_to_direntry from _add_message The code to map a filename to a direntry is something that we're going to want in a future _remove_message function, so put it in a new function _notmuch_database_filename_to_direntry .
2009-12-22 00:09:56 +01:00
*direntry = talloc_asprintf (ctx, "%u:%s", directory_id, basename);
return NOTMUCH_STATUS_SUCCESS;
}
lib: Abstract the extraction of a relative path from set_filename We'll soon be having multiple entry points that accept a filename path, so we want common code for getting a relative path from a potentially absolute path.
2009-12-19 21:32:11 +01:00
/* Given a legal 'path' for the database, return the relative path.
*
fix sum moar typos [comments in source code] Various typo fixes in comments within the source code. Signed-off-by: Pieter Praet <pieter@praet.org> Edited-by: Carl Worth <cworth@cworth.org> Restricted to just source-code comments, (and fixed fix of "descriptios" to "descriptors" rather than "descriptions").
2011-06-20 22:14:21 +02:00
* The return value will be a pointer to the original path contents,
lib: Abstract the extraction of a relative path from set_filename We'll soon be having multiple entry points that accept a filename path, so we want common code for getting a relative path from a potentially absolute path.
2009-12-19 21:32:11 +01:00
* and will be either the original string (if 'path' was relative) or
* a portion of the string (if path was absolute and begins with the
* database path).
*/
const char *
_notmuch_database_relative_path (notmuch_database_t *notmuch,
const char *path)
{
const char *db_path, *relative;
unsigned int db_path_len;
db_path = notmuch_database_get_path (notmuch);
db_path_len = strlen (db_path);
relative = path;
if (*relative == '/') {
while (*relative == '/' && *(relative+1) == '/')
relative++;
if (strncmp (relative, db_path, db_path_len) == 0)
{
relative += db_path_len;
while (*relative == '/')
relative++;
}
}
return relative;
}
lib/cli: Make notmuch_database_get_directory return a status code Previously, notmuch_database_get_directory had no way to indicate how it had failed. This changes its prototype to return a status code and set an out-argument to the retrieved directory, like similar functions in the library API. This does *not* change its currently broken behavior of creating directory objects when they don't exist, but it does document it and paves the way for fixing this. Also, it can now check for a read-only database and return NOTMUCH_STATUS_READ_ONLY_DATABASE instead of crashing. In the interest of atomicity, this also updates calls from the CLI so that notmuch still compiles.
2012-05-14 01:36:09 +02:00
notmuch_status_t
lib: Implement new notmuch_directory_t API. This new directory ojbect provides all the infrastructure needed to detect when files or directories are deleted or renamed. There's still code needed on top of this (within "notmuch new") to actually do that detection.
2010-01-05 22:29:23 +01:00
notmuch_database_get_directory (notmuch_database_t *notmuch,
lib/cli: Make notmuch_database_get_directory return a status code Previously, notmuch_database_get_directory had no way to indicate how it had failed. This changes its prototype to return a status code and set an out-argument to the retrieved directory, like similar functions in the library API. This does *not* change its currently broken behavior of creating directory objects when they don't exist, but it does document it and paves the way for fixing this. Also, it can now check for a read-only database and return NOTMUCH_STATUS_READ_ONLY_DATABASE instead of crashing. In the interest of atomicity, this also updates calls from the CLI so that notmuch still compiles.
2012-05-14 01:36:09 +02:00
const char *path,
notmuch_directory_t **directory)
Add notmuch_database_set_timestamp and notmuch_database_get_timestamp These will be very helpful to implement an efficient "notmuch new" command which imports new mail messages that have appeared.
2009-10-23 23:31:01 +02:00
{
lib: Implement new notmuch_directory_t API. This new directory ojbect provides all the infrastructure needed to detect when files or directories are deleted or renamed. There's still code needed on top of this (within "notmuch new") to actually do that detection.
2010-01-05 22:29:23 +01:00
notmuch_status_t status;
Add notmuch_database_set_timestamp and notmuch_database_get_timestamp These will be very helpful to implement an efficient "notmuch new" command which imports new mail messages that have appeared.
2009-10-23 23:31:01 +02:00
lib/cli: Make notmuch_database_get_directory return a status code Previously, notmuch_database_get_directory had no way to indicate how it had failed. This changes its prototype to return a status code and set an out-argument to the retrieved directory, like similar functions in the library API. This does *not* change its currently broken behavior of creating directory objects when they don't exist, but it does document it and paves the way for fixing this. Also, it can now check for a read-only database and return NOTMUCH_STATUS_READ_ONLY_DATABASE instead of crashing. In the interest of atomicity, this also updates calls from the CLI so that notmuch still compiles.
2012-05-14 01:36:09 +02:00
if (directory == NULL)
return NOTMUCH_STATUS_NULL_POINTER;
*directory = NULL;
lib: Audit all notmuch_database call for Xapian exception handling. Our current approach is for top-level entry poitns in the library to have try/catch blocks that catch any Xapian exception and print a message. Add a few missing blocks and fix up the documentation.
2010-04-24 16:22:34 +02:00
try {
lib: Make notmuch_database_get_directory return NULL if the directory is not found Using the new support from _notmuch_directory_create, this makes notmuch_database_get_directory a read-only operation that simply returns the directory object if it exists or NULL otherwise. This also means that notmuch_database_get_directory can work on read-only databases. This change breaks the directory mtime workaround in notmuch-new.c by fixing the exact issue it was working around. This permits mtime update races to prevent scans of changed directories, which non-deterministically breaks a few tests. The next patch fixes this.
2012-05-18 06:13:37 +02:00
*directory = _notmuch_directory_create (notmuch, path,
NOTMUCH_FIND_LOOKUP, &status);
lib: Audit all notmuch_database call for Xapian exception handling. Our current approach is for top-level entry poitns in the library to have try/catch blocks that catch any Xapian exception and print a message. Add a few missing blocks and fix up the documentation.
2010-04-24 16:22:34 +02:00
} catch (const Xapian::Error &error) {
lib: replace almost all fprintfs in library with _n_d_log This is not supposed to change any functionality from an end user point of view. Note that it will eliminate some output to stderr. The query debugging output is left as is; it doesn't really fit with the current primitive logging model. The remaining "bad" fprintf will need an internal API change.
2014-12-26 17:25:35 +01:00
_notmuch_database_log (notmuch, "A Xapian exception occurred getting directory: %s.\n",
lib: Audit all notmuch_database call for Xapian exception handling. Our current approach is for top-level entry poitns in the library to have try/catch blocks that catch any Xapian exception and print a message. Add a few missing blocks and fix up the documentation.
2010-04-24 16:22:34 +02:00
error.get_msg().c_str());
notmuch->exception_reported = TRUE;
lib/cli: Make notmuch_database_get_directory return a status code Previously, notmuch_database_get_directory had no way to indicate how it had failed. This changes its prototype to return a status code and set an out-argument to the retrieved directory, like similar functions in the library API. This does *not* change its currently broken behavior of creating directory objects when they don't exist, but it does document it and paves the way for fixing this. Also, it can now check for a read-only database and return NOTMUCH_STATUS_READ_ONLY_DATABASE instead of crashing. In the interest of atomicity, this also updates calls from the CLI so that notmuch still compiles.
2012-05-14 01:36:09 +02:00
status = NOTMUCH_STATUS_XAPIAN_EXCEPTION;
lib: Audit all notmuch_database call for Xapian exception handling. Our current approach is for top-level entry poitns in the library to have try/catch blocks that catch any Xapian exception and print a message. Add a few missing blocks and fix up the documentation.
2010-04-24 16:22:34 +02:00
}
lib/cli: Make notmuch_database_get_directory return a status code Previously, notmuch_database_get_directory had no way to indicate how it had failed. This changes its prototype to return a status code and set an out-argument to the retrieved directory, like similar functions in the library API. This does *not* change its currently broken behavior of creating directory objects when they don't exist, but it does document it and paves the way for fixing this. Also, it can now check for a read-only database and return NOTMUCH_STATUS_READ_ONLY_DATABASE instead of crashing. In the interest of atomicity, this also updates calls from the CLI so that notmuch still compiles.
2012-05-14 01:36:09 +02:00
return status;
Add notmuch_database_set_timestamp and notmuch_database_get_timestamp These will be very helpful to implement an efficient "notmuch new" command which imports new mail messages that have appeared.
2009-10-23 23:31:01 +02:00
}
Avoid database corruption by not adding partially-constructed mail documents. Previously we were using Xapian's add_document to allocate document ID values for notmuch_message_t objects. This had the drawback of adding a partially constructed mail document to the database. If notmuch was subsequently interrupted before fully populating this document, then later runs would be quite confused when seeing the partial documents. There are reports from the wild of people hitting internal errors of the form "Message ... has no thread ID" for example, (which is currently an unrecoverable error). We fix this by manually allocating document IDs without adding documents. With this change, we never call Xapian's add_document method, but only replace_document with either the current document ID of a message or a new one that we have allocated.
2010-06-04 19:16:53 +02:00
/* Allocate a document ID that satisfies the following criteria:
*
* 1. The ID does not exist for any document in the Xapian database
*
* 2. The ID was not previously returned from this function
*
* 3. The ID is the smallest integer satisfying (1) and (2)
*
* This function will trigger an internal error if these constraints
* cannot all be satisfied, (that is, the pool of available document
* IDs has been exhausted).
*/
unsigned int
_notmuch_database_generate_doc_id (notmuch_database_t *notmuch)
{
assert (notmuch->last_doc_id >= notmuch->xapian_db->get_lastdocid ());
notmuch->last_doc_id++;
if (notmuch->last_doc_id == 0)
compact: tidy formatting Notmuch compact code whitespace changes to match devel/STYLE.
2013-11-13 18:02:43 +01:00
INTERNAL_ERROR ("Xapian document IDs are exhausted.\n");
Avoid database corruption by not adding partially-constructed mail documents. Previously we were using Xapian's add_document to allocate document ID values for notmuch_message_t objects. This had the drawback of adding a partially constructed mail document to the database. If notmuch was subsequently interrupted before fully populating this document, then later runs would be quite confused when seeing the partial documents. There are reports from the wild of people hitting internal errors of the form "Message ... has no thread ID" for example, (which is currently an unrecoverable error). We fix this by manually allocating document IDs without adding documents. With this change, we never call Xapian's add_document method, but only replace_document with either the current document ID of a message or a new one that we have allocated.
2010-06-04 19:16:53 +02:00
return notmuch->last_doc_id;
}
Store thread ids for messages that we haven't seen yet This allows us to thread messages even when we receive them out of order, or never receive the root. The thread ids for messages that aren't present but are referred to are stored as metadata in the database and then retrieved if we ever get that message. When determining the thread id for a message we also check for this metadata so that we can thread descendants of a message together before we receive it. Edited by Carl Worth <cworth@cworth.org>: Split this portion of the commit from the earlier-applied portion adding test cases.
2010-03-13 22:27:57 +01:00
static const char *
_notmuch_database_generate_thread_id (notmuch_database_t *notmuch)
{
/* 16 bytes (+ terminator) for hexadecimal representation of
* a 64-bit integer. */
static char thread_id[17];
Xapian::WritableDatabase *db;
db = static_cast <Xapian::WritableDatabase *> (notmuch->xapian_db);
notmuch->last_thread_id++;
sprintf (thread_id, "%016" PRIx64, notmuch->last_thread_id);
db->set_metadata ("last_thread_id", thread_id);
return thread_id;
}
static char *
_get_metadata_thread_id_key (void *ctx, const char *message_id)
{
Add support (and tests) for messages with really long message IDs. Scott Henson reported an internal error that occurred when he tried to add a message that referenced another message with a message ID well over 300 characters in length. The bug here was running into a Xapian limit for the length of metadata key names, (which is even more restrictive than the Xapian limit for the length of terms). We fix this by noticing long message ID values and instead using a message ID of the form "notmuch-sha1-<sha1_sum_of_message_id>". That is, we use SHA1 to generate a compressed, (but still unique), version of the message ID. We add support to the test suite to exercise this fix. The tests add a message referencing the long message ID, then add the message with the long message ID, then finally add another message referencing the long ID. Each of these tests exercise different code paths where the special handling is implemented. A final test ensures that all three messages are stitched together into a single thread---guaranteeing that the three code paths all act consistently.
2010-06-04 21:39:36 +02:00
if (strlen (message_id) > NOTMUCH_MESSAGE_ID_MAX)
lib: Move message ID compression to _notmuch_message_create_for_message_id Previously, this was performed by notmuch_database_add_message. This happens to be the only caller currently (which is why this was safe), but we're about to introduce more callers, and it makes more sense to put responsibility for ID compression in the lower-level function rather than requiring each caller to handle it.
2014-10-07 01:17:07 +02:00
message_id = _notmuch_message_id_compressed (ctx, message_id);
Add support (and tests) for messages with really long message IDs. Scott Henson reported an internal error that occurred when he tried to add a message that referenced another message with a message ID well over 300 characters in length. The bug here was running into a Xapian limit for the length of metadata key names, (which is even more restrictive than the Xapian limit for the length of terms). We fix this by noticing long message ID values and instead using a message ID of the form "notmuch-sha1-<sha1_sum_of_message_id>". That is, we use SHA1 to generate a compressed, (but still unique), version of the message ID. We add support to the test suite to exercise this fix. The tests add a message referencing the long message ID, then add the message with the long message ID, then finally add another message referencing the long ID. Each of these tests exercise different code paths where the special handling is implemented. A final test ensures that all three messages are stitched together into a single thread---guaranteeing that the three code paths all act consistently.
2010-06-04 21:39:36 +02:00
return talloc_asprintf (ctx, NOTMUCH_METADATA_THREAD_ID_PREFIX "%s",
message_id);
Store thread ids for messages that we haven't seen yet This allows us to thread messages even when we receive them out of order, or never receive the root. The thread ids for messages that aren't present but are referred to are stored as metadata in the database and then retrieved if we ever get that message. When determining the thread id for a message we also check for this metadata so that we can thread descendants of a message together before we receive it. Edited by Carl Worth <cworth@cworth.org>: Split this portion of the commit from the earlier-applied portion adding test cases.
2010-03-13 22:27:57 +01:00
}
lib: Implement ghost-based thread linking This updates the thread linking code to use ghost messages instead of user metadata to link messages into threads. In contrast with the old approach, this is actually correct. Previously, thread merging updated only the thread IDs of message documents, not thread IDs stored in user metadata. As originally diagnosed by Mark Walters [1] and as demonstrated by the broken T260-thread-order test, this can cause notmuch to fail to link messages even though they're in the same thread. In principle the old approach could have been fixed by updating the user metadata thread IDs as well, but these are not indexed and hence this would have required a full scan of all stored thread IDs. Ghost messages solve this problem naturally by reusing the exact same thread ID and message ID representation and indexing as regular messages. Furthermore, thanks to this greater symmetry, ghost messages are also algorithmically simpler. We continue to support the old user metadata format, so this patch can't delete any code, but when we do remove support for the old format, several functions can simply be deleted. [1] id:8738h7kv2q.fsf@qmul.ac.uk
2014-10-23 14:30:37 +02:00
static notmuch_status_t
_resolve_message_id_to_thread_id_old (notmuch_database_t *notmuch,
void *ctx,
const char *message_id,
const char **thread_id_ret);
Change database to store only a single thread ID per message. Instead of supporting multiple thread IDs, we now merge together thread IDs if one message is ever found to belong to more than one thread. This allows for constructing complete threads when, for example, a child message doesn't include a complete list of References headers back to the beginning of the thread. It also simplifies dealing with mapping a message ID to a thread ID which is now a simple get_thread_id just like get_message_id, (and no longer an iterator-based thing like get_tags).
2009-10-25 22:54:13 +01:00
/* Find the thread ID to which the message with 'message_id' belongs.
add_message: Pull the thread-stitching portion out into new _notmuch_database_link_message The function was getting too long-winded before. Add since I'm about to change how we handle the thread linking, it's convenient to have it in an isolated function.
2009-10-25 19:03:55 +01:00
*
lib: make find_message{,by_filename) report errors Previously, the functions notmuch_database_find_message() and notmuch_database_find_message_by_filename() functions did not properly report error condition to the library user. For more information, read the thread on the notmuch mailing list starting with my mail "id:871uv2unfd.fsf@gmail.com" Make these functions accept a pointer to 'notmuch_message_t' as argument and return notmuch_status_t which may be used to check for any error condition. restore: Modify for the new notmuch_database_find_message() new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 06:55:29 +02:00
* Note: 'thread_id_ret' must not be NULL!
* On success '*thread_id_ret' is set to a newly talloced string belonging to
* 'ctx'.
add_message: Pull the thread-stitching portion out into new _notmuch_database_link_message The function was getting too long-winded before. Add since I'm about to change how we handle the thread linking, it's convenient to have it in an isolated function.
2009-10-25 19:03:55 +01:00
*
lib: Fix internal documentation of _resolve_message_id_to_thread_id We no longer return NULL, but instead generate a new thread ID for messages that we haven't seen yet.
2010-04-12 23:19:15 +02:00
* Note: If there is no message in the database with the given
* 'message_id' then a new thread_id will be allocated for this
lib: Implement ghost-based thread linking This updates the thread linking code to use ghost messages instead of user metadata to link messages into threads. In contrast with the old approach, this is actually correct. Previously, thread merging updated only the thread IDs of message documents, not thread IDs stored in user metadata. As originally diagnosed by Mark Walters [1] and as demonstrated by the broken T260-thread-order test, this can cause notmuch to fail to link messages even though they're in the same thread. In principle the old approach could have been fixed by updating the user metadata thread IDs as well, but these are not indexed and hence this would have required a full scan of all stored thread IDs. Ghost messages solve this problem naturally by reusing the exact same thread ID and message ID representation and indexing as regular messages. Furthermore, thanks to this greater symmetry, ghost messages are also algorithmically simpler. We continue to support the old user metadata format, so this patch can't delete any code, but when we do remove support for the old format, several functions can simply be deleted. [1] id:8738h7kv2q.fsf@qmul.ac.uk
2014-10-23 14:30:37 +02:00
* message ID and stored in the database metadata so that the
lib: Fix internal documentation of _resolve_message_id_to_thread_id We no longer return NULL, but instead generate a new thread ID for messages that we haven't seen yet.
2010-04-12 23:19:15 +02:00
* thread ID can be looked up if the message is added to the database
lib: Implement ghost-based thread linking This updates the thread linking code to use ghost messages instead of user metadata to link messages into threads. In contrast with the old approach, this is actually correct. Previously, thread merging updated only the thread IDs of message documents, not thread IDs stored in user metadata. As originally diagnosed by Mark Walters [1] and as demonstrated by the broken T260-thread-order test, this can cause notmuch to fail to link messages even though they're in the same thread. In principle the old approach could have been fixed by updating the user metadata thread IDs as well, but these are not indexed and hence this would have required a full scan of all stored thread IDs. Ghost messages solve this problem naturally by reusing the exact same thread ID and message ID representation and indexing as regular messages. Furthermore, thanks to this greater symmetry, ghost messages are also algorithmically simpler. We continue to support the old user metadata format, so this patch can't delete any code, but when we do remove support for the old format, several functions can simply be deleted. [1] id:8738h7kv2q.fsf@qmul.ac.uk
2014-10-23 14:30:37 +02:00
* later.
add_message: Pull the thread-stitching portion out into new _notmuch_database_link_message The function was getting too long-winded before. Add since I'm about to change how we handle the thread linking, it's convenient to have it in an isolated function.
2009-10-25 19:03:55 +01:00
*/
lib: make find_message{,by_filename) report errors Previously, the functions notmuch_database_find_message() and notmuch_database_find_message_by_filename() functions did not properly report error condition to the library user. For more information, read the thread on the notmuch mailing list starting with my mail "id:871uv2unfd.fsf@gmail.com" Make these functions accept a pointer to 'notmuch_message_t' as argument and return notmuch_status_t which may be used to check for any error condition. restore: Modify for the new notmuch_database_find_message() new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 06:55:29 +02:00
static notmuch_status_t
Change database to store only a single thread ID per message. Instead of supporting multiple thread IDs, we now merge together thread IDs if one message is ever found to belong to more than one thread. This allows for constructing complete threads when, for example, a child message doesn't include a complete list of References headers back to the beginning of the thread. It also simplifies dealing with mapping a message ID to a thread ID which is now a simple get_thread_id just like get_message_id, (and no longer an iterator-based thing like get_tags).
2009-10-25 22:54:13 +01:00
_resolve_message_id_to_thread_id (notmuch_database_t *notmuch,
void *ctx,
lib: make find_message{,by_filename) report errors Previously, the functions notmuch_database_find_message() and notmuch_database_find_message_by_filename() functions did not properly report error condition to the library user. For more information, read the thread on the notmuch mailing list starting with my mail "id:871uv2unfd.fsf@gmail.com" Make these functions accept a pointer to 'notmuch_message_t' as argument and return notmuch_status_t which may be used to check for any error condition. restore: Modify for the new notmuch_database_find_message() new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 06:55:29 +02:00
const char *message_id,
const char **thread_id_ret)
lib: Implement ghost-based thread linking This updates the thread linking code to use ghost messages instead of user metadata to link messages into threads. In contrast with the old approach, this is actually correct. Previously, thread merging updated only the thread IDs of message documents, not thread IDs stored in user metadata. As originally diagnosed by Mark Walters [1] and as demonstrated by the broken T260-thread-order test, this can cause notmuch to fail to link messages even though they're in the same thread. In principle the old approach could have been fixed by updating the user metadata thread IDs as well, but these are not indexed and hence this would have required a full scan of all stored thread IDs. Ghost messages solve this problem naturally by reusing the exact same thread ID and message ID representation and indexing as regular messages. Furthermore, thanks to this greater symmetry, ghost messages are also algorithmically simpler. We continue to support the old user metadata format, so this patch can't delete any code, but when we do remove support for the old format, several functions can simply be deleted. [1] id:8738h7kv2q.fsf@qmul.ac.uk
2014-10-23 14:30:37 +02:00
{
notmuch_private_status_t status;
notmuch_message_t *message;
if (! (notmuch->features & NOTMUCH_FEATURE_GHOSTS))
return _resolve_message_id_to_thread_id_old (notmuch, ctx, message_id,
thread_id_ret);
/* Look for this message (regular or ghost) */
message = _notmuch_message_create_for_message_id (
notmuch, message_id, &status);
if (status == NOTMUCH_PRIVATE_STATUS_SUCCESS) {
/* Message exists */
*thread_id_ret = talloc_steal (
ctx, notmuch_message_get_thread_id (message));
} else if (status == NOTMUCH_PRIVATE_STATUS_NO_DOCUMENT_FOUND) {
/* Message did not exist. Give it a fresh thread ID and
* populate this message as a ghost message. */
*thread_id_ret = talloc_strdup (
ctx, _notmuch_database_generate_thread_id (notmuch));
if (! *thread_id_ret) {
status = NOTMUCH_PRIVATE_STATUS_OUT_OF_MEMORY;
} else {
status = _notmuch_message_initialize_ghost (message, *thread_id_ret);
if (status == 0)
/* Commit the new ghost message */
_notmuch_message_sync (message);
}
} else {
/* Create failed. Fall through. */
}
notmuch_message_destroy (message);
return COERCE_STATUS (status, "Error creating ghost message");
}
/* Pre-ghost messages _resolve_message_id_to_thread_id */
static notmuch_status_t
_resolve_message_id_to_thread_id_old (notmuch_database_t *notmuch,
void *ctx,
const char *message_id,
const char **thread_id_ret)
Change database to store only a single thread ID per message. Instead of supporting multiple thread IDs, we now merge together thread IDs if one message is ever found to belong to more than one thread. This allows for constructing complete threads when, for example, a child message doesn't include a complete list of References headers back to the beginning of the thread. It also simplifies dealing with mapping a message ID to a thread ID which is now a simple get_thread_id just like get_message_id, (and no longer an iterator-based thing like get_tags).
2009-10-25 22:54:13 +01:00
{
lib: make find_message{,by_filename) report errors Previously, the functions notmuch_database_find_message() and notmuch_database_find_message_by_filename() functions did not properly report error condition to the library user. For more information, read the thread on the notmuch mailing list starting with my mail "id:871uv2unfd.fsf@gmail.com" Make these functions accept a pointer to 'notmuch_message_t' as argument and return notmuch_status_t which may be used to check for any error condition. restore: Modify for the new notmuch_database_find_message() new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 06:55:29 +02:00
notmuch_status_t status;
Change database to store only a single thread ID per message. Instead of supporting multiple thread IDs, we now merge together thread IDs if one message is ever found to belong to more than one thread. This allows for constructing complete threads when, for example, a child message doesn't include a complete list of References headers back to the beginning of the thread. It also simplifies dealing with mapping a message ID to a thread ID which is now a simple get_thread_id just like get_message_id, (and no longer an iterator-based thing like get_tags).
2009-10-25 22:54:13 +01:00
notmuch_message_t *message;
lib: Simplify code flow in _resolve_message_id_to_thread_id There are two primary cases in this function, (the message exists in the database or it does not). Previously the code for these two cases was split and intermingled with goto-spaghetti connections.
2010-04-12 23:29:36 +02:00
string thread_id_string;
char *metadata_key;
Xapian::WritableDatabase *db;
Change database to store only a single thread ID per message. Instead of supporting multiple thread IDs, we now merge together thread IDs if one message is ever found to belong to more than one thread. This allows for constructing complete threads when, for example, a child message doesn't include a complete list of References headers back to the beginning of the thread. It also simplifies dealing with mapping a message ID to a thread ID which is now a simple get_thread_id just like get_message_id, (and no longer an iterator-based thing like get_tags).
2009-10-25 22:54:13 +01:00
lib: make find_message{,by_filename) report errors Previously, the functions notmuch_database_find_message() and notmuch_database_find_message_by_filename() functions did not properly report error condition to the library user. For more information, read the thread on the notmuch mailing list starting with my mail "id:871uv2unfd.fsf@gmail.com" Make these functions accept a pointer to 'notmuch_message_t' as argument and return notmuch_status_t which may be used to check for any error condition. restore: Modify for the new notmuch_database_find_message() new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 06:55:29 +02:00
status = notmuch_database_find_message (notmuch, message_id, &message);
if (status)
return status;
Change database to store only a single thread ID per message. Instead of supporting multiple thread IDs, we now merge together thread IDs if one message is ever found to belong to more than one thread. This allows for constructing complete threads when, for example, a child message doesn't include a complete list of References headers back to the beginning of the thread. It also simplifies dealing with mapping a message ID to a thread ID which is now a simple get_thread_id just like get_message_id, (and no longer an iterator-based thing like get_tags).
2009-10-25 22:54:13 +01:00
lib: Simplify code flow in _resolve_message_id_to_thread_id There are two primary cases in this function, (the message exists in the database or it does not). Previously the code for these two cases was split and intermingled with goto-spaghetti connections.
2010-04-12 23:29:36 +02:00
if (message) {
lib: make find_message{,by_filename) report errors Previously, the functions notmuch_database_find_message() and notmuch_database_find_message_by_filename() functions did not properly report error condition to the library user. For more information, read the thread on the notmuch mailing list starting with my mail "id:871uv2unfd.fsf@gmail.com" Make these functions accept a pointer to 'notmuch_message_t' as argument and return notmuch_status_t which may be used to check for any error condition. restore: Modify for the new notmuch_database_find_message() new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 06:55:29 +02:00
*thread_id_ret = talloc_steal (ctx,
notmuch_message_get_thread_id (message));
Change database to store only a single thread ID per message. Instead of supporting multiple thread IDs, we now merge together thread IDs if one message is ever found to belong to more than one thread. This allows for constructing complete threads when, for example, a child message doesn't include a complete list of References headers back to the beginning of the thread. It also simplifies dealing with mapping a message ID to a thread ID which is now a simple get_thread_id just like get_message_id, (and no longer an iterator-based thing like get_tags).
2009-10-25 22:54:13 +01:00
notmuch_message_destroy (message);
lib: make find_message{,by_filename) report errors Previously, the functions notmuch_database_find_message() and notmuch_database_find_message_by_filename() functions did not properly report error condition to the library user. For more information, read the thread on the notmuch mailing list starting with my mail "id:871uv2unfd.fsf@gmail.com" Make these functions accept a pointer to 'notmuch_message_t' as argument and return notmuch_status_t which may be used to check for any error condition. restore: Modify for the new notmuch_database_find_message() new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 06:55:29 +02:00
return NOTMUCH_STATUS_SUCCESS;
lib: Simplify code flow in _resolve_message_id_to_thread_id There are two primary cases in this function, (the message exists in the database or it does not). Previously the code for these two cases was split and intermingled with goto-spaghetti connections.
2010-04-12 23:29:36 +02:00
}
/* Message has not been seen yet.
*
* We may have seen a reference to it already, in which case, we
* can return the thread ID stored in the metadata. Otherwise, we
* generate a new thread ID and store it there.
*/
db = static_cast <Xapian::WritableDatabase *> (notmuch->xapian_db);
metadata_key = _get_metadata_thread_id_key (ctx, message_id);
thread_id_string = notmuch->xapian_db->get_metadata (metadata_key);
if (thread_id_string.empty()) {
lib: make find_message{,by_filename) report errors Previously, the functions notmuch_database_find_message() and notmuch_database_find_message_by_filename() functions did not properly report error condition to the library user. For more information, read the thread on the notmuch mailing list starting with my mail "id:871uv2unfd.fsf@gmail.com" Make these functions accept a pointer to 'notmuch_message_t' as argument and return notmuch_status_t which may be used to check for any error condition. restore: Modify for the new notmuch_database_find_message() new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 06:55:29 +02:00
*thread_id_ret = talloc_strdup (ctx,
_notmuch_database_generate_thread_id (notmuch));
db->set_metadata (metadata_key, *thread_id_ret);
lib: Simplify code flow in _resolve_message_id_to_thread_id There are two primary cases in this function, (the message exists in the database or it does not). Previously the code for these two cases was split and intermingled with goto-spaghetti connections.
2010-04-12 23:29:36 +02:00
} else {
lib: make find_message{,by_filename) report errors Previously, the functions notmuch_database_find_message() and notmuch_database_find_message_by_filename() functions did not properly report error condition to the library user. For more information, read the thread on the notmuch mailing list starting with my mail "id:871uv2unfd.fsf@gmail.com" Make these functions accept a pointer to 'notmuch_message_t' as argument and return notmuch_status_t which may be used to check for any error condition. restore: Modify for the new notmuch_database_find_message() new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 06:55:29 +02:00
*thread_id_ret = talloc_strdup (ctx, thread_id_string.c_str());
lib: Simplify code flow in _resolve_message_id_to_thread_id There are two primary cases in this function, (the message exists in the database or it does not). Previously the code for these two cases was split and intermingled with goto-spaghetti connections.
2010-04-12 23:29:36 +02:00
}
talloc_free (metadata_key);
lib: make find_message{,by_filename) report errors Previously, the functions notmuch_database_find_message() and notmuch_database_find_message_by_filename() functions did not properly report error condition to the library user. For more information, read the thread on the notmuch mailing list starting with my mail "id:871uv2unfd.fsf@gmail.com" Make these functions accept a pointer to 'notmuch_message_t' as argument and return notmuch_status_t which may be used to check for any error condition. restore: Modify for the new notmuch_database_find_message() new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 06:55:29 +02:00
return NOTMUCH_STATUS_SUCCESS;
Change database to store only a single thread ID per message. Instead of supporting multiple thread IDs, we now merge together thread IDs if one message is ever found to belong to more than one thread. This allows for constructing complete threads when, for example, a child message doesn't include a complete list of References headers back to the beginning of the thread. It also simplifies dealing with mapping a message ID to a thread ID which is now a simple get_thread_id just like get_message_id, (and no longer an iterator-based thing like get_tags).
2009-10-25 22:54:13 +01:00
}
add_message: Pull the thread-stitching portion out into new _notmuch_database_link_message The function was getting too long-winded before. Add since I'm about to change how we handle the thread linking, it's convenient to have it in an isolated function.
2009-10-25 19:03:55 +01:00
static notmuch_status_t
Change database to store only a single thread ID per message. Instead of supporting multiple thread IDs, we now merge together thread IDs if one message is ever found to belong to more than one thread. This allows for constructing complete threads when, for example, a child message doesn't include a complete list of References headers back to the beginning of the thread. It also simplifies dealing with mapping a message ID to a thread ID which is now a simple get_thread_id just like get_message_id, (and no longer an iterator-based thing like get_tags).
2009-10-25 22:54:13 +01:00
_merge_threads (notmuch_database_t *notmuch,
const char *winner_thread_id,
const char *loser_thread_id)
{
Xapian::PostingIterator loser, loser_end;
notmuch_message_t *message = NULL;
notmuch_private_status_t private_status;
notmuch_status_t ret = NOTMUCH_STATUS_SUCCESS;
find_doc_ids (notmuch, "thread", loser_thread_id, &loser, &loser_end);
for ( ; loser != loser_end; loser++) {
message = _notmuch_message_create (notmuch, notmuch,
*loser, &private_status);
if (message == NULL) {
ret = COERCE_STATUS (private_status,
"Cannot find document for doc_id from query");
goto DONE;
}
_notmuch_message_remove_term (message, "thread", loser_thread_id);
_notmuch_message_add_term (message, "thread", winner_thread_id);
_notmuch_message_sync (message);
notmuch_message_destroy (message);
message = NULL;
}
DONE:
if (message)
notmuch_message_destroy (message);
return ret;
}
Remove all calls to g_strdup_printf Replacing them with calls to talloc_asprintf if possible, otherwise to asprintf (with it's painful error-handling leaving the pointer undefined).
2009-10-26 23:17:10 +01:00
static void
_my_talloc_free_for_g_hash (void *ptr)
{
talloc_free (ptr);
}
Change database to store only a single thread ID per message. Instead of supporting multiple thread IDs, we now merge together thread IDs if one message is ever found to belong to more than one thread. This allows for constructing complete threads when, for example, a child message doesn't include a complete list of References headers back to the beginning of the thread. It also simplifies dealing with mapping a message ID to a thread ID which is now a simple get_thread_id just like get_message_id, (and no longer an iterator-based thing like get_tags).
2009-10-25 22:54:13 +01:00
static notmuch_status_t
_notmuch_database_link_message_to_parents (notmuch_database_t *notmuch,
notmuch_message_t *message,
notmuch_message_file_t *message_file,
const char **thread_id)
add_message: Pull the thread-stitching portion out into new _notmuch_database_link_message The function was getting too long-winded before. Add since I'm about to change how we handle the thread linking, it's convenient to have it in an isolated function.
2009-10-25 19:03:55 +01:00
{
Change database to store only a single thread ID per message. Instead of supporting multiple thread IDs, we now merge together thread IDs if one message is ever found to belong to more than one thread. This allows for constructing complete threads when, for example, a child message doesn't include a complete list of References headers back to the beginning of the thread. It also simplifies dealing with mapping a message ID to a thread ID which is now a simple get_thread_id just like get_message_id, (and no longer an iterator-based thing like get_tags).
2009-10-25 22:54:13 +01:00
GHashTable *parents = NULL;
add_message: Don't add any self-references to the database. In our scheme it's illegal for any message to refer to itself, (nor would it be useful for anything anyway). Cut these self-references off at the source, before they trip up any internal errors.
2009-11-18 02:20:32 +01:00
const char *refs, *in_reply_to, *in_reply_to_message_id;
lib/database.cc: change how the parent of a message is calculated Presently, the code which finds the parent of a message as it is being added to the database assumes that the first Message-ID-like substring of the In-Reply-To header is the parent Message ID. Some mail clients, however, put stuff other than the Message-ID of the parent in the In-Reply-To header, such as the email address of the sender of the parent. This can fool notmuch. The updated algorithm prefers the last Message ID in the References header. The References header lists messages oldest-first, so the last Message ID is the parent (RFC2822, p. 24). The References header is also less likely to be in a non-standard syntax (http://cr.yp.to/immhf/thread.html, http://www.jwz.org/doc/threading.html). In case the References header is not to be found, fall back to the old behavior. V2 of this patch, incorporating feedback from Jani and (indirectly) Austin.
2013-03-06 04:31:49 +01:00
const char *last_ref_message_id, *this_message_id;
Change database to store only a single thread ID per message. Instead of supporting multiple thread IDs, we now merge together thread IDs if one message is ever found to belong to more than one thread. This allows for constructing complete threads when, for example, a child message doesn't include a complete list of References headers back to the beginning of the thread. It also simplifies dealing with mapping a message ID to a thread ID which is now a simple get_thread_id just like get_message_id, (and no longer an iterator-based thing like get_tags).
2009-10-25 22:54:13 +01:00
GList *l, *keys = NULL;
notmuch_status_t ret = NOTMUCH_STATUS_SUCCESS;
add_message: Pull the thread-stitching portion out into new _notmuch_database_link_message The function was getting too long-winded before. Add since I'm about to change how we handle the thread linking, it's convenient to have it in an isolated function.
2009-10-25 19:03:55 +01:00
Change database to store only a single thread ID per message. Instead of supporting multiple thread IDs, we now merge together thread IDs if one message is ever found to belong to more than one thread. This allows for constructing complete threads when, for example, a child message doesn't include a complete list of References headers back to the beginning of the thread. It also simplifies dealing with mapping a message ID to a thread ID which is now a simple get_thread_id just like get_message_id, (and no longer an iterator-based thing like get_tags).
2009-10-25 22:54:13 +01:00
parents = g_hash_table_new_full (g_str_hash, g_str_equal,
Remove all calls to g_strdup_printf Replacing them with calls to talloc_asprintf if possible, otherwise to asprintf (with it's painful error-handling leaving the pointer undefined).
2009-10-26 23:17:10 +01:00
_my_talloc_free_for_g_hash, NULL);
lib/database.cc: change how the parent of a message is calculated Presently, the code which finds the parent of a message as it is being added to the database assumes that the first Message-ID-like substring of the In-Reply-To header is the parent Message ID. Some mail clients, however, put stuff other than the Message-ID of the parent in the In-Reply-To header, such as the email address of the sender of the parent. This can fool notmuch. The updated algorithm prefers the last Message ID in the References header. The References header lists messages oldest-first, so the last Message ID is the parent (RFC2822, p. 24). The References header is also less likely to be in a non-standard syntax (http://cr.yp.to/immhf/thread.html, http://www.jwz.org/doc/threading.html). In case the References header is not to be found, fall back to the old behavior. V2 of this patch, incorporating feedback from Jani and (indirectly) Austin.
2013-03-06 04:31:49 +01:00
this_message_id = notmuch_message_get_message_id (message);
add_message: Pull the thread-stitching portion out into new _notmuch_database_link_message The function was getting too long-winded before. Add since I'm about to change how we handle the thread linking, it's convenient to have it in an isolated function.
2009-10-25 19:03:55 +01:00
lib: Start all function names in notmuch-private.h with As noted in devel/STYLE, every private library function should start with _notmuch. This patch corrects function naming that did not adhere to this style in lib/notmuch-private.h. In particular, the old function names that now begin with _notmuch are notmuch_sha1_of_file notmuch_sha1_of_string notmuch_message_file_close notmuch_message_file_get_header notmuch_message_file_open notmuch_message_get_author notmuch_message_set_author Signed-off-by: Charles Celerier <cceleri@cs.stanford.edu>
2014-05-13 11:44:05 +02:00
refs = _notmuch_message_file_get_header (message_file, "references");
lib/database.cc: change how the parent of a message is calculated Presently, the code which finds the parent of a message as it is being added to the database assumes that the first Message-ID-like substring of the In-Reply-To header is the parent Message ID. Some mail clients, however, put stuff other than the Message-ID of the parent in the In-Reply-To header, such as the email address of the sender of the parent. This can fool notmuch. The updated algorithm prefers the last Message ID in the References header. The References header lists messages oldest-first, so the last Message ID is the parent (RFC2822, p. 24). The References header is also less likely to be in a non-standard syntax (http://cr.yp.to/immhf/thread.html, http://www.jwz.org/doc/threading.html). In case the References header is not to be found, fall back to the old behavior. V2 of this patch, incorporating feedback from Jani and (indirectly) Austin.
2013-03-06 04:31:49 +01:00
last_ref_message_id = parse_references (message,
this_message_id,
parents, refs);
add_message: Pull the thread-stitching portion out into new _notmuch_database_link_message The function was getting too long-winded before. Add since I'm about to change how we handle the thread linking, it's convenient to have it in an isolated function.
2009-10-25 19:03:55 +01:00
lib: Start all function names in notmuch-private.h with As noted in devel/STYLE, every private library function should start with _notmuch. This patch corrects function naming that did not adhere to this style in lib/notmuch-private.h. In particular, the old function names that now begin with _notmuch are notmuch_sha1_of_file notmuch_sha1_of_string notmuch_message_file_close notmuch_message_file_get_header notmuch_message_file_open notmuch_message_get_author notmuch_message_set_author Signed-off-by: Charles Celerier <cceleri@cs.stanford.edu>
2014-05-13 11:44:05 +02:00
in_reply_to = _notmuch_message_file_get_header (message_file, "in-reply-to");
lib/database.cc: change how the parent of a message is calculated Presently, the code which finds the parent of a message as it is being added to the database assumes that the first Message-ID-like substring of the In-Reply-To header is the parent Message ID. Some mail clients, however, put stuff other than the Message-ID of the parent in the In-Reply-To header, such as the email address of the sender of the parent. This can fool notmuch. The updated algorithm prefers the last Message ID in the References header. The References header lists messages oldest-first, so the last Message ID is the parent (RFC2822, p. 24). The References header is also less likely to be in a non-standard syntax (http://cr.yp.to/immhf/thread.html, http://www.jwz.org/doc/threading.html). In case the References header is not to be found, fall back to the old behavior. V2 of this patch, incorporating feedback from Jani and (indirectly) Austin.
2013-03-06 04:31:49 +01:00
in_reply_to_message_id = parse_references (message,
this_message_id,
parents, in_reply_to);
/* For the parent of this message, use the last message ID of the
* References header, if available. If not, fall back to the
* first message ID in the In-Reply-To header. */
if (last_ref_message_id) {
lib: whitespace cleanup Cleaned the following whitespace in lib/* files: lib/index.cc: 1 line: trailing whitespace lib/database.cc 5 lines: 8 spaces at the beginning of line lib/notmuch-private.h: 4 lines: 8 spaces at the beginning of line lib/message.cc: 1 line: trailing whitespace lib/sha1.c: 1 line: empty lines at the end of file lib/query.cc: 2 lines: 8 spaces at the beginning of line lib/gen-version-script.sh: 1 line: trailing whitespace
2016-05-28 19:45:31 +02:00
_notmuch_message_add_term (message, "replyto",
last_ref_message_id);
lib/database.cc: change how the parent of a message is calculated Presently, the code which finds the parent of a message as it is being added to the database assumes that the first Message-ID-like substring of the In-Reply-To header is the parent Message ID. Some mail clients, however, put stuff other than the Message-ID of the parent in the In-Reply-To header, such as the email address of the sender of the parent. This can fool notmuch. The updated algorithm prefers the last Message ID in the References header. The References header lists messages oldest-first, so the last Message ID is the parent (RFC2822, p. 24). The References header is also less likely to be in a non-standard syntax (http://cr.yp.to/immhf/thread.html, http://www.jwz.org/doc/threading.html). In case the References header is not to be found, fall back to the old behavior. V2 of this patch, incorporating feedback from Jani and (indirectly) Austin.
2013-03-06 04:31:49 +01:00
} else if (in_reply_to_message_id) {
add_message: Don't add any self-references to the database. In our scheme it's illegal for any message to refer to itself, (nor would it be useful for anything anyway). Cut these self-references off at the source, before they trip up any internal errors.
2009-11-18 02:20:32 +01:00
_notmuch_message_add_term (message, "replyto",
lib/database.cc: change how the parent of a message is calculated Presently, the code which finds the parent of a message as it is being added to the database assumes that the first Message-ID-like substring of the In-Reply-To header is the parent Message ID. Some mail clients, however, put stuff other than the Message-ID of the parent in the In-Reply-To header, such as the email address of the sender of the parent. This can fool notmuch. The updated algorithm prefers the last Message ID in the References header. The References header lists messages oldest-first, so the last Message ID is the parent (RFC2822, p. 24). The References header is also less likely to be in a non-standard syntax (http://cr.yp.to/immhf/thread.html, http://www.jwz.org/doc/threading.html). In case the References header is not to be found, fall back to the old behavior. V2 of this patch, incorporating feedback from Jani and (indirectly) Austin.
2013-03-06 04:31:49 +01:00
in_reply_to_message_id);
add_message: Don't add any self-references to the database. In our scheme it's illegal for any message to refer to itself, (nor would it be useful for anything anyway). Cut these self-references off at the source, before they trip up any internal errors.
2009-11-18 02:20:32 +01:00
}
add_message: Pull the thread-stitching portion out into new _notmuch_database_link_message The function was getting too long-winded before. Add since I'm about to change how we handle the thread linking, it's convenient to have it in an isolated function.
2009-10-25 19:03:55 +01:00
Change database to store only a single thread ID per message. Instead of supporting multiple thread IDs, we now merge together thread IDs if one message is ever found to belong to more than one thread. This allows for constructing complete threads when, for example, a child message doesn't include a complete list of References headers back to the beginning of the thread. It also simplifies dealing with mapping a message ID to a thread ID which is now a simple get_thread_id just like get_message_id, (and no longer an iterator-based thing like get_tags).
2009-10-25 22:54:13 +01:00
keys = g_hash_table_get_keys (parents);
for (l = keys; l; l = l->next) {
char *parent_message_id;
notmuch: Quiet buildbot warnings. Cast away the result of various *write functions. Provide a default value for some variables to avoid "use before set" warnings.
2011-12-20 16:20:04 +01:00
const char *parent_thread_id = NULL;
Change database to store only a single thread ID per message. Instead of supporting multiple thread IDs, we now merge together thread IDs if one message is ever found to belong to more than one thread. This allows for constructing complete threads when, for example, a child message doesn't include a complete list of References headers back to the beginning of the thread. It also simplifies dealing with mapping a message ID to a thread ID which is now a simple get_thread_id just like get_message_id, (and no longer an iterator-based thing like get_tags).
2009-10-25 22:54:13 +01:00
parent_message_id = (char *) l->data;
lib: Always add reference terms to the database. Previously, we were only adding the reference terms for cases where the referenced message did not yet exist in the database. For thread presentation, it's useful to have the connection information provided by the references, even when the messages are present. So add this term unconditionally.
2010-04-13 00:45:40 +02:00
_notmuch_message_add_term (message, "reference",
parent_message_id);
lib: make find_message{,by_filename) report errors Previously, the functions notmuch_database_find_message() and notmuch_database_find_message_by_filename() functions did not properly report error condition to the library user. For more information, read the thread on the notmuch mailing list starting with my mail "id:871uv2unfd.fsf@gmail.com" Make these functions accept a pointer to 'notmuch_message_t' as argument and return notmuch_status_t which may be used to check for any error condition. restore: Modify for the new notmuch_database_find_message() new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 06:55:29 +02:00
ret = _resolve_message_id_to_thread_id (notmuch,
message,
parent_message_id,
&parent_thread_id);
if (ret)
goto DONE;
Change database to store only a single thread ID per message. Instead of supporting multiple thread IDs, we now merge together thread IDs if one message is ever found to belong to more than one thread. This allows for constructing complete threads when, for example, a child message doesn't include a complete list of References headers back to the beginning of the thread. It also simplifies dealing with mapping a message ID to a thread ID which is now a simple get_thread_id just like get_message_id, (and no longer an iterator-based thing like get_tags).
2009-10-25 22:54:13 +01:00
lib: Remove condition regarding a NULL parent_thread_id. A recent change guaranteed that a message ID can never be resolved to a NULL thread ID, so we don't need this extra case.
2010-04-13 00:54:03 +02:00
if (*thread_id == NULL) {
*thread_id = talloc_strdup (message, parent_thread_id);
_notmuch_message_add_term (message, "thread", *thread_id);
} else if (strcmp (*thread_id, parent_thread_id)) {
ret = _merge_threads (notmuch, *thread_id, parent_thread_id);
if (ret)
goto DONE;
Change database to store only a single thread ID per message. Instead of supporting multiple thread IDs, we now merge together thread IDs if one message is ever found to belong to more than one thread. This allows for constructing complete threads when, for example, a child message doesn't include a complete list of References headers back to the beginning of the thread. It also simplifies dealing with mapping a message ID to a thread ID which is now a simple get_thread_id just like get_message_id, (and no longer an iterator-based thing like get_tags).
2009-10-25 22:54:13 +01:00
}
}
add_message: Pull the thread-stitching portion out into new _notmuch_database_link_message The function was getting too long-winded before. Add since I'm about to change how we handle the thread linking, it's convenient to have it in an isolated function.
2009-10-25 19:03:55 +01:00
Change database to store only a single thread ID per message. Instead of supporting multiple thread IDs, we now merge together thread IDs if one message is ever found to belong to more than one thread. This allows for constructing complete threads when, for example, a child message doesn't include a complete list of References headers back to the beginning of the thread. It also simplifies dealing with mapping a message ID to a thread ID which is now a simple get_thread_id just like get_message_id, (and no longer an iterator-based thing like get_tags).
2009-10-25 22:54:13 +01:00
DONE:
if (keys)
g_list_free (keys);
if (parents)
g_hash_table_unref (parents);
add_message: Pull the thread-stitching portion out into new _notmuch_database_link_message The function was getting too long-winded before. Add since I'm about to change how we handle the thread linking, it's convenient to have it in an isolated function.
2009-10-25 19:03:55 +01:00
Change database to store only a single thread ID per message. Instead of supporting multiple thread IDs, we now merge together thread IDs if one message is ever found to belong to more than one thread. This allows for constructing complete threads when, for example, a child message doesn't include a complete list of References headers back to the beginning of the thread. It also simplifies dealing with mapping a message ID to a thread ID which is now a simple get_thread_id just like get_message_id, (and no longer an iterator-based thing like get_tags).
2009-10-25 22:54:13 +01:00
return ret;
}
add_message: Pull the thread-stitching portion out into new _notmuch_database_link_message The function was getting too long-winded before. Add since I'm about to change how we handle the thread linking, it's convenient to have it in an isolated function.
2009-10-25 19:03:55 +01:00
Change database to store only a single thread ID per message. Instead of supporting multiple thread IDs, we now merge together thread IDs if one message is ever found to belong to more than one thread. This allows for constructing complete threads when, for example, a child message doesn't include a complete list of References headers back to the beginning of the thread. It also simplifies dealing with mapping a message ID to a thread ID which is now a simple get_thread_id just like get_message_id, (and no longer an iterator-based thing like get_tags).
2009-10-25 22:54:13 +01:00
static notmuch_status_t
_notmuch_database_link_message_to_children (notmuch_database_t *notmuch,
notmuch_message_t *message,
const char **thread_id)
{
const char *message_id = notmuch_message_get_message_id (message);
Xapian::PostingIterator child, children_end;
notmuch_message_t *child_message = NULL;
const char *child_thread_id;
notmuch_status_t ret = NOTMUCH_STATUS_SUCCESS;
notmuch_private_status_t private_status;
database: Rename "ref" prefix name to "reference" Which is more consistent with the XREFERENCE prefix used in the terms in the database. Also remove some stale documentation describing the removal of resolved references from the database (we no longer do this).
2009-11-18 03:44:02 +01:00
find_doc_ids (notmuch, "reference", message_id, &child, &children_end);
Change database to store only a single thread ID per message. Instead of supporting multiple thread IDs, we now merge together thread IDs if one message is ever found to belong to more than one thread. This allows for constructing complete threads when, for example, a child message doesn't include a complete list of References headers back to the beginning of the thread. It also simplifies dealing with mapping a message ID to a thread ID which is now a simple get_thread_id just like get_message_id, (and no longer an iterator-based thing like get_tags).
2009-10-25 22:54:13 +01:00
for ( ; child != children_end; child++) {
add_message: Pull the thread-stitching portion out into new _notmuch_database_link_message The function was getting too long-winded before. Add since I'm about to change how we handle the thread linking, it's convenient to have it in an isolated function.
2009-10-25 19:03:55 +01:00
Change database to store only a single thread ID per message. Instead of supporting multiple thread IDs, we now merge together thread IDs if one message is ever found to belong to more than one thread. This allows for constructing complete threads when, for example, a child message doesn't include a complete list of References headers back to the beginning of the thread. It also simplifies dealing with mapping a message ID to a thread ID which is now a simple get_thread_id just like get_message_id, (and no longer an iterator-based thing like get_tags).
2009-10-25 22:54:13 +01:00
child_message = _notmuch_message_create (message, notmuch,
*child, &private_status);
if (child_message == NULL) {
ret = COERCE_STATUS (private_status,
"Cannot find document for doc_id from query");
goto DONE;
add_message: Pull the thread-stitching portion out into new _notmuch_database_link_message The function was getting too long-winded before. Add since I'm about to change how we handle the thread linking, it's convenient to have it in an isolated function.
2009-10-25 19:03:55 +01:00
}
Change database to store only a single thread ID per message. Instead of supporting multiple thread IDs, we now merge together thread IDs if one message is ever found to belong to more than one thread. This allows for constructing complete threads when, for example, a child message doesn't include a complete list of References headers back to the beginning of the thread. It also simplifies dealing with mapping a message ID to a thread ID which is now a simple get_thread_id just like get_message_id, (and no longer an iterator-based thing like get_tags).
2009-10-25 22:54:13 +01:00
child_thread_id = notmuch_message_get_thread_id (child_message);
if (*thread_id == NULL) {
*thread_id = talloc_strdup (message, child_thread_id);
_notmuch_message_add_term (message, "thread", *thread_id);
} else if (strcmp (*thread_id, child_thread_id)) {
database: Rename "ref" prefix name to "reference" Which is more consistent with the XREFERENCE prefix used in the terms in the database. Also remove some stale documentation describing the removal of resolved references from the database (we no longer do this).
2009-11-18 03:44:02 +01:00
_notmuch_message_remove_term (child_message, "reference",
Change database to store only a single thread ID per message. Instead of supporting multiple thread IDs, we now merge together thread IDs if one message is ever found to belong to more than one thread. This allows for constructing complete threads when, for example, a child message doesn't include a complete list of References headers back to the beginning of the thread. It also simplifies dealing with mapping a message ID to a thread ID which is now a simple get_thread_id just like get_message_id, (and no longer an iterator-based thing like get_tags).
2009-10-25 22:54:13 +01:00
message_id);
_notmuch_message_sync (child_message);
ret = _merge_threads (notmuch, *thread_id, child_thread_id);
if (ret)
goto DONE;
}
notmuch_message_destroy (child_message);
child_message = NULL;
add_message: Pull the thread-stitching portion out into new _notmuch_database_link_message The function was getting too long-winded before. Add since I'm about to change how we handle the thread linking, it's convenient to have it in an isolated function.
2009-10-25 19:03:55 +01:00
}
Change database to store only a single thread ID per message. Instead of supporting multiple thread IDs, we now merge together thread IDs if one message is ever found to belong to more than one thread. This allows for constructing complete threads when, for example, a child message doesn't include a complete list of References headers back to the beginning of the thread. It also simplifies dealing with mapping a message ID to a thread ID which is now a simple get_thread_id just like get_message_id, (and no longer an iterator-based thing like get_tags).
2009-10-25 22:54:13 +01:00
DONE:
if (child_message)
notmuch_message_destroy (child_message);
return ret;
}
add_message: Pull the thread-stitching portion out into new _notmuch_database_link_message The function was getting too long-winded before. Add since I'm about to change how we handle the thread linking, it's convenient to have it in an isolated function.
2009-10-25 19:03:55 +01:00
lib: Refactor _notmuch_database_link_message This moves the code to retrieve and clear the metadata thread ID out of _notmuch_database_link_message into its own function. This will simplify future changes.
2014-10-07 01:17:08 +02:00
/* Fetch and clear the stored thread_id for message, or NULL if none. */
static char *
_consume_metadata_thread_id (void *ctx, notmuch_database_t *notmuch,
notmuch_message_t *message)
{
const char *message_id;
string stored_id;
char *metadata_key;
message_id = notmuch_message_get_message_id (message);
metadata_key = _get_metadata_thread_id_key (ctx, message_id);
/* Check if we have already seen related messages to this one.
* If we have then use the thread_id that we stored at that time.
*/
stored_id = notmuch->xapian_db->get_metadata (metadata_key);
if (stored_id.empty ()) {
return NULL;
} else {
lib: whitespace cleanup Cleaned the following whitespace in lib/* files: lib/index.cc: 1 line: trailing whitespace lib/database.cc 5 lines: 8 spaces at the beginning of line lib/notmuch-private.h: 4 lines: 8 spaces at the beginning of line lib/message.cc: 1 line: trailing whitespace lib/sha1.c: 1 line: empty lines at the end of file lib/query.cc: 2 lines: 8 spaces at the beginning of line lib/gen-version-script.sh: 1 line: trailing whitespace
2016-05-28 19:45:31 +02:00
Xapian::WritableDatabase *db;
lib: Refactor _notmuch_database_link_message This moves the code to retrieve and clear the metadata thread ID out of _notmuch_database_link_message into its own function. This will simplify future changes.
2014-10-07 01:17:08 +02:00
db = static_cast <Xapian::WritableDatabase *> (notmuch->xapian_db);
/* Clear the metadata for this message ID. We don't need it
* anymore. */
lib: whitespace cleanup Cleaned the following whitespace in lib/* files: lib/index.cc: 1 line: trailing whitespace lib/database.cc 5 lines: 8 spaces at the beginning of line lib/notmuch-private.h: 4 lines: 8 spaces at the beginning of line lib/message.cc: 1 line: trailing whitespace lib/sha1.c: 1 line: empty lines at the end of file lib/query.cc: 2 lines: 8 spaces at the beginning of line lib/gen-version-script.sh: 1 line: trailing whitespace
2016-05-28 19:45:31 +02:00
db->set_metadata (metadata_key, "");
lib: Refactor _notmuch_database_link_message This moves the code to retrieve and clear the metadata thread ID out of _notmuch_database_link_message into its own function. This will simplify future changes.
2014-10-07 01:17:08 +02:00
lib: whitespace cleanup Cleaned the following whitespace in lib/* files: lib/index.cc: 1 line: trailing whitespace lib/database.cc 5 lines: 8 spaces at the beginning of line lib/notmuch-private.h: 4 lines: 8 spaces at the beginning of line lib/message.cc: 1 line: trailing whitespace lib/sha1.c: 1 line: empty lines at the end of file lib/query.cc: 2 lines: 8 spaces at the beginning of line lib/gen-version-script.sh: 1 line: trailing whitespace
2016-05-28 19:45:31 +02:00
return talloc_strdup (ctx, stored_id.c_str ());
lib: Refactor _notmuch_database_link_message This moves the code to retrieve and clear the metadata thread ID out of _notmuch_database_link_message into its own function. This will simplify future changes.
2014-10-07 01:17:08 +02:00
}
}
lib: Implement ghost-based thread linking This updates the thread linking code to use ghost messages instead of user metadata to link messages into threads. In contrast with the old approach, this is actually correct. Previously, thread merging updated only the thread IDs of message documents, not thread IDs stored in user metadata. As originally diagnosed by Mark Walters [1] and as demonstrated by the broken T260-thread-order test, this can cause notmuch to fail to link messages even though they're in the same thread. In principle the old approach could have been fixed by updating the user metadata thread IDs as well, but these are not indexed and hence this would have required a full scan of all stored thread IDs. Ghost messages solve this problem naturally by reusing the exact same thread ID and message ID representation and indexing as regular messages. Furthermore, thanks to this greater symmetry, ghost messages are also algorithmically simpler. We continue to support the old user metadata format, so this patch can't delete any code, but when we do remove support for the old format, several functions can simply be deleted. [1] id:8738h7kv2q.fsf@qmul.ac.uk
2014-10-23 14:30:37 +02:00
/* Given a blank or ghost 'message' and its corresponding
Change database to store only a single thread ID per message. Instead of supporting multiple thread IDs, we now merge together thread IDs if one message is ever found to belong to more than one thread. This allows for constructing complete threads when, for example, a child message doesn't include a complete list of References headers back to the beginning of the thread. It also simplifies dealing with mapping a message ID to a thread ID which is now a simple get_thread_id just like get_message_id, (and no longer an iterator-based thing like get_tags).
2009-10-25 22:54:13 +01:00
* 'message_file' link it to existing threads in the database.
*
lib: Implement ghost-based thread linking This updates the thread linking code to use ghost messages instead of user metadata to link messages into threads. In contrast with the old approach, this is actually correct. Previously, thread merging updated only the thread IDs of message documents, not thread IDs stored in user metadata. As originally diagnosed by Mark Walters [1] and as demonstrated by the broken T260-thread-order test, this can cause notmuch to fail to link messages even though they're in the same thread. In principle the old approach could have been fixed by updating the user metadata thread IDs as well, but these are not indexed and hence this would have required a full scan of all stored thread IDs. Ghost messages solve this problem naturally by reusing the exact same thread ID and message ID representation and indexing as regular messages. Furthermore, thanks to this greater symmetry, ghost messages are also algorithmically simpler. We continue to support the old user metadata format, so this patch can't delete any code, but when we do remove support for the old format, several functions can simply be deleted. [1] id:8738h7kv2q.fsf@qmul.ac.uk
2014-10-23 14:30:37 +02:00
* First, if is_ghost, this retrieves the thread ID already stored in
* the message (which will be the case if a message was previously
* added that referenced this one). If the message is blank
* (!is_ghost), it doesn't have a thread ID yet (we'll generate one
* later in this function). If the database does not support ghost
* messages, this checks for a thread ID stored in database metadata
* for this message ID.
lib: Fix internal documentation of _notmuch_database_link_message This function was recently modified, (to include a metadata lookup for a message's thread ID before looking for parent/child thread IDs), but the documentation wasn't updated. Fix that.
2010-04-12 23:35:25 +02:00
*
* Second, we look at 'message_file' and its link-relevant headers
* (References and In-Reply-To) for message IDs.
*
* Finally, we look in the database for existing message that
* reference 'message'.
*
lib: Implement ghost-based thread linking This updates the thread linking code to use ghost messages instead of user metadata to link messages into threads. In contrast with the old approach, this is actually correct. Previously, thread merging updated only the thread IDs of message documents, not thread IDs stored in user metadata. As originally diagnosed by Mark Walters [1] and as demonstrated by the broken T260-thread-order test, this can cause notmuch to fail to link messages even though they're in the same thread. In principle the old approach could have been fixed by updating the user metadata thread IDs as well, but these are not indexed and hence this would have required a full scan of all stored thread IDs. Ghost messages solve this problem naturally by reusing the exact same thread ID and message ID representation and indexing as regular messages. Furthermore, thanks to this greater symmetry, ghost messages are also algorithmically simpler. We continue to support the old user metadata format, so this patch can't delete any code, but when we do remove support for the old format, several functions can simply be deleted. [1] id:8738h7kv2q.fsf@qmul.ac.uk
2014-10-23 14:30:37 +02:00
* In all cases, we assign to the current message the first thread ID
lib: Remove unnecessary thread linking steps when using ghost messages Previously, it was necessary to link new messages to children to work around some (though not all) problems with the old metadata-based approach to stored thread IDs. With ghost messages, this is no longer necessary, so don't bother with child linking when ghost messages are in use.
2014-10-23 14:30:41 +02:00
* found. We will also merge any existing, distinct threads where this
* message belongs to both, (which is not uncommon when messages are
* processed out of order).
Change database to store only a single thread ID per message. Instead of supporting multiple thread IDs, we now merge together thread IDs if one message is ever found to belong to more than one thread. This allows for constructing complete threads when, for example, a child message doesn't include a complete list of References headers back to the beginning of the thread. It also simplifies dealing with mapping a message ID to a thread ID which is now a simple get_thread_id just like get_message_id, (and no longer an iterator-based thing like get_tags).
2009-10-25 22:54:13 +01:00
*
lib: Remove unnecessary thread linking steps when using ghost messages Previously, it was necessary to link new messages to children to work around some (though not all) problems with the old metadata-based approach to stored thread IDs. With ghost messages, this is no longer necessary, so don't bother with child linking when ghost messages are in use.
2014-10-23 14:30:41 +02:00
* Finally, if no thread ID has been found through referenced messages, we
lib: Fix internal documentation of _notmuch_database_link_message This function was recently modified, (to include a metadata lookup for a message's thread ID before looking for parent/child thread IDs), but the documentation wasn't updated. Fix that.
2010-04-12 23:35:25 +02:00
* call _notmuch_message_generate_thread_id to generate a new thread
* ID. This should only happen for new, top-level messages, (no
* References or In-Reply-To header in this message, and no previously
* added message refers to this message).
Change database to store only a single thread ID per message. Instead of supporting multiple thread IDs, we now merge together thread IDs if one message is ever found to belong to more than one thread. This allows for constructing complete threads when, for example, a child message doesn't include a complete list of References headers back to the beginning of the thread. It also simplifies dealing with mapping a message ID to a thread ID which is now a simple get_thread_id just like get_message_id, (and no longer an iterator-based thing like get_tags).
2009-10-25 22:54:13 +01:00
*/
static notmuch_status_t
_notmuch_database_link_message (notmuch_database_t *notmuch,
notmuch_message_t *message,
lib: Implement ghost-based thread linking This updates the thread linking code to use ghost messages instead of user metadata to link messages into threads. In contrast with the old approach, this is actually correct. Previously, thread merging updated only the thread IDs of message documents, not thread IDs stored in user metadata. As originally diagnosed by Mark Walters [1] and as demonstrated by the broken T260-thread-order test, this can cause notmuch to fail to link messages even though they're in the same thread. In principle the old approach could have been fixed by updating the user metadata thread IDs as well, but these are not indexed and hence this would have required a full scan of all stored thread IDs. Ghost messages solve this problem naturally by reusing the exact same thread ID and message ID representation and indexing as regular messages. Furthermore, thanks to this greater symmetry, ghost messages are also algorithmically simpler. We continue to support the old user metadata format, so this patch can't delete any code, but when we do remove support for the old format, several functions can simply be deleted. [1] id:8738h7kv2q.fsf@qmul.ac.uk
2014-10-23 14:30:37 +02:00
notmuch_message_file_t *message_file,
notmuch_bool_t is_ghost)
Change database to store only a single thread ID per message. Instead of supporting multiple thread IDs, we now merge together thread IDs if one message is ever found to belong to more than one thread. This allows for constructing complete threads when, for example, a child message doesn't include a complete list of References headers back to the beginning of the thread. It also simplifies dealing with mapping a message ID to a thread ID which is now a simple get_thread_id just like get_message_id, (and no longer an iterator-based thing like get_tags).
2009-10-25 22:54:13 +01:00
{
lib: Refactor _notmuch_database_link_message This moves the code to retrieve and clear the metadata thread ID out of _notmuch_database_link_message into its own function. This will simplify future changes.
2014-10-07 01:17:08 +02:00
void *local = talloc_new (NULL);
_notmuch_database_linke_message: Fix error-status propagation. The _notmuch_database_link_message_to_parents function was void in an earlier draft. Now, ensure that we don't miss any error return value from it.
2009-10-25 23:01:20 +01:00
notmuch_status_t status;
lib: Implement ghost-based thread linking This updates the thread linking code to use ghost messages instead of user metadata to link messages into threads. In contrast with the old approach, this is actually correct. Previously, thread merging updated only the thread IDs of message documents, not thread IDs stored in user metadata. As originally diagnosed by Mark Walters [1] and as demonstrated by the broken T260-thread-order test, this can cause notmuch to fail to link messages even though they're in the same thread. In principle the old approach could have been fixed by updating the user metadata thread IDs as well, but these are not indexed and hence this would have required a full scan of all stored thread IDs. Ghost messages solve this problem naturally by reusing the exact same thread ID and message ID representation and indexing as regular messages. Furthermore, thanks to this greater symmetry, ghost messages are also algorithmically simpler. We continue to support the old user metadata format, so this patch can't delete any code, but when we do remove support for the old format, several functions can simply be deleted. [1] id:8738h7kv2q.fsf@qmul.ac.uk
2014-10-23 14:30:37 +02:00
const char *thread_id = NULL;
lib: Fix line-wrapping in _notmuch_database_link_message. This function had some excessively long lines due to nested expressions. It's simple enough to un-nest these and have readable line lengths.
2010-04-12 23:41:34 +02:00
lib: Refactor _notmuch_database_link_message This moves the code to retrieve and clear the metadata thread ID out of _notmuch_database_link_message into its own function. This will simplify future changes.
2014-10-07 01:17:08 +02:00
/* Check if the message already had a thread ID */
lib: Implement ghost-based thread linking This updates the thread linking code to use ghost messages instead of user metadata to link messages into threads. In contrast with the old approach, this is actually correct. Previously, thread merging updated only the thread IDs of message documents, not thread IDs stored in user metadata. As originally diagnosed by Mark Walters [1] and as demonstrated by the broken T260-thread-order test, this can cause notmuch to fail to link messages even though they're in the same thread. In principle the old approach could have been fixed by updating the user metadata thread IDs as well, but these are not indexed and hence this would have required a full scan of all stored thread IDs. Ghost messages solve this problem naturally by reusing the exact same thread ID and message ID representation and indexing as regular messages. Furthermore, thanks to this greater symmetry, ghost messages are also algorithmically simpler. We continue to support the old user metadata format, so this patch can't delete any code, but when we do remove support for the old format, several functions can simply be deleted. [1] id:8738h7kv2q.fsf@qmul.ac.uk
2014-10-23 14:30:37 +02:00
if (notmuch->features & NOTMUCH_FEATURE_GHOSTS) {
if (is_ghost)
thread_id = notmuch_message_get_thread_id (message);
} else {
thread_id = _consume_metadata_thread_id (local, notmuch, message);
if (thread_id)
_notmuch_message_add_term (message, "thread", thread_id);
}
Change database to store only a single thread ID per message. Instead of supporting multiple thread IDs, we now merge together thread IDs if one message is ever found to belong to more than one thread. This allows for constructing complete threads when, for example, a child message doesn't include a complete list of References headers back to the beginning of the thread. It also simplifies dealing with mapping a message ID to a thread ID which is now a simple get_thread_id just like get_message_id, (and no longer an iterator-based thing like get_tags).
2009-10-25 22:54:13 +01:00
_notmuch_database_linke_message: Fix error-status propagation. The _notmuch_database_link_message_to_parents function was void in an earlier draft. Now, ensure that we don't miss any error return value from it.
2009-10-25 23:01:20 +01:00
status = _notmuch_database_link_message_to_parents (notmuch, message,
message_file,
&thread_id);
if (status)
lib: Refactor _notmuch_database_link_message This moves the code to retrieve and clear the metadata thread ID out of _notmuch_database_link_message into its own function. This will simplify future changes.
2014-10-07 01:17:08 +02:00
goto DONE;
Change database to store only a single thread ID per message. Instead of supporting multiple thread IDs, we now merge together thread IDs if one message is ever found to belong to more than one thread. This allows for constructing complete threads when, for example, a child message doesn't include a complete list of References headers back to the beginning of the thread. It also simplifies dealing with mapping a message ID to a thread ID which is now a simple get_thread_id just like get_message_id, (and no longer an iterator-based thing like get_tags).
2009-10-25 22:54:13 +01:00
lib: Remove unnecessary thread linking steps when using ghost messages Previously, it was necessary to link new messages to children to work around some (though not all) problems with the old metadata-based approach to stored thread IDs. With ghost messages, this is no longer necessary, so don't bother with child linking when ghost messages are in use.
2014-10-23 14:30:41 +02:00
if (! (notmuch->features & NOTMUCH_FEATURE_GHOSTS)) {
/* In general, it shouldn't be necessary to link children,
* since the earlier indexing of those children will have
* stored a thread ID for the missing parent. However, prior
* to ghost messages, these stored thread IDs were NOT
* rewritten during thread merging (and there was no
* performant way to do so), so if indexed children were
* pulled into a different thread ID by a merge, it was
* necessary to pull them *back* into the stored thread ID of
* the parent. With ghost messages, we just rewrite the
* stored thread IDs during merging, so this workaround isn't
* necessary. */
status = _notmuch_database_link_message_to_children (notmuch, message,
&thread_id);
if (status)
goto DONE;
}
Change database to store only a single thread ID per message. Instead of supporting multiple thread IDs, we now merge together thread IDs if one message is ever found to belong to more than one thread. This allows for constructing complete threads when, for example, a child message doesn't include a complete list of References headers back to the beginning of the thread. It also simplifies dealing with mapping a message ID to a thread ID which is now a simple get_thread_id just like get_message_id, (and no longer an iterator-based thing like get_tags).
2009-10-25 22:54:13 +01:00
Switch from random to sequential thread identifiers. The sequential identifiers have the advantage of being guaranteed to be unique (until we overflow a 64-bit unsigned integer), and also take up half as much space in the "notmuch search" output (16 columns rather than 32). This change also has the side effect of fixing a bug where notmuch could block on /dev/random at startup (waiting for some entropy to appear). This bug was hit hard by the test suite, (which could easily exhaust the available entropy on common systems---resulting in large delays of the test suite).
2010-02-08 20:33:33 +01:00
/* If not part of any existing thread, generate a new thread ID. */
if (thread_id == NULL) {
thread_id = _notmuch_database_generate_thread_id (notmuch);
_notmuch_message_add_term (message, "thread", thread_id);
}
Change database to store only a single thread ID per message. Instead of supporting multiple thread IDs, we now merge together thread IDs if one message is ever found to belong to more than one thread. This allows for constructing complete threads when, for example, a child message doesn't include a complete list of References headers back to the beginning of the thread. It also simplifies dealing with mapping a message ID to a thread ID which is now a simple get_thread_id just like get_message_id, (and no longer an iterator-based thing like get_tags).
2009-10-25 22:54:13 +01:00
lib: Refactor _notmuch_database_link_message This moves the code to retrieve and clear the metadata thread ID out of _notmuch_database_link_message into its own function. This will simplify future changes.
2014-10-07 01:17:08 +02:00
DONE:
talloc_free (local);
return status;
add_message: Pull the thread-stitching portion out into new _notmuch_database_link_message The function was getting too long-winded before. Add since I'm about to change how we handle the thread linking, it's convenient to have it in an isolated function.
2009-10-25 19:03:55 +01:00
}
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
notmuch_status_t
notmuch_database_add_message (notmuch_database_t *notmuch,
add_message: Add an optional parameter for getting the just-added message. We use this to implement the addition of "inbox" and "unread" tags for all messages added by "notmuch new".
2009-10-27 05:44:05 +01:00
const char *filename,
notmuch_message_t **message_ret)
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
{
add_message: Rename message to message_file I still don't like the name message_file at all, but we're about to start using a notmuch_message_t in this function so we need to do something to keep the identifiers separate for now. Eventually, it probably makes sense to push the message-parsing code from database.cc to message.cc.
2009-10-23 14:13:42 +02:00
notmuch_message_file_t *message_file;
add_message: Fix crash for file recognized as not email. This crash was introduced sometime recently, as previously things worked fine when notmuch detected that a file is not an email. We're definitely overdue for that test suite.
2009-11-09 22:43:59 +01:00
notmuch_message_t *message = NULL;
lib: Wrap notmuch_database_add_message in an atomic section. Adding a message may involve changes to multiple database documents, and thus needs to be done in a transaction. This makes add_message (and, I think, the whole library) atomicity-safe: library callers only needs to use atomic sections if they needs atomicity across multiple library calls.
2011-06-11 06:42:58 +02:00
notmuch_status_t ret = NOTMUCH_STATUS_SUCCESS, ret2;
Fix add_message and get_filename to strip/re-add the database path. We now store only a relative path inside the database so the database is not nicely relocatable.
2009-10-29 00:50:14 +01:00
notmuch_private_status_t private_status;
lib: Implement ghost-based thread linking This updates the thread linking code to use ghost messages instead of user metadata to link messages into threads. In contrast with the old approach, this is actually correct. Previously, thread merging updated only the thread IDs of message documents, not thread IDs stored in user metadata. As originally diagnosed by Mark Walters [1] and as demonstrated by the broken T260-thread-order test, this can cause notmuch to fail to link messages even though they're in the same thread. In principle the old approach could have been fixed by updating the user metadata thread IDs as well, but these are not indexed and hence this would have required a full scan of all stored thread IDs. Ghost messages solve this problem naturally by reusing the exact same thread ID and message ID representation and indexing as regular messages. Furthermore, thanks to this greater symmetry, ghost messages are also algorithmically simpler. We continue to support the old user metadata format, so this patch can't delete any code, but when we do remove support for the old format, several functions can simply be deleted. [1] id:8738h7kv2q.fsf@qmul.ac.uk
2014-10-23 14:30:37 +02:00
notmuch_bool_t is_ghost = false;
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
add_message: Pull the thread-stitching portion out into new _notmuch_database_link_message The function was getting too long-winded before. Add since I'm about to change how we handle the thread linking, it's convenient to have it in an isolated function.
2009-10-25 19:03:55 +01:00
const char *date, *header;
Fix add_message and get_filename to strip/re-add the database path. We now store only a relative path inside the database so the database is not nicely relocatable.
2009-10-29 00:50:14 +01:00
const char *from, *to, *subject;
add_message: Use sha-1 in place of overly long message ID. Since Xapian has a limit on the maximum length of a term, we have to check for that before trying to add the message ID as a term. This fixes the bug reported by Mike Hommey here: <20091120132625.GA19246@glandium.org> I've also constructed 20 files with a range of message ID lengths centered around the Xapian term-length limit which I'll use to seed a new test suite soon.
2009-11-22 04:03:49 +01:00
char *message_id = NULL;
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
notmuch_database_add_message: Do not return a message on failure. The recent, disastrous failure of "notmuch new" would have been avoided with this change. The new_command function was basically assuming that it would only get a message object on success so wasn't destroying the message in the other cases.
2009-10-28 00:19:20 +01:00
if (message_ret)
*message_ret = NULL;
lib: Consolidate checks for read-only database. Previously, many checks were deep in the library just before a cast operation. These have now been replaced with internal errors and new checks have instead been added at the beginning of all top-levelentry points requiring a read-write database. The new checks now also use a single function for checking and printing the error message. This will give us a convenient location to extend the check, (such as based on database version as well).
2010-01-07 19:19:44 +01:00
ret = _notmuch_database_ensure_writable (notmuch);
if (ret)
return ret;
lib: eliminate fprintf from _notmuch_message_file_open You may wonder why _notmuch_message_file_open_ctx has two parameters. This is because we need sometime to use a ctx which is a notmuch_message_t. While we could get the database from this, there is no easy way in C to tell type we are getting.
2014-12-26 18:34:49 +01:00
message_file = _notmuch_message_file_open (notmuch, filename);
lib: Consolidate checks for read-only database. Previously, many checks were deep in the library just before a cast operation. These have now been replaced with internal errors and new checks have instead been added at the beginning of all top-levelentry points requiring a read-write database. The new checks now also use a single function for checking and printing the error message. This will give us a convenient location to extend the check, (such as based on database version as well).
2010-01-07 19:19:44 +01:00
if (message_file == NULL)
return NOTMUCH_STATUS_FILE_ERROR;
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
lib: Wrap notmuch_database_add_message in an atomic section. Adding a message may involve changes to multiple database documents, and thus needs to be done in a transaction. This makes add_message (and, I think, the whole library) atomicity-safe: library callers only needs to use atomic sections if they needs atomicity across multiple library calls.
2011-06-11 06:42:58 +02:00
/* Adding a message may change many documents. Do this all
* atomically. */
ret = notmuch_database_begin_atomic (notmuch);
if (ret)
goto DONE;
lib: replace the header parser with gmime The notmuch library includes a full blown message header parser. Yet the same message headers are parsed by gmime during indexing. Switch to gmime parsing completely. These are the main changes: * Gmime stops header parsing at the first invalid header, and presumes the message body starts from there. The current parser is quite liberal in accepting broken headers. The change means we will be much pickier about accepting invalid messages. * The current parser converts tabs used in header folding to spaces. Gmime preserve the tabs. Due to a broken python library used in mailman, there are plenty of mailing lists that produce headers with tabs in header folding, and we'll see plenty of tabs. (This change has been mitigated in preparatory patches.) * For pure header parsing, the current parser is likely faster than gmime, which parses the whole message rather than just the headers. Since we parse the message and its headers using gmime for indexing anyway, this avoids and extra header parsing round when adding new messages. In case of duplicate messages, we'll end up parsing the full message although just headers would be sufficient. All in all this should still speed up 'notmuch new'. * Calls to notmuch_message_get_header() may be slightly slower than previously for headers that are not indexed in the database, due to parsing of the whole message. Within the notmuch code base, notmuch reply is the only such user.
2014-03-30 23:21:49 +02:00
/* Parse message up front to get better error status. */
ret = _notmuch_message_file_parse (message_file);
if (ret)
goto DONE;
Don't hash headers we won't end up using. Just saving a little work here.
2009-10-19 22:48:13 +02:00
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
try {
notmuch_database_add_message: Sanity check the file as the first thing This avoids us wasting a bunch of time doing an expensive SHA-1 over a large file only to discover later that it doesn't even *look* like an email message.
2009-10-28 21:35:10 +01:00
/* Before we do any real work, (especially before doing a
* potential SHA-1 computation on the entire file's contents),
* let's make sure that what we're looking at looks like an
* actual email message.
*/
lib: Start all function names in notmuch-private.h with As noted in devel/STYLE, every private library function should start with _notmuch. This patch corrects function naming that did not adhere to this style in lib/notmuch-private.h. In particular, the old function names that now begin with _notmuch are notmuch_sha1_of_file notmuch_sha1_of_string notmuch_message_file_close notmuch_message_file_get_header notmuch_message_file_open notmuch_message_get_author notmuch_message_set_author Signed-off-by: Charles Celerier <cceleri@cs.stanford.edu>
2014-05-13 11:44:05 +02:00
from = _notmuch_message_file_get_header (message_file, "from");
subject = _notmuch_message_file_get_header (message_file, "subject");
to = _notmuch_message_file_get_header (message_file, "to");
notmuch_database_add_message: Sanity check the file as the first thing This avoids us wasting a bunch of time doing an expensive SHA-1 over a large file only to discover later that it doesn't even *look* like an email message.
2009-10-28 21:35:10 +01:00
add_message: Re-fix handling of non-mail files. More fallout from _get_header now returning "" for missing headers. The bug here is that we would no longer detect that a file is not an email message and give up on it like we should. And this time, I actually audited all callers to notmuch_message_get_header, so hopefully we're done fixing this bug over and over.
2009-11-20 21:46:37 +01:00
if ((from == NULL || *from == '\0') &&
(subject == NULL || *subject == '\0') &&
(to == NULL || *to == '\0'))
notmuch_database_add_message: Sanity check the file as the first thing This avoids us wasting a bunch of time doing an expensive SHA-1 over a large file only to discover later that it doesn't even *look* like an email message.
2009-10-28 21:35:10 +01:00
{
ret = NOTMUCH_STATUS_FILE_NOT_EMAIL;
goto DONE;
}
/* Now that we're sure it's mail, the first order of business
* is to find a message ID (or else create one ourselves). */
add_message: Fix to not add multiple documents with the same message ID Here's the second big fix to message-ID handling, (the first was to generate message IDs when an email contained none). Now, with no document missing a message ID, and no two documents having the same message ID, we have a nice consistent database where the message ID can be used as a unique key.
2009-10-23 15:00:10 +02:00
lib: Start all function names in notmuch-private.h with As noted in devel/STYLE, every private library function should start with _notmuch. This patch corrects function naming that did not adhere to this style in lib/notmuch-private.h. In particular, the old function names that now begin with _notmuch are notmuch_sha1_of_file notmuch_sha1_of_string notmuch_message_file_close notmuch_message_file_get_header notmuch_message_file_open notmuch_message_get_author notmuch_message_set_author Signed-off-by: Charles Celerier <cceleri@cs.stanford.edu>
2014-05-13 11:44:05 +02:00
header = _notmuch_message_file_get_header (message_file, "message-id");
add_message: Properly handle missing Message-ID once again. There's been a fair amount of fallout from when we changed message_file_get_header from returning NULL to returning "" for missing headers. This is yet more fallout from that, (where we were accepting an empty message-ID rather than generating one like we want to).
2009-11-20 19:31:00 +01:00
if (header && *header != '\0') {
Export _parse_message_id to the library implementation. Not exported through the public interface, but the thread code is going to want to be able to parse In-Reply-To headers so needs access to this code.
2009-11-16 05:21:43 +01:00
message_id = _parse_message_id (message_file, header, NULL);
add_message: Use sha-1 in place of overly long message ID. Since Xapian has a limit on the maximum length of a term, we have to check for that before trying to add the message ID as a term. This fixes the bug reported by Mike Hommey here: <20091120132625.GA19246@glandium.org> I've also constructed 20 files with a range of message ID lengths centered around the Xapian term-length limit which I'll use to seed a new test suite soon.
2009-11-22 04:03:49 +01:00
notmuch: Switch from gmime to custom, ad-hoc parsing of headers. Since we're currently just trying to stitch together In-Reply-To and References headers we don't need that much sophistication. It's when we later add full-text searching that GMime will be useful. So for now, even though my own code here is surely very buggy compared to GMime it's also a lot faster. And speed is what we're after for the initial index creation.
2009-10-19 21:54:40 +02:00
/* So the header value isn't RFC-compliant, but it's
* better than no message-id at all. */
if (message_id == NULL)
Remove all calls to g_strdup_printf Replacing them with calls to talloc_asprintf if possible, otherwise to asprintf (with it's painful error-handling leaving the pointer undefined).
2009-10-26 23:17:10 +01:00
message_id = talloc_strdup (message_file, header);
add_message: Use sha-1 in place of overly long message ID. Since Xapian has a limit on the maximum length of a term, we have to check for that before trying to add the message ID as a term. This fixes the bug reported by Mike Hommey here: <20091120132625.GA19246@glandium.org> I've also constructed 20 files with a range of message ID lengths centered around the Xapian term-length limit which I'll use to seed a new test suite soon.
2009-11-22 04:03:49 +01:00
}
if (message_id == NULL ) {
Generate message ID (using SHA1) when a mail message contains none. This is important as we're using the message ID as the unique key in our database. So previously, all messages with no message ID would be treated as the same message---not good at all.
2009-10-23 00:31:56 +02:00
/* No message-id at all, let's generate one by taking a
* hash over the file's contents. */
lib: Start all function names in notmuch-private.h with As noted in devel/STYLE, every private library function should start with _notmuch. This patch corrects function naming that did not adhere to this style in lib/notmuch-private.h. In particular, the old function names that now begin with _notmuch are notmuch_sha1_of_file notmuch_sha1_of_string notmuch_message_file_close notmuch_message_file_get_header notmuch_message_file_open notmuch_message_get_author notmuch_message_set_author Signed-off-by: Charles Celerier <cceleri@cs.stanford.edu>
2014-05-13 11:44:05 +02:00
char *sha1 = _notmuch_sha1_of_file (filename);
Generate message ID (using SHA1) when a mail message contains none. This is important as we're using the message ID as the unique key in our database. So previously, all messages with no message ID would be treated as the same message---not good at all.
2009-10-23 00:31:56 +02:00
/* If that failed too, something is really wrong. Give up. */
if (sha1 == NULL) {
ret = NOTMUCH_STATUS_FILE_ERROR;
goto DONE;
}
Remove all calls to g_strdup_printf Replacing them with calls to talloc_asprintf if possible, otherwise to asprintf (with it's painful error-handling leaving the pointer undefined).
2009-10-26 23:17:10 +01:00
message_id = talloc_asprintf (message_file,
"notmuch-sha1-%s", sha1);
Generate message ID (using SHA1) when a mail message contains none. This is important as we're using the message ID as the unique key in our database. So previously, all messages with no message ID would be treated as the same message---not good at all.
2009-10-23 00:31:56 +02:00
free (sha1);
notmuch: Switch from gmime to custom, ad-hoc parsing of headers. Since we're currently just trying to stitch together In-Reply-To and References headers we don't need that much sophistication. It's when we later add full-text searching that GMime will be useful. So for now, even though my own code here is surely very buggy compared to GMime it's also a lot faster. And speed is what we're after for the initial index creation.
2009-10-19 21:54:40 +02:00
}
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
add_message: Fix to not add multiple documents with the same message ID Here's the second big fix to message-ID handling, (the first was to generate message IDs when an email contained none). Now, with no document missing a message ID, and no two documents having the same message ID, we have a nice consistent database where the message ID can be used as a unique key.
2009-10-23 15:00:10 +02:00
/* Now that we have a message ID, we get a message object,
* (which may or may not reference an existing document in the
* database). */
Fix broken commit. Oops. I should have actually compiled before pushing.
2009-11-17 18:04:14 +01:00
message = _notmuch_message_create_for_message_id (notmuch,
add_message: Propagate error status from notmuch_message_create_for_message_id What a great feeling to remove an XXX comment.
2009-10-25 17:47:21 +01:00
message_id,
Fix add_message and get_filename to strip/re-add the database path. We now store only a relative path inside the database so the database is not nicely relocatable.
2009-10-29 00:50:14 +01:00
&private_status);
Remove all calls to g_strdup_printf Replacing them with calls to talloc_asprintf if possible, otherwise to asprintf (with it's painful error-handling leaving the pointer undefined).
2009-10-26 23:17:10 +01:00
talloc_free (message_id);
add_message: Pull the thread-stitching portion out into new _notmuch_database_link_message The function was getting too long-winded before. Add since I'm about to change how we handle the thread linking, it's convenient to have it in an isolated function.
2009-10-25 19:03:55 +01:00
notmuch_database_add_message: Add missing error-value propagation. Thanks to Mike Hommey for doing the analysis that led to noticing that this was missing.
2009-11-20 21:02:11 +01:00
if (message == NULL) {
ret = COERCE_STATUS (private_status,
"Unexpected status value from _notmuch_message_create_for_message_id");
add_message: Propagate error status from notmuch_message_create_for_message_id What a great feeling to remove an XXX comment.
2009-10-25 17:47:21 +01:00
goto DONE;
notmuch_database_add_message: Add missing error-value propagation. Thanks to Mike Hommey for doing the analysis that led to noticing that this was missing.
2009-11-20 21:02:11 +01:00
}
add_message: Re-order the code a bit (find message-id first). We're preparing for being able to deal with files with duplicate message IDs here. The plan is to create a notmuch_message_t object in add_message that may or may not reference a document that exists in the database. So to do this, we have to find the message ID before we do any manipulation of the doc.
2009-10-23 14:30:37 +02:00
database: Allowing storing multiple filenames for a single message ID. The library interface is unchanged so far, (still just notmuch_database_add_message), but internally, the old _set_filename function is now _add_filename instead.
2009-12-21 21:08:46 +01:00
_notmuch_message_add_filename (message, filename);
lib: Implement ghost-based thread linking This updates the thread linking code to use ghost messages instead of user metadata to link messages into threads. In contrast with the old approach, this is actually correct. Previously, thread merging updated only the thread IDs of message documents, not thread IDs stored in user metadata. As originally diagnosed by Mark Walters [1] and as demonstrated by the broken T260-thread-order test, this can cause notmuch to fail to link messages even though they're in the same thread. In principle the old approach could have been fixed by updating the user metadata thread IDs as well, but these are not indexed and hence this would have required a full scan of all stored thread IDs. Ghost messages solve this problem naturally by reusing the exact same thread ID and message ID representation and indexing as regular messages. Furthermore, thanks to this greater symmetry, ghost messages are also algorithmically simpler. We continue to support the old user metadata format, so this patch can't delete any code, but when we do remove support for the old format, several functions can simply be deleted. [1] id:8738h7kv2q.fsf@qmul.ac.uk
2014-10-23 14:30:37 +02:00
/* Is this a newly created message object or a ghost
* message? We have to be slightly careful: if this is a
* blank message, it's not safe to call
* notmuch_message_get_flag yet. */
if (private_status == NOTMUCH_PRIVATE_STATUS_NO_DOCUMENT_FOUND ||
(is_ghost = notmuch_message_get_flag (
message, NOTMUCH_MESSAGE_FLAG_GHOST))) {
add_message: Fix to not add multiple documents with the same message ID Here's the second big fix to message-ID handling, (the first was to generate message IDs when an email contained none). Now, with no document missing a message ID, and no two documents having the same message ID, we have a nice consistent database where the message ID can be used as a unique key.
2009-10-23 15:00:10 +02:00
_notmuch_message_add_term (message, "type", "mail");
lib: Implement ghost-based thread linking This updates the thread linking code to use ghost messages instead of user metadata to link messages into threads. In contrast with the old approach, this is actually correct. Previously, thread merging updated only the thread IDs of message documents, not thread IDs stored in user metadata. As originally diagnosed by Mark Walters [1] and as demonstrated by the broken T260-thread-order test, this can cause notmuch to fail to link messages even though they're in the same thread. In principle the old approach could have been fixed by updating the user metadata thread IDs as well, but these are not indexed and hence this would have required a full scan of all stored thread IDs. Ghost messages solve this problem naturally by reusing the exact same thread ID and message ID representation and indexing as regular messages. Furthermore, thanks to this greater symmetry, ghost messages are also algorithmically simpler. We continue to support the old user metadata format, so this patch can't delete any code, but when we do remove support for the old format, several functions can simply be deleted. [1] id:8738h7kv2q.fsf@qmul.ac.uk
2014-10-23 14:30:37 +02:00
if (is_ghost)
/* Convert ghost message to a regular message */
_notmuch_message_remove_term (message, "type", "ghost");
add_message: Re-order the code a bit (find message-id first). We're preparing for being able to deal with files with duplicate message IDs here. The plan is to create a notmuch_message_t object in add_message that may or may not reference a document that exists in the database. So to do this, we have to find the message ID before we do any manipulation of the doc.
2009-10-23 14:30:37 +02:00
database: Allowing storing multiple filenames for a single message ID. The library interface is unchanged so far, (still just notmuch_database_add_message), but internally, the old _set_filename function is now _add_filename instead.
2009-12-21 21:08:46 +01:00
ret = _notmuch_database_link_message (notmuch, message,
lib: Implement ghost-based thread linking This updates the thread linking code to use ghost messages instead of user metadata to link messages into threads. In contrast with the old approach, this is actually correct. Previously, thread merging updated only the thread IDs of message documents, not thread IDs stored in user metadata. As originally diagnosed by Mark Walters [1] and as demonstrated by the broken T260-thread-order test, this can cause notmuch to fail to link messages even though they're in the same thread. In principle the old approach could have been fixed by updating the user metadata thread IDs as well, but these are not indexed and hence this would have required a full scan of all stored thread IDs. Ghost messages solve this problem naturally by reusing the exact same thread ID and message ID representation and indexing as regular messages. Furthermore, thanks to this greater symmetry, ghost messages are also algorithmically simpler. We continue to support the old user metadata format, so this patch can't delete any code, but when we do remove support for the old format, several functions can simply be deleted. [1] id:8738h7kv2q.fsf@qmul.ac.uk
2014-10-23 14:30:37 +02:00
message_file, is_ghost);
database: Allowing storing multiple filenames for a single message ID. The library interface is unchanged so far, (still just notmuch_database_add_message), but internally, the old _set_filename function is now _add_filename instead.
2009-12-21 21:08:46 +01:00
if (ret)
goto DONE;
add_message: Fix memory leak of thread_ids GPtrArray. We were properly feeing this memory when the thread-ids list was not empty, but leaking it when it was. Thanks, of course, to valgrind along with the G_SLICE=always-malloc environment variable which makes leak checking with glib almost bearable.
2009-10-20 22:05:45 +02:00
lib: Start all function names in notmuch-private.h with As noted in devel/STYLE, every private library function should start with _notmuch. This patch corrects function naming that did not adhere to this style in lib/notmuch-private.h. In particular, the old function names that now begin with _notmuch are notmuch_sha1_of_file notmuch_sha1_of_string notmuch_message_file_close notmuch_message_file_get_header notmuch_message_file_open notmuch_message_get_author notmuch_message_set_author Signed-off-by: Charles Celerier <cceleri@cs.stanford.edu>
2014-05-13 11:44:05 +02:00
date = _notmuch_message_file_get_header (message_file, "date");
Store "from" and "subject" headers in the database. This is a rebase and cleanup of Istvan Marko's patch from id:m3pqnj2j7a.fsf@zsu.kismala.com Search retrieves these headers for every message in the search results. Previously, this required opening and parsing every message file. Storing them directly in the database significantly reduces IO and computation, speeding up search by between 50% and 10X. Taking full advantage of this requires a database rebuild, but it will fall back to the old behavior for messages that do not have headers stored in the database.
2011-11-06 18:17:36 +01:00
_notmuch_message_set_header_values (message, date, from, subject);
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
lib: replace the header parser with gmime The notmuch library includes a full blown message header parser. Yet the same message headers are parsed by gmime during indexing. Switch to gmime parsing completely. These are the main changes: * Gmime stops header parsing at the first invalid header, and presumes the message body starts from there. The current parser is quite liberal in accepting broken headers. The change means we will be much pickier about accepting invalid messages. * The current parser converts tabs used in header folding to spaces. Gmime preserve the tabs. Due to a broken python library used in mailman, there are plenty of mailing lists that produce headers with tabs in header folding, and we'll see plenty of tabs. (This change has been mitigated in preparatory patches.) * For pure header parsing, the current parser is likely faster than gmime, which parses the whole message rather than just the headers. Since we parse the message and its headers using gmime for indexing anyway, this avoids and extra header parsing round when adding new messages. In case of duplicate messages, we'll end up parsing the full message although just headers would be sufficient. All in all this should still speed up 'notmuch new'. * Calls to notmuch_message_get_header() may be slightly slower than previously for headers that are not indexed in the database, due to parsing of the whole message. Within the notmuch code base, notmuch reply is the only such user.
2014-03-30 23:21:49 +02:00
ret = _notmuch_message_index_file (message, message_file);
lib: Reject multi-message mboxes and deprecate single-message mbox Previously, we would treat multi-message mboxes as one giant email, which, besides the obvious incorrect indexing, often led to out-of-memory errors for archival mboxes. Now we explicitly reject multi-message mboxes. For historical reasons, we retain support for single-message mboxes, but official deprecate this behavior.
2012-11-25 07:16:01 +01:00
if (ret)
goto DONE;
database: Allowing storing multiple filenames for a single message ID. The library interface is unchanged so far, (still just notmuch_database_add_message), but internally, the old _set_filename function is now _add_filename instead.
2009-12-21 21:08:46 +01:00
} else {
ret = NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID;
}
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
_notmuch_message_sync (message);
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
} catch (const Xapian::Error &error) {
lib: replace almost all fprintfs in library with _n_d_log This is not supposed to change any functionality from an end user point of view. Note that it will eliminate some output to stderr. The query debugging output is left as is; it doesn't really fit with the current primitive logging model. The remaining "bad" fprintf will need an internal API change.
2014-12-26 17:25:35 +01:00
_notmuch_database_log (notmuch, "A Xapian exception occurred adding message: %s.\n",
lib: Audit all notmuch_database call for Xapian exception handling. Our current approach is for top-level entry poitns in the library to have try/catch blocks that catch any Xapian exception and print a message. Add a few missing blocks and fix up the documentation.
2010-04-24 16:22:34 +02:00
error.get_msg().c_str());
Catch and optionally print about exception at database->flush. If an earlier exception occurred, then it's not unexpected for the flush to fail as well. So in that case, we'll silently catch the exception. Otherwise, make some noise about things going wrong at the time of flush.
2009-11-22 03:54:20 +01:00
notmuch->exception_reported = TRUE;
Generate message ID (using SHA1) when a mail message contains none. This is important as we're using the message ID as the unique key in our database. So previously, all messages with no message ID would be treated as the same message---not good at all.
2009-10-23 00:31:56 +02:00
ret = NOTMUCH_STATUS_XAPIAN_EXCEPTION;
goto DONE;
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
}
Generate message ID (using SHA1) when a mail message contains none. This is important as we're using the message ID as the unique key in our database. So previously, all messages with no message ID would be treated as the same message---not good at all.
2009-10-23 00:31:56 +02:00
DONE:
add_message: Add an optional parameter for getting the just-added message. We use this to implement the addition of "inbox" and "unread" tags for all messages added by "notmuch new".
2009-10-27 05:44:05 +01:00
if (message) {
lib: Return added message even if it already was in the database
2010-10-31 22:29:15 +01:00
if ((ret == NOTMUCH_STATUS_SUCCESS ||
ret == NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID) && message_ret)
add_message: Add an optional parameter for getting the just-added message. We use this to implement the addition of "inbox" and "unread" tags for all messages added by "notmuch new".
2009-10-27 05:44:05 +01:00
*message_ret = message;
else
notmuch_message_destroy (message);
}
add_message: Rename message to message_file I still don't like the name message_file at all, but we're about to start using a notmuch_message_t in this function so we need to do something to keep the identifiers separate for now. Eventually, it probably makes sense to push the message-parsing code from database.cc to message.cc.
2009-10-23 14:13:42 +02:00
if (message_file)
lib: Start all function names in notmuch-private.h with As noted in devel/STYLE, every private library function should start with _notmuch. This patch corrects function naming that did not adhere to this style in lib/notmuch-private.h. In particular, the old function names that now begin with _notmuch are notmuch_sha1_of_file notmuch_sha1_of_string notmuch_message_file_close notmuch_message_file_get_header notmuch_message_file_open notmuch_message_get_author notmuch_message_set_author Signed-off-by: Charles Celerier <cceleri@cs.stanford.edu>
2014-05-13 11:44:05 +02:00
_notmuch_message_file_close (message_file);
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
lib: Wrap notmuch_database_add_message in an atomic section. Adding a message may involve changes to multiple database documents, and thus needs to be done in a transaction. This makes add_message (and, I think, the whole library) atomicity-safe: library callers only needs to use atomic sections if they needs atomicity across multiple library calls.
2011-06-11 06:42:58 +02:00
ret2 = notmuch_database_end_atomic (notmuch);
if ((ret == NOTMUCH_STATUS_SUCCESS ||
ret == NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID) &&
ret2 != NOTMUCH_STATUS_SUCCESS)
ret = ret2;
Generate message ID (using SHA1) when a mail message contains none. This is important as we're using the message ID as the unique key in our database. So previously, all messages with no message ID would be treated as the same message---not good at all.
2009-10-23 00:31:56 +02:00
return ret;
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
}
notmuch: New function to retrieve all tags from the database. This patch adds a new function called notmuch_database_get_all_tags which can be used to obtain a list of all tags from the database (in other words, the list contains all tags from all messages). The function produces an alphabetically sorted list. To add support for the new function, we rip the guts off of notmuch_message_get_tags and put them in a new generic function called _notmuch_convert_tags. The generic function takes a Xapian::TermIterator as argument and uses the iterator to find tags. This makes the function usable with different Xapian objects. Function notmuch_message_get_tags is then reimplemented to call the generic function with message->doc.termlist_begin() as argument. Similarly, we implement notmuch_message_database_get_all_tags, the function calls the generic function with db->xapian_db->allterms_begin() as argument. Finally, notmuch_database_get_all_tags is exported through lib/notmuch.h Signed-off-by: Jan Janak <jan@ryngle.com>
2009-11-23 01:10:54 +01:00
database: Add new, public notmuch_database_remove_message This will allow applications to support the removal of messages, (such as when a file is deleted from the mail store). No removal support is provided yet in commands such as "notmuch new".
2009-12-22 00:14:32 +01:00
notmuch_status_t
notmuch_database_remove_message (notmuch_database_t *notmuch,
const char *filename)
lib: Add an API to find a message by filename. notmuch_database_find_message_by_filename is mostly stolen from notmuch_database_remove_message, so this patch also vastly simplfies the latter using the former. This API is also useful in its own right and will be used in a later patch for eager maildir flag synchronization.
2011-06-11 06:19:31 +02:00
{
lib: make find_message{,by_filename) report errors Previously, the functions notmuch_database_find_message() and notmuch_database_find_message_by_filename() functions did not properly report error condition to the library user. For more information, read the thread on the notmuch mailing list starting with my mail "id:871uv2unfd.fsf@gmail.com" Make these functions accept a pointer to 'notmuch_message_t' as argument and return notmuch_status_t which may be used to check for any error condition. restore: Modify for the new notmuch_database_find_message() new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 06:55:29 +02:00
notmuch_status_t status;
notmuch_message_t *message;
lib: Add an API to find a message by filename. notmuch_database_find_message_by_filename is mostly stolen from notmuch_database_remove_message, so this patch also vastly simplfies the latter using the former. This API is also useful in its own right and will be used in a later patch for eager maildir flag synchronization.
2011-06-11 06:19:31 +02:00
lib: make find_message{,by_filename) report errors Previously, the functions notmuch_database_find_message() and notmuch_database_find_message_by_filename() functions did not properly report error condition to the library user. For more information, read the thread on the notmuch mailing list starting with my mail "id:871uv2unfd.fsf@gmail.com" Make these functions accept a pointer to 'notmuch_message_t' as argument and return notmuch_status_t which may be used to check for any error condition. restore: Modify for the new notmuch_database_find_message() new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 06:55:29 +02:00
status = notmuch_database_find_message_by_filename (notmuch, filename,
&message);
if (status == NOTMUCH_STATUS_SUCCESS && message) {
lib: Add an API to find a message by filename. notmuch_database_find_message_by_filename is mostly stolen from notmuch_database_remove_message, so this patch also vastly simplfies the latter using the former. This API is also useful in its own right and will be used in a later patch for eager maildir flag synchronization.
2011-06-11 06:19:31 +02:00
status = _notmuch_message_remove_filename (message, filename);
if (status == NOTMUCH_STATUS_SUCCESS)
_notmuch_message_delete (message);
else if (status == NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID)
_notmuch_message_sync (message);
lib: destroy message object after message removal notmuch_database_remove_message() must call notmuch_message_destroy() once it is done handling message removal.
2011-10-03 22:27:32 +02:00
notmuch_message_destroy (message);
lib: Add an API to find a message by filename. notmuch_database_find_message_by_filename is mostly stolen from notmuch_database_remove_message, so this patch also vastly simplfies the latter using the former. This API is also useful in its own right and will be used in a later patch for eager maildir flag synchronization.
2011-06-11 06:19:31 +02:00
}
return status;
}
lib: make find_message{,by_filename) report errors Previously, the functions notmuch_database_find_message() and notmuch_database_find_message_by_filename() functions did not properly report error condition to the library user. For more information, read the thread on the notmuch mailing list starting with my mail "id:871uv2unfd.fsf@gmail.com" Make these functions accept a pointer to 'notmuch_message_t' as argument and return notmuch_status_t which may be used to check for any error condition. restore: Modify for the new notmuch_database_find_message() new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 06:55:29 +02:00
notmuch_status_t
lib: Add an API to find a message by filename. notmuch_database_find_message_by_filename is mostly stolen from notmuch_database_remove_message, so this patch also vastly simplfies the latter using the former. This API is also useful in its own right and will be used in a later patch for eager maildir flag synchronization.
2011-06-11 06:19:31 +02:00
notmuch_database_find_message_by_filename (notmuch_database_t *notmuch,
lib: make find_message{,by_filename) report errors Previously, the functions notmuch_database_find_message() and notmuch_database_find_message_by_filename() functions did not properly report error condition to the library user. For more information, read the thread on the notmuch mailing list starting with my mail "id:871uv2unfd.fsf@gmail.com" Make these functions accept a pointer to 'notmuch_message_t' as argument and return notmuch_status_t which may be used to check for any error condition. restore: Modify for the new notmuch_database_find_message() new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 06:55:29 +02:00
const char *filename,
notmuch_message_t **message_ret)
database: Add new, public notmuch_database_remove_message This will allow applications to support the removal of messages, (such as when a file is deleted from the mail store). No removal support is provided yet in commands such as "notmuch new".
2009-12-22 00:14:32 +01:00
{
lib: Audit all notmuch_database call for Xapian exception handling. Our current approach is for top-level entry poitns in the library to have try/catch blocks that catch any Xapian exception and print a message. Add a few missing blocks and fix up the documentation.
2010-04-24 16:22:34 +02:00
void *local;
lib: Implement new notmuch_directory_t API. This new directory ojbect provides all the infrastructure needed to detect when files or directories are deleted or renamed. There's still code needed on top of this (within "notmuch new") to actually do that detection.
2010-01-05 22:29:23 +01:00
const char *prefix = _find_prefix ("file-direntry");
database: Add new, public notmuch_database_remove_message This will allow applications to support the removal of messages, (such as when a file is deleted from the mail store). No removal support is provided yet in commands such as "notmuch new".
2009-12-22 00:14:32 +01:00
char *direntry, *term;
Xapian::PostingIterator i, end;
notmuch_status_t status;
lib: make find_message{,by_filename) report errors Previously, the functions notmuch_database_find_message() and notmuch_database_find_message_by_filename() functions did not properly report error condition to the library user. For more information, read the thread on the notmuch mailing list starting with my mail "id:871uv2unfd.fsf@gmail.com" Make these functions accept a pointer to 'notmuch_message_t' as argument and return notmuch_status_t which may be used to check for any error condition. restore: Modify for the new notmuch_database_find_message() new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 06:55:29 +02:00
if (message_ret == NULL)
return NOTMUCH_STATUS_NULL_POINTER;
lib: Return an error from operations that require an upgrade Previously, there was no protection against a caller invoking an operation on an old database version that would effectively corrupt the database by treating it like a newer version. According to notmuch.h, any caller that opens the database in read/write mode is supposed to check if the database needs upgrading and perform an upgrade if it does. This would protect against this, but nobody (even the CLI) actually does this. However, with features, it's easy to protect against incompatible operations on a fine-grained basis. This lightweight change allows callers to safely operate on old database versions, while preventing specific operations that would corrupt the database with an informative error message.
2014-08-25 19:26:08 +02:00
if (! (notmuch->features & NOTMUCH_FEATURE_FILE_TERMS))
return NOTMUCH_STATUS_UPGRADE_REQUIRED;
Fix error reporting in notmuch_database_find_message_by_filename Formerly it was possible for *message_ret to be left uninitialized. The documentation however clearly states that "[o]n any failure or when the message is not found, this function initializes '*message' to NULL". Signed-off-by: Justus Winter <4winter@informatik.uni-hamburg.de>
2012-03-17 17:41:27 +01:00
/* return NULL on any failure */
*message_ret = NULL;
lib: Audit all notmuch_database call for Xapian exception handling. Our current approach is for top-level entry poitns in the library to have try/catch blocks that catch any Xapian exception and print a message. Add a few missing blocks and fix up the documentation.
2010-04-24 16:22:34 +02:00
local = talloc_new (notmuch);
try {
lib: Perform the same transformation to _notmuch_database_filename_to_direntry Now _notmuch_database_filename_to_direntry takes a flags argument and can indicate if the necessary directory documents do not exist. Again, callers have been updated, but retain their original behavior.
2012-05-18 06:13:36 +02:00
status = _notmuch_database_filename_to_direntry (
lib: Make notmuch_database_find_message_by_filename not crash on read-only databases Previously, _notmuch_database_filename_to_direntry would abort with an internal error when called on a read-only database. Now that creating the directory document is optional, notmuch_database_find_message_by_filename can disable directory document creation (as it should) and, as a result, not abort on read-only databases.
2012-05-18 06:13:40 +02:00
local, notmuch, filename, NOTMUCH_FIND_LOOKUP, &direntry);
if (status || !direntry)
lib: make find_message{,by_filename) report errors Previously, the functions notmuch_database_find_message() and notmuch_database_find_message_by_filename() functions did not properly report error condition to the library user. For more information, read the thread on the notmuch mailing list starting with my mail "id:871uv2unfd.fsf@gmail.com" Make these functions accept a pointer to 'notmuch_message_t' as argument and return notmuch_status_t which may be used to check for any error condition. restore: Modify for the new notmuch_database_find_message() new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 06:55:29 +02:00
goto DONE;
database: Add new, public notmuch_database_remove_message This will allow applications to support the removal of messages, (such as when a file is deleted from the mail store). No removal support is provided yet in commands such as "notmuch new".
2009-12-22 00:14:32 +01:00
Tighten up a memory allocation. Using the local talloc context ensures that the memory we are using here will be freed shortly, (rather than hanging on for a long time with the notmuch database object).
2011-01-15 23:01:43 +01:00
term = talloc_asprintf (local, "%s%s", prefix, direntry);
database: Add new, public notmuch_database_remove_message This will allow applications to support the removal of messages, (such as when a file is deleted from the mail store). No removal support is provided yet in commands such as "notmuch new".
2009-12-22 00:14:32 +01:00
lib: Audit all notmuch_database call for Xapian exception handling. Our current approach is for top-level entry poitns in the library to have try/catch blocks that catch any Xapian exception and print a message. Add a few missing blocks and fix up the documentation.
2010-04-24 16:22:34 +02:00
find_doc_ids_for_term (notmuch, term, &i, &end);
database: Add new, public notmuch_database_remove_message This will allow applications to support the removal of messages, (such as when a file is deleted from the mail store). No removal support is provided yet in commands such as "notmuch new".
2009-12-22 00:14:32 +01:00
lib: Add an API to find a message by filename. notmuch_database_find_message_by_filename is mostly stolen from notmuch_database_remove_message, so this patch also vastly simplfies the latter using the former. This API is also useful in its own right and will be used in a later patch for eager maildir flag synchronization.
2011-06-11 06:19:31 +02:00
if (i != end) {
Add support for folder-based searching. A new "folder:" prefix in the query string can now be used to match the directories in which mail files are stored. The addition of this feature causes the recently added search-by-folder tests to now pass.
2011-01-15 23:09:04 +01:00
notmuch_private_status_t private_status;
database: Add new, public notmuch_database_remove_message This will allow applications to support the removal of messages, (such as when a file is deleted from the mail store). No removal support is provided yet in commands such as "notmuch new".
2009-12-22 00:14:32 +01:00
lib: make find_message{,by_filename) report errors Previously, the functions notmuch_database_find_message() and notmuch_database_find_message_by_filename() functions did not properly report error condition to the library user. For more information, read the thread on the notmuch mailing list starting with my mail "id:871uv2unfd.fsf@gmail.com" Make these functions accept a pointer to 'notmuch_message_t' as argument and return notmuch_status_t which may be used to check for any error condition. restore: Modify for the new notmuch_database_find_message() new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 06:55:29 +02:00
*message_ret = _notmuch_message_create (notmuch, notmuch, *i,
&private_status);
if (*message_ret == NULL)
status = NOTMUCH_STATUS_OUT_OF_MEMORY;
database: Add new, public notmuch_database_remove_message This will allow applications to support the removal of messages, (such as when a file is deleted from the mail store). No removal support is provided yet in commands such as "notmuch new".
2009-12-22 00:14:32 +01:00
}
lib: Audit all notmuch_database call for Xapian exception handling. Our current approach is for top-level entry poitns in the library to have try/catch blocks that catch any Xapian exception and print a message. Add a few missing blocks and fix up the documentation.
2010-04-24 16:22:34 +02:00
} catch (const Xapian::Error &error) {
lib: replace almost all fprintfs in library with _n_d_log This is not supposed to change any functionality from an end user point of view. Note that it will eliminate some output to stderr. The query debugging output is left as is; it doesn't really fit with the current primitive logging model. The remaining "bad" fprintf will need an internal API change.
2014-12-26 17:25:35 +01:00
_notmuch_database_log (notmuch, "Error: A Xapian exception occurred finding message by filename: %s\n",
lib: Audit all notmuch_database call for Xapian exception handling. Our current approach is for top-level entry poitns in the library to have try/catch blocks that catch any Xapian exception and print a message. Add a few missing blocks and fix up the documentation.
2010-04-24 16:22:34 +02:00
error.get_msg().c_str());
notmuch->exception_reported = TRUE;
lib: make find_message{,by_filename) report errors Previously, the functions notmuch_database_find_message() and notmuch_database_find_message_by_filename() functions did not properly report error condition to the library user. For more information, read the thread on the notmuch mailing list starting with my mail "id:871uv2unfd.fsf@gmail.com" Make these functions accept a pointer to 'notmuch_message_t' as argument and return notmuch_status_t which may be used to check for any error condition. restore: Modify for the new notmuch_database_find_message() new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 06:55:29 +02:00
status = NOTMUCH_STATUS_XAPIAN_EXCEPTION;
database: Add new, public notmuch_database_remove_message This will allow applications to support the removal of messages, (such as when a file is deleted from the mail store). No removal support is provided yet in commands such as "notmuch new".
2009-12-22 00:14:32 +01:00
}
lib: make find_message{,by_filename) report errors Previously, the functions notmuch_database_find_message() and notmuch_database_find_message_by_filename() functions did not properly report error condition to the library user. For more information, read the thread on the notmuch mailing list starting with my mail "id:871uv2unfd.fsf@gmail.com" Make these functions accept a pointer to 'notmuch_message_t' as argument and return notmuch_status_t which may be used to check for any error condition. restore: Modify for the new notmuch_database_find_message() new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 06:55:29 +02:00
DONE:
database: Add new, public notmuch_database_remove_message This will allow applications to support the removal of messages, (such as when a file is deleted from the mail store). No removal support is provided yet in commands such as "notmuch new".
2009-12-22 00:14:32 +01:00
talloc_free (local);
lib: make find_message{,by_filename) report errors Previously, the functions notmuch_database_find_message() and notmuch_database_find_message_by_filename() functions did not properly report error condition to the library user. For more information, read the thread on the notmuch mailing list starting with my mail "id:871uv2unfd.fsf@gmail.com" Make these functions accept a pointer to 'notmuch_message_t' as argument and return notmuch_status_t which may be used to check for any error condition. restore: Modify for the new notmuch_database_find_message() new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 06:55:29 +02:00
if (status && *message_ret) {
notmuch_message_destroy (*message_ret);
*message_ret = NULL;
}
return status;
database: Add new, public notmuch_database_remove_message This will allow applications to support the removal of messages, (such as when a file is deleted from the mail store). No removal support is provided yet in commands such as "notmuch new".
2009-12-22 00:14:32 +01:00
}
Add a generic function to get a list of terms with some prefix. Replace _notmuch_convert_tags with this and simplify _create_filenames_for_terms_with_prefix. This will also come in handy shortly to get the message file name list.
2010-12-09 06:32:35 +01:00
notmuch_string_list_t *
_notmuch_database_get_terms_with_prefix (void *ctx, Xapian::TermIterator &i,
Xapian::TermIterator &end,
const char *prefix)
notmuch: New function to retrieve all tags from the database. This patch adds a new function called notmuch_database_get_all_tags which can be used to obtain a list of all tags from the database (in other words, the list contains all tags from all messages). The function produces an alphabetically sorted list. To add support for the new function, we rip the guts off of notmuch_message_get_tags and put them in a new generic function called _notmuch_convert_tags. The generic function takes a Xapian::TermIterator as argument and uses the iterator to find tags. This makes the function usable with different Xapian objects. Function notmuch_message_get_tags is then reimplemented to call the generic function with message->doc.termlist_begin() as argument. Similarly, we implement notmuch_message_database_get_all_tags, the function calls the generic function with db->xapian_db->allterms_begin() as argument. Finally, notmuch_database_get_all_tags is exported through lib/notmuch.h Signed-off-by: Jan Janak <jan@ryngle.com>
2009-11-23 01:10:54 +01:00
{
Add a generic function to get a list of terms with some prefix. Replace _notmuch_convert_tags with this and simplify _create_filenames_for_terms_with_prefix. This will also come in handy shortly to get the message file name list.
2010-12-09 06:32:35 +01:00
int prefix_len = strlen (prefix);
Implement an internal generic string list and use it. This replaces the guts of the filename list and tag list, making those interfaces simple iterators over the generic string list. The directory, message filename, and tags-related code now build generic string lists and then wraps them in specific iterators. The real wins come in later patches, when we use these for even more generic functionality. As a nice side-effect, this also eliminates the annoying dependency on GList in the tag list.
2010-12-09 01:26:05 +01:00
notmuch_string_list_t *list;
notmuch: New function to retrieve all tags from the database. This patch adds a new function called notmuch_database_get_all_tags which can be used to obtain a list of all tags from the database (in other words, the list contains all tags from all messages). The function produces an alphabetically sorted list. To add support for the new function, we rip the guts off of notmuch_message_get_tags and put them in a new generic function called _notmuch_convert_tags. The generic function takes a Xapian::TermIterator as argument and uses the iterator to find tags. This makes the function usable with different Xapian objects. Function notmuch_message_get_tags is then reimplemented to call the generic function with message->doc.termlist_begin() as argument. Similarly, we implement notmuch_message_database_get_all_tags, the function calls the generic function with db->xapian_db->allterms_begin() as argument. Finally, notmuch_database_get_all_tags is exported through lib/notmuch.h Signed-off-by: Jan Janak <jan@ryngle.com>
2009-11-23 01:10:54 +01:00
Implement an internal generic string list and use it. This replaces the guts of the filename list and tag list, making those interfaces simple iterators over the generic string list. The directory, message filename, and tags-related code now build generic string lists and then wraps them in specific iterators. The real wins come in later patches, when we use these for even more generic functionality. As a nice side-effect, this also eliminates the annoying dependency on GList in the tag list.
2010-12-09 01:26:05 +01:00
list = _notmuch_string_list_create (ctx);
if (unlikely (list == NULL))
notmuch: New function to retrieve all tags from the database. This patch adds a new function called notmuch_database_get_all_tags which can be used to obtain a list of all tags from the database (in other words, the list contains all tags from all messages). The function produces an alphabetically sorted list. To add support for the new function, we rip the guts off of notmuch_message_get_tags and put them in a new generic function called _notmuch_convert_tags. The generic function takes a Xapian::TermIterator as argument and uses the iterator to find tags. This makes the function usable with different Xapian objects. Function notmuch_message_get_tags is then reimplemented to call the generic function with message->doc.termlist_begin() as argument. Similarly, we implement notmuch_message_database_get_all_tags, the function calls the generic function with db->xapian_db->allterms_begin() as argument. Finally, notmuch_database_get_all_tags is exported through lib/notmuch.h Signed-off-by: Jan Janak <jan@ryngle.com>
2009-11-23 01:10:54 +01:00
return NULL;
Add a generic function to get a list of terms with some prefix. Replace _notmuch_convert_tags with this and simplify _create_filenames_for_terms_with_prefix. This will also come in handy shortly to get the message file name list.
2010-12-09 06:32:35 +01:00
for (i.skip_to (prefix); i != end; i++) {
/* Terminate loop at first term without desired prefix. */
if (strncmp ((*i).c_str (), prefix, prefix_len))
notmuch: New function to retrieve all tags from the database. This patch adds a new function called notmuch_database_get_all_tags which can be used to obtain a list of all tags from the database (in other words, the list contains all tags from all messages). The function produces an alphabetically sorted list. To add support for the new function, we rip the guts off of notmuch_message_get_tags and put them in a new generic function called _notmuch_convert_tags. The generic function takes a Xapian::TermIterator as argument and uses the iterator to find tags. This makes the function usable with different Xapian objects. Function notmuch_message_get_tags is then reimplemented to call the generic function with message->doc.termlist_begin() as argument. Similarly, we implement notmuch_message_database_get_all_tags, the function calls the generic function with db->xapian_db->allterms_begin() as argument. Finally, notmuch_database_get_all_tags is exported through lib/notmuch.h Signed-off-by: Jan Janak <jan@ryngle.com>
2009-11-23 01:10:54 +01:00
break;
Add a generic function to get a list of terms with some prefix. Replace _notmuch_convert_tags with this and simplify _create_filenames_for_terms_with_prefix. This will also come in handy shortly to get the message file name list.
2010-12-09 06:32:35 +01:00
_notmuch_string_list_append (list, (*i).c_str () + prefix_len);
notmuch: New function to retrieve all tags from the database. This patch adds a new function called notmuch_database_get_all_tags which can be used to obtain a list of all tags from the database (in other words, the list contains all tags from all messages). The function produces an alphabetically sorted list. To add support for the new function, we rip the guts off of notmuch_message_get_tags and put them in a new generic function called _notmuch_convert_tags. The generic function takes a Xapian::TermIterator as argument and uses the iterator to find tags. This makes the function usable with different Xapian objects. Function notmuch_message_get_tags is then reimplemented to call the generic function with message->doc.termlist_begin() as argument. Similarly, we implement notmuch_message_database_get_all_tags, the function calls the generic function with db->xapian_db->allterms_begin() as argument. Finally, notmuch_database_get_all_tags is exported through lib/notmuch.h Signed-off-by: Jan Janak <jan@ryngle.com>
2009-11-23 01:10:54 +01:00
}
Add a generic function to get a list of terms with some prefix. Replace _notmuch_convert_tags with this and simplify _create_filenames_for_terms_with_prefix. This will also come in handy shortly to get the message file name list.
2010-12-09 06:32:35 +01:00
return list;
notmuch: New function to retrieve all tags from the database. This patch adds a new function called notmuch_database_get_all_tags which can be used to obtain a list of all tags from the database (in other words, the list contains all tags from all messages). The function produces an alphabetically sorted list. To add support for the new function, we rip the guts off of notmuch_message_get_tags and put them in a new generic function called _notmuch_convert_tags. The generic function takes a Xapian::TermIterator as argument and uses the iterator to find tags. This makes the function usable with different Xapian objects. Function notmuch_message_get_tags is then reimplemented to call the generic function with message->doc.termlist_begin() as argument. Similarly, we implement notmuch_message_database_get_all_tags, the function calls the generic function with db->xapian_db->allterms_begin() as argument. Finally, notmuch_database_get_all_tags is exported through lib/notmuch.h Signed-off-by: Jan Janak <jan@ryngle.com>
2009-11-23 01:10:54 +01:00
}
notmuch_tags_t *
notmuch_database_get_all_tags (notmuch_database_t *db)
{
Xapian::TermIterator i, end;
Add a generic function to get a list of terms with some prefix. Replace _notmuch_convert_tags with this and simplify _create_filenames_for_terms_with_prefix. This will also come in handy shortly to get the message file name list.
2010-12-09 06:32:35 +01:00
notmuch_string_list_t *tags;
lib: Audit all notmuch_database call for Xapian exception handling. Our current approach is for top-level entry poitns in the library to have try/catch blocks that catch any Xapian exception and print a message. Add a few missing blocks and fix up the documentation.
2010-04-24 16:22:34 +02:00
try {
i = db->xapian_db->allterms_begin();
end = db->xapian_db->allterms_end();
Add a generic function to get a list of terms with some prefix. Replace _notmuch_convert_tags with this and simplify _create_filenames_for_terms_with_prefix. This will also come in handy shortly to get the message file name list.
2010-12-09 06:32:35 +01:00
tags = _notmuch_database_get_terms_with_prefix (db, i, end,
_find_prefix ("tag"));
_notmuch_string_list_sort (tags);
return _notmuch_tags_create (db, tags);
lib: Audit all notmuch_database call for Xapian exception handling. Our current approach is for top-level entry poitns in the library to have try/catch blocks that catch any Xapian exception and print a message. Add a few missing blocks and fix up the documentation.
2010-04-24 16:22:34 +02:00
} catch (const Xapian::Error &error) {
lib: replace almost all fprintfs in library with _n_d_log This is not supposed to change any functionality from an end user point of view. Note that it will eliminate some output to stderr. The query debugging output is left as is; it doesn't really fit with the current primitive logging model. The remaining "bad" fprintf will need an internal API change.
2014-12-26 17:25:35 +01:00
_notmuch_database_log (db, "A Xapian exception occurred getting tags: %s.\n",
lib: Audit all notmuch_database call for Xapian exception handling. Our current approach is for top-level entry poitns in the library to have try/catch blocks that catch any Xapian exception and print a message. Add a few missing blocks and fix up the documentation.
2010-04-24 16:22:34 +02:00
error.get_msg().c_str());
db->exception_reported = TRUE;
return NULL;
}
notmuch: New function to retrieve all tags from the database. This patch adds a new function called notmuch_database_get_all_tags which can be used to obtain a list of all tags from the database (in other words, the list contains all tags from all messages). The function produces an alphabetically sorted list. To add support for the new function, we rip the guts off of notmuch_message_get_tags and put them in a new generic function called _notmuch_convert_tags. The generic function takes a Xapian::TermIterator as argument and uses the iterator to find tags. This makes the function usable with different Xapian objects. Function notmuch_message_get_tags is then reimplemented to call the generic function with message->doc.termlist_begin() as argument. Similarly, we implement notmuch_message_database_get_all_tags, the function calls the generic function with db->xapian_db->allterms_begin() as argument. Finally, notmuch_database_get_all_tags is exported through lib/notmuch.h Signed-off-by: Jan Janak <jan@ryngle.com>
2009-11-23 01:10:54 +01:00
}
lib: add a log function with output to a string in notmuch_database_t In principle in the future this could do something fancier than asprintf.
2014-12-26 09:01:01 +01:00
const char *
lib: constify argument to notmuch_database_status_string We don't modify the database struct, so no harm in committing to that.
2015-06-07 17:02:00 +02:00
notmuch_database_status_string (const notmuch_database_t *notmuch)
lib: add a log function with output to a string in notmuch_database_t In principle in the future this could do something fancier than asprintf.
2014-12-26 09:01:01 +01:00
{
return notmuch->status_string;
}
Reference in a new issue Copy permalink