debian/notmuch
1
0
Fork 0
mirror of https://git.notmuchmail.org/git/notmuch synced 2024-12-01 15:14:16 +01:00
Code Issues Projects Releases Packages Wiki Activity Actions
notmuch/lib/database.cc

2049 lines
60 KiB
C++
Raw Normal View History

notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
/* database.cc - The database interfaces of the notmuch mail library
*
* Copyright © 2009 Carl Worth
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
Use https instead of http where possible Many of the external links found in the notmuch source can be resolved using https instead of http. This changeset addresses as many as i could find, without touching the e-mail corpus or expected outputs found in tests.
2016-06-02 18:26:14 +02:00
* along with this program. If not, see https://www.gnu.org/licenses/ .
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
*
* Author: Carl Worth <cworth@cworth.org>
*/
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
#include "database-private.h"
lib: add date range query support Add a custom value range processor to enable date and time searches of the form date:since..until, where "since" and "until" are expressions understood by the previously added date/time parser, to restrict the results to messages within a particular time range (based on the Date: header). If "since" or "until" describes date/time at an accuracy of days or less, the values are rounded according to the accuracy, towards past for "since" and towards future for "until". For example, date:november..yesterday would match from the beginning of November until the end of yesterday. Expressions such as date:today..today means since the beginning of today until the end of today. Open-ended ranges are supported (since Xapian 1.2.1), i.e. you can specify date:..until or date:since.. to not limit the start or end date, respectively. CAVEATS: Xapian does not support spaces in range expressions. You can replace the spaces with '_', or (in most cases) '-', or (in some cases) leave the spaces out altogether. Entering date:expr without ".." (for example date:yesterday) will not work as you might expect. You can achieve the expected result by duplicating the expr both sides of ".." (for example date:yesterday..yesterday). Open-ended ranges won't work with pre-1.2.1 Xapian, but they don't produce an error either. Signed-off-by: Jani Nikula <jani@nikula.org>
2012-10-30 21:32:37 +01:00
#include "parse-time-vrp.h"
lib: add support for named queries This relies on the optional presense of xapian field processors, and the library config API.
2016-03-22 11:54:54 +01:00
#include "query-fp.h"
lib: add thread subqueries. This change allows queries of the form thread:{from:me} and thread:{from:jian} and not thread:{from:dave} This is still somewhat brute-force, but it's a big improvement over both the shell script solution and the previous proposal [1], because it does not build the whole thread structure just generate a query. A further potential optimization is to replace the calls to notmuch with more specialized Xapian code; in particular it's not likely that reading all of the message metadata is a win here. [1]: id:20170820213240.20526-1-david@tethera.net
2017-12-26 02:03:05 +01:00
#include "thread-fp.h"
lib: regexp matching in 'subject' and 'from' the idea is that you can run % notmuch search subject:/<your-favourite-regexp>/ % notmuch search from:/<your-favourite-regexp>/ or % notmuch search subject:"your usual phrase search" % notmuch search from:"usual phrase search" This feature is only available with recent Xapian, specifically support for field processors is needed. It should work with bindings, since it extends the query parser. This is easy to extend for other value slots, but currently the only value slots are date, message_id, from, subject, and last_mod. Date is already searchable; message_id is left for a followup commit. This was originally written by Austin Clements, and ported to Xapian field processors (from Austin's custom query parser) by yours truly.
2017-02-27 03:34:20 +01:00
#include "regexp-fields.h"
lib: Database version 3: Introduce fine-grained "features" Previously, our database schema was versioned by a single number. Each database schema change had to occur "atomically" in Notmuch's development history: before some commit, Notmuch used version N, after that commit, it used version N+1. Hence, each new schema version could introduce only one change, the task of developing a schema change fell on a single person, and it all had to happen and be perfect in a single commit series. This made introducing a new schema version hard. We've seen only two schema changes in the history of Notmuch. This commit introduces database schema version 3; hopefully the last schema version we'll need for a while. With this version, we switch from a single version number to "features": a set of named, independent aspects of the database schema. Features should make backwards compatibility easier. For many things, it should be easy to support databases both with and without a feature, which will allow us to make upgrades optional and will enable "unstable" features that can be developed and tested over time. Features also make forwards compatibility easier. The features recorded in a database include "compatibility flags," which can indicate to an older version of Notmuch when it must support a given feature to open the database for read or for write. This lets us replace the old vague "I don't recognize this version, so something might go wrong, but I promise to try my best" warnings upon opening a database with an unknown version with precise errors. If a database is safe to open for read/write despite unknown features, an older version will know that and issue no message at all. If the database is not safe to open for read/write because of unknown features, an older version will know that, too, and can tell the user exactly which required features it lacks support for.
2014-08-25 19:26:00 +02:00
#include "string-util.h"
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
#include <iostream>
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
#include <sys/time.h>
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
#include <sys/stat.h>
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
#include <signal.h>
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
#include <ftw.h>
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
#include <glib.h> /* g_free, GPtrArray, GHashTable */
#include <glib-object.h> /* g_type_init */
notmuch: Switch from gmime to custom, ad-hoc parsing of headers. Since we're currently just trying to stitch together In-Reply-To and References headers we don't need that much sophistication. It's when we later add full-text searching that GMime will be useful. So for now, even though my own code here is surely very buggy compared to GMime it's also a lot faster. And speed is what we're after for the initial index creation.
2009-10-19 21:54:40 +02:00
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
#include <gmime/gmime.h> /* g_mime_init */
lib: call g_mime_init() from notmuch_database_open() As reported in id:"CAEbOPGyuHnz4BPtDutnTPUHcP3eYcRCRkXhYoJR43RUMw671+g@mail.gmail.com" sometimes gmime tries to access a NULL pointer, e.g. g_mime_iconv_open() tries to access iconv_cache that is NULL if g_mime_init() is not called. This causes notmuch to segfault when calling gmime functions. Calling g_mime_init() initializes iconv_cache and others variables needed by gmime, making sure they are initialized when notmuch calls gmime functions. Test marked fix by db.
2011-12-31 05:37:41 +01:00
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
using namespace std;
Move the prefix-string arrays back into database.cc from message.cc Yes, I'm being wishy-washy here, moving code back and forth. But this is where these really do belong.
2009-10-25 05:52:48 +01:00
typedef struct {
const char *name;
const char *prefix;
lib: merge internal prefix tables Replace multiple tables with some flags in a single table. This makes the code in notmuch_database_open_verbose a bit shorter, and it should also make it easier to add other options to fields, e.g. regexp searching.
2017-02-17 04:07:49 +01:00
notmuch_field_flag_t flags;
Move the prefix-string arrays back into database.cc from message.cc Yes, I'm being wishy-washy here, moving code back and forth. But this is where these really do belong.
2009-10-25 05:52:48 +01:00
} prefix_t;
lib: Database version 3: Introduce fine-grained "features" Previously, our database schema was versioned by a single number. Each database schema change had to occur "atomically" in Notmuch's development history: before some commit, Notmuch used version N, after that commit, it used version N+1. Hence, each new schema version could introduce only one change, the task of developing a schema change fell on a single person, and it all had to happen and be perfect in a single commit series. This made introducing a new schema version hard. We've seen only two schema changes in the history of Notmuch. This commit introduces database schema version 3; hopefully the last schema version we'll need for a while. With this version, we switch from a single version number to "features": a set of named, independent aspects of the database schema. Features should make backwards compatibility easier. For many things, it should be easy to support databases both with and without a feature, which will allow us to make upgrades optional and will enable "unstable" features that can be developed and tested over time. Features also make forwards compatibility easier. The features recorded in a database include "compatibility flags," which can indicate to an older version of Notmuch when it must support a given feature to open the database for read or for write. This lets us replace the old vague "I don't recognize this version, so something might go wrong, but I promise to try my best" warnings upon opening a database with an unknown version with precise errors. If a database is safe to open for read/write despite unknown features, an older version will know that and issue no message at all. If the database is not safe to open for read/write because of unknown features, an older version will know that, too, and can tell the user exactly which required features it lacks support for.
2014-08-25 19:26:00 +02:00
#define NOTMUCH_DATABASE_VERSION 3
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
#define STRINGIFY(s) _SUB_STRINGIFY (s)
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
#define _SUB_STRINGIFY(s) #s
Use the Xapian::DB_RETRY_LOCK flag when available Xapian 1.3 has introduced the DB_RETRY_LOCK flag (Xapian bug 275). Detect it in configure and optionally use it. With this flag commands that need the write lock will wait for their turn instead of aborting when it's not immediately available. Amended by db: allow disabling in configure
2016-06-26 17:29:43 +02:00
#if HAVE_XAPIAN_DB_RETRY_LOCK
#define DB_ACTION (Xapian::DB_CREATE_OR_OPEN | Xapian::DB_RETRY_LOCK)
#else
#define DB_ACTION Xapian::DB_CREATE_OR_OPEN
#endif
lib/n_d_get_version: catch exceptions and clarify the API notmuch_database_get_version previously returned 0 on some errors, but did not document this. Luckily 0 is not a valid database version.
2020-07-15 00:31:10 +02:00
#define LOG_XAPIAN_EXCEPTION(message, error) _log_xapian_exception (__location__, message, error)
static void
_log_xapian_exception (const char *where, notmuch_database_t *notmuch, const Xapian::Error error) {
_notmuch_database_log (notmuch,
"A Xapian exception occurred at %s: %s\n",
where,
error.get_msg ().c_str ());
notmuch->exception_reported = true;
}
lib: encapsulate the use of notmuch_database_t field 'mode' The plan is to change the underlying representation.
2020-07-27 01:31:35 +02:00
notmuch_database_mode_t
_notmuch_database_mode (notmuch_database_t *notmuch)
{
lib: replace use of static_cast for writable databases static_cast is a bit tricky to understand and error prone, so add a second pointer to (potentially the same) Xapian database object that we know has the right subclass.
2020-07-27 01:31:36 +02:00
if (notmuch->writable_xapian_db)
return NOTMUCH_DATABASE_MODE_READ_WRITE;
else
return NOTMUCH_DATABASE_MODE_READ_ONLY;
lib: encapsulate the use of notmuch_database_t field 'mode' The plan is to change the underlying representation.
2020-07-27 01:31:35 +02:00
}
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
/* Here's the current schema for our database (for NOTMUCH_DATABASE_VERSION):
Add comment documenting our current database schema. I've got schemes to change this schema somewhat dramatically, so I want a place to be able to record and review those changes.
2009-10-25 16:57:09 +01:00
*
lib: Update database schema doc for ghost messages This describes the structure of ghost mail documents. Ghost messages are not yet implemented.
2014-10-23 14:30:34 +02:00
* We currently have three different types of documents (mail, ghost,
* and directory) and also some metadata.
Add comment documenting our current database schema. I've got schemes to change this schema somewhat dramatically, so I want a place to be able to record and review those changes.
2009-10-25 16:57:09 +01:00
*
lib: add clarification about the use of "prefix" in the docs.
2019-03-13 01:47:10 +01:00
* There are two kinds of prefixes used in notmuch. There are the
* human friendly 'prefix names' like "thread:", which are also used
* in the query parser, and the actual prefix terms in the database
* (e.g. "G"). The correspondence is maintained in the file scope data
* structure 'prefix_table'.
*
Add comment documenting our current database schema. I've got schemes to change this schema somewhat dramatically, so I want a place to be able to record and review those changes.
2009-10-25 16:57:09 +01:00
* Mail document
* -------------
lib: Fix slight misinformation in the database schema doc The database schema documentation made it sound like each mail document had exactly one on-disk message file, which hasn't been true for a long time.
2014-08-01 04:59:13 +02:00
* A mail document is associated with a particular email message. It
lib: drop comment about only indexing one file. Although the situation is complicated by the value fields (which are taken from a single file), this comment is now more false than true.
2019-03-13 01:47:09 +01:00
* is stored in one or more files on disk and is uniquely identified
* by its "id" field (which is generally the message ID). It is
* indexed with the following prefixed terms which the database uses
* to construct threads, etc.:
Add comment documenting our current database schema. I've got schemes to change this schema somewhat dramatically, so I want a place to be able to record and review those changes.
2009-10-25 16:57:09 +01:00
*
* Single terms of given prefix:
*
* type: mail
*
Add support (and tests) for messages with really long message IDs. Scott Henson reported an internal error that occurred when he tried to add a message that referenced another message with a message ID well over 300 characters in length. The bug here was running into a Xapian limit for the length of metadata key names, (which is even more restrictive than the Xapian limit for the length of terms). We fix this by noticing long message ID values and instead using a message ID of the form "notmuch-sha1-<sha1_sum_of_message_id>". That is, we use SHA1 to generate a compressed, (but still unique), version of the message ID. We add support to the test suite to exercise this fix. The tests add a message referencing the long message ID, then add the message with the long message ID, then finally add another message referencing the long ID. Each of these tests exercise different code paths where the special handling is implemented. A final test ensures that all three messages are stitched together into a single thread---guaranteeing that the three code paths all act consistently.
2010-06-04 21:39:36 +02:00
* id: Unique ID of mail. This is from the Message-ID header
* if present and not too long (see NOTMUCH_MESSAGE_ID_MAX).
* If it's present and too long, then we use
* "notmuch-sha1-<sha1_sum_of_message_id>".
* If this header is not present, we use
* "notmuch-sha1-<sha1_sum_of_entire_file>".
Add comment documenting our current database schema. I've got schemes to change this schema somewhat dramatically, so I want a place to be able to record and review those changes.
2009-10-25 16:57:09 +01:00
*
Change database to store only a single thread ID per message. Instead of supporting multiple thread IDs, we now merge together thread IDs if one message is ever found to belong to more than one thread. This allows for constructing complete threads when, for example, a child message doesn't include a complete list of References headers back to the beginning of the thread. It also simplifies dealing with mapping a message ID to a thread ID which is now a simple get_thread_id just like get_message_id, (and no longer an iterator-based thing like get_tags).
2009-10-25 22:54:13 +01:00
* thread: The ID of the thread to which the mail belongs
*
database: Add "replyto" to the database schema documentation. Maybe ths lack of this documentation is why I forgot we were actually storing this and wrote the ugly code to fetch In-Reply-To from message files rather than from the database.
2009-11-18 03:48:38 +01:00
* replyto: The ID from the In-Reply-To header of the mail (if any).
*
Add comment documenting our current database schema. I've got schemes to change this schema somewhat dramatically, so I want a place to be able to record and review those changes.
2009-10-25 16:57:09 +01:00
* Multiple terms of given prefix:
*
fix sum moar typos [comments in source code] Various typo fixes in comments within the source code. Signed-off-by: Pieter Praet <pieter@praet.org> Edited-by: Carl Worth <cworth@cworth.org> Restricted to just source-code comments, (and fixed fix of "descriptios" to "descriptors" rather than "descriptions").
2011-06-20 22:14:21 +02:00
* reference: All message IDs from In-Reply-To and References
database: Rename "ref" prefix name to "reference" Which is more consistent with the XREFERENCE prefix used in the terms in the database. Also remove some stale documentation describing the removal of resolved references from the database (we no longer do this).
2009-11-18 03:44:02 +01:00
* headers in the message.
Add comment documenting our current database schema. I've got schemes to change this schema somewhat dramatically, so I want a place to be able to record and review those changes.
2009-10-25 16:57:09 +01:00
*
database: Rename "ref" prefix name to "reference" Which is more consistent with the XREFERENCE prefix used in the terms in the database. Also remove some stale documentation describing the removal of resolved references from the database (we no longer do this).
2009-11-18 03:44:02 +01:00
* tag: Any tags associated with this message by the user.
Add comment documenting our current database schema. I've got schemes to change this schema somewhat dramatically, so I want a place to be able to record and review those changes.
2009-10-25 16:57:09 +01:00
*
lib: Implement new notmuch_directory_t API. This new directory ojbect provides all the infrastructure needed to detect when files or directories are deleted or renamed. There's still code needed on top of this (within "notmuch new") to actually do that detection.
2010-01-05 22:29:23 +01:00
* file-direntry: A colon-separated pair of values
* (INTEGER:STRING), where INTEGER is the
* document ID of a directory document, and
* STRING is the name of a file within that
* directory for this mail message.
database: Store mail filename as a new 'direntry' term, not as 'data'. Instead of storing the complete message filename in the data portion of a mail document we now store a 'direntry' term that contains the document ID of a directory document and also the basename of the message filename within that directory. This will allow us to easily store multple filenames for a single message, and will also allow us to find mail documents for files that previously existed in a directory but that have since been deleted.
2009-12-21 17:23:26 +01:00
*
add property: query prefix to search for specific properties We want to be able to query the properties directly, like: notmuch count property:foo=bar which should return a count of messages where the property with key "foo" has value equal to "bar".
2016-07-08 11:15:36 +02:00
* property: Has a property with key=value
* FIXME: if no = is present, should match on any value
*
Fix comments about what is stored in the database Commit 567bcbc2 introduced two new values for each message (content of the "From" and "Subject" headers), but the comments about the database schema had not been updated accordingly.
2011-12-13 18:11:41 +01:00
* A mail document also has four values:
Add comment documenting our current database schema. I've got schemes to change this schema somewhat dramatically, so I want a place to be able to record and review those changes.
2009-10-25 16:57:09 +01:00
*
* TIMESTAMP: The time_t value corresponding to the message's
* Date header.
*
* MESSAGE_ID: The unique ID of the mail mess (see "id" above)
*
Fix comments about what is stored in the database Commit 567bcbc2 introduced two new values for each message (content of the "From" and "Subject" headers), but the comments about the database schema had not been updated accordingly.
2011-12-13 18:11:41 +01:00
* FROM: The value of the "From" header
*
* SUBJECT: The value of the "Subject" header
*
lib: Add per-message last modification tracking This adds a new document value that stores the revision of the last modification to message metadata, where the revision number increases monotonically with each database commit. An alternative would be to store the wall-clock time of the last modification of each message. In principle this is simpler and has the advantage that any process can determine the current timestamp without support from libnotmuch. However, even assuming a computer's clock never goes backward and ignoring clock skew in networked environments, this has a fatal flaw. Xapian uses (optimistic) snapshot isolation, which means reads can be concurrent with writes. Given this, consider the following time line with a write and two read transactions: write |-X-A--------------| read 1 |---B---| read 2 |---| The write transaction modifies message X and records the wall-clock time of the modification at A. The writer hangs around for a while and later commits its change. Read 1 is concurrent with the write, so it doesn't see the change to X. It does some query and records the wall-clock time of its results at B. Transaction read 2 later starts after the write commits and queries for changes since wall-clock time B (say the reads are performing an incremental backup). Even though read 1 could not see the change to X, read 2 is told (correctly) that X has not changed since B, the time of the last read. In fact, X changed before wall-clock time A, but the change was not visible until *after* wall-clock time B, so read 2 misses the change to X. This is tricky to solve in full-blown snapshot isolation, but because Xapian serializes writes, we can use a simple, monotonically increasing database revision number. Furthermore, maintaining this revision number requires no more IO than a wall-clock time solution because Xapian already maintains statistics on the upper (and lower) bound of each value stream.
2014-10-13 08:20:01 +02:00
* LAST_MOD: The revision number as of the last tag or
* filename change.
*
lib: add 'body:' field, stop indexing headers twice. The new `body:` field (in Xapian terms) or prefix (in slightly sloppier notmuch) terms allows matching terms that occur only in the body. Unprefixed query terms should continue to match anywhere (header or body) in the message. This follows a suggestion of Olly Betts to use the facility (since Xapian 1.0.4) to add the same field with multiple prefixes. The double indexing of previous versions is thus replaced with a query time expension of unprefixed query terms to the various prefixed equivalent. Reindexing will be needed for 'body:' searches to work correctly; otherwise they will also match messages where the term occur in headers (demonstrated by the new tests in T530-upgrade.sh)
2019-03-19 01:39:21 +01:00
* The prefixed terms described above are also searchable without an
* explicit field name, but as of notmuch 0.29 this is due to
* query-parser setup, not extra terms in the database. In addition,
* terms from the content of the message are added without a prefix
* for use by the user in searching. Note that the prefix name "body"
* is used to refer to the empty prefix string in the database.
lib: update commentary about path/folder terms We missed this when we changed to binary fields.
2019-03-13 01:47:11 +01:00
*
* The path of the containing folder is added with the "folder" prefix
* (see _notmuch_message_add_folder_terms). Sub-paths of the the path
* of the mail message are added with the "path" prefix.
database: Add "replyto" to the database schema documentation. Maybe ths lack of this documentation is why I forgot we were actually storing this and wrote the ugly code to fetch In-Reply-To from message files rather than from the database.
2009-11-18 03:48:38 +01:00
*
database: Store mail filename as a new 'direntry' term, not as 'data'. Instead of storing the complete message filename in the data portion of a mail document we now store a 'direntry' term that contains the document ID of a directory document and also the basename of the message filename within that directory. This will allow us to easily store multple filenames for a single message, and will also allow us to find mail documents for files that previously existed in a directory but that have since been deleted.
2009-12-21 17:23:26 +01:00
* The data portion of a mail document is empty.
lib: Document that the filename is stored in the 'data' of a mail document Our database schema documentation previously didn't give any indication of where this most essential piece of information is stored.
2009-12-19 22:05:06 +01:00
*
lib: Update database schema doc for ghost messages This describes the structure of ghost mail documents. Ghost messages are not yet implemented.
2014-10-23 14:30:34 +02:00
* Ghost mail document [if NOTMUCH_FEATURE_GHOSTS]
* -----------------------------------------------
* A ghost mail document is like a mail document, but where we don't
* have the message content. These are used to track thread reference
* information for messages we haven't received.
*
* A ghost mail document has type: ghost; id and thread fields that
* are identical to the mail document fields; and a MESSAGE_ID value.
*
lib: Rename set/get_timestamp to set/get_directory_mtime. I've been suitably scolded by Keith for doing a premature generalization that ended up just making the documentation more convoluted. Fix that.
2009-12-17 23:33:34 +01:00
* Directory document
Add comment documenting our current database schema. I've got schemes to change this schema somewhat dramatically, so I want a place to be able to record and review those changes.
2009-10-25 16:57:09 +01:00
* ------------------
lib: Rename set/get_timestamp to set/get_directory_mtime. I've been suitably scolded by Keith for doing a premature generalization that ended up just making the documentation more convoluted. Fix that.
2009-12-17 23:33:34 +01:00
* A directory document is used by a client of the notmuch library to
Add comment documenting our current database schema. I've got schemes to change this schema somewhat dramatically, so I want a place to be able to record and review those changes.
2009-10-25 16:57:09 +01:00
* maintain data necessary to allow for efficient polling of mail
lib: Rename set/get_timestamp to set/get_directory_mtime. I've been suitably scolded by Keith for doing a premature generalization that ended up just making the documentation more convoluted. Fix that.
2009-12-17 23:33:34 +01:00
* directories.
Add comment documenting our current database schema. I've got schemes to change this schema somewhat dramatically, so I want a place to be able to record and review those changes.
2009-10-25 16:57:09 +01:00
*
lib: Implement new notmuch_directory_t API. This new directory ojbect provides all the infrastructure needed to detect when files or directories are deleted or renamed. There's still code needed on top of this (within "notmuch new") to actually do that detection.
2010-01-05 22:29:23 +01:00
* All directory documents contain one term:
Add comment documenting our current database schema. I've got schemes to change this schema somewhat dramatically, so I want a place to be able to record and review those changes.
2009-10-25 16:57:09 +01:00
*
database: Store directory paths as relative, not absolute. We were already storing relative mail filenames, so this is consistent with that. Additionally, it means that directory documents remain valid even if the database is relocated within its containing filesystem.
2009-12-19 22:11:00 +01:00
* directory: The directory path (relative to the database path)
database: Store directory path in 'data' of directory documents. We're planning to have mail documents refer to directory documents for the path of the containing directory. To support this, we need the path in the data, (since the path in the 'directory' term can be irretrievable as it will be the SHA1 sum of the path for a very long path).
2009-12-21 00:46:41 +01:00
* Or the SHA1 sum of the directory path (if the
* path itself is too long to fit in a Xapian
* term).
*
lib: Implement new notmuch_directory_t API. This new directory ojbect provides all the infrastructure needed to detect when files or directories are deleted or renamed. There's still code needed on top of this (within "notmuch new") to actually do that detection.
2010-01-05 22:29:23 +01:00
* And all directory documents for directories other than top-level
* directories also contain the following term:
Add comment documenting our current database schema. I've got schemes to change this schema somewhat dramatically, so I want a place to be able to record and review those changes.
2009-10-25 16:57:09 +01:00
*
lib: Implement new notmuch_directory_t API. This new directory ojbect provides all the infrastructure needed to detect when files or directories are deleted or renamed. There's still code needed on top of this (within "notmuch new") to actually do that detection.
2010-01-05 22:29:23 +01:00
* directory-direntry: A colon-separated pair of values
* (INTEGER:STRING), where INTEGER is the
* document ID of the parent directory
* document, and STRING is the name of this
* directory within that parent.
*
* All directory documents have a single value:
Add comment documenting our current database schema. I've got schemes to change this schema somewhat dramatically, so I want a place to be able to record and review those changes.
2009-10-25 16:57:09 +01:00
*
lib: Rename set/get_timestamp to set/get_directory_mtime. I've been suitably scolded by Keith for doing a premature generalization that ended up just making the documentation more convoluted. Fix that.
2009-12-17 23:33:34 +01:00
* TIMESTAMP: The mtime of the directory (at last scan)
database: Store directory path in 'data' of directory documents. We're planning to have mail documents refer to directory documents for the path of the containing directory. To support this, we need the path in the data, (since the path in the 'directory' term can be irretrievable as it will be the SHA1 sum of the path for a very long path).
2009-12-21 00:46:41 +01:00
*
* The data portion of a directory document contains the path of the
lib: Implement new notmuch_directory_t API. This new directory ojbect provides all the infrastructure needed to detect when files or directories are deleted or renamed. There's still code needed on top of this (within "notmuch new") to actually do that detection.
2010-01-05 22:29:23 +01:00
* directory (relative to the database path).
lib: Document the metadata stored within the Xapian database. We are currently storing "version", "last_thread_id", and "thread_id_*" values so document how each of these are used.
2010-04-13 00:15:14 +02:00
*
* Database metadata
* -----------------
* Xapian allows us to store arbitrary name-value pairs as
* "metadata". We currently use the following metadata names with the
* given meanings:
*
* version The database schema version, (which is distinct
* from both the notmuch package version (see
* notmuch --version) and the libnotmuch library
* version. The version is stored as an base-10
* ASCII integer. The initial database version
* was 1, (though a schema existed before that
* were no "version" database value existed at
fix sum moar typos [comments in source code] Various typo fixes in comments within the source code. Signed-off-by: Pieter Praet <pieter@praet.org> Edited-by: Carl Worth <cworth@cworth.org> Restricted to just source-code comments, (and fixed fix of "descriptios" to "descriptors" rather than "descriptions").
2011-06-20 22:14:21 +02:00
* all). Successive versions are allocated as
lib: Document the metadata stored within the Xapian database. We are currently storing "version", "last_thread_id", and "thread_id_*" values so document how each of these are used.
2010-04-13 00:15:14 +02:00
* changes are made to the database (such as by
* indexing new fields).
*
lib: Database version 3: Introduce fine-grained "features" Previously, our database schema was versioned by a single number. Each database schema change had to occur "atomically" in Notmuch's development history: before some commit, Notmuch used version N, after that commit, it used version N+1. Hence, each new schema version could introduce only one change, the task of developing a schema change fell on a single person, and it all had to happen and be perfect in a single commit series. This made introducing a new schema version hard. We've seen only two schema changes in the history of Notmuch. This commit introduces database schema version 3; hopefully the last schema version we'll need for a while. With this version, we switch from a single version number to "features": a set of named, independent aspects of the database schema. Features should make backwards compatibility easier. For many things, it should be easy to support databases both with and without a feature, which will allow us to make upgrades optional and will enable "unstable" features that can be developed and tested over time. Features also make forwards compatibility easier. The features recorded in a database include "compatibility flags," which can indicate to an older version of Notmuch when it must support a given feature to open the database for read or for write. This lets us replace the old vague "I don't recognize this version, so something might go wrong, but I promise to try my best" warnings upon opening a database with an unknown version with precise errors. If a database is safe to open for read/write despite unknown features, an older version will know that and issue no message at all. If the database is not safe to open for read/write because of unknown features, an older version will know that, too, and can tell the user exactly which required features it lacks support for.
2014-08-25 19:26:00 +02:00
* features The set of features supported by this
* database. This consists of a set of
* '\n'-separated lines, where each is a feature
* name, a '\t', and compatibility flags. If the
* compatibility flags contain 'w', then the
* opener must support this feature to safely
* write this database. If the compatibility
* flags contain 'r', then the opener must
* support this feature to read this database.
* Introduced in database version 3.
*
lib: Document the metadata stored within the Xapian database. We are currently storing "version", "last_thread_id", and "thread_id_*" values so document how each of these are used.
2010-04-13 00:15:14 +02:00
* last_thread_id The last thread ID generated. This is stored
* as a 16-byte hexadecimal ASCII representation
* of a 64-bit unsigned integer. The first ID
* generated is 1 and the value will be
* incremented for each thread ID.
*
lib: document config metadata This probably should have been part of 3458e3c89c, but I missed it.
2016-06-05 00:17:41 +02:00
* C* metadata keys starting with C indicate
* configuration data. It can be managed with the
* n_database_*config* API. There is a convention
* of hierarchical keys separated by '.' (e.g.
* query.notmuch stores the value for the named
* query 'notmuch'), but it is not enforced by the
* API.
*
lib: Update database schema doc for ghost messages This describes the structure of ghost mail documents. Ghost messages are not yet implemented.
2014-10-23 14:30:34 +02:00
* Obsolete metadata
* -----------------
*
* If ! NOTMUCH_FEATURE_GHOSTS, there are no ghost mail documents.
* Instead, the database has the following additional database
* metadata:
*
lib: Document the metadata stored within the Xapian database. We are currently storing "version", "last_thread_id", and "thread_id_*" values so document how each of these are used.
2010-04-13 00:15:14 +02:00
* thread_id_* A pre-allocated thread ID for a particular
fix sum moar typos [comments in source code] Various typo fixes in comments within the source code. Signed-off-by: Pieter Praet <pieter@praet.org> Edited-by: Carl Worth <cworth@cworth.org> Restricted to just source-code comments, (and fixed fix of "descriptios" to "descriptors" rather than "descriptions").
2011-06-20 22:14:21 +02:00
* message. This is actually an arbitrarily large
Add support (and tests) for messages with really long message IDs. Scott Henson reported an internal error that occurred when he tried to add a message that referenced another message with a message ID well over 300 characters in length. The bug here was running into a Xapian limit for the length of metadata key names, (which is even more restrictive than the Xapian limit for the length of terms). We fix this by noticing long message ID values and instead using a message ID of the form "notmuch-sha1-<sha1_sum_of_message_id>". That is, we use SHA1 to generate a compressed, (but still unique), version of the message ID. We add support to the test suite to exercise this fix. The tests add a message referencing the long message ID, then add the message with the long message ID, then finally add another message referencing the long ID. Each of these tests exercise different code paths where the special handling is implemented. A final test ensures that all three messages are stitched together into a single thread---guaranteeing that the three code paths all act consistently.
2010-06-04 21:39:36 +02:00
* family of metadata name. Any particular name is
* formed by concatenating "thread_id_" with a message
* ID (or the SHA1 sum of a message ID if it is very
* long---see description of 'id' in the mail
* document). The value stored is a thread ID.
lib: Document the metadata stored within the Xapian database. We are currently storing "version", "last_thread_id", and "thread_id_*" values so document how each of these are used.
2010-04-13 00:15:14 +02:00
*
* These thread ID metadata values are stored
* whenever a message references a parent message
* that does not yet exist in the database. A
* thread ID will be allocated and stored, and if
* the message is later added, the stored thread
* ID will be used (and the metadata value will
* be cleared).
*
* Even before a message is added, it's
* pre-allocated thread ID is useful so that all
* descendant messages that reference this common
* parent can be recognized as belonging to the
* same thread.
Add comment documenting our current database schema. I've got schemes to change this schema somewhat dramatically, so I want a place to be able to record and review those changes.
2009-10-25 16:57:09 +01:00
*/
Invent our own prefix values. We're now dropping all pretense of keeping the database directly compatible with sup's current xapian backend. (But perhaps someone might write a new nothmuch backend for sup in the future.) In coming up with the prefix values here, I tried to follow the conventions of http://xapian.org/docs/omega/termprefixes.html as closely as makes sense, (with some domain translation from "web" to "email archive").
2009-10-25 06:49:35 +01:00
/* With these prefix values we follow the conventions published here:
*
Use https instead of http where possible Many of the external links found in the notmuch source can be resolved using https instead of http. This changeset addresses as many as i could find, without touching the e-mail corpus or expected outputs found in tests.
2016-06-02 18:26:14 +02:00
* https://xapian.org/docs/omega/termprefixes.html
Invent our own prefix values. We're now dropping all pretense of keeping the database directly compatible with sup's current xapian backend. (But perhaps someone might write a new nothmuch backend for sup in the future.) In coming up with the prefix values here, I tried to follow the conventions of http://xapian.org/docs/omega/termprefixes.html as closely as makes sense, (with some domain translation from "web" to "email archive").
2009-10-25 06:49:35 +01:00
*
* as much as makes sense. Note that I took some liberty in matching
* the reserved prefix values to notmuch concepts, (for example, 'G'
* is documented as "newsGroup (or similar entity - e.g. a web forum
* name)", for which I think the thread is the closest analogue in
* notmuch. This in spite of the fact that we will eventually be
* storing mailing-list messages where 'G' for "mailing list name"
* might be even a closer analogue. I'm treating the single-character
* prefixes preferentially for core notmuch concepts (which will be
* nearly universal to all mail messages).
*/
lib: merge internal prefix tables Replace multiple tables with some flags in a single table. This makes the code in notmuch_database_open_verbose a bit shorter, and it should also make it easier to add other options to fields, e.g. regexp searching.
2017-02-17 04:07:49 +01:00
static const
prefix_t prefix_table[] = {
/* name term prefix flags */
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
{ "type", "T", NOTMUCH_FIELD_NO_FLAGS },
{ "reference", "XREFERENCE", NOTMUCH_FIELD_NO_FLAGS },
{ "replyto", "XREPLYTO", NOTMUCH_FIELD_NO_FLAGS },
{ "directory", "XDIRECTORY", NOTMUCH_FIELD_NO_FLAGS },
{ "file-direntry", "XFDIRENTRY", NOTMUCH_FIELD_NO_FLAGS },
{ "directory-direntry", "XDDIRENTRY", NOTMUCH_FIELD_NO_FLAGS },
{ "body", "", NOTMUCH_FIELD_EXTERNAL |
NOTMUCH_FIELD_PROBABILISTIC },
{ "thread", "G", NOTMUCH_FIELD_EXTERNAL |
NOTMUCH_FIELD_PROCESSOR },
{ "tag", "K", NOTMUCH_FIELD_EXTERNAL |
NOTMUCH_FIELD_PROCESSOR },
{ "is", "K", NOTMUCH_FIELD_EXTERNAL |
NOTMUCH_FIELD_PROCESSOR },
{ "id", "Q", NOTMUCH_FIELD_EXTERNAL },
{ "mid", "Q", NOTMUCH_FIELD_EXTERNAL |
NOTMUCH_FIELD_PROCESSOR },
{ "path", "P", NOTMUCH_FIELD_EXTERNAL |
NOTMUCH_FIELD_PROCESSOR },
{ "property", "XPROPERTY", NOTMUCH_FIELD_EXTERNAL },
lib: make folder: prefix literal In xapian terms, convert folder: prefix from probabilistic to boolean prefix, matching the paths, relative from the maildir root, of the message files, ignoring the maildir new and cur leaf directories. folder:foo matches all message files in foo, foo/new, and foo/cur. folder:foo/new does *not* match message files in foo/new. folder:"" matches all message files in the top level maildir and its new and cur subdirectories. This change constitutes a database change: bump the database version and add database upgrade support for folder: terms. The upgrade also adds path: terms. Finally, fix the folder search test for literal folder: search, as some of the folder: matching capabilities are lost in the probabilistic to boolean prefix change.
2014-02-08 20:20:42 +01:00
/*
lib: reword comment about XFOLDER: prefix I believe the current one is misleading, because in my experiments Xapian did not add : when prefix and term were both upper case. Indeed, it's hard to see how it could, because prefixes are added at a layer above Xapian in our code. See _notmuch_message_add_term for an example. Also try to explain why this is a good idea. As far as I can ascertain, this is more of an issue for a system trying to work with an unknown set of prefixes. Since notmuch has a fixed set of prefixes, and we can hopefully be trusted not to add XGOLD and XGOLDEN as prefixes, it is harder for problems to arise.
2016-08-18 10:06:54 +02:00
* Unconditionally add ':' to reduce potential ambiguity with
* overlapping prefixes and/or terms that start with capital
* letters. See Xapian document termprefixes.html for related
* discussion.
lib: make folder: prefix literal In xapian terms, convert folder: prefix from probabilistic to boolean prefix, matching the paths, relative from the maildir root, of the message files, ignoring the maildir new and cur leaf directories. folder:foo matches all message files in foo, foo/new, and foo/cur. folder:foo/new does *not* match message files in foo/new. folder:"" matches all message files in the top level maildir and its new and cur subdirectories. This change constitutes a database change: bump the database version and add database upgrade support for folder: terms. The upgrade also adds path: terms. Finally, fix the folder search test for literal folder: search, as some of the folder: matching capabilities are lost in the probabilistic to boolean prefix change.
2014-02-08 20:20:42 +01:00
*/
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
{ "folder", "XFOLDER:", NOTMUCH_FIELD_EXTERNAL |
NOTMUCH_FIELD_PROCESSOR },
{ "date", NULL, NOTMUCH_FIELD_EXTERNAL |
NOTMUCH_FIELD_PROCESSOR },
{ "query", NULL, NOTMUCH_FIELD_EXTERNAL |
NOTMUCH_FIELD_PROCESSOR },
{ "from", "XFROM", NOTMUCH_FIELD_EXTERNAL |
NOTMUCH_FIELD_PROBABILISTIC |
NOTMUCH_FIELD_PROCESSOR },
{ "to", "XTO", NOTMUCH_FIELD_EXTERNAL |
NOTMUCH_FIELD_PROBABILISTIC },
{ "attachment", "XATTACHMENT", NOTMUCH_FIELD_EXTERNAL |
NOTMUCH_FIELD_PROBABILISTIC },
{ "mimetype", "XMIMETYPE", NOTMUCH_FIELD_EXTERNAL |
NOTMUCH_FIELD_PROBABILISTIC },
{ "subject", "XSUBJECT", NOTMUCH_FIELD_EXTERNAL |
NOTMUCH_FIELD_PROBABILISTIC |
NOTMUCH_FIELD_PROCESSOR },
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
};
lib: create field processors from prefix table This is a bit more code than hardcoding the two existing field processors, but it should make it easy to add more.
2017-02-27 03:34:19 +01:00
static void
_setup_query_field_default (const prefix_t *prefix, notmuch_database_t *notmuch)
{
lib: add 'body:' field, stop indexing headers twice. The new `body:` field (in Xapian terms) or prefix (in slightly sloppier notmuch) terms allows matching terms that occur only in the body. Unprefixed query terms should continue to match anywhere (header or body) in the message. This follows a suggestion of Olly Betts to use the facility (since Xapian 1.0.4) to add the same field with multiple prefixes. The double indexing of previous versions is thus replaced with a query time expension of unprefixed query terms to the various prefixed equivalent. Reindexing will be needed for 'body:' searches to work correctly; otherwise they will also match messages where the term occur in headers (demonstrated by the new tests in T530-upgrade.sh)
2019-03-19 01:39:21 +01:00
if (prefix->prefix)
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
notmuch->query_parser->add_prefix ("", prefix->prefix);
lib: create field processors from prefix table This is a bit more code than hardcoding the two existing field processors, but it should make it easy to add more.
2017-02-27 03:34:19 +01:00
if (prefix->flags & NOTMUCH_FIELD_PROBABILISTIC)
notmuch->query_parser->add_prefix (prefix->name, prefix->prefix);
else
notmuch->query_parser->add_boolean_prefix (prefix->name, prefix->prefix);
}
lib/database: index user headers. This essentially involves calling _notmuch_message_gen_terms once for each user defined header.
2019-02-26 03:10:29 +01:00
notmuch_string_map_iterator_t *
_notmuch_database_user_headers (notmuch_database_t *notmuch)
{
return _notmuch_string_map_iterator_create (notmuch->user_header, "", false);
}
lib: setup user headers in query parser These tests will need to be updated if the Xapian query print/debug format changes.
2018-11-17 15:08:59 +01:00
const char *
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
_user_prefix (void *ctx, const char *name)
lib: setup user headers in query parser These tests will need to be updated if the Xapian query print/debug format changes.
2018-11-17 15:08:59 +01:00
{
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
return talloc_asprintf (ctx, "XU%s:", name);
lib: setup user headers in query parser These tests will need to be updated if the Xapian query print/debug format changes.
2018-11-17 15:08:59 +01:00
}
static notmuch_status_t
_setup_user_query_fields (notmuch_database_t *notmuch)
{
notmuch_config_list_t *list;
notmuch_status_t status;
lib: cache user prefixes in database object This will be used to avoid needing a database access to resolve a db prefix from the corresponding UI prefix (e.g. when indexing). Arguably the setup of the separate header map does not belong here, since it is about indexing rather than querying, but we currently don't have any other indexing setup to do.
2018-11-17 15:09:00 +01:00
notmuch->user_prefix = _notmuch_string_map_create (notmuch);
if (notmuch->user_prefix == NULL)
return NOTMUCH_STATUS_OUT_OF_MEMORY;
notmuch->user_header = _notmuch_string_map_create (notmuch);
if (notmuch->user_header == NULL)
return NOTMUCH_STATUS_OUT_OF_MEMORY;
lib: setup user headers in query parser These tests will need to be updated if the Xapian query print/debug format changes.
2018-11-17 15:08:59 +01:00
status = notmuch_database_get_config_list (notmuch, CONFIG_HEADER_PREFIX, &list);
if (status)
return status;
for (; notmuch_config_list_valid (list); notmuch_config_list_move_to_next (list)) {
prefix_t query_field;
const char *key = notmuch_config_list_key (list)
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
+ sizeof (CONFIG_HEADER_PREFIX) - 1;
lib: setup user headers in query parser These tests will need to be updated if the Xapian query print/debug format changes.
2018-11-17 15:08:59 +01:00
lib: cache user prefixes in database object This will be used to avoid needing a database access to resolve a db prefix from the corresponding UI prefix (e.g. when indexing). Arguably the setup of the separate header map does not belong here, since it is about indexing rather than querying, but we currently don't have any other indexing setup to do.
2018-11-17 15:09:00 +01:00
_notmuch_string_map_append (notmuch->user_prefix,
key,
_user_prefix (notmuch, key));
_notmuch_string_map_append (notmuch->user_header,
key,
notmuch_config_list_value (list));
lib: setup user headers in query parser These tests will need to be updated if the Xapian query print/debug format changes.
2018-11-17 15:08:59 +01:00
query_field.name = talloc_strdup (notmuch, key);
query_field.prefix = _user_prefix (notmuch, key);
query_field.flags = NOTMUCH_FIELD_PROBABILISTIC
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
| NOTMUCH_FIELD_EXTERNAL;
lib: setup user headers in query parser These tests will need to be updated if the Xapian query print/debug format changes.
2018-11-17 15:08:59 +01:00
_setup_query_field_default (&query_field, notmuch);
}
notmuch_config_list_destroy (list);
return NOTMUCH_STATUS_SUCCESS;
}
lib: create field processors from prefix table This is a bit more code than hardcoding the two existing field processors, but it should make it easy to add more.
2017-02-27 03:34:19 +01:00
static void
_setup_query_field (const prefix_t *prefix, notmuch_database_t *notmuch)
{
if (prefix->flags & NOTMUCH_FIELD_PROCESSOR) {
Xapian::FieldProcessor *fp;
if (STRNCMP_LITERAL (prefix->name, "date") == 0)
lib: migrate from Xapian ValueRangeProcessor to RangeProcessor This will be mandatory as of Xapian 1.5. The API is also more consistent with the FieldProcessor API, which helps code re-use a bit. Note that this switches to using the built-in Xapian support for prefixes on ranges (i.e. deleted code at beginning of ParseTimeRangeProcessor::operator(), added prefix to constructor). Another side effect of the migration is that we are generating smaller queries, using one OP_VALUE_RANGE instead of an AND of two OP_VALUE_* queries.
2020-07-07 12:56:46 +02:00
fp = (new DateFieldProcessor(NOTMUCH_VALUE_TIMESTAMP))->release ();
else if (STRNCMP_LITERAL(prefix->name, "query") == 0)
lib: create field processors from prefix table This is a bit more code than hardcoding the two existing field processors, but it should make it easy to add more.
2017-02-27 03:34:19 +01:00
fp = (new QueryFieldProcessor (*notmuch->query_parser, notmuch))->release ();
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
else if (STRNCMP_LITERAL (prefix->name, "thread") == 0)
lib: add thread subqueries. This change allows queries of the form thread:{from:me} and thread:{from:jian} and not thread:{from:dave} This is still somewhat brute-force, but it's a big improvement over both the shell script solution and the previous proposal [1], because it does not build the whole thread structure just generate a query. A further potential optimization is to replace the calls to notmuch with more specialized Xapian code; in particular it's not likely that reading all of the message metadata is a win here. [1]: id:20170820213240.20526-1-david@tethera.net
2017-12-26 02:03:05 +01:00
fp = (new ThreadFieldProcessor (*notmuch->query_parser, notmuch))->release ();
lib: create field processors from prefix table This is a bit more code than hardcoding the two existing field processors, but it should make it easy to add more.
2017-02-27 03:34:19 +01:00
else
lib: Add regexp searching for mid: prefix The bulk of the change is passing in the field options to the regexp field processor, so that we can properly handle the fallback (non-regexp case).
2017-02-15 13:58:28 +01:00
fp = (new RegexpFieldProcessor (prefix->name, prefix->flags,
*notmuch->query_parser, notmuch))->release ();
lib: create field processors from prefix table This is a bit more code than hardcoding the two existing field processors, but it should make it easy to add more.
2017-02-27 03:34:19 +01:00
/* we treat all field-processor fields as boolean in order to get the raw input */
lib: add 'body:' field, stop indexing headers twice. The new `body:` field (in Xapian terms) or prefix (in slightly sloppier notmuch) terms allows matching terms that occur only in the body. Unprefixed query terms should continue to match anywhere (header or body) in the message. This follows a suggestion of Olly Betts to use the facility (since Xapian 1.0.4) to add the same field with multiple prefixes. The double indexing of previous versions is thus replaced with a query time expension of unprefixed query terms to the various prefixed equivalent. Reindexing will be needed for 'body:' searches to work correctly; otherwise they will also match messages where the term occur in headers (demonstrated by the new tests in T530-upgrade.sh)
2019-03-19 01:39:21 +01:00
if (prefix->prefix)
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
notmuch->query_parser->add_prefix ("", prefix->prefix);
lib: create field processors from prefix table This is a bit more code than hardcoding the two existing field processors, but it should make it easy to add more.
2017-02-27 03:34:19 +01:00
notmuch->query_parser->add_boolean_prefix (prefix->name, fp);
} else {
_setup_query_field_default (prefix, notmuch);
}
}
Move the prefix-string arrays back into database.cc from message.cc Yes, I'm being wishy-washy here, moving code back and forth. But this is where these really do belong.
2009-10-25 05:52:48 +01:00
const char *
_find_prefix (const char *name)
{
unsigned int i;
lib: merge internal prefix tables Replace multiple tables with some flags in a single table. This makes the code in notmuch_database_open_verbose a bit shorter, and it should also make it easier to add other options to fields, e.g. regexp searching.
2017-02-17 04:07:49 +01:00
for (i = 0; i < ARRAY_SIZE (prefix_table); i++) {
if (strcmp (name, prefix_table[i].name) == 0)
return prefix_table[i].prefix;
lib/database.cc: coding style Carl claims he must have been distracted when he wrote this... Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
2009-11-22 00:14:39 +01:00
}
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
Add an INTERNAL_ERROR macro and use it for all internal errors. We were previously just doing fprintf;exit at each point, but I wanted to add file and line-number details to all messages, so it makes sense to use a single macro for that.
2009-10-25 18:22:41 +01:00
INTERNAL_ERROR ("No prefix exists for '%s'\n", name);
Move the prefix-string arrays back into database.cc from message.cc Yes, I'm being wishy-washy here, moving code back and forth. But this is where these really do belong.
2009-10-25 05:52:48 +01:00
return "";
}
lib: support user prefix names in term generation This should not change the indexing process yet as nothing calls _notmuch_message_gen_terms with a user prefix name. On the other hand, it should not break anything either. _notmuch_database_prefix does a linear walk of the list of (built-in) prefixes, followed by a logarithmic time search of the list of user prefixes. The latter is probably not really noticable.
2019-02-26 02:57:40 +01:00
/* Like find prefix, but include the possibility of user defined
* prefixes specific to this database */
const char *
_notmuch_database_prefix (notmuch_database_t *notmuch, const char *name)
{
unsigned int i;
/*XXX TODO: reduce code duplication */
for (i = 0; i < ARRAY_SIZE (prefix_table); i++) {
if (strcmp (name, prefix_table[i].name) == 0)
return prefix_table[i].prefix;
}
if (notmuch->user_prefix)
return _notmuch_string_map_get (notmuch->user_prefix, name);
return NULL;
}
Add notmuch_status_to_string function. Be kind and let the user print error messages, not just error codes.
2009-10-22 01:12:53 +02:00
const char *
notmuch_status_to_string (notmuch_status_t status)
{
switch (status) {
case NOTMUCH_STATUS_SUCCESS:
return "No error occurred";
Add -Wswitch-enum and fix warnings. Having to enumerate all the enum values at every switch is annoying, but this warning actually found a bug, (missing support for NOTMUCH_STATUS_OUT_OF_MEMORY in notmuch_status_to_string).
2009-10-26 00:03:45 +01:00
case NOTMUCH_STATUS_OUT_OF_MEMORY:
return "Out of memory";
Prefer READ_ONLY consistently over READONLY. Previously we had NOTMUCH_DATABASE_MODE_READ_ONLY but NOTMUCH_STATUS_READONLY_DATABASE which was ugly and confusing. Rename the latter to NOTMUCH_STATUS_READ_ONLY_DATABASE for consistency.
2010-01-07 19:29:05 +01:00
case NOTMUCH_STATUS_READ_ONLY_DATABASE:
Add missing comment for NOTMUCH_STATUS_READONLY_DATABASE. And adjust the string representation of the same to match.
2010-01-06 00:01:58 +01:00
return "Attempt to write to a read-only database";
Add notmuch_status_to_string function. Be kind and let the user print error messages, not just error codes.
2009-10-22 01:12:53 +02:00
case NOTMUCH_STATUS_XAPIAN_EXCEPTION:
return "A Xapian exception occurred";
Generate message ID (using SHA1) when a mail message contains none. This is important as we're using the message ID as the unique key in our database. So previously, all messages with no message ID would be treated as the same message---not good at all.
2009-10-23 00:31:56 +02:00
case NOTMUCH_STATUS_FILE_ERROR:
return "Something went wrong trying to read or write a file";
Add notmuch_status_to_string function. Be kind and let the user print error messages, not just error codes.
2009-10-22 01:12:53 +02:00
case NOTMUCH_STATUS_FILE_NOT_EMAIL:
return "File is not an email";
Add NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID And document that notmuch_database_add_message can return this value. This pushes the hard decision of what to do with duplicate messages out to the user, but that's OK. (We weren't really doing anything with these ourselves, and this way the user is at least informed of the issue, rather than it just getting papered over internally.)
2009-10-23 23:40:33 +02:00
case NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID:
return "Message ID is identical to a message in database";
Add notmuch_status_to_string function. Be kind and let the user print error messages, not just error codes.
2009-10-22 01:12:53 +02:00
case NOTMUCH_STATUS_NULL_POINTER:
return "Erroneous NULL pointer";
case NOTMUCH_STATUS_TAG_TOO_LONG:
Clarify documentation and error string for NOTMUCH_STATUS_TAG_TOO_LONG It's helpful to point out NOTMUCH_STATUS_TAG_MAX for users.
2009-10-23 23:34:21 +02:00
return "Tag value is too long (exceeds NOTMUCH_TAG_MAX)";
notmuch restore: Fix to remove all tags before adding tags. This means that the restore operation will now properly pick up the removal of tags indicated by the tag just not being present in the dump file. We added a few new public functions in order to support this: notmuch_message_freeze notmuch_message_remove_all_tags notmuch_message_thaw
2009-10-27 06:25:45 +01:00
case NOTMUCH_STATUS_UNBALANCED_FREEZE_THAW:
Typsos
2009-11-18 00:23:42 +01:00
return "Unbalanced number of calls to notmuch_message_freeze/thaw";
lib: Add support for nested atomic sections. notmuch_database_t now keeps a nesting count and we only start a transaction or commit for the outermost atomic section. Introduces a new error, NOTMUCH_STATUS_UNBALANCED_ATOMIC.
2011-06-11 05:35:06 +02:00
case NOTMUCH_STATUS_UNBALANCED_ATOMIC:
return "Unbalanced number of calls to notmuch_database_begin_atomic/end_atomic";
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
case NOTMUCH_STATUS_UNSUPPORTED_OPERATION:
return "Unsupported operation";
lib: Return an error from operations that require an upgrade Previously, there was no protection against a caller invoking an operation on an old database version that would effectively corrupt the database by treating it like a newer version. According to notmuch.h, any caller that opens the database in read/write mode is supposed to check if the database needs upgrading and perform an upgrade if it does. This would protect against this, but nobody (even the CLI) actually does this. However, with features, it's easy to protect against incompatible operations on a fine-grained basis. This lightweight change allows callers to safely operate on old database versions, while preventing specific operations that would corrupt the database with an informative error message.
2014-08-25 19:26:08 +02:00
case NOTMUCH_STATUS_UPGRADE_REQUIRED:
return "Operation requires a database upgrade";
lib: add NOTMUCH_STATUS_PATH_ERROR The difference with FILE_ERROR is that this is for things that are wrong with the path before looking at the disk. Add some 3 tests; two broken as a reminder to actually use this new code.
2015-06-10 07:58:44 +02:00
case NOTMUCH_STATUS_PATH_ERROR:
return "Path supplied is illegal for this function";
crypto: make shared crypto code behave library-like If we're going to reuse the crypto code across both the library and the client, then it needs to report error states properly and not write to stderr.
2017-10-17 21:09:56 +02:00
case NOTMUCH_STATUS_MALFORMED_CRYPTO_PROTOCOL:
return "Crypto protocol missing, malformed, or unintelligible";
case NOTMUCH_STATUS_FAILED_CRYPTO_CONTEXT_CREATION:
return "Crypto engine initialization failure";
case NOTMUCH_STATUS_UNKNOWN_CRYPTO_PROTOCOL:
return "Unknown crypto protocol";
Add notmuch_status_to_string function. Be kind and let the user print error messages, not just error codes.
2009-10-22 01:12:53 +02:00
default:
case NOTMUCH_STATUS_LAST_STATUS:
return "Unknown error status value";
}
}
lib: add a log function with output to a string in notmuch_database_t In principle in the future this could do something fancier than asprintf.
2014-12-26 09:01:01 +01:00
void
_notmuch_database_log (notmuch_database_t *notmuch,
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
const char *format,
...)
lib: add a log function with output to a string in notmuch_database_t In principle in the future this could do something fancier than asprintf.
2014-12-26 09:01:01 +01:00
{
va_list va_args;
va_start (va_args, format);
if (notmuch->status_string)
talloc_free (notmuch->status_string);
notmuch->status_string = talloc_vasprintf (notmuch, format, va_args);
lib: provide _notmuch_database_log_append _notmuch_database_log clears the log buffer each time. Rather than introducing more complicated semantics about for this function, provide a second function that does not clear the buffer. This is mainly a convenience function for callers constructing complex or multi-line log messages. The changes to query.cc are to make sure that the common code path of the new function is tested.
2016-07-15 12:25:41 +02:00
va_end (va_args);
}
void
_notmuch_database_log_append (notmuch_database_t *notmuch,
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
const char *format,
...)
lib: provide _notmuch_database_log_append _notmuch_database_log clears the log buffer each time. Rather than introducing more complicated semantics about for this function, provide a second function that does not clear the buffer. This is mainly a convenience function for callers constructing complex or multi-line log messages. The changes to query.cc are to make sure that the common code path of the new function is tested.
2016-07-15 12:25:41 +02:00
{
va_list va_args;
va_start (va_args, format);
if (notmuch->status_string)
notmuch->status_string = talloc_vasprintf_append (notmuch->status_string, format, va_args);
else
notmuch->status_string = talloc_vasprintf (notmuch, format, va_args);
lib: add a log function with output to a string in notmuch_database_t In principle in the future this could do something fancier than asprintf.
2014-12-26 09:01:01 +01:00
va_end (va_args);
}
database: Add new find_doc_ids_for_term interface. The existing find_doc_ids function is convenient when the caller doesn't want to be bothered constructing a term. But when the caller *does* have the term already, that interface is just wasteful. So we export a lower-level interface that maps a pre-constructed term to a document-ID iterators.
2009-12-22 00:12:52 +01:00
static void
find_doc_ids_for_term (notmuch_database_t *notmuch,
const char *term,
Xapian::PostingIterator *begin,
Xapian::PostingIterator *end)
{
*begin = notmuch->xapian_db->postlist_begin (term);
*end = notmuch->xapian_db->postlist_end (term);
}
lib: isolate n_d_add_message and helper functions into own file 'database.cc' is becoming a monster, and it's hard to follow what the various static functions are used for. It turns out that about 1/3 of this file notmuch_database_add_message and helper functions not used by any other function. This commit isolates this code into it's own file. Some side effects of this refactoring: - find_doc_ids becomes the non-static (but still private) _notmuch_database_find_doc_ids - a few instances of 'string' have 'std::' prepended, avoiding the need for 'using namespace std;' in the new file.
2017-06-04 14:32:24 +02:00
void
_notmuch_database_find_doc_ids (notmuch_database_t *notmuch,
const char *prefix_name,
const char *value,
Xapian::PostingIterator *begin,
Xapian::PostingIterator *end)
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
{
char *term;
Remove all calls to g_strdup_printf Replacing them with calls to talloc_asprintf if possible, otherwise to asprintf (with it's painful error-handling leaving the pointer undefined).
2009-10-26 23:17:10 +01:00
term = talloc_asprintf (notmuch, "%s%s",
_find_prefix (prefix_name), value);
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
database: Add new find_doc_ids_for_term interface. The existing find_doc_ids function is convenient when the caller doesn't want to be bothered constructing a term. But when the caller *does* have the term already, that interface is just wasteful. So we export a lower-level interface that maps a pre-constructed term to a document-ID iterators.
2009-12-22 00:12:52 +01:00
find_doc_ids_for_term (notmuch, term, begin, end);
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
Remove all calls to g_strdup_printf Replacing them with calls to talloc_asprintf if possible, otherwise to asprintf (with it's painful error-handling leaving the pointer undefined).
2009-10-26 23:17:10 +01:00
talloc_free (term);
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
}
lib: Implement new notmuch_directory_t API. This new directory ojbect provides all the infrastructure needed to detect when files or directories are deleted or renamed. There's still code needed on top of this (within "notmuch new") to actually do that detection.
2010-01-05 22:29:23 +01:00
notmuch_private_status_t
_notmuch_database_find_unique_doc_id (notmuch_database_t *notmuch,
const char *prefix_name,
const char *value,
unsigned int *doc_id)
database: Add private find_unique_doc_id and find_unique_document functions These are a generalization of the unique-ness testing of notmuch_database_find_message. More preparation for firectory timestamps.
2009-10-23 23:24:07 +02:00
{
Xapian::PostingIterator i, end;
lib: isolate n_d_add_message and helper functions into own file 'database.cc' is becoming a monster, and it's hard to follow what the various static functions are used for. It turns out that about 1/3 of this file notmuch_database_add_message and helper functions not used by any other function. This commit isolates this code into it's own file. Some side effects of this refactoring: - find_doc_ids becomes the non-static (but still private) _notmuch_database_find_doc_ids - a few instances of 'string' have 'std::' prepended, avoiding the need for 'using namespace std;' in the new file.
2017-06-04 14:32:24 +02:00
_notmuch_database_find_doc_ids (notmuch, prefix_name, value, &i, &end);
database: Add private find_unique_doc_id and find_unique_document functions These are a generalization of the unique-ness testing of notmuch_database_find_message. More preparation for firectory timestamps.
2009-10-23 23:24:07 +02:00
if (i == end) {
*doc_id = 0;
return NOTMUCH_PRIVATE_STATUS_NO_DOCUMENT_FOUND;
}
database: Make find_unique_doc_id enforce uniqueness (for a debug build) Catching any violation of this unique-ness constraint is very much in line with similar, existing INTERNAL_ERROR cases.
2009-12-22 00:11:32 +01:00
*doc_id = *i;
#if DEBUG_DATABASE_SANITY
i++;
if (i != end)
INTERNAL_ERROR ("Term %s:%s is not unique as expected.\n",
prefix_name, value);
#endif
return NOTMUCH_PRIVATE_STATUS_SUCCESS;
database: Add private find_unique_doc_id and find_unique_document functions These are a generalization of the unique-ness testing of notmuch_database_find_message. More preparation for firectory timestamps.
2009-10-23 23:24:07 +02:00
}
database: Similarly rename find_message_by_docid to find_document_for_doc_id Again preferring notmuch_database_t* over Xapian::Database*. Also, we're standardizing on "doc_id" rather than "docid" locally, (as an analoge to "message_id"), in spite of the "Xapian::docid" name, (which, fortunately, we can ignore and just us "unsigned int" instead).
2009-10-23 23:12:06 +02:00
static Xapian::Document
find_document_for_doc_id (notmuch_database_t *notmuch, unsigned doc_id)
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
{
database: Similarly rename find_message_by_docid to find_document_for_doc_id Again preferring notmuch_database_t* over Xapian::Database*. Also, we're standardizing on "doc_id" rather than "docid" locally, (as an analoge to "message_id"), in spite of the "Xapian::docid" name, (which, fortunately, we can ignore and just us "unsigned int" instead).
2009-10-23 23:12:06 +02:00
return notmuch->xapian_db->get_document (doc_id);
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
}
Add support (and tests) for messages with really long message IDs. Scott Henson reported an internal error that occurred when he tried to add a message that referenced another message with a message ID well over 300 characters in length. The bug here was running into a Xapian limit for the length of metadata key names, (which is even more restrictive than the Xapian limit for the length of terms). We fix this by noticing long message ID values and instead using a message ID of the form "notmuch-sha1-<sha1_sum_of_message_id>". That is, we use SHA1 to generate a compressed, (but still unique), version of the message ID. We add support to the test suite to exercise this fix. The tests add a message referencing the long message ID, then add the message with the long message ID, then finally add another message referencing the long ID. Each of these tests exercise different code paths where the special handling is implemented. A final test ensures that all three messages are stitched together into a single thread---guaranteeing that the three code paths all act consistently.
2010-06-04 21:39:36 +02:00
/* Generate a compressed version of 'message_id' of the form:
*
* notmuch-sha1-<sha1_sum_of_message_id>
*/
lib: Move message ID compression to _notmuch_message_create_for_message_id Previously, this was performed by notmuch_database_add_message. This happens to be the only caller currently (which is why this was safe), but we're about to introduce more callers, and it makes more sense to put responsibility for ID compression in the lower-level function rather than requiring each caller to handle it.
2014-10-07 01:17:07 +02:00
char *
_notmuch_message_id_compressed (void *ctx, const char *message_id)
Add support (and tests) for messages with really long message IDs. Scott Henson reported an internal error that occurred when he tried to add a message that referenced another message with a message ID well over 300 characters in length. The bug here was running into a Xapian limit for the length of metadata key names, (which is even more restrictive than the Xapian limit for the length of terms). We fix this by noticing long message ID values and instead using a message ID of the form "notmuch-sha1-<sha1_sum_of_message_id>". That is, we use SHA1 to generate a compressed, (but still unique), version of the message ID. We add support to the test suite to exercise this fix. The tests add a message referencing the long message ID, then add the message with the long message ID, then finally add another message referencing the long ID. Each of these tests exercise different code paths where the special handling is implemented. A final test ensures that all three messages are stitched together into a single thread---guaranteeing that the three code paths all act consistently.
2010-06-04 21:39:36 +02:00
{
char *sha1, *compressed;
lib: Start all function names in notmuch-private.h with As noted in devel/STYLE, every private library function should start with _notmuch. This patch corrects function naming that did not adhere to this style in lib/notmuch-private.h. In particular, the old function names that now begin with _notmuch are notmuch_sha1_of_file notmuch_sha1_of_string notmuch_message_file_close notmuch_message_file_get_header notmuch_message_file_open notmuch_message_get_author notmuch_message_set_author Signed-off-by: Charles Celerier <cceleri@cs.stanford.edu>
2014-05-13 11:44:05 +02:00
sha1 = _notmuch_sha1_of_string (message_id);
Add support (and tests) for messages with really long message IDs. Scott Henson reported an internal error that occurred when he tried to add a message that referenced another message with a message ID well over 300 characters in length. The bug here was running into a Xapian limit for the length of metadata key names, (which is even more restrictive than the Xapian limit for the length of terms). We fix this by noticing long message ID values and instead using a message ID of the form "notmuch-sha1-<sha1_sum_of_message_id>". That is, we use SHA1 to generate a compressed, (but still unique), version of the message ID. We add support to the test suite to exercise this fix. The tests add a message referencing the long message ID, then add the message with the long message ID, then finally add another message referencing the long ID. Each of these tests exercise different code paths where the special handling is implemented. A final test ensures that all three messages are stitched together into a single thread---guaranteeing that the three code paths all act consistently.
2010-06-04 21:39:36 +02:00
compressed = talloc_asprintf (ctx, "notmuch-sha1-%s", sha1);
free (sha1);
return compressed;
}
lib: make find_message{,by_filename) report errors Previously, the functions notmuch_database_find_message() and notmuch_database_find_message_by_filename() functions did not properly report error condition to the library user. For more information, read the thread on the notmuch mailing list starting with my mail "id:871uv2unfd.fsf@gmail.com" Make these functions accept a pointer to 'notmuch_message_t' as argument and return notmuch_status_t which may be used to check for any error condition. restore: Modify for the new notmuch_database_find_message() new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 06:55:29 +02:00
notmuch_status_t
database: Add new notmuch_database_find_message With this function, and the recently added support for notmuch_message_get_thread_ids, we now recode the find_thread_ids function to work just the way we expect a user of the public notmuch API to work. Not too bad really.
2009-10-22 00:37:51 +02:00
notmuch_database_find_message (notmuch_database_t *notmuch,
lib: make find_message{,by_filename) report errors Previously, the functions notmuch_database_find_message() and notmuch_database_find_message_by_filename() functions did not properly report error condition to the library user. For more information, read the thread on the notmuch mailing list starting with my mail "id:871uv2unfd.fsf@gmail.com" Make these functions accept a pointer to 'notmuch_message_t' as argument and return notmuch_status_t which may be used to check for any error condition. restore: Modify for the new notmuch_database_find_message() new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 06:55:29 +02:00
const char *message_id,
notmuch_message_t **message_ret)
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
{
database: Add private find_unique_doc_id and find_unique_document functions These are a generalization of the unique-ness testing of notmuch_database_find_message. More preparation for firectory timestamps.
2009-10-23 23:24:07 +02:00
notmuch_private_status_t status;
unsigned int doc_id;
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
lib: make find_message{,by_filename) report errors Previously, the functions notmuch_database_find_message() and notmuch_database_find_message_by_filename() functions did not properly report error condition to the library user. For more information, read the thread on the notmuch mailing list starting with my mail "id:871uv2unfd.fsf@gmail.com" Make these functions accept a pointer to 'notmuch_message_t' as argument and return notmuch_status_t which may be used to check for any error condition. restore: Modify for the new notmuch_database_find_message() new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 06:55:29 +02:00
if (message_ret == NULL)
return NOTMUCH_STATUS_NULL_POINTER;
Add support (and tests) for messages with really long message IDs. Scott Henson reported an internal error that occurred when he tried to add a message that referenced another message with a message ID well over 300 characters in length. The bug here was running into a Xapian limit for the length of metadata key names, (which is even more restrictive than the Xapian limit for the length of terms). We fix this by noticing long message ID values and instead using a message ID of the form "notmuch-sha1-<sha1_sum_of_message_id>". That is, we use SHA1 to generate a compressed, (but still unique), version of the message ID. We add support to the test suite to exercise this fix. The tests add a message referencing the long message ID, then add the message with the long message ID, then finally add another message referencing the long ID. Each of these tests exercise different code paths where the special handling is implemented. A final test ensures that all three messages are stitched together into a single thread---guaranteeing that the three code paths all act consistently.
2010-06-04 21:39:36 +02:00
if (strlen (message_id) > NOTMUCH_MESSAGE_ID_MAX)
lib: Move message ID compression to _notmuch_message_create_for_message_id Previously, this was performed by notmuch_database_add_message. This happens to be the only caller currently (which is why this was safe), but we're about to introduce more callers, and it makes more sense to put responsibility for ID compression in the lower-level function rather than requiring each caller to handle it.
2014-10-07 01:17:07 +02:00
message_id = _notmuch_message_id_compressed (notmuch, message_id);
Add support (and tests) for messages with really long message IDs. Scott Henson reported an internal error that occurred when he tried to add a message that referenced another message with a message ID well over 300 characters in length. The bug here was running into a Xapian limit for the length of metadata key names, (which is even more restrictive than the Xapian limit for the length of terms). We fix this by noticing long message ID values and instead using a message ID of the form "notmuch-sha1-<sha1_sum_of_message_id>". That is, we use SHA1 to generate a compressed, (but still unique), version of the message ID. We add support to the test suite to exercise this fix. The tests add a message referencing the long message ID, then add the message with the long message ID, then finally add another message referencing the long ID. Each of these tests exercise different code paths where the special handling is implemented. A final test ensures that all three messages are stitched together into a single thread---guaranteeing that the three code paths all act consistently.
2010-06-04 21:39:36 +02:00
lib: Audit all notmuch_database call for Xapian exception handling. Our current approach is for top-level entry poitns in the library to have try/catch blocks that catch any Xapian exception and print a message. Add a few missing blocks and fix up the documentation.
2010-04-24 16:22:34 +02:00
try {
status = _notmuch_database_find_unique_doc_id (notmuch, "id",
message_id, &doc_id);
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
lib: Audit all notmuch_database call for Xapian exception handling. Our current approach is for top-level entry poitns in the library to have try/catch blocks that catch any Xapian exception and print a message. Add a few missing blocks and fix up the documentation.
2010-04-24 16:22:34 +02:00
if (status == NOTMUCH_PRIVATE_STATUS_NO_DOCUMENT_FOUND)
lib: make find_message{,by_filename) report errors Previously, the functions notmuch_database_find_message() and notmuch_database_find_message_by_filename() functions did not properly report error condition to the library user. For more information, read the thread on the notmuch mailing list starting with my mail "id:871uv2unfd.fsf@gmail.com" Make these functions accept a pointer to 'notmuch_message_t' as argument and return notmuch_status_t which may be used to check for any error condition. restore: Modify for the new notmuch_database_find_message() new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 06:55:29 +02:00
*message_ret = NULL;
else {
*message_ret = _notmuch_message_create (notmuch, notmuch, doc_id,
NULL);
if (*message_ret == NULL)
return NOTMUCH_STATUS_OUT_OF_MEMORY;
}
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
lib: make find_message{,by_filename) report errors Previously, the functions notmuch_database_find_message() and notmuch_database_find_message_by_filename() functions did not properly report error condition to the library user. For more information, read the thread on the notmuch mailing list starting with my mail "id:871uv2unfd.fsf@gmail.com" Make these functions accept a pointer to 'notmuch_message_t' as argument and return notmuch_status_t which may be used to check for any error condition. restore: Modify for the new notmuch_database_find_message() new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 06:55:29 +02:00
return NOTMUCH_STATUS_SUCCESS;
lib: Audit all notmuch_database call for Xapian exception handling. Our current approach is for top-level entry poitns in the library to have try/catch blocks that catch any Xapian exception and print a message. Add a few missing blocks and fix up the documentation.
2010-04-24 16:22:34 +02:00
} catch (const Xapian::Error &error) {
lib: replace almost all fprintfs in library with _n_d_log This is not supposed to change any functionality from an end user point of view. Note that it will eliminate some output to stderr. The query debugging output is left as is; it doesn't really fit with the current primitive logging model. The remaining "bad" fprintf will need an internal API change.
2014-12-26 17:25:35 +01:00
_notmuch_database_log (notmuch, "A Xapian exception occurred finding message: %s.\n",
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
error.get_msg ().c_str ());
lib: convert notmuch_bool_t to stdbool internally C99 stdbool turned 18 this year. There really is no reason to use our own, except in the library interface for backward compatibility. Convert the lib internally to stdbool.
2017-10-07 10:44:05 +02:00
notmuch->exception_reported = true;
lib: make find_message{,by_filename) report errors Previously, the functions notmuch_database_find_message() and notmuch_database_find_message_by_filename() functions did not properly report error condition to the library user. For more information, read the thread on the notmuch mailing list starting with my mail "id:871uv2unfd.fsf@gmail.com" Make these functions accept a pointer to 'notmuch_message_t' as argument and return notmuch_status_t which may be used to check for any error condition. restore: Modify for the new notmuch_database_find_message() new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 06:55:29 +02:00
*message_ret = NULL;
return NOTMUCH_STATUS_XAPIAN_EXCEPTION;
lib: Audit all notmuch_database call for Xapian exception handling. Our current approach is for top-level entry poitns in the library to have try/catch blocks that catch any Xapian exception and print a message. Add a few missing blocks and fix up the documentation.
2010-04-24 16:22:34 +02:00
}
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
}
lib/cli: Make notmuch_database_create return a status code This is the notmuch_database_create equivalent of the previous change. In this case, there were places where errors were not being propagated correctly in notmuch_database_create or in calls to it. These have been fixed, using the new status value.
2012-04-30 18:25:34 +02:00
notmuch_status_t
notmuch_database_create (const char *path, notmuch_database_t **database)
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
{
char *status_string = NULL;
notmuch_status_t status;
status = notmuch_database_create_verbose (path, database,
&status_string);
if (status_string) {
fputs (status_string, stderr);
free (status_string);
}
return status;
}
notmuch_status_t
notmuch_database_create_verbose (const char *path,
notmuch_database_t **database,
char **status_string)
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
{
lib/cli: Make notmuch_database_create return a status code This is the notmuch_database_create equivalent of the previous change. In this case, there were places where errors were not being propagated correctly in notmuch_database_create or in calls to it. These have been fixed, using the new status value.
2012-04-30 18:25:34 +02:00
notmuch_status_t status = NOTMUCH_STATUS_SUCCESS;
notmuch_database_create/open: Fix to handle NULL as documented. When documenting these functions I described support for a NOTMUCH_BASE environment variable to be consulted in the case of a NULL path. Only, I had forgotten to actually write the code. This code exists now, with a new, exported function: notmuch_database_default_path
2009-10-20 18:56:25 +02:00
notmuch_database_t *notmuch = NULL;
char *notmuch_path = NULL;
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
char *message = NULL;
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
struct stat st;
int err;
notmuch_database_create/open: Fix to handle NULL as documented. When documenting these functions I described support for a NOTMUCH_BASE environment variable to be consulted in the case of a NULL path. Only, I had forgotten to actually write the code. This code exists now, with a new, exported function: notmuch_database_default_path
2009-10-20 18:56:25 +02:00
notmuch: Add a configuration system. This will allow for things like the database path to be specified without any cheesy NOTMUCH_BASE environment variable. It also will allow "notmuch reply" to recognize the user's email address when constructing a reply in order to do the right thing, (that is, to use the user's address to which mail was sent as From:, and not to reply to the user's own addresses). With this change, the "notmuch setup" command is now strictly for changing the configuration of notmuch. It no longer creates the database, but instead instructs the user to call "notmuch new" to do that.
2009-11-12 02:01:55 +01:00
if (path == NULL) {
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
message = strdup ("Error: Cannot create a database for a NULL path.\n");
lib/cli: Make notmuch_database_create return a status code This is the notmuch_database_create equivalent of the previous change. In this case, there were places where errors were not being propagated correctly in notmuch_database_create or in calls to it. These have been fixed, using the new status value.
2012-04-30 18:25:34 +02:00
status = NOTMUCH_STATUS_NULL_POINTER;
notmuch: Add a configuration system. This will allow for things like the database path to be specified without any cheesy NOTMUCH_BASE environment variable. It also will allow "notmuch reply" to recognize the user's email address when constructing a reply in order to do the right thing, (that is, to use the user's address to which mail was sent as From:, and not to reply to the user's own addresses). With this change, the "notmuch setup" command is now strictly for changing the configuration of notmuch. It no longer creates the database, but instead instructs the user to call "notmuch new" to do that.
2009-11-12 02:01:55 +01:00
goto DONE;
}
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
lib: reject relative paths in n_d_{create,open}_verbose There are many places in the notmuch code where the path is assumed to be absolute. If someone (TM) wants a project, one could remove these assumptions. In the mean time, prevent users from shooting themselves in the foot. Update test suite mark tests for this error as no longer broken, and also convert some tests that used relative paths for nonexistent directories.
2015-06-08 08:02:22 +02:00
if (path[0] != '/') {
message = strdup ("Error: Database path must be absolute.\n");
status = NOTMUCH_STATUS_PATH_ERROR;
goto DONE;
}
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
err = stat (path, &st);
if (err) {
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
IGNORE_RESULT (asprintf (&message, "Error: Cannot create database at %s: %s.\n",
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
path, strerror (errno)));
lib/cli: Make notmuch_database_create return a status code This is the notmuch_database_create equivalent of the previous change. In this case, there were places where errors were not being propagated correctly in notmuch_database_create or in calls to it. These have been fixed, using the new status value.
2012-04-30 18:25:34 +02:00
status = NOTMUCH_STATUS_FILE_ERROR;
notmuch_database_create/open: Fix to handle NULL as documented. When documenting these functions I described support for a NOTMUCH_BASE environment variable to be consulted in the case of a NULL path. Only, I had forgotten to actually write the code. This code exists now, with a new, exported function: notmuch_database_default_path
2009-10-20 18:56:25 +02:00
goto DONE;
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
}
if (! S_ISDIR (st.st_mode)) {
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
IGNORE_RESULT (asprintf (&message, "Error: Cannot create database at %s: "
"Not a directory.\n",
path));
lib/cli: Make notmuch_database_create return a status code This is the notmuch_database_create equivalent of the previous change. In this case, there were places where errors were not being propagated correctly in notmuch_database_create or in calls to it. These have been fixed, using the new status value.
2012-04-30 18:25:34 +02:00
status = NOTMUCH_STATUS_FILE_ERROR;
notmuch_database_create/open: Fix to handle NULL as documented. When documenting these functions I described support for a NOTMUCH_BASE environment variable to be consulted in the case of a NULL path. Only, I had forgotten to actually write the code. This code exists now, with a new, exported function: notmuch_database_default_path
2009-10-20 18:56:25 +02:00
goto DONE;
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
}
Remove all calls to g_strdup_printf Replacing them with calls to talloc_asprintf if possible, otherwise to asprintf (with it's painful error-handling leaving the pointer undefined).
2009-10-26 23:17:10 +01:00
notmuch_path = talloc_asprintf (NULL, "%s/%s", path, ".notmuch");
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
err = mkdir (notmuch_path, 0755);
if (err) {
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
IGNORE_RESULT (asprintf (&message, "Error: Cannot create directory %s: %s.\n",
notmuch_path, strerror (errno)));
lib/cli: Make notmuch_database_create return a status code This is the notmuch_database_create equivalent of the previous change. In this case, there were places where errors were not being propagated correctly in notmuch_database_create or in calls to it. These have been fixed, using the new status value.
2012-04-30 18:25:34 +02:00
status = NOTMUCH_STATUS_FILE_ERROR;
notmuch_database_create/open: Fix to handle NULL as documented. When documenting these functions I described support for a NOTMUCH_BASE environment variable to be consulted in the case of a NULL path. Only, I had forgotten to actually write the code. This code exists now, with a new, exported function: notmuch_database_default_path
2009-10-20 18:56:25 +02:00
goto DONE;
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
}
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
status = notmuch_database_open_verbose (path,
NOTMUCH_DATABASE_MODE_READ_WRITE,
&notmuch, &message);
lib/cli: Make notmuch_database_create return a status code This is the notmuch_database_create equivalent of the previous change. In this case, there were places where errors were not being propagated correctly in notmuch_database_create or in calls to it. These have been fixed, using the new status value.
2012-04-30 18:25:34 +02:00
if (status)
goto DONE;
lib: Database version 3: Introduce fine-grained "features" Previously, our database schema was versioned by a single number. Each database schema change had to occur "atomically" in Notmuch's development history: before some commit, Notmuch used version N, after that commit, it used version N+1. Hence, each new schema version could introduce only one change, the task of developing a schema change fell on a single person, and it all had to happen and be perfect in a single commit series. This made introducing a new schema version hard. We've seen only two schema changes in the history of Notmuch. This commit introduces database schema version 3; hopefully the last schema version we'll need for a while. With this version, we switch from a single version number to "features": a set of named, independent aspects of the database schema. Features should make backwards compatibility easier. For many things, it should be easy to support databases both with and without a feature, which will allow us to make upgrades optional and will enable "unstable" features that can be developed and tested over time. Features also make forwards compatibility easier. The features recorded in a database include "compatibility flags," which can indicate to an older version of Notmuch when it must support a given feature to open the database for read or for write. This lets us replace the old vague "I don't recognize this version, so something might go wrong, but I promise to try my best" warnings upon opening a database with an unknown version with precise errors. If a database is safe to open for read/write despite unknown features, an older version will know that and issue no message at all. If the database is not safe to open for read/write because of unknown features, an older version will know that, too, and can tell the user exactly which required features it lacks support for.
2014-08-25 19:26:00 +02:00
Add the NOTMUCH_FEATURE_INDEXED_MIMETYPES database feature This feature will exist in all newly created databases, but there is no upgrade provided for it. If this flag exists, it indicates that the database was created after the indexed MIME-types feature was added.
2015-01-23 00:43:37 +01:00
/* Upgrade doesn't add these feature to existing databases, but
* new databases have them. */
lib: Database version 3: Introduce fine-grained "features" Previously, our database schema was versioned by a single number. Each database schema change had to occur "atomically" in Notmuch's development history: before some commit, Notmuch used version N, after that commit, it used version N+1. Hence, each new schema version could introduce only one change, the task of developing a schema change fell on a single person, and it all had to happen and be perfect in a single commit series. This made introducing a new schema version hard. We've seen only two schema changes in the history of Notmuch. This commit introduces database schema version 3; hopefully the last schema version we'll need for a while. With this version, we switch from a single version number to "features": a set of named, independent aspects of the database schema. Features should make backwards compatibility easier. For many things, it should be easy to support databases both with and without a feature, which will allow us to make upgrades optional and will enable "unstable" features that can be developed and tested over time. Features also make forwards compatibility easier. The features recorded in a database include "compatibility flags," which can indicate to an older version of Notmuch when it must support a given feature to open the database for read or for write. This lets us replace the old vague "I don't recognize this version, so something might go wrong, but I promise to try my best" warnings upon opening a database with an unknown version with precise errors. If a database is safe to open for read/write despite unknown features, an older version will know that and issue no message at all. If the database is not safe to open for read/write because of unknown features, an older version will know that, too, and can tell the user exactly which required features it lacks support for.
2014-08-25 19:26:00 +02:00
notmuch->features |= NOTMUCH_FEATURE_FROM_SUBJECT_ID_VALUES;
Add the NOTMUCH_FEATURE_INDEXED_MIMETYPES database feature This feature will exist in all newly created databases, but there is no upgrade provided for it. If this flag exists, it indicates that the database was created after the indexed MIME-types feature was added.
2015-01-23 00:43:37 +01:00
notmuch->features |= NOTMUCH_FEATURE_INDEXED_MIMETYPES;
lib: add 'body:' field, stop indexing headers twice. The new `body:` field (in Xapian terms) or prefix (in slightly sloppier notmuch) terms allows matching terms that occur only in the body. Unprefixed query terms should continue to match anywhere (header or body) in the message. This follows a suggestion of Olly Betts to use the facility (since Xapian 1.0.4) to add the same field with multiple prefixes. The double indexing of previous versions is thus replaced with a query time expension of unprefixed query terms to the various prefixed equivalent. Reindexing will be needed for 'body:' searches to work correctly; otherwise they will also match messages where the term occur in headers (demonstrated by the new tests in T530-upgrade.sh)
2019-03-19 01:39:21 +01:00
notmuch->features |= NOTMUCH_FEATURE_UNPREFIX_BODY_ONLY;
lib: Database version 3: Introduce fine-grained "features" Previously, our database schema was versioned by a single number. Each database schema change had to occur "atomically" in Notmuch's development history: before some commit, Notmuch used version N, after that commit, it used version N+1. Hence, each new schema version could introduce only one change, the task of developing a schema change fell on a single person, and it all had to happen and be perfect in a single commit series. This made introducing a new schema version hard. We've seen only two schema changes in the history of Notmuch. This commit introduces database schema version 3; hopefully the last schema version we'll need for a while. With this version, we switch from a single version number to "features": a set of named, independent aspects of the database schema. Features should make backwards compatibility easier. For many things, it should be easy to support databases both with and without a feature, which will allow us to make upgrades optional and will enable "unstable" features that can be developed and tested over time. Features also make forwards compatibility easier. The features recorded in a database include "compatibility flags," which can indicate to an older version of Notmuch when it must support a given feature to open the database for read or for write. This lets us replace the old vague "I don't recognize this version, so something might go wrong, but I promise to try my best" warnings upon opening a database with an unknown version with precise errors. If a database is safe to open for read/write despite unknown features, an older version will know that and issue no message at all. If the database is not safe to open for read/write because of unknown features, an older version will know that, too, and can tell the user exactly which required features it lacks support for.
2014-08-25 19:26:00 +02:00
lib/cli: Make notmuch_database_create return a status code This is the notmuch_database_create equivalent of the previous change. In this case, there were places where errors were not being propagated correctly in notmuch_database_create or in calls to it. These have been fixed, using the new status value.
2012-04-30 18:25:34 +02:00
status = notmuch_database_upgrade (notmuch, NULL, NULL);
if (status) {
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
notmuch_database_close (notmuch);
lib/cli: Make notmuch_database_create return a status code This is the notmuch_database_create equivalent of the previous change. In this case, there were places where errors were not being propagated correctly in notmuch_database_create or in calls to it. These have been fixed, using the new status value.
2012-04-30 18:25:34 +02:00
notmuch = NULL;
}
notmuch_database_create/open: Fix to handle NULL as documented. When documenting these functions I described support for a NOTMUCH_BASE environment variable to be consulted in the case of a NULL path. Only, I had forgotten to actually write the code. This code exists now, with a new, exported function: notmuch_database_default_path
2009-10-20 18:56:25 +02:00
DONE:
if (notmuch_path)
Remove all calls to g_strdup_printf Replacing them with calls to talloc_asprintf if possible, otherwise to asprintf (with it's painful error-handling leaving the pointer undefined).
2009-10-26 23:17:10 +01:00
talloc_free (notmuch_path);
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
if (message) {
if (status_string)
*status_string = message;
else
free (message);
}
lib/cli: Make notmuch_database_create return a status code This is the notmuch_database_create equivalent of the previous change. In this case, there were places where errors were not being propagated correctly in notmuch_database_create or in calls to it. These have been fixed, using the new status value.
2012-04-30 18:25:34 +02:00
if (database)
*database = notmuch;
else
talloc_free (notmuch);
return status;
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
}
lib: Consolidate checks for read-only database. Previously, many checks were deep in the library just before a cast operation. These have now been replaced with internal errors and new checks have instead been added at the beginning of all top-levelentry points requiring a read-write database. The new checks now also use a single function for checking and printing the error message. This will give us a convenient location to extend the check, (such as based on database version as well).
2010-01-07 19:19:44 +01:00
notmuch_status_t
_notmuch_database_ensure_writable (notmuch_database_t *notmuch)
{
lib: encapsulate the use of notmuch_database_t field 'mode' The plan is to change the underlying representation.
2020-07-27 01:31:35 +02:00
if (_notmuch_database_mode (notmuch) == NOTMUCH_DATABASE_MODE_READ_ONLY) {
lib: replace almost all fprintfs in library with _n_d_log This is not supposed to change any functionality from an end user point of view. Note that it will eliminate some output to stderr. The query debugging output is left as is; it doesn't really fit with the current primitive logging model. The remaining "bad" fprintf will need an internal API change.
2014-12-26 17:25:35 +01:00
_notmuch_database_log (notmuch, "Cannot write to a read-only database.\n");
Prefer READ_ONLY consistently over READONLY. Previously we had NOTMUCH_DATABASE_MODE_READ_ONLY but NOTMUCH_STATUS_READONLY_DATABASE which was ugly and confusing. Rename the latter to NOTMUCH_STATUS_READ_ONLY_DATABASE for consistency.
2010-01-07 19:29:05 +01:00
return NOTMUCH_STATUS_READ_ONLY_DATABASE;
lib: Consolidate checks for read-only database. Previously, many checks were deep in the library just before a cast operation. These have now been replaced with internal errors and new checks have instead been added at the beginning of all top-levelentry points requiring a read-write database. The new checks now also use a single function for checking and printing the error message. This will give us a convenient location to extend the check, (such as based on database version as well).
2010-01-07 19:19:44 +01:00
}
return NOTMUCH_STATUS_SUCCESS;
}
lib: Add per-message last modification tracking This adds a new document value that stores the revision of the last modification to message metadata, where the revision number increases monotonically with each database commit. An alternative would be to store the wall-clock time of the last modification of each message. In principle this is simpler and has the advantage that any process can determine the current timestamp without support from libnotmuch. However, even assuming a computer's clock never goes backward and ignoring clock skew in networked environments, this has a fatal flaw. Xapian uses (optimistic) snapshot isolation, which means reads can be concurrent with writes. Given this, consider the following time line with a write and two read transactions: write |-X-A--------------| read 1 |---B---| read 2 |---| The write transaction modifies message X and records the wall-clock time of the modification at A. The writer hangs around for a while and later commits its change. Read 1 is concurrent with the write, so it doesn't see the change to X. It does some query and records the wall-clock time of its results at B. Transaction read 2 later starts after the write commits and queries for changes since wall-clock time B (say the reads are performing an incremental backup). Even though read 1 could not see the change to X, read 2 is told (correctly) that X has not changed since B, the time of the last read. In fact, X changed before wall-clock time A, but the change was not visible until *after* wall-clock time B, so read 2 misses the change to X. This is tricky to solve in full-blown snapshot isolation, but because Xapian serializes writes, we can use a simple, monotonically increasing database revision number. Furthermore, maintaining this revision number requires no more IO than a wall-clock time solution because Xapian already maintains statistics on the upper (and lower) bound of each value stream.
2014-10-13 08:20:01 +02:00
/* Allocate a revision number for the next change. */
unsigned long
_notmuch_database_new_revision (notmuch_database_t *notmuch)
{
unsigned long new_revision = notmuch->revision + 1;
/* If we're in an atomic section, hold off on updating the
* committed revision number until we commit the atomic section.
*/
if (notmuch->atomic_nesting)
lib: convert notmuch_bool_t to stdbool internally C99 stdbool turned 18 this year. There really is no reason to use our own, except in the library interface for backward compatibility. Convert the lib internally to stdbool.
2017-10-07 10:44:05 +02:00
notmuch->atomic_dirty = true;
lib: Add per-message last modification tracking This adds a new document value that stores the revision of the last modification to message metadata, where the revision number increases monotonically with each database commit. An alternative would be to store the wall-clock time of the last modification of each message. In principle this is simpler and has the advantage that any process can determine the current timestamp without support from libnotmuch. However, even assuming a computer's clock never goes backward and ignoring clock skew in networked environments, this has a fatal flaw. Xapian uses (optimistic) snapshot isolation, which means reads can be concurrent with writes. Given this, consider the following time line with a write and two read transactions: write |-X-A--------------| read 1 |---B---| read 2 |---| The write transaction modifies message X and records the wall-clock time of the modification at A. The writer hangs around for a while and later commits its change. Read 1 is concurrent with the write, so it doesn't see the change to X. It does some query and records the wall-clock time of its results at B. Transaction read 2 later starts after the write commits and queries for changes since wall-clock time B (say the reads are performing an incremental backup). Even though read 1 could not see the change to X, read 2 is told (correctly) that X has not changed since B, the time of the last read. In fact, X changed before wall-clock time A, but the change was not visible until *after* wall-clock time B, so read 2 misses the change to X. This is tricky to solve in full-blown snapshot isolation, but because Xapian serializes writes, we can use a simple, monotonically increasing database revision number. Furthermore, maintaining this revision number requires no more IO than a wall-clock time solution because Xapian already maintains statistics on the upper (and lower) bound of each value stream.
2014-10-13 08:20:01 +02:00
else
notmuch->revision = new_revision;
return new_revision;
}
lib/cli: Make notmuch_database_open return a status code It has been a long-standing issue that notmuch_database_open doesn't return any indication of why it failed. This patch changes its prototype to return a notmuch_status_t and set an out-argument to the database itself, like other functions that return both a status and an object. In the interest of atomicity, this also updates every use in the CLI so that notmuch still compiles. Since this patch does not update the bindings, the Python bindings test fails.
2012-04-30 18:25:33 +02:00
notmuch_status_t
Permit opening the notmuch database in read-only mode. We only rarely need to actually open the database for writing, but we always create a Xapian::WritableDatabase. This has the effect of preventing searches and like whilst updating the index. Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Acked-by: Carl Worth <cworth@cworth.org>
2009-11-21 20:54:25 +01:00
notmuch_database_open (const char *path,
lib/cli: Make notmuch_database_open return a status code It has been a long-standing issue that notmuch_database_open doesn't return any indication of why it failed. This patch changes its prototype to return a notmuch_status_t and set an out-argument to the database itself, like other functions that return both a status and an object. In the interest of atomicity, this also updates every use in the CLI so that notmuch still compiles. Since this patch does not update the bindings, the Python bindings test fails.
2012-04-30 18:25:33 +02:00
notmuch_database_mode_t mode,
notmuch_database_t **database)
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
{
char *status_string = NULL;
notmuch_status_t status;
status = notmuch_database_open_verbose (path, mode, database,
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
&status_string);
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
if (status_string) {
fputs (status_string, stderr);
free (status_string);
}
return status;
}
notmuch_status_t
notmuch_database_open_verbose (const char *path,
notmuch_database_mode_t mode,
notmuch_database_t **database,
char **status_string)
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
{
lib/cli: Make notmuch_database_open return a status code It has been a long-standing issue that notmuch_database_open doesn't return any indication of why it failed. This patch changes its prototype to return a notmuch_status_t and set an out-argument to the database itself, like other functions that return both a status and an object. In the interest of atomicity, this also updates every use in the CLI so that notmuch still compiles. Since this patch does not update the bindings, the Python bindings test fails.
2012-04-30 18:25:33 +02:00
notmuch_status_t status = NOTMUCH_STATUS_SUCCESS;
lib: Use talloc to simplify cleanup in notmuch_database_open Previously, we manually "free"d various pointers in notmuch_database_open. Use a local talloc context instead to simplify cleanup and eliminate various NULL pointer initializations and conditionals.
2012-01-29 06:50:10 +01:00
void *local = talloc_new (NULL);
notmuch_database_create/open: Fix to handle NULL as documented. When documenting these functions I described support for a NOTMUCH_BASE environment variable to be consulted in the case of a NULL path. Only, I had forgotten to actually write the code. This code exists now, with a new, exported function: notmuch_database_default_path
2009-10-20 18:56:25 +02:00
notmuch_database_t *notmuch = NULL;
lib: Database version 3: Introduce fine-grained "features" Previously, our database schema was versioned by a single number. Each database schema change had to occur "atomically" in Notmuch's development history: before some commit, Notmuch used version N, after that commit, it used version N+1. Hence, each new schema version could introduce only one change, the task of developing a schema change fell on a single person, and it all had to happen and be perfect in a single commit series. This made introducing a new schema version hard. We've seen only two schema changes in the history of Notmuch. This commit introduces database schema version 3; hopefully the last schema version we'll need for a while. With this version, we switch from a single version number to "features": a set of named, independent aspects of the database schema. Features should make backwards compatibility easier. For many things, it should be easy to support databases both with and without a feature, which will allow us to make upgrades optional and will enable "unstable" features that can be developed and tested over time. Features also make forwards compatibility easier. The features recorded in a database include "compatibility flags," which can indicate to an older version of Notmuch when it must support a given feature to open the database for read or for write. This lets us replace the old vague "I don't recognize this version, so something might go wrong, but I promise to try my best" warnings upon opening a database with an unknown version with precise errors. If a database is safe to open for read/write despite unknown features, an older version will know that and issue no message at all. If the database is not safe to open for read/write because of unknown features, an older version will know that, too, and can tell the user exactly which required features it lacks support for.
2014-08-25 19:26:00 +02:00
char *notmuch_path, *xapian_path, *incompat_features;
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
char *message = NULL;
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
struct stat st;
int err;
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
unsigned int i, version;
lib: call g_mime_init() from notmuch_database_open() As reported in id:"CAEbOPGyuHnz4BPtDutnTPUHcP3eYcRCRkXhYoJR43RUMw671+g@mail.gmail.com" sometimes gmime tries to access a NULL pointer, e.g. g_mime_iconv_open() tries to access iconv_cache that is NULL if g_mime_init() is not called. This causes notmuch to segfault when calling gmime functions. Calling g_mime_init() initializes iconv_cache and others variables needed by gmime, making sure they are initialized when notmuch calls gmime functions. Test marked fix by db.
2011-12-31 05:37:41 +01:00
static int initialized = 0;
notmuch_database_create/open: Fix to handle NULL as documented. When documenting these functions I described support for a NOTMUCH_BASE environment variable to be consulted in the case of a NULL path. Only, I had forgotten to actually write the code. This code exists now, with a new, exported function: notmuch_database_default_path
2009-10-20 18:56:25 +02:00
lib/cli: Make notmuch_database_open return a status code It has been a long-standing issue that notmuch_database_open doesn't return any indication of why it failed. This patch changes its prototype to return a notmuch_status_t and set an out-argument to the database itself, like other functions that return both a status and an object. In the interest of atomicity, this also updates every use in the CLI so that notmuch still compiles. Since this patch does not update the bindings, the Python bindings test fails.
2012-04-30 18:25:33 +02:00
if (path == NULL) {
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
message = strdup ("Error: Cannot open a database for a NULL path.\n");
lib/cli: Make notmuch_database_open return a status code It has been a long-standing issue that notmuch_database_open doesn't return any indication of why it failed. This patch changes its prototype to return a notmuch_status_t and set an out-argument to the database itself, like other functions that return both a status and an object. In the interest of atomicity, this also updates every use in the CLI so that notmuch still compiles. Since this patch does not update the bindings, the Python bindings test fails.
2012-04-30 18:25:33 +02:00
status = NOTMUCH_STATUS_NULL_POINTER;
goto DONE;
}
lib: reject relative paths in n_d_{create,open}_verbose There are many places in the notmuch code where the path is assumed to be absolute. If someone (TM) wants a project, one could remove these assumptions. In the mean time, prevent users from shooting themselves in the foot. Update test suite mark tests for this error as no longer broken, and also convert some tests that used relative paths for nonexistent directories.
2015-06-08 08:02:22 +02:00
if (path[0] != '/') {
message = strdup ("Error: Database path must be absolute.\n");
status = NOTMUCH_STATUS_PATH_ERROR;
goto DONE;
}
lib: Use talloc to simplify cleanup in notmuch_database_open Previously, we manually "free"d various pointers in notmuch_database_open. Use a local talloc context instead to simplify cleanup and eliminate various NULL pointer initializations and conditionals.
2012-01-29 06:50:10 +01:00
if (! (notmuch_path = talloc_asprintf (local, "%s/%s", path, ".notmuch"))) {
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
message = strdup ("Out of memory\n");
lib/cli: Make notmuch_database_open return a status code It has been a long-standing issue that notmuch_database_open doesn't return any indication of why it failed. This patch changes its prototype to return a notmuch_status_t and set an out-argument to the database itself, like other functions that return both a status and an object. In the interest of atomicity, this also updates every use in the CLI so that notmuch still compiles. Since this patch does not update the bindings, the Python bindings test fails.
2012-04-30 18:25:33 +02:00
status = NOTMUCH_STATUS_OUT_OF_MEMORY;
Remove all calls to g_strdup_printf Replacing them with calls to talloc_asprintf if possible, otherwise to asprintf (with it's painful error-handling leaving the pointer undefined).
2009-10-26 23:17:10 +01:00
goto DONE;
}
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
err = stat (notmuch_path, &st);
if (err) {
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
IGNORE_RESULT (asprintf (&message, "Error opening database at %s: %s\n",
notmuch_path, strerror (errno)));
lib/cli: Make notmuch_database_open return a status code It has been a long-standing issue that notmuch_database_open doesn't return any indication of why it failed. This patch changes its prototype to return a notmuch_status_t and set an out-argument to the database itself, like other functions that return both a status and an object. In the interest of atomicity, this also updates every use in the CLI so that notmuch still compiles. Since this patch does not update the bindings, the Python bindings test fails.
2012-04-30 18:25:33 +02:00
status = NOTMUCH_STATUS_FILE_ERROR;
notmuch_database_create/open: Fix to handle NULL as documented. When documenting these functions I described support for a NOTMUCH_BASE environment variable to be consulted in the case of a NULL path. Only, I had forgotten to actually write the code. This code exists now, with a new, exported function: notmuch_database_default_path
2009-10-20 18:56:25 +02:00
goto DONE;
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
}
lib: Use talloc to simplify cleanup in notmuch_database_open Previously, we manually "free"d various pointers in notmuch_database_open. Use a local talloc context instead to simplify cleanup and eliminate various NULL pointer initializations and conditionals.
2012-01-29 06:50:10 +01:00
if (! (xapian_path = talloc_asprintf (local, "%s/%s", notmuch_path, "xapian"))) {
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
message = strdup ("Out of memory\n");
lib/cli: Make notmuch_database_open return a status code It has been a long-standing issue that notmuch_database_open doesn't return any indication of why it failed. This patch changes its prototype to return a notmuch_status_t and set an out-argument to the database itself, like other functions that return both a status and an object. In the interest of atomicity, this also updates every use in the CLI so that notmuch still compiles. Since this patch does not update the bindings, the Python bindings test fails.
2012-04-30 18:25:33 +02:00
status = NOTMUCH_STATUS_OUT_OF_MEMORY;
Remove all calls to g_strdup_printf Replacing them with calls to talloc_asprintf if possible, otherwise to asprintf (with it's painful error-handling leaving the pointer undefined).
2009-10-26 23:17:10 +01:00
goto DONE;
}
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
lib: call g_type_init from notmuch_database_open We want to make sure g_type_init is called before any GObject functionality is used.
2011-12-04 20:20:39 +01:00
/* Initialize the GLib type system and threads */
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
#if ! GLIB_CHECK_VERSION (2, 35, 1)
lib: call g_type_init from notmuch_database_open We want to make sure g_type_init is called before any GObject functionality is used.
2011-12-04 20:20:39 +01:00
g_type_init ();
cli: Guard deprecated g_type_init calls g_type_init was deprecated in GLib 2.35.1. In order to compile cleanly, guard these with a suitable #if. (commit msg from https://bugs.freedesktop.org/attachment.cgi?id=73774 )
2013-06-07 20:41:13 +02:00
#endif
lib: call g_type_init from notmuch_database_open We want to make sure g_type_init is called before any GObject functionality is used.
2011-12-04 20:20:39 +01:00
lib: call g_mime_init() from notmuch_database_open() As reported in id:"CAEbOPGyuHnz4BPtDutnTPUHcP3eYcRCRkXhYoJR43RUMw671+g@mail.gmail.com" sometimes gmime tries to access a NULL pointer, e.g. g_mime_iconv_open() tries to access iconv_cache that is NULL if g_mime_init() is not called. This causes notmuch to segfault when calling gmime functions. Calling g_mime_init() initializes iconv_cache and others variables needed by gmime, making sure they are initialized when notmuch calls gmime functions. Test marked fix by db.
2011-12-31 05:37:41 +01:00
/* Initialize gmime */
if (! initialized) {
gmime-cleanup: drop unused gmime #defines and simplify g_mime_init () Several of these #defines were not actually used in the notmuch codebase any longer. And as of GMime 3.0, g_mime_init takes no arguments, so we can also drop the bogus RFC2047 argument that we were passing and then #defining away. signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-02 15:19:42 +02:00
g_mime_init ();
lib: call g_mime_init() from notmuch_database_open() As reported in id:"CAEbOPGyuHnz4BPtDutnTPUHcP3eYcRCRkXhYoJR43RUMw671+g@mail.gmail.com" sometimes gmime tries to access a NULL pointer, e.g. g_mime_iconv_open() tries to access iconv_cache that is NULL if g_mime_init() is not called. This causes notmuch to segfault when calling gmime functions. Calling g_mime_init() initializes iconv_cache and others variables needed by gmime, making sure they are initialized when notmuch calls gmime functions. Test marked fix by db.
2011-12-31 05:37:41 +01:00
initialized = 1;
}
lib: Don't delete uninitialized pointers In the error-handling paths of notmuch_database_open, we call notmuch_database_close, which "delete"s several objects referenced by the notmuch_database_t object. However, some of these pointers may be uninitialized, resulting in undefined behavior. Hence, allocate the notmuch_database_t with talloc_zero to make sure these pointers are NULL so that "delete"ing them is harmless.
2012-01-29 06:50:08 +01:00
notmuch = talloc_zero (NULL, notmuch_database_t);
lib: convert notmuch_bool_t to stdbool internally C99 stdbool turned 18 this year. There really is no reason to use our own, except in the library interface for backward compatibility. Convert the lib internally to stdbool.
2017-10-07 10:44:05 +02:00
notmuch->exception_reported = false;
lib: add a log function with output to a string in notmuch_database_t In principle in the future this could do something fancier than asprintf.
2014-12-26 09:01:01 +01:00
notmuch->status_string = NULL;
Convert notmuch_database_t to start using talloc. This will be handy as we can hang future talloc allocations off of the datbase now.
2009-10-21 23:00:37 +02:00
notmuch->path = talloc_strdup (notmuch, path);
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
strip_trailing (notmuch->path, '/');
Fix add_message and get_filename to strip/re-add the database path. We now store only a relative path inside the database so the database is not nicely relocatable.
2009-10-29 00:50:14 +01:00
lib: replace use of static_cast for writable databases static_cast is a bit tricky to understand and error prone, so add a second pointer to (potentially the same) Xapian database object that we know has the right subclass.
2020-07-27 01:31:36 +02:00
notmuch->writable_xapian_db = NULL;
lib: Add support for nested atomic sections. notmuch_database_t now keeps a nesting count and we only start a transaction or commit for the outermost atomic section. Introduces a new error, NOTMUCH_STATUS_UNBALANCED_ATOMIC.
2011-06-11 05:35:06 +02:00
notmuch->atomic_nesting = 0;
lib: add _notmuch_database_reopen The main expected use is to recover from a Xapian::DatabaseChanged exception.
2017-02-24 02:38:24 +01:00
notmuch->view = 1;
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
try {
Switch from random to sequential thread identifiers. The sequential identifiers have the advantage of being guaranteed to be unique (until we overflow a 64-bit unsigned integer), and also take up half as much space in the "notmuch search" output (16 columns rather than 32). This change also has the side effect of fixing a bug where notmuch could block on /dev/random at startup (waiting for some entropy to appear). This bug was hit hard by the test suite, (which could easily exhaust the available entropy on common systems---resulting in large delays of the test suite).
2010-02-08 20:33:33 +01:00
string last_thread_id;
lib: Add per-message last modification tracking This adds a new document value that stores the revision of the last modification to message metadata, where the revision number increases monotonically with each database commit. An alternative would be to store the wall-clock time of the last modification of each message. In principle this is simpler and has the advantage that any process can determine the current timestamp without support from libnotmuch. However, even assuming a computer's clock never goes backward and ignoring clock skew in networked environments, this has a fatal flaw. Xapian uses (optimistic) snapshot isolation, which means reads can be concurrent with writes. Given this, consider the following time line with a write and two read transactions: write |-X-A--------------| read 1 |---B---| read 2 |---| The write transaction modifies message X and records the wall-clock time of the modification at A. The writer hangs around for a while and later commits its change. Read 1 is concurrent with the write, so it doesn't see the change to X. It does some query and records the wall-clock time of its results at B. Transaction read 2 later starts after the write commits and queries for changes since wall-clock time B (say the reads are performing an incremental backup). Even though read 1 could not see the change to X, read 2 is told (correctly) that X has not changed since B, the time of the last read. In fact, X changed before wall-clock time A, but the change was not visible until *after* wall-clock time B, so read 2 misses the change to X. This is tricky to solve in full-blown snapshot isolation, but because Xapian serializes writes, we can use a simple, monotonically increasing database revision number. Furthermore, maintaining this revision number requires no more IO than a wall-clock time solution because Xapian already maintains statistics on the upper (and lower) bound of each value stream.
2014-10-13 08:20:01 +02:00
string last_mod;
Switch from random to sequential thread identifiers. The sequential identifiers have the advantage of being guaranteed to be unique (until we overflow a 64-bit unsigned integer), and also take up half as much space in the "notmuch search" output (16 columns rather than 32). This change also has the side effect of fixing a bug where notmuch could block on /dev/random at startup (waiting for some entropy to appear). This bug was hit hard by the test suite, (which could easily exhaust the available entropy on common systems---resulting in large delays of the test suite).
2010-02-08 20:33:33 +01:00
Rename NOTMUCH_DATABASE_MODE_WRITABLE to NOTMUCH_DATABASE_MODE_READ_WRITE And correspondingly, READONLY to READ_ONLY.
2009-11-21 22:10:18 +01:00
if (mode == NOTMUCH_DATABASE_MODE_READ_WRITE) {
lib: replace use of static_cast for writable databases static_cast is a bit tricky to understand and error prone, so add a second pointer to (potentially the same) Xapian database object that we know has the right subclass.
2020-07-27 01:31:36 +02:00
notmuch->writable_xapian_db = new Xapian::WritableDatabase (xapian_path,
DB_ACTION);
notmuch->xapian_db = notmuch->writable_xapian_db;
Permit opening the notmuch database in read-only mode. We only rarely need to actually open the database for writing, but we always create a Xapian::WritableDatabase. This has the effect of preventing searches and like whilst updating the index. Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Acked-by: Carl Worth <cworth@cworth.org>
2009-11-21 20:54:25 +01:00
} else {
notmuch->xapian_db = new Xapian::Database (xapian_path);
lib: Database version 3: Introduce fine-grained "features" Previously, our database schema was versioned by a single number. Each database schema change had to occur "atomically" in Notmuch's development history: before some commit, Notmuch used version N, after that commit, it used version N+1. Hence, each new schema version could introduce only one change, the task of developing a schema change fell on a single person, and it all had to happen and be perfect in a single commit series. This made introducing a new schema version hard. We've seen only two schema changes in the history of Notmuch. This commit introduces database schema version 3; hopefully the last schema version we'll need for a while. With this version, we switch from a single version number to "features": a set of named, independent aspects of the database schema. Features should make backwards compatibility easier. For many things, it should be easy to support databases both with and without a feature, which will allow us to make upgrades optional and will enable "unstable" features that can be developed and tested over time. Features also make forwards compatibility easier. The features recorded in a database include "compatibility flags," which can indicate to an older version of Notmuch when it must support a given feature to open the database for read or for write. This lets us replace the old vague "I don't recognize this version, so something might go wrong, but I promise to try my best" warnings upon opening a database with an unknown version with precise errors. If a database is safe to open for read/write despite unknown features, an older version will know that and issue no message at all. If the database is not safe to open for read/write because of unknown features, an older version will know that, too, and can tell the user exactly which required features it lacks support for.
2014-08-25 19:26:00 +02:00
}
/* Check version. As of database version 3, we represent
* changes in terms of features, so assume a version bump
* means a dramatically incompatible change. */
version = notmuch_database_get_version (notmuch);
if (version > NOTMUCH_DATABASE_VERSION) {
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
IGNORE_RESULT (asprintf (&message,
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
"Error: Notmuch database at %s\n"
" has a newer database format version (%u) than supported by this\n"
" version of notmuch (%u).\n",
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
notmuch_path, version, NOTMUCH_DATABASE_VERSION));
lib: Database version 3: Introduce fine-grained "features" Previously, our database schema was versioned by a single number. Each database schema change had to occur "atomically" in Notmuch's development history: before some commit, Notmuch used version N, after that commit, it used version N+1. Hence, each new schema version could introduce only one change, the task of developing a schema change fell on a single person, and it all had to happen and be perfect in a single commit series. This made introducing a new schema version hard. We've seen only two schema changes in the history of Notmuch. This commit introduces database schema version 3; hopefully the last schema version we'll need for a while. With this version, we switch from a single version number to "features": a set of named, independent aspects of the database schema. Features should make backwards compatibility easier. For many things, it should be easy to support databases both with and without a feature, which will allow us to make upgrades optional and will enable "unstable" features that can be developed and tested over time. Features also make forwards compatibility easier. The features recorded in a database include "compatibility flags," which can indicate to an older version of Notmuch when it must support a given feature to open the database for read or for write. This lets us replace the old vague "I don't recognize this version, so something might go wrong, but I promise to try my best" warnings upon opening a database with an unknown version with precise errors. If a database is safe to open for read/write despite unknown features, an older version will know that and issue no message at all. If the database is not safe to open for read/write because of unknown features, an older version will know that, too, and can tell the user exactly which required features it lacks support for.
2014-08-25 19:26:00 +02:00
notmuch_database_destroy (notmuch);
notmuch = NULL;
status = NOTMUCH_STATUS_FILE_ERROR;
goto DONE;
}
/* Check features. */
incompat_features = NULL;
lib: factor out feature name related code. database.cc is uncomfortably large, and some of the static data structures do not need to be shared as much as they are. This is a somewhat small piece to factor out, but it will turn out to be helpful to further refactoring.
2020-08-08 16:16:43 +02:00
notmuch->features = _notmuch_database_parse_features (
lib: Database version 3: Introduce fine-grained "features" Previously, our database schema was versioned by a single number. Each database schema change had to occur "atomically" in Notmuch's development history: before some commit, Notmuch used version N, after that commit, it used version N+1. Hence, each new schema version could introduce only one change, the task of developing a schema change fell on a single person, and it all had to happen and be perfect in a single commit series. This made introducing a new schema version hard. We've seen only two schema changes in the history of Notmuch. This commit introduces database schema version 3; hopefully the last schema version we'll need for a while. With this version, we switch from a single version number to "features": a set of named, independent aspects of the database schema. Features should make backwards compatibility easier. For many things, it should be easy to support databases both with and without a feature, which will allow us to make upgrades optional and will enable "unstable" features that can be developed and tested over time. Features also make forwards compatibility easier. The features recorded in a database include "compatibility flags," which can indicate to an older version of Notmuch when it must support a given feature to open the database for read or for write. This lets us replace the old vague "I don't recognize this version, so something might go wrong, but I promise to try my best" warnings upon opening a database with an unknown version with precise errors. If a database is safe to open for read/write despite unknown features, an older version will know that and issue no message at all. If the database is not safe to open for read/write because of unknown features, an older version will know that, too, and can tell the user exactly which required features it lacks support for.
2014-08-25 19:26:00 +02:00
local, notmuch->xapian_db->get_metadata ("features").c_str (),
version, mode == NOTMUCH_DATABASE_MODE_READ_WRITE ? 'w' : 'r',
&incompat_features);
if (incompat_features) {
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
IGNORE_RESULT (asprintf (&message,
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
"Error: Notmuch database at %s\n"
" requires features (%s)\n"
" not supported by this version of notmuch.\n",
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
notmuch_path, incompat_features));
lib: Database version 3: Introduce fine-grained "features" Previously, our database schema was versioned by a single number. Each database schema change had to occur "atomically" in Notmuch's development history: before some commit, Notmuch used version N, after that commit, it used version N+1. Hence, each new schema version could introduce only one change, the task of developing a schema change fell on a single person, and it all had to happen and be perfect in a single commit series. This made introducing a new schema version hard. We've seen only two schema changes in the history of Notmuch. This commit introduces database schema version 3; hopefully the last schema version we'll need for a while. With this version, we switch from a single version number to "features": a set of named, independent aspects of the database schema. Features should make backwards compatibility easier. For many things, it should be easy to support databases both with and without a feature, which will allow us to make upgrades optional and will enable "unstable" features that can be developed and tested over time. Features also make forwards compatibility easier. The features recorded in a database include "compatibility flags," which can indicate to an older version of Notmuch when it must support a given feature to open the database for read or for write. This lets us replace the old vague "I don't recognize this version, so something might go wrong, but I promise to try my best" warnings upon opening a database with an unknown version with precise errors. If a database is safe to open for read/write despite unknown features, an older version will know that and issue no message at all. If the database is not safe to open for read/write because of unknown features, an older version will know that, too, and can tell the user exactly which required features it lacks support for.
2014-08-25 19:26:00 +02:00
notmuch_database_destroy (notmuch);
notmuch = NULL;
status = NOTMUCH_STATUS_FILE_ERROR;
goto DONE;
Permit opening the notmuch database in read-only mode. We only rarely need to actually open the database for writing, but we always create a Xapian::WritableDatabase. This has the effect of preventing searches and like whilst updating the index. Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Acked-by: Carl Worth <cworth@cworth.org>
2009-11-21 20:54:25 +01:00
}
Switch from random to sequential thread identifiers. The sequential identifiers have the advantage of being guaranteed to be unique (until we overflow a 64-bit unsigned integer), and also take up half as much space in the "notmuch search" output (16 columns rather than 32). This change also has the side effect of fixing a bug where notmuch could block on /dev/random at startup (waiting for some entropy to appear). This bug was hit hard by the test suite, (which could easily exhaust the available entropy on common systems---resulting in large delays of the test suite).
2010-02-08 20:33:33 +01:00
Avoid database corruption by not adding partially-constructed mail documents. Previously we were using Xapian's add_document to allocate document ID values for notmuch_message_t objects. This had the drawback of adding a partially constructed mail document to the database. If notmuch was subsequently interrupted before fully populating this document, then later runs would be quite confused when seeing the partial documents. There are reports from the wild of people hitting internal errors of the form "Message ... has no thread ID" for example, (which is currently an unrecoverable error). We fix this by manually allocating document IDs without adding documents. With this change, we never call Xapian's add_document method, but only replace_document with either the current document ID of a message or a new one that we have allocated.
2010-06-04 19:16:53 +02:00
notmuch->last_doc_id = notmuch->xapian_db->get_lastdocid ();
Switch from random to sequential thread identifiers. The sequential identifiers have the advantage of being guaranteed to be unique (until we overflow a 64-bit unsigned integer), and also take up half as much space in the "notmuch search" output (16 columns rather than 32). This change also has the side effect of fixing a bug where notmuch could block on /dev/random at startup (waiting for some entropy to appear). This bug was hit hard by the test suite, (which could easily exhaust the available entropy on common systems---resulting in large delays of the test suite).
2010-02-08 20:33:33 +01:00
last_thread_id = notmuch->xapian_db->get_metadata ("last_thread_id");
if (last_thread_id.empty ()) {
notmuch->last_thread_id = 0;
} else {
const char *str;
char *end;
str = last_thread_id.c_str ();
notmuch->last_thread_id = strtoull (str, &end, 16);
if (*end != '\0')
INTERNAL_ERROR ("Malformed database last_thread_id: %s", str);
}
lib: Add per-message last modification tracking This adds a new document value that stores the revision of the last modification to message metadata, where the revision number increases monotonically with each database commit. An alternative would be to store the wall-clock time of the last modification of each message. In principle this is simpler and has the advantage that any process can determine the current timestamp without support from libnotmuch. However, even assuming a computer's clock never goes backward and ignoring clock skew in networked environments, this has a fatal flaw. Xapian uses (optimistic) snapshot isolation, which means reads can be concurrent with writes. Given this, consider the following time line with a write and two read transactions: write |-X-A--------------| read 1 |---B---| read 2 |---| The write transaction modifies message X and records the wall-clock time of the modification at A. The writer hangs around for a while and later commits its change. Read 1 is concurrent with the write, so it doesn't see the change to X. It does some query and records the wall-clock time of its results at B. Transaction read 2 later starts after the write commits and queries for changes since wall-clock time B (say the reads are performing an incremental backup). Even though read 1 could not see the change to X, read 2 is told (correctly) that X has not changed since B, the time of the last read. In fact, X changed before wall-clock time A, but the change was not visible until *after* wall-clock time B, so read 2 misses the change to X. This is tricky to solve in full-blown snapshot isolation, but because Xapian serializes writes, we can use a simple, monotonically increasing database revision number. Furthermore, maintaining this revision number requires no more IO than a wall-clock time solution because Xapian already maintains statistics on the upper (and lower) bound of each value stream.
2014-10-13 08:20:01 +02:00
/* Get current highest revision number. */
last_mod = notmuch->xapian_db->get_value_upper_bound (
NOTMUCH_VALUE_LAST_MOD);
if (last_mod.empty ())
notmuch->revision = 0;
else
notmuch->revision = Xapian::sortable_unserialise (last_mod);
lib: API to retrieve database revision and UUID This exposes the committed database revision to library users along with a UUID that can be used to detect when revision numbers are no longer comparable (e.g., because the database has been replaced).
2014-10-13 08:20:02 +02:00
notmuch->uuid = talloc_strdup (
notmuch, notmuch->xapian_db->get_uuid ().c_str ());
lib: Add per-message last modification tracking This adds a new document value that stores the revision of the last modification to message metadata, where the revision number increases monotonically with each database commit. An alternative would be to store the wall-clock time of the last modification of each message. In principle this is simpler and has the advantage that any process can determine the current timestamp without support from libnotmuch. However, even assuming a computer's clock never goes backward and ignoring clock skew in networked environments, this has a fatal flaw. Xapian uses (optimistic) snapshot isolation, which means reads can be concurrent with writes. Given this, consider the following time line with a write and two read transactions: write |-X-A--------------| read 1 |---B---| read 2 |---| The write transaction modifies message X and records the wall-clock time of the modification at A. The writer hangs around for a while and later commits its change. Read 1 is concurrent with the write, so it doesn't see the change to X. It does some query and records the wall-clock time of its results at B. Transaction read 2 later starts after the write commits and queries for changes since wall-clock time B (say the reads are performing an incremental backup). Even though read 1 could not see the change to X, read 2 is told (correctly) that X has not changed since B, the time of the last read. In fact, X changed before wall-clock time A, but the change was not visible until *after* wall-clock time B, so read 2 misses the change to X. This is tricky to solve in full-blown snapshot isolation, but because Xapian serializes writes, we can use a simple, monotonically increasing database revision number. Furthermore, maintaining this revision number requires no more IO than a wall-clock time solution because Xapian already maintains statistics on the upper (and lower) bound of each value stream.
2014-10-13 08:20:01 +02:00
notmuch dump: Fix the sorting of results. To properly support sorting in notmuch_query we know use an Enquire object. We also throw in a QueryParser too, so we're really close to being able to support arbitrary full-text searches. I took a look at the supported QueryParser syntax and chose a set of flags for everything I like, (such as supporting Boolean operators in either case ("AND" or "and"), supporting phrase searching, supporting + and - to include/preclude terms, and supporting a trailing * on any term as a wildcard).
2009-10-21 09:35:56 +02:00
notmuch->query_parser = new Xapian::QueryParser;
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
notmuch->term_gen = new Xapian::TermGenerator;
notmuch->term_gen->set_stemmer (Xapian::Stem ("english"));
lib: migrate from Xapian ValueRangeProcessor to RangeProcessor This will be mandatory as of Xapian 1.5. The API is also more consistent with the FieldProcessor API, which helps code re-use a bit. Note that this switches to using the built-in Xapian support for prefixes on ranges (i.e. deleted code at beginning of ParseTimeRangeProcessor::operator(), added prefix to constructor). Another side effect of the migration is that we are generating smaller queries, using one OP_VALUE_RANGE instead of an AND of two OP_VALUE_* queries.
2020-07-07 12:56:46 +02:00
notmuch->value_range_processor = new Xapian::NumberRangeProcessor (NOTMUCH_VALUE_TIMESTAMP);
notmuch->date_range_processor = new ParseTimeRangeProcessor (NOTMUCH_VALUE_TIMESTAMP, "date:");
notmuch->last_mod_range_processor = new Xapian::NumberRangeProcessor (NOTMUCH_VALUE_LAST_MOD, "lastmod:");
notmuch dump: Fix the sorting of results. To properly support sorting in notmuch_query we know use an Enquire object. We also throw in a QueryParser too, so we're really close to being able to support arbitrary full-text searches. I took a look at the supported QueryParser syntax and chose a set of flags for everything I like, (such as supporting Boolean operators in either case ("AND" or "and"), supporting phrase searching, supporting + and - to include/preclude terms, and supporting a trailing * on any term as a wildcard).
2009-10-21 09:35:56 +02:00
notmuch->query_parser->set_default_op (Xapian::Query::OP_AND);
notmuch->query_parser->set_database (*notmuch->xapian_db);
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
notmuch->query_parser->set_stemmer (Xapian::Stem ("english"));
notmuch->query_parser->set_stemming_strategy (Xapian::QueryParser::STEM_SOME);
lib: migrate from Xapian ValueRangeProcessor to RangeProcessor This will be mandatory as of Xapian 1.5. The API is also more consistent with the FieldProcessor API, which helps code re-use a bit. Note that this switches to using the built-in Xapian support for prefixes on ranges (i.e. deleted code at beginning of ParseTimeRangeProcessor::operator(), added prefix to constructor). Another side effect of the migration is that we are generating smaller queries, using one OP_VALUE_RANGE instead of an AND of two OP_VALUE_* queries.
2020-07-07 12:56:46 +02:00
notmuch->query_parser->add_rangeprocessor (notmuch->value_range_processor);
notmuch->query_parser->add_rangeprocessor (notmuch->date_range_processor);
notmuch->query_parser->add_rangeprocessor (notmuch->last_mod_range_processor);
Split BOOLEAN_PREFIX into INTERNAL and EXTERNAL subsets. The idea here is that only some of the prefix names (such as "id" and "tag") actually make sense in external user-supplied query strings. Other things like "type" are internal implementation details of how we store things in the database. So internal machinery will add those terms to the database and we don't need to support them in the string itself. With this, we can now simply loop over the external prefix values to let the quiery parser know about them. So as we add prefixes in the future, we'll only need to add them to this list.
2009-10-25 06:38:43 +01:00
lib: merge internal prefix tables Replace multiple tables with some flags in a single table. This makes the code in notmuch_database_open_verbose a bit shorter, and it should also make it easier to add other options to fields, e.g. regexp searching.
2017-02-17 04:07:49 +01:00
for (i = 0; i < ARRAY_SIZE (prefix_table); i++) {
const prefix_t *prefix = &prefix_table[i];
if (prefix->flags & NOTMUCH_FIELD_EXTERNAL) {
lib: create field processors from prefix table This is a bit more code than hardcoding the two existing field processors, but it should make it easy to add more.
2017-02-27 03:34:19 +01:00
_setup_query_field (prefix, notmuch);
lib: merge internal prefix tables Replace multiple tables with some flags in a single table. This makes the code in notmuch_database_open_verbose a bit shorter, and it should also make it easier to add other options to fields, e.g. regexp searching.
2017-02-17 04:07:49 +01:00
}
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
}
lib: setup user headers in query parser These tests will need to be updated if the Xapian query print/debug format changes.
2018-11-17 15:08:59 +01:00
status = _setup_user_query_fields (notmuch);
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
} catch (const Xapian::Error &error) {
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
IGNORE_RESULT (asprintf (&message, "A Xapian exception occurred opening database: %s\n",
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
error.get_msg ().c_str ()));
Split notmuch_database_close into two functions Formerly notmuch_database_close closed the xapian database and destroyed the talloc structure associated with the notmuch database object. Split notmuch_database_close into notmuch_database_close and notmuch_database_destroy. This makes it possible for long running programs to close the xapian database and thus release the lock associated with it without destroying the data structures obtained from it. This also makes the api more consistent since every other data structure has a destructor function. The comments in notmuch.h are a courtesy of Austin Clements. Signed-off-by: Justus Winter <4winter@informatik.uni-hamburg.de>
2012-04-25 15:20:16 +02:00
notmuch_database_destroy (notmuch);
Fix segfault in case of the database lock not being available. We were nicely reporting the lock-aquisition failure, but then marching along trying to use the database object and just crashing badly. So don't do that.
2009-10-28 07:57:37 +01:00
notmuch = NULL;
lib/cli: Make notmuch_database_open return a status code It has been a long-standing issue that notmuch_database_open doesn't return any indication of why it failed. This patch changes its prototype to return a notmuch_status_t and set an out-argument to the database itself, like other functions that return both a status and an object. In the interest of atomicity, this also updates every use in the CLI so that notmuch still compiles. Since this patch does not update the bindings, the Python bindings test fails.
2012-04-30 18:25:33 +02:00
status = NOTMUCH_STATUS_XAPIAN_EXCEPTION;
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
}
lib/database.cc: coding style Carl claims he must have been distracted when he wrote this... Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
2009-11-22 00:14:39 +01:00
notmuch_database_create/open: Fix to handle NULL as documented. When documenting these functions I described support for a NOTMUCH_BASE environment variable to be consulted in the case of a NULL path. Only, I had forgotten to actually write the code. This code exists now, with a new, exported function: notmuch_database_default_path
2009-10-20 18:56:25 +02:00
DONE:
lib: Use talloc to simplify cleanup in notmuch_database_open Previously, we manually "free"d various pointers in notmuch_database_open. Use a local talloc context instead to simplify cleanup and eliminate various NULL pointer initializations and conditionals.
2012-01-29 06:50:10 +01:00
talloc_free (local);
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
if (message) {
if (status_string)
*status_string = message;
else
free (message);
}
lib/cli: Make notmuch_database_open return a status code It has been a long-standing issue that notmuch_database_open doesn't return any indication of why it failed. This patch changes its prototype to return a notmuch_status_t and set an out-argument to the database itself, like other functions that return both a status and an object. In the interest of atomicity, this also updates every use in the CLI so that notmuch still compiles. Since this patch does not update the bindings, the Python bindings test fails.
2012-04-30 18:25:33 +02:00
if (database)
*database = notmuch;
else
talloc_free (notmuch);
lib: move deallocation of memory from n_d_close to n_d_destroy In order to mimic the "best effort" API of Xapian to provide information from a closed database when possible, do not destroy the Xapian database object too early. Because the pointer to a Xapian database is no longer nulled on close, introduce a flag to track whether the notmuch database is open or not.
2020-07-14 13:25:28 +02:00
if (notmuch)
notmuch->open = true;
lib/cli: Make notmuch_database_open return a status code It has been a long-standing issue that notmuch_database_open doesn't return any indication of why it failed. This patch changes its prototype to return a notmuch_status_t and set an out-argument to the database itself, like other functions that return both a status and an object. In the interest of atomicity, this also updates every use in the CLI so that notmuch still compiles. Since this patch does not update the bindings, the Python bindings test fails.
2012-04-30 18:25:33 +02:00
return status;
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
}
lib: add return status to database close and destroy notmuch_database_close may fail in Xapian ->flush() or ->close(), so report the status. Similarly for notmuch_database_destroy which calls close. This is required for notmuch insert to report error status if message indexing failed.
2014-04-16 14:59:16 +02:00
notmuch_status_t
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
notmuch_database_close (notmuch_database_t *notmuch)
{
lib: add return status to database close and destroy notmuch_database_close may fail in Xapian ->flush() or ->close(), so report the status. Similarly for notmuch_database_destroy which calls close. This is required for notmuch insert to report error status if message indexing failed.
2014-04-16 14:59:16 +02:00
notmuch_status_t status = NOTMUCH_STATUS_SUCCESS;
Actually close the xapian database in notmuch_database_close Formerly the xapian database object was deleted and closed in its destructor once the object was garbage collected. Explicitly call close() so that the database and the associated lock is released immediately. The comment is a courtesy of Austin Clements. Signed-off-by: Justus Winter <4winter@informatik.uni-hamburg.de>
2012-03-02 15:58:39 +01:00
/* Many Xapian objects (and thus notmuch objects) hold references to
* the database, so merely deleting the database may not suffice to
* close it. Thus, we explicitly close it here. */
lib: move deallocation of memory from n_d_close to n_d_destroy In order to mimic the "best effort" API of Xapian to provide information from a closed database when possible, do not destroy the Xapian database object too early. Because the pointer to a Xapian database is no longer nulled on close, introduce a flag to track whether the notmuch database is open or not.
2020-07-14 13:25:28 +02:00
if (notmuch->open) {
Actually close the xapian database in notmuch_database_close Formerly the xapian database object was deleted and closed in its destructor once the object was garbage collected. Explicitly call close() so that the database and the associated lock is released immediately. The comment is a courtesy of Austin Clements. Signed-off-by: Justus Winter <4winter@informatik.uni-hamburg.de>
2012-03-02 15:58:39 +01:00
try {
lib: Simplify close and codify aborting atomic section In Xapian, closing a database implicitly aborts any outstanding transaction and commits changes. For historical reasons, notmuch_database_close had grown to almost, but not quite duplicate this behavior. Before closing the database, it would explicitly (and unnecessarily) commit it. However, if there was an outstanding transaction (ie atomic section), commit would throw a Xapian exception, which notmuch_database_close would unnecessarily print to stderr, even though notmuch_database_close would ultimately abort the transaction anyway when it called close. This patch simplifies notmuch_database_close to explicitly abort any outstanding transaction and then just call Database::close. This works for both read-only and read/write databases, takes care of committing changes, unifies the exception handling path, and codifies aborting outstanding transactions. This is currently the only way to abort an atomic section (and may remain so, since it would be difficult to roll back things we may have cached from rolled-back modifications).
2014-10-02 21:19:08 +02:00
/* If there's an outstanding transaction, it's unclear if
* closing the Xapian database commits everything up to
* that transaction, or may discard committed (but
* unflushed) transactions. To be certain, explicitly
* cancel any outstanding transaction before closing. */
lib: encapsulate the use of notmuch_database_t field 'mode' The plan is to change the underlying representation.
2020-07-27 01:31:35 +02:00
if (_notmuch_database_mode (notmuch) == NOTMUCH_DATABASE_MODE_READ_WRITE &&
lib: Simplify close and codify aborting atomic section In Xapian, closing a database implicitly aborts any outstanding transaction and commits changes. For historical reasons, notmuch_database_close had grown to almost, but not quite duplicate this behavior. Before closing the database, it would explicitly (and unnecessarily) commit it. However, if there was an outstanding transaction (ie atomic section), commit would throw a Xapian exception, which notmuch_database_close would unnecessarily print to stderr, even though notmuch_database_close would ultimately abort the transaction anyway when it called close. This patch simplifies notmuch_database_close to explicitly abort any outstanding transaction and then just call Database::close. This works for both read-only and read/write databases, takes care of committing changes, unifies the exception handling path, and codifies aborting outstanding transactions. This is currently the only way to abort an atomic section (and may remain so, since it would be difficult to roll back things we may have cached from rolled-back modifications).
2014-10-02 21:19:08 +02:00
notmuch->atomic_nesting)
lib: replace use of static_cast for writable databases static_cast is a bit tricky to understand and error prone, so add a second pointer to (potentially the same) Xapian database object that we know has the right subclass.
2020-07-27 01:31:36 +02:00
notmuch->writable_xapian_db->cancel_transaction ();
lib: Simplify close and codify aborting atomic section In Xapian, closing a database implicitly aborts any outstanding transaction and commits changes. For historical reasons, notmuch_database_close had grown to almost, but not quite duplicate this behavior. Before closing the database, it would explicitly (and unnecessarily) commit it. However, if there was an outstanding transaction (ie atomic section), commit would throw a Xapian exception, which notmuch_database_close would unnecessarily print to stderr, even though notmuch_database_close would ultimately abort the transaction anyway when it called close. This patch simplifies notmuch_database_close to explicitly abort any outstanding transaction and then just call Database::close. This works for both read-only and read/write databases, takes care of committing changes, unifies the exception handling path, and codifies aborting outstanding transactions. This is currently the only way to abort an atomic section (and may remain so, since it would be difficult to roll back things we may have cached from rolled-back modifications).
2014-10-02 21:19:08 +02:00
/* Close the database. This implicitly flushes
* outstanding changes. */
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
notmuch->xapian_db->close ();
Actually close the xapian database in notmuch_database_close Formerly the xapian database object was deleted and closed in its destructor once the object was garbage collected. Explicitly call close() so that the database and the associated lock is released immediately. The comment is a courtesy of Austin Clements. Signed-off-by: Justus Winter <4winter@informatik.uni-hamburg.de>
2012-03-02 15:58:39 +01:00
} catch (const Xapian::Error &error) {
lib: Simplify close and codify aborting atomic section In Xapian, closing a database implicitly aborts any outstanding transaction and commits changes. For historical reasons, notmuch_database_close had grown to almost, but not quite duplicate this behavior. Before closing the database, it would explicitly (and unnecessarily) commit it. However, if there was an outstanding transaction (ie atomic section), commit would throw a Xapian exception, which notmuch_database_close would unnecessarily print to stderr, even though notmuch_database_close would ultimately abort the transaction anyway when it called close. This patch simplifies notmuch_database_close to explicitly abort any outstanding transaction and then just call Database::close. This works for both read-only and read/write databases, takes care of committing changes, unifies the exception handling path, and codifies aborting outstanding transactions. This is currently the only way to abort an atomic section (and may remain so, since it would be difficult to roll back things we may have cached from rolled-back modifications).
2014-10-02 21:19:08 +02:00
status = NOTMUCH_STATUS_XAPIAN_EXCEPTION;
if (! notmuch->exception_reported) {
lib: replace almost all fprintfs in library with _n_d_log This is not supposed to change any functionality from an end user point of view. Note that it will eliminate some output to stderr. The query debugging output is left as is; it doesn't really fit with the current primitive logging model. The remaining "bad" fprintf will need an internal API change.
2014-12-26 17:25:35 +01:00
_notmuch_database_log (notmuch, "Error: A Xapian exception occurred closing database: %s\n",
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
error.get_msg ().c_str ());
lib: Simplify close and codify aborting atomic section In Xapian, closing a database implicitly aborts any outstanding transaction and commits changes. For historical reasons, notmuch_database_close had grown to almost, but not quite duplicate this behavior. Before closing the database, it would explicitly (and unnecessarily) commit it. However, if there was an outstanding transaction (ie atomic section), commit would throw a Xapian exception, which notmuch_database_close would unnecessarily print to stderr, even though notmuch_database_close would ultimately abort the transaction anyway when it called close. This patch simplifies notmuch_database_close to explicitly abort any outstanding transaction and then just call Database::close. This works for both read-only and read/write databases, takes care of committing changes, unifies the exception handling path, and codifies aborting outstanding transactions. This is currently the only way to abort an atomic section (and may remain so, since it would be difficult to roll back things we may have cached from rolled-back modifications).
2014-10-02 21:19:08 +02:00
}
Actually close the xapian database in notmuch_database_close Formerly the xapian database object was deleted and closed in its destructor once the object was garbage collected. Explicitly call close() so that the database and the associated lock is released immediately. The comment is a courtesy of Austin Clements. Signed-off-by: Justus Winter <4winter@informatik.uni-hamburg.de>
2012-03-02 15:58:39 +01:00
}
}
lib: move deallocation of memory from n_d_close to n_d_destroy In order to mimic the "best effort" API of Xapian to provide information from a closed database when possible, do not destroy the Xapian database object too early. Because the pointer to a Xapian database is no longer nulled on close, introduce a flag to track whether the notmuch database is open or not.
2020-07-14 13:25:28 +02:00
notmuch->open = false;
lib: add return status to database close and destroy notmuch_database_close may fail in Xapian ->flush() or ->close(), so report the status. Similarly for notmuch_database_destroy which calls close. This is required for notmuch insert to report error status if message indexing failed.
2014-04-16 14:59:16 +02:00
return status;
Split notmuch_database_close into two functions Formerly notmuch_database_close closed the xapian database and destroyed the talloc structure associated with the notmuch database object. Split notmuch_database_close into notmuch_database_close and notmuch_database_destroy. This makes it possible for long running programs to close the xapian database and thus release the lock associated with it without destroying the data structures obtained from it. This also makes the api more consistent since every other data structure has a destructor function. The comments in notmuch.h are a courtesy of Austin Clements. Signed-off-by: Justus Winter <4winter@informatik.uni-hamburg.de>
2012-04-25 15:20:16 +02:00
}
lib: add _notmuch_database_reopen The main expected use is to recover from a Xapian::DatabaseChanged exception.
2017-02-24 02:38:24 +01:00
notmuch_status_t
_notmuch_database_reopen (notmuch_database_t *notmuch)
{
lib: encapsulate the use of notmuch_database_t field 'mode' The plan is to change the underlying representation.
2020-07-27 01:31:35 +02:00
if (_notmuch_database_mode (notmuch) != NOTMUCH_DATABASE_MODE_READ_ONLY)
lib: add _notmuch_database_reopen The main expected use is to recover from a Xapian::DatabaseChanged exception.
2017-02-24 02:38:24 +01:00
return NOTMUCH_STATUS_UNSUPPORTED_OPERATION;
try {
notmuch->xapian_db->reopen ();
} catch (const Xapian::Error &error) {
if (! notmuch->exception_reported) {
_notmuch_database_log (notmuch, "Error: A Xapian exception reopening database: %s\n",
error.get_msg ().c_str ());
lib: convert notmuch_bool_t to stdbool internally C99 stdbool turned 18 this year. There really is no reason to use our own, except in the library interface for backward compatibility. Convert the lib internally to stdbool.
2017-10-07 10:44:05 +02:00
notmuch->exception_reported = true;
lib: add _notmuch_database_reopen The main expected use is to recover from a Xapian::DatabaseChanged exception.
2017-02-24 02:38:24 +01:00
}
return NOTMUCH_STATUS_XAPIAN_EXCEPTION;
}
notmuch->view++;
return NOTMUCH_STATUS_SUCCESS;
}
compact: tidy formatting Notmuch compact code whitespace changes to match devel/STYLE.
2013-11-13 18:02:43 +01:00
static int
unlink_cb (const char *path,
unused (const struct stat *sb),
unused (int type),
unused (struct FTW *ftw))
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
{
compact: tidy formatting Notmuch compact code whitespace changes to match devel/STYLE.
2013-11-13 18:02:43 +01:00
return remove (path);
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
}
compact: tidy formatting Notmuch compact code whitespace changes to match devel/STYLE.
2013-11-13 18:02:43 +01:00
static int
rmtree (const char *path)
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
{
compact: tidy formatting Notmuch compact code whitespace changes to match devel/STYLE.
2013-11-13 18:02:43 +01:00
return nftw (path, unlink_cb, 64, FTW_DEPTH | FTW_PHYS);
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
}
class NotmuchCompactor : public Xapian::Compactor
{
notmuch_compact_status_cb_t status_cb;
lib: add closure parameter to compact status update callback This provides much more flexibility for the caller.
2013-11-03 13:24:45 +01:00
void *status_closure;
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
public:
lib: add closure parameter to compact status update callback This provides much more flexibility for the caller.
2013-11-03 13:24:45 +01:00
NotmuchCompactor(notmuch_compact_status_cb_t cb, void *closure) :
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
status_cb (cb), status_closure (closure)
{
}
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
virtual void
set_status (const std::string &table, const std::string &status)
{
compact: tidy formatting Notmuch compact code whitespace changes to match devel/STYLE.
2013-11-13 18:02:43 +01:00
char *msg;
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
if (status_cb == NULL)
return;
compact: tidy formatting Notmuch compact code whitespace changes to match devel/STYLE.
2013-11-13 18:02:43 +01:00
if (status.length () == 0)
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
msg = talloc_asprintf (NULL, "compacting table %s", table.c_str ());
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
else
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
msg = talloc_asprintf (NULL, " %s", status.c_str ());
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
if (msg == NULL) {
return;
}
compact: tidy formatting Notmuch compact code whitespace changes to match devel/STYLE.
2013-11-13 18:02:43 +01:00
status_cb (msg, status_closure);
talloc_free (msg);
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
}
};
/* Compacts the given database, optionally saving the original database
* in backup_path. Additionally, a callback function can be provided to
* give the user feedback on the progress of the (likely long-lived)
* compaction process.
*
* The backup path must point to a directory on the same volume as the
* original database. Passing a NULL backup_path will result in the
* uncompacted database being deleted after compaction has finished.
* Note that the database write lock will be held during the
* compaction process to protect data integrity.
*/
notmuch_status_t
compact: tidy formatting Notmuch compact code whitespace changes to match devel/STYLE.
2013-11-13 18:02:43 +01:00
notmuch_database_compact (const char *path,
const char *backup_path,
lib: add closure parameter to compact status update callback This provides much more flexibility for the caller.
2013-11-03 13:24:45 +01:00
notmuch_compact_status_cb_t status_cb,
void *closure)
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
{
lib: check talloc success in compact In line with the allocation checks all around.
2013-11-03 13:24:43 +01:00
void *local;
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
char *notmuch_path, *xapian_path, *compact_xapian_path;
notmuch_status_t ret = NOTMUCH_STATUS_SUCCESS;
notmuch_database_t *notmuch = NULL;
struct stat statbuf;
lib: convert notmuch_bool_t to stdbool internally C99 stdbool turned 18 this year. There really is no reason to use our own, except in the library interface for backward compatibility. Convert the lib internally to stdbool.
2017-10-07 10:44:05 +02:00
bool keep_backup;
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
char *message = NULL;
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
lib: check talloc success in compact In line with the allocation checks all around.
2013-11-03 13:24:43 +01:00
local = talloc_new (NULL);
if (! local)
return NOTMUCH_STATUS_OUT_OF_MEMORY;
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
ret = notmuch_database_open_verbose (path,
NOTMUCH_DATABASE_MODE_READ_WRITE,
&notmuch,
&message);
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
if (ret) {
lib: add "verbose" versions of notmuch_database_{open,create} The compatibility wrapper ensures that clients calling notmuch_database_open will receive consistent output for now. The changes to notmuch-{new,search} and test/symbol-test are just to make the test suite pass. The use of IGNORE_RESULT is justified by two things. 1) I don't know what else to do. 2) asprintf guarantees the output string is NULL if an error occurs, so at least we are not passing garbage back.
2014-12-27 19:12:49 +01:00
if (status_cb) status_cb (message, closure);
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
goto DONE;
}
if (! (notmuch_path = talloc_asprintf (local, "%s/%s", path, ".notmuch"))) {
ret = NOTMUCH_STATUS_OUT_OF_MEMORY;
goto DONE;
}
if (! (xapian_path = talloc_asprintf (local, "%s/%s", notmuch_path, "xapian"))) {
ret = NOTMUCH_STATUS_OUT_OF_MEMORY;
goto DONE;
}
if (! (compact_xapian_path = talloc_asprintf (local, "%s.compact", xapian_path))) {
ret = NOTMUCH_STATUS_OUT_OF_MEMORY;
goto DONE;
}
compact: preserve backup database until compacted database is in place It is less error prone and window of failure opportunity is smaller if the old (backup) database is always renamed (instead of sometimes rmtree'd) before new (compacted) database is put into its place. Finally rmtree() old database in case old database backup is not kept.
2013-11-14 23:03:25 +01:00
if (backup_path == NULL) {
if (! (backup_path = talloc_asprintf (local, "%s.old", xapian_path))) {
ret = NOTMUCH_STATUS_OUT_OF_MEMORY;
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
goto DONE;
}
lib: convert notmuch_bool_t to stdbool internally C99 stdbool turned 18 this year. There really is no reason to use our own, except in the library interface for backward compatibility. Convert the lib internally to stdbool.
2017-10-07 10:44:05 +02:00
keep_backup = false;
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
} else {
lib: convert notmuch_bool_t to stdbool internally C99 stdbool turned 18 this year. There really is no reason to use our own, except in the library interface for backward compatibility. Convert the lib internally to stdbool.
2017-10-07 10:44:05 +02:00
keep_backup = true;
compact: preserve backup database until compacted database is in place It is less error prone and window of failure opportunity is smaller if the old (backup) database is always renamed (instead of sometimes rmtree'd) before new (compacted) database is put into its place. Finally rmtree() old database in case old database backup is not kept.
2013-11-14 23:03:25 +01:00
}
if (stat (backup_path, &statbuf) != -1) {
lib: replace almost all fprintfs in library with _n_d_log This is not supposed to change any functionality from an end user point of view. Note that it will eliminate some output to stderr. The query debugging output is left as is; it doesn't really fit with the current primitive logging model. The remaining "bad" fprintf will need an internal API change.
2014-12-26 17:25:35 +01:00
_notmuch_database_log (notmuch, "Path already exists: %s\n", backup_path);
compact: preserve backup database until compacted database is in place It is less error prone and window of failure opportunity is smaller if the old (backup) database is always renamed (instead of sometimes rmtree'd) before new (compacted) database is put into its place. Finally rmtree() old database in case old database backup is not kept.
2013-11-14 23:03:25 +01:00
ret = NOTMUCH_STATUS_FILE_ERROR;
goto DONE;
}
if (errno != ENOENT) {
lib: replace almost all fprintfs in library with _n_d_log This is not supposed to change any functionality from an end user point of view. Note that it will eliminate some output to stderr. The query debugging output is left as is; it doesn't really fit with the current primitive logging model. The remaining "bad" fprintf will need an internal API change.
2014-12-26 17:25:35 +01:00
_notmuch_database_log (notmuch, "Unknown error while stat()ing path: %s\n",
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
strerror (errno));
compact: preserve backup database until compacted database is in place It is less error prone and window of failure opportunity is smaller if the old (backup) database is always renamed (instead of sometimes rmtree'd) before new (compacted) database is put into its place. Finally rmtree() old database in case old database backup is not kept.
2013-11-14 23:03:25 +01:00
ret = NOTMUCH_STATUS_FILE_ERROR;
goto DONE;
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
}
compact: unconditionally remove old wip database compact directory In case previous notmuch compact has been interrupted there is old work-in-progress database compact directory partially filled. Remove it just before starting to fill the directory with new files.
2013-11-14 23:03:26 +01:00
/* Unconditionally attempt to remove old work-in-progress database (if
* any). This is "protected" by database lock. If this fails due to write
* errors (etc), the following code will fail and provide error message.
*/
(void) rmtree (compact_xapian_path);
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
try {
compact: tidy formatting Notmuch compact code whitespace changes to match devel/STYLE.
2013-11-13 18:02:43 +01:00
NotmuchCompactor compactor (status_cb, closure);
lib: migrate to post Xapian 1.3.4 compact support The old API was deprecated in Xapian 1.3.4 and (will be) removed in 1.5.0
2020-07-07 12:56:45 +02:00
notmuch->xapian_db->compact (compact_xapian_path, Xapian::DBCOMPACT_NO_RENUMBER, 0, compactor);
compact: catch Xapian::Error consistently catch Xapian::Error in compact code in lib/database.cc to be consistent with other code in addition to not making software crash on uncaught other Xapian error.
2013-11-13 18:02:44 +01:00
} catch (const Xapian::Error &error) {
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
_notmuch_database_log (notmuch, "Error while compacting: %s\n", error.get_msg ().c_str ());
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
ret = NOTMUCH_STATUS_XAPIAN_EXCEPTION;
goto DONE;
}
compact: preserve backup database until compacted database is in place It is less error prone and window of failure opportunity is smaller if the old (backup) database is always renamed (instead of sometimes rmtree'd) before new (compacted) database is put into its place. Finally rmtree() old database in case old database backup is not kept.
2013-11-14 23:03:25 +01:00
if (rename (xapian_path, backup_path)) {
lib: replace almost all fprintfs in library with _n_d_log This is not supposed to change any functionality from an end user point of view. Note that it will eliminate some output to stderr. The query debugging output is left as is; it doesn't really fit with the current primitive logging model. The remaining "bad" fprintf will need an internal API change.
2014-12-26 17:25:35 +01:00
_notmuch_database_log (notmuch, "Error moving %s to %s: %s\n",
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
xapian_path, backup_path, strerror (errno));
compact: preserve backup database until compacted database is in place It is less error prone and window of failure opportunity is smaller if the old (backup) database is always renamed (instead of sometimes rmtree'd) before new (compacted) database is put into its place. Finally rmtree() old database in case old database backup is not kept.
2013-11-14 23:03:25 +01:00
ret = NOTMUCH_STATUS_FILE_ERROR;
goto DONE;
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
}
compact: tidy formatting Notmuch compact code whitespace changes to match devel/STYLE.
2013-11-13 18:02:43 +01:00
if (rename (compact_xapian_path, xapian_path)) {
lib: replace almost all fprintfs in library with _n_d_log This is not supposed to change any functionality from an end user point of view. Note that it will eliminate some output to stderr. The query debugging output is left as is; it doesn't really fit with the current primitive logging model. The remaining "bad" fprintf will need an internal API change.
2014-12-26 17:25:35 +01:00
_notmuch_database_log (notmuch, "Error moving %s to %s: %s\n",
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
compact_xapian_path, xapian_path, strerror (errno));
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
ret = NOTMUCH_STATUS_FILE_ERROR;
goto DONE;
}
compact: improve error messages on failures after compaction The error messages written during the steps replacing old database with new now includes relevant paths and strerror.
2013-11-14 23:03:27 +01:00
if (! keep_backup) {
if (rmtree (backup_path)) {
lib: replace almost all fprintfs in library with _n_d_log This is not supposed to change any functionality from an end user point of view. Note that it will eliminate some output to stderr. The query debugging output is left as is; it doesn't really fit with the current primitive logging model. The remaining "bad" fprintf will need an internal API change.
2014-12-26 17:25:35 +01:00
_notmuch_database_log (notmuch, "Error removing old database %s: %s\n",
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
backup_path, strerror (errno));
compact: improve error messages on failures after compaction The error messages written during the steps replacing old database with new now includes relevant paths and strerror.
2013-11-14 23:03:27 +01:00
ret = NOTMUCH_STATUS_FILE_ERROR;
goto DONE;
}
}
compact: preserve backup database until compacted database is in place It is less error prone and window of failure opportunity is smaller if the old (backup) database is always renamed (instead of sometimes rmtree'd) before new (compacted) database is put into its place. Finally rmtree() old database in case old database backup is not kept.
2013-11-14 23:03:25 +01:00
compact: tidy formatting Notmuch compact code whitespace changes to match devel/STYLE.
2013-11-13 18:02:43 +01:00
DONE:
lib: add return status to database close and destroy notmuch_database_close may fail in Xapian ->flush() or ->close(), so report the status. Similarly for notmuch_database_destroy which calls close. This is required for notmuch insert to report error status if message indexing failed.
2014-04-16 14:59:16 +02:00
if (notmuch) {
notmuch_status_t ret2;
lib: replace almost all fprintfs in library with _n_d_log This is not supposed to change any functionality from an end user point of view. Note that it will eliminate some output to stderr. The query debugging output is left as is; it doesn't really fit with the current primitive logging model. The remaining "bad" fprintf will need an internal API change.
2014-12-26 17:25:35 +01:00
const char *str = notmuch_database_status_string (notmuch);
if (status_cb && str)
status_cb (str, closure);
lib: add return status to database close and destroy notmuch_database_close may fail in Xapian ->flush() or ->close(), so report the status. Similarly for notmuch_database_destroy which calls close. This is required for notmuch insert to report error status if message indexing failed.
2014-04-16 14:59:16 +02:00
ret2 = notmuch_database_destroy (notmuch);
/* don't clobber previous error status */
if (ret == NOTMUCH_STATUS_SUCCESS && ret2 != NOTMUCH_STATUS_SUCCESS)
ret = ret2;
}
lib: do not leak the database in compaction Destroy instead of close the database after compaction, and also on error path, to not leak the database.
2013-11-03 13:24:44 +01:00
compact: tidy formatting Notmuch compact code whitespace changes to match devel/STYLE.
2013-11-13 18:02:43 +01:00
talloc_free (local);
lib: do not leak the database in compaction Destroy instead of close the database after compaction, and also on error path, to not leak the database.
2013-11-03 13:24:44 +01:00
database: Add notmuch_database_compact_close This function uses Xapian's Compactor machinery to compact the notmuch database. The compacted database is built in a temporary directory and later moved into place while the original uncompacted database is preserved. Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-02 22:30:46 +02:00
return ret;
}
lib: add return status to database close and destroy notmuch_database_close may fail in Xapian ->flush() or ->close(), so report the status. Similarly for notmuch_database_destroy which calls close. This is required for notmuch insert to report error status if message indexing failed.
2014-04-16 14:59:16 +02:00
notmuch_status_t
Split notmuch_database_close into two functions Formerly notmuch_database_close closed the xapian database and destroyed the talloc structure associated with the notmuch database object. Split notmuch_database_close into notmuch_database_close and notmuch_database_destroy. This makes it possible for long running programs to close the xapian database and thus release the lock associated with it without destroying the data structures obtained from it. This also makes the api more consistent since every other data structure has a destructor function. The comments in notmuch.h are a courtesy of Austin Clements. Signed-off-by: Justus Winter <4winter@informatik.uni-hamburg.de>
2012-04-25 15:20:16 +02:00
notmuch_database_destroy (notmuch_database_t *notmuch)
{
lib: add return status to database close and destroy notmuch_database_close may fail in Xapian ->flush() or ->close(), so report the status. Similarly for notmuch_database_destroy which calls close. This is required for notmuch insert to report error status if message indexing failed.
2014-04-16 14:59:16 +02:00
notmuch_status_t status;
status = notmuch_database_close (notmuch);
lib: move deallocation of memory from n_d_close to n_d_destroy In order to mimic the "best effort" API of Xapian to provide information from a closed database when possible, do not destroy the Xapian database object too early. Because the pointer to a Xapian database is no longer nulled on close, introduce a flag to track whether the notmuch database is open or not.
2020-07-14 13:25:28 +02:00
delete notmuch->term_gen;
notmuch->term_gen = NULL;
delete notmuch->query_parser;
notmuch->query_parser = NULL;
delete notmuch->xapian_db;
notmuch->xapian_db = NULL;
delete notmuch->value_range_processor;
notmuch->value_range_processor = NULL;
delete notmuch->date_range_processor;
notmuch->date_range_processor = NULL;
delete notmuch->last_mod_range_processor;
notmuch->last_mod_range_processor = NULL;
Convert notmuch_database_t to start using talloc. This will be handy as we can hang future talloc allocations off of the datbase now.
2009-10-21 23:00:37 +02:00
talloc_free (notmuch);
lib: add return status to database close and destroy notmuch_database_close may fail in Xapian ->flush() or ->close(), so report the status. Similarly for notmuch_database_destroy which calls close. This is required for notmuch insert to report error status if message indexing failed.
2014-04-16 14:59:16 +02:00
return status;
notmuch: Start actually adding messages to the index. This is the beginning of the notmuch library as well, with its interface in notmuch.h. So far we've got create, open, close, and add_message (all with a notmuch_database prefix). The current add_message function has already been whittled down from what we have in notmuch-index-message to add only references, message-id, and thread-id to the index, (that is---just enough to do thread-linkage but nothing for full-text searching). The concept here is to do something quickly so that the user can get some data into notmuch and start using it. (The most interesting stuff is then thread-linkage and labels like inbox and unread.) We can defer the full-text indexing of the body of the messages for later, (such as in the background while the user is reading mail). The initial thread-stitching step is still slower than I would like. We may have to stop using libgmime for this step as its overhead is not worth it for the simple case of just parsing the message-id, references, and in-reply-to headers.
2009-10-19 05:56:30 +02:00
}
const char *
notmuch_database_get_path (notmuch_database_t *notmuch)
{
return notmuch->path;
}
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
unsigned int
notmuch_database_get_version (notmuch_database_t *notmuch)
{
unsigned int version;
string version_string;
const char *str;
char *end;
lib/n_d_get_version: catch exceptions and clarify the API notmuch_database_get_version previously returned 0 on some errors, but did not document this. Luckily 0 is not a valid database version.
2020-07-15 00:31:10 +02:00
try {
version_string = notmuch->xapian_db->get_metadata ("version");
} catch (const Xapian::Error &error) {
LOG_XAPIAN_EXCEPTION (notmuch, error);
return 0;
}
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
if (version_string.empty ())
return 0;
str = version_string.c_str ();
if (str == NULL || *str == '\0')
return 0;
version = strtoul (str, &end, 10);
if (*end != '\0')
INTERNAL_ERROR ("Malformed database version: %s", str);
return version;
}
notmuch_bool_t
notmuch_database_needs_upgrade (notmuch_database_t *notmuch)
{
lib/n_d_needs_upgrade: handle error return from n_d_get_version Also clarify documentation of error return from n_d_needs_upgrade.
2020-07-17 00:28:22 +02:00
unsigned int version;
lib: encapsulate the use of notmuch_database_t field 'mode' The plan is to change the underlying representation.
2020-07-27 01:31:35 +02:00
if (_notmuch_database_mode (notmuch) != NOTMUCH_DATABASE_MODE_READ_WRITE)
lib/n_d_needs_upgrade: handle error return from n_d_get_version Also clarify documentation of error return from n_d_needs_upgrade.
2020-07-17 00:28:22 +02:00
return FALSE;
if (NOTMUCH_FEATURES_CURRENT & ~notmuch->features)
return TRUE;
version = notmuch_database_get_version (notmuch);
return (version > 0 && version < NOTMUCH_DATABASE_VERSION);
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
}
static volatile sig_atomic_t do_progress_notify = 0;
static void
handle_sigalrm (unused (int signal))
{
do_progress_notify = 1;
}
/* Upgrade the current database.
*
* After opening a database in read-write mode, the client should
* check if an upgrade is needed (notmuch_database_needs_upgrade) and
* if so, upgrade with this function before making any modifications.
*
* The optional progress_notify callback can be used by the caller to
* provide progress indication to the user. If non-NULL it will be
* called periodically with 'count' as the number of messages upgraded
* so far and 'total' the overall number of messages that will be
* converted.
*/
notmuch_status_t
notmuch_database_upgrade (notmuch_database_t *notmuch,
lib: migrate from Xapian ValueRangeProcessor to RangeProcessor This will be mandatory as of Xapian 1.5. The API is also more consistent with the FieldProcessor API, which helps code re-use a bit. Note that this switches to using the built-in Xapian support for prefixes on ranges (i.e. deleted code at beginning of ParseTimeRangeProcessor::operator(), added prefix to constructor). Another side effect of the migration is that we are generating smaller queries, using one OP_VALUE_RANGE instead of an AND of two OP_VALUE_* queries.
2020-07-07 12:56:46 +02:00
void (*progress_notify) (void *closure,
double progress),
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
void *closure)
{
lib: Database version 3: Introduce fine-grained "features" Previously, our database schema was versioned by a single number. Each database schema change had to occur "atomically" in Notmuch's development history: before some commit, Notmuch used version N, after that commit, it used version N+1. Hence, each new schema version could introduce only one change, the task of developing a schema change fell on a single person, and it all had to happen and be perfect in a single commit series. This made introducing a new schema version hard. We've seen only two schema changes in the history of Notmuch. This commit introduces database schema version 3; hopefully the last schema version we'll need for a while. With this version, we switch from a single version number to "features": a set of named, independent aspects of the database schema. Features should make backwards compatibility easier. For many things, it should be easy to support databases both with and without a feature, which will allow us to make upgrades optional and will enable "unstable" features that can be developed and tested over time. Features also make forwards compatibility easier. The features recorded in a database include "compatibility flags," which can indicate to an older version of Notmuch when it must support a given feature to open the database for read or for write. This lets us replace the old vague "I don't recognize this version, so something might go wrong, but I promise to try my best" warnings upon opening a database with an unknown version with precise errors. If a database is safe to open for read/write despite unknown features, an older version will know that and issue no message at all. If the database is not safe to open for read/write because of unknown features, an older version will know that, too, and can tell the user exactly which required features it lacks support for.
2014-08-25 19:26:00 +02:00
void *local = talloc_new (NULL);
lib: Use database features to drive upgrade Previously, we had database version information hard-coded in the upgrade code. Slightly re-organize the upgrade process around the set of new database features to be enabled by the upgrade.
2014-08-25 19:26:04 +02:00
Xapian::TermIterator t, t_end;
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
Xapian::WritableDatabase *db;
struct sigaction action;
struct itimerval timerval;
lib: convert notmuch_bool_t to stdbool internally C99 stdbool turned 18 this year. There really is no reason to use our own, except in the library interface for backward compatibility. Convert the lib internally to stdbool.
2017-10-07 10:44:05 +02:00
bool timer_is_active = false;
lib: Use database features to drive upgrade Previously, we had database version information hard-coded in the upgrade code. Slightly re-organize the upgrade process around the set of new database features to be enabled by the upgrade.
2014-08-25 19:26:04 +02:00
enum _notmuch_features target_features, new_features;
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
notmuch_status_t status;
lib: Implement upgrade to ghost messages feature Somehow this is the first upgrade pass that actually does *any* error checking, so this also adds the bit of necessary infrastructure to handle that.
2014-10-23 14:30:38 +02:00
notmuch_private_status_t private_status;
lib: move query variable to function scope This is a prelude to deallocating it (if necessary) on the error path.
2015-09-27 17:31:55 +02:00
notmuch_query_t *query = NULL;
notmuch new: Print upgrade progress report as a percentage. Previously we were printing a number of messages upgraded so far. The original motivation for this was to accurately reflect the fact that there are two passes, (so each message is processed twice and it's not accurate to represent with a single count). But as it turns out, the second pass takes zero time (relatively speaking) so we're still not accounting for it. If nothing else, the percentage-based reporting makes for a cleaner API for the progress_notify function.
2010-01-10 02:38:23 +01:00
unsigned int count = 0, total = 0;
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
status = _notmuch_database_ensure_writable (notmuch);
if (status)
return status;
lib: replace use of static_cast for writable databases static_cast is a bit tricky to understand and error prone, so add a second pointer to (potentially the same) Xapian database object that we know has the right subclass.
2020-07-27 01:31:36 +02:00
db = notmuch->writable_xapian_db;
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
lib: Use database features to drive upgrade Previously, we had database version information hard-coded in the upgrade code. Slightly re-organize the upgrade process around the set of new database features to be enabled by the upgrade.
2014-08-25 19:26:04 +02:00
target_features = notmuch->features | NOTMUCH_FEATURES_CURRENT;
new_features = NOTMUCH_FEATURES_CURRENT & ~notmuch->features;
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
lib: Fix endless upgrade problem 48db8c8 introduced a disagreement between when notmuch_database_needs_upgrade returned TRUE and when notmuch_database_upgrade actually performed an upgrade. As a result, if a database had a version less than 3, but no new features were required, notmuch new would call notmuch_database_upgrade to perform an upgrade, but notmuch_database_upgrade would return immediately without updating the database version. Hence, the next notmuch new would do the same, and so on. Fix this by ensuring that the upgrade-required logic is identical between the two.
2014-09-02 00:49:07 +02:00
if (! notmuch_database_needs_upgrade (notmuch))
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
return NOTMUCH_STATUS_SUCCESS;
if (progress_notify) {
cli: change "setup" to "set up" where used as a verb The word "setup" is a noun, not a verb. Change occurrences of "setup" where used as a verb to "set up".
2015-05-27 19:53:52 +02:00
/* Set up our handler for SIGALRM */
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
memset (&action, 0, sizeof (struct sigaction));
action.sa_handler = handle_sigalrm;
sigemptyset (&action.sa_mask);
action.sa_flags = SA_RESTART;
sigaction (SIGALRM, &action, NULL);
/* Then start a timer to send SIGALRM once per second. */
timerval.it_interval.tv_sec = 1;
timerval.it_interval.tv_usec = 0;
timerval.it_value.tv_sec = 1;
timerval.it_value.tv_usec = 0;
setitimer (ITIMER_REAL, &timerval, NULL);
lib: convert notmuch_bool_t to stdbool internally C99 stdbool turned 18 this year. There really is no reason to use our own, except in the library interface for backward compatibility. Convert the lib internally to stdbool.
2017-10-07 10:44:05 +02:00
timer_is_active = true;
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
}
lib: Report progress for combined upgrade operation Previously, some parts of upgrade didn't report progress and for others it was possible for the progress meter to restart at 0 part way through the upgrade because each stage was reported separately. Fix this by computing the total amount of work that needs to be done up-front and updating completed work monotonically.
2014-08-25 19:26:06 +02:00
/* Figure out how much total work we need to do. */
if (new_features &
lib: Add per-message last modification tracking This adds a new document value that stores the revision of the last modification to message metadata, where the revision number increases monotonically with each database commit. An alternative would be to store the wall-clock time of the last modification of each message. In principle this is simpler and has the advantage that any process can determine the current timestamp without support from libnotmuch. However, even assuming a computer's clock never goes backward and ignoring clock skew in networked environments, this has a fatal flaw. Xapian uses (optimistic) snapshot isolation, which means reads can be concurrent with writes. Given this, consider the following time line with a write and two read transactions: write |-X-A--------------| read 1 |---B---| read 2 |---| The write transaction modifies message X and records the wall-clock time of the modification at A. The writer hangs around for a while and later commits its change. Read 1 is concurrent with the write, so it doesn't see the change to X. It does some query and records the wall-clock time of its results at B. Transaction read 2 later starts after the write commits and queries for changes since wall-clock time B (say the reads are performing an incremental backup). Even though read 1 could not see the change to X, read 2 is told (correctly) that X has not changed since B, the time of the last read. In fact, X changed before wall-clock time A, but the change was not visible until *after* wall-clock time B, so read 2 misses the change to X. This is tricky to solve in full-blown snapshot isolation, but because Xapian serializes writes, we can use a simple, monotonically increasing database revision number. Furthermore, maintaining this revision number requires no more IO than a wall-clock time solution because Xapian already maintains statistics on the upper (and lower) bound of each value stream.
2014-10-13 08:20:01 +02:00
(NOTMUCH_FEATURE_FILE_TERMS | NOTMUCH_FEATURE_BOOL_FOLDER |
NOTMUCH_FEATURE_LAST_MOD)) {
lib: move query variable to function scope This is a prelude to deallocating it (if necessary) on the error path.
2015-09-27 17:31:55 +02:00
query = notmuch_query_create (notmuch, "");
lib: add versions of n_q_count_{message,threads} with status return Although I think it's a pretty bad idea to continue using the old API, this allows both a more gentle transition for clients of the library, and allows us to break one monolithic change into a series
2015-09-27 17:31:57 +02:00
unsigned msg_count;
lib: replace deprecated n_q_count_messages with status returning version This function was deprecated in notmuch 0.21. We re-use the name for a status returning version, and deprecate the _st name. One or two remaining uses of the (removed) non-status returning version fixed at the same time
2017-02-26 22:21:34 +01:00
status = notmuch_query_count_messages (query, &msg_count);
lib: add versions of n_q_count_{message,threads} with status return Although I think it's a pretty bad idea to continue using the old API, this allows both a more gentle transition for clients of the library, and allows us to break one monolithic change into a series
2015-09-27 17:31:57 +02:00
if (status)
goto DONE;
total += msg_count;
lib: Report progress for combined upgrade operation Previously, some parts of upgrade didn't report progress and for others it was possible for the progress meter to restart at 0 part way through the upgrade because each stage was reported separately. Fix this by computing the total amount of work that needs to be done up-front and updating completed work monotonically.
2014-08-25 19:26:06 +02:00
notmuch_query_destroy (query);
lib: add versions of n_q_count_{message,threads} with status return Although I think it's a pretty bad idea to continue using the old API, this allows both a more gentle transition for clients of the library, and allows us to break one monolithic change into a series
2015-09-27 17:31:57 +02:00
query = NULL;
lib: Report progress for combined upgrade operation Previously, some parts of upgrade didn't report progress and for others it was possible for the progress meter to restart at 0 part way through the upgrade because each stage was reported separately. Fix this by computing the total amount of work that needs to be done up-front and updating completed work monotonically.
2014-08-25 19:26:06 +02:00
}
if (new_features & NOTMUCH_FEATURE_DIRECTORY_DOCS) {
t_end = db->allterms_end ("XTIMESTAMP");
for (t = db->allterms_begin ("XTIMESTAMP"); t != t_end; t++)
++total;
}
lib: Implement upgrade to ghost messages feature Somehow this is the first upgrade pass that actually does *any* error checking, so this also adds the bit of necessary infrastructure to handle that.
2014-10-23 14:30:38 +02:00
if (new_features & NOTMUCH_FEATURE_GHOSTS) {
/* The ghost message upgrade converts all thread_id_*
* metadata values into ghost message documents. */
t_end = db->metadata_keys_end ("thread_id_");
for (t = db->metadata_keys_begin ("thread_id_"); t != t_end; ++t)
++total;
}
lib: Report progress for combined upgrade operation Previously, some parts of upgrade didn't report progress and for others it was possible for the progress meter to restart at 0 part way through the upgrade because each stage was reported separately. Fix this by computing the total amount of work that needs to be done up-front and updating completed work monotonically.
2014-08-25 19:26:06 +02:00
lib: Simplify upgrade code using a transaction Previously, the upgrade was organized as two passes -- an upgrade pass, and a separate cleanup pass -- so the database was always in a valid state. This change substantially simplifies this code by performing the upgrade in a transaction and combining both passes in to one. This 1) eliminates a lot of duplicate code between the passes, 2) speeds up the upgrade process, 3) makes progress reporting more accurate, 4) eliminates the potential for stale data if the upgrade is interrupted during the cleanup pass, and 5) makes it easier to reason about the safety of the upgrade code.
2014-08-25 19:26:03 +02:00
/* Perform the upgrade in a transaction. */
db->begin_transaction (true);
lib: Database version 3: Introduce fine-grained "features" Previously, our database schema was versioned by a single number. Each database schema change had to occur "atomically" in Notmuch's development history: before some commit, Notmuch used version N, after that commit, it used version N+1. Hence, each new schema version could introduce only one change, the task of developing a schema change fell on a single person, and it all had to happen and be perfect in a single commit series. This made introducing a new schema version hard. We've seen only two schema changes in the history of Notmuch. This commit introduces database schema version 3; hopefully the last schema version we'll need for a while. With this version, we switch from a single version number to "features": a set of named, independent aspects of the database schema. Features should make backwards compatibility easier. For many things, it should be easy to support databases both with and without a feature, which will allow us to make upgrades optional and will enable "unstable" features that can be developed and tested over time. Features also make forwards compatibility easier. The features recorded in a database include "compatibility flags," which can indicate to an older version of Notmuch when it must support a given feature to open the database for read or for write. This lets us replace the old vague "I don't recognize this version, so something might go wrong, but I promise to try my best" warnings upon opening a database with an unknown version with precise errors. If a database is safe to open for read/write despite unknown features, an older version will know that and issue no message at all. If the database is not safe to open for read/write because of unknown features, an older version will know that, too, and can tell the user exactly which required features it lacks support for.
2014-08-25 19:26:00 +02:00
/* Set the target features so we write out changes in the desired
* format. */
lib: Use database features to drive upgrade Previously, we had database version information hard-coded in the upgrade code. Slightly re-organize the upgrade process around the set of new database features to be enabled by the upgrade.
2014-08-25 19:26:04 +02:00
notmuch->features = target_features;
lib: Database version 3: Introduce fine-grained "features" Previously, our database schema was versioned by a single number. Each database schema change had to occur "atomically" in Notmuch's development history: before some commit, Notmuch used version N, after that commit, it used version N+1. Hence, each new schema version could introduce only one change, the task of developing a schema change fell on a single person, and it all had to happen and be perfect in a single commit series. This made introducing a new schema version hard. We've seen only two schema changes in the history of Notmuch. This commit introduces database schema version 3; hopefully the last schema version we'll need for a while. With this version, we switch from a single version number to "features": a set of named, independent aspects of the database schema. Features should make backwards compatibility easier. For many things, it should be easy to support databases both with and without a feature, which will allow us to make upgrades optional and will enable "unstable" features that can be developed and tested over time. Features also make forwards compatibility easier. The features recorded in a database include "compatibility flags," which can indicate to an older version of Notmuch when it must support a given feature to open the database for read or for write. This lets us replace the old vague "I don't recognize this version, so something might go wrong, but I promise to try my best" warnings upon opening a database with an unknown version with precise errors. If a database is safe to open for read/write despite unknown features, an older version will know that and issue no message at all. If the database is not safe to open for read/write because of unknown features, an older version will know that, too, and can tell the user exactly which required features it lacks support for.
2014-08-25 19:26:00 +02:00
lib: Reorganize upgrade around document types Rather than potentially making multiple passes over the same type of data in the database, reorganize upgrade around each type of data that may be upgraded. This eliminates code duplication, will make multi-version upgrades faster, and will let us improve progress reporting.
2014-08-25 19:26:05 +02:00
/* Perform per-message upgrades. */
if (new_features &
lib: Add per-message last modification tracking This adds a new document value that stores the revision of the last modification to message metadata, where the revision number increases monotonically with each database commit. An alternative would be to store the wall-clock time of the last modification of each message. In principle this is simpler and has the advantage that any process can determine the current timestamp without support from libnotmuch. However, even assuming a computer's clock never goes backward and ignoring clock skew in networked environments, this has a fatal flaw. Xapian uses (optimistic) snapshot isolation, which means reads can be concurrent with writes. Given this, consider the following time line with a write and two read transactions: write |-X-A--------------| read 1 |---B---| read 2 |---| The write transaction modifies message X and records the wall-clock time of the modification at A. The writer hangs around for a while and later commits its change. Read 1 is concurrent with the write, so it doesn't see the change to X. It does some query and records the wall-clock time of its results at B. Transaction read 2 later starts after the write commits and queries for changes since wall-clock time B (say the reads are performing an incremental backup). Even though read 1 could not see the change to X, read 2 is told (correctly) that X has not changed since B, the time of the last read. In fact, X changed before wall-clock time A, but the change was not visible until *after* wall-clock time B, so read 2 misses the change to X. This is tricky to solve in full-blown snapshot isolation, but because Xapian serializes writes, we can use a simple, monotonically increasing database revision number. Furthermore, maintaining this revision number requires no more IO than a wall-clock time solution because Xapian already maintains statistics on the upper (and lower) bound of each value stream.
2014-10-13 08:20:01 +02:00
(NOTMUCH_FEATURE_FILE_TERMS | NOTMUCH_FEATURE_BOOL_FOLDER |
NOTMUCH_FEATURE_LAST_MOD)) {
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
notmuch_messages_t *messages;
notmuch_message_t *message;
lib: Split the database upgrade into two phases for safer operation. The first phase copies data from the old format to the new format without deleting anything. This allows an old notmuch to still use the database if the upgrade process gets interrupted. The second phase performs the deletion (after updating the database version number). If the second phase is interrupted, there will be some unused data in the database, but it shouldn't cause any actual harm.
2010-01-09 20:13:12 +01:00
char *filename;
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
lib: move query variable to function scope This is a prelude to deallocating it (if necessary) on the error path.
2015-09-27 17:31:55 +02:00
query = notmuch_query_create (notmuch, "");
lib: replace deprecated n_q_search_messages with status returning version This function was deprecated in notmuch 0.21. We re-use the name for a status returning version, and deprecate the _st name.
2017-02-26 22:21:32 +01:00
status = notmuch_query_search_messages (query, &messages);
lib: migrate notmuch_database_upgrade to new query_search API Here we depend on the error path cleaning up query
2015-09-27 17:32:01 +02:00
if (status)
goto DONE;
for (;
lib: Rename iterator functions to prepare for reverse iteration. We rename 'has_more' to 'valid' so that it can function whether iterating in a forward or reverse direction. We also rename 'advance' to 'move_to_next' to setup parallel naming with the proposed functions 'move_to_first', 'move_to_last', and 'move_to_previous'.
2010-03-09 18:22:29 +01:00
notmuch_messages_valid (messages);
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
notmuch_messages_move_to_next (messages)) {
notmuch new: Fix progress notification on database upgrade. This was firing continuously rather than just once per second as intended.
2010-01-08 06:24:44 +01:00
if (do_progress_notify) {
notmuch new: Print upgrade progress report as a percentage. Previously we were printing a number of messages upgraded so far. The original motivation for this was to accurately reflect the fact that there are two passes, (so each message is processed twice and it's not accurate to represent with a single count). But as it turns out, the second pass takes zero time (relatively speaking) so we're still not accounting for it. If nothing else, the percentage-based reporting makes for a cleaner API for the progress_notify function.
2010-01-10 02:38:23 +01:00
progress_notify (closure, (double) count / total);
notmuch new: Fix progress notification on database upgrade. This was firing continuously rather than just once per second as intended.
2010-01-08 06:24:44 +01:00
do_progress_notify = 0;
}
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
message = notmuch_messages_get (messages);
lib: Reorganize upgrade around document types Rather than potentially making multiple passes over the same type of data in the database, reorganize upgrade around each type of data that may be upgraded. This eliminates code duplication, will make multi-version upgrades faster, and will let us improve progress reporting.
2014-08-25 19:26:05 +02:00
/* Before version 1, each message document had its
* filename in the data field. Copy that into the new
* format by calling notmuch_message_add_filename.
*/
if (new_features & NOTMUCH_FEATURE_FILE_TERMS) {
filename = _notmuch_message_talloc_copy_data (message);
if (filename && *filename != '\0') {
_notmuch_message_add_filename (message, filename);
_notmuch_message_clear_data (message);
}
talloc_free (filename);
lib: Split the database upgrade into two phases for safer operation. The first phase copies data from the old format to the new format without deleting anything. This allows an old notmuch to still use the database if the upgrade process gets interrupted. The second phase performs the deletion (after updating the database version number). If the second phase is interrupted, there will be some unused data in the database, but it shouldn't cause any actual harm.
2010-01-09 20:13:12 +01:00
}
lib: Reorganize upgrade around document types Rather than potentially making multiple passes over the same type of data in the database, reorganize upgrade around each type of data that may be upgraded. This eliminates code duplication, will make multi-version upgrades faster, and will let us improve progress reporting.
2014-08-25 19:26:05 +02:00
/* Prior to version 2, the "folder:" prefix was
* probabilistic and stemmed. Change it to the current
* boolean prefix. Add "path:" prefixes while at it.
*/
if (new_features & NOTMUCH_FEATURE_BOOL_FOLDER)
_notmuch_message_upgrade_folder (message);
lib: Add per-message last modification tracking This adds a new document value that stores the revision of the last modification to message metadata, where the revision number increases monotonically with each database commit. An alternative would be to store the wall-clock time of the last modification of each message. In principle this is simpler and has the advantage that any process can determine the current timestamp without support from libnotmuch. However, even assuming a computer's clock never goes backward and ignoring clock skew in networked environments, this has a fatal flaw. Xapian uses (optimistic) snapshot isolation, which means reads can be concurrent with writes. Given this, consider the following time line with a write and two read transactions: write |-X-A--------------| read 1 |---B---| read 2 |---| The write transaction modifies message X and records the wall-clock time of the modification at A. The writer hangs around for a while and later commits its change. Read 1 is concurrent with the write, so it doesn't see the change to X. It does some query and records the wall-clock time of its results at B. Transaction read 2 later starts after the write commits and queries for changes since wall-clock time B (say the reads are performing an incremental backup). Even though read 1 could not see the change to X, read 2 is told (correctly) that X has not changed since B, the time of the last read. In fact, X changed before wall-clock time A, but the change was not visible until *after* wall-clock time B, so read 2 misses the change to X. This is tricky to solve in full-blown snapshot isolation, but because Xapian serializes writes, we can use a simple, monotonically increasing database revision number. Furthermore, maintaining this revision number requires no more IO than a wall-clock time solution because Xapian already maintains statistics on the upper (and lower) bound of each value stream.
2014-10-13 08:20:01 +02:00
/* Prior to NOTMUCH_FEATURE_LAST_MOD, messages did not
* track modification revisions. Give all messages the
* next available revision; since we just started tracking
* revisions for this database, that will be 1.
*/
if (new_features & NOTMUCH_FEATURE_LAST_MOD)
_notmuch_message_upgrade_last_mod (message);
lib: Reorganize upgrade around document types Rather than potentially making multiple passes over the same type of data in the database, reorganize upgrade around each type of data that may be upgraded. This eliminates code duplication, will make multi-version upgrades faster, and will let us improve progress reporting.
2014-08-25 19:26:05 +02:00
_notmuch_message_sync (message);
lib: Split the database upgrade into two phases for safer operation. The first phase copies data from the old format to the new format without deleting anything. This allows an old notmuch to still use the database if the upgrade process gets interrupted. The second phase performs the deletion (after updating the database version number). If the second phase is interrupted, there will be some unused data in the database, but it shouldn't cause any actual harm.
2010-01-09 20:13:12 +01:00
notmuch_message_destroy (message);
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
count++;
}
lib: Split the database upgrade into two phases for safer operation. The first phase copies data from the old format to the new format without deleting anything. This allows an old notmuch to still use the database if the upgrade process gets interrupted. The second phase performs the deletion (after updating the database version number). If the second phase is interrupted, there will be some unused data in the database, but it shouldn't cause any actual harm.
2010-01-09 20:13:12 +01:00
notmuch_query_destroy (query);
lib: add versions of n_q_count_{message,threads} with status return Although I think it's a pretty bad idea to continue using the old API, this allows both a more gentle transition for clients of the library, and allows us to break one monolithic change into a series
2015-09-27 17:31:57 +02:00
query = NULL;
lib: Use database features to drive upgrade Previously, we had database version information hard-coded in the upgrade code. Slightly re-organize the upgrade process around the set of new database features to be enabled by the upgrade.
2014-08-25 19:26:04 +02:00
}
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
lib: Reorganize upgrade around document types Rather than potentially making multiple passes over the same type of data in the database, reorganize upgrade around each type of data that may be upgraded. This eliminates code duplication, will make multi-version upgrades faster, and will let us improve progress reporting.
2014-08-25 19:26:05 +02:00
/* Perform per-directory upgrades. */
/* Before version 1 we stored directory timestamps in
lib: Use database features to drive upgrade Previously, we had database version information hard-coded in the upgrade code. Slightly re-organize the upgrade process around the set of new database features to be enabled by the upgrade.
2014-08-25 19:26:04 +02:00
* XTIMESTAMP documents instead of the current XDIRECTORY
* documents. So copy those as well. */
if (new_features & NOTMUCH_FEATURE_DIRECTORY_DOCS) {
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
t_end = notmuch->xapian_db->allterms_end ("XTIMESTAMP");
for (t = notmuch->xapian_db->allterms_begin ("XTIMESTAMP");
t != t_end;
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
t++) {
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
Xapian::PostingIterator p, p_end;
std::string term = *t;
p_end = notmuch->xapian_db->postlist_end (term);
for (p = notmuch->xapian_db->postlist_begin (term);
p != p_end;
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
p++) {
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
Xapian::Document document;
time_t mtime;
notmuch_directory_t *directory;
notmuch new: Print upgrade progress report as a percentage. Previously we were printing a number of messages upgraded so far. The original motivation for this was to accurately reflect the fact that there are two passes, (so each message is processed twice and it's not accurate to represent with a single count). But as it turns out, the second pass takes zero time (relatively speaking) so we're still not accounting for it. If nothing else, the percentage-based reporting makes for a cleaner API for the progress_notify function.
2010-01-10 02:38:23 +01:00
if (do_progress_notify) {
progress_notify (closure, (double) count / total);
do_progress_notify = 0;
}
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
document = find_document_for_doc_id (notmuch, *p);
mtime = Xapian::sortable_unserialise (
document.get_value (NOTMUCH_VALUE_TIMESTAMP));
lib: rename _n_d_create to _n_d_find_or_create The error message and name were confusing when called in some "read only" context.
2020-07-19 12:36:47 +02:00
directory = _notmuch_directory_find_or_create (notmuch, term.c_str () + 10,
NOTMUCH_FIND_CREATE, &status);
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
notmuch_directory_set_mtime (directory, mtime);
notmuch_directory_destroy (directory);
lib: Simplify upgrade code using a transaction Previously, the upgrade was organized as two passes -- an upgrade pass, and a separate cleanup pass -- so the database was always in a valid state. This change substantially simplifies this code by performing the upgrade in a transaction and combining both passes in to one. This 1) eliminates a lot of duplicate code between the passes, 2) speeds up the upgrade process, 3) makes progress reporting more accurate, 4) eliminates the potential for stale data if the upgrade is interrupted during the cleanup pass, and 5) makes it easier to reason about the safety of the upgrade code.
2014-08-25 19:26:03 +02:00
db->delete_document (*p);
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
}
lib: Report progress for combined upgrade operation Previously, some parts of upgrade didn't report progress and for others it was possible for the progress meter to restart at 0 part way through the upgrade because each stage was reported separately. Fix this by computing the total amount of work that needs to be done up-front and updating completed work monotonically.
2014-08-25 19:26:06 +02:00
++count;
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
}
}
lib: Implement upgrade to ghost messages feature Somehow this is the first upgrade pass that actually does *any* error checking, so this also adds the bit of necessary infrastructure to handle that.
2014-10-23 14:30:38 +02:00
/* Perform metadata upgrades. */
/* Prior to NOTMUCH_FEATURE_GHOSTS, thread IDs for missing
* messages were stored as database metadata. Change these to
* ghost messages.
*/
if (new_features & NOTMUCH_FEATURE_GHOSTS) {
notmuch_message_t *message;
std::string message_id, thread_id;
t_end = db->metadata_keys_end (NOTMUCH_METADATA_THREAD_ID_PREFIX);
for (t = db->metadata_keys_begin (NOTMUCH_METADATA_THREAD_ID_PREFIX);
t != t_end; ++t) {
if (do_progress_notify) {
progress_notify (closure, (double) count / total);
do_progress_notify = 0;
}
message_id = (*t).substr (
strlen (NOTMUCH_METADATA_THREAD_ID_PREFIX));
thread_id = db->get_metadata (*t);
/* Create ghost message */
message = _notmuch_message_create_for_message_id (
notmuch, message_id.c_str (), &private_status);
if (private_status == NOTMUCH_PRIVATE_STATUS_SUCCESS) {
/* Document already exists; ignore the stored thread ID */
} else if (private_status ==
NOTMUCH_PRIVATE_STATUS_NO_DOCUMENT_FOUND) {
private_status = _notmuch_message_initialize_ghost (
message, thread_id.c_str ());
if (! private_status)
_notmuch_message_sync (message);
}
if (private_status) {
lib: replace almost all fprintfs in library with _n_d_log This is not supposed to change any functionality from an end user point of view. Note that it will eliminate some output to stderr. The query debugging output is left as is; it doesn't really fit with the current primitive logging model. The remaining "bad" fprintf will need an internal API change.
2014-12-26 17:25:35 +01:00
_notmuch_database_log (notmuch,
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
"Upgrade failed while creating ghost messages.\n");
lib: Implement upgrade to ghost messages feature Somehow this is the first upgrade pass that actually does *any* error checking, so this also adds the bit of necessary infrastructure to handle that.
2014-10-23 14:30:38 +02:00
status = COERCE_STATUS (private_status, "Unexpected status from _notmuch_message_initialize_ghost");
goto DONE;
}
/* Clear saved metadata thread ID */
db->set_metadata (*t, "");
++count;
}
}
status = NOTMUCH_STATUS_SUCCESS;
lib: factor out feature name related code. database.cc is uncomfortably large, and some of the static data structures do not need to be shared as much as they are. This is a somewhat small piece to factor out, but it will turn out to be helpful to further refactoring.
2020-08-08 16:16:43 +02:00
db->set_metadata ("features", _notmuch_database_print_features (local, notmuch->features));
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
db->set_metadata ("version", STRINGIFY (NOTMUCH_DATABASE_VERSION));
lib: Split the database upgrade into two phases for safer operation. The first phase copies data from the old format to the new format without deleting anything. This allows an old notmuch to still use the database if the upgrade process gets interrupted. The second phase performs the deletion (after updating the database version number). If the second phase is interrupted, there will be some unused data in the database, but it shouldn't cause any actual harm.
2010-01-09 20:13:12 +01:00
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
DONE:
lib: Implement upgrade to ghost messages feature Somehow this is the first upgrade pass that actually does *any* error checking, so this also adds the bit of necessary infrastructure to handle that.
2014-10-23 14:30:38 +02:00
if (status == NOTMUCH_STATUS_SUCCESS)
db->commit_transaction ();
else
db->cancel_transaction ();
lib: Split the database upgrade into two phases for safer operation. The first phase copies data from the old format to the new format without deleting anything. This allows an old notmuch to still use the database if the upgrade process gets interrupted. The second phase performs the deletion (after updating the database version number). If the second phase is interrupted, there will be some unused data in the database, but it shouldn't cause any actual harm.
2010-01-09 20:13:12 +01:00
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
if (timer_is_active) {
/* Now stop the timer. */
timerval.it_interval.tv_sec = 0;
timerval.it_interval.tv_usec = 0;
timerval.it_value.tv_sec = 0;
timerval.it_value.tv_usec = 0;
setitimer (ITIMER_REAL, &timerval, NULL);
/* And disable the signal handler. */
action.sa_handler = SIG_IGN;
sigaction (SIGALRM, &action, NULL);
}
lib: add versions of n_q_count_{message,threads} with status return Although I think it's a pretty bad idea to continue using the old API, this allows both a more gentle transition for clients of the library, and allows us to break one monolithic change into a series
2015-09-27 17:31:57 +02:00
if (query)
notmuch_query_destroy (query);
lib: Database version 3: Introduce fine-grained "features" Previously, our database schema was versioned by a single number. Each database schema change had to occur "atomically" in Notmuch's development history: before some commit, Notmuch used version N, after that commit, it used version N+1. Hence, each new schema version could introduce only one change, the task of developing a schema change fell on a single person, and it all had to happen and be perfect in a single commit series. This made introducing a new schema version hard. We've seen only two schema changes in the history of Notmuch. This commit introduces database schema version 3; hopefully the last schema version we'll need for a while. With this version, we switch from a single version number to "features": a set of named, independent aspects of the database schema. Features should make backwards compatibility easier. For many things, it should be easy to support databases both with and without a feature, which will allow us to make upgrades optional and will enable "unstable" features that can be developed and tested over time. Features also make forwards compatibility easier. The features recorded in a database include "compatibility flags," which can indicate to an older version of Notmuch when it must support a given feature to open the database for read or for write. This lets us replace the old vague "I don't recognize this version, so something might go wrong, but I promise to try my best" warnings upon opening a database with an unknown version with precise errors. If a database is safe to open for read/write despite unknown features, an older version will know that and issue no message at all. If the database is not safe to open for read/write because of unknown features, an older version will know that, too, and can tell the user exactly which required features it lacks support for.
2014-08-25 19:26:00 +02:00
talloc_free (local);
lib: Implement upgrade to ghost messages feature Somehow this is the first upgrade pass that actually does *any* error checking, so this also adds the bit of necessary infrastructure to handle that.
2014-10-23 14:30:38 +02:00
return status;
lib: Implement versioning in the database and provide upgrade function. The recent support for renames in the database is our first time (since notmuch has had more than a single user) that we have a database format change. To support smooth upgrades we now encode a database format version number in the Xapian metadata. Going forward notmuch will emit a warning if used to read from a database with a newer version than it natively supports, and will refuse to write to a database with a newer version. The library also provides functions to query the database format version: notmuch_database_get_version to ask if notmuch wants a newer version than that: notmuch_database_needs_upgrade and a function to actually perform that upgrade: notmuch_database_upgrade
2010-01-08 03:26:31 +01:00
}
lib: Add notmuch_database_{begin,end}_atomic. These operations translate into non-flushed Xapian transactions, allowing arbitrary groups of database operations to be performed atomically.
2011-01-29 17:25:24 +01:00
notmuch_status_t
notmuch_database_begin_atomic (notmuch_database_t *notmuch)
{
lib: encapsulate the use of notmuch_database_t field 'mode' The plan is to change the underlying representation.
2020-07-27 01:31:35 +02:00
if (_notmuch_database_mode (notmuch) == NOTMUCH_DATABASE_MODE_READ_ONLY ||
lib: Add support for nested atomic sections. notmuch_database_t now keeps a nesting count and we only start a transaction or commit for the outermost atomic section. Introduces a new error, NOTMUCH_STATUS_UNBALANCED_ATOMIC.
2011-06-11 05:35:06 +02:00
notmuch->atomic_nesting > 0)
goto DONE;
lib: Add notmuch_database_{begin,end}_atomic. These operations translate into non-flushed Xapian transactions, allowing arbitrary groups of database operations to be performed atomically.
2011-01-29 17:25:24 +01:00
lib/database.cc: fix misleading indentation Found by gcc 6.1.1 -Wmisleading-indentation option (set by -Wall).
2016-09-27 17:06:52 +02:00
if (notmuch_database_needs_upgrade (notmuch))
return NOTMUCH_STATUS_UPGRADE_REQUIRED;
forbid atomic transactions on writable, upgradable databases We can't (but currently do) allow upgrades within transactions because upgrades need their own transactions. We don't want to re-use the current transaction because bailing out of an upgrade would mean loosing all previous changes (because our "atomic" transactions don't commit before hand). This gives us two options: 1. Fail at the beginning of upgrade (tell the user to end the transaction, upgrade, and start over). 2. Don't allow the user to start the transaction. I went with the latter because: 1. There is no reason to call `begin_atomic` unless you intend to to write to the database and anyone intending to write to the database should upgrade it first. 2. This means that nothing inside an atomic transaction can ever fail with NOTMUCH_STATUS_UPGRADE_REQUIRED.
2015-10-25 22:30:39 +01:00
lib: Add notmuch_database_{begin,end}_atomic. These operations translate into non-flushed Xapian transactions, allowing arbitrary groups of database operations to be performed atomically.
2011-01-29 17:25:24 +01:00
try {
lib: replace use of static_cast for writable databases static_cast is a bit tricky to understand and error prone, so add a second pointer to (potentially the same) Xapian database object that we know has the right subclass.
2020-07-27 01:31:36 +02:00
notmuch->writable_xapian_db->begin_transaction (false);
lib: Add notmuch_database_{begin,end}_atomic. These operations translate into non-flushed Xapian transactions, allowing arbitrary groups of database operations to be performed atomically.
2011-01-29 17:25:24 +01:00
} catch (const Xapian::Error &error) {
lib: replace almost all fprintfs in library with _n_d_log This is not supposed to change any functionality from an end user point of view. Note that it will eliminate some output to stderr. The query debugging output is left as is; it doesn't really fit with the current primitive logging model. The remaining "bad" fprintf will need an internal API change.
2014-12-26 17:25:35 +01:00
_notmuch_database_log (notmuch, "A Xapian exception occurred beginning transaction: %s.\n",
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
error.get_msg ().c_str ());
lib: convert notmuch_bool_t to stdbool internally C99 stdbool turned 18 this year. There really is no reason to use our own, except in the library interface for backward compatibility. Convert the lib internally to stdbool.
2017-10-07 10:44:05 +02:00
notmuch->exception_reported = true;
lib: Add notmuch_database_{begin,end}_atomic. These operations translate into non-flushed Xapian transactions, allowing arbitrary groups of database operations to be performed atomically.
2011-01-29 17:25:24 +01:00
return NOTMUCH_STATUS_XAPIAN_EXCEPTION;
}
lib: Add support for nested atomic sections. notmuch_database_t now keeps a nesting count and we only start a transaction or commit for the outermost atomic section. Introduces a new error, NOTMUCH_STATUS_UNBALANCED_ATOMIC.
2011-06-11 05:35:06 +02:00
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
DONE:
lib: Add support for nested atomic sections. notmuch_database_t now keeps a nesting count and we only start a transaction or commit for the outermost atomic section. Introduces a new error, NOTMUCH_STATUS_UNBALANCED_ATOMIC.
2011-06-11 05:35:06 +02:00
notmuch->atomic_nesting++;
lib: Add notmuch_database_{begin,end}_atomic. These operations translate into non-flushed Xapian transactions, allowing arbitrary groups of database operations to be performed atomically.
2011-01-29 17:25:24 +01:00
return NOTMUCH_STATUS_SUCCESS;
}
notmuch_status_t
notmuch_database_end_atomic (notmuch_database_t *notmuch)
{
Xapian::WritableDatabase *db;
lib: Add support for nested atomic sections. notmuch_database_t now keeps a nesting count and we only start a transaction or commit for the outermost atomic section. Introduces a new error, NOTMUCH_STATUS_UNBALANCED_ATOMIC.
2011-06-11 05:35:06 +02:00
if (notmuch->atomic_nesting == 0)
return NOTMUCH_STATUS_UNBALANCED_ATOMIC;
lib: encapsulate the use of notmuch_database_t field 'mode' The plan is to change the underlying representation.
2020-07-27 01:31:35 +02:00
if (_notmuch_database_mode (notmuch) == NOTMUCH_DATABASE_MODE_READ_ONLY ||
lib: Add support for nested atomic sections. notmuch_database_t now keeps a nesting count and we only start a transaction or commit for the outermost atomic section. Introduces a new error, NOTMUCH_STATUS_UNBALANCED_ATOMIC.
2011-06-11 05:35:06 +02:00
notmuch->atomic_nesting > 1)
goto DONE;
lib: Add notmuch_database_{begin,end}_atomic. These operations translate into non-flushed Xapian transactions, allowing arbitrary groups of database operations to be performed atomically.
2011-01-29 17:25:24 +01:00
lib: replace use of static_cast for writable databases static_cast is a bit tricky to understand and error prone, so add a second pointer to (potentially the same) Xapian database object that we know has the right subclass.
2020-07-27 01:31:36 +02:00
db = notmuch->writable_xapian_db;
lib: Add notmuch_database_{begin,end}_atomic. These operations translate into non-flushed Xapian transactions, allowing arbitrary groups of database operations to be performed atomically.
2011-01-29 17:25:24 +01:00
try {
db->commit_transaction ();
/* This is a hack for testing. Xapian never flushes on a
* non-flushed commit, even if the flush threshold is 1.
* However, we rely on flushing to test atomicity. */
const char *thresh = getenv ("XAPIAN_FLUSH_THRESHOLD");
if (thresh && atoi (thresh) == 1)
lib: replace deprecated xapian call 'flush()' with 'commit()' This will make notmuch incompatible with Xapian before 1.1.0, which is more than 6 years old this point.
2016-10-05 13:34:33 +02:00
db->commit ();
lib: Add notmuch_database_{begin,end}_atomic. These operations translate into non-flushed Xapian transactions, allowing arbitrary groups of database operations to be performed atomically.
2011-01-29 17:25:24 +01:00
} catch (const Xapian::Error &error) {
lib: replace almost all fprintfs in library with _n_d_log This is not supposed to change any functionality from an end user point of view. Note that it will eliminate some output to stderr. The query debugging output is left as is; it doesn't really fit with the current primitive logging model. The remaining "bad" fprintf will need an internal API change.
2014-12-26 17:25:35 +01:00
_notmuch_database_log (notmuch, "A Xapian exception occurred committing transaction: %s.\n",
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
error.get_msg ().c_str ());
lib: convert notmuch_bool_t to stdbool internally C99 stdbool turned 18 this year. There really is no reason to use our own, except in the library interface for backward compatibility. Convert the lib internally to stdbool.
2017-10-07 10:44:05 +02:00
notmuch->exception_reported = true;
lib: Add notmuch_database_{begin,end}_atomic. These operations translate into non-flushed Xapian transactions, allowing arbitrary groups of database operations to be performed atomically.
2011-01-29 17:25:24 +01:00
return NOTMUCH_STATUS_XAPIAN_EXCEPTION;
}
lib: Add support for nested atomic sections. notmuch_database_t now keeps a nesting count and we only start a transaction or commit for the outermost atomic section. Introduces a new error, NOTMUCH_STATUS_UNBALANCED_ATOMIC.
2011-06-11 05:35:06 +02:00
lib: Add per-message last modification tracking This adds a new document value that stores the revision of the last modification to message metadata, where the revision number increases monotonically with each database commit. An alternative would be to store the wall-clock time of the last modification of each message. In principle this is simpler and has the advantage that any process can determine the current timestamp without support from libnotmuch. However, even assuming a computer's clock never goes backward and ignoring clock skew in networked environments, this has a fatal flaw. Xapian uses (optimistic) snapshot isolation, which means reads can be concurrent with writes. Given this, consider the following time line with a write and two read transactions: write |-X-A--------------| read 1 |---B---| read 2 |---| The write transaction modifies message X and records the wall-clock time of the modification at A. The writer hangs around for a while and later commits its change. Read 1 is concurrent with the write, so it doesn't see the change to X. It does some query and records the wall-clock time of its results at B. Transaction read 2 later starts after the write commits and queries for changes since wall-clock time B (say the reads are performing an incremental backup). Even though read 1 could not see the change to X, read 2 is told (correctly) that X has not changed since B, the time of the last read. In fact, X changed before wall-clock time A, but the change was not visible until *after* wall-clock time B, so read 2 misses the change to X. This is tricky to solve in full-blown snapshot isolation, but because Xapian serializes writes, we can use a simple, monotonically increasing database revision number. Furthermore, maintaining this revision number requires no more IO than a wall-clock time solution because Xapian already maintains statistics on the upper (and lower) bound of each value stream.
2014-10-13 08:20:01 +02:00
if (notmuch->atomic_dirty) {
++notmuch->revision;
lib: convert notmuch_bool_t to stdbool internally C99 stdbool turned 18 this year. There really is no reason to use our own, except in the library interface for backward compatibility. Convert the lib internally to stdbool.
2017-10-07 10:44:05 +02:00
notmuch->atomic_dirty = false;
lib: Add per-message last modification tracking This adds a new document value that stores the revision of the last modification to message metadata, where the revision number increases monotonically with each database commit. An alternative would be to store the wall-clock time of the last modification of each message. In principle this is simpler and has the advantage that any process can determine the current timestamp without support from libnotmuch. However, even assuming a computer's clock never goes backward and ignoring clock skew in networked environments, this has a fatal flaw. Xapian uses (optimistic) snapshot isolation, which means reads can be concurrent with writes. Given this, consider the following time line with a write and two read transactions: write |-X-A--------------| read 1 |---B---| read 2 |---| The write transaction modifies message X and records the wall-clock time of the modification at A. The writer hangs around for a while and later commits its change. Read 1 is concurrent with the write, so it doesn't see the change to X. It does some query and records the wall-clock time of its results at B. Transaction read 2 later starts after the write commits and queries for changes since wall-clock time B (say the reads are performing an incremental backup). Even though read 1 could not see the change to X, read 2 is told (correctly) that X has not changed since B, the time of the last read. In fact, X changed before wall-clock time A, but the change was not visible until *after* wall-clock time B, so read 2 misses the change to X. This is tricky to solve in full-blown snapshot isolation, but because Xapian serializes writes, we can use a simple, monotonically increasing database revision number. Furthermore, maintaining this revision number requires no more IO than a wall-clock time solution because Xapian already maintains statistics on the upper (and lower) bound of each value stream.
2014-10-13 08:20:01 +02:00
}
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
DONE:
lib: Add support for nested atomic sections. notmuch_database_t now keeps a nesting count and we only start a transaction or commit for the outermost atomic section. Introduces a new error, NOTMUCH_STATUS_UNBALANCED_ATOMIC.
2011-06-11 05:35:06 +02:00
notmuch->atomic_nesting--;
lib: Add notmuch_database_{begin,end}_atomic. These operations translate into non-flushed Xapian transactions, allowing arbitrary groups of database operations to be performed atomically.
2011-01-29 17:25:24 +01:00
return NOTMUCH_STATUS_SUCCESS;
}
lib: API to retrieve database revision and UUID This exposes the committed database revision to library users along with a UUID that can be used to detect when revision numbers are no longer comparable (e.g., because the database has been replaced).
2014-10-13 08:20:02 +02:00
unsigned long
notmuch_database_get_revision (notmuch_database_t *notmuch,
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
const char **uuid)
lib: API to retrieve database revision and UUID This exposes the committed database revision to library users along with a UUID that can be used to detect when revision numbers are no longer comparable (e.g., because the database has been replaced).
2014-10-13 08:20:02 +02:00
{
if (uuid)
*uuid = notmuch->uuid;
return notmuch->revision;
}
lib: Rename set/get_timestamp to set/get_directory_mtime. I've been suitably scolded by Keith for doing a premature generalization that ended up just making the documentation more convoluted. Fix that.
2009-12-17 23:33:34 +01:00
/* We allow the user to use arbitrarily long paths for directories. But
* we have a term-length limit. So if we exceed that, we'll use the
* SHA-1 of the path for the database term.
Add notmuch_database_set_timestamp and notmuch_database_get_timestamp These will be very helpful to implement an efficient "notmuch new" command which imports new mail messages that have appeared.
2009-10-23 23:31:01 +02:00
*
lib: Rename set/get_timestamp to set/get_directory_mtime. I've been suitably scolded by Keith for doing a premature generalization that ended up just making the documentation more convoluted. Fix that.
2009-12-17 23:33:34 +01:00
* Note: This function may return the original value of 'path'. If it
* does not, then the caller is responsible to free() the returned
* value.
Add notmuch_database_set_timestamp and notmuch_database_get_timestamp These will be very helpful to implement an efficient "notmuch new" command which imports new mail messages that have appeared.
2009-10-23 23:31:01 +02:00
*/
lib: Implement new notmuch_directory_t API. This new directory ojbect provides all the infrastructure needed to detect when files or directories are deleted or renamed. There's still code needed on top of this (within "notmuch new") to actually do that detection.
2010-01-05 22:29:23 +01:00
const char *
_notmuch_database_get_directory_db_path (const char *path)
Add notmuch_database_set_timestamp and notmuch_database_get_timestamp These will be very helpful to implement an efficient "notmuch new" command which imports new mail messages that have appeared.
2009-10-23 23:31:01 +02:00
{
lib: Rename set/get_timestamp to set/get_directory_mtime. I've been suitably scolded by Keith for doing a premature generalization that ended up just making the documentation more convoluted. Fix that.
2009-12-17 23:33:34 +01:00
int term_len = strlen (_find_prefix ("directory")) + strlen (path);
Fix timestamp generation to avoid overflowing the term limit The previous code was only correct as long as the timestamp prefix was only a single character. But with the recent change to a multi-character prefix, this broke. So fix it now.
2009-10-25 06:10:03 +01:00
if (term_len > NOTMUCH_TERM_MAX)
lib: Start all function names in notmuch-private.h with As noted in devel/STYLE, every private library function should start with _notmuch. This patch corrects function naming that did not adhere to this style in lib/notmuch-private.h. In particular, the old function names that now begin with _notmuch are notmuch_sha1_of_file notmuch_sha1_of_string notmuch_message_file_close notmuch_message_file_get_header notmuch_message_file_open notmuch_message_get_author notmuch_message_set_author Signed-off-by: Charles Celerier <cceleri@cs.stanford.edu>
2014-05-13 11:44:05 +02:00
return _notmuch_sha1_of_string (path);
Fix timestamp generation to avoid overflowing the term limit The previous code was only correct as long as the timestamp prefix was only a single character. But with the recent change to a multi-character prefix, this broke. So fix it now.
2009-10-25 06:10:03 +01:00
else
lib: Rename set/get_timestamp to set/get_directory_mtime. I've been suitably scolded by Keith for doing a premature generalization that ended up just making the documentation more convoluted. Fix that.
2009-12-17 23:33:34 +01:00
return path;
Add notmuch_database_set_timestamp and notmuch_database_get_timestamp These will be very helpful to implement an efficient "notmuch new" command which imports new mail messages that have appeared.
2009-10-23 23:31:01 +02:00
}
database: Split _find_parent_id into _split_path and _find_directory_id Some pending commits want the _split_path functionality separate from mapping a directory to a document ID. The split_path function now returns the basename as well as the directory name.
2009-12-21 17:14:52 +01:00
/* Given a path, split it into two parts: the directory part is all
* components except for the last, and the basename is that last
* component. Getting the return-value for either part is optional
* (the caller can pass NULL).
database: Export _notmuch_database_find_parent_id for internal use. We'll soon have mail documents referring to their parent directory's directory documents, so we'll need access to _find_parent_id in files such as message.cc.
2009-12-20 00:11:55 +01:00
*
* The original 'path' can represent either a regular file or a
database: Split _find_parent_id into _split_path and _find_directory_id Some pending commits want the _split_path functionality separate from mapping a directory to a document ID. The split_path function now returns the basename as well as the directory name.
2009-12-21 17:14:52 +01:00
* directory---the splitting will be carried out in the same way in
* either case. Trailing slashes on 'path' will be ignored, and any
database: Export _notmuch_database_find_parent_id for internal use. We'll soon have mail documents referring to their parent directory's directory documents, so we'll need access to _find_parent_id in files such as message.cc.
2009-12-20 00:11:55 +01:00
* cases of multiple '/' characters appearing in series will be
* treated as a single '/'.
*
database: Split _find_parent_id into _split_path and _find_directory_id Some pending commits want the _split_path functionality separate from mapping a directory to a document ID. The split_path function now returns the basename as well as the directory name.
2009-12-21 17:14:52 +01:00
* Allocation (if any) will have 'ctx' as the talloc owner. But
* pointers will be returned within the original path string whenever
* possible.
database: Export _notmuch_database_find_parent_id for internal use. We'll soon have mail documents referring to their parent directory's directory documents, so we'll need access to _find_parent_id in files such as message.cc.
2009-12-20 00:11:55 +01:00
*
database: Split _find_parent_id into _split_path and _find_directory_id Some pending commits want the _split_path functionality separate from mapping a directory to a document ID. The split_path function now returns the basename as well as the directory name.
2009-12-21 17:14:52 +01:00
* Note: If 'path' is non-empty and contains no non-trailing slash,
* (that is, consists of a filename with no parent directory), then
* the directory returned will be an empty string. However, if 'path'
* is an empty string, then both directory and basename will be
* returned as NULL.
database: Export _notmuch_database_find_parent_id for internal use. We'll soon have mail documents referring to their parent directory's directory documents, so we'll need access to _find_parent_id in files such as message.cc.
2009-12-20 00:11:55 +01:00
*/
database: Store the parent ID for each directory document. Storing the document ID of the parent of each directory document will allow us to find all child-directory documents for a given directory document. We will need this in order to detect directories that have been removed from the mail store, (though we aren't yet doing this).
2009-12-19 22:20:26 +01:00
notmuch_status_t
database: Split _find_parent_id into _split_path and _find_directory_id Some pending commits want the _split_path functionality separate from mapping a directory to a document ID. The split_path function now returns the basename as well as the directory name.
2009-12-21 17:14:52 +01:00
_notmuch_database_split_path (void *ctx,
const char *path,
const char **directory,
const char **basename)
database: Store the parent ID for each directory document. Storing the document ID of the parent of each directory document will allow us to find all child-directory documents for a given directory document. We will need this in order to detect directories that have been removed from the mail store, (though we aren't yet doing this).
2009-12-19 22:20:26 +01:00
{
database: Split _find_parent_id into _split_path and _find_directory_id Some pending commits want the _split_path functionality separate from mapping a directory to a document ID. The split_path function now returns the basename as well as the directory name.
2009-12-21 17:14:52 +01:00
const char *slash;
database: Store the parent ID for each directory document. Storing the document ID of the parent of each directory document will allow us to find all child-directory documents for a given directory document. We will need this in order to detect directories that have been removed from the mail store, (though we aren't yet doing this).
2009-12-19 22:20:26 +01:00
if (path == NULL || *path == '\0') {
database: Split _find_parent_id into _split_path and _find_directory_id Some pending commits want the _split_path functionality separate from mapping a directory to a document ID. The split_path function now returns the basename as well as the directory name.
2009-12-21 17:14:52 +01:00
if (directory)
*directory = NULL;
if (basename)
*basename = NULL;
database: Store the parent ID for each directory document. Storing the document ID of the parent of each directory document will allow us to find all child-directory documents for a given directory document. We will need this in order to detect directories that have been removed from the mail store, (though we aren't yet doing this).
2009-12-19 22:20:26 +01:00
return NOTMUCH_STATUS_SUCCESS;
}
/* Find the last slash (not counting a trailing slash), if any. */
slash = path + strlen (path) - 1;
/* First, skip trailing slashes. */
lib: clean up _notmuch_database_split_path Make the logic it a bit easier to read. No functional changes.
2016-04-10 21:43:23 +02:00
while (slash != path && *slash == '/')
database: Store the parent ID for each directory document. Storing the document ID of the parent of each directory document will allow us to find all child-directory documents for a given directory document. We will need this in order to detect directories that have been removed from the mail store, (though we aren't yet doing this).
2009-12-19 22:20:26 +01:00
--slash;
/* Then, find a slash. */
lib: clean up _notmuch_database_split_path Make the logic it a bit easier to read. No functional changes.
2016-04-10 21:43:23 +02:00
while (slash != path && *slash != '/') {
database: Split _find_parent_id into _split_path and _find_directory_id Some pending commits want the _split_path functionality separate from mapping a directory to a document ID. The split_path function now returns the basename as well as the directory name.
2009-12-21 17:14:52 +01:00
if (basename)
*basename = slash;
database: Store the parent ID for each directory document. Storing the document ID of the parent of each directory document will allow us to find all child-directory documents for a given directory document. We will need this in order to detect directories that have been removed from the mail store, (though we aren't yet doing this).
2009-12-19 22:20:26 +01:00
--slash;
}
/* Finally, skip multiple slashes. */
lib: clean up _notmuch_database_split_path Make the logic it a bit easier to read. No functional changes.
2016-04-10 21:43:23 +02:00
while (slash != path && *(slash - 1) == '/')
database: Store the parent ID for each directory document. Storing the document ID of the parent of each directory document will allow us to find all child-directory documents for a given directory document. We will need this in order to detect directories that have been removed from the mail store, (though we aren't yet doing this).
2009-12-19 22:20:26 +01:00
--slash;
database: Split _find_parent_id into _split_path and _find_directory_id Some pending commits want the _split_path functionality separate from mapping a directory to a document ID. The split_path function now returns the basename as well as the directory name.
2009-12-21 17:14:52 +01:00
if (slash == path) {
if (directory)
*directory = talloc_strdup (ctx, "");
if (basename)
*basename = path;
} else {
if (directory)
lib: fix handling of one character long directory names at top level The code to skip multiple slashes in _notmuch_database_split_path() skips back one character too much. This is compensated by a +1 in the length parameter to the strndup() call. Mostly this works fine, but if the path is to a file under a top level directory with one character long name, the directory part is mistaken to be part of the file name (slash == path in code). The returned directory name will be the empty string and the basename will be the full path, breaking the indexing logic in notmuch new. Fix the multiple slash skipping to keep the slash variable pointing at the last slash, and adjust strndup() accordingly. The bug was introduced in commit e890b0cf4011fd9fd77ebd87343379e4a778888b Author: Carl Worth <cworth@cworth.org> Date: Sat Dec 19 13:20:26 2009 -0800 database: Store the parent ID for each directory document. just a little over two months after the initial commit in the Notmuch code history, making this the longest living bug in Notmuch to date.
2016-04-10 21:43:22 +02:00
*directory = talloc_strndup (ctx, path, slash - path);
database: Split _find_parent_id into _split_path and _find_directory_id Some pending commits want the _split_path functionality separate from mapping a directory to a document ID. The split_path function now returns the basename as well as the directory name.
2009-12-21 17:14:52 +01:00
}
database: Store the parent ID for each directory document. Storing the document ID of the parent of each directory document will allow us to find all child-directory documents for a given directory document. We will need this in order to detect directories that have been removed from the mail store, (though we aren't yet doing this).
2009-12-19 22:20:26 +01:00
database: Split _find_parent_id into _split_path and _find_directory_id Some pending commits want the _split_path functionality separate from mapping a directory to a document ID. The split_path function now returns the basename as well as the directory name.
2009-12-21 17:14:52 +01:00
return NOTMUCH_STATUS_SUCCESS;
}
lib: Perform the same transformation to _notmuch_database_find_directory_id Now _notmuch_database_find_directory_id takes a flags argument, which it passes through to _notmuch_directory_create and can indicate if the directory does not exist. Again, callers have been updated, but retain their original behavior.
2012-05-18 06:13:35 +02:00
/* Find the document ID of the specified directory.
*
* If (flags & NOTMUCH_FIND_CREATE), a new directory document will be
* created if one does not exist for 'path'. Otherwise, if the
* directory document does not exist, this sets *directory_id to
* ((unsigned int)-1) and returns NOTMUCH_STATUS_SUCCESS.
*/
database: Split _find_parent_id into _split_path and _find_directory_id Some pending commits want the _split_path functionality separate from mapping a directory to a document ID. The split_path function now returns the basename as well as the directory name.
2009-12-21 17:14:52 +01:00
notmuch_status_t
_notmuch_database_find_directory_id (notmuch_database_t *notmuch,
const char *path,
lib: Perform the same transformation to _notmuch_database_find_directory_id Now _notmuch_database_find_directory_id takes a flags argument, which it passes through to _notmuch_directory_create and can indicate if the directory does not exist. Again, callers have been updated, but retain their original behavior.
2012-05-18 06:13:35 +02:00
notmuch_find_flags_t flags,
database: Split _find_parent_id into _split_path and _find_directory_id Some pending commits want the _split_path functionality separate from mapping a directory to a document ID. The split_path function now returns the basename as well as the directory name.
2009-12-21 17:14:52 +01:00
unsigned int *directory_id)
{
lib: Implement new notmuch_directory_t API. This new directory ojbect provides all the infrastructure needed to detect when files or directories are deleted or renamed. There's still code needed on top of this (within "notmuch new") to actually do that detection.
2010-01-05 22:29:23 +01:00
notmuch_directory_t *directory;
notmuch_status_t status;
database: Split _find_parent_id into _split_path and _find_directory_id Some pending commits want the _split_path functionality separate from mapping a directory to a document ID. The split_path function now returns the basename as well as the directory name.
2009-12-21 17:14:52 +01:00
if (path == NULL) {
*directory_id = 0;
return NOTMUCH_STATUS_SUCCESS;
}
lib: rename _n_d_create to _n_d_find_or_create The error message and name were confusing when called in some "read only" context.
2020-07-19 12:36:47 +02:00
directory = _notmuch_directory_find_or_create (notmuch, path, flags, &status);
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
if (status || ! directory) {
lib: Implement new notmuch_directory_t API. This new directory ojbect provides all the infrastructure needed to detect when files or directories are deleted or renamed. There's still code needed on top of this (within "notmuch new") to actually do that detection.
2010-01-05 22:29:23 +01:00
*directory_id = -1;
return status;
database: Store the parent ID for each directory document. Storing the document ID of the parent of each directory document will allow us to find all child-directory documents for a given directory document. We will need this in order to detect directories that have been removed from the mail store, (though we aren't yet doing this).
2009-12-19 22:20:26 +01:00
}
lib: Implement new notmuch_directory_t API. This new directory ojbect provides all the infrastructure needed to detect when files or directories are deleted or renamed. There's still code needed on top of this (within "notmuch new") to actually do that detection.
2010-01-05 22:29:23 +01:00
*directory_id = _notmuch_directory_get_document_id (directory);
database: Store the parent ID for each directory document. Storing the document ID of the parent of each directory document will allow us to find all child-directory documents for a given directory document. We will need this in order to detect directories that have been removed from the mail store, (though we aren't yet doing this).
2009-12-19 22:20:26 +01:00
lib: Implement new notmuch_directory_t API. This new directory ojbect provides all the infrastructure needed to detect when files or directories are deleted or renamed. There's still code needed on top of this (within "notmuch new") to actually do that detection.
2010-01-05 22:29:23 +01:00
notmuch_directory_destroy (directory);
database: Export _notmuch_database_find_parent_id for internal use. We'll soon have mail documents referring to their parent directory's directory documents, so we'll need access to _find_parent_id in files such as message.cc.
2009-12-20 00:11:55 +01:00
lib: Implement new notmuch_directory_t API. This new directory ojbect provides all the infrastructure needed to detect when files or directories are deleted or renamed. There's still code needed on top of this (within "notmuch new") to actually do that detection.
2010-01-05 22:29:23 +01:00
return NOTMUCH_STATUS_SUCCESS;
database: Store the parent ID for each directory document. Storing the document ID of the parent of each directory document will allow us to find all child-directory documents for a given directory document. We will need this in order to detect directories that have been removed from the mail store, (though we aren't yet doing this).
2009-12-19 22:20:26 +01:00
}
database: Store mail filename as a new 'direntry' term, not as 'data'. Instead of storing the complete message filename in the data portion of a mail document we now store a 'direntry' term that contains the document ID of a directory document and also the basename of the message filename within that directory. This will allow us to easily store multple filenames for a single message, and will also allow us to find mail documents for files that previously existed in a directory but that have since been deleted.
2009-12-21 17:23:26 +01:00
const char *
_notmuch_database_get_directory_path (void *ctx,
notmuch_database_t *notmuch,
unsigned int doc_id)
{
Xapian::Document document;
document = find_document_for_doc_id (notmuch, doc_id);
return talloc_strdup (ctx, document.get_data ().c_str ());
}
database: Abstract _filename_to_direntry from _add_message The code to map a filename to a direntry is something that we're going to want in a future _remove_message function, so put it in a new function _notmuch_database_filename_to_direntry .
2009-12-22 00:09:56 +01:00
/* Given a legal 'filename' for the database, (either relative to
* database path or absolute with initial components identical to
* database path), return a new string (with 'ctx' as the talloc
* owner) suitable for use as a direntry term value.
lib: Clarify internal documentation of _notmuch_database_filename_to_direntry The original wording made it sound like this function was just doing some string manipulation. But this function actually creates new directory documents as a side effect. So make that explicit in its documentation.
2010-01-07 18:31:58 +01:00
*
lib: Perform the same transformation to _notmuch_database_filename_to_direntry Now _notmuch_database_filename_to_direntry takes a flags argument and can indicate if the necessary directory documents do not exist. Again, callers have been updated, but retain their original behavior.
2012-05-18 06:13:36 +02:00
* If (flags & NOTMUCH_FIND_CREATE), the necessary directory documents
* will be created in the database as needed. Otherwise, if the
* necessary directory documents do not exist, this sets
* *direntry to NULL and returns NOTMUCH_STATUS_SUCCESS.
database: Abstract _filename_to_direntry from _add_message The code to map a filename to a direntry is something that we're going to want in a future _remove_message function, so put it in a new function _notmuch_database_filename_to_direntry .
2009-12-22 00:09:56 +01:00
*/
notmuch_status_t
_notmuch_database_filename_to_direntry (void *ctx,
notmuch_database_t *notmuch,
const char *filename,
lib: Perform the same transformation to _notmuch_database_filename_to_direntry Now _notmuch_database_filename_to_direntry takes a flags argument and can indicate if the necessary directory documents do not exist. Again, callers have been updated, but retain their original behavior.
2012-05-18 06:13:36 +02:00
notmuch_find_flags_t flags,
database: Abstract _filename_to_direntry from _add_message The code to map a filename to a direntry is something that we're going to want in a future _remove_message function, so put it in a new function _notmuch_database_filename_to_direntry .
2009-12-22 00:09:56 +01:00
char **direntry)
{
const char *relative, *directory, *basename;
Xapian::docid directory_id;
notmuch_status_t status;
relative = _notmuch_database_relative_path (notmuch, filename);
status = _notmuch_database_split_path (ctx, relative,
&directory, &basename);
if (status)
return status;
lib: Perform the same transformation to _notmuch_database_filename_to_direntry Now _notmuch_database_filename_to_direntry takes a flags argument and can indicate if the necessary directory documents do not exist. Again, callers have been updated, but retain their original behavior.
2012-05-18 06:13:36 +02:00
status = _notmuch_database_find_directory_id (notmuch, directory, flags,
database: Abstract _filename_to_direntry from _add_message The code to map a filename to a direntry is something that we're going to want in a future _remove_message function, so put it in a new function _notmuch_database_filename_to_direntry .
2009-12-22 00:09:56 +01:00
&directory_id);
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
if (status || directory_id == (unsigned int) -1) {
lib: Perform the same transformation to _notmuch_database_filename_to_direntry Now _notmuch_database_filename_to_direntry takes a flags argument and can indicate if the necessary directory documents do not exist. Again, callers have been updated, but retain their original behavior.
2012-05-18 06:13:36 +02:00
*direntry = NULL;
database: Abstract _filename_to_direntry from _add_message The code to map a filename to a direntry is something that we're going to want in a future _remove_message function, so put it in a new function _notmuch_database_filename_to_direntry .
2009-12-22 00:09:56 +01:00
return status;
lib: Perform the same transformation to _notmuch_database_filename_to_direntry Now _notmuch_database_filename_to_direntry takes a flags argument and can indicate if the necessary directory documents do not exist. Again, callers have been updated, but retain their original behavior.
2012-05-18 06:13:36 +02:00
}
database: Abstract _filename_to_direntry from _add_message The code to map a filename to a direntry is something that we're going to want in a future _remove_message function, so put it in a new function _notmuch_database_filename_to_direntry .
2009-12-22 00:09:56 +01:00
*direntry = talloc_asprintf (ctx, "%u:%s", directory_id, basename);
return NOTMUCH_STATUS_SUCCESS;
}
lib: Abstract the extraction of a relative path from set_filename We'll soon be having multiple entry points that accept a filename path, so we want common code for getting a relative path from a potentially absolute path.
2009-12-19 21:32:11 +01:00
/* Given a legal 'path' for the database, return the relative path.
*
fix sum moar typos [comments in source code] Various typo fixes in comments within the source code. Signed-off-by: Pieter Praet <pieter@praet.org> Edited-by: Carl Worth <cworth@cworth.org> Restricted to just source-code comments, (and fixed fix of "descriptios" to "descriptors" rather than "descriptions").
2011-06-20 22:14:21 +02:00
* The return value will be a pointer to the original path contents,
lib: Abstract the extraction of a relative path from set_filename We'll soon be having multiple entry points that accept a filename path, so we want common code for getting a relative path from a potentially absolute path.
2009-12-19 21:32:11 +01:00
* and will be either the original string (if 'path' was relative) or
* a portion of the string (if path was absolute and begins with the
* database path).
*/
const char *
_notmuch_database_relative_path (notmuch_database_t *notmuch,
const char *path)
{
const char *db_path, *relative;
unsigned int db_path_len;
db_path = notmuch_database_get_path (notmuch);
db_path_len = strlen (db_path);
relative = path;
if (*relative == '/') {
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
while (*relative == '/' && *(relative + 1) == '/')
lib: Abstract the extraction of a relative path from set_filename We'll soon be having multiple entry points that accept a filename path, so we want common code for getting a relative path from a potentially absolute path.
2009-12-19 21:32:11 +01:00
relative++;
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
if (strncmp (relative, db_path, db_path_len) == 0) {
lib: Abstract the extraction of a relative path from set_filename We'll soon be having multiple entry points that accept a filename path, so we want common code for getting a relative path from a potentially absolute path.
2009-12-19 21:32:11 +01:00
relative += db_path_len;
while (*relative == '/')
relative++;
}
}
return relative;
}
lib/cli: Make notmuch_database_get_directory return a status code Previously, notmuch_database_get_directory had no way to indicate how it had failed. This changes its prototype to return a status code and set an out-argument to the retrieved directory, like similar functions in the library API. This does *not* change its currently broken behavior of creating directory objects when they don't exist, but it does document it and paves the way for fixing this. Also, it can now check for a read-only database and return NOTMUCH_STATUS_READ_ONLY_DATABASE instead of crashing. In the interest of atomicity, this also updates calls from the CLI so that notmuch still compiles.
2012-05-14 01:36:09 +02:00
notmuch_status_t
lib: Implement new notmuch_directory_t API. This new directory ojbect provides all the infrastructure needed to detect when files or directories are deleted or renamed. There's still code needed on top of this (within "notmuch new") to actually do that detection.
2010-01-05 22:29:23 +01:00
notmuch_database_get_directory (notmuch_database_t *notmuch,
lib/cli: Make notmuch_database_get_directory return a status code Previously, notmuch_database_get_directory had no way to indicate how it had failed. This changes its prototype to return a status code and set an out-argument to the retrieved directory, like similar functions in the library API. This does *not* change its currently broken behavior of creating directory objects when they don't exist, but it does document it and paves the way for fixing this. Also, it can now check for a read-only database and return NOTMUCH_STATUS_READ_ONLY_DATABASE instead of crashing. In the interest of atomicity, this also updates calls from the CLI so that notmuch still compiles.
2012-05-14 01:36:09 +02:00
const char *path,
notmuch_directory_t **directory)
Add notmuch_database_set_timestamp and notmuch_database_get_timestamp These will be very helpful to implement an efficient "notmuch new" command which imports new mail messages that have appeared.
2009-10-23 23:31:01 +02:00
{
lib: Implement new notmuch_directory_t API. This new directory ojbect provides all the infrastructure needed to detect when files or directories are deleted or renamed. There's still code needed on top of this (within "notmuch new") to actually do that detection.
2010-01-05 22:29:23 +01:00
notmuch_status_t status;
Add notmuch_database_set_timestamp and notmuch_database_get_timestamp These will be very helpful to implement an efficient "notmuch new" command which imports new mail messages that have appeared.
2009-10-23 23:31:01 +02:00
lib/cli: Make notmuch_database_get_directory return a status code Previously, notmuch_database_get_directory had no way to indicate how it had failed. This changes its prototype to return a status code and set an out-argument to the retrieved directory, like similar functions in the library API. This does *not* change its currently broken behavior of creating directory objects when they don't exist, but it does document it and paves the way for fixing this. Also, it can now check for a read-only database and return NOTMUCH_STATUS_READ_ONLY_DATABASE instead of crashing. In the interest of atomicity, this also updates calls from the CLI so that notmuch still compiles.
2012-05-14 01:36:09 +02:00
if (directory == NULL)
return NOTMUCH_STATUS_NULL_POINTER;
*directory = NULL;
lib: Audit all notmuch_database call for Xapian exception handling. Our current approach is for top-level entry poitns in the library to have try/catch blocks that catch any Xapian exception and print a message. Add a few missing blocks and fix up the documentation.
2010-04-24 16:22:34 +02:00
try {
lib: rename _n_d_create to _n_d_find_or_create The error message and name were confusing when called in some "read only" context.
2020-07-19 12:36:47 +02:00
*directory = _notmuch_directory_find_or_create (notmuch, path,
NOTMUCH_FIND_LOOKUP, &status);
lib: Audit all notmuch_database call for Xapian exception handling. Our current approach is for top-level entry poitns in the library to have try/catch blocks that catch any Xapian exception and print a message. Add a few missing blocks and fix up the documentation.
2010-04-24 16:22:34 +02:00
} catch (const Xapian::Error &error) {
lib: replace almost all fprintfs in library with _n_d_log This is not supposed to change any functionality from an end user point of view. Note that it will eliminate some output to stderr. The query debugging output is left as is; it doesn't really fit with the current primitive logging model. The remaining "bad" fprintf will need an internal API change.
2014-12-26 17:25:35 +01:00
_notmuch_database_log (notmuch, "A Xapian exception occurred getting directory: %s.\n",
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
error.get_msg ().c_str ());
lib: convert notmuch_bool_t to stdbool internally C99 stdbool turned 18 this year. There really is no reason to use our own, except in the library interface for backward compatibility. Convert the lib internally to stdbool.
2017-10-07 10:44:05 +02:00
notmuch->exception_reported = true;
lib/cli: Make notmuch_database_get_directory return a status code Previously, notmuch_database_get_directory had no way to indicate how it had failed. This changes its prototype to return a status code and set an out-argument to the retrieved directory, like similar functions in the library API. This does *not* change its currently broken behavior of creating directory objects when they don't exist, but it does document it and paves the way for fixing this. Also, it can now check for a read-only database and return NOTMUCH_STATUS_READ_ONLY_DATABASE instead of crashing. In the interest of atomicity, this also updates calls from the CLI so that notmuch still compiles.
2012-05-14 01:36:09 +02:00
status = NOTMUCH_STATUS_XAPIAN_EXCEPTION;
lib: Audit all notmuch_database call for Xapian exception handling. Our current approach is for top-level entry poitns in the library to have try/catch blocks that catch any Xapian exception and print a message. Add a few missing blocks and fix up the documentation.
2010-04-24 16:22:34 +02:00
}
lib/cli: Make notmuch_database_get_directory return a status code Previously, notmuch_database_get_directory had no way to indicate how it had failed. This changes its prototype to return a status code and set an out-argument to the retrieved directory, like similar functions in the library API. This does *not* change its currently broken behavior of creating directory objects when they don't exist, but it does document it and paves the way for fixing this. Also, it can now check for a read-only database and return NOTMUCH_STATUS_READ_ONLY_DATABASE instead of crashing. In the interest of atomicity, this also updates calls from the CLI so that notmuch still compiles.
2012-05-14 01:36:09 +02:00
return status;
Add notmuch_database_set_timestamp and notmuch_database_get_timestamp These will be very helpful to implement an efficient "notmuch new" command which imports new mail messages that have appeared.
2009-10-23 23:31:01 +02:00
}
Avoid database corruption by not adding partially-constructed mail documents. Previously we were using Xapian's add_document to allocate document ID values for notmuch_message_t objects. This had the drawback of adding a partially constructed mail document to the database. If notmuch was subsequently interrupted before fully populating this document, then later runs would be quite confused when seeing the partial documents. There are reports from the wild of people hitting internal errors of the form "Message ... has no thread ID" for example, (which is currently an unrecoverable error). We fix this by manually allocating document IDs without adding documents. With this change, we never call Xapian's add_document method, but only replace_document with either the current document ID of a message or a new one that we have allocated.
2010-06-04 19:16:53 +02:00
/* Allocate a document ID that satisfies the following criteria:
*
* 1. The ID does not exist for any document in the Xapian database
*
* 2. The ID was not previously returned from this function
*
* 3. The ID is the smallest integer satisfying (1) and (2)
*
* This function will trigger an internal error if these constraints
* cannot all be satisfied, (that is, the pool of available document
* IDs has been exhausted).
*/
unsigned int
_notmuch_database_generate_doc_id (notmuch_database_t *notmuch)
{
assert (notmuch->last_doc_id >= notmuch->xapian_db->get_lastdocid ());
notmuch->last_doc_id++;
if (notmuch->last_doc_id == 0)
compact: tidy formatting Notmuch compact code whitespace changes to match devel/STYLE.
2013-11-13 18:02:43 +01:00
INTERNAL_ERROR ("Xapian document IDs are exhausted.\n");
Avoid database corruption by not adding partially-constructed mail documents. Previously we were using Xapian's add_document to allocate document ID values for notmuch_message_t objects. This had the drawback of adding a partially constructed mail document to the database. If notmuch was subsequently interrupted before fully populating this document, then later runs would be quite confused when seeing the partial documents. There are reports from the wild of people hitting internal errors of the form "Message ... has no thread ID" for example, (which is currently an unrecoverable error). We fix this by manually allocating document IDs without adding documents. With this change, we never call Xapian's add_document method, but only replace_document with either the current document ID of a message or a new one that we have allocated.
2010-06-04 19:16:53 +02:00
return notmuch->last_doc_id;
}
database: Add new, public notmuch_database_remove_message This will allow applications to support the removal of messages, (such as when a file is deleted from the mail store). No removal support is provided yet in commands such as "notmuch new".
2009-12-22 00:14:32 +01:00
notmuch_status_t
notmuch_database_remove_message (notmuch_database_t *notmuch,
const char *filename)
lib: Add an API to find a message by filename. notmuch_database_find_message_by_filename is mostly stolen from notmuch_database_remove_message, so this patch also vastly simplfies the latter using the former. This API is also useful in its own right and will be used in a later patch for eager maildir flag synchronization.
2011-06-11 06:19:31 +02:00
{
lib: make find_message{,by_filename) report errors Previously, the functions notmuch_database_find_message() and notmuch_database_find_message_by_filename() functions did not properly report error condition to the library user. For more information, read the thread on the notmuch mailing list starting with my mail "id:871uv2unfd.fsf@gmail.com" Make these functions accept a pointer to 'notmuch_message_t' as argument and return notmuch_status_t which may be used to check for any error condition. restore: Modify for the new notmuch_database_find_message() new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 06:55:29 +02:00
notmuch_status_t status;
notmuch_message_t *message;
lib: Add an API to find a message by filename. notmuch_database_find_message_by_filename is mostly stolen from notmuch_database_remove_message, so this patch also vastly simplfies the latter using the former. This API is also useful in its own right and will be used in a later patch for eager maildir flag synchronization.
2011-06-11 06:19:31 +02:00
lib: make find_message{,by_filename) report errors Previously, the functions notmuch_database_find_message() and notmuch_database_find_message_by_filename() functions did not properly report error condition to the library user. For more information, read the thread on the notmuch mailing list starting with my mail "id:871uv2unfd.fsf@gmail.com" Make these functions accept a pointer to 'notmuch_message_t' as argument and return notmuch_status_t which may be used to check for any error condition. restore: Modify for the new notmuch_database_find_message() new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 06:55:29 +02:00
status = notmuch_database_find_message_by_filename (notmuch, filename,
&message);
if (status == NOTMUCH_STATUS_SUCCESS && message) {
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
status = _notmuch_message_remove_filename (message, filename);
if (status == NOTMUCH_STATUS_SUCCESS)
_notmuch_message_delete (message);
else if (status == NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID)
_notmuch_message_sync (message);
lib: destroy message object after message removal notmuch_database_remove_message() must call notmuch_message_destroy() once it is done handling message removal.
2011-10-03 22:27:32 +02:00
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
notmuch_message_destroy (message);
lib: Add an API to find a message by filename. notmuch_database_find_message_by_filename is mostly stolen from notmuch_database_remove_message, so this patch also vastly simplfies the latter using the former. This API is also useful in its own right and will be used in a later patch for eager maildir flag synchronization.
2011-06-11 06:19:31 +02:00
}
return status;
}
lib: make find_message{,by_filename) report errors Previously, the functions notmuch_database_find_message() and notmuch_database_find_message_by_filename() functions did not properly report error condition to the library user. For more information, read the thread on the notmuch mailing list starting with my mail "id:871uv2unfd.fsf@gmail.com" Make these functions accept a pointer to 'notmuch_message_t' as argument and return notmuch_status_t which may be used to check for any error condition. restore: Modify for the new notmuch_database_find_message() new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 06:55:29 +02:00
notmuch_status_t
lib: Add an API to find a message by filename. notmuch_database_find_message_by_filename is mostly stolen from notmuch_database_remove_message, so this patch also vastly simplfies the latter using the former. This API is also useful in its own right and will be used in a later patch for eager maildir flag synchronization.
2011-06-11 06:19:31 +02:00
notmuch_database_find_message_by_filename (notmuch_database_t *notmuch,
lib: make find_message{,by_filename) report errors Previously, the functions notmuch_database_find_message() and notmuch_database_find_message_by_filename() functions did not properly report error condition to the library user. For more information, read the thread on the notmuch mailing list starting with my mail "id:871uv2unfd.fsf@gmail.com" Make these functions accept a pointer to 'notmuch_message_t' as argument and return notmuch_status_t which may be used to check for any error condition. restore: Modify for the new notmuch_database_find_message() new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 06:55:29 +02:00
const char *filename,
notmuch_message_t **message_ret)
database: Add new, public notmuch_database_remove_message This will allow applications to support the removal of messages, (such as when a file is deleted from the mail store). No removal support is provided yet in commands such as "notmuch new".
2009-12-22 00:14:32 +01:00
{
lib: Audit all notmuch_database call for Xapian exception handling. Our current approach is for top-level entry poitns in the library to have try/catch blocks that catch any Xapian exception and print a message. Add a few missing blocks and fix up the documentation.
2010-04-24 16:22:34 +02:00
void *local;
lib: Implement new notmuch_directory_t API. This new directory ojbect provides all the infrastructure needed to detect when files or directories are deleted or renamed. There's still code needed on top of this (within "notmuch new") to actually do that detection.
2010-01-05 22:29:23 +01:00
const char *prefix = _find_prefix ("file-direntry");
database: Add new, public notmuch_database_remove_message This will allow applications to support the removal of messages, (such as when a file is deleted from the mail store). No removal support is provided yet in commands such as "notmuch new".
2009-12-22 00:14:32 +01:00
char *direntry, *term;
Xapian::PostingIterator i, end;
notmuch_status_t status;
lib: make find_message{,by_filename) report errors Previously, the functions notmuch_database_find_message() and notmuch_database_find_message_by_filename() functions did not properly report error condition to the library user. For more information, read the thread on the notmuch mailing list starting with my mail "id:871uv2unfd.fsf@gmail.com" Make these functions accept a pointer to 'notmuch_message_t' as argument and return notmuch_status_t which may be used to check for any error condition. restore: Modify for the new notmuch_database_find_message() new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 06:55:29 +02:00
if (message_ret == NULL)
return NOTMUCH_STATUS_NULL_POINTER;
lib: Return an error from operations that require an upgrade Previously, there was no protection against a caller invoking an operation on an old database version that would effectively corrupt the database by treating it like a newer version. According to notmuch.h, any caller that opens the database in read/write mode is supposed to check if the database needs upgrading and perform an upgrade if it does. This would protect against this, but nobody (even the CLI) actually does this. However, with features, it's easy to protect against incompatible operations on a fine-grained basis. This lightweight change allows callers to safely operate on old database versions, while preventing specific operations that would corrupt the database with an informative error message.
2014-08-25 19:26:08 +02:00
if (! (notmuch->features & NOTMUCH_FEATURE_FILE_TERMS))
return NOTMUCH_STATUS_UPGRADE_REQUIRED;
Fix error reporting in notmuch_database_find_message_by_filename Formerly it was possible for *message_ret to be left uninitialized. The documentation however clearly states that "[o]n any failure or when the message is not found, this function initializes '*message' to NULL". Signed-off-by: Justus Winter <4winter@informatik.uni-hamburg.de>
2012-03-17 17:41:27 +01:00
/* return NULL on any failure */
*message_ret = NULL;
lib: Audit all notmuch_database call for Xapian exception handling. Our current approach is for top-level entry poitns in the library to have try/catch blocks that catch any Xapian exception and print a message. Add a few missing blocks and fix up the documentation.
2010-04-24 16:22:34 +02:00
local = talloc_new (notmuch);
try {
lib: Perform the same transformation to _notmuch_database_filename_to_direntry Now _notmuch_database_filename_to_direntry takes a flags argument and can indicate if the necessary directory documents do not exist. Again, callers have been updated, but retain their original behavior.
2012-05-18 06:13:36 +02:00
status = _notmuch_database_filename_to_direntry (
lib: Make notmuch_database_find_message_by_filename not crash on read-only databases Previously, _notmuch_database_filename_to_direntry would abort with an internal error when called on a read-only database. Now that creating the directory document is optional, notmuch_database_find_message_by_filename can disable directory document creation (as it should) and, as a result, not abort on read-only databases.
2012-05-18 06:13:40 +02:00
local, notmuch, filename, NOTMUCH_FIND_LOOKUP, &direntry);
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
if (status || ! direntry)
lib: make find_message{,by_filename) report errors Previously, the functions notmuch_database_find_message() and notmuch_database_find_message_by_filename() functions did not properly report error condition to the library user. For more information, read the thread on the notmuch mailing list starting with my mail "id:871uv2unfd.fsf@gmail.com" Make these functions accept a pointer to 'notmuch_message_t' as argument and return notmuch_status_t which may be used to check for any error condition. restore: Modify for the new notmuch_database_find_message() new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 06:55:29 +02:00
goto DONE;
database: Add new, public notmuch_database_remove_message This will allow applications to support the removal of messages, (such as when a file is deleted from the mail store). No removal support is provided yet in commands such as "notmuch new".
2009-12-22 00:14:32 +01:00
Tighten up a memory allocation. Using the local talloc context ensures that the memory we are using here will be freed shortly, (rather than hanging on for a long time with the notmuch database object).
2011-01-15 23:01:43 +01:00
term = talloc_asprintf (local, "%s%s", prefix, direntry);
database: Add new, public notmuch_database_remove_message This will allow applications to support the removal of messages, (such as when a file is deleted from the mail store). No removal support is provided yet in commands such as "notmuch new".
2009-12-22 00:14:32 +01:00
lib: Audit all notmuch_database call for Xapian exception handling. Our current approach is for top-level entry poitns in the library to have try/catch blocks that catch any Xapian exception and print a message. Add a few missing blocks and fix up the documentation.
2010-04-24 16:22:34 +02:00
find_doc_ids_for_term (notmuch, term, &i, &end);
database: Add new, public notmuch_database_remove_message This will allow applications to support the removal of messages, (such as when a file is deleted from the mail store). No removal support is provided yet in commands such as "notmuch new".
2009-12-22 00:14:32 +01:00
lib: Add an API to find a message by filename. notmuch_database_find_message_by_filename is mostly stolen from notmuch_database_remove_message, so this patch also vastly simplfies the latter using the former. This API is also useful in its own right and will be used in a later patch for eager maildir flag synchronization.
2011-06-11 06:19:31 +02:00
if (i != end) {
Add support for folder-based searching. A new "folder:" prefix in the query string can now be used to match the directories in which mail files are stored. The addition of this feature causes the recently added search-by-folder tests to now pass.
2011-01-15 23:09:04 +01:00
notmuch_private_status_t private_status;
database: Add new, public notmuch_database_remove_message This will allow applications to support the removal of messages, (such as when a file is deleted from the mail store). No removal support is provided yet in commands such as "notmuch new".
2009-12-22 00:14:32 +01:00
lib: make find_message{,by_filename) report errors Previously, the functions notmuch_database_find_message() and notmuch_database_find_message_by_filename() functions did not properly report error condition to the library user. For more information, read the thread on the notmuch mailing list starting with my mail "id:871uv2unfd.fsf@gmail.com" Make these functions accept a pointer to 'notmuch_message_t' as argument and return notmuch_status_t which may be used to check for any error condition. restore: Modify for the new notmuch_database_find_message() new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 06:55:29 +02:00
*message_ret = _notmuch_message_create (notmuch, notmuch, *i,
&private_status);
if (*message_ret == NULL)
status = NOTMUCH_STATUS_OUT_OF_MEMORY;
database: Add new, public notmuch_database_remove_message This will allow applications to support the removal of messages, (such as when a file is deleted from the mail store). No removal support is provided yet in commands such as "notmuch new".
2009-12-22 00:14:32 +01:00
}
lib: Audit all notmuch_database call for Xapian exception handling. Our current approach is for top-level entry poitns in the library to have try/catch blocks that catch any Xapian exception and print a message. Add a few missing blocks and fix up the documentation.
2010-04-24 16:22:34 +02:00
} catch (const Xapian::Error &error) {
lib: replace almost all fprintfs in library with _n_d_log This is not supposed to change any functionality from an end user point of view. Note that it will eliminate some output to stderr. The query debugging output is left as is; it doesn't really fit with the current primitive logging model. The remaining "bad" fprintf will need an internal API change.
2014-12-26 17:25:35 +01:00
_notmuch_database_log (notmuch, "Error: A Xapian exception occurred finding message by filename: %s\n",
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
error.get_msg ().c_str ());
lib: convert notmuch_bool_t to stdbool internally C99 stdbool turned 18 this year. There really is no reason to use our own, except in the library interface for backward compatibility. Convert the lib internally to stdbool.
2017-10-07 10:44:05 +02:00
notmuch->exception_reported = true;
lib: make find_message{,by_filename) report errors Previously, the functions notmuch_database_find_message() and notmuch_database_find_message_by_filename() functions did not properly report error condition to the library user. For more information, read the thread on the notmuch mailing list starting with my mail "id:871uv2unfd.fsf@gmail.com" Make these functions accept a pointer to 'notmuch_message_t' as argument and return notmuch_status_t which may be used to check for any error condition. restore: Modify for the new notmuch_database_find_message() new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 06:55:29 +02:00
status = NOTMUCH_STATUS_XAPIAN_EXCEPTION;
database: Add new, public notmuch_database_remove_message This will allow applications to support the removal of messages, (such as when a file is deleted from the mail store). No removal support is provided yet in commands such as "notmuch new".
2009-12-22 00:14:32 +01:00
}
lib: make find_message{,by_filename) report errors Previously, the functions notmuch_database_find_message() and notmuch_database_find_message_by_filename() functions did not properly report error condition to the library user. For more information, read the thread on the notmuch mailing list starting with my mail "id:871uv2unfd.fsf@gmail.com" Make these functions accept a pointer to 'notmuch_message_t' as argument and return notmuch_status_t which may be used to check for any error condition. restore: Modify for the new notmuch_database_find_message() new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 06:55:29 +02:00
DONE:
database: Add new, public notmuch_database_remove_message This will allow applications to support the removal of messages, (such as when a file is deleted from the mail store). No removal support is provided yet in commands such as "notmuch new".
2009-12-22 00:14:32 +01:00
talloc_free (local);
lib: make find_message{,by_filename) report errors Previously, the functions notmuch_database_find_message() and notmuch_database_find_message_by_filename() functions did not properly report error condition to the library user. For more information, read the thread on the notmuch mailing list starting with my mail "id:871uv2unfd.fsf@gmail.com" Make these functions accept a pointer to 'notmuch_message_t' as argument and return notmuch_status_t which may be used to check for any error condition. restore: Modify for the new notmuch_database_find_message() new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 06:55:29 +02:00
if (status && *message_ret) {
notmuch_message_destroy (*message_ret);
*message_ret = NULL;
}
return status;
database: Add new, public notmuch_database_remove_message This will allow applications to support the removal of messages, (such as when a file is deleted from the mail store). No removal support is provided yet in commands such as "notmuch new".
2009-12-22 00:14:32 +01:00
}
Add a generic function to get a list of terms with some prefix. Replace _notmuch_convert_tags with this and simplify _create_filenames_for_terms_with_prefix. This will also come in handy shortly to get the message file name list.
2010-12-09 06:32:35 +01:00
notmuch_string_list_t *
_notmuch_database_get_terms_with_prefix (void *ctx, Xapian::TermIterator &i,
Xapian::TermIterator &end,
const char *prefix)
notmuch: New function to retrieve all tags from the database. This patch adds a new function called notmuch_database_get_all_tags which can be used to obtain a list of all tags from the database (in other words, the list contains all tags from all messages). The function produces an alphabetically sorted list. To add support for the new function, we rip the guts off of notmuch_message_get_tags and put them in a new generic function called _notmuch_convert_tags. The generic function takes a Xapian::TermIterator as argument and uses the iterator to find tags. This makes the function usable with different Xapian objects. Function notmuch_message_get_tags is then reimplemented to call the generic function with message->doc.termlist_begin() as argument. Similarly, we implement notmuch_message_database_get_all_tags, the function calls the generic function with db->xapian_db->allterms_begin() as argument. Finally, notmuch_database_get_all_tags is exported through lib/notmuch.h Signed-off-by: Jan Janak <jan@ryngle.com>
2009-11-23 01:10:54 +01:00
{
Add a generic function to get a list of terms with some prefix. Replace _notmuch_convert_tags with this and simplify _create_filenames_for_terms_with_prefix. This will also come in handy shortly to get the message file name list.
2010-12-09 06:32:35 +01:00
int prefix_len = strlen (prefix);
Implement an internal generic string list and use it. This replaces the guts of the filename list and tag list, making those interfaces simple iterators over the generic string list. The directory, message filename, and tags-related code now build generic string lists and then wraps them in specific iterators. The real wins come in later patches, when we use these for even more generic functionality. As a nice side-effect, this also eliminates the annoying dependency on GList in the tag list.
2010-12-09 01:26:05 +01:00
notmuch_string_list_t *list;
notmuch: New function to retrieve all tags from the database. This patch adds a new function called notmuch_database_get_all_tags which can be used to obtain a list of all tags from the database (in other words, the list contains all tags from all messages). The function produces an alphabetically sorted list. To add support for the new function, we rip the guts off of notmuch_message_get_tags and put them in a new generic function called _notmuch_convert_tags. The generic function takes a Xapian::TermIterator as argument and uses the iterator to find tags. This makes the function usable with different Xapian objects. Function notmuch_message_get_tags is then reimplemented to call the generic function with message->doc.termlist_begin() as argument. Similarly, we implement notmuch_message_database_get_all_tags, the function calls the generic function with db->xapian_db->allterms_begin() as argument. Finally, notmuch_database_get_all_tags is exported through lib/notmuch.h Signed-off-by: Jan Janak <jan@ryngle.com>
2009-11-23 01:10:54 +01:00
Implement an internal generic string list and use it. This replaces the guts of the filename list and tag list, making those interfaces simple iterators over the generic string list. The directory, message filename, and tags-related code now build generic string lists and then wraps them in specific iterators. The real wins come in later patches, when we use these for even more generic functionality. As a nice side-effect, this also eliminates the annoying dependency on GList in the tag list.
2010-12-09 01:26:05 +01:00
list = _notmuch_string_list_create (ctx);
if (unlikely (list == NULL))
notmuch: New function to retrieve all tags from the database. This patch adds a new function called notmuch_database_get_all_tags which can be used to obtain a list of all tags from the database (in other words, the list contains all tags from all messages). The function produces an alphabetically sorted list. To add support for the new function, we rip the guts off of notmuch_message_get_tags and put them in a new generic function called _notmuch_convert_tags. The generic function takes a Xapian::TermIterator as argument and uses the iterator to find tags. This makes the function usable with different Xapian objects. Function notmuch_message_get_tags is then reimplemented to call the generic function with message->doc.termlist_begin() as argument. Similarly, we implement notmuch_message_database_get_all_tags, the function calls the generic function with db->xapian_db->allterms_begin() as argument. Finally, notmuch_database_get_all_tags is exported through lib/notmuch.h Signed-off-by: Jan Janak <jan@ryngle.com>
2009-11-23 01:10:54 +01:00
return NULL;
Add a generic function to get a list of terms with some prefix. Replace _notmuch_convert_tags with this and simplify _create_filenames_for_terms_with_prefix. This will also come in handy shortly to get the message file name list.
2010-12-09 06:32:35 +01:00
for (i.skip_to (prefix); i != end; i++) {
/* Terminate loop at first term without desired prefix. */
if (strncmp ((*i).c_str (), prefix, prefix_len))
notmuch: New function to retrieve all tags from the database. This patch adds a new function called notmuch_database_get_all_tags which can be used to obtain a list of all tags from the database (in other words, the list contains all tags from all messages). The function produces an alphabetically sorted list. To add support for the new function, we rip the guts off of notmuch_message_get_tags and put them in a new generic function called _notmuch_convert_tags. The generic function takes a Xapian::TermIterator as argument and uses the iterator to find tags. This makes the function usable with different Xapian objects. Function notmuch_message_get_tags is then reimplemented to call the generic function with message->doc.termlist_begin() as argument. Similarly, we implement notmuch_message_database_get_all_tags, the function calls the generic function with db->xapian_db->allterms_begin() as argument. Finally, notmuch_database_get_all_tags is exported through lib/notmuch.h Signed-off-by: Jan Janak <jan@ryngle.com>
2009-11-23 01:10:54 +01:00
break;
Add a generic function to get a list of terms with some prefix. Replace _notmuch_convert_tags with this and simplify _create_filenames_for_terms_with_prefix. This will also come in handy shortly to get the message file name list.
2010-12-09 06:32:35 +01:00
_notmuch_string_list_append (list, (*i).c_str () + prefix_len);
notmuch: New function to retrieve all tags from the database. This patch adds a new function called notmuch_database_get_all_tags which can be used to obtain a list of all tags from the database (in other words, the list contains all tags from all messages). The function produces an alphabetically sorted list. To add support for the new function, we rip the guts off of notmuch_message_get_tags and put them in a new generic function called _notmuch_convert_tags. The generic function takes a Xapian::TermIterator as argument and uses the iterator to find tags. This makes the function usable with different Xapian objects. Function notmuch_message_get_tags is then reimplemented to call the generic function with message->doc.termlist_begin() as argument. Similarly, we implement notmuch_message_database_get_all_tags, the function calls the generic function with db->xapian_db->allterms_begin() as argument. Finally, notmuch_database_get_all_tags is exported through lib/notmuch.h Signed-off-by: Jan Janak <jan@ryngle.com>
2009-11-23 01:10:54 +01:00
}
Add a generic function to get a list of terms with some prefix. Replace _notmuch_convert_tags with this and simplify _create_filenames_for_terms_with_prefix. This will also come in handy shortly to get the message file name list.
2010-12-09 06:32:35 +01:00
return list;
notmuch: New function to retrieve all tags from the database. This patch adds a new function called notmuch_database_get_all_tags which can be used to obtain a list of all tags from the database (in other words, the list contains all tags from all messages). The function produces an alphabetically sorted list. To add support for the new function, we rip the guts off of notmuch_message_get_tags and put them in a new generic function called _notmuch_convert_tags. The generic function takes a Xapian::TermIterator as argument and uses the iterator to find tags. This makes the function usable with different Xapian objects. Function notmuch_message_get_tags is then reimplemented to call the generic function with message->doc.termlist_begin() as argument. Similarly, we implement notmuch_message_database_get_all_tags, the function calls the generic function with db->xapian_db->allterms_begin() as argument. Finally, notmuch_database_get_all_tags is exported through lib/notmuch.h Signed-off-by: Jan Janak <jan@ryngle.com>
2009-11-23 01:10:54 +01:00
}
notmuch_tags_t *
notmuch_database_get_all_tags (notmuch_database_t *db)
{
Xapian::TermIterator i, end;
Add a generic function to get a list of terms with some prefix. Replace _notmuch_convert_tags with this and simplify _create_filenames_for_terms_with_prefix. This will also come in handy shortly to get the message file name list.
2010-12-09 06:32:35 +01:00
notmuch_string_list_t *tags;
lib: Audit all notmuch_database call for Xapian exception handling. Our current approach is for top-level entry poitns in the library to have try/catch blocks that catch any Xapian exception and print a message. Add a few missing blocks and fix up the documentation.
2010-04-24 16:22:34 +02:00
try {
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
i = db->xapian_db->allterms_begin ();
end = db->xapian_db->allterms_end ();
Add a generic function to get a list of terms with some prefix. Replace _notmuch_convert_tags with this and simplify _create_filenames_for_terms_with_prefix. This will also come in handy shortly to get the message file name list.
2010-12-09 06:32:35 +01:00
tags = _notmuch_database_get_terms_with_prefix (db, i, end,
_find_prefix ("tag"));
_notmuch_string_list_sort (tags);
return _notmuch_tags_create (db, tags);
lib: Audit all notmuch_database call for Xapian exception handling. Our current approach is for top-level entry poitns in the library to have try/catch blocks that catch any Xapian exception and print a message. Add a few missing blocks and fix up the documentation.
2010-04-24 16:22:34 +02:00
} catch (const Xapian::Error &error) {
lib: replace almost all fprintfs in library with _n_d_log This is not supposed to change any functionality from an end user point of view. Note that it will eliminate some output to stderr. The query debugging output is left as is; it doesn't really fit with the current primitive logging model. The remaining "bad" fprintf will need an internal API change.
2014-12-26 17:25:35 +01:00
_notmuch_database_log (db, "A Xapian exception occurred getting tags: %s.\n",
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
error.get_msg ().c_str ());
lib: convert notmuch_bool_t to stdbool internally C99 stdbool turned 18 this year. There really is no reason to use our own, except in the library interface for backward compatibility. Convert the lib internally to stdbool.
2017-10-07 10:44:05 +02:00
db->exception_reported = true;
lib: Audit all notmuch_database call for Xapian exception handling. Our current approach is for top-level entry poitns in the library to have try/catch blocks that catch any Xapian exception and print a message. Add a few missing blocks and fix up the documentation.
2010-04-24 16:22:34 +02:00
return NULL;
}
notmuch: New function to retrieve all tags from the database. This patch adds a new function called notmuch_database_get_all_tags which can be used to obtain a list of all tags from the database (in other words, the list contains all tags from all messages). The function produces an alphabetically sorted list. To add support for the new function, we rip the guts off of notmuch_message_get_tags and put them in a new generic function called _notmuch_convert_tags. The generic function takes a Xapian::TermIterator as argument and uses the iterator to find tags. This makes the function usable with different Xapian objects. Function notmuch_message_get_tags is then reimplemented to call the generic function with message->doc.termlist_begin() as argument. Similarly, we implement notmuch_message_database_get_all_tags, the function calls the generic function with db->xapian_db->allterms_begin() as argument. Finally, notmuch_database_get_all_tags is exported through lib/notmuch.h Signed-off-by: Jan Janak <jan@ryngle.com>
2009-11-23 01:10:54 +01:00
}
lib: add a log function with output to a string in notmuch_database_t In principle in the future this could do something fancier than asprintf.
2014-12-26 09:01:01 +01:00
const char *
lib: constify argument to notmuch_database_status_string We don't modify the database struct, so no harm in committing to that.
2015-06-07 17:02:00 +02:00
notmuch_database_status_string (const notmuch_database_t *notmuch)
lib: add a log function with output to a string in notmuch_database_t In principle in the future this could do something fancier than asprintf.
2014-12-26 09:01:01 +01:00
{
return notmuch->status_string;
}
Reference in a new issue Copy permalink