debian/notmuch
1
0
Fork 0
mirror of https://git.notmuchmail.org/git/notmuch synced 2024-11-22 10:58:10 +01:00
Code Issues Projects Releases Packages Wiki Activity Actions
notmuch/lib/database-private.h

256 lines
8.6 KiB
C
Raw Normal View History

Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
/* database-private.h - For peeking into the internals of notmuch_database_t
*
* Copyright © 2009 Carl Worth
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
Use https instead of http where possible Many of the external links found in the notmuch source can be resolved using https instead of http. This changeset addresses as many as i could find, without touching the e-mail corpus or expected outputs found in tests.
2016-06-02 18:26:14 +02:00
* along with this program. If not, see https://www.gnu.org/licenses/ .
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
*
* Author: Carl Worth <cworth@cworth.org>
*/
#ifndef NOTMUCH_DATABASE_PRIVATE_H
#define NOTMUCH_DATABASE_PRIVATE_H
Fix printf for when uint64_t != unsigned long long int Thanks to Michal Sojka <sojkam1@fel.cvut.cz> for pointing out the correct fix, which I verified in the freely-available WG14/N1124 draft (from the C99 working group) which is available here: http://www.open-std.org/JTC1/SC22/wg14/www/docs/n1124.pdf
2010-02-09 20:09:30 +01:00
/* According to WG14/N1124, a C++ implementation won't provide us a
* macro like PRIx64 (which gives a printf format string for
* formatting a uint64_t as hexadecimal) unless we define
* __STDC_FORMAT_MACROS before including inttypes.h. That's annoying,
* but there it is.
*/
#define __STDC_FORMAT_MACROS
#include <inttypes.h>
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
#include "notmuch-private.h"
lib: optionally silence Xapian deprecation warnings This is not ideal, but the new API is not available in Xapian 1.2.x, and it seems to soon to depend on Xapian >= 1.4
2016-10-05 13:34:34 +02:00
#ifdef SILENCE_XAPIAN_DEPRECATION_WARNINGS
#define XAPIAN_DEPRECATED(D) D
#endif
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
#include <xapian.h>
lib: Database version 3: Introduce fine-grained "features" Previously, our database schema was versioned by a single number. Each database schema change had to occur "atomically" in Notmuch's development history: before some commit, Notmuch used version N, after that commit, it used version N+1. Hence, each new schema version could introduce only one change, the task of developing a schema change fell on a single person, and it all had to happen and be perfect in a single commit series. This made introducing a new schema version hard. We've seen only two schema changes in the history of Notmuch. This commit introduces database schema version 3; hopefully the last schema version we'll need for a while. With this version, we switch from a single version number to "features": a set of named, independent aspects of the database schema. Features should make backwards compatibility easier. For many things, it should be easy to support databases both with and without a feature, which will allow us to make upgrades optional and will enable "unstable" features that can be developed and tested over time. Features also make forwards compatibility easier. The features recorded in a database include "compatibility flags," which can indicate to an older version of Notmuch when it must support a given feature to open the database for read or for write. This lets us replace the old vague "I don't recognize this version, so something might go wrong, but I promise to try my best" warnings upon opening a database with an unknown version with precise errors. If a database is safe to open for read/write despite unknown features, an older version will know that and issue no message at all. If the database is not safe to open for read/write because of unknown features, an older version will know that, too, and can tell the user exactly which required features it lacks support for.
2014-08-25 19:26:00 +02:00
/* Bit masks for _notmuch_database::features. Features are named,
* independent aspects of the database schema.
*
* A database stores the set of features that it "uses" (implicitly
* before database version 3 and explicitly as of version 3).
*
* A given library version will "recognize" a particular set of
* features; if a database uses a feature that the library does not
* recognize, the library will refuse to open it. It is assumed the
* set of recognized features grows monotonically over time. A
* library version will "implement" some subset of the recognized
* features: some operations may require that the database use (or not
* use) some feature, while other operations may support both
* databases that use and that don't use some feature.
*
* On disk, the database stores string names for these features (see
* the feature_names array). These enum bit values are never
* persisted to disk and may change freely.
*/
enum _notmuch_features {
/* If set, file names are stored in "file-direntry" terms. If
* unset, file names are stored in document data.
*
* Introduced: version 1. */
NOTMUCH_FEATURE_FILE_TERMS = 1 << 0,
/* If set, directory timestamps are stored in documents with
* XDIRECTORY terms and relative paths. If unset, directory
* timestamps are stored in documents with XTIMESTAMP terms and
* absolute paths.
*
* Introduced: version 1. */
NOTMUCH_FEATURE_DIRECTORY_DOCS = 1 << 1,
/* If set, the from, subject, and message-id headers are stored in
* message document values. If unset, message documents *may*
* have these values, but if the value is empty, it must be
* retrieved from the message file.
*
* Introduced: optional in version 1, required as of version 3.
*/
NOTMUCH_FEATURE_FROM_SUBJECT_ID_VALUES = 1 << 2,
/* If set, folder terms are boolean and path terms exist. If
* unset, folder terms are probabilistic and stemmed and path
* terms do not exist.
*
* Introduced: version 2. */
NOTMUCH_FEATURE_BOOL_FOLDER = 1 << 3,
lib: Add a ghost messages database feature This will be implemented over the next several patches. The feature is not yet "enabled" (this does not add it to NOTMUCH_FEATURES_CURRENT).
2014-10-23 14:30:33 +02:00
/* If set, missing messages are stored in ghost mail documents.
* If unset, thread IDs of ghost messages are stored as database
* metadata instead of in ghost documents.
*
* Introduced: version 3. */
NOTMUCH_FEATURE_GHOSTS = 1 << 4,
Add the NOTMUCH_FEATURE_INDEXED_MIMETYPES database feature This feature will exist in all newly created databases, but there is no upgrade provided for it. If this flag exists, it indicates that the database was created after the indexed MIME-types feature was added.
2015-01-23 00:43:37 +01:00
/* If set, then the database was created after the introduction of
* indexed mime types. If unset, then the database may contain a
* mixture of messages with indexed and non-indexed mime types.
*
* Introduced: version 3. */
NOTMUCH_FEATURE_INDEXED_MIMETYPES = 1 << 5,
lib: Add per-message last modification tracking This adds a new document value that stores the revision of the last modification to message metadata, where the revision number increases monotonically with each database commit. An alternative would be to store the wall-clock time of the last modification of each message. In principle this is simpler and has the advantage that any process can determine the current timestamp without support from libnotmuch. However, even assuming a computer's clock never goes backward and ignoring clock skew in networked environments, this has a fatal flaw. Xapian uses (optimistic) snapshot isolation, which means reads can be concurrent with writes. Given this, consider the following time line with a write and two read transactions: write |-X-A--------------| read 1 |---B---| read 2 |---| The write transaction modifies message X and records the wall-clock time of the modification at A. The writer hangs around for a while and later commits its change. Read 1 is concurrent with the write, so it doesn't see the change to X. It does some query and records the wall-clock time of its results at B. Transaction read 2 later starts after the write commits and queries for changes since wall-clock time B (say the reads are performing an incremental backup). Even though read 1 could not see the change to X, read 2 is told (correctly) that X has not changed since B, the time of the last read. In fact, X changed before wall-clock time A, but the change was not visible until *after* wall-clock time B, so read 2 misses the change to X. This is tricky to solve in full-blown snapshot isolation, but because Xapian serializes writes, we can use a simple, monotonically increasing database revision number. Furthermore, maintaining this revision number requires no more IO than a wall-clock time solution because Xapian already maintains statistics on the upper (and lower) bound of each value stream.
2014-10-13 08:20:01 +02:00
/* If set, messages store the revision number of the last
* modification in NOTMUCH_VALUE_LAST_MOD.
*
* Introduced: version 3. */
NOTMUCH_FEATURE_LAST_MOD = 1 << 6,
lib: Database version 3: Introduce fine-grained "features" Previously, our database schema was versioned by a single number. Each database schema change had to occur "atomically" in Notmuch's development history: before some commit, Notmuch used version N, after that commit, it used version N+1. Hence, each new schema version could introduce only one change, the task of developing a schema change fell on a single person, and it all had to happen and be perfect in a single commit series. This made introducing a new schema version hard. We've seen only two schema changes in the history of Notmuch. This commit introduces database schema version 3; hopefully the last schema version we'll need for a while. With this version, we switch from a single version number to "features": a set of named, independent aspects of the database schema. Features should make backwards compatibility easier. For many things, it should be easy to support databases both with and without a feature, which will allow us to make upgrades optional and will enable "unstable" features that can be developed and tested over time. Features also make forwards compatibility easier. The features recorded in a database include "compatibility flags," which can indicate to an older version of Notmuch when it must support a given feature to open the database for read or for write. This lets us replace the old vague "I don't recognize this version, so something might go wrong, but I promise to try my best" warnings upon opening a database with an unknown version with precise errors. If a database is safe to open for read/write despite unknown features, an older version will know that and issue no message at all. If the database is not safe to open for read/write because of unknown features, an older version will know that, too, and can tell the user exactly which required features it lacks support for.
2014-08-25 19:26:00 +02:00
};
/* In C++, a named enum is its own type, so define bitwise operators
* on _notmuch_features. */
inline _notmuch_features
operator|(_notmuch_features a, _notmuch_features b)
{
return static_cast<_notmuch_features>(
static_cast<unsigned>(a) | static_cast<unsigned>(b));
}
inline _notmuch_features
operator&(_notmuch_features a, _notmuch_features b)
{
return static_cast<_notmuch_features>(
static_cast<unsigned>(a) & static_cast<unsigned>(b));
}
inline _notmuch_features
operator~(_notmuch_features a)
{
return static_cast<_notmuch_features>(~static_cast<unsigned>(a));
}
inline _notmuch_features&
operator|=(_notmuch_features &a, _notmuch_features b)
{
a = a | b;
return a;
}
inline _notmuch_features&
operator&=(_notmuch_features &a, _notmuch_features b)
{
a = a & b;
return a;
}
lib: merge internal prefix tables Replace multiple tables with some flags in a single table. This makes the code in notmuch_database_open_verbose a bit shorter, and it should also make it easier to add other options to fields, e.g. regexp searching.
2017-02-17 04:07:49 +01:00
/*
* Configuration options for xapian database fields */
typedef enum notmuch_field_flags {
NOTMUCH_FIELD_NO_FLAGS = 0,
NOTMUCH_FIELD_EXTERNAL = 1 << 0,
lib: create field processors from prefix table This is a bit more code than hardcoding the two existing field processors, but it should make it easy to add more.
2017-02-27 03:34:19 +01:00
NOTMUCH_FIELD_PROBABILISTIC = 1 << 1,
NOTMUCH_FIELD_PROCESSOR = 1 << 2,
lib: merge internal prefix tables Replace multiple tables with some flags in a single table. This makes the code in notmuch_database_open_verbose a bit shorter, and it should also make it easier to add other options to fields, e.g. regexp searching.
2017-02-17 04:07:49 +01:00
} notmuch_field_flag_t;
/*
* define bitwise operators to hide casts */
inline notmuch_field_flag_t
operator|(notmuch_field_flag_t a, notmuch_field_flag_t b)
{
return static_cast<notmuch_field_flag_t>(
static_cast<unsigned>(a) | static_cast<unsigned>(b));
}
inline notmuch_field_flag_t
operator&(notmuch_field_flag_t a, notmuch_field_flag_t b)
{
return static_cast<notmuch_field_flag_t>(
static_cast<unsigned>(a) & static_cast<unsigned>(b));
}
lib: make a global constant for query parser flags It's already kindof gross that this is hardcoded in two different places. We will also need these later in field processors calling back into the query parser.
2016-03-22 11:54:53 +01:00
#define NOTMUCH_QUERY_PARSER_FLAGS (Xapian::QueryParser::FLAG_BOOLEAN | \
Xapian::QueryParser::FLAG_PHRASE | \
Xapian::QueryParser::FLAG_LOVEHATE | \
Xapian::QueryParser::FLAG_BOOLEAN_ANY_CASE | \
Xapian::QueryParser::FLAG_WILDCARD | \
Xapian::QueryParser::FLAG_PURE_NOT)
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
struct _notmuch_database {
Catch and optionally print about exception at database->flush. If an earlier exception occurred, then it's not unexpected for the flush to fail as well. So in that case, we'll silently catch the exception. Otherwise, make some noise about things going wrong at the time of flush.
2009-11-22 03:54:20 +01:00
notmuch_bool_t exception_reported;
Switch from random to sequential thread identifiers. The sequential identifiers have the advantage of being guaranteed to be unique (until we overflow a 64-bit unsigned integer), and also take up half as much space in the "notmuch search" output (16 columns rather than 32). This change also has the side effect of fixing a bug where notmuch could block on /dev/random at startup (waiting for some entropy to appear). This bug was hit hard by the test suite, (which could easily exhaust the available entropy on common systems---resulting in large delays of the test suite).
2010-02-08 20:33:33 +01:00
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
char *path;
Switch from random to sequential thread identifiers. The sequential identifiers have the advantage of being guaranteed to be unique (until we overflow a 64-bit unsigned integer), and also take up half as much space in the "notmuch search" output (16 columns rather than 32). This change also has the side effect of fixing a bug where notmuch could block on /dev/random at startup (waiting for some entropy to appear). This bug was hit hard by the test suite, (which could easily exhaust the available entropy on common systems---resulting in large delays of the test suite).
2010-02-08 20:33:33 +01:00
Permit opening the notmuch database in read-only mode. We only rarely need to actually open the database for writing, but we always create a Xapian::WritableDatabase. This has the effect of preventing searches and like whilst updating the index. Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Acked-by: Carl Worth <cworth@cworth.org>
2009-11-21 20:54:25 +01:00
notmuch_database_mode_t mode;
lib: Add support for nested atomic sections. notmuch_database_t now keeps a nesting count and we only start a transaction or commit for the outermost atomic section. Introduces a new error, NOTMUCH_STATUS_UNBALANCED_ATOMIC.
2011-06-11 05:35:06 +02:00
int atomic_nesting;
lib: Add per-message last modification tracking This adds a new document value that stores the revision of the last modification to message metadata, where the revision number increases monotonically with each database commit. An alternative would be to store the wall-clock time of the last modification of each message. In principle this is simpler and has the advantage that any process can determine the current timestamp without support from libnotmuch. However, even assuming a computer's clock never goes backward and ignoring clock skew in networked environments, this has a fatal flaw. Xapian uses (optimistic) snapshot isolation, which means reads can be concurrent with writes. Given this, consider the following time line with a write and two read transactions: write |-X-A--------------| read 1 |---B---| read 2 |---| The write transaction modifies message X and records the wall-clock time of the modification at A. The writer hangs around for a while and later commits its change. Read 1 is concurrent with the write, so it doesn't see the change to X. It does some query and records the wall-clock time of its results at B. Transaction read 2 later starts after the write commits and queries for changes since wall-clock time B (say the reads are performing an incremental backup). Even though read 1 could not see the change to X, read 2 is told (correctly) that X has not changed since B, the time of the last read. In fact, X changed before wall-clock time A, but the change was not visible until *after* wall-clock time B, so read 2 misses the change to X. This is tricky to solve in full-blown snapshot isolation, but because Xapian serializes writes, we can use a simple, monotonically increasing database revision number. Furthermore, maintaining this revision number requires no more IO than a wall-clock time solution because Xapian already maintains statistics on the upper (and lower) bound of each value stream.
2014-10-13 08:20:01 +02:00
/* TRUE if changes have been made in this atomic section */
notmuch_bool_t atomic_dirty;
Permit opening the notmuch database in read-only mode. We only rarely need to actually open the database for writing, but we always create a Xapian::WritableDatabase. This has the effect of preventing searches and like whilst updating the index. Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Acked-by: Carl Worth <cworth@cworth.org>
2009-11-21 20:54:25 +01:00
Xapian::Database *xapian_db;
Switch from random to sequential thread identifiers. The sequential identifiers have the advantage of being guaranteed to be unique (until we overflow a 64-bit unsigned integer), and also take up half as much space in the "notmuch search" output (16 columns rather than 32). This change also has the side effect of fixing a bug where notmuch could block on /dev/random at startup (waiting for some entropy to appear). This bug was hit hard by the test suite, (which could easily exhaust the available entropy on common systems---resulting in large delays of the test suite).
2010-02-08 20:33:33 +01:00
lib: Database version 3: Introduce fine-grained "features" Previously, our database schema was versioned by a single number. Each database schema change had to occur "atomically" in Notmuch's development history: before some commit, Notmuch used version N, after that commit, it used version N+1. Hence, each new schema version could introduce only one change, the task of developing a schema change fell on a single person, and it all had to happen and be perfect in a single commit series. This made introducing a new schema version hard. We've seen only two schema changes in the history of Notmuch. This commit introduces database schema version 3; hopefully the last schema version we'll need for a while. With this version, we switch from a single version number to "features": a set of named, independent aspects of the database schema. Features should make backwards compatibility easier. For many things, it should be easy to support databases both with and without a feature, which will allow us to make upgrades optional and will enable "unstable" features that can be developed and tested over time. Features also make forwards compatibility easier. The features recorded in a database include "compatibility flags," which can indicate to an older version of Notmuch when it must support a given feature to open the database for read or for write. This lets us replace the old vague "I don't recognize this version, so something might go wrong, but I promise to try my best" warnings upon opening a database with an unknown version with precise errors. If a database is safe to open for read/write despite unknown features, an older version will know that and issue no message at all. If the database is not safe to open for read/write because of unknown features, an older version will know that, too, and can tell the user exactly which required features it lacks support for.
2014-08-25 19:26:00 +02:00
/* Bit mask of features used by this database. This is a
* bitwise-OR of NOTMUCH_FEATURE_* values (above). */
enum _notmuch_features features;
Avoid database corruption by not adding partially-constructed mail documents. Previously we were using Xapian's add_document to allocate document ID values for notmuch_message_t objects. This had the drawback of adding a partially constructed mail document to the database. If notmuch was subsequently interrupted before fully populating this document, then later runs would be quite confused when seeing the partial documents. There are reports from the wild of people hitting internal errors of the form "Message ... has no thread ID" for example, (which is currently an unrecoverable error). We fix this by manually allocating document IDs without adding documents. With this change, we never call Xapian's add_document method, but only replace_document with either the current document ID of a message or a new one that we have allocated.
2010-06-04 19:16:53 +02:00
unsigned int last_doc_id;
Switch from random to sequential thread identifiers. The sequential identifiers have the advantage of being guaranteed to be unique (until we overflow a 64-bit unsigned integer), and also take up half as much space in the "notmuch search" output (16 columns rather than 32). This change also has the side effect of fixing a bug where notmuch could block on /dev/random at startup (waiting for some entropy to appear). This bug was hit hard by the test suite, (which could easily exhaust the available entropy on common systems---resulting in large delays of the test suite).
2010-02-08 20:33:33 +01:00
uint64_t last_thread_id;
lib: add a log function with output to a string in notmuch_database_t In principle in the future this could do something fancier than asprintf.
2014-12-26 09:01:01 +01:00
/* error reporting; this value persists only until the
* next library call. May be NULL */
char *status_string;
lib: Add per-message last modification tracking This adds a new document value that stores the revision of the last modification to message metadata, where the revision number increases monotonically with each database commit. An alternative would be to store the wall-clock time of the last modification of each message. In principle this is simpler and has the advantage that any process can determine the current timestamp without support from libnotmuch. However, even assuming a computer's clock never goes backward and ignoring clock skew in networked environments, this has a fatal flaw. Xapian uses (optimistic) snapshot isolation, which means reads can be concurrent with writes. Given this, consider the following time line with a write and two read transactions: write |-X-A--------------| read 1 |---B---| read 2 |---| The write transaction modifies message X and records the wall-clock time of the modification at A. The writer hangs around for a while and later commits its change. Read 1 is concurrent with the write, so it doesn't see the change to X. It does some query and records the wall-clock time of its results at B. Transaction read 2 later starts after the write commits and queries for changes since wall-clock time B (say the reads are performing an incremental backup). Even though read 1 could not see the change to X, read 2 is told (correctly) that X has not changed since B, the time of the last read. In fact, X changed before wall-clock time A, but the change was not visible until *after* wall-clock time B, so read 2 misses the change to X. This is tricky to solve in full-blown snapshot isolation, but because Xapian serializes writes, we can use a simple, monotonically increasing database revision number. Furthermore, maintaining this revision number requires no more IO than a wall-clock time solution because Xapian already maintains statistics on the upper (and lower) bound of each value stream.
2014-10-13 08:20:01 +02:00
/* Highest committed revision number. Modifications are recorded
* under a higher revision number, which can be generated with
* notmuch_database_new_revision. */
unsigned long revision;
lib: API to retrieve database revision and UUID This exposes the committed database revision to library users along with a UUID that can be used to detect when revision numbers are no longer comparable (e.g., because the database has been replaced).
2014-10-13 08:20:02 +02:00
const char *uuid;
lib: Add per-message last modification tracking This adds a new document value that stores the revision of the last modification to message metadata, where the revision number increases monotonically with each database commit. An alternative would be to store the wall-clock time of the last modification of each message. In principle this is simpler and has the advantage that any process can determine the current timestamp without support from libnotmuch. However, even assuming a computer's clock never goes backward and ignoring clock skew in networked environments, this has a fatal flaw. Xapian uses (optimistic) snapshot isolation, which means reads can be concurrent with writes. Given this, consider the following time line with a write and two read transactions: write |-X-A--------------| read 1 |---B---| read 2 |---| The write transaction modifies message X and records the wall-clock time of the modification at A. The writer hangs around for a while and later commits its change. Read 1 is concurrent with the write, so it doesn't see the change to X. It does some query and records the wall-clock time of its results at B. Transaction read 2 later starts after the write commits and queries for changes since wall-clock time B (say the reads are performing an incremental backup). Even though read 1 could not see the change to X, read 2 is told (correctly) that X has not changed since B, the time of the last read. In fact, X changed before wall-clock time A, but the change was not visible until *after* wall-clock time B, so read 2 misses the change to X. This is tricky to solve in full-blown snapshot isolation, but because Xapian serializes writes, we can use a simple, monotonically increasing database revision number. Furthermore, maintaining this revision number requires no more IO than a wall-clock time solution because Xapian already maintains statistics on the upper (and lower) bound of each value stream.
2014-10-13 08:20:01 +02:00
lib: add _notmuch_database_reopen The main expected use is to recover from a Xapian::DatabaseChanged exception.
2017-02-24 02:38:24 +01:00
/* Keep track of the number of times the database has been re-opened
* (or other global invalidations of notmuch's caching)
*/
unsigned long view;
notmuch dump: Fix the sorting of results. To properly support sorting in notmuch_query we know use an Enquire object. We also throw in a QueryParser too, so we're really close to being able to support arbitrary full-text searches. I took a look at the supported QueryParser syntax and chose a set of flags for everything I like, (such as supporting Boolean operators in either case ("AND" or "and"), supporting phrase searching, supporting + and - to include/preclude terms, and supporting a trailing * on any term as a wildcard).
2009-10-21 09:35:56 +02:00
Xapian::QueryParser *query_parser;
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
Xapian::TermGenerator *term_gen;
Add rudimentary date-based search. The rudimentary aspect here is that the date ranges are specified with UNIX timestamp values (number of seconds since 1970-01-01 UTC). One thing that can help here is using the date program to determins timestamps, such as: $(date +%s -d 2009-10-01)..$(date +%s) Long-term, we'll probably need to do our own query parsing to be able to support directly-specified dates and also relative expressions like "since:'2 months ago'".
2009-11-23 16:58:35 +01:00
Xapian::ValueRangeProcessor *value_range_processor;
lib: add date range query support Add a custom value range processor to enable date and time searches of the form date:since..until, where "since" and "until" are expressions understood by the previously added date/time parser, to restrict the results to messages within a particular time range (based on the Date: header). If "since" or "until" describes date/time at an accuracy of days or less, the values are rounded according to the accuracy, towards past for "since" and towards future for "until". For example, date:november..yesterday would match from the beginning of November until the end of yesterday. Expressions such as date:today..today means since the beginning of today until the end of today. Open-ended ranges are supported (since Xapian 1.2.1), i.e. you can specify date:..until or date:since.. to not limit the start or end date, respectively. CAVEATS: Xapian does not support spaces in range expressions. You can replace the spaces with '_', or (in most cases) '-', or (in some cases) leave the spaces out altogether. Entering date:expr without ".." (for example date:yesterday) will not work as you might expect. You can achieve the expected result by duplicating the expr both sides of ".." (for example date:yesterday..yesterday). Open-ended ranges won't work with pre-1.2.1 Xapian, but they don't produce an error either. Signed-off-by: Jani Nikula <jani@nikula.org>
2012-10-30 21:32:37 +01:00
Xapian::ValueRangeProcessor *date_range_processor;
lib: Add "lastmod:" queries for filtering by last modification The implementation is essentially the same as the date range search prior to Jani's fancy date parser.
2014-10-13 08:20:03 +02:00
Xapian::ValueRangeProcessor *last_mod_range_processor;
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
};
lib: Database version 3: Introduce fine-grained "features" Previously, our database schema was versioned by a single number. Each database schema change had to occur "atomically" in Notmuch's development history: before some commit, Notmuch used version N, after that commit, it used version N+1. Hence, each new schema version could introduce only one change, the task of developing a schema change fell on a single person, and it all had to happen and be perfect in a single commit series. This made introducing a new schema version hard. We've seen only two schema changes in the history of Notmuch. This commit introduces database schema version 3; hopefully the last schema version we'll need for a while. With this version, we switch from a single version number to "features": a set of named, independent aspects of the database schema. Features should make backwards compatibility easier. For many things, it should be easy to support databases both with and without a feature, which will allow us to make upgrades optional and will enable "unstable" features that can be developed and tested over time. Features also make forwards compatibility easier. The features recorded in a database include "compatibility flags," which can indicate to an older version of Notmuch when it must support a given feature to open the database for read or for write. This lets us replace the old vague "I don't recognize this version, so something might go wrong, but I promise to try my best" warnings upon opening a database with an unknown version with precise errors. If a database is safe to open for read/write despite unknown features, an older version will know that and issue no message at all. If the database is not safe to open for read/write because of unknown features, an older version will know that, too, and can tell the user exactly which required features it lacks support for.
2014-08-25 19:26:00 +02:00
/* Prior to database version 3, features were implied by the database
* version number, so hard-code them for earlier versions. */
#define NOTMUCH_FEATURES_V0 ((enum _notmuch_features)0)
#define NOTMUCH_FEATURES_V1 (NOTMUCH_FEATURES_V0 | NOTMUCH_FEATURE_FILE_TERMS | \
NOTMUCH_FEATURE_DIRECTORY_DOCS)
#define NOTMUCH_FEATURES_V2 (NOTMUCH_FEATURES_V1 | NOTMUCH_FEATURE_BOOL_FOLDER)
/* Current database features. If any of these are missing from a
* database, request an upgrade.
Add the NOTMUCH_FEATURE_INDEXED_MIMETYPES database feature This feature will exist in all newly created databases, but there is no upgrade provided for it. If this flag exists, it indicates that the database was created after the indexed MIME-types feature was added.
2015-01-23 00:43:37 +01:00
* NOTMUCH_FEATURE_FROM_SUBJECT_ID_VALUES and
* NOTMUCH_FEATURE_INDEXED_MIMETYPES are not included because upgrade
* doesn't currently introduce the features (though brand new databases
* will have it). */
lib: Database version 3: Introduce fine-grained "features" Previously, our database schema was versioned by a single number. Each database schema change had to occur "atomically" in Notmuch's development history: before some commit, Notmuch used version N, after that commit, it used version N+1. Hence, each new schema version could introduce only one change, the task of developing a schema change fell on a single person, and it all had to happen and be perfect in a single commit series. This made introducing a new schema version hard. We've seen only two schema changes in the history of Notmuch. This commit introduces database schema version 3; hopefully the last schema version we'll need for a while. With this version, we switch from a single version number to "features": a set of named, independent aspects of the database schema. Features should make backwards compatibility easier. For many things, it should be easy to support databases both with and without a feature, which will allow us to make upgrades optional and will enable "unstable" features that can be developed and tested over time. Features also make forwards compatibility easier. The features recorded in a database include "compatibility flags," which can indicate to an older version of Notmuch when it must support a given feature to open the database for read or for write. This lets us replace the old vague "I don't recognize this version, so something might go wrong, but I promise to try my best" warnings upon opening a database with an unknown version with precise errors. If a database is safe to open for read/write despite unknown features, an older version will know that and issue no message at all. If the database is not safe to open for read/write because of unknown features, an older version will know that, too, and can tell the user exactly which required features it lacks support for.
2014-08-25 19:26:00 +02:00
#define NOTMUCH_FEATURES_CURRENT \
(NOTMUCH_FEATURE_FILE_TERMS | NOTMUCH_FEATURE_DIRECTORY_DOCS | \
lib: Add per-message last modification tracking This adds a new document value that stores the revision of the last modification to message metadata, where the revision number increases monotonically with each database commit. An alternative would be to store the wall-clock time of the last modification of each message. In principle this is simpler and has the advantage that any process can determine the current timestamp without support from libnotmuch. However, even assuming a computer's clock never goes backward and ignoring clock skew in networked environments, this has a fatal flaw. Xapian uses (optimistic) snapshot isolation, which means reads can be concurrent with writes. Given this, consider the following time line with a write and two read transactions: write |-X-A--------------| read 1 |---B---| read 2 |---| The write transaction modifies message X and records the wall-clock time of the modification at A. The writer hangs around for a while and later commits its change. Read 1 is concurrent with the write, so it doesn't see the change to X. It does some query and records the wall-clock time of its results at B. Transaction read 2 later starts after the write commits and queries for changes since wall-clock time B (say the reads are performing an incremental backup). Even though read 1 could not see the change to X, read 2 is told (correctly) that X has not changed since B, the time of the last read. In fact, X changed before wall-clock time A, but the change was not visible until *after* wall-clock time B, so read 2 misses the change to X. This is tricky to solve in full-blown snapshot isolation, but because Xapian serializes writes, we can use a simple, monotonically increasing database revision number. Furthermore, maintaining this revision number requires no more IO than a wall-clock time solution because Xapian already maintains statistics on the upper (and lower) bound of each value stream.
2014-10-13 08:20:01 +02:00
NOTMUCH_FEATURE_BOOL_FOLDER | NOTMUCH_FEATURE_GHOSTS | \
NOTMUCH_FEATURE_LAST_MOD)
lib: Database version 3: Introduce fine-grained "features" Previously, our database schema was versioned by a single number. Each database schema change had to occur "atomically" in Notmuch's development history: before some commit, Notmuch used version N, after that commit, it used version N+1. Hence, each new schema version could introduce only one change, the task of developing a schema change fell on a single person, and it all had to happen and be perfect in a single commit series. This made introducing a new schema version hard. We've seen only two schema changes in the history of Notmuch. This commit introduces database schema version 3; hopefully the last schema version we'll need for a while. With this version, we switch from a single version number to "features": a set of named, independent aspects of the database schema. Features should make backwards compatibility easier. For many things, it should be easy to support databases both with and without a feature, which will allow us to make upgrades optional and will enable "unstable" features that can be developed and tested over time. Features also make forwards compatibility easier. The features recorded in a database include "compatibility flags," which can indicate to an older version of Notmuch when it must support a given feature to open the database for read or for write. This lets us replace the old vague "I don't recognize this version, so something might go wrong, but I promise to try my best" warnings upon opening a database with an unknown version with precise errors. If a database is safe to open for read/write despite unknown features, an older version will know that and issue no message at all. If the database is not safe to open for read/write because of unknown features, an older version will know that, too, and can tell the user exactly which required features it lacks support for.
2014-08-25 19:26:00 +02:00
Add a generic function to get a list of terms with some prefix. Replace _notmuch_convert_tags with this and simplify _create_filenames_for_terms_with_prefix. This will also come in handy shortly to get the message file name list.
2010-12-09 06:32:35 +01:00
/* Return the list of terms from the given iterator matching a prefix.
* The prefix will be stripped from the strings in the returned list.
* The list will be allocated using ctx as the talloc context.
notmuch: New function to retrieve all tags from the database. This patch adds a new function called notmuch_database_get_all_tags which can be used to obtain a list of all tags from the database (in other words, the list contains all tags from all messages). The function produces an alphabetically sorted list. To add support for the new function, we rip the guts off of notmuch_message_get_tags and put them in a new generic function called _notmuch_convert_tags. The generic function takes a Xapian::TermIterator as argument and uses the iterator to find tags. This makes the function usable with different Xapian objects. Function notmuch_message_get_tags is then reimplemented to call the generic function with message->doc.termlist_begin() as argument. Similarly, we implement notmuch_message_database_get_all_tags, the function calls the generic function with db->xapian_db->allterms_begin() as argument. Finally, notmuch_database_get_all_tags is exported through lib/notmuch.h Signed-off-by: Jan Janak <jan@ryngle.com>
2009-11-23 01:10:54 +01:00
*
* The function returns NULL on failure.
*/
Add a generic function to get a list of terms with some prefix. Replace _notmuch_convert_tags with this and simplify _create_filenames_for_terms_with_prefix. This will also come in handy shortly to get the message file name list.
2010-12-09 06:32:35 +01:00
notmuch_string_list_t *
_notmuch_database_get_terms_with_prefix (void *ctx, Xapian::TermIterator &i,
Xapian::TermIterator &end,
const char *prefix);
notmuch: New function to retrieve all tags from the database. This patch adds a new function called notmuch_database_get_all_tags which can be used to obtain a list of all tags from the database (in other words, the list contains all tags from all messages). The function produces an alphabetically sorted list. To add support for the new function, we rip the guts off of notmuch_message_get_tags and put them in a new generic function called _notmuch_convert_tags. The generic function takes a Xapian::TermIterator as argument and uses the iterator to find tags. This makes the function usable with different Xapian objects. Function notmuch_message_get_tags is then reimplemented to call the generic function with message->doc.termlist_begin() as argument. Similarly, we implement notmuch_message_database_get_all_tags, the function calls the generic function with db->xapian_db->allterms_begin() as argument. Finally, notmuch_database_get_all_tags is exported through lib/notmuch.h Signed-off-by: Jan Janak <jan@ryngle.com>
2009-11-23 01:10:54 +01:00
lib: isolate n_d_add_message and helper functions into own file 'database.cc' is becoming a monster, and it's hard to follow what the various static functions are used for. It turns out that about 1/3 of this file notmuch_database_add_message and helper functions not used by any other function. This commit isolates this code into it's own file. Some side effects of this refactoring: - find_doc_ids becomes the non-static (but still private) _notmuch_database_find_doc_ids - a few instances of 'string' have 'std::' prepended, avoiding the need for 'using namespace std;' in the new file.
2017-06-04 14:32:24 +02:00
void
_notmuch_database_find_doc_ids (notmuch_database_t *notmuch,
const char *prefix_name,
const char *value,
Xapian::PostingIterator *begin,
Xapian::PostingIterator *end);
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
#endif
Reference in a new issue Copy permalink