Commit graph

697 commits

Author SHA1 Message Date
David Bremner
4ce7591610 lib: paper over allocation difference
In gmime 3.0 this function is "transfer none", so no deallocation is
needed (or permitted)
2017-07-14 21:23:52 -03:00
David Bremner
eeb64cdeeb lib: add version of _n_m_f_get_combinded_header for gmime 3.0
The iterator is gone, so we need a new loop structure.
2017-07-14 21:23:52 -03:00
David Bremner
439c5896b6 lib: refactor _notmuch_messsage_file_get_combined_header
We need to rewrite the loop for gmime-3.0; move the loop body to its
own function to avoid code duplication.  Keep the common exit via
"goto DONE" to make this pure code movement.  It's important to note
that the existing exit path only deallocates the iterator.
2017-07-14 21:23:52 -03:00
David Bremner
c040464a7c lib: wrap use of g_mime_utils_header_decode_date
This changes return type in gmime 3.0
2017-07-14 21:23:52 -03:00
David Bremner
cbb2d5608e lib/cli: replace use of g_mime_message_get_sender
This function changes semantics in gmime-3.0 so make a new function
that provides the same functionality in both
2017-07-14 17:58:09 -03:00
David Bremner
6dd00d6486 lib/index: add simple html filter
The filter just drops all (HTML) tags. As an enabling change, pass the
content type to the filter constructor so we can decide which scanner
to user.
2017-07-01 12:32:27 -03:00
David Bremner
64f81f95a1 lib/index.cc: generalize filter state machine
To match things more complicated than fixed strings, we need states
with multiple out arrows.
2017-07-01 12:32:17 -03:00
David Bremner
4a085a5137 lib/index: separate state table definition from scanner.
We want to reuse the scanner definition with a different table.  This
is mainly code movement, and making the state table part of the filter
struct/class.
2017-07-01 12:32:03 -03:00
David Bremner
20c15bc820 lib/index: generalize name of indexing filter
In followup commits we will generalize the functionality of this
filter to deal with other types of non-indexable content.
2017-07-01 12:31:55 -03:00
Jani Nikula
30c475c1ef build: visibility=default for library structs is no longer needed
Commit d5523ead90 ("Mark some structures in the library interface
with visibility=default attribute.") fixed some mixed visibility
issues with structs. With the symbol default visibility reversed, this
is no longer a problem.
2017-05-13 08:38:18 -03:00
Jani Nikula
bc11759dd1 build: switch to hiding libnotmuch symbols by default
The dynamic generation of the linker version script for libnotmuch
exports has grown rather complicated.

Reverse the visibility control by hiding symbols by default using
-fvisibility=hidden, and explicitly exporting symbols in notmuch.h
using #pragma GCC visibility. (We could also use __attribute__
((visibility ("default"))) for each exported function, but the pragma
is more convenient.)

The above is not quite enough alone, as it would "leak" a number of
weak symbols from Xapian and C++ standard library. Combine it with a
small static version script that filters out everything except the
notmuch_* symbols that we explicitly exposed, and the C++ RTTI
typeinfo symbols for exception handling.

Finally, as the symbol hiding test can no longer look at the generated
symbol table, switch the test to parse the functions from notmuch.h.
2017-05-12 07:17:18 -03:00
Jani Nikula
d5ed9af0e4 build: do not export compat functions from lib
Commits 9db2145272 ("lib/gen-version-script.h: add getline and
getdelim to notmuch.sym if needed") and 3242e29e57 ("build: add
canonicalize_file_name to symbols exported from libnotmuch.so")
started exporting compat functions from libnotmuch so that the cli
could use them. But we shouldn't export such functions from the
library. They are not part of our ABI. Instead, the cli should include
its own copies of the compat functions.
2017-05-11 20:41:10 -03:00
David Bremner
11d47950c1 lib: Add regexp expansion for for tags and paths
From a UI perspective this looks similar to what was already provided
for from, subject, and mid, but the implementation is quite
different. It uses the database's list of terms to construct a term
based query equivalent to the passed regular expression.
2017-05-09 07:44:29 -03:00
David Bremner
eab365c742 lib: Add regexp searching for mid: prefix
The bulk of the change is passing in the field options to the regexp
field processor, so that we can properly handle the
fallback (non-regexp case).
2017-05-09 07:44:15 -03:00
Fredrik Fornwall
e565118172 Replace index(3) with strchr(3)
The index(3) function has been deprecated in POSIX since 2001 and
removed in 2008, and most code in notmuch already calls strchr(3).

This fixes a compilation error on Android whose libc does not have
index(3).
2017-04-20 06:59:22 -03:00
David Bremner
e1c1d33f37 Merge branch 'release'
Another regexp search fix.
2017-03-29 20:58:34 -03:00
David Bremner
cb84f84878 lib: handle empty string in regexp field processors
The non-field processor behaviour is is convert the corresponding
queries into a search for the unprefixed terms. This yields pretty
surprising results so I decided to generate a query that would match
the terms (i.e. none with that prefix) generated for an empty header.
2017-03-29 20:44:32 -03:00
David Bremner
d877240f4e Merge branch 'release'
wildcard search fixes, plus release busywork
2017-03-25 11:51:03 -03:00
David Bremner
38a56b98f9 lib: only trigger phrase processing for regexp fields when needed
The argument is that if the string passed to the field processor has
no spaces, then the added quotes won't have any benefit except for
disabling wildcards. But disabling wildcards doesn't seem very useful
in the normal Xapian query parser, since they're stripped before
generating terms anyway. It does mean that the query 'from:"foo*"' will
not be precisely equivalent to 'from:foo' as it is for the non
field-processor version.
2017-03-24 09:24:13 -03:00
David Bremner
242d5a3ed5 lib: make notmuch_query_add_tag_exclude return a status value
Since this is an ABI breaking change, but we already bumped the SONAME
for the next release
2017-03-22 08:47:13 -03:00
David Bremner
3721bd45d7 lib: replace deprecated n_q_count_threads with status returning version
This function was deprecated in notmuch 0.21.  We re-use the name for
a status returning version, and deprecate the _st name.
2017-03-22 08:35:07 -03:00
David Bremner
5ce8e0b11b lib: replace deprecated n_q_count_messages with status returning version
This function was deprecated in notmuch 0.21.  We re-use the name for
a status returning version, and deprecate the _st name. One or two
remaining uses of the (removed) non-status returning version fixed at
the same time
2017-03-22 08:35:07 -03:00
David Bremner
86cbd215eb lib: replace deprecated n_q_search_messages with status returning version
This function was deprecated in notmuch 0.21.  We re-use the name for
a status returning version, and deprecate the _st name.
2017-03-22 08:35:07 -03:00
David Bremner
1e982de508 lib: replace n_query_search_threads with status returning version
This function was deprecated in notmuch 0.21. We finally remove the
deprecated API, and rename the status returning version to the simpler
name. The status returning is kept as a deprecated alias.
2017-03-22 08:28:09 -03:00
David Bremner
fc63c15833 lib: bump SONAME to libnotmuch5
We plan a sequence of ABI breaking changes. Put the SONAME change in a
separate commit to make reordering easier.
2017-03-22 08:27:58 -03:00
David Bremner
c39f6361d0 rename libutil.a to libnotmuch_util.a
Apparently some systems (MacOS?) have a system library called libutil
and the name conflict causes problems. Since this library is quite
notmuch specific, rename it to something less generic.
2017-03-18 21:37:43 -03:00
David Bremner
a8a2705222 Merge branch 'release'
Merge in memory fixes
2017-03-18 21:02:42 -03:00
Tomi Ollila
06adc27668 lib/message.cc: fix Coverity finding (use after free)
The object where pointer to `data` was received was deleted before
it was used in _notmuch_string_list_append().

Relevant Coverity messages follow:

3: extract
Assigning: data = std::__cxx11::string(message->doc.()).c_str(),
which extracts wrapped state from temporary of type std::__cxx11::string.

4: dtor_free
The internal representation of temporary of type std::__cxx11::string
is freed by its destructor.

5: use after free:
Wrapper object use after free (WRAPPER_ESCAPE)
Using internal representation of destroyed object local data.
2017-03-18 20:59:46 -03:00
David Bremner
62822a4e2d lib: clamp return value of g_mime_utils_header_decode_date to >=0
For reasons not completely understood at this time, gmime (as of
2.6.22) is returning a date before 1900 on bad date input. Since this
confuses some other software, we clamp such dates to 0,
i.e. 1970-01-01.
2017-03-15 21:58:25 -03:00
Jani Nikula
d56a801b67 lib/database: reduce try block scope to things that really need it
No need to maintain the pure C stuff within a try block, it's arguably
confusing. This also reduces indent for a bunch of code. No functional
changes.
2017-03-10 09:21:05 -04:00
Olly Betts
81bd72cebb lib: Fix RegexpPostingSource
Remove incorrect skipping to first match from init(), and add explicit
skip_to() and check() methods to work around xapian-core bug (the
check() method will also improve speed when filtering by one of
these).
2017-03-07 19:44:36 -04:00
David Bremner
dfacfe14f3 lib: query make exclude handling non-destructive
We filter added exclude at add time, rather than modifying the query by
count search. As noted in the comments, there are several ignored
conditions here.
2017-03-04 20:47:25 -04:00
David Bremner
e209b71873 lib: centralize query parsing, store results.
The main goal is to prepare the way for non-destructive (or at least
less destructive) exclude tag handling. It does this by having a
pre-parsed query available for further processing. This also allows us
to provide slightly more precise error messages.
2017-03-04 20:47:25 -04:00
Jani Nikula
f3edc5dc86 lib: use delete[] to free buffer allocated using new[]
Fix warning caught by clang:

lib/regexp-fields.cc:41:2: warning: 'delete' applied to a pointer that was allocated
      with 'new[]'; did you mean 'delete[]'? [-Wmismatched-new-delete]
        delete buffer;
        ^
              []
lib/regexp-fields.cc:37:17: note: allocated with 'new[]' here
        char *buffer = new char[len];
                       ^
2017-03-04 20:42:39 -04:00
David Bremner
6cb1c617a7 lib: add mid: as a synonym for id:
mid: is the url scheme suggested by URL 2392. We also plan to
introduce more flexible searches for mid: than are possible with
id: (in order not to break assumptions about the special behaviour of
id:, e.g. identifying at most one message).
2017-03-03 17:46:48 -04:00
David Bremner
55524bb063 lib: regexp matching in 'subject' and 'from'
the idea is that you can run

% notmuch search subject:/<your-favourite-regexp>/
% notmuch search from:/<your-favourite-regexp>/

or

% notmuch search subject:"your usual phrase search"
% notmuch search from:"usual phrase search"

This feature is only available with recent Xapian, specifically
support for field processors is needed.

It should work with bindings, since it extends the query parser.

This is easy to extend for other value slots, but currently the only
value slots are date, message_id, from, subject, and last_mod. Date is
already searchable;  message_id is left for a followup commit.

This was originally written by Austin Clements, and ported to Xapian
field processors (from Austin's custom query parser) by yours truly.
2017-03-03 17:46:48 -04:00
David Bremner
31b8ce4558 lib: create field processors from prefix table
This is a bit more code than hardcoding the two existing field
processors, but it should make it easy to add more.
2017-03-03 07:15:13 -04:00
David Bremner
7bd63833bf lib/message.cc: use view number to invalidate cached metadata
Currently the view number is incremented by notmuch_database_reopen
2017-02-25 21:15:38 -04:00
David Bremner
e0b22c139c lib: handle DatabaseModifiedError in _n_message_ensure_metadata
The retries are hardcoded to a small number, and error handling aborts
than propagating errors from notmuch_database_reopen. These are both
somewhat justified by the assumption that most things that can go
wrong in Xapian::Database::reopen are rare and fatal. Here's the brief
discussion with Xapian upstream:

   24-02-2017 08:12:57 < bremner> any intuition about how likely
      Xapian::Database::reopen is to fail? I'm catching a
      DatabaseModifiedError somewhere where handling any further errors is
      tricky, and wondering about treating a failed reopen as as "the
      impossible happened, stopping"

   24-02-2017 16:22:34 < olly> bremner: there should not be much scope for
    failure - stuff like out of memory or disk errors, which are probably a
    good enough excuse to stop
2017-02-25 21:13:50 -04:00
David Bremner
e17a914b77 lib: add _notmuch_database_reopen
The main expected use is to recover from a Xapian::DatabaseChanged
exception.
2017-02-25 21:09:17 -04:00
David Bremner
e0e8586fc7 Merge branch 'release'
Merge in g_hash_table read-after-free fix
2017-02-23 09:08:15 -04:00
David Bremner
884dccf293 lib: make _notmuch_message_ensure_property_map static
It's not called outside message.cc
2017-02-23 08:54:36 -04:00
David Bremner
3db9e94b0e lib: make _notmuch_message_ensure_metadata static
It's not called anywhere outside message.cc.
2017-02-23 08:54:25 -04:00
David Bremner
4e649d000b lib: fix g_hash_table related read-after-free bug
The two g_hash_table functions (insert, add) have different behaviour
with respect to existing keys. g_hash_table_insert frees the new key,
while g_hash_table_add (which is really g_hash_table_replace in
disguise) frees the existing key. With this change 'ref' is live until
the end of the function (assuming single-threaded access to
'hash'). We can't guarantee it will continue to be live in the
future (i.e. there may be a future key duplication) so we copy it with
the allocation context passed to parse_references (in practice this is
the notmuch_message_t object whose parents we are finding).

Thanks to Tomi for the simpler approach to the problem based on
reading the fine glib manual.
2017-02-22 06:28:03 -04:00
David Bremner
0e037c34dd lib: Let Xapian manage the memory for FieldProcessors
It turns out this is exactly what release() is for; Xapian will
deallocate the objects when it's done with them.
2017-02-18 22:18:06 -04:00
David Bremner
e30fa4182f lib: merge internal prefix tables
Replace multiple tables with some flags in a single table. This makes
the code in notmuch_database_open_verbose a bit shorter, and it should
also make it easier to add other options to fields, e.g. regexp
searching.
2017-02-18 22:17:39 -04:00
David Bremner
70519319b5 lib: optimize counting documents
From #xapian

olly> bremner: btw, i noticed notmuch count see ms to request all the documents and then ignores them

bremner> hmm. There's something funny about the way that notmuch uses matches in general iirc

olly> it should be able to do: mset = enquire.get_mset (0, 0, notmuch->xapian_db->get_doccount ());
...
olly> get_matches_estimated() will be exact because check_at_least is the size of the database
2017-01-27 21:54:44 -04:00
Steven Allen
4a2ce7b570 docs: fix notmuch_message_properties_value documentation
It returns the value, not the key.
2017-01-15 14:25:00 -04:00
Jani Nikula
c906da9f60 lib: use glib for sha1 digests instead of embedding libsha1
We already depend on glib both directly and indirectly (via gmime). We
might as well make use of its facilities. Drop the embedded libsha1
and use glib for sha1 digests.
2017-01-08 10:50:38 -04:00
Jani Nikula
217404ff86 lib: fix the todo comment placement on NOTMUCH_STATUS_XAPIAN_EXCEPTION
The todo comment got separated from the status it's related to at
commit 3f32fd8a1c ("Add missing comment for
NOTMUCH_STATUS_READONLY_DATABASE."). Later, commit b65ca8e0ba ("lib:
modify notmuch.h for automatic document generation") moved it, but to
the wrong place. Fix the location.
2017-01-07 08:30:08 -04:00
David Bremner
0abcad7c0e lib: optionally silence Xapian deprecation warnings
This is not ideal, but the new API is not available in Xapian 1.2.x, and
it seems to soon to depend on Xapian >= 1.4
2016-11-15 07:47:55 -04:00
David Bremner
7f07a3f0ed lib: replace deprecated xapian call 'flush()' with 'commit()'
This will make notmuch incompatible with Xapian before 1.1.0, which is
more than 6 years old this point.
2016-10-25 18:13:52 -03:00
David Bremner
b2d6f07a02 lib: document API added in 0.23
The API was already documented, but for future readers note when the
functions were added,
2016-10-06 22:46:01 -03:00
David Bremner
af8903df34 require xapian >= 1.2.6
It seems that no-one tried to compile without Xapian compact support
since March of 2015, since that's when I introduced a syntax error in
that branch of the ifdef.

Given the choice of maintaining this underused branch of code, or
bumping the Xapian dependency to a version from 2011, it seems
reasonable to do the latter.
2016-10-06 22:45:46 -03:00
David Bremner
c2e74662bb lib: bump minor version to mark added symbols
This should not change the SONAME, and therefore won't change the
dynamic linking behaviour, but it may help some users debug missing
symbols in case their libnotmuch is too old.
2016-10-01 22:19:07 -03:00
Tomi Ollila
1c3a8e0898 lib/database.cc: fix misleading indentation
Found by gcc 6.1.1 -Wmisleading-indentation option (set by -Wall).
2016-09-28 08:14:08 -03:00
David Bremner
514a0a6a3b lib: add talloc reference from string map iterator to map
This is needed so that when the map is modified during traversal, and
thus unlinked by the database code, the map is not disposed of until the
iterator is done with it.
2016-09-24 10:08:45 -03:00
Daniel Kahn Gillmor
693ca8d8a8 add property: query prefix to search for specific properties
We want to be able to query the properties directly, like:

   notmuch count property:foo=bar

which should return a count of messages where the property with key
"foo" has value equal to "bar".
2016-09-21 18:14:25 -03:00
David Bremner
58fe8fce1d lib: iterator API for message properties
This is a thin wrapper around the string map iterator API just introduced.
2016-09-21 18:14:25 -03:00
David Bremner
b846bdb482 lib: extend private string map API with iterators
Support for prefix based iterators is perhaps overengineering, but I
wanted to mimic the existing database_config API.
2016-09-21 18:14:24 -03:00
David Bremner
b8bb6d7964 lib: basic message-property API
Initially, support get, set and removal of single key/value pair, as
well as removing all properties.
2016-09-21 18:14:24 -03:00
David Bremner
8b03ee1d5a lib: private string map (associative array) API
The choice of array implementation is deliberate, for future iterator support
2016-09-21 18:14:24 -03:00
David Bremner
4dfb69169e lib: read "property" terms from messages.
This is a first step towards providing an API to attach
arbitrary (key,value) pairs to messages and retrieve all of the values
for a given key.
2016-09-21 18:14:24 -03:00
David Bremner
59fed50a82 lib: update cached mtime in notmuch_directory_set_mtime
Without this change, the following code fails

  notmuch_directory_set_mtime(dir, 12345);
  assert(notmuch_directory_get_mtime(dir) == 12345);
2016-08-23 20:58:46 -03:00
David Bremner
9e177b236c lib: reword comment about XFOLDER: prefix
I believe the current one is misleading, because in my experiments
Xapian did not add : when prefix and term were both upper case. Indeed,
it's hard to see how it could, because prefixes are added at a layer
above Xapian in our code. See _notmuch_message_add_term for an example.

Also try to explain why this is a good idea.  As far as I can ascertain,
this is more of an issue for a system trying to work with an unknown set
of prefixes. Since notmuch has a fixed set of prefixes, and we can
hopefully be trusted not to add XGOLD and XGOLDEN as prefixes, it is
harder for problems to arise.
2016-08-18 05:11:37 -03:00
David Bremner
293186d6c6 lib: provide _notmuch_database_log_append
_notmuch_database_log clears the log buffer each time. Rather than
introducing more complicated semantics about for this function, provide
a second function that does not clear the buffer. This is mainly a
convenience function for callers constructing complex or multi-line log
messages.

The changes to query.cc are to make sure that the common code path of
the new function is tested.
2016-08-09 09:34:11 +09:00
David Bremner
3a45d29ed4 lib: add built_with handling for XAPIAN_DB_RETRY_LOCK
This support will be present only if the appropriate version of xapian
is available _and_ the user did not disable the feature when
building. So there really needs to be some way for the user to check.
2016-06-29 09:05:49 +02:00
Istvan Marko
9b60dc3cd9 Use the Xapian::DB_RETRY_LOCK flag when available
Xapian 1.3 has introduced the DB_RETRY_LOCK flag (Xapian bug
275). Detect it in configure and optionally use it. With this flag
commands that need the write lock will wait for their turn instead of
aborting when it's not immediately available.

Amended by db: allow disabling in configure
2016-06-29 09:03:34 +02:00
David Bremner
38f0d44a82 doc: forbid further operations on a closed database
We could add many null pointer checks, but currently I don't see a use
case that justifies it.
2016-06-28 23:20:38 +02:00
David Bremner
44cfa90bdc lib: fix definition of LIBNOTMUCH_CHECK_VERSION
Fix bug reported in id:20160606124522.g2y2eazhhrwjsa4h@flatcap.org

Although the C99 standard 6.10 is a little non-obvious on this point,
the docs for e.g. gcc are unambiguous. And indeed in practice with the
extra space, this code fails

#include <stdio.h>
#define foo (x) (x+1)

int main(int argc, char **argv){
  printf("%d\n",foo(1));
}
2016-06-11 13:01:44 -03:00
David Bremner
4291f32680 lib: fix memory leak of field processor objects
The field processor objects need to be deallocated explicitly just like
the range processors (or a talloc destructor defined).
2016-06-10 09:20:22 -03:00
David Bremner
ba0b95f846 lib: document config metadata
This probably should have been part of 3458e3c89c, but I missed it.
2016-06-07 07:51:57 -03:00
Daniel Kahn Gillmor
6a833a6e83 Use https instead of http where possible
Many of the external links found in the notmuch source can be resolved
using https instead of http.  This changeset addresses as many as i
could find, without touching the e-mail corpus or expected outputs
found in tests.
2016-06-05 08:32:17 -03:00
Tomi Ollila
cf09631a45 lib: whitespace cleanup
Cleaned the following whitespace in lib/* files:

lib/index.cc:              1 line:  trailing whitespace
lib/database.cc            5 lines: 8 spaces at the beginning of line
lib/notmuch-private.h:     4 lines: 8 spaces at the beginning of line
lib/message.cc:            1 line:  trailing whitespace
lib/sha1.c:                1 line:  empty lines at the end of file
lib/query.cc:              2 lines: 8 spaces at the beginning of line
lib/gen-version-script.sh: 1 line:  trailing whitespace
2016-06-05 08:23:28 -03:00
David Bremner
b9bf3f44ea lib: add support for named queries
This relies on the optional presense of xapian field processors, and the
library config API.
2016-05-25 07:40:44 -03:00
David Bremner
30caaf52b0 lib: make a global constant for query parser flags
It's already kindof gross that this is hardcoded in two different
places. We will also need these later in field processors calling back
into the query parser.
2016-05-25 07:40:44 -03:00
David Bremner
92e59568fa lib: config list iterators
Since xapian provides the ability to restrict the iterator to a given
prefix, we expose this ability to the user. Otherwise we mimic the other
iterator interfances in notmuch (e.g. tags.c).
2016-05-25 06:51:16 -03:00
David Bremner
3458e3c89c lib: provide config API
This is a thin wrapper around the Xapian metadata API. The job of this
layer is to keep the config key value pairs from colliding with other
metadata by transparently prefixing the keys, along with the usual glue
to provide a C interface.

The split of _get_config into two functions is to allow returning of the
return value with different memory ownership semantics.
2016-05-24 08:53:03 -03:00
David Bremner
792bea5aff lib/cli: add library API / CLI for compile time options
This is intentionally low tech; if we have more than two options it may
make sense to build up what infrastructure is provided.
2016-05-13 07:29:12 -03:00
David Bremner
bbf6069252 lib: optionally support single argument date: queries
This relies on the FieldProcessor API, which is only present in xapian
>= 1.3.
2016-05-08 08:17:07 -03:00
Daniel Kahn Gillmor
e366bb2227 complete ghost-on-removal-when-shared-thread-exists
To fully complete the ghost-on-removal-when-shared-thread-exists
proposal, we need to clear all ghost messages when the last active
message is removed from a thread.

Amended by db: Remove the last test of T530, as it no longer makes sense
if we are garbage collecting ghost messages.
2016-04-15 07:13:49 -03:00
Daniel Kahn Gillmor
1695415039 On deletion, replace with ghost when other active messages in thread
There is no need to add a ghost message upon deletion if there are no
other active messages in the thread.

Also, if the message being deleted was a ghost already, we can just go
ahead and delete it.
2016-04-15 07:07:23 -03:00
Daniel Kahn Gillmor
9eebae3da4 Introduce _notmuch_message_has_term()
It can be useful to easily tell if a given message has a given term
associated with it.
2016-04-15 07:07:23 -03:00
Daniel Kahn Gillmor
011fc41d4d Add internal functions to search for alternate doc types
Publicly we are only exposing the non-ghost documents (of "type"
"mail").  But internally we might want to inspect the ghost messages
as well.

This changeset adds two new private interfaces to queries to recover
information about alternate document types.
2016-04-15 07:07:23 -03:00
Daniel Kahn Gillmor
604d1e0977 fix thread breakage via ghost-on-removal
implement ghost-on-removal, the solution to T590-thread-breakage.sh
that just adds a ghost message after removing each message.

It leaks information about whether we've ever seen a given message id,
but it's a fairly simple implementation.

Note that _resolve_message_id_to_thread_id already introduces new
message_ids to the database, so i think just searching for a given
message ID may introduce the same metadata leakage.
2016-04-15 07:07:23 -03:00
Jani Nikula
54aeab1962 lib: clean up _notmuch_database_split_path
Make the logic it a bit easier to read. No functional changes.
2016-04-12 20:46:42 -03:00
Jani Nikula
a352d9ceaa lib: fix handling of one character long directory names at top level
The code to skip multiple slashes in _notmuch_database_split_path()
skips back one character too much. This is compensated by a +1 in the
length parameter to the strndup() call. Mostly this works fine, but if
the path is to a file under a top level directory with one character
long name, the directory part is mistaken to be part of the file name
(slash == path in code). The returned directory name will be the empty
string and the basename will be the full path, breaking the indexing
logic in notmuch new.

Fix the multiple slash skipping to keep the slash variable pointing at
the last slash, and adjust strndup() accordingly.

The bug was introduced in

commit e890b0cf40
Author: Carl Worth <cworth@cworth.org>
Date:   Sat Dec 19 13:20:26 2009 -0800

    database: Store the parent ID for each directory document.

just a little over two months after the initial commit in the Notmuch
code history, making this the longest living bug in Notmuch to date.
2016-04-12 20:40:19 -03:00
Tomi Ollila
342910a280 lib: NOTMUCH_DEPRECATED macro also for older compilers
Some compilers (older than gcc 4.5 and clang 2.9) do support
__attribute__ ((deprecated)) but not
__attribute__ ((deprecated("message"))).

Check if clang version is at least 3.0, or gcc version
is at least 4.5 to define NOTMUCH_DEPRECATED as the
latter variant above. Otherwise define NOTMUCH_DEPRECATED
as the former variant above.

For a bit simpler implementation clang 2.9 is not included
to use the newer variant. It is just one release, and the
older one works fine. Clang 3.0 was released around 2011-11
and gcc 5.1 2015-04-22 (therefore newer macro for gcc 4.5+)
2016-03-14 19:54:32 -03:00
Daniel Kahn Gillmor
07b6220a55 clean up stray apostrophe in comment
This is a nit-picky orthographical fix for an nit-picky ontological
comment.
2016-01-16 08:17:15 -04:00
Daniel Kahn Gillmor
e038b95ffe correct comment referring to notmuch_database_remove_message
notmuch_database_remove_message has no leading underscore in its name.
2016-01-16 08:16:51 -04:00
Steven Allen
c946356cdc forbid atomic transactions on writable, upgradable databases
We can't (but currently do) allow upgrades within transactions because
upgrades need their own transactions. We don't want to re-use the
current transaction because bailing out of an upgrade would mean loosing
all previous changes (because our "atomic" transactions don't commit
before hand). This gives us two options:

1. Fail at the beginning of upgrade (tell the user to end the
   transaction, upgrade, and start over).
2. Don't allow the user to start the transaction.

I went with the latter because:

1. There is no reason to call `begin_atomic` unless you intend to to
   write to the database and anyone intending to write to the database
   should upgrade it first.
2. This means that nothing inside an atomic transaction can ever fail
   with NOTMUCH_STATUS_UPGRADE_REQUIRED.
2015-11-23 08:15:37 -04:00
Jani Nikula
506b81679a lib: content disposition values are not case-sensitive
Per RFC 2183, the values for Content-Disposition values are not
case-sensitive. While at it, use the gmime function for getting at the
disposition string instead of referencing the field directly.

This fixes "attachment" tagging and filename term generation for
attachments while indexing.
2015-11-19 07:47:29 -04:00
Steven Allen
10e933a3bb Documentation: fix type name spelling 2015-10-27 08:07:31 -03:00
Jani Nikula
727fcd18c6 lib: add interface to delete directory documents
As mentioned in acd66cdec0 we don't have
an interface to delete directory documents, and they're left behind. Add
the interface.
2015-10-10 09:14:25 -03:00
David Bremner
7a20f26f91 lib: update doxygen comments to add @since for the new _st API
We should probably to this for all new functions introduced from now on.
2015-10-05 20:16:59 -03:00
David Bremner
378ba492a6 lib: migrate thread.cc to new query_search API
here we rely on thread_id_query being attached to the local talloc
context, so no new cleanup code is needed.
2015-10-05 19:53:53 -03:00
David Bremner
2501c2565c lib: migrate notmuch_database_upgrade to new query_search API
Here we depend on the error path cleaning up query
2015-10-05 19:53:11 -03:00
David Bremner
87ee9a53e3 lib: add versions of n_q_count_{message,threads} with status return
Although I think it's a pretty bad idea to continue using the old API,
this allows both a more gentle transition for clients of the library,
and allows us to break one monolithic change into a series
2015-10-05 19:44:07 -03:00
David Bremner
65a6b86873 lib: move query variable to function scope
This is a prelude to deallocating it (if necessary) on the error path.
2015-10-05 19:39:11 -03:00
Jani Nikula
23b8ed610a lib: add support for date:<expr>..! to mean date:<expr>..<expr>
It doesn't seem likely we can support simple date:<expr> expanding to
date:<expr>..<expr> any time soon. (This can be done with a future
version of Xapian, or with a custom query query parser.) In the mean
time, provide shorthand date:<expr>..! to mean the same. This is
useful, as the expansion takes place before interpetation, and we can
use, for example, date:yesterday..! to match from beginning of
yesterday to end of yesterday.

Idea from Mark Walters <markwalters1009@gmail.com>.
2015-09-25 21:55:24 -03:00
David Bremner
93ee4faa4d lib: constify arguments to notmuch_query_get_*
These functions are all just accessors, and it's pretty clear they don't
modify the query struct. This also fixes one warning I created when I
introduced status.c.
2015-09-23 08:58:19 -03:00
Jani Nikula
f460ad4e9a util: move strcase_equal and strcase_hash to util
For future use in both cli and lib.
2015-09-07 09:43:31 -03:00
David Bremner
bd5504ec10 lib: constify argument to notmuch_database_status_string
We don't modify the database struct, so no harm in committing to that.
2015-09-04 08:24:38 -03:00
David Bremner
110694b00b lib: note remaining uses of deprecated message search API
The two remaining cases in the lib seem to require more than a simple
replacement of the old call, with the new call plus a check of the
return value.
2015-09-04 08:08:18 -03:00
David Bremner
f16944c3b4 lib: remove use of notmuch_query_search_messages from query.cc
There is not too much point in worrying about the bad error reporting
here, because the count api is due for the same deprecation.
2015-09-04 08:06:08 -03:00
Austin Clements
cb08a2ee01 lib: Add "lastmod:" queries for filtering by last modification
The implementation is essentially the same as the date range search
prior to Jani's fancy date parser.
2015-08-14 18:23:49 +02:00
Austin Clements
98ee460eaa lib: API to retrieve database revision and UUID
This exposes the committed database revision to library users along
with a UUID that can be used to detect when revision numbers are no
longer comparable (e.g., because the database has been replaced).
2015-08-13 23:52:51 +02:00
Austin Clements
7f57b747b9 lib: Add per-message last modification tracking
This adds a new document value that stores the revision of the last
modification to message metadata, where the revision number increases
monotonically with each database commit.

An alternative would be to store the wall-clock time of the last
modification of each message.  In principle this is simpler and has
the advantage that any process can determine the current timestamp
without support from libnotmuch.  However, even assuming a computer's
clock never goes backward and ignoring clock skew in networked
environments, this has a fatal flaw.  Xapian uses (optimistic)
snapshot isolation, which means reads can be concurrent with writes.
Given this, consider the following time line with a write and two read
transactions:

   write  |-X-A--------------|
   read 1       |---B---|
   read 2                      |---|

The write transaction modifies message X and records the wall-clock
time of the modification at A.  The writer hangs around for a while
and later commits its change.  Read 1 is concurrent with the write, so
it doesn't see the change to X.  It does some query and records the
wall-clock time of its results at B.  Transaction read 2 later starts
after the write commits and queries for changes since wall-clock time
B (say the reads are performing an incremental backup).  Even though
read 1 could not see the change to X, read 2 is told (correctly) that
X has not changed since B, the time of the last read.  In fact, X
changed before wall-clock time A, but the change was not visible until
*after* wall-clock time B, so read 2 misses the change to X.

This is tricky to solve in full-blown snapshot isolation, but because
Xapian serializes writes, we can use a simple, monotonically
increasing database revision number.  Furthermore, maintaining this
revision number requires no more IO than a wall-clock time solution
because Xapian already maintains statistics on the upper (and lower)
bound of each value stream.
2015-08-13 23:52:51 +02:00
David Bremner
765556c1f1 build: extract library versions from notmuch.h
- Make lib/notmuch.h the canonical location for the library versioning
information.

- Since the release-check should never fail now, remove it to reduce
complexity.

- Make the version numbers in notmuch.h consistent with the (now
  deleted) ones in lib/Makefile.local
2015-08-10 13:53:55 +02:00
David Bremner
6b440a0adf lib: add public accessor for database from query
This is to make it easier for clients of the library to update to the
new error code returning versions of notmuch_query_search_messages
2015-08-04 09:11:34 +02:00
David Bremner
4fed7047b2 lib: deprecate notmuch_query_search_{threads, messages}
The CLI (and bindings) code should really be updated to use the new
status-code-returning versions. Here are some warnings to prod us (and
other clients) to do so.
2015-08-04 09:11:25 +02:00
David Bremner
7e2d0ef105 lib: define NOTMUCH_DEPRECATED macro, document its use.
This has been tested with gcc and clang.
2015-08-04 09:11:17 +02:00
Austin Clements
e6ad3a5dd4 lib: Only sync modified message documents
Previously, we updated the database copy of a message on every call to
_notmuch_message_sync, even if nothing had changed.  In particular,
this always happens on a thaw, so a freeze/thaw pair with no
modifications between still caused a database update.

We only modify message documents in a handful of places, so keep track
of whether the document has been modified and only sync it when
necessary.  This will be particularly important when we add message
revision tracking.
2015-08-04 08:54:46 +02:00
David Bremner
882ccb7e49 build: add "set -eu" to version script generation
It turns out that on certain systems like FreeBSD, c++filt is not
installed by default. It's basically OK if we fail the build in that
case, but what's really not OK is for the build to continue and
generate bad binaries.
2015-07-28 21:34:01 +02:00
David Bremner
53035dafe0 lib, ruby: make use of -Wl,--no-undefined configurable
In particular this is supposed to help build on systems (presumably
using a non-gnu ld) where this flag is not available.
2015-06-13 17:52:48 +02:00
David Bremner
32fd74b7aa lib: reject relative paths in n_d_{create,open}_verbose
There are many places in the notmuch code where the path is assumed to be absolute. If someone (TM) wants a project, one could remove these assumptions. In the mean time, prevent users from shooting themselves in the foot.

Update test suite mark tests for this error as no longer broken, and
also convert some tests that used relative paths for nonexistent
directories.
2015-06-12 07:34:50 +02:00
David Bremner
b59ad1a9cc lib: add NOTMUCH_STATUS_PATH_ERROR
The difference with FILE_ERROR is that this is for things that are
wrong with the path before looking at the disk.

Add some 3 tests; two broken as a reminder to actually use this new
code.
2015-06-12 07:34:47 +02:00
J. Lewis Muir
d08af93c65 cli: change "setup" to "set up" where used as a verb
The word "setup" is a noun, not a verb.  Change occurrences of "setup"
where used as a verb to "set up".
2015-05-31 19:14:42 +02:00
David Bremner
9d192da683 lib: eliminate fprintf from _notmuch_message_file_open
You may wonder why _notmuch_message_file_open_ctx has two parameters.
This is because we need sometime to use a ctx which is a
notmuch_message_t. While we could get the database from this, there is
no easy way in C to tell type we are getting.
2015-03-29 00:34:15 +01:00
David Bremner
736ac26407 lib: replace almost all fprintfs in library with _n_d_log
This is not supposed to change any functionality from an end user
point of view. Note that it will eliminate some output to stderr. The
query debugging output is left as is; it doesn't really fit with the
current primitive logging model. The remaining "bad" fprintf will need
an internal API change.
2015-03-29 00:34:15 +01:00
David Bremner
9b73a8bcc9 lib: add private function to extract the database for a message.
This is needed by logging in functions outside message.cc that take
only a notmuch_message_t object.
2015-03-29 00:34:15 +01:00
David Bremner
b53e1a2da7 lib: add a log function with output to a string in notmuch_database_t
In principle in the future this could do something fancier than
asprintf.
2015-03-29 00:34:15 +01:00
David Bremner
84d3b15d25 lib: add "verbose" versions of notmuch_database_{open,create}
The compatibility wrapper ensures that clients calling
notmuch_database_open will receive consistent output for now.

The changes to notmuch-{new,search} and test/symbol-test are just to
make the test suite pass.

The use of IGNORE_RESULT is justified by two things. 1) I don't know
what else to do.  2) asprintf guarantees the output string is NULL if
an error occurs, so at least we are not passing garbage back.
2015-03-29 00:34:15 +01:00
Jani Nikula
6d44e5ac47 lib: make notmuch_query_count_messages explicitely exact
The default is actually exact if no checkatleast parameter is
specified. This change makes that explicit, mainly for documentation,
but also to be safe in the unlikely event of a change of default.

[ commit message rewritten by db based on id:87lho0nlkk.fsf@nikula.org
]
2015-03-13 07:58:55 +01:00
David Bremner
90886f3640 lib: bump library minor version
This should have happened in commit 326e18856, but it didn't.
2015-03-07 20:15:06 +01:00
J. Lewis Muir
ee2d490280 lib: make notmuch shared library install_name be full path on Mac OS X
The install_name of libnotmuch.dylib on Mac OS X is what is written
into a program that links against it.  If it is just the name of the
shared library file, as opposed to the full path, the program won't be
able to find it when it runs and will abort.  Instead, the install_name
should be the full path to the shared library (in its final installed
location).

Why does Notmuch work without this patch when installed via Homebrew?
The answer is twofold.  One, /usr/local/lib is a special location in
which the dynamic linker will look by default to find shared libraries.
Homebrew highly recommends installing to /usr/local, and, assuming it
has been configured this way, the Notmuch library will end up installed
in /usr/local/lib, and the dynamic linker will find it.  Two, Homebrew
globally corrects all install names in dynamically shared libraries and
binaries for each package it installs.  So, even if the install names in
a package's binaries and libraries are incorrect, Homebrew corrects them
automatically, and no one ever knows.

Why does Notmuch work without this patch when installed via MacPorts?
The answer is that MacPorts applies a patch just like this patch to fix
the same problem.
2015-03-06 08:00:18 +01:00
David Bremner
326e188564 lib: bump SONAME minor version
This indicates upwardly compatible changes, namely adding new symbols.

Although we don't formally need to do this until the next release,
there is no hard in doing it now, as long as we don't bump the minor
version for every addition between now and the release.
2015-03-01 08:49:14 +01:00
David Bremner
7a0fc10367 lib: add new status reporting API for notmuch_query_search_{m,t}
This at least allows distinguishing between out of memory and Xapian
exceptions. Adding finer grained status codes would allow different
Xapian exceptions to be preserved.

Adding wrappers allows people to transition gradually to the new API,
at the cost of bloating the library API a bit.
2015-03-01 08:49:14 +01:00
Jani Nikula
08757767de lib: fix clang build warnings
Fix the following warning produced by clang 3.5.0:

lib/message.cc:899:4: warning: comparison of constant 64 with expression of type 'notmuch_message_flag_t' (aka '_notmuch_message_flag') is always true [-Wtautological-constant-out-of-range-compare]
        ! NOTMUCH_TEST_BIT (message->lazy_flags, flag))
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./lib/notmuch-private.h:70:6: note: expanded from macro 'NOTMUCH_TEST_BIT'
    (_NOTMUCH_VALID_BIT(bit) ? !!((val) & (1ull << (bit))) : 0)
     ^~~~~~~~~~~~~~~~~~~~~~~
./lib/notmuch-private.h:68:26: note: expanded from macro '_NOTMUCH_VALID_BIT'
    ((bit) >= 0 && (bit) < CHAR_BIT * sizeof (unsigned long long))
                   ~~~~~ ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2015-02-25 23:09:39 +01:00
Jani Nikula
41b870fba5 lib: abstract bit validity check in bit test/set/clear macros
Reduce duplication in the bit test/set/clear macros. No functional
changes.
2015-02-25 23:08:35 +01:00
Todd
b04bc967f9 Add indexing for the mimetype term
This adds the indexing support for the "mimetype:" term and removes
the broken test flag.  The indexing is probablistic in Xapian terms,
which gives a better experience to end users.  Standard content-types
of the form "foo/bar" are automatically interpreted as phrases in
Xapian due to the embedded slash.

Assume, separate messages with application/pdf and application/x-pdf
are indexed, then:

- mimetype:application/x-pdf will find only the application/x-pdf
- mimetype:application/pdf will find only the application/pdf
- mimetype:pdf will find both of the messages
2015-01-24 16:47:59 +01:00
Todd
0de999aab5 Add the NOTMUCH_FEATURE_INDEXED_MIMETYPES database feature
This feature will exist in all newly created databases, but there is
no upgrade provided for it.  If this flag exists, it indicates that
the database was created after the indexed MIME-types feature was
added.
2015-01-24 16:47:47 +01:00
Todd
ef5b4947d8 lib: Fix use after free
_thread_set_subject_from_message sometimes replaces the subject, making the
cur_subject point to free'd memory

==6550== ERROR: AddressSanitizer: heap-use-after-free on address 0x601a0000bec0 at pc 0x4464a4 bp 0x7fffa40be910 sp 0x7fffa40be908
READ of size 1 at 0x601a0000bec0 thread T0
    #0 0x4464a3 in _thread_add_matched_message /home/todd/.apps/notmuch/lib/thread.cc:369
    #1 0x443c2c in notmuch_threads_get /home/todd/.apps/notmuch/lib/query.cc:496
    #2 0x41d947 in do_search_threads /home/todd/.apps/notmuch/notmuch-search.c:131
    #3 0x40a3fe in main /home/todd/.apps/notmuch/notmuch.c:345
    #4 0x7f4e535b4ec4 in __libc_start_main /build/buildd/eglibc-2.19/csu/libc-start.c:287
    #5 0x40abe6 in _start ??:?
0x601a0000bec0 is located 96 bytes inside of 134-byte region [0x601a0000be60,0x601a0000bee6)
freed by thread T0 here:
    #0 0x7f4e54e6933a in __interceptor_free ??:?
    #1 0x7f4e54482fab in _talloc_free ??:?
previously allocated by thread T0 here:
    #0 0x7f4e54e6941a in malloc ??:?
    #1 0x7f4e54485b5d in talloc_strdup ??:?
2015-01-19 08:15:14 +01:00
Todd
0fc7884602 lib: Fix potential invalid read past an empty string
==22884== ERROR: AddressSanitizer: heap-buffer-overflow on address 0x601600008291 at pc 0x7ff6295680e5 bp 0x7fff4ab9aa40 sp 0x7fff4ab9aa08
READ of size 1 at 0x601600008291 thread T0
    #0 0x7ff6295680e4 in __interceptor_strcmp ??:?
    #1 0x44763b in _thread_add_message /home/todd/.apps/notmuch/lib/thread.cc:255
    #2 0x4459e8 in notmuch_threads_get /home/todd/.apps/notmuch/lib/query.cc:496
    #3 0x41e2a7 in do_search_threads /home/todd/.apps/notmuch/notmuch-search.c:131
    #4 0x40a408 in main /home/todd/.apps/notmuch/notmuch.c:345
    #5 0x7ff627cb9ec4 in __libc_start_main /build/buildd/eglibc-2.19/csu/libc-start.c:287
    #6 0x40abf3 in _start ??:?
0x601600008291 is located 0 bytes to the right of 97-byte region [0x601600008230,0x601600008291)
allocated by thread T0 here:
    #0 0x7ff62956e41a in malloc ??:?
    #1 0x7ff628b8ab5d in talloc_strdup ??:?
2015-01-19 08:15:05 +01:00
Jesse Rosenthal
67e368ac7b thread.cc: Avoid empty thread names if possible.
Currently the thread is named based on either the oldest or newest
matching message (depending on the search order). If this message has
an empty subject, though, the thread will show up with an empty
subject in the search results. (See the thread starting with
`id:1412371140-21051-1-git-send-email-david@tethera.net` for an
example.)

This changes the behavior so it will use a non-empty name for the
thread if possible. We name threads based on (a) non-empty matches for
the query, and (b) the search order. If the search order is
oldest-first (as in the default inbox) it chooses the oldest matching
non-empty message as the subject. If the search order is newest-first
it chooses the newest one.
2015-01-17 13:48:26 +01:00
David Bremner
105537a809 lib: convert two "iterator copy strings" into references.
Apparently this is a supported and even idiomatic way of keeping a
temporary object (e.g. like that returned from an operator
dereference) alive.
2015-01-02 17:18:42 +01:00
David Bremner
3d978a0d61 lib: another iterator-temporary/stale-pointer bug
Tamas Szakaly points out [1] that the bug fixed in 51b073c still
exists in at least one place. This change follows the suggestion of
[2] and creates a block scope temporary std::string to avoid the rules
of iterators temporaries.

[1]: id:20141226113755.GA64154@pamparam
[2]: id:20141226230655.GA41992@pamparam
2015-01-02 17:10:37 +01:00
Jani Nikula
c82a1745ac lib: drop the deprecation message for single-message mbox files
We generally do not support mbox files, but for historical reasons
we've supported single-message mbox files, with a deprecation
message. We've tried dropping the support altogether, but backed out
of it because we'd need to stop indexing them, while keeping support
for previously indexed files. This would be more complicated than
simply supporting single-message mbox files. Therefore, drop the
deprecation message, and just silently accept single-message mboxes.
2015-01-01 16:47:47 +01:00
Jesse Rosenthal
4135a1f8da lib: Use email address instead of empty real name.
Currently, if a From-header is of the form:

    "" <address@example.com>

the empty string will be treated as a valid real-name, and the entry
in the search results will be empty.

The new behavior here is that we treat an empty real-name field as if
it were null, so that the email address will be used in the search
results instead.

Signed-off-by: Jesse Rosenthal <jrosenthal@jhu.edu>
2014-12-07 13:36:08 +01:00
David Bremner
74bd4ea2ed lib: bump LIBNOTMUCH_MAJOR_VERSION to 4
This should have been done at the same time as the SONAME bump.
2014-11-06 00:10:03 +01:00
Austin Clements
70f15b37fb lib: Remove unnecessary thread linking steps when using ghost messages
Previously, it was necessary to link new messages to children to work
around some (though not all) problems with the old metadata-based
approach to stored thread IDs.  With ghost messages, this is no longer
necessary, so don't bother with child linking when ghost messages are
in use.
2014-10-25 19:46:19 +02:00
Austin Clements
ee476f1e76 lib: Enable ghost messages feature
This fixes the broken thread order test.
2014-10-25 19:31:27 +02:00
Austin Clements
d1e8c80b72 lib: Implement upgrade to ghost messages feature
Somehow this is the first upgrade pass that actually does *any* error
checking, so this also adds the bit of necessary infrastructure to
handle that.
2014-10-25 19:30:08 +02:00
Austin Clements
58a4277d3b lib: Implement ghost-based thread linking
This updates the thread linking code to use ghost messages instead of
user metadata to link messages into threads.

In contrast with the old approach, this is actually correct.
Previously, thread merging updated only the thread IDs of message
documents, not thread IDs stored in user metadata.  As originally
diagnosed by Mark Walters [1] and as demonstrated by the broken
T260-thread-order test, this can cause notmuch to fail to link
messages even though they're in the same thread.  In principle the old
approach could have been fixed by updating the user metadata thread
IDs as well, but these are not indexed and hence this would have
required a full scan of all stored thread IDs.  Ghost messages solve
this problem naturally by reusing the exact same thread ID and message
ID representation and indexing as regular messages.

Furthermore, thanks to this greater symmetry, ghost messages are also
algorithmically simpler.  We continue to support the old user metadata
format, so this patch can't delete any code, but when we do remove
support for the old format, several functions can simply be deleted.

[1] id:8738h7kv2q.fsf@qmul.ac.uk
2014-10-25 19:27:07 +02:00
Austin Clements
bc9c50602d lib: Internal support for querying and creating ghost messages
This updates the message abstraction to support ghost messages: it
adds a message flag that distinguishes regular messages from ghost
messages, and an internal function for initializing a newly created
(blank) message as a ghost message.
2014-10-25 19:26:54 +02:00
Austin Clements
d99491f274 lib: Introduce macros for bit operations
These macros help clarify basic bit-twiddling code and are written to
be robust against C undefined behavior of shift operators.
2014-10-25 19:26:43 +02:00
Austin Clements
d9f5da00bb lib: Update database schema doc for ghost messages
This describes the structure of ghost mail documents.  Ghost messages
are not yet implemented.
2014-10-25 19:26:03 +02:00
Austin Clements
1cdb96d3c4 lib: Add a ghost messages database feature
This will be implemented over the next several patches.  The feature
is not yet "enabled" (this does not add it to
NOTMUCH_FEATURES_CURRENT).
2014-10-25 19:25:54 +02:00
Austin Clements
7487e2e221 lib: Handle empty date value
In the interest of robustness, avoid undefined behavior of
sortable_unserialise if the date value is missing.  This shouldn't
happen now, but ghost messages will have blank date values.
2014-10-11 07:10:12 +02:00
Austin Clements
46b1b035a5 lib: Refactor _notmuch_database_link_message
This moves the code to retrieve and clear the metadata thread ID out
of _notmuch_database_link_message into its own function.  This will
simplify future changes.
2014-10-11 07:10:02 +02:00
Austin Clements
54ec8a0fd8 lib: Move message ID compression to _notmuch_message_create_for_message_id
Previously, this was performed by notmuch_database_add_message.  This
happens to be the only caller currently (which is why this was safe),
but we're about to introduce more callers, and it makes more sense to
put responsibility for ID compression in the lower-level function
rather than requiring each caller to handle it.
2014-10-11 07:09:54 +02:00
Austin Clements
cec601c4dd lib: Simplify close and codify aborting atomic section
In Xapian, closing a database implicitly aborts any outstanding
transaction and commits changes.  For historical reasons,
notmuch_database_close had grown to almost, but not quite duplicate
this behavior.  Before closing the database, it would explicitly (and
unnecessarily) commit it.  However, if there was an outstanding
transaction (ie atomic section), commit would throw a Xapian
exception, which notmuch_database_close would unnecessarily print to
stderr, even though notmuch_database_close would ultimately abort the
transaction anyway when it called close.

This patch simplifies notmuch_database_close to explicitly abort any
outstanding transaction and then just call Database::close.  This
works for both read-only and read/write databases, takes care of
committing changes, unifies the exception handling path, and codifies
aborting outstanding transactions.  This is currently the only way to
abort an atomic section (and may remain so, since it would be
difficult to roll back things we may have cached from rolled-back
modifications).
2014-10-03 08:58:58 +02:00
Jani Nikula
f42e2e43a0 lib: actually return failures from notmuch_message_tags_to_maildir_flags
The function takes great care to preserve the first error status it
encounters, yet fails to return that status to the caller. Fix it.
2014-09-24 20:19:34 +02:00
Peter Wang
6754ad9f9e lib: bump soname
Adding return values to notmuch_database_close and
notmuch_database_destroy may require bumping the soname.
2014-09-16 20:16:31 +02:00
Gaute Hope
2c9e120e0a notmuch_thread_get_authors: document match grouping with |
as stated in thread.cc:115

/* Construct an authors string from matched_authors_array and
 * authors_array. The string contains matched authors first, then
 * non-matched authors (with the two groups separated by '|'). Within
 * each group, authors are listed in date order. */

this is, however, not reflected in the public API documentation in
notmuch.h:970. This patch a paragraph explaining how | separates the
group of authors of messages matching the query and those of messages
that do not, but are still contained in the thread.
2014-09-13 08:43:35 +02:00
Austin Clements
cca05ac10e lib: Fix endless upgrade problem
48db8c8 introduced a disagreement between when
notmuch_database_needs_upgrade returned TRUE and when
notmuch_database_upgrade actually performed an upgrade.  As a result,
if a database had a version less than 3, but no new features were
required, notmuch new would call notmuch_database_upgrade to perform
an upgrade, but notmuch_database_upgrade would return immediately
without updating the database version.  Hence, the next notmuch new
would do the same, and so on.

Fix this by ensuring that the upgrade-required logic is identical
between the two.
2014-09-01 23:06:51 -07:00
Austin Clements
658a00e7c8 lib: Update doc of notmuch_database_{needs_upgrade,upgrade}
Clients are no longer required to call these functions after opening a
database in read/write mode (which is good, because almost none of
them do!).
2014-08-30 11:42:07 -07:00
Austin Clements
ec573cd54f lib: Return an error from operations that require an upgrade
Previously, there was no protection against a caller invoking an
operation on an old database version that would effectively corrupt
the database by treating it like a newer version.

According to notmuch.h, any caller that opens the database in
read/write mode is supposed to check if the database needs upgrading
and perform an upgrade if it does.  This would protect against this,
but nobody (even the CLI) actually does this.

However, with features, it's easy to protect against incompatible
operations on a fine-grained basis.  This lightweight change allows
callers to safely operate on old database versions, while preventing
specific operations that would corrupt the database with an
informative error message.
2014-08-30 11:39:41 -07:00
Austin Clements
5dbfed4a73 lib: Support empty header values in database
Commit 567bcbc2 introduced support for storing various headers in
document values.  However, doing so in a backwards-compatible way
meant that genuinely empty header values could not be distinguished
from the old behavior of not storing the headers at all, so these
required parsing the original message.

Now that we have database features, new databases can declare that all
messages have header values, so if we have this feature flag, we can
use the stored header value even if it's the empty string.

This requires slight cleanup to notmuch_message_get_header, since the
code previously couldn't distinguish between empty headers and headers
that are never stored in the database (previously this distinction
didn't matter).
2014-08-30 11:37:33 -07:00
Austin Clements
02fec226fc lib: Report progress for combined upgrade operation
Previously, some parts of upgrade didn't report progress and for
others it was possible for the progress meter to restart at 0 part way
through the upgrade because each stage was reported separately.

Fix this by computing the total amount of work that needs to be done
up-front and updating completed work monotonically.
2014-08-30 11:36:08 -07:00
Austin Clements
e0635bd003 lib: Reorganize upgrade around document types
Rather than potentially making multiple passes over the same type of
data in the database, reorganize upgrade around each type of data that
may be upgraded.  This eliminates code duplication, will make
multi-version upgrades faster, and will let us improve progress
reporting.
2014-08-30 11:24:11 -07:00
Austin Clements
48db8c8b60 lib: Use database features to drive upgrade
Previously, we had database version information hard-coded in the
upgrade code.  Slightly re-organize the upgrade process around the set
of new database features to be enabled by the upgrade.
2014-08-30 11:21:48 -07:00
Austin Clements
4a38588488 lib: Simplify upgrade code using a transaction
Previously, the upgrade was organized as two passes -- an upgrade
pass, and a separate cleanup pass -- so the database was always in a
valid state.  This change substantially simplifies this code by
performing the upgrade in a transaction and combining both passes in
to one.  This 1) eliminates a lot of duplicate code between the
passes, 2) speeds up the upgrade process, 3) makes progress reporting
more accurate, 4) eliminates the potential for stale data if the
upgrade is interrupted during the cleanup pass, and 5) makes it easier
to reason about the safety of the upgrade code.
2014-08-30 10:45:36 -07:00
Austin Clements
8363c90531 lib: Database version 3: Introduce fine-grained "features"
Previously, our database schema was versioned by a single number.
Each database schema change had to occur "atomically" in Notmuch's
development history: before some commit, Notmuch used version N, after
that commit, it used version N+1.  Hence, each new schema version
could introduce only one change, the task of developing a schema
change fell on a single person, and it all had to happen and be
perfect in a single commit series.  This made introducing a new schema
version hard.  We've seen only two schema changes in the history of
Notmuch.

This commit introduces database schema version 3; hopefully the last
schema version we'll need for a while.  With this version, we switch
from a single version number to "features": a set of named,
independent aspects of the database schema.

Features should make backwards compatibility easier.  For many things,
it should be easy to support databases both with and without a
feature, which will allow us to make upgrades optional and will enable
"unstable" features that can be developed and tested over time.

Features also make forwards compatibility easier.  The features
recorded in a database include "compatibility flags," which can
indicate to an older version of Notmuch when it must support a given
feature to open the database for read or for write.  This lets us
replace the old vague "I don't recognize this version, so something
might go wrong, but I promise to try my best" warnings upon opening a
database with an unknown version with precise errors.  If a database
is safe to open for read/write despite unknown features, an older
version will know that and issue no message at all.  If the database
is not safe to open for read/write because of unknown features, an
older version will know that, too, and can tell the user exactly which
required features it lacks support for.
2014-08-30 10:42:08 -07:00
Michal Sojka
028c56061e Make parsing of References and In-Reply-To header less error prone
According to RFC2822 References and In-Reply-To headers are supposed
to contain one or more Message-IDs, however older RFC822 allowed
almost any content. When both References and In-Reply-To headers ends
with something else that a Message-ID (see e.g. [1]), the thread
structure presented by notmuch is incorrect. The reason is that
notmuch treats this case as if the email contained no "replyto"
information (see _notmuch_database_link_message_to_parents).

This patch changes the parse_references() function to return the last
valid Message-ID encountered rather than NULL resulting from the last
hunk of text not being the Message-ID.

[1] https://lkml.org/lkml/headers/2014/5/19/864
2014-08-16 17:45:16 -07:00
Austin Clements
0c1292051e lib: Improve documentation of _notmuch_message_create_for_message_id
Clarify the state of the returned message when
_notmuch_message_create_for_message_id returns
NOTMUCH_PRIVATE_STATUS_NO_DOCUMENT_FOUND.
2014-08-05 08:14:15 -03:00
Austin Clements
1d652c8719 lib: Fix slight misinformation in the database schema doc
The database schema documentation made it sound like each mail
document had exactly one on-disk message file, which hasn't been true
for a long time.
2014-08-04 18:58:11 -03:00
Austin Clements
30de720ba0 lib: Invalidate message metadata in _notmuch_message_gen_terms
Previously, we invalidated stored message metadata in
_notmuch_message_add_term and _notmuch_message_remove_term, but not in
_notmuch_message_gen_terms.  This doesn't currently result in any bugs
because of our limited uses of _notmuch_message_gen_terms, but it may
could cause trouble in the future.
2014-08-04 18:57:55 -03:00
Charles Celerier
df8885f62c lib: Start all function names in notmuch-private.h with
As noted in devel/STYLE, every private library function should start
with _notmuch. This patch corrects function naming that did not adhere
to this style in lib/notmuch-private.h. In particular, the old function
names that now begin with _notmuch are

    notmuch_sha1_of_file
    notmuch_sha1_of_string
    notmuch_message_file_close
    notmuch_message_file_get_header
    notmuch_message_file_open
    notmuch_message_get_author
    notmuch_message_set_author

Signed-off-by: Charles Celerier <cceleri@cs.stanford.edu>
2014-07-13 12:25:29 -03:00
Jani Nikula
ab24e883b0 lib: add return status to database close and destroy
notmuch_database_close may fail in Xapian ->flush() or ->close(), so
report the status. Similarly for notmuch_database_destroy which calls
close.

This is required for notmuch insert to report error status if message
indexing failed.
2014-07-09 20:29:36 -03:00
Austin Clements
dc64ab6720 lib: Separate all phrases indexed by _notmuch_message_gen_terms
This adds a 100 termpos gap between all phrases indexed by
_notmuch_message_gen_terms.  This fixes a bug where terms from the end
of one header and the beginning of another header could match together
in a single phrase and a separate bug where term positions of
un-prefixed terms overlapped.

This fix only affects newly indexed messages.  Messages that are
already indexed won't benefit from this fix without re-indexing, but
the fix won't make things any worse for existing messages.
2014-06-18 18:03:18 -03:00
Austin Clements
44327ca86d lib: Index name and address of from/to headers as a phrase
Previously, we indexed the name and address parts of from/to headers
with two calls to _notmuch_message_gen_terms.  In general, this
indicates that these parts are separate phrases.  However, because of
an implementation quirk, the two calls to _notmuch_message_gen_terms
generated adjacent term positions for the prefixed terms, which
happens to be the right thing to do in this case, but the wrong thing
to do for all other calls.  Furthermore, _notmuch_message_gen_terms
produced potentially overlapping term positions for the un-prefixed
copies of the terms, which is simply wrong.

This change indexes both the name and address in a single call to
_notmuch_message_gen_terms, indicating that they should be part of a
single phrase.  This masks the problem with the un-prefixed terms
(fixing the two known-broken tests) and puts us in a position to fix
the unintentionally phrases generated by other calls to
_notmuch_message_gen_terms.
2014-06-18 17:55:14 -03:00
Jani Nikula
0cc0144875 lib: resurrect support for single-message mbox files
This is effectively a revert of

commit 6812136bf5
Author: Jani Nikula <jani@nikula.org>
Date:   Mon Mar 31 00:21:48 2014 +0300

    lib: drop support for single-message mbox files

The intention was to drop support for indexing new single-message mbox
files (and whether that was a good idea in the first place is
arguable). However this inadvertently broke support for reading
headers from previously indexed single-message mbox files, which is
far worse.

Distinguishing between the two cases would require more code than
simply bringing back support for single-message mbox files.
2014-06-13 22:59:04 -03:00
David Bremner
3242e29e57 build: add canonicalize_file_name to symbols exported from libnotmuch.so
This is needed for our compat version of canonicalize_file_name to be used.
2014-04-19 05:58:34 +09:00
Jani Nikula
473930bb6f lib: replace the header parser with gmime
The notmuch library includes a full blown message header parser. Yet
the same message headers are parsed by gmime during indexing. Switch
to gmime parsing completely.

These are the main changes:

* Gmime stops header parsing at the first invalid header, and presumes
  the message body starts from there. The current parser is quite
  liberal in accepting broken headers. The change means we will be
  much pickier about accepting invalid messages.

* The current parser converts tabs used in header folding to
  spaces. Gmime preserve the tabs. Due to a broken python library used
  in mailman, there are plenty of mailing lists that produce headers
  with tabs in header folding, and we'll see plenty of tabs. (This
  change has been mitigated in preparatory patches.)

* For pure header parsing, the current parser is likely faster than
  gmime, which parses the whole message rather than just the
  headers. Since we parse the message and its headers using gmime for
  indexing anyway, this avoids and extra header parsing round when
  adding new messages. In case of duplicate messages, we'll end up
  parsing the full message although just headers would be
  sufficient. All in all this should still speed up 'notmuch new'.

* Calls to notmuch_message_get_header() may be slightly slower than
  previously for headers that are not indexed in the database, due to
  parsing of the whole message. Within the notmuch code base, notmuch
  reply is the only such user.
2014-04-05 12:53:04 -03:00
Jani Nikula
6812136bf5 lib: drop support for single-message mbox files
We've supported mbox files containing a single message for historical
reasons, but the support has been deprecated, with a warning message
while indexing, since Notmuch 0.15. Finally drop the support, and
consider all mbox files non-email.
2014-04-05 12:52:42 -03:00
Jani Nikula
1fa8e40561 lib: make folder: prefix literal
In xapian terms, convert folder: prefix from probabilistic to boolean
prefix, matching the paths, relative from the maildir root, of the
message files, ignoring the maildir new and cur leaf directories.

folder:foo matches all message files in foo, foo/new, and foo/cur.

folder:foo/new does *not* match message files in foo/new.

folder:"" matches all message files in the top level maildir and its
new and cur subdirectories.

This change constitutes a database change: bump the database version
and add database upgrade support for folder: terms. The upgrade also
adds path: terms.

Finally, fix the folder search test for literal folder: search, as
some of the folder: matching capabilities are lost in the
probabilistic to boolean prefix change.
2014-03-11 19:51:22 -03:00
Jani Nikula
59823f9642 lib: add support for path: prefix searches
The path: prefix is a literal boolean prefix matching the paths,
relative from the maildir root, of the message files.

path:foo matches all message files in foo (but not in foo/new or
foo/cur).

path:foo/new matches all message files in foo/new.

path:"" matches all message files in the top level maildir.

path:foo/** matches all message files in foo and recursively in all
subdirectories of foo.

path:** matches all message files recursively, i.e. all messages.
2014-03-11 19:51:22 -03:00
Jani Nikula
4d150eba67 lib: refactor folder term update after filename removal
Abstract some blocks of code for reuse. No functional changes.
2014-03-11 19:51:22 -03:00
Gaute Hope
91aede05a3 doc: notmuch_result_move_to_next -> notmuch_tags_move_to_next
Fix typo in docs.
2014-02-13 08:43:34 -04:00
David Bremner
3b74537f49 lib: update documentation for notmuch_database_get_directory
Clarify that using the directory after destroying the corresponding
database is not permitted.

This is implicit in the description of notmuch_database_destroy, but
it doesn't hurt to be explicit, and we do express similar "ownership"
relationships at other places in the docs.
2014-01-26 09:36:38 -04:00
David Bremner
29f1252126 lib: make notmuch_threads_valid return FALSE when passed NULL
Without this patch, the example code in the header docs crashes for certain
invalid queries (see id:871u00oimv.fsf@approx.mit.edu)
2014-01-24 20:20:54 -04:00
Tomi Valkeinen
075d53dde5 lib: fix error handling
Currently if a Xapian exception happens in notmuch_message_get_header,
the exception is not caught leading to crash. In
notmuch_message_get_date the exception is caught, but an internal error
is raised, again leading to crash.

This patch fixes the error handling by making both functions catch the
Xapian exceptions, print an error and return NULL or 0.

The 'notmuch->exception_reported' is also set, as is done elsewhere,
even if I don't really get the idea of that field.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@iki.fi>
2014-01-18 14:47:35 -04:00
Jani Nikula
f94834407b lib: fix clang compiler warning
With some combination of clang and talloc, not using the return value
of talloc_steal() produces a warning. Ignore it, as talloc_steal() has
no failure modes per documentation.
2014-01-18 14:39:51 -04:00
Jani Nikula
b65ca8e0ba lib: modify notmuch.h for automatic document generation
Minimal changes to produce a sensible result.
2014-01-05 09:05:00 -04:00
Austin Clements
3d4b0930bd lib: Bump library version from 3.0.0 to 3.1.0
This version of the library introduces LIBNOTMUCH_CHECK_VERSION and
the *_VERSION macros.  Bumping the version number is also necessary to
make the comment on LIBNOTMUCH_CHECK_VERSION no longer a lie.
2013-12-07 10:31:19 +08:00
Austin Clements
8fb16e277e lib: Replace NOTMUCH_*_VERSION with LIBNOTMUCH_*_VERSION
This makes it clear that these macros refer to the *library* version,
and not to the notmuch application-level release.  Since there are no
consumers of these macros yet, this is now or never.
2013-12-07 10:31:11 +08:00
Austin Clements
4b14ccf0d6 lib: Make VERSION macros agree with soname version
We have two distinct "library version" numbers: the soname version and
the version macros.  We need both for different reasons: the version
macros enable easy compile-time version detection (and conditional
compilation), while the soname version enables runtime version
detection (which includes the version checking done by things like the
Python bindings).

However, currently, these two version numbers are different, which is
unnecessary and can lead to confusion (especially in things like
Debian, which include the soname version in the package name).  This
patch makes them the same by bumping the version macros up to agree
with the soname version.

(We should probably keep the version number in just one place so they
can't get out of sync, but that can be done in another patch.)
2013-12-07 10:31:04 +08:00
David Bremner
b9f0e6923d util: detect byte order
Unfortunately old versions of GCC and clang do not provide byte order
macros, so we re-invent them.

If UTIL_BYTE_ORDER is not defined or defined to 0, we fall back to
macros supported by recent versions of GCC and clang
2013-11-27 07:43:29 -04:00
David Bremner
20adfe39d7 lib: fix byte order test in libsha1.c
Previously PLATFORM_BYTE_ORDER and IS_LITTLE_ENDIAN were not defined,
so the little endian code was always compiled in.

This will have the effect that the "SHA1s" on big endian architectures
will change (i.e. become actual sha1s). So someone re-indexing their
database could conceivable lose tags on messages without a message-id
header.
2013-11-25 07:55:54 -04:00
Tomi Ollila
2fd7ef64ba compact: improve error messages on failures after compaction
The error messages written during the steps replacing old
database with new now includes relevant paths and strerror.
2013-11-19 20:15:02 -04:00
Tomi Ollila
6452ae0fcb compact: unconditionally remove old wip database compact directory
In case previous notmuch compact has been interrupted there is old
work-in-progress database compact directory partially filled. Remove
it just before starting to fill the directory with new files.
2013-11-19 20:14:28 -04:00
Tomi Ollila
cb6cc296e2 compact: preserve backup database until compacted database is in place
It is less error prone and window of failure opportunity is smaller
if the old (backup) database is always renamed (instead of sometimes
rmtree'd) before new (compacted) database is put into its place.
Finally rmtree() old database in case old database backup is not kept.
2013-11-19 20:13:25 -04:00
Tomi Ollila
19a89753ca compact: catch Xapian::Error consistently
catch Xapian::Error in compact code in lib/database.cc to be consistent
with other code in addition to not making software crash on uncaught
other Xapian error.
2013-11-17 20:25:43 -04:00
Tomi Ollila
4d5986e8ad compact: tidy formatting
Notmuch compact code whitespace changes to match devel/STYLE.
2013-11-17 20:25:25 -04:00
Austin Clements
215a8dc29c lib: Document extent of some return values
This documents the extent of the notmuch_messages_t* pointers returned
by notmuch_thread_get_toplevel_messages and
notmuch_thread_get_messages.
2013-11-08 20:14:08 -04:00
Jani Nikula
060990db14 lib: add library version check macro
There have been some plans for making build incompatible changes to
the library API. This is inconvenient, but it is much more so without
a way to easily conditional build against multiple versions of
notmuch.

The macro has been lifted from glib.
2013-11-07 19:15:06 -04:00
Jani Nikula
00d2ac2b41 lib: use the compaction backup path provided by the caller
The extra path component added by the lib is a magic value that the
caller just has to know. This is demonstrated by the current code,
which indeed has "xapian.old" both sides of the interface. Use the
backup path provided by the lib caller verbatim, without adding
anything to it.
2013-11-07 06:51:16 -04:00
David Bremner
20abbe89a3 lib: update documentation of callback functions for database_compact and database_upgrade.
Compact was missing callback documentation entirely, and upgrade did not discuss the
closure parameter.
2013-11-07 06:50:50 -04:00
Jani Nikula
180dba66e4 lib: add closure parameter to compact status update callback
This provides much more flexibility for the caller.
2013-11-07 06:46:42 -04:00