Commit graph

779 commits

Author SHA1 Message Date
Daniel Kahn Gillmor
d0da7a0a1c config: define new option index.try_decrypt
By default, notmuch won't try to decrypt on indexing.  With this
patch, we make it possible to indicate a per-database preference using
the config variable "index.try_decrypt", which by default will be
false.

At indexing time, the database needs some way to know its internal
defaults for how to index encrypted parts.  It shouldn't be contingent
on an external config file (since that can't be retrieved from the
database object itself), so we store it in the database.

This behaves similarly to the query.* configurations, which are also
stored in the database itself, so we're not introducing any new
dependencies by requiring that it be stored in the database.
2017-10-21 19:54:33 -03:00
Daniel Kahn Gillmor
4dfcc8c9b2 crypto: index encrypted parts when indexopts try_decrypt is set.
If we see index options that ask us to decrypt when indexing a
message, and we encounter an encrypted part, we'll try to descend into
it.

If we can decrypt, we add the property index.decryption=success.

If we can't decrypt (or recognize the encrypted type of mail), we add
the property index.decryption=failure.

Note that a single message may have both values of the
"index.decryption" property: "success" and "failure".  For example,
consider a message that includes multiple layers of encryption.  If we
manage to decrypt the outer layer ("index.decryption=success"), but
fail on the inner layer ("index.decryption=failure").

Because of the property name, this will be automatically cleared (and
possibly re-set) during re-indexing.  This means it will subsequently
correspond to the actual semantics of the stored index.
2017-10-21 19:53:19 -03:00
Daniel Kahn Gillmor
0bb05ff693 reindex: drop all properties named with prefix "index."
This allows us to create new properties that will be automatically set
during indexing, and cleared during re-indexing, just by choice of
property name.
2017-10-21 19:53:08 -03:00
Daniel Kahn Gillmor
20ff9de24d index: implement notmuch_indexopts_t with try_decrypt
This is currently mostly a wrapper around _notmuch_crypto_t that keeps
its internals private and doesn't expose any of the GMime API.
However, non-crypto indexing options might also be added later
(e.g. filters or other transformations).
2017-10-21 19:52:47 -03:00
Daniel Kahn Gillmor
0b9e1a2472 properties: add notmuch_message_remove_all_properties_with_prefix()
Subsequent patches will introduce a convention that properties whose
name starts with "index." will be stripped (and possibly re-added)
during re-indexing.  This patch lays the groundwork for doing that.
2017-10-20 07:58:43 -03:00
Daniel Kahn Gillmor
a18bbf7f15 crypto: make shared crypto code behave library-like
If we're going to reuse the crypto code across both the library and
the client, then it needs to report error states properly and not
write to stderr.
2017-10-20 07:58:20 -03:00
Jani Nikula
008a5e92eb lib: convert notmuch_bool_t to stdbool internally
C99 stdbool turned 18 this year. There really is no reason to use our
own, except in the library interface for backward
compatibility. Convert the lib internally to stdbool.
2017-10-09 22:27:16 -03:00
Daniel Kahn Gillmor
e3a6368e8d fix reference to notmuch_message_get_properties 2017-09-24 09:15:24 -03:00
Daniel Kahn Gillmor
f4ac5ecd5c lib: index the content-type of the parts of encrypted messages
This is a logical followup to "lib: index the content type of
signature parts", which will make it easier to record the message
structure of all messages.
2017-09-17 20:01:19 -03:00
Jani Nikula
55c047ee0b lib: index the content type of signature parts
It's useful (*) to be able to easily find messages with certain types
of signatures. Having the mimetype: prefix searches fail for some
content types is also genuinely surprising (*). Index the content type
of signature parts.

While at it, switch to the gmime convenience constants for content and
signature part indexes.

*) At least for developers of email software!
2017-09-17 20:01:00 -03:00
Jani Nikula
930d0aefb1 lib: abstract content type indexing
Make the follow-up change of indexing signature content types
easier. No functional changes.
2017-09-17 20:00:32 -03:00
Jani Nikula
eb29e26a99 build: fix out-of-tree builds, again
Broken, again, by yours truly in bc11759dd1 ("build: switch to
hiding libnotmuch symbols by default"). Reference notmuch.sym via
$(srctree).
2017-09-13 08:48:17 -03:00
Daniel Kahn Gillmor
3445385f95 fix documentation bug (leading quotes break documentation) 2017-09-05 21:54:46 -03:00
David Bremner
debfae20db lib: enforce that n_message_reindex takes headers from first file
This is still a bit stopgap to be only choosing one set of headers,
but this seems like a more defensible set of headers to choose.
2017-09-05 21:51:57 -03:00
David Bremner
0260ee371e lib&cli: use g_object_new instead of g_object_newv
'g_object_newv' is deprecated, and prints annoying warnings. The
warnings suggest using 'g_object_new_with_properties', but that's only
available since glib 2.55 (i.e. a month ago as of this writing).
Since we don't actuall pass any properties, it seems we can just call
'g_object_new'.
2017-09-04 08:04:44 -03:00
David Bremner
0a40ea4b48 lib: add notmuch_message_has_maildir_flag
I considered a higher level interface where the caller passes a tag
name rather than a flag character, but the role of the "unread" tag is
particularly confusing with such an interface.
2017-08-29 21:56:21 -03:00
David Bremner
8a8fb39b0c lib/message: split n_m_maildir_flags_tags, store maildir flags
In a future commit this will allow querying maildir flags seperately
from tags to allow resolving certain conflicts.
2017-08-29 21:51:10 -03:00
Daniel Kahn Gillmor
eb232ee0ab reindex: drop notmuch_param_t, use notmuch_indexopts_t instead
There are at least three places in notmuch that can trigger an
indexing action:

 * notmuch new
 * notmuch insert
 * notmuch reindex

I have plans to add some indexing options (e.g. indexing the cleartext
of encrypted parts, external filters, automated property injection)
that should properly be available in all places where indexing
happens.

I also want those indexing options to be exposed by (and constrained
by) the libnotmuch C API.

This isn't yet an API break because we've never made a release with
notmuch_param_t.

These indexing options are relevant in the listed places (and in the
libnotmuch analogues), but they aren't relevant in the other kinds of
functionality that notmuch offers (e.g. dump/restore, tagging, search,
show, reply).

So i think a generic "param" object isn't well-suited for this case.
In particular:

 * a param object sounds like it could contain parameters for some
   other (non-indexing) operation.  This sounds confusing -- why would
   i pass non-indexing parameters to a function that only does
   indexing?

 * bremner suggests online a generic param object would actually be
   passed as a list of param objects, argv-style.  In this case (at
   least in the obvious argv implementation), the params might be some
   sort of generic string.  This introduces a problem where the API of
   the library doesn't grow as new options are added, which means that
   when code outside the library tries to use a feature, it first has
   to test for it, and have code to handle it not being available.
   The indexopts approach proposed here instead makes it clear at
   compile time and at dynamic link time that there is an explicit
   dependency on that feature, which allows automated tools to keep
   track of what's needed and keeps the actual code simple.

My proposal adds the notmuch_indexopts_t as an opaque struct, so that
we can extend the list of options without causing ABI breakage.

The cost of this proposal appears to be that the "boilerplate" API
increases a little bit, with a generic constructor and destructor
function for the indexopts struct.

More patches will follow that make use of this indexopts approach.
2017-08-23 07:55:12 -03:00
Daniel Kahn Gillmor
b10ce6bc23 database: add n_d_index_file (deprecates n_d_add_message)
We need a way to pass parameters to the indexing functionality on the
first index, not just on reindexing.  The obvious place is in
notmuch_database_add_message.  But since modifying the argument list
would break both API and ABI, we needed a new name.

I considered notmuch_database_add_message_with_params(), but the
functionality we're talking about doesn't always add a message.  It
tries to index a specific file, possibly adding a message, but
possibly doing other things, like adding terms to an existing message,
or failing to deal with message objects entirely (e.g. because the
file didn't contain a message).

So i chose the function name notmuch_database_index_file.

I confess i'm a little concerned about confusing future notmuch
developers with the new name, since we already have a private
_notmuch_message_index_file function, and the two do rather different
things.  But i think the added clarity for people linking against the
future libnotmuch and the capacity for using index parameters makes
this a worthwhile tradeoff.  (that said, if anyone has another name
that they strongly prefer, i'd be happy to go with it)

This changeset also adjusts the tests so that we test whether the new,
preferred function returns bad values (since the deprecated function
just calls the new one).

We can keep the deprecated n_d_add_message function around as long as
we like, but at the next place where we're forced to break API or ABI
we can probably choose to drop the name relatively safely.

NOTE: there is probably more cleanup to do in the ruby and go bindings
to complete the deprecation directly.  I don't know those languages
well enough to attempt a fix; i don't know how to test them; and i
don't know the culture around those languages about API additions or
deprecations.
2017-08-23 07:38:37 -03:00
Yuri Volchkov
cec4a87539 database: move striping of trailing '/' into helper function
Stripping trailing character is not that uncommon
operation. Particularly, the next patch has to perform it as
well. Lets move it to the separate function to avoid code duplication.

Also the new function has a little improvement: if the character to
strip is repeated several times in the end of a string, function
strips them all.

Signed-off-by: Yuri Volchkov <yuri.volchkov@gmail.com>
2017-08-22 18:47:51 -03:00
Daniel Kahn Gillmor
55f9f6505e lib: clarify description of notmuch_database_add_message
Since we're accumulating the index when we add a new file to the
message, the semantics have slightly changed.  This tries to align the
documentation with the actual functionality.
2017-08-20 08:33:46 -03:00
Daniel Kahn Gillmor
5b93fa6e70 lib: add notmuch_message_reindex
This new function asks the database to reindex a given message.
The parameter `indexopts` is currently ignored, but is intended to
provide an extensible API to support e.g. changing the encryption or
filtering status (e.g. whether and how certain non-plaintext parts are
indexed).
2017-08-01 21:17:47 -04:00
David Bremner
34d7753992 lib: add _notmuch_message_remove_indexed_terms
Testing will be provided via use in notmuch_message_reindex
2017-08-01 21:17:47 -04:00
David Bremner
50340bcb78 lib: add notmuch_thread_get_total_files
This is relatively inexpensive in terms of run time and implementation
cost as we are already traversing the list of messages in a thread.
2017-08-01 21:17:47 -04:00
David Bremner
8a8e2b11c2 lib: add notmuch_message_count_files
This operation is relatively inexpensive, as the needed metadata is
already computed by our lazy metadata fetching. The goal is to support
better UI for messages with multipile files.
2017-08-01 21:17:47 -04:00
David Bremner
411675a6ce lib: index message files with duplicate message-ids
The corresponding xapian document just gets more terms added to it,
but this doesn't seem to break anything. Values on the other hand get
overwritten, which is a bit annoying, but arguably it is not worse to
take the values (from, subject, date) from the last file indexed
rather than the first.
2017-08-01 21:17:47 -04:00
David Bremner
4fdabd636e lib: refactor notmuch_database_add_message header parsing
This function is large and hard to understand and modify. Start to
break it down into meaningful pieces.
2017-08-01 21:17:47 -04:00
David Bremner
2f94b3090c lib: factor out message-id parsing to separate file.
This is really pure C string parsing, and doesn't need to be mixed in
with the Xapian/C++ layer. Although not strictly necessary, it also
makes it a bit more natural to call _parse_message_id from multiple
compilation units.
2017-08-01 21:17:47 -04:00
David Bremner
95b52e85b2 lib/n_d_add_message: refactor test for new/ghost messages
The switch is easier to understand than the side effects in the if
test. It also potentially allows us more flexibility in breaking up
this function into smaller pieces, since passing private_status around
is icky.
2017-08-01 21:17:47 -04:00
David Bremner
4034a7cec7 lib: isolate n_d_add_message and helper functions into own file
'database.cc' is becoming a monster, and it's hard to follow what the
various static functions are used for. It turns out that about 1/3 of
this file notmuch_database_add_message and helper functions not used
by any other function. This commit isolates this code into it's own
file.

Some side effects of this refactoring:

- find_doc_ids becomes the non-static (but still private)
  _notmuch_database_find_doc_ids
- a few instances of 'string' have 'std::' prepended, avoiding the
  need for 'using namespace std;' in the new file.
2017-08-01 21:17:47 -04:00
Daniel Kahn Gillmor
d55fffffd7 fix the generated documentation output 2017-07-18 06:53:57 -03:00
Daniel Kahn Gillmor
87bdfbc91f Fix orthography 2017-07-18 06:50:44 -03:00
David Bremner
4ce7591610 lib: paper over allocation difference
In gmime 3.0 this function is "transfer none", so no deallocation is
needed (or permitted)
2017-07-14 21:23:52 -03:00
David Bremner
eeb64cdeeb lib: add version of _n_m_f_get_combinded_header for gmime 3.0
The iterator is gone, so we need a new loop structure.
2017-07-14 21:23:52 -03:00
David Bremner
439c5896b6 lib: refactor _notmuch_messsage_file_get_combined_header
We need to rewrite the loop for gmime-3.0; move the loop body to its
own function to avoid code duplication.  Keep the common exit via
"goto DONE" to make this pure code movement.  It's important to note
that the existing exit path only deallocates the iterator.
2017-07-14 21:23:52 -03:00
David Bremner
c040464a7c lib: wrap use of g_mime_utils_header_decode_date
This changes return type in gmime 3.0
2017-07-14 21:23:52 -03:00
David Bremner
cbb2d5608e lib/cli: replace use of g_mime_message_get_sender
This function changes semantics in gmime-3.0 so make a new function
that provides the same functionality in both
2017-07-14 17:58:09 -03:00
David Bremner
6dd00d6486 lib/index: add simple html filter
The filter just drops all (HTML) tags. As an enabling change, pass the
content type to the filter constructor so we can decide which scanner
to user.
2017-07-01 12:32:27 -03:00
David Bremner
64f81f95a1 lib/index.cc: generalize filter state machine
To match things more complicated than fixed strings, we need states
with multiple out arrows.
2017-07-01 12:32:17 -03:00
David Bremner
4a085a5137 lib/index: separate state table definition from scanner.
We want to reuse the scanner definition with a different table.  This
is mainly code movement, and making the state table part of the filter
struct/class.
2017-07-01 12:32:03 -03:00
David Bremner
20c15bc820 lib/index: generalize name of indexing filter
In followup commits we will generalize the functionality of this
filter to deal with other types of non-indexable content.
2017-07-01 12:31:55 -03:00
Jani Nikula
30c475c1ef build: visibility=default for library structs is no longer needed
Commit d5523ead90 ("Mark some structures in the library interface
with visibility=default attribute.") fixed some mixed visibility
issues with structs. With the symbol default visibility reversed, this
is no longer a problem.
2017-05-13 08:38:18 -03:00
Jani Nikula
bc11759dd1 build: switch to hiding libnotmuch symbols by default
The dynamic generation of the linker version script for libnotmuch
exports has grown rather complicated.

Reverse the visibility control by hiding symbols by default using
-fvisibility=hidden, and explicitly exporting symbols in notmuch.h
using #pragma GCC visibility. (We could also use __attribute__
((visibility ("default"))) for each exported function, but the pragma
is more convenient.)

The above is not quite enough alone, as it would "leak" a number of
weak symbols from Xapian and C++ standard library. Combine it with a
small static version script that filters out everything except the
notmuch_* symbols that we explicitly exposed, and the C++ RTTI
typeinfo symbols for exception handling.

Finally, as the symbol hiding test can no longer look at the generated
symbol table, switch the test to parse the functions from notmuch.h.
2017-05-12 07:17:18 -03:00
Jani Nikula
d5ed9af0e4 build: do not export compat functions from lib
Commits 9db2145272 ("lib/gen-version-script.h: add getline and
getdelim to notmuch.sym if needed") and 3242e29e57 ("build: add
canonicalize_file_name to symbols exported from libnotmuch.so")
started exporting compat functions from libnotmuch so that the cli
could use them. But we shouldn't export such functions from the
library. They are not part of our ABI. Instead, the cli should include
its own copies of the compat functions.
2017-05-11 20:41:10 -03:00
David Bremner
11d47950c1 lib: Add regexp expansion for for tags and paths
From a UI perspective this looks similar to what was already provided
for from, subject, and mid, but the implementation is quite
different. It uses the database's list of terms to construct a term
based query equivalent to the passed regular expression.
2017-05-09 07:44:29 -03:00
David Bremner
eab365c742 lib: Add regexp searching for mid: prefix
The bulk of the change is passing in the field options to the regexp
field processor, so that we can properly handle the
fallback (non-regexp case).
2017-05-09 07:44:15 -03:00
Fredrik Fornwall
e565118172 Replace index(3) with strchr(3)
The index(3) function has been deprecated in POSIX since 2001 and
removed in 2008, and most code in notmuch already calls strchr(3).

This fixes a compilation error on Android whose libc does not have
index(3).
2017-04-20 06:59:22 -03:00
David Bremner
e1c1d33f37 Merge branch 'release'
Another regexp search fix.
2017-03-29 20:58:34 -03:00
David Bremner
cb84f84878 lib: handle empty string in regexp field processors
The non-field processor behaviour is is convert the corresponding
queries into a search for the unprefixed terms. This yields pretty
surprising results so I decided to generate a query that would match
the terms (i.e. none with that prefix) generated for an empty header.
2017-03-29 20:44:32 -03:00
David Bremner
d877240f4e Merge branch 'release'
wildcard search fixes, plus release busywork
2017-03-25 11:51:03 -03:00
David Bremner
38a56b98f9 lib: only trigger phrase processing for regexp fields when needed
The argument is that if the string passed to the field processor has
no spaces, then the added quotes won't have any benefit except for
disabling wildcards. But disabling wildcards doesn't seem very useful
in the normal Xapian query parser, since they're stripped before
generating terms anyway. It does mean that the query 'from:"foo*"' will
not be precisely equivalent to 'from:foo' as it is for the non
field-processor version.
2017-03-24 09:24:13 -03:00
David Bremner
242d5a3ed5 lib: make notmuch_query_add_tag_exclude return a status value
Since this is an ABI breaking change, but we already bumped the SONAME
for the next release
2017-03-22 08:47:13 -03:00
David Bremner
3721bd45d7 lib: replace deprecated n_q_count_threads with status returning version
This function was deprecated in notmuch 0.21.  We re-use the name for
a status returning version, and deprecate the _st name.
2017-03-22 08:35:07 -03:00
David Bremner
5ce8e0b11b lib: replace deprecated n_q_count_messages with status returning version
This function was deprecated in notmuch 0.21.  We re-use the name for
a status returning version, and deprecate the _st name. One or two
remaining uses of the (removed) non-status returning version fixed at
the same time
2017-03-22 08:35:07 -03:00
David Bremner
86cbd215eb lib: replace deprecated n_q_search_messages with status returning version
This function was deprecated in notmuch 0.21.  We re-use the name for
a status returning version, and deprecate the _st name.
2017-03-22 08:35:07 -03:00
David Bremner
1e982de508 lib: replace n_query_search_threads with status returning version
This function was deprecated in notmuch 0.21. We finally remove the
deprecated API, and rename the status returning version to the simpler
name. The status returning is kept as a deprecated alias.
2017-03-22 08:28:09 -03:00
David Bremner
fc63c15833 lib: bump SONAME to libnotmuch5
We plan a sequence of ABI breaking changes. Put the SONAME change in a
separate commit to make reordering easier.
2017-03-22 08:27:58 -03:00
David Bremner
c39f6361d0 rename libutil.a to libnotmuch_util.a
Apparently some systems (MacOS?) have a system library called libutil
and the name conflict causes problems. Since this library is quite
notmuch specific, rename it to something less generic.
2017-03-18 21:37:43 -03:00
David Bremner
a8a2705222 Merge branch 'release'
Merge in memory fixes
2017-03-18 21:02:42 -03:00
Tomi Ollila
06adc27668 lib/message.cc: fix Coverity finding (use after free)
The object where pointer to `data` was received was deleted before
it was used in _notmuch_string_list_append().

Relevant Coverity messages follow:

3: extract
Assigning: data = std::__cxx11::string(message->doc.()).c_str(),
which extracts wrapped state from temporary of type std::__cxx11::string.

4: dtor_free
The internal representation of temporary of type std::__cxx11::string
is freed by its destructor.

5: use after free:
Wrapper object use after free (WRAPPER_ESCAPE)
Using internal representation of destroyed object local data.
2017-03-18 20:59:46 -03:00
David Bremner
62822a4e2d lib: clamp return value of g_mime_utils_header_decode_date to >=0
For reasons not completely understood at this time, gmime (as of
2.6.22) is returning a date before 1900 on bad date input. Since this
confuses some other software, we clamp such dates to 0,
i.e. 1970-01-01.
2017-03-15 21:58:25 -03:00
Jani Nikula
d56a801b67 lib/database: reduce try block scope to things that really need it
No need to maintain the pure C stuff within a try block, it's arguably
confusing. This also reduces indent for a bunch of code. No functional
changes.
2017-03-10 09:21:05 -04:00
Olly Betts
81bd72cebb lib: Fix RegexpPostingSource
Remove incorrect skipping to first match from init(), and add explicit
skip_to() and check() methods to work around xapian-core bug (the
check() method will also improve speed when filtering by one of
these).
2017-03-07 19:44:36 -04:00
David Bremner
dfacfe14f3 lib: query make exclude handling non-destructive
We filter added exclude at add time, rather than modifying the query by
count search. As noted in the comments, there are several ignored
conditions here.
2017-03-04 20:47:25 -04:00
David Bremner
e209b71873 lib: centralize query parsing, store results.
The main goal is to prepare the way for non-destructive (or at least
less destructive) exclude tag handling. It does this by having a
pre-parsed query available for further processing. This also allows us
to provide slightly more precise error messages.
2017-03-04 20:47:25 -04:00
Jani Nikula
f3edc5dc86 lib: use delete[] to free buffer allocated using new[]
Fix warning caught by clang:

lib/regexp-fields.cc:41:2: warning: 'delete' applied to a pointer that was allocated
      with 'new[]'; did you mean 'delete[]'? [-Wmismatched-new-delete]
        delete buffer;
        ^
              []
lib/regexp-fields.cc:37:17: note: allocated with 'new[]' here
        char *buffer = new char[len];
                       ^
2017-03-04 20:42:39 -04:00
David Bremner
6cb1c617a7 lib: add mid: as a synonym for id:
mid: is the url scheme suggested by URL 2392. We also plan to
introduce more flexible searches for mid: than are possible with
id: (in order not to break assumptions about the special behaviour of
id:, e.g. identifying at most one message).
2017-03-03 17:46:48 -04:00
David Bremner
55524bb063 lib: regexp matching in 'subject' and 'from'
the idea is that you can run

% notmuch search subject:/<your-favourite-regexp>/
% notmuch search from:/<your-favourite-regexp>/

or

% notmuch search subject:"your usual phrase search"
% notmuch search from:"usual phrase search"

This feature is only available with recent Xapian, specifically
support for field processors is needed.

It should work with bindings, since it extends the query parser.

This is easy to extend for other value slots, but currently the only
value slots are date, message_id, from, subject, and last_mod. Date is
already searchable;  message_id is left for a followup commit.

This was originally written by Austin Clements, and ported to Xapian
field processors (from Austin's custom query parser) by yours truly.
2017-03-03 17:46:48 -04:00
David Bremner
31b8ce4558 lib: create field processors from prefix table
This is a bit more code than hardcoding the two existing field
processors, but it should make it easy to add more.
2017-03-03 07:15:13 -04:00
David Bremner
7bd63833bf lib/message.cc: use view number to invalidate cached metadata
Currently the view number is incremented by notmuch_database_reopen
2017-02-25 21:15:38 -04:00
David Bremner
e0b22c139c lib: handle DatabaseModifiedError in _n_message_ensure_metadata
The retries are hardcoded to a small number, and error handling aborts
than propagating errors from notmuch_database_reopen. These are both
somewhat justified by the assumption that most things that can go
wrong in Xapian::Database::reopen are rare and fatal. Here's the brief
discussion with Xapian upstream:

   24-02-2017 08:12:57 < bremner> any intuition about how likely
      Xapian::Database::reopen is to fail? I'm catching a
      DatabaseModifiedError somewhere where handling any further errors is
      tricky, and wondering about treating a failed reopen as as "the
      impossible happened, stopping"

   24-02-2017 16:22:34 < olly> bremner: there should not be much scope for
    failure - stuff like out of memory or disk errors, which are probably a
    good enough excuse to stop
2017-02-25 21:13:50 -04:00
David Bremner
e17a914b77 lib: add _notmuch_database_reopen
The main expected use is to recover from a Xapian::DatabaseChanged
exception.
2017-02-25 21:09:17 -04:00
David Bremner
e0e8586fc7 Merge branch 'release'
Merge in g_hash_table read-after-free fix
2017-02-23 09:08:15 -04:00
David Bremner
884dccf293 lib: make _notmuch_message_ensure_property_map static
It's not called outside message.cc
2017-02-23 08:54:36 -04:00
David Bremner
3db9e94b0e lib: make _notmuch_message_ensure_metadata static
It's not called anywhere outside message.cc.
2017-02-23 08:54:25 -04:00
David Bremner
4e649d000b lib: fix g_hash_table related read-after-free bug
The two g_hash_table functions (insert, add) have different behaviour
with respect to existing keys. g_hash_table_insert frees the new key,
while g_hash_table_add (which is really g_hash_table_replace in
disguise) frees the existing key. With this change 'ref' is live until
the end of the function (assuming single-threaded access to
'hash'). We can't guarantee it will continue to be live in the
future (i.e. there may be a future key duplication) so we copy it with
the allocation context passed to parse_references (in practice this is
the notmuch_message_t object whose parents we are finding).

Thanks to Tomi for the simpler approach to the problem based on
reading the fine glib manual.
2017-02-22 06:28:03 -04:00
David Bremner
0e037c34dd lib: Let Xapian manage the memory for FieldProcessors
It turns out this is exactly what release() is for; Xapian will
deallocate the objects when it's done with them.
2017-02-18 22:18:06 -04:00
David Bremner
e30fa4182f lib: merge internal prefix tables
Replace multiple tables with some flags in a single table. This makes
the code in notmuch_database_open_verbose a bit shorter, and it should
also make it easier to add other options to fields, e.g. regexp
searching.
2017-02-18 22:17:39 -04:00
David Bremner
70519319b5 lib: optimize counting documents
From #xapian

olly> bremner: btw, i noticed notmuch count see ms to request all the documents and then ignores them

bremner> hmm. There's something funny about the way that notmuch uses matches in general iirc

olly> it should be able to do: mset = enquire.get_mset (0, 0, notmuch->xapian_db->get_doccount ());
...
olly> get_matches_estimated() will be exact because check_at_least is the size of the database
2017-01-27 21:54:44 -04:00
Steven Allen
4a2ce7b570 docs: fix notmuch_message_properties_value documentation
It returns the value, not the key.
2017-01-15 14:25:00 -04:00
Jani Nikula
c906da9f60 lib: use glib for sha1 digests instead of embedding libsha1
We already depend on glib both directly and indirectly (via gmime). We
might as well make use of its facilities. Drop the embedded libsha1
and use glib for sha1 digests.
2017-01-08 10:50:38 -04:00
Jani Nikula
217404ff86 lib: fix the todo comment placement on NOTMUCH_STATUS_XAPIAN_EXCEPTION
The todo comment got separated from the status it's related to at
commit 3f32fd8a1c ("Add missing comment for
NOTMUCH_STATUS_READONLY_DATABASE."). Later, commit b65ca8e0ba ("lib:
modify notmuch.h for automatic document generation") moved it, but to
the wrong place. Fix the location.
2017-01-07 08:30:08 -04:00
David Bremner
0abcad7c0e lib: optionally silence Xapian deprecation warnings
This is not ideal, but the new API is not available in Xapian 1.2.x, and
it seems to soon to depend on Xapian >= 1.4
2016-11-15 07:47:55 -04:00
David Bremner
7f07a3f0ed lib: replace deprecated xapian call 'flush()' with 'commit()'
This will make notmuch incompatible with Xapian before 1.1.0, which is
more than 6 years old this point.
2016-10-25 18:13:52 -03:00
David Bremner
b2d6f07a02 lib: document API added in 0.23
The API was already documented, but for future readers note when the
functions were added,
2016-10-06 22:46:01 -03:00
David Bremner
af8903df34 require xapian >= 1.2.6
It seems that no-one tried to compile without Xapian compact support
since March of 2015, since that's when I introduced a syntax error in
that branch of the ifdef.

Given the choice of maintaining this underused branch of code, or
bumping the Xapian dependency to a version from 2011, it seems
reasonable to do the latter.
2016-10-06 22:45:46 -03:00
David Bremner
c2e74662bb lib: bump minor version to mark added symbols
This should not change the SONAME, and therefore won't change the
dynamic linking behaviour, but it may help some users debug missing
symbols in case their libnotmuch is too old.
2016-10-01 22:19:07 -03:00
Tomi Ollila
1c3a8e0898 lib/database.cc: fix misleading indentation
Found by gcc 6.1.1 -Wmisleading-indentation option (set by -Wall).
2016-09-28 08:14:08 -03:00
David Bremner
514a0a6a3b lib: add talloc reference from string map iterator to map
This is needed so that when the map is modified during traversal, and
thus unlinked by the database code, the map is not disposed of until the
iterator is done with it.
2016-09-24 10:08:45 -03:00
Daniel Kahn Gillmor
693ca8d8a8 add property: query prefix to search for specific properties
We want to be able to query the properties directly, like:

   notmuch count property:foo=bar

which should return a count of messages where the property with key
"foo" has value equal to "bar".
2016-09-21 18:14:25 -03:00
David Bremner
58fe8fce1d lib: iterator API for message properties
This is a thin wrapper around the string map iterator API just introduced.
2016-09-21 18:14:25 -03:00
David Bremner
b846bdb482 lib: extend private string map API with iterators
Support for prefix based iterators is perhaps overengineering, but I
wanted to mimic the existing database_config API.
2016-09-21 18:14:24 -03:00
David Bremner
b8bb6d7964 lib: basic message-property API
Initially, support get, set and removal of single key/value pair, as
well as removing all properties.
2016-09-21 18:14:24 -03:00
David Bremner
8b03ee1d5a lib: private string map (associative array) API
The choice of array implementation is deliberate, for future iterator support
2016-09-21 18:14:24 -03:00
David Bremner
4dfb69169e lib: read "property" terms from messages.
This is a first step towards providing an API to attach
arbitrary (key,value) pairs to messages and retrieve all of the values
for a given key.
2016-09-21 18:14:24 -03:00
David Bremner
59fed50a82 lib: update cached mtime in notmuch_directory_set_mtime
Without this change, the following code fails

  notmuch_directory_set_mtime(dir, 12345);
  assert(notmuch_directory_get_mtime(dir) == 12345);
2016-08-23 20:58:46 -03:00
David Bremner
9e177b236c lib: reword comment about XFOLDER: prefix
I believe the current one is misleading, because in my experiments
Xapian did not add : when prefix and term were both upper case. Indeed,
it's hard to see how it could, because prefixes are added at a layer
above Xapian in our code. See _notmuch_message_add_term for an example.

Also try to explain why this is a good idea.  As far as I can ascertain,
this is more of an issue for a system trying to work with an unknown set
of prefixes. Since notmuch has a fixed set of prefixes, and we can
hopefully be trusted not to add XGOLD and XGOLDEN as prefixes, it is
harder for problems to arise.
2016-08-18 05:11:37 -03:00
David Bremner
293186d6c6 lib: provide _notmuch_database_log_append
_notmuch_database_log clears the log buffer each time. Rather than
introducing more complicated semantics about for this function, provide
a second function that does not clear the buffer. This is mainly a
convenience function for callers constructing complex or multi-line log
messages.

The changes to query.cc are to make sure that the common code path of
the new function is tested.
2016-08-09 09:34:11 +09:00
David Bremner
3a45d29ed4 lib: add built_with handling for XAPIAN_DB_RETRY_LOCK
This support will be present only if the appropriate version of xapian
is available _and_ the user did not disable the feature when
building. So there really needs to be some way for the user to check.
2016-06-29 09:05:49 +02:00
Istvan Marko
9b60dc3cd9 Use the Xapian::DB_RETRY_LOCK flag when available
Xapian 1.3 has introduced the DB_RETRY_LOCK flag (Xapian bug
275). Detect it in configure and optionally use it. With this flag
commands that need the write lock will wait for their turn instead of
aborting when it's not immediately available.

Amended by db: allow disabling in configure
2016-06-29 09:03:34 +02:00
David Bremner
38f0d44a82 doc: forbid further operations on a closed database
We could add many null pointer checks, but currently I don't see a use
case that justifies it.
2016-06-28 23:20:38 +02:00
David Bremner
44cfa90bdc lib: fix definition of LIBNOTMUCH_CHECK_VERSION
Fix bug reported in id:20160606124522.g2y2eazhhrwjsa4h@flatcap.org

Although the C99 standard 6.10 is a little non-obvious on this point,
the docs for e.g. gcc are unambiguous. And indeed in practice with the
extra space, this code fails

#include <stdio.h>
#define foo (x) (x+1)

int main(int argc, char **argv){
  printf("%d\n",foo(1));
}
2016-06-11 13:01:44 -03:00
David Bremner
4291f32680 lib: fix memory leak of field processor objects
The field processor objects need to be deallocated explicitly just like
the range processors (or a talloc destructor defined).
2016-06-10 09:20:22 -03:00
David Bremner
ba0b95f846 lib: document config metadata
This probably should have been part of 3458e3c89c, but I missed it.
2016-06-07 07:51:57 -03:00
Daniel Kahn Gillmor
6a833a6e83 Use https instead of http where possible
Many of the external links found in the notmuch source can be resolved
using https instead of http.  This changeset addresses as many as i
could find, without touching the e-mail corpus or expected outputs
found in tests.
2016-06-05 08:32:17 -03:00
Tomi Ollila
cf09631a45 lib: whitespace cleanup
Cleaned the following whitespace in lib/* files:

lib/index.cc:              1 line:  trailing whitespace
lib/database.cc            5 lines: 8 spaces at the beginning of line
lib/notmuch-private.h:     4 lines: 8 spaces at the beginning of line
lib/message.cc:            1 line:  trailing whitespace
lib/sha1.c:                1 line:  empty lines at the end of file
lib/query.cc:              2 lines: 8 spaces at the beginning of line
lib/gen-version-script.sh: 1 line:  trailing whitespace
2016-06-05 08:23:28 -03:00
David Bremner
b9bf3f44ea lib: add support for named queries
This relies on the optional presense of xapian field processors, and the
library config API.
2016-05-25 07:40:44 -03:00
David Bremner
30caaf52b0 lib: make a global constant for query parser flags
It's already kindof gross that this is hardcoded in two different
places. We will also need these later in field processors calling back
into the query parser.
2016-05-25 07:40:44 -03:00
David Bremner
92e59568fa lib: config list iterators
Since xapian provides the ability to restrict the iterator to a given
prefix, we expose this ability to the user. Otherwise we mimic the other
iterator interfances in notmuch (e.g. tags.c).
2016-05-25 06:51:16 -03:00
David Bremner
3458e3c89c lib: provide config API
This is a thin wrapper around the Xapian metadata API. The job of this
layer is to keep the config key value pairs from colliding with other
metadata by transparently prefixing the keys, along with the usual glue
to provide a C interface.

The split of _get_config into two functions is to allow returning of the
return value with different memory ownership semantics.
2016-05-24 08:53:03 -03:00
David Bremner
792bea5aff lib/cli: add library API / CLI for compile time options
This is intentionally low tech; if we have more than two options it may
make sense to build up what infrastructure is provided.
2016-05-13 07:29:12 -03:00
David Bremner
bbf6069252 lib: optionally support single argument date: queries
This relies on the FieldProcessor API, which is only present in xapian
>= 1.3.
2016-05-08 08:17:07 -03:00
Daniel Kahn Gillmor
e366bb2227 complete ghost-on-removal-when-shared-thread-exists
To fully complete the ghost-on-removal-when-shared-thread-exists
proposal, we need to clear all ghost messages when the last active
message is removed from a thread.

Amended by db: Remove the last test of T530, as it no longer makes sense
if we are garbage collecting ghost messages.
2016-04-15 07:13:49 -03:00
Daniel Kahn Gillmor
1695415039 On deletion, replace with ghost when other active messages in thread
There is no need to add a ghost message upon deletion if there are no
other active messages in the thread.

Also, if the message being deleted was a ghost already, we can just go
ahead and delete it.
2016-04-15 07:07:23 -03:00
Daniel Kahn Gillmor
9eebae3da4 Introduce _notmuch_message_has_term()
It can be useful to easily tell if a given message has a given term
associated with it.
2016-04-15 07:07:23 -03:00
Daniel Kahn Gillmor
011fc41d4d Add internal functions to search for alternate doc types
Publicly we are only exposing the non-ghost documents (of "type"
"mail").  But internally we might want to inspect the ghost messages
as well.

This changeset adds two new private interfaces to queries to recover
information about alternate document types.
2016-04-15 07:07:23 -03:00
Daniel Kahn Gillmor
604d1e0977 fix thread breakage via ghost-on-removal
implement ghost-on-removal, the solution to T590-thread-breakage.sh
that just adds a ghost message after removing each message.

It leaks information about whether we've ever seen a given message id,
but it's a fairly simple implementation.

Note that _resolve_message_id_to_thread_id already introduces new
message_ids to the database, so i think just searching for a given
message ID may introduce the same metadata leakage.
2016-04-15 07:07:23 -03:00
Jani Nikula
54aeab1962 lib: clean up _notmuch_database_split_path
Make the logic it a bit easier to read. No functional changes.
2016-04-12 20:46:42 -03:00
Jani Nikula
a352d9ceaa lib: fix handling of one character long directory names at top level
The code to skip multiple slashes in _notmuch_database_split_path()
skips back one character too much. This is compensated by a +1 in the
length parameter to the strndup() call. Mostly this works fine, but if
the path is to a file under a top level directory with one character
long name, the directory part is mistaken to be part of the file name
(slash == path in code). The returned directory name will be the empty
string and the basename will be the full path, breaking the indexing
logic in notmuch new.

Fix the multiple slash skipping to keep the slash variable pointing at
the last slash, and adjust strndup() accordingly.

The bug was introduced in

commit e890b0cf40
Author: Carl Worth <cworth@cworth.org>
Date:   Sat Dec 19 13:20:26 2009 -0800

    database: Store the parent ID for each directory document.

just a little over two months after the initial commit in the Notmuch
code history, making this the longest living bug in Notmuch to date.
2016-04-12 20:40:19 -03:00
Tomi Ollila
342910a280 lib: NOTMUCH_DEPRECATED macro also for older compilers
Some compilers (older than gcc 4.5 and clang 2.9) do support
__attribute__ ((deprecated)) but not
__attribute__ ((deprecated("message"))).

Check if clang version is at least 3.0, or gcc version
is at least 4.5 to define NOTMUCH_DEPRECATED as the
latter variant above. Otherwise define NOTMUCH_DEPRECATED
as the former variant above.

For a bit simpler implementation clang 2.9 is not included
to use the newer variant. It is just one release, and the
older one works fine. Clang 3.0 was released around 2011-11
and gcc 5.1 2015-04-22 (therefore newer macro for gcc 4.5+)
2016-03-14 19:54:32 -03:00
Daniel Kahn Gillmor
07b6220a55 clean up stray apostrophe in comment
This is a nit-picky orthographical fix for an nit-picky ontological
comment.
2016-01-16 08:17:15 -04:00
Daniel Kahn Gillmor
e038b95ffe correct comment referring to notmuch_database_remove_message
notmuch_database_remove_message has no leading underscore in its name.
2016-01-16 08:16:51 -04:00
Steven Allen
c946356cdc forbid atomic transactions on writable, upgradable databases
We can't (but currently do) allow upgrades within transactions because
upgrades need their own transactions. We don't want to re-use the
current transaction because bailing out of an upgrade would mean loosing
all previous changes (because our "atomic" transactions don't commit
before hand). This gives us two options:

1. Fail at the beginning of upgrade (tell the user to end the
   transaction, upgrade, and start over).
2. Don't allow the user to start the transaction.

I went with the latter because:

1. There is no reason to call `begin_atomic` unless you intend to to
   write to the database and anyone intending to write to the database
   should upgrade it first.
2. This means that nothing inside an atomic transaction can ever fail
   with NOTMUCH_STATUS_UPGRADE_REQUIRED.
2015-11-23 08:15:37 -04:00
Jani Nikula
506b81679a lib: content disposition values are not case-sensitive
Per RFC 2183, the values for Content-Disposition values are not
case-sensitive. While at it, use the gmime function for getting at the
disposition string instead of referencing the field directly.

This fixes "attachment" tagging and filename term generation for
attachments while indexing.
2015-11-19 07:47:29 -04:00
Steven Allen
10e933a3bb Documentation: fix type name spelling 2015-10-27 08:07:31 -03:00
Jani Nikula
727fcd18c6 lib: add interface to delete directory documents
As mentioned in acd66cdec0 we don't have
an interface to delete directory documents, and they're left behind. Add
the interface.
2015-10-10 09:14:25 -03:00
David Bremner
7a20f26f91 lib: update doxygen comments to add @since for the new _st API
We should probably to this for all new functions introduced from now on.
2015-10-05 20:16:59 -03:00
David Bremner
378ba492a6 lib: migrate thread.cc to new query_search API
here we rely on thread_id_query being attached to the local talloc
context, so no new cleanup code is needed.
2015-10-05 19:53:53 -03:00
David Bremner
2501c2565c lib: migrate notmuch_database_upgrade to new query_search API
Here we depend on the error path cleaning up query
2015-10-05 19:53:11 -03:00
David Bremner
87ee9a53e3 lib: add versions of n_q_count_{message,threads} with status return
Although I think it's a pretty bad idea to continue using the old API,
this allows both a more gentle transition for clients of the library,
and allows us to break one monolithic change into a series
2015-10-05 19:44:07 -03:00
David Bremner
65a6b86873 lib: move query variable to function scope
This is a prelude to deallocating it (if necessary) on the error path.
2015-10-05 19:39:11 -03:00
Jani Nikula
23b8ed610a lib: add support for date:<expr>..! to mean date:<expr>..<expr>
It doesn't seem likely we can support simple date:<expr> expanding to
date:<expr>..<expr> any time soon. (This can be done with a future
version of Xapian, or with a custom query query parser.) In the mean
time, provide shorthand date:<expr>..! to mean the same. This is
useful, as the expansion takes place before interpetation, and we can
use, for example, date:yesterday..! to match from beginning of
yesterday to end of yesterday.

Idea from Mark Walters <markwalters1009@gmail.com>.
2015-09-25 21:55:24 -03:00
David Bremner
93ee4faa4d lib: constify arguments to notmuch_query_get_*
These functions are all just accessors, and it's pretty clear they don't
modify the query struct. This also fixes one warning I created when I
introduced status.c.
2015-09-23 08:58:19 -03:00
Jani Nikula
f460ad4e9a util: move strcase_equal and strcase_hash to util
For future use in both cli and lib.
2015-09-07 09:43:31 -03:00
David Bremner
bd5504ec10 lib: constify argument to notmuch_database_status_string
We don't modify the database struct, so no harm in committing to that.
2015-09-04 08:24:38 -03:00
David Bremner
110694b00b lib: note remaining uses of deprecated message search API
The two remaining cases in the lib seem to require more than a simple
replacement of the old call, with the new call plus a check of the
return value.
2015-09-04 08:08:18 -03:00
David Bremner
f16944c3b4 lib: remove use of notmuch_query_search_messages from query.cc
There is not too much point in worrying about the bad error reporting
here, because the count api is due for the same deprecation.
2015-09-04 08:06:08 -03:00
Austin Clements
cb08a2ee01 lib: Add "lastmod:" queries for filtering by last modification
The implementation is essentially the same as the date range search
prior to Jani's fancy date parser.
2015-08-14 18:23:49 +02:00
Austin Clements
98ee460eaa lib: API to retrieve database revision and UUID
This exposes the committed database revision to library users along
with a UUID that can be used to detect when revision numbers are no
longer comparable (e.g., because the database has been replaced).
2015-08-13 23:52:51 +02:00
Austin Clements
7f57b747b9 lib: Add per-message last modification tracking
This adds a new document value that stores the revision of the last
modification to message metadata, where the revision number increases
monotonically with each database commit.

An alternative would be to store the wall-clock time of the last
modification of each message.  In principle this is simpler and has
the advantage that any process can determine the current timestamp
without support from libnotmuch.  However, even assuming a computer's
clock never goes backward and ignoring clock skew in networked
environments, this has a fatal flaw.  Xapian uses (optimistic)
snapshot isolation, which means reads can be concurrent with writes.
Given this, consider the following time line with a write and two read
transactions:

   write  |-X-A--------------|
   read 1       |---B---|
   read 2                      |---|

The write transaction modifies message X and records the wall-clock
time of the modification at A.  The writer hangs around for a while
and later commits its change.  Read 1 is concurrent with the write, so
it doesn't see the change to X.  It does some query and records the
wall-clock time of its results at B.  Transaction read 2 later starts
after the write commits and queries for changes since wall-clock time
B (say the reads are performing an incremental backup).  Even though
read 1 could not see the change to X, read 2 is told (correctly) that
X has not changed since B, the time of the last read.  In fact, X
changed before wall-clock time A, but the change was not visible until
*after* wall-clock time B, so read 2 misses the change to X.

This is tricky to solve in full-blown snapshot isolation, but because
Xapian serializes writes, we can use a simple, monotonically
increasing database revision number.  Furthermore, maintaining this
revision number requires no more IO than a wall-clock time solution
because Xapian already maintains statistics on the upper (and lower)
bound of each value stream.
2015-08-13 23:52:51 +02:00
David Bremner
765556c1f1 build: extract library versions from notmuch.h
- Make lib/notmuch.h the canonical location for the library versioning
information.

- Since the release-check should never fail now, remove it to reduce
complexity.

- Make the version numbers in notmuch.h consistent with the (now
  deleted) ones in lib/Makefile.local
2015-08-10 13:53:55 +02:00
David Bremner
6b440a0adf lib: add public accessor for database from query
This is to make it easier for clients of the library to update to the
new error code returning versions of notmuch_query_search_messages
2015-08-04 09:11:34 +02:00
David Bremner
4fed7047b2 lib: deprecate notmuch_query_search_{threads, messages}
The CLI (and bindings) code should really be updated to use the new
status-code-returning versions. Here are some warnings to prod us (and
other clients) to do so.
2015-08-04 09:11:25 +02:00
David Bremner
7e2d0ef105 lib: define NOTMUCH_DEPRECATED macro, document its use.
This has been tested with gcc and clang.
2015-08-04 09:11:17 +02:00
Austin Clements
e6ad3a5dd4 lib: Only sync modified message documents
Previously, we updated the database copy of a message on every call to
_notmuch_message_sync, even if nothing had changed.  In particular,
this always happens on a thaw, so a freeze/thaw pair with no
modifications between still caused a database update.

We only modify message documents in a handful of places, so keep track
of whether the document has been modified and only sync it when
necessary.  This will be particularly important when we add message
revision tracking.
2015-08-04 08:54:46 +02:00
David Bremner
882ccb7e49 build: add "set -eu" to version script generation
It turns out that on certain systems like FreeBSD, c++filt is not
installed by default. It's basically OK if we fail the build in that
case, but what's really not OK is for the build to continue and
generate bad binaries.
2015-07-28 21:34:01 +02:00
David Bremner
53035dafe0 lib, ruby: make use of -Wl,--no-undefined configurable
In particular this is supposed to help build on systems (presumably
using a non-gnu ld) where this flag is not available.
2015-06-13 17:52:48 +02:00
David Bremner
32fd74b7aa lib: reject relative paths in n_d_{create,open}_verbose
There are many places in the notmuch code where the path is assumed to be absolute. If someone (TM) wants a project, one could remove these assumptions. In the mean time, prevent users from shooting themselves in the foot.

Update test suite mark tests for this error as no longer broken, and
also convert some tests that used relative paths for nonexistent
directories.
2015-06-12 07:34:50 +02:00
David Bremner
b59ad1a9cc lib: add NOTMUCH_STATUS_PATH_ERROR
The difference with FILE_ERROR is that this is for things that are
wrong with the path before looking at the disk.

Add some 3 tests; two broken as a reminder to actually use this new
code.
2015-06-12 07:34:47 +02:00
J. Lewis Muir
d08af93c65 cli: change "setup" to "set up" where used as a verb
The word "setup" is a noun, not a verb.  Change occurrences of "setup"
where used as a verb to "set up".
2015-05-31 19:14:42 +02:00
David Bremner
9d192da683 lib: eliminate fprintf from _notmuch_message_file_open
You may wonder why _notmuch_message_file_open_ctx has two parameters.
This is because we need sometime to use a ctx which is a
notmuch_message_t. While we could get the database from this, there is
no easy way in C to tell type we are getting.
2015-03-29 00:34:15 +01:00
David Bremner
736ac26407 lib: replace almost all fprintfs in library with _n_d_log
This is not supposed to change any functionality from an end user
point of view. Note that it will eliminate some output to stderr. The
query debugging output is left as is; it doesn't really fit with the
current primitive logging model. The remaining "bad" fprintf will need
an internal API change.
2015-03-29 00:34:15 +01:00
David Bremner
9b73a8bcc9 lib: add private function to extract the database for a message.
This is needed by logging in functions outside message.cc that take
only a notmuch_message_t object.
2015-03-29 00:34:15 +01:00
David Bremner
b53e1a2da7 lib: add a log function with output to a string in notmuch_database_t
In principle in the future this could do something fancier than
asprintf.
2015-03-29 00:34:15 +01:00
David Bremner
84d3b15d25 lib: add "verbose" versions of notmuch_database_{open,create}
The compatibility wrapper ensures that clients calling
notmuch_database_open will receive consistent output for now.

The changes to notmuch-{new,search} and test/symbol-test are just to
make the test suite pass.

The use of IGNORE_RESULT is justified by two things. 1) I don't know
what else to do.  2) asprintf guarantees the output string is NULL if
an error occurs, so at least we are not passing garbage back.
2015-03-29 00:34:15 +01:00
Jani Nikula
6d44e5ac47 lib: make notmuch_query_count_messages explicitely exact
The default is actually exact if no checkatleast parameter is
specified. This change makes that explicit, mainly for documentation,
but also to be safe in the unlikely event of a change of default.

[ commit message rewritten by db based on id:87lho0nlkk.fsf@nikula.org
]
2015-03-13 07:58:55 +01:00
David Bremner
90886f3640 lib: bump library minor version
This should have happened in commit 326e18856, but it didn't.
2015-03-07 20:15:06 +01:00
J. Lewis Muir
ee2d490280 lib: make notmuch shared library install_name be full path on Mac OS X
The install_name of libnotmuch.dylib on Mac OS X is what is written
into a program that links against it.  If it is just the name of the
shared library file, as opposed to the full path, the program won't be
able to find it when it runs and will abort.  Instead, the install_name
should be the full path to the shared library (in its final installed
location).

Why does Notmuch work without this patch when installed via Homebrew?
The answer is twofold.  One, /usr/local/lib is a special location in
which the dynamic linker will look by default to find shared libraries.
Homebrew highly recommends installing to /usr/local, and, assuming it
has been configured this way, the Notmuch library will end up installed
in /usr/local/lib, and the dynamic linker will find it.  Two, Homebrew
globally corrects all install names in dynamically shared libraries and
binaries for each package it installs.  So, even if the install names in
a package's binaries and libraries are incorrect, Homebrew corrects them
automatically, and no one ever knows.

Why does Notmuch work without this patch when installed via MacPorts?
The answer is that MacPorts applies a patch just like this patch to fix
the same problem.
2015-03-06 08:00:18 +01:00
David Bremner
326e188564 lib: bump SONAME minor version
This indicates upwardly compatible changes, namely adding new symbols.

Although we don't formally need to do this until the next release,
there is no hard in doing it now, as long as we don't bump the minor
version for every addition between now and the release.
2015-03-01 08:49:14 +01:00
David Bremner
7a0fc10367 lib: add new status reporting API for notmuch_query_search_{m,t}
This at least allows distinguishing between out of memory and Xapian
exceptions. Adding finer grained status codes would allow different
Xapian exceptions to be preserved.

Adding wrappers allows people to transition gradually to the new API,
at the cost of bloating the library API a bit.
2015-03-01 08:49:14 +01:00
Jani Nikula
08757767de lib: fix clang build warnings
Fix the following warning produced by clang 3.5.0:

lib/message.cc:899:4: warning: comparison of constant 64 with expression of type 'notmuch_message_flag_t' (aka '_notmuch_message_flag') is always true [-Wtautological-constant-out-of-range-compare]
        ! NOTMUCH_TEST_BIT (message->lazy_flags, flag))
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./lib/notmuch-private.h:70:6: note: expanded from macro 'NOTMUCH_TEST_BIT'
    (_NOTMUCH_VALID_BIT(bit) ? !!((val) & (1ull << (bit))) : 0)
     ^~~~~~~~~~~~~~~~~~~~~~~
./lib/notmuch-private.h:68:26: note: expanded from macro '_NOTMUCH_VALID_BIT'
    ((bit) >= 0 && (bit) < CHAR_BIT * sizeof (unsigned long long))
                   ~~~~~ ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2015-02-25 23:09:39 +01:00
Jani Nikula
41b870fba5 lib: abstract bit validity check in bit test/set/clear macros
Reduce duplication in the bit test/set/clear macros. No functional
changes.
2015-02-25 23:08:35 +01:00
Todd
b04bc967f9 Add indexing for the mimetype term
This adds the indexing support for the "mimetype:" term and removes
the broken test flag.  The indexing is probablistic in Xapian terms,
which gives a better experience to end users.  Standard content-types
of the form "foo/bar" are automatically interpreted as phrases in
Xapian due to the embedded slash.

Assume, separate messages with application/pdf and application/x-pdf
are indexed, then:

- mimetype:application/x-pdf will find only the application/x-pdf
- mimetype:application/pdf will find only the application/pdf
- mimetype:pdf will find both of the messages
2015-01-24 16:47:59 +01:00
Todd
0de999aab5 Add the NOTMUCH_FEATURE_INDEXED_MIMETYPES database feature
This feature will exist in all newly created databases, but there is
no upgrade provided for it.  If this flag exists, it indicates that
the database was created after the indexed MIME-types feature was
added.
2015-01-24 16:47:47 +01:00
Todd
ef5b4947d8 lib: Fix use after free
_thread_set_subject_from_message sometimes replaces the subject, making the
cur_subject point to free'd memory

==6550== ERROR: AddressSanitizer: heap-use-after-free on address 0x601a0000bec0 at pc 0x4464a4 bp 0x7fffa40be910 sp 0x7fffa40be908
READ of size 1 at 0x601a0000bec0 thread T0
    #0 0x4464a3 in _thread_add_matched_message /home/todd/.apps/notmuch/lib/thread.cc:369
    #1 0x443c2c in notmuch_threads_get /home/todd/.apps/notmuch/lib/query.cc:496
    #2 0x41d947 in do_search_threads /home/todd/.apps/notmuch/notmuch-search.c:131
    #3 0x40a3fe in main /home/todd/.apps/notmuch/notmuch.c:345
    #4 0x7f4e535b4ec4 in __libc_start_main /build/buildd/eglibc-2.19/csu/libc-start.c:287
    #5 0x40abe6 in _start ??:?
0x601a0000bec0 is located 96 bytes inside of 134-byte region [0x601a0000be60,0x601a0000bee6)
freed by thread T0 here:
    #0 0x7f4e54e6933a in __interceptor_free ??:?
    #1 0x7f4e54482fab in _talloc_free ??:?
previously allocated by thread T0 here:
    #0 0x7f4e54e6941a in malloc ??:?
    #1 0x7f4e54485b5d in talloc_strdup ??:?
2015-01-19 08:15:14 +01:00
Todd
0fc7884602 lib: Fix potential invalid read past an empty string
==22884== ERROR: AddressSanitizer: heap-buffer-overflow on address 0x601600008291 at pc 0x7ff6295680e5 bp 0x7fff4ab9aa40 sp 0x7fff4ab9aa08
READ of size 1 at 0x601600008291 thread T0
    #0 0x7ff6295680e4 in __interceptor_strcmp ??:?
    #1 0x44763b in _thread_add_message /home/todd/.apps/notmuch/lib/thread.cc:255
    #2 0x4459e8 in notmuch_threads_get /home/todd/.apps/notmuch/lib/query.cc:496
    #3 0x41e2a7 in do_search_threads /home/todd/.apps/notmuch/notmuch-search.c:131
    #4 0x40a408 in main /home/todd/.apps/notmuch/notmuch.c:345
    #5 0x7ff627cb9ec4 in __libc_start_main /build/buildd/eglibc-2.19/csu/libc-start.c:287
    #6 0x40abf3 in _start ??:?
0x601600008291 is located 0 bytes to the right of 97-byte region [0x601600008230,0x601600008291)
allocated by thread T0 here:
    #0 0x7ff62956e41a in malloc ??:?
    #1 0x7ff628b8ab5d in talloc_strdup ??:?
2015-01-19 08:15:05 +01:00
Jesse Rosenthal
67e368ac7b thread.cc: Avoid empty thread names if possible.
Currently the thread is named based on either the oldest or newest
matching message (depending on the search order). If this message has
an empty subject, though, the thread will show up with an empty
subject in the search results. (See the thread starting with
`id:1412371140-21051-1-git-send-email-david@tethera.net` for an
example.)

This changes the behavior so it will use a non-empty name for the
thread if possible. We name threads based on (a) non-empty matches for
the query, and (b) the search order. If the search order is
oldest-first (as in the default inbox) it chooses the oldest matching
non-empty message as the subject. If the search order is newest-first
it chooses the newest one.
2015-01-17 13:48:26 +01:00
David Bremner
105537a809 lib: convert two "iterator copy strings" into references.
Apparently this is a supported and even idiomatic way of keeping a
temporary object (e.g. like that returned from an operator
dereference) alive.
2015-01-02 17:18:42 +01:00
David Bremner
3d978a0d61 lib: another iterator-temporary/stale-pointer bug
Tamas Szakaly points out [1] that the bug fixed in 51b073c still
exists in at least one place. This change follows the suggestion of
[2] and creates a block scope temporary std::string to avoid the rules
of iterators temporaries.

[1]: id:20141226113755.GA64154@pamparam
[2]: id:20141226230655.GA41992@pamparam
2015-01-02 17:10:37 +01:00
Jani Nikula
c82a1745ac lib: drop the deprecation message for single-message mbox files
We generally do not support mbox files, but for historical reasons
we've supported single-message mbox files, with a deprecation
message. We've tried dropping the support altogether, but backed out
of it because we'd need to stop indexing them, while keeping support
for previously indexed files. This would be more complicated than
simply supporting single-message mbox files. Therefore, drop the
deprecation message, and just silently accept single-message mboxes.
2015-01-01 16:47:47 +01:00
Jesse Rosenthal
4135a1f8da lib: Use email address instead of empty real name.
Currently, if a From-header is of the form:

    "" <address@example.com>

the empty string will be treated as a valid real-name, and the entry
in the search results will be empty.

The new behavior here is that we treat an empty real-name field as if
it were null, so that the email address will be used in the search
results instead.

Signed-off-by: Jesse Rosenthal <jrosenthal@jhu.edu>
2014-12-07 13:36:08 +01:00
David Bremner
74bd4ea2ed lib: bump LIBNOTMUCH_MAJOR_VERSION to 4
This should have been done at the same time as the SONAME bump.
2014-11-06 00:10:03 +01:00
Austin Clements
70f15b37fb lib: Remove unnecessary thread linking steps when using ghost messages
Previously, it was necessary to link new messages to children to work
around some (though not all) problems with the old metadata-based
approach to stored thread IDs.  With ghost messages, this is no longer
necessary, so don't bother with child linking when ghost messages are
in use.
2014-10-25 19:46:19 +02:00
Austin Clements
ee476f1e76 lib: Enable ghost messages feature
This fixes the broken thread order test.
2014-10-25 19:31:27 +02:00
Austin Clements
d1e8c80b72 lib: Implement upgrade to ghost messages feature
Somehow this is the first upgrade pass that actually does *any* error
checking, so this also adds the bit of necessary infrastructure to
handle that.
2014-10-25 19:30:08 +02:00
Austin Clements
58a4277d3b lib: Implement ghost-based thread linking
This updates the thread linking code to use ghost messages instead of
user metadata to link messages into threads.

In contrast with the old approach, this is actually correct.
Previously, thread merging updated only the thread IDs of message
documents, not thread IDs stored in user metadata.  As originally
diagnosed by Mark Walters [1] and as demonstrated by the broken
T260-thread-order test, this can cause notmuch to fail to link
messages even though they're in the same thread.  In principle the old
approach could have been fixed by updating the user metadata thread
IDs as well, but these are not indexed and hence this would have
required a full scan of all stored thread IDs.  Ghost messages solve
this problem naturally by reusing the exact same thread ID and message
ID representation and indexing as regular messages.

Furthermore, thanks to this greater symmetry, ghost messages are also
algorithmically simpler.  We continue to support the old user metadata
format, so this patch can't delete any code, but when we do remove
support for the old format, several functions can simply be deleted.

[1] id:8738h7kv2q.fsf@qmul.ac.uk
2014-10-25 19:27:07 +02:00
Austin Clements
bc9c50602d lib: Internal support for querying and creating ghost messages
This updates the message abstraction to support ghost messages: it
adds a message flag that distinguishes regular messages from ghost
messages, and an internal function for initializing a newly created
(blank) message as a ghost message.
2014-10-25 19:26:54 +02:00
Austin Clements
d99491f274 lib: Introduce macros for bit operations
These macros help clarify basic bit-twiddling code and are written to
be robust against C undefined behavior of shift operators.
2014-10-25 19:26:43 +02:00
Austin Clements
d9f5da00bb lib: Update database schema doc for ghost messages
This describes the structure of ghost mail documents.  Ghost messages
are not yet implemented.
2014-10-25 19:26:03 +02:00
Austin Clements
1cdb96d3c4 lib: Add a ghost messages database feature
This will be implemented over the next several patches.  The feature
is not yet "enabled" (this does not add it to
NOTMUCH_FEATURES_CURRENT).
2014-10-25 19:25:54 +02:00
Austin Clements
7487e2e221 lib: Handle empty date value
In the interest of robustness, avoid undefined behavior of
sortable_unserialise if the date value is missing.  This shouldn't
happen now, but ghost messages will have blank date values.
2014-10-11 07:10:12 +02:00
Austin Clements
46b1b035a5 lib: Refactor _notmuch_database_link_message
This moves the code to retrieve and clear the metadata thread ID out
of _notmuch_database_link_message into its own function.  This will
simplify future changes.
2014-10-11 07:10:02 +02:00
Austin Clements
54ec8a0fd8 lib: Move message ID compression to _notmuch_message_create_for_message_id
Previously, this was performed by notmuch_database_add_message.  This
happens to be the only caller currently (which is why this was safe),
but we're about to introduce more callers, and it makes more sense to
put responsibility for ID compression in the lower-level function
rather than requiring each caller to handle it.
2014-10-11 07:09:54 +02:00
Austin Clements
cec601c4dd lib: Simplify close and codify aborting atomic section
In Xapian, closing a database implicitly aborts any outstanding
transaction and commits changes.  For historical reasons,
notmuch_database_close had grown to almost, but not quite duplicate
this behavior.  Before closing the database, it would explicitly (and
unnecessarily) commit it.  However, if there was an outstanding
transaction (ie atomic section), commit would throw a Xapian
exception, which notmuch_database_close would unnecessarily print to
stderr, even though notmuch_database_close would ultimately abort the
transaction anyway when it called close.

This patch simplifies notmuch_database_close to explicitly abort any
outstanding transaction and then just call Database::close.  This
works for both read-only and read/write databases, takes care of
committing changes, unifies the exception handling path, and codifies
aborting outstanding transactions.  This is currently the only way to
abort an atomic section (and may remain so, since it would be
difficult to roll back things we may have cached from rolled-back
modifications).
2014-10-03 08:58:58 +02:00
Jani Nikula
f42e2e43a0 lib: actually return failures from notmuch_message_tags_to_maildir_flags
The function takes great care to preserve the first error status it
encounters, yet fails to return that status to the caller. Fix it.
2014-09-24 20:19:34 +02:00
Peter Wang
6754ad9f9e lib: bump soname
Adding return values to notmuch_database_close and
notmuch_database_destroy may require bumping the soname.
2014-09-16 20:16:31 +02:00
Gaute Hope
2c9e120e0a notmuch_thread_get_authors: document match grouping with |
as stated in thread.cc:115

/* Construct an authors string from matched_authors_array and
 * authors_array. The string contains matched authors first, then
 * non-matched authors (with the two groups separated by '|'). Within
 * each group, authors are listed in date order. */

this is, however, not reflected in the public API documentation in
notmuch.h:970. This patch a paragraph explaining how | separates the
group of authors of messages matching the query and those of messages
that do not, but are still contained in the thread.
2014-09-13 08:43:35 +02:00
Austin Clements
cca05ac10e lib: Fix endless upgrade problem
48db8c8 introduced a disagreement between when
notmuch_database_needs_upgrade returned TRUE and when
notmuch_database_upgrade actually performed an upgrade.  As a result,
if a database had a version less than 3, but no new features were
required, notmuch new would call notmuch_database_upgrade to perform
an upgrade, but notmuch_database_upgrade would return immediately
without updating the database version.  Hence, the next notmuch new
would do the same, and so on.

Fix this by ensuring that the upgrade-required logic is identical
between the two.
2014-09-01 23:06:51 -07:00
Austin Clements
658a00e7c8 lib: Update doc of notmuch_database_{needs_upgrade,upgrade}
Clients are no longer required to call these functions after opening a
database in read/write mode (which is good, because almost none of
them do!).
2014-08-30 11:42:07 -07:00
Austin Clements
ec573cd54f lib: Return an error from operations that require an upgrade
Previously, there was no protection against a caller invoking an
operation on an old database version that would effectively corrupt
the database by treating it like a newer version.

According to notmuch.h, any caller that opens the database in
read/write mode is supposed to check if the database needs upgrading
and perform an upgrade if it does.  This would protect against this,
but nobody (even the CLI) actually does this.

However, with features, it's easy to protect against incompatible
operations on a fine-grained basis.  This lightweight change allows
callers to safely operate on old database versions, while preventing
specific operations that would corrupt the database with an
informative error message.
2014-08-30 11:39:41 -07:00
Austin Clements
5dbfed4a73 lib: Support empty header values in database
Commit 567bcbc2 introduced support for storing various headers in
document values.  However, doing so in a backwards-compatible way
meant that genuinely empty header values could not be distinguished
from the old behavior of not storing the headers at all, so these
required parsing the original message.

Now that we have database features, new databases can declare that all
messages have header values, so if we have this feature flag, we can
use the stored header value even if it's the empty string.

This requires slight cleanup to notmuch_message_get_header, since the
code previously couldn't distinguish between empty headers and headers
that are never stored in the database (previously this distinction
didn't matter).
2014-08-30 11:37:33 -07:00
Austin Clements
02fec226fc lib: Report progress for combined upgrade operation
Previously, some parts of upgrade didn't report progress and for
others it was possible for the progress meter to restart at 0 part way
through the upgrade because each stage was reported separately.

Fix this by computing the total amount of work that needs to be done
up-front and updating completed work monotonically.
2014-08-30 11:36:08 -07:00
Austin Clements
e0635bd003 lib: Reorganize upgrade around document types
Rather than potentially making multiple passes over the same type of
data in the database, reorganize upgrade around each type of data that
may be upgraded.  This eliminates code duplication, will make
multi-version upgrades faster, and will let us improve progress
reporting.
2014-08-30 11:24:11 -07:00
Austin Clements
48db8c8b60 lib: Use database features to drive upgrade
Previously, we had database version information hard-coded in the
upgrade code.  Slightly re-organize the upgrade process around the set
of new database features to be enabled by the upgrade.
2014-08-30 11:21:48 -07:00
Austin Clements
4a38588488 lib: Simplify upgrade code using a transaction
Previously, the upgrade was organized as two passes -- an upgrade
pass, and a separate cleanup pass -- so the database was always in a
valid state.  This change substantially simplifies this code by
performing the upgrade in a transaction and combining both passes in
to one.  This 1) eliminates a lot of duplicate code between the
passes, 2) speeds up the upgrade process, 3) makes progress reporting
more accurate, 4) eliminates the potential for stale data if the
upgrade is interrupted during the cleanup pass, and 5) makes it easier
to reason about the safety of the upgrade code.
2014-08-30 10:45:36 -07:00
Austin Clements
8363c90531 lib: Database version 3: Introduce fine-grained "features"
Previously, our database schema was versioned by a single number.
Each database schema change had to occur "atomically" in Notmuch's
development history: before some commit, Notmuch used version N, after
that commit, it used version N+1.  Hence, each new schema version
could introduce only one change, the task of developing a schema
change fell on a single person, and it all had to happen and be
perfect in a single commit series.  This made introducing a new schema
version hard.  We've seen only two schema changes in the history of
Notmuch.

This commit introduces database schema version 3; hopefully the last
schema version we'll need for a while.  With this version, we switch
from a single version number to "features": a set of named,
independent aspects of the database schema.

Features should make backwards compatibility easier.  For many things,
it should be easy to support databases both with and without a
feature, which will allow us to make upgrades optional and will enable
"unstable" features that can be developed and tested over time.

Features also make forwards compatibility easier.  The features
recorded in a database include "compatibility flags," which can
indicate to an older version of Notmuch when it must support a given
feature to open the database for read or for write.  This lets us
replace the old vague "I don't recognize this version, so something
might go wrong, but I promise to try my best" warnings upon opening a
database with an unknown version with precise errors.  If a database
is safe to open for read/write despite unknown features, an older
version will know that and issue no message at all.  If the database
is not safe to open for read/write because of unknown features, an
older version will know that, too, and can tell the user exactly which
required features it lacks support for.
2014-08-30 10:42:08 -07:00
Michal Sojka
028c56061e Make parsing of References and In-Reply-To header less error prone
According to RFC2822 References and In-Reply-To headers are supposed
to contain one or more Message-IDs, however older RFC822 allowed
almost any content. When both References and In-Reply-To headers ends
with something else that a Message-ID (see e.g. [1]), the thread
structure presented by notmuch is incorrect. The reason is that
notmuch treats this case as if the email contained no "replyto"
information (see _notmuch_database_link_message_to_parents).

This patch changes the parse_references() function to return the last
valid Message-ID encountered rather than NULL resulting from the last
hunk of text not being the Message-ID.

[1] https://lkml.org/lkml/headers/2014/5/19/864
2014-08-16 17:45:16 -07:00
Austin Clements
0c1292051e lib: Improve documentation of _notmuch_message_create_for_message_id
Clarify the state of the returned message when
_notmuch_message_create_for_message_id returns
NOTMUCH_PRIVATE_STATUS_NO_DOCUMENT_FOUND.
2014-08-05 08:14:15 -03:00
Austin Clements
1d652c8719 lib: Fix slight misinformation in the database schema doc
The database schema documentation made it sound like each mail
document had exactly one on-disk message file, which hasn't been true
for a long time.
2014-08-04 18:58:11 -03:00
Austin Clements
30de720ba0 lib: Invalidate message metadata in _notmuch_message_gen_terms
Previously, we invalidated stored message metadata in
_notmuch_message_add_term and _notmuch_message_remove_term, but not in
_notmuch_message_gen_terms.  This doesn't currently result in any bugs
because of our limited uses of _notmuch_message_gen_terms, but it may
could cause trouble in the future.
2014-08-04 18:57:55 -03:00
Charles Celerier
df8885f62c lib: Start all function names in notmuch-private.h with
As noted in devel/STYLE, every private library function should start
with _notmuch. This patch corrects function naming that did not adhere
to this style in lib/notmuch-private.h. In particular, the old function
names that now begin with _notmuch are

    notmuch_sha1_of_file
    notmuch_sha1_of_string
    notmuch_message_file_close
    notmuch_message_file_get_header
    notmuch_message_file_open
    notmuch_message_get_author
    notmuch_message_set_author

Signed-off-by: Charles Celerier <cceleri@cs.stanford.edu>
2014-07-13 12:25:29 -03:00
Jani Nikula
ab24e883b0 lib: add return status to database close and destroy
notmuch_database_close may fail in Xapian ->flush() or ->close(), so
report the status. Similarly for notmuch_database_destroy which calls
close.

This is required for notmuch insert to report error status if message
indexing failed.
2014-07-09 20:29:36 -03:00
Austin Clements
dc64ab6720 lib: Separate all phrases indexed by _notmuch_message_gen_terms
This adds a 100 termpos gap between all phrases indexed by
_notmuch_message_gen_terms.  This fixes a bug where terms from the end
of one header and the beginning of another header could match together
in a single phrase and a separate bug where term positions of
un-prefixed terms overlapped.

This fix only affects newly indexed messages.  Messages that are
already indexed won't benefit from this fix without re-indexing, but
the fix won't make things any worse for existing messages.
2014-06-18 18:03:18 -03:00
Austin Clements
44327ca86d lib: Index name and address of from/to headers as a phrase
Previously, we indexed the name and address parts of from/to headers
with two calls to _notmuch_message_gen_terms.  In general, this
indicates that these parts are separate phrases.  However, because of
an implementation quirk, the two calls to _notmuch_message_gen_terms
generated adjacent term positions for the prefixed terms, which
happens to be the right thing to do in this case, but the wrong thing
to do for all other calls.  Furthermore, _notmuch_message_gen_terms
produced potentially overlapping term positions for the un-prefixed
copies of the terms, which is simply wrong.

This change indexes both the name and address in a single call to
_notmuch_message_gen_terms, indicating that they should be part of a
single phrase.  This masks the problem with the un-prefixed terms
(fixing the two known-broken tests) and puts us in a position to fix
the unintentionally phrases generated by other calls to
_notmuch_message_gen_terms.
2014-06-18 17:55:14 -03:00
Jani Nikula
0cc0144875 lib: resurrect support for single-message mbox files
This is effectively a revert of

commit 6812136bf5
Author: Jani Nikula <jani@nikula.org>
Date:   Mon Mar 31 00:21:48 2014 +0300

    lib: drop support for single-message mbox files

The intention was to drop support for indexing new single-message mbox
files (and whether that was a good idea in the first place is
arguable). However this inadvertently broke support for reading
headers from previously indexed single-message mbox files, which is
far worse.

Distinguishing between the two cases would require more code than
simply bringing back support for single-message mbox files.
2014-06-13 22:59:04 -03:00
David Bremner
3242e29e57 build: add canonicalize_file_name to symbols exported from libnotmuch.so
This is needed for our compat version of canonicalize_file_name to be used.
2014-04-19 05:58:34 +09:00
Jani Nikula
473930bb6f lib: replace the header parser with gmime
The notmuch library includes a full blown message header parser. Yet
the same message headers are parsed by gmime during indexing. Switch
to gmime parsing completely.

These are the main changes:

* Gmime stops header parsing at the first invalid header, and presumes
  the message body starts from there. The current parser is quite
  liberal in accepting broken headers. The change means we will be
  much pickier about accepting invalid messages.

* The current parser converts tabs used in header folding to
  spaces. Gmime preserve the tabs. Due to a broken python library used
  in mailman, there are plenty of mailing lists that produce headers
  with tabs in header folding, and we'll see plenty of tabs. (This
  change has been mitigated in preparatory patches.)

* For pure header parsing, the current parser is likely faster than
  gmime, which parses the whole message rather than just the
  headers. Since we parse the message and its headers using gmime for
  indexing anyway, this avoids and extra header parsing round when
  adding new messages. In case of duplicate messages, we'll end up
  parsing the full message although just headers would be
  sufficient. All in all this should still speed up 'notmuch new'.

* Calls to notmuch_message_get_header() may be slightly slower than
  previously for headers that are not indexed in the database, due to
  parsing of the whole message. Within the notmuch code base, notmuch
  reply is the only such user.
2014-04-05 12:53:04 -03:00
Jani Nikula
6812136bf5 lib: drop support for single-message mbox files
We've supported mbox files containing a single message for historical
reasons, but the support has been deprecated, with a warning message
while indexing, since Notmuch 0.15. Finally drop the support, and
consider all mbox files non-email.
2014-04-05 12:52:42 -03:00
Jani Nikula
1fa8e40561 lib: make folder: prefix literal
In xapian terms, convert folder: prefix from probabilistic to boolean
prefix, matching the paths, relative from the maildir root, of the
message files, ignoring the maildir new and cur leaf directories.

folder:foo matches all message files in foo, foo/new, and foo/cur.

folder:foo/new does *not* match message files in foo/new.

folder:"" matches all message files in the top level maildir and its
new and cur subdirectories.

This change constitutes a database change: bump the database version
and add database upgrade support for folder: terms. The upgrade also
adds path: terms.

Finally, fix the folder search test for literal folder: search, as
some of the folder: matching capabilities are lost in the
probabilistic to boolean prefix change.
2014-03-11 19:51:22 -03:00
Jani Nikula
59823f9642 lib: add support for path: prefix searches
The path: prefix is a literal boolean prefix matching the paths,
relative from the maildir root, of the message files.

path:foo matches all message files in foo (but not in foo/new or
foo/cur).

path:foo/new matches all message files in foo/new.

path:"" matches all message files in the top level maildir.

path:foo/** matches all message files in foo and recursively in all
subdirectories of foo.

path:** matches all message files recursively, i.e. all messages.
2014-03-11 19:51:22 -03:00
Jani Nikula
4d150eba67 lib: refactor folder term update after filename removal
Abstract some blocks of code for reuse. No functional changes.
2014-03-11 19:51:22 -03:00
Gaute Hope
91aede05a3 doc: notmuch_result_move_to_next -> notmuch_tags_move_to_next
Fix typo in docs.
2014-02-13 08:43:34 -04:00
David Bremner
3b74537f49 lib: update documentation for notmuch_database_get_directory
Clarify that using the directory after destroying the corresponding
database is not permitted.

This is implicit in the description of notmuch_database_destroy, but
it doesn't hurt to be explicit, and we do express similar "ownership"
relationships at other places in the docs.
2014-01-26 09:36:38 -04:00
David Bremner
29f1252126 lib: make notmuch_threads_valid return FALSE when passed NULL
Without this patch, the example code in the header docs crashes for certain
invalid queries (see id:871u00oimv.fsf@approx.mit.edu)
2014-01-24 20:20:54 -04:00
Tomi Valkeinen
075d53dde5 lib: fix error handling
Currently if a Xapian exception happens in notmuch_message_get_header,
the exception is not caught leading to crash. In
notmuch_message_get_date the exception is caught, but an internal error
is raised, again leading to crash.

This patch fixes the error handling by making both functions catch the
Xapian exceptions, print an error and return NULL or 0.

The 'notmuch->exception_reported' is also set, as is done elsewhere,
even if I don't really get the idea of that field.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@iki.fi>
2014-01-18 14:47:35 -04:00
Jani Nikula
f94834407b lib: fix clang compiler warning
With some combination of clang and talloc, not using the return value
of talloc_steal() produces a warning. Ignore it, as talloc_steal() has
no failure modes per documentation.
2014-01-18 14:39:51 -04:00
Jani Nikula
b65ca8e0ba lib: modify notmuch.h for automatic document generation
Minimal changes to produce a sensible result.
2014-01-05 09:05:00 -04:00
Austin Clements
3d4b0930bd lib: Bump library version from 3.0.0 to 3.1.0
This version of the library introduces LIBNOTMUCH_CHECK_VERSION and
the *_VERSION macros.  Bumping the version number is also necessary to
make the comment on LIBNOTMUCH_CHECK_VERSION no longer a lie.
2013-12-07 10:31:19 +08:00
Austin Clements
8fb16e277e lib: Replace NOTMUCH_*_VERSION with LIBNOTMUCH_*_VERSION
This makes it clear that these macros refer to the *library* version,
and not to the notmuch application-level release.  Since there are no
consumers of these macros yet, this is now or never.
2013-12-07 10:31:11 +08:00
Austin Clements
4b14ccf0d6 lib: Make VERSION macros agree with soname version
We have two distinct "library version" numbers: the soname version and
the version macros.  We need both for different reasons: the version
macros enable easy compile-time version detection (and conditional
compilation), while the soname version enables runtime version
detection (which includes the version checking done by things like the
Python bindings).

However, currently, these two version numbers are different, which is
unnecessary and can lead to confusion (especially in things like
Debian, which include the soname version in the package name).  This
patch makes them the same by bumping the version macros up to agree
with the soname version.

(We should probably keep the version number in just one place so they
can't get out of sync, but that can be done in another patch.)
2013-12-07 10:31:04 +08:00
David Bremner
b9f0e6923d util: detect byte order
Unfortunately old versions of GCC and clang do not provide byte order
macros, so we re-invent them.

If UTIL_BYTE_ORDER is not defined or defined to 0, we fall back to
macros supported by recent versions of GCC and clang
2013-11-27 07:43:29 -04:00
David Bremner
20adfe39d7 lib: fix byte order test in libsha1.c
Previously PLATFORM_BYTE_ORDER and IS_LITTLE_ENDIAN were not defined,
so the little endian code was always compiled in.

This will have the effect that the "SHA1s" on big endian architectures
will change (i.e. become actual sha1s). So someone re-indexing their
database could conceivable lose tags on messages without a message-id
header.
2013-11-25 07:55:54 -04:00
Tomi Ollila
2fd7ef64ba compact: improve error messages on failures after compaction
The error messages written during the steps replacing old
database with new now includes relevant paths and strerror.
2013-11-19 20:15:02 -04:00
Tomi Ollila
6452ae0fcb compact: unconditionally remove old wip database compact directory
In case previous notmuch compact has been interrupted there is old
work-in-progress database compact directory partially filled. Remove
it just before starting to fill the directory with new files.
2013-11-19 20:14:28 -04:00
Tomi Ollila
cb6cc296e2 compact: preserve backup database until compacted database is in place
It is less error prone and window of failure opportunity is smaller
if the old (backup) database is always renamed (instead of sometimes
rmtree'd) before new (compacted) database is put into its place.
Finally rmtree() old database in case old database backup is not kept.
2013-11-19 20:13:25 -04:00
Tomi Ollila
19a89753ca compact: catch Xapian::Error consistently
catch Xapian::Error in compact code in lib/database.cc to be consistent
with other code in addition to not making software crash on uncaught
other Xapian error.
2013-11-17 20:25:43 -04:00
Tomi Ollila
4d5986e8ad compact: tidy formatting
Notmuch compact code whitespace changes to match devel/STYLE.
2013-11-17 20:25:25 -04:00
Austin Clements
215a8dc29c lib: Document extent of some return values
This documents the extent of the notmuch_messages_t* pointers returned
by notmuch_thread_get_toplevel_messages and
notmuch_thread_get_messages.
2013-11-08 20:14:08 -04:00
Jani Nikula
060990db14 lib: add library version check macro
There have been some plans for making build incompatible changes to
the library API. This is inconvenient, but it is much more so without
a way to easily conditional build against multiple versions of
notmuch.

The macro has been lifted from glib.
2013-11-07 19:15:06 -04:00
Jani Nikula
00d2ac2b41 lib: use the compaction backup path provided by the caller
The extra path component added by the lib is a magic value that the
caller just has to know. This is demonstrated by the current code,
which indeed has "xapian.old" both sides of the interface. Use the
backup path provided by the lib caller verbatim, without adding
anything to it.
2013-11-07 06:51:16 -04:00
David Bremner
20abbe89a3 lib: update documentation of callback functions for database_compact and database_upgrade.
Compact was missing callback documentation entirely, and upgrade did not discuss the
closure parameter.
2013-11-07 06:50:50 -04:00
Jani Nikula
180dba66e4 lib: add closure parameter to compact status update callback
This provides much more flexibility for the caller.
2013-11-07 06:46:42 -04:00
Jani Nikula
35ca5feb28 lib: do not leak the database in compaction
Destroy instead of close the database after compaction, and also on
error path, to not leak the database.
2013-11-07 06:46:25 -04:00
Jani Nikula
a95dbba156 lib: check talloc success in compact
In line with the allocation checks all around.
2013-11-06 17:49:46 -04:00
Jani Nikula
8e4e537cee lib: construct compactor within try block to catch any exceptions
Constructors may also throw exceptions. Catch them.
2013-11-06 17:49:36 -04:00
Felipe Contreras
fd733a4f5c query: bind queries to database objects
The queries don't really work after a database is closed, and we would
like them to be freed if the database is destroyed.

Acknowledged-by: David Bremner <david@tethera.net>
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2013-11-02 07:20:01 -06:00
Jani Nikula
8e4900b8a7 lib: fix build on !HAVE_XAPIAN_COMPACT
Minimal change to build notmuch against xapian that doesn't have
compaction support.
2013-10-30 21:16:22 -03:00
Ben Gamari
0bd11b654e database: Add notmuch_database_compact_close
This function uses Xapian's Compactor machinery to compact the notmuch
database. The compacted database is built in a temporary directory and
later moved into place while the original uncompacted database is
preserved.

Signed-off-by: Ben Gamari <bgamari.foss@gmail.com>
2013-10-09 21:46:49 -03:00
Jani Nikula
71521f06b0 lib/cli: pass GMIME_ENABLE_RFC2047_WORKAROUNDS to g_mime_init()
As explained by Jeffrey Stedfast, the author of GMime, quoted in [1]:

> Passing the GMIME_ENABLE_RFC2047_WORKAROUNDS flag to g_mime_init()
> *should* solve the decoding problem mentioned in the thread. This
> flag should be safe to pass into g_mime_init() without any bad side
> effects and my unit tests do test that code-path.

The thread being referred to is [2].

[1] id:87bo56viyo.fsf@nikula.org
[2] id:08cb1dcd-c5db-4e33-8b09-7730cb3d59a2@gmail.com
2013-09-14 14:13:43 -03:00
Louis Rilling
a9b2135c75 tags_to_maildir_flags: Don't rename if no flags change
notmuch_message_tags_to_maildir_flags() unconditionally moves messages from
maildir directory "new/" to maildir directory "cur/", which makes messages lose
their "new" status in the MUA. However some users want to keep this "new"
status after, for instance, an auto-tagging of new messages.

However, as Austin mentioned and according to the maildir specification,
messages living in "new/" are not allowed to have flags, even if mutt allows it
to happen. For this reason, this patch prevents moving messages from "new/" to
"cur/", only if no flags have to be changed. It's hopefully enough to satisfy
mutt (and maybe other MUAs showing the "new" status) users checking the "new"
status.

Changelog:
* v2: Fix bool type as well as NULL returned despite having no errors (Austin
      Clements)
* v4: Tag the related test (contributed by Michal Sojka) as working

Signed-off-by: Louis Rilling <l.rilling@av7.net>

[Condition for keeping messages in new/ was extended to satisfy all
 tests from the previous patch. -Michal Sojka]

[Added by David Bremner, to keep the tests passing at each commit]

update insert tests for new maildir synchronization rules

As of id:1355952747-27350-4-git-send-email-sojkam1@fel.cvut.cz
we are more conservative about moving messages from ./new to ./cur.
This updates the insert tests to match
2013-09-03 20:41:51 -03:00
Jani Nikula
db465e443f lib: fix clang build
Long story short, fix build on recent (3.2+) clang.

The long story for posterity follows.

gcc 4.6 added new warnings about structs with greater visibility than
their fields. The warnings were silenced by adjusting visibility in

commit d5523ead90
Author: Carl Worth <cworth@cworth.org>
Date:   Wed May 11 13:23:13 2011 -0700

    Mark some structures in the library interface with visibility=default attribute.

Later on,

commit 3b76adf9e2
Author: Austin Clements <amdragon@MIT.EDU>
Date:   Sat Jan 14 19:17:33 2012 -0500

    lib: Add support for automatically excluding tags from queries

changed visibility of struct _notmuch_string_list for the same reason, and

commit 1a53f9f116
Author: Mark Walters <markwalters1009@gmail.com>
Date:   Thu Mar 1 22:30:38 2012 +0000

    lib: Add the exclude flag to notmuch_query_search_threads

split the struct _notmuch_string_list and its typedef
notmuch_string_list_t as a way to make a forward declaration for
_notmuch_thread_create().

The subtle difference was that the struct definition now had 'visible'
in it, while the typedef didn't, and it was within the #pragma GCC
visibility push(hidden) block. This went unnoticed, as the then common
versions of clang didn't care about this.

A later change in clang (I did not dig into when this change was
introduced) caused the following error:

CXX  -O2 lib/database.o
In file included from lib/database.cc:21:
In file included from ./lib/database-private.h:33:
./lib/notmuch-private.h:479:8: error: visibility does not match previous declaration
struct visible _notmuch_string_list {
       ^
./lib/notmuch-private.h:67:33: note: expanded from macro 'visible'
                                ^
./lib/notmuch-private.h:52:13: note: previous attribute is here
            ^
1 error generated.
make: *** [lib/database.o] Error 1

This is slightly misleading due to the reference to the #pragma. The
real culprit is the typedef within the #pragma.

We could just add 'visible' to the typedef, or move the typedef
outside of the #pragma, and be done with it, but juggle the
declarations a bit to accommodate moving the typedef back with the
struct, and keep the visibility attribute in one place.

The problem was originally reported by Simonas Kazlauskas
<s@kazlauskas.me> in id:20130418102507.GA23688@godbox but I was only
able to reproduce and investigate now that I upgraded clang.
2013-09-01 07:06:54 -03:00
Austin Clements
3fbb518335 lib: Document which strings are returned in UTF-8
Any string that ultimately comes from notmuch_message_file_get_header
is in UTF-8.
2013-08-13 17:43:34 +02:00
Austin Clements
a7bf045fb1 lib: Correct out-of-date doc comment
notmuch_message_get_header started returning some headers straight
from the database in 567bcbc, but this comment explicitly claimed all
headers were read from the message file.
2013-08-13 17:43:07 +02:00
Mark Walters
d29d7e1ee2 lib: add NOTMUCH_EXCLUDE_FLAG to notmuch_exclude_t
Add NOTMUCH_EXCLUDE_FLAG to notmuch_exclude_t so that it can
cover all four values of search --exclude in the cli.

Previously the way to avoid any message being marked excluded was to
pass in an empty list of excluded tags: since we now have an explicit
option we might as well honour it.

The enum is in a slightly strange order as the existing FALSE/TRUE
options correspond to the new
NOTMUCH_EXCLUDE_FLAG/NOTMUCH_EXCLUDE_TRUE options so this means we do
not need to bump the version number.

Indeed, an example of this is that the cli count and show still use
FALSE/TRUE and still work.
2013-06-24 22:53:16 -07:00
Tomi Ollila
8d6aa603ef cli: Guard deprecated g_type_init calls
g_type_init was deprecated in GLib 2.35.1.  In order to compile
cleanly, guard these with a suitable #if.

(commit msg from https://bugs.freedesktop.org/attachment.cgi?id=73774 )
2013-06-08 20:42:33 -03:00
Mark Walters
38698d8659 lib: add --exclude=all option
Adds a exclude all option to the lib which means that excluded
messages are completely ignored (as if they had actually been
deleted).
2013-05-13 21:32:03 -03:00
Aaron Ecay
cf8aaafbad lib/database.cc: change how the parent of a message is calculated
Presently, the code which finds the parent of a message as it is being
added to the database assumes that the first Message-ID-like substring
of the In-Reply-To header is the parent Message ID.  Some mail clients,
however, put stuff other than the Message-ID of the parent in the
In-Reply-To header, such as the email address of the sender of the
parent.  This can fool notmuch.

The updated algorithm prefers the last Message ID in the References
header.  The References header lists messages oldest-first, so the last
Message ID is the parent (RFC2822, p. 24).  The References header is
also less likely to be in a non-standard
syntax (http://cr.yp.to/immhf/thread.html,
http://www.jwz.org/doc/threading.html).  In case the References header
is not to be found, fall back to the old behavior.

V2 of this patch, incorporating feedback from Jani and (indirectly)
Austin.
2013-05-13 21:29:13 -03:00
Vladimir Marek
51b073c6f2 lib/message.cc: stale pointer bug (v3)
Xapian::TermIterator::operator* returns std::string which is destroyed
as soon as (*i).c_str() finishes. The remembered pointer 'term' then
references invalid memory.

Signed-off-by: Vladimir Marek <vlmarek@volny.cz>
2013-05-03 21:17:56 -03:00
Adam Wolfe Gordon
f55b35b3df lib: Fix name reordering to handle commas without spaces
Notmuch automatically re-orders names of the format "Last, First" to
"First Last" when the associated email address is
First.Last@example.com. But, if a name is of the format "Last,First"
then notmuch will format the name as "irst Last". Handle any number of
spaces after the comma, including none.
2013-03-29 09:24:29 -04:00
Austin Clements
f29bcc59df lib: Add an iterator over all messages in a thread
Previously, getting the list of all messages in a thread required
recursively traversing the thread's message hierarchy, which was both
difficult and resulted in messages being out of order.  This adds a
public function to retrieve an iterator over all of the messages in a
thread in oldest-first order.
2013-02-18 20:20:59 -04:00