Commit graph

496 commits

Author SHA1 Message Date
David Bremner
4e649d000b lib: fix g_hash_table related read-after-free bug
The two g_hash_table functions (insert, add) have different behaviour
with respect to existing keys. g_hash_table_insert frees the new key,
while g_hash_table_add (which is really g_hash_table_replace in
disguise) frees the existing key. With this change 'ref' is live until
the end of the function (assuming single-threaded access to
'hash'). We can't guarantee it will continue to be live in the
future (i.e. there may be a future key duplication) so we copy it with
the allocation context passed to parse_references (in practice this is
the notmuch_message_t object whose parents we are finding).

Thanks to Tomi for the simpler approach to the problem based on
reading the fine glib manual.
2017-02-22 06:28:03 -04:00
David Bremner
b2d6f07a02 lib: document API added in 0.23
The API was already documented, but for future readers note when the
functions were added,
2016-10-06 22:46:01 -03:00
David Bremner
af8903df34 require xapian >= 1.2.6
It seems that no-one tried to compile without Xapian compact support
since March of 2015, since that's when I introduced a syntax error in
that branch of the ifdef.

Given the choice of maintaining this underused branch of code, or
bumping the Xapian dependency to a version from 2011, it seems
reasonable to do the latter.
2016-10-06 22:45:46 -03:00
David Bremner
c2e74662bb lib: bump minor version to mark added symbols
This should not change the SONAME, and therefore won't change the
dynamic linking behaviour, but it may help some users debug missing
symbols in case their libnotmuch is too old.
2016-10-01 22:19:07 -03:00
Tomi Ollila
1c3a8e0898 lib/database.cc: fix misleading indentation
Found by gcc 6.1.1 -Wmisleading-indentation option (set by -Wall).
2016-09-28 08:14:08 -03:00
David Bremner
514a0a6a3b lib: add talloc reference from string map iterator to map
This is needed so that when the map is modified during traversal, and
thus unlinked by the database code, the map is not disposed of until the
iterator is done with it.
2016-09-24 10:08:45 -03:00
Daniel Kahn Gillmor
693ca8d8a8 add property: query prefix to search for specific properties
We want to be able to query the properties directly, like:

   notmuch count property:foo=bar

which should return a count of messages where the property with key
"foo" has value equal to "bar".
2016-09-21 18:14:25 -03:00
David Bremner
58fe8fce1d lib: iterator API for message properties
This is a thin wrapper around the string map iterator API just introduced.
2016-09-21 18:14:25 -03:00
David Bremner
b846bdb482 lib: extend private string map API with iterators
Support for prefix based iterators is perhaps overengineering, but I
wanted to mimic the existing database_config API.
2016-09-21 18:14:24 -03:00
David Bremner
b8bb6d7964 lib: basic message-property API
Initially, support get, set and removal of single key/value pair, as
well as removing all properties.
2016-09-21 18:14:24 -03:00
David Bremner
8b03ee1d5a lib: private string map (associative array) API
The choice of array implementation is deliberate, for future iterator support
2016-09-21 18:14:24 -03:00
David Bremner
4dfb69169e lib: read "property" terms from messages.
This is a first step towards providing an API to attach
arbitrary (key,value) pairs to messages and retrieve all of the values
for a given key.
2016-09-21 18:14:24 -03:00
David Bremner
59fed50a82 lib: update cached mtime in notmuch_directory_set_mtime
Without this change, the following code fails

  notmuch_directory_set_mtime(dir, 12345);
  assert(notmuch_directory_get_mtime(dir) == 12345);
2016-08-23 20:58:46 -03:00
David Bremner
9e177b236c lib: reword comment about XFOLDER: prefix
I believe the current one is misleading, because in my experiments
Xapian did not add : when prefix and term were both upper case. Indeed,
it's hard to see how it could, because prefixes are added at a layer
above Xapian in our code. See _notmuch_message_add_term for an example.

Also try to explain why this is a good idea.  As far as I can ascertain,
this is more of an issue for a system trying to work with an unknown set
of prefixes. Since notmuch has a fixed set of prefixes, and we can
hopefully be trusted not to add XGOLD and XGOLDEN as prefixes, it is
harder for problems to arise.
2016-08-18 05:11:37 -03:00
David Bremner
293186d6c6 lib: provide _notmuch_database_log_append
_notmuch_database_log clears the log buffer each time. Rather than
introducing more complicated semantics about for this function, provide
a second function that does not clear the buffer. This is mainly a
convenience function for callers constructing complex or multi-line log
messages.

The changes to query.cc are to make sure that the common code path of
the new function is tested.
2016-08-09 09:34:11 +09:00
David Bremner
3a45d29ed4 lib: add built_with handling for XAPIAN_DB_RETRY_LOCK
This support will be present only if the appropriate version of xapian
is available _and_ the user did not disable the feature when
building. So there really needs to be some way for the user to check.
2016-06-29 09:05:49 +02:00
Istvan Marko
9b60dc3cd9 Use the Xapian::DB_RETRY_LOCK flag when available
Xapian 1.3 has introduced the DB_RETRY_LOCK flag (Xapian bug
275). Detect it in configure and optionally use it. With this flag
commands that need the write lock will wait for their turn instead of
aborting when it's not immediately available.

Amended by db: allow disabling in configure
2016-06-29 09:03:34 +02:00
David Bremner
38f0d44a82 doc: forbid further operations on a closed database
We could add many null pointer checks, but currently I don't see a use
case that justifies it.
2016-06-28 23:20:38 +02:00
David Bremner
44cfa90bdc lib: fix definition of LIBNOTMUCH_CHECK_VERSION
Fix bug reported in id:20160606124522.g2y2eazhhrwjsa4h@flatcap.org

Although the C99 standard 6.10 is a little non-obvious on this point,
the docs for e.g. gcc are unambiguous. And indeed in practice with the
extra space, this code fails

#include <stdio.h>
#define foo (x) (x+1)

int main(int argc, char **argv){
  printf("%d\n",foo(1));
}
2016-06-11 13:01:44 -03:00
David Bremner
4291f32680 lib: fix memory leak of field processor objects
The field processor objects need to be deallocated explicitly just like
the range processors (or a talloc destructor defined).
2016-06-10 09:20:22 -03:00
David Bremner
ba0b95f846 lib: document config metadata
This probably should have been part of 3458e3c89c, but I missed it.
2016-06-07 07:51:57 -03:00
Daniel Kahn Gillmor
6a833a6e83 Use https instead of http where possible
Many of the external links found in the notmuch source can be resolved
using https instead of http.  This changeset addresses as many as i
could find, without touching the e-mail corpus or expected outputs
found in tests.
2016-06-05 08:32:17 -03:00
Tomi Ollila
cf09631a45 lib: whitespace cleanup
Cleaned the following whitespace in lib/* files:

lib/index.cc:              1 line:  trailing whitespace
lib/database.cc            5 lines: 8 spaces at the beginning of line
lib/notmuch-private.h:     4 lines: 8 spaces at the beginning of line
lib/message.cc:            1 line:  trailing whitespace
lib/sha1.c:                1 line:  empty lines at the end of file
lib/query.cc:              2 lines: 8 spaces at the beginning of line
lib/gen-version-script.sh: 1 line:  trailing whitespace
2016-06-05 08:23:28 -03:00
David Bremner
b9bf3f44ea lib: add support for named queries
This relies on the optional presense of xapian field processors, and the
library config API.
2016-05-25 07:40:44 -03:00
David Bremner
30caaf52b0 lib: make a global constant for query parser flags
It's already kindof gross that this is hardcoded in two different
places. We will also need these later in field processors calling back
into the query parser.
2016-05-25 07:40:44 -03:00
David Bremner
92e59568fa lib: config list iterators
Since xapian provides the ability to restrict the iterator to a given
prefix, we expose this ability to the user. Otherwise we mimic the other
iterator interfances in notmuch (e.g. tags.c).
2016-05-25 06:51:16 -03:00
David Bremner
3458e3c89c lib: provide config API
This is a thin wrapper around the Xapian metadata API. The job of this
layer is to keep the config key value pairs from colliding with other
metadata by transparently prefixing the keys, along with the usual glue
to provide a C interface.

The split of _get_config into two functions is to allow returning of the
return value with different memory ownership semantics.
2016-05-24 08:53:03 -03:00
David Bremner
792bea5aff lib/cli: add library API / CLI for compile time options
This is intentionally low tech; if we have more than two options it may
make sense to build up what infrastructure is provided.
2016-05-13 07:29:12 -03:00
David Bremner
bbf6069252 lib: optionally support single argument date: queries
This relies on the FieldProcessor API, which is only present in xapian
>= 1.3.
2016-05-08 08:17:07 -03:00
Daniel Kahn Gillmor
e366bb2227 complete ghost-on-removal-when-shared-thread-exists
To fully complete the ghost-on-removal-when-shared-thread-exists
proposal, we need to clear all ghost messages when the last active
message is removed from a thread.

Amended by db: Remove the last test of T530, as it no longer makes sense
if we are garbage collecting ghost messages.
2016-04-15 07:13:49 -03:00
Daniel Kahn Gillmor
1695415039 On deletion, replace with ghost when other active messages in thread
There is no need to add a ghost message upon deletion if there are no
other active messages in the thread.

Also, if the message being deleted was a ghost already, we can just go
ahead and delete it.
2016-04-15 07:07:23 -03:00
Daniel Kahn Gillmor
9eebae3da4 Introduce _notmuch_message_has_term()
It can be useful to easily tell if a given message has a given term
associated with it.
2016-04-15 07:07:23 -03:00
Daniel Kahn Gillmor
011fc41d4d Add internal functions to search for alternate doc types
Publicly we are only exposing the non-ghost documents (of "type"
"mail").  But internally we might want to inspect the ghost messages
as well.

This changeset adds two new private interfaces to queries to recover
information about alternate document types.
2016-04-15 07:07:23 -03:00
Daniel Kahn Gillmor
604d1e0977 fix thread breakage via ghost-on-removal
implement ghost-on-removal, the solution to T590-thread-breakage.sh
that just adds a ghost message after removing each message.

It leaks information about whether we've ever seen a given message id,
but it's a fairly simple implementation.

Note that _resolve_message_id_to_thread_id already introduces new
message_ids to the database, so i think just searching for a given
message ID may introduce the same metadata leakage.
2016-04-15 07:07:23 -03:00
Jani Nikula
54aeab1962 lib: clean up _notmuch_database_split_path
Make the logic it a bit easier to read. No functional changes.
2016-04-12 20:46:42 -03:00
Jani Nikula
a352d9ceaa lib: fix handling of one character long directory names at top level
The code to skip multiple slashes in _notmuch_database_split_path()
skips back one character too much. This is compensated by a +1 in the
length parameter to the strndup() call. Mostly this works fine, but if
the path is to a file under a top level directory with one character
long name, the directory part is mistaken to be part of the file name
(slash == path in code). The returned directory name will be the empty
string and the basename will be the full path, breaking the indexing
logic in notmuch new.

Fix the multiple slash skipping to keep the slash variable pointing at
the last slash, and adjust strndup() accordingly.

The bug was introduced in

commit e890b0cf40
Author: Carl Worth <cworth@cworth.org>
Date:   Sat Dec 19 13:20:26 2009 -0800

    database: Store the parent ID for each directory document.

just a little over two months after the initial commit in the Notmuch
code history, making this the longest living bug in Notmuch to date.
2016-04-12 20:40:19 -03:00
Tomi Ollila
342910a280 lib: NOTMUCH_DEPRECATED macro also for older compilers
Some compilers (older than gcc 4.5 and clang 2.9) do support
__attribute__ ((deprecated)) but not
__attribute__ ((deprecated("message"))).

Check if clang version is at least 3.0, or gcc version
is at least 4.5 to define NOTMUCH_DEPRECATED as the
latter variant above. Otherwise define NOTMUCH_DEPRECATED
as the former variant above.

For a bit simpler implementation clang 2.9 is not included
to use the newer variant. It is just one release, and the
older one works fine. Clang 3.0 was released around 2011-11
and gcc 5.1 2015-04-22 (therefore newer macro for gcc 4.5+)
2016-03-14 19:54:32 -03:00
Daniel Kahn Gillmor
07b6220a55 clean up stray apostrophe in comment
This is a nit-picky orthographical fix for an nit-picky ontological
comment.
2016-01-16 08:17:15 -04:00
Daniel Kahn Gillmor
e038b95ffe correct comment referring to notmuch_database_remove_message
notmuch_database_remove_message has no leading underscore in its name.
2016-01-16 08:16:51 -04:00
Steven Allen
c946356cdc forbid atomic transactions on writable, upgradable databases
We can't (but currently do) allow upgrades within transactions because
upgrades need their own transactions. We don't want to re-use the
current transaction because bailing out of an upgrade would mean loosing
all previous changes (because our "atomic" transactions don't commit
before hand). This gives us two options:

1. Fail at the beginning of upgrade (tell the user to end the
   transaction, upgrade, and start over).
2. Don't allow the user to start the transaction.

I went with the latter because:

1. There is no reason to call `begin_atomic` unless you intend to to
   write to the database and anyone intending to write to the database
   should upgrade it first.
2. This means that nothing inside an atomic transaction can ever fail
   with NOTMUCH_STATUS_UPGRADE_REQUIRED.
2015-11-23 08:15:37 -04:00
Jani Nikula
506b81679a lib: content disposition values are not case-sensitive
Per RFC 2183, the values for Content-Disposition values are not
case-sensitive. While at it, use the gmime function for getting at the
disposition string instead of referencing the field directly.

This fixes "attachment" tagging and filename term generation for
attachments while indexing.
2015-11-19 07:47:29 -04:00
Steven Allen
10e933a3bb Documentation: fix type name spelling 2015-10-27 08:07:31 -03:00
Jani Nikula
727fcd18c6 lib: add interface to delete directory documents
As mentioned in acd66cdec0 we don't have
an interface to delete directory documents, and they're left behind. Add
the interface.
2015-10-10 09:14:25 -03:00
David Bremner
7a20f26f91 lib: update doxygen comments to add @since for the new _st API
We should probably to this for all new functions introduced from now on.
2015-10-05 20:16:59 -03:00
David Bremner
378ba492a6 lib: migrate thread.cc to new query_search API
here we rely on thread_id_query being attached to the local talloc
context, so no new cleanup code is needed.
2015-10-05 19:53:53 -03:00
David Bremner
2501c2565c lib: migrate notmuch_database_upgrade to new query_search API
Here we depend on the error path cleaning up query
2015-10-05 19:53:11 -03:00
David Bremner
87ee9a53e3 lib: add versions of n_q_count_{message,threads} with status return
Although I think it's a pretty bad idea to continue using the old API,
this allows both a more gentle transition for clients of the library,
and allows us to break one monolithic change into a series
2015-10-05 19:44:07 -03:00
David Bremner
65a6b86873 lib: move query variable to function scope
This is a prelude to deallocating it (if necessary) on the error path.
2015-10-05 19:39:11 -03:00
Jani Nikula
23b8ed610a lib: add support for date:<expr>..! to mean date:<expr>..<expr>
It doesn't seem likely we can support simple date:<expr> expanding to
date:<expr>..<expr> any time soon. (This can be done with a future
version of Xapian, or with a custom query query parser.) In the mean
time, provide shorthand date:<expr>..! to mean the same. This is
useful, as the expansion takes place before interpetation, and we can
use, for example, date:yesterday..! to match from beginning of
yesterday to end of yesterday.

Idea from Mark Walters <markwalters1009@gmail.com>.
2015-09-25 21:55:24 -03:00
David Bremner
93ee4faa4d lib: constify arguments to notmuch_query_get_*
These functions are all just accessors, and it's pretty clear they don't
modify the query struct. This also fixes one warning I created when I
introduced status.c.
2015-09-23 08:58:19 -03:00