Commit graph

544 commits

Author SHA1 Message Date
David Bremner
c040464a7c lib: wrap use of g_mime_utils_header_decode_date
This changes return type in gmime 3.0
2017-07-14 21:23:52 -03:00
David Bremner
cbb2d5608e lib/cli: replace use of g_mime_message_get_sender
This function changes semantics in gmime-3.0 so make a new function
that provides the same functionality in both
2017-07-14 17:58:09 -03:00
David Bremner
6dd00d6486 lib/index: add simple html filter
The filter just drops all (HTML) tags. As an enabling change, pass the
content type to the filter constructor so we can decide which scanner
to user.
2017-07-01 12:32:27 -03:00
David Bremner
64f81f95a1 lib/index.cc: generalize filter state machine
To match things more complicated than fixed strings, we need states
with multiple out arrows.
2017-07-01 12:32:17 -03:00
David Bremner
4a085a5137 lib/index: separate state table definition from scanner.
We want to reuse the scanner definition with a different table.  This
is mainly code movement, and making the state table part of the filter
struct/class.
2017-07-01 12:32:03 -03:00
David Bremner
20c15bc820 lib/index: generalize name of indexing filter
In followup commits we will generalize the functionality of this
filter to deal with other types of non-indexable content.
2017-07-01 12:31:55 -03:00
Jani Nikula
30c475c1ef build: visibility=default for library structs is no longer needed
Commit d5523ead90 ("Mark some structures in the library interface
with visibility=default attribute.") fixed some mixed visibility
issues with structs. With the symbol default visibility reversed, this
is no longer a problem.
2017-05-13 08:38:18 -03:00
Jani Nikula
bc11759dd1 build: switch to hiding libnotmuch symbols by default
The dynamic generation of the linker version script for libnotmuch
exports has grown rather complicated.

Reverse the visibility control by hiding symbols by default using
-fvisibility=hidden, and explicitly exporting symbols in notmuch.h
using #pragma GCC visibility. (We could also use __attribute__
((visibility ("default"))) for each exported function, but the pragma
is more convenient.)

The above is not quite enough alone, as it would "leak" a number of
weak symbols from Xapian and C++ standard library. Combine it with a
small static version script that filters out everything except the
notmuch_* symbols that we explicitly exposed, and the C++ RTTI
typeinfo symbols for exception handling.

Finally, as the symbol hiding test can no longer look at the generated
symbol table, switch the test to parse the functions from notmuch.h.
2017-05-12 07:17:18 -03:00
Jani Nikula
d5ed9af0e4 build: do not export compat functions from lib
Commits 9db2145272 ("lib/gen-version-script.h: add getline and
getdelim to notmuch.sym if needed") and 3242e29e57 ("build: add
canonicalize_file_name to symbols exported from libnotmuch.so")
started exporting compat functions from libnotmuch so that the cli
could use them. But we shouldn't export such functions from the
library. They are not part of our ABI. Instead, the cli should include
its own copies of the compat functions.
2017-05-11 20:41:10 -03:00
David Bremner
11d47950c1 lib: Add regexp expansion for for tags and paths
From a UI perspective this looks similar to what was already provided
for from, subject, and mid, but the implementation is quite
different. It uses the database's list of terms to construct a term
based query equivalent to the passed regular expression.
2017-05-09 07:44:29 -03:00
David Bremner
eab365c742 lib: Add regexp searching for mid: prefix
The bulk of the change is passing in the field options to the regexp
field processor, so that we can properly handle the
fallback (non-regexp case).
2017-05-09 07:44:15 -03:00
Fredrik Fornwall
e565118172 Replace index(3) with strchr(3)
The index(3) function has been deprecated in POSIX since 2001 and
removed in 2008, and most code in notmuch already calls strchr(3).

This fixes a compilation error on Android whose libc does not have
index(3).
2017-04-20 06:59:22 -03:00
David Bremner
e1c1d33f37 Merge branch 'release'
Another regexp search fix.
2017-03-29 20:58:34 -03:00
David Bremner
cb84f84878 lib: handle empty string in regexp field processors
The non-field processor behaviour is is convert the corresponding
queries into a search for the unprefixed terms. This yields pretty
surprising results so I decided to generate a query that would match
the terms (i.e. none with that prefix) generated for an empty header.
2017-03-29 20:44:32 -03:00
David Bremner
d877240f4e Merge branch 'release'
wildcard search fixes, plus release busywork
2017-03-25 11:51:03 -03:00
David Bremner
38a56b98f9 lib: only trigger phrase processing for regexp fields when needed
The argument is that if the string passed to the field processor has
no spaces, then the added quotes won't have any benefit except for
disabling wildcards. But disabling wildcards doesn't seem very useful
in the normal Xapian query parser, since they're stripped before
generating terms anyway. It does mean that the query 'from:"foo*"' will
not be precisely equivalent to 'from:foo' as it is for the non
field-processor version.
2017-03-24 09:24:13 -03:00
David Bremner
242d5a3ed5 lib: make notmuch_query_add_tag_exclude return a status value
Since this is an ABI breaking change, but we already bumped the SONAME
for the next release
2017-03-22 08:47:13 -03:00
David Bremner
3721bd45d7 lib: replace deprecated n_q_count_threads with status returning version
This function was deprecated in notmuch 0.21.  We re-use the name for
a status returning version, and deprecate the _st name.
2017-03-22 08:35:07 -03:00
David Bremner
5ce8e0b11b lib: replace deprecated n_q_count_messages with status returning version
This function was deprecated in notmuch 0.21.  We re-use the name for
a status returning version, and deprecate the _st name. One or two
remaining uses of the (removed) non-status returning version fixed at
the same time
2017-03-22 08:35:07 -03:00
David Bremner
86cbd215eb lib: replace deprecated n_q_search_messages with status returning version
This function was deprecated in notmuch 0.21.  We re-use the name for
a status returning version, and deprecate the _st name.
2017-03-22 08:35:07 -03:00
David Bremner
1e982de508 lib: replace n_query_search_threads with status returning version
This function was deprecated in notmuch 0.21. We finally remove the
deprecated API, and rename the status returning version to the simpler
name. The status returning is kept as a deprecated alias.
2017-03-22 08:28:09 -03:00
David Bremner
fc63c15833 lib: bump SONAME to libnotmuch5
We plan a sequence of ABI breaking changes. Put the SONAME change in a
separate commit to make reordering easier.
2017-03-22 08:27:58 -03:00
David Bremner
c39f6361d0 rename libutil.a to libnotmuch_util.a
Apparently some systems (MacOS?) have a system library called libutil
and the name conflict causes problems. Since this library is quite
notmuch specific, rename it to something less generic.
2017-03-18 21:37:43 -03:00
David Bremner
a8a2705222 Merge branch 'release'
Merge in memory fixes
2017-03-18 21:02:42 -03:00
Tomi Ollila
06adc27668 lib/message.cc: fix Coverity finding (use after free)
The object where pointer to `data` was received was deleted before
it was used in _notmuch_string_list_append().

Relevant Coverity messages follow:

3: extract
Assigning: data = std::__cxx11::string(message->doc.()).c_str(),
which extracts wrapped state from temporary of type std::__cxx11::string.

4: dtor_free
The internal representation of temporary of type std::__cxx11::string
is freed by its destructor.

5: use after free:
Wrapper object use after free (WRAPPER_ESCAPE)
Using internal representation of destroyed object local data.
2017-03-18 20:59:46 -03:00
David Bremner
62822a4e2d lib: clamp return value of g_mime_utils_header_decode_date to >=0
For reasons not completely understood at this time, gmime (as of
2.6.22) is returning a date before 1900 on bad date input. Since this
confuses some other software, we clamp such dates to 0,
i.e. 1970-01-01.
2017-03-15 21:58:25 -03:00
Jani Nikula
d56a801b67 lib/database: reduce try block scope to things that really need it
No need to maintain the pure C stuff within a try block, it's arguably
confusing. This also reduces indent for a bunch of code. No functional
changes.
2017-03-10 09:21:05 -04:00
Olly Betts
81bd72cebb lib: Fix RegexpPostingSource
Remove incorrect skipping to first match from init(), and add explicit
skip_to() and check() methods to work around xapian-core bug (the
check() method will also improve speed when filtering by one of
these).
2017-03-07 19:44:36 -04:00
David Bremner
dfacfe14f3 lib: query make exclude handling non-destructive
We filter added exclude at add time, rather than modifying the query by
count search. As noted in the comments, there are several ignored
conditions here.
2017-03-04 20:47:25 -04:00
David Bremner
e209b71873 lib: centralize query parsing, store results.
The main goal is to prepare the way for non-destructive (or at least
less destructive) exclude tag handling. It does this by having a
pre-parsed query available for further processing. This also allows us
to provide slightly more precise error messages.
2017-03-04 20:47:25 -04:00
Jani Nikula
f3edc5dc86 lib: use delete[] to free buffer allocated using new[]
Fix warning caught by clang:

lib/regexp-fields.cc:41:2: warning: 'delete' applied to a pointer that was allocated
      with 'new[]'; did you mean 'delete[]'? [-Wmismatched-new-delete]
        delete buffer;
        ^
              []
lib/regexp-fields.cc:37:17: note: allocated with 'new[]' here
        char *buffer = new char[len];
                       ^
2017-03-04 20:42:39 -04:00
David Bremner
6cb1c617a7 lib: add mid: as a synonym for id:
mid: is the url scheme suggested by URL 2392. We also plan to
introduce more flexible searches for mid: than are possible with
id: (in order not to break assumptions about the special behaviour of
id:, e.g. identifying at most one message).
2017-03-03 17:46:48 -04:00
David Bremner
55524bb063 lib: regexp matching in 'subject' and 'from'
the idea is that you can run

% notmuch search subject:/<your-favourite-regexp>/
% notmuch search from:/<your-favourite-regexp>/

or

% notmuch search subject:"your usual phrase search"
% notmuch search from:"usual phrase search"

This feature is only available with recent Xapian, specifically
support for field processors is needed.

It should work with bindings, since it extends the query parser.

This is easy to extend for other value slots, but currently the only
value slots are date, message_id, from, subject, and last_mod. Date is
already searchable;  message_id is left for a followup commit.

This was originally written by Austin Clements, and ported to Xapian
field processors (from Austin's custom query parser) by yours truly.
2017-03-03 17:46:48 -04:00
David Bremner
31b8ce4558 lib: create field processors from prefix table
This is a bit more code than hardcoding the two existing field
processors, but it should make it easy to add more.
2017-03-03 07:15:13 -04:00
David Bremner
7bd63833bf lib/message.cc: use view number to invalidate cached metadata
Currently the view number is incremented by notmuch_database_reopen
2017-02-25 21:15:38 -04:00
David Bremner
e0b22c139c lib: handle DatabaseModifiedError in _n_message_ensure_metadata
The retries are hardcoded to a small number, and error handling aborts
than propagating errors from notmuch_database_reopen. These are both
somewhat justified by the assumption that most things that can go
wrong in Xapian::Database::reopen are rare and fatal. Here's the brief
discussion with Xapian upstream:

   24-02-2017 08:12:57 < bremner> any intuition about how likely
      Xapian::Database::reopen is to fail? I'm catching a
      DatabaseModifiedError somewhere where handling any further errors is
      tricky, and wondering about treating a failed reopen as as "the
      impossible happened, stopping"

   24-02-2017 16:22:34 < olly> bremner: there should not be much scope for
    failure - stuff like out of memory or disk errors, which are probably a
    good enough excuse to stop
2017-02-25 21:13:50 -04:00
David Bremner
e17a914b77 lib: add _notmuch_database_reopen
The main expected use is to recover from a Xapian::DatabaseChanged
exception.
2017-02-25 21:09:17 -04:00
David Bremner
e0e8586fc7 Merge branch 'release'
Merge in g_hash_table read-after-free fix
2017-02-23 09:08:15 -04:00
David Bremner
884dccf293 lib: make _notmuch_message_ensure_property_map static
It's not called outside message.cc
2017-02-23 08:54:36 -04:00
David Bremner
3db9e94b0e lib: make _notmuch_message_ensure_metadata static
It's not called anywhere outside message.cc.
2017-02-23 08:54:25 -04:00
David Bremner
4e649d000b lib: fix g_hash_table related read-after-free bug
The two g_hash_table functions (insert, add) have different behaviour
with respect to existing keys. g_hash_table_insert frees the new key,
while g_hash_table_add (which is really g_hash_table_replace in
disguise) frees the existing key. With this change 'ref' is live until
the end of the function (assuming single-threaded access to
'hash'). We can't guarantee it will continue to be live in the
future (i.e. there may be a future key duplication) so we copy it with
the allocation context passed to parse_references (in practice this is
the notmuch_message_t object whose parents we are finding).

Thanks to Tomi for the simpler approach to the problem based on
reading the fine glib manual.
2017-02-22 06:28:03 -04:00
David Bremner
0e037c34dd lib: Let Xapian manage the memory for FieldProcessors
It turns out this is exactly what release() is for; Xapian will
deallocate the objects when it's done with them.
2017-02-18 22:18:06 -04:00
David Bremner
e30fa4182f lib: merge internal prefix tables
Replace multiple tables with some flags in a single table. This makes
the code in notmuch_database_open_verbose a bit shorter, and it should
also make it easier to add other options to fields, e.g. regexp
searching.
2017-02-18 22:17:39 -04:00
David Bremner
70519319b5 lib: optimize counting documents
From #xapian

olly> bremner: btw, i noticed notmuch count see ms to request all the documents and then ignores them

bremner> hmm. There's something funny about the way that notmuch uses matches in general iirc

olly> it should be able to do: mset = enquire.get_mset (0, 0, notmuch->xapian_db->get_doccount ());
...
olly> get_matches_estimated() will be exact because check_at_least is the size of the database
2017-01-27 21:54:44 -04:00
Steven Allen
4a2ce7b570 docs: fix notmuch_message_properties_value documentation
It returns the value, not the key.
2017-01-15 14:25:00 -04:00
Jani Nikula
c906da9f60 lib: use glib for sha1 digests instead of embedding libsha1
We already depend on glib both directly and indirectly (via gmime). We
might as well make use of its facilities. Drop the embedded libsha1
and use glib for sha1 digests.
2017-01-08 10:50:38 -04:00
Jani Nikula
217404ff86 lib: fix the todo comment placement on NOTMUCH_STATUS_XAPIAN_EXCEPTION
The todo comment got separated from the status it's related to at
commit 3f32fd8a1c ("Add missing comment for
NOTMUCH_STATUS_READONLY_DATABASE."). Later, commit b65ca8e0ba ("lib:
modify notmuch.h for automatic document generation") moved it, but to
the wrong place. Fix the location.
2017-01-07 08:30:08 -04:00
David Bremner
0abcad7c0e lib: optionally silence Xapian deprecation warnings
This is not ideal, but the new API is not available in Xapian 1.2.x, and
it seems to soon to depend on Xapian >= 1.4
2016-11-15 07:47:55 -04:00
David Bremner
7f07a3f0ed lib: replace deprecated xapian call 'flush()' with 'commit()'
This will make notmuch incompatible with Xapian before 1.1.0, which is
more than 6 years old this point.
2016-10-25 18:13:52 -03:00
David Bremner
b2d6f07a02 lib: document API added in 0.23
The API was already documented, but for future readers note when the
functions were added,
2016-10-06 22:46:01 -03:00