Commit graph

710 commits

Author SHA1 Message Date
Tomi Ollila
00cdfe1071 build: drop support for xapian versions less than 1.4
Xapian 1.4 is over 3 years old now (1.4.0 released 2016-06-24),
and 1.2 has been deprecated in Notmuch version 0.27 (2018-06-13).

Xapian 1.4 supports compaction, field processors and retry locking;
conditionals checking compaction and field processors were removed
but user may want to disable retry locking at configure time so it
is kept.
2020-04-23 21:28:45 -03:00
Daniel Kahn Gillmor
f2a85904f1 Correct doxygen framing for libnotmuch.h
Apparently doxygen needs its comments formatted in a specific way to
notice that the group is closed.

Without this fix, with doxygen 1.8.16-2 we see:

```
doxygen ./doc/doxygen.cfg
…/notmuch/lib/notmuch.h:2322: warning: end of file while inside a group
```

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-03-19 21:56:05 -03:00
David Bremner
dc2b5a031b notmuch release 0.29.3-1 for unstable (sid) [dgit]
[dgit distro=debian no-split --quilt=linear]
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEkiyHYXwaY0SiY6fqA0U5G1WqFSEFAl3eb60ACgkQA0U5G1Wq
 FSE1Ug//Wxc7AYeHc9ErjBOkK/oQjagGOI6cKsovHfyHDRqVULsDfGt/at7wBuS2
 7V1sfU5zttoNmltmamkao4T+f/dH70D5UJxWXu1vM0QDnKz+1U4g5jTGV9sDbqwA
 34XtewzZ4508GjmmhhCHLkwQN1zplI1phGfhkwDCWrWc5er8Yat9EroHvgJ2GYcA
 fvMe13hODPC4/R5jPyKHKERyFrcj5oeZPOAV6pnaNKq1qeNgOpKm1usldl73rqe5
 ehrqF5ZgMr7wVjrnK/9wV4x4FTUKTgjweLzwQ3pWkYj3tzDx5KBBM47IrIKwoLh1
 /IS4PY9znohi/Nyl8IScmIeDAVj6PYITj0lqaDCL8x0vTkuZ5f8sFiJORZcmderk
 CylvJUahr/shdV1YolBsh1TfavP4eKIz77MkfO77uTyZACPIyZF0iHlV+me8ixIJ
 IDi2BJ0Sp+pky6/211iJtbf43CUnXATfr3COraLtT/xEKfENA63lvJRL4y+6ahmr
 BrGNfUZTvnY1+K9ym8dm3wfLe49TThMM+zbkY+2b25puO6u7cIFXgjBhhIwC4rdY
 bCd3RewBngge8NzvHw//k5+fd4s7b2BfA/6rLGWlEm4g6eik2IALtDU9Qm3L1twS
 RVr1yLoXICoJNLX7hZlgaF3iv+LzMrWkyKgjaLKiZ8HYnDwKXJg=
 =qUN/
 -----END PGP SIGNATURE-----

Merge tag 'debian/0.29.3-1'

notmuch release 0.29.3-1 for unstable (sid) [dgit]

[dgit distro=debian no-split --quilt=linear]
2019-11-27 08:45:43 -04:00
David Bremner
8e22514842 lib: fix memory error in notmuch_config_list_value
The documentation for notmuch_config_list_key warns that that the
returned value will be destroyed by the next call to
notmuch_config_list_key, but it neglected to mention that calling
notmuch_config_list_value would also destroy it (by calling
notmuch_config_list_key). This is surprising, and caused a use after
free bug in _setup_user_query_fields (first noticed by an OpenBSD
porter, so kudos to the OpenBSD malloc implementation).  This change
fixes that use-after-free bug.
2019-11-27 07:58:09 -04:00
Daniel Kahn Gillmor
4b1a8fd183 index: repair "Mixed Up" messages before indexing.
When encountering a message that has been mangled in the "mixed up"
way by an intermediate MTA, notmuch should instead repair it and index
the repaired form.

When it does this, it also associates the index.repaired=mixedup
property with the message.  If a problem is found with this repair
process, or an improved repair process is proposed later, this should
make it easy for people to reindex the relevant message.  The property
will also hopefully make it easier to diagnose this particular problem
in the future.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-09-15 19:07:06 -04:00
Daniel Kahn Gillmor
9829533e92 index: avoid indexing legacy-display parts
When we notice a legacy-display part during indexing, it makes more
sense to avoid indexing it as part of the message body.

Given that the protected subject will already be indexed, there is no
need to index this part at all, so we skip over it.

If this happens during indexing, we set a property on the message:
index.repaired=skip-protected-headers-legacy-display

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-09-01 08:45:30 -03:00
Daniel Kahn Gillmor
ab0ae8b1c0 util/crypto: _n_m_crypto_potential_payload returns whether part is the payload
Our _notmuch_message_crypto_potential_payload implementation could
only return a failure if bad arguments were passed to it.  It is an
internal function, so if that happens it's an entirely internal bug
for notmuch.

It will be more useful for this function to return whether or not the
part is in fact a cryptographic payload, so we dispense with the
status return.

If some future change suggests adding a status return back, there are
only a handful of call sites, and no pressure to retain a stable API,
so it could be changed easily. But for now, go with the simpler
function.

We will use this return value in future patches, to make different
decisions based on whether a part is the cryptographic payload or not.
But for now, we just leave the places where it gets invoked marked
with (void) to show that the result is ignored.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-09-01 08:38:11 -03:00
Daniel Kahn Gillmor
1b29822cf5 repair: set up codebase for repair functionality
This adds no functionality directly, but is a useful starting point
for adding new repair functionality.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-09-01 08:20:25 -03:00
uncrustify
2b62ca2e3b lib: run uncrustify
This is the result of running

     $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc

in the lib directory
2019-06-14 07:41:27 -03:00
Daniel Kahn Gillmor
bcee870826 fix misspelling
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-06-11 07:24:31 -03:00
Daniel Kahn Gillmor
5c3a44681f indexing: record protected subject when indexing cleartext
When indexing the cleartext of an encrypted message, record any
protected subject in the database, which should make it findable and
visible in search.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-29 08:14:44 -03:00
David Bremner
adb53b0737 lib/database: index user headers.
This essentially involves calling _notmuch_message_gen_terms once for
each user defined header.
2019-05-25 07:21:13 -03:00
David Bremner
75bdce7952 lib: support user prefix names in term generation
This should not change the indexing process yet as nothing calls
_notmuch_message_gen_terms with a user prefix name. On the other hand,
it should not break anything either.

_notmuch_database_prefix does a linear walk of the list of (built-in)
prefixes, followed by a logarithmic time search of the list of user
prefixes. The latter is probably not really noticable.
2019-05-25 07:17:27 -03:00
David Bremner
b52cda90f0 lib: cache user prefixes in database object
This will be used to avoid needing a database access to resolve a db
prefix from the corresponding UI prefix (e.g. when indexing). Arguably
the setup of the separate header map does not belong here, since it is
about indexing rather than querying, but we currently don't have any
other indexing setup to do.
2019-05-25 07:08:20 -03:00
David Bremner
575493e785 lib: setup user headers in query parser
These tests will need to be updated if the Xapian
query print/debug format changes.
2019-05-25 06:56:16 -03:00
David Bremner
97939170b3 n_m_remove_indexed_terms: reduce number of Xapian API calls.
Previously this functioned scanned every term attached to a given
Xapian document. It turns out we know how to read only the terms we
need to preserve (and we might have already done so). This commit
replaces many calls to Xapian::Document::remove_term with one call to
::clear_terms, and a (typically much smaller) number of calls to
::add_term. Roughly speaking this is based on the assumption that most
messages have more text than they have tags.

According to the performance test suite, this yields a roughly 40%
speedup on "notmuch reindex '*'"
2019-05-23 08:00:56 -03:00
David Bremner
e19954fa18 lib/message-file: close stream in destructor
Without this,

$ make time-test OPTIONS=--small

leads to fatal errors from too many open files.

Thanks to st-gourichon-fid for bringing this problem to my attention in IRC.
2019-05-10 12:26:50 -03:00
David Bremner
852167479f lib/message_file: open gzipped files
Rather than storing the lower level stdio FILE object, we store a
GMime stream. This allows both transparent decompression, and passing
the stream into GMime for parsing. As a side effect, we can let GMime
close the underlying OS stream (indeed, that stream isn't visible here
anymore).

This change is enough to get notmuch-{new,search} working, but there is still
some work required for notmuch-show, to be done in a following commit.
2019-05-03 07:48:43 -03:00
Daniel Kahn Gillmor
e9b870b692 gmime-cleanup: pass NULL as default GMimeParserOptions
This is a functional change, not a straight translation, because we
are no longer directly invoking g_mime_parser_options_get_default(),
but the GMime source has indicated that the options parameter for
g_mime_parser_construct_message() is "nullable" since upstream commit
d0ebdd2ea3e6fa635a2a551c846e9bc8b6040353 (which itself precedes GMime
3.0).

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-03 06:58:00 -03:00
Daniel Kahn Gillmor
bbe3015b3e gmime-cleanup: pass NULL arguments explicitly where GMime 3.0 expects it
Several GMime 2.6 functions sprouted a change in the argument order in
GMime 3.0.  We had a compatibility layer here to be able to handle
compiling against both GMime 2.6 and 3.0.  Now that we're using 3.0
only, rip out the compatibility layer for those functions with changed
argument lists, and explicitly use the 3.0 argument lists.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-03 06:57:27 -03:00
Daniel Kahn Gillmor
582f255aeb gmime-cleanup: use GMime 3.0 function names
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-03 06:57:16 -03:00
Daniel Kahn Gillmor
58ee5d1bb5 gmime-cleanup: drop unused gmime #defines and simplify g_mime_init ()
Several of these #defines were not actually used in the notmuch
codebase any longer.  And as of GMime 3.0, g_mime_init takes no
arguments, so we can also drop the bogus RFC2047 argument that we were
passing and then #defining away.

signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-03 06:56:58 -03:00
Daniel Kahn Gillmor
b7ac4c05e1 gmime-cleanup: drop all arguments unused in GMime 3
This means dropping GMimeCryptoContext and notmuch_config arguments.

All the argument changes are to internal functions, so this is not an
API or ABI break.

We also get to drop the #define for g_mime_3_unused.

signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-03 06:56:38 -03:00
Daniel Kahn Gillmor
591a0787c2 gmime-cleanup: drop g_mime_2_6_unref
signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-03 06:56:28 -03:00
Daniel Kahn Gillmor
bb0b119358 gmime-cleanup: always support session keys
Our minimum version of GMime 3.0 always supports good session key
handling.

signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-03 06:55:32 -03:00
Daniel Kahn Gillmor
35e21bfb6f gmime-cleanup: remove GMime 2.6 variant codeblocks
signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-03 06:50:40 -03:00
Daniel Kahn Gillmor
f6da475e35 gmime-cleanup: drop unused gmime 2.6 content_type from _index_encrypted_mime_part
In _index_mime_part, we don't need to extract the content-type from
the part until just before we use it, so we also defer it lazily.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-03 06:50:27 -03:00
David Bremner
319dd95ebb lib: add 'body:' field, stop indexing headers twice.
The new `body:` field (in Xapian terms) or prefix (in slightly
sloppier notmuch) terms allows matching terms that occur only in the
body.

Unprefixed query terms should continue to match anywhere (header or
body) in the message.

This follows a suggestion of Olly Betts to use the facility (since
Xapian 1.0.4) to add the same field with multiple prefixes. The double
indexing of previous versions is thus replaced with a query time
expension of unprefixed query terms to the various prefixed
equivalent.

Reindexing will be needed for 'body:' searches to work correctly;
otherwise they will also match messages where the term occur in
headers (demonstrated by the new tests in T530-upgrade.sh)
2019-04-17 08:48:16 -03:00
David Bremner
b22386f353 lib: update commentary about path/folder terms
We missed this when we changed to binary fields.
2019-03-31 12:00:30 -03:00
David Bremner
6b0cf9b21a lib: add clarification about the use of "prefix" in the docs. 2019-03-31 11:59:59 -03:00
David Bremner
2717ff96a7 lib: drop comment about only indexing one file.
Although the situation is complicated by the value fields (which are
taken from a single file), this comment is now more false than true.
2019-03-31 11:59:46 -03:00
David Bremner
d25dcc589c lib: use phrase search for anything not ending in '*'
Anything that does not look like a wildcard should be safe to
quote. This should fix the problem searching for xapian keywords.
2019-03-28 14:34:37 -03:00
Luis Ressel
9f7e851263 Prepend regerror() messages with "regexp error: "
The exact error messages returned by regerror() aren't standardized;
relying on them isn't portable. Thus, add a a prefix to make clear that
the subsequent message is a regexp parsing error, and only look for this
prefix in the test suite, ignoring the rest of the message.
2019-03-11 22:24:55 -03:00
David Bremner
71eaa19350 Merge branch 'release'
Changes from 0.28.3
2019-03-06 08:53:26 -04:00
David Bremner
e88297c072 lib/string_map: fix return type of string_cmp
I can't figure out how checking the sign of a bool ever worked. The
following program demonstrates the problem (i.e. for me it prints 1).

 #include <stdio.h>
 #include <stdbool.h>
 int main(int argc, char **argv) {
    bool x;
    x = -1;
    printf("x = %d\n", x);
 }

This seems to be mandated by the C99 standard 6.3.1.2.
2019-03-05 21:46:41 -04:00
rhn
fac155815c docs: Use correct call to notmuch_query_search_threads in usage example
Amended by db: simplify (subjectively) the example.
2019-01-25 20:51:00 -04:00
rhn
45639881b5 lib: Explicitly state when replies will be destroyed
Without an explicit guarantee, it's not clear how to use the reference.
2019-01-25 20:34:57 -04:00
Daniel Kahn Gillmor
f5411574af index: explicitly follow GObject conventions
Use explicit labels for GTypeInfo member initializers, rather than
relying on comments and ordering.  This is both easier to read, and
harder to screw up.  This also makes it clear that we're mis-casting
GObject class initializers for gcc.

Without this patch, g++ 8.2.0-7 produces this warning:

CXX  -g -O2 lib/index.o
lib/index.cc: In function ‘GMimeFilter* notmuch_filter_discard_non_term_new(GMimeContentType*)’:
lib/index.cc:252:23: warning: cast between incompatible function types from ‘void (*)(NotmuchFilterDiscardNonTermClass*)’ {aka ‘void (*)(_NotmuchFilterDiscardNonTermClass*)’} to ‘GClassInitFunc’ {aka ‘void (*)(void*, void*)’} [-Wcast-function-type]
      (GClassInitFunc) notmuch_filter_discard_non_term_class_init,
                       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The definition of GClassInitFunc in
/usr/include/glib-2.0/gobject/gtype.h suggests that this function will
always be called with the class_data member of the GTypeInfo.  We set
that value to NULL in both GObject definitions in notmuch. So we mark
it as explicitly unused.

There is no functional change here, just code cleanup.
2018-10-21 10:21:26 -03:00
David Bremner
87934c432c lib: change parent strategy to use In-Reply-To if it looks sane
As reported by Sean Whitton, there are mailers (in particular the
Debian Bug Tracking System) that have sensible In-Reply-To headers,
but un-useful-for-notmuch References (in particular with the BTS, the
oldest reference is last). I looked at a sample of about 200K
messages, and only about 0.5% these had something other than a single
message-id in In-Reply-To. On this basis, if we see a single
message-id in In-Reply-To, consider that as authoritative.
2018-09-06 08:07:13 -03:00
David Bremner
b31e44c678 lib: add _notmuch_message_id_parse_strict
The idea is that if a message-id parses with this function, the MUA
generating it was probably sane, and in particular it's probably safe
to use the result as a parent from In-Reply-to.
2018-09-06 08:07:13 -03:00
David Bremner
46dce33abc lib/thread: change _resolve_thread_relationships to use depths
We (finally) implement the XXX comment. It requires a bit of care not
to reparent all of the possible toplevel messages.

_notmuch_messages_has_next is not ready to be a public function yet,
since it punts on the mset case. We know in the one case it is called,
the notmuch_messages_t is just a regular list / iterator.
2018-09-06 08:07:13 -03:00
David Bremner
21803df6ae lib/thread: rewrite _parent_or_toplevel to use depths
This is part 1/2 of changing the reparenting of alleged toplevel
messages to use a "deep" reference rather than just the first one
found.
2018-09-06 08:07:13 -03:00
David Bremner
0a7181dd16 lib: calculate message depth in thread
This will be used in reparenting messages without useful in-reply-to,
but with useful references
2018-09-06 08:07:13 -03:00
David Bremner
a330858284 lib/thread: initial use of references as for fallback parenting
This is mainly to lay out the structure of the final code. The problem
isn't really solved yet, although some very simple cases are
better (hence the fixed test). We need two passes through the messages
because we need to be careful not to re-parent too many messages and
end up without any toplevel messages.
2018-09-06 08:07:13 -03:00
David Bremner
ac2146118b use EMPTY_STRING in _parent_via_in_reply_to
This is a review suggestion [1] of Tomi. I decided not to squash it
so that the code movement remains clear.

[1]: id:m2pnxxgf5q.fsf@guru.guru-group.fi
2018-09-06 08:07:13 -03:00
David Bremner
b9d4eb0412 lib/thread: refactor in_reply_to test
This is not a complete win in code-size, but it makes the code (which
is about to get more complicated) easier to follow.
2018-09-06 08:07:13 -03:00
David Bremner
dc3cc18bf0 lib: add _notmuch_message_list_empty
There is no public notmuch_message_list_t public interface, so to this
is added to the private API. We use it immediately in thread.cc;
future commits will use it further.
2018-09-06 08:07:13 -03:00
David Bremner
040fd630bf lib/thread: add macro for debug printing of threading
This is analogous to DEBUG_DATABASE_SANITY, and is intended to help
debugging and to help users submit bug reports.
2018-09-06 08:07:12 -03:00
David Bremner
d0b844b358 lib: read reference terms into message struct.
The plan is to use these in resolving threads.
2018-09-06 08:07:12 -03:00
David Bremner
9b568e73e1 lib/thread: sort sibling messages by date
For non-root messages, this should not should anything currently, as
the messages are already added in date order. In the future we will
add some non-root messages in a second pass out of order and the
sorting will be useful. It does fix the order of multiple
root-messages (although it is overkill for that).
2018-09-06 08:07:12 -03:00
Daniel Kahn Gillmor
6a9f26b4a0 lib: make notmuch_message_get_database() take a const notmuch_message_t*
This is technically an API change, but it is not an ABI change, and
it's merely a statement that limits what the library can do.

This is in parallel to notmuch_query_get_database(), which also takes
a const pointer.
2018-05-26 07:32:01 -07:00
Daniel Kahn Gillmor
499bb78178 properties: add notmuch_message_count_properties
The user can already do this manually, of course, but (a) it's nice to
have a convenience function, and (b) exposing this interface means
that someone more clever with a _notmuch_string_map_t than i am can
write a more efficient version if they like, and it will just
accelerate the users of the convenience function.
2018-05-26 07:31:39 -07:00
David Bremner
4a6371f1d8 lib: bump minor version
This recognizes the addition of (at least)
notmuch_message_get_database to the API.
2018-05-26 07:31:27 -07:00
Daniel Kahn Gillmor
9088db76d8 lib: expose notmuch_message_get_database()
We've had _notmuch_message_database() internally for a while, and it's
useful.  It turns out to be useful on the other side of the library
interface as well (i'll use it later in this series for "notmuch
show"), so we expose it publicly now.
2018-05-26 07:30:32 -07:00
David Bremner
1883ab6650 drop use of register keyword
The performance benefits are dubious, and it's deprecated in C++11.
2018-05-14 22:18:05 -03:00
David Bremner
f0131af6c5 lib: define specialized get_thread_id for use in thread subquery
The observation is that we are only using the messages to get there
thread_id, which is kindof a pessimal access pattern for the current
notmuch_message_get_thread_id
2018-05-07 08:42:53 -03:00
David Bremner
5ad39ebf75 lib: add thread subqueries.
This change allows queries of the form

 thread:{from:me} and thread:{from:jian} and not thread:{from:dave}

This is still somewhat brute-force, but it's a big improvement over
both the shell script solution and the previous proposal [1], because it
does not build the whole thread structure just generate a
query. A further potential optimization is to replace the calls to
notmuch with more specialized Xapian code; in particular it's not
likely that reading all of the message metadata is a win here.

[1]: id:20170820213240.20526-1-david@tethera.net
2018-05-07 08:42:53 -03:00
Daniel Kahn Gillmor
c20a5eb805 move more http -> https
Correct URLs that have crept into the notmuch codebase with http://
when https:// is possible.

As part of this conversion, this changeset also indicates the current
preferred upstream URLs for both gmime and sup.  the new URLs are
https-enabled, the old ones are not.

This also fixes T310-emacs.sh, thanks to Bremner for catching it.
2018-05-03 20:59:20 -03:00
David Bremner
388edce0b7 Merge branch 'release'
minimal mset fix, for 0.26.2
2018-04-26 22:47:40 -03:00
David Bremner
64831e8016 lib: work around xapian bug with get_mset(0,0, x)
At least Fedora28 triggers this Xapian bug due to some toolchain change .

   https://bugzilla.redhat.com/show_bug.cgi?id=1546162

The underlying bug is fixed in xapian commit f92e2a936c1592, and
should be fixed in Xapian 1.4.6
2018-04-26 22:14:59 -03:00
David Bremner
963ccabe93 Merge branch 'release'
reference loop fixes to be included in 0.26.2
2018-04-25 17:36:32 -03:00
David Bremner
491b1f4b40 lib: choose oldest message when breaking reference loops
This preserves a sensible thread order
2018-04-23 23:00:20 -03:00
David Bremner
9293d6da27 lib: break reference loop by choosing arbitrary top level msg
Other parts of notmuch (e.g. notmuch show) expect each thread to
contain at least one top level message, and crash if this expectation
is not met.
2018-04-20 11:23:31 -03:00
David Bremner
5d510221d1 Merge branch 'release' 2018-04-07 17:43:01 -03:00
David Bremner
920f5d925e lib: bump LIBRARY_MINOR_VERSION
We added several new functions, at least

   notmuch_database_get_default_indexopts
   notmuch_database_index_file
   notmuch_indexopts_destroy
   notmuch_indexopts_get_decrypt_policy
   notmuch_indexopts_set_decrypt_policy
   notmuch_message_count_files
   notmuch_message_has_maildir_flag
   notmuch_message_reindex
   notmuch_message_remove_all_properties_with_prefix
   notmuch_thread_get_total_files
2018-04-02 08:06:53 -03:00
Daniel Kahn Gillmor
54982e520c fix typos 2018-01-04 20:35:58 -04:00
David Bremner
b09025bce2 Revert "lib: add thread subqueries."
This reverts commit 4f5bbaf7e2.
2017-12-28 10:05:55 -04:00
David Bremner
4f5bbaf7e2 lib: add thread subqueries.
This change allows queries of the form

 thread:{from:me} and thread:{from:jian} and not thread:{from:dave}

This is still somewhat brute-force, but it's a big improvement over
both the shell script solution and the previous proposal [1], because it
does not build the whole thread structure just generate a
query. A further potential optimization is to replace the calls to
notmuch with more specialized Xapian code; in particular it's not
likely that reading all of the message metadata is a win here.

[1]: id:20170820213240.20526-1-david@tethera.net
2017-12-25 20:40:28 -04:00
David Bremner
7cfa1c6961 lib: return "" rather than NULL from notmuch_thread_get_authors
The current behaviour is at best under-documented. The modified test in
T470-missing-headers.sh previously relied on printf doing the right
thing with NULL, which seems icky.

The use of talloc_strdup here is probably overkill, but it avoids
having to enforce that thread->authors is never mutated outside
_resolve_thread_authors_string.
2017-12-21 09:22:30 -04:00
Daniel Kahn Gillmor
fccebbaeef crypto: add --decrypt=nostash to avoid stashing session keys
Here's the configuration choice for people who want a cleartext index,
but don't want stashed session keys.

Interestingly, this "nostash" decryption policy is actually the same
policy that should be used by "notmuch show" and "notmuch reply",
since they never modify the index or database when they are invoked
with --decrypt.

We take advantage of this parallel to tune the behavior of those
programs so that we're not requesting session keys from GnuPG during
"show" and "reply" that we would then otherwise just throw away.
2017-12-08 08:08:47 -04:00
Daniel Kahn Gillmor
29648a137c crypto: actually stash session keys when decrypt=true
If you're going to store the cleartext index of an encrypted message,
in most situations you might just as well store the session key.
Doing this storage has efficiency and recoverability advantages.

Combined with a schedule of regular OpenPGP subkey rotation and
destruction, this can also offer security benefits, like "deletable
e-mail", which is the store-and-forward analog to "forward secrecy".

But wait, i hear you saying, i have a special need to store cleartext
indexes but it's really bad for me to store session keys!  Maybe
(let's imagine) i get lots of e-mails with incriminating photos
attached, and i want to be able to search for them by the text in the
e-mail, but i don't want someone with access to the index to be
actually able to see the photos themselves.

Fret not, the next patch in this series will support your wacky
uncommon use case.
2017-12-08 08:08:47 -04:00
Daniel Kahn Gillmor
6a9626a2fd cli/reindex: destroy stashed session keys when --decrypt=false
There are some situations where the user wants to get rid of the
cleartext index of a message.  For example, if they're indexing
encrypted messages normally, but suddenly they run across a message
that they really don't want any trace of in their index.

In that case, the natural thing to do is:

   notmuch reindex --decrypt=false id:whatever@example.biz

But of course, clearing the cleartext index without clearing the
stashed session key is just silly.  So we do the expected thing and
also destroy any stashed session keys while we're destroying the index
of the cleartext.

Note that stashed session keys are stored in the xapian database, but
xapian does not currently allow safe deletion (see
https://trac.xapian.org/ticket/742).

As a workaround, after removing session keys and cleartext material
from the database, the user probably should do something like "notmuch
compact" to try to purge whatever recoverable data is left in the
xapian freelist.  This problem really needs to be addressed within
xapian, though, if we want it fixed right.
2017-12-08 08:08:47 -04:00
Daniel Kahn Gillmor
076f86025d cli/new, insert, reindex: change index.decrypt to "auto" by default
The new "auto" decryption policy is not only good for "notmuch show"
and "notmuch reindex".  It's also useful for indexing messages --
there's no good reason to not try to go ahead and index the cleartext
of a message that we have a stashed session key for.

This change updates the defaults and tunes the test suite to make sure
that they have taken effect.
2017-12-08 08:08:46 -04:00
Daniel Kahn Gillmor
d137170b23 crypto: record whether an actual decryption attempt happened
In our consolidation of _notmuch_crypto_decrypt, the callers lost
track a little bit of whether any actual decryption was attempted.

Now that we have the more-subtle "auto" policy, it's possible that
_notmuch_crypto_decrypt could be called without having any actual
decryption take place.

This change lets the callers be a little bit smarter about whether or
not any decryption was actually attempted.
2017-12-08 08:08:46 -04:00
Daniel Kahn Gillmor
e4890b5bf9 crypto: new decryption policy "auto"
This new automatic decryption policy should make it possible to
decrypt messages that we have stashed session keys for, without
incurring a call to the user's asymmetric keys.
2017-12-08 08:07:53 -04:00
Daniel Kahn Gillmor
798aa789b5 lib: convert notmuch decryption policy to an enum
Future patches in this series will introduce new policies; this merely
readies the way for them.

We also convert --try-decrypt to a keyword argument instead of a boolean.
2017-12-08 08:07:02 -04:00
Daniel Kahn Gillmor
b62045a186 indexopts: change _try_decrypt to _decrypt_policy
This terminology makes it clearer what's going on at the API layer,
and paves the way for future changesets that offer more nuanced
decryption policy.
2017-12-08 08:06:03 -04:00
Daniel Kahn Gillmor
d3964e81ac indexing: Change from try_decrypt to decrypt
the command-line interface for indexing (reindex, new, insert) used
--try-decrypt; and the configuration records used index.try_decrypt.
But by comparison with "show" and "reply", there doesn't seem to be
any reason for the "try" prefix.

This changeset adjusts the command-line interface and the
configuration interface.

For the moment, i've left indexopts_{set,get}_try_decrypt alone.  The
subsequent changeset will address those.
2017-12-08 08:05:53 -04:00
Daniel Kahn Gillmor
a990585408 crypto: use stashed session-key properties for decryption, if available
When doing any decryption, if the notmuch database knows of any
session keys associated with the message in question, try them before
defaulting to using default symmetric crypto.

This changeset does the primary work in _notmuch_crypto_decrypt, which
grows some new parameters to handle it.

The primary advantage this patch offers is a significant speedup when
rendering large encrypted threads ("notmuch show") if session keys
happen to be cached.

Additionally, it permits message composition without access to
asymmetric secret keys ("notmuch reply"); and it permits recovering a
cleartext index when reindexing after a "notmuch restore" for those
messages that already have a session key stored.

Note that we may try multiple decryptions here (e.g. if there are
multiple session keys in the database), but we will ignore and throw
away all the GMime errors except for those that come from last
decryption attempt.  Since we don't necessarily know at the time of
the decryption that this *is* the last decryption attempt, we'll ask
for the errors each time anyway.

This does nothing if no session keys are stashed in the database,
which is fine.  Actually stashing session keys in the database will
come as a subsequent patch.
2017-12-04 21:48:31 -04:00
Daniel Kahn Gillmor
0ff13f862c configure: session key handling in gmime maps to built_with("session_key")
This flag should make it easier to write the code for session-key
handling.

Note that this only works for GMime 2.6.21 and later (the session key
interface wasn't available before then).  It should be fine to build
the rest of notmuch if this functionality isn't available.

Note that this also adds the "session_key" built_with() aspect to
libnotmuch.
2017-12-04 21:39:50 -04:00
Daniel Kahn Gillmor
5f2832ae21 crypto: add _notmuch_crypto_decrypt wrapper function
We will use this centralized function to consolidate the awkward
behavior around different gmime versions.

It's only invoked from two places: mime-node.c's
node_decrypt_and_verify() and lib/index.cc's
_index_encrypted_mime_part().

However, those two places have some markedly distinct logic, so the
interface for this _notmuch_crypto_decrypt function is going to get a
little bit clunky.  It's worthwhile, though, for the sake of keeping
these #if directives reasonably well-contained.
2017-12-04 21:39:24 -04:00
Daniel Kahn Gillmor
d0da7a0a1c config: define new option index.try_decrypt
By default, notmuch won't try to decrypt on indexing.  With this
patch, we make it possible to indicate a per-database preference using
the config variable "index.try_decrypt", which by default will be
false.

At indexing time, the database needs some way to know its internal
defaults for how to index encrypted parts.  It shouldn't be contingent
on an external config file (since that can't be retrieved from the
database object itself), so we store it in the database.

This behaves similarly to the query.* configurations, which are also
stored in the database itself, so we're not introducing any new
dependencies by requiring that it be stored in the database.
2017-10-21 19:54:33 -03:00
Daniel Kahn Gillmor
4dfcc8c9b2 crypto: index encrypted parts when indexopts try_decrypt is set.
If we see index options that ask us to decrypt when indexing a
message, and we encounter an encrypted part, we'll try to descend into
it.

If we can decrypt, we add the property index.decryption=success.

If we can't decrypt (or recognize the encrypted type of mail), we add
the property index.decryption=failure.

Note that a single message may have both values of the
"index.decryption" property: "success" and "failure".  For example,
consider a message that includes multiple layers of encryption.  If we
manage to decrypt the outer layer ("index.decryption=success"), but
fail on the inner layer ("index.decryption=failure").

Because of the property name, this will be automatically cleared (and
possibly re-set) during re-indexing.  This means it will subsequently
correspond to the actual semantics of the stored index.
2017-10-21 19:53:19 -03:00
Daniel Kahn Gillmor
0bb05ff693 reindex: drop all properties named with prefix "index."
This allows us to create new properties that will be automatically set
during indexing, and cleared during re-indexing, just by choice of
property name.
2017-10-21 19:53:08 -03:00
Daniel Kahn Gillmor
20ff9de24d index: implement notmuch_indexopts_t with try_decrypt
This is currently mostly a wrapper around _notmuch_crypto_t that keeps
its internals private and doesn't expose any of the GMime API.
However, non-crypto indexing options might also be added later
(e.g. filters or other transformations).
2017-10-21 19:52:47 -03:00
Daniel Kahn Gillmor
0b9e1a2472 properties: add notmuch_message_remove_all_properties_with_prefix()
Subsequent patches will introduce a convention that properties whose
name starts with "index." will be stripped (and possibly re-added)
during re-indexing.  This patch lays the groundwork for doing that.
2017-10-20 07:58:43 -03:00
Daniel Kahn Gillmor
a18bbf7f15 crypto: make shared crypto code behave library-like
If we're going to reuse the crypto code across both the library and
the client, then it needs to report error states properly and not
write to stderr.
2017-10-20 07:58:20 -03:00
Jani Nikula
008a5e92eb lib: convert notmuch_bool_t to stdbool internally
C99 stdbool turned 18 this year. There really is no reason to use our
own, except in the library interface for backward
compatibility. Convert the lib internally to stdbool.
2017-10-09 22:27:16 -03:00
Daniel Kahn Gillmor
e3a6368e8d fix reference to notmuch_message_get_properties 2017-09-24 09:15:24 -03:00
Daniel Kahn Gillmor
f4ac5ecd5c lib: index the content-type of the parts of encrypted messages
This is a logical followup to "lib: index the content type of
signature parts", which will make it easier to record the message
structure of all messages.
2017-09-17 20:01:19 -03:00
Jani Nikula
55c047ee0b lib: index the content type of signature parts
It's useful (*) to be able to easily find messages with certain types
of signatures. Having the mimetype: prefix searches fail for some
content types is also genuinely surprising (*). Index the content type
of signature parts.

While at it, switch to the gmime convenience constants for content and
signature part indexes.

*) At least for developers of email software!
2017-09-17 20:01:00 -03:00
Jani Nikula
930d0aefb1 lib: abstract content type indexing
Make the follow-up change of indexing signature content types
easier. No functional changes.
2017-09-17 20:00:32 -03:00
Jani Nikula
eb29e26a99 build: fix out-of-tree builds, again
Broken, again, by yours truly in bc11759dd1 ("build: switch to
hiding libnotmuch symbols by default"). Reference notmuch.sym via
$(srctree).
2017-09-13 08:48:17 -03:00
Daniel Kahn Gillmor
3445385f95 fix documentation bug (leading quotes break documentation) 2017-09-05 21:54:46 -03:00
David Bremner
debfae20db lib: enforce that n_message_reindex takes headers from first file
This is still a bit stopgap to be only choosing one set of headers,
but this seems like a more defensible set of headers to choose.
2017-09-05 21:51:57 -03:00
David Bremner
0260ee371e lib&cli: use g_object_new instead of g_object_newv
'g_object_newv' is deprecated, and prints annoying warnings. The
warnings suggest using 'g_object_new_with_properties', but that's only
available since glib 2.55 (i.e. a month ago as of this writing).
Since we don't actuall pass any properties, it seems we can just call
'g_object_new'.
2017-09-04 08:04:44 -03:00
David Bremner
0a40ea4b48 lib: add notmuch_message_has_maildir_flag
I considered a higher level interface where the caller passes a tag
name rather than a flag character, but the role of the "unread" tag is
particularly confusing with such an interface.
2017-08-29 21:56:21 -03:00
David Bremner
8a8fb39b0c lib/message: split n_m_maildir_flags_tags, store maildir flags
In a future commit this will allow querying maildir flags seperately
from tags to allow resolving certain conflicts.
2017-08-29 21:51:10 -03:00
Daniel Kahn Gillmor
eb232ee0ab reindex: drop notmuch_param_t, use notmuch_indexopts_t instead
There are at least three places in notmuch that can trigger an
indexing action:

 * notmuch new
 * notmuch insert
 * notmuch reindex

I have plans to add some indexing options (e.g. indexing the cleartext
of encrypted parts, external filters, automated property injection)
that should properly be available in all places where indexing
happens.

I also want those indexing options to be exposed by (and constrained
by) the libnotmuch C API.

This isn't yet an API break because we've never made a release with
notmuch_param_t.

These indexing options are relevant in the listed places (and in the
libnotmuch analogues), but they aren't relevant in the other kinds of
functionality that notmuch offers (e.g. dump/restore, tagging, search,
show, reply).

So i think a generic "param" object isn't well-suited for this case.
In particular:

 * a param object sounds like it could contain parameters for some
   other (non-indexing) operation.  This sounds confusing -- why would
   i pass non-indexing parameters to a function that only does
   indexing?

 * bremner suggests online a generic param object would actually be
   passed as a list of param objects, argv-style.  In this case (at
   least in the obvious argv implementation), the params might be some
   sort of generic string.  This introduces a problem where the API of
   the library doesn't grow as new options are added, which means that
   when code outside the library tries to use a feature, it first has
   to test for it, and have code to handle it not being available.
   The indexopts approach proposed here instead makes it clear at
   compile time and at dynamic link time that there is an explicit
   dependency on that feature, which allows automated tools to keep
   track of what's needed and keeps the actual code simple.

My proposal adds the notmuch_indexopts_t as an opaque struct, so that
we can extend the list of options without causing ABI breakage.

The cost of this proposal appears to be that the "boilerplate" API
increases a little bit, with a generic constructor and destructor
function for the indexopts struct.

More patches will follow that make use of this indexopts approach.
2017-08-23 07:55:12 -03:00
Daniel Kahn Gillmor
b10ce6bc23 database: add n_d_index_file (deprecates n_d_add_message)
We need a way to pass parameters to the indexing functionality on the
first index, not just on reindexing.  The obvious place is in
notmuch_database_add_message.  But since modifying the argument list
would break both API and ABI, we needed a new name.

I considered notmuch_database_add_message_with_params(), but the
functionality we're talking about doesn't always add a message.  It
tries to index a specific file, possibly adding a message, but
possibly doing other things, like adding terms to an existing message,
or failing to deal with message objects entirely (e.g. because the
file didn't contain a message).

So i chose the function name notmuch_database_index_file.

I confess i'm a little concerned about confusing future notmuch
developers with the new name, since we already have a private
_notmuch_message_index_file function, and the two do rather different
things.  But i think the added clarity for people linking against the
future libnotmuch and the capacity for using index parameters makes
this a worthwhile tradeoff.  (that said, if anyone has another name
that they strongly prefer, i'd be happy to go with it)

This changeset also adjusts the tests so that we test whether the new,
preferred function returns bad values (since the deprecated function
just calls the new one).

We can keep the deprecated n_d_add_message function around as long as
we like, but at the next place where we're forced to break API or ABI
we can probably choose to drop the name relatively safely.

NOTE: there is probably more cleanup to do in the ruby and go bindings
to complete the deprecation directly.  I don't know those languages
well enough to attempt a fix; i don't know how to test them; and i
don't know the culture around those languages about API additions or
deprecations.
2017-08-23 07:38:37 -03:00