This replaces two instances of Xapian::Query::MatchAll with the
equivalent but thread-safe alternative Xapian::Query(std::string()).
Xapian::Query::MatchAll maintains an internal pointer to a refcounted
Xapian::Internal::QueryTerm.
None of this is thread-safe but that wouldn't be an issue if
Xapian::Query::MatchAll wasn't static. Because it's static, the
refcounting goes awry when Notmuch is called from multiple threads.
This is actually documented by Xapian:
4715de3a9f/xapian-core/include/xapian/query.h (L65)
While static, Xapian::Query::MatchNothing is safe because it doesn't
maintain an internal object and as such, doesn't use references.
Two best-effort tests making use of TSan were added to showcase the
issue (I couldn't figure out a way to deterministically reproduce it
without making an unmaintainable mess).
First, when two databases are created in parallel, a query that uses
Xapian::Query::MatchAll is made (lib/query.cc), resulting in the
following backtrace on a segfault:
#0 0x00007ffff76822af in Xapian::Query::get_terms_begin (this=0x7fffe80137f0) at api/query.cc:141
#1 0x00007ffff7f933f5 in _notmuch_query_cache_terms (query=0x7fffe80137c0) at lib/query.cc:176
#2 0x00007ffff7f93784 in _notmuch_query_ensure_parsed_xapian (query=0x7fffe80137c0) at lib/query.cc:225
#3 0x00007ffff7f9381a in _notmuch_query_ensure_parsed (query=0x7fffe80137c0) at lib/query.cc:260
#4 0x00007ffff7f93bfe in _notmuch_query_search_documents (query=0x7fffe80137c0, type=0x7ffff7fa9b1e "mail", out=0x7ffff666da18) at lib/query.cc:361
#5 0x00007ffff7f93ba4 in notmuch_query_search_messages (query=0x7fffe80137c0, out=0x7ffff666da18) at lib/query.cc:349
#6 0x00007ffff7f83d98 in notmuch_database_upgrade (notmuch=0x7fffe8000bd0, progress_notify=0x0, closure=0x0) at lib/database.cc:934
#7 0x00007ffff7fa110f in notmuch_database_create_with_config (database_path=0x7ffff666dcb0 "/tmp/notmuch.MZ2AGr", config_path=0x7ffff7faab3c "", profile=0x0, database=0x0, status_string=0x7ffff666dc90) at lib/open.cc:754
#8 0x00007ffff7fa0d6f in notmuch_database_create_verbose (path=0x7ffff666dcb0 "/tmp/notmuch.MZ2AGr", database=0x0, status_string=0x7ffff666dc90) at lib/open.cc:653
#9 0x00007ffff7fa0ceb in notmuch_database_create (path=0x7ffff666dcb0 "/tmp/notmuch.MZ2AGr", database=0x0) at lib/open.cc:637
...
Second, some queries would make use of Xapian::Query::MatchAll
(lib/regexp-fields.cc), resulting in the following backtrace on a
segfault:
#0 0x00007f629828b690 in Xapian::Internal::QueryBranch::gather_terms (this=0x7f628800def0, void_terms=0x7f629726d5a0) at api/queryinternal.cc:1245
#1 0x00007f629828c260 in Xapian::Internal::QueryScaleWeight::gather_terms (this=0x7f628800df70, void_terms=0x7f629726d5a0) at api/queryinternal.cc:1434
#2 0x00007f629828b69f in Xapian::Internal::QueryBranch::gather_terms (this=0x7f628800dd90, void_terms=0x7f629726d5a0) at api/queryinternal.cc:1245
#3 0x00007f6298282571 in Xapian::Query::get_unique_terms_begin (this=0x7f628800dcd8) at api/query.cc:166
#4 0x00007f629841a59b in Xapian::Weight::Internal::accumulate_stats (this=0x7f628800dca0, subdb=..., rset=...) at weight/weightinternal.cc:86
#5 0x00007f62983c15ba in LocalSubMatch::prepare_match (this=0x7f628800df20, nowait=true, total_stats=...) at matcher/localsubmatch.cc:172
#6 0x00007f62983c8fcc in prepare_sub_matches (leaves=std::vector of length 1, capacity 1 = {...}, stats=...) at matcher/multimatch.cc:237
#7 0x00007f62983c98a3 in MultiMatch::MultiMatch (this=0x7f629726d9a0, db_=..., query_=..., qlen=3, omrset=0x0, collapse_max_=0, collapse_key_=4294967295, percent_cutoff_=0, weight_cutoff_=0, order_=Xapian::Enquire::ASCENDING, sort_key_=0, sort_by_=Xapian::Enquire::Internal::VAL, sort_value_forward_=true, time_limit_=0, stats=..., weight_=0x7f6288008d50, matchspies_=std::vector of length 0, capacity 0, have_sorter=false, have_mdecider=false) at matcher/multimatch.cc:353
#8 0x00007f629826fcba in Xapian::Enquire::Internal::get_mset (this=0x7f628800e0b0, first=0, maxitems=0, check_at_least=0, rset=0x0, mdecider=0x0) at api/omenquire.cc:569
#9 0x00007f629827181c in Xapian::Enquire::get_mset (this=0x7f629726db80, first=0, maxitems=0, check_at_least=0, rset=0x0, mdecider=0x0) at api/omenquire.cc:937
#10 0x00007f6298be529a in _notmuch_query_search_documents (query=0x7f6288009750, type=0x7f6298bfaafe "mail", out=0x7f629726dcc0) at lib/query.cc:447
#11 0x00007f6298be4ae8 in notmuch_query_search_messages (query=0x7f6288009750, out=0x7f629726dcc0) at lib/query.cc:349
...
Printing Xapian::Query::MatchAll->internal.px->_refs in these
circumstances can help quickly identifying this scenario.
This is motivated by some test frameworks (like Rust's Cargo) that
runs unit tests in parallel and would easily encounter this issue,
unless client code gates every call to Notmuch behind a lock.
This is what can be expected from the tests when they fail:
== stderr ==
+==================
+WARNING: ThreadSanitizer: data race (pid=207931)
+ Read of size 1 at 0x7b10000001a0 by thread T2:
+ #0 memcpy <null> (libtsan.so.2+0x62506)
+ #1 void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char*>(char*, char*, std::forward_iterator_tag) [clone .isra.0] <null> (libxapian.so.30+0x872b3)
+
+ Previous write of size 8 at 0x7b10000001a0 by thread T1:
+ #0 operator new(unsigned long) <null> (libtsan.so.2+0x8ba83)
+ #1 Xapian::Query::Query(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int, unsigned int) <null> (libxapian.so.30+0x855cd)
...
GnuPG upstream has supported pkg-config since gpgme version 1.13 and
gpg-error 1.33, and now prefers the use of pkg-config by default,
instead of relying on gpg-error-config and gpgme-config.
As of libgpg-error 1.46, upstream deliberately does not ship
gpg-error-config by default. As of gpgme 1.18.0, upstream does not
ship gpgme-config if gpg-error-config is also not present.
Both of these versions of upstream libraries are in debian unstable
now. To the extent that notmuch is dependent on GnuPG, it should
follow GnuPG upstream's lead.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
This replaces the old OpenPGPv4 key that is used in the test suite
with a more modern OpenPGPv4 key. All cryptographic artifacts in the
test suite are updated accordingly.
Having old cryptographic artifacts in the test suite presents a
problem once the old algorithms are rejected by contemporary
implementations.
For reference, this is the old key.
sec rsa1024 2011-02-05 [SC]
5AEAB11F5E33DCE875DDB75B6D92612D94E46381
uid [ unknown] Notmuch Test Suite <test_suite@notmuchmail.org> (INSECURE!)
ssb rsa1024 2011-02-05 [E]
And this is the new key. Note that is has the same shape, but uses
Ed25519 and Cv25519 instead of 1024-bit RSA.
sec ed25519 2022-09-07 [SC]
9A3AFE6C60065A148FD4B58A7E6ABE924645CC60
uid [ultimate] Notmuch Test Suite (INSECURE!) <test_suite@notmuchmail.org>
ssb cv25519 2022-09-07 [E]
When testing error handling, it is sometimes difficult to cover a
particular error path deterministically. Introduce a test function to
allow calling lower level functions directly.
7228fe68 ("configure: restructure gmime cert validity checker code",
2022-04-09) restructured generated C code to repurpose it later on. This
put usage of `validity` within an `#if`, resulting in an "unused
warning" if that `#if` is not executed.
Put the variable declariation inside the same if branch and, thus, quel
the warning.
Signed-off-by: Michael J Gruber <git@grubix.eu>
The extra machinery to check for the actual output format is justified
by the possibility that distros may patch this newer output format
into older versions of gmime.
Amended-by: Michael J Gruber <git@grubix.eu>
Signed-off-by: Michael J Gruber <git@grubix.eu>
Amended-again-by: db
This will correct the current use of an undefined variable when
setting LD_LIBRARY_PATH in doc/Makefile.local
It is tempting to try to replace the use of test/export-dirs.sh, but
this is not as simple as it looks, as NOTMUCH_BUILDDIR is used to
locate sh.config, so probably cannot also sensibly be used to define
it.
Since set -u is used, without bash or perl, configure would fail.
This has gone unnoticed as (almost) everyone always had both
bash and perl installed (and in $PATH).
Thanks to FreeBSD ports this bug became visible; this change is
verbatim copy of `patch-configure` in FreeBSD ports tree.
The main idea is to replace the hack of copying version.txt into the
bindings source with a generated _notmuch_config.py file.
This will mean that the bindings only build after configuring and
building notmuch itself. Given those constraints, "pip install ."
should work.
In order to make it easier to keep the whitespace consistent in the
configure script, use the same style defined in devel/STYLE for
C/C++.
Specifically, a line should begin with zero or more tabs followed
by fewer than eight spaces.
Presumably this will be no more difficult for people editing configure
than for people editing the C and C++ code.
As discussed at [1] we have received reports that the implicit check
using cffi.FFI().verify() is not reliable in all environments. Since
we already use pkg-config, and the python dev package should include a
.pc file [2], add an extra check using pkg-config. On at least
Debian, we have to know which version of python dev files with are
looking for, so calculate that first.
[1]: id:87im1g35ey.fsf@tethera.netid:87im1g35ey.fsf@tethera.net,
[2]: checked on Debian and Fedora
The configure part is essentially the same as the other checks using
pkg-config. Since the optional inclusion of this feature changes what
options are available to the user, include it in the "built_with"
pseudo-configuration keys.
For portability; the realpath command (e.g. from GNU coreutils)
is not so common outside Linux systems.
The "$(cd emacs && pwd -P)" replaces that realpath(1) execution
suitably in this context (using just bash(1) builtins).
Starting from xapian 1.3.5, xapian switched default backend to glass.
From 00cdfe10 (build: drop support for xapian versions less than 1.4,
2020-04-22), we only support xapian 1.4.0+. Effectively, we don't need
to check for default xapian backend anymore.
Further more, from 99a7aac8 (test: drop use of db_ending, 2020-07-29),
our test framework has become independence from default xapian.
Let's drop it.
Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
On different distro, pytest is suffixed with different patterns.
On the other hand, `python3-pytest' could be invoked correctly,
via `python3 -m pytest', the latter is used by our tests, now.
Switch to `$python -m pytest` to fix address all incompatible naming.
Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
It is getting unwieldy to pass configuration options on the
sphinx-build command line, and I anticipate further use of
conditionals.
As far as I could tell, execing a string is the idiomatic way to
emulate include in Python.
These are less crucial since we stopped generating new database
versions and relied primarily on features. They also rely on a
pre-generated v1 database which happens to be chert format. This
backend is not supported by Xapian 1.5.
Also drop the tool gen-testdb.sh, which is currently broken, due to
changes in the testing infrastructure.
If https://dev.gnupg.org/T3464 is unresolved in the version of gpgme
we are testing against, then we should know about it, because it
affects the behavior of notmuch.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Certain tests involving timestamps > 32 bits cannot pass with the
current libnotmuch API. We will avoid this issue for now by disabling
those tests on "old" architectures with 32 bit time_t.
Checking existence of pyconfig.h to determine whether CFFI-based
notmuch bindings are buildable is not enough; for example Fedora 32
ships pyconfig.h in python3-libs package, but python3-devel is required
to be installed for the bindings to build.
Executing cffi.FFI().verify() is pretty close to what is done in
bindings/python-cffi/notmuch2/_build.py to get the c code part of the
bindings built.
Also tell users what the consequences of a "No" answer is when
python version is less than 3.5, cffi or setuptools is missing,
or no pytest >= 3.0 is available.
The notmuch2 CFFI-based Python interface is not buildable unless
python3 dev package and python3 setuptools are installed.
Check that these exist in configure (and disable notmuch2 bindings
build if not) so that build of these bindings don't fail when make(1)
is executed.
This is a simple hack to enable out-of-tree builds, a concern raised
by Tomi in id:m24kzjib9a.fsf@guru.guru-group.fi
This change at least enables "make check" to complete without error,
but I'm sure it could be improved. I am not expert enough in
setuptools to know how.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Amended by db per id:87d06usa31.fsf@powell.devork.be
We already report the minimum version for Glib, zlib, and Xapian
development libraries. For consistency, report it for GMime as well.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
When checking cryptographic signatures, Notmuch relies on GMime to
tell it whether the certificate that signs a message has a valid User
ID or not.
If the User ID is not valid, then notmuch does not report the signer's
User ID to the user. This means that the consumer of notmuch's
cryptographic summary of a message (or of its protected headers) can
be confident in relaying the reported identity to the user.
However, some versions of GMime before 3.2.7 cannot report Certificate
validity for X.509 certificates. This is resolved upstream in GMime
at https://github.com/jstedfast/gmime/pull/90.
We adapt to this by marking tests of reported User IDs for
S/MIME-signed messages as known-broken if GMime is older than 3.2.7
and has not been patched.
If GMime >= 3.2.7 and certificate validity still doesn't work for
X.509 certs, then there has likely been a regression in GMime and we
should fail early, during ./configure.
To break out these specific User ID checks from other checks, i had to
split some tests into two parts, and reuse $output across the two
subtests.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Xapian 1.4 is over 3 years old now (1.4.0 released 2016-06-24),
and 1.2 has been deprecated in Notmuch version 0.27 (2018-06-13).
Xapian 1.4 supports compaction, field processors and retry locking;
conditionals checking compaction and field processors were removed
but user may want to disable retry locking at configure time so it
is kept.
In case zlib not found by pkg-config(1) the pkg-config information
is resolved by attempting to print ZLIB_VERSION from from zlib
installation if it exists anyway.
If above done successfully compat/zlib.pc is written for forthcoming
pkg-config execution.
Since `set -u` is in effect (since 124a67e96, 2016-05-06),
expanding unset $PKG_CONFIG_PATH (would have) failed whenever tried.
Now it is changed to set as "$PKG_CONFIG_PATH:compat" if PKG_CONFIG_PATH
is set and is non-empty string, plain "compat" otherwise.
Zsh searches in the $fpath array for completion functions. By default
this includes $(prefix)/share/zsh/site-functions but not the existing
value. The prefix for zsh and notmuch isn't guaranteed to be the same
but it normally will be making this a better default for
zsh_completion_dir.
The entire python-cffi test suite is considered as a single test at
the level of the notmuch test suite. This might or might not be ideal,
but it gets them run.
Whitespace in $NOTMUCH_SRCDIR (and $PWD) may work in builds,
but definitely will not work in tests. It would be difficult
to make tests support whitespace in test filename paths -- and
fragile to maintain if done.
So it is just easier and safer to disallow whitespace there.
In case of out of tree build $NOTMUCH_SRCDIR differs from $PWD
(current directory). Extend this whitespace, and also previously
made unsafe characters check to $PWD too.
While check for GMime session key extraction support... was made
out of tree build compatible, related (and some unrelated) unsafe
characters are now checked in notmuch source directory path.
The known unsafe characters in NOTMUCH_SRCDIR are:
- Single quote (') -- NOTMUCH_SRCDIR='${NOTMUCH_SRCDIR}'
is written to sh.config in configure line 1328.
- Double quote (") -- configure line 521 *now* writes "$srcdir"
into generated c source file ($NOTMUCH_SRCDIR includes $srcdir).
- Backslash (\) could also be problematic in configure line 521.
- The added $ and ` are potentially unsafe -- inside double quotes
in shell script those have special meaning.
Other characters don't expand inside double quoted strings.
The extra flexibility of having both HAVE_EMACS (for yes, there is an
emacs we can use) and WITH_EMACS (the user wants emacs support) lead
to confusion and bugs. We now just force WITH_EMACS to 0 if no
suitable emacs is detected.
When using a promiscuous linker, _check_session_keys was working fine.
But some OSes (including some versions of Ubuntu) have set their
linker to always link in "--as-needed" mode, which means that the
order of the objects linked is relevant. If a library is loaded
before it is needed, that library will no longer be linked in the
final outcome. _check_session_keys.c was failing on those systems.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
We never want ./configure to try to do something with an unassigned
variable. So, make the directory $TEMP_GPG at the start of the
testing of session-key handling, and clean it up afterwards as long as
the directory exists.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
There are a few changes bundled here:
* say "No." explicitly if there's a failure.
* try to avoid implying that gpgme-config is necessary to build
notmuch itself (it's not, though it may be useful if you need to
rebuild gmime).
* leave _check_session_keys and _check_session_keys.c around if
./configure fails, so that the user can play with it more easily
for debugging.
* let error messages show when _check_session_keys.c is built.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Amended by DB: use command -v instead of which.
GMime 3.0 and higher can extract session keys, but it will *not*
extract session keys if it was built with --disable-crypto, or if it
was built against GPGME version < 1.8.0.
Notmuch currently expects to be able to extract session keys, and
tests will fail if it is not possible, so we ensure that this is the
case during ./configure time.
Part of this feels awkward because notmuch doesn't directly depend on
gpg at all. Rather, it depends on GMime, and the current
implementation of GMime depends on GPGME for its crypto, and GPGME in
turn depends on gpg.
So the use of gpg in ./configure isn't actually introducing a new
dependency, though if a future version of GMime were ever to move away
from GnuPG, we might need to reconsider.
Note that this changeset depends on
id:20190506174327.13457-1-dkg@fifthhorseman.net , which supplies the
rfc822 message test/corpora/crypto/basic-encrypted.eml used in it.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
