According to my reading of RFC5322, there is an obsolete syntax for
Message-Id which permits folding whitespace (i.e. to be removed /
ignored by parsers). In [1] Paul Wise observed that notmuch removed
whitespace on indexing, but does not do any corresponding
normalization of queries. Mark the latter as a bug by adding a failing
test.
[1]: id:20230409044143.4041560-1-pabs3@bonedaddy.net
Py 3.12 finally pulled the plug on the `SafeConfigParser` class which
has been deprecated since py 3.2.
We use it in the legacy bindings only, so take the easy route of
importing `ConfigParser` as `SafeConfigParser` and monkey-patching so
that the class has the expected interface.
This prevents data loss when users configure the search cache Maildir to be a
real Maildir containing their real mail data, since the search cache Maildir
is expected to contain only symlinks to the real mail data.
Prevents: <ZCsQBNmbzwkvbpHA@localhost.localdomain>
This should be be slightly faster since it avoids forking a shell
and is less code in and less dependencies for the script.
Since String::ShellQuote isn't used elsewhere, drop mention of it.
Spaces need to be stripped when querying the Message-Id,
because notmuch stores them in Xapian with spaces stripped.
All double-quote characters need to be doubled to escape them,
otherwise they will be added as extra query terms outside the id.
We pre-parse into a list of compiled regular expressions to avoid
calling regexc on the hot (indexing) path. As explained in the code
comment, this cannot be done lazily with reasonable error reporting,
at least not without touching a lot of the code in index.cc.
This replaces two instances of Xapian::Query::MatchAll with the
equivalent but thread-safe alternative Xapian::Query(std::string()).
Xapian::Query::MatchAll maintains an internal pointer to a refcounted
Xapian::Internal::QueryTerm.
None of this is thread-safe but that wouldn't be an issue if
Xapian::Query::MatchAll wasn't static. Because it's static, the
refcounting goes awry when Notmuch is called from multiple threads.
This is actually documented by Xapian:
4715de3a9f/xapian-core/include/xapian/query.h (L65)
While static, Xapian::Query::MatchNothing is safe because it doesn't
maintain an internal object and as such, doesn't use references.
Two best-effort tests making use of TSan were added to showcase the
issue (I couldn't figure out a way to deterministically reproduce it
without making an unmaintainable mess).
First, when two databases are created in parallel, a query that uses
Xapian::Query::MatchAll is made (lib/query.cc), resulting in the
following backtrace on a segfault:
#0 0x00007ffff76822af in Xapian::Query::get_terms_begin (this=0x7fffe80137f0) at api/query.cc:141
#1 0x00007ffff7f933f5 in _notmuch_query_cache_terms (query=0x7fffe80137c0) at lib/query.cc:176
#2 0x00007ffff7f93784 in _notmuch_query_ensure_parsed_xapian (query=0x7fffe80137c0) at lib/query.cc:225
#3 0x00007ffff7f9381a in _notmuch_query_ensure_parsed (query=0x7fffe80137c0) at lib/query.cc:260
#4 0x00007ffff7f93bfe in _notmuch_query_search_documents (query=0x7fffe80137c0, type=0x7ffff7fa9b1e "mail", out=0x7ffff666da18) at lib/query.cc:361
#5 0x00007ffff7f93ba4 in notmuch_query_search_messages (query=0x7fffe80137c0, out=0x7ffff666da18) at lib/query.cc:349
#6 0x00007ffff7f83d98 in notmuch_database_upgrade (notmuch=0x7fffe8000bd0, progress_notify=0x0, closure=0x0) at lib/database.cc:934
#7 0x00007ffff7fa110f in notmuch_database_create_with_config (database_path=0x7ffff666dcb0 "/tmp/notmuch.MZ2AGr", config_path=0x7ffff7faab3c "", profile=0x0, database=0x0, status_string=0x7ffff666dc90) at lib/open.cc:754
#8 0x00007ffff7fa0d6f in notmuch_database_create_verbose (path=0x7ffff666dcb0 "/tmp/notmuch.MZ2AGr", database=0x0, status_string=0x7ffff666dc90) at lib/open.cc:653
#9 0x00007ffff7fa0ceb in notmuch_database_create (path=0x7ffff666dcb0 "/tmp/notmuch.MZ2AGr", database=0x0) at lib/open.cc:637
...
Second, some queries would make use of Xapian::Query::MatchAll
(lib/regexp-fields.cc), resulting in the following backtrace on a
segfault:
#0 0x00007f629828b690 in Xapian::Internal::QueryBranch::gather_terms (this=0x7f628800def0, void_terms=0x7f629726d5a0) at api/queryinternal.cc:1245
#1 0x00007f629828c260 in Xapian::Internal::QueryScaleWeight::gather_terms (this=0x7f628800df70, void_terms=0x7f629726d5a0) at api/queryinternal.cc:1434
#2 0x00007f629828b69f in Xapian::Internal::QueryBranch::gather_terms (this=0x7f628800dd90, void_terms=0x7f629726d5a0) at api/queryinternal.cc:1245
#3 0x00007f6298282571 in Xapian::Query::get_unique_terms_begin (this=0x7f628800dcd8) at api/query.cc:166
#4 0x00007f629841a59b in Xapian::Weight::Internal::accumulate_stats (this=0x7f628800dca0, subdb=..., rset=...) at weight/weightinternal.cc:86
#5 0x00007f62983c15ba in LocalSubMatch::prepare_match (this=0x7f628800df20, nowait=true, total_stats=...) at matcher/localsubmatch.cc:172
#6 0x00007f62983c8fcc in prepare_sub_matches (leaves=std::vector of length 1, capacity 1 = {...}, stats=...) at matcher/multimatch.cc:237
#7 0x00007f62983c98a3 in MultiMatch::MultiMatch (this=0x7f629726d9a0, db_=..., query_=..., qlen=3, omrset=0x0, collapse_max_=0, collapse_key_=4294967295, percent_cutoff_=0, weight_cutoff_=0, order_=Xapian::Enquire::ASCENDING, sort_key_=0, sort_by_=Xapian::Enquire::Internal::VAL, sort_value_forward_=true, time_limit_=0, stats=..., weight_=0x7f6288008d50, matchspies_=std::vector of length 0, capacity 0, have_sorter=false, have_mdecider=false) at matcher/multimatch.cc:353
#8 0x00007f629826fcba in Xapian::Enquire::Internal::get_mset (this=0x7f628800e0b0, first=0, maxitems=0, check_at_least=0, rset=0x0, mdecider=0x0) at api/omenquire.cc:569
#9 0x00007f629827181c in Xapian::Enquire::get_mset (this=0x7f629726db80, first=0, maxitems=0, check_at_least=0, rset=0x0, mdecider=0x0) at api/omenquire.cc:937
#10 0x00007f6298be529a in _notmuch_query_search_documents (query=0x7f6288009750, type=0x7f6298bfaafe "mail", out=0x7f629726dcc0) at lib/query.cc:447
#11 0x00007f6298be4ae8 in notmuch_query_search_messages (query=0x7f6288009750, out=0x7f629726dcc0) at lib/query.cc:349
...
Printing Xapian::Query::MatchAll->internal.px->_refs in these
circumstances can help quickly identifying this scenario.
This is motivated by some test frameworks (like Rust's Cargo) that
runs unit tests in parallel and would easily encounter this issue,
unless client code gates every call to Notmuch behind a lock.
This is what can be expected from the tests when they fail:
== stderr ==
+==================
+WARNING: ThreadSanitizer: data race (pid=207931)
+ Read of size 1 at 0x7b10000001a0 by thread T2:
+ #0 memcpy <null> (libtsan.so.2+0x62506)
+ #1 void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char*>(char*, char*, std::forward_iterator_tag) [clone .isra.0] <null> (libxapian.so.30+0x872b3)
+
+ Previous write of size 8 at 0x7b10000001a0 by thread T1:
+ #0 operator new(unsigned long) <null> (libtsan.so.2+0x8ba83)
+ #1 Xapian::Query::Query(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int, unsigned int) <null> (libxapian.so.30+0x855cd)
...
Fixes warning:
warning: ‘notmuch_database_open’ is deprecated: function deprecated as of libnotmuch 5.4
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
_notmuch_message_remove_all_properties wasn't syncing the message back
to the database but was still invalidating the metadata, giving the
impression the properties had actually been removed.
Also move the metadata invalidation to _notmuch_message_remove_terms
to be closer to what's done in _notmuch_message_modify_property and
_notmuch_message_remove_term.
notmuch_message_remove_all_properties should have removed the
testkey1 = testvalue1
property but hasn't. Delay the execution of the corresponding test
to avoid updating a few tests that actually relied on the broken
behavior.
These two functions don't fail gracefully when editing a removed
message:
BROKEN edit property on removed message without uncaught exception
--- T610-message-property.20.EXPECTED 2023-02-27 11:33:25.792764376 +0000
+++ T610-message-property.20.OUTPUT 2023-02-27 11:33:25.793764381 +0000
@@ -1,2 +1,3 @@
== stdout ==
== stderr ==
+terminate called after throwing an instance of 'Xapian::DocNotFoundError'
The other functions appear to be safe.
Previously we just crashed with an internal error. With this change,
the caller can handle it better. Update notmuch-new so that it doesn't
crash with "unknown error code" because of this change.
With this mode, one can fold trees in the notmuch-tree buffer as if
they were outlines, using all the commands provided by
outline-minor-mode. We also define a couple of movement commands
that, optional, will ensure that only the thread around point is
unfolded.
The implementation is based on registering a :level property in the
messages p-list, that is then used by outline-minor-mode to to
recognise headers.
Amended by db: Copy docstring to manual and edit for presentation. Add
two tests. Fix typo "wether".
GnuPG upstream has supported pkg-config since gpgme version 1.13 and
gpg-error 1.33, and now prefers the use of pkg-config by default,
instead of relying on gpg-error-config and gpgme-config.
As of libgpg-error 1.46, upstream deliberately does not ship
gpg-error-config by default. As of gpgme 1.18.0, upstream does not
ship gpgme-config if gpg-error-config is also not present.
Both of these versions of upstream libraries are in debian unstable
now. To the extent that notmuch is dependent on GnuPG, it should
follow GnuPG upstream's lead.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
The call to delete_document can throw exceptions (and can happen in
practice [1]), so catch the exception and extract the error
message. As a side effect, also move the call to _n_m_has_term inside
the try/catch. This should not change anything as that function
already traps any Xapian exceptions.
[1]: id:wwuk039sk2p.fsf@chaotikum.eu
In [1], Thomas Schneider reported an uncaught Xapian exception when
running out of disk space. We generate the same exception via database
corruption.
[1]: id:wwuk039sk2p.fsf@chaotikum.eu
We use notmuch search in two places in notmuch-git.py: to find which
tags have a given prefix, and to see if message with given id exists
locally. In both cases we do not want the presence of exclude tags
(e.g. deleted) to change the results.
`notmuch search` behaves differently depending on the output option: It
either outputs information pertaining to all threads with matching
messages (summary, threads) or to all matching messages (messages,
files, tags). The man page refres solely to the former in the main
description.
Help the user by clearly marking `summary` as the default output option.
Signed-off-by: Michael J Gruber <git@grubix.eu>
Although this has more steps than the previous regular expression
search and replace, it should be more robust against changes in the
headerline format, such as the inclusion of duplicate numbers (which
broke the previous version).
OTHER-HEADERS are expected to be passed as strings, to match the
implementation of `compose-mail'. But the "From" header is currently
expected to be passed as a symbol. Instead the "From" header can be
safely added after converting all the headers to symbols.
