Commit graph

7435 commits

Author SHA1 Message Date
Kevin Boulain
c1a23a64ae test: suppress all interceptors in glib
On ppc64el, races are detected by TSan:
  WARNING: ThreadSanitizer: data race (pid=4520)
    Read of size 8 at 0x7ffff20016c0 by thread T1:
      #0 strlen ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:386 (libtsan.so.2+0x77c0c)
      #1 strlen ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:378 (libtsan.so.2+0x77c0c)
      #2 g_strdup ../../../glib/gstrfuncs.c:362 (libglib-2.0.so.0+0xa4ac4)

    Previous write of size 8 at 0x7ffff20016c0 by thread T2:
      #0 malloc ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:647 (libtsan.so.2+0x471f0)
      #1 g_malloc ../../../glib/gmem.c:130 (libglib-2.0.so.0+0x7bb68)

    Location is heap block of size 20 at 0x7ffff20016c0 allocated by thread T2:
      #0 malloc ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:647 (libtsan.so.2+0x471f0)
      #1 g_malloc ../../../glib/gmem.c:130 (libglib-2.0.so.0+0x7bb68)

This appears to be a false positive in GLib, as explained at
https://gitlab.gnome.org/GNOME/glib/-/issues/1672#note_1831968
In short, a call to fstat fails under TSan and GLib's g_sterror will
intern the error message, which will be reused by other threads.

Since upstream appears to be aware that GLib doesn't play nicely with
TSan, suppress everything coming from the library instead of
maintaining a fine grained list.

Reported at
https://buildd.debian.org/status/fetch.php?pkg=notmuch&arch=ppc64el&ver=0.38%7Erc0-1&stamp=1692959868&raw=0
2023-09-02 07:45:30 -03:00
Michael J Gruber
a84dc2f7e6 compat: probe for strcasestr more thoroughly
Depending on compiler (gcc, g++, clang) and standard options (c99, c11),
string.h may or may not include strings.h, leading to possibly missing
or conflicting declarations of strcasestr.

Include both so that both detection and compilation phases use the same
(possibly optimised) implementations.

Suggested-by: Thomas Schneider <qsx@chaotikum.eu>
Suggested-by: Florian Weimer <fweimer@redhat.com>
Suggested-by: Tomi Ollila <tomi.ollila@iki.fi>
2023-08-27 19:39:23 -03:00
David Bremner
9a1126241d debian: changelog for 0.38~rc1-1 2023-08-26 08:31:49 -03:00
David Bremner
240e9fff90 version: bump to 0.38~rc1 2023-08-26 08:29:45 -03:00
David Bremner
90c6182825 debian: skip T810-tsan on ppc64el
Hopefully just a temporary measure.
2023-08-26 08:28:33 -03:00
David Bremner
25c933d7e6 debian: use architecture.mk
I plan to add further architecture dependent checks, which makes this
preferable to directly calling dpkg-architecture.
2023-08-26 08:22:33 -03:00
David Bremner
115d4d69eb test: minimize impact of native compilation.
Native compilation is kindof useless in the test suite because we
throw away the cache after every subtest.  The test suite could in
principle share an eln cache within a given test file; for now try to
minimize the amount of native-compilation. There is an intermittent
bug where emacs loses track of its default-directory; I suspect (but
have no proof) that bug is related to native compilation and/or race
conditions. This patch seems to prevent that bug (or at least reduce
its frequency).
2023-08-26 07:45:36 -03:00
David Bremner
8c8fda965f debian: set suite to experimental 2023-08-24 10:56:33 -03:00
David Bremner
5ed121bed9 doc: update copyright date 2023-08-24 08:03:46 -03:00
David Bremner
695f663f81 debian: changelog for 0.38~rc0-1 2023-08-24 08:00:55 -03:00
David Bremner
aa5674fce0 NEWS: start NEWS for 0.38 2023-08-24 07:58:51 -03:00
David Bremner
2f874fb2cc version: set up 0.38~rc0 2023-08-24 07:58:19 -03:00
David Bremner
368e0f61d5 test/setup: ignore blank lines in generated config
The presense of the blank lines between sections depends on the
version of glib. Strip them before comparison.
2023-08-21 19:30:54 -03:00
David Bremner
df45194d5f CLI/config: simulate top level comments when creating config
According to discussion on

          https://gitlab.gnome.org/GNOME/glib/-/issues/3078

it looks like upstream will stop supporting top of file comments.

It is questionable whether we really need this feature, but for now
update notmuch-config to simulate it.
2023-08-21 19:30:54 -03:00
David Bremner
1719b9e568 test/emacs: adapt to breaking change in Gnus defaults
As of Emacs 29.1, In-Reply-To is in the default value for
message-hidden-headers. We actually want to see that in the test
suite, so remove it again. To future proof the tests, fix a default
value for message-hidden-headers specifically for the test suite.
2023-08-20 14:32:02 -03:00
David Bremner
b6f144abe1 lib/n_d_remove_message: do not remove unique filename
It is wasteful to remove a filename term when the whole message
document is about to be removed from the database. Profiling with perf
shows this takes a significant portion of the time when cleaning up
removed files in the database.

The logic of n_d_remove_message becomes a bit more convoluted here in
order to make the change minimal.

It is possible that this function can be further optimized, since the
expansion of filename terms into filenames is probably not needed
here.
2023-07-22 07:15:59 -03:00
David Bremner
d93d49b6ae lib/message: check message type before deleting document
It isn't really clear how this worked before. Traversing the terms of
a document after deleting it from the database seems likely to be
undefined behaviour at best
2023-07-22 07:11:46 -03:00
David Bremner
08ca74d715 debian: add autopkgtests
We generate output to stderr for BROKEN tests, which are not failures,
so tell the test runner not to fail because of output on stderr.
2023-07-21 07:41:50 -03:00
David Bremner
ec26eeaeec test: support testing notmuch as installed
We put some effort into testing the built copy rather than some
installed copy. On the other hand for people like packagers, testing
the installed copy is also of interest.

When NOTMUCH_TEST_INSTALLED is set to a nonempty value, tests do not
require a built notmuch tree or running configure.

Some of the tests marked as broken when running against installed
notmuch are probably fixable.
2023-07-21 07:41:50 -03:00
David Bremner
73f3081160 test: Guess a value for NOTMUCH_PYTHON
python3 will work for many people, and reduce the friction to running
the tests without running configure first.
2023-07-21 07:07:43 -03:00
David Bremner
f6fcdf12da test: check for empty/missing files in test_expect_equal_message_body
Messages can have empty bodies, but empty files are not messages.
2023-07-21 07:07:43 -03:00
David Bremner
dfa43a1921 test: treat undefined feature variables as 0
When running the test suite without building first, it is desirable to
have the tests consider these variables being undefined as equivalent
to the feature not being present, and in particular for the tests not
to generate errors.
2023-07-21 06:58:16 -03:00
Tim Culverhouse
e9ff896f84 schemata: document 'excluded' field in structured output
Include the 'excluded' field in the structured output schema.

Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
2023-07-09 12:26:56 -03:00
David Bremner
f873790b6f perf-test: update corpus signature
In the decade (!) since this corpus was last updated, the keyserver
network is essentially dead, and I have migrated gpg keys. Bump the
version number as a clean way of switching signatures. Also update the
instructions to suggest using "--locate-external-key" to download the
public key. By default this uses WKD, which is now supported for my
UID.
2023-07-09 12:14:29 -03:00
David Bremner
a62b8a95c0 doc/lib: clarify ownership for notmuch_database_get_revision
The ownership is implicit in the const declaration (I think!), but
that does not show up in the doxygen generated API docs.
2023-07-09 12:08:28 -03:00
David Bremner
808fd6f258 test: add known broken test for message-id with embedded spaces.
According to my reading of RFC5322, there is an obsolete syntax for
Message-Id which permits folding whitespace (i.e. to be removed /
ignored by parsers). In [1] Paul Wise observed that notmuch removed
whitespace on indexing, but does not do any corresponding
normalization of queries. Mark the latter as a bug by adding a failing
test.

[1]: id:20230409044143.4041560-1-pabs3@bonedaddy.net
2023-07-09 11:59:43 -03:00
Michael J Gruber
3771832b01 python: adjust legacy bindings to py 3.12
Py 3.12 finally pulled the plug on the `SafeConfigParser` class which
has been deprecated since py 3.2.

We use it in the legacy bindings only, so take the easy route of
importing `ConfigParser` as `SafeConfigParser` and monkey-patching so
that the class has the expected interface.
2023-07-09 11:43:21 -03:00
Felipe Contreras
95a4bf3817 ruby: db: reorganize initializer
In order to make it more extensible.

No functional changes.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2023-05-29 07:51:02 -03:00
Paul Wise
239fdbbbf0 notmuch-mutt: check that the search cache Maildir is not a real Maildir
This prevents data loss when users configure the search cache Maildir to be a
real Maildir containing their real mail data, since the search cache Maildir
is expected to contain only symlinks to the real mail data.

Prevents: <ZCsQBNmbzwkvbpHA@localhost.localdomain>
2023-05-28 15:34:42 -03:00
Paul Wise
18e35950da notmuch-mutt: do not clear search cache Maildir when nothing is found
The previous results might be useful to the user but
an empty directory definitely isn't useful.
2023-05-28 15:33:11 -03:00
Paul Wise
85916d8a96 notmuch-mutt: clarify the empty Maildir function operates on search caches
Rename the function and adjust the documentation comment.
2023-05-28 15:32:11 -03:00
Paul Wise
93d936c5ae notmuch-mutt: replace extra command with notmuch thread search feature
This should be be slightly faster since it avoids forking a shell
and is less code in and less dependencies for the script.

Since String::ShellQuote isn't used elsewhere, drop mention of it.
2023-05-27 14:26:43 -03:00
Paul Wise
6d383d4049 notmuch-mutt: fix Xapian query construction
Spaces need to be stripped when querying the Message-Id,
because notmuch stores them in Xapian with spaces stripped.

All double-quote characters need to be doubled to escape them,
otherwise they will be added as extra query terms outside the id.
2023-05-27 14:24:40 -03:00
Paul Wise
d155f29eca notmuch-mutt: convert ISO-8859-15 copyright statement to UTF-8
Suggested-by: isutf8 & check-all-the-things
2023-05-25 08:43:14 -03:00
Felipe Contreras
afa45bd6b8 ruby: query: fix get sort
The order was wrong, right now `query.sort` doesn't return a number.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2023-05-25 07:02:59 -03:00
John Gliksberg
74c2c86769 vim: doc: minor spelling fix
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2023-05-07 14:27:33 -06:00
Felipe Contreras
3f4c61592f vim: doc: small fixes
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2023-05-07 14:27:10 -06:00
Jakub Wilk
1b878877d9 doc: fix typos 2023-04-13 11:14:58 -03:00
Felipe Contreras
e4d75fcc83 ruby: remove FileNames object
Not used anymore now that we return an array of strings directly.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2023-04-12 07:30:01 -03:00
Felipe Contreras
837426d7be ruby: filenames: return string array directly
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2023-04-12 07:29:48 -03:00
Felipe Contreras
777b02a7d7 ruby: add filenames helper
Right now it doesn't do much, but it will help for further
reorganization.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2023-04-12 07:05:23 -03:00
David Bremner
a554690d6a lib: index attachments with mime types matching index.as_text
Instead of skipping indexing all attachments, we check of a (user
configured) mime type that is indexable as text.
2023-04-02 19:24:43 -03:00
David Bremner
3f5809bf28 lib: parse index.as_text
We pre-parse into a list of compiled regular expressions to avoid
calling regexc on the hot (indexing) path.  As explained in the code
comment, this cannot be done lazily with reasonable error reporting,
at least not without touching a lot of the code in index.cc.
2023-04-02 19:22:36 -03:00
David Bremner
c6733a45c8 lib: add config key INDEX_AS_TEXT
Higher level processing as a list of regular expressions and
documentation will follow.
2023-04-02 19:21:37 -03:00
Felipe Contreras
44924a6a09 ruby: remove Tags object
Not used anymore now that we return an array of strings directly.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2023-04-02 19:11:10 -03:00
Felipe Contreras
3a52290609 ruby: tags: return string array directly
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2023-04-02 19:10:55 -03:00
Felipe Contreras
ae1336dea5 ruby: add tags helper
Right now it doesn't do much, but it will help for further
reorganization.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2023-04-02 19:00:49 -03:00
Kevin Boulain
6273966d0b lib: replace some uses of Query::MatchAll with a thread-safe alternative
This replaces two instances of Xapian::Query::MatchAll with the
equivalent but thread-safe alternative Xapian::Query(std::string()).
Xapian::Query::MatchAll maintains an internal pointer to a refcounted
Xapian::Internal::QueryTerm.

None of this is thread-safe but that wouldn't be an issue if
Xapian::Query::MatchAll wasn't static. Because it's static, the
refcounting goes awry when Notmuch is called from multiple threads.
This is actually documented by Xapian:
4715de3a9f/xapian-core/include/xapian/query.h (L65)

While static, Xapian::Query::MatchNothing is safe because it doesn't
maintain an internal object and as such, doesn't use references.

Two best-effort tests making use of TSan were added to showcase the
issue (I couldn't figure out a way to deterministically reproduce it
without making an unmaintainable mess).

First, when two databases are created in parallel, a query that uses
Xapian::Query::MatchAll is made (lib/query.cc), resulting in the
following backtrace on a segfault:
  #0  0x00007ffff76822af in Xapian::Query::get_terms_begin (this=0x7fffe80137f0) at api/query.cc:141
  #1  0x00007ffff7f933f5 in _notmuch_query_cache_terms (query=0x7fffe80137c0) at lib/query.cc:176
  #2  0x00007ffff7f93784 in _notmuch_query_ensure_parsed_xapian (query=0x7fffe80137c0) at lib/query.cc:225
  #3  0x00007ffff7f9381a in _notmuch_query_ensure_parsed (query=0x7fffe80137c0) at lib/query.cc:260
  #4  0x00007ffff7f93bfe in _notmuch_query_search_documents (query=0x7fffe80137c0, type=0x7ffff7fa9b1e "mail", out=0x7ffff666da18) at lib/query.cc:361
  #5  0x00007ffff7f93ba4 in notmuch_query_search_messages (query=0x7fffe80137c0, out=0x7ffff666da18) at lib/query.cc:349
  #6  0x00007ffff7f83d98 in notmuch_database_upgrade (notmuch=0x7fffe8000bd0, progress_notify=0x0, closure=0x0) at lib/database.cc:934
  #7  0x00007ffff7fa110f in notmuch_database_create_with_config (database_path=0x7ffff666dcb0 "/tmp/notmuch.MZ2AGr", config_path=0x7ffff7faab3c "", profile=0x0, database=0x0, status_string=0x7ffff666dc90) at lib/open.cc:754
  #8  0x00007ffff7fa0d6f in notmuch_database_create_verbose (path=0x7ffff666dcb0 "/tmp/notmuch.MZ2AGr", database=0x0, status_string=0x7ffff666dc90) at lib/open.cc:653
  #9  0x00007ffff7fa0ceb in notmuch_database_create (path=0x7ffff666dcb0 "/tmp/notmuch.MZ2AGr", database=0x0) at lib/open.cc:637
  ...

Second, some queries would make use of Xapian::Query::MatchAll
(lib/regexp-fields.cc), resulting in the following backtrace on a
segfault:
  #0  0x00007f629828b690 in Xapian::Internal::QueryBranch::gather_terms (this=0x7f628800def0, void_terms=0x7f629726d5a0) at api/queryinternal.cc:1245
  #1  0x00007f629828c260 in Xapian::Internal::QueryScaleWeight::gather_terms (this=0x7f628800df70, void_terms=0x7f629726d5a0) at api/queryinternal.cc:1434
  #2  0x00007f629828b69f in Xapian::Internal::QueryBranch::gather_terms (this=0x7f628800dd90, void_terms=0x7f629726d5a0) at api/queryinternal.cc:1245
  #3  0x00007f6298282571 in Xapian::Query::get_unique_terms_begin (this=0x7f628800dcd8) at api/query.cc:166
  #4  0x00007f629841a59b in Xapian::Weight::Internal::accumulate_stats (this=0x7f628800dca0, subdb=..., rset=...) at weight/weightinternal.cc:86
  #5  0x00007f62983c15ba in LocalSubMatch::prepare_match (this=0x7f628800df20, nowait=true, total_stats=...) at matcher/localsubmatch.cc:172
  #6  0x00007f62983c8fcc in prepare_sub_matches (leaves=std::vector of length 1, capacity 1 = {...}, stats=...) at matcher/multimatch.cc:237
  #7  0x00007f62983c98a3 in MultiMatch::MultiMatch (this=0x7f629726d9a0, db_=..., query_=..., qlen=3, omrset=0x0, collapse_max_=0, collapse_key_=4294967295, percent_cutoff_=0, weight_cutoff_=0, order_=Xapian::Enquire::ASCENDING, sort_key_=0, sort_by_=Xapian::Enquire::Internal::VAL, sort_value_forward_=true, time_limit_=0, stats=..., weight_=0x7f6288008d50, matchspies_=std::vector of length 0, capacity 0, have_sorter=false, have_mdecider=false) at matcher/multimatch.cc:353
  #8  0x00007f629826fcba in Xapian::Enquire::Internal::get_mset (this=0x7f628800e0b0, first=0, maxitems=0, check_at_least=0, rset=0x0, mdecider=0x0) at api/omenquire.cc:569
  #9  0x00007f629827181c in Xapian::Enquire::get_mset (this=0x7f629726db80, first=0, maxitems=0, check_at_least=0, rset=0x0, mdecider=0x0) at api/omenquire.cc:937
  #10 0x00007f6298be529a in _notmuch_query_search_documents (query=0x7f6288009750, type=0x7f6298bfaafe "mail", out=0x7f629726dcc0) at lib/query.cc:447
  #11 0x00007f6298be4ae8 in notmuch_query_search_messages (query=0x7f6288009750, out=0x7f629726dcc0) at lib/query.cc:349
  ...

Printing Xapian::Query::MatchAll->internal.px->_refs in these
circumstances can help quickly identifying this scenario.

This is motivated by some test frameworks (like Rust's Cargo) that
runs unit tests in parallel and would easily encounter this issue,
unless client code gates every call to Notmuch behind a lock.

This is what can be expected from the tests when they fail:
   == stderr ==
  +==================
  +WARNING: ThreadSanitizer: data race (pid=207931)
  +  Read of size 1 at 0x7b10000001a0 by thread T2:
  +    #0 memcpy <null> (libtsan.so.2+0x62506)
  +    #1 void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char*>(char*, char*, std::forward_iterator_tag) [clone .isra.0] <null> (libxapian.so.30+0x872b3)
  +
  +  Previous write of size 8 at 0x7b10000001a0 by thread T1:
  +    #0 operator new(unsigned long) <null> (libtsan.so.2+0x8ba83)
  +    #1 Xapian::Query::Query(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int, unsigned int) <null> (libxapian.so.30+0x855cd)
  ...
2023-03-31 08:11:39 -03:00
Felipe Contreras
4152e1bc20 ruby: database: make path arg optional
It can be automatically loaded from the configuration now.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2023-03-31 07:56:03 -03:00
Felipe Contreras
83ea220178 ruby: use database_open_with_config
Fixes warning:

  warning: ‘notmuch_database_open’ is deprecated: function deprecated as of libnotmuch 5.4

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2023-03-31 07:53:00 -03:00