Commit graph

1673 commits

Author SHA1 Message Date
David Bremner
112c53535a test: add known broken test for bad utf8 in config
We should ideally print an informative error message, but at the very
least we should not exit with success.
2023-09-23 08:22:51 -03:00
David Bremner
115d4d69eb test: minimize impact of native compilation.
Native compilation is kindof useless in the test suite because we
throw away the cache after every subtest.  The test suite could in
principle share an eln cache within a given test file; for now try to
minimize the amount of native-compilation. There is an intermittent
bug where emacs loses track of its default-directory; I suspect (but
have no proof) that bug is related to native compilation and/or race
conditions. This patch seems to prevent that bug (or at least reduce
its frequency).
2023-08-26 07:45:36 -03:00
David Bremner
368e0f61d5 test/setup: ignore blank lines in generated config
The presense of the blank lines between sections depends on the
version of glib. Strip them before comparison.
2023-08-21 19:30:54 -03:00
David Bremner
1719b9e568 test/emacs: adapt to breaking change in Gnus defaults
As of Emacs 29.1, In-Reply-To is in the default value for
message-hidden-headers. We actually want to see that in the test
suite, so remove it again. To future proof the tests, fix a default
value for message-hidden-headers specifically for the test suite.
2023-08-20 14:32:02 -03:00
David Bremner
ec26eeaeec test: support testing notmuch as installed
We put some effort into testing the built copy rather than some
installed copy. On the other hand for people like packagers, testing
the installed copy is also of interest.

When NOTMUCH_TEST_INSTALLED is set to a nonempty value, tests do not
require a built notmuch tree or running configure.

Some of the tests marked as broken when running against installed
notmuch are probably fixable.
2023-07-21 07:41:50 -03:00
David Bremner
73f3081160 test: Guess a value for NOTMUCH_PYTHON
python3 will work for many people, and reduce the friction to running
the tests without running configure first.
2023-07-21 07:07:43 -03:00
David Bremner
f6fcdf12da test: check for empty/missing files in test_expect_equal_message_body
Messages can have empty bodies, but empty files are not messages.
2023-07-21 07:07:43 -03:00
David Bremner
dfa43a1921 test: treat undefined feature variables as 0
When running the test suite without building first, it is desirable to
have the tests consider these variables being undefined as equivalent
to the feature not being present, and in particular for the tests not
to generate errors.
2023-07-21 06:58:16 -03:00
David Bremner
808fd6f258 test: add known broken test for message-id with embedded spaces.
According to my reading of RFC5322, there is an obsolete syntax for
Message-Id which permits folding whitespace (i.e. to be removed /
ignored by parsers). In [1] Paul Wise observed that notmuch removed
whitespace on indexing, but does not do any corresponding
normalization of queries. Mark the latter as a bug by adding a failing
test.

[1]: id:20230409044143.4041560-1-pabs3@bonedaddy.net
2023-07-09 11:59:43 -03:00
David Bremner
a554690d6a lib: index attachments with mime types matching index.as_text
Instead of skipping indexing all attachments, we check of a (user
configured) mime type that is indexable as text.
2023-04-02 19:24:43 -03:00
David Bremner
c6733a45c8 lib: add config key INDEX_AS_TEXT
Higher level processing as a list of regular expressions and
documentation will follow.
2023-04-02 19:21:37 -03:00
Kevin Boulain
6273966d0b lib: replace some uses of Query::MatchAll with a thread-safe alternative
This replaces two instances of Xapian::Query::MatchAll with the
equivalent but thread-safe alternative Xapian::Query(std::string()).
Xapian::Query::MatchAll maintains an internal pointer to a refcounted
Xapian::Internal::QueryTerm.

None of this is thread-safe but that wouldn't be an issue if
Xapian::Query::MatchAll wasn't static. Because it's static, the
refcounting goes awry when Notmuch is called from multiple threads.
This is actually documented by Xapian:
4715de3a9f/xapian-core/include/xapian/query.h (L65)

While static, Xapian::Query::MatchNothing is safe because it doesn't
maintain an internal object and as such, doesn't use references.

Two best-effort tests making use of TSan were added to showcase the
issue (I couldn't figure out a way to deterministically reproduce it
without making an unmaintainable mess).

First, when two databases are created in parallel, a query that uses
Xapian::Query::MatchAll is made (lib/query.cc), resulting in the
following backtrace on a segfault:
  #0  0x00007ffff76822af in Xapian::Query::get_terms_begin (this=0x7fffe80137f0) at api/query.cc:141
  #1  0x00007ffff7f933f5 in _notmuch_query_cache_terms (query=0x7fffe80137c0) at lib/query.cc:176
  #2  0x00007ffff7f93784 in _notmuch_query_ensure_parsed_xapian (query=0x7fffe80137c0) at lib/query.cc:225
  #3  0x00007ffff7f9381a in _notmuch_query_ensure_parsed (query=0x7fffe80137c0) at lib/query.cc:260
  #4  0x00007ffff7f93bfe in _notmuch_query_search_documents (query=0x7fffe80137c0, type=0x7ffff7fa9b1e "mail", out=0x7ffff666da18) at lib/query.cc:361
  #5  0x00007ffff7f93ba4 in notmuch_query_search_messages (query=0x7fffe80137c0, out=0x7ffff666da18) at lib/query.cc:349
  #6  0x00007ffff7f83d98 in notmuch_database_upgrade (notmuch=0x7fffe8000bd0, progress_notify=0x0, closure=0x0) at lib/database.cc:934
  #7  0x00007ffff7fa110f in notmuch_database_create_with_config (database_path=0x7ffff666dcb0 "/tmp/notmuch.MZ2AGr", config_path=0x7ffff7faab3c "", profile=0x0, database=0x0, status_string=0x7ffff666dc90) at lib/open.cc:754
  #8  0x00007ffff7fa0d6f in notmuch_database_create_verbose (path=0x7ffff666dcb0 "/tmp/notmuch.MZ2AGr", database=0x0, status_string=0x7ffff666dc90) at lib/open.cc:653
  #9  0x00007ffff7fa0ceb in notmuch_database_create (path=0x7ffff666dcb0 "/tmp/notmuch.MZ2AGr", database=0x0) at lib/open.cc:637
  ...

Second, some queries would make use of Xapian::Query::MatchAll
(lib/regexp-fields.cc), resulting in the following backtrace on a
segfault:
  #0  0x00007f629828b690 in Xapian::Internal::QueryBranch::gather_terms (this=0x7f628800def0, void_terms=0x7f629726d5a0) at api/queryinternal.cc:1245
  #1  0x00007f629828c260 in Xapian::Internal::QueryScaleWeight::gather_terms (this=0x7f628800df70, void_terms=0x7f629726d5a0) at api/queryinternal.cc:1434
  #2  0x00007f629828b69f in Xapian::Internal::QueryBranch::gather_terms (this=0x7f628800dd90, void_terms=0x7f629726d5a0) at api/queryinternal.cc:1245
  #3  0x00007f6298282571 in Xapian::Query::get_unique_terms_begin (this=0x7f628800dcd8) at api/query.cc:166
  #4  0x00007f629841a59b in Xapian::Weight::Internal::accumulate_stats (this=0x7f628800dca0, subdb=..., rset=...) at weight/weightinternal.cc:86
  #5  0x00007f62983c15ba in LocalSubMatch::prepare_match (this=0x7f628800df20, nowait=true, total_stats=...) at matcher/localsubmatch.cc:172
  #6  0x00007f62983c8fcc in prepare_sub_matches (leaves=std::vector of length 1, capacity 1 = {...}, stats=...) at matcher/multimatch.cc:237
  #7  0x00007f62983c98a3 in MultiMatch::MultiMatch (this=0x7f629726d9a0, db_=..., query_=..., qlen=3, omrset=0x0, collapse_max_=0, collapse_key_=4294967295, percent_cutoff_=0, weight_cutoff_=0, order_=Xapian::Enquire::ASCENDING, sort_key_=0, sort_by_=Xapian::Enquire::Internal::VAL, sort_value_forward_=true, time_limit_=0, stats=..., weight_=0x7f6288008d50, matchspies_=std::vector of length 0, capacity 0, have_sorter=false, have_mdecider=false) at matcher/multimatch.cc:353
  #8  0x00007f629826fcba in Xapian::Enquire::Internal::get_mset (this=0x7f628800e0b0, first=0, maxitems=0, check_at_least=0, rset=0x0, mdecider=0x0) at api/omenquire.cc:569
  #9  0x00007f629827181c in Xapian::Enquire::get_mset (this=0x7f629726db80, first=0, maxitems=0, check_at_least=0, rset=0x0, mdecider=0x0) at api/omenquire.cc:937
  #10 0x00007f6298be529a in _notmuch_query_search_documents (query=0x7f6288009750, type=0x7f6298bfaafe "mail", out=0x7f629726dcc0) at lib/query.cc:447
  #11 0x00007f6298be4ae8 in notmuch_query_search_messages (query=0x7f6288009750, out=0x7f629726dcc0) at lib/query.cc:349
  ...

Printing Xapian::Query::MatchAll->internal.px->_refs in these
circumstances can help quickly identifying this scenario.

This is motivated by some test frameworks (like Rust's Cargo) that
runs unit tests in parallel and would easily encounter this issue,
unless client code gates every call to Notmuch behind a lock.

This is what can be expected from the tests when they fail:
   == stderr ==
  +==================
  +WARNING: ThreadSanitizer: data race (pid=207931)
  +  Read of size 1 at 0x7b10000001a0 by thread T2:
  +    #0 memcpy <null> (libtsan.so.2+0x62506)
  +    #1 void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char*>(char*, char*, std::forward_iterator_tag) [clone .isra.0] <null> (libxapian.so.30+0x872b3)
  +
  +  Previous write of size 8 at 0x7b10000001a0 by thread T1:
  +    #0 operator new(unsigned long) <null> (libtsan.so.2+0x8ba83)
  +    #1 Xapian::Query::Query(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int, unsigned int) <null> (libxapian.so.30+0x855cd)
  ...
2023-03-31 08:11:39 -03:00
Felipe Contreras
4152e1bc20 ruby: database: make path arg optional
It can be automatically loaded from the configuration now.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2023-03-31 07:56:03 -03:00
Kevin Boulain
a95959c491 test: add test for notmuch_message_remove_all_properties_with_prefix
It wasn't covered, though it shares most of its implementation with
notmuch_message_remove_all_properties.
2023-03-30 08:07:57 -03:00
Kevin Boulain
fb55ff28a2 lib/message-property: sync removed properties to the database
_notmuch_message_remove_all_properties wasn't syncing the message back
to the database but was still invalidating the metadata, giving the
impression the properties had actually been removed.

Also move the metadata invalidation to _notmuch_message_remove_terms
to be closer to what's done in _notmuch_message_modify_property and
_notmuch_message_remove_term.
2023-03-30 08:01:09 -03:00
David Bremner
3363349967 test: reveal notmuch_message_remove_all_properties as broken
Close and re-open the database to show that the removal is not
committed to the database.
2023-03-30 07:56:17 -03:00
Kevin Boulain
e191d3c574 test: reorganize tests and mark a few of them as broken
notmuch_message_remove_all_properties should have removed the
  testkey1 = testvalue1
property but hasn't. Delay the execution of the corresponding test
to avoid updating a few tests that actually relied on the broken
behavior.
2023-03-30 07:49:07 -03:00
Kevin Boulain
552d9ec9f7 test: remove unnecessary sorting
The other tests rely on a stable output.
2023-03-30 07:48:27 -03:00
Kevin Boulain
ce59df5296 test: display key name in property tests
To make the tests a bit easier to understand.
2023-03-30 07:48:19 -03:00
Kevin Boulain
568f6bc3c2 lib/message-property: catch xapian exceptions
Since libnotmuch exposes a C interface there's no way for clients to
catch this.
Inspired by what's done for tags (see notmuch_message_remove_tag).
2023-03-30 07:08:47 -03:00
Kevin Boulain
c810312e24 test: uncaught exception when editing properties of a removed message
These two functions don't fail gracefully when editing a removed
message:
 BROKEN edit property on removed message without uncaught exception
        --- T610-message-property.20.EXPECTED   2023-02-27 11:33:25.792764376 +0000
        +++ T610-message-property.20.OUTPUT     2023-02-27 11:33:25.793764381 +0000
        @@ -1,2 +1,3 @@
         == stdout ==
         == stderr ==
        +terminate called after throwing an instance of 'Xapian::DocNotFoundError'

The other functions appear to be safe.
2023-03-30 07:06:48 -03:00
David Bremner
09f2ad8e85 lib: add better diagnostics for over long filenames.
Previously we just crashed with an internal error. With this change,
the caller can handle it better. Update notmuch-new so that it doesn't
crash with "unknown error code" because of this change.
2023-02-20 09:22:32 -04:00
David Bremner
4e6c6c8aac test: add known broken test for diagnostics from over long filenames.
Previously we tested over long directory names, add similar testing
for over long filenames.
2023-02-20 09:22:07 -04:00
jao
f63d14a8c1 emacs: notmuch-tree-outline-mode
With this mode, one can fold trees in the notmuch-tree buffer as if
they were outlines, using all the commands provided by
outline-minor-mode.  We also define a couple of movement commands
that, optional, will ensure that only the thread around point is
unfolded.

The implementation is based on registering a :level property in the
messages p-list, that is then used by outline-minor-mode to to
recognise headers.

Amended by db: Copy docstring to manual and edit for presentation. Add
two tests. Fix typo "wether".
2023-02-20 08:58:32 -04:00
David Bremner
f4dc32e71b test: mark some tests as broken when run as root.
File permission errors e.g., are hard to trigger as root.
2023-01-05 20:14:37 -04:00
David Bremner
1d5d0ae686 lib/message: move xapian call inside try/catch block in _n_m_delete
The call to delete_document can throw exceptions (and can happen in
practice [1]), so catch the exception and extract the error
message. As a side effect, also move the call to _n_m_has_term inside
the try/catch. This should not change anything as that function
already traps any Xapian exceptions.

[1]: id:wwuk039sk2p.fsf@chaotikum.eu
2022-12-27 11:59:46 -04:00
David Bremner
966f40086f test: add known broken test for exception handling in _n_m_delete
In [1], Thomas Schneider reported an uncaught Xapian exception when
running out of disk space. We generate the same exception via database
corruption.

[1]: id:wwuk039sk2p.fsf@chaotikum.eu
2022-12-27 11:59:38 -04:00
Robin Jarry
793f298091 cli: add options --offset and --limit to notmuch show
notmuch search does not output header values. However, when browsing
through a large email corpus, it can be time saving to be able to
paginate without running notmuch show for each message/thread.

Add --offset and --limit options to notmuch show. This is inspired from
commit 796b629c3b ("cli: add options --offset and --limit to notmuch
search").

Update man page, shell completion and add a test case to ensure it works
as expected.

Cc: Tim Culverhouse <tim@timculverhouse.com>
Cc: Tomi Ollila <tomi.ollila@iki.fi>
Signed-off-by: Robin Jarry <robin@jarry.cc>
2022-11-05 13:18:15 -04:00
Justus Winter
bf8aa34324 test: replace aging OpenPGP key used in the test suite
This replaces the old OpenPGPv4 key that is used in the test suite
with a more modern OpenPGPv4 key.  All cryptographic artifacts in the
test suite are updated accordingly.

Having old cryptographic artifacts in the test suite presents a
problem once the old algorithms are rejected by contemporary
implementations.

For reference, this is the old key.

  sec   rsa1024 2011-02-05 [SC]
        5AEAB11F5E33DCE875DDB75B6D92612D94E46381
  uid           [ unknown] Notmuch Test Suite <test_suite@notmuchmail.org> (INSECURE!)
  ssb   rsa1024 2011-02-05 [E]

And this is the new key.  Note that is has the same shape, but uses
Ed25519 and Cv25519 instead of 1024-bit RSA.

  sec   ed25519 2022-09-07 [SC]
        9A3AFE6C60065A148FD4B58A7E6ABE924645CC60
  uid           [ultimate] Notmuch Test Suite (INSECURE!) <test_suite@notmuchmail.org>
  ssb   cv25519 2022-09-07 [E]
2022-09-23 20:16:00 -03:00
Justus Winter
acb31939bb test: compute expected keyid from fingerprint 2022-09-19 22:19:35 -03:00
David Bremner
8eabd6388e test: add known broken test for indexing text/* attachments
The general problem of indexing attachments requires some help to turn
things into text, but (most?) text/* should be doable internally,
possibly with optimizations as for the text/html case.
2022-09-03 09:06:08 -03:00
David Bremner
a832f940e1 test: rename indexing corpus
The corpus is not really suitable for general indexing test since the
sole message is ignored (and will most likely continue to be ignored)
by notmuch-new.
2022-09-03 09:05:44 -03:00
David Bremner
2e5ef69fbf lib: add field processor for lastmod: prefix
By sharing the existing logic used by the sexp query parser, this
allows negative lastmod revisions to be interpreted as relative to the
most recent revision.
2022-09-03 08:43:33 -03:00
David Bremner
606d9b02e4 lib/sexp: provide relative lastmod queries
Test the relatively trivial logic changes for the sexp query parser
first before refactoring that logic to share with the infix query
parser.
2022-09-03 08:36:53 -03:00
David Bremner
84e4e130e2 lib/open: create database path in some cases
There is some duplication of code here, but not all of the locations
valid to find a database make sense to create. Furthermore we nead two
passes, so the control flow in _choose_database_path would get a bit
convoluted.
2022-09-03 08:24:43 -03:00
David Bremner
8ba3057d01 lib/open: return non-SUCCESS on missing database path
This simplifies the logic of creating the directory path when it doesn't
exist.
2022-09-03 08:24:43 -03:00
David Bremner
44c9338061 test/path-config: set database.mail_root but not database.path
The failing "create database" test replicates a bug reported by Sean
Whitton [1]. The other two failures also look related to the database
being (re)created in the wrong place.

[1]: id:87y1wqkw13.fsf@athena.silentflame.com.
2022-09-03 08:24:43 -03:00
David Bremner
687866111a test: add test for creating database in various configurations.
The existing database creation (via add_email_corpus) was always done
in the traditional configuration. The use of xapian-metadata is just
to portably ensure that there is a database created where we expect
there to be.
2022-09-03 08:24:43 -03:00
David Bremner
d273263d16 emacs/show: restrict inlined mimetypes on refresh.
This fixes the bug reported by Al [1]. Essentially apply the same fix
as [2] in a different place.

[1]: id:877d41nmr1.fsf@gmail.com
[2]: 90a7c1af36
2022-08-11 07:17:17 -03:00
David Bremner
4de3eb4c0e test/emacs: add known broken test for inlining on refresh.
This duplicates the bug reported by Al [1]

[1]: id:877d41nmr1.fsf@gmail.com
2022-08-11 07:17:17 -03:00
David Bremner
6d6d2a5fe7 emacs: add duplicate support to notmuch-show-view-raw-message
This fixes one of the bugs reported by Gregor in [1]

[1]: id:87edxw8jp4.fsf@no.workgroup
2022-08-07 07:05:49 -03:00
David Bremner
67f535332c test/emacs: add known broken test for viewing raw duplicate
This duplicates one of the bugs reported by Gregor in [1].

[1]: id:87edxw8jp4.fsf@no.workgroup
2022-08-07 07:05:49 -03:00
David Bremner
1849c68d0f emacs: update notmuch-show-get-filename to handle duplicates
This fixes one of the bugs reported by Gregor in [1].

[1]: id:87edxw8jp4.fsf@no.workgroup
2022-08-07 07:05:49 -03:00
David Bremner
2c7438fe51 test/emacs: add known broken test for stashing duplicate filename
This duplicates (no pun intended) one of the bugs reported by Gregor
in id:87edxw8jp4.fsf@no.workgroup
2022-08-07 07:05:49 -03:00
David Bremner
7b27119d8e tests: remove dead code from T055-path-config.sh
This case statement does nothing.
2022-08-06 09:16:13 -03:00
David Bremner
1ef7c75111 emacs/reply: reply to correct duplicate
Essentially we just need to arrange to pass the right --duplicate
argument to notmuch reply.

As a side-effect, correct the previously unused value of EXPECTED in
T453-emacs-reply.sh.
2022-07-30 08:46:22 -03:00
David Bremner
24decfc1dd CLI/reply: support --duplicate argument
We want the reply used to match that shown e.g. in the emacs
interface. As a first step provide that functionality on the command
line.

Schema does not need updating as the duplicate key was already
present (with a constant value of 1).
2022-07-30 08:45:15 -03:00
David Bremner
5ea5a5557d emacs/show: display count of duplicates in headerline
There is no real cost here, except screen real estate. Some people
might prefer hiding the duplicate count, but we leave that for a
future commit.
2022-07-30 08:44:14 -03:00
David Bremner
b3d466bf39 emacs/show: provide notmuch-show-choose-duplicate
This new command allows the user to interactively choose a different
duplicate (file) to display for a given message in
notmuch-show-mode. Since both tree and unthreaded view use
notmuch-show-mode, this provides the same facility there.
2022-07-30 08:42:12 -03:00
David Bremner
4612f3eb3d CLI/show: support --duplicate for structured output
This introduces a new mandatory key for message structures, namely
"duplicate". Per convention in devel/schemata this does _not_ increase
the format version. This means that clients are responsible for
checking that it exists, and not crashing if it does not.

The main functional change is teaching mime_node_open to understand a
'duplicate' argument.

Support for --duplicate in notmuch-reply would make sense, but we
defer it to a later commit.
2022-07-30 08:41:50 -03:00