Commit graph

1471 commits

Author SHA1 Message Date
David Bremner
1870b3ae4b lib/parse-sexp: support regular expressions
At least to the degree that the Xapian QueryParser based parser
also supports them. Support short alias 'rx' as it seems to make more
complex queries nicer to read.
2021-09-04 17:07:19 -07:00
David Bremner
0b98ad5e4e lib/query: generalize exclude handling to s-expression queries
In fact most of the code path is in common, only the caching of terms
in the query needs to be added for s-expression queries.
2021-09-04 17:07:19 -07:00
David Bremner
bafc307190 lib/parse-sexp: handle unprefixed terms.
This is equivalent to adding the same field name "" for multiple
prefixes in the Xapian query parser, but we have to explicitely
construct the resulting query.
2021-09-04 17:07:19 -07:00
David Bremner
0ca4ad2670 lib/parse-sexp: add '*' as syntactic sugar for '(starts-with "")'
Users that insist on using a literal '*' as a tag, can continue to do
so by quoting it when searching.
2021-09-04 17:07:19 -07:00
David Bremner
011d06f4d6 lib/parse-sexp: 'starts-with' wildcard searches
The many tests potentially overkill, but they could catch typos in the
prefixes table. As a simplifying assumption, for now we assume a
single argument to the wildcard operator, as this matches the Xapian
semantics. The name 'starts-with' is chosen to emphasize the supported
case of wildcards in currrent (1.4.x) Xapian.
2021-09-04 17:07:19 -07:00
David Bremner
8322f536f5 lib/parse-sexp: add term prefix backed fields
We use "boolean" to describe fields that should generate terms
literally without stemming or phrase splitting.  This terminology
might not be ideal but it is already enshrined in
notmuch-search-terms(7).
2021-09-04 17:07:19 -07:00
David Bremner
90d9c2ad5c lib/parse-sexp: support phrase queries.
Anything that is quoted or not purely word characters is considered a
phrase.  Phrases are not stemmed, because the stems do not have
positional information in the database. It is less efficient to scan
the term twice, but it avoids a second pass to add prefixes, so maybe
it balances out. In any case, it seems unlikely query parsing is very
often a bottleneck.
2021-09-04 17:07:19 -07:00
David Bremner
200e164dc7 lib/parse-sexp: support subject field
The broken tests are because we do not yet handle phrase searches.
2021-09-04 17:07:19 -07:00
David Bremner
f83cd2a05a lib/parse-sexp: support and, not, and or.
All operations and (Xapian) fields will eventually have an entry in
the prefixes table. The flags field is just a placeholder for now, but
will eventually distinguish between various kinds of prefixes.
2021-09-04 17:07:19 -07:00
David Bremner
a2785c3919 lib/parse-sexp: stem unquoted atoms
This is somewhat less DWIM than the Xapian query parser, but it has
the advantage of simplicity.
2021-09-04 17:07:19 -07:00
David Bremner
be7e83de96 lib/parse-sexp: parse single terms and the empty list.
There is not much of a parser here yet, but it already does some
useful error reporting. Most functionality sketched in the
documentation is not implemented yet; detailed documentation will
follow with the implementation.
2021-09-04 17:07:19 -07:00
David Bremner
e3b1a0a6a5 CLI/search+address: support sexpr queries
Initially support selection of query syntax in two subcommands to
enable testing.
2021-09-04 17:07:19 -07:00
David Bremner
a83ad52da4 configure: optional library sfsexp
The configure part is essentially the same as the other checks using
pkg-config. Since the optional inclusion of this feature changes what
options are available to the user, include it in the "built_with"
pseudo-configuration keys.
2021-09-04 17:07:19 -07:00
David Bremner
d447b694b4 CLI: make variable n_requested_db_uuid file scope.
It turns out that now that we pass an open database into the
subcommands, it is easy to check any requested uuid against the
database at the same time as we process the other shared
arguments. This results in overall less boilerplate code, as well as
making a CLI scope function and variable file scope in notmuch.c.
2021-09-04 17:07:19 -07:00
David Bremner
06b536dae0 test: known broken test for emacs display of message/rfc822 parts
There are at least 3 bugs present.

1) notmuch-show-insert-part-message/rfc822 assumes that message/rfc822
parts will have a ":content" property, but that turns out not to be
the case.

2) something deep in gnus wants gnus-newsgroup-charset, but that is
defined in gnus-sum, which is not loaded by default.

3) If gnus-sum is loaded, then the display of the message/rfc822 part
succeeds, but the buffer gets put into gnus-article-mode, which means
that, inter alia, notmuch text properties and keybindings get wiped.
2021-08-30 16:29:04 -07:00
David Bremner
e1c56f49de test/crypto: test message with rfc822 attachment.
This is intended to help track down a display problem in the emacs
front end
2021-08-30 16:24:59 -07:00
Hannu Hartikainen
717e3dcdc3 lib: consider all instances of Delivered-To header
When using notmuch-reply and guessing the From: address from
Delivered-To headers, I had the wrong address chosen today. This was
because the messages from the notmuch list contain these headers in this
order:

Delivered-To: hannu.hartikainen@gmail.com
...
Delivered-To: hannu@hrtk.in

In my .notmuch-config I have the following configuration:

primary_email=hannu@hrtk.in
other_email=hannu.hartikainen@gmail.com;...

Before this change, notmuch-reply would guess From: @gmail.com because
that is the first Delivered-To header present. After the change, the
primary address is chosen as I would expect.
2021-08-29 18:10:08 -07:00
Hannu Hartikainen
8c0bebedc5 reply: add test for multiple Delivered-To headers
Add a known broken subtest for guessing From: correctly when there are
multiple Delivered-To: headers. The address configured as primary_email
should get picked.
2021-08-29 18:07:59 -07:00
David Bremner
731697d671 CLI: define and use format version 5
This is a bit of a cheat, since the format does not actually
change. On the other hand it is fairly common to do something like
this to shared libary SONAMEs when the ABI changes in some subtle way.
It does rely on the format-version argument being early enough on the
command line to generate a sensible error message.
2021-08-22 07:05:13 -07:00
David Bremner
4f84c01b3a test/path-config: use test_expect_equal_file_nonempty
This is more robust against crashes when the expected output is also
generated by notmuch. In the case where the expected output is
explicit, it seems like overkill.
2021-08-21 14:56:13 -07:00
David Bremner
474a7f8e65 test: add test_expect_equal_file_nonempty
A common bug in tests is that the code used to generate the EXPECTED
file fails, generating no output. When the code generating the OUTPUT
file fails in the same way, the test passes, even though there is a
failure being hidden. Add a new test function that guards against
this.
2021-08-21 14:55:41 -07:00
David Bremner
78045301e7 test: factor out test_diff_file_
A following commit will use this new function in a public test
function.
2021-08-21 14:55:27 -07:00
David Bremner
f2e6f311c7 test/expect_equal_file: whitespace cleanup
No functional change, just reindent to Emacs defaults.
2021-08-21 14:51:04 -07:00
David Bremner
3df2281746 notmuch release 0.32.3-1 for unstable (sid) [dgit]
[dgit distro=debian no-split --quilt=linear]
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEkiyHYXwaY0SiY6fqA0U5G1WqFSEFAmEclwwACgkQA0U5G1Wq
 FSEdBw//cF+LAJy9qyWX8axQcSwo1/ixZkyxSCc0JVaDhhAal5GdVd3E+vDKZp3Q
 KWExBKncTAciB1CooKNurnvFr30s9MX+SrDK0rPCqXtObMRYS96AEDCvul9Z9mBT
 eE6thb66Y2Mrr2dyHiyG/GHrp9wces+sSnB6xM6bqO16c9UsMuqjTIIqlb0A7oP8
 eN/LLPAcVoXIMry3hVQWzv6bJWfqQK9Ho66p1fH+QswgeeKIzh5HR3ZnGdsBhnyN
 n3l9QPYuhN7/85cjWH4AE0/2EQLIF/Ewu86qYHzvOufc7oiNXRGpzQtMNqqur/da
 JIOizaylhLkFP39i/7CHvldCTx0FOCVhbHtFwb1Hm7rPqr8DhOvXyZbDHK/LkWtg
 jBKVFld4nNv/zm1uz6S3E5rDFsxuvpMVuesWKj6Yb051QvfGqVTnwRHP3EFxMyk8
 CVle5a3mMg8bnlN9o/WchB3z3ybijK4HLjGB+FWDZMUvhKXkfZL6GUeo/EQ8byaX
 mDEx5u56euqgNQS1PnIF/nTaOCbu75IqYAUSqwVXBcICi7LfYKDwGezBiEhoRiKt
 RQoulN9FNS+HuL+85LN4TdZyJOvD1Rc9xdT9TwuLr47nIc/KdAWUxyYSEo4PMiZ+
 NNiCatI5kPJrp0q1A5xDRkpFavLu5Mtu9ore+Cf1nA18iEFNQwc=
 =cECE
 -----END PGP SIGNATURE-----

Merge tag 'debian/0.32.3-1'

notmuch release 0.32.3-1 for unstable (sid) [dgit]

[dgit distro=debian no-split --quilt=linear]
2021-08-18 21:46:42 -07:00
David Bremner
686230712f CLI/config: restore "notmuch config get built_with.*"
We need to special case the config section "built_with" because it is
not (currently) handled by the library. This seems consist with the
other sub-sub-commands 'list' and 'set'.
2021-08-17 17:09:56 -07:00
David Bremner
202dde04e1 test/config: add tests for built_with
The "get" test is known broken because this functionality was dropped
during the 0.32 config rewrite.
2021-08-17 17:09:38 -07:00
David Bremner
d930011690 lib/open: look in MAILDIR for database, as documented.
This fixes the bug id:87bl9lx864.fsf@kisara.moe
2021-08-17 17:09:21 -07:00
David Bremner
8376e81441 test: add known broken tests for finding database via MAILDIR
This highlights a bug reported by several users, including
Mohsin Kaleem [1].

The inconsistent use of test_begin_subtest_known_broken is because
some of these tests pass even though the database cannot be
located. This problem is left for a future commit.

[1]: id:87bl9lx864.fsf@kisara.moe
2021-08-17 17:07:53 -07:00
David Bremner
3fc2e5f23b test: revert to mkdir -p to make results directory.
As suggested by id:m21r7al3mt.fsf@guru.guru-group.fi
2021-08-03 20:35:13 -03:00
David Bremner
c966fa6439 test: use --minversion to detect GNU Parallel.
Based on a suggestion of Ole Tange [1].

[1]: id:CA+4vN7x6jp4HCiEybZ=5g+2X6Pa7etBFX3Bbd=UYty37gR6wEQ@mail.gmail.com
2021-08-03 20:29:47 -03:00
jao
13d49b8e16 CLI/show: tests for the new --sort option
New unit tests for notmuch show --sort, covering the basic use cases.
2021-07-03 20:27:39 -03:00
Michael J Gruber
1768b0c416 test: deduplicate T590
Test numbers are a concise way to communicate about tests and to remeber
them. Currently, there is one pait of duplicates:

T590-libconfig.sh
T590-thread-breakage.sh

Renumber the latter one to 592 since this keeps the alphabetic order and
leaves room in between.

Signed-off-by: Michael J Gruber <git@grubix.eu>
2021-06-29 13:43:24 -03:00
David Bremner
37f84d6d21 emacs/tree use notmuch-show-single-message
This is more efficient than notmuch-show-only-matching-messages, since
we do not parse the potentially large thread structure to find a
single message.

This is only a partial fix for notmuch-tree view, because displaying
the thread structure in the tree-mode window still crashes on long
threads. It is however enough to make unthreaded view handle long
threads.
2021-06-27 14:41:23 -03:00
David Bremner
14326e6cb7 test: start test file for emacs unthreaded view.
Initial focus is on behaviour with large threads.

The second test replicates a bug reported by Alan Schmitt in
id:87lf7sojbq.fsf@m4x.org.
2021-06-27 14:29:08 -03:00
Felipe Contreras
814abafc3e ruby: add keyword arguments to db.query
That way we don't need pass them to the query object ourselves.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-06-27 14:13:03 -03:00
David Bremner
e2a3e5fa51 lib: autocommit after some number of completed transactions
This change addresses two known issues with large sets of changes to
the database.  The first is that as reported by Steven Allen [1],
notmuch commits are not "flushed" when they complete, which means that
if there is an open transaction when the database closes (or e.g. the
program crashes) then all changes since the last commit will be
discarded (nothing is irrecoverably lost for "notmuch new", as the
indexing process just restarts next time it is run).  This does not
really "fix" the issue reported in [1]; that seems rather difficult
given how transactions work in Xapian. On the other hand, with the
default settings, this should mean one only loses less than a minutes
worth of work.  The second issue is the occasionally reported "storm"
of disk writes when notmuch finishes. I don't yet have a test for
this, but I think committing as we go should reduce the amount of work
when finalizing the database.

[1]: id:20151025210215.GA3754@stebalien.com
2021-06-27 14:03:00 -03:00
David Bremner
8aabddb043 test: add known broken test for closing with open transaction
The expected output may need adjusting, but what is clear is that
saving none of the changes is not desirable.
2021-06-27 14:01:26 -03:00
David Bremner
2f608d2a94 lib/config: add NOTMUCH_CONFIG_AUTOCOMMIT
This will be used to control how often atomic transactions are
committed.
2021-06-27 13:59:42 -03:00
Daniel Kahn Gillmor
8c29a5da09 cli/show: produce "email" element in sigstatus
When the certificate that signs a message is known to be valid, GMime
is capable of reporting on the e-mail address embedded in the
certificate.

We pass this information along to the caller of "notmuch show", as
often only the e-mail address of the certificate has actually been
checked/verified.

Furthermore, signature verification should probably at some point
compare the e-mail address of the caller against the sender address of
the message itself.  Having to parse what gmime thinks is a "userid"
to extract an e-mail address seems clunky and unnecessary if gmime
already thinks it knows what the e-mail address is.

See id:878s41ax6t.fsf@fifthhorseman.net for more motivation and discussion.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2021-06-26 13:07:47 -03:00
David Bremner
7406abf5e0 test/libconfig: make indexing more consistent
The other tests have converted to two digit decimal, the commit does
the same to the remaining test that was printing out indices in hex.
2021-06-21 06:57:17 -03:00
David Bremner
8dbd5deb8d Merge branch 'release' 2021-06-20 17:15:20 -03:00
David Bremner
a19d2b0d2b test: check openssl prequisite for add_gpgsm_home
This is a fix for the test failures reported by Dan Čermák [1].

It is more robust to check for the prerequisite inside the function
that uses it, rather than in every test file that calls the function.

[1]: id:87k0n4fqgm.fsf@tethera.net
2021-06-08 17:15:46 -03:00
Tomi Ollila
6f0f83660e test: aggregate-results updates
notmuch-test will now call aggregate-results.sh with file list
that it compiles based on the test ran, and aggregate-results
will report failure is any of the test files are missing.

With this notmuch-test no longer has to exit in non-parallel
run if some test fail to write its report file -- so it works
as parallel tests in this sense.

Changed test_done() in test-lib.sh write report file in one write(2),
so there is (even) less chance it being partially written. Also,
now it writes 'total' last and aggregate-results.sh expects this
line to exist in all report files for reporting to be successful.

Added 'set -eu' to notmuch-test and modified code to work with
these settings. That makes it harder to get mistakes slipped
into committed code.
2021-06-07 20:16:33 -03:00
Jani Nikula
87234a8ef3 emacs: update default mailing list archives for stashing links
Gmane web interface is long gone, remove it. Make MARC the new
default. Update LKML to Lore, where it already redirects anyway. Also
add Notmuch web archive.
2021-06-05 15:39:53 -03:00
David Bremner
2c85cc29bd test: add known broken test for indexing RFC822 group names
Austin Clements diagnosed this indexing problem in [1].

[1]: id:20130711215207.GR2214@mit.edu
2021-06-05 15:31:03 -03:00
Tomi Ollila
572af27950 test: source $NOTMUCH_SRCDIR/test/test-lib-emacs.sh
Sourcing test-lib.sh will cd to TMP_DIRECTORY, so
relative path in $0 will not work in previous version
 . $(dirname "$0")/test-lib-emacs.sh

Now individual test scripts -- e.g. ./test/T310-emacs.sh
will work.
2021-06-03 09:29:27 -03:00
Felipe Contreras
85ae2bcf56 ruby: use notmuch_exclude_t enum
It exists since 2013, let's allow it to be used in Ruby.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-23 09:05:33 -03:00
David Bremner
f44c83c083 test: add regression test for large number of deletions
Prior to 9ad19e4454 there was an unhandled Xapian exception when
reindexing after a large number of deletes.  This test was used for
bisection, and will subsequently serve as a regression test.
2021-05-23 08:31:36 -03:00
Tomi Ollila
f2533b9e73 test: say_color() in one write(2)
say_color() used to call (builtin) printf (and tput(1) to stdout)
several times, which caused attempts to write messages with color
to have partial content (e.g. escape sequences) often intermixed
with other tests when parallel tests were run.

Now, with all output collected, then written out using one
printf, all strings with color print out correctly
((at least short) write(2)'s appear to write out "atomically").

While at it, used only one tput(1) execution to determine whether
color output works, and made bold/colors/sgr0 to tput(1) their
values once per test.
2021-05-23 08:05:15 -03:00
David Bremner
69c2c930ec test: quiet some extra debugging output
This output does not cause test failures, but may make it harder to
interpret the output.
2021-05-23 08:04:07 -03:00