Commit graph

1678 commits

Author SHA1 Message Date
David Bremner
551254eb76 lib/parse-sexp: apply macros
Macros implement lazy evaluation and lexical scope.  The former is
needed to make certain natural constructs work sensibly (e.g. (tag
,param)) but the latter is mainly future-proofing in case the DSL is
is extended to allow local bindings.

For technical background, see chapters 6 and 17 of [1] (or some other
intermediate programming languages textbook).

[1] http://cs.brown.edu/courses/cs173/2012/book/
2021-09-04 17:07:19 -07:00
David Bremner
9b9eb1d8bd lib/parse-sexp: support saved s-expression queries
It turns out there is not really much code in query-fp.cc useful for
supporting the new syntax. The code we could potentially factor out
amounts to calling notmuch_database_get_config; both the key
construction and the parsing of the results are specific to the query
syntax involved.
2021-09-04 17:07:19 -07:00
David Bremner
81b9dbd110 CLI/config support saving s-expression queries
This commit does not enable using saved s-expression queries, only
saving and retrieving them from the config file or the database. Use
in queries will be enabled in a following commit.
2021-09-04 17:07:19 -07:00
David Bremner
6ab2d9b1a2 lib/parse-sexp: handle saved queries
This provides functionality analogous to query: in the Xapian
QueryParser based parser. Perhaps counterintuitively, the saved
queries currently have to be in the original query syntax (i.e. not
s-expressions).
2021-09-04 17:07:19 -07:00
David Bremner
a07ef8abf5 lib/parse-sexp: parse user headers
One subtle aspect is the replacement of _find_prefix with
_notmuch_database_prefix, which understands user headers. Otherwise
the code mainly consists of creating a fake prefix record (since the
user prefixes are not in the prefix table) and error handling.
2021-09-04 17:07:19 -07:00
David Bremner
cc5992a304 lib/parse-sexp: support infix subqueries
This is necessary so that programs can take infix syntax queries from
a user and use the sexp query syntax to construct e.g. a refinement of
that query.
2021-09-04 17:07:19 -07:00
David Bremner
afe85e6578 lib/parse-sexp: expand queries
The code here is just gluing together _notmuch_query_expand with the
existing sexp parser infrastructure.
2021-09-04 17:07:19 -07:00
David Bremner
1870b3ae4b lib/parse-sexp: support regular expressions
At least to the degree that the Xapian QueryParser based parser
also supports them. Support short alias 'rx' as it seems to make more
complex queries nicer to read.
2021-09-04 17:07:19 -07:00
David Bremner
0b98ad5e4e lib/query: generalize exclude handling to s-expression queries
In fact most of the code path is in common, only the caching of terms
in the query needs to be added for s-expression queries.
2021-09-04 17:07:19 -07:00
David Bremner
bafc307190 lib/parse-sexp: handle unprefixed terms.
This is equivalent to adding the same field name "" for multiple
prefixes in the Xapian query parser, but we have to explicitely
construct the resulting query.
2021-09-04 17:07:19 -07:00
David Bremner
0ca4ad2670 lib/parse-sexp: add '*' as syntactic sugar for '(starts-with "")'
Users that insist on using a literal '*' as a tag, can continue to do
so by quoting it when searching.
2021-09-04 17:07:19 -07:00
David Bremner
011d06f4d6 lib/parse-sexp: 'starts-with' wildcard searches
The many tests potentially overkill, but they could catch typos in the
prefixes table. As a simplifying assumption, for now we assume a
single argument to the wildcard operator, as this matches the Xapian
semantics. The name 'starts-with' is chosen to emphasize the supported
case of wildcards in currrent (1.4.x) Xapian.
2021-09-04 17:07:19 -07:00
David Bremner
8322f536f5 lib/parse-sexp: add term prefix backed fields
We use "boolean" to describe fields that should generate terms
literally without stemming or phrase splitting.  This terminology
might not be ideal but it is already enshrined in
notmuch-search-terms(7).
2021-09-04 17:07:19 -07:00
David Bremner
90d9c2ad5c lib/parse-sexp: support phrase queries.
Anything that is quoted or not purely word characters is considered a
phrase.  Phrases are not stemmed, because the stems do not have
positional information in the database. It is less efficient to scan
the term twice, but it avoids a second pass to add prefixes, so maybe
it balances out. In any case, it seems unlikely query parsing is very
often a bottleneck.
2021-09-04 17:07:19 -07:00
David Bremner
200e164dc7 lib/parse-sexp: support subject field
The broken tests are because we do not yet handle phrase searches.
2021-09-04 17:07:19 -07:00
David Bremner
f83cd2a05a lib/parse-sexp: support and, not, and or.
All operations and (Xapian) fields will eventually have an entry in
the prefixes table. The flags field is just a placeholder for now, but
will eventually distinguish between various kinds of prefixes.
2021-09-04 17:07:19 -07:00
David Bremner
a2785c3919 lib/parse-sexp: stem unquoted atoms
This is somewhat less DWIM than the Xapian query parser, but it has
the advantage of simplicity.
2021-09-04 17:07:19 -07:00
David Bremner
be7e83de96 lib/parse-sexp: parse single terms and the empty list.
There is not much of a parser here yet, but it already does some
useful error reporting. Most functionality sketched in the
documentation is not implemented yet; detailed documentation will
follow with the implementation.
2021-09-04 17:07:19 -07:00
David Bremner
e3b1a0a6a5 CLI/search+address: support sexpr queries
Initially support selection of query syntax in two subcommands to
enable testing.
2021-09-04 17:07:19 -07:00
David Bremner
a83ad52da4 configure: optional library sfsexp
The configure part is essentially the same as the other checks using
pkg-config. Since the optional inclusion of this feature changes what
options are available to the user, include it in the "built_with"
pseudo-configuration keys.
2021-09-04 17:07:19 -07:00
David Bremner
d447b694b4 CLI: make variable n_requested_db_uuid file scope.
It turns out that now that we pass an open database into the
subcommands, it is easy to check any requested uuid against the
database at the same time as we process the other shared
arguments. This results in overall less boilerplate code, as well as
making a CLI scope function and variable file scope in notmuch.c.
2021-09-04 17:07:19 -07:00
David Bremner
06b536dae0 test: known broken test for emacs display of message/rfc822 parts
There are at least 3 bugs present.

1) notmuch-show-insert-part-message/rfc822 assumes that message/rfc822
parts will have a ":content" property, but that turns out not to be
the case.

2) something deep in gnus wants gnus-newsgroup-charset, but that is
defined in gnus-sum, which is not loaded by default.

3) If gnus-sum is loaded, then the display of the message/rfc822 part
succeeds, but the buffer gets put into gnus-article-mode, which means
that, inter alia, notmuch text properties and keybindings get wiped.
2021-08-30 16:29:04 -07:00
David Bremner
e1c56f49de test/crypto: test message with rfc822 attachment.
This is intended to help track down a display problem in the emacs
front end
2021-08-30 16:24:59 -07:00
Hannu Hartikainen
717e3dcdc3 lib: consider all instances of Delivered-To header
When using notmuch-reply and guessing the From: address from
Delivered-To headers, I had the wrong address chosen today. This was
because the messages from the notmuch list contain these headers in this
order:

Delivered-To: hannu.hartikainen@gmail.com
...
Delivered-To: hannu@hrtk.in

In my .notmuch-config I have the following configuration:

primary_email=hannu@hrtk.in
other_email=hannu.hartikainen@gmail.com;...

Before this change, notmuch-reply would guess From: @gmail.com because
that is the first Delivered-To header present. After the change, the
primary address is chosen as I would expect.
2021-08-29 18:10:08 -07:00
Hannu Hartikainen
8c0bebedc5 reply: add test for multiple Delivered-To headers
Add a known broken subtest for guessing From: correctly when there are
multiple Delivered-To: headers. The address configured as primary_email
should get picked.
2021-08-29 18:07:59 -07:00
David Bremner
731697d671 CLI: define and use format version 5
This is a bit of a cheat, since the format does not actually
change. On the other hand it is fairly common to do something like
this to shared libary SONAMEs when the ABI changes in some subtle way.
It does rely on the format-version argument being early enough on the
command line to generate a sensible error message.
2021-08-22 07:05:13 -07:00
David Bremner
4f84c01b3a test/path-config: use test_expect_equal_file_nonempty
This is more robust against crashes when the expected output is also
generated by notmuch. In the case where the expected output is
explicit, it seems like overkill.
2021-08-21 14:56:13 -07:00
David Bremner
474a7f8e65 test: add test_expect_equal_file_nonempty
A common bug in tests is that the code used to generate the EXPECTED
file fails, generating no output. When the code generating the OUTPUT
file fails in the same way, the test passes, even though there is a
failure being hidden. Add a new test function that guards against
this.
2021-08-21 14:55:41 -07:00
David Bremner
78045301e7 test: factor out test_diff_file_
A following commit will use this new function in a public test
function.
2021-08-21 14:55:27 -07:00
David Bremner
f2e6f311c7 test/expect_equal_file: whitespace cleanup
No functional change, just reindent to Emacs defaults.
2021-08-21 14:51:04 -07:00
David Bremner
3df2281746 notmuch release 0.32.3-1 for unstable (sid) [dgit]
[dgit distro=debian no-split --quilt=linear]
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEkiyHYXwaY0SiY6fqA0U5G1WqFSEFAmEclwwACgkQA0U5G1Wq
 FSEdBw//cF+LAJy9qyWX8axQcSwo1/ixZkyxSCc0JVaDhhAal5GdVd3E+vDKZp3Q
 KWExBKncTAciB1CooKNurnvFr30s9MX+SrDK0rPCqXtObMRYS96AEDCvul9Z9mBT
 eE6thb66Y2Mrr2dyHiyG/GHrp9wces+sSnB6xM6bqO16c9UsMuqjTIIqlb0A7oP8
 eN/LLPAcVoXIMry3hVQWzv6bJWfqQK9Ho66p1fH+QswgeeKIzh5HR3ZnGdsBhnyN
 n3l9QPYuhN7/85cjWH4AE0/2EQLIF/Ewu86qYHzvOufc7oiNXRGpzQtMNqqur/da
 JIOizaylhLkFP39i/7CHvldCTx0FOCVhbHtFwb1Hm7rPqr8DhOvXyZbDHK/LkWtg
 jBKVFld4nNv/zm1uz6S3E5rDFsxuvpMVuesWKj6Yb051QvfGqVTnwRHP3EFxMyk8
 CVle5a3mMg8bnlN9o/WchB3z3ybijK4HLjGB+FWDZMUvhKXkfZL6GUeo/EQ8byaX
 mDEx5u56euqgNQS1PnIF/nTaOCbu75IqYAUSqwVXBcICi7LfYKDwGezBiEhoRiKt
 RQoulN9FNS+HuL+85LN4TdZyJOvD1Rc9xdT9TwuLr47nIc/KdAWUxyYSEo4PMiZ+
 NNiCatI5kPJrp0q1A5xDRkpFavLu5Mtu9ore+Cf1nA18iEFNQwc=
 =cECE
 -----END PGP SIGNATURE-----

Merge tag 'debian/0.32.3-1'

notmuch release 0.32.3-1 for unstable (sid) [dgit]

[dgit distro=debian no-split --quilt=linear]
2021-08-18 21:46:42 -07:00
David Bremner
686230712f CLI/config: restore "notmuch config get built_with.*"
We need to special case the config section "built_with" because it is
not (currently) handled by the library. This seems consist with the
other sub-sub-commands 'list' and 'set'.
2021-08-17 17:09:56 -07:00
David Bremner
202dde04e1 test/config: add tests for built_with
The "get" test is known broken because this functionality was dropped
during the 0.32 config rewrite.
2021-08-17 17:09:38 -07:00
David Bremner
d930011690 lib/open: look in MAILDIR for database, as documented.
This fixes the bug id:87bl9lx864.fsf@kisara.moe
2021-08-17 17:09:21 -07:00
David Bremner
8376e81441 test: add known broken tests for finding database via MAILDIR
This highlights a bug reported by several users, including
Mohsin Kaleem [1].

The inconsistent use of test_begin_subtest_known_broken is because
some of these tests pass even though the database cannot be
located. This problem is left for a future commit.

[1]: id:87bl9lx864.fsf@kisara.moe
2021-08-17 17:07:53 -07:00
David Bremner
3fc2e5f23b test: revert to mkdir -p to make results directory.
As suggested by id:m21r7al3mt.fsf@guru.guru-group.fi
2021-08-03 20:35:13 -03:00
David Bremner
c966fa6439 test: use --minversion to detect GNU Parallel.
Based on a suggestion of Ole Tange [1].

[1]: id:CA+4vN7x6jp4HCiEybZ=5g+2X6Pa7etBFX3Bbd=UYty37gR6wEQ@mail.gmail.com
2021-08-03 20:29:47 -03:00
jao
13d49b8e16 CLI/show: tests for the new --sort option
New unit tests for notmuch show --sort, covering the basic use cases.
2021-07-03 20:27:39 -03:00
Michael J Gruber
1768b0c416 test: deduplicate T590
Test numbers are a concise way to communicate about tests and to remeber
them. Currently, there is one pait of duplicates:

T590-libconfig.sh
T590-thread-breakage.sh

Renumber the latter one to 592 since this keeps the alphabetic order and
leaves room in between.

Signed-off-by: Michael J Gruber <git@grubix.eu>
2021-06-29 13:43:24 -03:00
David Bremner
37f84d6d21 emacs/tree use notmuch-show-single-message
This is more efficient than notmuch-show-only-matching-messages, since
we do not parse the potentially large thread structure to find a
single message.

This is only a partial fix for notmuch-tree view, because displaying
the thread structure in the tree-mode window still crashes on long
threads. It is however enough to make unthreaded view handle long
threads.
2021-06-27 14:41:23 -03:00
David Bremner
14326e6cb7 test: start test file for emacs unthreaded view.
Initial focus is on behaviour with large threads.

The second test replicates a bug reported by Alan Schmitt in
id:87lf7sojbq.fsf@m4x.org.
2021-06-27 14:29:08 -03:00
Felipe Contreras
814abafc3e ruby: add keyword arguments to db.query
That way we don't need pass them to the query object ourselves.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-06-27 14:13:03 -03:00
David Bremner
e2a3e5fa51 lib: autocommit after some number of completed transactions
This change addresses two known issues with large sets of changes to
the database.  The first is that as reported by Steven Allen [1],
notmuch commits are not "flushed" when they complete, which means that
if there is an open transaction when the database closes (or e.g. the
program crashes) then all changes since the last commit will be
discarded (nothing is irrecoverably lost for "notmuch new", as the
indexing process just restarts next time it is run).  This does not
really "fix" the issue reported in [1]; that seems rather difficult
given how transactions work in Xapian. On the other hand, with the
default settings, this should mean one only loses less than a minutes
worth of work.  The second issue is the occasionally reported "storm"
of disk writes when notmuch finishes. I don't yet have a test for
this, but I think committing as we go should reduce the amount of work
when finalizing the database.

[1]: id:20151025210215.GA3754@stebalien.com
2021-06-27 14:03:00 -03:00
David Bremner
8aabddb043 test: add known broken test for closing with open transaction
The expected output may need adjusting, but what is clear is that
saving none of the changes is not desirable.
2021-06-27 14:01:26 -03:00
David Bremner
2f608d2a94 lib/config: add NOTMUCH_CONFIG_AUTOCOMMIT
This will be used to control how often atomic transactions are
committed.
2021-06-27 13:59:42 -03:00
Daniel Kahn Gillmor
8c29a5da09 cli/show: produce "email" element in sigstatus
When the certificate that signs a message is known to be valid, GMime
is capable of reporting on the e-mail address embedded in the
certificate.

We pass this information along to the caller of "notmuch show", as
often only the e-mail address of the certificate has actually been
checked/verified.

Furthermore, signature verification should probably at some point
compare the e-mail address of the caller against the sender address of
the message itself.  Having to parse what gmime thinks is a "userid"
to extract an e-mail address seems clunky and unnecessary if gmime
already thinks it knows what the e-mail address is.

See id:878s41ax6t.fsf@fifthhorseman.net for more motivation and discussion.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2021-06-26 13:07:47 -03:00
David Bremner
7406abf5e0 test/libconfig: make indexing more consistent
The other tests have converted to two digit decimal, the commit does
the same to the remaining test that was printing out indices in hex.
2021-06-21 06:57:17 -03:00
David Bremner
8dbd5deb8d Merge branch 'release' 2021-06-20 17:15:20 -03:00
David Bremner
a19d2b0d2b test: check openssl prequisite for add_gpgsm_home
This is a fix for the test failures reported by Dan Čermák [1].

It is more robust to check for the prerequisite inside the function
that uses it, rather than in every test file that calls the function.

[1]: id:87k0n4fqgm.fsf@tethera.net
2021-06-08 17:15:46 -03:00
Tomi Ollila
6f0f83660e test: aggregate-results updates
notmuch-test will now call aggregate-results.sh with file list
that it compiles based on the test ran, and aggregate-results
will report failure is any of the test files are missing.

With this notmuch-test no longer has to exit in non-parallel
run if some test fail to write its report file -- so it works
as parallel tests in this sense.

Changed test_done() in test-lib.sh write report file in one write(2),
so there is (even) less chance it being partially written. Also,
now it writes 'total' last and aggregate-results.sh expects this
line to exist in all report files for reporting to be successful.

Added 'set -eu' to notmuch-test and modified code to work with
these settings. That makes it harder to get mistakes slipped
into committed code.
2021-06-07 20:16:33 -03:00
Jani Nikula
87234a8ef3 emacs: update default mailing list archives for stashing links
Gmane web interface is long gone, remove it. Make MARC the new
default. Update LKML to Lore, where it already redirects anyway. Also
add Notmuch web archive.
2021-06-05 15:39:53 -03:00
David Bremner
2c85cc29bd test: add known broken test for indexing RFC822 group names
Austin Clements diagnosed this indexing problem in [1].

[1]: id:20130711215207.GR2214@mit.edu
2021-06-05 15:31:03 -03:00
Tomi Ollila
572af27950 test: source $NOTMUCH_SRCDIR/test/test-lib-emacs.sh
Sourcing test-lib.sh will cd to TMP_DIRECTORY, so
relative path in $0 will not work in previous version
 . $(dirname "$0")/test-lib-emacs.sh

Now individual test scripts -- e.g. ./test/T310-emacs.sh
will work.
2021-06-03 09:29:27 -03:00
Felipe Contreras
85ae2bcf56 ruby: use notmuch_exclude_t enum
It exists since 2013, let's allow it to be used in Ruby.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-23 09:05:33 -03:00
David Bremner
f44c83c083 test: add regression test for large number of deletions
Prior to 9ad19e4454 there was an unhandled Xapian exception when
reindexing after a large number of deletes.  This test was used for
bisection, and will subsequently serve as a regression test.
2021-05-23 08:31:36 -03:00
Tomi Ollila
f2533b9e73 test: say_color() in one write(2)
say_color() used to call (builtin) printf (and tput(1) to stdout)
several times, which caused attempts to write messages with color
to have partial content (e.g. escape sequences) often intermixed
with other tests when parallel tests were run.

Now, with all output collected, then written out using one
printf, all strings with color print out correctly
((at least short) write(2)'s appear to write out "atomically").

While at it, used only one tput(1) execution to determine whether
color output works, and made bold/colors/sgr0 to tput(1) their
values once per test.
2021-05-23 08:05:15 -03:00
David Bremner
69c2c930ec test: quiet some extra debugging output
This output does not cause test failures, but may make it harder to
interpret the output.
2021-05-23 08:04:07 -03:00
David Bremner
119a41b9ed test: ignore debugging messages
Previously building with "-DDEBUG" broke the test suite in several places.
2021-05-23 08:03:02 -03:00
Tomi Ollila
8675be1d30 test: replace notmuch_passwd_sanitize() with _libconfig_sanitize()
notmuch_passwd_sanitize() in test-lib.sh is too generic, it cannot
work in many cases...

The more specific version _libconfig_sanitize() replaces it in
T590-libconfig.sh and the code that uses it is modified to output
the keys (ascending numbers printed in hex) so the sanitizer knows
what to sanitize in which lines...

"@" + fqdn -> "@FQDN" replacement is used as fqdn could
-- in theory -- be substring of 'USERNAME'.

'user -> 'USER_FULL_NAME replacement to work in cases where user
is empty -- as only first ' is replaced that works as expected.

In addition to ".(none)" now also ".localdomain" is filtered from
USERNAME@FQDN.

/dev/fd/{n} is not defined in posix, but it is portable enough
(if it weren't it is easy to fix -- now code looks clearer).
2021-05-22 11:51:29 -03:00
David Bremner
702635d5f6 Merge branch 'release' 2021-05-22 09:34:55 -03:00
David Bremner
3f4de98e7c lib/n_d_index_file: re-use thread-id of existing message
This prevents the message document getting multiple thread-id terms
when there are multiple files with the same message-id.

This change shifts some thread ids, requiring adjustments to other tests.
2021-05-22 09:08:02 -03:00
David Bremner
349fc29803 test: add known broken test for duplicate thread-id terms
According to my bijection, this bug has been present since commit
411675a6ce in 2017. It is not completely clear what harm it causes in
regulary use, but it (at least) makes notmuch crash when compiled with
-DDEBUG_DATABASE_SANITY.
2021-05-22 09:02:58 -03:00
Felipe Contreras
92454bc093 test: split emacs functionality to its own file
This way it's easier to identify the tests that do require emacs stuff.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-17 07:29:04 -03:00
Felipe Contreras
bba4788aec test: emacs: fix a couple of shellcheck complaints
In test-lib-emacs.sh line 20:
    test_require_external_prereq ${TEST_EMACS} || ret=1
                                 ^-----------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
    test_require_external_prereq "${TEST_EMACS}" || ret=1

In test-lib-emacs.sh line 21:
    test_require_external_prereq ${TEST_EMACSCLIENT} || ret=1
                                 ^-----------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
    test_require_external_prereq "${TEST_EMACSCLIENT}" || ret=1

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-17 07:28:44 -03:00
Felipe Contreras
29ba523275 test: emacs: check for configured emacs
Commit d59d9c81 (test: Make the emacsclient binary user-configurable,
2012-11-27) modified the prereq check for the configured emacsclient,
but we probably want to do the same for emacs itself.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-17 07:28:33 -03:00
Felipe Contreras
cd9ec20a49 test: emacs: simplify missing dependencies check
No functional changes.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-17 07:28:23 -03:00
Felipe Contreras
4561f694e9 test: more style fixes
In order to fit the git coding style.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-17 07:28:01 -03:00
Felipe Contreras
645bea13cb test: trivial style cleanups
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-17 07:27:52 -03:00
David Bremner
c84ccb70f3 Merge branch 'release' 2021-05-15 09:10:58 -03:00
David Bremner
b3258244c8 lib/open: restore default database path of $HOME/mail
Although this default worked for "notmuch config get", it didn't work
most other places. Restore the previous functionality, with the
wrinkle that XDG locations will shadow $HOME/mail if they exist.

This fixes a bug reported by Jack Kamm in id:87eeefdc8b.fsf@gmail.com
2021-05-15 08:40:05 -03:00
Kyle Meyer
319dcfb50e emacs: restore tag-changes and query bindings for tag hooks
notmuch-before-tag-hook and notmuch-after-tag-hook are supposed to
have access to two dynamic variables, tag-changes and query, but these
were lost with the switch to lexical binding in fc4cda07 (emacs: use
lexical-bindings in all libraries, 2021-01-13).

Add a variant of Emacs's dlet (not available until Emacs 28) and use
it in notmuch-tag to expose tag-changes and query to the hooks.
2021-05-15 08:34:28 -03:00
David Bremner
a663783343 test: add known broken tests for notuch-{before,after}-tag-hook
These tests illustrate the bug reported in id:87v97ytd2s.fsf@fastmail.fm
2021-05-15 08:31:00 -03:00
David Bremner
25da4a1340 Merge branch 'release' 2021-05-12 08:42:28 -03:00
David Bremner
393c92b042 lib/notmuch_database_reopen: reload some database metadata
In some uses of reopen, new documents and threads maybe have been
added, and e.g. compaction may have changed the uuid.
2021-05-12 08:40:04 -03:00
Michael J Gruber
0729c4e81f test: change database from within pre-new hook
Due to the change in the config system, notmuch keeps a notmuch database
open when it would not do so before. Consequently, it can miss changes
to the database which are done from a hook (while notmuch holds the
databse in read only mode). When notmuch itself writes to the database
after that it uses wrong assumptions about the last used doc id etc.

Demonstrate this by triggering an assertion. (This new test succeeds
with notmuch 0.31.4.)

Signed-off-by: Michael J Gruber <git@grubix.eu>
Amended-by: db. Check for both messages
2021-05-12 08:34:34 -03:00
David Bremner
b4a4ed0df9 Merge branch 'release' 2021-05-10 11:36:56 -03:00
David Bremner
8bf3311084 test: test explicit configuration of backup directory
Including the relative path that was broken until a recent commit.
2021-05-10 11:13:07 -03:00
David Bremner
891b950219 test: test relative paths for database.hook_dir 2021-05-10 11:13:07 -03:00
David Bremner
1040e7aa07 lib/config: expand relative paths when reading from database
This makes the treatment of relative paths consistent between the
database and config files.
2021-05-10 11:12:58 -03:00
David Bremner
322a492c77 test: add known broken test for relative setting of mail_root
The behaviour should not change depending on where the configuration
is stored.
2021-05-10 11:11:42 -03:00
David Bremner
31098c4ae4 lib/config: canonicalize paths relative to $HOME.
Prior to 0.32, notmuch had the (undocumented) behaviour that it
expanded a relative value of database.path with respect to $HOME. In
0.32 this was special cased for database.path but broken for
database.mail_root, which causes problems for at least notmuch-new
when database.path is set to a relative path.

The change in T030-config.sh reflects a user visible, but hopefully
harmless behaviour change; the expanded form of the paths will now be
printed by notmuch config.
2021-05-10 11:08:18 -03:00
David Bremner
ef0b126619 test: add known broken test for relative database path in new
This test highlights a bug introduced in 0.32. The new split between
path and mail_root does not properly canonicalize relative paths in
the latter.
2021-05-10 11:06:40 -03:00
Felipe Contreras
ac64de450a test: ruby: simplify output comparison
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-05 17:00:37 -03:00
Felipe Contreras
faf5511c3e test: ruby: don't use instance variables
Local variables are perfectly fine.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-05 17:00:16 -03:00
Felipe Contreras
c4b23cf29a test: ruby: use much more standard Ruby idioms
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-05 16:40:15 -03:00
Felipe Contreras
63413a5563 test: ruby: use much more standard puts
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-02 08:12:15 -03:00
Felipe Contreras
2dbc5fdf5a test: ruby: simplify test_ruby()
We always do test_expect_equal_file, so do it in test_ruby() directly.

The only subtest where we don't (get non-existent file) can be easily
modified.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-02 08:12:05 -03:00
Felipe Contreras
9f6bc01824 test: ruby: simplify MAIL_DIR initialization
There's no need to complicate the script passing the MAIL_DIR
environment variable.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-02 08:11:56 -03:00
Felipe Contreras
c1b99d6f94 test: ruby: simplify MAIL_DIR check
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-02 08:11:42 -03:00
Felipe Contreras
a75a9a5aed test: ruby: refactor test_ruby()
There's no point in repeating the same initialization in all the tests.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-02 08:11:17 -03:00
Felipe Contreras
8cbd8e71d0 test: move test_ruby() inside the only client
Not much point in polluting the main library, and also will be useful to
modify it in tandem with the tests.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-02 08:11:04 -03:00
Felipe Contreras
fe9616aef1 test: add external prereqs to many emacs tests
The tests fail otherwise.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-01 21:15:27 -03:00
Felipe Contreras
049a652da7 test: add prereqs check in test_emacs_expect_t
test_emacs may update the external prereqs, in which case we want to
skip the test rather than fail.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-01 21:15:10 -03:00
Felipe Contreras
7579f3ac6b test: fix wrong SKIP messages
When the external prereqs are updated inside the body of the command
(e.g. test_emacs) the message in test_report_skip_ is wrong: it outputs
the body of the command instead of the subtest name.

We need to pass the same argument we pass to test_skip.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-01 21:14:55 -03:00
Felipe Contreras
0b48e65526 test: use correct fqdn in passwd_sanitize()
My fqdn is 'natae.localdomain', however, socket.getfqdn() returns
'localhost'.

To fetch the true fqdn we need socket.getaddrinfo().

For more information see: https://stackoverflow.com/a/11580042/10474

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-01 21:14:37 -03:00
Felipe Contreras
2add9ce722 test: unset NAME environment variable
Otherwise the output from the tests would be different.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-01 21:13:41 -03:00
Felipe Contreras
ff5f141bf2 test: fix passwd_sanitize()
If any of the variables is empty the output is completely messed up,
because replace("", "FOO") puts "FOO" before every single character.

I don't have my full name configured, and this is what I get:

  USER_FULL_NAME=USER_FULL_NAME=USER_FULL_NAME USER_FULL_NAMEsUSER_FULL_NAMEtUSER_FULL_NAMEdUSER_FULL_NAMEoUSER_FULL_NAMEuUSER_FULL_NAMEtUSER_FULL_NAME USER_FULL_NAME=USER_FULL_NAME=USER_FULL_NAME

Let's check for empty strings before doing any replace.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-01 21:13:23 -03:00
David Bremner
00487fa901 test: hide message from moreutils parallel.
The argument --version confuses moreutils parallel, but this is OK,
because its confusing does not include printing "GNU"
2021-04-26 09:00:11 -03:00
Tobias Backer Dirks
f2b5ad28e2 test: check for GNU parallel with --version options
The lastest versions of GNU parallel no longer make mention of GNU
within their help output. This causes the test script to mistakenly use
the moreutils parallel execution. In order to fix this, while
maintaining compatibility with previous versions of GNU parallel,
--version should be used.

Signed-off-by: Tobias Backer Dirks <omgitsaheadcrab@gmail.com>
2021-04-24 20:06:41 -03:00
David Bremner
27a2c2c80d test: replace notmuch_passwd_sanitize with python version
Apparently the -f option to hostname is not portable, and in fact it
does not seem to always behave reasonably in e.g. a chroot.

Python code originally due to Tomi [1], modified by yours truly.

[1]: id:m2lf9fbkug.fsf@guru.guru-group.fi
2021-04-24 12:11:22 -03:00
Đoàn Trần Công Danh
828a004921 test-lib: unset XDG_CONFIG_HOME
lib/open.cc:_load_key_file will only open xdg-config files in
$XDG_CONFIG_HOME if it's defined, $HOME/.config will be considered if
and only if XDG_CONFIG_HOME not defined.

Let's unset said variable before running the test.
2021-04-18 10:00:10 -03:00
David Bremner
5248f55d5f test: put shim at end of LD_PRELOAD path
Certain tools like the address-sanitizer fail if they are not the
first LD_PRELOADed library. It does not seem to matter for our shims,
as long as they are loaded before libnotmuch.
2021-04-16 09:06:39 -03:00
David Bremner
ec4b7efbe4 CLI/config: remove calls to notmuch_config_open from top level
This will allow simplifying the subcommand interface.

Change the internal API to notmuch_config_open to not tie it to the
implementation of subcommands in notmuch.c.

It also fixes a previously broken test, since notmuch_config_open does
not understand the notion of the empty string as a config file name.
2021-04-06 21:32:36 -03:00
David Bremner
77d4b26d3e CLI/config: support set/get with split configuration
There are two small code changes. The main one is to retrieve the
possibly updated config file name found during the database opening
call. The second change is to allow empty config file names, as
a (currently broken) way of specifying that configuration should only
be taken from the database.
2021-03-27 09:26:14 -03:00
David Bremner
217f819608 CLI+lib: detect missing database in split configurations.
Eventually we want to do all opening of databases in the top
level (main function). This means that detection of missing databases
needs to move out of subcommands. It also requires updating the
library to use the new NO_DATABASE status code.
2021-03-27 09:26:14 -03:00
David Bremner
dd9112e7d8 CLI/config: default to storing all config in external files
Previously the fact that some configuration options were only stored
in the database (and thus editing the config file had no effect) was a
source of user confusion. This was fixed with the series ending at
d9af0af164.

On the other hand, the underlying partition of config options into
those stored by default in the database and those stored in the config
file remained. This is also confusing, since now some invocations of
"notmuch config set" modify the config file, and others silently
modify the database instead.

With this commit, it is up to the user to decide which configuration
to modify. A new "--database" option is provided for notmuch config to
enable modifying the configuration information in the database;
otherwise the default is to update an external config file.
2021-03-27 09:26:14 -03:00
David Bremner
f055e94562 CLI/config: use notmuch_database_reopen
This allows `notmuch config` to use the common configuration
information carried by the notmuch_database_t opened at the top level.
2021-03-27 09:26:14 -03:00
David Bremner
b76da87d29 CLI/config: migrate notmuch_config_open to new config
notmuch_config_open will be preserved in the medium term for use by
the commands that are manipulating the config file directly (config
and setup)
2021-03-27 09:26:14 -03:00
David Bremner
3787fe6c84 CLI/config: switch "notmuch config list" to merged config
Use the database opened at the top level rather than opening another
notmuch_database_t.

Test output changes because keys are now listed in alphabetical order,
and because a missing database is no longer an error.
2021-03-27 09:26:14 -03:00
David Bremner
ee0e1cfb17 test/setup: check file output instead of notmuch config list
This allows verification of comments added to new / updated config
file.
2021-03-27 09:26:14 -03:00
David Bremner
ecb91d056c CLI/config: use merged config for "config get"
This commit starts the conversion of notmuch-config.c
functionality (as opposed to just interface) to the new config
framework.

The change to T030-config is because of the move of the
canonicalization database paths from the notmuch_config_t accessor to
the internal function _choose_database_path.
2021-03-27 09:26:14 -03:00
David Bremner
2e9ee47072 lib/config: set default for primary user email
This is mainly copying code from the CLI into the lib. The CLI copy
will be deleted in a later commit.
2021-03-27 09:26:14 -03:00
David Bremner
e81dc2072f lib/config: set defaults for user full name
This just copies code from from the CLI into the library. New test
infrastructure is needed because apparently we have never tested this
code path.
2021-03-27 09:26:14 -03:00
David Bremner
863b243185 lib/config: add config_pairs iterators
The layer of shims here seems a bit wasteful compared to just calling
the corresponding string map functions directly, but it allows control
over the API (calling with notmuch_database_t *) and flexibility for
future changes.
2021-03-27 09:26:14 -03:00
David Bremner
0c6db22930 lib/config: add notmuch_config_get_values_string
This is to support the less common (at least in the notmuch codebase)
case of accessing a ;-delimited list config value with an arbitrary
string key.
2021-03-27 09:26:14 -03:00
David Bremner
2fc40e24de lib: provide notmuch_database_load_config
This is mainly targetted at notmuch-config.c, or similar applications
that don't necessarily need both a configuration file and a database
to exist.
2021-03-27 09:26:14 -03:00
David Bremner
79a4b2e9e7 test: convert random-corpus to use n_d_open_with_config
Remove one more usage of notmuch_config_get_database_path
2021-03-27 09:26:14 -03:00
David Bremner
4e209ca99a CLI/new: use configuration variable for backup directory
The stat is essentially replaced by the mkdir for error detection
purposes.  This changes the default location for backups to make
things tidier, even in non-split configurations. Hopefully there is
not too many user scripts relying on the previous location.

Because the default location may not exist, replace the use of stat
for error detection with a call to mkdir.
2021-03-20 07:53:02 -03:00
David Bremner
a7873df331 lib/config: add configuration variable for backup directory
Like the hook directory, we primarily need a way to communicate this
directory between various components, but we may as well let the user
configure it.

Most of the diff is generalizing choose_hook_dir to work for both
backup and hook directories.
2021-03-20 07:51:36 -03:00
David Bremner
e3a4abc513 lib/open: fix hook directory calculation in split configuration
Choose sibling directory of xapian database, as .notmuch may not
exist.

libgen.h is already used in debugger.c, so it is not a new dependency
/ potential portability problem.
2021-03-20 07:50:13 -03:00
David Bremner
b30a59157d lib/compact: enable split config
This promotes _choose_xapian_path from static to extern linkage in
order to share between open.cc and database.cc.
2021-03-20 07:50:06 -03:00
David Bremner
74c3cc03c2 CLI/insert: support split database and mail root
The new test is in T055-path-config because it uses the helper
function split_config, and because it seems easier to put the
database path related tests in one place.
2021-03-20 07:44:08 -03:00
David Bremner
c82554193d lib/open: support XDG_DATA_HOME as a fallback database location.
This changes some error reporting, either intentionally by reporting
the highest level missing directory, or by side effect from looking in
XDG locations when given null database location.
2021-03-20 07:43:09 -03:00
David Bremner
2c879667b3 CLI/new: support split database and mail location
This adds new state variable for the mail root, and uses it most
places db_path was used. The notable exception is dumps from
backups. The latter will be dealt with properly in a future commit.
2021-03-20 07:42:06 -03:00
David Bremner
5ec6fd4dcf lib/open: check for split configuration when creating database.
The main functionality will be tested when notmuch-new is converted to
support split configuration. Here only the somewhat odd case of split
mail root which is actually symlinked to the database path is tested.
2021-03-20 07:41:04 -03:00
David Bremner
e823d05ae6 lib: support splitting mail from database location.
Introduce a new configuration value for the mail root, and use it to
locate mail messages in preference to the database.path (which
previously implied the mail messages were also in this location.

Initially only a subset of the CLI is tested in a split
configuration. Further changes will be needed for the remainder of the
CLI to work in split configurations.
2021-03-20 07:39:12 -03:00
David Bremner
986056bdbc lib/open: Use check for existing database by trial opening
This is a bit heavyweight for now, but it will make more sense when we
check multiple locations for the Xapian database.
2021-03-20 07:38:15 -03:00
David Bremner
cba540d6f5 lib/open: reuse directory checks from n_d_c_with_config
Make checks more uniform between creating new databases and opening
existing ones.
2021-03-20 07:24:59 -03:00
David Bremner
793d4305d3 lib/open: support NOTMUCH_DATABASE environment variable
The additional code is trivial, but making sure we get the priority of
various options correct takes a few tests.
2021-03-19 22:06:44 -03:00
David Bremner
6e6c319c26 CLI/show: complete conversion to new configuration framework.
In order to open the database in main() for this command, we may need
to re-open it in the (possibly less common) case where crypto options
require write access.
2021-03-19 22:06:19 -03:00
David Bremner
a9f74aeeb9 CLI/new: drop the write lock to run the pre-new hook.
This fixes a bug reported in [1]. In principle it could be possible
avoid one of these reopens, but it complicates the logic in main with
respect to creating new databases.

[1]: id:9C1993DF-84BD-4199-A9C8-BADA98498812@bubblegen.co.uk
2021-03-18 22:58:02 -03:00
David Bremner
7d286f0272 test: Add tests for write access to database from hooks.
Recent changes to configuration handling meant the pre-new hook was
run while the database was open read only, limiting what could be done
in the hook. Add some known broken tests for this problem, as well as
a regression test for write access from the post-new hook.
2021-03-18 22:54:13 -03:00
David Bremner
13efbd0e1c lib: support reopening databases for write access.
In the future Xapian will apparently support this more conveniently
for the cases other than READ_ONLY => READ_ONLY

Conceptually this function seems to fit better in lib/open.cc;
database.cc is still large enough that moving the function makes
sense.
2021-03-18 08:04:06 -03:00
David Bremner
9a5406cc94 test/setup: add check for config exists, but no database.
This code path is not currently tested, and will need updating if the
location of the xapian database changes.
2021-03-18 08:03:19 -03:00
David Bremner
dcd4df2cc0 test: add known broken test for long directory bug
In [1] Gregor Zattler explained the results of his hard work
tracking down a bug in notmuch with long directories. This test
duplicates the bug.

[1]: id:20210317194728.GB5561@no.workgroup
2021-03-18 07:57:38 -03:00
uncrustify
df4c66f85d test: run uncrustify
This is the result of running:

    $ uncrustify --replace --config ../devel/uncrustify.cfg *.cc *.c *.h

in the test directory.
2021-03-13 08:45:34 -04:00
David Bremner
97fadd0645 test: clean up some extra whitespace.
The extra space is mainly just untidy.
2021-03-12 07:19:14 -04:00
Tomi Ollila
d12d91f828 test: T020-compact.sh: fix work directory location
Fix use of $TEST_DIRECTORY ($NOTMUCH_BUILDDIR/test/) with use of
$TMP_DIRECTORY ($NOTMUCH_BUILDDIR/test/tmp.T020-compact/ in case
of T020-compact.sh) as root directory where to write test files
and directories.
2021-03-12 07:16:01 -04:00
David Bremner
4c79a2dabe notmuch 0.31.4 release
-----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEkiyHYXwaY0SiY6fqA0U5G1WqFSEFAmAuVjQACgkQA0U5G1Wq
 FSG6RA//bDdAtsG7QlywGONVX1FOSHxAgppDVRvSDXuluPmGgWvc2T80awbonfT5
 AHy9co41L9484QbOzd29d9Ttu5O39JdQbo+DCAJ6y2vm1M1dLCZw3HXaZGfxoUZo
 9L9Agxg/rYIEso7dAehOs3rGsia46aj2MDCVCUuA4DkVFhkbQ2QidIH+l3VPdYqm
 +1XmuJwyftO7hMifFd9W9ifO9wrBcK9WtJ23feufMhqfBsS0ItLysOhZQx+QdxSj
 7GuA6qX6V7XlIWpdohIOMKmT9tGHMDUo6Qk5m8aSc2XmKAybAXRc+qr0Kg8EpvBF
 1d9SqNptcXQL6rORJxluXR/aCuCb7m8YDgxFVSrFcp/M/twbpDC3WzjmAV+RZpDP
 GBKDH06IGaOMOj9GaYYWCe9loGROzOIT4y04Ckukit0AWpmdbaKnkAkPdrxrMWEu
 a5v5KkkMYT42q0PFU5bDjwvq+8Afmzt7oMO72zXz0mfuFNtFzZCUN/AG9LtHoRfk
 Q2ks/xBppmtCxWRAHVWc7f+Gk1OzI2PLVWnXIe9gxGimSpgT2QPUvZcxLHT/XWD5
 XRrrxtkPjKB1v6tWOFEMPI6WxE1iBzoC9AdG2h48ZP5drlLObcAxhAg5AvHJGJSr
 8D4iSa9cKjHD6qxDSXChzF6BF4aDNhF2Flq9Cs9tZb8nyt+Ix18=
 =mAE/
 -----END PGP SIGNATURE-----

Merge tag '0.31.4'

notmuch 0.31.4 release
2021-02-18 08:47:53 -04:00
David Bremner
a37d5f5976 test: Fix race condition in T568-lib-thread.sh
The assignment of thread-ids is (apparently) non-deterministic in a
way that mostly seems to show up on multicore machines. In my tests
the number is different from that previously assumed by this test
about 15% of the time on a 50 thread (25 core) Xeon.

Since message id's are fixed, use a message known to be in the thread
of interest to pick out the correct thread-id.
2021-02-18 07:15:47 -04:00
Daniel Kahn Gillmor
7061e41cd0 python: convert shebangs to python3
This is the last bit of "python" left in the notmuch codebase.

https://www.python.org/dev/peps/pep-0394/#recommendation encourages
"third-party distributors" to use more-specific shebang lines.  I'm
not certain that the notmuch project itself is a "third-party
contributor" but I think this is a safe way to encourage people to use
python3 when they're developing notmuch.

We already have python3 explicitly elsewhere in the codebase for
developers (in nmbug).

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2021-02-15 16:32:31 -04:00
David Bremner
f61d88c6f4 CLI: use configured hook directory
This enables support for hooks outside the database directory.
It relies strongly on configuration information being usable between
closing the database and destroying it.
2021-02-06 19:57:55 -04:00
David Bremner
0345bc57a0 lib/open: set HOOK_DIR on open
This is a simple two step path search.  Most error checking is
deferred until running the hooks.
2021-02-06 19:56:13 -04:00
David Bremner
4922416ccc lib/config: add HOOK_DIR
The hook directory configuration needs to be kept in synch with the
other configuration information, so add scaffolding to support this at
database opening time.
2021-02-06 19:54:09 -04:00
David Bremner
0d3bef312d cli/new: convert to new config framework
In addition to the same type of changes as converting other
subcommands, add the possibility of creating a database at the top
level. It would probably make sense to use this for insert as well.
2021-02-06 19:52:11 -04:00
David Bremner
ac67cd84ee lib: introduce notmuch_database_create_with_config
This takes a config path parameter, and can use that to decide the
new database location.
2021-02-06 19:48:34 -04:00
David Bremner
b860be6a76 lib/config: add NOTMUCH_CONFIG_NEW_IGNORE
This will be needed by (at least) the conversion of notmuch-new.c to
the new config framework
2021-02-06 19:39:33 -04:00
David Bremner
f118ef3c3d cli/compact: convert to new configuration framework
Switch to the newly created API function notmuch_database_compact_db,
which takes the database opened in main().
2021-02-06 19:36:06 -04:00
David Bremner
c56dcea7e2 cli/tag: convert to new config framework.
In addition to changing configuration access, change talloc context
for allocation.
2021-02-06 19:30:20 -04:00
David Bremner
6c28039ebb CLI/show: mostly switch show to new config framework
This will need some cleanup when the transition completes, and we stop
passing notmuch_config_t structs to the subcommands.

Unlike the general case, we open the database in the subcommand, since
we don't know whether it should be opened read/write until we parse
the command line arguments.

Add a test to make sure passing config file on the command line is not
broken by these or future config related changes.
2021-02-06 19:27:54 -04:00
David Bremner
d613d10ddd CLI/{search,address}: convert to new configuration framework
Since we are already passing a search context around as a kind of
parameter block, add a new talloc context to that to replace relying
on 'config'.

Convert notmuch-search and notmuch-address at the same time, because
they share some code.

Add a test to make sure we don't break passing configuration as a
command line argument.
2021-02-06 19:23:28 -04:00
David Bremner
8588719eb1 CLI/reply: convert to new config framework
This is messier than some of the other conversions because the
extensive use of 'config' as a talloc context.
2021-02-06 19:21:23 -04:00
David Bremner
0ab0b48be2 cli/reindex: convert new config framework
The only non-trivial part is switching the talloc context for
query_string_from args from 'config' to 'notmuch'.
2021-02-06 19:19:18 -04:00
David Bremner
f994f0e7df CLI/insert: convert to new config framework.
The new talloc context is needed to run the hook at the very end of
the function. That in turn is needed so that this process gives up the
write lock on the database.
2021-02-06 19:17:34 -04:00
David Bremner
66adcd4f53 CLI/restore: convert to new config framework
Switch one configuration check to new n_c_get_bool function, and
switch use of config as talloc context to notmuch.
2021-02-06 19:15:53 -04:00
David Bremner
53f27aaf73 cli/dump: convert to new config framework
This conversion is trivial because the only configuration information
accessed by dump is that stored in the database (in order to dump
it). We do need to be careful to keep the write lock on the database
to ensure dump consistency.
2021-02-06 19:12:34 -04:00
David Bremner
acc6331baa CLI/count: switch to new configuration framework
The main effort is changing from the old argv style config list
iterators to the new more opaque ones provided by the library (and
backed by the database+file config cache).
2021-02-06 19:11:06 -04:00
David Bremner
d071828bd5 lib/config: make values iterators restartable
This is relatively cheap, and makes it easier to transform existing
code which uses arrays of pointers to store configuration lists.
2021-02-06 19:09:39 -04:00
David Bremner
fd6f8e6c30 lib/config: add config values iterator
This is intended to avoid duplicating the string splitting and
traversal code for all clients of the config API.
2021-02-06 19:08:12 -04:00
David Bremner
06a64cf0aa lib/open: load default values for known configuration keys.
This emulates the behaviour of notmuch_config_open defined in the CLI,
in that it fills in default values if they are not otherwise defined.
2021-02-06 19:03:36 -04:00
David Bremner
d6bd87a712 lib/config: add notmuch_config_key_{get,set}
By using an enum we can have better error detection than copy pasting
key strings around.

The question of what layer this belongs in is a bit
tricky. Historically most of the keys are defined by the CLI. On the
other hand features like excludes are supported in the
library/bindings, and it makes sense to configure them from the
library as well.

The somewhat long prefix for notmuch_config_t is to avoid collisions
with the existing usage in notmuch-client.h.
2021-02-06 19:02:00 -04:00
David Bremner
39580e2d7f lib/open: add support for config profiles and default locations
Fill in the remainder of the documented functionality for
n_d_open_with_config with respect to config file location. Similar
searching default locations of the database file still needs to be
added.
2021-02-06 18:59:00 -04:00
David Bremner
e5f3c3ed50 lib: add stub for notmuch_database_open_with_config
Initially document the intended API and copy the code from
notmuch_database_open_verbose. Most of the documented functionality is
not there yet.
2021-02-06 18:57:35 -04:00
David Bremner
4743e87c2c lib: cache configuration information from database
The main goal is to allow configuration information to be temporarily
overridden by a separate config file. That will require further
changes not in this commit.

The performance impact is unclear, and will depend on the balance
between number of queries and number of distinct metadata items read
on the first call to n_d_get_config.
2021-02-06 18:56:05 -04:00
David Bremner
6a7b61b1d5 test: add (back) upgrade tests
In ee897cab8b the upgrade tests from pre v3 databases were
removed. The reasons for that are still valid, but we should still
test the code paths that do the upgrade, and it is relatively
straightforward to do that for v3 to v3 upgrades.
2021-01-17 13:24:04 -04:00
David Bremner
15d8067c0a test/T391-python-cffi
Make bindings test verbose. This helps in debugging.
2021-01-15 07:43:23 -04:00
David Bremner
003fdba7a4 test/T750-gzip: don't compress the xapian database
This causes mysterious failures in trying to detect if a database
exists.
2021-01-15 07:43:07 -04:00
Jonas Bernoulli
2ca941163d emacs: make headings outline-minor-mode compatible
`outline-minor-mode' treats comments that begin with three or more
semicolons as headings.  That makes it very convenient to navigate
code and to show/hide parts of a file.

Elips libraries typically have four top-level sections, e.g.:

;;; notmuch.el --- run notmuch within emacs...
;;; Commentary:...
;;; Code:...
;;; notmuch.el ends here

In this package many libraries lack a "Commentary:" section, which is
not optimal but okay for most libraries, except major entry points.

Depending on how one chooses to look at it, the "... ends here" line
is not really a heading that begins a section, because it should never
have a "section" body (after all it marks eof).

If the file is rather short, then I left "Code:" as the only section
that contains code.  Otherwise I split the file into multiple sibling
sections.  The "Code:" section continues to contain `require' and
`declare-function' forms and other such "front matter".

If and only if I have split the code into multiple sections anyway,
then I also added an additional section named just "_" before the
`provide' form and shortly before the "...end here" line.  This
section could also be called "Back matter", but I feel it would be
distracting to be that explicit about it.  (The IMO unnecessary but
unfortunately still obligatory "... ends here" line is already
distracting enough as far as I am concerned.)

Before this commit some libraries already uses section headings, some
of them consistently.  When a library already had some headings, then
this commit often sticks to that style, even at the cost inconsistent
styling across all libraries.

A very limited number of variable and function definitions have to be
moved around because they would otherwise end up in sections they do
not belong into.

Sections, including but not limited to their heading, can and should
be further improved in the future.
2021-01-13 07:10:27 -04:00
David Bremner
1331888374 Merge branch 'release' 2020-12-25 12:56:57 -04:00
David Bremner
6d5d28c593 test/T360-symbol-hiding.sh: trim extra output from readelf
readelf on (at least) ppc64le sometimes generates some extension to
the Ndx name inside '[]'. Remove this output to allow our simple
column based parsing to work.
2020-12-25 11:36:33 -04:00
David Bremner
76871fcf5e test: add regression test for searching with alternate config
Make sure upcoming changes to config handling do not break command
line specification.
2020-12-23 09:23:42 -04:00
David Bremner
59488ee929 test: use keys with group 'test' in T590-libconfig
In a future commit we want to interoperate better with glib KeyFiles,
which need groups for all keys.
2020-12-20 07:46:18 -04:00
David Bremner
82faa059b8 test/T360-symbol-hiding: use readelf in place of nm
It turns out that using nm -P isn't as portable as hoped. In
particular with some ELF ABIs (e.g. ppc64 ELFv1), the desired symbols
end up in the data section instead of text.

The test is currently only functional on ELF based architectures, so I
think it's legit to depend on readelf instead of nm.

The switch to readelf has the advantage that we can explicitely ask
for all of the symbols with global visibility, rather than grepping
for notmuch. That seems a more robust approach since it will catch any
strangely named global symbols.
2020-12-13 08:23:42 -04:00
David Bremner
85b5803869 test: fix syntax errors in erroring calls to notmuch insert
notmuch insert does not currently support passing a filename for the
input, so all of these tests have an extra error in addition to the
one being tested for.

Currently this does not make a difference because the error being
tested for is caught before the error of an extra command line
argument. In the future it might make a difference, and in any case it
is confusing.
2020-09-04 21:04:33 -03:00
David Bremner
fe449f779d test: fix uninitialized variable use in T562-lib-database
Fix a copy paste error of using the boolean ret as a notmuch_status_t,
and uninitialized.
2020-08-16 14:04:36 -03:00
David Bremner
25f9a42287 test: update README to reflect dropping upgrade tests
These test databases have been unneeded since ee897cab8.
2020-08-16 10:41:44 -03:00
Teemu Likonen
adb90b9bb6 Emacs: Indent first header line only when indentation is turned on
Previously in message-show mode message's first header line (From
header) was always indented, even if user had turned thread
indentation off with "<" (notmuch-show-toggle-thread-indentation)
command.

This change modifies notmuch-show-insert-headerline function so that
it doesn't indent the first header line if notmuch-show-indent-content
variable is nil.

This change also modifies tests so that they expect this new output
format:
test/emacs-show.expected-output/notmuch-show-indent-thread-content-off
2020-08-15 09:16:34 -03:00
Jonas Bernoulli
3665914f71 emacs: Do not abuse advice to monkey patch while testing
Use `cl-letf*' instead.
2020-08-09 21:17:06 -03:00
Jonas Bernoulli
96baa22318 emacs: Drop old advices that were only need for Emacs 23 2020-08-09 21:16:12 -03:00
Jonas Bernoulli
bb15524c12 test: Fix indentation
Fix it to consistently match the style we have configured in
".dir-locals.el".
2020-08-09 21:14:36 -03:00
Jonas Bernoulli
6c84dee531 Fix typos 2020-08-09 21:14:36 -03:00
Jonas Bernoulli
73b8f0b8d7 emacs: Various cosmetic changes 2020-08-09 21:14:36 -03:00
Jonas Bernoulli
c454135376 emacs: Use makefile-gmake-mode in Makefile*s
Use `makefile-gmake-mode' instead of `makefile-mode' because the
former also highlights ifdef et al. while the latter does not.

"./Makefile.global" and one "Makefile.local" failed to specify any
major mode at all but doing so is necessary because Emacs does not
automatically figure out that these are Makefiles (of any flavor).
2020-08-09 21:14:36 -03:00
Jonas Bernoulli
e1a700067a emacs: Use 'when' instead of 'if' when there is no ELSE part 2020-08-09 20:52:34 -03:00
Jonas Bernoulli
09f6533c37 emacs: Use 'unless' instead of 'when' and 'not'
Also use 'unless' in a few cases where previously 'if' was used with
'not' but without an ELSE part.
2020-08-09 20:51:26 -03:00
Jonas Bernoulli
caaa108760 emacs: Fix indentation 2020-08-09 20:48:09 -03:00
Đoàn Trần Công Danh
f5ae8040b3 T355: specify hash algorithm explicitly
On some systems (notably, the one shipped with LibreSSL),
default fingerprint digest algorithm is SHA256.

On other systems, users can change default digest algorithm by changing
default_md in /etc/ssl/default_md.

Let's ask openssl to provide us specific algorithm to make the test
more deterministic.

Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
2020-08-08 16:04:27 -03:00
David Bremner
0e4695abaa test: regression tests for n_indexopts_{get,set}_decrypt_policy
The main criteria is that they don't crash. Working with a closed
database is a bonus.
2020-08-03 21:05:29 -03:00
David Bremner
e3f88436b7 test: regression test for traversing config list with closed db
Also mention error return in API docs
2020-08-03 21:03:43 -03:00
David Bremner
a0b2a54aa7 test: regression test for n_d_get_config_list on closed db.
Exception is caught.
2020-08-03 21:01:53 -03:00
David Bremner
e8ffbf84ff test: regression test for n_directory_{get,set}_mtime
The mtime is cached, so closing the db is not a problem. Writing the
mtime throws an exception, which is caught.
2020-08-03 21:00:25 -03:00
David Bremner
f4f5db0f1a lib: fix return value for n_directory_delete
Falling out of the catch meant the error return was lost
2020-08-03 20:59:06 -03:00
David Bremner
b03cc6cf90 test: known broken test for n_directory_delete with closed db.
There is a return value bug in notmuch_directory_delete that is hiding
the exception.
2020-08-03 20:57:43 -03:00
David Bremner
337665e26a lib: catch exceptions in n_directory_get_child_files
Also clarify API in error case.
2020-08-03 20:56:09 -03:00
David Bremner
3890d98966 test: add known broken test for n_directory_get_child_files
This is a clone of the one for get_child_directories
2020-08-03 20:54:26 -03:00
David Bremner
9668583a49 lib: catch exceptions in n_directory_get_child_directories
Also clarify API in error case.
2020-08-03 20:53:04 -03:00
David Bremner
919b89156c test: add known broken test for n_directory_get_child_directories
Start a new test file (for the notmuch_directory_* API group) to hold
this test.
2020-08-03 20:51:40 -03:00
David Bremner
c151506773 test: regression test for n_m_get_filenames
Closing the database after the iterator is created is not a problem.
2020-08-03 20:50:15 -03:00
David Bremner
2b6e73d895 test: split header for lib-message tests.
This allows finer control over when to close the database.
2020-08-03 20:48:51 -03:00
David Bremner
e5e17926db test: add regression test for n_messages_collect_tags
Also test n_messages_destroy.
2020-08-03 20:47:18 -03:00
David Bremner
31dab9b1cd lib: return NULL from n_d_get_default_indexopts on error
This is a rare and probably serious programming error, so better not
to silently return a default value.
2020-08-03 20:45:58 -03:00
David Bremner
99a7aac8f2 test: drop use of db_ending
This will allow the dropping of the test for the default ending from
configure.
2020-08-01 20:27:30 -03:00
David Bremner
29a58ecf4a test: drop NOTMUCH_DEFAULT_XAPIAN_BACKEND from T360-symbol-test
Inspired by the suggestion of

         id:20200727154108.16269-3-congdanhqx@gmail.com

to drop the configuration test for the default backend. This version
is hopefully robust against backend changes.
2020-07-31 21:30:06 -03:00
David Bremner
00f48f397a test: destroy thread from closed database
Check for (non)-crash.
2020-07-31 07:41:00 -03:00
David Bremner
864f422f14 test: regression test for n_thread_get_tags
Code is taken from the API docs, with the twist that the underlying
database is closed. Not crashing is the main point.
2020-07-31 07:41:00 -03:00
David Bremner
43406e7d4c test: add regression tests for oldest and newest dates
These are strictly to watch for crashes, so don't be too fussy about
the actual timestamps.
2020-07-31 07:41:00 -03:00
David Bremner
c5017c76d9 test: add regression tests for n_t_get_{authors, subject}
This is returning explicitely cached data, so no database access is needed.
2020-07-31 07:41:00 -03:00
David Bremner
70b30066f6 test: add regression test for n_t_get_messages
This is similar to the case of toplevel messages. Currently everything
is cached, so no database access is necessary. This might change in
the future, but it should not crash in either case.
2020-07-31 07:41:00 -03:00
David Bremner
6ccc4338a4 test: add regression tests for n_thread_get_toplevel_messages
Include a test for the previously omitted non-trivial code path for
notmuch_thread_get_replies.
2020-07-31 07:41:00 -03:00
David Bremner
02ab473115 test: add regression test for n_thread_get_total_{messages,files}
This is returning cached info, so does not need to access the (closed)
database.
2020-07-31 07:41:00 -03:00
David Bremner
0baa581e6e test: regression test for n_t_get_thread_id
This is just cached data, so as long as we don't prematurely free
anything, it should be fine.
2020-07-31 07:41:00 -03:00
David Bremner
46468baa5a test: regression test for notmuch_query_destroy
As with other void API entries, not crashing counts as success.
2020-07-29 12:20:41 -03:00
David Bremner
451c0ce3fc test: regression tests for n_q_count_{messages, threads}
At least these exceptions are caught.
2020-07-29 12:20:30 -03:00
David Bremner
0a4c6cdeb1 test: regression test for n_q_search_messages
Exception handling matches notmuch_query_search_threads, at least for
this case.
2020-07-29 12:20:22 -03:00
David Bremner
25fc8662ed test: regression test for n_q_search_threads
At least this exception is caught.
2020-07-29 12:20:14 -03:00
David Bremner
4130b4e514 test: add regression test for n_q_add_tag_exclude
This relies on the change to not tear down the auxilary Xapian
objects, in particular the query parser, when the database is closed.
2020-07-29 12:20:06 -03:00
David Bremner
d5c84c8289 test: regression test for n_q_{set, get}_sort
We just want to make sure accessing the query owned by a closed db
does not cause a crash.
2020-07-29 12:19:57 -03:00
David Bremner
a292461e65 test: regression test for set_omit_excluded 2020-07-29 12:19:48 -03:00
David Bremner
768efe88d0 test: regression test for notmuch_query_get_database
Current functionality is too trivial to really justify a test, but
it's simpler just to test the complete API.
2020-07-29 12:19:39 -03:00
David Bremner
cbf9f890aa test: add regression test for n_q_{create,_get_query_string}
Start a new file of tests, to keep to the (emerging) scheme of one
notmuch_foo group per file
2020-07-29 12:19:09 -03:00
David Bremner
85da95e266 test: move notmuch_message_* tests to their own file
This is for consistency with the recently added tests for
notmuch_database_*.
2020-07-29 12:18:11 -03:00
Jonas Bernoulli
8b162b40f8 test: Explicitly state that we want to sign with sender
Since Emacs 27 'mml-secure-epg-sign' errors out if we don't opt-in to
signing as the sender using 'mml-secure-openpgp-sign-with-sender'.
2020-07-28 08:40:30 -03:00
Jonas Bernoulli
7b752e9eb4 test: Remove misguided emacs testing utilities
The goal of this abstraction was to save space.  But that failed as
the result actually was that four trivial lines got replace with 15
fairly complicated lines.  The opposite of what it was supposed to
do.

Also it made it harder to come up with the fix in the previous commit;
simply grepping for the relevant symbols did not work because they get
constructed at run-time instead of appearing in the source file.
2020-07-28 08:39:58 -03:00
Jonas Bernoulli
d5d8846c32 test: Deal with Emacs 27 switching to lexical scope by default
Starting with Emacs 27 undeclared variables in evaluated interactive
code uses lexical scope.  This includes code passed with '--eval' as
we do in the Emacs tests, which also happen to assume dynamic scope.

- This can affect variables defined by libraries that we use.  We
  let-bind such variables to change the behavior of functions which we
  then call with these bindings in effect.  If these libraries are not
  loaded beforehand, then the bindings are lexical and fail to have
  the effect we intended.

  At this time only 'smtpmail' has to be loaded explicitly (for the
  variables let-bound in emacs_deliver_message and emacs_fcc_message).

  'message' doesn't have to be loaded explicitly, because loading
  'notmuch' (in 'run_emacs') already takes care of that, indirectly.

- Our own testing-only variables also have to be declared explicitly.
  We should have done that anyway, but because of how and where these
  variables are used it was very easy to overlook that (i.e. it isn't
  something the byte-compiler ever looks at).  Not so in Emacs 27
  anymore; here this oversight caused four tests to fail.

  The numeric values of these variables get incremented by functions
  that we add to hooks that are run by many tests, not just the tests
  where we actually inspect the value and therefore take care to let-
  bind the values to 0 before we begin.  The global values therefore
  have to be numeric values as well.  I have chosen -100 instead of 0
  as the default in case someone writes a test that inspects the value
  but forgets to let-bind the value.  I hope that the unusual negative
  value that one is going to see in such a case will help debugging
  the issue.
2020-07-28 08:39:44 -03:00
David Bremner
24cf4381b8 test: add known broken test for n_d_get_default_indexopts
Xapian exceptions are swallowed and turned into default return value.
2020-07-22 19:52:55 -03:00
David Bremner
b268c8c071 lib: fix error return bug with n_d_set_config.
The catch block either needs to return, or the function needs to
return "status". Choose the latter for consistency with
n_d_get_config.
2020-07-22 19:52:55 -03:00
David Bremner
f6d74be840 test: add known broken test for n_d_set_config
Error status is currently lost.
2020-07-22 19:52:55 -03:00
David Bremner
a5e13d5218 test: add regression test for n_d_get_config
Xapian exceptions seem to handled OK, at least for this case.
2020-07-22 19:52:55 -03:00
David Bremner
dfab190a40 lib: add regresion test for n_d_get_all_tags
Existing error handling seems adequate, if not ideal.
2020-07-22 19:52:55 -03:00
David Bremner
64aa896151 lib: add regression test for n_d_find_message_by_filename
At least this Xapian exception is caught. Make sure it stays that way.
2020-07-22 19:52:55 -03:00
David Bremner
ce360ce4c9 lib: rename _n_d_create to _n_d_find_or_create
The error message and name were confusing when called in some "read
only" context.
2020-07-22 19:52:55 -03:00
David Bremner
97c887282a test: add known broken test for notmuch_database_remove_message
The current error message is a bit confusing; fix in next commit.
2020-07-22 19:52:55 -03:00
David Bremner
c477d7ce31 lib: convert relative filenames to absolute in n_d_index_file
The API docs promise to handle relative filenames, but the code did
not do it.

Also check for files outside the mail root, as implied by the API
description.

This fixes the bug reported at

     id:87sgdqo0rz.fsf@tethera.net
2020-07-22 19:52:55 -03:00
David Bremner
be3f4aec3f test: add known broken test for indexing relative path
The current code seems to look for the path relative to the current
working directory, rather than the mail store root.
2020-07-22 19:52:55 -03:00
David Bremner
9ab3e2e77d test: regression test for n_d_index_file closed db
Current exception handling seems OK, at least for this case.
2020-07-22 19:52:55 -03:00
David Bremner
1ca805a962 test: add regression test for n_d_get_directory
At least this exception is already handled correctly.
2020-07-22 19:52:55 -03:00
David Bremner
959cb4b7a7 test: regression test for n_d_get_revision
This function only accesses data cached by notmuch, so being closed is
not a problem.
2020-07-22 19:52:55 -03:00
David Bremner
bcb02dd8fc test: add regression test for n_d_{begin,end}_atomic
Xapian currently succeeds to begin/end a transaction on a closed database,
or at least does not throw an exception. Make the test robust against
this changing.
2020-07-22 19:52:55 -03:00
David Bremner
3f121d636e test: regression test for n_d_upgrade
The logic is that if it's acceptable to return SUCCESS for read only
database, it's acceptable for a closed one.
2020-07-22 19:52:55 -03:00
David Bremner
f25fc8e211 lib/n_d_needs_upgrade: handle error return from n_d_get_version
Also clarify documentation of error return from n_d_needs_upgrade.
2020-07-22 19:52:55 -03:00
David Bremner
cf90431cbf test: add known broken test for n_d_needs_upgrade
It's a bit arbitrary which value to return for errors, but the same
argument as for read only databases applies for errors.
2020-07-22 19:52:55 -03:00
David Bremner
37dc6045e4 test: add regression tests for notmuch database destroy
Either an open or closed database should be ok to destroy
2020-07-22 19:52:55 -03:00
David Bremner
964e783fd4 test: regression test for closing a closed database
This does not currently throw an error, and it should stay that way.
2020-07-22 19:52:55 -03:00
David Bremner
ab45654192 lib/n_d_get_version: catch exceptions and clarify the API
notmuch_database_get_version previously returned 0 on some errors, but
did not document this. Luckily 0 is not a valid database version.
2020-07-22 19:52:55 -03:00
David Bremner
920dc56e60 test: add known broken test for n_d_get_version on closed db
This should not crash, but it does currently.
2020-07-22 19:52:55 -03:00
David Bremner
f3a56b5f6d test: regression tests for n_d_status_string and n_d_get_path
These do not crash on a closed database, and we want to keep it that
way.

Start a new file of tests as T560-lib-error was starting to get unwieldy.
2020-07-22 19:52:55 -03:00
David Bremner
e9867b818b lib: fix exception messages for n_m_message_*
The original generic handler had an extra '%s' in the format
string. Update tests that failed to catch this because the template to
print status strings checked 'stat', which was not set.
2020-07-22 19:52:55 -03:00
David Bremner
1a8060b81f test: sanitize line numbers in exception reports
Prevent test suite churn when reported line numbers change.
2020-07-22 19:52:55 -03:00
David Bremner
765ca7bc08 lib: fix return value for n_m_reindex
Also update the documentation for the behaviour of n_m_get_thread_id
that this fix relies on.
2020-07-20 08:54:42 -03:00
David Bremner
a0c83e1fac test: add known broken test for n_m_reindex on closed db
This is another case where the code should not call INTERNAL_ERROR.
2020-07-20 08:54:42 -03:00