Commit graph

1619 commits

Author SHA1 Message Date
David Bremner
702635d5f6 Merge branch 'release' 2021-05-22 09:34:55 -03:00
David Bremner
3f4de98e7c lib/n_d_index_file: re-use thread-id of existing message
This prevents the message document getting multiple thread-id terms
when there are multiple files with the same message-id.

This change shifts some thread ids, requiring adjustments to other tests.
2021-05-22 09:08:02 -03:00
David Bremner
349fc29803 test: add known broken test for duplicate thread-id terms
According to my bijection, this bug has been present since commit
411675a6ce in 2017. It is not completely clear what harm it causes in
regulary use, but it (at least) makes notmuch crash when compiled with
-DDEBUG_DATABASE_SANITY.
2021-05-22 09:02:58 -03:00
Felipe Contreras
92454bc093 test: split emacs functionality to its own file
This way it's easier to identify the tests that do require emacs stuff.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-17 07:29:04 -03:00
Felipe Contreras
bba4788aec test: emacs: fix a couple of shellcheck complaints
In test-lib-emacs.sh line 20:
    test_require_external_prereq ${TEST_EMACS} || ret=1
                                 ^-----------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
    test_require_external_prereq "${TEST_EMACS}" || ret=1

In test-lib-emacs.sh line 21:
    test_require_external_prereq ${TEST_EMACSCLIENT} || ret=1
                                 ^-----------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
    test_require_external_prereq "${TEST_EMACSCLIENT}" || ret=1

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-17 07:28:44 -03:00
Felipe Contreras
29ba523275 test: emacs: check for configured emacs
Commit d59d9c81 (test: Make the emacsclient binary user-configurable,
2012-11-27) modified the prereq check for the configured emacsclient,
but we probably want to do the same for emacs itself.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-17 07:28:33 -03:00
Felipe Contreras
cd9ec20a49 test: emacs: simplify missing dependencies check
No functional changes.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-17 07:28:23 -03:00
Felipe Contreras
4561f694e9 test: more style fixes
In order to fit the git coding style.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-17 07:28:01 -03:00
Felipe Contreras
645bea13cb test: trivial style cleanups
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-17 07:27:52 -03:00
David Bremner
c84ccb70f3 Merge branch 'release' 2021-05-15 09:10:58 -03:00
David Bremner
b3258244c8 lib/open: restore default database path of $HOME/mail
Although this default worked for "notmuch config get", it didn't work
most other places. Restore the previous functionality, with the
wrinkle that XDG locations will shadow $HOME/mail if they exist.

This fixes a bug reported by Jack Kamm in id:87eeefdc8b.fsf@gmail.com
2021-05-15 08:40:05 -03:00
Kyle Meyer
319dcfb50e emacs: restore tag-changes and query bindings for tag hooks
notmuch-before-tag-hook and notmuch-after-tag-hook are supposed to
have access to two dynamic variables, tag-changes and query, but these
were lost with the switch to lexical binding in fc4cda07 (emacs: use
lexical-bindings in all libraries, 2021-01-13).

Add a variant of Emacs's dlet (not available until Emacs 28) and use
it in notmuch-tag to expose tag-changes and query to the hooks.
2021-05-15 08:34:28 -03:00
David Bremner
a663783343 test: add known broken tests for notuch-{before,after}-tag-hook
These tests illustrate the bug reported in id:87v97ytd2s.fsf@fastmail.fm
2021-05-15 08:31:00 -03:00
David Bremner
25da4a1340 Merge branch 'release' 2021-05-12 08:42:28 -03:00
David Bremner
393c92b042 lib/notmuch_database_reopen: reload some database metadata
In some uses of reopen, new documents and threads maybe have been
added, and e.g. compaction may have changed the uuid.
2021-05-12 08:40:04 -03:00
Michael J Gruber
0729c4e81f test: change database from within pre-new hook
Due to the change in the config system, notmuch keeps a notmuch database
open when it would not do so before. Consequently, it can miss changes
to the database which are done from a hook (while notmuch holds the
databse in read only mode). When notmuch itself writes to the database
after that it uses wrong assumptions about the last used doc id etc.

Demonstrate this by triggering an assertion. (This new test succeeds
with notmuch 0.31.4.)

Signed-off-by: Michael J Gruber <git@grubix.eu>
Amended-by: db. Check for both messages
2021-05-12 08:34:34 -03:00
David Bremner
b4a4ed0df9 Merge branch 'release' 2021-05-10 11:36:56 -03:00
David Bremner
8bf3311084 test: test explicit configuration of backup directory
Including the relative path that was broken until a recent commit.
2021-05-10 11:13:07 -03:00
David Bremner
891b950219 test: test relative paths for database.hook_dir 2021-05-10 11:13:07 -03:00
David Bremner
1040e7aa07 lib/config: expand relative paths when reading from database
This makes the treatment of relative paths consistent between the
database and config files.
2021-05-10 11:12:58 -03:00
David Bremner
322a492c77 test: add known broken test for relative setting of mail_root
The behaviour should not change depending on where the configuration
is stored.
2021-05-10 11:11:42 -03:00
David Bremner
31098c4ae4 lib/config: canonicalize paths relative to $HOME.
Prior to 0.32, notmuch had the (undocumented) behaviour that it
expanded a relative value of database.path with respect to $HOME. In
0.32 this was special cased for database.path but broken for
database.mail_root, which causes problems for at least notmuch-new
when database.path is set to a relative path.

The change in T030-config.sh reflects a user visible, but hopefully
harmless behaviour change; the expanded form of the paths will now be
printed by notmuch config.
2021-05-10 11:08:18 -03:00
David Bremner
ef0b126619 test: add known broken test for relative database path in new
This test highlights a bug introduced in 0.32. The new split between
path and mail_root does not properly canonicalize relative paths in
the latter.
2021-05-10 11:06:40 -03:00
Felipe Contreras
ac64de450a test: ruby: simplify output comparison
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-05 17:00:37 -03:00
Felipe Contreras
faf5511c3e test: ruby: don't use instance variables
Local variables are perfectly fine.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-05 17:00:16 -03:00
Felipe Contreras
c4b23cf29a test: ruby: use much more standard Ruby idioms
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-05 16:40:15 -03:00
Felipe Contreras
63413a5563 test: ruby: use much more standard puts
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-02 08:12:15 -03:00
Felipe Contreras
2dbc5fdf5a test: ruby: simplify test_ruby()
We always do test_expect_equal_file, so do it in test_ruby() directly.

The only subtest where we don't (get non-existent file) can be easily
modified.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-02 08:12:05 -03:00
Felipe Contreras
9f6bc01824 test: ruby: simplify MAIL_DIR initialization
There's no need to complicate the script passing the MAIL_DIR
environment variable.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-02 08:11:56 -03:00
Felipe Contreras
c1b99d6f94 test: ruby: simplify MAIL_DIR check
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-02 08:11:42 -03:00
Felipe Contreras
a75a9a5aed test: ruby: refactor test_ruby()
There's no point in repeating the same initialization in all the tests.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-02 08:11:17 -03:00
Felipe Contreras
8cbd8e71d0 test: move test_ruby() inside the only client
Not much point in polluting the main library, and also will be useful to
modify it in tandem with the tests.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-02 08:11:04 -03:00
Felipe Contreras
fe9616aef1 test: add external prereqs to many emacs tests
The tests fail otherwise.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-01 21:15:27 -03:00
Felipe Contreras
049a652da7 test: add prereqs check in test_emacs_expect_t
test_emacs may update the external prereqs, in which case we want to
skip the test rather than fail.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-01 21:15:10 -03:00
Felipe Contreras
7579f3ac6b test: fix wrong SKIP messages
When the external prereqs are updated inside the body of the command
(e.g. test_emacs) the message in test_report_skip_ is wrong: it outputs
the body of the command instead of the subtest name.

We need to pass the same argument we pass to test_skip.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-01 21:14:55 -03:00
Felipe Contreras
0b48e65526 test: use correct fqdn in passwd_sanitize()
My fqdn is 'natae.localdomain', however, socket.getfqdn() returns
'localhost'.

To fetch the true fqdn we need socket.getaddrinfo().

For more information see: https://stackoverflow.com/a/11580042/10474

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-01 21:14:37 -03:00
Felipe Contreras
2add9ce722 test: unset NAME environment variable
Otherwise the output from the tests would be different.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-01 21:13:41 -03:00
Felipe Contreras
ff5f141bf2 test: fix passwd_sanitize()
If any of the variables is empty the output is completely messed up,
because replace("", "FOO") puts "FOO" before every single character.

I don't have my full name configured, and this is what I get:

  USER_FULL_NAME=USER_FULL_NAME=USER_FULL_NAME USER_FULL_NAMEsUSER_FULL_NAMEtUSER_FULL_NAMEdUSER_FULL_NAMEoUSER_FULL_NAMEuUSER_FULL_NAMEtUSER_FULL_NAME USER_FULL_NAME=USER_FULL_NAME=USER_FULL_NAME

Let's check for empty strings before doing any replace.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2021-05-01 21:13:23 -03:00
David Bremner
00487fa901 test: hide message from moreutils parallel.
The argument --version confuses moreutils parallel, but this is OK,
because its confusing does not include printing "GNU"
2021-04-26 09:00:11 -03:00
Tobias Backer Dirks
f2b5ad28e2 test: check for GNU parallel with --version options
The lastest versions of GNU parallel no longer make mention of GNU
within their help output. This causes the test script to mistakenly use
the moreutils parallel execution. In order to fix this, while
maintaining compatibility with previous versions of GNU parallel,
--version should be used.

Signed-off-by: Tobias Backer Dirks <omgitsaheadcrab@gmail.com>
2021-04-24 20:06:41 -03:00
David Bremner
27a2c2c80d test: replace notmuch_passwd_sanitize with python version
Apparently the -f option to hostname is not portable, and in fact it
does not seem to always behave reasonably in e.g. a chroot.

Python code originally due to Tomi [1], modified by yours truly.

[1]: id:m2lf9fbkug.fsf@guru.guru-group.fi
2021-04-24 12:11:22 -03:00
Đoàn Trần Công Danh
828a004921 test-lib: unset XDG_CONFIG_HOME
lib/open.cc:_load_key_file will only open xdg-config files in
$XDG_CONFIG_HOME if it's defined, $HOME/.config will be considered if
and only if XDG_CONFIG_HOME not defined.

Let's unset said variable before running the test.
2021-04-18 10:00:10 -03:00
David Bremner
5248f55d5f test: put shim at end of LD_PRELOAD path
Certain tools like the address-sanitizer fail if they are not the
first LD_PRELOADed library. It does not seem to matter for our shims,
as long as they are loaded before libnotmuch.
2021-04-16 09:06:39 -03:00
David Bremner
ec4b7efbe4 CLI/config: remove calls to notmuch_config_open from top level
This will allow simplifying the subcommand interface.

Change the internal API to notmuch_config_open to not tie it to the
implementation of subcommands in notmuch.c.

It also fixes a previously broken test, since notmuch_config_open does
not understand the notion of the empty string as a config file name.
2021-04-06 21:32:36 -03:00
David Bremner
77d4b26d3e CLI/config: support set/get with split configuration
There are two small code changes. The main one is to retrieve the
possibly updated config file name found during the database opening
call. The second change is to allow empty config file names, as
a (currently broken) way of specifying that configuration should only
be taken from the database.
2021-03-27 09:26:14 -03:00
David Bremner
217f819608 CLI+lib: detect missing database in split configurations.
Eventually we want to do all opening of databases in the top
level (main function). This means that detection of missing databases
needs to move out of subcommands. It also requires updating the
library to use the new NO_DATABASE status code.
2021-03-27 09:26:14 -03:00
David Bremner
dd9112e7d8 CLI/config: default to storing all config in external files
Previously the fact that some configuration options were only stored
in the database (and thus editing the config file had no effect) was a
source of user confusion. This was fixed with the series ending at
d9af0af164.

On the other hand, the underlying partition of config options into
those stored by default in the database and those stored in the config
file remained. This is also confusing, since now some invocations of
"notmuch config set" modify the config file, and others silently
modify the database instead.

With this commit, it is up to the user to decide which configuration
to modify. A new "--database" option is provided for notmuch config to
enable modifying the configuration information in the database;
otherwise the default is to update an external config file.
2021-03-27 09:26:14 -03:00
David Bremner
f055e94562 CLI/config: use notmuch_database_reopen
This allows `notmuch config` to use the common configuration
information carried by the notmuch_database_t opened at the top level.
2021-03-27 09:26:14 -03:00
David Bremner
b76da87d29 CLI/config: migrate notmuch_config_open to new config
notmuch_config_open will be preserved in the medium term for use by
the commands that are manipulating the config file directly (config
and setup)
2021-03-27 09:26:14 -03:00
David Bremner
3787fe6c84 CLI/config: switch "notmuch config list" to merged config
Use the database opened at the top level rather than opening another
notmuch_database_t.

Test output changes because keys are now listed in alphabetical order,
and because a missing database is no longer an error.
2021-03-27 09:26:14 -03:00
David Bremner
ee0e1cfb17 test/setup: check file output instead of notmuch config list
This allows verification of comments added to new / updated config
file.
2021-03-27 09:26:14 -03:00
David Bremner
ecb91d056c CLI/config: use merged config for "config get"
This commit starts the conversion of notmuch-config.c
functionality (as opposed to just interface) to the new config
framework.

The change to T030-config is because of the move of the
canonicalization database paths from the notmuch_config_t accessor to
the internal function _choose_database_path.
2021-03-27 09:26:14 -03:00
David Bremner
2e9ee47072 lib/config: set default for primary user email
This is mainly copying code from the CLI into the lib. The CLI copy
will be deleted in a later commit.
2021-03-27 09:26:14 -03:00
David Bremner
e81dc2072f lib/config: set defaults for user full name
This just copies code from from the CLI into the library. New test
infrastructure is needed because apparently we have never tested this
code path.
2021-03-27 09:26:14 -03:00
David Bremner
863b243185 lib/config: add config_pairs iterators
The layer of shims here seems a bit wasteful compared to just calling
the corresponding string map functions directly, but it allows control
over the API (calling with notmuch_database_t *) and flexibility for
future changes.
2021-03-27 09:26:14 -03:00
David Bremner
0c6db22930 lib/config: add notmuch_config_get_values_string
This is to support the less common (at least in the notmuch codebase)
case of accessing a ;-delimited list config value with an arbitrary
string key.
2021-03-27 09:26:14 -03:00
David Bremner
2fc40e24de lib: provide notmuch_database_load_config
This is mainly targetted at notmuch-config.c, or similar applications
that don't necessarily need both a configuration file and a database
to exist.
2021-03-27 09:26:14 -03:00
David Bremner
79a4b2e9e7 test: convert random-corpus to use n_d_open_with_config
Remove one more usage of notmuch_config_get_database_path
2021-03-27 09:26:14 -03:00
David Bremner
4e209ca99a CLI/new: use configuration variable for backup directory
The stat is essentially replaced by the mkdir for error detection
purposes.  This changes the default location for backups to make
things tidier, even in non-split configurations. Hopefully there is
not too many user scripts relying on the previous location.

Because the default location may not exist, replace the use of stat
for error detection with a call to mkdir.
2021-03-20 07:53:02 -03:00
David Bremner
a7873df331 lib/config: add configuration variable for backup directory
Like the hook directory, we primarily need a way to communicate this
directory between various components, but we may as well let the user
configure it.

Most of the diff is generalizing choose_hook_dir to work for both
backup and hook directories.
2021-03-20 07:51:36 -03:00
David Bremner
e3a4abc513 lib/open: fix hook directory calculation in split configuration
Choose sibling directory of xapian database, as .notmuch may not
exist.

libgen.h is already used in debugger.c, so it is not a new dependency
/ potential portability problem.
2021-03-20 07:50:13 -03:00
David Bremner
b30a59157d lib/compact: enable split config
This promotes _choose_xapian_path from static to extern linkage in
order to share between open.cc and database.cc.
2021-03-20 07:50:06 -03:00
David Bremner
74c3cc03c2 CLI/insert: support split database and mail root
The new test is in T055-path-config because it uses the helper
function split_config, and because it seems easier to put the
database path related tests in one place.
2021-03-20 07:44:08 -03:00
David Bremner
c82554193d lib/open: support XDG_DATA_HOME as a fallback database location.
This changes some error reporting, either intentionally by reporting
the highest level missing directory, or by side effect from looking in
XDG locations when given null database location.
2021-03-20 07:43:09 -03:00
David Bremner
2c879667b3 CLI/new: support split database and mail location
This adds new state variable for the mail root, and uses it most
places db_path was used. The notable exception is dumps from
backups. The latter will be dealt with properly in a future commit.
2021-03-20 07:42:06 -03:00
David Bremner
5ec6fd4dcf lib/open: check for split configuration when creating database.
The main functionality will be tested when notmuch-new is converted to
support split configuration. Here only the somewhat odd case of split
mail root which is actually symlinked to the database path is tested.
2021-03-20 07:41:04 -03:00
David Bremner
e823d05ae6 lib: support splitting mail from database location.
Introduce a new configuration value for the mail root, and use it to
locate mail messages in preference to the database.path (which
previously implied the mail messages were also in this location.

Initially only a subset of the CLI is tested in a split
configuration. Further changes will be needed for the remainder of the
CLI to work in split configurations.
2021-03-20 07:39:12 -03:00
David Bremner
986056bdbc lib/open: Use check for existing database by trial opening
This is a bit heavyweight for now, but it will make more sense when we
check multiple locations for the Xapian database.
2021-03-20 07:38:15 -03:00
David Bremner
cba540d6f5 lib/open: reuse directory checks from n_d_c_with_config
Make checks more uniform between creating new databases and opening
existing ones.
2021-03-20 07:24:59 -03:00
David Bremner
793d4305d3 lib/open: support NOTMUCH_DATABASE environment variable
The additional code is trivial, but making sure we get the priority of
various options correct takes a few tests.
2021-03-19 22:06:44 -03:00
David Bremner
6e6c319c26 CLI/show: complete conversion to new configuration framework.
In order to open the database in main() for this command, we may need
to re-open it in the (possibly less common) case where crypto options
require write access.
2021-03-19 22:06:19 -03:00
David Bremner
a9f74aeeb9 CLI/new: drop the write lock to run the pre-new hook.
This fixes a bug reported in [1]. In principle it could be possible
avoid one of these reopens, but it complicates the logic in main with
respect to creating new databases.

[1]: id:9C1993DF-84BD-4199-A9C8-BADA98498812@bubblegen.co.uk
2021-03-18 22:58:02 -03:00
David Bremner
7d286f0272 test: Add tests for write access to database from hooks.
Recent changes to configuration handling meant the pre-new hook was
run while the database was open read only, limiting what could be done
in the hook. Add some known broken tests for this problem, as well as
a regression test for write access from the post-new hook.
2021-03-18 22:54:13 -03:00
David Bremner
13efbd0e1c lib: support reopening databases for write access.
In the future Xapian will apparently support this more conveniently
for the cases other than READ_ONLY => READ_ONLY

Conceptually this function seems to fit better in lib/open.cc;
database.cc is still large enough that moving the function makes
sense.
2021-03-18 08:04:06 -03:00
David Bremner
9a5406cc94 test/setup: add check for config exists, but no database.
This code path is not currently tested, and will need updating if the
location of the xapian database changes.
2021-03-18 08:03:19 -03:00
David Bremner
dcd4df2cc0 test: add known broken test for long directory bug
In [1] Gregor Zattler explained the results of his hard work
tracking down a bug in notmuch with long directories. This test
duplicates the bug.

[1]: id:20210317194728.GB5561@no.workgroup
2021-03-18 07:57:38 -03:00
uncrustify
df4c66f85d test: run uncrustify
This is the result of running:

    $ uncrustify --replace --config ../devel/uncrustify.cfg *.cc *.c *.h

in the test directory.
2021-03-13 08:45:34 -04:00
David Bremner
97fadd0645 test: clean up some extra whitespace.
The extra space is mainly just untidy.
2021-03-12 07:19:14 -04:00
Tomi Ollila
d12d91f828 test: T020-compact.sh: fix work directory location
Fix use of $TEST_DIRECTORY ($NOTMUCH_BUILDDIR/test/) with use of
$TMP_DIRECTORY ($NOTMUCH_BUILDDIR/test/tmp.T020-compact/ in case
of T020-compact.sh) as root directory where to write test files
and directories.
2021-03-12 07:16:01 -04:00
David Bremner
4c79a2dabe notmuch 0.31.4 release
-----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEkiyHYXwaY0SiY6fqA0U5G1WqFSEFAmAuVjQACgkQA0U5G1Wq
 FSG6RA//bDdAtsG7QlywGONVX1FOSHxAgppDVRvSDXuluPmGgWvc2T80awbonfT5
 AHy9co41L9484QbOzd29d9Ttu5O39JdQbo+DCAJ6y2vm1M1dLCZw3HXaZGfxoUZo
 9L9Agxg/rYIEso7dAehOs3rGsia46aj2MDCVCUuA4DkVFhkbQ2QidIH+l3VPdYqm
 +1XmuJwyftO7hMifFd9W9ifO9wrBcK9WtJ23feufMhqfBsS0ItLysOhZQx+QdxSj
 7GuA6qX6V7XlIWpdohIOMKmT9tGHMDUo6Qk5m8aSc2XmKAybAXRc+qr0Kg8EpvBF
 1d9SqNptcXQL6rORJxluXR/aCuCb7m8YDgxFVSrFcp/M/twbpDC3WzjmAV+RZpDP
 GBKDH06IGaOMOj9GaYYWCe9loGROzOIT4y04Ckukit0AWpmdbaKnkAkPdrxrMWEu
 a5v5KkkMYT42q0PFU5bDjwvq+8Afmzt7oMO72zXz0mfuFNtFzZCUN/AG9LtHoRfk
 Q2ks/xBppmtCxWRAHVWc7f+Gk1OzI2PLVWnXIe9gxGimSpgT2QPUvZcxLHT/XWD5
 XRrrxtkPjKB1v6tWOFEMPI6WxE1iBzoC9AdG2h48ZP5drlLObcAxhAg5AvHJGJSr
 8D4iSa9cKjHD6qxDSXChzF6BF4aDNhF2Flq9Cs9tZb8nyt+Ix18=
 =mAE/
 -----END PGP SIGNATURE-----

Merge tag '0.31.4'

notmuch 0.31.4 release
2021-02-18 08:47:53 -04:00
David Bremner
a37d5f5976 test: Fix race condition in T568-lib-thread.sh
The assignment of thread-ids is (apparently) non-deterministic in a
way that mostly seems to show up on multicore machines. In my tests
the number is different from that previously assumed by this test
about 15% of the time on a 50 thread (25 core) Xeon.

Since message id's are fixed, use a message known to be in the thread
of interest to pick out the correct thread-id.
2021-02-18 07:15:47 -04:00
Daniel Kahn Gillmor
7061e41cd0 python: convert shebangs to python3
This is the last bit of "python" left in the notmuch codebase.

https://www.python.org/dev/peps/pep-0394/#recommendation encourages
"third-party distributors" to use more-specific shebang lines.  I'm
not certain that the notmuch project itself is a "third-party
contributor" but I think this is a safe way to encourage people to use
python3 when they're developing notmuch.

We already have python3 explicitly elsewhere in the codebase for
developers (in nmbug).

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2021-02-15 16:32:31 -04:00
David Bremner
f61d88c6f4 CLI: use configured hook directory
This enables support for hooks outside the database directory.
It relies strongly on configuration information being usable between
closing the database and destroying it.
2021-02-06 19:57:55 -04:00
David Bremner
0345bc57a0 lib/open: set HOOK_DIR on open
This is a simple two step path search.  Most error checking is
deferred until running the hooks.
2021-02-06 19:56:13 -04:00
David Bremner
4922416ccc lib/config: add HOOK_DIR
The hook directory configuration needs to be kept in synch with the
other configuration information, so add scaffolding to support this at
database opening time.
2021-02-06 19:54:09 -04:00
David Bremner
0d3bef312d cli/new: convert to new config framework
In addition to the same type of changes as converting other
subcommands, add the possibility of creating a database at the top
level. It would probably make sense to use this for insert as well.
2021-02-06 19:52:11 -04:00
David Bremner
ac67cd84ee lib: introduce notmuch_database_create_with_config
This takes a config path parameter, and can use that to decide the
new database location.
2021-02-06 19:48:34 -04:00
David Bremner
b860be6a76 lib/config: add NOTMUCH_CONFIG_NEW_IGNORE
This will be needed by (at least) the conversion of notmuch-new.c to
the new config framework
2021-02-06 19:39:33 -04:00
David Bremner
f118ef3c3d cli/compact: convert to new configuration framework
Switch to the newly created API function notmuch_database_compact_db,
which takes the database opened in main().
2021-02-06 19:36:06 -04:00
David Bremner
c56dcea7e2 cli/tag: convert to new config framework.
In addition to changing configuration access, change talloc context
for allocation.
2021-02-06 19:30:20 -04:00
David Bremner
6c28039ebb CLI/show: mostly switch show to new config framework
This will need some cleanup when the transition completes, and we stop
passing notmuch_config_t structs to the subcommands.

Unlike the general case, we open the database in the subcommand, since
we don't know whether it should be opened read/write until we parse
the command line arguments.

Add a test to make sure passing config file on the command line is not
broken by these or future config related changes.
2021-02-06 19:27:54 -04:00
David Bremner
d613d10ddd CLI/{search,address}: convert to new configuration framework
Since we are already passing a search context around as a kind of
parameter block, add a new talloc context to that to replace relying
on 'config'.

Convert notmuch-search and notmuch-address at the same time, because
they share some code.

Add a test to make sure we don't break passing configuration as a
command line argument.
2021-02-06 19:23:28 -04:00
David Bremner
8588719eb1 CLI/reply: convert to new config framework
This is messier than some of the other conversions because the
extensive use of 'config' as a talloc context.
2021-02-06 19:21:23 -04:00
David Bremner
0ab0b48be2 cli/reindex: convert new config framework
The only non-trivial part is switching the talloc context for
query_string_from args from 'config' to 'notmuch'.
2021-02-06 19:19:18 -04:00
David Bremner
f994f0e7df CLI/insert: convert to new config framework.
The new talloc context is needed to run the hook at the very end of
the function. That in turn is needed so that this process gives up the
write lock on the database.
2021-02-06 19:17:34 -04:00
David Bremner
66adcd4f53 CLI/restore: convert to new config framework
Switch one configuration check to new n_c_get_bool function, and
switch use of config as talloc context to notmuch.
2021-02-06 19:15:53 -04:00
David Bremner
53f27aaf73 cli/dump: convert to new config framework
This conversion is trivial because the only configuration information
accessed by dump is that stored in the database (in order to dump
it). We do need to be careful to keep the write lock on the database
to ensure dump consistency.
2021-02-06 19:12:34 -04:00
David Bremner
acc6331baa CLI/count: switch to new configuration framework
The main effort is changing from the old argv style config list
iterators to the new more opaque ones provided by the library (and
backed by the database+file config cache).
2021-02-06 19:11:06 -04:00
David Bremner
d071828bd5 lib/config: make values iterators restartable
This is relatively cheap, and makes it easier to transform existing
code which uses arrays of pointers to store configuration lists.
2021-02-06 19:09:39 -04:00
David Bremner
fd6f8e6c30 lib/config: add config values iterator
This is intended to avoid duplicating the string splitting and
traversal code for all clients of the config API.
2021-02-06 19:08:12 -04:00
David Bremner
06a64cf0aa lib/open: load default values for known configuration keys.
This emulates the behaviour of notmuch_config_open defined in the CLI,
in that it fills in default values if they are not otherwise defined.
2021-02-06 19:03:36 -04:00
David Bremner
d6bd87a712 lib/config: add notmuch_config_key_{get,set}
By using an enum we can have better error detection than copy pasting
key strings around.

The question of what layer this belongs in is a bit
tricky. Historically most of the keys are defined by the CLI. On the
other hand features like excludes are supported in the
library/bindings, and it makes sense to configure them from the
library as well.

The somewhat long prefix for notmuch_config_t is to avoid collisions
with the existing usage in notmuch-client.h.
2021-02-06 19:02:00 -04:00
David Bremner
39580e2d7f lib/open: add support for config profiles and default locations
Fill in the remainder of the documented functionality for
n_d_open_with_config with respect to config file location. Similar
searching default locations of the database file still needs to be
added.
2021-02-06 18:59:00 -04:00
David Bremner
e5f3c3ed50 lib: add stub for notmuch_database_open_with_config
Initially document the intended API and copy the code from
notmuch_database_open_verbose. Most of the documented functionality is
not there yet.
2021-02-06 18:57:35 -04:00
David Bremner
4743e87c2c lib: cache configuration information from database
The main goal is to allow configuration information to be temporarily
overridden by a separate config file. That will require further
changes not in this commit.

The performance impact is unclear, and will depend on the balance
between number of queries and number of distinct metadata items read
on the first call to n_d_get_config.
2021-02-06 18:56:05 -04:00
David Bremner
6a7b61b1d5 test: add (back) upgrade tests
In ee897cab8b the upgrade tests from pre v3 databases were
removed. The reasons for that are still valid, but we should still
test the code paths that do the upgrade, and it is relatively
straightforward to do that for v3 to v3 upgrades.
2021-01-17 13:24:04 -04:00
David Bremner
15d8067c0a test/T391-python-cffi
Make bindings test verbose. This helps in debugging.
2021-01-15 07:43:23 -04:00
David Bremner
003fdba7a4 test/T750-gzip: don't compress the xapian database
This causes mysterious failures in trying to detect if a database
exists.
2021-01-15 07:43:07 -04:00
Jonas Bernoulli
2ca941163d emacs: make headings outline-minor-mode compatible
`outline-minor-mode' treats comments that begin with three or more
semicolons as headings.  That makes it very convenient to navigate
code and to show/hide parts of a file.

Elips libraries typically have four top-level sections, e.g.:

;;; notmuch.el --- run notmuch within emacs...
;;; Commentary:...
;;; Code:...
;;; notmuch.el ends here

In this package many libraries lack a "Commentary:" section, which is
not optimal but okay for most libraries, except major entry points.

Depending on how one chooses to look at it, the "... ends here" line
is not really a heading that begins a section, because it should never
have a "section" body (after all it marks eof).

If the file is rather short, then I left "Code:" as the only section
that contains code.  Otherwise I split the file into multiple sibling
sections.  The "Code:" section continues to contain `require' and
`declare-function' forms and other such "front matter".

If and only if I have split the code into multiple sections anyway,
then I also added an additional section named just "_" before the
`provide' form and shortly before the "...end here" line.  This
section could also be called "Back matter", but I feel it would be
distracting to be that explicit about it.  (The IMO unnecessary but
unfortunately still obligatory "... ends here" line is already
distracting enough as far as I am concerned.)

Before this commit some libraries already uses section headings, some
of them consistently.  When a library already had some headings, then
this commit often sticks to that style, even at the cost inconsistent
styling across all libraries.

A very limited number of variable and function definitions have to be
moved around because they would otherwise end up in sections they do
not belong into.

Sections, including but not limited to their heading, can and should
be further improved in the future.
2021-01-13 07:10:27 -04:00
David Bremner
1331888374 Merge branch 'release' 2020-12-25 12:56:57 -04:00
David Bremner
6d5d28c593 test/T360-symbol-hiding.sh: trim extra output from readelf
readelf on (at least) ppc64le sometimes generates some extension to
the Ndx name inside '[]'. Remove this output to allow our simple
column based parsing to work.
2020-12-25 11:36:33 -04:00
David Bremner
76871fcf5e test: add regression test for searching with alternate config
Make sure upcoming changes to config handling do not break command
line specification.
2020-12-23 09:23:42 -04:00
David Bremner
59488ee929 test: use keys with group 'test' in T590-libconfig
In a future commit we want to interoperate better with glib KeyFiles,
which need groups for all keys.
2020-12-20 07:46:18 -04:00
David Bremner
82faa059b8 test/T360-symbol-hiding: use readelf in place of nm
It turns out that using nm -P isn't as portable as hoped. In
particular with some ELF ABIs (e.g. ppc64 ELFv1), the desired symbols
end up in the data section instead of text.

The test is currently only functional on ELF based architectures, so I
think it's legit to depend on readelf instead of nm.

The switch to readelf has the advantage that we can explicitely ask
for all of the symbols with global visibility, rather than grepping
for notmuch. That seems a more robust approach since it will catch any
strangely named global symbols.
2020-12-13 08:23:42 -04:00
David Bremner
85b5803869 test: fix syntax errors in erroring calls to notmuch insert
notmuch insert does not currently support passing a filename for the
input, so all of these tests have an extra error in addition to the
one being tested for.

Currently this does not make a difference because the error being
tested for is caught before the error of an extra command line
argument. In the future it might make a difference, and in any case it
is confusing.
2020-09-04 21:04:33 -03:00
David Bremner
fe449f779d test: fix uninitialized variable use in T562-lib-database
Fix a copy paste error of using the boolean ret as a notmuch_status_t,
and uninitialized.
2020-08-16 14:04:36 -03:00
David Bremner
25f9a42287 test: update README to reflect dropping upgrade tests
These test databases have been unneeded since ee897cab8.
2020-08-16 10:41:44 -03:00
Teemu Likonen
adb90b9bb6 Emacs: Indent first header line only when indentation is turned on
Previously in message-show mode message's first header line (From
header) was always indented, even if user had turned thread
indentation off with "<" (notmuch-show-toggle-thread-indentation)
command.

This change modifies notmuch-show-insert-headerline function so that
it doesn't indent the first header line if notmuch-show-indent-content
variable is nil.

This change also modifies tests so that they expect this new output
format:
test/emacs-show.expected-output/notmuch-show-indent-thread-content-off
2020-08-15 09:16:34 -03:00
Jonas Bernoulli
3665914f71 emacs: Do not abuse advice to monkey patch while testing
Use `cl-letf*' instead.
2020-08-09 21:17:06 -03:00
Jonas Bernoulli
96baa22318 emacs: Drop old advices that were only need for Emacs 23 2020-08-09 21:16:12 -03:00
Jonas Bernoulli
bb15524c12 test: Fix indentation
Fix it to consistently match the style we have configured in
".dir-locals.el".
2020-08-09 21:14:36 -03:00
Jonas Bernoulli
6c84dee531 Fix typos 2020-08-09 21:14:36 -03:00
Jonas Bernoulli
73b8f0b8d7 emacs: Various cosmetic changes 2020-08-09 21:14:36 -03:00
Jonas Bernoulli
c454135376 emacs: Use makefile-gmake-mode in Makefile*s
Use `makefile-gmake-mode' instead of `makefile-mode' because the
former also highlights ifdef et al. while the latter does not.

"./Makefile.global" and one "Makefile.local" failed to specify any
major mode at all but doing so is necessary because Emacs does not
automatically figure out that these are Makefiles (of any flavor).
2020-08-09 21:14:36 -03:00
Jonas Bernoulli
e1a700067a emacs: Use 'when' instead of 'if' when there is no ELSE part 2020-08-09 20:52:34 -03:00
Jonas Bernoulli
09f6533c37 emacs: Use 'unless' instead of 'when' and 'not'
Also use 'unless' in a few cases where previously 'if' was used with
'not' but without an ELSE part.
2020-08-09 20:51:26 -03:00
Jonas Bernoulli
caaa108760 emacs: Fix indentation 2020-08-09 20:48:09 -03:00
Đoàn Trần Công Danh
f5ae8040b3 T355: specify hash algorithm explicitly
On some systems (notably, the one shipped with LibreSSL),
default fingerprint digest algorithm is SHA256.

On other systems, users can change default digest algorithm by changing
default_md in /etc/ssl/default_md.

Let's ask openssl to provide us specific algorithm to make the test
more deterministic.

Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
2020-08-08 16:04:27 -03:00
David Bremner
0e4695abaa test: regression tests for n_indexopts_{get,set}_decrypt_policy
The main criteria is that they don't crash. Working with a closed
database is a bonus.
2020-08-03 21:05:29 -03:00
David Bremner
e3f88436b7 test: regression test for traversing config list with closed db
Also mention error return in API docs
2020-08-03 21:03:43 -03:00
David Bremner
a0b2a54aa7 test: regression test for n_d_get_config_list on closed db.
Exception is caught.
2020-08-03 21:01:53 -03:00
David Bremner
e8ffbf84ff test: regression test for n_directory_{get,set}_mtime
The mtime is cached, so closing the db is not a problem. Writing the
mtime throws an exception, which is caught.
2020-08-03 21:00:25 -03:00
David Bremner
f4f5db0f1a lib: fix return value for n_directory_delete
Falling out of the catch meant the error return was lost
2020-08-03 20:59:06 -03:00
David Bremner
b03cc6cf90 test: known broken test for n_directory_delete with closed db.
There is a return value bug in notmuch_directory_delete that is hiding
the exception.
2020-08-03 20:57:43 -03:00
David Bremner
337665e26a lib: catch exceptions in n_directory_get_child_files
Also clarify API in error case.
2020-08-03 20:56:09 -03:00
David Bremner
3890d98966 test: add known broken test for n_directory_get_child_files
This is a clone of the one for get_child_directories
2020-08-03 20:54:26 -03:00
David Bremner
9668583a49 lib: catch exceptions in n_directory_get_child_directories
Also clarify API in error case.
2020-08-03 20:53:04 -03:00
David Bremner
919b89156c test: add known broken test for n_directory_get_child_directories
Start a new test file (for the notmuch_directory_* API group) to hold
this test.
2020-08-03 20:51:40 -03:00
David Bremner
c151506773 test: regression test for n_m_get_filenames
Closing the database after the iterator is created is not a problem.
2020-08-03 20:50:15 -03:00
David Bremner
2b6e73d895 test: split header for lib-message tests.
This allows finer control over when to close the database.
2020-08-03 20:48:51 -03:00
David Bremner
e5e17926db test: add regression test for n_messages_collect_tags
Also test n_messages_destroy.
2020-08-03 20:47:18 -03:00
David Bremner
31dab9b1cd lib: return NULL from n_d_get_default_indexopts on error
This is a rare and probably serious programming error, so better not
to silently return a default value.
2020-08-03 20:45:58 -03:00
David Bremner
99a7aac8f2 test: drop use of db_ending
This will allow the dropping of the test for the default ending from
configure.
2020-08-01 20:27:30 -03:00
David Bremner
29a58ecf4a test: drop NOTMUCH_DEFAULT_XAPIAN_BACKEND from T360-symbol-test
Inspired by the suggestion of

         id:20200727154108.16269-3-congdanhqx@gmail.com

to drop the configuration test for the default backend. This version
is hopefully robust against backend changes.
2020-07-31 21:30:06 -03:00
David Bremner
00f48f397a test: destroy thread from closed database
Check for (non)-crash.
2020-07-31 07:41:00 -03:00
David Bremner
864f422f14 test: regression test for n_thread_get_tags
Code is taken from the API docs, with the twist that the underlying
database is closed. Not crashing is the main point.
2020-07-31 07:41:00 -03:00
David Bremner
43406e7d4c test: add regression tests for oldest and newest dates
These are strictly to watch for crashes, so don't be too fussy about
the actual timestamps.
2020-07-31 07:41:00 -03:00
David Bremner
c5017c76d9 test: add regression tests for n_t_get_{authors, subject}
This is returning explicitely cached data, so no database access is needed.
2020-07-31 07:41:00 -03:00
David Bremner
70b30066f6 test: add regression test for n_t_get_messages
This is similar to the case of toplevel messages. Currently everything
is cached, so no database access is necessary. This might change in
the future, but it should not crash in either case.
2020-07-31 07:41:00 -03:00
David Bremner
6ccc4338a4 test: add regression tests for n_thread_get_toplevel_messages
Include a test for the previously omitted non-trivial code path for
notmuch_thread_get_replies.
2020-07-31 07:41:00 -03:00
David Bremner
02ab473115 test: add regression test for n_thread_get_total_{messages,files}
This is returning cached info, so does not need to access the (closed)
database.
2020-07-31 07:41:00 -03:00
David Bremner
0baa581e6e test: regression test for n_t_get_thread_id
This is just cached data, so as long as we don't prematurely free
anything, it should be fine.
2020-07-31 07:41:00 -03:00
David Bremner
46468baa5a test: regression test for notmuch_query_destroy
As with other void API entries, not crashing counts as success.
2020-07-29 12:20:41 -03:00
David Bremner
451c0ce3fc test: regression tests for n_q_count_{messages, threads}
At least these exceptions are caught.
2020-07-29 12:20:30 -03:00
David Bremner
0a4c6cdeb1 test: regression test for n_q_search_messages
Exception handling matches notmuch_query_search_threads, at least for
this case.
2020-07-29 12:20:22 -03:00
David Bremner
25fc8662ed test: regression test for n_q_search_threads
At least this exception is caught.
2020-07-29 12:20:14 -03:00
David Bremner
4130b4e514 test: add regression test for n_q_add_tag_exclude
This relies on the change to not tear down the auxilary Xapian
objects, in particular the query parser, when the database is closed.
2020-07-29 12:20:06 -03:00
David Bremner
d5c84c8289 test: regression test for n_q_{set, get}_sort
We just want to make sure accessing the query owned by a closed db
does not cause a crash.
2020-07-29 12:19:57 -03:00
David Bremner
a292461e65 test: regression test for set_omit_excluded 2020-07-29 12:19:48 -03:00
David Bremner
768efe88d0 test: regression test for notmuch_query_get_database
Current functionality is too trivial to really justify a test, but
it's simpler just to test the complete API.
2020-07-29 12:19:39 -03:00
David Bremner
cbf9f890aa test: add regression test for n_q_{create,_get_query_string}
Start a new file of tests, to keep to the (emerging) scheme of one
notmuch_foo group per file
2020-07-29 12:19:09 -03:00
David Bremner
85da95e266 test: move notmuch_message_* tests to their own file
This is for consistency with the recently added tests for
notmuch_database_*.
2020-07-29 12:18:11 -03:00
Jonas Bernoulli
8b162b40f8 test: Explicitly state that we want to sign with sender
Since Emacs 27 'mml-secure-epg-sign' errors out if we don't opt-in to
signing as the sender using 'mml-secure-openpgp-sign-with-sender'.
2020-07-28 08:40:30 -03:00
Jonas Bernoulli
7b752e9eb4 test: Remove misguided emacs testing utilities
The goal of this abstraction was to save space.  But that failed as
the result actually was that four trivial lines got replace with 15
fairly complicated lines.  The opposite of what it was supposed to
do.

Also it made it harder to come up with the fix in the previous commit;
simply grepping for the relevant symbols did not work because they get
constructed at run-time instead of appearing in the source file.
2020-07-28 08:39:58 -03:00
Jonas Bernoulli
d5d8846c32 test: Deal with Emacs 27 switching to lexical scope by default
Starting with Emacs 27 undeclared variables in evaluated interactive
code uses lexical scope.  This includes code passed with '--eval' as
we do in the Emacs tests, which also happen to assume dynamic scope.

- This can affect variables defined by libraries that we use.  We
  let-bind such variables to change the behavior of functions which we
  then call with these bindings in effect.  If these libraries are not
  loaded beforehand, then the bindings are lexical and fail to have
  the effect we intended.

  At this time only 'smtpmail' has to be loaded explicitly (for the
  variables let-bound in emacs_deliver_message and emacs_fcc_message).

  'message' doesn't have to be loaded explicitly, because loading
  'notmuch' (in 'run_emacs') already takes care of that, indirectly.

- Our own testing-only variables also have to be declared explicitly.
  We should have done that anyway, but because of how and where these
  variables are used it was very easy to overlook that (i.e. it isn't
  something the byte-compiler ever looks at).  Not so in Emacs 27
  anymore; here this oversight caused four tests to fail.

  The numeric values of these variables get incremented by functions
  that we add to hooks that are run by many tests, not just the tests
  where we actually inspect the value and therefore take care to let-
  bind the values to 0 before we begin.  The global values therefore
  have to be numeric values as well.  I have chosen -100 instead of 0
  as the default in case someone writes a test that inspects the value
  but forgets to let-bind the value.  I hope that the unusual negative
  value that one is going to see in such a case will help debugging
  the issue.
2020-07-28 08:39:44 -03:00
David Bremner
24cf4381b8 test: add known broken test for n_d_get_default_indexopts
Xapian exceptions are swallowed and turned into default return value.
2020-07-22 19:52:55 -03:00
David Bremner
b268c8c071 lib: fix error return bug with n_d_set_config.
The catch block either needs to return, or the function needs to
return "status". Choose the latter for consistency with
n_d_get_config.
2020-07-22 19:52:55 -03:00
David Bremner
f6d74be840 test: add known broken test for n_d_set_config
Error status is currently lost.
2020-07-22 19:52:55 -03:00
David Bremner
a5e13d5218 test: add regression test for n_d_get_config
Xapian exceptions seem to handled OK, at least for this case.
2020-07-22 19:52:55 -03:00
David Bremner
dfab190a40 lib: add regresion test for n_d_get_all_tags
Existing error handling seems adequate, if not ideal.
2020-07-22 19:52:55 -03:00
David Bremner
64aa896151 lib: add regression test for n_d_find_message_by_filename
At least this Xapian exception is caught. Make sure it stays that way.
2020-07-22 19:52:55 -03:00
David Bremner
ce360ce4c9 lib: rename _n_d_create to _n_d_find_or_create
The error message and name were confusing when called in some "read
only" context.
2020-07-22 19:52:55 -03:00
David Bremner
97c887282a test: add known broken test for notmuch_database_remove_message
The current error message is a bit confusing; fix in next commit.
2020-07-22 19:52:55 -03:00
David Bremner
c477d7ce31 lib: convert relative filenames to absolute in n_d_index_file
The API docs promise to handle relative filenames, but the code did
not do it.

Also check for files outside the mail root, as implied by the API
description.

This fixes the bug reported at

     id:87sgdqo0rz.fsf@tethera.net
2020-07-22 19:52:55 -03:00
David Bremner
be3f4aec3f test: add known broken test for indexing relative path
The current code seems to look for the path relative to the current
working directory, rather than the mail store root.
2020-07-22 19:52:55 -03:00
David Bremner
9ab3e2e77d test: regression test for n_d_index_file closed db
Current exception handling seems OK, at least for this case.
2020-07-22 19:52:55 -03:00
David Bremner
1ca805a962 test: add regression test for n_d_get_directory
At least this exception is already handled correctly.
2020-07-22 19:52:55 -03:00
David Bremner
959cb4b7a7 test: regression test for n_d_get_revision
This function only accesses data cached by notmuch, so being closed is
not a problem.
2020-07-22 19:52:55 -03:00
David Bremner
bcb02dd8fc test: add regression test for n_d_{begin,end}_atomic
Xapian currently succeeds to begin/end a transaction on a closed database,
or at least does not throw an exception. Make the test robust against
this changing.
2020-07-22 19:52:55 -03:00
David Bremner
3f121d636e test: regression test for n_d_upgrade
The logic is that if it's acceptable to return SUCCESS for read only
database, it's acceptable for a closed one.
2020-07-22 19:52:55 -03:00
David Bremner
f25fc8e211 lib/n_d_needs_upgrade: handle error return from n_d_get_version
Also clarify documentation of error return from n_d_needs_upgrade.
2020-07-22 19:52:55 -03:00
David Bremner
cf90431cbf test: add known broken test for n_d_needs_upgrade
It's a bit arbitrary which value to return for errors, but the same
argument as for read only databases applies for errors.
2020-07-22 19:52:55 -03:00
David Bremner
37dc6045e4 test: add regression tests for notmuch database destroy
Either an open or closed database should be ok to destroy
2020-07-22 19:52:55 -03:00
David Bremner
964e783fd4 test: regression test for closing a closed database
This does not currently throw an error, and it should stay that way.
2020-07-22 19:52:55 -03:00
David Bremner
ab45654192 lib/n_d_get_version: catch exceptions and clarify the API
notmuch_database_get_version previously returned 0 on some errors, but
did not document this. Luckily 0 is not a valid database version.
2020-07-22 19:52:55 -03:00
David Bremner
920dc56e60 test: add known broken test for n_d_get_version on closed db
This should not crash, but it does currently.
2020-07-22 19:52:55 -03:00
David Bremner
f3a56b5f6d test: regression tests for n_d_status_string and n_d_get_path
These do not crash on a closed database, and we want to keep it that
way.

Start a new file of tests as T560-lib-error was starting to get unwieldy.
2020-07-22 19:52:55 -03:00
David Bremner
e9867b818b lib: fix exception messages for n_m_message_*
The original generic handler had an extra '%s' in the format
string. Update tests that failed to catch this because the template to
print status strings checked 'stat', which was not set.
2020-07-22 19:52:55 -03:00
David Bremner
1a8060b81f test: sanitize line numbers in exception reports
Prevent test suite churn when reported line numbers change.
2020-07-22 19:52:55 -03:00
David Bremner
765ca7bc08 lib: fix return value for n_m_reindex
Also update the documentation for the behaviour of n_m_get_thread_id
that this fix relies on.
2020-07-20 08:54:42 -03:00
David Bremner
a0c83e1fac test: add known broken test for n_m_reindex on closed db
This is another case where the code should not call INTERNAL_ERROR.
2020-07-20 08:54:42 -03:00
David Bremner
a2279c322e test: regression for retrieving closed db from message
This is actually one of the few potentially useful things you can do
with a message belonging to a closed database, since in principle you
could re-open the database.
2020-07-20 08:54:42 -03:00
David Bremner
1578e2af6a test: regression test for destroying message with closed db
This should be fine because the message belongs to the
database (talloc context wise).
2020-07-20 08:54:42 -03:00
David Bremner
9244f58860 test: regression tests of n_m_freeze and n_m_thaw on closed db
Neither of these accesses the database, so should be safe. Add the
tests to catch any changes in exception throwing.
2020-07-20 08:54:42 -03:00
David Bremner
a2b90dc084 lib: handle xapian exception in n_m_remove_all_tags
At least the exception we already catch should be reported properly.
2020-07-20 08:54:42 -03:00
David Bremner
a4ac9941a8 test: add broken test for n_m_remove_all_tags
The Xapian exception is actually caught here, but the NULL return is
not dealt with properly.
2020-07-20 08:54:42 -03:00
David Bremner
e32f66d0e3 test: add regression test for n_m_maildir_flags_to_tags
This function currently catches at least the obvious Xapian exceptions
and we want to keep it that way.
2020-07-20 08:54:42 -03:00
David Bremner
b7572ceb14 lib: add notmuch_message_has_maildir_flag_st
Initially the new function is mainly tested indirectly via the
wrapper.
2020-07-20 08:54:42 -03:00
David Bremner
b21f0fcb6a test: add regression test for notmuch_message_has_maildir_flag
This passes the NULL return inside _ensure_maildir_flags does not
break anything. Probably this should be handled more explicitely.
2020-07-20 08:45:15 -03:00
David Bremner
2d04ed2631 lib: catch exceptions in n_m_get_flag, provide n_m_get_flag_st
It's not very nice to return FALSE for an error, so provide
notmuch_message_get_flag_st as a migration path.

Bump LIBNOTMUCH_MINOR_VERSION because the API is extended.
2020-07-18 09:52:27 -03:00
David Bremner
aa8e3f4487 lib: catch Xapian exceptions in n_m_remove_tag
The churn here is again mainly re-indentation.
2020-07-14 07:31:45 -03:00
David Bremner
0fc769aa4f test: add broken test for n_m_remove_tag
Exception will be caught in next commit.
2020-07-14 07:31:45 -03:00
David Bremner
33dd5fdc69 lib: catch Xapian exceptions in n_m_add_tag
This is mostly just (horizontal) code movement due to wrapping
everything in a try / catch.
2020-07-14 07:31:45 -03:00
David Bremner
503c035077 test: add known broken test for n_m_add_tag with closed db
Exception will be caught in next commit.
2020-07-14 07:31:45 -03:00
David Bremner
96befd0dd0 lib: catch Xapian exceptions in n_m_count_files
This will require some care for the caller to check the sign, and not
just add error returns into a running total.
2020-07-14 07:31:37 -03:00
David Bremner
6dc62d8bea test: add known broken test for n_m_count_files
This will be fixed in the next commit.
2020-07-14 07:31:30 -03:00
David Bremner
00f1abfdf4 lib: catch Xapian exceptions in n_m_get_tags
This allows the function to return an error value rather than
crashing.
2020-07-14 07:12:52 -03:00
David Bremner
d284dc00da test: add known broken test for n_m_get_tags
This will be fixed in the next commit.
2020-07-14 07:12:52 -03:00
David Bremner
6eaadb43ad lib: add regression test for n_m_get_date; clarify API
This function catches Xapian exceptions. The test is intended to make
sure it stays that way.
2020-07-14 07:12:52 -03:00
David Bremner
ef15946def test: add known broken for n_m_get_flag on closed db
Exception caught in next commit. Note that FLAG_GHOST is the only one
that triggers the I/O code path.
2020-07-13 07:19:22 -03:00
David Bremner
286161b703 lib: catch exceptions in n_m_get_filenames
This is essentially copied from the change to notmuch_message_get_filename
2020-07-13 07:19:22 -03:00
David Bremner
2c2ba7d6e2 test: add known broken test for n_m_get_filenames
This will be fixed in the next commit
2020-07-13 07:19:22 -03:00
David Bremner
a606cba32b lib/n_m_g_filename: catch Xapian exceptions, document NULL return
This is the same machinery as applied for

     notmuch_message_get_{thread,message}_id
2020-07-13 07:19:22 -03:00
David Bremner
46e8076281 lib: add known broken test for notmuch_message_get_filename
This will be fixed in the next commit
2020-07-13 07:19:22 -03:00
David Bremner
a962bd2bf8 lib/n_m_get_replies: doc return, initial regression test
We need to to set a query and retrieve the threads to meaningfully
test this function.
2020-07-13 07:19:22 -03:00
David Bremner
056794a60d test: add regression test for n_m_get_header
This function already catches Xapian exceptions, and we want to make
sure it stays that way.
2020-07-13 07:19:22 -03:00
David Bremner
cca551a736 test: drop use of assert in closed db tests
Instead of printing the same static string for each test, can replace
the assert with something simpler (or at least easier to integrate
into the test suite).
2020-07-13 07:19:22 -03:00
David Bremner
a280d2a1a1 test: remove unused backup_database calls
Since these backups are never restored, they should be safe to remove.
2020-07-13 07:19:22 -03:00
David Bremner
ee897cab8b test: drop upgrade from v1 tests
These are less crucial since we stopped generating new database
versions and relied primarily on features. They also rely on a
pre-generated v1 database which happens to be chert format. This
backend is not supported by Xapian 1.5.

Also drop the tool gen-testdb.sh, which is currently broken, due to
changes in the testing infrastructure.
2020-07-11 17:20:09 -03:00
David Bremner
dbdb860bb9 lib/message: catch exception in n_m_get_thread_id
This allows us to return an error value from the library.
2020-07-03 21:04:43 -03:00
David Bremner
bb51f3aa29 test: add known broken test for n_m_get_thread_id on closed db
This will be fixed in the next commit.
2020-07-03 21:04:02 -03:00
David Bremner
87d462a204 lib: catch error from closed db in n_m_get_message_id
By catching it at the library top level, we can return an error value.
2020-07-03 21:03:51 -03:00
David Bremner
2c17327ee5 test: add known broken test for error handling on closed database
Based on id:87d05je1j6.fsf@powell.devork.be
2020-07-03 21:01:39 -03:00
David Bremner
e9f8ffd882 test: fix python 3.8.4 related regression in T210-raw
It seems (at least in 3.8.4~rc1-1 on Debian) that set_content requires
at least one line.
2020-07-03 07:23:32 -03:00
Daniel Kahn Gillmor
18a1522948 tests: mark sig verification known-broken with session keys on buggy gpgme
We make use of the just-introduced configure test.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-07-02 21:23:28 -03:00
David Bremner
b46d842782 test: mark two tests broken on machines with 32 bit time_t
I haven't traced the code path as exhaustively for the SMIME test, but
the expiry date in question is larger then representable in a signed
32 bit integer.
2020-06-26 22:16:51 -03:00
Daniel Kahn Gillmor
b624b406ff tests/ruby: Ensure that test works for out-of-tree builds 2020-05-31 13:52:33 -03:00
Daniel Kahn Gillmor
ef0ab496b3 python-cffi: enable out-of-tree builds
This is a simple hack to enable out-of-tree builds, a concern raised
by Tomi in id:m24kzjib9a.fsf@guru.guru-group.fi

This change at least enables "make check" to complete without error,
but I'm sure it could be improved.  I am not expert enough in
setuptools to know how.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Amended by db per id:87d06usa31.fsf@powell.devork.be
2020-05-30 12:42:14 -03:00
Tomi Ollila
c9e55a712e test/test-lib.sh: fix two out of tree test issues
json_check_nodes.py exists in source tree, not in out of tree
build tree. Added -B to the execution so source tree is not
"polluted" by a .pyc file when json_check_nodes.py is executed.

When creating run_emacs.sh make it load .elc files from out of
tree build tree, not from source tree if such files existed.
If existed, those may be outdated, or even created by some other
emacs than the one that was used to build .elc files in out of
tree build dir.
2020-05-30 12:42:14 -03:00
Jonas Bernoulli
291ef68ede emacs: Use dolist' instead of mapcar' for side-effects
As recommended by the byte-compiler.
2020-05-26 20:23:14 -03:00
Daniel Kahn Gillmor
6cdf4b7e38 smime: Index cleartext of envelopedData when requested
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-05-22 22:12:00 -03:00
Daniel Kahn Gillmor
cb88b51fe5 smime: Pass PKCS#7 envelopedData to node_decrypt_and_verify
This change means we can support "notmuch show --decrypt=true" for
S/MIME encrypted messages, resolving several outstanding broken tests,
including all the remaining S/MIME protected header examples.

We do not yet handle indexing the cleartext of S/MIME encrypted
messages, though.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-05-22 22:11:51 -03:00
Daniel Kahn Gillmor
1b9f4a9863 cli/reply: Ignore PKCS#7 wrapper parts when replying
When composing a reply, no one wants to see this line in the proposed
message:

    Non-text part: application/pkcs7-mime

So we hide it, the same way we hide PGP/MIME cruft.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-05-22 22:11:25 -03:00
Daniel Kahn Gillmor
f12fb4d819 cli/show: If a leaf part has children, show them instead of omitting
Until we did PKCS#7 unwrapping, no leaf MIME part could have a child.

Now, we treat the unwrapped MIME part as the child of the PKCS#7
SignedData object.  So in that case, we want to show it instead of
deliberately omitting the content.

This fixes the test of the protected subject in
id:smime-onepart-signed@protected-headers.example.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-05-22 22:11:17 -03:00
Daniel Kahn Gillmor
5f4aceee26 cli: include wrapped part of PKCS#7 SignedData in the MIME tree
Unwrap a PKCS#7 SignedData part unconditionally when the cli is
traversing the MIME tree, and return it as a "child" of what would
otherwise be a leaf in the tree.

Unfortunately, this also breaks the JSON output.  We will fix that
next.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-05-22 22:11:07 -03:00
Daniel Kahn Gillmor
ad60e5d4e8 smime: Identify encrypted S/MIME parts during indexing
We don't handle them correctly yet, but we can at least mark them as
being encrypted.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-05-22 22:10:55 -03:00
Daniel Kahn Gillmor
38bd0df922 lib: index PKCS7 SignedData parts
When we are indexing, we should treat SignedData parts the same way
that we treat a multipart object, indexing the wrapped part as a
distinct MIME object.

Unfortunately, this means doing some sort of cryptographic
verification whose results we throw away, because GMime doesn't offer
us any way to unwrap without doing signature verification.

I've opened https://github.com/jstedfast/gmime/issues/67 to request
the capability from GMime but for now, we'll just accept the
additional performance hit.

As we do this indexing, we also apply the "signed" tag, by analogy
with how we handle multipart/signed messages.  These days, that kind
of change should probably be done with a property instead, but that's
a different set of changes.  This one is just for consistency.

Note that we are currently *only* handling signedData parts, which are
basically clearsigned messages.  PKCS#7 parts can also be
envelopedData and authEnvelopedData (which are effectively encryption
layers), and compressedData (which afaict isn't implemented anywhere,
i've never encountered it).  We're laying the groundwork for indexing
these other S/MIME types here, but we're only dealing with signedData
for now.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-05-22 22:10:46 -03:00
Daniel Kahn Gillmor
b14d9ae204 smime: tests of X.509 certificate validity are known-broken on GMime < 3.2.7
When checking cryptographic signatures, Notmuch relies on GMime to
tell it whether the certificate that signs a message has a valid User
ID or not.

If the User ID is not valid, then notmuch does not report the signer's
User ID to the user.  This means that the consumer of notmuch's
cryptographic summary of a message (or of its protected headers) can
be confident in relaying the reported identity to the user.

However, some versions of GMime before 3.2.7 cannot report Certificate
validity for X.509 certificates.  This is resolved upstream in GMime
at https://github.com/jstedfast/gmime/pull/90.

We adapt to this by marking tests of reported User IDs for
S/MIME-signed messages as known-broken if GMime is older than 3.2.7
and has not been patched.

If GMime >= 3.2.7 and certificate validity still doesn't work for
X.509 certs, then there has likely been a regression in GMime and we
should fail early, during ./configure.

To break out these specific User ID checks from other checks, i had to
split some tests into two parts, and reuse $output across the two
subtests.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-05-22 22:04:57 -03:00
Daniel Kahn Gillmor
627460d7bb test-lib: mark function variables as local
Several functions in test/test-lib.sh used variable names that are
also used outside of those functions (e.g. $output and $expected are
used in many of the test scripts), but they are not expected to
communicate via those variables.

We mark those variables "local" within test-lib.sh so that they do not
get clobbered when used outside test-lib.

We also move the local variable declarations to beginning of each
function, to avoid weird gotchas with local variable declarations as
described in https://tldp.org/LDP/abs/html/localvar.html.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-05-09 08:32:51 -03:00
David Bremner
45cfeb2e55 lib: replace STRNCMP_LITERAL in __message_remove_indexed_terms
strncmp looks for a prefix that matches, which is very much not what
we want here. This fixes the bug reported by Franz Fellner in
id:1588595993-ner-8.651@TPL520
2020-05-04 10:55:43 -03:00
David Bremner
accfee5c6e test: known broken test for reindex tag preservation
In id:1588595993-ner-8.651@TPL520 Franz Fellner reported that tags
starting with 'attachment' are removed by 'notmuch reindex'. This is
probably related to the use of STRNCMP_LITERAL in
_notmuch_message_remove_indexed_terms.
2020-05-04 10:49:43 -03:00
Daniel Kahn Gillmor
9055dfdae4 tests: disable CRL checks from gpgsm
GPGME has a strange failure mode when it is in offline mode, and/or
when certificates don't have any CRLs: in particular, it refuses to
accept the validity of any certificate other than a "root" cert.

This can be worked around by setting the `disable-crl-checks`
configuration variable for gpgsm.

I've reported this to the GPGME upstream at
https://dev.gnupg.org/T4883, but I have no idea how it will be
resolved.  In the meantime, we'll just work around it.

Note that this fixes the test for verification of
id:smime-multipart-signed@protected-headers.example, because
multipart/signed messages are already handled correctly (one-part
PKCS#7 messages will get fixed later).

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-04-30 18:02:38 -03:00
Daniel Kahn Gillmor
b415ec06c3 test/protected-headers: Add tests for S/MIME protected headers
Recognize the protected subject for S/MIME example protected header
messages.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-04-30 17:57:37 -03:00
Daniel Kahn Gillmor
b1a04bddc2 tests/smime: add tests for S/MIME SignedData
Add a simple S/MIME SignedData message, taken from an upcoming draft
of
https://datatracker.ietf.org/doc/draft-autocrypt-lamps-protected-headers/

RFC 8551 describes a SignedData, a one-part clearsigned object that is
more resistant to common patterns of MTA message munging than
multipart/signed (but has the downside that it is only readable by
clients that implement S/MIME).

To make sure sure notmuch can handle this kind of object, we want to
know a few things:

Already working:

 - Is the content of the SignedData object indexed?  It actually is
   right now because of dumb luck -- i think we're indexing the raw
   CMS object and it happens to contain the cleartext of the message
   in a way that we can consume it before passing it on to Xapian.
 - Are we accidentally indexing the embedded PKCS#7 certificates? We
   don't want to, and for some reason I don't understand, our indexing
   is actually skipping the embedded certificates already.  That's
   good!

Still need fixing:
 - do we know the MIME type of the embedded part?
 - do we know that the message is signed?
 - can notmuch-show read its content?
 - can notmuch-show indicate the signature validity?
 - can notmuch-reply properly quote and attribute content?

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-04-30 17:57:26 -03:00
Daniel Kahn Gillmor
488e91f42b test-lib.sh: add test_valid_json
This test does exactly what it says on the tin.  It expects JSON data
to be parseable by Python, at least.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-04-30 17:57:16 -03:00
Daniel Kahn Gillmor
92149485cb tests/smime: Test indexing cleartext of envelopedData
These tests describe some simple behavior we would expect to work if
we were to correctly index the cleartext of encrypted S/MIME messages
(PKCS#7 envelopedData).

Of course, they don't currently pass, so we mark them known-broken.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-04-30 17:56:13 -03:00
Daniel Kahn Gillmor
a8bf94af44 tests/smime: Verify cryptographic message status
When consuming a signed+encrypted S/MIME message generated by emacs,
we expect to see the same cryptographic properties for the message as
a whole.  This is not done correctly yet, so the test is marked as
known broken.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-04-30 17:56:02 -03:00
Daniel Kahn Gillmor
482af5a031 tests: Add S/MIME messages to protected-headers corpus
These sample messages are taken directly from the Protected Headers
draft:

https://www.ietf.org/id/draft-autocrypt-lamps-protected-headers-02.html

Note that this commit doesn't strictly pass the common git pre-commit
hook due to introducing some trailing whitespace.  That's just the
nature of the corpus, though.  We should have that trailing
whitespace, so I've made this commit with --no-verify.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-04-30 17:55:19 -03:00
Daniel Kahn Gillmor
7c7cebffe6 tests/smime: include secret key material for Bob
This is taken from the same Internet Draft that test/smime/ca.crt
comes from.  See that draft for more details.
https://www.ietf.org/id/draft-dkg-lamps-samples-02.html#name-pkcs12-object-for-bob

We don't use it yet, but it will be used to decrypt other messages in
the test suite.

Note that we include it here with an empty passphrase, rather than
with the passphrase "bob" that it is supplied with in the I-D.  The
underlying cryptographic material is the same, but this way we can
import cleanly into gpgsm without having a passphrase set on it (gpgsm
converts an empty-string passphrase into no passphrase at all on
import).

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-04-30 17:55:11 -03:00
Daniel Kahn Gillmor
51c43d0e92 test: Allow tests to have both gpg and gpgsm active at once
Without this fix, we couldn't run both add_gnupg_home and
add_gpgsm_home in the same test script.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-04-30 17:54:58 -03:00