This highlights a bug reported by several users, including
Mohsin Kaleem [1].
The inconsistent use of test_begin_subtest_known_broken is because
some of these tests pass even though the database cannot be
located. This problem is left for a future commit.
[1]: id:87bl9lx864.fsf@kisara.moe
Test numbers are a concise way to communicate about tests and to remeber
them. Currently, there is one pait of duplicates:
T590-libconfig.sh
T590-thread-breakage.sh
Renumber the latter one to 592 since this keeps the alphabetic order and
leaves room in between.
Signed-off-by: Michael J Gruber <git@grubix.eu>
This is more efficient than notmuch-show-only-matching-messages, since
we do not parse the potentially large thread structure to find a
single message.
This is only a partial fix for notmuch-tree view, because displaying
the thread structure in the tree-mode window still crashes on long
threads. It is however enough to make unthreaded view handle long
threads.
This change addresses two known issues with large sets of changes to
the database. The first is that as reported by Steven Allen [1],
notmuch commits are not "flushed" when they complete, which means that
if there is an open transaction when the database closes (or e.g. the
program crashes) then all changes since the last commit will be
discarded (nothing is irrecoverably lost for "notmuch new", as the
indexing process just restarts next time it is run). This does not
really "fix" the issue reported in [1]; that seems rather difficult
given how transactions work in Xapian. On the other hand, with the
default settings, this should mean one only loses less than a minutes
worth of work. The second issue is the occasionally reported "storm"
of disk writes when notmuch finishes. I don't yet have a test for
this, but I think committing as we go should reduce the amount of work
when finalizing the database.
[1]: id:20151025210215.GA3754@stebalien.com
When the certificate that signs a message is known to be valid, GMime
is capable of reporting on the e-mail address embedded in the
certificate.
We pass this information along to the caller of "notmuch show", as
often only the e-mail address of the certificate has actually been
checked/verified.
Furthermore, signature verification should probably at some point
compare the e-mail address of the caller against the sender address of
the message itself. Having to parse what gmime thinks is a "userid"
to extract an e-mail address seems clunky and unnecessary if gmime
already thinks it knows what the e-mail address is.
See id:878s41ax6t.fsf@fifthhorseman.net for more motivation and discussion.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
This is a fix for the test failures reported by Dan Čermák [1].
It is more robust to check for the prerequisite inside the function
that uses it, rather than in every test file that calls the function.
[1]: id:87k0n4fqgm.fsf@tethera.net
notmuch-test will now call aggregate-results.sh with file list
that it compiles based on the test ran, and aggregate-results
will report failure is any of the test files are missing.
With this notmuch-test no longer has to exit in non-parallel
run if some test fail to write its report file -- so it works
as parallel tests in this sense.
Changed test_done() in test-lib.sh write report file in one write(2),
so there is (even) less chance it being partially written. Also,
now it writes 'total' last and aggregate-results.sh expects this
line to exist in all report files for reporting to be successful.
Added 'set -eu' to notmuch-test and modified code to work with
these settings. That makes it harder to get mistakes slipped
into committed code.
Gmane web interface is long gone, remove it. Make MARC the new
default. Update LKML to Lore, where it already redirects anyway. Also
add Notmuch web archive.
Sourcing test-lib.sh will cd to TMP_DIRECTORY, so
relative path in $0 will not work in previous version
. $(dirname "$0")/test-lib-emacs.sh
Now individual test scripts -- e.g. ./test/T310-emacs.sh
will work.
Prior to 9ad19e4454 there was an unhandled Xapian exception when
reindexing after a large number of deletes. This test was used for
bisection, and will subsequently serve as a regression test.
say_color() used to call (builtin) printf (and tput(1) to stdout)
several times, which caused attempts to write messages with color
to have partial content (e.g. escape sequences) often intermixed
with other tests when parallel tests were run.
Now, with all output collected, then written out using one
printf, all strings with color print out correctly
((at least short) write(2)'s appear to write out "atomically").
While at it, used only one tput(1) execution to determine whether
color output works, and made bold/colors/sgr0 to tput(1) their
values once per test.
notmuch_passwd_sanitize() in test-lib.sh is too generic, it cannot
work in many cases...
The more specific version _libconfig_sanitize() replaces it in
T590-libconfig.sh and the code that uses it is modified to output
the keys (ascending numbers printed in hex) so the sanitizer knows
what to sanitize in which lines...
"@" + fqdn -> "@FQDN" replacement is used as fqdn could
-- in theory -- be substring of 'USERNAME'.
'user -> 'USER_FULL_NAME replacement to work in cases where user
is empty -- as only first ' is replaced that works as expected.
In addition to ".(none)" now also ".localdomain" is filtered from
USERNAME@FQDN.
/dev/fd/{n} is not defined in posix, but it is portable enough
(if it weren't it is easy to fix -- now code looks clearer).
This prevents the message document getting multiple thread-id terms
when there are multiple files with the same message-id.
This change shifts some thread ids, requiring adjustments to other tests.
According to my bijection, this bug has been present since commit
411675a6ce in 2017. It is not completely clear what harm it causes in
regulary use, but it (at least) makes notmuch crash when compiled with
-DDEBUG_DATABASE_SANITY.
In test-lib-emacs.sh line 20:
test_require_external_prereq ${TEST_EMACS} || ret=1
^-----------^ SC2086: Double quote to prevent globbing and word splitting.
Did you mean:
test_require_external_prereq "${TEST_EMACS}" || ret=1
In test-lib-emacs.sh line 21:
test_require_external_prereq ${TEST_EMACSCLIENT} || ret=1
^-----------------^ SC2086: Double quote to prevent globbing and word splitting.
Did you mean:
test_require_external_prereq "${TEST_EMACSCLIENT}" || ret=1
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
Commit d59d9c81 (test: Make the emacsclient binary user-configurable,
2012-11-27) modified the prereq check for the configured emacsclient,
but we probably want to do the same for emacs itself.
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
Although this default worked for "notmuch config get", it didn't work
most other places. Restore the previous functionality, with the
wrinkle that XDG locations will shadow $HOME/mail if they exist.
This fixes a bug reported by Jack Kamm in id:87eeefdc8b.fsf@gmail.com
notmuch-before-tag-hook and notmuch-after-tag-hook are supposed to
have access to two dynamic variables, tag-changes and query, but these
were lost with the switch to lexical binding in fc4cda07 (emacs: use
lexical-bindings in all libraries, 2021-01-13).
Add a variant of Emacs's dlet (not available until Emacs 28) and use
it in notmuch-tag to expose tag-changes and query to the hooks.
Due to the change in the config system, notmuch keeps a notmuch database
open when it would not do so before. Consequently, it can miss changes
to the database which are done from a hook (while notmuch holds the
databse in read only mode). When notmuch itself writes to the database
after that it uses wrong assumptions about the last used doc id etc.
Demonstrate this by triggering an assertion. (This new test succeeds
with notmuch 0.31.4.)
Signed-off-by: Michael J Gruber <git@grubix.eu>
Amended-by: db. Check for both messages
Prior to 0.32, notmuch had the (undocumented) behaviour that it
expanded a relative value of database.path with respect to $HOME. In
0.32 this was special cased for database.path but broken for
database.mail_root, which causes problems for at least notmuch-new
when database.path is set to a relative path.
The change in T030-config.sh reflects a user visible, but hopefully
harmless behaviour change; the expanded form of the paths will now be
printed by notmuch config.
We always do test_expect_equal_file, so do it in test_ruby() directly.
The only subtest where we don't (get non-existent file) can be easily
modified.
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
Not much point in polluting the main library, and also will be useful to
modify it in tandem with the tests.
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
test_emacs may update the external prereqs, in which case we want to
skip the test rather than fail.
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
When the external prereqs are updated inside the body of the command
(e.g. test_emacs) the message in test_report_skip_ is wrong: it outputs
the body of the command instead of the subtest name.
We need to pass the same argument we pass to test_skip.
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
My fqdn is 'natae.localdomain', however, socket.getfqdn() returns
'localhost'.
To fetch the true fqdn we need socket.getaddrinfo().
For more information see: https://stackoverflow.com/a/11580042/10474
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
If any of the variables is empty the output is completely messed up,
because replace("", "FOO") puts "FOO" before every single character.
I don't have my full name configured, and this is what I get:
USER_FULL_NAME=USER_FULL_NAME=USER_FULL_NAME USER_FULL_NAMEsUSER_FULL_NAMEtUSER_FULL_NAMEdUSER_FULL_NAMEoUSER_FULL_NAMEuUSER_FULL_NAMEtUSER_FULL_NAME USER_FULL_NAME=USER_FULL_NAME=USER_FULL_NAME
Let's check for empty strings before doing any replace.
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
The lastest versions of GNU parallel no longer make mention of GNU
within their help output. This causes the test script to mistakenly use
the moreutils parallel execution. In order to fix this, while
maintaining compatibility with previous versions of GNU parallel,
--version should be used.
Signed-off-by: Tobias Backer Dirks <omgitsaheadcrab@gmail.com>
Apparently the -f option to hostname is not portable, and in fact it
does not seem to always behave reasonably in e.g. a chroot.
Python code originally due to Tomi [1], modified by yours truly.
[1]: id:m2lf9fbkug.fsf@guru.guru-group.fi
lib/open.cc:_load_key_file will only open xdg-config files in
$XDG_CONFIG_HOME if it's defined, $HOME/.config will be considered if
and only if XDG_CONFIG_HOME not defined.
Let's unset said variable before running the test.
Certain tools like the address-sanitizer fail if they are not the
first LD_PRELOADed library. It does not seem to matter for our shims,
as long as they are loaded before libnotmuch.
This will allow simplifying the subcommand interface.
Change the internal API to notmuch_config_open to not tie it to the
implementation of subcommands in notmuch.c.
It also fixes a previously broken test, since notmuch_config_open does
not understand the notion of the empty string as a config file name.
There are two small code changes. The main one is to retrieve the
possibly updated config file name found during the database opening
call. The second change is to allow empty config file names, as
a (currently broken) way of specifying that configuration should only
be taken from the database.
Eventually we want to do all opening of databases in the top
level (main function). This means that detection of missing databases
needs to move out of subcommands. It also requires updating the
library to use the new NO_DATABASE status code.
Previously the fact that some configuration options were only stored
in the database (and thus editing the config file had no effect) was a
source of user confusion. This was fixed with the series ending at
d9af0af164.
On the other hand, the underlying partition of config options into
those stored by default in the database and those stored in the config
file remained. This is also confusing, since now some invocations of
"notmuch config set" modify the config file, and others silently
modify the database instead.
With this commit, it is up to the user to decide which configuration
to modify. A new "--database" option is provided for notmuch config to
enable modifying the configuration information in the database;
otherwise the default is to update an external config file.
Use the database opened at the top level rather than opening another
notmuch_database_t.
Test output changes because keys are now listed in alphabetical order,
and because a missing database is no longer an error.
This commit starts the conversion of notmuch-config.c
functionality (as opposed to just interface) to the new config
framework.
The change to T030-config is because of the move of the
canonicalization database paths from the notmuch_config_t accessor to
the internal function _choose_database_path.
The layer of shims here seems a bit wasteful compared to just calling
the corresponding string map functions directly, but it allows control
over the API (calling with notmuch_database_t *) and flexibility for
future changes.
The stat is essentially replaced by the mkdir for error detection
purposes. This changes the default location for backups to make
things tidier, even in non-split configurations. Hopefully there is
not too many user scripts relying on the previous location.
Because the default location may not exist, replace the use of stat
for error detection with a call to mkdir.
Like the hook directory, we primarily need a way to communicate this
directory between various components, but we may as well let the user
configure it.
Most of the diff is generalizing choose_hook_dir to work for both
backup and hook directories.
Choose sibling directory of xapian database, as .notmuch may not
exist.
libgen.h is already used in debugger.c, so it is not a new dependency
/ potential portability problem.
The new test is in T055-path-config because it uses the helper
function split_config, and because it seems easier to put the
database path related tests in one place.
This changes some error reporting, either intentionally by reporting
the highest level missing directory, or by side effect from looking in
XDG locations when given null database location.
This adds new state variable for the mail root, and uses it most
places db_path was used. The notable exception is dumps from
backups. The latter will be dealt with properly in a future commit.
The main functionality will be tested when notmuch-new is converted to
support split configuration. Here only the somewhat odd case of split
mail root which is actually symlinked to the database path is tested.
Introduce a new configuration value for the mail root, and use it to
locate mail messages in preference to the database.path (which
previously implied the mail messages were also in this location.
Initially only a subset of the CLI is tested in a split
configuration. Further changes will be needed for the remainder of the
CLI to work in split configurations.
In order to open the database in main() for this command, we may need
to re-open it in the (possibly less common) case where crypto options
require write access.
This fixes a bug reported in [1]. In principle it could be possible
avoid one of these reopens, but it complicates the logic in main with
respect to creating new databases.
[1]: id:9C1993DF-84BD-4199-A9C8-BADA98498812@bubblegen.co.uk
Recent changes to configuration handling meant the pre-new hook was
run while the database was open read only, limiting what could be done
in the hook. Add some known broken tests for this problem, as well as
a regression test for write access from the post-new hook.
In the future Xapian will apparently support this more conveniently
for the cases other than READ_ONLY => READ_ONLY
Conceptually this function seems to fit better in lib/open.cc;
database.cc is still large enough that moving the function makes
sense.
In [1] Gregor Zattler explained the results of his hard work
tracking down a bug in notmuch with long directories. This test
duplicates the bug.
[1]: id:20210317194728.GB5561@no.workgroup
Fix use of $TEST_DIRECTORY ($NOTMUCH_BUILDDIR/test/) with use of
$TMP_DIRECTORY ($NOTMUCH_BUILDDIR/test/tmp.T020-compact/ in case
of T020-compact.sh) as root directory where to write test files
and directories.
The assignment of thread-ids is (apparently) non-deterministic in a
way that mostly seems to show up on multicore machines. In my tests
the number is different from that previously assumed by this test
about 15% of the time on a 50 thread (25 core) Xeon.
Since message id's are fixed, use a message known to be in the thread
of interest to pick out the correct thread-id.
This is the last bit of "python" left in the notmuch codebase.
https://www.python.org/dev/peps/pep-0394/#recommendation encourages
"third-party distributors" to use more-specific shebang lines. I'm
not certain that the notmuch project itself is a "third-party
contributor" but I think this is a safe way to encourage people to use
python3 when they're developing notmuch.
We already have python3 explicitly elsewhere in the codebase for
developers (in nmbug).
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
This enables support for hooks outside the database directory.
It relies strongly on configuration information being usable between
closing the database and destroying it.
The hook directory configuration needs to be kept in synch with the
other configuration information, so add scaffolding to support this at
database opening time.
In addition to the same type of changes as converting other
subcommands, add the possibility of creating a database at the top
level. It would probably make sense to use this for insert as well.
This will need some cleanup when the transition completes, and we stop
passing notmuch_config_t structs to the subcommands.
Unlike the general case, we open the database in the subcommand, since
we don't know whether it should be opened read/write until we parse
the command line arguments.
Add a test to make sure passing config file on the command line is not
broken by these or future config related changes.
Since we are already passing a search context around as a kind of
parameter block, add a new talloc context to that to replace relying
on 'config'.
Convert notmuch-search and notmuch-address at the same time, because
they share some code.
Add a test to make sure we don't break passing configuration as a
command line argument.
The new talloc context is needed to run the hook at the very end of
the function. That in turn is needed so that this process gives up the
write lock on the database.
This conversion is trivial because the only configuration information
accessed by dump is that stored in the database (in order to dump
it). We do need to be careful to keep the write lock on the database
to ensure dump consistency.
The main effort is changing from the old argv style config list
iterators to the new more opaque ones provided by the library (and
backed by the database+file config cache).
By using an enum we can have better error detection than copy pasting
key strings around.
The question of what layer this belongs in is a bit
tricky. Historically most of the keys are defined by the CLI. On the
other hand features like excludes are supported in the
library/bindings, and it makes sense to configure them from the
library as well.
The somewhat long prefix for notmuch_config_t is to avoid collisions
with the existing usage in notmuch-client.h.
Fill in the remainder of the documented functionality for
n_d_open_with_config with respect to config file location. Similar
searching default locations of the database file still needs to be
added.
The main goal is to allow configuration information to be temporarily
overridden by a separate config file. That will require further
changes not in this commit.
The performance impact is unclear, and will depend on the balance
between number of queries and number of distinct metadata items read
on the first call to n_d_get_config.
In ee897cab8b the upgrade tests from pre v3 databases were
removed. The reasons for that are still valid, but we should still
test the code paths that do the upgrade, and it is relatively
straightforward to do that for v3 to v3 upgrades.
`outline-minor-mode' treats comments that begin with three or more
semicolons as headings. That makes it very convenient to navigate
code and to show/hide parts of a file.
Elips libraries typically have four top-level sections, e.g.:
;;; notmuch.el --- run notmuch within emacs...
;;; Commentary:...
;;; Code:...
;;; notmuch.el ends here
In this package many libraries lack a "Commentary:" section, which is
not optimal but okay for most libraries, except major entry points.
Depending on how one chooses to look at it, the "... ends here" line
is not really a heading that begins a section, because it should never
have a "section" body (after all it marks eof).
If the file is rather short, then I left "Code:" as the only section
that contains code. Otherwise I split the file into multiple sibling
sections. The "Code:" section continues to contain `require' and
`declare-function' forms and other such "front matter".
If and only if I have split the code into multiple sections anyway,
then I also added an additional section named just "_" before the
`provide' form and shortly before the "...end here" line. This
section could also be called "Back matter", but I feel it would be
distracting to be that explicit about it. (The IMO unnecessary but
unfortunately still obligatory "... ends here" line is already
distracting enough as far as I am concerned.)
Before this commit some libraries already uses section headings, some
of them consistently. When a library already had some headings, then
this commit often sticks to that style, even at the cost inconsistent
styling across all libraries.
A very limited number of variable and function definitions have to be
moved around because they would otherwise end up in sections they do
not belong into.
Sections, including but not limited to their heading, can and should
be further improved in the future.
readelf on (at least) ppc64le sometimes generates some extension to
the Ndx name inside '[]'. Remove this output to allow our simple
column based parsing to work.
It turns out that using nm -P isn't as portable as hoped. In
particular with some ELF ABIs (e.g. ppc64 ELFv1), the desired symbols
end up in the data section instead of text.
The test is currently only functional on ELF based architectures, so I
think it's legit to depend on readelf instead of nm.
The switch to readelf has the advantage that we can explicitely ask
for all of the symbols with global visibility, rather than grepping
for notmuch. That seems a more robust approach since it will catch any
strangely named global symbols.
notmuch insert does not currently support passing a filename for the
input, so all of these tests have an extra error in addition to the
one being tested for.
Currently this does not make a difference because the error being
tested for is caught before the error of an extra command line
argument. In the future it might make a difference, and in any case it
is confusing.
Previously in message-show mode message's first header line (From
header) was always indented, even if user had turned thread
indentation off with "<" (notmuch-show-toggle-thread-indentation)
command.
This change modifies notmuch-show-insert-headerline function so that
it doesn't indent the first header line if notmuch-show-indent-content
variable is nil.
This change also modifies tests so that they expect this new output
format:
test/emacs-show.expected-output/notmuch-show-indent-thread-content-off
Use `makefile-gmake-mode' instead of `makefile-mode' because the
former also highlights ifdef et al. while the latter does not.
"./Makefile.global" and one "Makefile.local" failed to specify any
major mode at all but doing so is necessary because Emacs does not
automatically figure out that these are Makefiles (of any flavor).
On some systems (notably, the one shipped with LibreSSL),
default fingerprint digest algorithm is SHA256.
On other systems, users can change default digest algorithm by changing
default_md in /etc/ssl/default_md.
Let's ask openssl to provide us specific algorithm to make the test
more deterministic.
Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
Inspired by the suggestion of
id:20200727154108.16269-3-congdanhqx@gmail.com
to drop the configuration test for the default backend. This version
is hopefully robust against backend changes.
This is similar to the case of toplevel messages. Currently everything
is cached, so no database access is necessary. This might change in
the future, but it should not crash in either case.
The goal of this abstraction was to save space. But that failed as
the result actually was that four trivial lines got replace with 15
fairly complicated lines. The opposite of what it was supposed to
do.
Also it made it harder to come up with the fix in the previous commit;
simply grepping for the relevant symbols did not work because they get
constructed at run-time instead of appearing in the source file.
Starting with Emacs 27 undeclared variables in evaluated interactive
code uses lexical scope. This includes code passed with '--eval' as
we do in the Emacs tests, which also happen to assume dynamic scope.
- This can affect variables defined by libraries that we use. We
let-bind such variables to change the behavior of functions which we
then call with these bindings in effect. If these libraries are not
loaded beforehand, then the bindings are lexical and fail to have
the effect we intended.
At this time only 'smtpmail' has to be loaded explicitly (for the
variables let-bound in emacs_deliver_message and emacs_fcc_message).
'message' doesn't have to be loaded explicitly, because loading
'notmuch' (in 'run_emacs') already takes care of that, indirectly.
- Our own testing-only variables also have to be declared explicitly.
We should have done that anyway, but because of how and where these
variables are used it was very easy to overlook that (i.e. it isn't
something the byte-compiler ever looks at). Not so in Emacs 27
anymore; here this oversight caused four tests to fail.
The numeric values of these variables get incremented by functions
that we add to hooks that are run by many tests, not just the tests
where we actually inspect the value and therefore take care to let-
bind the values to 0 before we begin. The global values therefore
have to be numeric values as well. I have chosen -100 instead of 0
as the default in case someone writes a test that inspects the value
but forgets to let-bind the value. I hope that the unusual negative
value that one is going to see in such a case will help debugging
the issue.
The API docs promise to handle relative filenames, but the code did
not do it.
Also check for files outside the mail root, as implied by the API
description.
This fixes the bug reported at
id:87sgdqo0rz.fsf@tethera.net
Xapian currently succeeds to begin/end a transaction on a closed database,
or at least does not throw an exception. Make the test robust against
this changing.
The original generic handler had an extra '%s' in the format
string. Update tests that failed to catch this because the template to
print status strings checked 'stat', which was not set.
This is actually one of the few potentially useful things you can do
with a message belonging to a closed database, since in principle you
could re-open the database.
It's not very nice to return FALSE for an error, so provide
notmuch_message_get_flag_st as a migration path.
Bump LIBNOTMUCH_MINOR_VERSION because the API is extended.
Instead of printing the same static string for each test, can replace
the assert with something simpler (or at least easier to integrate
into the test suite).
These are less crucial since we stopped generating new database
versions and relied primarily on features. They also rely on a
pre-generated v1 database which happens to be chert format. This
backend is not supported by Xapian 1.5.
Also drop the tool gen-testdb.sh, which is currently broken, due to
changes in the testing infrastructure.
I haven't traced the code path as exhaustively for the SMIME test, but
the expiry date in question is larger then representable in a signed
32 bit integer.
This is a simple hack to enable out-of-tree builds, a concern raised
by Tomi in id:m24kzjib9a.fsf@guru.guru-group.fi
This change at least enables "make check" to complete without error,
but I'm sure it could be improved. I am not expert enough in
setuptools to know how.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Amended by db per id:87d06usa31.fsf@powell.devork.be
json_check_nodes.py exists in source tree, not in out of tree
build tree. Added -B to the execution so source tree is not
"polluted" by a .pyc file when json_check_nodes.py is executed.
When creating run_emacs.sh make it load .elc files from out of
tree build tree, not from source tree if such files existed.
If existed, those may be outdated, or even created by some other
emacs than the one that was used to build .elc files in out of
tree build dir.
This change means we can support "notmuch show --decrypt=true" for
S/MIME encrypted messages, resolving several outstanding broken tests,
including all the remaining S/MIME protected header examples.
We do not yet handle indexing the cleartext of S/MIME encrypted
messages, though.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
When composing a reply, no one wants to see this line in the proposed
message:
Non-text part: application/pkcs7-mime
So we hide it, the same way we hide PGP/MIME cruft.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Until we did PKCS#7 unwrapping, no leaf MIME part could have a child.
Now, we treat the unwrapped MIME part as the child of the PKCS#7
SignedData object. So in that case, we want to show it instead of
deliberately omitting the content.
This fixes the test of the protected subject in
id:smime-onepart-signed@protected-headers.example.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Unwrap a PKCS#7 SignedData part unconditionally when the cli is
traversing the MIME tree, and return it as a "child" of what would
otherwise be a leaf in the tree.
Unfortunately, this also breaks the JSON output. We will fix that
next.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
When we are indexing, we should treat SignedData parts the same way
that we treat a multipart object, indexing the wrapped part as a
distinct MIME object.
Unfortunately, this means doing some sort of cryptographic
verification whose results we throw away, because GMime doesn't offer
us any way to unwrap without doing signature verification.
I've opened https://github.com/jstedfast/gmime/issues/67 to request
the capability from GMime but for now, we'll just accept the
additional performance hit.
As we do this indexing, we also apply the "signed" tag, by analogy
with how we handle multipart/signed messages. These days, that kind
of change should probably be done with a property instead, but that's
a different set of changes. This one is just for consistency.
Note that we are currently *only* handling signedData parts, which are
basically clearsigned messages. PKCS#7 parts can also be
envelopedData and authEnvelopedData (which are effectively encryption
layers), and compressedData (which afaict isn't implemented anywhere,
i've never encountered it). We're laying the groundwork for indexing
these other S/MIME types here, but we're only dealing with signedData
for now.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
When checking cryptographic signatures, Notmuch relies on GMime to
tell it whether the certificate that signs a message has a valid User
ID or not.
If the User ID is not valid, then notmuch does not report the signer's
User ID to the user. This means that the consumer of notmuch's
cryptographic summary of a message (or of its protected headers) can
be confident in relaying the reported identity to the user.
However, some versions of GMime before 3.2.7 cannot report Certificate
validity for X.509 certificates. This is resolved upstream in GMime
at https://github.com/jstedfast/gmime/pull/90.
We adapt to this by marking tests of reported User IDs for
S/MIME-signed messages as known-broken if GMime is older than 3.2.7
and has not been patched.
If GMime >= 3.2.7 and certificate validity still doesn't work for
X.509 certs, then there has likely been a regression in GMime and we
should fail early, during ./configure.
To break out these specific User ID checks from other checks, i had to
split some tests into two parts, and reuse $output across the two
subtests.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Several functions in test/test-lib.sh used variable names that are
also used outside of those functions (e.g. $output and $expected are
used in many of the test scripts), but they are not expected to
communicate via those variables.
We mark those variables "local" within test-lib.sh so that they do not
get clobbered when used outside test-lib.
We also move the local variable declarations to beginning of each
function, to avoid weird gotchas with local variable declarations as
described in https://tldp.org/LDP/abs/html/localvar.html.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
strncmp looks for a prefix that matches, which is very much not what
we want here. This fixes the bug reported by Franz Fellner in
id:1588595993-ner-8.651@TPL520
In id:1588595993-ner-8.651@TPL520 Franz Fellner reported that tags
starting with 'attachment' are removed by 'notmuch reindex'. This is
probably related to the use of STRNCMP_LITERAL in
_notmuch_message_remove_indexed_terms.
GPGME has a strange failure mode when it is in offline mode, and/or
when certificates don't have any CRLs: in particular, it refuses to
accept the validity of any certificate other than a "root" cert.
This can be worked around by setting the `disable-crl-checks`
configuration variable for gpgsm.
I've reported this to the GPGME upstream at
https://dev.gnupg.org/T4883, but I have no idea how it will be
resolved. In the meantime, we'll just work around it.
Note that this fixes the test for verification of
id:smime-multipart-signed@protected-headers.example, because
multipart/signed messages are already handled correctly (one-part
PKCS#7 messages will get fixed later).
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Add a simple S/MIME SignedData message, taken from an upcoming draft
of
https://datatracker.ietf.org/doc/draft-autocrypt-lamps-protected-headers/
RFC 8551 describes a SignedData, a one-part clearsigned object that is
more resistant to common patterns of MTA message munging than
multipart/signed (but has the downside that it is only readable by
clients that implement S/MIME).
To make sure sure notmuch can handle this kind of object, we want to
know a few things:
Already working:
- Is the content of the SignedData object indexed? It actually is
right now because of dumb luck -- i think we're indexing the raw
CMS object and it happens to contain the cleartext of the message
in a way that we can consume it before passing it on to Xapian.
- Are we accidentally indexing the embedded PKCS#7 certificates? We
don't want to, and for some reason I don't understand, our indexing
is actually skipping the embedded certificates already. That's
good!
Still need fixing:
- do we know the MIME type of the embedded part?
- do we know that the message is signed?
- can notmuch-show read its content?
- can notmuch-show indicate the signature validity?
- can notmuch-reply properly quote and attribute content?
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
This test does exactly what it says on the tin. It expects JSON data
to be parseable by Python, at least.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
These tests describe some simple behavior we would expect to work if
we were to correctly index the cleartext of encrypted S/MIME messages
(PKCS#7 envelopedData).
Of course, they don't currently pass, so we mark them known-broken.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
When consuming a signed+encrypted S/MIME message generated by emacs,
we expect to see the same cryptographic properties for the message as
a whole. This is not done correctly yet, so the test is marked as
known broken.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
These sample messages are taken directly from the Protected Headers
draft:
https://www.ietf.org/id/draft-autocrypt-lamps-protected-headers-02.html
Note that this commit doesn't strictly pass the common git pre-commit
hook due to introducing some trailing whitespace. That's just the
nature of the corpus, though. We should have that trailing
whitespace, so I've made this commit with --no-verify.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
This is taken from the same Internet Draft that test/smime/ca.crt
comes from. See that draft for more details.
https://www.ietf.org/id/draft-dkg-lamps-samples-02.html#name-pkcs12-object-for-bob
We don't use it yet, but it will be used to decrypt other messages in
the test suite.
Note that we include it here with an empty passphrase, rather than
with the passphrase "bob" that it is supplied with in the I-D. The
underlying cryptographic material is the same, but this way we can
import cleanly into gpgsm without having a passphrase set on it (gpgsm
converts an empty-string passphrase into no passphrase at all on
import).
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Without this fix, we couldn't run both add_gnupg_home and
add_gpgsm_home in the same test script.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
No functional change.
We no longer need to identify the key and cert to mml-mode when
sending an S/MIME message, so making a copy of key+cert.pem to
test_suite.pem is superfluous. Get rid of the extra file.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
The documentation for message mode clearly states that EasyPG (which
uses GnuPG) is the default and recommended way to use S/MIME with
mml-secure:
[0] https://www.gnu.org/software/emacs/manual/html_node/message/Using-S_002fMIME.html
To ensure that this mode works, we just need to import the secret key
in question into gpgsm in addition to the public key. gpgsm should be
able pick the right keys+certificates to use based on To/From headers,
so we don't have to specify anything manually in the #secure mml tag.
The import process from the OpenSSL-preferred form (cert+secretkey) is
rather ugly, because gpgsm wants to see a PKCS#12 object when
importing secret keys.
Note that EasyPG generates the more modern Content-Type:
application/pkcs7-signature instead of application/x-pkcs7-signature
for the detached signature.
We are also obliged to manually set gpgsm's include-certs setting to 1
because gpgsm defaults to send "everything but the root cert". In our
weird test case, the certificate we're using is self-signed, so it
*is* the root cert, which means that gpgsm doesn't include it by
default. Setting it to 1 forces inclusion of the signer's cert, which
satisfies openssl's smime subcommand. See https://dev.gnupg.org/T4878
for more details.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
This CA is useful for test suites and the like, but is not an
actually-secure CA, because its secret key material is also published.
I plan to use it for its intended purpose in the notmuch test suite.
It was copied from this Internet Draft:
https://tools.ietf.org/id/draft-dkg-lamps-samples-01.html#name-certificate-authority-certi
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
GnuPG's gpgsm, like gpg, should always be used with --batch when it is
invoked in a non-interactive environment.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Starting with Emacs 27 the old `cl' implementation is finally
considered obsolete. Previously its use was strongly discouraged
at run-time but one was still allowed to use it at compile-time.
For the most part the transition is very simple and boils down to
adding the "cl-" prefix to some symbols. A few replacements do not
follow that simple pattern; e.g. `first' is replaced with `car',
even though the alias `cl-first' exists, because the latter is not
idiomatic emacs-lisp.
In a few cases we start using `pcase-let' or `pcase-lambda' instead
of renaming e.g. `first' to `car'. That way we can remind the reader
of the meaning of the various parts of the data that is being
deconstructed.
An obsolete `lexical-let' and a `lexical-let*' are replaced with their
regular variants `let' and `let*' even though we do not at the same
time enable `lexical-binding' for that file. That is the right thing
to do because it does not actually make a difference in those cases
whether lexical bindings are used or not, and because this should be
enabled in a separate commit.
We need to explicitly depend on the `cl-lib' package because Emacs
24.1 and 24.2 lack that library. When using these releases we end
up using the backport from GNU Elpa.
We need to explicitly require the `pcase' library because
`pcase-dolist' was not autoloaded until Emacs 25.1.
This test extracts values from a (key,value) map where multiple entries
can have the same key, and the entries are sorted by key, but not by
value. The test incorrectly assumes that the values will be sorted as
well, so sort the output.
Xapian 1.4 is over 3 years old now (1.4.0 released 2016-06-24),
and 1.2 has been deprecated in Notmuch version 0.27 (2018-06-13).
Xapian 1.4 supports compaction, field processors and retry locking;
conditionals checking compaction and field processors were removed
but user may want to disable retry locking at configure time so it
is kept.
'qsx' reported a bug on #notmuch with notmuch-dump and large stored
queries. This test will pass (on my machine) if the value of `repeat'
is made smaller.
Reported-By: Thomas Schneider <qsx@chaotikum.eu>
In particular, timestamps beyond 2038 could overflow the sprinter
interface on systems where time_t is 64-bit but 'int' is a signed 32-bit
integer type.
The entire python-cffi test suite is considered as a single test at
the level of the notmuch test suite. This might or might not be ideal,
but it gets them run.
We want freeing the returned stream to also free these underlying
objects. Compare tests/test-filters.c in the gmime 3.2.x source, which
uses this same idiom.
Thanks to James Troup for the report and the fix.
When showing or replying to a message that has been mangled in transit
by an MTA in the "Mixed up" way, notmuch should instead use the
repaired form of the message.
Tracking the repaired GMimeObject for the lifetime of the mime_node so
that it is cleaned up properly is probably the trickiest part of this
patch, but the choices here are based on the idea that the
mime_node_context is the memory manager for the whole mime_node tree
in the first place, so new GMimeObject tree created on-the-fly during
message parsing should be disposed of in the same place.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
When encountering a message that has been mangled in the "mixed up"
way by an intermediate MTA, notmuch should instead repair it and index
the repaired form.
When it does this, it also associates the index.repaired=mixedup
property with the message. If a problem is found with this repair
process, or an improved repair process is proposed later, this should
make it easy for people to reindex the relevant message. The property
will also hopefully make it easier to diagnose this particular problem
in the future.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Some MTAs mangle e-mail messages in transit in ways that are
repairable.
Microsoft Exchange (in particular, the version running today on
Office365's mailservers) appears to mangle multipart/encrypted
messages in a way that makes them undecryptable by the recipient.
I've documented this in section 4.1 "Mixed-up encryption" of draft -00
of
https://tools.ietf.org/html/draft-dkg-openpgp-pgpmime-message-mangling
Fortunately, it's possible to repair such a message, and notmuch can
do that so that a user who receives an encrypted message from a user
of office365.com can still decrypt the message.
Enigmail already knows about this particular kind of mangling. It
describes it as "broken PGP email format probably caused by an old
Exchange server", and it tries to repair by directly changing the
message held by the user. if this kind of repair goes wrong, the
repair process can cause data loss
(https://sourceforge.net/p/enigmail/bugs/987/, yikes).
The tests introduced here are currently broken. In subsequent
patches, i'll introduce a non-destructive form of repair for notmuch
so that notmuch users can read mail that has been mangled in this way,
and the tests will succeed.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
When we notice a legacy-display part during indexing, it makes more
sense to avoid indexing it as part of the message body.
Given that the protected subject will already be indexed, there is no
need to index this part at all, so we skip over it.
If this happens during indexing, we set a property on the message:
index.repaired=skip-protected-headers-legacy-display
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
