Commit graph

6984 commits

Author SHA1 Message Date
Daniel Kahn Gillmor
b1a04bddc2 tests/smime: add tests for S/MIME SignedData
Add a simple S/MIME SignedData message, taken from an upcoming draft
of
https://datatracker.ietf.org/doc/draft-autocrypt-lamps-protected-headers/

RFC 8551 describes a SignedData, a one-part clearsigned object that is
more resistant to common patterns of MTA message munging than
multipart/signed (but has the downside that it is only readable by
clients that implement S/MIME).

To make sure sure notmuch can handle this kind of object, we want to
know a few things:

Already working:

 - Is the content of the SignedData object indexed?  It actually is
   right now because of dumb luck -- i think we're indexing the raw
   CMS object and it happens to contain the cleartext of the message
   in a way that we can consume it before passing it on to Xapian.
 - Are we accidentally indexing the embedded PKCS#7 certificates? We
   don't want to, and for some reason I don't understand, our indexing
   is actually skipping the embedded certificates already.  That's
   good!

Still need fixing:
 - do we know the MIME type of the embedded part?
 - do we know that the message is signed?
 - can notmuch-show read its content?
 - can notmuch-show indicate the signature validity?
 - can notmuch-reply properly quote and attribute content?

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-04-30 17:57:26 -03:00
Daniel Kahn Gillmor
488e91f42b test-lib.sh: add test_valid_json
This test does exactly what it says on the tin.  It expects JSON data
to be parseable by Python, at least.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-04-30 17:57:16 -03:00
Daniel Kahn Gillmor
92149485cb tests/smime: Test indexing cleartext of envelopedData
These tests describe some simple behavior we would expect to work if
we were to correctly index the cleartext of encrypted S/MIME messages
(PKCS#7 envelopedData).

Of course, they don't currently pass, so we mark them known-broken.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-04-30 17:56:13 -03:00
Daniel Kahn Gillmor
a8bf94af44 tests/smime: Verify cryptographic message status
When consuming a signed+encrypted S/MIME message generated by emacs,
we expect to see the same cryptographic properties for the message as
a whole.  This is not done correctly yet, so the test is marked as
known broken.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-04-30 17:56:02 -03:00
Daniel Kahn Gillmor
482af5a031 tests: Add S/MIME messages to protected-headers corpus
These sample messages are taken directly from the Protected Headers
draft:

https://www.ietf.org/id/draft-autocrypt-lamps-protected-headers-02.html

Note that this commit doesn't strictly pass the common git pre-commit
hook due to introducing some trailing whitespace.  That's just the
nature of the corpus, though.  We should have that trailing
whitespace, so I've made this commit with --no-verify.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-04-30 17:55:19 -03:00
Daniel Kahn Gillmor
7c7cebffe6 tests/smime: include secret key material for Bob
This is taken from the same Internet Draft that test/smime/ca.crt
comes from.  See that draft for more details.
https://www.ietf.org/id/draft-dkg-lamps-samples-02.html#name-pkcs12-object-for-bob

We don't use it yet, but it will be used to decrypt other messages in
the test suite.

Note that we include it here with an empty passphrase, rather than
with the passphrase "bob" that it is supplied with in the I-D.  The
underlying cryptographic material is the same, but this way we can
import cleanly into gpgsm without having a passphrase set on it (gpgsm
converts an empty-string passphrase into no passphrase at all on
import).

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-04-30 17:55:11 -03:00
Daniel Kahn Gillmor
51c43d0e92 test: Allow tests to have both gpg and gpgsm active at once
Without this fix, we couldn't run both add_gnupg_home and
add_gpgsm_home in the same test script.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-04-30 17:54:58 -03:00
Daniel Kahn Gillmor
7eaac52ddb tests/smime: avoid copying the key+cert.pem around
No functional change.

We no longer need to identify the key and cert to mml-mode when
sending an S/MIME message, so making a copy of key+cert.pem to
test_suite.pem is superfluous.  Get rid of the extra file.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-04-30 17:54:48 -03:00
Daniel Kahn Gillmor
2e351d10c2 tests/smime: Use gpgsm instead of openssl for mml creation of S/MIME msgs
The documentation for message mode clearly states that EasyPG (which
uses GnuPG) is the default and recommended way to use S/MIME with
mml-secure:

[0] https://www.gnu.org/software/emacs/manual/html_node/message/Using-S_002fMIME.html

To ensure that this mode works, we just need to import the secret key
in question into gpgsm in addition to the public key.  gpgsm should be
able pick the right keys+certificates to use based on To/From headers,
so we don't have to specify anything manually in the #secure mml tag.

The import process from the OpenSSL-preferred form (cert+secretkey) is
rather ugly, because gpgsm wants to see a PKCS#12 object when
importing secret keys.

Note that EasyPG generates the more modern Content-Type:
application/pkcs7-signature instead of application/x-pkcs7-signature
for the detached signature.

We are also obliged to manually set gpgsm's include-certs setting to 1
because gpgsm defaults to send "everything but the root cert".  In our
weird test case, the certificate we're using is self-signed, so it
*is* the root cert, which means that gpgsm doesn't include it by
default.  Setting it to 1 forces inclusion of the signer's cert, which
satisfies openssl's smime subcommand. See https://dev.gnupg.org/T4878
for more details.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-04-30 17:54:37 -03:00
Daniel Kahn Gillmor
f7921e6e1c tests/smime: consistently quote $GNUPGHOME
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-04-30 17:54:03 -03:00
Daniel Kahn Gillmor
1f21465205 tests/smime: Include the Sample LAMPS Certificate Authority
This CA is useful for test suites and the like, but is not an
actually-secure CA, because its secret key material is also published.

I plan to use it for its intended purpose in the notmuch test suite.

It was copied from this Internet Draft:

https://tools.ietf.org/id/draft-dkg-lamps-samples-01.html#name-certificate-authority-certi

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-04-30 17:53:54 -03:00
Daniel Kahn Gillmor
6d843b8199 tests/smime: Always use --batch with gpgsm
GnuPG's gpgsm, like gpg, should always be used with --batch when it is
invoked in a non-interactive environment.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-04-30 17:53:43 -03:00
Daniel Kahn Gillmor
055e0917d7 tests: move add_gpgsm_home to test-lib.sh
This allows us to test S/MIME messages in other tests.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-04-30 17:53:34 -03:00
David Bremner
ad9c2e91a0 util/zlib-extra: de-inline gzerror_str
It turns out that putting inline functions in C header files is not a
good idea, and can cause linking problems if the compiler decides not
to inline them.  In principle this is solvable by using a "static
inline" declaration, but this potentially makes a copy in every
compilation unit. Since we don't actually care about the performance
of this function, just use a non-inline function.
2020-04-28 10:35:44 -03:00
Jonas Bernoulli
11ac932a45 emacs: Use cl-lib' instead of deprecated cl'
Starting with Emacs 27 the old `cl' implementation is finally
considered obsolete.  Previously its use was strongly discouraged
at run-time but one was still allowed to use it at compile-time.

For the most part the transition is very simple and boils down to
adding the "cl-" prefix to some symbols.  A few replacements do not
follow that simple pattern; e.g. `first' is replaced with `car',
even though the alias `cl-first' exists, because the latter is not
idiomatic emacs-lisp.

In a few cases we start using `pcase-let' or `pcase-lambda' instead
of renaming e.g. `first' to `car'.  That way we can remind the reader
of the meaning of the various parts of the data that is being
deconstructed.

An obsolete `lexical-let' and a `lexical-let*' are replaced with their
regular variants `let' and `let*' even though we do not at the same
time enable `lexical-binding' for that file.  That is the right thing
to do because it does not actually make a difference in those cases
whether lexical bindings are used or not, and because this should be
enabled in a separate commit.

We need to explicitly depend on the `cl-lib' package because Emacs
24.1 and 24.2 lack that library.  When using these releases we end
up using the backport from GNU Elpa.

We need to explicitly require the `pcase' library because
`pcase-dolist' was not autoloaded until Emacs 25.1.
2020-04-27 07:36:10 -03:00
Olivier Taïbi
7b756d1e38 test: sort the output of the "prefix" test in T610-message-property
This test extracts values from a (key,value) map where multiple entries
can have the same key, and the entries are sorted by key, but not by
value.  The test incorrectly assumes that the values will be sorted as
well, so sort the output.
2020-04-23 21:31:37 -03:00
Tomi Ollila
00cdfe1071 build: drop support for xapian versions less than 1.4
Xapian 1.4 is over 3 years old now (1.4.0 released 2016-06-24),
and 1.2 has been deprecated in Notmuch version 0.27 (2018-06-13).

Xapian 1.4 supports compaction, field processors and retry locking;
conditionals checking compaction and field processors were removed
but user may want to disable retry locking at configure time so it
is kept.
2020-04-23 21:28:45 -03:00
Jonas Bernoulli
e02bb7e9fd emacs: Explicitly depend on Emacs 24
We use various things that were not available in earlier versions.
2020-04-16 07:58:27 -03:00
Jonas Bernoulli
b4b558ac38 emacs: Declare function notmuch-show-get-message-id 2020-04-16 07:58:20 -03:00
Jonas Bernoulli
89f7a3c3e4 gitignore: Ignore generated python-cffi files 2020-04-16 07:58:12 -03:00
Olivier Taïbi
b4f593e0e6 util: after gzgets(), Z_STREAM_END means EOF, not error
Context: I am compiling notmuch on OpenBSD which has a rather old zlib
1.2.3.  It seems that the behaviour of gzgets() changed slightly between
this version and more recent versions, but the manual does not reflect
that change.  Note that zlib's manual:
- does not specify which error code (Z_OK or Z_STREAM_END) is set when
  EOF is reached,
- does not indicate the meaning of Z_STREAM_END after gzgets(), but
  based on its meaning as a possible return value of inflate(), I would
  guess that it means EOF.

amended by db: tidy commit message
2020-04-16 07:55:23 -03:00
Olivier Taïbi
8c718a8190 cli/restore: gzerror() after gzclose_r() is a use after free
Calling gzerror() (indirectly via gzerror_str()) after gzclose_r is a
use after free, according to zlib's manual.

amended by db: tidied commit message
2020-04-16 07:52:42 -03:00
Keegan Carruthers-Smith
e083987338 emacs: use def instead of initial-input for notmuch-show-browse-urls
This is the non-deprecated way to use completing-read. Additionally
the old use was broken when using ivy for completing-read. For user's
using completing-read-default they won't see the default URL now, but
if they hit enter it will be visited. Alternatively they can select
it with M-n.

From the completing-read documentation for initial-input:

 This feature is deprecated--it is best to pass nil for INITIAL-INPUT
 and supply the default value DEF instead.  The user can yank the
 default value into the minibuffer easily using M-n.

Additionally collection is now all urls, rather than all but the
first. I'm not sure why "(cdr urls)" was previously done.
2020-04-14 12:29:47 -03:00
Keegan Carruthers-Smith
f28e0a9337 emacs: introduce notmuch-search-by-tag
This is like notmuch-search-filter-by-tag, but creates a new search
rather than filtering the current search. We add this to
notmuch-common-keymap since this can be used by many contexts. We bind
to the key "t", which is the same key used by
notmuch-search-filter-by-tag in notmuch-search-mode-map. This is done
intentionally since the keybinding for notmuch-search-mode-map can be
seen as a specialization of creating a new search.

This change was motivated for use in "notmuch-hello". It is a more
convenient way to search a tag than expanding the list of all tags. I
also noticed many saved searches people use are simply tags.
2020-04-14 12:29:31 -03:00
David Bremner
690e36bacd cli/dump: replace use of gzprintf with gzputs for config values
These can be large, and hit buffer limitations of gzprintf.
2020-04-13 17:14:50 -03:00
David Bremner
24ff33082a cli/dump: define GZPUTS and use it in notmuch-dump
Similarly to GZPRINTF, this is a drop in replacement that can be
improved where needd.
2020-04-13 17:14:50 -03:00
David Bremner
0d0918f604 cli/dump: define GZPRINTF macro and use it in place of gzprintf
This will at least catch errors, and can be replaced with more
sophisticated error handling where appropriate.
2020-04-13 17:14:50 -03:00
David Bremner
0211272843 status: add print_status_gzbytes
This is in the client code, rather than libnotmuch_util, because it
prints to stderr. Also it in pretends to generate notmuch status
codes.
2020-04-13 17:14:50 -03:00
David Bremner
2c1f783f5f don't pass NULL as second parameter to gzerror
Although (as of 1.2.11) zlib checks this parameter before writing to
it, the docs don't promise to keep doing so, so be safe.
2020-04-13 17:13:55 -03:00
David Bremner
d50f41c0fd test: add known_broken test for dumping large stored queries
'qsx' reported a bug on #notmuch with notmuch-dump and large stored
queries. This test will pass (on my machine) if the value of `repeat'
is made smaller.

Reported-By: Thomas Schneider <qsx@chaotikum.eu>
2020-04-13 09:35:14 -03:00
William Casarin
374217a01a emacs/tree: add x/X bindings
Add x and X binds to notmuch-tree for functionally that we have in
notmuch-show.

The notmuch-tree-quit binding is somewhat redundant, since it is
handled by notmuch-bury-or-kill-this-buffer which is bound to q.

Signed-off-by: William Casarin <jb55@jb55.com>
2020-04-06 07:17:55 -03:00
William Casarin
86f3cc265a emacs/tree: add notmuch-tree-archive-thread-then-exit
This is the notmuch-tree version of
notmuch-show-archive-thread-then-exit

Signed-off-by: William Casarin <jb55@jb55.com>
2020-04-06 07:17:34 -03:00
William Casarin
1abe5a0c5b emacs/tree: add notmuch-tree-archive-message-than-next-or-exit
This is the notmuch-tree version of
notmuch-show-archive-message-than-next-or-exit.

Signed-off-by: William Casarin <jb55@jb55.com>
2020-04-06 07:16:58 -03:00
William Casarin
c6f43306cc emacs/tree: add kill-both prefix argument to notmuch-tree-quit
This allows us to close both windows at the same time.

Signed-off-by: William Casarin <jb55@jb55.com>
2020-04-06 07:15:27 -03:00
William Casarin
d9888b301c emacs/tree: add notmuch-tree-matching-message
This functions removes some duplicate logic between
notmuch-tree-{next,prev}-matching-message

We do this because we will be adding some additional logic similar to
the notmuch-show-next-open-message function, and it will help if this
logic is all in one place.

Signed-off-by: William Casarin <jb55@jb55.com>
2020-04-06 07:15:07 -03:00
William Casarin
c0868d959d emacs/tree: add notmuch-tree-goto-matching-message
This function captures some common logic when jumping to matching
messages in notmuch-tree mode.

We also add a new return value (t or nil), that indicates if there was
a next matching message in the thread to show.

Signed-off-by: William Casarin <jb55@jb55.com>
2020-04-06 07:14:42 -03:00
William Casarin
8eec15a903 emacs/tree: return true if a thread was found in next-thread
This will allow us to pop back to parent buffers when there are no
more threads to jump to.

Signed-off-by: William Casarin <jb55@jb55.com>
2020-04-06 07:14:32 -03:00
Daniel Kahn Gillmor
144cf30e2c nmbug: explicitly prefer python3
nmbug and notmuch-report are developer tools.  It's 2018, and all
developers should have python3 available.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-04-03 09:33:44 -03:00
Daniel Kahn Gillmor
5bf5aa1cff debian: Add packaging for python3-notmuch2
Ship a new debian package for the notmuch2 CFFI-based Python interface
to notmuch.

Unlike the notmuch python module, the new notmuch2 module is no longer
arch-independent, because it builds and ships a shared object in
addition to the python code.

This patch encourages new downstream development to rely on notmuch2
instead of on notmuch, to get the benefits of the new module.

I welcome any suggested improvements to this packaging, but it appears
to me to be sufficient to get "import notmuch2" to work and do some
basic tests.
2020-04-01 22:50:27 -03:00
Daniel Kahn Gillmor
bd0b5abd5d doc: Drop obsolete MSCGEN_PATH, PERL_PATH from doxygen configuration
Since doxygen 1.8.16, MSCGEN_PATH and PERL_PATH are obsolete:

MSCGEN_PATH:
  873e0ccfbe
PERL_PATH:
  6d1535c38f

I don't think that the notmuch builds ever depended on them in the
first place, and including them in the default config yields the
following two warnings:

```
doxygen ./doc/doxygen.cfg
warning: Tag 'PERL_PATH' at line 267 of file './doc/doxygen.cfg' has become obsolete.
         To avoid this warning please remove this line from your configuration file or upgrade it using "doxygen -u"
warning: Tag 'MSCGEN_PATH' at line 272 of file './doc/doxygen.cfg' has become obsolete.
         To avoid this warning please remove this line from your configuration file or upgrade it using "doxygen -u"
```

Remove them to avoid the warnings.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-04-01 22:46:58 -03:00
Greg Anders
1fcf068e33 Make notmuch-mutt script more portable
The -D flag to install (used in the Makefile) is GNU-specific and does
not work on BSD distributions (i.e. macOS). Likewise with the xargs -r
flag. These changes use portable alternatives to these flags while
preserving the exact behavior.
2020-03-22 22:57:25 -03:00
Daniel Kahn Gillmor
ea16b5ba85 emacs: avoid warning about notmuch-show-get-message-id
Without this change, we see the following warning when compiling the
elisp:

```
EMACS emacs/notmuch-crypto.elc

In end of data:
emacs/notmuch-crypto.el:266:1:Warning: the function
    ‘notmuch-show-get-message-id’ is not known to be defined.
```

Thanks to Örjan Ekeberg and David Edmondson for their followup about
this.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-03-22 22:21:20 -03:00
Daniel Kahn Gillmor
4dccb99282 mime-node: Clean up unwrapped MIME parts correctly.
Avoid a memory leak in the notmuch command line.

gmime_multipart_encrypted_decrypt returns a GMimeObject marked by
GMime as "transfer full", so we are supposed to clean up after it.

When parsing a message, notmuch would leak one GMimeObject part per
multipart/encrypted MIME layer.  We clean it up by analogy with
cleaning up the signature list associated with a MIME node.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-03-22 22:21:03 -03:00
Daniel Kahn Gillmor
7de3d77d2d mime-node: rename decrypted_child to unwrapped_child
When walking the MIME tree, we might need to extract a new MIME
object.  Thus far, we've only done it when decrypting
multipart/encrypted messages, but PKCS#7 (RFC 8551, S/MIME) has
several other transformations that warrant a comparable form of
unwrapping.

Make this member re-usable for PKCS#7 unwrappings as well as
multipart/encrypted decryptions.

This change is just a naming change, it has no effect on function.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-03-22 22:20:52 -03:00
Mark Walters
98e9bda17f notmuch-hello/jump: allow saved searches to specify unthreaded mode
Saved searches in notmuch-hello and notmuch-jump can specify whether
to use search mode or tree mode. This adds an option for them to
specify unthreaded mode.
2020-03-19 22:08:07 -03:00
Mark Walters
c578c32e3b Add a U binding to switch to unthreaded from other views
We have shortcuts S and Z to let the user switch to Search view and
Tree view with the current search. Add U to let the user switch to
unthreaded view from the current search, and ensure that S and Z
switch from unthreaded to search and tree veiew respectively.
2020-03-19 22:07:51 -03:00
Mark Walters
c36e91d9d4 Unthreaded mode: allow user to choose different `show out' than tree
Tree mode allows the user to choose whether to use the split screen
displaying just the current message or a full screen displaying the
entire thread. As unthreaded mode is quite different in use the user
may want a different customisation for this mode.
2020-03-19 22:07:20 -03:00
Mark Walters
a82fb6e670 Unthreaded mode: allow different result format
It is likely that the user will want a different line format for
unthreaded mode from tree mode; in particular the thread structure
graphics are unnecessary in unthreaded mode.

Add a new customisable variable and set it to something sensible.
2020-03-19 22:07:06 -03:00
Mark Walters
63f4ba3057 Introduce unthreaded mode
This commit introduces a new 'unthreaded' search mode where each
matching message is shown on a separate line. It shares almost all of
its code with tree view. Subsequent commits will allow it to diverge
slightly in appearance.
2020-03-19 22:06:49 -03:00
Mark Walters
f140bbcb40 notmuch-show.c: add an option for messages to be returned unthreaded
This adds a --unthreaded option to notmuch show to tell it to return
the matching messages in an unthreaded order (so just by date).

To make it easier for users, in particular for notmuch-tree.el, we
output each message with the same "nesting" as if it were an entire
thread in its own right.

amended by db: s/status= /status = /
2020-03-19 22:05:47 -03:00