Commit graph

1200 commits

Author SHA1 Message Date
David Bremner
2cf38f8e1c test: known broken test file descriptor leak in gzip file open
James Troup reported this bug in id:87pnjsf9q5.fsf@canonical.com
2019-10-12 08:43:39 -03:00
Daniel Kahn Gillmor
23bcd00363 cli/{show,reply}: use repaired form of "Mixed Up" mangled messages
When showing or replying to a message that has been mangled in transit
by an MTA in the "Mixed up" way, notmuch should instead use the
repaired form of the message.

Tracking the repaired GMimeObject for the lifetime of the mime_node so
that it is cleaned up properly is probably the trickiest part of this
patch, but the choices here are based on the idea that the
mime_node_context is the memory manager for the whole mime_node tree
in the first place, so new GMimeObject tree created on-the-fly during
message parsing should be disposed of in the same place.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-09-15 19:07:06 -04:00
Daniel Kahn Gillmor
4b1a8fd183 index: repair "Mixed Up" messages before indexing.
When encountering a message that has been mangled in the "mixed up"
way by an intermediate MTA, notmuch should instead repair it and index
the repaired form.

When it does this, it also associates the index.repaired=mixedup
property with the message.  If a problem is found with this repair
process, or an improved repair process is proposed later, this should
make it easy for people to reindex the relevant message.  The property
will also hopefully make it easier to diagnose this particular problem
in the future.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-09-15 19:07:06 -04:00
Daniel Kahn Gillmor
cb522fb06e test: add test for "Mixed-Up Mime" message mangling
Some MTAs mangle e-mail messages in transit in ways that are
repairable.

Microsoft Exchange (in particular, the version running today on
Office365's mailservers) appears to mangle multipart/encrypted
messages in a way that makes them undecryptable by the recipient.

I've documented this in section 4.1 "Mixed-up encryption" of draft -00
of
https://tools.ietf.org/html/draft-dkg-openpgp-pgpmime-message-mangling

Fortunately, it's possible to repair such a message, and notmuch can
do that so that a user who receives an encrypted message from a user
of office365.com can still decrypt the message.

Enigmail already knows about this particular kind of mangling.  It
describes it as "broken PGP email format probably caused by an old
Exchange server", and it tries to repair by directly changing the
message held by the user.  if this kind of repair goes wrong, the
repair process can cause data loss
(https://sourceforge.net/p/enigmail/bugs/987/, yikes).

The tests introduced here are currently broken.  In subsequent
patches, i'll introduce a non-destructive form of repair for notmuch
so that notmuch users can read mail that has been mangled in this way,
and the tests will succeed.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-09-15 01:20:03 -04:00
Daniel Kahn Gillmor
9829533e92 index: avoid indexing legacy-display parts
When we notice a legacy-display part during indexing, it makes more
sense to avoid indexing it as part of the message body.

Given that the protected subject will already be indexed, there is no
need to index this part at all, so we skip over it.

If this happens during indexing, we set a property on the message:
index.repaired=skip-protected-headers-legacy-display

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-09-01 08:45:30 -03:00
Daniel Kahn Gillmor
c61e22d5cb cli/{show,reply}: skip over legacy-display parts
Make use of the previous changes to fast-forward past any
legacy-display parts during "notmuch show" and "notmuch reply".

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-09-01 08:44:17 -03:00
Daniel Kahn Gillmor
27b25e45dc test: avoid showing legacy-display parts
Enigmail generates a "legacy-display" part when it sends encrypted
mail with a protected Subject: header.  This part is intended to
display the Subject for mail user agents that are capable of
decryption, but do not know how to deal with embedded protected
headers.

This part is the first child of a two-part multipart/mixed
cryptographic payload within a cryptographic envelope that includes
encryption (that is, it is not just a cleartext signed message).  It
uses Content-Type: text/rfc822-headers.

That is:

A └┬╴multipart/encrypted
B  ├─╴application/pgp-encrypted
C  └┬╴application/octet-stream
*   ╤ <decryption>
D   └┬╴multipart/mixed; protected-headers=v1 (cryptographic payload)
E    ├─╴text/rfc822-headers; protected-headers=v1 (legacy-display part)
F    └─╴… (actual message body)

In discussions with jrollins, i've come to the conclusion that a
legacy-display part should be stripped entirely from "notmuch show"
and "notmuch reply" now that these tools can understand and interpret
protected headers.

You can tell when a message part is a protected header part this way:

 * is the payload (D) multipart/mixed with exactly two children?
 * is its first child (E) Content-Type: text/rfc822-headers?
 * does the first child (E) have the property protected-headers=v1?
 * do all the headers in the body of the first child (E) match
   the protected headers in the payload part (D) itself?

If this is the case, and we already know how to deal with the
protected header, then there is no reason to try to render the
legacy-display part itself for the user.

Furthermore, when indexing, if we are indexing properly, we should
avoid indexing the text in E as part of the message body.

'notmuch reply' is an interesting case: the standard use of 'notmuch
reply' will end up omitting all mention of protected Subject:.

The right fix is for the replying MUA to be able to protect its
headers, and for it to set them appropriately based on headers found
in the original message.

If a replying MUA is unable to protect headers, but still wants the
user to be able to see the original header, a replying MUA that
notices that the original message's subject differs from the proposed
reply subject may choose to include the original's subject in the
quoted/attributed text. (this would be a stopgap measure; it's not
even clear that there is user demand for it)

This test suite change indicates what we want to happen for this case
(the tests are currently broken), and includes three additional TODO
suggestions of subtle cases for anyone who wants to flesh out the test
suite even further.  (i believe all these cases should be already
fixed by the rest of this series, but haven't had time to write the
tests for the unusual cases)

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-09-01 08:32:56 -03:00
Tomi Ollila
b6e589f54f test: aggregate-results.sh: count test files where all tests skipped
Previously, when all tests were skipped on a test file, there were
no indication of this in the final results aggregate-results.sh
printed.
Now count of the files where all tests were skipped is printed.
2019-07-05 17:58:23 +02:00
Daniel Kahn Gillmor
bdc87f0d3e test: run uncrustify
This is the result of running:

    $ uncrustify --replace --config ../devel/uncrustify.cfg *.cc *.c *.h

in the test directory.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-06-29 21:18:15 +02:00
David Bremner
1959a95d25 test: replace use of gdb with LD_PRELOAD shims in T070-insert.sh
This removes the dependency of this test script on gdb, and
considerably speeds up the running of the tests.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-06-29 21:11:08 +02:00
David Bremner
6544a2e305 test: provide machinery to make and use test_shims
These can be used e.g. to override return values for functions, in
place of the existing scripting of gdb.

This prepends to LD_PRELOAD rather than clobbering it, thanks to a
suggestion from Tomi Ollila.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-06-29 21:10:44 +02:00
Tomi Ollila
00c63bf736 test: aggregate-results.sh: consistent style. zero forks.
- all variables in $((...)) without leading $
- all comparisons use -gt, -eq or -ne
- no -a nor -o inside [ ... ] expressions
- all indentation levels using one tab

Dropped unnecessary empty string check when reading results files.

Replaced pluralize() which was executed in subshell with
pluralize_s(). pluralize_s sets $s to 's' or '' based on value of
$1. Calls to pluralize_s are done in context of current shell, so
no forks to subshells executed.
2019-06-11 07:20:01 -03:00
Daniel Kahn Gillmor
bc396c967c test: signature verification during decryption (session keys)
When the user knows the signer's key, we want "notmuch show" to be
able to verify the signature of an encrypted and signed message
regardless of whether we are using a stashed session key or not.

I wrote this test because I was surprised to see signature
verification failing when viewing some encrypted messages after
upgrading to GPGME 1.13.0-1 in debian experimental.

The added tests here all pass with GPGME 1.12.0, but the final test
fails with 1.13.0, due to some buggy updates to GPGME upstream: see
https://dev.gnupg.org/T3464 for more details.

While the bug needs to be fixed in GPGME, notmuch's test suite needs
to make sure that GMime is doing what we expect it to do; i was a bit
surprised that it hadn't caught the problem, hence this patch.

I've fixed this bug in debian experimental with gpgme 1.13.0-2, so the
tests should pass on any debian system.  I've also fixed it in the
gpgme packages (1.13.0-2~ppa1) in the ubuntu xenial PPA
(ppa:notmuch/notmuch) that notmuch uses for Travis CI.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-06-08 20:14:00 -03:00
David Bremner
2c1e5c186e test: update test description.
I missed this fix in dkg's revisions.
2019-05-29 08:40:02 -03:00
Daniel Kahn Gillmor
1c704dd22d cli/reply: pull proposed subject line from the message, not the index
Protected subject lines were being emitted in reply when the cleartext
of documents was indexed.  create_reply_message() was pulling the
subject line from the index, rather than pulling it from the
GMimeMessage object that it already has on hand.

This one-line fix to notmuch-reply.c solves that problem, and doesn't
cause any additional tests to fail.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-29 08:17:33 -03:00
Daniel Kahn Gillmor
06dedd0a83 test: reply (in cli and emacs) should protect indexed sensitive headers
These tests are currently broken!  When a protected subject is indexed
in the clear, it leaks in the reply headers :(

For emacs, we set up separate tests for when the protected header is
indexed in the clear and when it is unindexed.  neither case should
leak, but the former wasn't tested yet.

We will fix the two broken tests in a subsequent patch.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-29 08:17:20 -03:00
Daniel Kahn Gillmor
cd8006886b test: emacs/show: ensure that protected headers appear as expected
This tests notmuch-show; headers appear appropriately based on the
setting of notmuch-crypto-process-mime.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-29 08:17:12 -03:00
Daniel Kahn Gillmor
5007595be8 test: ensure that protected headers appear in notmuch-emacs search as expected
We initially test only notmuch-search; tests for other functionality
come in different patchsets later.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-29 08:16:58 -03:00
Daniel Kahn Gillmor
809a34a870 test: try indexing nested messages and protected headers
We want to make sure that internally-forwarded messages don't end up
"bubbling up" when they aren't actually the cryptographic payload.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-29 08:15:28 -03:00
Daniel Kahn Gillmor
bfed02bb0b test: after reindexing, only legitimate protected subjects are searchable
This test scans for all the possible protected headers (including
bogus/broken ones) that are present in the protected-headers corpus,
trying to make sure that only the ones that are not broken or
malformed show up in a search after re-indexing.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-29 08:15:18 -03:00
Daniel Kahn Gillmor
b36248a26e test: protected headers should work when both encrypted and signed.
Up to this point, we've tested protected headers on messages that have
either been encrypted or signed, but not both.

This adds a couple tests of signed+encrypted messages, one where the
subject line is masked (outside subject line is "Subject Unavailable")
and another where it is not (outside Subject: matches inner Subject:)

See the discussion at
https://dkg.fifthhorseman.net/blog/e-mail-cryptography.html#protected-headers
for more details about the nuances between signed, stripped, and
stubbed headers.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-29 08:14:57 -03:00
Daniel Kahn Gillmor
5c3a44681f indexing: record protected subject when indexing cleartext
When indexing the cleartext of an encrypted message, record any
protected subject in the database, which should make it findable and
visible in search.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-29 08:14:44 -03:00
Daniel Kahn Gillmor
b7b553e732 cli/reply: ensure encrypted Subject: line does not leak in the clear
Now that we can decrypt headers, we want to make sure that clients
using "notmuch reply" to prepare a reply don't leak cleartext in their
subject lines.  In particular, the ["reply-headers"]["Subject"] should
by default show the external Subject.

A replying MUA that intends to protect the Subject line should show
the user the Subject from ["original"]["headers"]["Subject"] instead
of using ["reply-headers"]["Subject"].

This minor asymmetry with "notmuch show" is intentional.  While both
tools always render the cleartext subject line when they know it (in
["headers"]["Subject"] for "notmuch show" and in
["original"]["headers"]["Subject"] for "notmuch reply"), "notmuch
reply" should never leak something that should stay under encrypted
cover in "reply-headers".

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-29 08:14:32 -03:00
Daniel Kahn Gillmor
996ef5710c test: show cryptographic envelope information for signed mails
Make sure that we emit the correct cryptographic envelope status for
cleartext signed messages.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-29 08:13:06 -03:00
Daniel Kahn Gillmor
1c879f3939 test: add test for missing external subject
Adding another test to ensure that we handle protected headers
gracefully when no external subject is present.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-29 08:12:49 -03:00
Daniel Kahn Gillmor
56416a5470 cli/show: add information about which headers were protected
The header-mask member of the per-message crypto object allows a
clever UI frontend to mark whether a header was protected (or not).
And if it was protected, it contains enough information to show useful
detail to an interested user.  For example, an MUA could offer a "show
what this message's Subject looked like on the wire" feature in expert
mode.

As before, we only handle Subject for now, but we might be able to
handle other headers in the future.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>

Amended by db: tweaked schemata notation.
2019-05-29 08:11:50 -03:00
Daniel Kahn Gillmor
1c7fbbcc99 cli/show: emit payload subject instead of outside subject
Correctly fix the two outstanding tests so that the protected (hidden)
subject is properly reported.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-29 08:05:01 -03:00
Daniel Kahn Gillmor
528f526f69 cli/show: add tests for viewing protected headers
Here we add several variant e-mail messages, some of which have
correctly-structured protected headers, and some of which do not.  The
goal of the tests is to ensure that the right protected subjects get
reported.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-29 08:04:32 -03:00
Jameson Graef Rollins
03839a8110 test: new test framework to compare json parts
This makes it easier to write fairly compact, readable tests of json
output, without needing to sanitize away parts that we don't care
about.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-29 08:03:21 -03:00
Daniel Kahn Gillmor
80728a95e6 cli/show: emit headers after emitting body
This paves the way for emitting protected headers after verification
and decryption, because it means that the headers will only be emitted
after the body has been parsed.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-29 08:02:32 -03:00
Daniel Kahn Gillmor
73cebe6e72 test: report summary even when aborting
In certain cases of test suite failure, the summary report was not
being printed.  In particular, any failure on the parallel test suite,
and any aborted test in the serialized test suite would end up hiding
the summary.

It's better to always show the summary where we can (while preserving
the return code).  If we do abort due to this high-level failure,
though, we should also announce to the user that we're doing so as
close to the end of the process as possible, to make it easier to find
the problem.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-26 18:55:06 -03:00
Daniel Kahn Gillmor
4cb789aa09 cli/show: emit new whole-message crypto status output
This allows MUAs that don't want to think about per-mime-part
cryptographic status to have a simple high-level overview of the
message's cryptographic state.

Sensibly structured encrypted and/or signed messages will work fine
with this.  The only requirement for the simplest encryption + signing
is that the message have all of its encryption and signing protection
(the "cryptographic envelope") in a contiguous set of MIME layers at
the very outside of the message itself.

This is because messages with some subparts signed or encrypted, but
with other subparts with no cryptographic protection is very difficult
to reason about, and even harder for the user to make sense of or work
with.

For further characterization of the Cryptographic Envelope and some of
the usability tradeoffs, see here:

   https://dkg.fifthhorseman.net/blog/e-mail-cryptography.html#cryptographic-envelope
2019-05-26 08:20:23 -03:00
Daniel Kahn Gillmor
9300defd64 emacs: Drop content-free "Unknown signature status" button
When we have not been able to evaluate the signature status of a given
MIME part, showing a content-free (and interaction-free) "[ Unknown
signature status ]" button doesn't really help the user at all, and
takes up valuable screen real-estate.

A visual reminder that a given message is *not* signed isn't helpful
unless it is always present, in which case we'd want to see "[ Unknown
signature status ]" buttons on all messages, even ones that don't have
a signing structure, but i don't think we want that.

Amended by db to drop the unused initialization of 'label'
2019-05-25 13:02:02 -03:00
Daniel Kahn Gillmor
fa9d8b7026 test: allow disabling timeout with NOTMUCH_TEST_TIMEOUT=0
To aid in diagnosing test suite tooling that interacts poorly with
coreutils' timeout, it's handy to be able to bypass it entirely.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-25 08:26:41 -03:00
David Bremner
adb53b0737 lib/database: index user headers.
This essentially involves calling _notmuch_message_gen_terms once for
each user defined header.
2019-05-25 07:21:13 -03:00
David Bremner
575493e785 lib: setup user headers in query parser
These tests will need to be updated if the Xapian
query print/debug format changes.
2019-05-25 06:56:16 -03:00
David Bremner
4b9c03efc6 cli/config: check syntax of user configured field names
These restrictions are meant to prevent incompatibilities with the
Xapian query parser (which will split at non-word characters) and
clashes with future notmuch builtin fields.
2019-05-25 06:56:16 -03:00
David Bremner
7981bd050e cli/config: support user header index config
We don't do anything with this configuration information information
yet, but nonetheless add a couple of regression tests to make sure we
don't break standard functionality when we do use the configuration
information.
2019-05-25 06:56:16 -03:00
Tomi Ollila
3563079be3 test-lib.sh: colors to test output when parallel(1) is run on tty
Done via $COLORS_WITHOUT_TTY environment variable as passing options
to commands through parallel(1) does not look trivial.

Reorganized color checking in test-lib.sh a bit for this (perhaps
were not fully necessary but rest still an improvement):

  - color checking commands in subshell are not run before arg parsing
    (args may disable colors with --no-color)

  - [ -t 1 ] is checked before forking subshell
2019-05-23 08:00:31 -03:00
Tomi Ollila
a1aea7272e test-lib.sh: "tidied" emacs_deliver_message ()
Added initialization and checking of smtp_dummy_port
like it was done with smtp_dummy_pid.

Made those function-local variables.

One 8 spaces to tab consistency conversion.

And last, but definitely not least; while doing above
noticed that there were quite a few double-quoted strings
where $@ was in the middle of it -- replaced those with $*
for robustness ("...$@..." expands params to separate words,
"...$*..." params expands to single word).
2019-05-23 08:00:13 -03:00
Tomi Ollila
f33053023b test: redirect STDIN from /dev/null
Without this stdin may be anything that parent process provided for it.

Test processes might have tried to read something from it, which would
have caused undeterministic behavior.

E.g. gdb(1) tries to do tty related ioctls on fd 0 (and fd 1 and fd 2,
but those are redirected to 'test.output' before test runs).
2019-05-22 08:47:17 -03:00
David Bremner
f4708ce0b1 test/emacs: revert invalid-from test to pre-86f89385 behaviour
To the best of my understanding, this original behaviour was what
Carl's homebrew parser produced. With commit 86f89385 Austin switched
to using GMime (2.6). This produced arguably worse results, but since
the input was bad, we could live with it. Now with GMime 3.0 we are
getting the original results again, and there is no reason to consider
this test broken.
2019-05-20 16:31:28 -03:00
Daniel Kahn Gillmor
c88e030580 tests: fail and report when a parallel build fails (or times out)
When a parallel build fails (or when it times out, if timeout is
present), the test suite should not blithely succeed.  Catch these
failures and at least report them.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-20 14:48:56 -03:00
Daniel Kahn Gillmor
73bf7e532e tests: make timeout configurable with NOTMUCH_TEST_TIMEOUT (default: 2m)
The current 2 minute timeout is reasonable, but to exercise the test
suite or induce timeout failures, we might want to make it shorter.
This makes it configurable so you can run (for example):

    make check NOTMUCH_TEST_TIMEOUT=10s

We stick with the default of 2m.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-20 14:48:43 -03:00
Daniel Kahn Gillmor
9c0001de4b test: show what emacs sees of an encrypted message when crypto is disabled
Some users may set notmuch-crypto-process-mime to nil, disabling all
crypto use.  We should have a baseline for what that looks like.
2019-05-10 12:30:03 -03:00
Daniel Kahn Gillmor
bda0fecccd test: avoid unnecessary extraction of the test fingerprint
FINGERPRINT is already exported by add_gnupg_home, so this is
unnecessary.  This change also happens to get rid of the superfluous
check-trustdb spew from the test suite that looked like this:

gpg: checking the trustdb
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-10 06:56:34 -03:00
David Bremner
f2425a11a3 test: let the OS choose a port for smtp-dummy
This should avoid potential collisions if we start running multiple
smtp-dummy processes in parallel.
2019-05-10 06:56:22 -03:00
Daniel Kahn Gillmor
e1c8357c44 emacs: test notmuch-show during message decryption
We did not have a test showing what message decryption looks like
within notmuch-emacs.  This change gives us a baseline for future work
on the notmuch-emacs interface.

This differs from previous revisions of this patch in that it should
be insensitive to the order in which the local filesystem readdir()s
the underlying maildir.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-10 06:54:50 -03:00
Jameson Graef Rollins
eeff431996 tests: environment variable to specify that tests should be serialized
If NOTMUCH_TEST_SERIALIZE is non-null all tests will be run in series,
rather than in parallel.
2019-05-07 06:55:31 -03:00
Jameson Graef Rollins
908d930d22 tests: run all tests in parallel, if available
If either the moreutils or GNU parallel utility are available, run all
tests in parallel.  On my eight core machine this makes for a ~x7
speed-up in the full test suite (1m24s -> 12s).

The design of the test suite makes this parallelization trivial.
2019-05-07 06:54:09 -03:00
Jameson Graef Rollins
591388ccd1 tests: remove entangling corpus index optimization
The add_email_corpus test utility includes logic that tries to re-use
an index of the corpus if available.  This was seemingly done as an
optimization, so that every test that uses the corpus didn't have to
create it's own index of the corpus.  However, this has the perverse
side effect of entangling tests together, and breaks parallelization.

Forcing each test to do it's own index does increase the overall time
of the test slightly (~6%), but this will be more than made up for in
the next patch that introduces paraellization.
2019-05-07 06:53:57 -03:00
Jameson Graef Rollins
7f7af27bd8 tests: remove some redundant pre-cleanup of the corpus MAIL_DIR
add_email_corpus itself does an rm -rf $MAIL_DIR, so these are not necessary.
2019-05-07 06:52:35 -03:00
Daniel Kahn Gillmor
7d48604157 test/crypto: add_gnupg_home should have ultimate trust on "its own" key
The typical use case for gpg is that if you control a secret key, you
mark it with "ultimate" ownertrust.

The opaque --import-ownertrust mechanism is GnuPG's standard mechanism
to set up ultimate ownertrust (the ":6:" means "ultimate", for
whatever reason).

We adjust the test suite to match this change, inverting the sense of
one test: since the default is now that the user ID of the suite's own
key is valid, we change the test to make sure that the user ID is not
emitted when it is *not* valid.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-07 06:42:21 -03:00
Daniel Kahn Gillmor
93e699e5c8 test: simplify user ID handling
The user ID on the self-test is a little bit clunky-looking.  It also
may end up showing up elsewhere in the test suite.  Centralizing the
user ID in one place should make it easier to handle if it ever
changes, and should make tests easier to read.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-07 06:42:11 -03:00
Daniel Kahn Gillmor
9f05ceb994 test/crypto: clarify the difference between ownertrust and validity
This is a subtle difference, but the output of notmuch shouldn't ever
change based on ownertrust itself -- notmuch is intended to show valid
User IDs, and to avoid showing invalid User IDs.

It so happens that setting ownertrust of a key to ultimate sets all
associated user IDs to "full" validity, so the test is correct, but
just misnamed.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-07 06:42:01 -03:00
David Bremner
6682b4e686 Merge tag 0.28.4
No functionality changes merged, since the bug in question was already
fixed on master.
2019-05-05 16:38:51 -03:00
David Bremner
ae6b52488d cli/show: avoid empty write to stdout in format_part_raw
Previously if the input was exactly a multiple of the internal buffer
size, notmuch would attempt to fwrite nothing to stdout, but still
expected fwrite to return 1, causing a failure that looked like this:

  $ notmuch show --format=raw id:87o96f1cya.fsf@codeaurora.org
    ...entire message shown as expected..
  Error: Write failed
  $ echo $?
  1

To fix the problem don't call fwrite at all when there's nothing to
write.

Amended by db: add some tests of message sizes likely to cause this
problem.
2019-05-05 08:01:17 -03:00
David Bremner
103c11822e cli/notmuch-show: support gzipped files
This drops "file" from mime_node_context and just uses a local
variable. It also uses the new gzip aware utility routines recently
added to util/gmime-extra.c. The use of gzopen / gzfile in addition is
a bit icky, but the choice is between that, and providing yet another
readline implimentation that understands GMime streams.
2019-05-03 07:48:43 -03:00
David Bremner
852167479f lib/message_file: open gzipped files
Rather than storing the lower level stdio FILE object, we store a
GMime stream. This allows both transparent decompression, and passing
the stream into GMime for parsing. As a side effect, we can let GMime
close the underlying OS stream (indeed, that stream isn't visible here
anymore).

This change is enough to get notmuch-{new,search} working, but there is still
some work required for notmuch-show, to be done in a following commit.
2019-05-03 07:48:43 -03:00
Daniel Kahn Gillmor
7e6f55b268 gmime-cleanup: simplify T355-smime.sh
GMime 3.0 and later can handle User ID as expected.

signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-03 06:55:52 -03:00
Daniel Kahn Gillmor
652baa6fe6 gmime-cleanup: tests should only care about gmime 3
note that "notmuch-show for message with invalid From" is still broken
in T310-emacs.sh.  It would be good to debug what's going on there and
try to get it fixed!

signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-03 06:55:44 -03:00
Daniel Kahn Gillmor
bb0b119358 gmime-cleanup: always support session keys
Our minimum version of GMime 3.0 always supports good session key
handling.

signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-03 06:55:32 -03:00
Daniel Kahn Gillmor
096d45a878 gmime-cleanup: remove obsolete gpg_path configuration option and crypto contexts
Note that we do keep ignoring the gpg_path configuration option,
though, to avoid breakage of existing installations.  It is ignored
like any other unknown configuration option, but we at least document
that it is ignored so that people who find it in their legacy configs
can know that it's safe to drop.

signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-03 06:55:04 -03:00
Daniel Kahn Gillmor
afb0b11dcd crypto: Avoid pretending to verify signatures on unsigned encrypted mail
Unsigned encrypted mail shows up with a weird empty signature list.
If we successfully decrypted and there was no signature in it, we
should just not show a sigstatus at all.

The documentation for g_mime_decrypt_result_get_signatures says:

    a GMimeSignatureList or NULL if the stream was not signed.
2019-04-26 08:03:15 -03:00
Daniel Kahn Gillmor
22ec4a36e6 tests: move FINGERPRINT definition to add_gnupg_home
If a test has added a GnuPG homedir, it may well want to know the
fingerprint.  This saves us from having to redefine this magic string
in multiple places when more tests eventually use the GnuPG homedir.
2019-04-24 07:16:46 -03:00
David Bremner
319dd95ebb lib: add 'body:' field, stop indexing headers twice.
The new `body:` field (in Xapian terms) or prefix (in slightly
sloppier notmuch) terms allows matching terms that occur only in the
body.

Unprefixed query terms should continue to match anywhere (header or
body) in the message.

This follows a suggestion of Olly Betts to use the facility (since
Xapian 1.0.4) to add the same field with multiple prefixes. The double
indexing of previous versions is thus replaced with a query time
expension of unprefixed query terms to the various prefixed
equivalent.

Reindexing will be needed for 'body:' searches to work correctly;
otherwise they will also match messages where the term occur in
headers (demonstrated by the new tests in T530-upgrade.sh)
2019-04-17 08:48:16 -03:00
Örjan Ekeberg
bfe0f8dcc4 test: add test for checking forwarded messages
Add test of forwarding messages from within emacs.
The first test checks that a references header is properly
added to the new message.  The second test checks that the
send-hook of the forwarding message adds a forwarded-tag
to the original message.
2019-04-14 08:01:30 -03:00
David Edmondson
639d21d5b9 test: Add emacs attachment check tests. 2019-03-28 14:37:21 -03:00
David Bremner
d25dcc589c lib: use phrase search for anything not ending in '*'
Anything that does not look like a wildcard should be safe to
quote. This should fix the problem searching for xapian keywords.
2019-03-28 14:34:37 -03:00
David Bremner
168211c563 test: add two known broken tests searching for xapian keywords
Thanks to plujon for pointing out this problem on IRC. The underlying
issue is that the quotes are stripped before the field processors get
the query string, and the heuristic for putting them back is not quite
right.
2019-03-28 14:32:11 -03:00
Luis Ressel
9f7e851263 Prepend regerror() messages with "regexp error: "
The exact error messages returned by regerror() aren't standardized;
relying on them isn't portable. Thus, add a a prefix to make clear that
the subsequent message is a regexp parsing error, and only look for this
prefix in the test suite, ignoring the rest of the message.
2019-03-11 22:24:55 -03:00
Luis Ressel
a1e649276e test/T030-config: Separate stdout and stderr output
POSIX doesn't specify the flushing behaviour of the STDOUT stream, so
it's invalid to assume a particular order between the stdout and stderr
output. The current test breaks on musl due to this.
2019-03-11 22:13:54 -03:00
David Bremner
71eaa19350 Merge branch 'release'
Changes from 0.28.3
2019-03-06 08:53:26 -04:00
David Bremner
e88297c072 lib/string_map: fix return type of string_cmp
I can't figure out how checking the sign of a bool ever worked. The
following program demonstrates the problem (i.e. for me it prints 1).

 #include <stdio.h>
 #include <stdbool.h>
 int main(int argc, char **argv) {
    bool x;
    x = -1;
    printf("x = %d\n", x);
 }

This seems to be mandated by the C99 standard 6.3.1.2.
2019-03-05 21:46:41 -04:00
David Bremner
09595cf577 lib: Add known broken test for string_map binary search.
Because the string_map functions are not exported, we test via message
properties.
2019-03-05 21:46:41 -04:00
rhn
b350c688e1 test: Check for replies obeying lifetime guarantees
The test attempts to check that a message coming from a thread outlives its messages list and gets destroyed together with the thread.
2019-01-25 20:37:26 -04:00
David Edmondson
576cff7654 reply: Include sender as recipient if they were the original recipient
When generating a reply message, if the user was the originator and
only recipient of the original message, include the user as a
recipient of the reply.
2018-12-07 08:03:50 -04:00
Maxime Coste
baa4185c30 cli: notmuch show support for --include-html with --format=text 2018-11-24 09:12:30 -04:00
Maxime Coste
37e5bc00ae cli: notmuch show support for --body=false with --format=text 2018-11-24 09:11:42 -04:00
Daniel Kahn Gillmor
311f530013 test: git should ignore message-id-parse
b31e44c678 introduced message-id-parse
as a new binary created by the test suite.  It shows up as something
additional to git, but git ought to know to ignore it.
2018-10-08 12:50:03 -03:00
David Edmondson
e8cb6b2cd6 emacs: Call `notmuch-mua-send-hook' hooks when sending a message
Previously any hook functions attached to `notmuch-mua-send-hook' were
ignored.
2018-09-28 20:22:42 -03:00
David Edmondson
4e213fe9b4 test: Check that `notmuch-mua-send-hook' is called on sending a message 2018-09-28 20:22:33 -03:00
David Bremner
c846e15ffe test: make regexp test conditional on field processors
Normally we'd mark it broken, but perversely missing regexp support
actually makes the test pass.
2018-09-14 08:54:20 -03:00
David Bremner
c6ba5522a5 test: mark thread subqueries broken without field processors
Currently these tests just fail when notmuch is built against Xapian
1.2.x
2018-09-14 08:54:09 -03:00
David Edmondson
4cd5a0a3d5 test: Absolute and relative directory paths. 2018-09-08 20:19:42 -03:00
David Bremner
87934c432c lib: change parent strategy to use In-Reply-To if it looks sane
As reported by Sean Whitton, there are mailers (in particular the
Debian Bug Tracking System) that have sensible In-Reply-To headers,
but un-useful-for-notmuch References (in particular with the BTS, the
oldest reference is last). I looked at a sample of about 200K
messages, and only about 0.5% these had something other than a single
message-id in In-Reply-To. On this basis, if we see a single
message-id in In-Reply-To, consider that as authoritative.
2018-09-06 08:07:13 -03:00
David Bremner
b31e44c678 lib: add _notmuch_message_id_parse_strict
The idea is that if a message-id parses with this function, the MUA
generating it was probably sane, and in particular it's probably safe
to use the result as a parent from In-Reply-to.
2018-09-06 08:07:13 -03:00
David Bremner
35053c2b9a test/thread-replies: mangle In-Reply-To's
In a future commit, we will start trusting In-Reply-To's when they
look sane (i.e. a single message-id). Modify these tests so they will
keep passing (i.e. keep choosing References) when that happens.
2018-09-06 08:07:13 -03:00
David Bremner
ea08032ae4 test: add known broken test for good In-Reply-To / bad References
The current scheme of choosing the replyto (i.e. the default parent
for threading purposes) does not work well for mailers that put
the oldest Reference last.
2018-09-06 08:07:13 -03:00
David Bremner
46dce33abc lib/thread: change _resolve_thread_relationships to use depths
We (finally) implement the XXX comment. It requires a bit of care not
to reparent all of the possible toplevel messages.

_notmuch_messages_has_next is not ready to be a public function yet,
since it punts on the mset case. We know in the one case it is called,
the notmuch_messages_t is just a regular list / iterator.
2018-09-06 08:07:13 -03:00
David Bremner
a330858284 lib/thread: initial use of references as for fallback parenting
This is mainly to lay out the structure of the final code. The problem
isn't really solved yet, although some very simple cases are
better (hence the fixed test). We need two passes through the messages
because we need to be careful not to re-parent too many messages and
end up without any toplevel messages.
2018-09-06 08:07:13 -03:00
David Bremner
9b568e73e1 lib/thread: sort sibling messages by date
For non-root messages, this should not should anything currently, as
the messages are already added in date order. In the future we will
add some non-root messages in a second pass out of order and the
sorting will be useful. It does fix the order of multiple
root-messages (although it is overkill for that).
2018-09-06 08:07:12 -03:00
David Bremner
ccb52edb4c test: add known broken tests for "ghost roots"
This documents the bug discussed at

     id:87efgmmysi.fsf@len.workgroup

The underlying issue is that the reply to a ghost (missing) message is
falsely classified as a root message in _resolve_thread_relationships.

There are two pairs of tests; in each case the the first test is
simpler / more robust, but also easier to fool.
2018-09-06 08:07:12 -03:00
David Bremner
ebd131ac07 test: start threading test corpus
There are 3 threads here, two synthetic, and one anonymized one using
data from Gregor. They test various aspects of thread
ordering/construction in the presence of replies to ghost messages.
2018-09-06 08:07:12 -03:00
David Bremner
dfda1745bf test/tag: add test for updating during batch tag operations
This clarifies that the breakage seen with Xapian 1.4.6 does not have
to do with "funny" tags.

This test is "known broken", but only with xapian 1.4.6, and there's
curently no convenient way to mark that.
2018-08-29 06:37:48 -03:00
Sebastian Poeplau
0f08bf7166 lib: detect mislabeled Windows-1252 parts
Use GMime functionality to detect mislabeled messages and apply the
correct (Windows) encoding instead.
2018-08-29 06:34:39 -03:00
Sebastian Poeplau
7074bb8f80 test: add known broken test for mislabeled Windows-1252 encoding
Messages that contain Windows-1252 are frequently mislabeled as ISO
8859-1, which may result in non-printable characters when displaying
the message. The test asserts that such characters (in this case
curved quotes) are displayed correctly.
2018-08-29 06:28:13 -03:00
David Bremner
46cf1a98f3 test: fix hardcoded paths in T050-new.sh
The previous paths will fail almost everywhere.
2018-05-26 16:46:44 -07:00
Daniel Kahn Gillmor
aa605f7e8a cli/show: enable --decrypt=stash
Add fancy new feature, which makes "notmuch show" capable of actually
indexing messages that it just decrypted.

This enables a workflow where messages can come in in the background
and be indexed using "--decrypt=auto".  But when showing an encrypted
message for the first time, it gets automatically indexed.

This is something of a departure for "notmuch show" -- in particular,
because it requires read/write access to the database.  However, this
might be a common use case -- people get mail delivered and indexed in
the background, but only want access to their secret key to happen
when they're directly interacting with notmuch itself.

In such a scenario, they couldn't search newly-delivered, encrypted
messages, but they could search for them once they've read them.

Documentation of this new feature also uses a table form, similar to
that found in the description of index.decrypt in notmuch-config(1).

A notmuch UI that wants to facilitate this workflow while also
offering an interactive search interface might instead make use of
these additional commands while the user is at the console:

Count received encrypted messages (if > 0, there are some things we
haven't yet tried to index, and therefore can't yet search):

     notmuch count tag:encrypted and \
         not property:index.decryption=success and \
         not property:index.decryption=failure

Reindex those messages:

     notmuch reindex --try-decrypt=true tag:encrypted and \
         not property:index.decryption=success and \
         not property:index.decryption=failure
2018-05-26 07:43:30 -07:00
Daniel Kahn Gillmor
9d114a8552 test-lib: add notmuch_show_part for "notmuch show --format=text"
Thanks to David Bremner for this improved readability!
2018-05-26 07:42:28 -07:00