Inspired by the suggestion of
id:20200727154108.16269-3-congdanhqx@gmail.com
to drop the configuration test for the default backend. This version
is hopefully robust against backend changes.
This is similar to the case of toplevel messages. Currently everything
is cached, so no database access is necessary. This might change in
the future, but it should not crash in either case.
The goal of this abstraction was to save space. But that failed as
the result actually was that four trivial lines got replace with 15
fairly complicated lines. The opposite of what it was supposed to
do.
Also it made it harder to come up with the fix in the previous commit;
simply grepping for the relevant symbols did not work because they get
constructed at run-time instead of appearing in the source file.
Starting with Emacs 27 undeclared variables in evaluated interactive
code uses lexical scope. This includes code passed with '--eval' as
we do in the Emacs tests, which also happen to assume dynamic scope.
- This can affect variables defined by libraries that we use. We
let-bind such variables to change the behavior of functions which we
then call with these bindings in effect. If these libraries are not
loaded beforehand, then the bindings are lexical and fail to have
the effect we intended.
At this time only 'smtpmail' has to be loaded explicitly (for the
variables let-bound in emacs_deliver_message and emacs_fcc_message).
'message' doesn't have to be loaded explicitly, because loading
'notmuch' (in 'run_emacs') already takes care of that, indirectly.
- Our own testing-only variables also have to be declared explicitly.
We should have done that anyway, but because of how and where these
variables are used it was very easy to overlook that (i.e. it isn't
something the byte-compiler ever looks at). Not so in Emacs 27
anymore; here this oversight caused four tests to fail.
The numeric values of these variables get incremented by functions
that we add to hooks that are run by many tests, not just the tests
where we actually inspect the value and therefore take care to let-
bind the values to 0 before we begin. The global values therefore
have to be numeric values as well. I have chosen -100 instead of 0
as the default in case someone writes a test that inspects the value
but forgets to let-bind the value. I hope that the unusual negative
value that one is going to see in such a case will help debugging
the issue.
The API docs promise to handle relative filenames, but the code did
not do it.
Also check for files outside the mail root, as implied by the API
description.
This fixes the bug reported at
id:87sgdqo0rz.fsf@tethera.net
Xapian currently succeeds to begin/end a transaction on a closed database,
or at least does not throw an exception. Make the test robust against
this changing.
The original generic handler had an extra '%s' in the format
string. Update tests that failed to catch this because the template to
print status strings checked 'stat', which was not set.
This is actually one of the few potentially useful things you can do
with a message belonging to a closed database, since in principle you
could re-open the database.
It's not very nice to return FALSE for an error, so provide
notmuch_message_get_flag_st as a migration path.
Bump LIBNOTMUCH_MINOR_VERSION because the API is extended.
Instead of printing the same static string for each test, can replace
the assert with something simpler (or at least easier to integrate
into the test suite).
These are less crucial since we stopped generating new database
versions and relied primarily on features. They also rely on a
pre-generated v1 database which happens to be chert format. This
backend is not supported by Xapian 1.5.
Also drop the tool gen-testdb.sh, which is currently broken, due to
changes in the testing infrastructure.
I haven't traced the code path as exhaustively for the SMIME test, but
the expiry date in question is larger then representable in a signed
32 bit integer.
This is a simple hack to enable out-of-tree builds, a concern raised
by Tomi in id:m24kzjib9a.fsf@guru.guru-group.fi
This change at least enables "make check" to complete without error,
but I'm sure it could be improved. I am not expert enough in
setuptools to know how.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Amended by db per id:87d06usa31.fsf@powell.devork.be
json_check_nodes.py exists in source tree, not in out of tree
build tree. Added -B to the execution so source tree is not
"polluted" by a .pyc file when json_check_nodes.py is executed.
When creating run_emacs.sh make it load .elc files from out of
tree build tree, not from source tree if such files existed.
If existed, those may be outdated, or even created by some other
emacs than the one that was used to build .elc files in out of
tree build dir.
This change means we can support "notmuch show --decrypt=true" for
S/MIME encrypted messages, resolving several outstanding broken tests,
including all the remaining S/MIME protected header examples.
We do not yet handle indexing the cleartext of S/MIME encrypted
messages, though.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
When composing a reply, no one wants to see this line in the proposed
message:
Non-text part: application/pkcs7-mime
So we hide it, the same way we hide PGP/MIME cruft.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Until we did PKCS#7 unwrapping, no leaf MIME part could have a child.
Now, we treat the unwrapped MIME part as the child of the PKCS#7
SignedData object. So in that case, we want to show it instead of
deliberately omitting the content.
This fixes the test of the protected subject in
id:smime-onepart-signed@protected-headers.example.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Unwrap a PKCS#7 SignedData part unconditionally when the cli is
traversing the MIME tree, and return it as a "child" of what would
otherwise be a leaf in the tree.
Unfortunately, this also breaks the JSON output. We will fix that
next.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
When we are indexing, we should treat SignedData parts the same way
that we treat a multipart object, indexing the wrapped part as a
distinct MIME object.
Unfortunately, this means doing some sort of cryptographic
verification whose results we throw away, because GMime doesn't offer
us any way to unwrap without doing signature verification.
I've opened https://github.com/jstedfast/gmime/issues/67 to request
the capability from GMime but for now, we'll just accept the
additional performance hit.
As we do this indexing, we also apply the "signed" tag, by analogy
with how we handle multipart/signed messages. These days, that kind
of change should probably be done with a property instead, but that's
a different set of changes. This one is just for consistency.
Note that we are currently *only* handling signedData parts, which are
basically clearsigned messages. PKCS#7 parts can also be
envelopedData and authEnvelopedData (which are effectively encryption
layers), and compressedData (which afaict isn't implemented anywhere,
i've never encountered it). We're laying the groundwork for indexing
these other S/MIME types here, but we're only dealing with signedData
for now.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
When checking cryptographic signatures, Notmuch relies on GMime to
tell it whether the certificate that signs a message has a valid User
ID or not.
If the User ID is not valid, then notmuch does not report the signer's
User ID to the user. This means that the consumer of notmuch's
cryptographic summary of a message (or of its protected headers) can
be confident in relaying the reported identity to the user.
However, some versions of GMime before 3.2.7 cannot report Certificate
validity for X.509 certificates. This is resolved upstream in GMime
at https://github.com/jstedfast/gmime/pull/90.
We adapt to this by marking tests of reported User IDs for
S/MIME-signed messages as known-broken if GMime is older than 3.2.7
and has not been patched.
If GMime >= 3.2.7 and certificate validity still doesn't work for
X.509 certs, then there has likely been a regression in GMime and we
should fail early, during ./configure.
To break out these specific User ID checks from other checks, i had to
split some tests into two parts, and reuse $output across the two
subtests.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Several functions in test/test-lib.sh used variable names that are
also used outside of those functions (e.g. $output and $expected are
used in many of the test scripts), but they are not expected to
communicate via those variables.
We mark those variables "local" within test-lib.sh so that they do not
get clobbered when used outside test-lib.
We also move the local variable declarations to beginning of each
function, to avoid weird gotchas with local variable declarations as
described in https://tldp.org/LDP/abs/html/localvar.html.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
strncmp looks for a prefix that matches, which is very much not what
we want here. This fixes the bug reported by Franz Fellner in
id:1588595993-ner-8.651@TPL520
In id:1588595993-ner-8.651@TPL520 Franz Fellner reported that tags
starting with 'attachment' are removed by 'notmuch reindex'. This is
probably related to the use of STRNCMP_LITERAL in
_notmuch_message_remove_indexed_terms.
GPGME has a strange failure mode when it is in offline mode, and/or
when certificates don't have any CRLs: in particular, it refuses to
accept the validity of any certificate other than a "root" cert.
This can be worked around by setting the `disable-crl-checks`
configuration variable for gpgsm.
I've reported this to the GPGME upstream at
https://dev.gnupg.org/T4883, but I have no idea how it will be
resolved. In the meantime, we'll just work around it.
Note that this fixes the test for verification of
id:smime-multipart-signed@protected-headers.example, because
multipart/signed messages are already handled correctly (one-part
PKCS#7 messages will get fixed later).
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Add a simple S/MIME SignedData message, taken from an upcoming draft
of
https://datatracker.ietf.org/doc/draft-autocrypt-lamps-protected-headers/
RFC 8551 describes a SignedData, a one-part clearsigned object that is
more resistant to common patterns of MTA message munging than
multipart/signed (but has the downside that it is only readable by
clients that implement S/MIME).
To make sure sure notmuch can handle this kind of object, we want to
know a few things:
Already working:
- Is the content of the SignedData object indexed? It actually is
right now because of dumb luck -- i think we're indexing the raw
CMS object and it happens to contain the cleartext of the message
in a way that we can consume it before passing it on to Xapian.
- Are we accidentally indexing the embedded PKCS#7 certificates? We
don't want to, and for some reason I don't understand, our indexing
is actually skipping the embedded certificates already. That's
good!
Still need fixing:
- do we know the MIME type of the embedded part?
- do we know that the message is signed?
- can notmuch-show read its content?
- can notmuch-show indicate the signature validity?
- can notmuch-reply properly quote and attribute content?
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
This test does exactly what it says on the tin. It expects JSON data
to be parseable by Python, at least.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
These tests describe some simple behavior we would expect to work if
we were to correctly index the cleartext of encrypted S/MIME messages
(PKCS#7 envelopedData).
Of course, they don't currently pass, so we mark them known-broken.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
When consuming a signed+encrypted S/MIME message generated by emacs,
we expect to see the same cryptographic properties for the message as
a whole. This is not done correctly yet, so the test is marked as
known broken.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
These sample messages are taken directly from the Protected Headers
draft:
https://www.ietf.org/id/draft-autocrypt-lamps-protected-headers-02.html
Note that this commit doesn't strictly pass the common git pre-commit
hook due to introducing some trailing whitespace. That's just the
nature of the corpus, though. We should have that trailing
whitespace, so I've made this commit with --no-verify.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
This is taken from the same Internet Draft that test/smime/ca.crt
comes from. See that draft for more details.
https://www.ietf.org/id/draft-dkg-lamps-samples-02.html#name-pkcs12-object-for-bob
We don't use it yet, but it will be used to decrypt other messages in
the test suite.
Note that we include it here with an empty passphrase, rather than
with the passphrase "bob" that it is supplied with in the I-D. The
underlying cryptographic material is the same, but this way we can
import cleanly into gpgsm without having a passphrase set on it (gpgsm
converts an empty-string passphrase into no passphrase at all on
import).
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Without this fix, we couldn't run both add_gnupg_home and
add_gpgsm_home in the same test script.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
No functional change.
We no longer need to identify the key and cert to mml-mode when
sending an S/MIME message, so making a copy of key+cert.pem to
test_suite.pem is superfluous. Get rid of the extra file.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
The documentation for message mode clearly states that EasyPG (which
uses GnuPG) is the default and recommended way to use S/MIME with
mml-secure:
[0] https://www.gnu.org/software/emacs/manual/html_node/message/Using-S_002fMIME.html
To ensure that this mode works, we just need to import the secret key
in question into gpgsm in addition to the public key. gpgsm should be
able pick the right keys+certificates to use based on To/From headers,
so we don't have to specify anything manually in the #secure mml tag.
The import process from the OpenSSL-preferred form (cert+secretkey) is
rather ugly, because gpgsm wants to see a PKCS#12 object when
importing secret keys.
Note that EasyPG generates the more modern Content-Type:
application/pkcs7-signature instead of application/x-pkcs7-signature
for the detached signature.
We are also obliged to manually set gpgsm's include-certs setting to 1
because gpgsm defaults to send "everything but the root cert". In our
weird test case, the certificate we're using is self-signed, so it
*is* the root cert, which means that gpgsm doesn't include it by
default. Setting it to 1 forces inclusion of the signer's cert, which
satisfies openssl's smime subcommand. See https://dev.gnupg.org/T4878
for more details.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
This CA is useful for test suites and the like, but is not an
actually-secure CA, because its secret key material is also published.
I plan to use it for its intended purpose in the notmuch test suite.
It was copied from this Internet Draft:
https://tools.ietf.org/id/draft-dkg-lamps-samples-01.html#name-certificate-authority-certi
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
GnuPG's gpgsm, like gpg, should always be used with --batch when it is
invoked in a non-interactive environment.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Starting with Emacs 27 the old `cl' implementation is finally
considered obsolete. Previously its use was strongly discouraged
at run-time but one was still allowed to use it at compile-time.
For the most part the transition is very simple and boils down to
adding the "cl-" prefix to some symbols. A few replacements do not
follow that simple pattern; e.g. `first' is replaced with `car',
even though the alias `cl-first' exists, because the latter is not
idiomatic emacs-lisp.
In a few cases we start using `pcase-let' or `pcase-lambda' instead
of renaming e.g. `first' to `car'. That way we can remind the reader
of the meaning of the various parts of the data that is being
deconstructed.
An obsolete `lexical-let' and a `lexical-let*' are replaced with their
regular variants `let' and `let*' even though we do not at the same
time enable `lexical-binding' for that file. That is the right thing
to do because it does not actually make a difference in those cases
whether lexical bindings are used or not, and because this should be
enabled in a separate commit.
We need to explicitly depend on the `cl-lib' package because Emacs
24.1 and 24.2 lack that library. When using these releases we end
up using the backport from GNU Elpa.
We need to explicitly require the `pcase' library because
`pcase-dolist' was not autoloaded until Emacs 25.1.
This test extracts values from a (key,value) map where multiple entries
can have the same key, and the entries are sorted by key, but not by
value. The test incorrectly assumes that the values will be sorted as
well, so sort the output.
Xapian 1.4 is over 3 years old now (1.4.0 released 2016-06-24),
and 1.2 has been deprecated in Notmuch version 0.27 (2018-06-13).
Xapian 1.4 supports compaction, field processors and retry locking;
conditionals checking compaction and field processors were removed
but user may want to disable retry locking at configure time so it
is kept.
'qsx' reported a bug on #notmuch with notmuch-dump and large stored
queries. This test will pass (on my machine) if the value of `repeat'
is made smaller.
Reported-By: Thomas Schneider <qsx@chaotikum.eu>
In particular, timestamps beyond 2038 could overflow the sprinter
interface on systems where time_t is 64-bit but 'int' is a signed 32-bit
integer type.
The entire python-cffi test suite is considered as a single test at
the level of the notmuch test suite. This might or might not be ideal,
but it gets them run.
We want freeing the returned stream to also free these underlying
objects. Compare tests/test-filters.c in the gmime 3.2.x source, which
uses this same idiom.
Thanks to James Troup for the report and the fix.
When showing or replying to a message that has been mangled in transit
by an MTA in the "Mixed up" way, notmuch should instead use the
repaired form of the message.
Tracking the repaired GMimeObject for the lifetime of the mime_node so
that it is cleaned up properly is probably the trickiest part of this
patch, but the choices here are based on the idea that the
mime_node_context is the memory manager for the whole mime_node tree
in the first place, so new GMimeObject tree created on-the-fly during
message parsing should be disposed of in the same place.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
When encountering a message that has been mangled in the "mixed up"
way by an intermediate MTA, notmuch should instead repair it and index
the repaired form.
When it does this, it also associates the index.repaired=mixedup
property with the message. If a problem is found with this repair
process, or an improved repair process is proposed later, this should
make it easy for people to reindex the relevant message. The property
will also hopefully make it easier to diagnose this particular problem
in the future.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Some MTAs mangle e-mail messages in transit in ways that are
repairable.
Microsoft Exchange (in particular, the version running today on
Office365's mailservers) appears to mangle multipart/encrypted
messages in a way that makes them undecryptable by the recipient.
I've documented this in section 4.1 "Mixed-up encryption" of draft -00
of
https://tools.ietf.org/html/draft-dkg-openpgp-pgpmime-message-mangling
Fortunately, it's possible to repair such a message, and notmuch can
do that so that a user who receives an encrypted message from a user
of office365.com can still decrypt the message.
Enigmail already knows about this particular kind of mangling. It
describes it as "broken PGP email format probably caused by an old
Exchange server", and it tries to repair by directly changing the
message held by the user. if this kind of repair goes wrong, the
repair process can cause data loss
(https://sourceforge.net/p/enigmail/bugs/987/, yikes).
The tests introduced here are currently broken. In subsequent
patches, i'll introduce a non-destructive form of repair for notmuch
so that notmuch users can read mail that has been mangled in this way,
and the tests will succeed.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
When we notice a legacy-display part during indexing, it makes more
sense to avoid indexing it as part of the message body.
Given that the protected subject will already be indexed, there is no
need to index this part at all, so we skip over it.
If this happens during indexing, we set a property on the message:
index.repaired=skip-protected-headers-legacy-display
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Make use of the previous changes to fast-forward past any
legacy-display parts during "notmuch show" and "notmuch reply".
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Enigmail generates a "legacy-display" part when it sends encrypted
mail with a protected Subject: header. This part is intended to
display the Subject for mail user agents that are capable of
decryption, but do not know how to deal with embedded protected
headers.
This part is the first child of a two-part multipart/mixed
cryptographic payload within a cryptographic envelope that includes
encryption (that is, it is not just a cleartext signed message). It
uses Content-Type: text/rfc822-headers.
That is:
A └┬╴multipart/encrypted
B ├─╴application/pgp-encrypted
C └┬╴application/octet-stream
* ╤ <decryption>
D └┬╴multipart/mixed; protected-headers=v1 (cryptographic payload)
E ├─╴text/rfc822-headers; protected-headers=v1 (legacy-display part)
F └─╴… (actual message body)
In discussions with jrollins, i've come to the conclusion that a
legacy-display part should be stripped entirely from "notmuch show"
and "notmuch reply" now that these tools can understand and interpret
protected headers.
You can tell when a message part is a protected header part this way:
* is the payload (D) multipart/mixed with exactly two children?
* is its first child (E) Content-Type: text/rfc822-headers?
* does the first child (E) have the property protected-headers=v1?
* do all the headers in the body of the first child (E) match
the protected headers in the payload part (D) itself?
If this is the case, and we already know how to deal with the
protected header, then there is no reason to try to render the
legacy-display part itself for the user.
Furthermore, when indexing, if we are indexing properly, we should
avoid indexing the text in E as part of the message body.
'notmuch reply' is an interesting case: the standard use of 'notmuch
reply' will end up omitting all mention of protected Subject:.
The right fix is for the replying MUA to be able to protect its
headers, and for it to set them appropriately based on headers found
in the original message.
If a replying MUA is unable to protect headers, but still wants the
user to be able to see the original header, a replying MUA that
notices that the original message's subject differs from the proposed
reply subject may choose to include the original's subject in the
quoted/attributed text. (this would be a stopgap measure; it's not
even clear that there is user demand for it)
This test suite change indicates what we want to happen for this case
(the tests are currently broken), and includes three additional TODO
suggestions of subtle cases for anyone who wants to flesh out the test
suite even further. (i believe all these cases should be already
fixed by the rest of this series, but haven't had time to write the
tests for the unusual cases)
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Previously, when all tests were skipped on a test file, there were
no indication of this in the final results aggregate-results.sh
printed.
Now count of the files where all tests were skipped is printed.
This is the result of running:
$ uncrustify --replace --config ../devel/uncrustify.cfg *.cc *.c *.h
in the test directory.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
This removes the dependency of this test script on gdb, and
considerably speeds up the running of the tests.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
These can be used e.g. to override return values for functions, in
place of the existing scripting of gdb.
This prepends to LD_PRELOAD rather than clobbering it, thanks to a
suggestion from Tomi Ollila.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
- all variables in $((...)) without leading $
- all comparisons use -gt, -eq or -ne
- no -a nor -o inside [ ... ] expressions
- all indentation levels using one tab
Dropped unnecessary empty string check when reading results files.
Replaced pluralize() which was executed in subshell with
pluralize_s(). pluralize_s sets $s to 's' or '' based on value of
$1. Calls to pluralize_s are done in context of current shell, so
no forks to subshells executed.
When the user knows the signer's key, we want "notmuch show" to be
able to verify the signature of an encrypted and signed message
regardless of whether we are using a stashed session key or not.
I wrote this test because I was surprised to see signature
verification failing when viewing some encrypted messages after
upgrading to GPGME 1.13.0-1 in debian experimental.
The added tests here all pass with GPGME 1.12.0, but the final test
fails with 1.13.0, due to some buggy updates to GPGME upstream: see
https://dev.gnupg.org/T3464 for more details.
While the bug needs to be fixed in GPGME, notmuch's test suite needs
to make sure that GMime is doing what we expect it to do; i was a bit
surprised that it hadn't caught the problem, hence this patch.
I've fixed this bug in debian experimental with gpgme 1.13.0-2, so the
tests should pass on any debian system. I've also fixed it in the
gpgme packages (1.13.0-2~ppa1) in the ubuntu xenial PPA
(ppa:notmuch/notmuch) that notmuch uses for Travis CI.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Protected subject lines were being emitted in reply when the cleartext
of documents was indexed. create_reply_message() was pulling the
subject line from the index, rather than pulling it from the
GMimeMessage object that it already has on hand.
This one-line fix to notmuch-reply.c solves that problem, and doesn't
cause any additional tests to fail.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
These tests are currently broken! When a protected subject is indexed
in the clear, it leaks in the reply headers :(
For emacs, we set up separate tests for when the protected header is
indexed in the clear and when it is unindexed. neither case should
leak, but the former wasn't tested yet.
We will fix the two broken tests in a subsequent patch.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
This tests notmuch-show; headers appear appropriately based on the
setting of notmuch-crypto-process-mime.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
We initially test only notmuch-search; tests for other functionality
come in different patchsets later.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
We want to make sure that internally-forwarded messages don't end up
"bubbling up" when they aren't actually the cryptographic payload.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
This test scans for all the possible protected headers (including
bogus/broken ones) that are present in the protected-headers corpus,
trying to make sure that only the ones that are not broken or
malformed show up in a search after re-indexing.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Up to this point, we've tested protected headers on messages that have
either been encrypted or signed, but not both.
This adds a couple tests of signed+encrypted messages, one where the
subject line is masked (outside subject line is "Subject Unavailable")
and another where it is not (outside Subject: matches inner Subject:)
See the discussion at
https://dkg.fifthhorseman.net/blog/e-mail-cryptography.html#protected-headers
for more details about the nuances between signed, stripped, and
stubbed headers.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
When indexing the cleartext of an encrypted message, record any
protected subject in the database, which should make it findable and
visible in search.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Now that we can decrypt headers, we want to make sure that clients
using "notmuch reply" to prepare a reply don't leak cleartext in their
subject lines. In particular, the ["reply-headers"]["Subject"] should
by default show the external Subject.
A replying MUA that intends to protect the Subject line should show
the user the Subject from ["original"]["headers"]["Subject"] instead
of using ["reply-headers"]["Subject"].
This minor asymmetry with "notmuch show" is intentional. While both
tools always render the cleartext subject line when they know it (in
["headers"]["Subject"] for "notmuch show" and in
["original"]["headers"]["Subject"] for "notmuch reply"), "notmuch
reply" should never leak something that should stay under encrypted
cover in "reply-headers".
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Make sure that we emit the correct cryptographic envelope status for
cleartext signed messages.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Adding another test to ensure that we handle protected headers
gracefully when no external subject is present.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
The header-mask member of the per-message crypto object allows a
clever UI frontend to mark whether a header was protected (or not).
And if it was protected, it contains enough information to show useful
detail to an interested user. For example, an MUA could offer a "show
what this message's Subject looked like on the wire" feature in expert
mode.
As before, we only handle Subject for now, but we might be able to
handle other headers in the future.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Amended by db: tweaked schemata notation.
Correctly fix the two outstanding tests so that the protected (hidden)
subject is properly reported.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Here we add several variant e-mail messages, some of which have
correctly-structured protected headers, and some of which do not. The
goal of the tests is to ensure that the right protected subjects get
reported.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
This makes it easier to write fairly compact, readable tests of json
output, without needing to sanitize away parts that we don't care
about.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
This paves the way for emitting protected headers after verification
and decryption, because it means that the headers will only be emitted
after the body has been parsed.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
In certain cases of test suite failure, the summary report was not
being printed. In particular, any failure on the parallel test suite,
and any aborted test in the serialized test suite would end up hiding
the summary.
It's better to always show the summary where we can (while preserving
the return code). If we do abort due to this high-level failure,
though, we should also announce to the user that we're doing so as
close to the end of the process as possible, to make it easier to find
the problem.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
This allows MUAs that don't want to think about per-mime-part
cryptographic status to have a simple high-level overview of the
message's cryptographic state.
Sensibly structured encrypted and/or signed messages will work fine
with this. The only requirement for the simplest encryption + signing
is that the message have all of its encryption and signing protection
(the "cryptographic envelope") in a contiguous set of MIME layers at
the very outside of the message itself.
This is because messages with some subparts signed or encrypted, but
with other subparts with no cryptographic protection is very difficult
to reason about, and even harder for the user to make sense of or work
with.
For further characterization of the Cryptographic Envelope and some of
the usability tradeoffs, see here:
https://dkg.fifthhorseman.net/blog/e-mail-cryptography.html#cryptographic-envelope
When we have not been able to evaluate the signature status of a given
MIME part, showing a content-free (and interaction-free) "[ Unknown
signature status ]" button doesn't really help the user at all, and
takes up valuable screen real-estate.
A visual reminder that a given message is *not* signed isn't helpful
unless it is always present, in which case we'd want to see "[ Unknown
signature status ]" buttons on all messages, even ones that don't have
a signing structure, but i don't think we want that.
Amended by db to drop the unused initialization of 'label'
To aid in diagnosing test suite tooling that interacts poorly with
coreutils' timeout, it's handy to be able to bypass it entirely.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
These restrictions are meant to prevent incompatibilities with the
Xapian query parser (which will split at non-word characters) and
clashes with future notmuch builtin fields.
We don't do anything with this configuration information information
yet, but nonetheless add a couple of regression tests to make sure we
don't break standard functionality when we do use the configuration
information.
Done via $COLORS_WITHOUT_TTY environment variable as passing options
to commands through parallel(1) does not look trivial.
Reorganized color checking in test-lib.sh a bit for this (perhaps
were not fully necessary but rest still an improvement):
- color checking commands in subshell are not run before arg parsing
(args may disable colors with --no-color)
- [ -t 1 ] is checked before forking subshell
Added initialization and checking of smtp_dummy_port
like it was done with smtp_dummy_pid.
Made those function-local variables.
One 8 spaces to tab consistency conversion.
And last, but definitely not least; while doing above
noticed that there were quite a few double-quoted strings
where $@ was in the middle of it -- replaced those with $*
for robustness ("...$@..." expands params to separate words,
"...$*..." params expands to single word).
Without this stdin may be anything that parent process provided for it.
Test processes might have tried to read something from it, which would
have caused undeterministic behavior.
E.g. gdb(1) tries to do tty related ioctls on fd 0 (and fd 1 and fd 2,
but those are redirected to 'test.output' before test runs).
To the best of my understanding, this original behaviour was what
Carl's homebrew parser produced. With commit 86f89385 Austin switched
to using GMime (2.6). This produced arguably worse results, but since
the input was bad, we could live with it. Now with GMime 3.0 we are
getting the original results again, and there is no reason to consider
this test broken.
When a parallel build fails (or when it times out, if timeout is
present), the test suite should not blithely succeed. Catch these
failures and at least report them.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
The current 2 minute timeout is reasonable, but to exercise the test
suite or induce timeout failures, we might want to make it shorter.
This makes it configurable so you can run (for example):
make check NOTMUCH_TEST_TIMEOUT=10s
We stick with the default of 2m.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
FINGERPRINT is already exported by add_gnupg_home, so this is
unnecessary. This change also happens to get rid of the superfluous
check-trustdb spew from the test suite that looked like this:
gpg: checking the trustdb
gpg: marginals needed: 3 completes needed: 1 trust model: pgp
gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
We did not have a test showing what message decryption looks like
within notmuch-emacs. This change gives us a baseline for future work
on the notmuch-emacs interface.
This differs from previous revisions of this patch in that it should
be insensitive to the order in which the local filesystem readdir()s
the underlying maildir.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
If either the moreutils or GNU parallel utility are available, run all
tests in parallel. On my eight core machine this makes for a ~x7
speed-up in the full test suite (1m24s -> 12s).
The design of the test suite makes this parallelization trivial.
The add_email_corpus test utility includes logic that tries to re-use
an index of the corpus if available. This was seemingly done as an
optimization, so that every test that uses the corpus didn't have to
create it's own index of the corpus. However, this has the perverse
side effect of entangling tests together, and breaks parallelization.
Forcing each test to do it's own index does increase the overall time
of the test slightly (~6%), but this will be more than made up for in
the next patch that introduces paraellization.
The typical use case for gpg is that if you control a secret key, you
mark it with "ultimate" ownertrust.
The opaque --import-ownertrust mechanism is GnuPG's standard mechanism
to set up ultimate ownertrust (the ":6:" means "ultimate", for
whatever reason).
We adjust the test suite to match this change, inverting the sense of
one test: since the default is now that the user ID of the suite's own
key is valid, we change the test to make sure that the user ID is not
emitted when it is *not* valid.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
The user ID on the self-test is a little bit clunky-looking. It also
may end up showing up elsewhere in the test suite. Centralizing the
user ID in one place should make it easier to handle if it ever
changes, and should make tests easier to read.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
This is a subtle difference, but the output of notmuch shouldn't ever
change based on ownertrust itself -- notmuch is intended to show valid
User IDs, and to avoid showing invalid User IDs.
It so happens that setting ownertrust of a key to ultimate sets all
associated user IDs to "full" validity, so the test is correct, but
just misnamed.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Previously if the input was exactly a multiple of the internal buffer
size, notmuch would attempt to fwrite nothing to stdout, but still
expected fwrite to return 1, causing a failure that looked like this:
$ notmuch show --format=raw id:87o96f1cya.fsf@codeaurora.org
...entire message shown as expected..
Error: Write failed
$ echo $?
1
To fix the problem don't call fwrite at all when there's nothing to
write.
Amended by db: add some tests of message sizes likely to cause this
problem.
This drops "file" from mime_node_context and just uses a local
variable. It also uses the new gzip aware utility routines recently
added to util/gmime-extra.c. The use of gzopen / gzfile in addition is
a bit icky, but the choice is between that, and providing yet another
readline implimentation that understands GMime streams.
Rather than storing the lower level stdio FILE object, we store a
GMime stream. This allows both transparent decompression, and passing
the stream into GMime for parsing. As a side effect, we can let GMime
close the underlying OS stream (indeed, that stream isn't visible here
anymore).
This change is enough to get notmuch-{new,search} working, but there is still
some work required for notmuch-show, to be done in a following commit.
note that "notmuch-show for message with invalid From" is still broken
in T310-emacs.sh. It would be good to debug what's going on there and
try to get it fixed!
signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Note that we do keep ignoring the gpg_path configuration option,
though, to avoid breakage of existing installations. It is ignored
like any other unknown configuration option, but we at least document
that it is ignored so that people who find it in their legacy configs
can know that it's safe to drop.
signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Unsigned encrypted mail shows up with a weird empty signature list.
If we successfully decrypted and there was no signature in it, we
should just not show a sigstatus at all.
The documentation for g_mime_decrypt_result_get_signatures says:
a GMimeSignatureList or NULL if the stream was not signed.
If a test has added a GnuPG homedir, it may well want to know the
fingerprint. This saves us from having to redefine this magic string
in multiple places when more tests eventually use the GnuPG homedir.
The new `body:` field (in Xapian terms) or prefix (in slightly
sloppier notmuch) terms allows matching terms that occur only in the
body.
Unprefixed query terms should continue to match anywhere (header or
body) in the message.
This follows a suggestion of Olly Betts to use the facility (since
Xapian 1.0.4) to add the same field with multiple prefixes. The double
indexing of previous versions is thus replaced with a query time
expension of unprefixed query terms to the various prefixed
equivalent.
Reindexing will be needed for 'body:' searches to work correctly;
otherwise they will also match messages where the term occur in
headers (demonstrated by the new tests in T530-upgrade.sh)
Add test of forwarding messages from within emacs.
The first test checks that a references header is properly
added to the new message. The second test checks that the
send-hook of the forwarding message adds a forwarded-tag
to the original message.
Thanks to plujon for pointing out this problem on IRC. The underlying
issue is that the quotes are stripped before the field processors get
the query string, and the heuristic for putting them back is not quite
right.
The exact error messages returned by regerror() aren't standardized;
relying on them isn't portable. Thus, add a a prefix to make clear that
the subsequent message is a regexp parsing error, and only look for this
prefix in the test suite, ignoring the rest of the message.
POSIX doesn't specify the flushing behaviour of the STDOUT stream, so
it's invalid to assume a particular order between the stdout and stderr
output. The current test breaks on musl due to this.
I can't figure out how checking the sign of a bool ever worked. The
following program demonstrates the problem (i.e. for me it prints 1).
#include <stdio.h>
#include <stdbool.h>
int main(int argc, char **argv) {
bool x;
x = -1;
printf("x = %d\n", x);
}
This seems to be mandated by the C99 standard 6.3.1.2.
When generating a reply message, if the user was the originator and
only recipient of the original message, include the user as a
recipient of the reply.
b31e44c678 introduced message-id-parse
as a new binary created by the test suite. It shows up as something
additional to git, but git ought to know to ignore it.
As reported by Sean Whitton, there are mailers (in particular the
Debian Bug Tracking System) that have sensible In-Reply-To headers,
but un-useful-for-notmuch References (in particular with the BTS, the
oldest reference is last). I looked at a sample of about 200K
messages, and only about 0.5% these had something other than a single
message-id in In-Reply-To. On this basis, if we see a single
message-id in In-Reply-To, consider that as authoritative.
The idea is that if a message-id parses with this function, the MUA
generating it was probably sane, and in particular it's probably safe
to use the result as a parent from In-Reply-to.
In a future commit, we will start trusting In-Reply-To's when they
look sane (i.e. a single message-id). Modify these tests so they will
keep passing (i.e. keep choosing References) when that happens.
The current scheme of choosing the replyto (i.e. the default parent
for threading purposes) does not work well for mailers that put
the oldest Reference last.
We (finally) implement the XXX comment. It requires a bit of care not
to reparent all of the possible toplevel messages.
_notmuch_messages_has_next is not ready to be a public function yet,
since it punts on the mset case. We know in the one case it is called,
the notmuch_messages_t is just a regular list / iterator.
This is mainly to lay out the structure of the final code. The problem
isn't really solved yet, although some very simple cases are
better (hence the fixed test). We need two passes through the messages
because we need to be careful not to re-parent too many messages and
end up without any toplevel messages.
For non-root messages, this should not should anything currently, as
the messages are already added in date order. In the future we will
add some non-root messages in a second pass out of order and the
sorting will be useful. It does fix the order of multiple
root-messages (although it is overkill for that).
This documents the bug discussed at
id:87efgmmysi.fsf@len.workgroup
The underlying issue is that the reply to a ghost (missing) message is
falsely classified as a root message in _resolve_thread_relationships.
There are two pairs of tests; in each case the the first test is
simpler / more robust, but also easier to fool.
There are 3 threads here, two synthetic, and one anonymized one using
data from Gregor. They test various aspects of thread
ordering/construction in the presence of replies to ghost messages.
This clarifies that the breakage seen with Xapian 1.4.6 does not have
to do with "funny" tags.
This test is "known broken", but only with xapian 1.4.6, and there's
curently no convenient way to mark that.
Messages that contain Windows-1252 are frequently mislabeled as ISO
8859-1, which may result in non-printable characters when displaying
the message. The test asserts that such characters (in this case
curved quotes) are displayed correctly.
Add fancy new feature, which makes "notmuch show" capable of actually
indexing messages that it just decrypted.
This enables a workflow where messages can come in in the background
and be indexed using "--decrypt=auto". But when showing an encrypted
message for the first time, it gets automatically indexed.
This is something of a departure for "notmuch show" -- in particular,
because it requires read/write access to the database. However, this
might be a common use case -- people get mail delivered and indexed in
the background, but only want access to their secret key to happen
when they're directly interacting with notmuch itself.
In such a scenario, they couldn't search newly-delivered, encrypted
messages, but they could search for them once they've read them.
Documentation of this new feature also uses a table form, similar to
that found in the description of index.decrypt in notmuch-config(1).
A notmuch UI that wants to facilitate this workflow while also
offering an interactive search interface might instead make use of
these additional commands while the user is at the console:
Count received encrypted messages (if > 0, there are some things we
haven't yet tried to index, and therefore can't yet search):
notmuch count tag:encrypted and \
not property:index.decryption=success and \
not property:index.decryption=failure
Reindex those messages:
notmuch reindex --try-decrypt=true tag:encrypted and \
not property:index.decryption=success and \
not property:index.decryption=failure
Wherever the test relies on directories being scanned, this option
should be used to avoid skipping them due to mtimes on directories
matching the database.
Most of these just check that adding the flag does not break existing
functionality. The one test that does check the full-scan
functionality had to be rewritten to output debugging info.
This way, one can build for a different Ruby than $PATH/ruby
(e. g. different versions, or Ruby in other paths).
Signed-off-by: Thomas Schneider <qsx@chaotikum.eu>
This change allows queries of the form
thread:{from:me} and thread:{from:jian} and not thread:{from:dave}
This is still somewhat brute-force, but it's a big improvement over
both the shell script solution and the previous proposal [1], because it
does not build the whole thread structure just generate a
query. A further potential optimization is to replace the calls to
notmuch with more specialized Xapian code; in particular it's not
likely that reading all of the message metadata is a win here.
[1]: id:20170820213240.20526-1-david@tethera.net
Correct URLs that have crept into the notmuch codebase with http://
when https:// is possible.
As part of this conversion, this changeset also indicates the current
preferred upstream URLs for both gmime and sup. the new URLs are
https-enabled, the old ones are not.
This also fixes T310-emacs.sh, thanks to Bremner for catching it.
We expect this to give the same answer as the non-regexp subject
search. It does not because the regexp search relies on the value
slot, which currently contains only one subject.
These roughly replicate the equivalent C tests, although they rely on
the database state created by the former tests, since the python
bindings currently provide read-only access to properties.
The previous loop handling code chooses the last message in the
message list, which turns out to be the last in date order.
See the comment in _notmuch_thread_create.
In some cases (e.g. when building a publicly-visible e-mail archive)
it doesn't make any sense to restrict visibility of the message to the
current user account.
This adds a --world-readable boolean option for "notmuch insert", so
that those who want to archive their mail publicly can feed their
archiver with:
notmuch insert --world-readable
Other local delivery agents (postfix's local, and dovecot's lda) all
default to delivery in mode 0600 rather than relying on the user's
umask, so this fix doesn't change the default.
Also, this does not override the user's umask. if the umask is
already set tight, it will not become looser as the result of passing
--world-readable.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
This brings the --decrypt argument to "notmuch reply" into line with
the other --decrypt arguments (in "show", "new", "insert", and
"reindex"). This patch is really just about bringing consistency to
the user interface.
We also use the recommended form in the emacs MUA when replying, and
update test T350 to match.
We also expand tab completion for it, update the emacs bindings, and
update T350, T357, and T450 to match.
Make use of the bool-to-keyword backward-compatibility feature.
We might change some notmuch command line tools that used to be
booleans into keyword arguments.
In that case, there are some legacy tools that will expect to be able
to do "notmuch foo --bar" instead of "notmuch foo --bar=baz".
This patch makes it possible to support that older API, while
providing a warning and an encouragement to upgrade.
We adopt a pythonic idiom here with an optional argument, rather than
exposing the user to the C indexopts object directly.
This now includes a simple test to ensure that the decrypt_policy
argument works as expected.
This test passes with older versions of Xapian as well, because
neither query returns any results.
This should resolve the travis build failure at
https://travis-ci.org/notmuch/notmuch/builds/318571658
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
The current behaviour is at best under-documented. The modified test in
T470-missing-headers.sh previously relied on printf doing the right
thing with NULL, which seems icky.
The use of talloc_strdup here is probably overkill, but it avoids
having to enforce that thread->authors is never mutated outside
_resolve_thread_authors_string.
If we're building against a version of Xapian that doesn't offer
retrying the lock, we should be honest and describe the tests as
broken, rather than marking them as missing a test prerequisite.
missing test prerequisites should be for specific components of the
test harness that are missing, not for the backend library notmuch
uses.
Named queries don't work without Xapian FieldProcessor. Rather than
silently skipping them, we should explictly mark them as broken when
building against an older version of Xapian.
Previously, the test suite had simply silently skipped the absolute
date test if we're using an archaic version of Xapian. For
correctness, we should instead mark the test as broken.
This also changes from string to numeric comparison when checking
NOMTUCH_HAVE_XAPIAN_FIELD_PROCESSOR for consistency with other tests.
The old name has a bit of a feeling of hungarian notation. Also many
generators in the core are named with the suffix "s" to indicate
iterables: dict.items, dict.keys for example.
This test will pass if either the notmuch show mitigation code is
working correctly, or upstream emacs mime handling code has it's own
fix for https://bugs.gnu.org/28350.
When i'm trying to understand a message signature, i care that i know
who it came from (the "validity" of the identity associated with the
key), *not* whether i'm willing to accept the keyholder's other
identity assertions (the "trust" associated with the certificate).
We've been reporting User ID information based on the "trust"
associated with the certificate, because GMime didn't clearly expose
the validity of the User IDs.
This change relies on fixes made in GMime 3.0.3 and later which
include https://github.com/jstedfast/gmime/pull/18.
Here's the configuration choice for people who want a cleartext index,
but don't want stashed session keys.
Interestingly, this "nostash" decryption policy is actually the same
policy that should be used by "notmuch show" and "notmuch reply",
since they never modify the index or database when they are invoked
with --decrypt.
We take advantage of this parallel to tune the behavior of those
programs so that we're not requesting session keys from GnuPG during
"show" and "reply" that we would then otherwise just throw away.
If you're going to store the cleartext index of an encrypted message,
in most situations you might just as well store the session key.
Doing this storage has efficiency and recoverability advantages.
Combined with a schedule of regular OpenPGP subkey rotation and
destruction, this can also offer security benefits, like "deletable
e-mail", which is the store-and-forward analog to "forward secrecy".
But wait, i hear you saying, i have a special need to store cleartext
indexes but it's really bad for me to store session keys! Maybe
(let's imagine) i get lots of e-mails with incriminating photos
attached, and i want to be able to search for them by the text in the
e-mail, but i don't want someone with access to the index to be
actually able to see the photos themselves.
Fret not, the next patch in this series will support your wacky
uncommon use case.
There are some situations where the user wants to get rid of the
cleartext index of a message. For example, if they're indexing
encrypted messages normally, but suddenly they run across a message
that they really don't want any trace of in their index.
In that case, the natural thing to do is:
notmuch reindex --decrypt=false id:whatever@example.biz
But of course, clearing the cleartext index without clearing the
stashed session key is just silly. So we do the expected thing and
also destroy any stashed session keys while we're destroying the index
of the cleartext.
Note that stashed session keys are stored in the xapian database, but
xapian does not currently allow safe deletion (see
https://trac.xapian.org/ticket/742).
As a workaround, after removing session keys and cleartext material
from the database, the user probably should do something like "notmuch
compact" to try to purge whatever recoverable data is left in the
xapian freelist. This problem really needs to be addressed within
xapian, though, if we want it fixed right.
The new "auto" decryption policy is not only good for "notmuch show"
and "notmuch reindex". It's also useful for indexing messages --
there's no good reason to not try to go ahead and index the cleartext
of a message that we have a stashed session key for.
This change updates the defaults and tunes the test suite to make sure
that they have taken effect.
When showing a message, if the user doesn't specify --decrypt= at all,
but a stashed session key is known to notmuch, notmuch should just go
ahead and try to decrypt the message with the session key (without
bothering the user for access to their asymmetric secret key).
The user can disable this at the command line with --decrypt=false if
they really don't want to look at the e-mail that they've asked
notmuch to show them.
and of course, "notmuch show --decrypt" still works for accessing the
user's secret keys if necessary.
If the user doesn't specify --decrypt= at all, but a stashed session
key is known to notmuch, when replying to an encrypted message,
notmuch should just go ahead and decrypt.
The user can disable this at the command line with --decrypt=false,
though it's not clear why they would ever want to do that.
This new automatic decryption policy should make it possible to
decrypt messages that we have stashed session keys for, without
incurring a call to the user's asymmetric keys.
Future patches in this series will introduce new policies; this merely
readies the way for them.
We also convert --try-decrypt to a keyword argument instead of a boolean.
the command-line interface for indexing (reindex, new, insert) used
--try-decrypt; and the configuration records used index.try_decrypt.
But by comparison with "show" and "reply", there doesn't seem to be
any reason for the "try" prefix.
This changeset adjusts the command-line interface and the
configuration interface.
For the moment, i've left indexopts_{set,get}_try_decrypt alone. The
subsequent changeset will address those.
If the version of GMime we're building against doesn't support session
key extraction or re-use, mark the tests that rely on session key
capabilities as known-broken.
This should resolve test suite failures on ubuntu trusty and debian
jessie and earlier, which have GMime 2.6.20 -- session key support was
introduced in GMime 2.6.21.
If you've got a notmuch dump that includes stashed session keys for
every decrypted message, and you've got your message archive, you
should be able to get back to the same index that you had before.
Here we add a simple test that give some flavor of how that works.
In 8e7fb88237 Jani replaced the use of $(pwd -P) to find the
TEST_DIRECTORY in order to better support out of tree
builds. Unfortunately the performance-tests need a different value for
the variable and were thus broken.
This commit splits out the setting of this variable for the two sets
of tests. Performance tests still don't work out of tree, because
the handling of the downloaded corpus needs to be updated.
backup_database() and restore_database() used to store the backups in
the test specific temporary directory, through the current working
directory being there. Commit 8e7fb88237 ("test: use source and
build paths in test-lib-common.sh") started using a test specific
backup directories under the build tree test directory. This was in
error. Switch back to the old location, but using paths to the
location instead of relying on current working directory.
Reported by Daniel Kahn Gillmor <dkg@fifthhorseman.net>.
Enable override of the index.try_decrypt setting on a per-run basis
when invoking "notmuch reindex". This allows the possibility of (for
example) an emacs keybinding that adds the cleartext of the currently
shown decrypted message to the index, making it searchable in the
future.
It also enables one-time indexing of all messages matching some query,
like so:
notmuch reindex tag:encrypted and\
not property:index.decryption=success and\
from:alice@example.org
We also update the documentation and tab completion, and add a few
more tests.
Enable override of the index.try_decrypt setting on a per-message
basis when invoking "notmuch insert".
We also update the documentation and tab completion, and add more tests.
Enable override of the index.try_decrypt setting during "notmuch new"
on a per-invocation basis.
We update the documentation and tab completion, and also add a test.
Subsequent patches may want to send GNU-style --long-arguments to
notmuch new in the test suite, in particular when invoking
emacs_fcc_message. This changeset makes that possible.
The primary motivation here is to fix TMP_DIRECTORY cleanup prior to
running each test when the current working directory is not the test
subdirectory. Tests with failures would leave their TMP_DIRECTORY
directory behind for debugging, and repeated out-of-tree test runs
would have old temp directories. (This lead to e.g. T310-emacs.sh
hanging because emacs would prompt for overwriting files.)
We remove the likely anyway defunct --root test option while at it,
just to be on the safe side when doing 'rm -rf' on the TMP_DIRECTORY.
Turns out round trip tests didn't really round trip anything. Broken
by yours truly in 971cdc72cd ("test: make it possible to have
multiple corpora"). Ooops.
C99 stdbool turned 18 this year. There really is no reason to use our
own, except in the library interface for backward
compatibility. Convert the cli and test binaries to stdbool.
Several changes at once, just to not have to change the same lines
several times over:
- Use designated initializers to initialize opt desc arrays.
- Only initialize the needed fields.
- Remove arg_id (short options) as unused.
- Replace opt_type and output_var with several type safe output
variables, where the output variable being non-NULL determines the
type. Introduce checks to ensure only one is set. The downside is
some waste of const space per argument; this could be saved by
retaining opt_type and using a union, but that's still pretty
verbose.
- Fix some variables due to the type safety. Mostly a good thing, but
leads to some enums being changed to ints. This is pedantically
correct, but somewhat annoying. We could also cast, but that defeats
the purpose a bit.
- Terminate the opt desc arrays using {}.
The output variable type safety and the ability to add new fields for
just some output types or arguments are the big wins. For example, if
we wanted to add a variable to set when the argument is present, we
could do so for just the arguments that need it.
Beauty is in the eye of the beholder, but I think this looks nice when
defining the arguments, and reduces some of the verbosity we have
there.
This imports a message with ISO-8859-2 encoded characters, then opens
the database using the python bindings. We peek through all mesage
parts, afterwards print the message id.
Signed-off-by: Florian Klink <flokli@flokli.de>
Signed-off-by: Andreas Rammhold <andreas@rammhold.de>
Commit a7964c86d1 ("emacs: Sanitize authors and subjects in search
and show") added sanitization of header information for display. Do
the same for reply subjects.
This fixes the long-standing annoying artefact of certain versions of
mailman using tab as folding whitespace, leading to tabs in reply
subjects.
This is a logical followup to "lib: index the content type of
signature parts", which will make it easier to record the message
structure of all messages.
It's useful (*) to be able to easily find messages with certain types
of signatures. Having the mimetype: prefix searches fail for some
content types is also genuinely surprising (*). Index the content type
of signature parts.
While at it, switch to the gmime convenience constants for content and
signature part indexes.
*) At least for developers of email software!
Add known broken subtests for searching signed and encrypted messages
using mimetype: prefix search for the content-types of signed and
encrypted parts.
In [1] Mark showed that the the current code (d7a49e81) is not
consistent in it's handling of subjects of messages with duplicate
message-ids (or in notmuch-speak, of messages with multiple files).
notmuch-search uses indexing order and explicitedly preserves the
first. notmuch-show (apparently) uses alphabetical (or at least xapian
term order) of filenames. In a perfect world we would probably report
all subjects in the json output; at the very least we should be
consistent.
[1]: id:87378dny3d.fsf@qmul.ac.uk
In [1], Mark gave a test that was behaving strangly. This turns out to
be specific to reindexing. I suppose one could argue that picking the
lexicographically last file name is a defensible choice, but it's
almost as easy to take the first, which seems more intuitive. So mark
the current situation as broken.
[1]: id:1503859703-2973-1-git-send-email-markwalters1009@gmail.com
The existing test for notmuch search had the first in filename order
the same as the first indexed, which made it harder to understand what
the underlying behaviour is. Add a file with a lexicographically
smaller name, but later index time to clarify this.
The original intent of this test was to verify that notmuch show was
not crashing when the first file (where headers are being read from)
was deleted. Run the output through some sanitization so that as we
add and delete copies we don't have to update this test.
Changed "" quotes to '' as we're not supposed to dynamically
alter python program (via shell $variable expansion).
Added space to python program to match general python style.
Replaced $* with 'idiomatic' "$@" to serve as better example.
In [1], Vladimir Panteleev observed that the In-Reply-To and
References headers could be wrapped in the 'default' output format of
notmuch-reply, depending on the version of Emacs creating the
message. In my own experiments notmuch-reply sometimes wraps headers
with only one message-id if that message-id is long enough. However it
happens, this causes the previous approach using grep to fail.
Since I found the proposed unwrapping shell fragment in [1] a bit hard
to follow, I decided to write a little python script instead. Then
Tomi suggested a slight generalization of my script, and here we are.
[1] id:20170817175145.3204-7-notmuch@thecybershadow.net
On some system configurations, setting a breakpoint on the "add_file"
function then issuing "continue" in gdb causes the debugger to
seemingly jump over the add_file invocation. This results in a test
failure, as the "Handle files vanishing between scandir and add_file"
subtest expects add_file to be called and fail due to the vanishing
file. The compiler optimization level also plays a role - the problem
can be reproduced with CFLAGS having -O2 but not -Og.
This problem was observed manifesting as a test failure on Travis CI
configured with "dist: trusty" and "sudo: false". It was not
reproducible on a local Docker image of Travis' runtime environment,
so Travis' virtualization infrastructure likely plays a role as well.
* T050-new.sh: Breakpoint notmuch_database_add_message instead of
add_file to the same effect, and avoid bad gdb behaviour on Travis
CI.
Amended by db:
s/notmuch_database_add_message/notmuch_database_index_file/
Somehow the wrapper function doesn't work as a breakpoint; perhaps due
to inlining.
We reorder reading maildir flags to avoid overwriting 'new.tags'. The
inverted status of 'unread' means the maildir flag needs to be checked
a second time.
I backpedalled here on the idea of supporting 'new.tags' without
'unread' in the presence of maildir syncing. For files in 'new/', it
seems quite natural to tag them as 'unread'.
json.tool does not sort or otherwise normalize the order of JSON keys
in its output, which can result in test failures on some test systems.
Instead, use a one-line Python script passed to the interpreter
directly on its command line. Use sort_keys=True for json.dump to
ensure the key order is normalized. The script works with both Python
2 and 3.
* test/test-lib.sh: Update test_expect_equal_json.
We need a way to pass parameters to the indexing functionality on the
first index, not just on reindexing. The obvious place is in
notmuch_database_add_message. But since modifying the argument list
would break both API and ABI, we needed a new name.
I considered notmuch_database_add_message_with_params(), but the
functionality we're talking about doesn't always add a message. It
tries to index a specific file, possibly adding a message, but
possibly doing other things, like adding terms to an existing message,
or failing to deal with message objects entirely (e.g. because the
file didn't contain a message).
So i chose the function name notmuch_database_index_file.
I confess i'm a little concerned about confusing future notmuch
developers with the new name, since we already have a private
_notmuch_message_index_file function, and the two do rather different
things. But i think the added clarity for people linking against the
future libnotmuch and the capacity for using index parameters makes
this a worthwhile tradeoff. (that said, if anyone has another name
that they strongly prefer, i'd be happy to go with it)
This changeset also adjusts the tests so that we test whether the new,
preferred function returns bad values (since the deprecated function
just calls the new one).
We can keep the deprecated n_d_add_message function around as long as
we like, but at the next place where we're forced to break API or ABI
we can probably choose to drop the name relatively safely.
NOTE: there is probably more cleanup to do in the ruby and go bindings
to complete the deprecation directly. I don't know those languages
well enough to attempt a fix; i don't know how to test them; and i
don't know the culture around those languages about API additions or
deprecations.
This patch fixes the 'Deleted first duplicate file does not stop
notmuch show from working' test.
If a message to be shown has several duplicated files, and for some
reason the first file in the list is not available anymore, notmuch
will exit with an error.
This is clearly a problem in the database, but we are not going to let
this problem be a show-stopper. Let's walk through the list, and show
the first existing file.
Signed-off-by: Yuri Volchkov <yuri.volchkov@gmail.com>
This patch fixes the "Insert message into folder with trailing /"
test. The problem was insufficient path canonization.
From database's point of view, "Sent" and "Sent/" are different
folders. If user runs (note the last '/'):
notmuch insert --folder=maildir/Sent/ < test.msg
notmuch will create an extra XDIRECTORY record for the folder
'Sent/'. This means that database will have _TWO_ records for _ONE_
physical folder: 'Sent' and 'Sent/'. However, the 'notmuch new'
command will update only records related to the first one (the correct
one).
Now, if user moved the email file (e.g. from 'Sent/new' to
'Sent/cur'), 'notmuch new' will add a record about the new file, but
will not delete the old record.
Signed-off-by: Yuri Volchkov <yuri.volchkov@gmail.com>
From database's point of view, "Drafts" and "Drafts/" are different
folders
Signed-off-by: Yuri Volchkov <yuri.volchkov@gmail.com>
Amended: add test_subtest_known_broken (db)
In the commit 51cd69feb1 the part of the
test "test runs if prerequisite is satisfied" has been
removed. However, there was a remainder of that test - variable
'haveit'.
Kill it, to not to confuse people.
Signed-off-by: Yuri Volchkov <yuri.volchkov@gmail.com>
A leading / in paths in a .gitignore file matches the beginning of the
path, meaning that for patterns without slashes, git will match files
only in the current directory as opposed to in any subdirectory.
Prefix relevant paths with / in .gitignore files, to prevent
accidentally ignoring files in subdirectories and possibly slightly
improve the performance of "git status".
971cdc72cd renamed corpus.mail to
corpora.mail. Although 971cdc72cd
updated some of the remaining corpus.mail references, two remained,
causing the test suite to leave behind an unignored corpora.mail
directory.
The corresponding xapian document just gets more terms added to it,
but this doesn't seem to break anything. Values on the other hand get
overwritten, which is a bit annoying, but arguably it is not worse to
take the values (from, subject, date) from the last file indexed
rather than the first.
There are many other problems that could be tested, but these ones we
have some hope of fixing because it doesn't require UI changes, just
indexing changes.
There are some cases like remote usage where this might cause
problems, but those users can easily customize the variable. The
inconvenience seems to be outweighed by the security benefit for most
users.
gmime 3.0 no longer offers a means to set the path for gpg.
Users can set $PATH anyway if they want to pick a
differently-installed gpg (e.g. /usr/local/bin/gpg), so this isn't
much of a reduction in functionality.
The one main difference is for people who have tried to use "gpg2" to
make use of gpg 2.1, but that isn't usefully co-installable anyway.
The reply-to munging code might behave differently whether there's an
exact match on the strings or not, or whether the string is a raw
addr-spec instead of an name-addr. These tests cover those variations
(i also had to tweak json output further below when this new test was
added).
Since the error field is unused by the emacs front end, no changes are
needed other than bumping the format version number.
As it is, this is a bit overengineered, but it will reduce duplication
when we support gmime 3.0
This is another case where the behaviour of gmime-2.6 and gmime-3.0
seems to differ. It may be that we prefer the more lax parsing of the
previous version, but that should be tested separately.
Since gnupg 2.1.20, gpg-agent no longer shut itself down when
$GNUPGHOME directory is removed.
Add exit hooks to the test modules which execute `gpgconf --kill all`
Add exit hooks to execute `gpgconf --kill all` in the modules that
create $GNUPGHOME for gpg to work with.
New function at_exit_function registers given function to be called
at script termination.
Functions so registered are called in the reverse order of their
registration; no arguments are passed.
Function is called only once; re-adding with function name already
registered will remove previous registration.
New function rm_exit_function can be used to remove registration.
Modules (and possibly test-lib.sh functions) in future commits will
register such functions.
The dynamic generation of the linker version script for libnotmuch
exports has grown rather complicated.
Reverse the visibility control by hiding symbols by default using
-fvisibility=hidden, and explicitly exporting symbols in notmuch.h
using #pragma GCC visibility. (We could also use __attribute__
((visibility ("default"))) for each exported function, but the pragma
is more convenient.)
The above is not quite enough alone, as it would "leak" a number of
weak symbols from Xapian and C++ standard library. Combine it with a
small static version script that filters out everything except the
notmuch_* symbols that we explicitly exposed, and the C++ RTTI
typeinfo symbols for exception handling.
Finally, as the symbol hiding test can no longer look at the generated
symbol table, switch the test to parse the functions from notmuch.h.
From a UI perspective this looks similar to what was already provided
for from, subject, and mid, but the implementation is quite
different. It uses the database's list of terms to construct a term
based query equivalent to the passed regular expression.
'quite' on IRC reported that notmuch new was grinding to a halt during
initial indexing, and we eventually narrowed the problem down to some
html parts with large embedded images. These cause the number of terms
added to the Xapian database to explode (the first 400 messages
generated 4.6M unique terms), and of course the resulting terms are
not much use for searching.
The second test is sanity check for any "improved" indexing of HTML.
These 210 messages are in several long threads, which is good for
testing our threading code, and may be useful just as a larger test
corpus in the future.
The non-field processor behaviour is is convert the corresponding
queries into a search for the unprefixed terms. This yields pretty
surprising results so I decided to generate a query that would match
the terms (i.e. none with that prefix) generated for an empty header.
Make test-lib-common.sh load test-lib-<$PLATFORM>.sh to create
additional shim for platform specifics.
Use test-lib-FREEBSD.sh to call GNU utilities instead of native ones.
- amended by db following Tomi's suggestions
The argument is that if the string passed to the field processor has
no spaces, then the added quotes won't have any benefit except for
disabling wildcards. But disabling wildcards doesn't seem very useful
in the normal Xapian query parser, since they're stripped before
generating terms anyway. It does mean that the query 'from:"foo*"' will
not be precisely equivalent to 'from:foo' as it is for the non
field-processor version.
This was broken by the addition of regexp searching. The detection of
wildcards is not currently done in the recursive call to parse_query,
because of quoting issues.
This function was deprecated in notmuch 0.21. We re-use the name for
a status returning version, and deprecate the _st name. One or two
remaining uses of the (removed) non-status returning version fixed at
the same time
Apparently some systems (MacOS?) have a system library called libutil
and the name conflict causes problems. Since this library is quite
notmuch specific, rename it to something less generic.
Often Linux systems are shipped with wget(1) by default (and no curl).
Many BSDs, macOS, and e.g. some Linux minimal/container images
comes with curl(1) (and no wget).
Attempting to download with curl if wget is not available increases
the likelihood for this to succeed.
For reasons not completely understood at this time, gmime (as of
2.6.22) is returning a date before 1900 on bad date input. Since this
confuses some other software, we clamp such dates to 0,
i.e. 1970-01-01.
