Commit graph

1548 commits

Author SHA1 Message Date
Daniel Kahn Gillmor
4b1a8fd183 index: repair "Mixed Up" messages before indexing.
When encountering a message that has been mangled in the "mixed up"
way by an intermediate MTA, notmuch should instead repair it and index
the repaired form.

When it does this, it also associates the index.repaired=mixedup
property with the message.  If a problem is found with this repair
process, or an improved repair process is proposed later, this should
make it easy for people to reindex the relevant message.  The property
will also hopefully make it easier to diagnose this particular problem
in the future.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-09-15 19:07:06 -04:00
Daniel Kahn Gillmor
cb522fb06e test: add test for "Mixed-Up Mime" message mangling
Some MTAs mangle e-mail messages in transit in ways that are
repairable.

Microsoft Exchange (in particular, the version running today on
Office365's mailservers) appears to mangle multipart/encrypted
messages in a way that makes them undecryptable by the recipient.

I've documented this in section 4.1 "Mixed-up encryption" of draft -00
of
https://tools.ietf.org/html/draft-dkg-openpgp-pgpmime-message-mangling

Fortunately, it's possible to repair such a message, and notmuch can
do that so that a user who receives an encrypted message from a user
of office365.com can still decrypt the message.

Enigmail already knows about this particular kind of mangling.  It
describes it as "broken PGP email format probably caused by an old
Exchange server", and it tries to repair by directly changing the
message held by the user.  if this kind of repair goes wrong, the
repair process can cause data loss
(https://sourceforge.net/p/enigmail/bugs/987/, yikes).

The tests introduced here are currently broken.  In subsequent
patches, i'll introduce a non-destructive form of repair for notmuch
so that notmuch users can read mail that has been mangled in this way,
and the tests will succeed.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-09-15 01:20:03 -04:00
Daniel Kahn Gillmor
9829533e92 index: avoid indexing legacy-display parts
When we notice a legacy-display part during indexing, it makes more
sense to avoid indexing it as part of the message body.

Given that the protected subject will already be indexed, there is no
need to index this part at all, so we skip over it.

If this happens during indexing, we set a property on the message:
index.repaired=skip-protected-headers-legacy-display

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-09-01 08:45:30 -03:00
Daniel Kahn Gillmor
c61e22d5cb cli/{show,reply}: skip over legacy-display parts
Make use of the previous changes to fast-forward past any
legacy-display parts during "notmuch show" and "notmuch reply".

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-09-01 08:44:17 -03:00
Daniel Kahn Gillmor
27b25e45dc test: avoid showing legacy-display parts
Enigmail generates a "legacy-display" part when it sends encrypted
mail with a protected Subject: header.  This part is intended to
display the Subject for mail user agents that are capable of
decryption, but do not know how to deal with embedded protected
headers.

This part is the first child of a two-part multipart/mixed
cryptographic payload within a cryptographic envelope that includes
encryption (that is, it is not just a cleartext signed message).  It
uses Content-Type: text/rfc822-headers.

That is:

A └┬╴multipart/encrypted
B  ├─╴application/pgp-encrypted
C  └┬╴application/octet-stream
*   ╤ <decryption>
D   └┬╴multipart/mixed; protected-headers=v1 (cryptographic payload)
E    ├─╴text/rfc822-headers; protected-headers=v1 (legacy-display part)
F    └─╴… (actual message body)

In discussions with jrollins, i've come to the conclusion that a
legacy-display part should be stripped entirely from "notmuch show"
and "notmuch reply" now that these tools can understand and interpret
protected headers.

You can tell when a message part is a protected header part this way:

 * is the payload (D) multipart/mixed with exactly two children?
 * is its first child (E) Content-Type: text/rfc822-headers?
 * does the first child (E) have the property protected-headers=v1?
 * do all the headers in the body of the first child (E) match
   the protected headers in the payload part (D) itself?

If this is the case, and we already know how to deal with the
protected header, then there is no reason to try to render the
legacy-display part itself for the user.

Furthermore, when indexing, if we are indexing properly, we should
avoid indexing the text in E as part of the message body.

'notmuch reply' is an interesting case: the standard use of 'notmuch
reply' will end up omitting all mention of protected Subject:.

The right fix is for the replying MUA to be able to protect its
headers, and for it to set them appropriately based on headers found
in the original message.

If a replying MUA is unable to protect headers, but still wants the
user to be able to see the original header, a replying MUA that
notices that the original message's subject differs from the proposed
reply subject may choose to include the original's subject in the
quoted/attributed text. (this would be a stopgap measure; it's not
even clear that there is user demand for it)

This test suite change indicates what we want to happen for this case
(the tests are currently broken), and includes three additional TODO
suggestions of subtle cases for anyone who wants to flesh out the test
suite even further.  (i believe all these cases should be already
fixed by the rest of this series, but haven't had time to write the
tests for the unusual cases)

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-09-01 08:32:56 -03:00
Tomi Ollila
b6e589f54f test: aggregate-results.sh: count test files where all tests skipped
Previously, when all tests were skipped on a test file, there were
no indication of this in the final results aggregate-results.sh
printed.
Now count of the files where all tests were skipped is printed.
2019-07-05 17:58:23 +02:00
Daniel Kahn Gillmor
bdc87f0d3e test: run uncrustify
This is the result of running:

    $ uncrustify --replace --config ../devel/uncrustify.cfg *.cc *.c *.h

in the test directory.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-06-29 21:18:15 +02:00
David Bremner
1959a95d25 test: replace use of gdb with LD_PRELOAD shims in T070-insert.sh
This removes the dependency of this test script on gdb, and
considerably speeds up the running of the tests.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-06-29 21:11:08 +02:00
David Bremner
6544a2e305 test: provide machinery to make and use test_shims
These can be used e.g. to override return values for functions, in
place of the existing scripting of gdb.

This prepends to LD_PRELOAD rather than clobbering it, thanks to a
suggestion from Tomi Ollila.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-06-29 21:10:44 +02:00
Tomi Ollila
00c63bf736 test: aggregate-results.sh: consistent style. zero forks.
- all variables in $((...)) without leading $
- all comparisons use -gt, -eq or -ne
- no -a nor -o inside [ ... ] expressions
- all indentation levels using one tab

Dropped unnecessary empty string check when reading results files.

Replaced pluralize() which was executed in subshell with
pluralize_s(). pluralize_s sets $s to 's' or '' based on value of
$1. Calls to pluralize_s are done in context of current shell, so
no forks to subshells executed.
2019-06-11 07:20:01 -03:00
Daniel Kahn Gillmor
bc396c967c test: signature verification during decryption (session keys)
When the user knows the signer's key, we want "notmuch show" to be
able to verify the signature of an encrypted and signed message
regardless of whether we are using a stashed session key or not.

I wrote this test because I was surprised to see signature
verification failing when viewing some encrypted messages after
upgrading to GPGME 1.13.0-1 in debian experimental.

The added tests here all pass with GPGME 1.12.0, but the final test
fails with 1.13.0, due to some buggy updates to GPGME upstream: see
https://dev.gnupg.org/T3464 for more details.

While the bug needs to be fixed in GPGME, notmuch's test suite needs
to make sure that GMime is doing what we expect it to do; i was a bit
surprised that it hadn't caught the problem, hence this patch.

I've fixed this bug in debian experimental with gpgme 1.13.0-2, so the
tests should pass on any debian system.  I've also fixed it in the
gpgme packages (1.13.0-2~ppa1) in the ubuntu xenial PPA
(ppa:notmuch/notmuch) that notmuch uses for Travis CI.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-06-08 20:14:00 -03:00
David Bremner
2c1e5c186e test: update test description.
I missed this fix in dkg's revisions.
2019-05-29 08:40:02 -03:00
Daniel Kahn Gillmor
1c704dd22d cli/reply: pull proposed subject line from the message, not the index
Protected subject lines were being emitted in reply when the cleartext
of documents was indexed.  create_reply_message() was pulling the
subject line from the index, rather than pulling it from the
GMimeMessage object that it already has on hand.

This one-line fix to notmuch-reply.c solves that problem, and doesn't
cause any additional tests to fail.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-29 08:17:33 -03:00
Daniel Kahn Gillmor
06dedd0a83 test: reply (in cli and emacs) should protect indexed sensitive headers
These tests are currently broken!  When a protected subject is indexed
in the clear, it leaks in the reply headers :(

For emacs, we set up separate tests for when the protected header is
indexed in the clear and when it is unindexed.  neither case should
leak, but the former wasn't tested yet.

We will fix the two broken tests in a subsequent patch.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-29 08:17:20 -03:00
Daniel Kahn Gillmor
cd8006886b test: emacs/show: ensure that protected headers appear as expected
This tests notmuch-show; headers appear appropriately based on the
setting of notmuch-crypto-process-mime.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-29 08:17:12 -03:00
Daniel Kahn Gillmor
5007595be8 test: ensure that protected headers appear in notmuch-emacs search as expected
We initially test only notmuch-search; tests for other functionality
come in different patchsets later.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-29 08:16:58 -03:00
Daniel Kahn Gillmor
809a34a870 test: try indexing nested messages and protected headers
We want to make sure that internally-forwarded messages don't end up
"bubbling up" when they aren't actually the cryptographic payload.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-29 08:15:28 -03:00
Daniel Kahn Gillmor
bfed02bb0b test: after reindexing, only legitimate protected subjects are searchable
This test scans for all the possible protected headers (including
bogus/broken ones) that are present in the protected-headers corpus,
trying to make sure that only the ones that are not broken or
malformed show up in a search after re-indexing.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-29 08:15:18 -03:00
Daniel Kahn Gillmor
b36248a26e test: protected headers should work when both encrypted and signed.
Up to this point, we've tested protected headers on messages that have
either been encrypted or signed, but not both.

This adds a couple tests of signed+encrypted messages, one where the
subject line is masked (outside subject line is "Subject Unavailable")
and another where it is not (outside Subject: matches inner Subject:)

See the discussion at
https://dkg.fifthhorseman.net/blog/e-mail-cryptography.html#protected-headers
for more details about the nuances between signed, stripped, and
stubbed headers.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-29 08:14:57 -03:00
Daniel Kahn Gillmor
5c3a44681f indexing: record protected subject when indexing cleartext
When indexing the cleartext of an encrypted message, record any
protected subject in the database, which should make it findable and
visible in search.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-29 08:14:44 -03:00
Daniel Kahn Gillmor
b7b553e732 cli/reply: ensure encrypted Subject: line does not leak in the clear
Now that we can decrypt headers, we want to make sure that clients
using "notmuch reply" to prepare a reply don't leak cleartext in their
subject lines.  In particular, the ["reply-headers"]["Subject"] should
by default show the external Subject.

A replying MUA that intends to protect the Subject line should show
the user the Subject from ["original"]["headers"]["Subject"] instead
of using ["reply-headers"]["Subject"].

This minor asymmetry with "notmuch show" is intentional.  While both
tools always render the cleartext subject line when they know it (in
["headers"]["Subject"] for "notmuch show" and in
["original"]["headers"]["Subject"] for "notmuch reply"), "notmuch
reply" should never leak something that should stay under encrypted
cover in "reply-headers".

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-29 08:14:32 -03:00
Daniel Kahn Gillmor
996ef5710c test: show cryptographic envelope information for signed mails
Make sure that we emit the correct cryptographic envelope status for
cleartext signed messages.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-29 08:13:06 -03:00
Daniel Kahn Gillmor
1c879f3939 test: add test for missing external subject
Adding another test to ensure that we handle protected headers
gracefully when no external subject is present.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-29 08:12:49 -03:00
Daniel Kahn Gillmor
56416a5470 cli/show: add information about which headers were protected
The header-mask member of the per-message crypto object allows a
clever UI frontend to mark whether a header was protected (or not).
And if it was protected, it contains enough information to show useful
detail to an interested user.  For example, an MUA could offer a "show
what this message's Subject looked like on the wire" feature in expert
mode.

As before, we only handle Subject for now, but we might be able to
handle other headers in the future.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>

Amended by db: tweaked schemata notation.
2019-05-29 08:11:50 -03:00
Daniel Kahn Gillmor
1c7fbbcc99 cli/show: emit payload subject instead of outside subject
Correctly fix the two outstanding tests so that the protected (hidden)
subject is properly reported.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-29 08:05:01 -03:00
Daniel Kahn Gillmor
528f526f69 cli/show: add tests for viewing protected headers
Here we add several variant e-mail messages, some of which have
correctly-structured protected headers, and some of which do not.  The
goal of the tests is to ensure that the right protected subjects get
reported.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-29 08:04:32 -03:00
Jameson Graef Rollins
03839a8110 test: new test framework to compare json parts
This makes it easier to write fairly compact, readable tests of json
output, without needing to sanitize away parts that we don't care
about.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-29 08:03:21 -03:00
Daniel Kahn Gillmor
80728a95e6 cli/show: emit headers after emitting body
This paves the way for emitting protected headers after verification
and decryption, because it means that the headers will only be emitted
after the body has been parsed.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-29 08:02:32 -03:00
Daniel Kahn Gillmor
73cebe6e72 test: report summary even when aborting
In certain cases of test suite failure, the summary report was not
being printed.  In particular, any failure on the parallel test suite,
and any aborted test in the serialized test suite would end up hiding
the summary.

It's better to always show the summary where we can (while preserving
the return code).  If we do abort due to this high-level failure,
though, we should also announce to the user that we're doing so as
close to the end of the process as possible, to make it easier to find
the problem.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-26 18:55:06 -03:00
Daniel Kahn Gillmor
4cb789aa09 cli/show: emit new whole-message crypto status output
This allows MUAs that don't want to think about per-mime-part
cryptographic status to have a simple high-level overview of the
message's cryptographic state.

Sensibly structured encrypted and/or signed messages will work fine
with this.  The only requirement for the simplest encryption + signing
is that the message have all of its encryption and signing protection
(the "cryptographic envelope") in a contiguous set of MIME layers at
the very outside of the message itself.

This is because messages with some subparts signed or encrypted, but
with other subparts with no cryptographic protection is very difficult
to reason about, and even harder for the user to make sense of or work
with.

For further characterization of the Cryptographic Envelope and some of
the usability tradeoffs, see here:

   https://dkg.fifthhorseman.net/blog/e-mail-cryptography.html#cryptographic-envelope
2019-05-26 08:20:23 -03:00
Daniel Kahn Gillmor
9300defd64 emacs: Drop content-free "Unknown signature status" button
When we have not been able to evaluate the signature status of a given
MIME part, showing a content-free (and interaction-free) "[ Unknown
signature status ]" button doesn't really help the user at all, and
takes up valuable screen real-estate.

A visual reminder that a given message is *not* signed isn't helpful
unless it is always present, in which case we'd want to see "[ Unknown
signature status ]" buttons on all messages, even ones that don't have
a signing structure, but i don't think we want that.

Amended by db to drop the unused initialization of 'label'
2019-05-25 13:02:02 -03:00
Daniel Kahn Gillmor
fa9d8b7026 test: allow disabling timeout with NOTMUCH_TEST_TIMEOUT=0
To aid in diagnosing test suite tooling that interacts poorly with
coreutils' timeout, it's handy to be able to bypass it entirely.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-25 08:26:41 -03:00
David Bremner
adb53b0737 lib/database: index user headers.
This essentially involves calling _notmuch_message_gen_terms once for
each user defined header.
2019-05-25 07:21:13 -03:00
David Bremner
575493e785 lib: setup user headers in query parser
These tests will need to be updated if the Xapian
query print/debug format changes.
2019-05-25 06:56:16 -03:00
David Bremner
4b9c03efc6 cli/config: check syntax of user configured field names
These restrictions are meant to prevent incompatibilities with the
Xapian query parser (which will split at non-word characters) and
clashes with future notmuch builtin fields.
2019-05-25 06:56:16 -03:00
David Bremner
7981bd050e cli/config: support user header index config
We don't do anything with this configuration information information
yet, but nonetheless add a couple of regression tests to make sure we
don't break standard functionality when we do use the configuration
information.
2019-05-25 06:56:16 -03:00
Tomi Ollila
3563079be3 test-lib.sh: colors to test output when parallel(1) is run on tty
Done via $COLORS_WITHOUT_TTY environment variable as passing options
to commands through parallel(1) does not look trivial.

Reorganized color checking in test-lib.sh a bit for this (perhaps
were not fully necessary but rest still an improvement):

  - color checking commands in subshell are not run before arg parsing
    (args may disable colors with --no-color)

  - [ -t 1 ] is checked before forking subshell
2019-05-23 08:00:31 -03:00
Tomi Ollila
a1aea7272e test-lib.sh: "tidied" emacs_deliver_message ()
Added initialization and checking of smtp_dummy_port
like it was done with smtp_dummy_pid.

Made those function-local variables.

One 8 spaces to tab consistency conversion.

And last, but definitely not least; while doing above
noticed that there were quite a few double-quoted strings
where $@ was in the middle of it -- replaced those with $*
for robustness ("...$@..." expands params to separate words,
"...$*..." params expands to single word).
2019-05-23 08:00:13 -03:00
Tomi Ollila
f33053023b test: redirect STDIN from /dev/null
Without this stdin may be anything that parent process provided for it.

Test processes might have tried to read something from it, which would
have caused undeterministic behavior.

E.g. gdb(1) tries to do tty related ioctls on fd 0 (and fd 1 and fd 2,
but those are redirected to 'test.output' before test runs).
2019-05-22 08:47:17 -03:00
David Bremner
f4708ce0b1 test/emacs: revert invalid-from test to pre-86f89385 behaviour
To the best of my understanding, this original behaviour was what
Carl's homebrew parser produced. With commit 86f89385 Austin switched
to using GMime (2.6). This produced arguably worse results, but since
the input was bad, we could live with it. Now with GMime 3.0 we are
getting the original results again, and there is no reason to consider
this test broken.
2019-05-20 16:31:28 -03:00
Daniel Kahn Gillmor
c88e030580 tests: fail and report when a parallel build fails (or times out)
When a parallel build fails (or when it times out, if timeout is
present), the test suite should not blithely succeed.  Catch these
failures and at least report them.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-20 14:48:56 -03:00
Daniel Kahn Gillmor
73bf7e532e tests: make timeout configurable with NOTMUCH_TEST_TIMEOUT (default: 2m)
The current 2 minute timeout is reasonable, but to exercise the test
suite or induce timeout failures, we might want to make it shorter.
This makes it configurable so you can run (for example):

    make check NOTMUCH_TEST_TIMEOUT=10s

We stick with the default of 2m.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-20 14:48:43 -03:00
Daniel Kahn Gillmor
9c0001de4b test: show what emacs sees of an encrypted message when crypto is disabled
Some users may set notmuch-crypto-process-mime to nil, disabling all
crypto use.  We should have a baseline for what that looks like.
2019-05-10 12:30:03 -03:00
Daniel Kahn Gillmor
bda0fecccd test: avoid unnecessary extraction of the test fingerprint
FINGERPRINT is already exported by add_gnupg_home, so this is
unnecessary.  This change also happens to get rid of the superfluous
check-trustdb spew from the test suite that looked like this:

gpg: checking the trustdb
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-10 06:56:34 -03:00
David Bremner
f2425a11a3 test: let the OS choose a port for smtp-dummy
This should avoid potential collisions if we start running multiple
smtp-dummy processes in parallel.
2019-05-10 06:56:22 -03:00
Daniel Kahn Gillmor
e1c8357c44 emacs: test notmuch-show during message decryption
We did not have a test showing what message decryption looks like
within notmuch-emacs.  This change gives us a baseline for future work
on the notmuch-emacs interface.

This differs from previous revisions of this patch in that it should
be insensitive to the order in which the local filesystem readdir()s
the underlying maildir.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-10 06:54:50 -03:00
Jameson Graef Rollins
eeff431996 tests: environment variable to specify that tests should be serialized
If NOTMUCH_TEST_SERIALIZE is non-null all tests will be run in series,
rather than in parallel.
2019-05-07 06:55:31 -03:00
Jameson Graef Rollins
908d930d22 tests: run all tests in parallel, if available
If either the moreutils or GNU parallel utility are available, run all
tests in parallel.  On my eight core machine this makes for a ~x7
speed-up in the full test suite (1m24s -> 12s).

The design of the test suite makes this parallelization trivial.
2019-05-07 06:54:09 -03:00
Jameson Graef Rollins
591388ccd1 tests: remove entangling corpus index optimization
The add_email_corpus test utility includes logic that tries to re-use
an index of the corpus if available.  This was seemingly done as an
optimization, so that every test that uses the corpus didn't have to
create it's own index of the corpus.  However, this has the perverse
side effect of entangling tests together, and breaks parallelization.

Forcing each test to do it's own index does increase the overall time
of the test slightly (~6%), but this will be more than made up for in
the next patch that introduces paraellization.
2019-05-07 06:53:57 -03:00
Jameson Graef Rollins
7f7af27bd8 tests: remove some redundant pre-cleanup of the corpus MAIL_DIR
add_email_corpus itself does an rm -rf $MAIL_DIR, so these are not necessary.
2019-05-07 06:52:35 -03:00
Daniel Kahn Gillmor
7d48604157 test/crypto: add_gnupg_home should have ultimate trust on "its own" key
The typical use case for gpg is that if you control a secret key, you
mark it with "ultimate" ownertrust.

The opaque --import-ownertrust mechanism is GnuPG's standard mechanism
to set up ultimate ownertrust (the ":6:" means "ultimate", for
whatever reason).

We adjust the test suite to match this change, inverting the sense of
one test: since the default is now that the user ID of the suite's own
key is valid, we change the test to make sure that the user ID is not
emitted when it is *not* valid.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-07 06:42:21 -03:00
Daniel Kahn Gillmor
93e699e5c8 test: simplify user ID handling
The user ID on the self-test is a little bit clunky-looking.  It also
may end up showing up elsewhere in the test suite.  Centralizing the
user ID in one place should make it easier to handle if it ever
changes, and should make tests easier to read.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-07 06:42:11 -03:00
Daniel Kahn Gillmor
9f05ceb994 test/crypto: clarify the difference between ownertrust and validity
This is a subtle difference, but the output of notmuch shouldn't ever
change based on ownertrust itself -- notmuch is intended to show valid
User IDs, and to avoid showing invalid User IDs.

It so happens that setting ownertrust of a key to ultimate sets all
associated user IDs to "full" validity, so the test is correct, but
just misnamed.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-07 06:42:01 -03:00
David Bremner
6682b4e686 Merge tag 0.28.4
No functionality changes merged, since the bug in question was already
fixed on master.
2019-05-05 16:38:51 -03:00
David Bremner
ae6b52488d cli/show: avoid empty write to stdout in format_part_raw
Previously if the input was exactly a multiple of the internal buffer
size, notmuch would attempt to fwrite nothing to stdout, but still
expected fwrite to return 1, causing a failure that looked like this:

  $ notmuch show --format=raw id:87o96f1cya.fsf@codeaurora.org
    ...entire message shown as expected..
  Error: Write failed
  $ echo $?
  1

To fix the problem don't call fwrite at all when there's nothing to
write.

Amended by db: add some tests of message sizes likely to cause this
problem.
2019-05-05 08:01:17 -03:00
David Bremner
103c11822e cli/notmuch-show: support gzipped files
This drops "file" from mime_node_context and just uses a local
variable. It also uses the new gzip aware utility routines recently
added to util/gmime-extra.c. The use of gzopen / gzfile in addition is
a bit icky, but the choice is between that, and providing yet another
readline implimentation that understands GMime streams.
2019-05-03 07:48:43 -03:00
David Bremner
852167479f lib/message_file: open gzipped files
Rather than storing the lower level stdio FILE object, we store a
GMime stream. This allows both transparent decompression, and passing
the stream into GMime for parsing. As a side effect, we can let GMime
close the underlying OS stream (indeed, that stream isn't visible here
anymore).

This change is enough to get notmuch-{new,search} working, but there is still
some work required for notmuch-show, to be done in a following commit.
2019-05-03 07:48:43 -03:00
Daniel Kahn Gillmor
7e6f55b268 gmime-cleanup: simplify T355-smime.sh
GMime 3.0 and later can handle User ID as expected.

signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-03 06:55:52 -03:00
Daniel Kahn Gillmor
652baa6fe6 gmime-cleanup: tests should only care about gmime 3
note that "notmuch-show for message with invalid From" is still broken
in T310-emacs.sh.  It would be good to debug what's going on there and
try to get it fixed!

signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-03 06:55:44 -03:00
Daniel Kahn Gillmor
bb0b119358 gmime-cleanup: always support session keys
Our minimum version of GMime 3.0 always supports good session key
handling.

signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-03 06:55:32 -03:00
Daniel Kahn Gillmor
096d45a878 gmime-cleanup: remove obsolete gpg_path configuration option and crypto contexts
Note that we do keep ignoring the gpg_path configuration option,
though, to avoid breakage of existing installations.  It is ignored
like any other unknown configuration option, but we at least document
that it is ignored so that people who find it in their legacy configs
can know that it's safe to drop.

signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-03 06:55:04 -03:00
Daniel Kahn Gillmor
afb0b11dcd crypto: Avoid pretending to verify signatures on unsigned encrypted mail
Unsigned encrypted mail shows up with a weird empty signature list.
If we successfully decrypted and there was no signature in it, we
should just not show a sigstatus at all.

The documentation for g_mime_decrypt_result_get_signatures says:

    a GMimeSignatureList or NULL if the stream was not signed.
2019-04-26 08:03:15 -03:00
Daniel Kahn Gillmor
22ec4a36e6 tests: move FINGERPRINT definition to add_gnupg_home
If a test has added a GnuPG homedir, it may well want to know the
fingerprint.  This saves us from having to redefine this magic string
in multiple places when more tests eventually use the GnuPG homedir.
2019-04-24 07:16:46 -03:00
David Bremner
319dd95ebb lib: add 'body:' field, stop indexing headers twice.
The new `body:` field (in Xapian terms) or prefix (in slightly
sloppier notmuch) terms allows matching terms that occur only in the
body.

Unprefixed query terms should continue to match anywhere (header or
body) in the message.

This follows a suggestion of Olly Betts to use the facility (since
Xapian 1.0.4) to add the same field with multiple prefixes. The double
indexing of previous versions is thus replaced with a query time
expension of unprefixed query terms to the various prefixed
equivalent.

Reindexing will be needed for 'body:' searches to work correctly;
otherwise they will also match messages where the term occur in
headers (demonstrated by the new tests in T530-upgrade.sh)
2019-04-17 08:48:16 -03:00
Örjan Ekeberg
bfe0f8dcc4 test: add test for checking forwarded messages
Add test of forwarding messages from within emacs.
The first test checks that a references header is properly
added to the new message.  The second test checks that the
send-hook of the forwarding message adds a forwarded-tag
to the original message.
2019-04-14 08:01:30 -03:00
David Edmondson
639d21d5b9 test: Add emacs attachment check tests. 2019-03-28 14:37:21 -03:00
David Bremner
d25dcc589c lib: use phrase search for anything not ending in '*'
Anything that does not look like a wildcard should be safe to
quote. This should fix the problem searching for xapian keywords.
2019-03-28 14:34:37 -03:00
David Bremner
168211c563 test: add two known broken tests searching for xapian keywords
Thanks to plujon for pointing out this problem on IRC. The underlying
issue is that the quotes are stripped before the field processors get
the query string, and the heuristic for putting them back is not quite
right.
2019-03-28 14:32:11 -03:00
Luis Ressel
9f7e851263 Prepend regerror() messages with "regexp error: "
The exact error messages returned by regerror() aren't standardized;
relying on them isn't portable. Thus, add a a prefix to make clear that
the subsequent message is a regexp parsing error, and only look for this
prefix in the test suite, ignoring the rest of the message.
2019-03-11 22:24:55 -03:00
Luis Ressel
a1e649276e test/T030-config: Separate stdout and stderr output
POSIX doesn't specify the flushing behaviour of the STDOUT stream, so
it's invalid to assume a particular order between the stdout and stderr
output. The current test breaks on musl due to this.
2019-03-11 22:13:54 -03:00
David Bremner
71eaa19350 Merge branch 'release'
Changes from 0.28.3
2019-03-06 08:53:26 -04:00
David Bremner
e88297c072 lib/string_map: fix return type of string_cmp
I can't figure out how checking the sign of a bool ever worked. The
following program demonstrates the problem (i.e. for me it prints 1).

 #include <stdio.h>
 #include <stdbool.h>
 int main(int argc, char **argv) {
    bool x;
    x = -1;
    printf("x = %d\n", x);
 }

This seems to be mandated by the C99 standard 6.3.1.2.
2019-03-05 21:46:41 -04:00
David Bremner
09595cf577 lib: Add known broken test for string_map binary search.
Because the string_map functions are not exported, we test via message
properties.
2019-03-05 21:46:41 -04:00
rhn
b350c688e1 test: Check for replies obeying lifetime guarantees
The test attempts to check that a message coming from a thread outlives its messages list and gets destroyed together with the thread.
2019-01-25 20:37:26 -04:00
David Edmondson
576cff7654 reply: Include sender as recipient if they were the original recipient
When generating a reply message, if the user was the originator and
only recipient of the original message, include the user as a
recipient of the reply.
2018-12-07 08:03:50 -04:00
Maxime Coste
baa4185c30 cli: notmuch show support for --include-html with --format=text 2018-11-24 09:12:30 -04:00
Maxime Coste
37e5bc00ae cli: notmuch show support for --body=false with --format=text 2018-11-24 09:11:42 -04:00
Daniel Kahn Gillmor
311f530013 test: git should ignore message-id-parse
b31e44c678 introduced message-id-parse
as a new binary created by the test suite.  It shows up as something
additional to git, but git ought to know to ignore it.
2018-10-08 12:50:03 -03:00
David Edmondson
e8cb6b2cd6 emacs: Call `notmuch-mua-send-hook' hooks when sending a message
Previously any hook functions attached to `notmuch-mua-send-hook' were
ignored.
2018-09-28 20:22:42 -03:00
David Edmondson
4e213fe9b4 test: Check that `notmuch-mua-send-hook' is called on sending a message 2018-09-28 20:22:33 -03:00
David Bremner
c846e15ffe test: make regexp test conditional on field processors
Normally we'd mark it broken, but perversely missing regexp support
actually makes the test pass.
2018-09-14 08:54:20 -03:00
David Bremner
c6ba5522a5 test: mark thread subqueries broken without field processors
Currently these tests just fail when notmuch is built against Xapian
1.2.x
2018-09-14 08:54:09 -03:00
David Edmondson
4cd5a0a3d5 test: Absolute and relative directory paths. 2018-09-08 20:19:42 -03:00
David Bremner
87934c432c lib: change parent strategy to use In-Reply-To if it looks sane
As reported by Sean Whitton, there are mailers (in particular the
Debian Bug Tracking System) that have sensible In-Reply-To headers,
but un-useful-for-notmuch References (in particular with the BTS, the
oldest reference is last). I looked at a sample of about 200K
messages, and only about 0.5% these had something other than a single
message-id in In-Reply-To. On this basis, if we see a single
message-id in In-Reply-To, consider that as authoritative.
2018-09-06 08:07:13 -03:00
David Bremner
b31e44c678 lib: add _notmuch_message_id_parse_strict
The idea is that if a message-id parses with this function, the MUA
generating it was probably sane, and in particular it's probably safe
to use the result as a parent from In-Reply-to.
2018-09-06 08:07:13 -03:00
David Bremner
35053c2b9a test/thread-replies: mangle In-Reply-To's
In a future commit, we will start trusting In-Reply-To's when they
look sane (i.e. a single message-id). Modify these tests so they will
keep passing (i.e. keep choosing References) when that happens.
2018-09-06 08:07:13 -03:00
David Bremner
ea08032ae4 test: add known broken test for good In-Reply-To / bad References
The current scheme of choosing the replyto (i.e. the default parent
for threading purposes) does not work well for mailers that put
the oldest Reference last.
2018-09-06 08:07:13 -03:00
David Bremner
46dce33abc lib/thread: change _resolve_thread_relationships to use depths
We (finally) implement the XXX comment. It requires a bit of care not
to reparent all of the possible toplevel messages.

_notmuch_messages_has_next is not ready to be a public function yet,
since it punts on the mset case. We know in the one case it is called,
the notmuch_messages_t is just a regular list / iterator.
2018-09-06 08:07:13 -03:00
David Bremner
a330858284 lib/thread: initial use of references as for fallback parenting
This is mainly to lay out the structure of the final code. The problem
isn't really solved yet, although some very simple cases are
better (hence the fixed test). We need two passes through the messages
because we need to be careful not to re-parent too many messages and
end up without any toplevel messages.
2018-09-06 08:07:13 -03:00
David Bremner
9b568e73e1 lib/thread: sort sibling messages by date
For non-root messages, this should not should anything currently, as
the messages are already added in date order. In the future we will
add some non-root messages in a second pass out of order and the
sorting will be useful. It does fix the order of multiple
root-messages (although it is overkill for that).
2018-09-06 08:07:12 -03:00
David Bremner
ccb52edb4c test: add known broken tests for "ghost roots"
This documents the bug discussed at

     id:87efgmmysi.fsf@len.workgroup

The underlying issue is that the reply to a ghost (missing) message is
falsely classified as a root message in _resolve_thread_relationships.

There are two pairs of tests; in each case the the first test is
simpler / more robust, but also easier to fool.
2018-09-06 08:07:12 -03:00
David Bremner
ebd131ac07 test: start threading test corpus
There are 3 threads here, two synthetic, and one anonymized one using
data from Gregor. They test various aspects of thread
ordering/construction in the presence of replies to ghost messages.
2018-09-06 08:07:12 -03:00
David Bremner
dfda1745bf test/tag: add test for updating during batch tag operations
This clarifies that the breakage seen with Xapian 1.4.6 does not have
to do with "funny" tags.

This test is "known broken", but only with xapian 1.4.6, and there's
curently no convenient way to mark that.
2018-08-29 06:37:48 -03:00
Sebastian Poeplau
0f08bf7166 lib: detect mislabeled Windows-1252 parts
Use GMime functionality to detect mislabeled messages and apply the
correct (Windows) encoding instead.
2018-08-29 06:34:39 -03:00
Sebastian Poeplau
7074bb8f80 test: add known broken test for mislabeled Windows-1252 encoding
Messages that contain Windows-1252 are frequently mislabeled as ISO
8859-1, which may result in non-printable characters when displaying
the message. The test asserts that such characters (in this case
curved quotes) are displayed correctly.
2018-08-29 06:28:13 -03:00
David Bremner
46cf1a98f3 test: fix hardcoded paths in T050-new.sh
The previous paths will fail almost everywhere.
2018-05-26 16:46:44 -07:00
Daniel Kahn Gillmor
aa605f7e8a cli/show: enable --decrypt=stash
Add fancy new feature, which makes "notmuch show" capable of actually
indexing messages that it just decrypted.

This enables a workflow where messages can come in in the background
and be indexed using "--decrypt=auto".  But when showing an encrypted
message for the first time, it gets automatically indexed.

This is something of a departure for "notmuch show" -- in particular,
because it requires read/write access to the database.  However, this
might be a common use case -- people get mail delivered and indexed in
the background, but only want access to their secret key to happen
when they're directly interacting with notmuch itself.

In such a scenario, they couldn't search newly-delivered, encrypted
messages, but they could search for them once they've read them.

Documentation of this new feature also uses a table form, similar to
that found in the description of index.decrypt in notmuch-config(1).

A notmuch UI that wants to facilitate this workflow while also
offering an interactive search interface might instead make use of
these additional commands while the user is at the console:

Count received encrypted messages (if > 0, there are some things we
haven't yet tried to index, and therefore can't yet search):

     notmuch count tag:encrypted and \
         not property:index.decryption=success and \
         not property:index.decryption=failure

Reindex those messages:

     notmuch reindex --try-decrypt=true tag:encrypted and \
         not property:index.decryption=success and \
         not property:index.decryption=failure
2018-05-26 07:43:30 -07:00
Daniel Kahn Gillmor
9d114a8552 test-lib: add notmuch_show_part for "notmuch show --format=text"
Thanks to David Bremner for this improved readability!
2018-05-26 07:42:28 -07:00
David Bremner
84b2963726 test: use --full-scan in T050-new.sh
Wherever the test relies on directories being scanned, this option
should be used to avoid skipping them due to mtimes on directories
matching the database.
2018-05-22 09:29:59 -07:00
David Bremner
d1acff6153 test: add tests for notmuch new --full-scan
Most of these just check that adding the flag does not break existing
functionality. The one test that does check the full-scan
functionality had to be rewritten to output debugging info.
2018-05-22 09:29:48 -07:00
Thomas Schneider
b2e4778ea4 build: Allow user to specify ruby executable
This way, one can build for a different Ruby than $PATH/ruby
(e. g. different versions, or Ruby in other paths).

Signed-off-by: Thomas Schneider <qsx@chaotikum.eu>
2018-05-10 21:01:06 -03:00
David Bremner
5ad39ebf75 lib: add thread subqueries.
This change allows queries of the form

 thread:{from:me} and thread:{from:jian} and not thread:{from:dave}

This is still somewhat brute-force, but it's a big improvement over
both the shell script solution and the previous proposal [1], because it
does not build the whole thread structure just generate a
query. A further potential optimization is to replace the calls to
notmuch with more specialized Xapian code; in particular it's not
likely that reading all of the message metadata is a win here.

[1]: id:20170820213240.20526-1-david@tethera.net
2018-05-07 08:42:53 -03:00
Florian Klink
140e8fb118 T460-emacs-tree: sync $EXPECTED with test name
This syncs the $EXPECTED directory name with the tests filename.

Signed-off-by: Florian Klink <flokli@flokli.de>
2018-05-03 21:03:33 -03:00
Daniel Kahn Gillmor
c20a5eb805 move more http -> https
Correct URLs that have crept into the notmuch codebase with http://
when https:// is possible.

As part of this conversion, this changeset also indicates the current
preferred upstream URLs for both gmime and sup.  the new URLs are
https-enabled, the old ones are not.

This also fixes T310-emacs.sh, thanks to Bremner for catching it.
2018-05-03 20:59:20 -03:00
David Bremner
4ac23c5978 test: add known broken test for regexp search of second subject
We expect this to give the same answer as the non-regexp subject
search. It does not because the regexp search relies on the value
slot, which currently contains only one subject.
2018-05-03 07:44:49 -03:00
David Bremner
f3f4297c7f test: tests for python bindings get_property / get_properties
These roughly replicate the equivalent C tests, although they rely on
the database state created by the former tests, since the python
bindings currently provide read-only access to properties.
2018-05-02 20:30:51 -03:00
David Bremner
963ccabe93 Merge branch 'release'
reference loop fixes to be included in 0.26.2
2018-04-25 17:36:32 -03:00
David Bremner
e8ac065a50 test: re-enable disabled test in T700-reindex.sh
The extra test_done looks like a typo
2018-04-24 22:59:21 -03:00
David Bremner
491b1f4b40 lib: choose oldest message when breaking reference loops
This preserves a sensible thread order
2018-04-23 23:00:20 -03:00
David Bremner
4e085b6d92 test: add known broken test for thread ordering from a loop
The previous loop handling code chooses the last message in the
message list, which turns out to be the last in date order.
See the comment in _notmuch_thread_create.
2018-04-23 23:00:20 -03:00
David Bremner
9293d6da27 lib: break reference loop by choosing arbitrary top level msg
Other parts of notmuch (e.g. notmuch show) expect each thread to
contain at least one top level message, and crash if this expectation
is not met.
2018-04-20 11:23:31 -03:00
David Bremner
ab55ca8e0a test: add known broken test for indexing an In-Reply-To loop.
This documents the bug discussed in

     id:87d10042pu.fsf@curie.anarc.at
2018-04-20 11:23:31 -03:00
David Bremner
044cbd920c test: two new messages for the 'broken' corpus
These have an 'In-Reply-To' loop, which currently confuses "notmuch
new".
2018-04-20 11:23:31 -03:00
Georg Faerber
c117306f2d Fix typos as found by codespell
Signed-off-by: Georg Faerber <georg@riseup.net>
2018-03-24 20:09:54 -03:00
Daniel Kahn Gillmor
b6e3efde05 cli/insert: add --world-readable flag
In some cases (e.g. when building a publicly-visible e-mail archive)
it doesn't make any sense to restrict visibility of the message to the
current user account.

This adds a --world-readable boolean option for "notmuch insert", so
that those who want to archive their mail publicly can feed their
archiver with:

    notmuch insert --world-readable

Other local delivery agents (postfix's local, and dovecot's lda) all
default to delivery in mode 0600 rather than relying on the user's
umask, so this fix doesn't change the default.

Also, this does not override the user's umask.  if the umask is
already set tight, it will not become looser as the result of passing
--world-readable.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2018-03-24 20:08:11 -03:00
Daniel Kahn Gillmor
54982e520c fix typos 2018-01-04 20:35:58 -04:00
Daniel Kahn Gillmor
af8255fb71 cli/reply: make --decrypt take a keyword
This brings the --decrypt argument to "notmuch reply" into line with
the other --decrypt arguments (in "show", "new", "insert", and
"reindex").  This patch is really just about bringing consistency to
the user interface.

We also use the recommended form in the emacs MUA when replying, and
update test T350 to match.
2017-12-29 16:45:55 -04:00
Daniel Kahn Gillmor
8ea4a99d74 cli/show: make --decrypt take a keyword.
We also expand tab completion for it, update the emacs bindings, and
update T350, T357, and T450 to match.

Make use of the bool-to-keyword backward-compatibility feature.
2017-12-29 16:45:46 -04:00
Daniel Kahn Gillmor
0ada2a05c9 cli: some keyword options can be supplied with no argument
We might change some notmuch command line tools that used to be
booleans into keyword arguments.

In that case, there are some legacy tools that will expect to be able
to do "notmuch foo --bar" instead of "notmuch foo --bar=baz".

This patch makes it possible to support that older API, while
providing a warning and an encouragement to upgrade.
2017-12-29 16:45:35 -04:00
Daniel Kahn Gillmor
6aec7a76b9 python: add decrypt_policy argument to Database.index_file()
We adopt a pythonic idiom here with an optional argument, rather than
exposing the user to the C indexopts object directly.

This now includes a simple test to ensure that the decrypt_policy
argument works as expected.
2017-12-24 09:47:35 -04:00
Daniel Kahn Gillmor
b1de8e885d test: final named query test works regardless of Xapian FieldProcessor
This test passes with older versions of Xapian as well, because
neither query returns any results.

This should resolve the travis build failure at
https://travis-ci.org/notmuch/notmuch/builds/318571658

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2017-12-21 09:35:20 -04:00
David Bremner
7cfa1c6961 lib: return "" rather than NULL from notmuch_thread_get_authors
The current behaviour is at best under-documented. The modified test in
T470-missing-headers.sh previously relied on printf doing the right
thing with NULL, which seems icky.

The use of talloc_strdup here is probably overkill, but it avoids
having to enforce that thread->authors is never mutated outside
_resolve_thread_authors_string.
2017-12-21 09:22:30 -04:00
Daniel Kahn Gillmor
8e1c3fa5bb test: retrying lock tests are broken when built against older Xapian
If we're building against a version of Xapian that doesn't offer
retrying the lock, we should be honest and describe the tests as
broken, rather than marking them as missing a test prerequisite.

missing test prerequisites should be for specific components of the
test harness that are missing, not for the backend library notmuch
uses.
2017-12-19 07:14:13 -04:00
Daniel Kahn Gillmor
bace15c275 test: named query tests are broken when missing Xapian FieldProcessor
Named queries don't work without Xapian FieldProcessor.  Rather than
silently skipping them, we should explictly mark them as broken when
building against an older version of Xapian.
2017-12-19 07:14:07 -04:00
Daniel Kahn Gillmor
776a2982ae test: searching by absolute date is broken without Xapian FieldProcessor
Previously, the test suite had simply silently skipped the absolute
date test if we're using an archaic version of Xapian.  For
correctness, we should instead mark the test as broken.

This also changes from string to numeric comparison when checking
NOMTUCH_HAVE_XAPIAN_FIELD_PROCESSOR for consistency with other tests.
2017-12-19 07:14:03 -04:00
l-m-h@web.de
660f1a5a33 test: Add test to unset config items with the python bindings 2017-12-19 06:42:50 -04:00
l-m-h@web.de
3444c731d2 python: Rename get_config_list to get_configs
The old name has a bit of a feeling of hungarian notation.  Also many
generators in the core are named with the suffix "s" to indicate
iterables: dict.items, dict.keys for example.
2017-12-19 06:42:11 -04:00
l-m-h@web.de
7a07fd8625 test: Add tests for new python bindings
The tests where adopted from the tests for the corresponding C functions
in test/T590-libconfig.sh.
2017-12-19 06:41:23 -04:00
Jani Nikula
7354d6d8f6 test: test regexp based new.ignore
Just some basics.
2017-12-15 07:55:02 -04:00
Jani Nikula
f3fc97c000 cli: add support for only printing the addresses in notmuch address
The notmuch address output is much more useful for scripts with just
the addresses printed. Support this using the --output=address option.
2017-12-14 21:28:50 -04:00
Jani Nikula
ab4b033c70 test: expand argument parsing sanity checks
Test the various boolean formats and --no- prefixed boolean and
keyword flag arguments.
2017-12-13 08:31:03 -04:00
David Bremner
040c3236af Merge branch 'release'
Conflicts:
        NEWS

Add in NEWS from point release
2017-12-08 22:19:06 -04:00
David Bremner
151b2c3747 test/emacs: add exploit mitigation test
This test will pass if either the notmuch show mitigation code is
working correctly, or upstream emacs mime handling code has it's own
fix for https://bugs.gnu.org/28350.
2017-12-08 20:36:20 -04:00
Daniel Kahn Gillmor
cb855d8a9d crypto: signature verification reports valid User IDs
When i'm trying to understand a message signature, i care that i know
who it came from (the "validity" of the identity associated with the
key), *not* whether i'm willing to accept the keyholder's other
identity assertions (the "trust" associated with the certificate).

We've been reporting User ID information based on the "trust"
associated with the certificate, because GMime didn't clearly expose
the validity of the User IDs.

This change relies on fixes made in GMime 3.0.3 and later which
include https://github.com/jstedfast/gmime/pull/18.
2017-12-08 20:35:18 -04:00
Daniel Kahn Gillmor
fccebbaeef crypto: add --decrypt=nostash to avoid stashing session keys
Here's the configuration choice for people who want a cleartext index,
but don't want stashed session keys.

Interestingly, this "nostash" decryption policy is actually the same
policy that should be used by "notmuch show" and "notmuch reply",
since they never modify the index or database when they are invoked
with --decrypt.

We take advantage of this parallel to tune the behavior of those
programs so that we're not requesting session keys from GnuPG during
"show" and "reply" that we would then otherwise just throw away.
2017-12-08 08:08:47 -04:00
Daniel Kahn Gillmor
29648a137c crypto: actually stash session keys when decrypt=true
If you're going to store the cleartext index of an encrypted message,
in most situations you might just as well store the session key.
Doing this storage has efficiency and recoverability advantages.

Combined with a schedule of regular OpenPGP subkey rotation and
destruction, this can also offer security benefits, like "deletable
e-mail", which is the store-and-forward analog to "forward secrecy".

But wait, i hear you saying, i have a special need to store cleartext
indexes but it's really bad for me to store session keys!  Maybe
(let's imagine) i get lots of e-mails with incriminating photos
attached, and i want to be able to search for them by the text in the
e-mail, but i don't want someone with access to the index to be
actually able to see the photos themselves.

Fret not, the next patch in this series will support your wacky
uncommon use case.
2017-12-08 08:08:47 -04:00
Daniel Kahn Gillmor
6a9626a2fd cli/reindex: destroy stashed session keys when --decrypt=false
There are some situations where the user wants to get rid of the
cleartext index of a message.  For example, if they're indexing
encrypted messages normally, but suddenly they run across a message
that they really don't want any trace of in their index.

In that case, the natural thing to do is:

   notmuch reindex --decrypt=false id:whatever@example.biz

But of course, clearing the cleartext index without clearing the
stashed session key is just silly.  So we do the expected thing and
also destroy any stashed session keys while we're destroying the index
of the cleartext.

Note that stashed session keys are stored in the xapian database, but
xapian does not currently allow safe deletion (see
https://trac.xapian.org/ticket/742).

As a workaround, after removing session keys and cleartext material
from the database, the user probably should do something like "notmuch
compact" to try to purge whatever recoverable data is left in the
xapian freelist.  This problem really needs to be addressed within
xapian, though, if we want it fixed right.
2017-12-08 08:08:47 -04:00
Daniel Kahn Gillmor
076f86025d cli/new, insert, reindex: change index.decrypt to "auto" by default
The new "auto" decryption policy is not only good for "notmuch show"
and "notmuch reindex".  It's also useful for indexing messages --
there's no good reason to not try to go ahead and index the cleartext
of a message that we have a stashed session key for.

This change updates the defaults and tunes the test suite to make sure
that they have taken effect.
2017-12-08 08:08:46 -04:00
Daniel Kahn Gillmor
a1260896f6 cli/show: use decryption policy "auto" by default.
When showing a message, if the user doesn't specify --decrypt= at all,
but a stashed session key is known to notmuch, notmuch should just go
ahead and try to decrypt the message with the session key (without
bothering the user for access to their asymmetric secret key).

The user can disable this at the command line with --decrypt=false if
they really don't want to look at the e-mail that they've asked
notmuch to show them.

and of course, "notmuch show --decrypt" still works for accessing the
user's secret keys if necessary.
2017-12-08 08:08:46 -04:00
Daniel Kahn Gillmor
6802b333eb cli/reply: use decryption policy "auto" by default.
If the user doesn't specify --decrypt= at all, but a stashed session
key is known to notmuch, when replying to an encrypted message,
notmuch should just go ahead and decrypt.

The user can disable this at the command line with --decrypt=false,
though it's not clear why they would ever want to do that.
2017-12-08 08:08:46 -04:00
Daniel Kahn Gillmor
e4890b5bf9 crypto: new decryption policy "auto"
This new automatic decryption policy should make it possible to
decrypt messages that we have stashed session keys for, without
incurring a call to the user's asymmetric keys.
2017-12-08 08:07:53 -04:00
Daniel Kahn Gillmor
798aa789b5 lib: convert notmuch decryption policy to an enum
Future patches in this series will introduce new policies; this merely
readies the way for them.

We also convert --try-decrypt to a keyword argument instead of a boolean.
2017-12-08 08:07:02 -04:00
Daniel Kahn Gillmor
d3964e81ac indexing: Change from try_decrypt to decrypt
the command-line interface for indexing (reindex, new, insert) used
--try-decrypt; and the configuration records used index.try_decrypt.
But by comparison with "show" and "reply", there doesn't seem to be
any reason for the "try" prefix.

This changeset adjusts the command-line interface and the
configuration interface.

For the moment, i've left indexopts_{set,get}_try_decrypt alone.  The
subsequent changeset will address those.
2017-12-08 08:05:53 -04:00
Daniel Kahn Gillmor
9d23ebfcc3 test: session keys are known broken without session key support
If the version of GMime we're building against doesn't support session
key extraction or re-use, mark the tests that rely on session key
capabilities as known-broken.

This should resolve test suite failures on ubuntu trusty and debian
jessie and earlier, which have GMime 2.6.20 -- session key support was
introduced in GMime 2.6.21.
2017-12-05 20:09:25 -04:00
Daniel Kahn Gillmor
6d7249fec9 crypto: Test restore of cleartext index from stashed session keys
If you've got a notmuch dump that includes stashed session keys for
every decrypted message, and you've got your message archive, you
should be able to get back to the same index that you had before.

Here we add a simple test that give some flavor of how that works.
2017-12-04 21:53:05 -04:00
Daniel Kahn Gillmor
836ec85b0c test/corpora: add an encrypted message for index decryption tests 2017-12-04 21:53:05 -04:00
David Bremner
1e4e8e3bf5 test: unbreak performance tests
In 8e7fb88237 Jani replaced the use of $(pwd -P) to find the
TEST_DIRECTORY in order to better support out of tree
builds. Unfortunately the performance-tests need a different value for
the variable and were thus broken.

This commit splits out the setting of this variable for the two sets
of tests.  Performance tests still don't work out of tree, because
the handling of the downloaded corpus needs to be updated.
2017-12-04 21:04:38 -04:00
Jani Nikula
cf08295c50 test: test notmuch insert --folder=""
Test insert into top level folder.
2017-11-08 09:47:14 -05:00
Jani Nikula
1b91884296 test: fix test database backup/restore location
backup_database() and restore_database() used to store the backups in
the test specific temporary directory, through the current working
directory being there. Commit 8e7fb88237 ("test: use source and
build paths in test-lib-common.sh") started using a test specific
backup directories under the build tree test directory. This was in
error. Switch back to the old location, but using paths to the
location instead of relying on current working directory.

Reported by Daniel Kahn Gillmor <dkg@fifthhorseman.net>.
2017-10-29 09:30:30 -07:00
Daniel Kahn Gillmor
8116891744 cli/reindex: add --try-decrypt=(true|false)
Enable override of the index.try_decrypt setting on a per-run basis
when invoking "notmuch reindex".  This allows the possibility of (for
example) an emacs keybinding that adds the cleartext of the currently
shown decrypted message to the index, making it searchable in the
future.

It also enables one-time indexing of all messages matching some query,
like so:

    notmuch reindex tag:encrypted and\
       not property:index.decryption=success and\
       from:alice@example.org

We also update the documentation and tab completion, and add a few
more tests.
2017-10-21 19:58:52 -03:00
Daniel Kahn Gillmor
c5356b9ed5 cli/insert: add --try-decrypt=(true|false)
Enable override of the index.try_decrypt setting on a per-message
basis when invoking "notmuch insert".

We also update the documentation and tab completion, and add more tests.
2017-10-21 19:58:43 -03:00
Daniel Kahn Gillmor
35456d4b0c cli/new: add --try-decrypt=(true|false)
Enable override of the index.try_decrypt setting during "notmuch new"
on a per-invocation basis.

We update the documentation and tab completion, and also add a test.
2017-10-21 19:58:32 -03:00
Daniel Kahn Gillmor
92f318abe4 tests: emacs_fcc_message: allow passing --arguments to notmuch new
Subsequent patches may want to send GNU-style --long-arguments to
notmuch new in the test suite, in particular when invoking
emacs_fcc_message.  This changeset makes that possible.
2017-10-21 19:58:23 -03:00
Jani Nikula
d7929d61a2 test: use source path in add_gnupg_home
Make a distinction between source and build directories.
2017-10-21 16:34:04 -03:00
Jani Nikula
d455c8b74a test: remove --root option and fix TMP_DIRECTORY cleanup
The primary motivation here is to fix TMP_DIRECTORY cleanup prior to
running each test when the current working directory is not the test
subdirectory. Tests with failures would leave their TMP_DIRECTORY
directory behind for debugging, and repeated out-of-tree test runs
would have old temp directories. (This lead to e.g. T310-emacs.sh
hanging because emacs would prompt for overwriting files.)

We remove the likely anyway defunct --root test option while at it,
just to be on the safe side when doing 'rm -rf' on the TMP_DIRECTORY.
2017-10-21 11:32:20 -03:00
Jani Nikula
409d877364 test: use source and build paths in T590-thread-breakage.sh
Make a distinction between source and build directories.
2017-10-20 20:50:17 -03:00
Jani Nikula
3c23780396 test: use source and build paths in T480-hex-escaping.sh
Make a distinction between source and build directories.
2017-10-20 20:50:03 -03:00
Jani Nikula
1ab72b13cd test: use source and build paths in T4?0-emacs-*.sh
Make a distinction between source and build directories.
2017-10-20 20:49:50 -03:00
Jani Nikula
e216d9b7ab test: use source and build paths in T380-atomicity.sh
Make a distinction between source and build directories.
2017-10-20 20:49:37 -03:00
Jani Nikula
0364a6d74b test: use source and build paths in T360-symbol-hiding.sh
Make a distinction between source and build directories.
2017-10-20 20:49:26 -03:00
Jani Nikula
bc349ad316 test: use source and build paths in T355-smime.sh
Make a distinction between source and build directories.
2017-10-20 20:49:18 -03:00
Jani Nikula
f6b098b1a6 test: use source and build paths in T310-emacs.sh
Make a distinction between source and build directories.
2017-10-20 19:56:13 -03:00
Jani Nikula
b079133ca8 test: use source and build paths in T260-thread-order.sh
Make a distinction between source and build directories.
2017-10-20 19:55:57 -03:00
Jani Nikula
32c088b524 test: use source and build paths in T160-json.sh and T170-sexp.sh
Make a distinction between source and build directories.
2017-10-20 19:55:49 -03:00
Jani Nikula
fc12f6f07b test: use source and build paths in T000-basic.sh
Make a distinction between source and build directories.
2017-10-20 19:55:41 -03:00
Jani Nikula
a8cd1ec26b test: use source and build paths in notmuch-test
Make a distinction between source and build directories.
2017-10-20 19:55:31 -03:00
Jani Nikula
8dfa116010 test: mkdir and cd to $TMP_DIRECTORY instead of $test for testing
$TMP_DIRECTORY is a full path, while $test is not.
2017-10-20 19:55:10 -03:00
Jani Nikula
8e7fb88237 test: use source and build paths in test-lib-common.sh
Make a distinction between source and build directories.

The expectation is that anyone sourcing test-lib-common.sh has sourced
export-dirs.sh.
2017-10-20 19:54:43 -03:00
Jani Nikula
a9894fce06 test: use source and build paths in test-lib.sh
Make a distinction between source and build directories.
2017-10-20 19:54:08 -03:00
Jani Nikula
1f3c114f08 test: add a script to figure out source and build directories
Expect the tests to be run somewhere in the build directory
hierarchy. Allow for source and build directories to be passed through
the environment.
2017-10-20 19:53:55 -03:00
Jani Nikula
e1bd4f2363 test: check for notmuch binary in test-lib.sh
Move notmuch executable check into common code. Redundant for
notmuch-test runs, but works also for individual tests.
2017-10-20 19:53:31 -03:00
Jani Nikula
a863de1e43 test: use $(dirname "$0") for sourcing test-lib.sh
Don't assume the tests are always run from within the source tree.
2017-10-20 19:52:49 -03:00
Jani Nikula
0dd58ea7e7 build: use NOTMUCH_SRCDIR for referencing notmuch-test
Don't assume the tests are always run from within the source tree.
2017-10-20 19:52:24 -03:00
Jani Nikula
1c2152ce73 test: use corpora/default not corpus in T480-hex-escaping.sh
Turns out round trip tests didn't really round trip anything. Broken
by yours truly in 971cdc72cd ("test: make it possible to have
multiple corpora"). Ooops.
2017-10-20 19:52:14 -03:00
Daniel Kahn Gillmor
ee5df7d7db tests: prepare for more crypto tests (using add_gnupg_home)
Move add_gnupg_home to test-lib.sh to prepare it for reuse.
2017-10-14 10:53:42 -03:00
Jani Nikula
0f314c0c99 cli: convert notmuch_bool_t to stdbool
C99 stdbool turned 18 this year. There really is no reason to use our
own, except in the library interface for backward
compatibility. Convert the cli and test binaries to stdbool.
2017-10-09 22:24:02 -03:00
Jani Nikula
816097f05b hex-xcode: use notmuch_bool_t for boolean arguments
Pedantically correct, although they're the same underlying type.
2017-10-04 22:02:45 -03:00
Jani Nikula
9165df2135 test: expand argument parsing tests
Test and use the new .present field, only output the parameters
given. Test space between parameter name and value.
2017-10-04 22:01:50 -03:00
Jani Nikula
25960b5ecd test: add opt_inherit to arg-test
Just split the arguments to two opt desc arrays.
2017-10-04 22:01:15 -03:00
Jani Nikula
e8cba9da18 test: add boolean argument to arg-test
Surprisingly it's not there.
2017-10-04 22:00:54 -03:00
Jani Nikula
4a6721970a cli: use designated initializers for opt desc
Several changes at once, just to not have to change the same lines
several times over:

- Use designated initializers to initialize opt desc arrays.

- Only initialize the needed fields.

- Remove arg_id (short options) as unused.

- Replace opt_type and output_var with several type safe output
  variables, where the output variable being non-NULL determines the
  type. Introduce checks to ensure only one is set. The downside is
  some waste of const space per argument; this could be saved by
  retaining opt_type and using a union, but that's still pretty
  verbose.

- Fix some variables due to the type safety. Mostly a good thing, but
  leads to some enums being changed to ints. This is pedantically
  correct, but somewhat annoying. We could also cast, but that defeats
  the purpose a bit.

- Terminate the opt desc arrays using {}.

The output variable type safety and the ability to add new fields for
just some output types or arguments are the big wins. For example, if
we wanted to add a variable to set when the argument is present, we
could do so for just the arguments that need it.

Beauty is in the eye of the beholder, but I think this looks nice when
defining the arguments, and reduces some of the verbosity we have
there.
2017-10-04 22:00:42 -03:00
Florian Klink
073d05eec3 T390-python: add test for get_message_parts and special characters
This imports a message with ISO-8859-2 encoded characters, then opens
the database using the python bindings. We peek through all mesage
parts, afterwards print the message id.

Signed-off-by: Florian Klink <flokli@flokli.de>
Signed-off-by: Andreas Rammhold <andreas@rammhold.de>
2017-10-02 07:21:41 -03:00
Jani Nikula
b2b65b4e6d emacs: sanitize subject in replies
Commit a7964c86d1 ("emacs: Sanitize authors and subjects in search
and show") added sanitization of header information for display. Do
the same for reply subjects.

This fixes the long-standing annoying artefact of certain versions of
mailman using tab as folding whitespace, leading to tabs in reply
subjects.
2017-09-27 08:15:02 -03:00
Jani Nikula
77eb76d9df test: add emacs reply test for subjects with TAB
Expect TABs to be sanitized from the subject line. Known broken.
2017-09-27 08:10:31 -03:00
Daniel Kahn Gillmor
f4ac5ecd5c lib: index the content-type of the parts of encrypted messages
This is a logical followup to "lib: index the content type of
signature parts", which will make it easier to record the message
structure of all messages.
2017-09-17 20:01:19 -03:00
Jani Nikula
55c047ee0b lib: index the content type of signature parts
It's useful (*) to be able to easily find messages with certain types
of signatures. Having the mimetype: prefix searches fail for some
content types is also genuinely surprising (*). Index the content type
of signature parts.

While at it, switch to the gmime convenience constants for content and
signature part indexes.

*) At least for developers of email software!
2017-09-17 20:01:00 -03:00
Jani Nikula
30c2761465 test: signed and encrypted part content type indexing
Add known broken subtests for searching signed and encrypted messages
using mimetype: prefix search for the content-types of signed and
encrypted parts.
2017-09-17 20:00:43 -03:00
David Bremner
64e30aeb24 test/duplicate-mid: check for subject with notmuch-show
In [1] Mark showed that the the current code (d7a49e81) is not
consistent in it's handling of subjects of messages with duplicate
message-ids (or in notmuch-speak, of messages with multiple files).
notmuch-search uses indexing order and explicitedly preserves the
first. notmuch-show (apparently) uses alphabetical (or at least xapian
term order) of filenames. In a perfect world we would probably report
all subjects in the json output; at the very least we should be
consistent.

[1]: id:87378dny3d.fsf@qmul.ac.uk
2017-09-05 21:53:06 -03:00
David Bremner
debfae20db lib: enforce that n_message_reindex takes headers from first file
This is still a bit stopgap to be only choosing one set of headers,
but this seems like a more defensible set of headers to choose.
2017-09-05 21:51:57 -03:00
David Bremner
36a3d65034 test: known broken test for subject after reindexing
In [1], Mark gave a test that was behaving strangly. This turns out to
be specific to reindexing.  I suppose one could argue that picking the
lexicographically last file name is a defensible choice, but it's
almost as easy to take the first, which seems more intuitive. So mark
the current situation as broken.

[1]: id:1503859703-2973-1-git-send-email-markwalters1009@gmail.com
2017-09-05 21:51:21 -03:00
David Bremner
17aebb6e7d test/duplicate-mid: clarify index order vs filename order
The existing test for notmuch search had the first in filename order
the same as the first indexed, which made it harder to understand what
the underlying behaviour is. Add a file with a lexicographically
smaller name, but later index time to clarify this.
2017-09-05 21:51:07 -03:00
David Bremner
44914f1c3b test: make fallback to duplicate test more robust.
The original intent of this test was to verify that notmuch show was
not crashing when the first file (where headers are being read from)
was deleted. Run the output through some sanitization so that as we
add and delete copies we don't have to update this test.
2017-09-05 21:50:54 -03:00
Tomi Ollila
7526538808 test: notmuch_drop_mail_headers() style update
Changed "" quotes to '' as we're not supposed to dynamically
alter python program (via shell $variable expansion).

Added space to python program to match general python style.

Replaced $* with 'idiomatic' "$@" to serve as better example.
2017-09-04 08:03:52 -03:00
David Bremner
ec37900c5f test/crypto: remove headers more robustly
In [1], Vladimir Panteleev observed that the In-Reply-To and
References headers could be wrapped in the 'default' output format of
notmuch-reply, depending on the version of Emacs creating the
message. In my own experiments notmuch-reply sometimes wraps headers
with only one message-id if that message-id is long enough. However it
happens, this causes the previous approach using grep to fail.

Since I found the proposed unwrapping shell fragment in [1] a bit hard
to follow, I decided to write a little python script instead. Then
Tomi suggested a slight generalization of my script, and here we are.

[1] id:20170817175145.3204-7-notmuch@thecybershadow.net
2017-09-03 08:06:15 -03:00
Vladimir Panteleev
95b82bb326 test: Fix T050-new.sh on some Travis CI machines
On some system configurations, setting a breakpoint on the "add_file"
function then issuing "continue" in gdb causes the debugger to
seemingly jump over the add_file invocation. This results in a test
failure, as the "Handle files vanishing between scandir and add_file"
subtest expects add_file to be called and fail due to the vanishing
file. The compiler optimization level also plays a role - the problem
can be reproduced with CFLAGS having -O2 but not -Og.

This problem was observed manifesting as a test failure on Travis CI
configured with "dist: trusty" and "sudo: false". It was not
reproducible on a local Docker image of Travis' runtime environment,
so Travis' virtualization infrastructure likely plays a role as well.

* T050-new.sh: Breakpoint notmuch_database_add_message instead of
  add_file to the same effect, and avoid bad gdb behaviour on Travis
  CI.

Amended by db:

	s/notmuch_database_add_message/notmuch_database_index_file/

Somehow the wrapper function doesn't work as a breakpoint; perhaps due
to inlining.
2017-08-30 07:24:44 -03:00
David Bremner
536d695a31 test/smtp-dummy: convert to 'goto DONE' style
Clean up several cppcheck warnings of the form

      - test/smtp-dummy.c:170: error: Resource leak: output

Conform to overall notmuch code style.
2017-08-30 07:16:24 -03:00
David Bremner
1fa5bd282e test/smtp-dummy: uncrustify
For some reason lost in the mists of time this code was indented 8 spaces.
2017-08-30 07:14:18 -03:00
David Bremner
0082a55785 CLI/new: support maildir synced tags in new.tags
We reorder reading maildir flags to avoid overwriting 'new.tags'. The
inverted status of 'unread' means the maildir flag needs to be checked
a second time.

I backpedalled here on the idea of supporting 'new.tags' without
'unread' in the presence of maildir syncing.  For files in 'new/', it
seems quite natural to tag them as 'unread'.
2017-08-29 21:57:38 -03:00
David Edmondson
ca4852649e test: Perform T170 tests that don't require dtach before any that do.
This avoids the later tests seeing different versions of the database
depending on whether dtach is available.
2017-08-28 08:02:02 -03:00
Vladimir Panteleev
8c772eaa0d test: Use small Python script for JSON normalization instead of json.tool
json.tool does not sort or otherwise normalize the order of JSON keys
in its output, which can result in test failures on some test systems.

Instead, use a one-line Python script passed to the interpreter
directly on its command line. Use sort_keys=True for json.dump to
ensure the key order is normalized. The script works with both Python
2 and 3.

* test/test-lib.sh: Update test_expect_equal_json.
2017-08-23 08:07:45 -03:00
Daniel Kahn Gillmor
b10ce6bc23 database: add n_d_index_file (deprecates n_d_add_message)
We need a way to pass parameters to the indexing functionality on the
first index, not just on reindexing.  The obvious place is in
notmuch_database_add_message.  But since modifying the argument list
would break both API and ABI, we needed a new name.

I considered notmuch_database_add_message_with_params(), but the
functionality we're talking about doesn't always add a message.  It
tries to index a specific file, possibly adding a message, but
possibly doing other things, like adding terms to an existing message,
or failing to deal with message objects entirely (e.g. because the
file didn't contain a message).

So i chose the function name notmuch_database_index_file.

I confess i'm a little concerned about confusing future notmuch
developers with the new name, since we already have a private
_notmuch_message_index_file function, and the two do rather different
things.  But i think the added clarity for people linking against the
future libnotmuch and the capacity for using index parameters makes
this a worthwhile tradeoff.  (that said, if anyone has another name
that they strongly prefer, i'd be happy to go with it)

This changeset also adjusts the tests so that we test whether the new,
preferred function returns bad values (since the deprecated function
just calls the new one).

We can keep the deprecated n_d_add_message function around as long as
we like, but at the next place where we're forced to break API or ABI
we can probably choose to drop the name relatively safely.

NOTE: there is probably more cleanup to do in the ruby and go bindings
to complete the deprecation directly.  I don't know those languages
well enough to attempt a fix; i don't know how to test them; and i
don't know the culture around those languages about API additions or
deprecations.
2017-08-23 07:38:37 -03:00
Yuri Volchkov
09fa51303c show: workaround for the missing file problem
This patch fixes the 'Deleted first duplicate file does not stop
notmuch show from working' test.

If a message to be shown has several duplicated files, and for some
reason the first file in the list is not available anymore, notmuch
will exit with an error.

This is clearly a problem in the database, but we are not going to let
this problem be a show-stopper. Let's walk through the list, and show
the first existing file.

Signed-off-by: Yuri Volchkov <yuri.volchkov@gmail.com>
2017-08-22 18:48:29 -03:00
Yuri Volchkov
39cc84e9fc test: show id:<> works even if the first duplicate is deleted
Signed-off-by: Yuri Volchkov <yuri.volchkov@gmail.com>
2017-08-22 18:48:19 -03:00
Yuri Volchkov
33a170e116 insert: strip trailing / in folder path
This patch fixes the "Insert message into folder with trailing /"
test. The problem was insufficient path canonization.

From database's point of view, "Sent" and "Sent/" are different
folders. If user runs (note the last '/'):

    notmuch insert --folder=maildir/Sent/ < test.msg

notmuch will create an extra XDIRECTORY record for the folder
'Sent/'. This means that database will have _TWO_ records for _ONE_
physical folder: 'Sent' and 'Sent/'. However, the 'notmuch new'
command will update only records related to the first one (the correct
one).

Now, if user moved the email file (e.g. from 'Sent/new' to
'Sent/cur'), 'notmuch new' will add a record about the new file, but
will not delete the old record.

Signed-off-by: Yuri Volchkov <yuri.volchkov@gmail.com>
2017-08-22 18:48:09 -03:00
David Bremner
1092c747ef test: move generate_message, add_message into test-lib-common.sh
The plan is to use at least the former in the perf test suite.
2017-08-20 08:34:03 -03:00
Yuri Volchkov
cd8551d53a test: insert into the folder with trailing /
From database's point of view, "Drafts" and "Drafts/" are different
folders

Signed-off-by: Yuri Volchkov <yuri.volchkov@gmail.com>
Amended: add test_subtest_known_broken (db)
2017-08-20 08:28:55 -03:00
Yuri Volchkov
02761b2e5c test: remove remainder of previously killed basic test
In the commit 51cd69feb1 the part of the
test "test runs if prerequisite is satisfied" has been
removed. However, there was a remainder of that test - variable
'haveit'.

Kill it, to not to confuse people.

Signed-off-by: Yuri Volchkov <yuri.volchkov@gmail.com>
2017-08-18 19:45:17 -03:00
Vladimir Panteleev
ca4688e103 Use rooted paths in .gitignore files
A leading / in paths in a .gitignore file matches the beginning of the
path, meaning that for patterns without slashes, git will match files
only in the current directory as opposed to in any subdirectory.

Prefix relevant paths with / in .gitignore files, to prevent
accidentally ignoring files in subdirectories and possibly slightly
improve the performance of "git status".
2017-08-18 19:42:35 -03:00
Vladimir Panteleev
4d44976135 test: Update extant references to corpus.mail
971cdc72cd renamed corpus.mail to
corpora.mail. Although 971cdc72cd
updated some of the remaining corpus.mail references, two remained,
causing the test suite to leave behind an unignored corpora.mail
directory.
2017-08-18 19:42:19 -03:00
David Bremner
f385055528 test: add test for ,S message without 'unread' in new.tags
This is arguably overkill, but it helps to understand the complicated
interactions here between maildir tags and configuration.
2017-08-16 21:53:43 -03:00
David Bremner
e1f9ab4849 test: add missing quotes in maildir-sync test.
Oops. ';' is significant in the shell. Who knew.
2017-08-16 21:48:51 -03:00
David Bremner
0967e46475 test: add broken tests for maildir syncing
Users should be able to specify synced tags in new.tags
2017-08-06 15:40:28 -04:00
Daniel Kahn Gillmor
e5beec39d6 add "notmuch reindex" subcommand
This new subcommand takes a set of search terms, and re-indexes the
list of matching messages.
2017-08-01 21:17:47 -04:00
David Bremner
ed4a9082c0 cli/search: print total number of files matched in summary output.
The structured output formats already have all of the filenames. This
is an easy bit of UI change to make the multiple files visible.
2017-08-01 21:17:47 -04:00
David Bremner
411675a6ce lib: index message files with duplicate message-ids
The corresponding xapian document just gets more terms added to it,
but this doesn't seem to break anything. Values on the other hand get
overwritten, which is a bit annoying, but arguably it is not worse to
take the values (from, subject, date) from the last file indexed
rather than the first.
2017-08-01 21:17:47 -04:00
David Bremner
639aced9c9 test: add known broken tests for duplicate message id
There are many other problems that could be tested, but these ones we
have some hope of fixing because it doesn't require UI changes, just
indexing changes.
2017-08-01 21:17:47 -04:00
David Bremner
1d82110f83 emacs: convert remaining format-versions from 3 to 4
This is needed for consistent beheviour between notmuch built against
gmime-2.6 and gmime-3.0 w.r.t. error reporting.
2017-07-18 06:57:26 -03:00
Daniel Kahn Gillmor
87bdfbc91f Fix orthography 2017-07-18 06:50:44 -03:00
David Bremner
f1bcb6b7c2 emacs: change default for notmuch-crypto-process-mime to t
There are some cases like remote usage where this might cause
problems, but those users can easily customize the variable. The
inconvenience seems to be outweighed by the security benefit for most
users.
2017-07-15 09:11:32 -03:00
Daniel Kahn Gillmor
e142de643d config: deprecate/drop crypto.gpg_path under gmime 2.6/3.0
gmime 3.0 no longer offers a means to set the path for gpg.

Users can set $PATH anyway if they want to pick a
differently-installed gpg (e.g. /usr/local/bin/gpg), so this isn't
much of a reduction in functionality.

The one main difference is for people who have tried to use "gpg2" to
make use of gpg 2.1, but that isn't usefully co-installable anyway.
2017-07-14 21:23:52 -03:00
Daniel Kahn Gillmor
8492298a61 Add additional munged reply-to tests
The reply-to munging code might behave differently whether there's an
exact match on the strings or not, or whether the string is a raw
addr-spec instead of an name-addr.  These tests cover those variations
(i also had to tweak json output further below when this new test was
added).
2017-07-14 21:23:52 -03:00
Daniel Kahn Gillmor
23a4d09773 Ensure that "notmuch reply" succeeds during testing.
In some (bad!) cases, "notmuch reply" might fail, or might even
segfault.  If this happens, it indicates a bug, and the test suite
should notice it.
2017-07-14 21:23:52 -03:00
David Bremner
a58c75fdf6 test/crypto: mark extra space in userid as a bug in gmime-2.6
I can't see the space in the output of gpg -K --with-colons
2017-07-14 17:58:09 -03:00
David Bremner
a32dd81a55 cli: add space separator for keyword, string, and int arguments
Defer the complication of optional boolean arguments for later (never?).
2017-07-12 17:46:29 -03:00
David Bremner
de9baa29dc cli: change api of parse_option
The idea is to allow it (in a future commit) advance to the next argv
element to get a value
2017-07-12 17:44:15 -03:00
David Bremner
87ca29b7a7 test/crypto: add test for corrupted signatures
Have an empty list of signatures is not the nicest mechanism to signal
complete failure to verify, but it's the one we currently have.
2017-07-05 07:36:20 -03:00
David Bremner
4f48ae4a5f test: add test for modified pgp/mime signed message
This is a good thing to test in any case, but particularly as we hit
the dark corners of the gmime crypto API.
2017-07-05 07:36:20 -03:00
David Bremner
71f04b9a67 test: mark test as broken in gmime 3.0
Currently I'm not sure what the intent of this test is, so it's not
clear if the new answer is better or worse than the old one.
2017-07-05 07:13:02 -03:00
David Bremner
f7dd69d4c6 test: mark inclusion of headers as broken in gmime-2.x
The output of headers here reflects an underlying bug / quirk of
gmime-2.6.
2017-07-05 07:12:20 -03:00
David Bremner
3805898e3e test/multipart: reorganize creation of multipart message
We want to have the bodies of the multipart available in a file on
their own for planned modifications to tests.
2017-07-05 07:12:08 -03:00
David Bremner
e1b7d32db0 cli: implement structured output version 4
Since the error field is unused by the emacs front end, no changes are
needed other than bumping the format version number.

As it is, this is a bit overengineered, but it will reduce duplication
when we support gmime 3.0
2017-07-04 08:32:44 -03:00
David Bremner
b6f87c3085 emacs: convert to use format-version 3
Keeping the emacs interface running old format-versions makes it
impossible to test new format-versions or deprecate old ones.
2017-07-04 08:32:28 -03:00
David Bremner
6dd00d6486 lib/index: add simple html filter
The filter just drops all (HTML) tags. As an enabling change, pass the
content type to the filter constructor so we can decide which scanner
to user.
2017-07-01 12:32:27 -03:00
David Bremner
1ec634461e Merge branch 'release' 2017-06-01 07:54:38 -03:00
David Bremner
11fda0bf2f cli/dump: don't include tags when not asked for
Add in the analogous test for tags that is given for properties a few
lines below.
2017-06-01 07:07:32 -03:00
David Bremner
e68022416d test: add known broken test --include=properties
Reported in [1], --include=properties currently implies
--include=tags, but it should not.

[1] id:87y3u8vjbo.fsf@tethera.net
2017-06-01 07:03:06 -03:00
David Bremner
6ac3d8c27f test: define GMime version dependant breakage
We have some tests where the gmime 3 behaviour seems like a bug fix,
others where it's less clear, so we allow both possibilities.
2017-05-31 21:54:08 -03:00
David Bremner
51cf6dbab2 test/thread-naming: remove excess escaping from sender address.
This is another case where the behaviour of gmime-2.6 and gmime-3.0
seems to differ. It may be that we prefer the more lax parsing of the
previous version, but that should be tested separately.
2017-05-30 09:02:51 -03:00
Tomi Ollila
5f71a4d3e1 exit lingering gpg agents at the end of relevant tests
Since gnupg 2.1.20, gpg-agent no longer shut itself down when
$GNUPGHOME directory is removed.

Add exit hooks to the test modules which execute `gpgconf --kill all`

Add exit hooks to execute `gpgconf --kill all` in the modules that
create $GNUPGHOME for gpg to work with.
2017-05-26 07:26:06 -03:00
Tomi Ollila
eb157f8841 test-lib.sh: add "atexit" functionality
New function at_exit_function registers given function to be called
at script termination.

Functions so registered are called in the reverse order of their
registration; no arguments are passed.

Function is called only once; re-adding with function name already
registered will remove previous registration.

New function rm_exit_function can be used to remove registration.

Modules (and possibly test-lib.sh functions) in future commits will
register such functions.
2017-05-26 07:25:55 -03:00
Jani Nikula
bc11759dd1 build: switch to hiding libnotmuch symbols by default
The dynamic generation of the linker version script for libnotmuch
exports has grown rather complicated.

Reverse the visibility control by hiding symbols by default using
-fvisibility=hidden, and explicitly exporting symbols in notmuch.h
using #pragma GCC visibility. (We could also use __attribute__
((visibility ("default"))) for each exported function, but the pragma
is more convenient.)

The above is not quite enough alone, as it would "leak" a number of
weak symbols from Xapian and C++ standard library. Combine it with a
small static version script that filters out everything except the
notmuch_* symbols that we explicitly exposed, and the C++ RTTI
typeinfo symbols for exception handling.

Finally, as the symbol hiding test can no longer look at the generated
symbol table, switch the test to parse the functions from notmuch.h.
2017-05-12 07:17:18 -03:00
David Bremner
11d47950c1 lib: Add regexp expansion for for tags and paths
From a UI perspective this looks similar to what was already provided
for from, subject, and mid, but the implementation is quite
different. It uses the database's list of terms to construct a term
based query equivalent to the passed regular expression.
2017-05-09 07:44:29 -03:00
David Bremner
eab365c742 lib: Add regexp searching for mid: prefix
The bulk of the change is passing in the field options to the regexp
field processor, so that we can properly handle the
fallback (non-regexp case).
2017-05-09 07:44:15 -03:00
David Bremner
77c9ec1fdd test: add known broken test for indexing html
'quite' on IRC reported that notmuch new was grinding to a halt during
initial indexing, and we eventually narrowed the problem down to some
html parts with large embedded images. These cause the number of terms
added to the Xapian database to explode (the first 400 messages
generated 4.6M unique terms), and of course the resulting terms are
not much use for searching.

The second test is sanity check for any "improved" indexing of HTML.
2017-04-20 06:59:40 -03:00
David Bremner
e08f5f76e4 test: add 'lkml' corpus
These 210 messages are in several long threads, which is good for
testing our threading code, and may be useful just as a larger test
corpus in the future.
2017-04-13 21:55:43 -03:00
David Bremner
ce8c88824a test: remove debugging "output" 2017-04-06 14:37:11 -03:00
David Bremner
ec3937b5cd test: standardize argument order to test_expect_equal_file
It is annoying to debug failing tests when the interpretation of the
diffs is reversed for some tests.
2017-04-06 14:37:02 -03:00
David Bremner
33e5802195 Merge branch 'release'
Final 0.24.1 release
2017-04-01 09:36:34 -03:00
David Bremner
d47e184118 cli/dump: bump dump format version to 3
No changes to the format of the body, but the header format was
fixed, and version 2 headers probably shouldn't be relied on.
2017-04-01 09:10:15 -03:00
David Bremner
1f3c7916f8 cli/dump: fix bug in dump header
Fix copy paste error.
2017-04-01 09:09:33 -03:00
David Bremner
704bd3d8a2 test: add known broken test for dump header
Apparently nobody uses the list of "what was included in the dump"
2017-04-01 09:09:20 -03:00
David Bremner
c0a3bd956a test: add known broken test for dump header
Apparently nobody uses the list of "what was included in the dump"
2017-03-31 21:18:54 -03:00
David Bremner
e1c1d33f37 Merge branch 'release'
Another regexp search fix.
2017-03-29 20:58:34 -03:00
David Bremner
cb84f84878 lib: handle empty string in regexp field processors
The non-field processor behaviour is is convert the corresponding
queries into a search for the unprefixed terms. This yields pretty
surprising results so I decided to generate a query that would match
the terms (i.e. none with that prefix) generated for an empty header.
2017-03-29 20:44:32 -03:00
David Bremner
bc0c051e04 test: add known broken test for null from: and subject: query
These queries currently fail with field processors enabled because the
code expects a non-empty string.
2017-03-29 20:44:09 -03:00
David Bremner
d877240f4e Merge branch 'release'
wildcard search fixes, plus release busywork
2017-03-25 11:51:03 -03:00
Mikhail
9b7dbed58e tests: add compatibility layer
Make test-lib-common.sh load test-lib-<$PLATFORM>.sh to create
additional shim for platform specifics.

Use test-lib-FREEBSD.sh to call GNU utilities instead of native ones.

- amended by db following Tomi's suggestions
2017-03-25 07:56:52 -03:00
David Bremner
38a56b98f9 lib: only trigger phrase processing for regexp fields when needed
The argument is that if the string passed to the field processor has
no spaces, then the added quotes won't have any benefit except for
disabling wildcards. But disabling wildcards doesn't seem very useful
in the normal Xapian query parser, since they're stripped before
generating terms anyway. It does mean that the query 'from:"foo*"' will
not be precisely equivalent to 'from:foo' as it is for the non
field-processor version.
2017-03-24 09:24:13 -03:00
David Bremner
497b83780e test: add known broken tests wildcard search in from and subject
This was broken by the addition of regexp searching. The detection of
wildcards is not currently done in the recursive call to parse_query,
because of quoting issues.
2017-03-24 09:24:13 -03:00
David Bremner
5ce8e0b11b lib: replace deprecated n_q_count_messages with status returning version
This function was deprecated in notmuch 0.21.  We re-use the name for
a status returning version, and deprecate the _st name. One or two
remaining uses of the (removed) non-status returning version fixed at
the same time
2017-03-22 08:35:07 -03:00
David Bremner
86cbd215eb lib: replace deprecated n_q_search_messages with status returning version
This function was deprecated in notmuch 0.21.  We re-use the name for
a status returning version, and deprecate the _st name.
2017-03-22 08:35:07 -03:00
David Bremner
c39f6361d0 rename libutil.a to libnotmuch_util.a
Apparently some systems (MacOS?) have a system library called libutil
and the name conflict causes problems. Since this library is quite
notmuch specific, rename it to something less generic.
2017-03-18 21:37:43 -03:00
Tomi Ollila
9c3a27dee9 test-databases: use wget or curl to download test databases
Often Linux systems are shipped with wget(1) by default (and no curl).

Many BSDs, macOS, and e.g. some Linux minimal/container images
comes with curl(1) (and no wget).

Attempting to download with curl if wget is not available increases
the likelihood for this to succeed.
2017-03-15 22:06:12 -03:00
David Bremner
62822a4e2d lib: clamp return value of g_mime_utils_header_decode_date to >=0
For reasons not completely understood at this time, gmime (as of
2.6.22) is returning a date before 1900 on bad date input. Since this
confuses some other software, we clamp such dates to 0,
i.e. 1970-01-01.
2017-03-15 21:58:25 -03:00
David Bremner
8e7812d824 lib: add known broken test for parsing bad dates.
This reproduces the symptoms of bug report
id:20150422065630.6330.90536@hoothoot
2017-03-15 21:52:34 -03:00
Tomi Ollila
936c35da86 fix out of tree tests
Use $NOTMUCH_SRCDIR/ instead of $TEST_DIRECTORY/../ (in those 2 places)
where reference to source directory instead of build directory is
required.
2017-03-12 09:19:23 -03:00
Tomi Ollila
d6a735423b test: atomicity.py: improve exit probability on failure
Some gdb python exceptions on some os environments (e.g. macOS
Sierra, non-codesigned gdb) do not make gdb exit (but to drop down
to nonexistent command line?).
Mitigate this chance by explict SystemExit on all exceptions.
The contents of output file 'gdb.out' is unchanged.
2017-03-10 08:21:28 -04:00
Gaute Hope
1b5c6b2dea bindings/python: test python Database.get_revision ()
Here is a test for Database.get_revision (), based on
the first part of T570-revision-tracking.sh.
2017-03-09 10:21:02 -04:00
Jani Nikula
0497d695ca test: require test_begin_subtest before test_expect_code
Unify the subtests by requiring test_begin_subtest before
test_expect_code. (Similar change for test_expect_success has already
been done.)

This increases clarity in the test scripts by having a separate line
for the start of the subtest with the heading, and makes it possible
to simplify the test infrastructure by making all subtests similar.
2017-03-09 09:03:40 -04:00
Jani Nikula
d0cd253b37 test: require test_begin_subtest before test_expect_success
Unify the subtests by requiring test_begin_subtest before
test_expect_success. (Similar change for test_expect_code will
follow.)

This increases clarity in the test scripts by having a separate line
for the start of the subtest with the heading, and makes it possible
to simplify the test infrastructure by making all subtests similar.
2017-03-09 09:01:21 -04:00
Jani Nikula
b8f12bd371 test: drop the implicit prereq check mechanism from test_expect_*
The only place where we use the implicit prereq check is T000-basic.sh
where we check that it works. It's an added complication that we don't
use. Remove it.

The test_have_prereq function can still be used for the same effect in
subtests that use test_begin_subtest. For now, this will make it
impossible to have prereqs in one-line subtests that don't require
test_begin_subtest. This will be fixed in follow-up work.
2017-03-09 09:00:16 -04:00
Jani Nikula
e563b8cafd test: ensure test_begin_subtest has been called before test_expect_*
This is the expectation, increase robustness of the test suite by
requiring it.
2017-03-09 08:58:28 -04:00
Jani Nikula
13a59c4a8f test: only accept short and long options, not silly in-betweens
It's not notmuch style to accept sloppy parameter names.
2017-03-09 08:55:38 -04:00
Jani Nikula
de66230b46 test: remove unused and no-op --long-tests parameter
It's been unused since its introduction in commit 0083854b12 ("Copy
test framework from Git").
2017-03-09 08:55:17 -04:00
Jani Nikula
8af6fc5e6e test: remove unused test_external and test_external_without_stderr
They've been unused since their introduction in commit 0083854b12
("Copy test framework from Git"), only causing maintenance burden.
2017-03-09 08:55:06 -04:00
Jani Nikula
cfc0909511 test: remove unused filter functions
They've been unused since their introduction in commit 0083854b12
("Copy test framework from Git").
2017-03-09 08:54:53 -04:00
Jani Nikula
1db3244bf1 test: remove unused regexp convenience variables
They've been unused since their introduction in commit 0083854b12
("Copy test framework from Git").
2017-03-09 08:53:12 -04:00
Jani Nikula
14c0862bac test: suppress diff for broken test without V=1
Known broken tests are, well, known broken. Do not print the result
diff for them unless V=1 is specified. Now that the test description
is printed also when known broken tests fail, the user can also skip
to running the individual failing tests.
2017-03-09 08:48:08 -04:00
Jani Nikula
740f3296b8 test: print test description also for failing known broken tests
With the test description, the user can see the test script name, and
debug with that alone.
2017-03-09 08:47:57 -04:00
Jani Nikula
85df8b3915 test: shrink T590-thread-breakage test decription to one line
The test description is used for log output, I think the intention is
to keep it as a one-liner. Leave the rest of the long description as a
comment.
2017-03-04 21:21:11 -04:00
Mark Walters
a8f91b74e7 Test: emacs: test for folding long headers.
Headers of more than 998 characters should be folded when sending.
However, until recently, emacs did not do this.

This adds a (known broken) test for this when sending messages in
emacs. We will backport the fix to notmuch-emacs in the next
changeset.
2017-03-04 21:15:55 -04:00
David Bremner
dfacfe14f3 lib: query make exclude handling non-destructive
We filter added exclude at add time, rather than modifying the query by
count search. As noted in the comments, there are several ignored
conditions here.
2017-03-04 20:47:25 -04:00
David Bremner
e209b71873 lib: centralize query parsing, store results.
The main goal is to prepare the way for non-destructive (or at least
less destructive) exclude tag handling. It does this by having a
pre-parsed query available for further processing. This also allows us
to provide slightly more precise error messages.
2017-03-04 20:47:25 -04:00
David Bremner
6cb1c617a7 lib: add mid: as a synonym for id:
mid: is the url scheme suggested by URL 2392. We also plan to
introduce more flexible searches for mid: than are possible with
id: (in order not to break assumptions about the special behaviour of
id:, e.g. identifying at most one message).
2017-03-03 17:46:48 -04:00
David Bremner
55524bb063 lib: regexp matching in 'subject' and 'from'
the idea is that you can run

% notmuch search subject:/<your-favourite-regexp>/
% notmuch search from:/<your-favourite-regexp>/

or

% notmuch search subject:"your usual phrase search"
% notmuch search from:"usual phrase search"

This feature is only available with recent Xapian, specifically
support for field processors is needed.

It should work with bindings, since it extends the query parser.

This is easy to extend for other value slots, but currently the only
value slots are date, message_id, from, subject, and last_mod. Date is
already searchable;  message_id is left for a followup commit.

This was originally written by Austin Clements, and ported to Xapian
field processors (from Austin's custom query parser) by yours truly.
2017-03-03 17:46:48 -04:00
David Bremner
508b5c20fa Merge branch 'release'
Second gnugpg test suite fix
2017-02-28 21:21:07 -04:00
Jani Nikula
ea20a932f1 cli/show: add content-disposition to structured output message parts
Help the clients decide how to display parts.

Test updates by Mark Walters <markwalters1009@gmail.com>.
One more test fix by db
2017-02-28 08:03:00 -04:00
David Bremner
1044775fc0 test: move GNUPGHOME to TEST_TMPDIR
We already use this directory for dtach sockets, so it makes sense to
put gnupg sockets there as well. There doesn't seem to be a clean way
to put a fully functional socket in a different location than
GNUPGHOME.
2017-02-27 20:01:42 -04:00
David Bremner
5a42bb96c1 Revert "test: use gpgconf --create-socketdir if available"
This reverts commit e7b88e8b0a.

It turns out that this does not work well in environments without a
running systemd (or some other provider of /run/user)
2017-02-27 18:26:47 -04:00
David Bremner
bd29098e47 test: make T640 compatible with old C compilers
Apparently our test system does not use the same flags for compiling
tests as it does for compiling notmuch. Make the test compatible with
C89. Also remove one unused loop index.
2017-02-26 08:16:45 -04:00
Jani Nikula
75bb23f748 test: test format version difference between v2 and v3
Schemata v3 changed message filename field to a list. Test both
versions 2 and 3.
2017-02-26 07:42:02 -04:00
Jani Nikula
14c60cf168 cli/show: list all filenames of a message in the formatted output
Instead of just having the first filename for the message, list all
duplicate filenames of the message as a list in the formatted
outputs. This bumps the format version to 3.
2017-02-26 07:41:33 -04:00
David Bremner
e0b22c139c lib: handle DatabaseModifiedError in _n_message_ensure_metadata
The retries are hardcoded to a small number, and error handling aborts
than propagating errors from notmuch_database_reopen. These are both
somewhat justified by the assumption that most things that can go
wrong in Xapian::Database::reopen are rare and fatal. Here's the brief
discussion with Xapian upstream:

   24-02-2017 08:12:57 < bremner> any intuition about how likely
      Xapian::Database::reopen is to fail? I'm catching a
      DatabaseModifiedError somewhere where handling any further errors is
      tricky, and wondering about treating a failed reopen as as "the
      impossible happened, stopping"

   24-02-2017 16:22:34 < olly> bremner: there should not be much scope for
    failure - stuff like out of memory or disk errors, which are probably a
    good enough excuse to stop
2017-02-25 21:13:50 -04:00
David Bremner
ae731c183a test: add known broken test for uncaught DatabaseModifiedError
There are several of these to track down, but one that is in quite a
few code paths is _notmuch_message_ensure_metadata.
2017-02-25 21:07:18 -04:00
Tomi Ollila
2c5eed16b3 test: replaced use of python with $NOTMUCH_PYTHON (twice)
$NOTMUCH_PYTHON is sourced from sh.config, configured by
./configure and stated to be used as:

"Name of python command to use in configure and the test suite."
2017-02-23 08:55:24 -04:00
David Bremner
9951598d11 Merge branch 'release'
Merge changes to use gpgconf --create-socketdir
2017-02-21 08:16:07 -04:00
David Bremner
e7b88e8b0a test: use gpgconf --create-socketdir if available
This enables the shortened socket pathes in /run or equivalent. The
explicit call to gpgconf is needed for nonstandard GNUPGHOME settings.

(amended according to id:m2fujatr4k.fsf@guru.guru-group.fi)
2017-02-21 07:45:40 -04:00
Tomi Ollila
f8d5c9b80e test: wrap 'wc -l' results in arithmetic evaluation to strip whitespace
Some new unwrapped 'wc -l's have been added since Jani's 60e79e3a9f
2017-01-27 22:02:36 -04:00
Tomi Ollila
95efe2d484 test: allow user to choose which gdb to run tests with
The variable used for selecting gdb is TEST_GDB, consistent with
TEST_CC and TEST_EMACS{,CLIENT}.
2017-01-08 10:50:28 -04:00
David Bremner
71f1228a51 Merge branch 'release'
Merge changes for notmuch-insert error handling, debian bugfix release
2016-12-07 07:30:12 -04:00
David Bremner
d74c534570 cli/insert: return EX_TEMPFAIL for some errors
Attempt to distinguish between errors indicating misconfiguration or
programmer error, which we consider "permanent", in the sense that
automatic retries are unlikely to be useful, and those indicating
transient error conditions. We consider XAPIAN_EXCEPTION transient
because it covers the important special case of locking failure.
2016-12-07 07:00:40 -04:00
Tomi Ollila
27e293f653 test: gdb insert: redirect input inside gdb script
Running `gdb command < input` is not as reliable way to give input
to the command (some installations of gdb consume it). Use "set args"
gdb command to have input redirected at gdb 'run' time.
2016-12-07 07:00:39 -04:00
Jani Nikula
e682c8f8f8 test: check the handling of files vanishing between scandir and indexing
Add a file for scandir to find, but use gdb to remove it before it
gets indexed.
2016-12-03 07:18:49 -04:00
David Bremner
f781b7e7d2 test: clean up corpora.email
When we renamed corpus to corpora, we missed a spot.
2016-12-03 07:16:12 -04:00
David Bremner
d35c2c15f6 Merge branch 'release'
Initial set of changes for 0.23.3
2016-11-24 21:21:16 -04:00
Daniel Kahn Gillmor
d31161c212 tests: account for varying-size cryptographic signatures
GnuPG 2.1.16 is now injecting the full issuer fingerprint in its
signatures, which makes them about 32 octets larger when
ascii-armored.

This change in size means that the size of the MIME parts will vary
depending on the version of gpg that the user has installed.  at any
rate, the signature part should be non-zero (this is true for
basically any MIME part), so we just test for that instead of an exact
size.
2016-11-24 20:22:12 -04:00
Mark Walters
70c6ac377b Test: fix draft test for emacs23
emacs24 and emacs23 have different secure tag defaults: in particular,
mml-secure-message-sign only signs the part on emacs23 but the whole
message on emacs24. This difference makes one of the draft tests fail
(which causes a cascade of later failures) on emacs23. It seems that
travis uses emacs23 so it is useful to fix this.

We do this by forcing the whole message to be signed in either case --
the code snippet is extracted from mml-secure-message-sign on emacs24.
2016-11-14 18:01:36 -04:00
David Bremner
2dd96d6bd6 emacs: resume messages
Provide functionality to resume editing a message previously saved with
notmuch-draft-save, including decoding the X-Notmuch-Emacs-Secure
header.

Resume gets the raw file from notmuch and using the emacs function
mime-to-mml reconstructs the message (including attachments).

'e' is bound to resume a draft from show or tree mode.
2016-11-13 13:15:31 -04:00
David Bremner
90248f862b emacs: check drafts for encryption tags before saving
In general the user may not want to save plaintext copies of messages
that they are sending encrypted, so give them a chance to abort.
2016-11-13 13:15:31 -04:00
Mark Walters
d820b97477 emacs: postpone a message
This provides initial support for postponing in the emacs frontend;
resuming will follow in a later commit. On saving/postponing it uses
notmuch insert to put the message in the notmuch database

Current bindings are C-x C-s to save a draft, C-c C-p to postpone a
draft (save and exit compose buffer).

Previous drafts get tagged deleted on subsequent saves, or on the
message being sent.

Each draft gets its own message-id, and we use the namespace
draft-.... for draft message ids (so, at least for most people, drafts
are easily distinguisable).
2016-11-13 12:55:24 -04:00
Tomi Ollila
8a82ad716f test/test-lib.sh: execute basename(1)s lazier in test_expect_equal_file ()
Moved the 2 basename(1) executions to the test failure branch in
test_expect_equal_file ().

The output of basename(1) executions in function test_expect_equal_file ()
are only used when tests fails -- when all tests pass these 2 basename(1)
executions are no longer done at all.
2016-10-25 18:13:29 -03:00
David Bremner
d93e9cee13 Merge in test-suite, docstring fixes, release prep 2016-10-22 12:44:39 -03:00
Tomi Ollila
c008dd1352 test: use vt100 as "smart" terminal for known dumb/unknown terminals
Otherwise use whatever user environment has set for TERM so
that there is more chance to test on users' actual environments.
2016-10-22 12:18:43 -03:00
Tomi Ollila
a3ee732c52 test-lib.sh: rename $DTACH_TERM to $SMART_TERM 2016-10-22 12:13:55 -03:00
David Bremner
169e29d431 test: add known broken test for nondestructiveness of count
Thanks to Lucas (id:147263183913.27784.12274024193186585889@mbp) for the
bug report and the test case.

I decided to use the python version because the python bindings could
use more exercise.
2016-10-15 10:22:40 -03:00
Tomi Ollila
6a3ce94aaf test/T000-basic: replaced use of which(1) with shell builtin command -v
Some minimal chroot/container environments don't have which(1) installed.
2016-10-11 22:43:17 -03:00
Tomi Ollila
b57426a435 test: test-lib.sh: use $BASH instead of ${SHELL-sh} to relaunch
In case of the test script is to be relaunced under valgrind, or --tee
is requested, use the $BASH shell variable to locate the command
interpreter. The $SHELL variable is re-set by non-interactive shells
so in case the shell uses some other shell (e.g. zsh) for interactive
use these bash scripts continue to work.
2016-09-24 10:10:40 -03:00
David Bremner
5292e9b1ae test/emacs: add broken test for message replying to msg to self
This is a strange corner case where the removing of the user's address
from the To: header does the wrong thing. If we think it is
worth (eventually) fixing, this test can serve as a reminder.
2016-09-24 10:05:04 -03:00
David Bremner
1d4812fd61 test: fix printf format
notmuch_status_t is an integer type, printing it as a string is a very
bad idea.
2016-09-22 08:11:42 -03:00
Daniel Kahn Gillmor
693ca8d8a8 add property: query prefix to search for specific properties
We want to be able to query the properties directly, like:

   notmuch count property:foo=bar

which should return a count of messages where the property with key
"foo" has value equal to "bar".
2016-09-21 18:14:25 -03:00
David Bremner
651da30c09 cli: optionally restore message properties from dump file
This somewhat mimics the config line parsing, except there can be
arbitrarily many key value pairs, so one more level of looping is
required.
2016-09-21 18:14:25 -03:00
David Bremner
b7345d277e CLI: add properties to dump output
Part of providing extensibility via properties is to make sure that user
data is not lost. Thus we need to be able to dump and restore
properties.
2016-09-21 18:14:25 -03:00
David Bremner
58fe8fce1d lib: iterator API for message properties
This is a thin wrapper around the string map iterator API just introduced.
2016-09-21 18:14:25 -03:00
David Bremner
b8bb6d7964 lib: basic message-property API
Initially, support get, set and removal of single key/value pair, as
well as removing all properties.
2016-09-21 18:14:24 -03:00
Jani Nikula
a63e674b34 test: silence the output of notmuch new mid-test
Fix this during test run:

T470-missing-headers: Testing messages with missing headers
Added 2 new messages to the database.
2016-09-19 20:42:09 -03:00
Mark Walters
a448879e49 emacs: reply: remove wrong sig/enc status buttons
This stops the (usually incorrect) sigstatus and encstatus buttons
appearing when replying in emacs, and updates the test suite to match.

Overriding the status button functions is a little unusual but much
less intrusive than passing an argument all the way down the call
chain. It also makes it clear exactly what it does.

We also hide the application/pgp-encrypted part as it can only contain
"Version: 1". We do this in notmuch show, which means it also happens
when replying.
2016-09-18 10:57:27 -03:00
David Bremner
653c5ee221 test/crypto: test reply to encrypted message in emacs
This test considers the extra output about encryption/signature status
as a bug, to be fixed in the next commit
2016-09-18 10:56:54 -03:00
Jani Nikula
536b1f9df9 cli/reply: return internet address list from get header funcs
Pass in GMimeMessage to simplify To/Cc/Bcc headers. We'll eventually
remove the notmuch message passing altogether, but keep both for now
to not make too big changes at once.

Getting the headers from GMimeMessage using GMime functions fixes the
error on duplicate Cc headers reported by Daniel Kahn Gillmor
<dkg@fifthhorseman.net> in id:87d1ngv95p.fsf@alice.fifthhorseman.net.

Get rid of an intermediate function.

The small annoyance is the ownership differences in the address lists.
2016-09-17 09:11:07 -03:00
Jani Nikula
36416c74e0 test: add known broken test for reply to message with multiple Cc headers
As Daniel Kahn Gillmor <dkg@fifthhorseman.net> reports in
id:87d1ngv95p.fsf@alice.fifthhorseman.net, notmuch show combines
multiple Cc: fields into one, while notmuch reply does not. While such
messages are in violation of RFC 5322, it would be reasonable to
expect notmuch to be consistent. Add a known broken test to document
this expectation.

This also starts a new "broken" corpus for messages which are broken.

Details:

The original message is formatted using the message printing in
notmuch-show.c. For Cc:, it uses g_mime_message_get_recipients(),
which apparently combines all Cc: fields into one internally.

The addresses in the reply headers, OTOH, are based on headers queried
through libnotmuch. It boils down to g_mime_object_get_header() in
lib/message-file.c, which returns only the first occurence of header.
2016-09-17 08:41:29 -03:00
Jani Nikula
971cdc72cd test: make it possible to have multiple corpora
We largely use the corpus under test/corpus for
testing. Unfortunately, many of our tests have grown to depend on
having exactly this set of messages, making it hard to add new message
files for testing specific cases.

We do use a lot of add_message from within the tests, but it's not
possible to use that for adding broken messages, and adding several
messages at once can get unwieldy.

Move the basic corpus under tests/corpora/default, and make it
possible to add new, independent corpora along its side. This means
tons of renames with a few tweaks to add_email_corpus function in
test-lib.sh to let tests specify which corpus to use.
2016-09-17 08:39:34 -03:00
David Bremner
07dff49630 Merge branch 'release'
hand fixed conflicts with NEWS
2016-09-08 22:18:37 -03:00
David Bremner
7edded2740 test/crypto: hard code fingerprint
Originally the intent was to make the test more robust against changing
test keys. It turns out that (unscientifically) gpg --with-colons output
changes more often than our test key. Rather than making the script more
complex, just hard code the fingerprint.

This fixes Debian bug #847013; I expect similar test failures as other
distros adopt gnupg 2.1.15
2016-09-08 09:08:09 -03:00
Mark Walters
fce8146a8b emacs: maildir: add the actual insert code
With all the preparation it is now simple to add the actual insert
code. Since insert can fail for many reasons we let the user decide
interactively deal with it.

We modify test-lib.el to set file fcc, so that all the old tests and
emacs_fcc_message from test-lib.sh still work
2016-09-04 08:23:14 -03:00
David Bremner
cf8aabdd37 test: make gdb even quieter
gdb sometimes writes warnings to stdout, which we don't need/want, and
for some reason --batch-silent isn't enough to hide. So in this commit
we write them to a log file, which is probably better for debugging
anyway. To see an illustrative test failure before this change, run

% make
% touch notmuch-count.c
% cd test && ./T060-count.sh

(cherry picked from commit f45fa5bdd3)
2016-08-14 13:27:57 +09:00
Daniel Kahn Gillmor
1ba73d1437 Omit User-Agent: header by default
The User-Agent: header can be fun and interesting, but it also leaks
quite a bit of information about the user and their software stack.

This represents a potential security risk (attackers can target the
particular stack) and also an anonymity risk (a user trying to
preserve their anonymity by sending mail from a non-associated account
might reveal quite a lot of information if their choice of mail user
agent is exposed).

This change also avoids hiding the User-Agent header by default, so
that people who decide they want to send it will at least see it (and
can edit it if they want to) before sending.

It makes sense to have safer defaults.
2016-08-11 20:16:00 +09:00
Mark Walters
999d473299 emacs: wash: word-wrap bugfix
Previously notmuch-wash made the width of the text (approximately) the
window-width minus the depth in thread. This is correct for the
default indentation of 1 per message depth, but is incorrect for any
other setting of notmuch-show-indent-messages-width.

As notmuch-show-indent-messages-width is customisable, and notmuch-tree
sets it to zero to avoid indenting messages in the message pane, this
bug can show up in real use.

Two of the tests had to be updated: when
notmuch-show-indent-messages-width is 0, then the new (correct) word
wrapping happens later, when notmuch-show-indent-messages-width is 4,
then the new word wrapping happens sooner.
2016-08-09 09:34:25 +09:00
David Bremner
f45fa5bdd3 test: make gdb even quieter
gdb sometimes writes warnings to stdout, which we don't need/want, and
for some reason --batch-silent isn't enough to hide. So in this commit
we write them to a log file, which is probably better for debugging
anyway. To see an illustrative test failure before this change, run

% make
% touch notmuch-count.c
% cd test && ./T060-count.sh
2016-08-09 09:33:59 +09:00
David Bremner
48d33532bb test: don't use dump and restore in a pipeline
This has been wrong since bbbdf0478e, but the race condition was not
previously been (often?) triggered in the tests. With the DB_RETRY_LOCK
patches, it manifests itself as a deadlock.
2016-06-30 17:47:36 +02:00
David Bremner
3a45d29ed4 lib: add built_with handling for XAPIAN_DB_RETRY_LOCK
This support will be present only if the appropriate version of xapian
is available _and_ the user did not disable the feature when
building. So there really needs to be some way for the user to check.
2016-06-29 09:05:49 +02:00
David Bremner
6450813e8f test: initial tests for locking retry
Currently there's not much to test, so we simulate contention, and check
that the modifications to the database are serialized.
2016-06-29 09:05:28 +02:00
David Bremner
496dccf44f test: don't use dump and restore in a pipeline
This has been wrong since bbbdf0478e, but the race condition was not
previously been (often?) triggered in the tests. With the DB_RETRY_LOCK
patches, it manifests itself as a deadlock.
2016-06-29 09:03:14 +02:00
Tomi Ollila
be55acf4f0 test/notmuch-test: simple streamlining
A few simple things that applies well to test/notmuch-test.sh

- Shell does pathname expansion also without doing `echo ...` in subshell.

- Redirections >/dev/null 2>/dev/null hide (improbable but) potential
  serious errors; adding -f to rm instead.

- Inter-file capitalization consistency in comments.

- Unnecesary space removal.
2016-06-28 09:23:39 +02:00
David Bremner
cc1a6d2a94 test: factor out some boilerplate from C tests
The trick of having a common header file doesn't work to share between
test scripts, so make an include file in the test directory.

The use of #include <notmuch-test.h> looks slightly pretentious, but
the include file is not actually in the current (temporary) directory.
2016-06-13 21:52:33 -03:00
Tomi Ollila
2333a44ab7 test: test_python: set PYTHONPATH to the python execution environment
Place PYTHONPATH to the environment when python is executed in a way
that current shell environment is not affected. This also allows adding
the old value of PYTHONPATH to the end of the new value (otherwise it
would have been appended again and again when test_python is called).

At the same time, use -B option to avoid writing .pyc files to
bindings/python/* (which are not cleared out by distclean).

Drop the (unused) prefix code which preserved the original stdout of the
python program and opened sys.stdout to OUTPUT. In place of that there
is now note how (debug) information can be printed to original stdout.
2016-06-11 13:12:08 -03:00
Tomi Ollila
f0e5317bcd test: set LD_LIBRARY_PATH early and keep its old contents
Previously LD_LIBRARY_PATH was exported (and environment changed)
in the middle of test case execution, when a function setting it
was called.

Previously the old contents of LD_LIBRARY_PATH was lost (if any)
when it was re-set and exported. In some systems the old contents of
LD_LIBRARY_PATH was needed to e.g. locate suitable gmime library.
2016-06-11 13:11:49 -03:00
Tomi Ollila
36492f20a7 test: fix die() in test-lib-common.sh
In scripts that include test-lib-common.sh but not test-lib.sh
the die() implementation needs to be a bit different due to
fd redirection differences. test-lib-common.sh implements die()
only if it was not implemented already.
2016-06-11 13:03:44 -03:00
Tomi Ollila
d53b417d1f test: add function die () and have use of it in add_email_corpus ()
Added die() function to test-lib.sh with the following first use of it:

If notmuch new fails during email corpus addition the database is
most probably inexistent or broken and the added corpus would be
unusable while running single tests, giving misleading failures
("only" full 'make test' cleans out old corpus).
2016-06-11 13:03:09 -03:00
Tomi Ollila
1ca752fa36 test-lib.sh: renamed die...()s to trap...()s and exit...()
Now the function names more accurately describes what the functions do.
2016-06-11 13:03:00 -03:00
Daniel Kahn Gillmor
6a833a6e83 Use https instead of http where possible
Many of the external links found in the notmuch source can be resolved
using https instead of http.  This changeset addresses as many as i
could find, without touching the e-mail corpus or expected outputs
found in tests.
2016-06-05 08:32:17 -03:00
David Bremner
b9bf3f44ea lib: add support for named queries
This relies on the optional presense of xapian field processors, and the
library config API.
2016-05-25 07:40:44 -03:00
David Bremner
2d2a13966c CLI: add notmuch-config support for named queries
Most of the infrastructure here is general, only the validation/dispatch
is hardcoded to a particular prefix.

A notable change in behaviour is that notmuch-config now opens the
database e.g. on every call to list, which fails with an error message
if the database doesn't exit yet.
2016-05-25 07:40:44 -03:00
David Bremner
c6fcc555dd CLI: optionally restore config data.
The default to restore config data seems safest, especially since
currently we have no config data to mess up.
2016-05-25 07:40:44 -03:00
David Bremner
e042a25a3f CLI: add optional config data to dump output.
Note that it changes the default dump output format, but doesn't break
existing notmuch-restore. It might break user scripts though.
2016-05-25 07:07:56 -03:00
David Bremner
92e59568fa lib: config list iterators
Since xapian provides the ability to restrict the iterator to a given
prefix, we expose this ability to the user. Otherwise we mimic the other
iterator interfances in notmuch (e.g. tags.c).
2016-05-25 06:51:16 -03:00
David Bremner
3458e3c89c lib: provide config API
This is a thin wrapper around the Xapian metadata API. The job of this
layer is to keep the config key value pairs from colliding with other
metadata by transparently prefixing the keys, along with the usual glue
to provide a C interface.

The split of _get_config into two functions is to allow returning of the
return value with different memory ownership semantics.
2016-05-24 08:53:03 -03:00
Ludovic LANGE
7e6e23c36e ruby: add bindings for notmuch_database_get_all_tags
The Ruby bindings were missing a way to get all the tags of the
database. Now you should be able to access this with the public
instance method `all_tags` of your database object.

Example of use:
    notmuchdb = Notmuch::Database.new path, { :create => false,
    	:mode => Notmuch::MODE_READ_ONLY }

    my_tags = notmuchdb.all_tags

    my_tags.each { |tag|
      print tag
    }

    my_tags.destroy!

Amended by db: improve error reporting, add test
2016-05-19 08:02:43 -03:00
Tomi Ollila
b183f2635e test: copyright information updates
Files in test directories had only copyright of a single individual,
of which code was adapted here as a base of the test system.
Since then many Notmuch Developers have contributed to the test
system, which is now acknowledged with a constant string in some
of the test files.

The README file in test directory instructed new files contain a
copyright notice, but that has never been done (and it is also not
needed). To simplify things a bit (and lessen confusion) this
instruction is now removed.

As a side enchangement, all of the 3 entries in the whole source
tree cd'ing to `dirname` of "$0" now uses syntax cd "$(dirname "$0")".
This makes these particular lines work when current working directory
is e.g. /c/Program Files/notmuch/test/.
(Probably it would fail elsewhere, though.)
2016-05-19 07:56:58 -03:00
Tomi Ollila
eac2976389 cli: tell how many messages were precisely matched when expected 1 match
In case of notmuch reply and notmuch show --part=N it is required that
search terms match to one message. If match count was != 1, error
message "Error: search term did not match precisely one message"
was too vague to explain what happened.

By appending (matched <num> messages) to the error message it
makes the problem more understandable (e.g when <num> is '0'
user reckons the query had a typo in it).
2016-05-19 07:49:03 -03:00
David Bremner
792bea5aff lib/cli: add library API / CLI for compile time options
This is intentionally low tech; if we have more than two options it may
make sense to build up what infrastructure is provided.
2016-05-13 07:29:12 -03:00
David Bremner
bbf6069252 lib: optionally support single argument date: queries
This relies on the FieldProcessor API, which is only present in xapian
>= 1.3.
2016-05-08 08:17:07 -03:00
Daniel Kahn Gillmor
e366bb2227 complete ghost-on-removal-when-shared-thread-exists
To fully complete the ghost-on-removal-when-shared-thread-exists
proposal, we need to clear all ghost messages when the last active
message is removed from a thread.

Amended by db: Remove the last test of T530, as it no longer makes sense
if we are garbage collecting ghost messages.
2016-04-15 07:13:49 -03:00
Daniel Kahn Gillmor
604d1e0977 fix thread breakage via ghost-on-removal
implement ghost-on-removal, the solution to T590-thread-breakage.sh
that just adds a ghost message after removing each message.

It leaks information about whether we've ever seen a given message id,
but it's a fairly simple implementation.

Note that _resolve_message_id_to_thread_id already introduces new
message_ids to the database, so i think just searching for a given
message ID may introduce the same metadata leakage.
2016-04-15 07:07:23 -03:00
Daniel Kahn Gillmor
92559ee347 test thread breakage when messages are removed and re-added
This test (T590-thread-breakage.sh) has known-broken subtests.

If you have a two-message thread where message "B" is in-reply-to "A",
notmuch rightly sees this as a single thread.

But if you:

 * remove "A" from the message store
 * run "notmuch new"
 * add "A" back into the message store
 * re-run "notmuch new"

Then notmuch sees the messages as distinct threads.

This happens because if you insert "B" initially (before anything is
known about "A"), then a "ghost message" gets added to the database in
reference to "A" that is in the same thread, which "A" takes over when
it appears.

But if "A" is subsequently removed, no ghost message is retained, so
when "A" appears, it is treated as a new thread.

I see a few options to fix this:

ghost-on-removal
----------------

We could unilaterally add a ghost upon message removal.  This has a
few disadvantages: the message index would leak information about what
messages the user has ever been exposed to, and we also create a
perpetually-growing dataset -- the ghosts can never be removed.

ghost-on-removal-when-shared-thread-exists
------------------------------------------

We could add a ghost upon message removal iff there are other
non-ghost messages with the same thread ID.

We'd also need to remove all ghost messages that share a thread when
the last non-ghost message in that thread is removed.

This still has a bit of information leakage, though: the message index
would reveal that i've seen a newer message in a thread, even if i had
deleted it from my message store

track-dependencies
------------------

rather than a simple "ghost-message" we could store all the (A,B)
message-reference pairs internally, showing which messages A reference
which other messages B.

Then removal of message X would require deleting all message-reference
pairs (X,B), and only deleting a ghost message if no (A,X) reference
pair exists.

This requires modifying the database by adding a new and fairly weird
table that would need to be indexed by both columns.  I don't know
whether xapian has nice ways to do that.

scan-dependencies
-----------------

Without modifying the database, we could do something less efficient.

Upon removal of message X, we could scan the headers of all non-ghost
messages that share a thread with X.  If any of those messages refers
to X, we would add a ghost message.  If none of them do, then we would
just drop X entirely from the table.

---------------------

One risk of attempted fixes to this problem is that we could fail to
remove the search term indexes entirely.  This test contains
additional subtests to guard against that.

This test also ensures that the right number of ghost messages exist
in each situation; this will help us ensure we don't accumulate ghosts
indefinitely or leak too much information about what messages we've
seen or not seen, while still making it easy to reassemble threads
when messages come in out-of-order.
2016-04-15 07:07:23 -03:00
David Bremner
f68e776617 test: add test-binary to print the number of ghost messages
This one-liner seems preferable to the complications of depending on
delve, getting the binary name right and parsing the output.
2016-04-15 07:07:23 -03:00
Jani Nikula
a352d9ceaa lib: fix handling of one character long directory names at top level
The code to skip multiple slashes in _notmuch_database_split_path()
skips back one character too much. This is compensated by a +1 in the
length parameter to the strndup() call. Mostly this works fine, but if
the path is to a file under a top level directory with one character
long name, the directory part is mistaken to be part of the file name
(slash == path in code). The returned directory name will be the empty
string and the basename will be the full path, breaking the indexing
logic in notmuch new.

Fix the multiple slash skipping to keep the slash variable pointing at
the last slash, and adjust strndup() accordingly.

The bug was introduced in

commit e890b0cf40
Author: Carl Worth <cworth@cworth.org>
Date:   Sat Dec 19 13:20:26 2009 -0800

    database: Store the parent ID for each directory document.

just a little over two months after the initial commit in the Notmuch
code history, making this the longest living bug in Notmuch to date.
2016-04-12 20:40:19 -03:00
Jani Nikula
0f6b399d5b test: test one character long directory names at top level
Yes, it's broken. Reported by h01ger on IRC.
2016-04-12 20:37:08 -03:00
David Bremner
e311aad182 test: cope with glass backend file naming variations
In several places in the test suite we intentionally corrupt the Xapian
database in order to test error handling. This corruption is specific to
the on-disk organization of the database, and that changed with the
glass backend. We use the previously computed default backend to make
the tests adapt to changing names.
2016-04-12 20:21:09 -03:00
David Bremner
deb4e5567c configure: add test for default xapian backend
This is mainly for the test suite.  We already expect the tests to be
run in the same environment as configure was run, at least to get the
name of the python interpreter. So we are not really imposing a new
restriction.
2016-04-12 20:14:43 -03:00
David Bremner
45a0ab9e84 test/atomicity: guard chert-only optimization
This should potentially be updated to have an equivalent optimization
for the glass backend, but it in my unscientific tests, the glass backend
without the optimization is faster then the chert backend with.
2016-04-12 20:11:53 -03:00
David Bremner
45df509cb6 test: improve error handling in lib-error tests
There is at least one bug fixed here (missing parameter to printf), even
if exiting via segfault is considered OK.
2016-04-12 20:04:03 -03:00
David Edmondson
f8effea9a3 emacs: Always insert crypto buttons.
When no decryption or signature examination is
happening (i.e. `notmuch-crypto-process-mime' is `nil') insert buttons
that indicate this, rather than remaining silent.
2016-04-10 20:46:58 -03:00
Tomi Ollila
cc369779ab test/test-lib.el: revert setting notmuch-mua-reply-insert-header-p-function
in d27d90875d (2016-02-20) notmuch-mua-reply-insert-header-p-function
was set to notmuch-show-reply-insert-header-p-never as its default was
changed to something else. Now that default is set back to *-never so
this change done in d27d90875d is not needed anymore.
2016-03-26 09:03:54 -03:00
David Bremner
3dcd6e9a23 test: canonicalize quotes in xapian error messages
These differ between 1.2 and 1.3, so we transform the 1.2 backquote to a
single quote for compatibility
2016-03-14 19:54:32 -03:00
David Bremner
bfccfc31c7 test: replace single giant term with large phrase
We only need a long string, not a single long term to trigger batch
mode.  The giant term triggers a bug/incompatibility in Xapian 1.3.4
that throws an exception because it is longer than the Xapian term size
limit.
2016-03-14 19:54:32 -03:00
Jani Nikula
b8a136187a emacs: hello: promote '?' as the universal help key
Move the brief help text at the bottom of the hello screen to the
notmuch-hello-mode help, and promote '?' as the universal help key
across Notmuch. This unclutters the hello screen, and allows for a
more verbose description in the mode help. Hopefully, this change is
useful for both experienced and new users alike.

While at it, improve the links to Notmuch and hello screen
customization.
2016-03-14 19:54:32 -03:00
Tomi Ollila
f8cc320694 test/README: have matching test script file names
A while ago test script names were changed to format
Tddd-basename.sh. Update README to reflect that.

While at it, included some small requirements updates.
2016-03-04 20:45:19 -04:00
David Edmondson
d27d90875d emacs/mua: Let user specify which parts get a header in citations.
Add a customizable function specifying which parts get a header when
replying, and give some sensible possiblities. These are,

1) all parts except multipart/*. (Subparts of a multipart part do
receive a header button.)

2) only included text/* parts.

3) Exactly as in the show buffer.

4) None at all. This means the reply contains a mish-mash of all the
original message's parts.

In the test suite we set the choice to option 4 to match the
previous behaviour.
2016-02-21 08:37:48 -04:00
David Edmondson
e103f0a971 emacs/mua: Generate improved cited text for replies
Use the message display code to generate message text to cite in
replies.

For now we set insert-headers-p function to
notmuch-show-reply-insert-header-p-never so that, as before, we don't
insert part buttons.

With that choice of insert-headers-p function there is only one
failing test: this test has a text part (an email message) listed as
application/octet-stream. Notmuch show displays this part, but the
reply code omitted it as it had type application/octet-stream. The new
code correctly includes it. Thus update the expected output to match.
2016-02-21 08:37:26 -04:00
David Bremner
e2b7bd9bac test: fix typo in informational message
Thanks to Olly Betts for noticing
2016-02-13 12:32:05 -04:00
Jani Nikula
2355f1522a cli: crypto: S/MIME verification support
notmuch-show --verify will now also process S/MIME multiparts if
encountered. Requires gmime-2.6 and gpgsm.

Based on work by Jameson Graef Rollins <jrollins@finestructure.net>.
2016-01-29 17:45:46 -04:00
David Bremner
a530a1706a test: add broken S/MIME signature verification test for notmuch CLI
The test is pretty much cut and paste from the PGP/MIME version, with
obvious updates taken from notmuch output.  This also requires setting
up gpgsm infrastucture.
2016-01-29 17:45:46 -04:00
David Bremner
ef30770dfa test: initial tests for S/MIME and notmuch-emacs
Test the ability of notmuch-mua-mail to send S/MIME signed (and
encrypted) messages; this really relies on existing functionality in
message-mode.

The generated keys and messages will later be useful for testing the
notmuch CLI.
2016-01-29 17:45:46 -04:00
David Bremner
12534d5c4b test: refactor directory name sanitization
test_C and notmuch_search_files_sanitize were giving different output on
the same path, which is not technically wrong, but slightly
confusing.
2016-01-26 22:50:58 -04:00
Jani Nikula
b645f41605 cli: fix function name in notmuch new debug logging
add_files_recursive has been renamed add_files long ago.
2015-12-30 11:28:30 -04:00
Michal Sojka
a9fec12a08 test: Unset ALTERNATE_EDITOR before running emacsclient
ALTERNATE_EDITOR causes emacsclient to run an alternate editor if the
emacs server is not ready. This can collide with intended
functionality in test-lib.sh.

If the ALTERNATE_EDITOR is set but empty, emacsclient runs emacs
daemon and tries to connect to it. When this happens the emacs run by
test-lib.sh fails to start the server and the subsequent attempts to
use the server fail because the daemon started by emacsclient does not
know about notmuch-test-progn. This leads to test suite failure due to
time out on any emacs test.
2015-12-30 11:28:30 -04:00
Michal Sojka
c2dfaccc90 test: Always use paths without symlinks
When notmuch sources are at a symlinked path, some tests fail because
one part of the test uses physical path and another uses logical
path (with symlinks). For example the following test output is
produced when the test is started from /home/src/symlink-to-notmuch,
which is a symlink to /home/src/notmuch.

    FAIL   notmuch-fcc-dirs set to a string
        --- T310-emacs.26.OUTPUT        2015-12-29 08:54:29.055878637 +0000
        +++ T310-emacs.26.EXPECTED      2015-12-29 08:54:29.055878637 +0000
        @@ -1,5 +1,5 @@
         From: Notmuch Test Suite <test_suite@notmuchmail.org>
         To:
         Subject:
        -Fcc: /home/src/notmuch/test/tmp.T310-emacs/mail/sent-string
        +Fcc: /home/src/symlink-to-notmuch/test/tmp.T310-emacs/mail/sent-string
         --text follows this line--
    nil

This commit makes all paths in test scripts physical. With it, all
tests pass even when run from a symlinked directory.
2015-12-30 11:28:30 -04:00
David Bremner
d98c0854b8 test: add sanity tests for threading
These tests are inspired by a problem report

      id:CAJhTkNh7_hXDLsAGyD7nwkXV4ca6ymkLtFG945USvfqK4ZJEdQ@mail.gmail.com

Of course I can't duplicate the mentioned problem, it probably depends
on specific message data.
2015-11-23 08:08:26 -04:00
Daniel Schoepe
40325352db test: Tests for combining --batch and --remove-all
This patch adds some tests for combining --remove-all with --batch or
--input when calling notmuch-tag.
2015-11-19 07:51:14 -04:00
Jani Nikula
506b81679a lib: content disposition values are not case-sensitive
Per RFC 2183, the values for Content-Disposition values are not
case-sensitive. While at it, use the gmime function for getting at the
disposition string instead of referencing the field directly.

This fixes "attachment" tagging and filename term generation for
attachments while indexing.
2015-11-19 07:47:29 -04:00
David Bremner
c1c955786a tests: add test for case insensitive Content-Disposition
This is broken now, but will be fixed in the next commit
2015-11-19 07:46:03 -04:00
Jani Nikula
e26d99dc7b cli: delete directory documents on directory removal
There was a problem with the directory documents being left behind when
the filesystem directory was removed. This was worked around in [1].
However, that ignored the fact that the directory documents are also
still listed by notmuch_directory_get_child_directories() leading to
confusing results when running notmuch new. The directory documents are
found and queued for removal over and over again.

Fix the problem for real by removing the directory documents. This fixes
the tests flagged as broken in [2].

The (non-deterministic) hack test from [3] also still passes with this
change.

[1] commit acd66cdec0
[2] commit ed9ceda623
[3] id:1441445731-4362-1-git-send-email-jani@nikula.org
2015-10-14 07:55:41 -03:00
Jani Nikula
0e03ff3894 test: flag one more notmuch new test as broken
Drop the test update added in [1] and mark the test as broken, like the
tests flagged as broken in [2]. These all reflect the same underlying
breakage with (lack of) directory deletion.

[1] commit e4e04bbc32
[2] commit ed9ceda623
2015-10-10 09:13:02 -03:00
Jani Nikula
f998724182 test: add notmuch address --deduplicate=(no|mailbox|address) tests
First a simple smoke test first, next generate messages with multiple
email address variants and check the behaviour of deduplication
schemes with these.
2015-09-26 07:40:23 -03:00
Jani Nikula
23b8ed610a lib: add support for date:<expr>..! to mean date:<expr>..<expr>
It doesn't seem likely we can support simple date:<expr> expanding to
date:<expr>..<expr> any time soon. (This can be done with a future
version of Xapian, or with a custom query query parser.) In the mean
time, provide shorthand date:<expr>..! to mean the same. This is
useful, as the expansion takes place before interpetation, and we can
use, for example, date:yesterday..! to match from beginning of
yesterday to end of yesterday.

Idea from Mark Walters <markwalters1009@gmail.com>.
2015-09-25 21:55:24 -03:00
David Bremner
3cf7ed26c0 test suite: don't consider skipped individual tests as failing
It isn't completely clear what we want to do here, but

1) We currently don't fail if we skip a whole test file (mainly because
we neglect to count those skipped tests properly). This change at least
makes the two kinds of skipping consistent.

2) Automated build environments may have good reasons for building with
a minimal set of prereqs, and we don't want to discourage running our
test suite by breaking builds.
2015-09-25 09:18:22 -03:00
Tomi Ollila
3c46a9e6df make test: NOTMUCH_TEST_QUIET=1 is now the default
make test V=1 (or any other value than 0) and make test V=0
works similar way as build in general
2015-09-25 08:00:18 -03:00
Jani Nikula
adba9b0d50 test: notmuch address --deduplicate=no tests
Test not using address deduplication. Incorporate some trivial output
sorting tests here, as they seem to lack tests.
2015-09-24 07:34:25 -03:00
Jani Nikula
19ea288679 test: check argc in symbol-test
Check argc mainly to fix unused parameter warning:

test/symbol-test.cc:7:14: warning: unused parameter ‘argc’ [-Wunused-parameter]
  int main(int argc, char** argv) {
               ^

This makes more sense than telling the compiler it's unused on
purpose.
2015-09-24 07:28:51 -03:00
David Bremner
dca72205ab cli/count: update to use notmuch_query_search_messages_st
This brings back status information that may have been hidden by the
great library logging conversion.

Note the change of the internal API / return-value for count_files. The
other count calls to the lib will also get error handling when that API
is updated in the lib.
2015-09-20 08:04:51 -03:00
Jani Nikula
1008fc45da test: fix whitespace/indentation in symbol-test
Not of much consequence, but makes it nicer to do further edits.
2015-09-07 09:28:18 -03:00
David Bremner
9ee29ad6f0 test: move backup_database and restore_database to library
The plan is to re-use them in testing the same errors from the CLI
2015-09-04 08:22:42 -03:00
Austin Clements
cb08a2ee01 lib: Add "lastmod:" queries for filtering by last modification
The implementation is essentially the same as the date range search
prior to Jani's fancy date parser.
2015-08-14 18:23:49 +02:00
David Bremner
f76d8f82dd cli: add global option "--uuid"
The function notmuch_exit_if_unmatched_db_uuid is split from
notmuch_process_shared_options because it needs an open notmuch
database.

There are two exceptional cases in uuid handling.

1) notmuch config and notmuch setup don't currently open the database,
   so it doesn't make sense to check the UUID.

2) notmuch compact opens the database inside the library, so we either
   need to open the database just to check uuid, or change the API.
2015-08-14 18:23:49 +02:00
David Bremner
5a3b42fb8c cli/count: add --lastmod
In the short term we need a way to get lastmod information e.g. for
the test suite. In the long term we probably want to add lastmod
information to at least the structured output for several other
clients (e.g. show, search).
2015-08-14 18:07:55 +02:00
Austin Clements
98ee460eaa lib: API to retrieve database revision and UUID
This exposes the committed database revision to library users along
with a UUID that can be used to detect when revision numbers are no
longer comparable (e.g., because the database has been replaced).
2015-08-13 23:52:51 +02:00
Tomi Ollila
02a2eeb427 test: make script exit (1) if it "fails" to source (.) a file
The files (test) scripts source (with builtin command `.`) provides
information which the scripts depend, and without the `source` to
succeed allowing script to continue may lead to dangerous situations
(e.g. rm -rf "${undefined_variable}"/*).

At the end of all source (.) lines construct ' || exit 1' was added;
In our case the script script will exit if it cannot find (or read) the
file to be sourced. Additionally script would also exits if the last
command of the sourced file exited nonzero.
2015-08-07 21:56:39 +02:00
David Bremner
d0553ad524 emacs: make modifications to message Fcc vars buffer-local
Previously we globally modified these variables, which tended to cause
problems for people using message-mode, but not notmuch-mua-mail, to
send mail.

User visible changes:

- Calling notmuch-fcc-header-setup is no longer optional. OTOH, it
  seems to do the right thing if notmuch-fcc-dirs is set to nil.

- The Fcc header is visible during message composition

- The name in the mode line is changed, and no longer matches exactly
  the menu label.

- Previously notmuch-mua-send-and-exit was never called.  Either we
  misunderstood define-mail-user-agent, or it had a bug.  So there was
  no difference if the user called message-send-and-exit directly. Now
  there will be.

- User bindings to C-c C-c and C-c C-s in message-mode-map are
  overridden. The user can override them in notmuch-message-mode-map,
  but then they're on their own for Fcc handling.
2015-08-07 21:18:59 +02:00
David Bremner
ed9ceda623 test: add debugging output to notmuch-new tests, mark 5 as broken
I marked the tests where I really couldn't understand the output as
broken. It could also be that I don't understand how directory removal
is supposed to work.
2015-08-04 21:00:09 +02:00
David Bremner
e4e04bbc32 cli/new: add more debugging output
Try to narrow down what part of the code adds files and directories to
the queue(s) to be deleted.

Update one test. The output is slightly confusing, but I believe it is
correct, resulting from a directory being discovered but containing only ignored files.
2015-08-04 20:59:58 +02:00
David Bremner
e2fb06d369 test: don't claim single message mbox support is going away
We gave up on this.
2015-08-04 20:59:50 +02:00
David Bremner
23d86773b9 test: use the python interpreter in sh.config
The configure script chooses "python" if both python and python{2,3}
exist exists, so this could change the version of python used to run
the test suite.

The checking for ${NOTMUCH_PYTHON} in the test suite is arguably
over-engineering, since the configure step will fail if it can't find
it.
2015-08-04 09:19:18 +02:00
David Bremner
d9567dd5b5 test: add initial ruby tests
This is pretty much a line by line translation of the existing python
tests, with two new tests for the count API.
2015-06-14 18:00:52 +02:00
David Bremner
32fd74b7aa lib: reject relative paths in n_d_{create,open}_verbose
There are many places in the notmuch code where the path is assumed to be absolute. If someone (TM) wants a project, one could remove these assumptions. In the mean time, prevent users from shooting themselves in the foot.

Update test suite mark tests for this error as no longer broken, and
also convert some tests that used relative paths for nonexistent
directories.
2015-06-12 07:34:50 +02:00
David Bremner
b59ad1a9cc lib: add NOTMUCH_STATUS_PATH_ERROR
The difference with FILE_ERROR is that this is for things that are
wrong with the path before looking at the disk.

Add some 3 tests; two broken as a reminder to actually use this new
code.
2015-06-12 07:34:47 +02:00
David Bremner
783a559968 test: redirect man output to /dev/null
In the case the these tests fail, they generate a bunch of output;
this output is not very interesting because it is just the successful
output of a man page. It does however make it hard to see what tests are actually failing, even with NOTMUCH_TEST_QUIET
2015-06-08 16:42:26 +02:00
David Bremner
ee1f573204 test: make python tests compatible with python3
Making the test suite actually run them with python3 is left for
future work.
2015-06-08 16:41:53 +02:00
David Bremner
243d7e30dc Merge branch 'release'
20.1 plus NEWS fixup
2015-06-02 06:37:59 +02:00
David Bremner
447ad6b498 cli: add standard option processing to config, help and setup
In particular this fixes a recently encountered bug where the
"--config" argument to "notmuch setup" is silently ignored, which the
unpleasant consequence of overwriting the users config file.
2015-06-01 07:32:54 +02:00
J. Lewis Muir
d08af93c65 cli: change "setup" to "set up" where used as a verb
The word "setup" is a noun, not a verb.  Change occurrences of "setup"
where used as a verb to "set up".
2015-05-31 19:14:42 +02:00
David Bremner
5df1b1bb81 test: remove redundant 'file' command from gdb scripts.
Quoting Debian bug 787341

   It failed to build on arm64: the last ten tests in T070-insert
   failed.

   What's happening here is that GDB is segfaulting in response to
   the
   "file" command. GDB on arm64 can be a bit buggy.

   However, the "file" command is redundant here as GDB has already
   got
   the file from the --args on the command line.
2015-05-31 18:56:10 +02:00
David Bremner
294bb6de20 test: move nonexistent directory to something under notmuch's control.
Apparently some systems actually have a directory called /nonexist[ae]nt.
It's hard to fathom a good reason for that, but oh well. As long as we
don't create such a directory inside the notmuch source tree, the new
version should be more robust.
2015-04-08 23:43:55 +09:00
David Bremner
2e9ecb611a test: be consistent about spelling `nonexistent'
Apparently most of the misspellings are my fault.
2015-04-08 23:43:55 +09:00
Tomi Ollila
3d1483df12 test/thread-order: more robust loop exit in case of broken input
When creating $THREADS data it may end of not having 'None' at all
or the numbers in line output yields a loop.

To avoid loop the value in current array index is set to 'None'
so that if the same item is reached again the loop will end.

Also empty string as next array index will end the loop.
2015-04-03 09:27:57 +09:00
David Bremner
703c8f9511 test: make one error output test more robust.
Since notmuch_database_status_string can return NULL, passing it
directly to fputs is not a good idea.
2015-03-29 09:50:01 +02:00
David Bremner
6a99505625 test: add more error reporting tests
This second half of the error reporting tests actually uses the function
notmuch_database_status_string to retrieve the last logged error
2015-03-29 00:34:15 +01:00
David Bremner
736ac26407 lib: replace almost all fprintfs in library with _n_d_log
This is not supposed to change any functionality from an end user
point of view. Note that it will eliminate some output to stderr. The
query debugging output is left as is; it doesn't really fit with the
current primitive logging model. The remaining "bad" fprintf will need
an internal API change.
2015-03-29 00:34:15 +01:00
David Bremner
84d3b15d25 lib: add "verbose" versions of notmuch_database_{open,create}
The compatibility wrapper ensures that clients calling
notmuch_database_open will receive consistent output for now.

The changes to notmuch-{new,search} and test/symbol-test are just to
make the test suite pass.

The use of IGNORE_RESULT is justified by two things. 1) I don't know
what else to do.  2) asprintf guarantees the output string is NULL if
an error occurs, so at least we are not passing garbage back.
2015-03-29 00:34:15 +01:00
David Bremner
83298fa0f8 test: add error reporting tests
This first half of the tests is all that don't need to retrieve the
the error string explicitly from the notmuch database structure.
2015-03-29 00:34:15 +01:00
David Bremner
1e69afe8af test: add support for compiling and running C snippets
This is to limit the copy-pasta involved in running C tests. I decided
to keep things simple and not try to provide an actual C skeleton.

The setting of LD_LIBRARY_PATH is to force using the built libnotmuch
rather than any potential system one.
2015-03-29 00:33:56 +01:00
David Bremner
baf93369f6 test: Add two tests for error output from notmuch_database_open
This is arguably testing the same thing twice, but in the brave new
future where we don't use printf anymore, each subcommand will be
responsible for handling the output on it's own.
2015-03-15 20:27:20 +01:00
David Bremner
c883e632bf CLI: make gpg binary used by libgmime configurable.
Previously we set up a way for the top level notmuch command to choose
which gpg binary was invoked by libgmime. In this commit we add the
(mostly boilerplate) code to allow the notmuch-config command to read
and write this path, and use it in the appropriate struct.

Update tests for new default variable
2015-03-11 08:04:00 +01:00
Jani Nikula
6aeef2ee15 parse-time-string: fix setting and rounding of seconds
If seconds are not specified in the string to be parsed, they're not
set according to the reference time (in the no rounding case) nor
rounded properly (in the rounding up cases). Fix this.

The bug caused searches such as date:10:30..10:30 to match messages
with date exactly 10:30:00 only, and not in range 10:30:00..10:30:59
(inclusive) as documented.

Note that date searches referring "noon" or "5pm" will still be
interpreted as exact to the second.
2015-03-07 10:20:33 +01:00
Jani Nikula
952f46fedc test: extract and flag second rounding tests as broken
Extract and add some more tests relating to rounding seconds into a
separate subtest, and flag it as broken. If seconds are not specified,
the time is not set or rounded according to the interface defined in
parse-time-string/parse-time-string.h. Instead, seconds are always set
to 00 in reality, which is broken.
2015-03-06 21:36:20 +01:00
Jani Nikula
65dee41fb6 test: add some more time tests
Ensure the seconds are set properly.
2015-03-06 21:36:11 +01:00
Jani Nikula
fd80a9cac1 test: improve the time parsing tests
Set the reference time to 12:13:14 instead of 11:11:00 to ensure hours
and minutes are not mixed up, and seconds are really set instead of
defaulted to 00.
2015-03-06 21:35:58 +01:00
Todd
b04bc967f9 Add indexing for the mimetype term
This adds the indexing support for the "mimetype:" term and removes
the broken test flag.  The indexing is probablistic in Xapian terms,
which gives a better experience to end users.  Standard content-types
of the form "foo/bar" are automatically interpreted as phrases in
Xapian due to the embedded slash.

Assume, separate messages with application/pdf and application/x-pdf
are indexed, then:

- mimetype:application/x-pdf will find only the application/x-pdf
- mimetype:application/pdf will find only the application/pdf
- mimetype:pdf will find both of the messages
2015-01-24 16:47:59 +01:00
Todd
0829bd4e5d test: Add failing unit tests for indexed mime types
Adds three failing unit tests for searching of mime-types.

An attempt was made at adding a negative test (i.e. searching for a
non-existent mime-type and ensuring it didn't return a message), but
that test would always pass making it pointless.
2015-01-24 16:47:38 +01:00
Tomi Ollila
7fcd100a2f test: prepare test-lib.sh for possible test system debug session
When something in tests fails one possibility to test is to run
the test script as `bash -x TXXX-testname.sh`. As stderr (fd 2) was
redirected to separate file during test execution also this set -x
(xtrace) output would also go there.
test-lib.sh saves the stderr to fd 7 from where it can be restored,
and bash has BASH_XTRACEFD variable, which is now given the same value
7, making bash to output all xtrade information (consistently) there.

This lib file used to save fd's 1 & 2 to 6 & 7 (respectively) in
test_begin_subtest(), but as those needs to be set *before* XTRACEFD
variable is set those are now saved at the beginning of the lib (once).
This is safe and simple thing to do.
To make xtrace output more verbose PS4 variable was set to contain the
source file, line number and if execution is in function, that function
name. Setting this variable has no effect when not xtracing.

As it is known that fd 6 is redirected stdout, printing status can now
use that fd, instead of saving stdout to fd 5 and use it.
2015-01-19 08:15:39 +01:00
Jesse Rosenthal
62bc1d4717 thread-naming test: Test empty subject names.
We test all empty subjects, and then empty subjects followed by
non-empty subjects (searching both oldest- and newest-first).
2015-01-17 13:49:26 +01:00
Jesse Rosenthal
c0c007dd7f test-lib: Add dummy subject to force empty subject
At the moment, the test-lib fills in any missing headers. This makes
it impossible to test our handling of empty subjects. This will
allow us to use a special dummy subject -- `@FORCE_EMPTY` -- to force
the subject to remain empty.
2015-01-17 13:49:05 +01:00
Jani Nikula
c82a1745ac lib: drop the deprecation message for single-message mbox files
We generally do not support mbox files, but for historical reasons
we've supported single-message mbox files, with a deprecation
message. We've tried dropping the support altogether, but backed out
of it because we'd need to stop indexing them, while keeping support
for previously indexed files. This would be more complicated than
simply supporting single-message mbox files. Therefore, drop the
deprecation message, and just silently accept single-message mboxes.
2015-01-01 16:47:47 +01:00
Jesse Rosenthal
4135a1f8da lib: Use email address instead of empty real name.
Currently, if a From-header is of the form:

    "" <address@example.com>

the empty string will be treated as a valid real-name, and the entry
in the search results will be empty.

The new behavior here is that we treat an empty real-name field as if
it were null, so that the email address will be used in the search
results instead.

Signed-off-by: Jesse Rosenthal <jrosenthal@jhu.edu>
2014-12-07 13:36:08 +01:00
Jesse Rosenthal
6c1d626519 test: Add known-broken test for empty author name
We test for whether a quoted empty email address

    "" <address@example.com>

will show up as the address, instead of the empty string. This is
marked as known-broken, since the current behavior is to use the empty
string.

This is a new test file, since handling of unusual email addresses
doesn't seem to fit well in any of our existing tests.

Signed-off-by: Jesse Rosenthal <jrosenthal@jhu.edu>
2014-12-07 13:32:12 +01:00
Tomi Ollila
0d8251dbe2 cli: notmuch address option defaults update
When no --output option were given, change default to display senders
only.

When only --count option is given, display senders (in contrary to not
displaying anything).

Document that --count with --sort=**order** is not supported option
combination.

Clean up some whitespace in the documentation.

One test updated to have --output=count without sender nor recipient
output option.
2014-11-09 10:09:51 +01:00
Michal Sojka
a5a6859197 cli: address: Add --output=count
This output prints how many times was each address encountered during
search.
2014-11-05 23:25:05 +01:00
Michal Sojka
4176e527fc cli: address: Do not output duplicate addresses
This filters out duplicate addresses from address command output.

It also also adds tests for the address command.

The code here is an extended version of a patch from Jani Nikula.
2014-11-05 23:23:01 +01:00
Jesse Rosenthal
0d40b37954 test: Make gen-threads work with python3
python3 doesn't allow dictionaries to be initialized with non-string
keywords. This presents problems on systems in which "python" means
"python3". We instead initalize the dictionary using the dict
comprehension and then update it with the values from the tree. This
will work with both python2 and python3.
2014-11-02 19:43:39 +01:00
Jani Nikula
207f3bf821 test: add tests for --output=messages --duplicate=N
Basic smoke tests for the feature, nothing fancy.
2014-11-02 19:42:28 +01:00
Jani Nikula
ecc4a9a644 cli: Add support for parsing keyword-flag arguments
This allows having multiple --foo=bar --foo=baz options on the command
line, with the corresponding values OR'd together.

[Test added by Michal Sojka]
2014-11-01 08:02:21 +01:00
Jani Nikula
0f35ddcdfc test: use LDFLAGS in test/Makefile.local
Apparently the test binaries are built with minimal LDFLAGS, only
adding dependency specific LDFLAGS as needed. However because some of
the test binaries incorporate notmuch object files, it is necessary to
use the same link flags as notmuch. For example user provided
CFLAGS/CXXFLAGS/LDFLAGS with -fsanitize=undefined fails to build the
test binaries if the flags differ.
2014-10-31 17:51:22 +01:00