This conversion is trivial because the only configuration information
accessed by dump is that stored in the database (in order to dump
it). We do need to be careful to keep the write lock on the database
to ensure dump consistency.
The main effort is changing from the old argv style config list
iterators to the new more opaque ones provided by the library (and
backed by the database+file config cache).
By using an enum we can have better error detection than copy pasting
key strings around.
The question of what layer this belongs in is a bit
tricky. Historically most of the keys are defined by the CLI. On the
other hand features like excludes are supported in the
library/bindings, and it makes sense to configure them from the
library as well.
The somewhat long prefix for notmuch_config_t is to avoid collisions
with the existing usage in notmuch-client.h.
Fill in the remainder of the documented functionality for
n_d_open_with_config with respect to config file location. Similar
searching default locations of the database file still needs to be
added.
The main goal is to allow configuration information to be temporarily
overridden by a separate config file. That will require further
changes not in this commit.
The performance impact is unclear, and will depend on the balance
between number of queries and number of distinct metadata items read
on the first call to n_d_get_config.
In ee897cab8b the upgrade tests from pre v3 databases were
removed. The reasons for that are still valid, but we should still
test the code paths that do the upgrade, and it is relatively
straightforward to do that for v3 to v3 upgrades.
`outline-minor-mode' treats comments that begin with three or more
semicolons as headings. That makes it very convenient to navigate
code and to show/hide parts of a file.
Elips libraries typically have four top-level sections, e.g.:
;;; notmuch.el --- run notmuch within emacs...
;;; Commentary:...
;;; Code:...
;;; notmuch.el ends here
In this package many libraries lack a "Commentary:" section, which is
not optimal but okay for most libraries, except major entry points.
Depending on how one chooses to look at it, the "... ends here" line
is not really a heading that begins a section, because it should never
have a "section" body (after all it marks eof).
If the file is rather short, then I left "Code:" as the only section
that contains code. Otherwise I split the file into multiple sibling
sections. The "Code:" section continues to contain `require' and
`declare-function' forms and other such "front matter".
If and only if I have split the code into multiple sections anyway,
then I also added an additional section named just "_" before the
`provide' form and shortly before the "...end here" line. This
section could also be called "Back matter", but I feel it would be
distracting to be that explicit about it. (The IMO unnecessary but
unfortunately still obligatory "... ends here" line is already
distracting enough as far as I am concerned.)
Before this commit some libraries already uses section headings, some
of them consistently. When a library already had some headings, then
this commit often sticks to that style, even at the cost inconsistent
styling across all libraries.
A very limited number of variable and function definitions have to be
moved around because they would otherwise end up in sections they do
not belong into.
Sections, including but not limited to their heading, can and should
be further improved in the future.
readelf on (at least) ppc64le sometimes generates some extension to
the Ndx name inside '[]'. Remove this output to allow our simple
column based parsing to work.
It turns out that using nm -P isn't as portable as hoped. In
particular with some ELF ABIs (e.g. ppc64 ELFv1), the desired symbols
end up in the data section instead of text.
The test is currently only functional on ELF based architectures, so I
think it's legit to depend on readelf instead of nm.
The switch to readelf has the advantage that we can explicitely ask
for all of the symbols with global visibility, rather than grepping
for notmuch. That seems a more robust approach since it will catch any
strangely named global symbols.
notmuch insert does not currently support passing a filename for the
input, so all of these tests have an extra error in addition to the
one being tested for.
Currently this does not make a difference because the error being
tested for is caught before the error of an extra command line
argument. In the future it might make a difference, and in any case it
is confusing.
Previously in message-show mode message's first header line (From
header) was always indented, even if user had turned thread
indentation off with "<" (notmuch-show-toggle-thread-indentation)
command.
This change modifies notmuch-show-insert-headerline function so that
it doesn't indent the first header line if notmuch-show-indent-content
variable is nil.
This change also modifies tests so that they expect this new output
format:
test/emacs-show.expected-output/notmuch-show-indent-thread-content-off
Use `makefile-gmake-mode' instead of `makefile-mode' because the
former also highlights ifdef et al. while the latter does not.
"./Makefile.global" and one "Makefile.local" failed to specify any
major mode at all but doing so is necessary because Emacs does not
automatically figure out that these are Makefiles (of any flavor).
On some systems (notably, the one shipped with LibreSSL),
default fingerprint digest algorithm is SHA256.
On other systems, users can change default digest algorithm by changing
default_md in /etc/ssl/default_md.
Let's ask openssl to provide us specific algorithm to make the test
more deterministic.
Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
Inspired by the suggestion of
id:20200727154108.16269-3-congdanhqx@gmail.com
to drop the configuration test for the default backend. This version
is hopefully robust against backend changes.
This is similar to the case of toplevel messages. Currently everything
is cached, so no database access is necessary. This might change in
the future, but it should not crash in either case.
The goal of this abstraction was to save space. But that failed as
the result actually was that four trivial lines got replace with 15
fairly complicated lines. The opposite of what it was supposed to
do.
Also it made it harder to come up with the fix in the previous commit;
simply grepping for the relevant symbols did not work because they get
constructed at run-time instead of appearing in the source file.
Starting with Emacs 27 undeclared variables in evaluated interactive
code uses lexical scope. This includes code passed with '--eval' as
we do in the Emacs tests, which also happen to assume dynamic scope.
- This can affect variables defined by libraries that we use. We
let-bind such variables to change the behavior of functions which we
then call with these bindings in effect. If these libraries are not
loaded beforehand, then the bindings are lexical and fail to have
the effect we intended.
At this time only 'smtpmail' has to be loaded explicitly (for the
variables let-bound in emacs_deliver_message and emacs_fcc_message).
'message' doesn't have to be loaded explicitly, because loading
'notmuch' (in 'run_emacs') already takes care of that, indirectly.
- Our own testing-only variables also have to be declared explicitly.
We should have done that anyway, but because of how and where these
variables are used it was very easy to overlook that (i.e. it isn't
something the byte-compiler ever looks at). Not so in Emacs 27
anymore; here this oversight caused four tests to fail.
The numeric values of these variables get incremented by functions
that we add to hooks that are run by many tests, not just the tests
where we actually inspect the value and therefore take care to let-
bind the values to 0 before we begin. The global values therefore
have to be numeric values as well. I have chosen -100 instead of 0
as the default in case someone writes a test that inspects the value
but forgets to let-bind the value. I hope that the unusual negative
value that one is going to see in such a case will help debugging
the issue.
The API docs promise to handle relative filenames, but the code did
not do it.
Also check for files outside the mail root, as implied by the API
description.
This fixes the bug reported at
id:87sgdqo0rz.fsf@tethera.net
Xapian currently succeeds to begin/end a transaction on a closed database,
or at least does not throw an exception. Make the test robust against
this changing.
The original generic handler had an extra '%s' in the format
string. Update tests that failed to catch this because the template to
print status strings checked 'stat', which was not set.
This is actually one of the few potentially useful things you can do
with a message belonging to a closed database, since in principle you
could re-open the database.
It's not very nice to return FALSE for an error, so provide
notmuch_message_get_flag_st as a migration path.
Bump LIBNOTMUCH_MINOR_VERSION because the API is extended.
Instead of printing the same static string for each test, can replace
the assert with something simpler (or at least easier to integrate
into the test suite).
These are less crucial since we stopped generating new database
versions and relied primarily on features. They also rely on a
pre-generated v1 database which happens to be chert format. This
backend is not supported by Xapian 1.5.
Also drop the tool gen-testdb.sh, which is currently broken, due to
changes in the testing infrastructure.
I haven't traced the code path as exhaustively for the SMIME test, but
the expiry date in question is larger then representable in a signed
32 bit integer.
This is a simple hack to enable out-of-tree builds, a concern raised
by Tomi in id:m24kzjib9a.fsf@guru.guru-group.fi
This change at least enables "make check" to complete without error,
but I'm sure it could be improved. I am not expert enough in
setuptools to know how.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Amended by db per id:87d06usa31.fsf@powell.devork.be
json_check_nodes.py exists in source tree, not in out of tree
build tree. Added -B to the execution so source tree is not
"polluted" by a .pyc file when json_check_nodes.py is executed.
When creating run_emacs.sh make it load .elc files from out of
tree build tree, not from source tree if such files existed.
If existed, those may be outdated, or even created by some other
emacs than the one that was used to build .elc files in out of
tree build dir.
This change means we can support "notmuch show --decrypt=true" for
S/MIME encrypted messages, resolving several outstanding broken tests,
including all the remaining S/MIME protected header examples.
We do not yet handle indexing the cleartext of S/MIME encrypted
messages, though.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
When composing a reply, no one wants to see this line in the proposed
message:
Non-text part: application/pkcs7-mime
So we hide it, the same way we hide PGP/MIME cruft.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Until we did PKCS#7 unwrapping, no leaf MIME part could have a child.
Now, we treat the unwrapped MIME part as the child of the PKCS#7
SignedData object. So in that case, we want to show it instead of
deliberately omitting the content.
This fixes the test of the protected subject in
id:smime-onepart-signed@protected-headers.example.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Unwrap a PKCS#7 SignedData part unconditionally when the cli is
traversing the MIME tree, and return it as a "child" of what would
otherwise be a leaf in the tree.
Unfortunately, this also breaks the JSON output. We will fix that
next.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
When we are indexing, we should treat SignedData parts the same way
that we treat a multipart object, indexing the wrapped part as a
distinct MIME object.
Unfortunately, this means doing some sort of cryptographic
verification whose results we throw away, because GMime doesn't offer
us any way to unwrap without doing signature verification.
I've opened https://github.com/jstedfast/gmime/issues/67 to request
the capability from GMime but for now, we'll just accept the
additional performance hit.
As we do this indexing, we also apply the "signed" tag, by analogy
with how we handle multipart/signed messages. These days, that kind
of change should probably be done with a property instead, but that's
a different set of changes. This one is just for consistency.
Note that we are currently *only* handling signedData parts, which are
basically clearsigned messages. PKCS#7 parts can also be
envelopedData and authEnvelopedData (which are effectively encryption
layers), and compressedData (which afaict isn't implemented anywhere,
i've never encountered it). We're laying the groundwork for indexing
these other S/MIME types here, but we're only dealing with signedData
for now.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
When checking cryptographic signatures, Notmuch relies on GMime to
tell it whether the certificate that signs a message has a valid User
ID or not.
If the User ID is not valid, then notmuch does not report the signer's
User ID to the user. This means that the consumer of notmuch's
cryptographic summary of a message (or of its protected headers) can
be confident in relaying the reported identity to the user.
However, some versions of GMime before 3.2.7 cannot report Certificate
validity for X.509 certificates. This is resolved upstream in GMime
at https://github.com/jstedfast/gmime/pull/90.
We adapt to this by marking tests of reported User IDs for
S/MIME-signed messages as known-broken if GMime is older than 3.2.7
and has not been patched.
If GMime >= 3.2.7 and certificate validity still doesn't work for
X.509 certs, then there has likely been a regression in GMime and we
should fail early, during ./configure.
To break out these specific User ID checks from other checks, i had to
split some tests into two parts, and reuse $output across the two
subtests.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Several functions in test/test-lib.sh used variable names that are
also used outside of those functions (e.g. $output and $expected are
used in many of the test scripts), but they are not expected to
communicate via those variables.
We mark those variables "local" within test-lib.sh so that they do not
get clobbered when used outside test-lib.
We also move the local variable declarations to beginning of each
function, to avoid weird gotchas with local variable declarations as
described in https://tldp.org/LDP/abs/html/localvar.html.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
strncmp looks for a prefix that matches, which is very much not what
we want here. This fixes the bug reported by Franz Fellner in
id:1588595993-ner-8.651@TPL520
In id:1588595993-ner-8.651@TPL520 Franz Fellner reported that tags
starting with 'attachment' are removed by 'notmuch reindex'. This is
probably related to the use of STRNCMP_LITERAL in
_notmuch_message_remove_indexed_terms.
GPGME has a strange failure mode when it is in offline mode, and/or
when certificates don't have any CRLs: in particular, it refuses to
accept the validity of any certificate other than a "root" cert.
This can be worked around by setting the `disable-crl-checks`
configuration variable for gpgsm.
I've reported this to the GPGME upstream at
https://dev.gnupg.org/T4883, but I have no idea how it will be
resolved. In the meantime, we'll just work around it.
Note that this fixes the test for verification of
id:smime-multipart-signed@protected-headers.example, because
multipart/signed messages are already handled correctly (one-part
PKCS#7 messages will get fixed later).
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Add a simple S/MIME SignedData message, taken from an upcoming draft
of
https://datatracker.ietf.org/doc/draft-autocrypt-lamps-protected-headers/
RFC 8551 describes a SignedData, a one-part clearsigned object that is
more resistant to common patterns of MTA message munging than
multipart/signed (but has the downside that it is only readable by
clients that implement S/MIME).
To make sure sure notmuch can handle this kind of object, we want to
know a few things:
Already working:
- Is the content of the SignedData object indexed? It actually is
right now because of dumb luck -- i think we're indexing the raw
CMS object and it happens to contain the cleartext of the message
in a way that we can consume it before passing it on to Xapian.
- Are we accidentally indexing the embedded PKCS#7 certificates? We
don't want to, and for some reason I don't understand, our indexing
is actually skipping the embedded certificates already. That's
good!
Still need fixing:
- do we know the MIME type of the embedded part?
- do we know that the message is signed?
- can notmuch-show read its content?
- can notmuch-show indicate the signature validity?
- can notmuch-reply properly quote and attribute content?
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
This test does exactly what it says on the tin. It expects JSON data
to be parseable by Python, at least.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
These tests describe some simple behavior we would expect to work if
we were to correctly index the cleartext of encrypted S/MIME messages
(PKCS#7 envelopedData).
Of course, they don't currently pass, so we mark them known-broken.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
