Commit graph

6405 commits

Author SHA1 Message Date
David Bremner
949c27144e debian: changelog stanza for 0.25.1-1 2017-09-11 22:22:22 -03:00
David Bremner
aeeb966f95 NEWS: news entry for 0.25.1 2017-09-11 22:20:26 -03:00
David Bremner
0fa257cb74 version: bump to 0.25.1 2017-09-11 22:19:46 -03:00
Jani Nikula
9b05823838 emacs: override enriched-decode-display-prop for text/enriched display
Switch to a local version of enriched-decode-display-prop if we
encounter a text/enriched part. This is to mitigate
https://bugs.gnu.org/28350. Normally it would be prudent to remove the
override afterwards, but in this case just leave it in.

Notes from db:
      This doesn't disable text/enriched, just one feature of it.
2017-09-11 22:08:01 -03:00
Daniel Kahn Gillmor
3445385f95 fix documentation bug (leading quotes break documentation) 2017-09-05 21:54:46 -03:00
David Bremner
64e30aeb24 test/duplicate-mid: check for subject with notmuch-show
In [1] Mark showed that the the current code (d7a49e81) is not
consistent in it's handling of subjects of messages with duplicate
message-ids (or in notmuch-speak, of messages with multiple files).
notmuch-search uses indexing order and explicitedly preserves the
first. notmuch-show (apparently) uses alphabetical (or at least xapian
term order) of filenames. In a perfect world we would probably report
all subjects in the json output; at the very least we should be
consistent.

[1]: id:87378dny3d.fsf@qmul.ac.uk
2017-09-05 21:53:06 -03:00
David Bremner
debfae20db lib: enforce that n_message_reindex takes headers from first file
This is still a bit stopgap to be only choosing one set of headers,
but this seems like a more defensible set of headers to choose.
2017-09-05 21:51:57 -03:00
David Bremner
36a3d65034 test: known broken test for subject after reindexing
In [1], Mark gave a test that was behaving strangly. This turns out to
be specific to reindexing.  I suppose one could argue that picking the
lexicographically last file name is a defensible choice, but it's
almost as easy to take the first, which seems more intuitive. So mark
the current situation as broken.

[1]: id:1503859703-2973-1-git-send-email-markwalters1009@gmail.com
2017-09-05 21:51:21 -03:00
David Bremner
17aebb6e7d test/duplicate-mid: clarify index order vs filename order
The existing test for notmuch search had the first in filename order
the same as the first indexed, which made it harder to understand what
the underlying behaviour is. Add a file with a lexicographically
smaller name, but later index time to clarify this.
2017-09-05 21:51:07 -03:00
David Bremner
44914f1c3b test: make fallback to duplicate test more robust.
The original intent of this test was to verify that notmuch show was
not crashing when the first file (where headers are being read from)
was deleted. Run the output through some sanitization so that as we
add and delete copies we don't have to update this test.
2017-09-05 21:50:54 -03:00
Vladimir Panteleev
1348212cb0 emacs: Refuse requests to refresh tree views while a refresh is running
notmuch-tree did not protect against concurrent refreshes like
notmuch-search, meaning, hitting '=' (notmuch-refresh-this-buffer)
quickly will spawn multiple parallel notmuch processes, and clobber
the existing results in the current buffer.

* notmuch-tree.el: Add a guard to notmuch-tree-refresh-view similar to
  the one in notmuch-search.
2017-09-05 21:50:03 -03:00
David Bremner
0260ee371e lib&cli: use g_object_new instead of g_object_newv
'g_object_newv' is deprecated, and prints annoying warnings. The
warnings suggest using 'g_object_new_with_properties', but that's only
available since glib 2.55 (i.e. a month ago as of this writing).
Since we don't actuall pass any properties, it seems we can just call
'g_object_new'.
2017-09-04 08:04:44 -03:00
Tomi Ollila
7526538808 test: notmuch_drop_mail_headers() style update
Changed "" quotes to '' as we're not supposed to dynamically
alter python program (via shell $variable expansion).

Added space to python program to match general python style.

Replaced $* with 'idiomatic' "$@" to serve as better example.
2017-09-04 08:03:52 -03:00
David Bremner
ec37900c5f test/crypto: remove headers more robustly
In [1], Vladimir Panteleev observed that the In-Reply-To and
References headers could be wrapped in the 'default' output format of
notmuch-reply, depending on the version of Emacs creating the
message. In my own experiments notmuch-reply sometimes wraps headers
with only one message-id if that message-id is long enough. However it
happens, this causes the previous approach using grep to fail.

Since I found the proposed unwrapping shell fragment in [1] a bit hard
to follow, I decided to write a little python script instead. Then
Tomi suggested a slight generalization of my script, and here we are.

[1] id:20170817175145.3204-7-notmuch@thecybershadow.net
2017-09-03 08:06:15 -03:00
Jan Malakhovski
46bda291ba build: fix unbound variable in configure
Commentary by db:

This missing initialization caused configure to crash, and hence the
build to fail in environments without cppcheck.
2017-08-31 10:09:02 -03:00
David Bremner
b52e1e0f1d gitignore: add .stamps 2017-08-30 20:14:43 -03:00
Vladimir Panteleev
d4bb606834 .travis.yml: Switch to "sudo: false" for faster builds
Builds not requiring sudo access run in a container, which will have
better performance and less overhead on the Travis infrastructure.

Use the apt addon to install dependencies instead of explicit apt-get
commands.
2017-08-30 07:26:49 -03:00
Vladimir Panteleev
95b82bb326 test: Fix T050-new.sh on some Travis CI machines
On some system configurations, setting a breakpoint on the "add_file"
function then issuing "continue" in gdb causes the debugger to
seemingly jump over the add_file invocation. This results in a test
failure, as the "Handle files vanishing between scandir and add_file"
subtest expects add_file to be called and fail due to the vanishing
file. The compiler optimization level also plays a role - the problem
can be reproduced with CFLAGS having -O2 but not -Og.

This problem was observed manifesting as a test failure on Travis CI
configured with "dist: trusty" and "sudo: false". It was not
reproducible on a local Docker image of Travis' runtime environment,
so Travis' virtualization infrastructure likely plays a role as well.

* T050-new.sh: Breakpoint notmuch_database_add_message instead of
  add_file to the same effect, and avoid bad gdb behaviour on Travis
  CI.

Amended by db:

	s/notmuch_database_add_message/notmuch_database_index_file/

Somehow the wrapper function doesn't work as a breakpoint; perhaps due
to inlining.
2017-08-30 07:24:44 -03:00
Vladimir Panteleev
06cd6513fa .travis.yml: Replace manual zlib installation with "dist: trusty"
Travis now offers Ubuntu Trusty (14.04 LTS) VMs as test runners, which
is gradually becoming the default. We can opt in to using Trusty now
so that we no longer need to manually update zlib to a newer version.
2017-08-30 07:22:40 -03:00
Vladimir Panteleev
2268eeb90d .travis.yml: Add gdb and gpgsm dependencies
This allows running more tests which were previously skipped due to
missing dependencies.
2017-08-30 07:20:34 -03:00
Vladimir Panteleev
544d0e288d .travis.yml: Invoke `make download-test-databases' before running tests
Download the test message database used for the T530-upgrade.sh test.

If the additional load on the web server is undesired, Travis can be
instructed to cache the file.
2017-08-30 07:18:29 -03:00
David Bremner
536d695a31 test/smtp-dummy: convert to 'goto DONE' style
Clean up several cppcheck warnings of the form

      - test/smtp-dummy.c:170: error: Resource leak: output

Conform to overall notmuch code style.
2017-08-30 07:16:24 -03:00
David Bremner
1fa5bd282e test/smtp-dummy: uncrustify
For some reason lost in the mists of time this code was indented 8 spaces.
2017-08-30 07:14:18 -03:00
David Bremner
10071f8b50 cppcheck: call va_end in _internal_error
fix for:
    util/error_util.c:38: error: va_list 'va_args' was opened but not
    closed by va_end()

This makes the code more copy-pastable, if nothing else
2017-08-30 07:12:13 -03:00
David Bremner
c960bb4686 cppcheck: close files during shutdown
Fix the following cppcheck errors:

    notmuch-count.c:207: error: Resource leak: input
    notmuch-tag.c:238: error: Resource leak: input

We know that the program is shutting down here, but it does no harm to
clean up a bit.
2017-08-30 07:10:07 -03:00
David Bremner
af64530d3f build: add target to run cppcheck
The advantage of having a target as opposed to running cppcheck by
hand

  - reuse list of source files
  - output errors in a format parsable, e.g. by emacs
  - returns exit code 1 on any error, for possibly use in other
    targets.

For the moment, leave this as an optional target. If desired, it can
be added to e.g. the release targets in the same way as the test
target.

Using two levels of directory for the stamps is arguably
overengineering, but it doesn't really cost anything, and leaves open
the possibility of putting other kinds of stamp files there.

This only checks "new" source files (w.r.t. their last check). A future target
(cppcheck-all ?) could blow away the stamp files first.
2017-08-29 22:05:37 -03:00
Jani Nikula
8c7d979f5d build: add .deps to CLEAN instead of listing in clean target
Seems, uh, cleaner this way.
2017-08-29 22:05:37 -03:00
Jani Nikula
ab3ac525a0 build: add optional target parameter to quiet variable function
Sometimes using $@ as the target in the quiet build lines can be
confusing. Accept an optional second parameter in the quiet variable
function to specify the target.
2017-08-29 22:05:37 -03:00
Jani Nikula
e027711148 build: avoid an extra shell out in quiet variable function
$(word 1, $1) yields the same result as the more complicated
$(shell echo $1 | sed -e s'/ .*//')
2017-08-29 22:05:37 -03:00
David Bremner
0082a55785 CLI/new: support maildir synced tags in new.tags
We reorder reading maildir flags to avoid overwriting 'new.tags'. The
inverted status of 'unread' means the maildir flag needs to be checked
a second time.

I backpedalled here on the idea of supporting 'new.tags' without
'unread' in the presence of maildir syncing.  For files in 'new/', it
seems quite natural to tag them as 'unread'.
2017-08-29 21:57:38 -03:00
David Bremner
0a40ea4b48 lib: add notmuch_message_has_maildir_flag
I considered a higher level interface where the caller passes a tag
name rather than a flag character, but the role of the "unread" tag is
particularly confusing with such an interface.
2017-08-29 21:56:21 -03:00
David Bremner
8a8fb39b0c lib/message: split n_m_maildir_flags_tags, store maildir flags
In a future commit this will allow querying maildir flags seperately
from tags to allow resolving certain conflicts.
2017-08-29 21:51:10 -03:00
David Edmondson
ca4852649e test: Perform T170 tests that don't require dtach before any that do.
This avoids the later tests seeing different versions of the database
depending on whether dtach is available.
2017-08-28 08:02:02 -03:00
Vladimir Panteleev
d7a49e8199 notmuch-tag.el: Fix minor grammar error 2017-08-23 08:08:01 -03:00
Vladimir Panteleev
8c772eaa0d test: Use small Python script for JSON normalization instead of json.tool
json.tool does not sort or otherwise normalize the order of JSON keys
in its output, which can result in test failures on some test systems.

Instead, use a one-line Python script passed to the interpreter
directly on its command line. Use sort_keys=True for json.dump to
ensure the key order is normalized. The script works with both Python
2 and 3.

* test/test-lib.sh: Update test_expect_equal_json.
2017-08-23 08:07:45 -03:00
Daniel Kahn Gillmor
eb232ee0ab reindex: drop notmuch_param_t, use notmuch_indexopts_t instead
There are at least three places in notmuch that can trigger an
indexing action:

 * notmuch new
 * notmuch insert
 * notmuch reindex

I have plans to add some indexing options (e.g. indexing the cleartext
of encrypted parts, external filters, automated property injection)
that should properly be available in all places where indexing
happens.

I also want those indexing options to be exposed by (and constrained
by) the libnotmuch C API.

This isn't yet an API break because we've never made a release with
notmuch_param_t.

These indexing options are relevant in the listed places (and in the
libnotmuch analogues), but they aren't relevant in the other kinds of
functionality that notmuch offers (e.g. dump/restore, tagging, search,
show, reply).

So i think a generic "param" object isn't well-suited for this case.
In particular:

 * a param object sounds like it could contain parameters for some
   other (non-indexing) operation.  This sounds confusing -- why would
   i pass non-indexing parameters to a function that only does
   indexing?

 * bremner suggests online a generic param object would actually be
   passed as a list of param objects, argv-style.  In this case (at
   least in the obvious argv implementation), the params might be some
   sort of generic string.  This introduces a problem where the API of
   the library doesn't grow as new options are added, which means that
   when code outside the library tries to use a feature, it first has
   to test for it, and have code to handle it not being available.
   The indexopts approach proposed here instead makes it clear at
   compile time and at dynamic link time that there is an explicit
   dependency on that feature, which allows automated tools to keep
   track of what's needed and keeps the actual code simple.

My proposal adds the notmuch_indexopts_t as an opaque struct, so that
we can extend the list of options without causing ABI breakage.

The cost of this proposal appears to be that the "boilerplate" API
increases a little bit, with a generic constructor and destructor
function for the indexopts struct.

More patches will follow that make use of this indexopts approach.
2017-08-23 07:55:12 -03:00
Daniel Kahn Gillmor
b10ce6bc23 database: add n_d_index_file (deprecates n_d_add_message)
We need a way to pass parameters to the indexing functionality on the
first index, not just on reindexing.  The obvious place is in
notmuch_database_add_message.  But since modifying the argument list
would break both API and ABI, we needed a new name.

I considered notmuch_database_add_message_with_params(), but the
functionality we're talking about doesn't always add a message.  It
tries to index a specific file, possibly adding a message, but
possibly doing other things, like adding terms to an existing message,
or failing to deal with message objects entirely (e.g. because the
file didn't contain a message).

So i chose the function name notmuch_database_index_file.

I confess i'm a little concerned about confusing future notmuch
developers with the new name, since we already have a private
_notmuch_message_index_file function, and the two do rather different
things.  But i think the added clarity for people linking against the
future libnotmuch and the capacity for using index parameters makes
this a worthwhile tradeoff.  (that said, if anyone has another name
that they strongly prefer, i'd be happy to go with it)

This changeset also adjusts the tests so that we test whether the new,
preferred function returns bad values (since the deprecated function
just calls the new one).

We can keep the deprecated n_d_add_message function around as long as
we like, but at the next place where we're forced to break API or ABI
we can probably choose to drop the name relatively safely.

NOTE: there is probably more cleanup to do in the ruby and go bindings
to complete the deprecation directly.  I don't know those languages
well enough to attempt a fix; i don't know how to test them; and i
don't know the culture around those languages about API additions or
deprecations.
2017-08-23 07:38:37 -03:00
Yuri Volchkov
09fa51303c show: workaround for the missing file problem
This patch fixes the 'Deleted first duplicate file does not stop
notmuch show from working' test.

If a message to be shown has several duplicated files, and for some
reason the first file in the list is not available anymore, notmuch
will exit with an error.

This is clearly a problem in the database, but we are not going to let
this problem be a show-stopper. Let's walk through the list, and show
the first existing file.

Signed-off-by: Yuri Volchkov <yuri.volchkov@gmail.com>
2017-08-22 18:48:29 -03:00
Yuri Volchkov
39cc84e9fc test: show id:<> works even if the first duplicate is deleted
Signed-off-by: Yuri Volchkov <yuri.volchkov@gmail.com>
2017-08-22 18:48:19 -03:00
Yuri Volchkov
33a170e116 insert: strip trailing / in folder path
This patch fixes the "Insert message into folder with trailing /"
test. The problem was insufficient path canonization.

From database's point of view, "Sent" and "Sent/" are different
folders. If user runs (note the last '/'):

    notmuch insert --folder=maildir/Sent/ < test.msg

notmuch will create an extra XDIRECTORY record for the folder
'Sent/'. This means that database will have _TWO_ records for _ONE_
physical folder: 'Sent' and 'Sent/'. However, the 'notmuch new'
command will update only records related to the first one (the correct
one).

Now, if user moved the email file (e.g. from 'Sent/new' to
'Sent/cur'), 'notmuch new' will add a record about the new file, but
will not delete the old record.

Signed-off-by: Yuri Volchkov <yuri.volchkov@gmail.com>
2017-08-22 18:48:09 -03:00
Yuri Volchkov
cec4a87539 database: move striping of trailing '/' into helper function
Stripping trailing character is not that uncommon
operation. Particularly, the next patch has to perform it as
well. Lets move it to the separate function to avoid code duplication.

Also the new function has a little improvement: if the character to
strip is repeated several times in the end of a string, function
strips them all.

Signed-off-by: Yuri Volchkov <yuri.volchkov@gmail.com>
2017-08-22 18:47:51 -03:00
David Bremner
49d4f52f27 perf-test: add memory test for notmuch-insert
In the future it might be worthwhile selecting corpus messages to
insert, but that seems a bit overcomplicated for now
2017-08-20 08:48:43 -03:00
David Bremner
6da87a1369 perf-test: renumber tests
One test per number so ordering is clear.
2017-08-20 08:34:12 -03:00
David Bremner
1092c747ef test: move generate_message, add_message into test-lib-common.sh
The plan is to use at least the former in the perf test suite.
2017-08-20 08:34:03 -03:00
Daniel Kahn Gillmor
55f9f6505e lib: clarify description of notmuch_database_add_message
Since we're accumulating the index when we add a new file to the
message, the semantics have slightly changed.  This tries to align the
documentation with the actual functionality.
2017-08-20 08:33:46 -03:00
Vladimir Panteleev
fdf2b3007a emacs: Use make-process when available
make-process is a new function introduced in Emacs 25, which provides
greater control over process creation. Crucially, it allows
separately redirecting stderr directly to a buffer, which allows us to
avoid needing to use the shell to redirect to a temporary file in
order to correctly distinguish stdout and stderr.

* notmuch-lib.el: Use make-process when it is available; fall back to
  the previous method when not.
2017-08-20 08:33:09 -03:00
Vladimir Panteleev
69946c47c9 emacs: Refactor subprocess stderr propagation
Load subprocess error output to a string in the callers, and propagate
the error messages as a string parameter instead of a path to file
names.

Required to be able to avoid using temporary files for subprocess
error output.

* notmuch-lib.el: Update notmuch-check-async-exit-status,
  notmuch-check-exit-status: accept an err parameter instead of
  err-file; shift the responsibility of loading error messages from
  files up the call stack.
2017-08-20 08:32:47 -03:00
Vladimir Panteleev
938ec0f80c doc: Disable SmartyPants in generated manual pages
By default, Sphinx tries to pre-process text through SmartyPants,
which attempts to convert ASCII quotes and dashes to Unicode
characters. Unfortunately, this mangles technical text such as command
lines. For instance, this excerpt from notmuch-tag.rst:

  **notmuch** **tag** **--batch** [--input=<*filename*>]

got turned into:

  notmuch tag –batch [–input=<filename>]

That's an en-dash and an em-dash respectively.

Not only are these characters visually confusing and could easily be
mistaken for a single dash, copying and pasting such command lines
into a terminal is doomed to result in incomprehensible error
messages.

* doc/conf.py: Disable SmartyPants.
2017-08-20 08:32:29 -03:00
Jani Nikula
cb5253578d emacs: set query-context to nil if its "" or "*"
The queries "" and "*" are special cased in the notmuch library to
match all messages, but only if they're the entire query. They can't
be combined with other queries, such as "* AND foo", in which case
they "leak" down to the Xapian query parser.

Notmuch show and tree buffers inadvertently combine the thread query
with said special queries, causing incorrect collapsing of
messages. Handle the special queries specially. We already do a
similar thing in notmuch-search-filter.
2017-08-20 08:32:07 -03:00
Yuri Volchkov
cd8551d53a test: insert into the folder with trailing /
From database's point of view, "Drafts" and "Drafts/" are different
folders

Signed-off-by: Yuri Volchkov <yuri.volchkov@gmail.com>
Amended: add test_subtest_known_broken (db)
2017-08-20 08:28:55 -03:00