Add script to generate notmuch test databases using specified versions
of notmuch. This is useful for generating material for database
upgrade tests.
This reuses the test infrastructure to have a sandbox environment for
notmuch new etc.
Python dict() object does not have __values__() function which
OrderedDict().values() (the stub provided in nmbug-status) could call
to provide ordered list of values. By renaming this thinko to
values() will make our stub work as expected -- dict items listed out
in order those were added to the dictionary.
David [1] and Tomi [2] both feel that the user's choice of LANG is not
explicit enough to have such a strong effect on nmbug-status. For
example, cron jobs usually default to LANG=C, and that is going to
give you ASCII output:
$ LANG=C python -c 'import locale; print(locale.getpreferredencoding())'
ANSI_X3.4-1968
Trying to print Unicode author names (and other strings) in that
encoding would crash nmbug-status with a UnicodeEncodeError. To avoid
that, this patch hardcodes UTF-8, which can handle generic Unicode,
and is the preferred encoding (regardless of LANG settings) for
everyone who has chimed in on the list so far. I'd prefer trusting
LANG, but in the absence of any users that prefer non-UTF-8 encodings
I'm fine with this approach.
While we could achieve the same effect on the output content by
dropping the previous patch (nmbug-status: Encode output using the
user's locale), Tomi also wanted UTF-8 hardcoded as the config-file
encoding [2]. Keeping the output encoding patch and then adding this
to hardcode both the config-file and output encodings at once seems
the easiest route, now that fd29d3f (nmbug-status: Decode Popen output
using the user's locale, 2014-02-10) has landed in master.
[1]: id="877g8z4v4x.fsf@zancas.localnet"
http://article.gmane.org/gmane.mail.notmuch.general/17202
[2]: id="m2vbwj79lu.fsf@guru.guru-group.fi"
http://article.gmane.org/gmane.mail.notmuch.general/17209
Instead of always writing UTF-8, allow the user to configure the
output encoding using their locale. This is useful for previewing
output in the terminal, for poor souls that don't use UTF-8 locales
;).
We already had the tbody with a blank row separating threads (which is
not colored); this commit adds a bit of spacing to separate messages
within a thread. It will also add a bit of colored padding above the
first message and below the final message, but the main goal is to add
padding *between* two-row message blocks.
<--- new padding
thread-1, message-1, row-1 (class="message-first")
thread-1, message-1, row-2 (class="message-last")
<--- new padding
spacer tbody with a blank row
<--- new padding
thread-2, message-1, row-1 (class="message-first")
thread-2, message-1, row-2 (class="message-last")
<--- new padding
<--- new padding
thread-2, message-2, row-1 (class="message-first")
thread-2, message-2, row-2 (class="message-last")
<--- new padding
'message-id' and 'from' now have sensitive characters escaped using
xml.sax.saxutils.escape [1]. The 'subject' data was already being
converted to a link into Gmane; I've escape()d that too, so it doesn't
need to be handled ain the same block as 'message-id' and 'from'.
This prevents broken HTML by if subjects etc. contain characters that
would otherwise be interpreted as HTML markup.
[1]: http://docs.python.org/3/library/xml.sax.utils.html#xml.sax.saxutils.escape
Also allow manual id overrides from the JSON config. Sluggin avoids
errors like:
Bad value '#Possible bugs' for attribute href on element a:
Whitespace in fragment component. Use %20 in place of spaces.
from http://validator.w3.org.
I tried just quoting the titles (e.g. 'Possible%20bugs'), but that
didn't work (at least with Firefox 24.2.0). Slugging avoids any
ambiguity over when the quotes are expanded in the client. The specs
are unclear about quoting, saying only [1]:
Value: Any string, with the following restrictions:
must be at least one character long
must not contain any space characters
[1]: http://dev.w3.org/html5/markup/global-attributes.html#common.attrs.id
HTML 5 for the win :). I also de-namespaced the language; the HTML 5
spec allows a vestigial xml:lang attribute, but it's a no-op [1], so I
stripped it.
This shouldn't break anything at tethera, which already serves the
status as text/html:
$ wget -S http://nmbug.tethera.net/status/
--2014-02-02 21:20:39-- http://nmbug.tethera.net/status/
Resolving nmbug.tethera.net... 87.98.215.224
Connecting to nmbug.tethera.net|87.98.215.224|:80... connected.
HTTP request sent, awaiting response...
HTTP/1.1 200 OK
Vary: Accept-Encoding
Content-Type: text/html
...
This also matches the Content-Type in the generated HTML's http-equiv
meta.
[1]: http://www.w3.org/TR/html5/dom.html#the-lang-and-xml:lang-attributes
Tomi Ollila and David Bremner (and presumably others) are running
Python 2.6 on their nmbug-status boxes, so it makes sense to keep
support for that version. This commit adds a really minimal
OrderedDict stub (e.g. it doesn't handle key removal), but it gets the
job done for Page._get_threads. Once we reach a point where Python
2.6 is no longer important (it's already out of it's security-fix
window [1]), we can pull this stub back out.
[1]: http://www.python.org/download/releases/2.6.9/
I was having trouble understanding the logic of the longish print_view
function, so I refactored the output generation into modular bits.
The basic text rendering is handled by Page, which has enough hooks
that HtmlPage can borrow the logic and slot-in HTML generators.
By modularizing the logic it should also be easier to build other
renderers if folks want to customize the layout for other projects.
Timezones
=========
This commit has not effect on the output, except that some dates have
been converted from the sender's timezone to UTC due to:
- val = m.get_header(header)
- ...
- if header == 'date':
- val = str.join(' ', val.split(None)[1:4])
- val = str(datetime.datetime.strptime(val, '%d %b %Y').date())
...
+ value = str(datetime.datetime.utcfromtimestamp(
+ message.get_date()).date())
I also tweaked the HTML header date to be utcnow instead of the local
now() to make all times independent of the generator's local time.
This matches Gmane, which converts all Date headers to UTC (although
they use a 'GMT' suffix). Notmuch uses
g_mime_utils_header_decode_date to calculate the UTC timestamps, but
uses a NULL tz_offset which drops the information we'd need to get
back to the sender's local time [1]. With the generator's local time
arbitrarily different from the sender's and viewer's local time,
sticking with UTC seems the best bet.
[1]: https://developer.gnome.org/gmime/stable/gmime-gmime-utils.html#g-mime-utils-header-decode-date
Make this all one big string, using '...{date}...'.format(date=...) to
inject the date [1]. This syntax was added in Python 2.6, and is
preferred to %-formatting in Python 3 [1].
[1]: http://docs.python.org/2/library/stdtypes.html#str.format
The database in only used for notmuch.Query, so there's no need for
write access. This allows nmbug-status to run while the database is
being updated, without raising:
A Xapian exception occurred opening database: Unable to get write lock on …: already locked
Traceback (most recent call last):
File "./nmbug-status", line 182, in <module>
db = notmuch.Database(mode=notmuch.Database.MODE.READ_WRITE)
File "/…/notmuch/database.py", line 154, in __init__
self.open(path, mode)
File "/…/notmuch/database.py", line 214, in open
raise NotmuchError(status)
notmuch.errors.XapianError
The definitions of Thread, output_with_separator, and print_view were
between the main argparse and view-printing code. Group them together
with our existing read_config at the top of the module, which makes
for easier reading in the main section.
I also:
* Made 'headers' a print_view argument instead of a module-level
global. The list -> tuple conversion avoids having a mutable
default argument, which makes some people jumpy ;).
* Made 'db' a print_view argument instead of relying on the global
namespace to access it from print_view.
Now the suggested usage (listed by 'nmbug-status --help') is:
usage: nmbug-status [-h] [--text] [--config PATH] [--list-views]
[--get-query VIEW]
instead of the less obvious:
usage: nmbug-status [-h] [--text] [--config CONFIG] [--list-views]
[--get-query GET_QUERY]
Avoid:
$ ./nmbug-status --list-views
Traceback (most recent call last):
File "./nmbug-status", line 47, in <module>
'cat-file', 'blob', sha1+':status-config.json'],
TypeError: can't concat bytes to str
by explicitly converting the byte-stream read from Popen into a
Unicode string. On Python 2, this conversion is str -> unicode; on
Python 3 it is bytes -> str.
_ENCODING is derived from the user's locale (or system default) in an
attempt to match Git's output encoding. It may be more robust to skip
the encoding/decoding by using a Python wrapper like pygit2 [1] for
Git access. That's a fairly heavy dependency though, and using the
locale will probably work.
[1]: http://www.pygit2.org/
This is a pretty basic config to get started, generated using 'doxygen
-s -g' and mildly tweaked.
To generate the library man page man/man3/notmuch.3 from lib/notmuch.h
use:
$ doxygen devel/doxygen.cfg
NOTMUCH_VERSION_* macros in lib/notmuch.h are replaced with
LIBNOTMUCH_VERSION_* macros. Check that the values of those
match the LIBNOTMUCH_*_VERSION values in lib/Makefile.local.
New defines NOTMUCH_MAJOR_VERSION, NOTMUCH_MINOR_VERSION and
NOTMUCH_MICRO_VERSION were added to lib/notmuch.h.
Check that these match the current value defined in ./version.
This fixes races in thread-local and global tagging in notmuch-search
(e.g., "+", "-", "a", "*", etc.). Previously, these would modify tags
of new messages that arrived after the search. Now they only operate
on the messages that were in the threads when the search was
performed. This prevents surprises like archiving messages that
arrived in a thread after the search results were shown.
This eliminates `notmuch-search-find-thread-id-region(-search)'
because these functions strongly encouraged racy usage.
This fixes the two broken tests added by the previous patch.
These queries will match exactly the set of messages currently in the
thread, even if more messages later arrive. Two queries are provided:
one for matched messages and one for unmatched messages.
This can be used to fix race conditions with tagging threads from
search results. While tagging based on a thread: query can affect
messages that arrived after the search, tagging based on stable
queries affects only the messages the user was shown in the search UI.
Since we want clients to be able to depend on the presence of these
queries, this ushers in schema version 2.
Previously, the show schema and the search schema used different
"thread" non-terminals. While these schemata don't interact, this is
still confusing, so rename search's "thread" to "thread_summary". To
further limit confusion, prefix all top-level search non-terminals now
begin with "search_".
I find this script pretty useful when figuring out who to blame for
MIME rendering problems.
The notmuch repo will be the new primary home for this script, unless
and until a better home turns up.
After new notmuch release has been published the NEWS and manual
pages have been updated using these 2 programs.
Adding the tools to notmuch repository eases their use, adds more
transparency to the "process" and gives more people chance to
do the updates is one is unavailable to do it at the time being.
Current code does not distinguish between an empty string in the
NMBPREFIX environment variable and the variable being undefined. This
makes it impossible to define an empty prefix, if, e.g. somebody wants
to dump all of their tags with nmbug.
This is at least easier to understand than the magic hash. It may also
be a bit more robust, although it is hard to imagine these numbers
changing without many other changes in git.
This should be more robust with respect to tags with whitespace and
and other special characters. It also (hopefully) fixes a remaining
bug handling message-ids with whitespace. It should also be
noticeably faster for large sets of changes since it does one exec per
change set as opposed to one exec per tag changed.
This should make nmbug tolerate tags with whitespace and other special
characters it. At the moment this relies on _not_ passing calls to
notmuch tag through the shell, which is a documented feature of perl's
system function.
Check that the underlining '===...' for first (header) line in NEWS
file is of the same length as the header text and it is all '=':s.
-- extra execs removed by db.
There seems to be consensus to use presence in contrib as
documentation of limited support by the notmuch developers; in fact
nmbug is pretty integrated into our current development process, so
devel seems more appropriate.
M-RET notmuch-show-open-or-close-all opens all closed messages.
The archiving change is mentioned twice, remove dupe.
"notmuch search" supports --format=text0 to work with xargs -0
The commands are long deprecated, so removal is probably overdue. The
real motivation is to simplify argument handling for notmuch so that
we can migrate to the common argument parsing framework.
Version string has strict format requirements in release-check.sh:
only numbers and periods (in sane order) are accepted.
Mismatch there used to halt further execution.
In this case, checking versions like '*~rc1' for (more) problems
was not possible.
This 'fatal error' is now changed buffered error message like in
following tests, and is displayed at the end of execution.
If a leaf part's body content is omitted, return the encoded length and
transfer encoding in --format=json output. This information may be used
by the consumer, e.g. to decide whether to download a large attachment
over a slow link.
Returning the _encoded_ content length is more efficient than returning
the _decoded_ content length. Returning the transfer encoding allows
the consumer to estimate the decoded content length.
Currently there is a period of pain whenever we make
backward-incompatible changes to the structured output format, which
discourages not only backward-incompatible improvements to the format,
but also backwards-compatible additions that may not be "perfect". In
the end, these problems limit experimentation and innovation.
This series of patches introduces a way for CLI callers to request a
specific format version on the command line and to determine if the
CLI does not supported the requested version (and perhaps present a
useful diagnostic to the user). Since the caller requests a format
version, it's also possible for the CLI to support multiple
incompatible versions simultaneously, unlike the alternate approach of
including version information in the output.
This patch lays the groundwork by introducing a versioning convention,
standard exit codes, and a utility function to check the requested
version and produce standardized diagnostic messages and exit
statuses.
