David [1] and Tomi [2] both feel that the user's choice of LANG is not
explicit enough to have such a strong effect on nmbug-status. For
example, cron jobs usually default to LANG=C, and that is going to
give you ASCII output:
$ LANG=C python -c 'import locale; print(locale.getpreferredencoding())'
ANSI_X3.4-1968
Trying to print Unicode author names (and other strings) in that
encoding would crash nmbug-status with a UnicodeEncodeError. To avoid
that, this patch hardcodes UTF-8, which can handle generic Unicode,
and is the preferred encoding (regardless of LANG settings) for
everyone who has chimed in on the list so far. I'd prefer trusting
LANG, but in the absence of any users that prefer non-UTF-8 encodings
I'm fine with this approach.
While we could achieve the same effect on the output content by
dropping the previous patch (nmbug-status: Encode output using the
user's locale), Tomi also wanted UTF-8 hardcoded as the config-file
encoding [2]. Keeping the output encoding patch and then adding this
to hardcode both the config-file and output encodings at once seems
the easiest route, now that fd29d3f (nmbug-status: Decode Popen output
using the user's locale, 2014-02-10) has landed in master.
[1]: id="877g8z4v4x.fsf@zancas.localnet"
http://article.gmane.org/gmane.mail.notmuch.general/17202
[2]: id="m2vbwj79lu.fsf@guru.guru-group.fi"
http://article.gmane.org/gmane.mail.notmuch.general/17209
Instead of always writing UTF-8, allow the user to configure the
output encoding using their locale. This is useful for previewing
output in the terminal, for poor souls that don't use UTF-8 locales
;).
We already had the tbody with a blank row separating threads (which is
not colored); this commit adds a bit of spacing to separate messages
within a thread. It will also add a bit of colored padding above the
first message and below the final message, but the main goal is to add
padding *between* two-row message blocks.
<--- new padding
thread-1, message-1, row-1 (class="message-first")
thread-1, message-1, row-2 (class="message-last")
<--- new padding
spacer tbody with a blank row
<--- new padding
thread-2, message-1, row-1 (class="message-first")
thread-2, message-1, row-2 (class="message-last")
<--- new padding
<--- new padding
thread-2, message-2, row-1 (class="message-first")
thread-2, message-2, row-2 (class="message-last")
<--- new padding
'message-id' and 'from' now have sensitive characters escaped using
xml.sax.saxutils.escape [1]. The 'subject' data was already being
converted to a link into Gmane; I've escape()d that too, so it doesn't
need to be handled ain the same block as 'message-id' and 'from'.
This prevents broken HTML by if subjects etc. contain characters that
would otherwise be interpreted as HTML markup.
[1]: http://docs.python.org/3/library/xml.sax.utils.html#xml.sax.saxutils.escape
Also allow manual id overrides from the JSON config. Sluggin avoids
errors like:
Bad value '#Possible bugs' for attribute href on element a:
Whitespace in fragment component. Use %20 in place of spaces.
from http://validator.w3.org.
I tried just quoting the titles (e.g. 'Possible%20bugs'), but that
didn't work (at least with Firefox 24.2.0). Slugging avoids any
ambiguity over when the quotes are expanded in the client. The specs
are unclear about quoting, saying only [1]:
Value: Any string, with the following restrictions:
must be at least one character long
must not contain any space characters
[1]: http://dev.w3.org/html5/markup/global-attributes.html#common.attrs.id
HTML 5 for the win :). I also de-namespaced the language; the HTML 5
spec allows a vestigial xml:lang attribute, but it's a no-op [1], so I
stripped it.
This shouldn't break anything at tethera, which already serves the
status as text/html:
$ wget -S http://nmbug.tethera.net/status/
--2014-02-02 21:20:39-- http://nmbug.tethera.net/status/
Resolving nmbug.tethera.net... 87.98.215.224
Connecting to nmbug.tethera.net|87.98.215.224|:80... connected.
HTTP request sent, awaiting response...
HTTP/1.1 200 OK
Vary: Accept-Encoding
Content-Type: text/html
...
This also matches the Content-Type in the generated HTML's http-equiv
meta.
[1]: http://www.w3.org/TR/html5/dom.html#the-lang-and-xml:lang-attributes
Tomi Ollila and David Bremner (and presumably others) are running
Python 2.6 on their nmbug-status boxes, so it makes sense to keep
support for that version. This commit adds a really minimal
OrderedDict stub (e.g. it doesn't handle key removal), but it gets the
job done for Page._get_threads. Once we reach a point where Python
2.6 is no longer important (it's already out of it's security-fix
window [1]), we can pull this stub back out.
[1]: http://www.python.org/download/releases/2.6.9/
I was having trouble understanding the logic of the longish print_view
function, so I refactored the output generation into modular bits.
The basic text rendering is handled by Page, which has enough hooks
that HtmlPage can borrow the logic and slot-in HTML generators.
By modularizing the logic it should also be easier to build other
renderers if folks want to customize the layout for other projects.
Timezones
=========
This commit has not effect on the output, except that some dates have
been converted from the sender's timezone to UTC due to:
- val = m.get_header(header)
- ...
- if header == 'date':
- val = str.join(' ', val.split(None)[1:4])
- val = str(datetime.datetime.strptime(val, '%d %b %Y').date())
...
+ value = str(datetime.datetime.utcfromtimestamp(
+ message.get_date()).date())
I also tweaked the HTML header date to be utcnow instead of the local
now() to make all times independent of the generator's local time.
This matches Gmane, which converts all Date headers to UTC (although
they use a 'GMT' suffix). Notmuch uses
g_mime_utils_header_decode_date to calculate the UTC timestamps, but
uses a NULL tz_offset which drops the information we'd need to get
back to the sender's local time [1]. With the generator's local time
arbitrarily different from the sender's and viewer's local time,
sticking with UTC seems the best bet.
[1]: https://developer.gnome.org/gmime/stable/gmime-gmime-utils.html#g-mime-utils-header-decode-date
Make this all one big string, using '...{date}...'.format(date=...) to
inject the date [1]. This syntax was added in Python 2.6, and is
preferred to %-formatting in Python 3 [1].
[1]: http://docs.python.org/2/library/stdtypes.html#str.format
The database in only used for notmuch.Query, so there's no need for
write access. This allows nmbug-status to run while the database is
being updated, without raising:
A Xapian exception occurred opening database: Unable to get write lock on …: already locked
Traceback (most recent call last):
File "./nmbug-status", line 182, in <module>
db = notmuch.Database(mode=notmuch.Database.MODE.READ_WRITE)
File "/…/notmuch/database.py", line 154, in __init__
self.open(path, mode)
File "/…/notmuch/database.py", line 214, in open
raise NotmuchError(status)
notmuch.errors.XapianError
The definitions of Thread, output_with_separator, and print_view were
between the main argparse and view-printing code. Group them together
with our existing read_config at the top of the module, which makes
for easier reading in the main section.
I also:
* Made 'headers' a print_view argument instead of a module-level
global. The list -> tuple conversion avoids having a mutable
default argument, which makes some people jumpy ;).
* Made 'db' a print_view argument instead of relying on the global
namespace to access it from print_view.
Now the suggested usage (listed by 'nmbug-status --help') is:
usage: nmbug-status [-h] [--text] [--config PATH] [--list-views]
[--get-query VIEW]
instead of the less obvious:
usage: nmbug-status [-h] [--text] [--config CONFIG] [--list-views]
[--get-query GET_QUERY]
Avoid:
$ ./nmbug-status --list-views
Traceback (most recent call last):
File "./nmbug-status", line 47, in <module>
'cat-file', 'blob', sha1+':status-config.json'],
TypeError: can't concat bytes to str
by explicitly converting the byte-stream read from Popen into a
Unicode string. On Python 2, this conversion is str -> unicode; on
Python 3 it is bytes -> str.
_ENCODING is derived from the user's locale (or system default) in an
attempt to match Git's output encoding. It may be more robust to skip
the encoding/decoding by using a Python wrapper like pygit2 [1] for
Git access. That's a fairly heavy dependency though, and using the
locale will probably work.
[1]: http://www.pygit2.org/
There seems to be consensus to use presence in contrib as
documentation of limited support by the notmuch developers; in fact
nmbug is pretty integrated into our current development process, so
devel seems more appropriate.
2013-02-16 07:54:33 -04:00
Renamed from contrib/nmbug/nmbug-status (Browse further)
