Xapian provides an interator-based interface to all search results.
So it was natural to make notmuch_messages_t be iterator-based as
well. Which we did originally.
But we ran into a problem when we added two APIs, (_get_replies and
_get_toplevel_messages), that want to return a messages iterator
that's *not* based on a Xapian search result. My original compromise
was to use notmuch_message_list_t as the basis for all returned
messages iterators in the public interface.
This had the problem of introducing extra latency at the beginning
of a search for messages, (the call would block while iterating over
all results from Xapian, converting to a message list).
In this commit, we remove that initial conversion and instead provide
two alternate implementations of notmuch_messages_t (one on top of a
Xapian iterator and one on top of a message list).
With this change, I tested a "notmuch search" returning *many* results
as previously taking about 7 seconds before results started appearing,
and now taking only 2 seconds.
Previously, notmuch_query_search_threads would do all the work, so the
caller would block until all results were processed. Now, we do the
work as we go, as the caller iterates with notmuch_threads_next. This
means that once results start coming back from "notmuch search" they
just keep continually streaming.
There's still some initial blocking before the first results appear
because the notmuch_messages_t object has the same bug (for now).
This was a poor workaround around the fact that the existing
notmuch_threads_t object is implemented poorly. It's got a fine
iterartor-based interface, but the implementation does all of the
work up-front in _create rather than doing the work incrementally
while iterating.
So to start fixing this, first get rid of all the hacks we had working
around this. This drops the --first and --max-threads options from the
search command, (but hopefully nobody was using them
anyway---notmuch.el certainly wasn't).
The domain is alway case insensitive, but in principle the username is
case sensitive. Few systems actually enforce this so I think a good
default is to treat the entire address as case insensitive, it will
eliminate a lot of superfluous self-addressed messages and reply from
the correct address in these cases.
Signed-off-by: Jed Brown <jed@59A2.org>
This way, the user gets a steady (but bursty) stream of reults. We
double the chunk size each time since each successive chunk has to
redo work from all previous chunks.
Of course, the overall time is thereby slower, as the price we pay for
increased responsiveness. With a search returning about 17000 thread
results I measured a total time of 48.8 seconds before this change and
58.4 seconds afterwards.
The rudimentary aspect here is that the date ranges are specified with
UNIX timestamp values (number of seconds since 1970-01-01 UTC). One
thing that can help here is using the date program to determins
timestamps, such as:
$(date +%s -d 2009-10-01)..$(date +%s)
Long-term, we'll probably need to do our own query parsing to be able
to support directly-specified dates and also relative expressions like
"since:'2 months ago'".
Folder mode takes a (user-configurable) list of search patterns and
produces a list of those patterns and the count of messages that they
match. When an entry in this list is selected, a search window with
the defined search is opened. The set of folders is defined as a
list, each element contains the name of the folder and the query string
to count.
Signed-off-by: Keith Packard <keithp@keithp.com>
Getting the count of matching threads or messages is a fairly
expensive operation. Xapian provides a very efficient mechanism that
returns an approximate value, so use that for this new command.
This returns the number of matching messages, not threads, as that is
cheap to compute.
Signed-off-by: Keith Packard <keithp@keithp.com>
Selecting text in the search view isn't all that useful, so instead,
make mouse-1 clicks actually show the thread you click on. It's almost
like direct manipulation or something.
Signed-off-by: Keith Packard <keithp@keithp.com>
When running "notmuch new --verbose", ANSI escapes are used. This may not be
desirable when the output of the command is *not* being sent to a terminal
(e.g. when piping output into another command). In that case each file
processed is printed in a new line and ANSI escapes are not used at all.
I configured my database.path with a trailing /, and after running notmuch
new every notmuch search would fail with error messages like this:
Error opening /inbox/cur/1258565257.000211.mbox:2,S: No such file or directory
The actual bug was in the filename normalization for storage in the
database. The database.path was removed from the full filename, but if
the database.path from the config file contained a trailing /, the
relative file name would retain an extra leading /... which made it look
like an absolute path after it was read out from the DB.
Signed-off-by: Bart Trojanowski <bart@jukie.net>
I had these notes sitting in an uncommitted file that was cluttering
up my "git status" output. This cleans that up, and also shares the
ideas with the wider community.
This ensures that make clean always proceeds, even if the user
accidentally creates a file named 'clean'. Also, it ignores errors in
rm and other commands.
Signed-off-by: Jan Janak <jan@ryngle.com>
When you compose a new message, message mode carefully inserts your
mail signature at the bottom of the message; as notmuch constructs the
reply all by itself, this doesn't happen then. Use the message mode
function 'message-insert-signature' to add that to reply buffers.
Signed-off-by: Keith Packard <keithp@keithp.com>
For very large mail boxes, it is desirable to know which files are being
processed e.g. when a crash occurs to know which one was the cause. Also,
it may be interesting to have a better idea of how the operation is
progressing when processing mailboxes with big messages.
This patch adds support for printing messages as they are processed by
"notmuch new":
* The "new" command now supports a "--verbose" flag.
* When running in verbose mode, the file path of the message about to be
processed is printed in the following format:
current/total: /path/to/message/file
Where "current" is the number of messages processed so far and "total" is
the total count of files to be processed.
The status line is erased using an ANSI sequence "\033[K" (erase current
line from the cursor to the end of line) each time it is refreshed. This
should not pose a problem because nearly every terminal supports it.
* The signal handler for SIGALRM and the timer are not enabled when running
in verbose mode, because we are already printing progress with each file,
periodical reports are not neccessary.
Pass the message through the charset filter so that we can view
messages wrote in different charset encoding.
Signed-off-by: Kan-Ru Chen <kanru@kanru.info>
Check that the stdout is connected to an interactive terminal with
isatty() before installing the periodic timer to print progress reports.
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
I felt sorry for Carl trying to step through an exception from xapian
and suffering from the SIGALARMs..
We can detect if the user launched notmuch under a debugger by either
checking our cmdline for the presence of the gdb string or querying if
valgrind is controlling our process. For the latter we need to add a
compile time check for the valgrind development library, and so add the
initial support to build Makefile.config from configure.
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Reviewed-by: Carl Worth <cworth@cworth.org>
[ickle: And do not install the timer when under the debugger]
The rule here was written to assume that if the GZIP environment
variable was set that it would be the gzip binary to execute,
(similar to the CC and CXX variables). But GZIP is actually used
to pass arguments to gzip, so we have to use a different name.
Use the facilities of GNU make to create a magic function that will
on the first invocation print a description of how to enable verbose
compile lines and then print the quiet rule.
Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: Carl Worth <cworth@cworth.org>
Cc: Mikhail Gusarov <dottedmag@dottedmag.net>
[ickle: Rebased, and duplicate command string eliminated.]
[ickle: Fixed verbose bug pointed out by Mikhail]
Since Xapian has a limit on the maximum length of a term, we have
to check for that before trying to add the message ID as a term.
This fixes the bug reported by Mike Hommey here:
<20091120132625.GA19246@glandium.org>
I've also constructed 20 files with a range of message ID lengths
centered around the Xapian term-length limit which I'll use to seed a
new test suite soon.
If an earlier exception occurred, then it's not unexpected for the
flush to fail as well. So in that case, we'll silently catch the
exception. Otherwise, make some noise about things going wrong at the
time of flush.
While talloc is great we need to free the g_error by hand.
Tested-by: Stefan Schmidt <stefan@datenfreihafen.org>
Signed-off-by: Holger Freyther <zecke@selfish.org>
