debian/notmuch
1
0
Fork 0
mirror of https://git.notmuchmail.org/git/notmuch synced 2024-11-21 18:38:08 +01:00
Code Issues Projects Releases Packages Wiki Activity Actions
notmuch/lib/query.cc

459 lines
12 KiB
C++
Raw Normal View History

Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
/* query.cc - Support for searching a notmuch database
*
* Copyright © 2009 Carl Worth
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program. If not, see http://www.gnu.org/licenses/ .
*
* Author: Carl Worth <cworth@cworth.org>
*/
#include "notmuch-private.h"
#include "database-private.h"
Add an initial implementation of a notmuch_thread_t object. We've now got a new notmuch_query_search_threads and a notmuch_threads_result_t iterator. The thread object itself doesn't do much yet, (just allows one to get the thread_id), but that's at least enough to see that "notmuch search" is actually doing something now, (since it has been converted to print thread IDs instead of message IDs). And maybe that's all we need. Getting the messages belonging to a thread is as simple as a notmuch_query_search_messages with a string of "thread:<thread-id>". Though it would be convenient to add notmuch_thread_get_messages which could use the existing notmuch_message_results_t iterator. Now we just need an implementation of "notmuch show" and we'll have something somewhat usable.
2009-10-26 07:12:20 +01:00
#include <glib.h> /* GHashTable, GPtrArray */
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
struct _notmuch_query {
notmuch_database_t *notmuch;
const char *query_string;
notmuch_sort_t sort;
};
lib/messages.c: Make message searches stream as well. Xapian provides an interator-based interface to all search results. So it was natural to make notmuch_messages_t be iterator-based as well. Which we did originally. But we ran into a problem when we added two APIs, (_get_replies and _get_toplevel_messages), that want to return a messages iterator that's *not* based on a Xapian search result. My original compromise was to use notmuch_message_list_t as the basis for all returned messages iterators in the public interface. This had the problem of introducing extra latency at the beginning of a search for messages, (the call would block while iterating over all results from Xapian, converting to a message list). In this commit, we remove that initial conversion and instead provide two alternate implementations of notmuch_messages_t (one on top of a Xapian iterator and one on top of a message list). With this change, I tested a "notmuch search" returning *many* results as previously taking about 7 seconds before results started appearing, and now taking only 2 seconds.
2009-11-24 06:47:24 +01:00
typedef struct _notmuch_mset_messages {
notmuch_messages_t base;
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
notmuch_database_t *notmuch;
notmuch dump: Fix the sorting of results. To properly support sorting in notmuch_query we know use an Enquire object. We also throw in a QueryParser too, so we're really close to being able to support arbitrary full-text searches. I took a look at the supported QueryParser syntax and chose a set of flags for everything I like, (such as supporting Boolean operators in either case ("AND" or "and"), supporting phrase searching, supporting + and - to include/preclude terms, and supporting a trailing * on any term as a wildcard).
2009-10-21 09:35:56 +02:00
Xapian::MSetIterator iterator;
Xapian::MSetIterator iterator_end;
lib/messages.c: Make message searches stream as well. Xapian provides an interator-based interface to all search results. So it was natural to make notmuch_messages_t be iterator-based as well. Which we did originally. But we ran into a problem when we added two APIs, (_get_replies and _get_toplevel_messages), that want to return a messages iterator that's *not* based on a Xapian search result. My original compromise was to use notmuch_message_list_t as the basis for all returned messages iterators in the public interface. This had the problem of introducing extra latency at the beginning of a search for messages, (the call would block while iterating over all results from Xapian, converting to a message list). In this commit, we remove that initial conversion and instead provide two alternate implementations of notmuch_messages_t (one on top of a Xapian iterator and one on top of a message list). With this change, I tested a "notmuch search" returning *many* results as previously taking about 7 seconds before results started appearing, and now taking only 2 seconds.
2009-11-24 06:47:24 +01:00
} notmuch_mset_messages_t;
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
struct _notmuch_doc_id_set {
unsigned int *bitmap;
unsigned int bound;
};
Rename message_results/thread_results to messages/threads. Shorter naming without being any less clear. A definite win.
2009-11-01 00:32:30 +01:00
struct _notmuch_threads {
lib/query: Fix notmuch_threads_t to stream results rather than blocking. Previously, notmuch_query_search_threads would do all the work, so the caller would block until all results were processed. Now, we do the work as we go, as the caller iterates with notmuch_threads_next. This means that once results start coming back from "notmuch search" they just keep continually streaming. There's still some initial blocking before the first results appear because the notmuch_messages_t object has the same bug (for now).
2009-11-24 05:18:57 +01:00
notmuch_query_t *query;
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
/* The ordered list of doc ids matched by the query. */
GArray *doc_ids;
/* Our iterator's current position in doc_ids. */
unsigned int doc_id_pos;
/* The set of matched docid's that have not been assigned to a
* thread. Initially, this contains every docid in doc_ids. */
notmuch_doc_id_set_t match_set;
Add an initial implementation of a notmuch_thread_t object. We've now got a new notmuch_query_search_threads and a notmuch_threads_result_t iterator. The thread object itself doesn't do much yet, (just allows one to get the thread_id), but that's at least enough to see that "notmuch search" is actually doing something now, (since it has been converted to print thread IDs instead of message IDs). And maybe that's all we need. Getting the messages belonging to a thread is as simple as a notmuch_query_search_messages with a string of "thread:<thread-id>". Though it would be convenient to add notmuch_thread_get_messages which could use the existing notmuch_message_results_t iterator. Now we just need an implementation of "notmuch show" and we'll have something somewhat usable.
2009-10-26 07:12:20 +01:00
};
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
notmuch_query_t *
notmuch_query_create (notmuch_database_t *notmuch,
const char *query_string)
{
notmuch_query_t *query;
Add debugging code for examining query strings. It's nice that Xapian provides a little function to print a textual representation of the entire query tree. So now, if you compile like so: make CFLAGS=-DDEBUG_QUERY then you get a nice output of the query string received by the query module, and the final query actually being sent to Xapian.
2009-10-25 06:18:20 +01:00
#ifdef DEBUG_QUERY
fprintf (stderr, "Query string is:\n%s\n", query_string);
#endif
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
query = talloc (NULL, notmuch_query_t);
if (unlikely (query == NULL))
return NULL;
query->notmuch = notmuch;
query: Remove the magic NOTMUCH_QUERY_ALL Using the address of a static char* was clever, but really unnecessary. An empty string is much less magic, and even easier to understand as the way to query everything from the database.
2009-10-21 07:40:37 +02:00
query->query_string = talloc_strdup (query, query_string);
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
notmuch search: Change default search order to be newest messages first. This is what most people want for a _search_ command. It's often different for actually reading mail in an inbox, (where it makes more sense to have results displayed in chronological order), but in such a case, ther user is likely using an interface that can simply pass the --sort=oldest-first option to "notmuch search". Here we're also change the sort enum from NOTMUCH_SORT_DATE and NOTMUCH_SORT_DATE_REVERSE to NOTMUCH_SORT_OLDEST_FIRST and NOTMUCH_SORT_NEWEST_FIRST. Similarly we replace the --reverse option to "notmuch search" with two options: --sort=oldest-first and --sort=newest-first. Finally, these changes are all tracked in the emacs interface, (which has no change in its behavior).
2009-11-18 05:52:09 +01:00
query->sort = NOTMUCH_SORT_NEWEST_FIRST;
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
return query;
}
lib: Add two functions: notmuch_query_get_query_string and _get_sort It can be handy to be able to query these settings from an existing query object.
2010-10-28 19:30:26 +02:00
const char *
notmuch_query_get_query_string (notmuch_query_t *query)
{
return query->query_string;
}
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
void
notmuch_query_set_sort (notmuch_query_t *query, notmuch_sort_t sort)
{
query->sort = sort;
}
lib: Add two functions: notmuch_query_get_query_string and _get_sort It can be handy to be able to query these settings from an existing query object.
2010-10-28 19:30:26 +02:00
notmuch_sort_t
notmuch_query_get_sort (notmuch_query_t *query)
{
return query->sort;
}
lib/messages.c: Make message searches stream as well. Xapian provides an interator-based interface to all search results. So it was natural to make notmuch_messages_t be iterator-based as well. Which we did originally. But we ran into a problem when we added two APIs, (_get_replies and _get_toplevel_messages), that want to return a messages iterator that's *not* based on a Xapian search result. My original compromise was to use notmuch_message_list_t as the basis for all returned messages iterators in the public interface. This had the problem of introducing extra latency at the beginning of a search for messages, (the call would block while iterating over all results from Xapian, converting to a message list). In this commit, we remove that initial conversion and instead provide two alternate implementations of notmuch_messages_t (one on top of a Xapian iterator and one on top of a message list). With this change, I tested a "notmuch search" returning *many* results as previously taking about 7 seconds before results started appearing, and now taking only 2 seconds.
2009-11-24 06:47:24 +01:00
/* We end up having to call the destructors explicitly because we had
* to use "placement new" in order to initialize C++ objects within a
* block that we allocated with talloc. So C++ is making talloc
* slightly less simple to use, (we wouldn't need
* talloc_set_destructor at all otherwise).
*/
static int
_notmuch_messages_destructor (notmuch_mset_messages_t *messages)
{
messages->iterator.~MSetIterator ();
messages->iterator_end.~MSetIterator ();
return 0;
}
Rename message_results/thread_results to messages/threads. Shorter naming without being any less clear. A definite win.
2009-11-01 00:32:30 +01:00
notmuch_messages_t *
lib/query: Drop the first and max_messages arguments from search_messages. These only existed to support the chunky-searching hack, but that was recently dropped anyway.
2009-11-24 05:25:13 +01:00
notmuch_query_search_messages (notmuch_query_t *query)
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
{
notmuch dump: Fix the sorting of results. To properly support sorting in notmuch_query we know use an Enquire object. We also throw in a QueryParser too, so we're really close to being able to support arbitrary full-text searches. I took a look at the supported QueryParser syntax and chose a set of flags for everything I like, (such as supporting Boolean operators in either case ("AND" or "and"), supporting phrase searching, supporting + and - to include/preclude terms, and supporting a trailing * on any term as a wildcard).
2009-10-21 09:35:56 +02:00
notmuch_database_t *notmuch = query->notmuch;
const char *query_string = query->query_string;
lib/messages.c: Make message searches stream as well. Xapian provides an interator-based interface to all search results. So it was natural to make notmuch_messages_t be iterator-based as well. Which we did originally. But we ran into a problem when we added two APIs, (_get_replies and _get_toplevel_messages), that want to return a messages iterator that's *not* based on a Xapian search result. My original compromise was to use notmuch_message_list_t as the basis for all returned messages iterators in the public interface. This had the problem of introducing extra latency at the beginning of a search for messages, (the call would block while iterating over all results from Xapian, converting to a message list). In this commit, we remove that initial conversion and instead provide two alternate implementations of notmuch_messages_t (one on top of a Xapian iterator and one on top of a message list). With this change, I tested a "notmuch search" returning *many* results as previously taking about 7 seconds before results started appearing, and now taking only 2 seconds.
2009-11-24 06:47:24 +01:00
notmuch_mset_messages_t *messages;
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
lib/messages.c: Make message searches stream as well. Xapian provides an interator-based interface to all search results. So it was natural to make notmuch_messages_t be iterator-based as well. Which we did originally. But we ran into a problem when we added two APIs, (_get_replies and _get_toplevel_messages), that want to return a messages iterator that's *not* based on a Xapian search result. My original compromise was to use notmuch_message_list_t as the basis for all returned messages iterators in the public interface. This had the problem of introducing extra latency at the beginning of a search for messages, (the call would block while iterating over all results from Xapian, converting to a message list). In this commit, we remove that initial conversion and instead provide two alternate implementations of notmuch_messages_t (one on top of a Xapian iterator and one on top of a message list). With this change, I tested a "notmuch search" returning *many* results as previously taking about 7 seconds before results started appearing, and now taking only 2 seconds.
2009-11-24 06:47:24 +01:00
messages = talloc (query, notmuch_mset_messages_t);
if (unlikely (messages == NULL))
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
return NULL;
try {
lib/messages.c: Make message searches stream as well. Xapian provides an interator-based interface to all search results. So it was natural to make notmuch_messages_t be iterator-based as well. Which we did originally. But we ran into a problem when we added two APIs, (_get_replies and _get_toplevel_messages), that want to return a messages iterator that's *not* based on a Xapian search result. My original compromise was to use notmuch_message_list_t as the basis for all returned messages iterators in the public interface. This had the problem of introducing extra latency at the beginning of a search for messages, (the call would block while iterating over all results from Xapian, converting to a message list). In this commit, we remove that initial conversion and instead provide two alternate implementations of notmuch_messages_t (one on top of a Xapian iterator and one on top of a message list). With this change, I tested a "notmuch search" returning *many* results as previously taking about 7 seconds before results started appearing, and now taking only 2 seconds.
2009-11-24 06:47:24 +01:00
messages->base.is_of_list_type = FALSE;
messages->base.iterator = NULL;
messages->notmuch = notmuch;
new (&messages->iterator) Xapian::MSetIterator ();
new (&messages->iterator_end) Xapian::MSetIterator ();
talloc_set_destructor (messages, _notmuch_messages_destructor);
notmuch dump: Fix the sorting of results. To properly support sorting in notmuch_query we know use an Enquire object. We also throw in a QueryParser too, so we're really close to being able to support arbitrary full-text searches. I took a look at the supported QueryParser syntax and chose a set of flags for everything I like, (such as supporting Boolean operators in either case ("AND" or "and"), supporting phrase searching, supporting + and - to include/preclude terms, and supporting a trailing * on any term as a wildcard).
2009-10-21 09:35:56 +02:00
Xapian::Enquire enquire (*notmuch->xapian_db);
Use _find_prefix instead of hard-coded term in notmuch_query_search I'm planning to change prefix values soon, which would break code like this. So eliminate the fragility by going through our existing _find_prefix function.
2009-10-25 06:21:57 +01:00
Xapian::Query mail_query (talloc_asprintf (query, "%s%s",
_find_prefix ("type"),
"mail"));
notmuch dump: Fix the sorting of results. To properly support sorting in notmuch_query we know use an Enquire object. We also throw in a QueryParser too, so we're really close to being able to support arbitrary full-text searches. I took a look at the supported QueryParser syntax and chose a set of flags for everything I like, (such as supporting Boolean operators in either case ("AND" or "and"), supporting phrase searching, supporting + and - to include/preclude terms, and supporting a trailing * on any term as a wildcard).
2009-10-21 09:35:56 +02:00
Xapian::Query string_query, final_query;
Xapian::MSet mset;
Fix bit-twiddling brain damage in notmuch_query_search Here's the big bug that was preventing any searches from working at all like desired. I did the work to carefully pick out exactly the flags that I wanted, and then I threw it away by trying to combine them with & instead of | (so just passing 0 for flags instead). Much better now.
2009-10-25 06:20:13 +01:00
unsigned int flags = (Xapian::QueryParser::FLAG_BOOLEAN |
Xapian::QueryParser::FLAG_PHRASE |
Xapian::QueryParser::FLAG_LOVEHATE |
Xapian::QueryParser::FLAG_BOOLEAN_ANY_CASE |
Allow lone "not" search operators As suggested by Keith in FLAG_PURE_NOT allows for expressions like: notmuch search NOT tag:inbox Note that this way a search like: notmuch search foobar NOT tag:inbox should not be written instead: notmuch search foobar AND NOT tag:inbox In my opinion, the latter feels more natural and is somewhat more explicit. It gives a better clue of what the search is about instead of assuming that an implicit AND operator is there.
2009-11-19 01:07:22 +01:00
Xapian::QueryParser::FLAG_WILDCARD |
Xapian::QueryParser::FLAG_PURE_NOT);
notmuch dump: Fix the sorting of results. To properly support sorting in notmuch_query we know use an Enquire object. We also throw in a QueryParser too, so we're really close to being able to support arbitrary full-text searches. I took a look at the supported QueryParser syntax and chose a set of flags for everything I like, (such as supporting Boolean operators in either case ("AND" or "and"), supporting phrase searching, supporting + and - to include/preclude terms, and supporting a trailing * on any term as a wildcard).
2009-10-21 09:35:56 +02:00
lib: Handle "*" as a query string to match all messages. This seems like a generally useful thing to support, (but the previous support through an empty string was not convenient for some users, (such as the command-line client).
2010-04-10 01:40:31 +02:00
if (strcmp (query_string, "") == 0 ||
strcmp (query_string, "*") == 0)
{
notmuch dump: Fix the sorting of results. To properly support sorting in notmuch_query we know use an Enquire object. We also throw in a QueryParser too, so we're really close to being able to support arbitrary full-text searches. I took a look at the supported QueryParser syntax and chose a set of flags for everything I like, (such as supporting Boolean operators in either case ("AND" or "and"), supporting phrase searching, supporting + and - to include/preclude terms, and supporting a trailing * on any term as a wildcard).
2009-10-21 09:35:56 +02:00
final_query = mail_query;
} else {
string_query = notmuch->query_parser->
parse_query (query_string, flags);
final_query = Xapian::Query (Xapian::Query::OP_AND,
mail_query, string_query);
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
}
lib: Explicitly set BoolWeight when searching. All notmuch searches currently sort by value (either date or message ID) so it's just wasted effort for Xapian to compute relevance values for each result. We now explicitly tell Xapian that we're uninterested in the relevance values.
2010-01-09 20:16:40 +01:00
enquire.set_weighting_scheme (Xapian::BoolWeight());
notmuch dump: Fix the sorting of results. To properly support sorting in notmuch_query we know use an Enquire object. We also throw in a QueryParser too, so we're really close to being able to support arbitrary full-text searches. I took a look at the supported QueryParser syntax and chose a set of flags for everything I like, (such as supporting Boolean operators in either case ("AND" or "and"), supporting phrase searching, supporting + and - to include/preclude terms, and supporting a trailing * on any term as a wildcard).
2009-10-21 09:35:56 +02:00
switch (query->sort) {
notmuch search: Change default search order to be newest messages first. This is what most people want for a _search_ command. It's often different for actually reading mail in an inbox, (where it makes more sense to have results displayed in chronological order), but in such a case, ther user is likely using an interface that can simply pass the --sort=oldest-first option to "notmuch search". Here we're also change the sort enum from NOTMUCH_SORT_DATE and NOTMUCH_SORT_DATE_REVERSE to NOTMUCH_SORT_OLDEST_FIRST and NOTMUCH_SORT_NEWEST_FIRST. Similarly we replace the --reverse option to "notmuch search" with two options: --sort=oldest-first and --sort=newest-first. Finally, these changes are all tracked in the emacs interface, (which has no change in its behavior).
2009-11-18 05:52:09 +01:00
case NOTMUCH_SORT_OLDEST_FIRST:
Shuffle the value numbers around in the database. First, it's nice that for now we don't have any users yet, so we can make incompatible changes to the database layout like this without causing trouble. ;-) There are a few reasons for this change. First, we now use value 0 uniformly as a timestamp for both mail and timestamp documents, (which lets us cleanup an ugly and fragile bare 0 in the add_value and get_value calls in the timestamp code). Second, I want to drop the thread value entirely, so putting it at the end of the list means we can drop it as compatible change in the future. (I almost want to drop the message-ID value too, but it's nice to be able to sort on it to get diff-able output from "notmuch dump".) But the thread value we never use as a value, (we would never sort on it, for example). And it's totally redundant with the thread terms we store already. So expect it to disappear soon.
2009-10-25 07:05:08 +01:00
enquire.set_sort_by_value (NOTMUCH_VALUE_TIMESTAMP, FALSE);
notmuch dump: Fix the sorting of results. To properly support sorting in notmuch_query we know use an Enquire object. We also throw in a QueryParser too, so we're really close to being able to support arbitrary full-text searches. I took a look at the supported QueryParser syntax and chose a set of flags for everything I like, (such as supporting Boolean operators in either case ("AND" or "and"), supporting phrase searching, supporting + and - to include/preclude terms, and supporting a trailing * on any term as a wildcard).
2009-10-21 09:35:56 +02:00
break;
notmuch search: Change default search order to be newest messages first. This is what most people want for a _search_ command. It's often different for actually reading mail in an inbox, (where it makes more sense to have results displayed in chronological order), but in such a case, ther user is likely using an interface that can simply pass the --sort=oldest-first option to "notmuch search". Here we're also change the sort enum from NOTMUCH_SORT_DATE and NOTMUCH_SORT_DATE_REVERSE to NOTMUCH_SORT_OLDEST_FIRST and NOTMUCH_SORT_NEWEST_FIRST. Similarly we replace the --reverse option to "notmuch search" with two options: --sort=oldest-first and --sort=newest-first. Finally, these changes are all tracked in the emacs interface, (which has no change in its behavior).
2009-11-18 05:52:09 +01:00
case NOTMUCH_SORT_NEWEST_FIRST:
Shuffle the value numbers around in the database. First, it's nice that for now we don't have any users yet, so we can make incompatible changes to the database layout like this without causing trouble. ;-) There are a few reasons for this change. First, we now use value 0 uniformly as a timestamp for both mail and timestamp documents, (which lets us cleanup an ugly and fragile bare 0 in the add_value and get_value calls in the timestamp code). Second, I want to drop the thread value entirely, so putting it at the end of the list means we can drop it as compatible change in the future. (I almost want to drop the message-ID value too, but it's nice to be able to sort on it to get diff-able output from "notmuch dump".) But the thread value we never use as a value, (we would never sort on it, for example). And it's totally redundant with the thread terms we store already. So expect it to disappear soon.
2009-10-25 07:05:08 +01:00
enquire.set_sort_by_value (NOTMUCH_VALUE_TIMESTAMP, TRUE);
notmuch dump: Fix the sorting of results. To properly support sorting in notmuch_query we know use an Enquire object. We also throw in a QueryParser too, so we're really close to being able to support arbitrary full-text searches. I took a look at the supported QueryParser syntax and chose a set of flags for everything I like, (such as supporting Boolean operators in either case ("AND" or "and"), supporting phrase searching, supporting + and - to include/preclude terms, and supporting a trailing * on any term as a wildcard).
2009-10-21 09:35:56 +02:00
break;
case NOTMUCH_SORT_MESSAGE_ID:
enquire.set_sort_by_value (NOTMUCH_VALUE_MESSAGE_ID, FALSE);
break;
query.cc: allow to return query results unsorted Previously, we always sorted the returned results by some string value, (newest-to-oldest by default), however in some cases (as when applying tags to a search result) we are not interested in any special order. This introduces a NOTMUCH_SORT_UNSORTED value that does just that. It is not used at the moment anywhere in the code. Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2010-04-16 08:38:46 +02:00
case NOTMUCH_SORT_UNSORTED:
break;
notmuch dump: Fix the sorting of results. To properly support sorting in notmuch_query we know use an Enquire object. We also throw in a QueryParser too, so we're really close to being able to support arbitrary full-text searches. I took a look at the supported QueryParser syntax and chose a set of flags for everything I like, (such as supporting Boolean operators in either case ("AND" or "and"), supporting phrase searching, supporting + and - to include/preclude terms, and supporting a trailing * on any term as a wildcard).
2009-10-21 09:35:56 +02:00
}
Add debugging code for examining query strings. It's nice that Xapian provides a little function to print a textual representation of the entire query tree. So now, if you compile like so: make CFLAGS=-DDEBUG_QUERY then you get a nice output of the query string received by the query module, and the final query actually being sent to Xapian.
2009-10-25 06:18:20 +01:00
#if DEBUG_QUERY
fprintf (stderr, "Final query is:\n%s\n", final_query.get_description().c_str());
#endif
notmuch dump: Fix the sorting of results. To properly support sorting in notmuch_query we know use an Enquire object. We also throw in a QueryParser too, so we're really close to being able to support arbitrary full-text searches. I took a look at the supported QueryParser syntax and chose a set of flags for everything I like, (such as supporting Boolean operators in either case ("AND" or "and"), supporting phrase searching, supporting + and - to include/preclude terms, and supporting a trailing * on any term as a wildcard).
2009-10-21 09:35:56 +02:00
enquire.set_query (final_query);
lib/query: Drop the first and max_messages arguments from search_messages. These only existed to support the chunky-searching hack, but that was recently dropped anyway.
2009-11-24 05:25:13 +01:00
mset = enquire.get_mset (0, notmuch->xapian_db->get_doccount ());
notmuch dump: Fix the sorting of results. To properly support sorting in notmuch_query we know use an Enquire object. We also throw in a QueryParser too, so we're really close to being able to support arbitrary full-text searches. I took a look at the supported QueryParser syntax and chose a set of flags for everything I like, (such as supporting Boolean operators in either case ("AND" or "and"), supporting phrase searching, supporting + and - to include/preclude terms, and supporting a trailing * on any term as a wildcard).
2009-10-21 09:35:56 +02:00
lib/messages.c: Make message searches stream as well. Xapian provides an interator-based interface to all search results. So it was natural to make notmuch_messages_t be iterator-based as well. Which we did originally. But we ran into a problem when we added two APIs, (_get_replies and _get_toplevel_messages), that want to return a messages iterator that's *not* based on a Xapian search result. My original compromise was to use notmuch_message_list_t as the basis for all returned messages iterators in the public interface. This had the problem of introducing extra latency at the beginning of a search for messages, (the call would block while iterating over all results from Xapian, converting to a message list). In this commit, we remove that initial conversion and instead provide two alternate implementations of notmuch_messages_t (one on top of a Xapian iterator and one on top of a message list). With this change, I tested a "notmuch search" returning *many* results as previously taking about 7 seconds before results started appearing, and now taking only 2 seconds.
2009-11-24 06:47:24 +01:00
messages->iterator = mset.begin ();
messages->iterator_end = mset.end ();
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
lib: Ensure notmuch_query_search_messages returns NULL on an exception. Previously, this function may have segfaulted immediately after reporting the exception.
2010-04-24 16:25:12 +02:00
return &messages->base;
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
} catch (const Xapian::Error &error) {
Print information about where Xapian exception occurred. Previously, our Xapian exception reports where identical so they were hard to track down.
2009-11-22 03:41:30 +01:00
fprintf (stderr, "A Xapian exception occurred performing query: %s\n",
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
error.get_msg().c_str());
When a search query triggers a Xapian exception, log what the query was. In my script containing a series of queries to be run on new mail for setting up tags, it's nice to see which query I typed wrong. Signed-off-by: Eric Anholt <eric@anholt.net>
2009-11-20 14:00:42 +01:00
fprintf (stderr, "Query string was: %s\n", query->query_string);
Catch and optionally print about exception at database->flush. If an earlier exception occurred, then it's not unexpected for the flush to fail as well. So in that case, we'll silently catch the exception. Otherwise, make some noise about things going wrong at the time of flush.
2009-11-22 03:54:20 +01:00
notmuch->exception_reported = TRUE;
lib: Ensure notmuch_query_search_messages returns NULL on an exception. Previously, this function may have segfaulted immediately after reporting the exception.
2010-04-24 16:25:12 +02:00
talloc_free (messages);
return NULL;
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
}
lib/messages.c: Make message searches stream as well. Xapian provides an interator-based interface to all search results. So it was natural to make notmuch_messages_t be iterator-based as well. Which we did originally. But we ran into a problem when we added two APIs, (_get_replies and _get_toplevel_messages), that want to return a messages iterator that's *not* based on a Xapian search result. My original compromise was to use notmuch_message_list_t as the basis for all returned messages iterators in the public interface. This had the problem of introducing extra latency at the beginning of a search for messages, (the call would block while iterating over all results from Xapian, converting to a message list). In this commit, we remove that initial conversion and instead provide two alternate implementations of notmuch_messages_t (one on top of a Xapian iterator and one on top of a message list). With this change, I tested a "notmuch search" returning *many* results as previously taking about 7 seconds before results started appearing, and now taking only 2 seconds.
2009-11-24 06:47:24 +01:00
}
notmuch_bool_t
lib: Rename iterator functions to prepare for reverse iteration. We rename 'has_more' to 'valid' so that it can function whether iterating in a forward or reverse direction. We also rename 'advance' to 'move_to_next' to setup parallel naming with the proposed functions 'move_to_first', 'move_to_last', and 'move_to_previous'.
2010-03-09 18:22:29 +01:00
_notmuch_mset_messages_valid (notmuch_messages_t *messages)
lib/messages.c: Make message searches stream as well. Xapian provides an interator-based interface to all search results. So it was natural to make notmuch_messages_t be iterator-based as well. Which we did originally. But we ran into a problem when we added two APIs, (_get_replies and _get_toplevel_messages), that want to return a messages iterator that's *not* based on a Xapian search result. My original compromise was to use notmuch_message_list_t as the basis for all returned messages iterators in the public interface. This had the problem of introducing extra latency at the beginning of a search for messages, (the call would block while iterating over all results from Xapian, converting to a message list). In this commit, we remove that initial conversion and instead provide two alternate implementations of notmuch_messages_t (one on top of a Xapian iterator and one on top of a message list). With this change, I tested a "notmuch search" returning *many* results as previously taking about 7 seconds before results started appearing, and now taking only 2 seconds.
2009-11-24 06:47:24 +01:00
{
notmuch_mset_messages_t *mset_messages;
mset_messages = (notmuch_mset_messages_t *) messages;
return (mset_messages->iterator != mset_messages->iterator_end);
}
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
static Xapian::docid
_notmuch_mset_messages_get_doc_id (notmuch_messages_t *messages)
{
notmuch_mset_messages_t *mset_messages;
mset_messages = (notmuch_mset_messages_t *) messages;
if (! _notmuch_mset_messages_valid (&mset_messages->base))
return 0;
return *mset_messages->iterator;
}
lib/messages.c: Make message searches stream as well. Xapian provides an interator-based interface to all search results. So it was natural to make notmuch_messages_t be iterator-based as well. Which we did originally. But we ran into a problem when we added two APIs, (_get_replies and _get_toplevel_messages), that want to return a messages iterator that's *not* based on a Xapian search result. My original compromise was to use notmuch_message_list_t as the basis for all returned messages iterators in the public interface. This had the problem of introducing extra latency at the beginning of a search for messages, (the call would block while iterating over all results from Xapian, converting to a message list). In this commit, we remove that initial conversion and instead provide two alternate implementations of notmuch_messages_t (one on top of a Xapian iterator and one on top of a message list). With this change, I tested a "notmuch search" returning *many* results as previously taking about 7 seconds before results started appearing, and now taking only 2 seconds.
2009-11-24 06:47:24 +01:00
notmuch_message_t *
_notmuch_mset_messages_get (notmuch_messages_t *messages)
{
notmuch_message_t *message;
Xapian::docid doc_id;
notmuch_private_status_t status;
notmuch_mset_messages_t *mset_messages;
mset_messages = (notmuch_mset_messages_t *) messages;
lib: Rename iterator functions to prepare for reverse iteration. We rename 'has_more' to 'valid' so that it can function whether iterating in a forward or reverse direction. We also rename 'advance' to 'move_to_next' to setup parallel naming with the proposed functions 'move_to_first', 'move_to_last', and 'move_to_previous'.
2010-03-09 18:22:29 +01:00
if (! _notmuch_mset_messages_valid (&mset_messages->base))
lib/messages.c: Make message searches stream as well. Xapian provides an interator-based interface to all search results. So it was natural to make notmuch_messages_t be iterator-based as well. Which we did originally. But we ran into a problem when we added two APIs, (_get_replies and _get_toplevel_messages), that want to return a messages iterator that's *not* based on a Xapian search result. My original compromise was to use notmuch_message_list_t as the basis for all returned messages iterators in the public interface. This had the problem of introducing extra latency at the beginning of a search for messages, (the call would block while iterating over all results from Xapian, converting to a message list). In this commit, we remove that initial conversion and instead provide two alternate implementations of notmuch_messages_t (one on top of a Xapian iterator and one on top of a message list). With this change, I tested a "notmuch search" returning *many* results as previously taking about 7 seconds before results started appearing, and now taking only 2 seconds.
2009-11-24 06:47:24 +01:00
return NULL;
doc_id = *mset_messages->iterator;
message = _notmuch_message_create (mset_messages,
mset_messages->notmuch, doc_id,
&status);
if (message == NULL &&
status == NOTMUCH_PRIVATE_STATUS_NO_DOCUMENT_FOUND)
{
INTERNAL_ERROR ("a messages iterator contains a non-existent document ID.\n");
}
return message;
}
void
lib: Rename iterator functions to prepare for reverse iteration. We rename 'has_more' to 'valid' so that it can function whether iterating in a forward or reverse direction. We also rename 'advance' to 'move_to_next' to setup parallel naming with the proposed functions 'move_to_first', 'move_to_last', and 'move_to_previous'.
2010-03-09 18:22:29 +01:00
_notmuch_mset_messages_move_to_next (notmuch_messages_t *messages)
lib/messages.c: Make message searches stream as well. Xapian provides an interator-based interface to all search results. So it was natural to make notmuch_messages_t be iterator-based as well. Which we did originally. But we ran into a problem when we added two APIs, (_get_replies and _get_toplevel_messages), that want to return a messages iterator that's *not* based on a Xapian search result. My original compromise was to use notmuch_message_list_t as the basis for all returned messages iterators in the public interface. This had the problem of introducing extra latency at the beginning of a search for messages, (the call would block while iterating over all results from Xapian, converting to a message list). In this commit, we remove that initial conversion and instead provide two alternate implementations of notmuch_messages_t (one on top of a Xapian iterator and one on top of a message list). With this change, I tested a "notmuch search" returning *many* results as previously taking about 7 seconds before results started appearing, and now taking only 2 seconds.
2009-11-24 06:47:24 +01:00
{
notmuch_mset_messages_t *mset_messages;
mset_messages = (notmuch_mset_messages_t *) messages;
mset_messages->iterator++;
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
}
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
static notmuch_bool_t
_notmuch_doc_id_set_init (void *ctx,
notmuch_doc_id_set_t *doc_ids,
GArray *arr, unsigned int bound)
{
size_t count = (bound + sizeof (doc_ids->bitmap[0]) - 1) /
sizeof (doc_ids->bitmap[0]);
unsigned int *bitmap = talloc_zero_array (ctx, unsigned int, count);
if (bitmap == NULL)
return FALSE;
doc_ids->bitmap = bitmap;
doc_ids->bound = bound;
for (unsigned int i = 0; i < arr->len; i++) {
unsigned int doc_id = g_array_index(arr, unsigned int, i);
bitmap[doc_id / sizeof (bitmap[0])] |=
1 << (doc_id % sizeof (bitmap[0]));
}
return TRUE;
}
notmuch_bool_t
_notmuch_doc_id_set_contains (notmuch_doc_id_set_t *doc_ids,
unsigned int doc_id)
{
if (doc_id >= doc_ids->bound)
return FALSE;
return (doc_ids->bitmap[doc_id / sizeof (doc_ids->bitmap[0])] &
(1 << (doc_id % sizeof (doc_ids->bitmap[0])))) != 0;
}
void
_notmuch_doc_id_set_remove (notmuch_doc_id_set_t *doc_ids,
unsigned int doc_id)
{
if (doc_id < doc_ids->bound)
doc_ids->bitmap[doc_id / sizeof (doc_ids->bitmap[0])] &=
~(1 << (doc_id % sizeof (doc_ids->bitmap[0])));
}
Fix memory leak in notmuch_thread_results_t If we were using a talloc-based resizing array then this wouldn't have happened. Of course, thanks to valgrind for catching this.
2009-10-26 22:02:58 +01:00
/* Glib objects force use to use a talloc destructor as well, (but not
Rename message_results/thread_results to messages/threads. Shorter naming without being any less clear. A definite win.
2009-11-01 00:32:30 +01:00
* nearly as ugly as the for messages due to C++ objects). At
Fix memory leak in notmuch_thread_results_t If we were using a talloc-based resizing array then this wouldn't have happened. Of course, thanks to valgrind for catching this.
2009-10-26 22:02:58 +01:00
* this point, I'd really like to have some talloc-friendly
* equivalents for the few pieces of glib that I'm using. */
static int
Rename message_results/thread_results to messages/threads. Shorter naming without being any less clear. A definite win.
2009-11-01 00:32:30 +01:00
_notmuch_threads_destructor (notmuch_threads_t *threads)
Fix memory leak in notmuch_thread_results_t If we were using a talloc-based resizing array then this wouldn't have happened. Of course, thanks to valgrind for catching this.
2009-10-26 22:02:58 +01:00
{
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
if (threads->doc_ids)
g_array_unref (threads->doc_ids);
Fix memory leak in notmuch_thread_results_t If we were using a talloc-based resizing array then this wouldn't have happened. Of course, thanks to valgrind for catching this.
2009-10-26 22:02:58 +01:00
return 0;
}
Rename message_results/thread_results to messages/threads. Shorter naming without being any less clear. A definite win.
2009-11-01 00:32:30 +01:00
notmuch_threads_t *
notmuch search: Remove the chunked-searching hack. This was a poor workaround around the fact that the existing notmuch_threads_t object is implemented poorly. It's got a fine iterartor-based interface, but the implementation does all of the work up-front in _create rather than doing the work incrementally while iterating. So to start fixing this, first get rid of all the hacks we had working around this. This drops the --first and --max-threads options from the search command, (but hopefully nobody was using them anyway---notmuch.el certainly wasn't).
2009-11-24 05:12:57 +01:00
notmuch_query_search_threads (notmuch_query_t *query)
Add an initial implementation of a notmuch_thread_t object. We've now got a new notmuch_query_search_threads and a notmuch_threads_result_t iterator. The thread object itself doesn't do much yet, (just allows one to get the thread_id), but that's at least enough to see that "notmuch search" is actually doing something now, (since it has been converted to print thread IDs instead of message IDs). And maybe that's all we need. Getting the messages belonging to a thread is as simple as a notmuch_query_search_messages with a string of "thread:<thread-id>". Though it would be convenient to add notmuch_thread_get_messages which could use the existing notmuch_message_results_t iterator. Now we just need an implementation of "notmuch show" and we'll have something somewhat usable.
2009-10-26 07:12:20 +01:00
{
Rename message_results/thread_results to messages/threads. Shorter naming without being any less clear. A definite win.
2009-11-01 00:32:30 +01:00
notmuch_threads_t *threads;
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
notmuch_messages_t *messages;
Xapian::docid max_doc_id = 0;
Add an initial implementation of a notmuch_thread_t object. We've now got a new notmuch_query_search_threads and a notmuch_threads_result_t iterator. The thread object itself doesn't do much yet, (just allows one to get the thread_id), but that's at least enough to see that "notmuch search" is actually doing something now, (since it has been converted to print thread IDs instead of message IDs). And maybe that's all we need. Getting the messages belonging to a thread is as simple as a notmuch_query_search_messages with a string of "thread:<thread-id>". Though it would be convenient to add notmuch_thread_get_messages which could use the existing notmuch_message_results_t iterator. Now we just need an implementation of "notmuch show" and we'll have something somewhat usable.
2009-10-26 07:12:20 +01:00
Rename message_results/thread_results to messages/threads. Shorter naming without being any less clear. A definite win.
2009-11-01 00:32:30 +01:00
threads = talloc (query, notmuch_threads_t);
if (threads == NULL)
Add an initial implementation of a notmuch_thread_t object. We've now got a new notmuch_query_search_threads and a notmuch_threads_result_t iterator. The thread object itself doesn't do much yet, (just allows one to get the thread_id), but that's at least enough to see that "notmuch search" is actually doing something now, (since it has been converted to print thread IDs instead of message IDs). And maybe that's all we need. Getting the messages belonging to a thread is as simple as a notmuch_query_search_messages with a string of "thread:<thread-id>". Though it would be convenient to add notmuch_thread_get_messages which could use the existing notmuch_message_results_t iterator. Now we just need an implementation of "notmuch show" and we'll have something somewhat usable.
2009-10-26 07:12:20 +01:00
return NULL;
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
threads->doc_ids = NULL;
talloc_set_destructor (threads, _notmuch_threads_destructor);
Add an initial implementation of a notmuch_thread_t object. We've now got a new notmuch_query_search_threads and a notmuch_threads_result_t iterator. The thread object itself doesn't do much yet, (just allows one to get the thread_id), but that's at least enough to see that "notmuch search" is actually doing something now, (since it has been converted to print thread IDs instead of message IDs). And maybe that's all we need. Getting the messages belonging to a thread is as simple as a notmuch_query_search_messages with a string of "thread:<thread-id>". Though it would be convenient to add notmuch_thread_get_messages which could use the existing notmuch_message_results_t iterator. Now we just need an implementation of "notmuch show" and we'll have something somewhat usable.
2009-10-26 07:12:20 +01:00
lib/query: Fix notmuch_threads_t to stream results rather than blocking. Previously, notmuch_query_search_threads would do all the work, so the caller would block until all results were processed. Now, we do the work as we go, as the caller iterates with notmuch_threads_next. This means that once results start coming back from "notmuch search" they just keep continually streaming. There's still some initial blocking before the first results appear because the notmuch_messages_t object has the same bug (for now).
2009-11-24 05:18:57 +01:00
threads->query = query;
notmuch_query_search_threads: Avoid returning more threads than asked for. I thought it would be safe enough to return a few extra threads, (since we happened to already get the relevant messages out of the database). The problem is that then requires the caller to carefully read the number of threads returned and adjust its next "first" value accordingly. The interface is much simpler to use if we simply return exactly what is asked for and no more.
2009-11-13 05:31:22 +01:00
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
messages = notmuch_query_search_messages (query);
if (messages == NULL) {
lib: Fix notmuch_query_search_threads to return NULL on any Xapian exception. Previously, if the underlying search_messages hit an exception and returned NULL, this function would ignore that and return a non-NULL, (but empty) threads object. Fix this to properly propagate the error.
2010-10-23 02:55:07 +02:00
talloc_free (threads);
return NULL;
}
libnotmuch: Underlying support for doing partial-results searches. The library interface now allows the caller to do incremental searches, (such as one page of results at a time). Next we'll just need to hook this up to "notmuch search" and the emacs interface.
2009-11-13 01:47:27 +01:00
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
threads->doc_ids = g_array_new (FALSE, FALSE, sizeof (unsigned int));
while (notmuch_messages_valid (messages)) {
unsigned int doc_id = _notmuch_mset_messages_get_doc_id (messages);
g_array_append_val (threads->doc_ids, doc_id);
max_doc_id = MAX (max_doc_id, doc_id);
notmuch_messages_move_to_next (messages);
}
threads->doc_id_pos = 0;
Add an initial implementation of a notmuch_thread_t object. We've now got a new notmuch_query_search_threads and a notmuch_threads_result_t iterator. The thread object itself doesn't do much yet, (just allows one to get the thread_id), but that's at least enough to see that "notmuch search" is actually doing something now, (since it has been converted to print thread IDs instead of message IDs). And maybe that's all we need. Getting the messages belonging to a thread is as simple as a notmuch_query_search_messages with a string of "thread:<thread-id>". Though it would be convenient to add notmuch_thread_get_messages which could use the existing notmuch_message_results_t iterator. Now we just need an implementation of "notmuch show" and we'll have something somewhat usable.
2009-10-26 07:12:20 +01:00
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
talloc_free (messages);
if (! _notmuch_doc_id_set_init (threads, &threads->match_set,
threads->doc_ids, max_doc_id + 1)) {
talloc_free (threads);
return NULL;
}
Add an initial implementation of a notmuch_thread_t object. We've now got a new notmuch_query_search_threads and a notmuch_threads_result_t iterator. The thread object itself doesn't do much yet, (just allows one to get the thread_id), but that's at least enough to see that "notmuch search" is actually doing something now, (since it has been converted to print thread IDs instead of message IDs). And maybe that's all we need. Getting the messages belonging to a thread is as simple as a notmuch_query_search_messages with a string of "thread:<thread-id>". Though it would be convenient to add notmuch_thread_get_messages which could use the existing notmuch_message_results_t iterator. Now we just need an implementation of "notmuch show" and we'll have something somewhat usable.
2009-10-26 07:12:20 +01:00
Rename message_results/thread_results to messages/threads. Shorter naming without being any less clear. A definite win.
2009-11-01 00:32:30 +01:00
return threads;
Add an initial implementation of a notmuch_thread_t object. We've now got a new notmuch_query_search_threads and a notmuch_threads_result_t iterator. The thread object itself doesn't do much yet, (just allows one to get the thread_id), but that's at least enough to see that "notmuch search" is actually doing something now, (since it has been converted to print thread IDs instead of message IDs). And maybe that's all we need. Getting the messages belonging to a thread is as simple as a notmuch_query_search_messages with a string of "thread:<thread-id>". Though it would be convenient to add notmuch_thread_get_messages which could use the existing notmuch_message_results_t iterator. Now we just need an implementation of "notmuch show" and we'll have something somewhat usable.
2009-10-26 07:12:20 +01:00
}
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
void
notmuch_query_destroy (notmuch_query_t *query)
{
talloc_free (query);
}
Add an initial implementation of a notmuch_thread_t object. We've now got a new notmuch_query_search_threads and a notmuch_threads_result_t iterator. The thread object itself doesn't do much yet, (just allows one to get the thread_id), but that's at least enough to see that "notmuch search" is actually doing something now, (since it has been converted to print thread IDs instead of message IDs). And maybe that's all we need. Getting the messages belonging to a thread is as simple as a notmuch_query_search_messages with a string of "thread:<thread-id>". Though it would be convenient to add notmuch_thread_get_messages which could use the existing notmuch_message_results_t iterator. Now we just need an implementation of "notmuch show" and we'll have something somewhat usable.
2009-10-26 07:12:20 +01:00
notmuch_bool_t
lib: Rename iterator functions to prepare for reverse iteration. We rename 'has_more' to 'valid' so that it can function whether iterating in a forward or reverse direction. We also rename 'advance' to 'move_to_next' to setup parallel naming with the proposed functions 'move_to_first', 'move_to_last', and 'move_to_previous'.
2010-03-09 18:22:29 +01:00
notmuch_threads_valid (notmuch_threads_t *threads)
Add an initial implementation of a notmuch_thread_t object. We've now got a new notmuch_query_search_threads and a notmuch_threads_result_t iterator. The thread object itself doesn't do much yet, (just allows one to get the thread_id), but that's at least enough to see that "notmuch search" is actually doing something now, (since it has been converted to print thread IDs instead of message IDs). And maybe that's all we need. Getting the messages belonging to a thread is as simple as a notmuch_query_search_messages with a string of "thread:<thread-id>". Though it would be convenient to add notmuch_thread_get_messages which could use the existing notmuch_message_results_t iterator. Now we just need an implementation of "notmuch show" and we'll have something somewhat usable.
2009-10-26 07:12:20 +01:00
{
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
unsigned int doc_id;
lib/query: Fix notmuch_threads_t to stream results rather than blocking. Previously, notmuch_query_search_threads would do all the work, so the caller would block until all results were processed. Now, we do the work as we go, as the caller iterates with notmuch_threads_next. This means that once results start coming back from "notmuch search" they just keep continually streaming. There's still some initial blocking before the first results appear because the notmuch_messages_t object has the same bug (for now).
2009-11-24 05:18:57 +01:00
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
while (threads->doc_id_pos < threads->doc_ids->len) {
doc_id = g_array_index (threads->doc_ids, unsigned int,
threads->doc_id_pos);
if (_notmuch_doc_id_set_contains (&threads->match_set, doc_id))
break;
lib/query: Fix notmuch_threads_t to stream results rather than blocking. Previously, notmuch_query_search_threads would do all the work, so the caller would block until all results were processed. Now, we do the work as we go, as the caller iterates with notmuch_threads_next. This means that once results start coming back from "notmuch search" they just keep continually streaming. There's still some initial blocking before the first results appear because the notmuch_messages_t object has the same bug (for now).
2009-11-24 05:18:57 +01:00
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
threads->doc_id_pos++;
lib/query: Fix notmuch_threads_t to stream results rather than blocking. Previously, notmuch_query_search_threads would do all the work, so the caller would block until all results were processed. Now, we do the work as we go, as the caller iterates with notmuch_threads_next. This means that once results start coming back from "notmuch search" they just keep continually streaming. There's still some initial blocking before the first results appear because the notmuch_messages_t object has the same bug (for now).
2009-11-24 05:18:57 +01:00
}
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
return threads->doc_id_pos < threads->doc_ids->len;
Add an initial implementation of a notmuch_thread_t object. We've now got a new notmuch_query_search_threads and a notmuch_threads_result_t iterator. The thread object itself doesn't do much yet, (just allows one to get the thread_id), but that's at least enough to see that "notmuch search" is actually doing something now, (since it has been converted to print thread IDs instead of message IDs). And maybe that's all we need. Getting the messages belonging to a thread is as simple as a notmuch_query_search_messages with a string of "thread:<thread-id>". Though it would be convenient to add notmuch_thread_get_messages which could use the existing notmuch_message_results_t iterator. Now we just need an implementation of "notmuch show" and we'll have something somewhat usable.
2009-10-26 07:12:20 +01:00
}
notmuch_thread_t *
Rename message_results/thread_results to messages/threads. Shorter naming without being any less clear. A definite win.
2009-11-01 00:32:30 +01:00
notmuch_threads_get (notmuch_threads_t *threads)
Add an initial implementation of a notmuch_thread_t object. We've now got a new notmuch_query_search_threads and a notmuch_threads_result_t iterator. The thread object itself doesn't do much yet, (just allows one to get the thread_id), but that's at least enough to see that "notmuch search" is actually doing something now, (since it has been converted to print thread IDs instead of message IDs). And maybe that's all we need. Getting the messages belonging to a thread is as simple as a notmuch_query_search_messages with a string of "thread:<thread-id>". Though it would be convenient to add notmuch_thread_get_messages which could use the existing notmuch_message_results_t iterator. Now we just need an implementation of "notmuch show" and we'll have something somewhat usable.
2009-10-26 07:12:20 +01:00
{
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
unsigned int doc_id;
lib: Rename iterator functions to prepare for reverse iteration. We rename 'has_more' to 'valid' so that it can function whether iterating in a forward or reverse direction. We also rename 'advance' to 'move_to_next' to setup parallel naming with the proposed functions 'move_to_first', 'move_to_last', and 'move_to_previous'.
2010-03-09 18:22:29 +01:00
if (! notmuch_threads_valid (threads))
results_get: Fix to return NULL if past the end of the results We had documented both notmuch_thread_results_get and notmuch_message_results_get to return NULL if (! has_more) but we hadn't actually implemented that. Fix.
2009-10-26 13:14:51 +01:00
return NULL;
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
doc_id = g_array_index (threads->doc_ids, unsigned int,
threads->doc_id_pos);
lib/query: Fix notmuch_threads_t to stream results rather than blocking. Previously, notmuch_query_search_threads would do all the work, so the caller would block until all results were processed. Now, we do the work as we go, as the caller iterates with notmuch_threads_next. This means that once results start coming back from "notmuch search" they just keep continually streaming. There's still some initial blocking before the first results appear because the notmuch_messages_t object has the same bug (for now).
2009-11-24 05:18:57 +01:00
return _notmuch_thread_create (threads->query,
threads->query->notmuch,
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
doc_id,
&threads->match_set,
Name thread based on matching msgs instead of first msg. At the moment all threads are named based on the name of the first message in the thread. However, this can cause problems if people either start new threads by replying-all (as unfortunately, many out there do) or change the subject of their mails to reflect a shift in a thread on a list. This patch names threads based on (a) matches for the query, and (b) the search order. If the search order is oldest-first (as in the default inbox) it chooses the oldest matching message as the subject. If the search order is newest-first it chooses the newest one. Reply prefixes ("Re: ", "Aw: ", "Sv: ", "Vs: ") are ignored (case-insensitively) so a Re: won't change the subject. Note that this adds a "sort" argument to _notmuch_thread_create and _thread_add_matched_message, so that when constructing the thread we can be aware of the sort order. Signed-off-by: Jesse Rosenthal <jrosenthal@jhu.edu>
2010-04-17 19:59:22 +02:00
threads->query->sort);
Add an initial implementation of a notmuch_thread_t object. We've now got a new notmuch_query_search_threads and a notmuch_threads_result_t iterator. The thread object itself doesn't do much yet, (just allows one to get the thread_id), but that's at least enough to see that "notmuch search" is actually doing something now, (since it has been converted to print thread IDs instead of message IDs). And maybe that's all we need. Getting the messages belonging to a thread is as simple as a notmuch_query_search_messages with a string of "thread:<thread-id>". Though it would be convenient to add notmuch_thread_get_messages which could use the existing notmuch_message_results_t iterator. Now we just need an implementation of "notmuch show" and we'll have something somewhat usable.
2009-10-26 07:12:20 +01:00
}
void
lib: Rename iterator functions to prepare for reverse iteration. We rename 'has_more' to 'valid' so that it can function whether iterating in a forward or reverse direction. We also rename 'advance' to 'move_to_next' to setup parallel naming with the proposed functions 'move_to_first', 'move_to_last', and 'move_to_previous'.
2010-03-09 18:22:29 +01:00
notmuch_threads_move_to_next (notmuch_threads_t *threads)
Add an initial implementation of a notmuch_thread_t object. We've now got a new notmuch_query_search_threads and a notmuch_threads_result_t iterator. The thread object itself doesn't do much yet, (just allows one to get the thread_id), but that's at least enough to see that "notmuch search" is actually doing something now, (since it has been converted to print thread IDs instead of message IDs). And maybe that's all we need. Getting the messages belonging to a thread is as simple as a notmuch_query_search_messages with a string of "thread:<thread-id>". Though it would be convenient to add notmuch_thread_get_messages which could use the existing notmuch_message_results_t iterator. Now we just need an implementation of "notmuch show" and we'll have something somewhat usable.
2009-10-26 07:12:20 +01:00
{
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
threads->doc_id_pos++;
Add an initial implementation of a notmuch_thread_t object. We've now got a new notmuch_query_search_threads and a notmuch_threads_result_t iterator. The thread object itself doesn't do much yet, (just allows one to get the thread_id), but that's at least enough to see that "notmuch search" is actually doing something now, (since it has been converted to print thread IDs instead of message IDs). And maybe that's all we need. Getting the messages belonging to a thread is as simple as a notmuch_query_search_messages with a string of "thread:<thread-id>". Though it would be convenient to add notmuch_thread_get_messages which could use the existing notmuch_message_results_t iterator. Now we just need an implementation of "notmuch show" and we'll have something somewhat usable.
2009-10-26 07:12:20 +01:00
}
void
Rename message_results/thread_results to messages/threads. Shorter naming without being any less clear. A definite win.
2009-11-01 00:32:30 +01:00
notmuch_threads_destroy (notmuch_threads_t *threads)
Add an initial implementation of a notmuch_thread_t object. We've now got a new notmuch_query_search_threads and a notmuch_threads_result_t iterator. The thread object itself doesn't do much yet, (just allows one to get the thread_id), but that's at least enough to see that "notmuch search" is actually doing something now, (since it has been converted to print thread IDs instead of message IDs). And maybe that's all we need. Getting the messages belonging to a thread is as simple as a notmuch_query_search_messages with a string of "thread:<thread-id>". Though it would be convenient to add notmuch_thread_get_messages which could use the existing notmuch_message_results_t iterator. Now we just need an implementation of "notmuch show" and we'll have something somewhat usable.
2009-10-26 07:12:20 +01:00
{
Rename message_results/thread_results to messages/threads. Shorter naming without being any less clear. A definite win.
2009-11-01 00:32:30 +01:00
talloc_free (threads);
Add an initial implementation of a notmuch_thread_t object. We've now got a new notmuch_query_search_threads and a notmuch_threads_result_t iterator. The thread object itself doesn't do much yet, (just allows one to get the thread_id), but that's at least enough to see that "notmuch search" is actually doing something now, (since it has been converted to print thread IDs instead of message IDs). And maybe that's all we need. Getting the messages belonging to a thread is as simple as a notmuch_query_search_messages with a string of "thread:<thread-id>". Though it would be convenient to add notmuch_thread_get_messages which could use the existing notmuch_message_results_t iterator. Now we just need an implementation of "notmuch show" and we'll have something somewhat usable.
2009-10-26 07:12:20 +01:00
}
Add 'notmuch count' command to show the count of matching messages Getting the count of matching threads or messages is a fairly expensive operation. Xapian provides a very efficient mechanism that returns an approximate value, so use that for this new command. This returns the number of matching messages, not threads, as that is cheap to compute. Signed-off-by: Keith Packard <keithp@keithp.com>
2009-11-21 08:15:07 +01:00
unsigned
notmuch_query_count_messages (notmuch_query_t *query)
{
notmuch_database_t *notmuch = query->notmuch;
const char *query_string = query->query_string;
Silence compiler warning by initializing a variable. If Xapian threw an exception on notmuch_query_count_messages the count variable could be used uninitialized. Initialize count to solve the problem. Signed-off-by: Jeffrey C. Ollie <jeff@ocjtech.us>
2009-11-25 14:01:36 +01:00
Xapian::doccount count = 0;
Add 'notmuch count' command to show the count of matching messages Getting the count of matching threads or messages is a fairly expensive operation. Xapian provides a very efficient mechanism that returns an approximate value, so use that for this new command. This returns the number of matching messages, not threads, as that is cheap to compute. Signed-off-by: Keith Packard <keithp@keithp.com>
2009-11-21 08:15:07 +01:00
try {
Xapian::Enquire enquire (*notmuch->xapian_db);
Xapian::Query mail_query (talloc_asprintf (query, "%s%s",
_find_prefix ("type"),
"mail"));
Xapian::Query string_query, final_query;
Xapian::MSet mset;
unsigned int flags = (Xapian::QueryParser::FLAG_BOOLEAN |
Xapian::QueryParser::FLAG_PHRASE |
Xapian::QueryParser::FLAG_LOVEHATE |
Xapian::QueryParser::FLAG_BOOLEAN_ANY_CASE |
Xapian::QueryParser::FLAG_WILDCARD |
Xapian::QueryParser::FLAG_PURE_NOT);
lib: Handle "*" as a query string to match all messages. This seems like a generally useful thing to support, (but the previous support through an empty string was not convenient for some users, (such as the command-line client).
2010-04-10 01:40:31 +02:00
if (strcmp (query_string, "") == 0 ||
strcmp (query_string, "*") == 0)
{
Add 'notmuch count' command to show the count of matching messages Getting the count of matching threads or messages is a fairly expensive operation. Xapian provides a very efficient mechanism that returns an approximate value, so use that for this new command. This returns the number of matching messages, not threads, as that is cheap to compute. Signed-off-by: Keith Packard <keithp@keithp.com>
2009-11-21 08:15:07 +01:00
final_query = mail_query;
} else {
string_query = notmuch->query_parser->
parse_query (query_string, flags);
final_query = Xapian::Query (Xapian::Query::OP_AND,
mail_query, string_query);
}
enquire.set_weighting_scheme(Xapian::BoolWeight());
enquire.set_docid_order(Xapian::Enquire::ASCENDING);
#if DEBUG_QUERY
fprintf (stderr, "Final query is:\n%s\n", final_query.get_description().c_str());
#endif
enquire.set_query (final_query);
mset = enquire.get_mset (0, notmuch->xapian_db->get_doccount ());
count = mset.get_matches_estimated();
} catch (const Xapian::Error &error) {
fprintf (stderr, "A Xapian exception occurred: %s\n",
error.get_msg().c_str());
fprintf (stderr, "Query string was: %s\n", query->query_string);
}
return count;
}
Reference in a new issue Copy permalink