debian/notmuch
1
0
Fork 0
mirror of https://git.notmuchmail.org/git/notmuch synced 2024-11-21 18:38:08 +01:00
Code Issues Projects Releases Packages Wiki Activity Actions
notmuch/lib/query.cc

616 lines
17 KiB
C++
Raw Normal View History

Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
/* query.cc - Support for searching a notmuch database
*
* Copyright © 2009 Carl Worth
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program. If not, see http://www.gnu.org/licenses/ .
*
* Author: Carl Worth <cworth@cworth.org>
*/
#include "notmuch-private.h"
#include "database-private.h"
Add an initial implementation of a notmuch_thread_t object. We've now got a new notmuch_query_search_threads and a notmuch_threads_result_t iterator. The thread object itself doesn't do much yet, (just allows one to get the thread_id), but that's at least enough to see that "notmuch search" is actually doing something now, (since it has been converted to print thread IDs instead of message IDs). And maybe that's all we need. Getting the messages belonging to a thread is as simple as a notmuch_query_search_messages with a string of "thread:<thread-id>". Though it would be convenient to add notmuch_thread_get_messages which could use the existing notmuch_message_results_t iterator. Now we just need an implementation of "notmuch show" and we'll have something somewhat usable.
2009-10-26 07:12:20 +01:00
#include <glib.h> /* GHashTable, GPtrArray */
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
struct _notmuch_query {
notmuch_database_t *notmuch;
const char *query_string;
notmuch_sort_t sort;
lib: Add support for automatically excluding tags from queries This is useful for tags like "deleted" and "spam" that people generally want to exclude from query results. These exclusions will be overridden if a tag is explicitly mentioned in a query.
2012-01-15 01:17:33 +01:00
notmuch_string_list_t *exclude_terms;
lib: add --exclude=all option Adds a exclude all option to the lib which means that excluded messages are completely ignored (as if they had actually been deleted).
2013-05-11 21:50:02 +02:00
notmuch_exclude_t omit_excluded;
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
};
lib/messages.c: Make message searches stream as well. Xapian provides an interator-based interface to all search results. So it was natural to make notmuch_messages_t be iterator-based as well. Which we did originally. But we ran into a problem when we added two APIs, (_get_replies and _get_toplevel_messages), that want to return a messages iterator that's *not* based on a Xapian search result. My original compromise was to use notmuch_message_list_t as the basis for all returned messages iterators in the public interface. This had the problem of introducing extra latency at the beginning of a search for messages, (the call would block while iterating over all results from Xapian, converting to a message list). In this commit, we remove that initial conversion and instead provide two alternate implementations of notmuch_messages_t (one on top of a Xapian iterator and one on top of a message list). With this change, I tested a "notmuch search" returning *many* results as previously taking about 7 seconds before results started appearing, and now taking only 2 seconds.
2009-11-24 06:47:24 +01:00
typedef struct _notmuch_mset_messages {
notmuch_messages_t base;
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
notmuch_database_t *notmuch;
notmuch dump: Fix the sorting of results. To properly support sorting in notmuch_query we know use an Enquire object. We also throw in a QueryParser too, so we're really close to being able to support arbitrary full-text searches. I took a look at the supported QueryParser syntax and chose a set of flags for everything I like, (such as supporting Boolean operators in either case ("AND" or "and"), supporting phrase searching, supporting + and - to include/preclude terms, and supporting a trailing * on any term as a wildcard).
2009-10-21 09:35:56 +02:00
Xapian::MSetIterator iterator;
Xapian::MSetIterator iterator_end;
lib/messages.c: Make message searches stream as well. Xapian provides an interator-based interface to all search results. So it was natural to make notmuch_messages_t be iterator-based as well. Which we did originally. But we ran into a problem when we added two APIs, (_get_replies and _get_toplevel_messages), that want to return a messages iterator that's *not* based on a Xapian search result. My original compromise was to use notmuch_message_list_t as the basis for all returned messages iterators in the public interface. This had the problem of introducing extra latency at the beginning of a search for messages, (the call would block while iterating over all results from Xapian, converting to a message list). In this commit, we remove that initial conversion and instead provide two alternate implementations of notmuch_messages_t (one on top of a Xapian iterator and one on top of a message list). With this change, I tested a "notmuch search" returning *many* results as previously taking about 7 seconds before results started appearing, and now taking only 2 seconds.
2009-11-24 06:47:24 +01:00
} notmuch_mset_messages_t;
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
struct _notmuch_doc_id_set {
bitmap:improve memory usage using CHAR_BITS and unsigned CHAR Using char instead of int allows for simpler definitions of the DOCIDSET macros so the code is easier to understand and consistent with respect to memory-usage. Estimated reduction of memory-usage for bitmap about 8 times.
2013-02-13 16:32:57 +01:00
unsigned char *bitmap;
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
unsigned int bound;
};
bitmap:improve memory usage using CHAR_BITS and unsigned CHAR Using char instead of int allows for simpler definitions of the DOCIDSET macros so the code is easier to understand and consistent with respect to memory-usage. Estimated reduction of memory-usage for bitmap about 8 times.
2013-02-13 16:32:57 +01:00
#define DOCIDSET_WORD(bit) ((bit) / CHAR_BIT)
#define DOCIDSET_BIT(bit) ((bit) % CHAR_BIT)
Remove code repetition in the doc ID bitmap code. Remove the repeated "sizeof (doc_ids->bitmap[0])" that bothered cworth by instead defining macros to compute the word and bit offset of a given bit in the doc ID set bitmap.
2011-01-31 02:58:27 +01:00
Mark some structures in the library interface with visibility=default attribute. As of gcc 4.6, there are new warnings from -Wattributes along the lines of: warning: ‘_notmuch_messages’ declared with greater visibility than the type of its field ‘_notmuch_messages::iterator’ [-Wattributes] To squelch these, we decorate all such containing structs with __attribute__((visibility("default"))). We take care to let only the C++ compiler see this, (since the C compiler would otherwise warn about ignored visibility attributes on types).
2011-05-11 22:23:13 +02:00
struct visible _notmuch_threads {
lib/query: Fix notmuch_threads_t to stream results rather than blocking. Previously, notmuch_query_search_threads would do all the work, so the caller would block until all results were processed. Now, we do the work as we go, as the caller iterates with notmuch_threads_next. This means that once results start coming back from "notmuch search" they just keep continually streaming. There's still some initial blocking before the first results appear because the notmuch_messages_t object has the same bug (for now).
2009-11-24 05:18:57 +01:00
notmuch_query_t *query;
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
/* The ordered list of doc ids matched by the query. */
GArray *doc_ids;
/* Our iterator's current position in doc_ids. */
unsigned int doc_id_pos;
/* The set of matched docid's that have not been assigned to a
* thread. Initially, this contains every docid in doc_ids. */
notmuch_doc_id_set_t match_set;
Add an initial implementation of a notmuch_thread_t object. We've now got a new notmuch_query_search_threads and a notmuch_threads_result_t iterator. The thread object itself doesn't do much yet, (just allows one to get the thread_id), but that's at least enough to see that "notmuch search" is actually doing something now, (since it has been converted to print thread IDs instead of message IDs). And maybe that's all we need. Getting the messages belonging to a thread is as simple as a notmuch_query_search_messages with a string of "thread:<thread-id>". Though it would be convenient to add notmuch_thread_get_messages which could use the existing notmuch_message_results_t iterator. Now we just need an implementation of "notmuch show" and we'll have something somewhat usable.
2009-10-26 07:12:20 +01:00
};
lib: Make notmuch_query_search_messages set the exclude flag Add a flag NOTMUCH_MESSAGE_FLAG_EXCLUDED which is set by notmuch_query_search_messages for excluded messages. Also add an option omit_excluded_messages to the search that we do not want the excludes at all. This exclude flag will be added to notmuch_query_search threads in the next patch.
2012-03-01 23:30:37 +01:00
/* We need this in the message functions so forward declare. */
static notmuch_bool_t
_notmuch_doc_id_set_init (void *ctx,
notmuch_doc_id_set_t *doc_ids,
GArray *arr);
lib: Expose query debug output via an environment variable Allow query debugging to be enabled at run-time by setting the NOTMUCH_DEBUG_QUERY environment variable to a non-empty string. Previously, enabling query debugging required recompiling, but parsed queries are often useful for tracking down bugs in situations where recompiling is inconvenient.
2012-03-14 03:31:30 +01:00
static notmuch_bool_t
_debug_query (void)
{
char *env = getenv ("NOTMUCH_DEBUG_QUERY");
return (env && strcmp (env, "") != 0);
}
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
notmuch_query_t *
notmuch_query_create (notmuch_database_t *notmuch,
const char *query_string)
{
notmuch_query_t *query;
lib: Expose query debug output via an environment variable Allow query debugging to be enabled at run-time by setting the NOTMUCH_DEBUG_QUERY environment variable to a non-empty string. Previously, enabling query debugging required recompiling, but parsed queries are often useful for tracking down bugs in situations where recompiling is inconvenient.
2012-03-14 03:31:30 +01:00
if (_debug_query ())
fprintf (stderr, "Query string is:\n%s\n", query_string);
Add debugging code for examining query strings. It's nice that Xapian provides a little function to print a textual representation of the entire query tree. So now, if you compile like so: make CFLAGS=-DDEBUG_QUERY then you get a nice output of the query string received by the query module, and the final query actually being sent to Xapian.
2009-10-25 06:18:20 +01:00
query: bind queries to database objects The queries don't really work after a database is closed, and we would like them to be freed if the database is destroyed. Acknowledged-by: David Bremner <david@tethera.net> Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
2013-09-30 18:04:21 +02:00
query = talloc (notmuch, notmuch_query_t);
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
if (unlikely (query == NULL))
return NULL;
query->notmuch = notmuch;
query: Remove the magic NOTMUCH_QUERY_ALL Using the address of a static char* was clever, but really unnecessary. An empty string is much less magic, and even easier to understand as the way to query everything from the database.
2009-10-21 07:40:37 +02:00
query->query_string = talloc_strdup (query, query_string);
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
notmuch search: Change default search order to be newest messages first. This is what most people want for a _search_ command. It's often different for actually reading mail in an inbox, (where it makes more sense to have results displayed in chronological order), but in such a case, ther user is likely using an interface that can simply pass the --sort=oldest-first option to "notmuch search". Here we're also change the sort enum from NOTMUCH_SORT_DATE and NOTMUCH_SORT_DATE_REVERSE to NOTMUCH_SORT_OLDEST_FIRST and NOTMUCH_SORT_NEWEST_FIRST. Similarly we replace the --reverse option to "notmuch search" with two options: --sort=oldest-first and --sort=newest-first. Finally, these changes are all tracked in the emacs interface, (which has no change in its behavior).
2009-11-18 05:52:09 +01:00
query->sort = NOTMUCH_SORT_NEWEST_FIRST;
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
lib: Add support for automatically excluding tags from queries This is useful for tags like "deleted" and "spam" that people generally want to exclude from query results. These exclusions will be overridden if a tag is explicitly mentioned in a query.
2012-01-15 01:17:33 +01:00
query->exclude_terms = _notmuch_string_list_create (query);
lib: add --exclude=all option Adds a exclude all option to the lib which means that excluded messages are completely ignored (as if they had actually been deleted).
2013-05-11 21:50:02 +02:00
query->omit_excluded = NOTMUCH_EXCLUDE_TRUE;
lib: Make notmuch_query_search_messages set the exclude flag Add a flag NOTMUCH_MESSAGE_FLAG_EXCLUDED which is set by notmuch_query_search_messages for excluded messages. Also add an option omit_excluded_messages to the search that we do not want the excludes at all. This exclude flag will be added to notmuch_query_search threads in the next patch.
2012-03-01 23:30:37 +01:00
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
return query;
}
lib: Add two functions: notmuch_query_get_query_string and _get_sort It can be handy to be able to query these settings from an existing query object.
2010-10-28 19:30:26 +02:00
const char *
notmuch_query_get_query_string (notmuch_query_t *query)
{
return query->query_string;
}
lib: Make notmuch_query_search_messages set the exclude flag Add a flag NOTMUCH_MESSAGE_FLAG_EXCLUDED which is set by notmuch_query_search_messages for excluded messages. Also add an option omit_excluded_messages to the search that we do not want the excludes at all. This exclude flag will be added to notmuch_query_search threads in the next patch.
2012-03-01 23:30:37 +01:00
void
lib: add --exclude=all option Adds a exclude all option to the lib which means that excluded messages are completely ignored (as if they had actually been deleted).
2013-05-11 21:50:02 +02:00
notmuch_query_set_omit_excluded (notmuch_query_t *query,
notmuch_exclude_t omit_excluded)
lib: Make notmuch_query_search_messages set the exclude flag Add a flag NOTMUCH_MESSAGE_FLAG_EXCLUDED which is set by notmuch_query_search_messages for excluded messages. Also add an option omit_excluded_messages to the search that we do not want the excludes at all. This exclude flag will be added to notmuch_query_search threads in the next patch.
2012-03-01 23:30:37 +01:00
{
lib: change default for notmuch_query_set_omit_excluded
2012-04-07 18:10:03 +02:00
query->omit_excluded = omit_excluded;
lib: Make notmuch_query_search_messages set the exclude flag Add a flag NOTMUCH_MESSAGE_FLAG_EXCLUDED which is set by notmuch_query_search_messages for excluded messages. Also add an option omit_excluded_messages to the search that we do not want the excludes at all. This exclude flag will be added to notmuch_query_search threads in the next patch.
2012-03-01 23:30:37 +01:00
}
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
void
notmuch_query_set_sort (notmuch_query_t *query, notmuch_sort_t sort)
{
query->sort = sort;
}
lib: Add two functions: notmuch_query_get_query_string and _get_sort It can be handy to be able to query these settings from an existing query object.
2010-10-28 19:30:26 +02:00
notmuch_sort_t
notmuch_query_get_sort (notmuch_query_t *query)
{
return query->sort;
}
lib: Add support for automatically excluding tags from queries This is useful for tags like "deleted" and "spam" that people generally want to exclude from query results. These exclusions will be overridden if a tag is explicitly mentioned in a query.
2012-01-15 01:17:33 +01:00
void
notmuch_query_add_tag_exclude (notmuch_query_t *query, const char *tag)
{
char *term = talloc_asprintf (query, "%s%s", _find_prefix ("tag"), tag);
_notmuch_string_list_append (query->exclude_terms, term);
}
lib/messages.c: Make message searches stream as well. Xapian provides an interator-based interface to all search results. So it was natural to make notmuch_messages_t be iterator-based as well. Which we did originally. But we ran into a problem when we added two APIs, (_get_replies and _get_toplevel_messages), that want to return a messages iterator that's *not* based on a Xapian search result. My original compromise was to use notmuch_message_list_t as the basis for all returned messages iterators in the public interface. This had the problem of introducing extra latency at the beginning of a search for messages, (the call would block while iterating over all results from Xapian, converting to a message list). In this commit, we remove that initial conversion and instead provide two alternate implementations of notmuch_messages_t (one on top of a Xapian iterator and one on top of a message list). With this change, I tested a "notmuch search" returning *many* results as previously taking about 7 seconds before results started appearing, and now taking only 2 seconds.
2009-11-24 06:47:24 +01:00
/* We end up having to call the destructors explicitly because we had
* to use "placement new" in order to initialize C++ objects within a
* block that we allocated with talloc. So C++ is making talloc
* slightly less simple to use, (we wouldn't need
* talloc_set_destructor at all otherwise).
*/
static int
_notmuch_messages_destructor (notmuch_mset_messages_t *messages)
{
messages->iterator.~MSetIterator ();
messages->iterator_end.~MSetIterator ();
return 0;
}
lib: Rearrange the exclude code in query.cc Slightly refactor the exclude code to give the callers access to the exclude query itself. There should be no functional change.
2012-03-01 23:30:36 +01:00
/* Return a query that matches messages with the excluded tags
* registered with query. Any tags that explicitly appear in xquery
lib: Make notmuch_query_search_messages set the exclude flag Add a flag NOTMUCH_MESSAGE_FLAG_EXCLUDED which is set by notmuch_query_search_messages for excluded messages. Also add an option omit_excluded_messages to the search that we do not want the excludes at all. This exclude flag will be added to notmuch_query_search threads in the next patch.
2012-03-01 23:30:37 +01:00
* will not be excluded, and will be removed from the list of exclude
* tags. The caller of this function has to combine the returned
* query appropriately.*/
lib: Add support for automatically excluding tags from queries This is useful for tags like "deleted" and "spam" that people generally want to exclude from query results. These exclusions will be overridden if a tag is explicitly mentioned in a query.
2012-01-15 01:17:33 +01:00
static Xapian::Query
_notmuch_exclude_tags (notmuch_query_t *query, Xapian::Query xquery)
{
lib: Rearrange the exclude code in query.cc Slightly refactor the exclude code to give the callers access to the exclude query itself. There should be no functional change.
2012-03-01 23:30:36 +01:00
Xapian::Query exclude_query = Xapian::Query::MatchNothing;
lib: Add support for automatically excluding tags from queries This is useful for tags like "deleted" and "spam" that people generally want to exclude from query results. These exclusions will be overridden if a tag is explicitly mentioned in a query.
2012-01-15 01:17:33 +01:00
for (notmuch_string_node_t *term = query->exclude_terms->head; term;
term = term->next) {
Xapian::TermIterator it = xquery.get_terms_begin ();
Xapian::TermIterator end = xquery.get_terms_end ();
for (; it != end; it++) {
if ((*it).compare (term->string) == 0)
break;
}
if (it == end)
lib: Rearrange the exclude code in query.cc Slightly refactor the exclude code to give the callers access to the exclude query itself. There should be no functional change.
2012-03-01 23:30:36 +01:00
exclude_query = Xapian::Query (Xapian::Query::OP_OR,
exclude_query, Xapian::Query (term->string));
lib: Make notmuch_query_search_messages set the exclude flag Add a flag NOTMUCH_MESSAGE_FLAG_EXCLUDED which is set by notmuch_query_search_messages for excluded messages. Also add an option omit_excluded_messages to the search that we do not want the excludes at all. This exclude flag will be added to notmuch_query_search threads in the next patch.
2012-03-01 23:30:37 +01:00
else
term->string = talloc_strdup (query, "");
lib: Add support for automatically excluding tags from queries This is useful for tags like "deleted" and "spam" that people generally want to exclude from query results. These exclusions will be overridden if a tag is explicitly mentioned in a query.
2012-01-15 01:17:33 +01:00
}
lib: Rearrange the exclude code in query.cc Slightly refactor the exclude code to give the callers access to the exclude query itself. There should be no functional change.
2012-03-01 23:30:36 +01:00
return exclude_query;
lib: Add support for automatically excluding tags from queries This is useful for tags like "deleted" and "spam" that people generally want to exclude from query results. These exclusions will be overridden if a tag is explicitly mentioned in a query.
2012-01-15 01:17:33 +01:00
}
Rename message_results/thread_results to messages/threads. Shorter naming without being any less clear. A definite win.
2009-11-01 00:32:30 +01:00
notmuch_messages_t *
lib/query: Drop the first and max_messages arguments from search_messages. These only existed to support the chunky-searching hack, but that was recently dropped anyway.
2009-11-24 05:25:13 +01:00
notmuch_query_search_messages (notmuch_query_t *query)
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
{
notmuch dump: Fix the sorting of results. To properly support sorting in notmuch_query we know use an Enquire object. We also throw in a QueryParser too, so we're really close to being able to support arbitrary full-text searches. I took a look at the supported QueryParser syntax and chose a set of flags for everything I like, (such as supporting Boolean operators in either case ("AND" or "and"), supporting phrase searching, supporting + and - to include/preclude terms, and supporting a trailing * on any term as a wildcard).
2009-10-21 09:35:56 +02:00
notmuch_database_t *notmuch = query->notmuch;
const char *query_string = query->query_string;
lib/messages.c: Make message searches stream as well. Xapian provides an interator-based interface to all search results. So it was natural to make notmuch_messages_t be iterator-based as well. Which we did originally. But we ran into a problem when we added two APIs, (_get_replies and _get_toplevel_messages), that want to return a messages iterator that's *not* based on a Xapian search result. My original compromise was to use notmuch_message_list_t as the basis for all returned messages iterators in the public interface. This had the problem of introducing extra latency at the beginning of a search for messages, (the call would block while iterating over all results from Xapian, converting to a message list). In this commit, we remove that initial conversion and instead provide two alternate implementations of notmuch_messages_t (one on top of a Xapian iterator and one on top of a message list). With this change, I tested a "notmuch search" returning *many* results as previously taking about 7 seconds before results started appearing, and now taking only 2 seconds.
2009-11-24 06:47:24 +01:00
notmuch_mset_messages_t *messages;
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
lib/messages.c: Make message searches stream as well. Xapian provides an interator-based interface to all search results. So it was natural to make notmuch_messages_t be iterator-based as well. Which we did originally. But we ran into a problem when we added two APIs, (_get_replies and _get_toplevel_messages), that want to return a messages iterator that's *not* based on a Xapian search result. My original compromise was to use notmuch_message_list_t as the basis for all returned messages iterators in the public interface. This had the problem of introducing extra latency at the beginning of a search for messages, (the call would block while iterating over all results from Xapian, converting to a message list). In this commit, we remove that initial conversion and instead provide two alternate implementations of notmuch_messages_t (one on top of a Xapian iterator and one on top of a message list). With this change, I tested a "notmuch search" returning *many* results as previously taking about 7 seconds before results started appearing, and now taking only 2 seconds.
2009-11-24 06:47:24 +01:00
messages = talloc (query, notmuch_mset_messages_t);
if (unlikely (messages == NULL))
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
return NULL;
try {
lib/messages.c: Make message searches stream as well. Xapian provides an interator-based interface to all search results. So it was natural to make notmuch_messages_t be iterator-based as well. Which we did originally. But we ran into a problem when we added two APIs, (_get_replies and _get_toplevel_messages), that want to return a messages iterator that's *not* based on a Xapian search result. My original compromise was to use notmuch_message_list_t as the basis for all returned messages iterators in the public interface. This had the problem of introducing extra latency at the beginning of a search for messages, (the call would block while iterating over all results from Xapian, converting to a message list). In this commit, we remove that initial conversion and instead provide two alternate implementations of notmuch_messages_t (one on top of a Xapian iterator and one on top of a message list). With this change, I tested a "notmuch search" returning *many* results as previously taking about 7 seconds before results started appearing, and now taking only 2 seconds.
2009-11-24 06:47:24 +01:00
messages->base.is_of_list_type = FALSE;
messages->base.iterator = NULL;
messages->notmuch = notmuch;
new (&messages->iterator) Xapian::MSetIterator ();
new (&messages->iterator_end) Xapian::MSetIterator ();
talloc_set_destructor (messages, _notmuch_messages_destructor);
notmuch dump: Fix the sorting of results. To properly support sorting in notmuch_query we know use an Enquire object. We also throw in a QueryParser too, so we're really close to being able to support arbitrary full-text searches. I took a look at the supported QueryParser syntax and chose a set of flags for everything I like, (such as supporting Boolean operators in either case ("AND" or "and"), supporting phrase searching, supporting + and - to include/preclude terms, and supporting a trailing * on any term as a wildcard).
2009-10-21 09:35:56 +02:00
Xapian::Enquire enquire (*notmuch->xapian_db);
Use _find_prefix instead of hard-coded term in notmuch_query_search I'm planning to change prefix values soon, which would break code like this. So eliminate the fragility by going through our existing _find_prefix function.
2009-10-25 06:21:57 +01:00
Xapian::Query mail_query (talloc_asprintf (query, "%s%s",
_find_prefix ("type"),
"mail"));
lib: Rearrange the exclude code in query.cc Slightly refactor the exclude code to give the callers access to the exclude query itself. There should be no functional change.
2012-03-01 23:30:36 +01:00
Xapian::Query string_query, final_query, exclude_query;
notmuch dump: Fix the sorting of results. To properly support sorting in notmuch_query we know use an Enquire object. We also throw in a QueryParser too, so we're really close to being able to support arbitrary full-text searches. I took a look at the supported QueryParser syntax and chose a set of flags for everything I like, (such as supporting Boolean operators in either case ("AND" or "and"), supporting phrase searching, supporting + and - to include/preclude terms, and supporting a trailing * on any term as a wildcard).
2009-10-21 09:35:56 +02:00
Xapian::MSet mset;
lib: Make notmuch_query_search_messages set the exclude flag Add a flag NOTMUCH_MESSAGE_FLAG_EXCLUDED which is set by notmuch_query_search_messages for excluded messages. Also add an option omit_excluded_messages to the search that we do not want the excludes at all. This exclude flag will be added to notmuch_query_search threads in the next patch.
2012-03-01 23:30:37 +01:00
Xapian::MSetIterator iterator;
Fix bit-twiddling brain damage in notmuch_query_search Here's the big bug that was preventing any searches from working at all like desired. I did the work to carefully pick out exactly the flags that I wanted, and then I threw it away by trying to combine them with & instead of | (so just passing 0 for flags instead). Much better now.
2009-10-25 06:20:13 +01:00
unsigned int flags = (Xapian::QueryParser::FLAG_BOOLEAN |
Xapian::QueryParser::FLAG_PHRASE |
Xapian::QueryParser::FLAG_LOVEHATE |
Xapian::QueryParser::FLAG_BOOLEAN_ANY_CASE |
Allow lone "not" search operators As suggested by Keith in FLAG_PURE_NOT allows for expressions like: notmuch search NOT tag:inbox Note that this way a search like: notmuch search foobar NOT tag:inbox should not be written instead: notmuch search foobar AND NOT tag:inbox In my opinion, the latter feels more natural and is somewhat more explicit. It gives a better clue of what the search is about instead of assuming that an implicit AND operator is there.
2009-11-19 01:07:22 +01:00
Xapian::QueryParser::FLAG_WILDCARD |
Xapian::QueryParser::FLAG_PURE_NOT);
notmuch dump: Fix the sorting of results. To properly support sorting in notmuch_query we know use an Enquire object. We also throw in a QueryParser too, so we're really close to being able to support arbitrary full-text searches. I took a look at the supported QueryParser syntax and chose a set of flags for everything I like, (such as supporting Boolean operators in either case ("AND" or "and"), supporting phrase searching, supporting + and - to include/preclude terms, and supporting a trailing * on any term as a wildcard).
2009-10-21 09:35:56 +02:00
lib: Handle "*" as a query string to match all messages. This seems like a generally useful thing to support, (but the previous support through an empty string was not convenient for some users, (such as the command-line client).
2010-04-10 01:40:31 +02:00
if (strcmp (query_string, "") == 0 ||
strcmp (query_string, "*") == 0)
{
notmuch dump: Fix the sorting of results. To properly support sorting in notmuch_query we know use an Enquire object. We also throw in a QueryParser too, so we're really close to being able to support arbitrary full-text searches. I took a look at the supported QueryParser syntax and chose a set of flags for everything I like, (such as supporting Boolean operators in either case ("AND" or "and"), supporting phrase searching, supporting + and - to include/preclude terms, and supporting a trailing * on any term as a wildcard).
2009-10-21 09:35:56 +02:00
final_query = mail_query;
} else {
string_query = notmuch->query_parser->
parse_query (query_string, flags);
final_query = Xapian::Query (Xapian::Query::OP_AND,
mail_query, string_query);
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
}
lib: Make notmuch_query_search_messages set the exclude flag Add a flag NOTMUCH_MESSAGE_FLAG_EXCLUDED which is set by notmuch_query_search_messages for excluded messages. Also add an option omit_excluded_messages to the search that we do not want the excludes at all. This exclude flag will be added to notmuch_query_search threads in the next patch.
2012-03-01 23:30:37 +01:00
messages->base.excluded_doc_ids = NULL;
lib: add NOTMUCH_EXCLUDE_FLAG to notmuch_exclude_t Add NOTMUCH_EXCLUDE_FLAG to notmuch_exclude_t so that it can cover all four values of search --exclude in the cli. Previously the way to avoid any message being marked excluded was to pass in an empty list of excluded tags: since we now have an explicit option we might as well honour it. The enum is in a slightly strange order as the existing FALSE/TRUE options correspond to the new NOTMUCH_EXCLUDE_FLAG/NOTMUCH_EXCLUDE_TRUE options so this means we do not need to bump the version number. Indeed, an example of this is that the cli count and show still use FALSE/TRUE and still work.
2013-05-13 17:10:51 +02:00
if ((query->omit_excluded != NOTMUCH_EXCLUDE_FALSE) && (query->exclude_terms)) {
lib: Make notmuch_query_search_messages set the exclude flag Add a flag NOTMUCH_MESSAGE_FLAG_EXCLUDED which is set by notmuch_query_search_messages for excluded messages. Also add an option omit_excluded_messages to the search that we do not want the excludes at all. This exclude flag will be added to notmuch_query_search threads in the next patch.
2012-03-01 23:30:37 +01:00
exclude_query = _notmuch_exclude_tags (query, final_query);
lib: add NOTMUCH_EXCLUDE_FLAG to notmuch_exclude_t Add NOTMUCH_EXCLUDE_FLAG to notmuch_exclude_t so that it can cover all four values of search --exclude in the cli. Previously the way to avoid any message being marked excluded was to pass in an empty list of excluded tags: since we now have an explicit option we might as well honour it. The enum is in a slightly strange order as the existing FALSE/TRUE options correspond to the new NOTMUCH_EXCLUDE_FLAG/NOTMUCH_EXCLUDE_TRUE options so this means we do not need to bump the version number. Indeed, an example of this is that the cli count and show still use FALSE/TRUE and still work.
2013-05-13 17:10:51 +02:00
if (query->omit_excluded == NOTMUCH_EXCLUDE_TRUE ||
query->omit_excluded == NOTMUCH_EXCLUDE_ALL)
{
lib: Make notmuch_query_search_messages set the exclude flag Add a flag NOTMUCH_MESSAGE_FLAG_EXCLUDED which is set by notmuch_query_search_messages for excluded messages. Also add an option omit_excluded_messages to the search that we do not want the excludes at all. This exclude flag will be added to notmuch_query_search threads in the next patch.
2012-03-01 23:30:37 +01:00
final_query = Xapian::Query (Xapian::Query::OP_AND_NOT,
final_query, exclude_query);
lib: add NOTMUCH_EXCLUDE_FLAG to notmuch_exclude_t Add NOTMUCH_EXCLUDE_FLAG to notmuch_exclude_t so that it can cover all four values of search --exclude in the cli. Previously the way to avoid any message being marked excluded was to pass in an empty list of excluded tags: since we now have an explicit option we might as well honour it. The enum is in a slightly strange order as the existing FALSE/TRUE options correspond to the new NOTMUCH_EXCLUDE_FLAG/NOTMUCH_EXCLUDE_TRUE options so this means we do not need to bump the version number. Indeed, an example of this is that the cli count and show still use FALSE/TRUE and still work.
2013-05-13 17:10:51 +02:00
} else { /* NOTMUCH_EXCLUDE_FLAG */
lib: fix an exclude bug When the exclude tags contain a tag that does not occur anywhere in the Xapian database the exclusion fails. We modify the way the query is constructed to `work around' this. (In fact the new code is cleaner anyway.) It also seems to fix another exclusion failure bug reported by jrollins but we have not yet worked out why it helps in that case.
2012-03-14 13:26:54 +01:00
exclude_query = Xapian::Query (Xapian::Query::OP_AND,
exclude_query, final_query);
lib: Make notmuch_query_search_messages set the exclude flag Add a flag NOTMUCH_MESSAGE_FLAG_EXCLUDED which is set by notmuch_query_search_messages for excluded messages. Also add an option omit_excluded_messages to the search that we do not want the excludes at all. This exclude flag will be added to notmuch_query_search threads in the next patch.
2012-03-01 23:30:37 +01:00
enquire.set_weighting_scheme (Xapian::BoolWeight());
enquire.set_query (exclude_query);
mset = enquire.get_mset (0, notmuch->xapian_db->get_doccount ());
GArray *excluded_doc_ids = g_array_new (FALSE, FALSE, sizeof (unsigned int));
for (iterator = mset.begin (); iterator != mset.end (); iterator++) {
unsigned int doc_id = *iterator;
g_array_append_val (excluded_doc_ids, doc_id);
}
messages->base.excluded_doc_ids = talloc (messages, _notmuch_doc_id_set);
_notmuch_doc_id_set_init (query, messages->base.excluded_doc_ids,
excluded_doc_ids);
g_array_unref (excluded_doc_ids);
}
}
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
lib: Add support for automatically excluding tags from queries This is useful for tags like "deleted" and "spam" that people generally want to exclude from query results. These exclusions will be overridden if a tag is explicitly mentioned in a query.
2012-01-15 01:17:33 +01:00
lib: Explicitly set BoolWeight when searching. All notmuch searches currently sort by value (either date or message ID) so it's just wasted effort for Xapian to compute relevance values for each result. We now explicitly tell Xapian that we're uninterested in the relevance values.
2010-01-09 20:16:40 +01:00
enquire.set_weighting_scheme (Xapian::BoolWeight());
notmuch dump: Fix the sorting of results. To properly support sorting in notmuch_query we know use an Enquire object. We also throw in a QueryParser too, so we're really close to being able to support arbitrary full-text searches. I took a look at the supported QueryParser syntax and chose a set of flags for everything I like, (such as supporting Boolean operators in either case ("AND" or "and"), supporting phrase searching, supporting + and - to include/preclude terms, and supporting a trailing * on any term as a wildcard).
2009-10-21 09:35:56 +02:00
switch (query->sort) {
notmuch search: Change default search order to be newest messages first. This is what most people want for a _search_ command. It's often different for actually reading mail in an inbox, (where it makes more sense to have results displayed in chronological order), but in such a case, ther user is likely using an interface that can simply pass the --sort=oldest-first option to "notmuch search". Here we're also change the sort enum from NOTMUCH_SORT_DATE and NOTMUCH_SORT_DATE_REVERSE to NOTMUCH_SORT_OLDEST_FIRST and NOTMUCH_SORT_NEWEST_FIRST. Similarly we replace the --reverse option to "notmuch search" with two options: --sort=oldest-first and --sort=newest-first. Finally, these changes are all tracked in the emacs interface, (which has no change in its behavior).
2009-11-18 05:52:09 +01:00
case NOTMUCH_SORT_OLDEST_FIRST:
Shuffle the value numbers around in the database. First, it's nice that for now we don't have any users yet, so we can make incompatible changes to the database layout like this without causing trouble. ;-) There are a few reasons for this change. First, we now use value 0 uniformly as a timestamp for both mail and timestamp documents, (which lets us cleanup an ugly and fragile bare 0 in the add_value and get_value calls in the timestamp code). Second, I want to drop the thread value entirely, so putting it at the end of the list means we can drop it as compatible change in the future. (I almost want to drop the message-ID value too, but it's nice to be able to sort on it to get diff-able output from "notmuch dump".) But the thread value we never use as a value, (we would never sort on it, for example). And it's totally redundant with the thread terms we store already. So expect it to disappear soon.
2009-10-25 07:05:08 +01:00
enquire.set_sort_by_value (NOTMUCH_VALUE_TIMESTAMP, FALSE);
notmuch dump: Fix the sorting of results. To properly support sorting in notmuch_query we know use an Enquire object. We also throw in a QueryParser too, so we're really close to being able to support arbitrary full-text searches. I took a look at the supported QueryParser syntax and chose a set of flags for everything I like, (such as supporting Boolean operators in either case ("AND" or "and"), supporting phrase searching, supporting + and - to include/preclude terms, and supporting a trailing * on any term as a wildcard).
2009-10-21 09:35:56 +02:00
break;
notmuch search: Change default search order to be newest messages first. This is what most people want for a _search_ command. It's often different for actually reading mail in an inbox, (where it makes more sense to have results displayed in chronological order), but in such a case, ther user is likely using an interface that can simply pass the --sort=oldest-first option to "notmuch search". Here we're also change the sort enum from NOTMUCH_SORT_DATE and NOTMUCH_SORT_DATE_REVERSE to NOTMUCH_SORT_OLDEST_FIRST and NOTMUCH_SORT_NEWEST_FIRST. Similarly we replace the --reverse option to "notmuch search" with two options: --sort=oldest-first and --sort=newest-first. Finally, these changes are all tracked in the emacs interface, (which has no change in its behavior).
2009-11-18 05:52:09 +01:00
case NOTMUCH_SORT_NEWEST_FIRST:
Shuffle the value numbers around in the database. First, it's nice that for now we don't have any users yet, so we can make incompatible changes to the database layout like this without causing trouble. ;-) There are a few reasons for this change. First, we now use value 0 uniformly as a timestamp for both mail and timestamp documents, (which lets us cleanup an ugly and fragile bare 0 in the add_value and get_value calls in the timestamp code). Second, I want to drop the thread value entirely, so putting it at the end of the list means we can drop it as compatible change in the future. (I almost want to drop the message-ID value too, but it's nice to be able to sort on it to get diff-able output from "notmuch dump".) But the thread value we never use as a value, (we would never sort on it, for example). And it's totally redundant with the thread terms we store already. So expect it to disappear soon.
2009-10-25 07:05:08 +01:00
enquire.set_sort_by_value (NOTMUCH_VALUE_TIMESTAMP, TRUE);
notmuch dump: Fix the sorting of results. To properly support sorting in notmuch_query we know use an Enquire object. We also throw in a QueryParser too, so we're really close to being able to support arbitrary full-text searches. I took a look at the supported QueryParser syntax and chose a set of flags for everything I like, (such as supporting Boolean operators in either case ("AND" or "and"), supporting phrase searching, supporting + and - to include/preclude terms, and supporting a trailing * on any term as a wildcard).
2009-10-21 09:35:56 +02:00
break;
case NOTMUCH_SORT_MESSAGE_ID:
enquire.set_sort_by_value (NOTMUCH_VALUE_MESSAGE_ID, FALSE);
break;
query.cc: allow to return query results unsorted Previously, we always sorted the returned results by some string value, (newest-to-oldest by default), however in some cases (as when applying tags to a search result) we are not interested in any special order. This introduces a NOTMUCH_SORT_UNSORTED value that does just that. It is not used at the moment anywhere in the code. Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2010-04-16 08:38:46 +02:00
case NOTMUCH_SORT_UNSORTED:
break;
notmuch dump: Fix the sorting of results. To properly support sorting in notmuch_query we know use an Enquire object. We also throw in a QueryParser too, so we're really close to being able to support arbitrary full-text searches. I took a look at the supported QueryParser syntax and chose a set of flags for everything I like, (such as supporting Boolean operators in either case ("AND" or "and"), supporting phrase searching, supporting + and - to include/preclude terms, and supporting a trailing * on any term as a wildcard).
2009-10-21 09:35:56 +02:00
}
lib: Add exclude query debug output
2012-03-14 03:31:31 +01:00
if (_debug_query ()) {
fprintf (stderr, "Exclude query is:\n%s\n",
exclude_query.get_description ().c_str ());
lib: Expose query debug output via an environment variable Allow query debugging to be enabled at run-time by setting the NOTMUCH_DEBUG_QUERY environment variable to a non-empty string. Previously, enabling query debugging required recompiling, but parsed queries are often useful for tracking down bugs in situations where recompiling is inconvenient.
2012-03-14 03:31:30 +01:00
fprintf (stderr, "Final query is:\n%s\n",
final_query.get_description ().c_str ());
lib: Add exclude query debug output
2012-03-14 03:31:31 +01:00
}
Add debugging code for examining query strings. It's nice that Xapian provides a little function to print a textual representation of the entire query tree. So now, if you compile like so: make CFLAGS=-DDEBUG_QUERY then you get a nice output of the query string received by the query module, and the final query actually being sent to Xapian.
2009-10-25 06:18:20 +01:00
notmuch dump: Fix the sorting of results. To properly support sorting in notmuch_query we know use an Enquire object. We also throw in a QueryParser too, so we're really close to being able to support arbitrary full-text searches. I took a look at the supported QueryParser syntax and chose a set of flags for everything I like, (such as supporting Boolean operators in either case ("AND" or "and"), supporting phrase searching, supporting + and - to include/preclude terms, and supporting a trailing * on any term as a wildcard).
2009-10-21 09:35:56 +02:00
enquire.set_query (final_query);
lib/query: Drop the first and max_messages arguments from search_messages. These only existed to support the chunky-searching hack, but that was recently dropped anyway.
2009-11-24 05:25:13 +01:00
mset = enquire.get_mset (0, notmuch->xapian_db->get_doccount ());
notmuch dump: Fix the sorting of results. To properly support sorting in notmuch_query we know use an Enquire object. We also throw in a QueryParser too, so we're really close to being able to support arbitrary full-text searches. I took a look at the supported QueryParser syntax and chose a set of flags for everything I like, (such as supporting Boolean operators in either case ("AND" or "and"), supporting phrase searching, supporting + and - to include/preclude terms, and supporting a trailing * on any term as a wildcard).
2009-10-21 09:35:56 +02:00
lib/messages.c: Make message searches stream as well. Xapian provides an interator-based interface to all search results. So it was natural to make notmuch_messages_t be iterator-based as well. Which we did originally. But we ran into a problem when we added two APIs, (_get_replies and _get_toplevel_messages), that want to return a messages iterator that's *not* based on a Xapian search result. My original compromise was to use notmuch_message_list_t as the basis for all returned messages iterators in the public interface. This had the problem of introducing extra latency at the beginning of a search for messages, (the call would block while iterating over all results from Xapian, converting to a message list). In this commit, we remove that initial conversion and instead provide two alternate implementations of notmuch_messages_t (one on top of a Xapian iterator and one on top of a message list). With this change, I tested a "notmuch search" returning *many* results as previously taking about 7 seconds before results started appearing, and now taking only 2 seconds.
2009-11-24 06:47:24 +01:00
messages->iterator = mset.begin ();
messages->iterator_end = mset.end ();
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
lib: Ensure notmuch_query_search_messages returns NULL on an exception. Previously, this function may have segfaulted immediately after reporting the exception.
2010-04-24 16:25:12 +02:00
return &messages->base;
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
} catch (const Xapian::Error &error) {
Print information about where Xapian exception occurred. Previously, our Xapian exception reports where identical so they were hard to track down.
2009-11-22 03:41:30 +01:00
fprintf (stderr, "A Xapian exception occurred performing query: %s\n",
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
error.get_msg().c_str());
When a search query triggers a Xapian exception, log what the query was. In my script containing a series of queries to be run on new mail for setting up tags, it's nice to see which query I typed wrong. Signed-off-by: Eric Anholt <eric@anholt.net>
2009-11-20 14:00:42 +01:00
fprintf (stderr, "Query string was: %s\n", query->query_string);
Catch and optionally print about exception at database->flush. If an earlier exception occurred, then it's not unexpected for the flush to fail as well. So in that case, we'll silently catch the exception. Otherwise, make some noise about things going wrong at the time of flush.
2009-11-22 03:54:20 +01:00
notmuch->exception_reported = TRUE;
lib: Ensure notmuch_query_search_messages returns NULL on an exception. Previously, this function may have segfaulted immediately after reporting the exception.
2010-04-24 16:25:12 +02:00
talloc_free (messages);
return NULL;
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
}
lib/messages.c: Make message searches stream as well. Xapian provides an interator-based interface to all search results. So it was natural to make notmuch_messages_t be iterator-based as well. Which we did originally. But we ran into a problem when we added two APIs, (_get_replies and _get_toplevel_messages), that want to return a messages iterator that's *not* based on a Xapian search result. My original compromise was to use notmuch_message_list_t as the basis for all returned messages iterators in the public interface. This had the problem of introducing extra latency at the beginning of a search for messages, (the call would block while iterating over all results from Xapian, converting to a message list). In this commit, we remove that initial conversion and instead provide two alternate implementations of notmuch_messages_t (one on top of a Xapian iterator and one on top of a message list). With this change, I tested a "notmuch search" returning *many* results as previously taking about 7 seconds before results started appearing, and now taking only 2 seconds.
2009-11-24 06:47:24 +01:00
}
notmuch_bool_t
lib: Rename iterator functions to prepare for reverse iteration. We rename 'has_more' to 'valid' so that it can function whether iterating in a forward or reverse direction. We also rename 'advance' to 'move_to_next' to setup parallel naming with the proposed functions 'move_to_first', 'move_to_last', and 'move_to_previous'.
2010-03-09 18:22:29 +01:00
_notmuch_mset_messages_valid (notmuch_messages_t *messages)
lib/messages.c: Make message searches stream as well. Xapian provides an interator-based interface to all search results. So it was natural to make notmuch_messages_t be iterator-based as well. Which we did originally. But we ran into a problem when we added two APIs, (_get_replies and _get_toplevel_messages), that want to return a messages iterator that's *not* based on a Xapian search result. My original compromise was to use notmuch_message_list_t as the basis for all returned messages iterators in the public interface. This had the problem of introducing extra latency at the beginning of a search for messages, (the call would block while iterating over all results from Xapian, converting to a message list). In this commit, we remove that initial conversion and instead provide two alternate implementations of notmuch_messages_t (one on top of a Xapian iterator and one on top of a message list). With this change, I tested a "notmuch search" returning *many* results as previously taking about 7 seconds before results started appearing, and now taking only 2 seconds.
2009-11-24 06:47:24 +01:00
{
notmuch_mset_messages_t *mset_messages;
mset_messages = (notmuch_mset_messages_t *) messages;
return (mset_messages->iterator != mset_messages->iterator_end);
}
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
static Xapian::docid
_notmuch_mset_messages_get_doc_id (notmuch_messages_t *messages)
{
notmuch_mset_messages_t *mset_messages;
mset_messages = (notmuch_mset_messages_t *) messages;
if (! _notmuch_mset_messages_valid (&mset_messages->base))
return 0;
return *mset_messages->iterator;
}
lib/messages.c: Make message searches stream as well. Xapian provides an interator-based interface to all search results. So it was natural to make notmuch_messages_t be iterator-based as well. Which we did originally. But we ran into a problem when we added two APIs, (_get_replies and _get_toplevel_messages), that want to return a messages iterator that's *not* based on a Xapian search result. My original compromise was to use notmuch_message_list_t as the basis for all returned messages iterators in the public interface. This had the problem of introducing extra latency at the beginning of a search for messages, (the call would block while iterating over all results from Xapian, converting to a message list). In this commit, we remove that initial conversion and instead provide two alternate implementations of notmuch_messages_t (one on top of a Xapian iterator and one on top of a message list). With this change, I tested a "notmuch search" returning *many* results as previously taking about 7 seconds before results started appearing, and now taking only 2 seconds.
2009-11-24 06:47:24 +01:00
notmuch_message_t *
_notmuch_mset_messages_get (notmuch_messages_t *messages)
{
notmuch_message_t *message;
Xapian::docid doc_id;
notmuch_private_status_t status;
notmuch_mset_messages_t *mset_messages;
mset_messages = (notmuch_mset_messages_t *) messages;
lib: Rename iterator functions to prepare for reverse iteration. We rename 'has_more' to 'valid' so that it can function whether iterating in a forward or reverse direction. We also rename 'advance' to 'move_to_next' to setup parallel naming with the proposed functions 'move_to_first', 'move_to_last', and 'move_to_previous'.
2010-03-09 18:22:29 +01:00
if (! _notmuch_mset_messages_valid (&mset_messages->base))
lib/messages.c: Make message searches stream as well. Xapian provides an interator-based interface to all search results. So it was natural to make notmuch_messages_t be iterator-based as well. Which we did originally. But we ran into a problem when we added two APIs, (_get_replies and _get_toplevel_messages), that want to return a messages iterator that's *not* based on a Xapian search result. My original compromise was to use notmuch_message_list_t as the basis for all returned messages iterators in the public interface. This had the problem of introducing extra latency at the beginning of a search for messages, (the call would block while iterating over all results from Xapian, converting to a message list). In this commit, we remove that initial conversion and instead provide two alternate implementations of notmuch_messages_t (one on top of a Xapian iterator and one on top of a message list). With this change, I tested a "notmuch search" returning *many* results as previously taking about 7 seconds before results started appearing, and now taking only 2 seconds.
2009-11-24 06:47:24 +01:00
return NULL;
doc_id = *mset_messages->iterator;
message = _notmuch_message_create (mset_messages,
mset_messages->notmuch, doc_id,
&status);
if (message == NULL &&
status == NOTMUCH_PRIVATE_STATUS_NO_DOCUMENT_FOUND)
{
INTERNAL_ERROR ("a messages iterator contains a non-existent document ID.\n");
}
lib: Make notmuch_query_search_messages set the exclude flag Add a flag NOTMUCH_MESSAGE_FLAG_EXCLUDED which is set by notmuch_query_search_messages for excluded messages. Also add an option omit_excluded_messages to the search that we do not want the excludes at all. This exclude flag will be added to notmuch_query_search threads in the next patch.
2012-03-01 23:30:37 +01:00
if (messages->excluded_doc_ids &&
_notmuch_doc_id_set_contains (messages->excluded_doc_ids, doc_id))
notmuch_message_set_flag (message, NOTMUCH_MESSAGE_FLAG_EXCLUDED, TRUE);
lib/messages.c: Make message searches stream as well. Xapian provides an interator-based interface to all search results. So it was natural to make notmuch_messages_t be iterator-based as well. Which we did originally. But we ran into a problem when we added two APIs, (_get_replies and _get_toplevel_messages), that want to return a messages iterator that's *not* based on a Xapian search result. My original compromise was to use notmuch_message_list_t as the basis for all returned messages iterators in the public interface. This had the problem of introducing extra latency at the beginning of a search for messages, (the call would block while iterating over all results from Xapian, converting to a message list). In this commit, we remove that initial conversion and instead provide two alternate implementations of notmuch_messages_t (one on top of a Xapian iterator and one on top of a message list). With this change, I tested a "notmuch search" returning *many* results as previously taking about 7 seconds before results started appearing, and now taking only 2 seconds.
2009-11-24 06:47:24 +01:00
return message;
}
void
lib: Rename iterator functions to prepare for reverse iteration. We rename 'has_more' to 'valid' so that it can function whether iterating in a forward or reverse direction. We also rename 'advance' to 'move_to_next' to setup parallel naming with the proposed functions 'move_to_first', 'move_to_last', and 'move_to_previous'.
2010-03-09 18:22:29 +01:00
_notmuch_mset_messages_move_to_next (notmuch_messages_t *messages)
lib/messages.c: Make message searches stream as well. Xapian provides an interator-based interface to all search results. So it was natural to make notmuch_messages_t be iterator-based as well. Which we did originally. But we ran into a problem when we added two APIs, (_get_replies and _get_toplevel_messages), that want to return a messages iterator that's *not* based on a Xapian search result. My original compromise was to use notmuch_message_list_t as the basis for all returned messages iterators in the public interface. This had the problem of introducing extra latency at the beginning of a search for messages, (the call would block while iterating over all results from Xapian, converting to a message list). In this commit, we remove that initial conversion and instead provide two alternate implementations of notmuch_messages_t (one on top of a Xapian iterator and one on top of a message list). With this change, I tested a "notmuch search" returning *many* results as previously taking about 7 seconds before results started appearing, and now taking only 2 seconds.
2009-11-24 06:47:24 +01:00
{
notmuch_mset_messages_t *mset_messages;
mset_messages = (notmuch_mset_messages_t *) messages;
mset_messages->iterator++;
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
}
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
static notmuch_bool_t
_notmuch_doc_id_set_init (void *ctx,
notmuch_doc_id_set_t *doc_ids,
Simplify _notmuch_doc_id_set_init interface. Don't require the caller of _notmuch_doc_id_set_init to pass in a correct bound; instead compute it from the array. This simplifies the caller and makes this interface easier to use correctly.
2011-01-31 03:01:56 +01:00
GArray *arr)
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
{
Simplify _notmuch_doc_id_set_init interface. Don't require the caller of _notmuch_doc_id_set_init to pass in a correct bound; instead compute it from the array. This simplifies the caller and makes this interface easier to use correctly.
2011-01-31 03:01:56 +01:00
unsigned int max = 0;
bitmap:improve memory usage using CHAR_BITS and unsigned CHAR Using char instead of int allows for simpler definitions of the DOCIDSET macros so the code is easier to understand and consistent with respect to memory-usage. Estimated reduction of memory-usage for bitmap about 8 times.
2013-02-13 16:32:57 +01:00
unsigned char *bitmap;
Simplify _notmuch_doc_id_set_init interface. Don't require the caller of _notmuch_doc_id_set_init to pass in a correct bound; instead compute it from the array. This simplifies the caller and makes this interface easier to use correctly.
2011-01-31 03:01:56 +01:00
for (unsigned int i = 0; i < arr->len; i++)
max = MAX(max, g_array_index (arr, unsigned int, i));
bitmap:improve memory usage using CHAR_BITS and unsigned CHAR Using char instead of int allows for simpler definitions of the DOCIDSET macros so the code is easier to understand and consistent with respect to memory-usage. Estimated reduction of memory-usage for bitmap about 8 times.
2013-02-13 16:32:57 +01:00
bitmap = talloc_zero_array (ctx, unsigned char, DOCIDSET_WORD(max) + 1);
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
if (bitmap == NULL)
return FALSE;
doc_ids->bitmap = bitmap;
Simplify _notmuch_doc_id_set_init interface. Don't require the caller of _notmuch_doc_id_set_init to pass in a correct bound; instead compute it from the array. This simplifies the caller and makes this interface easier to use correctly.
2011-01-31 03:01:56 +01:00
doc_ids->bound = max + 1;
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
for (unsigned int i = 0; i < arr->len; i++) {
Remove code repetition in the doc ID bitmap code. Remove the repeated "sizeof (doc_ids->bitmap[0])" that bothered cworth by instead defining macros to compute the word and bit offset of a given bit in the doc ID set bitmap.
2011-01-31 02:58:27 +01:00
unsigned int doc_id = g_array_index (arr, unsigned int, i);
bitmap[DOCIDSET_WORD(doc_id)] |= 1 << DOCIDSET_BIT(doc_id);
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
}
return TRUE;
}
notmuch_bool_t
_notmuch_doc_id_set_contains (notmuch_doc_id_set_t *doc_ids,
unsigned int doc_id)
{
if (doc_id >= doc_ids->bound)
return FALSE;
Remove code repetition in the doc ID bitmap code. Remove the repeated "sizeof (doc_ids->bitmap[0])" that bothered cworth by instead defining macros to compute the word and bit offset of a given bit in the doc ID set bitmap.
2011-01-31 02:58:27 +01:00
return doc_ids->bitmap[DOCIDSET_WORD(doc_id)] & (1 << DOCIDSET_BIT(doc_id));
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
}
void
_notmuch_doc_id_set_remove (notmuch_doc_id_set_t *doc_ids,
unsigned int doc_id)
{
if (doc_id < doc_ids->bound)
Remove code repetition in the doc ID bitmap code. Remove the repeated "sizeof (doc_ids->bitmap[0])" that bothered cworth by instead defining macros to compute the word and bit offset of a given bit in the doc ID set bitmap.
2011-01-31 02:58:27 +01:00
doc_ids->bitmap[DOCIDSET_WORD(doc_id)] &= ~(1 << DOCIDSET_BIT(doc_id));
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
}
Fix memory leak in notmuch_thread_results_t If we were using a talloc-based resizing array then this wouldn't have happened. Of course, thanks to valgrind for catching this.
2009-10-26 22:02:58 +01:00
/* Glib objects force use to use a talloc destructor as well, (but not
Rename message_results/thread_results to messages/threads. Shorter naming without being any less clear. A definite win.
2009-11-01 00:32:30 +01:00
* nearly as ugly as the for messages due to C++ objects). At
Fix memory leak in notmuch_thread_results_t If we were using a talloc-based resizing array then this wouldn't have happened. Of course, thanks to valgrind for catching this.
2009-10-26 22:02:58 +01:00
* this point, I'd really like to have some talloc-friendly
* equivalents for the few pieces of glib that I'm using. */
static int
Rename message_results/thread_results to messages/threads. Shorter naming without being any less clear. A definite win.
2009-11-01 00:32:30 +01:00
_notmuch_threads_destructor (notmuch_threads_t *threads)
Fix memory leak in notmuch_thread_results_t If we were using a talloc-based resizing array then this wouldn't have happened. Of course, thanks to valgrind for catching this.
2009-10-26 22:02:58 +01:00
{
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
if (threads->doc_ids)
g_array_unref (threads->doc_ids);
Fix memory leak in notmuch_thread_results_t If we were using a talloc-based resizing array then this wouldn't have happened. Of course, thanks to valgrind for catching this.
2009-10-26 22:02:58 +01:00
return 0;
}
Rename message_results/thread_results to messages/threads. Shorter naming without being any less clear. A definite win.
2009-11-01 00:32:30 +01:00
notmuch_threads_t *
notmuch search: Remove the chunked-searching hack. This was a poor workaround around the fact that the existing notmuch_threads_t object is implemented poorly. It's got a fine iterartor-based interface, but the implementation does all of the work up-front in _create rather than doing the work incrementally while iterating. So to start fixing this, first get rid of all the hacks we had working around this. This drops the --first and --max-threads options from the search command, (but hopefully nobody was using them anyway---notmuch.el certainly wasn't).
2009-11-24 05:12:57 +01:00
notmuch_query_search_threads (notmuch_query_t *query)
Add an initial implementation of a notmuch_thread_t object. We've now got a new notmuch_query_search_threads and a notmuch_threads_result_t iterator. The thread object itself doesn't do much yet, (just allows one to get the thread_id), but that's at least enough to see that "notmuch search" is actually doing something now, (since it has been converted to print thread IDs instead of message IDs). And maybe that's all we need. Getting the messages belonging to a thread is as simple as a notmuch_query_search_messages with a string of "thread:<thread-id>". Though it would be convenient to add notmuch_thread_get_messages which could use the existing notmuch_message_results_t iterator. Now we just need an implementation of "notmuch show" and we'll have something somewhat usable.
2009-10-26 07:12:20 +01:00
{
Rename message_results/thread_results to messages/threads. Shorter naming without being any less clear. A definite win.
2009-11-01 00:32:30 +01:00
notmuch_threads_t *threads;
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
notmuch_messages_t *messages;
Add an initial implementation of a notmuch_thread_t object. We've now got a new notmuch_query_search_threads and a notmuch_threads_result_t iterator. The thread object itself doesn't do much yet, (just allows one to get the thread_id), but that's at least enough to see that "notmuch search" is actually doing something now, (since it has been converted to print thread IDs instead of message IDs). And maybe that's all we need. Getting the messages belonging to a thread is as simple as a notmuch_query_search_messages with a string of "thread:<thread-id>". Though it would be convenient to add notmuch_thread_get_messages which could use the existing notmuch_message_results_t iterator. Now we just need an implementation of "notmuch show" and we'll have something somewhat usable.
2009-10-26 07:12:20 +01:00
Rename message_results/thread_results to messages/threads. Shorter naming without being any less clear. A definite win.
2009-11-01 00:32:30 +01:00
threads = talloc (query, notmuch_threads_t);
if (threads == NULL)
Add an initial implementation of a notmuch_thread_t object. We've now got a new notmuch_query_search_threads and a notmuch_threads_result_t iterator. The thread object itself doesn't do much yet, (just allows one to get the thread_id), but that's at least enough to see that "notmuch search" is actually doing something now, (since it has been converted to print thread IDs instead of message IDs). And maybe that's all we need. Getting the messages belonging to a thread is as simple as a notmuch_query_search_messages with a string of "thread:<thread-id>". Though it would be convenient to add notmuch_thread_get_messages which could use the existing notmuch_message_results_t iterator. Now we just need an implementation of "notmuch show" and we'll have something somewhat usable.
2009-10-26 07:12:20 +01:00
return NULL;
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
threads->doc_ids = NULL;
talloc_set_destructor (threads, _notmuch_threads_destructor);
Add an initial implementation of a notmuch_thread_t object. We've now got a new notmuch_query_search_threads and a notmuch_threads_result_t iterator. The thread object itself doesn't do much yet, (just allows one to get the thread_id), but that's at least enough to see that "notmuch search" is actually doing something now, (since it has been converted to print thread IDs instead of message IDs). And maybe that's all we need. Getting the messages belonging to a thread is as simple as a notmuch_query_search_messages with a string of "thread:<thread-id>". Though it would be convenient to add notmuch_thread_get_messages which could use the existing notmuch_message_results_t iterator. Now we just need an implementation of "notmuch show" and we'll have something somewhat usable.
2009-10-26 07:12:20 +01:00
lib/query: Fix notmuch_threads_t to stream results rather than blocking. Previously, notmuch_query_search_threads would do all the work, so the caller would block until all results were processed. Now, we do the work as we go, as the caller iterates with notmuch_threads_next. This means that once results start coming back from "notmuch search" they just keep continually streaming. There's still some initial blocking before the first results appear because the notmuch_messages_t object has the same bug (for now).
2009-11-24 05:18:57 +01:00
threads->query = query;
notmuch_query_search_threads: Avoid returning more threads than asked for. I thought it would be safe enough to return a few extra threads, (since we happened to already get the relevant messages out of the database). The problem is that then requires the caller to carefully read the number of threads returned and adjust its next "first" value accordingly. The interface is much simpler to use if we simply return exactly what is asked for and no more.
2009-11-13 05:31:22 +01:00
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
messages = notmuch_query_search_messages (query);
if (messages == NULL) {
lib: Fix notmuch_query_search_threads to return NULL on any Xapian exception. Previously, if the underlying search_messages hit an exception and returned NULL, this function would ignore that and return a non-NULL, (but empty) threads object. Fix this to properly propagate the error.
2010-10-23 02:55:07 +02:00
talloc_free (threads);
return NULL;
}
libnotmuch: Underlying support for doing partial-results searches. The library interface now allows the caller to do incremental searches, (such as one page of results at a time). Next we'll just need to hook this up to "notmuch search" and the emacs interface.
2009-11-13 01:47:27 +01:00
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
threads->doc_ids = g_array_new (FALSE, FALSE, sizeof (unsigned int));
while (notmuch_messages_valid (messages)) {
unsigned int doc_id = _notmuch_mset_messages_get_doc_id (messages);
g_array_append_val (threads->doc_ids, doc_id);
notmuch_messages_move_to_next (messages);
}
threads->doc_id_pos = 0;
Add an initial implementation of a notmuch_thread_t object. We've now got a new notmuch_query_search_threads and a notmuch_threads_result_t iterator. The thread object itself doesn't do much yet, (just allows one to get the thread_id), but that's at least enough to see that "notmuch search" is actually doing something now, (since it has been converted to print thread IDs instead of message IDs). And maybe that's all we need. Getting the messages belonging to a thread is as simple as a notmuch_query_search_messages with a string of "thread:<thread-id>". Though it would be convenient to add notmuch_thread_get_messages which could use the existing notmuch_message_results_t iterator. Now we just need an implementation of "notmuch show" and we'll have something somewhat usable.
2009-10-26 07:12:20 +01:00
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
talloc_free (messages);
if (! _notmuch_doc_id_set_init (threads, &threads->match_set,
Simplify _notmuch_doc_id_set_init interface. Don't require the caller of _notmuch_doc_id_set_init to pass in a correct bound; instead compute it from the array. This simplifies the caller and makes this interface easier to use correctly.
2011-01-31 03:01:56 +01:00
threads->doc_ids)) {
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
talloc_free (threads);
return NULL;
}
Add an initial implementation of a notmuch_thread_t object. We've now got a new notmuch_query_search_threads and a notmuch_threads_result_t iterator. The thread object itself doesn't do much yet, (just allows one to get the thread_id), but that's at least enough to see that "notmuch search" is actually doing something now, (since it has been converted to print thread IDs instead of message IDs). And maybe that's all we need. Getting the messages belonging to a thread is as simple as a notmuch_query_search_messages with a string of "thread:<thread-id>". Though it would be convenient to add notmuch_thread_get_messages which could use the existing notmuch_message_results_t iterator. Now we just need an implementation of "notmuch show" and we'll have something somewhat usable.
2009-10-26 07:12:20 +01:00
Rename message_results/thread_results to messages/threads. Shorter naming without being any less clear. A definite win.
2009-11-01 00:32:30 +01:00
return threads;
Add an initial implementation of a notmuch_thread_t object. We've now got a new notmuch_query_search_threads and a notmuch_threads_result_t iterator. The thread object itself doesn't do much yet, (just allows one to get the thread_id), but that's at least enough to see that "notmuch search" is actually doing something now, (since it has been converted to print thread IDs instead of message IDs). And maybe that's all we need. Getting the messages belonging to a thread is as simple as a notmuch_query_search_messages with a string of "thread:<thread-id>". Though it would be convenient to add notmuch_thread_get_messages which could use the existing notmuch_message_results_t iterator. Now we just need an implementation of "notmuch show" and we'll have something somewhat usable.
2009-10-26 07:12:20 +01:00
}
Implement 'notmuch dump'. This is a fairly big milestone for notmuch. It's our first command to do anything besides building the index, so it proves we can actually read valid results out from the index. It also puts in place almost all of the API and infrastructure we will need to allow searching of the database. Finally, with this change we are now using talloc inside of notmuch which is truly a delight to use. And now that I figured out how to use C++ objects with talloc allocation, (it requires grotty parts of C++ such as "placement new" and "explicit destructors"), we are valgrind-clean for "notmuch dump", (as in "no leaks are possible").
2009-10-21 06:03:30 +02:00
void
notmuch_query_destroy (notmuch_query_t *query)
{
talloc_free (query);
}
Add an initial implementation of a notmuch_thread_t object. We've now got a new notmuch_query_search_threads and a notmuch_threads_result_t iterator. The thread object itself doesn't do much yet, (just allows one to get the thread_id), but that's at least enough to see that "notmuch search" is actually doing something now, (since it has been converted to print thread IDs instead of message IDs). And maybe that's all we need. Getting the messages belonging to a thread is as simple as a notmuch_query_search_messages with a string of "thread:<thread-id>". Though it would be convenient to add notmuch_thread_get_messages which could use the existing notmuch_message_results_t iterator. Now we just need an implementation of "notmuch show" and we'll have something somewhat usable.
2009-10-26 07:12:20 +01:00
notmuch_bool_t
lib: Rename iterator functions to prepare for reverse iteration. We rename 'has_more' to 'valid' so that it can function whether iterating in a forward or reverse direction. We also rename 'advance' to 'move_to_next' to setup parallel naming with the proposed functions 'move_to_first', 'move_to_last', and 'move_to_previous'.
2010-03-09 18:22:29 +01:00
notmuch_threads_valid (notmuch_threads_t *threads)
Add an initial implementation of a notmuch_thread_t object. We've now got a new notmuch_query_search_threads and a notmuch_threads_result_t iterator. The thread object itself doesn't do much yet, (just allows one to get the thread_id), but that's at least enough to see that "notmuch search" is actually doing something now, (since it has been converted to print thread IDs instead of message IDs). And maybe that's all we need. Getting the messages belonging to a thread is as simple as a notmuch_query_search_messages with a string of "thread:<thread-id>". Though it would be convenient to add notmuch_thread_get_messages which could use the existing notmuch_message_results_t iterator. Now we just need an implementation of "notmuch show" and we'll have something somewhat usable.
2009-10-26 07:12:20 +01:00
{
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
unsigned int doc_id;
lib/query: Fix notmuch_threads_t to stream results rather than blocking. Previously, notmuch_query_search_threads would do all the work, so the caller would block until all results were processed. Now, we do the work as we go, as the caller iterates with notmuch_threads_next. This means that once results start coming back from "notmuch search" they just keep continually streaming. There's still some initial blocking before the first results appear because the notmuch_messages_t object has the same bug (for now).
2009-11-24 05:18:57 +01:00
lib: make notmuch_threads_valid return FALSE when passed NULL Without this patch, the example code in the header docs crashes for certain invalid queries (see id:871u00oimv.fsf@approx.mit.edu)
2014-01-23 13:24:00 +01:00
if (! threads)
return FALSE;
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
while (threads->doc_id_pos < threads->doc_ids->len) {
doc_id = g_array_index (threads->doc_ids, unsigned int,
threads->doc_id_pos);
if (_notmuch_doc_id_set_contains (&threads->match_set, doc_id))
break;
lib/query: Fix notmuch_threads_t to stream results rather than blocking. Previously, notmuch_query_search_threads would do all the work, so the caller would block until all results were processed. Now, we do the work as we go, as the caller iterates with notmuch_threads_next. This means that once results start coming back from "notmuch search" they just keep continually streaming. There's still some initial blocking before the first results appear because the notmuch_messages_t object has the same bug (for now).
2009-11-24 05:18:57 +01:00
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
threads->doc_id_pos++;
lib/query: Fix notmuch_threads_t to stream results rather than blocking. Previously, notmuch_query_search_threads would do all the work, so the caller would block until all results were processed. Now, we do the work as we go, as the caller iterates with notmuch_threads_next. This means that once results start coming back from "notmuch search" they just keep continually streaming. There's still some initial blocking before the first results appear because the notmuch_messages_t object has the same bug (for now).
2009-11-24 05:18:57 +01:00
}
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
return threads->doc_id_pos < threads->doc_ids->len;
Add an initial implementation of a notmuch_thread_t object. We've now got a new notmuch_query_search_threads and a notmuch_threads_result_t iterator. The thread object itself doesn't do much yet, (just allows one to get the thread_id), but that's at least enough to see that "notmuch search" is actually doing something now, (since it has been converted to print thread IDs instead of message IDs). And maybe that's all we need. Getting the messages belonging to a thread is as simple as a notmuch_query_search_messages with a string of "thread:<thread-id>". Though it would be convenient to add notmuch_thread_get_messages which could use the existing notmuch_message_results_t iterator. Now we just need an implementation of "notmuch show" and we'll have something somewhat usable.
2009-10-26 07:12:20 +01:00
}
notmuch_thread_t *
Rename message_results/thread_results to messages/threads. Shorter naming without being any less clear. A definite win.
2009-11-01 00:32:30 +01:00
notmuch_threads_get (notmuch_threads_t *threads)
Add an initial implementation of a notmuch_thread_t object. We've now got a new notmuch_query_search_threads and a notmuch_threads_result_t iterator. The thread object itself doesn't do much yet, (just allows one to get the thread_id), but that's at least enough to see that "notmuch search" is actually doing something now, (since it has been converted to print thread IDs instead of message IDs). And maybe that's all we need. Getting the messages belonging to a thread is as simple as a notmuch_query_search_messages with a string of "thread:<thread-id>". Though it would be convenient to add notmuch_thread_get_messages which could use the existing notmuch_message_results_t iterator. Now we just need an implementation of "notmuch show" and we'll have something somewhat usable.
2009-10-26 07:12:20 +01:00
{
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
unsigned int doc_id;
lib: Rename iterator functions to prepare for reverse iteration. We rename 'has_more' to 'valid' so that it can function whether iterating in a forward or reverse direction. We also rename 'advance' to 'move_to_next' to setup parallel naming with the proposed functions 'move_to_first', 'move_to_last', and 'move_to_previous'.
2010-03-09 18:22:29 +01:00
if (! notmuch_threads_valid (threads))
results_get: Fix to return NULL if past the end of the results We had documented both notmuch_thread_results_get and notmuch_message_results_get to return NULL if (! has_more) but we hadn't actually implemented that. Fix.
2009-10-26 13:14:51 +01:00
return NULL;
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
doc_id = g_array_index (threads->doc_ids, unsigned int,
threads->doc_id_pos);
lib/query: Fix notmuch_threads_t to stream results rather than blocking. Previously, notmuch_query_search_threads would do all the work, so the caller would block until all results were processed. Now, we do the work as we go, as the caller iterates with notmuch_threads_next. This means that once results start coming back from "notmuch search" they just keep continually streaming. There's still some initial blocking before the first results appear because the notmuch_messages_t object has the same bug (for now).
2009-11-24 05:18:57 +01:00
return _notmuch_thread_create (threads->query,
threads->query->notmuch,
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
doc_id,
&threads->match_set,
lib: Add the exclude flag to notmuch_query_search_threads Add the NOTMUCH_MESSAGE_FLAG_EXCLUDED flag to notmuch_query_search_threads. Implemented by inspecting the tags directly in _notmuch_thread_create/_thread_add_message rather than as a Xapian query for speed reasons. Note notmuch_thread_get_matched_messages now returns the number of non-excluded matching messages. This API is not totally desirable but fixing it means breaking binary compatibility so we delay that.
2012-03-01 23:30:38 +01:00
threads->query->exclude_terms,
lib: add --exclude=all option Adds a exclude all option to the lib which means that excluded messages are completely ignored (as if they had actually been deleted).
2013-05-11 21:50:02 +02:00
threads->query->omit_excluded,
Name thread based on matching msgs instead of first msg. At the moment all threads are named based on the name of the first message in the thread. However, this can cause problems if people either start new threads by replying-all (as unfortunately, many out there do) or change the subject of their mails to reflect a shift in a thread on a list. This patch names threads based on (a) matches for the query, and (b) the search order. If the search order is oldest-first (as in the default inbox) it chooses the oldest matching message as the subject. If the search order is newest-first it chooses the newest one. Reply prefixes ("Re: ", "Aw: ", "Sv: ", "Vs: ") are ignored (case-insensitively) so a Re: won't change the subject. Note that this adds a "sort" argument to _notmuch_thread_create and _thread_add_matched_message, so that when constructing the thread we can be aware of the sort order. Signed-off-by: Jesse Rosenthal <jrosenthal@jhu.edu>
2010-04-17 19:59:22 +02:00
threads->query->sort);
Add an initial implementation of a notmuch_thread_t object. We've now got a new notmuch_query_search_threads and a notmuch_threads_result_t iterator. The thread object itself doesn't do much yet, (just allows one to get the thread_id), but that's at least enough to see that "notmuch search" is actually doing something now, (since it has been converted to print thread IDs instead of message IDs). And maybe that's all we need. Getting the messages belonging to a thread is as simple as a notmuch_query_search_messages with a string of "thread:<thread-id>". Though it would be convenient to add notmuch_thread_get_messages which could use the existing notmuch_message_results_t iterator. Now we just need an implementation of "notmuch show" and we'll have something somewhat usable.
2009-10-26 07:12:20 +01:00
}
void
lib: Rename iterator functions to prepare for reverse iteration. We rename 'has_more' to 'valid' so that it can function whether iterating in a forward or reverse direction. We also rename 'advance' to 'move_to_next' to setup parallel naming with the proposed functions 'move_to_first', 'move_to_last', and 'move_to_previous'.
2010-03-09 18:22:29 +01:00
notmuch_threads_move_to_next (notmuch_threads_t *threads)
Add an initial implementation of a notmuch_thread_t object. We've now got a new notmuch_query_search_threads and a notmuch_threads_result_t iterator. The thread object itself doesn't do much yet, (just allows one to get the thread_id), but that's at least enough to see that "notmuch search" is actually doing something now, (since it has been converted to print thread IDs instead of message IDs). And maybe that's all we need. Getting the messages belonging to a thread is as simple as a notmuch_query_search_messages with a string of "thread:<thread-id>". Though it would be convenient to add notmuch_thread_get_messages which could use the existing notmuch_message_results_t iterator. Now we just need an implementation of "notmuch show" and we'll have something somewhat usable.
2009-10-26 07:12:20 +01:00
{
Optimize thread search using matched docid sets. This reduces thread search's 1+2t Xapian queries (where t is the number of matched threads) to 1+t queries and constructs exactly one notmuch_message_t for each message instead of 2 to 3. notmuch_query_search_threads eagerly fetches the docids of all messages matching the user query instead of lazily constructing message objects and fetching thread ID's from term lists. _notmuch_thread_create takes a seed docid and the set of all matched docids and uses a single Xapian query to expand this docid to its containing thread, using the matched docid set to determine which messages in the thread match the user query instead of using a second Xapian query. This reduces the amount of time required to load my inbox from 4.523 seconds to 3.025 seconds (1.5X faster).
2010-11-17 20:28:26 +01:00
threads->doc_id_pos++;
Add an initial implementation of a notmuch_thread_t object. We've now got a new notmuch_query_search_threads and a notmuch_threads_result_t iterator. The thread object itself doesn't do much yet, (just allows one to get the thread_id), but that's at least enough to see that "notmuch search" is actually doing something now, (since it has been converted to print thread IDs instead of message IDs). And maybe that's all we need. Getting the messages belonging to a thread is as simple as a notmuch_query_search_messages with a string of "thread:<thread-id>". Though it would be convenient to add notmuch_thread_get_messages which could use the existing notmuch_message_results_t iterator. Now we just need an implementation of "notmuch show" and we'll have something somewhat usable.
2009-10-26 07:12:20 +01:00
}
void
Rename message_results/thread_results to messages/threads. Shorter naming without being any less clear. A definite win.
2009-11-01 00:32:30 +01:00
notmuch_threads_destroy (notmuch_threads_t *threads)
Add an initial implementation of a notmuch_thread_t object. We've now got a new notmuch_query_search_threads and a notmuch_threads_result_t iterator. The thread object itself doesn't do much yet, (just allows one to get the thread_id), but that's at least enough to see that "notmuch search" is actually doing something now, (since it has been converted to print thread IDs instead of message IDs). And maybe that's all we need. Getting the messages belonging to a thread is as simple as a notmuch_query_search_messages with a string of "thread:<thread-id>". Though it would be convenient to add notmuch_thread_get_messages which could use the existing notmuch_message_results_t iterator. Now we just need an implementation of "notmuch show" and we'll have something somewhat usable.
2009-10-26 07:12:20 +01:00
{
Rename message_results/thread_results to messages/threads. Shorter naming without being any less clear. A definite win.
2009-11-01 00:32:30 +01:00
talloc_free (threads);
Add an initial implementation of a notmuch_thread_t object. We've now got a new notmuch_query_search_threads and a notmuch_threads_result_t iterator. The thread object itself doesn't do much yet, (just allows one to get the thread_id), but that's at least enough to see that "notmuch search" is actually doing something now, (since it has been converted to print thread IDs instead of message IDs). And maybe that's all we need. Getting the messages belonging to a thread is as simple as a notmuch_query_search_messages with a string of "thread:<thread-id>". Though it would be convenient to add notmuch_thread_get_messages which could use the existing notmuch_message_results_t iterator. Now we just need an implementation of "notmuch show" and we'll have something somewhat usable.
2009-10-26 07:12:20 +01:00
}
Add 'notmuch count' command to show the count of matching messages Getting the count of matching threads or messages is a fairly expensive operation. Xapian provides a very efficient mechanism that returns an approximate value, so use that for this new command. This returns the number of matching messages, not threads, as that is cheap to compute. Signed-off-by: Keith Packard <keithp@keithp.com>
2009-11-21 08:15:07 +01:00
unsigned
notmuch_query_count_messages (notmuch_query_t *query)
{
notmuch_database_t *notmuch = query->notmuch;
const char *query_string = query->query_string;
Silence compiler warning by initializing a variable. If Xapian threw an exception on notmuch_query_count_messages the count variable could be used uninitialized. Initialize count to solve the problem. Signed-off-by: Jeffrey C. Ollie <jeff@ocjtech.us>
2009-11-25 14:01:36 +01:00
Xapian::doccount count = 0;
Add 'notmuch count' command to show the count of matching messages Getting the count of matching threads or messages is a fairly expensive operation. Xapian provides a very efficient mechanism that returns an approximate value, so use that for this new command. This returns the number of matching messages, not threads, as that is cheap to compute. Signed-off-by: Keith Packard <keithp@keithp.com>
2009-11-21 08:15:07 +01:00
try {
Xapian::Enquire enquire (*notmuch->xapian_db);
Xapian::Query mail_query (talloc_asprintf (query, "%s%s",
_find_prefix ("type"),
"mail"));
lib: Rearrange the exclude code in query.cc Slightly refactor the exclude code to give the callers access to the exclude query itself. There should be no functional change.
2012-03-01 23:30:36 +01:00
Xapian::Query string_query, final_query, exclude_query;
Add 'notmuch count' command to show the count of matching messages Getting the count of matching threads or messages is a fairly expensive operation. Xapian provides a very efficient mechanism that returns an approximate value, so use that for this new command. This returns the number of matching messages, not threads, as that is cheap to compute. Signed-off-by: Keith Packard <keithp@keithp.com>
2009-11-21 08:15:07 +01:00
Xapian::MSet mset;
unsigned int flags = (Xapian::QueryParser::FLAG_BOOLEAN |
Xapian::QueryParser::FLAG_PHRASE |
Xapian::QueryParser::FLAG_LOVEHATE |
Xapian::QueryParser::FLAG_BOOLEAN_ANY_CASE |
Xapian::QueryParser::FLAG_WILDCARD |
Xapian::QueryParser::FLAG_PURE_NOT);
lib: Handle "*" as a query string to match all messages. This seems like a generally useful thing to support, (but the previous support through an empty string was not convenient for some users, (such as the command-line client).
2010-04-10 01:40:31 +02:00
if (strcmp (query_string, "") == 0 ||
strcmp (query_string, "*") == 0)
{
Add 'notmuch count' command to show the count of matching messages Getting the count of matching threads or messages is a fairly expensive operation. Xapian provides a very efficient mechanism that returns an approximate value, so use that for this new command. This returns the number of matching messages, not threads, as that is cheap to compute. Signed-off-by: Keith Packard <keithp@keithp.com>
2009-11-21 08:15:07 +01:00
final_query = mail_query;
} else {
string_query = notmuch->query_parser->
parse_query (query_string, flags);
final_query = Xapian::Query (Xapian::Query::OP_AND,
mail_query, string_query);
}
lib: Rearrange the exclude code in query.cc Slightly refactor the exclude code to give the callers access to the exclude query itself. There should be no functional change.
2012-03-01 23:30:36 +01:00
exclude_query = _notmuch_exclude_tags (query, final_query);
final_query = Xapian::Query (Xapian::Query::OP_AND_NOT,
final_query, exclude_query);
lib: Add support for automatically excluding tags from queries This is useful for tags like "deleted" and "spam" that people generally want to exclude from query results. These exclusions will be overridden if a tag is explicitly mentioned in a query.
2012-01-15 01:17:33 +01:00
Add 'notmuch count' command to show the count of matching messages Getting the count of matching threads or messages is a fairly expensive operation. Xapian provides a very efficient mechanism that returns an approximate value, so use that for this new command. This returns the number of matching messages, not threads, as that is cheap to compute. Signed-off-by: Keith Packard <keithp@keithp.com>
2009-11-21 08:15:07 +01:00
enquire.set_weighting_scheme(Xapian::BoolWeight());
enquire.set_docid_order(Xapian::Enquire::ASCENDING);
lib: Add exclude query debug output
2012-03-14 03:31:31 +01:00
if (_debug_query ()) {
fprintf (stderr, "Exclude query is:\n%s\n",
exclude_query.get_description ().c_str ());
lib: Expose query debug output via an environment variable Allow query debugging to be enabled at run-time by setting the NOTMUCH_DEBUG_QUERY environment variable to a non-empty string. Previously, enabling query debugging required recompiling, but parsed queries are often useful for tracking down bugs in situations where recompiling is inconvenient.
2012-03-14 03:31:30 +01:00
fprintf (stderr, "Final query is:\n%s\n",
final_query.get_description ().c_str ());
lib: Add exclude query debug output
2012-03-14 03:31:31 +01:00
}
Add 'notmuch count' command to show the count of matching messages Getting the count of matching threads or messages is a fairly expensive operation. Xapian provides a very efficient mechanism that returns an approximate value, so use that for this new command. This returns the number of matching messages, not threads, as that is cheap to compute. Signed-off-by: Keith Packard <keithp@keithp.com>
2009-11-21 08:15:07 +01:00
enquire.set_query (final_query);
mset = enquire.get_mset (0, notmuch->xapian_db->get_doccount ());
count = mset.get_matches_estimated();
} catch (const Xapian::Error &error) {
fprintf (stderr, "A Xapian exception occurred: %s\n",
error.get_msg().c_str());
fprintf (stderr, "Query string was: %s\n", query->query_string);
}
return count;
}
lib: add function to get the number of threads matching a search Add function notmuch_query_count_threads() to get the number of threads matching a search. This is done by performing a search and figuring out the number of unique thread IDs in the matching messages, a significantly heavier operation than notmuch_query_count_messages(). Signed-off-by: Jani Nikula <jani@nikula.org>
2011-11-15 21:08:48 +01:00
unsigned
notmuch_query_count_threads (notmuch_query_t *query)
{
notmuch_messages_t *messages;
GHashTable *hash;
unsigned int count;
notmuch_sort_t sort;
sort = query->sort;
query->sort = NOTMUCH_SORT_UNSORTED;
messages = notmuch_query_search_messages (query);
query->sort = sort;
if (messages == NULL)
return 0;
hash = g_hash_table_new_full (g_str_hash, g_str_equal, NULL, NULL);
if (hash == NULL) {
talloc_free (messages);
return 0;
}
while (notmuch_messages_valid (messages)) {
notmuch_message_t *message = notmuch_messages_get (messages);
const char *thread_id = notmuch_message_get_thread_id (message);
char *thread_id_copy = talloc_strdup (messages, thread_id);
if (unlikely (thread_id_copy == NULL)) {
notmuch_message_destroy (message);
count = 0;
goto DONE;
}
g_hash_table_insert (hash, thread_id_copy, NULL);
notmuch_message_destroy (message);
notmuch_messages_move_to_next (messages);
}
count = g_hash_table_size (hash);
DONE:
g_hash_table_unref (hash);
talloc_free (messages);
return count;
}
Reference in a new issue Copy permalink