Commit graph

345 commits

Author SHA1 Message Date
Justus Winter
ea54c4fdc7 Fix error reporting in notmuch_database_find_message_by_filename
Formerly it was possible for *message_ret to be left
uninitialized. The documentation however clearly states that "[o]n any
failure or when the message is not found, this function initializes
'*message' to NULL".

Signed-off-by: Justus Winter <4winter@informatik.uni-hamburg.de>
2012-03-18 07:58:35 -03:00
David Bremner
ad6a6edb38 Merge branch 'release' 2012-03-03 11:56:36 -04:00
Justus Winter
cfc5f1059a Actually close the xapian database in notmuch_database_close
Formerly the xapian database object was deleted and closed in its
destructor once the object was garbage collected. Explicitly call
close() so that the database and the associated lock is released
immediately.

The comment is a courtesy of Austin Clements.

Signed-off-by: Justus Winter <4winter@informatik.uni-hamburg.de>
2012-03-03 11:30:07 -04:00
Mark Walters
1a53f9f116 lib: Add the exclude flag to notmuch_query_search_threads
Add the NOTMUCH_MESSAGE_FLAG_EXCLUDED flag to
notmuch_query_search_threads. Implemented by inspecting the tags
directly in _notmuch_thread_create/_thread_add_message rather than as
a Xapian query for speed reasons.

Note notmuch_thread_get_matched_messages now returns the number of
non-excluded matching messages. This API is not totally desirable but
fixing it means breaking binary compatibility so we delay that.
2012-03-02 08:28:39 -04:00
Mark Walters
c9eb94d7fb lib: Make notmuch_query_search_messages set the exclude flag
Add a flag NOTMUCH_MESSAGE_FLAG_EXCLUDED which is set by
notmuch_query_search_messages for excluded messages. Also add an
option omit_excluded_messages to the search that we do not want the
excludes at all.

This exclude flag will be added to notmuch_query_search threads in the
next patch.
2012-03-02 08:27:47 -04:00
Mark Walters
08f7b026a9 lib: Rearrange the exclude code in query.cc
Slightly refactor the exclude code to give the callers access to the
exclude query itself. There should be no functional change.
2012-03-02 08:27:36 -04:00
Michal Sojka
40edc971a8 Convert non-UTF-8 parts to UTF-8 before indexing them
This fixes a bug that didn't allow to search for non-ASCII words such
parts. The code here was copied from show_text_part_content(), because
the show command already does the needed conversion when showing the
message.
2012-02-29 07:41:39 -04:00
Justus Winter
e2e95caa51 Prevent segmentation fault in notmuch_database_close
Previously opening a notmuch database in read write mode that has been
locked resulted in the notmuch_database_open function executing
notmuch_database_close as a cleanup function. notmuch_database_close
failed to check whether the xapian database has in fact been created.

Add a check whether the xapian database object has actually been
created before trying to call its flush method.

Signed-off-by: Justus Winter <4winter@informatik.uni-hamburg.de>
2012-02-20 23:03:25 -04:00
Austin Clements
c32116d048 lib: Use talloc to simplify cleanup in notmuch_database_open
Previously, we manually "free"d various pointers in
notmuch_database_open.  Use a local talloc context instead to simplify
cleanup and eliminate various NULL pointer initializations and
conditionals.
2012-02-03 21:15:45 -04:00
Austin Clements
6c0adab23e lib: Release resources if notmuch_database_open fails
Previously, if a Xapian exception occurred in notmuch_database_open,
we failed to clean up the allocated notmuch_database_t object.
2012-02-03 21:15:26 -04:00
Austin Clements
a8ee1c75c3 lib: Don't delete uninitialized pointers
In the error-handling paths of notmuch_database_open, we call
notmuch_database_close, which "delete"s several objects referenced by
the notmuch_database_t object.  However, some of these pointers may be
uninitialized, resulting in undefined behavior.  Hence, allocate the
notmuch_database_t with talloc_zero to make sure these pointers are
NULL so that "delete"ing them is harmless.
2012-02-03 21:14:59 -04:00
Austin Clements
3b76adf9e2 lib: Add support for automatically excluding tags from queries
This is useful for tags like "deleted" and "spam" that people
generally want to exclude from query results.  These exclusions will
be overridden if a tag is explicitly mentioned in a query.
2012-01-16 21:06:35 -04:00
Jani Nikula
3f9d73884e lib: fix messages.c build warn
lib/messages.c: In function ‘notmuch_messages_move_to_next’:
lib/messages.c:131:2: warning: ISO C forbids ‘return’ with expression, in function returning void [-pedantic]

Signed-off-by: Jani Nikula <jani@nikula.org>
2012-01-10 06:33:23 -04:00
Kazuo Teramoto
442d405ad3 lib: call g_mime_init() from notmuch_database_open()
As reported in
id:"CAEbOPGyuHnz4BPtDutnTPUHcP3eYcRCRkXhYoJR43RUMw671+g@mail.gmail.com"
sometimes gmime tries to access a NULL pointer, e.g. g_mime_iconv_open()
tries to access iconv_cache that is NULL if g_mime_init() is not called.
This causes notmuch to segfault when calling gmime functions.

Calling g_mime_init() initializes iconv_cache and others variables needed
by gmime, making sure they are initialized when notmuch calls gmime
functions.

Test marked fix by db.
2011-12-31 23:08:15 -04:00
Jameson Graef Rollins
ac7f843064 Ignore encrypted parts when indexing.
It appears to be an oversight that encrypted parts were indexed
previously.  The terms generated from encrypted parts are meaningless
and do nothing but add bloat to the database.  It is not worth
indexing the encrypted content, just as it's not worth indexing the
signatures in signed parts.
2011-12-29 17:44:43 -04:00
Thomas Jost
824dad76b6 Fix comments about what is stored in the database
Commit 567bcbc2 introduced two new values for each message (content of the
"From" and "Subject" headers), but the comments about the database schema had
not been updated accordingly.
2011-12-23 15:08:34 -04:00
David Edmondson
77ec8108a1 notmuch: Quiet buildbot warnings.
Cast away the result of various *write functions. Provide a default
value for some variables to avoid "use before set" warnings.
2011-12-21 07:32:16 -04:00
David Bremner
f0e0053149 Merge branch 'release'
Conflicts:
	NEWS

Conflicts resolved by inserting the 0.10.2 stanza before 0.11
2011-12-06 19:39:33 -04:00
David Bremner
69dc421ab3 lib: call g_type_init from notmuch_database_open
We want to make sure g_type_init is called before any GObject
functionality is used.
2011-12-04 22:00:25 -04:00
Louis Rilling
b9360be2bd tags_to_maildir_flags: Cleanup double assignement
The for loop right after already does the job.

Signed-off-by: Louis Rilling <l.rilling@av7.net>
2011-11-21 20:32:32 -04:00
Louis Rilling
21b13c3932 lib: Kill last usage of C++ type bool
Signed-off-by: Louis Rilling <l.rilling@av7.net>
2011-11-21 20:32:07 -04:00
Tom Prince
dbceb50f47 Don't link libnotmuch if libutil isn't linked in properly.
For some reason, on my machine, the link is picking up
/usr/lib/libutil.so instead of util/libutil.a. This causes there to be
undefined symbols in libnotmuch, making it unuseable. This patch causes
the link to fail instead.
2011-11-19 15:18:05 -04:00
Jani Nikula
00c60fbcb3 lib: add function to get the number of threads matching a search
Add function notmuch_query_count_threads() to get the number of threads
matching a search. This is done by performing a search and figuring out the
number of unique thread IDs in the matching messages, a significantly
heavier operation than notmuch_query_count_messages().

Signed-off-by: Jani Nikula <jani@nikula.org>
2011-11-15 19:12:32 -04:00
Austin Clements
567bcbc294 Store "from" and "subject" headers in the database.
This is a rebase and cleanup of Istvan Marko's patch from
id:m3pqnj2j7a.fsf@zsu.kismala.com

Search retrieves these headers for every message in the search
results.  Previously, this required opening and parsing every message
file.  Storing them directly in the database significantly reduces IO
and computation, speeding up search by between 50% and 10X.

Taking full advantage of this requires a database rebuild, but it will
fall back to the old behavior for messages that do not have headers
stored in the database.
2011-11-14 17:10:58 -04:00
Tom Prince
74f16571e8 Link libutil using filenmae, rather than using -l.
glibc includes a libutil, so if the wrong -L options get passed, we
will pick up glibc's version, rather than our own.
2011-11-13 14:58:21 -04:00
David Bremner
606e4461f8 lib/database.cc: use flush Xapian method instead of commit
Apparently the method was renamed in Xapian 1.1.0 but the old method
name will stay around for a while. It seems better to stick with the
old name to make notmuch compile with older versions of Xapian, at
least for now.
2011-11-02 20:38:22 -03:00
David Bremner
1dedfc90f6 xutil.c: remove duplicate copies, create new library libutil.a to contain xutil.
We keep the lib/xutil.c version. As a consequence, also factor out
_internal_error and associated macros.  It might be overkill to make a
new file error_util.c for this, but _internal_error does not really
belong in database.cc.
2011-10-30 23:09:49 -03:00
Amadeusz Żołnowski
8d282adf53 Prefix lib/notmuch.h and lib/gen-version-script.sh with $(srcdir)
lib/notmuch.h and lib/gen-version-script.sh couldn't have been found
when building out of sources directory.
2011-10-24 21:56:23 -03:00
David Bremner
9e976fc60c lib: bump SONAME
Based on discussions with amdragon, tschwinge, and others on IRC, I concluded that

1) symbol versioning was probably overkill for libnotmuch
2) It was also probably GNU ld specific
3) Most importantly, nobody could tell me on short notice how exactly it works.

So since the change to the notmuch_database_find_message breaks the
previous ABI, we need to bump the SONAME.
2011-10-04 13:47:04 -03:00
Ali Polatel
02a3076711 lib: make find_message{,by_filename) report errors
Previously, the functions notmuch_database_find_message() and
notmuch_database_find_message_by_filename() functions did not properly
report error condition to the library user.

For more information, read the thread on the notmuch mailing list
starting with my mail "id:871uv2unfd.fsf@gmail.com"

Make these functions accept a pointer to 'notmuch_message_t' as argument
and return notmuch_status_t which may be used to check for any error
condition.

restore: Modify for the new notmuch_database_find_message()
new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 07:55:29 +03:00
Ali Polatel
9b3f16ce71 lib: destroy message object after message removal
notmuch_database_remove_message() must call notmuch_message_destroy()
once it is done handling message removal.
2011-10-04 07:39:57 +03:00
David Bremner
0ce713ae02 lib: bump library minor version because of new symbols.
This bump is because of the new symbols introduced by the atomicity
patches.
2011-09-24 20:26:42 -03:00
Austin Clements
35f4a0f18b lib: Improve notmuch_database_{add,remove}_message documentation.
State up front that these functions may add a filename to an existing
message or remove only a filename (and not the message), respectively.
Previously, this key information was buried in return value
documentation or in "notes", which made it seem secondary to these
functions' semantics.
2011-09-24 20:00:29 -03:00
Austin Clements
51c3c0b2d2 lib: Wrap notmuch_database_add_message in an atomic section.
Adding a message may involve changes to multiple database documents,
and thus needs to be done in a transaction.  This makes add_message
(and, I think, the whole library) atomicity-safe: library callers only
needs to use atomic sections if they needs atomicity across multiple
library calls.
2011-09-24 20:00:28 -03:00
Austin Clements
7a8046ced8 lib: Add an API to find a message by filename.
notmuch_database_find_message_by_filename is mostly stolen from
notmuch_database_remove_message, so this patch also vastly simplfies
the latter using the former.

This API is also useful in its own right and will be used in a later
patch for eager maildir flag synchronization.
2011-09-24 20:00:28 -03:00
Austin Clements
bfe4555325 lib: Remove message document directly after removing the last file name.
Previously, notmuch_database_remove_message would remove the message
file name, sync the change to the message document, re-find the
message document, and then delete it if there were no more file names.
An interruption after sync'ing would result in a file-name-less,
permanently un-removable zombie message that would produce errors and
odd results in searches.  We could wrap this in an atomic section, but
it's much simpler to eliminate the round-about approach and just
delete the message document instead of sync'ing it if we removed the
last filename.
2011-09-23 21:50:39 -04:00
Austin Clements
e4379c43e2 lib: Indicate if there are more filenames after removal.
Make _notmuch_message_remove_filename return
NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID if the message has more filenames
and fix callers to handle this.
2011-09-23 21:50:39 -04:00
Austin Clements
e59cc0031f lib: Add support for nested atomic sections.
notmuch_database_t now keeps a nesting count and we only start a
transaction or commit for the outermost atomic section.

Introduces a new error, NOTMUCH_STATUS_UNBALANCED_ATOMIC.
2011-09-23 21:50:38 -04:00
Austin Clements
957f1ba3fc lib: Add notmuch_database_{begin,end}_atomic.
These operations translate into non-flushed Xapian transactions,
allowing arbitrary groups of database operations to be performed
atomically.
2011-09-23 21:50:38 -04:00
Austin Clements
62445dd023 lib: Add missing status check in _notmuch_message_remove_filename.
Previously, this function would synchronize the folder list even if
removing the file name failed.  Now it returns immediately if removing
the file name fails.
2011-09-12 23:36:00 -03:00
David Bremner
9db2145272 lib/gen-version-script.h: add getline and getdelim to notmuch.sym if needed
If the configure script detects missing getline and/or getdelim
symbols, then notmuch will use it's own versions. This patch, based on
id:"87k49v12i5.fsf@pc44es141.cs.uni-magdeburg.de" by Matthias
Guedemann, adds the symbols to notmuch.sym as well so they are
properly exported from the library.
2011-09-05 07:44:14 -03:00
David Bremner
80212b1329 lib/gen-version-script.sh: replace --defined argument to nm with awk
OpenBSD nm apparently doesn't support --defined.

The awk condition is based on the assumption that all defined symbols
have some hex number in the first column.

Thanks to Matthias Guedemann reporting the problem, and an earlier
version of this patch.
2011-09-05 07:39:44 -03:00
David Bremner
4b55df8534 re-enable notmuch.sym generation using POSIX sed
Unfortunately Robin Green's patch 52e4dedf9a was lost when I created
gen-version-script.sh. This merges his changes manually into that
script. It turns out tabs seem not needed in version script
files, so I simplified a bit and removed the printf.

Thanks to Alexander Botero-Lowry for help and testing.
2011-08-21 16:50:39 -03:00
Thomas Jost
1c71d943cb libnotmuch: only build symbols list after the modules are built
If the notmuch.sym target does not explicitly depend on $(libnotmuch_modules),
gen-version-script.sh may be run before all the .o files are created, for
example when doing a parallel build on a machine with many cores.
2011-07-20 21:42:35 -03:00
David Bremner
a900ddaba6 Merge commit '0.6.1'
Conflicts:
	lib/Makefile.local

The conflicts are from three kinds of commits not merged into release:
    - typo fixes
    - removal of debug output
    - fix for CLEAN rule

That were never merged into the release branch.
2011-07-17 22:20:37 -03:00
David Bremner
feb22fef2e libnotmuch: export Xapian typeinfo symbols
The lack of such exporting seems to cause problems catching
exceptions, as suggested by

    http://gcc.gnu.org/wiki/Visibility

This manifested in the symbol-hiding test failing when notmuch was
compile with gcc 4.4.5. On i386, this further manifested as notmuch
new failing to run (crashing with an uncaught exception on first run).
2011-07-16 22:16:46 -03:00
Robin Green
52e4dedf9a Use POSIX sed invocation
Fixes broken build on DragonFly BSD

Signed-off-by: Robin Green <greenrd@greenrd.org>
2011-07-03 08:32:51 -03:00
Mark Anderson
d77c7cefe4 Fix folder: coherence issue
Add removal of all ZXFOLDER terms to removal of all XFOLDER terms for
each message filename removal.

The existing filename-list reindexing will put all the needed terms
back in.  Test search-folder-coherence now passes.

Signed-off-by:Mark Anderson <ma.skies@gmail.com>
(cherry picked from commit 8a856e5c38)
2011-06-29 21:58:53 -03:00
David Bremner
f4dae4e5a1 lib/Makefile.local: remove leftover debugging output.
The removed "echo $(libnotmuch_modules)" was strictly for debugging.

Thanks to Austin Clements for the hint.
2011-06-29 15:35:47 -07:00
Mark Anderson
8a856e5c38 Fix folder: coherence issue
Add removal of all ZXFOLDER terms to removal of all XFOLDER terms for
each message filename removal.

The existing filename-list reindexing will put all the needed terms
back in.  Test search-folder-coherence now passes.

Signed-off-by:Mark Anderson <ma.skies@gmail.com>
2011-06-29 14:13:16 -07:00
David Bremner
8d46f05304 libnotmuch: build symbols list without relying on gcc -aux-info.
Carl reports "gcc -aux-info notmuch.aux lib/notmuch.h" does not
generate notmuch.aux for him with Debian gcc 4.6.0-8. A small
modification of the original sed regular expression allows us to work
directly from lib/notmuch.h, rather than preprocessing with gcc.

As with most such simple regex based "parsing", this is quite
sensitive to the input format, and needs that each symbol to be
exported from libnotmuch should

- start with "notmuch_"
- be the first non-whitespace token on the line
- be followed by an open parenthesis.

(Cherry-picked from 51b7ab6968, with conflicts resolved by db)
2011-06-28 16:32:07 -03:00
David Bremner
51b7ab6968 libnotmuch: build symbols list without relying on gcc -aux-info.
Carl reports "gcc -aux-info notmuch.aux lib/notmuch.h" does not
generate notmuch.aux for him with Debian gcc 4.6.0-8. A small
modification of the original sed regular expression allows us to work
directly from lib/notmuch.h, rather than preprocessing with gcc.

As with most such simple regex based "parsing", this is quite
sensitive to the input format, and needs that each symbol to be
exported from libnotmuch should

- start with "notmuch_"
- be the first non-whitespace token on the line
- be followed by an open parenthesis.
2011-06-28 11:59:48 -07:00
David Bremner
891082befa libnotmuch: fix typos in CLEAN setting, add file
- c0961e6 introduced a missing slash between $(dir)$(LIBNAME) and missing
  $(dir) in front of libnotmuch.a
- cdf1c70a created a file $(dir)/notmuch.h.gch and neglected to
  add it to CLEAN
2011-06-28 11:59:48 -07:00
Pieter Praet
8bb6f7869c fix sum moar typos [comments in source code]
Various typo fixes in comments within the source code.

Signed-off-by: Pieter Praet <pieter@praet.org>

Edited-by: Carl Worth <cworth@cworth.org> Restricted to just
source-code comments, (and fixed fix of "descriptios" to "descriptors"
rather than "descriptions").
2011-06-23 15:58:39 -07:00
Pieter Praet
730acd4764 fix sum moar typos [build scripts, Makefiles]
Various typo fixes in comments within the Makefile and other build scripts.

Signed-off-by: Pieter Praet <pieter@praet.org>

Edited-by: Carl Worth <cworth@cworth.org> Restricted to just build files.
2011-06-23 15:44:59 -07:00
David Bremner
cdf1c70ab3 libnotmuch: add linker script to declare only notmuch_* symbols as global.
This is closely tied to gcc and particularly gnu ld, but I guess the
shared library linking code would need to be adjusted to work on a
non-gnu linker anyay.

I had to make a few not-obviously related changes to the
lib/Makefile.local to make this work: libnotmuch_modules is defined
with := and used in place of $^
(cherry picked from commit 014bf85b1c06ff49be2bde5a26433d2cf376cf70)
2011-06-23 07:04:56 -03:00
Stewart Smith
c86d77b16a Fix appending of Received headers
We're not properly concatenating the Received headers if we parse them
while requesting a header that isn't Received.

this fixes notmuch-reply address detection in a bunch of situations.
2011-06-10 17:03:14 -07:00
Anton Khirnov
d3fdb76c8d lib/message-file: plug three memleaks.
Signed-off-by: Jameson Graef Rollins <jrollins@finestructure.net>
2011-06-03 12:30:55 -07:00
Jameson Graef Rollins
1d6b49561f tag signed/encrypted during notmuch new
This patch adds the tag "signed" to messages with any multipart/signed
parts, and the tag "encrypted" to messages with any
multipart/encrypted parts.  This only occurs when messages are indexed
during notmuch new, so a database rebuild is required to have old
messages tagged.
2011-05-27 16:22:00 -07:00
Carl Worth
d5523ead90 Mark some structures in the library interface with visibility=default attribute.
As of gcc 4.6, there are new warnings from -Wattributes along the lines of:

	warning: ‘_notmuch_messages’ declared with greater visibility
	than the type of its field ‘_notmuch_messages::iterator’
	[-Wattributes]

To squelch these, we decorate all such containing structs with
__attribute__((visibility("default"))). We take care to let only the
C++ compiler see this, (since the C compiler would otherwise warn
about ignored visibility attributes on types).
2011-05-11 13:27:15 -07:00
Carl Worth
2f3a76c569 Remove some variables which were set but not used.
gcc (at least as of version 4.6.0) is kind enough to point these out to us,
(when given -Wunused-but-set-variable explicitly or implicitly via -Wunused
or -Wall).

One of these cases was a legitimately unused variable. Two were simply
variables (named ignored) we were assigning only to squelch a warning about
unused function return values. I don't seem to be getting those warnings
even without setting the ignored variable. And the gcc docs. say that the
correct way to squelch that warning is with a cast to (void) anyway.
2011-05-11 13:27:14 -07:00
Austin Clements
b599bbe672 Fixup string list author 2011-03-21 02:45:18 -04:00
Austin Clements
d19c5de17a Add the tag list to the unified message metadata pass.
Now each caller of notmuch_message_get_tags only gets a new iterator,
instead of a whole new list.  In principle this could cause problems
with iterating while modifying tags, but through the magic of talloc
references, we keep the old tag list alive even after the cache in the
message object is invalidated.

This reduces my index search from the 3.102 seconds before the unified
metadata pass to 1.811 seconds (1.7X faster).  Combined with the
thread search optimization in b3caef1f06,
that makes this query 2.5X faster than when I started.
2011-03-21 02:45:18 -04:00
Austin Clements
f271071330 Add the file name list to the unified message metadata pass.
Even if the caller never uses the file names, there is little cost to
simply fetching the file name terms.  However, retrieving the full
paths requires additional database work, so the expansion from terms
to full paths is performed lazily.

This also simplifies clearing the filename cache, since that's now
handled by the generic metadata cache code.

This further reduces my inbox search from 3.102 seconds before the
unified metadata pass to 2.206 seconds (1.4X faster).
2011-03-21 02:45:18 -04:00
Austin Clements
206938ec9b Add a generic function to get a list of terms with some prefix.
Replace _notmuch_convert_tags with this and simplify
_create_filenames_for_terms_with_prefix.  This will also come in handy
shortly to get the message file name list.
2011-03-21 02:45:18 -04:00
Austin Clements
f3c1eebfaf Implement an internal generic string list and use it.
This replaces the guts of the filename list and tag list, making those
interfaces simple iterators over the generic string list.  The
directory, message filename, and tags-related code now build generic
string lists and then wraps them in specific iterators.  The real wins
come in later patches, when we use these for even more generic
functionality.

As a nice side-effect, this also eliminates the annoying dependency on
GList in the tag list.
2011-03-21 02:45:18 -04:00
Austin Clements
d9b0ae918f Use a single unified pass to fetch scalar message metadata.
This performs a single pass over a message's term list to fetch the
thread ID, message ID, and reply-to, rather than requiring a pass for
each.  Xapian decompresses the term list anew for each iteration, so
this reduces the amount of time spent decompressing message metadata.

This reduces my inbox search from 3.102 seconds to 2.555 seconds (1.2X
faster).
2011-03-21 02:45:18 -04:00
Carl Worth
42b0409f8e Merge remote branch 'amdragon/search-perf-3' 2011-03-10 11:05:49 -08:00
Carl Worth
3e4a9d60a9 build: Add support for non-source-directory builds.
Such as:

     mkdir build
     cd build
     ../configure
     make

This is implemented by having the configure script set a srcdir
variable in Makefile.config, and then sprinkling $(srcdir) into
various make rules. We also use vpath directives to convince GNU make
to find the source files from the original source directory.
2011-03-09 15:10:03 -08:00
Austin Clements
56d19273fe Simplify _notmuch_doc_id_set_init interface.
Don't require the caller of _notmuch_doc_id_set_init to pass in a
correct bound; instead compute it from the array.  This simplifies the
caller and makes this interface easier to use correctly.
2011-01-30 21:01:56 -05:00
Austin Clements
e04e72f9dd Remove code repetition in the doc ID bitmap code.
Remove the repeated "sizeof (doc_ids->bitmap[0])" that bothered cworth
by instead defining macros to compute the word and bit offset of a
given bit in the doc ID set bitmap.
2011-01-30 20:58:27 -05:00
Carl Worth
1a915d1b38 Makefile: Quote variables used as filenames in shell commands
This allows support for filenames with spaces in them.
2011-01-26 23:36:52 +10:00
Carl Worth
db70f3f0c4 lib: Save and restore term position in message while indexing.
This fixes the recently addead search-position-overlap bug as
demonstrated in the test of the same name.
2011-01-26 15:59:19 +10:00
Carl Worth
b64afb1ab9 notmuch search: Clean up some memory leaks during search loop.
With talloc, we were already freeing all memory by the time we exited
the loop, but that didn't help with excess use of memory inside the
loop, (which was mostly from tallocing some objects with the incorrect
parent).

Thanks to Andrew Tridgell for sitting next to me and teaching me to
use talloc_report_full to find these leaks.
2011-01-26 15:59:19 +10:00
Carl Worth
99cfa27030 Add support for folder-based searching.
A new "folder:" prefix in the query string can now be used to match
the directories in which mail files are stored.

The addition of this feature causes the recently added
search-by-folder tests to now pass.
2011-01-15 15:37:43 -08:00
Carl Worth
8bef9ba922 Tighten up a memory allocation.
Using the local talloc context ensures that the memory we are using
here will be freed shortly, (rather than hanging on for a long time
with the notmuch database object).
2011-01-15 15:37:43 -08:00
Carl Worth
36161181df Correct some minor typos in a comment
Nothing too important here. Just some misspellings I noticed while reading
nearby code.
2011-01-15 15:37:43 -08:00
Austin Clements
b3caef1f06 Optimize thread search using matched docid sets.
This reduces thread search's 1+2t Xapian queries (where t is the
number of matched threads) to 1+t queries and constructs exactly one
notmuch_message_t for each message instead of 2 to 3.
notmuch_query_search_threads eagerly fetches the docids of all
messages matching the user query instead of lazily constructing
message objects and fetching thread ID's from term lists.
_notmuch_thread_create takes a seed docid and the set of all matched
docids and uses a single Xapian query to expand this docid to its
containing thread, using the matched docid set to determine which
messages in the thread match the user query instead of using a second
Xapian query.

This reduces the amount of time required to load my inbox from 4.523
seconds to 3.025 seconds (1.5X faster).
2010-12-07 16:40:05 -08:00
Carl Worth
b099fc9ea7 notmuch search: Fix to display authors in date order.
Previously, the authors of the thread were displayed in
reverse-chronological order, which was fairly confusing.
2010-12-07 16:34:52 -08:00
Carl Worth
a4ce6cfbc6 lib: Set thread subject at the same time as setting thread->{oldest,newest}
We really want to change the thread subject at the same time we set
the date, (if the sort order indicates this is necessary). The
previous code for setting the thread subject was sensitive on the
query sort when adding matching messages. An independent bug fix is
about to change that query sort order, so we remove the dependency on
it here.
2010-12-07 16:27:03 -08:00
Carl Worth
c7b4d15d0a Fix to index the "Re" term present in any subject.
This was a misfeature where notmuch had extra code that just threw
away legitimate information. It was never indexing an initial "Re"
term in a subject. But some users have legitimately wanted to search
for this term.

The original code was written this way merely for strict compatiblity
with the indexing performed by sup, but we're not taking advantage of
that now anyway.
2010-11-23 18:11:04 -08:00
Carl Worth
7278383005 lib: Fix missing initialization of status field.
This could have been a problematic bug. Fortuinately "gcc -O2" warns
about it.
2010-11-11 20:54:41 -08:00
Carl Worth
fe8eeaf4a5 lib: Add two missing static qualifiers
The debian packaging is nice enough to notice when we accidentally
leak private symbols to the public interface.
2010-11-11 20:53:21 -08:00
Carl Worth
d33dee5389 Increment library version to 1.3.0
For the addition of three functions:

	notmuch_message_get_filenames
	notmuch_message_tags_to_maildir_flags
	notmuch_message_maildir_flags_to_tags
2010-11-11 20:42:45 -08:00
Carl Worth
96d99c3837 tags_to_maildir_flags: Fix to preserve existing, unsupported flags
This is to prevent notmuch from destroying any information the user
has encoded as flags in the maildir filename. Tests are also added to
the test suite to verify the documented behavior.
2010-11-11 16:36:02 -08:00
Carl Worth
95dd5fe5d7 notmuch_message_tags_to_maildir_flags: Do nothing outside of "new" and "cur"
Some people use notmuch with non-maildir files, (for example, email
messages in MH format, or else cool things like using sluk[*] to suck
down feeds into a format that notmuch can index).

To better support uses like that, don't do any renaming for files that
are not in a directory named either "new" or "cur".

[*] https://github.com/krl/sluk/
2010-11-11 14:32:17 -08:00
Carl Worth
1ea0b8bfe1 lib: Be honest in the documentation of the maildir functions
I had originally hoped for better semantics, such as doing nothing in
non-maildir directories, and preserving unknown maildir flags that
happen to be present.

We could still do those things, of course, but for now, remove them
from the documentation since the implementation does not do these
things yet.
2010-11-11 04:51:55 -08:00
Carl Worth
37a8096fdc notmuch_message_tags_to_maildir_flags: Don't exit on failure to rename.
It is totally legitimate for a non-maildir directory to be named "new"
(and not have a directory next to it named "cur"). To support this
case at least, be silent about any rename failure.
2010-11-11 03:50:42 -08:00
Carl Worth
71a3201885 notmuch_message_tags_to_maildir_flags: Fix to rename multiple files
This function was documented as modifying every filename associated
with the message. Fix it to actually do that.
2010-11-11 03:47:11 -08:00
Carl Worth
8480a0a003 notmuch_filenames_create: Take a reference to underlying filename list
The notmuch_filenames_t object might easily outlive the original object
owning the filename list. So take a talloc reference to keep things safe.
2010-11-11 03:40:19 -08:00
Carl Worth
404db1de90 maildir_flags_to_tags: Avoid interpreting "no info" as "no flags set".
If a filename has no maildir info at all, (that is, it does not
contain the sequence ":2,"), we consider this distinct from a filename
with an empty maildir info, (the ":2," separator is present, but no
flags characters follow).

Specifically, we regard a missing info field as providing no
information, so tags will remain unchanged. On the other hand, an info
field that is present but has no flags set will cause various tags to
be cleared, (or in the case of "unread", added).

This fixes the "remove info" case of the maildir-sync tests in the
test suite.
2010-11-11 03:40:19 -08:00
Carl Worth
5a98880add lib: Document that absence of maildir flags can also remove/add tags
Previously the documentation of notmuch_message_maildir_flags_to_tags
suggested that the presence of a flag would cause tags to be added,
(or in the case of "unread", removed). But the case of absent maildir
flags was not explicitly described.

What we actually want, is that for supported flags, the absence of the
flag in all messages causes the corresponding tag to be removed,
(or in the case of "unread", added). So document that explicitly.

This is the case recently added to the test suite as a failing test,
(so we'll need to do bug fixing before the documentation is honest
here).
2010-11-11 03:40:19 -08:00
Carl Worth
81cbaafc0f Fix notmuch_message_tags_to_maildir_flags to effect rename immediately
We have tests to ensure that when the notmuch library renames a file
that that rename takes place immediately in the database, (without
requiring something like "notmuch new" to notice the change).

This was working when the code was first added, but recently broke in
the reworking of the maildir-synchronization interface since the
tags_to_maildir_flags function can no longer assume that it is being
called as part of _notmuch_message_sync.

Fortunately, the fix is as simple as adding an explicit call to
_notmuch_message_sync.
2010-11-11 03:40:19 -08:00
Carl Worth
4b6063397f Fix notmuch_message_maildir_flags_to_tags to iterate over filenames
As documented, this function now iterates over all filenames for the
message, computing a logical OR of the flags set on the filenames,
then uses the final result to set tags on the message.

This change fixes 3 of the 10 maildir-sync tests that have been
failing since being added.
2010-11-11 03:40:19 -08:00
Carl Worth
1d02dd64af lib: Add new, public notmuch_message_get_filenames
This augments the existing notmuch_message_get_filename by allowing
the caller access to all filenames in the case of multiple files for a
single message.

To support this, we split the iterator (notmuch_filenames_t) away from
the list storage (notmuch_filename_list_t) where previously these were
a single object (notmuch_filenames_t). Then, whenever the user asks
for a file or filename, the message object lazily creates a complete
notmuch_filename_list_t and then:

	For notmuch_message_get_filename, returns the first filename
	in the list.

	For notmuch_message_get_filenames, creates and returns a new
	iterator for the filename list.
2010-11-11 03:40:19 -08:00
Carl Worth
d87db88432 lib: Add new implementation of notmuch_filenames_t
The new implementation is simply a talloc-based list of strings. The
former support (a list of database terms with a common prefix) is
implemented by simply pre-iterating over the terms and populating the
list. This should provide no performance disadvantage as callers of
thigns like notmuch_directory_get_child_files are very likely to
always iterate over all filenames anyway.

This new implementation of notmuch_filenames_t is in preparation for
adding API to query all of the filenames for a single message.
2010-11-11 03:40:19 -08:00
Carl Worth
d422dcf0a2 lib: Remove the notion of TAGS_INVALID
This rather ugly hack was recently obviated by the removal of the
notmuch_database_set_maildir_sync function. Now, clients must make
explicit calls to do any syncrhonization between maildir flags and
tags. So the library no longer needs to worry about doing inconsistent
synchronization while a message is only partially added.
2010-11-11 03:40:19 -08:00
Carl Worth
bb74e9dff8 lib: Rework interface for maildir_flags synchronization
Instead of having an API for setting a library-wide flag for
synchronization (notmuch_database_set_maildir_sync) we instead
implement maildir synchronization with two new library functions:

	notmuch_message_maildir_flags_to_tags
  and   notmuch_message_tags_to_maildir_flags

These functions are nicely documented here, (though the implementation
does not quite match the documentation yet---as plainly evidenced by
the current results of the test suite).
2010-11-11 03:40:19 -08:00
Carl Worth
2c262042ac lib: Remove the synchronization of 'T' flag with "deleted" tag.
Tags in a notmuch database affect all messages with the identical
message-ID. But maildir tags affect individual files. And since
multiple files can contain the identical message-ID, there is not a
one-to-one correspondence between messages affected by tags and flags.

This is particularly dangerous with the 'T' (== "trashed") maildir
flag and the corresponding "deleted" tag in the notmuch
database. Since these flags/tags are often used to trigger
irreversible deletion operations, the lack of one-to-one
correspondence can be potentially dangerous.

For example, consider the following sequence:

  1. A third-party application is used to identify duplicate messages
     in the mail store, and mark all-but-one of each duplicate with
     the 'T' flag for subsequent deletion.

  2. A "notmuch new" operation reads that 'T' flag, adding the
     "deleted" flag to the corresponding messages within the notmuch
     database.

  3. A subsequent notmuch operation, (such as a "notmuch dump; notmuch
     restore" cycle) synchronized the "deleted" tag back to the mail
     store, applying the 'T' flag to all(!) filenames with duplicate
     message IDs.

  4. A third-party application reads the 'T' flags and irreversibly
     deletes all mail messages which had any duplicates(!).

In order to avoid this scenario, we simply refuse to synchronize the
'T' flag with the "deleted" tag. Instead, applications can set 'T' and
act on it to delete files, or can set "deleted" and act on it to
delete files. But in either case the semantics are clear and there is
never dangerous propagation through the one-to-many mapping of notmuch
message objects to files.
2010-11-11 02:35:03 -08:00
Michal Sojka
d9d3d3e6f0 Make maildir synchronization configurable
This adds group [maildir] and key 'synchronize_flags' to the
configuration file. Its value enables (true) or diables (false) the
synchronization between notmuch tags and maildir flags. By default,
the synchronization is disabled.
2010-11-10 13:09:32 -08:00