Commit graph

520 commits

Author SHA1 Message Date
Jani Nikula
90cd1bac4e lib: add date range query support
Add a custom value range processor to enable date and time searches of
the form date:since..until, where "since" and "until" are expressions
understood by the previously added date/time parser, to restrict the
results to messages within a particular time range (based on the Date:
header).

If "since" or "until" describes date/time at an accuracy of days or
less, the values are rounded according to the accuracy, towards past
for "since" and towards future for "until". For example,
date:november..yesterday would match from the beginning of November
until the end of yesterday. Expressions such as date:today..today
means since the beginning of today until the end of today.

Open-ended ranges are supported (since Xapian 1.2.1), i.e. you can
specify date:..until or date:since.. to not limit the start or end
date, respectively.

CAVEATS:

Xapian does not support spaces in range expressions. You can replace
the spaces with '_', or (in most cases) '-', or (in some cases) leave
the spaces out altogether.

Entering date:expr without ".." (for example date:yesterday) will not
work as you might expect. You can achieve the expected result by
duplicating the expr both sides of ".." (for example
date:yesterday..yesterday).

Open-ended ranges won't work with pre-1.2.1 Xapian, but they don't
produce an error either.

Signed-off-by: Jani Nikula <jani@nikula.org>
2012-10-31 16:55:32 -03:00
Jani Nikula
af6b4b29f9 build: build parse-time-string as part of the notmuch lib and static cli 2012-10-31 16:53:01 -03:00
Austin Clements
b04c062aee Support OpenBSD
OpenBSD's build flags are identical to FreeBSD, except that libraries
need to be explicitly linked against libc.  No code changes are
necessary.

From: Cody Cutler <ccutler@csail.mit.edu>
2012-10-27 09:35:47 -03:00
Justus Winter
faf6ede3ef Fix the COERCE_STATUS macro
Fix the COERCE_STATUS macro to handle _internal_error being declared
as void function.

Note that the function _internal_error does not return. Evaluating to
NOTMUCH_STATUS_SUCCESS is done purely to appease the compiler.

Signed-off-by: Justus Winter <4winter@informatik.uni-hamburg.de>
2012-09-27 12:51:51 -03:00
Sascha Silbe
8dd4e9770e lib: fix NULL checks for filenames iterators
The API documentation (notmuch.h) states that the parameter may be NULL,
but the implementation only checked the current element, potentially
dereferencing a NULL pointer in the process.

Signed-off-by: Sascha Silbe <sascha-pgp@silbe.org>
2012-09-01 23:03:11 -03:00
Austin Clements
b88030bda6 lib: Treat messages in new/ as maildir messages with no flags set
Previously, notmuch new only synchronized maildir flags to tags for
files with a maildir "info" part.  Since messages in new/ don't have
an info part, notmuch would ignore them for flag-to-tag
synchronization.

This patch makes notmuch consider messages in new/ to be legitimate
maildir messages that simply have no maildir flags set.  The most
visible effect of this is that such messages now automatically get the
unread tag.
2012-06-10 20:14:56 -03:00
Austin Clements
750231bae8 lib: Only synchronize maildir flags for messages in maildirs
Previously, we synchronized flags to tags for any message that looked
like it had maildir flags in its file name, regardless of whether it
was in a maildir-like directory structure.  This was asymmetric with
tag-to-flag synchronization, which only applied to messages in
directories named new/ and cur/ (introduced by 95dd5fe5).

This change makes our interpretation stricter and addresses this
asymmetry by only synchronizing flags to tags for messages in
directories named new/ or cur/.  It also prepares us to treat messages
in new/ as maildir messages, even though they lack maildir flags.
2012-06-10 20:13:58 -03:00
Austin Clements
93ab4c7d11 lib: Move _filename_is_in_maildir
This way notmuch_message_maildir_flags_to_tags can call it.  It makes
more sense for this to be just above all of the maildir
synchronization code rather than mixed in the middle.
2012-06-10 20:13:45 -03:00
Austin Clements
d9f61c26a1 lib: Don't needlessly create directory docs in _notmuch_message_remove_filename
Previously, if passed a filename with a directory that did not exist
in the database, _notmuch_message_remove_filename would needlessly
create that directory document.  Fix it so that doesn't happen.
2012-05-23 22:32:12 -03:00
Austin Clements
cdd698f969 lib: Make notmuch_database_find_message_by_filename not crash on read-only databases
Previously, _notmuch_database_filename_to_direntry would abort with an
internal error when called on a read-only database.  Now that creating
the directory document is optional,
notmuch_database_find_message_by_filename can disable directory
document creation (as it should) and, as a result, not abort on
read-only databases.
2012-05-23 22:31:47 -03:00
Austin Clements
fe1ca14104 lib: Make notmuch_database_get_directory return NULL if the directory is not found
Using the new support from _notmuch_directory_create, this makes
notmuch_database_get_directory a read-only operation that simply
returns the directory object if it exists or NULL otherwise.  This
also means that notmuch_database_get_directory can work on read-only
databases.

This change breaks the directory mtime workaround in notmuch-new.c by
fixing the exact issue it was working around.  This permits mtime
update races to prevent scans of changed directories, which
non-deterministically breaks a few tests.  The next patch fixes this.
2012-05-23 22:30:55 -03:00
Austin Clements
67ae2377a9 lib: Perform the same transformation to _notmuch_database_filename_to_direntry
Now _notmuch_database_filename_to_direntry takes a flags argument and
can indicate if the necessary directory documents do not exist.
Again, callers have been updated, but retain their original behavior.
2012-05-23 22:30:43 -03:00
Austin Clements
0c950146a1 lib: Perform the same transformation to _notmuch_database_find_directory_id
Now _notmuch_database_find_directory_id takes a flags argument, which
it passes through to _notmuch_directory_create and can indicate if the
directory does not exist.  Again, callers have been updated, but
retain their original behavior.
2012-05-23 22:30:32 -03:00
Austin Clements
f69314fbd3 lib: Make directory document creation optional for _notmuch_directory_create
Previously this function would create directory documents if they
didn't exist.  As a result, it could only be used on writable
databases.  This adds an argument to make creation optional and to
make this function work on read-only databases.  We use a flag
argument to avoid a bare boolean and to permit future expansion.

Both callers have been updated, but currently retain the old behavior.
We'll take advantage of the new argument in the following patches.
2012-05-23 22:30:20 -03:00
Austin Clements
7199d22f43 lib/cli: Make notmuch_database_get_directory return a status code
Previously, notmuch_database_get_directory had no way to indicate how
it had failed.  This changes its prototype to return a status code and
set an out-argument to the retrieved directory, like similar functions
in the library API.  This does *not* change its currently broken
behavior of creating directory objects when they don't exist, but it
does document it and paves the way for fixing this.  Also, it can now
check for a read-only database and return
NOTMUCH_STATUS_READ_ONLY_DATABASE instead of crashing.

In the interest of atomicity, this also updates calls from the CLI so
that notmuch still compiles.
2012-05-15 08:56:33 -03:00
Austin Clements
7bf5be75ae lib: Bump SO version from 2.0.0 to 3.0.0
We've changed the APIs of notmuch_database_open,
notmuch_database_create, and notmuch_database_close.

Amended by db: also bump string in bindings/python/notmuch/globals.py
2012-05-05 20:06:09 -03:00
Austin Clements
ba57294218 lib/cli: Make notmuch_database_create return a status code
This is the notmuch_database_create equivalent of the previous change.

In this case, there were places where errors were not being propagated
correctly in notmuch_database_create or in calls to it.  These have
been fixed, using the new status value.
2012-05-05 10:12:26 -03:00
Austin Clements
5fddc07dc3 lib/cli: Make notmuch_database_open return a status code
It has been a long-standing issue that notmuch_database_open doesn't
return any indication of why it failed.  This patch changes its
prototype to return a notmuch_status_t and set an out-argument to the
database itself, like other functions that return both a status and an
object.

In the interest of atomicity, this also updates every use in the CLI
so that notmuch still compiles.  Since this patch does not update the
bindings, the Python bindings test fails.
2012-05-05 10:11:57 -03:00
Justus Winter
7864350c93 Split notmuch_database_close into two functions
Formerly notmuch_database_close closed the xapian database and
destroyed the talloc structure associated with the notmuch database
object. Split notmuch_database_close into notmuch_database_close and
notmuch_database_destroy.

This makes it possible for long running programs to close the xapian
database and thus release the lock associated with it without
destroying the data structures obtained from it.

This also makes the api more consistent since every other data
structure has a destructor function.

The comments in notmuch.h are a courtesy of Austin Clements.

Signed-off-by: Justus Winter <4winter@informatik.uni-hamburg.de>
2012-04-28 09:21:13 -03:00
Jani Nikula
de0557477d lib: work around talloc_steal usage from C++ code
Implicit typecast from 'void *' to 'T *' is okay in C, but not in
C++. In talloc_steal, an explicit cast is provided for type safety in
some GCC versions. Otherwise, a cast is required. Provide a template
function for this to maintain type safety, and redefine talloc_steal
to use it.

The template must be outside the extern "C" block (NOTMUCH_BEGIN_DECLS
and NOTMUCH_END_DECLS), but keep it within the GCC visibility #pragma.

No functional changes, apart from making the library build with
compilers other than recent GCC.

Signed-off-by: Jani Nikula <jani@nikula.org>
2012-04-15 09:42:15 -03:00
Mark Walters
d6fbef4690 lib: change default for notmuch_query_set_omit_excluded 2012-04-07 22:58:33 -03:00
Mark Walters
c695534df5 lib: fix an exclude bug
When the exclude tags contain a tag that does not occur anywhere in
the Xapian database the exclusion fails. We modify the way the query
is constructed to `work around' this. (In fact the new code is cleaner
anyway.)

It also seems to fix another exclusion failure bug reported by
jrollins but we have not yet worked out why it helps in that case.
2012-03-18 09:14:23 -03:00
Austin Clements
28367a9bcd lib: Add exclude query debug output 2012-03-18 09:14:22 -03:00
Austin Clements
75a0552633 lib: Expose query debug output via an environment variable
Allow query debugging to be enabled at run-time by setting the
NOTMUCH_DEBUG_QUERY environment variable to a non-empty string.

Previously, enabling query debugging required recompiling, but parsed
queries are often useful for tracking down bugs in situations where
recompiling is inconvenient.
2012-03-18 09:14:22 -03:00
David Bremner
fc4e5248d2 notmuch 0.12~rc2 release
-----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.4.11 (GNU/Linux)
 
 iJwEAAECAAYFAk9lxXUACgkQTiiN/0Um85nz8gP+PdsYdn6jyizOc1LYrWqeccQK
 La9HA5f+pdimEVTE1vRPwJT1Vz0saphiqCzRxNaYTtrFKQZ8Cz3z0pPs6/1mDAhD
 myGd9r5y566TNIxG2OlJHxxnmhoml1LNMLOHpj7ZMnNhrojKa0HmibtXlMR5ZN27
 tFP+UZn0c6Ad0JomezE=
 =IEn+
 -----END PGP SIGNATURE-----

Merge tag '0.12_rc2'

notmuch 0.12~rc2 release
2012-03-18 08:36:33 -03:00
Justus Winter
ea54c4fdc7 Fix error reporting in notmuch_database_find_message_by_filename
Formerly it was possible for *message_ret to be left
uninitialized. The documentation however clearly states that "[o]n any
failure or when the message is not found, this function initializes
'*message' to NULL".

Signed-off-by: Justus Winter <4winter@informatik.uni-hamburg.de>
2012-03-18 07:58:35 -03:00
David Bremner
ad6a6edb38 Merge branch 'release' 2012-03-03 11:56:36 -04:00
Justus Winter
cfc5f1059a Actually close the xapian database in notmuch_database_close
Formerly the xapian database object was deleted and closed in its
destructor once the object was garbage collected. Explicitly call
close() so that the database and the associated lock is released
immediately.

The comment is a courtesy of Austin Clements.

Signed-off-by: Justus Winter <4winter@informatik.uni-hamburg.de>
2012-03-03 11:30:07 -04:00
Mark Walters
1a53f9f116 lib: Add the exclude flag to notmuch_query_search_threads
Add the NOTMUCH_MESSAGE_FLAG_EXCLUDED flag to
notmuch_query_search_threads. Implemented by inspecting the tags
directly in _notmuch_thread_create/_thread_add_message rather than as
a Xapian query for speed reasons.

Note notmuch_thread_get_matched_messages now returns the number of
non-excluded matching messages. This API is not totally desirable but
fixing it means breaking binary compatibility so we delay that.
2012-03-02 08:28:39 -04:00
Mark Walters
c9eb94d7fb lib: Make notmuch_query_search_messages set the exclude flag
Add a flag NOTMUCH_MESSAGE_FLAG_EXCLUDED which is set by
notmuch_query_search_messages for excluded messages. Also add an
option omit_excluded_messages to the search that we do not want the
excludes at all.

This exclude flag will be added to notmuch_query_search threads in the
next patch.
2012-03-02 08:27:47 -04:00
Mark Walters
08f7b026a9 lib: Rearrange the exclude code in query.cc
Slightly refactor the exclude code to give the callers access to the
exclude query itself. There should be no functional change.
2012-03-02 08:27:36 -04:00
Michal Sojka
40edc971a8 Convert non-UTF-8 parts to UTF-8 before indexing them
This fixes a bug that didn't allow to search for non-ASCII words such
parts. The code here was copied from show_text_part_content(), because
the show command already does the needed conversion when showing the
message.
2012-02-29 07:41:39 -04:00
Justus Winter
e2e95caa51 Prevent segmentation fault in notmuch_database_close
Previously opening a notmuch database in read write mode that has been
locked resulted in the notmuch_database_open function executing
notmuch_database_close as a cleanup function. notmuch_database_close
failed to check whether the xapian database has in fact been created.

Add a check whether the xapian database object has actually been
created before trying to call its flush method.

Signed-off-by: Justus Winter <4winter@informatik.uni-hamburg.de>
2012-02-20 23:03:25 -04:00
Austin Clements
c32116d048 lib: Use talloc to simplify cleanup in notmuch_database_open
Previously, we manually "free"d various pointers in
notmuch_database_open.  Use a local talloc context instead to simplify
cleanup and eliminate various NULL pointer initializations and
conditionals.
2012-02-03 21:15:45 -04:00
Austin Clements
6c0adab23e lib: Release resources if notmuch_database_open fails
Previously, if a Xapian exception occurred in notmuch_database_open,
we failed to clean up the allocated notmuch_database_t object.
2012-02-03 21:15:26 -04:00
Austin Clements
a8ee1c75c3 lib: Don't delete uninitialized pointers
In the error-handling paths of notmuch_database_open, we call
notmuch_database_close, which "delete"s several objects referenced by
the notmuch_database_t object.  However, some of these pointers may be
uninitialized, resulting in undefined behavior.  Hence, allocate the
notmuch_database_t with talloc_zero to make sure these pointers are
NULL so that "delete"ing them is harmless.
2012-02-03 21:14:59 -04:00
Austin Clements
3b76adf9e2 lib: Add support for automatically excluding tags from queries
This is useful for tags like "deleted" and "spam" that people
generally want to exclude from query results.  These exclusions will
be overridden if a tag is explicitly mentioned in a query.
2012-01-16 21:06:35 -04:00
Jani Nikula
3f9d73884e lib: fix messages.c build warn
lib/messages.c: In function ‘notmuch_messages_move_to_next’:
lib/messages.c:131:2: warning: ISO C forbids ‘return’ with expression, in function returning void [-pedantic]

Signed-off-by: Jani Nikula <jani@nikula.org>
2012-01-10 06:33:23 -04:00
Kazuo Teramoto
442d405ad3 lib: call g_mime_init() from notmuch_database_open()
As reported in
id:"CAEbOPGyuHnz4BPtDutnTPUHcP3eYcRCRkXhYoJR43RUMw671+g@mail.gmail.com"
sometimes gmime tries to access a NULL pointer, e.g. g_mime_iconv_open()
tries to access iconv_cache that is NULL if g_mime_init() is not called.
This causes notmuch to segfault when calling gmime functions.

Calling g_mime_init() initializes iconv_cache and others variables needed
by gmime, making sure they are initialized when notmuch calls gmime
functions.

Test marked fix by db.
2011-12-31 23:08:15 -04:00
Jameson Graef Rollins
ac7f843064 Ignore encrypted parts when indexing.
It appears to be an oversight that encrypted parts were indexed
previously.  The terms generated from encrypted parts are meaningless
and do nothing but add bloat to the database.  It is not worth
indexing the encrypted content, just as it's not worth indexing the
signatures in signed parts.
2011-12-29 17:44:43 -04:00
Thomas Jost
824dad76b6 Fix comments about what is stored in the database
Commit 567bcbc2 introduced two new values for each message (content of the
"From" and "Subject" headers), but the comments about the database schema had
not been updated accordingly.
2011-12-23 15:08:34 -04:00
David Edmondson
77ec8108a1 notmuch: Quiet buildbot warnings.
Cast away the result of various *write functions. Provide a default
value for some variables to avoid "use before set" warnings.
2011-12-21 07:32:16 -04:00
David Bremner
f0e0053149 Merge branch 'release'
Conflicts:
	NEWS

Conflicts resolved by inserting the 0.10.2 stanza before 0.11
2011-12-06 19:39:33 -04:00
David Bremner
69dc421ab3 lib: call g_type_init from notmuch_database_open
We want to make sure g_type_init is called before any GObject
functionality is used.
2011-12-04 22:00:25 -04:00
Louis Rilling
b9360be2bd tags_to_maildir_flags: Cleanup double assignement
The for loop right after already does the job.

Signed-off-by: Louis Rilling <l.rilling@av7.net>
2011-11-21 20:32:32 -04:00
Louis Rilling
21b13c3932 lib: Kill last usage of C++ type bool
Signed-off-by: Louis Rilling <l.rilling@av7.net>
2011-11-21 20:32:07 -04:00
Tom Prince
dbceb50f47 Don't link libnotmuch if libutil isn't linked in properly.
For some reason, on my machine, the link is picking up
/usr/lib/libutil.so instead of util/libutil.a. This causes there to be
undefined symbols in libnotmuch, making it unuseable. This patch causes
the link to fail instead.
2011-11-19 15:18:05 -04:00
Jani Nikula
00c60fbcb3 lib: add function to get the number of threads matching a search
Add function notmuch_query_count_threads() to get the number of threads
matching a search. This is done by performing a search and figuring out the
number of unique thread IDs in the matching messages, a significantly
heavier operation than notmuch_query_count_messages().

Signed-off-by: Jani Nikula <jani@nikula.org>
2011-11-15 19:12:32 -04:00
Austin Clements
567bcbc294 Store "from" and "subject" headers in the database.
This is a rebase and cleanup of Istvan Marko's patch from
id:m3pqnj2j7a.fsf@zsu.kismala.com

Search retrieves these headers for every message in the search
results.  Previously, this required opening and parsing every message
file.  Storing them directly in the database significantly reduces IO
and computation, speeding up search by between 50% and 10X.

Taking full advantage of this requires a database rebuild, but it will
fall back to the old behavior for messages that do not have headers
stored in the database.
2011-11-14 17:10:58 -04:00
Tom Prince
74f16571e8 Link libutil using filenmae, rather than using -l.
glibc includes a libutil, so if the wrong -L options get passed, we
will pick up glibc's version, rather than our own.
2011-11-13 14:58:21 -04:00
David Bremner
606e4461f8 lib/database.cc: use flush Xapian method instead of commit
Apparently the method was renamed in Xapian 1.1.0 but the old method
name will stay around for a while. It seems better to stick with the
old name to make notmuch compile with older versions of Xapian, at
least for now.
2011-11-02 20:38:22 -03:00
David Bremner
1dedfc90f6 xutil.c: remove duplicate copies, create new library libutil.a to contain xutil.
We keep the lib/xutil.c version. As a consequence, also factor out
_internal_error and associated macros.  It might be overkill to make a
new file error_util.c for this, but _internal_error does not really
belong in database.cc.
2011-10-30 23:09:49 -03:00
Amadeusz Żołnowski
8d282adf53 Prefix lib/notmuch.h and lib/gen-version-script.sh with $(srcdir)
lib/notmuch.h and lib/gen-version-script.sh couldn't have been found
when building out of sources directory.
2011-10-24 21:56:23 -03:00
David Bremner
9e976fc60c lib: bump SONAME
Based on discussions with amdragon, tschwinge, and others on IRC, I concluded that

1) symbol versioning was probably overkill for libnotmuch
2) It was also probably GNU ld specific
3) Most importantly, nobody could tell me on short notice how exactly it works.

So since the change to the notmuch_database_find_message breaks the
previous ABI, we need to bump the SONAME.
2011-10-04 13:47:04 -03:00
Ali Polatel
02a3076711 lib: make find_message{,by_filename) report errors
Previously, the functions notmuch_database_find_message() and
notmuch_database_find_message_by_filename() functions did not properly
report error condition to the library user.

For more information, read the thread on the notmuch mailing list
starting with my mail "id:871uv2unfd.fsf@gmail.com"

Make these functions accept a pointer to 'notmuch_message_t' as argument
and return notmuch_status_t which may be used to check for any error
condition.

restore: Modify for the new notmuch_database_find_message()
new: Modify for the new notmuch_database_find_message_by_filename()
2011-10-04 07:55:29 +03:00
Ali Polatel
9b3f16ce71 lib: destroy message object after message removal
notmuch_database_remove_message() must call notmuch_message_destroy()
once it is done handling message removal.
2011-10-04 07:39:57 +03:00
David Bremner
0ce713ae02 lib: bump library minor version because of new symbols.
This bump is because of the new symbols introduced by the atomicity
patches.
2011-09-24 20:26:42 -03:00
Austin Clements
35f4a0f18b lib: Improve notmuch_database_{add,remove}_message documentation.
State up front that these functions may add a filename to an existing
message or remove only a filename (and not the message), respectively.
Previously, this key information was buried in return value
documentation or in "notes", which made it seem secondary to these
functions' semantics.
2011-09-24 20:00:29 -03:00
Austin Clements
51c3c0b2d2 lib: Wrap notmuch_database_add_message in an atomic section.
Adding a message may involve changes to multiple database documents,
and thus needs to be done in a transaction.  This makes add_message
(and, I think, the whole library) atomicity-safe: library callers only
needs to use atomic sections if they needs atomicity across multiple
library calls.
2011-09-24 20:00:28 -03:00
Austin Clements
7a8046ced8 lib: Add an API to find a message by filename.
notmuch_database_find_message_by_filename is mostly stolen from
notmuch_database_remove_message, so this patch also vastly simplfies
the latter using the former.

This API is also useful in its own right and will be used in a later
patch for eager maildir flag synchronization.
2011-09-24 20:00:28 -03:00
Austin Clements
bfe4555325 lib: Remove message document directly after removing the last file name.
Previously, notmuch_database_remove_message would remove the message
file name, sync the change to the message document, re-find the
message document, and then delete it if there were no more file names.
An interruption after sync'ing would result in a file-name-less,
permanently un-removable zombie message that would produce errors and
odd results in searches.  We could wrap this in an atomic section, but
it's much simpler to eliminate the round-about approach and just
delete the message document instead of sync'ing it if we removed the
last filename.
2011-09-23 21:50:39 -04:00
Austin Clements
e4379c43e2 lib: Indicate if there are more filenames after removal.
Make _notmuch_message_remove_filename return
NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID if the message has more filenames
and fix callers to handle this.
2011-09-23 21:50:39 -04:00
Austin Clements
e59cc0031f lib: Add support for nested atomic sections.
notmuch_database_t now keeps a nesting count and we only start a
transaction or commit for the outermost atomic section.

Introduces a new error, NOTMUCH_STATUS_UNBALANCED_ATOMIC.
2011-09-23 21:50:38 -04:00
Austin Clements
957f1ba3fc lib: Add notmuch_database_{begin,end}_atomic.
These operations translate into non-flushed Xapian transactions,
allowing arbitrary groups of database operations to be performed
atomically.
2011-09-23 21:50:38 -04:00
Austin Clements
62445dd023 lib: Add missing status check in _notmuch_message_remove_filename.
Previously, this function would synchronize the folder list even if
removing the file name failed.  Now it returns immediately if removing
the file name fails.
2011-09-12 23:36:00 -03:00
David Bremner
9db2145272 lib/gen-version-script.h: add getline and getdelim to notmuch.sym if needed
If the configure script detects missing getline and/or getdelim
symbols, then notmuch will use it's own versions. This patch, based on
id:"87k49v12i5.fsf@pc44es141.cs.uni-magdeburg.de" by Matthias
Guedemann, adds the symbols to notmuch.sym as well so they are
properly exported from the library.
2011-09-05 07:44:14 -03:00
David Bremner
80212b1329 lib/gen-version-script.sh: replace --defined argument to nm with awk
OpenBSD nm apparently doesn't support --defined.

The awk condition is based on the assumption that all defined symbols
have some hex number in the first column.

Thanks to Matthias Guedemann reporting the problem, and an earlier
version of this patch.
2011-09-05 07:39:44 -03:00
David Bremner
4b55df8534 re-enable notmuch.sym generation using POSIX sed
Unfortunately Robin Green's patch 52e4dedf9a was lost when I created
gen-version-script.sh. This merges his changes manually into that
script. It turns out tabs seem not needed in version script
files, so I simplified a bit and removed the printf.

Thanks to Alexander Botero-Lowry for help and testing.
2011-08-21 16:50:39 -03:00
Thomas Jost
1c71d943cb libnotmuch: only build symbols list after the modules are built
If the notmuch.sym target does not explicitly depend on $(libnotmuch_modules),
gen-version-script.sh may be run before all the .o files are created, for
example when doing a parallel build on a machine with many cores.
2011-07-20 21:42:35 -03:00
David Bremner
a900ddaba6 Merge commit '0.6.1'
Conflicts:
	lib/Makefile.local

The conflicts are from three kinds of commits not merged into release:
    - typo fixes
    - removal of debug output
    - fix for CLEAN rule

That were never merged into the release branch.
2011-07-17 22:20:37 -03:00
David Bremner
feb22fef2e libnotmuch: export Xapian typeinfo symbols
The lack of such exporting seems to cause problems catching
exceptions, as suggested by

    http://gcc.gnu.org/wiki/Visibility

This manifested in the symbol-hiding test failing when notmuch was
compile with gcc 4.4.5. On i386, this further manifested as notmuch
new failing to run (crashing with an uncaught exception on first run).
2011-07-16 22:16:46 -03:00
Robin Green
52e4dedf9a Use POSIX sed invocation
Fixes broken build on DragonFly BSD

Signed-off-by: Robin Green <greenrd@greenrd.org>
2011-07-03 08:32:51 -03:00
Mark Anderson
d77c7cefe4 Fix folder: coherence issue
Add removal of all ZXFOLDER terms to removal of all XFOLDER terms for
each message filename removal.

The existing filename-list reindexing will put all the needed terms
back in.  Test search-folder-coherence now passes.

Signed-off-by:Mark Anderson <ma.skies@gmail.com>
(cherry picked from commit 8a856e5c38)
2011-06-29 21:58:53 -03:00
David Bremner
f4dae4e5a1 lib/Makefile.local: remove leftover debugging output.
The removed "echo $(libnotmuch_modules)" was strictly for debugging.

Thanks to Austin Clements for the hint.
2011-06-29 15:35:47 -07:00
Mark Anderson
8a856e5c38 Fix folder: coherence issue
Add removal of all ZXFOLDER terms to removal of all XFOLDER terms for
each message filename removal.

The existing filename-list reindexing will put all the needed terms
back in.  Test search-folder-coherence now passes.

Signed-off-by:Mark Anderson <ma.skies@gmail.com>
2011-06-29 14:13:16 -07:00
David Bremner
8d46f05304 libnotmuch: build symbols list without relying on gcc -aux-info.
Carl reports "gcc -aux-info notmuch.aux lib/notmuch.h" does not
generate notmuch.aux for him with Debian gcc 4.6.0-8. A small
modification of the original sed regular expression allows us to work
directly from lib/notmuch.h, rather than preprocessing with gcc.

As with most such simple regex based "parsing", this is quite
sensitive to the input format, and needs that each symbol to be
exported from libnotmuch should

- start with "notmuch_"
- be the first non-whitespace token on the line
- be followed by an open parenthesis.

(Cherry-picked from 51b7ab6968, with conflicts resolved by db)
2011-06-28 16:32:07 -03:00
David Bremner
51b7ab6968 libnotmuch: build symbols list without relying on gcc -aux-info.
Carl reports "gcc -aux-info notmuch.aux lib/notmuch.h" does not
generate notmuch.aux for him with Debian gcc 4.6.0-8. A small
modification of the original sed regular expression allows us to work
directly from lib/notmuch.h, rather than preprocessing with gcc.

As with most such simple regex based "parsing", this is quite
sensitive to the input format, and needs that each symbol to be
exported from libnotmuch should

- start with "notmuch_"
- be the first non-whitespace token on the line
- be followed by an open parenthesis.
2011-06-28 11:59:48 -07:00
David Bremner
891082befa libnotmuch: fix typos in CLEAN setting, add file
- c0961e6 introduced a missing slash between $(dir)$(LIBNAME) and missing
  $(dir) in front of libnotmuch.a
- cdf1c70a created a file $(dir)/notmuch.h.gch and neglected to
  add it to CLEAN
2011-06-28 11:59:48 -07:00
Pieter Praet
8bb6f7869c fix sum moar typos [comments in source code]
Various typo fixes in comments within the source code.

Signed-off-by: Pieter Praet <pieter@praet.org>

Edited-by: Carl Worth <cworth@cworth.org> Restricted to just
source-code comments, (and fixed fix of "descriptios" to "descriptors"
rather than "descriptions").
2011-06-23 15:58:39 -07:00
Pieter Praet
730acd4764 fix sum moar typos [build scripts, Makefiles]
Various typo fixes in comments within the Makefile and other build scripts.

Signed-off-by: Pieter Praet <pieter@praet.org>

Edited-by: Carl Worth <cworth@cworth.org> Restricted to just build files.
2011-06-23 15:44:59 -07:00
David Bremner
cdf1c70ab3 libnotmuch: add linker script to declare only notmuch_* symbols as global.
This is closely tied to gcc and particularly gnu ld, but I guess the
shared library linking code would need to be adjusted to work on a
non-gnu linker anyay.

I had to make a few not-obviously related changes to the
lib/Makefile.local to make this work: libnotmuch_modules is defined
with := and used in place of $^
(cherry picked from commit 014bf85b1c06ff49be2bde5a26433d2cf376cf70)
2011-06-23 07:04:56 -03:00
Stewart Smith
c86d77b16a Fix appending of Received headers
We're not properly concatenating the Received headers if we parse them
while requesting a header that isn't Received.

this fixes notmuch-reply address detection in a bunch of situations.
2011-06-10 17:03:14 -07:00
Anton Khirnov
d3fdb76c8d lib/message-file: plug three memleaks.
Signed-off-by: Jameson Graef Rollins <jrollins@finestructure.net>
2011-06-03 12:30:55 -07:00
Jameson Graef Rollins
1d6b49561f tag signed/encrypted during notmuch new
This patch adds the tag "signed" to messages with any multipart/signed
parts, and the tag "encrypted" to messages with any
multipart/encrypted parts.  This only occurs when messages are indexed
during notmuch new, so a database rebuild is required to have old
messages tagged.
2011-05-27 16:22:00 -07:00
Carl Worth
d5523ead90 Mark some structures in the library interface with visibility=default attribute.
As of gcc 4.6, there are new warnings from -Wattributes along the lines of:

	warning: ‘_notmuch_messages’ declared with greater visibility
	than the type of its field ‘_notmuch_messages::iterator’
	[-Wattributes]

To squelch these, we decorate all such containing structs with
__attribute__((visibility("default"))). We take care to let only the
C++ compiler see this, (since the C compiler would otherwise warn
about ignored visibility attributes on types).
2011-05-11 13:27:15 -07:00
Carl Worth
2f3a76c569 Remove some variables which were set but not used.
gcc (at least as of version 4.6.0) is kind enough to point these out to us,
(when given -Wunused-but-set-variable explicitly or implicitly via -Wunused
or -Wall).

One of these cases was a legitimately unused variable. Two were simply
variables (named ignored) we were assigning only to squelch a warning about
unused function return values. I don't seem to be getting those warnings
even without setting the ignored variable. And the gcc docs. say that the
correct way to squelch that warning is with a cast to (void) anyway.
2011-05-11 13:27:14 -07:00
Austin Clements
b599bbe672 Fixup string list author 2011-03-21 02:45:18 -04:00
Austin Clements
d19c5de17a Add the tag list to the unified message metadata pass.
Now each caller of notmuch_message_get_tags only gets a new iterator,
instead of a whole new list.  In principle this could cause problems
with iterating while modifying tags, but through the magic of talloc
references, we keep the old tag list alive even after the cache in the
message object is invalidated.

This reduces my index search from the 3.102 seconds before the unified
metadata pass to 1.811 seconds (1.7X faster).  Combined with the
thread search optimization in b3caef1f06,
that makes this query 2.5X faster than when I started.
2011-03-21 02:45:18 -04:00
Austin Clements
f271071330 Add the file name list to the unified message metadata pass.
Even if the caller never uses the file names, there is little cost to
simply fetching the file name terms.  However, retrieving the full
paths requires additional database work, so the expansion from terms
to full paths is performed lazily.

This also simplifies clearing the filename cache, since that's now
handled by the generic metadata cache code.

This further reduces my inbox search from 3.102 seconds before the
unified metadata pass to 2.206 seconds (1.4X faster).
2011-03-21 02:45:18 -04:00
Austin Clements
206938ec9b Add a generic function to get a list of terms with some prefix.
Replace _notmuch_convert_tags with this and simplify
_create_filenames_for_terms_with_prefix.  This will also come in handy
shortly to get the message file name list.
2011-03-21 02:45:18 -04:00
Austin Clements
f3c1eebfaf Implement an internal generic string list and use it.
This replaces the guts of the filename list and tag list, making those
interfaces simple iterators over the generic string list.  The
directory, message filename, and tags-related code now build generic
string lists and then wraps them in specific iterators.  The real wins
come in later patches, when we use these for even more generic
functionality.

As a nice side-effect, this also eliminates the annoying dependency on
GList in the tag list.
2011-03-21 02:45:18 -04:00
Austin Clements
d9b0ae918f Use a single unified pass to fetch scalar message metadata.
This performs a single pass over a message's term list to fetch the
thread ID, message ID, and reply-to, rather than requiring a pass for
each.  Xapian decompresses the term list anew for each iteration, so
this reduces the amount of time spent decompressing message metadata.

This reduces my inbox search from 3.102 seconds to 2.555 seconds (1.2X
faster).
2011-03-21 02:45:18 -04:00
Carl Worth
42b0409f8e Merge remote branch 'amdragon/search-perf-3' 2011-03-10 11:05:49 -08:00
Carl Worth
3e4a9d60a9 build: Add support for non-source-directory builds.
Such as:

     mkdir build
     cd build
     ../configure
     make

This is implemented by having the configure script set a srcdir
variable in Makefile.config, and then sprinkling $(srcdir) into
various make rules. We also use vpath directives to convince GNU make
to find the source files from the original source directory.
2011-03-09 15:10:03 -08:00
Austin Clements
56d19273fe Simplify _notmuch_doc_id_set_init interface.
Don't require the caller of _notmuch_doc_id_set_init to pass in a
correct bound; instead compute it from the array.  This simplifies the
caller and makes this interface easier to use correctly.
2011-01-30 21:01:56 -05:00
Austin Clements
e04e72f9dd Remove code repetition in the doc ID bitmap code.
Remove the repeated "sizeof (doc_ids->bitmap[0])" that bothered cworth
by instead defining macros to compute the word and bit offset of a
given bit in the doc ID set bitmap.
2011-01-30 20:58:27 -05:00
Carl Worth
1a915d1b38 Makefile: Quote variables used as filenames in shell commands
This allows support for filenames with spaces in them.
2011-01-26 23:36:52 +10:00
Carl Worth
db70f3f0c4 lib: Save and restore term position in message while indexing.
This fixes the recently addead search-position-overlap bug as
demonstrated in the test of the same name.
2011-01-26 15:59:19 +10:00
Carl Worth
b64afb1ab9 notmuch search: Clean up some memory leaks during search loop.
With talloc, we were already freeing all memory by the time we exited
the loop, but that didn't help with excess use of memory inside the
loop, (which was mostly from tallocing some objects with the incorrect
parent).

Thanks to Andrew Tridgell for sitting next to me and teaching me to
use talloc_report_full to find these leaks.
2011-01-26 15:59:19 +10:00
Carl Worth
99cfa27030 Add support for folder-based searching.
A new "folder:" prefix in the query string can now be used to match
the directories in which mail files are stored.

The addition of this feature causes the recently added
search-by-folder tests to now pass.
2011-01-15 15:37:43 -08:00
Carl Worth
8bef9ba922 Tighten up a memory allocation.
Using the local talloc context ensures that the memory we are using
here will be freed shortly, (rather than hanging on for a long time
with the notmuch database object).
2011-01-15 15:37:43 -08:00
Carl Worth
36161181df Correct some minor typos in a comment
Nothing too important here. Just some misspellings I noticed while reading
nearby code.
2011-01-15 15:37:43 -08:00
Austin Clements
b3caef1f06 Optimize thread search using matched docid sets.
This reduces thread search's 1+2t Xapian queries (where t is the
number of matched threads) to 1+t queries and constructs exactly one
notmuch_message_t for each message instead of 2 to 3.
notmuch_query_search_threads eagerly fetches the docids of all
messages matching the user query instead of lazily constructing
message objects and fetching thread ID's from term lists.
_notmuch_thread_create takes a seed docid and the set of all matched
docids and uses a single Xapian query to expand this docid to its
containing thread, using the matched docid set to determine which
messages in the thread match the user query instead of using a second
Xapian query.

This reduces the amount of time required to load my inbox from 4.523
seconds to 3.025 seconds (1.5X faster).
2010-12-07 16:40:05 -08:00
Carl Worth
b099fc9ea7 notmuch search: Fix to display authors in date order.
Previously, the authors of the thread were displayed in
reverse-chronological order, which was fairly confusing.
2010-12-07 16:34:52 -08:00
Carl Worth
a4ce6cfbc6 lib: Set thread subject at the same time as setting thread->{oldest,newest}
We really want to change the thread subject at the same time we set
the date, (if the sort order indicates this is necessary). The
previous code for setting the thread subject was sensitive on the
query sort when adding matching messages. An independent bug fix is
about to change that query sort order, so we remove the dependency on
it here.
2010-12-07 16:27:03 -08:00
Carl Worth
c7b4d15d0a Fix to index the "Re" term present in any subject.
This was a misfeature where notmuch had extra code that just threw
away legitimate information. It was never indexing an initial "Re"
term in a subject. But some users have legitimately wanted to search
for this term.

The original code was written this way merely for strict compatiblity
with the indexing performed by sup, but we're not taking advantage of
that now anyway.
2010-11-23 18:11:04 -08:00
Carl Worth
7278383005 lib: Fix missing initialization of status field.
This could have been a problematic bug. Fortuinately "gcc -O2" warns
about it.
2010-11-11 20:54:41 -08:00
Carl Worth
fe8eeaf4a5 lib: Add two missing static qualifiers
The debian packaging is nice enough to notice when we accidentally
leak private symbols to the public interface.
2010-11-11 20:53:21 -08:00
Carl Worth
d33dee5389 Increment library version to 1.3.0
For the addition of three functions:

	notmuch_message_get_filenames
	notmuch_message_tags_to_maildir_flags
	notmuch_message_maildir_flags_to_tags
2010-11-11 20:42:45 -08:00
Carl Worth
96d99c3837 tags_to_maildir_flags: Fix to preserve existing, unsupported flags
This is to prevent notmuch from destroying any information the user
has encoded as flags in the maildir filename. Tests are also added to
the test suite to verify the documented behavior.
2010-11-11 16:36:02 -08:00
Carl Worth
95dd5fe5d7 notmuch_message_tags_to_maildir_flags: Do nothing outside of "new" and "cur"
Some people use notmuch with non-maildir files, (for example, email
messages in MH format, or else cool things like using sluk[*] to suck
down feeds into a format that notmuch can index).

To better support uses like that, don't do any renaming for files that
are not in a directory named either "new" or "cur".

[*] https://github.com/krl/sluk/
2010-11-11 14:32:17 -08:00
Carl Worth
1ea0b8bfe1 lib: Be honest in the documentation of the maildir functions
I had originally hoped for better semantics, such as doing nothing in
non-maildir directories, and preserving unknown maildir flags that
happen to be present.

We could still do those things, of course, but for now, remove them
from the documentation since the implementation does not do these
things yet.
2010-11-11 04:51:55 -08:00
Carl Worth
37a8096fdc notmuch_message_tags_to_maildir_flags: Don't exit on failure to rename.
It is totally legitimate for a non-maildir directory to be named "new"
(and not have a directory next to it named "cur"). To support this
case at least, be silent about any rename failure.
2010-11-11 03:50:42 -08:00
Carl Worth
71a3201885 notmuch_message_tags_to_maildir_flags: Fix to rename multiple files
This function was documented as modifying every filename associated
with the message. Fix it to actually do that.
2010-11-11 03:47:11 -08:00
Carl Worth
8480a0a003 notmuch_filenames_create: Take a reference to underlying filename list
The notmuch_filenames_t object might easily outlive the original object
owning the filename list. So take a talloc reference to keep things safe.
2010-11-11 03:40:19 -08:00
Carl Worth
404db1de90 maildir_flags_to_tags: Avoid interpreting "no info" as "no flags set".
If a filename has no maildir info at all, (that is, it does not
contain the sequence ":2,"), we consider this distinct from a filename
with an empty maildir info, (the ":2," separator is present, but no
flags characters follow).

Specifically, we regard a missing info field as providing no
information, so tags will remain unchanged. On the other hand, an info
field that is present but has no flags set will cause various tags to
be cleared, (or in the case of "unread", added).

This fixes the "remove info" case of the maildir-sync tests in the
test suite.
2010-11-11 03:40:19 -08:00
Carl Worth
5a98880add lib: Document that absence of maildir flags can also remove/add tags
Previously the documentation of notmuch_message_maildir_flags_to_tags
suggested that the presence of a flag would cause tags to be added,
(or in the case of "unread", removed). But the case of absent maildir
flags was not explicitly described.

What we actually want, is that for supported flags, the absence of the
flag in all messages causes the corresponding tag to be removed,
(or in the case of "unread", added). So document that explicitly.

This is the case recently added to the test suite as a failing test,
(so we'll need to do bug fixing before the documentation is honest
here).
2010-11-11 03:40:19 -08:00
Carl Worth
81cbaafc0f Fix notmuch_message_tags_to_maildir_flags to effect rename immediately
We have tests to ensure that when the notmuch library renames a file
that that rename takes place immediately in the database, (without
requiring something like "notmuch new" to notice the change).

This was working when the code was first added, but recently broke in
the reworking of the maildir-synchronization interface since the
tags_to_maildir_flags function can no longer assume that it is being
called as part of _notmuch_message_sync.

Fortunately, the fix is as simple as adding an explicit call to
_notmuch_message_sync.
2010-11-11 03:40:19 -08:00
Carl Worth
4b6063397f Fix notmuch_message_maildir_flags_to_tags to iterate over filenames
As documented, this function now iterates over all filenames for the
message, computing a logical OR of the flags set on the filenames,
then uses the final result to set tags on the message.

This change fixes 3 of the 10 maildir-sync tests that have been
failing since being added.
2010-11-11 03:40:19 -08:00
Carl Worth
1d02dd64af lib: Add new, public notmuch_message_get_filenames
This augments the existing notmuch_message_get_filename by allowing
the caller access to all filenames in the case of multiple files for a
single message.

To support this, we split the iterator (notmuch_filenames_t) away from
the list storage (notmuch_filename_list_t) where previously these were
a single object (notmuch_filenames_t). Then, whenever the user asks
for a file or filename, the message object lazily creates a complete
notmuch_filename_list_t and then:

	For notmuch_message_get_filename, returns the first filename
	in the list.

	For notmuch_message_get_filenames, creates and returns a new
	iterator for the filename list.
2010-11-11 03:40:19 -08:00
Carl Worth
d87db88432 lib: Add new implementation of notmuch_filenames_t
The new implementation is simply a talloc-based list of strings. The
former support (a list of database terms with a common prefix) is
implemented by simply pre-iterating over the terms and populating the
list. This should provide no performance disadvantage as callers of
thigns like notmuch_directory_get_child_files are very likely to
always iterate over all filenames anyway.

This new implementation of notmuch_filenames_t is in preparation for
adding API to query all of the filenames for a single message.
2010-11-11 03:40:19 -08:00
Carl Worth
d422dcf0a2 lib: Remove the notion of TAGS_INVALID
This rather ugly hack was recently obviated by the removal of the
notmuch_database_set_maildir_sync function. Now, clients must make
explicit calls to do any syncrhonization between maildir flags and
tags. So the library no longer needs to worry about doing inconsistent
synchronization while a message is only partially added.
2010-11-11 03:40:19 -08:00
Carl Worth
bb74e9dff8 lib: Rework interface for maildir_flags synchronization
Instead of having an API for setting a library-wide flag for
synchronization (notmuch_database_set_maildir_sync) we instead
implement maildir synchronization with two new library functions:

	notmuch_message_maildir_flags_to_tags
  and   notmuch_message_tags_to_maildir_flags

These functions are nicely documented here, (though the implementation
does not quite match the documentation yet---as plainly evidenced by
the current results of the test suite).
2010-11-11 03:40:19 -08:00
Carl Worth
2c262042ac lib: Remove the synchronization of 'T' flag with "deleted" tag.
Tags in a notmuch database affect all messages with the identical
message-ID. But maildir tags affect individual files. And since
multiple files can contain the identical message-ID, there is not a
one-to-one correspondence between messages affected by tags and flags.

This is particularly dangerous with the 'T' (== "trashed") maildir
flag and the corresponding "deleted" tag in the notmuch
database. Since these flags/tags are often used to trigger
irreversible deletion operations, the lack of one-to-one
correspondence can be potentially dangerous.

For example, consider the following sequence:

  1. A third-party application is used to identify duplicate messages
     in the mail store, and mark all-but-one of each duplicate with
     the 'T' flag for subsequent deletion.

  2. A "notmuch new" operation reads that 'T' flag, adding the
     "deleted" flag to the corresponding messages within the notmuch
     database.

  3. A subsequent notmuch operation, (such as a "notmuch dump; notmuch
     restore" cycle) synchronized the "deleted" tag back to the mail
     store, applying the 'T' flag to all(!) filenames with duplicate
     message IDs.

  4. A third-party application reads the 'T' flags and irreversibly
     deletes all mail messages which had any duplicates(!).

In order to avoid this scenario, we simply refuse to synchronize the
'T' flag with the "deleted" tag. Instead, applications can set 'T' and
act on it to delete files, or can set "deleted" and act on it to
delete files. But in either case the semantics are clear and there is
never dangerous propagation through the one-to-many mapping of notmuch
message objects to files.
2010-11-11 02:35:03 -08:00
Michal Sojka
d9d3d3e6f0 Make maildir synchronization configurable
This adds group [maildir] and key 'synchronize_flags' to the
configuration file. Its value enables (true) or diables (false) the
synchronization between notmuch tags and maildir flags. By default,
the synchronization is disabled.
2010-11-10 13:09:32 -08:00
Michal Sojka
088801a14a Maildir synchronization
This patch allows bi-directional synchronization between maildir
flags and certain tags. The flag-to-tag mapping is defined by flag2tag
array.

The synchronization works this way:

1) Whenever notmuch new is executed, the following happens:
   o New messages are tagged with configured new_tags.
   o For new or renamed messages with maildir info present in the file
     name, the tags defined in flag2tag are either added or removed
     depending on the flags from the file name.

2) Whenever notmuch tag (or notmuch restore) is executed, a new set of
   flags based on the tags is constructed for every message and a new
   file name is prepared based on the old file name but with the new
   flags. If the flags differs and the old message was in 'new'
   directory then this is replaced with 'cur' in the new file name. If
   the new and old file names differ, the file is renamed and notmuch
   database is updated accordingly.

   The rename happens before the database is updated. In case of crash
   between rename and database update, the next run of notmuch new
   brings the database in sync with the mail store again.
2010-11-10 13:09:31 -08:00
Michal Sojka
b65a783195 lib: Return added message even if it already was in the database 2010-11-10 13:09:31 -08:00
Carl Worth
d064bd696c lib: Eliminate some redundant includes of xapian.h
Most files including this already include database-private.h which
includes xapian.h already.
2010-11-01 23:24:40 -07:00
Carl Worth
c81cecf620 lib: Add GCC visibility(hidden) pragmas to private header files.
This prevents any of the private functions from being leaked out
through the library interface (at least when compiling with a
recent-enough gcc to support the visibility pragma).
2010-11-01 22:35:48 -07:00
Carl Worth
67c3bc9db4 lib: Add some missing static qualifiers.
These various functions and data are all used only locally, so should
be marked static. Ensuring we get these right will avoid us accidentally
leaking unintended symbols through the library interface.
2010-11-01 21:58:43 -07:00
Carl Worth
4dd066ae6c Update library version to 1.2.0
This increment is for the recently-added functions:

	notmuch_query_get_query_string
	notmuch_query_get_sort

These were recently added to the library interface, but the library
version was not incremented at that time, (shame on me).
2010-11-01 16:13:44 -07:00
Michal Sojka
f7a688ec53 Do not call ldconfig when building Debian package
Hi,

If I want to build Debian package, it fails with the following message:

    ldconfig: Can't create temporary cache file /etc/ld.so.cache~: Permission denied
    make[1]: *** [install-lib] Error 1

The reason is that I build the package as a non-root user and make
install invokes ldconfig unconditionally. The following patch contains a
workaround, but I think that a more correct solution would be to check
the condition LIBDIR_IN_LDCONFIG directly when make install is invoked
rather than in configure as it is done now.

Signed-off-by: Michal Sojka <sojkam1@fel.cvut.cz>
2010-10-28 13:06:46 -07:00
Carl Worth
e83b40138e lib: Add two functions: notmuch_query_get_query_string and _get_sort
It can be handy to be able to query these settings from an existing
query object.
2010-10-28 10:30:26 -07:00
Carl Worth
f6cb896bc4 lib: Fix notmuch_query_search_threads to return NULL on any Xapian exception.
Previously, if the underlying search_messages hit an exception and returned
NULL, this function would ignore that and return a non-NULL, (but empty)
threads object. Fix this to properly propagate the error.
2010-10-22 17:56:58 -07:00
Carl Worth
8071c5cd64 lib: Fix "make install"
This has been broken since the addition of the test sub-directory to our
non-recursive make system.
2010-09-21 09:09:01 -07:00
Carl Worth
029830c1f1 lib: Fix use-after-free bug.
Thanks to the new git-based test suite, it's easy to run the whole
test suite in valgrind, (simply "make test OPTIONS="--valgrind"), and
doing so showed this obvious use-after-free bug, (triggered by the
thread-order tests).
2010-09-20 22:01:52 +00:00
Carl Worth
d64d0cc8d9 make install: Run ldconfig or install a DT_RUNPATH in binary as appropriate.
Various users were confused as to why they couldn't run notmuch
immediately after "make install", (with linker errors saying that
libnotmuch.so could not be found). The errors came from two different
causes:

1. The user had installed to a system library directory, but had not
   yet run ldconfig.

2. The user had installed to some non-system directory, and had not
   set the LD_LIBRARY_PATH variable.

With this change we fix both problems (on Linux) without the user
having to do anything additional. We first use ldconfig to find the
system library directories. If the user is installing to one of these,
then we run ldconfig as part of "make install".

For case (2) we use the -rpath and --enable-new-dtags linker options
to install a DT_RUNPATH entry in the binary. This entry tells the
dynamic linker where to find libnotmuch. Without the
--enable-new-dtags option only a DT_RPATH option would be installed,
(which has the drawback of not allowing any override with the
LD_LIBRARY_PATH variable).

Distributions (such as Debian and Fedora) don't want to see binaries
packaged with a DT_RPATH or DT_RUNPATH entry. This should be avoided
automatically as long as the packages install to standard locations,
(such as /usr/lib).
2010-06-04 16:52:56 -07:00
Carl Worth
7b78eb4af6 Add support (and tests) for messages with really long message IDs.
Scott Henson reported an internal error that occurred when he tried to
add a message that referenced another message with a message ID well
over 300 characters in length. The bug here was running into a Xapian
limit for the length of metadata key names, (which is even more
restrictive than the Xapian limit for the length of terms).

We fix this by noticing long message ID values and instead using a
message ID of the form "notmuch-sha1-<sha1_sum_of_message_id>". That
is, we use SHA1 to generate a compressed, (but still unique), version
of the message ID.

We add support to the test suite to exercise this fix. The tests add a
message referencing the long message ID, then add the message with the
long message ID, then finally add another message referencing the long
ID. Each of these tests exercise different code paths where the
special handling is implemented.

A final test ensures that all three messages are stitched together
into a single thread---guaranteeing that the three code paths all act
consistently.
2010-06-04 13:35:07 -07:00
Carl Worth
98845fdbb2 Avoid database corruption by not adding partially-constructed mail documents.
Previously we were using Xapian's add_document to allocate document ID
values for notmuch_message_t objects.  This had the drawback of adding
a partially constructed mail document to the database. If notmuch was
subsequently interrupted before fully populating this document, then
later runs would be quite confused when seeing the partial documents.

There are reports from the wild of people hitting internal errors of
the form "Message ... has no thread ID" for example, (which is
currently an unrecoverable error).

We fix this by manually allocating document IDs without adding
documents. With this change, we never call Xapian's add_document
method, but only replace_document with either the current document ID
of a message or a new one that we have allocated.
2010-06-04 10:16:53 -07:00
Carl Worth
361b9d4bd9 Fix misnamed function in internal documentation.
The documentation for several functions mentioned
_notmuch_message_set_sync which doesn't exist. Fix these to reference
_notmuch_message_sync instead.
2010-06-04 09:54:46 -07:00
Tomas Carnecky
a54cecfc8e Add support for the Solaris platform
Like on Mac OS X, the linker doesn't automatically resolve dependencies.

Signed-off-by: Tomas Carnecky <tom@dbservice.com>
2010-06-03 18:17:03 -07:00
Dirk Hohndel
a258cb32b3 Fix SEGV in _thread_cleanup_author if author ends with ', '
Admittedly, an author name ending in ',' guarantees this is spam, and
indeed this was triggered by a spam email, but that doesn't mean we
shouldn't handle this case correctly.
We now check that there is actually a component of the name (presumably
the first name) after the comma in the author name.

Signed-off-by: Dirk Hohndel <hohndel@infradead.org>
2010-04-27 16:34:27 -07:00
Carl Worth
0fb28c65f2 lib: Increment library version to 1.1.0
For the addition of the new NOTMUCH_SORT_UNSORTED value.
2010-04-27 02:02:14 -07:00
Carl Worth
c210d5632e lib: Re-implement moving of thread authors.
Just before releasing 0.3 we received reports of crashes that were
bisected to the commit adding thread-author moving. Sure enough,
valgrind pointed to buffer overruns in _thread_move_matched_author.

Rather than trying to make sense of all the by strncpy, strchr, +1,
and +2 of that code, I reimplemented thread-author ordering with a
pair of hash tables and an array.

Valgrind is at least happy now on the test cases it was complaining
about previously.
2010-04-27 01:48:03 -07:00
Dirk Hohndel
5b8b0377cb Make Received: header special in notmuch_message_file_get_header
With this patch the Received: header becomes special in the way
we treat headers - this is the only header for which we concatenate
all the instances we find (instead of just returning the first one).

This will be used in the From guessing code for replies as we need to
be able to walk ALL of the Received: headers in a message to have a
good chance to guess which mailbox this email was delivered to.

Signed-off-by: Dirk Hohndel <hohndel@infradead.org>
2010-04-26 14:44:06 -07:00
Dirk Hohndel
cd19671f51 Simple attempt to display author names in a friendlier way
This patch only addresses the typical Outlook/Exchange case
where we have "Last, First" <first.last@company.com> or
"Last, First MI" <first.mi.last@company.com>.

In the future we should be more fexible as to the formats
we recognize, but for now we address this one as it is the
Exchange default setting and therefore the most common one.

Signed-off-by: Dirk Hohndel <hohndel@infradead.org>
2010-04-26 11:45:29 -07:00
Dirk Hohndel
26d8d960ee Reorder displayed names of thread authors
When displaying threads as result of a search it makes sense to list those
authors first who match the search. The matching authors are separated from the
non-matching ones with a '|' instead of a ','

Imagine the default "+inbox" query. Those mails in the thread that
match the query are actually "new" (whatever that means). And some
people seem to think that it would be much better to see those author
names first. For example, imagine a long and drawn out thread that once
was started by me; you have long read the older part of the thread and
removed the inbox tag. Whenever a new email comes in on this thread,
prior to this patch the author column in the search display will first show
"Dirk Hohndel" - I think it should first show the actual author(s) of the new
mail(s).

Signed-off-by: Dirk Hohndel <hohndel@infradead.org>
2010-04-26 11:45:00 -07:00
Dirk Hohndel
57561414d7 Add authors member to message
message->authors contains the author's name (as we want to print it)
get / set methods are declared in notmuch-private.h

Signed-off-by: Dirk Hohndel <hohndel@infradead.org>
2010-04-26 11:44:49 -07:00
Carl Worth
138fd38afe lib: Ensure notmuch_query_search_messages returns NULL on an exception.
Previously, this function may have segfaulted immediately after
reporting the exception.
2010-04-24 07:27:50 -07:00
Carl Worth
e3e0e26806 lib: Document that notmuch_query_count_messages may return 0 if an exception occurs
This isn't a behavioral change---just a calrification in the documentation.
2010-04-24 07:27:50 -07:00
Carl Worth
9ef68f1444 lib: Audit all notmuch_database call for Xapian exception handling.
Our current approach is for top-level entry poitns in the library
to have try/catch blocks that catch any Xapian exception and print
a message. Add a few missing blocks and fix up the documentation.
2010-04-24 07:27:50 -07:00
Carl Worth
3dbef312fb lib: Audit calls to notmuch_message_get_header to handle NULL return
Sebastian Spaeth reported [*] a segfault within libnotmuch when
running notmuch operations while an asyncronous offlineimap job had
removed some files from the mail store. Avoid this by handling all
cases where notmuch_message_get_header could return NULL.

[*] See message id:87d3xqti3o.fsf@SSpaeth.de on notmuch@notmuchmail.org
2010-04-24 06:50:04 -07:00
Carl Worth
7c421b87b0 lib: Simplify code to set subject from matched message.
Simply moving the code from _add_matched_message to a new
_set_subject_from_message function.
2010-04-24 06:50:04 -07:00
Carl Worth
21965718a5 Revert "thread: Simplify code for assigning the subject."
This reverts commit 36e4459a32.

With the two previous reverts, this fixes the recent message-sorting
regression, so the test suite now passes again.
2010-04-22 14:01:41 -07:00
Carl Worth
a109966080 Revert "thread: Fix sort of search when constructing threads."
This reverts commit f43990ce13.
2010-04-22 14:00:33 -07:00
Carl Worth
6a0cba4ae0 Revert "thread: Removed unsed sort argument from _thread_add_matched_message"
This reverts commit 7fb56f9dc5.
2010-04-22 14:00:17 -07:00
Carl Worth
7fb56f9dc5 thread: Removed unsed sort argument from _thread_add_matched_message
The reworked solution for naming a thread based on the subject of
oldest/newest matching message no longer needs this argument.
2010-04-21 17:05:16 -07:00
Sebastian Spaeth
aadb15a002 query.cc: allow to return query results unsorted
Previously, we always sorted the returned results by some string value,
(newest-to-oldest by default), however in some cases (as when applying
tags to a search result) we are not interested in any special order.

This introduces a NOTMUCH_SORT_UNSORTED value that does just that. It is
not used at the moment anywhere in the code.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2010-04-21 16:06:05 -07:00
Carl Worth
f43990ce13 thread: Fix sort of search when constructing threads.
The thread-naming feature depends on the matched messages being passed
down in a precise order, (the order of the top-level search). We fix
the feature by passing that sort order down.
2010-04-21 15:52:28 -07:00
Carl Worth
36e4459a32 thread: Simplify code for assigning the subject.
We know that matched messages are always added in order, so we can
always just grab the subject from the first message. This is the same
approach that was used previously in _thread_add_message. That is, the
recent feature of renaming a thread based on the subject of the
"first" matched message is as simple as moving the subject assignment
from _thread_add_message to _thread_add_matched_message.
2010-04-21 15:06:02 -07:00
Jesse Rosenthal
4971b85641 Name thread based on matching msgs instead of first msg.
At the moment all threads are named based on the name of the first message
in the thread. However, this can cause problems if people either start
new threads by replying-all (as unfortunately, many out there do) or
change the subject of their mails to reflect a shift in a thread on a
list.

This patch names threads based on (a) matches for the query, and (b) the
search order. If the search order is oldest-first (as in the default
inbox) it chooses the oldest matching message as the subject. If the
search order is newest-first it chooses the newest one.

Reply prefixes ("Re: ", "Aw: ", "Sv: ", "Vs: ") are ignored
(case-insensitively) so a Re: won't change the subject.

Note that this adds a "sort" argument to _notmuch_thread_create and
_thread_add_matched_message, so that when constructing the thread we can
be aware of the sort order.

Signed-off-by: Jesse Rosenthal <jrosenthal@jhu.edu>
2010-04-21 14:56:53 -07:00
Carl Worth
c48dcc302c lib: search_threads: Fix nested search to handle original search of "*"
When constructing a thread, we usually run a nested query to find all
messages in the thread that match the original search string. However,
we need to have special-case handling of an original search string of
"*" now that that is a supported means of specifying all messages.

The special-case ends up bein quite simple---we do less work, (just
skipping the nested search since we know that all messages must
match). I had been wanting to write this identical code to more
efficiently handle "notmuch search thread:<foo>" which was previously
running two identical searches. So that case is now more efficient as
well.
2010-04-15 14:54:40 -07:00
Carl Worth
72ea1b71c6 Makefile: Add library version information on OS X.
This encodes the library version into the library, where the linking
binary can pick it up, and the linker can even enforce mismatches in
the minor release, (such as linking a binary against version 1.2 and
then attempting to run it against version 1.1).
2010-04-14 16:18:19 -07:00
Carl Worth
1036867897 Makefile: Fix library linking command for OS X
I'm not sure which system Aaron used, but on the machine I have access
to, (Darwin 8.11.0), the -shared and -dylib_install_name options are
not recognized. Instead I use -dynamic_lib and -install_name as
documented here:

http://www.finkproject.org/doc/porting/shared.php
2010-04-14 16:16:05 -07:00
Aaron Ecay
8c8079a8b1 Add infrastructure for building shared library on OS X.
This patch adds a configure check for OS X (actually Darwin),
and sets up the Makefiles to build a proper shared library on
that platform.

Signed-off-by: Aaron Ecay <aaronecay@gmail.com>
2010-04-14 16:10:27 -07:00
Carl Worth
f206408358 Makefile: Move compat sources from the client code to the library.
Since the library code needs these as well.
2010-04-14 16:03:18 -07:00
martin f. krafft
449a418c65 Do not segfault on empty mime parts
notmuch previously unconditionally checked mime parts for various
properties, but not for NULL, which is the case if libgmime encounters
an empty mime part.

Upon encounter of an empty mime part, the following is printed to
stderr (the second line due to my patch):

  (process:17197): gmime-CRITICAL **: g_mime_message_get_mime_part: assertion `GMIME_IS_MESSAGE (message)' failed
  Warning: Not indexing empty mime part.

This is probably a bug that should get addressed in libgmime, but for
not, my patch is an acceptable workaround.

Signed-off-by: martin f. krafft <madduck@madduck.net>
2010-04-13 08:49:06 -07:00
Michael Forney
9ddde6eb14 Fix typo in notmuch.h documentation regarding database open modes
Reviewed-by: Carl Worth <cworth@cworth.org>:

The original proposal for having different open modes used the name
WRITABLE. I didn't like that name, (easy to misspell as WRITEABLE even
for native English speakers). So we renamed it to READ_WRITE
immediately, but apparently some of the documentation held the old
name for a while.
2010-04-13 08:39:10 -07:00
Carl Worth
14073b8851 lib: Remove condition regarding a NULL parent_thread_id.
A recent change guaranteed that a message ID can never be resolved
to a NULL thread ID, so we don't need this extra case.
2010-04-12 15:54:03 -07:00
Carl Worth
071022c253 lib: Always add reference terms to the database.
Previously, we were only adding the reference terms for cases where
the referenced message did not yet exist in the database. For thread
presentation, it's useful to have the connection information provided
by the references, even when the messages are present. So add this
term unconditionally.
2010-04-12 15:45:40 -07:00
Carl Worth
328626d0fd lib: Document the metadata stored within the Xapian database.
We are currently storing "version", "last_thread_id", and
"thread_id_*" values so document how each of these are used.
2010-04-12 15:15:14 -07:00
Carl Worth
af49741228 lib: Fix line-wrapping in _notmuch_database_link_message.
This function had some excessively long lines due to nested
expressions. It's simple enough to un-nest these and have readable
line lengths.
2010-04-12 14:41:34 -07:00
Carl Worth
f8dc5c08e4 lib: Fix internal documentation of _notmuch_database_link_message
This function was recently modified, (to include a metadata lookup for
a message's thread ID before looking for parent/child thread IDs), but
the documentation wasn't updated. Fix that.
2010-04-12 14:35:25 -07:00
Carl Worth
5c20bdf035 lib: Simplify code flow in _resolve_message_id_to_thread_id
There are two primary cases in this function, (the message exists in
the database or it does not). Previously the code for these two cases
was split and intermingled with goto-spaghetti connections.
2010-04-12 14:29:36 -07:00
Carl Worth
e9bb90ba2c lib: Fix internal documentation of _resolve_message_id_to_thread_id
We no longer return NULL, but instead generate a new thread ID for
messages that we haven't seen yet.
2010-04-12 14:19:15 -07:00
James Westby
40ea73cf05 Store thread ids for messages that we haven't seen yet
This allows us to thread messages even when we receive them out of
order, or never receive the root.

The thread ids for messages that aren't present but are referred to are
stored as metadata in the database and then retrieved if we ever get
that message.

When determining the thread id for a message we also check for this
metadata so that we can thread descendants of a message together before
we receive it.

Edited by Carl Worth <cworth@cworth.org>: Split this portion of the
commit from the earlier-applied portion adding test cases.
2010-04-12 14:11:57 -07:00
Carl Worth
e100871981 lib: Handle "*" as a query string to match all messages.
This seems like a generally useful thing to support, (but the previous
support through an empty string was not convenient for some users,
(such as the command-line client).
2010-04-09 17:43:58 -07:00
Dirk Hohndel
4563f669ca fix obvious cut and paste error
the wrong variable is checked for success of an allocation

Signed-off-by: Dirk Hohndel <hohndel@infradead.org>
2010-04-06 18:55:56 -07:00
Dirk Hohndel
a48f368778 fix notmuch_message_file_get_header
fix notmuch_message_file_get_header to always return the first instance
 of the header you are looking for

Signed-off-by: Dirk Hohndel <hohndel@infradead.org>
2010-04-06 18:47:28 -07:00
Carl Worth
ae9d67fd81 Avoid needlessly linking final notmuch binary against libXapian.
The libnotmuch.so library already does, so we don't need to do
it again. (Thanks to a Debian debhelper warning for pointing this
out.)
2010-04-06 18:30:43 -07:00
Carl Worth
14e98e454e configure: Add support for a --includedir option
Very similar to the existing --libdir option.
2010-04-06 14:42:09 -07:00
Carl Worth
f89b3d16db Makefiles: Eliminate the useless quiet_* functions.
With the original quiet function, there's an actual purpose (hiding
excessively long compiler command lines so that warnings and errors
from the compiler can be seen).

But with things like quiet_symlink there's nothing quieter. In fact
"SYMLINK" is longer than "ln -sf". So all this is doing is hiding the
actual command from the user for no real benefit.

The only actual reason we implemented the quiet_* functions was to be
able to neatly right-align the command name and left-align the arguments.

Let's give up on that, and just left-align everything, simplifying the
Makefiles considerably. Now, the only instances of a captialized command
name in the output is if there's some actually shortening of the command
itself.
2010-04-06 14:36:31 -07:00
Carl Worth
da2403c310 RELEASING: Add this file describing the steps to make a release.
These steps might be changing a bit as we work on making the initial
0.1 release.
2010-04-05 15:29:54 -07:00
David Edmondson
d3884a5984 Makefile.local: Automatically use makefile mode
We add a magic line to the beginning of each Makefile.local file to
help the editor know that it should use makefile mode for editing the
file, (even though the filename isn't exactly "Makefile").

Edited-by: Carl Worth <cworth@cworth.org>: Expand treatment from
emacs/Makefile.local to each instance of Makefile.local.
2010-04-03 12:31:49 -07:00
Carl Worth
f689c83af4 Compile a static notmuch binary (but only install the shared version)
The idea here is to allow a new user of notmuch to be able to run
notmuch immediately after compiling, (without having to install
the shared library first). This also ensures that the test suite
tests the locally compiled library, and not whatever installled
version of the library the dynamic linker happens to find.
2010-04-01 15:03:40 -07:00
Michal Sojka
b884ab2ef1 Makefile: Create include directory when installing headers
When I wanted to create a debian package from the current master, make
install failed because of non-existent include directory. This patch
fixes this minor issue.
2010-04-01 05:13:21 -07:00
Carl Worth
c0961e6a82 lib: Switch to a 3-part version number for the library interface.
With a carefully documented description of how to increment the
various version components.
2010-04-01 00:41:25 -07:00
Carl Worth
33d5cc415e Makefiles: Make the install rules quiet like the compilation rules.
The output from make is looking better all the time, (though the
columns still aren't lined up).
2010-03-31 23:54:21 -07:00
Carl Worth
7b52b2c318 Move installation of library from top-level to lib/Makefile.local
We had a fairly ugly violation of modularity with the top-level
Makefile.local isntalling everything, (even when the build commands
for the library were down in lib/Makefile.local).
2010-03-31 22:54:15 -07:00
Saleem Abdulrasool
07378d0d14 Fix target dependencies for multiple jobs
Signed-off-by: Ingmar Vanhassel <ingmar@exherbo.org>
2010-03-31 17:41:28 -07:00
Ben Gamari
266ab595a2 Build and link against notmuch shared library, install notmuch.h
Signed-off-by: Ingmar Vanhassel <ingmar@exherbo.org>
2010-03-31 17:38:27 -07:00
Carl Worth
b957a1b029 emacs: Fix the notmuch-search-authors-width variable.
This variable existed previously, but wasn't actually used for anything.
2010-03-31 13:32:00 -07:00
Carl Worth
e002fe8a7a Clarify documentation of notmuch_database_add_message.
For the case of adding a file that already exist, (with the same
filename). In this case, nothing will happen to the database, but
that wasn't clear before.
2010-03-31 13:31:10 -07:00
Carl Worth
86232e62ab Makefile: Fix Makefiles to depend on all child Makefile fragments.
We were previously maintaining two lists of the child Makefile
fragments---one for the includes and another for the dependencies. So,
of course, they drifted and the dependency list wasn't up to date.

We fix this by adding a single subdirs variable, and then using GNU
Makefile substitution to generate both the include and the dependency
lists.

Some side effect of this change caused the '=' assignment of the dir
variable to not work anymore. I'm not sure why that is, but using ':='
makes sense here and fixes the problem.
2010-03-10 10:59:57 -08:00
Carl Worth
e3046c688b Add is:<tag> as a synonym for tag:<tag> in search terms.
I like the readability of this, it provides compatibility with people
trained in this syntax by sup, and it even saves one character.
2010-03-09 16:03:58 -08:00
Carl Worth
c446f22dee lib: Silence a compiler warning.
The original code was harmless, but apparently some compilers aren't
able to think deep enough to catch that.
2010-03-09 12:07:26 -08:00
Fernando Carrijo
7f2629520c Fix a few documentation typos in notmuch.h
Fix a few documentation typos in notmuch.h

Signed-off-by: Fernando Carrijo <fcarrijo@yahoo.com.br>
2010-03-09 10:32:58 -08:00
Fernando Carrijo
bc69bf09cb Update documentation of notmuch_query_create
Commit cd467caf renamed notmuch_query_search to notmuch_query_search_messages.
Commit 1ba3d46f created notmuch_query_search_threads. We better keep the docs
of notmuch_query_create consistent with those changes.

Signed-off-by: Fernando Carrijo <fcarrijo@yahoo.com.br>

Edited-by: Carl Worth to explicitly list the full name of each
function being referenced.
2010-03-09 10:29:38 -08:00
Carl Worth
64646841f7 lib: Document what move_to_next does at the end of the list.
Explicitly mention that there's an invalid position after the last
item in the list.
2010-03-09 09:24:45 -08:00
Carl Worth
4e5d2f22db lib: Rename iterator functions to prepare for reverse iteration.
We rename 'has_more' to 'valid' so that it can function whether
iterating in a forward or reverse direction. We also rename
'advance' to 'move_to_next' to setup parallel naming with
the proposed functions 'move_to_first', 'move_to_last', and
'move_to_previous'.
2010-03-09 09:22:29 -08:00
Carl Worth
e0a8dee8bc Fix printf for when uint64_t != unsigned long long int
Thanks to Michal Sojka <sojkam1@fel.cvut.cz> for pointing out the
correct fix, which I verified in the freely-available WG14/N1124 draft
(from the C99 working group) which is available here:

http://www.open-std.org/JTC1/SC22/wg14/www/docs/n1124.pdf
2010-02-09 11:14:16 -08:00
Carl Worth
9439b217c3 Switch from random to sequential thread identifiers.
The sequential identifiers have the advantage of being guaranteed to
be unique (until we overflow a 64-bit unsigned integer), and also take
up half as much space in the "notmuch search" output (16 columns
rather than 32).

This change also has the side effect of fixing a bug where notmuch
could block on /dev/random at startup (waiting for some entropy to
appear). This bug was hit hard by the test suite, (which could easily
exhaust the available entropy on common systems---resulting in large
delays of the test suite).
2010-02-09 11:14:11 -08:00
Carl Worth
7a9bacac67 notmuch.h: Fix a couple of typos in the documentation.
Obviously, the spell-checker isn't able to catch every mistake
I make.
2010-02-05 17:31:40 -08:00
Carl Worth
2bc0af15aa Eliminate some useless gobject boilerplate.
If we had external users of this filter then they might expect some of
these macros to exist. But since this is just internal, that's just
unneeded noise.
2010-02-04 17:26:00 -08:00
Carl Worth
3767c6f9f9 notmuch new: Don't index uuencoded data.
With modern MIME attachments, we're already avoiding indexing the
attachments. But for old-school uuencoded data in the mail, we have
been directly indexing the encoded data as terms, (which is not useful
at all---nobody will ever ytry to search based on the seemingly random
uuencoded data).

Additionally, indexing a modestly large uuencoded file seems to make
Xapian go insane, (consuming *lots* of memory).

We fix both problems by detecting uuencoded content and not performing
any indexing of it.
2010-02-04 17:08:11 -08:00
Carl Worth
c340c1bd11 notmuch new: Print upgrade progress report as a percentage.
Previously we were printing a number of messages upgraded so far. The
original motivation for this was to accurately reflect the fact that
there are two passes, (so each message is processed twice and it's not
accurate to represent with a single count). But as it turns out, the
second pass takes zero time (relatively speaking) so we're still not
accounting for it.

If nothing else, the percentage-based reporting makes for a cleaner
API for the progress_notify function.
2010-01-09 17:38:23 -08:00
Carl Worth
ccf2e0cc42 lib: Add non-content terms with a WDF value of 0.
The WDF is the "within-document frequency" value for a particular
term. It's intended to provide an indication of how frequent a term is
within a document, (for use in computing relevance). Xapian's term
generator already computes WDF values when we use that, (which we do
for indexing all mail content).

We don't use the term generator when adding single terms for things
that don't actually appear in the mail document, (such as tags, the
filename, etc.). In this case, the WDF value for these terms doesn't
matter much.

But Xapian's flint backend can be more efficient with changes to terms
that don't affect the document "length". So there's a performance
advantage for manipulating tags (with the flint backend) if the WDF of
these terms is 0.
2010-01-09 11:18:27 -08:00
Carl Worth
45b1856782 lib: Explicitly set BoolWeight when searching.
All notmuch searches currently sort by value (either date or message
ID) so it's just wasted effort for Xapian to compute relevance values
for each result. We now explicitly tell Xapian that we're uninterested
in the relevance values.
2010-01-09 11:16:40 -08:00
Carl Worth
d12801c8b4 lib: Split the database upgrade into two phases for safer operation.
The first phase copies data from the old format to the new format
without deleting anything. This allows an old notmuch to still use the
database if the upgrade process gets interrupted. The second phase
performs the deletion (after updating the database version number). If
the second phase is interrupted, there will be some unused data in the
database, but it shouldn't cause any actual harm.
2010-01-09 11:13:12 -08:00
Carl Worth
5fe5e802ab lib: Delete stale timestamp documents during database upgrade.
Once we move the timestamp to the new directory document, we don't
need the old one anymore.
2010-01-08 09:57:09 -08:00
Carl Worth
1c86b48329 notmuch new: Fix progress notification on database upgrade.
This was firing continuously rather than just once per second as
intended.
2010-01-07 21:24:44 -08:00
Carl Worth
909f52bd8c lib: Implement versioning in the database and provide upgrade function.
The recent support for renames in the database is our first time
(since notmuch has had more than a single user) that we have a
database format change. To support smooth upgrades we now encode a
database format version number in the Xapian metadata.

Going forward notmuch will emit a warning if used to read from a
database with a newer version than it natively supports, and will
refuse to write to a database with a newer version.

The library also provides functions to query the database format
version:

	notmuch_database_get_version

to ask if notmuch wants a newer version than that:

	notmuch_database_needs_upgrade

and a function to actually perform that upgrade:

	notmuch_database_upgrade
2010-01-07 18:26:31 -08:00
Carl Worth
807aef93d3 Prefer READ_ONLY consistently over READONLY.
Previously we had NOTMUCH_DATABASE_MODE_READ_ONLY but
NOTMUCH_STATUS_READONLY_DATABASE which was ugly and confusing. Rename
the latter to NOTMUCH_STATUS_READ_ONLY_DATABASE for consistency.
2010-01-07 10:29:05 -08:00
Carl Worth
f93b7218c3 lib: Consolidate checks for read-only database.
Previously, many checks were deep in the library just before a cast
operation. These have now been replaced with internal errors and new
checks have instead been added at the beginning of all top-levelentry
points requiring a read-write database.

The new checks now also use a single function for checking and
printing the error message. This will give us a convenient location to
extend the check, (such as based on database version as well).
2010-01-07 10:19:44 -08:00
Carl Worth
6ed606c19e lib: Clarify internal documentation of _notmuch_database_filename_to_direntry
The original wording made it sound like this function was just doing
some string manipulation. But this function actually creates new
directory documents as a side effect. So make that explicit in its
documentation.
2010-01-07 09:31:58 -08:00
Carl Worth
a274848f95 notmuch_message_get_filename: Support old-style filename storage.
When a notmuch database is upgraded to the new database format, (to
support file rename and deletion), any message documents corresponding
to deleted files will not currently be upgraded. This means that a
search matching these documents will find no filenames in the expected
place.

Go ahead and return the filename as originally stored, (rather than
aborting with an internal error), in this case.
2010-01-07 09:22:34 -08:00
Carl Worth
957ae198e7 lib: Treat NULL as a valid (and empty) notmuch_filenames_t iterator.
This will be convenient to avoid some special-casing in higher-level
code.
2010-01-06 14:35:11 -08:00
Carl Worth
4b418343f6 lib: Indicate whether notmuch_database_remove_message removed anything.
Similar to the return value of notmuch_database_add_message, we now
enhance the return value of notmuch_database_remove_message to
indicate whether the message document was entirely removed (SUCCESS)
or whether only this filename was removed and the document exists
under other filenamed (DUPLICATE_MESSAGE_ID).
2010-01-06 10:32:06 -08:00
Carl Worth
777cd23d9d lib: Update documentation of notmuch_database_add_message.
Previously, adding a filename with the same message ID as an existing
message would do nothing. But we recently fixed this to instead add
the new filename to the existing message document. So update the
documentation to match now.
2010-01-06 10:32:06 -08:00
Carl Worth
6ef6ddba80 Index content from citations and signatures.
In the presentation we often omit citations and signatures, but this
is not content that should be omitted from the index, (especially
when the citation detection is wrong---see cases where a line
beginning with "From" is corrupted to ">From" by mail processing
tools).
2010-01-06 10:32:06 -08:00
Carl Worth
341d49b061 Makefiles: Use .DEFAULT to support arbitrary targets from sub directories.
Taking advantage of the .DEFAULT construct means that we won't need to
explicitly list targets such as "clean", etc. in each sub-Makefile.
2010-01-06 10:32:06 -08:00
Carl Worth
3f32fd8a1c Add missing comment for NOTMUCH_STATUS_READONLY_DATABASE.
And adjust the string representation of the same to match.
2010-01-06 10:32:06 -08:00
Carl Worth
d807e28f43 lib: Implement new notmuch_directory_t API.
This new directory ojbect provides all the infrastructure needed to
detect when files or directories are deleted or renamed. There's still
code needed on top of this (within "notmuch new") to actually do that
detection.
2010-01-06 10:32:06 -08:00
Carl Worth
ba07fe1819 Revamp the proposed directory-tracking API slightly.
This commit contains my changes to the API proposed by Keith. Nothing
is dramatically different. There are minor things like changing
notmuch_files_t to notmuch_filenames_t and then various things needed
for completeness as noticed while implementing this, (such as
notmuch_directory_destroy and notmuch_directory_set_mtime).
2010-01-06 10:32:06 -08:00
Keith Packard
95deec1b27 Prototypes for directory tracking
There's no functionality here yet---just a sketch of what the
interface could look like.
2010-01-06 10:32:06 -08:00
Carl Worth
f11aaa3678 database: Add new, public notmuch_database_remove_message
This will allow applications to support the removal of messages, (such
as when a file is deleted from the mail store). No removal support is
provided yet in commands such as "notmuch new".
2010-01-06 10:32:06 -08:00
Carl Worth
44a74912c7 database: Add new find_doc_ids_for_term interface.
The existing find_doc_ids function is convenient when the caller
doesn't want to be bothered constructing a term. But when the caller
*does* have the term already, that interface is just wasteful. So we
export a lower-level interface that maps a pre-constructed term to a
document-ID iterators.
2010-01-06 10:32:06 -08:00
Carl Worth
d7e5f5827e database: Make find_unique_doc_id enforce uniqueness (for a debug build)
Catching any violation of this unique-ness constraint is very much in
line with similar, existing INTERNAL_ERROR cases.
2010-01-06 10:32:06 -08:00
Carl Worth
498edff503 database: Abstract _filename_to_direntry from _add_message
The code to map a filename to a direntry is something that we're going
to want in a future _remove_message function, so put it in a new
function _notmuch_database_filename_to_direntry .
2010-01-06 10:32:05 -08:00
Carl Worth
1376a90db6 database: Allowing storing multiple filenames for a single message ID.
The library interface is unchanged so far, (still just
notmuch_database_add_message), but internally, the old
_set_filename function is now _add_filename instead.
2010-01-06 10:32:05 -08:00
Carl Worth
6ca6c089e9 database: Store mail filename as a new 'direntry' term, not as 'data'.
Instead of storing the complete message filename in the data portion
of a mail document we now store a 'direntry' term that contains the
document ID of a directory document and also the basename of the
message filename within that directory. This will allow us to easily
store multple filenames for a single message, and will also allow us
to find mail documents for files that previously existed in a
directory but that have since been deleted.
2010-01-06 10:32:05 -08:00
Carl Worth
84742d86ab database: Split _find_parent_id into _split_path and _find_directory_id
Some pending commits want the _split_path functionality separate from
mapping a directory to a document ID. The split_path function now
returns the basename as well as the directory name.
2010-01-06 10:32:05 -08:00
Carl Worth
4c1cca888f database: Store directory path in 'data' of directory documents.
We're planning to have mail documents refer to directory documents for
the path of the containing directory. To support this, we need the
path in the data, (since the path in the 'directory' term can be
irretrievable as it will be the SHA1 sum of the path for a very long
path).
2010-01-06 10:32:05 -08:00
Carl Worth
406ec4b15d database: Export _notmuch_database_find_parent_id for internal use.
We'll soon have mail documents referring to their parent directory's
directory documents, so we'll need access to _find_parent_id in files
such as message.cc.
2010-01-06 10:32:05 -08:00
Carl Worth
e890b0cf40 database: Store the parent ID for each directory document.
Storing the document ID of the parent of each directory document will
allow us to find all child-directory documents for a given directory
document. We will need this in order to detect directories that have
been removed from the mail store, (though we aren't yet doing this).
2010-01-06 10:32:05 -08:00
Carl Worth
851c97aed7 database: Rename internal directory value from XTIMESTAMP to XDIRECTORY.
The recent change from storing absolute paths to relative paths means
that new directory documents will already be created, (and the old
ones will just linger stale in the database). Given that, we might as
well put a clean name on the term in the new documents, (and no real
flag day is needed).
2010-01-06 10:32:05 -08:00
Carl Worth
154bf7ac67 database: Store directory paths as relative, not absolute.
We were already storing relative mail filenames, so this is consistent
with that. Additionally, it means that directory documents remain
valid even if the database is relocated within its containing
filesystem.
2010-01-06 10:32:05 -08:00
Carl Worth
9257622da8 lib: Document that the filename is stored in the 'data' of a mail document
Our database schema documentation previously didn't give any
indication of where this most essential piece of information is
stored.
2010-01-06 10:32:05 -08:00
Carl Worth
50ae83a17f lib: Rename set/get_timestamp to set/get_directory_mtime.
I've been suitably scolded by Keith for doing a premature
generalization that ended up just making the documentation more
convoluted. Fix that.
2010-01-06 10:32:05 -08:00
Carl Worth
ba12bf1f26 lib: Abstract the extraction of a relative path from set_filename
We'll soon be having multiple entry points that accept a filename
path, so we want common code for getting a relative path from a
potentially absolute path.
2010-01-06 10:32:05 -08:00
Carl Worth
8c6b7d311c lib: Add missing value to notmuch_private_status_t enum.
And fix the initialization such that the private enum will always have
distinct values from the public enum even if we similarly miss the
addition of a new public value in the future.
2010-01-06 10:32:05 -08:00
Fernando Carrijo
db68eea013 Nuke the remainings of _notmuch_message_add_thread_id.
The function _notmuch_message_add_thread_id has been removed
from the private interface of notmuch. There's no reason for
one to keep a declaration of its prototype in the code base.
Also, lets update a commentary that referenced that function
and escaped from previous scrutiny.

Signed-off-by: Fernando Carrijo <fcarrijo@yahoo.com.br>
2009-12-09 12:09:55 -08:00
Jeffrey C. Ollie
95f97540a0 Remove unused notmuch_parse_date function prototype.
notmuch_parse_date is not implemented, so remove the unused function
prototype.

Signed-off-by: Jeffrey C. Ollie <jeff@ocjtech.us>
2009-12-03 17:07:22 -08:00
Carl Worth
880b21a097 Makefile: Incorporate getline implementation into the build.
It's unconditional for a very short time. We expect to soon be
building it only if necessary.
2009-12-01 16:33:17 -08:00
Carl Worth
8b445212e4 xutil: Implement xstrndup without relying on strndup.
Since we need to do this for portability, (some systems don't have a
strndup function), we might as well do it unconditionally. There's
almost no disadvantage to doing so, and this has the advantages of not
requiring a configure-time check nor having two different
implementations, one of which would often be less tested.
2009-12-01 12:51:39 -08:00
Carl Worth
e5316b320a lib/index: Fix memory leak for email addresses without names.
We carefully noted the fact that we had locally allocated the string
here, but then we neglected to free it. Switch to talloc instead
which makes it easier to get the behavior we want. It's simpler since
we can just call talloc_free unconditionally, without having to track
the state of whether we allocated the storage for name or not.
2009-12-01 12:40:13 -08:00
Carl Worth
64c8d6227a Avoid bogus internal error reporting duplicate In-Reply-To IDs.
This error was tirggered with a debugging build via:

	make CXXFLAGS="-DDEBUG"

and reported by David Bremner. The actual error is that I'm an
idiot that doesn't know how to use strcmp's return value. Of
course, the strcmp interface scores a negative 7 on Rusty Russell
ranking of bad interfaces:

http://ozlabs.org/~rusty/index.cgi/tech/2008-04-01.html
2009-11-28 10:01:22 -08:00
Bart Trojanowski
6da6566576 correct message flag enum value so that it matches the type
As per Carl's request, this patch corrects the only value defined under
the notmuch_message_flag_t enum typedef to match the name of the type.

Signed-off-by: Bart Trojanowski <bart@jukie.net>
2009-11-27 20:50:59 -08:00
David Bremner
12c91e8050 add missing comma in debugging code 2009-11-27 19:51:53 -08:00
Jeffrey C. Ollie
e991148b00 Silence compiler warning by initializing a variable.
If Xapian threw an exception on notmuch_query_count_messages the count
variable could be used uninitialized.  Initialize count to solve the
problem.

Signed-off-by: Jeffrey C. Ollie <jeff@ocjtech.us>
2009-11-27 18:38:06 -08:00