Commit graph

902 commits

Author SHA1 Message Date
Thomas Jost
1c71d943cb libnotmuch: only build symbols list after the modules are built
If the notmuch.sym target does not explicitly depend on $(libnotmuch_modules),
gen-version-script.sh may be run before all the .o files are created, for
example when doing a parallel build on a machine with many cores.
2011-07-20 21:42:35 -03:00
David Bremner
a900ddaba6 Merge commit '0.6.1'
Conflicts:
	lib/Makefile.local

The conflicts are from three kinds of commits not merged into release:
    - typo fixes
    - removal of debug output
    - fix for CLEAN rule

That were never merged into the release branch.
2011-07-17 22:20:37 -03:00
David Bremner
feb22fef2e libnotmuch: export Xapian typeinfo symbols
The lack of such exporting seems to cause problems catching
exceptions, as suggested by

    http://gcc.gnu.org/wiki/Visibility

This manifested in the symbol-hiding test failing when notmuch was
compile with gcc 4.4.5. On i386, this further manifested as notmuch
new failing to run (crashing with an uncaught exception on first run).
2011-07-16 22:16:46 -03:00
Robin Green
52e4dedf9a Use POSIX sed invocation
Fixes broken build on DragonFly BSD

Signed-off-by: Robin Green <greenrd@greenrd.org>
2011-07-03 08:32:51 -03:00
Mark Anderson
d77c7cefe4 Fix folder: coherence issue
Add removal of all ZXFOLDER terms to removal of all XFOLDER terms for
each message filename removal.

The existing filename-list reindexing will put all the needed terms
back in.  Test search-folder-coherence now passes.

Signed-off-by:Mark Anderson <ma.skies@gmail.com>
(cherry picked from commit 8a856e5c38)
2011-06-29 21:58:53 -03:00
David Bremner
f4dae4e5a1 lib/Makefile.local: remove leftover debugging output.
The removed "echo $(libnotmuch_modules)" was strictly for debugging.

Thanks to Austin Clements for the hint.
2011-06-29 15:35:47 -07:00
Mark Anderson
8a856e5c38 Fix folder: coherence issue
Add removal of all ZXFOLDER terms to removal of all XFOLDER terms for
each message filename removal.

The existing filename-list reindexing will put all the needed terms
back in.  Test search-folder-coherence now passes.

Signed-off-by:Mark Anderson <ma.skies@gmail.com>
2011-06-29 14:13:16 -07:00
David Bremner
8d46f05304 libnotmuch: build symbols list without relying on gcc -aux-info.
Carl reports "gcc -aux-info notmuch.aux lib/notmuch.h" does not
generate notmuch.aux for him with Debian gcc 4.6.0-8. A small
modification of the original sed regular expression allows us to work
directly from lib/notmuch.h, rather than preprocessing with gcc.

As with most such simple regex based "parsing", this is quite
sensitive to the input format, and needs that each symbol to be
exported from libnotmuch should

- start with "notmuch_"
- be the first non-whitespace token on the line
- be followed by an open parenthesis.

(Cherry-picked from 51b7ab6968, with conflicts resolved by db)
2011-06-28 16:32:07 -03:00
David Bremner
51b7ab6968 libnotmuch: build symbols list without relying on gcc -aux-info.
Carl reports "gcc -aux-info notmuch.aux lib/notmuch.h" does not
generate notmuch.aux for him with Debian gcc 4.6.0-8. A small
modification of the original sed regular expression allows us to work
directly from lib/notmuch.h, rather than preprocessing with gcc.

As with most such simple regex based "parsing", this is quite
sensitive to the input format, and needs that each symbol to be
exported from libnotmuch should

- start with "notmuch_"
- be the first non-whitespace token on the line
- be followed by an open parenthesis.
2011-06-28 11:59:48 -07:00
David Bremner
891082befa libnotmuch: fix typos in CLEAN setting, add file
- c0961e6 introduced a missing slash between $(dir)$(LIBNAME) and missing
  $(dir) in front of libnotmuch.a
- cdf1c70a created a file $(dir)/notmuch.h.gch and neglected to
  add it to CLEAN
2011-06-28 11:59:48 -07:00
Pieter Praet
8bb6f7869c fix sum moar typos [comments in source code]
Various typo fixes in comments within the source code.

Signed-off-by: Pieter Praet <pieter@praet.org>

Edited-by: Carl Worth <cworth@cworth.org> Restricted to just
source-code comments, (and fixed fix of "descriptios" to "descriptors"
rather than "descriptions").
2011-06-23 15:58:39 -07:00
Pieter Praet
730acd4764 fix sum moar typos [build scripts, Makefiles]
Various typo fixes in comments within the Makefile and other build scripts.

Signed-off-by: Pieter Praet <pieter@praet.org>

Edited-by: Carl Worth <cworth@cworth.org> Restricted to just build files.
2011-06-23 15:44:59 -07:00
David Bremner
cdf1c70ab3 libnotmuch: add linker script to declare only notmuch_* symbols as global.
This is closely tied to gcc and particularly gnu ld, but I guess the
shared library linking code would need to be adjusted to work on a
non-gnu linker anyay.

I had to make a few not-obviously related changes to the
lib/Makefile.local to make this work: libnotmuch_modules is defined
with := and used in place of $^
(cherry picked from commit 014bf85b1c06ff49be2bde5a26433d2cf376cf70)
2011-06-23 07:04:56 -03:00
Stewart Smith
c86d77b16a Fix appending of Received headers
We're not properly concatenating the Received headers if we parse them
while requesting a header that isn't Received.

this fixes notmuch-reply address detection in a bunch of situations.
2011-06-10 17:03:14 -07:00
Anton Khirnov
d3fdb76c8d lib/message-file: plug three memleaks.
Signed-off-by: Jameson Graef Rollins <jrollins@finestructure.net>
2011-06-03 12:30:55 -07:00
Jameson Graef Rollins
1d6b49561f tag signed/encrypted during notmuch new
This patch adds the tag "signed" to messages with any multipart/signed
parts, and the tag "encrypted" to messages with any
multipart/encrypted parts.  This only occurs when messages are indexed
during notmuch new, so a database rebuild is required to have old
messages tagged.
2011-05-27 16:22:00 -07:00
Carl Worth
d5523ead90 Mark some structures in the library interface with visibility=default attribute.
As of gcc 4.6, there are new warnings from -Wattributes along the lines of:

	warning: ‘_notmuch_messages’ declared with greater visibility
	than the type of its field ‘_notmuch_messages::iterator’
	[-Wattributes]

To squelch these, we decorate all such containing structs with
__attribute__((visibility("default"))). We take care to let only the
C++ compiler see this, (since the C compiler would otherwise warn
about ignored visibility attributes on types).
2011-05-11 13:27:15 -07:00
Carl Worth
2f3a76c569 Remove some variables which were set but not used.
gcc (at least as of version 4.6.0) is kind enough to point these out to us,
(when given -Wunused-but-set-variable explicitly or implicitly via -Wunused
or -Wall).

One of these cases was a legitimately unused variable. Two were simply
variables (named ignored) we were assigning only to squelch a warning about
unused function return values. I don't seem to be getting those warnings
even without setting the ignored variable. And the gcc docs. say that the
correct way to squelch that warning is with a cast to (void) anyway.
2011-05-11 13:27:14 -07:00
Austin Clements
b599bbe672 Fixup string list author 2011-03-21 02:45:18 -04:00
Austin Clements
d19c5de17a Add the tag list to the unified message metadata pass.
Now each caller of notmuch_message_get_tags only gets a new iterator,
instead of a whole new list.  In principle this could cause problems
with iterating while modifying tags, but through the magic of talloc
references, we keep the old tag list alive even after the cache in the
message object is invalidated.

This reduces my index search from the 3.102 seconds before the unified
metadata pass to 1.811 seconds (1.7X faster).  Combined with the
thread search optimization in b3caef1f06,
that makes this query 2.5X faster than when I started.
2011-03-21 02:45:18 -04:00
Austin Clements
f271071330 Add the file name list to the unified message metadata pass.
Even if the caller never uses the file names, there is little cost to
simply fetching the file name terms.  However, retrieving the full
paths requires additional database work, so the expansion from terms
to full paths is performed lazily.

This also simplifies clearing the filename cache, since that's now
handled by the generic metadata cache code.

This further reduces my inbox search from 3.102 seconds before the
unified metadata pass to 2.206 seconds (1.4X faster).
2011-03-21 02:45:18 -04:00
Austin Clements
206938ec9b Add a generic function to get a list of terms with some prefix.
Replace _notmuch_convert_tags with this and simplify
_create_filenames_for_terms_with_prefix.  This will also come in handy
shortly to get the message file name list.
2011-03-21 02:45:18 -04:00
Austin Clements
f3c1eebfaf Implement an internal generic string list and use it.
This replaces the guts of the filename list and tag list, making those
interfaces simple iterators over the generic string list.  The
directory, message filename, and tags-related code now build generic
string lists and then wraps them in specific iterators.  The real wins
come in later patches, when we use these for even more generic
functionality.

As a nice side-effect, this also eliminates the annoying dependency on
GList in the tag list.
2011-03-21 02:45:18 -04:00
Austin Clements
d9b0ae918f Use a single unified pass to fetch scalar message metadata.
This performs a single pass over a message's term list to fetch the
thread ID, message ID, and reply-to, rather than requiring a pass for
each.  Xapian decompresses the term list anew for each iteration, so
this reduces the amount of time spent decompressing message metadata.

This reduces my inbox search from 3.102 seconds to 2.555 seconds (1.2X
faster).
2011-03-21 02:45:18 -04:00
Carl Worth
42b0409f8e Merge remote branch 'amdragon/search-perf-3' 2011-03-10 11:05:49 -08:00
Carl Worth
3e4a9d60a9 build: Add support for non-source-directory builds.
Such as:

     mkdir build
     cd build
     ../configure
     make

This is implemented by having the configure script set a srcdir
variable in Makefile.config, and then sprinkling $(srcdir) into
various make rules. We also use vpath directives to convince GNU make
to find the source files from the original source directory.
2011-03-09 15:10:03 -08:00
Austin Clements
56d19273fe Simplify _notmuch_doc_id_set_init interface.
Don't require the caller of _notmuch_doc_id_set_init to pass in a
correct bound; instead compute it from the array.  This simplifies the
caller and makes this interface easier to use correctly.
2011-01-30 21:01:56 -05:00
Austin Clements
e04e72f9dd Remove code repetition in the doc ID bitmap code.
Remove the repeated "sizeof (doc_ids->bitmap[0])" that bothered cworth
by instead defining macros to compute the word and bit offset of a
given bit in the doc ID set bitmap.
2011-01-30 20:58:27 -05:00
Carl Worth
1a915d1b38 Makefile: Quote variables used as filenames in shell commands
This allows support for filenames with spaces in them.
2011-01-26 23:36:52 +10:00
Carl Worth
db70f3f0c4 lib: Save and restore term position in message while indexing.
This fixes the recently addead search-position-overlap bug as
demonstrated in the test of the same name.
2011-01-26 15:59:19 +10:00
Carl Worth
b64afb1ab9 notmuch search: Clean up some memory leaks during search loop.
With talloc, we were already freeing all memory by the time we exited
the loop, but that didn't help with excess use of memory inside the
loop, (which was mostly from tallocing some objects with the incorrect
parent).

Thanks to Andrew Tridgell for sitting next to me and teaching me to
use talloc_report_full to find these leaks.
2011-01-26 15:59:19 +10:00
Carl Worth
99cfa27030 Add support for folder-based searching.
A new "folder:" prefix in the query string can now be used to match
the directories in which mail files are stored.

The addition of this feature causes the recently added
search-by-folder tests to now pass.
2011-01-15 15:37:43 -08:00
Carl Worth
8bef9ba922 Tighten up a memory allocation.
Using the local talloc context ensures that the memory we are using
here will be freed shortly, (rather than hanging on for a long time
with the notmuch database object).
2011-01-15 15:37:43 -08:00
Carl Worth
36161181df Correct some minor typos in a comment
Nothing too important here. Just some misspellings I noticed while reading
nearby code.
2011-01-15 15:37:43 -08:00
Austin Clements
b3caef1f06 Optimize thread search using matched docid sets.
This reduces thread search's 1+2t Xapian queries (where t is the
number of matched threads) to 1+t queries and constructs exactly one
notmuch_message_t for each message instead of 2 to 3.
notmuch_query_search_threads eagerly fetches the docids of all
messages matching the user query instead of lazily constructing
message objects and fetching thread ID's from term lists.
_notmuch_thread_create takes a seed docid and the set of all matched
docids and uses a single Xapian query to expand this docid to its
containing thread, using the matched docid set to determine which
messages in the thread match the user query instead of using a second
Xapian query.

This reduces the amount of time required to load my inbox from 4.523
seconds to 3.025 seconds (1.5X faster).
2010-12-07 16:40:05 -08:00
Carl Worth
b099fc9ea7 notmuch search: Fix to display authors in date order.
Previously, the authors of the thread were displayed in
reverse-chronological order, which was fairly confusing.
2010-12-07 16:34:52 -08:00
Carl Worth
a4ce6cfbc6 lib: Set thread subject at the same time as setting thread->{oldest,newest}
We really want to change the thread subject at the same time we set
the date, (if the sort order indicates this is necessary). The
previous code for setting the thread subject was sensitive on the
query sort when adding matching messages. An independent bug fix is
about to change that query sort order, so we remove the dependency on
it here.
2010-12-07 16:27:03 -08:00
Carl Worth
c7b4d15d0a Fix to index the "Re" term present in any subject.
This was a misfeature where notmuch had extra code that just threw
away legitimate information. It was never indexing an initial "Re"
term in a subject. But some users have legitimately wanted to search
for this term.

The original code was written this way merely for strict compatiblity
with the indexing performed by sup, but we're not taking advantage of
that now anyway.
2010-11-23 18:11:04 -08:00
Carl Worth
7278383005 lib: Fix missing initialization of status field.
This could have been a problematic bug. Fortuinately "gcc -O2" warns
about it.
2010-11-11 20:54:41 -08:00
Carl Worth
fe8eeaf4a5 lib: Add two missing static qualifiers
The debian packaging is nice enough to notice when we accidentally
leak private symbols to the public interface.
2010-11-11 20:53:21 -08:00
Carl Worth
d33dee5389 Increment library version to 1.3.0
For the addition of three functions:

	notmuch_message_get_filenames
	notmuch_message_tags_to_maildir_flags
	notmuch_message_maildir_flags_to_tags
2010-11-11 20:42:45 -08:00
Carl Worth
96d99c3837 tags_to_maildir_flags: Fix to preserve existing, unsupported flags
This is to prevent notmuch from destroying any information the user
has encoded as flags in the maildir filename. Tests are also added to
the test suite to verify the documented behavior.
2010-11-11 16:36:02 -08:00
Carl Worth
95dd5fe5d7 notmuch_message_tags_to_maildir_flags: Do nothing outside of "new" and "cur"
Some people use notmuch with non-maildir files, (for example, email
messages in MH format, or else cool things like using sluk[*] to suck
down feeds into a format that notmuch can index).

To better support uses like that, don't do any renaming for files that
are not in a directory named either "new" or "cur".

[*] https://github.com/krl/sluk/
2010-11-11 14:32:17 -08:00
Carl Worth
1ea0b8bfe1 lib: Be honest in the documentation of the maildir functions
I had originally hoped for better semantics, such as doing nothing in
non-maildir directories, and preserving unknown maildir flags that
happen to be present.

We could still do those things, of course, but for now, remove them
from the documentation since the implementation does not do these
things yet.
2010-11-11 04:51:55 -08:00
Carl Worth
37a8096fdc notmuch_message_tags_to_maildir_flags: Don't exit on failure to rename.
It is totally legitimate for a non-maildir directory to be named "new"
(and not have a directory next to it named "cur"). To support this
case at least, be silent about any rename failure.
2010-11-11 03:50:42 -08:00
Carl Worth
71a3201885 notmuch_message_tags_to_maildir_flags: Fix to rename multiple files
This function was documented as modifying every filename associated
with the message. Fix it to actually do that.
2010-11-11 03:47:11 -08:00
Carl Worth
8480a0a003 notmuch_filenames_create: Take a reference to underlying filename list
The notmuch_filenames_t object might easily outlive the original object
owning the filename list. So take a talloc reference to keep things safe.
2010-11-11 03:40:19 -08:00
Carl Worth
404db1de90 maildir_flags_to_tags: Avoid interpreting "no info" as "no flags set".
If a filename has no maildir info at all, (that is, it does not
contain the sequence ":2,"), we consider this distinct from a filename
with an empty maildir info, (the ":2," separator is present, but no
flags characters follow).

Specifically, we regard a missing info field as providing no
information, so tags will remain unchanged. On the other hand, an info
field that is present but has no flags set will cause various tags to
be cleared, (or in the case of "unread", added).

This fixes the "remove info" case of the maildir-sync tests in the
test suite.
2010-11-11 03:40:19 -08:00
Carl Worth
5a98880add lib: Document that absence of maildir flags can also remove/add tags
Previously the documentation of notmuch_message_maildir_flags_to_tags
suggested that the presence of a flag would cause tags to be added,
(or in the case of "unread", removed). But the case of absent maildir
flags was not explicitly described.

What we actually want, is that for supported flags, the absence of the
flag in all messages causes the corresponding tag to be removed,
(or in the case of "unread", added). So document that explicitly.

This is the case recently added to the test suite as a failing test,
(so we'll need to do bug fixing before the documentation is honest
here).
2010-11-11 03:40:19 -08:00
Carl Worth
81cbaafc0f Fix notmuch_message_tags_to_maildir_flags to effect rename immediately
We have tests to ensure that when the notmuch library renames a file
that that rename takes place immediately in the database, (without
requiring something like "notmuch new" to notice the change).

This was working when the code was first added, but recently broke in
the reworking of the maildir-synchronization interface since the
tags_to_maildir_flags function can no longer assume that it is being
called as part of _notmuch_message_sync.

Fortunately, the fix is as simple as adding an explicit call to
_notmuch_message_sync.
2010-11-11 03:40:19 -08:00
Carl Worth
4b6063397f Fix notmuch_message_maildir_flags_to_tags to iterate over filenames
As documented, this function now iterates over all filenames for the
message, computing a logical OR of the flags set on the filenames,
then uses the final result to set tags on the message.

This change fixes 3 of the 10 maildir-sync tests that have been
failing since being added.
2010-11-11 03:40:19 -08:00
Carl Worth
1d02dd64af lib: Add new, public notmuch_message_get_filenames
This augments the existing notmuch_message_get_filename by allowing
the caller access to all filenames in the case of multiple files for a
single message.

To support this, we split the iterator (notmuch_filenames_t) away from
the list storage (notmuch_filename_list_t) where previously these were
a single object (notmuch_filenames_t). Then, whenever the user asks
for a file or filename, the message object lazily creates a complete
notmuch_filename_list_t and then:

	For notmuch_message_get_filename, returns the first filename
	in the list.

	For notmuch_message_get_filenames, creates and returns a new
	iterator for the filename list.
2010-11-11 03:40:19 -08:00
Carl Worth
d87db88432 lib: Add new implementation of notmuch_filenames_t
The new implementation is simply a talloc-based list of strings. The
former support (a list of database terms with a common prefix) is
implemented by simply pre-iterating over the terms and populating the
list. This should provide no performance disadvantage as callers of
thigns like notmuch_directory_get_child_files are very likely to
always iterate over all filenames anyway.

This new implementation of notmuch_filenames_t is in preparation for
adding API to query all of the filenames for a single message.
2010-11-11 03:40:19 -08:00
Carl Worth
d422dcf0a2 lib: Remove the notion of TAGS_INVALID
This rather ugly hack was recently obviated by the removal of the
notmuch_database_set_maildir_sync function. Now, clients must make
explicit calls to do any syncrhonization between maildir flags and
tags. So the library no longer needs to worry about doing inconsistent
synchronization while a message is only partially added.
2010-11-11 03:40:19 -08:00
Carl Worth
bb74e9dff8 lib: Rework interface for maildir_flags synchronization
Instead of having an API for setting a library-wide flag for
synchronization (notmuch_database_set_maildir_sync) we instead
implement maildir synchronization with two new library functions:

	notmuch_message_maildir_flags_to_tags
  and   notmuch_message_tags_to_maildir_flags

These functions are nicely documented here, (though the implementation
does not quite match the documentation yet---as plainly evidenced by
the current results of the test suite).
2010-11-11 03:40:19 -08:00
Carl Worth
2c262042ac lib: Remove the synchronization of 'T' flag with "deleted" tag.
Tags in a notmuch database affect all messages with the identical
message-ID. But maildir tags affect individual files. And since
multiple files can contain the identical message-ID, there is not a
one-to-one correspondence between messages affected by tags and flags.

This is particularly dangerous with the 'T' (== "trashed") maildir
flag and the corresponding "deleted" tag in the notmuch
database. Since these flags/tags are often used to trigger
irreversible deletion operations, the lack of one-to-one
correspondence can be potentially dangerous.

For example, consider the following sequence:

  1. A third-party application is used to identify duplicate messages
     in the mail store, and mark all-but-one of each duplicate with
     the 'T' flag for subsequent deletion.

  2. A "notmuch new" operation reads that 'T' flag, adding the
     "deleted" flag to the corresponding messages within the notmuch
     database.

  3. A subsequent notmuch operation, (such as a "notmuch dump; notmuch
     restore" cycle) synchronized the "deleted" tag back to the mail
     store, applying the 'T' flag to all(!) filenames with duplicate
     message IDs.

  4. A third-party application reads the 'T' flags and irreversibly
     deletes all mail messages which had any duplicates(!).

In order to avoid this scenario, we simply refuse to synchronize the
'T' flag with the "deleted" tag. Instead, applications can set 'T' and
act on it to delete files, or can set "deleted" and act on it to
delete files. But in either case the semantics are clear and there is
never dangerous propagation through the one-to-many mapping of notmuch
message objects to files.
2010-11-11 02:35:03 -08:00
Michal Sojka
d9d3d3e6f0 Make maildir synchronization configurable
This adds group [maildir] and key 'synchronize_flags' to the
configuration file. Its value enables (true) or diables (false) the
synchronization between notmuch tags and maildir flags. By default,
the synchronization is disabled.
2010-11-10 13:09:32 -08:00
Michal Sojka
088801a14a Maildir synchronization
This patch allows bi-directional synchronization between maildir
flags and certain tags. The flag-to-tag mapping is defined by flag2tag
array.

The synchronization works this way:

1) Whenever notmuch new is executed, the following happens:
   o New messages are tagged with configured new_tags.
   o For new or renamed messages with maildir info present in the file
     name, the tags defined in flag2tag are either added or removed
     depending on the flags from the file name.

2) Whenever notmuch tag (or notmuch restore) is executed, a new set of
   flags based on the tags is constructed for every message and a new
   file name is prepared based on the old file name but with the new
   flags. If the flags differs and the old message was in 'new'
   directory then this is replaced with 'cur' in the new file name. If
   the new and old file names differ, the file is renamed and notmuch
   database is updated accordingly.

   The rename happens before the database is updated. In case of crash
   between rename and database update, the next run of notmuch new
   brings the database in sync with the mail store again.
2010-11-10 13:09:31 -08:00
Michal Sojka
b65a783195 lib: Return added message even if it already was in the database 2010-11-10 13:09:31 -08:00
Carl Worth
d064bd696c lib: Eliminate some redundant includes of xapian.h
Most files including this already include database-private.h which
includes xapian.h already.
2010-11-01 23:24:40 -07:00
Carl Worth
c81cecf620 lib: Add GCC visibility(hidden) pragmas to private header files.
This prevents any of the private functions from being leaked out
through the library interface (at least when compiling with a
recent-enough gcc to support the visibility pragma).
2010-11-01 22:35:48 -07:00
Carl Worth
67c3bc9db4 lib: Add some missing static qualifiers.
These various functions and data are all used only locally, so should
be marked static. Ensuring we get these right will avoid us accidentally
leaking unintended symbols through the library interface.
2010-11-01 21:58:43 -07:00
Carl Worth
4dd066ae6c Update library version to 1.2.0
This increment is for the recently-added functions:

	notmuch_query_get_query_string
	notmuch_query_get_sort

These were recently added to the library interface, but the library
version was not incremented at that time, (shame on me).
2010-11-01 16:13:44 -07:00
Michal Sojka
f7a688ec53 Do not call ldconfig when building Debian package
Hi,

If I want to build Debian package, it fails with the following message:

    ldconfig: Can't create temporary cache file /etc/ld.so.cache~: Permission denied
    make[1]: *** [install-lib] Error 1

The reason is that I build the package as a non-root user and make
install invokes ldconfig unconditionally. The following patch contains a
workaround, but I think that a more correct solution would be to check
the condition LIBDIR_IN_LDCONFIG directly when make install is invoked
rather than in configure as it is done now.

Signed-off-by: Michal Sojka <sojkam1@fel.cvut.cz>
2010-10-28 13:06:46 -07:00
Carl Worth
e83b40138e lib: Add two functions: notmuch_query_get_query_string and _get_sort
It can be handy to be able to query these settings from an existing
query object.
2010-10-28 10:30:26 -07:00
Carl Worth
f6cb896bc4 lib: Fix notmuch_query_search_threads to return NULL on any Xapian exception.
Previously, if the underlying search_messages hit an exception and returned
NULL, this function would ignore that and return a non-NULL, (but empty)
threads object. Fix this to properly propagate the error.
2010-10-22 17:56:58 -07:00
Carl Worth
8071c5cd64 lib: Fix "make install"
This has been broken since the addition of the test sub-directory to our
non-recursive make system.
2010-09-21 09:09:01 -07:00
Carl Worth
029830c1f1 lib: Fix use-after-free bug.
Thanks to the new git-based test suite, it's easy to run the whole
test suite in valgrind, (simply "make test OPTIONS="--valgrind"), and
doing so showed this obvious use-after-free bug, (triggered by the
thread-order tests).
2010-09-20 22:01:52 +00:00
Carl Worth
d64d0cc8d9 make install: Run ldconfig or install a DT_RUNPATH in binary as appropriate.
Various users were confused as to why they couldn't run notmuch
immediately after "make install", (with linker errors saying that
libnotmuch.so could not be found). The errors came from two different
causes:

1. The user had installed to a system library directory, but had not
   yet run ldconfig.

2. The user had installed to some non-system directory, and had not
   set the LD_LIBRARY_PATH variable.

With this change we fix both problems (on Linux) without the user
having to do anything additional. We first use ldconfig to find the
system library directories. If the user is installing to one of these,
then we run ldconfig as part of "make install".

For case (2) we use the -rpath and --enable-new-dtags linker options
to install a DT_RUNPATH entry in the binary. This entry tells the
dynamic linker where to find libnotmuch. Without the
--enable-new-dtags option only a DT_RPATH option would be installed,
(which has the drawback of not allowing any override with the
LD_LIBRARY_PATH variable).

Distributions (such as Debian and Fedora) don't want to see binaries
packaged with a DT_RPATH or DT_RUNPATH entry. This should be avoided
automatically as long as the packages install to standard locations,
(such as /usr/lib).
2010-06-04 16:52:56 -07:00
Carl Worth
7b78eb4af6 Add support (and tests) for messages with really long message IDs.
Scott Henson reported an internal error that occurred when he tried to
add a message that referenced another message with a message ID well
over 300 characters in length. The bug here was running into a Xapian
limit for the length of metadata key names, (which is even more
restrictive than the Xapian limit for the length of terms).

We fix this by noticing long message ID values and instead using a
message ID of the form "notmuch-sha1-<sha1_sum_of_message_id>". That
is, we use SHA1 to generate a compressed, (but still unique), version
of the message ID.

We add support to the test suite to exercise this fix. The tests add a
message referencing the long message ID, then add the message with the
long message ID, then finally add another message referencing the long
ID. Each of these tests exercise different code paths where the
special handling is implemented.

A final test ensures that all three messages are stitched together
into a single thread---guaranteeing that the three code paths all act
consistently.
2010-06-04 13:35:07 -07:00
Carl Worth
98845fdbb2 Avoid database corruption by not adding partially-constructed mail documents.
Previously we were using Xapian's add_document to allocate document ID
values for notmuch_message_t objects.  This had the drawback of adding
a partially constructed mail document to the database. If notmuch was
subsequently interrupted before fully populating this document, then
later runs would be quite confused when seeing the partial documents.

There are reports from the wild of people hitting internal errors of
the form "Message ... has no thread ID" for example, (which is
currently an unrecoverable error).

We fix this by manually allocating document IDs without adding
documents. With this change, we never call Xapian's add_document
method, but only replace_document with either the current document ID
of a message or a new one that we have allocated.
2010-06-04 10:16:53 -07:00
Carl Worth
361b9d4bd9 Fix misnamed function in internal documentation.
The documentation for several functions mentioned
_notmuch_message_set_sync which doesn't exist. Fix these to reference
_notmuch_message_sync instead.
2010-06-04 09:54:46 -07:00
Tomas Carnecky
a54cecfc8e Add support for the Solaris platform
Like on Mac OS X, the linker doesn't automatically resolve dependencies.

Signed-off-by: Tomas Carnecky <tom@dbservice.com>
2010-06-03 18:17:03 -07:00
Dirk Hohndel
a258cb32b3 Fix SEGV in _thread_cleanup_author if author ends with ', '
Admittedly, an author name ending in ',' guarantees this is spam, and
indeed this was triggered by a spam email, but that doesn't mean we
shouldn't handle this case correctly.
We now check that there is actually a component of the name (presumably
the first name) after the comma in the author name.

Signed-off-by: Dirk Hohndel <hohndel@infradead.org>
2010-04-27 16:34:27 -07:00
Carl Worth
0fb28c65f2 lib: Increment library version to 1.1.0
For the addition of the new NOTMUCH_SORT_UNSORTED value.
2010-04-27 02:02:14 -07:00
Carl Worth
c210d5632e lib: Re-implement moving of thread authors.
Just before releasing 0.3 we received reports of crashes that were
bisected to the commit adding thread-author moving. Sure enough,
valgrind pointed to buffer overruns in _thread_move_matched_author.

Rather than trying to make sense of all the by strncpy, strchr, +1,
and +2 of that code, I reimplemented thread-author ordering with a
pair of hash tables and an array.

Valgrind is at least happy now on the test cases it was complaining
about previously.
2010-04-27 01:48:03 -07:00
Dirk Hohndel
5b8b0377cb Make Received: header special in notmuch_message_file_get_header
With this patch the Received: header becomes special in the way
we treat headers - this is the only header for which we concatenate
all the instances we find (instead of just returning the first one).

This will be used in the From guessing code for replies as we need to
be able to walk ALL of the Received: headers in a message to have a
good chance to guess which mailbox this email was delivered to.

Signed-off-by: Dirk Hohndel <hohndel@infradead.org>
2010-04-26 14:44:06 -07:00
Dirk Hohndel
cd19671f51 Simple attempt to display author names in a friendlier way
This patch only addresses the typical Outlook/Exchange case
where we have "Last, First" <first.last@company.com> or
"Last, First MI" <first.mi.last@company.com>.

In the future we should be more fexible as to the formats
we recognize, but for now we address this one as it is the
Exchange default setting and therefore the most common one.

Signed-off-by: Dirk Hohndel <hohndel@infradead.org>
2010-04-26 11:45:29 -07:00
Dirk Hohndel
26d8d960ee Reorder displayed names of thread authors
When displaying threads as result of a search it makes sense to list those
authors first who match the search. The matching authors are separated from the
non-matching ones with a '|' instead of a ','

Imagine the default "+inbox" query. Those mails in the thread that
match the query are actually "new" (whatever that means). And some
people seem to think that it would be much better to see those author
names first. For example, imagine a long and drawn out thread that once
was started by me; you have long read the older part of the thread and
removed the inbox tag. Whenever a new email comes in on this thread,
prior to this patch the author column in the search display will first show
"Dirk Hohndel" - I think it should first show the actual author(s) of the new
mail(s).

Signed-off-by: Dirk Hohndel <hohndel@infradead.org>
2010-04-26 11:45:00 -07:00
Dirk Hohndel
57561414d7 Add authors member to message
message->authors contains the author's name (as we want to print it)
get / set methods are declared in notmuch-private.h

Signed-off-by: Dirk Hohndel <hohndel@infradead.org>
2010-04-26 11:44:49 -07:00
Carl Worth
138fd38afe lib: Ensure notmuch_query_search_messages returns NULL on an exception.
Previously, this function may have segfaulted immediately after
reporting the exception.
2010-04-24 07:27:50 -07:00
Carl Worth
e3e0e26806 lib: Document that notmuch_query_count_messages may return 0 if an exception occurs
This isn't a behavioral change---just a calrification in the documentation.
2010-04-24 07:27:50 -07:00
Carl Worth
9ef68f1444 lib: Audit all notmuch_database call for Xapian exception handling.
Our current approach is for top-level entry poitns in the library
to have try/catch blocks that catch any Xapian exception and print
a message. Add a few missing blocks and fix up the documentation.
2010-04-24 07:27:50 -07:00
Carl Worth
3dbef312fb lib: Audit calls to notmuch_message_get_header to handle NULL return
Sebastian Spaeth reported [*] a segfault within libnotmuch when
running notmuch operations while an asyncronous offlineimap job had
removed some files from the mail store. Avoid this by handling all
cases where notmuch_message_get_header could return NULL.

[*] See message id:87d3xqti3o.fsf@SSpaeth.de on notmuch@notmuchmail.org
2010-04-24 06:50:04 -07:00
Carl Worth
7c421b87b0 lib: Simplify code to set subject from matched message.
Simply moving the code from _add_matched_message to a new
_set_subject_from_message function.
2010-04-24 06:50:04 -07:00
Carl Worth
21965718a5 Revert "thread: Simplify code for assigning the subject."
This reverts commit 36e4459a32.

With the two previous reverts, this fixes the recent message-sorting
regression, so the test suite now passes again.
2010-04-22 14:01:41 -07:00
Carl Worth
a109966080 Revert "thread: Fix sort of search when constructing threads."
This reverts commit f43990ce13.
2010-04-22 14:00:33 -07:00
Carl Worth
6a0cba4ae0 Revert "thread: Removed unsed sort argument from _thread_add_matched_message"
This reverts commit 7fb56f9dc5.
2010-04-22 14:00:17 -07:00
Carl Worth
7fb56f9dc5 thread: Removed unsed sort argument from _thread_add_matched_message
The reworked solution for naming a thread based on the subject of
oldest/newest matching message no longer needs this argument.
2010-04-21 17:05:16 -07:00
Sebastian Spaeth
aadb15a002 query.cc: allow to return query results unsorted
Previously, we always sorted the returned results by some string value,
(newest-to-oldest by default), however in some cases (as when applying
tags to a search result) we are not interested in any special order.

This introduces a NOTMUCH_SORT_UNSORTED value that does just that. It is
not used at the moment anywhere in the code.

Signed-off-by: Sebastian Spaeth <Sebastian@SSpaeth.de>
2010-04-21 16:06:05 -07:00
Carl Worth
f43990ce13 thread: Fix sort of search when constructing threads.
The thread-naming feature depends on the matched messages being passed
down in a precise order, (the order of the top-level search). We fix
the feature by passing that sort order down.
2010-04-21 15:52:28 -07:00
Carl Worth
36e4459a32 thread: Simplify code for assigning the subject.
We know that matched messages are always added in order, so we can
always just grab the subject from the first message. This is the same
approach that was used previously in _thread_add_message. That is, the
recent feature of renaming a thread based on the subject of the
"first" matched message is as simple as moving the subject assignment
from _thread_add_message to _thread_add_matched_message.
2010-04-21 15:06:02 -07:00
Jesse Rosenthal
4971b85641 Name thread based on matching msgs instead of first msg.
At the moment all threads are named based on the name of the first message
in the thread. However, this can cause problems if people either start
new threads by replying-all (as unfortunately, many out there do) or
change the subject of their mails to reflect a shift in a thread on a
list.

This patch names threads based on (a) matches for the query, and (b) the
search order. If the search order is oldest-first (as in the default
inbox) it chooses the oldest matching message as the subject. If the
search order is newest-first it chooses the newest one.

Reply prefixes ("Re: ", "Aw: ", "Sv: ", "Vs: ") are ignored
(case-insensitively) so a Re: won't change the subject.

Note that this adds a "sort" argument to _notmuch_thread_create and
_thread_add_matched_message, so that when constructing the thread we can
be aware of the sort order.

Signed-off-by: Jesse Rosenthal <jrosenthal@jhu.edu>
2010-04-21 14:56:53 -07:00
Carl Worth
c48dcc302c lib: search_threads: Fix nested search to handle original search of "*"
When constructing a thread, we usually run a nested query to find all
messages in the thread that match the original search string. However,
we need to have special-case handling of an original search string of
"*" now that that is a supported means of specifying all messages.

The special-case ends up bein quite simple---we do less work, (just
skipping the nested search since we know that all messages must
match). I had been wanting to write this identical code to more
efficiently handle "notmuch search thread:<foo>" which was previously
running two identical searches. So that case is now more efficient as
well.
2010-04-15 14:54:40 -07:00
Carl Worth
72ea1b71c6 Makefile: Add library version information on OS X.
This encodes the library version into the library, where the linking
binary can pick it up, and the linker can even enforce mismatches in
the minor release, (such as linking a binary against version 1.2 and
then attempting to run it against version 1.1).
2010-04-14 16:18:19 -07:00
Carl Worth
1036867897 Makefile: Fix library linking command for OS X
I'm not sure which system Aaron used, but on the machine I have access
to, (Darwin 8.11.0), the -shared and -dylib_install_name options are
not recognized. Instead I use -dynamic_lib and -install_name as
documented here:

http://www.finkproject.org/doc/porting/shared.php
2010-04-14 16:16:05 -07:00
Aaron Ecay
8c8079a8b1 Add infrastructure for building shared library on OS X.
This patch adds a configure check for OS X (actually Darwin),
and sets up the Makefiles to build a proper shared library on
that platform.

Signed-off-by: Aaron Ecay <aaronecay@gmail.com>
2010-04-14 16:10:27 -07:00
Carl Worth
f206408358 Makefile: Move compat sources from the client code to the library.
Since the library code needs these as well.
2010-04-14 16:03:18 -07:00
martin f. krafft
449a418c65 Do not segfault on empty mime parts
notmuch previously unconditionally checked mime parts for various
properties, but not for NULL, which is the case if libgmime encounters
an empty mime part.

Upon encounter of an empty mime part, the following is printed to
stderr (the second line due to my patch):

  (process:17197): gmime-CRITICAL **: g_mime_message_get_mime_part: assertion `GMIME_IS_MESSAGE (message)' failed
  Warning: Not indexing empty mime part.

This is probably a bug that should get addressed in libgmime, but for
not, my patch is an acceptable workaround.

Signed-off-by: martin f. krafft <madduck@madduck.net>
2010-04-13 08:49:06 -07:00
Michael Forney
9ddde6eb14 Fix typo in notmuch.h documentation regarding database open modes
Reviewed-by: Carl Worth <cworth@cworth.org>:

The original proposal for having different open modes used the name
WRITABLE. I didn't like that name, (easy to misspell as WRITEABLE even
for native English speakers). So we renamed it to READ_WRITE
immediately, but apparently some of the documentation held the old
name for a while.
2010-04-13 08:39:10 -07:00
Carl Worth
14073b8851 lib: Remove condition regarding a NULL parent_thread_id.
A recent change guaranteed that a message ID can never be resolved
to a NULL thread ID, so we don't need this extra case.
2010-04-12 15:54:03 -07:00
Carl Worth
071022c253 lib: Always add reference terms to the database.
Previously, we were only adding the reference terms for cases where
the referenced message did not yet exist in the database. For thread
presentation, it's useful to have the connection information provided
by the references, even when the messages are present. So add this
term unconditionally.
2010-04-12 15:45:40 -07:00
Carl Worth
328626d0fd lib: Document the metadata stored within the Xapian database.
We are currently storing "version", "last_thread_id", and
"thread_id_*" values so document how each of these are used.
2010-04-12 15:15:14 -07:00
Carl Worth
af49741228 lib: Fix line-wrapping in _notmuch_database_link_message.
This function had some excessively long lines due to nested
expressions. It's simple enough to un-nest these and have readable
line lengths.
2010-04-12 14:41:34 -07:00
Carl Worth
f8dc5c08e4 lib: Fix internal documentation of _notmuch_database_link_message
This function was recently modified, (to include a metadata lookup for
a message's thread ID before looking for parent/child thread IDs), but
the documentation wasn't updated. Fix that.
2010-04-12 14:35:25 -07:00
Carl Worth
5c20bdf035 lib: Simplify code flow in _resolve_message_id_to_thread_id
There are two primary cases in this function, (the message exists in
the database or it does not). Previously the code for these two cases
was split and intermingled with goto-spaghetti connections.
2010-04-12 14:29:36 -07:00
Carl Worth
e9bb90ba2c lib: Fix internal documentation of _resolve_message_id_to_thread_id
We no longer return NULL, but instead generate a new thread ID for
messages that we haven't seen yet.
2010-04-12 14:19:15 -07:00
James Westby
40ea73cf05 Store thread ids for messages that we haven't seen yet
This allows us to thread messages even when we receive them out of
order, or never receive the root.

The thread ids for messages that aren't present but are referred to are
stored as metadata in the database and then retrieved if we ever get
that message.

When determining the thread id for a message we also check for this
metadata so that we can thread descendants of a message together before
we receive it.

Edited by Carl Worth <cworth@cworth.org>: Split this portion of the
commit from the earlier-applied portion adding test cases.
2010-04-12 14:11:57 -07:00
Carl Worth
e100871981 lib: Handle "*" as a query string to match all messages.
This seems like a generally useful thing to support, (but the previous
support through an empty string was not convenient for some users,
(such as the command-line client).
2010-04-09 17:43:58 -07:00
Dirk Hohndel
4563f669ca fix obvious cut and paste error
the wrong variable is checked for success of an allocation

Signed-off-by: Dirk Hohndel <hohndel@infradead.org>
2010-04-06 18:55:56 -07:00
Dirk Hohndel
a48f368778 fix notmuch_message_file_get_header
fix notmuch_message_file_get_header to always return the first instance
 of the header you are looking for

Signed-off-by: Dirk Hohndel <hohndel@infradead.org>
2010-04-06 18:47:28 -07:00
Carl Worth
ae9d67fd81 Avoid needlessly linking final notmuch binary against libXapian.
The libnotmuch.so library already does, so we don't need to do
it again. (Thanks to a Debian debhelper warning for pointing this
out.)
2010-04-06 18:30:43 -07:00
Carl Worth
14e98e454e configure: Add support for a --includedir option
Very similar to the existing --libdir option.
2010-04-06 14:42:09 -07:00
Carl Worth
f89b3d16db Makefiles: Eliminate the useless quiet_* functions.
With the original quiet function, there's an actual purpose (hiding
excessively long compiler command lines so that warnings and errors
from the compiler can be seen).

But with things like quiet_symlink there's nothing quieter. In fact
"SYMLINK" is longer than "ln -sf". So all this is doing is hiding the
actual command from the user for no real benefit.

The only actual reason we implemented the quiet_* functions was to be
able to neatly right-align the command name and left-align the arguments.

Let's give up on that, and just left-align everything, simplifying the
Makefiles considerably. Now, the only instances of a captialized command
name in the output is if there's some actually shortening of the command
itself.
2010-04-06 14:36:31 -07:00
Carl Worth
da2403c310 RELEASING: Add this file describing the steps to make a release.
These steps might be changing a bit as we work on making the initial
0.1 release.
2010-04-05 15:29:54 -07:00
David Edmondson
d3884a5984 Makefile.local: Automatically use makefile mode
We add a magic line to the beginning of each Makefile.local file to
help the editor know that it should use makefile mode for editing the
file, (even though the filename isn't exactly "Makefile").

Edited-by: Carl Worth <cworth@cworth.org>: Expand treatment from
emacs/Makefile.local to each instance of Makefile.local.
2010-04-03 12:31:49 -07:00
Carl Worth
f689c83af4 Compile a static notmuch binary (but only install the shared version)
The idea here is to allow a new user of notmuch to be able to run
notmuch immediately after compiling, (without having to install
the shared library first). This also ensures that the test suite
tests the locally compiled library, and not whatever installled
version of the library the dynamic linker happens to find.
2010-04-01 15:03:40 -07:00
Michal Sojka
b884ab2ef1 Makefile: Create include directory when installing headers
When I wanted to create a debian package from the current master, make
install failed because of non-existent include directory. This patch
fixes this minor issue.
2010-04-01 05:13:21 -07:00
Carl Worth
c0961e6a82 lib: Switch to a 3-part version number for the library interface.
With a carefully documented description of how to increment the
various version components.
2010-04-01 00:41:25 -07:00
Carl Worth
33d5cc415e Makefiles: Make the install rules quiet like the compilation rules.
The output from make is looking better all the time, (though the
columns still aren't lined up).
2010-03-31 23:54:21 -07:00
Carl Worth
7b52b2c318 Move installation of library from top-level to lib/Makefile.local
We had a fairly ugly violation of modularity with the top-level
Makefile.local isntalling everything, (even when the build commands
for the library were down in lib/Makefile.local).
2010-03-31 22:54:15 -07:00
Saleem Abdulrasool
07378d0d14 Fix target dependencies for multiple jobs
Signed-off-by: Ingmar Vanhassel <ingmar@exherbo.org>
2010-03-31 17:41:28 -07:00
Ben Gamari
266ab595a2 Build and link against notmuch shared library, install notmuch.h
Signed-off-by: Ingmar Vanhassel <ingmar@exherbo.org>
2010-03-31 17:38:27 -07:00
Carl Worth
b957a1b029 emacs: Fix the notmuch-search-authors-width variable.
This variable existed previously, but wasn't actually used for anything.
2010-03-31 13:32:00 -07:00
Carl Worth
e002fe8a7a Clarify documentation of notmuch_database_add_message.
For the case of adding a file that already exist, (with the same
filename). In this case, nothing will happen to the database, but
that wasn't clear before.
2010-03-31 13:31:10 -07:00
Carl Worth
86232e62ab Makefile: Fix Makefiles to depend on all child Makefile fragments.
We were previously maintaining two lists of the child Makefile
fragments---one for the includes and another for the dependencies. So,
of course, they drifted and the dependency list wasn't up to date.

We fix this by adding a single subdirs variable, and then using GNU
Makefile substitution to generate both the include and the dependency
lists.

Some side effect of this change caused the '=' assignment of the dir
variable to not work anymore. I'm not sure why that is, but using ':='
makes sense here and fixes the problem.
2010-03-10 10:59:57 -08:00
Carl Worth
e3046c688b Add is:<tag> as a synonym for tag:<tag> in search terms.
I like the readability of this, it provides compatibility with people
trained in this syntax by sup, and it even saves one character.
2010-03-09 16:03:58 -08:00
Carl Worth
c446f22dee lib: Silence a compiler warning.
The original code was harmless, but apparently some compilers aren't
able to think deep enough to catch that.
2010-03-09 12:07:26 -08:00
Fernando Carrijo
7f2629520c Fix a few documentation typos in notmuch.h
Fix a few documentation typos in notmuch.h

Signed-off-by: Fernando Carrijo <fcarrijo@yahoo.com.br>
2010-03-09 10:32:58 -08:00
Fernando Carrijo
bc69bf09cb Update documentation of notmuch_query_create
Commit cd467caf renamed notmuch_query_search to notmuch_query_search_messages.
Commit 1ba3d46f created notmuch_query_search_threads. We better keep the docs
of notmuch_query_create consistent with those changes.

Signed-off-by: Fernando Carrijo <fcarrijo@yahoo.com.br>

Edited-by: Carl Worth to explicitly list the full name of each
function being referenced.
2010-03-09 10:29:38 -08:00
Carl Worth
64646841f7 lib: Document what move_to_next does at the end of the list.
Explicitly mention that there's an invalid position after the last
item in the list.
2010-03-09 09:24:45 -08:00
Carl Worth
4e5d2f22db lib: Rename iterator functions to prepare for reverse iteration.
We rename 'has_more' to 'valid' so that it can function whether
iterating in a forward or reverse direction. We also rename
'advance' to 'move_to_next' to setup parallel naming with
the proposed functions 'move_to_first', 'move_to_last', and
'move_to_previous'.
2010-03-09 09:22:29 -08:00
Carl Worth
e0a8dee8bc Fix printf for when uint64_t != unsigned long long int
Thanks to Michal Sojka <sojkam1@fel.cvut.cz> for pointing out the
correct fix, which I verified in the freely-available WG14/N1124 draft
(from the C99 working group) which is available here:

http://www.open-std.org/JTC1/SC22/wg14/www/docs/n1124.pdf
2010-02-09 11:14:16 -08:00
Carl Worth
9439b217c3 Switch from random to sequential thread identifiers.
The sequential identifiers have the advantage of being guaranteed to
be unique (until we overflow a 64-bit unsigned integer), and also take
up half as much space in the "notmuch search" output (16 columns
rather than 32).

This change also has the side effect of fixing a bug where notmuch
could block on /dev/random at startup (waiting for some entropy to
appear). This bug was hit hard by the test suite, (which could easily
exhaust the available entropy on common systems---resulting in large
delays of the test suite).
2010-02-09 11:14:11 -08:00
Carl Worth
7a9bacac67 notmuch.h: Fix a couple of typos in the documentation.
Obviously, the spell-checker isn't able to catch every mistake
I make.
2010-02-05 17:31:40 -08:00
Carl Worth
2bc0af15aa Eliminate some useless gobject boilerplate.
If we had external users of this filter then they might expect some of
these macros to exist. But since this is just internal, that's just
unneeded noise.
2010-02-04 17:26:00 -08:00
Carl Worth
3767c6f9f9 notmuch new: Don't index uuencoded data.
With modern MIME attachments, we're already avoiding indexing the
attachments. But for old-school uuencoded data in the mail, we have
been directly indexing the encoded data as terms, (which is not useful
at all---nobody will ever ytry to search based on the seemingly random
uuencoded data).

Additionally, indexing a modestly large uuencoded file seems to make
Xapian go insane, (consuming *lots* of memory).

We fix both problems by detecting uuencoded content and not performing
any indexing of it.
2010-02-04 17:08:11 -08:00
Carl Worth
c340c1bd11 notmuch new: Print upgrade progress report as a percentage.
Previously we were printing a number of messages upgraded so far. The
original motivation for this was to accurately reflect the fact that
there are two passes, (so each message is processed twice and it's not
accurate to represent with a single count). But as it turns out, the
second pass takes zero time (relatively speaking) so we're still not
accounting for it.

If nothing else, the percentage-based reporting makes for a cleaner
API for the progress_notify function.
2010-01-09 17:38:23 -08:00
Carl Worth
ccf2e0cc42 lib: Add non-content terms with a WDF value of 0.
The WDF is the "within-document frequency" value for a particular
term. It's intended to provide an indication of how frequent a term is
within a document, (for use in computing relevance). Xapian's term
generator already computes WDF values when we use that, (which we do
for indexing all mail content).

We don't use the term generator when adding single terms for things
that don't actually appear in the mail document, (such as tags, the
filename, etc.). In this case, the WDF value for these terms doesn't
matter much.

But Xapian's flint backend can be more efficient with changes to terms
that don't affect the document "length". So there's a performance
advantage for manipulating tags (with the flint backend) if the WDF of
these terms is 0.
2010-01-09 11:18:27 -08:00
Carl Worth
45b1856782 lib: Explicitly set BoolWeight when searching.
All notmuch searches currently sort by value (either date or message
ID) so it's just wasted effort for Xapian to compute relevance values
for each result. We now explicitly tell Xapian that we're uninterested
in the relevance values.
2010-01-09 11:16:40 -08:00
Carl Worth
d12801c8b4 lib: Split the database upgrade into two phases for safer operation.
The first phase copies data from the old format to the new format
without deleting anything. This allows an old notmuch to still use the
database if the upgrade process gets interrupted. The second phase
performs the deletion (after updating the database version number). If
the second phase is interrupted, there will be some unused data in the
database, but it shouldn't cause any actual harm.
2010-01-09 11:13:12 -08:00
Carl Worth
5fe5e802ab lib: Delete stale timestamp documents during database upgrade.
Once we move the timestamp to the new directory document, we don't
need the old one anymore.
2010-01-08 09:57:09 -08:00
Carl Worth
1c86b48329 notmuch new: Fix progress notification on database upgrade.
This was firing continuously rather than just once per second as
intended.
2010-01-07 21:24:44 -08:00
Carl Worth
909f52bd8c lib: Implement versioning in the database and provide upgrade function.
The recent support for renames in the database is our first time
(since notmuch has had more than a single user) that we have a
database format change. To support smooth upgrades we now encode a
database format version number in the Xapian metadata.

Going forward notmuch will emit a warning if used to read from a
database with a newer version than it natively supports, and will
refuse to write to a database with a newer version.

The library also provides functions to query the database format
version:

	notmuch_database_get_version

to ask if notmuch wants a newer version than that:

	notmuch_database_needs_upgrade

and a function to actually perform that upgrade:

	notmuch_database_upgrade
2010-01-07 18:26:31 -08:00
Carl Worth
807aef93d3 Prefer READ_ONLY consistently over READONLY.
Previously we had NOTMUCH_DATABASE_MODE_READ_ONLY but
NOTMUCH_STATUS_READONLY_DATABASE which was ugly and confusing. Rename
the latter to NOTMUCH_STATUS_READ_ONLY_DATABASE for consistency.
2010-01-07 10:29:05 -08:00
Carl Worth
f93b7218c3 lib: Consolidate checks for read-only database.
Previously, many checks were deep in the library just before a cast
operation. These have now been replaced with internal errors and new
checks have instead been added at the beginning of all top-levelentry
points requiring a read-write database.

The new checks now also use a single function for checking and
printing the error message. This will give us a convenient location to
extend the check, (such as based on database version as well).
2010-01-07 10:19:44 -08:00
Carl Worth
6ed606c19e lib: Clarify internal documentation of _notmuch_database_filename_to_direntry
The original wording made it sound like this function was just doing
some string manipulation. But this function actually creates new
directory documents as a side effect. So make that explicit in its
documentation.
2010-01-07 09:31:58 -08:00
Carl Worth
a274848f95 notmuch_message_get_filename: Support old-style filename storage.
When a notmuch database is upgraded to the new database format, (to
support file rename and deletion), any message documents corresponding
to deleted files will not currently be upgraded. This means that a
search matching these documents will find no filenames in the expected
place.

Go ahead and return the filename as originally stored, (rather than
aborting with an internal error), in this case.
2010-01-07 09:22:34 -08:00
Carl Worth
957ae198e7 lib: Treat NULL as a valid (and empty) notmuch_filenames_t iterator.
This will be convenient to avoid some special-casing in higher-level
code.
2010-01-06 14:35:11 -08:00
Carl Worth
4b418343f6 lib: Indicate whether notmuch_database_remove_message removed anything.
Similar to the return value of notmuch_database_add_message, we now
enhance the return value of notmuch_database_remove_message to
indicate whether the message document was entirely removed (SUCCESS)
or whether only this filename was removed and the document exists
under other filenamed (DUPLICATE_MESSAGE_ID).
2010-01-06 10:32:06 -08:00
Carl Worth
777cd23d9d lib: Update documentation of notmuch_database_add_message.
Previously, adding a filename with the same message ID as an existing
message would do nothing. But we recently fixed this to instead add
the new filename to the existing message document. So update the
documentation to match now.
2010-01-06 10:32:06 -08:00
Carl Worth
6ef6ddba80 Index content from citations and signatures.
In the presentation we often omit citations and signatures, but this
is not content that should be omitted from the index, (especially
when the citation detection is wrong---see cases where a line
beginning with "From" is corrupted to ">From" by mail processing
tools).
2010-01-06 10:32:06 -08:00
Carl Worth
341d49b061 Makefiles: Use .DEFAULT to support arbitrary targets from sub directories.
Taking advantage of the .DEFAULT construct means that we won't need to
explicitly list targets such as "clean", etc. in each sub-Makefile.
2010-01-06 10:32:06 -08:00
Carl Worth
3f32fd8a1c Add missing comment for NOTMUCH_STATUS_READONLY_DATABASE.
And adjust the string representation of the same to match.
2010-01-06 10:32:06 -08:00
Carl Worth
d807e28f43 lib: Implement new notmuch_directory_t API.
This new directory ojbect provides all the infrastructure needed to
detect when files or directories are deleted or renamed. There's still
code needed on top of this (within "notmuch new") to actually do that
detection.
2010-01-06 10:32:06 -08:00
Carl Worth
ba07fe1819 Revamp the proposed directory-tracking API slightly.
This commit contains my changes to the API proposed by Keith. Nothing
is dramatically different. There are minor things like changing
notmuch_files_t to notmuch_filenames_t and then various things needed
for completeness as noticed while implementing this, (such as
notmuch_directory_destroy and notmuch_directory_set_mtime).
2010-01-06 10:32:06 -08:00
Keith Packard
95deec1b27 Prototypes for directory tracking
There's no functionality here yet---just a sketch of what the
interface could look like.
2010-01-06 10:32:06 -08:00
Carl Worth
f11aaa3678 database: Add new, public notmuch_database_remove_message
This will allow applications to support the removal of messages, (such
as when a file is deleted from the mail store). No removal support is
provided yet in commands such as "notmuch new".
2010-01-06 10:32:06 -08:00
Carl Worth
44a74912c7 database: Add new find_doc_ids_for_term interface.
The existing find_doc_ids function is convenient when the caller
doesn't want to be bothered constructing a term. But when the caller
*does* have the term already, that interface is just wasteful. So we
export a lower-level interface that maps a pre-constructed term to a
document-ID iterators.
2010-01-06 10:32:06 -08:00
Carl Worth
d7e5f5827e database: Make find_unique_doc_id enforce uniqueness (for a debug build)
Catching any violation of this unique-ness constraint is very much in
line with similar, existing INTERNAL_ERROR cases.
2010-01-06 10:32:06 -08:00
Carl Worth
498edff503 database: Abstract _filename_to_direntry from _add_message
The code to map a filename to a direntry is something that we're going
to want in a future _remove_message function, so put it in a new
function _notmuch_database_filename_to_direntry .
2010-01-06 10:32:05 -08:00
Carl Worth
1376a90db6 database: Allowing storing multiple filenames for a single message ID.
The library interface is unchanged so far, (still just
notmuch_database_add_message), but internally, the old
_set_filename function is now _add_filename instead.
2010-01-06 10:32:05 -08:00
Carl Worth
6ca6c089e9 database: Store mail filename as a new 'direntry' term, not as 'data'.
Instead of storing the complete message filename in the data portion
of a mail document we now store a 'direntry' term that contains the
document ID of a directory document and also the basename of the
message filename within that directory. This will allow us to easily
store multple filenames for a single message, and will also allow us
to find mail documents for files that previously existed in a
directory but that have since been deleted.
2010-01-06 10:32:05 -08:00
Carl Worth
84742d86ab database: Split _find_parent_id into _split_path and _find_directory_id
Some pending commits want the _split_path functionality separate from
mapping a directory to a document ID. The split_path function now
returns the basename as well as the directory name.
2010-01-06 10:32:05 -08:00
Carl Worth
4c1cca888f database: Store directory path in 'data' of directory documents.
We're planning to have mail documents refer to directory documents for
the path of the containing directory. To support this, we need the
path in the data, (since the path in the 'directory' term can be
irretrievable as it will be the SHA1 sum of the path for a very long
path).
2010-01-06 10:32:05 -08:00
Carl Worth
406ec4b15d database: Export _notmuch_database_find_parent_id for internal use.
We'll soon have mail documents referring to their parent directory's
directory documents, so we'll need access to _find_parent_id in files
such as message.cc.
2010-01-06 10:32:05 -08:00
Carl Worth
e890b0cf40 database: Store the parent ID for each directory document.
Storing the document ID of the parent of each directory document will
allow us to find all child-directory documents for a given directory
document. We will need this in order to detect directories that have
been removed from the mail store, (though we aren't yet doing this).
2010-01-06 10:32:05 -08:00
Carl Worth
851c97aed7 database: Rename internal directory value from XTIMESTAMP to XDIRECTORY.
The recent change from storing absolute paths to relative paths means
that new directory documents will already be created, (and the old
ones will just linger stale in the database). Given that, we might as
well put a clean name on the term in the new documents, (and no real
flag day is needed).
2010-01-06 10:32:05 -08:00
Carl Worth
154bf7ac67 database: Store directory paths as relative, not absolute.
We were already storing relative mail filenames, so this is consistent
with that. Additionally, it means that directory documents remain
valid even if the database is relocated within its containing
filesystem.
2010-01-06 10:32:05 -08:00
Carl Worth
9257622da8 lib: Document that the filename is stored in the 'data' of a mail document
Our database schema documentation previously didn't give any
indication of where this most essential piece of information is
stored.
2010-01-06 10:32:05 -08:00
Carl Worth
50ae83a17f lib: Rename set/get_timestamp to set/get_directory_mtime.
I've been suitably scolded by Keith for doing a premature
generalization that ended up just making the documentation more
convoluted. Fix that.
2010-01-06 10:32:05 -08:00
Carl Worth
ba12bf1f26 lib: Abstract the extraction of a relative path from set_filename
We'll soon be having multiple entry points that accept a filename
path, so we want common code for getting a relative path from a
potentially absolute path.
2010-01-06 10:32:05 -08:00
Carl Worth
8c6b7d311c lib: Add missing value to notmuch_private_status_t enum.
And fix the initialization such that the private enum will always have
distinct values from the public enum even if we similarly miss the
addition of a new public value in the future.
2010-01-06 10:32:05 -08:00
Fernando Carrijo
db68eea013 Nuke the remainings of _notmuch_message_add_thread_id.
The function _notmuch_message_add_thread_id has been removed
from the private interface of notmuch. There's no reason for
one to keep a declaration of its prototype in the code base.
Also, lets update a commentary that referenced that function
and escaped from previous scrutiny.

Signed-off-by: Fernando Carrijo <fcarrijo@yahoo.com.br>
2009-12-09 12:09:55 -08:00
Jeffrey C. Ollie
95f97540a0 Remove unused notmuch_parse_date function prototype.
notmuch_parse_date is not implemented, so remove the unused function
prototype.

Signed-off-by: Jeffrey C. Ollie <jeff@ocjtech.us>
2009-12-03 17:07:22 -08:00
Carl Worth
880b21a097 Makefile: Incorporate getline implementation into the build.
It's unconditional for a very short time. We expect to soon be
building it only if necessary.
2009-12-01 16:33:17 -08:00
Carl Worth
8b445212e4 xutil: Implement xstrndup without relying on strndup.
Since we need to do this for portability, (some systems don't have a
strndup function), we might as well do it unconditionally. There's
almost no disadvantage to doing so, and this has the advantages of not
requiring a configure-time check nor having two different
implementations, one of which would often be less tested.
2009-12-01 12:51:39 -08:00
Carl Worth
e5316b320a lib/index: Fix memory leak for email addresses without names.
We carefully noted the fact that we had locally allocated the string
here, but then we neglected to free it. Switch to talloc instead
which makes it easier to get the behavior we want. It's simpler since
we can just call talloc_free unconditionally, without having to track
the state of whether we allocated the storage for name or not.
2009-12-01 12:40:13 -08:00
Carl Worth
64c8d6227a Avoid bogus internal error reporting duplicate In-Reply-To IDs.
This error was tirggered with a debugging build via:

	make CXXFLAGS="-DDEBUG"

and reported by David Bremner. The actual error is that I'm an
idiot that doesn't know how to use strcmp's return value. Of
course, the strcmp interface scores a negative 7 on Rusty Russell
ranking of bad interfaces:

http://ozlabs.org/~rusty/index.cgi/tech/2008-04-01.html
2009-11-28 10:01:22 -08:00
Bart Trojanowski
6da6566576 correct message flag enum value so that it matches the type
As per Carl's request, this patch corrects the only value defined under
the notmuch_message_flag_t enum typedef to match the name of the type.

Signed-off-by: Bart Trojanowski <bart@jukie.net>
2009-11-27 20:50:59 -08:00
David Bremner
12c91e8050 add missing comma in debugging code 2009-11-27 19:51:53 -08:00
Jeffrey C. Ollie
e991148b00 Silence compiler warning by initializing a variable.
If Xapian threw an exception on notmuch_query_count_messages the count
variable could be used uninitialized.  Initialize count to solve the
problem.

Signed-off-by: Jeffrey C. Ollie <jeff@ocjtech.us>
2009-11-27 18:38:06 -08:00
Bart Trojanowski
bede40987a have _notmuch_thread_create mark which messages matched the query
When _notmuch_thread_create() is given a query string, it can return more
messages than just those matching the query.  To distinguish those that
matched the query expression, the MATCHING_SEARCH flag is set
appropriately.

Signed-off-by: Bart Trojanowski <bart@jukie.net>
2009-11-27 17:07:06 -08:00
Bart Trojanowski
c57a0b4f8b message: add flags to notmuch_message_t
This patch allows for different flags, internal to notmuch, to be set on a
message object.  The patch does not define any such flags, just the
facilities to manage these flags.

Signed-off-by: Bart Trojanowski <bart@jukie.net>
2009-11-27 17:06:50 -08:00
Jan Janak
523a6f67ec lib: New function to collect tags from a list of messages.
This patch adds a new function that can be used to collect a list of
unique tags from a list of messages. 'notmuch search-tags' uses the
function to get a list of tags from messages matching a search-term,
but it has the potential to be used elsewhere so we put it in the lib.

Signed-off-by: Jan Janak <jan@ryngle.com>
2009-11-26 07:06:04 -08:00
Jan Janak
c3c52e464b notmuch: New function to retrieve all tags from the database.
This patch adds a new function called notmuch_database_get_all_tags
which can be used to obtain a list of all tags from the database
(in other words, the list contains all tags from all messages). The
function produces an alphabetically sorted list.

To add support for the new function, we rip the guts off of
notmuch_message_get_tags and put them in a new generic function
called _notmuch_convert_tags. The generic function takes a
Xapian::TermIterator as argument and uses the iterator to find tags.
This makes the function usable with different Xapian objects.

Function notmuch_message_get_tags is then reimplemented to call the
generic function with message->doc.termlist_begin() as argument.

Similarly, we implement notmuch_message_database_get_all_tags, the
function calls the generic function with db->xapian_db->allterms_begin()
as argument.

Finally, notmuch_database_get_all_tags is exported through
lib/notmuch.h

Signed-off-by: Jan Janak <jan@ryngle.com>
2009-11-26 07:01:52 -08:00
Carl Worth
70962fabf9 lib/messages.c: Make message searches stream as well.
Xapian provides an interator-based interface to all search results.
So it was natural to make notmuch_messages_t be iterator-based as
well. Which we did originally.

But we ran into a problem when we added two APIs, (_get_replies and
_get_toplevel_messages), that want to return a messages iterator
that's *not* based on a Xapian search result. My original compromise
was to use notmuch_message_list_t as the basis for all returned
messages iterators in the public interface.

This had the problem of introducing extra latency at the beginning
of a search for messages, (the call would block while iterating over
all results from Xapian, converting to a message list).

In this commit, we remove that initial conversion and instead provide
two alternate implementations of notmuch_messages_t (one on top of a
Xapian iterator and one on top of a message list).

With this change, I tested a "notmuch search" returning *many* results
as previously taking about 7 seconds before results started appearing,
and now taking only 2 seconds.
2009-11-24 11:33:09 -08:00
Carl Worth
94eb9aacd4 lib/query: Drop the first and max_messages arguments from search_messages.
These only existed to support the chunky-searching hack, but that
was recently dropped anyway.
2009-11-23 20:25:13 -08:00
Carl Worth
ba3554b804 lib/query: Fix notmuch_threads_t to stream results rather than blocking.
Previously, notmuch_query_search_threads would do all the work, so the
caller would block until all results were processed. Now, we do the
work as we go, as the caller iterates with notmuch_threads_next. This
means that once results start coming back from "notmuch search" they
just keep continually streaming.

There's still some initial blocking before the first results appear
because the notmuch_messages_t object has the same bug (for now).
2009-11-23 20:18:57 -08:00
Carl Worth
1fd8b7866f notmuch search: Remove the chunked-searching hack.
This was a poor workaround around the fact that the existing
notmuch_threads_t object is implemented poorly. It's got a fine
iterartor-based interface, but the implementation does all of the
work up-front in _create rather than doing the work incrementally
while iterating.

So to start fixing this, first get rid of all the hacks we had working
around this. This drops the --first and --max-threads options from the
search command, (but hopefully nobody was using them
anyway---notmuch.el certainly wasn't).
2009-11-23 20:17:37 -08:00
Carl Worth
793cbf8049 Add rudimentary date-based search.
The rudimentary aspect here is that the date ranges are specified with
UNIX timestamp values (number of seconds since 1970-01-01 UTC). One
thing that can help here is using the date program to determins
timestamps, such as:

	$(date +%s -d 2009-10-01)..$(date +%s)

Long-term, we'll probably need to do our own query parsing to be able
to support directly-specified dates and also relative expressions like
"since:'2 months ago'".
2009-11-23 17:17:08 +01:00
Keith Packard
53f8cc5651 Add 'notmuch count' command to show the count of matching messages
Getting the count of matching threads or messages is a fairly
expensive operation. Xapian provides a very efficient mechanism that
returns an approximate value, so use that for this new command.

This returns the number of matching messages, not threads, as that is
cheap to compute.

Signed-off-by: Keith Packard <keithp@keithp.com>
2009-11-23 06:33:54 +01:00
Bart Trojanowski
ceee152fca fix notmuch-new bug when database path ends with a trailing /
I configured my database.path with a trailing /, and after running notmuch
new every notmuch search would fail with error messages like this:

  Error opening /inbox/cur/1258565257.000211.mbox:2,S: No such file or directory

The actual bug was in the filename normalization for storage in the
database.  The database.path was removed from the full filename, but if
the database.path from the config file contained a trailing /, the
relative file name would retain an extra leading /... which made it look
like an absolute path after it was read out from the DB.

Signed-off-by: Bart Trojanowski <bart@jukie.net>
2009-11-23 04:37:01 +01:00
Chris Wilson
3e4ab913db lib/database.cc: coding style
Carl claims he must have been distracted when he wrote this...

Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
2009-11-22 05:26:59 +01:00
Chris Wilson
530df68258 Makefile: Magic silent rules.
Use the facilities of GNU make to create a magic function that will
on the first invocation print a description of how to enable verbose
compile lines and then print the quiet rule.

Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Signed-off-by: Carl Worth <cworth@cworth.org>
Cc: Mikhail Gusarov <dottedmag@dottedmag.net>

[ickle: Rebased, and duplicate command string eliminated.]
[ickle: Fixed verbose bug pointed out by Mikhail]
2009-11-22 04:29:29 +01:00
Carl Worth
5d56e931b9 add_message: Use sha-1 in place of overly long message ID.
Since Xapian has a limit on the maximum length of a term, we have
to check for that before trying to add the message ID as a term.

This fixes the bug reported by Mike Hommey here:

	<20091120132625.GA19246@glandium.org>

I've also constructed 20 files with a range of message ID lengths
centered around the Xapian term-length limit which I'll use to seed a
new test suite soon.
2009-11-22 04:03:49 +01:00
Carl Worth
f336ee034b get_timestamp: Ensure that return value is 0 in case of exception.
Just to be on the safe side of things.
2009-11-22 03:55:39 +01:00
Carl Worth
e2341cbc09 Catch and optionally print about exception at database->flush.
If an earlier exception occurred, then it's not unexpected for the
flush to fail as well. So in that case, we'll silently catch the
exception. Otherwise, make some noise about things going wrong at the
time of flush.
2009-11-22 03:54:20 +01:00
Carl Worth
717279fbcf Add a missing print after catching an exception.
Without this, trying to debug this exception was *really* confusing.
2009-11-22 03:52:55 +01:00
Carl Worth
591f901241 Print information about where Xapian exception occurred.
Previously, our Xapian exception reports where identical so they
were hard to track down.
2009-11-22 03:51:35 +01:00
Carl Worth
b725481cb3 Fix freak case problem that broke the compile.
I think I must have bumped some emacs keybinding that changed the case
of a word here.
2009-11-21 22:29:31 +01:00
Carl Worth
637f99d8f3 Rename NOTMUCH_DATABASE_MODE_WRITABLE to NOTMUCH_DATABASE_MODE_READ_WRITE
And correspondingly, READONLY to READ_ONLY.
2009-11-21 22:10:18 +01:00
Chris Wilson
f379aa5284 Permit opening the notmuch database in read-only mode.
We only rarely need to actually open the database for writing, but we
always create a Xapian::WritableDatabase. This has the effect of
preventing searches and like whilst updating the index.

Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk>
Acked-by: Carl Worth <cworth@cworth.org>
2009-11-21 22:04:49 +01:00
Eric Anholt
59c241ebd0 When a search query triggers a Xapian exception, log what the query was.
In my script containing a series of queries to be run on new mail for
setting up tags, it's nice to see which query I typed wrong.

Signed-off-by: Eric Anholt <eric@anholt.net>
2009-11-21 00:18:15 +01:00
Carl Worth
3ae12b1e28 add_message: Re-fix handling of non-mail files.
More fallout from _get_header now returning "" for missing headers.

The bug here is that we would no longer detect that a file is not an
email message and give up on it like we should.

And this time, I actually audited all callers to
notmuch_message_get_header, so hopefully we're done fixing this
bug over and over.
2009-11-20 21:46:37 +01:00
Carl Worth
656e4c413d notmuch_database_add_message: Add missing error-value propagation.
Thanks to Mike Hommey for doing the analysis that led to noticing that
this was missing.
2009-11-20 21:02:11 +01:00
Carl Worth
52292c5485 add_message: Properly handle missing Message-ID once again.
There's been a fair amount of fallout from when we changed
message_file_get_header from returning NULL to returning "" for
missing headers. This is yet more fallout from that, (where we were
accepting an empty message-ID rather than generating one like we want
to).
2009-11-20 19:36:01 +01:00
Carl Worth
31b54bc787 Avoid access of a Xapian iterator's object when there's nothing there.
This eliminates a crash when a message (either corrupted or a non-mail
file that wasn't properly detected as not being mail) has no In-Reply-To
header, (and so few terms that trying to skip to the prefix of the
In-Reply-To terms actually brings us to the end of the termlist).
2009-11-20 12:06:11 +01:00
Adrian Perez
e5da2b701f Allow lone "not" search operators
As suggested by Keith in FLAG_PURE_NOT allows for expressions like:

  notmuch search NOT tag:inbox

Note that this way a search like:

  notmuch search foobar NOT tag:inbox

should not be written instead:

  notmuch search foobar AND NOT tag:inbox

In my opinion, the latter feels more natural and is somewhat more explicit.
It gives a better clue of what the search is about instead of assuming that
an implicit AND operator is there.
2009-11-19 01:42:31 +01:00
Ingmar Vanhassel
2ce25b93a7 Typsos 2009-11-18 03:21:36 -08:00
Carl Worth
fc3a3be337 linke_message: Avoid segfault when In-Reply-to header is empty.
This was recently introduced in commit:

	64c03ae97f

which was adding extra checks to avoid adding a self-referencing
message.

How many times am I going to fix a dumb regression like this and say
"we really need a test suite" before I actually sit down and write the
test suite?
2009-11-18 01:36:30 -08:00
Carl Worth
3334865725 notmuch search: Change default search order to be newest messages first.
This is what most people want for a _search_ command. It's often
different for actually reading mail in an inbox, (where it makes more
sense to have results displayed in chronological order), but in such a
case, ther user is likely using an interface that can simply pass the
--sort=oldest-first option to "notmuch search".

Here we're also change the sort enum from NOTMUCH_SORT_DATE and
NOTMUCH_SORT_DATE_REVERSE to NOTMUCH_SORT_OLDEST_FIRST and
NOTMUCH_SORT_NEWEST_FIRST. Similarly we replace the --reverse option
to "notmuch search" with two options: --sort=oldest-first and
--sort=newest-first.

Finally, these changes are all tracked in the emacs interface, (which
has no change in its behavior).
2009-11-17 20:58:30 -08:00
Carl Worth
0da0131096 database: Make _parse_message_id static once again.
We had exposed this to the internal implementation for a short time,
(only while we had the silly code fetching In-Reply-To values from
message files instead of from the database). Make this private again
as it should be.
2009-11-17 18:50:13 -08:00
Carl Worth
c50891f449 database: Add "replyto" to the database schema documentation.
Maybe ths lack of this documentation is why I forgot we were actually
storing this and wrote the ugly code to fetch In-Reply-To from message
files rather than from the database.
2009-11-17 18:48:38 -08:00
Carl Worth
6e9fdf0abf database: Rename "ref" prefix name to "reference"
Which is more consistent with the XREFERENCE prefix used in the terms
in the database. Also remove some stale documentation describing the
removal of resolved references from the database (we no longer do
this).
2009-11-17 18:44:02 -08:00
Carl Worth
8cf72920e1 message_file_get_header: Use break where more clear than continue.
Calling continue here worked only because we set a flag before the
continue, and, check the flag at the beginning of the loop, and *then*
break. It's much more clear to just break in the first place.
2009-11-17 18:37:45 -08:00
Keith Packard
d025e89ac7 Fix "too many open files" bug by closing message files when done with them.
The message file header parsing code parses only enough of the file to
find the desired header fields, then it leaves the file open until the
next header parsing call or when the message is no longer in use. If a
large number of messages end up being active, this will quickly run
out of file descriptors.

Here, we add support to explicitly close the message file within a
message, (_notmuch_message_close) and call that from thread
construction code.

Signed-off-by: Keith Packard <keithp@keithp.com>

Edited-by: Carl Worth <cworth@cworth.org>:

Many portions of Keith's original patch have since been solved other
ways, (such as the code that changed the handling of the In-Reply-To
header). So the final version is clean enough that I think even Keith
would be happy to have his name on it.
2009-11-17 18:37:13 -08:00
Carl Worth
64c03ae97f add_message: Don't add any self-references to the database.
In our scheme it's illegal for any message to refer to itself, (nor
would it be useful for anything anyway). Cut these self-references off
at the source, before they trip up any internal errors.
2009-11-17 17:55:37 -08:00
Carl Worth
f7eaeff242 message_get_thread_id: Generate internal error if message has no thread ID.
This case was happening when a message had its own message ID in its
In-Reply-To header. The thread-resolution code would find the
partially constructed message, (with no thread ID yet), get garbage
from this function, and then march right along with that garbage.

With this commit, a self-cyclic message like this will now trigger an
internal error rather than marching along silienty. (And a subsequent
commit will remove the call to this function in this case.)
2009-11-17 17:42:32 -08:00
Carl Worth
24a25ffba9 Remove the talloc_owner argument from create_for_message_id.
This function has only one caller, and that one caller was passing the
same value for both talloc_owner and the notmuch database. Dropping
the redundant argument simplifies the documentation of this function
considerably.
2009-11-17 17:42:32 -08:00
Carl Worth
387828c435 get_in_reply_to: Implement via the database, not by opening mail file.
This reduces our reliance on open message_file objects, (so is a step
toward fixing the "too many open files" bug), but more importantly, it
means we don't load a self-referencing in-reply-to header, (since we
weed those out before adding any replyto terms to the database).
2009-11-17 17:40:19 -08:00
Carl Worth
12d3014d88 Fix broken commit.
Oops. I should have actually compiled before pushing.
2009-11-17 09:04:14 -08:00
Mikhail Gusarov
469ea9ebc6 Include <stdint.h> to get uint32_t in C++ file with gcc 4.4
Signed-off-by: Mikhail Gusarov <dottedmag@dottedmag.net>
2009-11-17 08:53:19 -08:00
Mikhail Gusarov
dc5a9d8eb2 Close message file after parsing message headers
Keeping unused files open helps to see "Too many open files" often.

Signed-off-by: Mikhail Gusarov <dottedmag@dottedmag.net>
2009-11-17 08:53:16 -08:00
Carl Worth
0dab6a2c1e add_message: Avoid a memory leak when user holds on to message return.
When this function was originally written, the 'message' object was
always destroyed locally, so I thought it would be good to use a NULL
talloc context to make it more obvious if there was any leak.

Since then, however, this function has been changed to optionally
return the added message, and in that case we *don't* free the message
locally, so let's let the database be the talloc context.
2009-11-17 08:50:14 -08:00
Carl Worth
933caf814f notmuch show: Implement proper thread ordering/nesting of messages.
We now properly analyze the in-reply-to headers to create a proper
tree representing the actual thread and present the messages in this
correct thread order. Also, there's a new "depth:" value added to the
"message{" header so that clients can format the thread as desired,
(such as by indenting replies).
2009-11-15 20:41:45 -08:00
Carl Worth
d136a1e2cf Add _notmuch_message_get_in_reply_to.
The existing notmuch_message_get_header is *almost* good enough for
this, except that we also need to remove the '<' and '>'
delimiters. We'll probably want to implement this function with
database storage in the future rather than loading the email message.
2009-11-15 20:36:51 -08:00
Carl Worth
b97756926f Remove obsolete notmuch_message_get_subject prototype.
This prototype has been sitting around for a while with no function
implementing it. I wonder if there's a compiler warning I could turn
on to catch these things.
2009-11-15 20:34:24 -08:00
Carl Worth
f970d8078c lib/messages: Add new notmuch_message_list_t to internal interface.
Previously, the notmuch_messages_t object was a linked list built on
top of a linked-list node with the odd name of notmuch_message_list_t.

Now, we've got much more sane naming with notmuch_message_list_t being
a list built on a linked-list node named notmuch_message_node_t. And
now the public notmuch_messages_t object is a separate iterator based
on notmuch_message_node_t. This means the interfaces for the new
notmuch_message_list_t object are now made available to the library
internals.
2009-11-15 20:31:30 -08:00
Carl Worth
9034e396b6 database: Fix a typo in a commit.
Nothing significant here, but we might as well not keep things
misspelled when we notice.
2009-11-15 20:23:27 -08:00
Carl Worth
9b1c6c250b Export _parse_message_id to the library implementation.
Not exported through the public interface, but the thread code is
going to want to be able to parse In-Reply-To headers so needs access
to this code.
2009-11-15 20:21:43 -08:00
Carl Worth
54be14098b _thread_add_messages: Remove unused variable.
I'm not sure how I let this warning go by unfixed for a while. Fix
it now.
2009-11-15 20:21:12 -08:00
Carl Worth
d3349358c6 lib: Move notmuch_messages_t code from query.cc to new messages.c
The new object is simply a linked-list of notmuch_message_t objects,
(unlike the old object which contained a couple of Xapian iterators).
This works now by the query code immediately iterator over all results
and creating notmuch_message_t objects for them, (rather than waiting
to create the objects until the notmuch_messages_get call as we did
earlier).

The point of this change is to allow other instances of lists of
messages, (such as in notmuch_thread_t), that are not directly related
to Xapian search results.
2009-11-14 23:05:17 -08:00
Carl Worth
c979fc5b05 notmuch_tags_advance: Make safe against excessive calls.
Previously, an excess call would have caused a crash. Now it simply
does nothing. Also, make notmuch_tags_get use a similar, consistent
early return for a NULL iterator.
2009-11-14 23:02:55 -08:00
Carl Worth
ed2643333c notmuch search: Fix thread dates to come only from matched messages.
We were properly sorting the threads based only on matched messages,
but we were displaying the date based on the total messages in the
thread, which led to inconsistent and very confusing results.
2009-11-12 23:10:04 -08:00
Carl Worth
f7b49d658a notmuch search: Add support for a --reverse option to reverse sort order.
Note that the difference between thread results in date order and
thread results in reverse-date order is not simply a matter of
reversing the final results. When sorting in date order, the threads
are sorted by the oldest message in the thread. When sorting in
reverse-date order, the threads are sorted by the newest message in
the thread.

This difference means that we might want an explicit option in the
interface to reverse the order, (even though the default will be to
display the inbox in date order and global searches in reverse-date
order).
2009-11-12 22:35:16 -08:00
Carl Worth
c168e24174 notmuch search: Print the number of matched/total messages for each thread.
Note that we don't print the number of *unread* messages, but instead
the number of messages that matched the search terms. This is in
keeping with our philosophy that the inbox is nothing more than a
search view. If we search for messages with an inbox tag, then that's
what we'll get a count of. (And if somebody does want to see unread
counts, then they can search for the "unread" tag.)

Getting the number of matched messages is really nice when doing
historical searches. For example in a search like:

	notmuch search tag:sent

(where the "sent" tag has been applied to all messages originating
from the user's email address)---here it's really nice to be able to
see a thread where the user just mentioned one point [1/13] vs. really
getting involved in the discussion [10/29].
2009-11-12 22:01:44 -08:00
Carl Worth
184c327692 Use $(MAKE) when invoking make from make.
Without this, things like MAKEFLAGS=-j4 can't work.
2009-11-12 22:00:51 -08:00
Carl Worth
c3c0966521 notmuch search: Avoid printing duplicate author names.
We add a hash to the thread object so that we can detect author names
that have already been added to the list, and avoid adding them
redundantly. This avoids the giant chain of "bugzilla-daemon,
bugzilla-daemon, bugzilla-daemon, bugzilla-daemon, ..." author lists
that we would get otherwise, for example.
2009-11-12 21:19:42 -08:00
Carl Worth
ec6d3506db notmuch search: Print all authors contributing to a thread.
We've now expanded the notmuch_thread_create function to fire off a
secondary database query to find all the messages that belong to this
particular thread. This allows us to now have the complete authors'
list for the thread, and will also make it trivial to print accurate
message counts for threads in the future.
2009-11-12 21:09:54 -08:00
Carl Worth
bbf4b8e4ae notmuch_query_search_threads: Avoid returning more threads than asked for.
I thought it would be safe enough to return a few extra threads,
(since we happened to already get the relevant messages out of the
database). The problem is that then requires the caller to carefully
read the number of threads returned and adjust its next "first" value
accordingly. The interface is much simpler to use if we simply return
exactly what is asked for and no more.
2009-11-12 20:31:22 -08:00
Carl Worth
e4a7c2b870 notmuch search: Fix a second bug in the change to support incremental searches.
The search was dropping the first thread from the results.

When am I going to break down and write a test suite? It's long
overdue now.
2009-11-12 20:12:16 -08:00
Carl Worth
523f1ce0a5 notmuch search: Fix to actually return something.
This serves me right for committing untested code. The
notmuch_query_search_threads was totally broken, (it didn't properly
treat -1 as being unlimited and instead returned no threads in that
case).
2009-11-12 20:09:12 -08:00
Carl Worth
93dcc3b695 libnotmuch: Underlying support for doing partial-results searches.
The library interface now allows the caller to do incremental searches,
(such as one page of results at a time). Next we'll just need to hook
this up to "notmuch search" and the emacs interface.
2009-11-12 16:47:27 -08:00
Carl Worth
2f4c587474 notmuch search: Print the names of author of matched emails.
It's important to have the names present for determining whether a
thread is worth reading or not. We may want to think about
abbreviating the list somehow if it is excessively long (or redundant
as in bugzilla-daemon, bugzilla-daemon, bugzilla-daemon, etc.).
2009-11-12 09:59:47 -08:00
Carl Worth
4d35c3544d Don't create "contact" terms in the database.
We never did export any interface to get at these, and when I went to
use these, I found them inadequate, (because I wanted to distinguish
address found in from: from those found in To:). Meanwhile, it was
easy enough to extract addresses with a search like:

	notmuch show tag:sent | grep ^To:

so the storage of contact terms was just wasting space. Stop that.
2009-11-12 09:38:24 -08:00
Carl Worth
091d18c54c notmuch show: Avoid segmentation for message with no subject.
It's safer to return an empty string rather than NULL for missing
header values.
2009-11-11 23:00:58 -08:00
Carl Worth
305e76bc0a notmuch: Add a configuration system.
This will allow for things like the database path to be specified
without any cheesy NOTMUCH_BASE environment variable. It also will
allow "notmuch reply" to recognize the user's email address when
constructing a reply in order to do the right thing, (that is, to use
the user's address to which mail was sent as From:, and not to reply
to the user's own addresses).

With this change, the "notmuch setup" command is now strictly for
changing the configuration of notmuch. It no longer creates the
database, but instead instructs the user to call "notmuch new" to do
that.
2009-11-11 17:01:55 -08:00
Carl Worth
946c94b6a1 Makefile: Make the top-level Makefile a little more independent.
Previously, the top-level Makefile was explicitly adding -I./lib to
the compiler flags. However, that's something that's much better done
from within the Makefile.local fragment within the lib directory
itself.
2009-11-10 08:34:54 -08:00
Carl Worth
97c7cffdc6 Makefile: Fix dependency generation to make .d files themselves dependent.
I saw this recommendation in the implementation notes for "Recursive
Make Considered Harmful" and then the further recommendation for
implementing the idea in the GNU make manual.

The idea is that if any of the files change then we need to regenerate
the dependency file before we regenerate any targets.

The approach from the GNU make manual is simpler in that it just uses
a sed script to fix up the output of an extra invocation of the
compiler, (as opposed to the approach in the implementation notes from
the paper's author which use a wrapper script for the compiler that's
always invoked rather than the compiler itself).
2009-11-10 08:04:54 -08:00
Carl Worth
c5dccd851a Implement a non-recursive make.
The idea here is that every Makefile at each lower level will be an
identical, tiny file that simply defers to a top-level make.

Meanwhile, the Makefile.local file at each level is a Makefile snippet
to be included at the top-level into a large, flat Makefile. As such,
it needs to define its rules with the entire relative directory to
each file, (typically in $(dir)). The local files can also append to
variables such as SRCS and CLEAN for files to be analyzed for
dependencies and to be cleaned.
2009-11-10 07:24:10 -08:00
Carl Worth
1465493210 libify: Move library sources down into lib directory.
A "make" invocation still works from the top-level, but not from
down inside the lib directory yet.
2009-11-09 16:24:03 -08:00