The compatibility wrapper ensures that clients calling
notmuch_database_open will receive consistent output for now.
The changes to notmuch-{new,search} and test/symbol-test are just to
make the test suite pass.
The use of IGNORE_RESULT is justified by two things. 1) I don't know
what else to do. 2) asprintf guarantees the output string is NULL if
an error occurs, so at least we are not passing garbage back.
The default is actually exact if no checkatleast parameter is
specified. This change makes that explicit, mainly for documentation,
but also to be safe in the unlikely event of a change of default.
[ commit message rewritten by db based on id:87lho0nlkk.fsf@nikula.org
]
The install_name of libnotmuch.dylib on Mac OS X is what is written
into a program that links against it. If it is just the name of the
shared library file, as opposed to the full path, the program won't be
able to find it when it runs and will abort. Instead, the install_name
should be the full path to the shared library (in its final installed
location).
Why does Notmuch work without this patch when installed via Homebrew?
The answer is twofold. One, /usr/local/lib is a special location in
which the dynamic linker will look by default to find shared libraries.
Homebrew highly recommends installing to /usr/local, and, assuming it
has been configured this way, the Notmuch library will end up installed
in /usr/local/lib, and the dynamic linker will find it. Two, Homebrew
globally corrects all install names in dynamically shared libraries and
binaries for each package it installs. So, even if the install names in
a package's binaries and libraries are incorrect, Homebrew corrects them
automatically, and no one ever knows.
Why does Notmuch work without this patch when installed via MacPorts?
The answer is that MacPorts applies a patch just like this patch to fix
the same problem.
This indicates upwardly compatible changes, namely adding new symbols.
Although we don't formally need to do this until the next release,
there is no hard in doing it now, as long as we don't bump the minor
version for every addition between now and the release.
This at least allows distinguishing between out of memory and Xapian
exceptions. Adding finer grained status codes would allow different
Xapian exceptions to be preserved.
Adding wrappers allows people to transition gradually to the new API,
at the cost of bloating the library API a bit.
This adds the indexing support for the "mimetype:" term and removes
the broken test flag. The indexing is probablistic in Xapian terms,
which gives a better experience to end users. Standard content-types
of the form "foo/bar" are automatically interpreted as phrases in
Xapian due to the embedded slash.
Assume, separate messages with application/pdf and application/x-pdf
are indexed, then:
- mimetype:application/x-pdf will find only the application/x-pdf
- mimetype:application/pdf will find only the application/pdf
- mimetype:pdf will find both of the messages
This feature will exist in all newly created databases, but there is
no upgrade provided for it. If this flag exists, it indicates that
the database was created after the indexed MIME-types feature was
added.
_thread_set_subject_from_message sometimes replaces the subject, making the
cur_subject point to free'd memory
==6550== ERROR: AddressSanitizer: heap-use-after-free on address 0x601a0000bec0 at pc 0x4464a4 bp 0x7fffa40be910 sp 0x7fffa40be908
READ of size 1 at 0x601a0000bec0 thread T0
#0 0x4464a3 in _thread_add_matched_message /home/todd/.apps/notmuch/lib/thread.cc:369
#1 0x443c2c in notmuch_threads_get /home/todd/.apps/notmuch/lib/query.cc:496
#2 0x41d947 in do_search_threads /home/todd/.apps/notmuch/notmuch-search.c:131
#3 0x40a3fe in main /home/todd/.apps/notmuch/notmuch.c:345
#4 0x7f4e535b4ec4 in __libc_start_main /build/buildd/eglibc-2.19/csu/libc-start.c:287
#5 0x40abe6 in _start ??:?
0x601a0000bec0 is located 96 bytes inside of 134-byte region [0x601a0000be60,0x601a0000bee6)
freed by thread T0 here:
#0 0x7f4e54e6933a in __interceptor_free ??:?
#1 0x7f4e54482fab in _talloc_free ??:?
previously allocated by thread T0 here:
#0 0x7f4e54e6941a in malloc ??:?
#1 0x7f4e54485b5d in talloc_strdup ??:?
==22884== ERROR: AddressSanitizer: heap-buffer-overflow on address 0x601600008291 at pc 0x7ff6295680e5 bp 0x7fff4ab9aa40 sp 0x7fff4ab9aa08
READ of size 1 at 0x601600008291 thread T0
#0 0x7ff6295680e4 in __interceptor_strcmp ??:?
#1 0x44763b in _thread_add_message /home/todd/.apps/notmuch/lib/thread.cc:255
#2 0x4459e8 in notmuch_threads_get /home/todd/.apps/notmuch/lib/query.cc:496
#3 0x41e2a7 in do_search_threads /home/todd/.apps/notmuch/notmuch-search.c:131
#4 0x40a408 in main /home/todd/.apps/notmuch/notmuch.c:345
#5 0x7ff627cb9ec4 in __libc_start_main /build/buildd/eglibc-2.19/csu/libc-start.c:287
#6 0x40abf3 in _start ??:?
0x601600008291 is located 0 bytes to the right of 97-byte region [0x601600008230,0x601600008291)
allocated by thread T0 here:
#0 0x7ff62956e41a in malloc ??:?
#1 0x7ff628b8ab5d in talloc_strdup ??:?
Currently the thread is named based on either the oldest or newest
matching message (depending on the search order). If this message has
an empty subject, though, the thread will show up with an empty
subject in the search results. (See the thread starting with
`id:1412371140-21051-1-git-send-email-david@tethera.net` for an
example.)
This changes the behavior so it will use a non-empty name for the
thread if possible. We name threads based on (a) non-empty matches for
the query, and (b) the search order. If the search order is
oldest-first (as in the default inbox) it chooses the oldest matching
non-empty message as the subject. If the search order is newest-first
it chooses the newest one.
Tamas Szakaly points out [1] that the bug fixed in 51b073c still
exists in at least one place. This change follows the suggestion of
[2] and creates a block scope temporary std::string to avoid the rules
of iterators temporaries.
[1]: id:20141226113755.GA64154@pamparam
[2]: id:20141226230655.GA41992@pamparam
We generally do not support mbox files, but for historical reasons
we've supported single-message mbox files, with a deprecation
message. We've tried dropping the support altogether, but backed out
of it because we'd need to stop indexing them, while keeping support
for previously indexed files. This would be more complicated than
simply supporting single-message mbox files. Therefore, drop the
deprecation message, and just silently accept single-message mboxes.
Currently, if a From-header is of the form:
"" <address@example.com>
the empty string will be treated as a valid real-name, and the entry
in the search results will be empty.
The new behavior here is that we treat an empty real-name field as if
it were null, so that the email address will be used in the search
results instead.
Signed-off-by: Jesse Rosenthal <jrosenthal@jhu.edu>
Previously, it was necessary to link new messages to children to work
around some (though not all) problems with the old metadata-based
approach to stored thread IDs. With ghost messages, this is no longer
necessary, so don't bother with child linking when ghost messages are
in use.
This updates the thread linking code to use ghost messages instead of
user metadata to link messages into threads.
In contrast with the old approach, this is actually correct.
Previously, thread merging updated only the thread IDs of message
documents, not thread IDs stored in user metadata. As originally
diagnosed by Mark Walters [1] and as demonstrated by the broken
T260-thread-order test, this can cause notmuch to fail to link
messages even though they're in the same thread. In principle the old
approach could have been fixed by updating the user metadata thread
IDs as well, but these are not indexed and hence this would have
required a full scan of all stored thread IDs. Ghost messages solve
this problem naturally by reusing the exact same thread ID and message
ID representation and indexing as regular messages.
Furthermore, thanks to this greater symmetry, ghost messages are also
algorithmically simpler. We continue to support the old user metadata
format, so this patch can't delete any code, but when we do remove
support for the old format, several functions can simply be deleted.
[1] id:8738h7kv2q.fsf@qmul.ac.uk
This updates the message abstraction to support ghost messages: it
adds a message flag that distinguishes regular messages from ghost
messages, and an internal function for initializing a newly created
(blank) message as a ghost message.
In the interest of robustness, avoid undefined behavior of
sortable_unserialise if the date value is missing. This shouldn't
happen now, but ghost messages will have blank date values.
This moves the code to retrieve and clear the metadata thread ID out
of _notmuch_database_link_message into its own function. This will
simplify future changes.
Previously, this was performed by notmuch_database_add_message. This
happens to be the only caller currently (which is why this was safe),
but we're about to introduce more callers, and it makes more sense to
put responsibility for ID compression in the lower-level function
rather than requiring each caller to handle it.
In Xapian, closing a database implicitly aborts any outstanding
transaction and commits changes. For historical reasons,
notmuch_database_close had grown to almost, but not quite duplicate
this behavior. Before closing the database, it would explicitly (and
unnecessarily) commit it. However, if there was an outstanding
transaction (ie atomic section), commit would throw a Xapian
exception, which notmuch_database_close would unnecessarily print to
stderr, even though notmuch_database_close would ultimately abort the
transaction anyway when it called close.
This patch simplifies notmuch_database_close to explicitly abort any
outstanding transaction and then just call Database::close. This
works for both read-only and read/write databases, takes care of
committing changes, unifies the exception handling path, and codifies
aborting outstanding transactions. This is currently the only way to
abort an atomic section (and may remain so, since it would be
difficult to roll back things we may have cached from rolled-back
modifications).
as stated in thread.cc:115
/* Construct an authors string from matched_authors_array and
* authors_array. The string contains matched authors first, then
* non-matched authors (with the two groups separated by '|'). Within
* each group, authors are listed in date order. */
this is, however, not reflected in the public API documentation in
notmuch.h:970. This patch a paragraph explaining how | separates the
group of authors of messages matching the query and those of messages
that do not, but are still contained in the thread.
48db8c8 introduced a disagreement between when
notmuch_database_needs_upgrade returned TRUE and when
notmuch_database_upgrade actually performed an upgrade. As a result,
if a database had a version less than 3, but no new features were
required, notmuch new would call notmuch_database_upgrade to perform
an upgrade, but notmuch_database_upgrade would return immediately
without updating the database version. Hence, the next notmuch new
would do the same, and so on.
Fix this by ensuring that the upgrade-required logic is identical
between the two.
Previously, there was no protection against a caller invoking an
operation on an old database version that would effectively corrupt
the database by treating it like a newer version.
According to notmuch.h, any caller that opens the database in
read/write mode is supposed to check if the database needs upgrading
and perform an upgrade if it does. This would protect against this,
but nobody (even the CLI) actually does this.
However, with features, it's easy to protect against incompatible
operations on a fine-grained basis. This lightweight change allows
callers to safely operate on old database versions, while preventing
specific operations that would corrupt the database with an
informative error message.
Commit 567bcbc2 introduced support for storing various headers in
document values. However, doing so in a backwards-compatible way
meant that genuinely empty header values could not be distinguished
from the old behavior of not storing the headers at all, so these
required parsing the original message.
Now that we have database features, new databases can declare that all
messages have header values, so if we have this feature flag, we can
use the stored header value even if it's the empty string.
This requires slight cleanup to notmuch_message_get_header, since the
code previously couldn't distinguish between empty headers and headers
that are never stored in the database (previously this distinction
didn't matter).
Previously, some parts of upgrade didn't report progress and for
others it was possible for the progress meter to restart at 0 part way
through the upgrade because each stage was reported separately.
Fix this by computing the total amount of work that needs to be done
up-front and updating completed work monotonically.
Rather than potentially making multiple passes over the same type of
data in the database, reorganize upgrade around each type of data that
may be upgraded. This eliminates code duplication, will make
multi-version upgrades faster, and will let us improve progress
reporting.
Previously, we had database version information hard-coded in the
upgrade code. Slightly re-organize the upgrade process around the set
of new database features to be enabled by the upgrade.
Previously, the upgrade was organized as two passes -- an upgrade
pass, and a separate cleanup pass -- so the database was always in a
valid state. This change substantially simplifies this code by
performing the upgrade in a transaction and combining both passes in
to one. This 1) eliminates a lot of duplicate code between the
passes, 2) speeds up the upgrade process, 3) makes progress reporting
more accurate, 4) eliminates the potential for stale data if the
upgrade is interrupted during the cleanup pass, and 5) makes it easier
to reason about the safety of the upgrade code.
Previously, our database schema was versioned by a single number.
Each database schema change had to occur "atomically" in Notmuch's
development history: before some commit, Notmuch used version N, after
that commit, it used version N+1. Hence, each new schema version
could introduce only one change, the task of developing a schema
change fell on a single person, and it all had to happen and be
perfect in a single commit series. This made introducing a new schema
version hard. We've seen only two schema changes in the history of
Notmuch.
This commit introduces database schema version 3; hopefully the last
schema version we'll need for a while. With this version, we switch
from a single version number to "features": a set of named,
independent aspects of the database schema.
Features should make backwards compatibility easier. For many things,
it should be easy to support databases both with and without a
feature, which will allow us to make upgrades optional and will enable
"unstable" features that can be developed and tested over time.
Features also make forwards compatibility easier. The features
recorded in a database include "compatibility flags," which can
indicate to an older version of Notmuch when it must support a given
feature to open the database for read or for write. This lets us
replace the old vague "I don't recognize this version, so something
might go wrong, but I promise to try my best" warnings upon opening a
database with an unknown version with precise errors. If a database
is safe to open for read/write despite unknown features, an older
version will know that and issue no message at all. If the database
is not safe to open for read/write because of unknown features, an
older version will know that, too, and can tell the user exactly which
required features it lacks support for.
According to RFC2822 References and In-Reply-To headers are supposed
to contain one or more Message-IDs, however older RFC822 allowed
almost any content. When both References and In-Reply-To headers ends
with something else that a Message-ID (see e.g. [1]), the thread
structure presented by notmuch is incorrect. The reason is that
notmuch treats this case as if the email contained no "replyto"
information (see _notmuch_database_link_message_to_parents).
This patch changes the parse_references() function to return the last
valid Message-ID encountered rather than NULL resulting from the last
hunk of text not being the Message-ID.
[1] https://lkml.org/lkml/headers/2014/5/19/864
Previously, we invalidated stored message metadata in
_notmuch_message_add_term and _notmuch_message_remove_term, but not in
_notmuch_message_gen_terms. This doesn't currently result in any bugs
because of our limited uses of _notmuch_message_gen_terms, but it may
could cause trouble in the future.
As noted in devel/STYLE, every private library function should start
with _notmuch. This patch corrects function naming that did not adhere
to this style in lib/notmuch-private.h. In particular, the old function
names that now begin with _notmuch are
notmuch_sha1_of_file
notmuch_sha1_of_string
notmuch_message_file_close
notmuch_message_file_get_header
notmuch_message_file_open
notmuch_message_get_author
notmuch_message_set_author
Signed-off-by: Charles Celerier <cceleri@cs.stanford.edu>
notmuch_database_close may fail in Xapian ->flush() or ->close(), so
report the status. Similarly for notmuch_database_destroy which calls
close.
This is required for notmuch insert to report error status if message
indexing failed.
This adds a 100 termpos gap between all phrases indexed by
_notmuch_message_gen_terms. This fixes a bug where terms from the end
of one header and the beginning of another header could match together
in a single phrase and a separate bug where term positions of
un-prefixed terms overlapped.
This fix only affects newly indexed messages. Messages that are
already indexed won't benefit from this fix without re-indexing, but
the fix won't make things any worse for existing messages.
