debian/notmuch
1
0
Fork 0
mirror of https://git.notmuchmail.org/git/notmuch synced 2024-11-22 19:08:09 +01:00
Code Issues Projects Releases Packages Wiki Activity Actions
notmuch/lib/index.cc

607 lines
18 KiB
C++
Raw Normal View History

Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
/*
* Copyright © 2009 Carl Worth
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
Use https instead of http where possible Many of the external links found in the notmuch source can be resolved using https instead of http. This changeset addresses as many as i could find, without touching the e-mail corpus or expected outputs found in tests.
2016-06-02 18:26:14 +02:00
* along with this program. If not, see https://www.gnu.org/licenses/ .
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
*
* Author: Carl Worth <cworth@cworth.org>
*/
#include "notmuch-private.h"
#include <gmime/gmime.h>
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
#include <gmime/gmime-filter.h>
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
#include <xapian.h>
lib/index: separate state table definition from scanner. We want to reuse the scanner definition with a different table. This is mainly code movement, and making the state table part of the filter struct/class.
2017-06-08 04:11:47 +02:00
typedef struct {
int state;
int a;
int b;
int next_if_match;
int next_if_not_match;
} scanner_state_t;
/* Simple, linear state-transition diagram for the uuencode filter.
*
* If the character being processed is within the range of [a, b]
* for the current state then we transition next_if_match
* state. If not, we transition to the next_if_not_match state.
*
* The final two states are special in that they are the states in
* which we discard data. */
static const int first_uuencode_skipping_state = 11;
static const scanner_state_t uuencode_states[] = {
{0, 'b', 'b', 1, 0},
{1, 'e', 'e', 2, 0},
{2, 'g', 'g', 3, 0},
{3, 'i', 'i', 4, 0},
{4, 'n', 'n', 5, 0},
{5, ' ', ' ', 6, 0},
{6, '0', '7', 7, 0},
{7, '0', '7', 8, 0},
{8, '0', '7', 9, 0},
{9, ' ', ' ', 10, 0},
{10, '\n', '\n', 11, 10},
{11, 'M', 'M', 12, 0},
{12, ' ', '`', 12, 11}
};
lib/index: add simple html filter The filter just drops all (HTML) tags. As an enabling change, pass the content type to the filter constructor so we can decide which scanner to user.
2017-06-08 04:11:49 +02:00
/* The following table is intended to implement this DFA (in 'dot'
format). Note that 2 and 3 are "hidden" states used to step through
the possible out edges of state 1.
digraph html_filter {
0 -> 1 [label="<"];
0 -> 0;
1 -> 4 [label="'"];
1 -> 5 [label="\""];
1 -> 0 [label=">"];
1 -> 1;
4 -> 1 [label="'"];
4 -> 4;
5 -> 1 [label="\""];
5 -> 5;
}
*/
static const int first_html_skipping_state = 1;
static const scanner_state_t html_states[] = {
{0, '<', '<', 1, 0},
{1, '\'', '\'', 4, 2}, /* scanning for quote or > */
{1, '"', '"', 5, 3},
{1, '>', '>', 0, 1},
{4, '\'', '\'', 1, 4}, /* inside single quotes */
{5, '"', '"', 1, 5}, /* inside double quotes */
};
Eliminate some useless gobject boilerplate. If we had external users of this filter then they might expect some of these macros to exist. But since this is just internal, that's just unneeded noise.
2010-02-05 02:26:00 +01:00
/* Oh, how I wish that gobject didn't require so much noisy boilerplate!
* (Though I have at least eliminated some of the stock set...) */
lib/index: generalize name of indexing filter In followup commits we will generalize the functionality of this filter to deal with other types of non-indexable content.
2017-06-08 04:11:46 +02:00
typedef struct _NotmuchFilterDiscardNonTerm NotmuchFilterDiscardNonTerm;
typedef struct _NotmuchFilterDiscardNonTermClass NotmuchFilterDiscardNonTermClass;
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
/**
lib/index: generalize name of indexing filter In followup commits we will generalize the functionality of this filter to deal with other types of non-indexable content.
2017-06-08 04:11:46 +02:00
* NotmuchFilterDiscardNonTerm:
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
*
* @parent_object: parent #GMimeFilter
* @encode: encoding vs decoding
* @state: State of the parser
*
* A filter to discard uuencoded portions of an email.
*
* A uuencoded portion is identified as beginning with a line
* matching:
*
* begin [0-7][0-7][0-7] .*
*
* After that detection, and beginning with the following line,
* characters will be discarded as long as the first character of each
* line begins with M and subsequent characters on the line are within
* the range of ASCII characters from ' ' to '`'.
*
* This is not a perfect UUencode filter. It's possible to have a
* message that will legitimately match that pattern, (so that some
* legitimate content is discarded). And for most UUencoded files, the
* final line of encoded data (the line not starting with M) will be
* indexed.
**/
lib/index: generalize name of indexing filter In followup commits we will generalize the functionality of this filter to deal with other types of non-indexable content.
2017-06-08 04:11:46 +02:00
struct _NotmuchFilterDiscardNonTerm {
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
GMimeFilter parent_object;
lib/index: add simple html filter The filter just drops all (HTML) tags. As an enabling change, pass the content type to the filter constructor so we can decide which scanner to user.
2017-06-08 04:11:49 +02:00
GMimeContentType *content_type;
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
int state;
lib/index: separate state table definition from scanner. We want to reuse the scanner definition with a different table. This is mainly code movement, and making the state table part of the filter struct/class.
2017-06-08 04:11:47 +02:00
int first_skipping_state;
const scanner_state_t *states;
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
};
lib/index: generalize name of indexing filter In followup commits we will generalize the functionality of this filter to deal with other types of non-indexable content.
2017-06-08 04:11:46 +02:00
struct _NotmuchFilterDiscardNonTermClass {
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
GMimeFilterClass parent_class;
};
lib/index: add simple html filter The filter just drops all (HTML) tags. As an enabling change, pass the content type to the filter constructor so we can decide which scanner to user.
2017-06-08 04:11:49 +02:00
static GMimeFilter *notmuch_filter_discard_non_term_new (GMimeContentType *content);
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
lib/index: generalize name of indexing filter In followup commits we will generalize the functionality of this filter to deal with other types of non-indexable content.
2017-06-08 04:11:46 +02:00
static void notmuch_filter_discard_non_term_finalize (GObject *object);
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
static GMimeFilter *filter_copy (GMimeFilter *filter);
static void filter_filter (GMimeFilter *filter, char *in, size_t len, size_t prespace,
char **out, size_t *outlen, size_t *outprespace);
static void filter_complete (GMimeFilter *filter, char *in, size_t len, size_t prespace,
char **out, size_t *outlen, size_t *outprespace);
static void filter_reset (GMimeFilter *filter);
static GMimeFilterClass *parent_class = NULL;
static void
lib/index: generalize name of indexing filter In followup commits we will generalize the functionality of this filter to deal with other types of non-indexable content.
2017-06-08 04:11:46 +02:00
notmuch_filter_discard_non_term_class_init (NotmuchFilterDiscardNonTermClass *klass)
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
{
GObjectClass *object_class = G_OBJECT_CLASS (klass);
GMimeFilterClass *filter_class = GMIME_FILTER_CLASS (klass);
parent_class = (GMimeFilterClass *) g_type_class_ref (GMIME_TYPE_FILTER);
lib/index: generalize name of indexing filter In followup commits we will generalize the functionality of this filter to deal with other types of non-indexable content.
2017-06-08 04:11:46 +02:00
object_class->finalize = notmuch_filter_discard_non_term_finalize;
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
filter_class->copy = filter_copy;
filter_class->filter = filter_filter;
filter_class->complete = filter_complete;
filter_class->reset = filter_reset;
}
static void
lib/index: generalize name of indexing filter In followup commits we will generalize the functionality of this filter to deal with other types of non-indexable content.
2017-06-08 04:11:46 +02:00
notmuch_filter_discard_non_term_finalize (GObject *object)
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
{
G_OBJECT_CLASS (parent_class)->finalize (object);
}
static GMimeFilter *
filter_copy (GMimeFilter *gmime_filter)
{
lib/index: add simple html filter The filter just drops all (HTML) tags. As an enabling change, pass the content type to the filter constructor so we can decide which scanner to user.
2017-06-08 04:11:49 +02:00
NotmuchFilterDiscardNonTerm *filter = (NotmuchFilterDiscardNonTerm *) gmime_filter;
return notmuch_filter_discard_non_term_new (filter->content_type);
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
}
static void
filter_filter (GMimeFilter *gmime_filter, char *inbuf, size_t inlen, size_t prespace,
char **outbuf, size_t *outlen, size_t *outprespace)
{
lib/index: generalize name of indexing filter In followup commits we will generalize the functionality of this filter to deal with other types of non-indexable content.
2017-06-08 04:11:46 +02:00
NotmuchFilterDiscardNonTerm *filter = (NotmuchFilterDiscardNonTerm *) gmime_filter;
lib/index: separate state table definition from scanner. We want to reuse the scanner definition with a different table. This is mainly code movement, and making the state table part of the filter struct/class.
2017-06-08 04:11:47 +02:00
const scanner_state_t *states = filter->states;
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
register const char *inptr = inbuf;
const char *inend = inbuf + inlen;
char *outptr;
(void) prespace;
int next;
lib: convert notmuch_bool_t to stdbool internally C99 stdbool turned 18 this year. There really is no reason to use our own, except in the library interface for backward compatibility. Convert the lib internally to stdbool.
2017-10-07 10:44:05 +02:00
g_mime_filter_set_size (gmime_filter, inlen, false);
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
outptr = gmime_filter->outbuf;
lib/index.cc: generalize filter state machine To match things more complicated than fixed strings, we need states with multiple out arrows.
2017-06-08 04:11:48 +02:00
next = filter->state;
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
while (inptr < inend) {
lib/index.cc: generalize filter state machine To match things more complicated than fixed strings, we need states with multiple out arrows.
2017-06-08 04:11:48 +02:00
/* Each state is defined by a contiguous set of rows of the
* state table marked by a common value for '.state'. The
* state numbers must be equal to the index of the first row
* in a given state; thus the loop condition here looks for a
* jump to a first row of a state, which is a real transition
* in the underlying DFA.
*/
do {
if (*inptr >= states[next].a && *inptr <= states[next].b) {
next = states[next].next_if_match;
} else {
next = states[next].next_if_not_match;
}
} while (next != states[next].state);
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
lib/index: separate state table definition from scanner. We want to reuse the scanner definition with a different table. This is mainly code movement, and making the state table part of the filter struct/class.
2017-06-08 04:11:47 +02:00
if (filter->state < filter->first_skipping_state)
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
*outptr++ = *inptr;
filter->state = next;
inptr++;
}
*outlen = outptr - gmime_filter->outbuf;
*outprespace = gmime_filter->outpre;
*outbuf = gmime_filter->outbuf;
}
static void
filter_complete (GMimeFilter *filter, char *inbuf, size_t inlen, size_t prespace,
char **outbuf, size_t *outlen, size_t *outprespace)
{
if (inbuf && inlen)
filter_filter (filter, inbuf, inlen, prespace, outbuf, outlen, outprespace);
}
static void
filter_reset (GMimeFilter *gmime_filter)
{
lib/index: generalize name of indexing filter In followup commits we will generalize the functionality of this filter to deal with other types of non-indexable content.
2017-06-08 04:11:46 +02:00
NotmuchFilterDiscardNonTerm *filter = (NotmuchFilterDiscardNonTerm *) gmime_filter;
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
filter->state = 0;
}
/**
lib/index: generalize name of indexing filter In followup commits we will generalize the functionality of this filter to deal with other types of non-indexable content.
2017-06-08 04:11:46 +02:00
* notmuch_filter_discard_non_term_new:
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
*
lib/index: generalize name of indexing filter In followup commits we will generalize the functionality of this filter to deal with other types of non-indexable content.
2017-06-08 04:11:46 +02:00
* Returns: a new #NotmuchFilterDiscardNonTerm filter.
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
**/
lib: Add some missing static qualifiers. These various functions and data are all used only locally, so should be marked static. Ensuring we get these right will avoid us accidentally leaking unintended symbols through the library interface.
2010-11-02 05:58:43 +01:00
static GMimeFilter *
lib/index: add simple html filter The filter just drops all (HTML) tags. As an enabling change, pass the content type to the filter constructor so we can decide which scanner to user.
2017-06-08 04:11:49 +02:00
notmuch_filter_discard_non_term_new (GMimeContentType *content_type)
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
{
Eliminate some useless gobject boilerplate. If we had external users of this filter then they might expect some of these macros to exist. But since this is just internal, that's just unneeded noise.
2010-02-05 02:26:00 +01:00
static GType type = 0;
lib/index: generalize name of indexing filter In followup commits we will generalize the functionality of this filter to deal with other types of non-indexable content.
2017-06-08 04:11:46 +02:00
NotmuchFilterDiscardNonTerm *filter;
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
Eliminate some useless gobject boilerplate. If we had external users of this filter then they might expect some of these macros to exist. But since this is just internal, that's just unneeded noise.
2010-02-05 02:26:00 +01:00
if (!type) {
static const GTypeInfo info = {
lib/index: generalize name of indexing filter In followup commits we will generalize the functionality of this filter to deal with other types of non-indexable content.
2017-06-08 04:11:46 +02:00
sizeof (NotmuchFilterDiscardNonTermClass),
Eliminate some useless gobject boilerplate. If we had external users of this filter then they might expect some of these macros to exist. But since this is just internal, that's just unneeded noise.
2010-02-05 02:26:00 +01:00
NULL, /* base_class_init */
NULL, /* base_class_finalize */
lib/index: generalize name of indexing filter In followup commits we will generalize the functionality of this filter to deal with other types of non-indexable content.
2017-06-08 04:11:46 +02:00
(GClassInitFunc) notmuch_filter_discard_non_term_class_init,
Eliminate some useless gobject boilerplate. If we had external users of this filter then they might expect some of these macros to exist. But since this is just internal, that's just unneeded noise.
2010-02-05 02:26:00 +01:00
NULL, /* class_finalize */
NULL, /* class_data */
lib/index: generalize name of indexing filter In followup commits we will generalize the functionality of this filter to deal with other types of non-indexable content.
2017-06-08 04:11:46 +02:00
sizeof (NotmuchFilterDiscardNonTerm),
Eliminate some useless gobject boilerplate. If we had external users of this filter then they might expect some of these macros to exist. But since this is just internal, that's just unneeded noise.
2010-02-05 02:26:00 +01:00
0, /* n_preallocs */
NULL, /* instance_init */
NULL /* value_table */
};
lib/index: generalize name of indexing filter In followup commits we will generalize the functionality of this filter to deal with other types of non-indexable content.
2017-06-08 04:11:46 +02:00
type = g_type_register_static (GMIME_TYPE_FILTER, "NotmuchFilterDiscardNonTerm", &info, (GTypeFlags) 0);
Eliminate some useless gobject boilerplate. If we had external users of this filter then they might expect some of these macros to exist. But since this is just internal, that's just unneeded noise.
2010-02-05 02:26:00 +01:00
}
lib&cli: use g_object_new instead of g_object_newv 'g_object_newv' is deprecated, and prints annoying warnings. The warnings suggest using 'g_object_new_with_properties', but that's only available since glib 2.55 (i.e. a month ago as of this writing). Since we don't actuall pass any properties, it seems we can just call 'g_object_new'.
2017-09-03 13:55:42 +02:00
filter = (NotmuchFilterDiscardNonTerm *) g_object_new (type, NULL);
lib/index: add simple html filter The filter just drops all (HTML) tags. As an enabling change, pass the content type to the filter constructor so we can decide which scanner to user.
2017-06-08 04:11:49 +02:00
filter->content_type = content_type;
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
filter->state = 0;
lib/index: add simple html filter The filter just drops all (HTML) tags. As an enabling change, pass the content type to the filter constructor so we can decide which scanner to user.
2017-06-08 04:11:49 +02:00
if (g_mime_content_type_is_type (content_type, "text", "html")) {
filter->states = html_states;
filter->first_skipping_state = first_html_skipping_state;
} else {
filter->states = uuencode_states;
filter->first_skipping_state = first_uuencode_skipping_state;
}
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
return (GMimeFilter *) filter;
}
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
/* We're finally down to a single (NAME + address) email "mailbox". */
static void
_index_address_mailbox (notmuch_message_t *message,
const char *prefix_name,
InternetAddress *address)
{
InternetAddressMailbox *mailbox = INTERNET_ADDRESS_MAILBOX (address);
lib: Index name and address of from/to headers as a phrase Previously, we indexed the name and address parts of from/to headers with two calls to _notmuch_message_gen_terms. In general, this indicates that these parts are separate phrases. However, because of an implementation quirk, the two calls to _notmuch_message_gen_terms generated adjacent term positions for the prefixed terms, which happens to be the right thing to do in this case, but the wrong thing to do for all other calls. Furthermore, _notmuch_message_gen_terms produced potentially overlapping term positions for the un-prefixed copies of the terms, which is simply wrong. This change indexes both the name and address in a single call to _notmuch_message_gen_terms, indicating that they should be part of a single phrase. This masks the problem with the un-prefixed terms (fixing the two known-broken tests) and puts us in a position to fix the unintentionally phrases generated by other calls to _notmuch_message_gen_terms.
2014-06-16 04:40:32 +02:00
const char *name, *addr, *combined;
database: Store mail filename as a new 'direntry' term, not as 'data'. Instead of storing the complete message filename in the data portion of a mail document we now store a 'direntry' term that contains the document ID of a directory document and also the basename of the message filename within that directory. This will allow us to easily store multple filenames for a single message, and will also allow us to find mail documents for files that previously existed in a directory but that have since been deleted.
2009-12-21 17:23:26 +01:00
void *local = talloc_new (message);
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
name = internet_address_get_name (address);
index: Store "Full Name <user@example.com>" addressses in the database We put these is as a separate term so that they can be extracted. We don't actually need this for searching, since typing an email address in as a search term will already trigger a phrase search that does exactly what's wanted.
2009-10-28 21:09:08 +01:00
addr = internet_address_mailbox_get_addr (mailbox);
lib: Index name and address of from/to headers as a phrase Previously, we indexed the name and address parts of from/to headers with two calls to _notmuch_message_gen_terms. In general, this indicates that these parts are separate phrases. However, because of an implementation quirk, the two calls to _notmuch_message_gen_terms generated adjacent term positions for the prefixed terms, which happens to be the right thing to do in this case, but the wrong thing to do for all other calls. Furthermore, _notmuch_message_gen_terms produced potentially overlapping term positions for the un-prefixed copies of the terms, which is simply wrong. This change indexes both the name and address in a single call to _notmuch_message_gen_terms, indicating that they should be part of a single phrase. This masks the problem with the un-prefixed terms (fixing the two known-broken tests) and puts us in a position to fix the unintentionally phrases generated by other calls to _notmuch_message_gen_terms.
2014-06-16 04:40:32 +02:00
/* Combine the name and address and index them as a phrase. */
if (name && addr)
combined = talloc_asprintf (local, "%s %s", name, addr);
else if (name)
combined = name;
else
combined = addr;
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
lib: Index name and address of from/to headers as a phrase Previously, we indexed the name and address parts of from/to headers with two calls to _notmuch_message_gen_terms. In general, this indicates that these parts are separate phrases. However, because of an implementation quirk, the two calls to _notmuch_message_gen_terms generated adjacent term positions for the prefixed terms, which happens to be the right thing to do in this case, but the wrong thing to do for all other calls. Furthermore, _notmuch_message_gen_terms produced potentially overlapping term positions for the un-prefixed copies of the terms, which is simply wrong. This change indexes both the name and address in a single call to _notmuch_message_gen_terms, indicating that they should be part of a single phrase. This masks the problem with the un-prefixed terms (fixing the two known-broken tests) and puts us in a position to fix the unintentionally phrases generated by other calls to _notmuch_message_gen_terms.
2014-06-16 04:40:32 +02:00
if (combined)
_notmuch_message_gen_terms (message, prefix_name, combined);
lib/index: Fix memory leak for email addresses without names. We carefully noted the fact that we had locally allocated the string here, but then we neglected to free it. Switch to talloc instead which makes it easier to get the behavior we want. It's simpler since we can just call talloc_free unconditionally, without having to track the state of whether we allocated the storage for name or not.
2009-12-01 21:40:13 +01:00
talloc_free (local);
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
}
static void
_index_address_list (notmuch_message_t *message,
const char *prefix_name,
InternetAddressList *addresses);
/* The outer loop over the InternetAddressList wasn't quite enough.
* There can actually be a tree here where a single member of the list
* is a "group" containing another list. Recurse please.
*/
static void
_index_address_group (notmuch_message_t *message,
const char *prefix_name,
InternetAddress *address)
{
InternetAddressGroup *group;
InternetAddressList *list;
group = INTERNET_ADDRESS_GROUP (address);
list = internet_address_group_get_members (group);
if (! list)
return;
_index_address_list (message, prefix_name, list);
}
static void
_index_address_list (notmuch_message_t *message,
const char *prefix_name,
InternetAddressList *addresses)
{
int i;
InternetAddress *address;
if (addresses == NULL)
return;
for (i = 0; i < internet_address_list_length (addresses); i++) {
address = internet_address_list_get_address (addresses, i);
if (INTERNET_ADDRESS_IS_MAILBOX (address)) {
_index_address_mailbox (message, prefix_name, address);
} else if (INTERNET_ADDRESS_IS_GROUP (address)) {
_index_address_group (message, prefix_name, address);
} else {
INTERNAL_ERROR ("GMime InternetAddress is neither a mailbox nor a group.\n");
}
}
}
lib: abstract content type indexing Make the follow-up change of indexing signature content types easier. No functional changes.
2017-09-16 16:59:02 +02:00
static void
_index_content_type (notmuch_message_t *message, GMimeObject *part)
{
GMimeContentType *content_type = g_mime_object_get_content_type (part);
if (content_type) {
char *mime_string = g_mime_content_type_to_string (content_type);
if (mime_string) {
_notmuch_message_gen_terms (message, "mimetype", mime_string);
g_free (mime_string);
}
}
}
crypto: index encrypted parts when indexopts try_decrypt is set. If we see index options that ask us to decrypt when indexing a message, and we encounter an encrypted part, we'll try to descend into it. If we can decrypt, we add the property index.decryption=success. If we can't decrypt (or recognize the encrypted type of mail), we add the property index.decryption=failure. Note that a single message may have both values of the "index.decryption" property: "success" and "failure". For example, consider a message that includes multiple layers of encryption. If we manage to decrypt the outer layer ("index.decryption=success"), but fail on the inner layer ("index.decryption=failure"). Because of the property name, this will be automatically cleared (and possibly re-set) during re-indexing. This means it will subsequently correspond to the actual semantics of the stored index.
2017-10-21 04:25:41 +02:00
static void
_index_encrypted_mime_part (notmuch_message_t *message, notmuch_indexopts_t *indexopts,
GMimeContentType *content_type,
GMimeMultipartEncrypted *part);
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
/* Callback to generate terms for each mime part of a message. */
static void
_index_mime_part (notmuch_message_t *message,
crypto: index encrypted parts when indexopts try_decrypt is set. If we see index options that ask us to decrypt when indexing a message, and we encounter an encrypted part, we'll try to descend into it. If we can decrypt, we add the property index.decryption=success. If we can't decrypt (or recognize the encrypted type of mail), we add the property index.decryption=failure. Note that a single message may have both values of the "index.decryption" property: "success" and "failure". For example, consider a message that includes multiple layers of encryption. If we manage to decrypt the outer layer ("index.decryption=success"), but fail on the inner layer ("index.decryption=failure"). Because of the property name, this will be automatically cleared (and possibly re-set) during re-indexing. This means it will subsequently correspond to the actual semantics of the stored index.
2017-10-21 04:25:41 +02:00
notmuch_indexopts_t *indexopts,
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
GMimeObject *part)
{
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
GMimeStream *stream, *filter;
lib/index: generalize name of indexing filter In followup commits we will generalize the functionality of this filter to deal with other types of non-indexable content.
2017-06-08 04:11:46 +02:00
GMimeFilter *discard_non_term_filter;
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
GMimeDataWrapper *wrapper;
GByteArray *byte_array;
GMimeContentDisposition *disposition;
lib: abstract content type indexing Make the follow-up change of indexing signature content types easier. No functional changes.
2017-09-16 16:59:02 +02:00
GMimeContentType *content_type;
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
char *body;
Convert non-UTF-8 parts to UTF-8 before indexing them This fixes a bug that didn't allow to search for non-ASCII words such parts. The code here was copied from show_text_part_content(), because the show command already does the needed conversion when showing the message.
2012-02-24 08:36:22 +01:00
const char *charset;
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
Do not segfault on empty mime parts notmuch previously unconditionally checked mime parts for various properties, but not for NULL, which is the case if libgmime encounters an empty mime part. Upon encounter of an empty mime part, the following is printed to stderr (the second line due to my patch): (process:17197): gmime-CRITICAL **: g_mime_message_get_mime_part: assertion `GMIME_IS_MESSAGE (message)' failed Warning: Not indexing empty mime part. This is probably a bug that should get addressed in libgmime, but for not, my patch is an acceptable workaround. Signed-off-by: martin f. krafft <madduck@madduck.net>
2010-03-02 16:31:28 +01:00
if (! part) {
lib: replace almost all fprintfs in library with _n_d_log This is not supposed to change any functionality from an end user point of view. Note that it will eliminate some output to stderr. The query debugging output is left as is; it doesn't really fit with the current primitive logging model. The remaining "bad" fprintf will need an internal API change.
2014-12-26 17:25:35 +01:00
_notmuch_database_log (_notmuch_message_database (message),
"Warning: Not indexing empty mime part.\n");
Do not segfault on empty mime parts notmuch previously unconditionally checked mime parts for various properties, but not for NULL, which is the case if libgmime encounters an empty mime part. Upon encounter of an empty mime part, the following is printed to stderr (the second line due to my patch): (process:17197): gmime-CRITICAL **: g_mime_message_get_mime_part: assertion `GMIME_IS_MESSAGE (message)' failed Warning: Not indexing empty mime part. This is probably a bug that should get addressed in libgmime, but for not, my patch is an acceptable workaround. Signed-off-by: martin f. krafft <madduck@madduck.net>
2010-03-02 16:31:28 +01:00
return;
}
lib: abstract content type indexing Make the follow-up change of indexing signature content types easier. No functional changes.
2017-09-16 16:59:02 +02:00
_index_content_type (message, part);
crypto: index encrypted parts when indexopts try_decrypt is set. If we see index options that ask us to decrypt when indexing a message, and we encounter an encrypted part, we'll try to descend into it. If we can decrypt, we add the property index.decryption=success. If we can't decrypt (or recognize the encrypted type of mail), we add the property index.decryption=failure. Note that a single message may have both values of the "index.decryption" property: "success" and "failure". For example, consider a message that includes multiple layers of encryption. If we manage to decrypt the outer layer ("index.decryption=success"), but fail on the inner layer ("index.decryption=failure"). Because of the property name, this will be automatically cleared (and possibly re-set) during re-indexing. This means it will subsequently correspond to the actual semantics of the stored index.
2017-10-21 04:25:41 +02:00
content_type = g_mime_object_get_content_type (part);
Add indexing for the mimetype term This adds the indexing support for the "mimetype:" term and removes the broken test flag. The indexing is probablistic in Xapian terms, which gives a better experience to end users. Standard content-types of the form "foo/bar" are automatically interpreted as phrases in Xapian due to the embedded slash. Assume, separate messages with application/pdf and application/x-pdf are indexed, then: - mimetype:application/x-pdf will find only the application/x-pdf - mimetype:application/pdf will find only the application/pdf - mimetype:pdf will find both of the messages
2015-01-23 00:43:38 +01:00
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
if (GMIME_IS_MULTIPART (part)) {
GMimeMultipart *multipart = GMIME_MULTIPART (part);
int i;
tag signed/encrypted during notmuch new This patch adds the tag "signed" to messages with any multipart/signed parts, and the tag "encrypted" to messages with any multipart/encrypted parts. This only occurs when messages are indexed during notmuch new, so a database rebuild is required to have old messages tagged.
2011-05-26 03:01:20 +02:00
if (GMIME_IS_MULTIPART_SIGNED (multipart))
_notmuch_message_add_term (message, "tag", "signed");
if (GMIME_IS_MULTIPART_ENCRYPTED (multipart))
_notmuch_message_add_term (message, "tag", "encrypted");
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
for (i = 0; i < g_mime_multipart_get_count (multipart); i++) {
if (GMIME_IS_MULTIPART_SIGNED (multipart)) {
lib: index the content type of signature parts It's useful (*) to be able to easily find messages with certain types of signatures. Having the mimetype: prefix searches fail for some content types is also genuinely surprising (*). Index the content type of signature parts. While at it, switch to the gmime convenience constants for content and signature part indexes. *) At least for developers of email software!
2017-09-16 16:59:04 +02:00
/* Don't index the signature, but index its content type. */
if (i == GMIME_MULTIPART_SIGNED_SIGNATURE) {
_index_content_type (message,
g_mime_multipart_get_part (multipart, i));
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
continue;
lib: index the content type of signature parts It's useful (*) to be able to easily find messages with certain types of signatures. Having the mimetype: prefix searches fail for some content types is also genuinely surprising (*). Index the content type of signature parts. While at it, switch to the gmime convenience constants for content and signature part indexes. *) At least for developers of email software!
2017-09-16 16:59:04 +02:00
} else if (i != GMIME_MULTIPART_SIGNED_CONTENT) {
lib: replace almost all fprintfs in library with _n_d_log This is not supposed to change any functionality from an end user point of view. Note that it will eliminate some output to stderr. The query debugging output is left as is; it doesn't really fit with the current primitive logging model. The remaining "bad" fprintf will need an internal API change.
2014-12-26 17:25:35 +01:00
_notmuch_database_log (_notmuch_message_database (message),
lib: index the content type of signature parts It's useful (*) to be able to easily find messages with certain types of signatures. Having the mimetype: prefix searches fail for some content types is also genuinely surprising (*). Index the content type of signature parts. While at it, switch to the gmime convenience constants for content and signature part indexes. *) At least for developers of email software!
2017-09-16 16:59:04 +02:00
"Warning: Unexpected extra parts of multipart/signed. Indexing anyway.\n");
}
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
}
Ignore encrypted parts when indexing. It appears to be an oversight that encrypted parts were indexed previously. The terms generated from encrypted parts are meaningless and do nothing but add bloat to the database. It is not worth indexing the encrypted content, just as it's not worth indexing the signatures in signed parts.
2011-12-28 21:14:29 +01:00
if (GMIME_IS_MULTIPART_ENCRYPTED (multipart)) {
lib: index the content-type of the parts of encrypted messages This is a logical followup to "lib: index the content type of signature parts", which will make it easier to record the message structure of all messages.
2017-09-16 16:59:05 +02:00
_index_content_type (message,
g_mime_multipart_get_part (multipart, i));
crypto: index encrypted parts when indexopts try_decrypt is set. If we see index options that ask us to decrypt when indexing a message, and we encounter an encrypted part, we'll try to descend into it. If we can decrypt, we add the property index.decryption=success. If we can't decrypt (or recognize the encrypted type of mail), we add the property index.decryption=failure. Note that a single message may have both values of the "index.decryption" property: "success" and "failure". For example, consider a message that includes multiple layers of encryption. If we manage to decrypt the outer layer ("index.decryption=success"), but fail on the inner layer ("index.decryption=failure"). Because of the property name, this will be automatically cleared (and possibly re-set) during re-indexing. This means it will subsequently correspond to the actual semantics of the stored index.
2017-10-21 04:25:41 +02:00
if (i == GMIME_MULTIPART_ENCRYPTED_CONTENT) {
_index_encrypted_mime_part(message, indexopts,
content_type,
GMIME_MULTIPART_ENCRYPTED (part));
} else {
if (i != GMIME_MULTIPART_ENCRYPTED_VERSION) {
_notmuch_database_log (_notmuch_message_database (message),
"Warning: Unexpected extra parts of multipart/encrypted.\n");
}
lib: index the content-type of the parts of encrypted messages This is a logical followup to "lib: index the content type of signature parts", which will make it easier to record the message structure of all messages.
2017-09-16 16:59:05 +02:00
}
Ignore encrypted parts when indexing. It appears to be an oversight that encrypted parts were indexed previously. The terms generated from encrypted parts are meaningless and do nothing but add bloat to the database. It is not worth indexing the encrypted content, just as it's not worth indexing the signatures in signed parts.
2011-12-28 21:14:29 +01:00
continue;
}
crypto: index encrypted parts when indexopts try_decrypt is set. If we see index options that ask us to decrypt when indexing a message, and we encounter an encrypted part, we'll try to descend into it. If we can decrypt, we add the property index.decryption=success. If we can't decrypt (or recognize the encrypted type of mail), we add the property index.decryption=failure. Note that a single message may have both values of the "index.decryption" property: "success" and "failure". For example, consider a message that includes multiple layers of encryption. If we manage to decrypt the outer layer ("index.decryption=success"), but fail on the inner layer ("index.decryption=failure"). Because of the property name, this will be automatically cleared (and possibly re-set) during re-indexing. This means it will subsequently correspond to the actual semantics of the stored index.
2017-10-21 04:25:41 +02:00
_index_mime_part (message, indexopts,
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
g_mime_multipart_get_part (multipart, i));
}
return;
}
if (GMIME_IS_MESSAGE_PART (part)) {
GMimeMessage *mime_message;
mime_message = g_mime_message_part_get_message (GMIME_MESSAGE_PART (part));
crypto: index encrypted parts when indexopts try_decrypt is set. If we see index options that ask us to decrypt when indexing a message, and we encounter an encrypted part, we'll try to descend into it. If we can decrypt, we add the property index.decryption=success. If we can't decrypt (or recognize the encrypted type of mail), we add the property index.decryption=failure. Note that a single message may have both values of the "index.decryption" property: "success" and "failure". For example, consider a message that includes multiple layers of encryption. If we manage to decrypt the outer layer ("index.decryption=success"), but fail on the inner layer ("index.decryption=failure"). Because of the property name, this will be automatically cleared (and possibly re-set) during re-indexing. This means it will subsequently correspond to the actual semantics of the stored index.
2017-10-21 04:25:41 +02:00
_index_mime_part (message, indexopts, g_mime_message_get_mime_part (mime_message));
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
return;
}
if (! (GMIME_IS_PART (part))) {
lib: replace almost all fprintfs in library with _n_d_log This is not supposed to change any functionality from an end user point of view. Note that it will eliminate some output to stderr. The query debugging output is left as is; it doesn't really fit with the current primitive logging model. The remaining "bad" fprintf will need an internal API change.
2014-12-26 17:25:35 +01:00
_notmuch_database_log (_notmuch_message_database (message),
"Warning: Not indexing unknown mime part: %s.\n",
g_type_name (G_OBJECT_TYPE (part)));
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
return;
}
disposition = g_mime_object_get_content_disposition (part);
if (disposition &&
lib: content disposition values are not case-sensitive Per RFC 2183, the values for Content-Disposition values are not case-sensitive. While at it, use the gmime function for getting at the disposition string instead of referencing the field directly. This fixes "attachment" tagging and filename term generation for attachments while indexing.
2015-09-26 11:35:21 +02:00
strcasecmp (g_mime_content_disposition_get_disposition (disposition),
GMIME_DISPOSITION_ATTACHMENT) == 0)
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
{
const char *filename = g_mime_part_get_filename (GMIME_PART (part));
_notmuch_message_add_term (message, "tag", "attachment");
_notmuch_message_gen_terms (message, "attachment", filename);
/* XXX: Would be nice to call out to something here to parse
* the attachment into text and then index that. */
return;
}
byte_array = g_byte_array_new ();
stream = g_mime_stream_mem_new_with_byte_array (byte_array);
lib: convert notmuch_bool_t to stdbool internally C99 stdbool turned 18 this year. There really is no reason to use our own, except in the library interface for backward compatibility. Convert the lib internally to stdbool.
2017-10-07 10:44:05 +02:00
g_mime_stream_mem_set_owner (GMIME_STREAM_MEM (stream), false);
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
filter = g_mime_stream_filter_new (stream);
lib: abstract content type indexing Make the follow-up change of indexing signature content types easier. No functional changes.
2017-09-16 16:59:02 +02:00
lib/index: add simple html filter The filter just drops all (HTML) tags. As an enabling change, pass the content type to the filter constructor so we can decide which scanner to user.
2017-06-08 04:11:49 +02:00
discard_non_term_filter = notmuch_filter_discard_non_term_new (content_type);
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
g_mime_stream_filter_add (GMIME_STREAM_FILTER (filter),
lib/index: generalize name of indexing filter In followup commits we will generalize the functionality of this filter to deal with other types of non-indexable content.
2017-06-08 04:11:46 +02:00
discard_non_term_filter);
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
Convert non-UTF-8 parts to UTF-8 before indexing them This fixes a bug that didn't allow to search for non-ASCII words such parts. The code here was copied from show_text_part_content(), because the show command already does the needed conversion when showing the message.
2012-02-24 08:36:22 +01:00
charset = g_mime_object_get_content_type_parameter (part, "charset");
if (charset) {
GMimeFilter *charset_filter;
charset_filter = g_mime_filter_charset_new (charset, "UTF-8");
/* This result can be NULL for things like "unknown-8bit".
* Don't set a NULL filter as that makes GMime print
* annoying assertion-failure messages on stderr. */
if (charset_filter) {
g_mime_stream_filter_add (GMIME_STREAM_FILTER (filter),
charset_filter);
g_object_unref (charset_filter);
}
}
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
wrapper = g_mime_part_get_content_object (GMIME_PART (part));
if (wrapper)
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
g_mime_data_wrapper_write_to_stream (wrapper, filter);
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
g_object_unref (stream);
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
g_object_unref (filter);
lib/index: generalize name of indexing filter In followup commits we will generalize the functionality of this filter to deal with other types of non-indexable content.
2017-06-08 04:11:46 +02:00
g_object_unref (discard_non_term_filter);
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
g_byte_array_append (byte_array, (guint8 *) "\0", 1);
lib: convert notmuch_bool_t to stdbool internally C99 stdbool turned 18 this year. There really is no reason to use our own, except in the library interface for backward compatibility. Convert the lib internally to stdbool.
2017-10-07 10:44:05 +02:00
body = (char *) g_byte_array_free (byte_array, false);
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
Index content from citations and signatures. In the presentation we often omit citations and signatures, but this is not content that should be omitted from the index, (especially when the citation detection is wrong---see cases where a line beginning with "From" is corrupted to ">From" by mail processing tools).
2010-01-06 04:14:07 +01:00
if (body) {
_notmuch_message_gen_terms (message, NULL, body);
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
Index content from citations and signatures. In the presentation we often omit citations and signatures, but this is not content that should be omitted from the index, (especially when the citation detection is wrong---see cases where a line beginning with "From" is corrupted to ">From" by mail processing tools).
2010-01-06 04:14:07 +01:00
free (body);
}
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
}
crypto: index encrypted parts when indexopts try_decrypt is set. If we see index options that ask us to decrypt when indexing a message, and we encounter an encrypted part, we'll try to descend into it. If we can decrypt, we add the property index.decryption=success. If we can't decrypt (or recognize the encrypted type of mail), we add the property index.decryption=failure. Note that a single message may have both values of the "index.decryption" property: "success" and "failure". For example, consider a message that includes multiple layers of encryption. If we manage to decrypt the outer layer ("index.decryption=success"), but fail on the inner layer ("index.decryption=failure"). Because of the property name, this will be automatically cleared (and possibly re-set) during re-indexing. This means it will subsequently correspond to the actual semantics of the stored index.
2017-10-21 04:25:41 +02:00
/* descend (if desired) into the cleartext part of an encrypted MIME
* part while indexing. */
static void
_index_encrypted_mime_part (notmuch_message_t *message,
notmuch_indexopts_t *indexopts,
g_mime_3_unused(GMimeContentType *content_type),
GMimeMultipartEncrypted *encrypted_data)
{
notmuch_status_t status;
GError *err = NULL;
notmuch_database_t * notmuch = NULL;
GMimeObject *clear = NULL;
if (!indexopts || !notmuch_indexopts_get_try_decrypt (indexopts))
return;
notmuch = _notmuch_message_database (message);
crypto: add _notmuch_crypto_decrypt wrapper function We will use this centralized function to consolidate the awkward behavior around different gmime versions. It's only invoked from two places: mime-node.c's node_decrypt_and_verify() and lib/index.cc's _index_encrypted_mime_part(). However, those two places have some markedly distinct logic, so the interface for this _notmuch_crypto_decrypt function is going to get a little bit clunky. It's worthwhile, though, for the sake of keeping these #if directives reasonably well-contained.
2017-11-30 09:59:27 +01:00
GMimeCryptoContext* crypto_ctx = NULL;
crypto: index encrypted parts when indexopts try_decrypt is set. If we see index options that ask us to decrypt when indexing a message, and we encounter an encrypted part, we'll try to descend into it. If we can decrypt, we add the property index.decryption=success. If we can't decrypt (or recognize the encrypted type of mail), we add the property index.decryption=failure. Note that a single message may have both values of the "index.decryption" property: "success" and "failure". For example, consider a message that includes multiple layers of encryption. If we manage to decrypt the outer layer ("index.decryption=success"), but fail on the inner layer ("index.decryption=failure"). Because of the property name, this will be automatically cleared (and possibly re-set) during re-indexing. This means it will subsequently correspond to the actual semantics of the stored index.
2017-10-21 04:25:41 +02:00
#if (GMIME_MAJOR_VERSION < 3)
{
const char *protocol = NULL;
protocol = g_mime_content_type_get_parameter (content_type, "protocol");
status = _notmuch_crypto_get_gmime_ctx_for_protocol (&(indexopts->crypto),
protocol, &crypto_ctx);
if (status) {
_notmuch_database_log (notmuch, "Warning: setup failed for decrypting "
"during indexing. (%d)\n", status);
status = notmuch_message_add_property (message, "index.decryption", "failure");
if (status)
_notmuch_database_log_append (notmuch, "failed to add index.decryption "
"property (%d)\n", status);
return;
}
}
#endif
crypto: add _notmuch_crypto_decrypt wrapper function We will use this centralized function to consolidate the awkward behavior around different gmime versions. It's only invoked from two places: mime-node.c's node_decrypt_and_verify() and lib/index.cc's _index_encrypted_mime_part(). However, those two places have some markedly distinct logic, so the interface for this _notmuch_crypto_decrypt function is going to get a little bit clunky. It's worthwhile, though, for the sake of keeping these #if directives reasonably well-contained.
2017-11-30 09:59:27 +01:00
clear = _notmuch_crypto_decrypt (crypto_ctx, encrypted_data, NULL, &err);
crypto: index encrypted parts when indexopts try_decrypt is set. If we see index options that ask us to decrypt when indexing a message, and we encounter an encrypted part, we'll try to descend into it. If we can decrypt, we add the property index.decryption=success. If we can't decrypt (or recognize the encrypted type of mail), we add the property index.decryption=failure. Note that a single message may have both values of the "index.decryption" property: "success" and "failure". For example, consider a message that includes multiple layers of encryption. If we manage to decrypt the outer layer ("index.decryption=success"), but fail on the inner layer ("index.decryption=failure"). Because of the property name, this will be automatically cleared (and possibly re-set) during re-indexing. This means it will subsequently correspond to the actual semantics of the stored index.
2017-10-21 04:25:41 +02:00
if (err) {
_notmuch_database_log (notmuch, "Failed to decrypt during indexing. (%d:%d) [%s]\n",
err->domain, err->code, err->message);
g_error_free(err);
/* Indicate that we failed to decrypt during indexing */
status = notmuch_message_add_property (message, "index.decryption", "failure");
if (status)
_notmuch_database_log_append (notmuch, "failed to add index.decryption "
"property (%d)\n", status);
return;
}
_index_mime_part (message, indexopts, clear);
g_object_unref (clear);
status = notmuch_message_add_property (message, "index.decryption", "success");
if (status)
_notmuch_database_log (notmuch, "failed to add index.decryption "
"property (%d)\n", status);
}
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
notmuch_status_t
_notmuch_message_index_file (notmuch_message_t *message,
crypto: index encrypted parts when indexopts try_decrypt is set. If we see index options that ask us to decrypt when indexing a message, and we encounter an encrypted part, we'll try to descend into it. If we can decrypt, we add the property index.decryption=success. If we can't decrypt (or recognize the encrypted type of mail), we add the property index.decryption=failure. Note that a single message may have both values of the "index.decryption" property: "success" and "failure". For example, consider a message that includes multiple layers of encryption. If we manage to decrypt the outer layer ("index.decryption=success"), but fail on the inner layer ("index.decryption=failure"). Because of the property name, this will be automatically cleared (and possibly re-set) during re-indexing. This means it will subsequently correspond to the actual semantics of the stored index.
2017-10-21 04:25:41 +02:00
notmuch_indexopts_t *indexopts,
lib: replace the header parser with gmime The notmuch library includes a full blown message header parser. Yet the same message headers are parsed by gmime during indexing. Switch to gmime parsing completely. These are the main changes: * Gmime stops header parsing at the first invalid header, and presumes the message body starts from there. The current parser is quite liberal in accepting broken headers. The change means we will be much pickier about accepting invalid messages. * The current parser converts tabs used in header folding to spaces. Gmime preserve the tabs. Due to a broken python library used in mailman, there are plenty of mailing lists that produce headers with tabs in header folding, and we'll see plenty of tabs. (This change has been mitigated in preparatory patches.) * For pure header parsing, the current parser is likely faster than gmime, which parses the whole message rather than just the headers. Since we parse the message and its headers using gmime for indexing anyway, this avoids and extra header parsing round when adding new messages. In case of duplicate messages, we'll end up parsing the full message although just headers would be sufficient. All in all this should still speed up 'notmuch new'. * Calls to notmuch_message_get_header() may be slightly slower than previously for headers that are not indexed in the database, due to parsing of the whole message. Within the notmuch code base, notmuch reply is the only such user.
2014-03-30 23:21:49 +02:00
notmuch_message_file_t *message_file)
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
{
lib: replace the header parser with gmime The notmuch library includes a full blown message header parser. Yet the same message headers are parsed by gmime during indexing. Switch to gmime parsing completely. These are the main changes: * Gmime stops header parsing at the first invalid header, and presumes the message body starts from there. The current parser is quite liberal in accepting broken headers. The change means we will be much pickier about accepting invalid messages. * The current parser converts tabs used in header folding to spaces. Gmime preserve the tabs. Due to a broken python library used in mailman, there are plenty of mailing lists that produce headers with tabs in header folding, and we'll see plenty of tabs. (This change has been mitigated in preparatory patches.) * For pure header parsing, the current parser is likely faster than gmime, which parses the whole message rather than just the headers. Since we parse the message and its headers using gmime for indexing anyway, this avoids and extra header parsing round when adding new messages. In case of duplicate messages, we'll end up parsing the full message although just headers would be sufficient. All in all this should still speed up 'notmuch new'. * Calls to notmuch_message_get_header() may be slightly slower than previously for headers that are not indexed in the database, due to parsing of the whole message. Within the notmuch code base, notmuch reply is the only such user.
2014-03-30 23:21:49 +02:00
GMimeMessage *mime_message;
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
InternetAddressList *addresses;
lib/cli: replace use of g_mime_message_get_sender This function changes semantics in gmime-3.0 so make a new function that provides the same functionality in both
2017-05-06 04:26:57 +02:00
const char *subject;
lib: replace the header parser with gmime The notmuch library includes a full blown message header parser. Yet the same message headers are parsed by gmime during indexing. Switch to gmime parsing completely. These are the main changes: * Gmime stops header parsing at the first invalid header, and presumes the message body starts from there. The current parser is quite liberal in accepting broken headers. The change means we will be much pickier about accepting invalid messages. * The current parser converts tabs used in header folding to spaces. Gmime preserve the tabs. Due to a broken python library used in mailman, there are plenty of mailing lists that produce headers with tabs in header folding, and we'll see plenty of tabs. (This change has been mitigated in preparatory patches.) * For pure header parsing, the current parser is likely faster than gmime, which parses the whole message rather than just the headers. Since we parse the message and its headers using gmime for indexing anyway, this avoids and extra header parsing round when adding new messages. In case of duplicate messages, we'll end up parsing the full message although just headers would be sufficient. All in all this should still speed up 'notmuch new'. * Calls to notmuch_message_get_header() may be slightly slower than previously for headers that are not indexed in the database, due to parsing of the whole message. Within the notmuch code base, notmuch reply is the only such user.
2014-03-30 23:21:49 +02:00
notmuch_status_t status;
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
lib: replace the header parser with gmime The notmuch library includes a full blown message header parser. Yet the same message headers are parsed by gmime during indexing. Switch to gmime parsing completely. These are the main changes: * Gmime stops header parsing at the first invalid header, and presumes the message body starts from there. The current parser is quite liberal in accepting broken headers. The change means we will be much pickier about accepting invalid messages. * The current parser converts tabs used in header folding to spaces. Gmime preserve the tabs. Due to a broken python library used in mailman, there are plenty of mailing lists that produce headers with tabs in header folding, and we'll see plenty of tabs. (This change has been mitigated in preparatory patches.) * For pure header parsing, the current parser is likely faster than gmime, which parses the whole message rather than just the headers. Since we parse the message and its headers using gmime for indexing anyway, this avoids and extra header parsing round when adding new messages. In case of duplicate messages, we'll end up parsing the full message although just headers would be sufficient. All in all this should still speed up 'notmuch new'. * Calls to notmuch_message_get_header() may be slightly slower than previously for headers that are not indexed in the database, due to parsing of the whole message. Within the notmuch code base, notmuch reply is the only such user.
2014-03-30 23:21:49 +02:00
status = _notmuch_message_file_get_mime_message (message_file,
&mime_message);
if (status)
return status;
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
lib/cli: replace use of g_mime_message_get_sender This function changes semantics in gmime-3.0 so make a new function that provides the same functionality in both
2017-05-06 04:26:57 +02:00
addresses = g_mime_message_get_from (mime_message);
_notmuch_message_index_file: unref (free) address lists from gmime. Apparently as of GMime 2.4, you don't need to call internet_address_list_destroy anymore, but you still need to call g_object_unref (from the GMime Changelog). On the medium performance corpus, valgrind shows "possibly lost" leakage in "notmuch new" dropping from 7M to 300k.
2012-12-11 04:33:40 +01:00
if (addresses) {
_index_address_list (message, "from", addresses);
lib: paper over allocation difference In gmime 3.0 this function is "transfer none", so no deallocation is needed (or permitted)
2017-05-20 14:05:22 +02:00
g_mime_2_6_unref (addresses);
_notmuch_message_index_file: unref (free) address lists from gmime. Apparently as of GMime 2.4, you don't need to call internet_address_list_destroy anymore, but you still need to call g_object_unref (from the GMime Changelog). On the medium performance corpus, valgrind shows "possibly lost" leakage in "notmuch new" dropping from 7M to 300k.
2012-12-11 04:33:40 +01:00
}
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
addresses = g_mime_message_get_all_recipients (mime_message);
_notmuch_message_index_file: unref (free) address lists from gmime. Apparently as of GMime 2.4, you don't need to call internet_address_list_destroy anymore, but you still need to call g_object_unref (from the GMime Changelog). On the medium performance corpus, valgrind shows "possibly lost" leakage in "notmuch new" dropping from 7M to 300k.
2012-12-11 04:33:40 +01:00
if (addresses) {
_index_address_list (message, "to", addresses);
g_object_unref (addresses);
}
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
subject = g_mime_message_get_subject (mime_message);
_notmuch_message_gen_terms (message, "subject", subject);
crypto: index encrypted parts when indexopts try_decrypt is set. If we see index options that ask us to decrypt when indexing a message, and we encounter an encrypted part, we'll try to descend into it. If we can decrypt, we add the property index.decryption=success. If we can't decrypt (or recognize the encrypted type of mail), we add the property index.decryption=failure. Note that a single message may have both values of the "index.decryption" property: "success" and "failure". For example, consider a message that includes multiple layers of encryption. If we manage to decrypt the outer layer ("index.decryption=success"), but fail on the inner layer ("index.decryption=failure"). Because of the property name, this will be automatically cleared (and possibly re-set) during re-indexing. This means it will subsequently correspond to the actual semantics of the stored index.
2017-10-21 04:25:41 +02:00
_index_mime_part (message, indexopts, g_mime_message_get_mime_part (mime_message));
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
lib: replace the header parser with gmime The notmuch library includes a full blown message header parser. Yet the same message headers are parsed by gmime during indexing. Switch to gmime parsing completely. These are the main changes: * Gmime stops header parsing at the first invalid header, and presumes the message body starts from there. The current parser is quite liberal in accepting broken headers. The change means we will be much pickier about accepting invalid messages. * The current parser converts tabs used in header folding to spaces. Gmime preserve the tabs. Due to a broken python library used in mailman, there are plenty of mailing lists that produce headers with tabs in header folding, and we'll see plenty of tabs. (This change has been mitigated in preparatory patches.) * For pure header parsing, the current parser is likely faster than gmime, which parses the whole message rather than just the headers. Since we parse the message and its headers using gmime for indexing anyway, this avoids and extra header parsing round when adding new messages. In case of duplicate messages, we'll end up parsing the full message although just headers would be sufficient. All in all this should still speed up 'notmuch new'. * Calls to notmuch_message_get_header() may be slightly slower than previously for headers that are not indexed in the database, due to parsing of the whole message. Within the notmuch code base, notmuch reply is the only such user.
2014-03-30 23:21:49 +02:00
return NOTMUCH_STATUS_SUCCESS;
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
}
Reference in a new issue Copy permalink