debian/notmuch
1
0
Fork 0
mirror of https://git.notmuchmail.org/git/notmuch synced 2024-11-24 11:58:10 +01:00
Code Issues Projects Releases Packages Wiki Activity Actions
notmuch/lib/index.cc

771 lines
24 KiB
C++
Raw Permalink Normal View History

Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
/*
* Copyright © 2009 Carl Worth
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
Use https instead of http where possible Many of the external links found in the notmuch source can be resolved using https instead of http. This changeset addresses as many as i could find, without touching the e-mail corpus or expected outputs found in tests.
2016-06-02 18:26:14 +02:00
* along with this program. If not, see https://www.gnu.org/licenses/ .
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
*
* Author: Carl Worth <cworth@cworth.org>
*/
#include "notmuch-private.h"
#include <gmime/gmime.h>
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
#include <gmime/gmime-filter.h>
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
#include <xapian.h>
lib/index: separate state table definition from scanner. We want to reuse the scanner definition with a different table. This is mainly code movement, and making the state table part of the filter struct/class.
2017-06-08 04:11:47 +02:00
typedef struct {
int state;
int a;
int b;
int next_if_match;
int next_if_not_match;
} scanner_state_t;
/* Simple, linear state-transition diagram for the uuencode filter.
*
* If the character being processed is within the range of [a, b]
* for the current state then we transition next_if_match
* state. If not, we transition to the next_if_not_match state.
*
* The final two states are special in that they are the states in
* which we discard data. */
static const int first_uuencode_skipping_state = 11;
static const scanner_state_t uuencode_states[] = {
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
{ 0, 'b', 'b', 1, 0 },
{ 1, 'e', 'e', 2, 0 },
{ 2, 'g', 'g', 3, 0 },
{ 3, 'i', 'i', 4, 0 },
{ 4, 'n', 'n', 5, 0 },
{ 5, ' ', ' ', 6, 0 },
{ 6, '0', '7', 7, 0 },
{ 7, '0', '7', 8, 0 },
{ 8, '0', '7', 9, 0 },
{ 9, ' ', ' ', 10, 0 },
{ 10, '\n', '\n', 11, 10 },
{ 11, 'M', 'M', 12, 0 },
{ 12, ' ', '`', 12, 11 }
lib/index: separate state table definition from scanner. We want to reuse the scanner definition with a different table. This is mainly code movement, and making the state table part of the filter struct/class.
2017-06-08 04:11:47 +02:00
};
lib/index: add simple html filter The filter just drops all (HTML) tags. As an enabling change, pass the content type to the filter constructor so we can decide which scanner to user.
2017-06-08 04:11:49 +02:00
/* The following table is intended to implement this DFA (in 'dot'
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
* format). Note that 2 and 3 are "hidden" states used to step through
* the possible out edges of state 1.
*
* digraph html_filter {
* 0 -> 1 [label="<"];
* 0 -> 0;
* 1 -> 4 [label="'"];
* 1 -> 5 [label="\""];
* 1 -> 0 [label=">"];
* 1 -> 1;
* 4 -> 1 [label="'"];
* 4 -> 4;
* 5 -> 1 [label="\""];
* 5 -> 5;
* }
*/
lib/index: add simple html filter The filter just drops all (HTML) tags. As an enabling change, pass the content type to the filter constructor so we can decide which scanner to user.
2017-06-08 04:11:49 +02:00
static const int first_html_skipping_state = 1;
static const scanner_state_t html_states[] = {
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
{ 0, '<', '<', 1, 0 },
{ 1, '\'', '\'', 4, 2 }, /* scanning for quote or > */
{ 1, '"', '"', 5, 3 },
{ 1, '>', '>', 0, 1 },
{ 4, '\'', '\'', 1, 4 }, /* inside single quotes */
{ 5, '"', '"', 1, 5 }, /* inside double quotes */
lib/index: add simple html filter The filter just drops all (HTML) tags. As an enabling change, pass the content type to the filter constructor so we can decide which scanner to user.
2017-06-08 04:11:49 +02:00
};
Eliminate some useless gobject boilerplate. If we had external users of this filter then they might expect some of these macros to exist. But since this is just internal, that's just unneeded noise.
2010-02-05 02:26:00 +01:00
/* Oh, how I wish that gobject didn't require so much noisy boilerplate!
* (Though I have at least eliminated some of the stock set...) */
lib/index: generalize name of indexing filter In followup commits we will generalize the functionality of this filter to deal with other types of non-indexable content.
2017-06-08 04:11:46 +02:00
typedef struct _NotmuchFilterDiscardNonTerm NotmuchFilterDiscardNonTerm;
typedef struct _NotmuchFilterDiscardNonTermClass NotmuchFilterDiscardNonTermClass;
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
/**
lib/index: generalize name of indexing filter In followup commits we will generalize the functionality of this filter to deal with other types of non-indexable content.
2017-06-08 04:11:46 +02:00
* NotmuchFilterDiscardNonTerm:
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
*
* @parent_object: parent #GMimeFilter
* @encode: encoding vs decoding
* @state: State of the parser
*
* A filter to discard uuencoded portions of an email.
*
* A uuencoded portion is identified as beginning with a line
* matching:
*
* begin [0-7][0-7][0-7] .*
*
* After that detection, and beginning with the following line,
* characters will be discarded as long as the first character of each
* line begins with M and subsequent characters on the line are within
* the range of ASCII characters from ' ' to '`'.
*
* This is not a perfect UUencode filter. It's possible to have a
* message that will legitimately match that pattern, (so that some
* legitimate content is discarded). And for most UUencoded files, the
* final line of encoded data (the line not starting with M) will be
* indexed.
**/
lib/index: generalize name of indexing filter In followup commits we will generalize the functionality of this filter to deal with other types of non-indexable content.
2017-06-08 04:11:46 +02:00
struct _NotmuchFilterDiscardNonTerm {
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
GMimeFilter parent_object;
lib/index: add simple html filter The filter just drops all (HTML) tags. As an enabling change, pass the content type to the filter constructor so we can decide which scanner to user.
2017-06-08 04:11:49 +02:00
GMimeContentType *content_type;
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
int state;
lib/index: separate state table definition from scanner. We want to reuse the scanner definition with a different table. This is mainly code movement, and making the state table part of the filter struct/class.
2017-06-08 04:11:47 +02:00
int first_skipping_state;
const scanner_state_t *states;
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
};
lib/index: generalize name of indexing filter In followup commits we will generalize the functionality of this filter to deal with other types of non-indexable content.
2017-06-08 04:11:46 +02:00
struct _NotmuchFilterDiscardNonTermClass {
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
GMimeFilterClass parent_class;
};
lib/index: add simple html filter The filter just drops all (HTML) tags. As an enabling change, pass the content type to the filter constructor so we can decide which scanner to user.
2017-06-08 04:11:49 +02:00
static GMimeFilter *notmuch_filter_discard_non_term_new (GMimeContentType *content);
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
lib/index: generalize name of indexing filter In followup commits we will generalize the functionality of this filter to deal with other types of non-indexable content.
2017-06-08 04:11:46 +02:00
static void notmuch_filter_discard_non_term_finalize (GObject *object);
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
static GMimeFilter *filter_copy (GMimeFilter *filter);
static void filter_filter (GMimeFilter *filter, char *in, size_t len, size_t prespace,
char **out, size_t *outlen, size_t *outprespace);
static void filter_complete (GMimeFilter *filter, char *in, size_t len, size_t prespace,
char **out, size_t *outlen, size_t *outprespace);
static void filter_reset (GMimeFilter *filter);
static GMimeFilterClass *parent_class = NULL;
static void
index: explicitly follow GObject conventions Use explicit labels for GTypeInfo member initializers, rather than relying on comments and ordering. This is both easier to read, and harder to screw up. This also makes it clear that we're mis-casting GObject class initializers for gcc. Without this patch, g++ 8.2.0-7 produces this warning: CXX -g -O2 lib/index.o lib/index.cc: In function ‘GMimeFilter* notmuch_filter_discard_non_term_new(GMimeContentType*)’: lib/index.cc:252:23: warning: cast between incompatible function types from ‘void (*)(NotmuchFilterDiscardNonTermClass*)’ {aka ‘void (*)(_NotmuchFilterDiscardNonTermClass*)’} to ‘GClassInitFunc’ {aka ‘void (*)(void*, void*)’} [-Wcast-function-type] (GClassInitFunc) notmuch_filter_discard_non_term_class_init, ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The definition of GClassInitFunc in /usr/include/glib-2.0/gobject/gtype.h suggests that this function will always be called with the class_data member of the GTypeInfo. We set that value to NULL in both GObject definitions in notmuch. So we mark it as explicitly unused. There is no functional change here, just code cleanup.
2018-10-07 03:37:57 +02:00
notmuch_filter_discard_non_term_class_init (NotmuchFilterDiscardNonTermClass *klass,
unused (void *class_data))
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
{
GObjectClass *object_class = G_OBJECT_CLASS (klass);
GMimeFilterClass *filter_class = GMIME_FILTER_CLASS (klass);
lib/index: generalize name of indexing filter In followup commits we will generalize the functionality of this filter to deal with other types of non-indexable content.
2017-06-08 04:11:46 +02:00
object_class->finalize = notmuch_filter_discard_non_term_finalize;
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
filter_class->copy = filter_copy;
filter_class->filter = filter_filter;
filter_class->complete = filter_complete;
filter_class->reset = filter_reset;
}
static void
lib/index: generalize name of indexing filter In followup commits we will generalize the functionality of this filter to deal with other types of non-indexable content.
2017-06-08 04:11:46 +02:00
notmuch_filter_discard_non_term_finalize (GObject *object)
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
{
G_OBJECT_CLASS (parent_class)->finalize (object);
}
static GMimeFilter *
filter_copy (GMimeFilter *gmime_filter)
{
lib/index: add simple html filter The filter just drops all (HTML) tags. As an enabling change, pass the content type to the filter constructor so we can decide which scanner to user.
2017-06-08 04:11:49 +02:00
NotmuchFilterDiscardNonTerm *filter = (NotmuchFilterDiscardNonTerm *) gmime_filter;
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
lib/index: add simple html filter The filter just drops all (HTML) tags. As an enabling change, pass the content type to the filter constructor so we can decide which scanner to user.
2017-06-08 04:11:49 +02:00
return notmuch_filter_discard_non_term_new (filter->content_type);
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
}
static void
filter_filter (GMimeFilter *gmime_filter, char *inbuf, size_t inlen, size_t prespace,
char **outbuf, size_t *outlen, size_t *outprespace)
{
lib/index: generalize name of indexing filter In followup commits we will generalize the functionality of this filter to deal with other types of non-indexable content.
2017-06-08 04:11:46 +02:00
NotmuchFilterDiscardNonTerm *filter = (NotmuchFilterDiscardNonTerm *) gmime_filter;
lib/index: separate state table definition from scanner. We want to reuse the scanner definition with a different table. This is mainly code movement, and making the state table part of the filter struct/class.
2017-06-08 04:11:47 +02:00
const scanner_state_t *states = filter->states;
drop use of register keyword The performance benefits are dubious, and it's deprecated in C++11.
2018-05-13 18:17:17 +02:00
const char *inptr = inbuf;
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
const char *inend = inbuf + inlen;
char *outptr;
(void) prespace;
int next;
lib: convert notmuch_bool_t to stdbool internally C99 stdbool turned 18 this year. There really is no reason to use our own, except in the library interface for backward compatibility. Convert the lib internally to stdbool.
2017-10-07 10:44:05 +02:00
g_mime_filter_set_size (gmime_filter, inlen, false);
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
outptr = gmime_filter->outbuf;
lib/index.cc: generalize filter state machine To match things more complicated than fixed strings, we need states with multiple out arrows.
2017-06-08 04:11:48 +02:00
next = filter->state;
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
while (inptr < inend) {
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
/* Each state is defined by a contiguous set of rows of the
lib/index.cc: generalize filter state machine To match things more complicated than fixed strings, we need states with multiple out arrows.
2017-06-08 04:11:48 +02:00
* state table marked by a common value for '.state'. The
* state numbers must be equal to the index of the first row
* in a given state; thus the loop condition here looks for a
* jump to a first row of a state, which is a real transition
* in the underlying DFA.
*/
do {
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
if (*inptr >= states[next].a && *inptr <= states[next].b) {
lib/index.cc: generalize filter state machine To match things more complicated than fixed strings, we need states with multiple out arrows.
2017-06-08 04:11:48 +02:00
next = states[next].next_if_match;
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
} else {
lib/index.cc: generalize filter state machine To match things more complicated than fixed strings, we need states with multiple out arrows.
2017-06-08 04:11:48 +02:00
next = states[next].next_if_not_match;
}
} while (next != states[next].state);
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
lib/index: separate state table definition from scanner. We want to reuse the scanner definition with a different table. This is mainly code movement, and making the state table part of the filter struct/class.
2017-06-08 04:11:47 +02:00
if (filter->state < filter->first_skipping_state)
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
*outptr++ = *inptr;
filter->state = next;
inptr++;
}
*outlen = outptr - gmime_filter->outbuf;
*outprespace = gmime_filter->outpre;
*outbuf = gmime_filter->outbuf;
}
static void
filter_complete (GMimeFilter *filter, char *inbuf, size_t inlen, size_t prespace,
char **outbuf, size_t *outlen, size_t *outprespace)
{
if (inbuf && inlen)
filter_filter (filter, inbuf, inlen, prespace, outbuf, outlen, outprespace);
}
static void
filter_reset (GMimeFilter *gmime_filter)
{
lib/index: generalize name of indexing filter In followup commits we will generalize the functionality of this filter to deal with other types of non-indexable content.
2017-06-08 04:11:46 +02:00
NotmuchFilterDiscardNonTerm *filter = (NotmuchFilterDiscardNonTerm *) gmime_filter;
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
filter->state = 0;
}
/**
lib/index: generalize name of indexing filter In followup commits we will generalize the functionality of this filter to deal with other types of non-indexable content.
2017-06-08 04:11:46 +02:00
* notmuch_filter_discard_non_term_new:
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
*
lib/index: generalize name of indexing filter In followup commits we will generalize the functionality of this filter to deal with other types of non-indexable content.
2017-06-08 04:11:46 +02:00
* Returns: a new #NotmuchFilterDiscardNonTerm filter.
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
**/
lib: make glib initialization thread-safe In principle this could be done without depending on C++11 features, but these features should be available since gcc 4.8.1, and this localized usage is easy to replace if it turns out to be problematic for portability.
2021-05-09 22:33:48 +02:00
static GType type = 0;
static const GTypeInfo info = {
.class_size = sizeof (NotmuchFilterDiscardNonTermClass),
.base_init = NULL,
.base_finalize = NULL,
.class_init = (GClassInitFunc) notmuch_filter_discard_non_term_class_init,
.class_finalize = NULL,
.class_data = NULL,
.instance_size = sizeof (NotmuchFilterDiscardNonTerm),
.n_preallocs = 0,
.instance_init = NULL,
.value_table = NULL,
};
void
_notmuch_filter_init () {
type = g_type_register_static (GMIME_TYPE_FILTER, "NotmuchFilterDiscardNonTerm", &info,
(GTypeFlags) 0);
parent_class = (GMimeFilterClass *) g_type_class_ref (GMIME_TYPE_FILTER);
}
lib: Add some missing static qualifiers. These various functions and data are all used only locally, so should be marked static. Ensuring we get these right will avoid us accidentally leaking unintended symbols through the library interface.
2010-11-02 05:58:43 +01:00
static GMimeFilter *
lib/index: add simple html filter The filter just drops all (HTML) tags. As an enabling change, pass the content type to the filter constructor so we can decide which scanner to user.
2017-06-08 04:11:49 +02:00
notmuch_filter_discard_non_term_new (GMimeContentType *content_type)
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
{
lib/index: generalize name of indexing filter In followup commits we will generalize the functionality of this filter to deal with other types of non-indexable content.
2017-06-08 04:11:46 +02:00
NotmuchFilterDiscardNonTerm *filter;
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
lib&cli: use g_object_new instead of g_object_newv 'g_object_newv' is deprecated, and prints annoying warnings. The warnings suggest using 'g_object_new_with_properties', but that's only available since glib 2.55 (i.e. a month ago as of this writing). Since we don't actuall pass any properties, it seems we can just call 'g_object_new'.
2017-09-03 13:55:42 +02:00
filter = (NotmuchFilterDiscardNonTerm *) g_object_new (type, NULL);
lib/index: add simple html filter The filter just drops all (HTML) tags. As an enabling change, pass the content type to the filter constructor so we can decide which scanner to user.
2017-06-08 04:11:49 +02:00
filter->content_type = content_type;
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
filter->state = 0;
lib/index: add simple html filter The filter just drops all (HTML) tags. As an enabling change, pass the content type to the filter constructor so we can decide which scanner to user.
2017-06-08 04:11:49 +02:00
if (g_mime_content_type_is_type (content_type, "text", "html")) {
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
filter->states = html_states;
filter->first_skipping_state = first_html_skipping_state;
lib/index: add simple html filter The filter just drops all (HTML) tags. As an enabling change, pass the content type to the filter constructor so we can decide which scanner to user.
2017-06-08 04:11:49 +02:00
} else {
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
filter->states = uuencode_states;
filter->first_skipping_state = first_uuencode_skipping_state;
lib/index: add simple html filter The filter just drops all (HTML) tags. As an enabling change, pass the content type to the filter constructor so we can decide which scanner to user.
2017-06-08 04:11:49 +02:00
}
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
return (GMimeFilter *) filter;
}
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
/* We're finally down to a single (NAME + address) email "mailbox". */
static void
_index_address_mailbox (notmuch_message_t *message,
const char *prefix_name,
InternetAddress *address)
{
InternetAddressMailbox *mailbox = INTERNET_ADDRESS_MAILBOX (address);
lib: Index name and address of from/to headers as a phrase Previously, we indexed the name and address parts of from/to headers with two calls to _notmuch_message_gen_terms. In general, this indicates that these parts are separate phrases. However, because of an implementation quirk, the two calls to _notmuch_message_gen_terms generated adjacent term positions for the prefixed terms, which happens to be the right thing to do in this case, but the wrong thing to do for all other calls. Furthermore, _notmuch_message_gen_terms produced potentially overlapping term positions for the un-prefixed copies of the terms, which is simply wrong. This change indexes both the name and address in a single call to _notmuch_message_gen_terms, indicating that they should be part of a single phrase. This masks the problem with the un-prefixed terms (fixing the two known-broken tests) and puts us in a position to fix the unintentionally phrases generated by other calls to _notmuch_message_gen_terms.
2014-06-16 04:40:32 +02:00
const char *name, *addr, *combined;
database: Store mail filename as a new 'direntry' term, not as 'data'. Instead of storing the complete message filename in the data portion of a mail document we now store a 'direntry' term that contains the document ID of a directory document and also the basename of the message filename within that directory. This will allow us to easily store multple filenames for a single message, and will also allow us to find mail documents for files that previously existed in a directory but that have since been deleted.
2009-12-21 17:23:26 +01:00
void *local = talloc_new (message);
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
name = internet_address_get_name (address);
index: Store "Full Name <user@example.com>" addressses in the database We put these is as a separate term so that they can be extracted. We don't actually need this for searching, since typing an email address in as a search term will already trigger a phrase search that does exactly what's wanted.
2009-10-28 21:09:08 +01:00
addr = internet_address_mailbox_get_addr (mailbox);
lib: Index name and address of from/to headers as a phrase Previously, we indexed the name and address parts of from/to headers with two calls to _notmuch_message_gen_terms. In general, this indicates that these parts are separate phrases. However, because of an implementation quirk, the two calls to _notmuch_message_gen_terms generated adjacent term positions for the prefixed terms, which happens to be the right thing to do in this case, but the wrong thing to do for all other calls. Furthermore, _notmuch_message_gen_terms produced potentially overlapping term positions for the un-prefixed copies of the terms, which is simply wrong. This change indexes both the name and address in a single call to _notmuch_message_gen_terms, indicating that they should be part of a single phrase. This masks the problem with the un-prefixed terms (fixing the two known-broken tests) and puts us in a position to fix the unintentionally phrases generated by other calls to _notmuch_message_gen_terms.
2014-06-16 04:40:32 +02:00
/* Combine the name and address and index them as a phrase. */
if (name && addr)
combined = talloc_asprintf (local, "%s %s", name, addr);
else if (name)
combined = name;
else
combined = addr;
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
lib: Index name and address of from/to headers as a phrase Previously, we indexed the name and address parts of from/to headers with two calls to _notmuch_message_gen_terms. In general, this indicates that these parts are separate phrases. However, because of an implementation quirk, the two calls to _notmuch_message_gen_terms generated adjacent term positions for the prefixed terms, which happens to be the right thing to do in this case, but the wrong thing to do for all other calls. Furthermore, _notmuch_message_gen_terms produced potentially overlapping term positions for the un-prefixed copies of the terms, which is simply wrong. This change indexes both the name and address in a single call to _notmuch_message_gen_terms, indicating that they should be part of a single phrase. This masks the problem with the un-prefixed terms (fixing the two known-broken tests) and puts us in a position to fix the unintentionally phrases generated by other calls to _notmuch_message_gen_terms.
2014-06-16 04:40:32 +02:00
if (combined)
_notmuch_message_gen_terms (message, prefix_name, combined);
lib/index: Fix memory leak for email addresses without names. We carefully noted the fact that we had locally allocated the string here, but then we neglected to free it. Switch to talloc instead which makes it easier to get the behavior we want. It's simpler since we can just call talloc_free unconditionally, without having to track the state of whether we allocated the storage for name or not.
2009-12-01 21:40:13 +01:00
talloc_free (local);
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
}
static void
_index_address_list (notmuch_message_t *message,
const char *prefix_name,
InternetAddressList *addresses);
/* The outer loop over the InternetAddressList wasn't quite enough.
* There can actually be a tree here where a single member of the list
* is a "group" containing another list. Recurse please.
*/
static void
_index_address_group (notmuch_message_t *message,
const char *prefix_name,
InternetAddress *address)
{
InternetAddressGroup *group;
InternetAddressList *list;
group = INTERNET_ADDRESS_GROUP (address);
list = internet_address_group_get_members (group);
if (! list)
return;
_index_address_list (message, prefix_name, list);
}
static void
_index_address_list (notmuch_message_t *message,
const char *prefix_name,
InternetAddressList *addresses)
{
int i;
InternetAddress *address;
if (addresses == NULL)
return;
for (i = 0; i < internet_address_list_length (addresses); i++) {
address = internet_address_list_get_address (addresses, i);
if (INTERNET_ADDRESS_IS_MAILBOX (address)) {
_index_address_mailbox (message, prefix_name, address);
} else if (INTERNET_ADDRESS_IS_GROUP (address)) {
_index_address_group (message, prefix_name, address);
} else {
INTERNAL_ERROR ("GMime InternetAddress is neither a mailbox nor a group.\n");
}
}
}
lib: abstract content type indexing Make the follow-up change of indexing signature content types easier. No functional changes.
2017-09-16 16:59:02 +02:00
static void
_index_content_type (notmuch_message_t *message, GMimeObject *part)
{
GMimeContentType *content_type = g_mime_object_get_content_type (part);
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
lib: abstract content type indexing Make the follow-up change of indexing signature content types easier. No functional changes.
2017-09-16 16:59:02 +02:00
if (content_type) {
gmime-cleanup: use GMime 3.0 function names Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-02 15:19:44 +02:00
char *mime_string = g_mime_content_type_get_mime_type (content_type);
lib: abstract content type indexing Make the follow-up change of indexing signature content types easier. No functional changes.
2017-09-16 16:59:02 +02:00
if (mime_string) {
_notmuch_message_gen_terms (message, "mimetype", mime_string);
g_free (mime_string);
}
}
}
crypto: index encrypted parts when indexopts try_decrypt is set. If we see index options that ask us to decrypt when indexing a message, and we encounter an encrypted part, we'll try to descend into it. If we can decrypt, we add the property index.decryption=success. If we can't decrypt (or recognize the encrypted type of mail), we add the property index.decryption=failure. Note that a single message may have both values of the "index.decryption" property: "success" and "failure". For example, consider a message that includes multiple layers of encryption. If we manage to decrypt the outer layer ("index.decryption=success"), but fail on the inner layer ("index.decryption=failure"). Because of the property name, this will be automatically cleared (and possibly re-set) during re-indexing. This means it will subsequently correspond to the actual semantics of the stored index.
2017-10-21 04:25:41 +02:00
static void
_index_encrypted_mime_part (notmuch_message_t *message, notmuch_indexopts_t *indexopts,
crypto: Make _notmuch_crypto_decrypt take a GMimeObject As we prepare to handle S/MIME-encrypted PKCS#7 EnvelopedData (which is not multipart), we don't want to be limited to passing only GMimeMultipartEncrypted MIME parts to _notmuch_crypto_decrypt. There is no functional change here, just a matter of adjusting how we pass arguments internally. Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-05-13 00:29:36 +02:00
GMimeObject *part,
indexing: record protected subject when indexing cleartext When indexing the cleartext of an encrypted message, record any protected subject in the database, which should make it findable and visible in search. Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-28 00:40:28 +02:00
_notmuch_message_crypto_t *msg_crypto);
crypto: index encrypted parts when indexopts try_decrypt is set. If we see index options that ask us to decrypt when indexing a message, and we encounter an encrypted part, we'll try to descend into it. If we can decrypt, we add the property index.decryption=success. If we can't decrypt (or recognize the encrypted type of mail), we add the property index.decryption=failure. Note that a single message may have both values of the "index.decryption" property: "success" and "failure". For example, consider a message that includes multiple layers of encryption. If we manage to decrypt the outer layer ("index.decryption=success"), but fail on the inner layer ("index.decryption=failure"). Because of the property name, this will be automatically cleared (and possibly re-set) during re-indexing. This means it will subsequently correspond to the actual semantics of the stored index.
2017-10-21 04:25:41 +02:00
lib: index PKCS7 SignedData parts When we are indexing, we should treat SignedData parts the same way that we treat a multipart object, indexing the wrapped part as a distinct MIME object. Unfortunately, this means doing some sort of cryptographic verification whose results we throw away, because GMime doesn't offer us any way to unwrap without doing signature verification. I've opened https://github.com/jstedfast/gmime/issues/67 to request the capability from GMime but for now, we'll just accept the additional performance hit. As we do this indexing, we also apply the "signed" tag, by analogy with how we handle multipart/signed messages. These days, that kind of change should probably be done with a property instead, but that's a different set of changes. This one is just for consistency. Note that we are currently *only* handling signedData parts, which are basically clearsigned messages. PKCS#7 parts can also be envelopedData and authEnvelopedData (which are effectively encryption layers), and compressedData (which afaict isn't implemented anywhere, i've never encountered it). We're laying the groundwork for indexing these other S/MIME types here, but we're only dealing with signedData for now. Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-05-13 00:29:31 +02:00
static void
_index_pkcs7_part (notmuch_message_t *message,
notmuch_indexopts_t *indexopts,
GMimeObject *part,
_notmuch_message_crypto_t *msg_crypto);
lib: index attachments with mime types matching index.as_text Instead of skipping indexing all attachments, we check of a (user configured) mime type that is indexable as text.
2023-01-06 01:02:06 +01:00
static bool
_indexable_as_text (notmuch_message_t *message, GMimeObject *part)
{
GMimeContentType *content_type = g_mime_object_get_content_type (part);
notmuch_database_t *notmuch = notmuch_message_get_database (message);
if (content_type) {
char *mime_string = g_mime_content_type_get_mime_type (content_type);
if (mime_string) {
bool ret = _notmuch_database_indexable_as_text (notmuch, mime_string);
g_free (mime_string);
return ret;
}
}
return false;
}
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
/* Callback to generate terms for each mime part of a message. */
static void
_index_mime_part (notmuch_message_t *message,
crypto: index encrypted parts when indexopts try_decrypt is set. If we see index options that ask us to decrypt when indexing a message, and we encounter an encrypted part, we'll try to descend into it. If we can decrypt, we add the property index.decryption=success. If we can't decrypt (or recognize the encrypted type of mail), we add the property index.decryption=failure. Note that a single message may have both values of the "index.decryption" property: "success" and "failure". For example, consider a message that includes multiple layers of encryption. If we manage to decrypt the outer layer ("index.decryption=success"), but fail on the inner layer ("index.decryption=failure"). Because of the property name, this will be automatically cleared (and possibly re-set) during re-indexing. This means it will subsequently correspond to the actual semantics of the stored index.
2017-10-21 04:25:41 +02:00
notmuch_indexopts_t *indexopts,
indexing: record protected subject when indexing cleartext When indexing the cleartext of an encrypted message, record any protected subject in the database, which should make it findable and visible in search. Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-28 00:40:28 +02:00
GMimeObject *part,
_notmuch_message_crypto_t *msg_crypto)
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
{
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
GMimeStream *stream, *filter;
lib/index: generalize name of indexing filter In followup commits we will generalize the functionality of this filter to deal with other types of non-indexable content.
2017-06-08 04:11:46 +02:00
GMimeFilter *discard_non_term_filter;
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
GMimeDataWrapper *wrapper;
GByteArray *byte_array;
GMimeContentDisposition *disposition;
lib: abstract content type indexing Make the follow-up change of indexing signature content types easier. No functional changes.
2017-09-16 16:59:02 +02:00
GMimeContentType *content_type;
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
char *body;
Convert non-UTF-8 parts to UTF-8 before indexing them This fixes a bug that didn't allow to search for non-ASCII words such parts. The code here was copied from show_text_part_content(), because the show command already does the needed conversion when showing the message.
2012-02-24 08:36:22 +01:00
const char *charset;
index: repair "Mixed Up" messages before indexing. When encountering a message that has been mangled in the "mixed up" way by an intermediate MTA, notmuch should instead repair it and index the repaired form. When it does this, it also associates the index.repaired=mixedup property with the message. If a problem is found with this repair process, or an improved repair process is proposed later, this should make it easy for people to reindex the relevant message. The property will also hopefully make it easier to diagnose this particular problem in the future. Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-28 20:42:26 +02:00
GMimeObject *repaired_part = NULL;
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
Do not segfault on empty mime parts notmuch previously unconditionally checked mime parts for various properties, but not for NULL, which is the case if libgmime encounters an empty mime part. Upon encounter of an empty mime part, the following is printed to stderr (the second line due to my patch): (process:17197): gmime-CRITICAL **: g_mime_message_get_mime_part: assertion `GMIME_IS_MESSAGE (message)' failed Warning: Not indexing empty mime part. This is probably a bug that should get addressed in libgmime, but for not, my patch is an acceptable workaround. Signed-off-by: martin f. krafft <madduck@madduck.net>
2010-03-02 16:31:28 +01:00
if (! part) {
lib: expose notmuch_message_get_database() We've had _notmuch_message_database() internally for a while, and it's useful. It turns out to be useful on the other side of the library interface as well (i'll use it later in this series for "notmuch show"), so we expose it publicly now.
2018-05-11 08:57:52 +02:00
_notmuch_database_log (notmuch_message_get_database (message),
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
"Warning: Not indexing empty mime part.\n");
index: repair "Mixed Up" messages before indexing. When encountering a message that has been mangled in the "mixed up" way by an intermediate MTA, notmuch should instead repair it and index the repaired form. When it does this, it also associates the index.repaired=mixedup property with the message. If a problem is found with this repair process, or an improved repair process is proposed later, this should make it easy for people to reindex the relevant message. The property will also hopefully make it easier to diagnose this particular problem in the future. Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-28 20:42:26 +02:00
goto DONE;
}
repaired_part = _notmuch_repair_mixed_up_mangled (part);
if (repaired_part) {
/* This was likely "Mixed Up" in transit! We will instead use
* the more likely-to-be-correct variant. */
notmuch_message_add_property (message, "index.repaired", "mixedup");
part = repaired_part;
Do not segfault on empty mime parts notmuch previously unconditionally checked mime parts for various properties, but not for NULL, which is the case if libgmime encounters an empty mime part. Upon encounter of an empty mime part, the following is printed to stderr (the second line due to my patch): (process:17197): gmime-CRITICAL **: g_mime_message_get_mime_part: assertion `GMIME_IS_MESSAGE (message)' failed Warning: Not indexing empty mime part. This is probably a bug that should get addressed in libgmime, but for not, my patch is an acceptable workaround. Signed-off-by: martin f. krafft <madduck@madduck.net>
2010-03-02 16:31:28 +01:00
}
lib: abstract content type indexing Make the follow-up change of indexing signature content types easier. No functional changes.
2017-09-16 16:59:02 +02:00
_index_content_type (message, part);
Add indexing for the mimetype term This adds the indexing support for the "mimetype:" term and removes the broken test flag. The indexing is probablistic in Xapian terms, which gives a better experience to end users. Standard content-types of the form "foo/bar" are automatically interpreted as phrases in Xapian due to the embedded slash. Assume, separate messages with application/pdf and application/x-pdf are indexed, then: - mimetype:application/x-pdf will find only the application/x-pdf - mimetype:application/pdf will find only the application/pdf - mimetype:pdf will find both of the messages
2015-01-23 00:43:38 +01:00
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
if (GMIME_IS_MULTIPART (part)) {
GMimeMultipart *multipart = GMIME_MULTIPART (part);
int i;
tag signed/encrypted during notmuch new This patch adds the tag "signed" to messages with any multipart/signed parts, and the tag "encrypted" to messages with any multipart/encrypted parts. This only occurs when messages are indexed during notmuch new, so a database rebuild is required to have old messages tagged.
2011-05-26 03:01:20 +02:00
if (GMIME_IS_MULTIPART_SIGNED (multipart))
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
_notmuch_message_add_term (message, "tag", "signed");
tag signed/encrypted during notmuch new This patch adds the tag "signed" to messages with any multipart/signed parts, and the tag "encrypted" to messages with any multipart/encrypted parts. This only occurs when messages are indexed during notmuch new, so a database rebuild is required to have old messages tagged.
2011-05-26 03:01:20 +02:00
if (GMIME_IS_MULTIPART_ENCRYPTED (multipart))
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
_notmuch_message_add_term (message, "tag", "encrypted");
tag signed/encrypted during notmuch new This patch adds the tag "signed" to messages with any multipart/signed parts, and the tag "encrypted" to messages with any multipart/encrypted parts. This only occurs when messages are indexed during notmuch new, so a database rebuild is required to have old messages tagged.
2011-05-26 03:01:20 +02:00
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
for (i = 0; i < g_mime_multipart_get_count (multipart); i++) {
indexing: record protected subject when indexing cleartext When indexing the cleartext of an encrypted message, record any protected subject in the database, which should make it findable and visible in search. Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-28 00:40:28 +02:00
GMimeObject *child;
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
if (GMIME_IS_MULTIPART_SIGNED (multipart)) {
lib: index the content type of signature parts It's useful (*) to be able to easily find messages with certain types of signatures. Having the mimetype: prefix searches fail for some content types is also genuinely surprising (*). Index the content type of signature parts. While at it, switch to the gmime convenience constants for content and signature part indexes. *) At least for developers of email software!
2017-09-16 16:59:04 +02:00
/* Don't index the signature, but index its content type. */
if (i == GMIME_MULTIPART_SIGNED_SIGNATURE) {
_index_content_type (message,
g_mime_multipart_get_part (multipart, i));
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
continue;
lib: index the content type of signature parts It's useful (*) to be able to easily find messages with certain types of signatures. Having the mimetype: prefix searches fail for some content types is also genuinely surprising (*). Index the content type of signature parts. While at it, switch to the gmime convenience constants for content and signature part indexes. *) At least for developers of email software!
2017-09-16 16:59:04 +02:00
} else if (i != GMIME_MULTIPART_SIGNED_CONTENT) {
lib: expose notmuch_message_get_database() We've had _notmuch_message_database() internally for a while, and it's useful. It turns out to be useful on the other side of the library interface as well (i'll use it later in this series for "notmuch show"), so we expose it publicly now.
2018-05-11 08:57:52 +02:00
_notmuch_database_log (notmuch_message_get_database (message),
lib: index the content type of signature parts It's useful (*) to be able to easily find messages with certain types of signatures. Having the mimetype: prefix searches fail for some content types is also genuinely surprising (*). Index the content type of signature parts. While at it, switch to the gmime convenience constants for content and signature part indexes. *) At least for developers of email software!
2017-09-16 16:59:04 +02:00
"Warning: Unexpected extra parts of multipart/signed. Indexing anyway.\n");
}
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
}
Ignore encrypted parts when indexing. It appears to be an oversight that encrypted parts were indexed previously. The terms generated from encrypted parts are meaningless and do nothing but add bloat to the database. It is not worth indexing the encrypted content, just as it's not worth indexing the signatures in signed parts.
2011-12-28 21:14:29 +01:00
if (GMIME_IS_MULTIPART_ENCRYPTED (multipart)) {
lib: index the content-type of the parts of encrypted messages This is a logical followup to "lib: index the content type of signature parts", which will make it easier to record the message structure of all messages.
2017-09-16 16:59:05 +02:00
_index_content_type (message,
g_mime_multipart_get_part (multipart, i));
crypto: index encrypted parts when indexopts try_decrypt is set. If we see index options that ask us to decrypt when indexing a message, and we encounter an encrypted part, we'll try to descend into it. If we can decrypt, we add the property index.decryption=success. If we can't decrypt (or recognize the encrypted type of mail), we add the property index.decryption=failure. Note that a single message may have both values of the "index.decryption" property: "success" and "failure". For example, consider a message that includes multiple layers of encryption. If we manage to decrypt the outer layer ("index.decryption=success"), but fail on the inner layer ("index.decryption=failure"). Because of the property name, this will be automatically cleared (and possibly re-set) during re-indexing. This means it will subsequently correspond to the actual semantics of the stored index.
2017-10-21 04:25:41 +02:00
if (i == GMIME_MULTIPART_ENCRYPTED_CONTENT) {
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
_index_encrypted_mime_part (message, indexopts,
crypto: Make _notmuch_crypto_decrypt take a GMimeObject As we prepare to handle S/MIME-encrypted PKCS#7 EnvelopedData (which is not multipart), we don't want to be limited to passing only GMimeMultipartEncrypted MIME parts to _notmuch_crypto_decrypt. There is no functional change here, just a matter of adjusting how we pass arguments internally. Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-05-13 00:29:36 +02:00
part,
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
msg_crypto);
crypto: index encrypted parts when indexopts try_decrypt is set. If we see index options that ask us to decrypt when indexing a message, and we encounter an encrypted part, we'll try to descend into it. If we can decrypt, we add the property index.decryption=success. If we can't decrypt (or recognize the encrypted type of mail), we add the property index.decryption=failure. Note that a single message may have both values of the "index.decryption" property: "success" and "failure". For example, consider a message that includes multiple layers of encryption. If we manage to decrypt the outer layer ("index.decryption=success"), but fail on the inner layer ("index.decryption=failure"). Because of the property name, this will be automatically cleared (and possibly re-set) during re-indexing. This means it will subsequently correspond to the actual semantics of the stored index.
2017-10-21 04:25:41 +02:00
} else {
if (i != GMIME_MULTIPART_ENCRYPTED_VERSION) {
lib: expose notmuch_message_get_database() We've had _notmuch_message_database() internally for a while, and it's useful. It turns out to be useful on the other side of the library interface as well (i'll use it later in this series for "notmuch show"), so we expose it publicly now.
2018-05-11 08:57:52 +02:00
_notmuch_database_log (notmuch_message_get_database (message),
crypto: index encrypted parts when indexopts try_decrypt is set. If we see index options that ask us to decrypt when indexing a message, and we encounter an encrypted part, we'll try to descend into it. If we can decrypt, we add the property index.decryption=success. If we can't decrypt (or recognize the encrypted type of mail), we add the property index.decryption=failure. Note that a single message may have both values of the "index.decryption" property: "success" and "failure". For example, consider a message that includes multiple layers of encryption. If we manage to decrypt the outer layer ("index.decryption=success"), but fail on the inner layer ("index.decryption=failure"). Because of the property name, this will be automatically cleared (and possibly re-set) during re-indexing. This means it will subsequently correspond to the actual semantics of the stored index.
2017-10-21 04:25:41 +02:00
"Warning: Unexpected extra parts of multipart/encrypted.\n");
}
lib: index the content-type of the parts of encrypted messages This is a logical followup to "lib: index the content type of signature parts", which will make it easier to record the message structure of all messages.
2017-09-16 16:59:05 +02:00
}
Ignore encrypted parts when indexing. It appears to be an oversight that encrypted parts were indexed previously. The terms generated from encrypted parts are meaningless and do nothing but add bloat to the database. It is not worth indexing the encrypted content, just as it's not worth indexing the signatures in signed parts.
2011-12-28 21:14:29 +01:00
continue;
}
indexing: record protected subject when indexing cleartext When indexing the cleartext of an encrypted message, record any protected subject in the database, which should make it findable and visible in search. Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-28 00:40:28 +02:00
child = g_mime_multipart_get_part (multipart, i);
index: avoid indexing legacy-display parts When we notice a legacy-display part during indexing, it makes more sense to avoid indexing it as part of the message body. Given that the protected subject will already be indexed, there is no need to index this part at all, so we skip over it. If this happens during indexing, we set a property on the message: index.repaired=skip-protected-headers-legacy-display Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-08-29 17:38:53 +02:00
GMimeObject *toindex = child;
if (_notmuch_message_crypto_potential_payload (msg_crypto, child, part, i) &&
msg_crypto->decryption_status == NOTMUCH_MESSAGE_DECRYPTED_FULL) {
toindex = _notmuch_repair_crypto_payload_skip_legacy_display (child);
if (toindex != child)
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2021-03-13 13:45:34 +01:00
notmuch_message_add_property (message, "index.repaired",
"skip-protected-headers-legacy-display");
index: avoid indexing legacy-display parts When we notice a legacy-display part during indexing, it makes more sense to avoid indexing it as part of the message body. Given that the protected subject will already be indexed, there is no need to index this part at all, so we skip over it. If this happens during indexing, we set a property on the message: index.repaired=skip-protected-headers-legacy-display Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-08-29 17:38:53 +02:00
}
_index_mime_part (message, indexopts, toindex, msg_crypto);
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
}
index: repair "Mixed Up" messages before indexing. When encountering a message that has been mangled in the "mixed up" way by an intermediate MTA, notmuch should instead repair it and index the repaired form. When it does this, it also associates the index.repaired=mixedup property with the message. If a problem is found with this repair process, or an improved repair process is proposed later, this should make it easy for people to reindex the relevant message. The property will also hopefully make it easier to diagnose this particular problem in the future. Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-28 20:42:26 +02:00
goto DONE;
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
}
if (GMIME_IS_MESSAGE_PART (part)) {
GMimeMessage *mime_message;
mime_message = g_mime_message_part_get_message (GMIME_MESSAGE_PART (part));
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2021-03-13 13:45:34 +01:00
_index_mime_part (message, indexopts, g_mime_message_get_mime_part (mime_message),
msg_crypto);
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
index: repair "Mixed Up" messages before indexing. When encountering a message that has been mangled in the "mixed up" way by an intermediate MTA, notmuch should instead repair it and index the repaired form. When it does this, it also associates the index.repaired=mixedup property with the message. If a problem is found with this repair process, or an improved repair process is proposed later, this should make it easy for people to reindex the relevant message. The property will also hopefully make it easier to diagnose this particular problem in the future. Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-28 20:42:26 +02:00
goto DONE;
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
}
lib: index PKCS7 SignedData parts When we are indexing, we should treat SignedData parts the same way that we treat a multipart object, indexing the wrapped part as a distinct MIME object. Unfortunately, this means doing some sort of cryptographic verification whose results we throw away, because GMime doesn't offer us any way to unwrap without doing signature verification. I've opened https://github.com/jstedfast/gmime/issues/67 to request the capability from GMime but for now, we'll just accept the additional performance hit. As we do this indexing, we also apply the "signed" tag, by analogy with how we handle multipart/signed messages. These days, that kind of change should probably be done with a property instead, but that's a different set of changes. This one is just for consistency. Note that we are currently *only* handling signedData parts, which are basically clearsigned messages. PKCS#7 parts can also be envelopedData and authEnvelopedData (which are effectively encryption layers), and compressedData (which afaict isn't implemented anywhere, i've never encountered it). We're laying the groundwork for indexing these other S/MIME types here, but we're only dealing with signedData for now. Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-05-13 00:29:31 +02:00
if (GMIME_IS_APPLICATION_PKCS7_MIME (part)) {
_index_pkcs7_part (message, indexopts, part, msg_crypto);
goto DONE;
}
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
if (! (GMIME_IS_PART (part))) {
lib: expose notmuch_message_get_database() We've had _notmuch_message_database() internally for a while, and it's useful. It turns out to be useful on the other side of the library interface as well (i'll use it later in this series for "notmuch show"), so we expose it publicly now.
2018-05-11 08:57:52 +02:00
_notmuch_database_log (notmuch_message_get_database (message),
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
"Warning: Not indexing unknown mime part: %s.\n",
g_type_name (G_OBJECT_TYPE (part)));
index: repair "Mixed Up" messages before indexing. When encountering a message that has been mangled in the "mixed up" way by an intermediate MTA, notmuch should instead repair it and index the repaired form. When it does this, it also associates the index.repaired=mixedup property with the message. If a problem is found with this repair process, or an improved repair process is proposed later, this should make it easy for people to reindex the relevant message. The property will also hopefully make it easier to diagnose this particular problem in the future. Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-28 20:42:26 +02:00
goto DONE;
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
}
disposition = g_mime_object_get_content_disposition (part);
if (disposition &&
lib: content disposition values are not case-sensitive Per RFC 2183, the values for Content-Disposition values are not case-sensitive. While at it, use the gmime function for getting at the disposition string instead of referencing the field directly. This fixes "attachment" tagging and filename term generation for attachments while indexing.
2015-09-26 11:35:21 +02:00
strcasecmp (g_mime_content_disposition_get_disposition (disposition),
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
GMIME_DISPOSITION_ATTACHMENT) == 0) {
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
const char *filename = g_mime_part_get_filename (GMIME_PART (part));
_notmuch_message_add_term (message, "tag", "attachment");
_notmuch_message_gen_terms (message, "attachment", filename);
lib: index attachments with mime types matching index.as_text Instead of skipping indexing all attachments, we check of a (user configured) mime type that is indexable as text.
2023-01-06 01:02:06 +01:00
if (! _indexable_as_text (message, part)) {
/* XXX: Would be nice to call out to something here to parse
* the attachment into text and then index that. */
goto DONE;
}
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
}
byte_array = g_byte_array_new ();
stream = g_mime_stream_mem_new_with_byte_array (byte_array);
lib: convert notmuch_bool_t to stdbool internally C99 stdbool turned 18 this year. There really is no reason to use our own, except in the library interface for backward compatibility. Convert the lib internally to stdbool.
2017-10-07 10:44:05 +02:00
g_mime_stream_mem_set_owner (GMIME_STREAM_MEM (stream), false);
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
filter = g_mime_stream_filter_new (stream);
lib: abstract content type indexing Make the follow-up change of indexing signature content types easier. No functional changes.
2017-09-16 16:59:02 +02:00
gmime-cleanup: drop unused gmime 2.6 content_type from _index_encrypted_mime_part In _index_mime_part, we don't need to extract the content-type from the part until just before we use it, so we also defer it lazily. Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-02 15:19:34 +02:00
content_type = g_mime_object_get_content_type (part);
lib/index: add simple html filter The filter just drops all (HTML) tags. As an enabling change, pass the content type to the filter constructor so we can decide which scanner to user.
2017-06-08 04:11:49 +02:00
discard_non_term_filter = notmuch_filter_discard_non_term_new (content_type);
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
g_mime_stream_filter_add (GMIME_STREAM_FILTER (filter),
lib/index: generalize name of indexing filter In followup commits we will generalize the functionality of this filter to deal with other types of non-indexable content.
2017-06-08 04:11:46 +02:00
discard_non_term_filter);
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
Convert non-UTF-8 parts to UTF-8 before indexing them This fixes a bug that didn't allow to search for non-ASCII words such parts. The code here was copied from show_text_part_content(), because the show command already does the needed conversion when showing the message.
2012-02-24 08:36:22 +01:00
charset = g_mime_object_get_content_type_parameter (part, "charset");
if (charset) {
GMimeFilter *charset_filter;
charset_filter = g_mime_filter_charset_new (charset, "UTF-8");
/* This result can be NULL for things like "unknown-8bit".
* Don't set a NULL filter as that makes GMime print
* annoying assertion-failure messages on stderr. */
if (charset_filter) {
g_mime_stream_filter_add (GMIME_STREAM_FILTER (filter),
charset_filter);
g_object_unref (charset_filter);
}
}
gmime-cleanup: use GMime 3.0 function names Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-02 15:19:44 +02:00
wrapper = g_mime_part_get_content (GMIME_PART (part));
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
if (wrapper)
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
g_mime_data_wrapper_write_to_stream (wrapper, filter);
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
g_object_unref (stream);
notmuch new: Don't index uuencoded data. With modern MIME attachments, we're already avoiding indexing the attachments. But for old-school uuencoded data in the mail, we have been directly indexing the encoded data as terms, (which is not useful at all---nobody will ever ytry to search based on the seemingly random uuencoded data). Additionally, indexing a modestly large uuencoded file seems to make Xapian go insane, (consuming *lots* of memory). We fix both problems by detecting uuencoded content and not performing any indexing of it.
2010-02-05 02:08:11 +01:00
g_object_unref (filter);
lib/index: generalize name of indexing filter In followup commits we will generalize the functionality of this filter to deal with other types of non-indexable content.
2017-06-08 04:11:46 +02:00
g_object_unref (discard_non_term_filter);
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
g_byte_array_append (byte_array, (guint8 *) "\0", 1);
lib: convert notmuch_bool_t to stdbool internally C99 stdbool turned 18 this year. There really is no reason to use our own, except in the library interface for backward compatibility. Convert the lib internally to stdbool.
2017-10-07 10:44:05 +02:00
body = (char *) g_byte_array_free (byte_array, false);
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
Index content from citations and signatures. In the presentation we often omit citations and signatures, but this is not content that should be omitted from the index, (especially when the citation detection is wrong---see cases where a line beginning with "From" is corrupted to ">From" by mail processing tools).
2010-01-06 04:14:07 +01:00
if (body) {
_notmuch_message_gen_terms (message, NULL, body);
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
Index content from citations and signatures. In the presentation we often omit citations and signatures, but this is not content that should be omitted from the index, (especially when the citation detection is wrong---see cases where a line beginning with "From" is corrupted to ">From" by mail processing tools).
2010-01-06 04:14:07 +01:00
free (body);
}
index: repair "Mixed Up" messages before indexing. When encountering a message that has been mangled in the "mixed up" way by an intermediate MTA, notmuch should instead repair it and index the repaired form. When it does this, it also associates the index.repaired=mixedup property with the message. If a problem is found with this repair process, or an improved repair process is proposed later, this should make it easy for people to reindex the relevant message. The property will also hopefully make it easier to diagnose this particular problem in the future. Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-28 20:42:26 +02:00
DONE:
if (repaired_part)
g_object_unref (repaired_part);
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
}
crypto: index encrypted parts when indexopts try_decrypt is set. If we see index options that ask us to decrypt when indexing a message, and we encounter an encrypted part, we'll try to descend into it. If we can decrypt, we add the property index.decryption=success. If we can't decrypt (or recognize the encrypted type of mail), we add the property index.decryption=failure. Note that a single message may have both values of the "index.decryption" property: "success" and "failure". For example, consider a message that includes multiple layers of encryption. If we manage to decrypt the outer layer ("index.decryption=success"), but fail on the inner layer ("index.decryption=failure"). Because of the property name, this will be automatically cleared (and possibly re-set) during re-indexing. This means it will subsequently correspond to the actual semantics of the stored index.
2017-10-21 04:25:41 +02:00
/* descend (if desired) into the cleartext part of an encrypted MIME
* part while indexing. */
static void
_index_encrypted_mime_part (notmuch_message_t *message,
notmuch_indexopts_t *indexopts,
crypto: Make _notmuch_crypto_decrypt take a GMimeObject As we prepare to handle S/MIME-encrypted PKCS#7 EnvelopedData (which is not multipart), we don't want to be limited to passing only GMimeMultipartEncrypted MIME parts to _notmuch_crypto_decrypt. There is no functional change here, just a matter of adjusting how we pass arguments internally. Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-05-13 00:29:36 +02:00
GMimeObject *encrypted_data,
indexing: record protected subject when indexing cleartext When indexing the cleartext of an encrypted message, record any protected subject in the database, which should make it findable and visible in search. Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-28 00:40:28 +02:00
_notmuch_message_crypto_t *msg_crypto)
crypto: index encrypted parts when indexopts try_decrypt is set. If we see index options that ask us to decrypt when indexing a message, and we encounter an encrypted part, we'll try to descend into it. If we can decrypt, we add the property index.decryption=success. If we can't decrypt (or recognize the encrypted type of mail), we add the property index.decryption=failure. Note that a single message may have both values of the "index.decryption" property: "success" and "failure". For example, consider a message that includes multiple layers of encryption. If we manage to decrypt the outer layer ("index.decryption=success"), but fail on the inner layer ("index.decryption=failure"). Because of the property name, this will be automatically cleared (and possibly re-set) during re-indexing. This means it will subsequently correspond to the actual semantics of the stored index.
2017-10-21 04:25:41 +02:00
{
notmuch_status_t status;
GError *err = NULL;
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
notmuch_database_t *notmuch = NULL;
crypto: index encrypted parts when indexopts try_decrypt is set. If we see index options that ask us to decrypt when indexing a message, and we encounter an encrypted part, we'll try to descend into it. If we can decrypt, we add the property index.decryption=success. If we can't decrypt (or recognize the encrypted type of mail), we add the property index.decryption=failure. Note that a single message may have both values of the "index.decryption" property: "success" and "failure". For example, consider a message that includes multiple layers of encryption. If we manage to decrypt the outer layer ("index.decryption=success"), but fail on the inner layer ("index.decryption=failure"). Because of the property name, this will be automatically cleared (and possibly re-set) during re-indexing. This means it will subsequently correspond to the actual semantics of the stored index.
2017-10-21 04:25:41 +02:00
GMimeObject *clear = NULL;
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
if (! indexopts || (notmuch_indexopts_get_decrypt_policy (indexopts) == NOTMUCH_DECRYPT_FALSE))
crypto: index encrypted parts when indexopts try_decrypt is set. If we see index options that ask us to decrypt when indexing a message, and we encounter an encrypted part, we'll try to descend into it. If we can decrypt, we add the property index.decryption=success. If we can't decrypt (or recognize the encrypted type of mail), we add the property index.decryption=failure. Note that a single message may have both values of the "index.decryption" property: "success" and "failure". For example, consider a message that includes multiple layers of encryption. If we manage to decrypt the outer layer ("index.decryption=success"), but fail on the inner layer ("index.decryption=failure"). Because of the property name, this will be automatically cleared (and possibly re-set) during re-indexing. This means it will subsequently correspond to the actual semantics of the stored index.
2017-10-21 04:25:41 +02:00
return;
lib: expose notmuch_message_get_database() We've had _notmuch_message_database() internally for a while, and it's useful. It turns out to be useful on the other side of the library interface as well (i'll use it later in this series for "notmuch show"), so we expose it publicly now.
2018-05-11 08:57:52 +02:00
notmuch = notmuch_message_get_database (message);
crypto: index encrypted parts when indexopts try_decrypt is set. If we see index options that ask us to decrypt when indexing a message, and we encounter an encrypted part, we'll try to descend into it. If we can decrypt, we add the property index.decryption=success. If we can't decrypt (or recognize the encrypted type of mail), we add the property index.decryption=failure. Note that a single message may have both values of the "index.decryption" property: "success" and "failure". For example, consider a message that includes multiple layers of encryption. If we manage to decrypt the outer layer ("index.decryption=success"), but fail on the inner layer ("index.decryption=failure"). Because of the property name, this will be automatically cleared (and possibly re-set) during re-indexing. This means it will subsequently correspond to the actual semantics of the stored index.
2017-10-21 04:25:41 +02:00
crypto: record whether an actual decryption attempt happened In our consolidation of _notmuch_crypto_decrypt, the callers lost track a little bit of whether any actual decryption was attempted. Now that we have the more-subtle "auto" policy, it's possible that _notmuch_crypto_decrypt could be called without having any actual decryption take place. This change lets the callers be a little bit smarter about whether or not any decryption was actually attempted.
2017-12-08 07:23:58 +01:00
bool attempted = false;
crypto: actually stash session keys when decrypt=true If you're going to store the cleartext index of an encrypted message, in most situations you might just as well store the session key. Doing this storage has efficiency and recoverability advantages. Combined with a schedule of regular OpenPGP subkey rotation and destruction, this can also offer security benefits, like "deletable e-mail", which is the store-and-forward analog to "forward secrecy". But wait, i hear you saying, i have a special need to store cleartext indexes but it's really bad for me to store session keys! Maybe (let's imagine) i get lots of e-mails with incriminating photos attached, and i want to be able to search for them by the text in the e-mail, but i don't want someone with access to the index to be actually able to see the photos themselves. Fret not, the next patch in this series will support your wacky uncommon use case.
2017-12-08 07:24:01 +01:00
GMimeDecryptResult *decrypt_result = NULL;
gmime-cleanup: always support session keys Our minimum version of GMime 3.0 always supports good session key handling. signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-02 15:19:37 +02:00
bool get_sk = (notmuch_indexopts_get_decrypt_policy (indexopts) == NOTMUCH_DECRYPT_TRUE);
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2021-03-13 13:45:34 +01:00
crypto: record whether an actual decryption attempt happened In our consolidation of _notmuch_crypto_decrypt, the callers lost track a little bit of whether any actual decryption was attempted. Now that we have the more-subtle "auto" policy, it's possible that _notmuch_crypto_decrypt could be called without having any actual decryption take place. This change lets the callers be a little bit smarter about whether or not any decryption was actually attempted.
2017-12-08 07:23:58 +01:00
clear = _notmuch_crypto_decrypt (&attempted, notmuch_indexopts_get_decrypt_policy (indexopts),
gmime-cleanup: drop all arguments unused in GMime 3 This means dropping GMimeCryptoContext and notmuch_config arguments. All the argument changes are to internal functions, so this is not an API or ABI break. We also get to drop the #define for g_mime_3_unused. signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-02 15:19:41 +02:00
message, encrypted_data, get_sk ? &decrypt_result : NULL, &err);
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
if (! attempted)
crypto: record whether an actual decryption attempt happened In our consolidation of _notmuch_crypto_decrypt, the callers lost track a little bit of whether any actual decryption was attempted. Now that we have the more-subtle "auto" policy, it's possible that _notmuch_crypto_decrypt could be called without having any actual decryption take place. This change lets the callers be a little bit smarter about whether or not any decryption was actually attempted.
2017-12-08 07:23:58 +01:00
return;
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
if (err || ! clear) {
crypto: actually stash session keys when decrypt=true If you're going to store the cleartext index of an encrypted message, in most situations you might just as well store the session key. Doing this storage has efficiency and recoverability advantages. Combined with a schedule of regular OpenPGP subkey rotation and destruction, this can also offer security benefits, like "deletable e-mail", which is the store-and-forward analog to "forward secrecy". But wait, i hear you saying, i have a special need to store cleartext indexes but it's really bad for me to store session keys! Maybe (let's imagine) i get lots of e-mails with incriminating photos attached, and i want to be able to search for them by the text in the e-mail, but i don't want someone with access to the index to be actually able to see the photos themselves. Fret not, the next patch in this series will support your wacky uncommon use case.
2017-12-08 07:24:01 +01:00
if (decrypt_result)
g_object_unref (decrypt_result);
crypto: record whether an actual decryption attempt happened In our consolidation of _notmuch_crypto_decrypt, the callers lost track a little bit of whether any actual decryption was attempted. Now that we have the more-subtle "auto" policy, it's possible that _notmuch_crypto_decrypt could be called without having any actual decryption take place. This change lets the callers be a little bit smarter about whether or not any decryption was actually attempted.
2017-12-08 07:23:58 +01:00
if (err) {
_notmuch_database_log (notmuch, "Failed to decrypt during indexing. (%d:%d) [%s]\n",
err->domain, err->code, err->message);
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2019-06-13 12:55:35 +02:00
g_error_free (err);
crypto: record whether an actual decryption attempt happened In our consolidation of _notmuch_crypto_decrypt, the callers lost track a little bit of whether any actual decryption was attempted. Now that we have the more-subtle "auto" policy, it's possible that _notmuch_crypto_decrypt could be called without having any actual decryption take place. This change lets the callers be a little bit smarter about whether or not any decryption was actually attempted.
2017-12-08 07:23:58 +01:00
} else {
_notmuch_database_log (notmuch, "Failed to decrypt during indexing. (unknown error)\n");
}
crypto: index encrypted parts when indexopts try_decrypt is set. If we see index options that ask us to decrypt when indexing a message, and we encounter an encrypted part, we'll try to descend into it. If we can decrypt, we add the property index.decryption=success. If we can't decrypt (or recognize the encrypted type of mail), we add the property index.decryption=failure. Note that a single message may have both values of the "index.decryption" property: "success" and "failure". For example, consider a message that includes multiple layers of encryption. If we manage to decrypt the outer layer ("index.decryption=success"), but fail on the inner layer ("index.decryption=failure"). Because of the property name, this will be automatically cleared (and possibly re-set) during re-indexing. This means it will subsequently correspond to the actual semantics of the stored index.
2017-10-21 04:25:41 +02:00
/* Indicate that we failed to decrypt during indexing */
status = notmuch_message_add_property (message, "index.decryption", "failure");
if (status)
_notmuch_database_log_append (notmuch, "failed to add index.decryption "
"property (%d)\n", status);
return;
}
crypto: actually stash session keys when decrypt=true If you're going to store the cleartext index of an encrypted message, in most situations you might just as well store the session key. Doing this storage has efficiency and recoverability advantages. Combined with a schedule of regular OpenPGP subkey rotation and destruction, this can also offer security benefits, like "deletable e-mail", which is the store-and-forward analog to "forward secrecy". But wait, i hear you saying, i have a special need to store cleartext indexes but it's really bad for me to store session keys! Maybe (let's imagine) i get lots of e-mails with incriminating photos attached, and i want to be able to search for them by the text in the e-mail, but i don't want someone with access to the index to be actually able to see the photos themselves. Fret not, the next patch in this series will support your wacky uncommon use case.
2017-12-08 07:24:01 +01:00
if (decrypt_result) {
indexing: record protected subject when indexing cleartext When indexing the cleartext of an encrypted message, record any protected subject in the database, which should make it findable and visible in search. Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-28 00:40:28 +02:00
status = _notmuch_message_crypto_successful_decryption (msg_crypto);
if (status)
_notmuch_database_log_append (notmuch, "failed to mark the message as decrypted (%s)\n",
notmuch_status_to_string (status));
crypto: actually stash session keys when decrypt=true If you're going to store the cleartext index of an encrypted message, in most situations you might just as well store the session key. Doing this storage has efficiency and recoverability advantages. Combined with a schedule of regular OpenPGP subkey rotation and destruction, this can also offer security benefits, like "deletable e-mail", which is the store-and-forward analog to "forward secrecy". But wait, i hear you saying, i have a special need to store cleartext indexes but it's really bad for me to store session keys! Maybe (let's imagine) i get lots of e-mails with incriminating photos attached, and i want to be able to search for them by the text in the e-mail, but i don't want someone with access to the index to be actually able to see the photos themselves. Fret not, the next patch in this series will support your wacky uncommon use case.
2017-12-08 07:24:01 +01:00
if (get_sk) {
status = notmuch_message_add_property (message, "session-key",
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2021-03-13 13:45:34 +01:00
g_mime_decrypt_result_get_session_key (
decrypt_result));
crypto: actually stash session keys when decrypt=true If you're going to store the cleartext index of an encrypted message, in most situations you might just as well store the session key. Doing this storage has efficiency and recoverability advantages. Combined with a schedule of regular OpenPGP subkey rotation and destruction, this can also offer security benefits, like "deletable e-mail", which is the store-and-forward analog to "forward secrecy". But wait, i hear you saying, i have a special need to store cleartext indexes but it's really bad for me to store session keys! Maybe (let's imagine) i get lots of e-mails with incriminating photos attached, and i want to be able to search for them by the text in the e-mail, but i don't want someone with access to the index to be actually able to see the photos themselves. Fret not, the next patch in this series will support your wacky uncommon use case.
2017-12-08 07:24:01 +01:00
if (status)
_notmuch_database_log (notmuch, "failed to add session-key "
"property (%d)\n", status);
}
g_object_unref (decrypt_result);
}
index: avoid indexing legacy-display parts When we notice a legacy-display part during indexing, it makes more sense to avoid indexing it as part of the message body. Given that the protected subject will already be indexed, there is no need to index this part at all, so we skip over it. If this happens during indexing, we set a property on the message: index.repaired=skip-protected-headers-legacy-display Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-08-29 17:38:53 +02:00
GMimeObject *toindex = clear;
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2021-03-13 13:45:34 +01:00
if (_notmuch_message_crypto_potential_payload (msg_crypto, clear, encrypted_data,
GMIME_MULTIPART_ENCRYPTED_CONTENT) &&
index: avoid indexing legacy-display parts When we notice a legacy-display part during indexing, it makes more sense to avoid indexing it as part of the message body. Given that the protected subject will already be indexed, there is no need to index this part at all, so we skip over it. If this happens during indexing, we set a property on the message: index.repaired=skip-protected-headers-legacy-display Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-08-29 17:38:53 +02:00
msg_crypto->decryption_status == NOTMUCH_MESSAGE_DECRYPTED_FULL) {
toindex = _notmuch_repair_crypto_payload_skip_legacy_display (clear);
if (toindex != clear)
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2021-03-13 13:45:34 +01:00
notmuch_message_add_property (message, "index.repaired",
"skip-protected-headers-legacy-display");
index: avoid indexing legacy-display parts When we notice a legacy-display part during indexing, it makes more sense to avoid indexing it as part of the message body. Given that the protected subject will already be indexed, there is no need to index this part at all, so we skip over it. If this happens during indexing, we set a property on the message: index.repaired=skip-protected-headers-legacy-display Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-08-29 17:38:53 +02:00
}
_index_mime_part (message, indexopts, toindex, msg_crypto);
crypto: index encrypted parts when indexopts try_decrypt is set. If we see index options that ask us to decrypt when indexing a message, and we encounter an encrypted part, we'll try to descend into it. If we can decrypt, we add the property index.decryption=success. If we can't decrypt (or recognize the encrypted type of mail), we add the property index.decryption=failure. Note that a single message may have both values of the "index.decryption" property: "success" and "failure". For example, consider a message that includes multiple layers of encryption. If we manage to decrypt the outer layer ("index.decryption=success"), but fail on the inner layer ("index.decryption=failure"). Because of the property name, this will be automatically cleared (and possibly re-set) during re-indexing. This means it will subsequently correspond to the actual semantics of the stored index.
2017-10-21 04:25:41 +02:00
g_object_unref (clear);
status = notmuch_message_add_property (message, "index.decryption", "success");
if (status)
_notmuch_database_log (notmuch, "failed to add index.decryption "
"property (%d)\n", status);
}
lib: index PKCS7 SignedData parts When we are indexing, we should treat SignedData parts the same way that we treat a multipart object, indexing the wrapped part as a distinct MIME object. Unfortunately, this means doing some sort of cryptographic verification whose results we throw away, because GMime doesn't offer us any way to unwrap without doing signature verification. I've opened https://github.com/jstedfast/gmime/issues/67 to request the capability from GMime but for now, we'll just accept the additional performance hit. As we do this indexing, we also apply the "signed" tag, by analogy with how we handle multipart/signed messages. These days, that kind of change should probably be done with a property instead, but that's a different set of changes. This one is just for consistency. Note that we are currently *only* handling signedData parts, which are basically clearsigned messages. PKCS#7 parts can also be envelopedData and authEnvelopedData (which are effectively encryption layers), and compressedData (which afaict isn't implemented anywhere, i've never encountered it). We're laying the groundwork for indexing these other S/MIME types here, but we're only dealing with signedData for now. Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-05-13 00:29:31 +02:00
static void
_index_pkcs7_part (notmuch_message_t *message,
notmuch_indexopts_t *indexopts,
GMimeObject *part,
_notmuch_message_crypto_t *msg_crypto)
{
GMimeApplicationPkcs7Mime *pkcs7;
GMimeSecureMimeType p7type;
GMimeObject *mimeobj = NULL;
GMimeSignatureList *sigs = NULL;
GError *err = NULL;
notmuch_database_t *notmuch = NULL;
pkcs7 = GMIME_APPLICATION_PKCS7_MIME (part);
p7type = g_mime_application_pkcs7_mime_get_smime_type (pkcs7);
notmuch = notmuch_message_get_database (message);
_index_content_type (message, part);
if (p7type == GMIME_SECURE_MIME_TYPE_SIGNED_DATA) {
sigs = g_mime_application_pkcs7_mime_verify (pkcs7, GMIME_VERIFY_NONE, &mimeobj, &err);
if (sigs == NULL) {
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2021-03-13 13:45:34 +01:00
_notmuch_database_log (notmuch,
"Failed to verify PKCS#7 SignedData during indexing. (%d:%d) [%s]\n",
lib: index PKCS7 SignedData parts When we are indexing, we should treat SignedData parts the same way that we treat a multipart object, indexing the wrapped part as a distinct MIME object. Unfortunately, this means doing some sort of cryptographic verification whose results we throw away, because GMime doesn't offer us any way to unwrap without doing signature verification. I've opened https://github.com/jstedfast/gmime/issues/67 to request the capability from GMime but for now, we'll just accept the additional performance hit. As we do this indexing, we also apply the "signed" tag, by analogy with how we handle multipart/signed messages. These days, that kind of change should probably be done with a property instead, but that's a different set of changes. This one is just for consistency. Note that we are currently *only* handling signedData parts, which are basically clearsigned messages. PKCS#7 parts can also be envelopedData and authEnvelopedData (which are effectively encryption layers), and compressedData (which afaict isn't implemented anywhere, i've never encountered it). We're laying the groundwork for indexing these other S/MIME types here, but we're only dealing with signedData for now. Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-05-13 00:29:31 +02:00
err->domain, err->code, err->message);
g_error_free (err);
goto DONE;
}
_notmuch_message_add_term (message, "tag", "signed");
GMimeObject *toindex = mimeobj;
if (_notmuch_message_crypto_potential_payload (msg_crypto, mimeobj, part, 0) &&
msg_crypto->decryption_status == NOTMUCH_MESSAGE_DECRYPTED_FULL) {
toindex = _notmuch_repair_crypto_payload_skip_legacy_display (mimeobj);
if (toindex != mimeobj)
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2021-03-13 13:45:34 +01:00
notmuch_message_add_property (message, "index.repaired",
"skip-protected-headers-legacy-display");
lib: index PKCS7 SignedData parts When we are indexing, we should treat SignedData parts the same way that we treat a multipart object, indexing the wrapped part as a distinct MIME object. Unfortunately, this means doing some sort of cryptographic verification whose results we throw away, because GMime doesn't offer us any way to unwrap without doing signature verification. I've opened https://github.com/jstedfast/gmime/issues/67 to request the capability from GMime but for now, we'll just accept the additional performance hit. As we do this indexing, we also apply the "signed" tag, by analogy with how we handle multipart/signed messages. These days, that kind of change should probably be done with a property instead, but that's a different set of changes. This one is just for consistency. Note that we are currently *only* handling signedData parts, which are basically clearsigned messages. PKCS#7 parts can also be envelopedData and authEnvelopedData (which are effectively encryption layers), and compressedData (which afaict isn't implemented anywhere, i've never encountered it). We're laying the groundwork for indexing these other S/MIME types here, but we're only dealing with signedData for now. Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-05-13 00:29:31 +02:00
}
_index_mime_part (message, indexopts, toindex, msg_crypto);
smime: Identify encrypted S/MIME parts during indexing We don't handle them correctly yet, but we can at least mark them as being encrypted. Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-05-13 00:29:32 +02:00
} else if (p7type == GMIME_SECURE_MIME_TYPE_ENVELOPED_DATA) {
_notmuch_message_add_term (message, "tag", "encrypted");
smime: Index cleartext of envelopedData when requested Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-05-13 00:29:39 +02:00
_index_encrypted_mime_part (message, indexopts,
part,
msg_crypto);
lib: index PKCS7 SignedData parts When we are indexing, we should treat SignedData parts the same way that we treat a multipart object, indexing the wrapped part as a distinct MIME object. Unfortunately, this means doing some sort of cryptographic verification whose results we throw away, because GMime doesn't offer us any way to unwrap without doing signature verification. I've opened https://github.com/jstedfast/gmime/issues/67 to request the capability from GMime but for now, we'll just accept the additional performance hit. As we do this indexing, we also apply the "signed" tag, by analogy with how we handle multipart/signed messages. These days, that kind of change should probably be done with a property instead, but that's a different set of changes. This one is just for consistency. Note that we are currently *only* handling signedData parts, which are basically clearsigned messages. PKCS#7 parts can also be envelopedData and authEnvelopedData (which are effectively encryption layers), and compressedData (which afaict isn't implemented anywhere, i've never encountered it). We're laying the groundwork for indexing these other S/MIME types here, but we're only dealing with signedData for now. Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-05-13 00:29:31 +02:00
} else {
_notmuch_database_log (notmuch, "Cannot currently handle PKCS#7 smime-type '%s'\n",
g_mime_object_get_content_type_parameter (part, "smime-type"));
}
lib: run uncrustify This is the result of running $ uncrustify --replace --config ../devel/uncrustify.cfg *.c *.h *.cc in the lib directory
2021-03-13 13:45:34 +01:00
DONE:
lib: index PKCS7 SignedData parts When we are indexing, we should treat SignedData parts the same way that we treat a multipart object, indexing the wrapped part as a distinct MIME object. Unfortunately, this means doing some sort of cryptographic verification whose results we throw away, because GMime doesn't offer us any way to unwrap without doing signature verification. I've opened https://github.com/jstedfast/gmime/issues/67 to request the capability from GMime but for now, we'll just accept the additional performance hit. As we do this indexing, we also apply the "signed" tag, by analogy with how we handle multipart/signed messages. These days, that kind of change should probably be done with a property instead, but that's a different set of changes. This one is just for consistency. Note that we are currently *only* handling signedData parts, which are basically clearsigned messages. PKCS#7 parts can also be envelopedData and authEnvelopedData (which are effectively encryption layers), and compressedData (which afaict isn't implemented anywhere, i've never encountered it). We're laying the groundwork for indexing these other S/MIME types here, but we're only dealing with signedData for now. Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2020-05-13 00:29:31 +02:00
if (mimeobj)
g_object_unref (mimeobj);
if (sigs)
g_object_unref (sigs);
}
lib/database: index user headers. This essentially involves calling _notmuch_message_gen_terms once for each user defined header.
2019-02-26 03:10:29 +01:00
static notmuch_status_t
_notmuch_message_index_user_headers (notmuch_message_t *message, GMimeMessage *mime_message)
{
notmuch_database_t *notmuch = notmuch_message_get_database (message);
notmuch_string_map_iterator_t *iter = _notmuch_database_user_headers (notmuch);
for (; _notmuch_string_map_iterator_valid (iter);
_notmuch_string_map_iterator_move_to_next (iter)) {
const char *prefix_name = _notmuch_string_map_iterator_key (iter);
const char *header_name = _notmuch_string_map_iterator_value (iter);
const char *header = g_mime_object_get_header (GMIME_OBJECT (mime_message), header_name);
if (header)
_notmuch_message_gen_terms (message, prefix_name, header);
}
if (iter)
_notmuch_string_map_iterator_destroy (iter);
return NOTMUCH_STATUS_SUCCESS;
}
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
notmuch_status_t
_notmuch_message_index_file (notmuch_message_t *message,
crypto: index encrypted parts when indexopts try_decrypt is set. If we see index options that ask us to decrypt when indexing a message, and we encounter an encrypted part, we'll try to descend into it. If we can decrypt, we add the property index.decryption=success. If we can't decrypt (or recognize the encrypted type of mail), we add the property index.decryption=failure. Note that a single message may have both values of the "index.decryption" property: "success" and "failure". For example, consider a message that includes multiple layers of encryption. If we manage to decrypt the outer layer ("index.decryption=success"), but fail on the inner layer ("index.decryption=failure"). Because of the property name, this will be automatically cleared (and possibly re-set) during re-indexing. This means it will subsequently correspond to the actual semantics of the stored index.
2017-10-21 04:25:41 +02:00
notmuch_indexopts_t *indexopts,
lib: replace the header parser with gmime The notmuch library includes a full blown message header parser. Yet the same message headers are parsed by gmime during indexing. Switch to gmime parsing completely. These are the main changes: * Gmime stops header parsing at the first invalid header, and presumes the message body starts from there. The current parser is quite liberal in accepting broken headers. The change means we will be much pickier about accepting invalid messages. * The current parser converts tabs used in header folding to spaces. Gmime preserve the tabs. Due to a broken python library used in mailman, there are plenty of mailing lists that produce headers with tabs in header folding, and we'll see plenty of tabs. (This change has been mitigated in preparatory patches.) * For pure header parsing, the current parser is likely faster than gmime, which parses the whole message rather than just the headers. Since we parse the message and its headers using gmime for indexing anyway, this avoids and extra header parsing round when adding new messages. In case of duplicate messages, we'll end up parsing the full message although just headers would be sufficient. All in all this should still speed up 'notmuch new'. * Calls to notmuch_message_get_header() may be slightly slower than previously for headers that are not indexed in the database, due to parsing of the whole message. Within the notmuch code base, notmuch reply is the only such user.
2014-03-30 23:21:49 +02:00
notmuch_message_file_t *message_file)
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
{
lib: replace the header parser with gmime The notmuch library includes a full blown message header parser. Yet the same message headers are parsed by gmime during indexing. Switch to gmime parsing completely. These are the main changes: * Gmime stops header parsing at the first invalid header, and presumes the message body starts from there. The current parser is quite liberal in accepting broken headers. The change means we will be much pickier about accepting invalid messages. * The current parser converts tabs used in header folding to spaces. Gmime preserve the tabs. Due to a broken python library used in mailman, there are plenty of mailing lists that produce headers with tabs in header folding, and we'll see plenty of tabs. (This change has been mitigated in preparatory patches.) * For pure header parsing, the current parser is likely faster than gmime, which parses the whole message rather than just the headers. Since we parse the message and its headers using gmime for indexing anyway, this avoids and extra header parsing round when adding new messages. In case of duplicate messages, we'll end up parsing the full message although just headers would be sufficient. All in all this should still speed up 'notmuch new'. * Calls to notmuch_message_get_header() may be slightly slower than previously for headers that are not indexed in the database, due to parsing of the whole message. Within the notmuch code base, notmuch reply is the only such user.
2014-03-30 23:21:49 +02:00
GMimeMessage *mime_message;
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
InternetAddressList *addresses;
lib/cli: replace use of g_mime_message_get_sender This function changes semantics in gmime-3.0 so make a new function that provides the same functionality in both
2017-05-06 04:26:57 +02:00
const char *subject;
lib: replace the header parser with gmime The notmuch library includes a full blown message header parser. Yet the same message headers are parsed by gmime during indexing. Switch to gmime parsing completely. These are the main changes: * Gmime stops header parsing at the first invalid header, and presumes the message body starts from there. The current parser is quite liberal in accepting broken headers. The change means we will be much pickier about accepting invalid messages. * The current parser converts tabs used in header folding to spaces. Gmime preserve the tabs. Due to a broken python library used in mailman, there are plenty of mailing lists that produce headers with tabs in header folding, and we'll see plenty of tabs. (This change has been mitigated in preparatory patches.) * For pure header parsing, the current parser is likely faster than gmime, which parses the whole message rather than just the headers. Since we parse the message and its headers using gmime for indexing anyway, this avoids and extra header parsing round when adding new messages. In case of duplicate messages, we'll end up parsing the full message although just headers would be sufficient. All in all this should still speed up 'notmuch new'. * Calls to notmuch_message_get_header() may be slightly slower than previously for headers that are not indexed in the database, due to parsing of the whole message. Within the notmuch code base, notmuch reply is the only such user.
2014-03-30 23:21:49 +02:00
notmuch_status_t status;
indexing: record protected subject when indexing cleartext When indexing the cleartext of an encrypted message, record any protected subject in the database, which should make it findable and visible in search. Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-28 00:40:28 +02:00
_notmuch_message_crypto_t *msg_crypto;
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
lib: replace the header parser with gmime The notmuch library includes a full blown message header parser. Yet the same message headers are parsed by gmime during indexing. Switch to gmime parsing completely. These are the main changes: * Gmime stops header parsing at the first invalid header, and presumes the message body starts from there. The current parser is quite liberal in accepting broken headers. The change means we will be much pickier about accepting invalid messages. * The current parser converts tabs used in header folding to spaces. Gmime preserve the tabs. Due to a broken python library used in mailman, there are plenty of mailing lists that produce headers with tabs in header folding, and we'll see plenty of tabs. (This change has been mitigated in preparatory patches.) * For pure header parsing, the current parser is likely faster than gmime, which parses the whole message rather than just the headers. Since we parse the message and its headers using gmime for indexing anyway, this avoids and extra header parsing round when adding new messages. In case of duplicate messages, we'll end up parsing the full message although just headers would be sufficient. All in all this should still speed up 'notmuch new'. * Calls to notmuch_message_get_header() may be slightly slower than previously for headers that are not indexed in the database, due to parsing of the whole message. Within the notmuch code base, notmuch reply is the only such user.
2014-03-30 23:21:49 +02:00
status = _notmuch_message_file_get_mime_message (message_file,
&mime_message);
if (status)
return status;
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
lib/cli: replace use of g_mime_message_get_sender This function changes semantics in gmime-3.0 so make a new function that provides the same functionality in both
2017-05-06 04:26:57 +02:00
addresses = g_mime_message_get_from (mime_message);
_notmuch_message_index_file: unref (free) address lists from gmime. Apparently as of GMime 2.4, you don't need to call internet_address_list_destroy anymore, but you still need to call g_object_unref (from the GMime Changelog). On the medium performance corpus, valgrind shows "possibly lost" leakage in "notmuch new" dropping from 7M to 300k.
2012-12-11 04:33:40 +01:00
if (addresses) {
_index_address_list (message, "from", addresses);
}
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
addresses = g_mime_message_get_all_recipients (mime_message);
_notmuch_message_index_file: unref (free) address lists from gmime. Apparently as of GMime 2.4, you don't need to call internet_address_list_destroy anymore, but you still need to call g_object_unref (from the GMime Changelog). On the medium performance corpus, valgrind shows "possibly lost" leakage in "notmuch new" dropping from 7M to 300k.
2012-12-11 04:33:40 +01:00
if (addresses) {
_index_address_list (message, "to", addresses);
g_object_unref (addresses);
}
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
subject = g_mime_message_get_subject (mime_message);
_notmuch_message_gen_terms (message, "subject", subject);
lib/database: index user headers. This essentially involves calling _notmuch_message_gen_terms once for each user defined header.
2019-02-26 03:10:29 +01:00
status = _notmuch_message_index_user_headers (message, mime_message);
indexing: record protected subject when indexing cleartext When indexing the cleartext of an encrypted message, record any protected subject in the database, which should make it findable and visible in search. Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
2019-05-28 00:40:28 +02:00
msg_crypto = _notmuch_message_crypto_new (NULL);
_index_mime_part (message, indexopts, g_mime_message_get_mime_part (mime_message), msg_crypto);
if (msg_crypto && msg_crypto->payload_subject) {
_notmuch_message_gen_terms (message, "subject", msg_crypto->payload_subject);
_notmuch_message_update_subject (message, msg_crypto->payload_subject);
}
talloc_free (msg_crypto);
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
lib: replace the header parser with gmime The notmuch library includes a full blown message header parser. Yet the same message headers are parsed by gmime during indexing. Switch to gmime parsing completely. These are the main changes: * Gmime stops header parsing at the first invalid header, and presumes the message body starts from there. The current parser is quite liberal in accepting broken headers. The change means we will be much pickier about accepting invalid messages. * The current parser converts tabs used in header folding to spaces. Gmime preserve the tabs. Due to a broken python library used in mailman, there are plenty of mailing lists that produce headers with tabs in header folding, and we'll see plenty of tabs. (This change has been mitigated in preparatory patches.) * For pure header parsing, the current parser is likely faster than gmime, which parses the whole message rather than just the headers. Since we parse the message and its headers using gmime for indexing anyway, this avoids and extra header parsing round when adding new messages. In case of duplicate messages, we'll end up parsing the full message although just headers would be sufficient. All in all this should still speed up 'notmuch new'. * Calls to notmuch_message_get_header() may be slightly slower than previously for headers that are not indexed in the database, due to parsing of the whole message. Within the notmuch code base, notmuch reply is the only such user.
2014-03-30 23:21:49 +02:00
return NOTMUCH_STATUS_SUCCESS;
Add full-text indexing using the GMime library for parsing. This is based on the old notmuch-index-message.cc from early in the history of notmuch, but considerably cleaned up now that we have some experience with Xapian and know just what we want to index, (rather than just blindly trying to index exactly what sup does). This does slow down notmuch_database_add_message a *lot*, but I've got some ideas for getting some time back.
2009-10-28 18:42:07 +01:00
}
Reference in a new issue Copy permalink