Commit graph

785 commits

Author SHA1 Message Date
David Bremner
b0a11dbc38 lib/{open,message}: make some internal functions static
They are not used outside their file, so being extern seems like an oversight
2021-06-05 15:40:00 -03:00
David Bremner
748352693c lib/thread: add common prefix to debug messages, join lines
This will simplify filtering these message, e.g. in the test suite.
2021-05-23 08:01:38 -03:00
David Bremner
702635d5f6 Merge branch 'release' 2021-05-22 09:34:55 -03:00
David Bremner
3f4de98e7c lib/n_d_index_file: re-use thread-id of existing message
This prevents the message document getting multiple thread-id terms
when there are multiple files with the same message-id.

This change shifts some thread ids, requiring adjustments to other tests.
2021-05-22 09:08:02 -03:00
David Bremner
c84ccb70f3 Merge branch 'release' 2021-05-15 09:10:58 -03:00
David Bremner
b3258244c8 lib/open: restore default database path of $HOME/mail
Although this default worked for "notmuch config get", it didn't work
most other places. Restore the previous functionality, with the
wrinkle that XDG locations will shadow $HOME/mail if they exist.

This fixes a bug reported by Jack Kamm in id:87eeefdc8b.fsf@gmail.com
2021-05-15 08:40:05 -03:00
David Bremner
b1b6798588 lib/message: mark flag2tag as const
This table is intended to be immutable
2021-05-14 06:39:12 -03:00
David Bremner
929386fad9 lib/generate_thread_id: move static buffer to notmuch_database_t
Work towards the goal of concurrent access to different Xapian
databases from the same process.
2021-05-14 06:38:19 -03:00
David Bremner
8410be8e08 lib: make glib initialization thread-safe
In principle this could be done without depending on C++11 features,
but these features should be available since gcc 4.8.1, and this
localized usage is easy to replace if it turns out to be problematic
for portability.
2021-05-13 22:21:57 -03:00
David Bremner
393c92b042 lib/notmuch_database_reopen: reload some database metadata
In some uses of reopen, new documents and threads maybe have been
added, and e.g. compaction may have changed the uuid.
2021-05-12 08:40:04 -03:00
David Bremner
1040e7aa07 lib/config: expand relative paths when reading from database
This makes the treatment of relative paths consistent between the
database and config files.
2021-05-10 11:12:58 -03:00
David Bremner
31098c4ae4 lib/config: canonicalize paths relative to $HOME.
Prior to 0.32, notmuch had the (undocumented) behaviour that it
expanded a relative value of database.path with respect to $HOME. In
0.32 this was special cased for database.path but broken for
database.mail_root, which causes problems for at least notmuch-new
when database.path is set to a relative path.

The change in T030-config.sh reflects a user visible, but hopefully
harmless behaviour change; the expanded form of the paths will now be
printed by notmuch config.
2021-05-10 11:08:18 -03:00
David Bremner
5f80e106d6 lib/config: remove early free in _get_email_from_passwd_file
This (obvious) bug was caused by cut&pasting the code from
notmuch-config.c into the library and adding on a return at the end.
2021-04-24 12:11:45 -03:00
Đoàn Trần Công Danh
441a327051 compat: rename {,notmuch_}canonicalize_file_name
When compat canonicalize_file_name was introduced, it was limited to
C code only because it was used by C code only during that time.

>From 5ec6fd4d, (lib/open: check for split configuration when creating
database., 2021-02-16), lib/open.cc, which is C++, relies on the
existent of canonicalize_file_name.

However, we can't blindly enable canonicalize_file_name for C++ code,
because different implementation has different additional signature for
C++ and users can arbitrarily add -DHAVE_CANONICALIZE_FILE_NAME=0 to
{C,CXX}FLAGS.

Let's move our implementation into a util library.

Helped-by: Tomi Ollila <tomi.ollila@iki.fi>
Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
2021-04-24 08:07:00 -03:00
David Bremner
084e60d54a lib/n_d_index_file: check return value from _n_m_add_filename
Ignoring this return value seems like a bad idea in general, and in
particular it has been hiding one or more bugs related to handling
long directory names.
2021-04-18 10:02:20 -03:00
David Bremner
9ad19e4454 lib: directly traverse postlists in _n_message_delete
This is intended to fix the slow behaviour of "notmuch new" (and possibly
"notmuch reindex") when large numbers of files are deleted.

The underlying issue [1] seems to be the Xapian glass backend spending
a large amount of time in db.has_positions when running queries with
large-ish amounts of unflushed changes.

This commit removes two uses of Xapian queries [2], and replaces them with
an approximation of what Xapian would do after optimizing the
queries. This avoids the calls to has_positions (which are in any case
un-needed because we are only using boolean terms here).

[1] Thanks to "andres" on IRC for narrowing down the performance
bottleneck.

[2] Thanks to Olly Betts of Xapian fame for talking me a through a fix
that does not require people to update Xapian.
2021-04-18 09:50:36 -03:00
David Bremner
f5d4349921 lib: provide notmuch_config_path
Since the library searches in several locations for a config file, the
caller does not know which of these is chosen in the usual case of
passing NULL as a config file. This changes provides an API for the
caller to retrieve the name of the config file chosen. It will be
tested in a following commit.
2021-03-27 09:26:14 -03:00
David Bremner
217f819608 CLI+lib: detect missing database in split configurations.
Eventually we want to do all opening of databases in the top
level (main function). This means that detection of missing databases
needs to move out of subcommands. It also requires updating the
library to use the new NO_DATABASE status code.
2021-03-27 09:26:14 -03:00
David Bremner
2e39ce6eb5 lib: add NOTMUCH_STATUS_NO_DATABASE
This will allow more precise return values from various open related functions.
2021-03-27 09:26:14 -03:00
David Bremner
b76da87d29 CLI/config: migrate notmuch_config_open to new config
notmuch_config_open will be preserved in the medium term for use by
the commands that are manipulating the config file directly (config
and setup)
2021-03-27 09:26:14 -03:00
David Bremner
b25e57e6cb lib/open: canonicalize relative path read from config file
This matches functionality in the the CLI function
notmuch_config_get_database_path, which was previously used in the CLI
code for all calls to open a database.
2021-03-27 09:26:14 -03:00
David Bremner
2e9ee47072 lib/config: set default for primary user email
This is mainly copying code from the CLI into the lib. The CLI copy
will be deleted in a later commit.
2021-03-27 09:26:14 -03:00
David Bremner
e81dc2072f lib/config: set defaults for user full name
This just copies code from from the CLI into the library. New test
infrastructure is needed because apparently we have never tested this
code path.
2021-03-27 09:26:14 -03:00
David Bremner
863b243185 lib/config: add config_pairs iterators
The layer of shims here seems a bit wasteful compared to just calling
the corresponding string map functions directly, but it allows control
over the API (calling with notmuch_database_t *) and flexibility for
future changes.
2021-03-27 09:26:14 -03:00
David Bremner
0c6db22930 lib/config: add notmuch_config_get_values_string
This is to support the less common (at least in the notmuch codebase)
case of accessing a ;-delimited list config value with an arbitrary
string key.
2021-03-27 09:26:14 -03:00
David Bremner
2fc40e24de lib: provide notmuch_database_load_config
This is mainly targetted at notmuch-config.c, or similar applications
that don't necessarily need both a configuration file and a database
to exist.
2021-03-27 09:26:14 -03:00
David Bremner
a3444e873f lib/open: pull _load_key_file out of _choose_database_path
Although this increases code duplication, it also increases
flexibility in handling various combinations of missing config file
and missing database.
2021-03-27 09:26:14 -03:00
David Bremner
a6a580e616 lib: add missing status strings 2021-03-27 09:26:14 -03:00
David Bremner
4a8d67e357 lib/open: fix leaks calling _trial_open
_trial_open can't know if the PATH_ERROR return value will cause the
error message to be returned from the library, so it's up the caller
to clean up if not.
2021-03-27 09:12:28 -03:00
David Bremner
a7873df331 lib/config: add configuration variable for backup directory
Like the hook directory, we primarily need a way to communicate this
directory between various components, but we may as well let the user
configure it.

Most of the diff is generalizing choose_hook_dir to work for both
backup and hook directories.
2021-03-20 07:51:36 -03:00
David Bremner
e3a4abc513 lib/open: fix hook directory calculation in split configuration
Choose sibling directory of xapian database, as .notmuch may not
exist.

libgen.h is already used in debugger.c, so it is not a new dependency
/ potential portability problem.
2021-03-20 07:50:13 -03:00
David Bremner
b30a59157d lib/compact: enable split config
This promotes _choose_xapian_path from static to extern linkage in
order to share between open.cc and database.cc.
2021-03-20 07:50:06 -03:00
David Bremner
c82554193d lib/open: support XDG_DATA_HOME as a fallback database location.
This changes some error reporting, either intentionally by reporting
the highest level missing directory, or by side effect from looking in
XDG locations when given null database location.
2021-03-20 07:43:09 -03:00
David Bremner
5ec6fd4dcf lib/open: check for split configuration when creating database.
The main functionality will be tested when notmuch-new is converted to
support split configuration. Here only the somewhat odd case of split
mail root which is actually symlinked to the database path is tested.
2021-03-20 07:41:04 -03:00
David Bremner
e823d05ae6 lib: support splitting mail from database location.
Introduce a new configuration value for the mail root, and use it to
locate mail messages in preference to the database.path (which
previously implied the mail messages were also in this location.

Initially only a subset of the CLI is tested in a split
configuration. Further changes will be needed for the remainder of the
CLI to work in split configurations.
2021-03-20 07:39:12 -03:00
David Bremner
986056bdbc lib/open: Use check for existing database by trial opening
This is a bit heavyweight for now, but it will make more sense when we
check multiple locations for the Xapian database.
2021-03-20 07:38:15 -03:00
David Bremner
aa59424812 lib/open: use _finish_open in n_d_create_with_config
This avoids reading the configuration file twice.
2021-03-20 07:33:25 -03:00
David Bremner
1be79fff50 lib/open: factor out the second half of n_d_open_with_config
The idea is to allow reuse in n_d_create_with_config. This is
primarily code movement, with some changes in error messages to reduce
the number of input parameters.
2021-03-20 07:30:39 -03:00
David Bremner
cba540d6f5 lib/open: reuse directory checks from n_d_c_with_config
Make checks more uniform between creating new databases and opening
existing ones.
2021-03-20 07:24:59 -03:00
David Bremner
82aa3f4d27 lib/open: factor out library initialization
This is slightly more tidy, but more importantly it allows for re-use
of this code in n_d_create_with_config. That re-use will be crucial
when we no longer call n_d_open_with_config from
n_d_create_with_config.
2021-03-20 07:24:24 -03:00
David Bremner
6251e2bb9e lib: remove "path" from notmuch struct
This removes duplication between the struct element and the
configuration string_map entry. Create a simple wrapper for setting
the database path that makes sure the trailing / is stripped.
2021-03-20 07:23:40 -03:00
David Bremner
1383481d4a lib/open: allocate notmuch_t struct early
This gives more flexibility in restructuring the database opening
code.
2021-03-20 07:21:26 -03:00
David Bremner
793d4305d3 lib/open: support NOTMUCH_DATABASE environment variable
The additional code is trivial, but making sure we get the priority of
various options correct takes a few tests.
2021-03-19 22:06:44 -03:00
David Bremner
13efbd0e1c lib: support reopening databases for write access.
In the future Xapian will apparently support this more conveniently
for the cases other than READ_ONLY => READ_ONLY

Conceptually this function seems to fit better in lib/open.cc;
database.cc is still large enough that moving the function makes
sense.
2021-03-18 08:04:06 -03:00
David Bremner
f0717aa380 lib: save path of xapian database in notmuch struct.
This will allow re-opening in a different mode (read/write
vs. read-only) with current Xapian API. It will also prove useful when
updating the compact functions to support more flexible database
location.
2021-03-18 08:03:48 -03:00
David Bremner
1121299905 lib: publish API for notmuch_database_reopen
Include the (currently unused) mode argument which will specify which
mode to re-open the database in. Functionality and docs to be
finalized in a followup commit.
2021-03-18 08:03:36 -03:00
David Bremner
6967dcbb02 lib/open: free GKeyFile
This fixes a small-to-medium (depending on size of config file) memory
leak.
2021-03-13 09:02:36 -04:00
David Bremner
50092a0375 lib/config: free memory from traversing GKeyFile
This fixes a few small memory leaks.
2021-03-13 09:01:51 -04:00
David Bremner
6d5deb76ca lib/open: free value from g_key_file_get_value
This fixes a small memory leak.
2021-03-13 08:58:21 -04:00
David Bremner
12301392d4 lib/open: use local talloc context in n_d_create_with_config
This better matches the memory allocation semantics in
notmuch_database_open_with_config.
2021-03-13 08:49:49 -04:00