debian/notmuch
1
0
Fork 0
mirror of https://git.notmuchmail.org/git/notmuch synced 2024-11-22 10:58:10 +01:00
Code Issues Projects Releases Packages Wiki Activity Actions
notmuch/emacs/notmuch-parser.el

195 lines
6.9 KiB
EmacsLisp
Raw Permalink Normal View History

emacs: use lexical-bindings in all libraries Doing so causes many new compile warnings. Some of these warnings concern genuine changes in behavior that have to be addressed right away. Many other warnings are due to unused variables. Nothing has changed here, except that the byte-compiler can now detect these pre-existing and harmless issues. We delay addressing these issues so that we can focus on the important ones here. A third group of warnings concern arguments that are not actually used inside the function but which cannot be removed because the functions signature is dictated by some outside convention. Silencing these warning is also delayed until subsequent commits.
2021-01-10 15:00:47 +01:00
;;; notmuch-parser.el --- streaming S-expression parser -*- lexical-binding: t -*-
emacs: Streaming S-expression parser This provides the same interface as the streaming JSON parser, but reads S-expressions incrementally. The only difference is that the `notmuch-sexp-parse-partial-list' helper does not handle interleaved error messages (since we now have the ability to separate these out at the invocation level), so it no longer takes an error function and does not need to do the horrible resynchronization that the JSON parser had to. Some implementation improvements have been made over the JSON parser. This uses a vector instead of a list for the parser data structure, since this allows faster access to elements (and modern versions of Emacs handle storage of small vectors efficiently). Private functions follow the "prefix--name" convention. And the implementation is much simpler overall because S-expressions are much easier to parse.
2013-06-01 02:40:06 +02:00
;;
;; Copyright © Austin Clements
;;
;; This file is part of Notmuch.
;;
;; Notmuch is free software: you can redistribute it and/or modify it
;; under the terms of the GNU General Public License as published by
;; the Free Software Foundation, either version 3 of the License, or
;; (at your option) any later version.
;;
;; Notmuch is distributed in the hope that it will be useful, but
;; WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
;; General Public License for more details.
;;
;; You should have received a copy of the GNU General Public License
Use https instead of http where possible Many of the external links found in the notmuch source can be resolved using https instead of http. This changeset addresses as many as i could find, without touching the e-mail corpus or expected outputs found in tests.
2016-06-02 18:26:14 +02:00
;; along with Notmuch. If not, see <https://www.gnu.org/licenses/>.
emacs: Streaming S-expression parser This provides the same interface as the streaming JSON parser, but reads S-expressions incrementally. The only difference is that the `notmuch-sexp-parse-partial-list' helper does not handle interleaved error messages (since we now have the ability to separate these out at the invocation level), so it no longer takes an error function and does not need to do the horrible resynchronization that the JSON parser had to. Some implementation improvements have been made over the JSON parser. This uses a vector instead of a list for the parser data structure, since this allows faster access to elements (and modern versions of Emacs handle storage of small vectors efficiently). Private functions follow the "prefix--name" convention. And the implementation is much simpler overall because S-expressions are much easier to parse.
2013-06-01 02:40:06 +02:00
;;
;; Authors: Austin Clements <aclements@csail.mit.edu>
emacs: Fix packaging Refer to (info "(elisp) Library Headers") for package conventions.
2016-04-13 09:58:47 +02:00
;;; Code:
emacs: improve how cl-lib and pcase are required We need to load `cl-lib' at run-time because we use more from it than just macros. Never-the-less many, but not all libraries required it only at compile-time, which we got away with because at least some libraries already required it at run-time as well. We use `cl-lib' and (currently to a lesser extend) `pcase' throughout the code-base, which means that we should require these features in most libraries. In the past we tried to only require these features in just the libraries that actually need them, without fully succeeding. We did not succeed in doing so because that means we would have to check every time that we use a function from these features whether they are already being required in the current library. An alternative would be to add the `require' forms at the top of every library but that is a bit annoying too. In order to make sure that these features are loaded when needed but also to keep the noise down we only require them in "notmuch-lib.el", which most other libraries require, and in most of the few libraries that do not do so, namely "notmuch-draft.el", "notmuch-message.el" and "notmuch-parser.el". ("coolj.el", "make-deps.el", various generated libraries, and "notmuch-compat.el" are left touched.)
2021-01-10 15:01:07 +01:00
(require 'cl-lib)
(require 'pcase)
emacs: make subr-x available in all libraries Like `cl-lib' and `pcase', which are already available in all libraries, `subr-x' also provided many useful functions that we would like to use. Making `subr-x' available in every library from the get-go means that we can use the functions it defines without having to double check every single time, whether the feature is already available in the current library.
2021-01-10 15:01:08 +01:00
(require 'subr-x)
emacs: Streaming S-expression parser This provides the same interface as the streaming JSON parser, but reads S-expressions incrementally. The only difference is that the `notmuch-sexp-parse-partial-list' helper does not handle interleaved error messages (since we now have the ability to separate these out at the invocation level), so it no longer takes an error function and does not need to do the horrible resynchronization that the JSON parser had to. Some implementation improvements have been made over the JSON parser. This uses a vector instead of a list for the parser data structure, since this allows faster access to elements (and modern versions of Emacs handle storage of small vectors efficiently). Private functions follow the "prefix--name" convention. And the implementation is much simpler overall because S-expressions are much easier to parse.
2013-06-01 02:40:06 +02:00
(defun notmuch-sexp-create-parser ()
"Return a new streaming S-expression parser.
This parser is designed to incrementally read an S-expression
whose structure is known to the caller. Like a typical
S-expression parsing interface, it provides a function to read a
complete S-expression from the input. However, it extends this
with an additional function that requires the next value in the
input to be a list and descends into it, allowing its elements to
be read one at a time or further descended into. Both functions
can return 'retry to indicate that not enough input is available.
The parser always consumes input from point in the current
buffer. Hence, the caller is allowed to delete any data before
point and may resynchronize after an error by moving point."
(vector 'notmuch-sexp-parser
emacs: Various cosmetic changes
2020-08-08 13:49:53 +02:00
0 ; List depth
nil ; Partial parse position marker
nil)) ; Partial parse state
emacs: Streaming S-expression parser This provides the same interface as the streaming JSON parser, but reads S-expressions incrementally. The only difference is that the `notmuch-sexp-parse-partial-list' helper does not handle interleaved error messages (since we now have the ability to separate these out at the invocation level), so it no longer takes an error function and does not need to do the horrible resynchronization that the JSON parser had to. Some implementation improvements have been made over the JSON parser. This uses a vector instead of a list for the parser data structure, since this allows faster access to elements (and modern versions of Emacs handle storage of small vectors efficiently). Private functions follow the "prefix--name" convention. And the implementation is much simpler overall because S-expressions are much easier to parse.
2013-06-01 02:40:06 +02:00
(defmacro notmuch-sexp--depth (sp) `(aref ,sp 1))
(defmacro notmuch-sexp--partial-pos (sp) `(aref ,sp 2))
(defmacro notmuch-sexp--partial-state (sp) `(aref ,sp 3))
(defun notmuch-sexp-read (sp)
"Consume and return the value at point in the current buffer.
Returns 'retry if there is insufficient input to parse a complete
value (though it may still move point over whitespace). If the
parser is currently inside a list and the next token ends the
list, this moves point just past the terminator and returns 'end.
Otherwise, this moves point to just past the end of the value and
returns the value."
(skip-chars-forward " \n\r\t")
(cond ((eobp) 'retry)
((= (char-after) ?\))
;; We've reached the end of a list
(if (= (notmuch-sexp--depth sp) 0)
;; .. but we weren't in a list. Let read signal the
;; error to be consistent with all other code paths.
(read (current-buffer))
;; Go up a level and return an end token
emacs: Use `cl-lib' instead of deprecated `cl' Starting with Emacs 27 the old `cl' implementation is finally considered obsolete. Previously its use was strongly discouraged at run-time but one was still allowed to use it at compile-time. For the most part the transition is very simple and boils down to adding the "cl-" prefix to some symbols. A few replacements do not follow that simple pattern; e.g. `first' is replaced with `car', even though the alias `cl-first' exists, because the latter is not idiomatic emacs-lisp. In a few cases we start using `pcase-let' or `pcase-lambda' instead of renaming e.g. `first' to `car'. That way we can remind the reader of the meaning of the various parts of the data that is being deconstructed. An obsolete `lexical-let' and a `lexical-let*' are replaced with their regular variants `let' and `let*' even though we do not at the same time enable `lexical-binding' for that file. That is the right thing to do because it does not actually make a difference in those cases whether lexical bindings are used or not, and because this should be enabled in a separate commit. We need to explicitly depend on the `cl-lib' package because Emacs 24.1 and 24.2 lack that library. When using these releases we end up using the backport from GNU Elpa. We need to explicitly require the `pcase' library because `pcase-dolist' was not autoloaded until Emacs 25.1.
2020-04-25 22:18:07 +02:00
(cl-decf (notmuch-sexp--depth sp))
emacs: Streaming S-expression parser This provides the same interface as the streaming JSON parser, but reads S-expressions incrementally. The only difference is that the `notmuch-sexp-parse-partial-list' helper does not handle interleaved error messages (since we now have the ability to separate these out at the invocation level), so it no longer takes an error function and does not need to do the horrible resynchronization that the JSON parser had to. Some implementation improvements have been made over the JSON parser. This uses a vector instead of a list for the parser data structure, since this allows faster access to elements (and modern versions of Emacs handle storage of small vectors efficiently). Private functions follow the "prefix--name" convention. And the implementation is much simpler overall because S-expressions are much easier to parse.
2013-06-01 02:40:06 +02:00
(forward-char)
'end))
((= (char-after) ?\()
;; We're at the beginning of a list. If we haven't started
;; a partial parse yet, attempt to read the list in its
;; entirety. If this fails, or we've started a partial
;; parse, extend the partial parse to figure out when we
;; have a complete list.
(catch 'return
emacs: Use 'unless' instead of 'when' and 'not' Also use 'unless' in a few cases where previously 'if' was used with 'not' but without an ELSE part.
2020-08-08 13:49:42 +02:00
(unless (notmuch-sexp--partial-state sp)
emacs: Streaming S-expression parser This provides the same interface as the streaming JSON parser, but reads S-expressions incrementally. The only difference is that the `notmuch-sexp-parse-partial-list' helper does not handle interleaved error messages (since we now have the ability to separate these out at the invocation level), so it no longer takes an error function and does not need to do the horrible resynchronization that the JSON parser had to. Some implementation improvements have been made over the JSON parser. This uses a vector instead of a list for the parser data structure, since this allows faster access to elements (and modern versions of Emacs handle storage of small vectors efficiently). Private functions follow the "prefix--name" convention. And the implementation is much simpler overall because S-expressions are much easier to parse.
2013-06-01 02:40:06 +02:00
(let ((start (point)))
(condition-case nil
(throw 'return (read (current-buffer)))
(end-of-file (goto-char start)))))
;; Extend the partial parse
(let (is-complete)
(save-excursion
(let* ((new-state (parse-partial-sexp
(or (notmuch-sexp--partial-pos sp) (point))
(point-max) 0 nil
(notmuch-sexp--partial-state sp)))
;; A complete value is available if we've
;; reached depth 0.
emacs: Use `cl-lib' instead of deprecated `cl' Starting with Emacs 27 the old `cl' implementation is finally considered obsolete. Previously its use was strongly discouraged at run-time but one was still allowed to use it at compile-time. For the most part the transition is very simple and boils down to adding the "cl-" prefix to some symbols. A few replacements do not follow that simple pattern; e.g. `first' is replaced with `car', even though the alias `cl-first' exists, because the latter is not idiomatic emacs-lisp. In a few cases we start using `pcase-let' or `pcase-lambda' instead of renaming e.g. `first' to `car'. That way we can remind the reader of the meaning of the various parts of the data that is being deconstructed. An obsolete `lexical-let' and a `lexical-let*' are replaced with their regular variants `let' and `let*' even though we do not at the same time enable `lexical-binding' for that file. That is the right thing to do because it does not actually make a difference in those cases whether lexical bindings are used or not, and because this should be enabled in a separate commit. We need to explicitly depend on the `cl-lib' package because Emacs 24.1 and 24.2 lack that library. When using these releases we end up using the backport from GNU Elpa. We need to explicitly require the `pcase' library because `pcase-dolist' was not autoloaded until Emacs 25.1.
2020-04-25 22:18:07 +02:00
(depth (car new-state)))
(cl-assert (>= depth 0))
emacs: Streaming S-expression parser This provides the same interface as the streaming JSON parser, but reads S-expressions incrementally. The only difference is that the `notmuch-sexp-parse-partial-list' helper does not handle interleaved error messages (since we now have the ability to separate these out at the invocation level), so it no longer takes an error function and does not need to do the horrible resynchronization that the JSON parser had to. Some implementation improvements have been made over the JSON parser. This uses a vector instead of a list for the parser data structure, since this allows faster access to elements (and modern versions of Emacs handle storage of small vectors efficiently). Private functions follow the "prefix--name" convention. And the implementation is much simpler overall because S-expressions are much easier to parse.
2013-06-01 02:40:06 +02:00
(if (= depth 0)
;; Reset partial parse state
(setf (notmuch-sexp--partial-state sp) nil
(notmuch-sexp--partial-pos sp) nil
is-complete t)
;; Update partial parse state
(setf (notmuch-sexp--partial-state sp) new-state
(notmuch-sexp--partial-pos sp) (point-marker)))))
(if is-complete
(read (current-buffer))
'retry))))
(t
;; Attempt to read a non-compound value
(let ((start (point)))
(condition-case nil
(let ((val (read (current-buffer))))
;; We got what looks like a complete read, but if
;; we reached the end of the buffer in the process,
;; we may not actually have all of the input we
;; need (unless it's a string, which is delimited).
(if (or (stringp val) (not (eobp)))
val
;; We can't be sure the input was complete
(goto-char start)
'retry))
(end-of-file
(goto-char start)
'retry))))))
(defun notmuch-sexp-begin-list (sp)
"Parse the beginning of a list value and enter the list.
Returns 'retry if there is insufficient input to parse the
beginning of the list. If this is able to parse the beginning of
a list, it moves point past the token that opens the list and
returns t. Later calls to `notmuch-sexp-read' will return the
elements inside the list. If the input in buffer is not the
beginning of a list, throw invalid-read-syntax."
(skip-chars-forward " \n\r\t")
(cond ((eobp) 'retry)
((= (char-after) ?\()
(forward-char)
emacs: Use `cl-lib' instead of deprecated `cl' Starting with Emacs 27 the old `cl' implementation is finally considered obsolete. Previously its use was strongly discouraged at run-time but one was still allowed to use it at compile-time. For the most part the transition is very simple and boils down to adding the "cl-" prefix to some symbols. A few replacements do not follow that simple pattern; e.g. `first' is replaced with `car', even though the alias `cl-first' exists, because the latter is not idiomatic emacs-lisp. In a few cases we start using `pcase-let' or `pcase-lambda' instead of renaming e.g. `first' to `car'. That way we can remind the reader of the meaning of the various parts of the data that is being deconstructed. An obsolete `lexical-let' and a `lexical-let*' are replaced with their regular variants `let' and `let*' even though we do not at the same time enable `lexical-binding' for that file. That is the right thing to do because it does not actually make a difference in those cases whether lexical bindings are used or not, and because this should be enabled in a separate commit. We need to explicitly depend on the `cl-lib' package because Emacs 24.1 and 24.2 lack that library. When using these releases we end up using the backport from GNU Elpa. We need to explicitly require the `pcase' library because `pcase-dolist' was not autoloaded until Emacs 25.1.
2020-04-25 22:18:07 +02:00
(cl-incf (notmuch-sexp--depth sp))
emacs: Streaming S-expression parser This provides the same interface as the streaming JSON parser, but reads S-expressions incrementally. The only difference is that the `notmuch-sexp-parse-partial-list' helper does not handle interleaved error messages (since we now have the ability to separate these out at the invocation level), so it no longer takes an error function and does not need to do the horrible resynchronization that the JSON parser had to. Some implementation improvements have been made over the JSON parser. This uses a vector instead of a list for the parser data structure, since this allows faster access to elements (and modern versions of Emacs handle storage of small vectors efficiently). Private functions follow the "prefix--name" convention. And the implementation is much simpler overall because S-expressions are much easier to parse.
2013-06-01 02:40:06 +02:00
t)
(t
;; Skip over the bad character like `read' does
(forward-char)
(signal 'invalid-read-syntax (list (string (char-before)))))))
(defvar notmuch-sexp--parser nil
"The buffer-local notmuch-sexp-parser instance.
Used by `notmuch-sexp-parse-partial-list'.")
(defvar notmuch-sexp--state nil
"The buffer-local `notmuch-sexp-parse-partial-list' state.")
(defun notmuch-sexp-parse-partial-list (result-function result-buffer)
"Incrementally parse an S-expression list from the current buffer.
This function consumes an S-expression list from the current
buffer, applying RESULT-FUNCTION in RESULT-BUFFER to each
complete value in the list. It operates incrementally and should
be called whenever the input buffer has been extended with
additional data. The caller just needs to ensure it does not
move point in the input buffer."
;; Set up the initial state
(unless (local-variable-p 'notmuch-sexp--parser)
emacs: use setq-local It is available since Emacs 24.3 and we require at least Emacs 25. It makes the variable buffer-local if it isn't always buffer-local anyway.
2020-11-16 22:28:39 +01:00
(setq-local notmuch-sexp--parser (notmuch-sexp-create-parser))
(setq-local notmuch-sexp--state 'begin))
emacs: Streaming S-expression parser This provides the same interface as the streaming JSON parser, but reads S-expressions incrementally. The only difference is that the `notmuch-sexp-parse-partial-list' helper does not handle interleaved error messages (since we now have the ability to separate these out at the invocation level), so it no longer takes an error function and does not need to do the horrible resynchronization that the JSON parser had to. Some implementation improvements have been made over the JSON parser. This uses a vector instead of a list for the parser data structure, since this allows faster access to elements (and modern versions of Emacs handle storage of small vectors efficiently). Private functions follow the "prefix--name" convention. And the implementation is much simpler overall because S-expressions are much easier to parse.
2013-06-01 02:40:06 +02:00
(let (done)
(while (not done)
emacs: Use `cl-lib' instead of deprecated `cl' Starting with Emacs 27 the old `cl' implementation is finally considered obsolete. Previously its use was strongly discouraged at run-time but one was still allowed to use it at compile-time. For the most part the transition is very simple and boils down to adding the "cl-" prefix to some symbols. A few replacements do not follow that simple pattern; e.g. `first' is replaced with `car', even though the alias `cl-first' exists, because the latter is not idiomatic emacs-lisp. In a few cases we start using `pcase-let' or `pcase-lambda' instead of renaming e.g. `first' to `car'. That way we can remind the reader of the meaning of the various parts of the data that is being deconstructed. An obsolete `lexical-let' and a `lexical-let*' are replaced with their regular variants `let' and `let*' even though we do not at the same time enable `lexical-binding' for that file. That is the right thing to do because it does not actually make a difference in those cases whether lexical bindings are used or not, and because this should be enabled in a separate commit. We need to explicitly depend on the `cl-lib' package because Emacs 24.1 and 24.2 lack that library. When using these releases we end up using the backport from GNU Elpa. We need to explicitly require the `pcase' library because `pcase-dolist' was not autoloaded until Emacs 25.1.
2020-04-25 22:18:07 +02:00
(cl-case notmuch-sexp--state
emacs: Streaming S-expression parser This provides the same interface as the streaming JSON parser, but reads S-expressions incrementally. The only difference is that the `notmuch-sexp-parse-partial-list' helper does not handle interleaved error messages (since we now have the ability to separate these out at the invocation level), so it no longer takes an error function and does not need to do the horrible resynchronization that the JSON parser had to. Some implementation improvements have been made over the JSON parser. This uses a vector instead of a list for the parser data structure, since this allows faster access to elements (and modern versions of Emacs handle storage of small vectors efficiently). Private functions follow the "prefix--name" convention. And the implementation is much simpler overall because S-expressions are much easier to parse.
2013-06-01 02:40:06 +02:00
(begin
;; Enter the list
(if (eq (notmuch-sexp-begin-list notmuch-sexp--parser) 'retry)
(setq done t)
(setq notmuch-sexp--state 'result)))
(result
;; Parse a result
(let ((result (notmuch-sexp-read notmuch-sexp--parser)))
emacs: Use `cl-lib' instead of deprecated `cl' Starting with Emacs 27 the old `cl' implementation is finally considered obsolete. Previously its use was strongly discouraged at run-time but one was still allowed to use it at compile-time. For the most part the transition is very simple and boils down to adding the "cl-" prefix to some symbols. A few replacements do not follow that simple pattern; e.g. `first' is replaced with `car', even though the alias `cl-first' exists, because the latter is not idiomatic emacs-lisp. In a few cases we start using `pcase-let' or `pcase-lambda' instead of renaming e.g. `first' to `car'. That way we can remind the reader of the meaning of the various parts of the data that is being deconstructed. An obsolete `lexical-let' and a `lexical-let*' are replaced with their regular variants `let' and `let*' even though we do not at the same time enable `lexical-binding' for that file. That is the right thing to do because it does not actually make a difference in those cases whether lexical bindings are used or not, and because this should be enabled in a separate commit. We need to explicitly depend on the `cl-lib' package because Emacs 24.1 and 24.2 lack that library. When using these releases we end up using the backport from GNU Elpa. We need to explicitly require the `pcase' library because `pcase-dolist' was not autoloaded until Emacs 25.1.
2020-04-25 22:18:07 +02:00
(cl-case result
emacs: Streaming S-expression parser This provides the same interface as the streaming JSON parser, but reads S-expressions incrementally. The only difference is that the `notmuch-sexp-parse-partial-list' helper does not handle interleaved error messages (since we now have the ability to separate these out at the invocation level), so it no longer takes an error function and does not need to do the horrible resynchronization that the JSON parser had to. Some implementation improvements have been made over the JSON parser. This uses a vector instead of a list for the parser data structure, since this allows faster access to elements (and modern versions of Emacs handle storage of small vectors efficiently). Private functions follow the "prefix--name" convention. And the implementation is much simpler overall because S-expressions are much easier to parse.
2013-06-01 02:40:06 +02:00
(retry (setq done t))
(end (setq notmuch-sexp--state 'end))
(t (with-current-buffer result-buffer
(funcall result-function result))))))
(end
emacs: inline notmuch-sexp-eof into only caller This function had a few issues. - Neither its name nor the old comment before it is called made it clear what it does. - It took one argument but didn't do anything with it. - It's doc-string made a few claims, which are untrue and generally focused on details instead of that its purpose is.
2021-01-10 15:00:50 +01:00
;; Skip over trailing whitespace.
(skip-chars-forward " \n\r\t")
;; Any trailing data is unexpected.
(unless (eobp)
(error "Trailing garbage following expression"))
emacs: Streaming S-expression parser This provides the same interface as the streaming JSON parser, but reads S-expressions incrementally. The only difference is that the `notmuch-sexp-parse-partial-list' helper does not handle interleaved error messages (since we now have the ability to separate these out at the invocation level), so it no longer takes an error function and does not need to do the horrible resynchronization that the JSON parser had to. Some implementation improvements have been made over the JSON parser. This uses a vector instead of a list for the parser data structure, since this allows faster access to elements (and modern versions of Emacs handle storage of small vectors efficiently). Private functions follow the "prefix--name" convention. And the implementation is much simpler overall because S-expressions are much easier to parse.
2013-06-01 02:40:06 +02:00
(setq done t)))))
;; Clear out what we've parsed
(delete-region (point-min) (point)))
(provide 'notmuch-parser)
emacs: Fix packaging Refer to (info "(elisp) Library Headers") for package conventions.
2016-04-13 09:58:47 +02:00
;;; notmuch-parser.el ends here
Reference in a new issue Copy permalink