This commit fixes a problem with the old method of handling
here-documents. We used to ignore here-documents in many places where
they are valid (e.g., between Boolean operators). They are now valid
after any 'NEWLINE' token. To achieve this, the handling logic was
moved out of the parser and into the lexer. We now use a hook
mechanism to manage the necessary communication between the parser and
lexer.
* geesh/parser.scm (read-here-docs): Return strings instead of tokens.
(make-lexer): Add here-docs-hook as a keyword argument, and call it
after reading here-documents.
(make-parser): Let command-hooks transform commands, and remove
here-document handling rules.
(parse): Handle merging here-documents into commands.
* tests/parser.scm: Add tests for many less-intuitive here-document
locations.
* geesh/parser.scm (make-lexer): Remove the read-sh/bracketed and
read-sh/backquoted parameters.
(parse): New function.
(->command-list): New function.
(read-sh/bracketed, read-sh/backquoted, read-sh, read-sh-all):
Simplify by using the new functions.
* geesh/parser.scm (make-parser): When parsing for loops, make all
word-lists actual lists, and make an empty word-list with the 'in'
keyword yield an empty list rather than a reference to '$@'.
* tests/parser.scm: Update tests.
This should have been in 2b05199562, but
it was overlooked.
* geesh/parser.scm (make-parser): Splice in arguments in the redirect
case (to be consistent with the non-redirect case).
* tests/parser.scm: Add a test for this.
This change is already in the syntax document.
* geesh/parser.scm (make-parser): Rename '<sh-define>' to '<sh-defun>'
and use the function name directly instead of making it a singleton
list.
* tests/parser.scm: Update tests.
* .dir-locals.el: Update indentation.
The AST generated by the parser did not match the syntax document.
However, neither were right. This commit updates the syntax document
to describe a more consistent form, and fixes the code to follow it.
* doc/syntax.txt (word): Make the children of '<sh-cmd-sub>' a
possibly empty list of 'list's.
* geesh/lexer.scm (get-bracketed-command): Splice in children for
'<sh-cmd-sub>'.
(get-backquoted-command): Ditto.
* geesh/parser.scm (read-sh/bracketed): Make sure that result is
always a list.
(read-sh/backquoted): Ditto.
* tests/lexer.scm: Update tests accordingly.
* tests/parser.scm: Ditto.
Commands with prefix redirects and no arguments or assignments would
generate '<sh-exec-let>' forms with an empty list for bindings.
* geesh/parser.scm (make-parser): Check for and handle an empty
assignment list.
* tests/parser.scm: Add test.
Commit b807b72a7c changed the AST format
when reading single commands by mistake. It caused 'read-sh' to
return a singleton list.
* geesh/parser.scm (make-parser): Restore old behaviour.
(read-sh-all): Convert the output from the parser into a list.
* tests/parser.scm: Add tests to make sure 'read-sh-all' always
returns a list.
* geesh/parser.scm (read-sh-all): New public function for reading all
commands from a port.
(make-parser): Simplify AST forms when reading multiple commands.
* tests/parser.scm: Test it.
* geesh/parser.scm (map+fold): New function.
(merge-here-docs): New function.
(remove-quotes): New function.
(read-here-docs): New function.
(make-lexer): Handle newlines and here-document operators specially,
and emit special 'HERE-DOC' and 'HERE-DOC-SEP' tokens.
(make-parser): Use new tokens to support here-documents.
* tests/parser.scm: Add tests for here-documents.
The right-hand side of a dup redirect ("<&" and ">&") may contain
expansions, so we cannot process it during parsing.
* geesh/parser.scm (process-dup-or-close-word): Remove function.
(make-parser): Return the raw word from a dup redirect instead of
trying to convert it into a number or a symbol.
* tests/parser.scm: Fix tests accordingly.
A reserved word should not be special when it is an argument to a
command. This commit makes the parser treat reserved words as normal
arguments. Note that this change exposed problems in many of the
parser tests, which relied on reserved words delimiting commands where
they should not. Those are now fixed.
* geesh/parser.scm (make-parser): Add reserved words to the default
'WORD*' rule, and use a new rule without reserved words for command
names.
* tests/parser.scm: Add a test for reserved words as arguments, and
fix old tests that relied on the old, incorrect behaviour.
* geesh/parser.scm (make-parser): Instead of using a nested list for
assignments, use a flat list.
* tests/parser.scm: Adjust existing test for this (which checks a
single assignment) and add a second test which checks multiple
assignments.
* geesh/parser.scm: New file.
* tests/parser.scm: New file.
* Makefile.am: Add them.
* .dir-locals.el: New file. Include indenting rules for Shell AST
forms and 'call-with-backquoted-input-port'.
Timothy Sample