     [file version.texi not included]

Copyright © 2018 Rutger EW van Beusekom

Copyright © 2018 Jan (janneke) Nieuwenhuizen

Copyright © 2019 Timothy Sample

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.  A copy of the license is included in the section entitled ``GNU Free Documentation License''.

Gash
****

This document describes Gash version VERSION, a POSIX-compatible shell written in Guile Scheme.

Introduction
************

Gash is a POSIX-compatible shell written in Guile Scheme.  It has two goals: bootstrapping GNU Bash and exposing the shell to Scheme.  In the following sections we will unpack that definition and explain all of its constituent parts.

What is a POSIX-compatible shell?
=================================

When talking about operating systems, a shell refers to the day-to-day user interface that a person would use to interact with the system.  That's what you might learn in computer science school, but in popular usage ``shell'' refers to a very particular type of interface---the one that developed along with the Unix operating system.  The Unix shell is a command-line interface designed mostly for invoking programs and connecting programs together.  Although we provide a whirlwind tour of shell features in Using Gash, this manual is not a good general introduction to using a Unix-like shell. For that, try Introduction to the Command Line (http://write.flossmanuals.net/command-line/introduction/).

So now we know what a Unix shell is, but what is a POSIX-compatible shell?  Over time, Unix inspired many similar systems, all of which were mostly the same, but not quite compatible with one another.  What worked on one Unix-like operating system might not work on another.  To make it easier to write portable software, everyone got together and described a minimal set of features that all Unix-like systems should provide.  They called it the Portable Operating System Interface, or POSIX for short.  Being a POSIX-compatible shell means that Gash should have at least all of the features required by POSIX. (Note that Gash is merely POSIX-compatible and not POSIX-certified.  This means we try our best, but we have not gotten an official stamp of approval from The Open Group, who owns the POSIX trademark and administers a certification program.)

What is Guile Scheme?
=====================

Scheme is a cool programming language from the 70's.  It is a very conceptually simple dialect of Lisp---a cool programming language from the 50's.  Why are we interested in using such an old programming language?  There are two reasons.  The first is that even by modern standards, Scheme is a very expressive, high-level language. Writing in such a language is a pleasure.  The second reason is that Scheme's simplicity makes it well-suited to the bootstrapping task (see What of Bash and its bootstraps?).

Guile is a particular implementation of Scheme.  It is the official extension language of the GNU project.  There are many implementations of Scheme, but Guile is our choice mainly out of a desire to integrate nicely with the GNU Guix project, which is also written in Guile.

What of Bash and its bootstraps?
================================

One of the most important functions of Gash is to bootstrap GNU Bash (see What is Bash? (Bash Reference Manual)). Bootstrapping is, briefly, going from having no software to having software.  What makes it hard is that you usually use software to build software, and that snake has a habit of eating its own tail. For example, Bash uses shell scripts to describe how it should be built, so you need a shell to build a shell.  What if this other shell needs shell scripts too?  Then you have a problem.  Normally this problem goes unnoticed, because you already have a shell that you got from somebody else, and that somebody got their shell from somebody, and this process works its way back to the dawn of time---er, the dawn of Unix time, which is the early 70's---when the first shell emerged from the primordial soup.

Where you really run head-long into this problem is when you have a complete, fine-grained description of a system's software dependency graph, like they have over at the GNU Guix project.  Then, if you follow the arrows back as far as you can, you can't miss the fact that the first step in building the system is ``download many megabytes of inscrutable binary programs from the Internet.''  There are a handful of people working to change this, and Gash is a small part of this larger project.  For all the details, see the Bootstrappable Builds website (https://bootstrappable.org/).

What does Scheme want from the shell?
=====================================

There is a long history of mixing Scheme and the shell.  Most famously, there is Scsh (https://scsh.net/), which is a hybrid language that tries to combine the brevity of the shell with the flexibility of Scheme.  There are a few other projects that mix Scheme or Lisp with shell syntax, and many projects mixing the shell with other popular programming languages.  A common goal of many of these projects is to provide a way forward for shell scripts that have become so large they are difficult to maintain.  Having a language that has a similar syntax to shell scripts but provides more ways to structure code would allow these scripts to continue to grow cleanly.

On the other hand, it can be useful to provide a shell-like interface for Scheme environments.  For instance, the Guix System boots into Guile and if anything goes wrong, it presents the user with a Scheme REPL.  It would be nice to be able to drive Guile using shell syntax, since most of what you need to do in that context is manipulate files and execute external utilities.  Another Guix example is adding build-system phases that are just shell scripts. (This one is a little heretical, but I imagine it being attractive as Guix becomes more popular.)

Ultimately, the future of Gash is yet to be explored, but there are plenty of spaces it could gracefully move into.  Patches are welcome!

Using Gash
**********

Gash is a mostly complete POSIX-compatible shell, so it should mostly work as you would expect.  However, it is missing a few POSIX features (see Missing features), and does not really have any features beyond POSIX.  Specifically, many of the ``Bashisms'' that you may be familiar with are not available. (See Major Differences From The Bourne Shell (Bash Reference Manual).)

In the following we cover the major features that work very well in Gash.  You should feel confident using these features as you would in any other POSIX-compatible shell.

All kinds of command invocation should just work.  This includes setting temporary variables, redirects, and here-documents.  All of the POSIX-specified ``special'' built-ins are available except for times.  Only a handful of other built-ins are provided (like cd and pwd).

All of the control structures are available and full-featured.  This means for, while, until, if, and case.  Functions and subshells also work fine.

Field splitting and quoting is well-tested and should work even in exceptional cases, like "$@" and when $IFS is manipulated.

You can set variables and mark them read-only or exported.  Many special variables are available, and about half of the variable operators (like ${VARIABLE+alternate}) work.

Both types of command substitution work (that is, $(...) and `...`), and can even be nested.

In general, Gash should not surprise you too much, and should run most POSIX-compatible shell scripts.

Invoking Gash
=============

The gash command is the sh interpreter.

     gash OPTION... FILE

The OPTIONs can be among the following:

-c, --command=STRING
     Evaluate STRING, and then exit.

-e, --errexit
     Exit immediately on any error.

-h, --help
     Display help on invoking Gash, and then exit.

-p, --parse
     Rather than evaluate a script, parse it and output its syntax tree.

-v, --version
     Display the current version of Gash, and then exit.

-x, --xtrace
     Print each command that is executed.

Missing features
================

In this section, we cover some of the features that are currently missing in Gash.  Particularly, we look at the features specified by POSIX that have not yet been implemented.  This list is not exhaustive, but covers the most glaring omissions.

  * Arithmetic substitution.

  * Asynchronous commands and job control.

  * Alias creation and substitution.

  * Certain special variables.  This includes $ENV; all the localization variables like $LANG, $LC_*, and $NLSPATH; the parent process ID variable ($PPID); and the prompt variables ($PS*).

  * Tilde expansion.

  * Variable pattern operators and assertion operators.  This means that ${FOO%pattern} and the like do not work, and neither does ${FOO?}.

  * Multi-line commands from the readline interface.  If you press Enter in the middle of a command (e.g., from within an unfinished quotation) Gash will not ask for more input but rather treat it as a syntax error and exit with an error message.

These features will be added to a future version of Gash.  If you would like to contribute to Gash by implementing one of these features, please get in touch by writing to gash-devel@nongnu.org (mailto:gash-devel@nongnu.org).

Hacking with Gash
*****************

Have you ever wanted to write Scheme code that can understand and manipulate shell scripts on the fly?  Have you ever wanted your Scheme scripts to have the breezy convenience of a shell script?  You've come to the right place!  In this chapter we will look at all the tools that Gash offers for exploring and illuminating the dark canyon that separates Scheme and the shell.

Parsing shell scripts
=====================

If you're reading this, you probably have some familiarity with Scheme, and you have probably heard of Scheme's read procedure. Gash exports a very similar function from the (gash parser) module.  It's called read-sh, and as you may have guessed, it reads shell code instead of Scheme code.

With Scheme code, the internal and external representations of the code are basically the same, but this is not true with shell code. You will have to learn and understand Gash's particular internal representation of shell code to use it effectively.  Fortunately, it is designed to be familiar to a Scheme programmer, and we provide many examples here to get you up to speed quickly.  At the end of this section, a formal definition of the internal representation is given for reference.

read-sh [port] Read a complete shell command from the input port PORT if it is specified, or from the current input port if PORT is not specified.  A complete command is essentially something an interactive shell would be able to execute immediately without prompting for more input.  Technically, it corresponds to the complete_command production rule defined in the POSIX specification.

For convenience, we also export the following procedure.

read-sh-all [port] Read all complete shell commands from the input port PORT if it is specified, or from the current input port if PORT is not specified.  This works much like writing your own loop using read-sh, but it should be more efficient.  As a bonus, it ensures that the result is always a list of commands.

Internal representation examples
--------------------------------

The easiest way to understand Gash's internal representation of shell is through examples, so let's take a look at a few.  To simplify the examples, we are going abuse notation a little bit.  Instead of writing

     (call-with-input-string "SHELL-EXAMPLE" read-sh)
     =>
     RESULT

we will simply write

     SHELL-EXAMPLE
     =>
     RESULT

The major benefit of doing this is that we do not have to escape the shell examples to be proper Scheme strings.

Simple commands
...............

The parser can read simple commands and split them into their constituent parts:

     echo foo
     =>
     (<sh-exec> "echo" "foo")

     echo $FOO
     =>
     (<sh-exec> "echo" (<sh-ref> "FOO"))

     echo foo$BAR baz
     =>
     (<sh-exec> "echo" ("foo" (<sh-ref> "BAR")) "baz")

It preserves quotes, since they are significant for field splitting and escaping special characters:

     echo "$FOO"
     =>
     (<sh-exec> "echo" (<sh-quote> (<sh-ref> "FOO")))

     echo *
     =>
     (<sh-exec> "echo" "*")

     echo '*'
     =>
     (<sh-exec> "echo" (<sh-quote> "*"))

It represents redirects and temporary variable assignments with a nod to Scheme:

     CC=gcc CFLAGS='-O2 -g' make
     =>
     (<sh-exec-let> (("CC" "gcc")
                     ("CFLAGS" (<sh-quote> "-O2 -g")))
       "make")

     wc < gash.texi
     =>
     (<sh-with-redirects> ((< 0 "gash.texi"))
       (<sh-exec> "wc"))

     POSIXLY_CORRECT=y sed N < file
     =>
     (<sh-with-redirects> ((< 0 "file"))
       (<sh-exec-let> (("POSIXLY_CORRECT" "y"))
         "sed" "N"))

     ;; Watch out for this one, though!
     echo foo >| file
     =>
     (<sh-with-redirects> ((>! 1 "file"))
       (<sh-exec> "echo" "foo"))

Here documents get processed by the parser, so they only have one form:

     cat <<EOF > hello.scm
     (display "Hello world!\n")
     EOF
     =>
     (<sh-with-redirects> ((<< 0 (<sh-quote> "(display \"Hello world!\\n\")\n"))
                           (> 1 "hello.scm"))
       (<sh-exec> "cat"))

     cat <<EOF > hello.sh
     hi=Howdy; echo $hi world!
     EOF
     =>
     (<sh-with-redirects> ((<< 0 (<sh-quote>
                                  ("hi=Howdy; echo " (<sh-ref> "hi") " world!\n")))
                           (> 1 "hello.sh"))
       (<sh-exec> "cat"))

     ;; That's not what we meant!  Let's try again.
     cat <<\EOF > hello.sh
     hi=Howdy; echo $hi world!
     EOF
     =>
     (<sh-with-redirects> ((<< 0 (<sh-quote> "hi=Howdy; echo $hi world!\n"))
                           (> 1 "hello.sh"))
       (<sh-exec> "cat"))

Compound commands
.................

Loops look like they do on the shell:

     while test $GASH = confusing
     do
         read-the-manual
     done
     =>
     (<sh-while> (<sh-exec> "test" (<sh-ref> "GASH") "=" "confusing")
       (<sh-exec> "read-the-manual"))

     until cows come home; do wait; done
     =>
     (<sh-until> (<sh-exec> "cows" "come" "home")
       (<sh-exec> "wait"))

     for cat in Murray Flash Boots
     do
         pet $cat
     done
     =>
     (<sh-for> ("cat" ("Murray" "Flash" "Boots"))
       (<sh-exec> "pet" (<sh-ref> "cat")))

If-statements and case-statements are a bit more like Scheme:

     if test $FLAVOR = chocolate
     then
         echo hooray!
     elif test $FLAVOR = vanilla
     then
         echo yum!
     else
         echo too fancy!
     fi
     =>
     (<sh-cond>
      ((<sh-exec> "test" (<sh-ref> "FLAVOR") "=" "chocolate")
       (<sh-exec> "echo" "hooray!"))
      ((<sh-exec> "test" (<sh-ref> "FLAVOR") "=" "vanilla")
       (<sh-exec> "echo" "yum!"))
      (<sh-else>
       (<sh-exec> "echo" "too" "fancy!")))

     case $FLAVOR
     in
         *mint*) echo blech! ;;
         *peanut*) echo so good! ;;
         *maple*) echo maybe! ;;
         *) echo sure why not! ;;
     esac
     =>
     (<sh-case> (<sh-ref> "FLAVOR")
       (("*mint*") (<sh-exec> "echo" "blech!"))
       (("*peanut*") (<sh-exec> "echo" "so" "good!"))
       (("*maple*") (<sh-exec> "echo" "maybe!"))
       (("*") (<sh-exec> "echo" "sure" "why" "not!")))

Brace groups and subshells are pretty straight-forward:

     { echo jump to the left;
       echo step to the right; }
     =>
     (<sh-begin>
      (<sh-exec> "echo" "jump" "to" "the" "left")
      (<sh-exec> "echo" "step" "to" "the" "right"))

     (cd tmp; cat ephemera.txt)
     =>
     (<sh-subshell>
      (<sh-exec> "cd" "tmp")
      (<sh-exec> "cat" "ephemera.txt"))

Function definitions
....................

Function definitions look more like Lisp:

     dance() {
       echo swing your hips now!
     }
     =>
     (<sh-defun> "dance"
       (<sh-exec> "echo" "swing" "your" "hips" "now!"))

     ;; Even this weird thing works!
     dance_to_the_beat() {
       sed 's/beat/swing hips/g'
     } < beats.txt
     =>
     (<sh-defun> "dance_to_the_beat"
       (<sh-with-redirects> ((< 0 "beats.txt"))
         (<sh-exec> "sed" (<sh-quote> "s/beat/move hips/g"))))

Pipelines and asynchronous commands
...................................

Pipelines collect commands into a list:

     source | sink
     =>
     (<sh-pipeline> (<sh-exec> "source") (<sh-exec> "sink"))

     cut -d ' ' -f 4 < ice-cream.txt \
       | grep chocolate \
       | sed 's/mint/peanut butter/g'
     =>
     (<sh-pipeline>
      (<sh-with-redirects> ((< 0 "flavors.txt"))
        (<sh-exec> "cut" "-d" (<sh-quote> " ") "-f" "4"))
      (<sh-exec> "grep" "chocolate")
      (<sh-exec> "sed" (<sh-quote> "s/mint/peanut butter/g")))

Asynchronous commands are simply regular commands wrapped with <sh-async>:

     helpful-daemon &
     =>
     (<sh-async> (<sh-exec> "helpful-daemon"))

     ;; So-so dancing.
     swing-hips ; flail-arms
     =>
     (<sh-begin>
      (<sh-exec> "swing-hips")
      (<sh-exec> "flail-arms"))

     ;; Great dancing.
     swing-hips & flail-arms
     =>
     (<sh-begin>
      (<sh-async> (<sh-exec> "swing-hips"))
      (<sh-exec> "flail-arms"))

Boolean expressions
...................

The logical ``and'' and ``or'' operators are binary, left-associative and have equal precedence:

     foo && bar || baz
     =>
     (<sh-or> (<sh-and> (<sh-exec> "foo")
                        (<sh-exec> "bar"))
              (<sh-exec> "baz"))

     foo || bar && baz
     =>
     (<sh-and> (<sh-or> (<sh-exec> "foo")
                        (<sh-exec> "bar"))
               (<sh-exec> "baz"))

The logical ``not'' operator wraps a command or pipeline:

     ! false
     =>
     (<sh-not> (<sh-exec> "false"))

     ! cat flavors.txt | grep mint
     =>
     (<sh-not> (<sh-pipeline>
                (<sh-exec> "cat" "flavors.txt")
                (<sh-exec> "grep" "mint")))

Variables and command substitution
..................................

Variable operators have names inspired by Scheme:

     echo ${FOO+bar}
     =>
     (<sh-exec> "echo" (<sh-ref-and> "FOO" "bar"))

     echo ${FOO:+bar}
     =>
     (<sh-exec> "echo" (<sh-ref-and*> "FOO" "bar"))

     echo ${FOO-bar}
     =>
     (<sh-exec> "echo" (<sh-ref-or> "FOO" "bar"))

     echo ${FOO=bar}
     =>
     (<sh-exec> "echo" (<sh-ref-or!> "FOO" "bar"))

Command substitution is a simple wrapper:

     echo $(cat message.txt)
     =>
     (<sh-exec> "echo" (<sh-cmd-sub> (<sh-exec> "cat" "message.txt")))

That's it for examples.  By this point, you should have a pretty good sense for what Gash is going to give you.  If you need to know more, you can always do your own experiments or read the formal definition in the next section.

Internal representation definition
----------------------------------

The internal representation of shell code in Gash has the following form.  (If Gash ever returns something that does not have this form, please file a bug!)

     sync  ::= exp
             | ('<sh-async> exp)

     exp   ::= pipe
             | ('<sh-and> exp1 exp2)
             | ('<sh-or> exp1 exp2)
             | ('<sh-not> pipe)

     pipe  ::= cmd*
             | ('<sh-pipeline> cmd* ...)

     cmd*  ::= cmd
             | ('<sh-defun> name sync ...)

     cmd   ::= ('<sh-exec> word ...)
             | ('<sh-exec-let> ((var var-word) ...) word ...)
             | ('<sh-with-redirects> (redir ...) [cmd])
             | ('<sh-set!> (var word) ...)
             | ('<sh-subshell> sync ...)
             | ('<sh-for> (name (word ...)) sync ...)
             | ('<sh-case> word ((pattern-word ...) sync ...) ...)
             | ('<sh-cond> (list sync ...) ... [('<sh-else> sync ...)])
             | ('<sh-while> list sync ...)
             | ('<sh-until> list sync ...)
             | ('<sh-begin> sync ...)

     redir ::= ('> fdes word)
             | ('< fdes word)
             | ('>& fdes word)
             | ('<& fdes word)
             | ('>> fdes word)
             | ('<> fdes word)
             | ('>! fdes word)
             | ('<< fdes word)

     word  ::= string
             | (word ...)
             | ('<sh-quote> word)
             | ('<sh-cmd-sub> sync ...)
             | ('<sh-ref> var)
             | ('<sh-ref-or> var [word])
             | ('<sh-ref-or*> var [word])
             | ('<sh-ref-or!> var [word])
             | ('<sh-ref-or!*> var [word])
             | ('<sh-ref-assert> var [word])
             | ('<sh-ref-assert*> var [word])
             | ('<sh-ref-and> var [word])
             | ('<sh-ref-and*> var [word])
             | ('<sh-ref-except-min> var [word])
             | ('<sh-ref-except-max> var [word])
             | ('<sh-ref-skip-min> var [word])
             | ('<sh-ref-skip-max> var [word])
             | ('<sh-ref-length> var)

     var   ::= string
             | ("LINENO" integer)

Using shell-like constructs from Scheme
=======================================

Gash also has some features for writing Scheme code with shell-like semantics.  Indeed, this is how Gash works internally.  It interprets shell code into calls into its shell interface.  The two modules that provide this interface are (gash shell) and (gash environment).  The shell module provides control structures with shell semantics.  For instance, you can use sh:for to update a given shell variable and run a thunk for each element of a list of strings.  The environment module allows you to manipulate shell variables and functions, as well as a number of other things.

Truthfully, these interfaces are not ready for public consumption yet. They are in need of some careful redesign to be useful outside of Gash itself.  If this sounds interesting to you, please get in touch by writing to gash-devel@nongnu.org (mailto:gash-devel@nongnu.org).

GNU Free Documentation License
******************************

     [file fdl-1.3.texi not included]