From 34feefa559c395147e21d10033c70e0fdef1eea5 Mon Sep 17 00:00:00 2001 From: Jan Nieuwenhuizen Date: Mon, 9 Dec 2019 07:26:16 +0100 Subject: [PATCH] doc: Release update. * BOOTSTRAP: Release update. * HACKING: Likewise. * INSTALL: Likewise. * NEWS: Likewise. * PORTING: Likewise. * README: Likewise. * ROADMAP: Likewise. * doc/mes.texi: Likewise. * doc/announce/ANNOUNCE-0.22: New file. --- BOOTSTRAP | 55 ++++++++++-- HACKING | 70 +++++----------- INSTALL | 23 +++++ NEWS | 36 ++++++++ PORTING | 22 +++++ README | 22 ++--- ROADMAP | 31 ++++--- doc/announce/ANNOUNCE-0.21 | 14 ++-- doc/announce/ANNOUNCE-0.22 | 162 ++++++++++++++++++++++++++++++++++++ doc/mes.texi | 166 +++++++++++++++++++++++-------------- 10 files changed, 455 insertions(+), 146 deletions(-) create mode 100644 doc/announce/ANNOUNCE-0.22 diff --git a/BOOTSTRAP b/BOOTSTRAP index c8b83cdd..c2d97dc3 100644 --- a/BOOTSTRAP +++ b/BOOTSTRAP @@ -8,24 +8,24 @@ Copyright © 2016,2017,2018,2019 Jan (janneke) Nieuwenhuizen notice and this notice are preserved. * What? -** Full source bootstrapping for GNU Guix +** Full source bootstrapping for the GNU System -A package in Guix is uniquely identified by the hash of its source code, -its dependencies, and its build recipe. +A package in GNU Guix is uniquely identified by the hash of its source +code, its dependencies, and its build recipe. Every package can be built from source, except for the bootstrap binaries. -*** From the Guix manual +*** From the GNU Guix manual - The distribution is fully “bootstrapped” and “self-contained”: each +The distribution is fully “bootstrapped” and “self-contained”: each package is built based solely on other packages in the distribution. The root of this dependency graph is a small set of “bootstrap binaries”, provided by the ‘(gnu packages bootstrap)’ module. For more information on bootstrapping, *note Bootstrapping::. -**** Guix bootstrap tarballs +**** Guix v1.0 bootstrap binary seed $ du -schx $(readlink $(guix build bootstrap-tarballs)/*) 2.1M /gnu/store/9623n4bq6iq5c8cwwdq99qb7d0xj93ym-binutils-static-stripped-tarball-2.28.1/binutils-static-stripped-2.28.1-x86_64-linux.tar.xz @@ -44,6 +44,42 @@ $ du -schx * 5.2M share 252M total +**** Guix Reduced Binary Seed bootstrap binary seed + +$ du -schx $(readlink $(guix build bootstrap-tarballs)/*) +5.7M /gnu/store/9f8gi8raqfx9j3l9d00qrrc0jg3r1kyj-guile-static-stripped-tarball-2.2.6/guile-static-stripped-2.2.6-x86_64-linux.tar.xz +80K /gnu/store/b6rjl52hibhmvyw4dg8678pwryhla0h2-linux-libre-headers-stripped-tarball-4.19.56/linux-libre-headers-stripped-4.19.56-x86_64-linux.tar.xz +12K /gnu/store/d7zlxsjcnqilmvqwx7scija9x9bjw8cw-mescc-tools-static-stripped-tarball-0.5.2-0.bb062b0/mescc-tools-static-stripped-0.5.2-0.bb062b0-x86_64-linux.tar.xz +428K /gnu/store/n7zc4kpi8ny6jlfaikkzxlwhc5fvr1vr-mes-minimal-stripped-tarball-0.19/mes-minimal-stripped-0.19-x86_64-linux.tar.xz +6.0M /gnu/store/nv4djwlrljfqmynqr2cqvfwz0ydx7kxb-static-binaries-tarball-0/static-binaries-0-x86_64-linux.tar.xz +13M total +$ for i in $(readlink $(guix build bootstrap-tarballs)/*);\ + do sudo tar xf $i; done +Password: +$ du -schx * +93M bin +700K include +38M lib +14M share +145M total + +**** Guix Scheme-only bootstrap binary seed + +$ du -schx $(readlink $(~/src/guix/wip-bootstrap/pre-inst-env guix build bootstrap-tarballs)/*) +5.7M /gnu/store/1mq2pcd2h7g54xpi2jrgj6ibbi4lgi3c-guile-static-stripped-tarball-2.2.6/guile-static-stripped-2.2.6-x86_64-linux.tar.xz +80K /gnu/store/bl1r2bpk6fam8r2gjvr5mvr48i3dm2hn-linux-libre-headers-stripped-tarball-4.19.56/linux-libre-headers-stripped-4.19.56-x86_64-linux.tar.xz +12K /gnu/store/w0dlz486dhb8aiq8pxm5akllz628fqin-mescc-tools-static-stripped-tarball-0.5.2-0.bb062b0/mescc-tools-static-stripped-0.5.2-0.bb062b0-x86_64-linux.tar.xz +428K /gnu/store/15j6l18q44ymlrh1cfp4s4hc9835xic5-mes-minimal-stripped-tarball-0.19/mes-minimal-stripped-0.19-x86_64-linux.tar.xz +6.2M total +$ for i in $(readlink $(~/src/guix/wip-bootstrap/pre-inst-env guix build bootstrap-tarballs)/*);\ + do sudo tar xf $i; done +$ du -schx * +4.9M bin +700K include +38M lib +14M share +57M total + * Why? ** Reproducibility is essential to Software Freedom @@ -92,20 +128,22 @@ be source. *** mes.c: a Scheme interpreter in ~5,000LOC of simple C *** mescc: a C compiler written in Scheme (uses Nyacc C99 parser in Scheme) -*** mes.M2: this Scheme interpreter in preprocessed M2 +*** mes.M2: this Scheme interpreter in bootstrappable M2 ** TinyCC https://gitlab.com/janneke/tinycc * TODO +** merge scheme-only bootstrap into Guix. +** reduced binary seed bootstrap into NixOS, Debian. ** remove or upstream patches from tcc-boot ** prepare src/mes.c for M2-Planet transpiler. ** ARM, the Hurd ** fix bootstrap-loops: (Nyacc?, mes.M2, psyntax.pp?) ** make GNU gcc (8.0?) bootstrappable again, remove [need for] tcc stage * DONE -** upstream mes-boot to Guix. +** reduced binary seed bootstrap into Guix. ** replace Guix bootstrap for x86_64. ** replace Guix bootstrap for x86. ** add full source gcc-4.7 package build. @@ -121,5 +159,6 @@ be source. ** tcc compiled with gcc is known to compile gcc * Contact +** bug-mes@gnu.org ** #bootstrappable, #guix on freenode ** bootstrappable.org diff --git a/HACKING b/HACKING index c4096c6b..fb6d7e95 100644 --- a/HACKING +++ b/HACKING @@ -1,64 +1,31 @@ -*- org -*- #+TITLE: Hacking GNU Mes -Copyright © 2016,2017,2018 Jan (janneke) Nieuwenhuizen - - Copying and distribution of this file, with or without modification, - are permitted in any medium without royalty provided the copyright - notice and this notice are preserved. - * SETUP guix environment -l guix.scm #64 bit + 32bit + CC=i686-unkown-linux-gnu-gcc ./configure or + ./configure - guix environment --system=i686-linux -l guix.scm #32 bit only - -or - - guix package --profile=~/.config/guix/mes --manifest=build-aux/manifest.scm - . ~/.config/guix/mes/etc/profile +The 64bit bootstrap is not supported; MesCC needs work to be able to +compile a functional tinycc. * BUILD -There are two major modes to build Mes: true bootstrap and -development. -** DEVELOPMENT BUILD -To help development we assume ./configure sets these variables for make +GNU Guile can be used as a replacement for Mes. It's faster and has +much better error handling. - CC -- gcc (or i686-unknown-linux-gnu-gcc sans libc) - GUILE -- guile - HEX2 -- hex2 - MES -- unset - M1 -- M1 - prefix -- "" + #+BEGIN_SRC bash + make MES=guile + make check MES=guile + #+END_SRC -Mes is supposed to serve as a full equivalent for Guile, however Mes -is still about 2 to 10 times slower than Guile. That's why we usually -don't use Mes during development, configure --with-cheating. +Configure to compile with Tiny CC -Gcc is used to verify the sanity of our C sources. - -i686-unknown-linux-gnu-gcc is used to compare hex/assembly, to test -the gcc variant of Mes C Libirary. -Target prefix: x86-mes-gcc. - -gcc -nostdinc,-nostdlib is used to compare hex/assembly, to test the -64bit variant of Mes C Library. Target prefix: x86_64-mes-gcc. - -Guile is used to develop MesCC, the C compiler in Scheme that during -bootstrapping will be executed by Mes. - -** BOOTSTRAP BUILD - - ./configure.sh [--prefix=PREFIX] - ./bootstrap.sh - ./install.sh - -In bootstrap mode, we don't have gcc (CC), we don't have a 32 bit gcc, -we have no guile (GUILE)...but we should have hex2, M1, and mes.M1. -That's a bootstrap problem which is currently ignored by using the -mes-seed package. mes.M1 will be produced by M2-Planet from mes.c. + #+BEGIN_SRC bash + ./configure CC=tcc --host=i686-unknown-linux-gnu --with-courage + #+END_SRC * DEBUG MES_DEBUG= mes @@ -83,7 +50,7 @@ mes-seed package. mes.M1 will be produced by M2-Planet from mes.c. * Bugs ** mes: performance, Mes is now 2-10x slower than Guile. -** mes/mescc lack support for the Hurd. +** mes/mescc lack support for fork/exec on the Hurd. ** mes: gcc-x86_64 compiled mes segfaults with small arena, or gc_up_arena. ** mes: gcc-x86 compiled, tests/srfi-13.test number->string INT-MIN fails: test: number->string INT-MIN: fail @@ -214,3 +181,10 @@ https://notabug.org/rain1/hex86/src/master/tests/hex0b3.hex86 https://jamey.thesharps.us/2016/07/15/testing-strategies-for-corrode/ ("Randomized testing with Csmith and C-Reduce") [10:58] ** linux syscalls: https://fedora.juszkiewicz.com.pl/syscalls.html +* legalese + +Copyright © 2016,2017,2018,2019 Jan (janneke) Nieuwenhuizen + + Copying and distribution of this file, with or without modification, + are permitted in any medium without royalty provided the copyright + notice and this notice are preserved. diff --git a/INSTALL b/INSTALL index 93da9350..dcea9ebd 100644 --- a/INSTALL +++ b/INSTALL @@ -41,6 +41,19 @@ similar way. make #+END_SRC +If you get something like + + #+BEGIN_SRC bash + mes.c:(.text+0x8da0): undefined reference to `__stack_chk_fail' + #+END_SRC + +then your compiler inserts stack protection code. The Mes C Library +does not support that. Turn it off lik this + + #+BEGIN_SRC bash + ./configure CFLAGS=-fno-stack-protector + #+END_SRC + ** Check it #+BEGIN_SRC bash @@ -86,3 +99,13 @@ https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/commencement.scm #+BEGIN_SRC bash sh install.sh #+END_SRC +* Regular build and bootstrap combine + +To build mes.c using MesCC, configure using --with-bootstrap: + + #+BEGIN_SRC bash + ./configure --with-bootstrap + make + #+END_SRC + +This creates bin/mes-gcc and bin/mes-mescc. diff --git a/NEWS b/NEWS index a9c09328..1c413875 100644 --- a/NEWS +++ b/NEWS @@ -10,6 +10,41 @@ Copyright © 2016,2017,2018,2019 Jan (janneke) Nieuwenhuizen Please send Mes bug reports to bug-mes@gnu.org. +* Changes in 0.22 since 0.21 +** Core +*** Mes now builds reproducibly with MesCC, cross distribution +Using --with-bootstrap on x86, a bin/mes-mescc is compiled with +mes+mescc, that shall have sha256sum + + 9e0bcb1633c58e7bc415f6ea27cee7951d6b0658e13cdc147e992b31a14625fb bin/mes-mescc + +This has been verified on GNU Guix, Debian GNU/Linux and NixOS. +*** Mes now builds reproducibly with Guile+MesCC vs Mes+MesCC. +*** Mes now builds reproducibly with Tiny CC-built MesCC. +*** configure now respects the user's CFLAGS, CPPFLAGS and LDFLAGS. +*** Mes now supports mescc-tools 0.5.2 next to 0.6.1. +This allows introducing the Scheme-only bootstrap without updating or +adding new bootstrap binary seeds into GNU Guix. +*** Mes now runs when configured --with-courage on the Hurd: + GNU debian 0.9 GNU-Mach 1.8+git20170609-486/Hurd-0.9 i686-AT386 GNU +Note that mescc does not run; fork and exec are not yet implemented. +*** Mes now configures --with-courage on x86-FreeBSD 12.1 +The initial ELF scaffold tests: exit-42.S, 0exit-42.hex2, +body-exit-42.hex2, hello-mes.S, 0hello-mes.hex2, body-hello-mes.hex2 +pass. +*** configure now has a --with-bootstrap option. +This adds the reproducible bootstrap build of bin/mes-mescc, using +mes+mescc. +*** configure prefers $CC over gcc; tcc over gcc, gcc over cc. +*** Mes now prints an error when attempting to read a file that does not exist. +*** Mes no longer depends on GIT. +** Noteworthy bug fixes +*** Several annoying build problems were fixed, for non-Guix systems. +*** A bug with buffered-read was fixed. +This means that bash-2.05 can now build glibc-2.2.5; notably it now +successfully executes make-syscall.sh. +*** A bug with execlp, execvp for file names containing a slash was fixed. +This should allow make-3.80 running scripts that have "#! ./move-if-change". * Changes in 0.21 since 0.20 ** Core *** Mes can now be bootstrapped with Gash and Gash Core Utils. @@ -48,6 +83,7 @@ take-while. *** 1 new function __mesabi_uldiv. ** Noteworthy bug fixes +*** map and for-each now support lists with unequal length. *** interger division has been fixed. *** isatty now looks at terminfo. *** signal now uses sigaction correctly for non-x86. diff --git a/PORTING b/PORTING index 8eeb418b..706ea13f 100644 --- a/PORTING +++ b/PORTING @@ -54,6 +54,28 @@ bootstrap for x86_64 uses x86 mes and that is not expected to change. The Hurd port can be found in wip-hurd on savannah. For development, we use a Debian GNU/Hurd vm. +* Porting GNU Mes to FreeBSD +The FreeBSD port can be found in wip-freebsd on savannah. + +* Porting the Reduced Binary Seed bootstrap to NixOS +The NixOS port of the Reduced Binary Seed bootstrap lives on the +mes-bootstrap branch in https://github.com/xwvvvvwx/nixpkgs + +* Porting the Reduced Binary Seed bootstrap to Debian GNU/Linux +To port the Reduced Binary Seed bootstrap to a traditional distribution +such as Debian, two things need to happen: the bootstrap must be ported +and the distribution build process needs to change to start from a only +binary seed. + +The porting aspect is probably easiest: we start by packaging GNU Mes as +a regular package. Having Mes built as a regular package, we can then +build a bootstrap Mes. This bootstrap Mes can be used to build tinycc, +and so on. + +Once we have shown that the Debian base system can be bootstrapped from +a Reduced Binary seed, thus significantly reducing the Trusted Computing +Base (TCB), strategic decisions about the build process can start. + * Legalese Copyright © 2019 Jan (janneke) Nieuwenhuizen <[[mailto:janneke@gnu.org][janneke@gnu.org]]> diff --git a/README b/README index 00280cc6..7607b6c8 100644 --- a/README +++ b/README @@ -4,20 +4,20 @@ #+SUBTITLE: Maxwell Equations of Software [[https://www.gnu.org/software/mes][GNU Mes]] is a Scheme interpreter and C compiler for bootstrapping the GNU -system. Since version 0.20 it brings a [[https://guix.gnu.org/blog/2019/guix-reduces-bootstrap-seed-by-50/][Reduced Binary Seed bootstrap]] to -[[https://www.gnu.org/software/guix][GNU Guix]]. This bootstrap has halved the size of opaque, uninspectable -binaries that were needed to bootstrap Guix 1.0. The final goal is to -help create a full source bootstrap as part of the [[http://bootstrappable.org][bootstrappable builds]] +System. Since version 0.22 it has again helped to halve the size of +opaque, uninspectable binary seeds that are currently being used in the +[[https://guix.gnu.org/blog/2019/guix-reduces-bootstrap-seed-by-50/][Reduced Binary Seed bootstrap]] of [[https://www.gnu.org/software/guix][GNU Guix]]. The final goal is to help +create a full source bootstrap as part of the [[http://bootstrappable.org][bootstrappable builds]] effort for UNIX-like operating systems. -Mes consists of a mutual self-hosting Scheme interpreter written in -~5,000 LOC of simple C, and a C compiler written in Scheme. This mes.c -is [[https://github.com/oriansj/mes-m2][being simplified]] to be transpiled by [[https://github.com/oriansj/m2-planet][M2-Planet]]. +The Scheme interpreter is written in ~5,000 LOC of simple C, and the C +compiler written in Scheme and these are mutual self-hosting. This +mes.c is [[https://github.com/oriansj/mes-m2][being simplified]] to be transpiled by [[https://github.com/oriansj/m2-planet][M2-Planet]]. -The Scheme interpreter has a Garbage Collector, a library of loadable -Scheme modules-- notably Dominique Boucher's [[https://github.com/schemeway/lalr-scm][LALR]], Pre-R6RS [[https://www.cs.indiana.edu/chezscheme/syntax-case/old-psyntax.html][portable -syntax-case]] with R7RS ellipsis, Matt Wette's [[https://www.nongnu.org/nyacc][Nyacc]] --and test suite just -enough to support a REPL and a C99 compiler: MesCC. +Mes has a Garbage Collector, a library of loadable Scheme modules-- +notably Dominique Boucher's [[https://github.com/schemeway/lalr-scm][LALR]], Pre-R6RS [[https://www.cs.indiana.edu/chezscheme/syntax-case/old-psyntax.html][portable syntax-case]] with +R7RS ellipsis, Matt Wette's [[https://www.nongnu.org/nyacc][Nyacc]] --and test suite, just enough to +support a REPL and a C99 compiler: mescc. Mes+MesCC can compile an only [[http://gitlab.com/janneke/tinycc][lightly patched TinyCC]] that is self-hosting. Using this tcc and the Mes C library we now have a diff --git a/ROADMAP b/ROADMAP index c4dd4d8b..ca476c45 100644 --- a/ROADMAP +++ b/ROADMAP @@ -9,7 +9,7 @@ Copyright © 2016,2017,2018,2019 Jan (janneke) Nieuwenhuizen The goals of the GNU Mes project are two-fold: - - to bootstrap the GNU system purely from source, without using any binary + - to bootstrap the GNU System purely from source, without using any binary seeds. - to aid the Reproducible-Builds.org effort; demonstrate the impact and @@ -23,23 +23,32 @@ on Freenode! * TODO ** release 0.x, unsorted + - Merge the Scheme-only bootstrap which uses Gash and Gash Core + Utilities to remove from bootstrap-binary seed : Awk, Bash, Core + Utilities, Grep, Gzip, Sed, Tar. - Introduce the Reduced Binaries Seed bootstrap to NixOS (Debian, - Gentoo, ...?) - - Full Source Bootstrap: compile Mes.c using M2-Planet. Jeremiah + Arch, Gentoo, ...?) + - Full Source Bootstrap: compile mes.c using M2-Planet. Jeremiah branched-out from mes; see https://github.com/oriansj/mes-m2, also see the - branches: wip-m2-merge and wip-m2. - - Scheme-only bootstrap bootstrap-binaries with Gash and Gash Core - Utilities: Awk, Bash, Core Utilities, Grep, Gzip, Make, Sed, Tar. + branches: wip-full-source-bootstrap, wip-m2-merge and wip-m2. - Full Guile-compatible module support: define-module, define-public, export, use-modules, ... - ARMv7/AArch64 support: Mes C Library and MesCC. - Simplify eval-apply and other core functionality (see mes-m2). -** after release 1.0 - - Replace the initial gcc-2.95.3 with gcc-4.6.4. + - Support native x86_64 bootstrap. + - Simplify MesCC ``MesCC should optimize ease of convincing us of its + correctness'' - tcc: remove or upstream patches from tcc-boot. - tcc: build 0.9.27 directly instead of via 0.9.26, see tinycc +** after release 1.0 + - Build Guix packages using Mes: run or simplify (guix build utils), + (guix build gnu-build-system). + - Build Guix packages without guix-daemon. See Ludovic's initrd build + work. + - Bootstrap support for the Hurd. Most needed now are fork and exec. + - Replace the initial gcc-2.95.3 with gcc-4.6.4. + - FreeBSD, *BSD? wip-bootstrappable@0.9.27 branch - - Bootstrap a `bootstrap-Guile' before bootstrapping tcc? - Skip tcc: Build gcc using MesCC. - better garbage collector. - mes/mescc: proper docstrings, api reference documentation. @@ -51,9 +60,9 @@ on Freenode! + get full source syntax-case up (Andre van Tonder?) https://srfi.schemers.org/srfi-72/srfi-72.html, or + ... drop it? - - Support the Hurd. There is a wip-hurd branch; most needed now are fork - and exec. * DONE + - 0.22 GNU Mes now builds reproducibly cross-platform, runs on the + Hurd and has better support for other distributions and kernels. - 0.21 GNU Mes now supports a Scheme-only bootstrap and is packaged in Debian GNU/Linux. - 0.20 GNU Mes brings the Reduced Binary Source bootstrap to Guix. diff --git a/doc/announce/ANNOUNCE-0.21 b/doc/announce/ANNOUNCE-0.21 index bc642b74..6cf75d44 100644 --- a/doc/announce/ANNOUNCE-0.21 +++ b/doc/announce/ANNOUNCE-0.21 @@ -37,18 +37,18 @@ Packages are available in Guix master. GNU Mes[0] brings a Reduced Binary Seed bootstrap[1] to GNU Guix[2]. This bootstrap has halved the size of opaque, uninspectable binaries - that were needed to bootstrap Guix. The final goal is to help create - a full source bootstrap as part of the bootstrappable builds[3] effort - for any interested UNIX-like operating system. + that were needed to bootstrap Guix 1.0. The final goal is to help + create a full source bootstrap as part of the bootstrappable builds[3] + effort for any interested UNIX-like operating system. - It consists of a mutual self-hosting Scheme interpreter written in - ~5,000 LOC of simple C and a Nyacc-based C compiler written in Scheme. - This mes.c is being simplified[4] to be transpiled by M2-Planet[5]. + Mes consists of a mutual self-hosting Scheme interpreter written in + ~5,000 LOC of simple C, and a C compiler written in Scheme. This + mes.c is being simplified[4] to be transpiled by M2-Planet[5]. The Scheme interpreter has a Garbage Collector, a library of loadable Scheme modules-- notably Dominique Boucher's LALR[6], Pre-R6RS [portable syntax-case[7] with R7RS ellipsis, Matt Wette's Nyacc[8] - --and test suite just enough to support a REPL and a C99 compiler: + --and test suite, just enough to support a REPL and a C99 compiler: MesCC. Mes+MesCC can compile an only lightly patched TinyCC[9] that is diff --git a/doc/announce/ANNOUNCE-0.22 b/doc/announce/ANNOUNCE-0.22 new file mode 100644 index 00000000..e1487932 --- /dev/null +++ b/doc/announce/ANNOUNCE-0.22 @@ -0,0 +1,162 @@ + +Subject: GNU Mes 0.22 released + +<#secure method=pgpmime mode=sign> + +We are pleased to announce the release of GNU Mes 0.22, representing +57 commits over 8 weeks. + +Mes can now be built reproducibly by mes+mescc or guile+mescc accross +distributions such as GNU Guix, Debian GNU/Linux and NixOS. Also, many +build issues were fixed. A big thank you to reproducible-builds.org, +Vagrant Cascadian, Jelle van der Waa, Hannes Mehnert and David Terry! + +Mes now runs on the Hurd--mescc does not run yet, many system calls are +missing, especially fork/exec--and initial scaffolding support for +creating FreeBSD binaries was added. + +This release better supports bringing the Scheme-only bootstrap to Guix: +it no longer requires an update to the Guix bootstrap seed. + +The Scheme-only bootstrap means that Mes can now be built with Gash and +the experimental Gash Core Utils instead of using GNU Awk, GNU Bash, the +GNU Core Utilities, GNU Grep, GNU Gzip, GNU Make, GNU Sed, and GNU Tar. +Also, the Mes C Library now supports bootstrapping those. + +We are excited that the Nlnet Foundation[12] is now sponsoring this +work! + +Next targets: + + - Introduce the Reduced Binaries Seed bootstrap to NixOS (Debian, + Arch, Gentoo, ...?) + - Merge the Scheme-only bootstrap, the wip-bootstrap branch (using + Guile, Gash and Gash Core Utils to remove awk, bash, core utilities, + grep, gzip, make, sed, tar, etc. from the Guix bootstrap binaries) + into Guix core-updates. + - ARM support + - Full Source Bootstrap: compile Mes.c using M2-Planet + - Add full Guile module support + - Reduced Binary Seed bootstrap for ARM + - Support mescc on the Hurd (fork/exec, ...) + +Packages are available in Guix master; the cross-distro reproducible +is built using + + guix build --system=i686-linux mes-rb5 + +* About + + GNU Mes[0] is a Scheme interpreter and C compiler for bootstrapping the GNU + System. Since version 0.22 it has again helped to halve the size of + opaque, uninspectable binary seeds that are currently being used in the + Reduced Binary Seed bootstrap[1] of GNU Guix[2]. The final goal is to help + create a full source bootstrap as part of the bootstrappable builds[3] + effort for UNIX-like operating systems. + + The Scheme interpreter is written in ~5,000 LOC of simple C, and the C + compiler written in Scheme and these are mutual self-hosting. This + mes.c is being simplified[4] to be transpiled by M2-Planet[5]. + + Mes has a Garbage Collector, a library of loadable Scheme modules-- + notably Dominique Boucher's LALR[6], Pre-R6RS [portable syntax-case[7] + with R7RS ellipsis, Matt Wette's Nyacc[8] --and test suite just enough + to support a REPL and a C99 compiler: MesCC. + + Mes+MesCC can compile an only lightly patched TinyCC[9] that is + self-hosting. Using this tcc and the Mes C library we now have a + Reduced Binary Seed bootstrap for the gnutools triplet: glibc-2.2.5, + binutils-2.20.1, gcc-2.95.3. This is enough to bootstrap Guix for + i686-linux and x86_64-linux. + + Mes is inspired by The Maxwell Equations of Software: LISP-1.5[10] -- John + McCarthy page 13, GNU Guix's source/binary packaging transparency and + Jeremiah Orians's stage0[11] ~500 byte self-hosting hex assembler. + +* Download + + git clone git://git.savannah.gnu.org/mes.git + + Here are the compressed sources and a GPG detached signature[*]: + https://ftp.gnu.org/gnu/mes/mes-0.22.tar.gz + https://ftp.gnu.org/gnu/mes/mes-0.22.tar.gz.sig + + Use a mirror for higher download bandwidth: + https://ftpmirror.gnu.org/mes/mes-0.22.tar.gz + https://ftpmirror.gnu.org/mes/mes-0.22.tar.gz.sig + + Here are the MD5 and SHA1 checksums: + + xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx mes-0.22.tar.gz + xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx mes-0.22.tar.gz + + [*] Use a .sig file to verify that the corresponding file (without the + .sig suffix) is intact. First, be sure to download both the .sig file + and the corresponding tarball. Then, run a command like this: + + gpg --verify mes-0.22.tar.gz.sig + + If that command fails because you don't have the required public key, + then run this command to import it: + + gpg --keyserver keys.gnupg.net --recv-keys 1A858392E331EAFDB8C27FFBF3C1A0D9C1D65273 + + and rerun the 'gpg --verify' command. + +* Get informed, get involved + + See https://bootstrappable.org + Join #bootstrappable on irc.freenode.net. + +* Changes in 0.22 since 0.21 + ** Core + *** Mes now builds reproducibly with MesCC, cross distribution + Using --with-bootstrap on x86, a bin/mes-mescc is compiled with + mes+mescc, that shall have sha256sum + + 9e0bcb1633c58e7bc415f6ea27cee7951d6b0658e13cdc147e992b31a14625fb bin/mes-mescc + + This has been verified on GNU Guix, Debian GNU/Linux and NixOS. + *** Mes now builds reproducibly with Guile+MesCC vs Mes+MesCC. + *** Mes now builds reproducibly with Tiny CC-built MesCC. + *** configure now respects the user's CFLAGS, CPPFLAGS and LDFLAGS. + *** Mes now supports mescc-tools 0.5.2 next to 0.6.1. + This allows introducing the Scheme-only bootstrap without updating or + adding new bootstrap binary seeds into GNU Guix. + *** Mes now runs when configured --with-courage on the Hurd: + GNU debian 0.9 GNU-Mach 1.8+git20170609-486/Hurd-0.9 i686-AT386 GNU + Note that mescc does not run; fork and exec are not yet implemented. + *** Mes now configures --with-courage on x86-FreeBSD 12.1 + The initial ELF scaffold tests: exit-42.S, 0exit-42.hex2, + body-exit-42.hex2, hello-mes.S, 0hello-mes.hex2, body-hello-mes.hex2 + pass. + *** configure now has a --with-bootstrap option. + This adds the reproducible bootstrap build of bin/mes-mescc, using + mes+mescc. + *** configure prefers $CC over gcc; tcc over gcc, gcc over cc. + *** Mes now prints an error when attempting to read a file that does not exist. + *** Mes no longer depends on GIT. + ** Noteworthy bug fixes + *** Several annoying build problems were fixed, for non-Guix systems. + *** A bug with buffered-read was fixed. + This means that bash-2.05 can now build glibc-2.2.5; notably it now + successfully executes make-syscall.sh. + *** A bug with execlp, execvp for file names containing a slash was fixed. + This should allow make-3.80 running scripts that have "#! ./move-if-change". + +Greetings, +janneke and Danny. + +[0] https://www.gnu.org/software/mes +[1] https://guix.gnu.org/blog/2019/guix-reduces-bootstrap-seed-by-50/ +[2] https://www.gnu.org/software/guix +[3] https://bootstrappable.org +[4] https://github.com/oriansj/mes-m2 +[5] https://github.com/oriansj/m2-planet +[6] https://github.com/schemeway/lalr-scm +[7] https://www.cs.indiana.edu/chezscheme/syntax-case/old-psyntax.html +[8] https://www.nongnu.org/nyacc +[9] https://gitlab.com/janneke/tinycc +[10] http://www.softwarepreservation.org/projects/LISP/book/LISP%201.5%20Programmers%20Manual.pdf +[11] https://savannah.nongnu.org/projects/stage0 +[12] https://nlnet.nl/project/GNUMes diff --git a/doc/mes.texi b/doc/mes.texi index 619f2fb8..0c2de61d 100644 --- a/doc/mes.texi +++ b/doc/mes.texi @@ -29,7 +29,7 @@ Documentation License''. * mes: (mes)Invoking mes. Running Mes, a minimalist Guile lookalike. -* mescc: (mes)Invoking MesCC. Running the MesCC bootstrap compiler. +* mescc: (mes)Invoking mescc. Running the MesCC bootstrap compiler. @end direntry @titlepage @@ -48,7 +48,7 @@ Edition @value{EDITION} @* @contents @c ********************************************************************* -@node Top +@node Top, Introduction, (dir), (dir) @top GNU Mes This document describes GNU Mes version @value{VERSION}, a bootstrappable @@ -73,6 +73,7 @@ Software Freedom * Reproducible Builds:: Reproducibility and free software. * Bootstrappable Builds:: The freedom to build a software without binary seed. * Full Source Bootstrap:: Software dependencies worthy of GNU. +* LISP as Maxwell's Equations of Software:: Auditable elegance. Installation @@ -83,15 +84,15 @@ Installation Bootstrapping * The Mes Bootstrap Process:: How Mes will make you yogurt from pure milk. -* Invoking Mes:: Running Mes, a minimalist Guile lookalike. -* Invoking MesCC:: Running the MesCC bootstrap compiler. +* Invoking mes:: Running Mes, a minimalist Guile lookalike. +* Invoking mescc:: Running the MesCC bootstrap compiler. * Invoking mesar:: -Invoking Mes +Invoking mes * Environment Variables:: If the bits won't change, change their habitat. -Invoking MesCC +Invoking mescc * MesCC Environment Variables:: There's no NIX like POSIX. @@ -108,7 +109,7 @@ Contributing @end menu @c ********************************************************************* -@node Introduction +@node Introduction, Installation, Top, Top @chapter Introduction @quotation @@ -187,9 +188,10 @@ Freedom 1. * Reproducible Builds:: Reproducibility and free software. * Bootstrappable Builds:: The freedom to build a software without binary seed. * Full Source Bootstrap:: Software dependencies worthy of GNU. +* LISP as Maxwell's Equations of Software:: Auditable elegance. @end menu -@node Reproducible Builds +@node Reproducible Builds, Bootstrappable Builds, Introduction, Introduction @section Reproducible Builds The current Reproducible Builds effort incubated in the Debian @@ -238,7 +240,7 @@ packages could be rebuild bit-identical from source. @author Martin Uecker @end quotation -@node Bootstrappable Builds +@node Bootstrappable Builds, Full Source Bootstrap, Reproducible Builds, Introduction @section Bootstrappable Builds Software distributions that take reproducible builds seriously are @@ -250,14 +252,14 @@ opaque ascii or binary @emph{seeds} that are injected during build time. Yes, a package may build reproduciblly from all inspectable sourcess...but what functionality is programmed in the opaque seed? -@subsection Bootstrap Binaries +@subsection Bootstrap Binary Seed Possibly one of the most harmless, but certainly by far the biggest binary seed that all software distributions inject are the so called -@emph{bootstrap binaries}. Bootstrap binaries are the initial binary +@emph{bootstrap binary seed}. Bootstrap binaries are the initial binary seeds that are used to start building the distribution. The GNU Guix operating system, version 1.0 had a relatively small -closure of bootstrap binaries: GNU binutils, GNU gcc, GNU Libc, GNU +closure of bootstrap binary seed: GNU binutils, GNU gcc, GNU Libc, GNU Guile, and ``Static binaries'' (think: bash, bzip2, coreutils, gawk, grep, gzip, patch, sed, tar, xz). @@ -287,17 +289,17 @@ $ du -schx * @end example During the Guix 1.1 development series we managed to create the first -reduction by 50% of the Guix @emph{bootstrap binaries}@footnote{See +reduction by 50% of the Guix @emph{bootstrap binary seed}@footnote{See @url{https://guix.gnu.org/blog/2019/guix-reduces-bootstrap-seed-by-50/}}. This was a very important step because the ~250MB @emph{seed} of binary code was practically non-auditable, which makes it hard to establish what source code produced them. -@node Full Source Bootstrap +@node Full Source Bootstrap, , Bootstrappable Builds, Introduction @section Full Source Bootstrap There is an obvious solution: we cannot allow any binary seeds in our -software stack. Not even in the bootstrap binaries. Maybe that is a +software stack. Not even in the bootstrap binary seed. Maybe that is a bit too strong: we want to have the absolute minimum of binary seeds and all binary seeds need to be inspectable and must be reviewed. How big would the absolute minimal set be? @@ -321,8 +323,8 @@ of the @var{hex} function @end example All computer programs look like this: an opaque list of computer codes. -The initial programs that we take for granted---the bootstrap -binaries---are about 250MB of such numbers: think 250,000 pages full of +The initial programs that we take for granted---the bootstrap binary +seed---are about 250MB of such numbers: think 250,000 pages full of numbers. Most computers work pretty well so apparently there is not a pressing @@ -407,7 +409,8 @@ nothing to all of the programs that our computer runs. Something that seemed to be an impossible dream is suddenly starting to look like ``just a couple years of work''. -@subsection LISP as Maxwell's Equations of Software +@node LISP as Maxwell's Equations of Software +@section LISP as Maxwell's Equations of Software As fate would have it, I stumbled upon this @url{https://queue.acm.org/detail.cfm?id=1039523, interview with Alan @@ -426,7 +429,7 @@ over. @end quotation Our starting point is @file{hex0}, a 500 byte hex assembler and we need -to somehow close the gap to building the bootstrap binaries, esp. GNU +to somehow close the gap to building the bootstrap binary seed, esp. GNU Gcc and the GNU C Library. What better way to do that than by leveraging the powers of LISP? @@ -435,8 +438,49 @@ will be indirectly bootstrapped from @file{hex0} and that wields the magical powers of LISP to close the bootstrap gap, asserting we can enjoy software Freedom 1. +@subsection Auditable Elegance + +@code{eval} and @code{apply} are mutual recursing functions that---using +a few helper functions---describe the core of the universe of computing. + +@lisp +(define (apply fn x a) + (cond + ((atom fn) + (cond + ((eq fn CAR) (caar x)) + ((eq fn CDR) (cdar x)) + ((eq fn CONS) (cons (car x) (cadr x))) + ((eq fn ATOM) (atom (car x))) + ((eq fn EQ) (eq (car x) (cadr x))) + (#t (apply (eval fn a) x a)))) + ((eq (car fn) LAMBDA) + (eval (caddr fn) (pairlis (cadr fn) x a))) + ((eq (car fn) LABEL) + (apply (caddr fn) x + (cons (cons (cadr fn) (caddr fn)) a))))) +@end lisp + +@lisp +(define (eval e a) + (cond + ((atom e) (cdr (assoc e a))) + ((atom (car e)) + (cond ((eq (car e) QUOTE) (cadr e)) + ((eq (car e) COND) (evcon (cdr e) a)) + (#t (apply (car e) (evlis (cdr e) a) a)))) + (#t (apply (car e) (evlis (cdr e) a) a)))) +@end lisp + +It will be a big day when our computers are fully bootstrapped from +source. It would be nice if that source code were readable, auditable +and elegant. To be honest, the elegance displayed above that we +achieved at the very start of the Mes project is currently hard to find. +It is our sincerest hope to bring back this level of quality and +elegance.. + @c ********************************************************************* -@node Installation +@node Installation, Bootstrapping, Introduction, Top @chapter Installation @cindex installing Mes @@ -452,7 +496,7 @@ to use it. * Running the Test Suites:: Testing Mes. @end menu -@node Regular Requirements +@node Regular Requirements, Bootstrap Requirements, Installation, Installation @section Regular Requirements This section lists requirements when building Mes from source. The @@ -476,7 +520,7 @@ Mes is compatible with GNU Guile, so it is possible to share the same Scheme code between both. Currently Mes only supports the minimal subset of R5RS and Guile extensions to run MesCC. -@node Bootstrap Requirements +@node Bootstrap Requirements, Running the Test Suites, Regular Requirements, Installation @section Bootstrap Requirements This section lists requirements when building Mes as a bootstrap @@ -501,7 +545,7 @@ Bootstrapping Mes depends on the following packages: @item @url{https://savannah.gnu.org/projects/nyacc/, NYACC}, 0.93.0 or later, including 0.99.0. @end itemize -@node Running the Test Suites +@node Running the Test Suites, , Bootstrap Requirements, Installation @section Running the Test Suites @cindex test suites @@ -544,7 +588,7 @@ CC=gcc CC32=i686-unknown-linux-gnu-gcc MES=guile \ build-aux/test.sh scaffold/tests/00-exit-0 @end example -@node Bootstrapping +@node Bootstrapping, Contributing, Installation, Top @chapter Bootstrapping @quotation @@ -584,12 +628,12 @@ responsibility. @menu * The Mes Bootstrap Process:: How Mes will make you yogurt from pure milk. -* Invoking Mes:: Running Mes, a minimalist Guile lookalike. -* Invoking MesCC:: Running the MesCC bootstrap compiler. +* Invoking mes:: Running Mes, a minimalist Guile lookalike. +* Invoking mescc:: Running the MesCC bootstrap compiler. * Invoking mesar:: @end menu -@node The Mes Bootstrap Process +@node The Mes Bootstrap Process, Invoking mes, Bootstrapping, Bootstrapping @section The Mes Bootstrap Process The Reduced Binary Seed bootstrap currently adopted by Guix@footnote{See @@ -667,8 +711,8 @@ Although we think these are less essential and thus less interesting than the GNU toolchain triplet that we focussed on initially, our next priority is to eleminate these one by one. -@node Invoking Mes -@section Invoking Mes +@node Invoking mes, Invoking mescc, The Mes Bootstrap Process, Bootstrapping +@section Invoking mes @cindex repl The @command{mes} command is the Scheme interpreter whose prime @@ -687,13 +731,13 @@ The @var{option}s can be among the following: @item -s @var{script} @var{arg}@dots{} @cindex script mode -By default, Mes will read a file named on the command line as a script. +By default, mes will read a file named on the command line as a script. Any command-line arguments @var{arg}@dots{} following @var{script} become the script's arguments; the @code{command-line} function returns a list of strings of the form @code{(@var{script} @var{arg}@dots{})}. Scripts are read and evaluated as Scheme source code just as the -@code{load} function would. After loading @var{script}, Mes exits. +@code{load} function would. After loading @var{script}, mes exits. @item -c @var{expr} @var{arg}@dots{} @cindex evaluate expression, command-line argument @@ -701,7 +745,7 @@ Evaluate @var{expr} as Scheme code, and then exit. Any command-line arguments @var{arg}@dots{}) following @var{expr} become command-line arguments; the @code{command-line} function returns a list of strings of the form @code{(@var{guile} @var{arg}@dots{})}, where @var{mes} is the -path of the Mes executable. +path of the mes executable. @item -- @var{arg}@dots{} Run interactively, prompting the user for expressions and evaluating @@ -709,7 +753,7 @@ them. Any command-line arguments @var{arg}@dots{} following the @option{--} become command-line arguments for the interactive session; the @code{command-line} function returns a list of strings of the form @code{(@var{guile} @var{arg}@dots{})}, where @var{mes} is the path of the -Mes executable. +mes executable. @item -L,--load-path=@var{directory} Add @var{directory} to the front of Mes module load path. The given @@ -732,10 +776,10 @@ and the command-line arguments---the list provided by the @code{command-line} function. @item -h@r{, }--help -Display help on invoking Mes, and then exit. +Display help on invoking mes, and then exit. @item -v@r{, }--version -Display the current version of Mes, and then exit. +Display the current version of mes%, and then exit. @end table @@ -743,7 +787,7 @@ Display the current version of Mes, and then exit. * Environment Variables:: If the bits won't change, change their habitat. @end menu -@node Environment Variables +@node Environment Variables, , Invoking mes, Invoking mes @subsection Environment Variables @cindex environment variables @cindex shell @@ -752,13 +796,13 @@ Display the current version of Mes, and then exit. @c Hmm, I expected this paragraph in the Guix manual? Here are the environment variables (see @pxref{Environment Variables,,, guile, Guile Reference}) that affect the run-time behavior of -Mes: +mes: @table @env @item MES_BOOT @vindex MES_BOOT -Set @env{MES_BOOT} to change the initial Scheme program that Mes runs. +Set @env{MES_BOOT} to change the initial Scheme program that mes runs. @item MES_ARENA @vindex MES_ARENA @@ -821,8 +865,8 @@ Mes uses @var{@strong{GUILE}_LOAD_PATH} for compatibility with Guile. @end table -@node Invoking MesCC -@section Invoking MesCC +@node Invoking mescc, Invoking mesar, Invoking mes, Bootstrapping +@section Invoking mescc @example mescc @var{option}@dots{} @file{FILE}@dots{} @@ -902,14 +946,14 @@ specify LANGUAGE of the following input files * MesCC Environment Variables:: There's no NIX like POSIX. @end menu -@node MesCC Environment Variables +@node MesCC Environment Variables, , Invoking mescc, Invoking mescc @subsection MesCC Environment Variables @table @env @item MES @vindex MES -Setting @env{MES} to a mes-compatible Scheme will run MesCC using that +Setting @env{MES} to a mes-compatible Scheme will run mescc using that @example MES=guile mescc -c scaffold/main.c @end example @@ -931,7 +975,7 @@ during the parsing phase. @end table -@node Invoking mesar +@node Invoking mesar, , Invoking mescc, Bootstrapping @section Invoking mesar @example @@ -961,7 +1005,7 @@ display version and exit @end table @c ********************************************************************* -@node Contributing +@node Contributing, Acknowledgments, Bootstrapping, Top @chapter Contributing @menu @@ -973,7 +1017,7 @@ display version and exit * Submitting Patches:: Share your work. @end menu -@node Building from Git +@node Building from Git, Running Mes From the Source Tree, Contributing, Contributing @section Building from Git If you want to hack GNU Mes itself, it is recommended to use the latest @@ -997,7 +1041,7 @@ Finally, you have to invoke @code{make check} to run tests installation instructions (@pxref{Installation}) or send a message to the @email{bug-mes@@gnu.org} mailing list. -@node Running Mes From the Source Tree +@node Running Mes From the Source Tree, Porting GNU Mes, Building from Git, Contributing @section Running Mes From the Source Tree First, you need to have an environment with all the dependencies @@ -1005,7 +1049,7 @@ available (@pxref{Building from Git}), and then simply prefix each command by @command{./pre-inst-env} (the @file{pre-inst-env} script lives in the top build tree of Mes). -@node Porting GNU Mes +@node Porting GNU Mes, The Perfect Setup, Running Mes From the Source Tree, Contributing @section Porting GNU Mes Mes was written for x86-linux. A 64 bit (x86_64) is almost done, only a @@ -1017,13 +1061,13 @@ An ARM (armv4/armv7l) linux port is underway. A port to GNU/Hurd Initial scaffold, built by @file{build-aux/build-scaffold.sh}: @example - @file{lib/x86-mes-gcc/exit-42.S} - @file{lib/x86-mes/elf32-0exit-42.hex2} - @file{lib/x86-mes/elf32-body-exit-42.hex2} + @file{lib/linux/x86-mes-gcc/exit-42.S} + @file{lib/linux/x86-mes/elf32-0exit-42.hex2} + @file{lib/linux/x86-mes/elf32-body-exit-42.hex2} - @file{lib/x86-mes-gcc/hello-mes.S} - @file{lib/x86-mes/elf32-0hello-mes.hex2} - @file{lib/x86-mes/elf32-body-hello-mes.hex2} + @file{lib/linux/x86-mes-gcc/hello-mes.S} + @file{lib/linux/x86-mes/elf32-0hello-mes.hex2} + @file{lib/linux/x86-mes/elf32-body-hello-mes.hex2} @end example Porting MesCC: @@ -1038,7 +1082,7 @@ Porting MesCC: @file{mes/module/mescc/i386/info.mes} @end example -@node The Perfect Setup +@node The Perfect Setup, Coding Style, Porting GNU Mes, Contributing @section The Perfect Setup The Perfect Setup to hack on Mes is basically the perfect setup used @@ -1053,7 +1097,7 @@ on-line documentation (docstrings), context-sensitive completion, @kbd{M-.} to jump to an object definition, a REPL to try out your code, and more (@pxref{Introduction,,, geiser, Geiser User Manual}). -@node Coding Style +@node Coding Style, Submitting Patches, The Perfect Setup, Contributing @section Coding Style In general our code follows the GNU Coding Standards (@pxref{Top,,, @@ -1089,7 +1133,7 @@ Additionally, in Mes we prefer to format @code{if} statements like this else)) @end example -@node Submitting Patches +@node Submitting Patches, , Coding Style, Contributing @section Submitting Patches Development is done using the Git distributed version control system. @@ -1139,7 +1183,7 @@ Join us on @code{#bootstrappable} on the Freenode IRC network or on Please send bug reports with full details to @email{bug-mes@@gnu.org}. @c ********************************************************************* -@node Acknowledgments +@node Acknowledgments, Resources, Contributing, Top @chapter Acknowledgments We would like to thank the following people for their help: Jeremiah @@ -1151,7 +1195,7 @@ LISP-1.5 and Alan Kay for their inspiring comment on @url{https://queue.acm.org/detail.cfm?id=1039523, Page 13}. @c ********************************************************************* -@node Resources +@node Resources, GNU Free Documentation License, Acknowledgments, Top @chapter Resources @itemize @@ -1185,17 +1229,17 @@ at the freenode IRC network. @end itemize @c ********************************************************************* -@node GNU Free Documentation License +@node GNU Free Documentation License, Concept Index, Resources, Top @appendix GNU Free Documentation License @cindex license, GNU Free Documentation License @include fdl-1.3.texi @c ********************************************************************* -@node Concept Index +@node Concept Index, Programming Index, GNU Free Documentation License, Top @unnumbered Concept Index @printindex cp -@node Programming Index +@node Programming Index, , Concept Index, Top @unnumbered Programming Index @syncodeindex tp fn @syncodeindex vr fn