andrius
/
live-bootstrap
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Merge pull request #461 from fosslinux/update-parts 

Update parts.rst
This commit is contained in:
Andrius Štikonas 2024-04-17 19:22:50 +01:00 committed by GitHub
parent 85026900ac cd74988efb
commit 3bbd639937
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 206 additions and 65 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

271
parts.rst
View File

@ -4,6 +4,7 @@
.. SPDX-FileCopyrightText: 2021 Paul Dersey <pdersey@gmail.com>
.. SPDX-FileCopyrightText: 2021-23 fosslinux <fosslinux@aussies.space>
.. SPDX-FileCopyrightText: 2021 Melg Eight <public.melg8@gmail.com>
.. SPDX-FileCopyrightText: 2024 Gábor Stefanik <netrolller.3d@gmail.com>
.. SPDX-License-Identifier: CC-BY-SA-4.0
@ -152,8 +153,9 @@ mescc-tools-extra
utilities ``cp`` and ``chown``. This allows us to have one unified
directory for our binaries. Furthermore, we also build ``sha256sum``, a
checksumming tool, that we use to ensure reproducibility and authenticity
of generated binaries. We also build initial ``untar``, ``ungz`` and ``unbz2``
utilities to deal with compressed archives.
of generated binaries. We also build initial ``untar``, ``ungz``, ``unxz``
and ``unbz2`` utilities to deal with compressed archives, as well as
``replace``, a trivial search-and-replace program.
live-bootstrap seed
===================
@ -179,8 +181,9 @@ The following directives are supported:
* ``build``, builds a particular package defined in ``steps/``.
* ``improve``, runs a script making a distinct and logical improvement to the
live bootstrap system.
* ``define``, define a variable evaluated from other constants/variables.
* ``define``, defines a variable evaluated from other constants/variables.
* ``jump``, moves into a new rootfs/kernel using a custom script.
* ``uninstall``, removes a previously built package for file.
checksum-transcriber 1.0
========================
@ -196,7 +199,7 @@ simple-patch 1.0
text block given to it, and replacing it with another text block. This is
sufficient for the early patching required before we have full proper GNU patch.
mes 0.25
mes 0.26
========
GNU ``mes`` is a scheme interpreter. It runs the sister project ``mescc``,
@ -209,6 +212,10 @@ to this part:
2. We then use this to recompile the Mes interpreter as well as building
the libc. This second interpreter is faster and less buggy.
The ``mescc`` component depends on the ``nyacc`` parsing library, version 1.00.2.
We use a modified version, 1.00.2-lb1, which incorporates Timothy Sample's
changes required to bootstrap ``mescc`` without relying on pregenerated files.
From this point until musl, we are capable of making non-standard and strange
libraries. All libraries are in ``/usr/lib/mes``, and includes are in
``/usr/include/mes``, as they are incompatible with musl.
@ -243,13 +250,13 @@ using older versions compilable by tinycc. Prior to this point, all tools
have been adapted significantly for the bootstrap; now, we will be using
old tooling instead.
fiwix 1.4.0-lb1
fiwix 1.5.0-lb1
===============
If the kernel bootstrap option is enabled then the Fiwix kernel is built next.
This is a Linux 2.0 clone which is much simpler to understand and build than
Linux. This version of Fiwix is a fork of 1.4.0 that contains many
modifications and enhancements to support live-boostrap.
Linux. This version of Fiwix is an intermediate release on top of 1.5.0 that
contains many modifications and enhancements to support live-boostrap.
lwext4 1.0.0-lb1
================
@ -266,20 +273,6 @@ kexec-fiwix
If the kernel bootstrap option is enabled then a C program `kexec-fiwix` is compiled
and run which places the Fiwix ram drive in memory and launches the Fiwix kernel.
esfu 1.0
========
This is an extremely crippled basic implementation of ``mount`` and ``mknod``.
Sufficient only for the next step.
early_mount_disk
================
When using kernel bootstrap, distfiles from this point exist on an external
disk. Using ``esfu``'s ``mount`` and ``mknod``, we are able to mount this disk.
This is unnecessary when not using kernel bootstrap as everything is done on the
disk.
make 3.82
=========
@ -329,13 +322,15 @@ we skip them.
The ``cp`` in this stage replaces the ``mescc-tools-extra`` ``cp``.
heirloom devtools
=================
byacc 20240109
==============
``lex`` and ``yacc`` from the Heirloom project. The Heirloom project is
a collection of standard UNIX utilities derived from code by Caldera and
Sun. Differently from the analogous utilities from the GNU project, they
can be compiled with a simple ``Makefile``.
The Berkeley Yacc parser generator, a public-domain implementation of the
``yacc`` utility. Differently from the analogous ``bison`` utility from the
GNU project, it can be compiled with a simple ``Makefile``.
Some code is backported from an earlier version of byacc, 20140101, because
of an incompatibility of newer versions with meslibc.
bash 2.05b
==========
@ -346,7 +341,19 @@ kaem, including proper POSIX sh support, globbing, etc.
Bash ships with a bison pre-generated file here which we delete.
Unfortunately, we have not bootstrapped bison but fortunately for us,
heirloom yacc is able to cope here.
Berkeley Yacc is able to cope here.
setup_repo
==========
This is a simple script that sets up the ``/external/repo`` directory to hold
binary tarballs of artifacts built in each step. It also creates ``base.tar.bz2``,
a tarball containing every artifact built before ``setup_repo``, which have no
individual repository tarballs corresponding to them.
From this point on, every package is built and installed into a temporary
directory, packaged from there into a repository tarball, and then installed
onto the live system from the newly built repository tarball.
update_env
==========
@ -354,17 +361,26 @@ update_env
This is a simple script that makes some small updates to the env file that were
not possible when using kaem.
flex 2.5.11
===========
merged_usr
==========
``flex`` is a tool for generating lexers or scanners: programs that
recognize lexical patterns.
Sets up symlinks from folders outside the ``/usr`` hierarchy to the corresponding
``/usr`` ones ("merged-usr" file system layout).
Unfortunately ``flex`` also depends on itself for compiling its own
scanner, so first flex 2.5.11 is compiled, with its scanner definition
manually modified so that it can be processed by lex from the Heirloom
project (the required modifications are mostly syntactical, plus a few
workarounds to avoid some flex advanced features).
populate_device_nodes
=====================
Sets up important device nodes under ``/dev``. These nodes are temporary, as once
the Linux kernel is started, ``devtmpfs`` is used to maintain ``/dev``.
open_console
============
In interactive mode only, sets up an interactive Bash console, accessible by
pressing Ctrl+Alt+F2. Because the early Bash doesn't support true interactive
operation, we emulate it using a REPL. A side effect of this is that after every
command entered, one must press Enter followed by Ctrl+D, rather than just Enter
as with a real interactive shell.
tcc 0.9.27 (patched)
====================
@ -424,6 +440,30 @@ m4 1.4.7
flex 2.6.4 and bison. It allows macros to be defined and files to be
generated from those macros.
heirloom devtools
=================
``lex`` from the Heirloom project. The Heirloom project is a collection
of standard UNIX utilities derived from code by Caldera and Sun.
Differently from the analogous utilities from the GNU project, they can
be compiled with a simple ``Makefile``.
Because issues with the Heirloom version of ``yacc`` compiled against musl,
we continue using Berkeley Yacc together with Heirloom ``lex`` for the next
few steps.
flex 2.5.11
===========
``flex`` is a tool for generating lexers or scanners: programs that
recognize lexical patterns.
Unfortunately ``flex`` also depends on itself for compiling its own
scanner, so first flex 2.5.11 is compiled, with its scanner definition
manually modified so that it can be processed by lex from the Heirloom
project (the required modifications are mostly syntactical, plus a few
workarounds to avoid some flex advanced features).
flex 2.6.4
==========
@ -690,12 +730,19 @@ GCC can build the latest as of the time of writing musl version.
We also don't need any of the TCC patches that we used before.
To accomodate Fiwix, there are patches to avoid syscalls set_thread_area and clone.
Linux headers 5.10.41
=====================
Linux headers 4.14.341-openela
==============================
This gets some headers out of the Linux kernel that are required to use the
kernel ABI, needed for ``util-linux``.
The version of the Linux kernel used comes from the Open Enterprise Linux
Association, who maintain this version as a continuation of the now ended
4.14 LTS release, to the same maintenance standards as the LTS.
Because this isn't directly available as an efficiently compressed tarball,
we start with the final LTS release, version 4.14.336, and apply the
differences using a patch file.
gcc 4.0.4
=========
@ -708,14 +755,13 @@ util-linux 2.19.1
This gives us access to a much less crippled version of ``mount`` and ``mknod``.
The latest version is not used because of autotools/GCC incompatibilities.
move_disk
=========
dhcpcd 10.0.1
=============
In ``kernel-bootstrap`` mode, we have been working off an initramfs for some
things up until now. At this point we are now capable of moving to it entirely,
so we do so.
``dhcpcd`` is a DHCP client daemon, which we later use to obtain an IP address.
Only wired interfaces are supported at the moment.
kbd-1.15
kbd 1.15
========
``kbd`` contains ``loadkeys`` which is required for building the Linux kernel.
@ -755,40 +801,100 @@ kind of soft-restart. It is only built for non-chroot mode, as we only use it
in non-chroot mode. It is used to boot the Linux kernel that will be built next
from the current Linux kernel (when using ``--kernel``).
Linux kernel 4.9.10
===================
clean_sources
=============
A script to remove source tarballs no longer needed for subsequent build steps from
distfiles. This frees space for the Linux kernel build.
clean_artifacts
===============
More space freeing in preparation for the kernel build. This time, we clear out any
artifacts left over from previous builds that weren't packaged in a proper repository
tarball or ``base.tar.bz2``
Linux kernel 4.14.341-openela
=============================
A lot going on here. This is the first (and currently only) time the Linux kernel
is built. Firstly, Linux kernel version 4.9.x is used because newer versions
require much more stringent requirements on the make, GCC, binutils versions.
However, the docs are also wrong, as the latest of the 4.9.x series does not
work with our version of binutils. However, a much earlier 4.9.10 does
(selected arbitrarily, could go newer but did not test), with a small amount
of patching. This is also modern enough for most hardware and to cause few
problems with software built afterwards. Secondly, the linux-libre scripts are used
to deblob the kernel. Every other pregenerated file is appended with ``_shipped``
is built. Firstly, Linux kernel version 4.14.y is used because newer versions
require much more stringent requirements on the make and GCC versions. This is also
modern enough for most hardware and to cause few problems with software built
afterwards. Secondly, since 4.14.y is no longer supported by kernel.org, we use the
last available kernel.org tarball, 4.14.336, and patch it up to 4.14.341-openela,
maintained by the Open Enterprise Linux Association as an ELTS version.
Pregenerated files in the kernel have file names appended with ``_shipped``
so we use a ``find`` command to remove those, which are automatically regenerated.
The kernel config was originally taken from Void Linux, and was then modified
for the requirements of live-bootstrap, including compiler features, drivers,
and removing modules. Modules are unused. They are difficult to transfer to
subsequent systems, and we do not have ``modprobe``.
The linux-libre scripts are no longer used to deblob the kernel, due to undesirable
modifications they make beyond just deblobbing. Instead, the remaining 4 drivers that
ship binary blobs in line with source code are stripped using a patch - neither of
these drivers are relevant to bootstrapping.
The kernel is built in 2 stages:
1. We build ``vmlinux`` and all of its dependencies.
2. After clearing away any unnecessary intermediate files, we build the final
``bzImage`` kernel. This is necessary because the whole build wouldn't fit
in Fiwix's initrd image all at once.
We then kexec to use the new Linux kernel, using ``kexec-tools`` for a Linux
kernel and ``kexec-linux`` for Fiwix.
move_disk
=========
In ``kernel-bootstrap`` mode, we have been working off an initramfs for some
things up until now. At this point we are now capable of moving to it entirely,
so we do so.
finalize_job_count
==================
In ``kernel-bootstrap`` mode, up until this point, we had no multiprocessor
support, and very limited RAM, so all builds used only one thread.
At this point, we allow the full selected thread count to take effect,
speeding up subsequent builds thanks to parallelization.
finalize_fhs
============
Sets up the file system as per the Filesystem Hierarchy Standard (FHS), creating
directories and mounting pseudo-filesystems as necessary.
open_console
============
In interactive mode only, sets up an interactive Bash console, accessible by
pressing Ctrl+Alt+F2, again. This is still the early Bash, requiring Ctrl+D.
swap
====
If enabled in ``bootstrap.cfg``, creates and activates a swap file under the
name ``/swapfile``.
musl 1.2.4
==========
At this point, it is guaranteed that we are running on Linux with thread support,
so we rebuild musl with thread support.
curl 8.5.0
==========
curl 8.5.0 and get_network
==========================
``curl`` is used to download files using various protocols including HTTP and HTTPS.
However, this first build does not support encrypted HTTPS yet. ``curl`` requires
Linux and musl with thread support, which are now available.
Once curl is built, we use dhcpcd to set up networking for downloading subsequent
source packages.
bash 5.2.15
===========
@ -797,6 +903,13 @@ and is built with GNU readline support so it can be used as a fully-featured
interactive shell. ``autoconf-2.69`` is used to regenerate the configure
script and ``bison`` is used to recreate some included generated files.
open_console
============
Now that we have a proper interactive shell available, open another interactive
console (only in interactive mode), this time accessible using Ctrl+Shift+F3, since
Ctrl+Shift+F2 is already occupied by our previous console, running the old Bash.
xz 5.4.1
========
@ -822,10 +935,10 @@ tar 1.34
Newer tar has better support for decompressing .tar.bz2 and .tar.xz archives.
It also deals better with modern tar archives with extra metadata.
coreutils 8.32
coreutils 9.4
==============
We build the latest available coreutils 8.32 which adds needed options to make
We build the latest available coreutils 9.4 which adds needed options to make
results of build metadata reproducible. For example, timestamps are changed with
``touch --no-dereference``.
@ -883,7 +996,7 @@ bison 2.3
This is an older version of bison required for the bison files in perl 5.10.1.
We backwards-bootstrap this from 3.4.1, using 3.4.1 to compile the bison files
in 2.3. This parser works sufficiently well for perl 5.10.5.
in 2.3. This parser works sufficiently well for perl 5.10.1.
bison 3.4.2
===========
@ -919,11 +1032,13 @@ libarchive 3.5.2
``libarchive`` is a C library used to read and write archives.
openssl 1.1.1l
openssl 3.0.13
==============
OpenSSL is a C library for secure communications/cryptography.
We do not use the latest 3.3.0 release because it causes lockups in curl.
curl 8.5.0
==========
@ -975,7 +1090,7 @@ We also add in two patchsets to the compiler;
* one to add support for musl shared library support
* one providing a few compiler flags/features that are required later to build GCC 10
binutils 2.38
binutils 2.41
=============
This version of binutils provides a more comprehensive set of programming tools for
@ -1190,14 +1305,14 @@ C/C++ standards available in GCC 4.7. Instead of manually configuring & compilin
every subdirectory, since we now have ``autogen`` available we are able to use
the top-level configure to build the project. We do not use GCC's bootstrap mode,
where GCC is recompiled with itself after being built, since we're just going
to use this GCC to compile GCC 12, it adds build time for little benefit.
to use this GCC to compile GCC 13, it adds build time for little benefit.
binutils 2.38 (pass 2)
binutils 2.41 (pass 2)
======================
We recompile Binutils with the full intended autogen top-level build system,
instead of the subdirectory build system used before. This creates a binutils
that functions completely correctly for the build of GCC 12 (eg, fixes the
that functions completely correctly for the build of GCC 13 (eg, fixes the
mistaken plugin loading support). Other modern features are added, including;
* threaded linking
@ -1281,3 +1396,29 @@ m4 1.4.19
We are in need of a newer version of m4 for some modern software. Attempts to
update m4 1.4.7 earlier in the bootstrap demonstrate some issues with Fiwix,
so we build a newer m4 at the end of the bootstrap instead.
cleanup_filesystem
==================
Remove any remaining loose build artifacts from ``steps``.
If preseeding was used, this step also removes the ``repo-preseeded`` directory.
null_time
=========
If FORCE_TIMESTAMPS is enabled, resets all file times in the file system to the
Unix epoch, to ensure maximum file system reproducibility.
update_checksums
================
If checksum updating is enabled, regenerates SHA256SUMS.pkgs to contain the actual
hashes of the packages just built.
after
=====
At the end of the bootstrap, executes any additional shell scripts placed in the
``/steps/after`` directory (if it exists), opens an interactive console (only in
interactive mode), and finally ensures a clean shutdown of the bootstrap system
(only needed in qemu and on bare metal).