Commit Graph

60 Commits

Author SHA1 Message Date
Dor Askayo 2227a2053a Mount a few missing device nodes in chroot bootstrap modes
The gap between the chroot and non-chroot bootstrap modes started in,
7ecad37 because BuildStream's sandbox was missing some device nodes.

With the latest version of buildbox-run-bubblewrap, a few of the
missing devices were made available to BuildStream's sandbox and
there is no longer a need to keep this gap. In fact, compatibilty
with newer BuildStream versions now requires these devices to be
present.

This change also partially reverts d0a5221, which had to remove
/dev/ptmx as a consequence of the aforementioned gap.
2023-07-16 01:41:01 +03:00
fosslinux d98f97e214 Introduce parallelism
By using --cores argument to rootfs.py, JOBS= is set in the
live-bootstrap environment, and -j${JOBS} is used on builds. This speeds
larger packages up significantly.

A fair number of packages do not build properly with parallelism. Most
of these, at least for now, are disabled with -j1.
2023-04-13 17:13:53 +10:00
Eduardo Sánchez Muñoz 043fdd448a Add `--fail` flag to curl download commands
By default, curls downloads the HTML error page and exits with code 0 when a server replies with a HTTP error code (e.g., 404), causing a SHA256 mismatch afterwards.

Adding the `--fail` flag makes curl exit with non-zero error code and print an error like "The requested URL returned error: 404", making it a bit easier to diagnose distfile download issues.

Also replaced `-L` with `--location` for the sake of script readability.
2023-03-19 12:28:09 +01:00
Bastian Bittorf 464433c911 cosmetic fix for --force-timestamps, avoid touching other filesystems
by using find-option -xdev and so: do not try to operate on
e.g. /proc which avoid messages like:

touch: setting times of '/proc/fs/nfsd': Operation not permitted
touch: setting times of '/proc/1': Operation not permitted

closes #264
2023-03-13 18:48:37 +01:00
fosslinux 76a5639b30 Strip created binaries & libraries 2023-03-06 17:35:35 +11:00
Andrius Štikonas d4eeba9d7a Improvements to early package creation:
* get_files produces list without extra ./ between extra subdirectories.
* Emtry directories are also included.
* Simplified src_pkg function.
* Use local variables instead of unsetting global variables in src_apply.
2023-02-22 10:27:56 +08:00
fosslinux 2577d2e704 Fix a packaging bug
Previously, symlinks to directories were followed early
in the bootstrap. This is incorrect behaviour.

While never actually encountered (hence no checksum changes),
this could be a problem.
2023-02-06 16:58:10 +11:00
Dor Askayo 74c852b166 Add /dev/shm for sysc
This allows named semaphores to be created. See sem_overview(7).
2023-02-02 01:55:33 +02:00
fosslinux d0a522113f Fix QEMU/chroot differences in outputs for Python 2023-01-18 08:20:34 +11:00
Andrius Štikonas 938d919503 Fix creation of binary tar packages.
Closes #223
2023-01-13 01:09:57 +00:00
fosslinux f4ebc241ac Fix linters (whoops!) 2023-01-07 19:12:50 +11:00
fosslinux b46b776404 Support .tgz files in extract phase 2023-01-05 21:20:18 +11:00
fosslinux d91deb50c9 Remove XBPS 2023-01-05 20:44:19 +11:00
fosslinux a89c76cc73 Fix musl-1.2.3 when umask on host system is not 022 2023-01-02 16:44:23 +11:00
rick-masters d2c669e0fc Always use file for checksums, mes built sha256sum doesn't support stdin. 2022-12-05 18:14:19 +00:00
Eduardo Sánchez Muñoz 6f4afb4e3f Make shellcheck happy 2022-11-28 20:10:30 +01:00
fosslinux 1c6c9a404d Hotfix shellcheck 2022-11-02 14:35:48 +11:00
fosslinux 31bc944dd3 Establish different XBPS packages for different versions 2022-10-30 21:07:57 +11:00
fosslinux c523eb9b47 Fix repo preseed in update_checksums mode 2022-10-30 21:07:57 +11:00
fosslinux 38b90d6a96 Make preseed work in sysc
- TODO: tar-1.34 does not work properly with preseed.
2022-10-30 21:07:57 +11:00
fosslinux 042deed5e5 Fix extract-only for old tar 2022-10-10 22:23:04 +11:00
fosslinux 10a55522a2 Change sources getting method.
- Rather than defining the urls where they are gotten (python sysa,
  python sysc, inside sysc), a spec file is now used that is easily
  interpretable and tool-independent.
- This is interpreted by rootfs.py and inside sysc.
- This is also used to make sources available and extract sources.
- Manual dirname selection is no longer required as is tarball renaming
  upon download - all of this is handled automatically.

Fixes #188
2022-10-02 09:48:20 +11:00
fosslinux 25f37ba926 Various fixes from rebase
This took a while so a bunch of stuff got broken in rebase.
2022-06-18 09:10:11 +10:00
fosslinux 0ce50a6393 Add options for;
- disk to be created (blank disk given to live-bootstrap) (default)
- disk to already exist but sources downloaded within live-bootstrap
- sources to be downloaded outside live-bootstrap (non-blank disk given
  to live-bootstrap)

Also migrate sysb to use sys_transfer in QEMU mode also.

Note that this means copy_sysc is now irrelevant. sysc is *always*
sourced from sysa.
2022-06-10 13:33:16 +10:00
fosslinux 8008de8e73 Add file downloading logic
Add urls to sysc, and modify some tarballs to http (non s) before ssl
2022-06-10 13:30:58 +10:00
Andrius Štikonas b0f0fa3a95 Fix return from bin_preseed. 2022-06-02 20:25:23 +01:00
Andrius Štikonas 0b3782d61b Allow preseeding with binary packages. 2022-05-29 00:04:57 +01:00
fosslinux fc0368aa6a
Merge pull request #169 from stikonas/links
Merge symlinks into tarball packages.
2022-05-20 22:46:30 +10:00
Andrius Štikonas 8f61b8400e Merge symlinks into tarball packages. 2022-05-20 09:39:57 +01:00
Dor Askayo 507f945419 Remove /dev/null only if it's not a character device 2022-05-19 21:55:25 +03:00
Andrius Štikonas 21f4174dc3 Fix packaging issues
* Bzip2 was manually installed directly into filesystem, so bzip2 package was empty.
  Fixed by installing it to destdir. bzip2 moves out its binary before installing
  its own package.
* sha256sum from stage0-posix was not checking any checksums because it does not
  accept piped input. Fixed by using temporary file.
* grep was broken for a short time (but with fixed bzip2 package this caused failures)
  due to touch creating grep file of zero size (egrep symlink was touched).
  Fixed by implementing touch -h

Resolves #156, resolves #166, resolves #167
2022-05-17 10:19:50 +01:00
Dor Askayo 3c60bdebc3 Improve reproducibility of tarball packages
This ensures the user running the bootstrap and its umask don't
affect the permission metadata in the tar archive.
2022-05-15 10:46:18 +03:00
Andrius Štikonas ef0030bbf3 Reinstall system from packages during sysa->sysc and sysb->sysc transitions. 2022-05-09 22:53:46 +01:00
Andrius Štikonas 424aa5be38 Move from mes-m2 to mes-0.24. 2022-05-05 23:59:52 +01:00
fosslinux c189dad168 Order early tarballs deterministically
Fixes #151
2022-05-05 17:44:47 +10:00
fosslinux f1600467a7 Variety of improvements
- Rename sources to distfiles for clarity.
- Per sys(a/c) distfiles to reduce rootfs.py processing and reduce RAM
  usage in sysa.
- Canonicalise early kaem mes/tcc files to kaem script conventions.
- Cleanup unused setup in python.
2022-05-05 17:44:47 +10:00
Andrius Štikonas 5b032cb46c rootfs.py refactoring.
Switch to bzip2 packages
Move most of the preprocessing done by rootfs.py
into kaem and bash scripts inside live-bootstrap.
2022-04-21 00:49:56 +01:00
fosslinux 5c88f1c87f Add sysb and sysc scaffolding.
Now that we have the Linux Kernel built, we move to a full-disk (rather
than initramfs) setup in sysc. However, we cannot assume the seed kernel
has support for mounting hard drives. So, first we need to kexec into
sysb, which is used as a jumping off point to create the hard drive for
sysc.

Additionally, since 2.6.16 does not have support for on-demand initramfs
(initramfs must be built into kernel), we will have to rebuild the linux
kernel within sysb without the initramfs.

All of this process is not performed for chroot mode. Instead, we skip
sysb and jump straight to sysc, copying over appropriate data.

The python scripts have been changed slightly. Each sys* inherits
SysGeneral, which contains various functions which are not specific to
any sys* and simplifies those files. rootfs now also handles sysb and
sysc.

bootstrap.cfg also gives an indication whether we are running in a
chroot to avoid attempting to kexec/mount within a chroot.
2021-08-27 14:54:08 +10:00
fosslinux 4c6e7afa42 Remove build directory
This extremely conserves storage space. Needed for larger packages to
fit within 4GB RAM limit of i386 no-PAE.
2021-08-27 14:54:07 +10:00
Andrius Štikonas c8784cbcd3 Split installed files out of /after into /image. 2021-06-29 17:49:34 +01:00
fosslinux c05c16e12c helpers.sh: add 5th argument, directory to cd into 2021-05-26 07:27:22 +10:00
fosslinux 338c85c8b8 Use absolute instead of relative paths in helpers.sh 2021-05-26 07:27:22 +10:00
fosslinux c168a3bd4c
Merge pull request #115 from melg8/master
Add coreutils 8.32 and fix creation dates
2021-05-24 07:34:23 +10:00
Melg Eight cf4ce56603
Add config file and force timestamp conditional option 2021-05-20 02:04:39 +03:00
Andrius Štikonas cd3fb17aa6 Unpack tarballs with --no-same-owner. 2021-05-08 13:53:42 +01:00
Andrius Štikonas a2c5745da4 helpers.sh: Add a helper function to call appropriate default stage function. 2021-05-03 23:48:06 +01:00
fosslinux 54234a5683 Add multiple tarball support, named args in rootfs.sh
1. Add named args to make this more clear and avoid unnessecarily
   setting default values for certain values ot set others positioned
   later.
2. Add support for multiple tarballs in rootfs.sh.
3. Add support for extracting multiple tarballs (name dosen't matter
   anymore) in helpers.sh.
4. Add comment for 4th arg to build.
2021-04-06 08:59:37 +10:00
Andrius Štikonas 70d1fc4163 Add support for DESTDIR variable in make install. 2021-04-02 12:24:40 +01:00
Paul Dersey 8565d26fef Update initial build of sed from version 4.0.7 to 4.0.9 2021-03-17 17:11:16 -04:00
Paul Dersey 6a5e57c2f8 helpers.sh: update default_src_unpack to support uncompressed tar files 2021-03-17 17:11:16 -04:00