Bug#1032610: reliably composable initramfs - zero-pad output to allow concatenation
Package: initramfs-tools-core
Version: 0.140
Severity: wishlist
File: /usr/sbin/mkinitramfs
Tags: patch
mkinitramfs supports compression that neither encodes size, nor is
guaranteed to signal the end of compressed archive. Linux refuses to
parse additional archives beyond the first compressed one - unless they
start at 4-byte boundary like uncompressed ones do.
I request
A) to modify mkninitramfs to append such padding and
B) to add tests to ensure concatenation keeps working (even after, say,
compression algo/lvl changes)
Rationale:
1. Documentation: After the requested change, inserting custom
configuration or installation prerequisites to provided installation
media will just work again. Append your changes, done.
2. It worked before: Without compression, still just works(tm). Should
never have been broken.
3. Debugging: With compression and before the requested change, raw cpio
and xz-compressed (len modulo 4 == 0) can be concatenated just fine. But
compressed-after-lz4 or raw-after-compressed makes the kernel ignore the
second half - and if that was not surprising enough, it *sometimes*
works, by chance.
4. Least surprise: With or without this change, alignment and/or EOF
signal might be added by the bootloader [10]. Since this can surprise
the person working on the image, it would be beneficial to preempt it in
ways visible in the filesystem.
Considerations:
5. Breaks reproducible builds: Post-change files are up to 0-3 bytes
bigger. See also #855357
6. Boot bugs: Unlikely, padding was always allowed; since the
introduction of lz4 even necessary.
7. Breakage in non-kernel Debian software: Unlikely; unmkinitramfs had
already been documented to not support multiple compressed archives.
Scripts would not depend on its behavior when parsing such anyway.
8. Breakage in bootloaders/firmware: Unlikely; all compression
algorithms (save xz) already sometimes produce the post-change
alignment, by (~25%) chance.
The attached sample code copies the desired bytes from /dev/zero. Also
attached is an autopkgtest script that calls qemu-system on the
concatenation of all currently supported compression algorithms in an
attempt to confirm that the kernel has not errorred out before parsing
the last one. It worked on my system, but even if it also works on
yours: beware that it is an amd64-only, non-EFI-only test.
dmesg samples, just to aid in detecting duplicates of this report
Initramfs unpacking failed: invalid magic at start of compressed archive
Initramfs unpacking failed: Decoding failed
Initramfs unpacking failed: broken padding
[10] see grub.git/grub-core/loader/linux.c@a8c473
[11] see linux.git/Documentation/filesystems/ramfs-rootfs-initramfs.rst
-- System Information:
Debian Release: 11.6
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'stable-security'), (500,
'stable')
Architecture: amd64 (x86_64)
Kernel: Linux 5.10.0-21-amd64 (SMP w/4 CPU threads)
Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8), LANGUAGE
not set
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages initramfs-tools-core depends on:
ii coreutils 8.32-4+b1
ii cpio 2.13+dfsg-4
ii e2fsprogs 1.46.2-2
ii klibc-utils 2.0.8-6.1
ii kmod 28-1
ii logsave 1.46.2-2
ii udev 247.3-7+deb11u1
Versions of packages initramfs-tools-core recommends:
ii busybox 1:1.30.1-6+b3
ii pigz 2.6-1
Versions of packages initramfs-tools-core suggests:
ii bash-completion 1:2.11-2
-- no debconf information
diff --git a/initramfs-tools-0.140/mkinitramfs b/initramfs-tools-0.140/mkinitramfs
index 9516992..332b148 100755
--- a/initramfs-tools-0.140/mkinitramfs
+++ b/initramfs-tools-0.140/mkinitramfs
@@ -451,4 +451,16 @@ if [ -s "${__TMPCPIOGZ}" ]; then
fi
} >"${outfile}" || exit 1
+# compressed cpio might create a total length breaking otherwise working concatenation
+# only done once, as early cpio is not compressed, thus unaffected
+# https://www.kernel.org/doc/html/latest/filesystems/ramfs-rootfs-initramfs.html
+# [dmesg] Initramfs unpacking failed: invalid magic at start of compressed archive
+# [dmesg] Initramfs unpacking failed: Decoding failed
+# [dmesg] Initramfs unpacking failed: broken padding
+outsize=$(wc -c <"${outfile}")
+pad=$(( 4 - (outsize % 4) + 4 ))
+# printf "\n\nCOMPRESS %s PADDING %d\n\n" "$compress" "$pad" >&2
+dd status=none bs=1 count=$pad if=/dev/zero >>"${outfile}" ||
+ { echo "E: mkinitramfs failure padding $?" >&2; exit 1; }
+
exit 0
diff --git a/initramfs-tools/initramfs-tools-0.140/debian/tests/amd64-appendcpio b/initramfs-tools-0.140/debian/tests/amd64-appendcpio
new file mode 100755
index 0000000..2293e06
--- /dev/null
+++ b/initramfs-tools-0.140/debian/tests/amd64-appendcpio
@@ -0,0 +1,69 @@
+#!/bin/sh -e
+
+SUPPORTED_FLAVOURS='amd64 generic'
+ROOTDISK_QEMU_IF=virtio
+ROOTDISK_LINUX_NAME=nonexistent
+. debian/tests/test-common
+
+# This test makes no attempt at forcing the failure to happen.
+# Archives aligned by accident prodice different results on each run
+# export SOURCE_DATE_EPOCH=0
+
+# FIXME: not actually configuration data
+truncate -s 0 "${CONFDIR}/globo"
+
+lastcompress=lzop
+for compress in lz4 xz gzip zstd bzip2 lzma "$lastcompress"; do
+cat >>"${CONFDIR}/initramfs.conf" <<EOF
+MODULES=list
+BUSYBOX=y
+COMPRESS=${compress}
+EOF
+cat >"${CONFDIR}/modules" <<EOF
+ext2
+EOF
+build_initramfs
+
+# TODO: lsinitramfs should be able to parse this too
+# challenge: exit code on garbage input - backwards-incompatible change needed
+# lsinitramfs "${INITRAMFS}" | grep -qw busybox
+
+(
+cd "${CONFDIR}"
+umask 022
+
+# simplest uncompressed, but we could not be certain kernel parsed this
+# printf "1\n" >offset
+# printf "offset\n" | cpio --format=newc --create --quiet --reproducible </dev/null >>"${INITRAMFS}"
+
+# uncompressed cpio that only when parsed correctly prints canary (if extracted by the kernel)
+mkdir -p scripts/init-premount
+# FIXME: consider full INIT_MESSAGE checking instead of rebooting right away
+cat >scripts/init-premount/ORDER <<EOF
+printf "\nsuccessful boot: %s %s\n" "canary" "${compress}"
+reboot -f
+EOF
+cpio --format=newc --create --quiet -R 0:0 --reproducible >>"${INITRAMFS}" <<'EOF'
+scripts
+scripts/init-premount
+scripts/init-premount/ORDER
+EOF
+)
+cat "${INITRAMFS}" >>"${CONFDIR}/globo"
+
+# endfor: $compress
+done
+
+INITRAMFS="${CONFDIR}/globo"
+
+# nocheck: qemu exits 1 either way - no root found=panic, or reboot
+# lib/decompress.c: pr_debug("Compressed data magic: ...")
+# init/do_mounts_rd.c: printk(KERN_NOTICE "RAMDISK: Loading ...")
+# init/initramfs.c: printk(KERN_INFO "Trying to unpack rootfs ...")
+run_qemu_nocheck_amd64 'loglevel=8 dyndbg="file lib/decompress.c +p"' 'dyndbg="file init/do_mounts_rd.c +p"' 'dyndbg="file init/initramfs.c +p"' panic=1 rootwait=1 rootdelay=1
+
+# look for dmesg like "Decoding failed"
+grep -q "Initramfs unpacking failed" "${OUTPUT}" && exit 1
+# Check that appended script ran - and that it mentions compression of last appended cpio
+# FIXME: compare directory listing against entire (not just lastitem) for..in array above
+grep -q "^successful boot: canary $lastcompress" "${OUTPUT}" || exit 1
diff --git a/initramfs-tools-0.140/debian/tests/control b/initramfs-tools-0.140/debian/tests/control
index b3efe72..7466779 100644
--- a/initramfs-tools-0.140/debian/tests/control
+++ b/initramfs-tools-0.140/debian/tests/control
@@ -10,6 +10,10 @@ Tests: amd64-busybox
Depends: @, qemu-system-x86, linux-image-amd64 | linux-image-generic:amd64, klibc-utils, busybox | busybox-initramfs, genext2fs
Restrictions: skip-not-installable
+Tests: amd64-appendcpio
+Depends: @, qemu-system-x86, linux-image-amd64 | linux-image-generic:amd64, klibc-utils, busybox | busybox-initramfs, lz4, xz-utils, lzma, lzip, bzip2, zstd, lzop
+Restrictions: skip-not-installable
+
Tests: amd64-ata-only
Depends: @, qemu-system-x86, linux-image-amd64 | linux-image-generic:amd64, klibc-utils, genext2fs
Restrictions: skip-not-installable
Reply to: