Bug#1031325: e2fsprogs 1.47.0 introduces a breaking change into Bookworm, breaking grub and making installations of Ubuntu and Debian releases via debootstrap impossible
On Wed, Feb 15, 2023 at 11:47:08AM +0200, Adrian Bunk wrote:
>
> For normal library dependencies
> Depends: libc6 (>= 2.34)
> will do the right thing automatically.
Sure, but dependencies only apply if you are using building packages.
If you are not building packages, but just moving binaries between
distribution versions --- which is analogous to what is going on here
when someone moves a file system from one distro version to another.
For example:
% echo 'int main() { printf("Hello, world\n"); return 0; }' > /tmp/foo.c ; gcc -w -o /tmp/foo /tmp/foo.c ; /tmp/foo ; schroot -c bullseye-amd64 /tmp/foo
Hello, world
/tmp/foo: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /tmp/foo)
> e2fsprogs adding Breaks against older versions of bootloaders and other
> software that lacks support for the new default might be a good idea.
>
> The situtation is different when a relationship cannot be reliably
> expressed by package dependencies.
Unfortunately, package dependencies won't address the concern raised
by Daniel. The mkfs.ext4, debootstrap, etc., are being run on a one
version of the distro, such as for example sid or Bookworm, but then
when the grub bootloader is installed --- presumably in a chroot, so
that you get the Bullseye grub on a Bullseye VM image --- it won't
know that the base file system was built on Sid or Bookworm. Of
course, if you run the mkfs.ext4 in a Bullseye chroot, everything will
work *just* *fine*. It's just that this isn't Daniel's workflow.
> For the kernel the answer to your "how long" is that packages should
> also work with the kernel from the previous and the kernel from the
> next Debian release.
This isn't a problem with the kernel. For the ext4 feature that
Daniel cited, metadata_csum_seed was first supported in 4.4 kernel
(note that Debian *stretch* used the 4.9 kernel and 4.4 was first
released in 2016 --- six years ago). On the file system utility side,
e2fsprogs 1.43 also support for metadata_csum_seed (again, support
going back to stretch).
The xfs bigtime feature, which was similarly enabled in Bookworm, was
supported in the kernel the xfsprogs for **many** years. In general,
file system developers do understand the concern of needing to wait
before enabling a feature by default in mke2fs.conf.
The issue here is in grub support. Part of the issue is that
admittedly, file system developers don't worry as much about
bootloaders as much as we do for kernels and the file system
utilities. The other is that the grub2 upstream simply hasn't been as
responsive or quick at doing releases. The fix landed in grub2
upstream in June of 2021. So all distributions tend to carry large
number of grub2 cherry-picks.
> If the feature stays enabled by default in bookworm:
> Everyone using grub is an x86 thing, for embedded ARM it is more common
> to use a once ported ancient u-boot on your hardware forever.[1]
> A bug against the release-notes pseudo-package with text describing
> the incompatibility and possible workarounds would be appreciated.
Would you prefer that we repurpose (and retitle) this bug, or open a
new one?
Here's a quick rough draft of what that text might look like, at least
for the ext4 metadata_csum_seed. (If it's helpful, I can try to also
write a blurb for XFS, although I'd want to run it by the XFS kernel
maintainer, Darrick Wong first.)
---- cut here ---
In the Bookworm release, the mkfs.ext4 will enable the orphan_file and
metadata_csum_seed feature. The orphan_file feature significantly
improves performance for workloads which are truncated or deleting
files in parallel. The orphan_file feature is an ext4 "compat"
feature, so file systems with the orphan_file feature can be mounted
on older kernels who do not understand the orphan_file feature, so
long as the file system was cleanly unmounted. If the file system was
not cleanly unmounted (for example, due to a crash or power failure),
the file system must be fsck'ed or mounted on a Bookworm system before
the file system is transferred to an older system.
The metadata_csum_seed feature allows the file system UUID to
be modified without needing to update all of the file system metadata,
which is often important in VM context where root file systems are
cloned from read-only images, and having unique UUID's is important to
avoid confusion when the block device might be mounted on another
whose root file system is similarly cloned from the same base image.
The metadata_csum_seed feature is an ext4 "incompat" feature, which
means that the kernel must understand the feature before it will be
able mount a file system with that feature. Fortunately, the Linux
kernels starting with 4.4 (and the Debian 9.0 "stretch" release)
support the metadata_csum_seed feature. So data disks formatted with
metadata_csum_seed can be mounted on systems with older distro
versions without concerns in nearly all case.
Unfortunately, grub2 support for the metadata_csum_seed feature is
only available starting in Bookworm. As a result, running mkfs.ext4
from Bookworm while building a bootable image for older distributions
(for example, for a VM or Bare Metal deployment), may result in an
image which will not be supported on a older version of Grub on those
distributions. There are a number of workarounds for Bookworm users
who want to build bootable image for older distributions.
* Most generally, just as running and installing grub needs to be
done in a chroot, it is best practice to run the mkfs command in a
chroot as well.
Beyond ext4's metadata_csum_seed feature, similar issues for other
file systems may cause bootability or stability concerns on older
distributions. For example, XFS's bigtime feature, which fixes
the 2038 bug for XFS timestamps, may cause bootability problem on
older distribution. There may be other file systems features (for
example, on btrfs) that technically be supported on older kernels,
but were not enabled by default on older distributions for
stability or data safety concerns. Using a chroot for the target
distribution while running mkfs will address all of these issues.
* The script creating the bootable image for an older distribution
can add the command-line option "-O ^metadata_csum_seed" to the
mkfs.ext4 invocation.
* After the bootable image has been created, the metadata_csum_seed
feature can be cleared using the command "tune2fs -O
^metadata_csum_seed" distro.img".
* The /etc/mke2fs.conf file controls what file system features and
parameters are used by default when creating ext4 file systems.
So if a Bookworm user would like to create ext4 file systems that
will be supported by versions of grub older than what is shipped
in Bookworm, they can find the metadata_csum_feature in the ext4
subsubsection in the [fs_types] section, and simply remove
metadata_csum_feature.
There are other file system featuers that are a bit more
specialized, such as the casefold feature, which the system
administrator can enable by default by editing the
/etc/mke2fs.conf file. Please see the mke2fs.conf(8) and ext4(5)
man pages for more details.
--- cut here ---
If this is too long, some of this could be moved into a NEWS entry in
e2fsprogs. Let me know...
Cheers,
- Ted
Reply to: