Hello world,
When we had our last tech-ctte meeting, 2020.10.21¹, I volunteered to
write up a summary of our positions regarding this bug. Then... Well,
life happened, and I have not had the time to sit down and write until
today -- A couple of hours before our next meeting. Several other
discussions have happened as well, most notably, the article by
Jonathan Corbet on this issue in LWN².
¹ http://meetbot.debian.net/debian-ctte/2020/debian-ctte.2020-10-21-17.58.html
http://meetbot.debian.net/debian-ctte/2020/debian-ctte.2020-10-21-17.58.log.html
² https://lwn.net/Articles/835599/
# Vendoring: Impedance mismatch with our long-established
# practices/traditions
######
I believe the issue of vendoring to be central to the discussion of
what Debian is, and what its role should be. We are very lucky to have
proponents of both sides of this issue in the Committee, and I'll try
to keep my ideas as centered as possible (of course, disclaimer: I do
hold a position, I really do not like vendoring; we have the luck of
having Elana in the ctte, as she is a developer of the affected
package, and Sean, who is a Debian Policy editor).
Elana started off with a very important and true point:
Elana>
I think this bug was sort of inevitable. Debian policy cruft is
clashing with how people actually build and distribute software
these days. we have an appeal to override a maintainer on the
basis of "policy" and the "Debian way being superior" without any
real technical examination of the merits of "the Debian way"
We are quite far from 1996, yes, and many languages have for a long
time delivered their own packaging systems, "freeing" the users from
the need of installing a myriad of little packages, and "freeing" the
distribution caregivers (and I don't only mean the developers, but
also the ftp-masters) and infrastructure from carrying this myriad of
little packages.
Elana and Sean seem to share the thought that each language ecosystem
could work with somewhat different rules:
Sean>
It might be reasonable to vendor like mad for something written in
Go but not for something written in Haskell, say
Elana>
Debian policy is specifically built around the distribution and
execution of dynamically linked C/C++ applications and
libraries. distros in general were. but modern software above that
base does not necessarily operate under the same assumptions and
it is silly to apply policy that was designed for applications
that are dynamically linked against programs that are statically
linked and are impossible to dynamically link
Simon mentioned that "our identity should be about shipping
high-quality packages, whatever that means", and mentioning that "our
packaging policy is designed for medium-sized packages", refereed back
to the discussions had long time ago over tiny Javascript snippets
packaged as packages, back in the starting days of the nodejs growth.
# DFSG-freeness checking
######
Sean and I shared the concern that excessive vendoring (say, having
tens to hundreds of libraries shipped as part of a single package)
place a very heavy burden on our ftp-masters when checking
DFSG-freeness; coupling this with the rapid change rate that
"new-wave" software development usually has, if adding / dropping
dependencies from already packaged software is usual practice,
DFSG-checks would not just have to be made in NEW, but as a continuous
process. Far from sustainable, and impossible to do by our ftp-master
team practices.
# Security issues
#######
The issue of security support was also mentioned: If many packages
start shipping tens or hundreds of vendored libraries, how can we
ensure security support? This has long been a pride point for Debian
and, in general, for the Linux distributions model. We package
independent libraries, dynamically link them, and security supports
"just happens" for the users. Now, what happens in languages as Go or
Rust, where linking is done statically? They would have to be at least
recompiled when any of their constitutive libraries is updated. But
what if the libraries are vendored-in? Security issues are prone to be
hidden from our sight.
I'm pasting here a bit of the discussion that happened later during
the meeting: having this discussion K8S-specific, Elana mentioned that
"that is a big part of the tension. sometimes, you have an upstream
where the authors are less resourced and responsive than the
downstream. this case is almost certainly the opposite".
At this point, we found some friction as to _what_ we are discussing
on: Is this bug specifically on Kubernetes, which should be taken as a
special case? Or would we try to rule as the Technical Committee on
vendoring in general in the Debian ecosystem? Elana repeated her
assurance that the Kubernetes team is thorough in their
freeness-checking and security practices; I insisted on "not
discussing about K8S, but about a bundling practice to which K8S
subscribes". I do fear that, if K8S is special-cased as an example of
well-written software, we will be standing on the verge of a slippery
slope; Sean stated we could keep the discussion specific to this
package:
Sean>
can't we qualify our response to this bug by saying we're
expressing a view on what's appropriate for this package and do
not intend to settle bigger vendoring questions for Debian?
To what I answered:
Gunnar>
Yes, but no matter what we do, we will be setting some
precedents. So, we have to keep the bigger picture in mind.
Elana mentioned that this issue is typical for Golang packages, but
"does not necessarily translate to e.g. JS packages". Simon mentions
that several properties of Golang's packaging are shared with Rust,
and given the amount of Rust in GNOME, Firefox or Chromium, "if we set
policies that rule out shipping Firefox, or chromium, or GNOME, then
we no longer have a useful distro".
Elana mentions several large Golang projects which share this amount
of vendoring with K8S, and said,
Elana>
disappointingly to me, most of these applications have basically
given up on OS packaging, they either provide their own debs via a
PPA or generally say "do not install using a distro package
manager ever"
# Test-passing assurance
######
Phil brings up an interesting point: Many users want the specific
library versions that were packaged with a given K8S release because,
given the scope of K8S (as a container orchestrator), test coverage
can be fundamental; Elana agrees "unless Debian wants to start running
its own suite of Kubernetes compliance tests". And, if K8S were to be
effectively recompiled every time one of its ~200 dependencies were to
be upgraded, said tests would have to happen on each of the users'
machines - Clearly not ideal, and a long deviation from standard
practice! (and... Well, quite unlikely to be able to perform installs
without, say, network activity - that would break other bits of Policy
for sure!)
There is a course of action that's unlikely to leave everybody happy,
but is worth considering: Phil said:
Phil>
that makes it seem like what we actually need is a decent way of
letting our users install the upstream binaries, rather than
trying to force those packages into Debian as a special case
Sean agreed with him; probably something as K8S due to its nature is
not suited for Debian inclusion (but we'd still have to ensure it's
easy to build and install, not from packages). But the quesion is
where to draw the line:
Elana>
is the future of Debian "here's a glibc platform base" ala
manylinux scope and then everything else is someone else's
problem? do we become some sort of nix-ish thing?
(...)
which is why I think the scope of applications included in Debian
needs to be wider than just a common base with flatpaks or similar
So... It's not like we reached a conclusion, but I do feel the meeting
was interesting and the discussion very much worthy. Yes, this will
surely end up in reinforcing the notion that the Technical Committee
is a slow and bureaucratic beast :-) But... well, I'll stop
apologizing. Only some minutes to go before the meeting, and I want to
give the rest of the Committee the opportunity to check if I didn't
misrepresent them or skip too much of their opinion.
Attachment:
signature.asc
Description: PGP signature