(subtitled: a tech proposal to make Manoj happy ;)
Hello once more, world,
This proposal presents a relatively minor change to the archive, but a
fairly serious change to the way we actually make use of the archive
as we develop a release.
The essence of this proposal is that we maintain a distribution somewhat
like frozen throughout the release cycle rather than just at the end
of it. This "frozen-like" distribution will only consist of packages
that are basically ready to be released, but will happily accept
improvements in functionality as well as bugfixes. The main focus of
this new distribution is to keep a track of what we'd be releasing if
we chose to release not just Real Soon Now, but *Right* Now.
This is a long proposal. My apologies, but I think the details are
important enough to warrant the excessive number of words.
(please note, this proposal attempts to focus on the technical
issues. However, most of the justification for the changes proposed
are philosophical and pyschological, so if you're not already familiar
with why a `prerelease' distribution would be useful, this proposal
might make more sense if you read the `Philosophy' section first)
The dists/ Hierarchy
====================
0. Introduction
---------------
At any one time, there are two distributions of Debian that are
relevant for the developers: the current release, that most of our
users should be running; and the next release.
Further, not every package is to be included in either distribution
immediately, and thus there are two incomplete distributions that are
also maintained, one that houses uninstalled updates to the current
distribution, and the other that houses uninstalled updates to the
next release.
1. Terms
--------
For some degree of clarity, some terms are used in a more specific
manner in this proposal. This proposal doesn't recommend that these
are appropriate formal meanings for these terms within the Debian
project, just that they are convenient for this proposal.
release -- A codenamed and numbered set of distributions of
Debian GNU/Linux.. Debian 1.3 (bo/i386), Debian 2.0
(hamm/i386,m68k), and Debian 2.1 (slink/i386,m68k)
are current examples of releases.
category -- One of 'main', 'contrib', 'non-free' or 'non-us'.
section -- eg 'base', 'admin', 'devel', 'libs', 'misc'.
stable -- The current release of Debian GNU/Linux.
prerelease -- The next release of Debian GNU/Linux.
stable-updates -- A collection of packages that may provide added
functionality to a system based on one of the stable
releases.
unstable -- A collection of packages that may provide added
functionality to a system based on the prerelease.
release-group -- The group of developer's who are most closely
involved with making the release. This is intended
to include people such as Brian White (the release
manager for hamm, who had the final say what made
it past the freeze and what didn't, etc), Richard
Braakman and Wichert Akkerman (who maintained the
"Hamm Bugs Stamp Out List"), and so forth.
stable-manager -- The guy who takes care of the horses.
Since we don't have any horses, in his spare time
he can also fill the release-group's role with
respect to the current stable release.
2. Layout
---------
Each release of Debian GNU/Linux should be identified by a static
codename, such as "hamm" or "slink". Each release should be
represented by a directory named by the codename under the dists
subdirectory, eg:
/pub/debian/dists/
bo/
hamm/
slink/
There should be symlinks "stable" and "prerelease", and they should
point to the current and development releases respectively, eg:
/pub/debian/dists/
stable -> hamm
prerelease -> slink
For the stable release, the should be an "-updates" directory, to
store updates to that release until they can be included in the
release itself. The directory itself should be named after the
codename of the release, but there should also be a symlink
"stable-updates" for easy identification, eg:
/pub/debian/dists/
bo-updates/
hamm-updates/
stable-updates -> hamm-updates
There should also be an "unstable" directory, for updates to the
prerelease release, and packages that are still in development and not
yet suitable for official release. (The intended use of this is for
packages such as gettext used to be -- that are not yet stable enough
for public release, but are useful enough to be made easily available
for testing purposes)
/pub/debian/dists/
unstable/
Each release should be arranged as follows:
/pub/debian/dists/slink/
main/
contrib/
non-free/
non-us/
disks/
cd-images/
Only the first four directories should be included in the `-updates'
releases. In unstable, the disks directory should also be included,
for development of as yet unreleased architectures. cd-images should
only be provided for codenamed releases.
The four categories should have a structure as follows:
/pub/debian/dists/slink/main/
binary-all/
binary-*/ -- one for each included architecture
source/
These should be subdivided further by section, and each section should
contain symlinks for each package to the appropriate package in the
package pool.
The various binary-* directories should contain in their top level,
the files:
/pub/debian/dists/slink/main/binary-*/
Packages
Packages.gz
Release
The source subdirectory should contain links to the source of each
package present in any of the binary-* subdirectories. This may mean
that one package has multiple versions of its source in the same
directory if different architectures are not synchronised.
The disks subdirectory should contain symlinks to the current
architecture specific disks directories in the package-pool, for each
architecture the release supports, eg:
/pub/debian/dists/slink/disks/
disks-i386 -> ../../../package-pool/disks-i386/2.0.10_...
disks-m68k -> ../../../package-pool/disks-m68k/2.1.13_...
The cd-images subdirectory should contain the Official CD images for
the release. That is, ISO-9660 images containing the binaries for each
supported architecture, and images containing the sources for each
supported architecture. These should be accompanied by detached GnuPG
and PGP signatures of the images made by the release manager and the
PGP or GnuPG signature of the developer who created the images.
In summary, the dists directory should contain:
/pub/debian/dists/
bo/
bo-updates/
hamm/
stable -> hamm
hamm-updates/
stable-updates -> hamm-updates
slink/
main/
binary-all/
binary-*/
Packages
Packages.gz
Release
section/
foobar.deb -> [package-pool]
source/
contrib/
non-free/
non-us/
disks/
disks-i386 -> [package-pool]
disks-m68k -> [package-pool]
cd-images/
[ISO images, and signatures]
prerelease -> slink
unstable/
3. Procedures
-------------
3.1 Uploading a package
-----------------------
...in order to improve on a package in prerelease
...in order to add a package to prerelease
In order to generally update a package included in prerelease the
maintainer must do two things:
* Upload the package into unstable for initial testing.
* After the package has received sufficient testing
(about a fortnight in unstable), and no release-
critical bugs have been found, the maintainer should
present it to the release group as a release
candidate.
NB: automatic reminders should be sent to the
maintainers of packages that have sat in unstable
for over a fortnight with no release-critical bugs,
but have not been submitted as a release candidate.
If a package is accepted as a release candidate by the release group,
it should be added to prerelease as soon as all its dependencies can
be satisfied (That is, a package, even though it is otherwise
perfectly ready for prerelease will not be moved until either all the
other packages it depends upon are in prerelease, or can be moved with
it).
[it is recommended that a "staging pool" be used to manage release
candidates. This should be maintained on master as an actual
distribution so that automated tools like pkgorder can be used to
ensure dependencies are met, etc]
...in order to fix a release-critical bug in prerelease
The only time when the above, overly cautious, procedure is not
workable, is when a package needs to be rushed into prerelease. This
is exactly when a release-critical bug is being fixed (whether there
be a bug report on it or no).
The correct procedure to follow in this case, should be that the
package is uploaded to "prerelease" with urgency=medium or high.
At this point the release group should be automatically informed that
the update is available, and presuming that it hasn't been uploaded to
prerelease by mistake, install it into prerlease ASAP.
If the release-critical fixes involved security fixes, then once the
package is installed, a mail should be sent to the -security-announce
list describing the problem, and similar information should be added
to the web site.
...in order to improve on a package in stable (new upstram
release &c)
...in order to fix a release-critical bug in stable
These two events should be similar to the corresponding changes to
prerelease, with "prerelease" changed to "stable", "unstable" changed
to "stable-updates", and the "release group" changed to the
"stable-manager".
More vigorous requirements than "a fortnight's testing" should
probably be applied before a package is moved to stable, but this is
left to the descretion of the stable manager.
...in order to test an alpha/beta package
The final case -- where a package needs to be tested, but isn't felt
ready for distribution yet (in particular if it crashes all the time,
introduces security holes that haven't yet been fixed, or similar) it
should be uploaded to unstable, but simply not presented to the
release group as a release candidate (The automatic reminder sent by
the release group should thus be ignored).
This is expected to be the usual place of residence of upstream
packages the upstream author does not wish publically distributed.
Once the package is ready for release, the release group should be
notified that it is now a release candidate, and the procedures for
moving a package to prerelease apply.
3.2 Releasing a release
-----------------------
In order to make a release of Debian GNU/Linux, the following should
occur:
* prerelease is frozen.
This involves no longer moving release candidates into
prerelease. This is in order to allow one or two fortnight's
thorough testing -- note that there should be no (or few)
release-critical bugs in prerelease to start with, so only
*very* minor churn is to be expected.
This is the beta release.
* prerelease is released.
This involves changing the stable symlink to point to the
former prerelease ("foo"), creating a "foo-updates"
directory, and pointing the stable-updates symlink at it,
and creating a new prerelease ("bar"), and pointing the
prerelease symlink at it,
The foo-updates directory should be initially empty.
The bar directory should be an exact copy of the foo
directory, with only the codename changed -- that is, we
will build the next release starting with the current one.
3.3 Removing a package
----------------------
...because it's buggy
If a package in prerelease or stable has release-critical bugs that
cannot (or at least have not, and will not) be fixed (and it's not
dpkg), that package should be removed -- either to be replaced with an
earlier version, or simply dropped.
This should be done by:
* notifying the maintainer by direct email
* removing the symlink from stable/prerelease
* if that package (possibly a later revision) is not in
stable-updates/unstable (as applicable) a symlink should
be made from that area.
* making a post to the appropriate groups as to why the
package was dropped, and where it may now be found.
...because it can't be distributed
4. Authority
------------
The archive-maintainers are responsible for incroporating uploads into
the archive. As described above, this involves adding that upload into
the package-pool, and providing a symlink from stable-updates/unstable
to the upload.
This should all be automated as much as possible, and where automation
is unreasonable, should still be done within 24 or 48 hours of upload.
The release group are responsible for moving packages from unstable to
prerelease, and as such need to:
* send reminders to the maintainers of packages that are
apparently in release-worthy condition
* verify that release-candidates are indeed release-worthy
(which should be basically "the maintainer says so, and
no one else has filed release-critical bugs" and hence
highly automatable)
* send reminders to the maintainers of packages in
prerelease that have release-critical bugs, that they
need to get fixed *now* or they'll be removed. (given a
week's grace?)
This should probably also involve maintaining a `Bug
Stamp Out List' for packages in unstable and prerelease.
* remove packages from prerelease and move them (back)
into unstable.
* determine an appropriate point at which to freeze
prerelease and go into beta testing.
* determine the appropriate point at which to stop beta
testing, and release.
5. What does it buy us?
-----------------------
There are two main improvements this proposal should bring about:
* First, releases should be *very* easy, yet remain at the high
standard Debian expects.
* Second, the prerelease distribution will form a far more
reliable base for people who want up-to-the-minute packages,
but can't afford the unreliabilities that following either
the current or the proposed unstable entails.
6. Philosophy
-------------
The philosophy behind maintaining a prerelease distribution, that is
forced to stay reasonably bug free in spite of constant updates, is
the zero defects policy Steve Maguire claims Microsoft follows in
his book _Debugging the Development Process_ (Microsoft Press, ISBN:
1-55615-650-2).
The theory is that developers should be forced to consider and fix
bugs throughout the entire development cycle, rather than simply at
the end. This is to avoid the scenario of gradually accumulating a
bunch of bugs that need to be worked on just when you think you've
finished the project.
Now obviously, there's no point trying to force anyone to do anything
with Debian, however the one thing we *can* do, is to make it clear
sooner which packages are going to fall under the `if there are
release-critical bugs, it doesn't get released' hammer.
Additionally, the classification of Debian users into the following
three groups was taken into consideration:
* those who require a robust, bug-free system and do not wish
to be bothered by constant change (due to the cost or
difficulty of obtaining updates, the difficulty of verifying
large upgrades still do what you want on mission critical
systems, or otherwise). Those who want security updates made
available, but would rather keep the wholesale upgrades down
to a couple of times a year at most.
* those who want to keep up with the latest packages, but
don't want to run into the more severe bugs that can come up
with actual development releases.
* those who want to run the absolute latest of everything, and
are willing to put up with some, possibly severe bugs, to
get it.
This doesn't consider people who want to install something then keep
getting security updates for the next five years, or people who want
the absolutely latest versions of everything, but no bugs whatsoever,
for what are probably obvious reasons.
7. Considerations
-----------------
7.1 Intrusiveness of Automated Reminders
----------------------------------------
One concern might be that the automated reminders would become unduly
annoying for alpha packages intended to be left in unstable.
The way these should behave, however, should minimise such
intrusiveness fairly successfully. In particular, if updates are
uploaded within a fortnight, then no reminder should be sent
(although, if the maintainer wishes one of the interim releases to be
placed in prerelease, then the maintainer can explicitly nominate that
version as a release candidate without having received a reminder).
If a package nominally under development isn't being updated, and
doesn't have any bugs, however, the occassional (fortnightly or
monthly) nag doesn't seem entirely inappropriate.
7.2 Foo Must be Removed!!
-------------------------
If a release-critical bug is found in a package that really can't be
removed, the release group should, of course, use there discretion in
acting on the above. Where possible it is probably best to revert back
to an older version of the package, but where it's not, living with
the bug may be more reasonable than harming the distribution.
7.3 Submitting a Package as a Release Candidate
-----------------------------------------------
In the usual case, the maintainer of a package should be the only one
submitting a package as a release candidate. For packages that have
been orphaned (Maintainer: set to debian-qa), this means anyone, and
for packages maintained by multiple developers, this means either of
those developers.
However, for non-maintainer uploads, and cases where the maintainer is
unavailable, any developer may submit a package as a release
candidate. In this case, the package maintainer should be notified of
this happening, and the release group should give the maintainer a
reasonable amount of time to object, if necessary (24 hours? a week?).
Note that this does not apply to release-critical updates, which are
submitted straight to prerelease, rather than going through the
staging area.
8. Acknowledgements
-------------------
The grouping of users presented in the Philosophy section is due to
Bdale Garbee.
The details and name of the "staging area" are due to Manoj Srivasta.
The term "category" for the main/contrib/non-free/non-us
`distributions', is from Jason Gunthorpe's "Release" files.
The name "prerelease" is due to Joseph Carter.
Most of the ideas in the proposal are due to discussions with:
Bill Mitchell <debian@pny-fmail.webquest.com>
Bdale Garbee <bdale@gag.com>
Brian White <bcwhite@debian.org>
Craig Sanders <cas@taz.net.au>
Dale Scheetz <dwarf@polaris.net>
David Engel <david@ods.com>
Guy Maor <maor@ece.utexas.edu>
Ian Jackson <ian@chiark.greenend.org.uk>
Klee Dienes <klee@alum.mit.edu>
Manoj Srivastava <srivasta@datasync.com>
Philip Hands <phil@hands.com>
"Rev. Joseph Carter" <knghtbrd@earthlink.net>
Richard Braakman <dark@debian.org>
Raul Miller <rdm@test.legislate.com>
Errors, ommissions and any lack of clarity or forethought in the above
are of course mine.
Awaiting your comments.
Cheers,
aj
--
Anthony Towns <aj@humbug.org.au> <http://azure.humbug.org.au/~aj/>
I don't speak for anyone save myself. PGP encrypted mail preferred.
Remember to breathe.
Attachment:
pgpMgkzWq9x8N.pgp
Description: PGP signature