Re: Moving to the FHS: not right now!
Hi,
>>"Wichert" == Wichert Akkerman <- Debian Project Leader <leader@debian.org>> writes:
Wichert> Luckily the constitution provides us with a way to solve this: the
Wichert> Technical Committee can be asked to decide on a strategy which people
Wichert> will have to follow. I hereby ask them to study this and come up with a
Wichert> strategy for moving to the FHS.
A draft proposal follows.
Problem statement:
The FSSTND location for package documentation, /usr/doc, has moved to
the location /usr/share/doc in the FHS. Now that we have formally
decided to adopt the FHS, we need to ensure the transition is as
smooth as we can make it.
Some of the constraints are:
* The transition may take a long time, going by previous
transitions, and not all packages are upgraded anywhere near
simultaneously.
I think that expecting _*all*_ packages to have moved before we
release potato is futile, unless we do not plan on releasing
potato for 18 months or so. We *_cannot_* expect to get FHS
compliance in place by potato.
* We should not break backwards compatibility during the transition
period. This is a quality of implementation issue
During the transition, we need to provide backwards
compatibility, firstly for programs like `dwww', and `dhelp', and
also for our users who have gotten used to looking under a single
dir (`/usr/doc/') for docs (`/usr/doc/`package''). During the
transition, the documentation could be in in two places, and that
is not good.
* There is no elegant way to piece wise move a directory
spanning multiple packages with dpkg. Basically any attempt
to provide 'legacy' symlinks that make /usr/doc/foo ==
/usr/share/doc/foo will either break upgrading, downgrading
or both, unless special care is taken
dpkg has no ability to resolve multiple paths to the same files in
it's database, so it sees /usr/share/doc/pkg/bar and
/usr/doc/pkg/bar as -different non overwriting- files and will end
up doing the wrong thing when it comes time to erase the old
versions files. Basically it will do:
create /usr/share/doc/pkg/bar
rm /usr/doc/pkg/bar
Which is perfectly sensible -IF- they did not happen to be the same
directory!!
What makes this directory move different from the other changes
mandated by the FHS is the human element: humans are used to
looking for documentation under one directory, namely,
/usr/doc/<package>, and having the documentation in two directories
is breaking that expectation.
What follows is an updated solution to this issue (the so called
symlink solution).
Several other solutions have also been proposed to address this
situation. I shall present a (hopefully unbiased) selection of
solutions, with arguments for and against each solution.
Proposed solution
--------------------
I propose that there be a symlink from /usr/doc/package =>
/usr/share/doc/package, managed by the package itself. Since there is
some concern that the packaging system does not deal well with
replacing a directory with a symbolic link, it is suggested that the
link be manipulated in the maintainer scripts postinst and postrm.
We create the postinst, prerm now, installing the symlink, Once
the move is over, we just have a prerm script removing the
symlink. another release, (potato+2), we stop bothering, since we
would have handled the most common case (and provide an base-files
postinst script removing the symlinks for upgrades at that point).
Yes, this is a long period, but not forever (which was one of
the major objections to this proposal)
So potato packages should have: (effectively)
postinst install upgrade:
if [ -d /usr/doc ]; then
if [ ! -e /usr/doc/$package -a -d /usr/share/doc/$package ]; then
ln -s /usr/share/doc/$package /usr/doc/$package
fi
fi
prerm remove upgrade:
if [ -d /usr/doc ]; then
if [ -L /usr/doc/$package ]; then
rm -f /usr/doc/$package
fi
fi
The prerm removes the symlink, and the postinst reinstates
it as necessary. The only problem is redundant symlinks when you're
doing partial upgrades to woody+1 (or later), which no longer need
any symlinks, from potato/woody. But they still all point at the
correct places. This is handled by a base-files postinst removing the
junk symlinks (details below).
This is how it works:
1. Policy 3.X mandates that packages that move the docs to
/usr/share/doc must provide a compatibility symlink in /usr/doc.
This shall allow packages to incrementally move to policy 3.X,
while providing compatibility with older systems.
(/usr/doc/package symlink is handled by package)
Thus, potato ships with a full /usr/doc/ (some of which is symlinks),
and a partial /usr/share/doc.
2. Post potato, we continue the transition, with the symlinks in
place. Before freeze, we file important bugs against any
package that has not been moved over (in one and a half
release periods, we may be actually able to accomplish this),
with NMU-fests to bring over the others.
Thus, potato+1 (woody) ships with a full /usr/share/doc, and a
/usr/doc full of symlinks.
3. At a later date, another policy (say, 4.X) shall ask for
packages to no longer provide the link (and possibly remove
links from /usr/doc). We can also provide a script (possibly
in base-files postinst) that rm's symlinks from within
/usr/doc. woody+1 may ship with such a script. (there was a
proposal as well that potato+2 (woody+1) ships with just the
prerms, and not the base file script, and potato+3 ships
with the base-file script, but I am not sure this long a
reversion period is required).
No dependency on this base-files shall be required, since they shall
work with or without a symlink in /usr/doc (and /usr/share/doc shall
be fully populated by then).
Thus, partial upgrades to potato and woody have a complete /usr/doc,
and full upgrades to woody have a complete /usr/share, and symlinks
throughout /usr/doc. Partial upgrades to anything beyond woody might
have old files left in /usr/doc, but they'll get moved when whoever
finally gets around to run an apt-get dist-upgrade.
I understand that the forest of symlinks is ugly, but it is
technically sound, it maintains backwards compatibility, it allows
incremental compliance to FHS, and caters to a hybrid system. I submit
that aesthetics takes a back seat to functionality.
I fail to see what the big problem with fairly obvious symlinks
existing on a system that's halfway between two significantly
different file system standards. I also can't see admins being
particularly bothered by the existence of such symlinks on such
systems.
OTOH, it *is* inconvenient to have documentation scattered
inconsistently between /usr/doc and /usr/share/doc. We've even had
user complaints on this list about it.
To the objection that it shall cause package to be modified twice, I
say that
* This is a complex transition, and may require this to meet the
constraints of the situation
* One modification of the packages is required anyway for compliance
with the FHS.
* The second modification, namely, the removal of the symbolic
link, shall be well into the future, and added to a future policy
change. It is conceivable that the packages may need to change
for policy updates in any case.
The symlink proposal is far from perfect --- adding postinst's and
prerm's to every package with the same half dozen lines of code is
really pretty lame. The *proper*, forward thinking solution would be
to add a feature to dpkg to run specific items of code before and
after each dpkg run.
But modifying dpkg is infeasible, and we've agreed to, among other
things, keep the needs of our users at the forefront of our
minds. And from a user's perspective, something that keeps the
system tidy in the normal case, and works *now*, is much better than
idealistic fantasies like a working dpkg.
Overview of other solutions proposed.
----------------------------------------------------------------------
* do nothing special
- means the admin, and all automated tools have to look in both places
or miss documentation
This essentially ignores the issues with the split locations
for the documentation. I think we can live with the problem, but as a
quality of implementation issue, it would be nice if we had a smoother
transition.
----------------------------------------------------------------------
* making a script that works out which of /usr/doc /usr/share/doc to use
- doesn't work for browsing by hand
- needs to be written/accessible from lots of different languages
- is significantly different to any other system on the planet
+ makes it easy to "support" /usr/local/doc or similar ("support"
in scare-quotes because I'm not really sure of the value of
such support in this case. YMMV)
[See Appendix A on details and commentary]
----------------------------------------------------------------------
* symlinks managed by postinst/prerm
- requires lots of packages to add postinsts/prerms for potato
and woody, and then to get rid of them for woody+1
- may leave crufty symlinks about on systems where (a) the admin
doesn't fix it (b) hasn't had the base packages upgraded to woody+1
+ allows incremental upgrades
+ allows for downgrades
+ does not involves a global change day
+ works with dpkg failings ;-)
[This was proposed and shot down because 4 people objected;]
[See Appendix B for details].
I think the symlinks proposal is the best one so far ---
it addresses the issue correctly, and it's drawbacks are both aesthetic
(crufty symlinks that don't damage the system; and extra postinsts)
and temporal (upgrading to woody+1 gets rid of both the symlinks and
the extra postinsts).
----------------------------------------------------------------------
* cronjob that adds/removes symlinks in cron.daily
- downgrading a package that uses /usr/share/doc to a prior version
that uses /usr/doc will cause dpkg to move files from
/usr/share/doc/foo to /usr/share/doc/foo via a symlink, which is
believed to be dangerous. (*)[proof in appendix C]
- may leave dangling symlinks / may not have symlinks for up to 24 hours
A cronjob is a bad idea because the links will persist for dpkg
operations and basically cause upgrades/downgrades to fail. This is
not a working solution.
----------------------------------------------------------------------
* mv /usr/doc/* /usr/share/doc
* Modify dpkg's internal databases (mainly the .list files in the
directory /var/lib/dpkg/info) so that they are in sync with the
previous changes.
+ would make the system to be /usr/share/doc-compliant.
+ would avoid to touch every maintainer script in every package.
+ would support upgrades and downgrades.
- fiddles with dpkg internals, against all warnings in the docs
- pointless unless the package is changed, or the next upgrade
creates the /usr/doc dir all over again
This isn't trivial, because you cannot be sure that /usr/doc and
/usr/share/doc are located at the same file system.
And don't miss the (few) packages which already moved to
/usr/share/doc (where some of them left back a .dhelp file in
/usr/doc/<package>). Also, doing it the way mv(1) does means
failures halfway through leave you with files in /usr/doc/foo and
/usr/share/doc/foo, which could be hard to deal with correctly.
If one uses tar/cpio to do the move, one should realize that if you
*do* have /usr and /usr/share on the same partition, you need to have
as much free space as /usr/doc takes up. Unless you do the move on
a directory by directory basis.
Also, fiddling with the dpkg database - bugs in the script can cause
catastrophic system failure by messing up the dpkg database (*) - must
be done by the admin by hand, rather than automatically as part of the
upgrade to potato/woody - script is difficult to get right on common
configurations
Messing with dpkg's database is *not* something to be done lightly.
Getting knowledgeable comment from the author's seems the *least* thing
that should be done.
This is not a working solution.
----------------------------------------------------------------------
* add support to dpkg for pre and post processing on a global basis
- requires dpkg modifications (*)
+ allows update-menus and similar programs to be implemented much more
cleanly.
This is not a working solution, given the timetable.
----------------------------------------------------------------------
* Stick with /usr/doc until potato is released, then begin a massive
migration, which may or may not involve symlinks.
- some people have already moved and may not want to move back.
- would need a policy amendment partially backing out of the FHS
move.
- requires all changes to be made in a single release cycle*
(or we're back in the same position we're in at the moment)
- delays the move to FHS compliance.
+ gives us a little more time to decide if we want crufty
symlinks, and if so, what's the best way to handle it.
+ no surprises to the user.
+ no changes to most packages till after potato's release.
----------------------------------------------------------------------
I consider the points marked with a (*) to be completely unacceptable.
By "completely unacceptable", I mean that it's unacceptable to just say
"we should do this". If you *can* do it (ie, write the program/make the
modifications), and then can show why the original fears were unfounded
(look, iwj even says this is okay!), then, naturally, there's no problem.
======================================================================
======================================================================
Appendix A
docdir() {
dirname $(grep "/doc/$1/copyright\$" /var/lib/dpkg/info/$1.list")
cddoc() {
cd $(docdir $1)
Now we're back to a single probe. We're just probing the
database, not the filesystem.
[This has to be in every environment, and needs to cater to all kinds
of shells, and shell scripts, and perl scripts, and ...]
> So we make it a script.
It doesn't just have to be in every user's environment, it has to be (or
at least, ought to be) easily available for arbitrary scripts/programs
to be able to access documentation. It thus needs to work conveniently
with csh, rc, perl, python and C.
It also adds another learning curve to Debian for, what appears to me
to be, very little benefit. Instead of just cd'ing to /usr/src/linux
and running make, we're meant to use kernel-package. Well, that's fine
--- kernel-package makes it a lot easier to deal with upgrading from
one kernel version to another keeping modules in sync and not leaving
cruft lying about on your disk. Instead of just cd'ing to /usr/doc and
poking around with your favourite utilities, you now have to either know
the name of the program in advance and use some special functions which
you've likely never heard of before.
Still does not answer less /usr/doc/<package>[TAB]
> Very true. There are always tradeoffs -- in this case, we gain
> flexibility.
To what end? Is /usr/local/doc particularly common anyway? We certainly
don't have any intention of keeping /usr/doc around, so having more than
one doc directory doesn't seem to help us.
Needless flexibility breeds bugs.
=================================================================================
Appendix B
Details of dpkg lossage with symlinks.
----------------------------------------------------------------------
All right dammit, here we go... built a package crap 1.0-1, here is the
listing:
/.
/usr
/usr/bin
/usr/sbin
/usr/lib
/usr/lib/crap
/usr/lib/crap/olddir
/usr/lib/crap/olddir/file
/usr/doc
/usr/doc/crap
/usr/doc/crap/changelog.Debian.gz
Now from /usr/lib/crap:
drwxr-xr-x 2 root root 1024 Aug 3 02:15 olddir/
-rwxr-xr-x 1 root root 633 Aug 3 02:14 olddir/file*
And then after I symlink it:
drwxr-xr-x 2 root root 1024 Aug 3 02:15 newdir/
-rwxr-xr-x 1 root root 633 Aug 3 02:14 newdir/file*
lrwxrwxrwx 1 root root 7 Aug 3 02:19 olddir -> newdir//
Install crap 1.0-2 ...
/.
/usr
/usr/bin
/usr/sbin
/usr/lib
/usr/lib/crap
/usr/lib/crap/newdir
/usr/lib/crap/newdir/file
/usr/doc
/usr/doc/crap
/usr/doc/crap/changelog.Debian.gz
root@icarus2:/usr/lib/crap# ls -lR
..:
total 1
drwxr-xr-x 2 root root 1024 Aug 3 02:21 newdir/
newdir:
total 0
Do you believe us yet? What more proof do you possibly need?
> (Note that I am explicitly avoiding actually installing or deinstalling an
> actual symlink which dpkg thinks is a directory ... with my proposal this
> can, in fact, happen, but only if users deinstall all packages that refer
> to the /usr/doc directory which is rather unlikely.)
Does not matter. dpkg breaks. It has been now demonstrated and proven.
> > Your script also has a cow if /usr/doc isn't -> share/doc, which is bad
> > because it may be necessary to use some symlink magic at some point. Not
> > that it isn't a moot point unless you fix dpkg first.
>
> This was intentional ... and in fact the script will merely warn you if
> this is the case. I was merely trying to KISS since this is a rather
> critical script.
>
> More reactions welcome !
It exits with 1, that's an error condition.
> I am happy to tell you that we agree completely on the behaviour of dpkg on
> your example. But you are ignoring a very important aspect of my proposal:
> THIS ONLY HAPPENS FOR DIRECTORIES INTERNAL TO PACKAGES. It happens because
> olddir is actually REMOVED by the deinstallation.
This doesn't seem to be the case.
* Create three packages:
test1 version 1.0 mimicing your average /usr/doc-using package
test1 version 2.0 mimicing your average /usr/share/doc-using package
test3 version 1.0 mimicing base-files
test1 1.0 has a file /my_usr/doc/test1/copyright,
and depends on test3
test1 2.0 has a file /my_usr/share/doc/test1/copyright,
and depends on test3
test3 1.0 has a file /my_usr/doc/copyright/GPL,
and a file /my_usr/share/doc/test3/copyright
* dpkg --install test3_1.0_all.deb
* mv /my_usr/doc/copyright /my_usr/share/doc/
* rmdir /my_usr/doc
* ln -s /my_usr/share/doc /my_usr/doc
* dpkg --install test1_1.0_all.deb
* dpkg --install test1_2.0_all.deb
* ls -l /my_usr/doc/test1 -> empty
* ls -l /my_usr/share/doc/test1 -> empty
* dpkg -L test1 | grep my_usr/share/doc -> not empty
The packages are available as:
http://www.debian.org/~ajt/test1_1.0_all.deb
http://www.debian.org/~ajt/test1_2.0_all.deb
http://www.debian.org/~ajt/test3_1.0_all.deb
Possibly I'm just misunderstanding what you're suggesting should be done
though. Can you give a sequence of commands that does whatever you're
suggesting, and still has those three packages survive unscathed?
----------------------------------------------------------------------
On Tue, Aug 03, 1999 at 09:02:53PM -0400, Michael Stone wrote:
> (And thus useless to me.) I don't argue that dpkg has some problems with
> symlinks if a package changes the path by which it references one of its
> files. It does not have problems (as far as I have found) if a package
> consistently uses the same path when referencing the file, whether or
> not that path passes through a symlink. That's the case I'm interested
> in and which I've been testing.
*OH*. No, that's completely correct.
> No, what I wrote works fine. The examples were irrelavent. You
> apparantly ignored the part where I said that all packages should put
> their stuff into /usr/doc without regard for whether it's a symlink or
> anything else.
This means packages will have to use /usr/doc instead of /usr/share/doc
until dpkg is fixed. "until dpkg is fixed" could be forever.
This seems like it might also require pre-dependencies on base-files for
every package (so that they don't accidently create /usr/doc as an
actual directory instead of as a symlink).
Hmmm. This also seems like it might require pre-dependencies from every
package against the new version of dpkg that handles following symlinks
correctly.
> 1. use both /usr/doc and /usr/share/doc; this upsets the partial
> upgrade people, and worries the UI people.
"upsets" and "worries" aren't really very precise. It results in a
"needlessly" inconsistent user interface for accessing documentation
for potato and partial upgrades from <=potato to >potato.
> 2. use both /usr/doc and /usr/share/doc but provide symlinks from
> each package in one to each package in the other by means of
> preinst/rm scripts; this upsets the anti-bloat people and horrifies
> the elegance people :)
It creates postinst and prerms where there never used to be any, but lets
them go away after woody.
> 3. use both /usr/doc and /usr/share/doc but provide symlinks from
> each package in one to each package in the other by means of a cron
> job or somesuch; this upsets the partial upgrade people, horrifies
> the elegance people, and terrifies the people who don't want cron
> jobs automatically deleting things on their systems.
It means documentation can be lost when doing a downgrade from a
/usr/share-using package to a prior, /usr/doc-using, version of that
package.
It means dangling symlinks could exist for up to 24 hours (or however
often the job runs). It means symlinks might not exist for the same
amount of time.
> 4. use both /usr/doc and /usr/share/doc but provide symlinks from
> each package in one to each package in the other by means of an
> optional script run manually by the admin; this upsets the partial
> upgrade people, worries the UI people, and horrifies the elegance
> people.
Same problems as the cron job, and not automatic.
> Don't explicitly use /usr/share/doc yet, and we can rig up a symlink
> to effectively use /usr/share/doc until we come up with a better
> solution; this upsets the policy-says-I-can-therefor-I-must people
> and dismays the people who have already converted their packages
Requires changes to dpkg, which don't have a particular history of being
made. Possibly requires lots of pre-dependencies that will last for ever.
----------------------------------------------------------------------
----------------------------------------------------------------------
Appendix C
Has no one seriously considered the mess that will happen if you
try to follow this path (namely, making each package manage the
transition by itself)? Think about all the typos (like "[-L foo]")
that people are going to make, the number of link-handling scripts
that bomb out or do the wrong thing, the number of unwanted symlinks
that will be lying around once the dust settles, and the crap that
will live in the maintainer scripts forever (well, a long time). Think
about how few maintainers test their scripts exhaustively (for
idempotency etc.) (when doing upgrade, new install, failed-upgrade,
configure, remove, purge, etc.).
I will oppose the proposal unless it contains code fragments that a)
have been thoroughly tested, b) have been shown to be the only real
solution, and c) will be mandated.
One question, does the FHS permit "/usr/share/doc -> ../doc" ? If so,
there is a very simple solution (forbid the use of /usr/share/doc and
have base-files contain this symlink; this does not address mount and
partition issues).
A better solution (than the proposed one) would be one that has all
the code in only one place and the opportunity to fail at only one
time. Alas, I do not have such a thing.
Unfortunately, various people have pre-empted the policy discussion
and have started using /usr/share/doc already.
======================================================================
--
A fool's brain digests philosophy into folly, science into
superstition, and art into pedantry. Hence University
education. Shaw
Manoj Srivastava <srivasta@debian.org> <http://www.debian.org/%7Esrivasta/>
Key C7261095 fingerprint = CB D9 F4 12 68 07 E4 05 CC 2D 27 12 1D F5 E8 6E
Reply to: