Re: itp: static bins / resolving static debian issues
Hi Marcus,
On Tue, Aug 24, 1999 at 01:39:44PM +0200, Marcus Brinkmann wrote:
> You are doing a dangerous thing here. You try to apply mathematical concepts
> to a matter of that is not of mathematical nature. You can't proof that the
> recovery proposals you propose are needed and should be included by default
> on every system. You can merely try to convince many people that they will
> help many people and try to convince the rest of us that they don't harm.
Yes. I don't think it was mathematical, I wanted to put it into a
formal structure only because several people had begun to propagate
the idea that there were no arguments, only opinions, behind what
I was saying. I didn't want to let that notion persist.
> * That we can improve the overall recovery possibilities does not mean that
> we should do so. We should only do so if the costs are low and the benefit
> is high.
Yes, I agree with this. If it can be shown that the costs are high, or
that the benefits are low, then I will resign. This was in my propositions.
> A default static bin package would increase the cost for everyone using a
> default system, and benefit only the few who know about it, know how to use
> it and want to use it instead other recovery measures. So it seems that
> the accumulated costs are too high and the benefit too low.
If you could optionally choose to remove it, but if it were there by
default, then those rare people who run such ancient hardware could do
a little bit of extra work and not have it.
In fact, I would be happy if it prompted you whether you wanted to have
it or not, with an explanation, and the default answer was "Yes".
People with obsolete hardware could answer "No".
> * That you can improve the overall recovery possibility for your systems by
> using static binaries does not mean that we should make it the default for
> all Debian systems.
> Nobody has denied that static binaries can help under some circumstances.
> But it seems that the people who would use them are few and they would
> only use them infrequently, so it is not a good way to improve the overall
> stability on all standard Debian systems.
See above.
> * Words are cheap. People who want to change the world do something about
> it.
> I suggest you make a rcovery package as you would like to see it. Priority
> "extra", section "admin".
> If the package is so useful how you claim, people will use it and it can
> climb up in the priority ladder. If it is really small and great and useful,
> it can become the default later, when it is finished and tested.
> There is no way Debian will promise you that it will be included by
> default on all standard Debian systems. We ca only evaluate existing
> software, not vapourware. Details maybe critical.
> "Put up or shut up."
There already is a static recovery package, sash. I am asking for it to be
raised in priority. Maybe sash doesn't have every single thing I want,
but it does provide a fair bit.
Let's start with that.
A .deb with some interesting modifications to sash (for use as root's
shell in conjunction with bash) were previously posted.
> > Proposition #1-- There are kinds of servers for which downtime is
> > considered unacceptable
>
> Debian is a general purpose distrbution, not a high end server distribution.
> You are free to base your distribution on Debian and make the default
> whatever you want.
I'm not arguing that it isn't general; I am arguing that there are kinds
of servers for which downtime is considered unacceptable. I am suggesting
that Debian should support these, by default, for the reasons you mentioned
above: I will show that the cost is low, and the benefit is high.
> > Proposition #3-- Debian should, by default, support servers where
> > downtime is unacceptable or difficult.
>
> This is wrong. It should support them, but NOT by default but by installing
> additional packages.
Why? Please give reasons.
> > Proof: Much of this portion is opinion; but I challenge you to try
> > adding a policy statement to the contrary.
>
> The contrary is NOT that Debian should not support such servers at all. The
> contrary is that it should not support them by DEFAULT. This IS already
> written in policy, read the definitions of the "required", "important" and
> "standard" priorities in section 2.2.
I know what the contrary is. Take your definition of the contrary, and
do what I challenged: have it added to the policy. Something like this:
Debian should not by DEFAULT be useful as a server which requires
high uptime, or which is administered remotely.
I don't think many would support that, but you are advocating that
this should be Debian policy when you insist that it not be the
default, no matter how negligible the cost.
> > I believe that
> > most Debian users (maybe not you) think this is important,
> > if only for the sake of Debian's reputation as a server OS.
>
> Believe is not strong enough. Your claim is null and void.
This goes with the text quoted above, not separate. It's the hypothesis
that is to be tested out by you attempting to make your view policy.
> > Proposition #4-- Failure of the C library can occur under Debian
>
> Proposition #4b-- Failure of the static recovery tools can occur under Debian
>
> So what do you propose to do about it? Add another layer of recovery tools?
I don't propose to do anything about it in proposition #4, I only establish
that it does in fact happen. This is apparently controversial, as several
people insisted it would never happen in a stable release.
You accept that it can happen, so my argument moves forward.
> There are so many more situations where static tools are not useful for
> recovery, so this does not back up your point at all. It does only proof
> that static tools are not utterly useless, which we all know already
> (otherwise we wouldn't talk to you at all).
"We can fix X, but since we can't fix Y, we won't fix X".
Comparing the number of failure points in total, with the number
of failure points removed by statics doesn't seem to be very interesting.
It is much more interesting to wonder whether the failure points that
are removed by statics are common enough failure points, or whether
they are not (regardless of how many other failure points there may be).
The dynamics failed two weeks ago in unstable due to an intricate
circular dependency (between bash and libreadline) that only affected
certain people, and in particular, did not affect people who were
upgrading systems that were already largely installed.
Also, I previously wiped them out by getting a bit too clever and
trying to force through an upgrade that the package manager told
me was a bad idea. It turned out the package manger was right--I
have enough experience not to override the package manager on a
production server, this was a desktop machine. But it did wipe out
my C library, and illustrate the fact that it's possible for an admin
to think they're doing something clever, when actually they're
making a huge mistake. (I figured I could manually install a subsequent
dependency, but as it turned out, by then it was too late.)
The point here is not to argue that they're frequent because they
happened to me; the point is to argue that they are possibilities
that can happen to anyone.
All of the above is somewhat loose so far, since who really knows how
many systems are important, or how often failures like this happen.
But in my mind it's this next point that nails it down. It's something
I've said many times, and so far nobody has responded to it:
-- Many people don't know how important their server is until it fails
plus
-- Many people don't know why statics are useful, even if they're
running a server where they're essential
plus
-- Some people, especially experienced Unix people, assume that
Debian provides statics by default
and
-- By the time you realize that you should have installed them,
it's too late, and your only option is to reboot (whatever the cost)
You can chalk all of these up to "lack of experience", either, lack
of experience with Unix in general, or lack of experience with
Debian in particular. The number of people with high level of
experience with Unix is very low compared to the number of people
who are still figuring it all out. (If you need proof, I'll pass your
name to a few recruitiers who are desperately trying to find any kind
of Unix administration experience no matter how shallow, to run
some fairly impressive sounding systems.)
Here's the point:
If you need static binaries, you REALLY need them. If you don't
need them, they won't cost you much. And since many people who
do need them don't know it (until things fail), it is important
that they be provided by default.
The very few people running Debian on a 386sx25 with 4M memory and
an 80MB hard drive can turn the default off without too much trouble,
and anyone installing onto a platform like that almost certainly
knows what they're doing (or at least, they'd better, because they're
going to face much stiffer challenges).
> > Proposition #6-- If a failure of the C library occurs, and the
> > servers are still running, then on a system where
> > downtime is considered unacceptable or difficult, you
> > should fix the problem without a reboot if that is
> > possible
> > Proposition #7-- Debian should include by default whatever is
> > required to fix the server as in #6, providing the
> > cost of including these services are not too high
>
> This is wrong. An additional requirement is that the benefit is high.
> Proposition #6 is a very narrow occurence, a situation which will occure on
> very few systems over time, so it is not suitable as default.
The benefit may be negligible to most people, but to the people who
need it, the benefit is enormously high.
> > Proposition #9-- Statics recovery tools are more fault tolerant
> > than dynamics equivalents
> >
> > Proof: Dynamics typically depend on a large number of files, and
> > the failure of any of those files will bring the dynamic
> > binary down.
>
> This is correct. It is also correct that static binaries are bigger and
> failure of any of the blocks will bring the static binary down.
But a failure in a block will only bring one static binary down. The
redundancy will help a lot. In another message someone asked me how I
would copy files if "cp" failed, and I was able to think up a huge
number of ways to do this. You can get equally creative over editing
files, and so forth.
Statics are highly redundant by their nature. They are not just one
level of redundancy, since each static binary exists on its own and
fails independent of all the others.
> The overall size of all static binaries will not be very much smaller than
> the overall size of dynmaic binaries plus shared library, so again the
> benefit of static binaries is much smaller than you claim.
No. A single block failure in the C library will bring down every dynamic
binary on the system. A single block failure in a static binary will bring
down just that binary.
The largest and most important static binary (sash) is a lot smaller
than the sum total of the size consumed by every critical dynamic binary.
There's a whole order of magnitude here.
> The number of files is almost irrelevant. The number of blocks occupied is a
> bit more relevant.
Not true. If what you are facing is a software error (rather than a hardware
error) then corruption is much more likely to hit inodes than blocks.
> Indeed, it is worse with static binaries. If you loose sash, your recovery
> environment becomes unusable. If I loose, let's say, mount, I still have all
> other tools.
This is totally misleading. Sure, losing mount is not so bad. The problem
with dynamic linking is that there are a large number of critical files
upon which EVERYTHING depends. When one of those goes down, you lose it
all. There are large, single points of failure, that eliminate any
chance for recovery.
> You are only shifting the weakness of the system from the libc6 library to
> the recovery tools. This is not such a big deal as you claim. You could also
> set up a redundant environment with dynamic bins, for example a chroot
> environment.
This is false.
You are not just shifting it around, you are massively increasing the
redundancy in your system. Each static binary stands on its own, and
fails on its own. You would have to knock out a large number of statics
to leave the admin with no options; with your second set of dynamic
libraries you have expanded from a single point of failure to two
points of failure.
For example, if I lost my static sash, and on top of that I lost all
my dynamic libraries, then there is still a chance I have a root shell
open, and can make use of the remaining, working static binaries
to fix my system. I have a chance at it anyway, whereas with dynamics
I have no chance.
I would probably accept the second set of dynamic C libraries, because
the probability of knocking out both C libraries is low enough--but if
you're going to add the redundancy, you might as well maximize it.
> Any system that Proposition #6 applies to will also need hardware
> measurements like RAID.
I don't think so. I think I only have to illustrate that there's a
group of people out there who wouldn't know to install it, for whom
it is massively important.
You're assuming that every admin is fully versed in every aspect
of administration; or that if they fail to read all the documentation
then it's their fault and they should be hung out to dry.
> > For example, bash depends on:
> > -- bash, libncurses.so.4 (symlink), libncurses.so.4.2,
> > libdl.so.2 (symlink), libdl.so.2.1.2, libc.so.6
> > (symlink), libc-2.1.2.so, ld-linux.so.2, and
> > until recently on libreadline as well (2 files)
> > whereas sash depends on only one file (sash). Thus there
> > are eight (previously 10) files upon which bash depends,
> > but only one on which sash depends.
>
> Thus if sash breaks, your whole recovery environment goes down. Great.
No. If sash breaks then I lose one of my recovery tools, the rest of
them continue to work. This is why static linking is ideal for
recovery tools. I lose my ability to create new root shells, and I
lose several key commands--I'm seriously hurt, but I'm not yet
out of the game. And to get to that state, I already had to have
lost my C library as well, or losing sash wouldn't be such an issue.
> > Proposition #11-- Static Recovery binaries do not use a lot of
> > system resources, such as memory, disk, or CPU
>
> This depends on the hardware you have.
Not for memory and CPU resources, since your static recovery tools simply
will not be loaded. And even if you use sash as your root shell, it
consumes less memory than each instance of a dynamically linked bash.
As for disk space, I had statics installed on a 200MB drive and it
was not a problem--so you have to get to a really, really small drive
before it's an issue. At that point what you are installing will
look NOTHING like a standard Debian install, so the space issue
is a complete non-issue. If you are already customizing the system
so that it will fit into such a small space, then you are probably
willing to turn the statics off. It's no extra work.
> As you want to make it the default, you have to add up the space occupied on
> ALL systems. This is much too high.
First, no, not all systems. People can turn it off. I said it should be
the default; I didn't say it should be absolutely impossible to shut off.
Secondly, no it isn't too high. It's a small amount of space, unless
you are installing onto a system that is already so small that you
need to do non-standard things anyway.
> > Proposition #12-- The presence of static recovery tools will
> > not pose any difficulties to ordinary use of the system
>
> Every byte counts on low end hardware, which we also support and which is
> more common than high end servers.
That was proposition #11, proposition #12 means that you will not
notice any functional difference. The static recovery tools can be
installed in such a way that nobody notices a difference in
functionality.
> > Proposition #13-- Static binaries can be provided with negligible cost
>
> Wrong. Only on the server systems you describe. Not on a 4 MB Ram i386 with
> a 80 MB hard disk.
Are you trying to convince me that you can install everything in "standard"
onto a 4MB ram i386 with an 80MB hard disk, and that if you were doing
that, you wouldn't already be doing some pretty non-standard stuff?
Seriously, this is a complete non-issue.
Static binaries CAN be provided with negligible cost. *NEGLIBLE*, as
in people will simply not notice the difference.
>
> > Conclusion: Debian should provide static binaries
>
> Yes, but NOT by default.
For the people who need it (who may not know that they need it) the
benefit is absolutely enormous. For others it might be useful. And
the cost is negligible.
So yes, by default.
> > These were desktop machines,
> > but no matter--the experience of needing statics was the same.
>
> Proposition: What works for you is not what works for everybody else.
> Proof: I did always well with a boot disk or a second root partition.
Right. But what is critical for many people should not be excluded,
unless you want to make it Debian policy that, by default, those
people are excluded. I don't think you want to do that, but you are
free to show me that I am wrong--propose a policy change asserting
that Debian should not be useful, by default, for remotely administered
servers, or for servers where a reboot is unacceptable.
> You seem to fail to understand the fundamental points:
>
> Make it optional, it can become the default later if it is good.
> Put up or shut up.
I don't misunderstand any fundamental points. I have argued, I feel
convincingly, that it must be the default, and that it is unacceptable
if it is merely optional.
I recognize that you made a seriously credible effor to show
shortcomings in my proposal, so this has been very productive. I
eagerly await your next round :-)
> The only thing people sets up is your world-saving attitude. Many people do
> well without static binaries, and instead wasting your time on trying to
> convince us otherwise you better work on something that can be used.
Yes, many people do well without static binaries. And those very same
people wouldn't notice the slightest difference if they had them. Further,
there is a small but significant group of people for whom the statics
are absolutely critical, and who may not know enough to realize that.
Justin
Reply to: