Debian Bug report logs - #35049
The bash segfault problem

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Richard Braakman <dark@xs4all.nl>

To: submit@bugs.debian.org

Subject: The bash segfault problem

Date: Thu, 25 Mar 1999 18:18:57 +0100 (CET)

Package: libc6
Version: 2.1.1-0.1

I have some preliminary results for this. 

The symptom: A bash compiled on a slink system will always segfault
immediately when run on a potato system.  This interferes with upgrades.

Cause of the segfault: bash tries to call stat, which is not resolved
by fixup.  It ends up trying to call a function at address 0.

The symbol "stat" is defined by the libreadline in slink, so a bash
compiled against that libreadline will try to use it.  The symbol
"stat" is _not_ defined by the libreadline in potato.  It is not
defined by any other library either.  So bash crashes.  I don't know
why this doesn't cause a pretty linker error message.

"stat" is normally not defined as a symbol, because <sys/stat.h> defines
an inline function for it that wraps __xstat.

I'm not exactly sure where it comes from in libreadline.  It's not
part of any of its .o files.  It also has no debug information, even
though libreadline is compiled with -g.  It's definitely the wrapper
for __xstat that is defined in <sys/stat.h>.  It's definitely in the
text section of libreadline.so, though it's not in libreadline.a.

My best guess is that something caused gcc to include a standalone
version in addition to inlining it, despite the "extern __inline__"
combination.

When I compile potato's bash (2.02.1-1.3) on a slink system, its
libreadline still has a "stat" symbol, but its bash executable does
not refer to this symbol.  When I compile slink's bash (2.01.1-4.1) on
a slink system, its bash executable does refer to this symbol.  I have
been unable to find the reason for this difference.

Note that all these comments apply to fstat and lstat too, it's just
that bash happens to crash on stat first.

A scan shows that quite a few binaries define a "stat" symbol in
their text section.

It does not matter for bash whether I compile on a slink system or a
potato system, but it does matter for libreadline.  If I compile
2.02.1 on a potato system, its libreadline will not define the stat
symbol.

Oh... once thing I noticed just now is that bash 2.02.1 compiles
with -D_FILE_OFFSET_BITS=64 when compiled on a potato system.
That may be it.  It means that both libreadline and bash will
use stat64 rather than stat.  Indeed, this is what makes potato's
libreadline fail to define stat -- it does define stat64 instead.
The bash that goes with it is still well-behaved and refers only to
__xstat64.

I hope this makes sense.  I'm too tired to continue now.  The easiest
solution may be to force libreadline to include a definition of stat,
by taking its address or something.

Richard Braakman

Message #10 received at 35049@bugs.debian.org (full text, mbox, reply):

From: Joost Kooij <joost@pc47.mpn.cp.philips.com>

To: debian-devel@lists.debian.org

Cc: 35049@bugs.debian.org

Subject: glibc2.1 upgrade braindamage (bash broken, failing postinsts)

Date: Fri, 26 Mar 1999 11:12:23 +0000 (/etc/localtime)

Hi,

The upgrade of libc from 2.0 to 2.1 caused some notable problems, some
minor, some major.  The cause of the problems appears to be described in
Bug#35049 ( <http://www.debian.org/Bugs/db/35/35049.html> .)

The problems started after doing a complete upgrade from stable slink to
unstable potato.  The upgrade seemingly went well for me, I did not follow
the entire process with fullest scrutiny, but when it had finished,
everything seemed fine. 

However, after I while, I found that "date" gave weird output: instead of
printing the timezone, it printed "/etc/localtime".  This is obviously not
correct and unfortunately I could not find a manpage about the file, so I 
knew neither where things had broken, nor where to fix them.

Silly timezone information in date being a minor problem (I don't know the
scope of damage caused by an incorrect timezone setting in general
though,) the effects on certain scripts were more annoying.  For instance,
dwww's daily cronjob began printing errors, because "/etc/localtime" made
a sed expression crash in dwww-build.

More worse, a colleague found his bash completely broken after the
upgrade.  The consequence was, that dpkg in effect broke also, as any
installation script using bash or sh would crash.  

To his luck, he still had a root shell open and we were able to unpack the
bash deb from potato manually and copy the bash executable to /bin, which
fixed bash.  After that, we could install the bash deb regularly.

When I read about Bug#35049, I figured where the cause of my timezone
problems might be.  I guess that libc6.postinst crashed, causing the
proper symlinks not to have been made.  Manually rerunning
"/var/lib/dpkg/info/libc6.postinst configure" cured the timezone problem.

I now wonder how many more installation scripts were not run due to bash
being broken at a certain stage in the upgrade.  Maybe I should run 
"find /var/lib/dpkg/info -name \*.postinst -exec {} configure" just to be
sure everything is configured okay.

There's a good chance that people will experience bugs because of the
abovementioned problems, please check if the breakage is not due to a
(couple of) installation scripts not having run correctly before filing
bugs to the package that doesn't behave.  

Anyway, apart from these "glitches", I'm quite content about my system 
running potato - thanks to everyone's fine efforts making this possible.

Cheers,


Joost

Message #15 received at 35049@bugs.debian.org (full text, mbox, reply):

From: Richard Braakman <dark@xs4all.nl>

To: 35049@bugs.debian.org

Cc: maor@debian.org

Subject: More information

Date: Fri, 26 Mar 1999 12:47:56 +0100 (CET)

(I'm Cc:ing the bash maintainer on this one.  Guy, please see the logs
for bug#35049 for the context.)

I found the difference.  bash 2.01.1 was compiled without
optimization, and bash 2.02.1 was compiled with -O2.  (This means we
shipped slink with an unoptimized bash, hmph.)

Without optimization, the stat functions defined in <sys/stat.h> are
not inlined.  They are referenced as external symbols, and the glibc
people expect that they will be found in libc_nonshared.a.

When a shared library is compiled without optimization, it links these
unshared definitions in, and for some reason exports them again.  They
are now shared, and unoptimized programs that use this library will
find the shared definitions rather than the static ones.

I think this is a bug in glibc, since it silently pollutes the binary
interface of such libraries.  It is aggravated by glibc2.1, because
the "stat" symbols can now be "stat64" symbols, silently changing the
binary interface.

Note that libreadline is always built non-optimized, for both bash
versions. When I looked for the reason, I found it in the debian/rules
file!  libreadline in bash 2.02.1-1.3 is specifically re-built without
optimization:

# Why override CFLAGS below?  Weird segfaults with the previous bash
# which was accidentally non-optimized and an optimized readline.
        rm lib/readline/{*.o,*.so}
        $(MAKE) -C lib/readline SOVERSION=$(soversion) CFLAGS=-g libreadline.so libhistory.so

Argh!  The problem was already known, but not properly fixed.

(libreadline in bash 2.01.1-4.1 was built non-optimized because
the whole package was; it did not have these lines in the rules file).

In the glibc sources, I see in the Makeconfig file that it was known
that these routines can end up in other shared libraries:

# Must build the routines as PIC, though, because they can end up in (users')
# shared objects.  We don't want to use CFLAGS-os because users may, for
# example, make that processor-specific.

(This is a comment for the code that sets compilation flags for
libc_nonshared.a)

What I still don't know is _why_ these definitions are unshared.
It seems to me that it can only cause problems.  What does it solve?

I see two ways to deal with the current situation.

Strategy 1

The symbol table pollution in libreadline is glibc's fault.  glibc
should deal with it by defining the necessary symbols dynamically.

This will fix bash, and prevent reoccurrence of this problem in
other libraries.  But if it is done only by Debian, it may cause binary
incompatibility with other platforms.

Strategy 2

For better or worse, libreadline's binary interface now includes
"stat" and "stat64" symbols (and lstat, etc...), and it must maintain
these symbols until its soname changes.  This might be done by copying
the defintions from <sys/stat.h>.

This may cause binary compatibility with other platforms, but no worse
so than we have already, and it will only affect binaries linked with
libreadline.  However, it does not fix other libraries that may have
this problem, and it remains to trap the unwary.

Message #20 received at 35049@bugs.debian.org (full text, mbox, reply):

From: Guy Maor <maor@debian.org>

To: Richard Braakman <dark@xs4all.nl>

Cc: 35049@bugs.debian.org

Subject: Re: More information

Date: 26 Mar 1999 09:07:01 -0800

Yes, you've identified the problem.  A version of bash was
accidentally released nonoptimized - I was trying to trace down a bug
and left out the -O symbols.  I've since had to maintain readline's
unoptimized state for binary compatibility.

Luckily, a new major version of readline was recently released, which
demands an soname change anyway.  So I can now solve this problem.


Guy

Message #25 received at 35049@bugs.debian.org (full text, mbox, reply):

From: Richard Braakman <dark@xs4all.nl>

To: maor@debian.org

Cc: 35049@bugs.debian.org

Subject: Re: More information

Date: Fri, 26 Mar 1999 18:39:41 +0100 (CET)

Guy Maor wrote:
> Yes, you've identified the problem.  A version of bash was
> accidentally released nonoptimized - I was trying to trace down a bug
> and left out the -O symbols.  I've since had to maintain readline's
> unoptimized state for binary compatibility.

(And this broke with glibc2.1, which introduced stat64)

> Luckily, a new major version of readline was recently released, which
> demands an soname change anyway.  So I can now solve this problem.

Great!  I expect this will fix the biggest slink -> potato upgrade problem.

I'll leave the bug open; I still think glibc is wrong for introducing
symbols in shared libraries this way.  But it's no longer release-critical.

Richard Braakman

Message #30 received at 35049@bugs.debian.org (full text, mbox, reply):

From: Daniel Jacobowitz <dan@debian.org>

To: Richard Braakman <dark@xs4all.nl>, 35049@bugs.debian.org

Cc: maor@debian.org

Subject: Re: Bug#35049: More information

Date: Fri, 26 Mar 1999 13:46:04 -0500

On Fri, Mar 26, 1999 at 12:47:56PM +0100, Richard Braakman wrote:
> (I'm Cc:ing the bash maintainer on this one.  Guy, please see the logs
> for bug#35049 for the context.)
> 
> I found the difference.  bash 2.01.1 was compiled without
> optimization, and bash 2.02.1 was compiled with -O2.  (This means we
> shipped slink with an unoptimized bash, hmph.)
> 
> Without optimization, the stat functions defined in <sys/stat.h> are
> not inlined.  They are referenced as external symbols, and the glibc
> people expect that they will be found in libc_nonshared.a.

<snip>

> What I still don't know is _why_ these definitions are unshared.
> It seems to me that it can only cause problems.  What does it solve?

[ Take this explanation with a grain of salt, I'm not sure I'm
remembering this correctly ]

They are unshared, if I remember correctly, to deal with interface
problems in changing stat() functionality.  Notice the definitions call
xstat, which is shared:

extern __inline__ int stat (__const char *__path,
                            struct stat *__statbuf) __THROW
{
  return __xstat (_STAT_VER, __path, __statbuf);
}

This is because the contents of struct stat show a higher-than-averagae
inclination to change.  _STAT_VER allows glibc to return the expected
struct stat; another piece of the binary compatibility effort.  Why
stat() needed a different solution than library versioning is unclear.

> I see two ways to deal with the current situation.
> 
> Strategy 1
> 
> The symbol table pollution in libreadline is glibc's fault.  glibc
> should deal with it by defining the necessary symbols dynamically.
> 
> This will fix bash, and prevent reoccurrence of this problem in
> other libraries.  But if it is done only by Debian, it may cause binary
> incompatibility with other platforms.

And I believe that Ulrich has in the past declined to make this change.


Dan

/--------------------------------\  /--------------------------------\
|       Daniel Jacobowitz        |__|     CMU, CS class of 2002      |
|   Debian GNU/Linux Developer    __   Part-Time Systems Programmer  |
|         dan@debian.org         |  |        drow@cs.cmu.edu         |
\--------------------------------/  \--------------------------------/

Message #35 received at 35049@bugs.debian.org (full text, mbox, reply):

From: Joel Klecker <jk@espy.org>

To: Richard Braakman <dark@xs4all.nl>, 35049@bugs.debian.org, maor@debian.org

Subject: Re: Bug#35049: More information

Date: Fri, 26 Mar 1999 11:18:09 -0800

At 18:39 +0100 1999-03-26, Richard Braakman wrote:
>Guy Maor wrote:
>> Yes, you've identified the problem.  A version of bash was
>> accidentally released nonoptimized - I was trying to trace down a bug
>> and left out the -O symbols.  I've since had to maintain readline's
>> unoptimized state for binary compatibility.
>
>(And this broke with glibc2.1, which introduced stat64)
>
>> Luckily, a new major version of readline was recently released, which
>> demands an soname change anyway.  So I can now solve this problem.
>
>Great!  I expect this will fix the biggest slink -> potato upgrade problem.
>
>I'll leave the bug open; I still think glibc is wrong for introducing
>symbols in shared libraries this way.  But it's no longer release-critical.

Upstream is aware of the issue, and I expect there will be a 
solution. In any event, I have solved the problem in my latest bash 
NMU by making libreadlineg2 conflict with bash (<< 2.02.1-1.2).

BTW, Guy, there is something I've been meaning to ask, have you ever 
attempted to send the /etc/inputrc changes to libreadline upstream?
--
Joel Klecker (aka Espy)                     <URL:http://web.espy.org/>
<URL:mailto:jk@espy.org>                  <URL:mailto:espy@debian.org>

Message #40 received at 35049@bugs.debian.org (full text, mbox, reply):

From: Richard Braakman <dark@xs4all.nl>

To: dan@debian.org

Cc: 35049@bugs.debian.org

Subject: Re: Bug#35049: More information

Date: Sun, 28 Mar 1999 13:07:41 +0200 (CEST)

Daniel Jacobowitz wrote:
> > Strategy 1
> > 
> > The symbol table pollution in libreadline is glibc's fault.  glibc
> > should deal with it by defining the necessary symbols dynamically.
> > 
> > This will fix bash, and prevent reoccurrence of this problem in
> > other libraries.  But if it is done only by Debian, it may cause binary
> > incompatibility with other platforms.
> 
> And I believe that Ulrich has in the past declined to make this change.

He prefers a solution that _does not work_?  Once a library is polluted,
all programs that link with it will use the stat in that library, and
the versioning is lost.

Richard Braakman

Message #45 received at 35049-close@bugs.debian.org (full text, mbox, reply):

From: Ben Collins <bcollins@debian.org>

To: 35049-close@bugs.debian.org

Subject: Bug#35049: fixed in glibc 2.1.94-1

Date: Tue, 26 Sep 2000 14:53:49 -0400

We believe that the bug you reported is fixed in the latest version of
glibc, which has been installed in the Debian FTP archive:
glibc_2.1.94-1.diff.gz
  to dists/woody/main/source/libs/glibc_2.1.94-1.diff.gz
  replacing glibc_2.1.3-10.diff.gz
glibc_2.1.94-1.dsc
  to dists/woody/main/source/libs/glibc_2.1.94-1.dsc
  replacing glibc_2.1.3-10.dsc
nscd_2.1.94-1_i386.deb
  to dists/woody/main/binary-i386/admin/nscd_2.1.94-1.deb
  replacing nscd_2.1.3-10.deb
i18ndata_2.1.94-1_all.deb
  to dists/woody/main/binary-all/admin/i18ndata_2.1.94-1.deb
  replacing i18ndata_2.1.3-10.deb
libc6-pic_2.1.94-1_i386.deb
  to dists/woody/main/binary-i386/devel/libc6-pic_2.1.94-1.deb
  replacing libc6-pic_2.1.3-10.deb
libc6-dbg_2.1.94-1_i386.deb
  to dists/woody/main/binary-i386/devel/libc6-dbg_2.1.94-1.deb
  replacing libc6-dbg_2.1.3-10.deb
libc6-dev_2.1.94-1_i386.deb
  to dists/woody/main/binary-i386/devel/libc6-dev_2.1.94-1.deb
  replacing libc6-dev_2.1.3-10.deb
libc6_2.1.94-1_i386.deb
  to dists/woody/main/binary-i386/base/libc6_2.1.94-1.deb
  replacing libc6_2.1.3-10.deb
glibc-doc_2.1.94-1_all.deb
  to dists/woody/main/binary-all/doc/glibc-doc_2.1.94-1.deb
  replacing glibc-doc_2.1.3-10.deb
locales_2.1.94-1_i386.deb
  to dists/woody/main/binary-i386/admin/locales_2.1.94-1.deb
  replacing locales_2.1.3-10.deb
glibc_2.1.94.orig.tar.gz
  to dists/woody/main/source/libs/glibc_2.1.94.orig.tar.gz
  replacing glibc_2.1.3.orig.tar.gz
libc6-prof_2.1.94-1_i386.deb
  to dists/woody/main/binary-i386/devel/libc6-prof_2.1.94-1.deb
  replacing libc6-prof_2.1.3-10.deb

Note that this package is not part of the released stable Debian
distribution.  It may have dependencies on other unreleased software,
or other instabilities.  Please take care if you wish to install it.
The update will eventually make its way into the next released Debian
distribution.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 35049@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Ben Collins <bcollins@debian.org> (supplier of updated glibc package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Format: 1.6
Date: Mon, 25 Sep 2000 11:30:45 -0400
Source: glibc
Binary: locales libc0.2-dbg glibc-doc nscd libc6.1-dbg libc0.2 libc6-dbg libc0.2-prof libc6.1 libc6 libc0.2-pic libc6.1-prof libc6-prof libc0.2-dev libc6.1-pic libc6-pic i18ndata libc6.1-dev libc6-dev
Architecture: source all i386
Version: 2.1.94-1
Distribution: unstable
Urgency: low
Maintainer: Ben Collins <bcollins@debian.org>
Description: 
 glibc-doc  - GNU C Library: Documentation
 i18ndata   - GNU C Library: National Language (locale) data [source]
 libc6      - GNU C Library: Shared libraries and Timezone data
 libc6-dbg  - GNU C Library: Libraries with debugging symbols
 libc6-dev  - GNU C Library: Development Libraries and Header Files.
 libc6-pic  - GNU C Library: PIC archive library
 libc6-prof - GNU C Library: Profiling Libraries.
 locales    - GNU C Library: National Language (locale) data [binary]
 nscd       - GNU C Library: Name Service Cache Daemon
Closes: 11300 17874 19264 20973 22301 26514 30987 32345 33759 35049 45903 46547 47560 48713 54154 61497 64865 65276 65683 65684
Changes: 
 glibc (2.1.94-1) unstable; urgency=low
 .
   * New maintainer, "Lector, si monumentum requiris, circumspice"
   * New upstream version, pre 2.2 now
     - crypt is now in glibc source, so is not a seperate tarball
     - removed nss1 compat tarball
     - db2 is gone aswell
     - without db2, upstream split nss_db from main source (*sigh*)
   * Added inetd to list of services to restart
   * modfl/fmodl: documented in info pages, closes: #17874
   * Just a quick list of bugs that I can verify do not exist any longer,
     closes: #45903, #26514, #46547, #32345, #30987, #48713
   * fcloseall: in the case of stdio/fcloseall.c, yes, it only ever
     returns success. However, we use libio/fcloseall.c, who does in fact
     have a chance to return EOF, closes: #20973
   * libio/libio.h: shows that the Stream Hooks do in fact take (void *) as
     the first argument as shown in the protos, closes: #61497
   * trunc/floor: documented correctly in this release, closes: #65683, #65684
   * Hurd maintainers say this can be closed, closes: #54154
   * I'm pretty sure this isn't an issue anymore, else potato wouldn't be
     releasing, closes: #35049
   * Sorry this isn't a glibc bug. The kernel handles error returns on a
     failed executable. Most likely this is bin_interp's problem, but I
     seriously doubt it will be fixed because of conventions, standards and
     the like, closes: #22301
   * keyenvoy: no longer compiled for linux (it seems), closes: #47560
   * infnan: is defined now, closes: #19264
   * libc5 bug, no longer applies, closes: #11300
   * Make sure we copy over asm-sparc for sparc aswell as "generate-asm.sh"
     script used to generate /usr/include/asm
   * Disable parallel build on sparc (broken for some reason, might not be
     sparc specific)
   * devpts.sh: used a more devfs friendly version from bug submitter,
     closes: #65276
   * libc/postinst: cannot reproduce problem, appears to be user error,
     closes: #64865
   * glibc-doc: this bug is no longer valid, closes: #33759
   * We now use ldconfig from libc6 for all archs, ldso will conform.
   * Change build deps to just "kernel-headers" for non-Hurd archs. With
     mips and other coming down the pipe, this is bound to get ugly if we
     specify the particular version for each.
   * Add checks to automatically detect proper kernel-headers, error out
     otherwise.
   * hurd: add Depends: hurd (>= 20000803) for libc0.2
   * libpthread: soname version is now 0.9
   * debian/libc/DEBIAN/shlibs: bump to 2.1.94
   * libc6: add temporary depend on libdb2
   * debian/rules: check for FAKEROOTKEY, and fail if it's there. We cannot
     build under fakeroot, we need real rewt. Fear my hacking skillz.
Files: 
 b12264380255b85ab75cd9f2ad251d6a 1056 libs required glibc_2.1.94-1.dsc
 aa9e35f334b7e101e2e463240c90dd42 8990988 libs required glibc_2.1.94.orig.tar.gz
 01126069e02a4636ebf0832d80708c2c 115450 libs required glibc_2.1.94-1.diff.gz
 e4a40dc28035546ff79b766b5d583129 2426038 base required libc6_2.1.94-1_i386.deb
 28434da6d804129d3a6ad7cc996c9ab4 1951728 devel standard libc6-dev_2.1.94-1_i386.deb
 ae86035879e9414a8782876d23a2b083 866092 devel extra libc6-prof_2.1.94-1_i386.deb
 34e6307a87e92156bf59fd41548c33cb 2209020 devel extra libc6-dbg_2.1.94-1_i386.deb
 a6c5521f539f1fb61eee4e10794b89d5 764650 devel extra libc6-pic_2.1.94-1_i386.deb
 53b85af2289f6d01ad1d3f2edd0d8a5f 566098 admin standard locales_2.1.94-1_i386.deb
 48093f5ff5d413a700b1f5f4954963f7 35056 admin optional nscd_2.1.94-1_i386.deb
 ee2f0c9080783cd3fdb06c2c0fcca3e0 2393990 admin extra i18ndata_2.1.94-1_all.deb
 066a2096aca1fd7dca507ba5bd4ebb2b 2460542 doc optional glibc-doc_2.1.94-1_all.deb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.2 (GNU/Linux)
Comment: Some people are fools, some are just foolish

iD8DBQE50BMpfNc/ZB4E7C0RAm6aAKCQV5z2RaPrdAs+qsnRxu8Ianvz5ACfcZNs
acsm0HMD7EU25DHZbHLSMuQ=
=RpJU
-----END PGP SIGNATURE-----

Message #50 received at 35049@bugs.debian.org (full text, mbox, reply):

From: Richard Braakman <dark@xs4all.nl>

To: 35049@bugs.debian.org

Subject: Re: Bug#35049 acknowledged by developer (The bash segfault problem)

Date: Wed, 27 Sep 2000 12:35:31 +0200

On Tue, Sep 26, 2000 at 02:07:52PM -0500, Debian Bug Tracking System wrote:
>    * I'm pretty sure this isn't an issue anymore, else potato wouldn't be
>      releasing, closes: #35049

Actually, the basic problem is still there and it will resurface the next
time someone distributes a shared library compiled without optimization.

Richard Braakman

Send a report that this bug log contains spam.