Hi,
I'm not yet writing bug reports about this because this is a rather
huge matrix of things which are bad and I suspect I haven't found all
of this type.
While working on plaso and updating dependencies of it which are too
old for plaso's most recent release, I noticed that e.g. the libevtx
source package holds these subdirectories:
[…]curity-tools/libevtx → \ls -1d lib*/
libbfio/
libcdata/
libcdirectory/
libcerror/
libcfile/
libclocale/
libcnotify/
libcpath/
libcsplit/
libcthreads/
libevtx/
libexe/
libfcache/
libfdata/
libfdatetime/
libfguid/
libfvalue/
libfwevt/
libfwnt/
libregf/
libuna/
libwrc/
(Build-Depends: debhelper-compat (= 11), pkg-config, dh-python,
python3-dev, libbfio-dev)
I already got slightly suspicious when I saw this. But I only became
aware of what is fundamentally broken here, as I looked into the
libregf source package (note that its name appears above, but not in
the Build-Depends):
/tmp/libregf-20201007 → \ls -1d lib*/
libbfio/
libcdata/
libcerror/
libcfile/
libclocale/
libcnotify/
libcpath/
libcsplit/
libcthreads/
libfcache/
libfdata/
libfdatetime/
libfwnt/
libregf/
libuna/
(Build-Depends: debhelper-compat (= 11), pkg-config, dh-python,
python3-dev, libbfio-dev, libfuse-dev)
Looking at the Build-Depends: Seems as if the embedded code copy of at
least libbfio-dev has been detected and it was tried to use the system
library version. But it looks to me on a short glance ("git grep bfio
debian/") as if the embedded code copy is used anyway.
To get a third view point, I took libfwnt:
/tmp/libfwnt-20181227 → \ls -1d lib*/
libcdata/
libcerror/
libcnotify/
libcthreads/
libfwnt/
(Build-Depends: debhelper (>= 11), dh-python, pkg-config, python3-dev)
To be on the safe side and to not only having found identical named
directories with just the same name by accident, I also checked a
rather specific file name from libfwnt:
→ apt-file search -I dsc libfwnt_security_descriptor.c
libesedb: /libfwnt/libfwnt_security_descriptor.c
libevt: /libfwnt/libfwnt_security_descriptor.c
libevtx: /libfwnt/libfwnt_security_descriptor.c
libfsntfs: /libfwnt/libfwnt_security_descriptor.c
libfwnt: /libfwnt/libfwnt_security_descriptor.c
libpff: /libfwnt/libfwnt_security_descriptor.c
libregf: /libfwnt/libfwnt_security_descriptor.c
libscca: /libfwnt/libfwnt_security_descriptor.c
So I searched a little bit more generic. In the unpacked libevtx
package I ran the following command to search for all the libraries it
seems to embed in other source packages:
[…]curity-tools/libevtx → for i in lib*/ ; do apt-file search -I dsc /${i}Makefile.am ; done
The output is attached, here's the summary by source package:
→ cat embedded-code-copies.txt | cut -f1 -d: | sort | uniq -c | sort -rn | cat -n
1 22 libevtx
2 21 libevt
3 16 libpff
4 16 libesedb
5 15 libscca
6 15 libregf
7 15 libfsntfs
8 15 libbde
9 14 libfvde
10 14 libfsxfs
11 14 libfshfs
12 14 libfsext
13 14 libfsapfs
14 14 libewf
15 13 libvslvm
16 13 libvmdk
17 13 libvhdi
18 13 libsmraw
19 13 libolecf
20 13 libmsiecf
21 13 libluksde
22 12 libvshadow
23 12 libqcow
24 12 liblnk
25 12 libcreg
26 10 libsigscan
27 10 libbfio
28 8 libfwsi
29 7 libsmdev
30 5 libfwnt
So we have 30 library packages of which each contains 4 to 21 strongly
suspected embedded code copies — and close to 370 suspected embedded
code copies overall. (One of the hits is likely always the library
itself, so I always substracted 1 from the numbers in the second
column. Oh, and by chance I stumbled upon the worst case and took the
least worse case as one of my examples.)
This looks at lot like all being upstream projects of
https://github.com/orgs/libyal/repositories, i.e. more or less Joachim
Metz — who already made a bad impression on me beacuse his by far most
favourite git commit message seems to be "Applied updates" <O>:
https://github.com/libyal/libevtx/commits/main
And indeed, e.g. on https://github.com/libyal/libevtx/wiki/Building he
writes:
> Read first
>
> GitHub will offer you the download options "Source code (zip)" and
> "Source code (tar.gz)". These are copies of the source, as-is, in the
> git repository and not suited for distribution. Instead it is highly
> recommend to use the provided source distribution package, which
> contains all the necessary dependencies.
<O>
Ok, so we could just use the Github generated tar ball since we use
autoconf anyway. And since Joachim Metz seems to be a fan of regular
release, we should be able to do a switch with a new upstream release,
so we don't have to fiddle with the version numbers due to having two
different tar balls for the same upstream version.
But there's a downside: Joachim Metz signs all his "all inclusive" tar
balls, but of course the git-generated lean tar balls are not signed,
so not using them would make us loose his PGP signature on the
released code tar balls. And keeping the embedded code copies in the
source package, keeping track of them and making sure that none of
them is used, seems quite some effort.
So I'm not blaming Hilko (who is in Cc and sole "Uploader" of at least
those packages I looked into closer so far), but we have a very
consistent issue of "meaning well is not the same as doing well" at
upstream. And actually, I don't know which is the best solution for
Debian to get out of this corner of embedded code copy hell, either.
Oh, and it seems that so far none of the embedded code copies in these
library packages seems to be known to or tracked by the Debian
Security Team (Cc'ed):
→ GET https://salsa.debian.org/security-tracker-team/security-tracker/raw/master/data/embedded-code-copies | fgrep -f <(cat embedded-code-copies.txt | cut -f1 -d: | sort -u)
I also tried to see if I can find any of them via
https://dedup.debian.net/, but to no avail. So I suspect that we
likely have many different versions of these libraries as embedded
code copy. What a fsckup…
P.S.: One more thought: Would it make sense to have lintian to detect
these embedded code copies instead of doing a mass bug filing? Then
again, their usage seems to be very restricted to other packages of
the Debian Security Tools Team due to their digital forensic
functionalities…
P.P.S.: I'm myself a member of the Debian Security Tools Team.
Regards, Axel
--
,''`. | Axel Beckert <abe@debian.org>, https://people.debian.org/~abe/
: :' : | Debian Developer, ftp.ch.debian.org Admin
`. `' | 4096R: 2517 B724 C5F6 CA99 5329 6E61 2FF9 CD59 6126 16B5
`- | 1024D: F067 EA27 26B9 C3FC 1486 202E C09E 1D89 9593 0EDE
libbde: /libbfio/Makefile.am libbfio: /libbfio/Makefile.am libcreg: /libbfio/Makefile.am libesedb: /libbfio/Makefile.am libevt: /libbfio/Makefile.am libevtx: /libbfio/Makefile.am libewf: /libbfio/Makefile.am libfsapfs: /libbfio/Makefile.am libfsext: /libbfio/Makefile.am libfshfs: /libbfio/Makefile.am libfsntfs: /libbfio/Makefile.am libfsxfs: /libbfio/Makefile.am libfvde: /libbfio/Makefile.am liblnk: /libbfio/Makefile.am libluksde: /libbfio/Makefile.am libmsiecf: /libbfio/Makefile.am libolecf: /libbfio/Makefile.am libpff: /libbfio/Makefile.am libqcow: /libbfio/Makefile.am libregf: /libbfio/Makefile.am libscca: /libbfio/Makefile.am libsigscan: /libbfio/Makefile.am libsmraw: /libbfio/Makefile.am libvhdi: /libbfio/Makefile.am libvmdk: /libbfio/Makefile.am libvshadow: /libbfio/Makefile.am libvslvm: /libbfio/Makefile.am libbde: /libcdata/Makefile.am libbfio: /libcdata/Makefile.am libcreg: /libcdata/Makefile.am libesedb: /libcdata/Makefile.am libevt: /libcdata/Makefile.am libevtx: /libcdata/Makefile.am libewf: /libcdata/Makefile.am libfsapfs: /libcdata/Makefile.am libfsext: /libcdata/Makefile.am libfshfs: /libcdata/Makefile.am libfsntfs: /libcdata/Makefile.am libfsxfs: /libcdata/Makefile.am libfvde: /libcdata/Makefile.am libfwnt: /libcdata/Makefile.am libfwsi: /libcdata/Makefile.am liblnk: /libcdata/Makefile.am libluksde: /libcdata/Makefile.am libmsiecf: /libcdata/Makefile.am libolecf: /libcdata/Makefile.am libpff: /libcdata/Makefile.am libqcow: /libcdata/Makefile.am libregf: /libcdata/Makefile.am libscca: /libcdata/Makefile.am libsigscan: /libcdata/Makefile.am libsmdev: /libcdata/Makefile.am libsmraw: /libcdata/Makefile.am libvhdi: /libcdata/Makefile.am libvmdk: /libcdata/Makefile.am libvshadow: /libcdata/Makefile.am libvslvm: /libcdata/Makefile.am libevt: /libcdirectory/Makefile.am libevtx: /libcdirectory/Makefile.am libbde: /libcerror/Makefile.am libbfio: /libcerror/Makefile.am libcreg: /libcerror/Makefile.am libesedb: /libcerror/Makefile.am libevt: /libcerror/Makefile.am libevtx: /libcerror/Makefile.am libewf: /libcerror/Makefile.am libfsapfs: /libcerror/Makefile.am libfsext: /libcerror/Makefile.am libfshfs: /libcerror/Makefile.am libfsntfs: /libcerror/Makefile.am libfsxfs: /libcerror/Makefile.am libfvde: /libcerror/Makefile.am libfwnt: /libcerror/Makefile.am libfwsi: /libcerror/Makefile.am liblnk: /libcerror/Makefile.am libluksde: /libcerror/Makefile.am libmsiecf: /libcerror/Makefile.am libolecf: /libcerror/Makefile.am libpff: /libcerror/Makefile.am libqcow: /libcerror/Makefile.am libregf: /libcerror/Makefile.am libscca: /libcerror/Makefile.am libsigscan: /libcerror/Makefile.am libsmdev: /libcerror/Makefile.am libsmraw: /libcerror/Makefile.am libvhdi: /libcerror/Makefile.am libvmdk: /libcerror/Makefile.am libvshadow: /libcerror/Makefile.am libvslvm: /libcerror/Makefile.am libbde: /libcfile/Makefile.am libbfio: /libcfile/Makefile.am libcreg: /libcfile/Makefile.am libesedb: /libcfile/Makefile.am libevt: /libcfile/Makefile.am libevtx: /libcfile/Makefile.am libewf: /libcfile/Makefile.am libfsapfs: /libcfile/Makefile.am libfsext: /libcfile/Makefile.am libfshfs: /libcfile/Makefile.am libfsntfs: /libcfile/Makefile.am libfsxfs: /libcfile/Makefile.am libfvde: /libcfile/Makefile.am liblnk: /libcfile/Makefile.am libluksde: /libcfile/Makefile.am libmsiecf: /libcfile/Makefile.am libolecf: /libcfile/Makefile.am libpff: /libcfile/Makefile.am libqcow: /libcfile/Makefile.am libregf: /libcfile/Makefile.am libscca: /libcfile/Makefile.am libsigscan: /libcfile/Makefile.am libsmdev: /libcfile/Makefile.am libsmraw: /libcfile/Makefile.am libvhdi: /libcfile/Makefile.am libvmdk: /libcfile/Makefile.am libvshadow: /libcfile/Makefile.am libvslvm: /libcfile/Makefile.am libbde: /libclocale/Makefile.am libbfio: /libclocale/Makefile.am libcreg: /libclocale/Makefile.am libesedb: /libclocale/Makefile.am libevt: /libclocale/Makefile.am libevtx: /libclocale/Makefile.am libewf: /libclocale/Makefile.am libfsapfs: /libclocale/Makefile.am libfsext: /libclocale/Makefile.am libfshfs: /libclocale/Makefile.am libfsntfs: /libclocale/Makefile.am libfsxfs: /libclocale/Makefile.am libfvde: /libclocale/Makefile.am libfwsi: /libclocale/Makefile.am liblnk: /libclocale/Makefile.am libluksde: /libclocale/Makefile.am libmsiecf: /libclocale/Makefile.am libolecf: /libclocale/Makefile.am libpff: /libclocale/Makefile.am libqcow: /libclocale/Makefile.am libregf: /libclocale/Makefile.am libscca: /libclocale/Makefile.am libsigscan: /libclocale/Makefile.am libsmdev: /libclocale/Makefile.am libsmraw: /libclocale/Makefile.am libvhdi: /libclocale/Makefile.am libvmdk: /libclocale/Makefile.am libvshadow: /libclocale/Makefile.am libvslvm: /libclocale/Makefile.am libbde: /libcnotify/Makefile.am libbfio: /libcnotify/Makefile.am libcreg: /libcnotify/Makefile.am libesedb: /libcnotify/Makefile.am libevt: /libcnotify/Makefile.am libevtx: /libcnotify/Makefile.am libewf: /libcnotify/Makefile.am libfsapfs: /libcnotify/Makefile.am libfsext: /libcnotify/Makefile.am libfshfs: /libcnotify/Makefile.am libfsntfs: /libcnotify/Makefile.am libfsxfs: /libcnotify/Makefile.am libfvde: /libcnotify/Makefile.am libfwnt: /libcnotify/Makefile.am libfwsi: /libcnotify/Makefile.am liblnk: /libcnotify/Makefile.am libluksde: /libcnotify/Makefile.am libmsiecf: /libcnotify/Makefile.am libolecf: /libcnotify/Makefile.am libpff: /libcnotify/Makefile.am libqcow: /libcnotify/Makefile.am libregf: /libcnotify/Makefile.am libscca: /libcnotify/Makefile.am libsigscan: /libcnotify/Makefile.am libsmdev: /libcnotify/Makefile.am libsmraw: /libcnotify/Makefile.am libvhdi: /libcnotify/Makefile.am libvmdk: /libcnotify/Makefile.am libvshadow: /libcnotify/Makefile.am libvslvm: /libcnotify/Makefile.am libbde: /libcpath/Makefile.am libbfio: /libcpath/Makefile.am libcreg: /libcpath/Makefile.am libesedb: /libcpath/Makefile.am libevt: /libcpath/Makefile.am libevtx: /libcpath/Makefile.am libewf: /libcpath/Makefile.am libfsapfs: /libcpath/Makefile.am libfsext: /libcpath/Makefile.am libfshfs: /libcpath/Makefile.am libfsntfs: /libcpath/Makefile.am libfsxfs: /libcpath/Makefile.am libfvde: /libcpath/Makefile.am liblnk: /libcpath/Makefile.am libluksde: /libcpath/Makefile.am libmsiecf: /libcpath/Makefile.am libolecf: /libcpath/Makefile.am libpff: /libcpath/Makefile.am libqcow: /libcpath/Makefile.am libregf: /libcpath/Makefile.am libscca: /libcpath/Makefile.am libsigscan: /libcpath/Makefile.am libsmraw: /libcpath/Makefile.am libvhdi: /libcpath/Makefile.am libvmdk: /libcpath/Makefile.am libvshadow: /libcpath/Makefile.am libvslvm: /libcpath/Makefile.am libbde: /libcsplit/Makefile.am libbfio: /libcsplit/Makefile.am libcreg: /libcsplit/Makefile.am libesedb: /libcsplit/Makefile.am libevt: /libcsplit/Makefile.am libevtx: /libcsplit/Makefile.am libewf: /libcsplit/Makefile.am libfsapfs: /libcsplit/Makefile.am libfsext: /libcsplit/Makefile.am libfshfs: /libcsplit/Makefile.am libfsntfs: /libcsplit/Makefile.am libfsxfs: /libcsplit/Makefile.am libfvde: /libcsplit/Makefile.am liblnk: /libcsplit/Makefile.am libluksde: /libcsplit/Makefile.am libmsiecf: /libcsplit/Makefile.am libolecf: /libcsplit/Makefile.am libpff: /libcsplit/Makefile.am libqcow: /libcsplit/Makefile.am libregf: /libcsplit/Makefile.am libscca: /libcsplit/Makefile.am libsigscan: /libcsplit/Makefile.am libsmraw: /libcsplit/Makefile.am libvhdi: /libcsplit/Makefile.am libvmdk: /libcsplit/Makefile.am libvshadow: /libcsplit/Makefile.am libvslvm: /libcsplit/Makefile.am libbde: /libcthreads/Makefile.am libbfio: /libcthreads/Makefile.am libcreg: /libcthreads/Makefile.am libesedb: /libcthreads/Makefile.am libevt: /libcthreads/Makefile.am libevtx: /libcthreads/Makefile.am libewf: /libcthreads/Makefile.am libfsapfs: /libcthreads/Makefile.am libfsext: /libcthreads/Makefile.am libfshfs: /libcthreads/Makefile.am libfsntfs: /libcthreads/Makefile.am libfsxfs: /libcthreads/Makefile.am libfvde: /libcthreads/Makefile.am libfwnt: /libcthreads/Makefile.am libfwsi: /libcthreads/Makefile.am liblnk: /libcthreads/Makefile.am libluksde: /libcthreads/Makefile.am libmsiecf: /libcthreads/Makefile.am libolecf: /libcthreads/Makefile.am libpff: /libcthreads/Makefile.am libqcow: /libcthreads/Makefile.am libregf: /libcthreads/Makefile.am libscca: /libcthreads/Makefile.am libsigscan: /libcthreads/Makefile.am libsmdev: /libcthreads/Makefile.am libsmraw: /libcthreads/Makefile.am libvhdi: /libcthreads/Makefile.am libvmdk: /libcthreads/Makefile.am libvshadow: /libcthreads/Makefile.am libvslvm: /libcthreads/Makefile.am libevtx: /libevtx/Makefile.am libevt: /libexe/Makefile.am libevtx: /libexe/Makefile.am libbde: /libfcache/Makefile.am libcreg: /libfcache/Makefile.am libesedb: /libfcache/Makefile.am libevt: /libfcache/Makefile.am libevtx: /libfcache/Makefile.am libewf: /libfcache/Makefile.am libfsapfs: /libfcache/Makefile.am libfsext: /libfcache/Makefile.am libfshfs: /libfcache/Makefile.am libfsntfs: /libfcache/Makefile.am libfsxfs: /libfcache/Makefile.am libfvde: /libfcache/Makefile.am libluksde: /libfcache/Makefile.am libpff: /libfcache/Makefile.am libqcow: /libfcache/Makefile.am libregf: /libfcache/Makefile.am libscca: /libfcache/Makefile.am libsmraw: /libfcache/Makefile.am libvhdi: /libfcache/Makefile.am libvmdk: /libfcache/Makefile.am libvslvm: /libfcache/Makefile.am libbde: /libfdata/Makefile.am libcreg: /libfdata/Makefile.am libesedb: /libfdata/Makefile.am libevt: /libfdata/Makefile.am libevtx: /libfdata/Makefile.am libewf: /libfdata/Makefile.am libfsapfs: /libfdata/Makefile.am libfsext: /libfdata/Makefile.am libfshfs: /libfdata/Makefile.am libfsntfs: /libfdata/Makefile.am libfsxfs: /libfdata/Makefile.am libfvde: /libfdata/Makefile.am libluksde: /libfdata/Makefile.am libpff: /libfdata/Makefile.am libqcow: /libfdata/Makefile.am libregf: /libfdata/Makefile.am libscca: /libfdata/Makefile.am libsmraw: /libfdata/Makefile.am libvhdi: /libfdata/Makefile.am libvmdk: /libfdata/Makefile.am libvslvm: /libfdata/Makefile.am libbde: /libfdatetime/Makefile.am libesedb: /libfdatetime/Makefile.am libevt: /libfdatetime/Makefile.am libevtx: /libfdatetime/Makefile.am libfsapfs: /libfdatetime/Makefile.am libfsext: /libfdatetime/Makefile.am libfshfs: /libfdatetime/Makefile.am libfsntfs: /libfdatetime/Makefile.am libfsxfs: /libfdatetime/Makefile.am libfwsi: /libfdatetime/Makefile.am liblnk: /libfdatetime/Makefile.am libmsiecf: /libfdatetime/Makefile.am libolecf: /libfdatetime/Makefile.am libpff: /libfdatetime/Makefile.am libregf: /libfdatetime/Makefile.am libscca: /libfdatetime/Makefile.am libvshadow: /libfdatetime/Makefile.am libbde: /libfguid/Makefile.am libesedb: /libfguid/Makefile.am libevt: /libfguid/Makefile.am libevtx: /libfguid/Makefile.am libewf: /libfguid/Makefile.am libfsapfs: /libfguid/Makefile.am libfsext: /libfguid/Makefile.am libfshfs: /libfguid/Makefile.am libfsntfs: /libfguid/Makefile.am libfsxfs: /libfguid/Makefile.am libfvde: /libfguid/Makefile.am libfwsi: /libfguid/Makefile.am liblnk: /libfguid/Makefile.am libluksde: /libfguid/Makefile.am libmsiecf: /libfguid/Makefile.am libolecf: /libfguid/Makefile.am libpff: /libfguid/Makefile.am libvhdi: /libfguid/Makefile.am libvshadow: /libfguid/Makefile.am libbde: /libfvalue/Makefile.am libesedb: /libfvalue/Makefile.am libevt: /libfvalue/Makefile.am libevtx: /libfvalue/Makefile.am libewf: /libfvalue/Makefile.am libfvde: /libfvalue/Makefile.am libmsiecf: /libfvalue/Makefile.am libolecf: /libfvalue/Makefile.am libpff: /libfvalue/Makefile.am libscca: /libfvalue/Makefile.am libsmraw: /libfvalue/Makefile.am libvmdk: /libfvalue/Makefile.am libvslvm: /libfvalue/Makefile.am libevt: /libfwevt/Makefile.am libevtx: /libfwevt/Makefile.am libesedb: /libfwnt/Makefile.am libevt: /libfwnt/Makefile.am libevtx: /libfwnt/Makefile.am libfsntfs: /libfwnt/Makefile.am libfwnt: /libfwnt/Makefile.am libpff: /libfwnt/Makefile.am libregf: /libfwnt/Makefile.am libscca: /libfwnt/Makefile.am libevt: /libregf/Makefile.am libevtx: /libregf/Makefile.am libregf: /libregf/Makefile.am libbde: /libuna/Makefile.am libbfio: /libuna/Makefile.am libcreg: /libuna/Makefile.am libesedb: /libuna/Makefile.am libevt: /libuna/Makefile.am libevtx: /libuna/Makefile.am libewf: /libuna/Makefile.am libfsapfs: /libuna/Makefile.am libfsext: /libuna/Makefile.am libfshfs: /libuna/Makefile.am libfsntfs: /libuna/Makefile.am libfsxfs: /libuna/Makefile.am libfvde: /libuna/Makefile.am libfwsi: /libuna/Makefile.am liblnk: /libuna/Makefile.am libluksde: /libuna/Makefile.am libmsiecf: /libuna/Makefile.am libolecf: /libuna/Makefile.am libpff: /libuna/Makefile.am libqcow: /libuna/Makefile.am libregf: /libuna/Makefile.am libscca: /libuna/Makefile.am libsigscan: /libuna/Makefile.am libsmdev: /libuna/Makefile.am libsmraw: /libuna/Makefile.am libvhdi: /libuna/Makefile.am libvmdk: /libuna/Makefile.am libvshadow: /libuna/Makefile.am libvslvm: /libuna/Makefile.am libevt: /libwrc/Makefile.am libevtx: /libwrc/Makefile.am
Attachment:
signature.asc
Description: PGP signature