--- Begin Message ---
- To: Debian Bug Tracking System <submit@bugs.debian.org>
- Subject: python-apt: relies on MD5 internally to download packages
- From: Cyril Brulebois <cyril@debamax.com>
- Date: Thu, 14 Nov 2019 01:36:14 +0100
- Message-id: <157369177449.9422.1213703436713711073.reportbug@armor.home>
Package: python-apt
Version: 1.8.4
Severity: serious
Justification: some people want to get rid of MD5Sum in indices
Hi,
While debugging a live-wrapper (lwr) failure that started occurring
(literally) overnight, I ended up discovering it was triggered by the
intel-microcode package's getting a security upgrade.
live-wrapper 0.10 isn't affected, but live-wrapper's master branch has
an extra commit that automatically enables security sources for stable
releases.
Here's the traceback for a simple build (with a local mirror but anyone
would do) with that master branch:
$ sudo lwr -d buster -m http://wodi.home/debian -f intel-microcode
[…]
DEBUG environment: LWR_MIRROR = 'http://wodi.home/debian'
DEBUG environment: LWR_EXTRA_PACKAGES = ''
DEBUG environment: LWR_BASE_DEBS = ''
DEBUG environment: LWR_DISTRIBUTION = 'buster'
DEBUG environment: LWR_FIRMWARE_PACKAGES = 'intel-microcode'
DEBUG environment: LWR_TASK_PACKAGES = ''
[…]
Downloading udebs for Debian Installer...
INFO Downloading udebs for Debian Installer...
Updating a local cache for amd64 buster ...
DEBUG Updating local cache...
CRITICAL Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/cliapp/app.py", line 193, in _run
self.process_args(args)
File "/usr/lib/python2.7/dist-packages/lwr/run.py", line 143, in process_args
self.start_ops()
File "/usr/lib/python2.7/dist-packages/lwr/run.py", line 286, in start_ops
apt_udeb.download_udebs(exclude_list)
File "/usr/lib/python2.7/dist-packages/lwr/apt_udeb.py", line 157, in download_udebs
self.download_apt_file(pkg_name, pool_dir, False)
File "/usr/lib/python2.7/dist-packages/lwr/apt_udeb.py", line 141, in download_apt_file
version.fetch_binary(destdir=pkg_dir)
File "/usr/lib/python2.7/dist-packages/apt/package.py", line 867, in fetch_binary
if _file_is_same(destfile, self.size, self._records.md5_hash):
SystemError: error return without exception set
After some debugging, it turned out that merely accessing the
self._records.md5_hash item is sufficient to reproduce this issue.
Looking at the current (as of 2019-11-14 00:27:00 UTC) indices for
buster/updates on security.debian.org, one can only see SHA256 entries
in Release and Packages files, which is likely the reason for
python-apt's explosion. I've asked #debian-ftp to add MD5Sum entries
back at least for buster/updates, and will file another bug report for
that in a moment to make sure it isn't lost.
Looking at even the most recent python-apt code in experimental (1.9.0),
MD5 still seems hardwired, e.g. in apt/packages.py's fetch_binary():
def fetch_binary(self, destdir='', progress=None):
# type: (str, AcquireProgress) -> str
"""Fetch the binary version of the package.
The parameter *destdir* specifies the directory where the package will
be fetched to.
The parameter *progress* may refer to an apt_pkg.AcquireProgress()
object. If not specified or None, apt.progress.text.AcquireProgress()
is used.
.. versionadded:: 0.7.10
"""
base = os.path.basename(self._records.filename)
destfile = os.path.join(destdir, base)
if _file_is_same(destfile, self.size, self._records.md5_hash):
logging.debug('Ignoring already existing file: %s' % destfile)
return os.path.abspath(destfile)
acq = apt_pkg.Acquire(progress or apt.progress.text.AcquireProgress())
acqfile = apt_pkg.AcquireFile(acq, self.uri, self._records.md5_hash, # type: ignore # TODO: Do not use MD5 # nopep8
self.size, base, destfile=destfile)
acq.run()
if acqfile.status != acqfile.STAT_DONE:
raise FetchError("The item %r could not be fetched: %s" %
(acqfile.destfile, acqfile.error_text))
return os.path.abspath(destfile)
Notice the TODO on the apt_pkg.AcquireFile(), but it would probably
break in the same way as in the live-wrapper case a few lines before, on
the self._records.md5_hash item.
The same goes for fetch_source().
Since getting rid of MD5Sum entirely is a topic that comes up on a
regular fashion (with fingers being pointed at jigdo in particular), it
looks to me python-apt should get some attention as well; hence filing
at serious severity. Feel free to adjust as required.
Cheers,
--
Cyril Brulebois -- Debian Consultant @ DEBAMAX -- https://debamax.com/
--- End Message ---
--- Begin Message ---
- To: 944696-close@bugs.debian.org
- Subject: Bug#944696: fixed in python-apt 1.9.1
- From: Julian Andres Klode <jak@debian.org>
- Date: Wed, 11 Dec 2019 15:54:22 +0000
- Message-id: <E1if4Ji-0007yZ-Bk@fasolo.debian.org>
Source: python-apt
Source-Version: 1.9.1
We believe that the bug you reported is fixed in the latest version of
python-apt, which is due to be installed in the Debian FTP archive.
A summary of the changes between this version and the previous one is
attached.
Thank you for reporting the bug, which will now be closed. If you
have further comments please address them to 944696@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.
Debian distribution maintenance software
pp.
Julian Andres Klode <jak@debian.org> (supplier of updated python-apt package)
(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Format: 1.8
Date: Wed, 11 Dec 2019 16:17:24 +0100
Source: python-apt
Architecture: source
Version: 1.9.1
Distribution: experimental
Urgency: medium
Maintainer: APT Development Team <deity@lists.debian.org>
Changed-By: Julian Andres Klode <jak@debian.org>
Closes: 944091 944696
Changes:
python-apt (1.9.1) experimental; urgency=medium
.
[ Julian Andres Klode ]
* Install type information according to PEP561
* Temporarily perform CI on eoan only
* Adjust to mypy 0.710
* doc/examples: Convert to Python 3
* Convert all shebangs to python3
* gitlab-ci: Run 3 separate test stages
* pep8: Fix overindent issues
* ./doc/examples/dependant-pkgs.py: Make it work again
* Fix segmentation fault for apt_pkg.Cache.policy
* test_policy: Fix pyflakes issue
* apt.Cache: cache apt.package.Origin objects by id
* Adjust for PY_SSIZE_T_CLEAN (Closes: #944091)
* Turn InstallProgress into a context manager to avoid leaking files
* AcquireFile: Remove md5 parameter
* AcquireFile: Accept HashStringList in hash parameter
* apt/package.py: Use all hashes when fetching packages
* Remove leftover MD5 use in Version.fetch_binary() (Closes: #944696)
* Fix type hints
* typehinting: Override TagSection.get()
* debian/control: Rules-Requires-Root: no, Standards-Version upgrade
* Update po template
* Fix pre-build script to actually work in Python 3
* Run the pre-build script
* lintian fixes
- fix typo in changelog error
- remove use of extra priority
- override uses-dpkg-database-directly for the root location switching magic
* Remove spurious python-debian test dependency (see bug #937579)
* Build-Depend on pycodestyle, not pep8
.
[ Raphaël Hertzog ]
* Add Kali templates
.
[ Colomban Wendling ]
* Fix InstallProgress for installing .deb files on Python >= 3.4
* Add a test case for checking the communication with dpkg works
.
[ Michael Vogt ]
* apt, python: make `mypy --strict` clean
.
[ Matthias Klose ]
* data/templates/Ubuntu.info.in: Add focal
.
[ Dave Jones ]
* Don't duplicate disabled sources during add() (LP: #1311056)
Checksums-Sha1:
5cd43cc947f283523b459b18819828bac22c034a 2437 python-apt_1.9.1.dsc
caefccdcde07b1afaead9f80e6623a22013ee575 330280 python-apt_1.9.1.tar.xz
e9f2d06870f218ff0d55b3b6ee38510c21f2874c 10175 python-apt_1.9.1_source.buildinfo
Checksums-Sha256:
f02d675cf2c9cd14b96fd6537c21670f3a164a1dac30511481059f37e6c75897 2437 python-apt_1.9.1.dsc
05603c0821d6102c9f80dff7f2b8b7c30516aec31f2864124a7d8c2242495755 330280 python-apt_1.9.1.tar.xz
697389844836e7578c3b5e971281372d7db41c43af19ca3037845da9aa6f5399 10175 python-apt_1.9.1_source.buildinfo
Files:
e3643bec22d65a7dce83762d4abbdeba 2437 python optional python-apt_1.9.1.dsc
5d63ec1006dd925c2e1d0301d74207da 330280 python optional python-apt_1.9.1.tar.xz
caf67cce65f3acb79635aa5604447121 10175 python optional python-apt_1.9.1_source.buildinfo
-----BEGIN PGP SIGNATURE-----
iQJDBAEBCgAtFiEET7WIqEwt3nmnTHeHb6RY3R2wP3EFAl3xC3wPHGpha0BkZWJp
YW4ub3JnAAoJEG+kWN0dsD9x4fIP/iu3ndeHxk6LUi0qSkrOOTRtKYwzc8GGtuT4
6XIbmTqLCHMAfUe7oGnGV2mGmWvjlnizSjmGC9mpY/ryi+QmMb4+6N6/fuBs1BG5
d4eAlN9IfaWbWIk6Z1r4qdoQR1OsZpyUNTtRw4yeIPkgriOExkRweZKTSgjrgKUn
1mNQqfgpQGNbiCeVe7Ti8LBeEm4DR+zZK/fk0QspDf10isX+DMX6vmHMvwSRyiA1
Pozm39sSjCTVC4cEHz/3Nmd+IQQSa+1o4UGwOBypMjnrNohsQdojMKJT2AQWa/0O
9wu7A6lN3IjS7DghCbSBUAr0+5uGd/m2bbgv7VCkoQcbe0nblsc7bOQMTVnfjfLt
SDVji2J5Fgso7AuIjF0ZZatCA/iaDV2u+T8MDMpfo6eAG4cnR5f+IIBMa3zm8Y2Z
Dqp4X+KaGI28xP3Sv6CtUEM8JyAbcnhXXOY8H9B0aTcb14UFyHYZGpMU8LLsv/Rs
3Y9lyFWnMyAnbWMiUEt7dbGou+Awu1XFzNc2qyqcfaUsnX1QlQdT/vm4INwajvNh
FqE/fwnVNtM8tbS95ZYIfHeMkcJd4Yy5mPIbEOiHh/ezRVY9sfq+3iPk+ZwFxuNX
SmXhvY9YQ5OW6mBNXLe1S4QrRRRbwWE5ydBMc/wsH5lm/fCqno9ox4nk385KNxqh
nreyLxHx
=J+a5
-----END PGP SIGNATURE-----
--- End Message ---