Hi all,
A typical shasta run would reserve almost all the memory on our
arm64 CI runners, but it turned out it is possible to pass
different parameters to the run, in such a way that the program
does not reserve anywhere as much memory as the original test
used to. This also happens to fix the same issue observed on
Ubuntu runners, worked around by skipping the command hogging
the memory.
Have a nice day, :)
Étienne.
----- Forwarded message from Debian Bug Tracking System <owner@bugs.debian.org> -----
Date: Mon, 04 Dec 2023 16:30:03 +0000
From: Debian Bug Tracking System <owner@bugs.debian.org>
To: Étienne Mollier <emollier@debian.org>
Subject: Bug#1053509: marked as done (shasta: autopkgtest regression on arm64: is memory suddenly not enough?)
X-Mailer: MIME-tools 5.509 (Entity 5.509)
Your message dated Mon, 04 Dec 2023 16:25:57 +0000
with message-id <E1rABld-00DUBq-QI@fasolo.debian.org>
and subject line Bug#1053509: fixed in shasta 0.11.1-3
has caused the Debian Bug report #1053509,
regarding shasta: autopkgtest regression on arm64: is memory suddenly not enough?
to be marked as done.
This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.
(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact owner@bugs.debian.org
immediately.)
--
1053509: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1053509
Debian Bug Tracking System
Contact owner@bugs.debian.org with problems
Date: Thu, 5 Oct 2023 13:39:14 +0200
From: Paul Gevers <elbrus@debian.org>
To: Debian Bug Tracking System <submit@bugs.debian.org>
Subject: shasta: autopkgtest regression on arm64: is memory suddenly not
enough?
User-Agent: Mozilla Thunderbird
Source: shasta
Version: 0.11.1-1
Severity: important
User: debian-ci@lists.debian.org
Usertags: regression
Dear maintainer(s),
Your package has an autopkgtest, great. However, it recently started failing
on arm64 in unstable, testing *and* stable. Until mid September 2023, most
runs passed (there are some historical rare failures where we don't have logs
anymore), but somewhere between 2023-09-17 and 2023-10-01 the tests started
failing (although, in testing we had some passes on 2023-09-28 and 2023-10-04,
interestingly these runs too *much* longer than normal (factor ~20)). I first
suspected that this regression aligned with my fix for bug 1050256 where I
switched the kernel of our hosts to backports, but there are multiple good
runs with that kernel. Today I learned that "Killed" in the log is often
coming from the kernel when a program is allocating too much memory and the
kernel kills the process. We have some striking logs of failures [1] where the
shasta test actually bails out itself for lack of memory.
So, there's a couple of things (weirdness and options):
1) why does it now suddenly start to (nearly always) fail across the board on
arm64 (in Debian, Ubuntu still seems fine), without changes to the
infrastructure that I know of?
2) do you also believe this is related to memory consumption?
3) If 2 == yes, what are the memory requirements for the test? The test
*could* test for that before it starts and bail out (restriction: skippable
with exit code 77 [2]) if the amount of memory available is too small.
4) just stop testing on arm64 (but again, in Ubuntu the test is still running
fine)
5) the recent glibc security update also came to my mind, but that's not *yet*
installed on our hosts, nor is it installed in the stable testbed.
The release team has announced [3] that failing autopkgtest on amd64 and arm64
are considered RC in testing. However, due to nature of the symptoms, I've
filed this as important for now.
Paul
[1]
https://ci.debian.net/data/autopkgtest/testing/arm64/s/shasta/38544284/log.gz
48s 2023-Oct-04 12:01:05.603191 Memory allocation failure during mremap call
for MemoryMapped::Vector.
48s This assembly requires more memory than available.
48s Rerun on a larger machine.
[2] https://salsa.debian.org/ci-team/autopkgtest/-/blob/master/doc/README.package-tests.rst
[3] https://lists.debian.org/debian-devel-announce/2019/07/msg00002.html
Date: Mon, 04 Dec 2023 16:25:57 +0000
From: Debian FTP Masters <ftpmaster@ftp-master.debian.org>
To: 1053509-close@bugs.debian.org
Subject: Bug#1053509: fixed in shasta 0.11.1-3
Source: shasta
Source-Version: 0.11.1-3
Done: Étienne Mollier <emollier@debian.org>
We believe that the bug you reported is fixed in the latest version of
shasta, which is due to be installed in the Debian FTP archive.
A summary of the changes between this version and the previous one is
attached.
Thank you for reporting the bug, which will now be closed. If you
have further comments please address them to 1053509@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.
Debian distribution maintenance software
pp.
Étienne Mollier <emollier@debian.org> (supplier of updated shasta package)
(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Format: 1.8
Date: Mon, 04 Dec 2023 16:48:02 +0100
Source: shasta
Architecture: source
Version: 0.11.1-3
Distribution: unstable
Urgency: medium
Maintainer: Debian Med Packaging Team <debian-med-packaging@lists.alioth.debian.org>
Changed-By: Étienne Mollier <emollier@debian.org>
Closes: 1053509
Changes:
shasta (0.11.1-3) unstable; urgency=medium
.
* d/t/run-unit-test: use older shasta config.
The original test shipped with Nanopore-Sep2020 configuration file
selected by shasta in the autopkgtest script. This caused shasta to
reserve 8GiB of RAM on CI test hosts, which is just the limit on the
arm64 runners. Dropping to the older Nanopore-Dec2019 configuration
file causes shasta to reserve memory as needed instead of prefetching
a huge amount of pages, causing out of memory conditions on runners.
(Closes: #1053509)
Checksums-Sha1:
040018ed0c9238a38ddc3230fd428cdf07daba72 2556 shasta_0.11.1-3.dsc
e6c60a18e47219771ad09396f77ff758dd814ac1 10540 shasta_0.11.1-3.debian.tar.xz
Checksums-Sha256:
2d3bb7c87946f64039363aa7e382b1a8073dbbe5a93698b2aaa81979f2dfaf2d 2556 shasta_0.11.1-3.dsc
33fbd5d5431870c04127884a618f7e4f43f3cd919c3267f57de383ca315a1f76 10540 shasta_0.11.1-3.debian.tar.xz
Files:
6517b386aa94537798421b2f8ac6cba0 2556 science optional shasta_0.11.1-3.dsc
4b6ba7240615780b0ba071683ab16c7b 10540 science optional shasta_0.11.1-3.debian.tar.xz
-----BEGIN PGP SIGNATURE-----
iQJIBAEBCgAyFiEEj5GyJ8fW8rGUjII2eTz2fo8NEdoFAmVt9mQUHGVtb2xsaWVy
QGRlYmlhbi5vcmcACgkQeTz2fo8NEdp5TQ/+OQMT7WV+pZBeKX/VT6NDY2MDue8c
hqtsqUzqNQDsDl2QvszEgwd3c4kBYbS60dpbEM+hIo/DHqAHBR94tSBsxWJwQJ9T
akZO7xAF6w9M+gNZ11sDnmC0bO/yk46S82ZqzqyV+XAea/aSNlqdC7fIHa/l8QLz
fR4l8Z1AdKBxHsSx76Vv6Y4tIfbvDL4d3R1rPyNXj9L4XAGUgV/S0/YOXR2ermLq
MGr4E4cGd717u8c+B8B2/x15HzwIv+53L3NbnXPSqtqimtA/itjPuqNsZm4E66Sn
Abzq4tbNusFYXSHYpE+ascQMYd3Pq5ZID2VbJxKYqwBUIBiWWOj4yzzMnz7MEh8Z
zT4PiLaes87oFNWOVxRMBw07bQKPHP94wQSSJ8bPim6oYUD/XrftIeLkcgr+P9r2
pINDENpOnne3+GgwxL2UYDfoDMAimN9owrTuiCTqjRKGpGfUs/3KV432rKA0DATb
MkdQjRD6Wtz5Kf8/cn1H6oS8LbS3bEbzgHThEy9Dyd66g0RlL8oWd3hrPGIGxZsA
quUcW8WGDypwyv3fXkut53XGgmu1aU8scAODQ4E6gLD8PVn1JzAP31Fvt0yvmrif
Jwl+U4g0bglGMQFdk/M7fciLFmMjp2smdY8hQEW2HyuaeDmKdmdJfiHk2ubFNamg
jytusyXTO8sbdeY=
=zFjP
-----END PGP SIGNATURE-----
----- End forwarded message -----
--
.''`. Étienne Mollier <emollier@debian.org>
: :' : gpg: 8f91 b227 c7d6 f2b1 948c 8236 793c f67e 8f0d 11da
`. `' sent from /dev/pts/2, please excuse my verbosity
`- on air: Karmamoi - Black Hole Era
Attachment:
signature.asc
Description: PGP signature