Hi Raphael, On 12-10-2023 19:03, Raphael Hertzog wrote:
[ Please keep me in CC as I'm not subscribed to debian-ci@lists.debian.org ]
Done.As on of the maintainers of the infrastructure, I have some ideas or recommendation to add too:
* make sure you can block tests
o per architecture and per suite
o ideally even per host, because in practice not all hosts will be
equal
o ideally even per backend as you intent to support multiple
o also *after* the job has been scheduled. As an example, recently
sbuild started to kill our worker service instances while we had
quite some backlog, which we had to fully clear to resolve the
issue.
* ensure you can change the timeouts of autopkgtest
o as the infra maintainer, e.g. you want to allow more time on slow
archs: armel and riscv64 at this moment
o ideally per source package, as some tests run successfully, but just
take long, or just take long to unpack (so maybe even per timeout
type). E.g. we have had to block libreoffice on nearly all our
architectures because it's too big to startup on some hosts (also
too slow disks). Ubuntu has a big_package list for this [1].
o as the requestor, maybe you are ahead of the infra maintainers
* ensure you can direct certain sources to certain hosts (or prevent
certain hosts from being used). E.g. for amd64 our hosts currently are
very different; one is a beast while the others are normal. Some test
fail on one class while work on the other, things like #cpus, amount
of memory, hardware options, they all matter.
o of course autopkgtest could grow support for this too, but that
could make tests flaky depending on to which host they are sent, so
I believe the infra needs this too.
* I once played with /tmp and /var/lib/lxc (IIRC) on tmpfs but backed
out because some tests fail if not run on a "real" fs. By now I regret
that I backed out, but *maybe* you want this configurable? I regret
because particularly ppc64el hosts are slow because they are waiting
for io.
* If you support *default backend* too, I suggest you have per package
options to "block" certain backends. You probably want to run on the
"cheapest" backend that enables all tests (which can be detected if
you already process the d/t/control file) tests, but maybe there are
multiple tests where a subset make it use an "expensive" backend
without much added value.
* Maybe autopkgtest should grow an restriction to have test only run
when explicitly declared on the command interface, to enable writing
tests that don't need to run by default, but are nice to have during
development.
And now I ran out of idea, probably more come later.
Paul
[1]
https://git.launchpad.net/~ubuntu-release/autopkgtest-cloud/+git/autopkgtest-package-configs/tree/big_packages
Attachment:
OpenPGP_signature.asc
Description: OpenPGP digital signature