Re: pybuild-autopkgtest (was: Notes from the DC22 Python Team BoF)

[Julian Gilbey <julian@d-and-j.net>]

On Wed, Jul 27, 2022 at 10:26:33AM -0400, Louis-Philippe Véronneau wrote:
> The way I see it:
> 
> 1. We should have a Lintian tag for packages not using the new
> pybuild-autodep8 autopkgtest. It would be even better if this tag would be a
> pointed hint that identified 'manually' written unit test autopkgtests that
> could be replaced.
> 
> This way, you get something like:
> 
> python-foo source: not-using-pybuil-autodep8 [debian/tests/unittests]
> 
> for python packages that have old 'manually' written unit test autopkgtests
> and:
> 
> python-foo source: not-using-pybuild-autodep8 [no-autopkgtest]
> 
> for python packages without any autopkgtest.
> 
> 2. lintian-brush (or something else, but I think lintian-brush is the right
> tool) would go over these packages to:
> 
> 2.1 Add the new autodep8 autopkgtests and build the package to see if they
> pass
> 2.2 Remove the "manual" unit test autopkgtests if 2.1 succeeds
> 2.3 Open a bug report if 2.1 fails

I'd be wary about 2.2 and 2.3.  I have several packages where I know
that an automated test will fail; there are all sorts of weird cases
where I've had to write tests manually.  I would also be quite cross
if manually crafted tests were automatically removed, especially in
cases such as Simon mentioned where they do things that that
automatically generated test does not do.  Another thing I could
imagine happening is that the automated test succeeds in a trivial way
- it succeeds but doesn't actually test much because of the nature of
the package.

On the other hand, a bug report saying something like the following
seems much more reasonable: "We've tested this package using the
automated autopkgtest system and it seems to work by adding the line
'Testsuite: autopkgtest-pkg-pybuild'; please check that the automated
tests cover all of the tests of your manually written debian/tests/*
and if so, then please remove them.  The autopkgtest-pkg-pybuild logs
are attached."  This would give the maintainer the chance to decide
how best to proceed.

Best wishes,

   Julian