Bug#788659: autopkgtest: Provide option to not implicitly run undeclared tests

[dom@earth.li (Dominic Hargreaves)]

On Mon, Jun 15, 2015 at 07:30:34AM +0200, Martin Pitt wrote:
> Hello Dominic,

Hello,

Thank you for your reply (and indeed your prompt implementation).

> Dominic Hargreaves [2015-06-14  0:18 +0100]:
> >   To allow test execution environments to discover packages which provide
> >   tests, their source packages should have a ``Testsuite:`` header
> >   containing ``autopkgtest`` (which is currently the only defined value).
> >   Multiple values get comma separated, as usual in control files.
> > 
> > My expectation having read this was that adt-run would only attempt
> > to test packages if the Testsuite header is present; however adt-run
> > actually appears to not look for this header at all;
> 
> That's correct. If you run adt-run on a source tree/dsc etc., it will
> only look for debian/tests/control (or autodep8). The above is just a
> recommendation ("should") and states the reason for that.
> 
> > it always processes tests for a package, including automagic test
> > suites set up by autodep8. This is probably fine in the case of
> > invoking adt-run with a single package, but not if you want to look
> > for regressions using adt-run; for example I ran it against all
> > packages I'd been rebuilding against perl 5.22, and found quite a
> > few failures which didn't appear to be related to perl 5.22 but were
> > instead tests which just didn't work.
> 
> Right, we get the same on Ubuntu CI. Over time the failures pile up, I
> figure it's quite difficult for the Perl team to look at/fix all
> these.
> 
> > Please could adt-run have an option to exit without running any tests
> > if the source package in question doesn't define any?
> 
> So to clarify, you want an option to disable autodep8, right? If a
> source package doesn't have tests and autodep8 does not generate any
> either, then adt-run will already report "no tests in this package"
> and exit with 8. With a perl package and --no-autodep8 it would then
> do the same.

I thought about 'no autodep8', but the semantics aren't quite correct,
since a package could declare Testsuite: autopkgtest, but rely on the
autodep8 behaviour. This is in fact the common case for pkg-perl packages,
as far as I can tell.

So by 'declare tests', I was really talking about having
Testsuite: autopkgest in the control file. Sorry if that wasn't clear.

Cheers,
Dominic.