Bug#1024971: pybuild: should fail when the result of running tests is "Ran 0 tests in 0.000s"

[Stefano Rivera]

Hi Louis-Philippe (2022.11.28_01:46:58_+0200)

Too often, a mistake or a misconfiguration leads to no tests being detected
when trying to run the upstream testsuite.
When this happens, the result of the test command typically looks like "Ran
0 tests in 0.000s".

A couple of years later:

We've implemented the feature in Python 3.12, unittest's runner now
exits with return value 5 if no tests were discovered, like pytest does.
https://github.com/python/cpython/issues/113661

We initially ignored this exit status in dh-python, to allow us to
transition to Python 3.12, without too much pain.

I've just done a rebuild test without ignoring it to see how much effort
it would take to be able to use this feature:

I built 6440 packages build-depending on dh-python in one way or
another. 1483 failed, and 1124 of them say "NO TESTS RAN" in the logs.

To fix these build failures, package maintainers would have these
options:

1. Get the build to run some unit tests (assuming they exist),
2. override_dh_auto_test with something noop,
3. export PYBUILD_DISABLE=test,
4. We could make this failure opt-in in dh-python. Maybe via an explicit
--test-unittest option that selects the unittest runner. If you don't
explicitly select this runner, you'd get an attempt to run tests by
with unittest, and no failure if no tests are found.

The downside of options 2 and 3 is that if the package does start
gaining an upstream test suite, nothing ever will attempt to run it. I
think that's OK?

If you want to experiment yourself, there's a version of dh-python in
experimental that will treat no tests as an error.

Should we file bugs for these packages? Or implement option 4?

Attached is a current dd-list.

Stefano

[Scott Kitterman]

Post by Stefano Rivera
Hi Louis-Philippe (2022.11.28_01:46:58_+0200)

Too often, a mistake or a misconfiguration leads to no tests being detected
when trying to run the upstream testsuite.
When this happens, the result of the test command typically looks like "Ran
0 tests in 0.000s".

We've implemented the feature in Python 3.12, unittest's runner now
exits with return value 5 if no tests were discovered, like pytest does.
https://github.com/python/cpython/issues/113661
We initially ignored this exit status in dh-python, to allow us to
transition to Python 3.12, without too much pain.
I've just done a rebuild test without ignoring it to see how much effort
I built 6440 packages build-depending on dh-python in one way or
another. 1483 failed, and 1124 of them say "NO TESTS RAN" in the logs.
To fix these build failures, package maintainers would have these
1. Get the build to run some unit tests (assuming they exist),
2. override_dh_auto_test with something noop,
3. export PYBUILD_DISABLE=test,
4. We could make this failure opt-in in dh-python. Maybe via an explicit
--test-unittest option that selects the unittest runner. If you don't
explicitly select this runner, you'd get an attempt to run tests by
with unittest, and no failure if no tests are found.
The downside of options 2 and 3 is that if the package does start
gaining an upstream test suite, nothing ever will attempt to run it. I
think that's OK?
If you want to experiment yourself, there's a version of dh-python in
experimental that will treat no tests as an error.
Should we file bugs for these packages? Or implement option 4?
Attached is a current dd-list.

Thanks. I think some variation of #4 is the right answer. Causing a thousand packages to FTBFS isn't great. I would find it easier to have an environment variable (similar to what you suggested for #3) instead of adding an override, but that's more of an implementation detail.

Scott K

[Louis-Philippe Véronneau]

Post by Scott Kitterman
Thanks. I think some variation of #4 is the right answer. Causing a thousand packages to FTBFS isn't great. I would find it easier to have an environment variable (similar to what you suggested for #3) instead of adding an override, but that's more of an implementation detail.
Scott K

Having an opt-in environment variable means all the packages using
unittest will eventually need to add it.

What is the end goal? Once we have enough buy-in we flip it around?

I agree breaking tons of packages isn't great, but I don't want us to
haul crud forever either :(

[Julian Gilbey]

Post by Stefano Rivera
[...]
We've implemented the feature in Python 3.12, unittest's runner now
exits with return value 5 if no tests were discovered, like pytest does.
https://github.com/python/cpython/issues/113661
[...]
I built 6440 packages build-depending on dh-python in one way or
another. 1483 failed, and 1124 of them say "NO TESTS RAN" in the logs.

My guess is that most of these 1124 have no tests at all, rather than
having a misconfigured setup. A unittest is the pybuild default test
framework, unittest is used and fails to find any tests, hence all of
these failures.

Post by Stefano Rivera
To fix these build failures, package maintainers would have these
1. Get the build to run some unit tests (assuming they exist),
2. override_dh_auto_test with something noop,
3. export PYBUILD_DISABLE=test,
4. We could make this failure opt-in in dh-python. Maybe via an explicit
--test-unittest option that selects the unittest runner. If you don't
explicitly select this runner, you'd get an attempt to run tests by
with unittest, and no failure if no tests are found.

I like option 4 for the above reason. But implementing this would
mean that all of the packages that currently *do* use unittest
(intentionally, but without having to code it explicitly as it's the
default) would suddenly not have any tests running until they
proactively add --test-unittest or set PYBUILD_TEST_UNITTEST = 1 or
similar. This seems like an unfortunate consequence.

Is there a way of looking at the logs of the packages that passed the
build to identify which ones successfully passed tests using unittest?
There is also the issue of packages that use unittest (as the default)
via autopkgtest-pkg-pybuild but override dh_auto_test during the
build, though that will be much rarer.

Best wishes,

Julian

[Stefano Rivera]

Re-sent, with the bug in CC

Hi Julian (2024.09.08_21:33:49_+0000)

Post by Julian Gilbey

Post by Stefano Rivera
I built 6440 packages build-depending on dh-python in one way or
another. 1483 failed, and 1124 of them say "NO TESTS RAN" in the logs.

My guess is that most of these 1124 have no tests at all, rather than
having a misconfigured setup. A unittest is the pybuild default test
framework, unittest is used and fails to find any tests, hence all of
these failures.

Yes, almost certainly.

Post by Julian Gilbey

Post by Stefano Rivera
4. We could make this failure opt-in in dh-python. Maybe via an explicit
--test-unittest option that selects the unittest runner. If you don't
explicitly select this runner, you'd get an attempt to run tests by
with unittest, and no failure if no tests are found.

I like option 4 for the above reason. But implementing this would
mean that all of the packages that currently *do* use unittest
(intentionally, but without having to code it explicitly as it's the
default) would suddenly not have any tests running until they
proactively add --test-unittest or set PYBUILD_TEST_UNITTEST = 1 or
similar. This seems like an unfortunate consequence.

I would leave unittest as the default runner, but without missing test
detection.

That's a slightly unexpected behaviour, but it makes the default case
work.

Downside is that you have to opt-in to missing test detection. Maybe we
can have a lintian tag for that?

Stefano

[Julian Gilbey]

Post by Stefano Rivera

Post by Stefano Rivera
Hi Julian (2024.09.08_21:33:49_+0000)

Post by Stefano Rivera
I built 6440 packages build-depending on dh-python in one way or
another. 1483 failed, and 1124 of them say "NO TESTS RAN" in the logs.

My guess is that most of these 1124 have no tests at all, rather than
having a misconfigured setup. A unittest is the pybuild default test
framework, unittest is used and fails to find any tests, hence all of
these failures.

Yes, almost certainly.
[...]
I would leave unittest as the default runner, but without missing test
detection.
That's a slightly unexpected behaviour, but it makes the default case
work.
Downside is that you have to opt-in to missing test detection. Maybe we
can have a lintian tag for that?

That seems a bit heavy to ask for.

Is there any way of identifying those packages that do genuinely use
unittest? If there are not that many of them, then implementing a
--test-unittest option would be a good way to go. I would imagine the
following timeline:

(1) --test-unittest is introduced as an option to explicitly select
unittest as the test framework. When --test-unittest is specified,
the test will fail if no tests are found. unittest is still used as a
fallback test framework; in this case, the dh_auto_test call will
succeed if no tests are run.

(2) Add some sort of warning for pybuild-using packages that run
dh_auto_test but haven't specified a test framework and for which
autodetection of the test framework fails. If there aren't any tests
to run, an empty override_dh_auto_test target should be specified.

(3) Stop using unittest as the default test framework, and fail if no
test framework has been specified or autodetected.

But that might be overkill for something which may not actually be
much of a problem.

Best wishes,

Julian

[Stefano Rivera]

Hi Julian (2024.09.09_15:19:51_+0000)

Post by Julian Gilbey
That seems a bit heavy to ask for.
Is there any way of identifying those packages that do genuinely use
unittest?

From 6438 build logs:
- 651 don't call dh_auto_test
- 2180 do something custom
- 1989 use pytest
- 25 use nose
- 18 use nose2
- 23 use tox
- 3 use stestr
- 1561 packages use pybuild's unittest runner
* 391 pass
* 1170 fail
+ 1139 NO TESTS RAN
+ 33 the test suite failed

(numbers don't quite add up, because this was a lot of grep | wc -l)

Post by Julian Gilbey
If there are not that many of them, then implementing a
--test-unittest option would be a good way to go. I would imagine the
(1) --test-unittest is introduced as an option to explicitly select
unittest as the test framework. When --test-unittest is specified,
the test will fail if no tests are found. unittest is still used as a
fallback test framework; in this case, the dh_auto_test call will
succeed if no tests are run.
(2) Add some sort of warning for pybuild-using packages that run
dh_auto_test but haven't specified a test framework and for which
autodetection of the test framework fails. If there aren't any tests
to run, an empty override_dh_auto_test target should be specified.
(3) Stop using unittest as the default test framework, and fail if no
test framework has been specified or autodetected.
But that might be overkill for something which may not actually be
much of a problem.

Yeah, that can work. We can also just abort after step 2.

Stefano

[Alexandre Detiste]

Hi,

From my personal experience, the most common cause of missing tests

is because d/watch follow pypi where the tarball is incomplete
and the typical fix is to switch to github tarball.

Could this "1139 NO TESTS RAN" correlated with d/watch ?

- 25 use nose

Even less as of today ;-)

Greetings

Hi Julian (2024.09.09_15:19:51_+0000)

Post by Julian Gilbey
That seems a bit heavy to ask for.
Is there any way of identifying those packages that do genuinely use
unittest?

- 651 don't call dh_auto_test
- 2180 do something custom
- 1989 use pytest
- 25 use nose
- 18 use nose2
- 23 use tox
- 3 use stestr
- 1561 packages use pybuild's unittest runner
* 391 pass
* 1170 fail
+ 1139 NO TESTS RAN
+ 33 the test suite failed
(numbers don't quite add up, because this was a lot of grep | wc -l)

[Stefano Rivera]

Hi Alexandre (2024.09.12_14:26:13_+0000)

Post by Alexandre Detiste

From my personal experience, the most common cause of missing tests

is because d/watch follow pypi where the tarball is incomplete
and the typical fix is to switch to github tarball.
Could this "1139 NO TESTS RAN" correlated with d/watch ?

It'll be part of them, yes.

Stefano

[Stefano Rivera]

Hi Julian (2024.09.08_21:33:49_+0000)

Post by Julian Gilbey

Post by Stefano Rivera
I built 6440 packages build-depending on dh-python in one way or
another. 1483 failed, and 1124 of them say "NO TESTS RAN" in the logs.

My guess is that most of these 1124 have no tests at all, rather than
having a misconfigured setup. A unittest is the pybuild default test
framework, unittest is used and fails to find any tests, hence all of
these failures.

Yes, almost certainly.

Post by Julian Gilbey

Post by Stefano Rivera
4. We could make this failure opt-in in dh-python. Maybe via an explicit
--test-unittest option that selects the unittest runner. If you don't
explicitly select this runner, you'd get an attempt to run tests by
with unittest, and no failure if no tests are found.

I like option 4 for the above reason. But implementing this would
mean that all of the packages that currently *do* use unittest
(intentionally, but without having to code it explicitly as it's the
default) would suddenly not have any tests running until they
proactively add --test-unittest or set PYBUILD_TEST_UNITTEST = 1 or
similar. This seems like an unfortunate consequence.

I would leave unittest as the default runner, but without missing test
detection.

That's a slightly unexpected behaviour, but it makes the default case
work.

Downside is that you have to opt-in to missing test detection. Maybe we
can have a lintian tag for that?

Stefano

[Timo Röhling]

Hi Stefano,

Post by Stefano Rivera
To fix these build failures, package maintainers would have these
1. Get the build to run some unit tests (assuming they exist),
2. override_dh_auto_test with something noop,
3. export PYBUILD_DISABLE=test,
4. We could make this failure opt-in in dh-python. Maybe via an explicit
--test-unittest option that selects the unittest runner. If you don't
explicitly select this runner, you'd get an attempt to run tests by
with unittest, and no failure if no tests are found.

Maybe we can start by making missing tests in autopkgtest fail?
As you need to opt-in via "Testsuite: autopkgtest-pkg-pybuild", we
can be quite certain that missing tests are not intentional.

I think that would work very nicely together with option 4 for the
build itself. You might even use the "Testsuite" setting as opt-in
flag, but that is probably taking it too far.


Cheers
Timo