Policy Change Proposal: Running the upstream test suite

[Louis-Philippe Véronneau]

Hello,

As discussed during the DebConf24 Python BoF, I'm submitting this change
to the policy to require the use of the upstream test suite, both during
the build process and as an autopkgtest.

You can find the MR here:

https://salsa.debian.org/python-team/tools/python-modules/-/merge_requests/24

People present at the BoF seemed to think this was a good idea. If you
don't please reply to this message and make yourself heard :)

Cheers,

[PICCA Frederic-Emmanuel]

I have one concerne when using pybuild autopkgtest.

It install by default the build dependencies. which is not what peoples are doing when they install the packages.
It should be possible to define the autopkgtest dependencies. This way we could catch missing dependencies in python-<modules> dependencies.

Is it possible ?


Cheers

Frederic

[Martin]

Quoting PICCA Frederic-Emmanuel

Post by PICCA Frederic-Emmanuel
It install by default the build dependencies. which is not what
peoples are doing when they install the packages.
It should be possible to define the autopkgtest dependencies. This
way we could catch missing dependencies in python-<modules>
dependencies.
Is it possible ?

I believe (didn't check so many packages), in most cases
installing the build dependencies is the better option:

1. There might be testing frameworks to install as b-d, but
not as Depend.

2. Many unit test suites check various variants and optional
behaviour, that might be separated in different binary
packages (think of myapp-postgres and myapp-mariadb, etc.)

Maybe as an option for some packages?

Or a different kind of test, running on top of piuparts?

[PICCA Frederic-Emmanuel]

My use case, is to check that all the Dependencies computer by dh_python3 from the build tools are indeed listed in the Depends of the binary package.

I think about package whcih provide oiptional dependencies via the extra. In that case the extra should be declare when calling dh_python3.

So I must install only the python3-<module> and the test framework to be sure that it is fonctionnal as installed.

It seems to me that the autopkgtest-pybuild plugin is not well suited for this purpose. (as it is now).

Cheers

Frederic

For piupart, I alsao have some test that should be perfore with one of my package

the upgrade of a database.

I should install the package, upgrade it and check that the new version has the right database upgraded.

Is it possible to test upgrade scenario with autopkgtests ?

[Andrey Rakhmatullin]

Post by PICCA Frederic-Emmanuel
My use case, is to check that all the Dependencies computer by dh_python3 from the build tools are indeed listed in the Depends of the binary package.

Maybe we indeed want a "minimal" autopkgtest environment, but many
upstream tests will fail in those and I don't see an automatic way to test
a random package in this way.

[PICCA Frederic-Emmanuel]

Post by Andrey Rakhmatullin
Maybe we indeed want a "minimal" autopkgtest environment, but many
upstream tests will fail in those and I don't see an automatic way to test
a random package in this way.

Even if not minimal, at least correspond to the upstream declares dependencies.

by 'declare' I am not even sure of the meaning.

dependencies of the test can be different from the required installed ones.

Maybe we should install only the python binaries and the dependencies marked <!nocheck>.

Is there a standard in the Python community for the test dependencies ?

[Scott Kitterman]

Post by PICCA Frederic-Emmanuel

Post by Andrey Rakhmatullin
Maybe we indeed want a "minimal" autopkgtest environment, but many
upstream tests will fail in those and I don't see an automatic way to test
a random package in this way.

Even if not minimal, at least correspond to the upstream declares dependencies.
by 'declare' I am not even sure of the meaning.
dependencies of the test can be different from the required installed ones.
Maybe we should install only the python binaries and the dependencies marked <!nocheck>.
Is there a standard in the Python community for the test dependencies ?

As usual, there are several.

Scott K

[Andrey Rakhmatullin]

Post by PICCA Frederic-Emmanuel

Post by Andrey Rakhmatullin
Maybe we indeed want a "minimal" autopkgtest environment, but many
upstream tests will fail in those and I don't see an automatic way to test
a random package in this way.

Even if not minimal, at least correspond to the upstream declares dependencies.

Those will correspond to build dependencies in a typical simple case, as
the upstream tests need all those optional deps, assuming you mean the
tests dependencies (and not the runtime dependencies which should already
be in Depends).

Post by PICCA Frederic-Emmanuel
Maybe we should install only the python binaries and the dependencies marked <!nocheck>.

In a typical simple case all B-Ds except sphinx stuff will be <!nocheck>
as you don't need anything beyond the build system to "build" a pure
Python module.

So this is exactly what you wanted to avoid.

[PICCA Frederic-Emmanuel]

Post by Andrey Rakhmatullin
Those will correspond to build dependencies in a typical simple case, as
the upstream tests need all those optional deps, assuming you mean the
tests dependencies (and not the runtime dependencies which should already
be in Depends).

Post by PICCA Frederic-Emmanuel
Maybe we should install only the python binaries and the dependencies marked <!nocheck>.

In a typical simple case all B-Ds except sphinx stuff will be <!nocheck>
as you don't need anything beyond the build system to "build" a pure
Python module.
So this is exactly what you wanted to avoid.

Yes, I want to be sure that all the runtime dependencies are rightfully declared in the Depends of the python module package.

sometimes the upstream forgot about dependencies, or mark them as optional, but they are not when running the tests...

Is it possible to achieve this automatically ? lintian ? dh-python ? python3-stdeb ?

Fred.

PS: In haskell we have a cabal-debian package which compute automatically the list of the Dependencies, from the cabal file, then it update the control file to reflect the new status of the package.

[Andrey Rakhmatullin]

Post by PICCA Frederic-Emmanuel
Yes, I want to be sure that all the runtime dependencies are rightfully declared in the Depends of the python module package.

Depending on what do you mean by "all" this is not always what we want. We
often don't put all modules imported by all parts of the shipped code into
Depends, instead using Recommends, Suggests or nothing.

Post by PICCA Frederic-Emmanuel
sometimes the upstream forgot about dependencies, or mark them as optional, but they are not when running the tests...

Similarly, all modules installed for tests shouldn't always go into
Depends ("all modules installed for tests" is ideally the same as "all
modules imported by all parts of the shipped code", plus test system
deps, and minus deps for e.g. Windows code that we ship but don't run or
test).

Post by PICCA Frederic-Emmanuel
Is it possible to achieve this automatically ? lintian ? dh-python ? python3-stdeb ?

Well our tools can look at the deps declared by the upstream, and normally
do that as a part of the build process. But that won't help when those
deps are not what we want.

[Julian Gilbey]

Post by Andrey Rakhmatullin

Post by PICCA Frederic-Emmanuel
Maybe we should install only the python binaries and the dependencies marked <!nocheck>.

In a typical simple case all B-Ds except sphinx stuff will be <!nocheck>
as you don't need anything beyond the build system to "build" a pure
Python module.

Erm, no, you need to B-D on all runtime dependencies so they are
correctly included in the computed python3:Depends substvar, unless
you list these runtime dependencies explictly by hand.

Best wishes,

Julian

[Andrey Rakhmatullin]

Post by Julian Gilbey

Post by Andrey Rakhmatullin

Post by PICCA Frederic-Emmanuel
Maybe we should install only the python binaries and the dependencies marked <!nocheck>.

In a typical simple case all B-Ds except sphinx stuff will be <!nocheck>
as you don't need anything beyond the build system to "build" a pure
Python module.

Erm, no, you need to B-D on all runtime dependencies so they are
correctly included in the computed python3:Depends substvar, unless
you list these runtime dependencies explictly by hand.

Those will be installed anyway via Depends, by definition, but also in my
experience you don't always need to do one of these two things. Maybe I'm
doing it wrong and my python3:Depends are not 100% correct though.

[Simon McVittie]

Post by Andrey Rakhmatullin

Post by PICCA Frederic-Emmanuel
My use case, is to check that all the Dependencies computer by dh_python3 from the build tools are indeed listed in the Depends of the binary package.

Maybe we indeed want a "minimal" autopkgtest environment

For a simple smoke-test that the Python packages import succesfully, isn't
this exactly autopkgtest-pkg-python?

For thorough test coverage, I don't think this is possible: in general,
the test suite will legitimately require installation of packages that are
only needed during testing (pytest or similar), or packages that are
optional for basic use of the code but mandatory for full test coverage
(those might be Recommends or Suggests).

Post by Andrey Rakhmatullin
but many
upstream tests will fail in those and I don't see an automatic way to test
a random package in this way

If we want to run upstream test-suites as autopkgtests, then I think
ideally we want the same test coverage as autopkgtest-pkg-python,
*and* the same test coverage as autopkgtest-pkg-pybuild, as separate
autopkgtest test-cases with different dependencies.

But I don't think we can easily have that, because the Testsuite field
only takes one value.

Would it be possible to make autopkgtest-pkg-pybuild absorb
the functionality of autopkgtest-pkg-python, so that it
generates a debian/control that does some simple smoke-tests (like
autopkgtest-pkg-python does), *and* runs the upstream test suite (like
autopkgtest-pkg-pybuild currently does)?

If not, then the proposed policy change would result in us gaining
coverage but also losing coverage.

smcv

[Andrey Rakhmatullin]

Post by Simon McVittie

Post by Andrey Rakhmatullin

Post by PICCA Frederic-Emmanuel
My use case, is to check that all the Dependencies computer by dh_python3 from the build tools are indeed listed in the Depends of the binary package.

Maybe we indeed want a "minimal" autopkgtest environment

For a simple smoke-test that the Python packages import succesfully, isn't
this exactly autopkgtest-pkg-python?

Yes.

Post by Simon McVittie
For thorough test coverage, I don't think this is possible: in general,
the test suite will legitimately require installation of packages that are
only needed during testing (pytest or similar), or packages that are
optional for basic use of the code but mandatory for full test coverage
(those might be Recommends or Suggests).

I was thinking about "running a subset of the test suite that doesn't need
those optional deps" (I didn't consider that this needs pytest and likely
various pytest plugins and their deps so it isn't strictly minimal) and
automatically identifying this subset is what I later said I think isn't
possible.

Post by Simon McVittie

Post by Andrey Rakhmatullin
but many
upstream tests will fail in those and I don't see an automatic way to test
a random package in this way

If we want to run upstream test-suites as autopkgtests, then I think
ideally we want the same test coverage as autopkgtest-pkg-python,
*and* the same test coverage as autopkgtest-pkg-pybuild, as separate
autopkgtest test-cases with different dependencies.

I always thought the smoke test is redundant if there is the upstream
tests one.
What do you think?

Post by Simon McVittie
But I don't think we can easily have that, because the Testsuite field
only takes one value.

It was easy (and achieved by accident) before autopkgtest-pkg-pybuild: you
could have upstream tests in debian/tests and the smoke test via Testsuite
or automatic discovery (I could never remember which combination gives you
both). But, again, I never thought this is desirable and was removing the
smoke test.

[Simon McVittie]

Post by Andrey Rakhmatullin

Post by Simon McVittie
If we want to run upstream test-suites as autopkgtests, then I think
ideally we want the same test coverage as autopkgtest-pkg-python,
*and* the same test coverage as autopkgtest-pkg-pybuild, as separate
autopkgtest test-cases with different dependencies.

I always thought the smoke test is redundant if there is the upstream
tests one.
What do you think?

I think it is not redundant, because it's a quick way to catch packaging
errors where the python3-foo binary package is missing a mandatory
dependency - which seems to be a somewhat common failure mode, because
the maintainer (who presumably has the Build-Depends installed) will
never notice it by testing on their own system.

For instance in this (hypothetical) package:

Source: python-foo
Build-Depends: dh-python, python3-all, python3-bar, ...

Package: python3-foo
Depends: ${misc:Depends}, ${python3:Depends}

Suppose python3-foo does an "import bar", therefore there should have
been a "Depends: python3-bar" which is missing here. The smoke test
would detect that.

(I know that ideally python3-bar would be part of the ${python3:Depends},
and with some upstream build systems under some phases of the moon,
it is; but I'm thinking here about the packages where that information
doesn't work or isn't propagated.)

I'm a fan of putting a small amount of effort into automated tests, and
then having running the tests before uploading save you from a class of
Severity: grave bugs forever :-)

Post by Andrey Rakhmatullin

Post by Simon McVittie
But I don't think we can easily have that, because the Testsuite field
only takes one value.

Actually it seems it's meant to be able to take more than one value, and
the fact that autodep8 doesn't support that appears to be considered a bug.
(#1042717, #1061620, #1077645)

smcv

[Scott Kitterman]

Post by Louis-Philippe Véronneau
Hello,
As discussed during the DebConf24 Python BoF, I'm submitting this change to the policy to require the use of the upstream test suite, both during the build process and as an autopkgtest.
https://salsa.debian.org/python-team/tools/python-modules/-/merge_requests/24
People present at the BoF seemed to think this was a good idea. If you don't please reply to this message and make yourself heard :)

I understand the theory and why it's a good idea to run the test suite. I don't think it ought to be a hard requirement. I have several packages where there's a test suite, but I don't run it:

1. The largest set is packages that need test only dependencies which are not packaged. When I am packaging something new which has a test suite, then I generally package any needed test depends. If those test depends also need test depends packaged, I generally stop and don't enable tests for things that are only in the archives to support tests. Noseofyeti is an example of this.

2. There's at least one case where Debian has customizations that cause the test suite to fail, but the failures don't seem to cause any real problems. If anyone wants to make it so the weasyprint test suite works on Debian, please knock yourself out.

3. I also maintain multiple packages which require network access to run their test suite, so they can't run tests during build, only autopkgtests.

While I agree that running tests is good, it's not a universally reasonable requirement.

If we add this as a hard requirement for team packages, what do people think will happen? Will people invest more volunteer time that they otherwise wouldn't have to add running tests? Will they leave it for someone else on the team to address? Will they remove packages from team maintenance?

Python policy is only enforceable within the team. For the rest of the archive, it's a statement of best practices for people to consider. Making this a hard requirement for team packages is a statement that we would rather such packages not be team maintained. I don't think that's a good idea.

Scott K

[Simon McVittie]

Post by Scott Kitterman
While I agree that running tests is good, it's not a universally reasonable requirement.

I agree. In a project as large as Debian, most requirements similar to
this one at least need the qualifier "... unless there is a reason why
we can't".

Post by Scott Kitterman
2. There's at least one case where Debian has customizations that
cause the test suite to fail, but the failures don't seem to cause any
real problems. If anyone wants to make it so the weasyprint test suite
works on Debian, please knock yourself out.

This seems like the situation is:

- a build-time test or autopkgtest test failure would be a
release-critical bug

- upstream tests failing always indicate a bug, but that bug might not
be RC:
- the tests might be broken, flaky or make assumptions that aren't
true in Debian (the tests have a bug, the package under test does not)
- or the tests might be showing us a genuine bug in the package
under test, but its severity might be anywhere between wishlist
and critical

- in the case of this specific package, Scott has assessed the severity of
the bug that is implied by the tests failing, and has decided that it's
Severity: normal or less

and that seems fine.

Ideally there would be a bug open in the BTS to document this limitation,
possibly only wishlist, possibly tagged help.

If part of the test suite is fine but a different part of the test suite
is problematic, I'd personally prefer to flag the problematic parts as
skipped/ignored, and run the rest (for example src:python3-mpd skips some
of its tests like this); but it's reasonable for opinions on this to vary,
and I don't know whether there would be anything left in this particular
package's test suite after skipping the problematic parts.

Post by Scott Kitterman
3. I also maintain multiple packages which require network access
to run their test suite, so they can't run tests during build, only
autopkgtests.

Running the test suite for these would be a Debian Policy violation
(usually a RC bug), and if Python team policy conflicts with Debian Policy,
it's Debian Policy that wins.

Post by Scott Kitterman
If we add this as a hard requirement for team packages, what do people
think will happen? Will people invest more volunteer time that they
otherwise wouldn't have to add running tests? Will they leave it for
someone else on the team to address? Will they remove packages from
team maintenance?

These are all valid questions. It's very easy to create new QA
requirements that are simple to achieve for simple packages, but for the
packages where they are not so simple, someone who feels that a particular
level of QA coverage is valuable will need to step up to do that work,
and threatening contributors with sanctions if they don't do particular
work ("we'll throw you out of the team" or "we'll delete the package
you've invested time in from Debian" or similar) should be rare and a
last resort.

We often say that Debian Policy is (and should be) a tool for getting
a better distribution, and not a stick to beat contributors with. I
think the same should be equally true for individual teams' self-imposed
policies.

smcv

[Julian Gilbey]

[...]

Here's another case: the package has to be fully installed for the
test suite to work. So the tests can't be run at build time, only at
autopkgtest time.

Best wishes,

Julian

[Louis-Philippe Véronneau]

Post by Scott Kitterman

Post by Louis-Philippe Véronneau
Hello,
As discussed during the DebConf24 Python BoF, I'm submitting this change to the policy to require the use of the upstream test suite, both during the build process and as an autopkgtest.
https://salsa.debian.org/python-team/tools/python-modules/-/merge_requests/24
People present at the BoF seemed to think this was a good idea. If you don't please reply to this message and make yourself heard :)

1. The largest set is packages that need test only dependencies which are not packaged. When I am packaging something new which has a test suite, then I generally package any needed test depends. If those test depends also need test depends packaged, I generally stop and don't enable tests for things that are only in the archives to support tests. Noseofyeti is an example of this.

That sounds like a valid technical reason not to run the tests to me :)

Post by Scott Kitterman
2. There's at least one case where Debian has customizations that cause the test suite to fail, but the failures don't seem to cause any real problems. If anyone wants to make it so the weasyprint test suite works on Debian, please knock yourself out.

Again, as long as you document that, I don't think it would go against
the proposed policy change.

Post by Scott Kitterman
3. I also maintain multiple packages which require network access to run their test suite, so they can't run tests during build, only autopkgtests.

Same.

[Scott Kitterman]

Post by Louis-Philippe Véronneau

Post by Scott Kitterman

Post by Louis-Philippe Véronneau
Hello,
As discussed during the DebConf24 Python BoF, I'm submitting this change to the policy to require the use of the upstream test suite, both during the build process and as an autopkgtest.
https://salsa.debian.org/python-team/tools/python-modules/-/merge_requests/24
People present at the BoF seemed to think this was a good idea. If you don't please reply to this message and make yourself heard :)

1. The largest set is packages that need test only dependencies which are not packaged. When I am packaging something new which has a test suite, then I generally package any needed test depends. If those test depends also need test depends packaged, I generally stop and don't enable tests for things that are only in the archives to support tests. Noseofyeti is an example of this.

That sounds like a valid technical reason not to run the tests to me :)

Post by Scott Kitterman
2. There's at least one case where Debian has customizations that cause the test suite to fail, but the failures don't seem to cause any real problems. If anyone wants to make it so the weasyprint test suite works on Debian, please knock yourself out.

Again, as long as you document that, I don't think it would go against the proposed policy change.

Post by Scott Kitterman
3. I also maintain multiple packages which require network access to run their test suite, so they can't run tests during build, only autopkgtests.

Same.

Except for #3, I don't get that from the wording in the MR. I don't think not worth the trouble is a technical reason. I think the real rule that's being proposed is that packages must run the test suit or document why not. I don't have a problem with that, but I don't think that's what it actually says now. I think if you were to change must to should and strike the word technical before reason, it would accomplish the same thing and be clearer. I could support that.

Scott K

[Louis-Philippe Véronneau]

Post by Scott Kitterman

Post by Louis-Philippe Véronneau

Post by Scott Kitterman

Post by Louis-Philippe Véronneau
Hello,
As discussed during the DebConf24 Python BoF, I'm submitting this change to the policy to require the use of the upstream test suite, both during the build process and as an autopkgtest.
https://salsa.debian.org/python-team/tools/python-modules/-/merge_requests/24
People present at the BoF seemed to think this was a good idea. If you don't please reply to this message and make yourself heard :)

1. The largest set is packages that need test only dependencies which are not packaged. When I am packaging something new which has a test suite, then I generally package any needed test depends. If those test depends also need test depends packaged, I generally stop and don't enable tests for things that are only in the archives to support tests. Noseofyeti is an example of this.

That sounds like a valid technical reason not to run the tests to me :)

Post by Scott Kitterman
2. There's at least one case where Debian has customizations that cause the test suite to fail, but the failures don't seem to cause any real problems. If anyone wants to make it so the weasyprint test suite works on Debian, please knock yourself out.

Again, as long as you document that, I don't think it would go against the proposed policy change.

Post by Scott Kitterman
3. I also maintain multiple packages which require network access to run their test suite, so they can't run tests during build, only autopkgtests.

Same.

Except for #3, I don't get that from the wording in the MR. I don't think not worth the trouble is a technical reason. I think the real rule that's being proposed is that packages must run the test suit or document why not. I don't have a problem with that, but I don't think that's what it actually says now. I think if you were to change must to should and strike the word technical before reason, it would accomplish the same thing and be clearer. I could support that.

Language is hard and I'm not a native English speaker :)

What I want to prevent is people not running tests because they don't
feel like it, even though it would not take them a large amount of
efforts to do so.

I'll strike "technical", as it seems others also interpreted this word
they way you have.

As for "MUST" vs "SHOULD", I believe the exception clause provides
enough leeway to justify a reasonable way out in case of $VALID_REASON.

"SHOULD" is not particularly strong and again, I fear it would let
people get away with not running tests although it wouldn't be much work
to do so...

[Scott Kitterman]

Post by Louis-Philippe Véronneau

Post by Scott Kitterman

Post by Louis-Philippe Véronneau

Post by Scott Kitterman

Post by Louis-Philippe Véronneau
Hello,
As discussed during the DebConf24 Python BoF, I'm submitting this change to the policy to require the use of the upstream test suite, both during the build process and as an autopkgtest.
https://salsa.debian.org/python-team/tools/python-modules/-/merge_requests/24
People present at the BoF seemed to think this was a good idea. If you don't please reply to this message and make yourself heard :)

1. The largest set is packages that need test only dependencies which are not packaged. When I am packaging something new which has a test suite, then I generally package any needed test depends. If those test depends also need test depends packaged, I generally stop and don't enable tests for things that are only in the archives to support tests. Noseofyeti is an example of this.

That sounds like a valid technical reason not to run the tests to me :)

Post by Scott Kitterman
2. There's at least one case where Debian has customizations that cause the test suite to fail, but the failures don't seem to cause any real problems. If anyone wants to make it so the weasyprint test suite works on Debian, please knock yourself out.

Again, as long as you document that, I don't think it would go against the proposed policy change.

Post by Scott Kitterman
3. I also maintain multiple packages which require network access to run their test suite, so they can't run tests during build, only autopkgtests.

Same.

Except for #3, I don't get that from the wording in the MR. I don't think not worth the trouble is a technical reason. I think the real rule that's being proposed is that packages must run the test suit or document why not. I don't have a problem with that, but I don't think that's what it actually says now. I think if you were to change must to should and strike the word technical before reason, it would accomplish the same thing and be clearer. I could support that.

Language is hard and I'm not a native English speaker :)
What I want to prevent is people not running tests because they don't feel like it, even though it would not take them a large amount of efforts to do so.
I'll strike "technical", as it seems others also interpreted this word they way you have.
As for "MUST" vs "SHOULD", I believe the exception clause provides enough leeway to justify a reasonable way out in case of $VALID_REASON.
"SHOULD" is not particularly strong and again, I fear it would let people get away with not running tests although it wouldn't be much work to do so...

I guess it depends on how you look at the word. In the context of technical specifications, I tend to think of terms as defined in RFC 2119 [1]. I think that's pretty close to what you are suggesting:

3. SHOULD This word, or the adjective "RECOMMENDED", mean that there
may exist valid reasons in particular circumstances to ignore a
particular item, but the full implications must be understood and
carefully weighed before choosing a different course.

The way RFC 2119 defines MUST doesn't leave much room for exceptions. If you want to be clear, I suggest SHOULD with a reference to RFC 2119. While not universal, it is a widely used reference for definitions of these terms in technical specifications.

Scott K

[1] https://datatracker.ietf.org/doc/html/rfc2119#section-3

[Louis-Philippe Véronneau]

Post by Scott Kitterman

Post by Louis-Philippe Véronneau

Post by Scott Kitterman

Post by Louis-Philippe Véronneau

Post by Scott Kitterman

Post by Louis-Philippe Véronneau
Hello,
As discussed during the DebConf24 Python BoF, I'm submitting this change to the policy to require the use of the upstream test suite, both during the build process and as an autopkgtest.
https://salsa.debian.org/python-team/tools/python-modules/-/merge_requests/24
People present at the BoF seemed to think this was a good idea. If you don't please reply to this message and make yourself heard :)

1. The largest set is packages that need test only dependencies which are not packaged. When I am packaging something new which has a test suite, then I generally package any needed test depends. If those test depends also need test depends packaged, I generally stop and don't enable tests for things that are only in the archives to support tests. Noseofyeti is an example of this.

That sounds like a valid technical reason not to run the tests to me :)

Post by Scott Kitterman
2. There's at least one case where Debian has customizations that cause the test suite to fail, but the failures don't seem to cause any real problems. If anyone wants to make it so the weasyprint test suite works on Debian, please knock yourself out.

Again, as long as you document that, I don't think it would go against the proposed policy change.

Post by Scott Kitterman
3. I also maintain multiple packages which require network access to run their test suite, so they can't run tests during build, only autopkgtests.

Same.

Except for #3, I don't get that from the wording in the MR. I don't think not worth the trouble is a technical reason. I think the real rule that's being proposed is that packages must run the test suit or document why not. I don't have a problem with that, but I don't think that's what it actually says now. I think if you were to change must to should and strike the word technical before reason, it would accomplish the same thing and be clearer. I could support that.

Language is hard and I'm not a native English speaker :)
What I want to prevent is people not running tests because they don't feel like it, even though it would not take them a large amount of efforts to do so.
I'll strike "technical", as it seems others also interpreted this word they way you have.
As for "MUST" vs "SHOULD", I believe the exception clause provides enough leeway to justify a reasonable way out in case of $VALID_REASON.
"SHOULD" is not particularly strong and again, I fear it would let people get away with not running tests although it wouldn't be much work to do so...

3. SHOULD This word, or the adjective "RECOMMENDED", mean that there
may exist valid reasons in particular circumstances to ignore a
particular item, but the full implications must be understood and
carefully weighed before choosing a different course.
The way RFC 2119 defines MUST doesn't leave much room for exceptions. If you want to be clear, I suggest SHOULD with a reference to RFC 2119. While not universal, it is a widely used reference for definitions of these terms in technical specifications.
Scott K
[1] https://datatracker.ietf.org/doc/html/rfc2119#section-3

Fair enough. I also made that change.

[Scott Kitterman]

Post by Louis-Philippe Véronneau

Post by Scott Kitterman

Post by Louis-Philippe Véronneau

Post by Scott Kitterman

Post by Louis-Philippe Véronneau

Post by Scott Kitterman

Post by Louis-Philippe Véronneau
Hello,
As discussed during the DebConf24 Python BoF, I'm submitting this change to the policy to require the use of the upstream test suite, both during the build process and as an autopkgtest.
https://salsa.debian.org/python-team/tools/python-modules/-/merge_requests/24
People present at the BoF seemed to think this was a good idea. If you don't please reply to this message and make yourself heard :)

1. The largest set is packages that need test only dependencies which are not packaged. When I am packaging something new which has a test suite, then I generally package any needed test depends. If those test depends also need test depends packaged, I generally stop and don't enable tests for things that are only in the archives to support tests. Noseofyeti is an example of this.

That sounds like a valid technical reason not to run the tests to me :)

Post by Scott Kitterman
2. There's at least one case where Debian has customizations that cause the test suite to fail, but the failures don't seem to cause any real problems. If anyone wants to make it so the weasyprint test suite works on Debian, please knock yourself out.

Again, as long as you document that, I don't think it would go against the proposed policy change.

Post by Scott Kitterman
3. I also maintain multiple packages which require network access to run their test suite, so they can't run tests during build, only autopkgtests.

Same.

Except for #3, I don't get that from the wording in the MR. I don't think not worth the trouble is a technical reason. I think the real rule that's being proposed is that packages must run the test suit or document why not. I don't have a problem with that, but I don't think that's what it actually says now. I think if you were to change must to should and strike the word technical before reason, it would accomplish the same thing and be clearer. I could support that.

Language is hard and I'm not a native English speaker :)
What I want to prevent is people not running tests because they don't feel like it, even though it would not take them a large amount of efforts to do so.
I'll strike "technical", as it seems others also interpreted this word they way you have.
As for "MUST" vs "SHOULD", I believe the exception clause provides enough leeway to justify a reasonable way out in case of $VALID_REASON.
"SHOULD" is not particularly strong and again, I fear it would let people get away with not running tests although it wouldn't be much work to do so...

3. SHOULD This word, or the adjective "RECOMMENDED", mean that there
may exist valid reasons in particular circumstances to ignore a
particular item, but the full implications must be understood and
carefully weighed before choosing a different course.
The way RFC 2119 defines MUST doesn't leave much room for exceptions. If you want to be clear, I suggest SHOULD with a reference to RFC 2119. While not universal, it is a widely used reference for definitions of these terms in technical specifications.
Scott K
[1] https://datatracker.ietf.org/doc/html/rfc2119#section-3

Fair enough. I also made that change.

Thanks,

Scott K

[t***@goirand.fr]

Post by Scott Kitterman

Post by Louis-Philippe Véronneau
Fair enough. I also made that change.

Thanks,
Scott K

You have my +1 with the current wording.


Cheers,


Thomas Goirand (zigo)

[Pierre-Elliott Bécue]

Hey,

Post by Louis-Philippe Véronneau
Hello,
As discussed during the DebConf24 Python BoF, I'm submitting this
change to the policy to require the use of the upstream test suite,
both during the build process and as an autopkgtest.
https://salsa.debian.org/python-team/tools/python-modules/-/merge_requests/24
People present at the BoF seemed to think this was a good idea. If you
don't please reply to this message and make yourself heard :)
Cheers,

I spend a lot of time trying to make the upstream tests working in
package building, but I'm against enforcing this.

There are a lot of packages with good code but crappy tests, and we
don't want to force ourselves maintaining these, and dropping the
packages seem a bit too much.

[Louis-Philippe Véronneau]

Post by Pierre-Elliott Bécue
Hey,

Post by Louis-Philippe Véronneau
Hello,
As discussed during the DebConf24 Python BoF, I'm submitting this
change to the policy to require the use of the upstream test suite,
both during the build process and as an autopkgtest.
https://salsa.debian.org/python-team/tools/python-modules/-/merge_requests/24
People present at the BoF seemed to think this was a good idea. If you
don't please reply to this message and make yourself heard :)
Cheers,

I spend a lot of time trying to make the upstream tests working in
package building, but I'm against enforcing this.
There are a lot of packages with good code but crappy tests, and we
don't want to force ourselves maintaining these, and dropping the
packages seem a bit too much.

I'd file that under "a technical issue [that] prevents you from running
the test suite and you believe the reason to be valid".

:)

[Louis-Philippe Véronneau]

Post by Louis-Philippe Véronneau
Hello,
As discussed during the DebConf24 Python BoF, I'm submitting this change
to the policy to require the use of the upstream test suite, both during
the build process and as an autopkgtest.
https://salsa.debian.org/python-team/tools/python-modules/-/
merge_requests/24
People present at the BoF seemed to think this was a good idea. If you
don't please reply to this message and make yourself heard :)
Cheers,

Hi,

It's been about 2 weeks since the policy change has been proposed. I
believe I have taken in account the feedback and the discussion seems to
have died down.

As the response from the team member was largely positive, I propose the
change be merged.

Cheers,

--
⢀⣎⠟⠻⢶⣊⠀
⣟⠁⢠⠒⠀⣿⡁ Louis-Philippe Véronneau
⢿⡄⠘⠷⠚⠋ ***@debian.org / veronneau.org
⠈⠳⣄

[Andreas Tille]

It's been about 2 weeks since the policy change has been proposed. I believe
I have taken in account the feedback and the discussion seems to have died
down.
As the response from the team member was largely positive, I propose the
change be merged.

Thank you for driving this forward
Andreas.

[Pierre-Elliott Bécue]

Hi,

Post by Louis-Philippe Véronneau

Post by Louis-Philippe Véronneau
Hello,
As discussed during the DebConf24 Python BoF, I'm submitting this
change to the policy to require the use of the upstream test suite,
both during the build process and as an autopkgtest.
https://salsa.debian.org/python-team/tools/python-modules/-/
merge_requests/24
People present at the BoF seemed to think this was a good idea. If
you don't please reply to this message and make yourself heard :)
Cheers,

Hi,
It's been about 2 weeks since the policy change has been proposed. I
believe I have taken in account the feedback and the discussion seems
to have died down.
As the response from the team member was largely positive, I propose
the change be merged.
Cheers,

I'm still against this, and I'm not sure yet whether or not it'll drive
me to leave the team. The first sentence is ok, but the second is not
ok with me.

Regards,

[Louis-Philippe Véronneau]

Post by Pierre-Elliott Bécue
Hi,

Post by Louis-Philippe Véronneau

Post by Louis-Philippe Véronneau
Hello,
As discussed during the DebConf24 Python BoF, I'm submitting this
change to the policy to require the use of the upstream test suite,
both during the build process and as an autopkgtest.
https://salsa.debian.org/python-team/tools/python-modules/-/
merge_requests/24
People present at the BoF seemed to think this was a good idea. If
you don't please reply to this message and make yourself heard :)
Cheers,

Hi,
It's been about 2 weeks since the policy change has been proposed. I
believe I have taken in account the feedback and the discussion seems
to have died down.
As the response from the team member was largely positive, I propose
the change be merged.
Cheers,

I'm still against this, and I'm not sure yet whether or not it'll drive
me to leave the team. The first sentence is ok, but the second is not
ok with me.

Can you be more explicit on the reasons why you dislike the second sentence?

It only says you should document why tests aren't ran in a package and
encourages you to open a bug report.

You certainly don't have to write ten paragraphs, a single sentence
summarizing the issue (and thus letting other people in the team help)
is enough IMO.

--
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Louis-Philippe Véronneau
⢿⡄⠘⠷⠚⠋ ***@debian.org / ve

[Graham Inggs]

Post by Louis-Philippe Véronneau
Can you be more explicit on the reasons why you dislike the second sentence?
It only says you should document why tests aren't ran in a package and
encourages you to open a bug report.

I thought the second sentence was:
"As such, the use of `Testsuite: autopkgtest-pkg-pybuild` is highly
recommended."

I would still like to add a note about adding <!nocheck> to
build-dependencies only needed for the tests.

It might be easier if I add that once this proposal is committed.