Correct command to skip multiple tests in autopkgtest?

[Soren Stoutner]

I am in the process of packaging pyinstaller.

https://pypi.org/project/pyinstaller/

Many of pyinstaller’s tests require the pyinstaller-hooks-contrib package,
which I am also in the process of packaging.

https://pypi.org/project/pyinstaller-hooks-contrib/

But pyinstaller-hooks-contrib depends on pyinstaller for its tests.

In order to bootstrap the introduction of these packages into Debian, I am
temporarily disabling the pyinstaller tests that depend on pyinstaller-hooks-
contrib using the following command in debian/rules:

export PYBUILD_TEST_ARGS=-k\
'not test_find_module\
and not test_metadata_searching\
and not test_Qt_QTranslate\
and not test_zeep\
and not test_scapy3\
and not test_zope_interface\
and not test_gevent\
and not test_user_preferred_locale\
and not test_pytz'

This works successfully during the build process:

1004 passed, 286 skipped, 25 deselected, 24 xfailed, 1 xpassed, 17 warnings in
6969.97s (1:56:09)

https://salsa.debian.org/soren/python-pyinstaller/-/jobs/6514811#L6635

But, for some reason, autopkgtest doesn’t like the format of the arguments.

I: pybuild base:311: cd /tmp/autopkgtest-lxc.ntf_swjm/downtmp/autopkgtest_tmp/
build; python3.12 -m pytest -k 'not test_find_module and not
test_metadata_searching and not test_Qt_QTranslate and not test_zeep and not
test_scapy3 and not test_zope_interface and not test_gevent and not
test_user_preferred_locale and not test_pytz'
usage: __main__.py [-h] [--link-only] {samedir,subdir,parentdir}
__main__.py: error: argument test_type: invalid choice: 'not test_find_module
and not test_metadata_searching and not test_Qt_QTranslate and not test_zeep
and not test_scapy3 and not test_zope_interface and not test_gevent and not
test_user_preferred_locale and not test_pytz' (choose from 'samedir',
'subdir', ‘parentdir’)

https://salsa.debian.org/soren/python-pyinstaller/-/jobs/6514819#L781

I have tried several syntax variations without success. Does anyone know the
correct syntax for skipping several tests in autopgktest?

[Soren Stoutner]

Post by Soren Stoutner
I: pybuild base:311: cd /tmp/autopkgtest-lxc.ntf_swjm/downtmp/

autopkgtest_tmp/

Post by Soren Stoutner
build; python3.12 -m pytest -k 'not test_find_module and not
test_metadata_searching and not test_Qt_QTranslate and not test_zeep and not
test_scapy3 and not test_zope_interface and not test_gevent and not
test_user_preferred_locale and not test_pytz'
usage: __main__.py [-h] [--link-only] {samedir,subdir,parentdir}
__main__.py: error: argument test_type: invalid choice: 'not test_find_module
and not test_metadata_searching and not test_Qt_QTranslate and not test_zeep
and not test_scapy3 and not test_zope_interface and not test_gevent and not
test_user_preferred_locale and not test_pytz' (choose from 'samedir',
'subdir', ‘parentdir’)

My experience with another package is that -k works correctly with autopkgtest
when skipping only one test. The problem appears to be something related to
parsing multiple tests to skip. I have tried a couple of syntax variations,
but so far haven’t been able to guess the right one.

[Julian Gilbey]

Post by Soren Stoutner

Post by Soren Stoutner
I: pybuild base:311: cd /tmp/autopkgtest-lxc.ntf_swjm/downtmp/

autopkgtest_tmp/

Post by Soren Stoutner
build; python3.12 -m pytest -k 'not test_find_module and not
test_metadata_searching and not test_Qt_QTranslate and not test_zeep and not
test_scapy3 and not test_zope_interface and not test_gevent and not
test_user_preferred_locale and not test_pytz'
usage: __main__.py [-h] [--link-only] {samedir,subdir,parentdir}
__main__.py: error: argument test_type: invalid choice: 'not test_find_module
and not test_metadata_searching and not test_Qt_QTranslate and not test_zeep
and not test_scapy3 and not test_zope_interface and not test_gevent and not
test_user_preferred_locale and not test_pytz' (choose from 'samedir',
'subdir', ‘parentdir’)

My experience with another package is that -k works correctly with autopkgtest
when skipping only one test. The problem appears to be something related to
parsing multiple tests to skip. I have tried a couple of syntax variations,
but so far haven’t been able to guess the right one.

You could have a look at what I've done in pydevd (soon to be removed
from the archive), which was a complex case. But I'd suggest using
the syntax

--deselect=tests/test_file.py::test_testname

instead of -k; it can be given multiple times, and as long as there
are no spaces or weird characters in the test names (for example,
tests/test_file.py::test_testname[this is a parameter]), it works fine
when stored as a shell variable and then expanded without quotes.

Best wishes,

Julian

[Soren Stoutner]

Post by Julian Gilbey
You could have a look at what I've done in pydevd (soon to be removed
from the archive), which was a complex case. But I'd suggest using
the syntax
--deselect=tests/test_file.py::test_testname
instead of -k; it can be given multiple times, and as long as there
are no spaces or weird characters in the test names (for example,
tests/test_file.py::test_testname[this is a parameter]), it works fine
when stored as a shell variable and then expanded without quotes.

Thanks for the suggestion, but I wasn’t able to get --deselect to work, either
during the build or in autopkgtest. I also have been unsuccessful in getting
--ignore or -k to work with autopkgtest, even though they both work during the
build.

My sense is that it is some bug related to the implementation of "Testsuite:
autopkgtest-pkg-pybuild”, but I don’t know enough about it to troubleshoot it.