python-trezor: assistance with disabling or modifying a build test that requires network access

[Soren Stoutner]

I am in the process of adopting and updating python-trezor. Upstream now
contains a test that involves downloading a binary firmware package from the
internet [1].

This test causes the build to fail with the following error message, both on
my local sbuild and on Salsa [2]:

requests.exceptions.ProxyError: HTTPSConnectionPool(host='data.trezor.io',
port=443): Max retries exceeded with url: /firmware/2/trezor-2.4.2.bin (Caused
by ProxyError('Unable to connect to proxy',
NewConnectionError('<urllib3.connection.HTTPSConnection object at
0x7f6d1440d040>: Failed to establish a new connection: [Errno 111] Connection
refused’)))

Loading the URL [3] in a browser allows downloading of the file. I do not know
why the test fails in my local sbuild as I don’t believe it prohibits network
access. But it should fail on the buildds, so it needs to be addressed.

I would appreciate some guidance from more experienced Python packagers on the
best way to proceed.

1. What is the best (idomatic) way to disable this test during build?

2. I would still like to run this test as an autopkgtest. My guess is it
will fail there for the same reasons are that it is failing here (probably
because it is looking for a proxy that doesn’t exist). Does anyone know the
best way to approach that? Does it cause a DFSG problem if it downloads a
binary firmware image? A cursory investigation would indicate the firmware
source code is available under DFSG-free licenses [4].

[1] https://salsa.debian.org/python-team/packages/python-trezor/-/blob/debian/
master/tests/test_firmware.py?ref_type=heads

[2] https://salsa.debian.org/python-team/packages/python-trezor/-/jobs/
6341154#L6882

[3] https://data.trezor.io/firmware/2/trezor-2.4.2.bin

[4] https://github.com/trezor/trezor-firmware/blob/main/LICENSE.md

[Andrey Rakhmatullin]

Post by Soren Stoutner
I am in the process of adopting and updating python-trezor. Upstream now
contains a test that involves downloading a binary firmware package from the
internet [1].
This test causes the build to fail with the following error message, both on
requests.exceptions.ProxyError: HTTPSConnectionPool(host='data.trezor.io',
port=443): Max retries exceeded with url: /firmware/2/trezor-2.4.2.bin (Caused
by ProxyError('Unable to connect to proxy',
NewConnectionError('<urllib3.connection.HTTPSConnection object at
0x7f6d1440d040>: Failed to establish a new connection: [Errno 111] Connection
refused’)))
Loading the URL [3] in a browser allows downloading of the file. I do not know
why the test fails in my local sbuild as I don’t believe it prohibits network
access. But it should fail on the buildds, so it needs to be addressed.

pybuild prohibits network access to all programs that honor http(s)_proxy
envvars. That's why the error mentions a proxy.

Post by Soren Stoutner
I would appreciate some guidance from more experienced Python packagers on the
best way to proceed.
1. What is the best (idomatic) way to disable this test during build?

In this case - just skip the whole file via appropriate PYBUILD_TEST_ARGS.

[Soren Stoutner]

Post by Andrey Rakhmatullin

Post by Soren Stoutner
1. What is the best (idomatic) way to disable this test during build?

In this case - just skip the whole file via appropriate PYBUILD_TEST_ARGS.

Thanks for the pointer.

[Soren Stoutner]

I have a further question about this test. When it runs in autopkgtests, it
looks for a file that exists in the larger upstream repository, but isn’t in
the python module tarball.

The test fails [1] with this error:

E FileNotFoundError: [Errno 2] No such file or directory: '/tmp/
autopkgtest-lxc.8j5j6b6j/downtmp/build.nmg/core/embed/vendorheader/T2T1/
vendorheader_satoshilabs_signed_prod.bin'


The file it is looking for appears to be this one [2], locating in the main
upstream project (with a slightly different path, but maybe it somehow resolves
it if present). As far as I can tell, this is a dummy binary firmware file used
to make sure trezor can read the headers correctly.

The main upstream repository is quite large. It has a subdirectory named
python [3] which is packaged on PyPI [4] and which uscan pulls from
pypi.debian.net.

My question is, what is the best way to handle this test? I can think of
several possibilities, and there are probably others I haven’t thought of.

1. Just exclude this test from autopkgtests and go on with my life (I assume
autopkgtest-pkg-pybuild has an easy way to do this). There are 111 other
tests that run successfully, this is a test that upstream should run with
every release, and it is testing the type of thing that is unlikely to work
for upstream but end up broken in Debian.

3. Include this file in the Debian directory somewhere like missing-sources
and run the test against that. Something about this just feels wrong,
especially because it is a signed binary file, not really a source file.

2. Modify the test to download this file from github. Python is not my
primary language, and I don’t know how difficult it would be to implement or
maintain, but I assume this would be possible.

3. Replace the source package with the entire trezor-firmware repository.
This seems like an overreach because it would introduce massive numbers of
files into the Debian source package that have nothing to do with the python
binary packages.

If nobody has any previous experience or strong opinions about situations like
these, I am inclined to just go with option 1.


[1] https://salsa.debian.org/python-team/packages/python-trezor/-/jobs/
6345561#L248

[2] https://github.com/trezor/trezor-firmware/blob/main/core/embed/models/
T2T1/vendorheader/vendorheader_satoshilabs_signed_prod.bin

[3] https://github.com/trezor/trezor-firmware/tree/main/python

[4] https://pypi.org/project/trezor/

Post by Andrey Rakhmatullin

Post by Soren Stoutner
I am in the process of adopting and updating python-trezor. Upstream now
contains a test that involves downloading a binary firmware package from the
internet [1].
This test causes the build to fail with the following error message, both on
requests.exceptions.ProxyError: HTTPSConnectionPool(host='data.trezor.io',
port=443): Max retries exceeded with url: /firmware/2/trezor-2.4.2.bin
(Caused by ProxyError('Unable to connect to proxy',
NewConnectionError('<urllib3.connection.HTTPSConnection object at
0x7f6d1440d040>: Failed to establish a new connection: [Errno 111] Connection
refused’)))
Loading the URL [3] in a browser allows downloading of the file. I do not
know why the test fails in my local sbuild as I don’t believe it prohibits
network access. But it should fail on the buildds, so it needs to be
addressed.

pybuild prohibits network access to all programs that honor http(s)_proxy
envvars. That's why the error mentions a proxy.

Post by Soren Stoutner
I would appreciate some guidance from more experienced Python packagers on
the best way to proceed.
1. What is the best (idomatic) way to disable this test during build?

In this case - just skip the whole file via appropriate PYBUILD_TEST_ARGS.

[Soren Stoutner]

Post by Soren Stoutner
1. Just exclude this test from autopkgtests and go on with my life (I assume
autopkgtest-pkg-pybuild has an easy way to do this). There are 111 other
tests that run successfully, this is a test that upstream should run with
every release, and it is testing the type of thing that is unlikely to work
for upstream but end up broken in Debian.

Is there an easy way to exclude one test using autopkgtest-pkg-pybuild?

[Andrey Rakhmatullin]

Post by Soren Stoutner

Post by Soren Stoutner
1. Just exclude this test from autopkgtests and go on with my life (I

assume

Post by Soren Stoutner
autopkgtest-pkg-pybuild has an easy way to do this). There are 111 other
tests that run successfully, this is a test that upstream should run with
every release, and it is testing the type of thing that is unlikely to work
for upstream but end up broken in Debian.

Is there an easy way to exclude one test using autopkgtest-pkg-pybuild?

Yes, exclude it from the build-time tests.
Why does it succeed there?

[Soren Stoutner]

Post by Andrey Rakhmatullin

Post by Soren Stoutner

Post by Soren Stoutner
1. Just exclude this test from autopkgtests and go on with my life (I

assume

Post by Soren Stoutner
autopkgtest-pkg-pybuild has an easy way to do this). There are 111 other
tests that run successfully, this is a test that upstream should run with
every release, and it is testing the type of thing that is unlikely to work
for upstream but end up broken in Debian.

Is there an easy way to exclude one test using autopkgtest-pkg-pybuild?

Yes, exclude it from the build-time tests.
Why does it succeed there?

It turns out I had some misconfiguration that caused the testing behavior to
act different than what I thought I had told it to do. It is now fixed. Thank
you for prompting me to look again more closely.