URL mangling in https://pypi.debian.net/

[Alexandre Detiste]

Hi,

I've noticed a recent pattern with archives published on PyPi :
the "-" we expect in the regexp specified in d/watch is now an underscore.

So the tracker got the false information that everything is up-to-date

With some horribly wretched code I can find some projects with updates pending.
https://paste.debian.net/1340327/

One field got duplicated in the output but I'm not running
the code again immediately because it can be considered abuse
by who run pypi.debian.net.

Ideas ?

Greetings



url , current version "up to date" in UDD , stem , (idem version),
version upstream, upstream tarball

https://pypi.debian.net/python-socketio 5.11.2-1 python-socketio
5.11.2-1 python_socketio-5.11.3.tar.gz
https://pypi.debian.net/python-socketio 5.11.2-1 python-socketio
5.11.2-1 python_socketio-5.11.4.tar.gz
https://pypi.debian.net/drf-haystack 1.8.13-1 drf-haystack 1.8.13-1
drf_haystack-1.9.tar.gz
https://pypi.debian.net/drf-haystack 1.8.13-1 drf-haystack 1.8.13-1
drf_haystack-1.9.1.tar.gz
https://pypi.debian.net/mpl-scatter-density 0.7-1 mpl-scatter-density
0.7-1 mpl_scatter_density-0.8.tar.gz
https://pypi.debian.net/requests-futures 1.0.1-1 requests-futures
1.0.1-1 requests_futures-1.0.2.tar.gz
https://pypi.debian.net/pytest-retry 1.6.2-2 pytest-retry 1.6.2-2
pytest_retry-1.6.3.tar.gz
https://pypi.debian.net/time-machine 2.14.1-1 time-machine 2.14.1-1
time_machine-2.14.2.tar.gz
https://pypi.debian.net/time-machine 2.14.1-1 time-machine 2.14.1-1
time_machine-2.15.0.tar.gz
https://pypi.debian.net/time-machine 2.14.1-1 time-machine 2.14.1-1
time_machine-2.16.0.tar.gz
https://pypi.debian.net/django-braces 1.15.0-4 django-braces 1.15.0-4
django_braces-1.16.0.tar.gz
https://pypi.debian.net/tkinter-tooltip 3.0.0-3 tkinter-tooltip
3.0.0-3 tkinter_tooltip-3.1.0.tar.gz
https://pypi.debian.net/tkinter-tooltip 3.0.0-3 tkinter-tooltip
3.0.0-3 tkinter_tooltip-3.1.1.tar.gz
https://pypi.debian.net/tkinter-tooltip 3.0.0-3 tkinter-tooltip
3.0.0-3 tkinter_tooltip-3.1.2.tar.gz
https://pypi.debian.net/django-downloadview 2.3.0-1
django-downloadview 2.3.0-1 django_downloadview-2.4.0.tar.gz
https://pypi.debian.net/msoffcrypto-tool 5.0.0-2 msoffcrypto-tool
5.0.0-2 msoffcrypto_tool-5.0.1.tar.gz
https://pypi.debian.net/msoffcrypto-tool 5.0.0-2 msoffcrypto-tool
5.0.0-2 msoffcrypto_tool-5.0.1rc1.tar.gz
https://pypi.debian.net/msoffcrypto-tool 5.0.0-2 msoffcrypto-tool
5.0.0-2 msoffcrypto_tool-5.1.0.tar.gz
https://pypi.debian.net/msoffcrypto-tool 5.0.0-2 msoffcrypto-tool
5.0.0-2 msoffcrypto_tool-5.1.1.tar.gz
https://pypi.debian.net/msoffcrypto-tool 5.0.0-2 msoffcrypto-tool
5.0.0-2 msoffcrypto_tool-5.2.0.tar.gz
https://pypi.debian.net/msoffcrypto-tool 5.0.0-2 msoffcrypto-tool
5.0.0-2 msoffcrypto_tool-5.3.0.tar.gz
https://pypi.debian.net/msoffcrypto-tool 5.0.0-2 msoffcrypto-tool
5.0.0-2 msoffcrypto_tool-5.3.1.tar.gz
https://pypi.debian.net/msoffcrypto-tool 5.0.0-2 msoffcrypto-tool
5.0.0-2 msoffcrypto_tool-5.4.0.tar.gz
https://pypi.debian.net/msoffcrypto-tool 5.0.0-2 msoffcrypto-tool
5.0.0-2 msoffcrypto_tool-5.4.1.tar.gz
https://pypi.debian.net/msoffcrypto-tool 5.0.0-2 msoffcrypto-tool
5.0.0-2 msoffcrypto_tool-5.4.2.tar.gz
https://pypi.debian.net/django-python3-ldap 0.15.6-1
django-python3-ldap 0.15.6-1 django_python3_ldap-0.15.7.tar.gz
https://pypi.debian.net/django-python3-ldap 0.15.6-1
django-python3-ldap 0.15.6-1 django_python3_ldap-0.15.8.tar.gz
https://pypi.debian.net/django-pipeline 3.0.0-2 django-pipeline
3.0.0-2 django_pipeline-3.1.0.tar.gz
https://pypi.debian.net/django-pipeline 3.0.0-2 django-pipeline
3.0.0-2 django_pipeline-4.0.0.tar.gz
https://pypi.debian.net/pytest-twisted 1.14.1-3 pytest-twisted
1.14.1-3 pytest_twisted-1.14.2.tar.gz
https://pypi.debian.net/pytest-twisted 1.14.1-3 pytest-twisted
1.14.1-3 pytest_twisted-1.14.3.tar.gz
https://pypi.debian.net/python-vlc 3.0.20123-1 python-vlc 3.0.20123-1
python_vlc-3.0.21201.tar.gz
https://pypi.debian.net/python-vlc 3.0.20123-1 python-vlc 3.0.20123-1
python_vlc-3.0.21203.tar.gz
https://pypi.debian.net/bids-validator 1.14.5-1 bids-validator
1.14.5-1 bids_validator-1.14.6.tar.gz
https://pypi.debian.net/bids-validator 1.14.5-1 bids-validator
1.14.5-1 bids_validator-1.14.7.dev0.tar.gz
https://pypi.debian.net/bids-validator 1.14.5-1 bids-validator
1.14.5-1 bids_validator-1.14.7.post0.tar.gz
https://pypi.debian.net/djangorestframework-gis 1.0-3
djangorestframework-gis 1.0-3 djangorestframework_gis-1.1.tar.gz
https://pypi.debian.net/pylint-plugin-utils 0.7-3 pylint-plugin-utils
0.7-3 pylint_plugin_utils-0.8.tar.gz
https://pypi.debian.net/pylint-plugin-utils 0.7-3 pylint-plugin-utils
0.7-3 pylint_plugin_utils-0.8.1.tar.gz
https://pypi.debian.net/pylint-plugin-utils 0.7-3 pylint-plugin-utils
0.7-3 pylint_plugin_utils-0.8.2.tar.gz
https://pypi.debian.net/sphinxcontrib-svg2pdfconverter 1.2.2-1
sphinxcontrib-svg2pdfconverter 1.2.2-1
sphinxcontrib_svg2pdfconverter-1.2.3.tar.gz
https://pypi.debian.net/orange-canvas-core 0.2.2-1 orange-canvas-core
0.2.2-1 orange_canvas_core-0.2.3.tar.gz
https://pypi.debian.net/orange-canvas-core 0.2.2-1 orange-canvas-core
0.2.2-1 orange_canvas_core-0.2.4.tar.gz
https://pypi.debian.net/orange-canvas-core 0.2.2-1 orange-canvas-core
0.2.2-1 orange_canvas_core-0.2.5.tar.gz
https://pypi.debian.net/django-titofisto 0.2.2-1 django-titofisto
0.2.2-1 django_titofisto-1.0.0.tar.gz
https://pypi.debian.net/django-titofisto 0.2.2-1 django-titofisto
0.2.2-1 django_titofisto-1.0.0.post0.tar.gz
https://pypi.debian.net/django-titofisto 0.2.2-1 django-titofisto
0.2.2-1 django_titofisto-1.1.0.tar.gz
https://pypi.debian.net/python-bugzilla 3.2.0-2 python-bugzilla
3.2.0-2 python_bugzilla-3.3.0.tar.gz
https://pypi.debian.net/diff-match-patch 20230430-1 diff-match-patch
20230430-1 diff_match_patch-20241021.tar.gz
https://pypi.debian.net/django-gravatar2 1.4.4-4 django-gravatar2
1.4.4-4 django_gravatar2-1.4.5.tar.gz
https://pypi.debian.net/extension-helpers 1.1.1-2 extension-helpers
1.1.1-2 extension_helpers-1.2.0.tar.gz
https://pypi.debian.net/python-binary-memcached 0.31.2+dfsg1-2
python-binary-memcached 0.31.2+dfsg1-2
python_binary_memcached-0.31.3.tar.gz
https://pypi.debian.net/google-auth-oauthlib 1.2.0-3
google-auth-oauthlib 1.2.0-3 google_auth_oauthlib-1.2.1.tar.gz
https://pypi.debian.net/cached-property 1.5.2-1 cached-property
1.5.2-1 cached_property-2.0.tar.gz
https://pypi.debian.net/cached-property 1.5.2-1 cached-property
1.5.2-1 cached_property-2.0.1.tar.gz
https://pypi.debian.net/pytest-localserver 0.8.1-2 pytest-localserver
0.8.1-2 pytest_localserver-0.9.0.tar.gz
https://pypi.debian.net/pytest-localserver 0.8.1-2 pytest-localserver
0.8.1-2 pytest_localserver-0.9.0.post0.tar.gz
https://pypi.debian.net/requests-file 2.0.0-1 requests-file 2.0.0-1
requests_file-2.1.0.tar.gz
https://pypi.debian.net/python-engineio 4.9.0-2 python-engineio
4.9.0-2 python_engineio-4.9.1.tar.gz
https://pypi.debian.net/python-engineio 4.9.0-2 python-engineio
4.9.0-2 python_engineio-4.10.0.tar.gz
https://pypi.debian.net/python-engineio 4.9.0-2 python-engineio
4.9.0-2 python_engineio-4.10.1.tar.gz
https://pypi.debian.net/python-engineio 4.9.0-2 python-engineio
4.9.0-2 python_engineio-4.11.0.tar.gz
https://pypi.debian.net/orange-widget-base 4.24.0-1 orange-widget-base
4.24.0-1 orange_widget_base-4.25.0.tar.gz
https://pypi.debian.net/django-ckeditor 6.7.1+ds-1 django-ckeditor
6.7.1+ds-1 django_ckeditor-6.7.2.tar.gz
https://pypi.debian.net/hypothesis-auto 1.1.4-4 hypothesis-auto
1.1.4-4 hypothesis_auto-1.1.5.tar.gz
https://pypi.debian.net/flake8-docstrings 1.6.0-2 flake8-docstrings
1.6.0-2 flake8_docstrings-1.7.0.tar.gz
https://pypi.debian.net/python-crontab 3.1.0-1 python-crontab 3.1.0-1
python_crontab-3.2.0.tar.gz
https://pypi.debian.net/edgegrid-python 1.3.1-2 edgegrid-python
1.3.1-2 edgegrid_python-2.0.0.tar.gz
https://pypi.debian.net/ibm-watson 8.0.0-1.1 ibm-watson 8.0.0-1.1
ibm_watson-8.1.0.tar.gz
https://pypi.debian.net/ibm-watson 8.0.0-1.1 ibm-watson 8.0.0-1.1
ibm_watson-9.0.0.tar.gz
https://pypi.debian.net/django-auth-ldap 4.8.0-1 django-auth-ldap
4.8.0-1 django_auth_ldap-5.0.0.tar.gz
https://pypi.debian.net/django-auth-ldap 4.8.0-1 django-auth-ldap
4.8.0-1 django_auth_ldap-5.1.0.tar.gz

[Dmitry Shachnev]

Hi Alexandre!

Post by Alexandre Detiste
Hi,
the "-" we expect in the regexp specified in d/watch is now an underscore.
So the tracker got the false information that everything is up-to-date
With some horribly wretched code I can find some projects with updates pending.
https://paste.debian.net/1340327/
One field got duplicated in the output but I'm not running
the code again immediately because it can be considered abuse
by who run pypi.debian.net.
Ideas ?

I think pypi.debian.net does not mangle the file names in any way, it just
takes them from upstream PyPI verbatim.

And the change from - to _ is caused by more build tools adopting this
specification [1], which says:

“In distribution names, any run of -_. characters (HYPHEN-MINUS, LOW LINE and
FULL STOP) should be replaced with _ (LOW LINE), and uppercase characters
should be replaced with corresponding lowercase ones.”

This link is for binary distributions, but there is a separate specification
for source distributions [2] which says that rules are the same.

[1]: https://packaging.python.org/en/latest/specifications/binary-distribution-format/#escaping-and-unicode
[2]: https://packaging.python.org/en/latest/specifications/source-distribution-format/#source-distribution-file-name

--
Dmitry Shachnev

[Alexandre Detiste]

Thank you very much for the explanation.

It's a quite general problem, but not so important;
and it only can be detected after upstream
has made at least one release with the new naming convention.

I'll see how to follow this in the long run.

Greetings

Post by Dmitry Shachnev
Hi Alexandre!

Post by Alexandre Detiste
the "-" we expect in the regexp specified in d/watch is now an underscore.

I think pypi.debian.net does not mangle the file names in any way, it just
takes them from upstream PyPI verbatim.

Indeed

Post by Dmitry Shachnev
And the change from - to _ is caused by more build tools adopting this
[1]: https://packaging.python.org/en/latest/specifications/binary-distribution-format/#escaping-and-unicode
[2]: https://packaging.python.org/en/latest/specifications/source-distribution-format/#source-distribution-file-name