blhc failing due to non-verbose build

[Soren Stoutner]

I am working on PyInstaller, which is mostly written in Python, but compiles a
bootloader written in c. blhc failes because the longs do not contain verbose
compile flags.

111:NONVERBOSE BUILD: [ 1/21] Compiling src/pyi_utils.c
112:NONVERBOSE BUILD: [ 2/21] Compiling src/pyi_main.c
113:NONVERBOSE BUILD: [ 3/21] Compiling src/pyi_exception_dialog.c
114:NONVERBOSE BUILD: [ 4/21] Compiling src/pyi_apple_events.c
115:NONVERBOSE BUILD: [ 5/21] Compiling src/pyi_splashlib.c
116:NONVERBOSE BUILD: [ 6/21] Compiling src/pyi_global_posix.c
117:NONVERBOSE BUILD: [ 7/21] Compiling src/pyi_multipkg.c

https://salsa.debian.org/python-team/packages/pyinstaller/-/jobs/6701977

The standard tricks for most C/C++ packages don’t work.

# Enable all build hardening flags.
export DEB_BUILD_MAINT_OPTIONS = hardening=+all

# Export the DPKG default build flags.
DPKG_EXPORT_BUILDFLAGS = 1

# Include the default build flags. <https://wiki.debian.org/Hardening#dpkg-buildflags>
include /usr/share/dpkg/buildflags.mk

# Append the CPPFLAGS to the standard CFLAGS and CXXFLAGS variables, which is
how CMake likes it. <https://wiki.debian.org/
Hardening#Notes_for_packages_using_CMake>
CFLAGS += $(CPPFLAGS)
CXXFLAGS += $(CPPFLAGS)

https://salsa.debian.org/python-team/packages/pyinstaller/-/blob/debian/
master/debian/rules?ref_type=heads

Does anyone know how to convince it to populate the logs with the full flags
used to compile the binaries?

[Bo YU]

Hi,

On Fri, Dec 6, 2024 at 10:01 AM Soren Stoutner <***@debian.org> wrote:
...

Post by Soren Stoutner
# Append the CPPFLAGS to the standard CFLAGS and CXXFLAGS variables, which is
how CMake likes it. <https://wiki.debian.org/
Hardening#Notes_for_packages_using_CMake>
CFLAGS += $(CPPFLAGS)
CXXFLAGS += $(CPPFLAGS)
https://salsa.debian.org/python-team/packages/pyinstaller/-/blob/debian/
master/debian/rules?ref_type=heads
Does anyone know how to convince it to populate the logs with the full flags
used to compile the binaries?

I would like to suggest you to look at the code:
https://sources.debian.org/src/structure-synth/1.5.0-7.1/debian/rules/#L6
or
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1075694#56

Even if it doesn't work, we can get inspiration on how to append extra
flags during the configure phase.

Hope this helps.

BR,
Bo

Post by Soren Stoutner
--
Soren Stoutner

[Simon McVittie]

Post by Soren Stoutner
I am working on PyInstaller, which is mostly written in Python, but compiles a
bootloader written in c. blhc failes because the [logs] do not contain verbose
compile flags.

You'll need to look at the implementation of the build for the C part, and
then do whatever is most appropriate for that build system.
additional_args = os.getenv('PYINSTALLER_BOOTLOADER_WAF_ARGS', '').strip().split()
cmd = [sys.executable, './waf', 'configure', 'all']
cmd += additional_args

so hopefully there is something you can add to
PYINSTALLER_BOOTLOADER_WAF_ARGS that would make waf verbose, analogous
to `ninja -v` or Autotools `V=1`?

After that, you'll also need to make sure that the intended build
options are actually used (I don't know whether waf uses CFLAGS, etc. by
default or has to be given them via waf-specific command-line options).
Looking at other packages that use a waf build system and implement build
flags correctly, if any such packages exist, will probably be useful.

We have had problems with waf in the past, both technical and social
(licensing-related and others), so please be careful to ensure that
this package meets Debian's quality standards and doesn't contain any
particularly attractive places to hide malware.

In particular, the recommended way to distribute waf-built code used to
be to vendor a generated script containing a bzip2-compressed tarball,
which is not straightforward to review or patch, and the ftp team does
not consider this to be acceptable in Debian [1]. Is this still the case,
or is PyInstaller redistributing waf as reviewable/patchable files in
something more closely resembling their preferred form for modification?

Has the maintainer of this package (possibly you, I don't know this
package's history) verified that the included copy of waf is something
that we can trust? From the fact that you didn't already know this
package is using waf, I would guess perhaps not?

I can't help wondering whether its upstream would be receptive to the idea
of using Meson with meson-python as a replacement for both setuptools
and waf, which is what I would personally do if I had a Python module
that needed to compile/include C code (and indeed is what I *did* do
in dbus-python).

Good luck!

smcv

[1] https://wiki.debian.org/UnpackWaf

[Soren Stoutner]

Simon,

Thank you for taking the time to look at this.

Post by Simon McVittie

Post by Soren Stoutner
I am working on PyInstaller, which is mostly written in Python, but

compiles

Post by Simon McVittie

Post by Soren Stoutner
a bootloader written in c. blhc failes because the [logs] do not contain
verbose compile flags.

You'll need to look at the implementation of the build for the C part, and
then do whatever is most appropriate for that build system.
additional_args = os.getenv('PYINSTALLER_BOOTLOADER_WAF_ARGS',
'').strip().split() cmd = [sys.executable, './waf', 'configure', 'all']
cmd += additional_args
so hopefully there is something you can add to
PYINSTALLER_BOOTLOADER_WAF_ARGS that would make waf verbose, analogous
to `ninja -v` or Autotools `V=1`?

This was a very helpful suggestion. I was able to produce a verbose build by
adding the following to debian/rules:

# Enable the verbose waf build argument so that blhc can analyze the build
flags. waf is the system that builds the bootloader from C code.
export PYINSTALLER_BOOTLOADER_WAF_ARGS = --verbose

Post by Simon McVittie
After that, you'll also need to make sure that the intended build
options are actually used (I don't know whether waf uses CFLAGS, etc. by
default or has to be given them via waf-specific command-line options).
Looking at other packages that use a waf build system and implement build
flags correctly, if any such packages exist, will probably be useful.

The above verbose flag then produces this output in the build logs:

[ 1/21] Compiling src/pyi_utils.c
00:07:52 runner ['/usr/lib/ccache/gcc', '-g', '-O2', '-Werror=implicit-
function-declaration', '-ffile-prefix-map=/builds/python-team/packages/
pyinstaller/debian/output/source_dir=.', '-fstack-protector-strong', '-fstack-
clash-protection', '-Wformat', '-Werror=format-security', '-fcf-protection',
'-Wdate-time', '-D_FORTIFY_SOURCE=2', '-m64', '-O2', '-Wall', '-Werror', '-
Wno-error=unused-variable', '-Wno-error=unused-function', '-Wno-error=unused-
but-set-variable', '-U_FORTIFY_SOURCE', '-Isrc', '-I../../src', '-Iwindows',
'-I../../windows', '-Izlib', '-I../../zlib', '-D_REENTRANT', '-D_BSD_SOURCE',
'-D_DEFAULT_SOURCE', '-D_FORTIFY_SOURCE=2', '-DHAVE_STDBOOL_H=1', '-
DHAVE_UNSETENV=1', '-DHAVE_MKDTEMP=1', '-DHAVE_DIRNAME=1', '-
DHAVE_BASENAME=1', '-DLAUNCH_DEBUG', '-DNDEBUG', '../../src/pyi_utils.c', '-
c', '-o/builds/python-team/packages/pyinstaller/debian/output/source_dir/
bootloader/build/debug/src/pyi_utils.c.1.o', '-Wdate-time', ‘-
D_FORTIFY_SOURCE=2’]

Blhc still reports the above as a NONVERBOSE build because there is a line
break, so the first line is flagged separate from the second line. It turns out
there is an existing blhc bug report for this, which I have added some
addition information to.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=976175

Post by Simon McVittie
We have had problems with waf in the past, both technical and social
(licensing-related and others), so please be careful to ensure that
this package meets Debian's quality standards and doesn't contain any
particularly attractive places to hide malware.
In particular, the recommended way to distribute waf-built code used to
be to vendor a generated script containing a bzip2-compressed tarball,
which is not straightforward to review or patch, and the ftp team does
not consider this to be acceptable in Debian [1]. Is this still the case,
or is PyInstaller redistributing waf as reviewable/patchable files in
something more closely resembling their preferred form for modification?
Has the maintainer of this package (possibly you, I don't know this
package's history) verified that the included copy of waf is something
that we can trust? From the fact that you didn't already know this
package is using waf, I would guess perhaps not?

I have not had any experience with waf before, and so am not aware of DFSG or
malware difficulties that other projects have faced. In the case of
PyInstaller, most of the waf code is contained in:

https://salsa.debian.org/python-team/packages/pyinstaller/-/tree/debian/
master/bootloader/waflib?ref_type=heads

It is written in Python and licensed under the BSD-3-clause. It is used to
compile the C code in:

https://salsa.debian.org/python-team/packages/pyinstaller/-/tree/debian/
master/bootloader/src?ref_type=heads

Which is licensed under the GPL-2+~with-bootloader-exception, which is the
main license of the project. The resulting bootloader (two files) is shipped
in the binary package in /usr/lib/python3/dist-packages/PyInstaller/
bootloader/Linux-64bit-intel/*.

None of this looks problematic to me. However, if there are any concerns I
have missed I would be very interested to hear of them before I submit
PyInstaller to the NEW queue.

[Scott Kitterman]

On December 7, 2024 5:29:39 PM UTC, Soren Stoutner <***@debian.org> wrote:
...

Post by Soren Stoutner
I have not had any experience with waf before, and so am not aware of DFSG or
malware difficulties that other projects have faced. In the case of
https://salsa.debian.org/python-team/packages/pyinstaller/-/tree/debian/
master/bootloader/waflib?ref_type=heads
It is written in Python and licensed under the BSD-3-clause. It is used to
https://salsa.debian.org/python-team/packages/pyinstaller/-/tree/debian/
master/bootloader/src?ref_type=heads
Which is licensed under the GPL-2+~with-bootloader-exception, which is the
main license of the project. The resulting bootloader (two files) is shipped
in the binary package in /usr/lib/python3/dist-packages/PyInstaller/
bootloader/Linux-64bit-intel/*.
None of this looks problematic to me. However, if there are any concerns I
have missed I would be very interested to hear of them before I submit
PyInstaller to the NEW queue.

Have a look at the waf entry in the FTP Team reject FAQ:

https://ftp-master.debian.org/REJECT-FAQ.html

Scott K

[Soren Stoutner]

Post by Scott Kitterman
...

Post by Soren Stoutner
I have not had any experience with waf before, and so am not aware of DFSG or
malware difficulties that other projects have faced. In the case of
https://salsa.debian.org/python-team/packages/pyinstaller/-/tree/debian/
master/bootloader/waflib?ref_type=heads
It is written in Python and licensed under the BSD-3-clause. It is used to
https://salsa.debian.org/python-team/packages/pyinstaller/-/tree/debian/
master/bootloader/src?ref_type=heads
Which is licensed under the GPL-2+~with-bootloader-exception, which is the
main license of the project. The resulting bootloader (two files) is shipped
in the binary package in /usr/lib/python3/dist-packages/PyInstaller/
bootloader/Linux-64bit-intel/*.
None of this looks problematic to me. However, if there are any concerns I
have missed I would be very interested to hear of them before I submit
PyInstaller to the NEW queue.

https://ftp-master.debian.org/REJECT-FAQ.html

"That's a special case of source code missing. Normally packages using waf as
build system contain a Python script with a compressed tarball embedded as a
binary blob, where it is not obvious how to get the actual source. As that's
not considered to be the preferred form of modification, it fails the DFSG. See
#645190 and https://wiki.debian.org/UnpackWaf for details.”

As I detailed in the previous email, that does not appear to be the case for
PyInstaller. There are no binary blobs that I have found (although I would be
interested in knowing if I have missed them). I do understand and agree with
such a concern. It just doesn’t appear to be how waf is used by PyInstaller.