[contrib] tools for inspecting package images#1015
Open
tonyaiuto wants to merge 28 commits intobazelbuild:mainfrom
Open
[contrib] tools for inspecting package images#1015tonyaiuto wants to merge 28 commits intobazelbuild:mainfrom
tonyaiuto wants to merge 28 commits intobazelbuild:mainfrom
Conversation
* Add the workflow to publish a release to the bcr. - Update .bcr metadata template - Update .bcr test matrix to drop bazel 7
- pure python - it requires lzma and gzip modules for the decompressor, we we need 3.? - cpio dumper is a separate tool. It should be a library method.
Updates `ossf/scorecard-action` from `v2` to `v2.4.0` and `actions/upload-artifact` from `v3` to `v4` to comply with GitHub's deprecation notice (https://github.blog/changelog/2024-04-16-deprecation-notice-v3-of-the-artifact-actions). As of January 22, 2025, workflows using v3 artifact actions fail with: ``` ##[error]This request has been automatically failed because it uses a deprecated version of `actions/upload-artifact: v3`. Learn more: https://github.blog/changelog/2024-04-16-deprecation-notice-v3-of-the-artifact-actions/. ```
This upgrade of `bazel_skylib` from version 1.7.1 to 1.9.0 eliminates a few deprecation warnings when executing Bazel commands: > WARNING: [...]: target '[...]' depends on deprecated target > '@@bazel_tools//src/conditions:host_windows_x64_constraint': No longer > used by Bazel and will be removed in the future. Migrate to toolchains > or define your own version of this setting. These warnings originated from `bazel_skylib` 1.7.1's use of the deprecated `@bazel_tools//src/conditions:host_windows` constraint in `copy_file` and `diff_test rules`. Since version 1.8.0, `bazel_skylib` migrated away from this deprecated constraint (see bazelbuild/bazel-skylib#574).
Bazel 9.0.0 flipped `--incompatible_strict_action_env` to `true` by default, restricting `PATH` to `/bin:/usr/bin:/usr/local/bin`. On macOS, this broke a few tests relying on the `md5sum` command: ``` Executing genrule //tests/deb:generate_md5sums failed: (Exit 127): [...] [...] /bin/bash: md5sum: command not found ``` This is because: 1. `md5sum` is not a native macOS command (macOS equivalent is `md5`), 2. Homebrew's `md5sum` is typically in `/opt/homebrew/bin`, which is no longer in the restricted `PATH`. Since the repo already maintains a cross-platform `//tests/util:md5` tool, the fix simply consists in leveraging it. Note on: ```diff - cmd = "for i in $(OUTS); do echo 1 >$$i; done", + cmd = "for i in $(OUTS); do echo $$i >$$i; done", ``` ... is only meant to get distinct md5 sums for the 2 input files used in tests.
Bazel 9 flipped `--incompatible_strict_action_env` to `true` by default (bazelbuild/bazel#27670), which means tests no longer inherit `PATH` from the host environment. This breaks subprocess calls that rely on `PATH` lookup: ``` ==================== Test output for //distro:packaging_test: E. ====================================================================== ERROR: testBuild (__main__.PackagingTest.testBuild) ---------------------------------------------------------------------- Traceback (most recent call last): [...] build_result = subprocess.check_output(['bazel', 'build', '--enable_workspace', ':dummy_tar']) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [...] FileNotFoundError: [Errno 2] No such file or directory: 'bazel' ``` This change therefore adds `env_inherit = ["PATH"]` to that very test, which matches the behavior of earlier Bazel versions and is consistent with the presence of a "noci" tag. Additionally, Bazel 9 removed `WORKSPACE` support entirely (bazelbuild/bazel#26131), requiring `bzlmod` with `MODULE.bazel`: ``` ==================== Test output for //distro:packaging_test: [...] WARNING: --enable_bzlmod is set, but no MODULE.bazel file was found at the workspace root. Bazel will create an empty MODULE.bazel file. Please consider migrating your external dependencies from WORKSPACE to MODULE.bazel. For more details, please refer to bazelbuild/bazel#18958. [...] ERROR: error loading package '': Unable to find package for @@[unknown repo 'not_named_rules_pkg' requested from @@]//pkg:tar.bzl: The repository '@@[unknown repo 'not_named_rules_pkg' requested from @@]' could not be resolved: No repository visible as '@not_named_rules_pkg' from main repository. [...] ERROR: Build did NOT complete successfully E. ====================================================================== ERROR: testBuild (__main__.PackagingTest.testBuild) ---------------------------------------------------------------------- [...] build_result = subprocess.check_output(['bazel', 'build'] + bazel_flags + [':dummy_tar']) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [...] subprocess.CalledProcessError: Command '['bazel', 'build', '--enable_workspace', ':dummy_tar']' returned non-zero exit status 1. ``` The test now detects the Bazel version at runtime and generates the appropriate setup: - Bazel 9+: `MODULE.bazel`, with `bazel_dep` and `archive_override`, - earlier versions, as before: `WORKSPACE`, with `http_archive` and `--enable_workspace` flag. For good measure, the change also addresses a leftover TODO by replacing the `tar` subprocess invocation with pure Python code (as otherwise done in the repo).
This enables RPM debuginfo package generation across different Linux
distributions (tested on Rocky and Ubuntu in CI) by implementing RPM
version-based detection if possible, and falling back to the existing OS
release-based detection otherwise.
RPM 4.18.0 introduced changes to make `%{buildsubdir}` independent of
the `%setup` macro:
- commit: `Make %{buildsubdir} more independent of %setup`
(rpm-software-management/rpm@6caca84c904423)
- since: `rpm-4.18.0-alpha1`
- release: https://rpm.org/wiki/Releases/4.18.0
While this change enabled setting `buildsubdir` as a regular macro
rather than a `spec` object property, it had the side effect of altering
the relative path handling in install scripts.
This requires different file path formats in the `%install` section:
- RPM < 4.18, corresponding to the project's "centos" type:
```
cp 'bazel-out/k8-fastbuild/bin/tests/rpm/test_debuginfo' '%{buildroot}/test_debuginfo'
```
- RPM >= 4.18, corresponding to the project's "fedora" type:
```
cp '../bazel-out/k8-fastbuild/bin/tests/rpm/test_debuginfo' '%{buildroot}/test_debuginfo'
```
Using the wrong "type" causes build failures:
- "fedora" with RPM < 4.18:
```
cp: cannot stat '../bazel-out/k8-fastbuild/bin/tests/rpm/test_debuginfo': No such file or directory
```
- "centos" with RPM >= 4.18:
```
rm: refusing to remove '.' or '..' directory: skipping '.'
```
The present change therefore implements version detection to
automatically select the appropriate "type".
**Additional fixes for cross-distribution compatibility**
RedHat distros have redhat-rpm-config with `%_enable_debug_packages`
that auto-invokes `%debug_package` (rpm-software-management/rpm#2204).
Debian/Ubuntu ship vanilla upstream RPM without this configuration:
```
output 'tests/rpm/test_debuginfo_rpm-debuginfo-1-0..rpm' was not created
```
That's why the change adds `%debug_package` only when applicable:
- `%{!?_enable_debug_packages:%debug_package}`.
Also, since RPM [4.14](https://rpm.org/wiki/Releases/4.14.0), unique
debug package filenames are enabled by default, leading to variadic
filenames being generated:
```
Executing tests from //tests/rpm:test_golden_debuginfo_rpm_contents
-----------------------------------------------------------------------------
29c29
< /usr/lib/debug/test_debuginfo-1-0.x86_64.debug
---
> /usr/lib/debug/test_debuginfo.debug
FAIL: files "tests/rpm/test_debuginfo_rpm_contents.txt" and "tests/rpm/test_debuginfo_rpm_contents.txt.golden" differ
```
That's why the change makes debug package filenames consistent across
distributions by means of:
- `%undefine _unique_debug_names` (safe no-op on older RPM versions).
Note: I also verified the change locally with:
- RPM 4.17.1 on Fedora 35 ("centos" type)
- RPM 4.18.2 on Ubuntu 24.04.3 ("fedora" type)
- RPM 4.19.1.1 on Fedora 40 ("fedora" type)
tonyaiuto
force-pushed
the
ptools
branch
2 times, most recently
from
1cfc182 to
ced837a
Compare
aiuto
changed the title
Collaborator
Author
|
The tests seem to be failing on unrelated issues, unable to load rules_java. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First cut at toolkit to do package inspection. This is for contrib right now, because I want to see more flight time with it before supporting it as a core feature.
I started talking about this in #1001
The goal is to create a set of tools that can read through various archive types so that it is easy to build task specific validators. Examples are
The rpm and cpio decoder are clean room implementations, that I wrote from the spec.
After that, I used a coding agent to generate most of the readers. It's grunt work and I was happy to farm it out.
I started a test suite in #1019.
We can do review the API and general utility here. When that is shaken out and landed, I'll update the tests to account for changes here. In the interim, I'm using this daily at work, so that gives me a pretty good set of real world tests.
README.md has more details.