You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@texera.apache.org by "bobbai00 (via GitHub)" <gi...@apache.org> on 2026/05/03 00:05:08 UTC
[PR] fix(licensing): preserve all versions in check_binary_deps multi-version diff [texera]
bobbai00 opened a new pull request, #4711:
URL: https://github.com/apache/texera/pull/4711
### What changes were proposed in this PR?
Fixes a latent bug in `bin/licensing/check_binary_deps.py` — the internal indexers used `dict[name, version]` (and `dict[artifact, (version, basename)]` for jars), so when the same name appeared with two different versions the second assignment silently overwrote the first. Concretely, on `main` today:
- The combined LICENSE-binary has **97 artifacts claimed at multiple versions** (e.g. `org.eclipse.jetty.jetty-server` at `9.4.20.v20190813` and `11.0.20`, `io.grpc.grpc-api` at `1.60.0` and `1.62.2`, `org.apache.hadoop.hadoop-auth` at `3.3.1` and `3.3.3`, ...).
- `_index_jar` collapses 566 entries → 460 — **106 entries are silently dropped**, including 6 of the 7 `netty-tcnative-boringssl-static` per-platform classifiers.
This is undetectable by current CI because #4632 split each ecosystem across services so any single CI invocation only sees one version per lib. The script itself is still wrong, and the multi-version claims in the per-module LICENSE-binary files are not actually being validated against bundled reality.
**Fix.** Switch indexers to multimaps:
- `_index_npm` / `_index_python` → `dict[str, set[str]]` (name → versions)
- `_index_jar` → `dict[str, dict[str, str]]` (artifact → version → basename)
Refactor `diff_simple` / `diff_jars` to emit per-version `added` / `stale` and a per-name `drift` shaped `(name, sorted_claimed, sorted_real)`. Render multi-version drift as
```
~ jetty-server: LICENSE-binary=9.4.20.v20190813, 11.0.20 bundled=9.4.20.v20190813, 11.0.21
```
falling back to the existing single-version form when there's only one version on each side. As a side benefit, `added`/`stale` lines now include the version (this regressed in #4693 which printed bare names).
### Any related issues, documentation, discussions?
Follow-up to #4693 (the original `--ignore-transitive-version` PR) and #4691 (parent task). Discovered and reported by @bobbai00 — without per-service CI splitting (#4632) this would manifest as spurious passes whenever two services bundled different versions of the same lib and the LICENSE-binary documented both.
### How was this PR tested?
Smoke-tested locally against the real combined LICENSE-binary (566 jar claims, 113 python claims):
| Scenario | Strict | `--ignore-transitive-version` |
| --- | --- | --- |
| Clean run (multi-version preserved) | exit 0 (566 entries indexed, was 460) | exit 0 |
| Drop one of two versions of a transitive artifact | exit 1 (drift) | exit 0 (informational drift, both versions shown) |
| Drop all versions of an artifact | exit 1 (stale) | exit 1 |
| Direct-dep version drift (single version) | exit 1 | exit 1 |
| Transitive-dep version drift (single version) | exit 1 | exit 0 |
Sample multi-version drift output:
```
DRIFT (transitive, informational) JVM jars:
~ org.glassfish.jersey.ext.jersey-metainf-services: LICENSE-binary=2.25.1, 3.0.12 bundled=3.0.12
```
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (claude-opus-4-7)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(licensing): preserve all versions in check_binary_deps multi-version diff [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on code in PR #4711:
URL: https://github.com/apache/texera/pull/4711#discussion_r3177766212
##########
.github/workflows/build.yml:
##########
@@ -172,6 +172,10 @@ jobs:
python-version: "3.11"
- name: Show Python
run: python --version || python3 --version
+ - name: Unit-test licensing scripts
+ # Guard the binary-dep checker's behavior before we rely on it in
+ # the jar / python license checks below. Stdlib only, no install.
+ run: python3 -m unittest discover -s bin/licensing -p "test_*.py" -v
Review Comment:
can we run it in another job? say python job? amber job is super lengthy. I want to save every second on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(licensing): preserve all versions in check_binary_deps multi-version diff [texera]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #4711:
URL: https://github.com/apache/texera/pull/4711#issuecomment-4365002131
## [Codecov](https://app.codecov.io/gh/apache/texera/pull/4711?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 85.61%. Comparing base ([`e3d96b0`](https://app.codecov.io/gh/apache/texera/commit/e3d96b0d83d81cbbc1239725daf0b742103f3f0a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`ecd1e27`](https://app.codecov.io/gh/apache/texera/commit/ecd1e27e31e214d8eec20598b1abeffb12aca797?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 1 commits behind head on main.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #4711 +/- ##
=============================================
+ Coverage 46.60% 85.61% +39.00%
=============================================
Files 996 158 -838
Lines 37939 7930 -30009
Branches 3753 0 -3753
=============================================
- Hits 17682 6789 -10893
+ Misses 19479 1141 -18338
+ Partials 778 0 -778
```
| [Flag](https://app.codecov.io/gh/apache/texera/pull/4711/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [access-control-service](https://app.codecov.io/gh/apache/texera/pull/4711/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [agent-service](https://app.codecov.io/gh/apache/texera/pull/4711/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [amber](https://app.codecov.io/gh/apache/texera/pull/4711/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [computing-unit-managing-service](https://app.codecov.io/gh/apache/texera/pull/4711/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [config-service](https://app.codecov.io/gh/apache/texera/pull/4711/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [file-service](https://app.codecov.io/gh/apache/texera/pull/4711/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [frontend](https://app.codecov.io/gh/apache/texera/pull/4711/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [python](https://app.codecov.io/gh/apache/texera/pull/4711/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `85.61% <ø> (+0.58%)` | :arrow_up: |
| [workflow-compiling-service](https://app.codecov.io/gh/apache/texera/pull/4711/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/texera/pull/4711?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(licensing): preserve all versions in check_binary_deps multi-version diff [texera]
Posted by "bobbai00 (via GitHub)" <gi...@apache.org>.
bobbai00 commented on PR #4711:
URL: https://github.com/apache/texera/pull/4711#issuecomment-4365584229
> can you add a test case for it? I think this tool is complex enough. we want to guard its behavior. you can add its run in python CI.
unit test added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(licensing): preserve all versions in check_binary_deps multi-version diff [texera]
Posted by "bobbai00 (via GitHub)" <gi...@apache.org>.
bobbai00 commented on code in PR #4711:
URL: https://github.com/apache/texera/pull/4711#discussion_r3177790685
##########
.github/workflows/build.yml:
##########
@@ -172,6 +172,10 @@ jobs:
python-version: "3.11"
- name: Show Python
run: python --version || python3 --version
+ - name: Unit-test licensing scripts
+ # Guard the binary-dep checker's behavior before we rely on it in
+ # the jar / python license checks below. Stdlib only, no install.
+ run: python3 -m unittest discover -s bin/licensing -p "test_*.py" -v
Review Comment:
moved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(licensing): preserve all versions in check_binary_deps multi-version diff [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #4711:
URL: https://github.com/apache/texera/pull/4711#issuecomment-4365004895
can you add a test case for it? I think this tool is complex enough. we want to guard its behavior. you can add its run in python CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(licensing): preserve all versions in check_binary_deps multi-version diff [texera]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #4711:
URL: https://github.com/apache/texera/pull/4711#issuecomment-4365696026
Backport to [`release/v1.1.0-incubating`](https://github.com/apache/texera/tree/release/v1.1.0-incubating) succeeded as [`4d90034`](https://github.com/apache/texera/commit/4d900340f66ab2635f222fd7a8191583ac6e9820). [Run](https://github.com/apache/texera/actions/runs/25273622168)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(licensing): preserve all versions in check_binary_deps multi-version diff [texera]
Posted by "bobbai00 (via GitHub)" <gi...@apache.org>.
bobbai00 merged PR #4711:
URL: https://github.com/apache/texera/pull/4711
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org