You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/07/28 20:45:24 UTC
[PR] fix: resolve Comet jar without hardcoding the SNAPSHOT qualifier [datafusion-comet]
andygrove opened a new pull request, #5108:
URL: https://github.com/apache/datafusion-comet/pull/5108
## Which issue does this PR close?
N/A
## Rationale for this change
The PyArrow UDF test harness locates the built Comet jar with a glob that hardcodes the `-SNAPSHOT` qualifier:
```
spark/target/comet-spark-{spark_tag}{scala_tag}-*-SNAPSHOT.jar
```
On a release branch the Maven version is the final release version with no `-SNAPSHOT` qualifier, so the glob matches nothing and every PyArrow UDF test errors out with `FileNotFoundError: Comet jar not found`. This is what happened on the 1.0.0 release branch in #5105, where the PyArrow UDF jobs were the only failing checks. The release branch runs the same workflows as `main`, so this class of breakage only surfaces after the release branch is cut and the version is bumped.
Two benchmark helpers have the same problem: the TPC docker build copies `comet-spark-spark3.5_2.12-*-SNAPSHOT.jar` out of the container, and `run_all_benchmarks.sh` defaults `COMET_JAR` to a hardcoded `0.14.0-SNAPSHOT` path that is already stale.
## What changes are included in this PR?
- `spark/src/test/resources/pyspark/conftest.py`: match any version suffix rather than only `-SNAPSHOT`, and widen the artifact filter so sources/tests/javadoc/shaded jars are still excluded.
- `benchmarks/tpc/infra/docker/Dockerfile.build-comet`: drop `-SNAPSHOT` from the jar copy glob.
- `benchmarks/pyspark/run_all_benchmarks.sh`: resolve the default `COMET_JAR` by glob instead of a hardcoded version, and fail with a clear message when no jar is found.
- `docs/source/contributor-guide/release_process.md`: enumerate the files that carry the Maven version (including the Gradle version catalog entry in the Iceberg diffs, which uses a different syntax than the Maven property in the Spark diffs), add a `git grep` verification step after the bump, and note that jar globs must not assume a `-SNAPSHOT` suffix.
## How are these changes tested?
Existing CI. The PyArrow UDF workflow triggers on `conftest.py`, so it exercises the new resolution logic on `main`, where the version still carries `-SNAPSHOT`. The non-SNAPSHOT path is verified on the 1.0.0 release branch, where the same `conftest.py` change turns the previously failing PyArrow UDF jobs green in #5105.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: resolve Comet jar without hardcoding the SNAPSHOT qualifier [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #5108:
URL: https://github.com/apache/datafusion-comet/pull/5108
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org