You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "FelixYBW (via GitHub)" <gi...@apache.org> on 2026/10/23 22:19:23 UTC
[PR] [VL][CI] Use the pre-built images instead of installing tools at runtime [gluten]
FelixYBW opened a new pull request, #13423:
URL: https://github.com/apache/gluten/pull/13423
## What changes are proposed in this pull request?
Follow-up to #13422, part of #12288. **Draft until #13422 is merged and `docker_image.yml` has published the images.** Until then this PR's own CI will fail, because the runtime images and PySpark venvs don't exist yet.
This removes the runtime package installs from the PR-triggered jobs:
- **Spark UTs** (`spark-test-spark34/35/35-scala213/40/41/42`, enhanced, ANSI):
- A `Select PySpark <version>` step adds `/opt/pyspark/<version>/bin` to `GITHUB_PATH`. It replaces installing python39/python3.11 plus `pip install pyspark pandas pyarrow`.
- The per-step `yum install -y java-17-openjdk-devel` is dropped; JDK 17 is baked into every centos image.
- **TPC jobs:**
- `tpc-test-ubuntu` runs on `apache/gluten:ubuntu-22.04-runtime`.
- `tpc-test-centos8` and the Celeborn job run on `apache/gluten:centos-8-runtime`.
- The ARM `tpc-test-centos8` runs on `apache/gluten:centos-9-runtime`.
- The tzdata, mirror-list and JDK install steps are removed. The jobs keep selecting a JDK through `JAVA_HOME`.
- Hadoop and MinIO now come from the image via `install-spark-deps.sh`.
- **Runner-hosted jobs** (OOM, random-kill, Kafka Spark 3.5) use the runner's preinstalled `JAVA_HOME_8_X64` / `JAVA_HOME_17_X64` instead of `apt-get install openjdk-*`.
- **Native build, `cpp-test-udf-test`, Delta and `velox_native_build.yml`:** the gh/jq/bison/python39, `clang-tools-extra` and `regex` installs are removed.
After this, the only remaining runtime installs in PR jobs are the cuDF CUDA toolkit swap and `pip install requests` in the runner-hosted ANSI analysis jobs.
This conflicts with #13421 in `velox_kafka.yml` and `velox_backend_enhanced.yml`. I'll rebase whichever lands second.
## How was this patch tested?
- `actionlint` on all changed workflows: no new findings. The 6 findings in `velox_backend_ansi.yml` already exist on `main`.
- An install-step audit over the changed workflows finds no `yum`/`dnf`/`apt-get`/`pip`/gh/jq installs left, apart from the two noted above.
- The image contents were smoke-tested in #13422. The end-to-end check is this PR's CI once the images are published.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude claude-opus-5-5
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org