You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "cloud-fan (via GitHub)" <gi...@apache.org> on 2026/01/22 03:33:04 UTC
[PR] [SPARK-55115][INFRA][MASTER] Use composable Dockerfile for release builds [spark]
cloud-fan opened a new pull request, #53905:
URL: https://github.com/apache/spark/pull/53905
### What changes were proposed in this pull request?
This PR refactors the release Docker image build process to use a composable Dockerfile approach:
1. **`Dockerfile.base`**: A shared base image containing common tools (Ubuntu 22.04, R packages, Ruby/bundler, TeX, Node.js)
2. **`Dockerfile`**: Branch-specific image that extends the base with Java/Python versions and packages for this branch
3. **`do-release-docker.sh`**: Updated to build the base image first, then the branch-specific image
### Why are the changes needed?
Currently, each branch maintains its own full Dockerfile which leads to:
- Duplicated common configuration across branches
- Difficulty keeping base tools (R packages, Ruby, etc.) in sync
- Expired GPG keys or outdated base images affecting all branches
With the composable approach:
- Common tools are defined once in `Dockerfile.base`
- Each branch only specifies its unique Java/Python requirements
- Updates to base tools can be applied consistently
### Branch-specific versions (master/4.2-SNAPSHOT)
| Component | Version |
|-----------|---------|
| Java | 17 |
| Python | 3.10 |
| pandas | 2.3.3 |
| grpcio | 1.76.0 |
| protobuf | 6.33.0 |
| Sphinx | 4.5.0 |
| docutils | <0.18.0 |
### Does this PR introduce _any_ user-facing change?
No. This only affects the release infrastructure.
### How was this patch tested?
Docker image built and verified successfully on remote machine.
### Was this patch authored or co-authored using generative AI tooling?
Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55115][INFRA] Use composable Dockerfile for release builds [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #53905: [SPARK-55115][INFRA] Use composable Dockerfile for release builds
URL: https://github.com/apache/spark/pull/53905
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55115][INFRA][MASTER] Use composable Dockerfile for release builds [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #53905:
URL: https://github.com/apache/spark/pull/53905#issuecomment-3782339857
## JIRA Issue Information
=== Improvement SPARK-55115 ===
Summary: Use master branch's Dockerfile for release builds
Assignee: None
Status: Open
Affected: ["4.2.0"]
---
*This comment was automatically generated by GitHub Actions*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55115][INFRA][MASTER] Use composable Dockerfile for release builds [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53905:
URL: https://github.com/apache/spark/pull/53905#discussion_r2715183166
##########
dev/create-release/do-release-docker.sh:
##########
@@ -45,7 +45,7 @@ Options are:
-n : dry run mode. Performs checks and local builds, but do not upload anything.
-t [tag] : tag for the spark-rm docker image to use for building (default: "latest").
-j [path] : path to local JDK installation to use for building. By default the script will
- use openjdk17 installed in the docker image.
+ use openjdk8 installed in the docker image.
Review Comment:
```suggestion
use openjdk17 installed in the docker image.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55115][INFRA] Use composable Dockerfile for release builds [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #53905:
URL: https://github.com/apache/spark/pull/53905#issuecomment-3789312896
This is release infra change and CI failures is unrelated, thanks for review, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55115][INFRA][MASTER] Use composable Dockerfile for release builds [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #53905:
URL: https://github.com/apache/spark/pull/53905#issuecomment-3783503626
cc @dongjoon-hyun @HyukjinKwon
I want to use the master-branch base docker image for all active branches, as we can't really "freeze" the docker file in old branches. Things will go stale over time, and it's more stable to use the actively maintained docker file in the master branch. Every time I try to build release from an old branch, the docker environment is always broken. For example, building docker image with branch-3.5 fails now.
I have PRs for all the active branches, and the base docker file is the same:
https://github.com/apache/spark/pull/53906
https://github.com/apache/spark/pull/53907
https://github.com/apache/spark/pull/53908
I've done tests locally for all of them, for all the release steps (except for uploading). The produced artifacts are also verified.
After these are merged, I'll update the github action release job to always use the master-branch base docker file: https://github.com/apache/spark/pull/53890
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55115][INFRA][MASTER] Use composable Dockerfile for release builds [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53905:
URL: https://github.com/apache/spark/pull/53905#discussion_r2715206300
##########
dev/create-release/do-release-docker.sh:
##########
@@ -45,7 +45,7 @@ Options are:
-n : dry run mode. Performs checks and local builds, but do not upload anything.
-t [tag] : tag for the spark-rm docker image to use for building (default: "latest").
-j [path] : path to local JDK installation to use for building. By default the script will
- use openjdk17 installed in the docker image.
+ use openjdk8 installed in the docker image.
Review Comment:
```suggestion
use openjdk17 installed in the docker image.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org