You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "cloud-fan (via GitHub)" <gi...@apache.org> on 2026/01/22 03:33:28 UTC
[PR] [SPARK-55115][INFRA][4.0] Use composable Dockerfile for release builds [spark]
cloud-fan opened a new pull request, #53907:
URL: https://github.com/apache/spark/pull/53907
### What changes were proposed in this pull request?
This PR refactors the release Docker image build process to use a composable Dockerfile approach:
1. **`Dockerfile.base`**: A shared base image containing common tools (Ubuntu 22.04, R packages, Ruby/bundler, TeX, Node.js)
2. **`Dockerfile`**: Branch-specific image that extends the base with Java/Python versions and packages for this branch
3. **`do-release-docker.sh`**: Updated to build the base image first, then the branch-specific image
### Why are the changes needed?
Currently, each branch maintains its own full Dockerfile which leads to:
- Duplicated common configuration across branches
- Difficulty keeping base tools (R packages, Ruby, etc.) in sync
- Expired GPG keys or outdated base images affecting all branches
With the composable approach:
- Common tools are defined once in `Dockerfile.base`
- Each branch only specifies its unique Java/Python requirements
- Updates to base tools can be applied consistently
### Branch-specific versions (4.0)
| Component | Version |
|-----------|---------|
| Java | 17 |
| Python | 3.9 (default), 3.10, PyPy 3.10 |
| pandas | 2.2.3 |
| grpcio | 1.67.0 |
| protobuf | 5.29.1 |
| Sphinx | 4.5.0 (on Python 3.9) |
| docutils | <0.18.0 |
**Note**: Branch 4.0 uses Python 3.9 as the default for documentation generation.
### Does this PR introduce _any_ user-facing change?
No. This only affects the release infrastructure.
### How was this patch tested?
Docker image built and verified successfully on remote machine.
### Was this patch authored or co-authored using generative AI tooling?
Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55115][INFRA][4.0] Use composable Dockerfile for release builds [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53907:
URL: https://github.com/apache/spark/pull/53907#discussion_r2715204914
##########
dev/create-release/do-release-docker.sh:
##########
@@ -45,7 +45,7 @@ Options are:
-n : dry run mode. Performs checks and local builds, but do not upload anything.
-t [tag] : tag for the spark-rm docker image to use for building (default: "latest").
-j [path] : path to local JDK installation to use for building. By default the script will
- use openjdk17 installed in the docker image.
+ use openjdk8 installed in the docker image.
Review Comment:
```suggestion
use openjdk17 installed in the docker image.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55115][INFRA][4.0] Use composable Dockerfile for release builds [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #53907:
URL: https://github.com/apache/spark/pull/53907#issuecomment-3782340618
## JIRA Issue Information
=== Improvement SPARK-55115 ===
Summary: Use master branch's Dockerfile for release builds
Assignee: None
Status: Open
Affected: ["4.2.0"]
---
*This comment was automatically generated by GitHub Actions*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55115][INFRA][4.0] Use composable Dockerfile for release builds [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53907:
URL: https://github.com/apache/spark/pull/53907#discussion_r2715186519
##########
dev/create-release/do-release-docker.sh:
##########
@@ -45,7 +45,7 @@ Options are:
-n : dry run mode. Performs checks and local builds, but do not upload anything.
-t [tag] : tag for the spark-rm docker image to use for building (default: "latest").
-j [path] : path to local JDK installation to use for building. By default the script will
- use openjdk17 installed in the docker image.
+ use openjdk8 installed in the docker image.
Review Comment:
```suggestion
use openjdk17 installed in the docker image.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55115][INFRA][4.0] Use composable Dockerfile for release builds [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #53907:
URL: https://github.com/apache/spark/pull/53907#issuecomment-3789324727
This is release infra change and CI failures is unrelated, thanks for review, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55115][INFRA][4.0] Use composable Dockerfile for release builds [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #53907: [SPARK-55115][INFRA][4.0] Use composable Dockerfile for release builds
URL: https://github.com/apache/spark/pull/53907
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org