You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "deepakpanda93 (via GitHub)" <gi...@apache.org> on 2026/08/03 12:52:37 UTC
[PR] feat(docker): merge image publishing into the docker build script [hudi]
deepakpanda93 opened a new pull request, #19489:
URL: https://github.com/apache/hudi/pull/19489
### Describe the issue this Pull Request addresses
Supersedes #14101, which was opened against a `docker/build_docker_images.sh` that has since been
rewritten on master. Rebasing it would have reverted that work, so this is a fresh port of the same
idea onto the current script. All review comments from #14101 are addressed here.
Publishing the Hudi demo images to the `apachehudi` Docker Hub organisation is a manual
`docker push` per image today, as documented under *Upload Updated Image to Repository on Docker
Hub*. There is no scripted path, so a release publishes nine images by hand across two tags each.
### Summary and Changelog
`docker/build_and_publish_docker_images.sh` replaces `docker/build_docker_images.sh` and does both
jobs, which is what @xushiyan asked for on #14101 rather than a second script duplicating the image
list and version variables.
- `--publish <true|false>`, default `false`, so a plain run only ever writes to the local Docker
daemon. Named rather than positional, so the command states what it is about to do.
- `--hadoop-version`, `--spark-version`, `--hive-version`, `--version-tag`, all optional.
- `--multi-arch <true|false>`, default `false`.
- The base image follows the Spark version, so a 4.x release selects `base_java17`.
- Every build passes `HADOOP_VERSION`, `SPARK_VERSION`, `HIVE_VERSION` and `BASE_IMAGE_TAG`. The
Dockerfiles carry differing `ARG` defaults, so without this an image can get a name that does not
describe its contents.
- The push step reports a per-tag summary and exits non-zero if any push failed, so a partial
publish is not read as success.
- `--help` prints the usage, including who publishing is for.
Two things worth reviewer attention:
**Default versions changed.** They were Hadoop 2.8.4 / Hive 2.3.10, but `setup_demo.sh` runs
`docker-compose_hadoop334_hive313_spark353`, so a bare build produced a set the demo never
referenced. Defaults are now Hadoop 3.3.4 / Hive 3.1.3 / Spark 3.5.3 to match. The old combination
is still reachable with the version flags.
**Multi-arch without publishing.** `buildx` cannot leave a multi-platform result nowhere: it either
pushes or loads. With `--publish true` it pushes, which is also what makes each image visible to the
next one, since every image after the base starts `FROM` the previous one. Without it the images are
loaded into the local **containerd** image store, using the builder from the current docker context.
Both are required: the legacy image store cannot hold a multi-platform image, and a
`docker-container` builder resolves `FROM` against registries only, so it silently builds the later
images against whatever is already on Docker Hub rather than against the run. The script checks for
the containerd store and stops with an explanation instead.
**One unrelated bug fixed because it blocked the above.** `base_java11` copied
`/etc/hadoop/mapred-site.xml.template` unconditionally, a file Hadoop 3.x does not ship, so Hadoop
3.x could not be built on the Java 11 base at all. The copy is now guarded. The guard is POSIX
`[ -f ... ]` rather than the `[[ -f ... ]]` used by the legacy `base` module, because the Dockerfile
`RUN` executes under `/bin/sh`, which is `dash` in these images; `[[` is not a command there, so the
condition is always false and the copy would be skipped even when the file exists. Happy to split
this into its own PR if reviewers prefer.
`docker/README.md` is updated throughout: a publishing section stating this is a maintainer action,
the version and multi-arch flags, and what keeping multi-arch images locally requires.
### Impact
Publishing the demo image set becomes one command for maintainers. A plain build now produces the
images `setup_demo.sh` actually consumes. Hadoop 3.x is buildable on the Java 11 base.
`build_docker_images.sh` is removed, so anyone invoking it by name will need to switch. Nothing in
the repository referenced it other than the README.
### Risk Level
low
### Documentation Update
`docker/README.md` is updated in this PR. No Hudi website change needed.
### Verification
Argument handling and the resulting `docker`/`buildx` command lines were checked by stubbing
`docker` on `PATH` and asserting what the script invokes:
| run | `docker build` | `buildx` | `buildx --push` | `buildx --load` | `docker push` |
| --- | --- | --- | --- | --- | --- |
| *(default)* | 9 | 0 | 0 | 0 | 0 |
| `--publish true` | 9 | 0 | 0 | 0 | 18 |
| `--multi-arch true` | 0 | 9 | 0 | 9 | 0 |
| `--multi-arch true --publish true` | 0 | 9 | 9 | 0 | 0 |
| `--publish yes`, `--publish` with no value, `--multi-arch yes` | rejected, exit 1 | | | | |
`--spark-version 4.0.1` selects `base_java17` and tags `hudi-hadoop_3.4.0-base-java17`;
`--version-tag rc1` overrides the `pom.xml`-derived tag.
Real builds: `base_java11` and `datanode` were built for `linux/amd64,linux/arm64` on Docker 28.1.1
with the containerd image store, confirming the Hadoop 3.3.4 guard and that `datanode` resolves its
parent from the local store. The full nine-image multi-arch run was not completed end to end, as it
is dominated by ~660 MB Apache mirror downloads per architecture.
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(docker): merge image publishing into the docker build script [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19489:
URL: https://github.com/apache/hudi/pull/19489#discussion_r3717718380
##########
docker/README.md:
##########
@@ -125,10 +127,28 @@ After new images are built, you can run the following script to bring up docker
./setup_demo.sh dev
```
-## Upload Updated Image to Repository on Docker Hub
+## Publishing Images to Docker Hub
+
+Publishing is a release action for Hudi maintainers, not part of the normal build or demo flow. It overwrites the
+images that every `setup_demo.sh` user pulls, so it needs write access to the `apachehudi/...` Docker Hub
+repositories and a prior `docker login`. If you are contributing a Dockerfile change, build without `--publish true` and
+let a maintainer publish.
-Once you have built the updated image locally, you can push the corresponding this repository of the image to the Docker
-Hud registry designated by its name or tag:
+`build_and_publish_docker_images.sh` publishes the whole image set it just built when given `--publish true`:
+
+```shell
+# Build the image set and publish it
+./build_and_publish_docker_images.sh --publish true
+```
+
+Each image is pushed under both the `latest` tag and the version tag, the latter taken from the root `pom.xml`
+unless `VERSION_TAG` overrides it. The script reports a per-tag summary and exits non-zero if any push fails, so a
Review Comment:
🤖 This section states the script "exits non-zero if any push fails, so a partial publish is not mistaken for a successful one," but the current `push_images()` in `build_and_publish_docker_images.sh` doesn't actually do that. Its failure branch only prints `Some pushes failed. Review logs above.` with no `exit 1`/`return 1`, and since each `docker push` runs inside an `if` (exempt from `set -e`) and the final command is an `echo`, the script exits 0 even when some images (or all, if inspect finds them missing) failed to push. Because publishing overwrites the shared `apachehudi/...` repos, a maintainer gating a release on the exit code would read a partial publish as success and leave the registry mixed (some images at the new tag, some stale). Could we either add an `exit 1` when `FAILURE_COUNT > 0` in the script, or soften this sentence to match the actual behavior? The per-tag summary claim is accurate; it's just the exit-code guarantee that isn't.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(docker): merge image publishing into the docker build script [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19489:
URL: https://github.com/apache/hudi/pull/19489#issuecomment-5166599340
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2c7464a3ca281d0c6922228d79607ba76dc55f3d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2c7464a3ca281d0c6922228d79607ba76dc55f3d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2c7464a3ca281d0c6922228d79607ba76dc55f3d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(docker): merge image publishing into the docker build script [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19489:
URL: https://github.com/apache/hudi/pull/19489#issuecomment-5180158205
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2c7464a3ca281d0c6922228d79607ba76dc55f3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15993",
"triggerID" : "2c7464a3ca281d0c6922228d79607ba76dc55f3d",
"triggerType" : "PUSH"
}, {
"hash" : "473008de97fe5e426821279b4bd022b2289effe6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16006",
"triggerID" : "473008de97fe5e426821279b4bd022b2289effe6",
"triggerType" : "PUSH"
}, {
"hash" : "4b64f420e91a858c5d43f3bba3e319e515366afe",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4b64f420e91a858c5d43f3bba3e319e515366afe",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 473008de97fe5e426821279b4bd022b2289effe6 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16006)
* 4b64f420e91a858c5d43f3bba3e319e515366afe UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(docker): merge image publishing into the docker build script [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19489:
URL: https://github.com/apache/hudi/pull/19489#issuecomment-5169137081
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2c7464a3ca281d0c6922228d79607ba76dc55f3d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15993",
"triggerID" : "2c7464a3ca281d0c6922228d79607ba76dc55f3d",
"triggerType" : "PUSH"
}, {
"hash" : "473008de97fe5e426821279b4bd022b2289effe6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "473008de97fe5e426821279b4bd022b2289effe6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2c7464a3ca281d0c6922228d79607ba76dc55f3d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15993)
* 473008de97fe5e426821279b4bd022b2289effe6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(docker): merge image publishing into the docker build script [hudi]
Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous closed pull request #19489: feat(docker): merge image publishing into the docker build script
URL: https://github.com/apache/hudi/pull/19489
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(docker): merge image publishing into the docker build script [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19489:
URL: https://github.com/apache/hudi/pull/19489#issuecomment-5166798693
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2c7464a3ca281d0c6922228d79607ba76dc55f3d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15993",
"triggerID" : "2c7464a3ca281d0c6922228d79607ba76dc55f3d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2c7464a3ca281d0c6922228d79607ba76dc55f3d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15993)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(docker): merge image publishing into the docker build script [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19489:
URL: https://github.com/apache/hudi/pull/19489#discussion_r3705824202
##########
docker/README.md:
##########
@@ -63,32 +64,39 @@ To build the Docker demo images with `docker` directly, rather than through the
`build_local_docker_images.sh` above, run the script from under `<HUDI_REPO_DIR>/docker`:
```shell
-# With no flags, builds Hadoop 2.8.4 / Spark 3.5.3 / Hive 2.3.10, matching
-# docker-compose_hadoop284_hive2310_spark353_{amd64,arm64}.yml
-./build_docker_images.sh
+# Builds Hadoop 3.3.4 / Spark 3.5.3 / Hive 3.1.3, matching
+# docker-compose_hadoop334_hive313_spark353_{amd64,arm64}.yml, which is the set setup_demo.sh runs
+./build_and_publish_docker_images.sh
```
-You can override the Hadoop, Spark, and Hive versions from the command line. If you plan to use `setup_demo.sh`,
-build the image set matching the default compose files first. For other flows, use one of the supported version
-combinations under `docker/compose`.
+To build one of the other version combinations under `docker/compose`, edit the version block near the top of the
+script before running it:
```shell
-# Matches setup_demo.sh and
-# docker-compose_hadoop334_hive313_spark353_{amd64,arm64}.yml
-./build_docker_images.sh --hadoop-version 3.3.4 --spark-version 3.5.3 --hive-version 3.1.3
-
-# Another supported combination is
-# docker-compose_hadoop340_hive313_spark401_{amd64,arm64}.yml
-./build_docker_images.sh --hadoop-version 3.4.0 --spark-version 4.0.1 --hive-version 3.1.3
+HADOOP_VERSION="3.3.4"
+SPARK_VERSION="3.5.3"
+HIVE_VERSION="3.1.3"
```
+The base image follows from `SPARK_VERSION`, so setting it to a 4.x release also switches the set to the Java 17
+base. If you plan to use `setup_demo.sh`, build the image set matching its compose file first.
+
`setup_demo.sh` currently defaults to `docker-compose_hadoop334_hive313_spark353_{amd64,arm64}.yml`. If you build a
different image set for the demo flow, update `COMPOSE_FILE_NAME` in `setup_demo.sh` to point to the matching compose
file before running the script. Run `./setup_demo.sh dev` to use your locally built images; a plain run pulls the
Docker Hub images over them.
-By default, the script builds images for the current machine architecture and derives the version tag from the root
-`pom.xml`. Use `--version-tag` to set an explicit tag if needed.
+The script builds images for the current machine architecture and tags each one `:latest` plus the Hudi version
+taken from the root `pom.xml`. Export `VERSION_TAG` to use a different second tag:
Review Comment:
🤖 This says to override the second tag by exporting `VERSION_TAG` (`VERSION_TAG=my-test ./build_and_publish_docker_images.sh`), but the PR description and the previous docs describe a `--version-tag <tag>` flag for the same purpose. It would help to state definitively whether the tag override is an environment variable, a `--version-tag` flag, or both — right now the two forms coexist in the same doc and readers won't know which one the script honors.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
docker/README.md:
##########
@@ -63,32 +64,39 @@ To build the Docker demo images with `docker` directly, rather than through the
`build_local_docker_images.sh` above, run the script from under `<HUDI_REPO_DIR>/docker`:
```shell
-# With no flags, builds Hadoop 2.8.4 / Spark 3.5.3 / Hive 2.3.10, matching
-# docker-compose_hadoop284_hive2310_spark353_{amd64,arm64}.yml
-./build_docker_images.sh
+# Builds Hadoop 3.3.4 / Spark 3.5.3 / Hive 3.1.3, matching
+# docker-compose_hadoop334_hive313_spark353_{amd64,arm64}.yml, which is the set setup_demo.sh runs
+./build_and_publish_docker_images.sh
```
-You can override the Hadoop, Spark, and Hive versions from the command line. If you plan to use `setup_demo.sh`,
-build the image set matching the default compose files first. For other flows, use one of the supported version
-combinations under `docker/compose`.
+To build one of the other version combinations under `docker/compose`, edit the version block near the top of the
Review Comment:
🤖 This section tells users to override versions by editing the `HADOOP_VERSION`/`SPARK_VERSION`/`HIVE_VERSION` block "near the top of the script," but the Multi-Arch example below (and the PR description) overrides the same values with `--hadoop-version 3.4.0 --spark-version 4.0.1 --hive-version 3.1.3` flags. These two instructions contradict each other. Since the script appears to accept `--hadoop-version`/`--spark-version`/`--hive-version` flags, it would be clearer (and less error-prone than hand-editing the script) to document the flags here as well, rather than telling users to modify the source.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(docker): merge image publishing into the docker build script [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19489:
URL: https://github.com/apache/hudi/pull/19489#issuecomment-5451355008
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2c7464a3ca281d0c6922228d79607ba76dc55f3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15993",
"triggerID" : "2c7464a3ca281d0c6922228d79607ba76dc55f3d",
"triggerType" : "PUSH"
}, {
"hash" : "473008de97fe5e426821279b4bd022b2289effe6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16006",
"triggerID" : "473008de97fe5e426821279b4bd022b2289effe6",
"triggerType" : "PUSH"
}, {
"hash" : "4b64f420e91a858c5d43f3bba3e319e515366afe",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16053",
"triggerID" : "4b64f420e91a858c5d43f3bba3e319e515366afe",
"triggerType" : "PUSH"
}, {
"hash" : "15a1b8e05f1da5b7c0141bcd428fa3ca22e15de1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16658",
"triggerID" : "15a1b8e05f1da5b7c0141bcd428fa3ca22e15de1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 15a1b8e05f1da5b7c0141bcd428fa3ca22e15de1 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16658)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(docker): merge image publishing into the docker build script [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19489:
URL: https://github.com/apache/hudi/pull/19489#issuecomment-5168875112
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19489?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 76.97%. Comparing base ([`637996c`](https://app.codecov.io/gh/apache/hudi/commit/637996c5ab2078dfd858a2a425da8f28af1ce789?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`2c7464a`](https://app.codecov.io/gh/apache/hudi/commit/2c7464a3ca281d0c6922228d79607ba76dc55f3d?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 2 commits behind head on master.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19489 +/- ##
=========================================
Coverage 76.97% 76.97%
Complexity 33855 33855
=========================================
Files 2575 2575
Lines 143378 143378
Branches 17573 17573
=========================================
Hits 110362 110362
Misses 24755 24755
Partials 8261 8261
```
| [Components](https://app.codecov.io/gh/apache/hudi/pull/19489/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hudi-common](https://app.codecov.io/gh/apache/hudi/pull/19489/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `82.27% <ø> (ø)` | |
| [hudi-client](https://app.codecov.io/gh/apache/hudi/pull/19489/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.83% <ø> (ø)` | |
| [hudi-flink](https://app.codecov.io/gh/apache/hudi/pull/19489/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.96% <ø> (ø)` | |
| [hudi-spark-datasource](https://app.codecov.io/gh/apache/hudi/pull/19489/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `75.10% <ø> (ø)` | |
| [hudi-utilities](https://app.codecov.io/gh/apache/hudi/pull/19489/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `73.65% <ø> (ø)` | |
| [hudi-cli](https://app.codecov.io/gh/apache/hudi/pull/19489/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `15.32% <ø> (ø)` | |
| [hudi-hadoop](https://app.codecov.io/gh/apache/hudi/pull/19489/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `63.49% <ø> (ø)` | |
| [hudi-sync](https://app.codecov.io/gh/apache/hudi/pull/19489/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.87% <ø> (ø)` | |
| [hudi-io](https://app.codecov.io/gh/apache/hudi/pull/19489/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `79.60% <ø> (ø)` | |
| [hudi-timeline-service](https://app.codecov.io/gh/apache/hudi/pull/19489/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `84.23% <ø> (ø)` | |
| [hudi-cloud](https://app.codecov.io/gh/apache/hudi/pull/19489/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `64.00% <ø> (ø)` | |
| [hudi-kafka-connect](https://app.codecov.io/gh/apache/hudi/pull/19489/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.20% <ø> (ø)` | |
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19489/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19489/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `43.75% <ø> (-0.02%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(docker): merge image publishing into the docker build script [hudi]
Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.
deepakpanda93 commented on code in PR #19489:
URL: https://github.com/apache/hudi/pull/19489#discussion_r3713204015
##########
docker/README.md:
##########
@@ -27,21 +27,15 @@ docker demo environment.
The `/hoodie` folder contains all the configs for assembling necessary docker images. The name and repository of each
docker image, e.g., `apachehudi/hudi-hadoop_2.8.4-prestobase_0.232`, is defined in the maven configuration file `pom.xml`.
-### Base images by Java version
+### Base image
-`build_docker_images.sh` auto-selects one of the two supported base images from `--spark-version`:
-
-| Base module | JDK | Used for |
-|---------------|---------|------------|
-| `base_java11` | Java 11 | Spark 3.x |
-| `base_java17` | Java 17 | Spark 4.0+ |
+`build_and_publish_docker_images.sh` builds the `base_java11` module, which is what Spark 3.x needs.
Review Comment:
Good catch, and the drift was mine — `473008de97fe` took the Java 17 base out of the README but left the `SPARK_MAJOR -ge 4` branch in the script, so the two disagreed.
Resolved in favour of the script, because that branch is live rather than dead: `docker/hoodie/hadoop/base_java17` exists and the selection is reachable through `--spark-version 4.x`. Verified by running it:
```
$ ./build_and_publish_docker_images.sh --version-tag t1 --spark-version 4.0.1 --publish false
Using Java 17 base image for Spark 4.0.1
Building hoodie/hadoop/base_java17 as:
- apachehudi/hudi-hadoop_3.3.4-base-java17:latest
```
The section now reads:
> `build_and_publish_docker_images.sh` picks the base module from `--spark-version`: `base_java11` for Spark 3.x, which is the default and what the compose files under `/compose` use, and `base_java17` for a 4.x version. The demo compose files are all Spark 3.x, so the Java 17 base is only reached if you ask for a 4.x `--spark-version` explicitly.
That keeps the docs truthful about what the script does without presenting Spark 4 as a supported demo path, which was the intent behind removing it in the first place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(docker): merge image publishing into the docker build script [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19489:
URL: https://github.com/apache/hudi/pull/19489#discussion_r3710738542
##########
docker/README.md:
##########
@@ -27,21 +27,15 @@ docker demo environment.
The `/hoodie` folder contains all the configs for assembling necessary docker images. The name and repository of each
docker image, e.g., `apachehudi/hudi-hadoop_2.8.4-prestobase_0.232`, is defined in the maven configuration file `pom.xml`.
-### Base images by Java version
+### Base image
-`build_docker_images.sh` auto-selects one of the two supported base images from `--spark-version`:
-
-| Base module | JDK | Used for |
-|---------------|---------|------------|
-| `base_java11` | Java 11 | Spark 3.x |
-| `base_java17` | Java 17 | Spark 4.0+ |
+`build_and_publish_docker_images.sh` builds the `base_java11` module, which is what Spark 3.x needs.
Review Comment:
🤖 This now says the script just "builds the `base_java11` module," but `build_and_publish_docker_images.sh` still auto-selects `base_java17` / `BASE_IMAGE_TAG=java17` when `--spark-version` is 4.x (the `SPARK_MAJOR -ge 4` branch), and that flag still accepts 4.x. Was dropping the Java 17 base from the docs intentional, or should this section still mention that a 4.x `--spark-version` switches to `base_java17`?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(docker): merge image publishing into the docker build script [hudi]
Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.
deepakpanda93 commented on code in PR #19489:
URL: https://github.com/apache/hudi/pull/19489#discussion_r3878840545
##########
docker/README.md:
##########
@@ -125,10 +127,28 @@ After new images are built, you can run the following script to bring up docker
./setup_demo.sh dev
```
-## Upload Updated Image to Repository on Docker Hub
+## Publishing Images to Docker Hub
+
+Publishing is a release action for Hudi maintainers, not part of the normal build or demo flow. It overwrites the
+images that every `setup_demo.sh` user pulls, so it needs write access to the `apachehudi/...` Docker Hub
+repositories and a prior `docker login`. If you are contributing a Dockerfile change, build without `--publish true` and
+let a maintainer publish.
-Once you have built the updated image locally, you can push the corresponding this repository of the image to the Docker
-Hud registry designated by its name or tag:
+`build_and_publish_docker_images.sh` publishes the whole image set it just built when given `--publish true`:
+
+```shell
+# Build the image set and publish it
+./build_and_publish_docker_images.sh --publish true
+```
+
+Each image is pushed under both the `latest` tag and the version tag, the latter taken from the root `pom.xml`
+unless `VERSION_TAG` overrides it. The script reports a per-tag summary and exits non-zero if any push fails, so a
Review Comment:
You are right, and I took the first option — added the `exit`, rather than softening the sentence, because the documented behaviour is the one worth having when publishing overwrites shared repositories.
Confirmed the mechanism before changing anything: `push_images()` ended on the `if/else` echo with no `return`/`exit`, each `docker push` sits inside an `if` so `set -e` skips it, and the missing-image branch only increments `FAILURE_COUNT` and `continue`s. The function therefore returned 0, and the script's last statement is the `if [ "$PUBLISH" = true ]` block, so the whole run exited 0.
Fix:
```sh
else
# Fail the script. Publishing overwrites the shared apachehudi/... repositories, so a partial
# push has to be distinguishable from a complete one by exit code alone: each docker push runs
# inside an if, which exempts it from set -e, so without this the script would report success
# after leaving the registry with some images at the new tag and some stale.
echo "Some pushes failed. Review logs above."
return 1
fi
```
`push_images` is called as a plain command, so `set -e` turns that non-zero return into a non-zero script exit.
Verified by extracting the real function into a harness with a stubbed `docker`, rather than only reading it. The partial case is the one you called out:
| scenario | before | after |
| --- | --- | --- |
| every push fails | 0 | **1** |
| partial publish (2 pushed, 2 failed) | **0** | **1** |
| all pushes succeed | 0 | 0 |
The "before" column is the current code with the `return 1` removed, so it reproduces the behaviour you described. The README sentence now matches the script, so no doc change was needed.
Two other things in this push, both from the rebase onto latest master:
- `bcaf2d48e6b7` modified `build_docker_images.sh`, the script this PR deletes, so git raised a modify/delete conflict. Resolving it as a plain delete would have dropped that work, so I ported it into `build_and_publish_docker_images.sh`: the `SPARK_VERSION` validation and the `hadoop-aws`/`aws-sdk`/`analyticsaccelerator-s3` selection keyed on the Spark major.minor, plus the three `--build-arg`s on both the buildx and single-arch paths. Worth being explicit that this was a silent failure mode rather than a build break: `spark_base/Dockerfile` defaults those args to the Spark 3 pairing (`3.3.4` / SDK v1), so without them a Spark 4 image would have built successfully with the wrong S3A jars.
- The README's S3A paragraph came from master naming the removed script; it now names the new one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(docker): merge image publishing into the docker build script [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19489:
URL: https://github.com/apache/hudi/pull/19489#issuecomment-5449941073
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2c7464a3ca281d0c6922228d79607ba76dc55f3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15993",
"triggerID" : "2c7464a3ca281d0c6922228d79607ba76dc55f3d",
"triggerType" : "PUSH"
}, {
"hash" : "473008de97fe5e426821279b4bd022b2289effe6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16006",
"triggerID" : "473008de97fe5e426821279b4bd022b2289effe6",
"triggerType" : "PUSH"
}, {
"hash" : "4b64f420e91a858c5d43f3bba3e319e515366afe",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16053",
"triggerID" : "4b64f420e91a858c5d43f3bba3e319e515366afe",
"triggerType" : "PUSH"
}, {
"hash" : "15a1b8e05f1da5b7c0141bcd428fa3ca22e15de1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16658",
"triggerID" : "15a1b8e05f1da5b7c0141bcd428fa3ca22e15de1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4b64f420e91a858c5d43f3bba3e319e515366afe Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16053)
* 15a1b8e05f1da5b7c0141bcd428fa3ca22e15de1 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16658)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(docker): merge image publishing into the docker build script [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19489:
URL: https://github.com/apache/hudi/pull/19489#issuecomment-5180229738
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2c7464a3ca281d0c6922228d79607ba76dc55f3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15993",
"triggerID" : "2c7464a3ca281d0c6922228d79607ba76dc55f3d",
"triggerType" : "PUSH"
}, {
"hash" : "473008de97fe5e426821279b4bd022b2289effe6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16006",
"triggerID" : "473008de97fe5e426821279b4bd022b2289effe6",
"triggerType" : "PUSH"
}, {
"hash" : "4b64f420e91a858c5d43f3bba3e319e515366afe",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16053",
"triggerID" : "4b64f420e91a858c5d43f3bba3e319e515366afe",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 473008de97fe5e426821279b4bd022b2289effe6 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16006)
* 4b64f420e91a858c5d43f3bba3e319e515366afe Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16053)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(docker): merge image publishing into the docker build script [hudi]
Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous commented on PR #19489:
URL: https://github.com/apache/hudi/pull/19489#issuecomment-6701220876
Temporarily closing this PR while we debug a GitHub settings error on master (#20471); it will be reopened shortly, no action needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(docker): merge image publishing into the docker build script [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19489:
URL: https://github.com/apache/hudi/pull/19489#issuecomment-5169192843
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2c7464a3ca281d0c6922228d79607ba76dc55f3d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15993",
"triggerID" : "2c7464a3ca281d0c6922228d79607ba76dc55f3d",
"triggerType" : "PUSH"
}, {
"hash" : "473008de97fe5e426821279b4bd022b2289effe6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16006",
"triggerID" : "473008de97fe5e426821279b4bd022b2289effe6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2c7464a3ca281d0c6922228d79607ba76dc55f3d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15993)
* 473008de97fe5e426821279b4bd022b2289effe6 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16006)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(docker): merge image publishing into the docker build script [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19489:
URL: https://github.com/apache/hudi/pull/19489#issuecomment-5173007582
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2c7464a3ca281d0c6922228d79607ba76dc55f3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15993",
"triggerID" : "2c7464a3ca281d0c6922228d79607ba76dc55f3d",
"triggerType" : "PUSH"
}, {
"hash" : "473008de97fe5e426821279b4bd022b2289effe6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16006",
"triggerID" : "473008de97fe5e426821279b4bd022b2289effe6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 473008de97fe5e426821279b4bd022b2289effe6 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16006)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(docker): merge image publishing into the docker build script [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19489:
URL: https://github.com/apache/hudi/pull/19489#issuecomment-5172505198
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2c7464a3ca281d0c6922228d79607ba76dc55f3d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15993",
"triggerID" : "2c7464a3ca281d0c6922228d79607ba76dc55f3d",
"triggerType" : "PUSH"
}, {
"hash" : "473008de97fe5e426821279b4bd022b2289effe6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16006",
"triggerID" : "473008de97fe5e426821279b4bd022b2289effe6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2c7464a3ca281d0c6922228d79607ba76dc55f3d Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15993)
* 473008de97fe5e426821279b4bd022b2289effe6 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16006)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(docker): merge image publishing into the docker build script [hudi]
Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.
deepakpanda93 commented on code in PR #19489:
URL: https://github.com/apache/hudi/pull/19489#discussion_r3713200799
##########
docker/README.md:
##########
@@ -63,32 +64,39 @@ To build the Docker demo images with `docker` directly, rather than through the
`build_local_docker_images.sh` above, run the script from under `<HUDI_REPO_DIR>/docker`:
```shell
-# With no flags, builds Hadoop 2.8.4 / Spark 3.5.3 / Hive 2.3.10, matching
-# docker-compose_hadoop284_hive2310_spark353_{amd64,arm64}.yml
-./build_docker_images.sh
+# Builds Hadoop 3.3.4 / Spark 3.5.3 / Hive 3.1.3, matching
+# docker-compose_hadoop334_hive313_spark353_{amd64,arm64}.yml, which is the set setup_demo.sh runs
+./build_and_publish_docker_images.sh
```
-You can override the Hadoop, Spark, and Hive versions from the command line. If you plan to use `setup_demo.sh`,
-build the image set matching the default compose files first. For other flows, use one of the supported version
-combinations under `docker/compose`.
+To build one of the other version combinations under `docker/compose`, edit the version block near the top of the
Review Comment:
You are right, and the flags are the better instruction — fixed.
The script does take `--hadoop-version`, `--spark-version` and `--hive-version` (option parsing at lines 84-86), so telling readers to hand edit the source was both inconsistent with the example below and worse advice. That section now reads:
```shell
./build_and_publish_docker_images.sh --hadoop-version 3.3.4 --spark-version 3.5.3 --hive-version 3.1.3
```
with a note that those are the defaults and each can be set on its own. No `edit the version block` instruction remains anywhere in the README.
##########
docker/README.md:
##########
@@ -63,32 +64,39 @@ To build the Docker demo images with `docker` directly, rather than through the
`build_local_docker_images.sh` above, run the script from under `<HUDI_REPO_DIR>/docker`:
```shell
-# With no flags, builds Hadoop 2.8.4 / Spark 3.5.3 / Hive 2.3.10, matching
-# docker-compose_hadoop284_hive2310_spark353_{amd64,arm64}.yml
-./build_docker_images.sh
+# Builds Hadoop 3.3.4 / Spark 3.5.3 / Hive 3.1.3, matching
+# docker-compose_hadoop334_hive313_spark353_{amd64,arm64}.yml, which is the set setup_demo.sh runs
+./build_and_publish_docker_images.sh
```
-You can override the Hadoop, Spark, and Hive versions from the command line. If you plan to use `setup_demo.sh`,
-build the image set matching the default compose files first. For other flows, use one of the supported version
-combinations under `docker/compose`.
+To build one of the other version combinations under `docker/compose`, edit the version block near the top of the
+script before running it:
```shell
-# Matches setup_demo.sh and
-# docker-compose_hadoop334_hive313_spark353_{amd64,arm64}.yml
-./build_docker_images.sh --hadoop-version 3.3.4 --spark-version 3.5.3 --hive-version 3.1.3
-
-# Another supported combination is
-# docker-compose_hadoop340_hive313_spark401_{amd64,arm64}.yml
-./build_docker_images.sh --hadoop-version 3.4.0 --spark-version 4.0.1 --hive-version 3.1.3
+HADOOP_VERSION="3.3.4"
+SPARK_VERSION="3.5.3"
+HIVE_VERSION="3.1.3"
```
+The base image follows from `SPARK_VERSION`, so setting it to a 4.x release also switches the set to the Java 17
+base. If you plan to use `setup_demo.sh`, build the image set matching its compose file first.
+
`setup_demo.sh` currently defaults to `docker-compose_hadoop334_hive313_spark353_{amd64,arm64}.yml`. If you build a
different image set for the demo flow, update `COMPOSE_FILE_NAME` in `setup_demo.sh` to point to the matching compose
file before running the script. Run `./setup_demo.sh dev` to use your locally built images; a plain run pulls the
Docker Hub images over them.
-By default, the script builds images for the current machine architecture and derives the version tag from the root
-`pom.xml`. Use `--version-tag` to set an explicit tag if needed.
+The script builds images for the current machine architecture and tags each one `:latest` plus the Hudi version
+taken from the root `pom.xml`. Export `VERSION_TAG` to use a different second tag:
Review Comment:
Both work, and the docs now say so along with the order they resolve in. The script does:
```sh
if [ -n "$VERSION_TAG_ARG" ]; then # --version-tag
VERSION_TAG="$VERSION_TAG_ARG"
elif [ -z "${VERSION_TAG:-}" ]; then # exported VERSION_TAG
VERSION_TAG=$(get_hudi_project_version ...)
fi
```
so it is flag, then environment variable, then the version from the root `pom.xml`. Confirmed by running it rather than only reading it:
| invocation | resulting tag |
| --- | --- |
| `--version-tag from-flag` | `from-flag` |
| `VERSION_TAG=from-env` | `from-env` |
| both together | `from-flag` |
The README now documents both forms side by side and states that the flag wins.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(docker): merge image publishing into the docker build script [hudi]
Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.
deepakpanda93 closed pull request #19489: feat(docker): merge image publishing into the docker build script
URL: https://github.com/apache/hudi/pull/19489
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(docker): merge image publishing into the docker build script [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19489:
URL: https://github.com/apache/hudi/pull/19489#issuecomment-5181836647
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2c7464a3ca281d0c6922228d79607ba76dc55f3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15993",
"triggerID" : "2c7464a3ca281d0c6922228d79607ba76dc55f3d",
"triggerType" : "PUSH"
}, {
"hash" : "473008de97fe5e426821279b4bd022b2289effe6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16006",
"triggerID" : "473008de97fe5e426821279b4bd022b2289effe6",
"triggerType" : "PUSH"
}, {
"hash" : "4b64f420e91a858c5d43f3bba3e319e515366afe",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16053",
"triggerID" : "4b64f420e91a858c5d43f3bba3e319e515366afe",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4b64f420e91a858c5d43f3bba3e319e515366afe Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16053)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(docker): merge image publishing into the docker build script [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19489:
URL: https://github.com/apache/hudi/pull/19489#issuecomment-5449891826
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2c7464a3ca281d0c6922228d79607ba76dc55f3d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15993",
"triggerID" : "2c7464a3ca281d0c6922228d79607ba76dc55f3d",
"triggerType" : "PUSH"
}, {
"hash" : "473008de97fe5e426821279b4bd022b2289effe6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16006",
"triggerID" : "473008de97fe5e426821279b4bd022b2289effe6",
"triggerType" : "PUSH"
}, {
"hash" : "4b64f420e91a858c5d43f3bba3e319e515366afe",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16053",
"triggerID" : "4b64f420e91a858c5d43f3bba3e319e515366afe",
"triggerType" : "PUSH"
}, {
"hash" : "15a1b8e05f1da5b7c0141bcd428fa3ca22e15de1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "15a1b8e05f1da5b7c0141bcd428fa3ca22e15de1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4b64f420e91a858c5d43f3bba3e319e515366afe Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16053)
* 15a1b8e05f1da5b7c0141bcd428fa3ca22e15de1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org