You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "xiangfu0 (via GitHub)" <gi...@apache.org> on 2026/04/13 10:13:57 UTC
[PR] Upgrade Docker base images and build tooling [pinot]
xiangfu0 opened a new pull request, #18178:
URL: https://github.com/apache/pinot/pull/18178
## Summary
This PR upgrades the Docker base images and build tooling to address EOL operating systems and outdated dependencies:
**Security / EOL fixes:**
- Migrate `amazoncorretto` build image from `debian:buster-slim` (EOL since Jun 2022) to `debian:bookworm-slim`; update Corretto GPG key import to modern `gpg --dearmor` approach
- Migrate `amazoncorretto` runtime image from `al2-jdk` to `al2023-jdk` (Amazon Linux 2 is EOL)
- Migrate `ms-openjdk` runtime base from `ubuntu:24.10` (non-LTS, ~9 month lifespan) to `ubuntu:24.04` LTS
**Build improvements:**
- Replace `libboost-all-dev` with `libboost-dev` in all build images to reduce image size
- Upgrade Maven from `3.9.9` to `3.9.14` in all build images
- Add `make -j$(nproc)` for parallel Thrift compilation; clean up temp build artifacts
- Make JDK version dynamic (`ARG JAVA_VERSION`) in `amazoncorretto` build image
- Add `-ntp` (no transfer progress) flag to Maven build in `Dockerfile.build`
- Pin build stage platform to `linux/amd64` in `Dockerfile.package` for cross-platform correctness
- Remove unnecessary packages (`vim`, `python-is-python3`, `linux-perf`) from runtime images
**Workflow improvements:**
- Expand base-image build matrix to JDK 11/17/21/25 (was only JDK 11)
- Add manifest push job to `build-pinot-base-docker-image` workflow
- Add Java 25 to `build-multi-arch-pinot-docker-image` matrix
- Add arm64 support to superset Docker build script
- Change default manifest tag from `11-amazoncorretto` to `17-amazoncorretto`
## Test plan
- [ ] Verify `build-pinot-base-docker-image` workflow builds successfully for all JDK versions and distros
- [ ] Verify `build-multi-arch-pinot-docker-image` workflow produces correct multi-arch manifests
- [ ] Verify Pinot starts correctly in the updated runtime containers
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Upgrade Docker base images and build tooling [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18178:
URL: https://github.com/apache/pinot/pull/18178#issuecomment-4236044387
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/18178?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 55.22%. Comparing base ([`22b3b6f`](https://app.codecov.io/gh/apache/pinot/commit/22b3b6fe6645dbab9e3e5ad8c43aad29348d7fab?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`1c28d71`](https://app.codecov.io/gh/apache/pinot/commit/1c28d712827f10eaa4162580e8a9a5dd265da954?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
> :exclamation: There is a different number of reports uploaded between BASE (22b3b6f) and HEAD (1c28d71). Click for more details.
>
> <details><summary>HEAD has 36 uploads less than BASE</summary>
>
>| Flag | BASE (22b3b6f) | HEAD (1c28d71) |
>|------|------|------|
>|java-21|5|1|
>|unittests1|2|1|
>|unittests|4|1|
>|temurin|10|1|
>|java-11|5|0|
>|unittests2|2|0|
>|integration|6|0|
>|integration1|2|0|
>|integration2|2|0|
>|custom-integration1|2|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18178 +/- ##
============================================
- Coverage 63.30% 55.22% -8.09%
+ Complexity 1627 837 -790
============================================
Files 3226 2531 -695
Lines 196636 145603 -51033
Branches 30401 23407 -6994
============================================
- Hits 124490 80406 -44084
+ Misses 62170 58256 -3914
+ Partials 9976 6941 -3035
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/18178/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/18178/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/18178/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/18178/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/18178/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-11](https://app.codecov.io/gh/apache/pinot/pull/18178/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/18178/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.22% <ø> (-8.07%)` | :arrow_down: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/18178/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.22% <ø> (-8.09%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/18178/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.22% <ø> (-8.09%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/18178/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.22% <ø> (-0.06%)` | :arrow_down: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/18178/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/18178?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Upgrade Docker base images and build tooling [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on PR #18178:
URL: https://github.com/apache/pinot/pull/18178#issuecomment-4247432707
> Have the Docker secrets already been updated in this repo?
yes, I added those
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Upgrade Docker base images and build tooling [pinot]
Posted by "yashmayya (via GitHub)" <gi...@apache.org>.
yashmayya commented on code in PR #18178:
URL: https://github.com/apache/pinot/pull/18178#discussion_r3082256682
##########
.github/workflows/scripts/docker/.pinot_base_docker_image_build_and_push.sh:
##########
@@ -30,11 +30,12 @@ fi
cd docker/images/pinot-base/pinot-base-${BASE_IMAGE_TYPE}
-docker buildx build \
Review Comment:
Why is this no longer needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Upgrade Docker base images and build tooling [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #18178:
URL: https://github.com/apache/pinot/pull/18178#discussion_r3083812162
##########
docker/images/pinot/README.md:
##########
@@ -71,12 +71,12 @@ For users on Mac M1 chips, they need to build the images with arm64 base image,
or just run the docker build script directly
```SHELL
-docker build -t pinot:latest --no-cache --network=host --build-arg PINOT_GIT_URL=https://github.com/apache/pinot.git --build-arg PINOT_BRANCH=master --build-arg JDK_VERSION=11 --build-arg OPENJDK_IMAGE=arm64v8/openjdk -f Dockerfile .
+docker build -t pinot:latest --no-cache --network=host --build-arg PINOT_GIT_REF=master --build-arg JDK_VERSION=11 --build-arg OPENJDK_IMAGE=arm64v8/openjdk -f Dockerfile .
```
Note that if you are not on arm64 machine, you can still build the image by turning on the experimental feature of docker, and add `--platform linux/arm64` into the `docker build ...` script, e.g.
```SHELL
-docker build -t pinot:latest --platform linux/arm64 --no-cache --network=host --build-arg PINOT_GIT_URL=https://github.com/apache/pinot.git --build-arg PINOT_BRANCH=master --build-arg JDK_VERSION=11 --build-arg OPENJDK_IMAGE=arm64v8/openjdk -f Dockerfile .
+docker build -t pinot:latest --platform linux/arm64 --no-cache --network=host --build-arg PINOT_GIT_REF=master --build-arg JDK_VERSION=11 --build-arg OPENJDK_IMAGE=arm64v8/openjdk -f Dockerfile .
```
Review Comment:
README arm64 build commands pass --build-arg OPENJDK_IMAGE=... but docker/images/pinot/Dockerfile does not reference OPENJDK_IMAGE anywhere, so these instructions are ineffective/misleading. Please update the documented build command(s) to match the current Dockerfile inputs (e.g., PINOT_BASE_IMAGE_TAG/JDK_VERSION) or remove the unused build-arg from the docs.
##########
.github/workflows/nightly-pull-upstream-master-branch.yml:
##########
@@ -0,0 +1,51 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+name: Sync with Upstream
+
+on:
+ schedule:
+ - cron: "*/5 * * * *" # run at 0 AM UTC
+ workflow_dispatch:
+ inputs: {}
+
+jobs:
+ sync:
+ runs-on: ubuntu-latest
+ steps:
+ - name: Checkout forked repository
+ uses: actions/checkout@v5
+ with:
+ ref: "master"
+
+ - name: Merge changes from upstream master to your branch
+ env:
+ GH_TOKEN: ${{ secrets.GH_TOKEN }}
+ run: |
+ git version
+ git clone https://xiangfu0:${GH_TOKEN}@github.com/apachepinot/pinot-fork.git pinot-fork
+ cd pinot-fork
+ git config --global user.name 'Pinot Bot'
+ git config --global user.email 'apachepinot@gmail.com'
+ git remote add upstream https://github.com/apache/pinot.git
+ git fetch upstream
+ git log --pretty=oneline | head -3
+ git pull --rebase upstream master
+ git status
+ git diff
+ git push origin master -f
Review Comment:
This workflow appears to be specific to a personal fork: it clones `apachepinot/pinot-fork` using a token embedded in the URL, force-pushes `master`, and runs every 5 minutes (comment says 0 AM). This is a serious security/ops risk for the upstream repo and is not mentioned in the PR description. Please remove this workflow from the Apache Pinot repo, or rework it to avoid hardcoded fork URLs, avoid force-push, and use a safe schedule.
```suggestion
- cron: "0 0 * * *" # run at 0 AM UTC
workflow_dispatch:
inputs: {}
permissions:
contents: write
jobs:
sync:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v5
with:
ref: "master"
- name: Merge changes from upstream master
run: |
git version
git config --global user.name 'Pinot Bot'
git config --global user.email 'apachepinot@gmail.com'
git remote get-url upstream >/dev/null 2>&1 || git remote add upstream https://github.com/apache/pinot.git
git fetch upstream master
git log --pretty=oneline | head -3
git merge --ff-only upstream/master
git status
git diff
git push origin HEAD:master
```
##########
.github/workflows/nightly-build-multi-arch-superset-docker-image.yml:
##########
@@ -0,0 +1,106 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+name: Nightly Multi Arch Superset Pinot Docker Image Build and Publish
+
+on:
+ schedule:
+ - cron: "0 0 * * *" # run at 0 AM UTC
+
+jobs:
+ generate-superset-info:
+ name: Generate Superset Build Info
+ runs-on: ubuntu-24.04
+ outputs:
+ tags: ${{ steps.gen-info.outputs.tags }}
+ steps:
+ - id: gen-info
+ env:
+ SUPERSET_IMAGE_TAG: master-py310
+ run: |
+ docker pull apache/superset:${SUPERSET_IMAGE_TAG}
+ COMMIT_ID=$(docker images apache/superset:${SUPERSET_IMAGE_TAG} --format "{{.ID}}")
+ DATE=$(date +%Y%m%d)
+ echo "tags=${COMMIT_ID}-${DATE},latest" >> $GITHUB_OUTPUT
+ build-superset-docker-image:
+ name: Build Superset ${{ matrix.arch }}
+ runs-on: ${{ matrix.runner }}
+ needs: generate-superset-info
+ strategy:
+ matrix:
+ arch: ["amd64", "arm64"]
+ include:
+ - arch: amd64
+ runner: ubuntu-24.04
+ superset_suffix: ""
+ - arch: arm64
+ runner: ubuntu-24.04-arm
+ superset_suffix: "-arm"
+ steps:
+ - name: Login to DockerHub
+ uses: docker/login-action@v4
+ with:
+ username: ${{ secrets.DOCKERHUB_USERNAME }}
+ password: ${{ secrets.DOCKERHUB_TOKEN }}
+ - uses: docker/setup-buildx-action@v4
+ name: Set up Docker Buildx
+ - uses: actions/checkout@v5
+ - name: Build and push
+ env:
+ DOCKER_IMAGE_NAME: apachepinot/pinot-superset
+ SUPERSET_IMAGE_TAG: "master-py310${{ matrix.superset_suffix }}"
+ TAGS: "${{ needs.generate-superset-info.outputs.tags }}"
+ ARCH: ${{ matrix.arch }}
+ run: |
+ cd docker/images/pinot-superset
+
+ DOCKER_BUILD_TAGS=""
+ for tag in $(echo ${TAGS} | tr "," " "); do
+ DOCKER_BUILD_TAGS+=" --tag ${DOCKER_IMAGE_NAME}:${tag}-${ARCH} "
+ done
+
+ docker build \
+ --no-cache \
+ --platform=linux/${ARCH} \
+ --file Dockerfile \
+ --build-arg SUPERSET_IMAGE_TAG=${SUPERSET_IMAGE_TAG} \
+ ${DOCKER_BUILD_TAGS} \
+ --push \
+ .
+ create-superset-manifest:
Review Comment:
The build step uses `docker build ... --push`, but `--push` is a buildx-only flag (`docker buildx build --push`). Even though buildx is set up, this command is still `docker build` and is likely to fail. Switch to `docker buildx build` (or use an explicit `--output=type=registry` flow) for pushing.
##########
.github/workflows/pinot_vuln_check.yml:
##########
@@ -44,8 +44,7 @@ jobs:
env:
DOCKER_FILE_BASE_DIR: "docker/images/pinot"
DOCKER_IMAGE_NAME: "apachepinot/pinot"
- PINOT_GIT_URL: ${{ github.event.inputs.gitUrl }}
- PINOT_BRANCH: ${{ env.GITHUB_REF }}
+ PINOT_GIT_REF: ${{ env.GITHUB_REF }}
Review Comment:
`PINOT_GIT_REF: ${{ env.GITHUB_REF }}` is unlikely to resolve as intended because `env` context only includes workflow-defined env vars, not runner-provided `GITHUB_REF`. This will typically pass an empty value and fall back to master in the script, causing the vulnerability check to scan the wrong code. Use `${{ github.sha }}` (works with `git checkout`) or `${{ github.ref }}` / `$GITHUB_REF` explicitly.
```suggestion
PINOT_GIT_REF: ${{ github.sha }}
```
##########
.github/workflows/scripts/docker/.pinot_compile_and_push_build_image.sh:
##########
@@ -0,0 +1,69 @@
+#!/bin/bash -x
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# Compiles Pinot from source on amd64 and pushes the build image to DockerHub
+# so that per-arch package jobs can pull it.
+#
+# The build image is tagged by JDK version only (not by JDK distro) because
+# JVM bytecode is identical regardless of the JDK vendor used to compile it.
+# Both amazoncorretto and ms-openjdk runtime images share the same build image.
+
+set -e
+
+if [ -z "${DOCKER_IMAGE_NAME}" ]; then
+ DOCKER_IMAGE_NAME="apachepinot/pinot"
+fi
+if [ -z "${DOCKER_FILE_BASE_DIR}" ]; then
+ echo "DOCKER_FILE_BASE_DIR is required" >&2
+ exit 1
+fi
+if [ -z "${JDK_VERSION}" ]; then
+ echo "JDK_VERSION is required" >&2
+ exit 1
+fi
+if [ -z "${PINOT_GIT_REF}" ]; then
+ echo "PINOT_GIT_REF is required" >&2
+ exit 1
+fi
+
+cd "${DOCKER_FILE_BASE_DIR}"
+
+# Tag the build image by JDK version only — distro-agnostic.
+# Include PINOT_GIT_REF (commit SHA) to avoid collisions between concurrent runs.
+PINOT_BUILD_IMAGE_TAG="${JDK_VERSION}-amd64"
+BUILD_IMAGE_REMOTE_TAG="${DOCKER_IMAGE_NAME}:build-${JDK_VERSION}-${PINOT_GIT_REF}"
+
+# Use amazoncorretto as the compile-time base; the resulting JARs are identical
+# to those produced by any other JDK vendor at the same version.
+COMPILE_BASE_IMAGE_TAG="${JDK_VERSION}-amazoncorretto"
+
+echo "Building docker image for platform: amd64 with tag: pinot-build:${PINOT_BUILD_IMAGE_TAG}"
+docker build \
+ --no-cache \
+ --platform amd64 \
+ --file Dockerfile.build \
+ --build-arg "PINOT_GIT_REF=${PINOT_GIT_REF}" \
+ --build-arg "JDK_VERSION=${JDK_VERSION}" \
+ --build-arg "PINOT_BASE_IMAGE_TAG=${COMPILE_BASE_IMAGE_TAG}" \
+ --tag "pinot-build:${PINOT_BUILD_IMAGE_TAG}" \
+ --tag "${BUILD_IMAGE_REMOTE_TAG}" \
+ .
Review Comment:
`docker build --platform amd64` is not a valid platform value for Docker/BuildKit (it should be `linux/amd64`). As written, this build step is likely to fail on CI. Use `--platform linux/amd64` (and similarly for other uses of `arm64`).
##########
.github/workflows/scripts/docker/.pinot_build_info_gen.sh:
##########
@@ -17,17 +17,14 @@
# specific language governing permissions and limitations
# under the License.
#
-if [ -z "${PINOT_GIT_URL}" ]; then
- PINOT_GIT_URL="https://github.com/apache/pinot.git"
-fi
-if [ -z "${PINOT_BRANCH}" ]; then
- PINOT_BRANCH="master"
+if [ -z "${PINOT_GIT_REF}" ]; then
+ PINOT_GIT_REF="master"
fi
# Get pinot commit id
ROOT_DIR=$(pwd)
rm -rf /tmp/pinot
-git clone -b ${PINOT_BRANCH} --single-branch ${PINOT_GIT_URL} /tmp/pinot
+git clone -b ${PINOT_GIT_REF} --single-branch https://github.com/apache/pinot.git /tmp/pinot
cd /tmp/pinot
Review Comment:
`git clone -b ${PINOT_GIT_REF} --single-branch ...` only supports branches/tags; it will fail when PINOT_GIT_REF is a commit SHA (which the workflows allow). Consider cloning the repo normally and running `git checkout ${PINOT_GIT_REF}` so any ref works consistently.
```suggestion
git clone https://github.com/apache/pinot.git /tmp/pinot
cd /tmp/pinot
git checkout "${PINOT_GIT_REF}"
```
##########
.github/workflows/scripts/docker/.pinot_docker_image_build_and_push.sh:
##########
@@ -34,7 +31,7 @@ fi
# Get pinot commit id
ROOT_DIR=`pwd`
rm -rf /tmp/pinot
-git clone -b ${PINOT_BRANCH} --single-branch ${PINOT_GIT_URL} /tmp/pinot
+git clone -b ${PINOT_GIT_REF} --single-branch https://github.com/apache/pinot.git /tmp/pinot
cd /tmp/pinot
Review Comment:
`git clone -b ${PINOT_GIT_REF} --single-branch ...` only works when PINOT_GIT_REF is a branch or tag; it will fail for commit SHAs even though the workflows/PR description treat this as a general git ref. To support branch/tag/SHA uniformly, clone without `-b` (or with a known branch) and then `git checkout ${PINOT_GIT_REF}` after cloning.
```suggestion
git clone https://github.com/apache/pinot.git /tmp/pinot
cd /tmp/pinot
git checkout ${PINOT_GIT_REF}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Upgrade Docker base images and build tooling [pinot]
Posted by "yashmayya (via GitHub)" <gi...@apache.org>.
yashmayya commented on code in PR #18178:
URL: https://github.com/apache/pinot/pull/18178#discussion_r3082231032
##########
.github/workflows/scripts/docker/.pinot_package_single_platform.sh:
##########
@@ -0,0 +1,84 @@
+#!/bin/bash -x
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# Packages Pinot for a single platform. Pulls the build image from DockerHub,
+# tags it locally, then builds and pushes the platform-specific package image.
+
+set -e
+
+if [ -z "${DOCKER_IMAGE_NAME}" ]; then
+ DOCKER_IMAGE_NAME="apachepinot/pinot"
+fi
+if [ -z "${BUILD_ARCH}" ]; then
+ echo "BUILD_ARCH is required (amd64 or arm64)" >&2
+ exit 1
+fi
+if [ -z "${DOCKER_FILE_BASE_DIR}" ]; then
+ echo "DOCKER_FILE_BASE_DIR is required" >&2
+ exit 1
+fi
+
+cd "${DOCKER_FILE_BASE_DIR}"
+
+# Pull the commit-scoped build image from DockerHub and tag it locally.
+# PINOT_BRANCH is the short commit SHA resolved by generate-build-info; including
+# it in the tag prevents concurrent nightly runs from overwriting each other.
+PINOT_BUILD_IMAGE_TAG="${BASE_IMAGE_TAG}-amd64"
+BUILD_IMAGE_REMOTE_TAG="${DOCKER_IMAGE_NAME}:build-${BASE_IMAGE_TAG}-${PINOT_BRANCH}"
+echo "Pulling build image: ${BUILD_IMAGE_REMOTE_TAG}"
+docker pull --platform linux/amd64 "${BUILD_IMAGE_REMOTE_TAG}"
+docker tag "${BUILD_IMAGE_REMOTE_TAG}" "pinot-build:${PINOT_BUILD_IMAGE_TAG}"
+
+declare -a tags=($(echo "${TAGS}" | tr "," " "))
+declare -a runtimeImages=($(echo "${RUNTIME_IMAGE_TAGS}" | tr "," " "))
+
+for runtimeImage in "${runtimeImages[@]}"; do
+ DOCKER_BUILD_TAGS=""
+ for tag in "${tags[@]}"; do
+ DOCKER_BUILD_TAGS+=" --tag ${DOCKER_IMAGE_NAME}:${tag}-${runtimeImage}-linux-${BUILD_ARCH} "
+
+ # 17-amazoncorretto is the canonical default runtime; promote it as the bare
+ # latest-linux-<arch> tag so multi-arch manifests can reference it.
+ if [ "${runtimeImage}" == "17-amazoncorretto" ]; then
+ if [ "${tag}" == "latest" ]; then
+ DOCKER_BUILD_TAGS+=" --tag ${DOCKER_IMAGE_NAME}:latest-linux-${BUILD_ARCH} "
Review Comment:
I'm not sure we should consider changing the Docker image JVM version to be a backward incompatible change?
cc - @Jackie-Jiang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Overhaul Docker base images and release workflows [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 merged PR #18178:
URL: https://github.com/apache/pinot/pull/18178
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Upgrade Docker base images and build tooling [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #18178:
URL: https://github.com/apache/pinot/pull/18178#discussion_r3082727657
##########
.github/workflows/scripts/docker/.pinot_package_single_platform.sh:
##########
@@ -0,0 +1,84 @@
+#!/bin/bash -x
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# Packages Pinot for a single platform. Pulls the build image from DockerHub,
+# tags it locally, then builds and pushes the platform-specific package image.
+
+set -e
+
+if [ -z "${DOCKER_IMAGE_NAME}" ]; then
+ DOCKER_IMAGE_NAME="apachepinot/pinot"
+fi
+if [ -z "${BUILD_ARCH}" ]; then
+ echo "BUILD_ARCH is required (amd64 or arm64)" >&2
+ exit 1
+fi
+if [ -z "${DOCKER_FILE_BASE_DIR}" ]; then
+ echo "DOCKER_FILE_BASE_DIR is required" >&2
+ exit 1
+fi
+
+cd "${DOCKER_FILE_BASE_DIR}"
+
+# Pull the commit-scoped build image from DockerHub and tag it locally.
+# PINOT_BRANCH is the short commit SHA resolved by generate-build-info; including
+# it in the tag prevents concurrent nightly runs from overwriting each other.
+PINOT_BUILD_IMAGE_TAG="${BASE_IMAGE_TAG}-amd64"
+BUILD_IMAGE_REMOTE_TAG="${DOCKER_IMAGE_NAME}:build-${BASE_IMAGE_TAG}-${PINOT_BRANCH}"
+echo "Pulling build image: ${BUILD_IMAGE_REMOTE_TAG}"
+docker pull --platform linux/amd64 "${BUILD_IMAGE_REMOTE_TAG}"
+docker tag "${BUILD_IMAGE_REMOTE_TAG}" "pinot-build:${PINOT_BUILD_IMAGE_TAG}"
+
+declare -a tags=($(echo "${TAGS}" | tr "," " "))
+declare -a runtimeImages=($(echo "${RUNTIME_IMAGE_TAGS}" | tr "," " "))
+
+for runtimeImage in "${runtimeImages[@]}"; do
+ DOCKER_BUILD_TAGS=""
+ for tag in "${tags[@]}"; do
+ DOCKER_BUILD_TAGS+=" --tag ${DOCKER_IMAGE_NAME}:${tag}-${runtimeImage}-linux-${BUILD_ARCH} "
+
+ # 17-amazoncorretto is the canonical default runtime; promote it as the bare
+ # latest-linux-<arch> tag so multi-arch manifests can reference it.
+ if [ "${runtimeImage}" == "17-amazoncorretto" ]; then
+ if [ "${tag}" == "latest" ]; then
+ DOCKER_BUILD_TAGS+=" --tag ${DOCKER_IMAGE_NAME}:latest-linux-${BUILD_ARCH} "
Review Comment:
yes, moved to 21-ms-openjdk as default.
Usually for docker users, they don't care java runtime.
For advanced users, there can pick image based on tags.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Upgrade Docker base images and build tooling [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #18178:
URL: https://github.com/apache/pinot/pull/18178#discussion_r3079295276
##########
.github/workflows/scripts/docker/.pinot_package_single_platform.sh:
##########
@@ -0,0 +1,84 @@
+#!/bin/bash -x
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# Packages Pinot for a single platform. Pulls the build image from DockerHub,
+# tags it locally, then builds and pushes the platform-specific package image.
+
+set -e
+
+if [ -z "${DOCKER_IMAGE_NAME}" ]; then
+ DOCKER_IMAGE_NAME="apachepinot/pinot"
+fi
+if [ -z "${BUILD_ARCH}" ]; then
+ echo "BUILD_ARCH is required (amd64 or arm64)" >&2
+ exit 1
+fi
+if [ -z "${DOCKER_FILE_BASE_DIR}" ]; then
+ echo "DOCKER_FILE_BASE_DIR is required" >&2
+ exit 1
+fi
+
+cd "${DOCKER_FILE_BASE_DIR}"
+
+# Pull the commit-scoped build image from DockerHub and tag it locally.
+# PINOT_BRANCH is the short commit SHA resolved by generate-build-info; including
+# it in the tag prevents concurrent nightly runs from overwriting each other.
+PINOT_BUILD_IMAGE_TAG="${BASE_IMAGE_TAG}-amd64"
+BUILD_IMAGE_REMOTE_TAG="${DOCKER_IMAGE_NAME}:build-${BASE_IMAGE_TAG}-${PINOT_BRANCH}"
+echo "Pulling build image: ${BUILD_IMAGE_REMOTE_TAG}"
+docker pull --platform linux/amd64 "${BUILD_IMAGE_REMOTE_TAG}"
+docker tag "${BUILD_IMAGE_REMOTE_TAG}" "pinot-build:${PINOT_BUILD_IMAGE_TAG}"
+
+declare -a tags=($(echo "${TAGS}" | tr "," " "))
+declare -a runtimeImages=($(echo "${RUNTIME_IMAGE_TAGS}" | tr "," " "))
+
+for runtimeImage in "${runtimeImages[@]}"; do
+ DOCKER_BUILD_TAGS=""
+ for tag in "${tags[@]}"; do
+ DOCKER_BUILD_TAGS+=" --tag ${DOCKER_IMAGE_NAME}:${tag}-${runtimeImage}-linux-${BUILD_ARCH} "
+
+ # 17-amazoncorretto is the canonical default runtime; promote it as the bare
+ # latest-linux-<arch> tag so multi-arch manifests can reference it.
+ if [ "${runtimeImage}" == "17-amazoncorretto" ]; then
+ if [ "${tag}" == "latest" ]; then
+ DOCKER_BUILD_TAGS+=" --tag ${DOCKER_IMAGE_NAME}:latest-linux-${BUILD_ARCH} "
Review Comment:
This repoints the public `latest` / `latest-linux-<arch>` tags from JDK 11 to 17. Any deployment pulling `apachepinot/pinot:latest` will silently change JVM major version on the next publish, which is a backward-incompatible default change. Please keep the existing `latest` alias on the 11 line and add a separate explicit 17 tag instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Upgrade Docker base images and build tooling [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #18178:
URL: https://github.com/apache/pinot/pull/18178#discussion_r3082716598
##########
.github/workflows/scripts/docker/.pinot_base_docker_image_build_and_push.sh:
##########
@@ -30,11 +30,12 @@ fi
cd docker/images/pinot-base/pinot-base-${BASE_IMAGE_TYPE}
-docker buildx build \
Review Comment:
github action now support both amd64 and arm64 machines, we don't need cross platform build
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Overhaul Docker base images and release workflows [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on PR #18178:
URL: https://github.com/apache/pinot/pull/18178#issuecomment-4253624097
Documentation PR has been created to document these Docker base image changes: https://github.com/pinot-contrib/pinot-docs/pull/742
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Upgrade Docker base images and build tooling [pinot]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #18178:
URL: https://github.com/apache/pinot/pull/18178#discussion_r3072300225
##########
.github/workflows/scripts/docker/.pinot_base_docker_image_build_and_push.sh:
##########
@@ -30,11 +30,12 @@ fi
cd docker/images/pinot-base/pinot-base-${BASE_IMAGE_TYPE}
-docker buildx build \
+docker build \
--no-cache \
--platform=${BUILD_PLATFORM} \
--file ${OPEN_JDK_DIST}.dockerfile \
- --tag apachepinot/pinot-base-${BASE_IMAGE_TYPE}:${TAG} \
+ --tag apachepinot/pinot-base-${BASE_IMAGE_TYPE}:${TAG}-${ARCH} \
--build-arg JAVA_VERSION=${JDK_VERSION:-11} \
+ --build-arg ARCH=${ARCH} \
--push \
.
Review Comment:
The script uses `docker build` together with `--platform` and `--push`. `--push` is only supported via Buildx (`docker buildx build --push`), so this will fail at runtime and break the workflow. Switch back to `docker buildx build` (or ensure Buildx is installed as the default builder) for pushing images built for a specific platform.
##########
.github/workflows/scripts/docker/.superset_multi_arch_docker_image_build_and_push.sh:
##########
@@ -40,20 +40,40 @@ else
declare -a tags=($(echo ${TAGS} | tr "," " "))
fi
-DOCKER_BUILD_TAGS=""
+DOCKER_BUILD_AMD64_TAGS=""
+DOCKER_BUILD_ARM64_TAGS=""
+DOCKER_AMEND_TAGS_CMD=""
for tag in "${tags[@]}"
do
echo "Plan to build and push docker images for: ${DOCKER_IMAGE_NAME}:${tag}"
- DOCKER_BUILD_TAGS+=" --tag ${DOCKER_IMAGE_NAME}:${tag} "
+ DOCKER_BUILD_AMD64_TAGS+=" --tag ${DOCKER_IMAGE_NAME}:${tag}-amd64 "
+ DOCKER_BUILD_ARM64_TAGS+=" --tag ${DOCKER_IMAGE_NAME}:${tag}-arm64 "
+ DOCKER_AMEND_TAGS_CMD=" --amend ${DOCKER_IMAGE_NAME}:${tag}-amd64 --amend ${DOCKER_IMAGE_NAME}:${tag}-arm64 "
done
Review Comment:
`DOCKER_AMEND_TAGS_CMD` gets overwritten on each iteration of the tag loop, so after the loop it only contains `--amend` entries for the *last* tag. The subsequent `docker manifest create ${DOCKER_IMAGE_NAME}:${tag} ${DOCKER_AMEND_TAGS_CMD}` will therefore build incorrect manifests for all earlier tags. Build the amend list per-tag inside the manifest loop (or append to `DOCKER_AMEND_TAGS_CMD` and index it per tag).
##########
.github/workflows/build-pinot-base-docker-image.yml:
##########
@@ -45,10 +46,33 @@ jobs:
- uses: docker/setup-buildx-action@v4
name: Set up Docker Buildx
- uses: actions/checkout@v5
- - name: Build and push the Docker image
+ - name: Build ${{ matrix.type }} ${{ matrix.jdk_version }}-${{ matrix.openJdkDist }}-${{ matrix.arch }} Pinot Base Docker Image
env:
OPEN_JDK_DIST: ${{ matrix.openJdkDist }}
- BASE_IMAGE_TYPE: ${{ matrix.baseImageType }}
+ BASE_IMAGE_TYPE: ${{ matrix.type }}
BUILD_PLATFORM: "linux/${{ matrix.arch }}"
- TAG: "11-${{ matrix.openJdkDist }}"
+ TAG: "${{ matrix.jdk_version }}-${{ matrix.openJdkDist }}"
+ JDK_VERSION: ${{ matrix.jdk_version }}
+ ARCH: ${{ matrix.arch }}
run: .github/workflows/scripts/docker/.pinot_base_docker_image_build_and_push.sh
+ push-pinot-build-docker-manifest:
+ name: Push Pinot Base Docker Manifest
+ runs-on: ubuntu-24.04
+ needs: build-pinot-build-docker-image
+ strategy:
+ matrix:
+ openJdkDist: ["amazoncorretto", "ms-openjdk"]
+ jdk_version: [11, 17, 21, 25]
+ type: [build, runtime]
+ steps:
+ - name: Login to DockerHub
+ uses: docker/login-action@v4
+ with:
+ username: ${{ secrets.DOCKERHUB_USERNAME }}
+ password: ${{ secrets.DOCKERHUB_TOKEN }}
+ - uses: actions/checkout@v5
+ - name: Push ${{ matrix.type }} ${{ matrix.jdk_version }}-${{ matrix.openJdkDist }} Pinot Base Docker manifest
+ env:
+ BASE_IMAGE_TYPE: ${{ matrix.type }}
+ TAG: "${{ matrix.jdk_version }}-${{ matrix.openJdkDist }}"
+ run: .github/workflows/scripts/docker/.pinot_base_docker_manifest_build_and_push.sh
Review Comment:
This workflow calls `.github/workflows/scripts/docker/.pinot_base_docker_manifest_build_and_push.sh`, but there is no such script in the repository (at least under `.github/workflows/scripts/docker/`). As written, the manifest-push job will fail. Add the missing script (or update the `run:` line to the correct existing manifest script).
```suggestion
run: .github/workflows/scripts/docker/pinot_base_docker_manifest_build_and_push.sh
```
##########
.github/workflows/scripts/docker/.superset_multi_arch_docker_image_build_and_push.sh:
##########
@@ -40,20 +40,40 @@ else
declare -a tags=($(echo ${TAGS} | tr "," " "))
fi
-DOCKER_BUILD_TAGS=""
+DOCKER_BUILD_AMD64_TAGS=""
+DOCKER_BUILD_ARM64_TAGS=""
+DOCKER_AMEND_TAGS_CMD=""
for tag in "${tags[@]}"
do
echo "Plan to build and push docker images for: ${DOCKER_IMAGE_NAME}:${tag}"
- DOCKER_BUILD_TAGS+=" --tag ${DOCKER_IMAGE_NAME}:${tag} "
+ DOCKER_BUILD_AMD64_TAGS+=" --tag ${DOCKER_IMAGE_NAME}:${tag}-amd64 "
+ DOCKER_BUILD_ARM64_TAGS+=" --tag ${DOCKER_IMAGE_NAME}:${tag}-arm64 "
+ DOCKER_AMEND_TAGS_CMD=" --amend ${DOCKER_IMAGE_NAME}:${tag}-amd64 --amend ${DOCKER_IMAGE_NAME}:${tag}-arm64 "
done
cd ${DOCKER_FILE_BASE_DIR}
docker build \
--no-cache \
- --platform=${BUILD_PLATFORM} \
+ --platform=linux/amd64 \
--file Dockerfile \
--build-arg SUPERSET_IMAGE_TAG=${SUPERSET_IMAGE_TAG} \
- ${DOCKER_BUILD_TAGS} \
+ ${DOCKER_BUILD_AMD64_TAGS} \
--push \
.
+
+docker build \
+ --no-cache \
+ --platform=linux/arm64 \
+ --file Dockerfile \
+ --build-arg SUPERSET_IMAGE_TAG=${SUPERSET_IMAGE_TAG}-${SUPERSET_ARM64_SUFFIX} \
+ ${DOCKER_BUILD_ARM64_TAGS} \
+ --push \
+ .
Review Comment:
These builds use `docker build` with `--platform` and `--push`. `--push` is a Buildx feature (`docker buildx build --push`), so this is likely to fail in CI and prevent publishing the images. Use `docker buildx build` (and ideally `--platform linux/amd64,linux/arm64` to build/push both in one invocation) to ensure consistent multi-arch behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Upgrade Docker base images and build tooling [pinot]
Posted by "yashmayya (via GitHub)" <gi...@apache.org>.
yashmayya commented on code in PR #18178:
URL: https://github.com/apache/pinot/pull/18178#discussion_r3082275896
##########
.github/workflows/scripts/docker/.pinot_package_single_platform.sh:
##########
@@ -0,0 +1,84 @@
+#!/bin/bash -x
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# Packages Pinot for a single platform. Pulls the build image from DockerHub,
+# tags it locally, then builds and pushes the platform-specific package image.
+
+set -e
+
+if [ -z "${DOCKER_IMAGE_NAME}" ]; then
+ DOCKER_IMAGE_NAME="apachepinot/pinot"
+fi
+if [ -z "${BUILD_ARCH}" ]; then
+ echo "BUILD_ARCH is required (amd64 or arm64)" >&2
+ exit 1
+fi
+if [ -z "${DOCKER_FILE_BASE_DIR}" ]; then
+ echo "DOCKER_FILE_BASE_DIR is required" >&2
+ exit 1
+fi
+
+cd "${DOCKER_FILE_BASE_DIR}"
+
+# Pull the commit-scoped build image from DockerHub and tag it locally.
+# PINOT_BRANCH is the short commit SHA resolved by generate-build-info; including
+# it in the tag prevents concurrent nightly runs from overwriting each other.
+PINOT_BUILD_IMAGE_TAG="${BASE_IMAGE_TAG}-amd64"
+BUILD_IMAGE_REMOTE_TAG="${DOCKER_IMAGE_NAME}:build-${BASE_IMAGE_TAG}-${PINOT_BRANCH}"
+echo "Pulling build image: ${BUILD_IMAGE_REMOTE_TAG}"
+docker pull --platform linux/amd64 "${BUILD_IMAGE_REMOTE_TAG}"
+docker tag "${BUILD_IMAGE_REMOTE_TAG}" "pinot-build:${PINOT_BUILD_IMAGE_TAG}"
+
+declare -a tags=($(echo "${TAGS}" | tr "," " "))
+declare -a runtimeImages=($(echo "${RUNTIME_IMAGE_TAGS}" | tr "," " "))
+
+for runtimeImage in "${runtimeImages[@]}"; do
+ DOCKER_BUILD_TAGS=""
+ for tag in "${tags[@]}"; do
+ DOCKER_BUILD_TAGS+=" --tag ${DOCKER_IMAGE_NAME}:${tag}-${runtimeImage}-linux-${BUILD_ARCH} "
+
+ # 17-amazoncorretto is the canonical default runtime; promote it as the bare
+ # latest-linux-<arch> tag so multi-arch manifests can reference it.
+ if [ "${runtimeImage}" == "17-amazoncorretto" ]; then
+ if [ "${tag}" == "latest" ]; then
+ DOCKER_BUILD_TAGS+=" --tag ${DOCKER_IMAGE_NAME}:latest-linux-${BUILD_ARCH} "
Review Comment:
Also is the default supposed to be `21-ms-openjdk` or `17-amazoncorretto`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org