You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2023/02/16 05:52:30 UTC
[spark] branch master updated: [SPARK-42462][K8S] Prevent `docker-image-tool.sh` from publishing OCI manifests
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 2ac70ae5381 [SPARK-42462][K8S] Prevent `docker-image-tool.sh` from publishing OCI manifests
2ac70ae5381 is described below
commit 2ac70ae5381333aa899d82f6cd4c3bbae524e1c2
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Wed Feb 15 21:52:17 2023 -0800
[SPARK-42462][K8S] Prevent `docker-image-tool.sh` from publishing OCI manifests
### What changes were proposed in this pull request?
This is found during Apache Spark 3.3.2 docker image publishing. It's not an Apache Spark but important for `docker-image-tool.sh` to provide backward compatibility during cross-building. This PR targets for all **future releases**, Apache Spark 3.4.0/3.3.3/3.2.4.
### Why are the changes needed?
Docker `buildx` v0.10.0 publishes OCI Manifests by default which is not supported by `docker manifest` command like the following.
https://github.com/docker/buildx/issues/1509
```
$ docker manifest inspect apache/spark:v3.3.2
no such manifest: docker.io/apache/spark:v3.3.2
```
Note that the published images are working on both AMD64/ARM64 machines, but `docker manifest` cannot be used. For example, we cannot create `latest` tag.
### Does this PR introduce _any_ user-facing change?
This will fix the regression of Docker `buildx`.
### How was this patch tested?
Manually builds the multi-arch image and check `manifest`.
```
$ docker manifest inspect apache/spark:v3.3.2
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
"manifests": [
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"size": 3444,
"digest": "sha256:30ae5023fc384ae3b68d2fb83adde44b1ece05f926cfceecac44204cdc9e79cb",
"platform": {
"architecture": "amd64",
"os": "linux"
}
},
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"size": 3444,
"digest": "sha256:aac13b5b5a681aefa91036d2acae91d30a743c2e78087c6df79af4de46a16e1b",
"platform": {
"architecture": "arm64",
"os": "linux"
}
}
]
}
```
Closes #40051 from dongjoon-hyun/SPARK-42462.
Authored-by: Dongjoon Hyun <do...@apache.org>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
bin/docker-image-tool.sh | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/bin/docker-image-tool.sh b/bin/docker-image-tool.sh
index 65b0f8d8898..a137a2fba52 100755
--- a/bin/docker-image-tool.sh
+++ b/bin/docker-image-tool.sh
@@ -181,7 +181,7 @@ function build {
error "Failed to build Spark JVM Docker image, please refer to Docker build output for details."
fi
if [ "${CROSS_BUILD}" != "false" ]; then
- (cd $(img_ctx_dir base) && docker buildx build $ARCHS $NOCACHEARG "${BUILD_ARGS[@]}" --push \
+ (cd $(img_ctx_dir base) && docker buildx build $ARCHS $NOCACHEARG "${BUILD_ARGS[@]}" --push --provenance=false \
-t $(image_ref spark) \
-f "$BASEDOCKERFILE" .)
fi
@@ -194,7 +194,7 @@ function build {
error "Failed to build PySpark Docker image, please refer to Docker build output for details."
fi
if [ "${CROSS_BUILD}" != "false" ]; then
- (cd $(img_ctx_dir pyspark) && docker buildx build $ARCHS $NOCACHEARG "${BINDING_BUILD_ARGS[@]}" --push \
+ (cd $(img_ctx_dir pyspark) && docker buildx build $ARCHS $NOCACHEARG "${BINDING_BUILD_ARGS[@]}" --push --provenance=false \
-t $(image_ref spark-py) \
-f "$PYDOCKERFILE" .)
fi
@@ -208,7 +208,7 @@ function build {
error "Failed to build SparkR Docker image, please refer to Docker build output for details."
fi
if [ "${CROSS_BUILD}" != "false" ]; then
- (cd $(img_ctx_dir sparkr) && docker buildx build $ARCHS $NOCACHEARG "${BINDING_BUILD_ARGS[@]}" --push \
+ (cd $(img_ctx_dir sparkr) && docker buildx build $ARCHS $NOCACHEARG "${BINDING_BUILD_ARGS[@]}" --push --provenance=false \
-t $(image_ref spark-r) \
-f "$RDOCKERFILE" .)
fi
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org