You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2026/09/28 19:45:08 UTC
(spark-kubernetes-operator) branch main updated: [SPARK-59836] Remove unused `libstdc++` package from Docker image
This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new 28e2f375 [SPARK-59836] Remove unused `libstdc++` package from Docker image
28e2f375 is described below
commit 28e2f375532560fa99e660882cde3b0f9294900a
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Mon Sep 28 12:45:01 2026 -0700
[SPARK-59836] Remove unused `libstdc++` package from Docker image
### What changes were proposed in this pull request?
This PR aims to remove the unused `libstdc++` package, together with its only dependency `libgcc`, from the final stage of the operator Docker image.
```diff
-RUN apk add --no-cache libstdc++ && \
- addgroup -S -g $SPARK_UID spark && \
+RUN addgroup -S -g $SPARK_UID spark && \
adduser -S -h $SPARK_OPERATOR_HOME -u $SPARK_UID -G spark spark && \
chown -R spark:spark $SPARK_OPERATOR_HOME
```
### Why are the changes needed?
SPARK-56131 (https://github.com/apache/spark-kubernetes-operator/pull/571) added `libstdc++` when the image switched to a `jlink` custom JRE on `alpine`. It has been shipped since [0.9.0](https://github.com/apache/spark-kubernetes-operator/releases/tag/0.9.0) (2026-05-14). However, nothing in the image uses it.
- **Custom JRE**: The Zulu musl JRE depends only on `musl` and `libz`. The GCC runtime is statically linked (see `legal/java.base/gcc.md`). `ldd` shows that neither `bin/java` nor any of the 29 libraries under `/opt/custom-jre` links `libstdc++` or `libgcc_s`.
- **JNI libraries in the operator jar**: The bundled Linux native libraries are C libraries. They depend only on glibc's `libc`, `libdl` and `librt` (plus `ld-linux-*`). None of them has a `NEEDED` entry or an undefined C++ runtime symbol for `libstdc++` or `libgcc_s`. Native libraries built with C++, such as RocksDB and LevelDB, are already excluded, and no class refers to `libstdc++` by name.
| Library | Linux architectures | `NEEDED` |
| - | - | - |
| Netty `epoll` 4.2.18.Final | `x86_64`, `aarch_64`, `riscv64` | `libc.so.6`, `libdl.so.2`, `librt.so.1` |
| `lz4-java` 1.11.0 | `aarch64`, `amd64`, `i386`, `ppc64le`, `riscv64`, `s390x` | `libc.so.6` (+ `ld-linux-*` on some architectures) |
| `commons-crypto` 1.1.0 | `aarch64`, `arm`, `armhf`, `x86`, `x86_64` | `libc.so.6` (+ `ld-linux-*` on some architectures) |
- **Runtime**: The current image has `libstdc++` installed. Even there, a running operator never maps `libstdc++.so.6` or `libgcc_s.so.1`, even after all the bundled JNI loaders are triggered.
As a result, the image has fewer OS packages. The `RUN` layer shrinks from 3.07MB to 45.1kB, and the image content shrinks from 114,457,385 bytes to 113,432,908 bytes.
```
$ docker images apache/spark-kubernetes-operator
IMAGE ID DISK USAGE CONTENT SIZE
apache/spark-kubernetes-operator:test-with-libstdcxx f6950e9aa608 268MB 114MB
apache/spark-kubernetes-operator:test-without-libstdcxx 331ac295cc52 264MB 113MB
```
### Does this PR introduce _any_ user-facing change?
No. The operator behavior is unchanged. Only the unused `libstdc++` and `libgcc` packages are removed from the image.
### How was this patch tested?
Pass the CIs. The `Build Operator Image CI` job builds the image with this `Dockerfile`, and the `K8s Integration Tests` run on it.
In addition, I manually built the images with and without this patch on `linux/arm64` (`alpine` 3.24.2) and compared them.
```
$ docker build --build-arg APP_VERSION=1.1.0-SNAPSHOT -t apache/spark-kubernetes-operator:test-without-libstdcxx -f build-tools/docker/Dockerfile .
$ docker run --rm apache/spark-kubernetes-operator:test-without-libstdcxx java -version
openjdk version "26.0.2.1" 2026-08-18
OpenJDK Runtime Environment Zulu26.32+203-CA (build 26.0.2.1+1)
OpenJDK 64-Bit Server VM Zulu26.32+203-CA (build 26.0.2.1+1, mixed mode)
```
The operator starts in the same way with and without this patch. It logs its version, registers the reconcilers, and retries on `UnknownHostException` when there is no K8s cluster.
```
$ docker run -d -e JAVA_TOOL_OPTIONS=-Dlog4j2.level=INFO apache/spark-kubernetes-operator:test-without-libstdcxx
$ docker logs <container>
main INFO Version: 1.1.0-SNAPSHOT
main INFO Java Version: 26.0.2.1+1
main INFO Built-in Spark Version: 4.2.0
...
main INFO Registered reconciler: 'sparkappreconciler' for resource: 'class org.apache.spark.k8s.operator.SparkApplication' for namespace(s): [default]
...
main INFO Registered reconciler: 'sparkclusterreconciler' for resource: 'class org.apache.spark.k8s.operator.SparkCluster' for namespace(s): [default]
...
Caused by: java.net.UnknownHostException: kubernetes.default.svc
...
vert.x-eventloop-thread-3 WARN listSyncAndWatch failed for spark.apache.org/v1/namespaces/default/sparkapplications, will retry
```
The shared libraries loaded by the running operator are identical in both images. `libstdc++` is not used even when it is installed.
```
$ docker exec <container> cat /proc/1/maps | awk '{print $6}' | grep '\.so' | sort -u
/lib/ld-musl-aarch64.so.1
/opt/custom-jre/lib/libjava.so
/opt/custom-jre/lib/libjimage.so
/opt/custom-jre/lib/libjli.so
/opt/custom-jre/lib/libmanagement.so
/opt/custom-jre/lib/libmanagement_ext.so
/opt/custom-jre/lib/libnet.so
/opt/custom-jre/lib/libnio.so
/opt/custom-jre/lib/libzip.so
/opt/custom-jre/lib/server/libjvm.so
/usr/lib/libz.so.1.3.2
```
In addition, triggering every bundled JNI loader gave identical results in both images:
- Netty `Epoll.isAvailable()` returns `true`.
- `lz4-java` and `commons-crypto` fail to load on `ld-linux-aarch64.so.1` and fall back to Java.
- Hadoop `NativeCodeLoader.isNativeCodeLoaded()` returns `false`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #902 from dongjoon-hyun/SPARK-59836.
Authored-by: Dongjoon Hyun <do...@apache.org>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
build-tools/docker/Dockerfile | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/build-tools/docker/Dockerfile b/build-tools/docker/Dockerfile
index d195bc9c..04c4c177 100644
--- a/build-tools/docker/Dockerfile
+++ b/build-tools/docker/Dockerfile
@@ -61,8 +61,7 @@ WORKDIR $SPARK_OPERATOR_WORK_DIR
COPY --from=jlink /opt/custom-jre $JAVA_HOME
-RUN apk add --no-cache libstdc++ && \
- addgroup -S -g $SPARK_UID spark && \
+RUN addgroup -S -g $SPARK_UID spark && \
adduser -S -h $SPARK_OPERATOR_HOME -u $SPARK_UID -G spark spark && \
chown -R spark:spark $SPARK_OPERATOR_HOME
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org