You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@sedona.apache.org by "jiayuasu (via GitHub)" <gi...@apache.org> on 2026/05/06 17:36:34 UTC

[PR] [CI] Trim docker image build context: exclude Maven target/, drop pip cache [sedona]

jiayuasu opened a new pull request, #2907:
URL: https://github.com/apache/sedona/pull/2907

   ## Did you read the Contributor Guide?
   
   - Yes, I have read the [Contributor Rules](https://sedona.apache.org/latest/community/rule/) and [Contributor Development Guide](https://sedona.apache.org/latest/community/develop/)
   
   ## Is this PR related to a ticket?
   
   - No: this is a CI / release-tooling update. The PR name follows the format `[CI] my subject`.
   
   ## What changes were proposed in this PR?
   
   Two related fixes that together restore the `apache/sedona` docker image download size to its pre-1.9.0 footprint.
   
   ### Background — what regressed
   
   Per the Docker Hub tags API, `apache/sedona:1.9.0` ships at **4.03 GB** compressed (per-arch, both amd64 and arm64) while `apache/sedona:1.8.1` is **2.97 GB**. A layer-by-layer comparison of the published manifests (37 layers per arch) shows all but one layer are identical to within a few MB:
   
   | Layer | 1.8.1 | 1.9.0 | Δ |
   |---|---|---|---|
   | L08 (`install-spark.sh`) | 1241 MB | 1239 MB | ~0 |
   | L10 (`pip install -r requirements.txt`) | 715 MB | 728 MB | +13 |
   | **L12 (`COPY ./spark-shaded/`)** | **0.7 MB** | **1104 MB** | **+1103** |
   | L18 (`install-zeppelin.sh`) | 504 MB | 504 MB | 0 |
   
   Layer 12 is `COPY ./spark-shaded/ ${SEDONA_HOME}/spark-shaded/` (line 57 of `docker/sedona-docker.dockerfile`). The 1.9.0 release was published from a tree that had Maven build outputs in `spark-shaded/target/` (likely from a prior `mvn package`); the existing `.dockerignore` allow-list `!spark-shaded/**` re-included everything under that directory, so the COPY swept in ~1.1 GB of JARs and test classes.
   
   The dockerfile's trailing `RUN rm -rf ${SEDONA_HOME}` *does* delete the content from the running container's filesystem, so `du` inside the container looks normal — but it cannot shrink the prior layer that already committed those bytes. The wasted ~1 GB stays in the published manifest and inflates every pull's download size.
   
   ### Fix 1 — `docker/sedona-docker.dockerfile.dockerignore`
   
   Re-exclude Maven and Python build outputs after the allow-list (last-match-wins). Even on a release machine that has stale `target/` directories, the Docker build context will not include them.
   
   ```diff
    *
    !docker/**
    !zeppelin/**
    !docs/usecases/**
    !python/**
    !spark-shaded/**
   +
   +# Re-exclude Maven build outputs and Python build artifacts so a tree
   +# that has had `mvn package` or `python -m build` run against it does
   +# not balloon the COPY layers ...
   +**/target/
   +python/build/
   +python/dist/
   +**/*.egg-info/
   +**/__pycache__/
   ```
   
   ### Fix 2 — `docker/sedona-docker.dockerfile`
   
   Pass `--no-cache-dir` to both `pip3 install` invocations. Without it, pip leaves ~439 MB of wheel downloads under `/root/.cache/pip` in the requirements.txt install layer — measured `du -sh /root/.cache` inside the running 1.9.0 image. `--no-cache-dir` skips that write and brings the pip layer down to roughly its installed-package size, with no runtime impact.
   
   ## How was this patch tested?
   
   1. **Probe build proves the deny rule actually fires.** Synthesized a 200 MB fake JAR at `spark-shaded/target/fake-jar.jar`, then built a one-liner `FROM alpine; COPY ./spark-shaded/ /shaded/` Dockerfile twice — once with the new `.dockerignore`, once with the old:
   
      | Variant | `/shaded` size in resulting image |
      |---|---|
      | With `**/target/` deny rule | **24 KB** (pom.xml + .gitignore only) |
      | Without the deny rule | **200 MB** (fake jar leaked in) |
   
   2. **Full image rebuild confirms `--no-cache-dir` shrinks the pip layer locally.** Local tree has no `target/` (clean checkout), so the deny rule is a no-op for our local build; the win comes purely from `--no-cache-dir`:
   
      | Image | Total size | `pip install -r` layer |
      |---|---|---|
      | `sedona:dev` (master @ HEAD) | 5.06 GB | 1.48 GB |
      | `sedona:trim` (this PR) | **4.81 GB** | **1.24 GB** |
   
   3. **Existing Docker-build CI matrix exercises the change.** The path filter (`docker/**`) widened in #2889 means this PR triggers `docker-build.yml`, which runs `./docker/build.sh ... local ...` and `docker/test-notebooks.sh` against the resulting image — so the existing 6-notebook test suite verifies the new dockerfile end-to-end.
   
   ## Expected impact on the next 1.9.0 re-publish
   
   Combining both fixes on a release-machine tree should drop the published image from 4.03 GB → ~3.0 GB compressed (roughly the 1.8.1 baseline). Recipe:
   
   ```bash
   git clean -fdX -- spark-shaded/ python/   # belt-and-suspenders, in case the new .dockerignore misses anything
   ./docker/build.sh 4.0.1 1.9.0 release 33.5
   ```
   
   `release` mode does `--platform linux/amd64,linux/arm64 --output type=registry`, so it pushes both arches to Docker Hub directly.
   
   ## Did this PR include necessary documentation updates?
   
   - No public API changes.
   - No documentation updates needed; the `.dockerignore` comment block explains the regression and the rationale for any future contributor running `git blame`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [CI] Trim docker image build context: exclude Maven target/, drop pip cache [sedona]

Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu merged PR #2907:
URL: https://github.com/apache/sedona/pull/2907


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [CI] Trim docker image build context: exclude Maven target/, drop pip cache [sedona]

Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on PR #2907:
URL: https://github.com/apache/sedona/pull/2907#issuecomment-4390702472

   Force-pushed `4ffad3e745` (was `1eda34f9ef`). Both Docker-build matrix legs went red on the previous commit because the `**/target/` deny rule excluded `spark-shaded/target/` from the build context — but the `latest` matrix entry needs that directory: `build.sh` runs `mvn clean install -DskipTests` first, then `install-sedona.sh` inside the image does `cp \${SEDONA_HOME}/spark-shaded/target/sedona-spark-shaded-*.jar \${SPARK_HOME}/jars/`. With `target/` excluded, the `cp` failed and so did install-sedona.sh.
   
   Pivoted the fix:
   
   | Old approach (red) | New approach |
   |---|---|
   | Add `**/target/` to `.dockerignore` | **Removed** — would break `latest` mode |
   | (n/a) | Add `else: rm -rf spark-shaded/target python/build python/dist` to `build.sh`'s `if SEDONA_VERSION = latest` block. Non-`latest` builds (the path the 1.9.0 release publisher uses) never touch `target/` since `install-sedona.sh` downloads from Maven Central instead. |
   | Add `--no-cache-dir` to pip | **Kept** — independent improvement |
   | Add `python/build/`, `python/dist/`, `**/*.egg-info/`, `**/__pycache__/` to `.dockerignore` | **Kept** — harmless defensive rules |
   
   Result:
   - The `latest` matrix entry still has `mvn`-built artifacts in `spark-shaded/target/` so install-sedona.sh's `cp` succeeds.
   - A non-`latest` release build (`./docker/build.sh 4.0.1 1.9.0 release 33.5`) cleans stale `target/` first, so the COPY layer stays small and the published image avoids the 1 GB regression.
   
   CI should now go green on both legs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [CI] Trim docker image build context: exclude Maven target/, drop pip cache [sedona]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #2907:
URL: https://github.com/apache/sedona/pull/2907#discussion_r3197032646


##########
docker/sedona-docker.dockerfile.dockerignore:
##########
@@ -7,3 +7,16 @@
 !docs/usecases/**
 !python/**
 !spark-shaded/**
+
+# Re-exclude Python build artifacts and pyc caches so a tree that has
+# had `python -m build` run against it does not balloon the COPY layers.
+# Note: `**/target/` is intentionally NOT excluded here because the
+# build.sh `latest` mode runs `mvn clean install -DskipTests` and then
+# install-sedona.sh inside the image copies the freshly-built shaded
+# JAR from `${SEDONA_HOME}/spark-shaded/target/`. Stale Maven `target/`
+# directories from a prior local build are cleaned up by build.sh
+# (non-latest branch) instead — see docker/build.sh.
+python/build/
+python/dist/
+**/*.egg-info/
+**/__pycache__/

Review Comment:
   This ignorefile intentionally keeps Maven `target/` in the build context (see note), which means a manual `docker build -f docker/sedona-docker.dockerfile .` for a *published* `sedona_version` (i.e., not using `docker/build.sh`) can still accidentally COPY a stale `spark-shaded/target/` into the image and bloat the resulting layers. Consider excluding `spark-shaded/target/` (or `**/target/`) and then explicitly re-including only `spark-shaded/target/sedona-spark-shaded-*.jar` for the `sedona_version=latest` path so both workflows are protected.
   



##########
docker/sedona-docker.dockerfile:
##########
@@ -43,14 +43,14 @@ ENV PYTHONPATH=${SPARK_HOME}/python
 # Set up OS libraries and PySpark
 RUN apt-get update
 RUN apt-get install -y openjdk-17-jdk-headless curl python3-pip maven
-RUN pip3 install pipenv --break-system-packages
+RUN pip3 install --no-cache-dir pipenv --break-system-packages
 COPY ./docker/install-spark.sh ${SEDONA_HOME}/docker/
 RUN chmod +x ${SEDONA_HOME}/docker/install-spark.sh
 RUN ${SEDONA_HOME}/docker/install-spark.sh ${spark_version} ${hadoop_s3_version} ${aws_sdk_version}
 
 # Install Python dependencies
 COPY docker/requirements.txt /opt/requirements.txt
-RUN pip3 install -r /opt/requirements.txt --break-system-packages
+RUN pip3 install --no-cache-dir -r /opt/requirements.txt --break-system-packages

Review Comment:
   `--no-cache-dir` is added for the pipenv and requirements installs, but later the Dockerfile runs `install-sedona.sh`, which performs additional `pip3 install ...` steps without `--no-cache-dir`. That can still repopulate `/root/.cache/pip` in a later layer, partially negating the size win. Consider setting `ENV PIP_NO_CACHE_DIR=1` (or updating the script’s pip invocations) so all pip installs during the build are cache-free.



##########
docker/build.sh:
##########
@@ -81,6 +81,17 @@ if [ "$SEDONA_VERSION" = "latest" ]; then
 
     # The compilation must take place outside Docker to avoid unnecessary maven packages
     mvn clean install -DskipTests -Dspark="${SEDONA_SPARK_VERSION}" -Dscala=2.13
+else
+    # When building against a published Sedona version, install-sedona.sh
+    # downloads the shaded JAR from Maven Central inside the container and
+    # never reads spark-shaded/target/. Any stale Maven artifacts in the
+    # local tree would still be pulled into the build context by the
+    # `COPY ./spark-shaded/` step, ship in the published manifest, and add
+    # to every pull's download size — even though the dockerfile's trailing
+    # `RUN rm -rf ${SEDONA_HOME}` deletes them from the runtime filesystem.
+    # apache/sedona:1.9.0 hit this regression and shipped 1.1 GB heavier than
+    # 1.8.1 (4.03 GB vs 2.97 GB compressed) for exactly this reason.
+    rm -rf spark-shaded/target python/build python/dist
 fi

Review Comment:
   This new cleanup deletes relative paths (`rm -rf spark-shaded/target ...`) without first ensuring the script is running from the repo root. If `docker/build.sh` is invoked from another working directory that happens to contain a `spark-shaded/target` path, it could delete unrelated files; if invoked from a different dir without that path, it won’t actually prevent the Docker context from including stale artifacts. Consider `cd`ing to the repo root at the top of the script (e.g., via the script’s directory or `git rev-parse --show-toplevel`) before running `rm` and `docker buildx build`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org