You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@systemds.apache.org by "alexanderschmi (via GitHub)" <gi...@apache.org> on 2025/06/17 13:24:33 UTC

[PR] [SYSTEMDS-3877] Docker image size reduction and safety improvement [systemds]

alexanderschmi opened a new pull request, #2274:
URL: https://github.com/apache/systemds/pull/2274

   ## Purpose
   Update Docker Images to run on Alpine Linux
   
   ## What Changed
   
   1. Switched from Ubuntu to Alpine
   2. Added multi stage build to reduce image size


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [SYSTEMDS-3877] Docker image size reduction and safety improvement [systemds]

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #2274:
URL: https://github.com/apache/systemds/pull/2274#issuecomment-2988261559

   ## [Codecov](https://app.codecov.io/gh/apache/systemds/pull/2274?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   All modified and coverable lines are covered by tests :white_check_mark:
   > Project coverage is 22.10%. Comparing base [(`a914c0d`)](https://app.codecov.io/gh/apache/systemds/commit/a914c0d0ca307afb357cb46bc0390f58b8544faf?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`5fde8ea`)](https://app.codecov.io/gh/apache/systemds/commit/5fde8ea1ce4123f56d99d3bf84faa1c5b3140e0b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@              Coverage Diff              @@
   ##               main    #2274       +/-   ##
   =============================================
   - Coverage     72.96%   22.10%   -50.86%     
   + Complexity    46090    14567    -31523     
   =============================================
     Files          1479     1479               
     Lines        172655   172655               
     Branches      33797    33797               
   =============================================
   - Hits         125972    38170    -87802     
   - Misses        37188   131119    +93931     
   + Partials       9495     3366     -6129     
   ```
   
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/systemds/pull/2274?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [SYSTEMDS-3877] Docker image size reduction and safety improvement [systemds]

Posted by "Baunsgaard (via GitHub)" <gi...@apache.org>.
Baunsgaard commented on PR #2274:
URL: https://github.com/apache/systemds/pull/2274#issuecomment-3377244660

   Hi @alexanderschmi ,
   
   Thanks for the PR, it is now merged, I only rolled back the default script of the docker image.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [SYSTEMDS-3877] Docker image size reduction and safety improvement [systemds]

Posted by "Baunsgaard (via GitHub)" <gi...@apache.org>.
Baunsgaard closed pull request #2274: [SYSTEMDS-3877] Docker image size reduction and safety improvement
URL: https://github.com/apache/systemds/pull/2274


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [SYSTEMDS-3877] Docker image size reduction and safety improvement [systemds]

Posted by "Baunsgaard (via GitHub)" <gi...@apache.org>.
Baunsgaard commented on PR #2274:
URL: https://github.com/apache/systemds/pull/2274#issuecomment-3018145898

   > ## Size Reduction
   > ### Before
   > 
   > ![Before](https://private-user-images.githubusercontent.com/51443671/458954908-eeff534c-4610-405f-a95a-a436b450c6b0.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NTEyNjk4MzEsIm5iZiI6MTc1MTI2OTUzMSwicGF0aCI6Ii81MTQ0MzY3MS80NTg5NTQ5MDgtZWVmZjUzNGMtNDYxMC00MDVmLWE5NWEtYTQzNmI0NTBjNmIwLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTA2MzAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwNjMwVDA3NDUzMVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTY5OGM2NDExN2Y4ZjUxZjhlNDFlYzlkZmE1M2JiYTBmYmQ2NWNjYTM4NTk1ODkyNzk5NzdhYjgyNWI4OWZlYmQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0._JpJFoxzNcdKVlTqXsX2OVcww3faZIjoVRhyyHijgrg)
   > ### After
   > 
   > ![After](https://private-user-images.githubusercontent.com/51443671/458954904-927920aa-a724-4a40-b58a-b1b5c38d2200.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NTEyNjk4MzEsIm5iZiI6MTc1MTI2OTUzMSwicGF0aCI6Ii81MTQ0MzY3MS80NTg5NTQ5MDQtOTI3OTIwYWEtYTcyNC00YTQwLWI1OGEtYjFiNWMzOGQyMjAwLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTA2MzAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwNjMwVDA3NDUzMVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTkwYzYzYTc2YzZjNGZkNjVkMzI0NzVmOGVhZjI3MGEzNmFkYmI3NWY2OWEwMWZmODY0NjUxZmM0NTA5ZjkwOWImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.ZjATaHi_LNbPYPG32qg5zyiRMS2nzhPislPYd2-uLYo)
   > 
   > We reduced the size of the main docker image by 1.11 GB.
   
   Hi @alexanderschmi 
   
   Thanks for the PR, it is a great improvement!
   
   Based on your pictures, did we move the SystemDS folder to the root? If so, would it not be better if we move it back to the previous position?
   
   Best regards
   Sebastian


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [SYSTEMDS-3877] Docker image size reduction and safety improvement [systemds]

Posted by "alexanderschmi (via GitHub)" <gi...@apache.org>.
alexanderschmi commented on PR #2274:
URL: https://github.com/apache/systemds/pull/2274#issuecomment-3048316537

   # Dockerfile Refactoring: `latest` & `testing-latest`
   
   ## Summary
   This pull request refactors the two main Docker images of SystemDS — `latest` and `testing-latest` — with the goal of reducing image size, improving build efficiency, and modernizing the base images.
   
   ---
   
   ## Changes in Detail
   
   ### `latest`
   - Switched from Ubuntu to Alpine  
     → The base operating system was changed to Alpine Linux to significantly reduce image size.  
     → All package installations were adapted to use `apk`, and unnecessary dependencies were removed.
   - Minimal, production-oriented setup  
     → Only essential runtime dependencies are included  
     → Usage of jre instead of jdk in final image because of multi-stage approach
     → Integration of native hadoop library
   
   ### `testing-latest`
   - Switched from Ubuntu to debian:bullseye-slim
     → The operating system was changed to reduce image size.
   - Migrated to a multi-stage build  
     → The previous single-stage build was replaced with a multi-stage setup 
     → The final testing image is smaller
   - Fully functional test environment  
     → All unit and integration tests run as before  
     → CI integration remains fully compatible
   
   ---
   
   ## Results
   - `latest`: 
       - smaller due to Alpine base (from 1.5 GiB to 460 MiB → reduced by 1.1 GB)
       - more secure through Alpine base and a default non-root user
   - `testing-latest`: 
       - smaller due to  bullseye-slim base (from 1.7 GiB to 1.4 GiB → reduced by 325 MB)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [SYSTEMDS-3877] Docker image size reduction and safety improvement [systemds]

Posted by "alexanderschmi (via GitHub)" <gi...@apache.org>.
alexanderschmi commented on PR #2274:
URL: https://github.com/apache/systemds/pull/2274#issuecomment-3005301604

   ## Size Reduction
   
   ### Before 
   
   ![Before](https://github.com/user-attachments/assets/eeff534c-4610-405f-a95a-a436b450c6b0)
   
   ### After 
   
   ![After](https://github.com/user-attachments/assets/927920aa-a724-4a40-b58a-b1b5c38d2200)
   
   We reduced the size of the main docker image by 1.11 GB. 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [SYSTEMDS-3877] Docker image size reduction and safety improvement [systemds]

Posted by "Baunsgaard (via GitHub)" <gi...@apache.org>.
Baunsgaard commented on PR #2274:
URL: https://github.com/apache/systemds/pull/2274#issuecomment-2980458543

   An example of B security:
   
   https://hub.docker.com/layers/apache/spark-connect-swift/pi/images/sha256-ff987ef8350ee3032268af288c2479728d7c6b6cfca626741bf1216093d57382
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [SYSTEMDS-3877] Docker image size reduction and safety improvement [systemds]

Posted by "Baunsgaard (via GitHub)" <gi...@apache.org>.
Baunsgaard commented on PR #2274:
URL: https://github.com/apache/systemds/pull/2274#issuecomment-2980410184

   Thanks for the PR, @alexanderschmi 
   
   While looking into the docker images could we also look at security vulnerabilities:
   
   https://hub.docker.com/repository/docker/apache/systemds/general
   ![Screenshot from 2025-06-17 15-29-52](https://github.com/user-attachments/assets/a9ff050d-6141-4640-b608-9c0431aeef2b)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [SYSTEMDS-3877] Docker image size reduction and safety improvement [systemds]

Posted by "Baunsgaard (via GitHub)" <gi...@apache.org>.
Baunsgaard commented on code in PR #2274:
URL: https://github.com/apache/systemds/pull/2274#discussion_r2192107569


##########
docker/sysds.Dockerfile:
##########
@@ -43,22 +41,25 @@ ENV SYSTEMDS_ROOT=/usr/src/systemds
 ENV PATH=$SYSTEMDS_ROOT/bin:$PATH
 ENV SYSDS_QUIET=1
 
-# Download Java and Mvn 
+# Download Mvn and JDK
 RUN mkdir -p /usr/lib/jvm \
 	&& wget -qO- \
-	https://github.com/adoptium/temurin17-binaries/releases/download/jdk-17.0.15%2B6/OpenJDK17U-jdk_x64_linux_hotspot_17.0.15_6.tar.gz | tar xzf - \
-	&& mv jdk-17.0.15+6 /usr/lib/jvm/jdk-17.0.15+6 \
+	https://github.com/adoptium/temurin17-binaries/releases/download/jdk-17.0.15%2B6/OpenJDK17U-jdk_x64_alpine-linux_hotspot_17.0.15_6.tar.gz  | tar xzf - \
+	&& mv jdk-17.0.15+6 $JAVA_HOME \
 	&& wget -qO- \
 	http://archive.apache.org/dist/maven/maven-3/$MAVEN_VERSION/binaries/apache-maven-$MAVEN_VERSION-bin.tar.gz | tar xzf - \ 
 	&& mv apache-maven-$MAVEN_VERSION /usr/lib/mvn
 
 # Build the system
-RUN git clone --depth 1 https://github.com/apache/systemds.git systemds && \
+RUN git clone -b SYSTEMDS-3877 --depth 1 https://github.com/alexanderschmi/systemds.git systemds && \

Review Comment:
   when cleaning up the PR, we need to change this line.



##########
docker/mountFolder/main.dml:
##########
@@ -19,4 +19,15 @@
 #
 #-------------------------------------------------------------
 
-print("Hello, World!");
\ No newline at end of file
+# kmeans_test.dml
+# Simple KMeans clustering test
+
+# Arguments
+X = read("test.bin")
+
+# Run KMeans
+[centroids, assignments] = kmeans(X=X, k=2, max_iter=100, eps=0.01)
+
+print(centroids)
+print(assignments)
+write(centroids, "centroids.bin")

Review Comment:
   lets add a newline in the end of this file.



##########
docker/mountFolder/.gitignore:
##########


Review Comment:
   remove this file, and instead edit the main .gitignore file in the root of the project.



##########
docker/mountFolder/main.dml:
##########
@@ -19,4 +19,15 @@
 #
 #-------------------------------------------------------------
 
-print("Hello, World!");
\ No newline at end of file
+# kmeans_test.dml
+# Simple KMeans clustering test
+
+# Arguments
+X = read("test.bin")

Review Comment:
   This will only work if there is a test.bin file. Please change back to the in place construction of the matrix.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [SYSTEMDS-3877] Docker image size reduction and safety improvement [systemds]

Posted by "Baunsgaard (via GitHub)" <gi...@apache.org>.
Baunsgaard commented on PR #2274:
URL: https://github.com/apache/systemds/pull/2274#issuecomment-3081251692

   LGTM, will merge once the images are built.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org