You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/10/18 09:16:16 UTC
[GitHub] [spark] Yikun opened a new pull request, #38298: [SPARK-40833][K8S] Cleanup apt lists cache
Yikun opened a new pull request, #38298:
URL: https://github.com/apache/spark/pull/38298
### What changes were proposed in this pull request?
Remove unused apt lists cache
### Why are the changes needed?
Clean cache to reduce docker image size.
This is also [recommanded](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#run) by docker community:
```
$ docker run --user 0:0 -ti apache/spark bash
root@5d1ca347279e:/opt/spark/work-dir# ls /var/lib/apt/lists/
auxfiles lock
deb.debian.org_debian_dists_bullseye-updates_InRelease partial
deb.debian.org_debian_dists_bullseye-updates_main_binary-arm64_Packages.lz4 security.debian.org_debian-security_dists_bullseye-security_InRelease
deb.debian.org_debian_dists_bullseye_InRelease security.debian.org_debian-security_dists_bullseye-security_main_binary-arm64_Packages.lz4
deb.debian.org_debian_dists_bullseye_main_binary-arm64_Packages.lz4
root@5d1ca347279e:/opt/spark/work-dir# du --max-depth=1 -h /var/lib/apt/lists/
4.0K /var/lib/apt/lists/partial
4.0K /var/lib/apt/lists/auxfiles
17M /var/lib/apt/lists/
```
### Does this PR introduce _any_ user-facing change?
Yes in some level, image size is reduced.
### How was this patch tested?
K8s CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on pull request #38298: [SPARK-40833][K8S] Cleanup apt lists cache
Posted by GitBox <gi...@apache.org>.
Yikun commented on PR #38298:
URL: https://github.com/apache/spark/pull/38298#issuecomment-1283387373
cc @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on pull request #38298: [SPARK-40833][K8S] Cleanup apt lists cache
Posted by GitBox <gi...@apache.org>.
Yikun commented on PR #38298:
URL: https://github.com/apache/spark/pull/38298#issuecomment-1284669252
@dongjoon-hyun OK, thanks, let's just keep different for now! If a submodule is really needed, we can do it in future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #38298: [SPARK-40833][K8S] Cleanup apt lists cache
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #38298: [SPARK-40833][K8S] Cleanup apt lists cache
URL: https://github.com/apache/spark/pull/38298
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #38298: [SPARK-40833][K8S] Cleanup apt lists cache
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #38298:
URL: https://github.com/apache/spark/pull/38298#issuecomment-1284671317
Ya, feel free to propose any changes if it helps you and don't blocked by me. You are now a committer with more power, @Yikun . :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #38298: [SPARK-40833][K8S] Cleanup apt lists cache
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #38298:
URL: https://github.com/apache/spark/pull/38298#issuecomment-1284361786
Ur, @Yikun . Why do you use a different style per repo?
- https://github.com/apache/spark-docker/pull/14/files
```
- rm -rf /var/cache/apt/*
+ rm -rf /var/cache/apt/* && \
+ rm -rf /var/lib/apt/lists/*
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on pull request #38298: [SPARK-40833][K8S] Cleanup apt lists cache
Posted by GitBox <gi...@apache.org>.
Yikun commented on PR #38298:
URL: https://github.com/apache/spark/pull/38298#issuecomment-1284656776
BTW, what do you think about the dockerfile being in sync between spark-docker and spark?
spark-docker contains some changes for non-k8s to meet the requirements of DOI. such as, remove print line in entrypoint, add users, expose port. Do we want to also sync them in spark repo?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #38298: [SPARK-40833][K8S] Cleanup apt lists cache
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on PR #38298:
URL: https://github.com/apache/spark/pull/38298#issuecomment-1284658665
We don't need to sync~ If we really want to sync, we had better make `spark-docker` git repo as a submodule of `spark` repo.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on pull request #38298: [SPARK-40833][K8S] Cleanup apt lists cache
Posted by GitBox <gi...@apache.org>.
Yikun commented on PR #38298:
URL: https://github.com/apache/spark/pull/38298#issuecomment-1284656370
@dongjoon-hyun After spark-docker PR merged, I found the Python docker file are using one line style, so keep the same style in spark repo.
I'd also like to sync this style to spark-docker. : ), from functions view it's same, put similar function cmd (rm in here) in one line is better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org