You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "zhengruifeng (via GitHub)" <gi...@apache.org> on 2023/08/01 02:37:07 UTC
[GitHub] [spark] zhengruifeng opened a new pull request, #42253: [SPARK-44619][INFRA] Free up disk space for container jobs
zhengruifeng opened a new pull request, #42253:
URL: https://github.com/apache/spark/pull/42253
### What changes were proposed in this pull request?
Free up disk space for container jobs
### Why are the changes needed?
increase the available disk space
### Does this PR introduce _any_ user-facing change?
No, infra-only
### How was this patch tested?
updated CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #42253: [SPARK-44619][INFRA] Free up disk space for pyspark container jobs
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #42253:
URL: https://github.com/apache/spark/pull/42253#discussion_r1280055884
##########
.github/workflows/build_and_test.yml:
##########
@@ -408,6 +408,14 @@ jobs:
key: pyspark-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
pyspark-coursier-
+ - name: Free up disk space
+ run: |
+ dpkg-query -Wf '${Installed-Size}\t${Package}\n' | sort -n | tail -n 100
+ df -h
+ rm -rf /__t/CodeQL || echo "fail to delete /__t/CodeQL"
+ rm -rf /__t/go || echo "fail to delete /__t/go"
+ rm -rf /__t/node || echo "fail to delete /__t/node"
Review Comment:
I can not find an official way to uninstall [CodeQL](https://codeql.github.com/docs/codeql-overview/about-codeql/#:~:text=In%20CodeQL%2C%20code%20is%20treated%20like%20data.%20Security,write%20your%20own%20to%20use%20in%20custom%20analyses.) (like pip/apt-get/etc)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #42253: [SPARK-44619][INFRA] Free up disk space for pyspark container jobs
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #42253:
URL: https://github.com/apache/spark/pull/42253#issuecomment-1661304234
let me take another look to check whether it is safe to uninstall CodeQL and what is the proper way to do
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #42253: [SPARK-44619][INFRA] Free up disk space for container jobs
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #42253:
URL: https://github.com/apache/spark/pull/42253#issuecomment-1664861475
thanks, merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #42253: [SPARK-44619][INFRA] Free up disk space for pyspark container jobs
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #42253:
URL: https://github.com/apache/spark/pull/42253#issuecomment-1661415437
Since the CI passes fine for the time being, we can just drop this too for now if this takes too much time to investigate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #42253: [SPARK-44619][INFRA] Free up disk space for pyspark container jobs
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #42253:
URL: https://github.com/apache/spark/pull/42253#discussion_r1280463097
##########
.github/workflows/build_and_test.yml:
##########
@@ -408,6 +408,14 @@ jobs:
key: pyspark-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
pyspark-coursier-
+ - name: Free up disk space
+ run: |
+ dpkg-query -Wf '${Installed-Size}\t${Package}\n' | sort -n | tail -n 100
+ df -h
+ rm -rf /__t/CodeQL || echo "fail to delete /__t/CodeQL"
+ rm -rf /__t/go || echo "fail to delete /__t/go"
+ rm -rf /__t/node || echo "fail to delete /__t/node"
Review Comment:
TBH, I don't find a good way to remove CodeQL/Go/Node
also cc @LuciferYang @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng closed pull request #42253: [SPARK-44619][INFRA] Free up disk space for container jobs
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng closed pull request #42253: [SPARK-44619][INFRA] Free up disk space for container jobs
URL: https://github.com/apache/spark/pull/42253
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a diff in pull request #42253: [SPARK-44619][INFRA] Free up disk space for pyspark container jobs
Posted by "Yikun (via GitHub)" <gi...@apache.org>.
Yikun commented on code in PR #42253:
URL: https://github.com/apache/spark/pull/42253#discussion_r1280479837
##########
dev/infra/Dockerfile:
##########
@@ -73,3 +74,8 @@ RUN python3.9 -m pip install grpcio protobuf googleapis-common-protos grpcio-sta
# Add torch as a testing dependency for TorchDistributor
RUN python3.9 -m pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu
RUN python3.9 -m pip install torcheval
+
+# Uninstall unused packages
+RUN $APT_REMOVE libgl1-mesa-dri
+RUN apt-get autoremove --purge -y
+RUN apt-get clean
Review Comment:
According to https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#run
```
Official Debian and Ubuntu images [automatically run apt-get clean](https://github.com/moby/moby/blob/03e2923e42446dbb830c654d0eec323a0b4ef02a/contrib/mkimage/debootstrap#L82-L105), so explicit invocation is not required.
```
It already be removed. But add `rm -rf /var/lib/apt/lists/*` to the end of apt install line might help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #42253: [SPARK-44619][INFRA] Free up disk space for container jobs
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #42253:
URL: https://github.com/apache/spark/pull/42253#issuecomment-1663750853
cc @HyukjinKwon @Yikun @LuciferYang would you mind taking another look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #42253: [SPARK-44619][INFRA] Free up disk space for pyspark container jobs
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #42253:
URL: https://github.com/apache/spark/pull/42253#discussion_r1280481054
##########
.github/workflows/build_and_test.yml:
##########
@@ -408,6 +408,14 @@ jobs:
key: pyspark-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
pyspark-coursier-
+ - name: Free up disk space
+ run: |
+ dpkg-query -Wf '${Installed-Size}\t${Package}\n' | sort -n | tail -n 100
+ df -h
+ rm -rf /__t/CodeQL || echo "fail to delete /__t/CodeQL"
+ rm -rf /__t/go || echo "fail to delete /__t/go"
+ rm -rf /__t/node || echo "fail to delete /__t/node"
Review Comment:
is `CodeQL` installed by the `Ubuntu 22.04` runner?
I am surprised that I don't find it in a non-container job (in https://github.com/apache/spark/pull/42241)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #42253: [SPARK-44619][INFRA] Free up disk space for container jobs
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #42253:
URL: https://github.com/apache/spark/pull/42253#discussion_r1280053406
##########
.github/workflows/build_and_test.yml:
##########
@@ -408,6 +408,14 @@ jobs:
key: pyspark-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
pyspark-coursier-
+ - name: Free up disk space
+ run: |
+ dpkg-query -Wf '${Installed-Size}\t${Package}\n' | sort -n | tail -n 100
+ df -h
+ rm -rf /__t/CodeQL || echo "fail to delete /__t/CodeQL"
+ rm -rf /__t/go || echo "fail to delete /__t/go"
+ rm -rf /__t/node || echo "fail to delete /__t/node"
Review Comment:
deleting those directories in `Dockerfile` takes no effect ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #42253: [WIP][SPARK-44619][INFRA] Free up disk space for pyspark container jobs
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #42253:
URL: https://github.com/apache/spark/pull/42253#discussion_r1280357149
##########
.github/workflows/build_and_test.yml:
##########
@@ -408,6 +408,14 @@ jobs:
key: pyspark-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
pyspark-coursier-
+ - name: Free up disk space
+ run: |
+ dpkg-query -Wf '${Installed-Size}\t${Package}\n' | sort -n | tail -n 100
+ df -h
+ rm -rf /__t/CodeQL || echo "fail to delete /__t/CodeQL"
+ rm -rf /__t/go || echo "fail to delete /__t/go"
+ rm -rf /__t/node || echo "fail to delete /__t/node"
Review Comment:
cc @Yikun do you happen to know can we remove `/__t/CodeQL` in this way?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #42253: [SPARK-44619][INFRA] Free up disk space for pyspark container jobs
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #42253:
URL: https://github.com/apache/spark/pull/42253#discussion_r1280463097
##########
.github/workflows/build_and_test.yml:
##########
@@ -408,6 +408,14 @@ jobs:
key: pyspark-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
pyspark-coursier-
+ - name: Free up disk space
+ run: |
+ dpkg-query -Wf '${Installed-Size}\t${Package}\n' | sort -n | tail -n 100
+ df -h
+ rm -rf /__t/CodeQL || echo "fail to delete /__t/CodeQL"
+ rm -rf /__t/go || echo "fail to delete /__t/go"
+ rm -rf /__t/node || echo "fail to delete /__t/node"
Review Comment:
TBH, I don't find a good way to uninstall CodeQL/Go/Node
also cc @LuciferYang @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #42253: [SPARK-44619][INFRA] Free up disk space for container jobs
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #42253:
URL: https://github.com/apache/spark/pull/42253#issuecomment-1663630035
Different from the non-container jobs, there are not many unneeded apt libraries in container jobs.
I discard the `apt-get remove` because:
1, uninstall `libgl1-mesa-dri` only free about 500MiB disk;
2, cause this new script can not work with job `lint`;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #42253: [SPARK-44619][INFRA] Free up disk space for pyspark container jobs
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #42253:
URL: https://github.com/apache/spark/pull/42253#discussion_r1281351440
##########
dev/infra/Dockerfile:
##########
@@ -73,3 +74,8 @@ RUN python3.9 -m pip install grpcio protobuf googleapis-common-protos grpcio-sta
# Add torch as a testing dependency for TorchDistributor
RUN python3.9 -m pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu
RUN python3.9 -m pip install torcheval
+
+# Uninstall unused packages
+RUN $APT_REMOVE libgl1-mesa-dri
+RUN apt-get autoremove --purge -y
+RUN apt-get clean
Review Comment:
since this PR aims to uninstall pkgs, I would like to send another PR to following this ` rm -rf /var/lib/apt/lists/* ` guide
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #42253: [SPARK-44619][INFRA] Free up disk space for container jobs
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #42253:
URL: https://github.com/apache/spark/pull/42253#discussion_r1280054159
##########
.github/workflows/build_and_test.yml:
##########
@@ -408,6 +408,14 @@ jobs:
key: pyspark-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
pyspark-coursier-
+ - name: Free up disk space
+ run: |
+ dpkg-query -Wf '${Installed-Size}\t${Package}\n' | sort -n | tail -n 100
+ df -h
+ rm -rf /__t/CodeQL || echo "fail to delete /__t/CodeQL"
+ rm -rf /__t/go || echo "fail to delete /__t/go"
+ rm -rf /__t/node || echo "fail to delete /__t/node"
Review Comment:
```
88M /__e/node12
55M /__e/node12_alpine
101M /__e/node16
89M /__e/node16_alpine
8.2G /__t/CodeQL
16K /__t/Java_Temurin-Hotspot_jdk
487M /__t/PyPy
1.2G /__t/Python
62M /__t/Ruby
1.2G /__t/go
379M /__t/node
16K /__w/_PipelineMapping
26M /__w/_actions
68K /__w/_temp
681M /__w/spark
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #42253: [SPARK-44619][INFRA] Free up disk space for pyspark container jobs
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #42253:
URL: https://github.com/apache/spark/pull/42253#issuecomment-1661437670
> Since the CI passes fine for the time being, we can just drop this too for now if this takes too much time to investigate.
nice!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Yikun commented on a diff in pull request #42253: [SPARK-44619][INFRA] Free up disk space for pyspark container jobs
Posted by "Yikun (via GitHub)" <gi...@apache.org>.
Yikun commented on code in PR #42253:
URL: https://github.com/apache/spark/pull/42253#discussion_r1280475423
##########
.github/workflows/build_and_test.yml:
##########
@@ -408,6 +408,14 @@ jobs:
key: pyspark-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
pyspark-coursier-
+ - name: Free up disk space
+ run: |
+ dpkg-query -Wf '${Installed-Size}\t${Package}\n' | sort -n | tail -n 100
+ df -h
+ rm -rf /__t/CodeQL || echo "fail to delete /__t/CodeQL"
+ rm -rf /__t/go || echo "fail to delete /__t/go"
+ rm -rf /__t/node || echo "fail to delete /__t/node"
Review Comment:
According the link:
https://github.com/actions/runner-images/blob/main/images/linux/Ubuntu2204-Readme.md#installed-apt-packages
seems only above package can be unintall by apt, for other one, we can only cleanup by this way, so this PR seems a okay way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #42253: [SPARK-44619][INFRA] Free up disk space for pyspark container jobs
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #42253:
URL: https://github.com/apache/spark/pull/42253#issuecomment-1662017994
let me focus on removing the tool directories first and skip the changes in dockerfile
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org