You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/11/08 21:23:10 UTC

[GitHub] [beam] pabloem opened a new pull request, #24040: Improving stale container cleanup script

pabloem opened a new pull request, #24040:
URL: https://github.com/apache/beam/pull/24040

   This is the expected outcome from running this script today: https://gist.github.com/pabloem/95cff6aca3e17fd09f4c792a54d14dde
   
   Should we run it?
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
    - [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
   
   To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/workflows/Python%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/workflows/Java%20Tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Go tests](https://github.com/apache/beam/workflows/Go%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] pabloem commented on pull request #24040: Improving stale container cleanup script

Posted by GitBox <gi...@apache.org>.
pabloem commented on PR #24040:
URL: https://github.com/apache/beam/pull/24040#issuecomment-1309553431

   hm I've just realized that this script is super slow if there's a lot of containers to delete (maybe it's okay?)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] github-actions[bot] commented on pull request #24040: Improving stale container cleanup script

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #24040:
URL: https://github.com/apache/beam/pull/24040#issuecomment-1307844696

   Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on a diff in pull request #24040: Improving stale container cleanup script

Posted by GitBox <gi...@apache.org>.
Abacn commented on code in PR #24040:
URL: https://github.com/apache/beam/pull/24040#discussion_r1018327333


##########
.test-infra/tools/stale_dataflow_prebuilt_image_cleaner.sh:
##########
@@ -19,15 +19,38 @@
 #
 set -euo pipefail
 
-STALE_IMAGES=$(gcloud container images list-tags \
-gcr.io/apache-beam-testing/prebuilt_beam_sdk/beam_python_prebuilt_sdk \
---sort-by=TIMESTAMP  --filter="timestamp.datetime < $(date --iso-8601=s -d '1 day ago')" \
---format="get(digest)")
+REPOSITORIES=(prebuilt_beam_sdk/beam_python_prebuilt_sdk beam-sdk beam_portability)
 
-if [[ ${STALE_IMAGES} ]]; then
-  for digest in ${STALE_IMAGES}; do
-    gcloud container images delete gcr.io/apache-beam-testing/prebuilt_beam_sdk/beam_python_prebuilt_sdk@"$digest" --force-delete-tags -q
+echo $REPOSITORIES
+
+for repository in ${REPOSITORIES[@]}; do
+  echo IMAGES FOR REPO ${repository}
+  IMAGE_NAMES+=$(gcloud container images list --repository=gcr.io/apache-beam-testing/${repository} --format="get(name)")
+  IMAGE_NAMES+=" "
+done
+
+echo $IMAGE_NAMES
+
+for image_name in ${IMAGE_NAMES[@]}; do
+  echo IMAGES FOR image ${image_name}
+  echo "Command" gcloud container images list-tags \
+  ${image_name} \
+  --sort-by=TIMESTAMP  --filter="timestamp.datetime < $(date --iso-8601=s -d '5 days ago')" \

Review Comment:
   ```suggestion
     --sort-by=TIMESTAMP  --filter="NOT tags:latest AND timestamp.datetime < $(date --iso-8601=s -d '5 days ago')" \
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] pabloem commented on pull request #24040: Improving stale container cleanup script

Posted by GitBox <gi...@apache.org>.
pabloem commented on PR #24040:
URL: https://github.com/apache/beam/pull/24040#issuecomment-1307960581

   that makes sense to me. I'll look into it...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #24040: Improving stale container cleanup script

Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #24040:
URL: https://github.com/apache/beam/pull/24040#issuecomment-1310562129

   btw if there is a worry we can reduce the frequency of this job to daily. Currently it runs 6 times a day which is a lot. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] pabloem commented on pull request #24040: Improving stale container cleanup script

Posted by GitBox <gi...@apache.org>.
pabloem commented on PR #24040:
URL: https://github.com/apache/beam/pull/24040#issuecomment-1307843433

   r: @Abacn 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #24040: Improving stale container cleanup script

Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #24040:
URL: https://github.com/apache/beam/pull/24040#issuecomment-1309620118

   > 
   
   It's okay, its even okay if the cron job timeout for first couple of times. I also tested the script locally (commented out the actual delete container call), it only runs a couple of seconds. So if they are not many thing to delete it should not be too slow.
   
   There are currently 30,000 containers which take time to clean up.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] pabloem merged pull request #24040: Improving stale container cleanup script

Posted by GitBox <gi...@apache.org>.
pabloem merged PR #24040:
URL: https://github.com/apache/beam/pull/24040


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #24040: Improving stale container cleanup script

Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #24040:
URL: https://github.com/apache/beam/pull/24040#issuecomment-1307866102

   Thanks for the quick solution!
   
   Two concerns:
   the 5 day filter is fine for prebuilt_sdk but may be aggressive for beam-sdk and beam_portability, given that the docker publish jobs have history to have failed for a couple of days time range: https://ci-beam.apache.org/job/beam_Publish_Beam_SDK_Snapshots/4095/  I would suggest a 10 day or 30 day filter for beam-sdk and beam_portability;
   
   also we may want to keep some latest but no longer produced images as our nightly napshot doing: https://repository.apache.org/content/groups/snapshots/org/apache/beam/ for example, flink-1.12 is no longer generated but we may still want to preserve the latest 5 (or other number) days containers.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] pabloem commented on pull request #24040: Improving stale container cleanup script

Posted by GitBox <gi...@apache.org>.
pabloem commented on PR #24040:
URL: https://github.com/apache/beam/pull/24040#issuecomment-1309228375

   I've tested this locally and your fix works as expeted (now all the image categories are parsed properly)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] pabloem commented on pull request #24040: Improving stale container cleanup script

Posted by GitBox <gi...@apache.org>.
pabloem commented on PR #24040:
URL: https://github.com/apache/beam/pull/24040#issuecomment-1309228993

   if this LGTY, I will run it on my machine (since the first run will delete many many containers) - and merge it


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] pabloem commented on pull request #24040: Improving stale container cleanup script

Posted by GitBox <gi...@apache.org>.
pabloem commented on PR #24040:
URL: https://github.com/apache/beam/pull/24040#issuecomment-1310688617

   thanks Yi! Let's see how it behaves over the next few days


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #24040: Improving stale container cleanup script

Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #24040:
URL: https://github.com/apache/beam/pull/24040#issuecomment-1308107114

   Thanks again. Looking into the gist file I see this log:
   ```
   IMAGES FOR image gcr.io/apache-beam-testing/beam-sdk/beam_python3.9_sdkgcr.io/apache-beam-testing/beam_portability/beam_flink1.10_job_server
   Command gcloud container images list-tags gcr.io/apache-beam-testing/beam-sdk/beam_python3.9_sdkgcr.io/apache-beam-testing/beam_portability/beam_flink1.10_job_server --sort-by=TIMESTAMP --filter=timestamp.datetime < 2022-11-03T14:18:54-07:00 --format=get(digest)
   IMAGES FOR image gcr.io/apache-beam-testing/beam_portability/beam_flink1.12_job_server
   ```
   looks like for some reason image name are concatenated and Python3.9 images are not listed


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on a diff in pull request #24040: Improving stale container cleanup script

Posted by GitBox <gi...@apache.org>.
Abacn commented on code in PR #24040:
URL: https://github.com/apache/beam/pull/24040#discussion_r1017336866


##########
.test-infra/tools/stale_dataflow_prebuilt_image_cleaner.sh:
##########
@@ -19,15 +19,37 @@
 #
 set -euo pipefail
 
-STALE_IMAGES=$(gcloud container images list-tags \
-gcr.io/apache-beam-testing/prebuilt_beam_sdk/beam_python_prebuilt_sdk \
---sort-by=TIMESTAMP  --filter="timestamp.datetime < $(date --iso-8601=s -d '1 day ago')" \
---format="get(digest)")
+REPOSITORIES=(prebuilt_beam_sdk/beam_python_prebuilt_sdk beam-sdk beam_portability)
 
-if [[ ${STALE_IMAGES} ]]; then
-  for digest in ${STALE_IMAGES}; do
-    gcloud container images delete gcr.io/apache-beam-testing/prebuilt_beam_sdk/beam_python_prebuilt_sdk@"$digest" --force-delete-tags -q
+echo $REPOSITORIES
+
+for repository in ${REPOSITORIES[@]}; do
+  echo IMAGES FOR REPO ${repository}
+  IMAGE_NAMES+=$(gcloud container images list --repository=gcr.io/apache-beam-testing/${repository} --format="get(name)")

Review Comment:
   ```suggestion
     IMAGE_NAMES+=$(gcloud container images list --repository=gcr.io/apache-beam-testing/${repository} --format="get(name)")
     IMAGE_NAMES+=" "
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on a diff in pull request #24040: Improving stale container cleanup script

Posted by GitBox <gi...@apache.org>.
Abacn commented on code in PR #24040:
URL: https://github.com/apache/beam/pull/24040#discussion_r1018330443


##########
.test-infra/tools/stale_dataflow_prebuilt_image_cleaner.sh:
##########
@@ -19,15 +19,38 @@
 #
 set -euo pipefail
 
-STALE_IMAGES=$(gcloud container images list-tags \
-gcr.io/apache-beam-testing/prebuilt_beam_sdk/beam_python_prebuilt_sdk \
---sort-by=TIMESTAMP  --filter="timestamp.datetime < $(date --iso-8601=s -d '1 day ago')" \
---format="get(digest)")
+REPOSITORIES=(prebuilt_beam_sdk/beam_python_prebuilt_sdk beam-sdk beam_portability)
 
-if [[ ${STALE_IMAGES} ]]; then
-  for digest in ${STALE_IMAGES}; do
-    gcloud container images delete gcr.io/apache-beam-testing/prebuilt_beam_sdk/beam_python_prebuilt_sdk@"$digest" --force-delete-tags -q
+echo $REPOSITORIES
+
+for repository in ${REPOSITORIES[@]}; do
+  echo IMAGES FOR REPO ${repository}
+  IMAGE_NAMES+=$(gcloud container images list --repository=gcr.io/apache-beam-testing/${repository} --format="get(name)")
+  IMAGE_NAMES+=" "
+done
+
+echo $IMAGE_NAMES
+
+for image_name in ${IMAGE_NAMES[@]}; do
+  echo IMAGES FOR image ${image_name}
+  echo "Command" gcloud container images list-tags \
+  ${image_name} \
+  --sort-by=TIMESTAMP  --filter="timestamp.datetime < $(date --iso-8601=s -d '5 days ago')" \

Review Comment:
   nit (does not affect functionality)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] pabloem commented on pull request #24040: Improving stale container cleanup script

Posted by GitBox <gi...@apache.org>.
pabloem commented on PR #24040:
URL: https://github.com/apache/beam/pull/24040#issuecomment-1307971668

   The difference in the expected outcome is that 6 fewer images are deleted:
   
   ```
   diff new_expected_outcome expected_outcome  | grep Delete
   > Deleting image. Command: gcloud container images delete gcr.io/apache-beam-testing/beam-sdk/beam_python3.6_sdk@sha256:3f0ff833838b5945f1a9551a51f6d3223d7f11d4777360c678641c12c1995862 --force-delete-tags -q
   > Deleting image. Command: gcloud container images delete gcr.io/apache-beam-testing/beam_portability/beam_flink1.12_job_server@sha256:911283585036550af2b3ab86dd87aab88a229103f63aabf080ebec9be3589a3b --force-delete-tags -q
   > Deleting image. Command: gcloud container images delete gcr.io/apache-beam-testing/beam_portability/beam_go_sdk@sha256:de12cb02d386c0efa1799b39c9a8a9f5035ad1bf29248440909e2223fea8c157 --force-delete-tags -q
   > Deleting image. Command: gcloud container images delete gcr.io/apache-beam-testing/beam_portability/beam_java_sdk@sha256:a602d9c25750a6c1351f7703977e03d5a45a0e7bb762958473ef68a873d1eacd --force-delete-tags -q
   > Deleting image. Command: gcloud container images delete gcr.io/apache-beam-testing/beam_portability/beam_python2.7_sdk@sha256:4feefecb1b41a4c7273e46266fab06a6215e6b985613bab88921a6edbcbdd48e --force-delete-tags -q
   > Deleting image. Command: gcloud container images delete gcr.io/apache-beam-testing/beam_portability/beam_python3.6_sdk@sha256:106ece6c7d8bd11f35250b72c71674735065b176332ecfffc01aa86a47504c7b --force-delete-tags -q
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] pabloem commented on pull request #24040: Improving stale container cleanup script

Posted by GitBox <gi...@apache.org>.
pabloem commented on PR #24040:
URL: https://github.com/apache/beam/pull/24040#issuecomment-1309216788

   wow great catch : )


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org