You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/06/22 11:53:17 UTC
[GitHub] [beam] TobKed opened a new pull request #12049: GitHub Actions periodic GCS clear [dependent on [BEAM-10184]]
TobKed opened a new pull request #12049:
URL: https://github.com/apache/beam/pull/12049
Once per month clear GCS folders which do not have corresponding branches.
### Before merging
Before merging it is required to setup related secrets:
* `GCP_SA_EMAIL`
* `GCP_SA_KEY`
* `GCP_BUCKET`
Depends on https://github.com/apache/beam/pull/11877
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] potiuk commented on a change in pull request #12049: [BEAM-10399] Periodic clear of GCS wheels staging bucket
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #12049:
URL: https://github.com/apache/beam/pull/12049#discussion_r448397786
##########
File path: .github/workflows/clear_staging_bucket.yml
##########
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: Clear staging bucket
+
+on:
+ schedule:
+ - cron: '0 0 1 * *'
+
+
+env:
+ GCP_PATH: "gs://beam-wheels-staging"
+
+
+jobs:
+ clear_staging_bucket:
+ name: Clear bucket
+ runs-on: ubuntu-latest
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v2
+ with:
+ fetch-depth: 0
+ - name: Authenticate on GCP
+ uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
+ with:
+ project_id: ${{ secrets.GCP_PROJECT_ID }}
+ service_account_email: ${{ secrets.GCP_SA_EMAIL }}
+ service_account_key: ${{ secrets.GCP_SA_KEY }}
+ - name: Delete folders without corresponding branch in repo
+ run: |
+ declare -a FOLDERS_TO_BE_DELETED=()
Review comment:
I think it's much better to put it all in separate bash script and call it from there. It's easier to analyse it this way
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay edited a comment on pull request #12049: [BEAM-10399] Periodic clear of GCS wheels staging bucket
Posted by GitBox <gi...@apache.org>.
aaltay edited a comment on pull request #12049:
URL: https://github.com/apache/beam/pull/12049#issuecomment-655635243
What files will be cleared this way? Release branches live forever. Would there be other branches that this script will clean?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TobKed closed pull request #12049: [BEAM-10399] Periodic clear of GCS wheels staging bucket
Posted by GitBox <gi...@apache.org>.
TobKed closed pull request #12049:
URL: https://github.com/apache/beam/pull/12049
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mik-laj commented on a change in pull request #12049: [BEAM-10399] Periodic clear of GCS wheels staging bucket
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #12049:
URL: https://github.com/apache/beam/pull/12049#discussion_r448427194
##########
File path: .github/workflows/clear_staging_bucket.yml
##########
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: Clear staging bucket
+
+on:
+ schedule:
+ - cron: '0 0 1 * *'
+
+
+env:
+ GCP_PATH: "gs://beam-wheels-staging"
+
+
+jobs:
+ clear_staging_bucket:
+ name: Clear bucket
+ runs-on: ubuntu-latest
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v2
+ with:
+ fetch-depth: 0
+ - name: Authenticate on GCP
+ uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
+ with:
+ project_id: ${{ secrets.GCP_PROJECT_ID }}
+ service_account_email: ${{ secrets.GCP_SA_EMAIL }}
+ service_account_key: ${{ secrets.GCP_SA_KEY }}
+ - name: Delete folders without corresponding branch in repo
+ run: |
+ declare -a FOLDERS_TO_BE_DELETED=()
Review comment:
I agree. The bash script contained in the YAML file cannot be validated by shellcheck.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TobKed commented on pull request #12049: [BEAM-10399] Periodic clear of GCS wheels staging bucket
Posted by GitBox <gi...@apache.org>.
TobKed commented on pull request #12049:
URL: https://github.com/apache/beam/pull/12049#issuecomment-657443946
At the moment only master and release branches upload wheels to GCS. and after learning that release branches are kept forever this PR seems has no sense.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #12049: [BEAM-10399] Periodic clear of GCS wheels staging bucket
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #12049:
URL: https://github.com/apache/beam/pull/12049#issuecomment-655635243
What files will be cleared this way? Release branches leave forever. Would there be other branches that this script will clean?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TobKed commented on pull request #12049: [BEAM-10399] Periodic clear of GCS wheels staging bucket
Posted by GitBox <gi...@apache.org>.
TobKed commented on pull request #12049:
URL: https://github.com/apache/beam/pull/12049#issuecomment-655590189
R: @kamilwu @aaltay
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TobKed commented on a change in pull request #12049: [BEAM-10399] Periodic clear of GCS wheels staging bucket
Posted by GitBox <gi...@apache.org>.
TobKed commented on a change in pull request #12049:
URL: https://github.com/apache/beam/pull/12049#discussion_r448909493
##########
File path: .github/workflows/clear_staging_bucket.yml
##########
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: Clear staging bucket
+
+on:
+ schedule:
+ - cron: '0 0 1 * *'
+
+
+env:
+ GCP_PATH: "gs://beam-wheels-staging"
+
+
+jobs:
+ clear_staging_bucket:
+ name: Clear bucket
+ runs-on: ubuntu-latest
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v2
+ with:
+ fetch-depth: 0
+ - name: Authenticate on GCP
+ uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
+ with:
+ project_id: ${{ secrets.GCP_PROJECT_ID }}
+ service_account_email: ${{ secrets.GCP_SA_EMAIL }}
+ service_account_key: ${{ secrets.GCP_SA_KEY }}
+ - name: Delete folders without corresponding branch in repo
+ run: |
+ declare -a FOLDERS_TO_BE_DELETED=()
Review comment:
Thanks for the suggestion. I moved it to the separate file. Looks much more nicer and it is easier do test script locally.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mik-laj commented on a change in pull request #12049: [BEAM-10399] Periodic clear of GCS wheels staging bucket
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #12049:
URL: https://github.com/apache/beam/pull/12049#discussion_r448427194
##########
File path: .github/workflows/clear_staging_bucket.yml
##########
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: Clear staging bucket
+
+on:
+ schedule:
+ - cron: '0 0 1 * *'
+
+
+env:
+ GCP_PATH: "gs://beam-wheels-staging"
+
+
+jobs:
+ clear_staging_bucket:
+ name: Clear bucket
+ runs-on: ubuntu-latest
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v2
+ with:
+ fetch-depth: 0
+ - name: Authenticate on GCP
+ uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
+ with:
+ project_id: ${{ secrets.GCP_PROJECT_ID }}
+ service_account_email: ${{ secrets.GCP_SA_EMAIL }}
+ service_account_key: ${{ secrets.GCP_SA_KEY }}
+ - name: Delete folders without corresponding branch in repo
+ run: |
+ declare -a FOLDERS_TO_BE_DELETED=()
Review comment:
I agree. The bash script contained in the YAML file cannot be verified by shellcheck.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org