You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/08/15 07:02:23 UTC
[GitHub] [beam] vchunikhin opened a new pull request, #22722: [Playground] [Infrastructure] Deleting the Cloud Storage Client
vchunikhin opened a new pull request, #22722:
URL: https://github.com/apache/beam/pull/22722
fixes #22720
@olehborysevych
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #22722: [Playground] [Infrastructure] Deleting the Cloud Storage Client
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on PR #22722:
URL: https://github.com/apache/beam/pull/22722#issuecomment-1233893407
# [Codecov](https://codecov.io/gh/apache/beam/pull/22722?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#22722](https://codecov.io/gh/apache/beam/pull/22722?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7e2897f) into [master](https://codecov.io/gh/apache/beam/commit/4844af152fface83961dd3f4e89022d1e5bef6d6?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (4844af1) will **increase** coverage by `0.00%`.
> The diff coverage is `66.56%`.
```diff
@@ Coverage Diff @@
## master #22722 +/- ##
========================================
Coverage 73.68% 73.68%
========================================
Files 713 714 +1
Lines 94988 95201 +213
========================================
+ Hits 69992 70153 +161
- Misses 23695 23751 +56
+ Partials 1301 1297 -4
```
| Flag | Coverage Δ | |
|---|---|---|
| go | `51.03% <29.72%> (-0.01%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/22722?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/runtime/exec/userstate.go](https://codecov.io/gh/apache/beam/pull/22722/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy91c2Vyc3RhdGUuZ28=) | `5.46% <0.00%> (-0.62%)` | :arrow_down: |
| [...thon/apache\_beam/ml/inference/pytorch\_inference.py](https://codecov.io/gh/apache/beam/pull/22722/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvaW5mZXJlbmNlL3B5dG9yY2hfaW5mZXJlbmNlLnB5) | `0.00% <0.00%> (ø)` | |
| [sdks/python/apache\_beam/typehints/schemas.py](https://codecov.io/gh/apache/beam/pull/22722/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3NjaGVtYXMucHk=) | `93.84% <ø> (ø)` | |
| [sdks/python/apache\_beam/typehints/\_\_init\_\_.py](https://codecov.io/gh/apache/beam/pull/22722/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL19faW5pdF9fLnB5) | `77.77% <66.66%> (-22.23%)` | :arrow_down: |
| [sdks/python/apache\_beam/dataframe/schemas.py](https://codecov.io/gh/apache/beam/pull/22722/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL3NjaGVtYXMucHk=) | `96.62% <92.30%> (-1.05%)` | :arrow_down: |
| [sdks/python/apache\_beam/dataframe/convert.py](https://codecov.io/gh/apache/beam/pull/22722/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZGF0YWZyYW1lL2NvbnZlcnQucHk=) | `91.20% <93.47%> (+0.83%)` | :arrow_up: |
| [...apache\_beam/typehints/pandas\_type\_compatibility.py](https://codecov.io/gh/apache/beam/pull/22722/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3BhbmRhc190eXBlX2NvbXBhdGliaWxpdHkucHk=) | `94.95% <94.95%> (ø)` | |
| [sdks/go/pkg/beam/core/state/state.go](https://codecov.io/gh/apache/beam/pull/22722/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3N0YXRlL3N0YXRlLmdv) | `83.60% <100.00%> (+2.74%)` | :arrow_up: |
| [sdks/python/apache\_beam/typehints/batch.py](https://codecov.io/gh/apache/beam/pull/22722/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL2JhdGNoLnB5) | `90.38% <100.00%> (+1.99%)` | :arrow_up: |
| [sdks/python/apache\_beam/typehints/row\_type.py](https://codecov.io/gh/apache/beam/pull/22722/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHlwZWhpbnRzL3Jvd190eXBlLnB5) | `100.00% <100.00%> (+2.98%)` | :arrow_up: |
| ... and [10 more](https://codecov.io/gh/apache/beam/pull/22722/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem merged pull request #22722: [Playground] [Infrastructure] Deleting the Cloud Storage Client
Posted by GitBox <gi...@apache.org>.
pabloem merged PR #22722:
URL: https://github.com/apache/beam/pull/22722
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] vchunikhin commented on pull request #22722: [Playground] [Infrastructure] Deleting the Cloud Storage Client
Posted by GitBox <gi...@apache.org>.
vchunikhin commented on PR #22722:
URL: https://github.com/apache/beam/pull/22722#issuecomment-1233919954
> can you please rebase? thanks!
Sure, i've merged with master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #22722: [Playground] [Infrastructure] Deleting the Cloud Storage Client
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #22722:
URL: https://github.com/apache/beam/pull/22722#issuecomment-1219416340
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #22722: [Playground] [Infrastructure] Deleting the Cloud Storage Client
Posted by GitBox <gi...@apache.org>.
pabloem commented on PR #22722:
URL: https://github.com/apache/beam/pull/22722#issuecomment-1231971296
can you please rebase? thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #22722: [Playground] [Infrastructure] Deleting the Cloud Storage Client
Posted by GitBox <gi...@apache.org>.
pabloem commented on PR #22722:
URL: https://github.com/apache/beam/pull/22722#issuecomment-1235893283
Run SQL_Java17 PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] vchunikhin commented on pull request #22722: [Playground] [Infrastructure] Deleting the Cloud Storage Client
Posted by GitBox <gi...@apache.org>.
vchunikhin commented on PR #22722:
URL: https://github.com/apache/beam/pull/22722#issuecomment-1219415504
R: @pabloem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] vchunikhin commented on a diff in pull request #22722: [Playground] [Infrastructure] Deleting the Cloud Storage Client
Posted by GitBox <gi...@apache.org>.
vchunikhin commented on code in PR #22722:
URL: https://github.com/apache/beam/pull/22722#discussion_r949046669
##########
.github/workflows/playground_deploy_examples.yml:
##########
@@ -133,6 +133,8 @@ jobs:
K8S_SERVER_ADDRESS=$(kubectl get svc -n $K8S_NAMESPACE -l "app=service-python" -o jsonpath='{.items[0].status.loadBalancer.ingress[0].ip}')
BEAM_ROOT_DIR="../../"
BEAM_EXAMPLE_CATEGORIES="../../playground/categories.yaml"
+ GOOGLE_CLOUD_PROJECT_VALUE=${{ secrets.GCP_PLAYGROUND_PROJECT_ID }}
Review Comment:
This env variable is needed to connect to the Cloud Datastore
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] vchunikhin commented on a diff in pull request #22722: [Playground] [Infrastructure] Deleting the Cloud Storage Client
Posted by GitBox <gi...@apache.org>.
vchunikhin commented on code in PR #22722:
URL: https://github.com/apache/beam/pull/22722#discussion_r951265402
##########
playground/backend/internal/preparers/java_preparers.go:
##########
@@ -35,7 +36,7 @@ const (
javaPublicClassNamePattern = "public class (.*?) [{|implements(.*)]"
pipelineNamePattern = `Pipeline\s([A-z|0-9_]*)\s=\sPipeline\.create`
graphSavePattern = "String dotString = org.apache.beam.runners.core.construction.renderer.PipelineDotRenderer.toDotString(%s);\n" +
- " try (java.io.PrintWriter out = new java.io.PrintWriter(\"Graph.dot\")) {\n " +
+ " try (java.io.PrintWriter out = new java.io.PrintWriter(\"graph.dot\")) {\n " +
Review Comment:
This update is necessary to save graphs to cache correctly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] vchunikhin commented on a diff in pull request #22722: [Playground] [Infrastructure] Deleting the Cloud Storage Client
Posted by GitBox <gi...@apache.org>.
vchunikhin commented on code in PR #22722:
URL: https://github.com/apache/beam/pull/22722#discussion_r951564115
##########
playground/infrastructure/datastore_client.py:
##########
@@ -106,6 +110,35 @@ def save_to_cloud_datastore(self, examples_from_rep: List[Example], sdk: Sdk):
self._datastore_client.delete_multi(pc_objs_keys_for_removing)
logging.info("Finish of deleting extra playground examples ...")
+ def save_catalogs(self):
Review Comment:
This method is needed in order to load catalogs to the Cloud Datastore. It helps to decrease the coupling between python scripts and the router service
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #22722: [Playground] [Infrastructure] Deleting the Cloud Storage Client
Posted by GitBox <gi...@apache.org>.
pabloem commented on PR #22722:
URL: https://github.com/apache/beam/pull/22722#issuecomment-1235893020
Run SQL PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #22722: [Playground] [Infrastructure] Deleting the Cloud Storage Client
Posted by GitBox <gi...@apache.org>.
pabloem commented on PR #22722:
URL: https://github.com/apache/beam/pull/22722#issuecomment-1235892324
lgtm thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] vchunikhin commented on a diff in pull request #22722: [Playground] [Infrastructure] Deleting the Cloud Storage Client
Posted by GitBox <gi...@apache.org>.
vchunikhin commented on code in PR #22722:
URL: https://github.com/apache/beam/pull/22722#discussion_r949046669
##########
.github/workflows/playground_deploy_examples.yml:
##########
@@ -133,6 +133,8 @@ jobs:
K8S_SERVER_ADDRESS=$(kubectl get svc -n $K8S_NAMESPACE -l "app=service-python" -o jsonpath='{.items[0].status.loadBalancer.ingress[0].ip}')
BEAM_ROOT_DIR="../../"
BEAM_EXAMPLE_CATEGORIES="../../playground/categories.yaml"
+ GOOGLE_CLOUD_PROJECT_VALUE=${{ secrets.GCP_PLAYGROUND_PROJECT_ID }}
Review Comment:
This env variable needs to connect to the Cloud Datastore
##########
playground/backend/containers/go/Dockerfile:
##########
@@ -16,27 +16,28 @@
# limitations under the License.
###############################################################################
#Dokerfile to set up the Beam Go SDK
-ARG BASE_IMAGE=golang:1.16-bullseye
+ARG BASE_IMAGE=golang:1.18-bullseye
Review Comment:
Since BEAM version was updated to the latest version, we have to update our golang version
##########
learning/katas/java/Core Transforms/CoGroupByKey/CoGroupByKey/src/org/apache/beam/learning/katas/coretransforms/cogroupbykey/Task.java:
##########
@@ -19,7 +19,7 @@
package org.apache.beam.learning.katas.coretransforms.cogroupbykey;
// beam-playground:
-// name: GroupByKey
+// name: JoinPCollections
Review Comment:
We forbid examples with the same names for one SDK. An example name should be unique
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #22722: [Playground] [Infrastructure] Deleting the Cloud Storage Client
Posted by GitBox <gi...@apache.org>.
pabloem commented on PR #22722:
URL: https://github.com/apache/beam/pull/22722#issuecomment-1235892819
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #22722: [Playground] [Infrastructure] Deleting the Cloud Storage Client
Posted by GitBox <gi...@apache.org>.
pabloem commented on PR #22722:
URL: https://github.com/apache/beam/pull/22722#issuecomment-1235900767
lgtm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org