You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "proost (via GitHub)" <gi...@apache.org> on 2024/09/15 15:11:43 UTC

[PR] feat: add topic existing validation [beam]

proost opened a new pull request, #32465:
URL: https://github.com/apache/beam/pull/32465

   **Please** add a meaningful description for your change here
   fix: https://github.com/apache/beam/issues/18027
   
   adding validation to check topic exist or not for PubsubIO
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [X] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
   
   To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/workflows/Python%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/workflows/Java%20Tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Go tests](https://github.com/apache/beam/workflows/Go%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: add topic existing validation [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32465:
URL: https://github.com/apache/beam/pull/32465#issuecomment-2351668755

   Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
   
   R: @m-trieu for label java.
   R: @ahmedabu98 for label io.
   
   Available commands:
   - `stop reviewer notifications` - opt out of the automated review tooling
   - `remind me after tests pass` - tag the comment author after tests pass
   - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
   
   The PR bot will only process comments in the main thread (not review comments).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: add topic existing validation [beam]

Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 merged PR #32465:
URL: https://github.com/apache/beam/pull/32465


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: add topic existing validation [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #32465:
URL: https://github.com/apache/beam/pull/32465#issuecomment-2377659324

   Well, I am going to revert the change for now, as introducing a required field in IO connector specification would generally breaks upgrade compatibility of pipelines (unless tested not to be the case).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: add topic existing validation [beam]

Posted by "proost (via GitHub)" <gi...@apache.org>.
proost commented on code in PR #32465:
URL: https://github.com/apache/beam/pull/32465#discussion_r1776461123


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubGrpcClient.java:
##########
@@ -372,6 +373,21 @@ public List<TopicPath> listTopics(ProjectPath project) throws IOException {
     return topics;
   }
 
+  @Override
+  public boolean isTopicExists(TopicPath topic) throws IOException {
+    GetTopicRequest request = GetTopicRequest.newBuilder().setTopic(topic.getPath()).build();
+    try {
+      publisherStub().getTopic(request);
+      return true;
+    } catch (StatusRuntimeException e) {
+      if (e.getStatus().getCode() == io.grpc.Status.Code.NOT_FOUND) {
+        return false;
+      }
+
+      throw e;

Review Comment:
   @Abacn 
   Oh, I missed that point. How about disabled as default? Checking topic existence is needed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: add topic existing validation [beam]

Posted by "proost (via GitHub)" <gi...@apache.org>.
proost commented on PR #32465:
URL: https://github.com/apache/beam/pull/32465#issuecomment-2351963887

   @ahmedabu98 
   I update "CHANGES.md".
   
   can you review again? because i notice that `Write` doesn't validate topic. when topic is given statically, check topic is valid or not.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: add topic existing validation [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #32465:
URL: https://github.com/apache/beam/pull/32465#issuecomment-2375536733

   This is breaking Java presubmit:
   
   ```
   testPubsubSinkOverride (org.apache.beam.runners.dataflow.DataflowRunnerTest) failed
   
   java.lang.RuntimeException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
   GET https://pubsub.googleapis.com/v1/projects/project/topics/topic
   {
     "code": 403,
     "errors": [
       {
         "domain": "global",
         "message": "Method doesn't allow unregistered callers (callers without established identity). Please use API Key or other form of API consumer identity to call this API.",
         "reason": "forbidden"
       }
     ],
     "message": "Method doesn't allow unregistered callers (callers without established identity). Please use API Key or other form of API consumer identity to call this API.",
     "status": "PERMISSION_DENIED"
   }
   	at org.apache.beam.sdk.io.gcp.pubsub.PubsubIO$Write.validate(PubsubIO.java:1658)
   ```
   
   #18027 was a very old Issue. Is it still needed here?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: add topic existing validation [beam]

Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #32465:
URL: https://github.com/apache/beam/pull/32465#discussion_r1760874735


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java:
##########
@@ -1390,6 +1392,7 @@ static <T> Builder<T> newBuilder(
       builder.setFormatFn(formatFn);
       builder.setBadRecordRouter(BadRecordRouter.THROWING_ROUTER);
       builder.setBadRecordErrorHandler(new DefaultErrorHandler<>());
+      builder.setValidate(true);

Review Comment:
   This is the entry method for all writes, so it's enough to `setValidate(true)` here and remove it from the other methods.
   
   It's better to only set it here, otherwise validation will be set to true even when someone doesn't want it. Take the following for example:
   
   ```java
   PubsubIO
     .writeStrings() // validate = true
     .withoutValidation() // validate = false
     .to("topic"); // validate = true
    ```



##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java:
##########
@@ -872,6 +875,7 @@ static <T> Builder<T> newBuilder(SerializableFunction<PubsubMessage, T> parseFn)
       builder.setNeedsOrderingKey(false);
       builder.setBadRecordRouter(BadRecordRouter.THROWING_ROUTER);
       builder.setBadRecordErrorHandler(new DefaultErrorHandler<>());
+      builder.setValidate(true);

Review Comment:
   Similar to the above comment, can we only `setValidate(true)` here and remove it from the other methods?



##########
CHANGES.md:
##########
@@ -62,7 +62,7 @@
 
 ## I/Os
 
-* Support for X source added (Java/Python) ([#X](https://github.com/apache/beam/issues/X)).
+* Check Pub/Sub topic is existing before Read/Write (Java) ([#32465](https://github.com/apache/beam/pull/32465))

Review Comment:
   ```suggestion
   * PubsubIO will validate that the Pub/Sub topic exists before running the Read/Write pipeline (Java) ([#32465](https://github.com/apache/beam/pull/32465))
   ```



##########
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIOTest.java:
##########
@@ -997,4 +997,114 @@ public void testWithoutValidation() throws IOException {
 
     read.validate(options);
   }
+
+  @Test
+  public void testWriteTopicValidationSuccess() throws Exception {
+    PubsubIO.writeStrings().to("projects/my-project/topics/abc");
+    PubsubIO.writeStrings().to("projects/my-project/topics/ABC");
+    PubsubIO.writeStrings().to("projects/my-project/topics/AbC-DeF");
+    PubsubIO.writeStrings().to("projects/my-project/topics/AbC-1234");
+    PubsubIO.writeStrings().to("projects/my-project/topics/AbC-1234-_.~%+-_.~%+-_.~%+-abc");
+    PubsubIO.writeStrings()
+        .to(
+            new StringBuilder()
+                .append("projects/my-project/topics/A-really-long-one-")
+                .append(
+                    "111111111111111111111111111111111111111111111111111111111111111111111111111111111")

Review Comment:
   nit: for these long strings, can we use something like `RandomStringUtils.randomAlphanumeric()` instead? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: add topic existing validation [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #32465:
URL: https://github.com/apache/beam/pull/32465#discussion_r1777361325


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubGrpcClient.java:
##########
@@ -372,6 +373,21 @@ public List<TopicPath> listTopics(ProjectPath project) throws IOException {
     return topics;
   }
 
+  @Override
+  public boolean isTopicExists(TopicPath topic) throws IOException {
+    GetTopicRequest request = GetTopicRequest.newBuilder().setTopic(topic.getPath()).build();
+    try {
+      publisherStub().getTopic(request);
+      return true;
+    } catch (StatusRuntimeException e) {
+      if (e.getStatus().getCode() == io.grpc.Status.Code.NOT_FOUND) {
+        return false;
+      }
+
+      throw e;

Review Comment:
   How about disabled as default?
   
   sounds good



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: add topic existing validation [beam]

Posted by "proost (via GitHub)" <gi...@apache.org>.
proost commented on code in PR #32465:
URL: https://github.com/apache/beam/pull/32465#discussion_r1761072557


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java:
##########
@@ -1390,6 +1392,7 @@ static <T> Builder<T> newBuilder(
       builder.setFormatFn(formatFn);
       builder.setBadRecordRouter(BadRecordRouter.THROWING_ROUTER);
       builder.setBadRecordErrorHandler(new DefaultErrorHandler<>());
+      builder.setValidate(true);

Review Comment:
   I agree. 
   [24859f8](https://github.com/apache/beam/pull/32465/commits/24859f869d28d9ae3efbcbf803c249981725725a)



##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java:
##########
@@ -872,6 +875,7 @@ static <T> Builder<T> newBuilder(SerializableFunction<PubsubMessage, T> parseFn)
       builder.setNeedsOrderingKey(false);
       builder.setBadRecordRouter(BadRecordRouter.THROWING_ROUTER);
       builder.setBadRecordErrorHandler(new DefaultErrorHandler<>());
+      builder.setValidate(true);

Review Comment:
   [24859f8](https://github.com/apache/beam/pull/32465/commits/24859f869d28d9ae3efbcbf803c249981725725a)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: add topic existing validation [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32465:
URL: https://github.com/apache/beam/pull/32465#issuecomment-2371100699

   Reminder, please take a look at this pr: @m-trieu @ahmedabu98 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: add topic existing validation [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #32465:
URL: https://github.com/apache/beam/pull/32465#discussion_r1776163515


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubGrpcClient.java:
##########
@@ -372,6 +373,21 @@ public List<TopicPath> listTopics(ProjectPath project) throws IOException {
     return topics;
   }
 
+  @Override
+  public boolean isTopicExists(TopicPath topic) throws IOException {
+    GetTopicRequest request = GetTopicRequest.newBuilder().setTopic(topic.getPath()).build();
+    try {
+      publisherStub().getTopic(request);
+      return true;
+    } catch (StatusRuntimeException e) {
+      if (e.getStatus().getCode() == io.grpc.Status.Code.NOT_FOUND) {
+        return false;
+      }
+
+      throw e;

Review Comment:
   This is a breaking change. Pipeline construction environment may not have access to the PubSub. Here it should either be fail safe or not enabled by default.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: add topic existing validation [beam]

Posted by "proost (via GitHub)" <gi...@apache.org>.
proost commented on code in PR #32465:
URL: https://github.com/apache/beam/pull/32465#discussion_r1761071799


##########
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIOTest.java:
##########
@@ -997,4 +997,114 @@ public void testWithoutValidation() throws IOException {
 
     read.validate(options);
   }
+
+  @Test
+  public void testWriteTopicValidationSuccess() throws Exception {
+    PubsubIO.writeStrings().to("projects/my-project/topics/abc");
+    PubsubIO.writeStrings().to("projects/my-project/topics/ABC");
+    PubsubIO.writeStrings().to("projects/my-project/topics/AbC-DeF");
+    PubsubIO.writeStrings().to("projects/my-project/topics/AbC-1234");
+    PubsubIO.writeStrings().to("projects/my-project/topics/AbC-1234-_.~%+-_.~%+-_.~%+-abc");
+    PubsubIO.writeStrings()
+        .to(
+            new StringBuilder()
+                .append("projects/my-project/topics/A-really-long-one-")
+                .append(
+                    "111111111111111111111111111111111111111111111111111111111111111111111111111111111")

Review Comment:
   [a09e32e](https://github.com/apache/beam/pull/32465/commits/a09e32e504cfd6db6c3d4e7d4447d6bfe9955216)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: add topic existing validation [beam]

Posted by "proost (via GitHub)" <gi...@apache.org>.
proost commented on code in PR #32465:
URL: https://github.com/apache/beam/pull/32465#discussion_r1761072988


##########
CHANGES.md:
##########
@@ -62,7 +62,7 @@
 
 ## I/Os
 
-* Support for X source added (Java/Python) ([#X](https://github.com/apache/beam/issues/X)).
+* Check Pub/Sub topic is existing before Read/Write (Java) ([#32465](https://github.com/apache/beam/pull/32465))

Review Comment:
   [c7e89b9](https://github.com/apache/beam/pull/32465/commits/c7e89b93f75016920b9b6f87730f3fb7d8b42327)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] fix: add topic existing validation [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #32465:
URL: https://github.com/apache/beam/pull/32465#discussion_r1776163515


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubGrpcClient.java:
##########
@@ -372,6 +373,21 @@ public List<TopicPath> listTopics(ProjectPath project) throws IOException {
     return topics;
   }
 
+  @Override
+  public boolean isTopicExists(TopicPath topic) throws IOException {
+    GetTopicRequest request = GetTopicRequest.newBuilder().setTopic(topic.getPath()).build();
+    try {
+      publisherStub().getTopic(request);
+      return true;
+    } catch (StatusRuntimeException e) {
+      if (e.getStatus().getCode() == io.grpc.Status.Code.NOT_FOUND) {
+        return false;
+      }
+
+      throw e;

Review Comment:
   This is a breaking change. Pipeline construction environment may not have access to the PubSub. Here it should either be fail safe or not enabled at default.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org