You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "jackdingilian (via GitHub)" <gi...@apache.org> on 2023/09/08 13:50:03 UTC

[GitHub] [beam] jackdingilian opened a new pull request, #28374: Add mechanism to override bigtable client for testing

jackdingilian opened a new pull request, #28374:
URL: https://github.com/apache/beam/pull/28374

   
   Adds `BigtableClientOverride` interface to provide a way to override client settings we need to change for testing. 
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
   
   To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/workflows/Python%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/workflows/Java%20Tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Go tests](https://github.com/apache/beam/workflows/Go%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] jackdingilian commented on a diff in pull request #28374: Add mechanism to override bigtable client for testing

Posted by "jackdingilian (via GitHub)" <gi...@apache.org>.
jackdingilian commented on code in PR #28374:
URL: https://github.com/apache/beam/pull/28374#discussion_r1327496574


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableConfig.java:
##########
@@ -156,6 +162,12 @@ public BigtableConfig withEmulator(String emulatorHost) {
     return toBuilder().setEmulatorHost(emulatorHost).build();
   }
 
+  @VisibleForTesting
+  BigtableConfig withBigtableClientOverride(BigtableClientOverride clientOverride) {
+    checkArgument(clientOverride != null, "clientOverride can not be null");

Review Comment:
   good point, I guess this check isn't really necessary. I'll remove it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] kennknowles commented on a diff in pull request #28374: Add mechanism to override bigtable client for testing

Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on code in PR #28374:
URL: https://github.com/apache/beam/pull/28374#discussion_r1327473112


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableConfig.java:
##########
@@ -156,6 +162,12 @@ public BigtableConfig withEmulator(String emulatorHost) {
     return toBuilder().setEmulatorHost(emulatorHost).build();
   }
 
+  @VisibleForTesting
+  BigtableConfig withBigtableClientOverride(BigtableClientOverride clientOverride) {
+    checkArgument(clientOverride != null, "clientOverride can not be null");

Review Comment:
   If someone has set the override and you want to clear it, how do you do that?



##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java:
##########
@@ -1988,6 +1989,24 @@ public ReadChangeStream withMetadataTableAppProfileId(String appProfileId) {
           .build();
     }
 
+    /**
+     * Returns a new {@link BigtableIO.ReadChangeStream} that overrides the config of data and/or
+     * admin client for streaming changes and for managing the metadata. For testing purposes only.
+     * Not intended for use.

Review Comment:
   Maybe this is useful also for regional endpoints or some such? (no action required - it is certainly always easier to make things more visible than less visible)



##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java:
##########
@@ -1988,6 +1989,24 @@ public ReadChangeStream withMetadataTableAppProfileId(String appProfileId) {
           .build();
     }
 
+    /**
+     * Returns a new {@link BigtableIO.ReadChangeStream} that overrides the config of data and/or
+     * admin client for streaming changes and for managing the metadata. For testing purposes only.
+     * Not intended for use.
+     *
+     * <p>Does not modify this object.
+     */
+    @VisibleForTesting

Review Comment:
   It says visible for testing, but there are no tests. It seems like you can test this by setting up the `PipelineOptions` in the test before you expand the `BigTableChangeStreamsAccessor`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] tonytanger commented on pull request #28374: Add mechanism to override bigtable client for testing

Posted by "tonytanger (via GitHub)" <gi...@apache.org>.
tonytanger commented on PR #28374:
URL: https://github.com/apache/beam/pull/28374#issuecomment-1711772324

   LGTM


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] kennknowles commented on a diff in pull request #28374: Add mechanism to override bigtable client for testing

Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on code in PR #28374:
URL: https://github.com/apache/beam/pull/28374#discussion_r1327629142


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableConfig.java:
##########
@@ -156,6 +162,12 @@ public BigtableConfig withEmulator(String emulatorHost) {
     return toBuilder().setEmulatorHost(emulatorHost).build();
   }
 
+  @VisibleForTesting
+  BigtableConfig withBigtableClientOverride(BigtableClientOverride clientOverride) {
+    checkArgument(clientOverride != null, "clientOverride can not be null");

Review Comment:
   OK that seems fine. It is backwards-compatible to start to allow nulls as a way to re-clear it. One reason to allow nulls is to allow someone to make a copy without forcing them to `if (!= null) { set it }` but just be able to do straight-line `set to same value, whether or not it is null`



##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java:
##########
@@ -1988,6 +1989,24 @@ public ReadChangeStream withMetadataTableAppProfileId(String appProfileId) {
           .build();
     }
 
+    /**
+     * Returns a new {@link BigtableIO.ReadChangeStream} that overrides the config of data and/or
+     * admin client for streaming changes and for managing the metadata. For testing purposes only.
+     * Not intended for use.
+     *
+     * <p>Does not modify this object.
+     */
+    @VisibleForTesting

Review Comment:
   Oh I misread the whole change and thought there were already options for this (those do tend to be useful for redirecting a whole test at a different bigtable env or local endpoint). Nevermind for now!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] github-actions[bot] commented on pull request #28374: Add mechanism to override bigtable client for testing

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28374:
URL: https://github.com/apache/beam/pull/28374#issuecomment-1711817925

   Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
   
   R: @kennknowles for label java.
   R: @chamikaramj for label io.
   
   Available commands:
   - `stop reviewer notifications` - opt out of the automated review tooling
   - `remind me after tests pass` - tag the comment author after tests pass
   - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
   
   The PR bot will only process comments in the main thread (not review comments).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] jackdingilian commented on pull request #28374: Add mechanism to override bigtable client for testing

Posted by "jackdingilian (via GitHub)" <gi...@apache.org>.
jackdingilian commented on PR #28374:
URL: https://github.com/apache/beam/pull/28374#issuecomment-1711707141

   @tonytanger 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] jackdingilian commented on a diff in pull request #28374: Add mechanism to override bigtable client for testing

Posted by "jackdingilian (via GitHub)" <gi...@apache.org>.
jackdingilian commented on code in PR #28374:
URL: https://github.com/apache/beam/pull/28374#discussion_r1327512731


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableConfig.java:
##########
@@ -156,6 +162,12 @@ public BigtableConfig withEmulator(String emulatorHost) {
     return toBuilder().setEmulatorHost(emulatorHost).build();
   }
 
+  @VisibleForTesting
+  BigtableConfig withBigtableClientOverride(BigtableClientOverride clientOverride) {
+    checkArgument(clientOverride != null, "clientOverride can not be null");

Review Comment:
   on second thought, all of the other withParameter functions here have similar null checks that wouldn't allow overrides.
   
   I'm leaning towards keeping this to be consistent. I don't think there's a good reason to ever set this and then clear it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] kennknowles merged pull request #28374: Add mechanism to override bigtable client for testing

Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles merged PR #28374:
URL: https://github.com/apache/beam/pull/28374


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] jackdingilian commented on a diff in pull request #28374: Add mechanism to override bigtable client for testing

Posted by "jackdingilian (via GitHub)" <gi...@apache.org>.
jackdingilian commented on code in PR #28374:
URL: https://github.com/apache/beam/pull/28374#discussion_r1327498919


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java:
##########
@@ -1988,6 +1989,24 @@ public ReadChangeStream withMetadataTableAppProfileId(String appProfileId) {
           .build();
     }
 
+    /**
+     * Returns a new {@link BigtableIO.ReadChangeStream} that overrides the config of data and/or
+     * admin client for streaming changes and for managing the metadata. For testing purposes only.
+     * Not intended for use.
+     *
+     * <p>Does not modify this object.
+     */
+    @VisibleForTesting

Review Comment:
   IIUC this would require exposing all of the override parameters as PipelineOptions to users? We use this to connect to our internal testing environment. It requires fairly complicated client settings overrides that I don't think would ever be useful to expose to users



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org