You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "tonytanger (via GitHub)" <gi...@apache.org> on 2023/02/07 18:46:59 UTC

[GitHub] [beam] tonytanger opened a new pull request, #25364: Create metadata table in initialize stage if table doesn't exist.

tonytanger opened a new pull request, #25364:
URL: https://github.com/apache/beam/pull/25364

   Add logic to create metadata table if it doesn't exist. The metadata table is used to communicate between DetectNewPartitions and ReadChangeStreamPartition stages in the connector. The table includes information regarding currently streaming partitions as well as new splits and merges of partitions.
   
   This PR also includes locally built jar of Cloud Bigtable Java Client that includes the new Change Stream API. We are still waiting for the API to merge and make its way to the latest version of Google Cloud BOM. So, in parallel, we will incrementally review and submit changes to a feature branch. Once the beam repo is updated to include the Cloud Bigtable's new API, we can rebase and merge into master.
   
   All the jars in `*/bigtable_local/*` will be removed and dependency conflicts in `sdks/java/io/google-cloud-platform/build.gradle` will be resolved.
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
   
   To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/workflows/Python%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/workflows/Java%20Tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Go tests](https://github.com/apache/beam/workflows/Go%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] pabloem commented on pull request #25364: Create metadata table in initialize stage if table doesn't exist.

Posted by "pabloem (via GitHub)" <gi...@apache.org>.
pabloem commented on PR #25364:
URL: https://github.com/apache/beam/pull/25364#issuecomment-1425086450

   Run Java_GCP_IO_Direct PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tonytanger commented on a diff in pull request #25364: Create metadata table in initialize stage if table doesn't exist.

Posted by "tonytanger (via GitHub)" <gi...@apache.org>.
tonytanger commented on code in PR #25364:
URL: https://github.com/apache/beam/pull/25364#discussion_r1100757973


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/changestreams/dofn/InitializeDoFn.java:
##########
@@ -53,6 +52,25 @@ public void processElement(OutputReceiver<com.google.cloud.Timestamp> receiver)
     LOG.info(daoFactory.getStreamTableDebugString());
     LOG.info(daoFactory.getMetadataTableDebugString());
     LOG.info("ChangeStreamName: " + daoFactory.getChangeStreamName());
+    if (!daoFactory
+        .getMetadataTableAdminDao()
+        .isAppProfileSingleClusterAndTransactional(this.metadataTableAppProfileId)) {
+      LOG.error(
+          "App profile id '"
+              + metadataTableAppProfileId
+              + "' provided to access metadata table needs to use single-cluster routing policy"
+              + " and allow single-row transactions.");
+      // Terminate this pipeline now.
+      return;

Review Comment:
   That's a good suggestion. I think fail fast is a good idea. The biggest worry that I have is the pipeline submitter might not have the necessary permissions to interact with Cloud Bigtable. Whereas the service account attached to the dataflow workers will. 
   
   I think I will need some extra time to evaluate this approach. So for now, I think we should go ahead for now without checking in `expand`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] pabloem commented on pull request #25364: Create metadata table in initialize stage if table doesn't exist.

Posted by "pabloem (via GitHub)" <gi...@apache.org>.
pabloem commented on PR #25364:
URL: https://github.com/apache/beam/pull/25364#issuecomment-1426084029

   Run Java_GCP_IO_Direct PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on pull request #25364: Create metadata table in initialize stage if table doesn't exist.

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #25364:
URL: https://github.com/apache/beam/pull/25364#issuecomment-1424239709

   Test failure looks relevant:
   ```
   org.apache.beam.sdk.io.gcp.bigtable.BigtableReadIT.testE2EBigtableRead
   java.lang.NoClassDefFoundError: com/google/bigtable/admin/v2/BigtableInstanceAdminGrpc
   	at com.google.cloud.bigtable.grpc.BigtableInstanceGrpcClient.<init>(BigtableInstanceGrpcClient.java:71)
   	at com.google.cloud.bigtable.grpc.BigtableSession.<init>(BigtableSession.java:296)
   	at org.apache.beam.sdk.io.gcp.bigtable.BigtableServiceImpl.tableExists(BigtableServiceImpl.java:112)
   Caused by: java.lang.ClassNotFoundException: com.google.bigtable.admin.v2.BigtableInstanceAdminGrpc
   
   org.apache.beam.sdk.io.gcp.bigtable.BigtableWriteIT.testE2EBigtableWrite
   java.lang.NoClassDefFoundError: com/google/bigtable/admin/v2/BigtableInstanceAdminGrpc
   	at com.google.cloud.bigtable.grpc.BigtableInstanceGrpcClient.<init>(BigtableInstanceGrpcClient.java:71)
   	at com.google.cloud.bigtable.grpc.BigtableSession.<init>(BigtableSession.java:296)
   	at org.apache.beam.sdk.io.gcp.bigtable.BigtableWriteIT.setup(BigtableWriteIT.java:95)
   Caused by: java.lang.ClassNotFoundException: com.google.bigtable.admin.v2.BigtableInstanceAdminGrpc
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on a diff in pull request #25364: Create metadata table in initialize stage if table doesn't exist.

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #25364:
URL: https://github.com/apache/beam/pull/25364#discussion_r1102098408


##########
sdks/java/io/google-cloud-platform/build.gradle:
##########
@@ -27,6 +33,22 @@ applyJavaNature(
   ],
 )
 
+// For resolving Cloud Bigtable dependencies from custom registry that includes Change Stream API.
+repositories {
+  maven {
+    url "artifactregistry://us-central1-maven.pkg.dev/cloud-bigtable-ecosystem/bigtable-change-streams-preview"
+  }
+}
+configurations.all {
+  resolutionStrategy {
+    force group: 'com.google.cloud', name: 'google-cloud-bigtable', version: '2.11.2-change-streams-preview1-SNAPSHOT'

Review Comment:
   ah I see, the base branch is not master, sorry for confusion



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] pabloem merged pull request #25364: Create metadata table in initialize stage if table doesn't exist.

Posted by "pabloem (via GitHub)" <gi...@apache.org>.
pabloem merged PR #25364:
URL: https://github.com/apache/beam/pull/25364


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tonytanger commented on a diff in pull request #25364: Create metadata table in initialize stage if table doesn't exist.

Posted by "tonytanger (via GitHub)" <gi...@apache.org>.
tonytanger commented on code in PR #25364:
URL: https://github.com/apache/beam/pull/25364#discussion_r1102101840


##########
sdks/java/io/google-cloud-platform/build.gradle:
##########
@@ -27,6 +33,22 @@ applyJavaNature(
   ],
 )
 
+// For resolving Cloud Bigtable dependencies from custom registry that includes Change Stream API.
+repositories {
+  maven {
+    url "artifactregistry://us-central1-maven.pkg.dev/cloud-bigtable-ecosystem/bigtable-change-streams-preview"
+  }
+}
+configurations.all {
+  resolutionStrategy {
+    force group: 'com.google.cloud', name: 'google-cloud-bigtable', version: '2.11.2-change-streams-preview1-SNAPSHOT'

Review Comment:
   Also created https://github.com/apache/beam/issues/25412 to track this feature work.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] pabloem commented on pull request #25364: Create metadata table in initialize stage if table doesn't exist.

Posted by "pabloem (via GitHub)" <gi...@apache.org>.
pabloem commented on PR #25364:
URL: https://github.com/apache/beam/pull/25364#issuecomment-1425086122

   Run Java_GCP_IO_Direct PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tonytanger commented on a diff in pull request #25364: Create metadata table in initialize stage if table doesn't exist.

Posted by "tonytanger (via GitHub)" <gi...@apache.org>.
tonytanger commented on code in PR #25364:
URL: https://github.com/apache/beam/pull/25364#discussion_r1102103343


##########
sdks/java/io/google-cloud-platform/build.gradle:
##########
@@ -18,7 +18,13 @@
 
 import groovy.json.JsonOutput
 
-plugins { id 'org.apache.beam.module' }
+plugins {
+  id 'org.apache.beam.module'
+  // For resolving Cloud Bigtable dependencies from custom registry that includes Change Stream API.
+  id "maven-publish"

Review Comment:
   The complication, and I couldn't figure this out, is the URL is `artifactregistry` which requires `com.google.cloud.artifactregistry.gradle-plugin` plugin. It's not obvious how to include the plugin in Repositories.groovy.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tonytanger commented on pull request #25364: Create metadata table in initialize stage if table doesn't exist.

Posted by "tonytanger (via GitHub)" <gi...@apache.org>.
tonytanger commented on PR #25364:
URL: https://github.com/apache/beam/pull/25364#issuecomment-1424750914

   Run Java PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] pabloem commented on pull request #25364: Create metadata table in initialize stage if table doesn't exist.

Posted by "pabloem (via GitHub)" <gi...@apache.org>.
pabloem commented on PR #25364:
URL: https://github.com/apache/beam/pull/25364#issuecomment-1425086215

   Run Portable_Python PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] pabloem commented on a diff in pull request #25364: Create metadata table in initialize stage if table doesn't exist.

Posted by "pabloem (via GitHub)" <gi...@apache.org>.
pabloem commented on code in PR #25364:
URL: https://github.com/apache/beam/pull/25364#discussion_r1100679077


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/changestreams/dofn/InitializeDoFn.java:
##########
@@ -53,6 +52,25 @@ public void processElement(OutputReceiver<com.google.cloud.Timestamp> receiver)
     LOG.info(daoFactory.getStreamTableDebugString());
     LOG.info(daoFactory.getMetadataTableDebugString());
     LOG.info("ChangeStreamName: " + daoFactory.getChangeStreamName());
+    if (!daoFactory
+        .getMetadataTableAdminDao()
+        .isAppProfileSingleClusterAndTransactional(this.metadataTableAppProfileId)) {
+      LOG.error(
+          "App profile id '"
+              + metadataTableAppProfileId
+              + "' provided to access metadata table needs to use single-cluster routing policy"
+              + " and allow single-row transactions.");
+      // Terminate this pipeline now.
+      return;

Review Comment:
   I suggest performing this check in the `expand` method - this would allow you to fail the pipeline submission and have an error thrown.
   
   When you perform the check at this point in the pipeline, the pipeline will terminate, but it will look 'normal' - users will need to notice the ERROR log.
   
   We can leave this check here, but I think you should make sure to perform the same check in the `expand` method so the pipeline submission fails with an error being thrown (rather than just logged)



##########
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/GcpApiSurfaceTest.java:
##########
@@ -124,9 +130,7 @@ public void testGcpApiSurface() throws Exception {
             classesInPackage("org.apache.commons.logging"),
             classesInPackage("org.codehaus.jackson"),
             classesInPackage("org.joda.time"),
-            Matchers.<Class<?>>equalTo(org.threeten.bp.Duration.class),
-            Matchers.<Class<?>>equalTo(org.threeten.bp.format.ResolverStyle.class),
-            classesInPackage("org.threeten.bp.temporal"),

Review Comment:
   for my edification - what is the reason for the removals here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] pabloem closed pull request #25364: Create metadata table in initialize stage if table doesn't exist.

Posted by "pabloem (via GitHub)" <gi...@apache.org>.
pabloem closed pull request #25364: Create metadata table in initialize stage if table doesn't exist.
URL: https://github.com/apache/beam/pull/25364


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tonytanger commented on pull request #25364: Create metadata table in initialize stage if table doesn't exist.

Posted by "tonytanger (via GitHub)" <gi...@apache.org>.
tonytanger commented on PR #25364:
URL: https://github.com/apache/beam/pull/25364#issuecomment-1426212087

   Run PythonLint PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tonytanger commented on pull request #25364: Create metadata table in initialize stage if table doesn't exist.

Posted by "tonytanger (via GitHub)" <gi...@apache.org>.
tonytanger commented on PR #25364:
URL: https://github.com/apache/beam/pull/25364#issuecomment-1426428935

   Run Java PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on a diff in pull request #25364: Create metadata table in initialize stage if table doesn't exist.

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #25364:
URL: https://github.com/apache/beam/pull/25364#discussion_r1102080972


##########
sdks/java/io/google-cloud-platform/build.gradle:
##########
@@ -18,7 +18,13 @@
 
 import groovy.json.JsonOutput
 
-plugins { id 'org.apache.beam.module' }
+plugins {
+  id 'org.apache.beam.module'
+  // For resolving Cloud Bigtable dependencies from custom registry that includes Change Stream API.
+  id "maven-publish"

Review Comment:
   maven-publish is handled in BeamModulePlugin and to my understanding should not be added here.
   
   move the below 
   ```
   repositories {
     maven {
   ```
   clause to https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/Repositories.groovy is the right approach



##########
sdks/java/io/google-cloud-platform/build.gradle:
##########
@@ -27,6 +33,22 @@ applyJavaNature(
   ],
 )
 
+// For resolving Cloud Bigtable dependencies from custom registry that includes Change Stream API.
+repositories {
+  maven {
+    url "artifactregistry://us-central1-maven.pkg.dev/cloud-bigtable-ecosystem/bigtable-change-streams-preview"
+  }
+}
+configurations.all {
+  resolutionStrategy {
+    force group: 'com.google.cloud', name: 'google-cloud-bigtable', version: '2.11.2-change-streams-preview1-SNAPSHOT'

Review Comment:
   since these are all SNAPSHOT versions, they should not appear in beam release versions. What is the time line that it will get released? If is within a month or so, could you please create a GitHub issue and set milestone to v2.46.0 as a release blocker?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tonytanger commented on pull request #25364: Create metadata table in initialize stage if table doesn't exist.

Posted by "tonytanger (via GitHub)" <gi...@apache.org>.
tonytanger commented on PR #25364:
URL: https://github.com/apache/beam/pull/25364#issuecomment-1424889547

   Resolved the class not found error. It was caused by all the messing around with dependencies that will be removed anyways once the Change Stream API merges into the Bigtable client. 
   
   The rest of the test failures seem unrelated to my changes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] pabloem commented on pull request #25364: Create metadata table in initialize stage if table doesn't exist.

Posted by "pabloem (via GitHub)" <gi...@apache.org>.
pabloem commented on PR #25364:
URL: https://github.com/apache/beam/pull/25364#issuecomment-1423343636

   Run Java PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] pabloem commented on pull request #25364: Create metadata table in initialize stage if table doesn't exist.

Posted by "pabloem (via GitHub)" <gi...@apache.org>.
pabloem commented on PR #25364:
URL: https://github.com/apache/beam/pull/25364#issuecomment-1425086276

   Run Python_Coverage PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tonytanger commented on pull request #25364: Create metadata table in initialize stage if table doesn't exist.

Posted by "tonytanger (via GitHub)" <gi...@apache.org>.
tonytanger commented on PR #25364:
URL: https://github.com/apache/beam/pull/25364#issuecomment-1426212004

   Run Java_MongoDb_IO_Direct PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] commented on pull request #25364: Create metadata table in initialize stage if table doesn't exist.

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #25364:
URL: https://github.com/apache/beam/pull/25364#issuecomment-1421314747

   # [Codecov](https://codecov.io/gh/apache/beam/pull/25364?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > :exclamation: No coverage uploaded for pull request base (`bigtable-cdc-feature-branch@67b5b36`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#section-missing-base-commit).
   > The diff coverage is `n/a`.
   
   ```diff
   @@                      Coverage Diff                       @@
   ##             bigtable-cdc-feature-branch   #25364   +/-   ##
   ==============================================================
     Coverage                               ?   73.04%           
   ==============================================================
     Files                                  ?      742           
     Lines                                  ?    98910           
     Branches                               ?        0           
   ==============================================================
     Hits                                   ?    72251           
     Misses                                 ?    25296           
     Partials                               ?     1363           
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | python | `82.45% <0.00%> (?)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] pabloem commented on pull request #25364: Create metadata table in initialize stage if table doesn't exist.

Posted by "pabloem (via GitHub)" <gi...@apache.org>.
pabloem commented on PR #25364:
URL: https://github.com/apache/beam/pull/25364#issuecomment-1423338476

   Run Java_GCP_IO_Direct PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] Abacn commented on a diff in pull request #25364: Create metadata table in initialize stage if table doesn't exist.

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #25364:
URL: https://github.com/apache/beam/pull/25364#discussion_r1102084742


##########
sdks/java/io/google-cloud-platform/build.gradle:
##########
@@ -27,6 +33,22 @@ applyJavaNature(
   ],
 )
 
+// For resolving Cloud Bigtable dependencies from custom registry that includes Change Stream API.
+repositories {
+  maven {
+    url "artifactregistry://us-central1-maven.pkg.dev/cloud-bigtable-ecosystem/bigtable-change-streams-preview"
+  }
+}
+configurations.all {
+  resolutionStrategy {
+    force group: 'com.google.cloud', name: 'google-cloud-bigtable', version: '2.11.2-change-streams-preview1-SNAPSHOT'

Review Comment:
   since these are all SNAPSHOT versions, they should not appear in beam release versions. What is the time line that it will get released? If is within a month or so, could you please create a GitHub issue and set milestone to v2.46.0 as a release blocker? Then add a TODO comment here and in repository.groovy



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tonytanger commented on a diff in pull request #25364: Create metadata table in initialize stage if table doesn't exist.

Posted by "tonytanger (via GitHub)" <gi...@apache.org>.
tonytanger commented on code in PR #25364:
URL: https://github.com/apache/beam/pull/25364#discussion_r1102090605


##########
sdks/java/io/google-cloud-platform/build.gradle:
##########
@@ -27,6 +33,22 @@ applyJavaNature(
   ],
 )
 
+// For resolving Cloud Bigtable dependencies from custom registry that includes Change Stream API.
+repositories {
+  maven {
+    url "artifactregistry://us-central1-maven.pkg.dev/cloud-bigtable-ecosystem/bigtable-change-streams-preview"
+  }
+}
+configurations.all {
+  resolutionStrategy {
+    force group: 'com.google.cloud', name: 'google-cloud-bigtable', version: '2.11.2-change-streams-preview1-SNAPSHOT'

Review Comment:
   This is merging into a feature branch. None of this will be included in when merging into master branch. This is a temporary work around to get code logic reviewed/compile/pass tests while we wait for Cloud Bigtable's java client to catch up to include the new API.
   
   We will remove all this before creating the PR to merge into master. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] pabloem commented on pull request #25364: Create metadata table in initialize stage if table doesn't exist.

Posted by "pabloem (via GitHub)" <gi...@apache.org>.
pabloem commented on PR #25364:
URL: https://github.com/apache/beam/pull/25364#issuecomment-1426535634

   Run Java PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tonytanger commented on a diff in pull request #25364: Create metadata table in initialize stage if table doesn't exist.

Posted by "tonytanger (via GitHub)" <gi...@apache.org>.
tonytanger commented on code in PR #25364:
URL: https://github.com/apache/beam/pull/25364#discussion_r1100685551


##########
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/GcpApiSurfaceTest.java:
##########
@@ -124,9 +130,7 @@ public void testGcpApiSurface() throws Exception {
             classesInPackage("org.apache.commons.logging"),
             classesInPackage("org.codehaus.jackson"),
             classesInPackage("org.joda.time"),
-            Matchers.<Class<?>>equalTo(org.threeten.bp.Duration.class),
-            Matchers.<Class<?>>equalTo(org.threeten.bp.format.ResolverStyle.class),
-            classesInPackage("org.threeten.bp.temporal"),

Review Comment:
   I add the entire `org.threeten.bp` which covers these more specific matchers. The bigtable data client uses many of the classes in `org.threeten.bp`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] pabloem commented on pull request #25364: Create metadata table in initialize stage if table doesn't exist.

Posted by "pabloem (via GitHub)" <gi...@apache.org>.
pabloem commented on PR #25364:
URL: https://github.com/apache/beam/pull/25364#issuecomment-1423632530

   Run Java PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tonytanger commented on pull request #25364: Create metadata table in initialize stage if table doesn't exist.

Posted by "tonytanger (via GitHub)" <gi...@apache.org>.
tonytanger commented on PR #25364:
URL: https://github.com/apache/beam/pull/25364#issuecomment-1421583142

   @pabloem 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] github-actions[bot] commented on pull request #25364: Create metadata table in initialize stage if table doesn't exist.

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25364:
URL: https://github.com/apache/beam/pull/25364#issuecomment-1421669543

   Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
   
   R: @robertwb for label java.
   R: @Abacn for label build.
   R: @pabloem for label io.
   
   Available commands:
   - `stop reviewer notifications` - opt out of the automated review tooling
   - `remind me after tests pass` - tag the comment author after tests pass
   - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
   
   The PR bot will only process comments in the main thread (not review comments).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org