[GitHub] [beam] pabloem commented on a diff in pull request #25364: Create metadata table in initialize stage if table doesn't exist.

["pabloem (via GitHub)" <gi...@apache.org>]

pabloem commented on code in PR #25364:
URL: https://github.com/apache/beam/pull/25364#discussion_r1100679077


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/changestreams/dofn/InitializeDoFn.java:
##########
@@ -53,6 +52,25 @@ public void processElement(OutputReceiver<com.google.cloud.Timestamp> receiver)
     LOG.info(daoFactory.getStreamTableDebugString());
     LOG.info(daoFactory.getMetadataTableDebugString());
     LOG.info("ChangeStreamName: " + daoFactory.getChangeStreamName());
+    if (!daoFactory
+        .getMetadataTableAdminDao()
+        .isAppProfileSingleClusterAndTransactional(this.metadataTableAppProfileId)) {
+      LOG.error(
+          "App profile id '"
+              + metadataTableAppProfileId
+              + "' provided to access metadata table needs to use single-cluster routing policy"
+              + " and allow single-row transactions.");
+      // Terminate this pipeline now.
+      return;

Review Comment:
   I suggest performing this check in the `expand` method - this would allow you to fail the pipeline submission and have an error thrown.
   
   When you perform the check at this point in the pipeline, the pipeline will terminate, but it will look 'normal' - users will need to notice the ERROR log.
   
   We can leave this check here, but I think you should make sure to perform the same check in the `expand` method so the pipeline submission fails with an error being thrown (rather than just logged)



##########
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/GcpApiSurfaceTest.java:
##########
@@ -124,9 +130,7 @@ public void testGcpApiSurface() throws Exception {
             classesInPackage("org.apache.commons.logging"),
             classesInPackage("org.codehaus.jackson"),
             classesInPackage("org.joda.time"),
-            Matchers.<Class<?>>equalTo(org.threeten.bp.Duration.class),
-            Matchers.<Class<?>>equalTo(org.threeten.bp.format.ResolverStyle.class),
-            classesInPackage("org.threeten.bp.temporal"),

Review Comment:
   for my edification - what is the reason for the removals here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org