[GitHub] [beam] ahmedabu98 commented on a diff in pull request #24041: Add random string at the end of BigQuery query job name to make it resilient to retries

[GitBox <gi...@apache.org>]

ahmedabu98 commented on code in PR #24041:
URL: https://github.com/apache/beam/pull/24041#discussion_r1018304092


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryQueryHelper.java:
##########
@@ -119,16 +119,21 @@ public static TableReference executeQuery(
       }
 
       // Step 2: Create a temporary dataset in the query location only if the user has not specified
-      // a temp dataset.
-      String queryJobId =
+      // a temp dataset. The temp table name may be deterministic but the query job ID has to be
+      // non-deterministic to protect against retries. If BigQuery sees a repeated query job ID, it
+      // will be skipped.
+      String tempTableID =
           BigQueryResourceNaming.createJobIdPrefix(options.getJobName(), stepUuid, JobType.QUERY);
+      String queryJobId =
+          BigQueryResourceNaming.createJobIdPrefix(
+              options.getJobName(), stepUuid, JobType.QUERY, BigQueryHelpers.randomUUIDString());
       Optional<String> queryTempDatasetOpt = Optional.ofNullable(queryTempDatasetId);
       TableReference queryResultTable =
           createTempTableReference(
               options.getBigQueryProject() == null
                   ? options.getProject()
                   : options.getBigQueryProject(),
-              queryJobId,
+              tempTableID,

Review Comment:
   The table name can stay deterministic, and probably it should because other parts of BigQuery connector recreate the temp table using just the stepUuid, e.g. for table deletion [here (export read)](https://github.com/apache/beam/blob/539fa9159ffc116b2e79e6de2804dfdd1c1e4722/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryQuerySourceDef.java#L131-L138) and [here (direct read)](https://github.com/apache/beam/blob/539fa9159ffc116b2e79e6de2804dfdd1c1e4722/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L1554-L1561).
   
   The problem is when the query job reference is retried with the same name, so having a non-deterministic name [here](https://github.com/apache/beam/blob/539fa9159ffc116b2e79e6de2804dfdd1c1e4722/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryQueryHelper.java#L166-L173) is the goal.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org