[GitHub] [beam] chamikaramj commented on a diff in pull request #24041: Add random string at the end of BigQuery query job name to make it resilient to retries

[GitBox <gi...@apache.org>]

chamikaramj commented on code in PR #24041:
URL: https://github.com/apache/beam/pull/24041#discussion_r1018290203


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryQueryHelper.java:
##########
@@ -119,16 +119,21 @@ public static TableReference executeQuery(
       }
 
       // Step 2: Create a temporary dataset in the query location only if the user has not specified
-      // a temp dataset.
-      String queryJobId =
+      // a temp dataset. The temp table name may be deterministic but the query job ID has to be
+      // non-deterministic to protect against retries. If BigQuery sees a repeated query job ID, it
+      // will be skipped.
+      String tempTableID =
           BigQueryResourceNaming.createJobIdPrefix(options.getJobName(), stepUuid, JobType.QUERY);
+      String queryJobId =
+          BigQueryResourceNaming.createJobIdPrefix(
+              options.getJobName(), stepUuid, JobType.QUERY, BigQueryHelpers.randomUUIDString());
       Optional<String> queryTempDatasetOpt = Optional.ofNullable(queryTempDatasetId);
       TableReference queryResultTable =
           createTempTableReference(
               options.getBigQueryProject() == null
                   ? options.getProject()
                   : options.getBigQueryProject(),
-              queryJobId,
+              tempTableID,

Review Comment:
   Should we be passing "queryJobId" here instead ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org