[GitHub] [beam] tomlynchRNA commented on a diff in pull request #25325: Remove from _KNOWN_TABLES on 404 insert to allow table re-creation for Python SDK

["tomlynchRNA (via GitHub)" <gi...@apache.org>]

tomlynchRNA commented on code in PR #25325:
URL: https://github.com/apache/beam/pull/25325#discussion_r1142942877


##########
sdks/python/apache_beam/io/gcp/bigquery.py:
##########
@@ -1551,17 +1552,30 @@ def _flush_batch(self, destination):
       insert_ids = [None for r in rows_and_insert_ids]
     else:
       insert_ids = [r[1] for r in rows_and_insert_ids]
-
     while True:
+      errors = []
+      passed = False
       start = time.time()
-      passed, errors = self.bigquery_wrapper.insert_rows(
-          project_id=table_reference.projectId,
-          dataset_id=table_reference.datasetId,
-          table_id=table_reference.tableId,
-          rows=rows,
-          insert_ids=insert_ids,
-          skip_invalid_rows=True,
-          ignore_unknown_values=self.ignore_unknown_columns)
+      try:
+        passed, errors = self.bigquery_wrapper.insert_rows(
+              project_id=table_reference.projectId,
+              dataset_id=table_reference.datasetId,
+              table_id=table_reference.tableId,
+              rows=rows,
+              insert_ids=insert_ids,
+              skip_invalid_rows=True,
+              ignore_unknown_values=self.ignore_unknown_columns)
+      except (ClientError, GoogleAPICallError) as e:
+        if e.code == 404 and destination in _KNOWN_TABLES:
+          _KNOWN_TABLES.remove(destination)

Review Comment:
   Can you clarify? I believe after raising, the bundle will retry, _create_table_if_needed is called again and will pass the check for the table name being in KNOWN_TABLES. Then the table may be created again (depending on create_disposition).
   
   I've also realised, if the create_disposition is 'CREATE_NEVER' and the insert_retry_strategy is set to 'RETRY_ALWAYS', the pipeline will be stuck in a loop, no?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org