[GitHub] [beam] damccorm opened a new issue, #21168: WriteTables leaves behind temporary tables on job failure

[GitBox <gi...@apache.org>]

damccorm opened a new issue, #21168:
URL: https://github.com/apache/beam/issues/21168

   I'm running a job that writes to a BigQuery table using `BigQueryIO.writeTableRows().to(
   new SerializableFunction<ValueInSingleWindow<TableRow\>, TableDestination\>)`.
    
   I'm noticing that when my job fails, it leaves behind temporary tables (`beam_bq_job_LOAD_*`) in the destination dataset. These tables are created by load jobs started here:
    
   [https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java#L273-L284)](https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java#L273-L284),)
    
   I'd like to specify a temporary dataset for these load job result tables, but I don't see a way to specify one using the Java SDK. It seems like the load job destination is inferred by changing the table id of the final destination:
    
   [https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java#L255](https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java#L255)
    
   which makes me think that the configuration I want to set doesn't exist. Is there a workaround to avoid having these tables be left around when the job fails? Could the option be added?
   
   Imported from Jira [BEAM-12986](https://issues.apache.org/jira/browse/BEAM-12986). Original Jira may contain additional context.
   Reported by: Wyszynski.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org