You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@beam.apache.org by "Chamikara Jayalath (JIRA)" <ji...@apache.org> on 2018/08/24 21:57:00 UTC

[jira] [Created] (BEAM-5216) BigQueryIO multi-partitioned write doesn't work for streaming writes

Chamikara Jayalath created BEAM-5216:
----------------------------------------

             Summary: BigQueryIO multi-partitioned write doesn't work for streaming writes
                 Key: BEAM-5216
                 URL: https://issues.apache.org/jira/browse/BEAM-5216
             Project: Beam
          Issue Type: Bug
          Components: io-java-gcp
            Reporter: Chamikara Jayalath


BigQueryIO performes multi-partitioned write (MultiPartitionsWriteTables step) when there's more data than the quota allowed by BigQuery (10k files or 11TB of data) to be written to a single BQ table.

 

When writing using load jobs in streaming mode (with a triggering frequency) we hit following location where we set CREATE_DISPOSITION to CREATE_NEVER for all panes other than the first one. This is fine when we are writing a single partition (all panes of a window should write to the same table) but when there are multiple partitions this is incorrect since we need to create temp tables for all panes.

[https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java#L165]

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)