[jira] [Work logged] (BEAM-3200) Streaming Pipeline throws RuntimeException when using DynamicDestinations and Method.FILE_LOADS

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BEAM-3200?focusedWorklogId=312306&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-312306 ]

ASF GitHub Bot logged work on BEAM-3200:
----------------------------------------

                Author: ASF GitHub Bot
            Created on: 13/Sep/19 20:32
            Start Date: 13/Sep/19 20:32
    Worklog Time Spent: 10m 
      Work Description: lukecwik commented on pull request #9556: [BEAM-3200, BEAM-3772] Bugfix setting CREATE_NEVER when writing to destination
URL: https://github.com/apache/beam/pull/9556#discussion_r324355297
 
 

 ##########
 File path: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java
 ##########
 @@ -199,7 +199,6 @@ public void processElement(ProcessContext c, BoundedWindow window) throws Except
         // If writing directly to the destination, then the table is created on the first write
 
 Review comment:
   ```suggestion
            // We must append data to the table after the first pane since the table may not have
   ```
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

            Worklog Id:     (was: 312306)
    Remaining Estimate: 0h
            Time Spent: 10m

> Streaming Pipeline throws RuntimeException when using DynamicDestinations and Method.FILE_LOADS
> -----------------------------------------------------------------------------------------------
>
>                 Key: BEAM-3200
>                 URL: https://issues.apache.org/jira/browse/BEAM-3200
>             Project: Beam
>          Issue Type: Bug
>          Components: io-java-gcp
>    Affects Versions: 2.2.0
>            Reporter: AJ
>            Priority: Critical
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> I am trying to use Method.FILE_LOADS for loading data into BQ in my streaming pipeline using RC3 release of 2.2.0. I am writing to around 500 tables using DynamicDestinations and I am also using withCreateDisposition(CreateDisposition.CREATE_IF_NEEDED). Everything works fine when the first time bigquery load jobs get triggered. But on subsequent triggers pipeline throws a RuntimeException about table not found even though I created the pipeline with CreateDisposition.CREATE_IF_NEEDED. The exact exception is:
> {code}
> java.lang.RuntimeException: Failed to create load job with id prefix 717aed9ed1ef4aa7a616e1132f8b7f6d_a0928cae3d670b32f01ab2d9fe5cc0ee_00001_00001, reached max retries: 3, last failed load job: {
>   "configuration" : {
>     "load" : {
>       "createDisposition" : "CREATE_NEVER",
>       "destinationTable" : {
>         "datasetId" : ...,
>         "projectId" : ...,
>         "tableId" : ....
>       },
>     "errors" : [ }
>       "message" : "Not found: Table ....,
>       "reason" : "notFound"
>     } ],
> {code}
> My theory is all the subsequent load jobs get trigged using CREATE_NEVER disposition and 
> this might be due to https://github.com/apache/beam/blob/release-2.2.0/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java#L140
> When using DynamicDestinations all the destination tables might not be known during the first trigger and hence the pipeline's create disposition should be respected.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)