You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/03 20:15:00 UTC

[GitHub] [beam] kennknowles opened a new issue, #18706: BigQueryIO - Can't use DynamicDestination with CREATE_IF_NEEDED for unbounded PCollection and FILE_LOADS

kennknowles opened a new issue, #18706:
URL: https://github.com/apache/beam/issues/18706

   My workflow : KAFKA -\> Dataflow streaming -\> BigQuery
   
   Given that having low-latency isn't important in my case, I use FILE_LOADS to reduce the costs. I'm using _BigQueryIO.Write_ with a _DynamicDestination_, which is a table with the current hour as a suffix.
   
   This _BigQueryIO.Write_ is configured like this :
   ```
   
   .withCreateDisposition(CreateDisposition.CREATE_IF_NEEDED)
   .withMethod(Method.FILE_LOADS)
   .withTriggeringFrequency(triggeringFrequency)
   .withNumFileShards(100)
   
   ```
   
   
   The first table is successfully created and is written to. But then the following tables are never created and I get these exceptions:
   
   ```
   
   (99e5cd8c66414e7a): java.lang.RuntimeException: Failed to create load job with id prefix 5047f71312a94bf3a42ee5d67feede75_5295fbf25e1a7534f85e25dcaa9f4986_00001_00023,
   reached max retries: 3, last failed load job: {
     "configuration" : {
       "load" : {
         "createDisposition"
   : "CREATE_NEVER",
         "destinationTable" : {
           "datasetId" : "dev_mydataset",
           "projectId"
   : "myproject-id",
           "tableId" : "mytable_20180302_16"
         },
   
   ```
   
   
   The _CreateDisposition_ used is _CREATE_NEVER_, contrary as _CREATE_IF_NEEDED_ as specified.
   
   Imported from Jira [BEAM-3772](https://issues.apache.org/jira/browse/BEAM-3772). Original Jira may contain additional context.
   Reported by: benjben.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] ragyibrahim commented on issue #18706: BigQueryIO - Can't use DynamicDestination with CREATE_IF_NEEDED for unbounded PCollection and FILE_LOADS

Posted by "ragyibrahim (via GitHub)" <gi...@apache.org>.
ragyibrahim commented on issue #18706:
URL: https://github.com/apache/beam/issues/18706#issuecomment-1409825259

   @kennknowles any updates on this issue? This is causing problems with us in production


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org