[GitHub] [beam] kennknowles opened a new issue, #18680: BigQuery Partitioned table creation/write fails when destination has partition decorator

[GitBox <gi...@apache.org>]

kennknowles opened a new issue, #18680:
URL: https://github.com/apache/beam/issues/18680

   Following is the code that writes to BigQuery: 
   ```
   
   BigQueryIO.writeTableRows()
    .to(destination)
    .withCreateDisposition(CREATE_IF_NEEDED)
    .withWriteDisposition(WRITE_APPEND)
   
   .withSchema(tableSchema)
    .expand(tableRows);
   ```
   
    
   
   Here's the destination's implementation: 
   ```
   
   public TableDestination apply(ValueInSingleWindow<TableRow> input) {
    String partition = timestampExtractor.apply(input.getValue())
   
   .toString(DateTimeFormat.forPattern("yyyyMMdd").withZoneUTC());
    TableReference tableReference = new
   TableReference();
    tableReference.setDatasetId(dataset);
    tableReference.setProjectId(projectId);
   
   tableReference.setTableId(String.format("%s_%s", table, partition));
    log.debug("Will write to BigQuery
   table: %s", tableReference);
    return new TableDestination(tableReference, null);
   }
   ```
   
    
   
   When the dataflow tries to write to this table, I see the following message:
   ```
   
   "errors" : [ {
    "domain" : "global",
    "message" : "Cannot read partition information from a table
   that is not partitioned: <project_id>:<dataset>.<table>$19730522",
    "reason" : "invalid"
    } ]
   ```
   
   So, it looks like it's not creating tables with partition in the first place? 
   
   Apache beam version : 2.2.0
   
   Imported from Jira [BEAM-3501](https://issues.apache.org/jira/browse/BEAM-3501). Original Jira may contain additional context.
   Reported by: darshanmehta2.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org