[jira] [Updated] (BEAM-3501) BigQuery Partitioned table creation/write fails when destination has partition decorator

["Beam JIRA Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BEAM-3501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Beam JIRA Bot updated BEAM-3501:
--------------------------------
    Labels:   (was: stale-P2)

> BigQuery Partitioned table creation/write fails when destination has partition decorator
> ----------------------------------------------------------------------------------------
>
>                 Key: BEAM-3501
>                 URL: https://issues.apache.org/jira/browse/BEAM-3501
>             Project: Beam
>          Issue Type: Bug
>          Components: io-java-gcp
>            Reporter: Darshan Mehta
>            Priority: P3
>
> Following is the code that writes to BigQuery: 
> {code:java}
> BigQueryIO.writeTableRows()
>  .to(destination)
>  .withCreateDisposition(CREATE_IF_NEEDED)
>  .withWriteDisposition(WRITE_APPEND)
>  .withSchema(tableSchema)
>  .expand(tableRows);{code}
>  
> Here's the destination's implementation: 
> {code:java}
> public TableDestination apply(ValueInSingleWindow<TableRow> input) {
>  String partition = timestampExtractor.apply(input.getValue())
>  .toString(DateTimeFormat.forPattern("yyyyMMdd").withZoneUTC());
>  TableReference tableReference = new TableReference();
>  tableReference.setDatasetId(dataset);
>  tableReference.setProjectId(projectId);
>  tableReference.setTableId(String.format("%s_%s", table, partition));
>  log.debug("Will write to BigQuery table: %s", tableReference);
>  return new TableDestination(tableReference, null);
> }{code}
>  
> When the dataflow tries to write to this table, I see the following message:
> {code:java}
> "errors" : [ {
>  "domain" : "global",
>  "message" : "Cannot read partition information from a table that is not partitioned: <project_id>:<dataset>.<table>$19730522",
>  "reason" : "invalid"
>  } ]{code}
> So, it looks like it's not creating tables with partition in the first place? 
> Apache beam version : 2.2.0



--
This message was sent by Atlassian Jira
(v8.3.4#803005)