You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "Peter Brumblay (JIRA)" <ji...@apache.org> on 2018/05/25 18:07:00 UTC
[jira] [Commented] (BEAM-4370) Cannot write to bigquery partitions
using DynamicDestination or SerializableFunction<>
[ https://issues.apache.org/jira/browse/BEAM-4370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16491095#comment-16491095 ]
Peter Brumblay commented on BEAM-4370:
--------------------------------------
Ok. I have figured it out through a lot of pain and suffering. This is NOT a bug in BigQueryIO.
The problem:
If you include a *malformed* table decorator, you get this error message from the BIGQUERY REST API:
"Invalid table ID \"merging_partitions$2018-03-03\". Table IDs must be alphanumeric (plus underscores) and must be at most 1024 characters long. Also, Table decorators cannot be used."
Note 2 things
1. The table decorator I'm trying to use is NOT in the correct format. it should NOT be $YYYY-MM-DD. It SHOULD be $YYYYMMDD.
2. I get an error message which is COMPLETELY misleading. Table decorators CAN and SHOULD be used when working with partitions.
This is a working example: [https://gist.github.com/pbrumblay/b49bf08e0b652319120dfb1c861f0936]
Closing.
> Cannot write to bigquery partitions using DynamicDestination or SerializableFunction<>
> --------------------------------------------------------------------------------------
>
> Key: BEAM-4370
> URL: https://issues.apache.org/jira/browse/BEAM-4370
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
> Affects Versions: 2.4.0
> Reporter: Peter Brumblay
> Assignee: Chamikara Jayalath
> Priority: Major
> Fix For: Not applicable
>
>
> When trying to use BigQueryIO.writeTableRows() with DynamicDestinations<TableRow, String> or
> SerializableFunction<ValueInSingleWindow<TableRow>, TableDestination>, the operation fails with an error message claiming that decorators are not allowed:
> "Invalid table ID <table>$<partition>. Table IDs must be alphanumeric (plus underscores) and must be at most 1024 characters long. Also, Table decorators cannot be used."
> This should work according to ...
> # [https://stackoverflow.com/questions/47351578/create-dynamic-side-outputs-in-apache-beam-dataflow#comment81668927_47351578] and [https://stackoverflow.com/a/43505535]
> # [https://shinesolutions.com/2017/12/05/fun-with-serializable-functions-and-dynamic-destinations-in-cloud-dataflow/]
> See the following for a complete example demonstrating the error with both scenarios:
> [https://github.com/pbrumblay/bigqueryioproblems/blob/master/src/main/java/com/fearlesstg/CannotWriteToPartitionedTable.java]
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)