[jira] [Created] (BEAM-4370) Cannot write to partitions using DynamicDestination or SerializableFunction<>

["Peter Brumblay (JIRA)" <ji...@apache.org>]

Peter Brumblay created BEAM-4370:
------------------------------------

             Summary: Cannot write to partitions using DynamicDestination or SerializableFunction<>
                 Key: BEAM-4370
                 URL: https://issues.apache.org/jira/browse/BEAM-4370
             Project: Beam
          Issue Type: Bug
          Components: io-java-gcp
    Affects Versions: 2.4.0
            Reporter: Peter Brumblay
            Assignee: Chamikara Jayalath


When trying to use BigQueryIO.writeTableRows() with DynamicDestinations<TableRow, String> or

SerializableFunction<ValueInSingleWindow<TableRow>, TableDestination>, the operation fails with an error message claiming that decorators are not allowed:

"Invalid table ID <table>$<partition>. Table IDs must be alphanumeric (plus underscores) and must be at most 1024 characters long. Also, Table decorators cannot be used."

This should work according to ...
 # [https://stackoverflow.com/questions/47351578/create-dynamic-side-outputs-in-apache-beam-dataflow#comment81668927_47351578] and [https://stackoverflow.com/a/43505535]
 # [https://shinesolutions.com/2017/12/05/fun-with-serializable-functions-and-dynamic-destinations-in-cloud-dataflow/]

See the following for a complete example demonstrating the error with both scenarios:

[https://github.com/pbrumblay/bigqueryioproblems/blob/master/src/main/java/com/fearlesstg/CannotWriteToPartitionedTable.java]

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)