[jira] [Commented] (BEAM-2390) allow user to use .setTimePartitioning in BigQueryIO.write

["Eric Johston (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16032409#comment-16032409 ] 

Eric Johston commented on BEAM-2390:
------------------------------------

My initial commit had some errors in that

a) TimePartitioning is not serializable, and
b) CreateTables#possibleCreate fails when TableDestination includes a partition with $

I've made changes to these by storing TimePartitioning in Json format similar to how the schemas are propagated. I've also modified the table creation such that when creating a table Beam only looks at the part before $. It seems to be working now (running this from my own fork in production)

> allow user to use .setTimePartitioning in BigQueryIO.write
> ----------------------------------------------------------
>
>                 Key: BEAM-2390
>                 URL: https://issues.apache.org/jira/browse/BEAM-2390
>             Project: Beam
>          Issue Type: Improvement
>          Components: beam-model-runner-api
>    Affects Versions: 2.0.0
>            Reporter: Eric Johston
>            Assignee: Kenneth Knowles
>              Labels: easyfix, features, newbie
>             Fix For: 2.0.0
>
>   Original Estimate: 2h
>  Remaining Estimate: 2h
>
> Currently when writing to a table with BigQueryIO sink, there is no way to create a new table that is date partitioned. This would be very useful, since currently the only way to do this is  by manually creating a table ahead of time. We should be able to leverage the automatic table creation functionality for date partitioned tables.
> The best way to do this would be to have a withTimePartitioning method in the BigQueryIO class.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)