BigQuery writer error: Clustering fields can only be set when TimePartitioning is set

[Kjetil Halvorsen <kj...@cognite.com>]

Hi,

In Beam 2.23.0 it seems that the BigQuery writer doesn't allow specifying
clustering without partitioning. This was a restriction in BigQuery, but it
has been lifted. It would be great if we can use the Beam BigQuery IO in
combination with writing clustered (but not partitioned) tables.

Best,
Kjetil Halvorsen

Re: BigQuery writer error: Clustering fields can only be set when TimePartitioning is set

[Brian Hulette <bh...@google.com>]

Thanks, I filed https://issues.apache.org/jira/browse/BEAM-10652 for this.
I imagine this would be a pretty straight-forward PR (just remove the check
at [1]?). The only tricky part would be making sure we test it adequately.

[1]
https://github.com/apache/beam/blob/5e0e798ddd827fd212ac89b8c6f6f2cf9e4b29a5/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L2511-L2515

On Wed, Aug 5, 2020 at 4:36 AM Kjetil Halvorsen <
kjetil.halvorsen@cognite.com> wrote:

> Hi,
>
> In Beam 2.23.0 it seems that the BigQuery writer doesn't allow specifying
> clustering without partitioning. This was a restriction in BigQuery, but it
> has been lifted. It would be great if we can use the Beam BigQuery IO in
> combination with writing clustered (but not partitioned) tables.
>
> Best,
> Kjetil Halvorsen
>
>
>