Why Avro(Generic Record) -> Beam Row -> BigQuery TableRow

[Theodore Siu <th...@google.com>]

Hi Everyone,

I am trying to insert Avro data into BigQuery using BigQueryIO. We use beam
because we have additional hydration steps for our avro files and we use a
group by in order to set up multiple load jobs in a window. I see a direct
conversion of Avro into a BigQuery TableRow but they are protected and only
available to Apache Beam Package, (see
https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtils.java
)

The public class seems to want the user to go from Avro(Generic Record) ->
Beam Row -> TableRow (
https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java).
I am guessing this is the suggested path of conversion but is there a
specific reason for this method?

Thank you for your time.
-Theo

Re: Why Avro(Generic Record) -> Beam Row -> BigQuery TableRow

[Chamikara Jayalath <ch...@google.com>]

These utilities are not a part of Beam public API and can change at any
point. Beam transform for writing to BigQuery by default expects a
PCollection of TableRow objects as input but can also take a format
function that converts a given type into TableRow objects.
https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L1489

So please consider either providing an appropriate format function or
converting your PCollection<T> into a PCollection<TableRow> using a ParDo
before the BigQueryIO.write() step.

Thanks,
Cham

On Fri, Jun 14, 2019 at 12:55 PM Theodore Siu <th...@google.com> wrote:

> Hi Everyone,
>
> I am trying to insert Avro data into BigQuery using BigQueryIO. We use
> beam because we have additional hydration steps for our avro files and we
> use a group by in order to set up multiple load jobs in a window. I see a
> direct conversion of Avro into a BigQuery TableRow but they are protected
> and only available to Apache Beam Package, (see
> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtils.java
> )
>
> The public class seems to want the user to go from Avro(Generic Record) ->
> Beam Row -> TableRow (
> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java).
> I am guessing this is the suggested path of conversion but is there a
> specific reason for this method?
>
> Thank you for your time.
> -Theo
>