DynamicDestination for AvroIO

[Anant Damle <an...@gmail.com>]

Hi Team,
I am working on a solution with Beam's Java SD, that requires:


   1. Read AVRO files using AvroIO.parseGenericRecords
   2. Modify the records (encrypt some attributes) and generate an updated
   Schema
   3. Write the modified record with updated Schema

I'm facing challenges in Step#3, as AvroIO doesn't have an option to read
Schema from a PCollectionView. I found that BigQueryIO.withSchemaFromView
<https://beam.apache.org/releases/javadoc/2.29.0/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.Write.html#withSchemaFromView-org.apache.beam.sdk.values.PCollectionView->
allows
one to provide Schema through a view.

What should be done for something like AvroIO.withSchemaFromView.
I'm looking for some guidance on how to achieve this. I would be happy to
build and contribute it into the Beam.

Thanks,
Anant

Re: DynamicDestination for AvroIO

[Reuven Lax <re...@google.com>]

I believe you can do this today using DynamicAvroDestinations

On Mon, Jun 7, 2021 at 11:50 AM Anant Damle <an...@gmail.com> wrote:

> Hi Team,
> I am working on a solution with Beam's Java SD, that requires:
>
>
>    1. Read AVRO files using AvroIO.parseGenericRecords
>    2. Modify the records (encrypt some attributes) and generate an
>    updated Schema
>    3. Write the modified record with updated Schema
>
> I'm facing challenges in Step#3, as AvroIO doesn't have an option to read
> Schema from a PCollectionView. I found that BigQueryIO.withSchemaFromView
> <https://beam.apache.org/releases/javadoc/2.29.0/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.Write.html#withSchemaFromView-org.apache.beam.sdk.values.PCollectionView-> allows
> one to provide Schema through a view.
>
> What should be done for something like AvroIO.withSchemaFromView.
> I'm looking for some guidance on how to achieve this. I would be happy to
> build and contribute it into the Beam.
>
> Thanks,
> Anant
>
>