You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Neville Li <ne...@gmail.com> on 2019/06/13 20:46:31 UTC
AvroIO read SpecificRecord with custom reader schema?
Hi,
Is it just me or is there no way for AvroIO to read SpecificRecords with a
custom reader schema?
AvroIO.read(Class<T> recordClass) will use the schema of T and there's no
way to override it.
Cheers,
Neville
Re: AvroIO read SpecificRecord with custom reader schema?
Posted by Ismaël Mejía <ie...@gmail.com>.
> There is an alternative approach with getting PCollection<Row> using AvroIO and then converting Row to SpecificRecord using new Schemas APIs
Yes but this will have a higher run time, no ? Maybe worth a JIRA + PR
for this feature.
On Fri, Jun 14, 2019 at 10:15 AM Gleb Kanterov <gl...@spotify.com> wrote:
>
> There is an alternative approach with getting PCollection<Row> using AvroIO and then converting Row to SpecificRecord using new Schemas APIs.
>
> On Fri, Jun 14, 2019 at 1:58 AM Chamikara Jayalath <ch...@google.com> wrote:
>>
>> I see. I don't think this is supported by AvroIO currently so your best bet will probably be to do a GenericRecord to SpecificRecord conversion in a ParDo that follows the read transform.
>>
>> On Thu, Jun 13, 2019 at 4:32 PM Neville Li <ne...@gmail.com> wrote:
>>>
>>> That gives me a GenericRecord which is not type safe.
>>> In my case I have the compiled SpecificRecord class i.e. MyRecord available, but would like to pass in a schema other than MyRecord.getClassSchema() to say populate a subset of the fields.
>>>
>>> On Thu, Jun 13, 2019 at 6:18 PM Chamikara Jayalath <ch...@google.com> wrote:
>>>>
>>>> Does AvroIO.readGenericRecords() work ?
>>>> https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java#L333
>>>>
>>>> Thanks,
>>>> Cham
>>>>
>>>> On Thu, Jun 13, 2019 at 1:46 PM Neville Li <ne...@gmail.com> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> Is it just me or is there no way for AvroIO to read SpecificRecords with a custom reader schema?
>>>>>
>>>>> AvroIO.read(Class<T> recordClass) will use the schema of T and there's no way to override it.
>>>>>
>>>>> Cheers,
>>>>> Neville
>
>
>
> --
> Cheers,
> Gleb
Re: AvroIO read SpecificRecord with custom reader schema?
Posted by Gleb Kanterov <gl...@spotify.com>.
There is an alternative approach with getting PCollection<Row> using AvroIO
and then converting Row to SpecificRecord using new Schemas APIs.
On Fri, Jun 14, 2019 at 1:58 AM Chamikara Jayalath <ch...@google.com>
wrote:
> I see. I don't think this is supported by AvroIO currently so your best
> bet will probably be to do a GenericRecord to SpecificRecord conversion in
> a ParDo that follows the read transform.
>
> On Thu, Jun 13, 2019 at 4:32 PM Neville Li <ne...@gmail.com> wrote:
>
>> That gives me a GenericRecord which is not type safe.
>> In my case I have the compiled SpecificRecord class i.e. MyRecord
>> available, but would like to pass in a schema other than
>> MyRecord.getClassSchema() to say populate a subset of the fields.
>>
>> On Thu, Jun 13, 2019 at 6:18 PM Chamikara Jayalath <ch...@google.com>
>> wrote:
>>
>>> Does AvroIO.readGenericRecords() work ?
>>>
>>> https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java#L333
>>>
>>> Thanks,
>>> Cham
>>>
>>> On Thu, Jun 13, 2019 at 1:46 PM Neville Li <ne...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> Is it just me or is there no way for AvroIO to read SpecificRecords
>>>> with a custom reader schema?
>>>>
>>>> AvroIO.read(Class<T> recordClass) will use the schema of T and there's
>>>> no way to override it.
>>>>
>>>> Cheers,
>>>> Neville
>>>>
>>>
--
Cheers,
Gleb
Re: AvroIO read SpecificRecord with custom reader schema?
Posted by Chamikara Jayalath <ch...@google.com>.
I see. I don't think this is supported by AvroIO currently so your best bet
will probably be to do a GenericRecord to SpecificRecord conversion in a
ParDo that follows the read transform.
On Thu, Jun 13, 2019 at 4:32 PM Neville Li <ne...@gmail.com> wrote:
> That gives me a GenericRecord which is not type safe.
> In my case I have the compiled SpecificRecord class i.e. MyRecord
> available, but would like to pass in a schema other than
> MyRecord.getClassSchema() to say populate a subset of the fields.
>
> On Thu, Jun 13, 2019 at 6:18 PM Chamikara Jayalath <ch...@google.com>
> wrote:
>
>> Does AvroIO.readGenericRecords() work ?
>>
>> https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java#L333
>>
>> Thanks,
>> Cham
>>
>> On Thu, Jun 13, 2019 at 1:46 PM Neville Li <ne...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Is it just me or is there no way for AvroIO to read SpecificRecords
>>> with a custom reader schema?
>>>
>>> AvroIO.read(Class<T> recordClass) will use the schema of T and there's
>>> no way to override it.
>>>
>>> Cheers,
>>> Neville
>>>
>>
Re: AvroIO read SpecificRecord with custom reader schema?
Posted by Neville Li <ne...@gmail.com>.
That gives me a GenericRecord which is not type safe.
In my case I have the compiled SpecificRecord class i.e. MyRecord
available, but would like to pass in a schema other than
MyRecord.getClassSchema() to say populate a subset of the fields.
On Thu, Jun 13, 2019 at 6:18 PM Chamikara Jayalath <ch...@google.com>
wrote:
> Does AvroIO.readGenericRecords() work ?
>
> https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java#L333
>
> Thanks,
> Cham
>
> On Thu, Jun 13, 2019 at 1:46 PM Neville Li <ne...@gmail.com> wrote:
>
>> Hi,
>>
>> Is it just me or is there no way for AvroIO to read SpecificRecords with
>> a custom reader schema?
>>
>> AvroIO.read(Class<T> recordClass) will use the schema of T and there's
>> no way to override it.
>>
>> Cheers,
>> Neville
>>
>
Re: AvroIO read SpecificRecord with custom reader schema?
Posted by Chamikara Jayalath <ch...@google.com>.
Does AvroIO.readGenericRecords() work ?
https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroIO.java#L333
Thanks,
Cham
On Thu, Jun 13, 2019 at 1:46 PM Neville Li <ne...@gmail.com> wrote:
> Hi,
>
> Is it just me or is there no way for AvroIO to read SpecificRecords with
> a custom reader schema?
>
> AvroIO.read(Class<T> recordClass) will use the schema of T and there's no
> way to override it.
>
> Cheers,
> Neville
>