Is there a way to skip field decoding without materializing the data?

[JohnGa <pk...@kubikconcepts.com>]

Hello,
I am using BinaryDecoder with a compiled schema.  Let's say there are 10
fields: F1, F2,....F10.  After reading F1, I want to skip to the end of the
record without decoding the fields from F2 to F10.  Is there a way to
accomplish this in Avro?

Thanks.



--
Sent from: http://apache-avro.679487.n3.nabble.com/Avro-Users-f679479.html

Re: Is there a way to skip field decoding without materializing the data?

[Doug Cutting <cu...@gmail.com>]

Yes, as Roger suggests, in Avro this is done by specifying a reader schema
that contains a subset of the fields written.

E.g.,  https://icircuit.net/avro-schema-projection/1446

Doug


On Fri, Apr 3, 2020 at 8:09 AM roger peppe <ro...@gmail.com> wrote:

> If you're using a custom codec, this is potentially possible if you're
> reading only a single record, not a sequence of records. I've been
> considering implementing this as an optimisation. However, I don't think
> it's possible to skip entirely when reading a sequence of records because
> there's no record header that encodes the record length.
>
> However if you read into a schema that only has F1, none of the other
> fields will be materialised, only read and discarded, AIUI (although I'm
> not familiar with the Java codec so it might work differently).
>
>   cheers, rog.
>
>
> On Tue, 31 Mar 2020, 15:59 JohnGa, <pk...@kubikconcepts.com> wrote:
>
>> Hello,
>> I am using BinaryDecoder with a compiled schema.  Let's say there are 10
>> fields: F1, F2,....F10.  After reading F1, I want to skip to the end of
>> the
>> record without decoding the fields from F2 to F10.  Is there a way to
>> accomplish this in Avro?
>>
>> Thanks.
>>
>>
>>
>> --
>> Sent from:
>> http://apache-avro.679487.n3.nabble.com/Avro-Users-f679479.html
>>
>

Re: Is there a way to skip field decoding without materializing the data?

[roger peppe <ro...@gmail.com>]

If you're using a custom codec, this is potentially possible if you're
reading only a single record, not a sequence of records. I've been
considering implementing this as an optimisation. However, I don't think
it's possible to skip entirely when reading a sequence of records because
there's no record header that encodes the record length.

However if you read into a schema that only has F1, none of the other
fields will be materialised, only read and discarded, AIUI (although I'm
not familiar with the Java codec so it might work differently).

  cheers, rog.


On Tue, 31 Mar 2020, 15:59 JohnGa, <pk...@kubikconcepts.com> wrote:

> Hello,
> I am using BinaryDecoder with a compiled schema.  Let's say there are 10
> fields: F1, F2,....F10.  After reading F1, I want to skip to the end of the
> record without decoding the fields from F2 to F10.  Is there a way to
> accomplish this in Avro?
>
> Thanks.
>
>
>
> --
> Sent from: http://apache-avro.679487.n3.nabble.com/Avro-Users-f679479.html
>