Overridden methods of dictionary

[Swapnil Chougule <th...@gmail.com>]

Hi Folks,

Abstract class Dictionary contains methods:
public Binary decodeToBinary(int id)
public int decodeToInt(int id)
public long decodeToLong(int id)
public float decodeToFloat(int id)
public double decodeToDouble(int id)
public boolean decodeToBoolean(int id)

These are subsequently overridden in respective dictionary implementation
like

PlainLongDictionary overrides "decodeToLong" method only
PlainIntegerDictionary overrides "decodeToInt" method only
& so on

Can we support type upcasting here ?
PlainLongDictionary overrides "decodeToLong" & "decodeToDouble" methods
PlainIntegerDictionary overrides "decodeToInt", "decodeToLong" &
"decodeToDouble" methods

Type up casting is valid use case.
It also needs some changes in ValidTypeMap.java &
SchemaCompatibilityValidator.java for Filter predicate.

Can parquet support this type upcasting feature? I came across such
scenario in one of my use case.

Thanks,
Swapnil

Re: Overridden methods of dictionary

[Ryan Blue <rb...@netflix.com.INVALID>]

Hi Swapnil,

I don't think it is a good idea for Parquet to implement casting at this
level. Parquet should return the data that was stored and make it possible
for higher-level components, like record materialization, to coerce types.

rb

On Mon, Feb 25, 2019 at 11:48 AM Swapnil Chougule <th...@gmail.com>
wrote:

> Hi Folks,
>
> Abstract class Dictionary contains methods:
> public Binary decodeToBinary(int id)
> public int decodeToInt(int id)
> public long decodeToLong(int id)
> public float decodeToFloat(int id)
> public double decodeToDouble(int id)
> public boolean decodeToBoolean(int id)
>
> These are subsequently overridden in respective dictionary implementation
> like
>
> PlainLongDictionary overrides "decodeToLong" method only
> PlainIntegerDictionary overrides "decodeToInt" method only
> & so on
>
> Can we support type upcasting here ?
> PlainLongDictionary overrides "decodeToLong" & "decodeToDouble" methods
> PlainIntegerDictionary overrides "decodeToInt", "decodeToLong" &
> "decodeToDouble" methods
>
> Type up casting is valid use case.
> It also needs some changes in ValidTypeMap.java &
> SchemaCompatibilityValidator.java for Filter predicate.
>
> Can parquet support this type upcasting feature? I came across such
> scenario in one of my use case.
>
> Thanks,
> Swapnil
>


-- 
Ryan Blue
Software Engineer
Netflix