You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@parquet.apache.org by Michael Heuer <he...@gmail.com> on 2019/04/24 18:30:28 UTC

add new custom encodings

Hello,

Might someone be able to point me to the Parquet APIs I would need to work with to add new custom encodings for String columns?  We have a few different bit-packing encodings that take advantage of the particular data stored in those columns.

Thank you in advance,

   michael

Re: add new custom encodings

Posted by Michael Heuer <he...@gmail.com>.

Hello Wes,

Something different; these encodings would take advantage of ambiguity in DNA sequences.

   michael


> On Apr 24, 2019, at 1:38 PM, Wes McKinney <we...@gmail.com> wrote:
> 
> hi Michael,
> 
> Are you proposing to use alternative value encodings beyond the ones in
> 
> https://github.com/apache/parquet-format/blob/master/Encodings.md
> 
> or something different?
> 
> - Wes
> 
> On Wed, Apr 24, 2019 at 1:30 PM Michael Heuer <he...@gmail.com> wrote:
>> 
>> Hello,
>> 
>> Might someone be able to point me to the Parquet APIs I would need to work with to add new custom encodings for String columns?  We have a few different bit-packing encodings that take advantage of the particular data stored in those columns.
>> 
>> Thank you in advance,
>> 
>>   michael

Re: add new custom encodings

Posted by Wes McKinney <we...@gmail.com>.

hi Michael,

Are you proposing to use alternative value encodings beyond the ones in

https://github.com/apache/parquet-format/blob/master/Encodings.md

or something different?

- Wes

On Wed, Apr 24, 2019 at 1:30 PM Michael Heuer <he...@gmail.com> wrote:
>
> Hello,
>
> Might someone be able to point me to the Parquet APIs I would need to work with to add new custom encodings for String columns?  We have a few different bit-packing encodings that take advantage of the particular data stored in those columns.
>
> Thank you in advance,
>
>    michael