[C++][Parquet]Delta Binary Encoding

[Brian Kiefer <bk...@uwalumni.com>]

Are there any near term plans to implement delta binary encoding for the
cpp parquet libs? What is the overall community interest in this?

I'd personally like to see it because of how well it performs with
time-series data. It also appears to be the last major encoding type not
implemented. Could the parquet Rust libs be helpful here to take some short
cuts? The Rust version is very feature complete, but the higher level
languages are based on the C++ system level implementation. Thanks for the
discussion in advance!

Re: [C++][Parquet]Delta Binary Encoding

[Micah Kornfield <em...@gmail.com>]

Hi Brian,

> Are there any near term plans to implement delta binary encoding for the
> cpp parquet libs?

I'm not aware of anyone working on this.


> What is the overall community interest in this?

I think we would like to see it done but I don't think anyone has had the
bandwidth to do it.  If you wanted to contribute something it would be
welcome.

Could the parquet Rust libs be helpful here to take some short cuts?

Not for an actual C++ implementation (other than perhaps adapting source
code).  I believe there are bindings through data fusion in python and
other languages that might make this feasible if you wanted to rig this
together but I don't think we'd ever bind the C++ implementation directly
to Rust.

Cheers,
Micah



On Thu, Jan 27, 2022 at 3:20 PM Brian Kiefer <bk...@uwalumni.com> wrote:

> Are there any near term plans to implement delta binary encoding for the
> cpp parquet libs? What is the overall community interest in this?
>
> I'd personally like to see it because of how well it performs with
> time-series data. It also appears to be the last major encoding type not
> implemented. Could the parquet Rust libs be helpful here to take some short
> cuts? The Rust version is very feature complete, but the higher level
> languages are based on the C++ system level implementation. Thanks for the
> discussion in advance!
>
>

Re: [C++][Parquet]Delta Binary Encoding

[Antoine Pitrou <an...@python.org>]

On Thu, 27 Jan 2022 18:19:43 -0500
Brian Kiefer <bk...@uwalumni.com> wrote:
> Are there any near term plans to implement delta binary encoding for the
> cpp parquet libs? What is the overall community interest in this?

I don't know about any near term plans, but since we already have
*decoding* support, adding encoding support would be very welcome.  It
needs someone to actively work on the issue, though.

Regards

Antoine.