You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by PRATEEK GAUR <pr...@gmail.com> on 2026/04/22 17:00:19 UTC
Re: [Parquet] ALP Encoding for Floating point data
Hi team,
Hope everyone is doing well. I got a chance to work through all the
remaining feedback and update the spec doc. Here are the new artifacts
1) Spec document :
https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
2) Spec document in parquet format repo :
https://github.com/apache/parquet-format/pull/557
3) Alp implementation in arrow c++ repo :
https://github.com/apache/arrow/pull/48345/changes
4) Alp implementation in parquet-java repo : Work for Vinoo and Julien
https://github.com/apache/parquet-java/pull/3397
5) PR with test and benchmarking artifacts in parquet-testing repo :
https://github.com/apache/parquet-testing/pull/100
And
- Go : Arnav just submitted an in progress implementation in Go.
https://github.com/apache/arrow-go/pull/704 (I haven't started looking
at it yet)
- Rust : I remember Andrew mentioned that this work is also in progress
(So 4 languages!)
*Arrow C++ implementation *
The PR is out and was also used by Antoine to report the numbers as
reported here. Micah and Konstantin have given 1 round of feedback and I'm
addressing them today. Please note that the default optimization flag for
compiling is O2 and not Q3. I got around 70% performance improvement in the
decoding speed when using the O3 flag.
*Parqet-MR Java implementation (working with Vinoo and Julien) and **Cross
Language testing*
Let me know if you have any questions or feedback.
Now pasting some performance numbers
Table 1: C++ ALP Double Decode — Spotify Columns (Graviton 3, ARM
Neoverse V1)
┌──────────────────┬──────────────┬──────────────┬─────────┐
│ Column │ -O2 (MB/s) │ -O3 (MB/s) │ Speedup │
├──────────────────┼──────────────┼──────────────┼─────────┤
│ valence │ 3,155 │ 5,523 │ 1.75x │
│ danceability │ 3,233 │ 5,685 │ 1.76x │
│ energy │ 3,197 │ 5,652 │ 1.77x │
│ loudness │ 3,186 │ 5,473 │ 1.72x │
└──────────────────┴──────────────┴──────────────┴─────────┘
On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <pr...@gmail.com> wrote:
> @Micah Kornfield <em...@gmail.com> : Got it.
>
> @Andrew Lamb <an...@gmail.com>
>
>
>> Do you think it would be good to start moving the spec development into
>> markdown format, in preparation for finalizing it?
>>
>
> Yes I'll update the numbers for some of the examples I have in the spec
> based
> on the updated header size. Then we should be good to go for the markdown
> format.
>
> Thanks everyone!
>
>
>>
>> Andrew
>>
>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <pr...@gmail.com> wrote:
>>
>> > Hi team,
>> >
>> > 1) Andrew
>> >
>> > - Thanks for working on test files. My PR did add all the test files
>> I
>> > used to benchmark on datasets. Maybe we can club it together. WIll
>> also
>> > aid
>> > cross language testing
>> > - Kosta Tarasov working on Rust implementation. This is great.
>> Thanks
>> >
>> >
>> > 2) Antoine
>> >
>> > - Thanks a lot for reporting the numbers on AMD. Looks like you are
>> > getting 8X the decoding performance of BSS. This is amazing!!.
>> > - Thanks for acknowledging the sampling design.
>> > - I agree with you on Fastlanes. In some crude experiments I didn't
>> get
>> > a good perf benefit from it on Graviton3 (but maybe there was
>> something
>> > wrong with my implementation).
>> > - Locking the 16bit exception encoding for the spec in this case.
>> > - Awesome I think we have solved for all open questions minus the
>> > version byte :). (will get back on this soon)
>> >
>> >
>> > 3) Micah
>> >
>> > - FastLanes : The current spec does allow for using FastLane with the
>> > configurable enum value for layout. We should be able to inject any
>> > layout
>> > in the current design.
>> >
>> >
>> > Working on resolving all remaining open comments on the spec this week.
>> >
>> > Best
>> > Prateek
>> >
>> >
>> > On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <st...@cloudera.com>
>> > wrote:
>> >
>> > > On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <em...@gmail.com>
>> > > wrote:
>> > >
>> > > >
>> > > >
>> > > > It looks like the actual issue described for ORC in the paper is
>> that
>> > it
>> > > > has multiple sub-encodings in a batch. This is different then the
>> > design
>> > > > proposed here where there is still fixed encoding per page in
>> parquet.
>> > > > Given reasonably sized pages I don't think branch misprediction
>> should
>> > > be a
>> > > > big issue for new encodings. I agree that we should be
>> conservative in
>> > > > general for adding new encodings.
>> > > >
>> > > >
>> > > +1
>> > >
>> >
>>
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Andrew Lamb <an...@gmail.com>.
I also just filed a ticket[1] to track adding the example files and linked
it around to try and give it a bit more visibility.
[1]: https://github.com/apache/parquet-testing/issues/105
On Wed, Apr 29, 2026 at 7:25 PM Curt Hagenlocher <cu...@hagenlocher.org>
wrote:
> Ah, thanks! I missed that.
>
> On Wed, Apr 29, 2026 at 4:24 PM Micah Kornfield <em...@gmail.com>
> wrote:
>
>> Hi Curt,
>>
>>> As part of the process of amending the Parquet format, perhaps it would
>>> be a good idea for early implementations to generate sample files and
>>> commit them to apache/parquet-testing: Apache Parquet Testing
>>> <https://github.com/apache/parquet-testing> for other implementations
>>> to leverage?
>>
>>
>> It got dropped in the thread but does
>> https://github.com/apache/parquet-testing/pull/100 address your concerns?
>>
>> Thanks,
>> Micah
>>
>> On Wed, Apr 29, 2026 at 4:20 PM Curt Hagenlocher <cu...@hagenlocher.org>
>> wrote:
>>
>>> As part of the process of amending the Parquet format, perhaps it would
>>> be a good idea for early implementations to generate sample files and
>>> commit them to apache/parquet-testing: Apache Parquet Testing
>>> <https://github.com/apache/parquet-testing> for other implementations
>>> to leverage?
>>>
>>> -Curt
>>>
>>> On Wed, Apr 29, 2026 at 4:11 PM PRATEEK GAUR <pr...@gmail.com> wrote:
>>>
>>>> Thanks Andrew and Micah for review feedback on the two PR's
>>>> 1) (c++ arrow repo) https://github.com/apache/arrow/pull/48345/changes
>>>> 2) (parquet-format repo)
>>>> https://github.com/apache/parquet-format/pull/557
>>>>
>>>> I have addressed all (unless I missed something) comments on the two
>>>> PR's.
>>>>
>>>> Best
>>>> Prateek
>>>>
>>>> On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <pr...@gmail.com>
>>>> wrote:
>>>>
>>>> > Thanks Andrew and Micah.
>>>> >
>>>> > `fair amount of feedback on at least the implementations`
>>>> > For the c++ I have already started addressing the feedback, I should
>>>> be
>>>> > done with that Monday/Tuesday.
>>>> > I think Vinoo too has been making good progress on the Java
>>>> implementation.
>>>> >
>>>> > Best
>>>> > Prateek
>>>> >
>>>> > On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <an...@gmail.com>
>>>> > wrote:
>>>> >
>>>> >> Got it. Thank you for the clarification -- I will try and look into
>>>> the
>>>> >> spec and the Rust implementation[1] in this next week
>>>> >>
>>>> >> [1]: https://github.com/apache/arrow-rs/pull/9372
>>>> >>
>>>> >> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <
>>>> emkornfield@gmail.com>
>>>> >> wrote:
>>>> >>
>>>> >>> Hi Andrew,
>>>> >>> I think there is a fair amount of feedback on at least the
>>>> >>> implementations, typically I think we've waited till they are close
>>>> to
>>>> >>> mergeable before a final vote. Otherwise I agree we are very close.
>>>> >>>
>>>> >>> -Micah
>>>> >>>
>>>> >>> On Saturday, April 25, 2026, Andrew Lamb <an...@gmail.com>
>>>> wrote:
>>>> >>>
>>>> >>>> Thanks Prateek,
>>>> >>>>
>>>> >>>> I think from this content it looks to me like we are ready to
>>>> start a
>>>> >>>> vote to explicitly accept ALP into Parquet
>>>> >>>>
>>>> >>>> Does anyone know of a reason we should postpone it for longer?
>>>> >>>> Perhaps someone needs some more time to review?
>>>> >>>>
>>>> >>>> Andrew
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <pr...@gmail.com>
>>>> >>>> wrote:
>>>> >>>>
>>>> >>>>> Hi team,
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> Hope everyone is doing well. I got a chance to work through all
>>>> the
>>>> >>>>> remaining feedback and update the spec doc. Here are the new
>>>> artifacts
>>>> >>>>>
>>>> >>>>> 1) Spec document :
>>>> >>>>>
>>>> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
>>>> >>>>>
>>>> >>>>> 2) Spec document in parquet format repo :
>>>> >>>>> https://github.com/apache/parquet-format/pull/557
>>>> >>>>>
>>>> >>>>> 3) Alp implementation in arrow c++ repo :
>>>> >>>>> https://github.com/apache/arrow/pull/48345/changes
>>>> >>>>>
>>>> >>>>> 4) Alp implementation in parquet-java repo : Work for Vinoo and
>>>> Julien
>>>> >>>>> https://github.com/apache/parquet-java/pull/3397
>>>> >>>>>
>>>> >>>>> 5) PR with test and benchmarking artifacts in parquet-testing
>>>> repo :
>>>> >>>>> https://github.com/apache/parquet-testing/pull/100
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> And
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> - Go : Arnav just submitted an in progress implementation in
>>>> Go.
>>>> >>>>> https://github.com/apache/arrow-go/pull/704 (I haven't started
>>>> >>>>> looking at it yet)
>>>> >>>>> - Rust : I remember Andrew mentioned that this work is also in
>>>> >>>>> progress (So 4 languages!)
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> *Arrow C++ implementation *
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> The PR is out and was also used by Antoine to report the numbers
>>>> as
>>>> >>>>> reported here. Micah and Konstantin have given 1 round of feedback
>>>> >>>>> and I'm addressing them today. Please note that the default
>>>> >>>>> optimization flag for compiling is O2 and not Q3. I got around 70%
>>>> >>>>> performance improvement in the decoding speed when using the O3
>>>> flag.
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> *Parqet-MR Java implementation (working with Vinoo and Julien)
>>>> and **Cross
>>>> >>>>> Language testing*
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> Let me know if you have any questions or feedback.
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> Now pasting some performance numbers
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> Table 1: C++ ALP Double Decode — Spotify Columns (Graviton 3,
>>>> ARM
>>>> >>>>> Neoverse V1)
>>>> >>>>>
>>>> >>>>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
>>>> >>>>>
>>>> >>>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │ Speedup │
>>>> >>>>>
>>>> >>>>> ├──────────────────┼──────────────┼──────────────┼─────────┤
>>>> >>>>>
>>>> >>>>> │ valence │ 3,155 │ 5,523 │ 1.75x │
>>>> >>>>>
>>>> >>>>> │ danceability │ 3,233 │ 5,685 │ 1.76x │
>>>> >>>>>
>>>> >>>>> │ energy │ 3,197 │ 5,652 │ 1.77x │
>>>> >>>>>
>>>> >>>>> │ loudness │ 3,186 │ 5,473 │ 1.72x │
>>>> >>>>>
>>>> >>>>> └──────────────────┴──────────────┴──────────────┴─────────┘
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <pr...@gmail.com>
>>>> >>>>> wrote:
>>>> >>>>>
>>>> >>>>>> @Micah Kornfield <em...@gmail.com> : Got it.
>>>> >>>>>>
>>>> >>>>>> @Andrew Lamb <an...@gmail.com>
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>>> Do you think it would be good to start moving the spec
>>>> development
>>>> >>>>>>> into
>>>> >>>>>>> markdown format, in preparation for finalizing it?
>>>> >>>>>>>
>>>> >>>>>>
>>>> >>>>>> Yes I'll update the numbers for some of the examples I have in
>>>> the
>>>> >>>>>> spec based
>>>> >>>>>> on the updated header size. Then we should be good to go for the
>>>> >>>>>> markdown format.
>>>> >>>>>>
>>>> >>>>>> Thanks everyone!
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>>>
>>>> >>>>>>> Andrew
>>>> >>>>>>>
>>>> >>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <
>>>> prtkgaur@gmail.com>
>>>> >>>>>>> wrote:
>>>> >>>>>>>
>>>> >>>>>>> > Hi team,
>>>> >>>>>>> >
>>>> >>>>>>> > 1) Andrew
>>>> >>>>>>> >
>>>> >>>>>>> > - Thanks for working on test files. My PR did add all the
>>>> test
>>>> >>>>>>> files I
>>>> >>>>>>> > used to benchmark on datasets. Maybe we can club it
>>>> together.
>>>> >>>>>>> WIll also
>>>> >>>>>>> > aid
>>>> >>>>>>> > cross language testing
>>>> >>>>>>> > - Kosta Tarasov working on Rust implementation. This is
>>>> great.
>>>> >>>>>>> Thanks
>>>> >>>>>>> >
>>>> >>>>>>> >
>>>> >>>>>>> > 2) Antoine
>>>> >>>>>>> >
>>>> >>>>>>> > - Thanks a lot for reporting the numbers on AMD. Looks
>>>> like you
>>>> >>>>>>> are
>>>> >>>>>>> > getting 8X the decoding performance of BSS. This is
>>>> amazing!!.
>>>> >>>>>>> > - Thanks for acknowledging the sampling design.
>>>> >>>>>>> > - I agree with you on Fastlanes. In some crude experiments
>>>> I
>>>> >>>>>>> didn't get
>>>> >>>>>>> > a good perf benefit from it on Graviton3 (but maybe there
>>>> was
>>>> >>>>>>> something
>>>> >>>>>>> > wrong with my implementation).
>>>> >>>>>>> > - Locking the 16bit exception encoding for the spec in this
>>>> >>>>>>> case.
>>>> >>>>>>> > - Awesome I think we have solved for all open questions
>>>> minus
>>>> >>>>>>> the
>>>> >>>>>>> > version byte :). (will get back on this soon)
>>>> >>>>>>> >
>>>> >>>>>>> >
>>>> >>>>>>> > 3) Micah
>>>> >>>>>>> >
>>>> >>>>>>> > - FastLanes : The current spec does allow for using
>>>> FastLane
>>>> >>>>>>> with the
>>>> >>>>>>> > configurable enum value for layout. We should be able to
>>>> inject
>>>> >>>>>>> any
>>>> >>>>>>> > layout
>>>> >>>>>>> > in the current design.
>>>> >>>>>>> >
>>>> >>>>>>> >
>>>> >>>>>>> > Working on resolving all remaining open comments on the spec
>>>> this
>>>> >>>>>>> week.
>>>> >>>>>>> >
>>>> >>>>>>> > Best
>>>> >>>>>>> > Prateek
>>>> >>>>>>> >
>>>> >>>>>>> >
>>>> >>>>>>> > On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <
>>>> >>>>>>> stevel@cloudera.com>
>>>> >>>>>>> > wrote:
>>>> >>>>>>> >
>>>> >>>>>>> > > On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
>>>> >>>>>>> emkornfield@gmail.com>
>>>> >>>>>>> > > wrote:
>>>> >>>>>>> > >
>>>> >>>>>>> > > >
>>>> >>>>>>> > > >
>>>> >>>>>>> > > > It looks like the actual issue described for ORC in the
>>>> paper
>>>> >>>>>>> is that
>>>> >>>>>>> > it
>>>> >>>>>>> > > > has multiple sub-encodings in a batch. This is different
>>>> then
>>>> >>>>>>> the
>>>> >>>>>>> > design
>>>> >>>>>>> > > > proposed here where there is still fixed encoding per
>>>> page in
>>>> >>>>>>> parquet.
>>>> >>>>>>> > > > Given reasonably sized pages I don't think branch
>>>> >>>>>>> misprediction should
>>>> >>>>>>> > > be a
>>>> >>>>>>> > > > big issue for new encodings. I agree that we should be
>>>> >>>>>>> conservative in
>>>> >>>>>>> > > > general for adding new encodings.
>>>> >>>>>>> > > >
>>>> >>>>>>> > > >
>>>> >>>>>>> > > +1
>>>> >>>>>>> > >
>>>> >>>>>>> >
>>>> >>>>>>>
>>>> >>>>>>
>>>>
>>>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Curt Hagenlocher <cu...@hagenlocher.org>.
Ah, thanks! I missed that.
On Wed, Apr 29, 2026 at 4:24 PM Micah Kornfield <em...@gmail.com>
wrote:
> Hi Curt,
>
>> As part of the process of amending the Parquet format, perhaps it would
>> be a good idea for early implementations to generate sample files and
>> commit them to apache/parquet-testing: Apache Parquet Testing
>> <https://github.com/apache/parquet-testing> for other implementations to
>> leverage?
>
>
> It got dropped in the thread but does
> https://github.com/apache/parquet-testing/pull/100 address your concerns?
>
> Thanks,
> Micah
>
> On Wed, Apr 29, 2026 at 4:20 PM Curt Hagenlocher <cu...@hagenlocher.org>
> wrote:
>
>> As part of the process of amending the Parquet format, perhaps it would
>> be a good idea for early implementations to generate sample files and
>> commit them to apache/parquet-testing: Apache Parquet Testing
>> <https://github.com/apache/parquet-testing> for other implementations to
>> leverage?
>>
>> -Curt
>>
>> On Wed, Apr 29, 2026 at 4:11 PM PRATEEK GAUR <pr...@gmail.com> wrote:
>>
>>> Thanks Andrew and Micah for review feedback on the two PR's
>>> 1) (c++ arrow repo) https://github.com/apache/arrow/pull/48345/changes
>>> 2) (parquet-format repo)
>>> https://github.com/apache/parquet-format/pull/557
>>>
>>> I have addressed all (unless I missed something) comments on the two
>>> PR's.
>>>
>>> Best
>>> Prateek
>>>
>>> On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <pr...@gmail.com> wrote:
>>>
>>> > Thanks Andrew and Micah.
>>> >
>>> > `fair amount of feedback on at least the implementations`
>>> > For the c++ I have already started addressing the feedback, I should be
>>> > done with that Monday/Tuesday.
>>> > I think Vinoo too has been making good progress on the Java
>>> implementation.
>>> >
>>> > Best
>>> > Prateek
>>> >
>>> > On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <an...@gmail.com>
>>> > wrote:
>>> >
>>> >> Got it. Thank you for the clarification -- I will try and look into
>>> the
>>> >> spec and the Rust implementation[1] in this next week
>>> >>
>>> >> [1]: https://github.com/apache/arrow-rs/pull/9372
>>> >>
>>> >> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <
>>> emkornfield@gmail.com>
>>> >> wrote:
>>> >>
>>> >>> Hi Andrew,
>>> >>> I think there is a fair amount of feedback on at least the
>>> >>> implementations, typically I think we've waited till they are close
>>> to
>>> >>> mergeable before a final vote. Otherwise I agree we are very close.
>>> >>>
>>> >>> -Micah
>>> >>>
>>> >>> On Saturday, April 25, 2026, Andrew Lamb <an...@gmail.com>
>>> wrote:
>>> >>>
>>> >>>> Thanks Prateek,
>>> >>>>
>>> >>>> I think from this content it looks to me like we are ready to start
>>> a
>>> >>>> vote to explicitly accept ALP into Parquet
>>> >>>>
>>> >>>> Does anyone know of a reason we should postpone it for longer?
>>> >>>> Perhaps someone needs some more time to review?
>>> >>>>
>>> >>>> Andrew
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <pr...@gmail.com>
>>> >>>> wrote:
>>> >>>>
>>> >>>>> Hi team,
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> Hope everyone is doing well. I got a chance to work through all the
>>> >>>>> remaining feedback and update the spec doc. Here are the new
>>> artifacts
>>> >>>>>
>>> >>>>> 1) Spec document :
>>> >>>>>
>>> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
>>> >>>>>
>>> >>>>> 2) Spec document in parquet format repo :
>>> >>>>> https://github.com/apache/parquet-format/pull/557
>>> >>>>>
>>> >>>>> 3) Alp implementation in arrow c++ repo :
>>> >>>>> https://github.com/apache/arrow/pull/48345/changes
>>> >>>>>
>>> >>>>> 4) Alp implementation in parquet-java repo : Work for Vinoo and
>>> Julien
>>> >>>>> https://github.com/apache/parquet-java/pull/3397
>>> >>>>>
>>> >>>>> 5) PR with test and benchmarking artifacts in parquet-testing repo
>>> :
>>> >>>>> https://github.com/apache/parquet-testing/pull/100
>>> >>>>>
>>> >>>>>
>>> >>>>> And
>>> >>>>>
>>> >>>>>
>>> >>>>> - Go : Arnav just submitted an in progress implementation in Go.
>>> >>>>> https://github.com/apache/arrow-go/pull/704 (I haven't started
>>> >>>>> looking at it yet)
>>> >>>>> - Rust : I remember Andrew mentioned that this work is also in
>>> >>>>> progress (So 4 languages!)
>>> >>>>>
>>> >>>>>
>>> >>>>> *Arrow C++ implementation *
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> The PR is out and was also used by Antoine to report the numbers as
>>> >>>>> reported here. Micah and Konstantin have given 1 round of feedback
>>> >>>>> and I'm addressing them today. Please note that the default
>>> >>>>> optimization flag for compiling is O2 and not Q3. I got around 70%
>>> >>>>> performance improvement in the decoding speed when using the O3
>>> flag.
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> *Parqet-MR Java implementation (working with Vinoo and Julien) and
>>> **Cross
>>> >>>>> Language testing*
>>> >>>>>
>>> >>>>>
>>> >>>>> Let me know if you have any questions or feedback.
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> Now pasting some performance numbers
>>> >>>>>
>>> >>>>>
>>> >>>>> Table 1: C++ ALP Double Decode — Spotify Columns (Graviton 3, ARM
>>> >>>>> Neoverse V1)
>>> >>>>>
>>> >>>>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
>>> >>>>>
>>> >>>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │ Speedup │
>>> >>>>>
>>> >>>>> ├──────────────────┼──────────────┼──────────────┼─────────┤
>>> >>>>>
>>> >>>>> │ valence │ 3,155 │ 5,523 │ 1.75x │
>>> >>>>>
>>> >>>>> │ danceability │ 3,233 │ 5,685 │ 1.76x │
>>> >>>>>
>>> >>>>> │ energy │ 3,197 │ 5,652 │ 1.77x │
>>> >>>>>
>>> >>>>> │ loudness │ 3,186 │ 5,473 │ 1.72x │
>>> >>>>>
>>> >>>>> └──────────────────┴──────────────┴──────────────┴─────────┘
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <pr...@gmail.com>
>>> >>>>> wrote:
>>> >>>>>
>>> >>>>>> @Micah Kornfield <em...@gmail.com> : Got it.
>>> >>>>>>
>>> >>>>>> @Andrew Lamb <an...@gmail.com>
>>> >>>>>>
>>> >>>>>>
>>> >>>>>>> Do you think it would be good to start moving the spec
>>> development
>>> >>>>>>> into
>>> >>>>>>> markdown format, in preparation for finalizing it?
>>> >>>>>>>
>>> >>>>>>
>>> >>>>>> Yes I'll update the numbers for some of the examples I have in the
>>> >>>>>> spec based
>>> >>>>>> on the updated header size. Then we should be good to go for the
>>> >>>>>> markdown format.
>>> >>>>>>
>>> >>>>>> Thanks everyone!
>>> >>>>>>
>>> >>>>>>
>>> >>>>>>>
>>> >>>>>>> Andrew
>>> >>>>>>>
>>> >>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <prtkgaur@gmail.com
>>> >
>>> >>>>>>> wrote:
>>> >>>>>>>
>>> >>>>>>> > Hi team,
>>> >>>>>>> >
>>> >>>>>>> > 1) Andrew
>>> >>>>>>> >
>>> >>>>>>> > - Thanks for working on test files. My PR did add all the
>>> test
>>> >>>>>>> files I
>>> >>>>>>> > used to benchmark on datasets. Maybe we can club it
>>> together.
>>> >>>>>>> WIll also
>>> >>>>>>> > aid
>>> >>>>>>> > cross language testing
>>> >>>>>>> > - Kosta Tarasov working on Rust implementation. This is
>>> great.
>>> >>>>>>> Thanks
>>> >>>>>>> >
>>> >>>>>>> >
>>> >>>>>>> > 2) Antoine
>>> >>>>>>> >
>>> >>>>>>> > - Thanks a lot for reporting the numbers on AMD. Looks like
>>> you
>>> >>>>>>> are
>>> >>>>>>> > getting 8X the decoding performance of BSS. This is
>>> amazing!!.
>>> >>>>>>> > - Thanks for acknowledging the sampling design.
>>> >>>>>>> > - I agree with you on Fastlanes. In some crude experiments I
>>> >>>>>>> didn't get
>>> >>>>>>> > a good perf benefit from it on Graviton3 (but maybe there
>>> was
>>> >>>>>>> something
>>> >>>>>>> > wrong with my implementation).
>>> >>>>>>> > - Locking the 16bit exception encoding for the spec in this
>>> >>>>>>> case.
>>> >>>>>>> > - Awesome I think we have solved for all open questions
>>> minus
>>> >>>>>>> the
>>> >>>>>>> > version byte :). (will get back on this soon)
>>> >>>>>>> >
>>> >>>>>>> >
>>> >>>>>>> > 3) Micah
>>> >>>>>>> >
>>> >>>>>>> > - FastLanes : The current spec does allow for using FastLane
>>> >>>>>>> with the
>>> >>>>>>> > configurable enum value for layout. We should be able to
>>> inject
>>> >>>>>>> any
>>> >>>>>>> > layout
>>> >>>>>>> > in the current design.
>>> >>>>>>> >
>>> >>>>>>> >
>>> >>>>>>> > Working on resolving all remaining open comments on the spec
>>> this
>>> >>>>>>> week.
>>> >>>>>>> >
>>> >>>>>>> > Best
>>> >>>>>>> > Prateek
>>> >>>>>>> >
>>> >>>>>>> >
>>> >>>>>>> > On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <
>>> >>>>>>> stevel@cloudera.com>
>>> >>>>>>> > wrote:
>>> >>>>>>> >
>>> >>>>>>> > > On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
>>> >>>>>>> emkornfield@gmail.com>
>>> >>>>>>> > > wrote:
>>> >>>>>>> > >
>>> >>>>>>> > > >
>>> >>>>>>> > > >
>>> >>>>>>> > > > It looks like the actual issue described for ORC in the
>>> paper
>>> >>>>>>> is that
>>> >>>>>>> > it
>>> >>>>>>> > > > has multiple sub-encodings in a batch. This is different
>>> then
>>> >>>>>>> the
>>> >>>>>>> > design
>>> >>>>>>> > > > proposed here where there is still fixed encoding per page
>>> in
>>> >>>>>>> parquet.
>>> >>>>>>> > > > Given reasonably sized pages I don't think branch
>>> >>>>>>> misprediction should
>>> >>>>>>> > > be a
>>> >>>>>>> > > > big issue for new encodings. I agree that we should be
>>> >>>>>>> conservative in
>>> >>>>>>> > > > general for adding new encodings.
>>> >>>>>>> > > >
>>> >>>>>>> > > >
>>> >>>>>>> > > +1
>>> >>>>>>> > >
>>> >>>>>>> >
>>> >>>>>>>
>>> >>>>>>
>>>
>>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Micah Kornfield <em...@gmail.com>.
Hi Curt,
> As part of the process of amending the Parquet format, perhaps it would be
> a good idea for early implementations to generate sample files and commit
> them to apache/parquet-testing: Apache Parquet Testing
> <https://github.com/apache/parquet-testing> for other implementations to
> leverage?
It got dropped in the thread but does
https://github.com/apache/parquet-testing/pull/100 address your concerns?
Thanks,
Micah
On Wed, Apr 29, 2026 at 4:20 PM Curt Hagenlocher <cu...@hagenlocher.org>
wrote:
> As part of the process of amending the Parquet format, perhaps it would be
> a good idea for early implementations to generate sample files and commit
> them to apache/parquet-testing: Apache Parquet Testing
> <https://github.com/apache/parquet-testing> for other implementations to
> leverage?
>
> -Curt
>
> On Wed, Apr 29, 2026 at 4:11 PM PRATEEK GAUR <pr...@gmail.com> wrote:
>
>> Thanks Andrew and Micah for review feedback on the two PR's
>> 1) (c++ arrow repo) https://github.com/apache/arrow/pull/48345/changes
>> 2) (parquet-format repo)
>> https://github.com/apache/parquet-format/pull/557
>>
>> I have addressed all (unless I missed something) comments on the two PR's.
>>
>> Best
>> Prateek
>>
>> On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <pr...@gmail.com> wrote:
>>
>> > Thanks Andrew and Micah.
>> >
>> > `fair amount of feedback on at least the implementations`
>> > For the c++ I have already started addressing the feedback, I should be
>> > done with that Monday/Tuesday.
>> > I think Vinoo too has been making good progress on the Java
>> implementation.
>> >
>> > Best
>> > Prateek
>> >
>> > On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <an...@gmail.com>
>> > wrote:
>> >
>> >> Got it. Thank you for the clarification -- I will try and look into the
>> >> spec and the Rust implementation[1] in this next week
>> >>
>> >> [1]: https://github.com/apache/arrow-rs/pull/9372
>> >>
>> >> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <
>> emkornfield@gmail.com>
>> >> wrote:
>> >>
>> >>> Hi Andrew,
>> >>> I think there is a fair amount of feedback on at least the
>> >>> implementations, typically I think we've waited till they are close to
>> >>> mergeable before a final vote. Otherwise I agree we are very close.
>> >>>
>> >>> -Micah
>> >>>
>> >>> On Saturday, April 25, 2026, Andrew Lamb <an...@gmail.com>
>> wrote:
>> >>>
>> >>>> Thanks Prateek,
>> >>>>
>> >>>> I think from this content it looks to me like we are ready to start a
>> >>>> vote to explicitly accept ALP into Parquet
>> >>>>
>> >>>> Does anyone know of a reason we should postpone it for longer?
>> >>>> Perhaps someone needs some more time to review?
>> >>>>
>> >>>> Andrew
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <pr...@gmail.com>
>> >>>> wrote:
>> >>>>
>> >>>>> Hi team,
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> Hope everyone is doing well. I got a chance to work through all the
>> >>>>> remaining feedback and update the spec doc. Here are the new
>> artifacts
>> >>>>>
>> >>>>> 1) Spec document :
>> >>>>>
>> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
>> >>>>>
>> >>>>> 2) Spec document in parquet format repo :
>> >>>>> https://github.com/apache/parquet-format/pull/557
>> >>>>>
>> >>>>> 3) Alp implementation in arrow c++ repo :
>> >>>>> https://github.com/apache/arrow/pull/48345/changes
>> >>>>>
>> >>>>> 4) Alp implementation in parquet-java repo : Work for Vinoo and
>> Julien
>> >>>>> https://github.com/apache/parquet-java/pull/3397
>> >>>>>
>> >>>>> 5) PR with test and benchmarking artifacts in parquet-testing repo :
>> >>>>> https://github.com/apache/parquet-testing/pull/100
>> >>>>>
>> >>>>>
>> >>>>> And
>> >>>>>
>> >>>>>
>> >>>>> - Go : Arnav just submitted an in progress implementation in Go.
>> >>>>> https://github.com/apache/arrow-go/pull/704 (I haven't started
>> >>>>> looking at it yet)
>> >>>>> - Rust : I remember Andrew mentioned that this work is also in
>> >>>>> progress (So 4 languages!)
>> >>>>>
>> >>>>>
>> >>>>> *Arrow C++ implementation *
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> The PR is out and was also used by Antoine to report the numbers as
>> >>>>> reported here. Micah and Konstantin have given 1 round of feedback
>> >>>>> and I'm addressing them today. Please note that the default
>> >>>>> optimization flag for compiling is O2 and not Q3. I got around 70%
>> >>>>> performance improvement in the decoding speed when using the O3
>> flag.
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> *Parqet-MR Java implementation (working with Vinoo and Julien) and
>> **Cross
>> >>>>> Language testing*
>> >>>>>
>> >>>>>
>> >>>>> Let me know if you have any questions or feedback.
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> Now pasting some performance numbers
>> >>>>>
>> >>>>>
>> >>>>> Table 1: C++ ALP Double Decode — Spotify Columns (Graviton 3, ARM
>> >>>>> Neoverse V1)
>> >>>>>
>> >>>>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
>> >>>>>
>> >>>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │ Speedup │
>> >>>>>
>> >>>>> ├──────────────────┼──────────────┼──────────────┼─────────┤
>> >>>>>
>> >>>>> │ valence │ 3,155 │ 5,523 │ 1.75x │
>> >>>>>
>> >>>>> │ danceability │ 3,233 │ 5,685 │ 1.76x │
>> >>>>>
>> >>>>> │ energy │ 3,197 │ 5,652 │ 1.77x │
>> >>>>>
>> >>>>> │ loudness │ 3,186 │ 5,473 │ 1.72x │
>> >>>>>
>> >>>>> └──────────────────┴──────────────┴──────────────┴─────────┘
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <pr...@gmail.com>
>> >>>>> wrote:
>> >>>>>
>> >>>>>> @Micah Kornfield <em...@gmail.com> : Got it.
>> >>>>>>
>> >>>>>> @Andrew Lamb <an...@gmail.com>
>> >>>>>>
>> >>>>>>
>> >>>>>>> Do you think it would be good to start moving the spec development
>> >>>>>>> into
>> >>>>>>> markdown format, in preparation for finalizing it?
>> >>>>>>>
>> >>>>>>
>> >>>>>> Yes I'll update the numbers for some of the examples I have in the
>> >>>>>> spec based
>> >>>>>> on the updated header size. Then we should be good to go for the
>> >>>>>> markdown format.
>> >>>>>>
>> >>>>>> Thanks everyone!
>> >>>>>>
>> >>>>>>
>> >>>>>>>
>> >>>>>>> Andrew
>> >>>>>>>
>> >>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <pr...@gmail.com>
>> >>>>>>> wrote:
>> >>>>>>>
>> >>>>>>> > Hi team,
>> >>>>>>> >
>> >>>>>>> > 1) Andrew
>> >>>>>>> >
>> >>>>>>> > - Thanks for working on test files. My PR did add all the
>> test
>> >>>>>>> files I
>> >>>>>>> > used to benchmark on datasets. Maybe we can club it together.
>> >>>>>>> WIll also
>> >>>>>>> > aid
>> >>>>>>> > cross language testing
>> >>>>>>> > - Kosta Tarasov working on Rust implementation. This is
>> great.
>> >>>>>>> Thanks
>> >>>>>>> >
>> >>>>>>> >
>> >>>>>>> > 2) Antoine
>> >>>>>>> >
>> >>>>>>> > - Thanks a lot for reporting the numbers on AMD. Looks like
>> you
>> >>>>>>> are
>> >>>>>>> > getting 8X the decoding performance of BSS. This is
>> amazing!!.
>> >>>>>>> > - Thanks for acknowledging the sampling design.
>> >>>>>>> > - I agree with you on Fastlanes. In some crude experiments I
>> >>>>>>> didn't get
>> >>>>>>> > a good perf benefit from it on Graviton3 (but maybe there was
>> >>>>>>> something
>> >>>>>>> > wrong with my implementation).
>> >>>>>>> > - Locking the 16bit exception encoding for the spec in this
>> >>>>>>> case.
>> >>>>>>> > - Awesome I think we have solved for all open questions minus
>> >>>>>>> the
>> >>>>>>> > version byte :). (will get back on this soon)
>> >>>>>>> >
>> >>>>>>> >
>> >>>>>>> > 3) Micah
>> >>>>>>> >
>> >>>>>>> > - FastLanes : The current spec does allow for using FastLane
>> >>>>>>> with the
>> >>>>>>> > configurable enum value for layout. We should be able to
>> inject
>> >>>>>>> any
>> >>>>>>> > layout
>> >>>>>>> > in the current design.
>> >>>>>>> >
>> >>>>>>> >
>> >>>>>>> > Working on resolving all remaining open comments on the spec
>> this
>> >>>>>>> week.
>> >>>>>>> >
>> >>>>>>> > Best
>> >>>>>>> > Prateek
>> >>>>>>> >
>> >>>>>>> >
>> >>>>>>> > On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <
>> >>>>>>> stevel@cloudera.com>
>> >>>>>>> > wrote:
>> >>>>>>> >
>> >>>>>>> > > On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
>> >>>>>>> emkornfield@gmail.com>
>> >>>>>>> > > wrote:
>> >>>>>>> > >
>> >>>>>>> > > >
>> >>>>>>> > > >
>> >>>>>>> > > > It looks like the actual issue described for ORC in the
>> paper
>> >>>>>>> is that
>> >>>>>>> > it
>> >>>>>>> > > > has multiple sub-encodings in a batch. This is different
>> then
>> >>>>>>> the
>> >>>>>>> > design
>> >>>>>>> > > > proposed here where there is still fixed encoding per page
>> in
>> >>>>>>> parquet.
>> >>>>>>> > > > Given reasonably sized pages I don't think branch
>> >>>>>>> misprediction should
>> >>>>>>> > > be a
>> >>>>>>> > > > big issue for new encodings. I agree that we should be
>> >>>>>>> conservative in
>> >>>>>>> > > > general for adding new encodings.
>> >>>>>>> > > >
>> >>>>>>> > > >
>> >>>>>>> > > +1
>> >>>>>>> > >
>> >>>>>>> >
>> >>>>>>>
>> >>>>>>
>>
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Curt Hagenlocher <cu...@hagenlocher.org>.
As part of the process of amending the Parquet format, perhaps it would be
a good idea for early implementations to generate sample files and commit
them to apache/parquet-testing: Apache Parquet Testing
<https://github.com/apache/parquet-testing> for other implementations to
leverage?
-Curt
On Wed, Apr 29, 2026 at 4:11 PM PRATEEK GAUR <pr...@gmail.com> wrote:
> Thanks Andrew and Micah for review feedback on the two PR's
> 1) (c++ arrow repo) https://github.com/apache/arrow/pull/48345/changes
> 2) (parquet-format repo) https://github.com/apache/parquet-format/pull/557
>
> I have addressed all (unless I missed something) comments on the two PR's.
>
> Best
> Prateek
>
> On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <pr...@gmail.com> wrote:
>
> > Thanks Andrew and Micah.
> >
> > `fair amount of feedback on at least the implementations`
> > For the c++ I have already started addressing the feedback, I should be
> > done with that Monday/Tuesday.
> > I think Vinoo too has been making good progress on the Java
> implementation.
> >
> > Best
> > Prateek
> >
> > On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <an...@gmail.com>
> > wrote:
> >
> >> Got it. Thank you for the clarification -- I will try and look into the
> >> spec and the Rust implementation[1] in this next week
> >>
> >> [1]: https://github.com/apache/arrow-rs/pull/9372
> >>
> >> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <emkornfield@gmail.com
> >
> >> wrote:
> >>
> >>> Hi Andrew,
> >>> I think there is a fair amount of feedback on at least the
> >>> implementations, typically I think we've waited till they are close to
> >>> mergeable before a final vote. Otherwise I agree we are very close.
> >>>
> >>> -Micah
> >>>
> >>> On Saturday, April 25, 2026, Andrew Lamb <an...@gmail.com>
> wrote:
> >>>
> >>>> Thanks Prateek,
> >>>>
> >>>> I think from this content it looks to me like we are ready to start a
> >>>> vote to explicitly accept ALP into Parquet
> >>>>
> >>>> Does anyone know of a reason we should postpone it for longer?
> >>>> Perhaps someone needs some more time to review?
> >>>>
> >>>> Andrew
> >>>>
> >>>>
> >>>>
> >>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <pr...@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> Hi team,
> >>>>>
> >>>>>
> >>>>>
> >>>>> Hope everyone is doing well. I got a chance to work through all the
> >>>>> remaining feedback and update the spec doc. Here are the new
> artifacts
> >>>>>
> >>>>> 1) Spec document :
> >>>>>
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
> >>>>>
> >>>>> 2) Spec document in parquet format repo :
> >>>>> https://github.com/apache/parquet-format/pull/557
> >>>>>
> >>>>> 3) Alp implementation in arrow c++ repo :
> >>>>> https://github.com/apache/arrow/pull/48345/changes
> >>>>>
> >>>>> 4) Alp implementation in parquet-java repo : Work for Vinoo and
> Julien
> >>>>> https://github.com/apache/parquet-java/pull/3397
> >>>>>
> >>>>> 5) PR with test and benchmarking artifacts in parquet-testing repo :
> >>>>> https://github.com/apache/parquet-testing/pull/100
> >>>>>
> >>>>>
> >>>>> And
> >>>>>
> >>>>>
> >>>>> - Go : Arnav just submitted an in progress implementation in Go.
> >>>>> https://github.com/apache/arrow-go/pull/704 (I haven't started
> >>>>> looking at it yet)
> >>>>> - Rust : I remember Andrew mentioned that this work is also in
> >>>>> progress (So 4 languages!)
> >>>>>
> >>>>>
> >>>>> *Arrow C++ implementation *
> >>>>>
> >>>>>
> >>>>>
> >>>>> The PR is out and was also used by Antoine to report the numbers as
> >>>>> reported here. Micah and Konstantin have given 1 round of feedback
> >>>>> and I'm addressing them today. Please note that the default
> >>>>> optimization flag for compiling is O2 and not Q3. I got around 70%
> >>>>> performance improvement in the decoding speed when using the O3 flag.
> >>>>>
> >>>>>
> >>>>>
> >>>>> *Parqet-MR Java implementation (working with Vinoo and Julien) and
> **Cross
> >>>>> Language testing*
> >>>>>
> >>>>>
> >>>>> Let me know if you have any questions or feedback.
> >>>>>
> >>>>>
> >>>>>
> >>>>> Now pasting some performance numbers
> >>>>>
> >>>>>
> >>>>> Table 1: C++ ALP Double Decode — Spotify Columns (Graviton 3, ARM
> >>>>> Neoverse V1)
> >>>>>
> >>>>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
> >>>>>
> >>>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │ Speedup │
> >>>>>
> >>>>> ├──────────────────┼──────────────┼──────────────┼─────────┤
> >>>>>
> >>>>> │ valence │ 3,155 │ 5,523 │ 1.75x │
> >>>>>
> >>>>> │ danceability │ 3,233 │ 5,685 │ 1.76x │
> >>>>>
> >>>>> │ energy │ 3,197 │ 5,652 │ 1.77x │
> >>>>>
> >>>>> │ loudness │ 3,186 │ 5,473 │ 1.72x │
> >>>>>
> >>>>> └──────────────────┴──────────────┴──────────────┴─────────┘
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <pr...@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>>> @Micah Kornfield <em...@gmail.com> : Got it.
> >>>>>>
> >>>>>> @Andrew Lamb <an...@gmail.com>
> >>>>>>
> >>>>>>
> >>>>>>> Do you think it would be good to start moving the spec development
> >>>>>>> into
> >>>>>>> markdown format, in preparation for finalizing it?
> >>>>>>>
> >>>>>>
> >>>>>> Yes I'll update the numbers for some of the examples I have in the
> >>>>>> spec based
> >>>>>> on the updated header size. Then we should be good to go for the
> >>>>>> markdown format.
> >>>>>>
> >>>>>> Thanks everyone!
> >>>>>>
> >>>>>>
> >>>>>>>
> >>>>>>> Andrew
> >>>>>>>
> >>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <pr...@gmail.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>> > Hi team,
> >>>>>>> >
> >>>>>>> > 1) Andrew
> >>>>>>> >
> >>>>>>> > - Thanks for working on test files. My PR did add all the test
> >>>>>>> files I
> >>>>>>> > used to benchmark on datasets. Maybe we can club it together.
> >>>>>>> WIll also
> >>>>>>> > aid
> >>>>>>> > cross language testing
> >>>>>>> > - Kosta Tarasov working on Rust implementation. This is
> great.
> >>>>>>> Thanks
> >>>>>>> >
> >>>>>>> >
> >>>>>>> > 2) Antoine
> >>>>>>> >
> >>>>>>> > - Thanks a lot for reporting the numbers on AMD. Looks like
> you
> >>>>>>> are
> >>>>>>> > getting 8X the decoding performance of BSS. This is amazing!!.
> >>>>>>> > - Thanks for acknowledging the sampling design.
> >>>>>>> > - I agree with you on Fastlanes. In some crude experiments I
> >>>>>>> didn't get
> >>>>>>> > a good perf benefit from it on Graviton3 (but maybe there was
> >>>>>>> something
> >>>>>>> > wrong with my implementation).
> >>>>>>> > - Locking the 16bit exception encoding for the spec in this
> >>>>>>> case.
> >>>>>>> > - Awesome I think we have solved for all open questions minus
> >>>>>>> the
> >>>>>>> > version byte :). (will get back on this soon)
> >>>>>>> >
> >>>>>>> >
> >>>>>>> > 3) Micah
> >>>>>>> >
> >>>>>>> > - FastLanes : The current spec does allow for using FastLane
> >>>>>>> with the
> >>>>>>> > configurable enum value for layout. We should be able to
> inject
> >>>>>>> any
> >>>>>>> > layout
> >>>>>>> > in the current design.
> >>>>>>> >
> >>>>>>> >
> >>>>>>> > Working on resolving all remaining open comments on the spec this
> >>>>>>> week.
> >>>>>>> >
> >>>>>>> > Best
> >>>>>>> > Prateek
> >>>>>>> >
> >>>>>>> >
> >>>>>>> > On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <
> >>>>>>> stevel@cloudera.com>
> >>>>>>> > wrote:
> >>>>>>> >
> >>>>>>> > > On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
> >>>>>>> emkornfield@gmail.com>
> >>>>>>> > > wrote:
> >>>>>>> > >
> >>>>>>> > > >
> >>>>>>> > > >
> >>>>>>> > > > It looks like the actual issue described for ORC in the paper
> >>>>>>> is that
> >>>>>>> > it
> >>>>>>> > > > has multiple sub-encodings in a batch. This is different
> then
> >>>>>>> the
> >>>>>>> > design
> >>>>>>> > > > proposed here where there is still fixed encoding per page in
> >>>>>>> parquet.
> >>>>>>> > > > Given reasonably sized pages I don't think branch
> >>>>>>> misprediction should
> >>>>>>> > > be a
> >>>>>>> > > > big issue for new encodings. I agree that we should be
> >>>>>>> conservative in
> >>>>>>> > > > general for adding new encodings.
> >>>>>>> > > >
> >>>>>>> > > >
> >>>>>>> > > +1
> >>>>>>> > >
> >>>>>>> >
> >>>>>>>
> >>>>>>
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Antoine Pitrou <an...@python.org>.
Le 12/11/2026 à 13:42, Gang Wu a écrit :
>
> I'm trying to improve the ALP implementations on the Parquet C++ side:
> https://github.com/apache/arrow/pull/55449. So I'm really interested
> if you have any document on performance comparison among different
> implementations.
When reviewing this PR, the question came up of why use such weird and
ad-hoc sampling schemes instead of actual pseudo-random sampling using a
good and fast PRNG.
I would be curious if this has been experimented with already
(pseudo-random sampling).
Regards
Antoine.
>
> Thanks!
> Gang
>
> On Thu, Nov 12, 2026 at 6:50 PM Gunnar Morling via dev <
> dev@parquet.apache.org> wrote:
>
>> Hey all,
>>
>> We've added ALP encoding and decoding support to Hardwood too [1], [2].
>> Interop with parquet-java's implementation looks good: Hardwood decodes
>> pages written by parquet-java's ALP writer bit-exactly and vice versa.
>>
>> Perhaps interesting: on the encoder side, we build the preset of a column
>> chunk from eight sample vectors distributed across the entire chunk (easy
>> in our case, as Hardwood's writer buffers whole chunks), whereas, IIUC,
>> parquet-java only samples within the first ~110K values of a chunk.
>> Depending on the data distribution, this can yield a more efficient
>> encoding, for instance when a high-cardinality column changes character
>> midway, e.g. moving from two to four decimal places (about 2.8× smaller in
>> a synthetic test with 1M doubles).
>>
>> Best,
>>
>> --Gunnar
Re: [Parquet] ALP Encoding for Floating point data
Posted by PRATEEK GAUR <pr...@gmail.com>.
Hi Antoine,
My initial assumption for this design was to aid faster sample
creation in a long stream of data. Thinking was
- Jump to a location in the stream
- Select some random values close to this location (within the 1024
proximity limit)
- Find the next location (equidistant) to jump to
However, perhaps that isn't needed as the page is small enough to go for
this design.
Random set of values might end up working equally well. I can give this a
try.
Best
Prateek
Re: [Parquet] ALP Encoding for Floating point data
Posted by PRATEEK GAUR <pr...@gmail.com>.
Hi Antoine,
My initial assumption for this design was to aid faster sample
creation in a long stream of data. Thinking was
- Jump to a location in the stream
- Select some random values close to this location (within the 1024
proximity limit)
- Find the next location (equidistant) to jump to
However, perhaps that isn't needed as the page is small enough to go for
this design.
Random set of values might end up working equally well. I can give this a
try.
Best
Prateek
Re: [Parquet] ALP Encoding for Floating point data
Posted by Antoine Pitrou <an...@python.org>.
Hi Prateek!
Le 13/11/2026 à 07:44, PRATEEK GAUR a écrit :
>
>> I would be curious if this has been experimented with already
>> (pseudo-random sampling).
>
> Thanks, Antoine. Your comment prompted me to revisit the Arrow C++ sampler
> and experiment
> with a simpler approach using fixed-seed pseudo-random sampling.
> - Pick evenly spaced value vectors across the page. This ensures values are
> picked from different
> parts of the input.
> - From each vector, pick 32 (fixed) pseudo-random values using a fixed seed
> so the result remains
> deterministic. This ensures a statistically meaningful sample while
> further reducing the sample
> size (to reduce CPU compute).
>
> Trying this on the dataset I didn't see any meaningful regression in
> compression ratio or
> decoding speed. So we can simplify the implementation.
Thank you, but my proposal goes a bit further: how about dropping the
"evenly spaced value vectors" and instead just sample the desired number
of values (for example 256) from the page?
Some PRNGs are extremely fast and I doubt the cost of sampling would be
significant compared to the actual cost of deriving ALP parameters from
the samples?
Regards
Antoine.
Re: [Parquet] ALP Encoding for Floating point data
Posted by PRATEEK GAUR <pr...@gmail.com>.
Hi Gunnar, Gang, Antoine,
> Interop with parquet-java's implementation looks good: Hardwood decodes
> pages written by parquet-java's ALP writer bit-exactly and vice versa.
Thanks, Gunnar. That is great to hear.
> on the encoder side, we build the preset of a column chunk from eight
> sample vectors distributed across the entire chunk [...] whereas, IIUC,
> parquet-java only samples within the first ~110K values of a chunk.
Yes that's right about parquet-java: its writer encodes one vector at a
time and fixes the preset
after sampling roughly the first 110K values.
Since Hardwood buffers the entire column chunk, distributing eight sample
vectors across it
makes sense. Arrow C++ instead draws samples from across each data page and
builds a fresh
preset for every page. This lets it adapt between pages, at the cost of
repeating preset
selection.
> I would be curious if this has been experimented with already
> (pseudo-random sampling).
Thanks, Antoine. Your comment prompted me to revisit the Arrow C++ sampler
and experiment
with a simpler approach using fixed-seed pseudo-random sampling.
- Pick evenly spaced value vectors across the page. This ensures values are
picked from different
parts of the input.
- From each vector, pick 32 (fixed) pseudo-random values using a fixed seed
so the result remains
deterministic. This ensures a statistically meaningful sample while
further reducing the sample
size (to reduce CPU compute).
Trying this on the dataset I didn't see any meaningful regression in
compression ratio or
decoding speed. So we can simplify the implementation.
> I'm trying to improve the ALP implementations on the Parquet C++ side:
> https://github.com/apache/arrow/pull/55449.
Thanks Gang, I haven't had a chance to look at your PR yet, but I'll get to
it soon. Since it
touches the same sampling code, I'm happy to compare notes once I've read
it.
> Hardwood decodes 5–6× faster than parquet-java
Thanks for this data point. Is fusing the one helping with speed up?
cc @Vinoo Ganesh <vi...@gmail.com> .
> encodes a page 2–4× faster once both presets exist.
That makes sense. As you noted, Hardwood evaluates the preset pairs using a
smaller sample,
while parquet-java evaluates all 1,024 values in each vector. That reduces
encoding work even
after the preset has been built.
Best,
Prateek
On Thu, Nov 12, 2026 at 9:42 AM Gunnar Morling via dev <
dev@parquet.apache.org> wrote:
> Hey Gang,
>
> Yeah, agreed, I think it's more similar in the sense that we take the
> samples for the preset from across the whole chunk instead of only from its
> first segment. AFAIK, DuckDB also resamples and then uses dedicated presets
> for specific segments, which can yield even better results but also is more
> costly.
>
> As for a perf comparison, I have a quick informal (LLM-coded)
> microbenchmark of parquet-java's and Hardwood's ALP decoders and encoders
> in this draft PR [1]. In that benchmark, in a Docker container on my
> laptop, Hardwood decodes 5–6× faster than parquet-java and encodes a page
> 2–4× faster once both presets exist (pages purely in memory, no
> compression, etc.). Based on some quick profiling, IIUC, it mostly comes
> down to bit unpacking on decode (generic 8-at-a-time BytePacker plus a
> separate decode pass vs. one fused loop with a single 8-byte load per
> value), and on encode to parquet-java evaluating each preset pair on the
> full vector rather than a sample.
>
> Numbers may look different elsewhere, so I'd be curious to learn about your
> results, also if there's something to do better in the benchmark itself,
> etc.
>
> Best,
>
> --Gunnar
>
> [1] https://github.com/hardwood-hq/hardwood/pull/1775
>
>
> On Thu, 12 Nov 2026 at 13:42, Gang Wu <us...@gmail.com> wrote:
>
> > Thanks Gunnar for sharing this!
> >
> > I think the encoder approach used by Hardwood looks more similar to the
> ALP
> > paper where all 102,400 values are buffered to decide best presets and
> then
> > encode every vector based on those presets.
> >
> > I'm trying to improve the ALP implementations on the Parquet C++ side:
> > https://github.com/apache/arrow/pull/55449. So I'm really interested
> > if you have any document on performance comparison among different
> > implementations.
> >
> > Thanks!
> > Gang
> >
> > On Thu, Nov 12, 2026 at 6:50 PM Gunnar Morling via dev <
> > dev@parquet.apache.org> wrote:
> >
> >> Hey all,
> >>
> >> We've added ALP encoding and decoding support to Hardwood too [1], [2].
> >> Interop with parquet-java's implementation looks good: Hardwood decodes
> >> pages written by parquet-java's ALP writer bit-exactly and vice versa.
> >>
> >> Perhaps interesting: on the encoder side, we build the preset of a
> column
> >> chunk from eight sample vectors distributed across the entire chunk
> (easy
> >> in our case, as Hardwood's writer buffers whole chunks), whereas, IIUC,
> >> parquet-java only samples within the first ~110K values of a chunk.
> >> Depending on the data distribution, this can yield a more efficient
> >> encoding, for instance when a high-cardinality column changes character
> >> midway, e.g. moving from two to four decimal places (about 2.8× smaller
> in
> >> a synthetic test with 1M doubles).
> >>
> >> Best,
> >>
> >> --Gunnar
> >>
> >> [1] https://github.com/hardwood-hq/hardwood/pull/1496
> >> [2] https://github.com/hardwood-hq/hardwood/pull/1769
> >>
> >>
> >> On Thu, 12 Nov 2026 at 07:01, PRATEEK GAUR <pr...@gmail.com> wrote:
> >>
> >> > Thanks everyone!
> >> >
> >> > On Mon, Nov 9, 2026 at 11:58 AM Vinoo Ganesh <vi...@gmail.com>
> >> > wrote:
> >> >
> >> > > Thanks everyone!
> >> > >
> >> > > Thanks,
> >> > > Vinoo Ganesh | vinoo.ganesh@gmail.com
> >> > >
> >> > > <Vi...@gmail.com>
> >> > >
> >> > >
> >> > > On Mon, Nov 9, 2026 at 2:49 PM Andrew Lamb <an...@gmail.com>
> >> > wrote:
> >> > >
> >> > > > The Java implementation was just merged[1]!
> >> > > >
> >> > > > Thank you Gang, Vinoo, Prateek and everyone who helped push that
> one
> >> > > along
> >> > > >
> >> > > > Andrew
> >> > > >
> >> > > > [1]:
> >> > https://github.com/apache/parquet-java/pull/3397#event-35545451335
> >> > > >
> >> > > > On Tue, Oct 13, 2026 at 12:52 PM PRATEEK GAUR <prtkgaur@gmail.com
> >
> >> > > wrote:
> >> > > >
> >> > > > > Hi team,
> >> > > > >
> >> > > > > Quick update re: the C++ implementation.
> >> > > > > Thanks to all the reviewers, we merged the C++ implementation
> >> today.
> >> > > > > There are a few follow-up items that I'll address in a
> subsequent
> >> PR.
> >> > > > >
> >> > > > > Thanks
> >> > > > > Prateek
> >> > > > >
> >> > > > > On Sun, Sep 13, 2026 at 6:54 PM Vinoo Ganesh <
> >> vinoo.ganesh@gmail.com
> >> > >
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Quick update re: the Java impl. Thanks for the feedback, on
> >> > > > > > https://github.com/apache/parquet-java/pull/3397, Russel! I
> >> pulled
> >> > > > > > https://github.com/apache/parquet-java/pull/3791 into its own
> >> PR
> >> > > > (thanks
> >> > > > > > for the quick approval too, Gang). Assuming no objections, I
> >> think
> >> > we
> >> > > > can
> >> > > > > > merge #3791, and then go ahead of #3397.
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > > Vinoo Ganesh | vinoo.ganesh@gmail.com
> >> > > > > >
> >> > > > > > <Vi...@gmail.com>
> >> > > > > >
> >> > > > > >
> >> > > > > > On Tue, Sep 8, 2026 at 10:54 AM PRATEEK GAUR <
> >> prtkgaur@gmail.com>
> >> > > > wrote:
> >> > > > > >
> >> > > > > > > [Rust] Awesome thanks Andrew, Kostas and Devin.
> >> > > > > > > [Java] Thanks Gang for the feedback on Java PR.
> >> > > > > > > [C++] Thanks Gang for another round of feedback on the C++
> >> one.
> >> > > > > > > [Go] Arnav and I will be working on the golang impl this
> week.
> >> > > > > > >
> >> > > > > > > This is great progress all around.
> >> > > > > > >
> >> > > > > > > On Sat, Sep 5, 2026 at 1:26 PM Vinoo Ganesh <
> >> > > vinoo.ganesh@gmail.com>
> >> > > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > Thanks all - I'm addressing Gang's comments on
> >> > > > > > > > https://github.com/apache/parquet-java/pull/3397 now, so
> >> > > hopefully
> >> > > > > > well
> >> > > > > > > > also be in a good state for merging the java
> implementation
> >> > soon.
> >> > > > > > > >
> >> > > > > > > > Thanks,
> >> > > > > > > > Vinoo Ganesh | vinoo.ganesh@gmail.com
> >> > > > > > > >
> >> > > > > > > > <Vi...@gmail.com>
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > On Sat, Sep 5, 2026 at 8:56 AM Andrew Lamb <
> >> > > andrewlamb11@gmail.com
> >> > > > >
> >> > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > We just merged the Rust implementation[1] from Kosta and
> >> > Devan.
> >> > > > > With
> >> > > > > > > the
> >> > > > > > > > > imminent release of Parquet Format 2.14.0 I think we'll
> be
> >> > well
> >> > > > on
> >> > > > > > our
> >> > > > > > > > way.
> >> > > > > > > > >
> >> > > > > > > > > Thanks again for pushing this along,
> >> > > > > > > > > Andrew
> >> > > > > > > > >
> >> > > > > > > > > [1]: https://github.com/apache/arrow-rs/pull/9372
> >> > > > > > > > >
> >> > > > > > > > > On Fri, Sep 4, 2026 at 5:42 PM PRATEEK GAUR <
> >> > > prtkgaur@gmail.com>
> >> > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > > Hi team,
> >> > > > > > > > > >
> >> > > > > > > > > > Thanks Micah for multiple rounds of review and thanks
> >> Gang
> >> > > for
> >> > > > > > > taking a
> >> > > > > > > > > > look at the PR and giving feedback. I've addressed all
> >> > > comments
> >> > > > > in
> >> > > > > > > the
> >> > > > > > > > > PR.
> >> > > > > > > > > > Looking forward to the next round of feedback (or
> >> > approval).
> >> > > > > Would
> >> > > > > > > like
> >> > > > > > > > > to
> >> > > > > > > > > > get it in soon.
> >> > > > > > > > > >
> >> > > > > > > > > > https://github.com/apache/arrow/pull/48345/changes
> >> > > > > > > > > >
> >> > > > > > > > > > With this done and after talking to Arnav, I plan to
> >> spend
> >> > > some
> >> > > > > > time
> >> > > > > > > on
> >> > > > > > > > > the
> >> > > > > > > > > > ALP implementation in the arrow-go repository helping
> >> > spread
> >> > > > the
> >> > > > > > > > support
> >> > > > > > > > > to
> >> > > > > > > > > > more languages.
> >> > > > > > > > > >
> >> > > > > > > > > > Best
> >> > > > > > > > > > Prateek
> >> > > > > > > > > >
> >> > > > > > > > > > On Sun, Aug 2, 2026 at 10:42 PM PRATEEK GAUR <
> >> > > > prtkgaur@gmail.com
> >> > > > > >
> >> > > > > > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > Thanks Divjot for bringing it up.
> >> > > > > > > > > > >
> >> > > > > > > > > > > I was under the same impression as Russell and
> Andrew
> >> > that
> >> > > > we'd
> >> > > > > > > have
> >> > > > > > > > it
> >> > > > > > > > > > > flag gated.
> >> > > > > > > > > > > With the new versioning proposal 'preview' might be
> a
> >> > good
> >> > > > tag
> >> > > > > > for
> >> > > > > > > > it.
> >> > > > > > > > > > >
> >> > > > > > > > > > > Best
> >> > > > > > > > > > > Prateek
> >> > > > > > > > > > >
> >> > > > > > > > > > > On Sat, Aug 1, 2026 at 12:39 AM Andrew Lamb <
> >> > > > > > > andrewlamb11@gmail.com>
> >> > > > > > > > > > > wrote:
> >> > > > > > > > > > >
> >> > > > > > > > > > >> Div: Here is a proposal to mark ALP in Preview: [1]
> >> > > (using a
> >> > > > > > > comment
> >> > > > > > > > > in
> >> > > > > > > > > > >> the
> >> > > > > > > > > > >> spec section).
> >> > > > > > > > > > >>
> >> > > > > > > > > > >> Russel: That is my understanding as well.
> >> > > > > > > > > > >>
> >> > > > > > > > > > >> Hopefully we are able to formalize the versioning
> >> > process
> >> > > > > > shortly
> >> > > > > > > so
> >> > > > > > > > > we
> >> > > > > > > > > > >> can
> >> > > > > > > > > > >> formalize the notion of "preview"
> >> > > > > > > > > > >>
> >> > > > > > > > > > >> Andrew
> >> > > > > > > > > > >>
> >> > > > > > > > > > >>
> >> > > > > > > > > > >> [1]:
> >> https://github.com/apache/parquet-format/pull/604
> >> > > > > > > > > > >>
> >> > > > > > > > > > >> On Fri, Jul 31, 2026 at 11:15 AM Russell Spitzer <
> >> > > > > > > > > > >> russell.spitzer@gmail.com>
> >> > > > > > > > > > >> wrote:
> >> > > > > > > > > > >>
> >> > > > > > > > > > >> > I thought we had agreed it would be merged, and
> >> > features
> >> > > > > like
> >> > > > > > > this
> >> > > > > > > > > > would
> >> > > > > > > > > > >> > be feature flag gated. So a V-Old writer could
> >> always
> >> > > opt
> >> > > > in
> >> > > > > > to
> >> > > > > > > > > using
> >> > > > > > > > > > >> this
> >> > > > > > > > > > >> > encoding but that must be a conscious effort if
> you
> >> > are
> >> > > > not
> >> > > > > > > using
> >> > > > > > > > > > VNext
> >> > > > > > > > > > >> of
> >> > > > > > > > > > >> > Parquet. If you are on V-Next the writer can
> >> choose to
> >> > > use
> >> > > > > ALP
> >> > > > > > > > > > whenever
> >> > > > > > > > > > >> it
> >> > > > > > > > > > >> > likes without user intervention.
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >> > On Fri, Jul 31, 2026 at 2:20 AM Divjot Arora via
> >> dev <
> >> > > > > > > > > > >> > dev@parquet.apache.org> wrote:
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >> >> Yes, that’s my understanding as well. However,
> as
> >> you
> >> > > > > noted,
> >> > > > > > > the
> >> > > > > > > > > > >> >> versioning
> >> > > > > > > > > > >> >> discussion is still in early stages and hasn’t
> >> > > prescribed
> >> > > > > > > > concrete
> >> > > > > > > > > > >> >> mechanisms yet. If ALP goes into parquet-format
> >> > 2.14.0,
> >> > > > > > there’s
> >> > > > > > > > no
> >> > > > > > > > > > >> >> indication that it’s in preview or that writers
> >> > > shouldn’t
> >> > > > > use
> >> > > > > > > it
> >> > > > > > > > by
> >> > > > > > > > > > >> >> default
> >> > > > > > > > > > >> >> yet. Given that we’ve held off on merging other
> >> > > > > incompatible
> >> > > > > > > > > changes
> >> > > > > > > > > > >> (e.g
> >> > > > > > > > > > >> >> removal of path_in_schema) until the versioning
> >> > > > discussion
> >> > > > > > > > closes,
> >> > > > > > > > > my
> >> > > > > > > > > > >> >> preference would be to leave ALP un-merged for
> the
> >> > time
> >> > > > > being
> >> > > > > > > as
> >> > > > > > > > > > well.
> >> > > > > > > > > > >> >>
> >> > > > > > > > > > >> >> — Div
> >> > > > > > > > > > >> >>
> >> > > > > > > > > > >> >> On Fri, Jul 31, 2026 at 08:59 Andrew Lamb <
> >> > > > > > > > andrewlamb11@gmail.com>
> >> > > > > > > > > > >> wrote:
> >> > > > > > > > > > >> >>
> >> > > > > > > > > > >> >> > After re-reading Julien's Versioning Proposal,
> >> my
> >> > > > > > > understanding
> >> > > > > > > > > is
> >> > > > > > > > > > >> that
> >> > > > > > > > > > >> >> if
> >> > > > > > > > > > >> >> > we adopt that system, we will somehow mark ALP
> >> in
> >> > > > > "Preview"
> >> > > > > > > in
> >> > > > > > > > > > >> >> > parquet-format for release Version 3.
> >> > > > > > > > > > >> >> >
> >> > > > > > > > > > >> >> > The versioning document doesn't seem to
> >> proscribe
> >> > the
> >> > > > > > > > particular
> >> > > > > > > > > > >> method
> >> > > > > > > > > > >> >> > for annotating "preview" features -- I think
> >> > clearly
> >> > > > > > marking
> >> > > > > > > > them
> >> > > > > > > > > > as
> >> > > > > > > > > > >> >> such
> >> > > > > > > > > > >> >> > in parquet.thrift as well as reflecting that
> on
> >> the
> >> > > > > > > > > > >> >> >
> >> > > https://parquet.apache.org/docs/file-format/versions/
> >> > > > > page
> >> > > > > > > > > somehow
> >> > > > > > > > > > >> >> would
> >> > > > > > > > > > >> >> > be adequate.
> >> > > > > > > > > > >> >> >
> >> > > > > > > > > > >> >> > Andrew
> >> > > > > > > > > > >> >> >
> >> > > > > > > > > > >> >> > On Thu, Jul 30, 2026 at 11:50 AM Andrew Lamb <
> >> > > > > > > > > > andrewlamb11@gmail.com
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >> >> > wrote:
> >> > > > > > > > > > >> >> >
> >> > > > > > > > > > >> >> >> In my mind ALP is in the same position as
> other
> >> > > > > encodings
> >> > > > > > > that
> >> > > > > > > > > > were
> >> > > > > > > > > > >> >> added
> >> > > > > > > > > > >> >> >> recently[1] (such as BYTE_STREAM_SPLIT for
> non
> >> > > > floating
> >> > > > > > > point
> >> > > > > > > > > > types)
> >> > > > > > > > > > >> >> >>
> >> > > > > > > > > > >> >> >> That is to say it will be available in the
> >> latest
> >> > > open
> >> > > > > > > source
> >> > > > > > > > > > >> writers
> >> > > > > > > > > > >> >> and
> >> > > > > > > > > > >> >> >> readers, but realistically not widely used in
> >> the
> >> > > > > > ecosystem
> >> > > > > > > > > where
> >> > > > > > > > > > >> >> >> compatibility is a concern (a large part)
> >> > > > > > > > > > >> >> >>
> >> > > > > > > > > > >> >> >> I view Julian's work[2][3] to gather
> consensus
> >> on
> >> > > how
> >> > > > to
> >> > > > > > > name
> >> > > > > > > > > > >> >> >> combinations of features (e.g. "versions") as
> >> the
> >> > > > > > mechanism
> >> > > > > > > by
> >> > > > > > > > > > >> which we
> >> > > > > > > > > > >> >> >> will accelerate the adoption of this change
> >> (along
> >> > > > with
> >> > > > > > > other
> >> > > > > > > > > > >> forward
> >> > > > > > > > > > >> >> >> incompatible changes) across the ecosystem.
> >> > > > > > > > > > >> >> >>
> >> > > > > > > > > > >> >> >> > Will we exclude this change when cutting
> >> > > > > parquet-format
> >> > > > > > > > > v2.14.0?
> >> > > > > > > > > > >> >> >>
> >> > > > > > > > > > >> >> >> No I don't think so -- I would expect it to
> go
> >> out
> >> > > in
> >> > > > > the
> >> > > > > > > next
> >> > > > > > > > > > >> >> >> parquet-format version as do other changes in
> >> the
> >> > > > spec.
> >> > > > > > > > > > >> >> >>
> >> > > > > > > > > > >> >> >> Andrew
> >> > > > > > > > > > >> >> >>
> >> > > > > > > > > > >> >> >> [1]:
> >> > > > > > https://parquet.apache.org/docs/file-format/versions/
> >> > > > > > > > > > >> >> >> [2]:
> >> > > > > > > > > > >>
> >> > > > > >
> >> https://lists.apache.org/thread/xmnj8h0h8ozmrgox5tydhhs7yz7q2hss
> >> > > > > > > > > > >> >> >> [3]:
> >> > > > > > > > > > >>
> >> > > > > >
> >> https://lists.apache.org/thread/qlf8lg90gqq41lllyy9mk2f0skqvftqj
> >> > > > > > > > > > >> >> >>
> >> > > > > > > > > > >> >> >>
> >> > > > > > > > > > >> >> >> On Thu, Jul 30, 2026 at 3:51 AM Divjot Arora
> >> via
> >> > > dev <
> >> > > > > > > > > > >> >> >> dev@parquet.apache.org> wrote:
> >> > > > > > > > > > >> >> >>
> >> > > > > > > > > > >> >> >>> Hi Andrew,
> >> > > > > > > > > > >> >> >>>
> >> > > > > > > > > > >> >> >>> How does this interact with the ongoing
> >> > versioning
> >> > > > > > > > discussion?
> >> > > > > > > > > > >> Given
> >> > > > > > > > > > >> >> that
> >> > > > > > > > > > >> >> >>> new encodings are not forward compatible, I
> >> would
> >> > > > > assume
> >> > > > > > > ALP
> >> > > > > > > > is
> >> > > > > > > > > > >> >> included
> >> > > > > > > > > > >> >> >>> in
> >> > > > > > > > > > >> >> >>> Parquet V<next> (V3?), but it's not clear to
> >> me
> >> > how
> >> > > > > that
> >> > > > > > > > works
> >> > > > > > > > > if
> >> > > > > > > > > > >> it's
> >> > > > > > > > > > >> >> >>> merged to the parquet-format master branch
> >> now.
> >> > > Will
> >> > > > we
> >> > > > > > > > exclude
> >> > > > > > > > > > >> this
> >> > > > > > > > > > >> >> >>> change
> >> > > > > > > > > > >> >> >>> when cutting parquet-format v2.14.0?
> >> > > > > > > > > > >> >> >>>
> >> > > > > > > > > > >> >> >>> -- Div
> >> > > > > > > > > > >> >> >>>
> >> > > > > > > > > > >> >> >>> On Thu, Jul 30, 2026 at 8:51 AM Andrew Lamb
> <
> >> > > > > > > > > > >> andrewlamb11@gmail.com>
> >> > > > > > > > > > >> >> >>> wrote:
> >> > > > > > > > > > >> >> >>>
> >> > > > > > > > > > >> >> >>> > Hello,
> >> > > > > > > > > > >> >> >>> >
> >> > > > > > > > > > >> >> >>> > As an update, thanks to the epic work from
> >> many
> >> > > > > > > > contributors,
> >> > > > > > > > > > >> >> >>> especially
> >> > > > > > > > > > >> >> >>> > Prateek, I just merged the parquet-format
> >> > change
> >> > > > for
> >> > > > > > > > ALP[1].
> >> > > > > > > > > We
> >> > > > > > > > > > >> >> plan to
> >> > > > > > > > > > >> >> >>> > make additional editorial The remaining
> >> items I
> >> > > > know
> >> > > > > of
> >> > > > > > > are
> >> > > > > > > > > > >> merging
> >> > > > > > > > > > >> >> the
> >> > > > > > > > > > >> >> >>> > examples from parquet-testing[2], and then
> >> of
> >> > > > course
> >> > > > > > the
> >> > > > > > > > > > various
> >> > > > > > > > > > >> >> >>> > implementations
> >> > > > > > > > > > >> >> >>> >
> >> > > > > > > > > > >> >> >>> > Thanks again everyone. This is a great
> step
> >> > > forward
> >> > > > > > > > > > >> >> >>> > Andrew
> >> > > > > > > > > > >> >> >>> >
> >> > > > > > > > > > >> >> >>> > [1]:
> >> > > > > https://github.com/apache/parquet-format/pull/557
> >> > > > > > > > > > >> >> >>> > [2]:
> >> > > > > > https://github.com/apache/parquet-testing/pull/100
> >> > > > > > > > > > >> >> >>> >
> >> > > > > > > > > > >> >> >>> > On Tue, Jul 21, 2026 at 4:39 PM Julien Le
> >> Dem <
> >> > > > > > > > > > julien@apache.org
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >> >> >>> wrote:
> >> > > > > > > > > > >> >> >>> >
> >> > > > > > > > > > >> >> >>> > > It was also in my spam folder... :(
> >> > > > > > > > > > >> >> >>> > > I've seen a bunch of apache list emails
> >> go to
> >> > > my
> >> > > > > spam
> >> > > > > > > > > > recently.
> >> > > > > > > > > > >> >> >>> > >
> >> > > > > > > > > > >> >> >>> > > On Tue, Jul 21, 2026 at 10:58 AM Russell
> >> > > Spitzer
> >> > > > <
> >> > > > > > > > > > >> >> >>> > > russell.spitzer@gmail.com>
> >> > > > > > > > > > >> >> >>> > > wrote:
> >> > > > > > > > > > >> >> >>> > >
> >> > > > > > > > > > >> >> >>> > > > Yes, my hope was that my vote would
> >> kick it
> >> > > out
> >> > > > > of
> >> > > > > > > spam
> >> > > > > > > > > but
> >> > > > > > > > > > >> we
> >> > > > > > > > > > >> >> >>> > definitely
> >> > > > > > > > > > >> >> >>> > > > had two other votes that also went to
> >> spam
> >> > > for
> >> > > > > me.
> >> > > > > > > > > > >> >> >>> > > >
> >> > > > > > > > > > >> >> >>> > > > For anyone looking the thread is here
> >> > > > > > > > > > >> >> >>> > > >
> >> > > > > > > > > > >> >>
> >> > > > > > >
> >> https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt
> >> > > > > > > > > > >> >> >>> > > >
> >> > > > > > > > > > >> >> >>> > > > On Tue, Jul 21, 2026 at 10:57 AM Micah
> >> > > > Kornfield
> >> > > > > <
> >> > > > > > > > > > >> >> >>> > emkornfield@gmail.com>
> >> > > > > > > > > > >> >> >>> > > > wrote:
> >> > > > > > > > > > >> >> >>> > > >
> >> > > > > > > > > > >> >> >>> > > > > I've seen other's votes not go to
> >> spam,
> >> > so
> >> > > > > > > hopefully
> >> > > > > > > > > > people
> >> > > > > > > > > > >> >> will
> >> > > > > > > > > > >> >> >>> see
> >> > > > > > > > > > >> >> >>> > it
> >> > > > > > > > > > >> >> >>> > > > in
> >> > > > > > > > > > >> >> >>> > > > > there inboxes (or see this thread
> and
> >> > look
> >> > > in
> >> > > > > > there
> >> > > > > > > > > > spam).
> >> > > > > > > > > > >> >> >>> > > > >
> >> > > > > > > > > > >> >> >>> > > > > On Tue, Jul 21, 2026 at 10:52 AM
> >> PRATEEK
> >> > > > GAUR <
> >> > > > > > > > > > >> >> >>> prtkgaur@gmail.com>
> >> > > > > > > > > > >> >> >>> > > > wrote:
> >> > > > > > > > > > >> >> >>> > > > >
> >> > > > > > > > > > >> >> >>> > > > > > Okay,
> >> > > > > > > > > > >> >> >>> > > > > >
> >> > > > > > > > > > >> >> >>> > > > > > I sent out the voting email
> >> yesterday
> >> > and
> >> > > > so
> >> > > > > > far
> >> > > > > > > > I've
> >> > > > > > > > > > >> heard
> >> > > > > > > > > > >> >> >>> from 3
> >> > > > > > > > > > >> >> >>> > > > people
> >> > > > > > > > > > >> >> >>> > > > > > that it ended up in their "spam
> >> > folder".
> >> > > > Dev
> >> > > > > > VPN
> >> > > > > > > to
> >> > > > > > > > > be
> >> > > > > > > > > > >> >> blamed
> >> > > > > > > > > > >> >> >>> for
> >> > > > > > > > > > >> >> >>> > > it?.
> >> > > > > > > > > > >> >> >>> > > > > >
> >> > > > > > > > > > >> >> >>> > > > > > What's the protocol here?
> >> > > > > > > > > > >> >> >>> > > > > > I let people hunt for the email in
> >> > their
> >> > > > spam
> >> > > > > > > > folder
> >> > > > > > > > > or
> >> > > > > > > > > > >> send
> >> > > > > > > > > > >> >> >>> out
> >> > > > > > > > > > >> >> >>> > > > another
> >> > > > > > > > > > >> >> >>> > > > > > one?
> >> > > > > > > > > > >> >> >>> > > > > >
> >> > > > > > > > > > >> >> >>> > > > > > Best
> >> > > > > > > > > > >> >> >>> > > > > > Prateek
> >> > > > > > > > > > >> >> >>> > > > > >
> >> > > > > > > > > > >> >> >>> > > > > > On Sun, Jul 19, 2026 at 10:47 PM
> >> > PRATEEK
> >> > > > > GAUR <
> >> > > > > > > > > > >> >> >>> prtkgaur@gmail.com>
> >> > > > > > > > > > >> >> >>> > > > > wrote:
> >> > > > > > > > > > >> >> >>> > > > > >
> >> > > > > > > > > > >> >> >>> > > > > > > Hi team,
> >> > > > > > > > > > >> >> >>> > > > > > >
> >> > > > > > > > > > >> >> >>> > > > > > > Vinoo has added the tests which
> >> were
> >> > > > > > requested
> >> > > > > > > in
> >> > > > > > > > > the
> >> > > > > > > > > > >> last
> >> > > > > > > > > > >> >> >>> round
> >> > > > > > > > > > >> >> >>> > of
> >> > > > > > > > > > >> >> >>> > > > > > review.
> >> > > > > > > > > > >> >> >>> > > > > > > In his own words.
> >> > > > > > > > > > >> >> >>> > > > > > >
> >> > > > > > > > > > >> >> >>> > > > > > > "
> >> > > > > > > > > > >> >> >>> > > > > > > The extreme value tests Micah
> >> wanted
> >> > > are
> >> > > > > done
> >> > > > > > > and
> >> > > > > > > > > > >> pushed.
> >> > > > > > > > > > >> >> I
> >> > > > > > > > > > >> >> >>> added
> >> > > > > > > > > > >> >> >>> > > > > > coverage
> >> > > > > > > > > > >> >> >>> > > > > > > for values that need the full
> FOR
> >> bit
> >> > > > width
> >> > > > > > > after
> >> > > > > > > > > the
> >> > > > > > > > > > >> >> frame
> >> > > > > > > > > > >> >> >>> of
> >> > > > > > > > > > >> >> >>> > > > > reference
> >> > > > > > > > > > >> >> >>> > > > > > is
> >> > > > > > > > > > >> >> >>> > > > > > > applied: 64-bit (the case where
> >> the
> >> > > > signed
> >> > > > > > max
> >> > > > > > > > > minus
> >> > > > > > > > > > >> min
> >> > > > > > > > > > >> >> >>> > > subtraction
> >> > > > > > > > > > >> >> >>> > > > > > > overflows), 63-bit (the
> >> non-overflow
> >> > > > case),
> >> > > > > > and
> >> > > > > > > > > > 32-bit
> >> > > > > > > > > > >> for
> >> > > > > > > > > > >> >> >>> > floats.
> >> > > > > > > > > > >> >> >>> > > > All
> >> > > > > > > > > > >> >> >>> > > > > > > lossless with zero exceptions. I
> >> also
> >> > > > > > verified
> >> > > > > > > > them
> >> > > > > > > > > > >> >> through
> >> > > > > > > > > > >> >> >>> Arrow
> >> > > > > > > > > > >> >> >>> > > C++
> >> > > > > > > > > > >> >> >>> > > > > ALP
> >> > > > > > > > > > >> >> >>> > > > > > > reader (the local build), so the
> >> > > > > Java-written
> >> > > > > > > > > extreme
> >> > > > > > > > > > >> >> columns
> >> > > > > > > > > > >> >> >>> > > decode
> >> > > > > > > > > > >> >> >>> > > > > > > correctly in C++, bit-exact
> >> against
> >> > the
> >> > > > > > > expected
> >> > > > > > > > > > >> values."
> >> > > > > > > > > > >> >> >>> > > > > > >
> >> > > > > > > > > > >> >> >>> > > > > > > State of the PR (#3397):
> >> > > > > > > > > > >> >> >>> > > > > > >
> >> > > > > > > > > > >> >> >>> > > > > > > - Merges cleanly with master
> (I
> >> > > pulled
> >> > > > > > > master
> >> > > > > > > > in
> >> > > > > > > > > > and
> >> > > > > > > > > > >> >> >>> resolved
> >> > > > > > > > > > >> >> >>> > > the
> >> > > > > > > > > > >> >> >>> > > > > > > conflicts).
> >> > > > > > > > > > >> >> >>> > > > > > > - 41 of 44 review threads
> >> > resolved.
> >> > > > The
> >> > > > > 3
> >> > > > > > > > > > remaining
> >> > > > > > > > > > >> are
> >> > > > > > > > > > >> >> >>> > > > intentional:
> >> > > > > > > > > > >> >> >>> > > > > > > one deferred perf nit, one
> that
> >> > > stays
> >> > > > > open
> >> > > > > > > > until
> >> > > > > > > > > > >> >> >>> > parquet-format
> >> > > > > > > > > > >> >> >>> > > > > ships
> >> > > > > > > > > > >> >> >>> > > > > > ALP,
> >> > > > > > > > > > >> >> >>> > > > > > >
> >> > > > > > > > > > >> >> >>> > > > > > > "
> >> > > > > > > > > > >> >> >>> > > > > > >
> >> > > > > > > > > > >> >> >>> > > > > > > With this both the
> >> > > > > > > > > > >> >> >>> > > > > > > 1) C++ PR :
> >> > > > > > > > > > >> >> >>>
> >> > https://github.com/apache/arrow/pull/48345/changes
> >> > > > > > > > > > >> >> >>> > > > > > > 2) Java PR :
> >> > > > > > > > > > >> >> >>>
> >> https://github.com/apache/parquet-java/pull/3397
> >> > > > > > > > > > >> >> >>> > > > > > >
> >> > > > > > > > > > >> >> >>> > > > > > > Are cross language tested and
> >> range
> >> > of
> >> > > > test
> >> > > > > > > > > > scenarios.
> >> > > > > > > > > > >> >> >>> > > > > > >
> >> > > > > > > > > > >> >> >>> > > > > > > Best
> >> > > > > > > > > > >> >> >>> > > > > > > Prateek
> >> > > > > > > > > > >> >> >>> > > > > > >
> >> > > > > > > > > > >> >> >>> > > > > > >
> >> > > > > > > > > > >> >> >>> > > > > > > On Mon, Jul 13, 2026 at 8:38 AM
> >> > PRATEEK
> >> > > > > GAUR
> >> > > > > > <
> >> > > > > > > > > > >> >> >>> prtkgaur@gmail.com
> >> > > > > > > > > > >> >> >>> > >
> >> > > > > > > > > > >> >> >>> > > > > wrote:
> >> > > > > > > > > > >> >> >>> > > > > > >
> >> > > > > > > > > > >> >> >>> > > > > > >> Thanks Matt,
> >> > > > > > > > > > >> >> >>> > > > > > >>
> >> > > > > > > > > > >> >> >>> > > > > > >> I'll be on it this week.
> >> > > > > > > > > > >> >> >>> > > > > > >>
> >> > > > > > > > > > >> >> >>> > > > > > >> Best
> >> > > > > > > > > > >> >> >>> > > > > > >> Prateek
> >> > > > > > > > > > >> >> >>> > > > > > >>
> >> > > > > > > > > > >> >> >>> > > > > > >> On Mon, Jul 13, 2026 at 8:36 AM
> >> Matt
> >> > > > > Topol <
> >> > > > > > > > > > >> >> >>> > > zotthewizard@gmail.com>
> >> > > > > > > > > > >> >> >>> > > > > > >> wrote:
> >> > > > > > > > > > >> >> >>> > > > > > >>
> >> > > > > > > > > > >> >> >>> > > > > > >>> There is also the Go
> >> implementation
> >> > > > > > submitted
> >> > > > > > > > by
> >> > > > > > > > > > >> Arnav (
> >> > > > > > > > > > >> >> >>> > > > > > >>>
> >> > > > > > https://github.com/apache/arrow-go/pull/704)
> >> > > > > > > > > which
> >> > > > > > > > > > >> is
> >> > > > > > > > > > >> >> >>> waiting
> >> > > > > > > > > > >> >> >>> > > for
> >> > > > > > > > > > >> >> >>> > > > > > >>> updates.
> >> > > > > > > > > > >> >> >>> > > > > > >>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> Just wanted to make sure it
> >> didn't
> >> > > get
> >> > > > > lost
> >> > > > > > > > here.
> >> > > > > > > > > > >> >> >>> > > > > > >>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> --Matt
> >> > > > > > > > > > >> >> >>> > > > > > >>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> On Mon, Jul 13, 2026 at
> 11:30 AM
> >> > > > PRATEEK
> >> > > > > > > GAUR <
> >> > > > > > > > > > >> >> >>> > > prtkgaur@gmail.com>
> >> > > > > > > > > > >> >> >>> > > > > > >>> wrote:
> >> > > > > > > > > > >> >> >>> > > > > > >>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > Hi all,
> >> > > > > > > > > > >> >> >>> > > > > > >>> >
> >> > > > > > > > > > >> >> >>> > > > > > >>> > I'd like to share a status
> >> update
> >> > > on
> >> > > > > the
> >> > > > > > > ALP
> >> > > > > > > > > > >> encoding
> >> > > > > > > > > > >> >> >>> effort
> >> > > > > > > > > > >> >> >>> > > and
> >> > > > > > > > > > >> >> >>> > > > > get
> >> > > > > > > > > > >> >> >>> > > > > > a
> >> > > > > > > > > > >> >> >>> > > > > > >>> feel
> >> > > > > > > > > > >> >> >>> > > > > > >>> > from the community before
> >> > starting
> >> > > a
> >> > > > > > formal
> >> > > > > > > > > vote.
> >> > > > > > > > > > >> >> >>> > > > > > >>> >
> >> > > > > > > > > > >> >> >>> > > > > > >>> > We now have the
> specification
> >> > plus
> >> > > > > > > > > > implementations
> >> > > > > > > > > > >> in
> >> > > > > > > > > > >> >> >>> both
> >> > > > > > > > > > >> >> >>> > > > > languages:
> >> > > > > > > > > > >> >> >>> > > > > > >>> >
> >> > > > > > > > > > >> >> >>> > > > > > >>> > - Spec:
> >> > > > > > > > apache/parquet-format#557 —
> >> > > > > > > > > > >> GH-533
> >> > > > > > > > > > >> >> >>> Add
> >> > > > > > > > > > >> >> >>> > ALP
> >> > > > > > > > > > >> >> >>> > > > > > encoding
> >> > > > > > > > > > >> >> >>> > > > > > >>> > specification
> >> > > > > > > > > > >> >> >>> > > > > > >>> > - C++ (Arrow):
> >> > > > apache/arrow#48345
> >> > > > > —
> >> > > > > > > > > GH-48701
> >> > > > > > > > > > >> >> >>> > > [C++][Parquet]
> >> > > > > > > > > > >> >> >>> > > > > Add
> >> > > > > > > > > > >> >> >>> > > > > > >>> ALPpd
> >> > > > > > > > > > >> >> >>> > > > > > >>> > encoding
> >> > > > > > > > > > >> >> >>> > > > > > >>> > - Java:
> >> > > > > > > apache/parquet-java#3397
> >> > > > > > > > —
> >> > > > > > > > > > >> Parquet
> >> > > > > > > > > > >> >> >>> Java
> >> > > > > > > > > > >> >> >>> > ALP
> >> > > > > > > > > > >> >> >>> > > > > > >>> > Implementation (by Vinoo)
> >> > > > > > > > > > >> >> >>> > > > > > >>> >
> >> > > > > > > > > > >> >> >>> > > > > > >>> > Current state:
> >> > > > > > > > > > >> >> >>> > > > > > >>> >
> >> > > > > > > > > > >> >> >>> > > > > > >>> > - The C++ implementation
> >> has
> >> > > been
> >> > > > > > > through
> >> > > > > > > > > 3-4
> >> > > > > > > > > > >> >> rounds
> >> > > > > > > > > > >> >> >>> of
> >> > > > > > > > > > >> >> >>> > > > review.
> >> > > > > > > > > > >> >> >>> > > > > > >>> > - The Java
> implementation
> >> has
> >> > > > been
> >> > > > > > > > tracking
> >> > > > > > > > > > >> well
> >> > > > > > > > > > >> >> and
> >> > > > > > > > > > >> >> >>> has
> >> > > > > > > > > > >> >> >>> > > had
> >> > > > > > > > > > >> >> >>> > > > > > >>> initial
> >> > > > > > > > > > >> >> >>> > > > > > >>> > reviews from
> >> contributors,
> >> > > with
> >> > > > > no
> >> > > > > > > > major
> >> > > > > > > > > > >> >> remaining
> >> > > > > > > > > > >> >> >>> > > issues.
> >> > > > > > > > > > >> >> >>> > > > > > >>> > - Cross-language
> >> > compatibility
> >> > > > > tests
> >> > > > > > > are
> >> > > > > > > > > > >> passing:
> >> > > > > > > > > > >> >> the
> >> > > > > > > > > > >> >> >>> > Arrow
> >> > > > > > > > > > >> >> >>> > > > C++
> >> > > > > > > > > > >> >> >>> > > > > > >>> decoder
> >> > > > > > > > > > >> >> >>> > > > > > >>> > reads Java-written
> data
> >> > > > > bit-exactly
> >> > > > > > > > > across
> >> > > > > > > > > > >> >> ~1.56M
> >> > > > > > > > > > >> >> >>> > values
> >> > > > > > > > > > >> >> >>> > > > and
> >> > > > > > > > > > >> >> >>> > > > > 18
> >> > > > > > > > > > >> >> >>> > > > > > >>> > fixtures, covering V1
> >> and
> >> > V2
> >> > > > > pages,
> >> > > > > > > > > > multiple
> >> > > > > > > > > > >> >> vector
> >> > > > > > > > > > >> >> >>> > > sizes,
> >> > > > > > > > > > >> >> >>> > > > > and
> >> > > > > > > > > > >> >> >>> > > > > > >>> > several
> >> > > > > > > > > > >> >> >>> > > > > > >>> > real datasets — zero
> >> > > > mismatches.
> >> > > > > > > > > > >> >> >>> > > > > > >>> >
> >> > > > > > > > > > >> >> >>> > > > > > >>> > As raised in the java-pr we
> >> will
> >> > > be
> >> > > > > > adding
> >> > > > > > > > > > >> coverage
> >> > > > > > > > > > >> >> for
> >> > > > > > > > > > >> >> >>> > > extreme
> >> > > > > > > > > > >> >> >>> > > > > > values
> >> > > > > > > > > > >> >> >>> > > > > > >>> > (those requiring
> >> > > > > > > > > > >> >> >>> > > > > > >>> > 63-64 bits after FOR is
> >> applied)
> >> > > > > before
> >> > > > > > we
> >> > > > > > > > > > close a
> >> > > > > > > > > > >> >> vote.
> >> > > > > > > > > > >> >> >>> > We'll
> >> > > > > > > > > > >> >> >>> > > > aim
> >> > > > > > > > > > >> >> >>> > > > > > to
> >> > > > > > > > > > >> >> >>> > > > > > >>> get
> >> > > > > > > > > > >> >> >>> > > > > > >>> > that
> >> > > > > > > > > > >> >> >>> > > > > > >>> > done in parallel.
> >> > > > > > > > > > >> >> >>> > > > > > >>> >
> >> > > > > > > > > > >> >> >>> > > > > > >>> > Plan: unless there are
> >> > objections,
> >> > > I
> >> > > > > > intend
> >> > > > > > > > to
> >> > > > > > > > > > >> start a
> >> > > > > > > > > > >> >> >>> vote
> >> > > > > > > > > > >> >> >>> > at
> >> > > > > > > > > > >> >> >>> > > > the
> >> > > > > > > > > > >> >> >>> > > > > > end
> >> > > > > > > > > > >> >> >>> > > > > > >>> of
> >> > > > > > > > > > >> >> >>> > > > > > >>> > this week/early
> >> > > > > > > > > > >> >> >>> > > > > > >>> > next week.
> >> > > > > > > > > > >> >> >>> > > > > > >>> >
> >> > > > > > > > > > >> >> >>> > > > > > >>> > Thanks
> >> > > > > > > > > > >> >> >>> > > > > > >>> > Prateek
> >> > > > > > > > > > >> >> >>> > > > > > >>> >
> >> > > > > > > > > > >> >> >>> > > > > > >>> > On Wed, Jul 1, 2026 at
> 8:50 AM
> >> > > > PRATEEK
> >> > > > > > > GAUR <
> >> > > > > > > > > > >> >> >>> > > prtkgaur@gmail.com>
> >> > > > > > > > > > >> >> >>> > > > > > >>> wrote:
> >> > > > > > > > > > >> >> >>> > > > > > >>> >
> >> > > > > > > > > > >> >> >>> > > > > > >>> > > Hi Team,
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >
> >> > > > > > > > > > >> >> >>> > > > > > >>> > > Just wanted to provide
> some
> >> > > updates
> >> > > > > on
> >> > > > > > > ALP.
> >> > > > > > > > > > >> >> >>> > > > > > >>> > > Micah and I have done 3-4
> >> > rounds
> >> > > of
> >> > > > > > > review
> >> > > > > > > > > for
> >> > > > > > > > > > >> the
> >> > > > > > > > > > >> >> c++
> >> > > > > > > > > > >> >> >>> PR :
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >
> >> > > > > > > https://github.com/apache/arrow/pull/48345
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >
> >> > > > > > > > > > >> >> >>> > > > > > >>> > > For the Java
> implementation
> >> > Vinoo
> >> > > > has
> >> > > > > > > been
> >> > > > > > > > > > >> working
> >> > > > > > > > > > >> >> on
> >> > > > > > > > > > >> >> >>> > > following
> >> > > > > > > > > > >> >> >>> > > > > PR
> >> > > > > > > > > > >> >> >>> > > > > > :
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >
> >> > > > > > > > > > https://github.com/apache/parquet-java/pull/3397
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >
> >> > > > > > > > > > >> >> >>> > > > > > >>> > > Best
> >> > > > > > > > > > >> >> >>> > > > > > >>> > > Prateek
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >
> >> > > > > > > > > > >> >> >>> > > > > > >>> > > On Tue, May 5, 2026 at
> >> 1:50 PM
> >> > > > Micah
> >> > > > > > > > > Kornfield
> >> > > > > > > > > > <
> >> > > > > > > > > > >> >> >>> > > > > > >>> emkornfield@gmail.com>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > > wrote:
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> Hi Antoine,
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > Apologies if the
> question
> >> > was
> >> > > > > > already
> >> > > > > > > > > asked,
> >> > > > > > > > > > >> but
> >> > > > > > > > > > >> >> >>> should
> >> > > > > > > > > > >> >> >>> > we
> >> > > > > > > > > > >> >> >>> > > > > care
> >> > > > > > > > > > >> >> >>> > > > > > >>> about
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can
> >> FLOAT +
> >> > > ALP
> >> > > > > be
> >> > > > > > > more
> >> > > > > > > > > > >> >> efficient
> >> > > > > > > > > > >> >> >>> than
> >> > > > > > > > > > >> >> >>> > > > > FLOAT16
> >> > > > > > > > > > >> >> >>> > > > > > +
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4
> >> for
> >> > > > > example?
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> It was. We thought we
> >> could
> >> > > defer
> >> > > > > it
> >> > > > > > > for
> >> > > > > > > > > the
> >> > > > > > > > > > >> >> >>> following
> >> > > > > > > > > > >> >> >>> > > > reasons:
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> 1. It's not clear there
> >> are a
> >> > > lot
> >> > > > > of
> >> > > > > > > easy
> >> > > > > > > > > > >> >> reference
> >> > > > > > > > > > >> >> >>> > > datasets
> >> > > > > > > > > > >> >> >>> > > > to
> >> > > > > > > > > > >> >> >>> > > > > > >>> test
> >> > > > > > > > > > >> >> >>> > > > > > >>> > its
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> effectiveness. It does
> >> look
> >> > > like
> >> > > > > > there
> >> > > > > > > > > might
> >> > > > > > > > > > be
> >> > > > > > > > > > >> >> one
> >> > > > > > > > > > >> >> >>> or
> >> > > > > > > > > > >> >> >>> > two
> >> > > > > > > > > > >> >> >>> > > on
> >> > > > > > > > > > >> >> >>> > > > > > >>> > huggingface
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> (e.g.
> >> > > > > > > > > > >> >> https://huggingface.co/datasets/kikitora/curdie
> >> > > > > > > > > > >> >> >>> ).
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> 2. It seemed likely that
> >> > float
> >> > > 16
> >> > > > > was
> >> > > > > > > > more
> >> > > > > > > > > > >> likely
> >> > > > > > > > > > >> >> >>> used
> >> > > > > > > > > > >> >> >>> > for
> >> > > > > > > > > > >> >> >>> > > > > values
> >> > > > > > > > > > >> >> >>> > > > > > >>> that
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> were less likely to
> reduce
> >> to
> >> > > > > decimal
> >> > > > > > > > > values.
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> 3. It could be added as
> an
> >> > > > > extension
> >> > > > > > > > later
> >> > > > > > > > > if
> >> > > > > > > > > > >> >> needed.
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> Cheers,
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> Micah
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> On Tue, May 5, 2026 at
> >> 1:40 PM
> >> > > > > Antoine
> >> > > > > > > > > Pitrou
> >> > > > > > > > > > <
> >> > > > > > > > > > >> >> >>> > > > > antoine@python.org
> >> > > > > > > > > > >> >> >>> > > > > > >
> >> > > > > > > > > > >> >> >>> > > > > > >>> > wrote:
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> >
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > Hello,
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> >
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > Apologies if the
> question
> >> > was
> >> > > > > > already
> >> > > > > > > > > asked,
> >> > > > > > > > > > >> but
> >> > > > > > > > > > >> >> >>> should
> >> > > > > > > > > > >> >> >>> > we
> >> > > > > > > > > > >> >> >>> > > > > care
> >> > > > > > > > > > >> >> >>> > > > > > >>> about
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can
> >> FLOAT +
> >> > > ALP
> >> > > > > be
> >> > > > > > > more
> >> > > > > > > > > > >> >> efficient
> >> > > > > > > > > > >> >> >>> than
> >> > > > > > > > > > >> >> >>> > > > > FLOAT16
> >> > > > > > > > > > >> >> >>> > > > > > +
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4
> >> for
> >> > > > > example?
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> >
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > Regards
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> >
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > Antoine.
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> >
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> >
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > Le 30/04/2026 à 01:10,
> >> > PRATEEK
> >> > > > > GAUR
> >> > > > > > a
> >> > > > > > > > > écrit
> >> > > > > > > > > > :
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > > Thanks Andrew and
> Micah
> >> > for
> >> > > > > review
> >> > > > > > > > > > feedback
> >> > > > > > > > > > >> on
> >> > > > > > > > > > >> >> >>> the two
> >> > > > > > > > > > >> >> >>> > > > PR's
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > > 1) (c++ arrow repo)
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >>
> >> > > > > > > > > > >> https://github.com/apache/arrow/pull/48345/changes
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > > 2) (parquet-format
> >> repo)
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> >
> >> > > > > > > > > > >> >>
> https://github.com/apache/parquet-format/pull/557
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > > I have addressed all
> >> > > (unless I
> >> > > > > > > missed
> >> > > > > > > > > > >> >> something)
> >> > > > > > > > > > >> >> >>> > > comments
> >> > > > > > > > > > >> >> >>> > > > on
> >> > > > > > > > > > >> >> >>> > > > > > >>> the two
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > PR's.
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > > Best
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > > Prateek
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > > On Sat, Apr 25, 2026
> at
> >> > > > 1:08 PM
> >> > > > > > > > PRATEEK
> >> > > > > > > > > > >> GAUR <
> >> > > > > > > > > > >> >> >>> > > > > > >>> prtkgaur@gmail.com>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> wrote:
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >> Thanks Andrew and
> >> Micah.
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >> `fair amount of
> >> feedback
> >> > on
> >> > > > at
> >> > > > > > > least
> >> > > > > > > > > the
> >> > > > > > > > > > >> >> >>> > > implementations`
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >> For the c++ I have
> >> > already
> >> > > > > > started
> >> > > > > > > > > > >> addressing
> >> > > > > > > > > > >> >> the
> >> > > > > > > > > > >> >> >>> > > > > feedback, I
> >> > > > > > > > > > >> >> >>> > > > > > >>> > should
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> be
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >> done with that
> >> > > > Monday/Tuesday.
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >> I think Vinoo too
> has
> >> > been
> >> > > > > making
> >> > > > > > > > good
> >> > > > > > > > > > >> >> progress
> >> > > > > > > > > > >> >> >>> on
> >> > > > > > > > > > >> >> >>> > the
> >> > > > > > > > > > >> >> >>> > > > Java
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > implementation.
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >> Best
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >> Prateek
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >> On Sat, Apr 25, 2026
> >> at
> >> > > > > 12:55 PM
> >> > > > > > > > Andrew
> >> > > > > > > > > > >> Lamb <
> >> > > > > > > > > > >> >> >>> > > > > > >>> > andrewlamb11@gmail.com
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> >
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >> wrote:
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>> Got it. Thank you
> for
> >> > the
> >> > > > > > > > > clarification
> >> > > > > > > > > > >> -- I
> >> > > > > > > > > > >> >> >>> will
> >> > > > > > > > > > >> >> >>> > try
> >> > > > > > > > > > >> >> >>> > > > and
> >> > > > > > > > > > >> >> >>> > > > > > look
> >> > > > > > > > > > >> >> >>> > > > > > >>> > into
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> the
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>> spec and the Rust
> >> > > > > > > implementation[1]
> >> > > > > > > > in
> >> > > > > > > > > > >> this
> >> > > > > > > > > > >> >> next
> >> > > > > > > > > > >> >> >>> > week
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>> [1]:
> >> > > > > > > > > > >> >> >>>
> https://github.com/apache/arrow-rs/pull/9372
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>> On Sat, Apr 25,
> 2026
> >> at
> >> > > > > 12:01 PM
> >> > > > > > > > Micah
> >> > > > > > > > > > >> >> >>> Kornfield <
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > emkornfield@gmail.com>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>> wrote:
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>> Hi Andrew,
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>> I think there is a
> >> fair
> >> > > > > amount
> >> > > > > > of
> >> > > > > > > > > > >> feedback
> >> > > > > > > > > > >> >> on
> >> > > > > > > > > > >> >> >>> at
> >> > > > > > > > > > >> >> >>> > > least
> >> > > > > > > > > > >> >> >>> > > > > the
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>> implementations,
> >> > > typically
> >> > > > I
> >> > > > > > > think
> >> > > > > > > > > > we've
> >> > > > > > > > > > >> >> waited
> >> > > > > > > > > > >> >> >>> > till
> >> > > > > > > > > > >> >> >>> > > > they
> >> > > > > > > > > > >> >> >>> > > > > > are
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> close to
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>> mergeable before a
> >> > final
> >> > > > > vote.
> >> > > > > > > > > > >> Otherwise I
> >> > > > > > > > > > >> >> >>> agree
> >> > > > > > > > > > >> >> >>> > we
> >> > > > > > > > > > >> >> >>> > > > are
> >> > > > > > > > > > >> >> >>> > > > > > very
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> close.
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>> -Micah
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>> On Saturday, April
> >> 25,
> >> > > > 2026,
> >> > > > > > > Andrew
> >> > > > > > > > > > Lamb
> >> > > > > > > > > > >> <
> >> > > > > > > > > > >> >> >>> > > > > > >>> andrewlamb11@gmail.com
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > wrote:
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> Thanks Prateek,
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> I think from this
> >> > > content
> >> > > > it
> >> > > > > > > looks
> >> > > > > > > > > to
> >> > > > > > > > > > me
> >> > > > > > > > > > >> >> like
> >> > > > > > > > > > >> >> >>> we
> >> > > > > > > > > > >> >> >>> > are
> >> > > > > > > > > > >> >> >>> > > > > ready
> >> > > > > > > > > > >> >> >>> > > > > > >>> to
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> start a
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> vote to
> explicitly
> >> > > accept
> >> > > > > ALP
> >> > > > > > > into
> >> > > > > > > > > > >> Parquet
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> Does anyone know
> >> of a
> >> > > > reason
> >> > > > > > we
> >> > > > > > > > > should
> >> > > > > > > > > > >> >> >>> postpone it
> >> > > > > > > > > > >> >> >>> > > for
> >> > > > > > > > > > >> >> >>> > > > > > >>> longer?
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> Perhaps someone
> >> needs
> >> > > some
> >> > > > > > more
> >> > > > > > > > time
> >> > > > > > > > > > to
> >> > > > > > > > > > >> >> >>> review?
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> Andrew
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> On Wed, Apr 22,
> >> 2026
> >> > at
> >> > > > > > 1:00 PM
> >> > > > > > > > > > PRATEEK
> >> > > > > > > > > > >> >> GAUR <
> >> > > > > > > > > > >> >> >>> > > > > > >>> > prtkgaur@gmail.com>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> wrote:
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Hi team,
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Hope everyone is
> >> > doing
> >> > > > > well.
> >> > > > > > I
> >> > > > > > > > got
> >> > > > > > > > > a
> >> > > > > > > > > > >> >> chance
> >> > > > > > > > > > >> >> >>> to
> >> > > > > > > > > > >> >> >>> > work
> >> > > > > > > > > > >> >> >>> > > > > > >>> through all
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> the
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> remaining
> feedback
> >> > and
> >> > > > > update
> >> > > > > > > the
> >> > > > > > > > > > spec
> >> > > > > > > > > > >> >> doc.
> >> > > > > > > > > > >> >> >>> Here
> >> > > > > > > > > > >> >> >>> > > are
> >> > > > > > > > > > >> >> >>> > > > > the
> >> > > > > > > > > > >> >> >>> > > > > > >>> new
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > artifacts
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 1) Spec
> document :
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> >
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >>
> >> > > > > > > > > > >> >> >>> > > > > > >>> >
> >> > > > > > > > > > >> >> >>> > > > > > >>>
> >> > > > > > > > > > >> >> >>> > > > > >
> >> > > > > > > > > > >> >> >>> > > > >
> >> > > > > > > > > > >> >> >>> > > >
> >> > > > > > > > > > >> >> >>> > >
> >> > > > > > > > > > >> >> >>> >
> >> > > > > > > > > > >> >> >>>
> >> > > > > > > > > > >> >>
> >> > > > > > > > > > >>
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > > https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-
> >> > > > 3fPtD_aWt/edit
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 2) Spec document
> >> in
> >> > > > parquet
> >> > > > > > > > format
> >> > > > > > > > > > >> repo :
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> >
> >> > > https://github.com/apache/parquet-format/pull/557
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 3) Alp
> >> implementation
> >> > > in
> >> > > > > > arrow
> >> > > > > > > > c++
> >> > > > > > > > > > >> repo :
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> >
> >> > > https://github.com/apache/arrow/pull/48345/changes
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 4) Alp
> >> implementation
> >> > > in
> >> > > > > > > > > parquet-java
> >> > > > > > > > > > >> >> repo :
> >> > > > > > > > > > >> >> >>> Work
> >> > > > > > > > > > >> >> >>> > > for
> >> > > > > > > > > > >> >> >>> > > > > > >>> Vinoo and
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > Julien
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> >
> >> > https://github.com/apache/parquet-java/pull/3397
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 5) PR with test
> >> and
> >> > > > > > > benchmarking
> >> > > > > > > > > > >> >> artifacts in
> >> > > > > > > > > > >> >> >>> > > > > > >>> parquet-testing
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> repo :
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> >
> >> > > https://github.com/apache/parquet-testing/pull/100
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> And
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> - Go : Arnav
> >> just
> >> > > > > > submitted
> >> > > > > > > > an
> >> > > > > > > > > in
> >> > > > > > > > > > >> >> >>> progress
> >> > > > > > > > > > >> >> >>> > > > > > >>> implementation
> >> > > > > > > > > > >> >> >>> > > > > > >>> > in
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> Go.
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> https://github.com/apache/arrow-go/pull/704
> >> > > > > > > > > > >> >> >>> > (I
> >> > > > > > > > > > >> >> >>> > > > > > haven't
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> started
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> looking at
> it
> >> > yet)
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> - Rust : I
> >> > remember
> >> > > > > > Andrew
> >> > > > > > > > > > >> mentioned
> >> > > > > > > > > > >> >> that
> >> > > > > > > > > > >> >> >>> > this
> >> > > > > > > > > > >> >> >>> > > > work
> >> > > > > > > > > > >> >> >>> > > > > > is
> >> > > > > > > > > > >> >> >>> > > > > > >>> also
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> in
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> progress
> (So 4
> >> > > > > > languages!)
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> *Arrow C++
> >> > > > implementation *
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> The PR is out
> and
> >> was
> >> > > > also
> >> > > > > > used
> >> > > > > > > > by
> >> > > > > > > > > > >> >> Antoine to
> >> > > > > > > > > > >> >> >>> > > report
> >> > > > > > > > > > >> >> >>> > > > > the
> >> > > > > > > > > > >> >> >>> > > > > > >>> > numbers
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> as
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> reported here.
> >> Micah
> >> > > and
> >> > > > > > > > Konstantin
> >> > > > > > > > > > >> have
> >> > > > > > > > > > >> >> >>> given 1
> >> > > > > > > > > > >> >> >>> > > > round
> >> > > > > > > > > > >> >> >>> > > > > of
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> feedback
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> and I'm
> addressing
> >> > them
> >> > > > > > today.
> >> > > > > > > > > Please
> >> > > > > > > > > > >> note
> >> > > > > > > > > > >> >> >>> that
> >> > > > > > > > > > >> >> >>> > the
> >> > > > > > > > > > >> >> >>> > > > > > default
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> optimization
> flag
> >> for
> >> > > > > > compiling
> >> > > > > > > > is
> >> > > > > > > > > O2
> >> > > > > > > > > > >> and
> >> > > > > > > > > > >> >> not
> >> > > > > > > > > > >> >> >>> > Q3. I
> >> > > > > > > > > > >> >> >>> > > > got
> >> > > > > > > > > > >> >> >>> > > > > > >>> around
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> 70%
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> performance
> >> > improvement
> >> > > > in
> >> > > > > > the
> >> > > > > > > > > > decoding
> >> > > > > > > > > > >> >> speed
> >> > > > > > > > > > >> >> >>> > when
> >> > > > > > > > > > >> >> >>> > > > > using
> >> > > > > > > > > > >> >> >>> > > > > > >>> the O3
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > flag.
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> *Parqet-MR Java
> >> > > > > > implementation
> >> > > > > > > > > > (working
> >> > > > > > > > > > >> >> with
> >> > > > > > > > > > >> >> >>> > Vinoo
> >> > > > > > > > > > >> >> >>> > > > and
> >> > > > > > > > > > >> >> >>> > > > > > >>> Julien)
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> and
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > **Cross
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Language
> testing*
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Let me know
> if
> >> > you
> >> > > > have
> >> > > > > > any
> >> > > > > > > > > > >> questions
> >> > > > > > > > > > >> >> or
> >> > > > > > > > > > >> >> >>> > > > feedback.
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Now pasting some
> >> > > > > performance
> >> > > > > > > > > numbers
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Table 1: C++
> >> ALP
> >> > > > Double
> >> > > > > > > > Decode —
> >> > > > > > > > > > >> >> Spotify
> >> > > > > > > > > > >> >> >>> > Columns
> >> > > > > > > > > > >> >> >>> > > > > > >>> (Graviton
> >> > > > > > > > > > >> >> >>> > > > > > >>> > 3,
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> ARM
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Neoverse V1)
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>>
> >> > > > > > > > > > >> >> >>>
> >> > > > > > > ┌──────────────────┬──────────────┬──────────────┬─────────┐
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ Column
> >> > │
> >> > > > > -O2
> >> > > > > > > > (MB/s)
> >> > > > > > > > > > │
> >> > > > > > > > > > >> >> -O3
> >> > > > > > > > > > >> >> >>> > > (MB/s) │
> >> > > > > > > > > > >> >> >>> > > > > > >>> Speedup │
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>>
> >> > > > > > > > > > >> >> >>>
> >> > > > > > > ├──────────────────┼──────────────┼──────────────┼─────────┤
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ valence
> >> > │
> >> > > > > > > 3,155
> >> > > > > > > > > > │
> >> > > > > > > > > > >> >> >>> 5,523
> >> > > > > > > > > > >> >> >>> > > │
> >> > > > > > > > > > >> >> >>> > > > > > >>> 1.75x │
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │
> danceability
> >> > │
> >> > > > > > > 3,233
> >> > > > > > > > > > │
> >> > > > > > > > > > >> >> >>> 5,685
> >> > > > > > > > > > >> >> >>> > > │
> >> > > > > > > > > > >> >> >>> > > > > > >>> 1.76x │
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ energy
> >> > │
> >> > > > > > > 3,197
> >> > > > > > > > > > │
> >> > > > > > > > > > >> >> >>> 5,652
> >> > > > > > > > > > >> >> >>> > > │
> >> > > > > > > > > > >> >> >>> > > > > > >>> 1.77x │
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ loudness
> >> > │
> >> > > > > > > 3,186
> >> > > > > > > > > > │
> >> > > > > > > > > > >> >> >>> 5,473
> >> > > > > > > > > > >> >> >>> > > │
> >> > > > > > > > > > >> >> >>> > > > > > >>> 1.72x │
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>>
> >> > > > > > > > > > >> >> >>>
> >> > > > > > > └──────────────────┴──────────────┴──────────────┴─────────┘
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> On Wed, Feb 25,
> >> 2026
> >> > at
> >> > > > > > 9:49 AM
> >> > > > > > > > > > PRATEEK
> >> > > > > > > > > > >> >> GAUR
> >> > > > > > > > > > >> >> >>> <
> >> > > > > > > > > > >> >> >>> > > > > > >>> > prtkgaur@gmail.com
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> >
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> wrote:
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> @Micah
> Kornfield
> >> <
> >> > > > > > > > > > >> emkornfield@gmail.com>
> >> > > > > > > > > > >> >> :
> >> > > > > > > > > > >> >> >>> Got
> >> > > > > > > > > > >> >> >>> > > it.
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> @Andrew Lamb <
> >> > > > > > > > > > andrewlamb11@gmail.com>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> Do you think
> it
> >> > would
> >> > > > be
> >> > > > > > good
> >> > > > > > > > to
> >> > > > > > > > > > >> start
> >> > > > > > > > > > >> >> >>> moving
> >> > > > > > > > > > >> >> >>> > the
> >> > > > > > > > > > >> >> >>> > > > > spec
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> development
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> into
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> markdown
> >> format, in
> >> > > > > > > preparation
> >> > > > > > > > > for
> >> > > > > > > > > > >> >> >>> finalizing
> >> > > > > > > > > > >> >> >>> > > it?
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> Yes I'll update
> >> the
> >> > > > > numbers
> >> > > > > > > for
> >> > > > > > > > > some
> >> > > > > > > > > > >> of
> >> > > > > > > > > > >> >> the
> >> > > > > > > > > > >> >> >>> > > > examples I
> >> > > > > > > > > > >> >> >>> > > > > > >>> have in
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> the
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> spec based
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> on the updated
> >> > header
> >> > > > > size.
> >> > > > > > > Then
> >> > > > > > > > > we
> >> > > > > > > > > > >> >> should
> >> > > > > > > > > > >> >> >>> be
> >> > > > > > > > > > >> >> >>> > good
> >> > > > > > > > > > >> >> >>> > > > to
> >> > > > > > > > > > >> >> >>> > > > > go
> >> > > > > > > > > > >> >> >>> > > > > > >>> for
> >> > > > > > > > > > >> >> >>> > > > > > >>> > the
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> markdown
> format.
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> Thanks
> everyone!
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> Andrew
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> On Tue, Feb
> 17,
> >> > 2026
> >> > > at
> >> > > > > > > 7:28 PM
> >> > > > > > > > > > >> PRATEEK
> >> > > > > > > > > > >> >> >>> GAUR <
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> prtkgaur@gmail.com>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> wrote:
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Hi team,
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 1) Andrew
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks
> >> for
> >> > > > working
> >> > > > > > on
> >> > > > > > > > test
> >> > > > > > > > > > >> files.
> >> > > > > > > > > > >> >> >>> My PR
> >> > > > > > > > > > >> >> >>> > > did
> >> > > > > > > > > > >> >> >>> > > > > add
> >> > > > > > > > > > >> >> >>> > > > > > >>> all
> >> > > > > > > > > > >> >> >>> > > > > > >>> > the
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > test
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> files I
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> used to
> >> > > benchmark
> >> > > > on
> >> > > > > > > > > datasets.
> >> > > > > > > > > > >> >> Maybe
> >> > > > > > > > > > >> >> >>> we
> >> > > > > > > > > > >> >> >>> > can
> >> > > > > > > > > > >> >> >>> > > > club
> >> > > > > > > > > > >> >> >>> > > > > > it
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> together.
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> WIll also
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> aid
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> cross
> >> language
> >> > > > > testing
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Kosta
> >> > Tarasov
> >> > > > > > working
> >> > > > > > > > on
> >> > > > > > > > > > Rust
> >> > > > > > > > > > >> >> >>> > > > implementation.
> >> > > > > > > > > > >> >> >>> > > > > > >>> This is
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > great.
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> Thanks
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 2) Antoine
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks
> a
> >> lot
> >> > > for
> >> > > > > > > > reporting
> >> > > > > > > > > > the
> >> > > > > > > > > > >> >> >>> numbers
> >> > > > > > > > > > >> >> >>> > on
> >> > > > > > > > > > >> >> >>> > > > AMD.
> >> > > > > > > > > > >> >> >>> > > > > > >>> Looks
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> like
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > you
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> are
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> getting
> 8X
> >> the
> >> > > > > > decoding
> >> > > > > > > > > > >> >> performance of
> >> > > > > > > > > > >> >> >>> > BSS.
> >> > > > > > > > > > >> >> >>> > > > This
> >> > > > > > > > > > >> >> >>> > > > > > is
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > amazing!!.
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks
> >> for
> >> > > > > > > acknowledging
> >> > > > > > > > > the
> >> > > > > > > > > > >> >> >>> sampling
> >> > > > > > > > > > >> >> >>> > > > design.
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - I agree
> >> with
> >> > > you
> >> > > > > on
> >> > > > > > > > > > >> Fastlanes. In
> >> > > > > > > > > > >> >> >>> some
> >> > > > > > > > > > >> >> >>> > > crude
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> experiments I
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> didn't get
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> a good
> perf
> >> > > > benefit
> >> > > > > > from
> >> > > > > > > > it
> >> > > > > > > > > on
> >> > > > > > > > > > >> >> >>> Graviton3
> >> > > > > > > > > > >> >> >>> > > (but
> >> > > > > > > > > > >> >> >>> > > > > > maybe
> >> > > > > > > > > > >> >> >>> > > > > > >>> > there
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> was
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> something
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> wrong
> with
> >> my
> >> > > > > > > > > implementation).
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Locking
> >> the
> >> > > > 16bit
> >> > > > > > > > > exception
> >> > > > > > > > > > >> >> >>> encoding for
> >> > > > > > > > > > >> >> >>> > > the
> >> > > > > > > > > > >> >> >>> > > > > > spec
> >> > > > > > > > > > >> >> >>> > > > > > >>> in
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> this
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> case.
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> -
> Awesome I
> >> > > think
> >> > > > we
> >> > > > > > > have
> >> > > > > > > > > > solved
> >> > > > > > > > > > >> >> for
> >> > > > > > > > > > >> >> >>> all
> >> > > > > > > > > > >> >> >>> > > open
> >> > > > > > > > > > >> >> >>> > > > > > >>> questions
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> minus
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> the
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> version
> >> byte
> >> > :).
> >> > > > > (will
> >> > > > > > > get
> >> > > > > > > > > > back
> >> > > > > > > > > > >> on
> >> > > > > > > > > > >> >> >>> this
> >> > > > > > > > > > >> >> >>> > > soon)
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 3) Micah
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> -
> >> FastLanes :
> >> > > The
> >> > > > > > > current
> >> > > > > > > > > spec
> >> > > > > > > > > > >> does
> >> > > > > > > > > > >> >> >>> allow
> >> > > > > > > > > > >> >> >>> > > for
> >> > > > > > > > > > >> >> >>> > > > > > using
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> FastLane
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> with the
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> configurable
> >> > > enum
> >> > > > > > value
> >> > > > > > > > for
> >> > > > > > > > > > >> >> layout. We
> >> > > > > > > > > > >> >> >>> > > should
> >> > > > > > > > > > >> >> >>> > > > be
> >> > > > > > > > > > >> >> >>> > > > > > >>> able to
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > inject
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> any
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> layout
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> in the
> >> current
> >> > > > > design.
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Working on
> >> > resolving
> >> > > > all
> >> > > > > > > > > remaining
> >> > > > > > > > > > >> open
> >> > > > > > > > > > >> >> >>> > comments
> >> > > > > > > > > > >> >> >>> > > > on
> >> > > > > > > > > > >> >> >>> > > > > > the
> >> > > > > > > > > > >> >> >>> > > > > > >>> spec
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> this
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> week.
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Best
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Prateek
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> On Tue, Feb
> 10,
> >> > 2026
> >> > > > at
> >> > > > > > > > 3:37 AM
> >> > > > > > > > > > >> Steve
> >> > > > > > > > > > >> >> >>> > Loughran <
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> >> > stevel@cloudera.com>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> wrote:
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> On Sun, 8
> Feb
> >> > 2026
> >> > > at
> >> > > > > > > 18:12,
> >> > > > > > > > > > Micah
> >> > > > > > > > > > >> >> >>> Kornfield
> >> > > > > > > > > > >> >> >>> > <
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> >> > > emkornfield@gmail.com>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> wrote:
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> It looks
> like
> >> > the
> >> > > > > actual
> >> > > > > > > > issue
> >> > > > > > > > > > >> >> >>> described for
> >> > > > > > > > > > >> >> >>> > > ORC
> >> > > > > > > > > > >> >> >>> > > > > in
> >> > > > > > > > > > >> >> >>> > > > > > >>> the
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> paper
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> is that
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> it
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> has
> multiple
> >> > > > > > sub-encodings
> >> > > > > > > > in
> >> > > > > > > > > a
> >> > > > > > > > > > >> >> batch.
> >> > > > > > > > > > >> >> >>> This
> >> > > > > > > > > > >> >> >>> > > is
> >> > > > > > > > > > >> >> >>> > > > > > >>> different
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> then
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> the
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> design
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> proposed
> here
> >> > > where
> >> > > > > > there
> >> > > > > > > is
> >> > > > > > > > > > still
> >> > > > > > > > > > >> >> fixed
> >> > > > > > > > > > >> >> >>> > > > encoding
> >> > > > > > > > > > >> >> >>> > > > > > per
> >> > > > > > > > > > >> >> >>> > > > > > >>> page
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> in
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> parquet.
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> Given
> >> reasonably
> >> > > > sized
> >> > > > > > > > pages I
> >> > > > > > > > > > >> don't
> >> > > > > > > > > > >> >> >>> think
> >> > > > > > > > > > >> >> >>> > > > branch
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> misprediction
> >> > should
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> be a
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> big issue
> for
> >> > new
> >> > > > > > > encodings.
> >> > > > > > > > > I
> >> > > > > > > > > > >> agree
> >> > > > > > > > > > >> >> >>> that
> >> > > > > > > > > > >> >> >>> > we
> >> > > > > > > > > > >> >> >>> > > > > should
> >> > > > > > > > > > >> >> >>> > > > > > >>> be
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> conservative
> in
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> general for
> >> > adding
> >> > > > new
> >> > > > > > > > > > encodings.
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> +1
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> >
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> >
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >> >
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >>
> >> > > > > > > > > > >> >> >>> > > > > > >>> > >
> >> > > > > > > > > > >> >> >>> > > > > > >>> >
> >> > > > > > > > > > >> >> >>> > > > > > >>>
> >> > > > > > > > > > >> >> >>> > > > > > >>
> >> > > > > > > > > > >> >> >>> > > > > >
> >> > > > > > > > > > >> >> >>> > > > >
> >> > > > > > > > > > >> >> >>> > > >
> >> > > > > > > > > > >> >> >>> > >
> >> > > > > > > > > > >> >> >>> >
> >> > > > > > > > > > >> >> >>>
> >> > > > > > > > > > >> >> >>
> >> > > > > > > > > > >> >>
> >> > > > > > > > > > >> >
> >> > > > > > > > > > >>
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Gunnar Morling via dev <de...@parquet.apache.org>.
Hey Gang,
Yeah, agreed, I think it's more similar in the sense that we take the
samples for the preset from across the whole chunk instead of only from its
first segment. AFAIK, DuckDB also resamples and then uses dedicated presets
for specific segments, which can yield even better results but also is more
costly.
As for a perf comparison, I have a quick informal (LLM-coded)
microbenchmark of parquet-java's and Hardwood's ALP decoders and encoders
in this draft PR [1]. In that benchmark, in a Docker container on my
laptop, Hardwood decodes 5–6× faster than parquet-java and encodes a page
2–4× faster once both presets exist (pages purely in memory, no
compression, etc.). Based on some quick profiling, IIUC, it mostly comes
down to bit unpacking on decode (generic 8-at-a-time BytePacker plus a
separate decode pass vs. one fused loop with a single 8-byte load per
value), and on encode to parquet-java evaluating each preset pair on the
full vector rather than a sample.
Numbers may look different elsewhere, so I'd be curious to learn about your
results, also if there's something to do better in the benchmark itself,
etc.
Best,
--Gunnar
[1] https://github.com/hardwood-hq/hardwood/pull/1775
On Thu, 12 Nov 2026 at 13:42, Gang Wu <us...@gmail.com> wrote:
> Thanks Gunnar for sharing this!
>
> I think the encoder approach used by Hardwood looks more similar to the ALP
> paper where all 102,400 values are buffered to decide best presets and then
> encode every vector based on those presets.
>
> I'm trying to improve the ALP implementations on the Parquet C++ side:
> https://github.com/apache/arrow/pull/55449. So I'm really interested
> if you have any document on performance comparison among different
> implementations.
>
> Thanks!
> Gang
>
> On Thu, Nov 12, 2026 at 6:50 PM Gunnar Morling via dev <
> dev@parquet.apache.org> wrote:
>
>> Hey all,
>>
>> We've added ALP encoding and decoding support to Hardwood too [1], [2].
>> Interop with parquet-java's implementation looks good: Hardwood decodes
>> pages written by parquet-java's ALP writer bit-exactly and vice versa.
>>
>> Perhaps interesting: on the encoder side, we build the preset of a column
>> chunk from eight sample vectors distributed across the entire chunk (easy
>> in our case, as Hardwood's writer buffers whole chunks), whereas, IIUC,
>> parquet-java only samples within the first ~110K values of a chunk.
>> Depending on the data distribution, this can yield a more efficient
>> encoding, for instance when a high-cardinality column changes character
>> midway, e.g. moving from two to four decimal places (about 2.8× smaller in
>> a synthetic test with 1M doubles).
>>
>> Best,
>>
>> --Gunnar
>>
>> [1] https://github.com/hardwood-hq/hardwood/pull/1496
>> [2] https://github.com/hardwood-hq/hardwood/pull/1769
>>
>>
>> On Thu, 12 Nov 2026 at 07:01, PRATEEK GAUR <pr...@gmail.com> wrote:
>>
>> > Thanks everyone!
>> >
>> > On Mon, Nov 9, 2026 at 11:58 AM Vinoo Ganesh <vi...@gmail.com>
>> > wrote:
>> >
>> > > Thanks everyone!
>> > >
>> > > Thanks,
>> > > Vinoo Ganesh | vinoo.ganesh@gmail.com
>> > >
>> > > <Vi...@gmail.com>
>> > >
>> > >
>> > > On Mon, Nov 9, 2026 at 2:49 PM Andrew Lamb <an...@gmail.com>
>> > wrote:
>> > >
>> > > > The Java implementation was just merged[1]!
>> > > >
>> > > > Thank you Gang, Vinoo, Prateek and everyone who helped push that one
>> > > along
>> > > >
>> > > > Andrew
>> > > >
>> > > > [1]:
>> > https://github.com/apache/parquet-java/pull/3397#event-35545451335
>> > > >
>> > > > On Tue, Oct 13, 2026 at 12:52 PM PRATEEK GAUR <pr...@gmail.com>
>> > > wrote:
>> > > >
>> > > > > Hi team,
>> > > > >
>> > > > > Quick update re: the C++ implementation.
>> > > > > Thanks to all the reviewers, we merged the C++ implementation
>> today.
>> > > > > There are a few follow-up items that I'll address in a subsequent
>> PR.
>> > > > >
>> > > > > Thanks
>> > > > > Prateek
>> > > > >
>> > > > > On Sun, Sep 13, 2026 at 6:54 PM Vinoo Ganesh <
>> vinoo.ganesh@gmail.com
>> > >
>> > > > > wrote:
>> > > > >
>> > > > > > Quick update re: the Java impl. Thanks for the feedback, on
>> > > > > > https://github.com/apache/parquet-java/pull/3397, Russel! I
>> pulled
>> > > > > > https://github.com/apache/parquet-java/pull/3791 into its own
>> PR
>> > > > (thanks
>> > > > > > for the quick approval too, Gang). Assuming no objections, I
>> think
>> > we
>> > > > can
>> > > > > > merge #3791, and then go ahead of #3397.
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Vinoo Ganesh | vinoo.ganesh@gmail.com
>> > > > > >
>> > > > > > <Vi...@gmail.com>
>> > > > > >
>> > > > > >
>> > > > > > On Tue, Sep 8, 2026 at 10:54 AM PRATEEK GAUR <
>> prtkgaur@gmail.com>
>> > > > wrote:
>> > > > > >
>> > > > > > > [Rust] Awesome thanks Andrew, Kostas and Devin.
>> > > > > > > [Java] Thanks Gang for the feedback on Java PR.
>> > > > > > > [C++] Thanks Gang for another round of feedback on the C++
>> one.
>> > > > > > > [Go] Arnav and I will be working on the golang impl this week.
>> > > > > > >
>> > > > > > > This is great progress all around.
>> > > > > > >
>> > > > > > > On Sat, Sep 5, 2026 at 1:26 PM Vinoo Ganesh <
>> > > vinoo.ganesh@gmail.com>
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > Thanks all - I'm addressing Gang's comments on
>> > > > > > > > https://github.com/apache/parquet-java/pull/3397 now, so
>> > > hopefully
>> > > > > > well
>> > > > > > > > also be in a good state for merging the java implementation
>> > soon.
>> > > > > > > >
>> > > > > > > > Thanks,
>> > > > > > > > Vinoo Ganesh | vinoo.ganesh@gmail.com
>> > > > > > > >
>> > > > > > > > <Vi...@gmail.com>
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Sat, Sep 5, 2026 at 8:56 AM Andrew Lamb <
>> > > andrewlamb11@gmail.com
>> > > > >
>> > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > We just merged the Rust implementation[1] from Kosta and
>> > Devan.
>> > > > > With
>> > > > > > > the
>> > > > > > > > > imminent release of Parquet Format 2.14.0 I think we'll be
>> > well
>> > > > on
>> > > > > > our
>> > > > > > > > way.
>> > > > > > > > >
>> > > > > > > > > Thanks again for pushing this along,
>> > > > > > > > > Andrew
>> > > > > > > > >
>> > > > > > > > > [1]: https://github.com/apache/arrow-rs/pull/9372
>> > > > > > > > >
>> > > > > > > > > On Fri, Sep 4, 2026 at 5:42 PM PRATEEK GAUR <
>> > > prtkgaur@gmail.com>
>> > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > Hi team,
>> > > > > > > > > >
>> > > > > > > > > > Thanks Micah for multiple rounds of review and thanks
>> Gang
>> > > for
>> > > > > > > taking a
>> > > > > > > > > > look at the PR and giving feedback. I've addressed all
>> > > comments
>> > > > > in
>> > > > > > > the
>> > > > > > > > > PR.
>> > > > > > > > > > Looking forward to the next round of feedback (or
>> > approval).
>> > > > > Would
>> > > > > > > like
>> > > > > > > > > to
>> > > > > > > > > > get it in soon.
>> > > > > > > > > >
>> > > > > > > > > > https://github.com/apache/arrow/pull/48345/changes
>> > > > > > > > > >
>> > > > > > > > > > With this done and after talking to Arnav, I plan to
>> spend
>> > > some
>> > > > > > time
>> > > > > > > on
>> > > > > > > > > the
>> > > > > > > > > > ALP implementation in the arrow-go repository helping
>> > spread
>> > > > the
>> > > > > > > > support
>> > > > > > > > > to
>> > > > > > > > > > more languages.
>> > > > > > > > > >
>> > > > > > > > > > Best
>> > > > > > > > > > Prateek
>> > > > > > > > > >
>> > > > > > > > > > On Sun, Aug 2, 2026 at 10:42 PM PRATEEK GAUR <
>> > > > prtkgaur@gmail.com
>> > > > > >
>> > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > Thanks Divjot for bringing it up.
>> > > > > > > > > > >
>> > > > > > > > > > > I was under the same impression as Russell and Andrew
>> > that
>> > > > we'd
>> > > > > > > have
>> > > > > > > > it
>> > > > > > > > > > > flag gated.
>> > > > > > > > > > > With the new versioning proposal 'preview' might be a
>> > good
>> > > > tag
>> > > > > > for
>> > > > > > > > it.
>> > > > > > > > > > >
>> > > > > > > > > > > Best
>> > > > > > > > > > > Prateek
>> > > > > > > > > > >
>> > > > > > > > > > > On Sat, Aug 1, 2026 at 12:39 AM Andrew Lamb <
>> > > > > > > andrewlamb11@gmail.com>
>> > > > > > > > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > >> Div: Here is a proposal to mark ALP in Preview: [1]
>> > > (using a
>> > > > > > > comment
>> > > > > > > > > in
>> > > > > > > > > > >> the
>> > > > > > > > > > >> spec section).
>> > > > > > > > > > >>
>> > > > > > > > > > >> Russel: That is my understanding as well.
>> > > > > > > > > > >>
>> > > > > > > > > > >> Hopefully we are able to formalize the versioning
>> > process
>> > > > > > shortly
>> > > > > > > so
>> > > > > > > > > we
>> > > > > > > > > > >> can
>> > > > > > > > > > >> formalize the notion of "preview"
>> > > > > > > > > > >>
>> > > > > > > > > > >> Andrew
>> > > > > > > > > > >>
>> > > > > > > > > > >>
>> > > > > > > > > > >> [1]:
>> https://github.com/apache/parquet-format/pull/604
>> > > > > > > > > > >>
>> > > > > > > > > > >> On Fri, Jul 31, 2026 at 11:15 AM Russell Spitzer <
>> > > > > > > > > > >> russell.spitzer@gmail.com>
>> > > > > > > > > > >> wrote:
>> > > > > > > > > > >>
>> > > > > > > > > > >> > I thought we had agreed it would be merged, and
>> > features
>> > > > > like
>> > > > > > > this
>> > > > > > > > > > would
>> > > > > > > > > > >> > be feature flag gated. So a V-Old writer could
>> always
>> > > opt
>> > > > in
>> > > > > > to
>> > > > > > > > > using
>> > > > > > > > > > >> this
>> > > > > > > > > > >> > encoding but that must be a conscious effort if you
>> > are
>> > > > not
>> > > > > > > using
>> > > > > > > > > > VNext
>> > > > > > > > > > >> of
>> > > > > > > > > > >> > Parquet. If you are on V-Next the writer can
>> choose to
>> > > use
>> > > > > ALP
>> > > > > > > > > > whenever
>> > > > > > > > > > >> it
>> > > > > > > > > > >> > likes without user intervention.
>> > > > > > > > > > >> >
>> > > > > > > > > > >> > On Fri, Jul 31, 2026 at 2:20 AM Divjot Arora via
>> dev <
>> > > > > > > > > > >> > dev@parquet.apache.org> wrote:
>> > > > > > > > > > >> >
>> > > > > > > > > > >> >> Yes, that’s my understanding as well. However, as
>> you
>> > > > > noted,
>> > > > > > > the
>> > > > > > > > > > >> >> versioning
>> > > > > > > > > > >> >> discussion is still in early stages and hasn’t
>> > > prescribed
>> > > > > > > > concrete
>> > > > > > > > > > >> >> mechanisms yet. If ALP goes into parquet-format
>> > 2.14.0,
>> > > > > > there’s
>> > > > > > > > no
>> > > > > > > > > > >> >> indication that it’s in preview or that writers
>> > > shouldn’t
>> > > > > use
>> > > > > > > it
>> > > > > > > > by
>> > > > > > > > > > >> >> default
>> > > > > > > > > > >> >> yet. Given that we’ve held off on merging other
>> > > > > incompatible
>> > > > > > > > > changes
>> > > > > > > > > > >> (e.g
>> > > > > > > > > > >> >> removal of path_in_schema) until the versioning
>> > > > discussion
>> > > > > > > > closes,
>> > > > > > > > > my
>> > > > > > > > > > >> >> preference would be to leave ALP un-merged for the
>> > time
>> > > > > being
>> > > > > > > as
>> > > > > > > > > > well.
>> > > > > > > > > > >> >>
>> > > > > > > > > > >> >> — Div
>> > > > > > > > > > >> >>
>> > > > > > > > > > >> >> On Fri, Jul 31, 2026 at 08:59 Andrew Lamb <
>> > > > > > > > andrewlamb11@gmail.com>
>> > > > > > > > > > >> wrote:
>> > > > > > > > > > >> >>
>> > > > > > > > > > >> >> > After re-reading Julien's Versioning Proposal,
>> my
>> > > > > > > understanding
>> > > > > > > > > is
>> > > > > > > > > > >> that
>> > > > > > > > > > >> >> if
>> > > > > > > > > > >> >> > we adopt that system, we will somehow mark ALP
>> in
>> > > > > "Preview"
>> > > > > > > in
>> > > > > > > > > > >> >> > parquet-format for release Version 3.
>> > > > > > > > > > >> >> >
>> > > > > > > > > > >> >> > The versioning document doesn't seem to
>> proscribe
>> > the
>> > > > > > > > particular
>> > > > > > > > > > >> method
>> > > > > > > > > > >> >> > for annotating "preview" features -- I think
>> > clearly
>> > > > > > marking
>> > > > > > > > them
>> > > > > > > > > > as
>> > > > > > > > > > >> >> such
>> > > > > > > > > > >> >> > in parquet.thrift as well as reflecting that on
>> the
>> > > > > > > > > > >> >> >
>> > > https://parquet.apache.org/docs/file-format/versions/
>> > > > > page
>> > > > > > > > > somehow
>> > > > > > > > > > >> >> would
>> > > > > > > > > > >> >> > be adequate.
>> > > > > > > > > > >> >> >
>> > > > > > > > > > >> >> > Andrew
>> > > > > > > > > > >> >> >
>> > > > > > > > > > >> >> > On Thu, Jul 30, 2026 at 11:50 AM Andrew Lamb <
>> > > > > > > > > > andrewlamb11@gmail.com
>> > > > > > > > > > >> >
>> > > > > > > > > > >> >> > wrote:
>> > > > > > > > > > >> >> >
>> > > > > > > > > > >> >> >> In my mind ALP is in the same position as other
>> > > > > encodings
>> > > > > > > that
>> > > > > > > > > > were
>> > > > > > > > > > >> >> added
>> > > > > > > > > > >> >> >> recently[1] (such as BYTE_STREAM_SPLIT for non
>> > > > floating
>> > > > > > > point
>> > > > > > > > > > types)
>> > > > > > > > > > >> >> >>
>> > > > > > > > > > >> >> >> That is to say it will be available in the
>> latest
>> > > open
>> > > > > > > source
>> > > > > > > > > > >> writers
>> > > > > > > > > > >> >> and
>> > > > > > > > > > >> >> >> readers, but realistically not widely used in
>> the
>> > > > > > ecosystem
>> > > > > > > > > where
>> > > > > > > > > > >> >> >> compatibility is a concern (a large part)
>> > > > > > > > > > >> >> >>
>> > > > > > > > > > >> >> >> I view Julian's work[2][3] to gather consensus
>> on
>> > > how
>> > > > to
>> > > > > > > name
>> > > > > > > > > > >> >> >> combinations of features (e.g. "versions") as
>> the
>> > > > > > mechanism
>> > > > > > > by
>> > > > > > > > > > >> which we
>> > > > > > > > > > >> >> >> will accelerate the adoption of this change
>> (along
>> > > > with
>> > > > > > > other
>> > > > > > > > > > >> forward
>> > > > > > > > > > >> >> >> incompatible changes) across the ecosystem.
>> > > > > > > > > > >> >> >>
>> > > > > > > > > > >> >> >> > Will we exclude this change when cutting
>> > > > > parquet-format
>> > > > > > > > > v2.14.0?
>> > > > > > > > > > >> >> >>
>> > > > > > > > > > >> >> >> No I don't think so -- I would expect it to go
>> out
>> > > in
>> > > > > the
>> > > > > > > next
>> > > > > > > > > > >> >> >> parquet-format version as do other changes in
>> the
>> > > > spec.
>> > > > > > > > > > >> >> >>
>> > > > > > > > > > >> >> >> Andrew
>> > > > > > > > > > >> >> >>
>> > > > > > > > > > >> >> >> [1]:
>> > > > > > https://parquet.apache.org/docs/file-format/versions/
>> > > > > > > > > > >> >> >> [2]:
>> > > > > > > > > > >>
>> > > > > >
>> https://lists.apache.org/thread/xmnj8h0h8ozmrgox5tydhhs7yz7q2hss
>> > > > > > > > > > >> >> >> [3]:
>> > > > > > > > > > >>
>> > > > > >
>> https://lists.apache.org/thread/qlf8lg90gqq41lllyy9mk2f0skqvftqj
>> > > > > > > > > > >> >> >>
>> > > > > > > > > > >> >> >>
>> > > > > > > > > > >> >> >> On Thu, Jul 30, 2026 at 3:51 AM Divjot Arora
>> via
>> > > dev <
>> > > > > > > > > > >> >> >> dev@parquet.apache.org> wrote:
>> > > > > > > > > > >> >> >>
>> > > > > > > > > > >> >> >>> Hi Andrew,
>> > > > > > > > > > >> >> >>>
>> > > > > > > > > > >> >> >>> How does this interact with the ongoing
>> > versioning
>> > > > > > > > discussion?
>> > > > > > > > > > >> Given
>> > > > > > > > > > >> >> that
>> > > > > > > > > > >> >> >>> new encodings are not forward compatible, I
>> would
>> > > > > assume
>> > > > > > > ALP
>> > > > > > > > is
>> > > > > > > > > > >> >> included
>> > > > > > > > > > >> >> >>> in
>> > > > > > > > > > >> >> >>> Parquet V<next> (V3?), but it's not clear to
>> me
>> > how
>> > > > > that
>> > > > > > > > works
>> > > > > > > > > if
>> > > > > > > > > > >> it's
>> > > > > > > > > > >> >> >>> merged to the parquet-format master branch
>> now.
>> > > Will
>> > > > we
>> > > > > > > > exclude
>> > > > > > > > > > >> this
>> > > > > > > > > > >> >> >>> change
>> > > > > > > > > > >> >> >>> when cutting parquet-format v2.14.0?
>> > > > > > > > > > >> >> >>>
>> > > > > > > > > > >> >> >>> -- Div
>> > > > > > > > > > >> >> >>>
>> > > > > > > > > > >> >> >>> On Thu, Jul 30, 2026 at 8:51 AM Andrew Lamb <
>> > > > > > > > > > >> andrewlamb11@gmail.com>
>> > > > > > > > > > >> >> >>> wrote:
>> > > > > > > > > > >> >> >>>
>> > > > > > > > > > >> >> >>> > Hello,
>> > > > > > > > > > >> >> >>> >
>> > > > > > > > > > >> >> >>> > As an update, thanks to the epic work from
>> many
>> > > > > > > > contributors,
>> > > > > > > > > > >> >> >>> especially
>> > > > > > > > > > >> >> >>> > Prateek, I just merged the parquet-format
>> > change
>> > > > for
>> > > > > > > > ALP[1].
>> > > > > > > > > We
>> > > > > > > > > > >> >> plan to
>> > > > > > > > > > >> >> >>> > make additional editorial The remaining
>> items I
>> > > > know
>> > > > > of
>> > > > > > > are
>> > > > > > > > > > >> merging
>> > > > > > > > > > >> >> the
>> > > > > > > > > > >> >> >>> > examples from parquet-testing[2], and then
>> of
>> > > > course
>> > > > > > the
>> > > > > > > > > > various
>> > > > > > > > > > >> >> >>> > implementations
>> > > > > > > > > > >> >> >>> >
>> > > > > > > > > > >> >> >>> > Thanks again everyone. This is a great step
>> > > forward
>> > > > > > > > > > >> >> >>> > Andrew
>> > > > > > > > > > >> >> >>> >
>> > > > > > > > > > >> >> >>> > [1]:
>> > > > > https://github.com/apache/parquet-format/pull/557
>> > > > > > > > > > >> >> >>> > [2]:
>> > > > > > https://github.com/apache/parquet-testing/pull/100
>> > > > > > > > > > >> >> >>> >
>> > > > > > > > > > >> >> >>> > On Tue, Jul 21, 2026 at 4:39 PM Julien Le
>> Dem <
>> > > > > > > > > > julien@apache.org
>> > > > > > > > > > >> >
>> > > > > > > > > > >> >> >>> wrote:
>> > > > > > > > > > >> >> >>> >
>> > > > > > > > > > >> >> >>> > > It was also in my spam folder... :(
>> > > > > > > > > > >> >> >>> > > I've seen a bunch of apache list emails
>> go to
>> > > my
>> > > > > spam
>> > > > > > > > > > recently.
>> > > > > > > > > > >> >> >>> > >
>> > > > > > > > > > >> >> >>> > > On Tue, Jul 21, 2026 at 10:58 AM Russell
>> > > Spitzer
>> > > > <
>> > > > > > > > > > >> >> >>> > > russell.spitzer@gmail.com>
>> > > > > > > > > > >> >> >>> > > wrote:
>> > > > > > > > > > >> >> >>> > >
>> > > > > > > > > > >> >> >>> > > > Yes, my hope was that my vote would
>> kick it
>> > > out
>> > > > > of
>> > > > > > > spam
>> > > > > > > > > but
>> > > > > > > > > > >> we
>> > > > > > > > > > >> >> >>> > definitely
>> > > > > > > > > > >> >> >>> > > > had two other votes that also went to
>> spam
>> > > for
>> > > > > me.
>> > > > > > > > > > >> >> >>> > > >
>> > > > > > > > > > >> >> >>> > > > For anyone looking the thread is here
>> > > > > > > > > > >> >> >>> > > >
>> > > > > > > > > > >> >>
>> > > > > > >
>> https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt
>> > > > > > > > > > >> >> >>> > > >
>> > > > > > > > > > >> >> >>> > > > On Tue, Jul 21, 2026 at 10:57 AM Micah
>> > > > Kornfield
>> > > > > <
>> > > > > > > > > > >> >> >>> > emkornfield@gmail.com>
>> > > > > > > > > > >> >> >>> > > > wrote:
>> > > > > > > > > > >> >> >>> > > >
>> > > > > > > > > > >> >> >>> > > > > I've seen other's votes not go to
>> spam,
>> > so
>> > > > > > > hopefully
>> > > > > > > > > > people
>> > > > > > > > > > >> >> will
>> > > > > > > > > > >> >> >>> see
>> > > > > > > > > > >> >> >>> > it
>> > > > > > > > > > >> >> >>> > > > in
>> > > > > > > > > > >> >> >>> > > > > there inboxes (or see this thread and
>> > look
>> > > in
>> > > > > > there
>> > > > > > > > > > spam).
>> > > > > > > > > > >> >> >>> > > > >
>> > > > > > > > > > >> >> >>> > > > > On Tue, Jul 21, 2026 at 10:52 AM
>> PRATEEK
>> > > > GAUR <
>> > > > > > > > > > >> >> >>> prtkgaur@gmail.com>
>> > > > > > > > > > >> >> >>> > > > wrote:
>> > > > > > > > > > >> >> >>> > > > >
>> > > > > > > > > > >> >> >>> > > > > > Okay,
>> > > > > > > > > > >> >> >>> > > > > >
>> > > > > > > > > > >> >> >>> > > > > > I sent out the voting email
>> yesterday
>> > and
>> > > > so
>> > > > > > far
>> > > > > > > > I've
>> > > > > > > > > > >> heard
>> > > > > > > > > > >> >> >>> from 3
>> > > > > > > > > > >> >> >>> > > > people
>> > > > > > > > > > >> >> >>> > > > > > that it ended up in their "spam
>> > folder".
>> > > > Dev
>> > > > > > VPN
>> > > > > > > to
>> > > > > > > > > be
>> > > > > > > > > > >> >> blamed
>> > > > > > > > > > >> >> >>> for
>> > > > > > > > > > >> >> >>> > > it?.
>> > > > > > > > > > >> >> >>> > > > > >
>> > > > > > > > > > >> >> >>> > > > > > What's the protocol here?
>> > > > > > > > > > >> >> >>> > > > > > I let people hunt for the email in
>> > their
>> > > > spam
>> > > > > > > > folder
>> > > > > > > > > or
>> > > > > > > > > > >> send
>> > > > > > > > > > >> >> >>> out
>> > > > > > > > > > >> >> >>> > > > another
>> > > > > > > > > > >> >> >>> > > > > > one?
>> > > > > > > > > > >> >> >>> > > > > >
>> > > > > > > > > > >> >> >>> > > > > > Best
>> > > > > > > > > > >> >> >>> > > > > > Prateek
>> > > > > > > > > > >> >> >>> > > > > >
>> > > > > > > > > > >> >> >>> > > > > > On Sun, Jul 19, 2026 at 10:47 PM
>> > PRATEEK
>> > > > > GAUR <
>> > > > > > > > > > >> >> >>> prtkgaur@gmail.com>
>> > > > > > > > > > >> >> >>> > > > > wrote:
>> > > > > > > > > > >> >> >>> > > > > >
>> > > > > > > > > > >> >> >>> > > > > > > Hi team,
>> > > > > > > > > > >> >> >>> > > > > > >
>> > > > > > > > > > >> >> >>> > > > > > > Vinoo has added the tests which
>> were
>> > > > > > requested
>> > > > > > > in
>> > > > > > > > > the
>> > > > > > > > > > >> last
>> > > > > > > > > > >> >> >>> round
>> > > > > > > > > > >> >> >>> > of
>> > > > > > > > > > >> >> >>> > > > > > review.
>> > > > > > > > > > >> >> >>> > > > > > > In his own words.
>> > > > > > > > > > >> >> >>> > > > > > >
>> > > > > > > > > > >> >> >>> > > > > > > "
>> > > > > > > > > > >> >> >>> > > > > > > The extreme value tests Micah
>> wanted
>> > > are
>> > > > > done
>> > > > > > > and
>> > > > > > > > > > >> pushed.
>> > > > > > > > > > >> >> I
>> > > > > > > > > > >> >> >>> added
>> > > > > > > > > > >> >> >>> > > > > > coverage
>> > > > > > > > > > >> >> >>> > > > > > > for values that need the full FOR
>> bit
>> > > > width
>> > > > > > > after
>> > > > > > > > > the
>> > > > > > > > > > >> >> frame
>> > > > > > > > > > >> >> >>> of
>> > > > > > > > > > >> >> >>> > > > > reference
>> > > > > > > > > > >> >> >>> > > > > > is
>> > > > > > > > > > >> >> >>> > > > > > > applied: 64-bit (the case where
>> the
>> > > > signed
>> > > > > > max
>> > > > > > > > > minus
>> > > > > > > > > > >> min
>> > > > > > > > > > >> >> >>> > > subtraction
>> > > > > > > > > > >> >> >>> > > > > > > overflows), 63-bit (the
>> non-overflow
>> > > > case),
>> > > > > > and
>> > > > > > > > > > 32-bit
>> > > > > > > > > > >> for
>> > > > > > > > > > >> >> >>> > floats.
>> > > > > > > > > > >> >> >>> > > > All
>> > > > > > > > > > >> >> >>> > > > > > > lossless with zero exceptions. I
>> also
>> > > > > > verified
>> > > > > > > > them
>> > > > > > > > > > >> >> through
>> > > > > > > > > > >> >> >>> Arrow
>> > > > > > > > > > >> >> >>> > > C++
>> > > > > > > > > > >> >> >>> > > > > ALP
>> > > > > > > > > > >> >> >>> > > > > > > reader (the local build), so the
>> > > > > Java-written
>> > > > > > > > > extreme
>> > > > > > > > > > >> >> columns
>> > > > > > > > > > >> >> >>> > > decode
>> > > > > > > > > > >> >> >>> > > > > > > correctly in C++, bit-exact
>> against
>> > the
>> > > > > > > expected
>> > > > > > > > > > >> values."
>> > > > > > > > > > >> >> >>> > > > > > >
>> > > > > > > > > > >> >> >>> > > > > > > State of the PR (#3397):
>> > > > > > > > > > >> >> >>> > > > > > >
>> > > > > > > > > > >> >> >>> > > > > > > - Merges cleanly with master (I
>> > > pulled
>> > > > > > > master
>> > > > > > > > in
>> > > > > > > > > > and
>> > > > > > > > > > >> >> >>> resolved
>> > > > > > > > > > >> >> >>> > > the
>> > > > > > > > > > >> >> >>> > > > > > > conflicts).
>> > > > > > > > > > >> >> >>> > > > > > > - 41 of 44 review threads
>> > resolved.
>> > > > The
>> > > > > 3
>> > > > > > > > > > remaining
>> > > > > > > > > > >> are
>> > > > > > > > > > >> >> >>> > > > intentional:
>> > > > > > > > > > >> >> >>> > > > > > > one deferred perf nit, one that
>> > > stays
>> > > > > open
>> > > > > > > > until
>> > > > > > > > > > >> >> >>> > parquet-format
>> > > > > > > > > > >> >> >>> > > > > ships
>> > > > > > > > > > >> >> >>> > > > > > ALP,
>> > > > > > > > > > >> >> >>> > > > > > >
>> > > > > > > > > > >> >> >>> > > > > > > "
>> > > > > > > > > > >> >> >>> > > > > > >
>> > > > > > > > > > >> >> >>> > > > > > > With this both the
>> > > > > > > > > > >> >> >>> > > > > > > 1) C++ PR :
>> > > > > > > > > > >> >> >>>
>> > https://github.com/apache/arrow/pull/48345/changes
>> > > > > > > > > > >> >> >>> > > > > > > 2) Java PR :
>> > > > > > > > > > >> >> >>>
>> https://github.com/apache/parquet-java/pull/3397
>> > > > > > > > > > >> >> >>> > > > > > >
>> > > > > > > > > > >> >> >>> > > > > > > Are cross language tested and
>> range
>> > of
>> > > > test
>> > > > > > > > > > scenarios.
>> > > > > > > > > > >> >> >>> > > > > > >
>> > > > > > > > > > >> >> >>> > > > > > > Best
>> > > > > > > > > > >> >> >>> > > > > > > Prateek
>> > > > > > > > > > >> >> >>> > > > > > >
>> > > > > > > > > > >> >> >>> > > > > > >
>> > > > > > > > > > >> >> >>> > > > > > > On Mon, Jul 13, 2026 at 8:38 AM
>> > PRATEEK
>> > > > > GAUR
>> > > > > > <
>> > > > > > > > > > >> >> >>> prtkgaur@gmail.com
>> > > > > > > > > > >> >> >>> > >
>> > > > > > > > > > >> >> >>> > > > > wrote:
>> > > > > > > > > > >> >> >>> > > > > > >
>> > > > > > > > > > >> >> >>> > > > > > >> Thanks Matt,
>> > > > > > > > > > >> >> >>> > > > > > >>
>> > > > > > > > > > >> >> >>> > > > > > >> I'll be on it this week.
>> > > > > > > > > > >> >> >>> > > > > > >>
>> > > > > > > > > > >> >> >>> > > > > > >> Best
>> > > > > > > > > > >> >> >>> > > > > > >> Prateek
>> > > > > > > > > > >> >> >>> > > > > > >>
>> > > > > > > > > > >> >> >>> > > > > > >> On Mon, Jul 13, 2026 at 8:36 AM
>> Matt
>> > > > > Topol <
>> > > > > > > > > > >> >> >>> > > zotthewizard@gmail.com>
>> > > > > > > > > > >> >> >>> > > > > > >> wrote:
>> > > > > > > > > > >> >> >>> > > > > > >>
>> > > > > > > > > > >> >> >>> > > > > > >>> There is also the Go
>> implementation
>> > > > > > submitted
>> > > > > > > > by
>> > > > > > > > > > >> Arnav (
>> > > > > > > > > > >> >> >>> > > > > > >>>
>> > > > > > https://github.com/apache/arrow-go/pull/704)
>> > > > > > > > > which
>> > > > > > > > > > >> is
>> > > > > > > > > > >> >> >>> waiting
>> > > > > > > > > > >> >> >>> > > for
>> > > > > > > > > > >> >> >>> > > > > > >>> updates.
>> > > > > > > > > > >> >> >>> > > > > > >>>
>> > > > > > > > > > >> >> >>> > > > > > >>> Just wanted to make sure it
>> didn't
>> > > get
>> > > > > lost
>> > > > > > > > here.
>> > > > > > > > > > >> >> >>> > > > > > >>>
>> > > > > > > > > > >> >> >>> > > > > > >>> --Matt
>> > > > > > > > > > >> >> >>> > > > > > >>>
>> > > > > > > > > > >> >> >>> > > > > > >>> On Mon, Jul 13, 2026 at 11:30 AM
>> > > > PRATEEK
>> > > > > > > GAUR <
>> > > > > > > > > > >> >> >>> > > prtkgaur@gmail.com>
>> > > > > > > > > > >> >> >>> > > > > > >>> wrote:
>> > > > > > > > > > >> >> >>> > > > > > >>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > Hi all,
>> > > > > > > > > > >> >> >>> > > > > > >>> >
>> > > > > > > > > > >> >> >>> > > > > > >>> > I'd like to share a status
>> update
>> > > on
>> > > > > the
>> > > > > > > ALP
>> > > > > > > > > > >> encoding
>> > > > > > > > > > >> >> >>> effort
>> > > > > > > > > > >> >> >>> > > and
>> > > > > > > > > > >> >> >>> > > > > get
>> > > > > > > > > > >> >> >>> > > > > > a
>> > > > > > > > > > >> >> >>> > > > > > >>> feel
>> > > > > > > > > > >> >> >>> > > > > > >>> > from the community before
>> > starting
>> > > a
>> > > > > > formal
>> > > > > > > > > vote.
>> > > > > > > > > > >> >> >>> > > > > > >>> >
>> > > > > > > > > > >> >> >>> > > > > > >>> > We now have the specification
>> > plus
>> > > > > > > > > > implementations
>> > > > > > > > > > >> in
>> > > > > > > > > > >> >> >>> both
>> > > > > > > > > > >> >> >>> > > > > languages:
>> > > > > > > > > > >> >> >>> > > > > > >>> >
>> > > > > > > > > > >> >> >>> > > > > > >>> > - Spec:
>> > > > > > > > apache/parquet-format#557 —
>> > > > > > > > > > >> GH-533
>> > > > > > > > > > >> >> >>> Add
>> > > > > > > > > > >> >> >>> > ALP
>> > > > > > > > > > >> >> >>> > > > > > encoding
>> > > > > > > > > > >> >> >>> > > > > > >>> > specification
>> > > > > > > > > > >> >> >>> > > > > > >>> > - C++ (Arrow):
>> > > > apache/arrow#48345
>> > > > > —
>> > > > > > > > > GH-48701
>> > > > > > > > > > >> >> >>> > > [C++][Parquet]
>> > > > > > > > > > >> >> >>> > > > > Add
>> > > > > > > > > > >> >> >>> > > > > > >>> ALPpd
>> > > > > > > > > > >> >> >>> > > > > > >>> > encoding
>> > > > > > > > > > >> >> >>> > > > > > >>> > - Java:
>> > > > > > > apache/parquet-java#3397
>> > > > > > > > —
>> > > > > > > > > > >> Parquet
>> > > > > > > > > > >> >> >>> Java
>> > > > > > > > > > >> >> >>> > ALP
>> > > > > > > > > > >> >> >>> > > > > > >>> > Implementation (by Vinoo)
>> > > > > > > > > > >> >> >>> > > > > > >>> >
>> > > > > > > > > > >> >> >>> > > > > > >>> > Current state:
>> > > > > > > > > > >> >> >>> > > > > > >>> >
>> > > > > > > > > > >> >> >>> > > > > > >>> > - The C++ implementation
>> has
>> > > been
>> > > > > > > through
>> > > > > > > > > 3-4
>> > > > > > > > > > >> >> rounds
>> > > > > > > > > > >> >> >>> of
>> > > > > > > > > > >> >> >>> > > > review.
>> > > > > > > > > > >> >> >>> > > > > > >>> > - The Java implementation
>> has
>> > > > been
>> > > > > > > > tracking
>> > > > > > > > > > >> well
>> > > > > > > > > > >> >> and
>> > > > > > > > > > >> >> >>> has
>> > > > > > > > > > >> >> >>> > > had
>> > > > > > > > > > >> >> >>> > > > > > >>> initial
>> > > > > > > > > > >> >> >>> > > > > > >>> > reviews from
>> contributors,
>> > > with
>> > > > > no
>> > > > > > > > major
>> > > > > > > > > > >> >> remaining
>> > > > > > > > > > >> >> >>> > > issues.
>> > > > > > > > > > >> >> >>> > > > > > >>> > - Cross-language
>> > compatibility
>> > > > > tests
>> > > > > > > are
>> > > > > > > > > > >> passing:
>> > > > > > > > > > >> >> the
>> > > > > > > > > > >> >> >>> > Arrow
>> > > > > > > > > > >> >> >>> > > > C++
>> > > > > > > > > > >> >> >>> > > > > > >>> decoder
>> > > > > > > > > > >> >> >>> > > > > > >>> > reads Java-written data
>> > > > > bit-exactly
>> > > > > > > > > across
>> > > > > > > > > > >> >> ~1.56M
>> > > > > > > > > > >> >> >>> > values
>> > > > > > > > > > >> >> >>> > > > and
>> > > > > > > > > > >> >> >>> > > > > 18
>> > > > > > > > > > >> >> >>> > > > > > >>> > fixtures, covering V1
>> and
>> > V2
>> > > > > pages,
>> > > > > > > > > > multiple
>> > > > > > > > > > >> >> vector
>> > > > > > > > > > >> >> >>> > > sizes,
>> > > > > > > > > > >> >> >>> > > > > and
>> > > > > > > > > > >> >> >>> > > > > > >>> > several
>> > > > > > > > > > >> >> >>> > > > > > >>> > real datasets — zero
>> > > > mismatches.
>> > > > > > > > > > >> >> >>> > > > > > >>> >
>> > > > > > > > > > >> >> >>> > > > > > >>> > As raised in the java-pr we
>> will
>> > > be
>> > > > > > adding
>> > > > > > > > > > >> coverage
>> > > > > > > > > > >> >> for
>> > > > > > > > > > >> >> >>> > > extreme
>> > > > > > > > > > >> >> >>> > > > > > values
>> > > > > > > > > > >> >> >>> > > > > > >>> > (those requiring
>> > > > > > > > > > >> >> >>> > > > > > >>> > 63-64 bits after FOR is
>> applied)
>> > > > > before
>> > > > > > we
>> > > > > > > > > > close a
>> > > > > > > > > > >> >> vote.
>> > > > > > > > > > >> >> >>> > We'll
>> > > > > > > > > > >> >> >>> > > > aim
>> > > > > > > > > > >> >> >>> > > > > > to
>> > > > > > > > > > >> >> >>> > > > > > >>> get
>> > > > > > > > > > >> >> >>> > > > > > >>> > that
>> > > > > > > > > > >> >> >>> > > > > > >>> > done in parallel.
>> > > > > > > > > > >> >> >>> > > > > > >>> >
>> > > > > > > > > > >> >> >>> > > > > > >>> > Plan: unless there are
>> > objections,
>> > > I
>> > > > > > intend
>> > > > > > > > to
>> > > > > > > > > > >> start a
>> > > > > > > > > > >> >> >>> vote
>> > > > > > > > > > >> >> >>> > at
>> > > > > > > > > > >> >> >>> > > > the
>> > > > > > > > > > >> >> >>> > > > > > end
>> > > > > > > > > > >> >> >>> > > > > > >>> of
>> > > > > > > > > > >> >> >>> > > > > > >>> > this week/early
>> > > > > > > > > > >> >> >>> > > > > > >>> > next week.
>> > > > > > > > > > >> >> >>> > > > > > >>> >
>> > > > > > > > > > >> >> >>> > > > > > >>> > Thanks
>> > > > > > > > > > >> >> >>> > > > > > >>> > Prateek
>> > > > > > > > > > >> >> >>> > > > > > >>> >
>> > > > > > > > > > >> >> >>> > > > > > >>> > On Wed, Jul 1, 2026 at 8:50 AM
>> > > > PRATEEK
>> > > > > > > GAUR <
>> > > > > > > > > > >> >> >>> > > prtkgaur@gmail.com>
>> > > > > > > > > > >> >> >>> > > > > > >>> wrote:
>> > > > > > > > > > >> >> >>> > > > > > >>> >
>> > > > > > > > > > >> >> >>> > > > > > >>> > > Hi Team,
>> > > > > > > > > > >> >> >>> > > > > > >>> > >
>> > > > > > > > > > >> >> >>> > > > > > >>> > > Just wanted to provide some
>> > > updates
>> > > > > on
>> > > > > > > ALP.
>> > > > > > > > > > >> >> >>> > > > > > >>> > > Micah and I have done 3-4
>> > rounds
>> > > of
>> > > > > > > review
>> > > > > > > > > for
>> > > > > > > > > > >> the
>> > > > > > > > > > >> >> c++
>> > > > > > > > > > >> >> >>> PR :
>> > > > > > > > > > >> >> >>> > > > > > >>> > >
>> > > > > > > https://github.com/apache/arrow/pull/48345
>> > > > > > > > > > >> >> >>> > > > > > >>> > >
>> > > > > > > > > > >> >> >>> > > > > > >>> > > For the Java implementation
>> > Vinoo
>> > > > has
>> > > > > > > been
>> > > > > > > > > > >> working
>> > > > > > > > > > >> >> on
>> > > > > > > > > > >> >> >>> > > following
>> > > > > > > > > > >> >> >>> > > > > PR
>> > > > > > > > > > >> >> >>> > > > > > :
>> > > > > > > > > > >> >> >>> > > > > > >>> > >
>> > > > > > > > > > https://github.com/apache/parquet-java/pull/3397
>> > > > > > > > > > >> >> >>> > > > > > >>> > >
>> > > > > > > > > > >> >> >>> > > > > > >>> > > Best
>> > > > > > > > > > >> >> >>> > > > > > >>> > > Prateek
>> > > > > > > > > > >> >> >>> > > > > > >>> > >
>> > > > > > > > > > >> >> >>> > > > > > >>> > > On Tue, May 5, 2026 at
>> 1:50 PM
>> > > > Micah
>> > > > > > > > > Kornfield
>> > > > > > > > > > <
>> > > > > > > > > > >> >> >>> > > > > > >>> emkornfield@gmail.com>
>> > > > > > > > > > >> >> >>> > > > > > >>> > > wrote:
>> > > > > > > > > > >> >> >>> > > > > > >>> > >
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> Hi Antoine,
>> > > > > > > > > > >> >> >>> > > > > > >>> > >>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > Apologies if the question
>> > was
>> > > > > > already
>> > > > > > > > > asked,
>> > > > > > > > > > >> but
>> > > > > > > > > > >> >> >>> should
>> > > > > > > > > > >> >> >>> > we
>> > > > > > > > > > >> >> >>> > > > > care
>> > > > > > > > > > >> >> >>> > > > > > >>> about
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can
>> FLOAT +
>> > > ALP
>> > > > > be
>> > > > > > > more
>> > > > > > > > > > >> >> efficient
>> > > > > > > > > > >> >> >>> than
>> > > > > > > > > > >> >> >>> > > > > FLOAT16
>> > > > > > > > > > >> >> >>> > > > > > +
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4
>> for
>> > > > > example?
>> > > > > > > > > > >> >> >>> > > > > > >>> > >>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> It was. We thought we
>> could
>> > > defer
>> > > > > it
>> > > > > > > for
>> > > > > > > > > the
>> > > > > > > > > > >> >> >>> following
>> > > > > > > > > > >> >> >>> > > > reasons:
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> 1. It's not clear there
>> are a
>> > > lot
>> > > > > of
>> > > > > > > easy
>> > > > > > > > > > >> >> reference
>> > > > > > > > > > >> >> >>> > > datasets
>> > > > > > > > > > >> >> >>> > > > to
>> > > > > > > > > > >> >> >>> > > > > > >>> test
>> > > > > > > > > > >> >> >>> > > > > > >>> > its
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> effectiveness. It does
>> look
>> > > like
>> > > > > > there
>> > > > > > > > > might
>> > > > > > > > > > be
>> > > > > > > > > > >> >> one
>> > > > > > > > > > >> >> >>> or
>> > > > > > > > > > >> >> >>> > two
>> > > > > > > > > > >> >> >>> > > on
>> > > > > > > > > > >> >> >>> > > > > > >>> > huggingface
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> (e.g.
>> > > > > > > > > > >> >> https://huggingface.co/datasets/kikitora/curdie
>> > > > > > > > > > >> >> >>> ).
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> 2. It seemed likely that
>> > float
>> > > 16
>> > > > > was
>> > > > > > > > more
>> > > > > > > > > > >> likely
>> > > > > > > > > > >> >> >>> used
>> > > > > > > > > > >> >> >>> > for
>> > > > > > > > > > >> >> >>> > > > > values
>> > > > > > > > > > >> >> >>> > > > > > >>> that
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> were less likely to reduce
>> to
>> > > > > decimal
>> > > > > > > > > values.
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> 3. It could be added as an
>> > > > > extension
>> > > > > > > > later
>> > > > > > > > > if
>> > > > > > > > > > >> >> needed.
>> > > > > > > > > > >> >> >>> > > > > > >>> > >>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> Cheers,
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> Micah
>> > > > > > > > > > >> >> >>> > > > > > >>> > >>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> On Tue, May 5, 2026 at
>> 1:40 PM
>> > > > > Antoine
>> > > > > > > > > Pitrou
>> > > > > > > > > > <
>> > > > > > > > > > >> >> >>> > > > > antoine@python.org
>> > > > > > > > > > >> >> >>> > > > > > >
>> > > > > > > > > > >> >> >>> > > > > > >>> > wrote:
>> > > > > > > > > > >> >> >>> > > > > > >>> > >>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> >
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > Hello,
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> >
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > Apologies if the question
>> > was
>> > > > > > already
>> > > > > > > > > asked,
>> > > > > > > > > > >> but
>> > > > > > > > > > >> >> >>> should
>> > > > > > > > > > >> >> >>> > we
>> > > > > > > > > > >> >> >>> > > > > care
>> > > > > > > > > > >> >> >>> > > > > > >>> about
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can
>> FLOAT +
>> > > ALP
>> > > > > be
>> > > > > > > more
>> > > > > > > > > > >> >> efficient
>> > > > > > > > > > >> >> >>> than
>> > > > > > > > > > >> >> >>> > > > > FLOAT16
>> > > > > > > > > > >> >> >>> > > > > > +
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4
>> for
>> > > > > example?
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> >
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > Regards
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> >
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > Antoine.
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> >
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> >
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > Le 30/04/2026 à 01:10,
>> > PRATEEK
>> > > > > GAUR
>> > > > > > a
>> > > > > > > > > écrit
>> > > > > > > > > > :
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > > Thanks Andrew and Micah
>> > for
>> > > > > review
>> > > > > > > > > > feedback
>> > > > > > > > > > >> on
>> > > > > > > > > > >> >> >>> the two
>> > > > > > > > > > >> >> >>> > > > PR's
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > > 1) (c++ arrow repo)
>> > > > > > > > > > >> >> >>> > > > > > >>> > >>
>> > > > > > > > > > >> https://github.com/apache/arrow/pull/48345/changes
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > > 2) (parquet-format
>> repo)
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> >
>> > > > > > > > > > >> >> https://github.com/apache/parquet-format/pull/557
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > > I have addressed all
>> > > (unless I
>> > > > > > > missed
>> > > > > > > > > > >> >> something)
>> > > > > > > > > > >> >> >>> > > comments
>> > > > > > > > > > >> >> >>> > > > on
>> > > > > > > > > > >> >> >>> > > > > > >>> the two
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > PR's.
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > > Best
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > > Prateek
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > > On Sat, Apr 25, 2026 at
>> > > > 1:08 PM
>> > > > > > > > PRATEEK
>> > > > > > > > > > >> GAUR <
>> > > > > > > > > > >> >> >>> > > > > > >>> prtkgaur@gmail.com>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> wrote:
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >> Thanks Andrew and
>> Micah.
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >> `fair amount of
>> feedback
>> > on
>> > > > at
>> > > > > > > least
>> > > > > > > > > the
>> > > > > > > > > > >> >> >>> > > implementations`
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >> For the c++ I have
>> > already
>> > > > > > started
>> > > > > > > > > > >> addressing
>> > > > > > > > > > >> >> the
>> > > > > > > > > > >> >> >>> > > > > feedback, I
>> > > > > > > > > > >> >> >>> > > > > > >>> > should
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> be
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >> done with that
>> > > > Monday/Tuesday.
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >> I think Vinoo too has
>> > been
>> > > > > making
>> > > > > > > > good
>> > > > > > > > > > >> >> progress
>> > > > > > > > > > >> >> >>> on
>> > > > > > > > > > >> >> >>> > the
>> > > > > > > > > > >> >> >>> > > > Java
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > implementation.
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >> Best
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >> Prateek
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >> On Sat, Apr 25, 2026
>> at
>> > > > > 12:55 PM
>> > > > > > > > Andrew
>> > > > > > > > > > >> Lamb <
>> > > > > > > > > > >> >> >>> > > > > > >>> > andrewlamb11@gmail.com
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> >
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >> wrote:
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>> Got it. Thank you for
>> > the
>> > > > > > > > > clarification
>> > > > > > > > > > >> -- I
>> > > > > > > > > > >> >> >>> will
>> > > > > > > > > > >> >> >>> > try
>> > > > > > > > > > >> >> >>> > > > and
>> > > > > > > > > > >> >> >>> > > > > > look
>> > > > > > > > > > >> >> >>> > > > > > >>> > into
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> the
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>> spec and the Rust
>> > > > > > > implementation[1]
>> > > > > > > > in
>> > > > > > > > > > >> this
>> > > > > > > > > > >> >> next
>> > > > > > > > > > >> >> >>> > week
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>> [1]:
>> > > > > > > > > > >> >> >>> https://github.com/apache/arrow-rs/pull/9372
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>> On Sat, Apr 25, 2026
>> at
>> > > > > 12:01 PM
>> > > > > > > > Micah
>> > > > > > > > > > >> >> >>> Kornfield <
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > emkornfield@gmail.com>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>> wrote:
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>> Hi Andrew,
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>> I think there is a
>> fair
>> > > > > amount
>> > > > > > of
>> > > > > > > > > > >> feedback
>> > > > > > > > > > >> >> on
>> > > > > > > > > > >> >> >>> at
>> > > > > > > > > > >> >> >>> > > least
>> > > > > > > > > > >> >> >>> > > > > the
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>> implementations,
>> > > typically
>> > > > I
>> > > > > > > think
>> > > > > > > > > > we've
>> > > > > > > > > > >> >> waited
>> > > > > > > > > > >> >> >>> > till
>> > > > > > > > > > >> >> >>> > > > they
>> > > > > > > > > > >> >> >>> > > > > > are
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> close to
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>> mergeable before a
>> > final
>> > > > > vote.
>> > > > > > > > > > >> Otherwise I
>> > > > > > > > > > >> >> >>> agree
>> > > > > > > > > > >> >> >>> > we
>> > > > > > > > > > >> >> >>> > > > are
>> > > > > > > > > > >> >> >>> > > > > > very
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> close.
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>> -Micah
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>> On Saturday, April
>> 25,
>> > > > 2026,
>> > > > > > > Andrew
>> > > > > > > > > > Lamb
>> > > > > > > > > > >> <
>> > > > > > > > > > >> >> >>> > > > > > >>> andrewlamb11@gmail.com
>> > > > > > > > > > >> >> >>> > > > > > >>> > >
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > wrote:
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> Thanks Prateek,
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> I think from this
>> > > content
>> > > > it
>> > > > > > > looks
>> > > > > > > > > to
>> > > > > > > > > > me
>> > > > > > > > > > >> >> like
>> > > > > > > > > > >> >> >>> we
>> > > > > > > > > > >> >> >>> > are
>> > > > > > > > > > >> >> >>> > > > > ready
>> > > > > > > > > > >> >> >>> > > > > > >>> to
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> start a
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> vote to explicitly
>> > > accept
>> > > > > ALP
>> > > > > > > into
>> > > > > > > > > > >> Parquet
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> Does anyone know
>> of a
>> > > > reason
>> > > > > > we
>> > > > > > > > > should
>> > > > > > > > > > >> >> >>> postpone it
>> > > > > > > > > > >> >> >>> > > for
>> > > > > > > > > > >> >> >>> > > > > > >>> longer?
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> Perhaps someone
>> needs
>> > > some
>> > > > > > more
>> > > > > > > > time
>> > > > > > > > > > to
>> > > > > > > > > > >> >> >>> review?
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> Andrew
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> On Wed, Apr 22,
>> 2026
>> > at
>> > > > > > 1:00 PM
>> > > > > > > > > > PRATEEK
>> > > > > > > > > > >> >> GAUR <
>> > > > > > > > > > >> >> >>> > > > > > >>> > prtkgaur@gmail.com>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> wrote:
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Hi team,
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Hope everyone is
>> > doing
>> > > > > well.
>> > > > > > I
>> > > > > > > > got
>> > > > > > > > > a
>> > > > > > > > > > >> >> chance
>> > > > > > > > > > >> >> >>> to
>> > > > > > > > > > >> >> >>> > work
>> > > > > > > > > > >> >> >>> > > > > > >>> through all
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> the
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> remaining feedback
>> > and
>> > > > > update
>> > > > > > > the
>> > > > > > > > > > spec
>> > > > > > > > > > >> >> doc.
>> > > > > > > > > > >> >> >>> Here
>> > > > > > > > > > >> >> >>> > > are
>> > > > > > > > > > >> >> >>> > > > > the
>> > > > > > > > > > >> >> >>> > > > > > >>> new
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > artifacts
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 1) Spec document :
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> >
>> > > > > > > > > > >> >> >>> > > > > > >>> > >>
>> > > > > > > > > > >> >> >>> > > > > > >>> >
>> > > > > > > > > > >> >> >>> > > > > > >>>
>> > > > > > > > > > >> >> >>> > > > > >
>> > > > > > > > > > >> >> >>> > > > >
>> > > > > > > > > > >> >> >>> > > >
>> > > > > > > > > > >> >> >>> > >
>> > > > > > > > > > >> >> >>> >
>> > > > > > > > > > >> >> >>>
>> > > > > > > > > > >> >>
>> > > > > > > > > > >>
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > > https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-
>> > > > 3fPtD_aWt/edit
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 2) Spec document
>> in
>> > > > parquet
>> > > > > > > > format
>> > > > > > > > > > >> repo :
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> >
>> > > https://github.com/apache/parquet-format/pull/557
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 3) Alp
>> implementation
>> > > in
>> > > > > > arrow
>> > > > > > > > c++
>> > > > > > > > > > >> repo :
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> >
>> > > https://github.com/apache/arrow/pull/48345/changes
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 4) Alp
>> implementation
>> > > in
>> > > > > > > > > parquet-java
>> > > > > > > > > > >> >> repo :
>> > > > > > > > > > >> >> >>> Work
>> > > > > > > > > > >> >> >>> > > for
>> > > > > > > > > > >> >> >>> > > > > > >>> Vinoo and
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > Julien
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> >
>> > https://github.com/apache/parquet-java/pull/3397
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 5) PR with test
>> and
>> > > > > > > benchmarking
>> > > > > > > > > > >> >> artifacts in
>> > > > > > > > > > >> >> >>> > > > > > >>> parquet-testing
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> repo :
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> >
>> > > https://github.com/apache/parquet-testing/pull/100
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> And
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> - Go : Arnav
>> just
>> > > > > > submitted
>> > > > > > > > an
>> > > > > > > > > in
>> > > > > > > > > > >> >> >>> progress
>> > > > > > > > > > >> >> >>> > > > > > >>> implementation
>> > > > > > > > > > >> >> >>> > > > > > >>> > in
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> Go.
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> https://github.com/apache/arrow-go/pull/704
>> > > > > > > > > > >> >> >>> > (I
>> > > > > > > > > > >> >> >>> > > > > > haven't
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> started
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> looking at it
>> > yet)
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> - Rust : I
>> > remember
>> > > > > > Andrew
>> > > > > > > > > > >> mentioned
>> > > > > > > > > > >> >> that
>> > > > > > > > > > >> >> >>> > this
>> > > > > > > > > > >> >> >>> > > > work
>> > > > > > > > > > >> >> >>> > > > > > is
>> > > > > > > > > > >> >> >>> > > > > > >>> also
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> in
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> progress (So 4
>> > > > > > languages!)
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> *Arrow C++
>> > > > implementation *
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> The PR is out and
>> was
>> > > > also
>> > > > > > used
>> > > > > > > > by
>> > > > > > > > > > >> >> Antoine to
>> > > > > > > > > > >> >> >>> > > report
>> > > > > > > > > > >> >> >>> > > > > the
>> > > > > > > > > > >> >> >>> > > > > > >>> > numbers
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> as
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> reported here.
>> Micah
>> > > and
>> > > > > > > > Konstantin
>> > > > > > > > > > >> have
>> > > > > > > > > > >> >> >>> given 1
>> > > > > > > > > > >> >> >>> > > > round
>> > > > > > > > > > >> >> >>> > > > > of
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> feedback
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> and I'm addressing
>> > them
>> > > > > > today.
>> > > > > > > > > Please
>> > > > > > > > > > >> note
>> > > > > > > > > > >> >> >>> that
>> > > > > > > > > > >> >> >>> > the
>> > > > > > > > > > >> >> >>> > > > > > default
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> optimization flag
>> for
>> > > > > > compiling
>> > > > > > > > is
>> > > > > > > > > O2
>> > > > > > > > > > >> and
>> > > > > > > > > > >> >> not
>> > > > > > > > > > >> >> >>> > Q3. I
>> > > > > > > > > > >> >> >>> > > > got
>> > > > > > > > > > >> >> >>> > > > > > >>> around
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> 70%
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> performance
>> > improvement
>> > > > in
>> > > > > > the
>> > > > > > > > > > decoding
>> > > > > > > > > > >> >> speed
>> > > > > > > > > > >> >> >>> > when
>> > > > > > > > > > >> >> >>> > > > > using
>> > > > > > > > > > >> >> >>> > > > > > >>> the O3
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > flag.
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> *Parqet-MR Java
>> > > > > > implementation
>> > > > > > > > > > (working
>> > > > > > > > > > >> >> with
>> > > > > > > > > > >> >> >>> > Vinoo
>> > > > > > > > > > >> >> >>> > > > and
>> > > > > > > > > > >> >> >>> > > > > > >>> Julien)
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> and
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > **Cross
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Language testing*
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Let me know if
>> > you
>> > > > have
>> > > > > > any
>> > > > > > > > > > >> questions
>> > > > > > > > > > >> >> or
>> > > > > > > > > > >> >> >>> > > > feedback.
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Now pasting some
>> > > > > performance
>> > > > > > > > > numbers
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Table 1: C++
>> ALP
>> > > > Double
>> > > > > > > > Decode —
>> > > > > > > > > > >> >> Spotify
>> > > > > > > > > > >> >> >>> > Columns
>> > > > > > > > > > >> >> >>> > > > > > >>> (Graviton
>> > > > > > > > > > >> >> >>> > > > > > >>> > 3,
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> ARM
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Neoverse V1)
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>>
>> > > > > > > > > > >> >> >>>
>> > > > > > > ┌──────────────────┬──────────────┬──────────────┬─────────┐
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ Column
>> > │
>> > > > > -O2
>> > > > > > > > (MB/s)
>> > > > > > > > > > │
>> > > > > > > > > > >> >> -O3
>> > > > > > > > > > >> >> >>> > > (MB/s) │
>> > > > > > > > > > >> >> >>> > > > > > >>> Speedup │
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>>
>> > > > > > > > > > >> >> >>>
>> > > > > > > ├──────────────────┼──────────────┼──────────────┼─────────┤
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ valence
>> > │
>> > > > > > > 3,155
>> > > > > > > > > > │
>> > > > > > > > > > >> >> >>> 5,523
>> > > > > > > > > > >> >> >>> > > │
>> > > > > > > > > > >> >> >>> > > > > > >>> 1.75x │
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ danceability
>> > │
>> > > > > > > 3,233
>> > > > > > > > > > │
>> > > > > > > > > > >> >> >>> 5,685
>> > > > > > > > > > >> >> >>> > > │
>> > > > > > > > > > >> >> >>> > > > > > >>> 1.76x │
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ energy
>> > │
>> > > > > > > 3,197
>> > > > > > > > > > │
>> > > > > > > > > > >> >> >>> 5,652
>> > > > > > > > > > >> >> >>> > > │
>> > > > > > > > > > >> >> >>> > > > > > >>> 1.77x │
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ loudness
>> > │
>> > > > > > > 3,186
>> > > > > > > > > > │
>> > > > > > > > > > >> >> >>> 5,473
>> > > > > > > > > > >> >> >>> > > │
>> > > > > > > > > > >> >> >>> > > > > > >>> 1.72x │
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>>
>> > > > > > > > > > >> >> >>>
>> > > > > > > └──────────────────┴──────────────┴──────────────┴─────────┘
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> On Wed, Feb 25,
>> 2026
>> > at
>> > > > > > 9:49 AM
>> > > > > > > > > > PRATEEK
>> > > > > > > > > > >> >> GAUR
>> > > > > > > > > > >> >> >>> <
>> > > > > > > > > > >> >> >>> > > > > > >>> > prtkgaur@gmail.com
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> >
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> wrote:
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> @Micah Kornfield
>> <
>> > > > > > > > > > >> emkornfield@gmail.com>
>> > > > > > > > > > >> >> :
>> > > > > > > > > > >> >> >>> Got
>> > > > > > > > > > >> >> >>> > > it.
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> @Andrew Lamb <
>> > > > > > > > > > andrewlamb11@gmail.com>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> Do you think it
>> > would
>> > > > be
>> > > > > > good
>> > > > > > > > to
>> > > > > > > > > > >> start
>> > > > > > > > > > >> >> >>> moving
>> > > > > > > > > > >> >> >>> > the
>> > > > > > > > > > >> >> >>> > > > > spec
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> development
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> into
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> markdown
>> format, in
>> > > > > > > preparation
>> > > > > > > > > for
>> > > > > > > > > > >> >> >>> finalizing
>> > > > > > > > > > >> >> >>> > > it?
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> Yes I'll update
>> the
>> > > > > numbers
>> > > > > > > for
>> > > > > > > > > some
>> > > > > > > > > > >> of
>> > > > > > > > > > >> >> the
>> > > > > > > > > > >> >> >>> > > > examples I
>> > > > > > > > > > >> >> >>> > > > > > >>> have in
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> the
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> spec based
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> on the updated
>> > header
>> > > > > size.
>> > > > > > > Then
>> > > > > > > > > we
>> > > > > > > > > > >> >> should
>> > > > > > > > > > >> >> >>> be
>> > > > > > > > > > >> >> >>> > good
>> > > > > > > > > > >> >> >>> > > > to
>> > > > > > > > > > >> >> >>> > > > > go
>> > > > > > > > > > >> >> >>> > > > > > >>> for
>> > > > > > > > > > >> >> >>> > > > > > >>> > the
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> markdown format.
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> Thanks everyone!
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> Andrew
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> On Tue, Feb 17,
>> > 2026
>> > > at
>> > > > > > > 7:28 PM
>> > > > > > > > > > >> PRATEEK
>> > > > > > > > > > >> >> >>> GAUR <
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> prtkgaur@gmail.com>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> wrote:
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Hi team,
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 1) Andrew
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks
>> for
>> > > > working
>> > > > > > on
>> > > > > > > > test
>> > > > > > > > > > >> files.
>> > > > > > > > > > >> >> >>> My PR
>> > > > > > > > > > >> >> >>> > > did
>> > > > > > > > > > >> >> >>> > > > > add
>> > > > > > > > > > >> >> >>> > > > > > >>> all
>> > > > > > > > > > >> >> >>> > > > > > >>> > the
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > test
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> files I
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> used to
>> > > benchmark
>> > > > on
>> > > > > > > > > datasets.
>> > > > > > > > > > >> >> Maybe
>> > > > > > > > > > >> >> >>> we
>> > > > > > > > > > >> >> >>> > can
>> > > > > > > > > > >> >> >>> > > > club
>> > > > > > > > > > >> >> >>> > > > > > it
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> together.
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> WIll also
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> aid
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> cross
>> language
>> > > > > testing
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Kosta
>> > Tarasov
>> > > > > > working
>> > > > > > > > on
>> > > > > > > > > > Rust
>> > > > > > > > > > >> >> >>> > > > implementation.
>> > > > > > > > > > >> >> >>> > > > > > >>> This is
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > great.
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> Thanks
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 2) Antoine
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks a
>> lot
>> > > for
>> > > > > > > > reporting
>> > > > > > > > > > the
>> > > > > > > > > > >> >> >>> numbers
>> > > > > > > > > > >> >> >>> > on
>> > > > > > > > > > >> >> >>> > > > AMD.
>> > > > > > > > > > >> >> >>> > > > > > >>> Looks
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> like
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > you
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> are
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> getting 8X
>> the
>> > > > > > decoding
>> > > > > > > > > > >> >> performance of
>> > > > > > > > > > >> >> >>> > BSS.
>> > > > > > > > > > >> >> >>> > > > This
>> > > > > > > > > > >> >> >>> > > > > > is
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > amazing!!.
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks
>> for
>> > > > > > > acknowledging
>> > > > > > > > > the
>> > > > > > > > > > >> >> >>> sampling
>> > > > > > > > > > >> >> >>> > > > design.
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - I agree
>> with
>> > > you
>> > > > > on
>> > > > > > > > > > >> Fastlanes. In
>> > > > > > > > > > >> >> >>> some
>> > > > > > > > > > >> >> >>> > > crude
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> experiments I
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> didn't get
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> a good perf
>> > > > benefit
>> > > > > > from
>> > > > > > > > it
>> > > > > > > > > on
>> > > > > > > > > > >> >> >>> Graviton3
>> > > > > > > > > > >> >> >>> > > (but
>> > > > > > > > > > >> >> >>> > > > > > maybe
>> > > > > > > > > > >> >> >>> > > > > > >>> > there
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> was
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> something
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> wrong with
>> my
>> > > > > > > > > implementation).
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Locking
>> the
>> > > > 16bit
>> > > > > > > > > exception
>> > > > > > > > > > >> >> >>> encoding for
>> > > > > > > > > > >> >> >>> > > the
>> > > > > > > > > > >> >> >>> > > > > > spec
>> > > > > > > > > > >> >> >>> > > > > > >>> in
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> this
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> case.
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Awesome I
>> > > think
>> > > > we
>> > > > > > > have
>> > > > > > > > > > solved
>> > > > > > > > > > >> >> for
>> > > > > > > > > > >> >> >>> all
>> > > > > > > > > > >> >> >>> > > open
>> > > > > > > > > > >> >> >>> > > > > > >>> questions
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> minus
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> the
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> version
>> byte
>> > :).
>> > > > > (will
>> > > > > > > get
>> > > > > > > > > > back
>> > > > > > > > > > >> on
>> > > > > > > > > > >> >> >>> this
>> > > > > > > > > > >> >> >>> > > soon)
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 3) Micah
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> -
>> FastLanes :
>> > > The
>> > > > > > > current
>> > > > > > > > > spec
>> > > > > > > > > > >> does
>> > > > > > > > > > >> >> >>> allow
>> > > > > > > > > > >> >> >>> > > for
>> > > > > > > > > > >> >> >>> > > > > > using
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> FastLane
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> with the
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> configurable
>> > > enum
>> > > > > > value
>> > > > > > > > for
>> > > > > > > > > > >> >> layout. We
>> > > > > > > > > > >> >> >>> > > should
>> > > > > > > > > > >> >> >>> > > > be
>> > > > > > > > > > >> >> >>> > > > > > >>> able to
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > inject
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> any
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> layout
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> in the
>> current
>> > > > > design.
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Working on
>> > resolving
>> > > > all
>> > > > > > > > > remaining
>> > > > > > > > > > >> open
>> > > > > > > > > > >> >> >>> > comments
>> > > > > > > > > > >> >> >>> > > > on
>> > > > > > > > > > >> >> >>> > > > > > the
>> > > > > > > > > > >> >> >>> > > > > > >>> spec
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> this
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> week.
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Best
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Prateek
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> On Tue, Feb 10,
>> > 2026
>> > > > at
>> > > > > > > > 3:37 AM
>> > > > > > > > > > >> Steve
>> > > > > > > > > > >> >> >>> > Loughran <
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
>> > stevel@cloudera.com>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> wrote:
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> On Sun, 8 Feb
>> > 2026
>> > > at
>> > > > > > > 18:12,
>> > > > > > > > > > Micah
>> > > > > > > > > > >> >> >>> Kornfield
>> > > > > > > > > > >> >> >>> > <
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
>> > > emkornfield@gmail.com>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> wrote:
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> It looks like
>> > the
>> > > > > actual
>> > > > > > > > issue
>> > > > > > > > > > >> >> >>> described for
>> > > > > > > > > > >> >> >>> > > ORC
>> > > > > > > > > > >> >> >>> > > > > in
>> > > > > > > > > > >> >> >>> > > > > > >>> the
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> paper
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> is that
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> it
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> has multiple
>> > > > > > sub-encodings
>> > > > > > > > in
>> > > > > > > > > a
>> > > > > > > > > > >> >> batch.
>> > > > > > > > > > >> >> >>> This
>> > > > > > > > > > >> >> >>> > > is
>> > > > > > > > > > >> >> >>> > > > > > >>> different
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> then
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> the
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> design
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> proposed here
>> > > where
>> > > > > > there
>> > > > > > > is
>> > > > > > > > > > still
>> > > > > > > > > > >> >> fixed
>> > > > > > > > > > >> >> >>> > > > encoding
>> > > > > > > > > > >> >> >>> > > > > > per
>> > > > > > > > > > >> >> >>> > > > > > >>> page
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> in
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> parquet.
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> Given
>> reasonably
>> > > > sized
>> > > > > > > > pages I
>> > > > > > > > > > >> don't
>> > > > > > > > > > >> >> >>> think
>> > > > > > > > > > >> >> >>> > > > branch
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> misprediction
>> > should
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> be a
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> big issue for
>> > new
>> > > > > > > encodings.
>> > > > > > > > > I
>> > > > > > > > > > >> agree
>> > > > > > > > > > >> >> >>> that
>> > > > > > > > > > >> >> >>> > we
>> > > > > > > > > > >> >> >>> > > > > should
>> > > > > > > > > > >> >> >>> > > > > > >>> be
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> conservative in
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> general for
>> > adding
>> > > > new
>> > > > > > > > > > encodings.
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> +1
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> >
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> >
>> > > > > > > > > > >> >> >>> > > > > > >>> > >> >
>> > > > > > > > > > >> >> >>> > > > > > >>> > >>
>> > > > > > > > > > >> >> >>> > > > > > >>> > >
>> > > > > > > > > > >> >> >>> > > > > > >>> >
>> > > > > > > > > > >> >> >>> > > > > > >>>
>> > > > > > > > > > >> >> >>> > > > > > >>
>> > > > > > > > > > >> >> >>> > > > > >
>> > > > > > > > > > >> >> >>> > > > >
>> > > > > > > > > > >> >> >>> > > >
>> > > > > > > > > > >> >> >>> > >
>> > > > > > > > > > >> >> >>> >
>> > > > > > > > > > >> >> >>>
>> > > > > > > > > > >> >> >>
>> > > > > > > > > > >> >>
>> > > > > > > > > > >> >
>> > > > > > > > > > >>
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Gang Wu <us...@gmail.com>.
Thanks Gunnar for sharing this!
I think the encoder approach used by Hardwood looks more similar to the ALP
paper where all 102,400 values are buffered to decide best presets and then
encode every vector based on those presets.
I'm trying to improve the ALP implementations on the Parquet C++ side:
https://github.com/apache/arrow/pull/55449. So I'm really interested
if you have any document on performance comparison among different
implementations.
Thanks!
Gang
On Thu, Nov 12, 2026 at 6:50 PM Gunnar Morling via dev <
dev@parquet.apache.org> wrote:
> Hey all,
>
> We've added ALP encoding and decoding support to Hardwood too [1], [2].
> Interop with parquet-java's implementation looks good: Hardwood decodes
> pages written by parquet-java's ALP writer bit-exactly and vice versa.
>
> Perhaps interesting: on the encoder side, we build the preset of a column
> chunk from eight sample vectors distributed across the entire chunk (easy
> in our case, as Hardwood's writer buffers whole chunks), whereas, IIUC,
> parquet-java only samples within the first ~110K values of a chunk.
> Depending on the data distribution, this can yield a more efficient
> encoding, for instance when a high-cardinality column changes character
> midway, e.g. moving from two to four decimal places (about 2.8× smaller in
> a synthetic test with 1M doubles).
>
> Best,
>
> --Gunnar
>
> [1] https://github.com/hardwood-hq/hardwood/pull/1496
> [2] https://github.com/hardwood-hq/hardwood/pull/1769
>
>
> On Thu, 12 Nov 2026 at 07:01, PRATEEK GAUR <pr...@gmail.com> wrote:
>
> > Thanks everyone!
> >
> > On Mon, Nov 9, 2026 at 11:58 AM Vinoo Ganesh <vi...@gmail.com>
> > wrote:
> >
> > > Thanks everyone!
> > >
> > > Thanks,
> > > Vinoo Ganesh | vinoo.ganesh@gmail.com
> > >
> > > <Vi...@gmail.com>
> > >
> > >
> > > On Mon, Nov 9, 2026 at 2:49 PM Andrew Lamb <an...@gmail.com>
> > wrote:
> > >
> > > > The Java implementation was just merged[1]!
> > > >
> > > > Thank you Gang, Vinoo, Prateek and everyone who helped push that one
> > > along
> > > >
> > > > Andrew
> > > >
> > > > [1]:
> > https://github.com/apache/parquet-java/pull/3397#event-35545451335
> > > >
> > > > On Tue, Oct 13, 2026 at 12:52 PM PRATEEK GAUR <pr...@gmail.com>
> > > wrote:
> > > >
> > > > > Hi team,
> > > > >
> > > > > Quick update re: the C++ implementation.
> > > > > Thanks to all the reviewers, we merged the C++ implementation
> today.
> > > > > There are a few follow-up items that I'll address in a subsequent
> PR.
> > > > >
> > > > > Thanks
> > > > > Prateek
> > > > >
> > > > > On Sun, Sep 13, 2026 at 6:54 PM Vinoo Ganesh <
> vinoo.ganesh@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Quick update re: the Java impl. Thanks for the feedback, on
> > > > > > https://github.com/apache/parquet-java/pull/3397, Russel! I
> pulled
> > > > > > https://github.com/apache/parquet-java/pull/3791 into its own PR
> > > > (thanks
> > > > > > for the quick approval too, Gang). Assuming no objections, I
> think
> > we
> > > > can
> > > > > > merge #3791, and then go ahead of #3397.
> > > > > >
> > > > > > Thanks,
> > > > > > Vinoo Ganesh | vinoo.ganesh@gmail.com
> > > > > >
> > > > > > <Vi...@gmail.com>
> > > > > >
> > > > > >
> > > > > > On Tue, Sep 8, 2026 at 10:54 AM PRATEEK GAUR <prtkgaur@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > > > [Rust] Awesome thanks Andrew, Kostas and Devin.
> > > > > > > [Java] Thanks Gang for the feedback on Java PR.
> > > > > > > [C++] Thanks Gang for another round of feedback on the C++ one.
> > > > > > > [Go] Arnav and I will be working on the golang impl this week.
> > > > > > >
> > > > > > > This is great progress all around.
> > > > > > >
> > > > > > > On Sat, Sep 5, 2026 at 1:26 PM Vinoo Ganesh <
> > > vinoo.ganesh@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Thanks all - I'm addressing Gang's comments on
> > > > > > > > https://github.com/apache/parquet-java/pull/3397 now, so
> > > hopefully
> > > > > > well
> > > > > > > > also be in a good state for merging the java implementation
> > soon.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Vinoo Ganesh | vinoo.ganesh@gmail.com
> > > > > > > >
> > > > > > > > <Vi...@gmail.com>
> > > > > > > >
> > > > > > > >
> > > > > > > > On Sat, Sep 5, 2026 at 8:56 AM Andrew Lamb <
> > > andrewlamb11@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > We just merged the Rust implementation[1] from Kosta and
> > Devan.
> > > > > With
> > > > > > > the
> > > > > > > > > imminent release of Parquet Format 2.14.0 I think we'll be
> > well
> > > > on
> > > > > > our
> > > > > > > > way.
> > > > > > > > >
> > > > > > > > > Thanks again for pushing this along,
> > > > > > > > > Andrew
> > > > > > > > >
> > > > > > > > > [1]: https://github.com/apache/arrow-rs/pull/9372
> > > > > > > > >
> > > > > > > > > On Fri, Sep 4, 2026 at 5:42 PM PRATEEK GAUR <
> > > prtkgaur@gmail.com>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi team,
> > > > > > > > > >
> > > > > > > > > > Thanks Micah for multiple rounds of review and thanks
> Gang
> > > for
> > > > > > > taking a
> > > > > > > > > > look at the PR and giving feedback. I've addressed all
> > > comments
> > > > > in
> > > > > > > the
> > > > > > > > > PR.
> > > > > > > > > > Looking forward to the next round of feedback (or
> > approval).
> > > > > Would
> > > > > > > like
> > > > > > > > > to
> > > > > > > > > > get it in soon.
> > > > > > > > > >
> > > > > > > > > > https://github.com/apache/arrow/pull/48345/changes
> > > > > > > > > >
> > > > > > > > > > With this done and after talking to Arnav, I plan to
> spend
> > > some
> > > > > > time
> > > > > > > on
> > > > > > > > > the
> > > > > > > > > > ALP implementation in the arrow-go repository helping
> > spread
> > > > the
> > > > > > > > support
> > > > > > > > > to
> > > > > > > > > > more languages.
> > > > > > > > > >
> > > > > > > > > > Best
> > > > > > > > > > Prateek
> > > > > > > > > >
> > > > > > > > > > On Sun, Aug 2, 2026 at 10:42 PM PRATEEK GAUR <
> > > > prtkgaur@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Thanks Divjot for bringing it up.
> > > > > > > > > > >
> > > > > > > > > > > I was under the same impression as Russell and Andrew
> > that
> > > > we'd
> > > > > > > have
> > > > > > > > it
> > > > > > > > > > > flag gated.
> > > > > > > > > > > With the new versioning proposal 'preview' might be a
> > good
> > > > tag
> > > > > > for
> > > > > > > > it.
> > > > > > > > > > >
> > > > > > > > > > > Best
> > > > > > > > > > > Prateek
> > > > > > > > > > >
> > > > > > > > > > > On Sat, Aug 1, 2026 at 12:39 AM Andrew Lamb <
> > > > > > > andrewlamb11@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >> Div: Here is a proposal to mark ALP in Preview: [1]
> > > (using a
> > > > > > > comment
> > > > > > > > > in
> > > > > > > > > > >> the
> > > > > > > > > > >> spec section).
> > > > > > > > > > >>
> > > > > > > > > > >> Russel: That is my understanding as well.
> > > > > > > > > > >>
> > > > > > > > > > >> Hopefully we are able to formalize the versioning
> > process
> > > > > > shortly
> > > > > > > so
> > > > > > > > > we
> > > > > > > > > > >> can
> > > > > > > > > > >> formalize the notion of "preview"
> > > > > > > > > > >>
> > > > > > > > > > >> Andrew
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >> [1]:
> https://github.com/apache/parquet-format/pull/604
> > > > > > > > > > >>
> > > > > > > > > > >> On Fri, Jul 31, 2026 at 11:15 AM Russell Spitzer <
> > > > > > > > > > >> russell.spitzer@gmail.com>
> > > > > > > > > > >> wrote:
> > > > > > > > > > >>
> > > > > > > > > > >> > I thought we had agreed it would be merged, and
> > features
> > > > > like
> > > > > > > this
> > > > > > > > > > would
> > > > > > > > > > >> > be feature flag gated. So a V-Old writer could
> always
> > > opt
> > > > in
> > > > > > to
> > > > > > > > > using
> > > > > > > > > > >> this
> > > > > > > > > > >> > encoding but that must be a conscious effort if you
> > are
> > > > not
> > > > > > > using
> > > > > > > > > > VNext
> > > > > > > > > > >> of
> > > > > > > > > > >> > Parquet. If you are on V-Next the writer can choose
> to
> > > use
> > > > > ALP
> > > > > > > > > > whenever
> > > > > > > > > > >> it
> > > > > > > > > > >> > likes without user intervention.
> > > > > > > > > > >> >
> > > > > > > > > > >> > On Fri, Jul 31, 2026 at 2:20 AM Divjot Arora via
> dev <
> > > > > > > > > > >> > dev@parquet.apache.org> wrote:
> > > > > > > > > > >> >
> > > > > > > > > > >> >> Yes, that’s my understanding as well. However, as
> you
> > > > > noted,
> > > > > > > the
> > > > > > > > > > >> >> versioning
> > > > > > > > > > >> >> discussion is still in early stages and hasn’t
> > > prescribed
> > > > > > > > concrete
> > > > > > > > > > >> >> mechanisms yet. If ALP goes into parquet-format
> > 2.14.0,
> > > > > > there’s
> > > > > > > > no
> > > > > > > > > > >> >> indication that it’s in preview or that writers
> > > shouldn’t
> > > > > use
> > > > > > > it
> > > > > > > > by
> > > > > > > > > > >> >> default
> > > > > > > > > > >> >> yet. Given that we’ve held off on merging other
> > > > > incompatible
> > > > > > > > > changes
> > > > > > > > > > >> (e.g
> > > > > > > > > > >> >> removal of path_in_schema) until the versioning
> > > > discussion
> > > > > > > > closes,
> > > > > > > > > my
> > > > > > > > > > >> >> preference would be to leave ALP un-merged for the
> > time
> > > > > being
> > > > > > > as
> > > > > > > > > > well.
> > > > > > > > > > >> >>
> > > > > > > > > > >> >> — Div
> > > > > > > > > > >> >>
> > > > > > > > > > >> >> On Fri, Jul 31, 2026 at 08:59 Andrew Lamb <
> > > > > > > > andrewlamb11@gmail.com>
> > > > > > > > > > >> wrote:
> > > > > > > > > > >> >>
> > > > > > > > > > >> >> > After re-reading Julien's Versioning Proposal, my
> > > > > > > understanding
> > > > > > > > > is
> > > > > > > > > > >> that
> > > > > > > > > > >> >> if
> > > > > > > > > > >> >> > we adopt that system, we will somehow mark ALP in
> > > > > "Preview"
> > > > > > > in
> > > > > > > > > > >> >> > parquet-format for release Version 3.
> > > > > > > > > > >> >> >
> > > > > > > > > > >> >> > The versioning document doesn't seem to proscribe
> > the
> > > > > > > > particular
> > > > > > > > > > >> method
> > > > > > > > > > >> >> > for annotating "preview" features -- I think
> > clearly
> > > > > > marking
> > > > > > > > them
> > > > > > > > > > as
> > > > > > > > > > >> >> such
> > > > > > > > > > >> >> > in parquet.thrift as well as reflecting that on
> the
> > > > > > > > > > >> >> >
> > > https://parquet.apache.org/docs/file-format/versions/
> > > > > page
> > > > > > > > > somehow
> > > > > > > > > > >> >> would
> > > > > > > > > > >> >> > be adequate.
> > > > > > > > > > >> >> >
> > > > > > > > > > >> >> > Andrew
> > > > > > > > > > >> >> >
> > > > > > > > > > >> >> > On Thu, Jul 30, 2026 at 11:50 AM Andrew Lamb <
> > > > > > > > > > andrewlamb11@gmail.com
> > > > > > > > > > >> >
> > > > > > > > > > >> >> > wrote:
> > > > > > > > > > >> >> >
> > > > > > > > > > >> >> >> In my mind ALP is in the same position as other
> > > > > encodings
> > > > > > > that
> > > > > > > > > > were
> > > > > > > > > > >> >> added
> > > > > > > > > > >> >> >> recently[1] (such as BYTE_STREAM_SPLIT for non
> > > > floating
> > > > > > > point
> > > > > > > > > > types)
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >> >> That is to say it will be available in the
> latest
> > > open
> > > > > > > source
> > > > > > > > > > >> writers
> > > > > > > > > > >> >> and
> > > > > > > > > > >> >> >> readers, but realistically not widely used in
> the
> > > > > > ecosystem
> > > > > > > > > where
> > > > > > > > > > >> >> >> compatibility is a concern (a large part)
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >> >> I view Julian's work[2][3] to gather consensus
> on
> > > how
> > > > to
> > > > > > > name
> > > > > > > > > > >> >> >> combinations of features (e.g. "versions") as
> the
> > > > > > mechanism
> > > > > > > by
> > > > > > > > > > >> which we
> > > > > > > > > > >> >> >> will accelerate the adoption of this change
> (along
> > > > with
> > > > > > > other
> > > > > > > > > > >> forward
> > > > > > > > > > >> >> >> incompatible changes) across the ecosystem.
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >> >> > Will we exclude this change when cutting
> > > > > parquet-format
> > > > > > > > > v2.14.0?
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >> >> No I don't think so -- I would expect it to go
> out
> > > in
> > > > > the
> > > > > > > next
> > > > > > > > > > >> >> >> parquet-format version as do other changes in
> the
> > > > spec.
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >> >> Andrew
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >> >> [1]:
> > > > > > https://parquet.apache.org/docs/file-format/versions/
> > > > > > > > > > >> >> >> [2]:
> > > > > > > > > > >>
> > > > > > https://lists.apache.org/thread/xmnj8h0h8ozmrgox5tydhhs7yz7q2hss
> > > > > > > > > > >> >> >> [3]:
> > > > > > > > > > >>
> > > > > > https://lists.apache.org/thread/qlf8lg90gqq41lllyy9mk2f0skqvftqj
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >> >> On Thu, Jul 30, 2026 at 3:51 AM Divjot Arora via
> > > dev <
> > > > > > > > > > >> >> >> dev@parquet.apache.org> wrote:
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >> >>> Hi Andrew,
> > > > > > > > > > >> >> >>>
> > > > > > > > > > >> >> >>> How does this interact with the ongoing
> > versioning
> > > > > > > > discussion?
> > > > > > > > > > >> Given
> > > > > > > > > > >> >> that
> > > > > > > > > > >> >> >>> new encodings are not forward compatible, I
> would
> > > > > assume
> > > > > > > ALP
> > > > > > > > is
> > > > > > > > > > >> >> included
> > > > > > > > > > >> >> >>> in
> > > > > > > > > > >> >> >>> Parquet V<next> (V3?), but it's not clear to me
> > how
> > > > > that
> > > > > > > > works
> > > > > > > > > if
> > > > > > > > > > >> it's
> > > > > > > > > > >> >> >>> merged to the parquet-format master branch now.
> > > Will
> > > > we
> > > > > > > > exclude
> > > > > > > > > > >> this
> > > > > > > > > > >> >> >>> change
> > > > > > > > > > >> >> >>> when cutting parquet-format v2.14.0?
> > > > > > > > > > >> >> >>>
> > > > > > > > > > >> >> >>> -- Div
> > > > > > > > > > >> >> >>>
> > > > > > > > > > >> >> >>> On Thu, Jul 30, 2026 at 8:51 AM Andrew Lamb <
> > > > > > > > > > >> andrewlamb11@gmail.com>
> > > > > > > > > > >> >> >>> wrote:
> > > > > > > > > > >> >> >>>
> > > > > > > > > > >> >> >>> > Hello,
> > > > > > > > > > >> >> >>> >
> > > > > > > > > > >> >> >>> > As an update, thanks to the epic work from
> many
> > > > > > > > contributors,
> > > > > > > > > > >> >> >>> especially
> > > > > > > > > > >> >> >>> > Prateek, I just merged the parquet-format
> > change
> > > > for
> > > > > > > > ALP[1].
> > > > > > > > > We
> > > > > > > > > > >> >> plan to
> > > > > > > > > > >> >> >>> > make additional editorial The remaining
> items I
> > > > know
> > > > > of
> > > > > > > are
> > > > > > > > > > >> merging
> > > > > > > > > > >> >> the
> > > > > > > > > > >> >> >>> > examples from parquet-testing[2], and then of
> > > > course
> > > > > > the
> > > > > > > > > > various
> > > > > > > > > > >> >> >>> > implementations
> > > > > > > > > > >> >> >>> >
> > > > > > > > > > >> >> >>> > Thanks again everyone. This is a great step
> > > forward
> > > > > > > > > > >> >> >>> > Andrew
> > > > > > > > > > >> >> >>> >
> > > > > > > > > > >> >> >>> > [1]:
> > > > > https://github.com/apache/parquet-format/pull/557
> > > > > > > > > > >> >> >>> > [2]:
> > > > > > https://github.com/apache/parquet-testing/pull/100
> > > > > > > > > > >> >> >>> >
> > > > > > > > > > >> >> >>> > On Tue, Jul 21, 2026 at 4:39 PM Julien Le
> Dem <
> > > > > > > > > > julien@apache.org
> > > > > > > > > > >> >
> > > > > > > > > > >> >> >>> wrote:
> > > > > > > > > > >> >> >>> >
> > > > > > > > > > >> >> >>> > > It was also in my spam folder... :(
> > > > > > > > > > >> >> >>> > > I've seen a bunch of apache list emails go
> to
> > > my
> > > > > spam
> > > > > > > > > > recently.
> > > > > > > > > > >> >> >>> > >
> > > > > > > > > > >> >> >>> > > On Tue, Jul 21, 2026 at 10:58 AM Russell
> > > Spitzer
> > > > <
> > > > > > > > > > >> >> >>> > > russell.spitzer@gmail.com>
> > > > > > > > > > >> >> >>> > > wrote:
> > > > > > > > > > >> >> >>> > >
> > > > > > > > > > >> >> >>> > > > Yes, my hope was that my vote would kick
> it
> > > out
> > > > > of
> > > > > > > spam
> > > > > > > > > but
> > > > > > > > > > >> we
> > > > > > > > > > >> >> >>> > definitely
> > > > > > > > > > >> >> >>> > > > had two other votes that also went to
> spam
> > > for
> > > > > me.
> > > > > > > > > > >> >> >>> > > >
> > > > > > > > > > >> >> >>> > > > For anyone looking the thread is here
> > > > > > > > > > >> >> >>> > > >
> > > > > > > > > > >> >>
> > > > > > >
> https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt
> > > > > > > > > > >> >> >>> > > >
> > > > > > > > > > >> >> >>> > > > On Tue, Jul 21, 2026 at 10:57 AM Micah
> > > > Kornfield
> > > > > <
> > > > > > > > > > >> >> >>> > emkornfield@gmail.com>
> > > > > > > > > > >> >> >>> > > > wrote:
> > > > > > > > > > >> >> >>> > > >
> > > > > > > > > > >> >> >>> > > > > I've seen other's votes not go to spam,
> > so
> > > > > > > hopefully
> > > > > > > > > > people
> > > > > > > > > > >> >> will
> > > > > > > > > > >> >> >>> see
> > > > > > > > > > >> >> >>> > it
> > > > > > > > > > >> >> >>> > > > in
> > > > > > > > > > >> >> >>> > > > > there inboxes (or see this thread and
> > look
> > > in
> > > > > > there
> > > > > > > > > > spam).
> > > > > > > > > > >> >> >>> > > > >
> > > > > > > > > > >> >> >>> > > > > On Tue, Jul 21, 2026 at 10:52 AM
> PRATEEK
> > > > GAUR <
> > > > > > > > > > >> >> >>> prtkgaur@gmail.com>
> > > > > > > > > > >> >> >>> > > > wrote:
> > > > > > > > > > >> >> >>> > > > >
> > > > > > > > > > >> >> >>> > > > > > Okay,
> > > > > > > > > > >> >> >>> > > > > >
> > > > > > > > > > >> >> >>> > > > > > I sent out the voting email yesterday
> > and
> > > > so
> > > > > > far
> > > > > > > > I've
> > > > > > > > > > >> heard
> > > > > > > > > > >> >> >>> from 3
> > > > > > > > > > >> >> >>> > > > people
> > > > > > > > > > >> >> >>> > > > > > that it ended up in their "spam
> > folder".
> > > > Dev
> > > > > > VPN
> > > > > > > to
> > > > > > > > > be
> > > > > > > > > > >> >> blamed
> > > > > > > > > > >> >> >>> for
> > > > > > > > > > >> >> >>> > > it?.
> > > > > > > > > > >> >> >>> > > > > >
> > > > > > > > > > >> >> >>> > > > > > What's the protocol here?
> > > > > > > > > > >> >> >>> > > > > > I let people hunt for the email in
> > their
> > > > spam
> > > > > > > > folder
> > > > > > > > > or
> > > > > > > > > > >> send
> > > > > > > > > > >> >> >>> out
> > > > > > > > > > >> >> >>> > > > another
> > > > > > > > > > >> >> >>> > > > > > one?
> > > > > > > > > > >> >> >>> > > > > >
> > > > > > > > > > >> >> >>> > > > > > Best
> > > > > > > > > > >> >> >>> > > > > > Prateek
> > > > > > > > > > >> >> >>> > > > > >
> > > > > > > > > > >> >> >>> > > > > > On Sun, Jul 19, 2026 at 10:47 PM
> > PRATEEK
> > > > > GAUR <
> > > > > > > > > > >> >> >>> prtkgaur@gmail.com>
> > > > > > > > > > >> >> >>> > > > > wrote:
> > > > > > > > > > >> >> >>> > > > > >
> > > > > > > > > > >> >> >>> > > > > > > Hi team,
> > > > > > > > > > >> >> >>> > > > > > >
> > > > > > > > > > >> >> >>> > > > > > > Vinoo has added the tests which
> were
> > > > > > requested
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > >> last
> > > > > > > > > > >> >> >>> round
> > > > > > > > > > >> >> >>> > of
> > > > > > > > > > >> >> >>> > > > > > review.
> > > > > > > > > > >> >> >>> > > > > > > In his own words.
> > > > > > > > > > >> >> >>> > > > > > >
> > > > > > > > > > >> >> >>> > > > > > > "
> > > > > > > > > > >> >> >>> > > > > > > The extreme value tests Micah
> wanted
> > > are
> > > > > done
> > > > > > > and
> > > > > > > > > > >> pushed.
> > > > > > > > > > >> >> I
> > > > > > > > > > >> >> >>> added
> > > > > > > > > > >> >> >>> > > > > > coverage
> > > > > > > > > > >> >> >>> > > > > > > for values that need the full FOR
> bit
> > > > width
> > > > > > > after
> > > > > > > > > the
> > > > > > > > > > >> >> frame
> > > > > > > > > > >> >> >>> of
> > > > > > > > > > >> >> >>> > > > > reference
> > > > > > > > > > >> >> >>> > > > > > is
> > > > > > > > > > >> >> >>> > > > > > > applied: 64-bit (the case where the
> > > > signed
> > > > > > max
> > > > > > > > > minus
> > > > > > > > > > >> min
> > > > > > > > > > >> >> >>> > > subtraction
> > > > > > > > > > >> >> >>> > > > > > > overflows), 63-bit (the
> non-overflow
> > > > case),
> > > > > > and
> > > > > > > > > > 32-bit
> > > > > > > > > > >> for
> > > > > > > > > > >> >> >>> > floats.
> > > > > > > > > > >> >> >>> > > > All
> > > > > > > > > > >> >> >>> > > > > > > lossless with zero exceptions. I
> also
> > > > > > verified
> > > > > > > > them
> > > > > > > > > > >> >> through
> > > > > > > > > > >> >> >>> Arrow
> > > > > > > > > > >> >> >>> > > C++
> > > > > > > > > > >> >> >>> > > > > ALP
> > > > > > > > > > >> >> >>> > > > > > > reader (the local build), so the
> > > > > Java-written
> > > > > > > > > extreme
> > > > > > > > > > >> >> columns
> > > > > > > > > > >> >> >>> > > decode
> > > > > > > > > > >> >> >>> > > > > > > correctly in C++, bit-exact against
> > the
> > > > > > > expected
> > > > > > > > > > >> values."
> > > > > > > > > > >> >> >>> > > > > > >
> > > > > > > > > > >> >> >>> > > > > > > State of the PR (#3397):
> > > > > > > > > > >> >> >>> > > > > > >
> > > > > > > > > > >> >> >>> > > > > > > - Merges cleanly with master (I
> > > pulled
> > > > > > > master
> > > > > > > > in
> > > > > > > > > > and
> > > > > > > > > > >> >> >>> resolved
> > > > > > > > > > >> >> >>> > > the
> > > > > > > > > > >> >> >>> > > > > > > conflicts).
> > > > > > > > > > >> >> >>> > > > > > > - 41 of 44 review threads
> > resolved.
> > > > The
> > > > > 3
> > > > > > > > > > remaining
> > > > > > > > > > >> are
> > > > > > > > > > >> >> >>> > > > intentional:
> > > > > > > > > > >> >> >>> > > > > > > one deferred perf nit, one that
> > > stays
> > > > > open
> > > > > > > > until
> > > > > > > > > > >> >> >>> > parquet-format
> > > > > > > > > > >> >> >>> > > > > ships
> > > > > > > > > > >> >> >>> > > > > > ALP,
> > > > > > > > > > >> >> >>> > > > > > >
> > > > > > > > > > >> >> >>> > > > > > > "
> > > > > > > > > > >> >> >>> > > > > > >
> > > > > > > > > > >> >> >>> > > > > > > With this both the
> > > > > > > > > > >> >> >>> > > > > > > 1) C++ PR :
> > > > > > > > > > >> >> >>>
> > https://github.com/apache/arrow/pull/48345/changes
> > > > > > > > > > >> >> >>> > > > > > > 2) Java PR :
> > > > > > > > > > >> >> >>>
> https://github.com/apache/parquet-java/pull/3397
> > > > > > > > > > >> >> >>> > > > > > >
> > > > > > > > > > >> >> >>> > > > > > > Are cross language tested and range
> > of
> > > > test
> > > > > > > > > > scenarios.
> > > > > > > > > > >> >> >>> > > > > > >
> > > > > > > > > > >> >> >>> > > > > > > Best
> > > > > > > > > > >> >> >>> > > > > > > Prateek
> > > > > > > > > > >> >> >>> > > > > > >
> > > > > > > > > > >> >> >>> > > > > > >
> > > > > > > > > > >> >> >>> > > > > > > On Mon, Jul 13, 2026 at 8:38 AM
> > PRATEEK
> > > > > GAUR
> > > > > > <
> > > > > > > > > > >> >> >>> prtkgaur@gmail.com
> > > > > > > > > > >> >> >>> > >
> > > > > > > > > > >> >> >>> > > > > wrote:
> > > > > > > > > > >> >> >>> > > > > > >
> > > > > > > > > > >> >> >>> > > > > > >> Thanks Matt,
> > > > > > > > > > >> >> >>> > > > > > >>
> > > > > > > > > > >> >> >>> > > > > > >> I'll be on it this week.
> > > > > > > > > > >> >> >>> > > > > > >>
> > > > > > > > > > >> >> >>> > > > > > >> Best
> > > > > > > > > > >> >> >>> > > > > > >> Prateek
> > > > > > > > > > >> >> >>> > > > > > >>
> > > > > > > > > > >> >> >>> > > > > > >> On Mon, Jul 13, 2026 at 8:36 AM
> Matt
> > > > > Topol <
> > > > > > > > > > >> >> >>> > > zotthewizard@gmail.com>
> > > > > > > > > > >> >> >>> > > > > > >> wrote:
> > > > > > > > > > >> >> >>> > > > > > >>
> > > > > > > > > > >> >> >>> > > > > > >>> There is also the Go
> implementation
> > > > > > submitted
> > > > > > > > by
> > > > > > > > > > >> Arnav (
> > > > > > > > > > >> >> >>> > > > > > >>>
> > > > > > https://github.com/apache/arrow-go/pull/704)
> > > > > > > > > which
> > > > > > > > > > >> is
> > > > > > > > > > >> >> >>> waiting
> > > > > > > > > > >> >> >>> > > for
> > > > > > > > > > >> >> >>> > > > > > >>> updates.
> > > > > > > > > > >> >> >>> > > > > > >>>
> > > > > > > > > > >> >> >>> > > > > > >>> Just wanted to make sure it
> didn't
> > > get
> > > > > lost
> > > > > > > > here.
> > > > > > > > > > >> >> >>> > > > > > >>>
> > > > > > > > > > >> >> >>> > > > > > >>> --Matt
> > > > > > > > > > >> >> >>> > > > > > >>>
> > > > > > > > > > >> >> >>> > > > > > >>> On Mon, Jul 13, 2026 at 11:30 AM
> > > > PRATEEK
> > > > > > > GAUR <
> > > > > > > > > > >> >> >>> > > prtkgaur@gmail.com>
> > > > > > > > > > >> >> >>> > > > > > >>> wrote:
> > > > > > > > > > >> >> >>> > > > > > >>>
> > > > > > > > > > >> >> >>> > > > > > >>> > Hi all,
> > > > > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > > > > >> >> >>> > > > > > >>> > I'd like to share a status
> update
> > > on
> > > > > the
> > > > > > > ALP
> > > > > > > > > > >> encoding
> > > > > > > > > > >> >> >>> effort
> > > > > > > > > > >> >> >>> > > and
> > > > > > > > > > >> >> >>> > > > > get
> > > > > > > > > > >> >> >>> > > > > > a
> > > > > > > > > > >> >> >>> > > > > > >>> feel
> > > > > > > > > > >> >> >>> > > > > > >>> > from the community before
> > starting
> > > a
> > > > > > formal
> > > > > > > > > vote.
> > > > > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > > > > >> >> >>> > > > > > >>> > We now have the specification
> > plus
> > > > > > > > > > implementations
> > > > > > > > > > >> in
> > > > > > > > > > >> >> >>> both
> > > > > > > > > > >> >> >>> > > > > languages:
> > > > > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > > > > >> >> >>> > > > > > >>> > - Spec:
> > > > > > > > apache/parquet-format#557 —
> > > > > > > > > > >> GH-533
> > > > > > > > > > >> >> >>> Add
> > > > > > > > > > >> >> >>> > ALP
> > > > > > > > > > >> >> >>> > > > > > encoding
> > > > > > > > > > >> >> >>> > > > > > >>> > specification
> > > > > > > > > > >> >> >>> > > > > > >>> > - C++ (Arrow):
> > > > apache/arrow#48345
> > > > > —
> > > > > > > > > GH-48701
> > > > > > > > > > >> >> >>> > > [C++][Parquet]
> > > > > > > > > > >> >> >>> > > > > Add
> > > > > > > > > > >> >> >>> > > > > > >>> ALPpd
> > > > > > > > > > >> >> >>> > > > > > >>> > encoding
> > > > > > > > > > >> >> >>> > > > > > >>> > - Java:
> > > > > > > apache/parquet-java#3397
> > > > > > > > —
> > > > > > > > > > >> Parquet
> > > > > > > > > > >> >> >>> Java
> > > > > > > > > > >> >> >>> > ALP
> > > > > > > > > > >> >> >>> > > > > > >>> > Implementation (by Vinoo)
> > > > > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > > > > >> >> >>> > > > > > >>> > Current state:
> > > > > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > > > > >> >> >>> > > > > > >>> > - The C++ implementation
> has
> > > been
> > > > > > > through
> > > > > > > > > 3-4
> > > > > > > > > > >> >> rounds
> > > > > > > > > > >> >> >>> of
> > > > > > > > > > >> >> >>> > > > review.
> > > > > > > > > > >> >> >>> > > > > > >>> > - The Java implementation
> has
> > > > been
> > > > > > > > tracking
> > > > > > > > > > >> well
> > > > > > > > > > >> >> and
> > > > > > > > > > >> >> >>> has
> > > > > > > > > > >> >> >>> > > had
> > > > > > > > > > >> >> >>> > > > > > >>> initial
> > > > > > > > > > >> >> >>> > > > > > >>> > reviews from
> contributors,
> > > with
> > > > > no
> > > > > > > > major
> > > > > > > > > > >> >> remaining
> > > > > > > > > > >> >> >>> > > issues.
> > > > > > > > > > >> >> >>> > > > > > >>> > - Cross-language
> > compatibility
> > > > > tests
> > > > > > > are
> > > > > > > > > > >> passing:
> > > > > > > > > > >> >> the
> > > > > > > > > > >> >> >>> > Arrow
> > > > > > > > > > >> >> >>> > > > C++
> > > > > > > > > > >> >> >>> > > > > > >>> decoder
> > > > > > > > > > >> >> >>> > > > > > >>> > reads Java-written data
> > > > > bit-exactly
> > > > > > > > > across
> > > > > > > > > > >> >> ~1.56M
> > > > > > > > > > >> >> >>> > values
> > > > > > > > > > >> >> >>> > > > and
> > > > > > > > > > >> >> >>> > > > > 18
> > > > > > > > > > >> >> >>> > > > > > >>> > fixtures, covering V1 and
> > V2
> > > > > pages,
> > > > > > > > > > multiple
> > > > > > > > > > >> >> vector
> > > > > > > > > > >> >> >>> > > sizes,
> > > > > > > > > > >> >> >>> > > > > and
> > > > > > > > > > >> >> >>> > > > > > >>> > several
> > > > > > > > > > >> >> >>> > > > > > >>> > real datasets — zero
> > > > mismatches.
> > > > > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > > > > >> >> >>> > > > > > >>> > As raised in the java-pr we
> will
> > > be
> > > > > > adding
> > > > > > > > > > >> coverage
> > > > > > > > > > >> >> for
> > > > > > > > > > >> >> >>> > > extreme
> > > > > > > > > > >> >> >>> > > > > > values
> > > > > > > > > > >> >> >>> > > > > > >>> > (those requiring
> > > > > > > > > > >> >> >>> > > > > > >>> > 63-64 bits after FOR is
> applied)
> > > > > before
> > > > > > we
> > > > > > > > > > close a
> > > > > > > > > > >> >> vote.
> > > > > > > > > > >> >> >>> > We'll
> > > > > > > > > > >> >> >>> > > > aim
> > > > > > > > > > >> >> >>> > > > > > to
> > > > > > > > > > >> >> >>> > > > > > >>> get
> > > > > > > > > > >> >> >>> > > > > > >>> > that
> > > > > > > > > > >> >> >>> > > > > > >>> > done in parallel.
> > > > > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > > > > >> >> >>> > > > > > >>> > Plan: unless there are
> > objections,
> > > I
> > > > > > intend
> > > > > > > > to
> > > > > > > > > > >> start a
> > > > > > > > > > >> >> >>> vote
> > > > > > > > > > >> >> >>> > at
> > > > > > > > > > >> >> >>> > > > the
> > > > > > > > > > >> >> >>> > > > > > end
> > > > > > > > > > >> >> >>> > > > > > >>> of
> > > > > > > > > > >> >> >>> > > > > > >>> > this week/early
> > > > > > > > > > >> >> >>> > > > > > >>> > next week.
> > > > > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > > > > >> >> >>> > > > > > >>> > Thanks
> > > > > > > > > > >> >> >>> > > > > > >>> > Prateek
> > > > > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > > > > >> >> >>> > > > > > >>> > On Wed, Jul 1, 2026 at 8:50 AM
> > > > PRATEEK
> > > > > > > GAUR <
> > > > > > > > > > >> >> >>> > > prtkgaur@gmail.com>
> > > > > > > > > > >> >> >>> > > > > > >>> wrote:
> > > > > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > > > > >> >> >>> > > > > > >>> > > Hi Team,
> > > > > > > > > > >> >> >>> > > > > > >>> > >
> > > > > > > > > > >> >> >>> > > > > > >>> > > Just wanted to provide some
> > > updates
> > > > > on
> > > > > > > ALP.
> > > > > > > > > > >> >> >>> > > > > > >>> > > Micah and I have done 3-4
> > rounds
> > > of
> > > > > > > review
> > > > > > > > > for
> > > > > > > > > > >> the
> > > > > > > > > > >> >> c++
> > > > > > > > > > >> >> >>> PR :
> > > > > > > > > > >> >> >>> > > > > > >>> > >
> > > > > > > https://github.com/apache/arrow/pull/48345
> > > > > > > > > > >> >> >>> > > > > > >>> > >
> > > > > > > > > > >> >> >>> > > > > > >>> > > For the Java implementation
> > Vinoo
> > > > has
> > > > > > > been
> > > > > > > > > > >> working
> > > > > > > > > > >> >> on
> > > > > > > > > > >> >> >>> > > following
> > > > > > > > > > >> >> >>> > > > > PR
> > > > > > > > > > >> >> >>> > > > > > :
> > > > > > > > > > >> >> >>> > > > > > >>> > >
> > > > > > > > > > https://github.com/apache/parquet-java/pull/3397
> > > > > > > > > > >> >> >>> > > > > > >>> > >
> > > > > > > > > > >> >> >>> > > > > > >>> > > Best
> > > > > > > > > > >> >> >>> > > > > > >>> > > Prateek
> > > > > > > > > > >> >> >>> > > > > > >>> > >
> > > > > > > > > > >> >> >>> > > > > > >>> > > On Tue, May 5, 2026 at
> 1:50 PM
> > > > Micah
> > > > > > > > > Kornfield
> > > > > > > > > > <
> > > > > > > > > > >> >> >>> > > > > > >>> emkornfield@gmail.com>
> > > > > > > > > > >> >> >>> > > > > > >>> > > wrote:
> > > > > > > > > > >> >> >>> > > > > > >>> > >
> > > > > > > > > > >> >> >>> > > > > > >>> > >> Hi Antoine,
> > > > > > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > Apologies if the question
> > was
> > > > > > already
> > > > > > > > > asked,
> > > > > > > > > > >> but
> > > > > > > > > > >> >> >>> should
> > > > > > > > > > >> >> >>> > we
> > > > > > > > > > >> >> >>> > > > > care
> > > > > > > > > > >> >> >>> > > > > > >>> about
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can
> FLOAT +
> > > ALP
> > > > > be
> > > > > > > more
> > > > > > > > > > >> >> efficient
> > > > > > > > > > >> >> >>> than
> > > > > > > > > > >> >> >>> > > > > FLOAT16
> > > > > > > > > > >> >> >>> > > > > > +
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4
> for
> > > > > example?
> > > > > > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> It was. We thought we could
> > > defer
> > > > > it
> > > > > > > for
> > > > > > > > > the
> > > > > > > > > > >> >> >>> following
> > > > > > > > > > >> >> >>> > > > reasons:
> > > > > > > > > > >> >> >>> > > > > > >>> > >> 1. It's not clear there
> are a
> > > lot
> > > > > of
> > > > > > > easy
> > > > > > > > > > >> >> reference
> > > > > > > > > > >> >> >>> > > datasets
> > > > > > > > > > >> >> >>> > > > to
> > > > > > > > > > >> >> >>> > > > > > >>> test
> > > > > > > > > > >> >> >>> > > > > > >>> > its
> > > > > > > > > > >> >> >>> > > > > > >>> > >> effectiveness. It does look
> > > like
> > > > > > there
> > > > > > > > > might
> > > > > > > > > > be
> > > > > > > > > > >> >> one
> > > > > > > > > > >> >> >>> or
> > > > > > > > > > >> >> >>> > two
> > > > > > > > > > >> >> >>> > > on
> > > > > > > > > > >> >> >>> > > > > > >>> > huggingface
> > > > > > > > > > >> >> >>> > > > > > >>> > >> (e.g.
> > > > > > > > > > >> >> https://huggingface.co/datasets/kikitora/curdie
> > > > > > > > > > >> >> >>> ).
> > > > > > > > > > >> >> >>> > > > > > >>> > >> 2. It seemed likely that
> > float
> > > 16
> > > > > was
> > > > > > > > more
> > > > > > > > > > >> likely
> > > > > > > > > > >> >> >>> used
> > > > > > > > > > >> >> >>> > for
> > > > > > > > > > >> >> >>> > > > > values
> > > > > > > > > > >> >> >>> > > > > > >>> that
> > > > > > > > > > >> >> >>> > > > > > >>> > >> were less likely to reduce
> to
> > > > > decimal
> > > > > > > > > values.
> > > > > > > > > > >> >> >>> > > > > > >>> > >> 3. It could be added as an
> > > > > extension
> > > > > > > > later
> > > > > > > > > if
> > > > > > > > > > >> >> needed.
> > > > > > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> Cheers,
> > > > > > > > > > >> >> >>> > > > > > >>> > >> Micah
> > > > > > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> On Tue, May 5, 2026 at
> 1:40 PM
> > > > > Antoine
> > > > > > > > > Pitrou
> > > > > > > > > > <
> > > > > > > > > > >> >> >>> > > > > antoine@python.org
> > > > > > > > > > >> >> >>> > > > > > >
> > > > > > > > > > >> >> >>> > > > > > >>> > wrote:
> > > > > > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > Hello,
> > > > > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > Apologies if the question
> > was
> > > > > > already
> > > > > > > > > asked,
> > > > > > > > > > >> but
> > > > > > > > > > >> >> >>> should
> > > > > > > > > > >> >> >>> > we
> > > > > > > > > > >> >> >>> > > > > care
> > > > > > > > > > >> >> >>> > > > > > >>> about
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can
> FLOAT +
> > > ALP
> > > > > be
> > > > > > > more
> > > > > > > > > > >> >> efficient
> > > > > > > > > > >> >> >>> than
> > > > > > > > > > >> >> >>> > > > > FLOAT16
> > > > > > > > > > >> >> >>> > > > > > +
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4
> for
> > > > > example?
> > > > > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > Regards
> > > > > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > Antoine.
> > > > > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > Le 30/04/2026 à 01:10,
> > PRATEEK
> > > > > GAUR
> > > > > > a
> > > > > > > > > écrit
> > > > > > > > > > :
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > > Thanks Andrew and Micah
> > for
> > > > > review
> > > > > > > > > > feedback
> > > > > > > > > > >> on
> > > > > > > > > > >> >> >>> the two
> > > > > > > > > > >> >> >>> > > > PR's
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > > 1) (c++ arrow repo)
> > > > > > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > > > > > >> https://github.com/apache/arrow/pull/48345/changes
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > > 2) (parquet-format repo)
> > > > > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > > > > >> >> https://github.com/apache/parquet-format/pull/557
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > > I have addressed all
> > > (unless I
> > > > > > > missed
> > > > > > > > > > >> >> something)
> > > > > > > > > > >> >> >>> > > comments
> > > > > > > > > > >> >> >>> > > > on
> > > > > > > > > > >> >> >>> > > > > > >>> the two
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > PR's.
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > > Best
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > > Prateek
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > > On Sat, Apr 25, 2026 at
> > > > 1:08 PM
> > > > > > > > PRATEEK
> > > > > > > > > > >> GAUR <
> > > > > > > > > > >> >> >>> > > > > > >>> prtkgaur@gmail.com>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> wrote:
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >> Thanks Andrew and
> Micah.
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >> `fair amount of
> feedback
> > on
> > > > at
> > > > > > > least
> > > > > > > > > the
> > > > > > > > > > >> >> >>> > > implementations`
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >> For the c++ I have
> > already
> > > > > > started
> > > > > > > > > > >> addressing
> > > > > > > > > > >> >> the
> > > > > > > > > > >> >> >>> > > > > feedback, I
> > > > > > > > > > >> >> >>> > > > > > >>> > should
> > > > > > > > > > >> >> >>> > > > > > >>> > >> be
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >> done with that
> > > > Monday/Tuesday.
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >> I think Vinoo too has
> > been
> > > > > making
> > > > > > > > good
> > > > > > > > > > >> >> progress
> > > > > > > > > > >> >> >>> on
> > > > > > > > > > >> >> >>> > the
> > > > > > > > > > >> >> >>> > > > Java
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > implementation.
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >> Best
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >> Prateek
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >> On Sat, Apr 25, 2026 at
> > > > > 12:55 PM
> > > > > > > > Andrew
> > > > > > > > > > >> Lamb <
> > > > > > > > > > >> >> >>> > > > > > >>> > andrewlamb11@gmail.com
> > > > > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >> wrote:
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>> Got it. Thank you for
> > the
> > > > > > > > > clarification
> > > > > > > > > > >> -- I
> > > > > > > > > > >> >> >>> will
> > > > > > > > > > >> >> >>> > try
> > > > > > > > > > >> >> >>> > > > and
> > > > > > > > > > >> >> >>> > > > > > look
> > > > > > > > > > >> >> >>> > > > > > >>> > into
> > > > > > > > > > >> >> >>> > > > > > >>> > >> the
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>> spec and the Rust
> > > > > > > implementation[1]
> > > > > > > > in
> > > > > > > > > > >> this
> > > > > > > > > > >> >> next
> > > > > > > > > > >> >> >>> > week
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>> [1]:
> > > > > > > > > > >> >> >>> https://github.com/apache/arrow-rs/pull/9372
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>> On Sat, Apr 25, 2026
> at
> > > > > 12:01 PM
> > > > > > > > Micah
> > > > > > > > > > >> >> >>> Kornfield <
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > emkornfield@gmail.com>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>> wrote:
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>> Hi Andrew,
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>> I think there is a
> fair
> > > > > amount
> > > > > > of
> > > > > > > > > > >> feedback
> > > > > > > > > > >> >> on
> > > > > > > > > > >> >> >>> at
> > > > > > > > > > >> >> >>> > > least
> > > > > > > > > > >> >> >>> > > > > the
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>> implementations,
> > > typically
> > > > I
> > > > > > > think
> > > > > > > > > > we've
> > > > > > > > > > >> >> waited
> > > > > > > > > > >> >> >>> > till
> > > > > > > > > > >> >> >>> > > > they
> > > > > > > > > > >> >> >>> > > > > > are
> > > > > > > > > > >> >> >>> > > > > > >>> > >> close to
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>> mergeable before a
> > final
> > > > > vote.
> > > > > > > > > > >> Otherwise I
> > > > > > > > > > >> >> >>> agree
> > > > > > > > > > >> >> >>> > we
> > > > > > > > > > >> >> >>> > > > are
> > > > > > > > > > >> >> >>> > > > > > very
> > > > > > > > > > >> >> >>> > > > > > >>> > >> close.
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>> -Micah
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>> On Saturday, April
> 25,
> > > > 2026,
> > > > > > > Andrew
> > > > > > > > > > Lamb
> > > > > > > > > > >> <
> > > > > > > > > > >> >> >>> > > > > > >>> andrewlamb11@gmail.com
> > > > > > > > > > >> >> >>> > > > > > >>> > >
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > wrote:
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> Thanks Prateek,
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> I think from this
> > > content
> > > > it
> > > > > > > looks
> > > > > > > > > to
> > > > > > > > > > me
> > > > > > > > > > >> >> like
> > > > > > > > > > >> >> >>> we
> > > > > > > > > > >> >> >>> > are
> > > > > > > > > > >> >> >>> > > > > ready
> > > > > > > > > > >> >> >>> > > > > > >>> to
> > > > > > > > > > >> >> >>> > > > > > >>> > >> start a
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> vote to explicitly
> > > accept
> > > > > ALP
> > > > > > > into
> > > > > > > > > > >> Parquet
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> Does anyone know of
> a
> > > > reason
> > > > > > we
> > > > > > > > > should
> > > > > > > > > > >> >> >>> postpone it
> > > > > > > > > > >> >> >>> > > for
> > > > > > > > > > >> >> >>> > > > > > >>> longer?
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> Perhaps someone
> needs
> > > some
> > > > > > more
> > > > > > > > time
> > > > > > > > > > to
> > > > > > > > > > >> >> >>> review?
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> Andrew
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> On Wed, Apr 22, 2026
> > at
> > > > > > 1:00 PM
> > > > > > > > > > PRATEEK
> > > > > > > > > > >> >> GAUR <
> > > > > > > > > > >> >> >>> > > > > > >>> > prtkgaur@gmail.com>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> wrote:
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Hi team,
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Hope everyone is
> > doing
> > > > > well.
> > > > > > I
> > > > > > > > got
> > > > > > > > > a
> > > > > > > > > > >> >> chance
> > > > > > > > > > >> >> >>> to
> > > > > > > > > > >> >> >>> > work
> > > > > > > > > > >> >> >>> > > > > > >>> through all
> > > > > > > > > > >> >> >>> > > > > > >>> > >> the
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> remaining feedback
> > and
> > > > > update
> > > > > > > the
> > > > > > > > > > spec
> > > > > > > > > > >> >> doc.
> > > > > > > > > > >> >> >>> Here
> > > > > > > > > > >> >> >>> > > are
> > > > > > > > > > >> >> >>> > > > > the
> > > > > > > > > > >> >> >>> > > > > > >>> new
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > artifacts
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 1) Spec document :
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > > > > >> >> >>> > > > > > >>>
> > > > > > > > > > >> >> >>> > > > > >
> > > > > > > > > > >> >> >>> > > > >
> > > > > > > > > > >> >> >>> > > >
> > > > > > > > > > >> >> >>> > >
> > > > > > > > > > >> >> >>> >
> > > > > > > > > > >> >> >>>
> > > > > > > > > > >> >>
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-
> > > > 3fPtD_aWt/edit
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 2) Spec document in
> > > > parquet
> > > > > > > > format
> > > > > > > > > > >> repo :
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> >
> > > https://github.com/apache/parquet-format/pull/557
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 3) Alp
> implementation
> > > in
> > > > > > arrow
> > > > > > > > c++
> > > > > > > > > > >> repo :
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> >
> > > https://github.com/apache/arrow/pull/48345/changes
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 4) Alp
> implementation
> > > in
> > > > > > > > > parquet-java
> > > > > > > > > > >> >> repo :
> > > > > > > > > > >> >> >>> Work
> > > > > > > > > > >> >> >>> > > for
> > > > > > > > > > >> >> >>> > > > > > >>> Vinoo and
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > Julien
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> >
> > https://github.com/apache/parquet-java/pull/3397
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 5) PR with test and
> > > > > > > benchmarking
> > > > > > > > > > >> >> artifacts in
> > > > > > > > > > >> >> >>> > > > > > >>> parquet-testing
> > > > > > > > > > >> >> >>> > > > > > >>> > >> repo :
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> >
> > > https://github.com/apache/parquet-testing/pull/100
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> And
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> - Go : Arnav
> just
> > > > > > submitted
> > > > > > > > an
> > > > > > > > > in
> > > > > > > > > > >> >> >>> progress
> > > > > > > > > > >> >> >>> > > > > > >>> implementation
> > > > > > > > > > >> >> >>> > > > > > >>> > in
> > > > > > > > > > >> >> >>> > > > > > >>> > >> Go.
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> https://github.com/apache/arrow-go/pull/704
> > > > > > > > > > >> >> >>> > (I
> > > > > > > > > > >> >> >>> > > > > > haven't
> > > > > > > > > > >> >> >>> > > > > > >>> > >> started
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> looking at it
> > yet)
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> - Rust : I
> > remember
> > > > > > Andrew
> > > > > > > > > > >> mentioned
> > > > > > > > > > >> >> that
> > > > > > > > > > >> >> >>> > this
> > > > > > > > > > >> >> >>> > > > work
> > > > > > > > > > >> >> >>> > > > > > is
> > > > > > > > > > >> >> >>> > > > > > >>> also
> > > > > > > > > > >> >> >>> > > > > > >>> > >> in
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> progress (So 4
> > > > > > languages!)
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> *Arrow C++
> > > > implementation *
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> The PR is out and
> was
> > > > also
> > > > > > used
> > > > > > > > by
> > > > > > > > > > >> >> Antoine to
> > > > > > > > > > >> >> >>> > > report
> > > > > > > > > > >> >> >>> > > > > the
> > > > > > > > > > >> >> >>> > > > > > >>> > numbers
> > > > > > > > > > >> >> >>> > > > > > >>> > >> as
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> reported here.
> Micah
> > > and
> > > > > > > > Konstantin
> > > > > > > > > > >> have
> > > > > > > > > > >> >> >>> given 1
> > > > > > > > > > >> >> >>> > > > round
> > > > > > > > > > >> >> >>> > > > > of
> > > > > > > > > > >> >> >>> > > > > > >>> > >> feedback
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> and I'm addressing
> > them
> > > > > > today.
> > > > > > > > > Please
> > > > > > > > > > >> note
> > > > > > > > > > >> >> >>> that
> > > > > > > > > > >> >> >>> > the
> > > > > > > > > > >> >> >>> > > > > > default
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> optimization flag
> for
> > > > > > compiling
> > > > > > > > is
> > > > > > > > > O2
> > > > > > > > > > >> and
> > > > > > > > > > >> >> not
> > > > > > > > > > >> >> >>> > Q3. I
> > > > > > > > > > >> >> >>> > > > got
> > > > > > > > > > >> >> >>> > > > > > >>> around
> > > > > > > > > > >> >> >>> > > > > > >>> > >> 70%
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> performance
> > improvement
> > > > in
> > > > > > the
> > > > > > > > > > decoding
> > > > > > > > > > >> >> speed
> > > > > > > > > > >> >> >>> > when
> > > > > > > > > > >> >> >>> > > > > using
> > > > > > > > > > >> >> >>> > > > > > >>> the O3
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > flag.
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> *Parqet-MR Java
> > > > > > implementation
> > > > > > > > > > (working
> > > > > > > > > > >> >> with
> > > > > > > > > > >> >> >>> > Vinoo
> > > > > > > > > > >> >> >>> > > > and
> > > > > > > > > > >> >> >>> > > > > > >>> Julien)
> > > > > > > > > > >> >> >>> > > > > > >>> > >> and
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > **Cross
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Language testing*
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Let me know if
> > you
> > > > have
> > > > > > any
> > > > > > > > > > >> questions
> > > > > > > > > > >> >> or
> > > > > > > > > > >> >> >>> > > > feedback.
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Now pasting some
> > > > > performance
> > > > > > > > > numbers
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Table 1: C++ ALP
> > > > Double
> > > > > > > > Decode —
> > > > > > > > > > >> >> Spotify
> > > > > > > > > > >> >> >>> > Columns
> > > > > > > > > > >> >> >>> > > > > > >>> (Graviton
> > > > > > > > > > >> >> >>> > > > > > >>> > 3,
> > > > > > > > > > >> >> >>> > > > > > >>> > >> ARM
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Neoverse V1)
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>>
> > > > > > > > > > >> >> >>>
> > > > > > > ┌──────────────────┬──────────────┬──────────────┬─────────┐
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ Column
> > │
> > > > > -O2
> > > > > > > > (MB/s)
> > > > > > > > > > │
> > > > > > > > > > >> >> -O3
> > > > > > > > > > >> >> >>> > > (MB/s) │
> > > > > > > > > > >> >> >>> > > > > > >>> Speedup │
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>>
> > > > > > > > > > >> >> >>>
> > > > > > > ├──────────────────┼──────────────┼──────────────┼─────────┤
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ valence
> > │
> > > > > > > 3,155
> > > > > > > > > > │
> > > > > > > > > > >> >> >>> 5,523
> > > > > > > > > > >> >> >>> > > │
> > > > > > > > > > >> >> >>> > > > > > >>> 1.75x │
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ danceability
> > │
> > > > > > > 3,233
> > > > > > > > > > │
> > > > > > > > > > >> >> >>> 5,685
> > > > > > > > > > >> >> >>> > > │
> > > > > > > > > > >> >> >>> > > > > > >>> 1.76x │
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ energy
> > │
> > > > > > > 3,197
> > > > > > > > > > │
> > > > > > > > > > >> >> >>> 5,652
> > > > > > > > > > >> >> >>> > > │
> > > > > > > > > > >> >> >>> > > > > > >>> 1.77x │
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ loudness
> > │
> > > > > > > 3,186
> > > > > > > > > > │
> > > > > > > > > > >> >> >>> 5,473
> > > > > > > > > > >> >> >>> > > │
> > > > > > > > > > >> >> >>> > > > > > >>> 1.72x │
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>>
> > > > > > > > > > >> >> >>>
> > > > > > > └──────────────────┴──────────────┴──────────────┴─────────┘
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> On Wed, Feb 25,
> 2026
> > at
> > > > > > 9:49 AM
> > > > > > > > > > PRATEEK
> > > > > > > > > > >> >> GAUR
> > > > > > > > > > >> >> >>> <
> > > > > > > > > > >> >> >>> > > > > > >>> > prtkgaur@gmail.com
> > > > > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> wrote:
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> @Micah Kornfield <
> > > > > > > > > > >> emkornfield@gmail.com>
> > > > > > > > > > >> >> :
> > > > > > > > > > >> >> >>> Got
> > > > > > > > > > >> >> >>> > > it.
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> @Andrew Lamb <
> > > > > > > > > > andrewlamb11@gmail.com>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> Do you think it
> > would
> > > > be
> > > > > > good
> > > > > > > > to
> > > > > > > > > > >> start
> > > > > > > > > > >> >> >>> moving
> > > > > > > > > > >> >> >>> > the
> > > > > > > > > > >> >> >>> > > > > spec
> > > > > > > > > > >> >> >>> > > > > > >>> > >> development
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> into
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> markdown format,
> in
> > > > > > > preparation
> > > > > > > > > for
> > > > > > > > > > >> >> >>> finalizing
> > > > > > > > > > >> >> >>> > > it?
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> Yes I'll update
> the
> > > > > numbers
> > > > > > > for
> > > > > > > > > some
> > > > > > > > > > >> of
> > > > > > > > > > >> >> the
> > > > > > > > > > >> >> >>> > > > examples I
> > > > > > > > > > >> >> >>> > > > > > >>> have in
> > > > > > > > > > >> >> >>> > > > > > >>> > >> the
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> spec based
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> on the updated
> > header
> > > > > size.
> > > > > > > Then
> > > > > > > > > we
> > > > > > > > > > >> >> should
> > > > > > > > > > >> >> >>> be
> > > > > > > > > > >> >> >>> > good
> > > > > > > > > > >> >> >>> > > > to
> > > > > > > > > > >> >> >>> > > > > go
> > > > > > > > > > >> >> >>> > > > > > >>> for
> > > > > > > > > > >> >> >>> > > > > > >>> > the
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> markdown format.
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> Thanks everyone!
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> Andrew
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> On Tue, Feb 17,
> > 2026
> > > at
> > > > > > > 7:28 PM
> > > > > > > > > > >> PRATEEK
> > > > > > > > > > >> >> >>> GAUR <
> > > > > > > > > > >> >> >>> > > > > > >>> > >> prtkgaur@gmail.com>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> wrote:
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Hi team,
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 1) Andrew
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for
> > > > working
> > > > > > on
> > > > > > > > test
> > > > > > > > > > >> files.
> > > > > > > > > > >> >> >>> My PR
> > > > > > > > > > >> >> >>> > > did
> > > > > > > > > > >> >> >>> > > > > add
> > > > > > > > > > >> >> >>> > > > > > >>> all
> > > > > > > > > > >> >> >>> > > > > > >>> > the
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > test
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> files I
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> used to
> > > benchmark
> > > > on
> > > > > > > > > datasets.
> > > > > > > > > > >> >> Maybe
> > > > > > > > > > >> >> >>> we
> > > > > > > > > > >> >> >>> > can
> > > > > > > > > > >> >> >>> > > > club
> > > > > > > > > > >> >> >>> > > > > > it
> > > > > > > > > > >> >> >>> > > > > > >>> > >> together.
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> WIll also
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> aid
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> cross
> language
> > > > > testing
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Kosta
> > Tarasov
> > > > > > working
> > > > > > > > on
> > > > > > > > > > Rust
> > > > > > > > > > >> >> >>> > > > implementation.
> > > > > > > > > > >> >> >>> > > > > > >>> This is
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > great.
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> Thanks
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 2) Antoine
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks a
> lot
> > > for
> > > > > > > > reporting
> > > > > > > > > > the
> > > > > > > > > > >> >> >>> numbers
> > > > > > > > > > >> >> >>> > on
> > > > > > > > > > >> >> >>> > > > AMD.
> > > > > > > > > > >> >> >>> > > > > > >>> Looks
> > > > > > > > > > >> >> >>> > > > > > >>> > >> like
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > you
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> are
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> getting 8X
> the
> > > > > > decoding
> > > > > > > > > > >> >> performance of
> > > > > > > > > > >> >> >>> > BSS.
> > > > > > > > > > >> >> >>> > > > This
> > > > > > > > > > >> >> >>> > > > > > is
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > amazing!!.
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for
> > > > > > > acknowledging
> > > > > > > > > the
> > > > > > > > > > >> >> >>> sampling
> > > > > > > > > > >> >> >>> > > > design.
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - I agree
> with
> > > you
> > > > > on
> > > > > > > > > > >> Fastlanes. In
> > > > > > > > > > >> >> >>> some
> > > > > > > > > > >> >> >>> > > crude
> > > > > > > > > > >> >> >>> > > > > > >>> > >> experiments I
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> didn't get
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> a good perf
> > > > benefit
> > > > > > from
> > > > > > > > it
> > > > > > > > > on
> > > > > > > > > > >> >> >>> Graviton3
> > > > > > > > > > >> >> >>> > > (but
> > > > > > > > > > >> >> >>> > > > > > maybe
> > > > > > > > > > >> >> >>> > > > > > >>> > there
> > > > > > > > > > >> >> >>> > > > > > >>> > >> was
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> something
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> wrong with
> my
> > > > > > > > > implementation).
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Locking
> the
> > > > 16bit
> > > > > > > > > exception
> > > > > > > > > > >> >> >>> encoding for
> > > > > > > > > > >> >> >>> > > the
> > > > > > > > > > >> >> >>> > > > > > spec
> > > > > > > > > > >> >> >>> > > > > > >>> in
> > > > > > > > > > >> >> >>> > > > > > >>> > >> this
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> case.
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Awesome I
> > > think
> > > > we
> > > > > > > have
> > > > > > > > > > solved
> > > > > > > > > > >> >> for
> > > > > > > > > > >> >> >>> all
> > > > > > > > > > >> >> >>> > > open
> > > > > > > > > > >> >> >>> > > > > > >>> questions
> > > > > > > > > > >> >> >>> > > > > > >>> > >> minus
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> the
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> version byte
> > :).
> > > > > (will
> > > > > > > get
> > > > > > > > > > back
> > > > > > > > > > >> on
> > > > > > > > > > >> >> >>> this
> > > > > > > > > > >> >> >>> > > soon)
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 3) Micah
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - FastLanes
> :
> > > The
> > > > > > > current
> > > > > > > > > spec
> > > > > > > > > > >> does
> > > > > > > > > > >> >> >>> allow
> > > > > > > > > > >> >> >>> > > for
> > > > > > > > > > >> >> >>> > > > > > using
> > > > > > > > > > >> >> >>> > > > > > >>> > >> FastLane
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> with the
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> configurable
> > > enum
> > > > > > value
> > > > > > > > for
> > > > > > > > > > >> >> layout. We
> > > > > > > > > > >> >> >>> > > should
> > > > > > > > > > >> >> >>> > > > be
> > > > > > > > > > >> >> >>> > > > > > >>> able to
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > inject
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> any
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> layout
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> in the
> current
> > > > > design.
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Working on
> > resolving
> > > > all
> > > > > > > > > remaining
> > > > > > > > > > >> open
> > > > > > > > > > >> >> >>> > comments
> > > > > > > > > > >> >> >>> > > > on
> > > > > > > > > > >> >> >>> > > > > > the
> > > > > > > > > > >> >> >>> > > > > > >>> spec
> > > > > > > > > > >> >> >>> > > > > > >>> > >> this
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> week.
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Best
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Prateek
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> On Tue, Feb 10,
> > 2026
> > > > at
> > > > > > > > 3:37 AM
> > > > > > > > > > >> Steve
> > > > > > > > > > >> >> >>> > Loughran <
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > stevel@cloudera.com>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> wrote:
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> On Sun, 8 Feb
> > 2026
> > > at
> > > > > > > 18:12,
> > > > > > > > > > Micah
> > > > > > > > > > >> >> >>> Kornfield
> > > > > > > > > > >> >> >>> > <
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > emkornfield@gmail.com>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> wrote:
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> It looks like
> > the
> > > > > actual
> > > > > > > > issue
> > > > > > > > > > >> >> >>> described for
> > > > > > > > > > >> >> >>> > > ORC
> > > > > > > > > > >> >> >>> > > > > in
> > > > > > > > > > >> >> >>> > > > > > >>> the
> > > > > > > > > > >> >> >>> > > > > > >>> > >> paper
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> is that
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> it
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> has multiple
> > > > > > sub-encodings
> > > > > > > > in
> > > > > > > > > a
> > > > > > > > > > >> >> batch.
> > > > > > > > > > >> >> >>> This
> > > > > > > > > > >> >> >>> > > is
> > > > > > > > > > >> >> >>> > > > > > >>> different
> > > > > > > > > > >> >> >>> > > > > > >>> > >> then
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> the
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> design
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> proposed here
> > > where
> > > > > > there
> > > > > > > is
> > > > > > > > > > still
> > > > > > > > > > >> >> fixed
> > > > > > > > > > >> >> >>> > > > encoding
> > > > > > > > > > >> >> >>> > > > > > per
> > > > > > > > > > >> >> >>> > > > > > >>> page
> > > > > > > > > > >> >> >>> > > > > > >>> > >> in
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> parquet.
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> Given
> reasonably
> > > > sized
> > > > > > > > pages I
> > > > > > > > > > >> don't
> > > > > > > > > > >> >> >>> think
> > > > > > > > > > >> >> >>> > > > branch
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> misprediction
> > should
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> be a
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> big issue for
> > new
> > > > > > > encodings.
> > > > > > > > > I
> > > > > > > > > > >> agree
> > > > > > > > > > >> >> >>> that
> > > > > > > > > > >> >> >>> > we
> > > > > > > > > > >> >> >>> > > > > should
> > > > > > > > > > >> >> >>> > > > > > >>> be
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> conservative in
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> general for
> > adding
> > > > new
> > > > > > > > > > encodings.
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> +1
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > > > > > > >> >> >>> > > > > > >>> > >> > >
> > > > > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > > > > > >> >> >>> > > > > > >>> > >
> > > > > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > > > > >> >> >>> > > > > > >>>
> > > > > > > > > > >> >> >>> > > > > > >>
> > > > > > > > > > >> >> >>> > > > > >
> > > > > > > > > > >> >> >>> > > > >
> > > > > > > > > > >> >> >>> > > >
> > > > > > > > > > >> >> >>> > >
> > > > > > > > > > >> >> >>> >
> > > > > > > > > > >> >> >>>
> > > > > > > > > > >> >> >>
> > > > > > > > > > >> >>
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Gunnar Morling via dev <de...@parquet.apache.org>.
Hey all,
We've added ALP encoding and decoding support to Hardwood too [1], [2].
Interop with parquet-java's implementation looks good: Hardwood decodes
pages written by parquet-java's ALP writer bit-exactly and vice versa.
Perhaps interesting: on the encoder side, we build the preset of a column
chunk from eight sample vectors distributed across the entire chunk (easy
in our case, as Hardwood's writer buffers whole chunks), whereas, IIUC,
parquet-java only samples within the first ~110K values of a chunk.
Depending on the data distribution, this can yield a more efficient
encoding, for instance when a high-cardinality column changes character
midway, e.g. moving from two to four decimal places (about 2.8× smaller in
a synthetic test with 1M doubles).
Best,
--Gunnar
[1] https://github.com/hardwood-hq/hardwood/pull/1496
[2] https://github.com/hardwood-hq/hardwood/pull/1769
On Thu, 12 Nov 2026 at 07:01, PRATEEK GAUR <pr...@gmail.com> wrote:
> Thanks everyone!
>
> On Mon, Nov 9, 2026 at 11:58 AM Vinoo Ganesh <vi...@gmail.com>
> wrote:
>
> > Thanks everyone!
> >
> > Thanks,
> > Vinoo Ganesh | vinoo.ganesh@gmail.com
> >
> > <Vi...@gmail.com>
> >
> >
> > On Mon, Nov 9, 2026 at 2:49 PM Andrew Lamb <an...@gmail.com>
> wrote:
> >
> > > The Java implementation was just merged[1]!
> > >
> > > Thank you Gang, Vinoo, Prateek and everyone who helped push that one
> > along
> > >
> > > Andrew
> > >
> > > [1]:
> https://github.com/apache/parquet-java/pull/3397#event-35545451335
> > >
> > > On Tue, Oct 13, 2026 at 12:52 PM PRATEEK GAUR <pr...@gmail.com>
> > wrote:
> > >
> > > > Hi team,
> > > >
> > > > Quick update re: the C++ implementation.
> > > > Thanks to all the reviewers, we merged the C++ implementation today.
> > > > There are a few follow-up items that I'll address in a subsequent PR.
> > > >
> > > > Thanks
> > > > Prateek
> > > >
> > > > On Sun, Sep 13, 2026 at 6:54 PM Vinoo Ganesh <vinoo.ganesh@gmail.com
> >
> > > > wrote:
> > > >
> > > > > Quick update re: the Java impl. Thanks for the feedback, on
> > > > > https://github.com/apache/parquet-java/pull/3397, Russel! I pulled
> > > > > https://github.com/apache/parquet-java/pull/3791 into its own PR
> > > (thanks
> > > > > for the quick approval too, Gang). Assuming no objections, I think
> we
> > > can
> > > > > merge #3791, and then go ahead of #3397.
> > > > >
> > > > > Thanks,
> > > > > Vinoo Ganesh | vinoo.ganesh@gmail.com
> > > > >
> > > > > <Vi...@gmail.com>
> > > > >
> > > > >
> > > > > On Tue, Sep 8, 2026 at 10:54 AM PRATEEK GAUR <pr...@gmail.com>
> > > wrote:
> > > > >
> > > > > > [Rust] Awesome thanks Andrew, Kostas and Devin.
> > > > > > [Java] Thanks Gang for the feedback on Java PR.
> > > > > > [C++] Thanks Gang for another round of feedback on the C++ one.
> > > > > > [Go] Arnav and I will be working on the golang impl this week.
> > > > > >
> > > > > > This is great progress all around.
> > > > > >
> > > > > > On Sat, Sep 5, 2026 at 1:26 PM Vinoo Ganesh <
> > vinoo.ganesh@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks all - I'm addressing Gang's comments on
> > > > > > > https://github.com/apache/parquet-java/pull/3397 now, so
> > hopefully
> > > > > well
> > > > > > > also be in a good state for merging the java implementation
> soon.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Vinoo Ganesh | vinoo.ganesh@gmail.com
> > > > > > >
> > > > > > > <Vi...@gmail.com>
> > > > > > >
> > > > > > >
> > > > > > > On Sat, Sep 5, 2026 at 8:56 AM Andrew Lamb <
> > andrewlamb11@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > We just merged the Rust implementation[1] from Kosta and
> Devan.
> > > > With
> > > > > > the
> > > > > > > > imminent release of Parquet Format 2.14.0 I think we'll be
> well
> > > on
> > > > > our
> > > > > > > way.
> > > > > > > >
> > > > > > > > Thanks again for pushing this along,
> > > > > > > > Andrew
> > > > > > > >
> > > > > > > > [1]: https://github.com/apache/arrow-rs/pull/9372
> > > > > > > >
> > > > > > > > On Fri, Sep 4, 2026 at 5:42 PM PRATEEK GAUR <
> > prtkgaur@gmail.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi team,
> > > > > > > > >
> > > > > > > > > Thanks Micah for multiple rounds of review and thanks Gang
> > for
> > > > > > taking a
> > > > > > > > > look at the PR and giving feedback. I've addressed all
> > comments
> > > > in
> > > > > > the
> > > > > > > > PR.
> > > > > > > > > Looking forward to the next round of feedback (or
> approval).
> > > > Would
> > > > > > like
> > > > > > > > to
> > > > > > > > > get it in soon.
> > > > > > > > >
> > > > > > > > > https://github.com/apache/arrow/pull/48345/changes
> > > > > > > > >
> > > > > > > > > With this done and after talking to Arnav, I plan to spend
> > some
> > > > > time
> > > > > > on
> > > > > > > > the
> > > > > > > > > ALP implementation in the arrow-go repository helping
> spread
> > > the
> > > > > > > support
> > > > > > > > to
> > > > > > > > > more languages.
> > > > > > > > >
> > > > > > > > > Best
> > > > > > > > > Prateek
> > > > > > > > >
> > > > > > > > > On Sun, Aug 2, 2026 at 10:42 PM PRATEEK GAUR <
> > > prtkgaur@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Thanks Divjot for bringing it up.
> > > > > > > > > >
> > > > > > > > > > I was under the same impression as Russell and Andrew
> that
> > > we'd
> > > > > > have
> > > > > > > it
> > > > > > > > > > flag gated.
> > > > > > > > > > With the new versioning proposal 'preview' might be a
> good
> > > tag
> > > > > for
> > > > > > > it.
> > > > > > > > > >
> > > > > > > > > > Best
> > > > > > > > > > Prateek
> > > > > > > > > >
> > > > > > > > > > On Sat, Aug 1, 2026 at 12:39 AM Andrew Lamb <
> > > > > > andrewlamb11@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> Div: Here is a proposal to mark ALP in Preview: [1]
> > (using a
> > > > > > comment
> > > > > > > > in
> > > > > > > > > >> the
> > > > > > > > > >> spec section).
> > > > > > > > > >>
> > > > > > > > > >> Russel: That is my understanding as well.
> > > > > > > > > >>
> > > > > > > > > >> Hopefully we are able to formalize the versioning
> process
> > > > > shortly
> > > > > > so
> > > > > > > > we
> > > > > > > > > >> can
> > > > > > > > > >> formalize the notion of "preview"
> > > > > > > > > >>
> > > > > > > > > >> Andrew
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >> [1]: https://github.com/apache/parquet-format/pull/604
> > > > > > > > > >>
> > > > > > > > > >> On Fri, Jul 31, 2026 at 11:15 AM Russell Spitzer <
> > > > > > > > > >> russell.spitzer@gmail.com>
> > > > > > > > > >> wrote:
> > > > > > > > > >>
> > > > > > > > > >> > I thought we had agreed it would be merged, and
> features
> > > > like
> > > > > > this
> > > > > > > > > would
> > > > > > > > > >> > be feature flag gated. So a V-Old writer could always
> > opt
> > > in
> > > > > to
> > > > > > > > using
> > > > > > > > > >> this
> > > > > > > > > >> > encoding but that must be a conscious effort if you
> are
> > > not
> > > > > > using
> > > > > > > > > VNext
> > > > > > > > > >> of
> > > > > > > > > >> > Parquet. If you are on V-Next the writer can choose to
> > use
> > > > ALP
> > > > > > > > > whenever
> > > > > > > > > >> it
> > > > > > > > > >> > likes without user intervention.
> > > > > > > > > >> >
> > > > > > > > > >> > On Fri, Jul 31, 2026 at 2:20 AM Divjot Arora via dev <
> > > > > > > > > >> > dev@parquet.apache.org> wrote:
> > > > > > > > > >> >
> > > > > > > > > >> >> Yes, that’s my understanding as well. However, as you
> > > > noted,
> > > > > > the
> > > > > > > > > >> >> versioning
> > > > > > > > > >> >> discussion is still in early stages and hasn’t
> > prescribed
> > > > > > > concrete
> > > > > > > > > >> >> mechanisms yet. If ALP goes into parquet-format
> 2.14.0,
> > > > > there’s
> > > > > > > no
> > > > > > > > > >> >> indication that it’s in preview or that writers
> > shouldn’t
> > > > use
> > > > > > it
> > > > > > > by
> > > > > > > > > >> >> default
> > > > > > > > > >> >> yet. Given that we’ve held off on merging other
> > > > incompatible
> > > > > > > > changes
> > > > > > > > > >> (e.g
> > > > > > > > > >> >> removal of path_in_schema) until the versioning
> > > discussion
> > > > > > > closes,
> > > > > > > > my
> > > > > > > > > >> >> preference would be to leave ALP un-merged for the
> time
> > > > being
> > > > > > as
> > > > > > > > > well.
> > > > > > > > > >> >>
> > > > > > > > > >> >> — Div
> > > > > > > > > >> >>
> > > > > > > > > >> >> On Fri, Jul 31, 2026 at 08:59 Andrew Lamb <
> > > > > > > andrewlamb11@gmail.com>
> > > > > > > > > >> wrote:
> > > > > > > > > >> >>
> > > > > > > > > >> >> > After re-reading Julien's Versioning Proposal, my
> > > > > > understanding
> > > > > > > > is
> > > > > > > > > >> that
> > > > > > > > > >> >> if
> > > > > > > > > >> >> > we adopt that system, we will somehow mark ALP in
> > > > "Preview"
> > > > > > in
> > > > > > > > > >> >> > parquet-format for release Version 3.
> > > > > > > > > >> >> >
> > > > > > > > > >> >> > The versioning document doesn't seem to proscribe
> the
> > > > > > > particular
> > > > > > > > > >> method
> > > > > > > > > >> >> > for annotating "preview" features -- I think
> clearly
> > > > > marking
> > > > > > > them
> > > > > > > > > as
> > > > > > > > > >> >> such
> > > > > > > > > >> >> > in parquet.thrift as well as reflecting that on the
> > > > > > > > > >> >> >
> > https://parquet.apache.org/docs/file-format/versions/
> > > > page
> > > > > > > > somehow
> > > > > > > > > >> >> would
> > > > > > > > > >> >> > be adequate.
> > > > > > > > > >> >> >
> > > > > > > > > >> >> > Andrew
> > > > > > > > > >> >> >
> > > > > > > > > >> >> > On Thu, Jul 30, 2026 at 11:50 AM Andrew Lamb <
> > > > > > > > > andrewlamb11@gmail.com
> > > > > > > > > >> >
> > > > > > > > > >> >> > wrote:
> > > > > > > > > >> >> >
> > > > > > > > > >> >> >> In my mind ALP is in the same position as other
> > > > encodings
> > > > > > that
> > > > > > > > > were
> > > > > > > > > >> >> added
> > > > > > > > > >> >> >> recently[1] (such as BYTE_STREAM_SPLIT for non
> > > floating
> > > > > > point
> > > > > > > > > types)
> > > > > > > > > >> >> >>
> > > > > > > > > >> >> >> That is to say it will be available in the latest
> > open
> > > > > > source
> > > > > > > > > >> writers
> > > > > > > > > >> >> and
> > > > > > > > > >> >> >> readers, but realistically not widely used in the
> > > > > ecosystem
> > > > > > > > where
> > > > > > > > > >> >> >> compatibility is a concern (a large part)
> > > > > > > > > >> >> >>
> > > > > > > > > >> >> >> I view Julian's work[2][3] to gather consensus on
> > how
> > > to
> > > > > > name
> > > > > > > > > >> >> >> combinations of features (e.g. "versions") as the
> > > > > mechanism
> > > > > > by
> > > > > > > > > >> which we
> > > > > > > > > >> >> >> will accelerate the adoption of this change (along
> > > with
> > > > > > other
> > > > > > > > > >> forward
> > > > > > > > > >> >> >> incompatible changes) across the ecosystem.
> > > > > > > > > >> >> >>
> > > > > > > > > >> >> >> > Will we exclude this change when cutting
> > > > parquet-format
> > > > > > > > v2.14.0?
> > > > > > > > > >> >> >>
> > > > > > > > > >> >> >> No I don't think so -- I would expect it to go out
> > in
> > > > the
> > > > > > next
> > > > > > > > > >> >> >> parquet-format version as do other changes in the
> > > spec.
> > > > > > > > > >> >> >>
> > > > > > > > > >> >> >> Andrew
> > > > > > > > > >> >> >>
> > > > > > > > > >> >> >> [1]:
> > > > > https://parquet.apache.org/docs/file-format/versions/
> > > > > > > > > >> >> >> [2]:
> > > > > > > > > >>
> > > > > https://lists.apache.org/thread/xmnj8h0h8ozmrgox5tydhhs7yz7q2hss
> > > > > > > > > >> >> >> [3]:
> > > > > > > > > >>
> > > > > https://lists.apache.org/thread/qlf8lg90gqq41lllyy9mk2f0skqvftqj
> > > > > > > > > >> >> >>
> > > > > > > > > >> >> >>
> > > > > > > > > >> >> >> On Thu, Jul 30, 2026 at 3:51 AM Divjot Arora via
> > dev <
> > > > > > > > > >> >> >> dev@parquet.apache.org> wrote:
> > > > > > > > > >> >> >>
> > > > > > > > > >> >> >>> Hi Andrew,
> > > > > > > > > >> >> >>>
> > > > > > > > > >> >> >>> How does this interact with the ongoing
> versioning
> > > > > > > discussion?
> > > > > > > > > >> Given
> > > > > > > > > >> >> that
> > > > > > > > > >> >> >>> new encodings are not forward compatible, I would
> > > > assume
> > > > > > ALP
> > > > > > > is
> > > > > > > > > >> >> included
> > > > > > > > > >> >> >>> in
> > > > > > > > > >> >> >>> Parquet V<next> (V3?), but it's not clear to me
> how
> > > > that
> > > > > > > works
> > > > > > > > if
> > > > > > > > > >> it's
> > > > > > > > > >> >> >>> merged to the parquet-format master branch now.
> > Will
> > > we
> > > > > > > exclude
> > > > > > > > > >> this
> > > > > > > > > >> >> >>> change
> > > > > > > > > >> >> >>> when cutting parquet-format v2.14.0?
> > > > > > > > > >> >> >>>
> > > > > > > > > >> >> >>> -- Div
> > > > > > > > > >> >> >>>
> > > > > > > > > >> >> >>> On Thu, Jul 30, 2026 at 8:51 AM Andrew Lamb <
> > > > > > > > > >> andrewlamb11@gmail.com>
> > > > > > > > > >> >> >>> wrote:
> > > > > > > > > >> >> >>>
> > > > > > > > > >> >> >>> > Hello,
> > > > > > > > > >> >> >>> >
> > > > > > > > > >> >> >>> > As an update, thanks to the epic work from many
> > > > > > > contributors,
> > > > > > > > > >> >> >>> especially
> > > > > > > > > >> >> >>> > Prateek, I just merged the parquet-format
> change
> > > for
> > > > > > > ALP[1].
> > > > > > > > We
> > > > > > > > > >> >> plan to
> > > > > > > > > >> >> >>> > make additional editorial The remaining items I
> > > know
> > > > of
> > > > > > are
> > > > > > > > > >> merging
> > > > > > > > > >> >> the
> > > > > > > > > >> >> >>> > examples from parquet-testing[2], and then of
> > > course
> > > > > the
> > > > > > > > > various
> > > > > > > > > >> >> >>> > implementations
> > > > > > > > > >> >> >>> >
> > > > > > > > > >> >> >>> > Thanks again everyone. This is a great step
> > forward
> > > > > > > > > >> >> >>> > Andrew
> > > > > > > > > >> >> >>> >
> > > > > > > > > >> >> >>> > [1]:
> > > > https://github.com/apache/parquet-format/pull/557
> > > > > > > > > >> >> >>> > [2]:
> > > > > https://github.com/apache/parquet-testing/pull/100
> > > > > > > > > >> >> >>> >
> > > > > > > > > >> >> >>> > On Tue, Jul 21, 2026 at 4:39 PM Julien Le Dem <
> > > > > > > > > julien@apache.org
> > > > > > > > > >> >
> > > > > > > > > >> >> >>> wrote:
> > > > > > > > > >> >> >>> >
> > > > > > > > > >> >> >>> > > It was also in my spam folder... :(
> > > > > > > > > >> >> >>> > > I've seen a bunch of apache list emails go to
> > my
> > > > spam
> > > > > > > > > recently.
> > > > > > > > > >> >> >>> > >
> > > > > > > > > >> >> >>> > > On Tue, Jul 21, 2026 at 10:58 AM Russell
> > Spitzer
> > > <
> > > > > > > > > >> >> >>> > > russell.spitzer@gmail.com>
> > > > > > > > > >> >> >>> > > wrote:
> > > > > > > > > >> >> >>> > >
> > > > > > > > > >> >> >>> > > > Yes, my hope was that my vote would kick it
> > out
> > > > of
> > > > > > spam
> > > > > > > > but
> > > > > > > > > >> we
> > > > > > > > > >> >> >>> > definitely
> > > > > > > > > >> >> >>> > > > had two other votes that also went to spam
> > for
> > > > me.
> > > > > > > > > >> >> >>> > > >
> > > > > > > > > >> >> >>> > > > For anyone looking the thread is here
> > > > > > > > > >> >> >>> > > >
> > > > > > > > > >> >>
> > > > > > https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt
> > > > > > > > > >> >> >>> > > >
> > > > > > > > > >> >> >>> > > > On Tue, Jul 21, 2026 at 10:57 AM Micah
> > > Kornfield
> > > > <
> > > > > > > > > >> >> >>> > emkornfield@gmail.com>
> > > > > > > > > >> >> >>> > > > wrote:
> > > > > > > > > >> >> >>> > > >
> > > > > > > > > >> >> >>> > > > > I've seen other's votes not go to spam,
> so
> > > > > > hopefully
> > > > > > > > > people
> > > > > > > > > >> >> will
> > > > > > > > > >> >> >>> see
> > > > > > > > > >> >> >>> > it
> > > > > > > > > >> >> >>> > > > in
> > > > > > > > > >> >> >>> > > > > there inboxes (or see this thread and
> look
> > in
> > > > > there
> > > > > > > > > spam).
> > > > > > > > > >> >> >>> > > > >
> > > > > > > > > >> >> >>> > > > > On Tue, Jul 21, 2026 at 10:52 AM PRATEEK
> > > GAUR <
> > > > > > > > > >> >> >>> prtkgaur@gmail.com>
> > > > > > > > > >> >> >>> > > > wrote:
> > > > > > > > > >> >> >>> > > > >
> > > > > > > > > >> >> >>> > > > > > Okay,
> > > > > > > > > >> >> >>> > > > > >
> > > > > > > > > >> >> >>> > > > > > I sent out the voting email yesterday
> and
> > > so
> > > > > far
> > > > > > > I've
> > > > > > > > > >> heard
> > > > > > > > > >> >> >>> from 3
> > > > > > > > > >> >> >>> > > > people
> > > > > > > > > >> >> >>> > > > > > that it ended up in their "spam
> folder".
> > > Dev
> > > > > VPN
> > > > > > to
> > > > > > > > be
> > > > > > > > > >> >> blamed
> > > > > > > > > >> >> >>> for
> > > > > > > > > >> >> >>> > > it?.
> > > > > > > > > >> >> >>> > > > > >
> > > > > > > > > >> >> >>> > > > > > What's the protocol here?
> > > > > > > > > >> >> >>> > > > > > I let people hunt for the email in
> their
> > > spam
> > > > > > > folder
> > > > > > > > or
> > > > > > > > > >> send
> > > > > > > > > >> >> >>> out
> > > > > > > > > >> >> >>> > > > another
> > > > > > > > > >> >> >>> > > > > > one?
> > > > > > > > > >> >> >>> > > > > >
> > > > > > > > > >> >> >>> > > > > > Best
> > > > > > > > > >> >> >>> > > > > > Prateek
> > > > > > > > > >> >> >>> > > > > >
> > > > > > > > > >> >> >>> > > > > > On Sun, Jul 19, 2026 at 10:47 PM
> PRATEEK
> > > > GAUR <
> > > > > > > > > >> >> >>> prtkgaur@gmail.com>
> > > > > > > > > >> >> >>> > > > > wrote:
> > > > > > > > > >> >> >>> > > > > >
> > > > > > > > > >> >> >>> > > > > > > Hi team,
> > > > > > > > > >> >> >>> > > > > > >
> > > > > > > > > >> >> >>> > > > > > > Vinoo has added the tests which were
> > > > > requested
> > > > > > in
> > > > > > > > the
> > > > > > > > > >> last
> > > > > > > > > >> >> >>> round
> > > > > > > > > >> >> >>> > of
> > > > > > > > > >> >> >>> > > > > > review.
> > > > > > > > > >> >> >>> > > > > > > In his own words.
> > > > > > > > > >> >> >>> > > > > > >
> > > > > > > > > >> >> >>> > > > > > > "
> > > > > > > > > >> >> >>> > > > > > > The extreme value tests Micah wanted
> > are
> > > > done
> > > > > > and
> > > > > > > > > >> pushed.
> > > > > > > > > >> >> I
> > > > > > > > > >> >> >>> added
> > > > > > > > > >> >> >>> > > > > > coverage
> > > > > > > > > >> >> >>> > > > > > > for values that need the full FOR bit
> > > width
> > > > > > after
> > > > > > > > the
> > > > > > > > > >> >> frame
> > > > > > > > > >> >> >>> of
> > > > > > > > > >> >> >>> > > > > reference
> > > > > > > > > >> >> >>> > > > > > is
> > > > > > > > > >> >> >>> > > > > > > applied: 64-bit (the case where the
> > > signed
> > > > > max
> > > > > > > > minus
> > > > > > > > > >> min
> > > > > > > > > >> >> >>> > > subtraction
> > > > > > > > > >> >> >>> > > > > > > overflows), 63-bit (the non-overflow
> > > case),
> > > > > and
> > > > > > > > > 32-bit
> > > > > > > > > >> for
> > > > > > > > > >> >> >>> > floats.
> > > > > > > > > >> >> >>> > > > All
> > > > > > > > > >> >> >>> > > > > > > lossless with zero exceptions. I also
> > > > > verified
> > > > > > > them
> > > > > > > > > >> >> through
> > > > > > > > > >> >> >>> Arrow
> > > > > > > > > >> >> >>> > > C++
> > > > > > > > > >> >> >>> > > > > ALP
> > > > > > > > > >> >> >>> > > > > > > reader (the local build), so the
> > > > Java-written
> > > > > > > > extreme
> > > > > > > > > >> >> columns
> > > > > > > > > >> >> >>> > > decode
> > > > > > > > > >> >> >>> > > > > > > correctly in C++, bit-exact against
> the
> > > > > > expected
> > > > > > > > > >> values."
> > > > > > > > > >> >> >>> > > > > > >
> > > > > > > > > >> >> >>> > > > > > > State of the PR (#3397):
> > > > > > > > > >> >> >>> > > > > > >
> > > > > > > > > >> >> >>> > > > > > > - Merges cleanly with master (I
> > pulled
> > > > > > master
> > > > > > > in
> > > > > > > > > and
> > > > > > > > > >> >> >>> resolved
> > > > > > > > > >> >> >>> > > the
> > > > > > > > > >> >> >>> > > > > > > conflicts).
> > > > > > > > > >> >> >>> > > > > > > - 41 of 44 review threads
> resolved.
> > > The
> > > > 3
> > > > > > > > > remaining
> > > > > > > > > >> are
> > > > > > > > > >> >> >>> > > > intentional:
> > > > > > > > > >> >> >>> > > > > > > one deferred perf nit, one that
> > stays
> > > > open
> > > > > > > until
> > > > > > > > > >> >> >>> > parquet-format
> > > > > > > > > >> >> >>> > > > > ships
> > > > > > > > > >> >> >>> > > > > > ALP,
> > > > > > > > > >> >> >>> > > > > > >
> > > > > > > > > >> >> >>> > > > > > > "
> > > > > > > > > >> >> >>> > > > > > >
> > > > > > > > > >> >> >>> > > > > > > With this both the
> > > > > > > > > >> >> >>> > > > > > > 1) C++ PR :
> > > > > > > > > >> >> >>>
> https://github.com/apache/arrow/pull/48345/changes
> > > > > > > > > >> >> >>> > > > > > > 2) Java PR :
> > > > > > > > > >> >> >>> https://github.com/apache/parquet-java/pull/3397
> > > > > > > > > >> >> >>> > > > > > >
> > > > > > > > > >> >> >>> > > > > > > Are cross language tested and range
> of
> > > test
> > > > > > > > > scenarios.
> > > > > > > > > >> >> >>> > > > > > >
> > > > > > > > > >> >> >>> > > > > > > Best
> > > > > > > > > >> >> >>> > > > > > > Prateek
> > > > > > > > > >> >> >>> > > > > > >
> > > > > > > > > >> >> >>> > > > > > >
> > > > > > > > > >> >> >>> > > > > > > On Mon, Jul 13, 2026 at 8:38 AM
> PRATEEK
> > > > GAUR
> > > > > <
> > > > > > > > > >> >> >>> prtkgaur@gmail.com
> > > > > > > > > >> >> >>> > >
> > > > > > > > > >> >> >>> > > > > wrote:
> > > > > > > > > >> >> >>> > > > > > >
> > > > > > > > > >> >> >>> > > > > > >> Thanks Matt,
> > > > > > > > > >> >> >>> > > > > > >>
> > > > > > > > > >> >> >>> > > > > > >> I'll be on it this week.
> > > > > > > > > >> >> >>> > > > > > >>
> > > > > > > > > >> >> >>> > > > > > >> Best
> > > > > > > > > >> >> >>> > > > > > >> Prateek
> > > > > > > > > >> >> >>> > > > > > >>
> > > > > > > > > >> >> >>> > > > > > >> On Mon, Jul 13, 2026 at 8:36 AM Matt
> > > > Topol <
> > > > > > > > > >> >> >>> > > zotthewizard@gmail.com>
> > > > > > > > > >> >> >>> > > > > > >> wrote:
> > > > > > > > > >> >> >>> > > > > > >>
> > > > > > > > > >> >> >>> > > > > > >>> There is also the Go implementation
> > > > > submitted
> > > > > > > by
> > > > > > > > > >> Arnav (
> > > > > > > > > >> >> >>> > > > > > >>>
> > > > > https://github.com/apache/arrow-go/pull/704)
> > > > > > > > which
> > > > > > > > > >> is
> > > > > > > > > >> >> >>> waiting
> > > > > > > > > >> >> >>> > > for
> > > > > > > > > >> >> >>> > > > > > >>> updates.
> > > > > > > > > >> >> >>> > > > > > >>>
> > > > > > > > > >> >> >>> > > > > > >>> Just wanted to make sure it didn't
> > get
> > > > lost
> > > > > > > here.
> > > > > > > > > >> >> >>> > > > > > >>>
> > > > > > > > > >> >> >>> > > > > > >>> --Matt
> > > > > > > > > >> >> >>> > > > > > >>>
> > > > > > > > > >> >> >>> > > > > > >>> On Mon, Jul 13, 2026 at 11:30 AM
> > > PRATEEK
> > > > > > GAUR <
> > > > > > > > > >> >> >>> > > prtkgaur@gmail.com>
> > > > > > > > > >> >> >>> > > > > > >>> wrote:
> > > > > > > > > >> >> >>> > > > > > >>>
> > > > > > > > > >> >> >>> > > > > > >>> > Hi all,
> > > > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > > > >> >> >>> > > > > > >>> > I'd like to share a status update
> > on
> > > > the
> > > > > > ALP
> > > > > > > > > >> encoding
> > > > > > > > > >> >> >>> effort
> > > > > > > > > >> >> >>> > > and
> > > > > > > > > >> >> >>> > > > > get
> > > > > > > > > >> >> >>> > > > > > a
> > > > > > > > > >> >> >>> > > > > > >>> feel
> > > > > > > > > >> >> >>> > > > > > >>> > from the community before
> starting
> > a
> > > > > formal
> > > > > > > > vote.
> > > > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > > > >> >> >>> > > > > > >>> > We now have the specification
> plus
> > > > > > > > > implementations
> > > > > > > > > >> in
> > > > > > > > > >> >> >>> both
> > > > > > > > > >> >> >>> > > > > languages:
> > > > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > > > >> >> >>> > > > > > >>> > - Spec:
> > > > > > > apache/parquet-format#557 —
> > > > > > > > > >> GH-533
> > > > > > > > > >> >> >>> Add
> > > > > > > > > >> >> >>> > ALP
> > > > > > > > > >> >> >>> > > > > > encoding
> > > > > > > > > >> >> >>> > > > > > >>> > specification
> > > > > > > > > >> >> >>> > > > > > >>> > - C++ (Arrow):
> > > apache/arrow#48345
> > > > —
> > > > > > > > GH-48701
> > > > > > > > > >> >> >>> > > [C++][Parquet]
> > > > > > > > > >> >> >>> > > > > Add
> > > > > > > > > >> >> >>> > > > > > >>> ALPpd
> > > > > > > > > >> >> >>> > > > > > >>> > encoding
> > > > > > > > > >> >> >>> > > > > > >>> > - Java:
> > > > > > apache/parquet-java#3397
> > > > > > > —
> > > > > > > > > >> Parquet
> > > > > > > > > >> >> >>> Java
> > > > > > > > > >> >> >>> > ALP
> > > > > > > > > >> >> >>> > > > > > >>> > Implementation (by Vinoo)
> > > > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > > > >> >> >>> > > > > > >>> > Current state:
> > > > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > > > >> >> >>> > > > > > >>> > - The C++ implementation has
> > been
> > > > > > through
> > > > > > > > 3-4
> > > > > > > > > >> >> rounds
> > > > > > > > > >> >> >>> of
> > > > > > > > > >> >> >>> > > > review.
> > > > > > > > > >> >> >>> > > > > > >>> > - The Java implementation has
> > > been
> > > > > > > tracking
> > > > > > > > > >> well
> > > > > > > > > >> >> and
> > > > > > > > > >> >> >>> has
> > > > > > > > > >> >> >>> > > had
> > > > > > > > > >> >> >>> > > > > > >>> initial
> > > > > > > > > >> >> >>> > > > > > >>> > reviews from contributors,
> > with
> > > > no
> > > > > > > major
> > > > > > > > > >> >> remaining
> > > > > > > > > >> >> >>> > > issues.
> > > > > > > > > >> >> >>> > > > > > >>> > - Cross-language
> compatibility
> > > > tests
> > > > > > are
> > > > > > > > > >> passing:
> > > > > > > > > >> >> the
> > > > > > > > > >> >> >>> > Arrow
> > > > > > > > > >> >> >>> > > > C++
> > > > > > > > > >> >> >>> > > > > > >>> decoder
> > > > > > > > > >> >> >>> > > > > > >>> > reads Java-written data
> > > > bit-exactly
> > > > > > > > across
> > > > > > > > > >> >> ~1.56M
> > > > > > > > > >> >> >>> > values
> > > > > > > > > >> >> >>> > > > and
> > > > > > > > > >> >> >>> > > > > 18
> > > > > > > > > >> >> >>> > > > > > >>> > fixtures, covering V1 and
> V2
> > > > pages,
> > > > > > > > > multiple
> > > > > > > > > >> >> vector
> > > > > > > > > >> >> >>> > > sizes,
> > > > > > > > > >> >> >>> > > > > and
> > > > > > > > > >> >> >>> > > > > > >>> > several
> > > > > > > > > >> >> >>> > > > > > >>> > real datasets — zero
> > > mismatches.
> > > > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > > > >> >> >>> > > > > > >>> > As raised in the java-pr we will
> > be
> > > > > adding
> > > > > > > > > >> coverage
> > > > > > > > > >> >> for
> > > > > > > > > >> >> >>> > > extreme
> > > > > > > > > >> >> >>> > > > > > values
> > > > > > > > > >> >> >>> > > > > > >>> > (those requiring
> > > > > > > > > >> >> >>> > > > > > >>> > 63-64 bits after FOR is applied)
> > > > before
> > > > > we
> > > > > > > > > close a
> > > > > > > > > >> >> vote.
> > > > > > > > > >> >> >>> > We'll
> > > > > > > > > >> >> >>> > > > aim
> > > > > > > > > >> >> >>> > > > > > to
> > > > > > > > > >> >> >>> > > > > > >>> get
> > > > > > > > > >> >> >>> > > > > > >>> > that
> > > > > > > > > >> >> >>> > > > > > >>> > done in parallel.
> > > > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > > > >> >> >>> > > > > > >>> > Plan: unless there are
> objections,
> > I
> > > > > intend
> > > > > > > to
> > > > > > > > > >> start a
> > > > > > > > > >> >> >>> vote
> > > > > > > > > >> >> >>> > at
> > > > > > > > > >> >> >>> > > > the
> > > > > > > > > >> >> >>> > > > > > end
> > > > > > > > > >> >> >>> > > > > > >>> of
> > > > > > > > > >> >> >>> > > > > > >>> > this week/early
> > > > > > > > > >> >> >>> > > > > > >>> > next week.
> > > > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > > > >> >> >>> > > > > > >>> > Thanks
> > > > > > > > > >> >> >>> > > > > > >>> > Prateek
> > > > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > > > >> >> >>> > > > > > >>> > On Wed, Jul 1, 2026 at 8:50 AM
> > > PRATEEK
> > > > > > GAUR <
> > > > > > > > > >> >> >>> > > prtkgaur@gmail.com>
> > > > > > > > > >> >> >>> > > > > > >>> wrote:
> > > > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > > > >> >> >>> > > > > > >>> > > Hi Team,
> > > > > > > > > >> >> >>> > > > > > >>> > >
> > > > > > > > > >> >> >>> > > > > > >>> > > Just wanted to provide some
> > updates
> > > > on
> > > > > > ALP.
> > > > > > > > > >> >> >>> > > > > > >>> > > Micah and I have done 3-4
> rounds
> > of
> > > > > > review
> > > > > > > > for
> > > > > > > > > >> the
> > > > > > > > > >> >> c++
> > > > > > > > > >> >> >>> PR :
> > > > > > > > > >> >> >>> > > > > > >>> > >
> > > > > > https://github.com/apache/arrow/pull/48345
> > > > > > > > > >> >> >>> > > > > > >>> > >
> > > > > > > > > >> >> >>> > > > > > >>> > > For the Java implementation
> Vinoo
> > > has
> > > > > > been
> > > > > > > > > >> working
> > > > > > > > > >> >> on
> > > > > > > > > >> >> >>> > > following
> > > > > > > > > >> >> >>> > > > > PR
> > > > > > > > > >> >> >>> > > > > > :
> > > > > > > > > >> >> >>> > > > > > >>> > >
> > > > > > > > > https://github.com/apache/parquet-java/pull/3397
> > > > > > > > > >> >> >>> > > > > > >>> > >
> > > > > > > > > >> >> >>> > > > > > >>> > > Best
> > > > > > > > > >> >> >>> > > > > > >>> > > Prateek
> > > > > > > > > >> >> >>> > > > > > >>> > >
> > > > > > > > > >> >> >>> > > > > > >>> > > On Tue, May 5, 2026 at 1:50 PM
> > > Micah
> > > > > > > > Kornfield
> > > > > > > > > <
> > > > > > > > > >> >> >>> > > > > > >>> emkornfield@gmail.com>
> > > > > > > > > >> >> >>> > > > > > >>> > > wrote:
> > > > > > > > > >> >> >>> > > > > > >>> > >
> > > > > > > > > >> >> >>> > > > > > >>> > >> Hi Antoine,
> > > > > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > Apologies if the question
> was
> > > > > already
> > > > > > > > asked,
> > > > > > > > > >> but
> > > > > > > > > >> >> >>> should
> > > > > > > > > >> >> >>> > we
> > > > > > > > > >> >> >>> > > > > care
> > > > > > > > > >> >> >>> > > > > > >>> about
> > > > > > > > > >> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT +
> > ALP
> > > > be
> > > > > > more
> > > > > > > > > >> >> efficient
> > > > > > > > > >> >> >>> than
> > > > > > > > > >> >> >>> > > > > FLOAT16
> > > > > > > > > >> >> >>> > > > > > +
> > > > > > > > > >> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for
> > > > example?
> > > > > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > > > > >> >> >>> > > > > > >>> > >> It was. We thought we could
> > defer
> > > > it
> > > > > > for
> > > > > > > > the
> > > > > > > > > >> >> >>> following
> > > > > > > > > >> >> >>> > > > reasons:
> > > > > > > > > >> >> >>> > > > > > >>> > >> 1. It's not clear there are a
> > lot
> > > > of
> > > > > > easy
> > > > > > > > > >> >> reference
> > > > > > > > > >> >> >>> > > datasets
> > > > > > > > > >> >> >>> > > > to
> > > > > > > > > >> >> >>> > > > > > >>> test
> > > > > > > > > >> >> >>> > > > > > >>> > its
> > > > > > > > > >> >> >>> > > > > > >>> > >> effectiveness. It does look
> > like
> > > > > there
> > > > > > > > might
> > > > > > > > > be
> > > > > > > > > >> >> one
> > > > > > > > > >> >> >>> or
> > > > > > > > > >> >> >>> > two
> > > > > > > > > >> >> >>> > > on
> > > > > > > > > >> >> >>> > > > > > >>> > huggingface
> > > > > > > > > >> >> >>> > > > > > >>> > >> (e.g.
> > > > > > > > > >> >> https://huggingface.co/datasets/kikitora/curdie
> > > > > > > > > >> >> >>> ).
> > > > > > > > > >> >> >>> > > > > > >>> > >> 2. It seemed likely that
> float
> > 16
> > > > was
> > > > > > > more
> > > > > > > > > >> likely
> > > > > > > > > >> >> >>> used
> > > > > > > > > >> >> >>> > for
> > > > > > > > > >> >> >>> > > > > values
> > > > > > > > > >> >> >>> > > > > > >>> that
> > > > > > > > > >> >> >>> > > > > > >>> > >> were less likely to reduce to
> > > > decimal
> > > > > > > > values.
> > > > > > > > > >> >> >>> > > > > > >>> > >> 3. It could be added as an
> > > > extension
> > > > > > > later
> > > > > > > > if
> > > > > > > > > >> >> needed.
> > > > > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > > > > >> >> >>> > > > > > >>> > >> Cheers,
> > > > > > > > > >> >> >>> > > > > > >>> > >> Micah
> > > > > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > > > > >> >> >>> > > > > > >>> > >> On Tue, May 5, 2026 at 1:40 PM
> > > > Antoine
> > > > > > > > Pitrou
> > > > > > > > > <
> > > > > > > > > >> >> >>> > > > > antoine@python.org
> > > > > > > > > >> >> >>> > > > > > >
> > > > > > > > > >> >> >>> > > > > > >>> > wrote:
> > > > > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > > > >> >> >>> > > > > > >>> > >> > Hello,
> > > > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > > > >> >> >>> > > > > > >>> > >> > Apologies if the question
> was
> > > > > already
> > > > > > > > asked,
> > > > > > > > > >> but
> > > > > > > > > >> >> >>> should
> > > > > > > > > >> >> >>> > we
> > > > > > > > > >> >> >>> > > > > care
> > > > > > > > > >> >> >>> > > > > > >>> about
> > > > > > > > > >> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT +
> > ALP
> > > > be
> > > > > > more
> > > > > > > > > >> >> efficient
> > > > > > > > > >> >> >>> than
> > > > > > > > > >> >> >>> > > > > FLOAT16
> > > > > > > > > >> >> >>> > > > > > +
> > > > > > > > > >> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for
> > > > example?
> > > > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > > > >> >> >>> > > > > > >>> > >> > Regards
> > > > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > > > >> >> >>> > > > > > >>> > >> > Antoine.
> > > > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > > > >> >> >>> > > > > > >>> > >> > Le 30/04/2026 à 01:10,
> PRATEEK
> > > > GAUR
> > > > > a
> > > > > > > > écrit
> > > > > > > > > :
> > > > > > > > > >> >> >>> > > > > > >>> > >> > > Thanks Andrew and Micah
> for
> > > > review
> > > > > > > > > feedback
> > > > > > > > > >> on
> > > > > > > > > >> >> >>> the two
> > > > > > > > > >> >> >>> > > > PR's
> > > > > > > > > >> >> >>> > > > > > >>> > >> > > 1) (c++ arrow repo)
> > > > > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > > > > >> https://github.com/apache/arrow/pull/48345/changes
> > > > > > > > > >> >> >>> > > > > > >>> > >> > > 2) (parquet-format repo)
> > > > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > > > >> >> https://github.com/apache/parquet-format/pull/557
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >
> > > > > > > > > >> >> >>> > > > > > >>> > >> > > I have addressed all
> > (unless I
> > > > > > missed
> > > > > > > > > >> >> something)
> > > > > > > > > >> >> >>> > > comments
> > > > > > > > > >> >> >>> > > > on
> > > > > > > > > >> >> >>> > > > > > >>> the two
> > > > > > > > > >> >> >>> > > > > > >>> > >> > PR's.
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >
> > > > > > > > > >> >> >>> > > > > > >>> > >> > > Best
> > > > > > > > > >> >> >>> > > > > > >>> > >> > > Prateek
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >
> > > > > > > > > >> >> >>> > > > > > >>> > >> > > On Sat, Apr 25, 2026 at
> > > 1:08 PM
> > > > > > > PRATEEK
> > > > > > > > > >> GAUR <
> > > > > > > > > >> >> >>> > > > > > >>> prtkgaur@gmail.com>
> > > > > > > > > >> >> >>> > > > > > >>> > >> wrote:
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >> Thanks Andrew and Micah.
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >> `fair amount of feedback
> on
> > > at
> > > > > > least
> > > > > > > > the
> > > > > > > > > >> >> >>> > > implementations`
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >> For the c++ I have
> already
> > > > > started
> > > > > > > > > >> addressing
> > > > > > > > > >> >> the
> > > > > > > > > >> >> >>> > > > > feedback, I
> > > > > > > > > >> >> >>> > > > > > >>> > should
> > > > > > > > > >> >> >>> > > > > > >>> > >> be
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >> done with that
> > > Monday/Tuesday.
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >> I think Vinoo too has
> been
> > > > making
> > > > > > > good
> > > > > > > > > >> >> progress
> > > > > > > > > >> >> >>> on
> > > > > > > > > >> >> >>> > the
> > > > > > > > > >> >> >>> > > > Java
> > > > > > > > > >> >> >>> > > > > > >>> > >> > implementation.
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >> Best
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >> Prateek
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >> On Sat, Apr 25, 2026 at
> > > > 12:55 PM
> > > > > > > Andrew
> > > > > > > > > >> Lamb <
> > > > > > > > > >> >> >>> > > > > > >>> > andrewlamb11@gmail.com
> > > > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >> wrote:
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>> Got it. Thank you for
> the
> > > > > > > > clarification
> > > > > > > > > >> -- I
> > > > > > > > > >> >> >>> will
> > > > > > > > > >> >> >>> > try
> > > > > > > > > >> >> >>> > > > and
> > > > > > > > > >> >> >>> > > > > > look
> > > > > > > > > >> >> >>> > > > > > >>> > into
> > > > > > > > > >> >> >>> > > > > > >>> > >> the
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>> spec and the Rust
> > > > > > implementation[1]
> > > > > > > in
> > > > > > > > > >> this
> > > > > > > > > >> >> next
> > > > > > > > > >> >> >>> > week
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>> [1]:
> > > > > > > > > >> >> >>> https://github.com/apache/arrow-rs/pull/9372
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>> On Sat, Apr 25, 2026 at
> > > > 12:01 PM
> > > > > > > Micah
> > > > > > > > > >> >> >>> Kornfield <
> > > > > > > > > >> >> >>> > > > > > >>> > >> > emkornfield@gmail.com>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>> wrote:
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>> Hi Andrew,
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>> I think there is a fair
> > > > amount
> > > > > of
> > > > > > > > > >> feedback
> > > > > > > > > >> >> on
> > > > > > > > > >> >> >>> at
> > > > > > > > > >> >> >>> > > least
> > > > > > > > > >> >> >>> > > > > the
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>> implementations,
> > typically
> > > I
> > > > > > think
> > > > > > > > > we've
> > > > > > > > > >> >> waited
> > > > > > > > > >> >> >>> > till
> > > > > > > > > >> >> >>> > > > they
> > > > > > > > > >> >> >>> > > > > > are
> > > > > > > > > >> >> >>> > > > > > >>> > >> close to
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>> mergeable before a
> final
> > > > vote.
> > > > > > > > > >> Otherwise I
> > > > > > > > > >> >> >>> agree
> > > > > > > > > >> >> >>> > we
> > > > > > > > > >> >> >>> > > > are
> > > > > > > > > >> >> >>> > > > > > very
> > > > > > > > > >> >> >>> > > > > > >>> > >> close.
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>> -Micah
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>> On Saturday, April 25,
> > > 2026,
> > > > > > Andrew
> > > > > > > > > Lamb
> > > > > > > > > >> <
> > > > > > > > > >> >> >>> > > > > > >>> andrewlamb11@gmail.com
> > > > > > > > > >> >> >>> > > > > > >>> > >
> > > > > > > > > >> >> >>> > > > > > >>> > >> > wrote:
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> Thanks Prateek,
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> I think from this
> > content
> > > it
> > > > > > looks
> > > > > > > > to
> > > > > > > > > me
> > > > > > > > > >> >> like
> > > > > > > > > >> >> >>> we
> > > > > > > > > >> >> >>> > are
> > > > > > > > > >> >> >>> > > > > ready
> > > > > > > > > >> >> >>> > > > > > >>> to
> > > > > > > > > >> >> >>> > > > > > >>> > >> start a
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> vote to explicitly
> > accept
> > > > ALP
> > > > > > into
> > > > > > > > > >> Parquet
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> Does anyone know of a
> > > reason
> > > > > we
> > > > > > > > should
> > > > > > > > > >> >> >>> postpone it
> > > > > > > > > >> >> >>> > > for
> > > > > > > > > >> >> >>> > > > > > >>> longer?
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> Perhaps someone needs
> > some
> > > > > more
> > > > > > > time
> > > > > > > > > to
> > > > > > > > > >> >> >>> review?
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> Andrew
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> On Wed, Apr 22, 2026
> at
> > > > > 1:00 PM
> > > > > > > > > PRATEEK
> > > > > > > > > >> >> GAUR <
> > > > > > > > > >> >> >>> > > > > > >>> > prtkgaur@gmail.com>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> wrote:
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Hi team,
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Hope everyone is
> doing
> > > > well.
> > > > > I
> > > > > > > got
> > > > > > > > a
> > > > > > > > > >> >> chance
> > > > > > > > > >> >> >>> to
> > > > > > > > > >> >> >>> > work
> > > > > > > > > >> >> >>> > > > > > >>> through all
> > > > > > > > > >> >> >>> > > > > > >>> > >> the
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> remaining feedback
> and
> > > > update
> > > > > > the
> > > > > > > > > spec
> > > > > > > > > >> >> doc.
> > > > > > > > > >> >> >>> Here
> > > > > > > > > >> >> >>> > > are
> > > > > > > > > >> >> >>> > > > > the
> > > > > > > > > >> >> >>> > > > > > >>> new
> > > > > > > > > >> >> >>> > > > > > >>> > >> > artifacts
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 1) Spec document :
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > > > >> >> >>> > > > > > >>>
> > > > > > > > > >> >> >>> > > > > >
> > > > > > > > > >> >> >>> > > > >
> > > > > > > > > >> >> >>> > > >
> > > > > > > > > >> >> >>> > >
> > > > > > > > > >> >> >>> >
> > > > > > > > > >> >> >>>
> > > > > > > > > >> >>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-
> > > 3fPtD_aWt/edit
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 2) Spec document in
> > > parquet
> > > > > > > format
> > > > > > > > > >> repo :
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> >
> > https://github.com/apache/parquet-format/pull/557
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 3) Alp implementation
> > in
> > > > > arrow
> > > > > > > c++
> > > > > > > > > >> repo :
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> >
> > https://github.com/apache/arrow/pull/48345/changes
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 4) Alp implementation
> > in
> > > > > > > > parquet-java
> > > > > > > > > >> >> repo :
> > > > > > > > > >> >> >>> Work
> > > > > > > > > >> >> >>> > > for
> > > > > > > > > >> >> >>> > > > > > >>> Vinoo and
> > > > > > > > > >> >> >>> > > > > > >>> > >> > Julien
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> >
> https://github.com/apache/parquet-java/pull/3397
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 5) PR with test and
> > > > > > benchmarking
> > > > > > > > > >> >> artifacts in
> > > > > > > > > >> >> >>> > > > > > >>> parquet-testing
> > > > > > > > > >> >> >>> > > > > > >>> > >> repo :
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> >
> > https://github.com/apache/parquet-testing/pull/100
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> And
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> - Go : Arnav just
> > > > > submitted
> > > > > > > an
> > > > > > > > in
> > > > > > > > > >> >> >>> progress
> > > > > > > > > >> >> >>> > > > > > >>> implementation
> > > > > > > > > >> >> >>> > > > > > >>> > in
> > > > > > > > > >> >> >>> > > > > > >>> > >> Go.
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> https://github.com/apache/arrow-go/pull/704
> > > > > > > > > >> >> >>> > (I
> > > > > > > > > >> >> >>> > > > > > haven't
> > > > > > > > > >> >> >>> > > > > > >>> > >> started
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> looking at it
> yet)
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> - Rust : I
> remember
> > > > > Andrew
> > > > > > > > > >> mentioned
> > > > > > > > > >> >> that
> > > > > > > > > >> >> >>> > this
> > > > > > > > > >> >> >>> > > > work
> > > > > > > > > >> >> >>> > > > > > is
> > > > > > > > > >> >> >>> > > > > > >>> also
> > > > > > > > > >> >> >>> > > > > > >>> > >> in
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> progress (So 4
> > > > > languages!)
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> *Arrow C++
> > > implementation *
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> The PR is out and was
> > > also
> > > > > used
> > > > > > > by
> > > > > > > > > >> >> Antoine to
> > > > > > > > > >> >> >>> > > report
> > > > > > > > > >> >> >>> > > > > the
> > > > > > > > > >> >> >>> > > > > > >>> > numbers
> > > > > > > > > >> >> >>> > > > > > >>> > >> as
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> reported here. Micah
> > and
> > > > > > > Konstantin
> > > > > > > > > >> have
> > > > > > > > > >> >> >>> given 1
> > > > > > > > > >> >> >>> > > > round
> > > > > > > > > >> >> >>> > > > > of
> > > > > > > > > >> >> >>> > > > > > >>> > >> feedback
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> and I'm addressing
> them
> > > > > today.
> > > > > > > > Please
> > > > > > > > > >> note
> > > > > > > > > >> >> >>> that
> > > > > > > > > >> >> >>> > the
> > > > > > > > > >> >> >>> > > > > > default
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> optimization flag for
> > > > > compiling
> > > > > > > is
> > > > > > > > O2
> > > > > > > > > >> and
> > > > > > > > > >> >> not
> > > > > > > > > >> >> >>> > Q3. I
> > > > > > > > > >> >> >>> > > > got
> > > > > > > > > >> >> >>> > > > > > >>> around
> > > > > > > > > >> >> >>> > > > > > >>> > >> 70%
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> performance
> improvement
> > > in
> > > > > the
> > > > > > > > > decoding
> > > > > > > > > >> >> speed
> > > > > > > > > >> >> >>> > when
> > > > > > > > > >> >> >>> > > > > using
> > > > > > > > > >> >> >>> > > > > > >>> the O3
> > > > > > > > > >> >> >>> > > > > > >>> > >> > flag.
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> *Parqet-MR Java
> > > > > implementation
> > > > > > > > > (working
> > > > > > > > > >> >> with
> > > > > > > > > >> >> >>> > Vinoo
> > > > > > > > > >> >> >>> > > > and
> > > > > > > > > >> >> >>> > > > > > >>> Julien)
> > > > > > > > > >> >> >>> > > > > > >>> > >> and
> > > > > > > > > >> >> >>> > > > > > >>> > >> > **Cross
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Language testing*
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Let me know if
> you
> > > have
> > > > > any
> > > > > > > > > >> questions
> > > > > > > > > >> >> or
> > > > > > > > > >> >> >>> > > > feedback.
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Now pasting some
> > > > performance
> > > > > > > > numbers
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Table 1: C++ ALP
> > > Double
> > > > > > > Decode —
> > > > > > > > > >> >> Spotify
> > > > > > > > > >> >> >>> > Columns
> > > > > > > > > >> >> >>> > > > > > >>> (Graviton
> > > > > > > > > >> >> >>> > > > > > >>> > 3,
> > > > > > > > > >> >> >>> > > > > > >>> > >> ARM
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Neoverse V1)
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>>
> > > > > > > > > >> >> >>>
> > > > > > ┌──────────────────┬──────────────┬──────────────┬─────────┐
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ Column
> │
> > > > -O2
> > > > > > > (MB/s)
> > > > > > > > > │
> > > > > > > > > >> >> -O3
> > > > > > > > > >> >> >>> > > (MB/s) │
> > > > > > > > > >> >> >>> > > > > > >>> Speedup │
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>>
> > > > > > > > > >> >> >>>
> > > > > > ├──────────────────┼──────────────┼──────────────┼─────────┤
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ valence
> │
> > > > > > 3,155
> > > > > > > > > │
> > > > > > > > > >> >> >>> 5,523
> > > > > > > > > >> >> >>> > > │
> > > > > > > > > >> >> >>> > > > > > >>> 1.75x │
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ danceability
> │
> > > > > > 3,233
> > > > > > > > > │
> > > > > > > > > >> >> >>> 5,685
> > > > > > > > > >> >> >>> > > │
> > > > > > > > > >> >> >>> > > > > > >>> 1.76x │
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ energy
> │
> > > > > > 3,197
> > > > > > > > > │
> > > > > > > > > >> >> >>> 5,652
> > > > > > > > > >> >> >>> > > │
> > > > > > > > > >> >> >>> > > > > > >>> 1.77x │
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ loudness
> │
> > > > > > 3,186
> > > > > > > > > │
> > > > > > > > > >> >> >>> 5,473
> > > > > > > > > >> >> >>> > > │
> > > > > > > > > >> >> >>> > > > > > >>> 1.72x │
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>>
> > > > > > > > > >> >> >>>
> > > > > > └──────────────────┴──────────────┴──────────────┴─────────┘
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> On Wed, Feb 25, 2026
> at
> > > > > 9:49 AM
> > > > > > > > > PRATEEK
> > > > > > > > > >> >> GAUR
> > > > > > > > > >> >> >>> <
> > > > > > > > > >> >> >>> > > > > > >>> > prtkgaur@gmail.com
> > > > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> wrote:
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> @Micah Kornfield <
> > > > > > > > > >> emkornfield@gmail.com>
> > > > > > > > > >> >> :
> > > > > > > > > >> >> >>> Got
> > > > > > > > > >> >> >>> > > it.
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> @Andrew Lamb <
> > > > > > > > > andrewlamb11@gmail.com>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> Do you think it
> would
> > > be
> > > > > good
> > > > > > > to
> > > > > > > > > >> start
> > > > > > > > > >> >> >>> moving
> > > > > > > > > >> >> >>> > the
> > > > > > > > > >> >> >>> > > > > spec
> > > > > > > > > >> >> >>> > > > > > >>> > >> development
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> into
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> markdown format, in
> > > > > > preparation
> > > > > > > > for
> > > > > > > > > >> >> >>> finalizing
> > > > > > > > > >> >> >>> > > it?
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> Yes I'll update the
> > > > numbers
> > > > > > for
> > > > > > > > some
> > > > > > > > > >> of
> > > > > > > > > >> >> the
> > > > > > > > > >> >> >>> > > > examples I
> > > > > > > > > >> >> >>> > > > > > >>> have in
> > > > > > > > > >> >> >>> > > > > > >>> > >> the
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> spec based
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> on the updated
> header
> > > > size.
> > > > > > Then
> > > > > > > > we
> > > > > > > > > >> >> should
> > > > > > > > > >> >> >>> be
> > > > > > > > > >> >> >>> > good
> > > > > > > > > >> >> >>> > > > to
> > > > > > > > > >> >> >>> > > > > go
> > > > > > > > > >> >> >>> > > > > > >>> for
> > > > > > > > > >> >> >>> > > > > > >>> > the
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> markdown format.
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> Thanks everyone!
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> Andrew
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> On Tue, Feb 17,
> 2026
> > at
> > > > > > 7:28 PM
> > > > > > > > > >> PRATEEK
> > > > > > > > > >> >> >>> GAUR <
> > > > > > > > > >> >> >>> > > > > > >>> > >> prtkgaur@gmail.com>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> wrote:
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Hi team,
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 1) Andrew
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for
> > > working
> > > > > on
> > > > > > > test
> > > > > > > > > >> files.
> > > > > > > > > >> >> >>> My PR
> > > > > > > > > >> >> >>> > > did
> > > > > > > > > >> >> >>> > > > > add
> > > > > > > > > >> >> >>> > > > > > >>> all
> > > > > > > > > >> >> >>> > > > > > >>> > the
> > > > > > > > > >> >> >>> > > > > > >>> > >> > test
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> files I
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> used to
> > benchmark
> > > on
> > > > > > > > datasets.
> > > > > > > > > >> >> Maybe
> > > > > > > > > >> >> >>> we
> > > > > > > > > >> >> >>> > can
> > > > > > > > > >> >> >>> > > > club
> > > > > > > > > >> >> >>> > > > > > it
> > > > > > > > > >> >> >>> > > > > > >>> > >> together.
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> WIll also
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> aid
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> cross language
> > > > testing
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Kosta
> Tarasov
> > > > > working
> > > > > > > on
> > > > > > > > > Rust
> > > > > > > > > >> >> >>> > > > implementation.
> > > > > > > > > >> >> >>> > > > > > >>> This is
> > > > > > > > > >> >> >>> > > > > > >>> > >> > great.
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> Thanks
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 2) Antoine
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks a lot
> > for
> > > > > > > reporting
> > > > > > > > > the
> > > > > > > > > >> >> >>> numbers
> > > > > > > > > >> >> >>> > on
> > > > > > > > > >> >> >>> > > > AMD.
> > > > > > > > > >> >> >>> > > > > > >>> Looks
> > > > > > > > > >> >> >>> > > > > > >>> > >> like
> > > > > > > > > >> >> >>> > > > > > >>> > >> > you
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> are
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> getting 8X the
> > > > > decoding
> > > > > > > > > >> >> performance of
> > > > > > > > > >> >> >>> > BSS.
> > > > > > > > > >> >> >>> > > > This
> > > > > > > > > >> >> >>> > > > > > is
> > > > > > > > > >> >> >>> > > > > > >>> > >> > amazing!!.
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for
> > > > > > acknowledging
> > > > > > > > the
> > > > > > > > > >> >> >>> sampling
> > > > > > > > > >> >> >>> > > > design.
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - I agree with
> > you
> > > > on
> > > > > > > > > >> Fastlanes. In
> > > > > > > > > >> >> >>> some
> > > > > > > > > >> >> >>> > > crude
> > > > > > > > > >> >> >>> > > > > > >>> > >> experiments I
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> didn't get
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> a good perf
> > > benefit
> > > > > from
> > > > > > > it
> > > > > > > > on
> > > > > > > > > >> >> >>> Graviton3
> > > > > > > > > >> >> >>> > > (but
> > > > > > > > > >> >> >>> > > > > > maybe
> > > > > > > > > >> >> >>> > > > > > >>> > there
> > > > > > > > > >> >> >>> > > > > > >>> > >> was
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> something
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> wrong with my
> > > > > > > > implementation).
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Locking the
> > > 16bit
> > > > > > > > exception
> > > > > > > > > >> >> >>> encoding for
> > > > > > > > > >> >> >>> > > the
> > > > > > > > > >> >> >>> > > > > > spec
> > > > > > > > > >> >> >>> > > > > > >>> in
> > > > > > > > > >> >> >>> > > > > > >>> > >> this
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> case.
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Awesome I
> > think
> > > we
> > > > > > have
> > > > > > > > > solved
> > > > > > > > > >> >> for
> > > > > > > > > >> >> >>> all
> > > > > > > > > >> >> >>> > > open
> > > > > > > > > >> >> >>> > > > > > >>> questions
> > > > > > > > > >> >> >>> > > > > > >>> > >> minus
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> the
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> version byte
> :).
> > > > (will
> > > > > > get
> > > > > > > > > back
> > > > > > > > > >> on
> > > > > > > > > >> >> >>> this
> > > > > > > > > >> >> >>> > > soon)
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 3) Micah
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - FastLanes :
> > The
> > > > > > current
> > > > > > > > spec
> > > > > > > > > >> does
> > > > > > > > > >> >> >>> allow
> > > > > > > > > >> >> >>> > > for
> > > > > > > > > >> >> >>> > > > > > using
> > > > > > > > > >> >> >>> > > > > > >>> > >> FastLane
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> with the
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> configurable
> > enum
> > > > > value
> > > > > > > for
> > > > > > > > > >> >> layout. We
> > > > > > > > > >> >> >>> > > should
> > > > > > > > > >> >> >>> > > > be
> > > > > > > > > >> >> >>> > > > > > >>> able to
> > > > > > > > > >> >> >>> > > > > > >>> > >> > inject
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> any
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> layout
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> in the current
> > > > design.
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Working on
> resolving
> > > all
> > > > > > > > remaining
> > > > > > > > > >> open
> > > > > > > > > >> >> >>> > comments
> > > > > > > > > >> >> >>> > > > on
> > > > > > > > > >> >> >>> > > > > > the
> > > > > > > > > >> >> >>> > > > > > >>> spec
> > > > > > > > > >> >> >>> > > > > > >>> > >> this
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> week.
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Best
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Prateek
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> On Tue, Feb 10,
> 2026
> > > at
> > > > > > > 3:37 AM
> > > > > > > > > >> Steve
> > > > > > > > > >> >> >>> > Loughran <
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> stevel@cloudera.com>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> wrote:
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> On Sun, 8 Feb
> 2026
> > at
> > > > > > 18:12,
> > > > > > > > > Micah
> > > > > > > > > >> >> >>> Kornfield
> > > > > > > > > >> >> >>> > <
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > emkornfield@gmail.com>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> wrote:
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> It looks like
> the
> > > > actual
> > > > > > > issue
> > > > > > > > > >> >> >>> described for
> > > > > > > > > >> >> >>> > > ORC
> > > > > > > > > >> >> >>> > > > > in
> > > > > > > > > >> >> >>> > > > > > >>> the
> > > > > > > > > >> >> >>> > > > > > >>> > >> paper
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> is that
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> it
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> has multiple
> > > > > sub-encodings
> > > > > > > in
> > > > > > > > a
> > > > > > > > > >> >> batch.
> > > > > > > > > >> >> >>> This
> > > > > > > > > >> >> >>> > > is
> > > > > > > > > >> >> >>> > > > > > >>> different
> > > > > > > > > >> >> >>> > > > > > >>> > >> then
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> the
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> design
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> proposed here
> > where
> > > > > there
> > > > > > is
> > > > > > > > > still
> > > > > > > > > >> >> fixed
> > > > > > > > > >> >> >>> > > > encoding
> > > > > > > > > >> >> >>> > > > > > per
> > > > > > > > > >> >> >>> > > > > > >>> page
> > > > > > > > > >> >> >>> > > > > > >>> > >> in
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> parquet.
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> Given reasonably
> > > sized
> > > > > > > pages I
> > > > > > > > > >> don't
> > > > > > > > > >> >> >>> think
> > > > > > > > > >> >> >>> > > > branch
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> misprediction
> should
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> be a
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> big issue for
> new
> > > > > > encodings.
> > > > > > > > I
> > > > > > > > > >> agree
> > > > > > > > > >> >> >>> that
> > > > > > > > > >> >> >>> > we
> > > > > > > > > >> >> >>> > > > > should
> > > > > > > > > >> >> >>> > > > > > >>> be
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> conservative in
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> general for
> adding
> > > new
> > > > > > > > > encodings.
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> +1
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > > > > > >> >> >>> > > > > > >>> > >> > >
> > > > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > > > > >> >> >>> > > > > > >>> > >
> > > > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > > > >> >> >>> > > > > > >>>
> > > > > > > > > >> >> >>> > > > > > >>
> > > > > > > > > >> >> >>> > > > > >
> > > > > > > > > >> >> >>> > > > >
> > > > > > > > > >> >> >>> > > >
> > > > > > > > > >> >> >>> > >
> > > > > > > > > >> >> >>> >
> > > > > > > > > >> >> >>>
> > > > > > > > > >> >> >>
> > > > > > > > > >> >>
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by PRATEEK GAUR <pr...@gmail.com>.
Thanks everyone!
On Mon, Nov 9, 2026 at 11:58 AM Vinoo Ganesh <vi...@gmail.com> wrote:
> Thanks everyone!
>
> Thanks,
> Vinoo Ganesh | vinoo.ganesh@gmail.com
>
> <Vi...@gmail.com>
>
>
> On Mon, Nov 9, 2026 at 2:49 PM Andrew Lamb <an...@gmail.com> wrote:
>
> > The Java implementation was just merged[1]!
> >
> > Thank you Gang, Vinoo, Prateek and everyone who helped push that one
> along
> >
> > Andrew
> >
> > [1]: https://github.com/apache/parquet-java/pull/3397#event-35545451335
> >
> > On Tue, Oct 13, 2026 at 12:52 PM PRATEEK GAUR <pr...@gmail.com>
> wrote:
> >
> > > Hi team,
> > >
> > > Quick update re: the C++ implementation.
> > > Thanks to all the reviewers, we merged the C++ implementation today.
> > > There are a few follow-up items that I'll address in a subsequent PR.
> > >
> > > Thanks
> > > Prateek
> > >
> > > On Sun, Sep 13, 2026 at 6:54 PM Vinoo Ganesh <vi...@gmail.com>
> > > wrote:
> > >
> > > > Quick update re: the Java impl. Thanks for the feedback, on
> > > > https://github.com/apache/parquet-java/pull/3397, Russel! I pulled
> > > > https://github.com/apache/parquet-java/pull/3791 into its own PR
> > (thanks
> > > > for the quick approval too, Gang). Assuming no objections, I think we
> > can
> > > > merge #3791, and then go ahead of #3397.
> > > >
> > > > Thanks,
> > > > Vinoo Ganesh | vinoo.ganesh@gmail.com
> > > >
> > > > <Vi...@gmail.com>
> > > >
> > > >
> > > > On Tue, Sep 8, 2026 at 10:54 AM PRATEEK GAUR <pr...@gmail.com>
> > wrote:
> > > >
> > > > > [Rust] Awesome thanks Andrew, Kostas and Devin.
> > > > > [Java] Thanks Gang for the feedback on Java PR.
> > > > > [C++] Thanks Gang for another round of feedback on the C++ one.
> > > > > [Go] Arnav and I will be working on the golang impl this week.
> > > > >
> > > > > This is great progress all around.
> > > > >
> > > > > On Sat, Sep 5, 2026 at 1:26 PM Vinoo Ganesh <
> vinoo.ganesh@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Thanks all - I'm addressing Gang's comments on
> > > > > > https://github.com/apache/parquet-java/pull/3397 now, so
> hopefully
> > > > well
> > > > > > also be in a good state for merging the java implementation soon.
> > > > > >
> > > > > > Thanks,
> > > > > > Vinoo Ganesh | vinoo.ganesh@gmail.com
> > > > > >
> > > > > > <Vi...@gmail.com>
> > > > > >
> > > > > >
> > > > > > On Sat, Sep 5, 2026 at 8:56 AM Andrew Lamb <
> andrewlamb11@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > We just merged the Rust implementation[1] from Kosta and Devan.
> > > With
> > > > > the
> > > > > > > imminent release of Parquet Format 2.14.0 I think we'll be well
> > on
> > > > our
> > > > > > way.
> > > > > > >
> > > > > > > Thanks again for pushing this along,
> > > > > > > Andrew
> > > > > > >
> > > > > > > [1]: https://github.com/apache/arrow-rs/pull/9372
> > > > > > >
> > > > > > > On Fri, Sep 4, 2026 at 5:42 PM PRATEEK GAUR <
> prtkgaur@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi team,
> > > > > > > >
> > > > > > > > Thanks Micah for multiple rounds of review and thanks Gang
> for
> > > > > taking a
> > > > > > > > look at the PR and giving feedback. I've addressed all
> comments
> > > in
> > > > > the
> > > > > > > PR.
> > > > > > > > Looking forward to the next round of feedback (or approval).
> > > Would
> > > > > like
> > > > > > > to
> > > > > > > > get it in soon.
> > > > > > > >
> > > > > > > > https://github.com/apache/arrow/pull/48345/changes
> > > > > > > >
> > > > > > > > With this done and after talking to Arnav, I plan to spend
> some
> > > > time
> > > > > on
> > > > > > > the
> > > > > > > > ALP implementation in the arrow-go repository helping spread
> > the
> > > > > > support
> > > > > > > to
> > > > > > > > more languages.
> > > > > > > >
> > > > > > > > Best
> > > > > > > > Prateek
> > > > > > > >
> > > > > > > > On Sun, Aug 2, 2026 at 10:42 PM PRATEEK GAUR <
> > prtkgaur@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks Divjot for bringing it up.
> > > > > > > > >
> > > > > > > > > I was under the same impression as Russell and Andrew that
> > we'd
> > > > > have
> > > > > > it
> > > > > > > > > flag gated.
> > > > > > > > > With the new versioning proposal 'preview' might be a good
> > tag
> > > > for
> > > > > > it.
> > > > > > > > >
> > > > > > > > > Best
> > > > > > > > > Prateek
> > > > > > > > >
> > > > > > > > > On Sat, Aug 1, 2026 at 12:39 AM Andrew Lamb <
> > > > > andrewlamb11@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Div: Here is a proposal to mark ALP in Preview: [1]
> (using a
> > > > > comment
> > > > > > > in
> > > > > > > > >> the
> > > > > > > > >> spec section).
> > > > > > > > >>
> > > > > > > > >> Russel: That is my understanding as well.
> > > > > > > > >>
> > > > > > > > >> Hopefully we are able to formalize the versioning process
> > > > shortly
> > > > > so
> > > > > > > we
> > > > > > > > >> can
> > > > > > > > >> formalize the notion of "preview"
> > > > > > > > >>
> > > > > > > > >> Andrew
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> [1]: https://github.com/apache/parquet-format/pull/604
> > > > > > > > >>
> > > > > > > > >> On Fri, Jul 31, 2026 at 11:15 AM Russell Spitzer <
> > > > > > > > >> russell.spitzer@gmail.com>
> > > > > > > > >> wrote:
> > > > > > > > >>
> > > > > > > > >> > I thought we had agreed it would be merged, and features
> > > like
> > > > > this
> > > > > > > > would
> > > > > > > > >> > be feature flag gated. So a V-Old writer could always
> opt
> > in
> > > > to
> > > > > > > using
> > > > > > > > >> this
> > > > > > > > >> > encoding but that must be a conscious effort if you are
> > not
> > > > > using
> > > > > > > > VNext
> > > > > > > > >> of
> > > > > > > > >> > Parquet. If you are on V-Next the writer can choose to
> use
> > > ALP
> > > > > > > > whenever
> > > > > > > > >> it
> > > > > > > > >> > likes without user intervention.
> > > > > > > > >> >
> > > > > > > > >> > On Fri, Jul 31, 2026 at 2:20 AM Divjot Arora via dev <
> > > > > > > > >> > dev@parquet.apache.org> wrote:
> > > > > > > > >> >
> > > > > > > > >> >> Yes, that’s my understanding as well. However, as you
> > > noted,
> > > > > the
> > > > > > > > >> >> versioning
> > > > > > > > >> >> discussion is still in early stages and hasn’t
> prescribed
> > > > > > concrete
> > > > > > > > >> >> mechanisms yet. If ALP goes into parquet-format 2.14.0,
> > > > there’s
> > > > > > no
> > > > > > > > >> >> indication that it’s in preview or that writers
> shouldn’t
> > > use
> > > > > it
> > > > > > by
> > > > > > > > >> >> default
> > > > > > > > >> >> yet. Given that we’ve held off on merging other
> > > incompatible
> > > > > > > changes
> > > > > > > > >> (e.g
> > > > > > > > >> >> removal of path_in_schema) until the versioning
> > discussion
> > > > > > closes,
> > > > > > > my
> > > > > > > > >> >> preference would be to leave ALP un-merged for the time
> > > being
> > > > > as
> > > > > > > > well.
> > > > > > > > >> >>
> > > > > > > > >> >> — Div
> > > > > > > > >> >>
> > > > > > > > >> >> On Fri, Jul 31, 2026 at 08:59 Andrew Lamb <
> > > > > > andrewlamb11@gmail.com>
> > > > > > > > >> wrote:
> > > > > > > > >> >>
> > > > > > > > >> >> > After re-reading Julien's Versioning Proposal, my
> > > > > understanding
> > > > > > > is
> > > > > > > > >> that
> > > > > > > > >> >> if
> > > > > > > > >> >> > we adopt that system, we will somehow mark ALP in
> > > "Preview"
> > > > > in
> > > > > > > > >> >> > parquet-format for release Version 3.
> > > > > > > > >> >> >
> > > > > > > > >> >> > The versioning document doesn't seem to proscribe the
> > > > > > particular
> > > > > > > > >> method
> > > > > > > > >> >> > for annotating "preview" features -- I think clearly
> > > > marking
> > > > > > them
> > > > > > > > as
> > > > > > > > >> >> such
> > > > > > > > >> >> > in parquet.thrift as well as reflecting that on the
> > > > > > > > >> >> >
> https://parquet.apache.org/docs/file-format/versions/
> > > page
> > > > > > > somehow
> > > > > > > > >> >> would
> > > > > > > > >> >> > be adequate.
> > > > > > > > >> >> >
> > > > > > > > >> >> > Andrew
> > > > > > > > >> >> >
> > > > > > > > >> >> > On Thu, Jul 30, 2026 at 11:50 AM Andrew Lamb <
> > > > > > > > andrewlamb11@gmail.com
> > > > > > > > >> >
> > > > > > > > >> >> > wrote:
> > > > > > > > >> >> >
> > > > > > > > >> >> >> In my mind ALP is in the same position as other
> > > encodings
> > > > > that
> > > > > > > > were
> > > > > > > > >> >> added
> > > > > > > > >> >> >> recently[1] (such as BYTE_STREAM_SPLIT for non
> > floating
> > > > > point
> > > > > > > > types)
> > > > > > > > >> >> >>
> > > > > > > > >> >> >> That is to say it will be available in the latest
> open
> > > > > source
> > > > > > > > >> writers
> > > > > > > > >> >> and
> > > > > > > > >> >> >> readers, but realistically not widely used in the
> > > > ecosystem
> > > > > > > where
> > > > > > > > >> >> >> compatibility is a concern (a large part)
> > > > > > > > >> >> >>
> > > > > > > > >> >> >> I view Julian's work[2][3] to gather consensus on
> how
> > to
> > > > > name
> > > > > > > > >> >> >> combinations of features (e.g. "versions") as the
> > > > mechanism
> > > > > by
> > > > > > > > >> which we
> > > > > > > > >> >> >> will accelerate the adoption of this change (along
> > with
> > > > > other
> > > > > > > > >> forward
> > > > > > > > >> >> >> incompatible changes) across the ecosystem.
> > > > > > > > >> >> >>
> > > > > > > > >> >> >> > Will we exclude this change when cutting
> > > parquet-format
> > > > > > > v2.14.0?
> > > > > > > > >> >> >>
> > > > > > > > >> >> >> No I don't think so -- I would expect it to go out
> in
> > > the
> > > > > next
> > > > > > > > >> >> >> parquet-format version as do other changes in the
> > spec.
> > > > > > > > >> >> >>
> > > > > > > > >> >> >> Andrew
> > > > > > > > >> >> >>
> > > > > > > > >> >> >> [1]:
> > > > https://parquet.apache.org/docs/file-format/versions/
> > > > > > > > >> >> >> [2]:
> > > > > > > > >>
> > > > https://lists.apache.org/thread/xmnj8h0h8ozmrgox5tydhhs7yz7q2hss
> > > > > > > > >> >> >> [3]:
> > > > > > > > >>
> > > > https://lists.apache.org/thread/qlf8lg90gqq41lllyy9mk2f0skqvftqj
> > > > > > > > >> >> >>
> > > > > > > > >> >> >>
> > > > > > > > >> >> >> On Thu, Jul 30, 2026 at 3:51 AM Divjot Arora via
> dev <
> > > > > > > > >> >> >> dev@parquet.apache.org> wrote:
> > > > > > > > >> >> >>
> > > > > > > > >> >> >>> Hi Andrew,
> > > > > > > > >> >> >>>
> > > > > > > > >> >> >>> How does this interact with the ongoing versioning
> > > > > > discussion?
> > > > > > > > >> Given
> > > > > > > > >> >> that
> > > > > > > > >> >> >>> new encodings are not forward compatible, I would
> > > assume
> > > > > ALP
> > > > > > is
> > > > > > > > >> >> included
> > > > > > > > >> >> >>> in
> > > > > > > > >> >> >>> Parquet V<next> (V3?), but it's not clear to me how
> > > that
> > > > > > works
> > > > > > > if
> > > > > > > > >> it's
> > > > > > > > >> >> >>> merged to the parquet-format master branch now.
> Will
> > we
> > > > > > exclude
> > > > > > > > >> this
> > > > > > > > >> >> >>> change
> > > > > > > > >> >> >>> when cutting parquet-format v2.14.0?
> > > > > > > > >> >> >>>
> > > > > > > > >> >> >>> -- Div
> > > > > > > > >> >> >>>
> > > > > > > > >> >> >>> On Thu, Jul 30, 2026 at 8:51 AM Andrew Lamb <
> > > > > > > > >> andrewlamb11@gmail.com>
> > > > > > > > >> >> >>> wrote:
> > > > > > > > >> >> >>>
> > > > > > > > >> >> >>> > Hello,
> > > > > > > > >> >> >>> >
> > > > > > > > >> >> >>> > As an update, thanks to the epic work from many
> > > > > > contributors,
> > > > > > > > >> >> >>> especially
> > > > > > > > >> >> >>> > Prateek, I just merged the parquet-format change
> > for
> > > > > > ALP[1].
> > > > > > > We
> > > > > > > > >> >> plan to
> > > > > > > > >> >> >>> > make additional editorial The remaining items I
> > know
> > > of
> > > > > are
> > > > > > > > >> merging
> > > > > > > > >> >> the
> > > > > > > > >> >> >>> > examples from parquet-testing[2], and then of
> > course
> > > > the
> > > > > > > > various
> > > > > > > > >> >> >>> > implementations
> > > > > > > > >> >> >>> >
> > > > > > > > >> >> >>> > Thanks again everyone. This is a great step
> forward
> > > > > > > > >> >> >>> > Andrew
> > > > > > > > >> >> >>> >
> > > > > > > > >> >> >>> > [1]:
> > > https://github.com/apache/parquet-format/pull/557
> > > > > > > > >> >> >>> > [2]:
> > > > https://github.com/apache/parquet-testing/pull/100
> > > > > > > > >> >> >>> >
> > > > > > > > >> >> >>> > On Tue, Jul 21, 2026 at 4:39 PM Julien Le Dem <
> > > > > > > > julien@apache.org
> > > > > > > > >> >
> > > > > > > > >> >> >>> wrote:
> > > > > > > > >> >> >>> >
> > > > > > > > >> >> >>> > > It was also in my spam folder... :(
> > > > > > > > >> >> >>> > > I've seen a bunch of apache list emails go to
> my
> > > spam
> > > > > > > > recently.
> > > > > > > > >> >> >>> > >
> > > > > > > > >> >> >>> > > On Tue, Jul 21, 2026 at 10:58 AM Russell
> Spitzer
> > <
> > > > > > > > >> >> >>> > > russell.spitzer@gmail.com>
> > > > > > > > >> >> >>> > > wrote:
> > > > > > > > >> >> >>> > >
> > > > > > > > >> >> >>> > > > Yes, my hope was that my vote would kick it
> out
> > > of
> > > > > spam
> > > > > > > but
> > > > > > > > >> we
> > > > > > > > >> >> >>> > definitely
> > > > > > > > >> >> >>> > > > had two other votes that also went to spam
> for
> > > me.
> > > > > > > > >> >> >>> > > >
> > > > > > > > >> >> >>> > > > For anyone looking the thread is here
> > > > > > > > >> >> >>> > > >
> > > > > > > > >> >>
> > > > > https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt
> > > > > > > > >> >> >>> > > >
> > > > > > > > >> >> >>> > > > On Tue, Jul 21, 2026 at 10:57 AM Micah
> > Kornfield
> > > <
> > > > > > > > >> >> >>> > emkornfield@gmail.com>
> > > > > > > > >> >> >>> > > > wrote:
> > > > > > > > >> >> >>> > > >
> > > > > > > > >> >> >>> > > > > I've seen other's votes not go to spam, so
> > > > > hopefully
> > > > > > > > people
> > > > > > > > >> >> will
> > > > > > > > >> >> >>> see
> > > > > > > > >> >> >>> > it
> > > > > > > > >> >> >>> > > > in
> > > > > > > > >> >> >>> > > > > there inboxes (or see this thread and look
> in
> > > > there
> > > > > > > > spam).
> > > > > > > > >> >> >>> > > > >
> > > > > > > > >> >> >>> > > > > On Tue, Jul 21, 2026 at 10:52 AM PRATEEK
> > GAUR <
> > > > > > > > >> >> >>> prtkgaur@gmail.com>
> > > > > > > > >> >> >>> > > > wrote:
> > > > > > > > >> >> >>> > > > >
> > > > > > > > >> >> >>> > > > > > Okay,
> > > > > > > > >> >> >>> > > > > >
> > > > > > > > >> >> >>> > > > > > I sent out the voting email yesterday and
> > so
> > > > far
> > > > > > I've
> > > > > > > > >> heard
> > > > > > > > >> >> >>> from 3
> > > > > > > > >> >> >>> > > > people
> > > > > > > > >> >> >>> > > > > > that it ended up in their "spam folder".
> > Dev
> > > > VPN
> > > > > to
> > > > > > > be
> > > > > > > > >> >> blamed
> > > > > > > > >> >> >>> for
> > > > > > > > >> >> >>> > > it?.
> > > > > > > > >> >> >>> > > > > >
> > > > > > > > >> >> >>> > > > > > What's the protocol here?
> > > > > > > > >> >> >>> > > > > > I let people hunt for the email in their
> > spam
> > > > > > folder
> > > > > > > or
> > > > > > > > >> send
> > > > > > > > >> >> >>> out
> > > > > > > > >> >> >>> > > > another
> > > > > > > > >> >> >>> > > > > > one?
> > > > > > > > >> >> >>> > > > > >
> > > > > > > > >> >> >>> > > > > > Best
> > > > > > > > >> >> >>> > > > > > Prateek
> > > > > > > > >> >> >>> > > > > >
> > > > > > > > >> >> >>> > > > > > On Sun, Jul 19, 2026 at 10:47 PM PRATEEK
> > > GAUR <
> > > > > > > > >> >> >>> prtkgaur@gmail.com>
> > > > > > > > >> >> >>> > > > > wrote:
> > > > > > > > >> >> >>> > > > > >
> > > > > > > > >> >> >>> > > > > > > Hi team,
> > > > > > > > >> >> >>> > > > > > >
> > > > > > > > >> >> >>> > > > > > > Vinoo has added the tests which were
> > > > requested
> > > > > in
> > > > > > > the
> > > > > > > > >> last
> > > > > > > > >> >> >>> round
> > > > > > > > >> >> >>> > of
> > > > > > > > >> >> >>> > > > > > review.
> > > > > > > > >> >> >>> > > > > > > In his own words.
> > > > > > > > >> >> >>> > > > > > >
> > > > > > > > >> >> >>> > > > > > > "
> > > > > > > > >> >> >>> > > > > > > The extreme value tests Micah wanted
> are
> > > done
> > > > > and
> > > > > > > > >> pushed.
> > > > > > > > >> >> I
> > > > > > > > >> >> >>> added
> > > > > > > > >> >> >>> > > > > > coverage
> > > > > > > > >> >> >>> > > > > > > for values that need the full FOR bit
> > width
> > > > > after
> > > > > > > the
> > > > > > > > >> >> frame
> > > > > > > > >> >> >>> of
> > > > > > > > >> >> >>> > > > > reference
> > > > > > > > >> >> >>> > > > > > is
> > > > > > > > >> >> >>> > > > > > > applied: 64-bit (the case where the
> > signed
> > > > max
> > > > > > > minus
> > > > > > > > >> min
> > > > > > > > >> >> >>> > > subtraction
> > > > > > > > >> >> >>> > > > > > > overflows), 63-bit (the non-overflow
> > case),
> > > > and
> > > > > > > > 32-bit
> > > > > > > > >> for
> > > > > > > > >> >> >>> > floats.
> > > > > > > > >> >> >>> > > > All
> > > > > > > > >> >> >>> > > > > > > lossless with zero exceptions. I also
> > > > verified
> > > > > > them
> > > > > > > > >> >> through
> > > > > > > > >> >> >>> Arrow
> > > > > > > > >> >> >>> > > C++
> > > > > > > > >> >> >>> > > > > ALP
> > > > > > > > >> >> >>> > > > > > > reader (the local build), so the
> > > Java-written
> > > > > > > extreme
> > > > > > > > >> >> columns
> > > > > > > > >> >> >>> > > decode
> > > > > > > > >> >> >>> > > > > > > correctly in C++, bit-exact against the
> > > > > expected
> > > > > > > > >> values."
> > > > > > > > >> >> >>> > > > > > >
> > > > > > > > >> >> >>> > > > > > > State of the PR (#3397):
> > > > > > > > >> >> >>> > > > > > >
> > > > > > > > >> >> >>> > > > > > > - Merges cleanly with master (I
> pulled
> > > > > master
> > > > > > in
> > > > > > > > and
> > > > > > > > >> >> >>> resolved
> > > > > > > > >> >> >>> > > the
> > > > > > > > >> >> >>> > > > > > > conflicts).
> > > > > > > > >> >> >>> > > > > > > - 41 of 44 review threads resolved.
> > The
> > > 3
> > > > > > > > remaining
> > > > > > > > >> are
> > > > > > > > >> >> >>> > > > intentional:
> > > > > > > > >> >> >>> > > > > > > one deferred perf nit, one that
> stays
> > > open
> > > > > > until
> > > > > > > > >> >> >>> > parquet-format
> > > > > > > > >> >> >>> > > > > ships
> > > > > > > > >> >> >>> > > > > > ALP,
> > > > > > > > >> >> >>> > > > > > >
> > > > > > > > >> >> >>> > > > > > > "
> > > > > > > > >> >> >>> > > > > > >
> > > > > > > > >> >> >>> > > > > > > With this both the
> > > > > > > > >> >> >>> > > > > > > 1) C++ PR :
> > > > > > > > >> >> >>> https://github.com/apache/arrow/pull/48345/changes
> > > > > > > > >> >> >>> > > > > > > 2) Java PR :
> > > > > > > > >> >> >>> https://github.com/apache/parquet-java/pull/3397
> > > > > > > > >> >> >>> > > > > > >
> > > > > > > > >> >> >>> > > > > > > Are cross language tested and range of
> > test
> > > > > > > > scenarios.
> > > > > > > > >> >> >>> > > > > > >
> > > > > > > > >> >> >>> > > > > > > Best
> > > > > > > > >> >> >>> > > > > > > Prateek
> > > > > > > > >> >> >>> > > > > > >
> > > > > > > > >> >> >>> > > > > > >
> > > > > > > > >> >> >>> > > > > > > On Mon, Jul 13, 2026 at 8:38 AM PRATEEK
> > > GAUR
> > > > <
> > > > > > > > >> >> >>> prtkgaur@gmail.com
> > > > > > > > >> >> >>> > >
> > > > > > > > >> >> >>> > > > > wrote:
> > > > > > > > >> >> >>> > > > > > >
> > > > > > > > >> >> >>> > > > > > >> Thanks Matt,
> > > > > > > > >> >> >>> > > > > > >>
> > > > > > > > >> >> >>> > > > > > >> I'll be on it this week.
> > > > > > > > >> >> >>> > > > > > >>
> > > > > > > > >> >> >>> > > > > > >> Best
> > > > > > > > >> >> >>> > > > > > >> Prateek
> > > > > > > > >> >> >>> > > > > > >>
> > > > > > > > >> >> >>> > > > > > >> On Mon, Jul 13, 2026 at 8:36 AM Matt
> > > Topol <
> > > > > > > > >> >> >>> > > zotthewizard@gmail.com>
> > > > > > > > >> >> >>> > > > > > >> wrote:
> > > > > > > > >> >> >>> > > > > > >>
> > > > > > > > >> >> >>> > > > > > >>> There is also the Go implementation
> > > > submitted
> > > > > > by
> > > > > > > > >> Arnav (
> > > > > > > > >> >> >>> > > > > > >>>
> > > > https://github.com/apache/arrow-go/pull/704)
> > > > > > > which
> > > > > > > > >> is
> > > > > > > > >> >> >>> waiting
> > > > > > > > >> >> >>> > > for
> > > > > > > > >> >> >>> > > > > > >>> updates.
> > > > > > > > >> >> >>> > > > > > >>>
> > > > > > > > >> >> >>> > > > > > >>> Just wanted to make sure it didn't
> get
> > > lost
> > > > > > here.
> > > > > > > > >> >> >>> > > > > > >>>
> > > > > > > > >> >> >>> > > > > > >>> --Matt
> > > > > > > > >> >> >>> > > > > > >>>
> > > > > > > > >> >> >>> > > > > > >>> On Mon, Jul 13, 2026 at 11:30 AM
> > PRATEEK
> > > > > GAUR <
> > > > > > > > >> >> >>> > > prtkgaur@gmail.com>
> > > > > > > > >> >> >>> > > > > > >>> wrote:
> > > > > > > > >> >> >>> > > > > > >>>
> > > > > > > > >> >> >>> > > > > > >>> > Hi all,
> > > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > > >> >> >>> > > > > > >>> > I'd like to share a status update
> on
> > > the
> > > > > ALP
> > > > > > > > >> encoding
> > > > > > > > >> >> >>> effort
> > > > > > > > >> >> >>> > > and
> > > > > > > > >> >> >>> > > > > get
> > > > > > > > >> >> >>> > > > > > a
> > > > > > > > >> >> >>> > > > > > >>> feel
> > > > > > > > >> >> >>> > > > > > >>> > from the community before starting
> a
> > > > formal
> > > > > > > vote.
> > > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > > >> >> >>> > > > > > >>> > We now have the specification plus
> > > > > > > > implementations
> > > > > > > > >> in
> > > > > > > > >> >> >>> both
> > > > > > > > >> >> >>> > > > > languages:
> > > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > > >> >> >>> > > > > > >>> > - Spec:
> > > > > > apache/parquet-format#557 —
> > > > > > > > >> GH-533
> > > > > > > > >> >> >>> Add
> > > > > > > > >> >> >>> > ALP
> > > > > > > > >> >> >>> > > > > > encoding
> > > > > > > > >> >> >>> > > > > > >>> > specification
> > > > > > > > >> >> >>> > > > > > >>> > - C++ (Arrow):
> > apache/arrow#48345
> > > —
> > > > > > > GH-48701
> > > > > > > > >> >> >>> > > [C++][Parquet]
> > > > > > > > >> >> >>> > > > > Add
> > > > > > > > >> >> >>> > > > > > >>> ALPpd
> > > > > > > > >> >> >>> > > > > > >>> > encoding
> > > > > > > > >> >> >>> > > > > > >>> > - Java:
> > > > > apache/parquet-java#3397
> > > > > > —
> > > > > > > > >> Parquet
> > > > > > > > >> >> >>> Java
> > > > > > > > >> >> >>> > ALP
> > > > > > > > >> >> >>> > > > > > >>> > Implementation (by Vinoo)
> > > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > > >> >> >>> > > > > > >>> > Current state:
> > > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > > >> >> >>> > > > > > >>> > - The C++ implementation has
> been
> > > > > through
> > > > > > > 3-4
> > > > > > > > >> >> rounds
> > > > > > > > >> >> >>> of
> > > > > > > > >> >> >>> > > > review.
> > > > > > > > >> >> >>> > > > > > >>> > - The Java implementation has
> > been
> > > > > > tracking
> > > > > > > > >> well
> > > > > > > > >> >> and
> > > > > > > > >> >> >>> has
> > > > > > > > >> >> >>> > > had
> > > > > > > > >> >> >>> > > > > > >>> initial
> > > > > > > > >> >> >>> > > > > > >>> > reviews from contributors,
> with
> > > no
> > > > > > major
> > > > > > > > >> >> remaining
> > > > > > > > >> >> >>> > > issues.
> > > > > > > > >> >> >>> > > > > > >>> > - Cross-language compatibility
> > > tests
> > > > > are
> > > > > > > > >> passing:
> > > > > > > > >> >> the
> > > > > > > > >> >> >>> > Arrow
> > > > > > > > >> >> >>> > > > C++
> > > > > > > > >> >> >>> > > > > > >>> decoder
> > > > > > > > >> >> >>> > > > > > >>> > reads Java-written data
> > > bit-exactly
> > > > > > > across
> > > > > > > > >> >> ~1.56M
> > > > > > > > >> >> >>> > values
> > > > > > > > >> >> >>> > > > and
> > > > > > > > >> >> >>> > > > > 18
> > > > > > > > >> >> >>> > > > > > >>> > fixtures, covering V1 and V2
> > > pages,
> > > > > > > > multiple
> > > > > > > > >> >> vector
> > > > > > > > >> >> >>> > > sizes,
> > > > > > > > >> >> >>> > > > > and
> > > > > > > > >> >> >>> > > > > > >>> > several
> > > > > > > > >> >> >>> > > > > > >>> > real datasets — zero
> > mismatches.
> > > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > > >> >> >>> > > > > > >>> > As raised in the java-pr we will
> be
> > > > adding
> > > > > > > > >> coverage
> > > > > > > > >> >> for
> > > > > > > > >> >> >>> > > extreme
> > > > > > > > >> >> >>> > > > > > values
> > > > > > > > >> >> >>> > > > > > >>> > (those requiring
> > > > > > > > >> >> >>> > > > > > >>> > 63-64 bits after FOR is applied)
> > > before
> > > > we
> > > > > > > > close a
> > > > > > > > >> >> vote.
> > > > > > > > >> >> >>> > We'll
> > > > > > > > >> >> >>> > > > aim
> > > > > > > > >> >> >>> > > > > > to
> > > > > > > > >> >> >>> > > > > > >>> get
> > > > > > > > >> >> >>> > > > > > >>> > that
> > > > > > > > >> >> >>> > > > > > >>> > done in parallel.
> > > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > > >> >> >>> > > > > > >>> > Plan: unless there are objections,
> I
> > > > intend
> > > > > > to
> > > > > > > > >> start a
> > > > > > > > >> >> >>> vote
> > > > > > > > >> >> >>> > at
> > > > > > > > >> >> >>> > > > the
> > > > > > > > >> >> >>> > > > > > end
> > > > > > > > >> >> >>> > > > > > >>> of
> > > > > > > > >> >> >>> > > > > > >>> > this week/early
> > > > > > > > >> >> >>> > > > > > >>> > next week.
> > > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > > >> >> >>> > > > > > >>> > Thanks
> > > > > > > > >> >> >>> > > > > > >>> > Prateek
> > > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > > >> >> >>> > > > > > >>> > On Wed, Jul 1, 2026 at 8:50 AM
> > PRATEEK
> > > > > GAUR <
> > > > > > > > >> >> >>> > > prtkgaur@gmail.com>
> > > > > > > > >> >> >>> > > > > > >>> wrote:
> > > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > > >> >> >>> > > > > > >>> > > Hi Team,
> > > > > > > > >> >> >>> > > > > > >>> > >
> > > > > > > > >> >> >>> > > > > > >>> > > Just wanted to provide some
> updates
> > > on
> > > > > ALP.
> > > > > > > > >> >> >>> > > > > > >>> > > Micah and I have done 3-4 rounds
> of
> > > > > review
> > > > > > > for
> > > > > > > > >> the
> > > > > > > > >> >> c++
> > > > > > > > >> >> >>> PR :
> > > > > > > > >> >> >>> > > > > > >>> > >
> > > > > https://github.com/apache/arrow/pull/48345
> > > > > > > > >> >> >>> > > > > > >>> > >
> > > > > > > > >> >> >>> > > > > > >>> > > For the Java implementation Vinoo
> > has
> > > > > been
> > > > > > > > >> working
> > > > > > > > >> >> on
> > > > > > > > >> >> >>> > > following
> > > > > > > > >> >> >>> > > > > PR
> > > > > > > > >> >> >>> > > > > > :
> > > > > > > > >> >> >>> > > > > > >>> > >
> > > > > > > > https://github.com/apache/parquet-java/pull/3397
> > > > > > > > >> >> >>> > > > > > >>> > >
> > > > > > > > >> >> >>> > > > > > >>> > > Best
> > > > > > > > >> >> >>> > > > > > >>> > > Prateek
> > > > > > > > >> >> >>> > > > > > >>> > >
> > > > > > > > >> >> >>> > > > > > >>> > > On Tue, May 5, 2026 at 1:50 PM
> > Micah
> > > > > > > Kornfield
> > > > > > > > <
> > > > > > > > >> >> >>> > > > > > >>> emkornfield@gmail.com>
> > > > > > > > >> >> >>> > > > > > >>> > > wrote:
> > > > > > > > >> >> >>> > > > > > >>> > >
> > > > > > > > >> >> >>> > > > > > >>> > >> Hi Antoine,
> > > > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > > > >> >> >>> > > > > > >>> > >> > Apologies if the question was
> > > > already
> > > > > > > asked,
> > > > > > > > >> but
> > > > > > > > >> >> >>> should
> > > > > > > > >> >> >>> > we
> > > > > > > > >> >> >>> > > > > care
> > > > > > > > >> >> >>> > > > > > >>> about
> > > > > > > > >> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT +
> ALP
> > > be
> > > > > more
> > > > > > > > >> >> efficient
> > > > > > > > >> >> >>> than
> > > > > > > > >> >> >>> > > > > FLOAT16
> > > > > > > > >> >> >>> > > > > > +
> > > > > > > > >> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for
> > > example?
> > > > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > > > >> >> >>> > > > > > >>> > >> It was. We thought we could
> defer
> > > it
> > > > > for
> > > > > > > the
> > > > > > > > >> >> >>> following
> > > > > > > > >> >> >>> > > > reasons:
> > > > > > > > >> >> >>> > > > > > >>> > >> 1. It's not clear there are a
> lot
> > > of
> > > > > easy
> > > > > > > > >> >> reference
> > > > > > > > >> >> >>> > > datasets
> > > > > > > > >> >> >>> > > > to
> > > > > > > > >> >> >>> > > > > > >>> test
> > > > > > > > >> >> >>> > > > > > >>> > its
> > > > > > > > >> >> >>> > > > > > >>> > >> effectiveness. It does look
> like
> > > > there
> > > > > > > might
> > > > > > > > be
> > > > > > > > >> >> one
> > > > > > > > >> >> >>> or
> > > > > > > > >> >> >>> > two
> > > > > > > > >> >> >>> > > on
> > > > > > > > >> >> >>> > > > > > >>> > huggingface
> > > > > > > > >> >> >>> > > > > > >>> > >> (e.g.
> > > > > > > > >> >> https://huggingface.co/datasets/kikitora/curdie
> > > > > > > > >> >> >>> ).
> > > > > > > > >> >> >>> > > > > > >>> > >> 2. It seemed likely that float
> 16
> > > was
> > > > > > more
> > > > > > > > >> likely
> > > > > > > > >> >> >>> used
> > > > > > > > >> >> >>> > for
> > > > > > > > >> >> >>> > > > > values
> > > > > > > > >> >> >>> > > > > > >>> that
> > > > > > > > >> >> >>> > > > > > >>> > >> were less likely to reduce to
> > > decimal
> > > > > > > values.
> > > > > > > > >> >> >>> > > > > > >>> > >> 3. It could be added as an
> > > extension
> > > > > > later
> > > > > > > if
> > > > > > > > >> >> needed.
> > > > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > > > >> >> >>> > > > > > >>> > >> Cheers,
> > > > > > > > >> >> >>> > > > > > >>> > >> Micah
> > > > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > > > >> >> >>> > > > > > >>> > >> On Tue, May 5, 2026 at 1:40 PM
> > > Antoine
> > > > > > > Pitrou
> > > > > > > > <
> > > > > > > > >> >> >>> > > > > antoine@python.org
> > > > > > > > >> >> >>> > > > > > >
> > > > > > > > >> >> >>> > > > > > >>> > wrote:
> > > > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > > >> >> >>> > > > > > >>> > >> > Hello,
> > > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > > >> >> >>> > > > > > >>> > >> > Apologies if the question was
> > > > already
> > > > > > > asked,
> > > > > > > > >> but
> > > > > > > > >> >> >>> should
> > > > > > > > >> >> >>> > we
> > > > > > > > >> >> >>> > > > > care
> > > > > > > > >> >> >>> > > > > > >>> about
> > > > > > > > >> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT +
> ALP
> > > be
> > > > > more
> > > > > > > > >> >> efficient
> > > > > > > > >> >> >>> than
> > > > > > > > >> >> >>> > > > > FLOAT16
> > > > > > > > >> >> >>> > > > > > +
> > > > > > > > >> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for
> > > example?
> > > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > > >> >> >>> > > > > > >>> > >> > Regards
> > > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > > >> >> >>> > > > > > >>> > >> > Antoine.
> > > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > > >> >> >>> > > > > > >>> > >> > Le 30/04/2026 à 01:10, PRATEEK
> > > GAUR
> > > > a
> > > > > > > écrit
> > > > > > > > :
> > > > > > > > >> >> >>> > > > > > >>> > >> > > Thanks Andrew and Micah for
> > > review
> > > > > > > > feedback
> > > > > > > > >> on
> > > > > > > > >> >> >>> the two
> > > > > > > > >> >> >>> > > > PR's
> > > > > > > > >> >> >>> > > > > > >>> > >> > > 1) (c++ arrow repo)
> > > > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > > > >> https://github.com/apache/arrow/pull/48345/changes
> > > > > > > > >> >> >>> > > > > > >>> > >> > > 2) (parquet-format repo)
> > > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > > >> >> https://github.com/apache/parquet-format/pull/557
> > > > > > > > >> >> >>> > > > > > >>> > >> > >
> > > > > > > > >> >> >>> > > > > > >>> > >> > > I have addressed all
> (unless I
> > > > > missed
> > > > > > > > >> >> something)
> > > > > > > > >> >> >>> > > comments
> > > > > > > > >> >> >>> > > > on
> > > > > > > > >> >> >>> > > > > > >>> the two
> > > > > > > > >> >> >>> > > > > > >>> > >> > PR's.
> > > > > > > > >> >> >>> > > > > > >>> > >> > >
> > > > > > > > >> >> >>> > > > > > >>> > >> > > Best
> > > > > > > > >> >> >>> > > > > > >>> > >> > > Prateek
> > > > > > > > >> >> >>> > > > > > >>> > >> > >
> > > > > > > > >> >> >>> > > > > > >>> > >> > > On Sat, Apr 25, 2026 at
> > 1:08 PM
> > > > > > PRATEEK
> > > > > > > > >> GAUR <
> > > > > > > > >> >> >>> > > > > > >>> prtkgaur@gmail.com>
> > > > > > > > >> >> >>> > > > > > >>> > >> wrote:
> > > > > > > > >> >> >>> > > > > > >>> > >> > >
> > > > > > > > >> >> >>> > > > > > >>> > >> > >> Thanks Andrew and Micah.
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >> `fair amount of feedback on
> > at
> > > > > least
> > > > > > > the
> > > > > > > > >> >> >>> > > implementations`
> > > > > > > > >> >> >>> > > > > > >>> > >> > >> For the c++ I have already
> > > > started
> > > > > > > > >> addressing
> > > > > > > > >> >> the
> > > > > > > > >> >> >>> > > > > feedback, I
> > > > > > > > >> >> >>> > > > > > >>> > should
> > > > > > > > >> >> >>> > > > > > >>> > >> be
> > > > > > > > >> >> >>> > > > > > >>> > >> > >> done with that
> > Monday/Tuesday.
> > > > > > > > >> >> >>> > > > > > >>> > >> > >> I think Vinoo too has been
> > > making
> > > > > > good
> > > > > > > > >> >> progress
> > > > > > > > >> >> >>> on
> > > > > > > > >> >> >>> > the
> > > > > > > > >> >> >>> > > > Java
> > > > > > > > >> >> >>> > > > > > >>> > >> > implementation.
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >> Best
> > > > > > > > >> >> >>> > > > > > >>> > >> > >> Prateek
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >> On Sat, Apr 25, 2026 at
> > > 12:55 PM
> > > > > > Andrew
> > > > > > > > >> Lamb <
> > > > > > > > >> >> >>> > > > > > >>> > andrewlamb11@gmail.com
> > > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > > >> >> >>> > > > > > >>> > >> > >> wrote:
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>> Got it. Thank you for the
> > > > > > > clarification
> > > > > > > > >> -- I
> > > > > > > > >> >> >>> will
> > > > > > > > >> >> >>> > try
> > > > > > > > >> >> >>> > > > and
> > > > > > > > >> >> >>> > > > > > look
> > > > > > > > >> >> >>> > > > > > >>> > into
> > > > > > > > >> >> >>> > > > > > >>> > >> the
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>> spec and the Rust
> > > > > implementation[1]
> > > > > > in
> > > > > > > > >> this
> > > > > > > > >> >> next
> > > > > > > > >> >> >>> > week
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>> [1]:
> > > > > > > > >> >> >>> https://github.com/apache/arrow-rs/pull/9372
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>> On Sat, Apr 25, 2026 at
> > > 12:01 PM
> > > > > > Micah
> > > > > > > > >> >> >>> Kornfield <
> > > > > > > > >> >> >>> > > > > > >>> > >> > emkornfield@gmail.com>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>> wrote:
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>> Hi Andrew,
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>> I think there is a fair
> > > amount
> > > > of
> > > > > > > > >> feedback
> > > > > > > > >> >> on
> > > > > > > > >> >> >>> at
> > > > > > > > >> >> >>> > > least
> > > > > > > > >> >> >>> > > > > the
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>> implementations,
> typically
> > I
> > > > > think
> > > > > > > > we've
> > > > > > > > >> >> waited
> > > > > > > > >> >> >>> > till
> > > > > > > > >> >> >>> > > > they
> > > > > > > > >> >> >>> > > > > > are
> > > > > > > > >> >> >>> > > > > > >>> > >> close to
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>> mergeable before a final
> > > vote.
> > > > > > > > >> Otherwise I
> > > > > > > > >> >> >>> agree
> > > > > > > > >> >> >>> > we
> > > > > > > > >> >> >>> > > > are
> > > > > > > > >> >> >>> > > > > > very
> > > > > > > > >> >> >>> > > > > > >>> > >> close.
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>> -Micah
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>> On Saturday, April 25,
> > 2026,
> > > > > Andrew
> > > > > > > > Lamb
> > > > > > > > >> <
> > > > > > > > >> >> >>> > > > > > >>> andrewlamb11@gmail.com
> > > > > > > > >> >> >>> > > > > > >>> > >
> > > > > > > > >> >> >>> > > > > > >>> > >> > wrote:
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> Thanks Prateek,
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> I think from this
> content
> > it
> > > > > looks
> > > > > > > to
> > > > > > > > me
> > > > > > > > >> >> like
> > > > > > > > >> >> >>> we
> > > > > > > > >> >> >>> > are
> > > > > > > > >> >> >>> > > > > ready
> > > > > > > > >> >> >>> > > > > > >>> to
> > > > > > > > >> >> >>> > > > > > >>> > >> start a
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> vote to explicitly
> accept
> > > ALP
> > > > > into
> > > > > > > > >> Parquet
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> Does anyone know of a
> > reason
> > > > we
> > > > > > > should
> > > > > > > > >> >> >>> postpone it
> > > > > > > > >> >> >>> > > for
> > > > > > > > >> >> >>> > > > > > >>> longer?
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> Perhaps someone needs
> some
> > > > more
> > > > > > time
> > > > > > > > to
> > > > > > > > >> >> >>> review?
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> Andrew
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> On Wed, Apr 22, 2026 at
> > > > 1:00 PM
> > > > > > > > PRATEEK
> > > > > > > > >> >> GAUR <
> > > > > > > > >> >> >>> > > > > > >>> > prtkgaur@gmail.com>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> wrote:
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Hi team,
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Hope everyone is doing
> > > well.
> > > > I
> > > > > > got
> > > > > > > a
> > > > > > > > >> >> chance
> > > > > > > > >> >> >>> to
> > > > > > > > >> >> >>> > work
> > > > > > > > >> >> >>> > > > > > >>> through all
> > > > > > > > >> >> >>> > > > > > >>> > >> the
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> remaining feedback and
> > > update
> > > > > the
> > > > > > > > spec
> > > > > > > > >> >> doc.
> > > > > > > > >> >> >>> Here
> > > > > > > > >> >> >>> > > are
> > > > > > > > >> >> >>> > > > > the
> > > > > > > > >> >> >>> > > > > > >>> new
> > > > > > > > >> >> >>> > > > > > >>> > >> > artifacts
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 1) Spec document :
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > > >> >> >>> > > > > > >>>
> > > > > > > > >> >> >>> > > > > >
> > > > > > > > >> >> >>> > > > >
> > > > > > > > >> >> >>> > > >
> > > > > > > > >> >> >>> > >
> > > > > > > > >> >> >>> >
> > > > > > > > >> >> >>>
> > > > > > > > >> >>
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-
> > 3fPtD_aWt/edit
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 2) Spec document in
> > parquet
> > > > > > format
> > > > > > > > >> repo :
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> >
> https://github.com/apache/parquet-format/pull/557
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 3) Alp implementation
> in
> > > > arrow
> > > > > > c++
> > > > > > > > >> repo :
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> >
> https://github.com/apache/arrow/pull/48345/changes
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 4) Alp implementation
> in
> > > > > > > parquet-java
> > > > > > > > >> >> repo :
> > > > > > > > >> >> >>> Work
> > > > > > > > >> >> >>> > > for
> > > > > > > > >> >> >>> > > > > > >>> Vinoo and
> > > > > > > > >> >> >>> > > > > > >>> > >> > Julien
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > https://github.com/apache/parquet-java/pull/3397
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 5) PR with test and
> > > > > benchmarking
> > > > > > > > >> >> artifacts in
> > > > > > > > >> >> >>> > > > > > >>> parquet-testing
> > > > > > > > >> >> >>> > > > > > >>> > >> repo :
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> >
> https://github.com/apache/parquet-testing/pull/100
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> And
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> - Go : Arnav just
> > > > submitted
> > > > > > an
> > > > > > > in
> > > > > > > > >> >> >>> progress
> > > > > > > > >> >> >>> > > > > > >>> implementation
> > > > > > > > >> >> >>> > > > > > >>> > in
> > > > > > > > >> >> >>> > > > > > >>> > >> Go.
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> https://github.com/apache/arrow-go/pull/704
> > > > > > > > >> >> >>> > (I
> > > > > > > > >> >> >>> > > > > > haven't
> > > > > > > > >> >> >>> > > > > > >>> > >> started
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> looking at it yet)
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> - Rust : I remember
> > > > Andrew
> > > > > > > > >> mentioned
> > > > > > > > >> >> that
> > > > > > > > >> >> >>> > this
> > > > > > > > >> >> >>> > > > work
> > > > > > > > >> >> >>> > > > > > is
> > > > > > > > >> >> >>> > > > > > >>> also
> > > > > > > > >> >> >>> > > > > > >>> > >> in
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> progress (So 4
> > > > languages!)
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> *Arrow C++
> > implementation *
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> The PR is out and was
> > also
> > > > used
> > > > > > by
> > > > > > > > >> >> Antoine to
> > > > > > > > >> >> >>> > > report
> > > > > > > > >> >> >>> > > > > the
> > > > > > > > >> >> >>> > > > > > >>> > numbers
> > > > > > > > >> >> >>> > > > > > >>> > >> as
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> reported here. Micah
> and
> > > > > > Konstantin
> > > > > > > > >> have
> > > > > > > > >> >> >>> given 1
> > > > > > > > >> >> >>> > > > round
> > > > > > > > >> >> >>> > > > > of
> > > > > > > > >> >> >>> > > > > > >>> > >> feedback
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> and I'm addressing them
> > > > today.
> > > > > > > Please
> > > > > > > > >> note
> > > > > > > > >> >> >>> that
> > > > > > > > >> >> >>> > the
> > > > > > > > >> >> >>> > > > > > default
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> optimization flag for
> > > > compiling
> > > > > > is
> > > > > > > O2
> > > > > > > > >> and
> > > > > > > > >> >> not
> > > > > > > > >> >> >>> > Q3. I
> > > > > > > > >> >> >>> > > > got
> > > > > > > > >> >> >>> > > > > > >>> around
> > > > > > > > >> >> >>> > > > > > >>> > >> 70%
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> performance improvement
> > in
> > > > the
> > > > > > > > decoding
> > > > > > > > >> >> speed
> > > > > > > > >> >> >>> > when
> > > > > > > > >> >> >>> > > > > using
> > > > > > > > >> >> >>> > > > > > >>> the O3
> > > > > > > > >> >> >>> > > > > > >>> > >> > flag.
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> *Parqet-MR Java
> > > > implementation
> > > > > > > > (working
> > > > > > > > >> >> with
> > > > > > > > >> >> >>> > Vinoo
> > > > > > > > >> >> >>> > > > and
> > > > > > > > >> >> >>> > > > > > >>> Julien)
> > > > > > > > >> >> >>> > > > > > >>> > >> and
> > > > > > > > >> >> >>> > > > > > >>> > >> > **Cross
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Language testing*
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Let me know if you
> > have
> > > > any
> > > > > > > > >> questions
> > > > > > > > >> >> or
> > > > > > > > >> >> >>> > > > feedback.
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Now pasting some
> > > performance
> > > > > > > numbers
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Table 1: C++ ALP
> > Double
> > > > > > Decode —
> > > > > > > > >> >> Spotify
> > > > > > > > >> >> >>> > Columns
> > > > > > > > >> >> >>> > > > > > >>> (Graviton
> > > > > > > > >> >> >>> > > > > > >>> > 3,
> > > > > > > > >> >> >>> > > > > > >>> > >> ARM
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Neoverse V1)
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>>
> > > > > > > > >> >> >>>
> > > > > ┌──────────────────┬──────────────┬──────────────┬─────────┐
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ Column │
> > > -O2
> > > > > > (MB/s)
> > > > > > > > │
> > > > > > > > >> >> -O3
> > > > > > > > >> >> >>> > > (MB/s) │
> > > > > > > > >> >> >>> > > > > > >>> Speedup │
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>>
> > > > > > > > >> >> >>>
> > > > > ├──────────────────┼──────────────┼──────────────┼─────────┤
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ valence │
> > > > > 3,155
> > > > > > > > │
> > > > > > > > >> >> >>> 5,523
> > > > > > > > >> >> >>> > > │
> > > > > > > > >> >> >>> > > > > > >>> 1.75x │
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ danceability │
> > > > > 3,233
> > > > > > > > │
> > > > > > > > >> >> >>> 5,685
> > > > > > > > >> >> >>> > > │
> > > > > > > > >> >> >>> > > > > > >>> 1.76x │
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ energy │
> > > > > 3,197
> > > > > > > > │
> > > > > > > > >> >> >>> 5,652
> > > > > > > > >> >> >>> > > │
> > > > > > > > >> >> >>> > > > > > >>> 1.77x │
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ loudness │
> > > > > 3,186
> > > > > > > > │
> > > > > > > > >> >> >>> 5,473
> > > > > > > > >> >> >>> > > │
> > > > > > > > >> >> >>> > > > > > >>> 1.72x │
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>>
> > > > > > > > >> >> >>>
> > > > > └──────────────────┴──────────────┴──────────────┴─────────┘
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> On Wed, Feb 25, 2026 at
> > > > 9:49 AM
> > > > > > > > PRATEEK
> > > > > > > > >> >> GAUR
> > > > > > > > >> >> >>> <
> > > > > > > > >> >> >>> > > > > > >>> > prtkgaur@gmail.com
> > > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> wrote:
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> @Micah Kornfield <
> > > > > > > > >> emkornfield@gmail.com>
> > > > > > > > >> >> :
> > > > > > > > >> >> >>> Got
> > > > > > > > >> >> >>> > > it.
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> @Andrew Lamb <
> > > > > > > > andrewlamb11@gmail.com>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> Do you think it would
> > be
> > > > good
> > > > > > to
> > > > > > > > >> start
> > > > > > > > >> >> >>> moving
> > > > > > > > >> >> >>> > the
> > > > > > > > >> >> >>> > > > > spec
> > > > > > > > >> >> >>> > > > > > >>> > >> development
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> into
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> markdown format, in
> > > > > preparation
> > > > > > > for
> > > > > > > > >> >> >>> finalizing
> > > > > > > > >> >> >>> > > it?
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> Yes I'll update the
> > > numbers
> > > > > for
> > > > > > > some
> > > > > > > > >> of
> > > > > > > > >> >> the
> > > > > > > > >> >> >>> > > > examples I
> > > > > > > > >> >> >>> > > > > > >>> have in
> > > > > > > > >> >> >>> > > > > > >>> > >> the
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> spec based
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> on the updated header
> > > size.
> > > > > Then
> > > > > > > we
> > > > > > > > >> >> should
> > > > > > > > >> >> >>> be
> > > > > > > > >> >> >>> > good
> > > > > > > > >> >> >>> > > > to
> > > > > > > > >> >> >>> > > > > go
> > > > > > > > >> >> >>> > > > > > >>> for
> > > > > > > > >> >> >>> > > > > > >>> > the
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> markdown format.
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> Thanks everyone!
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> Andrew
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> On Tue, Feb 17, 2026
> at
> > > > > 7:28 PM
> > > > > > > > >> PRATEEK
> > > > > > > > >> >> >>> GAUR <
> > > > > > > > >> >> >>> > > > > > >>> > >> prtkgaur@gmail.com>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> wrote:
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Hi team,
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 1) Andrew
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for
> > working
> > > > on
> > > > > > test
> > > > > > > > >> files.
> > > > > > > > >> >> >>> My PR
> > > > > > > > >> >> >>> > > did
> > > > > > > > >> >> >>> > > > > add
> > > > > > > > >> >> >>> > > > > > >>> all
> > > > > > > > >> >> >>> > > > > > >>> > the
> > > > > > > > >> >> >>> > > > > > >>> > >> > test
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> files I
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> used to
> benchmark
> > on
> > > > > > > datasets.
> > > > > > > > >> >> Maybe
> > > > > > > > >> >> >>> we
> > > > > > > > >> >> >>> > can
> > > > > > > > >> >> >>> > > > club
> > > > > > > > >> >> >>> > > > > > it
> > > > > > > > >> >> >>> > > > > > >>> > >> together.
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> WIll also
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> aid
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> cross language
> > > testing
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Kosta Tarasov
> > > > working
> > > > > > on
> > > > > > > > Rust
> > > > > > > > >> >> >>> > > > implementation.
> > > > > > > > >> >> >>> > > > > > >>> This is
> > > > > > > > >> >> >>> > > > > > >>> > >> > great.
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> Thanks
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 2) Antoine
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks a lot
> for
> > > > > > reporting
> > > > > > > > the
> > > > > > > > >> >> >>> numbers
> > > > > > > > >> >> >>> > on
> > > > > > > > >> >> >>> > > > AMD.
> > > > > > > > >> >> >>> > > > > > >>> Looks
> > > > > > > > >> >> >>> > > > > > >>> > >> like
> > > > > > > > >> >> >>> > > > > > >>> > >> > you
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> are
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> getting 8X the
> > > > decoding
> > > > > > > > >> >> performance of
> > > > > > > > >> >> >>> > BSS.
> > > > > > > > >> >> >>> > > > This
> > > > > > > > >> >> >>> > > > > > is
> > > > > > > > >> >> >>> > > > > > >>> > >> > amazing!!.
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for
> > > > > acknowledging
> > > > > > > the
> > > > > > > > >> >> >>> sampling
> > > > > > > > >> >> >>> > > > design.
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - I agree with
> you
> > > on
> > > > > > > > >> Fastlanes. In
> > > > > > > > >> >> >>> some
> > > > > > > > >> >> >>> > > crude
> > > > > > > > >> >> >>> > > > > > >>> > >> experiments I
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> didn't get
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> a good perf
> > benefit
> > > > from
> > > > > > it
> > > > > > > on
> > > > > > > > >> >> >>> Graviton3
> > > > > > > > >> >> >>> > > (but
> > > > > > > > >> >> >>> > > > > > maybe
> > > > > > > > >> >> >>> > > > > > >>> > there
> > > > > > > > >> >> >>> > > > > > >>> > >> was
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> something
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> wrong with my
> > > > > > > implementation).
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Locking the
> > 16bit
> > > > > > > exception
> > > > > > > > >> >> >>> encoding for
> > > > > > > > >> >> >>> > > the
> > > > > > > > >> >> >>> > > > > > spec
> > > > > > > > >> >> >>> > > > > > >>> in
> > > > > > > > >> >> >>> > > > > > >>> > >> this
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> case.
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Awesome I
> think
> > we
> > > > > have
> > > > > > > > solved
> > > > > > > > >> >> for
> > > > > > > > >> >> >>> all
> > > > > > > > >> >> >>> > > open
> > > > > > > > >> >> >>> > > > > > >>> questions
> > > > > > > > >> >> >>> > > > > > >>> > >> minus
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> the
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> version byte :).
> > > (will
> > > > > get
> > > > > > > > back
> > > > > > > > >> on
> > > > > > > > >> >> >>> this
> > > > > > > > >> >> >>> > > soon)
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 3) Micah
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - FastLanes :
> The
> > > > > current
> > > > > > > spec
> > > > > > > > >> does
> > > > > > > > >> >> >>> allow
> > > > > > > > >> >> >>> > > for
> > > > > > > > >> >> >>> > > > > > using
> > > > > > > > >> >> >>> > > > > > >>> > >> FastLane
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> with the
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> configurable
> enum
> > > > value
> > > > > > for
> > > > > > > > >> >> layout. We
> > > > > > > > >> >> >>> > > should
> > > > > > > > >> >> >>> > > > be
> > > > > > > > >> >> >>> > > > > > >>> able to
> > > > > > > > >> >> >>> > > > > > >>> > >> > inject
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> any
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> layout
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> in the current
> > > design.
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Working on resolving
> > all
> > > > > > > remaining
> > > > > > > > >> open
> > > > > > > > >> >> >>> > comments
> > > > > > > > >> >> >>> > > > on
> > > > > > > > >> >> >>> > > > > > the
> > > > > > > > >> >> >>> > > > > > >>> spec
> > > > > > > > >> >> >>> > > > > > >>> > >> this
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> week.
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Best
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Prateek
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> On Tue, Feb 10, 2026
> > at
> > > > > > 3:37 AM
> > > > > > > > >> Steve
> > > > > > > > >> >> >>> > Loughran <
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> stevel@cloudera.com>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> wrote:
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026
> at
> > > > > 18:12,
> > > > > > > > Micah
> > > > > > > > >> >> >>> Kornfield
> > > > > > > > >> >> >>> > <
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> emkornfield@gmail.com>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> wrote:
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> It looks like the
> > > actual
> > > > > > issue
> > > > > > > > >> >> >>> described for
> > > > > > > > >> >> >>> > > ORC
> > > > > > > > >> >> >>> > > > > in
> > > > > > > > >> >> >>> > > > > > >>> the
> > > > > > > > >> >> >>> > > > > > >>> > >> paper
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> is that
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> it
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> has multiple
> > > > sub-encodings
> > > > > > in
> > > > > > > a
> > > > > > > > >> >> batch.
> > > > > > > > >> >> >>> This
> > > > > > > > >> >> >>> > > is
> > > > > > > > >> >> >>> > > > > > >>> different
> > > > > > > > >> >> >>> > > > > > >>> > >> then
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> the
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> design
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> proposed here
> where
> > > > there
> > > > > is
> > > > > > > > still
> > > > > > > > >> >> fixed
> > > > > > > > >> >> >>> > > > encoding
> > > > > > > > >> >> >>> > > > > > per
> > > > > > > > >> >> >>> > > > > > >>> page
> > > > > > > > >> >> >>> > > > > > >>> > >> in
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> parquet.
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> Given reasonably
> > sized
> > > > > > pages I
> > > > > > > > >> don't
> > > > > > > > >> >> >>> think
> > > > > > > > >> >> >>> > > > branch
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> misprediction should
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> be a
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> big issue for new
> > > > > encodings.
> > > > > > > I
> > > > > > > > >> agree
> > > > > > > > >> >> >>> that
> > > > > > > > >> >> >>> > we
> > > > > > > > >> >> >>> > > > > should
> > > > > > > > >> >> >>> > > > > > >>> be
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> conservative in
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> general for adding
> > new
> > > > > > > > encodings.
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> +1
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > > > > >> >> >>> > > > > > >>> > >> > >
> > > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > > > >> >> >>> > > > > > >>> > >
> > > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > > >> >> >>> > > > > > >>>
> > > > > > > > >> >> >>> > > > > > >>
> > > > > > > > >> >> >>> > > > > >
> > > > > > > > >> >> >>> > > > >
> > > > > > > > >> >> >>> > > >
> > > > > > > > >> >> >>> > >
> > > > > > > > >> >> >>> >
> > > > > > > > >> >> >>>
> > > > > > > > >> >> >>
> > > > > > > > >> >>
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Vinoo Ganesh <vi...@gmail.com>.
Thanks everyone!
Thanks,
Vinoo Ganesh | vinoo.ganesh@gmail.com
<Vi...@gmail.com>
On Mon, Nov 9, 2026 at 2:49 PM Andrew Lamb <an...@gmail.com> wrote:
> The Java implementation was just merged[1]!
>
> Thank you Gang, Vinoo, Prateek and everyone who helped push that one along
>
> Andrew
>
> [1]: https://github.com/apache/parquet-java/pull/3397#event-35545451335
>
> On Tue, Oct 13, 2026 at 12:52 PM PRATEEK GAUR <pr...@gmail.com> wrote:
>
> > Hi team,
> >
> > Quick update re: the C++ implementation.
> > Thanks to all the reviewers, we merged the C++ implementation today.
> > There are a few follow-up items that I'll address in a subsequent PR.
> >
> > Thanks
> > Prateek
> >
> > On Sun, Sep 13, 2026 at 6:54 PM Vinoo Ganesh <vi...@gmail.com>
> > wrote:
> >
> > > Quick update re: the Java impl. Thanks for the feedback, on
> > > https://github.com/apache/parquet-java/pull/3397, Russel! I pulled
> > > https://github.com/apache/parquet-java/pull/3791 into its own PR
> (thanks
> > > for the quick approval too, Gang). Assuming no objections, I think we
> can
> > > merge #3791, and then go ahead of #3397.
> > >
> > > Thanks,
> > > Vinoo Ganesh | vinoo.ganesh@gmail.com
> > >
> > > <Vi...@gmail.com>
> > >
> > >
> > > On Tue, Sep 8, 2026 at 10:54 AM PRATEEK GAUR <pr...@gmail.com>
> wrote:
> > >
> > > > [Rust] Awesome thanks Andrew, Kostas and Devin.
> > > > [Java] Thanks Gang for the feedback on Java PR.
> > > > [C++] Thanks Gang for another round of feedback on the C++ one.
> > > > [Go] Arnav and I will be working on the golang impl this week.
> > > >
> > > > This is great progress all around.
> > > >
> > > > On Sat, Sep 5, 2026 at 1:26 PM Vinoo Ganesh <vi...@gmail.com>
> > > > wrote:
> > > >
> > > > > Thanks all - I'm addressing Gang's comments on
> > > > > https://github.com/apache/parquet-java/pull/3397 now, so hopefully
> > > well
> > > > > also be in a good state for merging the java implementation soon.
> > > > >
> > > > > Thanks,
> > > > > Vinoo Ganesh | vinoo.ganesh@gmail.com
> > > > >
> > > > > <Vi...@gmail.com>
> > > > >
> > > > >
> > > > > On Sat, Sep 5, 2026 at 8:56 AM Andrew Lamb <andrewlamb11@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > We just merged the Rust implementation[1] from Kosta and Devan.
> > With
> > > > the
> > > > > > imminent release of Parquet Format 2.14.0 I think we'll be well
> on
> > > our
> > > > > way.
> > > > > >
> > > > > > Thanks again for pushing this along,
> > > > > > Andrew
> > > > > >
> > > > > > [1]: https://github.com/apache/arrow-rs/pull/9372
> > > > > >
> > > > > > On Fri, Sep 4, 2026 at 5:42 PM PRATEEK GAUR <pr...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > Hi team,
> > > > > > >
> > > > > > > Thanks Micah for multiple rounds of review and thanks Gang for
> > > > taking a
> > > > > > > look at the PR and giving feedback. I've addressed all comments
> > in
> > > > the
> > > > > > PR.
> > > > > > > Looking forward to the next round of feedback (or approval).
> > Would
> > > > like
> > > > > > to
> > > > > > > get it in soon.
> > > > > > >
> > > > > > > https://github.com/apache/arrow/pull/48345/changes
> > > > > > >
> > > > > > > With this done and after talking to Arnav, I plan to spend some
> > > time
> > > > on
> > > > > > the
> > > > > > > ALP implementation in the arrow-go repository helping spread
> the
> > > > > support
> > > > > > to
> > > > > > > more languages.
> > > > > > >
> > > > > > > Best
> > > > > > > Prateek
> > > > > > >
> > > > > > > On Sun, Aug 2, 2026 at 10:42 PM PRATEEK GAUR <
> prtkgaur@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Thanks Divjot for bringing it up.
> > > > > > > >
> > > > > > > > I was under the same impression as Russell and Andrew that
> we'd
> > > > have
> > > > > it
> > > > > > > > flag gated.
> > > > > > > > With the new versioning proposal 'preview' might be a good
> tag
> > > for
> > > > > it.
> > > > > > > >
> > > > > > > > Best
> > > > > > > > Prateek
> > > > > > > >
> > > > > > > > On Sat, Aug 1, 2026 at 12:39 AM Andrew Lamb <
> > > > andrewlamb11@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Div: Here is a proposal to mark ALP in Preview: [1] (using a
> > > > comment
> > > > > > in
> > > > > > > >> the
> > > > > > > >> spec section).
> > > > > > > >>
> > > > > > > >> Russel: That is my understanding as well.
> > > > > > > >>
> > > > > > > >> Hopefully we are able to formalize the versioning process
> > > shortly
> > > > so
> > > > > > we
> > > > > > > >> can
> > > > > > > >> formalize the notion of "preview"
> > > > > > > >>
> > > > > > > >> Andrew
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> [1]: https://github.com/apache/parquet-format/pull/604
> > > > > > > >>
> > > > > > > >> On Fri, Jul 31, 2026 at 11:15 AM Russell Spitzer <
> > > > > > > >> russell.spitzer@gmail.com>
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >> > I thought we had agreed it would be merged, and features
> > like
> > > > this
> > > > > > > would
> > > > > > > >> > be feature flag gated. So a V-Old writer could always opt
> in
> > > to
> > > > > > using
> > > > > > > >> this
> > > > > > > >> > encoding but that must be a conscious effort if you are
> not
> > > > using
> > > > > > > VNext
> > > > > > > >> of
> > > > > > > >> > Parquet. If you are on V-Next the writer can choose to use
> > ALP
> > > > > > > whenever
> > > > > > > >> it
> > > > > > > >> > likes without user intervention.
> > > > > > > >> >
> > > > > > > >> > On Fri, Jul 31, 2026 at 2:20 AM Divjot Arora via dev <
> > > > > > > >> > dev@parquet.apache.org> wrote:
> > > > > > > >> >
> > > > > > > >> >> Yes, that’s my understanding as well. However, as you
> > noted,
> > > > the
> > > > > > > >> >> versioning
> > > > > > > >> >> discussion is still in early stages and hasn’t prescribed
> > > > > concrete
> > > > > > > >> >> mechanisms yet. If ALP goes into parquet-format 2.14.0,
> > > there’s
> > > > > no
> > > > > > > >> >> indication that it’s in preview or that writers shouldn’t
> > use
> > > > it
> > > > > by
> > > > > > > >> >> default
> > > > > > > >> >> yet. Given that we’ve held off on merging other
> > incompatible
> > > > > > changes
> > > > > > > >> (e.g
> > > > > > > >> >> removal of path_in_schema) until the versioning
> discussion
> > > > > closes,
> > > > > > my
> > > > > > > >> >> preference would be to leave ALP un-merged for the time
> > being
> > > > as
> > > > > > > well.
> > > > > > > >> >>
> > > > > > > >> >> — Div
> > > > > > > >> >>
> > > > > > > >> >> On Fri, Jul 31, 2026 at 08:59 Andrew Lamb <
> > > > > andrewlamb11@gmail.com>
> > > > > > > >> wrote:
> > > > > > > >> >>
> > > > > > > >> >> > After re-reading Julien's Versioning Proposal, my
> > > > understanding
> > > > > > is
> > > > > > > >> that
> > > > > > > >> >> if
> > > > > > > >> >> > we adopt that system, we will somehow mark ALP in
> > "Preview"
> > > > in
> > > > > > > >> >> > parquet-format for release Version 3.
> > > > > > > >> >> >
> > > > > > > >> >> > The versioning document doesn't seem to proscribe the
> > > > > particular
> > > > > > > >> method
> > > > > > > >> >> > for annotating "preview" features -- I think clearly
> > > marking
> > > > > them
> > > > > > > as
> > > > > > > >> >> such
> > > > > > > >> >> > in parquet.thrift as well as reflecting that on the
> > > > > > > >> >> > https://parquet.apache.org/docs/file-format/versions/
> > page
> > > > > > somehow
> > > > > > > >> >> would
> > > > > > > >> >> > be adequate.
> > > > > > > >> >> >
> > > > > > > >> >> > Andrew
> > > > > > > >> >> >
> > > > > > > >> >> > On Thu, Jul 30, 2026 at 11:50 AM Andrew Lamb <
> > > > > > > andrewlamb11@gmail.com
> > > > > > > >> >
> > > > > > > >> >> > wrote:
> > > > > > > >> >> >
> > > > > > > >> >> >> In my mind ALP is in the same position as other
> > encodings
> > > > that
> > > > > > > were
> > > > > > > >> >> added
> > > > > > > >> >> >> recently[1] (such as BYTE_STREAM_SPLIT for non
> floating
> > > > point
> > > > > > > types)
> > > > > > > >> >> >>
> > > > > > > >> >> >> That is to say it will be available in the latest open
> > > > source
> > > > > > > >> writers
> > > > > > > >> >> and
> > > > > > > >> >> >> readers, but realistically not widely used in the
> > > ecosystem
> > > > > > where
> > > > > > > >> >> >> compatibility is a concern (a large part)
> > > > > > > >> >> >>
> > > > > > > >> >> >> I view Julian's work[2][3] to gather consensus on how
> to
> > > > name
> > > > > > > >> >> >> combinations of features (e.g. "versions") as the
> > > mechanism
> > > > by
> > > > > > > >> which we
> > > > > > > >> >> >> will accelerate the adoption of this change (along
> with
> > > > other
> > > > > > > >> forward
> > > > > > > >> >> >> incompatible changes) across the ecosystem.
> > > > > > > >> >> >>
> > > > > > > >> >> >> > Will we exclude this change when cutting
> > parquet-format
> > > > > > v2.14.0?
> > > > > > > >> >> >>
> > > > > > > >> >> >> No I don't think so -- I would expect it to go out in
> > the
> > > > next
> > > > > > > >> >> >> parquet-format version as do other changes in the
> spec.
> > > > > > > >> >> >>
> > > > > > > >> >> >> Andrew
> > > > > > > >> >> >>
> > > > > > > >> >> >> [1]:
> > > https://parquet.apache.org/docs/file-format/versions/
> > > > > > > >> >> >> [2]:
> > > > > > > >>
> > > https://lists.apache.org/thread/xmnj8h0h8ozmrgox5tydhhs7yz7q2hss
> > > > > > > >> >> >> [3]:
> > > > > > > >>
> > > https://lists.apache.org/thread/qlf8lg90gqq41lllyy9mk2f0skqvftqj
> > > > > > > >> >> >>
> > > > > > > >> >> >>
> > > > > > > >> >> >> On Thu, Jul 30, 2026 at 3:51 AM Divjot Arora via dev <
> > > > > > > >> >> >> dev@parquet.apache.org> wrote:
> > > > > > > >> >> >>
> > > > > > > >> >> >>> Hi Andrew,
> > > > > > > >> >> >>>
> > > > > > > >> >> >>> How does this interact with the ongoing versioning
> > > > > discussion?
> > > > > > > >> Given
> > > > > > > >> >> that
> > > > > > > >> >> >>> new encodings are not forward compatible, I would
> > assume
> > > > ALP
> > > > > is
> > > > > > > >> >> included
> > > > > > > >> >> >>> in
> > > > > > > >> >> >>> Parquet V<next> (V3?), but it's not clear to me how
> > that
> > > > > works
> > > > > > if
> > > > > > > >> it's
> > > > > > > >> >> >>> merged to the parquet-format master branch now. Will
> we
> > > > > exclude
> > > > > > > >> this
> > > > > > > >> >> >>> change
> > > > > > > >> >> >>> when cutting parquet-format v2.14.0?
> > > > > > > >> >> >>>
> > > > > > > >> >> >>> -- Div
> > > > > > > >> >> >>>
> > > > > > > >> >> >>> On Thu, Jul 30, 2026 at 8:51 AM Andrew Lamb <
> > > > > > > >> andrewlamb11@gmail.com>
> > > > > > > >> >> >>> wrote:
> > > > > > > >> >> >>>
> > > > > > > >> >> >>> > Hello,
> > > > > > > >> >> >>> >
> > > > > > > >> >> >>> > As an update, thanks to the epic work from many
> > > > > contributors,
> > > > > > > >> >> >>> especially
> > > > > > > >> >> >>> > Prateek, I just merged the parquet-format change
> for
> > > > > ALP[1].
> > > > > > We
> > > > > > > >> >> plan to
> > > > > > > >> >> >>> > make additional editorial The remaining items I
> know
> > of
> > > > are
> > > > > > > >> merging
> > > > > > > >> >> the
> > > > > > > >> >> >>> > examples from parquet-testing[2], and then of
> course
> > > the
> > > > > > > various
> > > > > > > >> >> >>> > implementations
> > > > > > > >> >> >>> >
> > > > > > > >> >> >>> > Thanks again everyone. This is a great step forward
> > > > > > > >> >> >>> > Andrew
> > > > > > > >> >> >>> >
> > > > > > > >> >> >>> > [1]:
> > https://github.com/apache/parquet-format/pull/557
> > > > > > > >> >> >>> > [2]:
> > > https://github.com/apache/parquet-testing/pull/100
> > > > > > > >> >> >>> >
> > > > > > > >> >> >>> > On Tue, Jul 21, 2026 at 4:39 PM Julien Le Dem <
> > > > > > > julien@apache.org
> > > > > > > >> >
> > > > > > > >> >> >>> wrote:
> > > > > > > >> >> >>> >
> > > > > > > >> >> >>> > > It was also in my spam folder... :(
> > > > > > > >> >> >>> > > I've seen a bunch of apache list emails go to my
> > spam
> > > > > > > recently.
> > > > > > > >> >> >>> > >
> > > > > > > >> >> >>> > > On Tue, Jul 21, 2026 at 10:58 AM Russell Spitzer
> <
> > > > > > > >> >> >>> > > russell.spitzer@gmail.com>
> > > > > > > >> >> >>> > > wrote:
> > > > > > > >> >> >>> > >
> > > > > > > >> >> >>> > > > Yes, my hope was that my vote would kick it out
> > of
> > > > spam
> > > > > > but
> > > > > > > >> we
> > > > > > > >> >> >>> > definitely
> > > > > > > >> >> >>> > > > had two other votes that also went to spam for
> > me.
> > > > > > > >> >> >>> > > >
> > > > > > > >> >> >>> > > > For anyone looking the thread is here
> > > > > > > >> >> >>> > > >
> > > > > > > >> >>
> > > > https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt
> > > > > > > >> >> >>> > > >
> > > > > > > >> >> >>> > > > On Tue, Jul 21, 2026 at 10:57 AM Micah
> Kornfield
> > <
> > > > > > > >> >> >>> > emkornfield@gmail.com>
> > > > > > > >> >> >>> > > > wrote:
> > > > > > > >> >> >>> > > >
> > > > > > > >> >> >>> > > > > I've seen other's votes not go to spam, so
> > > > hopefully
> > > > > > > people
> > > > > > > >> >> will
> > > > > > > >> >> >>> see
> > > > > > > >> >> >>> > it
> > > > > > > >> >> >>> > > > in
> > > > > > > >> >> >>> > > > > there inboxes (or see this thread and look in
> > > there
> > > > > > > spam).
> > > > > > > >> >> >>> > > > >
> > > > > > > >> >> >>> > > > > On Tue, Jul 21, 2026 at 10:52 AM PRATEEK
> GAUR <
> > > > > > > >> >> >>> prtkgaur@gmail.com>
> > > > > > > >> >> >>> > > > wrote:
> > > > > > > >> >> >>> > > > >
> > > > > > > >> >> >>> > > > > > Okay,
> > > > > > > >> >> >>> > > > > >
> > > > > > > >> >> >>> > > > > > I sent out the voting email yesterday and
> so
> > > far
> > > > > I've
> > > > > > > >> heard
> > > > > > > >> >> >>> from 3
> > > > > > > >> >> >>> > > > people
> > > > > > > >> >> >>> > > > > > that it ended up in their "spam folder".
> Dev
> > > VPN
> > > > to
> > > > > > be
> > > > > > > >> >> blamed
> > > > > > > >> >> >>> for
> > > > > > > >> >> >>> > > it?.
> > > > > > > >> >> >>> > > > > >
> > > > > > > >> >> >>> > > > > > What's the protocol here?
> > > > > > > >> >> >>> > > > > > I let people hunt for the email in their
> spam
> > > > > folder
> > > > > > or
> > > > > > > >> send
> > > > > > > >> >> >>> out
> > > > > > > >> >> >>> > > > another
> > > > > > > >> >> >>> > > > > > one?
> > > > > > > >> >> >>> > > > > >
> > > > > > > >> >> >>> > > > > > Best
> > > > > > > >> >> >>> > > > > > Prateek
> > > > > > > >> >> >>> > > > > >
> > > > > > > >> >> >>> > > > > > On Sun, Jul 19, 2026 at 10:47 PM PRATEEK
> > GAUR <
> > > > > > > >> >> >>> prtkgaur@gmail.com>
> > > > > > > >> >> >>> > > > > wrote:
> > > > > > > >> >> >>> > > > > >
> > > > > > > >> >> >>> > > > > > > Hi team,
> > > > > > > >> >> >>> > > > > > >
> > > > > > > >> >> >>> > > > > > > Vinoo has added the tests which were
> > > requested
> > > > in
> > > > > > the
> > > > > > > >> last
> > > > > > > >> >> >>> round
> > > > > > > >> >> >>> > of
> > > > > > > >> >> >>> > > > > > review.
> > > > > > > >> >> >>> > > > > > > In his own words.
> > > > > > > >> >> >>> > > > > > >
> > > > > > > >> >> >>> > > > > > > "
> > > > > > > >> >> >>> > > > > > > The extreme value tests Micah wanted are
> > done
> > > > and
> > > > > > > >> pushed.
> > > > > > > >> >> I
> > > > > > > >> >> >>> added
> > > > > > > >> >> >>> > > > > > coverage
> > > > > > > >> >> >>> > > > > > > for values that need the full FOR bit
> width
> > > > after
> > > > > > the
> > > > > > > >> >> frame
> > > > > > > >> >> >>> of
> > > > > > > >> >> >>> > > > > reference
> > > > > > > >> >> >>> > > > > > is
> > > > > > > >> >> >>> > > > > > > applied: 64-bit (the case where the
> signed
> > > max
> > > > > > minus
> > > > > > > >> min
> > > > > > > >> >> >>> > > subtraction
> > > > > > > >> >> >>> > > > > > > overflows), 63-bit (the non-overflow
> case),
> > > and
> > > > > > > 32-bit
> > > > > > > >> for
> > > > > > > >> >> >>> > floats.
> > > > > > > >> >> >>> > > > All
> > > > > > > >> >> >>> > > > > > > lossless with zero exceptions. I also
> > > verified
> > > > > them
> > > > > > > >> >> through
> > > > > > > >> >> >>> Arrow
> > > > > > > >> >> >>> > > C++
> > > > > > > >> >> >>> > > > > ALP
> > > > > > > >> >> >>> > > > > > > reader (the local build), so the
> > Java-written
> > > > > > extreme
> > > > > > > >> >> columns
> > > > > > > >> >> >>> > > decode
> > > > > > > >> >> >>> > > > > > > correctly in C++, bit-exact against the
> > > > expected
> > > > > > > >> values."
> > > > > > > >> >> >>> > > > > > >
> > > > > > > >> >> >>> > > > > > > State of the PR (#3397):
> > > > > > > >> >> >>> > > > > > >
> > > > > > > >> >> >>> > > > > > > - Merges cleanly with master (I pulled
> > > > master
> > > > > in
> > > > > > > and
> > > > > > > >> >> >>> resolved
> > > > > > > >> >> >>> > > the
> > > > > > > >> >> >>> > > > > > > conflicts).
> > > > > > > >> >> >>> > > > > > > - 41 of 44 review threads resolved.
> The
> > 3
> > > > > > > remaining
> > > > > > > >> are
> > > > > > > >> >> >>> > > > intentional:
> > > > > > > >> >> >>> > > > > > > one deferred perf nit, one that stays
> > open
> > > > > until
> > > > > > > >> >> >>> > parquet-format
> > > > > > > >> >> >>> > > > > ships
> > > > > > > >> >> >>> > > > > > ALP,
> > > > > > > >> >> >>> > > > > > >
> > > > > > > >> >> >>> > > > > > > "
> > > > > > > >> >> >>> > > > > > >
> > > > > > > >> >> >>> > > > > > > With this both the
> > > > > > > >> >> >>> > > > > > > 1) C++ PR :
> > > > > > > >> >> >>> https://github.com/apache/arrow/pull/48345/changes
> > > > > > > >> >> >>> > > > > > > 2) Java PR :
> > > > > > > >> >> >>> https://github.com/apache/parquet-java/pull/3397
> > > > > > > >> >> >>> > > > > > >
> > > > > > > >> >> >>> > > > > > > Are cross language tested and range of
> test
> > > > > > > scenarios.
> > > > > > > >> >> >>> > > > > > >
> > > > > > > >> >> >>> > > > > > > Best
> > > > > > > >> >> >>> > > > > > > Prateek
> > > > > > > >> >> >>> > > > > > >
> > > > > > > >> >> >>> > > > > > >
> > > > > > > >> >> >>> > > > > > > On Mon, Jul 13, 2026 at 8:38 AM PRATEEK
> > GAUR
> > > <
> > > > > > > >> >> >>> prtkgaur@gmail.com
> > > > > > > >> >> >>> > >
> > > > > > > >> >> >>> > > > > wrote:
> > > > > > > >> >> >>> > > > > > >
> > > > > > > >> >> >>> > > > > > >> Thanks Matt,
> > > > > > > >> >> >>> > > > > > >>
> > > > > > > >> >> >>> > > > > > >> I'll be on it this week.
> > > > > > > >> >> >>> > > > > > >>
> > > > > > > >> >> >>> > > > > > >> Best
> > > > > > > >> >> >>> > > > > > >> Prateek
> > > > > > > >> >> >>> > > > > > >>
> > > > > > > >> >> >>> > > > > > >> On Mon, Jul 13, 2026 at 8:36 AM Matt
> > Topol <
> > > > > > > >> >> >>> > > zotthewizard@gmail.com>
> > > > > > > >> >> >>> > > > > > >> wrote:
> > > > > > > >> >> >>> > > > > > >>
> > > > > > > >> >> >>> > > > > > >>> There is also the Go implementation
> > > submitted
> > > > > by
> > > > > > > >> Arnav (
> > > > > > > >> >> >>> > > > > > >>>
> > > https://github.com/apache/arrow-go/pull/704)
> > > > > > which
> > > > > > > >> is
> > > > > > > >> >> >>> waiting
> > > > > > > >> >> >>> > > for
> > > > > > > >> >> >>> > > > > > >>> updates.
> > > > > > > >> >> >>> > > > > > >>>
> > > > > > > >> >> >>> > > > > > >>> Just wanted to make sure it didn't get
> > lost
> > > > > here.
> > > > > > > >> >> >>> > > > > > >>>
> > > > > > > >> >> >>> > > > > > >>> --Matt
> > > > > > > >> >> >>> > > > > > >>>
> > > > > > > >> >> >>> > > > > > >>> On Mon, Jul 13, 2026 at 11:30 AM
> PRATEEK
> > > > GAUR <
> > > > > > > >> >> >>> > > prtkgaur@gmail.com>
> > > > > > > >> >> >>> > > > > > >>> wrote:
> > > > > > > >> >> >>> > > > > > >>>
> > > > > > > >> >> >>> > > > > > >>> > Hi all,
> > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > >> >> >>> > > > > > >>> > I'd like to share a status update on
> > the
> > > > ALP
> > > > > > > >> encoding
> > > > > > > >> >> >>> effort
> > > > > > > >> >> >>> > > and
> > > > > > > >> >> >>> > > > > get
> > > > > > > >> >> >>> > > > > > a
> > > > > > > >> >> >>> > > > > > >>> feel
> > > > > > > >> >> >>> > > > > > >>> > from the community before starting a
> > > formal
> > > > > > vote.
> > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > >> >> >>> > > > > > >>> > We now have the specification plus
> > > > > > > implementations
> > > > > > > >> in
> > > > > > > >> >> >>> both
> > > > > > > >> >> >>> > > > > languages:
> > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > >> >> >>> > > > > > >>> > - Spec:
> > > > > apache/parquet-format#557 —
> > > > > > > >> GH-533
> > > > > > > >> >> >>> Add
> > > > > > > >> >> >>> > ALP
> > > > > > > >> >> >>> > > > > > encoding
> > > > > > > >> >> >>> > > > > > >>> > specification
> > > > > > > >> >> >>> > > > > > >>> > - C++ (Arrow):
> apache/arrow#48345
> > —
> > > > > > GH-48701
> > > > > > > >> >> >>> > > [C++][Parquet]
> > > > > > > >> >> >>> > > > > Add
> > > > > > > >> >> >>> > > > > > >>> ALPpd
> > > > > > > >> >> >>> > > > > > >>> > encoding
> > > > > > > >> >> >>> > > > > > >>> > - Java:
> > > > apache/parquet-java#3397
> > > > > —
> > > > > > > >> Parquet
> > > > > > > >> >> >>> Java
> > > > > > > >> >> >>> > ALP
> > > > > > > >> >> >>> > > > > > >>> > Implementation (by Vinoo)
> > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > >> >> >>> > > > > > >>> > Current state:
> > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > >> >> >>> > > > > > >>> > - The C++ implementation has been
> > > > through
> > > > > > 3-4
> > > > > > > >> >> rounds
> > > > > > > >> >> >>> of
> > > > > > > >> >> >>> > > > review.
> > > > > > > >> >> >>> > > > > > >>> > - The Java implementation has
> been
> > > > > tracking
> > > > > > > >> well
> > > > > > > >> >> and
> > > > > > > >> >> >>> has
> > > > > > > >> >> >>> > > had
> > > > > > > >> >> >>> > > > > > >>> initial
> > > > > > > >> >> >>> > > > > > >>> > reviews from contributors, with
> > no
> > > > > major
> > > > > > > >> >> remaining
> > > > > > > >> >> >>> > > issues.
> > > > > > > >> >> >>> > > > > > >>> > - Cross-language compatibility
> > tests
> > > > are
> > > > > > > >> passing:
> > > > > > > >> >> the
> > > > > > > >> >> >>> > Arrow
> > > > > > > >> >> >>> > > > C++
> > > > > > > >> >> >>> > > > > > >>> decoder
> > > > > > > >> >> >>> > > > > > >>> > reads Java-written data
> > bit-exactly
> > > > > > across
> > > > > > > >> >> ~1.56M
> > > > > > > >> >> >>> > values
> > > > > > > >> >> >>> > > > and
> > > > > > > >> >> >>> > > > > 18
> > > > > > > >> >> >>> > > > > > >>> > fixtures, covering V1 and V2
> > pages,
> > > > > > > multiple
> > > > > > > >> >> vector
> > > > > > > >> >> >>> > > sizes,
> > > > > > > >> >> >>> > > > > and
> > > > > > > >> >> >>> > > > > > >>> > several
> > > > > > > >> >> >>> > > > > > >>> > real datasets — zero
> mismatches.
> > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > >> >> >>> > > > > > >>> > As raised in the java-pr we will be
> > > adding
> > > > > > > >> coverage
> > > > > > > >> >> for
> > > > > > > >> >> >>> > > extreme
> > > > > > > >> >> >>> > > > > > values
> > > > > > > >> >> >>> > > > > > >>> > (those requiring
> > > > > > > >> >> >>> > > > > > >>> > 63-64 bits after FOR is applied)
> > before
> > > we
> > > > > > > close a
> > > > > > > >> >> vote.
> > > > > > > >> >> >>> > We'll
> > > > > > > >> >> >>> > > > aim
> > > > > > > >> >> >>> > > > > > to
> > > > > > > >> >> >>> > > > > > >>> get
> > > > > > > >> >> >>> > > > > > >>> > that
> > > > > > > >> >> >>> > > > > > >>> > done in parallel.
> > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > >> >> >>> > > > > > >>> > Plan: unless there are objections, I
> > > intend
> > > > > to
> > > > > > > >> start a
> > > > > > > >> >> >>> vote
> > > > > > > >> >> >>> > at
> > > > > > > >> >> >>> > > > the
> > > > > > > >> >> >>> > > > > > end
> > > > > > > >> >> >>> > > > > > >>> of
> > > > > > > >> >> >>> > > > > > >>> > this week/early
> > > > > > > >> >> >>> > > > > > >>> > next week.
> > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > >> >> >>> > > > > > >>> > Thanks
> > > > > > > >> >> >>> > > > > > >>> > Prateek
> > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > >> >> >>> > > > > > >>> > On Wed, Jul 1, 2026 at 8:50 AM
> PRATEEK
> > > > GAUR <
> > > > > > > >> >> >>> > > prtkgaur@gmail.com>
> > > > > > > >> >> >>> > > > > > >>> wrote:
> > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > >> >> >>> > > > > > >>> > > Hi Team,
> > > > > > > >> >> >>> > > > > > >>> > >
> > > > > > > >> >> >>> > > > > > >>> > > Just wanted to provide some updates
> > on
> > > > ALP.
> > > > > > > >> >> >>> > > > > > >>> > > Micah and I have done 3-4 rounds of
> > > > review
> > > > > > for
> > > > > > > >> the
> > > > > > > >> >> c++
> > > > > > > >> >> >>> PR :
> > > > > > > >> >> >>> > > > > > >>> > >
> > > > https://github.com/apache/arrow/pull/48345
> > > > > > > >> >> >>> > > > > > >>> > >
> > > > > > > >> >> >>> > > > > > >>> > > For the Java implementation Vinoo
> has
> > > > been
> > > > > > > >> working
> > > > > > > >> >> on
> > > > > > > >> >> >>> > > following
> > > > > > > >> >> >>> > > > > PR
> > > > > > > >> >> >>> > > > > > :
> > > > > > > >> >> >>> > > > > > >>> > >
> > > > > > > https://github.com/apache/parquet-java/pull/3397
> > > > > > > >> >> >>> > > > > > >>> > >
> > > > > > > >> >> >>> > > > > > >>> > > Best
> > > > > > > >> >> >>> > > > > > >>> > > Prateek
> > > > > > > >> >> >>> > > > > > >>> > >
> > > > > > > >> >> >>> > > > > > >>> > > On Tue, May 5, 2026 at 1:50 PM
> Micah
> > > > > > Kornfield
> > > > > > > <
> > > > > > > >> >> >>> > > > > > >>> emkornfield@gmail.com>
> > > > > > > >> >> >>> > > > > > >>> > > wrote:
> > > > > > > >> >> >>> > > > > > >>> > >
> > > > > > > >> >> >>> > > > > > >>> > >> Hi Antoine,
> > > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > > >> >> >>> > > > > > >>> > >> > Apologies if the question was
> > > already
> > > > > > asked,
> > > > > > > >> but
> > > > > > > >> >> >>> should
> > > > > > > >> >> >>> > we
> > > > > > > >> >> >>> > > > > care
> > > > > > > >> >> >>> > > > > > >>> about
> > > > > > > >> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP
> > be
> > > > more
> > > > > > > >> >> efficient
> > > > > > > >> >> >>> than
> > > > > > > >> >> >>> > > > > FLOAT16
> > > > > > > >> >> >>> > > > > > +
> > > > > > > >> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for
> > example?
> > > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > > >> >> >>> > > > > > >>> > >> It was. We thought we could defer
> > it
> > > > for
> > > > > > the
> > > > > > > >> >> >>> following
> > > > > > > >> >> >>> > > > reasons:
> > > > > > > >> >> >>> > > > > > >>> > >> 1. It's not clear there are a lot
> > of
> > > > easy
> > > > > > > >> >> reference
> > > > > > > >> >> >>> > > datasets
> > > > > > > >> >> >>> > > > to
> > > > > > > >> >> >>> > > > > > >>> test
> > > > > > > >> >> >>> > > > > > >>> > its
> > > > > > > >> >> >>> > > > > > >>> > >> effectiveness. It does look like
> > > there
> > > > > > might
> > > > > > > be
> > > > > > > >> >> one
> > > > > > > >> >> >>> or
> > > > > > > >> >> >>> > two
> > > > > > > >> >> >>> > > on
> > > > > > > >> >> >>> > > > > > >>> > huggingface
> > > > > > > >> >> >>> > > > > > >>> > >> (e.g.
> > > > > > > >> >> https://huggingface.co/datasets/kikitora/curdie
> > > > > > > >> >> >>> ).
> > > > > > > >> >> >>> > > > > > >>> > >> 2. It seemed likely that float 16
> > was
> > > > > more
> > > > > > > >> likely
> > > > > > > >> >> >>> used
> > > > > > > >> >> >>> > for
> > > > > > > >> >> >>> > > > > values
> > > > > > > >> >> >>> > > > > > >>> that
> > > > > > > >> >> >>> > > > > > >>> > >> were less likely to reduce to
> > decimal
> > > > > > values.
> > > > > > > >> >> >>> > > > > > >>> > >> 3. It could be added as an
> > extension
> > > > > later
> > > > > > if
> > > > > > > >> >> needed.
> > > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > > >> >> >>> > > > > > >>> > >> Cheers,
> > > > > > > >> >> >>> > > > > > >>> > >> Micah
> > > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > > >> >> >>> > > > > > >>> > >> On Tue, May 5, 2026 at 1:40 PM
> > Antoine
> > > > > > Pitrou
> > > > > > > <
> > > > > > > >> >> >>> > > > > antoine@python.org
> > > > > > > >> >> >>> > > > > > >
> > > > > > > >> >> >>> > > > > > >>> > wrote:
> > > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > >> >> >>> > > > > > >>> > >> > Hello,
> > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > >> >> >>> > > > > > >>> > >> > Apologies if the question was
> > > already
> > > > > > asked,
> > > > > > > >> but
> > > > > > > >> >> >>> should
> > > > > > > >> >> >>> > we
> > > > > > > >> >> >>> > > > > care
> > > > > > > >> >> >>> > > > > > >>> about
> > > > > > > >> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP
> > be
> > > > more
> > > > > > > >> >> efficient
> > > > > > > >> >> >>> than
> > > > > > > >> >> >>> > > > > FLOAT16
> > > > > > > >> >> >>> > > > > > +
> > > > > > > >> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for
> > example?
> > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > >> >> >>> > > > > > >>> > >> > Regards
> > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > >> >> >>> > > > > > >>> > >> > Antoine.
> > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > >> >> >>> > > > > > >>> > >> > Le 30/04/2026 à 01:10, PRATEEK
> > GAUR
> > > a
> > > > > > écrit
> > > > > > > :
> > > > > > > >> >> >>> > > > > > >>> > >> > > Thanks Andrew and Micah for
> > review
> > > > > > > feedback
> > > > > > > >> on
> > > > > > > >> >> >>> the two
> > > > > > > >> >> >>> > > > PR's
> > > > > > > >> >> >>> > > > > > >>> > >> > > 1) (c++ arrow repo)
> > > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > > >> https://github.com/apache/arrow/pull/48345/changes
> > > > > > > >> >> >>> > > > > > >>> > >> > > 2) (parquet-format repo)
> > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > >> >> https://github.com/apache/parquet-format/pull/557
> > > > > > > >> >> >>> > > > > > >>> > >> > >
> > > > > > > >> >> >>> > > > > > >>> > >> > > I have addressed all (unless I
> > > > missed
> > > > > > > >> >> something)
> > > > > > > >> >> >>> > > comments
> > > > > > > >> >> >>> > > > on
> > > > > > > >> >> >>> > > > > > >>> the two
> > > > > > > >> >> >>> > > > > > >>> > >> > PR's.
> > > > > > > >> >> >>> > > > > > >>> > >> > >
> > > > > > > >> >> >>> > > > > > >>> > >> > > Best
> > > > > > > >> >> >>> > > > > > >>> > >> > > Prateek
> > > > > > > >> >> >>> > > > > > >>> > >> > >
> > > > > > > >> >> >>> > > > > > >>> > >> > > On Sat, Apr 25, 2026 at
> 1:08 PM
> > > > > PRATEEK
> > > > > > > >> GAUR <
> > > > > > > >> >> >>> > > > > > >>> prtkgaur@gmail.com>
> > > > > > > >> >> >>> > > > > > >>> > >> wrote:
> > > > > > > >> >> >>> > > > > > >>> > >> > >
> > > > > > > >> >> >>> > > > > > >>> > >> > >> Thanks Andrew and Micah.
> > > > > > > >> >> >>> > > > > > >>> > >> > >>
> > > > > > > >> >> >>> > > > > > >>> > >> > >> `fair amount of feedback on
> at
> > > > least
> > > > > > the
> > > > > > > >> >> >>> > > implementations`
> > > > > > > >> >> >>> > > > > > >>> > >> > >> For the c++ I have already
> > > started
> > > > > > > >> addressing
> > > > > > > >> >> the
> > > > > > > >> >> >>> > > > > feedback, I
> > > > > > > >> >> >>> > > > > > >>> > should
> > > > > > > >> >> >>> > > > > > >>> > >> be
> > > > > > > >> >> >>> > > > > > >>> > >> > >> done with that
> Monday/Tuesday.
> > > > > > > >> >> >>> > > > > > >>> > >> > >> I think Vinoo too has been
> > making
> > > > > good
> > > > > > > >> >> progress
> > > > > > > >> >> >>> on
> > > > > > > >> >> >>> > the
> > > > > > > >> >> >>> > > > Java
> > > > > > > >> >> >>> > > > > > >>> > >> > implementation.
> > > > > > > >> >> >>> > > > > > >>> > >> > >>
> > > > > > > >> >> >>> > > > > > >>> > >> > >> Best
> > > > > > > >> >> >>> > > > > > >>> > >> > >> Prateek
> > > > > > > >> >> >>> > > > > > >>> > >> > >>
> > > > > > > >> >> >>> > > > > > >>> > >> > >> On Sat, Apr 25, 2026 at
> > 12:55 PM
> > > > > Andrew
> > > > > > > >> Lamb <
> > > > > > > >> >> >>> > > > > > >>> > andrewlamb11@gmail.com
> > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > >> >> >>> > > > > > >>> > >> > >> wrote:
> > > > > > > >> >> >>> > > > > > >>> > >> > >>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>> Got it. Thank you for the
> > > > > > clarification
> > > > > > > >> -- I
> > > > > > > >> >> >>> will
> > > > > > > >> >> >>> > try
> > > > > > > >> >> >>> > > > and
> > > > > > > >> >> >>> > > > > > look
> > > > > > > >> >> >>> > > > > > >>> > into
> > > > > > > >> >> >>> > > > > > >>> > >> the
> > > > > > > >> >> >>> > > > > > >>> > >> > >>> spec and the Rust
> > > > implementation[1]
> > > > > in
> > > > > > > >> this
> > > > > > > >> >> next
> > > > > > > >> >> >>> > week
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>> [1]:
> > > > > > > >> >> >>> https://github.com/apache/arrow-rs/pull/9372
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>> On Sat, Apr 25, 2026 at
> > 12:01 PM
> > > > > Micah
> > > > > > > >> >> >>> Kornfield <
> > > > > > > >> >> >>> > > > > > >>> > >> > emkornfield@gmail.com>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>> wrote:
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>> Hi Andrew,
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>> I think there is a fair
> > amount
> > > of
> > > > > > > >> feedback
> > > > > > > >> >> on
> > > > > > > >> >> >>> at
> > > > > > > >> >> >>> > > least
> > > > > > > >> >> >>> > > > > the
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>> implementations, typically
> I
> > > > think
> > > > > > > we've
> > > > > > > >> >> waited
> > > > > > > >> >> >>> > till
> > > > > > > >> >> >>> > > > they
> > > > > > > >> >> >>> > > > > > are
> > > > > > > >> >> >>> > > > > > >>> > >> close to
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>> mergeable before a final
> > vote.
> > > > > > > >> Otherwise I
> > > > > > > >> >> >>> agree
> > > > > > > >> >> >>> > we
> > > > > > > >> >> >>> > > > are
> > > > > > > >> >> >>> > > > > > very
> > > > > > > >> >> >>> > > > > > >>> > >> close.
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>> -Micah
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>> On Saturday, April 25,
> 2026,
> > > > Andrew
> > > > > > > Lamb
> > > > > > > >> <
> > > > > > > >> >> >>> > > > > > >>> andrewlamb11@gmail.com
> > > > > > > >> >> >>> > > > > > >>> > >
> > > > > > > >> >> >>> > > > > > >>> > >> > wrote:
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> Thanks Prateek,
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> I think from this content
> it
> > > > looks
> > > > > > to
> > > > > > > me
> > > > > > > >> >> like
> > > > > > > >> >> >>> we
> > > > > > > >> >> >>> > are
> > > > > > > >> >> >>> > > > > ready
> > > > > > > >> >> >>> > > > > > >>> to
> > > > > > > >> >> >>> > > > > > >>> > >> start a
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> vote to explicitly accept
> > ALP
> > > > into
> > > > > > > >> Parquet
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> Does anyone know of a
> reason
> > > we
> > > > > > should
> > > > > > > >> >> >>> postpone it
> > > > > > > >> >> >>> > > for
> > > > > > > >> >> >>> > > > > > >>> longer?
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> Perhaps someone needs some
> > > more
> > > > > time
> > > > > > > to
> > > > > > > >> >> >>> review?
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> Andrew
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> On Wed, Apr 22, 2026 at
> > > 1:00 PM
> > > > > > > PRATEEK
> > > > > > > >> >> GAUR <
> > > > > > > >> >> >>> > > > > > >>> > prtkgaur@gmail.com>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>> wrote:
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Hi team,
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Hope everyone is doing
> > well.
> > > I
> > > > > got
> > > > > > a
> > > > > > > >> >> chance
> > > > > > > >> >> >>> to
> > > > > > > >> >> >>> > work
> > > > > > > >> >> >>> > > > > > >>> through all
> > > > > > > >> >> >>> > > > > > >>> > >> the
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> remaining feedback and
> > update
> > > > the
> > > > > > > spec
> > > > > > > >> >> doc.
> > > > > > > >> >> >>> Here
> > > > > > > >> >> >>> > > are
> > > > > > > >> >> >>> > > > > the
> > > > > > > >> >> >>> > > > > > >>> new
> > > > > > > >> >> >>> > > > > > >>> > >> > artifacts
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 1) Spec document :
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > >> >> >>> > > > > > >>>
> > > > > > > >> >> >>> > > > > >
> > > > > > > >> >> >>> > > > >
> > > > > > > >> >> >>> > > >
> > > > > > > >> >> >>> > >
> > > > > > > >> >> >>> >
> > > > > > > >> >> >>>
> > > > > > > >> >>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-
> 3fPtD_aWt/edit
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 2) Spec document in
> parquet
> > > > > format
> > > > > > > >> repo :
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > https://github.com/apache/parquet-format/pull/557
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 3) Alp implementation in
> > > arrow
> > > > > c++
> > > > > > > >> repo :
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > https://github.com/apache/arrow/pull/48345/changes
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 4) Alp implementation in
> > > > > > parquet-java
> > > > > > > >> >> repo :
> > > > > > > >> >> >>> Work
> > > > > > > >> >> >>> > > for
> > > > > > > >> >> >>> > > > > > >>> Vinoo and
> > > > > > > >> >> >>> > > > > > >>> > >> > Julien
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > https://github.com/apache/parquet-java/pull/3397
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 5) PR with test and
> > > > benchmarking
> > > > > > > >> >> artifacts in
> > > > > > > >> >> >>> > > > > > >>> parquet-testing
> > > > > > > >> >> >>> > > > > > >>> > >> repo :
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > https://github.com/apache/parquet-testing/pull/100
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> And
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> - Go : Arnav just
> > > submitted
> > > > > an
> > > > > > in
> > > > > > > >> >> >>> progress
> > > > > > > >> >> >>> > > > > > >>> implementation
> > > > > > > >> >> >>> > > > > > >>> > in
> > > > > > > >> >> >>> > > > > > >>> > >> Go.
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> https://github.com/apache/arrow-go/pull/704
> > > > > > > >> >> >>> > (I
> > > > > > > >> >> >>> > > > > > haven't
> > > > > > > >> >> >>> > > > > > >>> > >> started
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> looking at it yet)
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> - Rust : I remember
> > > Andrew
> > > > > > > >> mentioned
> > > > > > > >> >> that
> > > > > > > >> >> >>> > this
> > > > > > > >> >> >>> > > > work
> > > > > > > >> >> >>> > > > > > is
> > > > > > > >> >> >>> > > > > > >>> also
> > > > > > > >> >> >>> > > > > > >>> > >> in
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> progress (So 4
> > > languages!)
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> *Arrow C++
> implementation *
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> The PR is out and was
> also
> > > used
> > > > > by
> > > > > > > >> >> Antoine to
> > > > > > > >> >> >>> > > report
> > > > > > > >> >> >>> > > > > the
> > > > > > > >> >> >>> > > > > > >>> > numbers
> > > > > > > >> >> >>> > > > > > >>> > >> as
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> reported here. Micah and
> > > > > Konstantin
> > > > > > > >> have
> > > > > > > >> >> >>> given 1
> > > > > > > >> >> >>> > > > round
> > > > > > > >> >> >>> > > > > of
> > > > > > > >> >> >>> > > > > > >>> > >> feedback
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> and I'm addressing them
> > > today.
> > > > > > Please
> > > > > > > >> note
> > > > > > > >> >> >>> that
> > > > > > > >> >> >>> > the
> > > > > > > >> >> >>> > > > > > default
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> optimization flag for
> > > compiling
> > > > > is
> > > > > > O2
> > > > > > > >> and
> > > > > > > >> >> not
> > > > > > > >> >> >>> > Q3. I
> > > > > > > >> >> >>> > > > got
> > > > > > > >> >> >>> > > > > > >>> around
> > > > > > > >> >> >>> > > > > > >>> > >> 70%
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> performance improvement
> in
> > > the
> > > > > > > decoding
> > > > > > > >> >> speed
> > > > > > > >> >> >>> > when
> > > > > > > >> >> >>> > > > > using
> > > > > > > >> >> >>> > > > > > >>> the O3
> > > > > > > >> >> >>> > > > > > >>> > >> > flag.
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> *Parqet-MR Java
> > > implementation
> > > > > > > (working
> > > > > > > >> >> with
> > > > > > > >> >> >>> > Vinoo
> > > > > > > >> >> >>> > > > and
> > > > > > > >> >> >>> > > > > > >>> Julien)
> > > > > > > >> >> >>> > > > > > >>> > >> and
> > > > > > > >> >> >>> > > > > > >>> > >> > **Cross
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Language testing*
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Let me know if you
> have
> > > any
> > > > > > > >> questions
> > > > > > > >> >> or
> > > > > > > >> >> >>> > > > feedback.
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Now pasting some
> > performance
> > > > > > numbers
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Table 1: C++ ALP
> Double
> > > > > Decode —
> > > > > > > >> >> Spotify
> > > > > > > >> >> >>> > Columns
> > > > > > > >> >> >>> > > > > > >>> (Graviton
> > > > > > > >> >> >>> > > > > > >>> > 3,
> > > > > > > >> >> >>> > > > > > >>> > >> ARM
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Neoverse V1)
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>>
> > > > > > > >> >> >>>
> > > > ┌──────────────────┬──────────────┬──────────────┬─────────┐
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ Column │
> > -O2
> > > > > (MB/s)
> > > > > > > │
> > > > > > > >> >> -O3
> > > > > > > >> >> >>> > > (MB/s) │
> > > > > > > >> >> >>> > > > > > >>> Speedup │
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>>
> > > > > > > >> >> >>>
> > > > ├──────────────────┼──────────────┼──────────────┼─────────┤
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ valence │
> > > > 3,155
> > > > > > > │
> > > > > > > >> >> >>> 5,523
> > > > > > > >> >> >>> > > │
> > > > > > > >> >> >>> > > > > > >>> 1.75x │
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ danceability │
> > > > 3,233
> > > > > > > │
> > > > > > > >> >> >>> 5,685
> > > > > > > >> >> >>> > > │
> > > > > > > >> >> >>> > > > > > >>> 1.76x │
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ energy │
> > > > 3,197
> > > > > > > │
> > > > > > > >> >> >>> 5,652
> > > > > > > >> >> >>> > > │
> > > > > > > >> >> >>> > > > > > >>> 1.77x │
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ loudness │
> > > > 3,186
> > > > > > > │
> > > > > > > >> >> >>> 5,473
> > > > > > > >> >> >>> > > │
> > > > > > > >> >> >>> > > > > > >>> 1.72x │
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>>
> > > > > > > >> >> >>>
> > > > └──────────────────┴──────────────┴──────────────┴─────────┘
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> On Wed, Feb 25, 2026 at
> > > 9:49 AM
> > > > > > > PRATEEK
> > > > > > > >> >> GAUR
> > > > > > > >> >> >>> <
> > > > > > > >> >> >>> > > > > > >>> > prtkgaur@gmail.com
> > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> wrote:
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> @Micah Kornfield <
> > > > > > > >> emkornfield@gmail.com>
> > > > > > > >> >> :
> > > > > > > >> >> >>> Got
> > > > > > > >> >> >>> > > it.
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> @Andrew Lamb <
> > > > > > > andrewlamb11@gmail.com>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> Do you think it would
> be
> > > good
> > > > > to
> > > > > > > >> start
> > > > > > > >> >> >>> moving
> > > > > > > >> >> >>> > the
> > > > > > > >> >> >>> > > > > spec
> > > > > > > >> >> >>> > > > > > >>> > >> development
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> into
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> markdown format, in
> > > > preparation
> > > > > > for
> > > > > > > >> >> >>> finalizing
> > > > > > > >> >> >>> > > it?
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> Yes I'll update the
> > numbers
> > > > for
> > > > > > some
> > > > > > > >> of
> > > > > > > >> >> the
> > > > > > > >> >> >>> > > > examples I
> > > > > > > >> >> >>> > > > > > >>> have in
> > > > > > > >> >> >>> > > > > > >>> > >> the
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> spec based
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> on the updated header
> > size.
> > > > Then
> > > > > > we
> > > > > > > >> >> should
> > > > > > > >> >> >>> be
> > > > > > > >> >> >>> > good
> > > > > > > >> >> >>> > > > to
> > > > > > > >> >> >>> > > > > go
> > > > > > > >> >> >>> > > > > > >>> for
> > > > > > > >> >> >>> > > > > > >>> > the
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> markdown format.
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> Thanks everyone!
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> Andrew
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at
> > > > 7:28 PM
> > > > > > > >> PRATEEK
> > > > > > > >> >> >>> GAUR <
> > > > > > > >> >> >>> > > > > > >>> > >> prtkgaur@gmail.com>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> wrote:
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Hi team,
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 1) Andrew
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for
> working
> > > on
> > > > > test
> > > > > > > >> files.
> > > > > > > >> >> >>> My PR
> > > > > > > >> >> >>> > > did
> > > > > > > >> >> >>> > > > > add
> > > > > > > >> >> >>> > > > > > >>> all
> > > > > > > >> >> >>> > > > > > >>> > the
> > > > > > > >> >> >>> > > > > > >>> > >> > test
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> files I
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> used to benchmark
> on
> > > > > > datasets.
> > > > > > > >> >> Maybe
> > > > > > > >> >> >>> we
> > > > > > > >> >> >>> > can
> > > > > > > >> >> >>> > > > club
> > > > > > > >> >> >>> > > > > > it
> > > > > > > >> >> >>> > > > > > >>> > >> together.
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> WIll also
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> aid
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> cross language
> > testing
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Kosta Tarasov
> > > working
> > > > > on
> > > > > > > Rust
> > > > > > > >> >> >>> > > > implementation.
> > > > > > > >> >> >>> > > > > > >>> This is
> > > > > > > >> >> >>> > > > > > >>> > >> > great.
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> Thanks
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 2) Antoine
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks a lot for
> > > > > reporting
> > > > > > > the
> > > > > > > >> >> >>> numbers
> > > > > > > >> >> >>> > on
> > > > > > > >> >> >>> > > > AMD.
> > > > > > > >> >> >>> > > > > > >>> Looks
> > > > > > > >> >> >>> > > > > > >>> > >> like
> > > > > > > >> >> >>> > > > > > >>> > >> > you
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> are
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> getting 8X the
> > > decoding
> > > > > > > >> >> performance of
> > > > > > > >> >> >>> > BSS.
> > > > > > > >> >> >>> > > > This
> > > > > > > >> >> >>> > > > > > is
> > > > > > > >> >> >>> > > > > > >>> > >> > amazing!!.
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for
> > > > acknowledging
> > > > > > the
> > > > > > > >> >> >>> sampling
> > > > > > > >> >> >>> > > > design.
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - I agree with you
> > on
> > > > > > > >> Fastlanes. In
> > > > > > > >> >> >>> some
> > > > > > > >> >> >>> > > crude
> > > > > > > >> >> >>> > > > > > >>> > >> experiments I
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> didn't get
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> a good perf
> benefit
> > > from
> > > > > it
> > > > > > on
> > > > > > > >> >> >>> Graviton3
> > > > > > > >> >> >>> > > (but
> > > > > > > >> >> >>> > > > > > maybe
> > > > > > > >> >> >>> > > > > > >>> > there
> > > > > > > >> >> >>> > > > > > >>> > >> was
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> something
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> wrong with my
> > > > > > implementation).
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Locking the
> 16bit
> > > > > > exception
> > > > > > > >> >> >>> encoding for
> > > > > > > >> >> >>> > > the
> > > > > > > >> >> >>> > > > > > spec
> > > > > > > >> >> >>> > > > > > >>> in
> > > > > > > >> >> >>> > > > > > >>> > >> this
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> case.
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Awesome I think
> we
> > > > have
> > > > > > > solved
> > > > > > > >> >> for
> > > > > > > >> >> >>> all
> > > > > > > >> >> >>> > > open
> > > > > > > >> >> >>> > > > > > >>> questions
> > > > > > > >> >> >>> > > > > > >>> > >> minus
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> the
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> version byte :).
> > (will
> > > > get
> > > > > > > back
> > > > > > > >> on
> > > > > > > >> >> >>> this
> > > > > > > >> >> >>> > > soon)
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 3) Micah
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - FastLanes : The
> > > > current
> > > > > > spec
> > > > > > > >> does
> > > > > > > >> >> >>> allow
> > > > > > > >> >> >>> > > for
> > > > > > > >> >> >>> > > > > > using
> > > > > > > >> >> >>> > > > > > >>> > >> FastLane
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> with the
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> configurable enum
> > > value
> > > > > for
> > > > > > > >> >> layout. We
> > > > > > > >> >> >>> > > should
> > > > > > > >> >> >>> > > > be
> > > > > > > >> >> >>> > > > > > >>> able to
> > > > > > > >> >> >>> > > > > > >>> > >> > inject
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> any
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> layout
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> in the current
> > design.
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Working on resolving
> all
> > > > > > remaining
> > > > > > > >> open
> > > > > > > >> >> >>> > comments
> > > > > > > >> >> >>> > > > on
> > > > > > > >> >> >>> > > > > > the
> > > > > > > >> >> >>> > > > > > >>> spec
> > > > > > > >> >> >>> > > > > > >>> > >> this
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> week.
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Best
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Prateek
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> On Tue, Feb 10, 2026
> at
> > > > > 3:37 AM
> > > > > > > >> Steve
> > > > > > > >> >> >>> > Loughran <
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> stevel@cloudera.com>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> wrote:
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at
> > > > 18:12,
> > > > > > > Micah
> > > > > > > >> >> >>> Kornfield
> > > > > > > >> >> >>> > <
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> emkornfield@gmail.com>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> wrote:
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> It looks like the
> > actual
> > > > > issue
> > > > > > > >> >> >>> described for
> > > > > > > >> >> >>> > > ORC
> > > > > > > >> >> >>> > > > > in
> > > > > > > >> >> >>> > > > > > >>> the
> > > > > > > >> >> >>> > > > > > >>> > >> paper
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> is that
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> it
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> has multiple
> > > sub-encodings
> > > > > in
> > > > > > a
> > > > > > > >> >> batch.
> > > > > > > >> >> >>> This
> > > > > > > >> >> >>> > > is
> > > > > > > >> >> >>> > > > > > >>> different
> > > > > > > >> >> >>> > > > > > >>> > >> then
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> the
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> design
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> proposed here where
> > > there
> > > > is
> > > > > > > still
> > > > > > > >> >> fixed
> > > > > > > >> >> >>> > > > encoding
> > > > > > > >> >> >>> > > > > > per
> > > > > > > >> >> >>> > > > > > >>> page
> > > > > > > >> >> >>> > > > > > >>> > >> in
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> parquet.
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> Given reasonably
> sized
> > > > > pages I
> > > > > > > >> don't
> > > > > > > >> >> >>> think
> > > > > > > >> >> >>> > > > branch
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> misprediction should
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> be a
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> big issue for new
> > > > encodings.
> > > > > > I
> > > > > > > >> agree
> > > > > > > >> >> >>> that
> > > > > > > >> >> >>> > we
> > > > > > > >> >> >>> > > > > should
> > > > > > > >> >> >>> > > > > > >>> be
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> conservative in
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> general for adding
> new
> > > > > > > encodings.
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> +1
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > > > >> >> >>> > > > > > >>> > >> > >
> > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > > >> >> >>> > > > > > >>> > >
> > > > > > > >> >> >>> > > > > > >>> >
> > > > > > > >> >> >>> > > > > > >>>
> > > > > > > >> >> >>> > > > > > >>
> > > > > > > >> >> >>> > > > > >
> > > > > > > >> >> >>> > > > >
> > > > > > > >> >> >>> > > >
> > > > > > > >> >> >>> > >
> > > > > > > >> >> >>> >
> > > > > > > >> >> >>>
> > > > > > > >> >> >>
> > > > > > > >> >>
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Andrew Lamb <an...@gmail.com>.
The Java implementation was just merged[1]!
Thank you Gang, Vinoo, Prateek and everyone who helped push that one along
Andrew
[1]: https://github.com/apache/parquet-java/pull/3397#event-35545451335
On Tue, Oct 13, 2026 at 12:52 PM PRATEEK GAUR <pr...@gmail.com> wrote:
> Hi team,
>
> Quick update re: the C++ implementation.
> Thanks to all the reviewers, we merged the C++ implementation today.
> There are a few follow-up items that I'll address in a subsequent PR.
>
> Thanks
> Prateek
>
> On Sun, Sep 13, 2026 at 6:54 PM Vinoo Ganesh <vi...@gmail.com>
> wrote:
>
> > Quick update re: the Java impl. Thanks for the feedback, on
> > https://github.com/apache/parquet-java/pull/3397, Russel! I pulled
> > https://github.com/apache/parquet-java/pull/3791 into its own PR (thanks
> > for the quick approval too, Gang). Assuming no objections, I think we can
> > merge #3791, and then go ahead of #3397.
> >
> > Thanks,
> > Vinoo Ganesh | vinoo.ganesh@gmail.com
> >
> > <Vi...@gmail.com>
> >
> >
> > On Tue, Sep 8, 2026 at 10:54 AM PRATEEK GAUR <pr...@gmail.com> wrote:
> >
> > > [Rust] Awesome thanks Andrew, Kostas and Devin.
> > > [Java] Thanks Gang for the feedback on Java PR.
> > > [C++] Thanks Gang for another round of feedback on the C++ one.
> > > [Go] Arnav and I will be working on the golang impl this week.
> > >
> > > This is great progress all around.
> > >
> > > On Sat, Sep 5, 2026 at 1:26 PM Vinoo Ganesh <vi...@gmail.com>
> > > wrote:
> > >
> > > > Thanks all - I'm addressing Gang's comments on
> > > > https://github.com/apache/parquet-java/pull/3397 now, so hopefully
> > well
> > > > also be in a good state for merging the java implementation soon.
> > > >
> > > > Thanks,
> > > > Vinoo Ganesh | vinoo.ganesh@gmail.com
> > > >
> > > > <Vi...@gmail.com>
> > > >
> > > >
> > > > On Sat, Sep 5, 2026 at 8:56 AM Andrew Lamb <an...@gmail.com>
> > > wrote:
> > > >
> > > > > We just merged the Rust implementation[1] from Kosta and Devan.
> With
> > > the
> > > > > imminent release of Parquet Format 2.14.0 I think we'll be well on
> > our
> > > > way.
> > > > >
> > > > > Thanks again for pushing this along,
> > > > > Andrew
> > > > >
> > > > > [1]: https://github.com/apache/arrow-rs/pull/9372
> > > > >
> > > > > On Fri, Sep 4, 2026 at 5:42 PM PRATEEK GAUR <pr...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hi team,
> > > > > >
> > > > > > Thanks Micah for multiple rounds of review and thanks Gang for
> > > taking a
> > > > > > look at the PR and giving feedback. I've addressed all comments
> in
> > > the
> > > > > PR.
> > > > > > Looking forward to the next round of feedback (or approval).
> Would
> > > like
> > > > > to
> > > > > > get it in soon.
> > > > > >
> > > > > > https://github.com/apache/arrow/pull/48345/changes
> > > > > >
> > > > > > With this done and after talking to Arnav, I plan to spend some
> > time
> > > on
> > > > > the
> > > > > > ALP implementation in the arrow-go repository helping spread the
> > > > support
> > > > > to
> > > > > > more languages.
> > > > > >
> > > > > > Best
> > > > > > Prateek
> > > > > >
> > > > > > On Sun, Aug 2, 2026 at 10:42 PM PRATEEK GAUR <prtkgaur@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > > > Thanks Divjot for bringing it up.
> > > > > > >
> > > > > > > I was under the same impression as Russell and Andrew that we'd
> > > have
> > > > it
> > > > > > > flag gated.
> > > > > > > With the new versioning proposal 'preview' might be a good tag
> > for
> > > > it.
> > > > > > >
> > > > > > > Best
> > > > > > > Prateek
> > > > > > >
> > > > > > > On Sat, Aug 1, 2026 at 12:39 AM Andrew Lamb <
> > > andrewlamb11@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Div: Here is a proposal to mark ALP in Preview: [1] (using a
> > > comment
> > > > > in
> > > > > > >> the
> > > > > > >> spec section).
> > > > > > >>
> > > > > > >> Russel: That is my understanding as well.
> > > > > > >>
> > > > > > >> Hopefully we are able to formalize the versioning process
> > shortly
> > > so
> > > > > we
> > > > > > >> can
> > > > > > >> formalize the notion of "preview"
> > > > > > >>
> > > > > > >> Andrew
> > > > > > >>
> > > > > > >>
> > > > > > >> [1]: https://github.com/apache/parquet-format/pull/604
> > > > > > >>
> > > > > > >> On Fri, Jul 31, 2026 at 11:15 AM Russell Spitzer <
> > > > > > >> russell.spitzer@gmail.com>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > I thought we had agreed it would be merged, and features
> like
> > > this
> > > > > > would
> > > > > > >> > be feature flag gated. So a V-Old writer could always opt in
> > to
> > > > > using
> > > > > > >> this
> > > > > > >> > encoding but that must be a conscious effort if you are not
> > > using
> > > > > > VNext
> > > > > > >> of
> > > > > > >> > Parquet. If you are on V-Next the writer can choose to use
> ALP
> > > > > > whenever
> > > > > > >> it
> > > > > > >> > likes without user intervention.
> > > > > > >> >
> > > > > > >> > On Fri, Jul 31, 2026 at 2:20 AM Divjot Arora via dev <
> > > > > > >> > dev@parquet.apache.org> wrote:
> > > > > > >> >
> > > > > > >> >> Yes, that’s my understanding as well. However, as you
> noted,
> > > the
> > > > > > >> >> versioning
> > > > > > >> >> discussion is still in early stages and hasn’t prescribed
> > > > concrete
> > > > > > >> >> mechanisms yet. If ALP goes into parquet-format 2.14.0,
> > there’s
> > > > no
> > > > > > >> >> indication that it’s in preview or that writers shouldn’t
> use
> > > it
> > > > by
> > > > > > >> >> default
> > > > > > >> >> yet. Given that we’ve held off on merging other
> incompatible
> > > > > changes
> > > > > > >> (e.g
> > > > > > >> >> removal of path_in_schema) until the versioning discussion
> > > > closes,
> > > > > my
> > > > > > >> >> preference would be to leave ALP un-merged for the time
> being
> > > as
> > > > > > well.
> > > > > > >> >>
> > > > > > >> >> — Div
> > > > > > >> >>
> > > > > > >> >> On Fri, Jul 31, 2026 at 08:59 Andrew Lamb <
> > > > andrewlamb11@gmail.com>
> > > > > > >> wrote:
> > > > > > >> >>
> > > > > > >> >> > After re-reading Julien's Versioning Proposal, my
> > > understanding
> > > > > is
> > > > > > >> that
> > > > > > >> >> if
> > > > > > >> >> > we adopt that system, we will somehow mark ALP in
> "Preview"
> > > in
> > > > > > >> >> > parquet-format for release Version 3.
> > > > > > >> >> >
> > > > > > >> >> > The versioning document doesn't seem to proscribe the
> > > > particular
> > > > > > >> method
> > > > > > >> >> > for annotating "preview" features -- I think clearly
> > marking
> > > > them
> > > > > > as
> > > > > > >> >> such
> > > > > > >> >> > in parquet.thrift as well as reflecting that on the
> > > > > > >> >> > https://parquet.apache.org/docs/file-format/versions/
> page
> > > > > somehow
> > > > > > >> >> would
> > > > > > >> >> > be adequate.
> > > > > > >> >> >
> > > > > > >> >> > Andrew
> > > > > > >> >> >
> > > > > > >> >> > On Thu, Jul 30, 2026 at 11:50 AM Andrew Lamb <
> > > > > > andrewlamb11@gmail.com
> > > > > > >> >
> > > > > > >> >> > wrote:
> > > > > > >> >> >
> > > > > > >> >> >> In my mind ALP is in the same position as other
> encodings
> > > that
> > > > > > were
> > > > > > >> >> added
> > > > > > >> >> >> recently[1] (such as BYTE_STREAM_SPLIT for non floating
> > > point
> > > > > > types)
> > > > > > >> >> >>
> > > > > > >> >> >> That is to say it will be available in the latest open
> > > source
> > > > > > >> writers
> > > > > > >> >> and
> > > > > > >> >> >> readers, but realistically not widely used in the
> > ecosystem
> > > > > where
> > > > > > >> >> >> compatibility is a concern (a large part)
> > > > > > >> >> >>
> > > > > > >> >> >> I view Julian's work[2][3] to gather consensus on how to
> > > name
> > > > > > >> >> >> combinations of features (e.g. "versions") as the
> > mechanism
> > > by
> > > > > > >> which we
> > > > > > >> >> >> will accelerate the adoption of this change (along with
> > > other
> > > > > > >> forward
> > > > > > >> >> >> incompatible changes) across the ecosystem.
> > > > > > >> >> >>
> > > > > > >> >> >> > Will we exclude this change when cutting
> parquet-format
> > > > > v2.14.0?
> > > > > > >> >> >>
> > > > > > >> >> >> No I don't think so -- I would expect it to go out in
> the
> > > next
> > > > > > >> >> >> parquet-format version as do other changes in the spec.
> > > > > > >> >> >>
> > > > > > >> >> >> Andrew
> > > > > > >> >> >>
> > > > > > >> >> >> [1]:
> > https://parquet.apache.org/docs/file-format/versions/
> > > > > > >> >> >> [2]:
> > > > > > >>
> > https://lists.apache.org/thread/xmnj8h0h8ozmrgox5tydhhs7yz7q2hss
> > > > > > >> >> >> [3]:
> > > > > > >>
> > https://lists.apache.org/thread/qlf8lg90gqq41lllyy9mk2f0skqvftqj
> > > > > > >> >> >>
> > > > > > >> >> >>
> > > > > > >> >> >> On Thu, Jul 30, 2026 at 3:51 AM Divjot Arora via dev <
> > > > > > >> >> >> dev@parquet.apache.org> wrote:
> > > > > > >> >> >>
> > > > > > >> >> >>> Hi Andrew,
> > > > > > >> >> >>>
> > > > > > >> >> >>> How does this interact with the ongoing versioning
> > > > discussion?
> > > > > > >> Given
> > > > > > >> >> that
> > > > > > >> >> >>> new encodings are not forward compatible, I would
> assume
> > > ALP
> > > > is
> > > > > > >> >> included
> > > > > > >> >> >>> in
> > > > > > >> >> >>> Parquet V<next> (V3?), but it's not clear to me how
> that
> > > > works
> > > > > if
> > > > > > >> it's
> > > > > > >> >> >>> merged to the parquet-format master branch now. Will we
> > > > exclude
> > > > > > >> this
> > > > > > >> >> >>> change
> > > > > > >> >> >>> when cutting parquet-format v2.14.0?
> > > > > > >> >> >>>
> > > > > > >> >> >>> -- Div
> > > > > > >> >> >>>
> > > > > > >> >> >>> On Thu, Jul 30, 2026 at 8:51 AM Andrew Lamb <
> > > > > > >> andrewlamb11@gmail.com>
> > > > > > >> >> >>> wrote:
> > > > > > >> >> >>>
> > > > > > >> >> >>> > Hello,
> > > > > > >> >> >>> >
> > > > > > >> >> >>> > As an update, thanks to the epic work from many
> > > > contributors,
> > > > > > >> >> >>> especially
> > > > > > >> >> >>> > Prateek, I just merged the parquet-format change for
> > > > ALP[1].
> > > > > We
> > > > > > >> >> plan to
> > > > > > >> >> >>> > make additional editorial The remaining items I know
> of
> > > are
> > > > > > >> merging
> > > > > > >> >> the
> > > > > > >> >> >>> > examples from parquet-testing[2], and then of course
> > the
> > > > > > various
> > > > > > >> >> >>> > implementations
> > > > > > >> >> >>> >
> > > > > > >> >> >>> > Thanks again everyone. This is a great step forward
> > > > > > >> >> >>> > Andrew
> > > > > > >> >> >>> >
> > > > > > >> >> >>> > [1]:
> https://github.com/apache/parquet-format/pull/557
> > > > > > >> >> >>> > [2]:
> > https://github.com/apache/parquet-testing/pull/100
> > > > > > >> >> >>> >
> > > > > > >> >> >>> > On Tue, Jul 21, 2026 at 4:39 PM Julien Le Dem <
> > > > > > julien@apache.org
> > > > > > >> >
> > > > > > >> >> >>> wrote:
> > > > > > >> >> >>> >
> > > > > > >> >> >>> > > It was also in my spam folder... :(
> > > > > > >> >> >>> > > I've seen a bunch of apache list emails go to my
> spam
> > > > > > recently.
> > > > > > >> >> >>> > >
> > > > > > >> >> >>> > > On Tue, Jul 21, 2026 at 10:58 AM Russell Spitzer <
> > > > > > >> >> >>> > > russell.spitzer@gmail.com>
> > > > > > >> >> >>> > > wrote:
> > > > > > >> >> >>> > >
> > > > > > >> >> >>> > > > Yes, my hope was that my vote would kick it out
> of
> > > spam
> > > > > but
> > > > > > >> we
> > > > > > >> >> >>> > definitely
> > > > > > >> >> >>> > > > had two other votes that also went to spam for
> me.
> > > > > > >> >> >>> > > >
> > > > > > >> >> >>> > > > For anyone looking the thread is here
> > > > > > >> >> >>> > > >
> > > > > > >> >>
> > > https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt
> > > > > > >> >> >>> > > >
> > > > > > >> >> >>> > > > On Tue, Jul 21, 2026 at 10:57 AM Micah Kornfield
> <
> > > > > > >> >> >>> > emkornfield@gmail.com>
> > > > > > >> >> >>> > > > wrote:
> > > > > > >> >> >>> > > >
> > > > > > >> >> >>> > > > > I've seen other's votes not go to spam, so
> > > hopefully
> > > > > > people
> > > > > > >> >> will
> > > > > > >> >> >>> see
> > > > > > >> >> >>> > it
> > > > > > >> >> >>> > > > in
> > > > > > >> >> >>> > > > > there inboxes (or see this thread and look in
> > there
> > > > > > spam).
> > > > > > >> >> >>> > > > >
> > > > > > >> >> >>> > > > > On Tue, Jul 21, 2026 at 10:52 AM PRATEEK GAUR <
> > > > > > >> >> >>> prtkgaur@gmail.com>
> > > > > > >> >> >>> > > > wrote:
> > > > > > >> >> >>> > > > >
> > > > > > >> >> >>> > > > > > Okay,
> > > > > > >> >> >>> > > > > >
> > > > > > >> >> >>> > > > > > I sent out the voting email yesterday and so
> > far
> > > > I've
> > > > > > >> heard
> > > > > > >> >> >>> from 3
> > > > > > >> >> >>> > > > people
> > > > > > >> >> >>> > > > > > that it ended up in their "spam folder". Dev
> > VPN
> > > to
> > > > > be
> > > > > > >> >> blamed
> > > > > > >> >> >>> for
> > > > > > >> >> >>> > > it?.
> > > > > > >> >> >>> > > > > >
> > > > > > >> >> >>> > > > > > What's the protocol here?
> > > > > > >> >> >>> > > > > > I let people hunt for the email in their spam
> > > > folder
> > > > > or
> > > > > > >> send
> > > > > > >> >> >>> out
> > > > > > >> >> >>> > > > another
> > > > > > >> >> >>> > > > > > one?
> > > > > > >> >> >>> > > > > >
> > > > > > >> >> >>> > > > > > Best
> > > > > > >> >> >>> > > > > > Prateek
> > > > > > >> >> >>> > > > > >
> > > > > > >> >> >>> > > > > > On Sun, Jul 19, 2026 at 10:47 PM PRATEEK
> GAUR <
> > > > > > >> >> >>> prtkgaur@gmail.com>
> > > > > > >> >> >>> > > > > wrote:
> > > > > > >> >> >>> > > > > >
> > > > > > >> >> >>> > > > > > > Hi team,
> > > > > > >> >> >>> > > > > > >
> > > > > > >> >> >>> > > > > > > Vinoo has added the tests which were
> > requested
> > > in
> > > > > the
> > > > > > >> last
> > > > > > >> >> >>> round
> > > > > > >> >> >>> > of
> > > > > > >> >> >>> > > > > > review.
> > > > > > >> >> >>> > > > > > > In his own words.
> > > > > > >> >> >>> > > > > > >
> > > > > > >> >> >>> > > > > > > "
> > > > > > >> >> >>> > > > > > > The extreme value tests Micah wanted are
> done
> > > and
> > > > > > >> pushed.
> > > > > > >> >> I
> > > > > > >> >> >>> added
> > > > > > >> >> >>> > > > > > coverage
> > > > > > >> >> >>> > > > > > > for values that need the full FOR bit width
> > > after
> > > > > the
> > > > > > >> >> frame
> > > > > > >> >> >>> of
> > > > > > >> >> >>> > > > > reference
> > > > > > >> >> >>> > > > > > is
> > > > > > >> >> >>> > > > > > > applied: 64-bit (the case where the signed
> > max
> > > > > minus
> > > > > > >> min
> > > > > > >> >> >>> > > subtraction
> > > > > > >> >> >>> > > > > > > overflows), 63-bit (the non-overflow case),
> > and
> > > > > > 32-bit
> > > > > > >> for
> > > > > > >> >> >>> > floats.
> > > > > > >> >> >>> > > > All
> > > > > > >> >> >>> > > > > > > lossless with zero exceptions. I also
> > verified
> > > > them
> > > > > > >> >> through
> > > > > > >> >> >>> Arrow
> > > > > > >> >> >>> > > C++
> > > > > > >> >> >>> > > > > ALP
> > > > > > >> >> >>> > > > > > > reader (the local build), so the
> Java-written
> > > > > extreme
> > > > > > >> >> columns
> > > > > > >> >> >>> > > decode
> > > > > > >> >> >>> > > > > > > correctly in C++, bit-exact against the
> > > expected
> > > > > > >> values."
> > > > > > >> >> >>> > > > > > >
> > > > > > >> >> >>> > > > > > > State of the PR (#3397):
> > > > > > >> >> >>> > > > > > >
> > > > > > >> >> >>> > > > > > > - Merges cleanly with master (I pulled
> > > master
> > > > in
> > > > > > and
> > > > > > >> >> >>> resolved
> > > > > > >> >> >>> > > the
> > > > > > >> >> >>> > > > > > > conflicts).
> > > > > > >> >> >>> > > > > > > - 41 of 44 review threads resolved. The
> 3
> > > > > > remaining
> > > > > > >> are
> > > > > > >> >> >>> > > > intentional:
> > > > > > >> >> >>> > > > > > > one deferred perf nit, one that stays
> open
> > > > until
> > > > > > >> >> >>> > parquet-format
> > > > > > >> >> >>> > > > > ships
> > > > > > >> >> >>> > > > > > ALP,
> > > > > > >> >> >>> > > > > > >
> > > > > > >> >> >>> > > > > > > "
> > > > > > >> >> >>> > > > > > >
> > > > > > >> >> >>> > > > > > > With this both the
> > > > > > >> >> >>> > > > > > > 1) C++ PR :
> > > > > > >> >> >>> https://github.com/apache/arrow/pull/48345/changes
> > > > > > >> >> >>> > > > > > > 2) Java PR :
> > > > > > >> >> >>> https://github.com/apache/parquet-java/pull/3397
> > > > > > >> >> >>> > > > > > >
> > > > > > >> >> >>> > > > > > > Are cross language tested and range of test
> > > > > > scenarios.
> > > > > > >> >> >>> > > > > > >
> > > > > > >> >> >>> > > > > > > Best
> > > > > > >> >> >>> > > > > > > Prateek
> > > > > > >> >> >>> > > > > > >
> > > > > > >> >> >>> > > > > > >
> > > > > > >> >> >>> > > > > > > On Mon, Jul 13, 2026 at 8:38 AM PRATEEK
> GAUR
> > <
> > > > > > >> >> >>> prtkgaur@gmail.com
> > > > > > >> >> >>> > >
> > > > > > >> >> >>> > > > > wrote:
> > > > > > >> >> >>> > > > > > >
> > > > > > >> >> >>> > > > > > >> Thanks Matt,
> > > > > > >> >> >>> > > > > > >>
> > > > > > >> >> >>> > > > > > >> I'll be on it this week.
> > > > > > >> >> >>> > > > > > >>
> > > > > > >> >> >>> > > > > > >> Best
> > > > > > >> >> >>> > > > > > >> Prateek
> > > > > > >> >> >>> > > > > > >>
> > > > > > >> >> >>> > > > > > >> On Mon, Jul 13, 2026 at 8:36 AM Matt
> Topol <
> > > > > > >> >> >>> > > zotthewizard@gmail.com>
> > > > > > >> >> >>> > > > > > >> wrote:
> > > > > > >> >> >>> > > > > > >>
> > > > > > >> >> >>> > > > > > >>> There is also the Go implementation
> > submitted
> > > > by
> > > > > > >> Arnav (
> > > > > > >> >> >>> > > > > > >>>
> > https://github.com/apache/arrow-go/pull/704)
> > > > > which
> > > > > > >> is
> > > > > > >> >> >>> waiting
> > > > > > >> >> >>> > > for
> > > > > > >> >> >>> > > > > > >>> updates.
> > > > > > >> >> >>> > > > > > >>>
> > > > > > >> >> >>> > > > > > >>> Just wanted to make sure it didn't get
> lost
> > > > here.
> > > > > > >> >> >>> > > > > > >>>
> > > > > > >> >> >>> > > > > > >>> --Matt
> > > > > > >> >> >>> > > > > > >>>
> > > > > > >> >> >>> > > > > > >>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK
> > > GAUR <
> > > > > > >> >> >>> > > prtkgaur@gmail.com>
> > > > > > >> >> >>> > > > > > >>> wrote:
> > > > > > >> >> >>> > > > > > >>>
> > > > > > >> >> >>> > > > > > >>> > Hi all,
> > > > > > >> >> >>> > > > > > >>> >
> > > > > > >> >> >>> > > > > > >>> > I'd like to share a status update on
> the
> > > ALP
> > > > > > >> encoding
> > > > > > >> >> >>> effort
> > > > > > >> >> >>> > > and
> > > > > > >> >> >>> > > > > get
> > > > > > >> >> >>> > > > > > a
> > > > > > >> >> >>> > > > > > >>> feel
> > > > > > >> >> >>> > > > > > >>> > from the community before starting a
> > formal
> > > > > vote.
> > > > > > >> >> >>> > > > > > >>> >
> > > > > > >> >> >>> > > > > > >>> > We now have the specification plus
> > > > > > implementations
> > > > > > >> in
> > > > > > >> >> >>> both
> > > > > > >> >> >>> > > > > languages:
> > > > > > >> >> >>> > > > > > >>> >
> > > > > > >> >> >>> > > > > > >>> > - Spec:
> > > > apache/parquet-format#557 —
> > > > > > >> GH-533
> > > > > > >> >> >>> Add
> > > > > > >> >> >>> > ALP
> > > > > > >> >> >>> > > > > > encoding
> > > > > > >> >> >>> > > > > > >>> > specification
> > > > > > >> >> >>> > > > > > >>> > - C++ (Arrow): apache/arrow#48345
> —
> > > > > GH-48701
> > > > > > >> >> >>> > > [C++][Parquet]
> > > > > > >> >> >>> > > > > Add
> > > > > > >> >> >>> > > > > > >>> ALPpd
> > > > > > >> >> >>> > > > > > >>> > encoding
> > > > > > >> >> >>> > > > > > >>> > - Java:
> > > apache/parquet-java#3397
> > > > —
> > > > > > >> Parquet
> > > > > > >> >> >>> Java
> > > > > > >> >> >>> > ALP
> > > > > > >> >> >>> > > > > > >>> > Implementation (by Vinoo)
> > > > > > >> >> >>> > > > > > >>> >
> > > > > > >> >> >>> > > > > > >>> > Current state:
> > > > > > >> >> >>> > > > > > >>> >
> > > > > > >> >> >>> > > > > > >>> > - The C++ implementation has been
> > > through
> > > > > 3-4
> > > > > > >> >> rounds
> > > > > > >> >> >>> of
> > > > > > >> >> >>> > > > review.
> > > > > > >> >> >>> > > > > > >>> > - The Java implementation has been
> > > > tracking
> > > > > > >> well
> > > > > > >> >> and
> > > > > > >> >> >>> has
> > > > > > >> >> >>> > > had
> > > > > > >> >> >>> > > > > > >>> initial
> > > > > > >> >> >>> > > > > > >>> > reviews from contributors, with
> no
> > > > major
> > > > > > >> >> remaining
> > > > > > >> >> >>> > > issues.
> > > > > > >> >> >>> > > > > > >>> > - Cross-language compatibility
> tests
> > > are
> > > > > > >> passing:
> > > > > > >> >> the
> > > > > > >> >> >>> > Arrow
> > > > > > >> >> >>> > > > C++
> > > > > > >> >> >>> > > > > > >>> decoder
> > > > > > >> >> >>> > > > > > >>> > reads Java-written data
> bit-exactly
> > > > > across
> > > > > > >> >> ~1.56M
> > > > > > >> >> >>> > values
> > > > > > >> >> >>> > > > and
> > > > > > >> >> >>> > > > > 18
> > > > > > >> >> >>> > > > > > >>> > fixtures, covering V1 and V2
> pages,
> > > > > > multiple
> > > > > > >> >> vector
> > > > > > >> >> >>> > > sizes,
> > > > > > >> >> >>> > > > > and
> > > > > > >> >> >>> > > > > > >>> > several
> > > > > > >> >> >>> > > > > > >>> > real datasets — zero mismatches.
> > > > > > >> >> >>> > > > > > >>> >
> > > > > > >> >> >>> > > > > > >>> > As raised in the java-pr we will be
> > adding
> > > > > > >> coverage
> > > > > > >> >> for
> > > > > > >> >> >>> > > extreme
> > > > > > >> >> >>> > > > > > values
> > > > > > >> >> >>> > > > > > >>> > (those requiring
> > > > > > >> >> >>> > > > > > >>> > 63-64 bits after FOR is applied)
> before
> > we
> > > > > > close a
> > > > > > >> >> vote.
> > > > > > >> >> >>> > We'll
> > > > > > >> >> >>> > > > aim
> > > > > > >> >> >>> > > > > > to
> > > > > > >> >> >>> > > > > > >>> get
> > > > > > >> >> >>> > > > > > >>> > that
> > > > > > >> >> >>> > > > > > >>> > done in parallel.
> > > > > > >> >> >>> > > > > > >>> >
> > > > > > >> >> >>> > > > > > >>> > Plan: unless there are objections, I
> > intend
> > > > to
> > > > > > >> start a
> > > > > > >> >> >>> vote
> > > > > > >> >> >>> > at
> > > > > > >> >> >>> > > > the
> > > > > > >> >> >>> > > > > > end
> > > > > > >> >> >>> > > > > > >>> of
> > > > > > >> >> >>> > > > > > >>> > this week/early
> > > > > > >> >> >>> > > > > > >>> > next week.
> > > > > > >> >> >>> > > > > > >>> >
> > > > > > >> >> >>> > > > > > >>> > Thanks
> > > > > > >> >> >>> > > > > > >>> > Prateek
> > > > > > >> >> >>> > > > > > >>> >
> > > > > > >> >> >>> > > > > > >>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK
> > > GAUR <
> > > > > > >> >> >>> > > prtkgaur@gmail.com>
> > > > > > >> >> >>> > > > > > >>> wrote:
> > > > > > >> >> >>> > > > > > >>> >
> > > > > > >> >> >>> > > > > > >>> > > Hi Team,
> > > > > > >> >> >>> > > > > > >>> > >
> > > > > > >> >> >>> > > > > > >>> > > Just wanted to provide some updates
> on
> > > ALP.
> > > > > > >> >> >>> > > > > > >>> > > Micah and I have done 3-4 rounds of
> > > review
> > > > > for
> > > > > > >> the
> > > > > > >> >> c++
> > > > > > >> >> >>> PR :
> > > > > > >> >> >>> > > > > > >>> > >
> > > https://github.com/apache/arrow/pull/48345
> > > > > > >> >> >>> > > > > > >>> > >
> > > > > > >> >> >>> > > > > > >>> > > For the Java implementation Vinoo has
> > > been
> > > > > > >> working
> > > > > > >> >> on
> > > > > > >> >> >>> > > following
> > > > > > >> >> >>> > > > > PR
> > > > > > >> >> >>> > > > > > :
> > > > > > >> >> >>> > > > > > >>> > >
> > > > > > https://github.com/apache/parquet-java/pull/3397
> > > > > > >> >> >>> > > > > > >>> > >
> > > > > > >> >> >>> > > > > > >>> > > Best
> > > > > > >> >> >>> > > > > > >>> > > Prateek
> > > > > > >> >> >>> > > > > > >>> > >
> > > > > > >> >> >>> > > > > > >>> > > On Tue, May 5, 2026 at 1:50 PM Micah
> > > > > Kornfield
> > > > > > <
> > > > > > >> >> >>> > > > > > >>> emkornfield@gmail.com>
> > > > > > >> >> >>> > > > > > >>> > > wrote:
> > > > > > >> >> >>> > > > > > >>> > >
> > > > > > >> >> >>> > > > > > >>> > >> Hi Antoine,
> > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > >> >> >>> > > > > > >>> > >> > Apologies if the question was
> > already
> > > > > asked,
> > > > > > >> but
> > > > > > >> >> >>> should
> > > > > > >> >> >>> > we
> > > > > > >> >> >>> > > > > care
> > > > > > >> >> >>> > > > > > >>> about
> > > > > > >> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP
> be
> > > more
> > > > > > >> >> efficient
> > > > > > >> >> >>> than
> > > > > > >> >> >>> > > > > FLOAT16
> > > > > > >> >> >>> > > > > > +
> > > > > > >> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for
> example?
> > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > >> >> >>> > > > > > >>> > >> It was. We thought we could defer
> it
> > > for
> > > > > the
> > > > > > >> >> >>> following
> > > > > > >> >> >>> > > > reasons:
> > > > > > >> >> >>> > > > > > >>> > >> 1. It's not clear there are a lot
> of
> > > easy
> > > > > > >> >> reference
> > > > > > >> >> >>> > > datasets
> > > > > > >> >> >>> > > > to
> > > > > > >> >> >>> > > > > > >>> test
> > > > > > >> >> >>> > > > > > >>> > its
> > > > > > >> >> >>> > > > > > >>> > >> effectiveness. It does look like
> > there
> > > > > might
> > > > > > be
> > > > > > >> >> one
> > > > > > >> >> >>> or
> > > > > > >> >> >>> > two
> > > > > > >> >> >>> > > on
> > > > > > >> >> >>> > > > > > >>> > huggingface
> > > > > > >> >> >>> > > > > > >>> > >> (e.g.
> > > > > > >> >> https://huggingface.co/datasets/kikitora/curdie
> > > > > > >> >> >>> ).
> > > > > > >> >> >>> > > > > > >>> > >> 2. It seemed likely that float 16
> was
> > > > more
> > > > > > >> likely
> > > > > > >> >> >>> used
> > > > > > >> >> >>> > for
> > > > > > >> >> >>> > > > > values
> > > > > > >> >> >>> > > > > > >>> that
> > > > > > >> >> >>> > > > > > >>> > >> were less likely to reduce to
> decimal
> > > > > values.
> > > > > > >> >> >>> > > > > > >>> > >> 3. It could be added as an
> extension
> > > > later
> > > > > if
> > > > > > >> >> needed.
> > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > >> >> >>> > > > > > >>> > >> Cheers,
> > > > > > >> >> >>> > > > > > >>> > >> Micah
> > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > >> >> >>> > > > > > >>> > >> On Tue, May 5, 2026 at 1:40 PM
> Antoine
> > > > > Pitrou
> > > > > > <
> > > > > > >> >> >>> > > > > antoine@python.org
> > > > > > >> >> >>> > > > > > >
> > > > > > >> >> >>> > > > > > >>> > wrote:
> > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > >> >> >>> > > > > > >>> > >> > Hello,
> > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > >> >> >>> > > > > > >>> > >> > Apologies if the question was
> > already
> > > > > asked,
> > > > > > >> but
> > > > > > >> >> >>> should
> > > > > > >> >> >>> > we
> > > > > > >> >> >>> > > > > care
> > > > > > >> >> >>> > > > > > >>> about
> > > > > > >> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP
> be
> > > more
> > > > > > >> >> efficient
> > > > > > >> >> >>> than
> > > > > > >> >> >>> > > > > FLOAT16
> > > > > > >> >> >>> > > > > > +
> > > > > > >> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for
> example?
> > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > >> >> >>> > > > > > >>> > >> > Regards
> > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > >> >> >>> > > > > > >>> > >> > Antoine.
> > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > >> >> >>> > > > > > >>> > >> > Le 30/04/2026 à 01:10, PRATEEK
> GAUR
> > a
> > > > > écrit
> > > > > > :
> > > > > > >> >> >>> > > > > > >>> > >> > > Thanks Andrew and Micah for
> review
> > > > > > feedback
> > > > > > >> on
> > > > > > >> >> >>> the two
> > > > > > >> >> >>> > > > PR's
> > > > > > >> >> >>> > > > > > >>> > >> > > 1) (c++ arrow repo)
> > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > >> https://github.com/apache/arrow/pull/48345/changes
> > > > > > >> >> >>> > > > > > >>> > >> > > 2) (parquet-format repo)
> > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > >> >> https://github.com/apache/parquet-format/pull/557
> > > > > > >> >> >>> > > > > > >>> > >> > >
> > > > > > >> >> >>> > > > > > >>> > >> > > I have addressed all (unless I
> > > missed
> > > > > > >> >> something)
> > > > > > >> >> >>> > > comments
> > > > > > >> >> >>> > > > on
> > > > > > >> >> >>> > > > > > >>> the two
> > > > > > >> >> >>> > > > > > >>> > >> > PR's.
> > > > > > >> >> >>> > > > > > >>> > >> > >
> > > > > > >> >> >>> > > > > > >>> > >> > > Best
> > > > > > >> >> >>> > > > > > >>> > >> > > Prateek
> > > > > > >> >> >>> > > > > > >>> > >> > >
> > > > > > >> >> >>> > > > > > >>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM
> > > > PRATEEK
> > > > > > >> GAUR <
> > > > > > >> >> >>> > > > > > >>> prtkgaur@gmail.com>
> > > > > > >> >> >>> > > > > > >>> > >> wrote:
> > > > > > >> >> >>> > > > > > >>> > >> > >
> > > > > > >> >> >>> > > > > > >>> > >> > >> Thanks Andrew and Micah.
> > > > > > >> >> >>> > > > > > >>> > >> > >>
> > > > > > >> >> >>> > > > > > >>> > >> > >> `fair amount of feedback on at
> > > least
> > > > > the
> > > > > > >> >> >>> > > implementations`
> > > > > > >> >> >>> > > > > > >>> > >> > >> For the c++ I have already
> > started
> > > > > > >> addressing
> > > > > > >> >> the
> > > > > > >> >> >>> > > > > feedback, I
> > > > > > >> >> >>> > > > > > >>> > should
> > > > > > >> >> >>> > > > > > >>> > >> be
> > > > > > >> >> >>> > > > > > >>> > >> > >> done with that Monday/Tuesday.
> > > > > > >> >> >>> > > > > > >>> > >> > >> I think Vinoo too has been
> making
> > > > good
> > > > > > >> >> progress
> > > > > > >> >> >>> on
> > > > > > >> >> >>> > the
> > > > > > >> >> >>> > > > Java
> > > > > > >> >> >>> > > > > > >>> > >> > implementation.
> > > > > > >> >> >>> > > > > > >>> > >> > >>
> > > > > > >> >> >>> > > > > > >>> > >> > >> Best
> > > > > > >> >> >>> > > > > > >>> > >> > >> Prateek
> > > > > > >> >> >>> > > > > > >>> > >> > >>
> > > > > > >> >> >>> > > > > > >>> > >> > >> On Sat, Apr 25, 2026 at
> 12:55 PM
> > > > Andrew
> > > > > > >> Lamb <
> > > > > > >> >> >>> > > > > > >>> > andrewlamb11@gmail.com
> > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > >> >> >>> > > > > > >>> > >> > >> wrote:
> > > > > > >> >> >>> > > > > > >>> > >> > >>
> > > > > > >> >> >>> > > > > > >>> > >> > >>> Got it. Thank you for the
> > > > > clarification
> > > > > > >> -- I
> > > > > > >> >> >>> will
> > > > > > >> >> >>> > try
> > > > > > >> >> >>> > > > and
> > > > > > >> >> >>> > > > > > look
> > > > > > >> >> >>> > > > > > >>> > into
> > > > > > >> >> >>> > > > > > >>> > >> the
> > > > > > >> >> >>> > > > > > >>> > >> > >>> spec and the Rust
> > > implementation[1]
> > > > in
> > > > > > >> this
> > > > > > >> >> next
> > > > > > >> >> >>> > week
> > > > > > >> >> >>> > > > > > >>> > >> > >>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>> [1]:
> > > > > > >> >> >>> https://github.com/apache/arrow-rs/pull/9372
> > > > > > >> >> >>> > > > > > >>> > >> > >>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>> On Sat, Apr 25, 2026 at
> 12:01 PM
> > > > Micah
> > > > > > >> >> >>> Kornfield <
> > > > > > >> >> >>> > > > > > >>> > >> > emkornfield@gmail.com>
> > > > > > >> >> >>> > > > > > >>> > >> > >>> wrote:
> > > > > > >> >> >>> > > > > > >>> > >> > >>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>> Hi Andrew,
> > > > > > >> >> >>> > > > > > >>> > >> > >>>> I think there is a fair
> amount
> > of
> > > > > > >> feedback
> > > > > > >> >> on
> > > > > > >> >> >>> at
> > > > > > >> >> >>> > > least
> > > > > > >> >> >>> > > > > the
> > > > > > >> >> >>> > > > > > >>> > >> > >>>> implementations, typically I
> > > think
> > > > > > we've
> > > > > > >> >> waited
> > > > > > >> >> >>> > till
> > > > > > >> >> >>> > > > they
> > > > > > >> >> >>> > > > > > are
> > > > > > >> >> >>> > > > > > >>> > >> close to
> > > > > > >> >> >>> > > > > > >>> > >> > >>>> mergeable before a final
> vote.
> > > > > > >> Otherwise I
> > > > > > >> >> >>> agree
> > > > > > >> >> >>> > we
> > > > > > >> >> >>> > > > are
> > > > > > >> >> >>> > > > > > very
> > > > > > >> >> >>> > > > > > >>> > >> close.
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>> -Micah
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>> On Saturday, April 25, 2026,
> > > Andrew
> > > > > > Lamb
> > > > > > >> <
> > > > > > >> >> >>> > > > > > >>> andrewlamb11@gmail.com
> > > > > > >> >> >>> > > > > > >>> > >
> > > > > > >> >> >>> > > > > > >>> > >> > wrote:
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>> Thanks Prateek,
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>> I think from this content it
> > > looks
> > > > > to
> > > > > > me
> > > > > > >> >> like
> > > > > > >> >> >>> we
> > > > > > >> >> >>> > are
> > > > > > >> >> >>> > > > > ready
> > > > > > >> >> >>> > > > > > >>> to
> > > > > > >> >> >>> > > > > > >>> > >> start a
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>> vote to explicitly accept
> ALP
> > > into
> > > > > > >> Parquet
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>> Does anyone know of a reason
> > we
> > > > > should
> > > > > > >> >> >>> postpone it
> > > > > > >> >> >>> > > for
> > > > > > >> >> >>> > > > > > >>> longer?
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>> Perhaps someone needs some
> > more
> > > > time
> > > > > > to
> > > > > > >> >> >>> review?
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>> Andrew
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>> On Wed, Apr 22, 2026 at
> > 1:00 PM
> > > > > > PRATEEK
> > > > > > >> >> GAUR <
> > > > > > >> >> >>> > > > > > >>> > prtkgaur@gmail.com>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>> wrote:
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Hi team,
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Hope everyone is doing
> well.
> > I
> > > > got
> > > > > a
> > > > > > >> >> chance
> > > > > > >> >> >>> to
> > > > > > >> >> >>> > work
> > > > > > >> >> >>> > > > > > >>> through all
> > > > > > >> >> >>> > > > > > >>> > >> the
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> remaining feedback and
> update
> > > the
> > > > > > spec
> > > > > > >> >> doc.
> > > > > > >> >> >>> Here
> > > > > > >> >> >>> > > are
> > > > > > >> >> >>> > > > > the
> > > > > > >> >> >>> > > > > > >>> new
> > > > > > >> >> >>> > > > > > >>> > >> > artifacts
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 1) Spec document :
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > >> >> >>> > > > > > >>> >
> > > > > > >> >> >>> > > > > > >>>
> > > > > > >> >> >>> > > > > >
> > > > > > >> >> >>> > > > >
> > > > > > >> >> >>> > > >
> > > > > > >> >> >>> > >
> > > > > > >> >> >>> >
> > > > > > >> >> >>>
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 2) Spec document in parquet
> > > > format
> > > > > > >> repo :
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > https://github.com/apache/parquet-format/pull/557
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 3) Alp implementation in
> > arrow
> > > > c++
> > > > > > >> repo :
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > https://github.com/apache/arrow/pull/48345/changes
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 4) Alp implementation in
> > > > > parquet-java
> > > > > > >> >> repo :
> > > > > > >> >> >>> Work
> > > > > > >> >> >>> > > for
> > > > > > >> >> >>> > > > > > >>> Vinoo and
> > > > > > >> >> >>> > > > > > >>> > >> > Julien
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > https://github.com/apache/parquet-java/pull/3397
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 5) PR with test and
> > > benchmarking
> > > > > > >> >> artifacts in
> > > > > > >> >> >>> > > > > > >>> parquet-testing
> > > > > > >> >> >>> > > > > > >>> > >> repo :
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > https://github.com/apache/parquet-testing/pull/100
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> And
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> - Go : Arnav just
> > submitted
> > > > an
> > > > > in
> > > > > > >> >> >>> progress
> > > > > > >> >> >>> > > > > > >>> implementation
> > > > > > >> >> >>> > > > > > >>> > in
> > > > > > >> >> >>> > > > > > >>> > >> Go.
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> https://github.com/apache/arrow-go/pull/704
> > > > > > >> >> >>> > (I
> > > > > > >> >> >>> > > > > > haven't
> > > > > > >> >> >>> > > > > > >>> > >> started
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> looking at it yet)
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> - Rust : I remember
> > Andrew
> > > > > > >> mentioned
> > > > > > >> >> that
> > > > > > >> >> >>> > this
> > > > > > >> >> >>> > > > work
> > > > > > >> >> >>> > > > > > is
> > > > > > >> >> >>> > > > > > >>> also
> > > > > > >> >> >>> > > > > > >>> > >> in
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> progress (So 4
> > languages!)
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> *Arrow C++ implementation *
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> The PR is out and was also
> > used
> > > > by
> > > > > > >> >> Antoine to
> > > > > > >> >> >>> > > report
> > > > > > >> >> >>> > > > > the
> > > > > > >> >> >>> > > > > > >>> > numbers
> > > > > > >> >> >>> > > > > > >>> > >> as
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> reported here. Micah and
> > > > Konstantin
> > > > > > >> have
> > > > > > >> >> >>> given 1
> > > > > > >> >> >>> > > > round
> > > > > > >> >> >>> > > > > of
> > > > > > >> >> >>> > > > > > >>> > >> feedback
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> and I'm addressing them
> > today.
> > > > > Please
> > > > > > >> note
> > > > > > >> >> >>> that
> > > > > > >> >> >>> > the
> > > > > > >> >> >>> > > > > > default
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> optimization flag for
> > compiling
> > > > is
> > > > > O2
> > > > > > >> and
> > > > > > >> >> not
> > > > > > >> >> >>> > Q3. I
> > > > > > >> >> >>> > > > got
> > > > > > >> >> >>> > > > > > >>> around
> > > > > > >> >> >>> > > > > > >>> > >> 70%
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> performance improvement in
> > the
> > > > > > decoding
> > > > > > >> >> speed
> > > > > > >> >> >>> > when
> > > > > > >> >> >>> > > > > using
> > > > > > >> >> >>> > > > > > >>> the O3
> > > > > > >> >> >>> > > > > > >>> > >> > flag.
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> *Parqet-MR Java
> > implementation
> > > > > > (working
> > > > > > >> >> with
> > > > > > >> >> >>> > Vinoo
> > > > > > >> >> >>> > > > and
> > > > > > >> >> >>> > > > > > >>> Julien)
> > > > > > >> >> >>> > > > > > >>> > >> and
> > > > > > >> >> >>> > > > > > >>> > >> > **Cross
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Language testing*
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Let me know if you have
> > any
> > > > > > >> questions
> > > > > > >> >> or
> > > > > > >> >> >>> > > > feedback.
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Now pasting some
> performance
> > > > > numbers
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Table 1: C++ ALP Double
> > > > Decode —
> > > > > > >> >> Spotify
> > > > > > >> >> >>> > Columns
> > > > > > >> >> >>> > > > > > >>> (Graviton
> > > > > > >> >> >>> > > > > > >>> > 3,
> > > > > > >> >> >>> > > > > > >>> > >> ARM
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Neoverse V1)
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>>
> > > > > > >> >> >>>
> > > ┌──────────────────┬──────────────┬──────────────┬─────────┐
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ Column │
> -O2
> > > > (MB/s)
> > > > > > │
> > > > > > >> >> -O3
> > > > > > >> >> >>> > > (MB/s) │
> > > > > > >> >> >>> > > > > > >>> Speedup │
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>>
> > > > > > >> >> >>>
> > > ├──────────────────┼──────────────┼──────────────┼─────────┤
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ valence │
> > > 3,155
> > > > > > │
> > > > > > >> >> >>> 5,523
> > > > > > >> >> >>> > > │
> > > > > > >> >> >>> > > > > > >>> 1.75x │
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ danceability │
> > > 3,233
> > > > > > │
> > > > > > >> >> >>> 5,685
> > > > > > >> >> >>> > > │
> > > > > > >> >> >>> > > > > > >>> 1.76x │
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ energy │
> > > 3,197
> > > > > > │
> > > > > > >> >> >>> 5,652
> > > > > > >> >> >>> > > │
> > > > > > >> >> >>> > > > > > >>> 1.77x │
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ loudness │
> > > 3,186
> > > > > > │
> > > > > > >> >> >>> 5,473
> > > > > > >> >> >>> > > │
> > > > > > >> >> >>> > > > > > >>> 1.72x │
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>>
> > > > > > >> >> >>>
> > > └──────────────────┴──────────────┴──────────────┴─────────┘
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> On Wed, Feb 25, 2026 at
> > 9:49 AM
> > > > > > PRATEEK
> > > > > > >> >> GAUR
> > > > > > >> >> >>> <
> > > > > > >> >> >>> > > > > > >>> > prtkgaur@gmail.com
> > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>> wrote:
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> @Micah Kornfield <
> > > > > > >> emkornfield@gmail.com>
> > > > > > >> >> :
> > > > > > >> >> >>> Got
> > > > > > >> >> >>> > > it.
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> @Andrew Lamb <
> > > > > > andrewlamb11@gmail.com>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> Do you think it would be
> > good
> > > > to
> > > > > > >> start
> > > > > > >> >> >>> moving
> > > > > > >> >> >>> > the
> > > > > > >> >> >>> > > > > spec
> > > > > > >> >> >>> > > > > > >>> > >> development
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> into
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> markdown format, in
> > > preparation
> > > > > for
> > > > > > >> >> >>> finalizing
> > > > > > >> >> >>> > > it?
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> Yes I'll update the
> numbers
> > > for
> > > > > some
> > > > > > >> of
> > > > > > >> >> the
> > > > > > >> >> >>> > > > examples I
> > > > > > >> >> >>> > > > > > >>> have in
> > > > > > >> >> >>> > > > > > >>> > >> the
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> spec based
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> on the updated header
> size.
> > > Then
> > > > > we
> > > > > > >> >> should
> > > > > > >> >> >>> be
> > > > > > >> >> >>> > good
> > > > > > >> >> >>> > > > to
> > > > > > >> >> >>> > > > > go
> > > > > > >> >> >>> > > > > > >>> for
> > > > > > >> >> >>> > > > > > >>> > the
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> markdown format.
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> Thanks everyone!
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> Andrew
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at
> > > 7:28 PM
> > > > > > >> PRATEEK
> > > > > > >> >> >>> GAUR <
> > > > > > >> >> >>> > > > > > >>> > >> prtkgaur@gmail.com>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> wrote:
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Hi team,
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 1) Andrew
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for working
> > on
> > > > test
> > > > > > >> files.
> > > > > > >> >> >>> My PR
> > > > > > >> >> >>> > > did
> > > > > > >> >> >>> > > > > add
> > > > > > >> >> >>> > > > > > >>> all
> > > > > > >> >> >>> > > > > > >>> > the
> > > > > > >> >> >>> > > > > > >>> > >> > test
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> files I
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> used to benchmark on
> > > > > datasets.
> > > > > > >> >> Maybe
> > > > > > >> >> >>> we
> > > > > > >> >> >>> > can
> > > > > > >> >> >>> > > > club
> > > > > > >> >> >>> > > > > > it
> > > > > > >> >> >>> > > > > > >>> > >> together.
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> WIll also
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> aid
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> cross language
> testing
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Kosta Tarasov
> > working
> > > > on
> > > > > > Rust
> > > > > > >> >> >>> > > > implementation.
> > > > > > >> >> >>> > > > > > >>> This is
> > > > > > >> >> >>> > > > > > >>> > >> > great.
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> Thanks
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 2) Antoine
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks a lot for
> > > > reporting
> > > > > > the
> > > > > > >> >> >>> numbers
> > > > > > >> >> >>> > on
> > > > > > >> >> >>> > > > AMD.
> > > > > > >> >> >>> > > > > > >>> Looks
> > > > > > >> >> >>> > > > > > >>> > >> like
> > > > > > >> >> >>> > > > > > >>> > >> > you
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> are
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> getting 8X the
> > decoding
> > > > > > >> >> performance of
> > > > > > >> >> >>> > BSS.
> > > > > > >> >> >>> > > > This
> > > > > > >> >> >>> > > > > > is
> > > > > > >> >> >>> > > > > > >>> > >> > amazing!!.
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for
> > > acknowledging
> > > > > the
> > > > > > >> >> >>> sampling
> > > > > > >> >> >>> > > > design.
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - I agree with you
> on
> > > > > > >> Fastlanes. In
> > > > > > >> >> >>> some
> > > > > > >> >> >>> > > crude
> > > > > > >> >> >>> > > > > > >>> > >> experiments I
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> didn't get
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> a good perf benefit
> > from
> > > > it
> > > > > on
> > > > > > >> >> >>> Graviton3
> > > > > > >> >> >>> > > (but
> > > > > > >> >> >>> > > > > > maybe
> > > > > > >> >> >>> > > > > > >>> > there
> > > > > > >> >> >>> > > > > > >>> > >> was
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> something
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> wrong with my
> > > > > implementation).
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Locking the 16bit
> > > > > exception
> > > > > > >> >> >>> encoding for
> > > > > > >> >> >>> > > the
> > > > > > >> >> >>> > > > > > spec
> > > > > > >> >> >>> > > > > > >>> in
> > > > > > >> >> >>> > > > > > >>> > >> this
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> case.
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Awesome I think we
> > > have
> > > > > > solved
> > > > > > >> >> for
> > > > > > >> >> >>> all
> > > > > > >> >> >>> > > open
> > > > > > >> >> >>> > > > > > >>> questions
> > > > > > >> >> >>> > > > > > >>> > >> minus
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> the
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> version byte :).
> (will
> > > get
> > > > > > back
> > > > > > >> on
> > > > > > >> >> >>> this
> > > > > > >> >> >>> > > soon)
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 3) Micah
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - FastLanes : The
> > > current
> > > > > spec
> > > > > > >> does
> > > > > > >> >> >>> allow
> > > > > > >> >> >>> > > for
> > > > > > >> >> >>> > > > > > using
> > > > > > >> >> >>> > > > > > >>> > >> FastLane
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> with the
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> configurable enum
> > value
> > > > for
> > > > > > >> >> layout. We
> > > > > > >> >> >>> > > should
> > > > > > >> >> >>> > > > be
> > > > > > >> >> >>> > > > > > >>> able to
> > > > > > >> >> >>> > > > > > >>> > >> > inject
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> any
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> layout
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> in the current
> design.
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Working on resolving all
> > > > > remaining
> > > > > > >> open
> > > > > > >> >> >>> > comments
> > > > > > >> >> >>> > > > on
> > > > > > >> >> >>> > > > > > the
> > > > > > >> >> >>> > > > > > >>> spec
> > > > > > >> >> >>> > > > > > >>> > >> this
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> week.
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Best
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Prateek
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at
> > > > 3:37 AM
> > > > > > >> Steve
> > > > > > >> >> >>> > Loughran <
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> stevel@cloudera.com>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> wrote:
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at
> > > 18:12,
> > > > > > Micah
> > > > > > >> >> >>> Kornfield
> > > > > > >> >> >>> > <
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> emkornfield@gmail.com>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> wrote:
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> It looks like the
> actual
> > > > issue
> > > > > > >> >> >>> described for
> > > > > > >> >> >>> > > ORC
> > > > > > >> >> >>> > > > > in
> > > > > > >> >> >>> > > > > > >>> the
> > > > > > >> >> >>> > > > > > >>> > >> paper
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> is that
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> it
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> has multiple
> > sub-encodings
> > > > in
> > > > > a
> > > > > > >> >> batch.
> > > > > > >> >> >>> This
> > > > > > >> >> >>> > > is
> > > > > > >> >> >>> > > > > > >>> different
> > > > > > >> >> >>> > > > > > >>> > >> then
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> the
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> design
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> proposed here where
> > there
> > > is
> > > > > > still
> > > > > > >> >> fixed
> > > > > > >> >> >>> > > > encoding
> > > > > > >> >> >>> > > > > > per
> > > > > > >> >> >>> > > > > > >>> page
> > > > > > >> >> >>> > > > > > >>> > >> in
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> parquet.
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> Given reasonably sized
> > > > pages I
> > > > > > >> don't
> > > > > > >> >> >>> think
> > > > > > >> >> >>> > > > branch
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> misprediction should
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> be a
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> big issue for new
> > > encodings.
> > > > > I
> > > > > > >> agree
> > > > > > >> >> >>> that
> > > > > > >> >> >>> > we
> > > > > > >> >> >>> > > > > should
> > > > > > >> >> >>> > > > > > >>> be
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> conservative in
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> general for adding new
> > > > > > encodings.
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> +1
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > > >> >> >>> > > > > > >>> > >> > >
> > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > >> >> >>> > > > > > >>> > >> >
> > > > > > >> >> >>> > > > > > >>> > >>
> > > > > > >> >> >>> > > > > > >>> > >
> > > > > > >> >> >>> > > > > > >>> >
> > > > > > >> >> >>> > > > > > >>>
> > > > > > >> >> >>> > > > > > >>
> > > > > > >> >> >>> > > > > >
> > > > > > >> >> >>> > > > >
> > > > > > >> >> >>> > > >
> > > > > > >> >> >>> > >
> > > > > > >> >> >>> >
> > > > > > >> >> >>>
> > > > > > >> >> >>
> > > > > > >> >>
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by PRATEEK GAUR <pr...@gmail.com>.
Hi team,
Quick update re: the C++ implementation.
Thanks to all the reviewers, we merged the C++ implementation today.
There are a few follow-up items that I'll address in a subsequent PR.
Thanks
Prateek
On Sun, Sep 13, 2026 at 6:54 PM Vinoo Ganesh <vi...@gmail.com> wrote:
> Quick update re: the Java impl. Thanks for the feedback, on
> https://github.com/apache/parquet-java/pull/3397, Russel! I pulled
> https://github.com/apache/parquet-java/pull/3791 into its own PR (thanks
> for the quick approval too, Gang). Assuming no objections, I think we can
> merge #3791, and then go ahead of #3397.
>
> Thanks,
> Vinoo Ganesh | vinoo.ganesh@gmail.com
>
> <Vi...@gmail.com>
>
>
> On Tue, Sep 8, 2026 at 10:54 AM PRATEEK GAUR <pr...@gmail.com> wrote:
>
> > [Rust] Awesome thanks Andrew, Kostas and Devin.
> > [Java] Thanks Gang for the feedback on Java PR.
> > [C++] Thanks Gang for another round of feedback on the C++ one.
> > [Go] Arnav and I will be working on the golang impl this week.
> >
> > This is great progress all around.
> >
> > On Sat, Sep 5, 2026 at 1:26 PM Vinoo Ganesh <vi...@gmail.com>
> > wrote:
> >
> > > Thanks all - I'm addressing Gang's comments on
> > > https://github.com/apache/parquet-java/pull/3397 now, so hopefully
> well
> > > also be in a good state for merging the java implementation soon.
> > >
> > > Thanks,
> > > Vinoo Ganesh | vinoo.ganesh@gmail.com
> > >
> > > <Vi...@gmail.com>
> > >
> > >
> > > On Sat, Sep 5, 2026 at 8:56 AM Andrew Lamb <an...@gmail.com>
> > wrote:
> > >
> > > > We just merged the Rust implementation[1] from Kosta and Devan. With
> > the
> > > > imminent release of Parquet Format 2.14.0 I think we'll be well on
> our
> > > way.
> > > >
> > > > Thanks again for pushing this along,
> > > > Andrew
> > > >
> > > > [1]: https://github.com/apache/arrow-rs/pull/9372
> > > >
> > > > On Fri, Sep 4, 2026 at 5:42 PM PRATEEK GAUR <pr...@gmail.com>
> > wrote:
> > > >
> > > > > Hi team,
> > > > >
> > > > > Thanks Micah for multiple rounds of review and thanks Gang for
> > taking a
> > > > > look at the PR and giving feedback. I've addressed all comments in
> > the
> > > > PR.
> > > > > Looking forward to the next round of feedback (or approval). Would
> > like
> > > > to
> > > > > get it in soon.
> > > > >
> > > > > https://github.com/apache/arrow/pull/48345/changes
> > > > >
> > > > > With this done and after talking to Arnav, I plan to spend some
> time
> > on
> > > > the
> > > > > ALP implementation in the arrow-go repository helping spread the
> > > support
> > > > to
> > > > > more languages.
> > > > >
> > > > > Best
> > > > > Prateek
> > > > >
> > > > > On Sun, Aug 2, 2026 at 10:42 PM PRATEEK GAUR <pr...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Thanks Divjot for bringing it up.
> > > > > >
> > > > > > I was under the same impression as Russell and Andrew that we'd
> > have
> > > it
> > > > > > flag gated.
> > > > > > With the new versioning proposal 'preview' might be a good tag
> for
> > > it.
> > > > > >
> > > > > > Best
> > > > > > Prateek
> > > > > >
> > > > > > On Sat, Aug 1, 2026 at 12:39 AM Andrew Lamb <
> > andrewlamb11@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > >> Div: Here is a proposal to mark ALP in Preview: [1] (using a
> > comment
> > > > in
> > > > > >> the
> > > > > >> spec section).
> > > > > >>
> > > > > >> Russel: That is my understanding as well.
> > > > > >>
> > > > > >> Hopefully we are able to formalize the versioning process
> shortly
> > so
> > > > we
> > > > > >> can
> > > > > >> formalize the notion of "preview"
> > > > > >>
> > > > > >> Andrew
> > > > > >>
> > > > > >>
> > > > > >> [1]: https://github.com/apache/parquet-format/pull/604
> > > > > >>
> > > > > >> On Fri, Jul 31, 2026 at 11:15 AM Russell Spitzer <
> > > > > >> russell.spitzer@gmail.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > I thought we had agreed it would be merged, and features like
> > this
> > > > > would
> > > > > >> > be feature flag gated. So a V-Old writer could always opt in
> to
> > > > using
> > > > > >> this
> > > > > >> > encoding but that must be a conscious effort if you are not
> > using
> > > > > VNext
> > > > > >> of
> > > > > >> > Parquet. If you are on V-Next the writer can choose to use ALP
> > > > > whenever
> > > > > >> it
> > > > > >> > likes without user intervention.
> > > > > >> >
> > > > > >> > On Fri, Jul 31, 2026 at 2:20 AM Divjot Arora via dev <
> > > > > >> > dev@parquet.apache.org> wrote:
> > > > > >> >
> > > > > >> >> Yes, that’s my understanding as well. However, as you noted,
> > the
> > > > > >> >> versioning
> > > > > >> >> discussion is still in early stages and hasn’t prescribed
> > > concrete
> > > > > >> >> mechanisms yet. If ALP goes into parquet-format 2.14.0,
> there’s
> > > no
> > > > > >> >> indication that it’s in preview or that writers shouldn’t use
> > it
> > > by
> > > > > >> >> default
> > > > > >> >> yet. Given that we’ve held off on merging other incompatible
> > > > changes
> > > > > >> (e.g
> > > > > >> >> removal of path_in_schema) until the versioning discussion
> > > closes,
> > > > my
> > > > > >> >> preference would be to leave ALP un-merged for the time being
> > as
> > > > > well.
> > > > > >> >>
> > > > > >> >> — Div
> > > > > >> >>
> > > > > >> >> On Fri, Jul 31, 2026 at 08:59 Andrew Lamb <
> > > andrewlamb11@gmail.com>
> > > > > >> wrote:
> > > > > >> >>
> > > > > >> >> > After re-reading Julien's Versioning Proposal, my
> > understanding
> > > > is
> > > > > >> that
> > > > > >> >> if
> > > > > >> >> > we adopt that system, we will somehow mark ALP in "Preview"
> > in
> > > > > >> >> > parquet-format for release Version 3.
> > > > > >> >> >
> > > > > >> >> > The versioning document doesn't seem to proscribe the
> > > particular
> > > > > >> method
> > > > > >> >> > for annotating "preview" features -- I think clearly
> marking
> > > them
> > > > > as
> > > > > >> >> such
> > > > > >> >> > in parquet.thrift as well as reflecting that on the
> > > > > >> >> > https://parquet.apache.org/docs/file-format/versions/ page
> > > > somehow
> > > > > >> >> would
> > > > > >> >> > be adequate.
> > > > > >> >> >
> > > > > >> >> > Andrew
> > > > > >> >> >
> > > > > >> >> > On Thu, Jul 30, 2026 at 11:50 AM Andrew Lamb <
> > > > > andrewlamb11@gmail.com
> > > > > >> >
> > > > > >> >> > wrote:
> > > > > >> >> >
> > > > > >> >> >> In my mind ALP is in the same position as other encodings
> > that
> > > > > were
> > > > > >> >> added
> > > > > >> >> >> recently[1] (such as BYTE_STREAM_SPLIT for non floating
> > point
> > > > > types)
> > > > > >> >> >>
> > > > > >> >> >> That is to say it will be available in the latest open
> > source
> > > > > >> writers
> > > > > >> >> and
> > > > > >> >> >> readers, but realistically not widely used in the
> ecosystem
> > > > where
> > > > > >> >> >> compatibility is a concern (a large part)
> > > > > >> >> >>
> > > > > >> >> >> I view Julian's work[2][3] to gather consensus on how to
> > name
> > > > > >> >> >> combinations of features (e.g. "versions") as the
> mechanism
> > by
> > > > > >> which we
> > > > > >> >> >> will accelerate the adoption of this change (along with
> > other
> > > > > >> forward
> > > > > >> >> >> incompatible changes) across the ecosystem.
> > > > > >> >> >>
> > > > > >> >> >> > Will we exclude this change when cutting parquet-format
> > > > v2.14.0?
> > > > > >> >> >>
> > > > > >> >> >> No I don't think so -- I would expect it to go out in the
> > next
> > > > > >> >> >> parquet-format version as do other changes in the spec.
> > > > > >> >> >>
> > > > > >> >> >> Andrew
> > > > > >> >> >>
> > > > > >> >> >> [1]:
> https://parquet.apache.org/docs/file-format/versions/
> > > > > >> >> >> [2]:
> > > > > >>
> https://lists.apache.org/thread/xmnj8h0h8ozmrgox5tydhhs7yz7q2hss
> > > > > >> >> >> [3]:
> > > > > >>
> https://lists.apache.org/thread/qlf8lg90gqq41lllyy9mk2f0skqvftqj
> > > > > >> >> >>
> > > > > >> >> >>
> > > > > >> >> >> On Thu, Jul 30, 2026 at 3:51 AM Divjot Arora via dev <
> > > > > >> >> >> dev@parquet.apache.org> wrote:
> > > > > >> >> >>
> > > > > >> >> >>> Hi Andrew,
> > > > > >> >> >>>
> > > > > >> >> >>> How does this interact with the ongoing versioning
> > > discussion?
> > > > > >> Given
> > > > > >> >> that
> > > > > >> >> >>> new encodings are not forward compatible, I would assume
> > ALP
> > > is
> > > > > >> >> included
> > > > > >> >> >>> in
> > > > > >> >> >>> Parquet V<next> (V3?), but it's not clear to me how that
> > > works
> > > > if
> > > > > >> it's
> > > > > >> >> >>> merged to the parquet-format master branch now. Will we
> > > exclude
> > > > > >> this
> > > > > >> >> >>> change
> > > > > >> >> >>> when cutting parquet-format v2.14.0?
> > > > > >> >> >>>
> > > > > >> >> >>> -- Div
> > > > > >> >> >>>
> > > > > >> >> >>> On Thu, Jul 30, 2026 at 8:51 AM Andrew Lamb <
> > > > > >> andrewlamb11@gmail.com>
> > > > > >> >> >>> wrote:
> > > > > >> >> >>>
> > > > > >> >> >>> > Hello,
> > > > > >> >> >>> >
> > > > > >> >> >>> > As an update, thanks to the epic work from many
> > > contributors,
> > > > > >> >> >>> especially
> > > > > >> >> >>> > Prateek, I just merged the parquet-format change for
> > > ALP[1].
> > > > We
> > > > > >> >> plan to
> > > > > >> >> >>> > make additional editorial The remaining items I know of
> > are
> > > > > >> merging
> > > > > >> >> the
> > > > > >> >> >>> > examples from parquet-testing[2], and then of course
> the
> > > > > various
> > > > > >> >> >>> > implementations
> > > > > >> >> >>> >
> > > > > >> >> >>> > Thanks again everyone. This is a great step forward
> > > > > >> >> >>> > Andrew
> > > > > >> >> >>> >
> > > > > >> >> >>> > [1]: https://github.com/apache/parquet-format/pull/557
> > > > > >> >> >>> > [2]:
> https://github.com/apache/parquet-testing/pull/100
> > > > > >> >> >>> >
> > > > > >> >> >>> > On Tue, Jul 21, 2026 at 4:39 PM Julien Le Dem <
> > > > > julien@apache.org
> > > > > >> >
> > > > > >> >> >>> wrote:
> > > > > >> >> >>> >
> > > > > >> >> >>> > > It was also in my spam folder... :(
> > > > > >> >> >>> > > I've seen a bunch of apache list emails go to my spam
> > > > > recently.
> > > > > >> >> >>> > >
> > > > > >> >> >>> > > On Tue, Jul 21, 2026 at 10:58 AM Russell Spitzer <
> > > > > >> >> >>> > > russell.spitzer@gmail.com>
> > > > > >> >> >>> > > wrote:
> > > > > >> >> >>> > >
> > > > > >> >> >>> > > > Yes, my hope was that my vote would kick it out of
> > spam
> > > > but
> > > > > >> we
> > > > > >> >> >>> > definitely
> > > > > >> >> >>> > > > had two other votes that also went to spam for me.
> > > > > >> >> >>> > > >
> > > > > >> >> >>> > > > For anyone looking the thread is here
> > > > > >> >> >>> > > >
> > > > > >> >>
> > https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt
> > > > > >> >> >>> > > >
> > > > > >> >> >>> > > > On Tue, Jul 21, 2026 at 10:57 AM Micah Kornfield <
> > > > > >> >> >>> > emkornfield@gmail.com>
> > > > > >> >> >>> > > > wrote:
> > > > > >> >> >>> > > >
> > > > > >> >> >>> > > > > I've seen other's votes not go to spam, so
> > hopefully
> > > > > people
> > > > > >> >> will
> > > > > >> >> >>> see
> > > > > >> >> >>> > it
> > > > > >> >> >>> > > > in
> > > > > >> >> >>> > > > > there inboxes (or see this thread and look in
> there
> > > > > spam).
> > > > > >> >> >>> > > > >
> > > > > >> >> >>> > > > > On Tue, Jul 21, 2026 at 10:52 AM PRATEEK GAUR <
> > > > > >> >> >>> prtkgaur@gmail.com>
> > > > > >> >> >>> > > > wrote:
> > > > > >> >> >>> > > > >
> > > > > >> >> >>> > > > > > Okay,
> > > > > >> >> >>> > > > > >
> > > > > >> >> >>> > > > > > I sent out the voting email yesterday and so
> far
> > > I've
> > > > > >> heard
> > > > > >> >> >>> from 3
> > > > > >> >> >>> > > > people
> > > > > >> >> >>> > > > > > that it ended up in their "spam folder". Dev
> VPN
> > to
> > > > be
> > > > > >> >> blamed
> > > > > >> >> >>> for
> > > > > >> >> >>> > > it?.
> > > > > >> >> >>> > > > > >
> > > > > >> >> >>> > > > > > What's the protocol here?
> > > > > >> >> >>> > > > > > I let people hunt for the email in their spam
> > > folder
> > > > or
> > > > > >> send
> > > > > >> >> >>> out
> > > > > >> >> >>> > > > another
> > > > > >> >> >>> > > > > > one?
> > > > > >> >> >>> > > > > >
> > > > > >> >> >>> > > > > > Best
> > > > > >> >> >>> > > > > > Prateek
> > > > > >> >> >>> > > > > >
> > > > > >> >> >>> > > > > > On Sun, Jul 19, 2026 at 10:47 PM PRATEEK GAUR <
> > > > > >> >> >>> prtkgaur@gmail.com>
> > > > > >> >> >>> > > > > wrote:
> > > > > >> >> >>> > > > > >
> > > > > >> >> >>> > > > > > > Hi team,
> > > > > >> >> >>> > > > > > >
> > > > > >> >> >>> > > > > > > Vinoo has added the tests which were
> requested
> > in
> > > > the
> > > > > >> last
> > > > > >> >> >>> round
> > > > > >> >> >>> > of
> > > > > >> >> >>> > > > > > review.
> > > > > >> >> >>> > > > > > > In his own words.
> > > > > >> >> >>> > > > > > >
> > > > > >> >> >>> > > > > > > "
> > > > > >> >> >>> > > > > > > The extreme value tests Micah wanted are done
> > and
> > > > > >> pushed.
> > > > > >> >> I
> > > > > >> >> >>> added
> > > > > >> >> >>> > > > > > coverage
> > > > > >> >> >>> > > > > > > for values that need the full FOR bit width
> > after
> > > > the
> > > > > >> >> frame
> > > > > >> >> >>> of
> > > > > >> >> >>> > > > > reference
> > > > > >> >> >>> > > > > > is
> > > > > >> >> >>> > > > > > > applied: 64-bit (the case where the signed
> max
> > > > minus
> > > > > >> min
> > > > > >> >> >>> > > subtraction
> > > > > >> >> >>> > > > > > > overflows), 63-bit (the non-overflow case),
> and
> > > > > 32-bit
> > > > > >> for
> > > > > >> >> >>> > floats.
> > > > > >> >> >>> > > > All
> > > > > >> >> >>> > > > > > > lossless with zero exceptions. I also
> verified
> > > them
> > > > > >> >> through
> > > > > >> >> >>> Arrow
> > > > > >> >> >>> > > C++
> > > > > >> >> >>> > > > > ALP
> > > > > >> >> >>> > > > > > > reader (the local build), so the Java-written
> > > > extreme
> > > > > >> >> columns
> > > > > >> >> >>> > > decode
> > > > > >> >> >>> > > > > > > correctly in C++, bit-exact against the
> > expected
> > > > > >> values."
> > > > > >> >> >>> > > > > > >
> > > > > >> >> >>> > > > > > > State of the PR (#3397):
> > > > > >> >> >>> > > > > > >
> > > > > >> >> >>> > > > > > > - Merges cleanly with master (I pulled
> > master
> > > in
> > > > > and
> > > > > >> >> >>> resolved
> > > > > >> >> >>> > > the
> > > > > >> >> >>> > > > > > > conflicts).
> > > > > >> >> >>> > > > > > > - 41 of 44 review threads resolved. The 3
> > > > > remaining
> > > > > >> are
> > > > > >> >> >>> > > > intentional:
> > > > > >> >> >>> > > > > > > one deferred perf nit, one that stays open
> > > until
> > > > > >> >> >>> > parquet-format
> > > > > >> >> >>> > > > > ships
> > > > > >> >> >>> > > > > > ALP,
> > > > > >> >> >>> > > > > > >
> > > > > >> >> >>> > > > > > > "
> > > > > >> >> >>> > > > > > >
> > > > > >> >> >>> > > > > > > With this both the
> > > > > >> >> >>> > > > > > > 1) C++ PR :
> > > > > >> >> >>> https://github.com/apache/arrow/pull/48345/changes
> > > > > >> >> >>> > > > > > > 2) Java PR :
> > > > > >> >> >>> https://github.com/apache/parquet-java/pull/3397
> > > > > >> >> >>> > > > > > >
> > > > > >> >> >>> > > > > > > Are cross language tested and range of test
> > > > > scenarios.
> > > > > >> >> >>> > > > > > >
> > > > > >> >> >>> > > > > > > Best
> > > > > >> >> >>> > > > > > > Prateek
> > > > > >> >> >>> > > > > > >
> > > > > >> >> >>> > > > > > >
> > > > > >> >> >>> > > > > > > On Mon, Jul 13, 2026 at 8:38 AM PRATEEK GAUR
> <
> > > > > >> >> >>> prtkgaur@gmail.com
> > > > > >> >> >>> > >
> > > > > >> >> >>> > > > > wrote:
> > > > > >> >> >>> > > > > > >
> > > > > >> >> >>> > > > > > >> Thanks Matt,
> > > > > >> >> >>> > > > > > >>
> > > > > >> >> >>> > > > > > >> I'll be on it this week.
> > > > > >> >> >>> > > > > > >>
> > > > > >> >> >>> > > > > > >> Best
> > > > > >> >> >>> > > > > > >> Prateek
> > > > > >> >> >>> > > > > > >>
> > > > > >> >> >>> > > > > > >> On Mon, Jul 13, 2026 at 8:36 AM Matt Topol <
> > > > > >> >> >>> > > zotthewizard@gmail.com>
> > > > > >> >> >>> > > > > > >> wrote:
> > > > > >> >> >>> > > > > > >>
> > > > > >> >> >>> > > > > > >>> There is also the Go implementation
> submitted
> > > by
> > > > > >> Arnav (
> > > > > >> >> >>> > > > > > >>>
> https://github.com/apache/arrow-go/pull/704)
> > > > which
> > > > > >> is
> > > > > >> >> >>> waiting
> > > > > >> >> >>> > > for
> > > > > >> >> >>> > > > > > >>> updates.
> > > > > >> >> >>> > > > > > >>>
> > > > > >> >> >>> > > > > > >>> Just wanted to make sure it didn't get lost
> > > here.
> > > > > >> >> >>> > > > > > >>>
> > > > > >> >> >>> > > > > > >>> --Matt
> > > > > >> >> >>> > > > > > >>>
> > > > > >> >> >>> > > > > > >>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK
> > GAUR <
> > > > > >> >> >>> > > prtkgaur@gmail.com>
> > > > > >> >> >>> > > > > > >>> wrote:
> > > > > >> >> >>> > > > > > >>>
> > > > > >> >> >>> > > > > > >>> > Hi all,
> > > > > >> >> >>> > > > > > >>> >
> > > > > >> >> >>> > > > > > >>> > I'd like to share a status update on the
> > ALP
> > > > > >> encoding
> > > > > >> >> >>> effort
> > > > > >> >> >>> > > and
> > > > > >> >> >>> > > > > get
> > > > > >> >> >>> > > > > > a
> > > > > >> >> >>> > > > > > >>> feel
> > > > > >> >> >>> > > > > > >>> > from the community before starting a
> formal
> > > > vote.
> > > > > >> >> >>> > > > > > >>> >
> > > > > >> >> >>> > > > > > >>> > We now have the specification plus
> > > > > implementations
> > > > > >> in
> > > > > >> >> >>> both
> > > > > >> >> >>> > > > > languages:
> > > > > >> >> >>> > > > > > >>> >
> > > > > >> >> >>> > > > > > >>> > - Spec:
> > > apache/parquet-format#557 —
> > > > > >> GH-533
> > > > > >> >> >>> Add
> > > > > >> >> >>> > ALP
> > > > > >> >> >>> > > > > > encoding
> > > > > >> >> >>> > > > > > >>> > specification
> > > > > >> >> >>> > > > > > >>> > - C++ (Arrow): apache/arrow#48345 —
> > > > GH-48701
> > > > > >> >> >>> > > [C++][Parquet]
> > > > > >> >> >>> > > > > Add
> > > > > >> >> >>> > > > > > >>> ALPpd
> > > > > >> >> >>> > > > > > >>> > encoding
> > > > > >> >> >>> > > > > > >>> > - Java:
> > apache/parquet-java#3397
> > > —
> > > > > >> Parquet
> > > > > >> >> >>> Java
> > > > > >> >> >>> > ALP
> > > > > >> >> >>> > > > > > >>> > Implementation (by Vinoo)
> > > > > >> >> >>> > > > > > >>> >
> > > > > >> >> >>> > > > > > >>> > Current state:
> > > > > >> >> >>> > > > > > >>> >
> > > > > >> >> >>> > > > > > >>> > - The C++ implementation has been
> > through
> > > > 3-4
> > > > > >> >> rounds
> > > > > >> >> >>> of
> > > > > >> >> >>> > > > review.
> > > > > >> >> >>> > > > > > >>> > - The Java implementation has been
> > > tracking
> > > > > >> well
> > > > > >> >> and
> > > > > >> >> >>> has
> > > > > >> >> >>> > > had
> > > > > >> >> >>> > > > > > >>> initial
> > > > > >> >> >>> > > > > > >>> > reviews from contributors, with no
> > > major
> > > > > >> >> remaining
> > > > > >> >> >>> > > issues.
> > > > > >> >> >>> > > > > > >>> > - Cross-language compatibility tests
> > are
> > > > > >> passing:
> > > > > >> >> the
> > > > > >> >> >>> > Arrow
> > > > > >> >> >>> > > > C++
> > > > > >> >> >>> > > > > > >>> decoder
> > > > > >> >> >>> > > > > > >>> > reads Java-written data bit-exactly
> > > > across
> > > > > >> >> ~1.56M
> > > > > >> >> >>> > values
> > > > > >> >> >>> > > > and
> > > > > >> >> >>> > > > > 18
> > > > > >> >> >>> > > > > > >>> > fixtures, covering V1 and V2 pages,
> > > > > multiple
> > > > > >> >> vector
> > > > > >> >> >>> > > sizes,
> > > > > >> >> >>> > > > > and
> > > > > >> >> >>> > > > > > >>> > several
> > > > > >> >> >>> > > > > > >>> > real datasets — zero mismatches.
> > > > > >> >> >>> > > > > > >>> >
> > > > > >> >> >>> > > > > > >>> > As raised in the java-pr we will be
> adding
> > > > > >> coverage
> > > > > >> >> for
> > > > > >> >> >>> > > extreme
> > > > > >> >> >>> > > > > > values
> > > > > >> >> >>> > > > > > >>> > (those requiring
> > > > > >> >> >>> > > > > > >>> > 63-64 bits after FOR is applied) before
> we
> > > > > close a
> > > > > >> >> vote.
> > > > > >> >> >>> > We'll
> > > > > >> >> >>> > > > aim
> > > > > >> >> >>> > > > > > to
> > > > > >> >> >>> > > > > > >>> get
> > > > > >> >> >>> > > > > > >>> > that
> > > > > >> >> >>> > > > > > >>> > done in parallel.
> > > > > >> >> >>> > > > > > >>> >
> > > > > >> >> >>> > > > > > >>> > Plan: unless there are objections, I
> intend
> > > to
> > > > > >> start a
> > > > > >> >> >>> vote
> > > > > >> >> >>> > at
> > > > > >> >> >>> > > > the
> > > > > >> >> >>> > > > > > end
> > > > > >> >> >>> > > > > > >>> of
> > > > > >> >> >>> > > > > > >>> > this week/early
> > > > > >> >> >>> > > > > > >>> > next week.
> > > > > >> >> >>> > > > > > >>> >
> > > > > >> >> >>> > > > > > >>> > Thanks
> > > > > >> >> >>> > > > > > >>> > Prateek
> > > > > >> >> >>> > > > > > >>> >
> > > > > >> >> >>> > > > > > >>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK
> > GAUR <
> > > > > >> >> >>> > > prtkgaur@gmail.com>
> > > > > >> >> >>> > > > > > >>> wrote:
> > > > > >> >> >>> > > > > > >>> >
> > > > > >> >> >>> > > > > > >>> > > Hi Team,
> > > > > >> >> >>> > > > > > >>> > >
> > > > > >> >> >>> > > > > > >>> > > Just wanted to provide some updates on
> > ALP.
> > > > > >> >> >>> > > > > > >>> > > Micah and I have done 3-4 rounds of
> > review
> > > > for
> > > > > >> the
> > > > > >> >> c++
> > > > > >> >> >>> PR :
> > > > > >> >> >>> > > > > > >>> > >
> > https://github.com/apache/arrow/pull/48345
> > > > > >> >> >>> > > > > > >>> > >
> > > > > >> >> >>> > > > > > >>> > > For the Java implementation Vinoo has
> > been
> > > > > >> working
> > > > > >> >> on
> > > > > >> >> >>> > > following
> > > > > >> >> >>> > > > > PR
> > > > > >> >> >>> > > > > > :
> > > > > >> >> >>> > > > > > >>> > >
> > > > > https://github.com/apache/parquet-java/pull/3397
> > > > > >> >> >>> > > > > > >>> > >
> > > > > >> >> >>> > > > > > >>> > > Best
> > > > > >> >> >>> > > > > > >>> > > Prateek
> > > > > >> >> >>> > > > > > >>> > >
> > > > > >> >> >>> > > > > > >>> > > On Tue, May 5, 2026 at 1:50 PM Micah
> > > > Kornfield
> > > > > <
> > > > > >> >> >>> > > > > > >>> emkornfield@gmail.com>
> > > > > >> >> >>> > > > > > >>> > > wrote:
> > > > > >> >> >>> > > > > > >>> > >
> > > > > >> >> >>> > > > > > >>> > >> Hi Antoine,
> > > > > >> >> >>> > > > > > >>> > >>
> > > > > >> >> >>> > > > > > >>> > >> > Apologies if the question was
> already
> > > > asked,
> > > > > >> but
> > > > > >> >> >>> should
> > > > > >> >> >>> > we
> > > > > >> >> >>> > > > > care
> > > > > >> >> >>> > > > > > >>> about
> > > > > >> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be
> > more
> > > > > >> >> efficient
> > > > > >> >> >>> than
> > > > > >> >> >>> > > > > FLOAT16
> > > > > >> >> >>> > > > > > +
> > > > > >> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> > > > > >> >> >>> > > > > > >>> > >>
> > > > > >> >> >>> > > > > > >>> > >> It was. We thought we could defer it
> > for
> > > > the
> > > > > >> >> >>> following
> > > > > >> >> >>> > > > reasons:
> > > > > >> >> >>> > > > > > >>> > >> 1. It's not clear there are a lot of
> > easy
> > > > > >> >> reference
> > > > > >> >> >>> > > datasets
> > > > > >> >> >>> > > > to
> > > > > >> >> >>> > > > > > >>> test
> > > > > >> >> >>> > > > > > >>> > its
> > > > > >> >> >>> > > > > > >>> > >> effectiveness. It does look like
> there
> > > > might
> > > > > be
> > > > > >> >> one
> > > > > >> >> >>> or
> > > > > >> >> >>> > two
> > > > > >> >> >>> > > on
> > > > > >> >> >>> > > > > > >>> > huggingface
> > > > > >> >> >>> > > > > > >>> > >> (e.g.
> > > > > >> >> https://huggingface.co/datasets/kikitora/curdie
> > > > > >> >> >>> ).
> > > > > >> >> >>> > > > > > >>> > >> 2. It seemed likely that float 16 was
> > > more
> > > > > >> likely
> > > > > >> >> >>> used
> > > > > >> >> >>> > for
> > > > > >> >> >>> > > > > values
> > > > > >> >> >>> > > > > > >>> that
> > > > > >> >> >>> > > > > > >>> > >> were less likely to reduce to decimal
> > > > values.
> > > > > >> >> >>> > > > > > >>> > >> 3. It could be added as an extension
> > > later
> > > > if
> > > > > >> >> needed.
> > > > > >> >> >>> > > > > > >>> > >>
> > > > > >> >> >>> > > > > > >>> > >> Cheers,
> > > > > >> >> >>> > > > > > >>> > >> Micah
> > > > > >> >> >>> > > > > > >>> > >>
> > > > > >> >> >>> > > > > > >>> > >>
> > > > > >> >> >>> > > > > > >>> > >> On Tue, May 5, 2026 at 1:40 PM Antoine
> > > > Pitrou
> > > > > <
> > > > > >> >> >>> > > > > antoine@python.org
> > > > > >> >> >>> > > > > > >
> > > > > >> >> >>> > > > > > >>> > wrote:
> > > > > >> >> >>> > > > > > >>> > >>
> > > > > >> >> >>> > > > > > >>> > >> >
> > > > > >> >> >>> > > > > > >>> > >> > Hello,
> > > > > >> >> >>> > > > > > >>> > >> >
> > > > > >> >> >>> > > > > > >>> > >> > Apologies if the question was
> already
> > > > asked,
> > > > > >> but
> > > > > >> >> >>> should
> > > > > >> >> >>> > we
> > > > > >> >> >>> > > > > care
> > > > > >> >> >>> > > > > > >>> about
> > > > > >> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be
> > more
> > > > > >> >> efficient
> > > > > >> >> >>> than
> > > > > >> >> >>> > > > > FLOAT16
> > > > > >> >> >>> > > > > > +
> > > > > >> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> > > > > >> >> >>> > > > > > >>> > >> >
> > > > > >> >> >>> > > > > > >>> > >> > Regards
> > > > > >> >> >>> > > > > > >>> > >> >
> > > > > >> >> >>> > > > > > >>> > >> > Antoine.
> > > > > >> >> >>> > > > > > >>> > >> >
> > > > > >> >> >>> > > > > > >>> > >> >
> > > > > >> >> >>> > > > > > >>> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR
> a
> > > > écrit
> > > > > :
> > > > > >> >> >>> > > > > > >>> > >> > > Thanks Andrew and Micah for review
> > > > > feedback
> > > > > >> on
> > > > > >> >> >>> the two
> > > > > >> >> >>> > > > PR's
> > > > > >> >> >>> > > > > > >>> > >> > > 1) (c++ arrow repo)
> > > > > >> >> >>> > > > > > >>> > >>
> > > > > >> https://github.com/apache/arrow/pull/48345/changes
> > > > > >> >> >>> > > > > > >>> > >> > > 2) (parquet-format repo)
> > > > > >> >> >>> > > > > > >>> > >> >
> > > > > >> >> https://github.com/apache/parquet-format/pull/557
> > > > > >> >> >>> > > > > > >>> > >> > >
> > > > > >> >> >>> > > > > > >>> > >> > > I have addressed all (unless I
> > missed
> > > > > >> >> something)
> > > > > >> >> >>> > > comments
> > > > > >> >> >>> > > > on
> > > > > >> >> >>> > > > > > >>> the two
> > > > > >> >> >>> > > > > > >>> > >> > PR's.
> > > > > >> >> >>> > > > > > >>> > >> > >
> > > > > >> >> >>> > > > > > >>> > >> > > Best
> > > > > >> >> >>> > > > > > >>> > >> > > Prateek
> > > > > >> >> >>> > > > > > >>> > >> > >
> > > > > >> >> >>> > > > > > >>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM
> > > PRATEEK
> > > > > >> GAUR <
> > > > > >> >> >>> > > > > > >>> prtkgaur@gmail.com>
> > > > > >> >> >>> > > > > > >>> > >> wrote:
> > > > > >> >> >>> > > > > > >>> > >> > >
> > > > > >> >> >>> > > > > > >>> > >> > >> Thanks Andrew and Micah.
> > > > > >> >> >>> > > > > > >>> > >> > >>
> > > > > >> >> >>> > > > > > >>> > >> > >> `fair amount of feedback on at
> > least
> > > > the
> > > > > >> >> >>> > > implementations`
> > > > > >> >> >>> > > > > > >>> > >> > >> For the c++ I have already
> started
> > > > > >> addressing
> > > > > >> >> the
> > > > > >> >> >>> > > > > feedback, I
> > > > > >> >> >>> > > > > > >>> > should
> > > > > >> >> >>> > > > > > >>> > >> be
> > > > > >> >> >>> > > > > > >>> > >> > >> done with that Monday/Tuesday.
> > > > > >> >> >>> > > > > > >>> > >> > >> I think Vinoo too has been making
> > > good
> > > > > >> >> progress
> > > > > >> >> >>> on
> > > > > >> >> >>> > the
> > > > > >> >> >>> > > > Java
> > > > > >> >> >>> > > > > > >>> > >> > implementation.
> > > > > >> >> >>> > > > > > >>> > >> > >>
> > > > > >> >> >>> > > > > > >>> > >> > >> Best
> > > > > >> >> >>> > > > > > >>> > >> > >> Prateek
> > > > > >> >> >>> > > > > > >>> > >> > >>
> > > > > >> >> >>> > > > > > >>> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM
> > > Andrew
> > > > > >> Lamb <
> > > > > >> >> >>> > > > > > >>> > andrewlamb11@gmail.com
> > > > > >> >> >>> > > > > > >>> > >> >
> > > > > >> >> >>> > > > > > >>> > >> > >> wrote:
> > > > > >> >> >>> > > > > > >>> > >> > >>
> > > > > >> >> >>> > > > > > >>> > >> > >>> Got it. Thank you for the
> > > > clarification
> > > > > >> -- I
> > > > > >> >> >>> will
> > > > > >> >> >>> > try
> > > > > >> >> >>> > > > and
> > > > > >> >> >>> > > > > > look
> > > > > >> >> >>> > > > > > >>> > into
> > > > > >> >> >>> > > > > > >>> > >> the
> > > > > >> >> >>> > > > > > >>> > >> > >>> spec and the Rust
> > implementation[1]
> > > in
> > > > > >> this
> > > > > >> >> next
> > > > > >> >> >>> > week
> > > > > >> >> >>> > > > > > >>> > >> > >>>
> > > > > >> >> >>> > > > > > >>> > >> > >>> [1]:
> > > > > >> >> >>> https://github.com/apache/arrow-rs/pull/9372
> > > > > >> >> >>> > > > > > >>> > >> > >>>
> > > > > >> >> >>> > > > > > >>> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM
> > > Micah
> > > > > >> >> >>> Kornfield <
> > > > > >> >> >>> > > > > > >>> > >> > emkornfield@gmail.com>
> > > > > >> >> >>> > > > > > >>> > >> > >>> wrote:
> > > > > >> >> >>> > > > > > >>> > >> > >>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>> Hi Andrew,
> > > > > >> >> >>> > > > > > >>> > >> > >>>> I think there is a fair amount
> of
> > > > > >> feedback
> > > > > >> >> on
> > > > > >> >> >>> at
> > > > > >> >> >>> > > least
> > > > > >> >> >>> > > > > the
> > > > > >> >> >>> > > > > > >>> > >> > >>>> implementations, typically I
> > think
> > > > > we've
> > > > > >> >> waited
> > > > > >> >> >>> > till
> > > > > >> >> >>> > > > they
> > > > > >> >> >>> > > > > > are
> > > > > >> >> >>> > > > > > >>> > >> close to
> > > > > >> >> >>> > > > > > >>> > >> > >>>> mergeable before a final vote.
> > > > > >> Otherwise I
> > > > > >> >> >>> agree
> > > > > >> >> >>> > we
> > > > > >> >> >>> > > > are
> > > > > >> >> >>> > > > > > very
> > > > > >> >> >>> > > > > > >>> > >> close.
> > > > > >> >> >>> > > > > > >>> > >> > >>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>> -Micah
> > > > > >> >> >>> > > > > > >>> > >> > >>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>> On Saturday, April 25, 2026,
> > Andrew
> > > > > Lamb
> > > > > >> <
> > > > > >> >> >>> > > > > > >>> andrewlamb11@gmail.com
> > > > > >> >> >>> > > > > > >>> > >
> > > > > >> >> >>> > > > > > >>> > >> > wrote:
> > > > > >> >> >>> > > > > > >>> > >> > >>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>> Thanks Prateek,
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>> I think from this content it
> > looks
> > > > to
> > > > > me
> > > > > >> >> like
> > > > > >> >> >>> we
> > > > > >> >> >>> > are
> > > > > >> >> >>> > > > > ready
> > > > > >> >> >>> > > > > > >>> to
> > > > > >> >> >>> > > > > > >>> > >> start a
> > > > > >> >> >>> > > > > > >>> > >> > >>>>> vote to explicitly accept ALP
> > into
> > > > > >> Parquet
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>> Does anyone know of a reason
> we
> > > > should
> > > > > >> >> >>> postpone it
> > > > > >> >> >>> > > for
> > > > > >> >> >>> > > > > > >>> longer?
> > > > > >> >> >>> > > > > > >>> > >> > >>>>> Perhaps someone needs some
> more
> > > time
> > > > > to
> > > > > >> >> >>> review?
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>> Andrew
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>> On Wed, Apr 22, 2026 at
> 1:00 PM
> > > > > PRATEEK
> > > > > >> >> GAUR <
> > > > > >> >> >>> > > > > > >>> > prtkgaur@gmail.com>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>> wrote:
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Hi team,
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Hope everyone is doing well.
> I
> > > got
> > > > a
> > > > > >> >> chance
> > > > > >> >> >>> to
> > > > > >> >> >>> > work
> > > > > >> >> >>> > > > > > >>> through all
> > > > > >> >> >>> > > > > > >>> > >> the
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> remaining feedback and update
> > the
> > > > > spec
> > > > > >> >> doc.
> > > > > >> >> >>> Here
> > > > > >> >> >>> > > are
> > > > > >> >> >>> > > > > the
> > > > > >> >> >>> > > > > > >>> new
> > > > > >> >> >>> > > > > > >>> > >> > artifacts
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 1) Spec document :
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> >
> > > > > >> >> >>> > > > > > >>> > >>
> > > > > >> >> >>> > > > > > >>> >
> > > > > >> >> >>> > > > > > >>>
> > > > > >> >> >>> > > > > >
> > > > > >> >> >>> > > > >
> > > > > >> >> >>> > > >
> > > > > >> >> >>> > >
> > > > > >> >> >>> >
> > > > > >> >> >>>
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 2) Spec document in parquet
> > > format
> > > > > >> repo :
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > https://github.com/apache/parquet-format/pull/557
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 3) Alp implementation in
> arrow
> > > c++
> > > > > >> repo :
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > https://github.com/apache/arrow/pull/48345/changes
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 4) Alp implementation in
> > > > parquet-java
> > > > > >> >> repo :
> > > > > >> >> >>> Work
> > > > > >> >> >>> > > for
> > > > > >> >> >>> > > > > > >>> Vinoo and
> > > > > >> >> >>> > > > > > >>> > >> > Julien
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > https://github.com/apache/parquet-java/pull/3397
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 5) PR with test and
> > benchmarking
> > > > > >> >> artifacts in
> > > > > >> >> >>> > > > > > >>> parquet-testing
> > > > > >> >> >>> > > > > > >>> > >> repo :
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > https://github.com/apache/parquet-testing/pull/100
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> And
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> - Go : Arnav just
> submitted
> > > an
> > > > in
> > > > > >> >> >>> progress
> > > > > >> >> >>> > > > > > >>> implementation
> > > > > >> >> >>> > > > > > >>> > in
> > > > > >> >> >>> > > > > > >>> > >> Go.
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> https://github.com/apache/arrow-go/pull/704
> > > > > >> >> >>> > (I
> > > > > >> >> >>> > > > > > haven't
> > > > > >> >> >>> > > > > > >>> > >> started
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> looking at it yet)
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> - Rust : I remember
> Andrew
> > > > > >> mentioned
> > > > > >> >> that
> > > > > >> >> >>> > this
> > > > > >> >> >>> > > > work
> > > > > >> >> >>> > > > > > is
> > > > > >> >> >>> > > > > > >>> also
> > > > > >> >> >>> > > > > > >>> > >> in
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> progress (So 4
> languages!)
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> *Arrow C++ implementation *
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> The PR is out and was also
> used
> > > by
> > > > > >> >> Antoine to
> > > > > >> >> >>> > > report
> > > > > >> >> >>> > > > > the
> > > > > >> >> >>> > > > > > >>> > numbers
> > > > > >> >> >>> > > > > > >>> > >> as
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> reported here. Micah and
> > > Konstantin
> > > > > >> have
> > > > > >> >> >>> given 1
> > > > > >> >> >>> > > > round
> > > > > >> >> >>> > > > > of
> > > > > >> >> >>> > > > > > >>> > >> feedback
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> and I'm addressing them
> today.
> > > > Please
> > > > > >> note
> > > > > >> >> >>> that
> > > > > >> >> >>> > the
> > > > > >> >> >>> > > > > > default
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> optimization flag for
> compiling
> > > is
> > > > O2
> > > > > >> and
> > > > > >> >> not
> > > > > >> >> >>> > Q3. I
> > > > > >> >> >>> > > > got
> > > > > >> >> >>> > > > > > >>> around
> > > > > >> >> >>> > > > > > >>> > >> 70%
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> performance improvement in
> the
> > > > > decoding
> > > > > >> >> speed
> > > > > >> >> >>> > when
> > > > > >> >> >>> > > > > using
> > > > > >> >> >>> > > > > > >>> the O3
> > > > > >> >> >>> > > > > > >>> > >> > flag.
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> *Parqet-MR Java
> implementation
> > > > > (working
> > > > > >> >> with
> > > > > >> >> >>> > Vinoo
> > > > > >> >> >>> > > > and
> > > > > >> >> >>> > > > > > >>> Julien)
> > > > > >> >> >>> > > > > > >>> > >> and
> > > > > >> >> >>> > > > > > >>> > >> > **Cross
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Language testing*
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Let me know if you have
> any
> > > > > >> questions
> > > > > >> >> or
> > > > > >> >> >>> > > > feedback.
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Now pasting some performance
> > > > numbers
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Table 1: C++ ALP Double
> > > Decode —
> > > > > >> >> Spotify
> > > > > >> >> >>> > Columns
> > > > > >> >> >>> > > > > > >>> (Graviton
> > > > > >> >> >>> > > > > > >>> > 3,
> > > > > >> >> >>> > > > > > >>> > >> ARM
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Neoverse V1)
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>>
> > > > > >> >> >>>
> > ┌──────────────────┬──────────────┬──────────────┬─────────┐
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ Column │ -O2
> > > (MB/s)
> > > > > │
> > > > > >> >> -O3
> > > > > >> >> >>> > > (MB/s) │
> > > > > >> >> >>> > > > > > >>> Speedup │
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>>
> > > > > >> >> >>>
> > ├──────────────────┼──────────────┼──────────────┼─────────┤
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ valence │
> > 3,155
> > > > > │
> > > > > >> >> >>> 5,523
> > > > > >> >> >>> > > │
> > > > > >> >> >>> > > > > > >>> 1.75x │
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ danceability │
> > 3,233
> > > > > │
> > > > > >> >> >>> 5,685
> > > > > >> >> >>> > > │
> > > > > >> >> >>> > > > > > >>> 1.76x │
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ energy │
> > 3,197
> > > > > │
> > > > > >> >> >>> 5,652
> > > > > >> >> >>> > > │
> > > > > >> >> >>> > > > > > >>> 1.77x │
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ loudness │
> > 3,186
> > > > > │
> > > > > >> >> >>> 5,473
> > > > > >> >> >>> > > │
> > > > > >> >> >>> > > > > > >>> 1.72x │
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>>
> > > > > >> >> >>>
> > └──────────────────┴──────────────┴──────────────┴─────────┘
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> On Wed, Feb 25, 2026 at
> 9:49 AM
> > > > > PRATEEK
> > > > > >> >> GAUR
> > > > > >> >> >>> <
> > > > > >> >> >>> > > > > > >>> > prtkgaur@gmail.com
> > > > > >> >> >>> > > > > > >>> > >> >
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> wrote:
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> @Micah Kornfield <
> > > > > >> emkornfield@gmail.com>
> > > > > >> >> :
> > > > > >> >> >>> Got
> > > > > >> >> >>> > > it.
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> @Andrew Lamb <
> > > > > andrewlamb11@gmail.com>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> Do you think it would be
> good
> > > to
> > > > > >> start
> > > > > >> >> >>> moving
> > > > > >> >> >>> > the
> > > > > >> >> >>> > > > > spec
> > > > > >> >> >>> > > > > > >>> > >> development
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> into
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> markdown format, in
> > preparation
> > > > for
> > > > > >> >> >>> finalizing
> > > > > >> >> >>> > > it?
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> Yes I'll update the numbers
> > for
> > > > some
> > > > > >> of
> > > > > >> >> the
> > > > > >> >> >>> > > > examples I
> > > > > >> >> >>> > > > > > >>> have in
> > > > > >> >> >>> > > > > > >>> > >> the
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> spec based
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> on the updated header size.
> > Then
> > > > we
> > > > > >> >> should
> > > > > >> >> >>> be
> > > > > >> >> >>> > good
> > > > > >> >> >>> > > > to
> > > > > >> >> >>> > > > > go
> > > > > >> >> >>> > > > > > >>> for
> > > > > >> >> >>> > > > > > >>> > the
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> markdown format.
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> Thanks everyone!
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> Andrew
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at
> > 7:28 PM
> > > > > >> PRATEEK
> > > > > >> >> >>> GAUR <
> > > > > >> >> >>> > > > > > >>> > >> prtkgaur@gmail.com>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> wrote:
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Hi team,
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 1) Andrew
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for working
> on
> > > test
> > > > > >> files.
> > > > > >> >> >>> My PR
> > > > > >> >> >>> > > did
> > > > > >> >> >>> > > > > add
> > > > > >> >> >>> > > > > > >>> all
> > > > > >> >> >>> > > > > > >>> > the
> > > > > >> >> >>> > > > > > >>> > >> > test
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> files I
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> used to benchmark on
> > > > datasets.
> > > > > >> >> Maybe
> > > > > >> >> >>> we
> > > > > >> >> >>> > can
> > > > > >> >> >>> > > > club
> > > > > >> >> >>> > > > > > it
> > > > > >> >> >>> > > > > > >>> > >> together.
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> WIll also
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> aid
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> cross language testing
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Kosta Tarasov
> working
> > > on
> > > > > Rust
> > > > > >> >> >>> > > > implementation.
> > > > > >> >> >>> > > > > > >>> This is
> > > > > >> >> >>> > > > > > >>> > >> > great.
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> Thanks
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 2) Antoine
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks a lot for
> > > reporting
> > > > > the
> > > > > >> >> >>> numbers
> > > > > >> >> >>> > on
> > > > > >> >> >>> > > > AMD.
> > > > > >> >> >>> > > > > > >>> Looks
> > > > > >> >> >>> > > > > > >>> > >> like
> > > > > >> >> >>> > > > > > >>> > >> > you
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> are
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> getting 8X the
> decoding
> > > > > >> >> performance of
> > > > > >> >> >>> > BSS.
> > > > > >> >> >>> > > > This
> > > > > >> >> >>> > > > > > is
> > > > > >> >> >>> > > > > > >>> > >> > amazing!!.
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for
> > acknowledging
> > > > the
> > > > > >> >> >>> sampling
> > > > > >> >> >>> > > > design.
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - I agree with you on
> > > > > >> Fastlanes. In
> > > > > >> >> >>> some
> > > > > >> >> >>> > > crude
> > > > > >> >> >>> > > > > > >>> > >> experiments I
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> didn't get
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> a good perf benefit
> from
> > > it
> > > > on
> > > > > >> >> >>> Graviton3
> > > > > >> >> >>> > > (but
> > > > > >> >> >>> > > > > > maybe
> > > > > >> >> >>> > > > > > >>> > there
> > > > > >> >> >>> > > > > > >>> > >> was
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> something
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> wrong with my
> > > > implementation).
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Locking the 16bit
> > > > exception
> > > > > >> >> >>> encoding for
> > > > > >> >> >>> > > the
> > > > > >> >> >>> > > > > > spec
> > > > > >> >> >>> > > > > > >>> in
> > > > > >> >> >>> > > > > > >>> > >> this
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> case.
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Awesome I think we
> > have
> > > > > solved
> > > > > >> >> for
> > > > > >> >> >>> all
> > > > > >> >> >>> > > open
> > > > > >> >> >>> > > > > > >>> questions
> > > > > >> >> >>> > > > > > >>> > >> minus
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> the
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> version byte :). (will
> > get
> > > > > back
> > > > > >> on
> > > > > >> >> >>> this
> > > > > >> >> >>> > > soon)
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 3) Micah
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - FastLanes : The
> > current
> > > > spec
> > > > > >> does
> > > > > >> >> >>> allow
> > > > > >> >> >>> > > for
> > > > > >> >> >>> > > > > > using
> > > > > >> >> >>> > > > > > >>> > >> FastLane
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> with the
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> configurable enum
> value
> > > for
> > > > > >> >> layout. We
> > > > > >> >> >>> > > should
> > > > > >> >> >>> > > > be
> > > > > >> >> >>> > > > > > >>> able to
> > > > > >> >> >>> > > > > > >>> > >> > inject
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> any
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> layout
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> in the current design.
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Working on resolving all
> > > > remaining
> > > > > >> open
> > > > > >> >> >>> > comments
> > > > > >> >> >>> > > > on
> > > > > >> >> >>> > > > > > the
> > > > > >> >> >>> > > > > > >>> spec
> > > > > >> >> >>> > > > > > >>> > >> this
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> week.
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Best
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Prateek
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at
> > > 3:37 AM
> > > > > >> Steve
> > > > > >> >> >>> > Loughran <
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> stevel@cloudera.com>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> wrote:
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at
> > 18:12,
> > > > > Micah
> > > > > >> >> >>> Kornfield
> > > > > >> >> >>> > <
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> emkornfield@gmail.com>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> wrote:
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> It looks like the actual
> > > issue
> > > > > >> >> >>> described for
> > > > > >> >> >>> > > ORC
> > > > > >> >> >>> > > > > in
> > > > > >> >> >>> > > > > > >>> the
> > > > > >> >> >>> > > > > > >>> > >> paper
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> is that
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> it
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> has multiple
> sub-encodings
> > > in
> > > > a
> > > > > >> >> batch.
> > > > > >> >> >>> This
> > > > > >> >> >>> > > is
> > > > > >> >> >>> > > > > > >>> different
> > > > > >> >> >>> > > > > > >>> > >> then
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> the
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> design
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> proposed here where
> there
> > is
> > > > > still
> > > > > >> >> fixed
> > > > > >> >> >>> > > > encoding
> > > > > >> >> >>> > > > > > per
> > > > > >> >> >>> > > > > > >>> page
> > > > > >> >> >>> > > > > > >>> > >> in
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> parquet.
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> Given reasonably sized
> > > pages I
> > > > > >> don't
> > > > > >> >> >>> think
> > > > > >> >> >>> > > > branch
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> misprediction should
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> be a
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> big issue for new
> > encodings.
> > > > I
> > > > > >> agree
> > > > > >> >> >>> that
> > > > > >> >> >>> > we
> > > > > >> >> >>> > > > > should
> > > > > >> >> >>> > > > > > >>> be
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> conservative in
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> general for adding new
> > > > > encodings.
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> +1
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >
> > > > > >> >> >>> > > > > > >>> > >> >
> > > > > >> >> >>> > > > > > >>> > >> >
> > > > > >> >> >>> > > > > > >>> > >> >
> > > > > >> >> >>> > > > > > >>> > >>
> > > > > >> >> >>> > > > > > >>> > >
> > > > > >> >> >>> > > > > > >>> >
> > > > > >> >> >>> > > > > > >>>
> > > > > >> >> >>> > > > > > >>
> > > > > >> >> >>> > > > > >
> > > > > >> >> >>> > > > >
> > > > > >> >> >>> > > >
> > > > > >> >> >>> > >
> > > > > >> >> >>> >
> > > > > >> >> >>>
> > > > > >> >> >>
> > > > > >> >>
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Vinoo Ganesh <vi...@gmail.com>.
Quick update re: the Java impl. Thanks for the feedback, on
https://github.com/apache/parquet-java/pull/3397, Russel! I pulled
https://github.com/apache/parquet-java/pull/3791 into its own PR (thanks
for the quick approval too, Gang). Assuming no objections, I think we can
merge #3791, and then go ahead of #3397.
Thanks,
Vinoo Ganesh | vinoo.ganesh@gmail.com
<Vi...@gmail.com>
On Tue, Sep 8, 2026 at 10:54 AM PRATEEK GAUR <pr...@gmail.com> wrote:
> [Rust] Awesome thanks Andrew, Kostas and Devin.
> [Java] Thanks Gang for the feedback on Java PR.
> [C++] Thanks Gang for another round of feedback on the C++ one.
> [Go] Arnav and I will be working on the golang impl this week.
>
> This is great progress all around.
>
> On Sat, Sep 5, 2026 at 1:26 PM Vinoo Ganesh <vi...@gmail.com>
> wrote:
>
> > Thanks all - I'm addressing Gang's comments on
> > https://github.com/apache/parquet-java/pull/3397 now, so hopefully well
> > also be in a good state for merging the java implementation soon.
> >
> > Thanks,
> > Vinoo Ganesh | vinoo.ganesh@gmail.com
> >
> > <Vi...@gmail.com>
> >
> >
> > On Sat, Sep 5, 2026 at 8:56 AM Andrew Lamb <an...@gmail.com>
> wrote:
> >
> > > We just merged the Rust implementation[1] from Kosta and Devan. With
> the
> > > imminent release of Parquet Format 2.14.0 I think we'll be well on our
> > way.
> > >
> > > Thanks again for pushing this along,
> > > Andrew
> > >
> > > [1]: https://github.com/apache/arrow-rs/pull/9372
> > >
> > > On Fri, Sep 4, 2026 at 5:42 PM PRATEEK GAUR <pr...@gmail.com>
> wrote:
> > >
> > > > Hi team,
> > > >
> > > > Thanks Micah for multiple rounds of review and thanks Gang for
> taking a
> > > > look at the PR and giving feedback. I've addressed all comments in
> the
> > > PR.
> > > > Looking forward to the next round of feedback (or approval). Would
> like
> > > to
> > > > get it in soon.
> > > >
> > > > https://github.com/apache/arrow/pull/48345/changes
> > > >
> > > > With this done and after talking to Arnav, I plan to spend some time
> on
> > > the
> > > > ALP implementation in the arrow-go repository helping spread the
> > support
> > > to
> > > > more languages.
> > > >
> > > > Best
> > > > Prateek
> > > >
> > > > On Sun, Aug 2, 2026 at 10:42 PM PRATEEK GAUR <pr...@gmail.com>
> > wrote:
> > > >
> > > > > Thanks Divjot for bringing it up.
> > > > >
> > > > > I was under the same impression as Russell and Andrew that we'd
> have
> > it
> > > > > flag gated.
> > > > > With the new versioning proposal 'preview' might be a good tag for
> > it.
> > > > >
> > > > > Best
> > > > > Prateek
> > > > >
> > > > > On Sat, Aug 1, 2026 at 12:39 AM Andrew Lamb <
> andrewlamb11@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> Div: Here is a proposal to mark ALP in Preview: [1] (using a
> comment
> > > in
> > > > >> the
> > > > >> spec section).
> > > > >>
> > > > >> Russel: That is my understanding as well.
> > > > >>
> > > > >> Hopefully we are able to formalize the versioning process shortly
> so
> > > we
> > > > >> can
> > > > >> formalize the notion of "preview"
> > > > >>
> > > > >> Andrew
> > > > >>
> > > > >>
> > > > >> [1]: https://github.com/apache/parquet-format/pull/604
> > > > >>
> > > > >> On Fri, Jul 31, 2026 at 11:15 AM Russell Spitzer <
> > > > >> russell.spitzer@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >> > I thought we had agreed it would be merged, and features like
> this
> > > > would
> > > > >> > be feature flag gated. So a V-Old writer could always opt in to
> > > using
> > > > >> this
> > > > >> > encoding but that must be a conscious effort if you are not
> using
> > > > VNext
> > > > >> of
> > > > >> > Parquet. If you are on V-Next the writer can choose to use ALP
> > > > whenever
> > > > >> it
> > > > >> > likes without user intervention.
> > > > >> >
> > > > >> > On Fri, Jul 31, 2026 at 2:20 AM Divjot Arora via dev <
> > > > >> > dev@parquet.apache.org> wrote:
> > > > >> >
> > > > >> >> Yes, that’s my understanding as well. However, as you noted,
> the
> > > > >> >> versioning
> > > > >> >> discussion is still in early stages and hasn’t prescribed
> > concrete
> > > > >> >> mechanisms yet. If ALP goes into parquet-format 2.14.0, there’s
> > no
> > > > >> >> indication that it’s in preview or that writers shouldn’t use
> it
> > by
> > > > >> >> default
> > > > >> >> yet. Given that we’ve held off on merging other incompatible
> > > changes
> > > > >> (e.g
> > > > >> >> removal of path_in_schema) until the versioning discussion
> > closes,
> > > my
> > > > >> >> preference would be to leave ALP un-merged for the time being
> as
> > > > well.
> > > > >> >>
> > > > >> >> — Div
> > > > >> >>
> > > > >> >> On Fri, Jul 31, 2026 at 08:59 Andrew Lamb <
> > andrewlamb11@gmail.com>
> > > > >> wrote:
> > > > >> >>
> > > > >> >> > After re-reading Julien's Versioning Proposal, my
> understanding
> > > is
> > > > >> that
> > > > >> >> if
> > > > >> >> > we adopt that system, we will somehow mark ALP in "Preview"
> in
> > > > >> >> > parquet-format for release Version 3.
> > > > >> >> >
> > > > >> >> > The versioning document doesn't seem to proscribe the
> > particular
> > > > >> method
> > > > >> >> > for annotating "preview" features -- I think clearly marking
> > them
> > > > as
> > > > >> >> such
> > > > >> >> > in parquet.thrift as well as reflecting that on the
> > > > >> >> > https://parquet.apache.org/docs/file-format/versions/ page
> > > somehow
> > > > >> >> would
> > > > >> >> > be adequate.
> > > > >> >> >
> > > > >> >> > Andrew
> > > > >> >> >
> > > > >> >> > On Thu, Jul 30, 2026 at 11:50 AM Andrew Lamb <
> > > > andrewlamb11@gmail.com
> > > > >> >
> > > > >> >> > wrote:
> > > > >> >> >
> > > > >> >> >> In my mind ALP is in the same position as other encodings
> that
> > > > were
> > > > >> >> added
> > > > >> >> >> recently[1] (such as BYTE_STREAM_SPLIT for non floating
> point
> > > > types)
> > > > >> >> >>
> > > > >> >> >> That is to say it will be available in the latest open
> source
> > > > >> writers
> > > > >> >> and
> > > > >> >> >> readers, but realistically not widely used in the ecosystem
> > > where
> > > > >> >> >> compatibility is a concern (a large part)
> > > > >> >> >>
> > > > >> >> >> I view Julian's work[2][3] to gather consensus on how to
> name
> > > > >> >> >> combinations of features (e.g. "versions") as the mechanism
> by
> > > > >> which we
> > > > >> >> >> will accelerate the adoption of this change (along with
> other
> > > > >> forward
> > > > >> >> >> incompatible changes) across the ecosystem.
> > > > >> >> >>
> > > > >> >> >> > Will we exclude this change when cutting parquet-format
> > > v2.14.0?
> > > > >> >> >>
> > > > >> >> >> No I don't think so -- I would expect it to go out in the
> next
> > > > >> >> >> parquet-format version as do other changes in the spec.
> > > > >> >> >>
> > > > >> >> >> Andrew
> > > > >> >> >>
> > > > >> >> >> [1]: https://parquet.apache.org/docs/file-format/versions/
> > > > >> >> >> [2]:
> > > > >> https://lists.apache.org/thread/xmnj8h0h8ozmrgox5tydhhs7yz7q2hss
> > > > >> >> >> [3]:
> > > > >> https://lists.apache.org/thread/qlf8lg90gqq41lllyy9mk2f0skqvftqj
> > > > >> >> >>
> > > > >> >> >>
> > > > >> >> >> On Thu, Jul 30, 2026 at 3:51 AM Divjot Arora via dev <
> > > > >> >> >> dev@parquet.apache.org> wrote:
> > > > >> >> >>
> > > > >> >> >>> Hi Andrew,
> > > > >> >> >>>
> > > > >> >> >>> How does this interact with the ongoing versioning
> > discussion?
> > > > >> Given
> > > > >> >> that
> > > > >> >> >>> new encodings are not forward compatible, I would assume
> ALP
> > is
> > > > >> >> included
> > > > >> >> >>> in
> > > > >> >> >>> Parquet V<next> (V3?), but it's not clear to me how that
> > works
> > > if
> > > > >> it's
> > > > >> >> >>> merged to the parquet-format master branch now. Will we
> > exclude
> > > > >> this
> > > > >> >> >>> change
> > > > >> >> >>> when cutting parquet-format v2.14.0?
> > > > >> >> >>>
> > > > >> >> >>> -- Div
> > > > >> >> >>>
> > > > >> >> >>> On Thu, Jul 30, 2026 at 8:51 AM Andrew Lamb <
> > > > >> andrewlamb11@gmail.com>
> > > > >> >> >>> wrote:
> > > > >> >> >>>
> > > > >> >> >>> > Hello,
> > > > >> >> >>> >
> > > > >> >> >>> > As an update, thanks to the epic work from many
> > contributors,
> > > > >> >> >>> especially
> > > > >> >> >>> > Prateek, I just merged the parquet-format change for
> > ALP[1].
> > > We
> > > > >> >> plan to
> > > > >> >> >>> > make additional editorial The remaining items I know of
> are
> > > > >> merging
> > > > >> >> the
> > > > >> >> >>> > examples from parquet-testing[2], and then of course the
> > > > various
> > > > >> >> >>> > implementations
> > > > >> >> >>> >
> > > > >> >> >>> > Thanks again everyone. This is a great step forward
> > > > >> >> >>> > Andrew
> > > > >> >> >>> >
> > > > >> >> >>> > [1]: https://github.com/apache/parquet-format/pull/557
> > > > >> >> >>> > [2]: https://github.com/apache/parquet-testing/pull/100
> > > > >> >> >>> >
> > > > >> >> >>> > On Tue, Jul 21, 2026 at 4:39 PM Julien Le Dem <
> > > > julien@apache.org
> > > > >> >
> > > > >> >> >>> wrote:
> > > > >> >> >>> >
> > > > >> >> >>> > > It was also in my spam folder... :(
> > > > >> >> >>> > > I've seen a bunch of apache list emails go to my spam
> > > > recently.
> > > > >> >> >>> > >
> > > > >> >> >>> > > On Tue, Jul 21, 2026 at 10:58 AM Russell Spitzer <
> > > > >> >> >>> > > russell.spitzer@gmail.com>
> > > > >> >> >>> > > wrote:
> > > > >> >> >>> > >
> > > > >> >> >>> > > > Yes, my hope was that my vote would kick it out of
> spam
> > > but
> > > > >> we
> > > > >> >> >>> > definitely
> > > > >> >> >>> > > > had two other votes that also went to spam for me.
> > > > >> >> >>> > > >
> > > > >> >> >>> > > > For anyone looking the thread is here
> > > > >> >> >>> > > >
> > > > >> >>
> https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt
> > > > >> >> >>> > > >
> > > > >> >> >>> > > > On Tue, Jul 21, 2026 at 10:57 AM Micah Kornfield <
> > > > >> >> >>> > emkornfield@gmail.com>
> > > > >> >> >>> > > > wrote:
> > > > >> >> >>> > > >
> > > > >> >> >>> > > > > I've seen other's votes not go to spam, so
> hopefully
> > > > people
> > > > >> >> will
> > > > >> >> >>> see
> > > > >> >> >>> > it
> > > > >> >> >>> > > > in
> > > > >> >> >>> > > > > there inboxes (or see this thread and look in there
> > > > spam).
> > > > >> >> >>> > > > >
> > > > >> >> >>> > > > > On Tue, Jul 21, 2026 at 10:52 AM PRATEEK GAUR <
> > > > >> >> >>> prtkgaur@gmail.com>
> > > > >> >> >>> > > > wrote:
> > > > >> >> >>> > > > >
> > > > >> >> >>> > > > > > Okay,
> > > > >> >> >>> > > > > >
> > > > >> >> >>> > > > > > I sent out the voting email yesterday and so far
> > I've
> > > > >> heard
> > > > >> >> >>> from 3
> > > > >> >> >>> > > > people
> > > > >> >> >>> > > > > > that it ended up in their "spam folder". Dev VPN
> to
> > > be
> > > > >> >> blamed
> > > > >> >> >>> for
> > > > >> >> >>> > > it?.
> > > > >> >> >>> > > > > >
> > > > >> >> >>> > > > > > What's the protocol here?
> > > > >> >> >>> > > > > > I let people hunt for the email in their spam
> > folder
> > > or
> > > > >> send
> > > > >> >> >>> out
> > > > >> >> >>> > > > another
> > > > >> >> >>> > > > > > one?
> > > > >> >> >>> > > > > >
> > > > >> >> >>> > > > > > Best
> > > > >> >> >>> > > > > > Prateek
> > > > >> >> >>> > > > > >
> > > > >> >> >>> > > > > > On Sun, Jul 19, 2026 at 10:47 PM PRATEEK GAUR <
> > > > >> >> >>> prtkgaur@gmail.com>
> > > > >> >> >>> > > > > wrote:
> > > > >> >> >>> > > > > >
> > > > >> >> >>> > > > > > > Hi team,
> > > > >> >> >>> > > > > > >
> > > > >> >> >>> > > > > > > Vinoo has added the tests which were requested
> in
> > > the
> > > > >> last
> > > > >> >> >>> round
> > > > >> >> >>> > of
> > > > >> >> >>> > > > > > review.
> > > > >> >> >>> > > > > > > In his own words.
> > > > >> >> >>> > > > > > >
> > > > >> >> >>> > > > > > > "
> > > > >> >> >>> > > > > > > The extreme value tests Micah wanted are done
> and
> > > > >> pushed.
> > > > >> >> I
> > > > >> >> >>> added
> > > > >> >> >>> > > > > > coverage
> > > > >> >> >>> > > > > > > for values that need the full FOR bit width
> after
> > > the
> > > > >> >> frame
> > > > >> >> >>> of
> > > > >> >> >>> > > > > reference
> > > > >> >> >>> > > > > > is
> > > > >> >> >>> > > > > > > applied: 64-bit (the case where the signed max
> > > minus
> > > > >> min
> > > > >> >> >>> > > subtraction
> > > > >> >> >>> > > > > > > overflows), 63-bit (the non-overflow case), and
> > > > 32-bit
> > > > >> for
> > > > >> >> >>> > floats.
> > > > >> >> >>> > > > All
> > > > >> >> >>> > > > > > > lossless with zero exceptions. I also verified
> > them
> > > > >> >> through
> > > > >> >> >>> Arrow
> > > > >> >> >>> > > C++
> > > > >> >> >>> > > > > ALP
> > > > >> >> >>> > > > > > > reader (the local build), so the Java-written
> > > extreme
> > > > >> >> columns
> > > > >> >> >>> > > decode
> > > > >> >> >>> > > > > > > correctly in C++, bit-exact against the
> expected
> > > > >> values."
> > > > >> >> >>> > > > > > >
> > > > >> >> >>> > > > > > > State of the PR (#3397):
> > > > >> >> >>> > > > > > >
> > > > >> >> >>> > > > > > > - Merges cleanly with master (I pulled
> master
> > in
> > > > and
> > > > >> >> >>> resolved
> > > > >> >> >>> > > the
> > > > >> >> >>> > > > > > > conflicts).
> > > > >> >> >>> > > > > > > - 41 of 44 review threads resolved. The 3
> > > > remaining
> > > > >> are
> > > > >> >> >>> > > > intentional:
> > > > >> >> >>> > > > > > > one deferred perf nit, one that stays open
> > until
> > > > >> >> >>> > parquet-format
> > > > >> >> >>> > > > > ships
> > > > >> >> >>> > > > > > ALP,
> > > > >> >> >>> > > > > > >
> > > > >> >> >>> > > > > > > "
> > > > >> >> >>> > > > > > >
> > > > >> >> >>> > > > > > > With this both the
> > > > >> >> >>> > > > > > > 1) C++ PR :
> > > > >> >> >>> https://github.com/apache/arrow/pull/48345/changes
> > > > >> >> >>> > > > > > > 2) Java PR :
> > > > >> >> >>> https://github.com/apache/parquet-java/pull/3397
> > > > >> >> >>> > > > > > >
> > > > >> >> >>> > > > > > > Are cross language tested and range of test
> > > > scenarios.
> > > > >> >> >>> > > > > > >
> > > > >> >> >>> > > > > > > Best
> > > > >> >> >>> > > > > > > Prateek
> > > > >> >> >>> > > > > > >
> > > > >> >> >>> > > > > > >
> > > > >> >> >>> > > > > > > On Mon, Jul 13, 2026 at 8:38 AM PRATEEK GAUR <
> > > > >> >> >>> prtkgaur@gmail.com
> > > > >> >> >>> > >
> > > > >> >> >>> > > > > wrote:
> > > > >> >> >>> > > > > > >
> > > > >> >> >>> > > > > > >> Thanks Matt,
> > > > >> >> >>> > > > > > >>
> > > > >> >> >>> > > > > > >> I'll be on it this week.
> > > > >> >> >>> > > > > > >>
> > > > >> >> >>> > > > > > >> Best
> > > > >> >> >>> > > > > > >> Prateek
> > > > >> >> >>> > > > > > >>
> > > > >> >> >>> > > > > > >> On Mon, Jul 13, 2026 at 8:36 AM Matt Topol <
> > > > >> >> >>> > > zotthewizard@gmail.com>
> > > > >> >> >>> > > > > > >> wrote:
> > > > >> >> >>> > > > > > >>
> > > > >> >> >>> > > > > > >>> There is also the Go implementation submitted
> > by
> > > > >> Arnav (
> > > > >> >> >>> > > > > > >>> https://github.com/apache/arrow-go/pull/704)
> > > which
> > > > >> is
> > > > >> >> >>> waiting
> > > > >> >> >>> > > for
> > > > >> >> >>> > > > > > >>> updates.
> > > > >> >> >>> > > > > > >>>
> > > > >> >> >>> > > > > > >>> Just wanted to make sure it didn't get lost
> > here.
> > > > >> >> >>> > > > > > >>>
> > > > >> >> >>> > > > > > >>> --Matt
> > > > >> >> >>> > > > > > >>>
> > > > >> >> >>> > > > > > >>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK
> GAUR <
> > > > >> >> >>> > > prtkgaur@gmail.com>
> > > > >> >> >>> > > > > > >>> wrote:
> > > > >> >> >>> > > > > > >>>
> > > > >> >> >>> > > > > > >>> > Hi all,
> > > > >> >> >>> > > > > > >>> >
> > > > >> >> >>> > > > > > >>> > I'd like to share a status update on the
> ALP
> > > > >> encoding
> > > > >> >> >>> effort
> > > > >> >> >>> > > and
> > > > >> >> >>> > > > > get
> > > > >> >> >>> > > > > > a
> > > > >> >> >>> > > > > > >>> feel
> > > > >> >> >>> > > > > > >>> > from the community before starting a formal
> > > vote.
> > > > >> >> >>> > > > > > >>> >
> > > > >> >> >>> > > > > > >>> > We now have the specification plus
> > > > implementations
> > > > >> in
> > > > >> >> >>> both
> > > > >> >> >>> > > > > languages:
> > > > >> >> >>> > > > > > >>> >
> > > > >> >> >>> > > > > > >>> > - Spec:
> > apache/parquet-format#557 —
> > > > >> GH-533
> > > > >> >> >>> Add
> > > > >> >> >>> > ALP
> > > > >> >> >>> > > > > > encoding
> > > > >> >> >>> > > > > > >>> > specification
> > > > >> >> >>> > > > > > >>> > - C++ (Arrow): apache/arrow#48345 —
> > > GH-48701
> > > > >> >> >>> > > [C++][Parquet]
> > > > >> >> >>> > > > > Add
> > > > >> >> >>> > > > > > >>> ALPpd
> > > > >> >> >>> > > > > > >>> > encoding
> > > > >> >> >>> > > > > > >>> > - Java:
> apache/parquet-java#3397
> > —
> > > > >> Parquet
> > > > >> >> >>> Java
> > > > >> >> >>> > ALP
> > > > >> >> >>> > > > > > >>> > Implementation (by Vinoo)
> > > > >> >> >>> > > > > > >>> >
> > > > >> >> >>> > > > > > >>> > Current state:
> > > > >> >> >>> > > > > > >>> >
> > > > >> >> >>> > > > > > >>> > - The C++ implementation has been
> through
> > > 3-4
> > > > >> >> rounds
> > > > >> >> >>> of
> > > > >> >> >>> > > > review.
> > > > >> >> >>> > > > > > >>> > - The Java implementation has been
> > tracking
> > > > >> well
> > > > >> >> and
> > > > >> >> >>> has
> > > > >> >> >>> > > had
> > > > >> >> >>> > > > > > >>> initial
> > > > >> >> >>> > > > > > >>> > reviews from contributors, with no
> > major
> > > > >> >> remaining
> > > > >> >> >>> > > issues.
> > > > >> >> >>> > > > > > >>> > - Cross-language compatibility tests
> are
> > > > >> passing:
> > > > >> >> the
> > > > >> >> >>> > Arrow
> > > > >> >> >>> > > > C++
> > > > >> >> >>> > > > > > >>> decoder
> > > > >> >> >>> > > > > > >>> > reads Java-written data bit-exactly
> > > across
> > > > >> >> ~1.56M
> > > > >> >> >>> > values
> > > > >> >> >>> > > > and
> > > > >> >> >>> > > > > 18
> > > > >> >> >>> > > > > > >>> > fixtures, covering V1 and V2 pages,
> > > > multiple
> > > > >> >> vector
> > > > >> >> >>> > > sizes,
> > > > >> >> >>> > > > > and
> > > > >> >> >>> > > > > > >>> > several
> > > > >> >> >>> > > > > > >>> > real datasets — zero mismatches.
> > > > >> >> >>> > > > > > >>> >
> > > > >> >> >>> > > > > > >>> > As raised in the java-pr we will be adding
> > > > >> coverage
> > > > >> >> for
> > > > >> >> >>> > > extreme
> > > > >> >> >>> > > > > > values
> > > > >> >> >>> > > > > > >>> > (those requiring
> > > > >> >> >>> > > > > > >>> > 63-64 bits after FOR is applied) before we
> > > > close a
> > > > >> >> vote.
> > > > >> >> >>> > We'll
> > > > >> >> >>> > > > aim
> > > > >> >> >>> > > > > > to
> > > > >> >> >>> > > > > > >>> get
> > > > >> >> >>> > > > > > >>> > that
> > > > >> >> >>> > > > > > >>> > done in parallel.
> > > > >> >> >>> > > > > > >>> >
> > > > >> >> >>> > > > > > >>> > Plan: unless there are objections, I intend
> > to
> > > > >> start a
> > > > >> >> >>> vote
> > > > >> >> >>> > at
> > > > >> >> >>> > > > the
> > > > >> >> >>> > > > > > end
> > > > >> >> >>> > > > > > >>> of
> > > > >> >> >>> > > > > > >>> > this week/early
> > > > >> >> >>> > > > > > >>> > next week.
> > > > >> >> >>> > > > > > >>> >
> > > > >> >> >>> > > > > > >>> > Thanks
> > > > >> >> >>> > > > > > >>> > Prateek
> > > > >> >> >>> > > > > > >>> >
> > > > >> >> >>> > > > > > >>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK
> GAUR <
> > > > >> >> >>> > > prtkgaur@gmail.com>
> > > > >> >> >>> > > > > > >>> wrote:
> > > > >> >> >>> > > > > > >>> >
> > > > >> >> >>> > > > > > >>> > > Hi Team,
> > > > >> >> >>> > > > > > >>> > >
> > > > >> >> >>> > > > > > >>> > > Just wanted to provide some updates on
> ALP.
> > > > >> >> >>> > > > > > >>> > > Micah and I have done 3-4 rounds of
> review
> > > for
> > > > >> the
> > > > >> >> c++
> > > > >> >> >>> PR :
> > > > >> >> >>> > > > > > >>> > >
> https://github.com/apache/arrow/pull/48345
> > > > >> >> >>> > > > > > >>> > >
> > > > >> >> >>> > > > > > >>> > > For the Java implementation Vinoo has
> been
> > > > >> working
> > > > >> >> on
> > > > >> >> >>> > > following
> > > > >> >> >>> > > > > PR
> > > > >> >> >>> > > > > > :
> > > > >> >> >>> > > > > > >>> > >
> > > > https://github.com/apache/parquet-java/pull/3397
> > > > >> >> >>> > > > > > >>> > >
> > > > >> >> >>> > > > > > >>> > > Best
> > > > >> >> >>> > > > > > >>> > > Prateek
> > > > >> >> >>> > > > > > >>> > >
> > > > >> >> >>> > > > > > >>> > > On Tue, May 5, 2026 at 1:50 PM Micah
> > > Kornfield
> > > > <
> > > > >> >> >>> > > > > > >>> emkornfield@gmail.com>
> > > > >> >> >>> > > > > > >>> > > wrote:
> > > > >> >> >>> > > > > > >>> > >
> > > > >> >> >>> > > > > > >>> > >> Hi Antoine,
> > > > >> >> >>> > > > > > >>> > >>
> > > > >> >> >>> > > > > > >>> > >> > Apologies if the question was already
> > > asked,
> > > > >> but
> > > > >> >> >>> should
> > > > >> >> >>> > we
> > > > >> >> >>> > > > > care
> > > > >> >> >>> > > > > > >>> about
> > > > >> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be
> more
> > > > >> >> efficient
> > > > >> >> >>> than
> > > > >> >> >>> > > > > FLOAT16
> > > > >> >> >>> > > > > > +
> > > > >> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> > > > >> >> >>> > > > > > >>> > >>
> > > > >> >> >>> > > > > > >>> > >> It was. We thought we could defer it
> for
> > > the
> > > > >> >> >>> following
> > > > >> >> >>> > > > reasons:
> > > > >> >> >>> > > > > > >>> > >> 1. It's not clear there are a lot of
> easy
> > > > >> >> reference
> > > > >> >> >>> > > datasets
> > > > >> >> >>> > > > to
> > > > >> >> >>> > > > > > >>> test
> > > > >> >> >>> > > > > > >>> > its
> > > > >> >> >>> > > > > > >>> > >> effectiveness. It does look like there
> > > might
> > > > be
> > > > >> >> one
> > > > >> >> >>> or
> > > > >> >> >>> > two
> > > > >> >> >>> > > on
> > > > >> >> >>> > > > > > >>> > huggingface
> > > > >> >> >>> > > > > > >>> > >> (e.g.
> > > > >> >> https://huggingface.co/datasets/kikitora/curdie
> > > > >> >> >>> ).
> > > > >> >> >>> > > > > > >>> > >> 2. It seemed likely that float 16 was
> > more
> > > > >> likely
> > > > >> >> >>> used
> > > > >> >> >>> > for
> > > > >> >> >>> > > > > values
> > > > >> >> >>> > > > > > >>> that
> > > > >> >> >>> > > > > > >>> > >> were less likely to reduce to decimal
> > > values.
> > > > >> >> >>> > > > > > >>> > >> 3. It could be added as an extension
> > later
> > > if
> > > > >> >> needed.
> > > > >> >> >>> > > > > > >>> > >>
> > > > >> >> >>> > > > > > >>> > >> Cheers,
> > > > >> >> >>> > > > > > >>> > >> Micah
> > > > >> >> >>> > > > > > >>> > >>
> > > > >> >> >>> > > > > > >>> > >>
> > > > >> >> >>> > > > > > >>> > >> On Tue, May 5, 2026 at 1:40 PM Antoine
> > > Pitrou
> > > > <
> > > > >> >> >>> > > > > antoine@python.org
> > > > >> >> >>> > > > > > >
> > > > >> >> >>> > > > > > >>> > wrote:
> > > > >> >> >>> > > > > > >>> > >>
> > > > >> >> >>> > > > > > >>> > >> >
> > > > >> >> >>> > > > > > >>> > >> > Hello,
> > > > >> >> >>> > > > > > >>> > >> >
> > > > >> >> >>> > > > > > >>> > >> > Apologies if the question was already
> > > asked,
> > > > >> but
> > > > >> >> >>> should
> > > > >> >> >>> > we
> > > > >> >> >>> > > > > care
> > > > >> >> >>> > > > > > >>> about
> > > > >> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be
> more
> > > > >> >> efficient
> > > > >> >> >>> than
> > > > >> >> >>> > > > > FLOAT16
> > > > >> >> >>> > > > > > +
> > > > >> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> > > > >> >> >>> > > > > > >>> > >> >
> > > > >> >> >>> > > > > > >>> > >> > Regards
> > > > >> >> >>> > > > > > >>> > >> >
> > > > >> >> >>> > > > > > >>> > >> > Antoine.
> > > > >> >> >>> > > > > > >>> > >> >
> > > > >> >> >>> > > > > > >>> > >> >
> > > > >> >> >>> > > > > > >>> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a
> > > écrit
> > > > :
> > > > >> >> >>> > > > > > >>> > >> > > Thanks Andrew and Micah for review
> > > > feedback
> > > > >> on
> > > > >> >> >>> the two
> > > > >> >> >>> > > > PR's
> > > > >> >> >>> > > > > > >>> > >> > > 1) (c++ arrow repo)
> > > > >> >> >>> > > > > > >>> > >>
> > > > >> https://github.com/apache/arrow/pull/48345/changes
> > > > >> >> >>> > > > > > >>> > >> > > 2) (parquet-format repo)
> > > > >> >> >>> > > > > > >>> > >> >
> > > > >> >> https://github.com/apache/parquet-format/pull/557
> > > > >> >> >>> > > > > > >>> > >> > >
> > > > >> >> >>> > > > > > >>> > >> > > I have addressed all (unless I
> missed
> > > > >> >> something)
> > > > >> >> >>> > > comments
> > > > >> >> >>> > > > on
> > > > >> >> >>> > > > > > >>> the two
> > > > >> >> >>> > > > > > >>> > >> > PR's.
> > > > >> >> >>> > > > > > >>> > >> > >
> > > > >> >> >>> > > > > > >>> > >> > > Best
> > > > >> >> >>> > > > > > >>> > >> > > Prateek
> > > > >> >> >>> > > > > > >>> > >> > >
> > > > >> >> >>> > > > > > >>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM
> > PRATEEK
> > > > >> GAUR <
> > > > >> >> >>> > > > > > >>> prtkgaur@gmail.com>
> > > > >> >> >>> > > > > > >>> > >> wrote:
> > > > >> >> >>> > > > > > >>> > >> > >
> > > > >> >> >>> > > > > > >>> > >> > >> Thanks Andrew and Micah.
> > > > >> >> >>> > > > > > >>> > >> > >>
> > > > >> >> >>> > > > > > >>> > >> > >> `fair amount of feedback on at
> least
> > > the
> > > > >> >> >>> > > implementations`
> > > > >> >> >>> > > > > > >>> > >> > >> For the c++ I have already started
> > > > >> addressing
> > > > >> >> the
> > > > >> >> >>> > > > > feedback, I
> > > > >> >> >>> > > > > > >>> > should
> > > > >> >> >>> > > > > > >>> > >> be
> > > > >> >> >>> > > > > > >>> > >> > >> done with that Monday/Tuesday.
> > > > >> >> >>> > > > > > >>> > >> > >> I think Vinoo too has been making
> > good
> > > > >> >> progress
> > > > >> >> >>> on
> > > > >> >> >>> > the
> > > > >> >> >>> > > > Java
> > > > >> >> >>> > > > > > >>> > >> > implementation.
> > > > >> >> >>> > > > > > >>> > >> > >>
> > > > >> >> >>> > > > > > >>> > >> > >> Best
> > > > >> >> >>> > > > > > >>> > >> > >> Prateek
> > > > >> >> >>> > > > > > >>> > >> > >>
> > > > >> >> >>> > > > > > >>> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM
> > Andrew
> > > > >> Lamb <
> > > > >> >> >>> > > > > > >>> > andrewlamb11@gmail.com
> > > > >> >> >>> > > > > > >>> > >> >
> > > > >> >> >>> > > > > > >>> > >> > >> wrote:
> > > > >> >> >>> > > > > > >>> > >> > >>
> > > > >> >> >>> > > > > > >>> > >> > >>> Got it. Thank you for the
> > > clarification
> > > > >> -- I
> > > > >> >> >>> will
> > > > >> >> >>> > try
> > > > >> >> >>> > > > and
> > > > >> >> >>> > > > > > look
> > > > >> >> >>> > > > > > >>> > into
> > > > >> >> >>> > > > > > >>> > >> the
> > > > >> >> >>> > > > > > >>> > >> > >>> spec and the Rust
> implementation[1]
> > in
> > > > >> this
> > > > >> >> next
> > > > >> >> >>> > week
> > > > >> >> >>> > > > > > >>> > >> > >>>
> > > > >> >> >>> > > > > > >>> > >> > >>> [1]:
> > > > >> >> >>> https://github.com/apache/arrow-rs/pull/9372
> > > > >> >> >>> > > > > > >>> > >> > >>>
> > > > >> >> >>> > > > > > >>> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM
> > Micah
> > > > >> >> >>> Kornfield <
> > > > >> >> >>> > > > > > >>> > >> > emkornfield@gmail.com>
> > > > >> >> >>> > > > > > >>> > >> > >>> wrote:
> > > > >> >> >>> > > > > > >>> > >> > >>>
> > > > >> >> >>> > > > > > >>> > >> > >>>> Hi Andrew,
> > > > >> >> >>> > > > > > >>> > >> > >>>> I think there is a fair amount of
> > > > >> feedback
> > > > >> >> on
> > > > >> >> >>> at
> > > > >> >> >>> > > least
> > > > >> >> >>> > > > > the
> > > > >> >> >>> > > > > > >>> > >> > >>>> implementations, typically I
> think
> > > > we've
> > > > >> >> waited
> > > > >> >> >>> > till
> > > > >> >> >>> > > > they
> > > > >> >> >>> > > > > > are
> > > > >> >> >>> > > > > > >>> > >> close to
> > > > >> >> >>> > > > > > >>> > >> > >>>> mergeable before a final vote.
> > > > >> Otherwise I
> > > > >> >> >>> agree
> > > > >> >> >>> > we
> > > > >> >> >>> > > > are
> > > > >> >> >>> > > > > > very
> > > > >> >> >>> > > > > > >>> > >> close.
> > > > >> >> >>> > > > > > >>> > >> > >>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>> -Micah
> > > > >> >> >>> > > > > > >>> > >> > >>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>> On Saturday, April 25, 2026,
> Andrew
> > > > Lamb
> > > > >> <
> > > > >> >> >>> > > > > > >>> andrewlamb11@gmail.com
> > > > >> >> >>> > > > > > >>> > >
> > > > >> >> >>> > > > > > >>> > >> > wrote:
> > > > >> >> >>> > > > > > >>> > >> > >>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>> Thanks Prateek,
> > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>> I think from this content it
> looks
> > > to
> > > > me
> > > > >> >> like
> > > > >> >> >>> we
> > > > >> >> >>> > are
> > > > >> >> >>> > > > > ready
> > > > >> >> >>> > > > > > >>> to
> > > > >> >> >>> > > > > > >>> > >> start a
> > > > >> >> >>> > > > > > >>> > >> > >>>>> vote to explicitly accept ALP
> into
> > > > >> Parquet
> > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>> Does anyone know of a reason we
> > > should
> > > > >> >> >>> postpone it
> > > > >> >> >>> > > for
> > > > >> >> >>> > > > > > >>> longer?
> > > > >> >> >>> > > > > > >>> > >> > >>>>> Perhaps someone needs some more
> > time
> > > > to
> > > > >> >> >>> review?
> > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>> Andrew
> > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM
> > > > PRATEEK
> > > > >> >> GAUR <
> > > > >> >> >>> > > > > > >>> > prtkgaur@gmail.com>
> > > > >> >> >>> > > > > > >>> > >> > >>>>> wrote:
> > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> Hi team,
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> Hope everyone is doing well. I
> > got
> > > a
> > > > >> >> chance
> > > > >> >> >>> to
> > > > >> >> >>> > work
> > > > >> >> >>> > > > > > >>> through all
> > > > >> >> >>> > > > > > >>> > >> the
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> remaining feedback and update
> the
> > > > spec
> > > > >> >> doc.
> > > > >> >> >>> Here
> > > > >> >> >>> > > are
> > > > >> >> >>> > > > > the
> > > > >> >> >>> > > > > > >>> new
> > > > >> >> >>> > > > > > >>> > >> > artifacts
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> 1) Spec document :
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> >
> > > > >> >> >>> > > > > > >>> > >>
> > > > >> >> >>> > > > > > >>> >
> > > > >> >> >>> > > > > > >>>
> > > > >> >> >>> > > > > >
> > > > >> >> >>> > > > >
> > > > >> >> >>> > > >
> > > > >> >> >>> > >
> > > > >> >> >>> >
> > > > >> >> >>>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> 2) Spec document in parquet
> > format
> > > > >> repo :
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > https://github.com/apache/parquet-format/pull/557
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> 3) Alp implementation in arrow
> > c++
> > > > >> repo :
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > https://github.com/apache/arrow/pull/48345/changes
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> 4) Alp implementation in
> > > parquet-java
> > > > >> >> repo :
> > > > >> >> >>> Work
> > > > >> >> >>> > > for
> > > > >> >> >>> > > > > > >>> Vinoo and
> > > > >> >> >>> > > > > > >>> > >> > Julien
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > https://github.com/apache/parquet-java/pull/3397
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> 5) PR with test and
> benchmarking
> > > > >> >> artifacts in
> > > > >> >> >>> > > > > > >>> parquet-testing
> > > > >> >> >>> > > > > > >>> > >> repo :
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > https://github.com/apache/parquet-testing/pull/100
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> And
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> - Go : Arnav just submitted
> > an
> > > in
> > > > >> >> >>> progress
> > > > >> >> >>> > > > > > >>> implementation
> > > > >> >> >>> > > > > > >>> > in
> > > > >> >> >>> > > > > > >>> > >> Go.
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> https://github.com/apache/arrow-go/pull/704
> > > > >> >> >>> > (I
> > > > >> >> >>> > > > > > haven't
> > > > >> >> >>> > > > > > >>> > >> started
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> looking at it yet)
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> - Rust : I remember Andrew
> > > > >> mentioned
> > > > >> >> that
> > > > >> >> >>> > this
> > > > >> >> >>> > > > work
> > > > >> >> >>> > > > > > is
> > > > >> >> >>> > > > > > >>> also
> > > > >> >> >>> > > > > > >>> > >> in
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> progress (So 4 languages!)
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> *Arrow C++ implementation *
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> The PR is out and was also used
> > by
> > > > >> >> Antoine to
> > > > >> >> >>> > > report
> > > > >> >> >>> > > > > the
> > > > >> >> >>> > > > > > >>> > numbers
> > > > >> >> >>> > > > > > >>> > >> as
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> reported here. Micah and
> > Konstantin
> > > > >> have
> > > > >> >> >>> given 1
> > > > >> >> >>> > > > round
> > > > >> >> >>> > > > > of
> > > > >> >> >>> > > > > > >>> > >> feedback
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> and I'm addressing them today.
> > > Please
> > > > >> note
> > > > >> >> >>> that
> > > > >> >> >>> > the
> > > > >> >> >>> > > > > > default
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> optimization flag for compiling
> > is
> > > O2
> > > > >> and
> > > > >> >> not
> > > > >> >> >>> > Q3. I
> > > > >> >> >>> > > > got
> > > > >> >> >>> > > > > > >>> around
> > > > >> >> >>> > > > > > >>> > >> 70%
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> performance improvement in the
> > > > decoding
> > > > >> >> speed
> > > > >> >> >>> > when
> > > > >> >> >>> > > > > using
> > > > >> >> >>> > > > > > >>> the O3
> > > > >> >> >>> > > > > > >>> > >> > flag.
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> *Parqet-MR Java implementation
> > > > (working
> > > > >> >> with
> > > > >> >> >>> > Vinoo
> > > > >> >> >>> > > > and
> > > > >> >> >>> > > > > > >>> Julien)
> > > > >> >> >>> > > > > > >>> > >> and
> > > > >> >> >>> > > > > > >>> > >> > **Cross
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> Language testing*
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> Let me know if you have any
> > > > >> questions
> > > > >> >> or
> > > > >> >> >>> > > > feedback.
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> Now pasting some performance
> > > numbers
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> Table 1: C++ ALP Double
> > Decode —
> > > > >> >> Spotify
> > > > >> >> >>> > Columns
> > > > >> >> >>> > > > > > >>> (Graviton
> > > > >> >> >>> > > > > > >>> > 3,
> > > > >> >> >>> > > > > > >>> > >> ARM
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> Neoverse V1)
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>>
> > > > >> >> >>>
> ┌──────────────────┬──────────────┬──────────────┬─────────┐
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ Column │ -O2
> > (MB/s)
> > > > │
> > > > >> >> -O3
> > > > >> >> >>> > > (MB/s) │
> > > > >> >> >>> > > > > > >>> Speedup │
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>>
> > > > >> >> >>>
> ├──────────────────┼──────────────┼──────────────┼─────────┤
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ valence │
> 3,155
> > > > │
> > > > >> >> >>> 5,523
> > > > >> >> >>> > > │
> > > > >> >> >>> > > > > > >>> 1.75x │
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ danceability │
> 3,233
> > > > │
> > > > >> >> >>> 5,685
> > > > >> >> >>> > > │
> > > > >> >> >>> > > > > > >>> 1.76x │
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ energy │
> 3,197
> > > > │
> > > > >> >> >>> 5,652
> > > > >> >> >>> > > │
> > > > >> >> >>> > > > > > >>> 1.77x │
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ loudness │
> 3,186
> > > > │
> > > > >> >> >>> 5,473
> > > > >> >> >>> > > │
> > > > >> >> >>> > > > > > >>> 1.72x │
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>>
> > > > >> >> >>>
> └──────────────────┴──────────────┴──────────────┴─────────┘
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM
> > > > PRATEEK
> > > > >> >> GAUR
> > > > >> >> >>> <
> > > > >> >> >>> > > > > > >>> > prtkgaur@gmail.com
> > > > >> >> >>> > > > > > >>> > >> >
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> wrote:
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>> @Micah Kornfield <
> > > > >> emkornfield@gmail.com>
> > > > >> >> :
> > > > >> >> >>> Got
> > > > >> >> >>> > > it.
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>> @Andrew Lamb <
> > > > andrewlamb11@gmail.com>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> Do you think it would be good
> > to
> > > > >> start
> > > > >> >> >>> moving
> > > > >> >> >>> > the
> > > > >> >> >>> > > > > spec
> > > > >> >> >>> > > > > > >>> > >> development
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> into
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> markdown format, in
> preparation
> > > for
> > > > >> >> >>> finalizing
> > > > >> >> >>> > > it?
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>> Yes I'll update the numbers
> for
> > > some
> > > > >> of
> > > > >> >> the
> > > > >> >> >>> > > > examples I
> > > > >> >> >>> > > > > > >>> have in
> > > > >> >> >>> > > > > > >>> > >> the
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>> spec based
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>> on the updated header size.
> Then
> > > we
> > > > >> >> should
> > > > >> >> >>> be
> > > > >> >> >>> > good
> > > > >> >> >>> > > > to
> > > > >> >> >>> > > > > go
> > > > >> >> >>> > > > > > >>> for
> > > > >> >> >>> > > > > > >>> > the
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>> markdown format.
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>> Thanks everyone!
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> Andrew
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at
> 7:28 PM
> > > > >> PRATEEK
> > > > >> >> >>> GAUR <
> > > > >> >> >>> > > > > > >>> > >> prtkgaur@gmail.com>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> wrote:
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Hi team,
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 1) Andrew
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for working on
> > test
> > > > >> files.
> > > > >> >> >>> My PR
> > > > >> >> >>> > > did
> > > > >> >> >>> > > > > add
> > > > >> >> >>> > > > > > >>> all
> > > > >> >> >>> > > > > > >>> > the
> > > > >> >> >>> > > > > > >>> > >> > test
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> files I
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> used to benchmark on
> > > datasets.
> > > > >> >> Maybe
> > > > >> >> >>> we
> > > > >> >> >>> > can
> > > > >> >> >>> > > > club
> > > > >> >> >>> > > > > > it
> > > > >> >> >>> > > > > > >>> > >> together.
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> WIll also
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> aid
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> cross language testing
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Kosta Tarasov working
> > on
> > > > Rust
> > > > >> >> >>> > > > implementation.
> > > > >> >> >>> > > > > > >>> This is
> > > > >> >> >>> > > > > > >>> > >> > great.
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> Thanks
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 2) Antoine
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks a lot for
> > reporting
> > > > the
> > > > >> >> >>> numbers
> > > > >> >> >>> > on
> > > > >> >> >>> > > > AMD.
> > > > >> >> >>> > > > > > >>> Looks
> > > > >> >> >>> > > > > > >>> > >> like
> > > > >> >> >>> > > > > > >>> > >> > you
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> are
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> getting 8X the decoding
> > > > >> >> performance of
> > > > >> >> >>> > BSS.
> > > > >> >> >>> > > > This
> > > > >> >> >>> > > > > > is
> > > > >> >> >>> > > > > > >>> > >> > amazing!!.
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for
> acknowledging
> > > the
> > > > >> >> >>> sampling
> > > > >> >> >>> > > > design.
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - I agree with you on
> > > > >> Fastlanes. In
> > > > >> >> >>> some
> > > > >> >> >>> > > crude
> > > > >> >> >>> > > > > > >>> > >> experiments I
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> didn't get
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> a good perf benefit from
> > it
> > > on
> > > > >> >> >>> Graviton3
> > > > >> >> >>> > > (but
> > > > >> >> >>> > > > > > maybe
> > > > >> >> >>> > > > > > >>> > there
> > > > >> >> >>> > > > > > >>> > >> was
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> something
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> wrong with my
> > > implementation).
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Locking the 16bit
> > > exception
> > > > >> >> >>> encoding for
> > > > >> >> >>> > > the
> > > > >> >> >>> > > > > > spec
> > > > >> >> >>> > > > > > >>> in
> > > > >> >> >>> > > > > > >>> > >> this
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> case.
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Awesome I think we
> have
> > > > solved
> > > > >> >> for
> > > > >> >> >>> all
> > > > >> >> >>> > > open
> > > > >> >> >>> > > > > > >>> questions
> > > > >> >> >>> > > > > > >>> > >> minus
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> the
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> version byte :). (will
> get
> > > > back
> > > > >> on
> > > > >> >> >>> this
> > > > >> >> >>> > > soon)
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 3) Micah
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - FastLanes : The
> current
> > > spec
> > > > >> does
> > > > >> >> >>> allow
> > > > >> >> >>> > > for
> > > > >> >> >>> > > > > > using
> > > > >> >> >>> > > > > > >>> > >> FastLane
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> with the
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> configurable enum value
> > for
> > > > >> >> layout. We
> > > > >> >> >>> > > should
> > > > >> >> >>> > > > be
> > > > >> >> >>> > > > > > >>> able to
> > > > >> >> >>> > > > > > >>> > >> > inject
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> any
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> layout
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> in the current design.
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Working on resolving all
> > > remaining
> > > > >> open
> > > > >> >> >>> > comments
> > > > >> >> >>> > > > on
> > > > >> >> >>> > > > > > the
> > > > >> >> >>> > > > > > >>> spec
> > > > >> >> >>> > > > > > >>> > >> this
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> week.
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Best
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Prateek
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at
> > 3:37 AM
> > > > >> Steve
> > > > >> >> >>> > Loughran <
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> stevel@cloudera.com>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> wrote:
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at
> 18:12,
> > > > Micah
> > > > >> >> >>> Kornfield
> > > > >> >> >>> > <
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> emkornfield@gmail.com>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> wrote:
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> It looks like the actual
> > issue
> > > > >> >> >>> described for
> > > > >> >> >>> > > ORC
> > > > >> >> >>> > > > > in
> > > > >> >> >>> > > > > > >>> the
> > > > >> >> >>> > > > > > >>> > >> paper
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> is that
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> it
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> has multiple sub-encodings
> > in
> > > a
> > > > >> >> batch.
> > > > >> >> >>> This
> > > > >> >> >>> > > is
> > > > >> >> >>> > > > > > >>> different
> > > > >> >> >>> > > > > > >>> > >> then
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> the
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> design
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> proposed here where there
> is
> > > > still
> > > > >> >> fixed
> > > > >> >> >>> > > > encoding
> > > > >> >> >>> > > > > > per
> > > > >> >> >>> > > > > > >>> page
> > > > >> >> >>> > > > > > >>> > >> in
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> parquet.
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> Given reasonably sized
> > pages I
> > > > >> don't
> > > > >> >> >>> think
> > > > >> >> >>> > > > branch
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> misprediction should
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> be a
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> big issue for new
> encodings.
> > > I
> > > > >> agree
> > > > >> >> >>> that
> > > > >> >> >>> > we
> > > > >> >> >>> > > > > should
> > > > >> >> >>> > > > > > >>> be
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> conservative in
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> general for adding new
> > > > encodings.
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> +1
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >
> > > > >> >> >>> > > > > > >>> > >> >
> > > > >> >> >>> > > > > > >>> > >> >
> > > > >> >> >>> > > > > > >>> > >> >
> > > > >> >> >>> > > > > > >>> > >>
> > > > >> >> >>> > > > > > >>> > >
> > > > >> >> >>> > > > > > >>> >
> > > > >> >> >>> > > > > > >>>
> > > > >> >> >>> > > > > > >>
> > > > >> >> >>> > > > > >
> > > > >> >> >>> > > > >
> > > > >> >> >>> > > >
> > > > >> >> >>> > >
> > > > >> >> >>> >
> > > > >> >> >>>
> > > > >> >> >>
> > > > >> >>
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by PRATEEK GAUR <pr...@gmail.com>.
[Rust] Awesome thanks Andrew, Kostas and Devin.
[Java] Thanks Gang for the feedback on Java PR.
[C++] Thanks Gang for another round of feedback on the C++ one.
[Go] Arnav and I will be working on the golang impl this week.
This is great progress all around.
On Sat, Sep 5, 2026 at 1:26 PM Vinoo Ganesh <vi...@gmail.com> wrote:
> Thanks all - I'm addressing Gang's comments on
> https://github.com/apache/parquet-java/pull/3397 now, so hopefully well
> also be in a good state for merging the java implementation soon.
>
> Thanks,
> Vinoo Ganesh | vinoo.ganesh@gmail.com
>
> <Vi...@gmail.com>
>
>
> On Sat, Sep 5, 2026 at 8:56 AM Andrew Lamb <an...@gmail.com> wrote:
>
> > We just merged the Rust implementation[1] from Kosta and Devan. With the
> > imminent release of Parquet Format 2.14.0 I think we'll be well on our
> way.
> >
> > Thanks again for pushing this along,
> > Andrew
> >
> > [1]: https://github.com/apache/arrow-rs/pull/9372
> >
> > On Fri, Sep 4, 2026 at 5:42 PM PRATEEK GAUR <pr...@gmail.com> wrote:
> >
> > > Hi team,
> > >
> > > Thanks Micah for multiple rounds of review and thanks Gang for taking a
> > > look at the PR and giving feedback. I've addressed all comments in the
> > PR.
> > > Looking forward to the next round of feedback (or approval). Would like
> > to
> > > get it in soon.
> > >
> > > https://github.com/apache/arrow/pull/48345/changes
> > >
> > > With this done and after talking to Arnav, I plan to spend some time on
> > the
> > > ALP implementation in the arrow-go repository helping spread the
> support
> > to
> > > more languages.
> > >
> > > Best
> > > Prateek
> > >
> > > On Sun, Aug 2, 2026 at 10:42 PM PRATEEK GAUR <pr...@gmail.com>
> wrote:
> > >
> > > > Thanks Divjot for bringing it up.
> > > >
> > > > I was under the same impression as Russell and Andrew that we'd have
> it
> > > > flag gated.
> > > > With the new versioning proposal 'preview' might be a good tag for
> it.
> > > >
> > > > Best
> > > > Prateek
> > > >
> > > > On Sat, Aug 1, 2026 at 12:39 AM Andrew Lamb <an...@gmail.com>
> > > > wrote:
> > > >
> > > >> Div: Here is a proposal to mark ALP in Preview: [1] (using a comment
> > in
> > > >> the
> > > >> spec section).
> > > >>
> > > >> Russel: That is my understanding as well.
> > > >>
> > > >> Hopefully we are able to formalize the versioning process shortly so
> > we
> > > >> can
> > > >> formalize the notion of "preview"
> > > >>
> > > >> Andrew
> > > >>
> > > >>
> > > >> [1]: https://github.com/apache/parquet-format/pull/604
> > > >>
> > > >> On Fri, Jul 31, 2026 at 11:15 AM Russell Spitzer <
> > > >> russell.spitzer@gmail.com>
> > > >> wrote:
> > > >>
> > > >> > I thought we had agreed it would be merged, and features like this
> > > would
> > > >> > be feature flag gated. So a V-Old writer could always opt in to
> > using
> > > >> this
> > > >> > encoding but that must be a conscious effort if you are not using
> > > VNext
> > > >> of
> > > >> > Parquet. If you are on V-Next the writer can choose to use ALP
> > > whenever
> > > >> it
> > > >> > likes without user intervention.
> > > >> >
> > > >> > On Fri, Jul 31, 2026 at 2:20 AM Divjot Arora via dev <
> > > >> > dev@parquet.apache.org> wrote:
> > > >> >
> > > >> >> Yes, that’s my understanding as well. However, as you noted, the
> > > >> >> versioning
> > > >> >> discussion is still in early stages and hasn’t prescribed
> concrete
> > > >> >> mechanisms yet. If ALP goes into parquet-format 2.14.0, there’s
> no
> > > >> >> indication that it’s in preview or that writers shouldn’t use it
> by
> > > >> >> default
> > > >> >> yet. Given that we’ve held off on merging other incompatible
> > changes
> > > >> (e.g
> > > >> >> removal of path_in_schema) until the versioning discussion
> closes,
> > my
> > > >> >> preference would be to leave ALP un-merged for the time being as
> > > well.
> > > >> >>
> > > >> >> — Div
> > > >> >>
> > > >> >> On Fri, Jul 31, 2026 at 08:59 Andrew Lamb <
> andrewlamb11@gmail.com>
> > > >> wrote:
> > > >> >>
> > > >> >> > After re-reading Julien's Versioning Proposal, my understanding
> > is
> > > >> that
> > > >> >> if
> > > >> >> > we adopt that system, we will somehow mark ALP in "Preview" in
> > > >> >> > parquet-format for release Version 3.
> > > >> >> >
> > > >> >> > The versioning document doesn't seem to proscribe the
> particular
> > > >> method
> > > >> >> > for annotating "preview" features -- I think clearly marking
> them
> > > as
> > > >> >> such
> > > >> >> > in parquet.thrift as well as reflecting that on the
> > > >> >> > https://parquet.apache.org/docs/file-format/versions/ page
> > somehow
> > > >> >> would
> > > >> >> > be adequate.
> > > >> >> >
> > > >> >> > Andrew
> > > >> >> >
> > > >> >> > On Thu, Jul 30, 2026 at 11:50 AM Andrew Lamb <
> > > andrewlamb11@gmail.com
> > > >> >
> > > >> >> > wrote:
> > > >> >> >
> > > >> >> >> In my mind ALP is in the same position as other encodings that
> > > were
> > > >> >> added
> > > >> >> >> recently[1] (such as BYTE_STREAM_SPLIT for non floating point
> > > types)
> > > >> >> >>
> > > >> >> >> That is to say it will be available in the latest open source
> > > >> writers
> > > >> >> and
> > > >> >> >> readers, but realistically not widely used in the ecosystem
> > where
> > > >> >> >> compatibility is a concern (a large part)
> > > >> >> >>
> > > >> >> >> I view Julian's work[2][3] to gather consensus on how to name
> > > >> >> >> combinations of features (e.g. "versions") as the mechanism by
> > > >> which we
> > > >> >> >> will accelerate the adoption of this change (along with other
> > > >> forward
> > > >> >> >> incompatible changes) across the ecosystem.
> > > >> >> >>
> > > >> >> >> > Will we exclude this change when cutting parquet-format
> > v2.14.0?
> > > >> >> >>
> > > >> >> >> No I don't think so -- I would expect it to go out in the next
> > > >> >> >> parquet-format version as do other changes in the spec.
> > > >> >> >>
> > > >> >> >> Andrew
> > > >> >> >>
> > > >> >> >> [1]: https://parquet.apache.org/docs/file-format/versions/
> > > >> >> >> [2]:
> > > >> https://lists.apache.org/thread/xmnj8h0h8ozmrgox5tydhhs7yz7q2hss
> > > >> >> >> [3]:
> > > >> https://lists.apache.org/thread/qlf8lg90gqq41lllyy9mk2f0skqvftqj
> > > >> >> >>
> > > >> >> >>
> > > >> >> >> On Thu, Jul 30, 2026 at 3:51 AM Divjot Arora via dev <
> > > >> >> >> dev@parquet.apache.org> wrote:
> > > >> >> >>
> > > >> >> >>> Hi Andrew,
> > > >> >> >>>
> > > >> >> >>> How does this interact with the ongoing versioning
> discussion?
> > > >> Given
> > > >> >> that
> > > >> >> >>> new encodings are not forward compatible, I would assume ALP
> is
> > > >> >> included
> > > >> >> >>> in
> > > >> >> >>> Parquet V<next> (V3?), but it's not clear to me how that
> works
> > if
> > > >> it's
> > > >> >> >>> merged to the parquet-format master branch now. Will we
> exclude
> > > >> this
> > > >> >> >>> change
> > > >> >> >>> when cutting parquet-format v2.14.0?
> > > >> >> >>>
> > > >> >> >>> -- Div
> > > >> >> >>>
> > > >> >> >>> On Thu, Jul 30, 2026 at 8:51 AM Andrew Lamb <
> > > >> andrewlamb11@gmail.com>
> > > >> >> >>> wrote:
> > > >> >> >>>
> > > >> >> >>> > Hello,
> > > >> >> >>> >
> > > >> >> >>> > As an update, thanks to the epic work from many
> contributors,
> > > >> >> >>> especially
> > > >> >> >>> > Prateek, I just merged the parquet-format change for
> ALP[1].
> > We
> > > >> >> plan to
> > > >> >> >>> > make additional editorial The remaining items I know of are
> > > >> merging
> > > >> >> the
> > > >> >> >>> > examples from parquet-testing[2], and then of course the
> > > various
> > > >> >> >>> > implementations
> > > >> >> >>> >
> > > >> >> >>> > Thanks again everyone. This is a great step forward
> > > >> >> >>> > Andrew
> > > >> >> >>> >
> > > >> >> >>> > [1]: https://github.com/apache/parquet-format/pull/557
> > > >> >> >>> > [2]: https://github.com/apache/parquet-testing/pull/100
> > > >> >> >>> >
> > > >> >> >>> > On Tue, Jul 21, 2026 at 4:39 PM Julien Le Dem <
> > > julien@apache.org
> > > >> >
> > > >> >> >>> wrote:
> > > >> >> >>> >
> > > >> >> >>> > > It was also in my spam folder... :(
> > > >> >> >>> > > I've seen a bunch of apache list emails go to my spam
> > > recently.
> > > >> >> >>> > >
> > > >> >> >>> > > On Tue, Jul 21, 2026 at 10:58 AM Russell Spitzer <
> > > >> >> >>> > > russell.spitzer@gmail.com>
> > > >> >> >>> > > wrote:
> > > >> >> >>> > >
> > > >> >> >>> > > > Yes, my hope was that my vote would kick it out of spam
> > but
> > > >> we
> > > >> >> >>> > definitely
> > > >> >> >>> > > > had two other votes that also went to spam for me.
> > > >> >> >>> > > >
> > > >> >> >>> > > > For anyone looking the thread is here
> > > >> >> >>> > > >
> > > >> >> https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt
> > > >> >> >>> > > >
> > > >> >> >>> > > > On Tue, Jul 21, 2026 at 10:57 AM Micah Kornfield <
> > > >> >> >>> > emkornfield@gmail.com>
> > > >> >> >>> > > > wrote:
> > > >> >> >>> > > >
> > > >> >> >>> > > > > I've seen other's votes not go to spam, so hopefully
> > > people
> > > >> >> will
> > > >> >> >>> see
> > > >> >> >>> > it
> > > >> >> >>> > > > in
> > > >> >> >>> > > > > there inboxes (or see this thread and look in there
> > > spam).
> > > >> >> >>> > > > >
> > > >> >> >>> > > > > On Tue, Jul 21, 2026 at 10:52 AM PRATEEK GAUR <
> > > >> >> >>> prtkgaur@gmail.com>
> > > >> >> >>> > > > wrote:
> > > >> >> >>> > > > >
> > > >> >> >>> > > > > > Okay,
> > > >> >> >>> > > > > >
> > > >> >> >>> > > > > > I sent out the voting email yesterday and so far
> I've
> > > >> heard
> > > >> >> >>> from 3
> > > >> >> >>> > > > people
> > > >> >> >>> > > > > > that it ended up in their "spam folder". Dev VPN to
> > be
> > > >> >> blamed
> > > >> >> >>> for
> > > >> >> >>> > > it?.
> > > >> >> >>> > > > > >
> > > >> >> >>> > > > > > What's the protocol here?
> > > >> >> >>> > > > > > I let people hunt for the email in their spam
> folder
> > or
> > > >> send
> > > >> >> >>> out
> > > >> >> >>> > > > another
> > > >> >> >>> > > > > > one?
> > > >> >> >>> > > > > >
> > > >> >> >>> > > > > > Best
> > > >> >> >>> > > > > > Prateek
> > > >> >> >>> > > > > >
> > > >> >> >>> > > > > > On Sun, Jul 19, 2026 at 10:47 PM PRATEEK GAUR <
> > > >> >> >>> prtkgaur@gmail.com>
> > > >> >> >>> > > > > wrote:
> > > >> >> >>> > > > > >
> > > >> >> >>> > > > > > > Hi team,
> > > >> >> >>> > > > > > >
> > > >> >> >>> > > > > > > Vinoo has added the tests which were requested in
> > the
> > > >> last
> > > >> >> >>> round
> > > >> >> >>> > of
> > > >> >> >>> > > > > > review.
> > > >> >> >>> > > > > > > In his own words.
> > > >> >> >>> > > > > > >
> > > >> >> >>> > > > > > > "
> > > >> >> >>> > > > > > > The extreme value tests Micah wanted are done and
> > > >> pushed.
> > > >> >> I
> > > >> >> >>> added
> > > >> >> >>> > > > > > coverage
> > > >> >> >>> > > > > > > for values that need the full FOR bit width after
> > the
> > > >> >> frame
> > > >> >> >>> of
> > > >> >> >>> > > > > reference
> > > >> >> >>> > > > > > is
> > > >> >> >>> > > > > > > applied: 64-bit (the case where the signed max
> > minus
> > > >> min
> > > >> >> >>> > > subtraction
> > > >> >> >>> > > > > > > overflows), 63-bit (the non-overflow case), and
> > > 32-bit
> > > >> for
> > > >> >> >>> > floats.
> > > >> >> >>> > > > All
> > > >> >> >>> > > > > > > lossless with zero exceptions. I also verified
> them
> > > >> >> through
> > > >> >> >>> Arrow
> > > >> >> >>> > > C++
> > > >> >> >>> > > > > ALP
> > > >> >> >>> > > > > > > reader (the local build), so the Java-written
> > extreme
> > > >> >> columns
> > > >> >> >>> > > decode
> > > >> >> >>> > > > > > > correctly in C++, bit-exact against the expected
> > > >> values."
> > > >> >> >>> > > > > > >
> > > >> >> >>> > > > > > > State of the PR (#3397):
> > > >> >> >>> > > > > > >
> > > >> >> >>> > > > > > > - Merges cleanly with master (I pulled master
> in
> > > and
> > > >> >> >>> resolved
> > > >> >> >>> > > the
> > > >> >> >>> > > > > > > conflicts).
> > > >> >> >>> > > > > > > - 41 of 44 review threads resolved. The 3
> > > remaining
> > > >> are
> > > >> >> >>> > > > intentional:
> > > >> >> >>> > > > > > > one deferred perf nit, one that stays open
> until
> > > >> >> >>> > parquet-format
> > > >> >> >>> > > > > ships
> > > >> >> >>> > > > > > ALP,
> > > >> >> >>> > > > > > >
> > > >> >> >>> > > > > > > "
> > > >> >> >>> > > > > > >
> > > >> >> >>> > > > > > > With this both the
> > > >> >> >>> > > > > > > 1) C++ PR :
> > > >> >> >>> https://github.com/apache/arrow/pull/48345/changes
> > > >> >> >>> > > > > > > 2) Java PR :
> > > >> >> >>> https://github.com/apache/parquet-java/pull/3397
> > > >> >> >>> > > > > > >
> > > >> >> >>> > > > > > > Are cross language tested and range of test
> > > scenarios.
> > > >> >> >>> > > > > > >
> > > >> >> >>> > > > > > > Best
> > > >> >> >>> > > > > > > Prateek
> > > >> >> >>> > > > > > >
> > > >> >> >>> > > > > > >
> > > >> >> >>> > > > > > > On Mon, Jul 13, 2026 at 8:38 AM PRATEEK GAUR <
> > > >> >> >>> prtkgaur@gmail.com
> > > >> >> >>> > >
> > > >> >> >>> > > > > wrote:
> > > >> >> >>> > > > > > >
> > > >> >> >>> > > > > > >> Thanks Matt,
> > > >> >> >>> > > > > > >>
> > > >> >> >>> > > > > > >> I'll be on it this week.
> > > >> >> >>> > > > > > >>
> > > >> >> >>> > > > > > >> Best
> > > >> >> >>> > > > > > >> Prateek
> > > >> >> >>> > > > > > >>
> > > >> >> >>> > > > > > >> On Mon, Jul 13, 2026 at 8:36 AM Matt Topol <
> > > >> >> >>> > > zotthewizard@gmail.com>
> > > >> >> >>> > > > > > >> wrote:
> > > >> >> >>> > > > > > >>
> > > >> >> >>> > > > > > >>> There is also the Go implementation submitted
> by
> > > >> Arnav (
> > > >> >> >>> > > > > > >>> https://github.com/apache/arrow-go/pull/704)
> > which
> > > >> is
> > > >> >> >>> waiting
> > > >> >> >>> > > for
> > > >> >> >>> > > > > > >>> updates.
> > > >> >> >>> > > > > > >>>
> > > >> >> >>> > > > > > >>> Just wanted to make sure it didn't get lost
> here.
> > > >> >> >>> > > > > > >>>
> > > >> >> >>> > > > > > >>> --Matt
> > > >> >> >>> > > > > > >>>
> > > >> >> >>> > > > > > >>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK GAUR <
> > > >> >> >>> > > prtkgaur@gmail.com>
> > > >> >> >>> > > > > > >>> wrote:
> > > >> >> >>> > > > > > >>>
> > > >> >> >>> > > > > > >>> > Hi all,
> > > >> >> >>> > > > > > >>> >
> > > >> >> >>> > > > > > >>> > I'd like to share a status update on the ALP
> > > >> encoding
> > > >> >> >>> effort
> > > >> >> >>> > > and
> > > >> >> >>> > > > > get
> > > >> >> >>> > > > > > a
> > > >> >> >>> > > > > > >>> feel
> > > >> >> >>> > > > > > >>> > from the community before starting a formal
> > vote.
> > > >> >> >>> > > > > > >>> >
> > > >> >> >>> > > > > > >>> > We now have the specification plus
> > > implementations
> > > >> in
> > > >> >> >>> both
> > > >> >> >>> > > > > languages:
> > > >> >> >>> > > > > > >>> >
> > > >> >> >>> > > > > > >>> > - Spec:
> apache/parquet-format#557 —
> > > >> GH-533
> > > >> >> >>> Add
> > > >> >> >>> > ALP
> > > >> >> >>> > > > > > encoding
> > > >> >> >>> > > > > > >>> > specification
> > > >> >> >>> > > > > > >>> > - C++ (Arrow): apache/arrow#48345 —
> > GH-48701
> > > >> >> >>> > > [C++][Parquet]
> > > >> >> >>> > > > > Add
> > > >> >> >>> > > > > > >>> ALPpd
> > > >> >> >>> > > > > > >>> > encoding
> > > >> >> >>> > > > > > >>> > - Java: apache/parquet-java#3397
> —
> > > >> Parquet
> > > >> >> >>> Java
> > > >> >> >>> > ALP
> > > >> >> >>> > > > > > >>> > Implementation (by Vinoo)
> > > >> >> >>> > > > > > >>> >
> > > >> >> >>> > > > > > >>> > Current state:
> > > >> >> >>> > > > > > >>> >
> > > >> >> >>> > > > > > >>> > - The C++ implementation has been through
> > 3-4
> > > >> >> rounds
> > > >> >> >>> of
> > > >> >> >>> > > > review.
> > > >> >> >>> > > > > > >>> > - The Java implementation has been
> tracking
> > > >> well
> > > >> >> and
> > > >> >> >>> has
> > > >> >> >>> > > had
> > > >> >> >>> > > > > > >>> initial
> > > >> >> >>> > > > > > >>> > reviews from contributors, with no
> major
> > > >> >> remaining
> > > >> >> >>> > > issues.
> > > >> >> >>> > > > > > >>> > - Cross-language compatibility tests are
> > > >> passing:
> > > >> >> the
> > > >> >> >>> > Arrow
> > > >> >> >>> > > > C++
> > > >> >> >>> > > > > > >>> decoder
> > > >> >> >>> > > > > > >>> > reads Java-written data bit-exactly
> > across
> > > >> >> ~1.56M
> > > >> >> >>> > values
> > > >> >> >>> > > > and
> > > >> >> >>> > > > > 18
> > > >> >> >>> > > > > > >>> > fixtures, covering V1 and V2 pages,
> > > multiple
> > > >> >> vector
> > > >> >> >>> > > sizes,
> > > >> >> >>> > > > > and
> > > >> >> >>> > > > > > >>> > several
> > > >> >> >>> > > > > > >>> > real datasets — zero mismatches.
> > > >> >> >>> > > > > > >>> >
> > > >> >> >>> > > > > > >>> > As raised in the java-pr we will be adding
> > > >> coverage
> > > >> >> for
> > > >> >> >>> > > extreme
> > > >> >> >>> > > > > > values
> > > >> >> >>> > > > > > >>> > (those requiring
> > > >> >> >>> > > > > > >>> > 63-64 bits after FOR is applied) before we
> > > close a
> > > >> >> vote.
> > > >> >> >>> > We'll
> > > >> >> >>> > > > aim
> > > >> >> >>> > > > > > to
> > > >> >> >>> > > > > > >>> get
> > > >> >> >>> > > > > > >>> > that
> > > >> >> >>> > > > > > >>> > done in parallel.
> > > >> >> >>> > > > > > >>> >
> > > >> >> >>> > > > > > >>> > Plan: unless there are objections, I intend
> to
> > > >> start a
> > > >> >> >>> vote
> > > >> >> >>> > at
> > > >> >> >>> > > > the
> > > >> >> >>> > > > > > end
> > > >> >> >>> > > > > > >>> of
> > > >> >> >>> > > > > > >>> > this week/early
> > > >> >> >>> > > > > > >>> > next week.
> > > >> >> >>> > > > > > >>> >
> > > >> >> >>> > > > > > >>> > Thanks
> > > >> >> >>> > > > > > >>> > Prateek
> > > >> >> >>> > > > > > >>> >
> > > >> >> >>> > > > > > >>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <
> > > >> >> >>> > > prtkgaur@gmail.com>
> > > >> >> >>> > > > > > >>> wrote:
> > > >> >> >>> > > > > > >>> >
> > > >> >> >>> > > > > > >>> > > Hi Team,
> > > >> >> >>> > > > > > >>> > >
> > > >> >> >>> > > > > > >>> > > Just wanted to provide some updates on ALP.
> > > >> >> >>> > > > > > >>> > > Micah and I have done 3-4 rounds of review
> > for
> > > >> the
> > > >> >> c++
> > > >> >> >>> PR :
> > > >> >> >>> > > > > > >>> > > https://github.com/apache/arrow/pull/48345
> > > >> >> >>> > > > > > >>> > >
> > > >> >> >>> > > > > > >>> > > For the Java implementation Vinoo has been
> > > >> working
> > > >> >> on
> > > >> >> >>> > > following
> > > >> >> >>> > > > > PR
> > > >> >> >>> > > > > > :
> > > >> >> >>> > > > > > >>> > >
> > > https://github.com/apache/parquet-java/pull/3397
> > > >> >> >>> > > > > > >>> > >
> > > >> >> >>> > > > > > >>> > > Best
> > > >> >> >>> > > > > > >>> > > Prateek
> > > >> >> >>> > > > > > >>> > >
> > > >> >> >>> > > > > > >>> > > On Tue, May 5, 2026 at 1:50 PM Micah
> > Kornfield
> > > <
> > > >> >> >>> > > > > > >>> emkornfield@gmail.com>
> > > >> >> >>> > > > > > >>> > > wrote:
> > > >> >> >>> > > > > > >>> > >
> > > >> >> >>> > > > > > >>> > >> Hi Antoine,
> > > >> >> >>> > > > > > >>> > >>
> > > >> >> >>> > > > > > >>> > >> > Apologies if the question was already
> > asked,
> > > >> but
> > > >> >> >>> should
> > > >> >> >>> > we
> > > >> >> >>> > > > > care
> > > >> >> >>> > > > > > >>> about
> > > >> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more
> > > >> >> efficient
> > > >> >> >>> than
> > > >> >> >>> > > > > FLOAT16
> > > >> >> >>> > > > > > +
> > > >> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> > > >> >> >>> > > > > > >>> > >>
> > > >> >> >>> > > > > > >>> > >> It was. We thought we could defer it for
> > the
> > > >> >> >>> following
> > > >> >> >>> > > > reasons:
> > > >> >> >>> > > > > > >>> > >> 1. It's not clear there are a lot of easy
> > > >> >> reference
> > > >> >> >>> > > datasets
> > > >> >> >>> > > > to
> > > >> >> >>> > > > > > >>> test
> > > >> >> >>> > > > > > >>> > its
> > > >> >> >>> > > > > > >>> > >> effectiveness. It does look like there
> > might
> > > be
> > > >> >> one
> > > >> >> >>> or
> > > >> >> >>> > two
> > > >> >> >>> > > on
> > > >> >> >>> > > > > > >>> > huggingface
> > > >> >> >>> > > > > > >>> > >> (e.g.
> > > >> >> https://huggingface.co/datasets/kikitora/curdie
> > > >> >> >>> ).
> > > >> >> >>> > > > > > >>> > >> 2. It seemed likely that float 16 was
> more
> > > >> likely
> > > >> >> >>> used
> > > >> >> >>> > for
> > > >> >> >>> > > > > values
> > > >> >> >>> > > > > > >>> that
> > > >> >> >>> > > > > > >>> > >> were less likely to reduce to decimal
> > values.
> > > >> >> >>> > > > > > >>> > >> 3. It could be added as an extension
> later
> > if
> > > >> >> needed.
> > > >> >> >>> > > > > > >>> > >>
> > > >> >> >>> > > > > > >>> > >> Cheers,
> > > >> >> >>> > > > > > >>> > >> Micah
> > > >> >> >>> > > > > > >>> > >>
> > > >> >> >>> > > > > > >>> > >>
> > > >> >> >>> > > > > > >>> > >> On Tue, May 5, 2026 at 1:40 PM Antoine
> > Pitrou
> > > <
> > > >> >> >>> > > > > antoine@python.org
> > > >> >> >>> > > > > > >
> > > >> >> >>> > > > > > >>> > wrote:
> > > >> >> >>> > > > > > >>> > >>
> > > >> >> >>> > > > > > >>> > >> >
> > > >> >> >>> > > > > > >>> > >> > Hello,
> > > >> >> >>> > > > > > >>> > >> >
> > > >> >> >>> > > > > > >>> > >> > Apologies if the question was already
> > asked,
> > > >> but
> > > >> >> >>> should
> > > >> >> >>> > we
> > > >> >> >>> > > > > care
> > > >> >> >>> > > > > > >>> about
> > > >> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more
> > > >> >> efficient
> > > >> >> >>> than
> > > >> >> >>> > > > > FLOAT16
> > > >> >> >>> > > > > > +
> > > >> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> > > >> >> >>> > > > > > >>> > >> >
> > > >> >> >>> > > > > > >>> > >> > Regards
> > > >> >> >>> > > > > > >>> > >> >
> > > >> >> >>> > > > > > >>> > >> > Antoine.
> > > >> >> >>> > > > > > >>> > >> >
> > > >> >> >>> > > > > > >>> > >> >
> > > >> >> >>> > > > > > >>> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a
> > écrit
> > > :
> > > >> >> >>> > > > > > >>> > >> > > Thanks Andrew and Micah for review
> > > feedback
> > > >> on
> > > >> >> >>> the two
> > > >> >> >>> > > > PR's
> > > >> >> >>> > > > > > >>> > >> > > 1) (c++ arrow repo)
> > > >> >> >>> > > > > > >>> > >>
> > > >> https://github.com/apache/arrow/pull/48345/changes
> > > >> >> >>> > > > > > >>> > >> > > 2) (parquet-format repo)
> > > >> >> >>> > > > > > >>> > >> >
> > > >> >> https://github.com/apache/parquet-format/pull/557
> > > >> >> >>> > > > > > >>> > >> > >
> > > >> >> >>> > > > > > >>> > >> > > I have addressed all (unless I missed
> > > >> >> something)
> > > >> >> >>> > > comments
> > > >> >> >>> > > > on
> > > >> >> >>> > > > > > >>> the two
> > > >> >> >>> > > > > > >>> > >> > PR's.
> > > >> >> >>> > > > > > >>> > >> > >
> > > >> >> >>> > > > > > >>> > >> > > Best
> > > >> >> >>> > > > > > >>> > >> > > Prateek
> > > >> >> >>> > > > > > >>> > >> > >
> > > >> >> >>> > > > > > >>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM
> PRATEEK
> > > >> GAUR <
> > > >> >> >>> > > > > > >>> prtkgaur@gmail.com>
> > > >> >> >>> > > > > > >>> > >> wrote:
> > > >> >> >>> > > > > > >>> > >> > >
> > > >> >> >>> > > > > > >>> > >> > >> Thanks Andrew and Micah.
> > > >> >> >>> > > > > > >>> > >> > >>
> > > >> >> >>> > > > > > >>> > >> > >> `fair amount of feedback on at least
> > the
> > > >> >> >>> > > implementations`
> > > >> >> >>> > > > > > >>> > >> > >> For the c++ I have already started
> > > >> addressing
> > > >> >> the
> > > >> >> >>> > > > > feedback, I
> > > >> >> >>> > > > > > >>> > should
> > > >> >> >>> > > > > > >>> > >> be
> > > >> >> >>> > > > > > >>> > >> > >> done with that Monday/Tuesday.
> > > >> >> >>> > > > > > >>> > >> > >> I think Vinoo too has been making
> good
> > > >> >> progress
> > > >> >> >>> on
> > > >> >> >>> > the
> > > >> >> >>> > > > Java
> > > >> >> >>> > > > > > >>> > >> > implementation.
> > > >> >> >>> > > > > > >>> > >> > >>
> > > >> >> >>> > > > > > >>> > >> > >> Best
> > > >> >> >>> > > > > > >>> > >> > >> Prateek
> > > >> >> >>> > > > > > >>> > >> > >>
> > > >> >> >>> > > > > > >>> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM
> Andrew
> > > >> Lamb <
> > > >> >> >>> > > > > > >>> > andrewlamb11@gmail.com
> > > >> >> >>> > > > > > >>> > >> >
> > > >> >> >>> > > > > > >>> > >> > >> wrote:
> > > >> >> >>> > > > > > >>> > >> > >>
> > > >> >> >>> > > > > > >>> > >> > >>> Got it. Thank you for the
> > clarification
> > > >> -- I
> > > >> >> >>> will
> > > >> >> >>> > try
> > > >> >> >>> > > > and
> > > >> >> >>> > > > > > look
> > > >> >> >>> > > > > > >>> > into
> > > >> >> >>> > > > > > >>> > >> the
> > > >> >> >>> > > > > > >>> > >> > >>> spec and the Rust implementation[1]
> in
> > > >> this
> > > >> >> next
> > > >> >> >>> > week
> > > >> >> >>> > > > > > >>> > >> > >>>
> > > >> >> >>> > > > > > >>> > >> > >>> [1]:
> > > >> >> >>> https://github.com/apache/arrow-rs/pull/9372
> > > >> >> >>> > > > > > >>> > >> > >>>
> > > >> >> >>> > > > > > >>> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM
> Micah
> > > >> >> >>> Kornfield <
> > > >> >> >>> > > > > > >>> > >> > emkornfield@gmail.com>
> > > >> >> >>> > > > > > >>> > >> > >>> wrote:
> > > >> >> >>> > > > > > >>> > >> > >>>
> > > >> >> >>> > > > > > >>> > >> > >>>> Hi Andrew,
> > > >> >> >>> > > > > > >>> > >> > >>>> I think there is a fair amount of
> > > >> feedback
> > > >> >> on
> > > >> >> >>> at
> > > >> >> >>> > > least
> > > >> >> >>> > > > > the
> > > >> >> >>> > > > > > >>> > >> > >>>> implementations, typically I think
> > > we've
> > > >> >> waited
> > > >> >> >>> > till
> > > >> >> >>> > > > they
> > > >> >> >>> > > > > > are
> > > >> >> >>> > > > > > >>> > >> close to
> > > >> >> >>> > > > > > >>> > >> > >>>> mergeable before a final vote.
> > > >> Otherwise I
> > > >> >> >>> agree
> > > >> >> >>> > we
> > > >> >> >>> > > > are
> > > >> >> >>> > > > > > very
> > > >> >> >>> > > > > > >>> > >> close.
> > > >> >> >>> > > > > > >>> > >> > >>>>
> > > >> >> >>> > > > > > >>> > >> > >>>> -Micah
> > > >> >> >>> > > > > > >>> > >> > >>>>
> > > >> >> >>> > > > > > >>> > >> > >>>> On Saturday, April 25, 2026, Andrew
> > > Lamb
> > > >> <
> > > >> >> >>> > > > > > >>> andrewlamb11@gmail.com
> > > >> >> >>> > > > > > >>> > >
> > > >> >> >>> > > > > > >>> > >> > wrote:
> > > >> >> >>> > > > > > >>> > >> > >>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>> Thanks Prateek,
> > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>> I think from this content it looks
> > to
> > > me
> > > >> >> like
> > > >> >> >>> we
> > > >> >> >>> > are
> > > >> >> >>> > > > > ready
> > > >> >> >>> > > > > > >>> to
> > > >> >> >>> > > > > > >>> > >> start a
> > > >> >> >>> > > > > > >>> > >> > >>>>> vote to explicitly accept ALP into
> > > >> Parquet
> > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>> Does anyone know of a reason we
> > should
> > > >> >> >>> postpone it
> > > >> >> >>> > > for
> > > >> >> >>> > > > > > >>> longer?
> > > >> >> >>> > > > > > >>> > >> > >>>>> Perhaps someone needs some more
> time
> > > to
> > > >> >> >>> review?
> > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>> Andrew
> > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM
> > > PRATEEK
> > > >> >> GAUR <
> > > >> >> >>> > > > > > >>> > prtkgaur@gmail.com>
> > > >> >> >>> > > > > > >>> > >> > >>>>> wrote:
> > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>> Hi team,
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>> Hope everyone is doing well. I
> got
> > a
> > > >> >> chance
> > > >> >> >>> to
> > > >> >> >>> > work
> > > >> >> >>> > > > > > >>> through all
> > > >> >> >>> > > > > > >>> > >> the
> > > >> >> >>> > > > > > >>> > >> > >>>>>> remaining feedback and update the
> > > spec
> > > >> >> doc.
> > > >> >> >>> Here
> > > >> >> >>> > > are
> > > >> >> >>> > > > > the
> > > >> >> >>> > > > > > >>> new
> > > >> >> >>> > > > > > >>> > >> > artifacts
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>> 1) Spec document :
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> >
> > > >> >> >>> > > > > > >>> > >>
> > > >> >> >>> > > > > > >>> >
> > > >> >> >>> > > > > > >>>
> > > >> >> >>> > > > > >
> > > >> >> >>> > > > >
> > > >> >> >>> > > >
> > > >> >> >>> > >
> > > >> >> >>> >
> > > >> >> >>>
> > > >> >>
> > > >>
> > >
> >
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>> 2) Spec document in parquet
> format
> > > >> repo :
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > https://github.com/apache/parquet-format/pull/557
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>> 3) Alp implementation in arrow
> c++
> > > >> repo :
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > https://github.com/apache/arrow/pull/48345/changes
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>> 4) Alp implementation in
> > parquet-java
> > > >> >> repo :
> > > >> >> >>> Work
> > > >> >> >>> > > for
> > > >> >> >>> > > > > > >>> Vinoo and
> > > >> >> >>> > > > > > >>> > >> > Julien
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > https://github.com/apache/parquet-java/pull/3397
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>> 5) PR with test and benchmarking
> > > >> >> artifacts in
> > > >> >> >>> > > > > > >>> parquet-testing
> > > >> >> >>> > > > > > >>> > >> repo :
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > https://github.com/apache/parquet-testing/pull/100
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>> And
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>> - Go : Arnav just submitted
> an
> > in
> > > >> >> >>> progress
> > > >> >> >>> > > > > > >>> implementation
> > > >> >> >>> > > > > > >>> > in
> > > >> >> >>> > > > > > >>> > >> Go.
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> https://github.com/apache/arrow-go/pull/704
> > > >> >> >>> > (I
> > > >> >> >>> > > > > > haven't
> > > >> >> >>> > > > > > >>> > >> started
> > > >> >> >>> > > > > > >>> > >> > >>>>>> looking at it yet)
> > > >> >> >>> > > > > > >>> > >> > >>>>>> - Rust : I remember Andrew
> > > >> mentioned
> > > >> >> that
> > > >> >> >>> > this
> > > >> >> >>> > > > work
> > > >> >> >>> > > > > > is
> > > >> >> >>> > > > > > >>> also
> > > >> >> >>> > > > > > >>> > >> in
> > > >> >> >>> > > > > > >>> > >> > >>>>>> progress (So 4 languages!)
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>> *Arrow C++ implementation *
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>> The PR is out and was also used
> by
> > > >> >> Antoine to
> > > >> >> >>> > > report
> > > >> >> >>> > > > > the
> > > >> >> >>> > > > > > >>> > numbers
> > > >> >> >>> > > > > > >>> > >> as
> > > >> >> >>> > > > > > >>> > >> > >>>>>> reported here. Micah and
> Konstantin
> > > >> have
> > > >> >> >>> given 1
> > > >> >> >>> > > > round
> > > >> >> >>> > > > > of
> > > >> >> >>> > > > > > >>> > >> feedback
> > > >> >> >>> > > > > > >>> > >> > >>>>>> and I'm addressing them today.
> > Please
> > > >> note
> > > >> >> >>> that
> > > >> >> >>> > the
> > > >> >> >>> > > > > > default
> > > >> >> >>> > > > > > >>> > >> > >>>>>> optimization flag for compiling
> is
> > O2
> > > >> and
> > > >> >> not
> > > >> >> >>> > Q3. I
> > > >> >> >>> > > > got
> > > >> >> >>> > > > > > >>> around
> > > >> >> >>> > > > > > >>> > >> 70%
> > > >> >> >>> > > > > > >>> > >> > >>>>>> performance improvement in the
> > > decoding
> > > >> >> speed
> > > >> >> >>> > when
> > > >> >> >>> > > > > using
> > > >> >> >>> > > > > > >>> the O3
> > > >> >> >>> > > > > > >>> > >> > flag.
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>> *Parqet-MR Java implementation
> > > (working
> > > >> >> with
> > > >> >> >>> > Vinoo
> > > >> >> >>> > > > and
> > > >> >> >>> > > > > > >>> Julien)
> > > >> >> >>> > > > > > >>> > >> and
> > > >> >> >>> > > > > > >>> > >> > **Cross
> > > >> >> >>> > > > > > >>> > >> > >>>>>> Language testing*
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>> Let me know if you have any
> > > >> questions
> > > >> >> or
> > > >> >> >>> > > > feedback.
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>> Now pasting some performance
> > numbers
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>> Table 1: C++ ALP Double
> Decode —
> > > >> >> Spotify
> > > >> >> >>> > Columns
> > > >> >> >>> > > > > > >>> (Graviton
> > > >> >> >>> > > > > > >>> > 3,
> > > >> >> >>> > > > > > >>> > >> ARM
> > > >> >> >>> > > > > > >>> > >> > >>>>>> Neoverse V1)
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>>
> > > >> >> >>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>> │ Column │ -O2
> (MB/s)
> > > │
> > > >> >> -O3
> > > >> >> >>> > > (MB/s) │
> > > >> >> >>> > > > > > >>> Speedup │
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>>
> > > >> >> >>> ├──────────────────┼──────────────┼──────────────┼─────────┤
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>> │ valence │ 3,155
> > > │
> > > >> >> >>> 5,523
> > > >> >> >>> > > │
> > > >> >> >>> > > > > > >>> 1.75x │
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>> │ danceability │ 3,233
> > > │
> > > >> >> >>> 5,685
> > > >> >> >>> > > │
> > > >> >> >>> > > > > > >>> 1.76x │
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>> │ energy │ 3,197
> > > │
> > > >> >> >>> 5,652
> > > >> >> >>> > > │
> > > >> >> >>> > > > > > >>> 1.77x │
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>> │ loudness │ 3,186
> > > │
> > > >> >> >>> 5,473
> > > >> >> >>> > > │
> > > >> >> >>> > > > > > >>> 1.72x │
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>>
> > > >> >> >>> └──────────────────┴──────────────┴──────────────┴─────────┘
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM
> > > PRATEEK
> > > >> >> GAUR
> > > >> >> >>> <
> > > >> >> >>> > > > > > >>> > prtkgaur@gmail.com
> > > >> >> >>> > > > > > >>> > >> >
> > > >> >> >>> > > > > > >>> > >> > >>>>>> wrote:
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>> @Micah Kornfield <
> > > >> emkornfield@gmail.com>
> > > >> >> :
> > > >> >> >>> Got
> > > >> >> >>> > > it.
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>> @Andrew Lamb <
> > > andrewlamb11@gmail.com>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> Do you think it would be good
> to
> > > >> start
> > > >> >> >>> moving
> > > >> >> >>> > the
> > > >> >> >>> > > > > spec
> > > >> >> >>> > > > > > >>> > >> development
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> into
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> markdown format, in preparation
> > for
> > > >> >> >>> finalizing
> > > >> >> >>> > > it?
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>> Yes I'll update the numbers for
> > some
> > > >> of
> > > >> >> the
> > > >> >> >>> > > > examples I
> > > >> >> >>> > > > > > >>> have in
> > > >> >> >>> > > > > > >>> > >> the
> > > >> >> >>> > > > > > >>> > >> > >>>>>>> spec based
> > > >> >> >>> > > > > > >>> > >> > >>>>>>> on the updated header size. Then
> > we
> > > >> >> should
> > > >> >> >>> be
> > > >> >> >>> > good
> > > >> >> >>> > > > to
> > > >> >> >>> > > > > go
> > > >> >> >>> > > > > > >>> for
> > > >> >> >>> > > > > > >>> > the
> > > >> >> >>> > > > > > >>> > >> > >>>>>>> markdown format.
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>> Thanks everyone!
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> Andrew
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM
> > > >> PRATEEK
> > > >> >> >>> GAUR <
> > > >> >> >>> > > > > > >>> > >> prtkgaur@gmail.com>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> wrote:
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Hi team,
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 1) Andrew
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for working on
> test
> > > >> files.
> > > >> >> >>> My PR
> > > >> >> >>> > > did
> > > >> >> >>> > > > > add
> > > >> >> >>> > > > > > >>> all
> > > >> >> >>> > > > > > >>> > the
> > > >> >> >>> > > > > > >>> > >> > test
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> files I
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> used to benchmark on
> > datasets.
> > > >> >> Maybe
> > > >> >> >>> we
> > > >> >> >>> > can
> > > >> >> >>> > > > club
> > > >> >> >>> > > > > > it
> > > >> >> >>> > > > > > >>> > >> together.
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> WIll also
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> aid
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> cross language testing
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Kosta Tarasov working
> on
> > > Rust
> > > >> >> >>> > > > implementation.
> > > >> >> >>> > > > > > >>> This is
> > > >> >> >>> > > > > > >>> > >> > great.
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> Thanks
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 2) Antoine
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks a lot for
> reporting
> > > the
> > > >> >> >>> numbers
> > > >> >> >>> > on
> > > >> >> >>> > > > AMD.
> > > >> >> >>> > > > > > >>> Looks
> > > >> >> >>> > > > > > >>> > >> like
> > > >> >> >>> > > > > > >>> > >> > you
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> are
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> getting 8X the decoding
> > > >> >> performance of
> > > >> >> >>> > BSS.
> > > >> >> >>> > > > This
> > > >> >> >>> > > > > > is
> > > >> >> >>> > > > > > >>> > >> > amazing!!.
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for acknowledging
> > the
> > > >> >> >>> sampling
> > > >> >> >>> > > > design.
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - I agree with you on
> > > >> Fastlanes. In
> > > >> >> >>> some
> > > >> >> >>> > > crude
> > > >> >> >>> > > > > > >>> > >> experiments I
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> didn't get
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> a good perf benefit from
> it
> > on
> > > >> >> >>> Graviton3
> > > >> >> >>> > > (but
> > > >> >> >>> > > > > > maybe
> > > >> >> >>> > > > > > >>> > there
> > > >> >> >>> > > > > > >>> > >> was
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> something
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> wrong with my
> > implementation).
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Locking the 16bit
> > exception
> > > >> >> >>> encoding for
> > > >> >> >>> > > the
> > > >> >> >>> > > > > > spec
> > > >> >> >>> > > > > > >>> in
> > > >> >> >>> > > > > > >>> > >> this
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> case.
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Awesome I think we have
> > > solved
> > > >> >> for
> > > >> >> >>> all
> > > >> >> >>> > > open
> > > >> >> >>> > > > > > >>> questions
> > > >> >> >>> > > > > > >>> > >> minus
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> the
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> version byte :). (will get
> > > back
> > > >> on
> > > >> >> >>> this
> > > >> >> >>> > > soon)
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 3) Micah
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - FastLanes : The current
> > spec
> > > >> does
> > > >> >> >>> allow
> > > >> >> >>> > > for
> > > >> >> >>> > > > > > using
> > > >> >> >>> > > > > > >>> > >> FastLane
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> with the
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> configurable enum value
> for
> > > >> >> layout. We
> > > >> >> >>> > > should
> > > >> >> >>> > > > be
> > > >> >> >>> > > > > > >>> able to
> > > >> >> >>> > > > > > >>> > >> > inject
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> any
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> layout
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> in the current design.
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Working on resolving all
> > remaining
> > > >> open
> > > >> >> >>> > comments
> > > >> >> >>> > > > on
> > > >> >> >>> > > > > > the
> > > >> >> >>> > > > > > >>> spec
> > > >> >> >>> > > > > > >>> > >> this
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> week.
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Best
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Prateek
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at
> 3:37 AM
> > > >> Steve
> > > >> >> >>> > Loughran <
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> stevel@cloudera.com>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> wrote:
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12,
> > > Micah
> > > >> >> >>> Kornfield
> > > >> >> >>> > <
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> emkornfield@gmail.com>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> wrote:
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> It looks like the actual
> issue
> > > >> >> >>> described for
> > > >> >> >>> > > ORC
> > > >> >> >>> > > > > in
> > > >> >> >>> > > > > > >>> the
> > > >> >> >>> > > > > > >>> > >> paper
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> is that
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> it
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> has multiple sub-encodings
> in
> > a
> > > >> >> batch.
> > > >> >> >>> This
> > > >> >> >>> > > is
> > > >> >> >>> > > > > > >>> different
> > > >> >> >>> > > > > > >>> > >> then
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> the
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> design
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> proposed here where there is
> > > still
> > > >> >> fixed
> > > >> >> >>> > > > encoding
> > > >> >> >>> > > > > > per
> > > >> >> >>> > > > > > >>> page
> > > >> >> >>> > > > > > >>> > >> in
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> parquet.
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> Given reasonably sized
> pages I
> > > >> don't
> > > >> >> >>> think
> > > >> >> >>> > > > branch
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> misprediction should
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> be a
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> big issue for new encodings.
> > I
> > > >> agree
> > > >> >> >>> that
> > > >> >> >>> > we
> > > >> >> >>> > > > > should
> > > >> >> >>> > > > > > >>> be
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> conservative in
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> general for adding new
> > > encodings.
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> +1
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >
> > > >> >> >>> > > > > > >>> > >> >
> > > >> >> >>> > > > > > >>> > >> >
> > > >> >> >>> > > > > > >>> > >> >
> > > >> >> >>> > > > > > >>> > >>
> > > >> >> >>> > > > > > >>> > >
> > > >> >> >>> > > > > > >>> >
> > > >> >> >>> > > > > > >>>
> > > >> >> >>> > > > > > >>
> > > >> >> >>> > > > > >
> > > >> >> >>> > > > >
> > > >> >> >>> > > >
> > > >> >> >>> > >
> > > >> >> >>> >
> > > >> >> >>>
> > > >> >> >>
> > > >> >>
> > > >> >
> > > >>
> > > >
> > >
> >
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Vinoo Ganesh <vi...@gmail.com>.
Thanks all - I'm addressing Gang's comments on
https://github.com/apache/parquet-java/pull/3397 now, so hopefully well
also be in a good state for merging the java implementation soon.
Thanks,
Vinoo Ganesh | vinoo.ganesh@gmail.com
<Vi...@gmail.com>
On Sat, Sep 5, 2026 at 8:56 AM Andrew Lamb <an...@gmail.com> wrote:
> We just merged the Rust implementation[1] from Kosta and Devan. With the
> imminent release of Parquet Format 2.14.0 I think we'll be well on our way.
>
> Thanks again for pushing this along,
> Andrew
>
> [1]: https://github.com/apache/arrow-rs/pull/9372
>
> On Fri, Sep 4, 2026 at 5:42 PM PRATEEK GAUR <pr...@gmail.com> wrote:
>
> > Hi team,
> >
> > Thanks Micah for multiple rounds of review and thanks Gang for taking a
> > look at the PR and giving feedback. I've addressed all comments in the
> PR.
> > Looking forward to the next round of feedback (or approval). Would like
> to
> > get it in soon.
> >
> > https://github.com/apache/arrow/pull/48345/changes
> >
> > With this done and after talking to Arnav, I plan to spend some time on
> the
> > ALP implementation in the arrow-go repository helping spread the support
> to
> > more languages.
> >
> > Best
> > Prateek
> >
> > On Sun, Aug 2, 2026 at 10:42 PM PRATEEK GAUR <pr...@gmail.com> wrote:
> >
> > > Thanks Divjot for bringing it up.
> > >
> > > I was under the same impression as Russell and Andrew that we'd have it
> > > flag gated.
> > > With the new versioning proposal 'preview' might be a good tag for it.
> > >
> > > Best
> > > Prateek
> > >
> > > On Sat, Aug 1, 2026 at 12:39 AM Andrew Lamb <an...@gmail.com>
> > > wrote:
> > >
> > >> Div: Here is a proposal to mark ALP in Preview: [1] (using a comment
> in
> > >> the
> > >> spec section).
> > >>
> > >> Russel: That is my understanding as well.
> > >>
> > >> Hopefully we are able to formalize the versioning process shortly so
> we
> > >> can
> > >> formalize the notion of "preview"
> > >>
> > >> Andrew
> > >>
> > >>
> > >> [1]: https://github.com/apache/parquet-format/pull/604
> > >>
> > >> On Fri, Jul 31, 2026 at 11:15 AM Russell Spitzer <
> > >> russell.spitzer@gmail.com>
> > >> wrote:
> > >>
> > >> > I thought we had agreed it would be merged, and features like this
> > would
> > >> > be feature flag gated. So a V-Old writer could always opt in to
> using
> > >> this
> > >> > encoding but that must be a conscious effort if you are not using
> > VNext
> > >> of
> > >> > Parquet. If you are on V-Next the writer can choose to use ALP
> > whenever
> > >> it
> > >> > likes without user intervention.
> > >> >
> > >> > On Fri, Jul 31, 2026 at 2:20 AM Divjot Arora via dev <
> > >> > dev@parquet.apache.org> wrote:
> > >> >
> > >> >> Yes, that’s my understanding as well. However, as you noted, the
> > >> >> versioning
> > >> >> discussion is still in early stages and hasn’t prescribed concrete
> > >> >> mechanisms yet. If ALP goes into parquet-format 2.14.0, there’s no
> > >> >> indication that it’s in preview or that writers shouldn’t use it by
> > >> >> default
> > >> >> yet. Given that we’ve held off on merging other incompatible
> changes
> > >> (e.g
> > >> >> removal of path_in_schema) until the versioning discussion closes,
> my
> > >> >> preference would be to leave ALP un-merged for the time being as
> > well.
> > >> >>
> > >> >> — Div
> > >> >>
> > >> >> On Fri, Jul 31, 2026 at 08:59 Andrew Lamb <an...@gmail.com>
> > >> wrote:
> > >> >>
> > >> >> > After re-reading Julien's Versioning Proposal, my understanding
> is
> > >> that
> > >> >> if
> > >> >> > we adopt that system, we will somehow mark ALP in "Preview" in
> > >> >> > parquet-format for release Version 3.
> > >> >> >
> > >> >> > The versioning document doesn't seem to proscribe the particular
> > >> method
> > >> >> > for annotating "preview" features -- I think clearly marking them
> > as
> > >> >> such
> > >> >> > in parquet.thrift as well as reflecting that on the
> > >> >> > https://parquet.apache.org/docs/file-format/versions/ page
> somehow
> > >> >> would
> > >> >> > be adequate.
> > >> >> >
> > >> >> > Andrew
> > >> >> >
> > >> >> > On Thu, Jul 30, 2026 at 11:50 AM Andrew Lamb <
> > andrewlamb11@gmail.com
> > >> >
> > >> >> > wrote:
> > >> >> >
> > >> >> >> In my mind ALP is in the same position as other encodings that
> > were
> > >> >> added
> > >> >> >> recently[1] (such as BYTE_STREAM_SPLIT for non floating point
> > types)
> > >> >> >>
> > >> >> >> That is to say it will be available in the latest open source
> > >> writers
> > >> >> and
> > >> >> >> readers, but realistically not widely used in the ecosystem
> where
> > >> >> >> compatibility is a concern (a large part)
> > >> >> >>
> > >> >> >> I view Julian's work[2][3] to gather consensus on how to name
> > >> >> >> combinations of features (e.g. "versions") as the mechanism by
> > >> which we
> > >> >> >> will accelerate the adoption of this change (along with other
> > >> forward
> > >> >> >> incompatible changes) across the ecosystem.
> > >> >> >>
> > >> >> >> > Will we exclude this change when cutting parquet-format
> v2.14.0?
> > >> >> >>
> > >> >> >> No I don't think so -- I would expect it to go out in the next
> > >> >> >> parquet-format version as do other changes in the spec.
> > >> >> >>
> > >> >> >> Andrew
> > >> >> >>
> > >> >> >> [1]: https://parquet.apache.org/docs/file-format/versions/
> > >> >> >> [2]:
> > >> https://lists.apache.org/thread/xmnj8h0h8ozmrgox5tydhhs7yz7q2hss
> > >> >> >> [3]:
> > >> https://lists.apache.org/thread/qlf8lg90gqq41lllyy9mk2f0skqvftqj
> > >> >> >>
> > >> >> >>
> > >> >> >> On Thu, Jul 30, 2026 at 3:51 AM Divjot Arora via dev <
> > >> >> >> dev@parquet.apache.org> wrote:
> > >> >> >>
> > >> >> >>> Hi Andrew,
> > >> >> >>>
> > >> >> >>> How does this interact with the ongoing versioning discussion?
> > >> Given
> > >> >> that
> > >> >> >>> new encodings are not forward compatible, I would assume ALP is
> > >> >> included
> > >> >> >>> in
> > >> >> >>> Parquet V<next> (V3?), but it's not clear to me how that works
> if
> > >> it's
> > >> >> >>> merged to the parquet-format master branch now. Will we exclude
> > >> this
> > >> >> >>> change
> > >> >> >>> when cutting parquet-format v2.14.0?
> > >> >> >>>
> > >> >> >>> -- Div
> > >> >> >>>
> > >> >> >>> On Thu, Jul 30, 2026 at 8:51 AM Andrew Lamb <
> > >> andrewlamb11@gmail.com>
> > >> >> >>> wrote:
> > >> >> >>>
> > >> >> >>> > Hello,
> > >> >> >>> >
> > >> >> >>> > As an update, thanks to the epic work from many contributors,
> > >> >> >>> especially
> > >> >> >>> > Prateek, I just merged the parquet-format change for ALP[1].
> We
> > >> >> plan to
> > >> >> >>> > make additional editorial The remaining items I know of are
> > >> merging
> > >> >> the
> > >> >> >>> > examples from parquet-testing[2], and then of course the
> > various
> > >> >> >>> > implementations
> > >> >> >>> >
> > >> >> >>> > Thanks again everyone. This is a great step forward
> > >> >> >>> > Andrew
> > >> >> >>> >
> > >> >> >>> > [1]: https://github.com/apache/parquet-format/pull/557
> > >> >> >>> > [2]: https://github.com/apache/parquet-testing/pull/100
> > >> >> >>> >
> > >> >> >>> > On Tue, Jul 21, 2026 at 4:39 PM Julien Le Dem <
> > julien@apache.org
> > >> >
> > >> >> >>> wrote:
> > >> >> >>> >
> > >> >> >>> > > It was also in my spam folder... :(
> > >> >> >>> > > I've seen a bunch of apache list emails go to my spam
> > recently.
> > >> >> >>> > >
> > >> >> >>> > > On Tue, Jul 21, 2026 at 10:58 AM Russell Spitzer <
> > >> >> >>> > > russell.spitzer@gmail.com>
> > >> >> >>> > > wrote:
> > >> >> >>> > >
> > >> >> >>> > > > Yes, my hope was that my vote would kick it out of spam
> but
> > >> we
> > >> >> >>> > definitely
> > >> >> >>> > > > had two other votes that also went to spam for me.
> > >> >> >>> > > >
> > >> >> >>> > > > For anyone looking the thread is here
> > >> >> >>> > > >
> > >> >> https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt
> > >> >> >>> > > >
> > >> >> >>> > > > On Tue, Jul 21, 2026 at 10:57 AM Micah Kornfield <
> > >> >> >>> > emkornfield@gmail.com>
> > >> >> >>> > > > wrote:
> > >> >> >>> > > >
> > >> >> >>> > > > > I've seen other's votes not go to spam, so hopefully
> > people
> > >> >> will
> > >> >> >>> see
> > >> >> >>> > it
> > >> >> >>> > > > in
> > >> >> >>> > > > > there inboxes (or see this thread and look in there
> > spam).
> > >> >> >>> > > > >
> > >> >> >>> > > > > On Tue, Jul 21, 2026 at 10:52 AM PRATEEK GAUR <
> > >> >> >>> prtkgaur@gmail.com>
> > >> >> >>> > > > wrote:
> > >> >> >>> > > > >
> > >> >> >>> > > > > > Okay,
> > >> >> >>> > > > > >
> > >> >> >>> > > > > > I sent out the voting email yesterday and so far I've
> > >> heard
> > >> >> >>> from 3
> > >> >> >>> > > > people
> > >> >> >>> > > > > > that it ended up in their "spam folder". Dev VPN to
> be
> > >> >> blamed
> > >> >> >>> for
> > >> >> >>> > > it?.
> > >> >> >>> > > > > >
> > >> >> >>> > > > > > What's the protocol here?
> > >> >> >>> > > > > > I let people hunt for the email in their spam folder
> or
> > >> send
> > >> >> >>> out
> > >> >> >>> > > > another
> > >> >> >>> > > > > > one?
> > >> >> >>> > > > > >
> > >> >> >>> > > > > > Best
> > >> >> >>> > > > > > Prateek
> > >> >> >>> > > > > >
> > >> >> >>> > > > > > On Sun, Jul 19, 2026 at 10:47 PM PRATEEK GAUR <
> > >> >> >>> prtkgaur@gmail.com>
> > >> >> >>> > > > > wrote:
> > >> >> >>> > > > > >
> > >> >> >>> > > > > > > Hi team,
> > >> >> >>> > > > > > >
> > >> >> >>> > > > > > > Vinoo has added the tests which were requested in
> the
> > >> last
> > >> >> >>> round
> > >> >> >>> > of
> > >> >> >>> > > > > > review.
> > >> >> >>> > > > > > > In his own words.
> > >> >> >>> > > > > > >
> > >> >> >>> > > > > > > "
> > >> >> >>> > > > > > > The extreme value tests Micah wanted are done and
> > >> pushed.
> > >> >> I
> > >> >> >>> added
> > >> >> >>> > > > > > coverage
> > >> >> >>> > > > > > > for values that need the full FOR bit width after
> the
> > >> >> frame
> > >> >> >>> of
> > >> >> >>> > > > > reference
> > >> >> >>> > > > > > is
> > >> >> >>> > > > > > > applied: 64-bit (the case where the signed max
> minus
> > >> min
> > >> >> >>> > > subtraction
> > >> >> >>> > > > > > > overflows), 63-bit (the non-overflow case), and
> > 32-bit
> > >> for
> > >> >> >>> > floats.
> > >> >> >>> > > > All
> > >> >> >>> > > > > > > lossless with zero exceptions. I also verified them
> > >> >> through
> > >> >> >>> Arrow
> > >> >> >>> > > C++
> > >> >> >>> > > > > ALP
> > >> >> >>> > > > > > > reader (the local build), so the Java-written
> extreme
> > >> >> columns
> > >> >> >>> > > decode
> > >> >> >>> > > > > > > correctly in C++, bit-exact against the expected
> > >> values."
> > >> >> >>> > > > > > >
> > >> >> >>> > > > > > > State of the PR (#3397):
> > >> >> >>> > > > > > >
> > >> >> >>> > > > > > > - Merges cleanly with master (I pulled master in
> > and
> > >> >> >>> resolved
> > >> >> >>> > > the
> > >> >> >>> > > > > > > conflicts).
> > >> >> >>> > > > > > > - 41 of 44 review threads resolved. The 3
> > remaining
> > >> are
> > >> >> >>> > > > intentional:
> > >> >> >>> > > > > > > one deferred perf nit, one that stays open until
> > >> >> >>> > parquet-format
> > >> >> >>> > > > > ships
> > >> >> >>> > > > > > ALP,
> > >> >> >>> > > > > > >
> > >> >> >>> > > > > > > "
> > >> >> >>> > > > > > >
> > >> >> >>> > > > > > > With this both the
> > >> >> >>> > > > > > > 1) C++ PR :
> > >> >> >>> https://github.com/apache/arrow/pull/48345/changes
> > >> >> >>> > > > > > > 2) Java PR :
> > >> >> >>> https://github.com/apache/parquet-java/pull/3397
> > >> >> >>> > > > > > >
> > >> >> >>> > > > > > > Are cross language tested and range of test
> > scenarios.
> > >> >> >>> > > > > > >
> > >> >> >>> > > > > > > Best
> > >> >> >>> > > > > > > Prateek
> > >> >> >>> > > > > > >
> > >> >> >>> > > > > > >
> > >> >> >>> > > > > > > On Mon, Jul 13, 2026 at 8:38 AM PRATEEK GAUR <
> > >> >> >>> prtkgaur@gmail.com
> > >> >> >>> > >
> > >> >> >>> > > > > wrote:
> > >> >> >>> > > > > > >
> > >> >> >>> > > > > > >> Thanks Matt,
> > >> >> >>> > > > > > >>
> > >> >> >>> > > > > > >> I'll be on it this week.
> > >> >> >>> > > > > > >>
> > >> >> >>> > > > > > >> Best
> > >> >> >>> > > > > > >> Prateek
> > >> >> >>> > > > > > >>
> > >> >> >>> > > > > > >> On Mon, Jul 13, 2026 at 8:36 AM Matt Topol <
> > >> >> >>> > > zotthewizard@gmail.com>
> > >> >> >>> > > > > > >> wrote:
> > >> >> >>> > > > > > >>
> > >> >> >>> > > > > > >>> There is also the Go implementation submitted by
> > >> Arnav (
> > >> >> >>> > > > > > >>> https://github.com/apache/arrow-go/pull/704)
> which
> > >> is
> > >> >> >>> waiting
> > >> >> >>> > > for
> > >> >> >>> > > > > > >>> updates.
> > >> >> >>> > > > > > >>>
> > >> >> >>> > > > > > >>> Just wanted to make sure it didn't get lost here.
> > >> >> >>> > > > > > >>>
> > >> >> >>> > > > > > >>> --Matt
> > >> >> >>> > > > > > >>>
> > >> >> >>> > > > > > >>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK GAUR <
> > >> >> >>> > > prtkgaur@gmail.com>
> > >> >> >>> > > > > > >>> wrote:
> > >> >> >>> > > > > > >>>
> > >> >> >>> > > > > > >>> > Hi all,
> > >> >> >>> > > > > > >>> >
> > >> >> >>> > > > > > >>> > I'd like to share a status update on the ALP
> > >> encoding
> > >> >> >>> effort
> > >> >> >>> > > and
> > >> >> >>> > > > > get
> > >> >> >>> > > > > > a
> > >> >> >>> > > > > > >>> feel
> > >> >> >>> > > > > > >>> > from the community before starting a formal
> vote.
> > >> >> >>> > > > > > >>> >
> > >> >> >>> > > > > > >>> > We now have the specification plus
> > implementations
> > >> in
> > >> >> >>> both
> > >> >> >>> > > > > languages:
> > >> >> >>> > > > > > >>> >
> > >> >> >>> > > > > > >>> > - Spec: apache/parquet-format#557 —
> > >> GH-533
> > >> >> >>> Add
> > >> >> >>> > ALP
> > >> >> >>> > > > > > encoding
> > >> >> >>> > > > > > >>> > specification
> > >> >> >>> > > > > > >>> > - C++ (Arrow): apache/arrow#48345 —
> GH-48701
> > >> >> >>> > > [C++][Parquet]
> > >> >> >>> > > > > Add
> > >> >> >>> > > > > > >>> ALPpd
> > >> >> >>> > > > > > >>> > encoding
> > >> >> >>> > > > > > >>> > - Java: apache/parquet-java#3397 —
> > >> Parquet
> > >> >> >>> Java
> > >> >> >>> > ALP
> > >> >> >>> > > > > > >>> > Implementation (by Vinoo)
> > >> >> >>> > > > > > >>> >
> > >> >> >>> > > > > > >>> > Current state:
> > >> >> >>> > > > > > >>> >
> > >> >> >>> > > > > > >>> > - The C++ implementation has been through
> 3-4
> > >> >> rounds
> > >> >> >>> of
> > >> >> >>> > > > review.
> > >> >> >>> > > > > > >>> > - The Java implementation has been tracking
> > >> well
> > >> >> and
> > >> >> >>> has
> > >> >> >>> > > had
> > >> >> >>> > > > > > >>> initial
> > >> >> >>> > > > > > >>> > reviews from contributors, with no major
> > >> >> remaining
> > >> >> >>> > > issues.
> > >> >> >>> > > > > > >>> > - Cross-language compatibility tests are
> > >> passing:
> > >> >> the
> > >> >> >>> > Arrow
> > >> >> >>> > > > C++
> > >> >> >>> > > > > > >>> decoder
> > >> >> >>> > > > > > >>> > reads Java-written data bit-exactly
> across
> > >> >> ~1.56M
> > >> >> >>> > values
> > >> >> >>> > > > and
> > >> >> >>> > > > > 18
> > >> >> >>> > > > > > >>> > fixtures, covering V1 and V2 pages,
> > multiple
> > >> >> vector
> > >> >> >>> > > sizes,
> > >> >> >>> > > > > and
> > >> >> >>> > > > > > >>> > several
> > >> >> >>> > > > > > >>> > real datasets — zero mismatches.
> > >> >> >>> > > > > > >>> >
> > >> >> >>> > > > > > >>> > As raised in the java-pr we will be adding
> > >> coverage
> > >> >> for
> > >> >> >>> > > extreme
> > >> >> >>> > > > > > values
> > >> >> >>> > > > > > >>> > (those requiring
> > >> >> >>> > > > > > >>> > 63-64 bits after FOR is applied) before we
> > close a
> > >> >> vote.
> > >> >> >>> > We'll
> > >> >> >>> > > > aim
> > >> >> >>> > > > > > to
> > >> >> >>> > > > > > >>> get
> > >> >> >>> > > > > > >>> > that
> > >> >> >>> > > > > > >>> > done in parallel.
> > >> >> >>> > > > > > >>> >
> > >> >> >>> > > > > > >>> > Plan: unless there are objections, I intend to
> > >> start a
> > >> >> >>> vote
> > >> >> >>> > at
> > >> >> >>> > > > the
> > >> >> >>> > > > > > end
> > >> >> >>> > > > > > >>> of
> > >> >> >>> > > > > > >>> > this week/early
> > >> >> >>> > > > > > >>> > next week.
> > >> >> >>> > > > > > >>> >
> > >> >> >>> > > > > > >>> > Thanks
> > >> >> >>> > > > > > >>> > Prateek
> > >> >> >>> > > > > > >>> >
> > >> >> >>> > > > > > >>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <
> > >> >> >>> > > prtkgaur@gmail.com>
> > >> >> >>> > > > > > >>> wrote:
> > >> >> >>> > > > > > >>> >
> > >> >> >>> > > > > > >>> > > Hi Team,
> > >> >> >>> > > > > > >>> > >
> > >> >> >>> > > > > > >>> > > Just wanted to provide some updates on ALP.
> > >> >> >>> > > > > > >>> > > Micah and I have done 3-4 rounds of review
> for
> > >> the
> > >> >> c++
> > >> >> >>> PR :
> > >> >> >>> > > > > > >>> > > https://github.com/apache/arrow/pull/48345
> > >> >> >>> > > > > > >>> > >
> > >> >> >>> > > > > > >>> > > For the Java implementation Vinoo has been
> > >> working
> > >> >> on
> > >> >> >>> > > following
> > >> >> >>> > > > > PR
> > >> >> >>> > > > > > :
> > >> >> >>> > > > > > >>> > >
> > https://github.com/apache/parquet-java/pull/3397
> > >> >> >>> > > > > > >>> > >
> > >> >> >>> > > > > > >>> > > Best
> > >> >> >>> > > > > > >>> > > Prateek
> > >> >> >>> > > > > > >>> > >
> > >> >> >>> > > > > > >>> > > On Tue, May 5, 2026 at 1:50 PM Micah
> Kornfield
> > <
> > >> >> >>> > > > > > >>> emkornfield@gmail.com>
> > >> >> >>> > > > > > >>> > > wrote:
> > >> >> >>> > > > > > >>> > >
> > >> >> >>> > > > > > >>> > >> Hi Antoine,
> > >> >> >>> > > > > > >>> > >>
> > >> >> >>> > > > > > >>> > >> > Apologies if the question was already
> asked,
> > >> but
> > >> >> >>> should
> > >> >> >>> > we
> > >> >> >>> > > > > care
> > >> >> >>> > > > > > >>> about
> > >> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more
> > >> >> efficient
> > >> >> >>> than
> > >> >> >>> > > > > FLOAT16
> > >> >> >>> > > > > > +
> > >> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> > >> >> >>> > > > > > >>> > >>
> > >> >> >>> > > > > > >>> > >> It was. We thought we could defer it for
> the
> > >> >> >>> following
> > >> >> >>> > > > reasons:
> > >> >> >>> > > > > > >>> > >> 1. It's not clear there are a lot of easy
> > >> >> reference
> > >> >> >>> > > datasets
> > >> >> >>> > > > to
> > >> >> >>> > > > > > >>> test
> > >> >> >>> > > > > > >>> > its
> > >> >> >>> > > > > > >>> > >> effectiveness. It does look like there
> might
> > be
> > >> >> one
> > >> >> >>> or
> > >> >> >>> > two
> > >> >> >>> > > on
> > >> >> >>> > > > > > >>> > huggingface
> > >> >> >>> > > > > > >>> > >> (e.g.
> > >> >> https://huggingface.co/datasets/kikitora/curdie
> > >> >> >>> ).
> > >> >> >>> > > > > > >>> > >> 2. It seemed likely that float 16 was more
> > >> likely
> > >> >> >>> used
> > >> >> >>> > for
> > >> >> >>> > > > > values
> > >> >> >>> > > > > > >>> that
> > >> >> >>> > > > > > >>> > >> were less likely to reduce to decimal
> values.
> > >> >> >>> > > > > > >>> > >> 3. It could be added as an extension later
> if
> > >> >> needed.
> > >> >> >>> > > > > > >>> > >>
> > >> >> >>> > > > > > >>> > >> Cheers,
> > >> >> >>> > > > > > >>> > >> Micah
> > >> >> >>> > > > > > >>> > >>
> > >> >> >>> > > > > > >>> > >>
> > >> >> >>> > > > > > >>> > >> On Tue, May 5, 2026 at 1:40 PM Antoine
> Pitrou
> > <
> > >> >> >>> > > > > antoine@python.org
> > >> >> >>> > > > > > >
> > >> >> >>> > > > > > >>> > wrote:
> > >> >> >>> > > > > > >>> > >>
> > >> >> >>> > > > > > >>> > >> >
> > >> >> >>> > > > > > >>> > >> > Hello,
> > >> >> >>> > > > > > >>> > >> >
> > >> >> >>> > > > > > >>> > >> > Apologies if the question was already
> asked,
> > >> but
> > >> >> >>> should
> > >> >> >>> > we
> > >> >> >>> > > > > care
> > >> >> >>> > > > > > >>> about
> > >> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more
> > >> >> efficient
> > >> >> >>> than
> > >> >> >>> > > > > FLOAT16
> > >> >> >>> > > > > > +
> > >> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> > >> >> >>> > > > > > >>> > >> >
> > >> >> >>> > > > > > >>> > >> > Regards
> > >> >> >>> > > > > > >>> > >> >
> > >> >> >>> > > > > > >>> > >> > Antoine.
> > >> >> >>> > > > > > >>> > >> >
> > >> >> >>> > > > > > >>> > >> >
> > >> >> >>> > > > > > >>> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a
> écrit
> > :
> > >> >> >>> > > > > > >>> > >> > > Thanks Andrew and Micah for review
> > feedback
> > >> on
> > >> >> >>> the two
> > >> >> >>> > > > PR's
> > >> >> >>> > > > > > >>> > >> > > 1) (c++ arrow repo)
> > >> >> >>> > > > > > >>> > >>
> > >> https://github.com/apache/arrow/pull/48345/changes
> > >> >> >>> > > > > > >>> > >> > > 2) (parquet-format repo)
> > >> >> >>> > > > > > >>> > >> >
> > >> >> https://github.com/apache/parquet-format/pull/557
> > >> >> >>> > > > > > >>> > >> > >
> > >> >> >>> > > > > > >>> > >> > > I have addressed all (unless I missed
> > >> >> something)
> > >> >> >>> > > comments
> > >> >> >>> > > > on
> > >> >> >>> > > > > > >>> the two
> > >> >> >>> > > > > > >>> > >> > PR's.
> > >> >> >>> > > > > > >>> > >> > >
> > >> >> >>> > > > > > >>> > >> > > Best
> > >> >> >>> > > > > > >>> > >> > > Prateek
> > >> >> >>> > > > > > >>> > >> > >
> > >> >> >>> > > > > > >>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK
> > >> GAUR <
> > >> >> >>> > > > > > >>> prtkgaur@gmail.com>
> > >> >> >>> > > > > > >>> > >> wrote:
> > >> >> >>> > > > > > >>> > >> > >
> > >> >> >>> > > > > > >>> > >> > >> Thanks Andrew and Micah.
> > >> >> >>> > > > > > >>> > >> > >>
> > >> >> >>> > > > > > >>> > >> > >> `fair amount of feedback on at least
> the
> > >> >> >>> > > implementations`
> > >> >> >>> > > > > > >>> > >> > >> For the c++ I have already started
> > >> addressing
> > >> >> the
> > >> >> >>> > > > > feedback, I
> > >> >> >>> > > > > > >>> > should
> > >> >> >>> > > > > > >>> > >> be
> > >> >> >>> > > > > > >>> > >> > >> done with that Monday/Tuesday.
> > >> >> >>> > > > > > >>> > >> > >> I think Vinoo too has been making good
> > >> >> progress
> > >> >> >>> on
> > >> >> >>> > the
> > >> >> >>> > > > Java
> > >> >> >>> > > > > > >>> > >> > implementation.
> > >> >> >>> > > > > > >>> > >> > >>
> > >> >> >>> > > > > > >>> > >> > >> Best
> > >> >> >>> > > > > > >>> > >> > >> Prateek
> > >> >> >>> > > > > > >>> > >> > >>
> > >> >> >>> > > > > > >>> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew
> > >> Lamb <
> > >> >> >>> > > > > > >>> > andrewlamb11@gmail.com
> > >> >> >>> > > > > > >>> > >> >
> > >> >> >>> > > > > > >>> > >> > >> wrote:
> > >> >> >>> > > > > > >>> > >> > >>
> > >> >> >>> > > > > > >>> > >> > >>> Got it. Thank you for the
> clarification
> > >> -- I
> > >> >> >>> will
> > >> >> >>> > try
> > >> >> >>> > > > and
> > >> >> >>> > > > > > look
> > >> >> >>> > > > > > >>> > into
> > >> >> >>> > > > > > >>> > >> the
> > >> >> >>> > > > > > >>> > >> > >>> spec and the Rust implementation[1] in
> > >> this
> > >> >> next
> > >> >> >>> > week
> > >> >> >>> > > > > > >>> > >> > >>>
> > >> >> >>> > > > > > >>> > >> > >>> [1]:
> > >> >> >>> https://github.com/apache/arrow-rs/pull/9372
> > >> >> >>> > > > > > >>> > >> > >>>
> > >> >> >>> > > > > > >>> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah
> > >> >> >>> Kornfield <
> > >> >> >>> > > > > > >>> > >> > emkornfield@gmail.com>
> > >> >> >>> > > > > > >>> > >> > >>> wrote:
> > >> >> >>> > > > > > >>> > >> > >>>
> > >> >> >>> > > > > > >>> > >> > >>>> Hi Andrew,
> > >> >> >>> > > > > > >>> > >> > >>>> I think there is a fair amount of
> > >> feedback
> > >> >> on
> > >> >> >>> at
> > >> >> >>> > > least
> > >> >> >>> > > > > the
> > >> >> >>> > > > > > >>> > >> > >>>> implementations, typically I think
> > we've
> > >> >> waited
> > >> >> >>> > till
> > >> >> >>> > > > they
> > >> >> >>> > > > > > are
> > >> >> >>> > > > > > >>> > >> close to
> > >> >> >>> > > > > > >>> > >> > >>>> mergeable before a final vote.
> > >> Otherwise I
> > >> >> >>> agree
> > >> >> >>> > we
> > >> >> >>> > > > are
> > >> >> >>> > > > > > very
> > >> >> >>> > > > > > >>> > >> close.
> > >> >> >>> > > > > > >>> > >> > >>>>
> > >> >> >>> > > > > > >>> > >> > >>>> -Micah
> > >> >> >>> > > > > > >>> > >> > >>>>
> > >> >> >>> > > > > > >>> > >> > >>>> On Saturday, April 25, 2026, Andrew
> > Lamb
> > >> <
> > >> >> >>> > > > > > >>> andrewlamb11@gmail.com
> > >> >> >>> > > > > > >>> > >
> > >> >> >>> > > > > > >>> > >> > wrote:
> > >> >> >>> > > > > > >>> > >> > >>>>
> > >> >> >>> > > > > > >>> > >> > >>>>> Thanks Prateek,
> > >> >> >>> > > > > > >>> > >> > >>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>> I think from this content it looks
> to
> > me
> > >> >> like
> > >> >> >>> we
> > >> >> >>> > are
> > >> >> >>> > > > > ready
> > >> >> >>> > > > > > >>> to
> > >> >> >>> > > > > > >>> > >> start a
> > >> >> >>> > > > > > >>> > >> > >>>>> vote to explicitly accept ALP into
> > >> Parquet
> > >> >> >>> > > > > > >>> > >> > >>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>> Does anyone know of a reason we
> should
> > >> >> >>> postpone it
> > >> >> >>> > > for
> > >> >> >>> > > > > > >>> longer?
> > >> >> >>> > > > > > >>> > >> > >>>>> Perhaps someone needs some more time
> > to
> > >> >> >>> review?
> > >> >> >>> > > > > > >>> > >> > >>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>> Andrew
> > >> >> >>> > > > > > >>> > >> > >>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM
> > PRATEEK
> > >> >> GAUR <
> > >> >> >>> > > > > > >>> > prtkgaur@gmail.com>
> > >> >> >>> > > > > > >>> > >> > >>>>> wrote:
> > >> >> >>> > > > > > >>> > >> > >>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>> Hi team,
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>> Hope everyone is doing well. I got
> a
> > >> >> chance
> > >> >> >>> to
> > >> >> >>> > work
> > >> >> >>> > > > > > >>> through all
> > >> >> >>> > > > > > >>> > >> the
> > >> >> >>> > > > > > >>> > >> > >>>>>> remaining feedback and update the
> > spec
> > >> >> doc.
> > >> >> >>> Here
> > >> >> >>> > > are
> > >> >> >>> > > > > the
> > >> >> >>> > > > > > >>> new
> > >> >> >>> > > > > > >>> > >> > artifacts
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>> 1) Spec document :
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> >
> > >> >> >>> > > > > > >>> > >>
> > >> >> >>> > > > > > >>> >
> > >> >> >>> > > > > > >>>
> > >> >> >>> > > > > >
> > >> >> >>> > > > >
> > >> >> >>> > > >
> > >> >> >>> > >
> > >> >> >>> >
> > >> >> >>>
> > >> >>
> > >>
> >
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>> 2) Spec document in parquet format
> > >> repo :
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > https://github.com/apache/parquet-format/pull/557
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>> 3) Alp implementation in arrow c++
> > >> repo :
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > https://github.com/apache/arrow/pull/48345/changes
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>> 4) Alp implementation in
> parquet-java
> > >> >> repo :
> > >> >> >>> Work
> > >> >> >>> > > for
> > >> >> >>> > > > > > >>> Vinoo and
> > >> >> >>> > > > > > >>> > >> > Julien
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > https://github.com/apache/parquet-java/pull/3397
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>> 5) PR with test and benchmarking
> > >> >> artifacts in
> > >> >> >>> > > > > > >>> parquet-testing
> > >> >> >>> > > > > > >>> > >> repo :
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > https://github.com/apache/parquet-testing/pull/100
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>> And
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>> - Go : Arnav just submitted an
> in
> > >> >> >>> progress
> > >> >> >>> > > > > > >>> implementation
> > >> >> >>> > > > > > >>> > in
> > >> >> >>> > > > > > >>> > >> Go.
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> https://github.com/apache/arrow-go/pull/704
> > >> >> >>> > (I
> > >> >> >>> > > > > > haven't
> > >> >> >>> > > > > > >>> > >> started
> > >> >> >>> > > > > > >>> > >> > >>>>>> looking at it yet)
> > >> >> >>> > > > > > >>> > >> > >>>>>> - Rust : I remember Andrew
> > >> mentioned
> > >> >> that
> > >> >> >>> > this
> > >> >> >>> > > > work
> > >> >> >>> > > > > > is
> > >> >> >>> > > > > > >>> also
> > >> >> >>> > > > > > >>> > >> in
> > >> >> >>> > > > > > >>> > >> > >>>>>> progress (So 4 languages!)
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>> *Arrow C++ implementation *
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>> The PR is out and was also used by
> > >> >> Antoine to
> > >> >> >>> > > report
> > >> >> >>> > > > > the
> > >> >> >>> > > > > > >>> > numbers
> > >> >> >>> > > > > > >>> > >> as
> > >> >> >>> > > > > > >>> > >> > >>>>>> reported here. Micah and Konstantin
> > >> have
> > >> >> >>> given 1
> > >> >> >>> > > > round
> > >> >> >>> > > > > of
> > >> >> >>> > > > > > >>> > >> feedback
> > >> >> >>> > > > > > >>> > >> > >>>>>> and I'm addressing them today.
> Please
> > >> note
> > >> >> >>> that
> > >> >> >>> > the
> > >> >> >>> > > > > > default
> > >> >> >>> > > > > > >>> > >> > >>>>>> optimization flag for compiling is
> O2
> > >> and
> > >> >> not
> > >> >> >>> > Q3. I
> > >> >> >>> > > > got
> > >> >> >>> > > > > > >>> around
> > >> >> >>> > > > > > >>> > >> 70%
> > >> >> >>> > > > > > >>> > >> > >>>>>> performance improvement in the
> > decoding
> > >> >> speed
> > >> >> >>> > when
> > >> >> >>> > > > > using
> > >> >> >>> > > > > > >>> the O3
> > >> >> >>> > > > > > >>> > >> > flag.
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>> *Parqet-MR Java implementation
> > (working
> > >> >> with
> > >> >> >>> > Vinoo
> > >> >> >>> > > > and
> > >> >> >>> > > > > > >>> Julien)
> > >> >> >>> > > > > > >>> > >> and
> > >> >> >>> > > > > > >>> > >> > **Cross
> > >> >> >>> > > > > > >>> > >> > >>>>>> Language testing*
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>> Let me know if you have any
> > >> questions
> > >> >> or
> > >> >> >>> > > > feedback.
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>> Now pasting some performance
> numbers
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>> Table 1: C++ ALP Double Decode —
> > >> >> Spotify
> > >> >> >>> > Columns
> > >> >> >>> > > > > > >>> (Graviton
> > >> >> >>> > > > > > >>> > 3,
> > >> >> >>> > > > > > >>> > >> ARM
> > >> >> >>> > > > > > >>> > >> > >>>>>> Neoverse V1)
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>>
> > >> >> >>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>> │ Column │ -O2 (MB/s)
> > │
> > >> >> -O3
> > >> >> >>> > > (MB/s) │
> > >> >> >>> > > > > > >>> Speedup │
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>>
> > >> >> >>> ├──────────────────┼──────────────┼──────────────┼─────────┤
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>> │ valence │ 3,155
> > │
> > >> >> >>> 5,523
> > >> >> >>> > > │
> > >> >> >>> > > > > > >>> 1.75x │
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>> │ danceability │ 3,233
> > │
> > >> >> >>> 5,685
> > >> >> >>> > > │
> > >> >> >>> > > > > > >>> 1.76x │
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>> │ energy │ 3,197
> > │
> > >> >> >>> 5,652
> > >> >> >>> > > │
> > >> >> >>> > > > > > >>> 1.77x │
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>> │ loudness │ 3,186
> > │
> > >> >> >>> 5,473
> > >> >> >>> > > │
> > >> >> >>> > > > > > >>> 1.72x │
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>>
> > >> >> >>> └──────────────────┴──────────────┴──────────────┴─────────┘
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM
> > PRATEEK
> > >> >> GAUR
> > >> >> >>> <
> > >> >> >>> > > > > > >>> > prtkgaur@gmail.com
> > >> >> >>> > > > > > >>> > >> >
> > >> >> >>> > > > > > >>> > >> > >>>>>> wrote:
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>> @Micah Kornfield <
> > >> emkornfield@gmail.com>
> > >> >> :
> > >> >> >>> Got
> > >> >> >>> > > it.
> > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>> @Andrew Lamb <
> > andrewlamb11@gmail.com>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> Do you think it would be good to
> > >> start
> > >> >> >>> moving
> > >> >> >>> > the
> > >> >> >>> > > > > spec
> > >> >> >>> > > > > > >>> > >> development
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> into
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> markdown format, in preparation
> for
> > >> >> >>> finalizing
> > >> >> >>> > > it?
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>> Yes I'll update the numbers for
> some
> > >> of
> > >> >> the
> > >> >> >>> > > > examples I
> > >> >> >>> > > > > > >>> have in
> > >> >> >>> > > > > > >>> > >> the
> > >> >> >>> > > > > > >>> > >> > >>>>>>> spec based
> > >> >> >>> > > > > > >>> > >> > >>>>>>> on the updated header size. Then
> we
> > >> >> should
> > >> >> >>> be
> > >> >> >>> > good
> > >> >> >>> > > > to
> > >> >> >>> > > > > go
> > >> >> >>> > > > > > >>> for
> > >> >> >>> > > > > > >>> > the
> > >> >> >>> > > > > > >>> > >> > >>>>>>> markdown format.
> > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>> Thanks everyone!
> > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> Andrew
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM
> > >> PRATEEK
> > >> >> >>> GAUR <
> > >> >> >>> > > > > > >>> > >> prtkgaur@gmail.com>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> wrote:
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Hi team,
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 1) Andrew
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for working on test
> > >> files.
> > >> >> >>> My PR
> > >> >> >>> > > did
> > >> >> >>> > > > > add
> > >> >> >>> > > > > > >>> all
> > >> >> >>> > > > > > >>> > the
> > >> >> >>> > > > > > >>> > >> > test
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> files I
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> used to benchmark on
> datasets.
> > >> >> Maybe
> > >> >> >>> we
> > >> >> >>> > can
> > >> >> >>> > > > club
> > >> >> >>> > > > > > it
> > >> >> >>> > > > > > >>> > >> together.
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> WIll also
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> aid
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> cross language testing
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Kosta Tarasov working on
> > Rust
> > >> >> >>> > > > implementation.
> > >> >> >>> > > > > > >>> This is
> > >> >> >>> > > > > > >>> > >> > great.
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> Thanks
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 2) Antoine
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks a lot for reporting
> > the
> > >> >> >>> numbers
> > >> >> >>> > on
> > >> >> >>> > > > AMD.
> > >> >> >>> > > > > > >>> Looks
> > >> >> >>> > > > > > >>> > >> like
> > >> >> >>> > > > > > >>> > >> > you
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> are
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> getting 8X the decoding
> > >> >> performance of
> > >> >> >>> > BSS.
> > >> >> >>> > > > This
> > >> >> >>> > > > > > is
> > >> >> >>> > > > > > >>> > >> > amazing!!.
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for acknowledging
> the
> > >> >> >>> sampling
> > >> >> >>> > > > design.
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - I agree with you on
> > >> Fastlanes. In
> > >> >> >>> some
> > >> >> >>> > > crude
> > >> >> >>> > > > > > >>> > >> experiments I
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> didn't get
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> a good perf benefit from it
> on
> > >> >> >>> Graviton3
> > >> >> >>> > > (but
> > >> >> >>> > > > > > maybe
> > >> >> >>> > > > > > >>> > there
> > >> >> >>> > > > > > >>> > >> was
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> something
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> wrong with my
> implementation).
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Locking the 16bit
> exception
> > >> >> >>> encoding for
> > >> >> >>> > > the
> > >> >> >>> > > > > > spec
> > >> >> >>> > > > > > >>> in
> > >> >> >>> > > > > > >>> > >> this
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> case.
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Awesome I think we have
> > solved
> > >> >> for
> > >> >> >>> all
> > >> >> >>> > > open
> > >> >> >>> > > > > > >>> questions
> > >> >> >>> > > > > > >>> > >> minus
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> the
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> version byte :). (will get
> > back
> > >> on
> > >> >> >>> this
> > >> >> >>> > > soon)
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 3) Micah
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - FastLanes : The current
> spec
> > >> does
> > >> >> >>> allow
> > >> >> >>> > > for
> > >> >> >>> > > > > > using
> > >> >> >>> > > > > > >>> > >> FastLane
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> with the
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> configurable enum value for
> > >> >> layout. We
> > >> >> >>> > > should
> > >> >> >>> > > > be
> > >> >> >>> > > > > > >>> able to
> > >> >> >>> > > > > > >>> > >> > inject
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> any
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> layout
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> in the current design.
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Working on resolving all
> remaining
> > >> open
> > >> >> >>> > comments
> > >> >> >>> > > > on
> > >> >> >>> > > > > > the
> > >> >> >>> > > > > > >>> spec
> > >> >> >>> > > > > > >>> > >> this
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> week.
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Best
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Prateek
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM
> > >> Steve
> > >> >> >>> > Loughran <
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> stevel@cloudera.com>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> wrote:
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12,
> > Micah
> > >> >> >>> Kornfield
> > >> >> >>> > <
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> emkornfield@gmail.com>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> wrote:
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> It looks like the actual issue
> > >> >> >>> described for
> > >> >> >>> > > ORC
> > >> >> >>> > > > > in
> > >> >> >>> > > > > > >>> the
> > >> >> >>> > > > > > >>> > >> paper
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> is that
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> it
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> has multiple sub-encodings in
> a
> > >> >> batch.
> > >> >> >>> This
> > >> >> >>> > > is
> > >> >> >>> > > > > > >>> different
> > >> >> >>> > > > > > >>> > >> then
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> the
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> design
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> proposed here where there is
> > still
> > >> >> fixed
> > >> >> >>> > > > encoding
> > >> >> >>> > > > > > per
> > >> >> >>> > > > > > >>> page
> > >> >> >>> > > > > > >>> > >> in
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> parquet.
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> Given reasonably sized pages I
> > >> don't
> > >> >> >>> think
> > >> >> >>> > > > branch
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> misprediction should
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> be a
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> big issue for new encodings.
> I
> > >> agree
> > >> >> >>> that
> > >> >> >>> > we
> > >> >> >>> > > > > should
> > >> >> >>> > > > > > >>> be
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> conservative in
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> general for adding new
> > encodings.
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> +1
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > >> >> >>> > > > > > >>> > >> > >
> > >> >> >>> > > > > > >>> > >> >
> > >> >> >>> > > > > > >>> > >> >
> > >> >> >>> > > > > > >>> > >> >
> > >> >> >>> > > > > > >>> > >>
> > >> >> >>> > > > > > >>> > >
> > >> >> >>> > > > > > >>> >
> > >> >> >>> > > > > > >>>
> > >> >> >>> > > > > > >>
> > >> >> >>> > > > > >
> > >> >> >>> > > > >
> > >> >> >>> > > >
> > >> >> >>> > >
> > >> >> >>> >
> > >> >> >>>
> > >> >> >>
> > >> >>
> > >> >
> > >>
> > >
> >
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Andrew Lamb <an...@gmail.com>.
We just merged the Rust implementation[1] from Kosta and Devan. With the
imminent release of Parquet Format 2.14.0 I think we'll be well on our way.
Thanks again for pushing this along,
Andrew
[1]: https://github.com/apache/arrow-rs/pull/9372
On Fri, Sep 4, 2026 at 5:42 PM PRATEEK GAUR <pr...@gmail.com> wrote:
> Hi team,
>
> Thanks Micah for multiple rounds of review and thanks Gang for taking a
> look at the PR and giving feedback. I've addressed all comments in the PR.
> Looking forward to the next round of feedback (or approval). Would like to
> get it in soon.
>
> https://github.com/apache/arrow/pull/48345/changes
>
> With this done and after talking to Arnav, I plan to spend some time on the
> ALP implementation in the arrow-go repository helping spread the support to
> more languages.
>
> Best
> Prateek
>
> On Sun, Aug 2, 2026 at 10:42 PM PRATEEK GAUR <pr...@gmail.com> wrote:
>
> > Thanks Divjot for bringing it up.
> >
> > I was under the same impression as Russell and Andrew that we'd have it
> > flag gated.
> > With the new versioning proposal 'preview' might be a good tag for it.
> >
> > Best
> > Prateek
> >
> > On Sat, Aug 1, 2026 at 12:39 AM Andrew Lamb <an...@gmail.com>
> > wrote:
> >
> >> Div: Here is a proposal to mark ALP in Preview: [1] (using a comment in
> >> the
> >> spec section).
> >>
> >> Russel: That is my understanding as well.
> >>
> >> Hopefully we are able to formalize the versioning process shortly so we
> >> can
> >> formalize the notion of "preview"
> >>
> >> Andrew
> >>
> >>
> >> [1]: https://github.com/apache/parquet-format/pull/604
> >>
> >> On Fri, Jul 31, 2026 at 11:15 AM Russell Spitzer <
> >> russell.spitzer@gmail.com>
> >> wrote:
> >>
> >> > I thought we had agreed it would be merged, and features like this
> would
> >> > be feature flag gated. So a V-Old writer could always opt in to using
> >> this
> >> > encoding but that must be a conscious effort if you are not using
> VNext
> >> of
> >> > Parquet. If you are on V-Next the writer can choose to use ALP
> whenever
> >> it
> >> > likes without user intervention.
> >> >
> >> > On Fri, Jul 31, 2026 at 2:20 AM Divjot Arora via dev <
> >> > dev@parquet.apache.org> wrote:
> >> >
> >> >> Yes, that’s my understanding as well. However, as you noted, the
> >> >> versioning
> >> >> discussion is still in early stages and hasn’t prescribed concrete
> >> >> mechanisms yet. If ALP goes into parquet-format 2.14.0, there’s no
> >> >> indication that it’s in preview or that writers shouldn’t use it by
> >> >> default
> >> >> yet. Given that we’ve held off on merging other incompatible changes
> >> (e.g
> >> >> removal of path_in_schema) until the versioning discussion closes, my
> >> >> preference would be to leave ALP un-merged for the time being as
> well.
> >> >>
> >> >> — Div
> >> >>
> >> >> On Fri, Jul 31, 2026 at 08:59 Andrew Lamb <an...@gmail.com>
> >> wrote:
> >> >>
> >> >> > After re-reading Julien's Versioning Proposal, my understanding is
> >> that
> >> >> if
> >> >> > we adopt that system, we will somehow mark ALP in "Preview" in
> >> >> > parquet-format for release Version 3.
> >> >> >
> >> >> > The versioning document doesn't seem to proscribe the particular
> >> method
> >> >> > for annotating "preview" features -- I think clearly marking them
> as
> >> >> such
> >> >> > in parquet.thrift as well as reflecting that on the
> >> >> > https://parquet.apache.org/docs/file-format/versions/ page somehow
> >> >> would
> >> >> > be adequate.
> >> >> >
> >> >> > Andrew
> >> >> >
> >> >> > On Thu, Jul 30, 2026 at 11:50 AM Andrew Lamb <
> andrewlamb11@gmail.com
> >> >
> >> >> > wrote:
> >> >> >
> >> >> >> In my mind ALP is in the same position as other encodings that
> were
> >> >> added
> >> >> >> recently[1] (such as BYTE_STREAM_SPLIT for non floating point
> types)
> >> >> >>
> >> >> >> That is to say it will be available in the latest open source
> >> writers
> >> >> and
> >> >> >> readers, but realistically not widely used in the ecosystem where
> >> >> >> compatibility is a concern (a large part)
> >> >> >>
> >> >> >> I view Julian's work[2][3] to gather consensus on how to name
> >> >> >> combinations of features (e.g. "versions") as the mechanism by
> >> which we
> >> >> >> will accelerate the adoption of this change (along with other
> >> forward
> >> >> >> incompatible changes) across the ecosystem.
> >> >> >>
> >> >> >> > Will we exclude this change when cutting parquet-format v2.14.0?
> >> >> >>
> >> >> >> No I don't think so -- I would expect it to go out in the next
> >> >> >> parquet-format version as do other changes in the spec.
> >> >> >>
> >> >> >> Andrew
> >> >> >>
> >> >> >> [1]: https://parquet.apache.org/docs/file-format/versions/
> >> >> >> [2]:
> >> https://lists.apache.org/thread/xmnj8h0h8ozmrgox5tydhhs7yz7q2hss
> >> >> >> [3]:
> >> https://lists.apache.org/thread/qlf8lg90gqq41lllyy9mk2f0skqvftqj
> >> >> >>
> >> >> >>
> >> >> >> On Thu, Jul 30, 2026 at 3:51 AM Divjot Arora via dev <
> >> >> >> dev@parquet.apache.org> wrote:
> >> >> >>
> >> >> >>> Hi Andrew,
> >> >> >>>
> >> >> >>> How does this interact with the ongoing versioning discussion?
> >> Given
> >> >> that
> >> >> >>> new encodings are not forward compatible, I would assume ALP is
> >> >> included
> >> >> >>> in
> >> >> >>> Parquet V<next> (V3?), but it's not clear to me how that works if
> >> it's
> >> >> >>> merged to the parquet-format master branch now. Will we exclude
> >> this
> >> >> >>> change
> >> >> >>> when cutting parquet-format v2.14.0?
> >> >> >>>
> >> >> >>> -- Div
> >> >> >>>
> >> >> >>> On Thu, Jul 30, 2026 at 8:51 AM Andrew Lamb <
> >> andrewlamb11@gmail.com>
> >> >> >>> wrote:
> >> >> >>>
> >> >> >>> > Hello,
> >> >> >>> >
> >> >> >>> > As an update, thanks to the epic work from many contributors,
> >> >> >>> especially
> >> >> >>> > Prateek, I just merged the parquet-format change for ALP[1]. We
> >> >> plan to
> >> >> >>> > make additional editorial The remaining items I know of are
> >> merging
> >> >> the
> >> >> >>> > examples from parquet-testing[2], and then of course the
> various
> >> >> >>> > implementations
> >> >> >>> >
> >> >> >>> > Thanks again everyone. This is a great step forward
> >> >> >>> > Andrew
> >> >> >>> >
> >> >> >>> > [1]: https://github.com/apache/parquet-format/pull/557
> >> >> >>> > [2]: https://github.com/apache/parquet-testing/pull/100
> >> >> >>> >
> >> >> >>> > On Tue, Jul 21, 2026 at 4:39 PM Julien Le Dem <
> julien@apache.org
> >> >
> >> >> >>> wrote:
> >> >> >>> >
> >> >> >>> > > It was also in my spam folder... :(
> >> >> >>> > > I've seen a bunch of apache list emails go to my spam
> recently.
> >> >> >>> > >
> >> >> >>> > > On Tue, Jul 21, 2026 at 10:58 AM Russell Spitzer <
> >> >> >>> > > russell.spitzer@gmail.com>
> >> >> >>> > > wrote:
> >> >> >>> > >
> >> >> >>> > > > Yes, my hope was that my vote would kick it out of spam but
> >> we
> >> >> >>> > definitely
> >> >> >>> > > > had two other votes that also went to spam for me.
> >> >> >>> > > >
> >> >> >>> > > > For anyone looking the thread is here
> >> >> >>> > > >
> >> >> https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt
> >> >> >>> > > >
> >> >> >>> > > > On Tue, Jul 21, 2026 at 10:57 AM Micah Kornfield <
> >> >> >>> > emkornfield@gmail.com>
> >> >> >>> > > > wrote:
> >> >> >>> > > >
> >> >> >>> > > > > I've seen other's votes not go to spam, so hopefully
> people
> >> >> will
> >> >> >>> see
> >> >> >>> > it
> >> >> >>> > > > in
> >> >> >>> > > > > there inboxes (or see this thread and look in there
> spam).
> >> >> >>> > > > >
> >> >> >>> > > > > On Tue, Jul 21, 2026 at 10:52 AM PRATEEK GAUR <
> >> >> >>> prtkgaur@gmail.com>
> >> >> >>> > > > wrote:
> >> >> >>> > > > >
> >> >> >>> > > > > > Okay,
> >> >> >>> > > > > >
> >> >> >>> > > > > > I sent out the voting email yesterday and so far I've
> >> heard
> >> >> >>> from 3
> >> >> >>> > > > people
> >> >> >>> > > > > > that it ended up in their "spam folder". Dev VPN to be
> >> >> blamed
> >> >> >>> for
> >> >> >>> > > it?.
> >> >> >>> > > > > >
> >> >> >>> > > > > > What's the protocol here?
> >> >> >>> > > > > > I let people hunt for the email in their spam folder or
> >> send
> >> >> >>> out
> >> >> >>> > > > another
> >> >> >>> > > > > > one?
> >> >> >>> > > > > >
> >> >> >>> > > > > > Best
> >> >> >>> > > > > > Prateek
> >> >> >>> > > > > >
> >> >> >>> > > > > > On Sun, Jul 19, 2026 at 10:47 PM PRATEEK GAUR <
> >> >> >>> prtkgaur@gmail.com>
> >> >> >>> > > > > wrote:
> >> >> >>> > > > > >
> >> >> >>> > > > > > > Hi team,
> >> >> >>> > > > > > >
> >> >> >>> > > > > > > Vinoo has added the tests which were requested in the
> >> last
> >> >> >>> round
> >> >> >>> > of
> >> >> >>> > > > > > review.
> >> >> >>> > > > > > > In his own words.
> >> >> >>> > > > > > >
> >> >> >>> > > > > > > "
> >> >> >>> > > > > > > The extreme value tests Micah wanted are done and
> >> pushed.
> >> >> I
> >> >> >>> added
> >> >> >>> > > > > > coverage
> >> >> >>> > > > > > > for values that need the full FOR bit width after the
> >> >> frame
> >> >> >>> of
> >> >> >>> > > > > reference
> >> >> >>> > > > > > is
> >> >> >>> > > > > > > applied: 64-bit (the case where the signed max minus
> >> min
> >> >> >>> > > subtraction
> >> >> >>> > > > > > > overflows), 63-bit (the non-overflow case), and
> 32-bit
> >> for
> >> >> >>> > floats.
> >> >> >>> > > > All
> >> >> >>> > > > > > > lossless with zero exceptions. I also verified them
> >> >> through
> >> >> >>> Arrow
> >> >> >>> > > C++
> >> >> >>> > > > > ALP
> >> >> >>> > > > > > > reader (the local build), so the Java-written extreme
> >> >> columns
> >> >> >>> > > decode
> >> >> >>> > > > > > > correctly in C++, bit-exact against the expected
> >> values."
> >> >> >>> > > > > > >
> >> >> >>> > > > > > > State of the PR (#3397):
> >> >> >>> > > > > > >
> >> >> >>> > > > > > > - Merges cleanly with master (I pulled master in
> and
> >> >> >>> resolved
> >> >> >>> > > the
> >> >> >>> > > > > > > conflicts).
> >> >> >>> > > > > > > - 41 of 44 review threads resolved. The 3
> remaining
> >> are
> >> >> >>> > > > intentional:
> >> >> >>> > > > > > > one deferred perf nit, one that stays open until
> >> >> >>> > parquet-format
> >> >> >>> > > > > ships
> >> >> >>> > > > > > ALP,
> >> >> >>> > > > > > >
> >> >> >>> > > > > > > "
> >> >> >>> > > > > > >
> >> >> >>> > > > > > > With this both the
> >> >> >>> > > > > > > 1) C++ PR :
> >> >> >>> https://github.com/apache/arrow/pull/48345/changes
> >> >> >>> > > > > > > 2) Java PR :
> >> >> >>> https://github.com/apache/parquet-java/pull/3397
> >> >> >>> > > > > > >
> >> >> >>> > > > > > > Are cross language tested and range of test
> scenarios.
> >> >> >>> > > > > > >
> >> >> >>> > > > > > > Best
> >> >> >>> > > > > > > Prateek
> >> >> >>> > > > > > >
> >> >> >>> > > > > > >
> >> >> >>> > > > > > > On Mon, Jul 13, 2026 at 8:38 AM PRATEEK GAUR <
> >> >> >>> prtkgaur@gmail.com
> >> >> >>> > >
> >> >> >>> > > > > wrote:
> >> >> >>> > > > > > >
> >> >> >>> > > > > > >> Thanks Matt,
> >> >> >>> > > > > > >>
> >> >> >>> > > > > > >> I'll be on it this week.
> >> >> >>> > > > > > >>
> >> >> >>> > > > > > >> Best
> >> >> >>> > > > > > >> Prateek
> >> >> >>> > > > > > >>
> >> >> >>> > > > > > >> On Mon, Jul 13, 2026 at 8:36 AM Matt Topol <
> >> >> >>> > > zotthewizard@gmail.com>
> >> >> >>> > > > > > >> wrote:
> >> >> >>> > > > > > >>
> >> >> >>> > > > > > >>> There is also the Go implementation submitted by
> >> Arnav (
> >> >> >>> > > > > > >>> https://github.com/apache/arrow-go/pull/704) which
> >> is
> >> >> >>> waiting
> >> >> >>> > > for
> >> >> >>> > > > > > >>> updates.
> >> >> >>> > > > > > >>>
> >> >> >>> > > > > > >>> Just wanted to make sure it didn't get lost here.
> >> >> >>> > > > > > >>>
> >> >> >>> > > > > > >>> --Matt
> >> >> >>> > > > > > >>>
> >> >> >>> > > > > > >>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK GAUR <
> >> >> >>> > > prtkgaur@gmail.com>
> >> >> >>> > > > > > >>> wrote:
> >> >> >>> > > > > > >>>
> >> >> >>> > > > > > >>> > Hi all,
> >> >> >>> > > > > > >>> >
> >> >> >>> > > > > > >>> > I'd like to share a status update on the ALP
> >> encoding
> >> >> >>> effort
> >> >> >>> > > and
> >> >> >>> > > > > get
> >> >> >>> > > > > > a
> >> >> >>> > > > > > >>> feel
> >> >> >>> > > > > > >>> > from the community before starting a formal vote.
> >> >> >>> > > > > > >>> >
> >> >> >>> > > > > > >>> > We now have the specification plus
> implementations
> >> in
> >> >> >>> both
> >> >> >>> > > > > languages:
> >> >> >>> > > > > > >>> >
> >> >> >>> > > > > > >>> > - Spec: apache/parquet-format#557 —
> >> GH-533
> >> >> >>> Add
> >> >> >>> > ALP
> >> >> >>> > > > > > encoding
> >> >> >>> > > > > > >>> > specification
> >> >> >>> > > > > > >>> > - C++ (Arrow): apache/arrow#48345 — GH-48701
> >> >> >>> > > [C++][Parquet]
> >> >> >>> > > > > Add
> >> >> >>> > > > > > >>> ALPpd
> >> >> >>> > > > > > >>> > encoding
> >> >> >>> > > > > > >>> > - Java: apache/parquet-java#3397 —
> >> Parquet
> >> >> >>> Java
> >> >> >>> > ALP
> >> >> >>> > > > > > >>> > Implementation (by Vinoo)
> >> >> >>> > > > > > >>> >
> >> >> >>> > > > > > >>> > Current state:
> >> >> >>> > > > > > >>> >
> >> >> >>> > > > > > >>> > - The C++ implementation has been through 3-4
> >> >> rounds
> >> >> >>> of
> >> >> >>> > > > review.
> >> >> >>> > > > > > >>> > - The Java implementation has been tracking
> >> well
> >> >> and
> >> >> >>> has
> >> >> >>> > > had
> >> >> >>> > > > > > >>> initial
> >> >> >>> > > > > > >>> > reviews from contributors, with no major
> >> >> remaining
> >> >> >>> > > issues.
> >> >> >>> > > > > > >>> > - Cross-language compatibility tests are
> >> passing:
> >> >> the
> >> >> >>> > Arrow
> >> >> >>> > > > C++
> >> >> >>> > > > > > >>> decoder
> >> >> >>> > > > > > >>> > reads Java-written data bit-exactly across
> >> >> ~1.56M
> >> >> >>> > values
> >> >> >>> > > > and
> >> >> >>> > > > > 18
> >> >> >>> > > > > > >>> > fixtures, covering V1 and V2 pages,
> multiple
> >> >> vector
> >> >> >>> > > sizes,
> >> >> >>> > > > > and
> >> >> >>> > > > > > >>> > several
> >> >> >>> > > > > > >>> > real datasets — zero mismatches.
> >> >> >>> > > > > > >>> >
> >> >> >>> > > > > > >>> > As raised in the java-pr we will be adding
> >> coverage
> >> >> for
> >> >> >>> > > extreme
> >> >> >>> > > > > > values
> >> >> >>> > > > > > >>> > (those requiring
> >> >> >>> > > > > > >>> > 63-64 bits after FOR is applied) before we
> close a
> >> >> vote.
> >> >> >>> > We'll
> >> >> >>> > > > aim
> >> >> >>> > > > > > to
> >> >> >>> > > > > > >>> get
> >> >> >>> > > > > > >>> > that
> >> >> >>> > > > > > >>> > done in parallel.
> >> >> >>> > > > > > >>> >
> >> >> >>> > > > > > >>> > Plan: unless there are objections, I intend to
> >> start a
> >> >> >>> vote
> >> >> >>> > at
> >> >> >>> > > > the
> >> >> >>> > > > > > end
> >> >> >>> > > > > > >>> of
> >> >> >>> > > > > > >>> > this week/early
> >> >> >>> > > > > > >>> > next week.
> >> >> >>> > > > > > >>> >
> >> >> >>> > > > > > >>> > Thanks
> >> >> >>> > > > > > >>> > Prateek
> >> >> >>> > > > > > >>> >
> >> >> >>> > > > > > >>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <
> >> >> >>> > > prtkgaur@gmail.com>
> >> >> >>> > > > > > >>> wrote:
> >> >> >>> > > > > > >>> >
> >> >> >>> > > > > > >>> > > Hi Team,
> >> >> >>> > > > > > >>> > >
> >> >> >>> > > > > > >>> > > Just wanted to provide some updates on ALP.
> >> >> >>> > > > > > >>> > > Micah and I have done 3-4 rounds of review for
> >> the
> >> >> c++
> >> >> >>> PR :
> >> >> >>> > > > > > >>> > > https://github.com/apache/arrow/pull/48345
> >> >> >>> > > > > > >>> > >
> >> >> >>> > > > > > >>> > > For the Java implementation Vinoo has been
> >> working
> >> >> on
> >> >> >>> > > following
> >> >> >>> > > > > PR
> >> >> >>> > > > > > :
> >> >> >>> > > > > > >>> > >
> https://github.com/apache/parquet-java/pull/3397
> >> >> >>> > > > > > >>> > >
> >> >> >>> > > > > > >>> > > Best
> >> >> >>> > > > > > >>> > > Prateek
> >> >> >>> > > > > > >>> > >
> >> >> >>> > > > > > >>> > > On Tue, May 5, 2026 at 1:50 PM Micah Kornfield
> <
> >> >> >>> > > > > > >>> emkornfield@gmail.com>
> >> >> >>> > > > > > >>> > > wrote:
> >> >> >>> > > > > > >>> > >
> >> >> >>> > > > > > >>> > >> Hi Antoine,
> >> >> >>> > > > > > >>> > >>
> >> >> >>> > > > > > >>> > >> > Apologies if the question was already asked,
> >> but
> >> >> >>> should
> >> >> >>> > we
> >> >> >>> > > > > care
> >> >> >>> > > > > > >>> about
> >> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more
> >> >> efficient
> >> >> >>> than
> >> >> >>> > > > > FLOAT16
> >> >> >>> > > > > > +
> >> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> >> >> >>> > > > > > >>> > >>
> >> >> >>> > > > > > >>> > >> It was. We thought we could defer it for the
> >> >> >>> following
> >> >> >>> > > > reasons:
> >> >> >>> > > > > > >>> > >> 1. It's not clear there are a lot of easy
> >> >> reference
> >> >> >>> > > datasets
> >> >> >>> > > > to
> >> >> >>> > > > > > >>> test
> >> >> >>> > > > > > >>> > its
> >> >> >>> > > > > > >>> > >> effectiveness. It does look like there might
> be
> >> >> one
> >> >> >>> or
> >> >> >>> > two
> >> >> >>> > > on
> >> >> >>> > > > > > >>> > huggingface
> >> >> >>> > > > > > >>> > >> (e.g.
> >> >> https://huggingface.co/datasets/kikitora/curdie
> >> >> >>> ).
> >> >> >>> > > > > > >>> > >> 2. It seemed likely that float 16 was more
> >> likely
> >> >> >>> used
> >> >> >>> > for
> >> >> >>> > > > > values
> >> >> >>> > > > > > >>> that
> >> >> >>> > > > > > >>> > >> were less likely to reduce to decimal values.
> >> >> >>> > > > > > >>> > >> 3. It could be added as an extension later if
> >> >> needed.
> >> >> >>> > > > > > >>> > >>
> >> >> >>> > > > > > >>> > >> Cheers,
> >> >> >>> > > > > > >>> > >> Micah
> >> >> >>> > > > > > >>> > >>
> >> >> >>> > > > > > >>> > >>
> >> >> >>> > > > > > >>> > >> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou
> <
> >> >> >>> > > > > antoine@python.org
> >> >> >>> > > > > > >
> >> >> >>> > > > > > >>> > wrote:
> >> >> >>> > > > > > >>> > >>
> >> >> >>> > > > > > >>> > >> >
> >> >> >>> > > > > > >>> > >> > Hello,
> >> >> >>> > > > > > >>> > >> >
> >> >> >>> > > > > > >>> > >> > Apologies if the question was already asked,
> >> but
> >> >> >>> should
> >> >> >>> > we
> >> >> >>> > > > > care
> >> >> >>> > > > > > >>> about
> >> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more
> >> >> efficient
> >> >> >>> than
> >> >> >>> > > > > FLOAT16
> >> >> >>> > > > > > +
> >> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> >> >> >>> > > > > > >>> > >> >
> >> >> >>> > > > > > >>> > >> > Regards
> >> >> >>> > > > > > >>> > >> >
> >> >> >>> > > > > > >>> > >> > Antoine.
> >> >> >>> > > > > > >>> > >> >
> >> >> >>> > > > > > >>> > >> >
> >> >> >>> > > > > > >>> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit
> :
> >> >> >>> > > > > > >>> > >> > > Thanks Andrew and Micah for review
> feedback
> >> on
> >> >> >>> the two
> >> >> >>> > > > PR's
> >> >> >>> > > > > > >>> > >> > > 1) (c++ arrow repo)
> >> >> >>> > > > > > >>> > >>
> >> https://github.com/apache/arrow/pull/48345/changes
> >> >> >>> > > > > > >>> > >> > > 2) (parquet-format repo)
> >> >> >>> > > > > > >>> > >> >
> >> >> https://github.com/apache/parquet-format/pull/557
> >> >> >>> > > > > > >>> > >> > >
> >> >> >>> > > > > > >>> > >> > > I have addressed all (unless I missed
> >> >> something)
> >> >> >>> > > comments
> >> >> >>> > > > on
> >> >> >>> > > > > > >>> the two
> >> >> >>> > > > > > >>> > >> > PR's.
> >> >> >>> > > > > > >>> > >> > >
> >> >> >>> > > > > > >>> > >> > > Best
> >> >> >>> > > > > > >>> > >> > > Prateek
> >> >> >>> > > > > > >>> > >> > >
> >> >> >>> > > > > > >>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK
> >> GAUR <
> >> >> >>> > > > > > >>> prtkgaur@gmail.com>
> >> >> >>> > > > > > >>> > >> wrote:
> >> >> >>> > > > > > >>> > >> > >
> >> >> >>> > > > > > >>> > >> > >> Thanks Andrew and Micah.
> >> >> >>> > > > > > >>> > >> > >>
> >> >> >>> > > > > > >>> > >> > >> `fair amount of feedback on at least the
> >> >> >>> > > implementations`
> >> >> >>> > > > > > >>> > >> > >> For the c++ I have already started
> >> addressing
> >> >> the
> >> >> >>> > > > > feedback, I
> >> >> >>> > > > > > >>> > should
> >> >> >>> > > > > > >>> > >> be
> >> >> >>> > > > > > >>> > >> > >> done with that Monday/Tuesday.
> >> >> >>> > > > > > >>> > >> > >> I think Vinoo too has been making good
> >> >> progress
> >> >> >>> on
> >> >> >>> > the
> >> >> >>> > > > Java
> >> >> >>> > > > > > >>> > >> > implementation.
> >> >> >>> > > > > > >>> > >> > >>
> >> >> >>> > > > > > >>> > >> > >> Best
> >> >> >>> > > > > > >>> > >> > >> Prateek
> >> >> >>> > > > > > >>> > >> > >>
> >> >> >>> > > > > > >>> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew
> >> Lamb <
> >> >> >>> > > > > > >>> > andrewlamb11@gmail.com
> >> >> >>> > > > > > >>> > >> >
> >> >> >>> > > > > > >>> > >> > >> wrote:
> >> >> >>> > > > > > >>> > >> > >>
> >> >> >>> > > > > > >>> > >> > >>> Got it. Thank you for the clarification
> >> -- I
> >> >> >>> will
> >> >> >>> > try
> >> >> >>> > > > and
> >> >> >>> > > > > > look
> >> >> >>> > > > > > >>> > into
> >> >> >>> > > > > > >>> > >> the
> >> >> >>> > > > > > >>> > >> > >>> spec and the Rust implementation[1] in
> >> this
> >> >> next
> >> >> >>> > week
> >> >> >>> > > > > > >>> > >> > >>>
> >> >> >>> > > > > > >>> > >> > >>> [1]:
> >> >> >>> https://github.com/apache/arrow-rs/pull/9372
> >> >> >>> > > > > > >>> > >> > >>>
> >> >> >>> > > > > > >>> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah
> >> >> >>> Kornfield <
> >> >> >>> > > > > > >>> > >> > emkornfield@gmail.com>
> >> >> >>> > > > > > >>> > >> > >>> wrote:
> >> >> >>> > > > > > >>> > >> > >>>
> >> >> >>> > > > > > >>> > >> > >>>> Hi Andrew,
> >> >> >>> > > > > > >>> > >> > >>>> I think there is a fair amount of
> >> feedback
> >> >> on
> >> >> >>> at
> >> >> >>> > > least
> >> >> >>> > > > > the
> >> >> >>> > > > > > >>> > >> > >>>> implementations, typically I think
> we've
> >> >> waited
> >> >> >>> > till
> >> >> >>> > > > they
> >> >> >>> > > > > > are
> >> >> >>> > > > > > >>> > >> close to
> >> >> >>> > > > > > >>> > >> > >>>> mergeable before a final vote.
> >> Otherwise I
> >> >> >>> agree
> >> >> >>> > we
> >> >> >>> > > > are
> >> >> >>> > > > > > very
> >> >> >>> > > > > > >>> > >> close.
> >> >> >>> > > > > > >>> > >> > >>>>
> >> >> >>> > > > > > >>> > >> > >>>> -Micah
> >> >> >>> > > > > > >>> > >> > >>>>
> >> >> >>> > > > > > >>> > >> > >>>> On Saturday, April 25, 2026, Andrew
> Lamb
> >> <
> >> >> >>> > > > > > >>> andrewlamb11@gmail.com
> >> >> >>> > > > > > >>> > >
> >> >> >>> > > > > > >>> > >> > wrote:
> >> >> >>> > > > > > >>> > >> > >>>>
> >> >> >>> > > > > > >>> > >> > >>>>> Thanks Prateek,
> >> >> >>> > > > > > >>> > >> > >>>>>
> >> >> >>> > > > > > >>> > >> > >>>>> I think from this content it looks to
> me
> >> >> like
> >> >> >>> we
> >> >> >>> > are
> >> >> >>> > > > > ready
> >> >> >>> > > > > > >>> to
> >> >> >>> > > > > > >>> > >> start a
> >> >> >>> > > > > > >>> > >> > >>>>> vote to explicitly accept ALP into
> >> Parquet
> >> >> >>> > > > > > >>> > >> > >>>>>
> >> >> >>> > > > > > >>> > >> > >>>>> Does anyone know of a reason we should
> >> >> >>> postpone it
> >> >> >>> > > for
> >> >> >>> > > > > > >>> longer?
> >> >> >>> > > > > > >>> > >> > >>>>> Perhaps someone needs some more time
> to
> >> >> >>> review?
> >> >> >>> > > > > > >>> > >> > >>>>>
> >> >> >>> > > > > > >>> > >> > >>>>> Andrew
> >> >> >>> > > > > > >>> > >> > >>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>
> >> >> >>> > > > > > >>> > >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM
> PRATEEK
> >> >> GAUR <
> >> >> >>> > > > > > >>> > prtkgaur@gmail.com>
> >> >> >>> > > > > > >>> > >> > >>>>> wrote:
> >> >> >>> > > > > > >>> > >> > >>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>> Hi team,
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>> Hope everyone is doing well. I got a
> >> >> chance
> >> >> >>> to
> >> >> >>> > work
> >> >> >>> > > > > > >>> through all
> >> >> >>> > > > > > >>> > >> the
> >> >> >>> > > > > > >>> > >> > >>>>>> remaining feedback and update the
> spec
> >> >> doc.
> >> >> >>> Here
> >> >> >>> > > are
> >> >> >>> > > > > the
> >> >> >>> > > > > > >>> new
> >> >> >>> > > > > > >>> > >> > artifacts
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>> 1) Spec document :
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> >
> >> >> >>> > > > > > >>> > >>
> >> >> >>> > > > > > >>> >
> >> >> >>> > > > > > >>>
> >> >> >>> > > > > >
> >> >> >>> > > > >
> >> >> >>> > > >
> >> >> >>> > >
> >> >> >>> >
> >> >> >>>
> >> >>
> >>
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>> 2) Spec document in parquet format
> >> repo :
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > https://github.com/apache/parquet-format/pull/557
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>> 3) Alp implementation in arrow c++
> >> repo :
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > https://github.com/apache/arrow/pull/48345/changes
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>> 4) Alp implementation in parquet-java
> >> >> repo :
> >> >> >>> Work
> >> >> >>> > > for
> >> >> >>> > > > > > >>> Vinoo and
> >> >> >>> > > > > > >>> > >> > Julien
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > https://github.com/apache/parquet-java/pull/3397
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>> 5) PR with test and benchmarking
> >> >> artifacts in
> >> >> >>> > > > > > >>> parquet-testing
> >> >> >>> > > > > > >>> > >> repo :
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > https://github.com/apache/parquet-testing/pull/100
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>> And
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>> - Go : Arnav just submitted an in
> >> >> >>> progress
> >> >> >>> > > > > > >>> implementation
> >> >> >>> > > > > > >>> > in
> >> >> >>> > > > > > >>> > >> Go.
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> https://github.com/apache/arrow-go/pull/704
> >> >> >>> > (I
> >> >> >>> > > > > > haven't
> >> >> >>> > > > > > >>> > >> started
> >> >> >>> > > > > > >>> > >> > >>>>>> looking at it yet)
> >> >> >>> > > > > > >>> > >> > >>>>>> - Rust : I remember Andrew
> >> mentioned
> >> >> that
> >> >> >>> > this
> >> >> >>> > > > work
> >> >> >>> > > > > > is
> >> >> >>> > > > > > >>> also
> >> >> >>> > > > > > >>> > >> in
> >> >> >>> > > > > > >>> > >> > >>>>>> progress (So 4 languages!)
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>> *Arrow C++ implementation *
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>> The PR is out and was also used by
> >> >> Antoine to
> >> >> >>> > > report
> >> >> >>> > > > > the
> >> >> >>> > > > > > >>> > numbers
> >> >> >>> > > > > > >>> > >> as
> >> >> >>> > > > > > >>> > >> > >>>>>> reported here. Micah and Konstantin
> >> have
> >> >> >>> given 1
> >> >> >>> > > > round
> >> >> >>> > > > > of
> >> >> >>> > > > > > >>> > >> feedback
> >> >> >>> > > > > > >>> > >> > >>>>>> and I'm addressing them today. Please
> >> note
> >> >> >>> that
> >> >> >>> > the
> >> >> >>> > > > > > default
> >> >> >>> > > > > > >>> > >> > >>>>>> optimization flag for compiling is O2
> >> and
> >> >> not
> >> >> >>> > Q3. I
> >> >> >>> > > > got
> >> >> >>> > > > > > >>> around
> >> >> >>> > > > > > >>> > >> 70%
> >> >> >>> > > > > > >>> > >> > >>>>>> performance improvement in the
> decoding
> >> >> speed
> >> >> >>> > when
> >> >> >>> > > > > using
> >> >> >>> > > > > > >>> the O3
> >> >> >>> > > > > > >>> > >> > flag.
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>> *Parqet-MR Java implementation
> (working
> >> >> with
> >> >> >>> > Vinoo
> >> >> >>> > > > and
> >> >> >>> > > > > > >>> Julien)
> >> >> >>> > > > > > >>> > >> and
> >> >> >>> > > > > > >>> > >> > **Cross
> >> >> >>> > > > > > >>> > >> > >>>>>> Language testing*
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>> Let me know if you have any
> >> questions
> >> >> or
> >> >> >>> > > > feedback.
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>> Now pasting some performance numbers
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>> Table 1: C++ ALP Double Decode —
> >> >> Spotify
> >> >> >>> > Columns
> >> >> >>> > > > > > >>> (Graviton
> >> >> >>> > > > > > >>> > 3,
> >> >> >>> > > > > > >>> > >> ARM
> >> >> >>> > > > > > >>> > >> > >>>>>> Neoverse V1)
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>>
> >> >> >>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>> │ Column │ -O2 (MB/s)
> │
> >> >> -O3
> >> >> >>> > > (MB/s) │
> >> >> >>> > > > > > >>> Speedup │
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>>
> >> >> >>> ├──────────────────┼──────────────┼──────────────┼─────────┤
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>> │ valence │ 3,155
> │
> >> >> >>> 5,523
> >> >> >>> > > │
> >> >> >>> > > > > > >>> 1.75x │
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>> │ danceability │ 3,233
> │
> >> >> >>> 5,685
> >> >> >>> > > │
> >> >> >>> > > > > > >>> 1.76x │
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>> │ energy │ 3,197
> │
> >> >> >>> 5,652
> >> >> >>> > > │
> >> >> >>> > > > > > >>> 1.77x │
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>> │ loudness │ 3,186
> │
> >> >> >>> 5,473
> >> >> >>> > > │
> >> >> >>> > > > > > >>> 1.72x │
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>>
> >> >> >>> └──────────────────┴──────────────┴──────────────┴─────────┘
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM
> PRATEEK
> >> >> GAUR
> >> >> >>> <
> >> >> >>> > > > > > >>> > prtkgaur@gmail.com
> >> >> >>> > > > > > >>> > >> >
> >> >> >>> > > > > > >>> > >> > >>>>>> wrote:
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>> @Micah Kornfield <
> >> emkornfield@gmail.com>
> >> >> :
> >> >> >>> Got
> >> >> >>> > > it.
> >> >> >>> > > > > > >>> > >> > >>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>> @Andrew Lamb <
> andrewlamb11@gmail.com>
> >> >> >>> > > > > > >>> > >> > >>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>> Do you think it would be good to
> >> start
> >> >> >>> moving
> >> >> >>> > the
> >> >> >>> > > > > spec
> >> >> >>> > > > > > >>> > >> development
> >> >> >>> > > > > > >>> > >> > >>>>>>>> into
> >> >> >>> > > > > > >>> > >> > >>>>>>>> markdown format, in preparation for
> >> >> >>> finalizing
> >> >> >>> > > it?
> >> >> >>> > > > > > >>> > >> > >>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>> Yes I'll update the numbers for some
> >> of
> >> >> the
> >> >> >>> > > > examples I
> >> >> >>> > > > > > >>> have in
> >> >> >>> > > > > > >>> > >> the
> >> >> >>> > > > > > >>> > >> > >>>>>>> spec based
> >> >> >>> > > > > > >>> > >> > >>>>>>> on the updated header size. Then we
> >> >> should
> >> >> >>> be
> >> >> >>> > good
> >> >> >>> > > > to
> >> >> >>> > > > > go
> >> >> >>> > > > > > >>> for
> >> >> >>> > > > > > >>> > the
> >> >> >>> > > > > > >>> > >> > >>>>>>> markdown format.
> >> >> >>> > > > > > >>> > >> > >>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>> Thanks everyone!
> >> >> >>> > > > > > >>> > >> > >>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>> Andrew
> >> >> >>> > > > > > >>> > >> > >>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM
> >> PRATEEK
> >> >> >>> GAUR <
> >> >> >>> > > > > > >>> > >> prtkgaur@gmail.com>
> >> >> >>> > > > > > >>> > >> > >>>>>>>> wrote:
> >> >> >>> > > > > > >>> > >> > >>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> Hi team,
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> 1) Andrew
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for working on test
> >> files.
> >> >> >>> My PR
> >> >> >>> > > did
> >> >> >>> > > > > add
> >> >> >>> > > > > > >>> all
> >> >> >>> > > > > > >>> > the
> >> >> >>> > > > > > >>> > >> > test
> >> >> >>> > > > > > >>> > >> > >>>>>>>> files I
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> used to benchmark on datasets.
> >> >> Maybe
> >> >> >>> we
> >> >> >>> > can
> >> >> >>> > > > club
> >> >> >>> > > > > > it
> >> >> >>> > > > > > >>> > >> together.
> >> >> >>> > > > > > >>> > >> > >>>>>>>> WIll also
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> aid
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> cross language testing
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Kosta Tarasov working on
> Rust
> >> >> >>> > > > implementation.
> >> >> >>> > > > > > >>> This is
> >> >> >>> > > > > > >>> > >> > great.
> >> >> >>> > > > > > >>> > >> > >>>>>>>> Thanks
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> 2) Antoine
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks a lot for reporting
> the
> >> >> >>> numbers
> >> >> >>> > on
> >> >> >>> > > > AMD.
> >> >> >>> > > > > > >>> Looks
> >> >> >>> > > > > > >>> > >> like
> >> >> >>> > > > > > >>> > >> > you
> >> >> >>> > > > > > >>> > >> > >>>>>>>> are
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> getting 8X the decoding
> >> >> performance of
> >> >> >>> > BSS.
> >> >> >>> > > > This
> >> >> >>> > > > > > is
> >> >> >>> > > > > > >>> > >> > amazing!!.
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for acknowledging the
> >> >> >>> sampling
> >> >> >>> > > > design.
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> - I agree with you on
> >> Fastlanes. In
> >> >> >>> some
> >> >> >>> > > crude
> >> >> >>> > > > > > >>> > >> experiments I
> >> >> >>> > > > > > >>> > >> > >>>>>>>> didn't get
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> a good perf benefit from it on
> >> >> >>> Graviton3
> >> >> >>> > > (but
> >> >> >>> > > > > > maybe
> >> >> >>> > > > > > >>> > there
> >> >> >>> > > > > > >>> > >> was
> >> >> >>> > > > > > >>> > >> > >>>>>>>> something
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> wrong with my implementation).
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Locking the 16bit exception
> >> >> >>> encoding for
> >> >> >>> > > the
> >> >> >>> > > > > > spec
> >> >> >>> > > > > > >>> in
> >> >> >>> > > > > > >>> > >> this
> >> >> >>> > > > > > >>> > >> > >>>>>>>> case.
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Awesome I think we have
> solved
> >> >> for
> >> >> >>> all
> >> >> >>> > > open
> >> >> >>> > > > > > >>> questions
> >> >> >>> > > > > > >>> > >> minus
> >> >> >>> > > > > > >>> > >> > >>>>>>>> the
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> version byte :). (will get
> back
> >> on
> >> >> >>> this
> >> >> >>> > > soon)
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> 3) Micah
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> - FastLanes : The current spec
> >> does
> >> >> >>> allow
> >> >> >>> > > for
> >> >> >>> > > > > > using
> >> >> >>> > > > > > >>> > >> FastLane
> >> >> >>> > > > > > >>> > >> > >>>>>>>> with the
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> configurable enum value for
> >> >> layout. We
> >> >> >>> > > should
> >> >> >>> > > > be
> >> >> >>> > > > > > >>> able to
> >> >> >>> > > > > > >>> > >> > inject
> >> >> >>> > > > > > >>> > >> > >>>>>>>> any
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> layout
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> in the current design.
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> Working on resolving all remaining
> >> open
> >> >> >>> > comments
> >> >> >>> > > > on
> >> >> >>> > > > > > the
> >> >> >>> > > > > > >>> spec
> >> >> >>> > > > > > >>> > >> this
> >> >> >>> > > > > > >>> > >> > >>>>>>>> week.
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> Best
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> Prateek
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM
> >> Steve
> >> >> >>> > Loughran <
> >> >> >>> > > > > > >>> > >> > >>>>>>>> stevel@cloudera.com>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> wrote:
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12,
> Micah
> >> >> >>> Kornfield
> >> >> >>> > <
> >> >> >>> > > > > > >>> > >> > >>>>>>>> emkornfield@gmail.com>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>> wrote:
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> It looks like the actual issue
> >> >> >>> described for
> >> >> >>> > > ORC
> >> >> >>> > > > > in
> >> >> >>> > > > > > >>> the
> >> >> >>> > > > > > >>> > >> paper
> >> >> >>> > > > > > >>> > >> > >>>>>>>> is that
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> it
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> has multiple sub-encodings in a
> >> >> batch.
> >> >> >>> This
> >> >> >>> > > is
> >> >> >>> > > > > > >>> different
> >> >> >>> > > > > > >>> > >> then
> >> >> >>> > > > > > >>> > >> > >>>>>>>> the
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> design
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> proposed here where there is
> still
> >> >> fixed
> >> >> >>> > > > encoding
> >> >> >>> > > > > > per
> >> >> >>> > > > > > >>> page
> >> >> >>> > > > > > >>> > >> in
> >> >> >>> > > > > > >>> > >> > >>>>>>>> parquet.
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> Given reasonably sized pages I
> >> don't
> >> >> >>> think
> >> >> >>> > > > branch
> >> >> >>> > > > > > >>> > >> > >>>>>>>> misprediction should
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>> be a
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> big issue for new encodings. I
> >> agree
> >> >> >>> that
> >> >> >>> > we
> >> >> >>> > > > > should
> >> >> >>> > > > > > >>> be
> >> >> >>> > > > > > >>> > >> > >>>>>>>> conservative in
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> general for adding new
> encodings.
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>> +1
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>
> >> >> >>> > > > > > >>> > >> > >
> >> >> >>> > > > > > >>> > >> >
> >> >> >>> > > > > > >>> > >> >
> >> >> >>> > > > > > >>> > >> >
> >> >> >>> > > > > > >>> > >>
> >> >> >>> > > > > > >>> > >
> >> >> >>> > > > > > >>> >
> >> >> >>> > > > > > >>>
> >> >> >>> > > > > > >>
> >> >> >>> > > > > >
> >> >> >>> > > > >
> >> >> >>> > > >
> >> >> >>> > >
> >> >> >>> >
> >> >> >>>
> >> >> >>
> >> >>
> >> >
> >>
> >
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by PRATEEK GAUR <pr...@gmail.com>.
Hi team,
Thanks Micah for multiple rounds of review and thanks Gang for taking a
look at the PR and giving feedback. I've addressed all comments in the PR.
Looking forward to the next round of feedback (or approval). Would like to
get it in soon.
https://github.com/apache/arrow/pull/48345/changes
With this done and after talking to Arnav, I plan to spend some time on the
ALP implementation in the arrow-go repository helping spread the support to
more languages.
Best
Prateek
On Sun, Aug 2, 2026 at 10:42 PM PRATEEK GAUR <pr...@gmail.com> wrote:
> Thanks Divjot for bringing it up.
>
> I was under the same impression as Russell and Andrew that we'd have it
> flag gated.
> With the new versioning proposal 'preview' might be a good tag for it.
>
> Best
> Prateek
>
> On Sat, Aug 1, 2026 at 12:39 AM Andrew Lamb <an...@gmail.com>
> wrote:
>
>> Div: Here is a proposal to mark ALP in Preview: [1] (using a comment in
>> the
>> spec section).
>>
>> Russel: That is my understanding as well.
>>
>> Hopefully we are able to formalize the versioning process shortly so we
>> can
>> formalize the notion of "preview"
>>
>> Andrew
>>
>>
>> [1]: https://github.com/apache/parquet-format/pull/604
>>
>> On Fri, Jul 31, 2026 at 11:15 AM Russell Spitzer <
>> russell.spitzer@gmail.com>
>> wrote:
>>
>> > I thought we had agreed it would be merged, and features like this would
>> > be feature flag gated. So a V-Old writer could always opt in to using
>> this
>> > encoding but that must be a conscious effort if you are not using VNext
>> of
>> > Parquet. If you are on V-Next the writer can choose to use ALP whenever
>> it
>> > likes without user intervention.
>> >
>> > On Fri, Jul 31, 2026 at 2:20 AM Divjot Arora via dev <
>> > dev@parquet.apache.org> wrote:
>> >
>> >> Yes, that’s my understanding as well. However, as you noted, the
>> >> versioning
>> >> discussion is still in early stages and hasn’t prescribed concrete
>> >> mechanisms yet. If ALP goes into parquet-format 2.14.0, there’s no
>> >> indication that it’s in preview or that writers shouldn’t use it by
>> >> default
>> >> yet. Given that we’ve held off on merging other incompatible changes
>> (e.g
>> >> removal of path_in_schema) until the versioning discussion closes, my
>> >> preference would be to leave ALP un-merged for the time being as well.
>> >>
>> >> — Div
>> >>
>> >> On Fri, Jul 31, 2026 at 08:59 Andrew Lamb <an...@gmail.com>
>> wrote:
>> >>
>> >> > After re-reading Julien's Versioning Proposal, my understanding is
>> that
>> >> if
>> >> > we adopt that system, we will somehow mark ALP in "Preview" in
>> >> > parquet-format for release Version 3.
>> >> >
>> >> > The versioning document doesn't seem to proscribe the particular
>> method
>> >> > for annotating "preview" features -- I think clearly marking them as
>> >> such
>> >> > in parquet.thrift as well as reflecting that on the
>> >> > https://parquet.apache.org/docs/file-format/versions/ page somehow
>> >> would
>> >> > be adequate.
>> >> >
>> >> > Andrew
>> >> >
>> >> > On Thu, Jul 30, 2026 at 11:50 AM Andrew Lamb <andrewlamb11@gmail.com
>> >
>> >> > wrote:
>> >> >
>> >> >> In my mind ALP is in the same position as other encodings that were
>> >> added
>> >> >> recently[1] (such as BYTE_STREAM_SPLIT for non floating point types)
>> >> >>
>> >> >> That is to say it will be available in the latest open source
>> writers
>> >> and
>> >> >> readers, but realistically not widely used in the ecosystem where
>> >> >> compatibility is a concern (a large part)
>> >> >>
>> >> >> I view Julian's work[2][3] to gather consensus on how to name
>> >> >> combinations of features (e.g. "versions") as the mechanism by
>> which we
>> >> >> will accelerate the adoption of this change (along with other
>> forward
>> >> >> incompatible changes) across the ecosystem.
>> >> >>
>> >> >> > Will we exclude this change when cutting parquet-format v2.14.0?
>> >> >>
>> >> >> No I don't think so -- I would expect it to go out in the next
>> >> >> parquet-format version as do other changes in the spec.
>> >> >>
>> >> >> Andrew
>> >> >>
>> >> >> [1]: https://parquet.apache.org/docs/file-format/versions/
>> >> >> [2]:
>> https://lists.apache.org/thread/xmnj8h0h8ozmrgox5tydhhs7yz7q2hss
>> >> >> [3]:
>> https://lists.apache.org/thread/qlf8lg90gqq41lllyy9mk2f0skqvftqj
>> >> >>
>> >> >>
>> >> >> On Thu, Jul 30, 2026 at 3:51 AM Divjot Arora via dev <
>> >> >> dev@parquet.apache.org> wrote:
>> >> >>
>> >> >>> Hi Andrew,
>> >> >>>
>> >> >>> How does this interact with the ongoing versioning discussion?
>> Given
>> >> that
>> >> >>> new encodings are not forward compatible, I would assume ALP is
>> >> included
>> >> >>> in
>> >> >>> Parquet V<next> (V3?), but it's not clear to me how that works if
>> it's
>> >> >>> merged to the parquet-format master branch now. Will we exclude
>> this
>> >> >>> change
>> >> >>> when cutting parquet-format v2.14.0?
>> >> >>>
>> >> >>> -- Div
>> >> >>>
>> >> >>> On Thu, Jul 30, 2026 at 8:51 AM Andrew Lamb <
>> andrewlamb11@gmail.com>
>> >> >>> wrote:
>> >> >>>
>> >> >>> > Hello,
>> >> >>> >
>> >> >>> > As an update, thanks to the epic work from many contributors,
>> >> >>> especially
>> >> >>> > Prateek, I just merged the parquet-format change for ALP[1]. We
>> >> plan to
>> >> >>> > make additional editorial The remaining items I know of are
>> merging
>> >> the
>> >> >>> > examples from parquet-testing[2], and then of course the various
>> >> >>> > implementations
>> >> >>> >
>> >> >>> > Thanks again everyone. This is a great step forward
>> >> >>> > Andrew
>> >> >>> >
>> >> >>> > [1]: https://github.com/apache/parquet-format/pull/557
>> >> >>> > [2]: https://github.com/apache/parquet-testing/pull/100
>> >> >>> >
>> >> >>> > On Tue, Jul 21, 2026 at 4:39 PM Julien Le Dem <julien@apache.org
>> >
>> >> >>> wrote:
>> >> >>> >
>> >> >>> > > It was also in my spam folder... :(
>> >> >>> > > I've seen a bunch of apache list emails go to my spam recently.
>> >> >>> > >
>> >> >>> > > On Tue, Jul 21, 2026 at 10:58 AM Russell Spitzer <
>> >> >>> > > russell.spitzer@gmail.com>
>> >> >>> > > wrote:
>> >> >>> > >
>> >> >>> > > > Yes, my hope was that my vote would kick it out of spam but
>> we
>> >> >>> > definitely
>> >> >>> > > > had two other votes that also went to spam for me.
>> >> >>> > > >
>> >> >>> > > > For anyone looking the thread is here
>> >> >>> > > >
>> >> https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt
>> >> >>> > > >
>> >> >>> > > > On Tue, Jul 21, 2026 at 10:57 AM Micah Kornfield <
>> >> >>> > emkornfield@gmail.com>
>> >> >>> > > > wrote:
>> >> >>> > > >
>> >> >>> > > > > I've seen other's votes not go to spam, so hopefully people
>> >> will
>> >> >>> see
>> >> >>> > it
>> >> >>> > > > in
>> >> >>> > > > > there inboxes (or see this thread and look in there spam).
>> >> >>> > > > >
>> >> >>> > > > > On Tue, Jul 21, 2026 at 10:52 AM PRATEEK GAUR <
>> >> >>> prtkgaur@gmail.com>
>> >> >>> > > > wrote:
>> >> >>> > > > >
>> >> >>> > > > > > Okay,
>> >> >>> > > > > >
>> >> >>> > > > > > I sent out the voting email yesterday and so far I've
>> heard
>> >> >>> from 3
>> >> >>> > > > people
>> >> >>> > > > > > that it ended up in their "spam folder". Dev VPN to be
>> >> blamed
>> >> >>> for
>> >> >>> > > it?.
>> >> >>> > > > > >
>> >> >>> > > > > > What's the protocol here?
>> >> >>> > > > > > I let people hunt for the email in their spam folder or
>> send
>> >> >>> out
>> >> >>> > > > another
>> >> >>> > > > > > one?
>> >> >>> > > > > >
>> >> >>> > > > > > Best
>> >> >>> > > > > > Prateek
>> >> >>> > > > > >
>> >> >>> > > > > > On Sun, Jul 19, 2026 at 10:47 PM PRATEEK GAUR <
>> >> >>> prtkgaur@gmail.com>
>> >> >>> > > > > wrote:
>> >> >>> > > > > >
>> >> >>> > > > > > > Hi team,
>> >> >>> > > > > > >
>> >> >>> > > > > > > Vinoo has added the tests which were requested in the
>> last
>> >> >>> round
>> >> >>> > of
>> >> >>> > > > > > review.
>> >> >>> > > > > > > In his own words.
>> >> >>> > > > > > >
>> >> >>> > > > > > > "
>> >> >>> > > > > > > The extreme value tests Micah wanted are done and
>> pushed.
>> >> I
>> >> >>> added
>> >> >>> > > > > > coverage
>> >> >>> > > > > > > for values that need the full FOR bit width after the
>> >> frame
>> >> >>> of
>> >> >>> > > > > reference
>> >> >>> > > > > > is
>> >> >>> > > > > > > applied: 64-bit (the case where the signed max minus
>> min
>> >> >>> > > subtraction
>> >> >>> > > > > > > overflows), 63-bit (the non-overflow case), and 32-bit
>> for
>> >> >>> > floats.
>> >> >>> > > > All
>> >> >>> > > > > > > lossless with zero exceptions. I also verified them
>> >> through
>> >> >>> Arrow
>> >> >>> > > C++
>> >> >>> > > > > ALP
>> >> >>> > > > > > > reader (the local build), so the Java-written extreme
>> >> columns
>> >> >>> > > decode
>> >> >>> > > > > > > correctly in C++, bit-exact against the expected
>> values."
>> >> >>> > > > > > >
>> >> >>> > > > > > > State of the PR (#3397):
>> >> >>> > > > > > >
>> >> >>> > > > > > > - Merges cleanly with master (I pulled master in and
>> >> >>> resolved
>> >> >>> > > the
>> >> >>> > > > > > > conflicts).
>> >> >>> > > > > > > - 41 of 44 review threads resolved. The 3 remaining
>> are
>> >> >>> > > > intentional:
>> >> >>> > > > > > > one deferred perf nit, one that stays open until
>> >> >>> > parquet-format
>> >> >>> > > > > ships
>> >> >>> > > > > > ALP,
>> >> >>> > > > > > >
>> >> >>> > > > > > > "
>> >> >>> > > > > > >
>> >> >>> > > > > > > With this both the
>> >> >>> > > > > > > 1) C++ PR :
>> >> >>> https://github.com/apache/arrow/pull/48345/changes
>> >> >>> > > > > > > 2) Java PR :
>> >> >>> https://github.com/apache/parquet-java/pull/3397
>> >> >>> > > > > > >
>> >> >>> > > > > > > Are cross language tested and range of test scenarios.
>> >> >>> > > > > > >
>> >> >>> > > > > > > Best
>> >> >>> > > > > > > Prateek
>> >> >>> > > > > > >
>> >> >>> > > > > > >
>> >> >>> > > > > > > On Mon, Jul 13, 2026 at 8:38 AM PRATEEK GAUR <
>> >> >>> prtkgaur@gmail.com
>> >> >>> > >
>> >> >>> > > > > wrote:
>> >> >>> > > > > > >
>> >> >>> > > > > > >> Thanks Matt,
>> >> >>> > > > > > >>
>> >> >>> > > > > > >> I'll be on it this week.
>> >> >>> > > > > > >>
>> >> >>> > > > > > >> Best
>> >> >>> > > > > > >> Prateek
>> >> >>> > > > > > >>
>> >> >>> > > > > > >> On Mon, Jul 13, 2026 at 8:36 AM Matt Topol <
>> >> >>> > > zotthewizard@gmail.com>
>> >> >>> > > > > > >> wrote:
>> >> >>> > > > > > >>
>> >> >>> > > > > > >>> There is also the Go implementation submitted by
>> Arnav (
>> >> >>> > > > > > >>> https://github.com/apache/arrow-go/pull/704) which
>> is
>> >> >>> waiting
>> >> >>> > > for
>> >> >>> > > > > > >>> updates.
>> >> >>> > > > > > >>>
>> >> >>> > > > > > >>> Just wanted to make sure it didn't get lost here.
>> >> >>> > > > > > >>>
>> >> >>> > > > > > >>> --Matt
>> >> >>> > > > > > >>>
>> >> >>> > > > > > >>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK GAUR <
>> >> >>> > > prtkgaur@gmail.com>
>> >> >>> > > > > > >>> wrote:
>> >> >>> > > > > > >>>
>> >> >>> > > > > > >>> > Hi all,
>> >> >>> > > > > > >>> >
>> >> >>> > > > > > >>> > I'd like to share a status update on the ALP
>> encoding
>> >> >>> effort
>> >> >>> > > and
>> >> >>> > > > > get
>> >> >>> > > > > > a
>> >> >>> > > > > > >>> feel
>> >> >>> > > > > > >>> > from the community before starting a formal vote.
>> >> >>> > > > > > >>> >
>> >> >>> > > > > > >>> > We now have the specification plus implementations
>> in
>> >> >>> both
>> >> >>> > > > > languages:
>> >> >>> > > > > > >>> >
>> >> >>> > > > > > >>> > - Spec: apache/parquet-format#557 —
>> GH-533
>> >> >>> Add
>> >> >>> > ALP
>> >> >>> > > > > > encoding
>> >> >>> > > > > > >>> > specification
>> >> >>> > > > > > >>> > - C++ (Arrow): apache/arrow#48345 — GH-48701
>> >> >>> > > [C++][Parquet]
>> >> >>> > > > > Add
>> >> >>> > > > > > >>> ALPpd
>> >> >>> > > > > > >>> > encoding
>> >> >>> > > > > > >>> > - Java: apache/parquet-java#3397 —
>> Parquet
>> >> >>> Java
>> >> >>> > ALP
>> >> >>> > > > > > >>> > Implementation (by Vinoo)
>> >> >>> > > > > > >>> >
>> >> >>> > > > > > >>> > Current state:
>> >> >>> > > > > > >>> >
>> >> >>> > > > > > >>> > - The C++ implementation has been through 3-4
>> >> rounds
>> >> >>> of
>> >> >>> > > > review.
>> >> >>> > > > > > >>> > - The Java implementation has been tracking
>> well
>> >> and
>> >> >>> has
>> >> >>> > > had
>> >> >>> > > > > > >>> initial
>> >> >>> > > > > > >>> > reviews from contributors, with no major
>> >> remaining
>> >> >>> > > issues.
>> >> >>> > > > > > >>> > - Cross-language compatibility tests are
>> passing:
>> >> the
>> >> >>> > Arrow
>> >> >>> > > > C++
>> >> >>> > > > > > >>> decoder
>> >> >>> > > > > > >>> > reads Java-written data bit-exactly across
>> >> ~1.56M
>> >> >>> > values
>> >> >>> > > > and
>> >> >>> > > > > 18
>> >> >>> > > > > > >>> > fixtures, covering V1 and V2 pages, multiple
>> >> vector
>> >> >>> > > sizes,
>> >> >>> > > > > and
>> >> >>> > > > > > >>> > several
>> >> >>> > > > > > >>> > real datasets — zero mismatches.
>> >> >>> > > > > > >>> >
>> >> >>> > > > > > >>> > As raised in the java-pr we will be adding
>> coverage
>> >> for
>> >> >>> > > extreme
>> >> >>> > > > > > values
>> >> >>> > > > > > >>> > (those requiring
>> >> >>> > > > > > >>> > 63-64 bits after FOR is applied) before we close a
>> >> vote.
>> >> >>> > We'll
>> >> >>> > > > aim
>> >> >>> > > > > > to
>> >> >>> > > > > > >>> get
>> >> >>> > > > > > >>> > that
>> >> >>> > > > > > >>> > done in parallel.
>> >> >>> > > > > > >>> >
>> >> >>> > > > > > >>> > Plan: unless there are objections, I intend to
>> start a
>> >> >>> vote
>> >> >>> > at
>> >> >>> > > > the
>> >> >>> > > > > > end
>> >> >>> > > > > > >>> of
>> >> >>> > > > > > >>> > this week/early
>> >> >>> > > > > > >>> > next week.
>> >> >>> > > > > > >>> >
>> >> >>> > > > > > >>> > Thanks
>> >> >>> > > > > > >>> > Prateek
>> >> >>> > > > > > >>> >
>> >> >>> > > > > > >>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <
>> >> >>> > > prtkgaur@gmail.com>
>> >> >>> > > > > > >>> wrote:
>> >> >>> > > > > > >>> >
>> >> >>> > > > > > >>> > > Hi Team,
>> >> >>> > > > > > >>> > >
>> >> >>> > > > > > >>> > > Just wanted to provide some updates on ALP.
>> >> >>> > > > > > >>> > > Micah and I have done 3-4 rounds of review for
>> the
>> >> c++
>> >> >>> PR :
>> >> >>> > > > > > >>> > > https://github.com/apache/arrow/pull/48345
>> >> >>> > > > > > >>> > >
>> >> >>> > > > > > >>> > > For the Java implementation Vinoo has been
>> working
>> >> on
>> >> >>> > > following
>> >> >>> > > > > PR
>> >> >>> > > > > > :
>> >> >>> > > > > > >>> > > https://github.com/apache/parquet-java/pull/3397
>> >> >>> > > > > > >>> > >
>> >> >>> > > > > > >>> > > Best
>> >> >>> > > > > > >>> > > Prateek
>> >> >>> > > > > > >>> > >
>> >> >>> > > > > > >>> > > On Tue, May 5, 2026 at 1:50 PM Micah Kornfield <
>> >> >>> > > > > > >>> emkornfield@gmail.com>
>> >> >>> > > > > > >>> > > wrote:
>> >> >>> > > > > > >>> > >
>> >> >>> > > > > > >>> > >> Hi Antoine,
>> >> >>> > > > > > >>> > >>
>> >> >>> > > > > > >>> > >> > Apologies if the question was already asked,
>> but
>> >> >>> should
>> >> >>> > we
>> >> >>> > > > > care
>> >> >>> > > > > > >>> about
>> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more
>> >> efficient
>> >> >>> than
>> >> >>> > > > > FLOAT16
>> >> >>> > > > > > +
>> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
>> >> >>> > > > > > >>> > >>
>> >> >>> > > > > > >>> > >> It was. We thought we could defer it for the
>> >> >>> following
>> >> >>> > > > reasons:
>> >> >>> > > > > > >>> > >> 1. It's not clear there are a lot of easy
>> >> reference
>> >> >>> > > datasets
>> >> >>> > > > to
>> >> >>> > > > > > >>> test
>> >> >>> > > > > > >>> > its
>> >> >>> > > > > > >>> > >> effectiveness. It does look like there might be
>> >> one
>> >> >>> or
>> >> >>> > two
>> >> >>> > > on
>> >> >>> > > > > > >>> > huggingface
>> >> >>> > > > > > >>> > >> (e.g.
>> >> https://huggingface.co/datasets/kikitora/curdie
>> >> >>> ).
>> >> >>> > > > > > >>> > >> 2. It seemed likely that float 16 was more
>> likely
>> >> >>> used
>> >> >>> > for
>> >> >>> > > > > values
>> >> >>> > > > > > >>> that
>> >> >>> > > > > > >>> > >> were less likely to reduce to decimal values.
>> >> >>> > > > > > >>> > >> 3. It could be added as an extension later if
>> >> needed.
>> >> >>> > > > > > >>> > >>
>> >> >>> > > > > > >>> > >> Cheers,
>> >> >>> > > > > > >>> > >> Micah
>> >> >>> > > > > > >>> > >>
>> >> >>> > > > > > >>> > >>
>> >> >>> > > > > > >>> > >> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <
>> >> >>> > > > > antoine@python.org
>> >> >>> > > > > > >
>> >> >>> > > > > > >>> > wrote:
>> >> >>> > > > > > >>> > >>
>> >> >>> > > > > > >>> > >> >
>> >> >>> > > > > > >>> > >> > Hello,
>> >> >>> > > > > > >>> > >> >
>> >> >>> > > > > > >>> > >> > Apologies if the question was already asked,
>> but
>> >> >>> should
>> >> >>> > we
>> >> >>> > > > > care
>> >> >>> > > > > > >>> about
>> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more
>> >> efficient
>> >> >>> than
>> >> >>> > > > > FLOAT16
>> >> >>> > > > > > +
>> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
>> >> >>> > > > > > >>> > >> >
>> >> >>> > > > > > >>> > >> > Regards
>> >> >>> > > > > > >>> > >> >
>> >> >>> > > > > > >>> > >> > Antoine.
>> >> >>> > > > > > >>> > >> >
>> >> >>> > > > > > >>> > >> >
>> >> >>> > > > > > >>> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
>> >> >>> > > > > > >>> > >> > > Thanks Andrew and Micah for review feedback
>> on
>> >> >>> the two
>> >> >>> > > > PR's
>> >> >>> > > > > > >>> > >> > > 1) (c++ arrow repo)
>> >> >>> > > > > > >>> > >>
>> https://github.com/apache/arrow/pull/48345/changes
>> >> >>> > > > > > >>> > >> > > 2) (parquet-format repo)
>> >> >>> > > > > > >>> > >> >
>> >> https://github.com/apache/parquet-format/pull/557
>> >> >>> > > > > > >>> > >> > >
>> >> >>> > > > > > >>> > >> > > I have addressed all (unless I missed
>> >> something)
>> >> >>> > > comments
>> >> >>> > > > on
>> >> >>> > > > > > >>> the two
>> >> >>> > > > > > >>> > >> > PR's.
>> >> >>> > > > > > >>> > >> > >
>> >> >>> > > > > > >>> > >> > > Best
>> >> >>> > > > > > >>> > >> > > Prateek
>> >> >>> > > > > > >>> > >> > >
>> >> >>> > > > > > >>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK
>> GAUR <
>> >> >>> > > > > > >>> prtkgaur@gmail.com>
>> >> >>> > > > > > >>> > >> wrote:
>> >> >>> > > > > > >>> > >> > >
>> >> >>> > > > > > >>> > >> > >> Thanks Andrew and Micah.
>> >> >>> > > > > > >>> > >> > >>
>> >> >>> > > > > > >>> > >> > >> `fair amount of feedback on at least the
>> >> >>> > > implementations`
>> >> >>> > > > > > >>> > >> > >> For the c++ I have already started
>> addressing
>> >> the
>> >> >>> > > > > feedback, I
>> >> >>> > > > > > >>> > should
>> >> >>> > > > > > >>> > >> be
>> >> >>> > > > > > >>> > >> > >> done with that Monday/Tuesday.
>> >> >>> > > > > > >>> > >> > >> I think Vinoo too has been making good
>> >> progress
>> >> >>> on
>> >> >>> > the
>> >> >>> > > > Java
>> >> >>> > > > > > >>> > >> > implementation.
>> >> >>> > > > > > >>> > >> > >>
>> >> >>> > > > > > >>> > >> > >> Best
>> >> >>> > > > > > >>> > >> > >> Prateek
>> >> >>> > > > > > >>> > >> > >>
>> >> >>> > > > > > >>> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew
>> Lamb <
>> >> >>> > > > > > >>> > andrewlamb11@gmail.com
>> >> >>> > > > > > >>> > >> >
>> >> >>> > > > > > >>> > >> > >> wrote:
>> >> >>> > > > > > >>> > >> > >>
>> >> >>> > > > > > >>> > >> > >>> Got it. Thank you for the clarification
>> -- I
>> >> >>> will
>> >> >>> > try
>> >> >>> > > > and
>> >> >>> > > > > > look
>> >> >>> > > > > > >>> > into
>> >> >>> > > > > > >>> > >> the
>> >> >>> > > > > > >>> > >> > >>> spec and the Rust implementation[1] in
>> this
>> >> next
>> >> >>> > week
>> >> >>> > > > > > >>> > >> > >>>
>> >> >>> > > > > > >>> > >> > >>> [1]:
>> >> >>> https://github.com/apache/arrow-rs/pull/9372
>> >> >>> > > > > > >>> > >> > >>>
>> >> >>> > > > > > >>> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah
>> >> >>> Kornfield <
>> >> >>> > > > > > >>> > >> > emkornfield@gmail.com>
>> >> >>> > > > > > >>> > >> > >>> wrote:
>> >> >>> > > > > > >>> > >> > >>>
>> >> >>> > > > > > >>> > >> > >>>> Hi Andrew,
>> >> >>> > > > > > >>> > >> > >>>> I think there is a fair amount of
>> feedback
>> >> on
>> >> >>> at
>> >> >>> > > least
>> >> >>> > > > > the
>> >> >>> > > > > > >>> > >> > >>>> implementations, typically I think we've
>> >> waited
>> >> >>> > till
>> >> >>> > > > they
>> >> >>> > > > > > are
>> >> >>> > > > > > >>> > >> close to
>> >> >>> > > > > > >>> > >> > >>>> mergeable before a final vote.
>> Otherwise I
>> >> >>> agree
>> >> >>> > we
>> >> >>> > > > are
>> >> >>> > > > > > very
>> >> >>> > > > > > >>> > >> close.
>> >> >>> > > > > > >>> > >> > >>>>
>> >> >>> > > > > > >>> > >> > >>>> -Micah
>> >> >>> > > > > > >>> > >> > >>>>
>> >> >>> > > > > > >>> > >> > >>>> On Saturday, April 25, 2026, Andrew Lamb
>> <
>> >> >>> > > > > > >>> andrewlamb11@gmail.com
>> >> >>> > > > > > >>> > >
>> >> >>> > > > > > >>> > >> > wrote:
>> >> >>> > > > > > >>> > >> > >>>>
>> >> >>> > > > > > >>> > >> > >>>>> Thanks Prateek,
>> >> >>> > > > > > >>> > >> > >>>>>
>> >> >>> > > > > > >>> > >> > >>>>> I think from this content it looks to me
>> >> like
>> >> >>> we
>> >> >>> > are
>> >> >>> > > > > ready
>> >> >>> > > > > > >>> to
>> >> >>> > > > > > >>> > >> start a
>> >> >>> > > > > > >>> > >> > >>>>> vote to explicitly accept ALP into
>> Parquet
>> >> >>> > > > > > >>> > >> > >>>>>
>> >> >>> > > > > > >>> > >> > >>>>> Does anyone know of a reason we should
>> >> >>> postpone it
>> >> >>> > > for
>> >> >>> > > > > > >>> longer?
>> >> >>> > > > > > >>> > >> > >>>>> Perhaps someone needs some more time to
>> >> >>> review?
>> >> >>> > > > > > >>> > >> > >>>>>
>> >> >>> > > > > > >>> > >> > >>>>> Andrew
>> >> >>> > > > > > >>> > >> > >>>>>
>> >> >>> > > > > > >>> > >> > >>>>>
>> >> >>> > > > > > >>> > >> > >>>>>
>> >> >>> > > > > > >>> > >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK
>> >> GAUR <
>> >> >>> > > > > > >>> > prtkgaur@gmail.com>
>> >> >>> > > > > > >>> > >> > >>>>> wrote:
>> >> >>> > > > > > >>> > >> > >>>>>
>> >> >>> > > > > > >>> > >> > >>>>>> Hi team,
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>> Hope everyone is doing well. I got a
>> >> chance
>> >> >>> to
>> >> >>> > work
>> >> >>> > > > > > >>> through all
>> >> >>> > > > > > >>> > >> the
>> >> >>> > > > > > >>> > >> > >>>>>> remaining feedback and update the spec
>> >> doc.
>> >> >>> Here
>> >> >>> > > are
>> >> >>> > > > > the
>> >> >>> > > > > > >>> new
>> >> >>> > > > > > >>> > >> > artifacts
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>> 1) Spec document :
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> >
>> >> >>> > > > > > >>> > >>
>> >> >>> > > > > > >>> >
>> >> >>> > > > > > >>>
>> >> >>> > > > > >
>> >> >>> > > > >
>> >> >>> > > >
>> >> >>> > >
>> >> >>> >
>> >> >>>
>> >>
>> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>> 2) Spec document in parquet format
>> repo :
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > https://github.com/apache/parquet-format/pull/557
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>> 3) Alp implementation in arrow c++
>> repo :
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > https://github.com/apache/arrow/pull/48345/changes
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>> 4) Alp implementation in parquet-java
>> >> repo :
>> >> >>> Work
>> >> >>> > > for
>> >> >>> > > > > > >>> Vinoo and
>> >> >>> > > > > > >>> > >> > Julien
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > https://github.com/apache/parquet-java/pull/3397
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>> 5) PR with test and benchmarking
>> >> artifacts in
>> >> >>> > > > > > >>> parquet-testing
>> >> >>> > > > > > >>> > >> repo :
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > https://github.com/apache/parquet-testing/pull/100
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>> And
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>> - Go : Arnav just submitted an in
>> >> >>> progress
>> >> >>> > > > > > >>> implementation
>> >> >>> > > > > > >>> > in
>> >> >>> > > > > > >>> > >> Go.
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> https://github.com/apache/arrow-go/pull/704
>> >> >>> > (I
>> >> >>> > > > > > haven't
>> >> >>> > > > > > >>> > >> started
>> >> >>> > > > > > >>> > >> > >>>>>> looking at it yet)
>> >> >>> > > > > > >>> > >> > >>>>>> - Rust : I remember Andrew
>> mentioned
>> >> that
>> >> >>> > this
>> >> >>> > > > work
>> >> >>> > > > > > is
>> >> >>> > > > > > >>> also
>> >> >>> > > > > > >>> > >> in
>> >> >>> > > > > > >>> > >> > >>>>>> progress (So 4 languages!)
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>> *Arrow C++ implementation *
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>> The PR is out and was also used by
>> >> Antoine to
>> >> >>> > > report
>> >> >>> > > > > the
>> >> >>> > > > > > >>> > numbers
>> >> >>> > > > > > >>> > >> as
>> >> >>> > > > > > >>> > >> > >>>>>> reported here. Micah and Konstantin
>> have
>> >> >>> given 1
>> >> >>> > > > round
>> >> >>> > > > > of
>> >> >>> > > > > > >>> > >> feedback
>> >> >>> > > > > > >>> > >> > >>>>>> and I'm addressing them today. Please
>> note
>> >> >>> that
>> >> >>> > the
>> >> >>> > > > > > default
>> >> >>> > > > > > >>> > >> > >>>>>> optimization flag for compiling is O2
>> and
>> >> not
>> >> >>> > Q3. I
>> >> >>> > > > got
>> >> >>> > > > > > >>> around
>> >> >>> > > > > > >>> > >> 70%
>> >> >>> > > > > > >>> > >> > >>>>>> performance improvement in the decoding
>> >> speed
>> >> >>> > when
>> >> >>> > > > > using
>> >> >>> > > > > > >>> the O3
>> >> >>> > > > > > >>> > >> > flag.
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>> *Parqet-MR Java implementation (working
>> >> with
>> >> >>> > Vinoo
>> >> >>> > > > and
>> >> >>> > > > > > >>> Julien)
>> >> >>> > > > > > >>> > >> and
>> >> >>> > > > > > >>> > >> > **Cross
>> >> >>> > > > > > >>> > >> > >>>>>> Language testing*
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>> Let me know if you have any
>> questions
>> >> or
>> >> >>> > > > feedback.
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>> Now pasting some performance numbers
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>> Table 1: C++ ALP Double Decode —
>> >> Spotify
>> >> >>> > Columns
>> >> >>> > > > > > >>> (Graviton
>> >> >>> > > > > > >>> > 3,
>> >> >>> > > > > > >>> > >> ARM
>> >> >>> > > > > > >>> > >> > >>>>>> Neoverse V1)
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>>
>> >> >>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>> │ Column │ -O2 (MB/s) │
>> >> -O3
>> >> >>> > > (MB/s) │
>> >> >>> > > > > > >>> Speedup │
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>>
>> >> >>> ├──────────────────┼──────────────┼──────────────┼─────────┤
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>> │ valence │ 3,155 │
>> >> >>> 5,523
>> >> >>> > > │
>> >> >>> > > > > > >>> 1.75x │
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>> │ danceability │ 3,233 │
>> >> >>> 5,685
>> >> >>> > > │
>> >> >>> > > > > > >>> 1.76x │
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>> │ energy │ 3,197 │
>> >> >>> 5,652
>> >> >>> > > │
>> >> >>> > > > > > >>> 1.77x │
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>> │ loudness │ 3,186 │
>> >> >>> 5,473
>> >> >>> > > │
>> >> >>> > > > > > >>> 1.72x │
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>>
>> >> >>> └──────────────────┴──────────────┴──────────────┴─────────┘
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK
>> >> GAUR
>> >> >>> <
>> >> >>> > > > > > >>> > prtkgaur@gmail.com
>> >> >>> > > > > > >>> > >> >
>> >> >>> > > > > > >>> > >> > >>>>>> wrote:
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>> @Micah Kornfield <
>> emkornfield@gmail.com>
>> >> :
>> >> >>> Got
>> >> >>> > > it.
>> >> >>> > > > > > >>> > >> > >>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>> @Andrew Lamb <an...@gmail.com>
>> >> >>> > > > > > >>> > >> > >>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>> Do you think it would be good to
>> start
>> >> >>> moving
>> >> >>> > the
>> >> >>> > > > > spec
>> >> >>> > > > > > >>> > >> development
>> >> >>> > > > > > >>> > >> > >>>>>>>> into
>> >> >>> > > > > > >>> > >> > >>>>>>>> markdown format, in preparation for
>> >> >>> finalizing
>> >> >>> > > it?
>> >> >>> > > > > > >>> > >> > >>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>> Yes I'll update the numbers for some
>> of
>> >> the
>> >> >>> > > > examples I
>> >> >>> > > > > > >>> have in
>> >> >>> > > > > > >>> > >> the
>> >> >>> > > > > > >>> > >> > >>>>>>> spec based
>> >> >>> > > > > > >>> > >> > >>>>>>> on the updated header size. Then we
>> >> should
>> >> >>> be
>> >> >>> > good
>> >> >>> > > > to
>> >> >>> > > > > go
>> >> >>> > > > > > >>> for
>> >> >>> > > > > > >>> > the
>> >> >>> > > > > > >>> > >> > >>>>>>> markdown format.
>> >> >>> > > > > > >>> > >> > >>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>> Thanks everyone!
>> >> >>> > > > > > >>> > >> > >>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>> Andrew
>> >> >>> > > > > > >>> > >> > >>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM
>> PRATEEK
>> >> >>> GAUR <
>> >> >>> > > > > > >>> > >> prtkgaur@gmail.com>
>> >> >>> > > > > > >>> > >> > >>>>>>>> wrote:
>> >> >>> > > > > > >>> > >> > >>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>> Hi team,
>> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>> 1) Andrew
>> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for working on test
>> files.
>> >> >>> My PR
>> >> >>> > > did
>> >> >>> > > > > add
>> >> >>> > > > > > >>> all
>> >> >>> > > > > > >>> > the
>> >> >>> > > > > > >>> > >> > test
>> >> >>> > > > > > >>> > >> > >>>>>>>> files I
>> >> >>> > > > > > >>> > >> > >>>>>>>>> used to benchmark on datasets.
>> >> Maybe
>> >> >>> we
>> >> >>> > can
>> >> >>> > > > club
>> >> >>> > > > > > it
>> >> >>> > > > > > >>> > >> together.
>> >> >>> > > > > > >>> > >> > >>>>>>>> WIll also
>> >> >>> > > > > > >>> > >> > >>>>>>>>> aid
>> >> >>> > > > > > >>> > >> > >>>>>>>>> cross language testing
>> >> >>> > > > > > >>> > >> > >>>>>>>>> - Kosta Tarasov working on Rust
>> >> >>> > > > implementation.
>> >> >>> > > > > > >>> This is
>> >> >>> > > > > > >>> > >> > great.
>> >> >>> > > > > > >>> > >> > >>>>>>>> Thanks
>> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>> 2) Antoine
>> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks a lot for reporting the
>> >> >>> numbers
>> >> >>> > on
>> >> >>> > > > AMD.
>> >> >>> > > > > > >>> Looks
>> >> >>> > > > > > >>> > >> like
>> >> >>> > > > > > >>> > >> > you
>> >> >>> > > > > > >>> > >> > >>>>>>>> are
>> >> >>> > > > > > >>> > >> > >>>>>>>>> getting 8X the decoding
>> >> performance of
>> >> >>> > BSS.
>> >> >>> > > > This
>> >> >>> > > > > > is
>> >> >>> > > > > > >>> > >> > amazing!!.
>> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for acknowledging the
>> >> >>> sampling
>> >> >>> > > > design.
>> >> >>> > > > > > >>> > >> > >>>>>>>>> - I agree with you on
>> Fastlanes. In
>> >> >>> some
>> >> >>> > > crude
>> >> >>> > > > > > >>> > >> experiments I
>> >> >>> > > > > > >>> > >> > >>>>>>>> didn't get
>> >> >>> > > > > > >>> > >> > >>>>>>>>> a good perf benefit from it on
>> >> >>> Graviton3
>> >> >>> > > (but
>> >> >>> > > > > > maybe
>> >> >>> > > > > > >>> > there
>> >> >>> > > > > > >>> > >> was
>> >> >>> > > > > > >>> > >> > >>>>>>>> something
>> >> >>> > > > > > >>> > >> > >>>>>>>>> wrong with my implementation).
>> >> >>> > > > > > >>> > >> > >>>>>>>>> - Locking the 16bit exception
>> >> >>> encoding for
>> >> >>> > > the
>> >> >>> > > > > > spec
>> >> >>> > > > > > >>> in
>> >> >>> > > > > > >>> > >> this
>> >> >>> > > > > > >>> > >> > >>>>>>>> case.
>> >> >>> > > > > > >>> > >> > >>>>>>>>> - Awesome I think we have solved
>> >> for
>> >> >>> all
>> >> >>> > > open
>> >> >>> > > > > > >>> questions
>> >> >>> > > > > > >>> > >> minus
>> >> >>> > > > > > >>> > >> > >>>>>>>> the
>> >> >>> > > > > > >>> > >> > >>>>>>>>> version byte :). (will get back
>> on
>> >> >>> this
>> >> >>> > > soon)
>> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>> 3) Micah
>> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>> - FastLanes : The current spec
>> does
>> >> >>> allow
>> >> >>> > > for
>> >> >>> > > > > > using
>> >> >>> > > > > > >>> > >> FastLane
>> >> >>> > > > > > >>> > >> > >>>>>>>> with the
>> >> >>> > > > > > >>> > >> > >>>>>>>>> configurable enum value for
>> >> layout. We
>> >> >>> > > should
>> >> >>> > > > be
>> >> >>> > > > > > >>> able to
>> >> >>> > > > > > >>> > >> > inject
>> >> >>> > > > > > >>> > >> > >>>>>>>> any
>> >> >>> > > > > > >>> > >> > >>>>>>>>> layout
>> >> >>> > > > > > >>> > >> > >>>>>>>>> in the current design.
>> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>> Working on resolving all remaining
>> open
>> >> >>> > comments
>> >> >>> > > > on
>> >> >>> > > > > > the
>> >> >>> > > > > > >>> spec
>> >> >>> > > > > > >>> > >> this
>> >> >>> > > > > > >>> > >> > >>>>>>>> week.
>> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>> Best
>> >> >>> > > > > > >>> > >> > >>>>>>>>> Prateek
>> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM
>> Steve
>> >> >>> > Loughran <
>> >> >>> > > > > > >>> > >> > >>>>>>>> stevel@cloudera.com>
>> >> >>> > > > > > >>> > >> > >>>>>>>>> wrote:
>> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah
>> >> >>> Kornfield
>> >> >>> > <
>> >> >>> > > > > > >>> > >> > >>>>>>>> emkornfield@gmail.com>
>> >> >>> > > > > > >>> > >> > >>>>>>>>>> wrote:
>> >> >>> > > > > > >>> > >> > >>>>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>>>> It looks like the actual issue
>> >> >>> described for
>> >> >>> > > ORC
>> >> >>> > > > > in
>> >> >>> > > > > > >>> the
>> >> >>> > > > > > >>> > >> paper
>> >> >>> > > > > > >>> > >> > >>>>>>>> is that
>> >> >>> > > > > > >>> > >> > >>>>>>>>> it
>> >> >>> > > > > > >>> > >> > >>>>>>>>>>> has multiple sub-encodings in a
>> >> batch.
>> >> >>> This
>> >> >>> > > is
>> >> >>> > > > > > >>> different
>> >> >>> > > > > > >>> > >> then
>> >> >>> > > > > > >>> > >> > >>>>>>>> the
>> >> >>> > > > > > >>> > >> > >>>>>>>>> design
>> >> >>> > > > > > >>> > >> > >>>>>>>>>>> proposed here where there is still
>> >> fixed
>> >> >>> > > > encoding
>> >> >>> > > > > > per
>> >> >>> > > > > > >>> page
>> >> >>> > > > > > >>> > >> in
>> >> >>> > > > > > >>> > >> > >>>>>>>> parquet.
>> >> >>> > > > > > >>> > >> > >>>>>>>>>>> Given reasonably sized pages I
>> don't
>> >> >>> think
>> >> >>> > > > branch
>> >> >>> > > > > > >>> > >> > >>>>>>>> misprediction should
>> >> >>> > > > > > >>> > >> > >>>>>>>>>> be a
>> >> >>> > > > > > >>> > >> > >>>>>>>>>>> big issue for new encodings. I
>> agree
>> >> >>> that
>> >> >>> > we
>> >> >>> > > > > should
>> >> >>> > > > > > >>> be
>> >> >>> > > > > > >>> > >> > >>>>>>>> conservative in
>> >> >>> > > > > > >>> > >> > >>>>>>>>>>> general for adding new encodings.
>> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>>> +1
>> >> >>> > > > > > >>> > >> > >>>>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>
>> >> >>> > > > > > >>> > >> > >
>> >> >>> > > > > > >>> > >> >
>> >> >>> > > > > > >>> > >> >
>> >> >>> > > > > > >>> > >> >
>> >> >>> > > > > > >>> > >>
>> >> >>> > > > > > >>> > >
>> >> >>> > > > > > >>> >
>> >> >>> > > > > > >>>
>> >> >>> > > > > > >>
>> >> >>> > > > > >
>> >> >>> > > > >
>> >> >>> > > >
>> >> >>> > >
>> >> >>> >
>> >> >>>
>> >> >>
>> >>
>> >
>>
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by PRATEEK GAUR <pr...@gmail.com>.
Thanks Divjot for bringing it up.
I was under the same impression as Russell and Andrew that we'd have it
flag gated.
With the new versioning proposal 'preview' might be a good tag for it.
Best
Prateek
On Sat, Aug 1, 2026 at 12:39 AM Andrew Lamb <an...@gmail.com> wrote:
> Div: Here is a proposal to mark ALP in Preview: [1] (using a comment in the
> spec section).
>
> Russel: That is my understanding as well.
>
> Hopefully we are able to formalize the versioning process shortly so we can
> formalize the notion of "preview"
>
> Andrew
>
>
> [1]: https://github.com/apache/parquet-format/pull/604
>
> On Fri, Jul 31, 2026 at 11:15 AM Russell Spitzer <
> russell.spitzer@gmail.com>
> wrote:
>
> > I thought we had agreed it would be merged, and features like this would
> > be feature flag gated. So a V-Old writer could always opt in to using
> this
> > encoding but that must be a conscious effort if you are not using VNext
> of
> > Parquet. If you are on V-Next the writer can choose to use ALP whenever
> it
> > likes without user intervention.
> >
> > On Fri, Jul 31, 2026 at 2:20 AM Divjot Arora via dev <
> > dev@parquet.apache.org> wrote:
> >
> >> Yes, that’s my understanding as well. However, as you noted, the
> >> versioning
> >> discussion is still in early stages and hasn’t prescribed concrete
> >> mechanisms yet. If ALP goes into parquet-format 2.14.0, there’s no
> >> indication that it’s in preview or that writers shouldn’t use it by
> >> default
> >> yet. Given that we’ve held off on merging other incompatible changes
> (e.g
> >> removal of path_in_schema) until the versioning discussion closes, my
> >> preference would be to leave ALP un-merged for the time being as well.
> >>
> >> — Div
> >>
> >> On Fri, Jul 31, 2026 at 08:59 Andrew Lamb <an...@gmail.com>
> wrote:
> >>
> >> > After re-reading Julien's Versioning Proposal, my understanding is
> that
> >> if
> >> > we adopt that system, we will somehow mark ALP in "Preview" in
> >> > parquet-format for release Version 3.
> >> >
> >> > The versioning document doesn't seem to proscribe the particular
> method
> >> > for annotating "preview" features -- I think clearly marking them as
> >> such
> >> > in parquet.thrift as well as reflecting that on the
> >> > https://parquet.apache.org/docs/file-format/versions/ page somehow
> >> would
> >> > be adequate.
> >> >
> >> > Andrew
> >> >
> >> > On Thu, Jul 30, 2026 at 11:50 AM Andrew Lamb <an...@gmail.com>
> >> > wrote:
> >> >
> >> >> In my mind ALP is in the same position as other encodings that were
> >> added
> >> >> recently[1] (such as BYTE_STREAM_SPLIT for non floating point types)
> >> >>
> >> >> That is to say it will be available in the latest open source writers
> >> and
> >> >> readers, but realistically not widely used in the ecosystem where
> >> >> compatibility is a concern (a large part)
> >> >>
> >> >> I view Julian's work[2][3] to gather consensus on how to name
> >> >> combinations of features (e.g. "versions") as the mechanism by which
> we
> >> >> will accelerate the adoption of this change (along with other forward
> >> >> incompatible changes) across the ecosystem.
> >> >>
> >> >> > Will we exclude this change when cutting parquet-format v2.14.0?
> >> >>
> >> >> No I don't think so -- I would expect it to go out in the next
> >> >> parquet-format version as do other changes in the spec.
> >> >>
> >> >> Andrew
> >> >>
> >> >> [1]: https://parquet.apache.org/docs/file-format/versions/
> >> >> [2]:
> https://lists.apache.org/thread/xmnj8h0h8ozmrgox5tydhhs7yz7q2hss
> >> >> [3]:
> https://lists.apache.org/thread/qlf8lg90gqq41lllyy9mk2f0skqvftqj
> >> >>
> >> >>
> >> >> On Thu, Jul 30, 2026 at 3:51 AM Divjot Arora via dev <
> >> >> dev@parquet.apache.org> wrote:
> >> >>
> >> >>> Hi Andrew,
> >> >>>
> >> >>> How does this interact with the ongoing versioning discussion? Given
> >> that
> >> >>> new encodings are not forward compatible, I would assume ALP is
> >> included
> >> >>> in
> >> >>> Parquet V<next> (V3?), but it's not clear to me how that works if
> it's
> >> >>> merged to the parquet-format master branch now. Will we exclude this
> >> >>> change
> >> >>> when cutting parquet-format v2.14.0?
> >> >>>
> >> >>> -- Div
> >> >>>
> >> >>> On Thu, Jul 30, 2026 at 8:51 AM Andrew Lamb <andrewlamb11@gmail.com
> >
> >> >>> wrote:
> >> >>>
> >> >>> > Hello,
> >> >>> >
> >> >>> > As an update, thanks to the epic work from many contributors,
> >> >>> especially
> >> >>> > Prateek, I just merged the parquet-format change for ALP[1]. We
> >> plan to
> >> >>> > make additional editorial The remaining items I know of are
> merging
> >> the
> >> >>> > examples from parquet-testing[2], and then of course the various
> >> >>> > implementations
> >> >>> >
> >> >>> > Thanks again everyone. This is a great step forward
> >> >>> > Andrew
> >> >>> >
> >> >>> > [1]: https://github.com/apache/parquet-format/pull/557
> >> >>> > [2]: https://github.com/apache/parquet-testing/pull/100
> >> >>> >
> >> >>> > On Tue, Jul 21, 2026 at 4:39 PM Julien Le Dem <ju...@apache.org>
> >> >>> wrote:
> >> >>> >
> >> >>> > > It was also in my spam folder... :(
> >> >>> > > I've seen a bunch of apache list emails go to my spam recently.
> >> >>> > >
> >> >>> > > On Tue, Jul 21, 2026 at 10:58 AM Russell Spitzer <
> >> >>> > > russell.spitzer@gmail.com>
> >> >>> > > wrote:
> >> >>> > >
> >> >>> > > > Yes, my hope was that my vote would kick it out of spam but we
> >> >>> > definitely
> >> >>> > > > had two other votes that also went to spam for me.
> >> >>> > > >
> >> >>> > > > For anyone looking the thread is here
> >> >>> > > >
> >> https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt
> >> >>> > > >
> >> >>> > > > On Tue, Jul 21, 2026 at 10:57 AM Micah Kornfield <
> >> >>> > emkornfield@gmail.com>
> >> >>> > > > wrote:
> >> >>> > > >
> >> >>> > > > > I've seen other's votes not go to spam, so hopefully people
> >> will
> >> >>> see
> >> >>> > it
> >> >>> > > > in
> >> >>> > > > > there inboxes (or see this thread and look in there spam).
> >> >>> > > > >
> >> >>> > > > > On Tue, Jul 21, 2026 at 10:52 AM PRATEEK GAUR <
> >> >>> prtkgaur@gmail.com>
> >> >>> > > > wrote:
> >> >>> > > > >
> >> >>> > > > > > Okay,
> >> >>> > > > > >
> >> >>> > > > > > I sent out the voting email yesterday and so far I've
> heard
> >> >>> from 3
> >> >>> > > > people
> >> >>> > > > > > that it ended up in their "spam folder". Dev VPN to be
> >> blamed
> >> >>> for
> >> >>> > > it?.
> >> >>> > > > > >
> >> >>> > > > > > What's the protocol here?
> >> >>> > > > > > I let people hunt for the email in their spam folder or
> send
> >> >>> out
> >> >>> > > > another
> >> >>> > > > > > one?
> >> >>> > > > > >
> >> >>> > > > > > Best
> >> >>> > > > > > Prateek
> >> >>> > > > > >
> >> >>> > > > > > On Sun, Jul 19, 2026 at 10:47 PM PRATEEK GAUR <
> >> >>> prtkgaur@gmail.com>
> >> >>> > > > > wrote:
> >> >>> > > > > >
> >> >>> > > > > > > Hi team,
> >> >>> > > > > > >
> >> >>> > > > > > > Vinoo has added the tests which were requested in the
> last
> >> >>> round
> >> >>> > of
> >> >>> > > > > > review.
> >> >>> > > > > > > In his own words.
> >> >>> > > > > > >
> >> >>> > > > > > > "
> >> >>> > > > > > > The extreme value tests Micah wanted are done and
> pushed.
> >> I
> >> >>> added
> >> >>> > > > > > coverage
> >> >>> > > > > > > for values that need the full FOR bit width after the
> >> frame
> >> >>> of
> >> >>> > > > > reference
> >> >>> > > > > > is
> >> >>> > > > > > > applied: 64-bit (the case where the signed max minus min
> >> >>> > > subtraction
> >> >>> > > > > > > overflows), 63-bit (the non-overflow case), and 32-bit
> for
> >> >>> > floats.
> >> >>> > > > All
> >> >>> > > > > > > lossless with zero exceptions. I also verified them
> >> through
> >> >>> Arrow
> >> >>> > > C++
> >> >>> > > > > ALP
> >> >>> > > > > > > reader (the local build), so the Java-written extreme
> >> columns
> >> >>> > > decode
> >> >>> > > > > > > correctly in C++, bit-exact against the expected
> values."
> >> >>> > > > > > >
> >> >>> > > > > > > State of the PR (#3397):
> >> >>> > > > > > >
> >> >>> > > > > > > - Merges cleanly with master (I pulled master in and
> >> >>> resolved
> >> >>> > > the
> >> >>> > > > > > > conflicts).
> >> >>> > > > > > > - 41 of 44 review threads resolved. The 3 remaining
> are
> >> >>> > > > intentional:
> >> >>> > > > > > > one deferred perf nit, one that stays open until
> >> >>> > parquet-format
> >> >>> > > > > ships
> >> >>> > > > > > ALP,
> >> >>> > > > > > >
> >> >>> > > > > > > "
> >> >>> > > > > > >
> >> >>> > > > > > > With this both the
> >> >>> > > > > > > 1) C++ PR :
> >> >>> https://github.com/apache/arrow/pull/48345/changes
> >> >>> > > > > > > 2) Java PR :
> >> >>> https://github.com/apache/parquet-java/pull/3397
> >> >>> > > > > > >
> >> >>> > > > > > > Are cross language tested and range of test scenarios.
> >> >>> > > > > > >
> >> >>> > > > > > > Best
> >> >>> > > > > > > Prateek
> >> >>> > > > > > >
> >> >>> > > > > > >
> >> >>> > > > > > > On Mon, Jul 13, 2026 at 8:38 AM PRATEEK GAUR <
> >> >>> prtkgaur@gmail.com
> >> >>> > >
> >> >>> > > > > wrote:
> >> >>> > > > > > >
> >> >>> > > > > > >> Thanks Matt,
> >> >>> > > > > > >>
> >> >>> > > > > > >> I'll be on it this week.
> >> >>> > > > > > >>
> >> >>> > > > > > >> Best
> >> >>> > > > > > >> Prateek
> >> >>> > > > > > >>
> >> >>> > > > > > >> On Mon, Jul 13, 2026 at 8:36 AM Matt Topol <
> >> >>> > > zotthewizard@gmail.com>
> >> >>> > > > > > >> wrote:
> >> >>> > > > > > >>
> >> >>> > > > > > >>> There is also the Go implementation submitted by
> Arnav (
> >> >>> > > > > > >>> https://github.com/apache/arrow-go/pull/704) which is
> >> >>> waiting
> >> >>> > > for
> >> >>> > > > > > >>> updates.
> >> >>> > > > > > >>>
> >> >>> > > > > > >>> Just wanted to make sure it didn't get lost here.
> >> >>> > > > > > >>>
> >> >>> > > > > > >>> --Matt
> >> >>> > > > > > >>>
> >> >>> > > > > > >>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK GAUR <
> >> >>> > > prtkgaur@gmail.com>
> >> >>> > > > > > >>> wrote:
> >> >>> > > > > > >>>
> >> >>> > > > > > >>> > Hi all,
> >> >>> > > > > > >>> >
> >> >>> > > > > > >>> > I'd like to share a status update on the ALP
> encoding
> >> >>> effort
> >> >>> > > and
> >> >>> > > > > get
> >> >>> > > > > > a
> >> >>> > > > > > >>> feel
> >> >>> > > > > > >>> > from the community before starting a formal vote.
> >> >>> > > > > > >>> >
> >> >>> > > > > > >>> > We now have the specification plus implementations
> in
> >> >>> both
> >> >>> > > > > languages:
> >> >>> > > > > > >>> >
> >> >>> > > > > > >>> > - Spec: apache/parquet-format#557 —
> GH-533
> >> >>> Add
> >> >>> > ALP
> >> >>> > > > > > encoding
> >> >>> > > > > > >>> > specification
> >> >>> > > > > > >>> > - C++ (Arrow): apache/arrow#48345 — GH-48701
> >> >>> > > [C++][Parquet]
> >> >>> > > > > Add
> >> >>> > > > > > >>> ALPpd
> >> >>> > > > > > >>> > encoding
> >> >>> > > > > > >>> > - Java: apache/parquet-java#3397 —
> Parquet
> >> >>> Java
> >> >>> > ALP
> >> >>> > > > > > >>> > Implementation (by Vinoo)
> >> >>> > > > > > >>> >
> >> >>> > > > > > >>> > Current state:
> >> >>> > > > > > >>> >
> >> >>> > > > > > >>> > - The C++ implementation has been through 3-4
> >> rounds
> >> >>> of
> >> >>> > > > review.
> >> >>> > > > > > >>> > - The Java implementation has been tracking well
> >> and
> >> >>> has
> >> >>> > > had
> >> >>> > > > > > >>> initial
> >> >>> > > > > > >>> > reviews from contributors, with no major
> >> remaining
> >> >>> > > issues.
> >> >>> > > > > > >>> > - Cross-language compatibility tests are
> passing:
> >> the
> >> >>> > Arrow
> >> >>> > > > C++
> >> >>> > > > > > >>> decoder
> >> >>> > > > > > >>> > reads Java-written data bit-exactly across
> >> ~1.56M
> >> >>> > values
> >> >>> > > > and
> >> >>> > > > > 18
> >> >>> > > > > > >>> > fixtures, covering V1 and V2 pages, multiple
> >> vector
> >> >>> > > sizes,
> >> >>> > > > > and
> >> >>> > > > > > >>> > several
> >> >>> > > > > > >>> > real datasets — zero mismatches.
> >> >>> > > > > > >>> >
> >> >>> > > > > > >>> > As raised in the java-pr we will be adding coverage
> >> for
> >> >>> > > extreme
> >> >>> > > > > > values
> >> >>> > > > > > >>> > (those requiring
> >> >>> > > > > > >>> > 63-64 bits after FOR is applied) before we close a
> >> vote.
> >> >>> > We'll
> >> >>> > > > aim
> >> >>> > > > > > to
> >> >>> > > > > > >>> get
> >> >>> > > > > > >>> > that
> >> >>> > > > > > >>> > done in parallel.
> >> >>> > > > > > >>> >
> >> >>> > > > > > >>> > Plan: unless there are objections, I intend to
> start a
> >> >>> vote
> >> >>> > at
> >> >>> > > > the
> >> >>> > > > > > end
> >> >>> > > > > > >>> of
> >> >>> > > > > > >>> > this week/early
> >> >>> > > > > > >>> > next week.
> >> >>> > > > > > >>> >
> >> >>> > > > > > >>> > Thanks
> >> >>> > > > > > >>> > Prateek
> >> >>> > > > > > >>> >
> >> >>> > > > > > >>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <
> >> >>> > > prtkgaur@gmail.com>
> >> >>> > > > > > >>> wrote:
> >> >>> > > > > > >>> >
> >> >>> > > > > > >>> > > Hi Team,
> >> >>> > > > > > >>> > >
> >> >>> > > > > > >>> > > Just wanted to provide some updates on ALP.
> >> >>> > > > > > >>> > > Micah and I have done 3-4 rounds of review for the
> >> c++
> >> >>> PR :
> >> >>> > > > > > >>> > > https://github.com/apache/arrow/pull/48345
> >> >>> > > > > > >>> > >
> >> >>> > > > > > >>> > > For the Java implementation Vinoo has been working
> >> on
> >> >>> > > following
> >> >>> > > > > PR
> >> >>> > > > > > :
> >> >>> > > > > > >>> > > https://github.com/apache/parquet-java/pull/3397
> >> >>> > > > > > >>> > >
> >> >>> > > > > > >>> > > Best
> >> >>> > > > > > >>> > > Prateek
> >> >>> > > > > > >>> > >
> >> >>> > > > > > >>> > > On Tue, May 5, 2026 at 1:50 PM Micah Kornfield <
> >> >>> > > > > > >>> emkornfield@gmail.com>
> >> >>> > > > > > >>> > > wrote:
> >> >>> > > > > > >>> > >
> >> >>> > > > > > >>> > >> Hi Antoine,
> >> >>> > > > > > >>> > >>
> >> >>> > > > > > >>> > >> > Apologies if the question was already asked,
> but
> >> >>> should
> >> >>> > we
> >> >>> > > > > care
> >> >>> > > > > > >>> about
> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more
> >> efficient
> >> >>> than
> >> >>> > > > > FLOAT16
> >> >>> > > > > > +
> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> >> >>> > > > > > >>> > >>
> >> >>> > > > > > >>> > >> It was. We thought we could defer it for the
> >> >>> following
> >> >>> > > > reasons:
> >> >>> > > > > > >>> > >> 1. It's not clear there are a lot of easy
> >> reference
> >> >>> > > datasets
> >> >>> > > > to
> >> >>> > > > > > >>> test
> >> >>> > > > > > >>> > its
> >> >>> > > > > > >>> > >> effectiveness. It does look like there might be
> >> one
> >> >>> or
> >> >>> > two
> >> >>> > > on
> >> >>> > > > > > >>> > huggingface
> >> >>> > > > > > >>> > >> (e.g.
> >> https://huggingface.co/datasets/kikitora/curdie
> >> >>> ).
> >> >>> > > > > > >>> > >> 2. It seemed likely that float 16 was more
> likely
> >> >>> used
> >> >>> > for
> >> >>> > > > > values
> >> >>> > > > > > >>> that
> >> >>> > > > > > >>> > >> were less likely to reduce to decimal values.
> >> >>> > > > > > >>> > >> 3. It could be added as an extension later if
> >> needed.
> >> >>> > > > > > >>> > >>
> >> >>> > > > > > >>> > >> Cheers,
> >> >>> > > > > > >>> > >> Micah
> >> >>> > > > > > >>> > >>
> >> >>> > > > > > >>> > >>
> >> >>> > > > > > >>> > >> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <
> >> >>> > > > > antoine@python.org
> >> >>> > > > > > >
> >> >>> > > > > > >>> > wrote:
> >> >>> > > > > > >>> > >>
> >> >>> > > > > > >>> > >> >
> >> >>> > > > > > >>> > >> > Hello,
> >> >>> > > > > > >>> > >> >
> >> >>> > > > > > >>> > >> > Apologies if the question was already asked,
> but
> >> >>> should
> >> >>> > we
> >> >>> > > > > care
> >> >>> > > > > > >>> about
> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more
> >> efficient
> >> >>> than
> >> >>> > > > > FLOAT16
> >> >>> > > > > > +
> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> >> >>> > > > > > >>> > >> >
> >> >>> > > > > > >>> > >> > Regards
> >> >>> > > > > > >>> > >> >
> >> >>> > > > > > >>> > >> > Antoine.
> >> >>> > > > > > >>> > >> >
> >> >>> > > > > > >>> > >> >
> >> >>> > > > > > >>> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
> >> >>> > > > > > >>> > >> > > Thanks Andrew and Micah for review feedback
> on
> >> >>> the two
> >> >>> > > > PR's
> >> >>> > > > > > >>> > >> > > 1) (c++ arrow repo)
> >> >>> > > > > > >>> > >>
> https://github.com/apache/arrow/pull/48345/changes
> >> >>> > > > > > >>> > >> > > 2) (parquet-format repo)
> >> >>> > > > > > >>> > >> >
> >> https://github.com/apache/parquet-format/pull/557
> >> >>> > > > > > >>> > >> > >
> >> >>> > > > > > >>> > >> > > I have addressed all (unless I missed
> >> something)
> >> >>> > > comments
> >> >>> > > > on
> >> >>> > > > > > >>> the two
> >> >>> > > > > > >>> > >> > PR's.
> >> >>> > > > > > >>> > >> > >
> >> >>> > > > > > >>> > >> > > Best
> >> >>> > > > > > >>> > >> > > Prateek
> >> >>> > > > > > >>> > >> > >
> >> >>> > > > > > >>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR
> <
> >> >>> > > > > > >>> prtkgaur@gmail.com>
> >> >>> > > > > > >>> > >> wrote:
> >> >>> > > > > > >>> > >> > >
> >> >>> > > > > > >>> > >> > >> Thanks Andrew and Micah.
> >> >>> > > > > > >>> > >> > >>
> >> >>> > > > > > >>> > >> > >> `fair amount of feedback on at least the
> >> >>> > > implementations`
> >> >>> > > > > > >>> > >> > >> For the c++ I have already started
> addressing
> >> the
> >> >>> > > > > feedback, I
> >> >>> > > > > > >>> > should
> >> >>> > > > > > >>> > >> be
> >> >>> > > > > > >>> > >> > >> done with that Monday/Tuesday.
> >> >>> > > > > > >>> > >> > >> I think Vinoo too has been making good
> >> progress
> >> >>> on
> >> >>> > the
> >> >>> > > > Java
> >> >>> > > > > > >>> > >> > implementation.
> >> >>> > > > > > >>> > >> > >>
> >> >>> > > > > > >>> > >> > >> Best
> >> >>> > > > > > >>> > >> > >> Prateek
> >> >>> > > > > > >>> > >> > >>
> >> >>> > > > > > >>> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew
> Lamb <
> >> >>> > > > > > >>> > andrewlamb11@gmail.com
> >> >>> > > > > > >>> > >> >
> >> >>> > > > > > >>> > >> > >> wrote:
> >> >>> > > > > > >>> > >> > >>
> >> >>> > > > > > >>> > >> > >>> Got it. Thank you for the clarification --
> I
> >> >>> will
> >> >>> > try
> >> >>> > > > and
> >> >>> > > > > > look
> >> >>> > > > > > >>> > into
> >> >>> > > > > > >>> > >> the
> >> >>> > > > > > >>> > >> > >>> spec and the Rust implementation[1] in this
> >> next
> >> >>> > week
> >> >>> > > > > > >>> > >> > >>>
> >> >>> > > > > > >>> > >> > >>> [1]:
> >> >>> https://github.com/apache/arrow-rs/pull/9372
> >> >>> > > > > > >>> > >> > >>>
> >> >>> > > > > > >>> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah
> >> >>> Kornfield <
> >> >>> > > > > > >>> > >> > emkornfield@gmail.com>
> >> >>> > > > > > >>> > >> > >>> wrote:
> >> >>> > > > > > >>> > >> > >>>
> >> >>> > > > > > >>> > >> > >>>> Hi Andrew,
> >> >>> > > > > > >>> > >> > >>>> I think there is a fair amount of feedback
> >> on
> >> >>> at
> >> >>> > > least
> >> >>> > > > > the
> >> >>> > > > > > >>> > >> > >>>> implementations, typically I think we've
> >> waited
> >> >>> > till
> >> >>> > > > they
> >> >>> > > > > > are
> >> >>> > > > > > >>> > >> close to
> >> >>> > > > > > >>> > >> > >>>> mergeable before a final vote. Otherwise
> I
> >> >>> agree
> >> >>> > we
> >> >>> > > > are
> >> >>> > > > > > very
> >> >>> > > > > > >>> > >> close.
> >> >>> > > > > > >>> > >> > >>>>
> >> >>> > > > > > >>> > >> > >>>> -Micah
> >> >>> > > > > > >>> > >> > >>>>
> >> >>> > > > > > >>> > >> > >>>> On Saturday, April 25, 2026, Andrew Lamb <
> >> >>> > > > > > >>> andrewlamb11@gmail.com
> >> >>> > > > > > >>> > >
> >> >>> > > > > > >>> > >> > wrote:
> >> >>> > > > > > >>> > >> > >>>>
> >> >>> > > > > > >>> > >> > >>>>> Thanks Prateek,
> >> >>> > > > > > >>> > >> > >>>>>
> >> >>> > > > > > >>> > >> > >>>>> I think from this content it looks to me
> >> like
> >> >>> we
> >> >>> > are
> >> >>> > > > > ready
> >> >>> > > > > > >>> to
> >> >>> > > > > > >>> > >> start a
> >> >>> > > > > > >>> > >> > >>>>> vote to explicitly accept ALP into
> Parquet
> >> >>> > > > > > >>> > >> > >>>>>
> >> >>> > > > > > >>> > >> > >>>>> Does anyone know of a reason we should
> >> >>> postpone it
> >> >>> > > for
> >> >>> > > > > > >>> longer?
> >> >>> > > > > > >>> > >> > >>>>> Perhaps someone needs some more time to
> >> >>> review?
> >> >>> > > > > > >>> > >> > >>>>>
> >> >>> > > > > > >>> > >> > >>>>> Andrew
> >> >>> > > > > > >>> > >> > >>>>>
> >> >>> > > > > > >>> > >> > >>>>>
> >> >>> > > > > > >>> > >> > >>>>>
> >> >>> > > > > > >>> > >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK
> >> GAUR <
> >> >>> > > > > > >>> > prtkgaur@gmail.com>
> >> >>> > > > > > >>> > >> > >>>>> wrote:
> >> >>> > > > > > >>> > >> > >>>>>
> >> >>> > > > > > >>> > >> > >>>>>> Hi team,
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>> Hope everyone is doing well. I got a
> >> chance
> >> >>> to
> >> >>> > work
> >> >>> > > > > > >>> through all
> >> >>> > > > > > >>> > >> the
> >> >>> > > > > > >>> > >> > >>>>>> remaining feedback and update the spec
> >> doc.
> >> >>> Here
> >> >>> > > are
> >> >>> > > > > the
> >> >>> > > > > > >>> new
> >> >>> > > > > > >>> > >> > artifacts
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>> 1) Spec document :
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> >
> >> >>> > > > > > >>> > >>
> >> >>> > > > > > >>> >
> >> >>> > > > > > >>>
> >> >>> > > > > >
> >> >>> > > > >
> >> >>> > > >
> >> >>> > >
> >> >>> >
> >> >>>
> >>
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>> 2) Spec document in parquet format repo
> :
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > https://github.com/apache/parquet-format/pull/557
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>> 3) Alp implementation in arrow c++ repo
> :
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > https://github.com/apache/arrow/pull/48345/changes
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>> 4) Alp implementation in parquet-java
> >> repo :
> >> >>> Work
> >> >>> > > for
> >> >>> > > > > > >>> Vinoo and
> >> >>> > > > > > >>> > >> > Julien
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > https://github.com/apache/parquet-java/pull/3397
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>> 5) PR with test and benchmarking
> >> artifacts in
> >> >>> > > > > > >>> parquet-testing
> >> >>> > > > > > >>> > >> repo :
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > https://github.com/apache/parquet-testing/pull/100
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>> And
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>> - Go : Arnav just submitted an in
> >> >>> progress
> >> >>> > > > > > >>> implementation
> >> >>> > > > > > >>> > in
> >> >>> > > > > > >>> > >> Go.
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> https://github.com/apache/arrow-go/pull/704
> >> >>> > (I
> >> >>> > > > > > haven't
> >> >>> > > > > > >>> > >> started
> >> >>> > > > > > >>> > >> > >>>>>> looking at it yet)
> >> >>> > > > > > >>> > >> > >>>>>> - Rust : I remember Andrew mentioned
> >> that
> >> >>> > this
> >> >>> > > > work
> >> >>> > > > > > is
> >> >>> > > > > > >>> also
> >> >>> > > > > > >>> > >> in
> >> >>> > > > > > >>> > >> > >>>>>> progress (So 4 languages!)
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>> *Arrow C++ implementation *
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>> The PR is out and was also used by
> >> Antoine to
> >> >>> > > report
> >> >>> > > > > the
> >> >>> > > > > > >>> > numbers
> >> >>> > > > > > >>> > >> as
> >> >>> > > > > > >>> > >> > >>>>>> reported here. Micah and Konstantin have
> >> >>> given 1
> >> >>> > > > round
> >> >>> > > > > of
> >> >>> > > > > > >>> > >> feedback
> >> >>> > > > > > >>> > >> > >>>>>> and I'm addressing them today. Please
> note
> >> >>> that
> >> >>> > the
> >> >>> > > > > > default
> >> >>> > > > > > >>> > >> > >>>>>> optimization flag for compiling is O2
> and
> >> not
> >> >>> > Q3. I
> >> >>> > > > got
> >> >>> > > > > > >>> around
> >> >>> > > > > > >>> > >> 70%
> >> >>> > > > > > >>> > >> > >>>>>> performance improvement in the decoding
> >> speed
> >> >>> > when
> >> >>> > > > > using
> >> >>> > > > > > >>> the O3
> >> >>> > > > > > >>> > >> > flag.
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>> *Parqet-MR Java implementation (working
> >> with
> >> >>> > Vinoo
> >> >>> > > > and
> >> >>> > > > > > >>> Julien)
> >> >>> > > > > > >>> > >> and
> >> >>> > > > > > >>> > >> > **Cross
> >> >>> > > > > > >>> > >> > >>>>>> Language testing*
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>> Let me know if you have any
> questions
> >> or
> >> >>> > > > feedback.
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>> Now pasting some performance numbers
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>> Table 1: C++ ALP Double Decode —
> >> Spotify
> >> >>> > Columns
> >> >>> > > > > > >>> (Graviton
> >> >>> > > > > > >>> > 3,
> >> >>> > > > > > >>> > >> ARM
> >> >>> > > > > > >>> > >> > >>>>>> Neoverse V1)
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>>
> >> >>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>> │ Column │ -O2 (MB/s) │
> >> -O3
> >> >>> > > (MB/s) │
> >> >>> > > > > > >>> Speedup │
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>>
> >> >>> ├──────────────────┼──────────────┼──────────────┼─────────┤
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>> │ valence │ 3,155 │
> >> >>> 5,523
> >> >>> > > │
> >> >>> > > > > > >>> 1.75x │
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>> │ danceability │ 3,233 │
> >> >>> 5,685
> >> >>> > > │
> >> >>> > > > > > >>> 1.76x │
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>> │ energy │ 3,197 │
> >> >>> 5,652
> >> >>> > > │
> >> >>> > > > > > >>> 1.77x │
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>> │ loudness │ 3,186 │
> >> >>> 5,473
> >> >>> > > │
> >> >>> > > > > > >>> 1.72x │
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>>
> >> >>> └──────────────────┴──────────────┴──────────────┴─────────┘
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK
> >> GAUR
> >> >>> <
> >> >>> > > > > > >>> > prtkgaur@gmail.com
> >> >>> > > > > > >>> > >> >
> >> >>> > > > > > >>> > >> > >>>>>> wrote:
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>> @Micah Kornfield <
> emkornfield@gmail.com>
> >> :
> >> >>> Got
> >> >>> > > it.
> >> >>> > > > > > >>> > >> > >>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>> @Andrew Lamb <an...@gmail.com>
> >> >>> > > > > > >>> > >> > >>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>> Do you think it would be good to start
> >> >>> moving
> >> >>> > the
> >> >>> > > > > spec
> >> >>> > > > > > >>> > >> development
> >> >>> > > > > > >>> > >> > >>>>>>>> into
> >> >>> > > > > > >>> > >> > >>>>>>>> markdown format, in preparation for
> >> >>> finalizing
> >> >>> > > it?
> >> >>> > > > > > >>> > >> > >>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>> Yes I'll update the numbers for some of
> >> the
> >> >>> > > > examples I
> >> >>> > > > > > >>> have in
> >> >>> > > > > > >>> > >> the
> >> >>> > > > > > >>> > >> > >>>>>>> spec based
> >> >>> > > > > > >>> > >> > >>>>>>> on the updated header size. Then we
> >> should
> >> >>> be
> >> >>> > good
> >> >>> > > > to
> >> >>> > > > > go
> >> >>> > > > > > >>> for
> >> >>> > > > > > >>> > the
> >> >>> > > > > > >>> > >> > >>>>>>> markdown format.
> >> >>> > > > > > >>> > >> > >>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>> Thanks everyone!
> >> >>> > > > > > >>> > >> > >>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>> Andrew
> >> >>> > > > > > >>> > >> > >>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM
> PRATEEK
> >> >>> GAUR <
> >> >>> > > > > > >>> > >> prtkgaur@gmail.com>
> >> >>> > > > > > >>> > >> > >>>>>>>> wrote:
> >> >>> > > > > > >>> > >> > >>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>> Hi team,
> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>> 1) Andrew
> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for working on test
> files.
> >> >>> My PR
> >> >>> > > did
> >> >>> > > > > add
> >> >>> > > > > > >>> all
> >> >>> > > > > > >>> > the
> >> >>> > > > > > >>> > >> > test
> >> >>> > > > > > >>> > >> > >>>>>>>> files I
> >> >>> > > > > > >>> > >> > >>>>>>>>> used to benchmark on datasets.
> >> Maybe
> >> >>> we
> >> >>> > can
> >> >>> > > > club
> >> >>> > > > > > it
> >> >>> > > > > > >>> > >> together.
> >> >>> > > > > > >>> > >> > >>>>>>>> WIll also
> >> >>> > > > > > >>> > >> > >>>>>>>>> aid
> >> >>> > > > > > >>> > >> > >>>>>>>>> cross language testing
> >> >>> > > > > > >>> > >> > >>>>>>>>> - Kosta Tarasov working on Rust
> >> >>> > > > implementation.
> >> >>> > > > > > >>> This is
> >> >>> > > > > > >>> > >> > great.
> >> >>> > > > > > >>> > >> > >>>>>>>> Thanks
> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>> 2) Antoine
> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks a lot for reporting the
> >> >>> numbers
> >> >>> > on
> >> >>> > > > AMD.
> >> >>> > > > > > >>> Looks
> >> >>> > > > > > >>> > >> like
> >> >>> > > > > > >>> > >> > you
> >> >>> > > > > > >>> > >> > >>>>>>>> are
> >> >>> > > > > > >>> > >> > >>>>>>>>> getting 8X the decoding
> >> performance of
> >> >>> > BSS.
> >> >>> > > > This
> >> >>> > > > > > is
> >> >>> > > > > > >>> > >> > amazing!!.
> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for acknowledging the
> >> >>> sampling
> >> >>> > > > design.
> >> >>> > > > > > >>> > >> > >>>>>>>>> - I agree with you on Fastlanes.
> In
> >> >>> some
> >> >>> > > crude
> >> >>> > > > > > >>> > >> experiments I
> >> >>> > > > > > >>> > >> > >>>>>>>> didn't get
> >> >>> > > > > > >>> > >> > >>>>>>>>> a good perf benefit from it on
> >> >>> Graviton3
> >> >>> > > (but
> >> >>> > > > > > maybe
> >> >>> > > > > > >>> > there
> >> >>> > > > > > >>> > >> was
> >> >>> > > > > > >>> > >> > >>>>>>>> something
> >> >>> > > > > > >>> > >> > >>>>>>>>> wrong with my implementation).
> >> >>> > > > > > >>> > >> > >>>>>>>>> - Locking the 16bit exception
> >> >>> encoding for
> >> >>> > > the
> >> >>> > > > > > spec
> >> >>> > > > > > >>> in
> >> >>> > > > > > >>> > >> this
> >> >>> > > > > > >>> > >> > >>>>>>>> case.
> >> >>> > > > > > >>> > >> > >>>>>>>>> - Awesome I think we have solved
> >> for
> >> >>> all
> >> >>> > > open
> >> >>> > > > > > >>> questions
> >> >>> > > > > > >>> > >> minus
> >> >>> > > > > > >>> > >> > >>>>>>>> the
> >> >>> > > > > > >>> > >> > >>>>>>>>> version byte :). (will get back
> on
> >> >>> this
> >> >>> > > soon)
> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>> 3) Micah
> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>> - FastLanes : The current spec
> does
> >> >>> allow
> >> >>> > > for
> >> >>> > > > > > using
> >> >>> > > > > > >>> > >> FastLane
> >> >>> > > > > > >>> > >> > >>>>>>>> with the
> >> >>> > > > > > >>> > >> > >>>>>>>>> configurable enum value for
> >> layout. We
> >> >>> > > should
> >> >>> > > > be
> >> >>> > > > > > >>> able to
> >> >>> > > > > > >>> > >> > inject
> >> >>> > > > > > >>> > >> > >>>>>>>> any
> >> >>> > > > > > >>> > >> > >>>>>>>>> layout
> >> >>> > > > > > >>> > >> > >>>>>>>>> in the current design.
> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>> Working on resolving all remaining
> open
> >> >>> > comments
> >> >>> > > > on
> >> >>> > > > > > the
> >> >>> > > > > > >>> spec
> >> >>> > > > > > >>> > >> this
> >> >>> > > > > > >>> > >> > >>>>>>>> week.
> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>> Best
> >> >>> > > > > > >>> > >> > >>>>>>>>> Prateek
> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve
> >> >>> > Loughran <
> >> >>> > > > > > >>> > >> > >>>>>>>> stevel@cloudera.com>
> >> >>> > > > > > >>> > >> > >>>>>>>>> wrote:
> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah
> >> >>> Kornfield
> >> >>> > <
> >> >>> > > > > > >>> > >> > >>>>>>>> emkornfield@gmail.com>
> >> >>> > > > > > >>> > >> > >>>>>>>>>> wrote:
> >> >>> > > > > > >>> > >> > >>>>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>>>> It looks like the actual issue
> >> >>> described for
> >> >>> > > ORC
> >> >>> > > > > in
> >> >>> > > > > > >>> the
> >> >>> > > > > > >>> > >> paper
> >> >>> > > > > > >>> > >> > >>>>>>>> is that
> >> >>> > > > > > >>> > >> > >>>>>>>>> it
> >> >>> > > > > > >>> > >> > >>>>>>>>>>> has multiple sub-encodings in a
> >> batch.
> >> >>> This
> >> >>> > > is
> >> >>> > > > > > >>> different
> >> >>> > > > > > >>> > >> then
> >> >>> > > > > > >>> > >> > >>>>>>>> the
> >> >>> > > > > > >>> > >> > >>>>>>>>> design
> >> >>> > > > > > >>> > >> > >>>>>>>>>>> proposed here where there is still
> >> fixed
> >> >>> > > > encoding
> >> >>> > > > > > per
> >> >>> > > > > > >>> page
> >> >>> > > > > > >>> > >> in
> >> >>> > > > > > >>> > >> > >>>>>>>> parquet.
> >> >>> > > > > > >>> > >> > >>>>>>>>>>> Given reasonably sized pages I
> don't
> >> >>> think
> >> >>> > > > branch
> >> >>> > > > > > >>> > >> > >>>>>>>> misprediction should
> >> >>> > > > > > >>> > >> > >>>>>>>>>> be a
> >> >>> > > > > > >>> > >> > >>>>>>>>>>> big issue for new encodings. I
> agree
> >> >>> that
> >> >>> > we
> >> >>> > > > > should
> >> >>> > > > > > >>> be
> >> >>> > > > > > >>> > >> > >>>>>>>> conservative in
> >> >>> > > > > > >>> > >> > >>>>>>>>>>> general for adding new encodings.
> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>>> +1
> >> >>> > > > > > >>> > >> > >>>>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>
> >> >>> > > > > > >>> > >> > >
> >> >>> > > > > > >>> > >> >
> >> >>> > > > > > >>> > >> >
> >> >>> > > > > > >>> > >> >
> >> >>> > > > > > >>> > >>
> >> >>> > > > > > >>> > >
> >> >>> > > > > > >>> >
> >> >>> > > > > > >>>
> >> >>> > > > > > >>
> >> >>> > > > > >
> >> >>> > > > >
> >> >>> > > >
> >> >>> > >
> >> >>> >
> >> >>>
> >> >>
> >>
> >
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Andrew Lamb <an...@gmail.com>.
Div: Here is a proposal to mark ALP in Preview: [1] (using a comment in the
spec section).
Russel: That is my understanding as well.
Hopefully we are able to formalize the versioning process shortly so we can
formalize the notion of "preview"
Andrew
[1]: https://github.com/apache/parquet-format/pull/604
On Fri, Jul 31, 2026 at 11:15 AM Russell Spitzer <ru...@gmail.com>
wrote:
> I thought we had agreed it would be merged, and features like this would
> be feature flag gated. So a V-Old writer could always opt in to using this
> encoding but that must be a conscious effort if you are not using VNext of
> Parquet. If you are on V-Next the writer can choose to use ALP whenever it
> likes without user intervention.
>
> On Fri, Jul 31, 2026 at 2:20 AM Divjot Arora via dev <
> dev@parquet.apache.org> wrote:
>
>> Yes, that’s my understanding as well. However, as you noted, the
>> versioning
>> discussion is still in early stages and hasn’t prescribed concrete
>> mechanisms yet. If ALP goes into parquet-format 2.14.0, there’s no
>> indication that it’s in preview or that writers shouldn’t use it by
>> default
>> yet. Given that we’ve held off on merging other incompatible changes (e.g
>> removal of path_in_schema) until the versioning discussion closes, my
>> preference would be to leave ALP un-merged for the time being as well.
>>
>> — Div
>>
>> On Fri, Jul 31, 2026 at 08:59 Andrew Lamb <an...@gmail.com> wrote:
>>
>> > After re-reading Julien's Versioning Proposal, my understanding is that
>> if
>> > we adopt that system, we will somehow mark ALP in "Preview" in
>> > parquet-format for release Version 3.
>> >
>> > The versioning document doesn't seem to proscribe the particular method
>> > for annotating "preview" features -- I think clearly marking them as
>> such
>> > in parquet.thrift as well as reflecting that on the
>> > https://parquet.apache.org/docs/file-format/versions/ page somehow
>> would
>> > be adequate.
>> >
>> > Andrew
>> >
>> > On Thu, Jul 30, 2026 at 11:50 AM Andrew Lamb <an...@gmail.com>
>> > wrote:
>> >
>> >> In my mind ALP is in the same position as other encodings that were
>> added
>> >> recently[1] (such as BYTE_STREAM_SPLIT for non floating point types)
>> >>
>> >> That is to say it will be available in the latest open source writers
>> and
>> >> readers, but realistically not widely used in the ecosystem where
>> >> compatibility is a concern (a large part)
>> >>
>> >> I view Julian's work[2][3] to gather consensus on how to name
>> >> combinations of features (e.g. "versions") as the mechanism by which we
>> >> will accelerate the adoption of this change (along with other forward
>> >> incompatible changes) across the ecosystem.
>> >>
>> >> > Will we exclude this change when cutting parquet-format v2.14.0?
>> >>
>> >> No I don't think so -- I would expect it to go out in the next
>> >> parquet-format version as do other changes in the spec.
>> >>
>> >> Andrew
>> >>
>> >> [1]: https://parquet.apache.org/docs/file-format/versions/
>> >> [2]: https://lists.apache.org/thread/xmnj8h0h8ozmrgox5tydhhs7yz7q2hss
>> >> [3]: https://lists.apache.org/thread/qlf8lg90gqq41lllyy9mk2f0skqvftqj
>> >>
>> >>
>> >> On Thu, Jul 30, 2026 at 3:51 AM Divjot Arora via dev <
>> >> dev@parquet.apache.org> wrote:
>> >>
>> >>> Hi Andrew,
>> >>>
>> >>> How does this interact with the ongoing versioning discussion? Given
>> that
>> >>> new encodings are not forward compatible, I would assume ALP is
>> included
>> >>> in
>> >>> Parquet V<next> (V3?), but it's not clear to me how that works if it's
>> >>> merged to the parquet-format master branch now. Will we exclude this
>> >>> change
>> >>> when cutting parquet-format v2.14.0?
>> >>>
>> >>> -- Div
>> >>>
>> >>> On Thu, Jul 30, 2026 at 8:51 AM Andrew Lamb <an...@gmail.com>
>> >>> wrote:
>> >>>
>> >>> > Hello,
>> >>> >
>> >>> > As an update, thanks to the epic work from many contributors,
>> >>> especially
>> >>> > Prateek, I just merged the parquet-format change for ALP[1]. We
>> plan to
>> >>> > make additional editorial The remaining items I know of are merging
>> the
>> >>> > examples from parquet-testing[2], and then of course the various
>> >>> > implementations
>> >>> >
>> >>> > Thanks again everyone. This is a great step forward
>> >>> > Andrew
>> >>> >
>> >>> > [1]: https://github.com/apache/parquet-format/pull/557
>> >>> > [2]: https://github.com/apache/parquet-testing/pull/100
>> >>> >
>> >>> > On Tue, Jul 21, 2026 at 4:39 PM Julien Le Dem <ju...@apache.org>
>> >>> wrote:
>> >>> >
>> >>> > > It was also in my spam folder... :(
>> >>> > > I've seen a bunch of apache list emails go to my spam recently.
>> >>> > >
>> >>> > > On Tue, Jul 21, 2026 at 10:58 AM Russell Spitzer <
>> >>> > > russell.spitzer@gmail.com>
>> >>> > > wrote:
>> >>> > >
>> >>> > > > Yes, my hope was that my vote would kick it out of spam but we
>> >>> > definitely
>> >>> > > > had two other votes that also went to spam for me.
>> >>> > > >
>> >>> > > > For anyone looking the thread is here
>> >>> > > >
>> https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt
>> >>> > > >
>> >>> > > > On Tue, Jul 21, 2026 at 10:57 AM Micah Kornfield <
>> >>> > emkornfield@gmail.com>
>> >>> > > > wrote:
>> >>> > > >
>> >>> > > > > I've seen other's votes not go to spam, so hopefully people
>> will
>> >>> see
>> >>> > it
>> >>> > > > in
>> >>> > > > > there inboxes (or see this thread and look in there spam).
>> >>> > > > >
>> >>> > > > > On Tue, Jul 21, 2026 at 10:52 AM PRATEEK GAUR <
>> >>> prtkgaur@gmail.com>
>> >>> > > > wrote:
>> >>> > > > >
>> >>> > > > > > Okay,
>> >>> > > > > >
>> >>> > > > > > I sent out the voting email yesterday and so far I've heard
>> >>> from 3
>> >>> > > > people
>> >>> > > > > > that it ended up in their "spam folder". Dev VPN to be
>> blamed
>> >>> for
>> >>> > > it?.
>> >>> > > > > >
>> >>> > > > > > What's the protocol here?
>> >>> > > > > > I let people hunt for the email in their spam folder or send
>> >>> out
>> >>> > > > another
>> >>> > > > > > one?
>> >>> > > > > >
>> >>> > > > > > Best
>> >>> > > > > > Prateek
>> >>> > > > > >
>> >>> > > > > > On Sun, Jul 19, 2026 at 10:47 PM PRATEEK GAUR <
>> >>> prtkgaur@gmail.com>
>> >>> > > > > wrote:
>> >>> > > > > >
>> >>> > > > > > > Hi team,
>> >>> > > > > > >
>> >>> > > > > > > Vinoo has added the tests which were requested in the last
>> >>> round
>> >>> > of
>> >>> > > > > > review.
>> >>> > > > > > > In his own words.
>> >>> > > > > > >
>> >>> > > > > > > "
>> >>> > > > > > > The extreme value tests Micah wanted are done and pushed.
>> I
>> >>> added
>> >>> > > > > > coverage
>> >>> > > > > > > for values that need the full FOR bit width after the
>> frame
>> >>> of
>> >>> > > > > reference
>> >>> > > > > > is
>> >>> > > > > > > applied: 64-bit (the case where the signed max minus min
>> >>> > > subtraction
>> >>> > > > > > > overflows), 63-bit (the non-overflow case), and 32-bit for
>> >>> > floats.
>> >>> > > > All
>> >>> > > > > > > lossless with zero exceptions. I also verified them
>> through
>> >>> Arrow
>> >>> > > C++
>> >>> > > > > ALP
>> >>> > > > > > > reader (the local build), so the Java-written extreme
>> columns
>> >>> > > decode
>> >>> > > > > > > correctly in C++, bit-exact against the expected values."
>> >>> > > > > > >
>> >>> > > > > > > State of the PR (#3397):
>> >>> > > > > > >
>> >>> > > > > > > - Merges cleanly with master (I pulled master in and
>> >>> resolved
>> >>> > > the
>> >>> > > > > > > conflicts).
>> >>> > > > > > > - 41 of 44 review threads resolved. The 3 remaining are
>> >>> > > > intentional:
>> >>> > > > > > > one deferred perf nit, one that stays open until
>> >>> > parquet-format
>> >>> > > > > ships
>> >>> > > > > > ALP,
>> >>> > > > > > >
>> >>> > > > > > > "
>> >>> > > > > > >
>> >>> > > > > > > With this both the
>> >>> > > > > > > 1) C++ PR :
>> >>> https://github.com/apache/arrow/pull/48345/changes
>> >>> > > > > > > 2) Java PR :
>> >>> https://github.com/apache/parquet-java/pull/3397
>> >>> > > > > > >
>> >>> > > > > > > Are cross language tested and range of test scenarios.
>> >>> > > > > > >
>> >>> > > > > > > Best
>> >>> > > > > > > Prateek
>> >>> > > > > > >
>> >>> > > > > > >
>> >>> > > > > > > On Mon, Jul 13, 2026 at 8:38 AM PRATEEK GAUR <
>> >>> prtkgaur@gmail.com
>> >>> > >
>> >>> > > > > wrote:
>> >>> > > > > > >
>> >>> > > > > > >> Thanks Matt,
>> >>> > > > > > >>
>> >>> > > > > > >> I'll be on it this week.
>> >>> > > > > > >>
>> >>> > > > > > >> Best
>> >>> > > > > > >> Prateek
>> >>> > > > > > >>
>> >>> > > > > > >> On Mon, Jul 13, 2026 at 8:36 AM Matt Topol <
>> >>> > > zotthewizard@gmail.com>
>> >>> > > > > > >> wrote:
>> >>> > > > > > >>
>> >>> > > > > > >>> There is also the Go implementation submitted by Arnav (
>> >>> > > > > > >>> https://github.com/apache/arrow-go/pull/704) which is
>> >>> waiting
>> >>> > > for
>> >>> > > > > > >>> updates.
>> >>> > > > > > >>>
>> >>> > > > > > >>> Just wanted to make sure it didn't get lost here.
>> >>> > > > > > >>>
>> >>> > > > > > >>> --Matt
>> >>> > > > > > >>>
>> >>> > > > > > >>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK GAUR <
>> >>> > > prtkgaur@gmail.com>
>> >>> > > > > > >>> wrote:
>> >>> > > > > > >>>
>> >>> > > > > > >>> > Hi all,
>> >>> > > > > > >>> >
>> >>> > > > > > >>> > I'd like to share a status update on the ALP encoding
>> >>> effort
>> >>> > > and
>> >>> > > > > get
>> >>> > > > > > a
>> >>> > > > > > >>> feel
>> >>> > > > > > >>> > from the community before starting a formal vote.
>> >>> > > > > > >>> >
>> >>> > > > > > >>> > We now have the specification plus implementations in
>> >>> both
>> >>> > > > > languages:
>> >>> > > > > > >>> >
>> >>> > > > > > >>> > - Spec: apache/parquet-format#557 — GH-533
>> >>> Add
>> >>> > ALP
>> >>> > > > > > encoding
>> >>> > > > > > >>> > specification
>> >>> > > > > > >>> > - C++ (Arrow): apache/arrow#48345 — GH-48701
>> >>> > > [C++][Parquet]
>> >>> > > > > Add
>> >>> > > > > > >>> ALPpd
>> >>> > > > > > >>> > encoding
>> >>> > > > > > >>> > - Java: apache/parquet-java#3397 — Parquet
>> >>> Java
>> >>> > ALP
>> >>> > > > > > >>> > Implementation (by Vinoo)
>> >>> > > > > > >>> >
>> >>> > > > > > >>> > Current state:
>> >>> > > > > > >>> >
>> >>> > > > > > >>> > - The C++ implementation has been through 3-4
>> rounds
>> >>> of
>> >>> > > > review.
>> >>> > > > > > >>> > - The Java implementation has been tracking well
>> and
>> >>> has
>> >>> > > had
>> >>> > > > > > >>> initial
>> >>> > > > > > >>> > reviews from contributors, with no major
>> remaining
>> >>> > > issues.
>> >>> > > > > > >>> > - Cross-language compatibility tests are passing:
>> the
>> >>> > Arrow
>> >>> > > > C++
>> >>> > > > > > >>> decoder
>> >>> > > > > > >>> > reads Java-written data bit-exactly across
>> ~1.56M
>> >>> > values
>> >>> > > > and
>> >>> > > > > 18
>> >>> > > > > > >>> > fixtures, covering V1 and V2 pages, multiple
>> vector
>> >>> > > sizes,
>> >>> > > > > and
>> >>> > > > > > >>> > several
>> >>> > > > > > >>> > real datasets — zero mismatches.
>> >>> > > > > > >>> >
>> >>> > > > > > >>> > As raised in the java-pr we will be adding coverage
>> for
>> >>> > > extreme
>> >>> > > > > > values
>> >>> > > > > > >>> > (those requiring
>> >>> > > > > > >>> > 63-64 bits after FOR is applied) before we close a
>> vote.
>> >>> > We'll
>> >>> > > > aim
>> >>> > > > > > to
>> >>> > > > > > >>> get
>> >>> > > > > > >>> > that
>> >>> > > > > > >>> > done in parallel.
>> >>> > > > > > >>> >
>> >>> > > > > > >>> > Plan: unless there are objections, I intend to start a
>> >>> vote
>> >>> > at
>> >>> > > > the
>> >>> > > > > > end
>> >>> > > > > > >>> of
>> >>> > > > > > >>> > this week/early
>> >>> > > > > > >>> > next week.
>> >>> > > > > > >>> >
>> >>> > > > > > >>> > Thanks
>> >>> > > > > > >>> > Prateek
>> >>> > > > > > >>> >
>> >>> > > > > > >>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <
>> >>> > > prtkgaur@gmail.com>
>> >>> > > > > > >>> wrote:
>> >>> > > > > > >>> >
>> >>> > > > > > >>> > > Hi Team,
>> >>> > > > > > >>> > >
>> >>> > > > > > >>> > > Just wanted to provide some updates on ALP.
>> >>> > > > > > >>> > > Micah and I have done 3-4 rounds of review for the
>> c++
>> >>> PR :
>> >>> > > > > > >>> > > https://github.com/apache/arrow/pull/48345
>> >>> > > > > > >>> > >
>> >>> > > > > > >>> > > For the Java implementation Vinoo has been working
>> on
>> >>> > > following
>> >>> > > > > PR
>> >>> > > > > > :
>> >>> > > > > > >>> > > https://github.com/apache/parquet-java/pull/3397
>> >>> > > > > > >>> > >
>> >>> > > > > > >>> > > Best
>> >>> > > > > > >>> > > Prateek
>> >>> > > > > > >>> > >
>> >>> > > > > > >>> > > On Tue, May 5, 2026 at 1:50 PM Micah Kornfield <
>> >>> > > > > > >>> emkornfield@gmail.com>
>> >>> > > > > > >>> > > wrote:
>> >>> > > > > > >>> > >
>> >>> > > > > > >>> > >> Hi Antoine,
>> >>> > > > > > >>> > >>
>> >>> > > > > > >>> > >> > Apologies if the question was already asked, but
>> >>> should
>> >>> > we
>> >>> > > > > care
>> >>> > > > > > >>> about
>> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more
>> efficient
>> >>> than
>> >>> > > > > FLOAT16
>> >>> > > > > > +
>> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
>> >>> > > > > > >>> > >>
>> >>> > > > > > >>> > >> It was. We thought we could defer it for the
>> >>> following
>> >>> > > > reasons:
>> >>> > > > > > >>> > >> 1. It's not clear there are a lot of easy
>> reference
>> >>> > > datasets
>> >>> > > > to
>> >>> > > > > > >>> test
>> >>> > > > > > >>> > its
>> >>> > > > > > >>> > >> effectiveness. It does look like there might be
>> one
>> >>> or
>> >>> > two
>> >>> > > on
>> >>> > > > > > >>> > huggingface
>> >>> > > > > > >>> > >> (e.g.
>> https://huggingface.co/datasets/kikitora/curdie
>> >>> ).
>> >>> > > > > > >>> > >> 2. It seemed likely that float 16 was more likely
>> >>> used
>> >>> > for
>> >>> > > > > values
>> >>> > > > > > >>> that
>> >>> > > > > > >>> > >> were less likely to reduce to decimal values.
>> >>> > > > > > >>> > >> 3. It could be added as an extension later if
>> needed.
>> >>> > > > > > >>> > >>
>> >>> > > > > > >>> > >> Cheers,
>> >>> > > > > > >>> > >> Micah
>> >>> > > > > > >>> > >>
>> >>> > > > > > >>> > >>
>> >>> > > > > > >>> > >> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <
>> >>> > > > > antoine@python.org
>> >>> > > > > > >
>> >>> > > > > > >>> > wrote:
>> >>> > > > > > >>> > >>
>> >>> > > > > > >>> > >> >
>> >>> > > > > > >>> > >> > Hello,
>> >>> > > > > > >>> > >> >
>> >>> > > > > > >>> > >> > Apologies if the question was already asked, but
>> >>> should
>> >>> > we
>> >>> > > > > care
>> >>> > > > > > >>> about
>> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more
>> efficient
>> >>> than
>> >>> > > > > FLOAT16
>> >>> > > > > > +
>> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
>> >>> > > > > > >>> > >> >
>> >>> > > > > > >>> > >> > Regards
>> >>> > > > > > >>> > >> >
>> >>> > > > > > >>> > >> > Antoine.
>> >>> > > > > > >>> > >> >
>> >>> > > > > > >>> > >> >
>> >>> > > > > > >>> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
>> >>> > > > > > >>> > >> > > Thanks Andrew and Micah for review feedback on
>> >>> the two
>> >>> > > > PR's
>> >>> > > > > > >>> > >> > > 1) (c++ arrow repo)
>> >>> > > > > > >>> > >> https://github.com/apache/arrow/pull/48345/changes
>> >>> > > > > > >>> > >> > > 2) (parquet-format repo)
>> >>> > > > > > >>> > >> >
>> https://github.com/apache/parquet-format/pull/557
>> >>> > > > > > >>> > >> > >
>> >>> > > > > > >>> > >> > > I have addressed all (unless I missed
>> something)
>> >>> > > comments
>> >>> > > > on
>> >>> > > > > > >>> the two
>> >>> > > > > > >>> > >> > PR's.
>> >>> > > > > > >>> > >> > >
>> >>> > > > > > >>> > >> > > Best
>> >>> > > > > > >>> > >> > > Prateek
>> >>> > > > > > >>> > >> > >
>> >>> > > > > > >>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <
>> >>> > > > > > >>> prtkgaur@gmail.com>
>> >>> > > > > > >>> > >> wrote:
>> >>> > > > > > >>> > >> > >
>> >>> > > > > > >>> > >> > >> Thanks Andrew and Micah.
>> >>> > > > > > >>> > >> > >>
>> >>> > > > > > >>> > >> > >> `fair amount of feedback on at least the
>> >>> > > implementations`
>> >>> > > > > > >>> > >> > >> For the c++ I have already started addressing
>> the
>> >>> > > > > feedback, I
>> >>> > > > > > >>> > should
>> >>> > > > > > >>> > >> be
>> >>> > > > > > >>> > >> > >> done with that Monday/Tuesday.
>> >>> > > > > > >>> > >> > >> I think Vinoo too has been making good
>> progress
>> >>> on
>> >>> > the
>> >>> > > > Java
>> >>> > > > > > >>> > >> > implementation.
>> >>> > > > > > >>> > >> > >>
>> >>> > > > > > >>> > >> > >> Best
>> >>> > > > > > >>> > >> > >> Prateek
>> >>> > > > > > >>> > >> > >>
>> >>> > > > > > >>> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <
>> >>> > > > > > >>> > andrewlamb11@gmail.com
>> >>> > > > > > >>> > >> >
>> >>> > > > > > >>> > >> > >> wrote:
>> >>> > > > > > >>> > >> > >>
>> >>> > > > > > >>> > >> > >>> Got it. Thank you for the clarification -- I
>> >>> will
>> >>> > try
>> >>> > > > and
>> >>> > > > > > look
>> >>> > > > > > >>> > into
>> >>> > > > > > >>> > >> the
>> >>> > > > > > >>> > >> > >>> spec and the Rust implementation[1] in this
>> next
>> >>> > week
>> >>> > > > > > >>> > >> > >>>
>> >>> > > > > > >>> > >> > >>> [1]:
>> >>> https://github.com/apache/arrow-rs/pull/9372
>> >>> > > > > > >>> > >> > >>>
>> >>> > > > > > >>> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah
>> >>> Kornfield <
>> >>> > > > > > >>> > >> > emkornfield@gmail.com>
>> >>> > > > > > >>> > >> > >>> wrote:
>> >>> > > > > > >>> > >> > >>>
>> >>> > > > > > >>> > >> > >>>> Hi Andrew,
>> >>> > > > > > >>> > >> > >>>> I think there is a fair amount of feedback
>> on
>> >>> at
>> >>> > > least
>> >>> > > > > the
>> >>> > > > > > >>> > >> > >>>> implementations, typically I think we've
>> waited
>> >>> > till
>> >>> > > > they
>> >>> > > > > > are
>> >>> > > > > > >>> > >> close to
>> >>> > > > > > >>> > >> > >>>> mergeable before a final vote. Otherwise I
>> >>> agree
>> >>> > we
>> >>> > > > are
>> >>> > > > > > very
>> >>> > > > > > >>> > >> close.
>> >>> > > > > > >>> > >> > >>>>
>> >>> > > > > > >>> > >> > >>>> -Micah
>> >>> > > > > > >>> > >> > >>>>
>> >>> > > > > > >>> > >> > >>>> On Saturday, April 25, 2026, Andrew Lamb <
>> >>> > > > > > >>> andrewlamb11@gmail.com
>> >>> > > > > > >>> > >
>> >>> > > > > > >>> > >> > wrote:
>> >>> > > > > > >>> > >> > >>>>
>> >>> > > > > > >>> > >> > >>>>> Thanks Prateek,
>> >>> > > > > > >>> > >> > >>>>>
>> >>> > > > > > >>> > >> > >>>>> I think from this content it looks to me
>> like
>> >>> we
>> >>> > are
>> >>> > > > > ready
>> >>> > > > > > >>> to
>> >>> > > > > > >>> > >> start a
>> >>> > > > > > >>> > >> > >>>>> vote to explicitly accept ALP into Parquet
>> >>> > > > > > >>> > >> > >>>>>
>> >>> > > > > > >>> > >> > >>>>> Does anyone know of a reason we should
>> >>> postpone it
>> >>> > > for
>> >>> > > > > > >>> longer?
>> >>> > > > > > >>> > >> > >>>>> Perhaps someone needs some more time to
>> >>> review?
>> >>> > > > > > >>> > >> > >>>>>
>> >>> > > > > > >>> > >> > >>>>> Andrew
>> >>> > > > > > >>> > >> > >>>>>
>> >>> > > > > > >>> > >> > >>>>>
>> >>> > > > > > >>> > >> > >>>>>
>> >>> > > > > > >>> > >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK
>> GAUR <
>> >>> > > > > > >>> > prtkgaur@gmail.com>
>> >>> > > > > > >>> > >> > >>>>> wrote:
>> >>> > > > > > >>> > >> > >>>>>
>> >>> > > > > > >>> > >> > >>>>>> Hi team,
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>> Hope everyone is doing well. I got a
>> chance
>> >>> to
>> >>> > work
>> >>> > > > > > >>> through all
>> >>> > > > > > >>> > >> the
>> >>> > > > > > >>> > >> > >>>>>> remaining feedback and update the spec
>> doc.
>> >>> Here
>> >>> > > are
>> >>> > > > > the
>> >>> > > > > > >>> new
>> >>> > > > > > >>> > >> > artifacts
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>> 1) Spec document :
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> >
>> >>> > > > > > >>> > >>
>> >>> > > > > > >>> >
>> >>> > > > > > >>>
>> >>> > > > > >
>> >>> > > > >
>> >>> > > >
>> >>> > >
>> >>> >
>> >>>
>> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>> 2) Spec document in parquet format repo :
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > https://github.com/apache/parquet-format/pull/557
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>> 3) Alp implementation in arrow c++ repo :
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > https://github.com/apache/arrow/pull/48345/changes
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>> 4) Alp implementation in parquet-java
>> repo :
>> >>> Work
>> >>> > > for
>> >>> > > > > > >>> Vinoo and
>> >>> > > > > > >>> > >> > Julien
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > https://github.com/apache/parquet-java/pull/3397
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>> 5) PR with test and benchmarking
>> artifacts in
>> >>> > > > > > >>> parquet-testing
>> >>> > > > > > >>> > >> repo :
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > https://github.com/apache/parquet-testing/pull/100
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>> And
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>> - Go : Arnav just submitted an in
>> >>> progress
>> >>> > > > > > >>> implementation
>> >>> > > > > > >>> > in
>> >>> > > > > > >>> > >> Go.
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> https://github.com/apache/arrow-go/pull/704
>> >>> > (I
>> >>> > > > > > haven't
>> >>> > > > > > >>> > >> started
>> >>> > > > > > >>> > >> > >>>>>> looking at it yet)
>> >>> > > > > > >>> > >> > >>>>>> - Rust : I remember Andrew mentioned
>> that
>> >>> > this
>> >>> > > > work
>> >>> > > > > > is
>> >>> > > > > > >>> also
>> >>> > > > > > >>> > >> in
>> >>> > > > > > >>> > >> > >>>>>> progress (So 4 languages!)
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>> *Arrow C++ implementation *
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>> The PR is out and was also used by
>> Antoine to
>> >>> > > report
>> >>> > > > > the
>> >>> > > > > > >>> > numbers
>> >>> > > > > > >>> > >> as
>> >>> > > > > > >>> > >> > >>>>>> reported here. Micah and Konstantin have
>> >>> given 1
>> >>> > > > round
>> >>> > > > > of
>> >>> > > > > > >>> > >> feedback
>> >>> > > > > > >>> > >> > >>>>>> and I'm addressing them today. Please note
>> >>> that
>> >>> > the
>> >>> > > > > > default
>> >>> > > > > > >>> > >> > >>>>>> optimization flag for compiling is O2 and
>> not
>> >>> > Q3. I
>> >>> > > > got
>> >>> > > > > > >>> around
>> >>> > > > > > >>> > >> 70%
>> >>> > > > > > >>> > >> > >>>>>> performance improvement in the decoding
>> speed
>> >>> > when
>> >>> > > > > using
>> >>> > > > > > >>> the O3
>> >>> > > > > > >>> > >> > flag.
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>> *Parqet-MR Java implementation (working
>> with
>> >>> > Vinoo
>> >>> > > > and
>> >>> > > > > > >>> Julien)
>> >>> > > > > > >>> > >> and
>> >>> > > > > > >>> > >> > **Cross
>> >>> > > > > > >>> > >> > >>>>>> Language testing*
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>> Let me know if you have any questions
>> or
>> >>> > > > feedback.
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>> Now pasting some performance numbers
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>> Table 1: C++ ALP Double Decode —
>> Spotify
>> >>> > Columns
>> >>> > > > > > >>> (Graviton
>> >>> > > > > > >>> > 3,
>> >>> > > > > > >>> > >> ARM
>> >>> > > > > > >>> > >> > >>>>>> Neoverse V1)
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>>
>> >>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>> │ Column │ -O2 (MB/s) │
>> -O3
>> >>> > > (MB/s) │
>> >>> > > > > > >>> Speedup │
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>>
>> >>> ├──────────────────┼──────────────┼──────────────┼─────────┤
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>> │ valence │ 3,155 │
>> >>> 5,523
>> >>> > > │
>> >>> > > > > > >>> 1.75x │
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>> │ danceability │ 3,233 │
>> >>> 5,685
>> >>> > > │
>> >>> > > > > > >>> 1.76x │
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>> │ energy │ 3,197 │
>> >>> 5,652
>> >>> > > │
>> >>> > > > > > >>> 1.77x │
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>> │ loudness │ 3,186 │
>> >>> 5,473
>> >>> > > │
>> >>> > > > > > >>> 1.72x │
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>>
>> >>> └──────────────────┴──────────────┴──────────────┴─────────┘
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK
>> GAUR
>> >>> <
>> >>> > > > > > >>> > prtkgaur@gmail.com
>> >>> > > > > > >>> > >> >
>> >>> > > > > > >>> > >> > >>>>>> wrote:
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>>> @Micah Kornfield <em...@gmail.com>
>> :
>> >>> Got
>> >>> > > it.
>> >>> > > > > > >>> > >> > >>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>> @Andrew Lamb <an...@gmail.com>
>> >>> > > > > > >>> > >> > >>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>> Do you think it would be good to start
>> >>> moving
>> >>> > the
>> >>> > > > > spec
>> >>> > > > > > >>> > >> development
>> >>> > > > > > >>> > >> > >>>>>>>> into
>> >>> > > > > > >>> > >> > >>>>>>>> markdown format, in preparation for
>> >>> finalizing
>> >>> > > it?
>> >>> > > > > > >>> > >> > >>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>> Yes I'll update the numbers for some of
>> the
>> >>> > > > examples I
>> >>> > > > > > >>> have in
>> >>> > > > > > >>> > >> the
>> >>> > > > > > >>> > >> > >>>>>>> spec based
>> >>> > > > > > >>> > >> > >>>>>>> on the updated header size. Then we
>> should
>> >>> be
>> >>> > good
>> >>> > > > to
>> >>> > > > > go
>> >>> > > > > > >>> for
>> >>> > > > > > >>> > the
>> >>> > > > > > >>> > >> > >>>>>>> markdown format.
>> >>> > > > > > >>> > >> > >>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>> Thanks everyone!
>> >>> > > > > > >>> > >> > >>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>> Andrew
>> >>> > > > > > >>> > >> > >>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK
>> >>> GAUR <
>> >>> > > > > > >>> > >> prtkgaur@gmail.com>
>> >>> > > > > > >>> > >> > >>>>>>>> wrote:
>> >>> > > > > > >>> > >> > >>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>> Hi team,
>> >>> > > > > > >>> > >> > >>>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>> 1) Andrew
>> >>> > > > > > >>> > >> > >>>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for working on test files.
>> >>> My PR
>> >>> > > did
>> >>> > > > > add
>> >>> > > > > > >>> all
>> >>> > > > > > >>> > the
>> >>> > > > > > >>> > >> > test
>> >>> > > > > > >>> > >> > >>>>>>>> files I
>> >>> > > > > > >>> > >> > >>>>>>>>> used to benchmark on datasets.
>> Maybe
>> >>> we
>> >>> > can
>> >>> > > > club
>> >>> > > > > > it
>> >>> > > > > > >>> > >> together.
>> >>> > > > > > >>> > >> > >>>>>>>> WIll also
>> >>> > > > > > >>> > >> > >>>>>>>>> aid
>> >>> > > > > > >>> > >> > >>>>>>>>> cross language testing
>> >>> > > > > > >>> > >> > >>>>>>>>> - Kosta Tarasov working on Rust
>> >>> > > > implementation.
>> >>> > > > > > >>> This is
>> >>> > > > > > >>> > >> > great.
>> >>> > > > > > >>> > >> > >>>>>>>> Thanks
>> >>> > > > > > >>> > >> > >>>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>> 2) Antoine
>> >>> > > > > > >>> > >> > >>>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks a lot for reporting the
>> >>> numbers
>> >>> > on
>> >>> > > > AMD.
>> >>> > > > > > >>> Looks
>> >>> > > > > > >>> > >> like
>> >>> > > > > > >>> > >> > you
>> >>> > > > > > >>> > >> > >>>>>>>> are
>> >>> > > > > > >>> > >> > >>>>>>>>> getting 8X the decoding
>> performance of
>> >>> > BSS.
>> >>> > > > This
>> >>> > > > > > is
>> >>> > > > > > >>> > >> > amazing!!.
>> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for acknowledging the
>> >>> sampling
>> >>> > > > design.
>> >>> > > > > > >>> > >> > >>>>>>>>> - I agree with you on Fastlanes. In
>> >>> some
>> >>> > > crude
>> >>> > > > > > >>> > >> experiments I
>> >>> > > > > > >>> > >> > >>>>>>>> didn't get
>> >>> > > > > > >>> > >> > >>>>>>>>> a good perf benefit from it on
>> >>> Graviton3
>> >>> > > (but
>> >>> > > > > > maybe
>> >>> > > > > > >>> > there
>> >>> > > > > > >>> > >> was
>> >>> > > > > > >>> > >> > >>>>>>>> something
>> >>> > > > > > >>> > >> > >>>>>>>>> wrong with my implementation).
>> >>> > > > > > >>> > >> > >>>>>>>>> - Locking the 16bit exception
>> >>> encoding for
>> >>> > > the
>> >>> > > > > > spec
>> >>> > > > > > >>> in
>> >>> > > > > > >>> > >> this
>> >>> > > > > > >>> > >> > >>>>>>>> case.
>> >>> > > > > > >>> > >> > >>>>>>>>> - Awesome I think we have solved
>> for
>> >>> all
>> >>> > > open
>> >>> > > > > > >>> questions
>> >>> > > > > > >>> > >> minus
>> >>> > > > > > >>> > >> > >>>>>>>> the
>> >>> > > > > > >>> > >> > >>>>>>>>> version byte :). (will get back on
>> >>> this
>> >>> > > soon)
>> >>> > > > > > >>> > >> > >>>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>> 3) Micah
>> >>> > > > > > >>> > >> > >>>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>> - FastLanes : The current spec does
>> >>> allow
>> >>> > > for
>> >>> > > > > > using
>> >>> > > > > > >>> > >> FastLane
>> >>> > > > > > >>> > >> > >>>>>>>> with the
>> >>> > > > > > >>> > >> > >>>>>>>>> configurable enum value for
>> layout. We
>> >>> > > should
>> >>> > > > be
>> >>> > > > > > >>> able to
>> >>> > > > > > >>> > >> > inject
>> >>> > > > > > >>> > >> > >>>>>>>> any
>> >>> > > > > > >>> > >> > >>>>>>>>> layout
>> >>> > > > > > >>> > >> > >>>>>>>>> in the current design.
>> >>> > > > > > >>> > >> > >>>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>> Working on resolving all remaining open
>> >>> > comments
>> >>> > > > on
>> >>> > > > > > the
>> >>> > > > > > >>> spec
>> >>> > > > > > >>> > >> this
>> >>> > > > > > >>> > >> > >>>>>>>> week.
>> >>> > > > > > >>> > >> > >>>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>> Best
>> >>> > > > > > >>> > >> > >>>>>>>>> Prateek
>> >>> > > > > > >>> > >> > >>>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve
>> >>> > Loughran <
>> >>> > > > > > >>> > >> > >>>>>>>> stevel@cloudera.com>
>> >>> > > > > > >>> > >> > >>>>>>>>> wrote:
>> >>> > > > > > >>> > >> > >>>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah
>> >>> Kornfield
>> >>> > <
>> >>> > > > > > >>> > >> > >>>>>>>> emkornfield@gmail.com>
>> >>> > > > > > >>> > >> > >>>>>>>>>> wrote:
>> >>> > > > > > >>> > >> > >>>>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>>>> It looks like the actual issue
>> >>> described for
>> >>> > > ORC
>> >>> > > > > in
>> >>> > > > > > >>> the
>> >>> > > > > > >>> > >> paper
>> >>> > > > > > >>> > >> > >>>>>>>> is that
>> >>> > > > > > >>> > >> > >>>>>>>>> it
>> >>> > > > > > >>> > >> > >>>>>>>>>>> has multiple sub-encodings in a
>> batch.
>> >>> This
>> >>> > > is
>> >>> > > > > > >>> different
>> >>> > > > > > >>> > >> then
>> >>> > > > > > >>> > >> > >>>>>>>> the
>> >>> > > > > > >>> > >> > >>>>>>>>> design
>> >>> > > > > > >>> > >> > >>>>>>>>>>> proposed here where there is still
>> fixed
>> >>> > > > encoding
>> >>> > > > > > per
>> >>> > > > > > >>> page
>> >>> > > > > > >>> > >> in
>> >>> > > > > > >>> > >> > >>>>>>>> parquet.
>> >>> > > > > > >>> > >> > >>>>>>>>>>> Given reasonably sized pages I don't
>> >>> think
>> >>> > > > branch
>> >>> > > > > > >>> > >> > >>>>>>>> misprediction should
>> >>> > > > > > >>> > >> > >>>>>>>>>> be a
>> >>> > > > > > >>> > >> > >>>>>>>>>>> big issue for new encodings. I agree
>> >>> that
>> >>> > we
>> >>> > > > > should
>> >>> > > > > > >>> be
>> >>> > > > > > >>> > >> > >>>>>>>> conservative in
>> >>> > > > > > >>> > >> > >>>>>>>>>>> general for adding new encodings.
>> >>> > > > > > >>> > >> > >>>>>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>>> +1
>> >>> > > > > > >>> > >> > >>>>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>
>> >>> > > > > > >>> > >> > >
>> >>> > > > > > >>> > >> >
>> >>> > > > > > >>> > >> >
>> >>> > > > > > >>> > >> >
>> >>> > > > > > >>> > >>
>> >>> > > > > > >>> > >
>> >>> > > > > > >>> >
>> >>> > > > > > >>>
>> >>> > > > > > >>
>> >>> > > > > >
>> >>> > > > >
>> >>> > > >
>> >>> > >
>> >>> >
>> >>>
>> >>
>>
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Russell Spitzer <ru...@gmail.com>.
I thought we had agreed it would be merged, and features like this would be
feature flag gated. So a V-Old writer could always opt in to using this
encoding but that must be a conscious effort if you are not using VNext of
Parquet. If you are on V-Next the writer can choose to use ALP whenever it
likes without user intervention.
On Fri, Jul 31, 2026 at 2:20 AM Divjot Arora via dev <de...@parquet.apache.org>
wrote:
> Yes, that’s my understanding as well. However, as you noted, the versioning
> discussion is still in early stages and hasn’t prescribed concrete
> mechanisms yet. If ALP goes into parquet-format 2.14.0, there’s no
> indication that it’s in preview or that writers shouldn’t use it by default
> yet. Given that we’ve held off on merging other incompatible changes (e.g
> removal of path_in_schema) until the versioning discussion closes, my
> preference would be to leave ALP un-merged for the time being as well.
>
> — Div
>
> On Fri, Jul 31, 2026 at 08:59 Andrew Lamb <an...@gmail.com> wrote:
>
> > After re-reading Julien's Versioning Proposal, my understanding is that
> if
> > we adopt that system, we will somehow mark ALP in "Preview" in
> > parquet-format for release Version 3.
> >
> > The versioning document doesn't seem to proscribe the particular method
> > for annotating "preview" features -- I think clearly marking them as such
> > in parquet.thrift as well as reflecting that on the
> > https://parquet.apache.org/docs/file-format/versions/ page somehow would
> > be adequate.
> >
> > Andrew
> >
> > On Thu, Jul 30, 2026 at 11:50 AM Andrew Lamb <an...@gmail.com>
> > wrote:
> >
> >> In my mind ALP is in the same position as other encodings that were
> added
> >> recently[1] (such as BYTE_STREAM_SPLIT for non floating point types)
> >>
> >> That is to say it will be available in the latest open source writers
> and
> >> readers, but realistically not widely used in the ecosystem where
> >> compatibility is a concern (a large part)
> >>
> >> I view Julian's work[2][3] to gather consensus on how to name
> >> combinations of features (e.g. "versions") as the mechanism by which we
> >> will accelerate the adoption of this change (along with other forward
> >> incompatible changes) across the ecosystem.
> >>
> >> > Will we exclude this change when cutting parquet-format v2.14.0?
> >>
> >> No I don't think so -- I would expect it to go out in the next
> >> parquet-format version as do other changes in the spec.
> >>
> >> Andrew
> >>
> >> [1]: https://parquet.apache.org/docs/file-format/versions/
> >> [2]: https://lists.apache.org/thread/xmnj8h0h8ozmrgox5tydhhs7yz7q2hss
> >> [3]: https://lists.apache.org/thread/qlf8lg90gqq41lllyy9mk2f0skqvftqj
> >>
> >>
> >> On Thu, Jul 30, 2026 at 3:51 AM Divjot Arora via dev <
> >> dev@parquet.apache.org> wrote:
> >>
> >>> Hi Andrew,
> >>>
> >>> How does this interact with the ongoing versioning discussion? Given
> that
> >>> new encodings are not forward compatible, I would assume ALP is
> included
> >>> in
> >>> Parquet V<next> (V3?), but it's not clear to me how that works if it's
> >>> merged to the parquet-format master branch now. Will we exclude this
> >>> change
> >>> when cutting parquet-format v2.14.0?
> >>>
> >>> -- Div
> >>>
> >>> On Thu, Jul 30, 2026 at 8:51 AM Andrew Lamb <an...@gmail.com>
> >>> wrote:
> >>>
> >>> > Hello,
> >>> >
> >>> > As an update, thanks to the epic work from many contributors,
> >>> especially
> >>> > Prateek, I just merged the parquet-format change for ALP[1]. We plan
> to
> >>> > make additional editorial The remaining items I know of are merging
> the
> >>> > examples from parquet-testing[2], and then of course the various
> >>> > implementations
> >>> >
> >>> > Thanks again everyone. This is a great step forward
> >>> > Andrew
> >>> >
> >>> > [1]: https://github.com/apache/parquet-format/pull/557
> >>> > [2]: https://github.com/apache/parquet-testing/pull/100
> >>> >
> >>> > On Tue, Jul 21, 2026 at 4:39 PM Julien Le Dem <ju...@apache.org>
> >>> wrote:
> >>> >
> >>> > > It was also in my spam folder... :(
> >>> > > I've seen a bunch of apache list emails go to my spam recently.
> >>> > >
> >>> > > On Tue, Jul 21, 2026 at 10:58 AM Russell Spitzer <
> >>> > > russell.spitzer@gmail.com>
> >>> > > wrote:
> >>> > >
> >>> > > > Yes, my hope was that my vote would kick it out of spam but we
> >>> > definitely
> >>> > > > had two other votes that also went to spam for me.
> >>> > > >
> >>> > > > For anyone looking the thread is here
> >>> > > > https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt
> >>> > > >
> >>> > > > On Tue, Jul 21, 2026 at 10:57 AM Micah Kornfield <
> >>> > emkornfield@gmail.com>
> >>> > > > wrote:
> >>> > > >
> >>> > > > > I've seen other's votes not go to spam, so hopefully people
> will
> >>> see
> >>> > it
> >>> > > > in
> >>> > > > > there inboxes (or see this thread and look in there spam).
> >>> > > > >
> >>> > > > > On Tue, Jul 21, 2026 at 10:52 AM PRATEEK GAUR <
> >>> prtkgaur@gmail.com>
> >>> > > > wrote:
> >>> > > > >
> >>> > > > > > Okay,
> >>> > > > > >
> >>> > > > > > I sent out the voting email yesterday and so far I've heard
> >>> from 3
> >>> > > > people
> >>> > > > > > that it ended up in their "spam folder". Dev VPN to be blamed
> >>> for
> >>> > > it?.
> >>> > > > > >
> >>> > > > > > What's the protocol here?
> >>> > > > > > I let people hunt for the email in their spam folder or send
> >>> out
> >>> > > > another
> >>> > > > > > one?
> >>> > > > > >
> >>> > > > > > Best
> >>> > > > > > Prateek
> >>> > > > > >
> >>> > > > > > On Sun, Jul 19, 2026 at 10:47 PM PRATEEK GAUR <
> >>> prtkgaur@gmail.com>
> >>> > > > > wrote:
> >>> > > > > >
> >>> > > > > > > Hi team,
> >>> > > > > > >
> >>> > > > > > > Vinoo has added the tests which were requested in the last
> >>> round
> >>> > of
> >>> > > > > > review.
> >>> > > > > > > In his own words.
> >>> > > > > > >
> >>> > > > > > > "
> >>> > > > > > > The extreme value tests Micah wanted are done and pushed. I
> >>> added
> >>> > > > > > coverage
> >>> > > > > > > for values that need the full FOR bit width after the frame
> >>> of
> >>> > > > > reference
> >>> > > > > > is
> >>> > > > > > > applied: 64-bit (the case where the signed max minus min
> >>> > > subtraction
> >>> > > > > > > overflows), 63-bit (the non-overflow case), and 32-bit for
> >>> > floats.
> >>> > > > All
> >>> > > > > > > lossless with zero exceptions. I also verified them through
> >>> Arrow
> >>> > > C++
> >>> > > > > ALP
> >>> > > > > > > reader (the local build), so the Java-written extreme
> columns
> >>> > > decode
> >>> > > > > > > correctly in C++, bit-exact against the expected values."
> >>> > > > > > >
> >>> > > > > > > State of the PR (#3397):
> >>> > > > > > >
> >>> > > > > > > - Merges cleanly with master (I pulled master in and
> >>> resolved
> >>> > > the
> >>> > > > > > > conflicts).
> >>> > > > > > > - 41 of 44 review threads resolved. The 3 remaining are
> >>> > > > intentional:
> >>> > > > > > > one deferred perf nit, one that stays open until
> >>> > parquet-format
> >>> > > > > ships
> >>> > > > > > ALP,
> >>> > > > > > >
> >>> > > > > > > "
> >>> > > > > > >
> >>> > > > > > > With this both the
> >>> > > > > > > 1) C++ PR :
> >>> https://github.com/apache/arrow/pull/48345/changes
> >>> > > > > > > 2) Java PR :
> >>> https://github.com/apache/parquet-java/pull/3397
> >>> > > > > > >
> >>> > > > > > > Are cross language tested and range of test scenarios.
> >>> > > > > > >
> >>> > > > > > > Best
> >>> > > > > > > Prateek
> >>> > > > > > >
> >>> > > > > > >
> >>> > > > > > > On Mon, Jul 13, 2026 at 8:38 AM PRATEEK GAUR <
> >>> prtkgaur@gmail.com
> >>> > >
> >>> > > > > wrote:
> >>> > > > > > >
> >>> > > > > > >> Thanks Matt,
> >>> > > > > > >>
> >>> > > > > > >> I'll be on it this week.
> >>> > > > > > >>
> >>> > > > > > >> Best
> >>> > > > > > >> Prateek
> >>> > > > > > >>
> >>> > > > > > >> On Mon, Jul 13, 2026 at 8:36 AM Matt Topol <
> >>> > > zotthewizard@gmail.com>
> >>> > > > > > >> wrote:
> >>> > > > > > >>
> >>> > > > > > >>> There is also the Go implementation submitted by Arnav (
> >>> > > > > > >>> https://github.com/apache/arrow-go/pull/704) which is
> >>> waiting
> >>> > > for
> >>> > > > > > >>> updates.
> >>> > > > > > >>>
> >>> > > > > > >>> Just wanted to make sure it didn't get lost here.
> >>> > > > > > >>>
> >>> > > > > > >>> --Matt
> >>> > > > > > >>>
> >>> > > > > > >>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK GAUR <
> >>> > > prtkgaur@gmail.com>
> >>> > > > > > >>> wrote:
> >>> > > > > > >>>
> >>> > > > > > >>> > Hi all,
> >>> > > > > > >>> >
> >>> > > > > > >>> > I'd like to share a status update on the ALP encoding
> >>> effort
> >>> > > and
> >>> > > > > get
> >>> > > > > > a
> >>> > > > > > >>> feel
> >>> > > > > > >>> > from the community before starting a formal vote.
> >>> > > > > > >>> >
> >>> > > > > > >>> > We now have the specification plus implementations in
> >>> both
> >>> > > > > languages:
> >>> > > > > > >>> >
> >>> > > > > > >>> > - Spec: apache/parquet-format#557 — GH-533
> >>> Add
> >>> > ALP
> >>> > > > > > encoding
> >>> > > > > > >>> > specification
> >>> > > > > > >>> > - C++ (Arrow): apache/arrow#48345 — GH-48701
> >>> > > [C++][Parquet]
> >>> > > > > Add
> >>> > > > > > >>> ALPpd
> >>> > > > > > >>> > encoding
> >>> > > > > > >>> > - Java: apache/parquet-java#3397 — Parquet
> >>> Java
> >>> > ALP
> >>> > > > > > >>> > Implementation (by Vinoo)
> >>> > > > > > >>> >
> >>> > > > > > >>> > Current state:
> >>> > > > > > >>> >
> >>> > > > > > >>> > - The C++ implementation has been through 3-4
> rounds
> >>> of
> >>> > > > review.
> >>> > > > > > >>> > - The Java implementation has been tracking well
> and
> >>> has
> >>> > > had
> >>> > > > > > >>> initial
> >>> > > > > > >>> > reviews from contributors, with no major
> remaining
> >>> > > issues.
> >>> > > > > > >>> > - Cross-language compatibility tests are passing:
> the
> >>> > Arrow
> >>> > > > C++
> >>> > > > > > >>> decoder
> >>> > > > > > >>> > reads Java-written data bit-exactly across ~1.56M
> >>> > values
> >>> > > > and
> >>> > > > > 18
> >>> > > > > > >>> > fixtures, covering V1 and V2 pages, multiple
> vector
> >>> > > sizes,
> >>> > > > > and
> >>> > > > > > >>> > several
> >>> > > > > > >>> > real datasets — zero mismatches.
> >>> > > > > > >>> >
> >>> > > > > > >>> > As raised in the java-pr we will be adding coverage
> for
> >>> > > extreme
> >>> > > > > > values
> >>> > > > > > >>> > (those requiring
> >>> > > > > > >>> > 63-64 bits after FOR is applied) before we close a
> vote.
> >>> > We'll
> >>> > > > aim
> >>> > > > > > to
> >>> > > > > > >>> get
> >>> > > > > > >>> > that
> >>> > > > > > >>> > done in parallel.
> >>> > > > > > >>> >
> >>> > > > > > >>> > Plan: unless there are objections, I intend to start a
> >>> vote
> >>> > at
> >>> > > > the
> >>> > > > > > end
> >>> > > > > > >>> of
> >>> > > > > > >>> > this week/early
> >>> > > > > > >>> > next week.
> >>> > > > > > >>> >
> >>> > > > > > >>> > Thanks
> >>> > > > > > >>> > Prateek
> >>> > > > > > >>> >
> >>> > > > > > >>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <
> >>> > > prtkgaur@gmail.com>
> >>> > > > > > >>> wrote:
> >>> > > > > > >>> >
> >>> > > > > > >>> > > Hi Team,
> >>> > > > > > >>> > >
> >>> > > > > > >>> > > Just wanted to provide some updates on ALP.
> >>> > > > > > >>> > > Micah and I have done 3-4 rounds of review for the
> c++
> >>> PR :
> >>> > > > > > >>> > > https://github.com/apache/arrow/pull/48345
> >>> > > > > > >>> > >
> >>> > > > > > >>> > > For the Java implementation Vinoo has been working on
> >>> > > following
> >>> > > > > PR
> >>> > > > > > :
> >>> > > > > > >>> > > https://github.com/apache/parquet-java/pull/3397
> >>> > > > > > >>> > >
> >>> > > > > > >>> > > Best
> >>> > > > > > >>> > > Prateek
> >>> > > > > > >>> > >
> >>> > > > > > >>> > > On Tue, May 5, 2026 at 1:50 PM Micah Kornfield <
> >>> > > > > > >>> emkornfield@gmail.com>
> >>> > > > > > >>> > > wrote:
> >>> > > > > > >>> > >
> >>> > > > > > >>> > >> Hi Antoine,
> >>> > > > > > >>> > >>
> >>> > > > > > >>> > >> > Apologies if the question was already asked, but
> >>> should
> >>> > we
> >>> > > > > care
> >>> > > > > > >>> about
> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient
> >>> than
> >>> > > > > FLOAT16
> >>> > > > > > +
> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> >>> > > > > > >>> > >>
> >>> > > > > > >>> > >> It was. We thought we could defer it for the
> >>> following
> >>> > > > reasons:
> >>> > > > > > >>> > >> 1. It's not clear there are a lot of easy reference
> >>> > > datasets
> >>> > > > to
> >>> > > > > > >>> test
> >>> > > > > > >>> > its
> >>> > > > > > >>> > >> effectiveness. It does look like there might be one
> >>> or
> >>> > two
> >>> > > on
> >>> > > > > > >>> > huggingface
> >>> > > > > > >>> > >> (e.g.
> https://huggingface.co/datasets/kikitora/curdie
> >>> ).
> >>> > > > > > >>> > >> 2. It seemed likely that float 16 was more likely
> >>> used
> >>> > for
> >>> > > > > values
> >>> > > > > > >>> that
> >>> > > > > > >>> > >> were less likely to reduce to decimal values.
> >>> > > > > > >>> > >> 3. It could be added as an extension later if
> needed.
> >>> > > > > > >>> > >>
> >>> > > > > > >>> > >> Cheers,
> >>> > > > > > >>> > >> Micah
> >>> > > > > > >>> > >>
> >>> > > > > > >>> > >>
> >>> > > > > > >>> > >> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <
> >>> > > > > antoine@python.org
> >>> > > > > > >
> >>> > > > > > >>> > wrote:
> >>> > > > > > >>> > >>
> >>> > > > > > >>> > >> >
> >>> > > > > > >>> > >> > Hello,
> >>> > > > > > >>> > >> >
> >>> > > > > > >>> > >> > Apologies if the question was already asked, but
> >>> should
> >>> > we
> >>> > > > > care
> >>> > > > > > >>> about
> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient
> >>> than
> >>> > > > > FLOAT16
> >>> > > > > > +
> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> >>> > > > > > >>> > >> >
> >>> > > > > > >>> > >> > Regards
> >>> > > > > > >>> > >> >
> >>> > > > > > >>> > >> > Antoine.
> >>> > > > > > >>> > >> >
> >>> > > > > > >>> > >> >
> >>> > > > > > >>> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
> >>> > > > > > >>> > >> > > Thanks Andrew and Micah for review feedback on
> >>> the two
> >>> > > > PR's
> >>> > > > > > >>> > >> > > 1) (c++ arrow repo)
> >>> > > > > > >>> > >> https://github.com/apache/arrow/pull/48345/changes
> >>> > > > > > >>> > >> > > 2) (parquet-format repo)
> >>> > > > > > >>> > >> > https://github.com/apache/parquet-format/pull/557
> >>> > > > > > >>> > >> > >
> >>> > > > > > >>> > >> > > I have addressed all (unless I missed something)
> >>> > > comments
> >>> > > > on
> >>> > > > > > >>> the two
> >>> > > > > > >>> > >> > PR's.
> >>> > > > > > >>> > >> > >
> >>> > > > > > >>> > >> > > Best
> >>> > > > > > >>> > >> > > Prateek
> >>> > > > > > >>> > >> > >
> >>> > > > > > >>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <
> >>> > > > > > >>> prtkgaur@gmail.com>
> >>> > > > > > >>> > >> wrote:
> >>> > > > > > >>> > >> > >
> >>> > > > > > >>> > >> > >> Thanks Andrew and Micah.
> >>> > > > > > >>> > >> > >>
> >>> > > > > > >>> > >> > >> `fair amount of feedback on at least the
> >>> > > implementations`
> >>> > > > > > >>> > >> > >> For the c++ I have already started addressing
> the
> >>> > > > > feedback, I
> >>> > > > > > >>> > should
> >>> > > > > > >>> > >> be
> >>> > > > > > >>> > >> > >> done with that Monday/Tuesday.
> >>> > > > > > >>> > >> > >> I think Vinoo too has been making good progress
> >>> on
> >>> > the
> >>> > > > Java
> >>> > > > > > >>> > >> > implementation.
> >>> > > > > > >>> > >> > >>
> >>> > > > > > >>> > >> > >> Best
> >>> > > > > > >>> > >> > >> Prateek
> >>> > > > > > >>> > >> > >>
> >>> > > > > > >>> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <
> >>> > > > > > >>> > andrewlamb11@gmail.com
> >>> > > > > > >>> > >> >
> >>> > > > > > >>> > >> > >> wrote:
> >>> > > > > > >>> > >> > >>
> >>> > > > > > >>> > >> > >>> Got it. Thank you for the clarification -- I
> >>> will
> >>> > try
> >>> > > > and
> >>> > > > > > look
> >>> > > > > > >>> > into
> >>> > > > > > >>> > >> the
> >>> > > > > > >>> > >> > >>> spec and the Rust implementation[1] in this
> next
> >>> > week
> >>> > > > > > >>> > >> > >>>
> >>> > > > > > >>> > >> > >>> [1]:
> >>> https://github.com/apache/arrow-rs/pull/9372
> >>> > > > > > >>> > >> > >>>
> >>> > > > > > >>> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah
> >>> Kornfield <
> >>> > > > > > >>> > >> > emkornfield@gmail.com>
> >>> > > > > > >>> > >> > >>> wrote:
> >>> > > > > > >>> > >> > >>>
> >>> > > > > > >>> > >> > >>>> Hi Andrew,
> >>> > > > > > >>> > >> > >>>> I think there is a fair amount of feedback on
> >>> at
> >>> > > least
> >>> > > > > the
> >>> > > > > > >>> > >> > >>>> implementations, typically I think we've
> waited
> >>> > till
> >>> > > > they
> >>> > > > > > are
> >>> > > > > > >>> > >> close to
> >>> > > > > > >>> > >> > >>>> mergeable before a final vote. Otherwise I
> >>> agree
> >>> > we
> >>> > > > are
> >>> > > > > > very
> >>> > > > > > >>> > >> close.
> >>> > > > > > >>> > >> > >>>>
> >>> > > > > > >>> > >> > >>>> -Micah
> >>> > > > > > >>> > >> > >>>>
> >>> > > > > > >>> > >> > >>>> On Saturday, April 25, 2026, Andrew Lamb <
> >>> > > > > > >>> andrewlamb11@gmail.com
> >>> > > > > > >>> > >
> >>> > > > > > >>> > >> > wrote:
> >>> > > > > > >>> > >> > >>>>
> >>> > > > > > >>> > >> > >>>>> Thanks Prateek,
> >>> > > > > > >>> > >> > >>>>>
> >>> > > > > > >>> > >> > >>>>> I think from this content it looks to me
> like
> >>> we
> >>> > are
> >>> > > > > ready
> >>> > > > > > >>> to
> >>> > > > > > >>> > >> start a
> >>> > > > > > >>> > >> > >>>>> vote to explicitly accept ALP into Parquet
> >>> > > > > > >>> > >> > >>>>>
> >>> > > > > > >>> > >> > >>>>> Does anyone know of a reason we should
> >>> postpone it
> >>> > > for
> >>> > > > > > >>> longer?
> >>> > > > > > >>> > >> > >>>>> Perhaps someone needs some more time to
> >>> review?
> >>> > > > > > >>> > >> > >>>>>
> >>> > > > > > >>> > >> > >>>>> Andrew
> >>> > > > > > >>> > >> > >>>>>
> >>> > > > > > >>> > >> > >>>>>
> >>> > > > > > >>> > >> > >>>>>
> >>> > > > > > >>> > >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK
> GAUR <
> >>> > > > > > >>> > prtkgaur@gmail.com>
> >>> > > > > > >>> > >> > >>>>> wrote:
> >>> > > > > > >>> > >> > >>>>>
> >>> > > > > > >>> > >> > >>>>>> Hi team,
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>> Hope everyone is doing well. I got a chance
> >>> to
> >>> > work
> >>> > > > > > >>> through all
> >>> > > > > > >>> > >> the
> >>> > > > > > >>> > >> > >>>>>> remaining feedback and update the spec doc.
> >>> Here
> >>> > > are
> >>> > > > > the
> >>> > > > > > >>> new
> >>> > > > > > >>> > >> > artifacts
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>> 1) Spec document :
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> >
> >>> > > > > > >>> > >>
> >>> > > > > > >>> >
> >>> > > > > > >>>
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>> 2) Spec document in parquet format repo :
> >>> > > > > > >>> > >> > >>>>>>
> >>> > https://github.com/apache/parquet-format/pull/557
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>> 3) Alp implementation in arrow c++ repo :
> >>> > > > > > >>> > >> > >>>>>>
> >>> > https://github.com/apache/arrow/pull/48345/changes
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>> 4) Alp implementation in parquet-java repo
> :
> >>> Work
> >>> > > for
> >>> > > > > > >>> Vinoo and
> >>> > > > > > >>> > >> > Julien
> >>> > > > > > >>> > >> > >>>>>>
> >>> > https://github.com/apache/parquet-java/pull/3397
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>> 5) PR with test and benchmarking artifacts
> in
> >>> > > > > > >>> parquet-testing
> >>> > > > > > >>> > >> repo :
> >>> > > > > > >>> > >> > >>>>>>
> >>> > https://github.com/apache/parquet-testing/pull/100
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>> And
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>> - Go : Arnav just submitted an in
> >>> progress
> >>> > > > > > >>> implementation
> >>> > > > > > >>> > in
> >>> > > > > > >>> > >> Go.
> >>> > > > > > >>> > >> > >>>>>>
> >>> https://github.com/apache/arrow-go/pull/704
> >>> > (I
> >>> > > > > > haven't
> >>> > > > > > >>> > >> started
> >>> > > > > > >>> > >> > >>>>>> looking at it yet)
> >>> > > > > > >>> > >> > >>>>>> - Rust : I remember Andrew mentioned
> that
> >>> > this
> >>> > > > work
> >>> > > > > > is
> >>> > > > > > >>> also
> >>> > > > > > >>> > >> in
> >>> > > > > > >>> > >> > >>>>>> progress (So 4 languages!)
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>> *Arrow C++ implementation *
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>> The PR is out and was also used by Antoine
> to
> >>> > > report
> >>> > > > > the
> >>> > > > > > >>> > numbers
> >>> > > > > > >>> > >> as
> >>> > > > > > >>> > >> > >>>>>> reported here. Micah and Konstantin have
> >>> given 1
> >>> > > > round
> >>> > > > > of
> >>> > > > > > >>> > >> feedback
> >>> > > > > > >>> > >> > >>>>>> and I'm addressing them today. Please note
> >>> that
> >>> > the
> >>> > > > > > default
> >>> > > > > > >>> > >> > >>>>>> optimization flag for compiling is O2 and
> not
> >>> > Q3. I
> >>> > > > got
> >>> > > > > > >>> around
> >>> > > > > > >>> > >> 70%
> >>> > > > > > >>> > >> > >>>>>> performance improvement in the decoding
> speed
> >>> > when
> >>> > > > > using
> >>> > > > > > >>> the O3
> >>> > > > > > >>> > >> > flag.
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>> *Parqet-MR Java implementation (working
> with
> >>> > Vinoo
> >>> > > > and
> >>> > > > > > >>> Julien)
> >>> > > > > > >>> > >> and
> >>> > > > > > >>> > >> > **Cross
> >>> > > > > > >>> > >> > >>>>>> Language testing*
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>> Let me know if you have any questions
> or
> >>> > > > feedback.
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>> Now pasting some performance numbers
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>> Table 1: C++ ALP Double Decode — Spotify
> >>> > Columns
> >>> > > > > > >>> (Graviton
> >>> > > > > > >>> > 3,
> >>> > > > > > >>> > >> ARM
> >>> > > > > > >>> > >> > >>>>>> Neoverse V1)
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>>
> >>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>> │ Column │ -O2 (MB/s) │ -O3
> >>> > > (MB/s) │
> >>> > > > > > >>> Speedup │
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>>
> >>> ├──────────────────┼──────────────┼──────────────┼─────────┤
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>> │ valence │ 3,155 │
> >>> 5,523
> >>> > > │
> >>> > > > > > >>> 1.75x │
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>> │ danceability │ 3,233 │
> >>> 5,685
> >>> > > │
> >>> > > > > > >>> 1.76x │
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>> │ energy │ 3,197 │
> >>> 5,652
> >>> > > │
> >>> > > > > > >>> 1.77x │
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>> │ loudness │ 3,186 │
> >>> 5,473
> >>> > > │
> >>> > > > > > >>> 1.72x │
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>>
> >>> └──────────────────┴──────────────┴──────────────┴─────────┘
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK
> GAUR
> >>> <
> >>> > > > > > >>> > prtkgaur@gmail.com
> >>> > > > > > >>> > >> >
> >>> > > > > > >>> > >> > >>>>>> wrote:
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>>> @Micah Kornfield <em...@gmail.com>
> :
> >>> Got
> >>> > > it.
> >>> > > > > > >>> > >> > >>>>>>>
> >>> > > > > > >>> > >> > >>>>>>> @Andrew Lamb <an...@gmail.com>
> >>> > > > > > >>> > >> > >>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>> Do you think it would be good to start
> >>> moving
> >>> > the
> >>> > > > > spec
> >>> > > > > > >>> > >> development
> >>> > > > > > >>> > >> > >>>>>>>> into
> >>> > > > > > >>> > >> > >>>>>>>> markdown format, in preparation for
> >>> finalizing
> >>> > > it?
> >>> > > > > > >>> > >> > >>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>
> >>> > > > > > >>> > >> > >>>>>>> Yes I'll update the numbers for some of
> the
> >>> > > > examples I
> >>> > > > > > >>> have in
> >>> > > > > > >>> > >> the
> >>> > > > > > >>> > >> > >>>>>>> spec based
> >>> > > > > > >>> > >> > >>>>>>> on the updated header size. Then we should
> >>> be
> >>> > good
> >>> > > > to
> >>> > > > > go
> >>> > > > > > >>> for
> >>> > > > > > >>> > the
> >>> > > > > > >>> > >> > >>>>>>> markdown format.
> >>> > > > > > >>> > >> > >>>>>>>
> >>> > > > > > >>> > >> > >>>>>>> Thanks everyone!
> >>> > > > > > >>> > >> > >>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>> Andrew
> >>> > > > > > >>> > >> > >>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK
> >>> GAUR <
> >>> > > > > > >>> > >> prtkgaur@gmail.com>
> >>> > > > > > >>> > >> > >>>>>>>> wrote:
> >>> > > > > > >>> > >> > >>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>> Hi team,
> >>> > > > > > >>> > >> > >>>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>> 1) Andrew
> >>> > > > > > >>> > >> > >>>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for working on test files.
> >>> My PR
> >>> > > did
> >>> > > > > add
> >>> > > > > > >>> all
> >>> > > > > > >>> > the
> >>> > > > > > >>> > >> > test
> >>> > > > > > >>> > >> > >>>>>>>> files I
> >>> > > > > > >>> > >> > >>>>>>>>> used to benchmark on datasets. Maybe
> >>> we
> >>> > can
> >>> > > > club
> >>> > > > > > it
> >>> > > > > > >>> > >> together.
> >>> > > > > > >>> > >> > >>>>>>>> WIll also
> >>> > > > > > >>> > >> > >>>>>>>>> aid
> >>> > > > > > >>> > >> > >>>>>>>>> cross language testing
> >>> > > > > > >>> > >> > >>>>>>>>> - Kosta Tarasov working on Rust
> >>> > > > implementation.
> >>> > > > > > >>> This is
> >>> > > > > > >>> > >> > great.
> >>> > > > > > >>> > >> > >>>>>>>> Thanks
> >>> > > > > > >>> > >> > >>>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>> 2) Antoine
> >>> > > > > > >>> > >> > >>>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks a lot for reporting the
> >>> numbers
> >>> > on
> >>> > > > AMD.
> >>> > > > > > >>> Looks
> >>> > > > > > >>> > >> like
> >>> > > > > > >>> > >> > you
> >>> > > > > > >>> > >> > >>>>>>>> are
> >>> > > > > > >>> > >> > >>>>>>>>> getting 8X the decoding performance
> of
> >>> > BSS.
> >>> > > > This
> >>> > > > > > is
> >>> > > > > > >>> > >> > amazing!!.
> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for acknowledging the
> >>> sampling
> >>> > > > design.
> >>> > > > > > >>> > >> > >>>>>>>>> - I agree with you on Fastlanes. In
> >>> some
> >>> > > crude
> >>> > > > > > >>> > >> experiments I
> >>> > > > > > >>> > >> > >>>>>>>> didn't get
> >>> > > > > > >>> > >> > >>>>>>>>> a good perf benefit from it on
> >>> Graviton3
> >>> > > (but
> >>> > > > > > maybe
> >>> > > > > > >>> > there
> >>> > > > > > >>> > >> was
> >>> > > > > > >>> > >> > >>>>>>>> something
> >>> > > > > > >>> > >> > >>>>>>>>> wrong with my implementation).
> >>> > > > > > >>> > >> > >>>>>>>>> - Locking the 16bit exception
> >>> encoding for
> >>> > > the
> >>> > > > > > spec
> >>> > > > > > >>> in
> >>> > > > > > >>> > >> this
> >>> > > > > > >>> > >> > >>>>>>>> case.
> >>> > > > > > >>> > >> > >>>>>>>>> - Awesome I think we have solved for
> >>> all
> >>> > > open
> >>> > > > > > >>> questions
> >>> > > > > > >>> > >> minus
> >>> > > > > > >>> > >> > >>>>>>>> the
> >>> > > > > > >>> > >> > >>>>>>>>> version byte :). (will get back on
> >>> this
> >>> > > soon)
> >>> > > > > > >>> > >> > >>>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>> 3) Micah
> >>> > > > > > >>> > >> > >>>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>> - FastLanes : The current spec does
> >>> allow
> >>> > > for
> >>> > > > > > using
> >>> > > > > > >>> > >> FastLane
> >>> > > > > > >>> > >> > >>>>>>>> with the
> >>> > > > > > >>> > >> > >>>>>>>>> configurable enum value for layout.
> We
> >>> > > should
> >>> > > > be
> >>> > > > > > >>> able to
> >>> > > > > > >>> > >> > inject
> >>> > > > > > >>> > >> > >>>>>>>> any
> >>> > > > > > >>> > >> > >>>>>>>>> layout
> >>> > > > > > >>> > >> > >>>>>>>>> in the current design.
> >>> > > > > > >>> > >> > >>>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>> Working on resolving all remaining open
> >>> > comments
> >>> > > > on
> >>> > > > > > the
> >>> > > > > > >>> spec
> >>> > > > > > >>> > >> this
> >>> > > > > > >>> > >> > >>>>>>>> week.
> >>> > > > > > >>> > >> > >>>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>> Best
> >>> > > > > > >>> > >> > >>>>>>>>> Prateek
> >>> > > > > > >>> > >> > >>>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve
> >>> > Loughran <
> >>> > > > > > >>> > >> > >>>>>>>> stevel@cloudera.com>
> >>> > > > > > >>> > >> > >>>>>>>>> wrote:
> >>> > > > > > >>> > >> > >>>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah
> >>> Kornfield
> >>> > <
> >>> > > > > > >>> > >> > >>>>>>>> emkornfield@gmail.com>
> >>> > > > > > >>> > >> > >>>>>>>>>> wrote:
> >>> > > > > > >>> > >> > >>>>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>>>> It looks like the actual issue
> >>> described for
> >>> > > ORC
> >>> > > > > in
> >>> > > > > > >>> the
> >>> > > > > > >>> > >> paper
> >>> > > > > > >>> > >> > >>>>>>>> is that
> >>> > > > > > >>> > >> > >>>>>>>>> it
> >>> > > > > > >>> > >> > >>>>>>>>>>> has multiple sub-encodings in a batch.
> >>> This
> >>> > > is
> >>> > > > > > >>> different
> >>> > > > > > >>> > >> then
> >>> > > > > > >>> > >> > >>>>>>>> the
> >>> > > > > > >>> > >> > >>>>>>>>> design
> >>> > > > > > >>> > >> > >>>>>>>>>>> proposed here where there is still
> fixed
> >>> > > > encoding
> >>> > > > > > per
> >>> > > > > > >>> page
> >>> > > > > > >>> > >> in
> >>> > > > > > >>> > >> > >>>>>>>> parquet.
> >>> > > > > > >>> > >> > >>>>>>>>>>> Given reasonably sized pages I don't
> >>> think
> >>> > > > branch
> >>> > > > > > >>> > >> > >>>>>>>> misprediction should
> >>> > > > > > >>> > >> > >>>>>>>>>> be a
> >>> > > > > > >>> > >> > >>>>>>>>>>> big issue for new encodings. I agree
> >>> that
> >>> > we
> >>> > > > > should
> >>> > > > > > >>> be
> >>> > > > > > >>> > >> > >>>>>>>> conservative in
> >>> > > > > > >>> > >> > >>>>>>>>>>> general for adding new encodings.
> >>> > > > > > >>> > >> > >>>>>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>>> +1
> >>> > > > > > >>> > >> > >>>>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>
> >>> > > > > > >>> > >> > >
> >>> > > > > > >>> > >> >
> >>> > > > > > >>> > >> >
> >>> > > > > > >>> > >> >
> >>> > > > > > >>> > >>
> >>> > > > > > >>> > >
> >>> > > > > > >>> >
> >>> > > > > > >>>
> >>> > > > > > >>
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> >>
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Divjot Arora via dev <de...@parquet.apache.org>.
Yes, that’s my understanding as well. However, as you noted, the versioning
discussion is still in early stages and hasn’t prescribed concrete
mechanisms yet. If ALP goes into parquet-format 2.14.0, there’s no
indication that it’s in preview or that writers shouldn’t use it by default
yet. Given that we’ve held off on merging other incompatible changes (e.g
removal of path_in_schema) until the versioning discussion closes, my
preference would be to leave ALP un-merged for the time being as well.
— Div
On Fri, Jul 31, 2026 at 08:59 Andrew Lamb <an...@gmail.com> wrote:
> After re-reading Julien's Versioning Proposal, my understanding is that if
> we adopt that system, we will somehow mark ALP in "Preview" in
> parquet-format for release Version 3.
>
> The versioning document doesn't seem to proscribe the particular method
> for annotating "preview" features -- I think clearly marking them as such
> in parquet.thrift as well as reflecting that on the
> https://parquet.apache.org/docs/file-format/versions/ page somehow would
> be adequate.
>
> Andrew
>
> On Thu, Jul 30, 2026 at 11:50 AM Andrew Lamb <an...@gmail.com>
> wrote:
>
>> In my mind ALP is in the same position as other encodings that were added
>> recently[1] (such as BYTE_STREAM_SPLIT for non floating point types)
>>
>> That is to say it will be available in the latest open source writers and
>> readers, but realistically not widely used in the ecosystem where
>> compatibility is a concern (a large part)
>>
>> I view Julian's work[2][3] to gather consensus on how to name
>> combinations of features (e.g. "versions") as the mechanism by which we
>> will accelerate the adoption of this change (along with other forward
>> incompatible changes) across the ecosystem.
>>
>> > Will we exclude this change when cutting parquet-format v2.14.0?
>>
>> No I don't think so -- I would expect it to go out in the next
>> parquet-format version as do other changes in the spec.
>>
>> Andrew
>>
>> [1]: https://parquet.apache.org/docs/file-format/versions/
>> [2]: https://lists.apache.org/thread/xmnj8h0h8ozmrgox5tydhhs7yz7q2hss
>> [3]: https://lists.apache.org/thread/qlf8lg90gqq41lllyy9mk2f0skqvftqj
>>
>>
>> On Thu, Jul 30, 2026 at 3:51 AM Divjot Arora via dev <
>> dev@parquet.apache.org> wrote:
>>
>>> Hi Andrew,
>>>
>>> How does this interact with the ongoing versioning discussion? Given that
>>> new encodings are not forward compatible, I would assume ALP is included
>>> in
>>> Parquet V<next> (V3?), but it's not clear to me how that works if it's
>>> merged to the parquet-format master branch now. Will we exclude this
>>> change
>>> when cutting parquet-format v2.14.0?
>>>
>>> -- Div
>>>
>>> On Thu, Jul 30, 2026 at 8:51 AM Andrew Lamb <an...@gmail.com>
>>> wrote:
>>>
>>> > Hello,
>>> >
>>> > As an update, thanks to the epic work from many contributors,
>>> especially
>>> > Prateek, I just merged the parquet-format change for ALP[1]. We plan to
>>> > make additional editorial The remaining items I know of are merging the
>>> > examples from parquet-testing[2], and then of course the various
>>> > implementations
>>> >
>>> > Thanks again everyone. This is a great step forward
>>> > Andrew
>>> >
>>> > [1]: https://github.com/apache/parquet-format/pull/557
>>> > [2]: https://github.com/apache/parquet-testing/pull/100
>>> >
>>> > On Tue, Jul 21, 2026 at 4:39 PM Julien Le Dem <ju...@apache.org>
>>> wrote:
>>> >
>>> > > It was also in my spam folder... :(
>>> > > I've seen a bunch of apache list emails go to my spam recently.
>>> > >
>>> > > On Tue, Jul 21, 2026 at 10:58 AM Russell Spitzer <
>>> > > russell.spitzer@gmail.com>
>>> > > wrote:
>>> > >
>>> > > > Yes, my hope was that my vote would kick it out of spam but we
>>> > definitely
>>> > > > had two other votes that also went to spam for me.
>>> > > >
>>> > > > For anyone looking the thread is here
>>> > > > https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt
>>> > > >
>>> > > > On Tue, Jul 21, 2026 at 10:57 AM Micah Kornfield <
>>> > emkornfield@gmail.com>
>>> > > > wrote:
>>> > > >
>>> > > > > I've seen other's votes not go to spam, so hopefully people will
>>> see
>>> > it
>>> > > > in
>>> > > > > there inboxes (or see this thread and look in there spam).
>>> > > > >
>>> > > > > On Tue, Jul 21, 2026 at 10:52 AM PRATEEK GAUR <
>>> prtkgaur@gmail.com>
>>> > > > wrote:
>>> > > > >
>>> > > > > > Okay,
>>> > > > > >
>>> > > > > > I sent out the voting email yesterday and so far I've heard
>>> from 3
>>> > > > people
>>> > > > > > that it ended up in their "spam folder". Dev VPN to be blamed
>>> for
>>> > > it?.
>>> > > > > >
>>> > > > > > What's the protocol here?
>>> > > > > > I let people hunt for the email in their spam folder or send
>>> out
>>> > > > another
>>> > > > > > one?
>>> > > > > >
>>> > > > > > Best
>>> > > > > > Prateek
>>> > > > > >
>>> > > > > > On Sun, Jul 19, 2026 at 10:47 PM PRATEEK GAUR <
>>> prtkgaur@gmail.com>
>>> > > > > wrote:
>>> > > > > >
>>> > > > > > > Hi team,
>>> > > > > > >
>>> > > > > > > Vinoo has added the tests which were requested in the last
>>> round
>>> > of
>>> > > > > > review.
>>> > > > > > > In his own words.
>>> > > > > > >
>>> > > > > > > "
>>> > > > > > > The extreme value tests Micah wanted are done and pushed. I
>>> added
>>> > > > > > coverage
>>> > > > > > > for values that need the full FOR bit width after the frame
>>> of
>>> > > > > reference
>>> > > > > > is
>>> > > > > > > applied: 64-bit (the case where the signed max minus min
>>> > > subtraction
>>> > > > > > > overflows), 63-bit (the non-overflow case), and 32-bit for
>>> > floats.
>>> > > > All
>>> > > > > > > lossless with zero exceptions. I also verified them through
>>> Arrow
>>> > > C++
>>> > > > > ALP
>>> > > > > > > reader (the local build), so the Java-written extreme columns
>>> > > decode
>>> > > > > > > correctly in C++, bit-exact against the expected values."
>>> > > > > > >
>>> > > > > > > State of the PR (#3397):
>>> > > > > > >
>>> > > > > > > - Merges cleanly with master (I pulled master in and
>>> resolved
>>> > > the
>>> > > > > > > conflicts).
>>> > > > > > > - 41 of 44 review threads resolved. The 3 remaining are
>>> > > > intentional:
>>> > > > > > > one deferred perf nit, one that stays open until
>>> > parquet-format
>>> > > > > ships
>>> > > > > > ALP,
>>> > > > > > >
>>> > > > > > > "
>>> > > > > > >
>>> > > > > > > With this both the
>>> > > > > > > 1) C++ PR :
>>> https://github.com/apache/arrow/pull/48345/changes
>>> > > > > > > 2) Java PR :
>>> https://github.com/apache/parquet-java/pull/3397
>>> > > > > > >
>>> > > > > > > Are cross language tested and range of test scenarios.
>>> > > > > > >
>>> > > > > > > Best
>>> > > > > > > Prateek
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > On Mon, Jul 13, 2026 at 8:38 AM PRATEEK GAUR <
>>> prtkgaur@gmail.com
>>> > >
>>> > > > > wrote:
>>> > > > > > >
>>> > > > > > >> Thanks Matt,
>>> > > > > > >>
>>> > > > > > >> I'll be on it this week.
>>> > > > > > >>
>>> > > > > > >> Best
>>> > > > > > >> Prateek
>>> > > > > > >>
>>> > > > > > >> On Mon, Jul 13, 2026 at 8:36 AM Matt Topol <
>>> > > zotthewizard@gmail.com>
>>> > > > > > >> wrote:
>>> > > > > > >>
>>> > > > > > >>> There is also the Go implementation submitted by Arnav (
>>> > > > > > >>> https://github.com/apache/arrow-go/pull/704) which is
>>> waiting
>>> > > for
>>> > > > > > >>> updates.
>>> > > > > > >>>
>>> > > > > > >>> Just wanted to make sure it didn't get lost here.
>>> > > > > > >>>
>>> > > > > > >>> --Matt
>>> > > > > > >>>
>>> > > > > > >>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK GAUR <
>>> > > prtkgaur@gmail.com>
>>> > > > > > >>> wrote:
>>> > > > > > >>>
>>> > > > > > >>> > Hi all,
>>> > > > > > >>> >
>>> > > > > > >>> > I'd like to share a status update on the ALP encoding
>>> effort
>>> > > and
>>> > > > > get
>>> > > > > > a
>>> > > > > > >>> feel
>>> > > > > > >>> > from the community before starting a formal vote.
>>> > > > > > >>> >
>>> > > > > > >>> > We now have the specification plus implementations in
>>> both
>>> > > > > languages:
>>> > > > > > >>> >
>>> > > > > > >>> > - Spec: apache/parquet-format#557 — GH-533
>>> Add
>>> > ALP
>>> > > > > > encoding
>>> > > > > > >>> > specification
>>> > > > > > >>> > - C++ (Arrow): apache/arrow#48345 — GH-48701
>>> > > [C++][Parquet]
>>> > > > > Add
>>> > > > > > >>> ALPpd
>>> > > > > > >>> > encoding
>>> > > > > > >>> > - Java: apache/parquet-java#3397 — Parquet
>>> Java
>>> > ALP
>>> > > > > > >>> > Implementation (by Vinoo)
>>> > > > > > >>> >
>>> > > > > > >>> > Current state:
>>> > > > > > >>> >
>>> > > > > > >>> > - The C++ implementation has been through 3-4 rounds
>>> of
>>> > > > review.
>>> > > > > > >>> > - The Java implementation has been tracking well and
>>> has
>>> > > had
>>> > > > > > >>> initial
>>> > > > > > >>> > reviews from contributors, with no major remaining
>>> > > issues.
>>> > > > > > >>> > - Cross-language compatibility tests are passing: the
>>> > Arrow
>>> > > > C++
>>> > > > > > >>> decoder
>>> > > > > > >>> > reads Java-written data bit-exactly across ~1.56M
>>> > values
>>> > > > and
>>> > > > > 18
>>> > > > > > >>> > fixtures, covering V1 and V2 pages, multiple vector
>>> > > sizes,
>>> > > > > and
>>> > > > > > >>> > several
>>> > > > > > >>> > real datasets — zero mismatches.
>>> > > > > > >>> >
>>> > > > > > >>> > As raised in the java-pr we will be adding coverage for
>>> > > extreme
>>> > > > > > values
>>> > > > > > >>> > (those requiring
>>> > > > > > >>> > 63-64 bits after FOR is applied) before we close a vote.
>>> > We'll
>>> > > > aim
>>> > > > > > to
>>> > > > > > >>> get
>>> > > > > > >>> > that
>>> > > > > > >>> > done in parallel.
>>> > > > > > >>> >
>>> > > > > > >>> > Plan: unless there are objections, I intend to start a
>>> vote
>>> > at
>>> > > > the
>>> > > > > > end
>>> > > > > > >>> of
>>> > > > > > >>> > this week/early
>>> > > > > > >>> > next week.
>>> > > > > > >>> >
>>> > > > > > >>> > Thanks
>>> > > > > > >>> > Prateek
>>> > > > > > >>> >
>>> > > > > > >>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <
>>> > > prtkgaur@gmail.com>
>>> > > > > > >>> wrote:
>>> > > > > > >>> >
>>> > > > > > >>> > > Hi Team,
>>> > > > > > >>> > >
>>> > > > > > >>> > > Just wanted to provide some updates on ALP.
>>> > > > > > >>> > > Micah and I have done 3-4 rounds of review for the c++
>>> PR :
>>> > > > > > >>> > > https://github.com/apache/arrow/pull/48345
>>> > > > > > >>> > >
>>> > > > > > >>> > > For the Java implementation Vinoo has been working on
>>> > > following
>>> > > > > PR
>>> > > > > > :
>>> > > > > > >>> > > https://github.com/apache/parquet-java/pull/3397
>>> > > > > > >>> > >
>>> > > > > > >>> > > Best
>>> > > > > > >>> > > Prateek
>>> > > > > > >>> > >
>>> > > > > > >>> > > On Tue, May 5, 2026 at 1:50 PM Micah Kornfield <
>>> > > > > > >>> emkornfield@gmail.com>
>>> > > > > > >>> > > wrote:
>>> > > > > > >>> > >
>>> > > > > > >>> > >> Hi Antoine,
>>> > > > > > >>> > >>
>>> > > > > > >>> > >> > Apologies if the question was already asked, but
>>> should
>>> > we
>>> > > > > care
>>> > > > > > >>> about
>>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient
>>> than
>>> > > > > FLOAT16
>>> > > > > > +
>>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
>>> > > > > > >>> > >>
>>> > > > > > >>> > >> It was. We thought we could defer it for the
>>> following
>>> > > > reasons:
>>> > > > > > >>> > >> 1. It's not clear there are a lot of easy reference
>>> > > datasets
>>> > > > to
>>> > > > > > >>> test
>>> > > > > > >>> > its
>>> > > > > > >>> > >> effectiveness. It does look like there might be one
>>> or
>>> > two
>>> > > on
>>> > > > > > >>> > huggingface
>>> > > > > > >>> > >> (e.g. https://huggingface.co/datasets/kikitora/curdie
>>> ).
>>> > > > > > >>> > >> 2. It seemed likely that float 16 was more likely
>>> used
>>> > for
>>> > > > > values
>>> > > > > > >>> that
>>> > > > > > >>> > >> were less likely to reduce to decimal values.
>>> > > > > > >>> > >> 3. It could be added as an extension later if needed.
>>> > > > > > >>> > >>
>>> > > > > > >>> > >> Cheers,
>>> > > > > > >>> > >> Micah
>>> > > > > > >>> > >>
>>> > > > > > >>> > >>
>>> > > > > > >>> > >> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <
>>> > > > > antoine@python.org
>>> > > > > > >
>>> > > > > > >>> > wrote:
>>> > > > > > >>> > >>
>>> > > > > > >>> > >> >
>>> > > > > > >>> > >> > Hello,
>>> > > > > > >>> > >> >
>>> > > > > > >>> > >> > Apologies if the question was already asked, but
>>> should
>>> > we
>>> > > > > care
>>> > > > > > >>> about
>>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient
>>> than
>>> > > > > FLOAT16
>>> > > > > > +
>>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
>>> > > > > > >>> > >> >
>>> > > > > > >>> > >> > Regards
>>> > > > > > >>> > >> >
>>> > > > > > >>> > >> > Antoine.
>>> > > > > > >>> > >> >
>>> > > > > > >>> > >> >
>>> > > > > > >>> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
>>> > > > > > >>> > >> > > Thanks Andrew and Micah for review feedback on
>>> the two
>>> > > > PR's
>>> > > > > > >>> > >> > > 1) (c++ arrow repo)
>>> > > > > > >>> > >> https://github.com/apache/arrow/pull/48345/changes
>>> > > > > > >>> > >> > > 2) (parquet-format repo)
>>> > > > > > >>> > >> > https://github.com/apache/parquet-format/pull/557
>>> > > > > > >>> > >> > >
>>> > > > > > >>> > >> > > I have addressed all (unless I missed something)
>>> > > comments
>>> > > > on
>>> > > > > > >>> the two
>>> > > > > > >>> > >> > PR's.
>>> > > > > > >>> > >> > >
>>> > > > > > >>> > >> > > Best
>>> > > > > > >>> > >> > > Prateek
>>> > > > > > >>> > >> > >
>>> > > > > > >>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <
>>> > > > > > >>> prtkgaur@gmail.com>
>>> > > > > > >>> > >> wrote:
>>> > > > > > >>> > >> > >
>>> > > > > > >>> > >> > >> Thanks Andrew and Micah.
>>> > > > > > >>> > >> > >>
>>> > > > > > >>> > >> > >> `fair amount of feedback on at least the
>>> > > implementations`
>>> > > > > > >>> > >> > >> For the c++ I have already started addressing the
>>> > > > > feedback, I
>>> > > > > > >>> > should
>>> > > > > > >>> > >> be
>>> > > > > > >>> > >> > >> done with that Monday/Tuesday.
>>> > > > > > >>> > >> > >> I think Vinoo too has been making good progress
>>> on
>>> > the
>>> > > > Java
>>> > > > > > >>> > >> > implementation.
>>> > > > > > >>> > >> > >>
>>> > > > > > >>> > >> > >> Best
>>> > > > > > >>> > >> > >> Prateek
>>> > > > > > >>> > >> > >>
>>> > > > > > >>> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <
>>> > > > > > >>> > andrewlamb11@gmail.com
>>> > > > > > >>> > >> >
>>> > > > > > >>> > >> > >> wrote:
>>> > > > > > >>> > >> > >>
>>> > > > > > >>> > >> > >>> Got it. Thank you for the clarification -- I
>>> will
>>> > try
>>> > > > and
>>> > > > > > look
>>> > > > > > >>> > into
>>> > > > > > >>> > >> the
>>> > > > > > >>> > >> > >>> spec and the Rust implementation[1] in this next
>>> > week
>>> > > > > > >>> > >> > >>>
>>> > > > > > >>> > >> > >>> [1]:
>>> https://github.com/apache/arrow-rs/pull/9372
>>> > > > > > >>> > >> > >>>
>>> > > > > > >>> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah
>>> Kornfield <
>>> > > > > > >>> > >> > emkornfield@gmail.com>
>>> > > > > > >>> > >> > >>> wrote:
>>> > > > > > >>> > >> > >>>
>>> > > > > > >>> > >> > >>>> Hi Andrew,
>>> > > > > > >>> > >> > >>>> I think there is a fair amount of feedback on
>>> at
>>> > > least
>>> > > > > the
>>> > > > > > >>> > >> > >>>> implementations, typically I think we've waited
>>> > till
>>> > > > they
>>> > > > > > are
>>> > > > > > >>> > >> close to
>>> > > > > > >>> > >> > >>>> mergeable before a final vote. Otherwise I
>>> agree
>>> > we
>>> > > > are
>>> > > > > > very
>>> > > > > > >>> > >> close.
>>> > > > > > >>> > >> > >>>>
>>> > > > > > >>> > >> > >>>> -Micah
>>> > > > > > >>> > >> > >>>>
>>> > > > > > >>> > >> > >>>> On Saturday, April 25, 2026, Andrew Lamb <
>>> > > > > > >>> andrewlamb11@gmail.com
>>> > > > > > >>> > >
>>> > > > > > >>> > >> > wrote:
>>> > > > > > >>> > >> > >>>>
>>> > > > > > >>> > >> > >>>>> Thanks Prateek,
>>> > > > > > >>> > >> > >>>>>
>>> > > > > > >>> > >> > >>>>> I think from this content it looks to me like
>>> we
>>> > are
>>> > > > > ready
>>> > > > > > >>> to
>>> > > > > > >>> > >> start a
>>> > > > > > >>> > >> > >>>>> vote to explicitly accept ALP into Parquet
>>> > > > > > >>> > >> > >>>>>
>>> > > > > > >>> > >> > >>>>> Does anyone know of a reason we should
>>> postpone it
>>> > > for
>>> > > > > > >>> longer?
>>> > > > > > >>> > >> > >>>>> Perhaps someone needs some more time to
>>> review?
>>> > > > > > >>> > >> > >>>>>
>>> > > > > > >>> > >> > >>>>> Andrew
>>> > > > > > >>> > >> > >>>>>
>>> > > > > > >>> > >> > >>>>>
>>> > > > > > >>> > >> > >>>>>
>>> > > > > > >>> > >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <
>>> > > > > > >>> > prtkgaur@gmail.com>
>>> > > > > > >>> > >> > >>>>> wrote:
>>> > > > > > >>> > >> > >>>>>
>>> > > > > > >>> > >> > >>>>>> Hi team,
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>> Hope everyone is doing well. I got a chance
>>> to
>>> > work
>>> > > > > > >>> through all
>>> > > > > > >>> > >> the
>>> > > > > > >>> > >> > >>>>>> remaining feedback and update the spec doc.
>>> Here
>>> > > are
>>> > > > > the
>>> > > > > > >>> new
>>> > > > > > >>> > >> > artifacts
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>> 1) Spec document :
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> >
>>> > > > > > >>> > >>
>>> > > > > > >>> >
>>> > > > > > >>>
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>> 2) Spec document in parquet format repo :
>>> > > > > > >>> > >> > >>>>>>
>>> > https://github.com/apache/parquet-format/pull/557
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>> 3) Alp implementation in arrow c++ repo :
>>> > > > > > >>> > >> > >>>>>>
>>> > https://github.com/apache/arrow/pull/48345/changes
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>> 4) Alp implementation in parquet-java repo :
>>> Work
>>> > > for
>>> > > > > > >>> Vinoo and
>>> > > > > > >>> > >> > Julien
>>> > > > > > >>> > >> > >>>>>>
>>> > https://github.com/apache/parquet-java/pull/3397
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>> 5) PR with test and benchmarking artifacts in
>>> > > > > > >>> parquet-testing
>>> > > > > > >>> > >> repo :
>>> > > > > > >>> > >> > >>>>>>
>>> > https://github.com/apache/parquet-testing/pull/100
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>> And
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>> - Go : Arnav just submitted an in
>>> progress
>>> > > > > > >>> implementation
>>> > > > > > >>> > in
>>> > > > > > >>> > >> Go.
>>> > > > > > >>> > >> > >>>>>>
>>> https://github.com/apache/arrow-go/pull/704
>>> > (I
>>> > > > > > haven't
>>> > > > > > >>> > >> started
>>> > > > > > >>> > >> > >>>>>> looking at it yet)
>>> > > > > > >>> > >> > >>>>>> - Rust : I remember Andrew mentioned that
>>> > this
>>> > > > work
>>> > > > > > is
>>> > > > > > >>> also
>>> > > > > > >>> > >> in
>>> > > > > > >>> > >> > >>>>>> progress (So 4 languages!)
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>> *Arrow C++ implementation *
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>> The PR is out and was also used by Antoine to
>>> > > report
>>> > > > > the
>>> > > > > > >>> > numbers
>>> > > > > > >>> > >> as
>>> > > > > > >>> > >> > >>>>>> reported here. Micah and Konstantin have
>>> given 1
>>> > > > round
>>> > > > > of
>>> > > > > > >>> > >> feedback
>>> > > > > > >>> > >> > >>>>>> and I'm addressing them today. Please note
>>> that
>>> > the
>>> > > > > > default
>>> > > > > > >>> > >> > >>>>>> optimization flag for compiling is O2 and not
>>> > Q3. I
>>> > > > got
>>> > > > > > >>> around
>>> > > > > > >>> > >> 70%
>>> > > > > > >>> > >> > >>>>>> performance improvement in the decoding speed
>>> > when
>>> > > > > using
>>> > > > > > >>> the O3
>>> > > > > > >>> > >> > flag.
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>> *Parqet-MR Java implementation (working with
>>> > Vinoo
>>> > > > and
>>> > > > > > >>> Julien)
>>> > > > > > >>> > >> and
>>> > > > > > >>> > >> > **Cross
>>> > > > > > >>> > >> > >>>>>> Language testing*
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>> Let me know if you have any questions or
>>> > > > feedback.
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>> Now pasting some performance numbers
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>> Table 1: C++ ALP Double Decode — Spotify
>>> > Columns
>>> > > > > > >>> (Graviton
>>> > > > > > >>> > 3,
>>> > > > > > >>> > >> ARM
>>> > > > > > >>> > >> > >>>>>> Neoverse V1)
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>>
>>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>> │ Column │ -O2 (MB/s) │ -O3
>>> > > (MB/s) │
>>> > > > > > >>> Speedup │
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>>
>>> ├──────────────────┼──────────────┼──────────────┼─────────┤
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>> │ valence │ 3,155 │
>>> 5,523
>>> > > │
>>> > > > > > >>> 1.75x │
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>> │ danceability │ 3,233 │
>>> 5,685
>>> > > │
>>> > > > > > >>> 1.76x │
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>> │ energy │ 3,197 │
>>> 5,652
>>> > > │
>>> > > > > > >>> 1.77x │
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>> │ loudness │ 3,186 │
>>> 5,473
>>> > > │
>>> > > > > > >>> 1.72x │
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>>
>>> └──────────────────┴──────────────┴──────────────┴─────────┘
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR
>>> <
>>> > > > > > >>> > prtkgaur@gmail.com
>>> > > > > > >>> > >> >
>>> > > > > > >>> > >> > >>>>>> wrote:
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>>> @Micah Kornfield <em...@gmail.com> :
>>> Got
>>> > > it.
>>> > > > > > >>> > >> > >>>>>>>
>>> > > > > > >>> > >> > >>>>>>> @Andrew Lamb <an...@gmail.com>
>>> > > > > > >>> > >> > >>>>>>>
>>> > > > > > >>> > >> > >>>>>>>
>>> > > > > > >>> > >> > >>>>>>>> Do you think it would be good to start
>>> moving
>>> > the
>>> > > > > spec
>>> > > > > > >>> > >> development
>>> > > > > > >>> > >> > >>>>>>>> into
>>> > > > > > >>> > >> > >>>>>>>> markdown format, in preparation for
>>> finalizing
>>> > > it?
>>> > > > > > >>> > >> > >>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>
>>> > > > > > >>> > >> > >>>>>>> Yes I'll update the numbers for some of the
>>> > > > examples I
>>> > > > > > >>> have in
>>> > > > > > >>> > >> the
>>> > > > > > >>> > >> > >>>>>>> spec based
>>> > > > > > >>> > >> > >>>>>>> on the updated header size. Then we should
>>> be
>>> > good
>>> > > > to
>>> > > > > go
>>> > > > > > >>> for
>>> > > > > > >>> > the
>>> > > > > > >>> > >> > >>>>>>> markdown format.
>>> > > > > > >>> > >> > >>>>>>>
>>> > > > > > >>> > >> > >>>>>>> Thanks everyone!
>>> > > > > > >>> > >> > >>>>>>>
>>> > > > > > >>> > >> > >>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>> Andrew
>>> > > > > > >>> > >> > >>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK
>>> GAUR <
>>> > > > > > >>> > >> prtkgaur@gmail.com>
>>> > > > > > >>> > >> > >>>>>>>> wrote:
>>> > > > > > >>> > >> > >>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>> Hi team,
>>> > > > > > >>> > >> > >>>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>> 1) Andrew
>>> > > > > > >>> > >> > >>>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>> - Thanks for working on test files.
>>> My PR
>>> > > did
>>> > > > > add
>>> > > > > > >>> all
>>> > > > > > >>> > the
>>> > > > > > >>> > >> > test
>>> > > > > > >>> > >> > >>>>>>>> files I
>>> > > > > > >>> > >> > >>>>>>>>> used to benchmark on datasets. Maybe
>>> we
>>> > can
>>> > > > club
>>> > > > > > it
>>> > > > > > >>> > >> together.
>>> > > > > > >>> > >> > >>>>>>>> WIll also
>>> > > > > > >>> > >> > >>>>>>>>> aid
>>> > > > > > >>> > >> > >>>>>>>>> cross language testing
>>> > > > > > >>> > >> > >>>>>>>>> - Kosta Tarasov working on Rust
>>> > > > implementation.
>>> > > > > > >>> This is
>>> > > > > > >>> > >> > great.
>>> > > > > > >>> > >> > >>>>>>>> Thanks
>>> > > > > > >>> > >> > >>>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>> 2) Antoine
>>> > > > > > >>> > >> > >>>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>> - Thanks a lot for reporting the
>>> numbers
>>> > on
>>> > > > AMD.
>>> > > > > > >>> Looks
>>> > > > > > >>> > >> like
>>> > > > > > >>> > >> > you
>>> > > > > > >>> > >> > >>>>>>>> are
>>> > > > > > >>> > >> > >>>>>>>>> getting 8X the decoding performance of
>>> > BSS.
>>> > > > This
>>> > > > > > is
>>> > > > > > >>> > >> > amazing!!.
>>> > > > > > >>> > >> > >>>>>>>>> - Thanks for acknowledging the
>>> sampling
>>> > > > design.
>>> > > > > > >>> > >> > >>>>>>>>> - I agree with you on Fastlanes. In
>>> some
>>> > > crude
>>> > > > > > >>> > >> experiments I
>>> > > > > > >>> > >> > >>>>>>>> didn't get
>>> > > > > > >>> > >> > >>>>>>>>> a good perf benefit from it on
>>> Graviton3
>>> > > (but
>>> > > > > > maybe
>>> > > > > > >>> > there
>>> > > > > > >>> > >> was
>>> > > > > > >>> > >> > >>>>>>>> something
>>> > > > > > >>> > >> > >>>>>>>>> wrong with my implementation).
>>> > > > > > >>> > >> > >>>>>>>>> - Locking the 16bit exception
>>> encoding for
>>> > > the
>>> > > > > > spec
>>> > > > > > >>> in
>>> > > > > > >>> > >> this
>>> > > > > > >>> > >> > >>>>>>>> case.
>>> > > > > > >>> > >> > >>>>>>>>> - Awesome I think we have solved for
>>> all
>>> > > open
>>> > > > > > >>> questions
>>> > > > > > >>> > >> minus
>>> > > > > > >>> > >> > >>>>>>>> the
>>> > > > > > >>> > >> > >>>>>>>>> version byte :). (will get back on
>>> this
>>> > > soon)
>>> > > > > > >>> > >> > >>>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>> 3) Micah
>>> > > > > > >>> > >> > >>>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>> - FastLanes : The current spec does
>>> allow
>>> > > for
>>> > > > > > using
>>> > > > > > >>> > >> FastLane
>>> > > > > > >>> > >> > >>>>>>>> with the
>>> > > > > > >>> > >> > >>>>>>>>> configurable enum value for layout. We
>>> > > should
>>> > > > be
>>> > > > > > >>> able to
>>> > > > > > >>> > >> > inject
>>> > > > > > >>> > >> > >>>>>>>> any
>>> > > > > > >>> > >> > >>>>>>>>> layout
>>> > > > > > >>> > >> > >>>>>>>>> in the current design.
>>> > > > > > >>> > >> > >>>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>> Working on resolving all remaining open
>>> > comments
>>> > > > on
>>> > > > > > the
>>> > > > > > >>> spec
>>> > > > > > >>> > >> this
>>> > > > > > >>> > >> > >>>>>>>> week.
>>> > > > > > >>> > >> > >>>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>> Best
>>> > > > > > >>> > >> > >>>>>>>>> Prateek
>>> > > > > > >>> > >> > >>>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve
>>> > Loughran <
>>> > > > > > >>> > >> > >>>>>>>> stevel@cloudera.com>
>>> > > > > > >>> > >> > >>>>>>>>> wrote:
>>> > > > > > >>> > >> > >>>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah
>>> Kornfield
>>> > <
>>> > > > > > >>> > >> > >>>>>>>> emkornfield@gmail.com>
>>> > > > > > >>> > >> > >>>>>>>>>> wrote:
>>> > > > > > >>> > >> > >>>>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>>>> It looks like the actual issue
>>> described for
>>> > > ORC
>>> > > > > in
>>> > > > > > >>> the
>>> > > > > > >>> > >> paper
>>> > > > > > >>> > >> > >>>>>>>> is that
>>> > > > > > >>> > >> > >>>>>>>>> it
>>> > > > > > >>> > >> > >>>>>>>>>>> has multiple sub-encodings in a batch.
>>> This
>>> > > is
>>> > > > > > >>> different
>>> > > > > > >>> > >> then
>>> > > > > > >>> > >> > >>>>>>>> the
>>> > > > > > >>> > >> > >>>>>>>>> design
>>> > > > > > >>> > >> > >>>>>>>>>>> proposed here where there is still fixed
>>> > > > encoding
>>> > > > > > per
>>> > > > > > >>> page
>>> > > > > > >>> > >> in
>>> > > > > > >>> > >> > >>>>>>>> parquet.
>>> > > > > > >>> > >> > >>>>>>>>>>> Given reasonably sized pages I don't
>>> think
>>> > > > branch
>>> > > > > > >>> > >> > >>>>>>>> misprediction should
>>> > > > > > >>> > >> > >>>>>>>>>> be a
>>> > > > > > >>> > >> > >>>>>>>>>>> big issue for new encodings. I agree
>>> that
>>> > we
>>> > > > > should
>>> > > > > > >>> be
>>> > > > > > >>> > >> > >>>>>>>> conservative in
>>> > > > > > >>> > >> > >>>>>>>>>>> general for adding new encodings.
>>> > > > > > >>> > >> > >>>>>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>>> +1
>>> > > > > > >>> > >> > >>>>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>
>>> > > > > > >>> > >> > >
>>> > > > > > >>> > >> >
>>> > > > > > >>> > >> >
>>> > > > > > >>> > >> >
>>> > > > > > >>> > >>
>>> > > > > > >>> > >
>>> > > > > > >>> >
>>> > > > > > >>>
>>> > > > > > >>
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>>
>>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Andrew Lamb <an...@gmail.com>.
After re-reading Julien's Versioning Proposal, my understanding is that if
we adopt that system, we will somehow mark ALP in "Preview" in
parquet-format for release Version 3.
The versioning document doesn't seem to proscribe the particular method for
annotating "preview" features -- I think clearly marking them as such in
parquet.thrift as well as reflecting that on the
https://parquet.apache.org/docs/file-format/versions/ page somehow would be
adequate.
Andrew
On Thu, Jul 30, 2026 at 11:50 AM Andrew Lamb <an...@gmail.com> wrote:
> In my mind ALP is in the same position as other encodings that were added
> recently[1] (such as BYTE_STREAM_SPLIT for non floating point types)
>
> That is to say it will be available in the latest open source writers and
> readers, but realistically not widely used in the ecosystem where
> compatibility is a concern (a large part)
>
> I view Julian's work[2][3] to gather consensus on how to name combinations
> of features (e.g. "versions") as the mechanism by which we will accelerate
> the adoption of this change (along with other forward incompatible changes)
> across the ecosystem.
>
> > Will we exclude this change when cutting parquet-format v2.14.0?
>
> No I don't think so -- I would expect it to go out in the next
> parquet-format version as do other changes in the spec.
>
> Andrew
>
> [1]: https://parquet.apache.org/docs/file-format/versions/
> [2]: https://lists.apache.org/thread/xmnj8h0h8ozmrgox5tydhhs7yz7q2hss
> [3]: https://lists.apache.org/thread/qlf8lg90gqq41lllyy9mk2f0skqvftqj
>
>
> On Thu, Jul 30, 2026 at 3:51 AM Divjot Arora via dev <
> dev@parquet.apache.org> wrote:
>
>> Hi Andrew,
>>
>> How does this interact with the ongoing versioning discussion? Given that
>> new encodings are not forward compatible, I would assume ALP is included
>> in
>> Parquet V<next> (V3?), but it's not clear to me how that works if it's
>> merged to the parquet-format master branch now. Will we exclude this
>> change
>> when cutting parquet-format v2.14.0?
>>
>> -- Div
>>
>> On Thu, Jul 30, 2026 at 8:51 AM Andrew Lamb <an...@gmail.com>
>> wrote:
>>
>> > Hello,
>> >
>> > As an update, thanks to the epic work from many contributors, especially
>> > Prateek, I just merged the parquet-format change for ALP[1]. We plan to
>> > make additional editorial The remaining items I know of are merging the
>> > examples from parquet-testing[2], and then of course the various
>> > implementations
>> >
>> > Thanks again everyone. This is a great step forward
>> > Andrew
>> >
>> > [1]: https://github.com/apache/parquet-format/pull/557
>> > [2]: https://github.com/apache/parquet-testing/pull/100
>> >
>> > On Tue, Jul 21, 2026 at 4:39 PM Julien Le Dem <ju...@apache.org>
>> wrote:
>> >
>> > > It was also in my spam folder... :(
>> > > I've seen a bunch of apache list emails go to my spam recently.
>> > >
>> > > On Tue, Jul 21, 2026 at 10:58 AM Russell Spitzer <
>> > > russell.spitzer@gmail.com>
>> > > wrote:
>> > >
>> > > > Yes, my hope was that my vote would kick it out of spam but we
>> > definitely
>> > > > had two other votes that also went to spam for me.
>> > > >
>> > > > For anyone looking the thread is here
>> > > > https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt
>> > > >
>> > > > On Tue, Jul 21, 2026 at 10:57 AM Micah Kornfield <
>> > emkornfield@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > I've seen other's votes not go to spam, so hopefully people will
>> see
>> > it
>> > > > in
>> > > > > there inboxes (or see this thread and look in there spam).
>> > > > >
>> > > > > On Tue, Jul 21, 2026 at 10:52 AM PRATEEK GAUR <prtkgaur@gmail.com
>> >
>> > > > wrote:
>> > > > >
>> > > > > > Okay,
>> > > > > >
>> > > > > > I sent out the voting email yesterday and so far I've heard
>> from 3
>> > > > people
>> > > > > > that it ended up in their "spam folder". Dev VPN to be blamed
>> for
>> > > it?.
>> > > > > >
>> > > > > > What's the protocol here?
>> > > > > > I let people hunt for the email in their spam folder or send out
>> > > > another
>> > > > > > one?
>> > > > > >
>> > > > > > Best
>> > > > > > Prateek
>> > > > > >
>> > > > > > On Sun, Jul 19, 2026 at 10:47 PM PRATEEK GAUR <
>> prtkgaur@gmail.com>
>> > > > > wrote:
>> > > > > >
>> > > > > > > Hi team,
>> > > > > > >
>> > > > > > > Vinoo has added the tests which were requested in the last
>> round
>> > of
>> > > > > > review.
>> > > > > > > In his own words.
>> > > > > > >
>> > > > > > > "
>> > > > > > > The extreme value tests Micah wanted are done and pushed. I
>> added
>> > > > > > coverage
>> > > > > > > for values that need the full FOR bit width after the frame of
>> > > > > reference
>> > > > > > is
>> > > > > > > applied: 64-bit (the case where the signed max minus min
>> > > subtraction
>> > > > > > > overflows), 63-bit (the non-overflow case), and 32-bit for
>> > floats.
>> > > > All
>> > > > > > > lossless with zero exceptions. I also verified them through
>> Arrow
>> > > C++
>> > > > > ALP
>> > > > > > > reader (the local build), so the Java-written extreme columns
>> > > decode
>> > > > > > > correctly in C++, bit-exact against the expected values."
>> > > > > > >
>> > > > > > > State of the PR (#3397):
>> > > > > > >
>> > > > > > > - Merges cleanly with master (I pulled master in and
>> resolved
>> > > the
>> > > > > > > conflicts).
>> > > > > > > - 41 of 44 review threads resolved. The 3 remaining are
>> > > > intentional:
>> > > > > > > one deferred perf nit, one that stays open until
>> > parquet-format
>> > > > > ships
>> > > > > > ALP,
>> > > > > > >
>> > > > > > > "
>> > > > > > >
>> > > > > > > With this both the
>> > > > > > > 1) C++ PR :
>> https://github.com/apache/arrow/pull/48345/changes
>> > > > > > > 2) Java PR : https://github.com/apache/parquet-java/pull/3397
>> > > > > > >
>> > > > > > > Are cross language tested and range of test scenarios.
>> > > > > > >
>> > > > > > > Best
>> > > > > > > Prateek
>> > > > > > >
>> > > > > > >
>> > > > > > > On Mon, Jul 13, 2026 at 8:38 AM PRATEEK GAUR <
>> prtkgaur@gmail.com
>> > >
>> > > > > wrote:
>> > > > > > >
>> > > > > > >> Thanks Matt,
>> > > > > > >>
>> > > > > > >> I'll be on it this week.
>> > > > > > >>
>> > > > > > >> Best
>> > > > > > >> Prateek
>> > > > > > >>
>> > > > > > >> On Mon, Jul 13, 2026 at 8:36 AM Matt Topol <
>> > > zotthewizard@gmail.com>
>> > > > > > >> wrote:
>> > > > > > >>
>> > > > > > >>> There is also the Go implementation submitted by Arnav (
>> > > > > > >>> https://github.com/apache/arrow-go/pull/704) which is
>> waiting
>> > > for
>> > > > > > >>> updates.
>> > > > > > >>>
>> > > > > > >>> Just wanted to make sure it didn't get lost here.
>> > > > > > >>>
>> > > > > > >>> --Matt
>> > > > > > >>>
>> > > > > > >>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK GAUR <
>> > > prtkgaur@gmail.com>
>> > > > > > >>> wrote:
>> > > > > > >>>
>> > > > > > >>> > Hi all,
>> > > > > > >>> >
>> > > > > > >>> > I'd like to share a status update on the ALP encoding
>> effort
>> > > and
>> > > > > get
>> > > > > > a
>> > > > > > >>> feel
>> > > > > > >>> > from the community before starting a formal vote.
>> > > > > > >>> >
>> > > > > > >>> > We now have the specification plus implementations in both
>> > > > > languages:
>> > > > > > >>> >
>> > > > > > >>> > - Spec: apache/parquet-format#557 — GH-533 Add
>> > ALP
>> > > > > > encoding
>> > > > > > >>> > specification
>> > > > > > >>> > - C++ (Arrow): apache/arrow#48345 — GH-48701
>> > > [C++][Parquet]
>> > > > > Add
>> > > > > > >>> ALPpd
>> > > > > > >>> > encoding
>> > > > > > >>> > - Java: apache/parquet-java#3397 — Parquet
>> Java
>> > ALP
>> > > > > > >>> > Implementation (by Vinoo)
>> > > > > > >>> >
>> > > > > > >>> > Current state:
>> > > > > > >>> >
>> > > > > > >>> > - The C++ implementation has been through 3-4 rounds
>> of
>> > > > review.
>> > > > > > >>> > - The Java implementation has been tracking well and
>> has
>> > > had
>> > > > > > >>> initial
>> > > > > > >>> > reviews from contributors, with no major remaining
>> > > issues.
>> > > > > > >>> > - Cross-language compatibility tests are passing: the
>> > Arrow
>> > > > C++
>> > > > > > >>> decoder
>> > > > > > >>> > reads Java-written data bit-exactly across ~1.56M
>> > values
>> > > > and
>> > > > > 18
>> > > > > > >>> > fixtures, covering V1 and V2 pages, multiple vector
>> > > sizes,
>> > > > > and
>> > > > > > >>> > several
>> > > > > > >>> > real datasets — zero mismatches.
>> > > > > > >>> >
>> > > > > > >>> > As raised in the java-pr we will be adding coverage for
>> > > extreme
>> > > > > > values
>> > > > > > >>> > (those requiring
>> > > > > > >>> > 63-64 bits after FOR is applied) before we close a vote.
>> > We'll
>> > > > aim
>> > > > > > to
>> > > > > > >>> get
>> > > > > > >>> > that
>> > > > > > >>> > done in parallel.
>> > > > > > >>> >
>> > > > > > >>> > Plan: unless there are objections, I intend to start a
>> vote
>> > at
>> > > > the
>> > > > > > end
>> > > > > > >>> of
>> > > > > > >>> > this week/early
>> > > > > > >>> > next week.
>> > > > > > >>> >
>> > > > > > >>> > Thanks
>> > > > > > >>> > Prateek
>> > > > > > >>> >
>> > > > > > >>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <
>> > > prtkgaur@gmail.com>
>> > > > > > >>> wrote:
>> > > > > > >>> >
>> > > > > > >>> > > Hi Team,
>> > > > > > >>> > >
>> > > > > > >>> > > Just wanted to provide some updates on ALP.
>> > > > > > >>> > > Micah and I have done 3-4 rounds of review for the c++
>> PR :
>> > > > > > >>> > > https://github.com/apache/arrow/pull/48345
>> > > > > > >>> > >
>> > > > > > >>> > > For the Java implementation Vinoo has been working on
>> > > following
>> > > > > PR
>> > > > > > :
>> > > > > > >>> > > https://github.com/apache/parquet-java/pull/3397
>> > > > > > >>> > >
>> > > > > > >>> > > Best
>> > > > > > >>> > > Prateek
>> > > > > > >>> > >
>> > > > > > >>> > > On Tue, May 5, 2026 at 1:50 PM Micah Kornfield <
>> > > > > > >>> emkornfield@gmail.com>
>> > > > > > >>> > > wrote:
>> > > > > > >>> > >
>> > > > > > >>> > >> Hi Antoine,
>> > > > > > >>> > >>
>> > > > > > >>> > >> > Apologies if the question was already asked, but
>> should
>> > we
>> > > > > care
>> > > > > > >>> about
>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient
>> than
>> > > > > FLOAT16
>> > > > > > +
>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
>> > > > > > >>> > >>
>> > > > > > >>> > >> It was. We thought we could defer it for the following
>> > > > reasons:
>> > > > > > >>> > >> 1. It's not clear there are a lot of easy reference
>> > > datasets
>> > > > to
>> > > > > > >>> test
>> > > > > > >>> > its
>> > > > > > >>> > >> effectiveness. It does look like there might be one or
>> > two
>> > > on
>> > > > > > >>> > huggingface
>> > > > > > >>> > >> (e.g. https://huggingface.co/datasets/kikitora/curdie
>> ).
>> > > > > > >>> > >> 2. It seemed likely that float 16 was more likely used
>> > for
>> > > > > values
>> > > > > > >>> that
>> > > > > > >>> > >> were less likely to reduce to decimal values.
>> > > > > > >>> > >> 3. It could be added as an extension later if needed.
>> > > > > > >>> > >>
>> > > > > > >>> > >> Cheers,
>> > > > > > >>> > >> Micah
>> > > > > > >>> > >>
>> > > > > > >>> > >>
>> > > > > > >>> > >> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <
>> > > > > antoine@python.org
>> > > > > > >
>> > > > > > >>> > wrote:
>> > > > > > >>> > >>
>> > > > > > >>> > >> >
>> > > > > > >>> > >> > Hello,
>> > > > > > >>> > >> >
>> > > > > > >>> > >> > Apologies if the question was already asked, but
>> should
>> > we
>> > > > > care
>> > > > > > >>> about
>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient
>> than
>> > > > > FLOAT16
>> > > > > > +
>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
>> > > > > > >>> > >> >
>> > > > > > >>> > >> > Regards
>> > > > > > >>> > >> >
>> > > > > > >>> > >> > Antoine.
>> > > > > > >>> > >> >
>> > > > > > >>> > >> >
>> > > > > > >>> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
>> > > > > > >>> > >> > > Thanks Andrew and Micah for review feedback on the
>> two
>> > > > PR's
>> > > > > > >>> > >> > > 1) (c++ arrow repo)
>> > > > > > >>> > >> https://github.com/apache/arrow/pull/48345/changes
>> > > > > > >>> > >> > > 2) (parquet-format repo)
>> > > > > > >>> > >> > https://github.com/apache/parquet-format/pull/557
>> > > > > > >>> > >> > >
>> > > > > > >>> > >> > > I have addressed all (unless I missed something)
>> > > comments
>> > > > on
>> > > > > > >>> the two
>> > > > > > >>> > >> > PR's.
>> > > > > > >>> > >> > >
>> > > > > > >>> > >> > > Best
>> > > > > > >>> > >> > > Prateek
>> > > > > > >>> > >> > >
>> > > > > > >>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <
>> > > > > > >>> prtkgaur@gmail.com>
>> > > > > > >>> > >> wrote:
>> > > > > > >>> > >> > >
>> > > > > > >>> > >> > >> Thanks Andrew and Micah.
>> > > > > > >>> > >> > >>
>> > > > > > >>> > >> > >> `fair amount of feedback on at least the
>> > > implementations`
>> > > > > > >>> > >> > >> For the c++ I have already started addressing the
>> > > > > feedback, I
>> > > > > > >>> > should
>> > > > > > >>> > >> be
>> > > > > > >>> > >> > >> done with that Monday/Tuesday.
>> > > > > > >>> > >> > >> I think Vinoo too has been making good progress on
>> > the
>> > > > Java
>> > > > > > >>> > >> > implementation.
>> > > > > > >>> > >> > >>
>> > > > > > >>> > >> > >> Best
>> > > > > > >>> > >> > >> Prateek
>> > > > > > >>> > >> > >>
>> > > > > > >>> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <
>> > > > > > >>> > andrewlamb11@gmail.com
>> > > > > > >>> > >> >
>> > > > > > >>> > >> > >> wrote:
>> > > > > > >>> > >> > >>
>> > > > > > >>> > >> > >>> Got it. Thank you for the clarification -- I will
>> > try
>> > > > and
>> > > > > > look
>> > > > > > >>> > into
>> > > > > > >>> > >> the
>> > > > > > >>> > >> > >>> spec and the Rust implementation[1] in this next
>> > week
>> > > > > > >>> > >> > >>>
>> > > > > > >>> > >> > >>> [1]:
>> https://github.com/apache/arrow-rs/pull/9372
>> > > > > > >>> > >> > >>>
>> > > > > > >>> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield
>> <
>> > > > > > >>> > >> > emkornfield@gmail.com>
>> > > > > > >>> > >> > >>> wrote:
>> > > > > > >>> > >> > >>>
>> > > > > > >>> > >> > >>>> Hi Andrew,
>> > > > > > >>> > >> > >>>> I think there is a fair amount of feedback on at
>> > > least
>> > > > > the
>> > > > > > >>> > >> > >>>> implementations, typically I think we've waited
>> > till
>> > > > they
>> > > > > > are
>> > > > > > >>> > >> close to
>> > > > > > >>> > >> > >>>> mergeable before a final vote. Otherwise I
>> agree
>> > we
>> > > > are
>> > > > > > very
>> > > > > > >>> > >> close.
>> > > > > > >>> > >> > >>>>
>> > > > > > >>> > >> > >>>> -Micah
>> > > > > > >>> > >> > >>>>
>> > > > > > >>> > >> > >>>> On Saturday, April 25, 2026, Andrew Lamb <
>> > > > > > >>> andrewlamb11@gmail.com
>> > > > > > >>> > >
>> > > > > > >>> > >> > wrote:
>> > > > > > >>> > >> > >>>>
>> > > > > > >>> > >> > >>>>> Thanks Prateek,
>> > > > > > >>> > >> > >>>>>
>> > > > > > >>> > >> > >>>>> I think from this content it looks to me like
>> we
>> > are
>> > > > > ready
>> > > > > > >>> to
>> > > > > > >>> > >> start a
>> > > > > > >>> > >> > >>>>> vote to explicitly accept ALP into Parquet
>> > > > > > >>> > >> > >>>>>
>> > > > > > >>> > >> > >>>>> Does anyone know of a reason we should
>> postpone it
>> > > for
>> > > > > > >>> longer?
>> > > > > > >>> > >> > >>>>> Perhaps someone needs some more time to review?
>> > > > > > >>> > >> > >>>>>
>> > > > > > >>> > >> > >>>>> Andrew
>> > > > > > >>> > >> > >>>>>
>> > > > > > >>> > >> > >>>>>
>> > > > > > >>> > >> > >>>>>
>> > > > > > >>> > >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <
>> > > > > > >>> > prtkgaur@gmail.com>
>> > > > > > >>> > >> > >>>>> wrote:
>> > > > > > >>> > >> > >>>>>
>> > > > > > >>> > >> > >>>>>> Hi team,
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>> Hope everyone is doing well. I got a chance to
>> > work
>> > > > > > >>> through all
>> > > > > > >>> > >> the
>> > > > > > >>> > >> > >>>>>> remaining feedback and update the spec doc.
>> Here
>> > > are
>> > > > > the
>> > > > > > >>> new
>> > > > > > >>> > >> > artifacts
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>> 1) Spec document :
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> >
>> > > > > > >>> > >>
>> > > > > > >>> >
>> > > > > > >>>
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>> 2) Spec document in parquet format repo :
>> > > > > > >>> > >> > >>>>>>
>> > https://github.com/apache/parquet-format/pull/557
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>> 3) Alp implementation in arrow c++ repo :
>> > > > > > >>> > >> > >>>>>>
>> > https://github.com/apache/arrow/pull/48345/changes
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>> 4) Alp implementation in parquet-java repo :
>> Work
>> > > for
>> > > > > > >>> Vinoo and
>> > > > > > >>> > >> > Julien
>> > > > > > >>> > >> > >>>>>>
>> > https://github.com/apache/parquet-java/pull/3397
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>> 5) PR with test and benchmarking artifacts in
>> > > > > > >>> parquet-testing
>> > > > > > >>> > >> repo :
>> > > > > > >>> > >> > >>>>>>
>> > https://github.com/apache/parquet-testing/pull/100
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>> And
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>> - Go : Arnav just submitted an in progress
>> > > > > > >>> implementation
>> > > > > > >>> > in
>> > > > > > >>> > >> Go.
>> > > > > > >>> > >> > >>>>>>
>> https://github.com/apache/arrow-go/pull/704
>> > (I
>> > > > > > haven't
>> > > > > > >>> > >> started
>> > > > > > >>> > >> > >>>>>> looking at it yet)
>> > > > > > >>> > >> > >>>>>> - Rust : I remember Andrew mentioned that
>> > this
>> > > > work
>> > > > > > is
>> > > > > > >>> also
>> > > > > > >>> > >> in
>> > > > > > >>> > >> > >>>>>> progress (So 4 languages!)
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>> *Arrow C++ implementation *
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>> The PR is out and was also used by Antoine to
>> > > report
>> > > > > the
>> > > > > > >>> > numbers
>> > > > > > >>> > >> as
>> > > > > > >>> > >> > >>>>>> reported here. Micah and Konstantin have
>> given 1
>> > > > round
>> > > > > of
>> > > > > > >>> > >> feedback
>> > > > > > >>> > >> > >>>>>> and I'm addressing them today. Please note
>> that
>> > the
>> > > > > > default
>> > > > > > >>> > >> > >>>>>> optimization flag for compiling is O2 and not
>> > Q3. I
>> > > > got
>> > > > > > >>> around
>> > > > > > >>> > >> 70%
>> > > > > > >>> > >> > >>>>>> performance improvement in the decoding speed
>> > when
>> > > > > using
>> > > > > > >>> the O3
>> > > > > > >>> > >> > flag.
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>> *Parqet-MR Java implementation (working with
>> > Vinoo
>> > > > and
>> > > > > > >>> Julien)
>> > > > > > >>> > >> and
>> > > > > > >>> > >> > **Cross
>> > > > > > >>> > >> > >>>>>> Language testing*
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>> Let me know if you have any questions or
>> > > > feedback.
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>> Now pasting some performance numbers
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>> Table 1: C++ ALP Double Decode — Spotify
>> > Columns
>> > > > > > >>> (Graviton
>> > > > > > >>> > 3,
>> > > > > > >>> > >> ARM
>> > > > > > >>> > >> > >>>>>> Neoverse V1)
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>> │ Column │ -O2 (MB/s) │ -O3
>> > > (MB/s) │
>> > > > > > >>> Speedup │
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> ├──────────────────┼──────────────┼──────────────┼─────────┤
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>> │ valence │ 3,155 │
>> 5,523
>> > > │
>> > > > > > >>> 1.75x │
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>> │ danceability │ 3,233 │
>> 5,685
>> > > │
>> > > > > > >>> 1.76x │
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>> │ energy │ 3,197 │
>> 5,652
>> > > │
>> > > > > > >>> 1.77x │
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>> │ loudness │ 3,186 │
>> 5,473
>> > > │
>> > > > > > >>> 1.72x │
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> └──────────────────┴──────────────┴──────────────┴─────────┘
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <
>> > > > > > >>> > prtkgaur@gmail.com
>> > > > > > >>> > >> >
>> > > > > > >>> > >> > >>>>>> wrote:
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>>> @Micah Kornfield <em...@gmail.com> :
>> Got
>> > > it.
>> > > > > > >>> > >> > >>>>>>>
>> > > > > > >>> > >> > >>>>>>> @Andrew Lamb <an...@gmail.com>
>> > > > > > >>> > >> > >>>>>>>
>> > > > > > >>> > >> > >>>>>>>
>> > > > > > >>> > >> > >>>>>>>> Do you think it would be good to start
>> moving
>> > the
>> > > > > spec
>> > > > > > >>> > >> development
>> > > > > > >>> > >> > >>>>>>>> into
>> > > > > > >>> > >> > >>>>>>>> markdown format, in preparation for
>> finalizing
>> > > it?
>> > > > > > >>> > >> > >>>>>>>>
>> > > > > > >>> > >> > >>>>>>>
>> > > > > > >>> > >> > >>>>>>> Yes I'll update the numbers for some of the
>> > > > examples I
>> > > > > > >>> have in
>> > > > > > >>> > >> the
>> > > > > > >>> > >> > >>>>>>> spec based
>> > > > > > >>> > >> > >>>>>>> on the updated header size. Then we should be
>> > good
>> > > > to
>> > > > > go
>> > > > > > >>> for
>> > > > > > >>> > the
>> > > > > > >>> > >> > >>>>>>> markdown format.
>> > > > > > >>> > >> > >>>>>>>
>> > > > > > >>> > >> > >>>>>>> Thanks everyone!
>> > > > > > >>> > >> > >>>>>>>
>> > > > > > >>> > >> > >>>>>>>
>> > > > > > >>> > >> > >>>>>>>>
>> > > > > > >>> > >> > >>>>>>>> Andrew
>> > > > > > >>> > >> > >>>>>>>>
>> > > > > > >>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK
>> GAUR <
>> > > > > > >>> > >> prtkgaur@gmail.com>
>> > > > > > >>> > >> > >>>>>>>> wrote:
>> > > > > > >>> > >> > >>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>> Hi team,
>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>> 1) Andrew
>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>> - Thanks for working on test files. My
>> PR
>> > > did
>> > > > > add
>> > > > > > >>> all
>> > > > > > >>> > the
>> > > > > > >>> > >> > test
>> > > > > > >>> > >> > >>>>>>>> files I
>> > > > > > >>> > >> > >>>>>>>>> used to benchmark on datasets. Maybe we
>> > can
>> > > > club
>> > > > > > it
>> > > > > > >>> > >> together.
>> > > > > > >>> > >> > >>>>>>>> WIll also
>> > > > > > >>> > >> > >>>>>>>>> aid
>> > > > > > >>> > >> > >>>>>>>>> cross language testing
>> > > > > > >>> > >> > >>>>>>>>> - Kosta Tarasov working on Rust
>> > > > implementation.
>> > > > > > >>> This is
>> > > > > > >>> > >> > great.
>> > > > > > >>> > >> > >>>>>>>> Thanks
>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>> 2) Antoine
>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>> - Thanks a lot for reporting the
>> numbers
>> > on
>> > > > AMD.
>> > > > > > >>> Looks
>> > > > > > >>> > >> like
>> > > > > > >>> > >> > you
>> > > > > > >>> > >> > >>>>>>>> are
>> > > > > > >>> > >> > >>>>>>>>> getting 8X the decoding performance of
>> > BSS.
>> > > > This
>> > > > > > is
>> > > > > > >>> > >> > amazing!!.
>> > > > > > >>> > >> > >>>>>>>>> - Thanks for acknowledging the sampling
>> > > > design.
>> > > > > > >>> > >> > >>>>>>>>> - I agree with you on Fastlanes. In
>> some
>> > > crude
>> > > > > > >>> > >> experiments I
>> > > > > > >>> > >> > >>>>>>>> didn't get
>> > > > > > >>> > >> > >>>>>>>>> a good perf benefit from it on
>> Graviton3
>> > > (but
>> > > > > > maybe
>> > > > > > >>> > there
>> > > > > > >>> > >> was
>> > > > > > >>> > >> > >>>>>>>> something
>> > > > > > >>> > >> > >>>>>>>>> wrong with my implementation).
>> > > > > > >>> > >> > >>>>>>>>> - Locking the 16bit exception encoding
>> for
>> > > the
>> > > > > > spec
>> > > > > > >>> in
>> > > > > > >>> > >> this
>> > > > > > >>> > >> > >>>>>>>> case.
>> > > > > > >>> > >> > >>>>>>>>> - Awesome I think we have solved for
>> all
>> > > open
>> > > > > > >>> questions
>> > > > > > >>> > >> minus
>> > > > > > >>> > >> > >>>>>>>> the
>> > > > > > >>> > >> > >>>>>>>>> version byte :). (will get back on this
>> > > soon)
>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>> 3) Micah
>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>> - FastLanes : The current spec does
>> allow
>> > > for
>> > > > > > using
>> > > > > > >>> > >> FastLane
>> > > > > > >>> > >> > >>>>>>>> with the
>> > > > > > >>> > >> > >>>>>>>>> configurable enum value for layout. We
>> > > should
>> > > > be
>> > > > > > >>> able to
>> > > > > > >>> > >> > inject
>> > > > > > >>> > >> > >>>>>>>> any
>> > > > > > >>> > >> > >>>>>>>>> layout
>> > > > > > >>> > >> > >>>>>>>>> in the current design.
>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>> Working on resolving all remaining open
>> > comments
>> > > > on
>> > > > > > the
>> > > > > > >>> spec
>> > > > > > >>> > >> this
>> > > > > > >>> > >> > >>>>>>>> week.
>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>> Best
>> > > > > > >>> > >> > >>>>>>>>> Prateek
>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve
>> > Loughran <
>> > > > > > >>> > >> > >>>>>>>> stevel@cloudera.com>
>> > > > > > >>> > >> > >>>>>>>>> wrote:
>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah
>> Kornfield
>> > <
>> > > > > > >>> > >> > >>>>>>>> emkornfield@gmail.com>
>> > > > > > >>> > >> > >>>>>>>>>> wrote:
>> > > > > > >>> > >> > >>>>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>>>> It looks like the actual issue described
>> for
>> > > ORC
>> > > > > in
>> > > > > > >>> the
>> > > > > > >>> > >> paper
>> > > > > > >>> > >> > >>>>>>>> is that
>> > > > > > >>> > >> > >>>>>>>>> it
>> > > > > > >>> > >> > >>>>>>>>>>> has multiple sub-encodings in a batch.
>> This
>> > > is
>> > > > > > >>> different
>> > > > > > >>> > >> then
>> > > > > > >>> > >> > >>>>>>>> the
>> > > > > > >>> > >> > >>>>>>>>> design
>> > > > > > >>> > >> > >>>>>>>>>>> proposed here where there is still fixed
>> > > > encoding
>> > > > > > per
>> > > > > > >>> page
>> > > > > > >>> > >> in
>> > > > > > >>> > >> > >>>>>>>> parquet.
>> > > > > > >>> > >> > >>>>>>>>>>> Given reasonably sized pages I don't
>> think
>> > > > branch
>> > > > > > >>> > >> > >>>>>>>> misprediction should
>> > > > > > >>> > >> > >>>>>>>>>> be a
>> > > > > > >>> > >> > >>>>>>>>>>> big issue for new encodings. I agree
>> that
>> > we
>> > > > > should
>> > > > > > >>> be
>> > > > > > >>> > >> > >>>>>>>> conservative in
>> > > > > > >>> > >> > >>>>>>>>>>> general for adding new encodings.
>> > > > > > >>> > >> > >>>>>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>>> +1
>> > > > > > >>> > >> > >>>>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>
>> > > > > > >>> > >> > >>>>>>>
>> > > > > > >>> > >> > >
>> > > > > > >>> > >> >
>> > > > > > >>> > >> >
>> > > > > > >>> > >> >
>> > > > > > >>> > >>
>> > > > > > >>> > >
>> > > > > > >>> >
>> > > > > > >>>
>> > > > > > >>
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Andrew Lamb <an...@gmail.com>.
In my mind ALP is in the same position as other encodings that were added
recently[1] (such as BYTE_STREAM_SPLIT for non floating point types)
That is to say it will be available in the latest open source writers and
readers, but realistically not widely used in the ecosystem where
compatibility is a concern (a large part)
I view Julian's work[2][3] to gather consensus on how to name combinations
of features (e.g. "versions") as the mechanism by which we will accelerate
the adoption of this change (along with other forward incompatible changes)
across the ecosystem.
> Will we exclude this change when cutting parquet-format v2.14.0?
No I don't think so -- I would expect it to go out in the next
parquet-format version as do other changes in the spec.
Andrew
[1]: https://parquet.apache.org/docs/file-format/versions/
[2]: https://lists.apache.org/thread/xmnj8h0h8ozmrgox5tydhhs7yz7q2hss
[3]: https://lists.apache.org/thread/qlf8lg90gqq41lllyy9mk2f0skqvftqj
On Thu, Jul 30, 2026 at 3:51 AM Divjot Arora via dev <de...@parquet.apache.org>
wrote:
> Hi Andrew,
>
> How does this interact with the ongoing versioning discussion? Given that
> new encodings are not forward compatible, I would assume ALP is included in
> Parquet V<next> (V3?), but it's not clear to me how that works if it's
> merged to the parquet-format master branch now. Will we exclude this change
> when cutting parquet-format v2.14.0?
>
> -- Div
>
> On Thu, Jul 30, 2026 at 8:51 AM Andrew Lamb <an...@gmail.com>
> wrote:
>
> > Hello,
> >
> > As an update, thanks to the epic work from many contributors, especially
> > Prateek, I just merged the parquet-format change for ALP[1]. We plan to
> > make additional editorial The remaining items I know of are merging the
> > examples from parquet-testing[2], and then of course the various
> > implementations
> >
> > Thanks again everyone. This is a great step forward
> > Andrew
> >
> > [1]: https://github.com/apache/parquet-format/pull/557
> > [2]: https://github.com/apache/parquet-testing/pull/100
> >
> > On Tue, Jul 21, 2026 at 4:39 PM Julien Le Dem <ju...@apache.org> wrote:
> >
> > > It was also in my spam folder... :(
> > > I've seen a bunch of apache list emails go to my spam recently.
> > >
> > > On Tue, Jul 21, 2026 at 10:58 AM Russell Spitzer <
> > > russell.spitzer@gmail.com>
> > > wrote:
> > >
> > > > Yes, my hope was that my vote would kick it out of spam but we
> > definitely
> > > > had two other votes that also went to spam for me.
> > > >
> > > > For anyone looking the thread is here
> > > > https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt
> > > >
> > > > On Tue, Jul 21, 2026 at 10:57 AM Micah Kornfield <
> > emkornfield@gmail.com>
> > > > wrote:
> > > >
> > > > > I've seen other's votes not go to spam, so hopefully people will
> see
> > it
> > > > in
> > > > > there inboxes (or see this thread and look in there spam).
> > > > >
> > > > > On Tue, Jul 21, 2026 at 10:52 AM PRATEEK GAUR <pr...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > Okay,
> > > > > >
> > > > > > I sent out the voting email yesterday and so far I've heard from
> 3
> > > > people
> > > > > > that it ended up in their "spam folder". Dev VPN to be blamed for
> > > it?.
> > > > > >
> > > > > > What's the protocol here?
> > > > > > I let people hunt for the email in their spam folder or send out
> > > > another
> > > > > > one?
> > > > > >
> > > > > > Best
> > > > > > Prateek
> > > > > >
> > > > > > On Sun, Jul 19, 2026 at 10:47 PM PRATEEK GAUR <
> prtkgaur@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Hi team,
> > > > > > >
> > > > > > > Vinoo has added the tests which were requested in the last
> round
> > of
> > > > > > review.
> > > > > > > In his own words.
> > > > > > >
> > > > > > > "
> > > > > > > The extreme value tests Micah wanted are done and pushed. I
> added
> > > > > > coverage
> > > > > > > for values that need the full FOR bit width after the frame of
> > > > > reference
> > > > > > is
> > > > > > > applied: 64-bit (the case where the signed max minus min
> > > subtraction
> > > > > > > overflows), 63-bit (the non-overflow case), and 32-bit for
> > floats.
> > > > All
> > > > > > > lossless with zero exceptions. I also verified them through
> Arrow
> > > C++
> > > > > ALP
> > > > > > > reader (the local build), so the Java-written extreme columns
> > > decode
> > > > > > > correctly in C++, bit-exact against the expected values."
> > > > > > >
> > > > > > > State of the PR (#3397):
> > > > > > >
> > > > > > > - Merges cleanly with master (I pulled master in and
> resolved
> > > the
> > > > > > > conflicts).
> > > > > > > - 41 of 44 review threads resolved. The 3 remaining are
> > > > intentional:
> > > > > > > one deferred perf nit, one that stays open until
> > parquet-format
> > > > > ships
> > > > > > ALP,
> > > > > > >
> > > > > > > "
> > > > > > >
> > > > > > > With this both the
> > > > > > > 1) C++ PR : https://github.com/apache/arrow/pull/48345/changes
> > > > > > > 2) Java PR : https://github.com/apache/parquet-java/pull/3397
> > > > > > >
> > > > > > > Are cross language tested and range of test scenarios.
> > > > > > >
> > > > > > > Best
> > > > > > > Prateek
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Jul 13, 2026 at 8:38 AM PRATEEK GAUR <
> prtkgaur@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > >> Thanks Matt,
> > > > > > >>
> > > > > > >> I'll be on it this week.
> > > > > > >>
> > > > > > >> Best
> > > > > > >> Prateek
> > > > > > >>
> > > > > > >> On Mon, Jul 13, 2026 at 8:36 AM Matt Topol <
> > > zotthewizard@gmail.com>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >>> There is also the Go implementation submitted by Arnav (
> > > > > > >>> https://github.com/apache/arrow-go/pull/704) which is
> waiting
> > > for
> > > > > > >>> updates.
> > > > > > >>>
> > > > > > >>> Just wanted to make sure it didn't get lost here.
> > > > > > >>>
> > > > > > >>> --Matt
> > > > > > >>>
> > > > > > >>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK GAUR <
> > > prtkgaur@gmail.com>
> > > > > > >>> wrote:
> > > > > > >>>
> > > > > > >>> > Hi all,
> > > > > > >>> >
> > > > > > >>> > I'd like to share a status update on the ALP encoding
> effort
> > > and
> > > > > get
> > > > > > a
> > > > > > >>> feel
> > > > > > >>> > from the community before starting a formal vote.
> > > > > > >>> >
> > > > > > >>> > We now have the specification plus implementations in both
> > > > > languages:
> > > > > > >>> >
> > > > > > >>> > - Spec: apache/parquet-format#557 — GH-533 Add
> > ALP
> > > > > > encoding
> > > > > > >>> > specification
> > > > > > >>> > - C++ (Arrow): apache/arrow#48345 — GH-48701
> > > [C++][Parquet]
> > > > > Add
> > > > > > >>> ALPpd
> > > > > > >>> > encoding
> > > > > > >>> > - Java: apache/parquet-java#3397 — Parquet Java
> > ALP
> > > > > > >>> > Implementation (by Vinoo)
> > > > > > >>> >
> > > > > > >>> > Current state:
> > > > > > >>> >
> > > > > > >>> > - The C++ implementation has been through 3-4 rounds of
> > > > review.
> > > > > > >>> > - The Java implementation has been tracking well and
> has
> > > had
> > > > > > >>> initial
> > > > > > >>> > reviews from contributors, with no major remaining
> > > issues.
> > > > > > >>> > - Cross-language compatibility tests are passing: the
> > Arrow
> > > > C++
> > > > > > >>> decoder
> > > > > > >>> > reads Java-written data bit-exactly across ~1.56M
> > values
> > > > and
> > > > > 18
> > > > > > >>> > fixtures, covering V1 and V2 pages, multiple vector
> > > sizes,
> > > > > and
> > > > > > >>> > several
> > > > > > >>> > real datasets — zero mismatches.
> > > > > > >>> >
> > > > > > >>> > As raised in the java-pr we will be adding coverage for
> > > extreme
> > > > > > values
> > > > > > >>> > (those requiring
> > > > > > >>> > 63-64 bits after FOR is applied) before we close a vote.
> > We'll
> > > > aim
> > > > > > to
> > > > > > >>> get
> > > > > > >>> > that
> > > > > > >>> > done in parallel.
> > > > > > >>> >
> > > > > > >>> > Plan: unless there are objections, I intend to start a vote
> > at
> > > > the
> > > > > > end
> > > > > > >>> of
> > > > > > >>> > this week/early
> > > > > > >>> > next week.
> > > > > > >>> >
> > > > > > >>> > Thanks
> > > > > > >>> > Prateek
> > > > > > >>> >
> > > > > > >>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <
> > > prtkgaur@gmail.com>
> > > > > > >>> wrote:
> > > > > > >>> >
> > > > > > >>> > > Hi Team,
> > > > > > >>> > >
> > > > > > >>> > > Just wanted to provide some updates on ALP.
> > > > > > >>> > > Micah and I have done 3-4 rounds of review for the c++
> PR :
> > > > > > >>> > > https://github.com/apache/arrow/pull/48345
> > > > > > >>> > >
> > > > > > >>> > > For the Java implementation Vinoo has been working on
> > > following
> > > > > PR
> > > > > > :
> > > > > > >>> > > https://github.com/apache/parquet-java/pull/3397
> > > > > > >>> > >
> > > > > > >>> > > Best
> > > > > > >>> > > Prateek
> > > > > > >>> > >
> > > > > > >>> > > On Tue, May 5, 2026 at 1:50 PM Micah Kornfield <
> > > > > > >>> emkornfield@gmail.com>
> > > > > > >>> > > wrote:
> > > > > > >>> > >
> > > > > > >>> > >> Hi Antoine,
> > > > > > >>> > >>
> > > > > > >>> > >> > Apologies if the question was already asked, but
> should
> > we
> > > > > care
> > > > > > >>> about
> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient
> than
> > > > > FLOAT16
> > > > > > +
> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> > > > > > >>> > >>
> > > > > > >>> > >> It was. We thought we could defer it for the following
> > > > reasons:
> > > > > > >>> > >> 1. It's not clear there are a lot of easy reference
> > > datasets
> > > > to
> > > > > > >>> test
> > > > > > >>> > its
> > > > > > >>> > >> effectiveness. It does look like there might be one or
> > two
> > > on
> > > > > > >>> > huggingface
> > > > > > >>> > >> (e.g. https://huggingface.co/datasets/kikitora/curdie).
> > > > > > >>> > >> 2. It seemed likely that float 16 was more likely used
> > for
> > > > > values
> > > > > > >>> that
> > > > > > >>> > >> were less likely to reduce to decimal values.
> > > > > > >>> > >> 3. It could be added as an extension later if needed.
> > > > > > >>> > >>
> > > > > > >>> > >> Cheers,
> > > > > > >>> > >> Micah
> > > > > > >>> > >>
> > > > > > >>> > >>
> > > > > > >>> > >> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <
> > > > > antoine@python.org
> > > > > > >
> > > > > > >>> > wrote:
> > > > > > >>> > >>
> > > > > > >>> > >> >
> > > > > > >>> > >> > Hello,
> > > > > > >>> > >> >
> > > > > > >>> > >> > Apologies if the question was already asked, but
> should
> > we
> > > > > care
> > > > > > >>> about
> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient
> than
> > > > > FLOAT16
> > > > > > +
> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> > > > > > >>> > >> >
> > > > > > >>> > >> > Regards
> > > > > > >>> > >> >
> > > > > > >>> > >> > Antoine.
> > > > > > >>> > >> >
> > > > > > >>> > >> >
> > > > > > >>> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
> > > > > > >>> > >> > > Thanks Andrew and Micah for review feedback on the
> two
> > > > PR's
> > > > > > >>> > >> > > 1) (c++ arrow repo)
> > > > > > >>> > >> https://github.com/apache/arrow/pull/48345/changes
> > > > > > >>> > >> > > 2) (parquet-format repo)
> > > > > > >>> > >> > https://github.com/apache/parquet-format/pull/557
> > > > > > >>> > >> > >
> > > > > > >>> > >> > > I have addressed all (unless I missed something)
> > > comments
> > > > on
> > > > > > >>> the two
> > > > > > >>> > >> > PR's.
> > > > > > >>> > >> > >
> > > > > > >>> > >> > > Best
> > > > > > >>> > >> > > Prateek
> > > > > > >>> > >> > >
> > > > > > >>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <
> > > > > > >>> prtkgaur@gmail.com>
> > > > > > >>> > >> wrote:
> > > > > > >>> > >> > >
> > > > > > >>> > >> > >> Thanks Andrew and Micah.
> > > > > > >>> > >> > >>
> > > > > > >>> > >> > >> `fair amount of feedback on at least the
> > > implementations`
> > > > > > >>> > >> > >> For the c++ I have already started addressing the
> > > > > feedback, I
> > > > > > >>> > should
> > > > > > >>> > >> be
> > > > > > >>> > >> > >> done with that Monday/Tuesday.
> > > > > > >>> > >> > >> I think Vinoo too has been making good progress on
> > the
> > > > Java
> > > > > > >>> > >> > implementation.
> > > > > > >>> > >> > >>
> > > > > > >>> > >> > >> Best
> > > > > > >>> > >> > >> Prateek
> > > > > > >>> > >> > >>
> > > > > > >>> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <
> > > > > > >>> > andrewlamb11@gmail.com
> > > > > > >>> > >> >
> > > > > > >>> > >> > >> wrote:
> > > > > > >>> > >> > >>
> > > > > > >>> > >> > >>> Got it. Thank you for the clarification -- I will
> > try
> > > > and
> > > > > > look
> > > > > > >>> > into
> > > > > > >>> > >> the
> > > > > > >>> > >> > >>> spec and the Rust implementation[1] in this next
> > week
> > > > > > >>> > >> > >>>
> > > > > > >>> > >> > >>> [1]: https://github.com/apache/arrow-rs/pull/9372
> > > > > > >>> > >> > >>>
> > > > > > >>> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <
> > > > > > >>> > >> > emkornfield@gmail.com>
> > > > > > >>> > >> > >>> wrote:
> > > > > > >>> > >> > >>>
> > > > > > >>> > >> > >>>> Hi Andrew,
> > > > > > >>> > >> > >>>> I think there is a fair amount of feedback on at
> > > least
> > > > > the
> > > > > > >>> > >> > >>>> implementations, typically I think we've waited
> > till
> > > > they
> > > > > > are
> > > > > > >>> > >> close to
> > > > > > >>> > >> > >>>> mergeable before a final vote. Otherwise I agree
> > we
> > > > are
> > > > > > very
> > > > > > >>> > >> close.
> > > > > > >>> > >> > >>>>
> > > > > > >>> > >> > >>>> -Micah
> > > > > > >>> > >> > >>>>
> > > > > > >>> > >> > >>>> On Saturday, April 25, 2026, Andrew Lamb <
> > > > > > >>> andrewlamb11@gmail.com
> > > > > > >>> > >
> > > > > > >>> > >> > wrote:
> > > > > > >>> > >> > >>>>
> > > > > > >>> > >> > >>>>> Thanks Prateek,
> > > > > > >>> > >> > >>>>>
> > > > > > >>> > >> > >>>>> I think from this content it looks to me like we
> > are
> > > > > ready
> > > > > > >>> to
> > > > > > >>> > >> start a
> > > > > > >>> > >> > >>>>> vote to explicitly accept ALP into Parquet
> > > > > > >>> > >> > >>>>>
> > > > > > >>> > >> > >>>>> Does anyone know of a reason we should postpone
> it
> > > for
> > > > > > >>> longer?
> > > > > > >>> > >> > >>>>> Perhaps someone needs some more time to review?
> > > > > > >>> > >> > >>>>>
> > > > > > >>> > >> > >>>>> Andrew
> > > > > > >>> > >> > >>>>>
> > > > > > >>> > >> > >>>>>
> > > > > > >>> > >> > >>>>>
> > > > > > >>> > >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <
> > > > > > >>> > prtkgaur@gmail.com>
> > > > > > >>> > >> > >>>>> wrote:
> > > > > > >>> > >> > >>>>>
> > > > > > >>> > >> > >>>>>> Hi team,
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>> Hope everyone is doing well. I got a chance to
> > work
> > > > > > >>> through all
> > > > > > >>> > >> the
> > > > > > >>> > >> > >>>>>> remaining feedback and update the spec doc.
> Here
> > > are
> > > > > the
> > > > > > >>> new
> > > > > > >>> > >> > artifacts
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>> 1) Spec document :
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> >
> > > > > > >>> > >>
> > > > > > >>> >
> > > > > > >>>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>> 2) Spec document in parquet format repo :
> > > > > > >>> > >> > >>>>>>
> > https://github.com/apache/parquet-format/pull/557
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>> 3) Alp implementation in arrow c++ repo :
> > > > > > >>> > >> > >>>>>>
> > https://github.com/apache/arrow/pull/48345/changes
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>> 4) Alp implementation in parquet-java repo :
> Work
> > > for
> > > > > > >>> Vinoo and
> > > > > > >>> > >> > Julien
> > > > > > >>> > >> > >>>>>>
> > https://github.com/apache/parquet-java/pull/3397
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>> 5) PR with test and benchmarking artifacts in
> > > > > > >>> parquet-testing
> > > > > > >>> > >> repo :
> > > > > > >>> > >> > >>>>>>
> > https://github.com/apache/parquet-testing/pull/100
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>> And
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>> - Go : Arnav just submitted an in progress
> > > > > > >>> implementation
> > > > > > >>> > in
> > > > > > >>> > >> Go.
> > > > > > >>> > >> > >>>>>>
> https://github.com/apache/arrow-go/pull/704
> > (I
> > > > > > haven't
> > > > > > >>> > >> started
> > > > > > >>> > >> > >>>>>> looking at it yet)
> > > > > > >>> > >> > >>>>>> - Rust : I remember Andrew mentioned that
> > this
> > > > work
> > > > > > is
> > > > > > >>> also
> > > > > > >>> > >> in
> > > > > > >>> > >> > >>>>>> progress (So 4 languages!)
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>> *Arrow C++ implementation *
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>> The PR is out and was also used by Antoine to
> > > report
> > > > > the
> > > > > > >>> > numbers
> > > > > > >>> > >> as
> > > > > > >>> > >> > >>>>>> reported here. Micah and Konstantin have given
> 1
> > > > round
> > > > > of
> > > > > > >>> > >> feedback
> > > > > > >>> > >> > >>>>>> and I'm addressing them today. Please note that
> > the
> > > > > > default
> > > > > > >>> > >> > >>>>>> optimization flag for compiling is O2 and not
> > Q3. I
> > > > got
> > > > > > >>> around
> > > > > > >>> > >> 70%
> > > > > > >>> > >> > >>>>>> performance improvement in the decoding speed
> > when
> > > > > using
> > > > > > >>> the O3
> > > > > > >>> > >> > flag.
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>> *Parqet-MR Java implementation (working with
> > Vinoo
> > > > and
> > > > > > >>> Julien)
> > > > > > >>> > >> and
> > > > > > >>> > >> > **Cross
> > > > > > >>> > >> > >>>>>> Language testing*
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>> Let me know if you have any questions or
> > > > feedback.
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>> Now pasting some performance numbers
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>> Table 1: C++ ALP Double Decode — Spotify
> > Columns
> > > > > > >>> (Graviton
> > > > > > >>> > 3,
> > > > > > >>> > >> ARM
> > > > > > >>> > >> > >>>>>> Neoverse V1)
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>> │ Column │ -O2 (MB/s) │ -O3
> > > (MB/s) │
> > > > > > >>> Speedup │
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> ├──────────────────┼──────────────┼──────────────┼─────────┤
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>> │ valence │ 3,155 │
> 5,523
> > > │
> > > > > > >>> 1.75x │
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>> │ danceability │ 3,233 │
> 5,685
> > > │
> > > > > > >>> 1.76x │
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>> │ energy │ 3,197 │
> 5,652
> > > │
> > > > > > >>> 1.77x │
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>> │ loudness │ 3,186 │
> 5,473
> > > │
> > > > > > >>> 1.72x │
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> └──────────────────┴──────────────┴──────────────┴─────────┘
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <
> > > > > > >>> > prtkgaur@gmail.com
> > > > > > >>> > >> >
> > > > > > >>> > >> > >>>>>> wrote:
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>>> @Micah Kornfield <em...@gmail.com> :
> Got
> > > it.
> > > > > > >>> > >> > >>>>>>>
> > > > > > >>> > >> > >>>>>>> @Andrew Lamb <an...@gmail.com>
> > > > > > >>> > >> > >>>>>>>
> > > > > > >>> > >> > >>>>>>>
> > > > > > >>> > >> > >>>>>>>> Do you think it would be good to start moving
> > the
> > > > > spec
> > > > > > >>> > >> development
> > > > > > >>> > >> > >>>>>>>> into
> > > > > > >>> > >> > >>>>>>>> markdown format, in preparation for
> finalizing
> > > it?
> > > > > > >>> > >> > >>>>>>>>
> > > > > > >>> > >> > >>>>>>>
> > > > > > >>> > >> > >>>>>>> Yes I'll update the numbers for some of the
> > > > examples I
> > > > > > >>> have in
> > > > > > >>> > >> the
> > > > > > >>> > >> > >>>>>>> spec based
> > > > > > >>> > >> > >>>>>>> on the updated header size. Then we should be
> > good
> > > > to
> > > > > go
> > > > > > >>> for
> > > > > > >>> > the
> > > > > > >>> > >> > >>>>>>> markdown format.
> > > > > > >>> > >> > >>>>>>>
> > > > > > >>> > >> > >>>>>>> Thanks everyone!
> > > > > > >>> > >> > >>>>>>>
> > > > > > >>> > >> > >>>>>>>
> > > > > > >>> > >> > >>>>>>>>
> > > > > > >>> > >> > >>>>>>>> Andrew
> > > > > > >>> > >> > >>>>>>>>
> > > > > > >>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR
> <
> > > > > > >>> > >> prtkgaur@gmail.com>
> > > > > > >>> > >> > >>>>>>>> wrote:
> > > > > > >>> > >> > >>>>>>>>
> > > > > > >>> > >> > >>>>>>>>> Hi team,
> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >>> > >> > >>>>>>>>> 1) Andrew
> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >>> > >> > >>>>>>>>> - Thanks for working on test files. My
> PR
> > > did
> > > > > add
> > > > > > >>> all
> > > > > > >>> > the
> > > > > > >>> > >> > test
> > > > > > >>> > >> > >>>>>>>> files I
> > > > > > >>> > >> > >>>>>>>>> used to benchmark on datasets. Maybe we
> > can
> > > > club
> > > > > > it
> > > > > > >>> > >> together.
> > > > > > >>> > >> > >>>>>>>> WIll also
> > > > > > >>> > >> > >>>>>>>>> aid
> > > > > > >>> > >> > >>>>>>>>> cross language testing
> > > > > > >>> > >> > >>>>>>>>> - Kosta Tarasov working on Rust
> > > > implementation.
> > > > > > >>> This is
> > > > > > >>> > >> > great.
> > > > > > >>> > >> > >>>>>>>> Thanks
> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >>> > >> > >>>>>>>>> 2) Antoine
> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >>> > >> > >>>>>>>>> - Thanks a lot for reporting the numbers
> > on
> > > > AMD.
> > > > > > >>> Looks
> > > > > > >>> > >> like
> > > > > > >>> > >> > you
> > > > > > >>> > >> > >>>>>>>> are
> > > > > > >>> > >> > >>>>>>>>> getting 8X the decoding performance of
> > BSS.
> > > > This
> > > > > > is
> > > > > > >>> > >> > amazing!!.
> > > > > > >>> > >> > >>>>>>>>> - Thanks for acknowledging the sampling
> > > > design.
> > > > > > >>> > >> > >>>>>>>>> - I agree with you on Fastlanes. In some
> > > crude
> > > > > > >>> > >> experiments I
> > > > > > >>> > >> > >>>>>>>> didn't get
> > > > > > >>> > >> > >>>>>>>>> a good perf benefit from it on Graviton3
> > > (but
> > > > > > maybe
> > > > > > >>> > there
> > > > > > >>> > >> was
> > > > > > >>> > >> > >>>>>>>> something
> > > > > > >>> > >> > >>>>>>>>> wrong with my implementation).
> > > > > > >>> > >> > >>>>>>>>> - Locking the 16bit exception encoding
> for
> > > the
> > > > > > spec
> > > > > > >>> in
> > > > > > >>> > >> this
> > > > > > >>> > >> > >>>>>>>> case.
> > > > > > >>> > >> > >>>>>>>>> - Awesome I think we have solved for all
> > > open
> > > > > > >>> questions
> > > > > > >>> > >> minus
> > > > > > >>> > >> > >>>>>>>> the
> > > > > > >>> > >> > >>>>>>>>> version byte :). (will get back on this
> > > soon)
> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >>> > >> > >>>>>>>>> 3) Micah
> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >>> > >> > >>>>>>>>> - FastLanes : The current spec does
> allow
> > > for
> > > > > > using
> > > > > > >>> > >> FastLane
> > > > > > >>> > >> > >>>>>>>> with the
> > > > > > >>> > >> > >>>>>>>>> configurable enum value for layout. We
> > > should
> > > > be
> > > > > > >>> able to
> > > > > > >>> > >> > inject
> > > > > > >>> > >> > >>>>>>>> any
> > > > > > >>> > >> > >>>>>>>>> layout
> > > > > > >>> > >> > >>>>>>>>> in the current design.
> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >>> > >> > >>>>>>>>> Working on resolving all remaining open
> > comments
> > > > on
> > > > > > the
> > > > > > >>> spec
> > > > > > >>> > >> this
> > > > > > >>> > >> > >>>>>>>> week.
> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >>> > >> > >>>>>>>>> Best
> > > > > > >>> > >> > >>>>>>>>> Prateek
> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve
> > Loughran <
> > > > > > >>> > >> > >>>>>>>> stevel@cloudera.com>
> > > > > > >>> > >> > >>>>>>>>> wrote:
> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah
> Kornfield
> > <
> > > > > > >>> > >> > >>>>>>>> emkornfield@gmail.com>
> > > > > > >>> > >> > >>>>>>>>>> wrote:
> > > > > > >>> > >> > >>>>>>>>>>
> > > > > > >>> > >> > >>>>>>>>>>>
> > > > > > >>> > >> > >>>>>>>>>>>
> > > > > > >>> > >> > >>>>>>>>>>> It looks like the actual issue described
> for
> > > ORC
> > > > > in
> > > > > > >>> the
> > > > > > >>> > >> paper
> > > > > > >>> > >> > >>>>>>>> is that
> > > > > > >>> > >> > >>>>>>>>> it
> > > > > > >>> > >> > >>>>>>>>>>> has multiple sub-encodings in a batch.
> This
> > > is
> > > > > > >>> different
> > > > > > >>> > >> then
> > > > > > >>> > >> > >>>>>>>> the
> > > > > > >>> > >> > >>>>>>>>> design
> > > > > > >>> > >> > >>>>>>>>>>> proposed here where there is still fixed
> > > > encoding
> > > > > > per
> > > > > > >>> page
> > > > > > >>> > >> in
> > > > > > >>> > >> > >>>>>>>> parquet.
> > > > > > >>> > >> > >>>>>>>>>>> Given reasonably sized pages I don't think
> > > > branch
> > > > > > >>> > >> > >>>>>>>> misprediction should
> > > > > > >>> > >> > >>>>>>>>>> be a
> > > > > > >>> > >> > >>>>>>>>>>> big issue for new encodings. I agree that
> > we
> > > > > should
> > > > > > >>> be
> > > > > > >>> > >> > >>>>>>>> conservative in
> > > > > > >>> > >> > >>>>>>>>>>> general for adding new encodings.
> > > > > > >>> > >> > >>>>>>>>>>>
> > > > > > >>> > >> > >>>>>>>>>>>
> > > > > > >>> > >> > >>>>>>>>>> +1
> > > > > > >>> > >> > >>>>>>>>>>
> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >>> > >> > >>>>>>>>
> > > > > > >>> > >> > >>>>>>>
> > > > > > >>> > >> > >
> > > > > > >>> > >> >
> > > > > > >>> > >> >
> > > > > > >>> > >> >
> > > > > > >>> > >>
> > > > > > >>> > >
> > > > > > >>> >
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Divjot Arora via dev <de...@parquet.apache.org>.
Hi Andrew,
How does this interact with the ongoing versioning discussion? Given that
new encodings are not forward compatible, I would assume ALP is included in
Parquet V<next> (V3?), but it's not clear to me how that works if it's
merged to the parquet-format master branch now. Will we exclude this change
when cutting parquet-format v2.14.0?
-- Div
On Thu, Jul 30, 2026 at 8:51 AM Andrew Lamb <an...@gmail.com> wrote:
> Hello,
>
> As an update, thanks to the epic work from many contributors, especially
> Prateek, I just merged the parquet-format change for ALP[1]. We plan to
> make additional editorial The remaining items I know of are merging the
> examples from parquet-testing[2], and then of course the various
> implementations
>
> Thanks again everyone. This is a great step forward
> Andrew
>
> [1]: https://github.com/apache/parquet-format/pull/557
> [2]: https://github.com/apache/parquet-testing/pull/100
>
> On Tue, Jul 21, 2026 at 4:39 PM Julien Le Dem <ju...@apache.org> wrote:
>
> > It was also in my spam folder... :(
> > I've seen a bunch of apache list emails go to my spam recently.
> >
> > On Tue, Jul 21, 2026 at 10:58 AM Russell Spitzer <
> > russell.spitzer@gmail.com>
> > wrote:
> >
> > > Yes, my hope was that my vote would kick it out of spam but we
> definitely
> > > had two other votes that also went to spam for me.
> > >
> > > For anyone looking the thread is here
> > > https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt
> > >
> > > On Tue, Jul 21, 2026 at 10:57 AM Micah Kornfield <
> emkornfield@gmail.com>
> > > wrote:
> > >
> > > > I've seen other's votes not go to spam, so hopefully people will see
> it
> > > in
> > > > there inboxes (or see this thread and look in there spam).
> > > >
> > > > On Tue, Jul 21, 2026 at 10:52 AM PRATEEK GAUR <pr...@gmail.com>
> > > wrote:
> > > >
> > > > > Okay,
> > > > >
> > > > > I sent out the voting email yesterday and so far I've heard from 3
> > > people
> > > > > that it ended up in their "spam folder". Dev VPN to be blamed for
> > it?.
> > > > >
> > > > > What's the protocol here?
> > > > > I let people hunt for the email in their spam folder or send out
> > > another
> > > > > one?
> > > > >
> > > > > Best
> > > > > Prateek
> > > > >
> > > > > On Sun, Jul 19, 2026 at 10:47 PM PRATEEK GAUR <pr...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > Hi team,
> > > > > >
> > > > > > Vinoo has added the tests which were requested in the last round
> of
> > > > > review.
> > > > > > In his own words.
> > > > > >
> > > > > > "
> > > > > > The extreme value tests Micah wanted are done and pushed. I added
> > > > > coverage
> > > > > > for values that need the full FOR bit width after the frame of
> > > > reference
> > > > > is
> > > > > > applied: 64-bit (the case where the signed max minus min
> > subtraction
> > > > > > overflows), 63-bit (the non-overflow case), and 32-bit for
> floats.
> > > All
> > > > > > lossless with zero exceptions. I also verified them through Arrow
> > C++
> > > > ALP
> > > > > > reader (the local build), so the Java-written extreme columns
> > decode
> > > > > > correctly in C++, bit-exact against the expected values."
> > > > > >
> > > > > > State of the PR (#3397):
> > > > > >
> > > > > > - Merges cleanly with master (I pulled master in and resolved
> > the
> > > > > > conflicts).
> > > > > > - 41 of 44 review threads resolved. The 3 remaining are
> > > intentional:
> > > > > > one deferred perf nit, one that stays open until
> parquet-format
> > > > ships
> > > > > ALP,
> > > > > >
> > > > > > "
> > > > > >
> > > > > > With this both the
> > > > > > 1) C++ PR : https://github.com/apache/arrow/pull/48345/changes
> > > > > > 2) Java PR : https://github.com/apache/parquet-java/pull/3397
> > > > > >
> > > > > > Are cross language tested and range of test scenarios.
> > > > > >
> > > > > > Best
> > > > > > Prateek
> > > > > >
> > > > > >
> > > > > > On Mon, Jul 13, 2026 at 8:38 AM PRATEEK GAUR <prtkgaur@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > >> Thanks Matt,
> > > > > >>
> > > > > >> I'll be on it this week.
> > > > > >>
> > > > > >> Best
> > > > > >> Prateek
> > > > > >>
> > > > > >> On Mon, Jul 13, 2026 at 8:36 AM Matt Topol <
> > zotthewizard@gmail.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >>> There is also the Go implementation submitted by Arnav (
> > > > > >>> https://github.com/apache/arrow-go/pull/704) which is waiting
> > for
> > > > > >>> updates.
> > > > > >>>
> > > > > >>> Just wanted to make sure it didn't get lost here.
> > > > > >>>
> > > > > >>> --Matt
> > > > > >>>
> > > > > >>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK GAUR <
> > prtkgaur@gmail.com>
> > > > > >>> wrote:
> > > > > >>>
> > > > > >>> > Hi all,
> > > > > >>> >
> > > > > >>> > I'd like to share a status update on the ALP encoding effort
> > and
> > > > get
> > > > > a
> > > > > >>> feel
> > > > > >>> > from the community before starting a formal vote.
> > > > > >>> >
> > > > > >>> > We now have the specification plus implementations in both
> > > > languages:
> > > > > >>> >
> > > > > >>> > - Spec: apache/parquet-format#557 — GH-533 Add
> ALP
> > > > > encoding
> > > > > >>> > specification
> > > > > >>> > - C++ (Arrow): apache/arrow#48345 — GH-48701
> > [C++][Parquet]
> > > > Add
> > > > > >>> ALPpd
> > > > > >>> > encoding
> > > > > >>> > - Java: apache/parquet-java#3397 — Parquet Java
> ALP
> > > > > >>> > Implementation (by Vinoo)
> > > > > >>> >
> > > > > >>> > Current state:
> > > > > >>> >
> > > > > >>> > - The C++ implementation has been through 3-4 rounds of
> > > review.
> > > > > >>> > - The Java implementation has been tracking well and has
> > had
> > > > > >>> initial
> > > > > >>> > reviews from contributors, with no major remaining
> > issues.
> > > > > >>> > - Cross-language compatibility tests are passing: the
> Arrow
> > > C++
> > > > > >>> decoder
> > > > > >>> > reads Java-written data bit-exactly across ~1.56M
> values
> > > and
> > > > 18
> > > > > >>> > fixtures, covering V1 and V2 pages, multiple vector
> > sizes,
> > > > and
> > > > > >>> > several
> > > > > >>> > real datasets — zero mismatches.
> > > > > >>> >
> > > > > >>> > As raised in the java-pr we will be adding coverage for
> > extreme
> > > > > values
> > > > > >>> > (those requiring
> > > > > >>> > 63-64 bits after FOR is applied) before we close a vote.
> We'll
> > > aim
> > > > > to
> > > > > >>> get
> > > > > >>> > that
> > > > > >>> > done in parallel.
> > > > > >>> >
> > > > > >>> > Plan: unless there are objections, I intend to start a vote
> at
> > > the
> > > > > end
> > > > > >>> of
> > > > > >>> > this week/early
> > > > > >>> > next week.
> > > > > >>> >
> > > > > >>> > Thanks
> > > > > >>> > Prateek
> > > > > >>> >
> > > > > >>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <
> > prtkgaur@gmail.com>
> > > > > >>> wrote:
> > > > > >>> >
> > > > > >>> > > Hi Team,
> > > > > >>> > >
> > > > > >>> > > Just wanted to provide some updates on ALP.
> > > > > >>> > > Micah and I have done 3-4 rounds of review for the c++ PR :
> > > > > >>> > > https://github.com/apache/arrow/pull/48345
> > > > > >>> > >
> > > > > >>> > > For the Java implementation Vinoo has been working on
> > following
> > > > PR
> > > > > :
> > > > > >>> > > https://github.com/apache/parquet-java/pull/3397
> > > > > >>> > >
> > > > > >>> > > Best
> > > > > >>> > > Prateek
> > > > > >>> > >
> > > > > >>> > > On Tue, May 5, 2026 at 1:50 PM Micah Kornfield <
> > > > > >>> emkornfield@gmail.com>
> > > > > >>> > > wrote:
> > > > > >>> > >
> > > > > >>> > >> Hi Antoine,
> > > > > >>> > >>
> > > > > >>> > >> > Apologies if the question was already asked, but should
> we
> > > > care
> > > > > >>> about
> > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than
> > > > FLOAT16
> > > > > +
> > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> > > > > >>> > >>
> > > > > >>> > >> It was. We thought we could defer it for the following
> > > reasons:
> > > > > >>> > >> 1. It's not clear there are a lot of easy reference
> > datasets
> > > to
> > > > > >>> test
> > > > > >>> > its
> > > > > >>> > >> effectiveness. It does look like there might be one or
> two
> > on
> > > > > >>> > huggingface
> > > > > >>> > >> (e.g. https://huggingface.co/datasets/kikitora/curdie).
> > > > > >>> > >> 2. It seemed likely that float 16 was more likely used
> for
> > > > values
> > > > > >>> that
> > > > > >>> > >> were less likely to reduce to decimal values.
> > > > > >>> > >> 3. It could be added as an extension later if needed.
> > > > > >>> > >>
> > > > > >>> > >> Cheers,
> > > > > >>> > >> Micah
> > > > > >>> > >>
> > > > > >>> > >>
> > > > > >>> > >> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <
> > > > antoine@python.org
> > > > > >
> > > > > >>> > wrote:
> > > > > >>> > >>
> > > > > >>> > >> >
> > > > > >>> > >> > Hello,
> > > > > >>> > >> >
> > > > > >>> > >> > Apologies if the question was already asked, but should
> we
> > > > care
> > > > > >>> about
> > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than
> > > > FLOAT16
> > > > > +
> > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> > > > > >>> > >> >
> > > > > >>> > >> > Regards
> > > > > >>> > >> >
> > > > > >>> > >> > Antoine.
> > > > > >>> > >> >
> > > > > >>> > >> >
> > > > > >>> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
> > > > > >>> > >> > > Thanks Andrew and Micah for review feedback on the two
> > > PR's
> > > > > >>> > >> > > 1) (c++ arrow repo)
> > > > > >>> > >> https://github.com/apache/arrow/pull/48345/changes
> > > > > >>> > >> > > 2) (parquet-format repo)
> > > > > >>> > >> > https://github.com/apache/parquet-format/pull/557
> > > > > >>> > >> > >
> > > > > >>> > >> > > I have addressed all (unless I missed something)
> > comments
> > > on
> > > > > >>> the two
> > > > > >>> > >> > PR's.
> > > > > >>> > >> > >
> > > > > >>> > >> > > Best
> > > > > >>> > >> > > Prateek
> > > > > >>> > >> > >
> > > > > >>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <
> > > > > >>> prtkgaur@gmail.com>
> > > > > >>> > >> wrote:
> > > > > >>> > >> > >
> > > > > >>> > >> > >> Thanks Andrew and Micah.
> > > > > >>> > >> > >>
> > > > > >>> > >> > >> `fair amount of feedback on at least the
> > implementations`
> > > > > >>> > >> > >> For the c++ I have already started addressing the
> > > > feedback, I
> > > > > >>> > should
> > > > > >>> > >> be
> > > > > >>> > >> > >> done with that Monday/Tuesday.
> > > > > >>> > >> > >> I think Vinoo too has been making good progress on
> the
> > > Java
> > > > > >>> > >> > implementation.
> > > > > >>> > >> > >>
> > > > > >>> > >> > >> Best
> > > > > >>> > >> > >> Prateek
> > > > > >>> > >> > >>
> > > > > >>> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <
> > > > > >>> > andrewlamb11@gmail.com
> > > > > >>> > >> >
> > > > > >>> > >> > >> wrote:
> > > > > >>> > >> > >>
> > > > > >>> > >> > >>> Got it. Thank you for the clarification -- I will
> try
> > > and
> > > > > look
> > > > > >>> > into
> > > > > >>> > >> the
> > > > > >>> > >> > >>> spec and the Rust implementation[1] in this next
> week
> > > > > >>> > >> > >>>
> > > > > >>> > >> > >>> [1]: https://github.com/apache/arrow-rs/pull/9372
> > > > > >>> > >> > >>>
> > > > > >>> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <
> > > > > >>> > >> > emkornfield@gmail.com>
> > > > > >>> > >> > >>> wrote:
> > > > > >>> > >> > >>>
> > > > > >>> > >> > >>>> Hi Andrew,
> > > > > >>> > >> > >>>> I think there is a fair amount of feedback on at
> > least
> > > > the
> > > > > >>> > >> > >>>> implementations, typically I think we've waited
> till
> > > they
> > > > > are
> > > > > >>> > >> close to
> > > > > >>> > >> > >>>> mergeable before a final vote. Otherwise I agree
> we
> > > are
> > > > > very
> > > > > >>> > >> close.
> > > > > >>> > >> > >>>>
> > > > > >>> > >> > >>>> -Micah
> > > > > >>> > >> > >>>>
> > > > > >>> > >> > >>>> On Saturday, April 25, 2026, Andrew Lamb <
> > > > > >>> andrewlamb11@gmail.com
> > > > > >>> > >
> > > > > >>> > >> > wrote:
> > > > > >>> > >> > >>>>
> > > > > >>> > >> > >>>>> Thanks Prateek,
> > > > > >>> > >> > >>>>>
> > > > > >>> > >> > >>>>> I think from this content it looks to me like we
> are
> > > > ready
> > > > > >>> to
> > > > > >>> > >> start a
> > > > > >>> > >> > >>>>> vote to explicitly accept ALP into Parquet
> > > > > >>> > >> > >>>>>
> > > > > >>> > >> > >>>>> Does anyone know of a reason we should postpone it
> > for
> > > > > >>> longer?
> > > > > >>> > >> > >>>>> Perhaps someone needs some more time to review?
> > > > > >>> > >> > >>>>>
> > > > > >>> > >> > >>>>> Andrew
> > > > > >>> > >> > >>>>>
> > > > > >>> > >> > >>>>>
> > > > > >>> > >> > >>>>>
> > > > > >>> > >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <
> > > > > >>> > prtkgaur@gmail.com>
> > > > > >>> > >> > >>>>> wrote:
> > > > > >>> > >> > >>>>>
> > > > > >>> > >> > >>>>>> Hi team,
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>> Hope everyone is doing well. I got a chance to
> work
> > > > > >>> through all
> > > > > >>> > >> the
> > > > > >>> > >> > >>>>>> remaining feedback and update the spec doc. Here
> > are
> > > > the
> > > > > >>> new
> > > > > >>> > >> > artifacts
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>> 1) Spec document :
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> >
> > > > > >>> > >>
> > > > > >>> >
> > > > > >>>
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>> 2) Spec document in parquet format repo :
> > > > > >>> > >> > >>>>>>
> https://github.com/apache/parquet-format/pull/557
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>> 3) Alp implementation in arrow c++ repo :
> > > > > >>> > >> > >>>>>>
> https://github.com/apache/arrow/pull/48345/changes
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>> 4) Alp implementation in parquet-java repo : Work
> > for
> > > > > >>> Vinoo and
> > > > > >>> > >> > Julien
> > > > > >>> > >> > >>>>>>
> https://github.com/apache/parquet-java/pull/3397
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>> 5) PR with test and benchmarking artifacts in
> > > > > >>> parquet-testing
> > > > > >>> > >> repo :
> > > > > >>> > >> > >>>>>>
> https://github.com/apache/parquet-testing/pull/100
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>> And
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>> - Go : Arnav just submitted an in progress
> > > > > >>> implementation
> > > > > >>> > in
> > > > > >>> > >> Go.
> > > > > >>> > >> > >>>>>> https://github.com/apache/arrow-go/pull/704
> (I
> > > > > haven't
> > > > > >>> > >> started
> > > > > >>> > >> > >>>>>> looking at it yet)
> > > > > >>> > >> > >>>>>> - Rust : I remember Andrew mentioned that
> this
> > > work
> > > > > is
> > > > > >>> also
> > > > > >>> > >> in
> > > > > >>> > >> > >>>>>> progress (So 4 languages!)
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>> *Arrow C++ implementation *
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>> The PR is out and was also used by Antoine to
> > report
> > > > the
> > > > > >>> > numbers
> > > > > >>> > >> as
> > > > > >>> > >> > >>>>>> reported here. Micah and Konstantin have given 1
> > > round
> > > > of
> > > > > >>> > >> feedback
> > > > > >>> > >> > >>>>>> and I'm addressing them today. Please note that
> the
> > > > > default
> > > > > >>> > >> > >>>>>> optimization flag for compiling is O2 and not
> Q3. I
> > > got
> > > > > >>> around
> > > > > >>> > >> 70%
> > > > > >>> > >> > >>>>>> performance improvement in the decoding speed
> when
> > > > using
> > > > > >>> the O3
> > > > > >>> > >> > flag.
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>> *Parqet-MR Java implementation (working with
> Vinoo
> > > and
> > > > > >>> Julien)
> > > > > >>> > >> and
> > > > > >>> > >> > **Cross
> > > > > >>> > >> > >>>>>> Language testing*
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>> Let me know if you have any questions or
> > > feedback.
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>> Now pasting some performance numbers
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>> Table 1: C++ ALP Double Decode — Spotify
> Columns
> > > > > >>> (Graviton
> > > > > >>> > 3,
> > > > > >>> > >> ARM
> > > > > >>> > >> > >>>>>> Neoverse V1)
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>>
> > > > > >>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>> │ Column │ -O2 (MB/s) │ -O3
> > (MB/s) │
> > > > > >>> Speedup │
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>>
> > > > > >>> ├──────────────────┼──────────────┼──────────────┼─────────┤
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>> │ valence │ 3,155 │ 5,523
> > │
> > > > > >>> 1.75x │
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>> │ danceability │ 3,233 │ 5,685
> > │
> > > > > >>> 1.76x │
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>> │ energy │ 3,197 │ 5,652
> > │
> > > > > >>> 1.77x │
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>> │ loudness │ 3,186 │ 5,473
> > │
> > > > > >>> 1.72x │
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>>
> > > > > >>> └──────────────────┴──────────────┴──────────────┴─────────┘
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <
> > > > > >>> > prtkgaur@gmail.com
> > > > > >>> > >> >
> > > > > >>> > >> > >>>>>> wrote:
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>>> @Micah Kornfield <em...@gmail.com> : Got
> > it.
> > > > > >>> > >> > >>>>>>>
> > > > > >>> > >> > >>>>>>> @Andrew Lamb <an...@gmail.com>
> > > > > >>> > >> > >>>>>>>
> > > > > >>> > >> > >>>>>>>
> > > > > >>> > >> > >>>>>>>> Do you think it would be good to start moving
> the
> > > > spec
> > > > > >>> > >> development
> > > > > >>> > >> > >>>>>>>> into
> > > > > >>> > >> > >>>>>>>> markdown format, in preparation for finalizing
> > it?
> > > > > >>> > >> > >>>>>>>>
> > > > > >>> > >> > >>>>>>>
> > > > > >>> > >> > >>>>>>> Yes I'll update the numbers for some of the
> > > examples I
> > > > > >>> have in
> > > > > >>> > >> the
> > > > > >>> > >> > >>>>>>> spec based
> > > > > >>> > >> > >>>>>>> on the updated header size. Then we should be
> good
> > > to
> > > > go
> > > > > >>> for
> > > > > >>> > the
> > > > > >>> > >> > >>>>>>> markdown format.
> > > > > >>> > >> > >>>>>>>
> > > > > >>> > >> > >>>>>>> Thanks everyone!
> > > > > >>> > >> > >>>>>>>
> > > > > >>> > >> > >>>>>>>
> > > > > >>> > >> > >>>>>>>>
> > > > > >>> > >> > >>>>>>>> Andrew
> > > > > >>> > >> > >>>>>>>>
> > > > > >>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <
> > > > > >>> > >> prtkgaur@gmail.com>
> > > > > >>> > >> > >>>>>>>> wrote:
> > > > > >>> > >> > >>>>>>>>
> > > > > >>> > >> > >>>>>>>>> Hi team,
> > > > > >>> > >> > >>>>>>>>>
> > > > > >>> > >> > >>>>>>>>> 1) Andrew
> > > > > >>> > >> > >>>>>>>>>
> > > > > >>> > >> > >>>>>>>>> - Thanks for working on test files. My PR
> > did
> > > > add
> > > > > >>> all
> > > > > >>> > the
> > > > > >>> > >> > test
> > > > > >>> > >> > >>>>>>>> files I
> > > > > >>> > >> > >>>>>>>>> used to benchmark on datasets. Maybe we
> can
> > > club
> > > > > it
> > > > > >>> > >> together.
> > > > > >>> > >> > >>>>>>>> WIll also
> > > > > >>> > >> > >>>>>>>>> aid
> > > > > >>> > >> > >>>>>>>>> cross language testing
> > > > > >>> > >> > >>>>>>>>> - Kosta Tarasov working on Rust
> > > implementation.
> > > > > >>> This is
> > > > > >>> > >> > great.
> > > > > >>> > >> > >>>>>>>> Thanks
> > > > > >>> > >> > >>>>>>>>>
> > > > > >>> > >> > >>>>>>>>>
> > > > > >>> > >> > >>>>>>>>> 2) Antoine
> > > > > >>> > >> > >>>>>>>>>
> > > > > >>> > >> > >>>>>>>>> - Thanks a lot for reporting the numbers
> on
> > > AMD.
> > > > > >>> Looks
> > > > > >>> > >> like
> > > > > >>> > >> > you
> > > > > >>> > >> > >>>>>>>> are
> > > > > >>> > >> > >>>>>>>>> getting 8X the decoding performance of
> BSS.
> > > This
> > > > > is
> > > > > >>> > >> > amazing!!.
> > > > > >>> > >> > >>>>>>>>> - Thanks for acknowledging the sampling
> > > design.
> > > > > >>> > >> > >>>>>>>>> - I agree with you on Fastlanes. In some
> > crude
> > > > > >>> > >> experiments I
> > > > > >>> > >> > >>>>>>>> didn't get
> > > > > >>> > >> > >>>>>>>>> a good perf benefit from it on Graviton3
> > (but
> > > > > maybe
> > > > > >>> > there
> > > > > >>> > >> was
> > > > > >>> > >> > >>>>>>>> something
> > > > > >>> > >> > >>>>>>>>> wrong with my implementation).
> > > > > >>> > >> > >>>>>>>>> - Locking the 16bit exception encoding for
> > the
> > > > > spec
> > > > > >>> in
> > > > > >>> > >> this
> > > > > >>> > >> > >>>>>>>> case.
> > > > > >>> > >> > >>>>>>>>> - Awesome I think we have solved for all
> > open
> > > > > >>> questions
> > > > > >>> > >> minus
> > > > > >>> > >> > >>>>>>>> the
> > > > > >>> > >> > >>>>>>>>> version byte :). (will get back on this
> > soon)
> > > > > >>> > >> > >>>>>>>>>
> > > > > >>> > >> > >>>>>>>>>
> > > > > >>> > >> > >>>>>>>>> 3) Micah
> > > > > >>> > >> > >>>>>>>>>
> > > > > >>> > >> > >>>>>>>>> - FastLanes : The current spec does allow
> > for
> > > > > using
> > > > > >>> > >> FastLane
> > > > > >>> > >> > >>>>>>>> with the
> > > > > >>> > >> > >>>>>>>>> configurable enum value for layout. We
> > should
> > > be
> > > > > >>> able to
> > > > > >>> > >> > inject
> > > > > >>> > >> > >>>>>>>> any
> > > > > >>> > >> > >>>>>>>>> layout
> > > > > >>> > >> > >>>>>>>>> in the current design.
> > > > > >>> > >> > >>>>>>>>>
> > > > > >>> > >> > >>>>>>>>>
> > > > > >>> > >> > >>>>>>>>> Working on resolving all remaining open
> comments
> > > on
> > > > > the
> > > > > >>> spec
> > > > > >>> > >> this
> > > > > >>> > >> > >>>>>>>> week.
> > > > > >>> > >> > >>>>>>>>>
> > > > > >>> > >> > >>>>>>>>> Best
> > > > > >>> > >> > >>>>>>>>> Prateek
> > > > > >>> > >> > >>>>>>>>>
> > > > > >>> > >> > >>>>>>>>>
> > > > > >>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve
> Loughran <
> > > > > >>> > >> > >>>>>>>> stevel@cloudera.com>
> > > > > >>> > >> > >>>>>>>>> wrote:
> > > > > >>> > >> > >>>>>>>>>
> > > > > >>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah Kornfield
> <
> > > > > >>> > >> > >>>>>>>> emkornfield@gmail.com>
> > > > > >>> > >> > >>>>>>>>>> wrote:
> > > > > >>> > >> > >>>>>>>>>>
> > > > > >>> > >> > >>>>>>>>>>>
> > > > > >>> > >> > >>>>>>>>>>>
> > > > > >>> > >> > >>>>>>>>>>> It looks like the actual issue described for
> > ORC
> > > > in
> > > > > >>> the
> > > > > >>> > >> paper
> > > > > >>> > >> > >>>>>>>> is that
> > > > > >>> > >> > >>>>>>>>> it
> > > > > >>> > >> > >>>>>>>>>>> has multiple sub-encodings in a batch. This
> > is
> > > > > >>> different
> > > > > >>> > >> then
> > > > > >>> > >> > >>>>>>>> the
> > > > > >>> > >> > >>>>>>>>> design
> > > > > >>> > >> > >>>>>>>>>>> proposed here where there is still fixed
> > > encoding
> > > > > per
> > > > > >>> page
> > > > > >>> > >> in
> > > > > >>> > >> > >>>>>>>> parquet.
> > > > > >>> > >> > >>>>>>>>>>> Given reasonably sized pages I don't think
> > > branch
> > > > > >>> > >> > >>>>>>>> misprediction should
> > > > > >>> > >> > >>>>>>>>>> be a
> > > > > >>> > >> > >>>>>>>>>>> big issue for new encodings. I agree that
> we
> > > > should
> > > > > >>> be
> > > > > >>> > >> > >>>>>>>> conservative in
> > > > > >>> > >> > >>>>>>>>>>> general for adding new encodings.
> > > > > >>> > >> > >>>>>>>>>>>
> > > > > >>> > >> > >>>>>>>>>>>
> > > > > >>> > >> > >>>>>>>>>> +1
> > > > > >>> > >> > >>>>>>>>>>
> > > > > >>> > >> > >>>>>>>>>
> > > > > >>> > >> > >>>>>>>>
> > > > > >>> > >> > >>>>>>>
> > > > > >>> > >> > >
> > > > > >>> > >> >
> > > > > >>> > >> >
> > > > > >>> > >> >
> > > > > >>> > >>
> > > > > >>> > >
> > > > > >>> >
> > > > > >>>
> > > > > >>
> > > > >
> > > >
> > >
> >
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Andrew Lamb <an...@gmail.com>.
Hello,
As an update, thanks to the epic work from many contributors, especially
Prateek, I just merged the parquet-format change for ALP[1]. We plan to
make additional editorial The remaining items I know of are merging the
examples from parquet-testing[2], and then of course the various
implementations
Thanks again everyone. This is a great step forward
Andrew
[1]: https://github.com/apache/parquet-format/pull/557
[2]: https://github.com/apache/parquet-testing/pull/100
On Tue, Jul 21, 2026 at 4:39 PM Julien Le Dem <ju...@apache.org> wrote:
> It was also in my spam folder... :(
> I've seen a bunch of apache list emails go to my spam recently.
>
> On Tue, Jul 21, 2026 at 10:58 AM Russell Spitzer <
> russell.spitzer@gmail.com>
> wrote:
>
> > Yes, my hope was that my vote would kick it out of spam but we definitely
> > had two other votes that also went to spam for me.
> >
> > For anyone looking the thread is here
> > https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt
> >
> > On Tue, Jul 21, 2026 at 10:57 AM Micah Kornfield <em...@gmail.com>
> > wrote:
> >
> > > I've seen other's votes not go to spam, so hopefully people will see it
> > in
> > > there inboxes (or see this thread and look in there spam).
> > >
> > > On Tue, Jul 21, 2026 at 10:52 AM PRATEEK GAUR <pr...@gmail.com>
> > wrote:
> > >
> > > > Okay,
> > > >
> > > > I sent out the voting email yesterday and so far I've heard from 3
> > people
> > > > that it ended up in their "spam folder". Dev VPN to be blamed for
> it?.
> > > >
> > > > What's the protocol here?
> > > > I let people hunt for the email in their spam folder or send out
> > another
> > > > one?
> > > >
> > > > Best
> > > > Prateek
> > > >
> > > > On Sun, Jul 19, 2026 at 10:47 PM PRATEEK GAUR <pr...@gmail.com>
> > > wrote:
> > > >
> > > > > Hi team,
> > > > >
> > > > > Vinoo has added the tests which were requested in the last round of
> > > > review.
> > > > > In his own words.
> > > > >
> > > > > "
> > > > > The extreme value tests Micah wanted are done and pushed. I added
> > > > coverage
> > > > > for values that need the full FOR bit width after the frame of
> > > reference
> > > > is
> > > > > applied: 64-bit (the case where the signed max minus min
> subtraction
> > > > > overflows), 63-bit (the non-overflow case), and 32-bit for floats.
> > All
> > > > > lossless with zero exceptions. I also verified them through Arrow
> C++
> > > ALP
> > > > > reader (the local build), so the Java-written extreme columns
> decode
> > > > > correctly in C++, bit-exact against the expected values."
> > > > >
> > > > > State of the PR (#3397):
> > > > >
> > > > > - Merges cleanly with master (I pulled master in and resolved
> the
> > > > > conflicts).
> > > > > - 41 of 44 review threads resolved. The 3 remaining are
> > intentional:
> > > > > one deferred perf nit, one that stays open until parquet-format
> > > ships
> > > > ALP,
> > > > >
> > > > > "
> > > > >
> > > > > With this both the
> > > > > 1) C++ PR : https://github.com/apache/arrow/pull/48345/changes
> > > > > 2) Java PR : https://github.com/apache/parquet-java/pull/3397
> > > > >
> > > > > Are cross language tested and range of test scenarios.
> > > > >
> > > > > Best
> > > > > Prateek
> > > > >
> > > > >
> > > > > On Mon, Jul 13, 2026 at 8:38 AM PRATEEK GAUR <pr...@gmail.com>
> > > wrote:
> > > > >
> > > > >> Thanks Matt,
> > > > >>
> > > > >> I'll be on it this week.
> > > > >>
> > > > >> Best
> > > > >> Prateek
> > > > >>
> > > > >> On Mon, Jul 13, 2026 at 8:36 AM Matt Topol <
> zotthewizard@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >>> There is also the Go implementation submitted by Arnav (
> > > > >>> https://github.com/apache/arrow-go/pull/704) which is waiting
> for
> > > > >>> updates.
> > > > >>>
> > > > >>> Just wanted to make sure it didn't get lost here.
> > > > >>>
> > > > >>> --Matt
> > > > >>>
> > > > >>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK GAUR <
> prtkgaur@gmail.com>
> > > > >>> wrote:
> > > > >>>
> > > > >>> > Hi all,
> > > > >>> >
> > > > >>> > I'd like to share a status update on the ALP encoding effort
> and
> > > get
> > > > a
> > > > >>> feel
> > > > >>> > from the community before starting a formal vote.
> > > > >>> >
> > > > >>> > We now have the specification plus implementations in both
> > > languages:
> > > > >>> >
> > > > >>> > - Spec: apache/parquet-format#557 — GH-533 Add ALP
> > > > encoding
> > > > >>> > specification
> > > > >>> > - C++ (Arrow): apache/arrow#48345 — GH-48701
> [C++][Parquet]
> > > Add
> > > > >>> ALPpd
> > > > >>> > encoding
> > > > >>> > - Java: apache/parquet-java#3397 — Parquet Java ALP
> > > > >>> > Implementation (by Vinoo)
> > > > >>> >
> > > > >>> > Current state:
> > > > >>> >
> > > > >>> > - The C++ implementation has been through 3-4 rounds of
> > review.
> > > > >>> > - The Java implementation has been tracking well and has
> had
> > > > >>> initial
> > > > >>> > reviews from contributors, with no major remaining
> issues.
> > > > >>> > - Cross-language compatibility tests are passing: the Arrow
> > C++
> > > > >>> decoder
> > > > >>> > reads Java-written data bit-exactly across ~1.56M values
> > and
> > > 18
> > > > >>> > fixtures, covering V1 and V2 pages, multiple vector
> sizes,
> > > and
> > > > >>> > several
> > > > >>> > real datasets — zero mismatches.
> > > > >>> >
> > > > >>> > As raised in the java-pr we will be adding coverage for
> extreme
> > > > values
> > > > >>> > (those requiring
> > > > >>> > 63-64 bits after FOR is applied) before we close a vote. We'll
> > aim
> > > > to
> > > > >>> get
> > > > >>> > that
> > > > >>> > done in parallel.
> > > > >>> >
> > > > >>> > Plan: unless there are objections, I intend to start a vote at
> > the
> > > > end
> > > > >>> of
> > > > >>> > this week/early
> > > > >>> > next week.
> > > > >>> >
> > > > >>> > Thanks
> > > > >>> > Prateek
> > > > >>> >
> > > > >>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <
> prtkgaur@gmail.com>
> > > > >>> wrote:
> > > > >>> >
> > > > >>> > > Hi Team,
> > > > >>> > >
> > > > >>> > > Just wanted to provide some updates on ALP.
> > > > >>> > > Micah and I have done 3-4 rounds of review for the c++ PR :
> > > > >>> > > https://github.com/apache/arrow/pull/48345
> > > > >>> > >
> > > > >>> > > For the Java implementation Vinoo has been working on
> following
> > > PR
> > > > :
> > > > >>> > > https://github.com/apache/parquet-java/pull/3397
> > > > >>> > >
> > > > >>> > > Best
> > > > >>> > > Prateek
> > > > >>> > >
> > > > >>> > > On Tue, May 5, 2026 at 1:50 PM Micah Kornfield <
> > > > >>> emkornfield@gmail.com>
> > > > >>> > > wrote:
> > > > >>> > >
> > > > >>> > >> Hi Antoine,
> > > > >>> > >>
> > > > >>> > >> > Apologies if the question was already asked, but should we
> > > care
> > > > >>> about
> > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than
> > > FLOAT16
> > > > +
> > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> > > > >>> > >>
> > > > >>> > >> It was. We thought we could defer it for the following
> > reasons:
> > > > >>> > >> 1. It's not clear there are a lot of easy reference
> datasets
> > to
> > > > >>> test
> > > > >>> > its
> > > > >>> > >> effectiveness. It does look like there might be one or two
> on
> > > > >>> > huggingface
> > > > >>> > >> (e.g. https://huggingface.co/datasets/kikitora/curdie).
> > > > >>> > >> 2. It seemed likely that float 16 was more likely used for
> > > values
> > > > >>> that
> > > > >>> > >> were less likely to reduce to decimal values.
> > > > >>> > >> 3. It could be added as an extension later if needed.
> > > > >>> > >>
> > > > >>> > >> Cheers,
> > > > >>> > >> Micah
> > > > >>> > >>
> > > > >>> > >>
> > > > >>> > >> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <
> > > antoine@python.org
> > > > >
> > > > >>> > wrote:
> > > > >>> > >>
> > > > >>> > >> >
> > > > >>> > >> > Hello,
> > > > >>> > >> >
> > > > >>> > >> > Apologies if the question was already asked, but should we
> > > care
> > > > >>> about
> > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than
> > > FLOAT16
> > > > +
> > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> > > > >>> > >> >
> > > > >>> > >> > Regards
> > > > >>> > >> >
> > > > >>> > >> > Antoine.
> > > > >>> > >> >
> > > > >>> > >> >
> > > > >>> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
> > > > >>> > >> > > Thanks Andrew and Micah for review feedback on the two
> > PR's
> > > > >>> > >> > > 1) (c++ arrow repo)
> > > > >>> > >> https://github.com/apache/arrow/pull/48345/changes
> > > > >>> > >> > > 2) (parquet-format repo)
> > > > >>> > >> > https://github.com/apache/parquet-format/pull/557
> > > > >>> > >> > >
> > > > >>> > >> > > I have addressed all (unless I missed something)
> comments
> > on
> > > > >>> the two
> > > > >>> > >> > PR's.
> > > > >>> > >> > >
> > > > >>> > >> > > Best
> > > > >>> > >> > > Prateek
> > > > >>> > >> > >
> > > > >>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <
> > > > >>> prtkgaur@gmail.com>
> > > > >>> > >> wrote:
> > > > >>> > >> > >
> > > > >>> > >> > >> Thanks Andrew and Micah.
> > > > >>> > >> > >>
> > > > >>> > >> > >> `fair amount of feedback on at least the
> implementations`
> > > > >>> > >> > >> For the c++ I have already started addressing the
> > > feedback, I
> > > > >>> > should
> > > > >>> > >> be
> > > > >>> > >> > >> done with that Monday/Tuesday.
> > > > >>> > >> > >> I think Vinoo too has been making good progress on the
> > Java
> > > > >>> > >> > implementation.
> > > > >>> > >> > >>
> > > > >>> > >> > >> Best
> > > > >>> > >> > >> Prateek
> > > > >>> > >> > >>
> > > > >>> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <
> > > > >>> > andrewlamb11@gmail.com
> > > > >>> > >> >
> > > > >>> > >> > >> wrote:
> > > > >>> > >> > >>
> > > > >>> > >> > >>> Got it. Thank you for the clarification -- I will try
> > and
> > > > look
> > > > >>> > into
> > > > >>> > >> the
> > > > >>> > >> > >>> spec and the Rust implementation[1] in this next week
> > > > >>> > >> > >>>
> > > > >>> > >> > >>> [1]: https://github.com/apache/arrow-rs/pull/9372
> > > > >>> > >> > >>>
> > > > >>> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <
> > > > >>> > >> > emkornfield@gmail.com>
> > > > >>> > >> > >>> wrote:
> > > > >>> > >> > >>>
> > > > >>> > >> > >>>> Hi Andrew,
> > > > >>> > >> > >>>> I think there is a fair amount of feedback on at
> least
> > > the
> > > > >>> > >> > >>>> implementations, typically I think we've waited till
> > they
> > > > are
> > > > >>> > >> close to
> > > > >>> > >> > >>>> mergeable before a final vote. Otherwise I agree we
> > are
> > > > very
> > > > >>> > >> close.
> > > > >>> > >> > >>>>
> > > > >>> > >> > >>>> -Micah
> > > > >>> > >> > >>>>
> > > > >>> > >> > >>>> On Saturday, April 25, 2026, Andrew Lamb <
> > > > >>> andrewlamb11@gmail.com
> > > > >>> > >
> > > > >>> > >> > wrote:
> > > > >>> > >> > >>>>
> > > > >>> > >> > >>>>> Thanks Prateek,
> > > > >>> > >> > >>>>>
> > > > >>> > >> > >>>>> I think from this content it looks to me like we are
> > > ready
> > > > >>> to
> > > > >>> > >> start a
> > > > >>> > >> > >>>>> vote to explicitly accept ALP into Parquet
> > > > >>> > >> > >>>>>
> > > > >>> > >> > >>>>> Does anyone know of a reason we should postpone it
> for
> > > > >>> longer?
> > > > >>> > >> > >>>>> Perhaps someone needs some more time to review?
> > > > >>> > >> > >>>>>
> > > > >>> > >> > >>>>> Andrew
> > > > >>> > >> > >>>>>
> > > > >>> > >> > >>>>>
> > > > >>> > >> > >>>>>
> > > > >>> > >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <
> > > > >>> > prtkgaur@gmail.com>
> > > > >>> > >> > >>>>> wrote:
> > > > >>> > >> > >>>>>
> > > > >>> > >> > >>>>>> Hi team,
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>> Hope everyone is doing well. I got a chance to work
> > > > >>> through all
> > > > >>> > >> the
> > > > >>> > >> > >>>>>> remaining feedback and update the spec doc. Here
> are
> > > the
> > > > >>> new
> > > > >>> > >> > artifacts
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>> 1) Spec document :
> > > > >>> > >> > >>>>>>
> > > > >>> > >> >
> > > > >>> > >>
> > > > >>> >
> > > > >>>
> > > >
> > >
> >
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>> 2) Spec document in parquet format repo :
> > > > >>> > >> > >>>>>> https://github.com/apache/parquet-format/pull/557
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>> 3) Alp implementation in arrow c++ repo :
> > > > >>> > >> > >>>>>> https://github.com/apache/arrow/pull/48345/changes
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>> 4) Alp implementation in parquet-java repo : Work
> for
> > > > >>> Vinoo and
> > > > >>> > >> > Julien
> > > > >>> > >> > >>>>>> https://github.com/apache/parquet-java/pull/3397
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>> 5) PR with test and benchmarking artifacts in
> > > > >>> parquet-testing
> > > > >>> > >> repo :
> > > > >>> > >> > >>>>>> https://github.com/apache/parquet-testing/pull/100
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>> And
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>> - Go : Arnav just submitted an in progress
> > > > >>> implementation
> > > > >>> > in
> > > > >>> > >> Go.
> > > > >>> > >> > >>>>>> https://github.com/apache/arrow-go/pull/704 (I
> > > > haven't
> > > > >>> > >> started
> > > > >>> > >> > >>>>>> looking at it yet)
> > > > >>> > >> > >>>>>> - Rust : I remember Andrew mentioned that this
> > work
> > > > is
> > > > >>> also
> > > > >>> > >> in
> > > > >>> > >> > >>>>>> progress (So 4 languages!)
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>> *Arrow C++ implementation *
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>> The PR is out and was also used by Antoine to
> report
> > > the
> > > > >>> > numbers
> > > > >>> > >> as
> > > > >>> > >> > >>>>>> reported here. Micah and Konstantin have given 1
> > round
> > > of
> > > > >>> > >> feedback
> > > > >>> > >> > >>>>>> and I'm addressing them today. Please note that the
> > > > default
> > > > >>> > >> > >>>>>> optimization flag for compiling is O2 and not Q3. I
> > got
> > > > >>> around
> > > > >>> > >> 70%
> > > > >>> > >> > >>>>>> performance improvement in the decoding speed when
> > > using
> > > > >>> the O3
> > > > >>> > >> > flag.
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>> *Parqet-MR Java implementation (working with Vinoo
> > and
> > > > >>> Julien)
> > > > >>> > >> and
> > > > >>> > >> > **Cross
> > > > >>> > >> > >>>>>> Language testing*
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>> Let me know if you have any questions or
> > feedback.
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>> Now pasting some performance numbers
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>> Table 1: C++ ALP Double Decode — Spotify Columns
> > > > >>> (Graviton
> > > > >>> > 3,
> > > > >>> > >> ARM
> > > > >>> > >> > >>>>>> Neoverse V1)
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>>
> > > > >>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>> │ Column │ -O2 (MB/s) │ -O3
> (MB/s) │
> > > > >>> Speedup │
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>>
> > > > >>> ├──────────────────┼──────────────┼──────────────┼─────────┤
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>> │ valence │ 3,155 │ 5,523
> │
> > > > >>> 1.75x │
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>> │ danceability │ 3,233 │ 5,685
> │
> > > > >>> 1.76x │
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>> │ energy │ 3,197 │ 5,652
> │
> > > > >>> 1.77x │
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>> │ loudness │ 3,186 │ 5,473
> │
> > > > >>> 1.72x │
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>>
> > > > >>> └──────────────────┴──────────────┴──────────────┴─────────┘
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <
> > > > >>> > prtkgaur@gmail.com
> > > > >>> > >> >
> > > > >>> > >> > >>>>>> wrote:
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>>> @Micah Kornfield <em...@gmail.com> : Got
> it.
> > > > >>> > >> > >>>>>>>
> > > > >>> > >> > >>>>>>> @Andrew Lamb <an...@gmail.com>
> > > > >>> > >> > >>>>>>>
> > > > >>> > >> > >>>>>>>
> > > > >>> > >> > >>>>>>>> Do you think it would be good to start moving the
> > > spec
> > > > >>> > >> development
> > > > >>> > >> > >>>>>>>> into
> > > > >>> > >> > >>>>>>>> markdown format, in preparation for finalizing
> it?
> > > > >>> > >> > >>>>>>>>
> > > > >>> > >> > >>>>>>>
> > > > >>> > >> > >>>>>>> Yes I'll update the numbers for some of the
> > examples I
> > > > >>> have in
> > > > >>> > >> the
> > > > >>> > >> > >>>>>>> spec based
> > > > >>> > >> > >>>>>>> on the updated header size. Then we should be good
> > to
> > > go
> > > > >>> for
> > > > >>> > the
> > > > >>> > >> > >>>>>>> markdown format.
> > > > >>> > >> > >>>>>>>
> > > > >>> > >> > >>>>>>> Thanks everyone!
> > > > >>> > >> > >>>>>>>
> > > > >>> > >> > >>>>>>>
> > > > >>> > >> > >>>>>>>>
> > > > >>> > >> > >>>>>>>> Andrew
> > > > >>> > >> > >>>>>>>>
> > > > >>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <
> > > > >>> > >> prtkgaur@gmail.com>
> > > > >>> > >> > >>>>>>>> wrote:
> > > > >>> > >> > >>>>>>>>
> > > > >>> > >> > >>>>>>>>> Hi team,
> > > > >>> > >> > >>>>>>>>>
> > > > >>> > >> > >>>>>>>>> 1) Andrew
> > > > >>> > >> > >>>>>>>>>
> > > > >>> > >> > >>>>>>>>> - Thanks for working on test files. My PR
> did
> > > add
> > > > >>> all
> > > > >>> > the
> > > > >>> > >> > test
> > > > >>> > >> > >>>>>>>> files I
> > > > >>> > >> > >>>>>>>>> used to benchmark on datasets. Maybe we can
> > club
> > > > it
> > > > >>> > >> together.
> > > > >>> > >> > >>>>>>>> WIll also
> > > > >>> > >> > >>>>>>>>> aid
> > > > >>> > >> > >>>>>>>>> cross language testing
> > > > >>> > >> > >>>>>>>>> - Kosta Tarasov working on Rust
> > implementation.
> > > > >>> This is
> > > > >>> > >> > great.
> > > > >>> > >> > >>>>>>>> Thanks
> > > > >>> > >> > >>>>>>>>>
> > > > >>> > >> > >>>>>>>>>
> > > > >>> > >> > >>>>>>>>> 2) Antoine
> > > > >>> > >> > >>>>>>>>>
> > > > >>> > >> > >>>>>>>>> - Thanks a lot for reporting the numbers on
> > AMD.
> > > > >>> Looks
> > > > >>> > >> like
> > > > >>> > >> > you
> > > > >>> > >> > >>>>>>>> are
> > > > >>> > >> > >>>>>>>>> getting 8X the decoding performance of BSS.
> > This
> > > > is
> > > > >>> > >> > amazing!!.
> > > > >>> > >> > >>>>>>>>> - Thanks for acknowledging the sampling
> > design.
> > > > >>> > >> > >>>>>>>>> - I agree with you on Fastlanes. In some
> crude
> > > > >>> > >> experiments I
> > > > >>> > >> > >>>>>>>> didn't get
> > > > >>> > >> > >>>>>>>>> a good perf benefit from it on Graviton3
> (but
> > > > maybe
> > > > >>> > there
> > > > >>> > >> was
> > > > >>> > >> > >>>>>>>> something
> > > > >>> > >> > >>>>>>>>> wrong with my implementation).
> > > > >>> > >> > >>>>>>>>> - Locking the 16bit exception encoding for
> the
> > > > spec
> > > > >>> in
> > > > >>> > >> this
> > > > >>> > >> > >>>>>>>> case.
> > > > >>> > >> > >>>>>>>>> - Awesome I think we have solved for all
> open
> > > > >>> questions
> > > > >>> > >> minus
> > > > >>> > >> > >>>>>>>> the
> > > > >>> > >> > >>>>>>>>> version byte :). (will get back on this
> soon)
> > > > >>> > >> > >>>>>>>>>
> > > > >>> > >> > >>>>>>>>>
> > > > >>> > >> > >>>>>>>>> 3) Micah
> > > > >>> > >> > >>>>>>>>>
> > > > >>> > >> > >>>>>>>>> - FastLanes : The current spec does allow
> for
> > > > using
> > > > >>> > >> FastLane
> > > > >>> > >> > >>>>>>>> with the
> > > > >>> > >> > >>>>>>>>> configurable enum value for layout. We
> should
> > be
> > > > >>> able to
> > > > >>> > >> > inject
> > > > >>> > >> > >>>>>>>> any
> > > > >>> > >> > >>>>>>>>> layout
> > > > >>> > >> > >>>>>>>>> in the current design.
> > > > >>> > >> > >>>>>>>>>
> > > > >>> > >> > >>>>>>>>>
> > > > >>> > >> > >>>>>>>>> Working on resolving all remaining open comments
> > on
> > > > the
> > > > >>> spec
> > > > >>> > >> this
> > > > >>> > >> > >>>>>>>> week.
> > > > >>> > >> > >>>>>>>>>
> > > > >>> > >> > >>>>>>>>> Best
> > > > >>> > >> > >>>>>>>>> Prateek
> > > > >>> > >> > >>>>>>>>>
> > > > >>> > >> > >>>>>>>>>
> > > > >>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <
> > > > >>> > >> > >>>>>>>> stevel@cloudera.com>
> > > > >>> > >> > >>>>>>>>> wrote:
> > > > >>> > >> > >>>>>>>>>
> > > > >>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
> > > > >>> > >> > >>>>>>>> emkornfield@gmail.com>
> > > > >>> > >> > >>>>>>>>>> wrote:
> > > > >>> > >> > >>>>>>>>>>
> > > > >>> > >> > >>>>>>>>>>>
> > > > >>> > >> > >>>>>>>>>>>
> > > > >>> > >> > >>>>>>>>>>> It looks like the actual issue described for
> ORC
> > > in
> > > > >>> the
> > > > >>> > >> paper
> > > > >>> > >> > >>>>>>>> is that
> > > > >>> > >> > >>>>>>>>> it
> > > > >>> > >> > >>>>>>>>>>> has multiple sub-encodings in a batch. This
> is
> > > > >>> different
> > > > >>> > >> then
> > > > >>> > >> > >>>>>>>> the
> > > > >>> > >> > >>>>>>>>> design
> > > > >>> > >> > >>>>>>>>>>> proposed here where there is still fixed
> > encoding
> > > > per
> > > > >>> page
> > > > >>> > >> in
> > > > >>> > >> > >>>>>>>> parquet.
> > > > >>> > >> > >>>>>>>>>>> Given reasonably sized pages I don't think
> > branch
> > > > >>> > >> > >>>>>>>> misprediction should
> > > > >>> > >> > >>>>>>>>>> be a
> > > > >>> > >> > >>>>>>>>>>> big issue for new encodings. I agree that we
> > > should
> > > > >>> be
> > > > >>> > >> > >>>>>>>> conservative in
> > > > >>> > >> > >>>>>>>>>>> general for adding new encodings.
> > > > >>> > >> > >>>>>>>>>>>
> > > > >>> > >> > >>>>>>>>>>>
> > > > >>> > >> > >>>>>>>>>> +1
> > > > >>> > >> > >>>>>>>>>>
> > > > >>> > >> > >>>>>>>>>
> > > > >>> > >> > >>>>>>>>
> > > > >>> > >> > >>>>>>>
> > > > >>> > >> > >
> > > > >>> > >> >
> > > > >>> > >> >
> > > > >>> > >> >
> > > > >>> > >>
> > > > >>> > >
> > > > >>> >
> > > > >>>
> > > > >>
> > > >
> > >
> >
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Julien Le Dem <ju...@apache.org>.
It was also in my spam folder... :(
I've seen a bunch of apache list emails go to my spam recently.
On Tue, Jul 21, 2026 at 10:58 AM Russell Spitzer <ru...@gmail.com>
wrote:
> Yes, my hope was that my vote would kick it out of spam but we definitely
> had two other votes that also went to spam for me.
>
> For anyone looking the thread is here
> https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt
>
> On Tue, Jul 21, 2026 at 10:57 AM Micah Kornfield <em...@gmail.com>
> wrote:
>
> > I've seen other's votes not go to spam, so hopefully people will see it
> in
> > there inboxes (or see this thread and look in there spam).
> >
> > On Tue, Jul 21, 2026 at 10:52 AM PRATEEK GAUR <pr...@gmail.com>
> wrote:
> >
> > > Okay,
> > >
> > > I sent out the voting email yesterday and so far I've heard from 3
> people
> > > that it ended up in their "spam folder". Dev VPN to be blamed for it?.
> > >
> > > What's the protocol here?
> > > I let people hunt for the email in their spam folder or send out
> another
> > > one?
> > >
> > > Best
> > > Prateek
> > >
> > > On Sun, Jul 19, 2026 at 10:47 PM PRATEEK GAUR <pr...@gmail.com>
> > wrote:
> > >
> > > > Hi team,
> > > >
> > > > Vinoo has added the tests which were requested in the last round of
> > > review.
> > > > In his own words.
> > > >
> > > > "
> > > > The extreme value tests Micah wanted are done and pushed. I added
> > > coverage
> > > > for values that need the full FOR bit width after the frame of
> > reference
> > > is
> > > > applied: 64-bit (the case where the signed max minus min subtraction
> > > > overflows), 63-bit (the non-overflow case), and 32-bit for floats.
> All
> > > > lossless with zero exceptions. I also verified them through Arrow C++
> > ALP
> > > > reader (the local build), so the Java-written extreme columns decode
> > > > correctly in C++, bit-exact against the expected values."
> > > >
> > > > State of the PR (#3397):
> > > >
> > > > - Merges cleanly with master (I pulled master in and resolved the
> > > > conflicts).
> > > > - 41 of 44 review threads resolved. The 3 remaining are
> intentional:
> > > > one deferred perf nit, one that stays open until parquet-format
> > ships
> > > ALP,
> > > >
> > > > "
> > > >
> > > > With this both the
> > > > 1) C++ PR : https://github.com/apache/arrow/pull/48345/changes
> > > > 2) Java PR : https://github.com/apache/parquet-java/pull/3397
> > > >
> > > > Are cross language tested and range of test scenarios.
> > > >
> > > > Best
> > > > Prateek
> > > >
> > > >
> > > > On Mon, Jul 13, 2026 at 8:38 AM PRATEEK GAUR <pr...@gmail.com>
> > wrote:
> > > >
> > > >> Thanks Matt,
> > > >>
> > > >> I'll be on it this week.
> > > >>
> > > >> Best
> > > >> Prateek
> > > >>
> > > >> On Mon, Jul 13, 2026 at 8:36 AM Matt Topol <zo...@gmail.com>
> > > >> wrote:
> > > >>
> > > >>> There is also the Go implementation submitted by Arnav (
> > > >>> https://github.com/apache/arrow-go/pull/704) which is waiting for
> > > >>> updates.
> > > >>>
> > > >>> Just wanted to make sure it didn't get lost here.
> > > >>>
> > > >>> --Matt
> > > >>>
> > > >>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK GAUR <pr...@gmail.com>
> > > >>> wrote:
> > > >>>
> > > >>> > Hi all,
> > > >>> >
> > > >>> > I'd like to share a status update on the ALP encoding effort and
> > get
> > > a
> > > >>> feel
> > > >>> > from the community before starting a formal vote.
> > > >>> >
> > > >>> > We now have the specification plus implementations in both
> > languages:
> > > >>> >
> > > >>> > - Spec: apache/parquet-format#557 — GH-533 Add ALP
> > > encoding
> > > >>> > specification
> > > >>> > - C++ (Arrow): apache/arrow#48345 — GH-48701 [C++][Parquet]
> > Add
> > > >>> ALPpd
> > > >>> > encoding
> > > >>> > - Java: apache/parquet-java#3397 — Parquet Java ALP
> > > >>> > Implementation (by Vinoo)
> > > >>> >
> > > >>> > Current state:
> > > >>> >
> > > >>> > - The C++ implementation has been through 3-4 rounds of
> review.
> > > >>> > - The Java implementation has been tracking well and has had
> > > >>> initial
> > > >>> > reviews from contributors, with no major remaining issues.
> > > >>> > - Cross-language compatibility tests are passing: the Arrow
> C++
> > > >>> decoder
> > > >>> > reads Java-written data bit-exactly across ~1.56M values
> and
> > 18
> > > >>> > fixtures, covering V1 and V2 pages, multiple vector sizes,
> > and
> > > >>> > several
> > > >>> > real datasets — zero mismatches.
> > > >>> >
> > > >>> > As raised in the java-pr we will be adding coverage for extreme
> > > values
> > > >>> > (those requiring
> > > >>> > 63-64 bits after FOR is applied) before we close a vote. We'll
> aim
> > > to
> > > >>> get
> > > >>> > that
> > > >>> > done in parallel.
> > > >>> >
> > > >>> > Plan: unless there are objections, I intend to start a vote at
> the
> > > end
> > > >>> of
> > > >>> > this week/early
> > > >>> > next week.
> > > >>> >
> > > >>> > Thanks
> > > >>> > Prateek
> > > >>> >
> > > >>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <pr...@gmail.com>
> > > >>> wrote:
> > > >>> >
> > > >>> > > Hi Team,
> > > >>> > >
> > > >>> > > Just wanted to provide some updates on ALP.
> > > >>> > > Micah and I have done 3-4 rounds of review for the c++ PR :
> > > >>> > > https://github.com/apache/arrow/pull/48345
> > > >>> > >
> > > >>> > > For the Java implementation Vinoo has been working on following
> > PR
> > > :
> > > >>> > > https://github.com/apache/parquet-java/pull/3397
> > > >>> > >
> > > >>> > > Best
> > > >>> > > Prateek
> > > >>> > >
> > > >>> > > On Tue, May 5, 2026 at 1:50 PM Micah Kornfield <
> > > >>> emkornfield@gmail.com>
> > > >>> > > wrote:
> > > >>> > >
> > > >>> > >> Hi Antoine,
> > > >>> > >>
> > > >>> > >> > Apologies if the question was already asked, but should we
> > care
> > > >>> about
> > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than
> > FLOAT16
> > > +
> > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> > > >>> > >>
> > > >>> > >> It was. We thought we could defer it for the following
> reasons:
> > > >>> > >> 1. It's not clear there are a lot of easy reference datasets
> to
> > > >>> test
> > > >>> > its
> > > >>> > >> effectiveness. It does look like there might be one or two on
> > > >>> > huggingface
> > > >>> > >> (e.g. https://huggingface.co/datasets/kikitora/curdie).
> > > >>> > >> 2. It seemed likely that float 16 was more likely used for
> > values
> > > >>> that
> > > >>> > >> were less likely to reduce to decimal values.
> > > >>> > >> 3. It could be added as an extension later if needed.
> > > >>> > >>
> > > >>> > >> Cheers,
> > > >>> > >> Micah
> > > >>> > >>
> > > >>> > >>
> > > >>> > >> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <
> > antoine@python.org
> > > >
> > > >>> > wrote:
> > > >>> > >>
> > > >>> > >> >
> > > >>> > >> > Hello,
> > > >>> > >> >
> > > >>> > >> > Apologies if the question was already asked, but should we
> > care
> > > >>> about
> > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than
> > FLOAT16
> > > +
> > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> > > >>> > >> >
> > > >>> > >> > Regards
> > > >>> > >> >
> > > >>> > >> > Antoine.
> > > >>> > >> >
> > > >>> > >> >
> > > >>> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
> > > >>> > >> > > Thanks Andrew and Micah for review feedback on the two
> PR's
> > > >>> > >> > > 1) (c++ arrow repo)
> > > >>> > >> https://github.com/apache/arrow/pull/48345/changes
> > > >>> > >> > > 2) (parquet-format repo)
> > > >>> > >> > https://github.com/apache/parquet-format/pull/557
> > > >>> > >> > >
> > > >>> > >> > > I have addressed all (unless I missed something) comments
> on
> > > >>> the two
> > > >>> > >> > PR's.
> > > >>> > >> > >
> > > >>> > >> > > Best
> > > >>> > >> > > Prateek
> > > >>> > >> > >
> > > >>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <
> > > >>> prtkgaur@gmail.com>
> > > >>> > >> wrote:
> > > >>> > >> > >
> > > >>> > >> > >> Thanks Andrew and Micah.
> > > >>> > >> > >>
> > > >>> > >> > >> `fair amount of feedback on at least the implementations`
> > > >>> > >> > >> For the c++ I have already started addressing the
> > feedback, I
> > > >>> > should
> > > >>> > >> be
> > > >>> > >> > >> done with that Monday/Tuesday.
> > > >>> > >> > >> I think Vinoo too has been making good progress on the
> Java
> > > >>> > >> > implementation.
> > > >>> > >> > >>
> > > >>> > >> > >> Best
> > > >>> > >> > >> Prateek
> > > >>> > >> > >>
> > > >>> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <
> > > >>> > andrewlamb11@gmail.com
> > > >>> > >> >
> > > >>> > >> > >> wrote:
> > > >>> > >> > >>
> > > >>> > >> > >>> Got it. Thank you for the clarification -- I will try
> and
> > > look
> > > >>> > into
> > > >>> > >> the
> > > >>> > >> > >>> spec and the Rust implementation[1] in this next week
> > > >>> > >> > >>>
> > > >>> > >> > >>> [1]: https://github.com/apache/arrow-rs/pull/9372
> > > >>> > >> > >>>
> > > >>> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <
> > > >>> > >> > emkornfield@gmail.com>
> > > >>> > >> > >>> wrote:
> > > >>> > >> > >>>
> > > >>> > >> > >>>> Hi Andrew,
> > > >>> > >> > >>>> I think there is a fair amount of feedback on at least
> > the
> > > >>> > >> > >>>> implementations, typically I think we've waited till
> they
> > > are
> > > >>> > >> close to
> > > >>> > >> > >>>> mergeable before a final vote. Otherwise I agree we
> are
> > > very
> > > >>> > >> close.
> > > >>> > >> > >>>>
> > > >>> > >> > >>>> -Micah
> > > >>> > >> > >>>>
> > > >>> > >> > >>>> On Saturday, April 25, 2026, Andrew Lamb <
> > > >>> andrewlamb11@gmail.com
> > > >>> > >
> > > >>> > >> > wrote:
> > > >>> > >> > >>>>
> > > >>> > >> > >>>>> Thanks Prateek,
> > > >>> > >> > >>>>>
> > > >>> > >> > >>>>> I think from this content it looks to me like we are
> > ready
> > > >>> to
> > > >>> > >> start a
> > > >>> > >> > >>>>> vote to explicitly accept ALP into Parquet
> > > >>> > >> > >>>>>
> > > >>> > >> > >>>>> Does anyone know of a reason we should postpone it for
> > > >>> longer?
> > > >>> > >> > >>>>> Perhaps someone needs some more time to review?
> > > >>> > >> > >>>>>
> > > >>> > >> > >>>>> Andrew
> > > >>> > >> > >>>>>
> > > >>> > >> > >>>>>
> > > >>> > >> > >>>>>
> > > >>> > >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <
> > > >>> > prtkgaur@gmail.com>
> > > >>> > >> > >>>>> wrote:
> > > >>> > >> > >>>>>
> > > >>> > >> > >>>>>> Hi team,
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>> Hope everyone is doing well. I got a chance to work
> > > >>> through all
> > > >>> > >> the
> > > >>> > >> > >>>>>> remaining feedback and update the spec doc. Here are
> > the
> > > >>> new
> > > >>> > >> > artifacts
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>> 1) Spec document :
> > > >>> > >> > >>>>>>
> > > >>> > >> >
> > > >>> > >>
> > > >>> >
> > > >>>
> > >
> >
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>> 2) Spec document in parquet format repo :
> > > >>> > >> > >>>>>> https://github.com/apache/parquet-format/pull/557
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>> 3) Alp implementation in arrow c++ repo :
> > > >>> > >> > >>>>>> https://github.com/apache/arrow/pull/48345/changes
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>> 4) Alp implementation in parquet-java repo : Work for
> > > >>> Vinoo and
> > > >>> > >> > Julien
> > > >>> > >> > >>>>>> https://github.com/apache/parquet-java/pull/3397
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>> 5) PR with test and benchmarking artifacts in
> > > >>> parquet-testing
> > > >>> > >> repo :
> > > >>> > >> > >>>>>> https://github.com/apache/parquet-testing/pull/100
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>> And
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>> - Go : Arnav just submitted an in progress
> > > >>> implementation
> > > >>> > in
> > > >>> > >> Go.
> > > >>> > >> > >>>>>> https://github.com/apache/arrow-go/pull/704 (I
> > > haven't
> > > >>> > >> started
> > > >>> > >> > >>>>>> looking at it yet)
> > > >>> > >> > >>>>>> - Rust : I remember Andrew mentioned that this
> work
> > > is
> > > >>> also
> > > >>> > >> in
> > > >>> > >> > >>>>>> progress (So 4 languages!)
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>> *Arrow C++ implementation *
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>> The PR is out and was also used by Antoine to report
> > the
> > > >>> > numbers
> > > >>> > >> as
> > > >>> > >> > >>>>>> reported here. Micah and Konstantin have given 1
> round
> > of
> > > >>> > >> feedback
> > > >>> > >> > >>>>>> and I'm addressing them today. Please note that the
> > > default
> > > >>> > >> > >>>>>> optimization flag for compiling is O2 and not Q3. I
> got
> > > >>> around
> > > >>> > >> 70%
> > > >>> > >> > >>>>>> performance improvement in the decoding speed when
> > using
> > > >>> the O3
> > > >>> > >> > flag.
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>> *Parqet-MR Java implementation (working with Vinoo
> and
> > > >>> Julien)
> > > >>> > >> and
> > > >>> > >> > **Cross
> > > >>> > >> > >>>>>> Language testing*
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>> Let me know if you have any questions or
> feedback.
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>> Now pasting some performance numbers
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>> Table 1: C++ ALP Double Decode — Spotify Columns
> > > >>> (Graviton
> > > >>> > 3,
> > > >>> > >> ARM
> > > >>> > >> > >>>>>> Neoverse V1)
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>>
> > > >>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │
> > > >>> Speedup │
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>>
> > > >>> ├──────────────────┼──────────────┼──────────────┼─────────┤
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>> │ valence │ 3,155 │ 5,523 │
> > > >>> 1.75x │
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>> │ danceability │ 3,233 │ 5,685 │
> > > >>> 1.76x │
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>> │ energy │ 3,197 │ 5,652 │
> > > >>> 1.77x │
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>> │ loudness │ 3,186 │ 5,473 │
> > > >>> 1.72x │
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>>
> > > >>> └──────────────────┴──────────────┴──────────────┴─────────┘
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <
> > > >>> > prtkgaur@gmail.com
> > > >>> > >> >
> > > >>> > >> > >>>>>> wrote:
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>>> @Micah Kornfield <em...@gmail.com> : Got it.
> > > >>> > >> > >>>>>>>
> > > >>> > >> > >>>>>>> @Andrew Lamb <an...@gmail.com>
> > > >>> > >> > >>>>>>>
> > > >>> > >> > >>>>>>>
> > > >>> > >> > >>>>>>>> Do you think it would be good to start moving the
> > spec
> > > >>> > >> development
> > > >>> > >> > >>>>>>>> into
> > > >>> > >> > >>>>>>>> markdown format, in preparation for finalizing it?
> > > >>> > >> > >>>>>>>>
> > > >>> > >> > >>>>>>>
> > > >>> > >> > >>>>>>> Yes I'll update the numbers for some of the
> examples I
> > > >>> have in
> > > >>> > >> the
> > > >>> > >> > >>>>>>> spec based
> > > >>> > >> > >>>>>>> on the updated header size. Then we should be good
> to
> > go
> > > >>> for
> > > >>> > the
> > > >>> > >> > >>>>>>> markdown format.
> > > >>> > >> > >>>>>>>
> > > >>> > >> > >>>>>>> Thanks everyone!
> > > >>> > >> > >>>>>>>
> > > >>> > >> > >>>>>>>
> > > >>> > >> > >>>>>>>>
> > > >>> > >> > >>>>>>>> Andrew
> > > >>> > >> > >>>>>>>>
> > > >>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <
> > > >>> > >> prtkgaur@gmail.com>
> > > >>> > >> > >>>>>>>> wrote:
> > > >>> > >> > >>>>>>>>
> > > >>> > >> > >>>>>>>>> Hi team,
> > > >>> > >> > >>>>>>>>>
> > > >>> > >> > >>>>>>>>> 1) Andrew
> > > >>> > >> > >>>>>>>>>
> > > >>> > >> > >>>>>>>>> - Thanks for working on test files. My PR did
> > add
> > > >>> all
> > > >>> > the
> > > >>> > >> > test
> > > >>> > >> > >>>>>>>> files I
> > > >>> > >> > >>>>>>>>> used to benchmark on datasets. Maybe we can
> club
> > > it
> > > >>> > >> together.
> > > >>> > >> > >>>>>>>> WIll also
> > > >>> > >> > >>>>>>>>> aid
> > > >>> > >> > >>>>>>>>> cross language testing
> > > >>> > >> > >>>>>>>>> - Kosta Tarasov working on Rust
> implementation.
> > > >>> This is
> > > >>> > >> > great.
> > > >>> > >> > >>>>>>>> Thanks
> > > >>> > >> > >>>>>>>>>
> > > >>> > >> > >>>>>>>>>
> > > >>> > >> > >>>>>>>>> 2) Antoine
> > > >>> > >> > >>>>>>>>>
> > > >>> > >> > >>>>>>>>> - Thanks a lot for reporting the numbers on
> AMD.
> > > >>> Looks
> > > >>> > >> like
> > > >>> > >> > you
> > > >>> > >> > >>>>>>>> are
> > > >>> > >> > >>>>>>>>> getting 8X the decoding performance of BSS.
> This
> > > is
> > > >>> > >> > amazing!!.
> > > >>> > >> > >>>>>>>>> - Thanks for acknowledging the sampling
> design.
> > > >>> > >> > >>>>>>>>> - I agree with you on Fastlanes. In some crude
> > > >>> > >> experiments I
> > > >>> > >> > >>>>>>>> didn't get
> > > >>> > >> > >>>>>>>>> a good perf benefit from it on Graviton3 (but
> > > maybe
> > > >>> > there
> > > >>> > >> was
> > > >>> > >> > >>>>>>>> something
> > > >>> > >> > >>>>>>>>> wrong with my implementation).
> > > >>> > >> > >>>>>>>>> - Locking the 16bit exception encoding for the
> > > spec
> > > >>> in
> > > >>> > >> this
> > > >>> > >> > >>>>>>>> case.
> > > >>> > >> > >>>>>>>>> - Awesome I think we have solved for all open
> > > >>> questions
> > > >>> > >> minus
> > > >>> > >> > >>>>>>>> the
> > > >>> > >> > >>>>>>>>> version byte :). (will get back on this soon)
> > > >>> > >> > >>>>>>>>>
> > > >>> > >> > >>>>>>>>>
> > > >>> > >> > >>>>>>>>> 3) Micah
> > > >>> > >> > >>>>>>>>>
> > > >>> > >> > >>>>>>>>> - FastLanes : The current spec does allow for
> > > using
> > > >>> > >> FastLane
> > > >>> > >> > >>>>>>>> with the
> > > >>> > >> > >>>>>>>>> configurable enum value for layout. We should
> be
> > > >>> able to
> > > >>> > >> > inject
> > > >>> > >> > >>>>>>>> any
> > > >>> > >> > >>>>>>>>> layout
> > > >>> > >> > >>>>>>>>> in the current design.
> > > >>> > >> > >>>>>>>>>
> > > >>> > >> > >>>>>>>>>
> > > >>> > >> > >>>>>>>>> Working on resolving all remaining open comments
> on
> > > the
> > > >>> spec
> > > >>> > >> this
> > > >>> > >> > >>>>>>>> week.
> > > >>> > >> > >>>>>>>>>
> > > >>> > >> > >>>>>>>>> Best
> > > >>> > >> > >>>>>>>>> Prateek
> > > >>> > >> > >>>>>>>>>
> > > >>> > >> > >>>>>>>>>
> > > >>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <
> > > >>> > >> > >>>>>>>> stevel@cloudera.com>
> > > >>> > >> > >>>>>>>>> wrote:
> > > >>> > >> > >>>>>>>>>
> > > >>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
> > > >>> > >> > >>>>>>>> emkornfield@gmail.com>
> > > >>> > >> > >>>>>>>>>> wrote:
> > > >>> > >> > >>>>>>>>>>
> > > >>> > >> > >>>>>>>>>>>
> > > >>> > >> > >>>>>>>>>>>
> > > >>> > >> > >>>>>>>>>>> It looks like the actual issue described for ORC
> > in
> > > >>> the
> > > >>> > >> paper
> > > >>> > >> > >>>>>>>> is that
> > > >>> > >> > >>>>>>>>> it
> > > >>> > >> > >>>>>>>>>>> has multiple sub-encodings in a batch. This is
> > > >>> different
> > > >>> > >> then
> > > >>> > >> > >>>>>>>> the
> > > >>> > >> > >>>>>>>>> design
> > > >>> > >> > >>>>>>>>>>> proposed here where there is still fixed
> encoding
> > > per
> > > >>> page
> > > >>> > >> in
> > > >>> > >> > >>>>>>>> parquet.
> > > >>> > >> > >>>>>>>>>>> Given reasonably sized pages I don't think
> branch
> > > >>> > >> > >>>>>>>> misprediction should
> > > >>> > >> > >>>>>>>>>> be a
> > > >>> > >> > >>>>>>>>>>> big issue for new encodings. I agree that we
> > should
> > > >>> be
> > > >>> > >> > >>>>>>>> conservative in
> > > >>> > >> > >>>>>>>>>>> general for adding new encodings.
> > > >>> > >> > >>>>>>>>>>>
> > > >>> > >> > >>>>>>>>>>>
> > > >>> > >> > >>>>>>>>>> +1
> > > >>> > >> > >>>>>>>>>>
> > > >>> > >> > >>>>>>>>>
> > > >>> > >> > >>>>>>>>
> > > >>> > >> > >>>>>>>
> > > >>> > >> > >
> > > >>> > >> >
> > > >>> > >> >
> > > >>> > >> >
> > > >>> > >>
> > > >>> > >
> > > >>> >
> > > >>>
> > > >>
> > >
> >
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Russell Spitzer <ru...@gmail.com>.
Yes, my hope was that my vote would kick it out of spam but we definitely
had two other votes that also went to spam for me.
For anyone looking the thread is here
https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt
On Tue, Jul 21, 2026 at 10:57 AM Micah Kornfield <em...@gmail.com>
wrote:
> I've seen other's votes not go to spam, so hopefully people will see it in
> there inboxes (or see this thread and look in there spam).
>
> On Tue, Jul 21, 2026 at 10:52 AM PRATEEK GAUR <pr...@gmail.com> wrote:
>
> > Okay,
> >
> > I sent out the voting email yesterday and so far I've heard from 3 people
> > that it ended up in their "spam folder". Dev VPN to be blamed for it?.
> >
> > What's the protocol here?
> > I let people hunt for the email in their spam folder or send out another
> > one?
> >
> > Best
> > Prateek
> >
> > On Sun, Jul 19, 2026 at 10:47 PM PRATEEK GAUR <pr...@gmail.com>
> wrote:
> >
> > > Hi team,
> > >
> > > Vinoo has added the tests which were requested in the last round of
> > review.
> > > In his own words.
> > >
> > > "
> > > The extreme value tests Micah wanted are done and pushed. I added
> > coverage
> > > for values that need the full FOR bit width after the frame of
> reference
> > is
> > > applied: 64-bit (the case where the signed max minus min subtraction
> > > overflows), 63-bit (the non-overflow case), and 32-bit for floats. All
> > > lossless with zero exceptions. I also verified them through Arrow C++
> ALP
> > > reader (the local build), so the Java-written extreme columns decode
> > > correctly in C++, bit-exact against the expected values."
> > >
> > > State of the PR (#3397):
> > >
> > > - Merges cleanly with master (I pulled master in and resolved the
> > > conflicts).
> > > - 41 of 44 review threads resolved. The 3 remaining are intentional:
> > > one deferred perf nit, one that stays open until parquet-format
> ships
> > ALP,
> > >
> > > "
> > >
> > > With this both the
> > > 1) C++ PR : https://github.com/apache/arrow/pull/48345/changes
> > > 2) Java PR : https://github.com/apache/parquet-java/pull/3397
> > >
> > > Are cross language tested and range of test scenarios.
> > >
> > > Best
> > > Prateek
> > >
> > >
> > > On Mon, Jul 13, 2026 at 8:38 AM PRATEEK GAUR <pr...@gmail.com>
> wrote:
> > >
> > >> Thanks Matt,
> > >>
> > >> I'll be on it this week.
> > >>
> > >> Best
> > >> Prateek
> > >>
> > >> On Mon, Jul 13, 2026 at 8:36 AM Matt Topol <zo...@gmail.com>
> > >> wrote:
> > >>
> > >>> There is also the Go implementation submitted by Arnav (
> > >>> https://github.com/apache/arrow-go/pull/704) which is waiting for
> > >>> updates.
> > >>>
> > >>> Just wanted to make sure it didn't get lost here.
> > >>>
> > >>> --Matt
> > >>>
> > >>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK GAUR <pr...@gmail.com>
> > >>> wrote:
> > >>>
> > >>> > Hi all,
> > >>> >
> > >>> > I'd like to share a status update on the ALP encoding effort and
> get
> > a
> > >>> feel
> > >>> > from the community before starting a formal vote.
> > >>> >
> > >>> > We now have the specification plus implementations in both
> languages:
> > >>> >
> > >>> > - Spec: apache/parquet-format#557 — GH-533 Add ALP
> > encoding
> > >>> > specification
> > >>> > - C++ (Arrow): apache/arrow#48345 — GH-48701 [C++][Parquet]
> Add
> > >>> ALPpd
> > >>> > encoding
> > >>> > - Java: apache/parquet-java#3397 — Parquet Java ALP
> > >>> > Implementation (by Vinoo)
> > >>> >
> > >>> > Current state:
> > >>> >
> > >>> > - The C++ implementation has been through 3-4 rounds of review.
> > >>> > - The Java implementation has been tracking well and has had
> > >>> initial
> > >>> > reviews from contributors, with no major remaining issues.
> > >>> > - Cross-language compatibility tests are passing: the Arrow C++
> > >>> decoder
> > >>> > reads Java-written data bit-exactly across ~1.56M values and
> 18
> > >>> > fixtures, covering V1 and V2 pages, multiple vector sizes,
> and
> > >>> > several
> > >>> > real datasets — zero mismatches.
> > >>> >
> > >>> > As raised in the java-pr we will be adding coverage for extreme
> > values
> > >>> > (those requiring
> > >>> > 63-64 bits after FOR is applied) before we close a vote. We'll aim
> > to
> > >>> get
> > >>> > that
> > >>> > done in parallel.
> > >>> >
> > >>> > Plan: unless there are objections, I intend to start a vote at the
> > end
> > >>> of
> > >>> > this week/early
> > >>> > next week.
> > >>> >
> > >>> > Thanks
> > >>> > Prateek
> > >>> >
> > >>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <pr...@gmail.com>
> > >>> wrote:
> > >>> >
> > >>> > > Hi Team,
> > >>> > >
> > >>> > > Just wanted to provide some updates on ALP.
> > >>> > > Micah and I have done 3-4 rounds of review for the c++ PR :
> > >>> > > https://github.com/apache/arrow/pull/48345
> > >>> > >
> > >>> > > For the Java implementation Vinoo has been working on following
> PR
> > :
> > >>> > > https://github.com/apache/parquet-java/pull/3397
> > >>> > >
> > >>> > > Best
> > >>> > > Prateek
> > >>> > >
> > >>> > > On Tue, May 5, 2026 at 1:50 PM Micah Kornfield <
> > >>> emkornfield@gmail.com>
> > >>> > > wrote:
> > >>> > >
> > >>> > >> Hi Antoine,
> > >>> > >>
> > >>> > >> > Apologies if the question was already asked, but should we
> care
> > >>> about
> > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than
> FLOAT16
> > +
> > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> > >>> > >>
> > >>> > >> It was. We thought we could defer it for the following reasons:
> > >>> > >> 1. It's not clear there are a lot of easy reference datasets to
> > >>> test
> > >>> > its
> > >>> > >> effectiveness. It does look like there might be one or two on
> > >>> > huggingface
> > >>> > >> (e.g. https://huggingface.co/datasets/kikitora/curdie).
> > >>> > >> 2. It seemed likely that float 16 was more likely used for
> values
> > >>> that
> > >>> > >> were less likely to reduce to decimal values.
> > >>> > >> 3. It could be added as an extension later if needed.
> > >>> > >>
> > >>> > >> Cheers,
> > >>> > >> Micah
> > >>> > >>
> > >>> > >>
> > >>> > >> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <
> antoine@python.org
> > >
> > >>> > wrote:
> > >>> > >>
> > >>> > >> >
> > >>> > >> > Hello,
> > >>> > >> >
> > >>> > >> > Apologies if the question was already asked, but should we
> care
> > >>> about
> > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than
> FLOAT16
> > +
> > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> > >>> > >> >
> > >>> > >> > Regards
> > >>> > >> >
> > >>> > >> > Antoine.
> > >>> > >> >
> > >>> > >> >
> > >>> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
> > >>> > >> > > Thanks Andrew and Micah for review feedback on the two PR's
> > >>> > >> > > 1) (c++ arrow repo)
> > >>> > >> https://github.com/apache/arrow/pull/48345/changes
> > >>> > >> > > 2) (parquet-format repo)
> > >>> > >> > https://github.com/apache/parquet-format/pull/557
> > >>> > >> > >
> > >>> > >> > > I have addressed all (unless I missed something) comments on
> > >>> the two
> > >>> > >> > PR's.
> > >>> > >> > >
> > >>> > >> > > Best
> > >>> > >> > > Prateek
> > >>> > >> > >
> > >>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <
> > >>> prtkgaur@gmail.com>
> > >>> > >> wrote:
> > >>> > >> > >
> > >>> > >> > >> Thanks Andrew and Micah.
> > >>> > >> > >>
> > >>> > >> > >> `fair amount of feedback on at least the implementations`
> > >>> > >> > >> For the c++ I have already started addressing the
> feedback, I
> > >>> > should
> > >>> > >> be
> > >>> > >> > >> done with that Monday/Tuesday.
> > >>> > >> > >> I think Vinoo too has been making good progress on the Java
> > >>> > >> > implementation.
> > >>> > >> > >>
> > >>> > >> > >> Best
> > >>> > >> > >> Prateek
> > >>> > >> > >>
> > >>> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <
> > >>> > andrewlamb11@gmail.com
> > >>> > >> >
> > >>> > >> > >> wrote:
> > >>> > >> > >>
> > >>> > >> > >>> Got it. Thank you for the clarification -- I will try and
> > look
> > >>> > into
> > >>> > >> the
> > >>> > >> > >>> spec and the Rust implementation[1] in this next week
> > >>> > >> > >>>
> > >>> > >> > >>> [1]: https://github.com/apache/arrow-rs/pull/9372
> > >>> > >> > >>>
> > >>> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <
> > >>> > >> > emkornfield@gmail.com>
> > >>> > >> > >>> wrote:
> > >>> > >> > >>>
> > >>> > >> > >>>> Hi Andrew,
> > >>> > >> > >>>> I think there is a fair amount of feedback on at least
> the
> > >>> > >> > >>>> implementations, typically I think we've waited till they
> > are
> > >>> > >> close to
> > >>> > >> > >>>> mergeable before a final vote. Otherwise I agree we are
> > very
> > >>> > >> close.
> > >>> > >> > >>>>
> > >>> > >> > >>>> -Micah
> > >>> > >> > >>>>
> > >>> > >> > >>>> On Saturday, April 25, 2026, Andrew Lamb <
> > >>> andrewlamb11@gmail.com
> > >>> > >
> > >>> > >> > wrote:
> > >>> > >> > >>>>
> > >>> > >> > >>>>> Thanks Prateek,
> > >>> > >> > >>>>>
> > >>> > >> > >>>>> I think from this content it looks to me like we are
> ready
> > >>> to
> > >>> > >> start a
> > >>> > >> > >>>>> vote to explicitly accept ALP into Parquet
> > >>> > >> > >>>>>
> > >>> > >> > >>>>> Does anyone know of a reason we should postpone it for
> > >>> longer?
> > >>> > >> > >>>>> Perhaps someone needs some more time to review?
> > >>> > >> > >>>>>
> > >>> > >> > >>>>> Andrew
> > >>> > >> > >>>>>
> > >>> > >> > >>>>>
> > >>> > >> > >>>>>
> > >>> > >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <
> > >>> > prtkgaur@gmail.com>
> > >>> > >> > >>>>> wrote:
> > >>> > >> > >>>>>
> > >>> > >> > >>>>>> Hi team,
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>> Hope everyone is doing well. I got a chance to work
> > >>> through all
> > >>> > >> the
> > >>> > >> > >>>>>> remaining feedback and update the spec doc. Here are
> the
> > >>> new
> > >>> > >> > artifacts
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>> 1) Spec document :
> > >>> > >> > >>>>>>
> > >>> > >> >
> > >>> > >>
> > >>> >
> > >>>
> >
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>> 2) Spec document in parquet format repo :
> > >>> > >> > >>>>>> https://github.com/apache/parquet-format/pull/557
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>> 3) Alp implementation in arrow c++ repo :
> > >>> > >> > >>>>>> https://github.com/apache/arrow/pull/48345/changes
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>> 4) Alp implementation in parquet-java repo : Work for
> > >>> Vinoo and
> > >>> > >> > Julien
> > >>> > >> > >>>>>> https://github.com/apache/parquet-java/pull/3397
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>> 5) PR with test and benchmarking artifacts in
> > >>> parquet-testing
> > >>> > >> repo :
> > >>> > >> > >>>>>> https://github.com/apache/parquet-testing/pull/100
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>> And
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>> - Go : Arnav just submitted an in progress
> > >>> implementation
> > >>> > in
> > >>> > >> Go.
> > >>> > >> > >>>>>> https://github.com/apache/arrow-go/pull/704 (I
> > haven't
> > >>> > >> started
> > >>> > >> > >>>>>> looking at it yet)
> > >>> > >> > >>>>>> - Rust : I remember Andrew mentioned that this work
> > is
> > >>> also
> > >>> > >> in
> > >>> > >> > >>>>>> progress (So 4 languages!)
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>> *Arrow C++ implementation *
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>> The PR is out and was also used by Antoine to report
> the
> > >>> > numbers
> > >>> > >> as
> > >>> > >> > >>>>>> reported here. Micah and Konstantin have given 1 round
> of
> > >>> > >> feedback
> > >>> > >> > >>>>>> and I'm addressing them today. Please note that the
> > default
> > >>> > >> > >>>>>> optimization flag for compiling is O2 and not Q3. I got
> > >>> around
> > >>> > >> 70%
> > >>> > >> > >>>>>> performance improvement in the decoding speed when
> using
> > >>> the O3
> > >>> > >> > flag.
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>> *Parqet-MR Java implementation (working with Vinoo and
> > >>> Julien)
> > >>> > >> and
> > >>> > >> > **Cross
> > >>> > >> > >>>>>> Language testing*
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>> Let me know if you have any questions or feedback.
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>> Now pasting some performance numbers
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>> Table 1: C++ ALP Double Decode — Spotify Columns
> > >>> (Graviton
> > >>> > 3,
> > >>> > >> ARM
> > >>> > >> > >>>>>> Neoverse V1)
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>>
> > >>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │
> > >>> Speedup │
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>>
> > >>> ├──────────────────┼──────────────┼──────────────┼─────────┤
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>> │ valence │ 3,155 │ 5,523 │
> > >>> 1.75x │
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>> │ danceability │ 3,233 │ 5,685 │
> > >>> 1.76x │
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>> │ energy │ 3,197 │ 5,652 │
> > >>> 1.77x │
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>> │ loudness │ 3,186 │ 5,473 │
> > >>> 1.72x │
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>>
> > >>> └──────────────────┴──────────────┴──────────────┴─────────┘
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <
> > >>> > prtkgaur@gmail.com
> > >>> > >> >
> > >>> > >> > >>>>>> wrote:
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>>> @Micah Kornfield <em...@gmail.com> : Got it.
> > >>> > >> > >>>>>>>
> > >>> > >> > >>>>>>> @Andrew Lamb <an...@gmail.com>
> > >>> > >> > >>>>>>>
> > >>> > >> > >>>>>>>
> > >>> > >> > >>>>>>>> Do you think it would be good to start moving the
> spec
> > >>> > >> development
> > >>> > >> > >>>>>>>> into
> > >>> > >> > >>>>>>>> markdown format, in preparation for finalizing it?
> > >>> > >> > >>>>>>>>
> > >>> > >> > >>>>>>>
> > >>> > >> > >>>>>>> Yes I'll update the numbers for some of the examples I
> > >>> have in
> > >>> > >> the
> > >>> > >> > >>>>>>> spec based
> > >>> > >> > >>>>>>> on the updated header size. Then we should be good to
> go
> > >>> for
> > >>> > the
> > >>> > >> > >>>>>>> markdown format.
> > >>> > >> > >>>>>>>
> > >>> > >> > >>>>>>> Thanks everyone!
> > >>> > >> > >>>>>>>
> > >>> > >> > >>>>>>>
> > >>> > >> > >>>>>>>>
> > >>> > >> > >>>>>>>> Andrew
> > >>> > >> > >>>>>>>>
> > >>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <
> > >>> > >> prtkgaur@gmail.com>
> > >>> > >> > >>>>>>>> wrote:
> > >>> > >> > >>>>>>>>
> > >>> > >> > >>>>>>>>> Hi team,
> > >>> > >> > >>>>>>>>>
> > >>> > >> > >>>>>>>>> 1) Andrew
> > >>> > >> > >>>>>>>>>
> > >>> > >> > >>>>>>>>> - Thanks for working on test files. My PR did
> add
> > >>> all
> > >>> > the
> > >>> > >> > test
> > >>> > >> > >>>>>>>> files I
> > >>> > >> > >>>>>>>>> used to benchmark on datasets. Maybe we can club
> > it
> > >>> > >> together.
> > >>> > >> > >>>>>>>> WIll also
> > >>> > >> > >>>>>>>>> aid
> > >>> > >> > >>>>>>>>> cross language testing
> > >>> > >> > >>>>>>>>> - Kosta Tarasov working on Rust implementation.
> > >>> This is
> > >>> > >> > great.
> > >>> > >> > >>>>>>>> Thanks
> > >>> > >> > >>>>>>>>>
> > >>> > >> > >>>>>>>>>
> > >>> > >> > >>>>>>>>> 2) Antoine
> > >>> > >> > >>>>>>>>>
> > >>> > >> > >>>>>>>>> - Thanks a lot for reporting the numbers on AMD.
> > >>> Looks
> > >>> > >> like
> > >>> > >> > you
> > >>> > >> > >>>>>>>> are
> > >>> > >> > >>>>>>>>> getting 8X the decoding performance of BSS. This
> > is
> > >>> > >> > amazing!!.
> > >>> > >> > >>>>>>>>> - Thanks for acknowledging the sampling design.
> > >>> > >> > >>>>>>>>> - I agree with you on Fastlanes. In some crude
> > >>> > >> experiments I
> > >>> > >> > >>>>>>>> didn't get
> > >>> > >> > >>>>>>>>> a good perf benefit from it on Graviton3 (but
> > maybe
> > >>> > there
> > >>> > >> was
> > >>> > >> > >>>>>>>> something
> > >>> > >> > >>>>>>>>> wrong with my implementation).
> > >>> > >> > >>>>>>>>> - Locking the 16bit exception encoding for the
> > spec
> > >>> in
> > >>> > >> this
> > >>> > >> > >>>>>>>> case.
> > >>> > >> > >>>>>>>>> - Awesome I think we have solved for all open
> > >>> questions
> > >>> > >> minus
> > >>> > >> > >>>>>>>> the
> > >>> > >> > >>>>>>>>> version byte :). (will get back on this soon)
> > >>> > >> > >>>>>>>>>
> > >>> > >> > >>>>>>>>>
> > >>> > >> > >>>>>>>>> 3) Micah
> > >>> > >> > >>>>>>>>>
> > >>> > >> > >>>>>>>>> - FastLanes : The current spec does allow for
> > using
> > >>> > >> FastLane
> > >>> > >> > >>>>>>>> with the
> > >>> > >> > >>>>>>>>> configurable enum value for layout. We should be
> > >>> able to
> > >>> > >> > inject
> > >>> > >> > >>>>>>>> any
> > >>> > >> > >>>>>>>>> layout
> > >>> > >> > >>>>>>>>> in the current design.
> > >>> > >> > >>>>>>>>>
> > >>> > >> > >>>>>>>>>
> > >>> > >> > >>>>>>>>> Working on resolving all remaining open comments on
> > the
> > >>> spec
> > >>> > >> this
> > >>> > >> > >>>>>>>> week.
> > >>> > >> > >>>>>>>>>
> > >>> > >> > >>>>>>>>> Best
> > >>> > >> > >>>>>>>>> Prateek
> > >>> > >> > >>>>>>>>>
> > >>> > >> > >>>>>>>>>
> > >>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <
> > >>> > >> > >>>>>>>> stevel@cloudera.com>
> > >>> > >> > >>>>>>>>> wrote:
> > >>> > >> > >>>>>>>>>
> > >>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
> > >>> > >> > >>>>>>>> emkornfield@gmail.com>
> > >>> > >> > >>>>>>>>>> wrote:
> > >>> > >> > >>>>>>>>>>
> > >>> > >> > >>>>>>>>>>>
> > >>> > >> > >>>>>>>>>>>
> > >>> > >> > >>>>>>>>>>> It looks like the actual issue described for ORC
> in
> > >>> the
> > >>> > >> paper
> > >>> > >> > >>>>>>>> is that
> > >>> > >> > >>>>>>>>> it
> > >>> > >> > >>>>>>>>>>> has multiple sub-encodings in a batch. This is
> > >>> different
> > >>> > >> then
> > >>> > >> > >>>>>>>> the
> > >>> > >> > >>>>>>>>> design
> > >>> > >> > >>>>>>>>>>> proposed here where there is still fixed encoding
> > per
> > >>> page
> > >>> > >> in
> > >>> > >> > >>>>>>>> parquet.
> > >>> > >> > >>>>>>>>>>> Given reasonably sized pages I don't think branch
> > >>> > >> > >>>>>>>> misprediction should
> > >>> > >> > >>>>>>>>>> be a
> > >>> > >> > >>>>>>>>>>> big issue for new encodings. I agree that we
> should
> > >>> be
> > >>> > >> > >>>>>>>> conservative in
> > >>> > >> > >>>>>>>>>>> general for adding new encodings.
> > >>> > >> > >>>>>>>>>>>
> > >>> > >> > >>>>>>>>>>>
> > >>> > >> > >>>>>>>>>> +1
> > >>> > >> > >>>>>>>>>>
> > >>> > >> > >>>>>>>>>
> > >>> > >> > >>>>>>>>
> > >>> > >> > >>>>>>>
> > >>> > >> > >
> > >>> > >> >
> > >>> > >> >
> > >>> > >> >
> > >>> > >>
> > >>> > >
> > >>> >
> > >>>
> > >>
> >
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Micah Kornfield <em...@gmail.com>.
I've seen other's votes not go to spam, so hopefully people will see it in
there inboxes (or see this thread and look in there spam).
On Tue, Jul 21, 2026 at 10:52 AM PRATEEK GAUR <pr...@gmail.com> wrote:
> Okay,
>
> I sent out the voting email yesterday and so far I've heard from 3 people
> that it ended up in their "spam folder". Dev VPN to be blamed for it?.
>
> What's the protocol here?
> I let people hunt for the email in their spam folder or send out another
> one?
>
> Best
> Prateek
>
> On Sun, Jul 19, 2026 at 10:47 PM PRATEEK GAUR <pr...@gmail.com> wrote:
>
> > Hi team,
> >
> > Vinoo has added the tests which were requested in the last round of
> review.
> > In his own words.
> >
> > "
> > The extreme value tests Micah wanted are done and pushed. I added
> coverage
> > for values that need the full FOR bit width after the frame of reference
> is
> > applied: 64-bit (the case where the signed max minus min subtraction
> > overflows), 63-bit (the non-overflow case), and 32-bit for floats. All
> > lossless with zero exceptions. I also verified them through Arrow C++ ALP
> > reader (the local build), so the Java-written extreme columns decode
> > correctly in C++, bit-exact against the expected values."
> >
> > State of the PR (#3397):
> >
> > - Merges cleanly with master (I pulled master in and resolved the
> > conflicts).
> > - 41 of 44 review threads resolved. The 3 remaining are intentional:
> > one deferred perf nit, one that stays open until parquet-format ships
> ALP,
> >
> > "
> >
> > With this both the
> > 1) C++ PR : https://github.com/apache/arrow/pull/48345/changes
> > 2) Java PR : https://github.com/apache/parquet-java/pull/3397
> >
> > Are cross language tested and range of test scenarios.
> >
> > Best
> > Prateek
> >
> >
> > On Mon, Jul 13, 2026 at 8:38 AM PRATEEK GAUR <pr...@gmail.com> wrote:
> >
> >> Thanks Matt,
> >>
> >> I'll be on it this week.
> >>
> >> Best
> >> Prateek
> >>
> >> On Mon, Jul 13, 2026 at 8:36 AM Matt Topol <zo...@gmail.com>
> >> wrote:
> >>
> >>> There is also the Go implementation submitted by Arnav (
> >>> https://github.com/apache/arrow-go/pull/704) which is waiting for
> >>> updates.
> >>>
> >>> Just wanted to make sure it didn't get lost here.
> >>>
> >>> --Matt
> >>>
> >>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK GAUR <pr...@gmail.com>
> >>> wrote:
> >>>
> >>> > Hi all,
> >>> >
> >>> > I'd like to share a status update on the ALP encoding effort and get
> a
> >>> feel
> >>> > from the community before starting a formal vote.
> >>> >
> >>> > We now have the specification plus implementations in both languages:
> >>> >
> >>> > - Spec: apache/parquet-format#557 — GH-533 Add ALP
> encoding
> >>> > specification
> >>> > - C++ (Arrow): apache/arrow#48345 — GH-48701 [C++][Parquet] Add
> >>> ALPpd
> >>> > encoding
> >>> > - Java: apache/parquet-java#3397 — Parquet Java ALP
> >>> > Implementation (by Vinoo)
> >>> >
> >>> > Current state:
> >>> >
> >>> > - The C++ implementation has been through 3-4 rounds of review.
> >>> > - The Java implementation has been tracking well and has had
> >>> initial
> >>> > reviews from contributors, with no major remaining issues.
> >>> > - Cross-language compatibility tests are passing: the Arrow C++
> >>> decoder
> >>> > reads Java-written data bit-exactly across ~1.56M values and 18
> >>> > fixtures, covering V1 and V2 pages, multiple vector sizes, and
> >>> > several
> >>> > real datasets — zero mismatches.
> >>> >
> >>> > As raised in the java-pr we will be adding coverage for extreme
> values
> >>> > (those requiring
> >>> > 63-64 bits after FOR is applied) before we close a vote. We'll aim
> to
> >>> get
> >>> > that
> >>> > done in parallel.
> >>> >
> >>> > Plan: unless there are objections, I intend to start a vote at the
> end
> >>> of
> >>> > this week/early
> >>> > next week.
> >>> >
> >>> > Thanks
> >>> > Prateek
> >>> >
> >>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <pr...@gmail.com>
> >>> wrote:
> >>> >
> >>> > > Hi Team,
> >>> > >
> >>> > > Just wanted to provide some updates on ALP.
> >>> > > Micah and I have done 3-4 rounds of review for the c++ PR :
> >>> > > https://github.com/apache/arrow/pull/48345
> >>> > >
> >>> > > For the Java implementation Vinoo has been working on following PR
> :
> >>> > > https://github.com/apache/parquet-java/pull/3397
> >>> > >
> >>> > > Best
> >>> > > Prateek
> >>> > >
> >>> > > On Tue, May 5, 2026 at 1:50 PM Micah Kornfield <
> >>> emkornfield@gmail.com>
> >>> > > wrote:
> >>> > >
> >>> > >> Hi Antoine,
> >>> > >>
> >>> > >> > Apologies if the question was already asked, but should we care
> >>> about
> >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16
> +
> >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> >>> > >>
> >>> > >> It was. We thought we could defer it for the following reasons:
> >>> > >> 1. It's not clear there are a lot of easy reference datasets to
> >>> test
> >>> > its
> >>> > >> effectiveness. It does look like there might be one or two on
> >>> > huggingface
> >>> > >> (e.g. https://huggingface.co/datasets/kikitora/curdie).
> >>> > >> 2. It seemed likely that float 16 was more likely used for values
> >>> that
> >>> > >> were less likely to reduce to decimal values.
> >>> > >> 3. It could be added as an extension later if needed.
> >>> > >>
> >>> > >> Cheers,
> >>> > >> Micah
> >>> > >>
> >>> > >>
> >>> > >> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <antoine@python.org
> >
> >>> > wrote:
> >>> > >>
> >>> > >> >
> >>> > >> > Hello,
> >>> > >> >
> >>> > >> > Apologies if the question was already asked, but should we care
> >>> about
> >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16
> +
> >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> >>> > >> >
> >>> > >> > Regards
> >>> > >> >
> >>> > >> > Antoine.
> >>> > >> >
> >>> > >> >
> >>> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
> >>> > >> > > Thanks Andrew and Micah for review feedback on the two PR's
> >>> > >> > > 1) (c++ arrow repo)
> >>> > >> https://github.com/apache/arrow/pull/48345/changes
> >>> > >> > > 2) (parquet-format repo)
> >>> > >> > https://github.com/apache/parquet-format/pull/557
> >>> > >> > >
> >>> > >> > > I have addressed all (unless I missed something) comments on
> >>> the two
> >>> > >> > PR's.
> >>> > >> > >
> >>> > >> > > Best
> >>> > >> > > Prateek
> >>> > >> > >
> >>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <
> >>> prtkgaur@gmail.com>
> >>> > >> wrote:
> >>> > >> > >
> >>> > >> > >> Thanks Andrew and Micah.
> >>> > >> > >>
> >>> > >> > >> `fair amount of feedback on at least the implementations`
> >>> > >> > >> For the c++ I have already started addressing the feedback, I
> >>> > should
> >>> > >> be
> >>> > >> > >> done with that Monday/Tuesday.
> >>> > >> > >> I think Vinoo too has been making good progress on the Java
> >>> > >> > implementation.
> >>> > >> > >>
> >>> > >> > >> Best
> >>> > >> > >> Prateek
> >>> > >> > >>
> >>> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <
> >>> > andrewlamb11@gmail.com
> >>> > >> >
> >>> > >> > >> wrote:
> >>> > >> > >>
> >>> > >> > >>> Got it. Thank you for the clarification -- I will try and
> look
> >>> > into
> >>> > >> the
> >>> > >> > >>> spec and the Rust implementation[1] in this next week
> >>> > >> > >>>
> >>> > >> > >>> [1]: https://github.com/apache/arrow-rs/pull/9372
> >>> > >> > >>>
> >>> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <
> >>> > >> > emkornfield@gmail.com>
> >>> > >> > >>> wrote:
> >>> > >> > >>>
> >>> > >> > >>>> Hi Andrew,
> >>> > >> > >>>> I think there is a fair amount of feedback on at least the
> >>> > >> > >>>> implementations, typically I think we've waited till they
> are
> >>> > >> close to
> >>> > >> > >>>> mergeable before a final vote. Otherwise I agree we are
> very
> >>> > >> close.
> >>> > >> > >>>>
> >>> > >> > >>>> -Micah
> >>> > >> > >>>>
> >>> > >> > >>>> On Saturday, April 25, 2026, Andrew Lamb <
> >>> andrewlamb11@gmail.com
> >>> > >
> >>> > >> > wrote:
> >>> > >> > >>>>
> >>> > >> > >>>>> Thanks Prateek,
> >>> > >> > >>>>>
> >>> > >> > >>>>> I think from this content it looks to me like we are ready
> >>> to
> >>> > >> start a
> >>> > >> > >>>>> vote to explicitly accept ALP into Parquet
> >>> > >> > >>>>>
> >>> > >> > >>>>> Does anyone know of a reason we should postpone it for
> >>> longer?
> >>> > >> > >>>>> Perhaps someone needs some more time to review?
> >>> > >> > >>>>>
> >>> > >> > >>>>> Andrew
> >>> > >> > >>>>>
> >>> > >> > >>>>>
> >>> > >> > >>>>>
> >>> > >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <
> >>> > prtkgaur@gmail.com>
> >>> > >> > >>>>> wrote:
> >>> > >> > >>>>>
> >>> > >> > >>>>>> Hi team,
> >>> > >> > >>>>>>
> >>> > >> > >>>>>>
> >>> > >> > >>>>>>
> >>> > >> > >>>>>> Hope everyone is doing well. I got a chance to work
> >>> through all
> >>> > >> the
> >>> > >> > >>>>>> remaining feedback and update the spec doc. Here are the
> >>> new
> >>> > >> > artifacts
> >>> > >> > >>>>>>
> >>> > >> > >>>>>> 1) Spec document :
> >>> > >> > >>>>>>
> >>> > >> >
> >>> > >>
> >>> >
> >>>
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
> >>> > >> > >>>>>>
> >>> > >> > >>>>>> 2) Spec document in parquet format repo :
> >>> > >> > >>>>>> https://github.com/apache/parquet-format/pull/557
> >>> > >> > >>>>>>
> >>> > >> > >>>>>> 3) Alp implementation in arrow c++ repo :
> >>> > >> > >>>>>> https://github.com/apache/arrow/pull/48345/changes
> >>> > >> > >>>>>>
> >>> > >> > >>>>>> 4) Alp implementation in parquet-java repo : Work for
> >>> Vinoo and
> >>> > >> > Julien
> >>> > >> > >>>>>> https://github.com/apache/parquet-java/pull/3397
> >>> > >> > >>>>>>
> >>> > >> > >>>>>> 5) PR with test and benchmarking artifacts in
> >>> parquet-testing
> >>> > >> repo :
> >>> > >> > >>>>>> https://github.com/apache/parquet-testing/pull/100
> >>> > >> > >>>>>>
> >>> > >> > >>>>>>
> >>> > >> > >>>>>> And
> >>> > >> > >>>>>>
> >>> > >> > >>>>>>
> >>> > >> > >>>>>> - Go : Arnav just submitted an in progress
> >>> implementation
> >>> > in
> >>> > >> Go.
> >>> > >> > >>>>>> https://github.com/apache/arrow-go/pull/704 (I
> haven't
> >>> > >> started
> >>> > >> > >>>>>> looking at it yet)
> >>> > >> > >>>>>> - Rust : I remember Andrew mentioned that this work
> is
> >>> also
> >>> > >> in
> >>> > >> > >>>>>> progress (So 4 languages!)
> >>> > >> > >>>>>>
> >>> > >> > >>>>>>
> >>> > >> > >>>>>> *Arrow C++ implementation *
> >>> > >> > >>>>>>
> >>> > >> > >>>>>>
> >>> > >> > >>>>>>
> >>> > >> > >>>>>> The PR is out and was also used by Antoine to report the
> >>> > numbers
> >>> > >> as
> >>> > >> > >>>>>> reported here. Micah and Konstantin have given 1 round of
> >>> > >> feedback
> >>> > >> > >>>>>> and I'm addressing them today. Please note that the
> default
> >>> > >> > >>>>>> optimization flag for compiling is O2 and not Q3. I got
> >>> around
> >>> > >> 70%
> >>> > >> > >>>>>> performance improvement in the decoding speed when using
> >>> the O3
> >>> > >> > flag.
> >>> > >> > >>>>>>
> >>> > >> > >>>>>>
> >>> > >> > >>>>>>
> >>> > >> > >>>>>> *Parqet-MR Java implementation (working with Vinoo and
> >>> Julien)
> >>> > >> and
> >>> > >> > **Cross
> >>> > >> > >>>>>> Language testing*
> >>> > >> > >>>>>>
> >>> > >> > >>>>>>
> >>> > >> > >>>>>> Let me know if you have any questions or feedback.
> >>> > >> > >>>>>>
> >>> > >> > >>>>>>
> >>> > >> > >>>>>>
> >>> > >> > >>>>>> Now pasting some performance numbers
> >>> > >> > >>>>>>
> >>> > >> > >>>>>>
> >>> > >> > >>>>>> Table 1: C++ ALP Double Decode — Spotify Columns
> >>> (Graviton
> >>> > 3,
> >>> > >> ARM
> >>> > >> > >>>>>> Neoverse V1)
> >>> > >> > >>>>>>
> >>> > >> > >>>>>>
> >>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
> >>> > >> > >>>>>>
> >>> > >> > >>>>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │
> >>> Speedup │
> >>> > >> > >>>>>>
> >>> > >> > >>>>>>
> >>> ├──────────────────┼──────────────┼──────────────┼─────────┤
> >>> > >> > >>>>>>
> >>> > >> > >>>>>> │ valence │ 3,155 │ 5,523 │
> >>> 1.75x │
> >>> > >> > >>>>>>
> >>> > >> > >>>>>> │ danceability │ 3,233 │ 5,685 │
> >>> 1.76x │
> >>> > >> > >>>>>>
> >>> > >> > >>>>>> │ energy │ 3,197 │ 5,652 │
> >>> 1.77x │
> >>> > >> > >>>>>>
> >>> > >> > >>>>>> │ loudness │ 3,186 │ 5,473 │
> >>> 1.72x │
> >>> > >> > >>>>>>
> >>> > >> > >>>>>>
> >>> └──────────────────┴──────────────┴──────────────┴─────────┘
> >>> > >> > >>>>>>
> >>> > >> > >>>>>>
> >>> > >> > >>>>>>
> >>> > >> > >>>>>>
> >>> > >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <
> >>> > prtkgaur@gmail.com
> >>> > >> >
> >>> > >> > >>>>>> wrote:
> >>> > >> > >>>>>>
> >>> > >> > >>>>>>> @Micah Kornfield <em...@gmail.com> : Got it.
> >>> > >> > >>>>>>>
> >>> > >> > >>>>>>> @Andrew Lamb <an...@gmail.com>
> >>> > >> > >>>>>>>
> >>> > >> > >>>>>>>
> >>> > >> > >>>>>>>> Do you think it would be good to start moving the spec
> >>> > >> development
> >>> > >> > >>>>>>>> into
> >>> > >> > >>>>>>>> markdown format, in preparation for finalizing it?
> >>> > >> > >>>>>>>>
> >>> > >> > >>>>>>>
> >>> > >> > >>>>>>> Yes I'll update the numbers for some of the examples I
> >>> have in
> >>> > >> the
> >>> > >> > >>>>>>> spec based
> >>> > >> > >>>>>>> on the updated header size. Then we should be good to go
> >>> for
> >>> > the
> >>> > >> > >>>>>>> markdown format.
> >>> > >> > >>>>>>>
> >>> > >> > >>>>>>> Thanks everyone!
> >>> > >> > >>>>>>>
> >>> > >> > >>>>>>>
> >>> > >> > >>>>>>>>
> >>> > >> > >>>>>>>> Andrew
> >>> > >> > >>>>>>>>
> >>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <
> >>> > >> prtkgaur@gmail.com>
> >>> > >> > >>>>>>>> wrote:
> >>> > >> > >>>>>>>>
> >>> > >> > >>>>>>>>> Hi team,
> >>> > >> > >>>>>>>>>
> >>> > >> > >>>>>>>>> 1) Andrew
> >>> > >> > >>>>>>>>>
> >>> > >> > >>>>>>>>> - Thanks for working on test files. My PR did add
> >>> all
> >>> > the
> >>> > >> > test
> >>> > >> > >>>>>>>> files I
> >>> > >> > >>>>>>>>> used to benchmark on datasets. Maybe we can club
> it
> >>> > >> together.
> >>> > >> > >>>>>>>> WIll also
> >>> > >> > >>>>>>>>> aid
> >>> > >> > >>>>>>>>> cross language testing
> >>> > >> > >>>>>>>>> - Kosta Tarasov working on Rust implementation.
> >>> This is
> >>> > >> > great.
> >>> > >> > >>>>>>>> Thanks
> >>> > >> > >>>>>>>>>
> >>> > >> > >>>>>>>>>
> >>> > >> > >>>>>>>>> 2) Antoine
> >>> > >> > >>>>>>>>>
> >>> > >> > >>>>>>>>> - Thanks a lot for reporting the numbers on AMD.
> >>> Looks
> >>> > >> like
> >>> > >> > you
> >>> > >> > >>>>>>>> are
> >>> > >> > >>>>>>>>> getting 8X the decoding performance of BSS. This
> is
> >>> > >> > amazing!!.
> >>> > >> > >>>>>>>>> - Thanks for acknowledging the sampling design.
> >>> > >> > >>>>>>>>> - I agree with you on Fastlanes. In some crude
> >>> > >> experiments I
> >>> > >> > >>>>>>>> didn't get
> >>> > >> > >>>>>>>>> a good perf benefit from it on Graviton3 (but
> maybe
> >>> > there
> >>> > >> was
> >>> > >> > >>>>>>>> something
> >>> > >> > >>>>>>>>> wrong with my implementation).
> >>> > >> > >>>>>>>>> - Locking the 16bit exception encoding for the
> spec
> >>> in
> >>> > >> this
> >>> > >> > >>>>>>>> case.
> >>> > >> > >>>>>>>>> - Awesome I think we have solved for all open
> >>> questions
> >>> > >> minus
> >>> > >> > >>>>>>>> the
> >>> > >> > >>>>>>>>> version byte :). (will get back on this soon)
> >>> > >> > >>>>>>>>>
> >>> > >> > >>>>>>>>>
> >>> > >> > >>>>>>>>> 3) Micah
> >>> > >> > >>>>>>>>>
> >>> > >> > >>>>>>>>> - FastLanes : The current spec does allow for
> using
> >>> > >> FastLane
> >>> > >> > >>>>>>>> with the
> >>> > >> > >>>>>>>>> configurable enum value for layout. We should be
> >>> able to
> >>> > >> > inject
> >>> > >> > >>>>>>>> any
> >>> > >> > >>>>>>>>> layout
> >>> > >> > >>>>>>>>> in the current design.
> >>> > >> > >>>>>>>>>
> >>> > >> > >>>>>>>>>
> >>> > >> > >>>>>>>>> Working on resolving all remaining open comments on
> the
> >>> spec
> >>> > >> this
> >>> > >> > >>>>>>>> week.
> >>> > >> > >>>>>>>>>
> >>> > >> > >>>>>>>>> Best
> >>> > >> > >>>>>>>>> Prateek
> >>> > >> > >>>>>>>>>
> >>> > >> > >>>>>>>>>
> >>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <
> >>> > >> > >>>>>>>> stevel@cloudera.com>
> >>> > >> > >>>>>>>>> wrote:
> >>> > >> > >>>>>>>>>
> >>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
> >>> > >> > >>>>>>>> emkornfield@gmail.com>
> >>> > >> > >>>>>>>>>> wrote:
> >>> > >> > >>>>>>>>>>
> >>> > >> > >>>>>>>>>>>
> >>> > >> > >>>>>>>>>>>
> >>> > >> > >>>>>>>>>>> It looks like the actual issue described for ORC in
> >>> the
> >>> > >> paper
> >>> > >> > >>>>>>>> is that
> >>> > >> > >>>>>>>>> it
> >>> > >> > >>>>>>>>>>> has multiple sub-encodings in a batch. This is
> >>> different
> >>> > >> then
> >>> > >> > >>>>>>>> the
> >>> > >> > >>>>>>>>> design
> >>> > >> > >>>>>>>>>>> proposed here where there is still fixed encoding
> per
> >>> page
> >>> > >> in
> >>> > >> > >>>>>>>> parquet.
> >>> > >> > >>>>>>>>>>> Given reasonably sized pages I don't think branch
> >>> > >> > >>>>>>>> misprediction should
> >>> > >> > >>>>>>>>>> be a
> >>> > >> > >>>>>>>>>>> big issue for new encodings. I agree that we should
> >>> be
> >>> > >> > >>>>>>>> conservative in
> >>> > >> > >>>>>>>>>>> general for adding new encodings.
> >>> > >> > >>>>>>>>>>>
> >>> > >> > >>>>>>>>>>>
> >>> > >> > >>>>>>>>>> +1
> >>> > >> > >>>>>>>>>>
> >>> > >> > >>>>>>>>>
> >>> > >> > >>>>>>>>
> >>> > >> > >>>>>>>
> >>> > >> > >
> >>> > >> >
> >>> > >> >
> >>> > >> >
> >>> > >>
> >>> > >
> >>> >
> >>>
> >>
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by PRATEEK GAUR <pr...@gmail.com>.
Okay,
I sent out the voting email yesterday and so far I've heard from 3 people
that it ended up in their "spam folder". Dev VPN to be blamed for it?.
What's the protocol here?
I let people hunt for the email in their spam folder or send out another
one?
Best
Prateek
On Sun, Jul 19, 2026 at 10:47 PM PRATEEK GAUR <pr...@gmail.com> wrote:
> Hi team,
>
> Vinoo has added the tests which were requested in the last round of review.
> In his own words.
>
> "
> The extreme value tests Micah wanted are done and pushed. I added coverage
> for values that need the full FOR bit width after the frame of reference is
> applied: 64-bit (the case where the signed max minus min subtraction
> overflows), 63-bit (the non-overflow case), and 32-bit for floats. All
> lossless with zero exceptions. I also verified them through Arrow C++ ALP
> reader (the local build), so the Java-written extreme columns decode
> correctly in C++, bit-exact against the expected values."
>
> State of the PR (#3397):
>
> - Merges cleanly with master (I pulled master in and resolved the
> conflicts).
> - 41 of 44 review threads resolved. The 3 remaining are intentional:
> one deferred perf nit, one that stays open until parquet-format ships ALP,
>
> "
>
> With this both the
> 1) C++ PR : https://github.com/apache/arrow/pull/48345/changes
> 2) Java PR : https://github.com/apache/parquet-java/pull/3397
>
> Are cross language tested and range of test scenarios.
>
> Best
> Prateek
>
>
> On Mon, Jul 13, 2026 at 8:38 AM PRATEEK GAUR <pr...@gmail.com> wrote:
>
>> Thanks Matt,
>>
>> I'll be on it this week.
>>
>> Best
>> Prateek
>>
>> On Mon, Jul 13, 2026 at 8:36 AM Matt Topol <zo...@gmail.com>
>> wrote:
>>
>>> There is also the Go implementation submitted by Arnav (
>>> https://github.com/apache/arrow-go/pull/704) which is waiting for
>>> updates.
>>>
>>> Just wanted to make sure it didn't get lost here.
>>>
>>> --Matt
>>>
>>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK GAUR <pr...@gmail.com>
>>> wrote:
>>>
>>> > Hi all,
>>> >
>>> > I'd like to share a status update on the ALP encoding effort and get a
>>> feel
>>> > from the community before starting a formal vote.
>>> >
>>> > We now have the specification plus implementations in both languages:
>>> >
>>> > - Spec: apache/parquet-format#557 — GH-533 Add ALP encoding
>>> > specification
>>> > - C++ (Arrow): apache/arrow#48345 — GH-48701 [C++][Parquet] Add
>>> ALPpd
>>> > encoding
>>> > - Java: apache/parquet-java#3397 — Parquet Java ALP
>>> > Implementation (by Vinoo)
>>> >
>>> > Current state:
>>> >
>>> > - The C++ implementation has been through 3-4 rounds of review.
>>> > - The Java implementation has been tracking well and has had
>>> initial
>>> > reviews from contributors, with no major remaining issues.
>>> > - Cross-language compatibility tests are passing: the Arrow C++
>>> decoder
>>> > reads Java-written data bit-exactly across ~1.56M values and 18
>>> > fixtures, covering V1 and V2 pages, multiple vector sizes, and
>>> > several
>>> > real datasets — zero mismatches.
>>> >
>>> > As raised in the java-pr we will be adding coverage for extreme values
>>> > (those requiring
>>> > 63-64 bits after FOR is applied) before we close a vote. We'll aim to
>>> get
>>> > that
>>> > done in parallel.
>>> >
>>> > Plan: unless there are objections, I intend to start a vote at the end
>>> of
>>> > this week/early
>>> > next week.
>>> >
>>> > Thanks
>>> > Prateek
>>> >
>>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <pr...@gmail.com>
>>> wrote:
>>> >
>>> > > Hi Team,
>>> > >
>>> > > Just wanted to provide some updates on ALP.
>>> > > Micah and I have done 3-4 rounds of review for the c++ PR :
>>> > > https://github.com/apache/arrow/pull/48345
>>> > >
>>> > > For the Java implementation Vinoo has been working on following PR :
>>> > > https://github.com/apache/parquet-java/pull/3397
>>> > >
>>> > > Best
>>> > > Prateek
>>> > >
>>> > > On Tue, May 5, 2026 at 1:50 PM Micah Kornfield <
>>> emkornfield@gmail.com>
>>> > > wrote:
>>> > >
>>> > >> Hi Antoine,
>>> > >>
>>> > >> > Apologies if the question was already asked, but should we care
>>> about
>>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
>>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
>>> > >>
>>> > >> It was. We thought we could defer it for the following reasons:
>>> > >> 1. It's not clear there are a lot of easy reference datasets to
>>> test
>>> > its
>>> > >> effectiveness. It does look like there might be one or two on
>>> > huggingface
>>> > >> (e.g. https://huggingface.co/datasets/kikitora/curdie).
>>> > >> 2. It seemed likely that float 16 was more likely used for values
>>> that
>>> > >> were less likely to reduce to decimal values.
>>> > >> 3. It could be added as an extension later if needed.
>>> > >>
>>> > >> Cheers,
>>> > >> Micah
>>> > >>
>>> > >>
>>> > >> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <an...@python.org>
>>> > wrote:
>>> > >>
>>> > >> >
>>> > >> > Hello,
>>> > >> >
>>> > >> > Apologies if the question was already asked, but should we care
>>> about
>>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
>>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
>>> > >> >
>>> > >> > Regards
>>> > >> >
>>> > >> > Antoine.
>>> > >> >
>>> > >> >
>>> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
>>> > >> > > Thanks Andrew and Micah for review feedback on the two PR's
>>> > >> > > 1) (c++ arrow repo)
>>> > >> https://github.com/apache/arrow/pull/48345/changes
>>> > >> > > 2) (parquet-format repo)
>>> > >> > https://github.com/apache/parquet-format/pull/557
>>> > >> > >
>>> > >> > > I have addressed all (unless I missed something) comments on
>>> the two
>>> > >> > PR's.
>>> > >> > >
>>> > >> > > Best
>>> > >> > > Prateek
>>> > >> > >
>>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <
>>> prtkgaur@gmail.com>
>>> > >> wrote:
>>> > >> > >
>>> > >> > >> Thanks Andrew and Micah.
>>> > >> > >>
>>> > >> > >> `fair amount of feedback on at least the implementations`
>>> > >> > >> For the c++ I have already started addressing the feedback, I
>>> > should
>>> > >> be
>>> > >> > >> done with that Monday/Tuesday.
>>> > >> > >> I think Vinoo too has been making good progress on the Java
>>> > >> > implementation.
>>> > >> > >>
>>> > >> > >> Best
>>> > >> > >> Prateek
>>> > >> > >>
>>> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <
>>> > andrewlamb11@gmail.com
>>> > >> >
>>> > >> > >> wrote:
>>> > >> > >>
>>> > >> > >>> Got it. Thank you for the clarification -- I will try and look
>>> > into
>>> > >> the
>>> > >> > >>> spec and the Rust implementation[1] in this next week
>>> > >> > >>>
>>> > >> > >>> [1]: https://github.com/apache/arrow-rs/pull/9372
>>> > >> > >>>
>>> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <
>>> > >> > emkornfield@gmail.com>
>>> > >> > >>> wrote:
>>> > >> > >>>
>>> > >> > >>>> Hi Andrew,
>>> > >> > >>>> I think there is a fair amount of feedback on at least the
>>> > >> > >>>> implementations, typically I think we've waited till they are
>>> > >> close to
>>> > >> > >>>> mergeable before a final vote. Otherwise I agree we are very
>>> > >> close.
>>> > >> > >>>>
>>> > >> > >>>> -Micah
>>> > >> > >>>>
>>> > >> > >>>> On Saturday, April 25, 2026, Andrew Lamb <
>>> andrewlamb11@gmail.com
>>> > >
>>> > >> > wrote:
>>> > >> > >>>>
>>> > >> > >>>>> Thanks Prateek,
>>> > >> > >>>>>
>>> > >> > >>>>> I think from this content it looks to me like we are ready
>>> to
>>> > >> start a
>>> > >> > >>>>> vote to explicitly accept ALP into Parquet
>>> > >> > >>>>>
>>> > >> > >>>>> Does anyone know of a reason we should postpone it for
>>> longer?
>>> > >> > >>>>> Perhaps someone needs some more time to review?
>>> > >> > >>>>>
>>> > >> > >>>>> Andrew
>>> > >> > >>>>>
>>> > >> > >>>>>
>>> > >> > >>>>>
>>> > >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <
>>> > prtkgaur@gmail.com>
>>> > >> > >>>>> wrote:
>>> > >> > >>>>>
>>> > >> > >>>>>> Hi team,
>>> > >> > >>>>>>
>>> > >> > >>>>>>
>>> > >> > >>>>>>
>>> > >> > >>>>>> Hope everyone is doing well. I got a chance to work
>>> through all
>>> > >> the
>>> > >> > >>>>>> remaining feedback and update the spec doc. Here are the
>>> new
>>> > >> > artifacts
>>> > >> > >>>>>>
>>> > >> > >>>>>> 1) Spec document :
>>> > >> > >>>>>>
>>> > >> >
>>> > >>
>>> >
>>> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
>>> > >> > >>>>>>
>>> > >> > >>>>>> 2) Spec document in parquet format repo :
>>> > >> > >>>>>> https://github.com/apache/parquet-format/pull/557
>>> > >> > >>>>>>
>>> > >> > >>>>>> 3) Alp implementation in arrow c++ repo :
>>> > >> > >>>>>> https://github.com/apache/arrow/pull/48345/changes
>>> > >> > >>>>>>
>>> > >> > >>>>>> 4) Alp implementation in parquet-java repo : Work for
>>> Vinoo and
>>> > >> > Julien
>>> > >> > >>>>>> https://github.com/apache/parquet-java/pull/3397
>>> > >> > >>>>>>
>>> > >> > >>>>>> 5) PR with test and benchmarking artifacts in
>>> parquet-testing
>>> > >> repo :
>>> > >> > >>>>>> https://github.com/apache/parquet-testing/pull/100
>>> > >> > >>>>>>
>>> > >> > >>>>>>
>>> > >> > >>>>>> And
>>> > >> > >>>>>>
>>> > >> > >>>>>>
>>> > >> > >>>>>> - Go : Arnav just submitted an in progress
>>> implementation
>>> > in
>>> > >> Go.
>>> > >> > >>>>>> https://github.com/apache/arrow-go/pull/704 (I haven't
>>> > >> started
>>> > >> > >>>>>> looking at it yet)
>>> > >> > >>>>>> - Rust : I remember Andrew mentioned that this work is
>>> also
>>> > >> in
>>> > >> > >>>>>> progress (So 4 languages!)
>>> > >> > >>>>>>
>>> > >> > >>>>>>
>>> > >> > >>>>>> *Arrow C++ implementation *
>>> > >> > >>>>>>
>>> > >> > >>>>>>
>>> > >> > >>>>>>
>>> > >> > >>>>>> The PR is out and was also used by Antoine to report the
>>> > numbers
>>> > >> as
>>> > >> > >>>>>> reported here. Micah and Konstantin have given 1 round of
>>> > >> feedback
>>> > >> > >>>>>> and I'm addressing them today. Please note that the default
>>> > >> > >>>>>> optimization flag for compiling is O2 and not Q3. I got
>>> around
>>> > >> 70%
>>> > >> > >>>>>> performance improvement in the decoding speed when using
>>> the O3
>>> > >> > flag.
>>> > >> > >>>>>>
>>> > >> > >>>>>>
>>> > >> > >>>>>>
>>> > >> > >>>>>> *Parqet-MR Java implementation (working with Vinoo and
>>> Julien)
>>> > >> and
>>> > >> > **Cross
>>> > >> > >>>>>> Language testing*
>>> > >> > >>>>>>
>>> > >> > >>>>>>
>>> > >> > >>>>>> Let me know if you have any questions or feedback.
>>> > >> > >>>>>>
>>> > >> > >>>>>>
>>> > >> > >>>>>>
>>> > >> > >>>>>> Now pasting some performance numbers
>>> > >> > >>>>>>
>>> > >> > >>>>>>
>>> > >> > >>>>>> Table 1: C++ ALP Double Decode — Spotify Columns
>>> (Graviton
>>> > 3,
>>> > >> ARM
>>> > >> > >>>>>> Neoverse V1)
>>> > >> > >>>>>>
>>> > >> > >>>>>>
>>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
>>> > >> > >>>>>>
>>> > >> > >>>>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │
>>> Speedup │
>>> > >> > >>>>>>
>>> > >> > >>>>>>
>>> ├──────────────────┼──────────────┼──────────────┼─────────┤
>>> > >> > >>>>>>
>>> > >> > >>>>>> │ valence │ 3,155 │ 5,523 │
>>> 1.75x │
>>> > >> > >>>>>>
>>> > >> > >>>>>> │ danceability │ 3,233 │ 5,685 │
>>> 1.76x │
>>> > >> > >>>>>>
>>> > >> > >>>>>> │ energy │ 3,197 │ 5,652 │
>>> 1.77x │
>>> > >> > >>>>>>
>>> > >> > >>>>>> │ loudness │ 3,186 │ 5,473 │
>>> 1.72x │
>>> > >> > >>>>>>
>>> > >> > >>>>>>
>>> └──────────────────┴──────────────┴──────────────┴─────────┘
>>> > >> > >>>>>>
>>> > >> > >>>>>>
>>> > >> > >>>>>>
>>> > >> > >>>>>>
>>> > >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <
>>> > prtkgaur@gmail.com
>>> > >> >
>>> > >> > >>>>>> wrote:
>>> > >> > >>>>>>
>>> > >> > >>>>>>> @Micah Kornfield <em...@gmail.com> : Got it.
>>> > >> > >>>>>>>
>>> > >> > >>>>>>> @Andrew Lamb <an...@gmail.com>
>>> > >> > >>>>>>>
>>> > >> > >>>>>>>
>>> > >> > >>>>>>>> Do you think it would be good to start moving the spec
>>> > >> development
>>> > >> > >>>>>>>> into
>>> > >> > >>>>>>>> markdown format, in preparation for finalizing it?
>>> > >> > >>>>>>>>
>>> > >> > >>>>>>>
>>> > >> > >>>>>>> Yes I'll update the numbers for some of the examples I
>>> have in
>>> > >> the
>>> > >> > >>>>>>> spec based
>>> > >> > >>>>>>> on the updated header size. Then we should be good to go
>>> for
>>> > the
>>> > >> > >>>>>>> markdown format.
>>> > >> > >>>>>>>
>>> > >> > >>>>>>> Thanks everyone!
>>> > >> > >>>>>>>
>>> > >> > >>>>>>>
>>> > >> > >>>>>>>>
>>> > >> > >>>>>>>> Andrew
>>> > >> > >>>>>>>>
>>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <
>>> > >> prtkgaur@gmail.com>
>>> > >> > >>>>>>>> wrote:
>>> > >> > >>>>>>>>
>>> > >> > >>>>>>>>> Hi team,
>>> > >> > >>>>>>>>>
>>> > >> > >>>>>>>>> 1) Andrew
>>> > >> > >>>>>>>>>
>>> > >> > >>>>>>>>> - Thanks for working on test files. My PR did add
>>> all
>>> > the
>>> > >> > test
>>> > >> > >>>>>>>> files I
>>> > >> > >>>>>>>>> used to benchmark on datasets. Maybe we can club it
>>> > >> together.
>>> > >> > >>>>>>>> WIll also
>>> > >> > >>>>>>>>> aid
>>> > >> > >>>>>>>>> cross language testing
>>> > >> > >>>>>>>>> - Kosta Tarasov working on Rust implementation.
>>> This is
>>> > >> > great.
>>> > >> > >>>>>>>> Thanks
>>> > >> > >>>>>>>>>
>>> > >> > >>>>>>>>>
>>> > >> > >>>>>>>>> 2) Antoine
>>> > >> > >>>>>>>>>
>>> > >> > >>>>>>>>> - Thanks a lot for reporting the numbers on AMD.
>>> Looks
>>> > >> like
>>> > >> > you
>>> > >> > >>>>>>>> are
>>> > >> > >>>>>>>>> getting 8X the decoding performance of BSS. This is
>>> > >> > amazing!!.
>>> > >> > >>>>>>>>> - Thanks for acknowledging the sampling design.
>>> > >> > >>>>>>>>> - I agree with you on Fastlanes. In some crude
>>> > >> experiments I
>>> > >> > >>>>>>>> didn't get
>>> > >> > >>>>>>>>> a good perf benefit from it on Graviton3 (but maybe
>>> > there
>>> > >> was
>>> > >> > >>>>>>>> something
>>> > >> > >>>>>>>>> wrong with my implementation).
>>> > >> > >>>>>>>>> - Locking the 16bit exception encoding for the spec
>>> in
>>> > >> this
>>> > >> > >>>>>>>> case.
>>> > >> > >>>>>>>>> - Awesome I think we have solved for all open
>>> questions
>>> > >> minus
>>> > >> > >>>>>>>> the
>>> > >> > >>>>>>>>> version byte :). (will get back on this soon)
>>> > >> > >>>>>>>>>
>>> > >> > >>>>>>>>>
>>> > >> > >>>>>>>>> 3) Micah
>>> > >> > >>>>>>>>>
>>> > >> > >>>>>>>>> - FastLanes : The current spec does allow for using
>>> > >> FastLane
>>> > >> > >>>>>>>> with the
>>> > >> > >>>>>>>>> configurable enum value for layout. We should be
>>> able to
>>> > >> > inject
>>> > >> > >>>>>>>> any
>>> > >> > >>>>>>>>> layout
>>> > >> > >>>>>>>>> in the current design.
>>> > >> > >>>>>>>>>
>>> > >> > >>>>>>>>>
>>> > >> > >>>>>>>>> Working on resolving all remaining open comments on the
>>> spec
>>> > >> this
>>> > >> > >>>>>>>> week.
>>> > >> > >>>>>>>>>
>>> > >> > >>>>>>>>> Best
>>> > >> > >>>>>>>>> Prateek
>>> > >> > >>>>>>>>>
>>> > >> > >>>>>>>>>
>>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <
>>> > >> > >>>>>>>> stevel@cloudera.com>
>>> > >> > >>>>>>>>> wrote:
>>> > >> > >>>>>>>>>
>>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
>>> > >> > >>>>>>>> emkornfield@gmail.com>
>>> > >> > >>>>>>>>>> wrote:
>>> > >> > >>>>>>>>>>
>>> > >> > >>>>>>>>>>>
>>> > >> > >>>>>>>>>>>
>>> > >> > >>>>>>>>>>> It looks like the actual issue described for ORC in
>>> the
>>> > >> paper
>>> > >> > >>>>>>>> is that
>>> > >> > >>>>>>>>> it
>>> > >> > >>>>>>>>>>> has multiple sub-encodings in a batch. This is
>>> different
>>> > >> then
>>> > >> > >>>>>>>> the
>>> > >> > >>>>>>>>> design
>>> > >> > >>>>>>>>>>> proposed here where there is still fixed encoding per
>>> page
>>> > >> in
>>> > >> > >>>>>>>> parquet.
>>> > >> > >>>>>>>>>>> Given reasonably sized pages I don't think branch
>>> > >> > >>>>>>>> misprediction should
>>> > >> > >>>>>>>>>> be a
>>> > >> > >>>>>>>>>>> big issue for new encodings. I agree that we should
>>> be
>>> > >> > >>>>>>>> conservative in
>>> > >> > >>>>>>>>>>> general for adding new encodings.
>>> > >> > >>>>>>>>>>>
>>> > >> > >>>>>>>>>>>
>>> > >> > >>>>>>>>>> +1
>>> > >> > >>>>>>>>>>
>>> > >> > >>>>>>>>>
>>> > >> > >>>>>>>>
>>> > >> > >>>>>>>
>>> > >> > >
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >>
>>> > >
>>> >
>>>
>>
Re: [Parquet] ALP Encoding for Floating point data
Posted by PRATEEK GAUR <pr...@gmail.com>.
Hi team,
Vinoo has added the tests which were requested in the last round of review.
In his own words.
"
The extreme value tests Micah wanted are done and pushed. I added coverage
for values that need the full FOR bit width after the frame of reference is
applied: 64-bit (the case where the signed max minus min subtraction
overflows), 63-bit (the non-overflow case), and 32-bit for floats. All
lossless with zero exceptions. I also verified them through Arrow C++ ALP
reader (the local build), so the Java-written extreme columns decode
correctly in C++, bit-exact against the expected values."
State of the PR (#3397):
- Merges cleanly with master (I pulled master in and resolved the
conflicts).
- 41 of 44 review threads resolved. The 3 remaining are intentional: one
deferred perf nit, one that stays open until parquet-format ships ALP,
"
With this both the
1) C++ PR : https://github.com/apache/arrow/pull/48345/changes
2) Java PR : https://github.com/apache/parquet-java/pull/3397
Are cross language tested and range of test scenarios.
Best
Prateek
On Mon, Jul 13, 2026 at 8:38 AM PRATEEK GAUR <pr...@gmail.com> wrote:
> Thanks Matt,
>
> I'll be on it this week.
>
> Best
> Prateek
>
> On Mon, Jul 13, 2026 at 8:36 AM Matt Topol <zo...@gmail.com> wrote:
>
>> There is also the Go implementation submitted by Arnav (
>> https://github.com/apache/arrow-go/pull/704) which is waiting for
>> updates.
>>
>> Just wanted to make sure it didn't get lost here.
>>
>> --Matt
>>
>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK GAUR <pr...@gmail.com> wrote:
>>
>> > Hi all,
>> >
>> > I'd like to share a status update on the ALP encoding effort and get a
>> feel
>> > from the community before starting a formal vote.
>> >
>> > We now have the specification plus implementations in both languages:
>> >
>> > - Spec: apache/parquet-format#557 — GH-533 Add ALP encoding
>> > specification
>> > - C++ (Arrow): apache/arrow#48345 — GH-48701 [C++][Parquet] Add
>> ALPpd
>> > encoding
>> > - Java: apache/parquet-java#3397 — Parquet Java ALP
>> > Implementation (by Vinoo)
>> >
>> > Current state:
>> >
>> > - The C++ implementation has been through 3-4 rounds of review.
>> > - The Java implementation has been tracking well and has had initial
>> > reviews from contributors, with no major remaining issues.
>> > - Cross-language compatibility tests are passing: the Arrow C++
>> decoder
>> > reads Java-written data bit-exactly across ~1.56M values and 18
>> > fixtures, covering V1 and V2 pages, multiple vector sizes, and
>> > several
>> > real datasets — zero mismatches.
>> >
>> > As raised in the java-pr we will be adding coverage for extreme values
>> > (those requiring
>> > 63-64 bits after FOR is applied) before we close a vote. We'll aim to
>> get
>> > that
>> > done in parallel.
>> >
>> > Plan: unless there are objections, I intend to start a vote at the end
>> of
>> > this week/early
>> > next week.
>> >
>> > Thanks
>> > Prateek
>> >
>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <pr...@gmail.com> wrote:
>> >
>> > > Hi Team,
>> > >
>> > > Just wanted to provide some updates on ALP.
>> > > Micah and I have done 3-4 rounds of review for the c++ PR :
>> > > https://github.com/apache/arrow/pull/48345
>> > >
>> > > For the Java implementation Vinoo has been working on following PR :
>> > > https://github.com/apache/parquet-java/pull/3397
>> > >
>> > > Best
>> > > Prateek
>> > >
>> > > On Tue, May 5, 2026 at 1:50 PM Micah Kornfield <emkornfield@gmail.com
>> >
>> > > wrote:
>> > >
>> > >> Hi Antoine,
>> > >>
>> > >> > Apologies if the question was already asked, but should we care
>> about
>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
>> > >>
>> > >> It was. We thought we could defer it for the following reasons:
>> > >> 1. It's not clear there are a lot of easy reference datasets to test
>> > its
>> > >> effectiveness. It does look like there might be one or two on
>> > huggingface
>> > >> (e.g. https://huggingface.co/datasets/kikitora/curdie).
>> > >> 2. It seemed likely that float 16 was more likely used for values
>> that
>> > >> were less likely to reduce to decimal values.
>> > >> 3. It could be added as an extension later if needed.
>> > >>
>> > >> Cheers,
>> > >> Micah
>> > >>
>> > >>
>> > >> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <an...@python.org>
>> > wrote:
>> > >>
>> > >> >
>> > >> > Hello,
>> > >> >
>> > >> > Apologies if the question was already asked, but should we care
>> about
>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
>> > >> >
>> > >> > Regards
>> > >> >
>> > >> > Antoine.
>> > >> >
>> > >> >
>> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
>> > >> > > Thanks Andrew and Micah for review feedback on the two PR's
>> > >> > > 1) (c++ arrow repo)
>> > >> https://github.com/apache/arrow/pull/48345/changes
>> > >> > > 2) (parquet-format repo)
>> > >> > https://github.com/apache/parquet-format/pull/557
>> > >> > >
>> > >> > > I have addressed all (unless I missed something) comments on the
>> two
>> > >> > PR's.
>> > >> > >
>> > >> > > Best
>> > >> > > Prateek
>> > >> > >
>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <prtkgaur@gmail.com
>> >
>> > >> wrote:
>> > >> > >
>> > >> > >> Thanks Andrew and Micah.
>> > >> > >>
>> > >> > >> `fair amount of feedback on at least the implementations`
>> > >> > >> For the c++ I have already started addressing the feedback, I
>> > should
>> > >> be
>> > >> > >> done with that Monday/Tuesday.
>> > >> > >> I think Vinoo too has been making good progress on the Java
>> > >> > implementation.
>> > >> > >>
>> > >> > >> Best
>> > >> > >> Prateek
>> > >> > >>
>> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <
>> > andrewlamb11@gmail.com
>> > >> >
>> > >> > >> wrote:
>> > >> > >>
>> > >> > >>> Got it. Thank you for the clarification -- I will try and look
>> > into
>> > >> the
>> > >> > >>> spec and the Rust implementation[1] in this next week
>> > >> > >>>
>> > >> > >>> [1]: https://github.com/apache/arrow-rs/pull/9372
>> > >> > >>>
>> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <
>> > >> > emkornfield@gmail.com>
>> > >> > >>> wrote:
>> > >> > >>>
>> > >> > >>>> Hi Andrew,
>> > >> > >>>> I think there is a fair amount of feedback on at least the
>> > >> > >>>> implementations, typically I think we've waited till they are
>> > >> close to
>> > >> > >>>> mergeable before a final vote. Otherwise I agree we are very
>> > >> close.
>> > >> > >>>>
>> > >> > >>>> -Micah
>> > >> > >>>>
>> > >> > >>>> On Saturday, April 25, 2026, Andrew Lamb <
>> andrewlamb11@gmail.com
>> > >
>> > >> > wrote:
>> > >> > >>>>
>> > >> > >>>>> Thanks Prateek,
>> > >> > >>>>>
>> > >> > >>>>> I think from this content it looks to me like we are ready to
>> > >> start a
>> > >> > >>>>> vote to explicitly accept ALP into Parquet
>> > >> > >>>>>
>> > >> > >>>>> Does anyone know of a reason we should postpone it for
>> longer?
>> > >> > >>>>> Perhaps someone needs some more time to review?
>> > >> > >>>>>
>> > >> > >>>>> Andrew
>> > >> > >>>>>
>> > >> > >>>>>
>> > >> > >>>>>
>> > >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <
>> > prtkgaur@gmail.com>
>> > >> > >>>>> wrote:
>> > >> > >>>>>
>> > >> > >>>>>> Hi team,
>> > >> > >>>>>>
>> > >> > >>>>>>
>> > >> > >>>>>>
>> > >> > >>>>>> Hope everyone is doing well. I got a chance to work through
>> all
>> > >> the
>> > >> > >>>>>> remaining feedback and update the spec doc. Here are the new
>> > >> > artifacts
>> > >> > >>>>>>
>> > >> > >>>>>> 1) Spec document :
>> > >> > >>>>>>
>> > >> >
>> > >>
>> >
>> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
>> > >> > >>>>>>
>> > >> > >>>>>> 2) Spec document in parquet format repo :
>> > >> > >>>>>> https://github.com/apache/parquet-format/pull/557
>> > >> > >>>>>>
>> > >> > >>>>>> 3) Alp implementation in arrow c++ repo :
>> > >> > >>>>>> https://github.com/apache/arrow/pull/48345/changes
>> > >> > >>>>>>
>> > >> > >>>>>> 4) Alp implementation in parquet-java repo : Work for Vinoo
>> and
>> > >> > Julien
>> > >> > >>>>>> https://github.com/apache/parquet-java/pull/3397
>> > >> > >>>>>>
>> > >> > >>>>>> 5) PR with test and benchmarking artifacts in
>> parquet-testing
>> > >> repo :
>> > >> > >>>>>> https://github.com/apache/parquet-testing/pull/100
>> > >> > >>>>>>
>> > >> > >>>>>>
>> > >> > >>>>>> And
>> > >> > >>>>>>
>> > >> > >>>>>>
>> > >> > >>>>>> - Go : Arnav just submitted an in progress
>> implementation
>> > in
>> > >> Go.
>> > >> > >>>>>> https://github.com/apache/arrow-go/pull/704 (I haven't
>> > >> started
>> > >> > >>>>>> looking at it yet)
>> > >> > >>>>>> - Rust : I remember Andrew mentioned that this work is
>> also
>> > >> in
>> > >> > >>>>>> progress (So 4 languages!)
>> > >> > >>>>>>
>> > >> > >>>>>>
>> > >> > >>>>>> *Arrow C++ implementation *
>> > >> > >>>>>>
>> > >> > >>>>>>
>> > >> > >>>>>>
>> > >> > >>>>>> The PR is out and was also used by Antoine to report the
>> > numbers
>> > >> as
>> > >> > >>>>>> reported here. Micah and Konstantin have given 1 round of
>> > >> feedback
>> > >> > >>>>>> and I'm addressing them today. Please note that the default
>> > >> > >>>>>> optimization flag for compiling is O2 and not Q3. I got
>> around
>> > >> 70%
>> > >> > >>>>>> performance improvement in the decoding speed when using
>> the O3
>> > >> > flag.
>> > >> > >>>>>>
>> > >> > >>>>>>
>> > >> > >>>>>>
>> > >> > >>>>>> *Parqet-MR Java implementation (working with Vinoo and
>> Julien)
>> > >> and
>> > >> > **Cross
>> > >> > >>>>>> Language testing*
>> > >> > >>>>>>
>> > >> > >>>>>>
>> > >> > >>>>>> Let me know if you have any questions or feedback.
>> > >> > >>>>>>
>> > >> > >>>>>>
>> > >> > >>>>>>
>> > >> > >>>>>> Now pasting some performance numbers
>> > >> > >>>>>>
>> > >> > >>>>>>
>> > >> > >>>>>> Table 1: C++ ALP Double Decode — Spotify Columns
>> (Graviton
>> > 3,
>> > >> ARM
>> > >> > >>>>>> Neoverse V1)
>> > >> > >>>>>>
>> > >> > >>>>>>
>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
>> > >> > >>>>>>
>> > >> > >>>>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │
>> Speedup │
>> > >> > >>>>>>
>> > >> > >>>>>>
>> ├──────────────────┼──────────────┼──────────────┼─────────┤
>> > >> > >>>>>>
>> > >> > >>>>>> │ valence │ 3,155 │ 5,523 │
>> 1.75x │
>> > >> > >>>>>>
>> > >> > >>>>>> │ danceability │ 3,233 │ 5,685 │
>> 1.76x │
>> > >> > >>>>>>
>> > >> > >>>>>> │ energy │ 3,197 │ 5,652 │
>> 1.77x │
>> > >> > >>>>>>
>> > >> > >>>>>> │ loudness │ 3,186 │ 5,473 │
>> 1.72x │
>> > >> > >>>>>>
>> > >> > >>>>>>
>> └──────────────────┴──────────────┴──────────────┴─────────┘
>> > >> > >>>>>>
>> > >> > >>>>>>
>> > >> > >>>>>>
>> > >> > >>>>>>
>> > >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <
>> > prtkgaur@gmail.com
>> > >> >
>> > >> > >>>>>> wrote:
>> > >> > >>>>>>
>> > >> > >>>>>>> @Micah Kornfield <em...@gmail.com> : Got it.
>> > >> > >>>>>>>
>> > >> > >>>>>>> @Andrew Lamb <an...@gmail.com>
>> > >> > >>>>>>>
>> > >> > >>>>>>>
>> > >> > >>>>>>>> Do you think it would be good to start moving the spec
>> > >> development
>> > >> > >>>>>>>> into
>> > >> > >>>>>>>> markdown format, in preparation for finalizing it?
>> > >> > >>>>>>>>
>> > >> > >>>>>>>
>> > >> > >>>>>>> Yes I'll update the numbers for some of the examples I
>> have in
>> > >> the
>> > >> > >>>>>>> spec based
>> > >> > >>>>>>> on the updated header size. Then we should be good to go
>> for
>> > the
>> > >> > >>>>>>> markdown format.
>> > >> > >>>>>>>
>> > >> > >>>>>>> Thanks everyone!
>> > >> > >>>>>>>
>> > >> > >>>>>>>
>> > >> > >>>>>>>>
>> > >> > >>>>>>>> Andrew
>> > >> > >>>>>>>>
>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <
>> > >> prtkgaur@gmail.com>
>> > >> > >>>>>>>> wrote:
>> > >> > >>>>>>>>
>> > >> > >>>>>>>>> Hi team,
>> > >> > >>>>>>>>>
>> > >> > >>>>>>>>> 1) Andrew
>> > >> > >>>>>>>>>
>> > >> > >>>>>>>>> - Thanks for working on test files. My PR did add all
>> > the
>> > >> > test
>> > >> > >>>>>>>> files I
>> > >> > >>>>>>>>> used to benchmark on datasets. Maybe we can club it
>> > >> together.
>> > >> > >>>>>>>> WIll also
>> > >> > >>>>>>>>> aid
>> > >> > >>>>>>>>> cross language testing
>> > >> > >>>>>>>>> - Kosta Tarasov working on Rust implementation.
>> This is
>> > >> > great.
>> > >> > >>>>>>>> Thanks
>> > >> > >>>>>>>>>
>> > >> > >>>>>>>>>
>> > >> > >>>>>>>>> 2) Antoine
>> > >> > >>>>>>>>>
>> > >> > >>>>>>>>> - Thanks a lot for reporting the numbers on AMD.
>> Looks
>> > >> like
>> > >> > you
>> > >> > >>>>>>>> are
>> > >> > >>>>>>>>> getting 8X the decoding performance of BSS. This is
>> > >> > amazing!!.
>> > >> > >>>>>>>>> - Thanks for acknowledging the sampling design.
>> > >> > >>>>>>>>> - I agree with you on Fastlanes. In some crude
>> > >> experiments I
>> > >> > >>>>>>>> didn't get
>> > >> > >>>>>>>>> a good perf benefit from it on Graviton3 (but maybe
>> > there
>> > >> was
>> > >> > >>>>>>>> something
>> > >> > >>>>>>>>> wrong with my implementation).
>> > >> > >>>>>>>>> - Locking the 16bit exception encoding for the spec
>> in
>> > >> this
>> > >> > >>>>>>>> case.
>> > >> > >>>>>>>>> - Awesome I think we have solved for all open
>> questions
>> > >> minus
>> > >> > >>>>>>>> the
>> > >> > >>>>>>>>> version byte :). (will get back on this soon)
>> > >> > >>>>>>>>>
>> > >> > >>>>>>>>>
>> > >> > >>>>>>>>> 3) Micah
>> > >> > >>>>>>>>>
>> > >> > >>>>>>>>> - FastLanes : The current spec does allow for using
>> > >> FastLane
>> > >> > >>>>>>>> with the
>> > >> > >>>>>>>>> configurable enum value for layout. We should be
>> able to
>> > >> > inject
>> > >> > >>>>>>>> any
>> > >> > >>>>>>>>> layout
>> > >> > >>>>>>>>> in the current design.
>> > >> > >>>>>>>>>
>> > >> > >>>>>>>>>
>> > >> > >>>>>>>>> Working on resolving all remaining open comments on the
>> spec
>> > >> this
>> > >> > >>>>>>>> week.
>> > >> > >>>>>>>>>
>> > >> > >>>>>>>>> Best
>> > >> > >>>>>>>>> Prateek
>> > >> > >>>>>>>>>
>> > >> > >>>>>>>>>
>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <
>> > >> > >>>>>>>> stevel@cloudera.com>
>> > >> > >>>>>>>>> wrote:
>> > >> > >>>>>>>>>
>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
>> > >> > >>>>>>>> emkornfield@gmail.com>
>> > >> > >>>>>>>>>> wrote:
>> > >> > >>>>>>>>>>
>> > >> > >>>>>>>>>>>
>> > >> > >>>>>>>>>>>
>> > >> > >>>>>>>>>>> It looks like the actual issue described for ORC in the
>> > >> paper
>> > >> > >>>>>>>> is that
>> > >> > >>>>>>>>> it
>> > >> > >>>>>>>>>>> has multiple sub-encodings in a batch. This is
>> different
>> > >> then
>> > >> > >>>>>>>> the
>> > >> > >>>>>>>>> design
>> > >> > >>>>>>>>>>> proposed here where there is still fixed encoding per
>> page
>> > >> in
>> > >> > >>>>>>>> parquet.
>> > >> > >>>>>>>>>>> Given reasonably sized pages I don't think branch
>> > >> > >>>>>>>> misprediction should
>> > >> > >>>>>>>>>> be a
>> > >> > >>>>>>>>>>> big issue for new encodings. I agree that we should be
>> > >> > >>>>>>>> conservative in
>> > >> > >>>>>>>>>>> general for adding new encodings.
>> > >> > >>>>>>>>>>>
>> > >> > >>>>>>>>>>>
>> > >> > >>>>>>>>>> +1
>> > >> > >>>>>>>>>>
>> > >> > >>>>>>>>>
>> > >> > >>>>>>>>
>> > >> > >>>>>>>
>> > >> > >
>> > >> >
>> > >> >
>> > >> >
>> > >>
>> > >
>> >
>>
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by PRATEEK GAUR <pr...@gmail.com>.
Thanks Matt,
I'll be on it this week.
Best
Prateek
On Mon, Jul 13, 2026 at 8:36 AM Matt Topol <zo...@gmail.com> wrote:
> There is also the Go implementation submitted by Arnav (
> https://github.com/apache/arrow-go/pull/704) which is waiting for updates.
>
> Just wanted to make sure it didn't get lost here.
>
> --Matt
>
> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK GAUR <pr...@gmail.com> wrote:
>
> > Hi all,
> >
> > I'd like to share a status update on the ALP encoding effort and get a
> feel
> > from the community before starting a formal vote.
> >
> > We now have the specification plus implementations in both languages:
> >
> > - Spec: apache/parquet-format#557 — GH-533 Add ALP encoding
> > specification
> > - C++ (Arrow): apache/arrow#48345 — GH-48701 [C++][Parquet] Add
> ALPpd
> > encoding
> > - Java: apache/parquet-java#3397 — Parquet Java ALP
> > Implementation (by Vinoo)
> >
> > Current state:
> >
> > - The C++ implementation has been through 3-4 rounds of review.
> > - The Java implementation has been tracking well and has had initial
> > reviews from contributors, with no major remaining issues.
> > - Cross-language compatibility tests are passing: the Arrow C++
> decoder
> > reads Java-written data bit-exactly across ~1.56M values and 18
> > fixtures, covering V1 and V2 pages, multiple vector sizes, and
> > several
> > real datasets — zero mismatches.
> >
> > As raised in the java-pr we will be adding coverage for extreme values
> > (those requiring
> > 63-64 bits after FOR is applied) before we close a vote. We'll aim to
> get
> > that
> > done in parallel.
> >
> > Plan: unless there are objections, I intend to start a vote at the end of
> > this week/early
> > next week.
> >
> > Thanks
> > Prateek
> >
> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <pr...@gmail.com> wrote:
> >
> > > Hi Team,
> > >
> > > Just wanted to provide some updates on ALP.
> > > Micah and I have done 3-4 rounds of review for the c++ PR :
> > > https://github.com/apache/arrow/pull/48345
> > >
> > > For the Java implementation Vinoo has been working on following PR :
> > > https://github.com/apache/parquet-java/pull/3397
> > >
> > > Best
> > > Prateek
> > >
> > > On Tue, May 5, 2026 at 1:50 PM Micah Kornfield <em...@gmail.com>
> > > wrote:
> > >
> > >> Hi Antoine,
> > >>
> > >> > Apologies if the question was already asked, but should we care
> about
> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> > >>
> > >> It was. We thought we could defer it for the following reasons:
> > >> 1. It's not clear there are a lot of easy reference datasets to test
> > its
> > >> effectiveness. It does look like there might be one or two on
> > huggingface
> > >> (e.g. https://huggingface.co/datasets/kikitora/curdie).
> > >> 2. It seemed likely that float 16 was more likely used for values
> that
> > >> were less likely to reduce to decimal values.
> > >> 3. It could be added as an extension later if needed.
> > >>
> > >> Cheers,
> > >> Micah
> > >>
> > >>
> > >> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <an...@python.org>
> > wrote:
> > >>
> > >> >
> > >> > Hello,
> > >> >
> > >> > Apologies if the question was already asked, but should we care
> about
> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> > >> >
> > >> > Regards
> > >> >
> > >> > Antoine.
> > >> >
> > >> >
> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
> > >> > > Thanks Andrew and Micah for review feedback on the two PR's
> > >> > > 1) (c++ arrow repo)
> > >> https://github.com/apache/arrow/pull/48345/changes
> > >> > > 2) (parquet-format repo)
> > >> > https://github.com/apache/parquet-format/pull/557
> > >> > >
> > >> > > I have addressed all (unless I missed something) comments on the
> two
> > >> > PR's.
> > >> > >
> > >> > > Best
> > >> > > Prateek
> > >> > >
> > >> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <pr...@gmail.com>
> > >> wrote:
> > >> > >
> > >> > >> Thanks Andrew and Micah.
> > >> > >>
> > >> > >> `fair amount of feedback on at least the implementations`
> > >> > >> For the c++ I have already started addressing the feedback, I
> > should
> > >> be
> > >> > >> done with that Monday/Tuesday.
> > >> > >> I think Vinoo too has been making good progress on the Java
> > >> > implementation.
> > >> > >>
> > >> > >> Best
> > >> > >> Prateek
> > >> > >>
> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <
> > andrewlamb11@gmail.com
> > >> >
> > >> > >> wrote:
> > >> > >>
> > >> > >>> Got it. Thank you for the clarification -- I will try and look
> > into
> > >> the
> > >> > >>> spec and the Rust implementation[1] in this next week
> > >> > >>>
> > >> > >>> [1]: https://github.com/apache/arrow-rs/pull/9372
> > >> > >>>
> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <
> > >> > emkornfield@gmail.com>
> > >> > >>> wrote:
> > >> > >>>
> > >> > >>>> Hi Andrew,
> > >> > >>>> I think there is a fair amount of feedback on at least the
> > >> > >>>> implementations, typically I think we've waited till they are
> > >> close to
> > >> > >>>> mergeable before a final vote. Otherwise I agree we are very
> > >> close.
> > >> > >>>>
> > >> > >>>> -Micah
> > >> > >>>>
> > >> > >>>> On Saturday, April 25, 2026, Andrew Lamb <
> andrewlamb11@gmail.com
> > >
> > >> > wrote:
> > >> > >>>>
> > >> > >>>>> Thanks Prateek,
> > >> > >>>>>
> > >> > >>>>> I think from this content it looks to me like we are ready to
> > >> start a
> > >> > >>>>> vote to explicitly accept ALP into Parquet
> > >> > >>>>>
> > >> > >>>>> Does anyone know of a reason we should postpone it for longer?
> > >> > >>>>> Perhaps someone needs some more time to review?
> > >> > >>>>>
> > >> > >>>>> Andrew
> > >> > >>>>>
> > >> > >>>>>
> > >> > >>>>>
> > >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <
> > prtkgaur@gmail.com>
> > >> > >>>>> wrote:
> > >> > >>>>>
> > >> > >>>>>> Hi team,
> > >> > >>>>>>
> > >> > >>>>>>
> > >> > >>>>>>
> > >> > >>>>>> Hope everyone is doing well. I got a chance to work through
> all
> > >> the
> > >> > >>>>>> remaining feedback and update the spec doc. Here are the new
> > >> > artifacts
> > >> > >>>>>>
> > >> > >>>>>> 1) Spec document :
> > >> > >>>>>>
> > >> >
> > >>
> >
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
> > >> > >>>>>>
> > >> > >>>>>> 2) Spec document in parquet format repo :
> > >> > >>>>>> https://github.com/apache/parquet-format/pull/557
> > >> > >>>>>>
> > >> > >>>>>> 3) Alp implementation in arrow c++ repo :
> > >> > >>>>>> https://github.com/apache/arrow/pull/48345/changes
> > >> > >>>>>>
> > >> > >>>>>> 4) Alp implementation in parquet-java repo : Work for Vinoo
> and
> > >> > Julien
> > >> > >>>>>> https://github.com/apache/parquet-java/pull/3397
> > >> > >>>>>>
> > >> > >>>>>> 5) PR with test and benchmarking artifacts in parquet-testing
> > >> repo :
> > >> > >>>>>> https://github.com/apache/parquet-testing/pull/100
> > >> > >>>>>>
> > >> > >>>>>>
> > >> > >>>>>> And
> > >> > >>>>>>
> > >> > >>>>>>
> > >> > >>>>>> - Go : Arnav just submitted an in progress implementation
> > in
> > >> Go.
> > >> > >>>>>> https://github.com/apache/arrow-go/pull/704 (I haven't
> > >> started
> > >> > >>>>>> looking at it yet)
> > >> > >>>>>> - Rust : I remember Andrew mentioned that this work is
> also
> > >> in
> > >> > >>>>>> progress (So 4 languages!)
> > >> > >>>>>>
> > >> > >>>>>>
> > >> > >>>>>> *Arrow C++ implementation *
> > >> > >>>>>>
> > >> > >>>>>>
> > >> > >>>>>>
> > >> > >>>>>> The PR is out and was also used by Antoine to report the
> > numbers
> > >> as
> > >> > >>>>>> reported here. Micah and Konstantin have given 1 round of
> > >> feedback
> > >> > >>>>>> and I'm addressing them today. Please note that the default
> > >> > >>>>>> optimization flag for compiling is O2 and not Q3. I got
> around
> > >> 70%
> > >> > >>>>>> performance improvement in the decoding speed when using the
> O3
> > >> > flag.
> > >> > >>>>>>
> > >> > >>>>>>
> > >> > >>>>>>
> > >> > >>>>>> *Parqet-MR Java implementation (working with Vinoo and
> Julien)
> > >> and
> > >> > **Cross
> > >> > >>>>>> Language testing*
> > >> > >>>>>>
> > >> > >>>>>>
> > >> > >>>>>> Let me know if you have any questions or feedback.
> > >> > >>>>>>
> > >> > >>>>>>
> > >> > >>>>>>
> > >> > >>>>>> Now pasting some performance numbers
> > >> > >>>>>>
> > >> > >>>>>>
> > >> > >>>>>> Table 1: C++ ALP Double Decode — Spotify Columns (Graviton
> > 3,
> > >> ARM
> > >> > >>>>>> Neoverse V1)
> > >> > >>>>>>
> > >> > >>>>>>
> ┌──────────────────┬──────────────┬──────────────┬─────────┐
> > >> > >>>>>>
> > >> > >>>>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │
> Speedup │
> > >> > >>>>>>
> > >> > >>>>>>
> ├──────────────────┼──────────────┼──────────────┼─────────┤
> > >> > >>>>>>
> > >> > >>>>>> │ valence │ 3,155 │ 5,523 │
> 1.75x │
> > >> > >>>>>>
> > >> > >>>>>> │ danceability │ 3,233 │ 5,685 │
> 1.76x │
> > >> > >>>>>>
> > >> > >>>>>> │ energy │ 3,197 │ 5,652 │
> 1.77x │
> > >> > >>>>>>
> > >> > >>>>>> │ loudness │ 3,186 │ 5,473 │
> 1.72x │
> > >> > >>>>>>
> > >> > >>>>>>
> └──────────────────┴──────────────┴──────────────┴─────────┘
> > >> > >>>>>>
> > >> > >>>>>>
> > >> > >>>>>>
> > >> > >>>>>>
> > >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <
> > prtkgaur@gmail.com
> > >> >
> > >> > >>>>>> wrote:
> > >> > >>>>>>
> > >> > >>>>>>> @Micah Kornfield <em...@gmail.com> : Got it.
> > >> > >>>>>>>
> > >> > >>>>>>> @Andrew Lamb <an...@gmail.com>
> > >> > >>>>>>>
> > >> > >>>>>>>
> > >> > >>>>>>>> Do you think it would be good to start moving the spec
> > >> development
> > >> > >>>>>>>> into
> > >> > >>>>>>>> markdown format, in preparation for finalizing it?
> > >> > >>>>>>>>
> > >> > >>>>>>>
> > >> > >>>>>>> Yes I'll update the numbers for some of the examples I have
> in
> > >> the
> > >> > >>>>>>> spec based
> > >> > >>>>>>> on the updated header size. Then we should be good to go for
> > the
> > >> > >>>>>>> markdown format.
> > >> > >>>>>>>
> > >> > >>>>>>> Thanks everyone!
> > >> > >>>>>>>
> > >> > >>>>>>>
> > >> > >>>>>>>>
> > >> > >>>>>>>> Andrew
> > >> > >>>>>>>>
> > >> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <
> > >> prtkgaur@gmail.com>
> > >> > >>>>>>>> wrote:
> > >> > >>>>>>>>
> > >> > >>>>>>>>> Hi team,
> > >> > >>>>>>>>>
> > >> > >>>>>>>>> 1) Andrew
> > >> > >>>>>>>>>
> > >> > >>>>>>>>> - Thanks for working on test files. My PR did add all
> > the
> > >> > test
> > >> > >>>>>>>> files I
> > >> > >>>>>>>>> used to benchmark on datasets. Maybe we can club it
> > >> together.
> > >> > >>>>>>>> WIll also
> > >> > >>>>>>>>> aid
> > >> > >>>>>>>>> cross language testing
> > >> > >>>>>>>>> - Kosta Tarasov working on Rust implementation. This
> is
> > >> > great.
> > >> > >>>>>>>> Thanks
> > >> > >>>>>>>>>
> > >> > >>>>>>>>>
> > >> > >>>>>>>>> 2) Antoine
> > >> > >>>>>>>>>
> > >> > >>>>>>>>> - Thanks a lot for reporting the numbers on AMD. Looks
> > >> like
> > >> > you
> > >> > >>>>>>>> are
> > >> > >>>>>>>>> getting 8X the decoding performance of BSS. This is
> > >> > amazing!!.
> > >> > >>>>>>>>> - Thanks for acknowledging the sampling design.
> > >> > >>>>>>>>> - I agree with you on Fastlanes. In some crude
> > >> experiments I
> > >> > >>>>>>>> didn't get
> > >> > >>>>>>>>> a good perf benefit from it on Graviton3 (but maybe
> > there
> > >> was
> > >> > >>>>>>>> something
> > >> > >>>>>>>>> wrong with my implementation).
> > >> > >>>>>>>>> - Locking the 16bit exception encoding for the spec in
> > >> this
> > >> > >>>>>>>> case.
> > >> > >>>>>>>>> - Awesome I think we have solved for all open
> questions
> > >> minus
> > >> > >>>>>>>> the
> > >> > >>>>>>>>> version byte :). (will get back on this soon)
> > >> > >>>>>>>>>
> > >> > >>>>>>>>>
> > >> > >>>>>>>>> 3) Micah
> > >> > >>>>>>>>>
> > >> > >>>>>>>>> - FastLanes : The current spec does allow for using
> > >> FastLane
> > >> > >>>>>>>> with the
> > >> > >>>>>>>>> configurable enum value for layout. We should be able
> to
> > >> > inject
> > >> > >>>>>>>> any
> > >> > >>>>>>>>> layout
> > >> > >>>>>>>>> in the current design.
> > >> > >>>>>>>>>
> > >> > >>>>>>>>>
> > >> > >>>>>>>>> Working on resolving all remaining open comments on the
> spec
> > >> this
> > >> > >>>>>>>> week.
> > >> > >>>>>>>>>
> > >> > >>>>>>>>> Best
> > >> > >>>>>>>>> Prateek
> > >> > >>>>>>>>>
> > >> > >>>>>>>>>
> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <
> > >> > >>>>>>>> stevel@cloudera.com>
> > >> > >>>>>>>>> wrote:
> > >> > >>>>>>>>>
> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
> > >> > >>>>>>>> emkornfield@gmail.com>
> > >> > >>>>>>>>>> wrote:
> > >> > >>>>>>>>>>
> > >> > >>>>>>>>>>>
> > >> > >>>>>>>>>>>
> > >> > >>>>>>>>>>> It looks like the actual issue described for ORC in the
> > >> paper
> > >> > >>>>>>>> is that
> > >> > >>>>>>>>> it
> > >> > >>>>>>>>>>> has multiple sub-encodings in a batch. This is
> different
> > >> then
> > >> > >>>>>>>> the
> > >> > >>>>>>>>> design
> > >> > >>>>>>>>>>> proposed here where there is still fixed encoding per
> page
> > >> in
> > >> > >>>>>>>> parquet.
> > >> > >>>>>>>>>>> Given reasonably sized pages I don't think branch
> > >> > >>>>>>>> misprediction should
> > >> > >>>>>>>>>> be a
> > >> > >>>>>>>>>>> big issue for new encodings. I agree that we should be
> > >> > >>>>>>>> conservative in
> > >> > >>>>>>>>>>> general for adding new encodings.
> > >> > >>>>>>>>>>>
> > >> > >>>>>>>>>>>
> > >> > >>>>>>>>>> +1
> > >> > >>>>>>>>>>
> > >> > >>>>>>>>>
> > >> > >>>>>>>>
> > >> > >>>>>>>
> > >> > >
> > >> >
> > >> >
> > >> >
> > >>
> > >
> >
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Matt Topol <zo...@gmail.com>.
There is also the Go implementation submitted by Arnav (
https://github.com/apache/arrow-go/pull/704) which is waiting for updates.
Just wanted to make sure it didn't get lost here.
--Matt
On Mon, Jul 13, 2026 at 11:30 AM PRATEEK GAUR <pr...@gmail.com> wrote:
> Hi all,
>
> I'd like to share a status update on the ALP encoding effort and get a feel
> from the community before starting a formal vote.
>
> We now have the specification plus implementations in both languages:
>
> - Spec: apache/parquet-format#557 — GH-533 Add ALP encoding
> specification
> - C++ (Arrow): apache/arrow#48345 — GH-48701 [C++][Parquet] Add ALPpd
> encoding
> - Java: apache/parquet-java#3397 — Parquet Java ALP
> Implementation (by Vinoo)
>
> Current state:
>
> - The C++ implementation has been through 3-4 rounds of review.
> - The Java implementation has been tracking well and has had initial
> reviews from contributors, with no major remaining issues.
> - Cross-language compatibility tests are passing: the Arrow C++ decoder
> reads Java-written data bit-exactly across ~1.56M values and 18
> fixtures, covering V1 and V2 pages, multiple vector sizes, and
> several
> real datasets — zero mismatches.
>
> As raised in the java-pr we will be adding coverage for extreme values
> (those requiring
> 63-64 bits after FOR is applied) before we close a vote. We'll aim to get
> that
> done in parallel.
>
> Plan: unless there are objections, I intend to start a vote at the end of
> this week/early
> next week.
>
> Thanks
> Prateek
>
> On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <pr...@gmail.com> wrote:
>
> > Hi Team,
> >
> > Just wanted to provide some updates on ALP.
> > Micah and I have done 3-4 rounds of review for the c++ PR :
> > https://github.com/apache/arrow/pull/48345
> >
> > For the Java implementation Vinoo has been working on following PR :
> > https://github.com/apache/parquet-java/pull/3397
> >
> > Best
> > Prateek
> >
> > On Tue, May 5, 2026 at 1:50 PM Micah Kornfield <em...@gmail.com>
> > wrote:
> >
> >> Hi Antoine,
> >>
> >> > Apologies if the question was already asked, but should we care about
> >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
> >> > BYTE_STREAM_SPLIT + LZ4 for example?
> >>
> >> It was. We thought we could defer it for the following reasons:
> >> 1. It's not clear there are a lot of easy reference datasets to test
> its
> >> effectiveness. It does look like there might be one or two on
> huggingface
> >> (e.g. https://huggingface.co/datasets/kikitora/curdie).
> >> 2. It seemed likely that float 16 was more likely used for values that
> >> were less likely to reduce to decimal values.
> >> 3. It could be added as an extension later if needed.
> >>
> >> Cheers,
> >> Micah
> >>
> >>
> >> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <an...@python.org>
> wrote:
> >>
> >> >
> >> > Hello,
> >> >
> >> > Apologies if the question was already asked, but should we care about
> >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
> >> > BYTE_STREAM_SPLIT + LZ4 for example?
> >> >
> >> > Regards
> >> >
> >> > Antoine.
> >> >
> >> >
> >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
> >> > > Thanks Andrew and Micah for review feedback on the two PR's
> >> > > 1) (c++ arrow repo)
> >> https://github.com/apache/arrow/pull/48345/changes
> >> > > 2) (parquet-format repo)
> >> > https://github.com/apache/parquet-format/pull/557
> >> > >
> >> > > I have addressed all (unless I missed something) comments on the two
> >> > PR's.
> >> > >
> >> > > Best
> >> > > Prateek
> >> > >
> >> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <pr...@gmail.com>
> >> wrote:
> >> > >
> >> > >> Thanks Andrew and Micah.
> >> > >>
> >> > >> `fair amount of feedback on at least the implementations`
> >> > >> For the c++ I have already started addressing the feedback, I
> should
> >> be
> >> > >> done with that Monday/Tuesday.
> >> > >> I think Vinoo too has been making good progress on the Java
> >> > implementation.
> >> > >>
> >> > >> Best
> >> > >> Prateek
> >> > >>
> >> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <
> andrewlamb11@gmail.com
> >> >
> >> > >> wrote:
> >> > >>
> >> > >>> Got it. Thank you for the clarification -- I will try and look
> into
> >> the
> >> > >>> spec and the Rust implementation[1] in this next week
> >> > >>>
> >> > >>> [1]: https://github.com/apache/arrow-rs/pull/9372
> >> > >>>
> >> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <
> >> > emkornfield@gmail.com>
> >> > >>> wrote:
> >> > >>>
> >> > >>>> Hi Andrew,
> >> > >>>> I think there is a fair amount of feedback on at least the
> >> > >>>> implementations, typically I think we've waited till they are
> >> close to
> >> > >>>> mergeable before a final vote. Otherwise I agree we are very
> >> close.
> >> > >>>>
> >> > >>>> -Micah
> >> > >>>>
> >> > >>>> On Saturday, April 25, 2026, Andrew Lamb <andrewlamb11@gmail.com
> >
> >> > wrote:
> >> > >>>>
> >> > >>>>> Thanks Prateek,
> >> > >>>>>
> >> > >>>>> I think from this content it looks to me like we are ready to
> >> start a
> >> > >>>>> vote to explicitly accept ALP into Parquet
> >> > >>>>>
> >> > >>>>> Does anyone know of a reason we should postpone it for longer?
> >> > >>>>> Perhaps someone needs some more time to review?
> >> > >>>>>
> >> > >>>>> Andrew
> >> > >>>>>
> >> > >>>>>
> >> > >>>>>
> >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <
> prtkgaur@gmail.com>
> >> > >>>>> wrote:
> >> > >>>>>
> >> > >>>>>> Hi team,
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>> Hope everyone is doing well. I got a chance to work through all
> >> the
> >> > >>>>>> remaining feedback and update the spec doc. Here are the new
> >> > artifacts
> >> > >>>>>>
> >> > >>>>>> 1) Spec document :
> >> > >>>>>>
> >> >
> >>
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
> >> > >>>>>>
> >> > >>>>>> 2) Spec document in parquet format repo :
> >> > >>>>>> https://github.com/apache/parquet-format/pull/557
> >> > >>>>>>
> >> > >>>>>> 3) Alp implementation in arrow c++ repo :
> >> > >>>>>> https://github.com/apache/arrow/pull/48345/changes
> >> > >>>>>>
> >> > >>>>>> 4) Alp implementation in parquet-java repo : Work for Vinoo and
> >> > Julien
> >> > >>>>>> https://github.com/apache/parquet-java/pull/3397
> >> > >>>>>>
> >> > >>>>>> 5) PR with test and benchmarking artifacts in parquet-testing
> >> repo :
> >> > >>>>>> https://github.com/apache/parquet-testing/pull/100
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>> And
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>> - Go : Arnav just submitted an in progress implementation
> in
> >> Go.
> >> > >>>>>> https://github.com/apache/arrow-go/pull/704 (I haven't
> >> started
> >> > >>>>>> looking at it yet)
> >> > >>>>>> - Rust : I remember Andrew mentioned that this work is also
> >> in
> >> > >>>>>> progress (So 4 languages!)
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>> *Arrow C++ implementation *
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>> The PR is out and was also used by Antoine to report the
> numbers
> >> as
> >> > >>>>>> reported here. Micah and Konstantin have given 1 round of
> >> feedback
> >> > >>>>>> and I'm addressing them today. Please note that the default
> >> > >>>>>> optimization flag for compiling is O2 and not Q3. I got around
> >> 70%
> >> > >>>>>> performance improvement in the decoding speed when using the O3
> >> > flag.
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>> *Parqet-MR Java implementation (working with Vinoo and Julien)
> >> and
> >> > **Cross
> >> > >>>>>> Language testing*
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>> Let me know if you have any questions or feedback.
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>> Now pasting some performance numbers
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>> Table 1: C++ ALP Double Decode — Spotify Columns (Graviton
> 3,
> >> ARM
> >> > >>>>>> Neoverse V1)
> >> > >>>>>>
> >> > >>>>>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
> >> > >>>>>>
> >> > >>>>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │ Speedup │
> >> > >>>>>>
> >> > >>>>>> ├──────────────────┼──────────────┼──────────────┼─────────┤
> >> > >>>>>>
> >> > >>>>>> │ valence │ 3,155 │ 5,523 │ 1.75x │
> >> > >>>>>>
> >> > >>>>>> │ danceability │ 3,233 │ 5,685 │ 1.76x │
> >> > >>>>>>
> >> > >>>>>> │ energy │ 3,197 │ 5,652 │ 1.77x │
> >> > >>>>>>
> >> > >>>>>> │ loudness │ 3,186 │ 5,473 │ 1.72x │
> >> > >>>>>>
> >> > >>>>>> └──────────────────┴──────────────┴──────────────┴─────────┘
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <
> prtkgaur@gmail.com
> >> >
> >> > >>>>>> wrote:
> >> > >>>>>>
> >> > >>>>>>> @Micah Kornfield <em...@gmail.com> : Got it.
> >> > >>>>>>>
> >> > >>>>>>> @Andrew Lamb <an...@gmail.com>
> >> > >>>>>>>
> >> > >>>>>>>
> >> > >>>>>>>> Do you think it would be good to start moving the spec
> >> development
> >> > >>>>>>>> into
> >> > >>>>>>>> markdown format, in preparation for finalizing it?
> >> > >>>>>>>>
> >> > >>>>>>>
> >> > >>>>>>> Yes I'll update the numbers for some of the examples I have in
> >> the
> >> > >>>>>>> spec based
> >> > >>>>>>> on the updated header size. Then we should be good to go for
> the
> >> > >>>>>>> markdown format.
> >> > >>>>>>>
> >> > >>>>>>> Thanks everyone!
> >> > >>>>>>>
> >> > >>>>>>>
> >> > >>>>>>>>
> >> > >>>>>>>> Andrew
> >> > >>>>>>>>
> >> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <
> >> prtkgaur@gmail.com>
> >> > >>>>>>>> wrote:
> >> > >>>>>>>>
> >> > >>>>>>>>> Hi team,
> >> > >>>>>>>>>
> >> > >>>>>>>>> 1) Andrew
> >> > >>>>>>>>>
> >> > >>>>>>>>> - Thanks for working on test files. My PR did add all
> the
> >> > test
> >> > >>>>>>>> files I
> >> > >>>>>>>>> used to benchmark on datasets. Maybe we can club it
> >> together.
> >> > >>>>>>>> WIll also
> >> > >>>>>>>>> aid
> >> > >>>>>>>>> cross language testing
> >> > >>>>>>>>> - Kosta Tarasov working on Rust implementation. This is
> >> > great.
> >> > >>>>>>>> Thanks
> >> > >>>>>>>>>
> >> > >>>>>>>>>
> >> > >>>>>>>>> 2) Antoine
> >> > >>>>>>>>>
> >> > >>>>>>>>> - Thanks a lot for reporting the numbers on AMD. Looks
> >> like
> >> > you
> >> > >>>>>>>> are
> >> > >>>>>>>>> getting 8X the decoding performance of BSS. This is
> >> > amazing!!.
> >> > >>>>>>>>> - Thanks for acknowledging the sampling design.
> >> > >>>>>>>>> - I agree with you on Fastlanes. In some crude
> >> experiments I
> >> > >>>>>>>> didn't get
> >> > >>>>>>>>> a good perf benefit from it on Graviton3 (but maybe
> there
> >> was
> >> > >>>>>>>> something
> >> > >>>>>>>>> wrong with my implementation).
> >> > >>>>>>>>> - Locking the 16bit exception encoding for the spec in
> >> this
> >> > >>>>>>>> case.
> >> > >>>>>>>>> - Awesome I think we have solved for all open questions
> >> minus
> >> > >>>>>>>> the
> >> > >>>>>>>>> version byte :). (will get back on this soon)
> >> > >>>>>>>>>
> >> > >>>>>>>>>
> >> > >>>>>>>>> 3) Micah
> >> > >>>>>>>>>
> >> > >>>>>>>>> - FastLanes : The current spec does allow for using
> >> FastLane
> >> > >>>>>>>> with the
> >> > >>>>>>>>> configurable enum value for layout. We should be able to
> >> > inject
> >> > >>>>>>>> any
> >> > >>>>>>>>> layout
> >> > >>>>>>>>> in the current design.
> >> > >>>>>>>>>
> >> > >>>>>>>>>
> >> > >>>>>>>>> Working on resolving all remaining open comments on the spec
> >> this
> >> > >>>>>>>> week.
> >> > >>>>>>>>>
> >> > >>>>>>>>> Best
> >> > >>>>>>>>> Prateek
> >> > >>>>>>>>>
> >> > >>>>>>>>>
> >> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <
> >> > >>>>>>>> stevel@cloudera.com>
> >> > >>>>>>>>> wrote:
> >> > >>>>>>>>>
> >> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
> >> > >>>>>>>> emkornfield@gmail.com>
> >> > >>>>>>>>>> wrote:
> >> > >>>>>>>>>>
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>> It looks like the actual issue described for ORC in the
> >> paper
> >> > >>>>>>>> is that
> >> > >>>>>>>>> it
> >> > >>>>>>>>>>> has multiple sub-encodings in a batch. This is different
> >> then
> >> > >>>>>>>> the
> >> > >>>>>>>>> design
> >> > >>>>>>>>>>> proposed here where there is still fixed encoding per page
> >> in
> >> > >>>>>>>> parquet.
> >> > >>>>>>>>>>> Given reasonably sized pages I don't think branch
> >> > >>>>>>>> misprediction should
> >> > >>>>>>>>>> be a
> >> > >>>>>>>>>>> big issue for new encodings. I agree that we should be
> >> > >>>>>>>> conservative in
> >> > >>>>>>>>>>> general for adding new encodings.
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>> +1
> >> > >>>>>>>>>>
> >> > >>>>>>>>>
> >> > >>>>>>>>
> >> > >>>>>>>
> >> > >
> >> >
> >> >
> >> >
> >>
> >
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by PRATEEK GAUR <pr...@gmail.com>.
Hi all,
I'd like to share a status update on the ALP encoding effort and get a feel
from the community before starting a formal vote.
We now have the specification plus implementations in both languages:
- Spec: apache/parquet-format#557 — GH-533 Add ALP encoding
specification
- C++ (Arrow): apache/arrow#48345 — GH-48701 [C++][Parquet] Add ALPpd
encoding
- Java: apache/parquet-java#3397 — Parquet Java ALP
Implementation (by Vinoo)
Current state:
- The C++ implementation has been through 3-4 rounds of review.
- The Java implementation has been tracking well and has had initial
reviews from contributors, with no major remaining issues.
- Cross-language compatibility tests are passing: the Arrow C++ decoder
reads Java-written data bit-exactly across ~1.56M values and 18
fixtures, covering V1 and V2 pages, multiple vector sizes, and several
real datasets — zero mismatches.
As raised in the java-pr we will be adding coverage for extreme values
(those requiring
63-64 bits after FOR is applied) before we close a vote. We'll aim to get
that
done in parallel.
Plan: unless there are objections, I intend to start a vote at the end of
this week/early
next week.
Thanks
Prateek
On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <pr...@gmail.com> wrote:
> Hi Team,
>
> Just wanted to provide some updates on ALP.
> Micah and I have done 3-4 rounds of review for the c++ PR :
> https://github.com/apache/arrow/pull/48345
>
> For the Java implementation Vinoo has been working on following PR :
> https://github.com/apache/parquet-java/pull/3397
>
> Best
> Prateek
>
> On Tue, May 5, 2026 at 1:50 PM Micah Kornfield <em...@gmail.com>
> wrote:
>
>> Hi Antoine,
>>
>> > Apologies if the question was already asked, but should we care about
>> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
>> > BYTE_STREAM_SPLIT + LZ4 for example?
>>
>> It was. We thought we could defer it for the following reasons:
>> 1. It's not clear there are a lot of easy reference datasets to test its
>> effectiveness. It does look like there might be one or two on huggingface
>> (e.g. https://huggingface.co/datasets/kikitora/curdie).
>> 2. It seemed likely that float 16 was more likely used for values that
>> were less likely to reduce to decimal values.
>> 3. It could be added as an extension later if needed.
>>
>> Cheers,
>> Micah
>>
>>
>> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <an...@python.org> wrote:
>>
>> >
>> > Hello,
>> >
>> > Apologies if the question was already asked, but should we care about
>> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
>> > BYTE_STREAM_SPLIT + LZ4 for example?
>> >
>> > Regards
>> >
>> > Antoine.
>> >
>> >
>> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
>> > > Thanks Andrew and Micah for review feedback on the two PR's
>> > > 1) (c++ arrow repo)
>> https://github.com/apache/arrow/pull/48345/changes
>> > > 2) (parquet-format repo)
>> > https://github.com/apache/parquet-format/pull/557
>> > >
>> > > I have addressed all (unless I missed something) comments on the two
>> > PR's.
>> > >
>> > > Best
>> > > Prateek
>> > >
>> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <pr...@gmail.com>
>> wrote:
>> > >
>> > >> Thanks Andrew and Micah.
>> > >>
>> > >> `fair amount of feedback on at least the implementations`
>> > >> For the c++ I have already started addressing the feedback, I should
>> be
>> > >> done with that Monday/Tuesday.
>> > >> I think Vinoo too has been making good progress on the Java
>> > implementation.
>> > >>
>> > >> Best
>> > >> Prateek
>> > >>
>> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <andrewlamb11@gmail.com
>> >
>> > >> wrote:
>> > >>
>> > >>> Got it. Thank you for the clarification -- I will try and look into
>> the
>> > >>> spec and the Rust implementation[1] in this next week
>> > >>>
>> > >>> [1]: https://github.com/apache/arrow-rs/pull/9372
>> > >>>
>> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <
>> > emkornfield@gmail.com>
>> > >>> wrote:
>> > >>>
>> > >>>> Hi Andrew,
>> > >>>> I think there is a fair amount of feedback on at least the
>> > >>>> implementations, typically I think we've waited till they are
>> close to
>> > >>>> mergeable before a final vote. Otherwise I agree we are very
>> close.
>> > >>>>
>> > >>>> -Micah
>> > >>>>
>> > >>>> On Saturday, April 25, 2026, Andrew Lamb <an...@gmail.com>
>> > wrote:
>> > >>>>
>> > >>>>> Thanks Prateek,
>> > >>>>>
>> > >>>>> I think from this content it looks to me like we are ready to
>> start a
>> > >>>>> vote to explicitly accept ALP into Parquet
>> > >>>>>
>> > >>>>> Does anyone know of a reason we should postpone it for longer?
>> > >>>>> Perhaps someone needs some more time to review?
>> > >>>>>
>> > >>>>> Andrew
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <pr...@gmail.com>
>> > >>>>> wrote:
>> > >>>>>
>> > >>>>>> Hi team,
>> > >>>>>>
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> Hope everyone is doing well. I got a chance to work through all
>> the
>> > >>>>>> remaining feedback and update the spec doc. Here are the new
>> > artifacts
>> > >>>>>>
>> > >>>>>> 1) Spec document :
>> > >>>>>>
>> >
>> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
>> > >>>>>>
>> > >>>>>> 2) Spec document in parquet format repo :
>> > >>>>>> https://github.com/apache/parquet-format/pull/557
>> > >>>>>>
>> > >>>>>> 3) Alp implementation in arrow c++ repo :
>> > >>>>>> https://github.com/apache/arrow/pull/48345/changes
>> > >>>>>>
>> > >>>>>> 4) Alp implementation in parquet-java repo : Work for Vinoo and
>> > Julien
>> > >>>>>> https://github.com/apache/parquet-java/pull/3397
>> > >>>>>>
>> > >>>>>> 5) PR with test and benchmarking artifacts in parquet-testing
>> repo :
>> > >>>>>> https://github.com/apache/parquet-testing/pull/100
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> And
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> - Go : Arnav just submitted an in progress implementation in
>> Go.
>> > >>>>>> https://github.com/apache/arrow-go/pull/704 (I haven't
>> started
>> > >>>>>> looking at it yet)
>> > >>>>>> - Rust : I remember Andrew mentioned that this work is also
>> in
>> > >>>>>> progress (So 4 languages!)
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> *Arrow C++ implementation *
>> > >>>>>>
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> The PR is out and was also used by Antoine to report the numbers
>> as
>> > >>>>>> reported here. Micah and Konstantin have given 1 round of
>> feedback
>> > >>>>>> and I'm addressing them today. Please note that the default
>> > >>>>>> optimization flag for compiling is O2 and not Q3. I got around
>> 70%
>> > >>>>>> performance improvement in the decoding speed when using the O3
>> > flag.
>> > >>>>>>
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> *Parqet-MR Java implementation (working with Vinoo and Julien)
>> and
>> > **Cross
>> > >>>>>> Language testing*
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> Let me know if you have any questions or feedback.
>> > >>>>>>
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> Now pasting some performance numbers
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> Table 1: C++ ALP Double Decode — Spotify Columns (Graviton 3,
>> ARM
>> > >>>>>> Neoverse V1)
>> > >>>>>>
>> > >>>>>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
>> > >>>>>>
>> > >>>>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │ Speedup │
>> > >>>>>>
>> > >>>>>> ├──────────────────┼──────────────┼──────────────┼─────────┤
>> > >>>>>>
>> > >>>>>> │ valence │ 3,155 │ 5,523 │ 1.75x │
>> > >>>>>>
>> > >>>>>> │ danceability │ 3,233 │ 5,685 │ 1.76x │
>> > >>>>>>
>> > >>>>>> │ energy │ 3,197 │ 5,652 │ 1.77x │
>> > >>>>>>
>> > >>>>>> │ loudness │ 3,186 │ 5,473 │ 1.72x │
>> > >>>>>>
>> > >>>>>> └──────────────────┴──────────────┴──────────────┴─────────┘
>> > >>>>>>
>> > >>>>>>
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <prtkgaur@gmail.com
>> >
>> > >>>>>> wrote:
>> > >>>>>>
>> > >>>>>>> @Micah Kornfield <em...@gmail.com> : Got it.
>> > >>>>>>>
>> > >>>>>>> @Andrew Lamb <an...@gmail.com>
>> > >>>>>>>
>> > >>>>>>>
>> > >>>>>>>> Do you think it would be good to start moving the spec
>> development
>> > >>>>>>>> into
>> > >>>>>>>> markdown format, in preparation for finalizing it?
>> > >>>>>>>>
>> > >>>>>>>
>> > >>>>>>> Yes I'll update the numbers for some of the examples I have in
>> the
>> > >>>>>>> spec based
>> > >>>>>>> on the updated header size. Then we should be good to go for the
>> > >>>>>>> markdown format.
>> > >>>>>>>
>> > >>>>>>> Thanks everyone!
>> > >>>>>>>
>> > >>>>>>>
>> > >>>>>>>>
>> > >>>>>>>> Andrew
>> > >>>>>>>>
>> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <
>> prtkgaur@gmail.com>
>> > >>>>>>>> wrote:
>> > >>>>>>>>
>> > >>>>>>>>> Hi team,
>> > >>>>>>>>>
>> > >>>>>>>>> 1) Andrew
>> > >>>>>>>>>
>> > >>>>>>>>> - Thanks for working on test files. My PR did add all the
>> > test
>> > >>>>>>>> files I
>> > >>>>>>>>> used to benchmark on datasets. Maybe we can club it
>> together.
>> > >>>>>>>> WIll also
>> > >>>>>>>>> aid
>> > >>>>>>>>> cross language testing
>> > >>>>>>>>> - Kosta Tarasov working on Rust implementation. This is
>> > great.
>> > >>>>>>>> Thanks
>> > >>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>> 2) Antoine
>> > >>>>>>>>>
>> > >>>>>>>>> - Thanks a lot for reporting the numbers on AMD. Looks
>> like
>> > you
>> > >>>>>>>> are
>> > >>>>>>>>> getting 8X the decoding performance of BSS. This is
>> > amazing!!.
>> > >>>>>>>>> - Thanks for acknowledging the sampling design.
>> > >>>>>>>>> - I agree with you on Fastlanes. In some crude
>> experiments I
>> > >>>>>>>> didn't get
>> > >>>>>>>>> a good perf benefit from it on Graviton3 (but maybe there
>> was
>> > >>>>>>>> something
>> > >>>>>>>>> wrong with my implementation).
>> > >>>>>>>>> - Locking the 16bit exception encoding for the spec in
>> this
>> > >>>>>>>> case.
>> > >>>>>>>>> - Awesome I think we have solved for all open questions
>> minus
>> > >>>>>>>> the
>> > >>>>>>>>> version byte :). (will get back on this soon)
>> > >>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>> 3) Micah
>> > >>>>>>>>>
>> > >>>>>>>>> - FastLanes : The current spec does allow for using
>> FastLane
>> > >>>>>>>> with the
>> > >>>>>>>>> configurable enum value for layout. We should be able to
>> > inject
>> > >>>>>>>> any
>> > >>>>>>>>> layout
>> > >>>>>>>>> in the current design.
>> > >>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>> Working on resolving all remaining open comments on the spec
>> this
>> > >>>>>>>> week.
>> > >>>>>>>>>
>> > >>>>>>>>> Best
>> > >>>>>>>>> Prateek
>> > >>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <
>> > >>>>>>>> stevel@cloudera.com>
>> > >>>>>>>>> wrote:
>> > >>>>>>>>>
>> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
>> > >>>>>>>> emkornfield@gmail.com>
>> > >>>>>>>>>> wrote:
>> > >>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> It looks like the actual issue described for ORC in the
>> paper
>> > >>>>>>>> is that
>> > >>>>>>>>> it
>> > >>>>>>>>>>> has multiple sub-encodings in a batch. This is different
>> then
>> > >>>>>>>> the
>> > >>>>>>>>> design
>> > >>>>>>>>>>> proposed here where there is still fixed encoding per page
>> in
>> > >>>>>>>> parquet.
>> > >>>>>>>>>>> Given reasonably sized pages I don't think branch
>> > >>>>>>>> misprediction should
>> > >>>>>>>>>> be a
>> > >>>>>>>>>>> big issue for new encodings. I agree that we should be
>> > >>>>>>>> conservative in
>> > >>>>>>>>>>> general for adding new encodings.
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>> +1
>> > >>>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>
>> > >
>> >
>> >
>> >
>>
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by PRATEEK GAUR <pr...@gmail.com>.
Hi all,
I'd like to share a status update on the ALP encoding effort and get a feel
from the community before starting a formal vote.
We now have the specification plus implementations in both languages:
- Spec: apache/parquet-format#557 — GH-533 Add ALP encoding
specification
- C++ (Arrow): apache/arrow#48345 — GH-48701 [C++][Parquet] Add ALPpd
encoding
- Java: apache/parquet-java#3397 — Parquet Java ALP
Implementation (by Vinoo)
Current state:
- The C++ implementation has been through 3-4 rounds of review.
- The Java implementation has been tracking well and has had initial
reviews from contributors, with no major remaining issues.
- Cross-language compatibility tests are passing: the Arrow C++ decoder
reads Java-written data bit-exactly across ~1.56M values and 18
fixtures, covering V1 and V2 pages, multiple vector sizes, and several
real datasets — zero mismatches.
As raised in the java-pr we will be adding coverage for extreme values
(those requiring
63-64 bits after FOR is applied) before we close a vote. We'll aim to get
that
done in parallel.
Plan: unless there are objections, I intend to start a vote at the end of
this week/early
next week.
Thanks
Prateek
On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <pr...@gmail.com> wrote:
> Hi Team,
>
> Just wanted to provide some updates on ALP.
> Micah and I have done 3-4 rounds of review for the c++ PR :
> https://github.com/apache/arrow/pull/48345
>
> For the Java implementation Vinoo has been working on following PR :
> https://github.com/apache/parquet-java/pull/3397
>
> Best
> Prateek
>
> On Tue, May 5, 2026 at 1:50 PM Micah Kornfield <em...@gmail.com>
> wrote:
>
>> Hi Antoine,
>>
>> > Apologies if the question was already asked, but should we care about
>> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
>> > BYTE_STREAM_SPLIT + LZ4 for example?
>>
>> It was. We thought we could defer it for the following reasons:
>> 1. It's not clear there are a lot of easy reference datasets to test its
>> effectiveness. It does look like there might be one or two on huggingface
>> (e.g. https://huggingface.co/datasets/kikitora/curdie).
>> 2. It seemed likely that float 16 was more likely used for values that
>> were less likely to reduce to decimal values.
>> 3. It could be added as an extension later if needed.
>>
>> Cheers,
>> Micah
>>
>>
>> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <an...@python.org> wrote:
>>
>> >
>> > Hello,
>> >
>> > Apologies if the question was already asked, but should we care about
>> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
>> > BYTE_STREAM_SPLIT + LZ4 for example?
>> >
>> > Regards
>> >
>> > Antoine.
>> >
>> >
>> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
>> > > Thanks Andrew and Micah for review feedback on the two PR's
>> > > 1) (c++ arrow repo)
>> https://github.com/apache/arrow/pull/48345/changes
>> > > 2) (parquet-format repo)
>> > https://github.com/apache/parquet-format/pull/557
>> > >
>> > > I have addressed all (unless I missed something) comments on the two
>> > PR's.
>> > >
>> > > Best
>> > > Prateek
>> > >
>> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <pr...@gmail.com>
>> wrote:
>> > >
>> > >> Thanks Andrew and Micah.
>> > >>
>> > >> `fair amount of feedback on at least the implementations`
>> > >> For the c++ I have already started addressing the feedback, I should
>> be
>> > >> done with that Monday/Tuesday.
>> > >> I think Vinoo too has been making good progress on the Java
>> > implementation.
>> > >>
>> > >> Best
>> > >> Prateek
>> > >>
>> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <andrewlamb11@gmail.com
>> >
>> > >> wrote:
>> > >>
>> > >>> Got it. Thank you for the clarification -- I will try and look into
>> the
>> > >>> spec and the Rust implementation[1] in this next week
>> > >>>
>> > >>> [1]: https://github.com/apache/arrow-rs/pull/9372
>> > >>>
>> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <
>> > emkornfield@gmail.com>
>> > >>> wrote:
>> > >>>
>> > >>>> Hi Andrew,
>> > >>>> I think there is a fair amount of feedback on at least the
>> > >>>> implementations, typically I think we've waited till they are
>> close to
>> > >>>> mergeable before a final vote. Otherwise I agree we are very
>> close.
>> > >>>>
>> > >>>> -Micah
>> > >>>>
>> > >>>> On Saturday, April 25, 2026, Andrew Lamb <an...@gmail.com>
>> > wrote:
>> > >>>>
>> > >>>>> Thanks Prateek,
>> > >>>>>
>> > >>>>> I think from this content it looks to me like we are ready to
>> start a
>> > >>>>> vote to explicitly accept ALP into Parquet
>> > >>>>>
>> > >>>>> Does anyone know of a reason we should postpone it for longer?
>> > >>>>> Perhaps someone needs some more time to review?
>> > >>>>>
>> > >>>>> Andrew
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <pr...@gmail.com>
>> > >>>>> wrote:
>> > >>>>>
>> > >>>>>> Hi team,
>> > >>>>>>
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> Hope everyone is doing well. I got a chance to work through all
>> the
>> > >>>>>> remaining feedback and update the spec doc. Here are the new
>> > artifacts
>> > >>>>>>
>> > >>>>>> 1) Spec document :
>> > >>>>>>
>> >
>> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
>> > >>>>>>
>> > >>>>>> 2) Spec document in parquet format repo :
>> > >>>>>> https://github.com/apache/parquet-format/pull/557
>> > >>>>>>
>> > >>>>>> 3) Alp implementation in arrow c++ repo :
>> > >>>>>> https://github.com/apache/arrow/pull/48345/changes
>> > >>>>>>
>> > >>>>>> 4) Alp implementation in parquet-java repo : Work for Vinoo and
>> > Julien
>> > >>>>>> https://github.com/apache/parquet-java/pull/3397
>> > >>>>>>
>> > >>>>>> 5) PR with test and benchmarking artifacts in parquet-testing
>> repo :
>> > >>>>>> https://github.com/apache/parquet-testing/pull/100
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> And
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> - Go : Arnav just submitted an in progress implementation in
>> Go.
>> > >>>>>> https://github.com/apache/arrow-go/pull/704 (I haven't
>> started
>> > >>>>>> looking at it yet)
>> > >>>>>> - Rust : I remember Andrew mentioned that this work is also
>> in
>> > >>>>>> progress (So 4 languages!)
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> *Arrow C++ implementation *
>> > >>>>>>
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> The PR is out and was also used by Antoine to report the numbers
>> as
>> > >>>>>> reported here. Micah and Konstantin have given 1 round of
>> feedback
>> > >>>>>> and I'm addressing them today. Please note that the default
>> > >>>>>> optimization flag for compiling is O2 and not Q3. I got around
>> 70%
>> > >>>>>> performance improvement in the decoding speed when using the O3
>> > flag.
>> > >>>>>>
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> *Parqet-MR Java implementation (working with Vinoo and Julien)
>> and
>> > **Cross
>> > >>>>>> Language testing*
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> Let me know if you have any questions or feedback.
>> > >>>>>>
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> Now pasting some performance numbers
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> Table 1: C++ ALP Double Decode — Spotify Columns (Graviton 3,
>> ARM
>> > >>>>>> Neoverse V1)
>> > >>>>>>
>> > >>>>>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
>> > >>>>>>
>> > >>>>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │ Speedup │
>> > >>>>>>
>> > >>>>>> ├──────────────────┼──────────────┼──────────────┼─────────┤
>> > >>>>>>
>> > >>>>>> │ valence │ 3,155 │ 5,523 │ 1.75x │
>> > >>>>>>
>> > >>>>>> │ danceability │ 3,233 │ 5,685 │ 1.76x │
>> > >>>>>>
>> > >>>>>> │ energy │ 3,197 │ 5,652 │ 1.77x │
>> > >>>>>>
>> > >>>>>> │ loudness │ 3,186 │ 5,473 │ 1.72x │
>> > >>>>>>
>> > >>>>>> └──────────────────┴──────────────┴──────────────┴─────────┘
>> > >>>>>>
>> > >>>>>>
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <prtkgaur@gmail.com
>> >
>> > >>>>>> wrote:
>> > >>>>>>
>> > >>>>>>> @Micah Kornfield <em...@gmail.com> : Got it.
>> > >>>>>>>
>> > >>>>>>> @Andrew Lamb <an...@gmail.com>
>> > >>>>>>>
>> > >>>>>>>
>> > >>>>>>>> Do you think it would be good to start moving the spec
>> development
>> > >>>>>>>> into
>> > >>>>>>>> markdown format, in preparation for finalizing it?
>> > >>>>>>>>
>> > >>>>>>>
>> > >>>>>>> Yes I'll update the numbers for some of the examples I have in
>> the
>> > >>>>>>> spec based
>> > >>>>>>> on the updated header size. Then we should be good to go for the
>> > >>>>>>> markdown format.
>> > >>>>>>>
>> > >>>>>>> Thanks everyone!
>> > >>>>>>>
>> > >>>>>>>
>> > >>>>>>>>
>> > >>>>>>>> Andrew
>> > >>>>>>>>
>> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <
>> prtkgaur@gmail.com>
>> > >>>>>>>> wrote:
>> > >>>>>>>>
>> > >>>>>>>>> Hi team,
>> > >>>>>>>>>
>> > >>>>>>>>> 1) Andrew
>> > >>>>>>>>>
>> > >>>>>>>>> - Thanks for working on test files. My PR did add all the
>> > test
>> > >>>>>>>> files I
>> > >>>>>>>>> used to benchmark on datasets. Maybe we can club it
>> together.
>> > >>>>>>>> WIll also
>> > >>>>>>>>> aid
>> > >>>>>>>>> cross language testing
>> > >>>>>>>>> - Kosta Tarasov working on Rust implementation. This is
>> > great.
>> > >>>>>>>> Thanks
>> > >>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>> 2) Antoine
>> > >>>>>>>>>
>> > >>>>>>>>> - Thanks a lot for reporting the numbers on AMD. Looks
>> like
>> > you
>> > >>>>>>>> are
>> > >>>>>>>>> getting 8X the decoding performance of BSS. This is
>> > amazing!!.
>> > >>>>>>>>> - Thanks for acknowledging the sampling design.
>> > >>>>>>>>> - I agree with you on Fastlanes. In some crude
>> experiments I
>> > >>>>>>>> didn't get
>> > >>>>>>>>> a good perf benefit from it on Graviton3 (but maybe there
>> was
>> > >>>>>>>> something
>> > >>>>>>>>> wrong with my implementation).
>> > >>>>>>>>> - Locking the 16bit exception encoding for the spec in
>> this
>> > >>>>>>>> case.
>> > >>>>>>>>> - Awesome I think we have solved for all open questions
>> minus
>> > >>>>>>>> the
>> > >>>>>>>>> version byte :). (will get back on this soon)
>> > >>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>> 3) Micah
>> > >>>>>>>>>
>> > >>>>>>>>> - FastLanes : The current spec does allow for using
>> FastLane
>> > >>>>>>>> with the
>> > >>>>>>>>> configurable enum value for layout. We should be able to
>> > inject
>> > >>>>>>>> any
>> > >>>>>>>>> layout
>> > >>>>>>>>> in the current design.
>> > >>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>> Working on resolving all remaining open comments on the spec
>> this
>> > >>>>>>>> week.
>> > >>>>>>>>>
>> > >>>>>>>>> Best
>> > >>>>>>>>> Prateek
>> > >>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <
>> > >>>>>>>> stevel@cloudera.com>
>> > >>>>>>>>> wrote:
>> > >>>>>>>>>
>> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
>> > >>>>>>>> emkornfield@gmail.com>
>> > >>>>>>>>>> wrote:
>> > >>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> It looks like the actual issue described for ORC in the
>> paper
>> > >>>>>>>> is that
>> > >>>>>>>>> it
>> > >>>>>>>>>>> has multiple sub-encodings in a batch. This is different
>> then
>> > >>>>>>>> the
>> > >>>>>>>>> design
>> > >>>>>>>>>>> proposed here where there is still fixed encoding per page
>> in
>> > >>>>>>>> parquet.
>> > >>>>>>>>>>> Given reasonably sized pages I don't think branch
>> > >>>>>>>> misprediction should
>> > >>>>>>>>>> be a
>> > >>>>>>>>>>> big issue for new encodings. I agree that we should be
>> > >>>>>>>> conservative in
>> > >>>>>>>>>>> general for adding new encodings.
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>> +1
>> > >>>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>
>> > >
>> >
>> >
>> >
>>
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by PRATEEK GAUR <pr...@gmail.com>.
Hi Team,
Just wanted to provide some updates on ALP.
Micah and I have done 3-4 rounds of review for the c++ PR :
https://github.com/apache/arrow/pull/48345
For the Java implementation Vinoo has been working on following PR :
https://github.com/apache/parquet-java/pull/3397
Best
Prateek
On Tue, May 5, 2026 at 1:50 PM Micah Kornfield <em...@gmail.com>
wrote:
> Hi Antoine,
>
> > Apologies if the question was already asked, but should we care about
> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
> > BYTE_STREAM_SPLIT + LZ4 for example?
>
> It was. We thought we could defer it for the following reasons:
> 1. It's not clear there are a lot of easy reference datasets to test its
> effectiveness. It does look like there might be one or two on huggingface
> (e.g. https://huggingface.co/datasets/kikitora/curdie).
> 2. It seemed likely that float 16 was more likely used for values that
> were less likely to reduce to decimal values.
> 3. It could be added as an extension later if needed.
>
> Cheers,
> Micah
>
>
> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <an...@python.org> wrote:
>
> >
> > Hello,
> >
> > Apologies if the question was already asked, but should we care about
> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
> > BYTE_STREAM_SPLIT + LZ4 for example?
> >
> > Regards
> >
> > Antoine.
> >
> >
> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
> > > Thanks Andrew and Micah for review feedback on the two PR's
> > > 1) (c++ arrow repo) https://github.com/apache/arrow/pull/48345/changes
> > > 2) (parquet-format repo)
> > https://github.com/apache/parquet-format/pull/557
> > >
> > > I have addressed all (unless I missed something) comments on the two
> > PR's.
> > >
> > > Best
> > > Prateek
> > >
> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <pr...@gmail.com>
> wrote:
> > >
> > >> Thanks Andrew and Micah.
> > >>
> > >> `fair amount of feedback on at least the implementations`
> > >> For the c++ I have already started addressing the feedback, I should
> be
> > >> done with that Monday/Tuesday.
> > >> I think Vinoo too has been making good progress on the Java
> > implementation.
> > >>
> > >> Best
> > >> Prateek
> > >>
> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <an...@gmail.com>
> > >> wrote:
> > >>
> > >>> Got it. Thank you for the clarification -- I will try and look into
> the
> > >>> spec and the Rust implementation[1] in this next week
> > >>>
> > >>> [1]: https://github.com/apache/arrow-rs/pull/9372
> > >>>
> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <
> > emkornfield@gmail.com>
> > >>> wrote:
> > >>>
> > >>>> Hi Andrew,
> > >>>> I think there is a fair amount of feedback on at least the
> > >>>> implementations, typically I think we've waited till they are close
> to
> > >>>> mergeable before a final vote. Otherwise I agree we are very close.
> > >>>>
> > >>>> -Micah
> > >>>>
> > >>>> On Saturday, April 25, 2026, Andrew Lamb <an...@gmail.com>
> > wrote:
> > >>>>
> > >>>>> Thanks Prateek,
> > >>>>>
> > >>>>> I think from this content it looks to me like we are ready to
> start a
> > >>>>> vote to explicitly accept ALP into Parquet
> > >>>>>
> > >>>>> Does anyone know of a reason we should postpone it for longer?
> > >>>>> Perhaps someone needs some more time to review?
> > >>>>>
> > >>>>> Andrew
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <pr...@gmail.com>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Hi team,
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> Hope everyone is doing well. I got a chance to work through all
> the
> > >>>>>> remaining feedback and update the spec doc. Here are the new
> > artifacts
> > >>>>>>
> > >>>>>> 1) Spec document :
> > >>>>>>
> >
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
> > >>>>>>
> > >>>>>> 2) Spec document in parquet format repo :
> > >>>>>> https://github.com/apache/parquet-format/pull/557
> > >>>>>>
> > >>>>>> 3) Alp implementation in arrow c++ repo :
> > >>>>>> https://github.com/apache/arrow/pull/48345/changes
> > >>>>>>
> > >>>>>> 4) Alp implementation in parquet-java repo : Work for Vinoo and
> > Julien
> > >>>>>> https://github.com/apache/parquet-java/pull/3397
> > >>>>>>
> > >>>>>> 5) PR with test and benchmarking artifacts in parquet-testing
> repo :
> > >>>>>> https://github.com/apache/parquet-testing/pull/100
> > >>>>>>
> > >>>>>>
> > >>>>>> And
> > >>>>>>
> > >>>>>>
> > >>>>>> - Go : Arnav just submitted an in progress implementation in
> Go.
> > >>>>>> https://github.com/apache/arrow-go/pull/704 (I haven't
> started
> > >>>>>> looking at it yet)
> > >>>>>> - Rust : I remember Andrew mentioned that this work is also in
> > >>>>>> progress (So 4 languages!)
> > >>>>>>
> > >>>>>>
> > >>>>>> *Arrow C++ implementation *
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> The PR is out and was also used by Antoine to report the numbers
> as
> > >>>>>> reported here. Micah and Konstantin have given 1 round of feedback
> > >>>>>> and I'm addressing them today. Please note that the default
> > >>>>>> optimization flag for compiling is O2 and not Q3. I got around 70%
> > >>>>>> performance improvement in the decoding speed when using the O3
> > flag.
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> *Parqet-MR Java implementation (working with Vinoo and Julien) and
> > **Cross
> > >>>>>> Language testing*
> > >>>>>>
> > >>>>>>
> > >>>>>> Let me know if you have any questions or feedback.
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> Now pasting some performance numbers
> > >>>>>>
> > >>>>>>
> > >>>>>> Table 1: C++ ALP Double Decode — Spotify Columns (Graviton 3,
> ARM
> > >>>>>> Neoverse V1)
> > >>>>>>
> > >>>>>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
> > >>>>>>
> > >>>>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │ Speedup │
> > >>>>>>
> > >>>>>> ├──────────────────┼──────────────┼──────────────┼─────────┤
> > >>>>>>
> > >>>>>> │ valence │ 3,155 │ 5,523 │ 1.75x │
> > >>>>>>
> > >>>>>> │ danceability │ 3,233 │ 5,685 │ 1.76x │
> > >>>>>>
> > >>>>>> │ energy │ 3,197 │ 5,652 │ 1.77x │
> > >>>>>>
> > >>>>>> │ loudness │ 3,186 │ 5,473 │ 1.72x │
> > >>>>>>
> > >>>>>> └──────────────────┴──────────────┴──────────────┴─────────┘
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <pr...@gmail.com>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> @Micah Kornfield <em...@gmail.com> : Got it.
> > >>>>>>>
> > >>>>>>> @Andrew Lamb <an...@gmail.com>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>> Do you think it would be good to start moving the spec
> development
> > >>>>>>>> into
> > >>>>>>>> markdown format, in preparation for finalizing it?
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>> Yes I'll update the numbers for some of the examples I have in
> the
> > >>>>>>> spec based
> > >>>>>>> on the updated header size. Then we should be good to go for the
> > >>>>>>> markdown format.
> > >>>>>>>
> > >>>>>>> Thanks everyone!
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>>
> > >>>>>>>> Andrew
> > >>>>>>>>
> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <
> prtkgaur@gmail.com>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Hi team,
> > >>>>>>>>>
> > >>>>>>>>> 1) Andrew
> > >>>>>>>>>
> > >>>>>>>>> - Thanks for working on test files. My PR did add all the
> > test
> > >>>>>>>> files I
> > >>>>>>>>> used to benchmark on datasets. Maybe we can club it
> together.
> > >>>>>>>> WIll also
> > >>>>>>>>> aid
> > >>>>>>>>> cross language testing
> > >>>>>>>>> - Kosta Tarasov working on Rust implementation. This is
> > great.
> > >>>>>>>> Thanks
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> 2) Antoine
> > >>>>>>>>>
> > >>>>>>>>> - Thanks a lot for reporting the numbers on AMD. Looks like
> > you
> > >>>>>>>> are
> > >>>>>>>>> getting 8X the decoding performance of BSS. This is
> > amazing!!.
> > >>>>>>>>> - Thanks for acknowledging the sampling design.
> > >>>>>>>>> - I agree with you on Fastlanes. In some crude experiments
> I
> > >>>>>>>> didn't get
> > >>>>>>>>> a good perf benefit from it on Graviton3 (but maybe there
> was
> > >>>>>>>> something
> > >>>>>>>>> wrong with my implementation).
> > >>>>>>>>> - Locking the 16bit exception encoding for the spec in this
> > >>>>>>>> case.
> > >>>>>>>>> - Awesome I think we have solved for all open questions
> minus
> > >>>>>>>> the
> > >>>>>>>>> version byte :). (will get back on this soon)
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> 3) Micah
> > >>>>>>>>>
> > >>>>>>>>> - FastLanes : The current spec does allow for using
> FastLane
> > >>>>>>>> with the
> > >>>>>>>>> configurable enum value for layout. We should be able to
> > inject
> > >>>>>>>> any
> > >>>>>>>>> layout
> > >>>>>>>>> in the current design.
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> Working on resolving all remaining open comments on the spec
> this
> > >>>>>>>> week.
> > >>>>>>>>>
> > >>>>>>>>> Best
> > >>>>>>>>> Prateek
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <
> > >>>>>>>> stevel@cloudera.com>
> > >>>>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
> > >>>>>>>> emkornfield@gmail.com>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> It looks like the actual issue described for ORC in the paper
> > >>>>>>>> is that
> > >>>>>>>>> it
> > >>>>>>>>>>> has multiple sub-encodings in a batch. This is different
> then
> > >>>>>>>> the
> > >>>>>>>>> design
> > >>>>>>>>>>> proposed here where there is still fixed encoding per page in
> > >>>>>>>> parquet.
> > >>>>>>>>>>> Given reasonably sized pages I don't think branch
> > >>>>>>>> misprediction should
> > >>>>>>>>>> be a
> > >>>>>>>>>>> big issue for new encodings. I agree that we should be
> > >>>>>>>> conservative in
> > >>>>>>>>>>> general for adding new encodings.
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>> +1
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >
> >
> >
> >
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by PRATEEK GAUR <pr...@gmail.com>.
Hi Team,
Just wanted to provide some updates on ALP.
Micah and I have done 3-4 rounds of review for the c++ PR :
https://github.com/apache/arrow/pull/48345
For the Java implementation Vinoo has been working on following PR :
https://github.com/apache/parquet-java/pull/3397
Best
Prateek
On Tue, May 5, 2026 at 1:50 PM Micah Kornfield <em...@gmail.com>
wrote:
> Hi Antoine,
>
> > Apologies if the question was already asked, but should we care about
> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
> > BYTE_STREAM_SPLIT + LZ4 for example?
>
> It was. We thought we could defer it for the following reasons:
> 1. It's not clear there are a lot of easy reference datasets to test its
> effectiveness. It does look like there might be one or two on huggingface
> (e.g. https://huggingface.co/datasets/kikitora/curdie).
> 2. It seemed likely that float 16 was more likely used for values that
> were less likely to reduce to decimal values.
> 3. It could be added as an extension later if needed.
>
> Cheers,
> Micah
>
>
> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <an...@python.org> wrote:
>
> >
> > Hello,
> >
> > Apologies if the question was already asked, but should we care about
> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
> > BYTE_STREAM_SPLIT + LZ4 for example?
> >
> > Regards
> >
> > Antoine.
> >
> >
> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
> > > Thanks Andrew and Micah for review feedback on the two PR's
> > > 1) (c++ arrow repo) https://github.com/apache/arrow/pull/48345/changes
> > > 2) (parquet-format repo)
> > https://github.com/apache/parquet-format/pull/557
> > >
> > > I have addressed all (unless I missed something) comments on the two
> > PR's.
> > >
> > > Best
> > > Prateek
> > >
> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <pr...@gmail.com>
> wrote:
> > >
> > >> Thanks Andrew and Micah.
> > >>
> > >> `fair amount of feedback on at least the implementations`
> > >> For the c++ I have already started addressing the feedback, I should
> be
> > >> done with that Monday/Tuesday.
> > >> I think Vinoo too has been making good progress on the Java
> > implementation.
> > >>
> > >> Best
> > >> Prateek
> > >>
> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <an...@gmail.com>
> > >> wrote:
> > >>
> > >>> Got it. Thank you for the clarification -- I will try and look into
> the
> > >>> spec and the Rust implementation[1] in this next week
> > >>>
> > >>> [1]: https://github.com/apache/arrow-rs/pull/9372
> > >>>
> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <
> > emkornfield@gmail.com>
> > >>> wrote:
> > >>>
> > >>>> Hi Andrew,
> > >>>> I think there is a fair amount of feedback on at least the
> > >>>> implementations, typically I think we've waited till they are close
> to
> > >>>> mergeable before a final vote. Otherwise I agree we are very close.
> > >>>>
> > >>>> -Micah
> > >>>>
> > >>>> On Saturday, April 25, 2026, Andrew Lamb <an...@gmail.com>
> > wrote:
> > >>>>
> > >>>>> Thanks Prateek,
> > >>>>>
> > >>>>> I think from this content it looks to me like we are ready to
> start a
> > >>>>> vote to explicitly accept ALP into Parquet
> > >>>>>
> > >>>>> Does anyone know of a reason we should postpone it for longer?
> > >>>>> Perhaps someone needs some more time to review?
> > >>>>>
> > >>>>> Andrew
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <pr...@gmail.com>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Hi team,
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> Hope everyone is doing well. I got a chance to work through all
> the
> > >>>>>> remaining feedback and update the spec doc. Here are the new
> > artifacts
> > >>>>>>
> > >>>>>> 1) Spec document :
> > >>>>>>
> >
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
> > >>>>>>
> > >>>>>> 2) Spec document in parquet format repo :
> > >>>>>> https://github.com/apache/parquet-format/pull/557
> > >>>>>>
> > >>>>>> 3) Alp implementation in arrow c++ repo :
> > >>>>>> https://github.com/apache/arrow/pull/48345/changes
> > >>>>>>
> > >>>>>> 4) Alp implementation in parquet-java repo : Work for Vinoo and
> > Julien
> > >>>>>> https://github.com/apache/parquet-java/pull/3397
> > >>>>>>
> > >>>>>> 5) PR with test and benchmarking artifacts in parquet-testing
> repo :
> > >>>>>> https://github.com/apache/parquet-testing/pull/100
> > >>>>>>
> > >>>>>>
> > >>>>>> And
> > >>>>>>
> > >>>>>>
> > >>>>>> - Go : Arnav just submitted an in progress implementation in
> Go.
> > >>>>>> https://github.com/apache/arrow-go/pull/704 (I haven't
> started
> > >>>>>> looking at it yet)
> > >>>>>> - Rust : I remember Andrew mentioned that this work is also in
> > >>>>>> progress (So 4 languages!)
> > >>>>>>
> > >>>>>>
> > >>>>>> *Arrow C++ implementation *
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> The PR is out and was also used by Antoine to report the numbers
> as
> > >>>>>> reported here. Micah and Konstantin have given 1 round of feedback
> > >>>>>> and I'm addressing them today. Please note that the default
> > >>>>>> optimization flag for compiling is O2 and not Q3. I got around 70%
> > >>>>>> performance improvement in the decoding speed when using the O3
> > flag.
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> *Parqet-MR Java implementation (working with Vinoo and Julien) and
> > **Cross
> > >>>>>> Language testing*
> > >>>>>>
> > >>>>>>
> > >>>>>> Let me know if you have any questions or feedback.
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> Now pasting some performance numbers
> > >>>>>>
> > >>>>>>
> > >>>>>> Table 1: C++ ALP Double Decode — Spotify Columns (Graviton 3,
> ARM
> > >>>>>> Neoverse V1)
> > >>>>>>
> > >>>>>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
> > >>>>>>
> > >>>>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │ Speedup │
> > >>>>>>
> > >>>>>> ├──────────────────┼──────────────┼──────────────┼─────────┤
> > >>>>>>
> > >>>>>> │ valence │ 3,155 │ 5,523 │ 1.75x │
> > >>>>>>
> > >>>>>> │ danceability │ 3,233 │ 5,685 │ 1.76x │
> > >>>>>>
> > >>>>>> │ energy │ 3,197 │ 5,652 │ 1.77x │
> > >>>>>>
> > >>>>>> │ loudness │ 3,186 │ 5,473 │ 1.72x │
> > >>>>>>
> > >>>>>> └──────────────────┴──────────────┴──────────────┴─────────┘
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <pr...@gmail.com>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> @Micah Kornfield <em...@gmail.com> : Got it.
> > >>>>>>>
> > >>>>>>> @Andrew Lamb <an...@gmail.com>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>> Do you think it would be good to start moving the spec
> development
> > >>>>>>>> into
> > >>>>>>>> markdown format, in preparation for finalizing it?
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>> Yes I'll update the numbers for some of the examples I have in
> the
> > >>>>>>> spec based
> > >>>>>>> on the updated header size. Then we should be good to go for the
> > >>>>>>> markdown format.
> > >>>>>>>
> > >>>>>>> Thanks everyone!
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>>
> > >>>>>>>> Andrew
> > >>>>>>>>
> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <
> prtkgaur@gmail.com>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Hi team,
> > >>>>>>>>>
> > >>>>>>>>> 1) Andrew
> > >>>>>>>>>
> > >>>>>>>>> - Thanks for working on test files. My PR did add all the
> > test
> > >>>>>>>> files I
> > >>>>>>>>> used to benchmark on datasets. Maybe we can club it
> together.
> > >>>>>>>> WIll also
> > >>>>>>>>> aid
> > >>>>>>>>> cross language testing
> > >>>>>>>>> - Kosta Tarasov working on Rust implementation. This is
> > great.
> > >>>>>>>> Thanks
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> 2) Antoine
> > >>>>>>>>>
> > >>>>>>>>> - Thanks a lot for reporting the numbers on AMD. Looks like
> > you
> > >>>>>>>> are
> > >>>>>>>>> getting 8X the decoding performance of BSS. This is
> > amazing!!.
> > >>>>>>>>> - Thanks for acknowledging the sampling design.
> > >>>>>>>>> - I agree with you on Fastlanes. In some crude experiments
> I
> > >>>>>>>> didn't get
> > >>>>>>>>> a good perf benefit from it on Graviton3 (but maybe there
> was
> > >>>>>>>> something
> > >>>>>>>>> wrong with my implementation).
> > >>>>>>>>> - Locking the 16bit exception encoding for the spec in this
> > >>>>>>>> case.
> > >>>>>>>>> - Awesome I think we have solved for all open questions
> minus
> > >>>>>>>> the
> > >>>>>>>>> version byte :). (will get back on this soon)
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> 3) Micah
> > >>>>>>>>>
> > >>>>>>>>> - FastLanes : The current spec does allow for using
> FastLane
> > >>>>>>>> with the
> > >>>>>>>>> configurable enum value for layout. We should be able to
> > inject
> > >>>>>>>> any
> > >>>>>>>>> layout
> > >>>>>>>>> in the current design.
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> Working on resolving all remaining open comments on the spec
> this
> > >>>>>>>> week.
> > >>>>>>>>>
> > >>>>>>>>> Best
> > >>>>>>>>> Prateek
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <
> > >>>>>>>> stevel@cloudera.com>
> > >>>>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
> > >>>>>>>> emkornfield@gmail.com>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> It looks like the actual issue described for ORC in the paper
> > >>>>>>>> is that
> > >>>>>>>>> it
> > >>>>>>>>>>> has multiple sub-encodings in a batch. This is different
> then
> > >>>>>>>> the
> > >>>>>>>>> design
> > >>>>>>>>>>> proposed here where there is still fixed encoding per page in
> > >>>>>>>> parquet.
> > >>>>>>>>>>> Given reasonably sized pages I don't think branch
> > >>>>>>>> misprediction should
> > >>>>>>>>>> be a
> > >>>>>>>>>>> big issue for new encodings. I agree that we should be
> > >>>>>>>> conservative in
> > >>>>>>>>>>> general for adding new encodings.
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>> +1
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >
> >
> >
> >
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Micah Kornfield <em...@gmail.com>.
Hi Antoine,
> Apologies if the question was already asked, but should we care about
> FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
> BYTE_STREAM_SPLIT + LZ4 for example?
It was. We thought we could defer it for the following reasons:
1. It's not clear there are a lot of easy reference datasets to test its
effectiveness. It does look like there might be one or two on huggingface
(e.g. https://huggingface.co/datasets/kikitora/curdie).
2. It seemed likely that float 16 was more likely used for values that
were less likely to reduce to decimal values.
3. It could be added as an extension later if needed.
Cheers,
Micah
On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <an...@python.org> wrote:
>
> Hello,
>
> Apologies if the question was already asked, but should we care about
> FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
> BYTE_STREAM_SPLIT + LZ4 for example?
>
> Regards
>
> Antoine.
>
>
> Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
> > Thanks Andrew and Micah for review feedback on the two PR's
> > 1) (c++ arrow repo) https://github.com/apache/arrow/pull/48345/changes
> > 2) (parquet-format repo)
> https://github.com/apache/parquet-format/pull/557
> >
> > I have addressed all (unless I missed something) comments on the two
> PR's.
> >
> > Best
> > Prateek
> >
> > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <pr...@gmail.com> wrote:
> >
> >> Thanks Andrew and Micah.
> >>
> >> `fair amount of feedback on at least the implementations`
> >> For the c++ I have already started addressing the feedback, I should be
> >> done with that Monday/Tuesday.
> >> I think Vinoo too has been making good progress on the Java
> implementation.
> >>
> >> Best
> >> Prateek
> >>
> >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <an...@gmail.com>
> >> wrote:
> >>
> >>> Got it. Thank you for the clarification -- I will try and look into the
> >>> spec and the Rust implementation[1] in this next week
> >>>
> >>> [1]: https://github.com/apache/arrow-rs/pull/9372
> >>>
> >>> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <
> emkornfield@gmail.com>
> >>> wrote:
> >>>
> >>>> Hi Andrew,
> >>>> I think there is a fair amount of feedback on at least the
> >>>> implementations, typically I think we've waited till they are close to
> >>>> mergeable before a final vote. Otherwise I agree we are very close.
> >>>>
> >>>> -Micah
> >>>>
> >>>> On Saturday, April 25, 2026, Andrew Lamb <an...@gmail.com>
> wrote:
> >>>>
> >>>>> Thanks Prateek,
> >>>>>
> >>>>> I think from this content it looks to me like we are ready to start a
> >>>>> vote to explicitly accept ALP into Parquet
> >>>>>
> >>>>> Does anyone know of a reason we should postpone it for longer?
> >>>>> Perhaps someone needs some more time to review?
> >>>>>
> >>>>> Andrew
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <pr...@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>>> Hi team,
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Hope everyone is doing well. I got a chance to work through all the
> >>>>>> remaining feedback and update the spec doc. Here are the new
> artifacts
> >>>>>>
> >>>>>> 1) Spec document :
> >>>>>>
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
> >>>>>>
> >>>>>> 2) Spec document in parquet format repo :
> >>>>>> https://github.com/apache/parquet-format/pull/557
> >>>>>>
> >>>>>> 3) Alp implementation in arrow c++ repo :
> >>>>>> https://github.com/apache/arrow/pull/48345/changes
> >>>>>>
> >>>>>> 4) Alp implementation in parquet-java repo : Work for Vinoo and
> Julien
> >>>>>> https://github.com/apache/parquet-java/pull/3397
> >>>>>>
> >>>>>> 5) PR with test and benchmarking artifacts in parquet-testing repo :
> >>>>>> https://github.com/apache/parquet-testing/pull/100
> >>>>>>
> >>>>>>
> >>>>>> And
> >>>>>>
> >>>>>>
> >>>>>> - Go : Arnav just submitted an in progress implementation in Go.
> >>>>>> https://github.com/apache/arrow-go/pull/704 (I haven't started
> >>>>>> looking at it yet)
> >>>>>> - Rust : I remember Andrew mentioned that this work is also in
> >>>>>> progress (So 4 languages!)
> >>>>>>
> >>>>>>
> >>>>>> *Arrow C++ implementation *
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> The PR is out and was also used by Antoine to report the numbers as
> >>>>>> reported here. Micah and Konstantin have given 1 round of feedback
> >>>>>> and I'm addressing them today. Please note that the default
> >>>>>> optimization flag for compiling is O2 and not Q3. I got around 70%
> >>>>>> performance improvement in the decoding speed when using the O3
> flag.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> *Parqet-MR Java implementation (working with Vinoo and Julien) and
> **Cross
> >>>>>> Language testing*
> >>>>>>
> >>>>>>
> >>>>>> Let me know if you have any questions or feedback.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Now pasting some performance numbers
> >>>>>>
> >>>>>>
> >>>>>> Table 1: C++ ALP Double Decode — Spotify Columns (Graviton 3, ARM
> >>>>>> Neoverse V1)
> >>>>>>
> >>>>>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
> >>>>>>
> >>>>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │ Speedup │
> >>>>>>
> >>>>>> ├──────────────────┼──────────────┼──────────────┼─────────┤
> >>>>>>
> >>>>>> │ valence │ 3,155 │ 5,523 │ 1.75x │
> >>>>>>
> >>>>>> │ danceability │ 3,233 │ 5,685 │ 1.76x │
> >>>>>>
> >>>>>> │ energy │ 3,197 │ 5,652 │ 1.77x │
> >>>>>>
> >>>>>> │ loudness │ 3,186 │ 5,473 │ 1.72x │
> >>>>>>
> >>>>>> └──────────────────┴──────────────┴──────────────┴─────────┘
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <pr...@gmail.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> @Micah Kornfield <em...@gmail.com> : Got it.
> >>>>>>>
> >>>>>>> @Andrew Lamb <an...@gmail.com>
> >>>>>>>
> >>>>>>>
> >>>>>>>> Do you think it would be good to start moving the spec development
> >>>>>>>> into
> >>>>>>>> markdown format, in preparation for finalizing it?
> >>>>>>>>
> >>>>>>>
> >>>>>>> Yes I'll update the numbers for some of the examples I have in the
> >>>>>>> spec based
> >>>>>>> on the updated header size. Then we should be good to go for the
> >>>>>>> markdown format.
> >>>>>>>
> >>>>>>> Thanks everyone!
> >>>>>>>
> >>>>>>>
> >>>>>>>>
> >>>>>>>> Andrew
> >>>>>>>>
> >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <pr...@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Hi team,
> >>>>>>>>>
> >>>>>>>>> 1) Andrew
> >>>>>>>>>
> >>>>>>>>> - Thanks for working on test files. My PR did add all the
> test
> >>>>>>>> files I
> >>>>>>>>> used to benchmark on datasets. Maybe we can club it together.
> >>>>>>>> WIll also
> >>>>>>>>> aid
> >>>>>>>>> cross language testing
> >>>>>>>>> - Kosta Tarasov working on Rust implementation. This is
> great.
> >>>>>>>> Thanks
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> 2) Antoine
> >>>>>>>>>
> >>>>>>>>> - Thanks a lot for reporting the numbers on AMD. Looks like
> you
> >>>>>>>> are
> >>>>>>>>> getting 8X the decoding performance of BSS. This is
> amazing!!.
> >>>>>>>>> - Thanks for acknowledging the sampling design.
> >>>>>>>>> - I agree with you on Fastlanes. In some crude experiments I
> >>>>>>>> didn't get
> >>>>>>>>> a good perf benefit from it on Graviton3 (but maybe there was
> >>>>>>>> something
> >>>>>>>>> wrong with my implementation).
> >>>>>>>>> - Locking the 16bit exception encoding for the spec in this
> >>>>>>>> case.
> >>>>>>>>> - Awesome I think we have solved for all open questions minus
> >>>>>>>> the
> >>>>>>>>> version byte :). (will get back on this soon)
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> 3) Micah
> >>>>>>>>>
> >>>>>>>>> - FastLanes : The current spec does allow for using FastLane
> >>>>>>>> with the
> >>>>>>>>> configurable enum value for layout. We should be able to
> inject
> >>>>>>>> any
> >>>>>>>>> layout
> >>>>>>>>> in the current design.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Working on resolving all remaining open comments on the spec this
> >>>>>>>> week.
> >>>>>>>>>
> >>>>>>>>> Best
> >>>>>>>>> Prateek
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <
> >>>>>>>> stevel@cloudera.com>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
> >>>>>>>> emkornfield@gmail.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> It looks like the actual issue described for ORC in the paper
> >>>>>>>> is that
> >>>>>>>>> it
> >>>>>>>>>>> has multiple sub-encodings in a batch. This is different then
> >>>>>>>> the
> >>>>>>>>> design
> >>>>>>>>>>> proposed here where there is still fixed encoding per page in
> >>>>>>>> parquet.
> >>>>>>>>>>> Given reasonably sized pages I don't think branch
> >>>>>>>> misprediction should
> >>>>>>>>>> be a
> >>>>>>>>>>> big issue for new encodings. I agree that we should be
> >>>>>>>> conservative in
> >>>>>>>>>>> general for adding new encodings.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>> +1
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >
>
>
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Micah Kornfield <em...@gmail.com>.
Hi Antoine,
> Apologies if the question was already asked, but should we care about
> FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
> BYTE_STREAM_SPLIT + LZ4 for example?
It was. We thought we could defer it for the following reasons:
1. It's not clear there are a lot of easy reference datasets to test its
effectiveness. It does look like there might be one or two on huggingface
(e.g. https://huggingface.co/datasets/kikitora/curdie).
2. It seemed likely that float 16 was more likely used for values that
were less likely to reduce to decimal values.
3. It could be added as an extension later if needed.
Cheers,
Micah
On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <an...@python.org> wrote:
>
> Hello,
>
> Apologies if the question was already asked, but should we care about
> FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
> BYTE_STREAM_SPLIT + LZ4 for example?
>
> Regards
>
> Antoine.
>
>
> Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
> > Thanks Andrew and Micah for review feedback on the two PR's
> > 1) (c++ arrow repo) https://github.com/apache/arrow/pull/48345/changes
> > 2) (parquet-format repo)
> https://github.com/apache/parquet-format/pull/557
> >
> > I have addressed all (unless I missed something) comments on the two
> PR's.
> >
> > Best
> > Prateek
> >
> > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <pr...@gmail.com> wrote:
> >
> >> Thanks Andrew and Micah.
> >>
> >> `fair amount of feedback on at least the implementations`
> >> For the c++ I have already started addressing the feedback, I should be
> >> done with that Monday/Tuesday.
> >> I think Vinoo too has been making good progress on the Java
> implementation.
> >>
> >> Best
> >> Prateek
> >>
> >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <an...@gmail.com>
> >> wrote:
> >>
> >>> Got it. Thank you for the clarification -- I will try and look into the
> >>> spec and the Rust implementation[1] in this next week
> >>>
> >>> [1]: https://github.com/apache/arrow-rs/pull/9372
> >>>
> >>> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <
> emkornfield@gmail.com>
> >>> wrote:
> >>>
> >>>> Hi Andrew,
> >>>> I think there is a fair amount of feedback on at least the
> >>>> implementations, typically I think we've waited till they are close to
> >>>> mergeable before a final vote. Otherwise I agree we are very close.
> >>>>
> >>>> -Micah
> >>>>
> >>>> On Saturday, April 25, 2026, Andrew Lamb <an...@gmail.com>
> wrote:
> >>>>
> >>>>> Thanks Prateek,
> >>>>>
> >>>>> I think from this content it looks to me like we are ready to start a
> >>>>> vote to explicitly accept ALP into Parquet
> >>>>>
> >>>>> Does anyone know of a reason we should postpone it for longer?
> >>>>> Perhaps someone needs some more time to review?
> >>>>>
> >>>>> Andrew
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <pr...@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>>> Hi team,
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Hope everyone is doing well. I got a chance to work through all the
> >>>>>> remaining feedback and update the spec doc. Here are the new
> artifacts
> >>>>>>
> >>>>>> 1) Spec document :
> >>>>>>
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
> >>>>>>
> >>>>>> 2) Spec document in parquet format repo :
> >>>>>> https://github.com/apache/parquet-format/pull/557
> >>>>>>
> >>>>>> 3) Alp implementation in arrow c++ repo :
> >>>>>> https://github.com/apache/arrow/pull/48345/changes
> >>>>>>
> >>>>>> 4) Alp implementation in parquet-java repo : Work for Vinoo and
> Julien
> >>>>>> https://github.com/apache/parquet-java/pull/3397
> >>>>>>
> >>>>>> 5) PR with test and benchmarking artifacts in parquet-testing repo :
> >>>>>> https://github.com/apache/parquet-testing/pull/100
> >>>>>>
> >>>>>>
> >>>>>> And
> >>>>>>
> >>>>>>
> >>>>>> - Go : Arnav just submitted an in progress implementation in Go.
> >>>>>> https://github.com/apache/arrow-go/pull/704 (I haven't started
> >>>>>> looking at it yet)
> >>>>>> - Rust : I remember Andrew mentioned that this work is also in
> >>>>>> progress (So 4 languages!)
> >>>>>>
> >>>>>>
> >>>>>> *Arrow C++ implementation *
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> The PR is out and was also used by Antoine to report the numbers as
> >>>>>> reported here. Micah and Konstantin have given 1 round of feedback
> >>>>>> and I'm addressing them today. Please note that the default
> >>>>>> optimization flag for compiling is O2 and not Q3. I got around 70%
> >>>>>> performance improvement in the decoding speed when using the O3
> flag.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> *Parqet-MR Java implementation (working with Vinoo and Julien) and
> **Cross
> >>>>>> Language testing*
> >>>>>>
> >>>>>>
> >>>>>> Let me know if you have any questions or feedback.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Now pasting some performance numbers
> >>>>>>
> >>>>>>
> >>>>>> Table 1: C++ ALP Double Decode — Spotify Columns (Graviton 3, ARM
> >>>>>> Neoverse V1)
> >>>>>>
> >>>>>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
> >>>>>>
> >>>>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │ Speedup │
> >>>>>>
> >>>>>> ├──────────────────┼──────────────┼──────────────┼─────────┤
> >>>>>>
> >>>>>> │ valence │ 3,155 │ 5,523 │ 1.75x │
> >>>>>>
> >>>>>> │ danceability │ 3,233 │ 5,685 │ 1.76x │
> >>>>>>
> >>>>>> │ energy │ 3,197 │ 5,652 │ 1.77x │
> >>>>>>
> >>>>>> │ loudness │ 3,186 │ 5,473 │ 1.72x │
> >>>>>>
> >>>>>> └──────────────────┴──────────────┴──────────────┴─────────┘
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <pr...@gmail.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> @Micah Kornfield <em...@gmail.com> : Got it.
> >>>>>>>
> >>>>>>> @Andrew Lamb <an...@gmail.com>
> >>>>>>>
> >>>>>>>
> >>>>>>>> Do you think it would be good to start moving the spec development
> >>>>>>>> into
> >>>>>>>> markdown format, in preparation for finalizing it?
> >>>>>>>>
> >>>>>>>
> >>>>>>> Yes I'll update the numbers for some of the examples I have in the
> >>>>>>> spec based
> >>>>>>> on the updated header size. Then we should be good to go for the
> >>>>>>> markdown format.
> >>>>>>>
> >>>>>>> Thanks everyone!
> >>>>>>>
> >>>>>>>
> >>>>>>>>
> >>>>>>>> Andrew
> >>>>>>>>
> >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <pr...@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Hi team,
> >>>>>>>>>
> >>>>>>>>> 1) Andrew
> >>>>>>>>>
> >>>>>>>>> - Thanks for working on test files. My PR did add all the
> test
> >>>>>>>> files I
> >>>>>>>>> used to benchmark on datasets. Maybe we can club it together.
> >>>>>>>> WIll also
> >>>>>>>>> aid
> >>>>>>>>> cross language testing
> >>>>>>>>> - Kosta Tarasov working on Rust implementation. This is
> great.
> >>>>>>>> Thanks
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> 2) Antoine
> >>>>>>>>>
> >>>>>>>>> - Thanks a lot for reporting the numbers on AMD. Looks like
> you
> >>>>>>>> are
> >>>>>>>>> getting 8X the decoding performance of BSS. This is
> amazing!!.
> >>>>>>>>> - Thanks for acknowledging the sampling design.
> >>>>>>>>> - I agree with you on Fastlanes. In some crude experiments I
> >>>>>>>> didn't get
> >>>>>>>>> a good perf benefit from it on Graviton3 (but maybe there was
> >>>>>>>> something
> >>>>>>>>> wrong with my implementation).
> >>>>>>>>> - Locking the 16bit exception encoding for the spec in this
> >>>>>>>> case.
> >>>>>>>>> - Awesome I think we have solved for all open questions minus
> >>>>>>>> the
> >>>>>>>>> version byte :). (will get back on this soon)
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> 3) Micah
> >>>>>>>>>
> >>>>>>>>> - FastLanes : The current spec does allow for using FastLane
> >>>>>>>> with the
> >>>>>>>>> configurable enum value for layout. We should be able to
> inject
> >>>>>>>> any
> >>>>>>>>> layout
> >>>>>>>>> in the current design.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Working on resolving all remaining open comments on the spec this
> >>>>>>>> week.
> >>>>>>>>>
> >>>>>>>>> Best
> >>>>>>>>> Prateek
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <
> >>>>>>>> stevel@cloudera.com>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
> >>>>>>>> emkornfield@gmail.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> It looks like the actual issue described for ORC in the paper
> >>>>>>>> is that
> >>>>>>>>> it
> >>>>>>>>>>> has multiple sub-encodings in a batch. This is different then
> >>>>>>>> the
> >>>>>>>>> design
> >>>>>>>>>>> proposed here where there is still fixed encoding per page in
> >>>>>>>> parquet.
> >>>>>>>>>>> Given reasonably sized pages I don't think branch
> >>>>>>>> misprediction should
> >>>>>>>>>> be a
> >>>>>>>>>>> big issue for new encodings. I agree that we should be
> >>>>>>>> conservative in
> >>>>>>>>>>> general for adding new encodings.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>> +1
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >
>
>
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Antoine Pitrou <an...@python.org>.
Hello,
Apologies if the question was already asked, but should we care about
FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
BYTE_STREAM_SPLIT + LZ4 for example?
Regards
Antoine.
Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
> Thanks Andrew and Micah for review feedback on the two PR's
> 1) (c++ arrow repo) https://github.com/apache/arrow/pull/48345/changes
> 2) (parquet-format repo) https://github.com/apache/parquet-format/pull/557
>
> I have addressed all (unless I missed something) comments on the two PR's.
>
> Best
> Prateek
>
> On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <pr...@gmail.com> wrote:
>
>> Thanks Andrew and Micah.
>>
>> `fair amount of feedback on at least the implementations`
>> For the c++ I have already started addressing the feedback, I should be
>> done with that Monday/Tuesday.
>> I think Vinoo too has been making good progress on the Java implementation.
>>
>> Best
>> Prateek
>>
>> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <an...@gmail.com>
>> wrote:
>>
>>> Got it. Thank you for the clarification -- I will try and look into the
>>> spec and the Rust implementation[1] in this next week
>>>
>>> [1]: https://github.com/apache/arrow-rs/pull/9372
>>>
>>> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <em...@gmail.com>
>>> wrote:
>>>
>>>> Hi Andrew,
>>>> I think there is a fair amount of feedback on at least the
>>>> implementations, typically I think we've waited till they are close to
>>>> mergeable before a final vote. Otherwise I agree we are very close.
>>>>
>>>> -Micah
>>>>
>>>> On Saturday, April 25, 2026, Andrew Lamb <an...@gmail.com> wrote:
>>>>
>>>>> Thanks Prateek,
>>>>>
>>>>> I think from this content it looks to me like we are ready to start a
>>>>> vote to explicitly accept ALP into Parquet
>>>>>
>>>>> Does anyone know of a reason we should postpone it for longer?
>>>>> Perhaps someone needs some more time to review?
>>>>>
>>>>> Andrew
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <pr...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi team,
>>>>>>
>>>>>>
>>>>>>
>>>>>> Hope everyone is doing well. I got a chance to work through all the
>>>>>> remaining feedback and update the spec doc. Here are the new artifacts
>>>>>>
>>>>>> 1) Spec document :
>>>>>> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
>>>>>>
>>>>>> 2) Spec document in parquet format repo :
>>>>>> https://github.com/apache/parquet-format/pull/557
>>>>>>
>>>>>> 3) Alp implementation in arrow c++ repo :
>>>>>> https://github.com/apache/arrow/pull/48345/changes
>>>>>>
>>>>>> 4) Alp implementation in parquet-java repo : Work for Vinoo and Julien
>>>>>> https://github.com/apache/parquet-java/pull/3397
>>>>>>
>>>>>> 5) PR with test and benchmarking artifacts in parquet-testing repo :
>>>>>> https://github.com/apache/parquet-testing/pull/100
>>>>>>
>>>>>>
>>>>>> And
>>>>>>
>>>>>>
>>>>>> - Go : Arnav just submitted an in progress implementation in Go.
>>>>>> https://github.com/apache/arrow-go/pull/704 (I haven't started
>>>>>> looking at it yet)
>>>>>> - Rust : I remember Andrew mentioned that this work is also in
>>>>>> progress (So 4 languages!)
>>>>>>
>>>>>>
>>>>>> *Arrow C++ implementation *
>>>>>>
>>>>>>
>>>>>>
>>>>>> The PR is out and was also used by Antoine to report the numbers as
>>>>>> reported here. Micah and Konstantin have given 1 round of feedback
>>>>>> and I'm addressing them today. Please note that the default
>>>>>> optimization flag for compiling is O2 and not Q3. I got around 70%
>>>>>> performance improvement in the decoding speed when using the O3 flag.
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Parqet-MR Java implementation (working with Vinoo and Julien) and **Cross
>>>>>> Language testing*
>>>>>>
>>>>>>
>>>>>> Let me know if you have any questions or feedback.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Now pasting some performance numbers
>>>>>>
>>>>>>
>>>>>> Table 1: C++ ALP Double Decode — Spotify Columns (Graviton 3, ARM
>>>>>> Neoverse V1)
>>>>>>
>>>>>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
>>>>>>
>>>>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │ Speedup │
>>>>>>
>>>>>> ├──────────────────┼──────────────┼──────────────┼─────────┤
>>>>>>
>>>>>> │ valence │ 3,155 │ 5,523 │ 1.75x │
>>>>>>
>>>>>> │ danceability │ 3,233 │ 5,685 │ 1.76x │
>>>>>>
>>>>>> │ energy │ 3,197 │ 5,652 │ 1.77x │
>>>>>>
>>>>>> │ loudness │ 3,186 │ 5,473 │ 1.72x │
>>>>>>
>>>>>> └──────────────────┴──────────────┴──────────────┴─────────┘
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <pr...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> @Micah Kornfield <em...@gmail.com> : Got it.
>>>>>>>
>>>>>>> @Andrew Lamb <an...@gmail.com>
>>>>>>>
>>>>>>>
>>>>>>>> Do you think it would be good to start moving the spec development
>>>>>>>> into
>>>>>>>> markdown format, in preparation for finalizing it?
>>>>>>>>
>>>>>>>
>>>>>>> Yes I'll update the numbers for some of the examples I have in the
>>>>>>> spec based
>>>>>>> on the updated header size. Then we should be good to go for the
>>>>>>> markdown format.
>>>>>>>
>>>>>>> Thanks everyone!
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Andrew
>>>>>>>>
>>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <pr...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi team,
>>>>>>>>>
>>>>>>>>> 1) Andrew
>>>>>>>>>
>>>>>>>>> - Thanks for working on test files. My PR did add all the test
>>>>>>>> files I
>>>>>>>>> used to benchmark on datasets. Maybe we can club it together.
>>>>>>>> WIll also
>>>>>>>>> aid
>>>>>>>>> cross language testing
>>>>>>>>> - Kosta Tarasov working on Rust implementation. This is great.
>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2) Antoine
>>>>>>>>>
>>>>>>>>> - Thanks a lot for reporting the numbers on AMD. Looks like you
>>>>>>>> are
>>>>>>>>> getting 8X the decoding performance of BSS. This is amazing!!.
>>>>>>>>> - Thanks for acknowledging the sampling design.
>>>>>>>>> - I agree with you on Fastlanes. In some crude experiments I
>>>>>>>> didn't get
>>>>>>>>> a good perf benefit from it on Graviton3 (but maybe there was
>>>>>>>> something
>>>>>>>>> wrong with my implementation).
>>>>>>>>> - Locking the 16bit exception encoding for the spec in this
>>>>>>>> case.
>>>>>>>>> - Awesome I think we have solved for all open questions minus
>>>>>>>> the
>>>>>>>>> version byte :). (will get back on this soon)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 3) Micah
>>>>>>>>>
>>>>>>>>> - FastLanes : The current spec does allow for using FastLane
>>>>>>>> with the
>>>>>>>>> configurable enum value for layout. We should be able to inject
>>>>>>>> any
>>>>>>>>> layout
>>>>>>>>> in the current design.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Working on resolving all remaining open comments on the spec this
>>>>>>>> week.
>>>>>>>>>
>>>>>>>>> Best
>>>>>>>>> Prateek
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <
>>>>>>>> stevel@cloudera.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
>>>>>>>> emkornfield@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> It looks like the actual issue described for ORC in the paper
>>>>>>>> is that
>>>>>>>>> it
>>>>>>>>>>> has multiple sub-encodings in a batch. This is different then
>>>>>>>> the
>>>>>>>>> design
>>>>>>>>>>> proposed here where there is still fixed encoding per page in
>>>>>>>> parquet.
>>>>>>>>>>> Given reasonably sized pages I don't think branch
>>>>>>>> misprediction should
>>>>>>>>>> be a
>>>>>>>>>>> big issue for new encodings. I agree that we should be
>>>>>>>> conservative in
>>>>>>>>>>> general for adding new encodings.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> +1
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by PRATEEK GAUR <pr...@gmail.com>.
Thanks Andrew and Micah for review feedback on the two PR's
1) (c++ arrow repo) https://github.com/apache/arrow/pull/48345/changes
2) (parquet-format repo) https://github.com/apache/parquet-format/pull/557
I have addressed all (unless I missed something) comments on the two PR's.
Best
Prateek
On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <pr...@gmail.com> wrote:
> Thanks Andrew and Micah.
>
> `fair amount of feedback on at least the implementations`
> For the c++ I have already started addressing the feedback, I should be
> done with that Monday/Tuesday.
> I think Vinoo too has been making good progress on the Java implementation.
>
> Best
> Prateek
>
> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <an...@gmail.com>
> wrote:
>
>> Got it. Thank you for the clarification -- I will try and look into the
>> spec and the Rust implementation[1] in this next week
>>
>> [1]: https://github.com/apache/arrow-rs/pull/9372
>>
>> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <em...@gmail.com>
>> wrote:
>>
>>> Hi Andrew,
>>> I think there is a fair amount of feedback on at least the
>>> implementations, typically I think we've waited till they are close to
>>> mergeable before a final vote. Otherwise I agree we are very close.
>>>
>>> -Micah
>>>
>>> On Saturday, April 25, 2026, Andrew Lamb <an...@gmail.com> wrote:
>>>
>>>> Thanks Prateek,
>>>>
>>>> I think from this content it looks to me like we are ready to start a
>>>> vote to explicitly accept ALP into Parquet
>>>>
>>>> Does anyone know of a reason we should postpone it for longer?
>>>> Perhaps someone needs some more time to review?
>>>>
>>>> Andrew
>>>>
>>>>
>>>>
>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <pr...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi team,
>>>>>
>>>>>
>>>>>
>>>>> Hope everyone is doing well. I got a chance to work through all the
>>>>> remaining feedback and update the spec doc. Here are the new artifacts
>>>>>
>>>>> 1) Spec document :
>>>>> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
>>>>>
>>>>> 2) Spec document in parquet format repo :
>>>>> https://github.com/apache/parquet-format/pull/557
>>>>>
>>>>> 3) Alp implementation in arrow c++ repo :
>>>>> https://github.com/apache/arrow/pull/48345/changes
>>>>>
>>>>> 4) Alp implementation in parquet-java repo : Work for Vinoo and Julien
>>>>> https://github.com/apache/parquet-java/pull/3397
>>>>>
>>>>> 5) PR with test and benchmarking artifacts in parquet-testing repo :
>>>>> https://github.com/apache/parquet-testing/pull/100
>>>>>
>>>>>
>>>>> And
>>>>>
>>>>>
>>>>> - Go : Arnav just submitted an in progress implementation in Go.
>>>>> https://github.com/apache/arrow-go/pull/704 (I haven't started
>>>>> looking at it yet)
>>>>> - Rust : I remember Andrew mentioned that this work is also in
>>>>> progress (So 4 languages!)
>>>>>
>>>>>
>>>>> *Arrow C++ implementation *
>>>>>
>>>>>
>>>>>
>>>>> The PR is out and was also used by Antoine to report the numbers as
>>>>> reported here. Micah and Konstantin have given 1 round of feedback
>>>>> and I'm addressing them today. Please note that the default
>>>>> optimization flag for compiling is O2 and not Q3. I got around 70%
>>>>> performance improvement in the decoding speed when using the O3 flag.
>>>>>
>>>>>
>>>>>
>>>>> *Parqet-MR Java implementation (working with Vinoo and Julien) and **Cross
>>>>> Language testing*
>>>>>
>>>>>
>>>>> Let me know if you have any questions or feedback.
>>>>>
>>>>>
>>>>>
>>>>> Now pasting some performance numbers
>>>>>
>>>>>
>>>>> Table 1: C++ ALP Double Decode — Spotify Columns (Graviton 3, ARM
>>>>> Neoverse V1)
>>>>>
>>>>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
>>>>>
>>>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │ Speedup │
>>>>>
>>>>> ├──────────────────┼──────────────┼──────────────┼─────────┤
>>>>>
>>>>> │ valence │ 3,155 │ 5,523 │ 1.75x │
>>>>>
>>>>> │ danceability │ 3,233 │ 5,685 │ 1.76x │
>>>>>
>>>>> │ energy │ 3,197 │ 5,652 │ 1.77x │
>>>>>
>>>>> │ loudness │ 3,186 │ 5,473 │ 1.72x │
>>>>>
>>>>> └──────────────────┴──────────────┴──────────────┴─────────┘
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <pr...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> @Micah Kornfield <em...@gmail.com> : Got it.
>>>>>>
>>>>>> @Andrew Lamb <an...@gmail.com>
>>>>>>
>>>>>>
>>>>>>> Do you think it would be good to start moving the spec development
>>>>>>> into
>>>>>>> markdown format, in preparation for finalizing it?
>>>>>>>
>>>>>>
>>>>>> Yes I'll update the numbers for some of the examples I have in the
>>>>>> spec based
>>>>>> on the updated header size. Then we should be good to go for the
>>>>>> markdown format.
>>>>>>
>>>>>> Thanks everyone!
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Andrew
>>>>>>>
>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <pr...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> > Hi team,
>>>>>>> >
>>>>>>> > 1) Andrew
>>>>>>> >
>>>>>>> > - Thanks for working on test files. My PR did add all the test
>>>>>>> files I
>>>>>>> > used to benchmark on datasets. Maybe we can club it together.
>>>>>>> WIll also
>>>>>>> > aid
>>>>>>> > cross language testing
>>>>>>> > - Kosta Tarasov working on Rust implementation. This is great.
>>>>>>> Thanks
>>>>>>> >
>>>>>>> >
>>>>>>> > 2) Antoine
>>>>>>> >
>>>>>>> > - Thanks a lot for reporting the numbers on AMD. Looks like you
>>>>>>> are
>>>>>>> > getting 8X the decoding performance of BSS. This is amazing!!.
>>>>>>> > - Thanks for acknowledging the sampling design.
>>>>>>> > - I agree with you on Fastlanes. In some crude experiments I
>>>>>>> didn't get
>>>>>>> > a good perf benefit from it on Graviton3 (but maybe there was
>>>>>>> something
>>>>>>> > wrong with my implementation).
>>>>>>> > - Locking the 16bit exception encoding for the spec in this
>>>>>>> case.
>>>>>>> > - Awesome I think we have solved for all open questions minus
>>>>>>> the
>>>>>>> > version byte :). (will get back on this soon)
>>>>>>> >
>>>>>>> >
>>>>>>> > 3) Micah
>>>>>>> >
>>>>>>> > - FastLanes : The current spec does allow for using FastLane
>>>>>>> with the
>>>>>>> > configurable enum value for layout. We should be able to inject
>>>>>>> any
>>>>>>> > layout
>>>>>>> > in the current design.
>>>>>>> >
>>>>>>> >
>>>>>>> > Working on resolving all remaining open comments on the spec this
>>>>>>> week.
>>>>>>> >
>>>>>>> > Best
>>>>>>> > Prateek
>>>>>>> >
>>>>>>> >
>>>>>>> > On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <
>>>>>>> stevel@cloudera.com>
>>>>>>> > wrote:
>>>>>>> >
>>>>>>> > > On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
>>>>>>> emkornfield@gmail.com>
>>>>>>> > > wrote:
>>>>>>> > >
>>>>>>> > > >
>>>>>>> > > >
>>>>>>> > > > It looks like the actual issue described for ORC in the paper
>>>>>>> is that
>>>>>>> > it
>>>>>>> > > > has multiple sub-encodings in a batch. This is different then
>>>>>>> the
>>>>>>> > design
>>>>>>> > > > proposed here where there is still fixed encoding per page in
>>>>>>> parquet.
>>>>>>> > > > Given reasonably sized pages I don't think branch
>>>>>>> misprediction should
>>>>>>> > > be a
>>>>>>> > > > big issue for new encodings. I agree that we should be
>>>>>>> conservative in
>>>>>>> > > > general for adding new encodings.
>>>>>>> > > >
>>>>>>> > > >
>>>>>>> > > +1
>>>>>>> > >
>>>>>>> >
>>>>>>>
>>>>>>
Re: [Parquet] ALP Encoding for Floating point data
Posted by PRATEEK GAUR <pr...@gmail.com>.
Thanks Andrew and Micah.
`fair amount of feedback on at least the implementations`
For the c++ I have already started addressing the feedback, I should be
done with that Monday/Tuesday.
I think Vinoo too has been making good progress on the Java implementation.
Best
Prateek
On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <an...@gmail.com> wrote:
> Got it. Thank you for the clarification -- I will try and look into the
> spec and the Rust implementation[1] in this next week
>
> [1]: https://github.com/apache/arrow-rs/pull/9372
>
> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <em...@gmail.com>
> wrote:
>
>> Hi Andrew,
>> I think there is a fair amount of feedback on at least the
>> implementations, typically I think we've waited till they are close to
>> mergeable before a final vote. Otherwise I agree we are very close.
>>
>> -Micah
>>
>> On Saturday, April 25, 2026, Andrew Lamb <an...@gmail.com> wrote:
>>
>>> Thanks Prateek,
>>>
>>> I think from this content it looks to me like we are ready to start a
>>> vote to explicitly accept ALP into Parquet
>>>
>>> Does anyone know of a reason we should postpone it for longer?
>>> Perhaps someone needs some more time to review?
>>>
>>> Andrew
>>>
>>>
>>>
>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <pr...@gmail.com> wrote:
>>>
>>>> Hi team,
>>>>
>>>>
>>>>
>>>> Hope everyone is doing well. I got a chance to work through all the
>>>> remaining feedback and update the spec doc. Here are the new artifacts
>>>>
>>>> 1) Spec document :
>>>> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
>>>>
>>>> 2) Spec document in parquet format repo :
>>>> https://github.com/apache/parquet-format/pull/557
>>>>
>>>> 3) Alp implementation in arrow c++ repo :
>>>> https://github.com/apache/arrow/pull/48345/changes
>>>>
>>>> 4) Alp implementation in parquet-java repo : Work for Vinoo and Julien
>>>> https://github.com/apache/parquet-java/pull/3397
>>>>
>>>> 5) PR with test and benchmarking artifacts in parquet-testing repo :
>>>> https://github.com/apache/parquet-testing/pull/100
>>>>
>>>>
>>>> And
>>>>
>>>>
>>>> - Go : Arnav just submitted an in progress implementation in Go.
>>>> https://github.com/apache/arrow-go/pull/704 (I haven't started
>>>> looking at it yet)
>>>> - Rust : I remember Andrew mentioned that this work is also in
>>>> progress (So 4 languages!)
>>>>
>>>>
>>>> *Arrow C++ implementation *
>>>>
>>>>
>>>>
>>>> The PR is out and was also used by Antoine to report the numbers as
>>>> reported here. Micah and Konstantin have given 1 round of feedback and
>>>> I'm addressing them today. Please note that the default optimization
>>>> flag for compiling is O2 and not Q3. I got around 70% performance
>>>> improvement in the decoding speed when using the O3 flag.
>>>>
>>>>
>>>>
>>>> *Parqet-MR Java implementation (working with Vinoo and Julien) and **Cross
>>>> Language testing*
>>>>
>>>>
>>>> Let me know if you have any questions or feedback.
>>>>
>>>>
>>>>
>>>> Now pasting some performance numbers
>>>>
>>>>
>>>> Table 1: C++ ALP Double Decode — Spotify Columns (Graviton 3, ARM
>>>> Neoverse V1)
>>>>
>>>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
>>>>
>>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │ Speedup │
>>>>
>>>> ├──────────────────┼──────────────┼──────────────┼─────────┤
>>>>
>>>> │ valence │ 3,155 │ 5,523 │ 1.75x │
>>>>
>>>> │ danceability │ 3,233 │ 5,685 │ 1.76x │
>>>>
>>>> │ energy │ 3,197 │ 5,652 │ 1.77x │
>>>>
>>>> │ loudness │ 3,186 │ 5,473 │ 1.72x │
>>>>
>>>> └──────────────────┴──────────────┴──────────────┴─────────┘
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <pr...@gmail.com>
>>>> wrote:
>>>>
>>>>> @Micah Kornfield <em...@gmail.com> : Got it.
>>>>>
>>>>> @Andrew Lamb <an...@gmail.com>
>>>>>
>>>>>
>>>>>> Do you think it would be good to start moving the spec development
>>>>>> into
>>>>>> markdown format, in preparation for finalizing it?
>>>>>>
>>>>>
>>>>> Yes I'll update the numbers for some of the examples I have in the
>>>>> spec based
>>>>> on the updated header size. Then we should be good to go for the
>>>>> markdown format.
>>>>>
>>>>> Thanks everyone!
>>>>>
>>>>>
>>>>>>
>>>>>> Andrew
>>>>>>
>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <pr...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> > Hi team,
>>>>>> >
>>>>>> > 1) Andrew
>>>>>> >
>>>>>> > - Thanks for working on test files. My PR did add all the test
>>>>>> files I
>>>>>> > used to benchmark on datasets. Maybe we can club it together.
>>>>>> WIll also
>>>>>> > aid
>>>>>> > cross language testing
>>>>>> > - Kosta Tarasov working on Rust implementation. This is great.
>>>>>> Thanks
>>>>>> >
>>>>>> >
>>>>>> > 2) Antoine
>>>>>> >
>>>>>> > - Thanks a lot for reporting the numbers on AMD. Looks like you
>>>>>> are
>>>>>> > getting 8X the decoding performance of BSS. This is amazing!!.
>>>>>> > - Thanks for acknowledging the sampling design.
>>>>>> > - I agree with you on Fastlanes. In some crude experiments I
>>>>>> didn't get
>>>>>> > a good perf benefit from it on Graviton3 (but maybe there was
>>>>>> something
>>>>>> > wrong with my implementation).
>>>>>> > - Locking the 16bit exception encoding for the spec in this case.
>>>>>> > - Awesome I think we have solved for all open questions minus the
>>>>>> > version byte :). (will get back on this soon)
>>>>>> >
>>>>>> >
>>>>>> > 3) Micah
>>>>>> >
>>>>>> > - FastLanes : The current spec does allow for using FastLane
>>>>>> with the
>>>>>> > configurable enum value for layout. We should be able to inject
>>>>>> any
>>>>>> > layout
>>>>>> > in the current design.
>>>>>> >
>>>>>> >
>>>>>> > Working on resolving all remaining open comments on the spec this
>>>>>> week.
>>>>>> >
>>>>>> > Best
>>>>>> > Prateek
>>>>>> >
>>>>>> >
>>>>>> > On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <stevel@cloudera.com
>>>>>> >
>>>>>> > wrote:
>>>>>> >
>>>>>> > > On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
>>>>>> emkornfield@gmail.com>
>>>>>> > > wrote:
>>>>>> > >
>>>>>> > > >
>>>>>> > > >
>>>>>> > > > It looks like the actual issue described for ORC in the paper
>>>>>> is that
>>>>>> > it
>>>>>> > > > has multiple sub-encodings in a batch. This is different then
>>>>>> the
>>>>>> > design
>>>>>> > > > proposed here where there is still fixed encoding per page in
>>>>>> parquet.
>>>>>> > > > Given reasonably sized pages I don't think branch misprediction
>>>>>> should
>>>>>> > > be a
>>>>>> > > > big issue for new encodings. I agree that we should be
>>>>>> conservative in
>>>>>> > > > general for adding new encodings.
>>>>>> > > >
>>>>>> > > >
>>>>>> > > +1
>>>>>> > >
>>>>>> >
>>>>>>
>>>>>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Andrew Lamb <an...@gmail.com>.
Got it. Thank you for the clarification -- I will try and look into the
spec and the Rust implementation[1] in this next week
[1]: https://github.com/apache/arrow-rs/pull/9372
On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <em...@gmail.com>
wrote:
> Hi Andrew,
> I think there is a fair amount of feedback on at least the
> implementations, typically I think we've waited till they are close to
> mergeable before a final vote. Otherwise I agree we are very close.
>
> -Micah
>
> On Saturday, April 25, 2026, Andrew Lamb <an...@gmail.com> wrote:
>
>> Thanks Prateek,
>>
>> I think from this content it looks to me like we are ready to start a
>> vote to explicitly accept ALP into Parquet
>>
>> Does anyone know of a reason we should postpone it for longer?
>> Perhaps someone needs some more time to review?
>>
>> Andrew
>>
>>
>>
>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <pr...@gmail.com> wrote:
>>
>>> Hi team,
>>>
>>>
>>>
>>> Hope everyone is doing well. I got a chance to work through all the
>>> remaining feedback and update the spec doc. Here are the new artifacts
>>>
>>> 1) Spec document :
>>> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
>>>
>>> 2) Spec document in parquet format repo :
>>> https://github.com/apache/parquet-format/pull/557
>>>
>>> 3) Alp implementation in arrow c++ repo :
>>> https://github.com/apache/arrow/pull/48345/changes
>>>
>>> 4) Alp implementation in parquet-java repo : Work for Vinoo and Julien
>>> https://github.com/apache/parquet-java/pull/3397
>>>
>>> 5) PR with test and benchmarking artifacts in parquet-testing repo :
>>> https://github.com/apache/parquet-testing/pull/100
>>>
>>>
>>> And
>>>
>>>
>>> - Go : Arnav just submitted an in progress implementation in Go.
>>> https://github.com/apache/arrow-go/pull/704 (I haven't started
>>> looking at it yet)
>>> - Rust : I remember Andrew mentioned that this work is also in
>>> progress (So 4 languages!)
>>>
>>>
>>> *Arrow C++ implementation *
>>>
>>>
>>>
>>> The PR is out and was also used by Antoine to report the numbers as
>>> reported here. Micah and Konstantin have given 1 round of feedback and
>>> I'm addressing them today. Please note that the default optimization
>>> flag for compiling is O2 and not Q3. I got around 70% performance
>>> improvement in the decoding speed when using the O3 flag.
>>>
>>>
>>>
>>> *Parqet-MR Java implementation (working with Vinoo and Julien) and **Cross
>>> Language testing*
>>>
>>>
>>> Let me know if you have any questions or feedback.
>>>
>>>
>>>
>>> Now pasting some performance numbers
>>>
>>>
>>> Table 1: C++ ALP Double Decode — Spotify Columns (Graviton 3, ARM
>>> Neoverse V1)
>>>
>>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
>>>
>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │ Speedup │
>>>
>>> ├──────────────────┼──────────────┼──────────────┼─────────┤
>>>
>>> │ valence │ 3,155 │ 5,523 │ 1.75x │
>>>
>>> │ danceability │ 3,233 │ 5,685 │ 1.76x │
>>>
>>> │ energy │ 3,197 │ 5,652 │ 1.77x │
>>>
>>> │ loudness │ 3,186 │ 5,473 │ 1.72x │
>>>
>>> └──────────────────┴──────────────┴──────────────┴─────────┘
>>>
>>>
>>>
>>>
>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <pr...@gmail.com> wrote:
>>>
>>>> @Micah Kornfield <em...@gmail.com> : Got it.
>>>>
>>>> @Andrew Lamb <an...@gmail.com>
>>>>
>>>>
>>>>> Do you think it would be good to start moving the spec development into
>>>>> markdown format, in preparation for finalizing it?
>>>>>
>>>>
>>>> Yes I'll update the numbers for some of the examples I have in the spec
>>>> based
>>>> on the updated header size. Then we should be good to go for the
>>>> markdown format.
>>>>
>>>> Thanks everyone!
>>>>
>>>>
>>>>>
>>>>> Andrew
>>>>>
>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <pr...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> > Hi team,
>>>>> >
>>>>> > 1) Andrew
>>>>> >
>>>>> > - Thanks for working on test files. My PR did add all the test
>>>>> files I
>>>>> > used to benchmark on datasets. Maybe we can club it together.
>>>>> WIll also
>>>>> > aid
>>>>> > cross language testing
>>>>> > - Kosta Tarasov working on Rust implementation. This is great.
>>>>> Thanks
>>>>> >
>>>>> >
>>>>> > 2) Antoine
>>>>> >
>>>>> > - Thanks a lot for reporting the numbers on AMD. Looks like you
>>>>> are
>>>>> > getting 8X the decoding performance of BSS. This is amazing!!.
>>>>> > - Thanks for acknowledging the sampling design.
>>>>> > - I agree with you on Fastlanes. In some crude experiments I
>>>>> didn't get
>>>>> > a good perf benefit from it on Graviton3 (but maybe there was
>>>>> something
>>>>> > wrong with my implementation).
>>>>> > - Locking the 16bit exception encoding for the spec in this case.
>>>>> > - Awesome I think we have solved for all open questions minus the
>>>>> > version byte :). (will get back on this soon)
>>>>> >
>>>>> >
>>>>> > 3) Micah
>>>>> >
>>>>> > - FastLanes : The current spec does allow for using FastLane with
>>>>> the
>>>>> > configurable enum value for layout. We should be able to inject
>>>>> any
>>>>> > layout
>>>>> > in the current design.
>>>>> >
>>>>> >
>>>>> > Working on resolving all remaining open comments on the spec this
>>>>> week.
>>>>> >
>>>>> > Best
>>>>> > Prateek
>>>>> >
>>>>> >
>>>>> > On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <st...@cloudera.com>
>>>>> > wrote:
>>>>> >
>>>>> > > On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
>>>>> emkornfield@gmail.com>
>>>>> > > wrote:
>>>>> > >
>>>>> > > >
>>>>> > > >
>>>>> > > > It looks like the actual issue described for ORC in the paper is
>>>>> that
>>>>> > it
>>>>> > > > has multiple sub-encodings in a batch. This is different then
>>>>> the
>>>>> > design
>>>>> > > > proposed here where there is still fixed encoding per page in
>>>>> parquet.
>>>>> > > > Given reasonably sized pages I don't think branch misprediction
>>>>> should
>>>>> > > be a
>>>>> > > > big issue for new encodings. I agree that we should be
>>>>> conservative in
>>>>> > > > general for adding new encodings.
>>>>> > > >
>>>>> > > >
>>>>> > > +1
>>>>> > >
>>>>> >
>>>>>
>>>>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Micah Kornfield <em...@gmail.com>.
Hi Andrew,
I think there is a fair amount of feedback on at least the implementations,
typically I think we've waited till they are close to mergeable before a
final vote. Otherwise I agree we are very close.
-Micah
On Saturday, April 25, 2026, Andrew Lamb <an...@gmail.com> wrote:
> Thanks Prateek,
>
> I think from this content it looks to me like we are ready to start a vote
> to explicitly accept ALP into Parquet
>
> Does anyone know of a reason we should postpone it for longer?
> Perhaps someone needs some more time to review?
>
> Andrew
>
>
>
> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <pr...@gmail.com> wrote:
>
>> Hi team,
>>
>>
>>
>> Hope everyone is doing well. I got a chance to work through all the
>> remaining feedback and update the spec doc. Here are the new artifacts
>>
>> 1) Spec document : https://docs.google.com/document/d/
>> 1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
>>
>> 2) Spec document in parquet format repo : https://github.com/apache/
>> parquet-format/pull/557
>>
>> 3) Alp implementation in arrow c++ repo : https://github.com/apache/
>> arrow/pull/48345/changes
>>
>> 4) Alp implementation in parquet-java repo : Work for Vinoo and Julien
>> https://github.com/apache/parquet-java/pull/3397
>>
>> 5) PR with test and benchmarking artifacts in parquet-testing repo :
>> https://github.com/apache/parquet-testing/pull/100
>>
>>
>> And
>>
>>
>> - Go : Arnav just submitted an in progress implementation in Go.
>> https://github.com/apache/arrow-go/pull/704
>> <https://github.com/apache/arrow-go/pull/704> (I haven't started
>> looking at it yet)
>> - Rust : I remember Andrew mentioned that this work is also in
>> progress (So 4 languages!)
>>
>>
>> *Arrow C++ implementation *
>>
>>
>>
>> The PR is out and was also used by Antoine to report the numbers as
>> reported here. Micah and Konstantin have given 1 round of feedback and
>> I'm addressing them today. Please note that the default optimization
>> flag for compiling is O2 and not Q3. I got around 70% performance
>> improvement in the decoding speed when using the O3 flag.
>>
>>
>>
>> *Parqet-MR Java implementation (working with Vinoo and Julien) and **Cross
>> Language testing*
>>
>>
>> Let me know if you have any questions or feedback.
>>
>>
>>
>> Now pasting some performance numbers
>>
>>
>> Table 1: C++ ALP Double Decode — Spotify Columns (Graviton 3, ARM
>> Neoverse V1)
>>
>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
>>
>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │ Speedup │
>>
>> ├──────────────────┼──────────────┼──────────────┼─────────┤
>>
>> │ valence │ 3,155 │ 5,523 │ 1.75x │
>>
>> │ danceability │ 3,233 │ 5,685 │ 1.76x │
>>
>> │ energy │ 3,197 │ 5,652 │ 1.77x │
>>
>> │ loudness │ 3,186 │ 5,473 │ 1.72x │
>>
>> └──────────────────┴──────────────┴──────────────┴─────────┘
>>
>>
>>
>>
>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <pr...@gmail.com> wrote:
>>
>>> @Micah Kornfield <em...@gmail.com> : Got it.
>>>
>>> @Andrew Lamb <an...@gmail.com>
>>>
>>>
>>>> Do you think it would be good to start moving the spec development into
>>>> markdown format, in preparation for finalizing it?
>>>>
>>>
>>> Yes I'll update the numbers for some of the examples I have in the spec
>>> based
>>> on the updated header size. Then we should be good to go for the
>>> markdown format.
>>>
>>> Thanks everyone!
>>>
>>>
>>>>
>>>> Andrew
>>>>
>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <pr...@gmail.com>
>>>> wrote:
>>>>
>>>> > Hi team,
>>>> >
>>>> > 1) Andrew
>>>> >
>>>> > - Thanks for working on test files. My PR did add all the test
>>>> files I
>>>> > used to benchmark on datasets. Maybe we can club it together. WIll
>>>> also
>>>> > aid
>>>> > cross language testing
>>>> > - Kosta Tarasov working on Rust implementation. This is great.
>>>> Thanks
>>>> >
>>>> >
>>>> > 2) Antoine
>>>> >
>>>> > - Thanks a lot for reporting the numbers on AMD. Looks like you are
>>>> > getting 8X the decoding performance of BSS. This is amazing!!.
>>>> > - Thanks for acknowledging the sampling design.
>>>> > - I agree with you on Fastlanes. In some crude experiments I
>>>> didn't get
>>>> > a good perf benefit from it on Graviton3 (but maybe there was
>>>> something
>>>> > wrong with my implementation).
>>>> > - Locking the 16bit exception encoding for the spec in this case.
>>>> > - Awesome I think we have solved for all open questions minus the
>>>> > version byte :). (will get back on this soon)
>>>> >
>>>> >
>>>> > 3) Micah
>>>> >
>>>> > - FastLanes : The current spec does allow for using FastLane with
>>>> the
>>>> > configurable enum value for layout. We should be able to inject any
>>>> > layout
>>>> > in the current design.
>>>> >
>>>> >
>>>> > Working on resolving all remaining open comments on the spec this
>>>> week.
>>>> >
>>>> > Best
>>>> > Prateek
>>>> >
>>>> >
>>>> > On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <st...@cloudera.com>
>>>> > wrote:
>>>> >
>>>> > > On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <emkornfield@gmail.com
>>>> >
>>>> > > wrote:
>>>> > >
>>>> > > >
>>>> > > >
>>>> > > > It looks like the actual issue described for ORC in the paper is
>>>> that
>>>> > it
>>>> > > > has multiple sub-encodings in a batch. This is different then the
>>>> > design
>>>> > > > proposed here where there is still fixed encoding per page in
>>>> parquet.
>>>> > > > Given reasonably sized pages I don't think branch misprediction
>>>> should
>>>> > > be a
>>>> > > > big issue for new encodings. I agree that we should be
>>>> conservative in
>>>> > > > general for adding new encodings.
>>>> > > >
>>>> > > >
>>>> > > +1
>>>> > >
>>>> >
>>>>
>>>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Andrew Lamb <an...@gmail.com>.
Thanks Prateek,
I think from this content it looks to me like we are ready to start a vote
to explicitly accept ALP into Parquet
Does anyone know of a reason we should postpone it for longer?
Perhaps someone needs some more time to review?
Andrew
On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <pr...@gmail.com> wrote:
> Hi team,
>
>
>
> Hope everyone is doing well. I got a chance to work through all the
> remaining feedback and update the spec doc. Here are the new artifacts
>
> 1) Spec document :
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
>
> 2) Spec document in parquet format repo :
> https://github.com/apache/parquet-format/pull/557
>
> 3) Alp implementation in arrow c++ repo :
> https://github.com/apache/arrow/pull/48345/changes
>
> 4) Alp implementation in parquet-java repo : Work for Vinoo and Julien
> https://github.com/apache/parquet-java/pull/3397
>
> 5) PR with test and benchmarking artifacts in parquet-testing repo :
> https://github.com/apache/parquet-testing/pull/100
>
>
> And
>
>
> - Go : Arnav just submitted an in progress implementation in Go.
> https://github.com/apache/arrow-go/pull/704 (I haven't started looking
> at it yet)
> - Rust : I remember Andrew mentioned that this work is also in
> progress (So 4 languages!)
>
>
> *Arrow C++ implementation *
>
>
>
> The PR is out and was also used by Antoine to report the numbers as
> reported here. Micah and Konstantin have given 1 round of feedback and
> I'm addressing them today. Please note that the default optimization flag
> for compiling is O2 and not Q3. I got around 70% performance improvement in
> the decoding speed when using the O3 flag.
>
>
>
> *Parqet-MR Java implementation (working with Vinoo and Julien) and **Cross
> Language testing*
>
>
> Let me know if you have any questions or feedback.
>
>
>
> Now pasting some performance numbers
>
>
> Table 1: C++ ALP Double Decode — Spotify Columns (Graviton 3, ARM
> Neoverse V1)
>
> ┌──────────────────┬──────────────┬──────────────┬─────────┐
>
> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │ Speedup │
>
> ├──────────────────┼──────────────┼──────────────┼─────────┤
>
> │ valence │ 3,155 │ 5,523 │ 1.75x │
>
> │ danceability │ 3,233 │ 5,685 │ 1.76x │
>
> │ energy │ 3,197 │ 5,652 │ 1.77x │
>
> │ loudness │ 3,186 │ 5,473 │ 1.72x │
>
> └──────────────────┴──────────────┴──────────────┴─────────┘
>
>
>
>
> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <pr...@gmail.com> wrote:
>
>> @Micah Kornfield <em...@gmail.com> : Got it.
>>
>> @Andrew Lamb <an...@gmail.com>
>>
>>
>>> Do you think it would be good to start moving the spec development into
>>> markdown format, in preparation for finalizing it?
>>>
>>
>> Yes I'll update the numbers for some of the examples I have in the spec
>> based
>> on the updated header size. Then we should be good to go for the markdown
>> format.
>>
>> Thanks everyone!
>>
>>
>>>
>>> Andrew
>>>
>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <pr...@gmail.com> wrote:
>>>
>>> > Hi team,
>>> >
>>> > 1) Andrew
>>> >
>>> > - Thanks for working on test files. My PR did add all the test
>>> files I
>>> > used to benchmark on datasets. Maybe we can club it together. WIll
>>> also
>>> > aid
>>> > cross language testing
>>> > - Kosta Tarasov working on Rust implementation. This is great.
>>> Thanks
>>> >
>>> >
>>> > 2) Antoine
>>> >
>>> > - Thanks a lot for reporting the numbers on AMD. Looks like you are
>>> > getting 8X the decoding performance of BSS. This is amazing!!.
>>> > - Thanks for acknowledging the sampling design.
>>> > - I agree with you on Fastlanes. In some crude experiments I didn't
>>> get
>>> > a good perf benefit from it on Graviton3 (but maybe there was
>>> something
>>> > wrong with my implementation).
>>> > - Locking the 16bit exception encoding for the spec in this case.
>>> > - Awesome I think we have solved for all open questions minus the
>>> > version byte :). (will get back on this soon)
>>> >
>>> >
>>> > 3) Micah
>>> >
>>> > - FastLanes : The current spec does allow for using FastLane with
>>> the
>>> > configurable enum value for layout. We should be able to inject any
>>> > layout
>>> > in the current design.
>>> >
>>> >
>>> > Working on resolving all remaining open comments on the spec this week.
>>> >
>>> > Best
>>> > Prateek
>>> >
>>> >
>>> > On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <st...@cloudera.com>
>>> > wrote:
>>> >
>>> > > On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <em...@gmail.com>
>>> > > wrote:
>>> > >
>>> > > >
>>> > > >
>>> > > > It looks like the actual issue described for ORC in the paper is
>>> that
>>> > it
>>> > > > has multiple sub-encodings in a batch. This is different then the
>>> > design
>>> > > > proposed here where there is still fixed encoding per page in
>>> parquet.
>>> > > > Given reasonably sized pages I don't think branch misprediction
>>> should
>>> > > be a
>>> > > > big issue for new encodings. I agree that we should be
>>> conservative in
>>> > > > general for adding new encodings.
>>> > > >
>>> > > >
>>> > > +1
>>> > >
>>> >
>>>
>>