You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by PRATEEK GAUR <pr...@gmail.com> on 2026/02/03 22:39:37 UTC
Re: [Parquet] ALP Encoding for Floating point data
Hi Antoine and Micah,
Apologies for getting back on this a little late.
*Running Perf tests*
@Antoine Pitrou <an...@python.org> were you able to figure out the steps
to run the tests?
*Sampling Frequency*
We want to pick the right parameters to encode the values with. That is
what the Spec requires.
From the implementation perspective you raise a good point that did cross my
mind that 'practically we don't want to sample for every page', for
performance
reasons. My thinking is each engine is free to decide this.
1) Do it at page level if data is changing often
2) Provide fixed presets via config
3) Do it once per encoder (per column, as Micah pointed out)
4) Provide a fancy config.
I agree with Micah here '*I think maybe should clarify that the*
*encoding algorithm in a specification is a recommendation'*
*Number of values to pick for sampling*
'why does this have to be a constant'
You are right, it doesn't need to be a constant, hence the spec doesn't say
so.
Everything that is segregated out in the AlpContants(c++ impl) file can be
changed by
configuration.
(Did I get your question right @Antoine Pitrou <an...@python.org> ?)
*Alp Version in Header*
Micah's point
*`* I'd suggest modelling fundamentally different algorithms with the
top level encoding enum, and have versioning/control bits where we believe
we will likely want to iterate`
Yes this is exactly what is happening here. An enum to add AlpRd(and more)
and version control to iterate anything fundamental (like a layout change
of the
metadata).
Re-stating the points so that scrolling is not needed.
1. Change of integer encoding (see debate in this thread on FOR vs
Delta). We also want to get fast lanes in at some point. I think an
enum inside the page for versioning makes sense, as it allows for easier
composability.
2. Change in structure to exceptions (e.g. G-ALP). G-ALP comes with some
trade-offs, so it is not clear if it is something everyone would want to
enable.
3. Offset indexes to vectors
4. Different floating point encoding algorithms (e.g. AlpRd + AlpBSS)
For eg at this point I do see that both bitpacking of exceptions, as
pointed by
Antoine, or plain ub2 encoding should work equally well . I was running
some benchmarks
here and I was getting read speeds of around 20 GB/s for 10bit packed
values which is
quite good enough (graviton 3 processor).
But for simplicity (and the fact that we won't get really large vectors, my
inclination
is towards ub2 values) but I want to keep the path open to possibly have
bipacking
as an option as the workloads evolve to a level we haven't thought about
yet. We
can always add a new encoding but I don't see a path to having 20+ top level
encodings. Again I don't have a very strong bias towards keeping it or
removing it
but my thinking right now is let's have the flexibility and make it easier
for people
to evolve this encoding behind a version byte.
Best
Prateek
PS : I probably have addressed all open threads raised by Antoine and Micah.
(but I may have missed something)
On Thu, Jan 29, 2026 at 10:52 PM Micah Kornfield <em...@gmail.com>
wrote:
> Hi Antoine and Prateek,
>
> > > In Parquet C++, encoding happens at page level, and I would guess other
> > > implementations do something similar. Sampling cannot reasonably be
> done
> > > at a higher level, that would require invasive architectural changes.
>
>
> At least in C++ I believe we cache the encoder at a column level [1] (I
> believe the same is true for java). I think this implies one could sample
> for the first page more or less, and then resample on some regular cadence
> (or if compression degrades too much)? In general, the exact approach used
> in implementations can vary here, so I think maybe should clarify that the
> encoding algorithm in a specification is a recommendation, and we
> concentrate the discussion on implementation to the individual language
> binding PRs.
>
> 1) Addition of AlpRd (if that takes time to get in for read doubles). (This
> > is easily addable with provided AlgorithmEnum)
> > 2) Addition of AlpRd + Modified BSS (suggested by Azim) (This is easily
> > addable with provided AlgorithmEnum)
> > 3) Addition of different encoding (This is easily addable with provided
> > AlgorithmEnum).
>
>
> In short, I'd suggest modelling fundamentally different algorithms with the
> top level encoding enum, and have versioning/control bits where we believe
> we will likely want to iterate.
>
> Long version, based on my understanding of current open design points the
> following extensions have been discussed:
>
> 1. Change of integer encoding (see debate in this thread on FOR vs
> Delta). We also want to get fastlanes in at some point. I think an
> enum inside the page for versioning makes sense, as it allows for easier
> composability.
> 2. Change in structure to exceptions (e.g. G-ALP). G-ALP comes with some
> trade-offs, so it is not clear if it is something everyone would want to
> enable.
> 3. Offset indexes to vectors
> 4. Different floating point encoding algorithms (e.g. AlpRd + AlpBSS)
>
> 1 is almost definitely an extension point and I think it pays to version
> this within the page (if we decide to allow delta at some point, and then
> do fast lanes we start having a lot of ALP-pseudecimal enums, that might
> not add a lot of value). This would get worse if we multiply this by any
> future exception layouts. 3. Feels like we can just make a decision now on
> whether to add them, and if not added now can probably be added if we get
> to cascaded encodings, or if really needed as a separate enum (but I'm open
> to control bit here).
>
> For 4, these feel like they should be a top level enum for versioning, they
> are fundamentally different algorithms with different code to decode them.
> The trade-off here is that the current writers need to have some more
> invasive changes to have better fallback or choice of initial encoder (but
> this needs to happen anyways). We can easily have multiple page encodings
> with a row-group from a reader's perspective
>
> For any other future changes that don't fall into these buckets a new
> top-level enum is always an escape hatch (and we can learn from our
> mistakes). Another take is it is just one byte per page, which would take a
> while to add up even at Parquet's scale, but on the balance I'd lean
> towards YAGNI.
>
> Cheers,
> Micah
>
> [1]
>
> https://github.com/apache/arrow/blob/main/cpp/src/parquet/column_writer.cc#L1290
>
>
> On Tue, Jan 27, 2026 at 9:47 AM PRATEEK GAUR <pr...@gmail.com> wrote:
>
> > Hi Antoine,
> >
> > Thanks. Replying inline to the discussion threads.
> >
> >
> > > > https://github.com/apache/parquet-testing/pull/100/files.
> > > > You might have to checkout that branch to be able to run the
> > benchmarks.
> > >
> >
> > Yeah just cherry-picking the branch should work. Let me me know if it
> ends
> > up
> > being some kind of permission issue.
> >
> >
> > >
> > > Hmm, I'll take a look at that later and come back.
> > >
> > > > * the encoding of integers uses a custom framing with
> > frame-of-reference
> > > >> encoding inside it, but Parquet implementations already implement
> > > >> DELTA_BINARY_PACKED which should have similar characteristics, so
> why
> > > >> not reuse that?
> > >
> > > Thanks. At worse it seems it would need one more bit per integer than
> > > the proposed FOR scheme (it might need less if delta encoding allows
> > > reducing entropy among the encoded integers). I'm not sure how that
> > > makes it "fail".
> > >
> >
> >
> > My bad. I wanted to add another point.
> > 1) So I looked at at a few other examples and in most cases FOR ended up
> > using
> > let's bits per value. My thinking is that for 1M (1B values) , it
> will
> > add up.
> > 2) Another major point was that the bitunpacker + FOR was much much
> faster
> > than
> > DeltaBitPack decoder. BitUnpacker+FOR was easily SIMDable but
> > DeltaBitUnpack
> > not so much. I vaguely remember the difference being around 2x. I can
> > try and
> > compute the numbers again. But today the entire bottleneck in the
> > decoder shows
> > up in bit-unpacking.
> >
> >
> > > >> * there are a lot of fields in the headers that look a bit
> superfluous
> > > >> (though of course those bits are relatively cheap); for example, why
> > > >> have a format "version" while we could define a new encoding for
> > > >> incompatible evolutions?
> > > >
> > > > We discussed this point in the Spec
> > > > <
> > >
> >
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit>
> > > > document a lot and have gravitated
> > > > towards a versioning scheme for easier evolution.
> > >
> >
> > > * Meta-answer:
> > >
> > > I don't see any such discussion in the comments and the document
> doesn't
> > > state any rationale for it.
> > >
> > > As a reference, Python PEPs include a discussion of rejected
> > > alternatives so that people don't ask the same questions over and over
> > > (see https://peps.python.org/pep-0810/#alternate-implementation-ideas
> > > for an example).
> > >
> > > * Actual answer:
> > >
> > > My problem with this is that it will complicate understanding and
> > > communicating the feature set supported by each Parquet implementation.
> > > "ALP" will not be a single spec but an evolving one with its own
> version
> > > numbers. I'm not sure why that is better than adding a "ALP2" if we
> even
> > > want to evolve the spec.
> > >
> > > It will also complicate APIs that currently accept encoding numbers,
> > > such as
> > >
> > >
> >
> https://github.com/apache/arrow/blob/5272a68c134deea82040f2f29bb6257ad7b52be0/cpp/src/parquet/properties.h#L221
> > >
> > > We need a clear explanation of what makes ALP so special that it
> *needs*
> > > its own versioning scheme. Otherwise we should remove it IMHO.
> > >
> >
> > Based on the feedback I think it is more of a clarification/API
> > complication discussion.
> > I was thinking along these lines.
> >
> > Things that I thought will had to change but then we added enough
> > flexibility in metadata structs to allow for these.
> > 1) Addition of AlpRd (if that takes time to get in for read doubles).
> (This
> > is easily addable with provided AlgorithmEnum)
> > 2) Addition of AlpRd + Modified BSS (suggested by Azim) (This is easily
> > addable with provided AlgorithmEnum)
> > 3) Addition of different encoding (This is easily addable with provided
> > AlgorithmEnum).
> >
> > Things that I think might need a version field.
> > 1) Right now FOR and ALP structs which are interleaved store certain set
> of
> > fields, given there have been lots of
> > in coming suggestions my thinking was that having a version will
> allow
> > us to easily change them with minor
> > changes to the code in all writers/readers (maybe that is a very bit
> > task).
> >
> >
> > >
> > > >> * the "Total Encoded Element count" duplicates information already
> in
> > > >> the page header, with risks of inconsistent values (including
> security
> > > >> risks that require specific care in implementations)
> > > >
> > > > 'num_elements' : let me re-read and get back on this.
> > >
> > > Ok, Micah's answer cleared that up.
> > >
> >
> > Cool.
> >
> >
> > >
> > > >> * what happens if the number of exceptions is above 65535? their
> > indices
> > > >> are coded as 16-bit uints. How about using the same encoding as for
> > > >> bit-packed integers (e.g. DELTA_BINARY_PACKED), which will also
> remove
> > > >> the 65535 limitation.
> > > >
> > > > So, I don't see a need for a vector larger than 65535. With that
> large
> > > > vectors the
> > > > overhead of metadata is small and you might as well break it into
> > > multiple
> > > > vectors. I'm gonna give it some more thought and get back.
> > >
> > > Ah, yes, I think you're right. Let's forget this :-)
> > >
> >
> >
> > :).
> >
> >
> > >
> > > > Sampling process should be statistically significant. It should pick
> > > enough
> > > > values
> > > > and not have bias towards just the values towards the start. ALP
> > > algorithm
> > > > ensures
> > > > that and tries to balance between not spending enough cycles to get
> > right
> > > > parameters
> > > > and picking incorrect parameters.
> > > >
> > > > For a very large row group we can change the constant and have it
> > > > selects over a larger data set.
> > > > Or one can do it at page level too. Happy to discuss more on this.
> > >
> > > In Parquet C++, encoding happens at page level, and I would guess other
> > > implementations do something similar. Sampling cannot reasonably be
> done
> > > at a higher level, that would require invasive architectural changes.
> > >
> > > But this raises another question: why does this have to be a constant?
> > > When encoding a page, you know the population size (i.e. the actual
> > > number of values that will be encoded). You don't need to estimate it
> > > with a constant.
> > >
> >
> > Some engines might decide to write large pages (large number of values
> in a
> > page) ~2B.
> > One might not want to select all the values towards sampling process to
> > keep sampling
> > a slightly light weight operation. Also as @Andrew Lamb
> > <an...@gmail.com> pointed out at times engines
> > already know what the best values of hyper parameters are might want to
> > provide it as
> > input via an epic to send in pre-populated presets. This can happen
> across
> > multiple pages
> > or multiple row groups after having calculated things for the first batch
> > of data. This might
> > further make sampling more light weight. I would gravitate towards
> > flexibility here :).
> >
> >
> > Thanks for this discussion
> > Prateek
> >
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Andrew Lamb <an...@gmail.com>.
I started creating some test .parquet files using the C++ implementation
[1] to help other implementations test compatibility. I currently have a
really simple file here [2] if anyone wants to play around with it.
I plan to expand it out over the next few days (e.g. include exceptions,
f32, etc)
Finally, I am pretty stoked to report that Kosta Tarasov has started
working on a Rust implementation[3]
Andrew
[1]: https://github.com/apache/arrow/pull/49154
[2]: https://github.com/user-attachments/files/25097841/single_f64_ALP.zip
[3]: https://github.com/apache/arrow-rs/issues/8748
On Tue, Feb 3, 2026 at 6:03 PM Micah Kornfield <em...@gmail.com>
wrote:
> Thanks Prateek,
>
>> *Alp Version in Header*
>> Micah's point
>> *`* I'd suggest modelling fundamentally different algorithms with the
>> top level encoding enum, and have versioning/control bits where we believe
>> we will likely want to iterate`
>> Yes this is exactly what is happening here. An enum to add AlpRd(and more)
>> and version control to iterate anything fundamental (like a layout change
>> of the
>> metadata).
>
>
> I think there might still be some misalignment on which enum we are
> talking about. I was referring to an enum in parquet.thrift for encoding
> <https://github.com/apache/parquet-format/blame/master/src/main/thrift/parquet.thrift#L630> [1].
> Specifically for the current proposal for ALP would ALP_PSEUDODECIMAL. For
> AlpRd we would and ALP_RD (or something similar). If for some reason we
> need ALP_PSEUDODECIMAL, we always have ALP_PSEUDO_DECIMAL_2. Given the
> other extension points, I hope we wouldn't need ALP_PSEUDO_DECIMAL_2 (or at
> least we get a better name) for a long time.
>
> I think the main technical trade-off here is how the thrift parsers across
> bindings handle unknown enum values (e.g. if these encodings show in the
> footer in stats, would that make the whole file unreadable, and do we care).
>
> Cheers,
> Micah
>
>
> [1]
> https://github.com/apache/parquet-format/blame/master/src/main/thrift/parquet.thrift#L630
>
>
>
>
> On Tue, Feb 3, 2026 at 2:40 PM PRATEEK GAUR <pr...@gmail.com> wrote:
>
>> Hi Antoine and Micah,
>>
>> Apologies for getting back on this a little late.
>>
>> *Running Perf tests*
>> @Antoine Pitrou <an...@python.org> were you able to figure out the
>> steps
>> to run the tests?
>>
>> *Sampling Frequency*
>> We want to pick the right parameters to encode the values with. That is
>> what the Spec requires.
>> From the implementation perspective you raise a good point that did cross
>> my
>> mind that 'practically we don't want to sample for every page', for
>> performance
>> reasons. My thinking is each engine is free to decide this.
>> 1) Do it at page level if data is changing often
>> 2) Provide fixed presets via config
>> 3) Do it once per encoder (per column, as Micah pointed out)
>> 4) Provide a fancy config.
>> I agree with Micah here '*I think maybe should clarify that the*
>> *encoding algorithm in a specification is a recommendation'*
>>
>>
>> *Number of values to pick for sampling*
>> 'why does this have to be a constant'
>> You are right, it doesn't need to be a constant, hence the spec doesn't
>> say
>> so.
>> Everything that is segregated out in the AlpContants(c++ impl) file can be
>> changed by
>> configuration.
>> (Did I get your question right @Antoine Pitrou <an...@python.org> ?)
>>
>> *Alp Version in Header*
>> Micah's point
>> *`* I'd suggest modelling fundamentally different algorithms with the
>> top level encoding enum, and have versioning/control bits where we believe
>> we will likely want to iterate`
>> Yes this is exactly what is happening here. An enum to add AlpRd(and more)
>> and version control to iterate anything fundamental (like a layout change
>> of the
>> metadata).
>>
>> Re-stating the points so that scrolling is not needed.
>> 1. Change of integer encoding (see debate in this thread on FOR vs
>> Delta). We also want to get fast lanes in at some point. I think an
>> enum inside the page for versioning makes sense, as it allows for easier
>> composability.
>> 2. Change in structure to exceptions (e.g. G-ALP). G-ALP comes with some
>> trade-offs, so it is not clear if it is something everyone would want to
>> enable.
>> 3. Offset indexes to vectors
>> 4. Different floating point encoding algorithms (e.g. AlpRd + AlpBSS)
>>
>> For eg at this point I do see that both bitpacking of exceptions, as
>> pointed by
>> Antoine, or plain ub2 encoding should work equally well . I was running
>> some benchmarks
>> here and I was getting read speeds of around 20 GB/s for 10bit packed
>> values which is
>> quite good enough (graviton 3 processor).
>> But for simplicity (and the fact that we won't get really large vectors,
>> my
>> inclination
>> is towards ub2 values) but I want to keep the path open to possibly have
>> bipacking
>> as an option as the workloads evolve to a level we haven't thought about
>> yet. We
>> can always add a new encoding but I don't see a path to having 20+ top
>> level
>> encodings. Again I don't have a very strong bias towards keeping it or
>> removing it
>> but my thinking right now is let's have the flexibility and make it easier
>> for people
>> to evolve this encoding behind a version byte.
>>
>>
>> Best
>> Prateek
>> PS : I probably have addressed all open threads raised by Antoine and
>> Micah.
>> (but I may have missed something)
>>
>>
>>
>>
>>
>> On Thu, Jan 29, 2026 at 10:52 PM Micah Kornfield <em...@gmail.com>
>> wrote:
>>
>> > Hi Antoine and Prateek,
>> >
>> > > > In Parquet C++, encoding happens at page level, and I would guess
>> other
>> > > > implementations do something similar. Sampling cannot reasonably be
>> > done
>> > > > at a higher level, that would require invasive architectural
>> changes.
>> >
>> >
>> > At least in C++ I believe we cache the encoder at a column level [1] (I
>> > believe the same is true for java). I think this implies one could
>> sample
>> > for the first page more or less, and then resample on some regular
>> cadence
>> > (or if compression degrades too much)? In general, the exact approach
>> used
>> > in implementations can vary here, so I think maybe should clarify that
>> the
>> > encoding algorithm in a specification is a recommendation, and we
>> > concentrate the discussion on implementation to the individual language
>> > binding PRs.
>> >
>> > 1) Addition of AlpRd (if that takes time to get in for read doubles).
>> (This
>> > > is easily addable with provided AlgorithmEnum)
>> > > 2) Addition of AlpRd + Modified BSS (suggested by Azim) (This is
>> easily
>> > > addable with provided AlgorithmEnum)
>> > > 3) Addition of different encoding (This is easily addable with
>> provided
>> > > AlgorithmEnum).
>> >
>> >
>> > In short, I'd suggest modelling fundamentally different algorithms with
>> the
>> > top level encoding enum, and have versioning/control bits where we
>> believe
>> > we will likely want to iterate.
>> >
>> > Long version, based on my understanding of current open design points
>> the
>> > following extensions have been discussed:
>> >
>> > 1. Change of integer encoding (see debate in this thread on FOR vs
>> > Delta). We also want to get fastlanes in at some point. I think an
>> > enum inside the page for versioning makes sense, as it allows for easier
>> > composability.
>> > 2. Change in structure to exceptions (e.g. G-ALP). G-ALP comes with
>> some
>> > trade-offs, so it is not clear if it is something everyone would want to
>> > enable.
>> > 3. Offset indexes to vectors
>> > 4. Different floating point encoding algorithms (e.g. AlpRd + AlpBSS)
>> >
>> > 1 is almost definitely an extension point and I think it pays to version
>> > this within the page (if we decide to allow delta at some point, and
>> then
>> > do fast lanes we start having a lot of ALP-pseudecimal enums, that might
>> > not add a lot of value). This would get worse if we multiply this by
>> any
>> > future exception layouts. 3. Feels like we can just make a decision
>> now on
>> > whether to add them, and if not added now can probably be added if we
>> get
>> > to cascaded encodings, or if really needed as a separate enum (but I'm
>> open
>> > to control bit here).
>> >
>> > For 4, these feel like they should be a top level enum for versioning,
>> they
>> > are fundamentally different algorithms with different code to decode
>> them.
>> > The trade-off here is that the current writers need to have some more
>> > invasive changes to have better fallback or choice of initial encoder
>> (but
>> > this needs to happen anyways). We can easily have multiple page
>> encodings
>> > with a row-group from a reader's perspective
>> >
>> > For any other future changes that don't fall into these buckets a new
>> > top-level enum is always an escape hatch (and we can learn from our
>> > mistakes). Another take is it is just one byte per page, which would
>> take a
>> > while to add up even at Parquet's scale, but on the balance I'd lean
>> > towards YAGNI.
>> >
>> > Cheers,
>> > Micah
>> >
>> > [1]
>> >
>> >
>> https://github.com/apache/arrow/blob/main/cpp/src/parquet/column_writer.cc#L1290
>> >
>> >
>> > On Tue, Jan 27, 2026 at 9:47 AM PRATEEK GAUR <pr...@gmail.com>
>> wrote:
>> >
>> > > Hi Antoine,
>> > >
>> > > Thanks. Replying inline to the discussion threads.
>> > >
>> > >
>> > > > > https://github.com/apache/parquet-testing/pull/100/files.
>> > > > > You might have to checkout that branch to be able to run the
>> > > benchmarks.
>> > > >
>> > >
>> > > Yeah just cherry-picking the branch should work. Let me me know if it
>> > ends
>> > > up
>> > > being some kind of permission issue.
>> > >
>> > >
>> > > >
>> > > > Hmm, I'll take a look at that later and come back.
>> > > >
>> > > > > * the encoding of integers uses a custom framing with
>> > > frame-of-reference
>> > > > >> encoding inside it, but Parquet implementations already implement
>> > > > >> DELTA_BINARY_PACKED which should have similar characteristics, so
>> > why
>> > > > >> not reuse that?
>> > > >
>> > > > Thanks. At worse it seems it would need one more bit per integer
>> than
>> > > > the proposed FOR scheme (it might need less if delta encoding allows
>> > > > reducing entropy among the encoded integers). I'm not sure how that
>> > > > makes it "fail".
>> > > >
>> > >
>> > >
>> > > My bad. I wanted to add another point.
>> > > 1) So I looked at at a few other examples and in most cases FOR ended
>> up
>> > > using
>> > > let's bits per value. My thinking is that for 1M (1B values) , it
>> > will
>> > > add up.
>> > > 2) Another major point was that the bitunpacker + FOR was much much
>> > faster
>> > > than
>> > > DeltaBitPack decoder. BitUnpacker+FOR was easily SIMDable but
>> > > DeltaBitUnpack
>> > > not so much. I vaguely remember the difference being around 2x. I
>> can
>> > > try and
>> > > compute the numbers again. But today the entire bottleneck in the
>> > > decoder shows
>> > > up in bit-unpacking.
>> > >
>> > >
>> > > > >> * there are a lot of fields in the headers that look a bit
>> > superfluous
>> > > > >> (though of course those bits are relatively cheap); for example,
>> why
>> > > > >> have a format "version" while we could define a new encoding for
>> > > > >> incompatible evolutions?
>> > > > >
>> > > > > We discussed this point in the Spec
>> > > > > <
>> > > >
>> > >
>> >
>> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
>> >
>> > > > > document a lot and have gravitated
>> > > > > towards a versioning scheme for easier evolution.
>> > > >
>> > >
>> > > > * Meta-answer:
>> > > >
>> > > > I don't see any such discussion in the comments and the document
>> > doesn't
>> > > > state any rationale for it.
>> > > >
>> > > > As a reference, Python PEPs include a discussion of rejected
>> > > > alternatives so that people don't ask the same questions over and
>> over
>> > > > (see
>> https://peps.python.org/pep-0810/#alternate-implementation-ideas
>> > > > for an example).
>> > > >
>> > > > * Actual answer:
>> > > >
>> > > > My problem with this is that it will complicate understanding and
>> > > > communicating the feature set supported by each Parquet
>> implementation.
>> > > > "ALP" will not be a single spec but an evolving one with its own
>> > version
>> > > > numbers. I'm not sure why that is better than adding a "ALP2" if we
>> > even
>> > > > want to evolve the spec.
>> > > >
>> > > > It will also complicate APIs that currently accept encoding numbers,
>> > > > such as
>> > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/arrow/blob/5272a68c134deea82040f2f29bb6257ad7b52be0/cpp/src/parquet/properties.h#L221
>> > > >
>> > > > We need a clear explanation of what makes ALP so special that it
>> > *needs*
>> > > > its own versioning scheme. Otherwise we should remove it IMHO.
>> > > >
>> > >
>> > > Based on the feedback I think it is more of a clarification/API
>> > > complication discussion.
>> > > I was thinking along these lines.
>> > >
>> > > Things that I thought will had to change but then we added enough
>> > > flexibility in metadata structs to allow for these.
>> > > 1) Addition of AlpRd (if that takes time to get in for read doubles).
>> > (This
>> > > is easily addable with provided AlgorithmEnum)
>> > > 2) Addition of AlpRd + Modified BSS (suggested by Azim) (This is
>> easily
>> > > addable with provided AlgorithmEnum)
>> > > 3) Addition of different encoding (This is easily addable with
>> provided
>> > > AlgorithmEnum).
>> > >
>> > > Things that I think might need a version field.
>> > > 1) Right now FOR and ALP structs which are interleaved store certain
>> set
>> > of
>> > > fields, given there have been lots of
>> > > in coming suggestions my thinking was that having a version will
>> > allow
>> > > us to easily change them with minor
>> > > changes to the code in all writers/readers (maybe that is a very
>> bit
>> > > task).
>> > >
>> > >
>> > > >
>> > > > >> * the "Total Encoded Element count" duplicates information
>> already
>> > in
>> > > > >> the page header, with risks of inconsistent values (including
>> > security
>> > > > >> risks that require specific care in implementations)
>> > > > >
>> > > > > 'num_elements' : let me re-read and get back on this.
>> > > >
>> > > > Ok, Micah's answer cleared that up.
>> > > >
>> > >
>> > > Cool.
>> > >
>> > >
>> > > >
>> > > > >> * what happens if the number of exceptions is above 65535? their
>> > > indices
>> > > > >> are coded as 16-bit uints. How about using the same encoding as
>> for
>> > > > >> bit-packed integers (e.g. DELTA_BINARY_PACKED), which will also
>> > remove
>> > > > >> the 65535 limitation.
>> > > > >
>> > > > > So, I don't see a need for a vector larger than 65535. With that
>> > large
>> > > > > vectors the
>> > > > > overhead of metadata is small and you might as well break it into
>> > > > multiple
>> > > > > vectors. I'm gonna give it some more thought and get back.
>> > > >
>> > > > Ah, yes, I think you're right. Let's forget this :-)
>> > > >
>> > >
>> > >
>> > > :).
>> > >
>> > >
>> > > >
>> > > > > Sampling process should be statistically significant. It should
>> pick
>> > > > enough
>> > > > > values
>> > > > > and not have bias towards just the values towards the start. ALP
>> > > > algorithm
>> > > > > ensures
>> > > > > that and tries to balance between not spending enough cycles to
>> get
>> > > right
>> > > > > parameters
>> > > > > and picking incorrect parameters.
>> > > > >
>> > > > > For a very large row group we can change the constant and have it
>> > > > > selects over a larger data set.
>> > > > > Or one can do it at page level too. Happy to discuss more on this.
>> > > >
>> > > > In Parquet C++, encoding happens at page level, and I would guess
>> other
>> > > > implementations do something similar. Sampling cannot reasonably be
>> > done
>> > > > at a higher level, that would require invasive architectural
>> changes.
>> > > >
>> > > > But this raises another question: why does this have to be a
>> constant?
>> > > > When encoding a page, you know the population size (i.e. the actual
>> > > > number of values that will be encoded). You don't need to estimate
>> it
>> > > > with a constant.
>> > > >
>> > >
>> > > Some engines might decide to write large pages (large number of values
>> > in a
>> > > page) ~2B.
>> > > One might not want to select all the values towards sampling process
>> to
>> > > keep sampling
>> > > a slightly light weight operation. Also as @Andrew Lamb
>> > > <an...@gmail.com> pointed out at times engines
>> > > already know what the best values of hyper parameters are might want
>> to
>> > > provide it as
>> > > input via an epic to send in pre-populated presets. This can happen
>> > across
>> > > multiple pages
>> > > or multiple row groups after having calculated things for the first
>> batch
>> > > of data. This might
>> > > further make sampling more light weight. I would gravitate towards
>> > > flexibility here :).
>> > >
>> > >
>> > > Thanks for this discussion
>> > > Prateek
>> > >
>> >
>>
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Andrew Lamb <an...@gmail.com>.
I started creating some test .parquet files using the C++ implementation
[1] to help other implementations test compatibility. I currently have a
really simple file here [2] if anyone wants to play around with it.
I plan to expand it out over the next few days (e.g. include exceptions,
f32, etc)
Finally, I am pretty stoked to report that Kosta Tarasov has started
working on a Rust implementation[3]
Andrew
[1]: https://github.com/apache/arrow/pull/49154
[2]: https://github.com/user-attachments/files/25097841/single_f64_ALP.zip
[3]: https://github.com/apache/arrow-rs/issues/8748
On Tue, Feb 3, 2026 at 6:03 PM Micah Kornfield <em...@gmail.com>
wrote:
> Thanks Prateek,
>
>> *Alp Version in Header*
>> Micah's point
>> *`* I'd suggest modelling fundamentally different algorithms with the
>> top level encoding enum, and have versioning/control bits where we believe
>> we will likely want to iterate`
>> Yes this is exactly what is happening here. An enum to add AlpRd(and more)
>> and version control to iterate anything fundamental (like a layout change
>> of the
>> metadata).
>
>
> I think there might still be some misalignment on which enum we are
> talking about. I was referring to an enum in parquet.thrift for encoding
> <https://github.com/apache/parquet-format/blame/master/src/main/thrift/parquet.thrift#L630> [1].
> Specifically for the current proposal for ALP would ALP_PSEUDODECIMAL. For
> AlpRd we would and ALP_RD (or something similar). If for some reason we
> need ALP_PSEUDODECIMAL, we always have ALP_PSEUDO_DECIMAL_2. Given the
> other extension points, I hope we wouldn't need ALP_PSEUDO_DECIMAL_2 (or at
> least we get a better name) for a long time.
>
> I think the main technical trade-off here is how the thrift parsers across
> bindings handle unknown enum values (e.g. if these encodings show in the
> footer in stats, would that make the whole file unreadable, and do we care).
>
> Cheers,
> Micah
>
>
> [1]
> https://github.com/apache/parquet-format/blame/master/src/main/thrift/parquet.thrift#L630
>
>
>
>
> On Tue, Feb 3, 2026 at 2:40 PM PRATEEK GAUR <pr...@gmail.com> wrote:
>
>> Hi Antoine and Micah,
>>
>> Apologies for getting back on this a little late.
>>
>> *Running Perf tests*
>> @Antoine Pitrou <an...@python.org> were you able to figure out the
>> steps
>> to run the tests?
>>
>> *Sampling Frequency*
>> We want to pick the right parameters to encode the values with. That is
>> what the Spec requires.
>> From the implementation perspective you raise a good point that did cross
>> my
>> mind that 'practically we don't want to sample for every page', for
>> performance
>> reasons. My thinking is each engine is free to decide this.
>> 1) Do it at page level if data is changing often
>> 2) Provide fixed presets via config
>> 3) Do it once per encoder (per column, as Micah pointed out)
>> 4) Provide a fancy config.
>> I agree with Micah here '*I think maybe should clarify that the*
>> *encoding algorithm in a specification is a recommendation'*
>>
>>
>> *Number of values to pick for sampling*
>> 'why does this have to be a constant'
>> You are right, it doesn't need to be a constant, hence the spec doesn't
>> say
>> so.
>> Everything that is segregated out in the AlpContants(c++ impl) file can be
>> changed by
>> configuration.
>> (Did I get your question right @Antoine Pitrou <an...@python.org> ?)
>>
>> *Alp Version in Header*
>> Micah's point
>> *`* I'd suggest modelling fundamentally different algorithms with the
>> top level encoding enum, and have versioning/control bits where we believe
>> we will likely want to iterate`
>> Yes this is exactly what is happening here. An enum to add AlpRd(and more)
>> and version control to iterate anything fundamental (like a layout change
>> of the
>> metadata).
>>
>> Re-stating the points so that scrolling is not needed.
>> 1. Change of integer encoding (see debate in this thread on FOR vs
>> Delta). We also want to get fast lanes in at some point. I think an
>> enum inside the page for versioning makes sense, as it allows for easier
>> composability.
>> 2. Change in structure to exceptions (e.g. G-ALP). G-ALP comes with some
>> trade-offs, so it is not clear if it is something everyone would want to
>> enable.
>> 3. Offset indexes to vectors
>> 4. Different floating point encoding algorithms (e.g. AlpRd + AlpBSS)
>>
>> For eg at this point I do see that both bitpacking of exceptions, as
>> pointed by
>> Antoine, or plain ub2 encoding should work equally well . I was running
>> some benchmarks
>> here and I was getting read speeds of around 20 GB/s for 10bit packed
>> values which is
>> quite good enough (graviton 3 processor).
>> But for simplicity (and the fact that we won't get really large vectors,
>> my
>> inclination
>> is towards ub2 values) but I want to keep the path open to possibly have
>> bipacking
>> as an option as the workloads evolve to a level we haven't thought about
>> yet. We
>> can always add a new encoding but I don't see a path to having 20+ top
>> level
>> encodings. Again I don't have a very strong bias towards keeping it or
>> removing it
>> but my thinking right now is let's have the flexibility and make it easier
>> for people
>> to evolve this encoding behind a version byte.
>>
>>
>> Best
>> Prateek
>> PS : I probably have addressed all open threads raised by Antoine and
>> Micah.
>> (but I may have missed something)
>>
>>
>>
>>
>>
>> On Thu, Jan 29, 2026 at 10:52 PM Micah Kornfield <em...@gmail.com>
>> wrote:
>>
>> > Hi Antoine and Prateek,
>> >
>> > > > In Parquet C++, encoding happens at page level, and I would guess
>> other
>> > > > implementations do something similar. Sampling cannot reasonably be
>> > done
>> > > > at a higher level, that would require invasive architectural
>> changes.
>> >
>> >
>> > At least in C++ I believe we cache the encoder at a column level [1] (I
>> > believe the same is true for java). I think this implies one could
>> sample
>> > for the first page more or less, and then resample on some regular
>> cadence
>> > (or if compression degrades too much)? In general, the exact approach
>> used
>> > in implementations can vary here, so I think maybe should clarify that
>> the
>> > encoding algorithm in a specification is a recommendation, and we
>> > concentrate the discussion on implementation to the individual language
>> > binding PRs.
>> >
>> > 1) Addition of AlpRd (if that takes time to get in for read doubles).
>> (This
>> > > is easily addable with provided AlgorithmEnum)
>> > > 2) Addition of AlpRd + Modified BSS (suggested by Azim) (This is
>> easily
>> > > addable with provided AlgorithmEnum)
>> > > 3) Addition of different encoding (This is easily addable with
>> provided
>> > > AlgorithmEnum).
>> >
>> >
>> > In short, I'd suggest modelling fundamentally different algorithms with
>> the
>> > top level encoding enum, and have versioning/control bits where we
>> believe
>> > we will likely want to iterate.
>> >
>> > Long version, based on my understanding of current open design points
>> the
>> > following extensions have been discussed:
>> >
>> > 1. Change of integer encoding (see debate in this thread on FOR vs
>> > Delta). We also want to get fastlanes in at some point. I think an
>> > enum inside the page for versioning makes sense, as it allows for easier
>> > composability.
>> > 2. Change in structure to exceptions (e.g. G-ALP). G-ALP comes with
>> some
>> > trade-offs, so it is not clear if it is something everyone would want to
>> > enable.
>> > 3. Offset indexes to vectors
>> > 4. Different floating point encoding algorithms (e.g. AlpRd + AlpBSS)
>> >
>> > 1 is almost definitely an extension point and I think it pays to version
>> > this within the page (if we decide to allow delta at some point, and
>> then
>> > do fast lanes we start having a lot of ALP-pseudecimal enums, that might
>> > not add a lot of value). This would get worse if we multiply this by
>> any
>> > future exception layouts. 3. Feels like we can just make a decision
>> now on
>> > whether to add them, and if not added now can probably be added if we
>> get
>> > to cascaded encodings, or if really needed as a separate enum (but I'm
>> open
>> > to control bit here).
>> >
>> > For 4, these feel like they should be a top level enum for versioning,
>> they
>> > are fundamentally different algorithms with different code to decode
>> them.
>> > The trade-off here is that the current writers need to have some more
>> > invasive changes to have better fallback or choice of initial encoder
>> (but
>> > this needs to happen anyways). We can easily have multiple page
>> encodings
>> > with a row-group from a reader's perspective
>> >
>> > For any other future changes that don't fall into these buckets a new
>> > top-level enum is always an escape hatch (and we can learn from our
>> > mistakes). Another take is it is just one byte per page, which would
>> take a
>> > while to add up even at Parquet's scale, but on the balance I'd lean
>> > towards YAGNI.
>> >
>> > Cheers,
>> > Micah
>> >
>> > [1]
>> >
>> >
>> https://github.com/apache/arrow/blob/main/cpp/src/parquet/column_writer.cc#L1290
>> >
>> >
>> > On Tue, Jan 27, 2026 at 9:47 AM PRATEEK GAUR <pr...@gmail.com>
>> wrote:
>> >
>> > > Hi Antoine,
>> > >
>> > > Thanks. Replying inline to the discussion threads.
>> > >
>> > >
>> > > > > https://github.com/apache/parquet-testing/pull/100/files.
>> > > > > You might have to checkout that branch to be able to run the
>> > > benchmarks.
>> > > >
>> > >
>> > > Yeah just cherry-picking the branch should work. Let me me know if it
>> > ends
>> > > up
>> > > being some kind of permission issue.
>> > >
>> > >
>> > > >
>> > > > Hmm, I'll take a look at that later and come back.
>> > > >
>> > > > > * the encoding of integers uses a custom framing with
>> > > frame-of-reference
>> > > > >> encoding inside it, but Parquet implementations already implement
>> > > > >> DELTA_BINARY_PACKED which should have similar characteristics, so
>> > why
>> > > > >> not reuse that?
>> > > >
>> > > > Thanks. At worse it seems it would need one more bit per integer
>> than
>> > > > the proposed FOR scheme (it might need less if delta encoding allows
>> > > > reducing entropy among the encoded integers). I'm not sure how that
>> > > > makes it "fail".
>> > > >
>> > >
>> > >
>> > > My bad. I wanted to add another point.
>> > > 1) So I looked at at a few other examples and in most cases FOR ended
>> up
>> > > using
>> > > let's bits per value. My thinking is that for 1M (1B values) , it
>> > will
>> > > add up.
>> > > 2) Another major point was that the bitunpacker + FOR was much much
>> > faster
>> > > than
>> > > DeltaBitPack decoder. BitUnpacker+FOR was easily SIMDable but
>> > > DeltaBitUnpack
>> > > not so much. I vaguely remember the difference being around 2x. I
>> can
>> > > try and
>> > > compute the numbers again. But today the entire bottleneck in the
>> > > decoder shows
>> > > up in bit-unpacking.
>> > >
>> > >
>> > > > >> * there are a lot of fields in the headers that look a bit
>> > superfluous
>> > > > >> (though of course those bits are relatively cheap); for example,
>> why
>> > > > >> have a format "version" while we could define a new encoding for
>> > > > >> incompatible evolutions?
>> > > > >
>> > > > > We discussed this point in the Spec
>> > > > > <
>> > > >
>> > >
>> >
>> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
>> >
>> > > > > document a lot and have gravitated
>> > > > > towards a versioning scheme for easier evolution.
>> > > >
>> > >
>> > > > * Meta-answer:
>> > > >
>> > > > I don't see any such discussion in the comments and the document
>> > doesn't
>> > > > state any rationale for it.
>> > > >
>> > > > As a reference, Python PEPs include a discussion of rejected
>> > > > alternatives so that people don't ask the same questions over and
>> over
>> > > > (see
>> https://peps.python.org/pep-0810/#alternate-implementation-ideas
>> > > > for an example).
>> > > >
>> > > > * Actual answer:
>> > > >
>> > > > My problem with this is that it will complicate understanding and
>> > > > communicating the feature set supported by each Parquet
>> implementation.
>> > > > "ALP" will not be a single spec but an evolving one with its own
>> > version
>> > > > numbers. I'm not sure why that is better than adding a "ALP2" if we
>> > even
>> > > > want to evolve the spec.
>> > > >
>> > > > It will also complicate APIs that currently accept encoding numbers,
>> > > > such as
>> > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/arrow/blob/5272a68c134deea82040f2f29bb6257ad7b52be0/cpp/src/parquet/properties.h#L221
>> > > >
>> > > > We need a clear explanation of what makes ALP so special that it
>> > *needs*
>> > > > its own versioning scheme. Otherwise we should remove it IMHO.
>> > > >
>> > >
>> > > Based on the feedback I think it is more of a clarification/API
>> > > complication discussion.
>> > > I was thinking along these lines.
>> > >
>> > > Things that I thought will had to change but then we added enough
>> > > flexibility in metadata structs to allow for these.
>> > > 1) Addition of AlpRd (if that takes time to get in for read doubles).
>> > (This
>> > > is easily addable with provided AlgorithmEnum)
>> > > 2) Addition of AlpRd + Modified BSS (suggested by Azim) (This is
>> easily
>> > > addable with provided AlgorithmEnum)
>> > > 3) Addition of different encoding (This is easily addable with
>> provided
>> > > AlgorithmEnum).
>> > >
>> > > Things that I think might need a version field.
>> > > 1) Right now FOR and ALP structs which are interleaved store certain
>> set
>> > of
>> > > fields, given there have been lots of
>> > > in coming suggestions my thinking was that having a version will
>> > allow
>> > > us to easily change them with minor
>> > > changes to the code in all writers/readers (maybe that is a very
>> bit
>> > > task).
>> > >
>> > >
>> > > >
>> > > > >> * the "Total Encoded Element count" duplicates information
>> already
>> > in
>> > > > >> the page header, with risks of inconsistent values (including
>> > security
>> > > > >> risks that require specific care in implementations)
>> > > > >
>> > > > > 'num_elements' : let me re-read and get back on this.
>> > > >
>> > > > Ok, Micah's answer cleared that up.
>> > > >
>> > >
>> > > Cool.
>> > >
>> > >
>> > > >
>> > > > >> * what happens if the number of exceptions is above 65535? their
>> > > indices
>> > > > >> are coded as 16-bit uints. How about using the same encoding as
>> for
>> > > > >> bit-packed integers (e.g. DELTA_BINARY_PACKED), which will also
>> > remove
>> > > > >> the 65535 limitation.
>> > > > >
>> > > > > So, I don't see a need for a vector larger than 65535. With that
>> > large
>> > > > > vectors the
>> > > > > overhead of metadata is small and you might as well break it into
>> > > > multiple
>> > > > > vectors. I'm gonna give it some more thought and get back.
>> > > >
>> > > > Ah, yes, I think you're right. Let's forget this :-)
>> > > >
>> > >
>> > >
>> > > :).
>> > >
>> > >
>> > > >
>> > > > > Sampling process should be statistically significant. It should
>> pick
>> > > > enough
>> > > > > values
>> > > > > and not have bias towards just the values towards the start. ALP
>> > > > algorithm
>> > > > > ensures
>> > > > > that and tries to balance between not spending enough cycles to
>> get
>> > > right
>> > > > > parameters
>> > > > > and picking incorrect parameters.
>> > > > >
>> > > > > For a very large row group we can change the constant and have it
>> > > > > selects over a larger data set.
>> > > > > Or one can do it at page level too. Happy to discuss more on this.
>> > > >
>> > > > In Parquet C++, encoding happens at page level, and I would guess
>> other
>> > > > implementations do something similar. Sampling cannot reasonably be
>> > done
>> > > > at a higher level, that would require invasive architectural
>> changes.
>> > > >
>> > > > But this raises another question: why does this have to be a
>> constant?
>> > > > When encoding a page, you know the population size (i.e. the actual
>> > > > number of values that will be encoded). You don't need to estimate
>> it
>> > > > with a constant.
>> > > >
>> > >
>> > > Some engines might decide to write large pages (large number of values
>> > in a
>> > > page) ~2B.
>> > > One might not want to select all the values towards sampling process
>> to
>> > > keep sampling
>> > > a slightly light weight operation. Also as @Andrew Lamb
>> > > <an...@gmail.com> pointed out at times engines
>> > > already know what the best values of hyper parameters are might want
>> to
>> > > provide it as
>> > > input via an epic to send in pre-populated presets. This can happen
>> > across
>> > > multiple pages
>> > > or multiple row groups after having calculated things for the first
>> batch
>> > > of data. This might
>> > > further make sampling more light weight. I would gravitate towards
>> > > flexibility here :).
>> > >
>> > >
>> > > Thanks for this discussion
>> > > Prateek
>> > >
>> >
>>
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Micah Kornfield <em...@gmail.com>.
Thanks Prateek,
> *Alp Version in Header*
> Micah's point
> *`* I'd suggest modelling fundamentally different algorithms with the
> top level encoding enum, and have versioning/control bits where we believe
> we will likely want to iterate`
> Yes this is exactly what is happening here. An enum to add AlpRd(and more)
> and version control to iterate anything fundamental (like a layout change
> of the
> metadata).
I think there might still be some misalignment on which enum we are talking
about. I was referring to an enum in parquet.thrift for encoding
<https://github.com/apache/parquet-format/blame/master/src/main/thrift/parquet.thrift#L630>
[1].
Specifically for the current proposal for ALP would ALP_PSEUDODECIMAL. For
AlpRd we would and ALP_RD (or something similar). If for some reason we
need ALP_PSEUDODECIMAL, we always have ALP_PSEUDO_DECIMAL_2. Given the
other extension points, I hope we wouldn't need ALP_PSEUDO_DECIMAL_2 (or at
least we get a better name) for a long time.
I think the main technical trade-off here is how the thrift parsers across
bindings handle unknown enum values (e.g. if these encodings show in the
footer in stats, would that make the whole file unreadable, and do we care).
Cheers,
Micah
[1]
https://github.com/apache/parquet-format/blame/master/src/main/thrift/parquet.thrift#L630
On Tue, Feb 3, 2026 at 2:40 PM PRATEEK GAUR <pr...@gmail.com> wrote:
> Hi Antoine and Micah,
>
> Apologies for getting back on this a little late.
>
> *Running Perf tests*
> @Antoine Pitrou <an...@python.org> were you able to figure out the steps
> to run the tests?
>
> *Sampling Frequency*
> We want to pick the right parameters to encode the values with. That is
> what the Spec requires.
> From the implementation perspective you raise a good point that did cross
> my
> mind that 'practically we don't want to sample for every page', for
> performance
> reasons. My thinking is each engine is free to decide this.
> 1) Do it at page level if data is changing often
> 2) Provide fixed presets via config
> 3) Do it once per encoder (per column, as Micah pointed out)
> 4) Provide a fancy config.
> I agree with Micah here '*I think maybe should clarify that the*
> *encoding algorithm in a specification is a recommendation'*
>
>
> *Number of values to pick for sampling*
> 'why does this have to be a constant'
> You are right, it doesn't need to be a constant, hence the spec doesn't say
> so.
> Everything that is segregated out in the AlpContants(c++ impl) file can be
> changed by
> configuration.
> (Did I get your question right @Antoine Pitrou <an...@python.org> ?)
>
> *Alp Version in Header*
> Micah's point
> *`* I'd suggest modelling fundamentally different algorithms with the
> top level encoding enum, and have versioning/control bits where we believe
> we will likely want to iterate`
> Yes this is exactly what is happening here. An enum to add AlpRd(and more)
> and version control to iterate anything fundamental (like a layout change
> of the
> metadata).
>
> Re-stating the points so that scrolling is not needed.
> 1. Change of integer encoding (see debate in this thread on FOR vs
> Delta). We also want to get fast lanes in at some point. I think an
> enum inside the page for versioning makes sense, as it allows for easier
> composability.
> 2. Change in structure to exceptions (e.g. G-ALP). G-ALP comes with some
> trade-offs, so it is not clear if it is something everyone would want to
> enable.
> 3. Offset indexes to vectors
> 4. Different floating point encoding algorithms (e.g. AlpRd + AlpBSS)
>
> For eg at this point I do see that both bitpacking of exceptions, as
> pointed by
> Antoine, or plain ub2 encoding should work equally well . I was running
> some benchmarks
> here and I was getting read speeds of around 20 GB/s for 10bit packed
> values which is
> quite good enough (graviton 3 processor).
> But for simplicity (and the fact that we won't get really large vectors, my
> inclination
> is towards ub2 values) but I want to keep the path open to possibly have
> bipacking
> as an option as the workloads evolve to a level we haven't thought about
> yet. We
> can always add a new encoding but I don't see a path to having 20+ top
> level
> encodings. Again I don't have a very strong bias towards keeping it or
> removing it
> but my thinking right now is let's have the flexibility and make it easier
> for people
> to evolve this encoding behind a version byte.
>
>
> Best
> Prateek
> PS : I probably have addressed all open threads raised by Antoine and
> Micah.
> (but I may have missed something)
>
>
>
>
>
> On Thu, Jan 29, 2026 at 10:52 PM Micah Kornfield <em...@gmail.com>
> wrote:
>
> > Hi Antoine and Prateek,
> >
> > > > In Parquet C++, encoding happens at page level, and I would guess
> other
> > > > implementations do something similar. Sampling cannot reasonably be
> > done
> > > > at a higher level, that would require invasive architectural changes.
> >
> >
> > At least in C++ I believe we cache the encoder at a column level [1] (I
> > believe the same is true for java). I think this implies one could sample
> > for the first page more or less, and then resample on some regular
> cadence
> > (or if compression degrades too much)? In general, the exact approach
> used
> > in implementations can vary here, so I think maybe should clarify that
> the
> > encoding algorithm in a specification is a recommendation, and we
> > concentrate the discussion on implementation to the individual language
> > binding PRs.
> >
> > 1) Addition of AlpRd (if that takes time to get in for read doubles).
> (This
> > > is easily addable with provided AlgorithmEnum)
> > > 2) Addition of AlpRd + Modified BSS (suggested by Azim) (This is easily
> > > addable with provided AlgorithmEnum)
> > > 3) Addition of different encoding (This is easily addable with provided
> > > AlgorithmEnum).
> >
> >
> > In short, I'd suggest modelling fundamentally different algorithms with
> the
> > top level encoding enum, and have versioning/control bits where we
> believe
> > we will likely want to iterate.
> >
> > Long version, based on my understanding of current open design points the
> > following extensions have been discussed:
> >
> > 1. Change of integer encoding (see debate in this thread on FOR vs
> > Delta). We also want to get fastlanes in at some point. I think an
> > enum inside the page for versioning makes sense, as it allows for easier
> > composability.
> > 2. Change in structure to exceptions (e.g. G-ALP). G-ALP comes with
> some
> > trade-offs, so it is not clear if it is something everyone would want to
> > enable.
> > 3. Offset indexes to vectors
> > 4. Different floating point encoding algorithms (e.g. AlpRd + AlpBSS)
> >
> > 1 is almost definitely an extension point and I think it pays to version
> > this within the page (if we decide to allow delta at some point, and then
> > do fast lanes we start having a lot of ALP-pseudecimal enums, that might
> > not add a lot of value). This would get worse if we multiply this by any
> > future exception layouts. 3. Feels like we can just make a decision now
> on
> > whether to add them, and if not added now can probably be added if we get
> > to cascaded encodings, or if really needed as a separate enum (but I'm
> open
> > to control bit here).
> >
> > For 4, these feel like they should be a top level enum for versioning,
> they
> > are fundamentally different algorithms with different code to decode
> them.
> > The trade-off here is that the current writers need to have some more
> > invasive changes to have better fallback or choice of initial encoder
> (but
> > this needs to happen anyways). We can easily have multiple page
> encodings
> > with a row-group from a reader's perspective
> >
> > For any other future changes that don't fall into these buckets a new
> > top-level enum is always an escape hatch (and we can learn from our
> > mistakes). Another take is it is just one byte per page, which would
> take a
> > while to add up even at Parquet's scale, but on the balance I'd lean
> > towards YAGNI.
> >
> > Cheers,
> > Micah
> >
> > [1]
> >
> >
> https://github.com/apache/arrow/blob/main/cpp/src/parquet/column_writer.cc#L1290
> >
> >
> > On Tue, Jan 27, 2026 at 9:47 AM PRATEEK GAUR <pr...@gmail.com> wrote:
> >
> > > Hi Antoine,
> > >
> > > Thanks. Replying inline to the discussion threads.
> > >
> > >
> > > > > https://github.com/apache/parquet-testing/pull/100/files.
> > > > > You might have to checkout that branch to be able to run the
> > > benchmarks.
> > > >
> > >
> > > Yeah just cherry-picking the branch should work. Let me me know if it
> > ends
> > > up
> > > being some kind of permission issue.
> > >
> > >
> > > >
> > > > Hmm, I'll take a look at that later and come back.
> > > >
> > > > > * the encoding of integers uses a custom framing with
> > > frame-of-reference
> > > > >> encoding inside it, but Parquet implementations already implement
> > > > >> DELTA_BINARY_PACKED which should have similar characteristics, so
> > why
> > > > >> not reuse that?
> > > >
> > > > Thanks. At worse it seems it would need one more bit per integer than
> > > > the proposed FOR scheme (it might need less if delta encoding allows
> > > > reducing entropy among the encoded integers). I'm not sure how that
> > > > makes it "fail".
> > > >
> > >
> > >
> > > My bad. I wanted to add another point.
> > > 1) So I looked at at a few other examples and in most cases FOR ended
> up
> > > using
> > > let's bits per value. My thinking is that for 1M (1B values) , it
> > will
> > > add up.
> > > 2) Another major point was that the bitunpacker + FOR was much much
> > faster
> > > than
> > > DeltaBitPack decoder. BitUnpacker+FOR was easily SIMDable but
> > > DeltaBitUnpack
> > > not so much. I vaguely remember the difference being around 2x. I
> can
> > > try and
> > > compute the numbers again. But today the entire bottleneck in the
> > > decoder shows
> > > up in bit-unpacking.
> > >
> > >
> > > > >> * there are a lot of fields in the headers that look a bit
> > superfluous
> > > > >> (though of course those bits are relatively cheap); for example,
> why
> > > > >> have a format "version" while we could define a new encoding for
> > > > >> incompatible evolutions?
> > > > >
> > > > > We discussed this point in the Spec
> > > > > <
> > > >
> > >
> >
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit>
> > > > > document a lot and have gravitated
> > > > > towards a versioning scheme for easier evolution.
> > > >
> > >
> > > > * Meta-answer:
> > > >
> > > > I don't see any such discussion in the comments and the document
> > doesn't
> > > > state any rationale for it.
> > > >
> > > > As a reference, Python PEPs include a discussion of rejected
> > > > alternatives so that people don't ask the same questions over and
> over
> > > > (see
> https://peps.python.org/pep-0810/#alternate-implementation-ideas
> > > > for an example).
> > > >
> > > > * Actual answer:
> > > >
> > > > My problem with this is that it will complicate understanding and
> > > > communicating the feature set supported by each Parquet
> implementation.
> > > > "ALP" will not be a single spec but an evolving one with its own
> > version
> > > > numbers. I'm not sure why that is better than adding a "ALP2" if we
> > even
> > > > want to evolve the spec.
> > > >
> > > > It will also complicate APIs that currently accept encoding numbers,
> > > > such as
> > > >
> > > >
> > >
> >
> https://github.com/apache/arrow/blob/5272a68c134deea82040f2f29bb6257ad7b52be0/cpp/src/parquet/properties.h#L221
> > > >
> > > > We need a clear explanation of what makes ALP so special that it
> > *needs*
> > > > its own versioning scheme. Otherwise we should remove it IMHO.
> > > >
> > >
> > > Based on the feedback I think it is more of a clarification/API
> > > complication discussion.
> > > I was thinking along these lines.
> > >
> > > Things that I thought will had to change but then we added enough
> > > flexibility in metadata structs to allow for these.
> > > 1) Addition of AlpRd (if that takes time to get in for read doubles).
> > (This
> > > is easily addable with provided AlgorithmEnum)
> > > 2) Addition of AlpRd + Modified BSS (suggested by Azim) (This is easily
> > > addable with provided AlgorithmEnum)
> > > 3) Addition of different encoding (This is easily addable with provided
> > > AlgorithmEnum).
> > >
> > > Things that I think might need a version field.
> > > 1) Right now FOR and ALP structs which are interleaved store certain
> set
> > of
> > > fields, given there have been lots of
> > > in coming suggestions my thinking was that having a version will
> > allow
> > > us to easily change them with minor
> > > changes to the code in all writers/readers (maybe that is a very
> bit
> > > task).
> > >
> > >
> > > >
> > > > >> * the "Total Encoded Element count" duplicates information already
> > in
> > > > >> the page header, with risks of inconsistent values (including
> > security
> > > > >> risks that require specific care in implementations)
> > > > >
> > > > > 'num_elements' : let me re-read and get back on this.
> > > >
> > > > Ok, Micah's answer cleared that up.
> > > >
> > >
> > > Cool.
> > >
> > >
> > > >
> > > > >> * what happens if the number of exceptions is above 65535? their
> > > indices
> > > > >> are coded as 16-bit uints. How about using the same encoding as
> for
> > > > >> bit-packed integers (e.g. DELTA_BINARY_PACKED), which will also
> > remove
> > > > >> the 65535 limitation.
> > > > >
> > > > > So, I don't see a need for a vector larger than 65535. With that
> > large
> > > > > vectors the
> > > > > overhead of metadata is small and you might as well break it into
> > > > multiple
> > > > > vectors. I'm gonna give it some more thought and get back.
> > > >
> > > > Ah, yes, I think you're right. Let's forget this :-)
> > > >
> > >
> > >
> > > :).
> > >
> > >
> > > >
> > > > > Sampling process should be statistically significant. It should
> pick
> > > > enough
> > > > > values
> > > > > and not have bias towards just the values towards the start. ALP
> > > > algorithm
> > > > > ensures
> > > > > that and tries to balance between not spending enough cycles to get
> > > right
> > > > > parameters
> > > > > and picking incorrect parameters.
> > > > >
> > > > > For a very large row group we can change the constant and have it
> > > > > selects over a larger data set.
> > > > > Or one can do it at page level too. Happy to discuss more on this.
> > > >
> > > > In Parquet C++, encoding happens at page level, and I would guess
> other
> > > > implementations do something similar. Sampling cannot reasonably be
> > done
> > > > at a higher level, that would require invasive architectural changes.
> > > >
> > > > But this raises another question: why does this have to be a
> constant?
> > > > When encoding a page, you know the population size (i.e. the actual
> > > > number of values that will be encoded). You don't need to estimate it
> > > > with a constant.
> > > >
> > >
> > > Some engines might decide to write large pages (large number of values
> > in a
> > > page) ~2B.
> > > One might not want to select all the values towards sampling process to
> > > keep sampling
> > > a slightly light weight operation. Also as @Andrew Lamb
> > > <an...@gmail.com> pointed out at times engines
> > > already know what the best values of hyper parameters are might want to
> > > provide it as
> > > input via an epic to send in pre-populated presets. This can happen
> > across
> > > multiple pages
> > > or multiple row groups after having calculated things for the first
> batch
> > > of data. This might
> > > further make sampling more light weight. I would gravitate towards
> > > flexibility here :).
> > >
> > >
> > > Thanks for this discussion
> > > Prateek
> > >
> >
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Micah Kornfield <em...@gmail.com>.
>
> Now the issue is that the header was exactly 8 bytes :). Do I keep 1 byte
> as padding or move the header to 7 bytes?
> Any recommendations?
I think it is fine for it to be 7 bytes, despite the asymmetry. We aren't
likely to be loading these on aligned boundaries IIRC, and we are using
Safe loading anyways?
On Tue, Feb 24, 2026 at 12:52 PM PRATEEK GAUR <pr...@gmail.com> wrote:
> Hi team,
>
> Regarding discussion on presence of version field concern that was raised
> by Antoine.
>
> Antoine suggested removing the version field from the ALP header.
>
> 0
>
> version
>
> 1 byte
>
> uint8
>
> Format version (current is 1)
>
>
> Concern raised was along the lines of it complicating the API.
>
> Having thought through this more I now agree with Antoine.
>
> In a little more formal words.
> *Parquet's standard practice is to handle format evolution at the Thrift
> metadata level by introducing a new*
> *Encoding enum, **rather than embedding version numbers within the page's
> binary data stream. **Managing*
> *versioning through metadata allows readers to immediately identify
> unsupported formats and fail gracefully*
> *before they even begin reading or decoding the page. *
>
> We touched upon it in the last parquet meeting and I remember people in the
> meeting also agreed with the suggestion.
>
> Now the issue is that the header was exactly 8 bytes :). Do I keep 1 byte
> as padding or move the header to 7 bytes?
> Any recommendations?
>
> On Tue, Feb 17, 2026 at 4:27 PM PRATEEK GAUR <pr...@gmail.com> wrote:
>
> > Hi team,
> >
> > 1) Andrew
> >
> > - Thanks for working on test files. My PR did add all the test files I
> > used to benchmark on datasets. Maybe we can club it together. WIll
> also aid
> > cross language testing
> > - Kosta Tarasov working on Rust implementation. This is great. Thanks
> >
> >
> > 2) Antoine
> >
> > - Thanks a lot for reporting the numbers on AMD. Looks like you are
> > getting 8X the decoding performance of BSS. This is amazing!!.
> > - Thanks for acknowledging the sampling design.
> > - I agree with you on Fastlanes. In some crude experiments I didn't
> > get a good perf benefit from it on Graviton3 (but maybe there was
> something
> > wrong with my implementation).
> > - Locking the 16bit exception encoding for the spec in this case.
> > - Awesome I think we have solved for all open questions minus the
> > version byte :). (will get back on this soon)
> >
> >
> > 3) Micah
> >
> > - FastLanes : The current spec does allow for using FastLane with the
> > configurable enum value for layout. We should be able to inject any
> layout
> > in the current design.
> >
> >
> > Working on resolving all remaining open comments on the spec this week.
> >
> > Best
> > Prateek
> >
> >
> > On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <st...@cloudera.com>
> > wrote:
> >
> >> On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <em...@gmail.com>
> >> wrote:
> >>
> >> >
> >> >
> >> > It looks like the actual issue described for ORC in the paper is that
> it
> >> > has multiple sub-encodings in a batch. This is different then the
> >> design
> >> > proposed here where there is still fixed encoding per page in parquet.
> >> > Given reasonably sized pages I don't think branch misprediction should
> >> be a
> >> > big issue for new encodings. I agree that we should be conservative
> in
> >> > general for adding new encodings.
> >> >
> >> >
> >> +1
> >>
> >
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by PRATEEK GAUR <pr...@gmail.com>.
Hi team,
Regarding discussion on presence of version field concern that was raised
by Antoine.
Antoine suggested removing the version field from the ALP header.
0
version
1 byte
uint8
Format version (current is 1)
Concern raised was along the lines of it complicating the API.
Having thought through this more I now agree with Antoine.
In a little more formal words.
*Parquet's standard practice is to handle format evolution at the Thrift
metadata level by introducing a new*
*Encoding enum, **rather than embedding version numbers within the page's
binary data stream. **Managing*
*versioning through metadata allows readers to immediately identify
unsupported formats and fail gracefully*
*before they even begin reading or decoding the page. *
We touched upon it in the last parquet meeting and I remember people in the
meeting also agreed with the suggestion.
Now the issue is that the header was exactly 8 bytes :). Do I keep 1 byte
as padding or move the header to 7 bytes?
Any recommendations?
On Tue, Feb 17, 2026 at 4:27 PM PRATEEK GAUR <pr...@gmail.com> wrote:
> Hi team,
>
> 1) Andrew
>
> - Thanks for working on test files. My PR did add all the test files I
> used to benchmark on datasets. Maybe we can club it together. WIll also aid
> cross language testing
> - Kosta Tarasov working on Rust implementation. This is great. Thanks
>
>
> 2) Antoine
>
> - Thanks a lot for reporting the numbers on AMD. Looks like you are
> getting 8X the decoding performance of BSS. This is amazing!!.
> - Thanks for acknowledging the sampling design.
> - I agree with you on Fastlanes. In some crude experiments I didn't
> get a good perf benefit from it on Graviton3 (but maybe there was something
> wrong with my implementation).
> - Locking the 16bit exception encoding for the spec in this case.
> - Awesome I think we have solved for all open questions minus the
> version byte :). (will get back on this soon)
>
>
> 3) Micah
>
> - FastLanes : The current spec does allow for using FastLane with the
> configurable enum value for layout. We should be able to inject any layout
> in the current design.
>
>
> Working on resolving all remaining open comments on the spec this week.
>
> Best
> Prateek
>
>
> On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <st...@cloudera.com>
> wrote:
>
>> On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <em...@gmail.com>
>> wrote:
>>
>> >
>> >
>> > It looks like the actual issue described for ORC in the paper is that it
>> > has multiple sub-encodings in a batch. This is different then the
>> design
>> > proposed here where there is still fixed encoding per page in parquet.
>> > Given reasonably sized pages I don't think branch misprediction should
>> be a
>> > big issue for new encodings. I agree that we should be conservative in
>> > general for adding new encodings.
>> >
>> >
>> +1
>>
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Andrew Lamb <an...@gmail.com>.
I also just filed a ticket[1] to track adding the example files and linked
it around to try and give it a bit more visibility.
[1]: https://github.com/apache/parquet-testing/issues/105
On Wed, Apr 29, 2026 at 7:25 PM Curt Hagenlocher <cu...@hagenlocher.org>
wrote:
> Ah, thanks! I missed that.
>
> On Wed, Apr 29, 2026 at 4:24 PM Micah Kornfield <em...@gmail.com>
> wrote:
>
>> Hi Curt,
>>
>>> As part of the process of amending the Parquet format, perhaps it would
>>> be a good idea for early implementations to generate sample files and
>>> commit them to apache/parquet-testing: Apache Parquet Testing
>>> <https://github.com/apache/parquet-testing> for other implementations
>>> to leverage?
>>
>>
>> It got dropped in the thread but does
>> https://github.com/apache/parquet-testing/pull/100 address your concerns?
>>
>> Thanks,
>> Micah
>>
>> On Wed, Apr 29, 2026 at 4:20 PM Curt Hagenlocher <cu...@hagenlocher.org>
>> wrote:
>>
>>> As part of the process of amending the Parquet format, perhaps it would
>>> be a good idea for early implementations to generate sample files and
>>> commit them to apache/parquet-testing: Apache Parquet Testing
>>> <https://github.com/apache/parquet-testing> for other implementations
>>> to leverage?
>>>
>>> -Curt
>>>
>>> On Wed, Apr 29, 2026 at 4:11 PM PRATEEK GAUR <pr...@gmail.com> wrote:
>>>
>>>> Thanks Andrew and Micah for review feedback on the two PR's
>>>> 1) (c++ arrow repo) https://github.com/apache/arrow/pull/48345/changes
>>>> 2) (parquet-format repo)
>>>> https://github.com/apache/parquet-format/pull/557
>>>>
>>>> I have addressed all (unless I missed something) comments on the two
>>>> PR's.
>>>>
>>>> Best
>>>> Prateek
>>>>
>>>> On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <pr...@gmail.com>
>>>> wrote:
>>>>
>>>> > Thanks Andrew and Micah.
>>>> >
>>>> > `fair amount of feedback on at least the implementations`
>>>> > For the c++ I have already started addressing the feedback, I should
>>>> be
>>>> > done with that Monday/Tuesday.
>>>> > I think Vinoo too has been making good progress on the Java
>>>> implementation.
>>>> >
>>>> > Best
>>>> > Prateek
>>>> >
>>>> > On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <an...@gmail.com>
>>>> > wrote:
>>>> >
>>>> >> Got it. Thank you for the clarification -- I will try and look into
>>>> the
>>>> >> spec and the Rust implementation[1] in this next week
>>>> >>
>>>> >> [1]: https://github.com/apache/arrow-rs/pull/9372
>>>> >>
>>>> >> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <
>>>> emkornfield@gmail.com>
>>>> >> wrote:
>>>> >>
>>>> >>> Hi Andrew,
>>>> >>> I think there is a fair amount of feedback on at least the
>>>> >>> implementations, typically I think we've waited till they are close
>>>> to
>>>> >>> mergeable before a final vote. Otherwise I agree we are very close.
>>>> >>>
>>>> >>> -Micah
>>>> >>>
>>>> >>> On Saturday, April 25, 2026, Andrew Lamb <an...@gmail.com>
>>>> wrote:
>>>> >>>
>>>> >>>> Thanks Prateek,
>>>> >>>>
>>>> >>>> I think from this content it looks to me like we are ready to
>>>> start a
>>>> >>>> vote to explicitly accept ALP into Parquet
>>>> >>>>
>>>> >>>> Does anyone know of a reason we should postpone it for longer?
>>>> >>>> Perhaps someone needs some more time to review?
>>>> >>>>
>>>> >>>> Andrew
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <pr...@gmail.com>
>>>> >>>> wrote:
>>>> >>>>
>>>> >>>>> Hi team,
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> Hope everyone is doing well. I got a chance to work through all
>>>> the
>>>> >>>>> remaining feedback and update the spec doc. Here are the new
>>>> artifacts
>>>> >>>>>
>>>> >>>>> 1) Spec document :
>>>> >>>>>
>>>> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
>>>> >>>>>
>>>> >>>>> 2) Spec document in parquet format repo :
>>>> >>>>> https://github.com/apache/parquet-format/pull/557
>>>> >>>>>
>>>> >>>>> 3) Alp implementation in arrow c++ repo :
>>>> >>>>> https://github.com/apache/arrow/pull/48345/changes
>>>> >>>>>
>>>> >>>>> 4) Alp implementation in parquet-java repo : Work for Vinoo and
>>>> Julien
>>>> >>>>> https://github.com/apache/parquet-java/pull/3397
>>>> >>>>>
>>>> >>>>> 5) PR with test and benchmarking artifacts in parquet-testing
>>>> repo :
>>>> >>>>> https://github.com/apache/parquet-testing/pull/100
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> And
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> - Go : Arnav just submitted an in progress implementation in
>>>> Go.
>>>> >>>>> https://github.com/apache/arrow-go/pull/704 (I haven't started
>>>> >>>>> looking at it yet)
>>>> >>>>> - Rust : I remember Andrew mentioned that this work is also in
>>>> >>>>> progress (So 4 languages!)
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> *Arrow C++ implementation *
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> The PR is out and was also used by Antoine to report the numbers
>>>> as
>>>> >>>>> reported here. Micah and Konstantin have given 1 round of feedback
>>>> >>>>> and I'm addressing them today. Please note that the default
>>>> >>>>> optimization flag for compiling is O2 and not Q3. I got around 70%
>>>> >>>>> performance improvement in the decoding speed when using the O3
>>>> flag.
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> *Parqet-MR Java implementation (working with Vinoo and Julien)
>>>> and **Cross
>>>> >>>>> Language testing*
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> Let me know if you have any questions or feedback.
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> Now pasting some performance numbers
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> Table 1: C++ ALP Double Decode — Spotify Columns (Graviton 3,
>>>> ARM
>>>> >>>>> Neoverse V1)
>>>> >>>>>
>>>> >>>>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
>>>> >>>>>
>>>> >>>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │ Speedup │
>>>> >>>>>
>>>> >>>>> ├──────────────────┼──────────────┼──────────────┼─────────┤
>>>> >>>>>
>>>> >>>>> │ valence │ 3,155 │ 5,523 │ 1.75x │
>>>> >>>>>
>>>> >>>>> │ danceability │ 3,233 │ 5,685 │ 1.76x │
>>>> >>>>>
>>>> >>>>> │ energy │ 3,197 │ 5,652 │ 1.77x │
>>>> >>>>>
>>>> >>>>> │ loudness │ 3,186 │ 5,473 │ 1.72x │
>>>> >>>>>
>>>> >>>>> └──────────────────┴──────────────┴──────────────┴─────────┘
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <pr...@gmail.com>
>>>> >>>>> wrote:
>>>> >>>>>
>>>> >>>>>> @Micah Kornfield <em...@gmail.com> : Got it.
>>>> >>>>>>
>>>> >>>>>> @Andrew Lamb <an...@gmail.com>
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>>> Do you think it would be good to start moving the spec
>>>> development
>>>> >>>>>>> into
>>>> >>>>>>> markdown format, in preparation for finalizing it?
>>>> >>>>>>>
>>>> >>>>>>
>>>> >>>>>> Yes I'll update the numbers for some of the examples I have in
>>>> the
>>>> >>>>>> spec based
>>>> >>>>>> on the updated header size. Then we should be good to go for the
>>>> >>>>>> markdown format.
>>>> >>>>>>
>>>> >>>>>> Thanks everyone!
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>>>
>>>> >>>>>>> Andrew
>>>> >>>>>>>
>>>> >>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <
>>>> prtkgaur@gmail.com>
>>>> >>>>>>> wrote:
>>>> >>>>>>>
>>>> >>>>>>> > Hi team,
>>>> >>>>>>> >
>>>> >>>>>>> > 1) Andrew
>>>> >>>>>>> >
>>>> >>>>>>> > - Thanks for working on test files. My PR did add all the
>>>> test
>>>> >>>>>>> files I
>>>> >>>>>>> > used to benchmark on datasets. Maybe we can club it
>>>> together.
>>>> >>>>>>> WIll also
>>>> >>>>>>> > aid
>>>> >>>>>>> > cross language testing
>>>> >>>>>>> > - Kosta Tarasov working on Rust implementation. This is
>>>> great.
>>>> >>>>>>> Thanks
>>>> >>>>>>> >
>>>> >>>>>>> >
>>>> >>>>>>> > 2) Antoine
>>>> >>>>>>> >
>>>> >>>>>>> > - Thanks a lot for reporting the numbers on AMD. Looks
>>>> like you
>>>> >>>>>>> are
>>>> >>>>>>> > getting 8X the decoding performance of BSS. This is
>>>> amazing!!.
>>>> >>>>>>> > - Thanks for acknowledging the sampling design.
>>>> >>>>>>> > - I agree with you on Fastlanes. In some crude experiments
>>>> I
>>>> >>>>>>> didn't get
>>>> >>>>>>> > a good perf benefit from it on Graviton3 (but maybe there
>>>> was
>>>> >>>>>>> something
>>>> >>>>>>> > wrong with my implementation).
>>>> >>>>>>> > - Locking the 16bit exception encoding for the spec in this
>>>> >>>>>>> case.
>>>> >>>>>>> > - Awesome I think we have solved for all open questions
>>>> minus
>>>> >>>>>>> the
>>>> >>>>>>> > version byte :). (will get back on this soon)
>>>> >>>>>>> >
>>>> >>>>>>> >
>>>> >>>>>>> > 3) Micah
>>>> >>>>>>> >
>>>> >>>>>>> > - FastLanes : The current spec does allow for using
>>>> FastLane
>>>> >>>>>>> with the
>>>> >>>>>>> > configurable enum value for layout. We should be able to
>>>> inject
>>>> >>>>>>> any
>>>> >>>>>>> > layout
>>>> >>>>>>> > in the current design.
>>>> >>>>>>> >
>>>> >>>>>>> >
>>>> >>>>>>> > Working on resolving all remaining open comments on the spec
>>>> this
>>>> >>>>>>> week.
>>>> >>>>>>> >
>>>> >>>>>>> > Best
>>>> >>>>>>> > Prateek
>>>> >>>>>>> >
>>>> >>>>>>> >
>>>> >>>>>>> > On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <
>>>> >>>>>>> stevel@cloudera.com>
>>>> >>>>>>> > wrote:
>>>> >>>>>>> >
>>>> >>>>>>> > > On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
>>>> >>>>>>> emkornfield@gmail.com>
>>>> >>>>>>> > > wrote:
>>>> >>>>>>> > >
>>>> >>>>>>> > > >
>>>> >>>>>>> > > >
>>>> >>>>>>> > > > It looks like the actual issue described for ORC in the
>>>> paper
>>>> >>>>>>> is that
>>>> >>>>>>> > it
>>>> >>>>>>> > > > has multiple sub-encodings in a batch. This is different
>>>> then
>>>> >>>>>>> the
>>>> >>>>>>> > design
>>>> >>>>>>> > > > proposed here where there is still fixed encoding per
>>>> page in
>>>> >>>>>>> parquet.
>>>> >>>>>>> > > > Given reasonably sized pages I don't think branch
>>>> >>>>>>> misprediction should
>>>> >>>>>>> > > be a
>>>> >>>>>>> > > > big issue for new encodings. I agree that we should be
>>>> >>>>>>> conservative in
>>>> >>>>>>> > > > general for adding new encodings.
>>>> >>>>>>> > > >
>>>> >>>>>>> > > >
>>>> >>>>>>> > > +1
>>>> >>>>>>> > >
>>>> >>>>>>> >
>>>> >>>>>>>
>>>> >>>>>>
>>>>
>>>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Curt Hagenlocher <cu...@hagenlocher.org>.
Ah, thanks! I missed that.
On Wed, Apr 29, 2026 at 4:24 PM Micah Kornfield <em...@gmail.com>
wrote:
> Hi Curt,
>
>> As part of the process of amending the Parquet format, perhaps it would
>> be a good idea for early implementations to generate sample files and
>> commit them to apache/parquet-testing: Apache Parquet Testing
>> <https://github.com/apache/parquet-testing> for other implementations to
>> leverage?
>
>
> It got dropped in the thread but does
> https://github.com/apache/parquet-testing/pull/100 address your concerns?
>
> Thanks,
> Micah
>
> On Wed, Apr 29, 2026 at 4:20 PM Curt Hagenlocher <cu...@hagenlocher.org>
> wrote:
>
>> As part of the process of amending the Parquet format, perhaps it would
>> be a good idea for early implementations to generate sample files and
>> commit them to apache/parquet-testing: Apache Parquet Testing
>> <https://github.com/apache/parquet-testing> for other implementations to
>> leverage?
>>
>> -Curt
>>
>> On Wed, Apr 29, 2026 at 4:11 PM PRATEEK GAUR <pr...@gmail.com> wrote:
>>
>>> Thanks Andrew and Micah for review feedback on the two PR's
>>> 1) (c++ arrow repo) https://github.com/apache/arrow/pull/48345/changes
>>> 2) (parquet-format repo)
>>> https://github.com/apache/parquet-format/pull/557
>>>
>>> I have addressed all (unless I missed something) comments on the two
>>> PR's.
>>>
>>> Best
>>> Prateek
>>>
>>> On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <pr...@gmail.com> wrote:
>>>
>>> > Thanks Andrew and Micah.
>>> >
>>> > `fair amount of feedback on at least the implementations`
>>> > For the c++ I have already started addressing the feedback, I should be
>>> > done with that Monday/Tuesday.
>>> > I think Vinoo too has been making good progress on the Java
>>> implementation.
>>> >
>>> > Best
>>> > Prateek
>>> >
>>> > On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <an...@gmail.com>
>>> > wrote:
>>> >
>>> >> Got it. Thank you for the clarification -- I will try and look into
>>> the
>>> >> spec and the Rust implementation[1] in this next week
>>> >>
>>> >> [1]: https://github.com/apache/arrow-rs/pull/9372
>>> >>
>>> >> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <
>>> emkornfield@gmail.com>
>>> >> wrote:
>>> >>
>>> >>> Hi Andrew,
>>> >>> I think there is a fair amount of feedback on at least the
>>> >>> implementations, typically I think we've waited till they are close
>>> to
>>> >>> mergeable before a final vote. Otherwise I agree we are very close.
>>> >>>
>>> >>> -Micah
>>> >>>
>>> >>> On Saturday, April 25, 2026, Andrew Lamb <an...@gmail.com>
>>> wrote:
>>> >>>
>>> >>>> Thanks Prateek,
>>> >>>>
>>> >>>> I think from this content it looks to me like we are ready to start
>>> a
>>> >>>> vote to explicitly accept ALP into Parquet
>>> >>>>
>>> >>>> Does anyone know of a reason we should postpone it for longer?
>>> >>>> Perhaps someone needs some more time to review?
>>> >>>>
>>> >>>> Andrew
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <pr...@gmail.com>
>>> >>>> wrote:
>>> >>>>
>>> >>>>> Hi team,
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> Hope everyone is doing well. I got a chance to work through all the
>>> >>>>> remaining feedback and update the spec doc. Here are the new
>>> artifacts
>>> >>>>>
>>> >>>>> 1) Spec document :
>>> >>>>>
>>> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
>>> >>>>>
>>> >>>>> 2) Spec document in parquet format repo :
>>> >>>>> https://github.com/apache/parquet-format/pull/557
>>> >>>>>
>>> >>>>> 3) Alp implementation in arrow c++ repo :
>>> >>>>> https://github.com/apache/arrow/pull/48345/changes
>>> >>>>>
>>> >>>>> 4) Alp implementation in parquet-java repo : Work for Vinoo and
>>> Julien
>>> >>>>> https://github.com/apache/parquet-java/pull/3397
>>> >>>>>
>>> >>>>> 5) PR with test and benchmarking artifacts in parquet-testing repo
>>> :
>>> >>>>> https://github.com/apache/parquet-testing/pull/100
>>> >>>>>
>>> >>>>>
>>> >>>>> And
>>> >>>>>
>>> >>>>>
>>> >>>>> - Go : Arnav just submitted an in progress implementation in Go.
>>> >>>>> https://github.com/apache/arrow-go/pull/704 (I haven't started
>>> >>>>> looking at it yet)
>>> >>>>> - Rust : I remember Andrew mentioned that this work is also in
>>> >>>>> progress (So 4 languages!)
>>> >>>>>
>>> >>>>>
>>> >>>>> *Arrow C++ implementation *
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> The PR is out and was also used by Antoine to report the numbers as
>>> >>>>> reported here. Micah and Konstantin have given 1 round of feedback
>>> >>>>> and I'm addressing them today. Please note that the default
>>> >>>>> optimization flag for compiling is O2 and not Q3. I got around 70%
>>> >>>>> performance improvement in the decoding speed when using the O3
>>> flag.
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> *Parqet-MR Java implementation (working with Vinoo and Julien) and
>>> **Cross
>>> >>>>> Language testing*
>>> >>>>>
>>> >>>>>
>>> >>>>> Let me know if you have any questions or feedback.
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> Now pasting some performance numbers
>>> >>>>>
>>> >>>>>
>>> >>>>> Table 1: C++ ALP Double Decode — Spotify Columns (Graviton 3, ARM
>>> >>>>> Neoverse V1)
>>> >>>>>
>>> >>>>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
>>> >>>>>
>>> >>>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │ Speedup │
>>> >>>>>
>>> >>>>> ├──────────────────┼──────────────┼──────────────┼─────────┤
>>> >>>>>
>>> >>>>> │ valence │ 3,155 │ 5,523 │ 1.75x │
>>> >>>>>
>>> >>>>> │ danceability │ 3,233 │ 5,685 │ 1.76x │
>>> >>>>>
>>> >>>>> │ energy │ 3,197 │ 5,652 │ 1.77x │
>>> >>>>>
>>> >>>>> │ loudness │ 3,186 │ 5,473 │ 1.72x │
>>> >>>>>
>>> >>>>> └──────────────────┴──────────────┴──────────────┴─────────┘
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <pr...@gmail.com>
>>> >>>>> wrote:
>>> >>>>>
>>> >>>>>> @Micah Kornfield <em...@gmail.com> : Got it.
>>> >>>>>>
>>> >>>>>> @Andrew Lamb <an...@gmail.com>
>>> >>>>>>
>>> >>>>>>
>>> >>>>>>> Do you think it would be good to start moving the spec
>>> development
>>> >>>>>>> into
>>> >>>>>>> markdown format, in preparation for finalizing it?
>>> >>>>>>>
>>> >>>>>>
>>> >>>>>> Yes I'll update the numbers for some of the examples I have in the
>>> >>>>>> spec based
>>> >>>>>> on the updated header size. Then we should be good to go for the
>>> >>>>>> markdown format.
>>> >>>>>>
>>> >>>>>> Thanks everyone!
>>> >>>>>>
>>> >>>>>>
>>> >>>>>>>
>>> >>>>>>> Andrew
>>> >>>>>>>
>>> >>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <prtkgaur@gmail.com
>>> >
>>> >>>>>>> wrote:
>>> >>>>>>>
>>> >>>>>>> > Hi team,
>>> >>>>>>> >
>>> >>>>>>> > 1) Andrew
>>> >>>>>>> >
>>> >>>>>>> > - Thanks for working on test files. My PR did add all the
>>> test
>>> >>>>>>> files I
>>> >>>>>>> > used to benchmark on datasets. Maybe we can club it
>>> together.
>>> >>>>>>> WIll also
>>> >>>>>>> > aid
>>> >>>>>>> > cross language testing
>>> >>>>>>> > - Kosta Tarasov working on Rust implementation. This is
>>> great.
>>> >>>>>>> Thanks
>>> >>>>>>> >
>>> >>>>>>> >
>>> >>>>>>> > 2) Antoine
>>> >>>>>>> >
>>> >>>>>>> > - Thanks a lot for reporting the numbers on AMD. Looks like
>>> you
>>> >>>>>>> are
>>> >>>>>>> > getting 8X the decoding performance of BSS. This is
>>> amazing!!.
>>> >>>>>>> > - Thanks for acknowledging the sampling design.
>>> >>>>>>> > - I agree with you on Fastlanes. In some crude experiments I
>>> >>>>>>> didn't get
>>> >>>>>>> > a good perf benefit from it on Graviton3 (but maybe there
>>> was
>>> >>>>>>> something
>>> >>>>>>> > wrong with my implementation).
>>> >>>>>>> > - Locking the 16bit exception encoding for the spec in this
>>> >>>>>>> case.
>>> >>>>>>> > - Awesome I think we have solved for all open questions
>>> minus
>>> >>>>>>> the
>>> >>>>>>> > version byte :). (will get back on this soon)
>>> >>>>>>> >
>>> >>>>>>> >
>>> >>>>>>> > 3) Micah
>>> >>>>>>> >
>>> >>>>>>> > - FastLanes : The current spec does allow for using FastLane
>>> >>>>>>> with the
>>> >>>>>>> > configurable enum value for layout. We should be able to
>>> inject
>>> >>>>>>> any
>>> >>>>>>> > layout
>>> >>>>>>> > in the current design.
>>> >>>>>>> >
>>> >>>>>>> >
>>> >>>>>>> > Working on resolving all remaining open comments on the spec
>>> this
>>> >>>>>>> week.
>>> >>>>>>> >
>>> >>>>>>> > Best
>>> >>>>>>> > Prateek
>>> >>>>>>> >
>>> >>>>>>> >
>>> >>>>>>> > On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <
>>> >>>>>>> stevel@cloudera.com>
>>> >>>>>>> > wrote:
>>> >>>>>>> >
>>> >>>>>>> > > On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
>>> >>>>>>> emkornfield@gmail.com>
>>> >>>>>>> > > wrote:
>>> >>>>>>> > >
>>> >>>>>>> > > >
>>> >>>>>>> > > >
>>> >>>>>>> > > > It looks like the actual issue described for ORC in the
>>> paper
>>> >>>>>>> is that
>>> >>>>>>> > it
>>> >>>>>>> > > > has multiple sub-encodings in a batch. This is different
>>> then
>>> >>>>>>> the
>>> >>>>>>> > design
>>> >>>>>>> > > > proposed here where there is still fixed encoding per page
>>> in
>>> >>>>>>> parquet.
>>> >>>>>>> > > > Given reasonably sized pages I don't think branch
>>> >>>>>>> misprediction should
>>> >>>>>>> > > be a
>>> >>>>>>> > > > big issue for new encodings. I agree that we should be
>>> >>>>>>> conservative in
>>> >>>>>>> > > > general for adding new encodings.
>>> >>>>>>> > > >
>>> >>>>>>> > > >
>>> >>>>>>> > > +1
>>> >>>>>>> > >
>>> >>>>>>> >
>>> >>>>>>>
>>> >>>>>>
>>>
>>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Micah Kornfield <em...@gmail.com>.
Hi Curt,
> As part of the process of amending the Parquet format, perhaps it would be
> a good idea for early implementations to generate sample files and commit
> them to apache/parquet-testing: Apache Parquet Testing
> <https://github.com/apache/parquet-testing> for other implementations to
> leverage?
It got dropped in the thread but does
https://github.com/apache/parquet-testing/pull/100 address your concerns?
Thanks,
Micah
On Wed, Apr 29, 2026 at 4:20 PM Curt Hagenlocher <cu...@hagenlocher.org>
wrote:
> As part of the process of amending the Parquet format, perhaps it would be
> a good idea for early implementations to generate sample files and commit
> them to apache/parquet-testing: Apache Parquet Testing
> <https://github.com/apache/parquet-testing> for other implementations to
> leverage?
>
> -Curt
>
> On Wed, Apr 29, 2026 at 4:11 PM PRATEEK GAUR <pr...@gmail.com> wrote:
>
>> Thanks Andrew and Micah for review feedback on the two PR's
>> 1) (c++ arrow repo) https://github.com/apache/arrow/pull/48345/changes
>> 2) (parquet-format repo)
>> https://github.com/apache/parquet-format/pull/557
>>
>> I have addressed all (unless I missed something) comments on the two PR's.
>>
>> Best
>> Prateek
>>
>> On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <pr...@gmail.com> wrote:
>>
>> > Thanks Andrew and Micah.
>> >
>> > `fair amount of feedback on at least the implementations`
>> > For the c++ I have already started addressing the feedback, I should be
>> > done with that Monday/Tuesday.
>> > I think Vinoo too has been making good progress on the Java
>> implementation.
>> >
>> > Best
>> > Prateek
>> >
>> > On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <an...@gmail.com>
>> > wrote:
>> >
>> >> Got it. Thank you for the clarification -- I will try and look into the
>> >> spec and the Rust implementation[1] in this next week
>> >>
>> >> [1]: https://github.com/apache/arrow-rs/pull/9372
>> >>
>> >> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <
>> emkornfield@gmail.com>
>> >> wrote:
>> >>
>> >>> Hi Andrew,
>> >>> I think there is a fair amount of feedback on at least the
>> >>> implementations, typically I think we've waited till they are close to
>> >>> mergeable before a final vote. Otherwise I agree we are very close.
>> >>>
>> >>> -Micah
>> >>>
>> >>> On Saturday, April 25, 2026, Andrew Lamb <an...@gmail.com>
>> wrote:
>> >>>
>> >>>> Thanks Prateek,
>> >>>>
>> >>>> I think from this content it looks to me like we are ready to start a
>> >>>> vote to explicitly accept ALP into Parquet
>> >>>>
>> >>>> Does anyone know of a reason we should postpone it for longer?
>> >>>> Perhaps someone needs some more time to review?
>> >>>>
>> >>>> Andrew
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <pr...@gmail.com>
>> >>>> wrote:
>> >>>>
>> >>>>> Hi team,
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> Hope everyone is doing well. I got a chance to work through all the
>> >>>>> remaining feedback and update the spec doc. Here are the new
>> artifacts
>> >>>>>
>> >>>>> 1) Spec document :
>> >>>>>
>> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
>> >>>>>
>> >>>>> 2) Spec document in parquet format repo :
>> >>>>> https://github.com/apache/parquet-format/pull/557
>> >>>>>
>> >>>>> 3) Alp implementation in arrow c++ repo :
>> >>>>> https://github.com/apache/arrow/pull/48345/changes
>> >>>>>
>> >>>>> 4) Alp implementation in parquet-java repo : Work for Vinoo and
>> Julien
>> >>>>> https://github.com/apache/parquet-java/pull/3397
>> >>>>>
>> >>>>> 5) PR with test and benchmarking artifacts in parquet-testing repo :
>> >>>>> https://github.com/apache/parquet-testing/pull/100
>> >>>>>
>> >>>>>
>> >>>>> And
>> >>>>>
>> >>>>>
>> >>>>> - Go : Arnav just submitted an in progress implementation in Go.
>> >>>>> https://github.com/apache/arrow-go/pull/704 (I haven't started
>> >>>>> looking at it yet)
>> >>>>> - Rust : I remember Andrew mentioned that this work is also in
>> >>>>> progress (So 4 languages!)
>> >>>>>
>> >>>>>
>> >>>>> *Arrow C++ implementation *
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> The PR is out and was also used by Antoine to report the numbers as
>> >>>>> reported here. Micah and Konstantin have given 1 round of feedback
>> >>>>> and I'm addressing them today. Please note that the default
>> >>>>> optimization flag for compiling is O2 and not Q3. I got around 70%
>> >>>>> performance improvement in the decoding speed when using the O3
>> flag.
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> *Parqet-MR Java implementation (working with Vinoo and Julien) and
>> **Cross
>> >>>>> Language testing*
>> >>>>>
>> >>>>>
>> >>>>> Let me know if you have any questions or feedback.
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> Now pasting some performance numbers
>> >>>>>
>> >>>>>
>> >>>>> Table 1: C++ ALP Double Decode — Spotify Columns (Graviton 3, ARM
>> >>>>> Neoverse V1)
>> >>>>>
>> >>>>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
>> >>>>>
>> >>>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │ Speedup │
>> >>>>>
>> >>>>> ├──────────────────┼──────────────┼──────────────┼─────────┤
>> >>>>>
>> >>>>> │ valence │ 3,155 │ 5,523 │ 1.75x │
>> >>>>>
>> >>>>> │ danceability │ 3,233 │ 5,685 │ 1.76x │
>> >>>>>
>> >>>>> │ energy │ 3,197 │ 5,652 │ 1.77x │
>> >>>>>
>> >>>>> │ loudness │ 3,186 │ 5,473 │ 1.72x │
>> >>>>>
>> >>>>> └──────────────────┴──────────────┴──────────────┴─────────┘
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <pr...@gmail.com>
>> >>>>> wrote:
>> >>>>>
>> >>>>>> @Micah Kornfield <em...@gmail.com> : Got it.
>> >>>>>>
>> >>>>>> @Andrew Lamb <an...@gmail.com>
>> >>>>>>
>> >>>>>>
>> >>>>>>> Do you think it would be good to start moving the spec development
>> >>>>>>> into
>> >>>>>>> markdown format, in preparation for finalizing it?
>> >>>>>>>
>> >>>>>>
>> >>>>>> Yes I'll update the numbers for some of the examples I have in the
>> >>>>>> spec based
>> >>>>>> on the updated header size. Then we should be good to go for the
>> >>>>>> markdown format.
>> >>>>>>
>> >>>>>> Thanks everyone!
>> >>>>>>
>> >>>>>>
>> >>>>>>>
>> >>>>>>> Andrew
>> >>>>>>>
>> >>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <pr...@gmail.com>
>> >>>>>>> wrote:
>> >>>>>>>
>> >>>>>>> > Hi team,
>> >>>>>>> >
>> >>>>>>> > 1) Andrew
>> >>>>>>> >
>> >>>>>>> > - Thanks for working on test files. My PR did add all the
>> test
>> >>>>>>> files I
>> >>>>>>> > used to benchmark on datasets. Maybe we can club it together.
>> >>>>>>> WIll also
>> >>>>>>> > aid
>> >>>>>>> > cross language testing
>> >>>>>>> > - Kosta Tarasov working on Rust implementation. This is
>> great.
>> >>>>>>> Thanks
>> >>>>>>> >
>> >>>>>>> >
>> >>>>>>> > 2) Antoine
>> >>>>>>> >
>> >>>>>>> > - Thanks a lot for reporting the numbers on AMD. Looks like
>> you
>> >>>>>>> are
>> >>>>>>> > getting 8X the decoding performance of BSS. This is
>> amazing!!.
>> >>>>>>> > - Thanks for acknowledging the sampling design.
>> >>>>>>> > - I agree with you on Fastlanes. In some crude experiments I
>> >>>>>>> didn't get
>> >>>>>>> > a good perf benefit from it on Graviton3 (but maybe there was
>> >>>>>>> something
>> >>>>>>> > wrong with my implementation).
>> >>>>>>> > - Locking the 16bit exception encoding for the spec in this
>> >>>>>>> case.
>> >>>>>>> > - Awesome I think we have solved for all open questions minus
>> >>>>>>> the
>> >>>>>>> > version byte :). (will get back on this soon)
>> >>>>>>> >
>> >>>>>>> >
>> >>>>>>> > 3) Micah
>> >>>>>>> >
>> >>>>>>> > - FastLanes : The current spec does allow for using FastLane
>> >>>>>>> with the
>> >>>>>>> > configurable enum value for layout. We should be able to
>> inject
>> >>>>>>> any
>> >>>>>>> > layout
>> >>>>>>> > in the current design.
>> >>>>>>> >
>> >>>>>>> >
>> >>>>>>> > Working on resolving all remaining open comments on the spec
>> this
>> >>>>>>> week.
>> >>>>>>> >
>> >>>>>>> > Best
>> >>>>>>> > Prateek
>> >>>>>>> >
>> >>>>>>> >
>> >>>>>>> > On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <
>> >>>>>>> stevel@cloudera.com>
>> >>>>>>> > wrote:
>> >>>>>>> >
>> >>>>>>> > > On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
>> >>>>>>> emkornfield@gmail.com>
>> >>>>>>> > > wrote:
>> >>>>>>> > >
>> >>>>>>> > > >
>> >>>>>>> > > >
>> >>>>>>> > > > It looks like the actual issue described for ORC in the
>> paper
>> >>>>>>> is that
>> >>>>>>> > it
>> >>>>>>> > > > has multiple sub-encodings in a batch. This is different
>> then
>> >>>>>>> the
>> >>>>>>> > design
>> >>>>>>> > > > proposed here where there is still fixed encoding per page
>> in
>> >>>>>>> parquet.
>> >>>>>>> > > > Given reasonably sized pages I don't think branch
>> >>>>>>> misprediction should
>> >>>>>>> > > be a
>> >>>>>>> > > > big issue for new encodings. I agree that we should be
>> >>>>>>> conservative in
>> >>>>>>> > > > general for adding new encodings.
>> >>>>>>> > > >
>> >>>>>>> > > >
>> >>>>>>> > > +1
>> >>>>>>> > >
>> >>>>>>> >
>> >>>>>>>
>> >>>>>>
>>
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Curt Hagenlocher <cu...@hagenlocher.org>.
As part of the process of amending the Parquet format, perhaps it would be
a good idea for early implementations to generate sample files and commit
them to apache/parquet-testing: Apache Parquet Testing
<https://github.com/apache/parquet-testing> for other implementations to
leverage?
-Curt
On Wed, Apr 29, 2026 at 4:11 PM PRATEEK GAUR <pr...@gmail.com> wrote:
> Thanks Andrew and Micah for review feedback on the two PR's
> 1) (c++ arrow repo) https://github.com/apache/arrow/pull/48345/changes
> 2) (parquet-format repo) https://github.com/apache/parquet-format/pull/557
>
> I have addressed all (unless I missed something) comments on the two PR's.
>
> Best
> Prateek
>
> On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <pr...@gmail.com> wrote:
>
> > Thanks Andrew and Micah.
> >
> > `fair amount of feedback on at least the implementations`
> > For the c++ I have already started addressing the feedback, I should be
> > done with that Monday/Tuesday.
> > I think Vinoo too has been making good progress on the Java
> implementation.
> >
> > Best
> > Prateek
> >
> > On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <an...@gmail.com>
> > wrote:
> >
> >> Got it. Thank you for the clarification -- I will try and look into the
> >> spec and the Rust implementation[1] in this next week
> >>
> >> [1]: https://github.com/apache/arrow-rs/pull/9372
> >>
> >> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <emkornfield@gmail.com
> >
> >> wrote:
> >>
> >>> Hi Andrew,
> >>> I think there is a fair amount of feedback on at least the
> >>> implementations, typically I think we've waited till they are close to
> >>> mergeable before a final vote. Otherwise I agree we are very close.
> >>>
> >>> -Micah
> >>>
> >>> On Saturday, April 25, 2026, Andrew Lamb <an...@gmail.com>
> wrote:
> >>>
> >>>> Thanks Prateek,
> >>>>
> >>>> I think from this content it looks to me like we are ready to start a
> >>>> vote to explicitly accept ALP into Parquet
> >>>>
> >>>> Does anyone know of a reason we should postpone it for longer?
> >>>> Perhaps someone needs some more time to review?
> >>>>
> >>>> Andrew
> >>>>
> >>>>
> >>>>
> >>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <pr...@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> Hi team,
> >>>>>
> >>>>>
> >>>>>
> >>>>> Hope everyone is doing well. I got a chance to work through all the
> >>>>> remaining feedback and update the spec doc. Here are the new
> artifacts
> >>>>>
> >>>>> 1) Spec document :
> >>>>>
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
> >>>>>
> >>>>> 2) Spec document in parquet format repo :
> >>>>> https://github.com/apache/parquet-format/pull/557
> >>>>>
> >>>>> 3) Alp implementation in arrow c++ repo :
> >>>>> https://github.com/apache/arrow/pull/48345/changes
> >>>>>
> >>>>> 4) Alp implementation in parquet-java repo : Work for Vinoo and
> Julien
> >>>>> https://github.com/apache/parquet-java/pull/3397
> >>>>>
> >>>>> 5) PR with test and benchmarking artifacts in parquet-testing repo :
> >>>>> https://github.com/apache/parquet-testing/pull/100
> >>>>>
> >>>>>
> >>>>> And
> >>>>>
> >>>>>
> >>>>> - Go : Arnav just submitted an in progress implementation in Go.
> >>>>> https://github.com/apache/arrow-go/pull/704 (I haven't started
> >>>>> looking at it yet)
> >>>>> - Rust : I remember Andrew mentioned that this work is also in
> >>>>> progress (So 4 languages!)
> >>>>>
> >>>>>
> >>>>> *Arrow C++ implementation *
> >>>>>
> >>>>>
> >>>>>
> >>>>> The PR is out and was also used by Antoine to report the numbers as
> >>>>> reported here. Micah and Konstantin have given 1 round of feedback
> >>>>> and I'm addressing them today. Please note that the default
> >>>>> optimization flag for compiling is O2 and not Q3. I got around 70%
> >>>>> performance improvement in the decoding speed when using the O3 flag.
> >>>>>
> >>>>>
> >>>>>
> >>>>> *Parqet-MR Java implementation (working with Vinoo and Julien) and
> **Cross
> >>>>> Language testing*
> >>>>>
> >>>>>
> >>>>> Let me know if you have any questions or feedback.
> >>>>>
> >>>>>
> >>>>>
> >>>>> Now pasting some performance numbers
> >>>>>
> >>>>>
> >>>>> Table 1: C++ ALP Double Decode — Spotify Columns (Graviton 3, ARM
> >>>>> Neoverse V1)
> >>>>>
> >>>>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
> >>>>>
> >>>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │ Speedup │
> >>>>>
> >>>>> ├──────────────────┼──────────────┼──────────────┼─────────┤
> >>>>>
> >>>>> │ valence │ 3,155 │ 5,523 │ 1.75x │
> >>>>>
> >>>>> │ danceability │ 3,233 │ 5,685 │ 1.76x │
> >>>>>
> >>>>> │ energy │ 3,197 │ 5,652 │ 1.77x │
> >>>>>
> >>>>> │ loudness │ 3,186 │ 5,473 │ 1.72x │
> >>>>>
> >>>>> └──────────────────┴──────────────┴──────────────┴─────────┘
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <pr...@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>>> @Micah Kornfield <em...@gmail.com> : Got it.
> >>>>>>
> >>>>>> @Andrew Lamb <an...@gmail.com>
> >>>>>>
> >>>>>>
> >>>>>>> Do you think it would be good to start moving the spec development
> >>>>>>> into
> >>>>>>> markdown format, in preparation for finalizing it?
> >>>>>>>
> >>>>>>
> >>>>>> Yes I'll update the numbers for some of the examples I have in the
> >>>>>> spec based
> >>>>>> on the updated header size. Then we should be good to go for the
> >>>>>> markdown format.
> >>>>>>
> >>>>>> Thanks everyone!
> >>>>>>
> >>>>>>
> >>>>>>>
> >>>>>>> Andrew
> >>>>>>>
> >>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <pr...@gmail.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>> > Hi team,
> >>>>>>> >
> >>>>>>> > 1) Andrew
> >>>>>>> >
> >>>>>>> > - Thanks for working on test files. My PR did add all the test
> >>>>>>> files I
> >>>>>>> > used to benchmark on datasets. Maybe we can club it together.
> >>>>>>> WIll also
> >>>>>>> > aid
> >>>>>>> > cross language testing
> >>>>>>> > - Kosta Tarasov working on Rust implementation. This is
> great.
> >>>>>>> Thanks
> >>>>>>> >
> >>>>>>> >
> >>>>>>> > 2) Antoine
> >>>>>>> >
> >>>>>>> > - Thanks a lot for reporting the numbers on AMD. Looks like
> you
> >>>>>>> are
> >>>>>>> > getting 8X the decoding performance of BSS. This is amazing!!.
> >>>>>>> > - Thanks for acknowledging the sampling design.
> >>>>>>> > - I agree with you on Fastlanes. In some crude experiments I
> >>>>>>> didn't get
> >>>>>>> > a good perf benefit from it on Graviton3 (but maybe there was
> >>>>>>> something
> >>>>>>> > wrong with my implementation).
> >>>>>>> > - Locking the 16bit exception encoding for the spec in this
> >>>>>>> case.
> >>>>>>> > - Awesome I think we have solved for all open questions minus
> >>>>>>> the
> >>>>>>> > version byte :). (will get back on this soon)
> >>>>>>> >
> >>>>>>> >
> >>>>>>> > 3) Micah
> >>>>>>> >
> >>>>>>> > - FastLanes : The current spec does allow for using FastLane
> >>>>>>> with the
> >>>>>>> > configurable enum value for layout. We should be able to
> inject
> >>>>>>> any
> >>>>>>> > layout
> >>>>>>> > in the current design.
> >>>>>>> >
> >>>>>>> >
> >>>>>>> > Working on resolving all remaining open comments on the spec this
> >>>>>>> week.
> >>>>>>> >
> >>>>>>> > Best
> >>>>>>> > Prateek
> >>>>>>> >
> >>>>>>> >
> >>>>>>> > On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <
> >>>>>>> stevel@cloudera.com>
> >>>>>>> > wrote:
> >>>>>>> >
> >>>>>>> > > On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
> >>>>>>> emkornfield@gmail.com>
> >>>>>>> > > wrote:
> >>>>>>> > >
> >>>>>>> > > >
> >>>>>>> > > >
> >>>>>>> > > > It looks like the actual issue described for ORC in the paper
> >>>>>>> is that
> >>>>>>> > it
> >>>>>>> > > > has multiple sub-encodings in a batch. This is different
> then
> >>>>>>> the
> >>>>>>> > design
> >>>>>>> > > > proposed here where there is still fixed encoding per page in
> >>>>>>> parquet.
> >>>>>>> > > > Given reasonably sized pages I don't think branch
> >>>>>>> misprediction should
> >>>>>>> > > be a
> >>>>>>> > > > big issue for new encodings. I agree that we should be
> >>>>>>> conservative in
> >>>>>>> > > > general for adding new encodings.
> >>>>>>> > > >
> >>>>>>> > > >
> >>>>>>> > > +1
> >>>>>>> > >
> >>>>>>> >
> >>>>>>>
> >>>>>>
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by PRATEEK GAUR <pr...@gmail.com>.
Hi team,
Quick update re: the C++ implementation.
Thanks to all the reviewers, we merged the C++ implementation today.
There are a few follow-up items that I'll address in a subsequent PR.
Thanks
Prateek
On Sun, Sep 13, 2026 at 6:54 PM Vinoo Ganesh <vi...@gmail.com> wrote:
> Quick update re: the Java impl. Thanks for the feedback, on
> https://github.com/apache/parquet-java/pull/3397, Russel! I pulled
> https://github.com/apache/parquet-java/pull/3791 into its own PR (thanks
> for the quick approval too, Gang). Assuming no objections, I think we can
> merge #3791, and then go ahead of #3397.
>
> Thanks,
> Vinoo Ganesh | vinoo.ganesh@gmail.com
>
> <Vi...@gmail.com>
>
>
> On Tue, Sep 8, 2026 at 10:54 AM PRATEEK GAUR <pr...@gmail.com> wrote:
>
> > [Rust] Awesome thanks Andrew, Kostas and Devin.
> > [Java] Thanks Gang for the feedback on Java PR.
> > [C++] Thanks Gang for another round of feedback on the C++ one.
> > [Go] Arnav and I will be working on the golang impl this week.
> >
> > This is great progress all around.
> >
> > On Sat, Sep 5, 2026 at 1:26 PM Vinoo Ganesh <vi...@gmail.com>
> > wrote:
> >
> > > Thanks all - I'm addressing Gang's comments on
> > > https://github.com/apache/parquet-java/pull/3397 now, so hopefully
> well
> > > also be in a good state for merging the java implementation soon.
> > >
> > > Thanks,
> > > Vinoo Ganesh | vinoo.ganesh@gmail.com
> > >
> > > <Vi...@gmail.com>
> > >
> > >
> > > On Sat, Sep 5, 2026 at 8:56 AM Andrew Lamb <an...@gmail.com>
> > wrote:
> > >
> > > > We just merged the Rust implementation[1] from Kosta and Devan. With
> > the
> > > > imminent release of Parquet Format 2.14.0 I think we'll be well on
> our
> > > way.
> > > >
> > > > Thanks again for pushing this along,
> > > > Andrew
> > > >
> > > > [1]: https://github.com/apache/arrow-rs/pull/9372
> > > >
> > > > On Fri, Sep 4, 2026 at 5:42 PM PRATEEK GAUR <pr...@gmail.com>
> > wrote:
> > > >
> > > > > Hi team,
> > > > >
> > > > > Thanks Micah for multiple rounds of review and thanks Gang for
> > taking a
> > > > > look at the PR and giving feedback. I've addressed all comments in
> > the
> > > > PR.
> > > > > Looking forward to the next round of feedback (or approval). Would
> > like
> > > > to
> > > > > get it in soon.
> > > > >
> > > > > https://github.com/apache/arrow/pull/48345/changes
> > > > >
> > > > > With this done and after talking to Arnav, I plan to spend some
> time
> > on
> > > > the
> > > > > ALP implementation in the arrow-go repository helping spread the
> > > support
> > > > to
> > > > > more languages.
> > > > >
> > > > > Best
> > > > > Prateek
> > > > >
> > > > > On Sun, Aug 2, 2026 at 10:42 PM PRATEEK GAUR <pr...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Thanks Divjot for bringing it up.
> > > > > >
> > > > > > I was under the same impression as Russell and Andrew that we'd
> > have
> > > it
> > > > > > flag gated.
> > > > > > With the new versioning proposal 'preview' might be a good tag
> for
> > > it.
> > > > > >
> > > > > > Best
> > > > > > Prateek
> > > > > >
> > > > > > On Sat, Aug 1, 2026 at 12:39 AM Andrew Lamb <
> > andrewlamb11@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > >> Div: Here is a proposal to mark ALP in Preview: [1] (using a
> > comment
> > > > in
> > > > > >> the
> > > > > >> spec section).
> > > > > >>
> > > > > >> Russel: That is my understanding as well.
> > > > > >>
> > > > > >> Hopefully we are able to formalize the versioning process
> shortly
> > so
> > > > we
> > > > > >> can
> > > > > >> formalize the notion of "preview"
> > > > > >>
> > > > > >> Andrew
> > > > > >>
> > > > > >>
> > > > > >> [1]: https://github.com/apache/parquet-format/pull/604
> > > > > >>
> > > > > >> On Fri, Jul 31, 2026 at 11:15 AM Russell Spitzer <
> > > > > >> russell.spitzer@gmail.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > I thought we had agreed it would be merged, and features like
> > this
> > > > > would
> > > > > >> > be feature flag gated. So a V-Old writer could always opt in
> to
> > > > using
> > > > > >> this
> > > > > >> > encoding but that must be a conscious effort if you are not
> > using
> > > > > VNext
> > > > > >> of
> > > > > >> > Parquet. If you are on V-Next the writer can choose to use ALP
> > > > > whenever
> > > > > >> it
> > > > > >> > likes without user intervention.
> > > > > >> >
> > > > > >> > On Fri, Jul 31, 2026 at 2:20 AM Divjot Arora via dev <
> > > > > >> > dev@parquet.apache.org> wrote:
> > > > > >> >
> > > > > >> >> Yes, that’s my understanding as well. However, as you noted,
> > the
> > > > > >> >> versioning
> > > > > >> >> discussion is still in early stages and hasn’t prescribed
> > > concrete
> > > > > >> >> mechanisms yet. If ALP goes into parquet-format 2.14.0,
> there’s
> > > no
> > > > > >> >> indication that it’s in preview or that writers shouldn’t use
> > it
> > > by
> > > > > >> >> default
> > > > > >> >> yet. Given that we’ve held off on merging other incompatible
> > > > changes
> > > > > >> (e.g
> > > > > >> >> removal of path_in_schema) until the versioning discussion
> > > closes,
> > > > my
> > > > > >> >> preference would be to leave ALP un-merged for the time being
> > as
> > > > > well.
> > > > > >> >>
> > > > > >> >> — Div
> > > > > >> >>
> > > > > >> >> On Fri, Jul 31, 2026 at 08:59 Andrew Lamb <
> > > andrewlamb11@gmail.com>
> > > > > >> wrote:
> > > > > >> >>
> > > > > >> >> > After re-reading Julien's Versioning Proposal, my
> > understanding
> > > > is
> > > > > >> that
> > > > > >> >> if
> > > > > >> >> > we adopt that system, we will somehow mark ALP in "Preview"
> > in
> > > > > >> >> > parquet-format for release Version 3.
> > > > > >> >> >
> > > > > >> >> > The versioning document doesn't seem to proscribe the
> > > particular
> > > > > >> method
> > > > > >> >> > for annotating "preview" features -- I think clearly
> marking
> > > them
> > > > > as
> > > > > >> >> such
> > > > > >> >> > in parquet.thrift as well as reflecting that on the
> > > > > >> >> > https://parquet.apache.org/docs/file-format/versions/ page
> > > > somehow
> > > > > >> >> would
> > > > > >> >> > be adequate.
> > > > > >> >> >
> > > > > >> >> > Andrew
> > > > > >> >> >
> > > > > >> >> > On Thu, Jul 30, 2026 at 11:50 AM Andrew Lamb <
> > > > > andrewlamb11@gmail.com
> > > > > >> >
> > > > > >> >> > wrote:
> > > > > >> >> >
> > > > > >> >> >> In my mind ALP is in the same position as other encodings
> > that
> > > > > were
> > > > > >> >> added
> > > > > >> >> >> recently[1] (such as BYTE_STREAM_SPLIT for non floating
> > point
> > > > > types)
> > > > > >> >> >>
> > > > > >> >> >> That is to say it will be available in the latest open
> > source
> > > > > >> writers
> > > > > >> >> and
> > > > > >> >> >> readers, but realistically not widely used in the
> ecosystem
> > > > where
> > > > > >> >> >> compatibility is a concern (a large part)
> > > > > >> >> >>
> > > > > >> >> >> I view Julian's work[2][3] to gather consensus on how to
> > name
> > > > > >> >> >> combinations of features (e.g. "versions") as the
> mechanism
> > by
> > > > > >> which we
> > > > > >> >> >> will accelerate the adoption of this change (along with
> > other
> > > > > >> forward
> > > > > >> >> >> incompatible changes) across the ecosystem.
> > > > > >> >> >>
> > > > > >> >> >> > Will we exclude this change when cutting parquet-format
> > > > v2.14.0?
> > > > > >> >> >>
> > > > > >> >> >> No I don't think so -- I would expect it to go out in the
> > next
> > > > > >> >> >> parquet-format version as do other changes in the spec.
> > > > > >> >> >>
> > > > > >> >> >> Andrew
> > > > > >> >> >>
> > > > > >> >> >> [1]:
> https://parquet.apache.org/docs/file-format/versions/
> > > > > >> >> >> [2]:
> > > > > >>
> https://lists.apache.org/thread/xmnj8h0h8ozmrgox5tydhhs7yz7q2hss
> > > > > >> >> >> [3]:
> > > > > >>
> https://lists.apache.org/thread/qlf8lg90gqq41lllyy9mk2f0skqvftqj
> > > > > >> >> >>
> > > > > >> >> >>
> > > > > >> >> >> On Thu, Jul 30, 2026 at 3:51 AM Divjot Arora via dev <
> > > > > >> >> >> dev@parquet.apache.org> wrote:
> > > > > >> >> >>
> > > > > >> >> >>> Hi Andrew,
> > > > > >> >> >>>
> > > > > >> >> >>> How does this interact with the ongoing versioning
> > > discussion?
> > > > > >> Given
> > > > > >> >> that
> > > > > >> >> >>> new encodings are not forward compatible, I would assume
> > ALP
> > > is
> > > > > >> >> included
> > > > > >> >> >>> in
> > > > > >> >> >>> Parquet V<next> (V3?), but it's not clear to me how that
> > > works
> > > > if
> > > > > >> it's
> > > > > >> >> >>> merged to the parquet-format master branch now. Will we
> > > exclude
> > > > > >> this
> > > > > >> >> >>> change
> > > > > >> >> >>> when cutting parquet-format v2.14.0?
> > > > > >> >> >>>
> > > > > >> >> >>> -- Div
> > > > > >> >> >>>
> > > > > >> >> >>> On Thu, Jul 30, 2026 at 8:51 AM Andrew Lamb <
> > > > > >> andrewlamb11@gmail.com>
> > > > > >> >> >>> wrote:
> > > > > >> >> >>>
> > > > > >> >> >>> > Hello,
> > > > > >> >> >>> >
> > > > > >> >> >>> > As an update, thanks to the epic work from many
> > > contributors,
> > > > > >> >> >>> especially
> > > > > >> >> >>> > Prateek, I just merged the parquet-format change for
> > > ALP[1].
> > > > We
> > > > > >> >> plan to
> > > > > >> >> >>> > make additional editorial The remaining items I know of
> > are
> > > > > >> merging
> > > > > >> >> the
> > > > > >> >> >>> > examples from parquet-testing[2], and then of course
> the
> > > > > various
> > > > > >> >> >>> > implementations
> > > > > >> >> >>> >
> > > > > >> >> >>> > Thanks again everyone. This is a great step forward
> > > > > >> >> >>> > Andrew
> > > > > >> >> >>> >
> > > > > >> >> >>> > [1]: https://github.com/apache/parquet-format/pull/557
> > > > > >> >> >>> > [2]:
> https://github.com/apache/parquet-testing/pull/100
> > > > > >> >> >>> >
> > > > > >> >> >>> > On Tue, Jul 21, 2026 at 4:39 PM Julien Le Dem <
> > > > > julien@apache.org
> > > > > >> >
> > > > > >> >> >>> wrote:
> > > > > >> >> >>> >
> > > > > >> >> >>> > > It was also in my spam folder... :(
> > > > > >> >> >>> > > I've seen a bunch of apache list emails go to my spam
> > > > > recently.
> > > > > >> >> >>> > >
> > > > > >> >> >>> > > On Tue, Jul 21, 2026 at 10:58 AM Russell Spitzer <
> > > > > >> >> >>> > > russell.spitzer@gmail.com>
> > > > > >> >> >>> > > wrote:
> > > > > >> >> >>> > >
> > > > > >> >> >>> > > > Yes, my hope was that my vote would kick it out of
> > spam
> > > > but
> > > > > >> we
> > > > > >> >> >>> > definitely
> > > > > >> >> >>> > > > had two other votes that also went to spam for me.
> > > > > >> >> >>> > > >
> > > > > >> >> >>> > > > For anyone looking the thread is here
> > > > > >> >> >>> > > >
> > > > > >> >>
> > https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt
> > > > > >> >> >>> > > >
> > > > > >> >> >>> > > > On Tue, Jul 21, 2026 at 10:57 AM Micah Kornfield <
> > > > > >> >> >>> > emkornfield@gmail.com>
> > > > > >> >> >>> > > > wrote:
> > > > > >> >> >>> > > >
> > > > > >> >> >>> > > > > I've seen other's votes not go to spam, so
> > hopefully
> > > > > people
> > > > > >> >> will
> > > > > >> >> >>> see
> > > > > >> >> >>> > it
> > > > > >> >> >>> > > > in
> > > > > >> >> >>> > > > > there inboxes (or see this thread and look in
> there
> > > > > spam).
> > > > > >> >> >>> > > > >
> > > > > >> >> >>> > > > > On Tue, Jul 21, 2026 at 10:52 AM PRATEEK GAUR <
> > > > > >> >> >>> prtkgaur@gmail.com>
> > > > > >> >> >>> > > > wrote:
> > > > > >> >> >>> > > > >
> > > > > >> >> >>> > > > > > Okay,
> > > > > >> >> >>> > > > > >
> > > > > >> >> >>> > > > > > I sent out the voting email yesterday and so
> far
> > > I've
> > > > > >> heard
> > > > > >> >> >>> from 3
> > > > > >> >> >>> > > > people
> > > > > >> >> >>> > > > > > that it ended up in their "spam folder". Dev
> VPN
> > to
> > > > be
> > > > > >> >> blamed
> > > > > >> >> >>> for
> > > > > >> >> >>> > > it?.
> > > > > >> >> >>> > > > > >
> > > > > >> >> >>> > > > > > What's the protocol here?
> > > > > >> >> >>> > > > > > I let people hunt for the email in their spam
> > > folder
> > > > or
> > > > > >> send
> > > > > >> >> >>> out
> > > > > >> >> >>> > > > another
> > > > > >> >> >>> > > > > > one?
> > > > > >> >> >>> > > > > >
> > > > > >> >> >>> > > > > > Best
> > > > > >> >> >>> > > > > > Prateek
> > > > > >> >> >>> > > > > >
> > > > > >> >> >>> > > > > > On Sun, Jul 19, 2026 at 10:47 PM PRATEEK GAUR <
> > > > > >> >> >>> prtkgaur@gmail.com>
> > > > > >> >> >>> > > > > wrote:
> > > > > >> >> >>> > > > > >
> > > > > >> >> >>> > > > > > > Hi team,
> > > > > >> >> >>> > > > > > >
> > > > > >> >> >>> > > > > > > Vinoo has added the tests which were
> requested
> > in
> > > > the
> > > > > >> last
> > > > > >> >> >>> round
> > > > > >> >> >>> > of
> > > > > >> >> >>> > > > > > review.
> > > > > >> >> >>> > > > > > > In his own words.
> > > > > >> >> >>> > > > > > >
> > > > > >> >> >>> > > > > > > "
> > > > > >> >> >>> > > > > > > The extreme value tests Micah wanted are done
> > and
> > > > > >> pushed.
> > > > > >> >> I
> > > > > >> >> >>> added
> > > > > >> >> >>> > > > > > coverage
> > > > > >> >> >>> > > > > > > for values that need the full FOR bit width
> > after
> > > > the
> > > > > >> >> frame
> > > > > >> >> >>> of
> > > > > >> >> >>> > > > > reference
> > > > > >> >> >>> > > > > > is
> > > > > >> >> >>> > > > > > > applied: 64-bit (the case where the signed
> max
> > > > minus
> > > > > >> min
> > > > > >> >> >>> > > subtraction
> > > > > >> >> >>> > > > > > > overflows), 63-bit (the non-overflow case),
> and
> > > > > 32-bit
> > > > > >> for
> > > > > >> >> >>> > floats.
> > > > > >> >> >>> > > > All
> > > > > >> >> >>> > > > > > > lossless with zero exceptions. I also
> verified
> > > them
> > > > > >> >> through
> > > > > >> >> >>> Arrow
> > > > > >> >> >>> > > C++
> > > > > >> >> >>> > > > > ALP
> > > > > >> >> >>> > > > > > > reader (the local build), so the Java-written
> > > > extreme
> > > > > >> >> columns
> > > > > >> >> >>> > > decode
> > > > > >> >> >>> > > > > > > correctly in C++, bit-exact against the
> > expected
> > > > > >> values."
> > > > > >> >> >>> > > > > > >
> > > > > >> >> >>> > > > > > > State of the PR (#3397):
> > > > > >> >> >>> > > > > > >
> > > > > >> >> >>> > > > > > > - Merges cleanly with master (I pulled
> > master
> > > in
> > > > > and
> > > > > >> >> >>> resolved
> > > > > >> >> >>> > > the
> > > > > >> >> >>> > > > > > > conflicts).
> > > > > >> >> >>> > > > > > > - 41 of 44 review threads resolved. The 3
> > > > > remaining
> > > > > >> are
> > > > > >> >> >>> > > > intentional:
> > > > > >> >> >>> > > > > > > one deferred perf nit, one that stays open
> > > until
> > > > > >> >> >>> > parquet-format
> > > > > >> >> >>> > > > > ships
> > > > > >> >> >>> > > > > > ALP,
> > > > > >> >> >>> > > > > > >
> > > > > >> >> >>> > > > > > > "
> > > > > >> >> >>> > > > > > >
> > > > > >> >> >>> > > > > > > With this both the
> > > > > >> >> >>> > > > > > > 1) C++ PR :
> > > > > >> >> >>> https://github.com/apache/arrow/pull/48345/changes
> > > > > >> >> >>> > > > > > > 2) Java PR :
> > > > > >> >> >>> https://github.com/apache/parquet-java/pull/3397
> > > > > >> >> >>> > > > > > >
> > > > > >> >> >>> > > > > > > Are cross language tested and range of test
> > > > > scenarios.
> > > > > >> >> >>> > > > > > >
> > > > > >> >> >>> > > > > > > Best
> > > > > >> >> >>> > > > > > > Prateek
> > > > > >> >> >>> > > > > > >
> > > > > >> >> >>> > > > > > >
> > > > > >> >> >>> > > > > > > On Mon, Jul 13, 2026 at 8:38 AM PRATEEK GAUR
> <
> > > > > >> >> >>> prtkgaur@gmail.com
> > > > > >> >> >>> > >
> > > > > >> >> >>> > > > > wrote:
> > > > > >> >> >>> > > > > > >
> > > > > >> >> >>> > > > > > >> Thanks Matt,
> > > > > >> >> >>> > > > > > >>
> > > > > >> >> >>> > > > > > >> I'll be on it this week.
> > > > > >> >> >>> > > > > > >>
> > > > > >> >> >>> > > > > > >> Best
> > > > > >> >> >>> > > > > > >> Prateek
> > > > > >> >> >>> > > > > > >>
> > > > > >> >> >>> > > > > > >> On Mon, Jul 13, 2026 at 8:36 AM Matt Topol <
> > > > > >> >> >>> > > zotthewizard@gmail.com>
> > > > > >> >> >>> > > > > > >> wrote:
> > > > > >> >> >>> > > > > > >>
> > > > > >> >> >>> > > > > > >>> There is also the Go implementation
> submitted
> > > by
> > > > > >> Arnav (
> > > > > >> >> >>> > > > > > >>>
> https://github.com/apache/arrow-go/pull/704)
> > > > which
> > > > > >> is
> > > > > >> >> >>> waiting
> > > > > >> >> >>> > > for
> > > > > >> >> >>> > > > > > >>> updates.
> > > > > >> >> >>> > > > > > >>>
> > > > > >> >> >>> > > > > > >>> Just wanted to make sure it didn't get lost
> > > here.
> > > > > >> >> >>> > > > > > >>>
> > > > > >> >> >>> > > > > > >>> --Matt
> > > > > >> >> >>> > > > > > >>>
> > > > > >> >> >>> > > > > > >>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK
> > GAUR <
> > > > > >> >> >>> > > prtkgaur@gmail.com>
> > > > > >> >> >>> > > > > > >>> wrote:
> > > > > >> >> >>> > > > > > >>>
> > > > > >> >> >>> > > > > > >>> > Hi all,
> > > > > >> >> >>> > > > > > >>> >
> > > > > >> >> >>> > > > > > >>> > I'd like to share a status update on the
> > ALP
> > > > > >> encoding
> > > > > >> >> >>> effort
> > > > > >> >> >>> > > and
> > > > > >> >> >>> > > > > get
> > > > > >> >> >>> > > > > > a
> > > > > >> >> >>> > > > > > >>> feel
> > > > > >> >> >>> > > > > > >>> > from the community before starting a
> formal
> > > > vote.
> > > > > >> >> >>> > > > > > >>> >
> > > > > >> >> >>> > > > > > >>> > We now have the specification plus
> > > > > implementations
> > > > > >> in
> > > > > >> >> >>> both
> > > > > >> >> >>> > > > > languages:
> > > > > >> >> >>> > > > > > >>> >
> > > > > >> >> >>> > > > > > >>> > - Spec:
> > > apache/parquet-format#557 —
> > > > > >> GH-533
> > > > > >> >> >>> Add
> > > > > >> >> >>> > ALP
> > > > > >> >> >>> > > > > > encoding
> > > > > >> >> >>> > > > > > >>> > specification
> > > > > >> >> >>> > > > > > >>> > - C++ (Arrow): apache/arrow#48345 —
> > > > GH-48701
> > > > > >> >> >>> > > [C++][Parquet]
> > > > > >> >> >>> > > > > Add
> > > > > >> >> >>> > > > > > >>> ALPpd
> > > > > >> >> >>> > > > > > >>> > encoding
> > > > > >> >> >>> > > > > > >>> > - Java:
> > apache/parquet-java#3397
> > > —
> > > > > >> Parquet
> > > > > >> >> >>> Java
> > > > > >> >> >>> > ALP
> > > > > >> >> >>> > > > > > >>> > Implementation (by Vinoo)
> > > > > >> >> >>> > > > > > >>> >
> > > > > >> >> >>> > > > > > >>> > Current state:
> > > > > >> >> >>> > > > > > >>> >
> > > > > >> >> >>> > > > > > >>> > - The C++ implementation has been
> > through
> > > > 3-4
> > > > > >> >> rounds
> > > > > >> >> >>> of
> > > > > >> >> >>> > > > review.
> > > > > >> >> >>> > > > > > >>> > - The Java implementation has been
> > > tracking
> > > > > >> well
> > > > > >> >> and
> > > > > >> >> >>> has
> > > > > >> >> >>> > > had
> > > > > >> >> >>> > > > > > >>> initial
> > > > > >> >> >>> > > > > > >>> > reviews from contributors, with no
> > > major
> > > > > >> >> remaining
> > > > > >> >> >>> > > issues.
> > > > > >> >> >>> > > > > > >>> > - Cross-language compatibility tests
> > are
> > > > > >> passing:
> > > > > >> >> the
> > > > > >> >> >>> > Arrow
> > > > > >> >> >>> > > > C++
> > > > > >> >> >>> > > > > > >>> decoder
> > > > > >> >> >>> > > > > > >>> > reads Java-written data bit-exactly
> > > > across
> > > > > >> >> ~1.56M
> > > > > >> >> >>> > values
> > > > > >> >> >>> > > > and
> > > > > >> >> >>> > > > > 18
> > > > > >> >> >>> > > > > > >>> > fixtures, covering V1 and V2 pages,
> > > > > multiple
> > > > > >> >> vector
> > > > > >> >> >>> > > sizes,
> > > > > >> >> >>> > > > > and
> > > > > >> >> >>> > > > > > >>> > several
> > > > > >> >> >>> > > > > > >>> > real datasets — zero mismatches.
> > > > > >> >> >>> > > > > > >>> >
> > > > > >> >> >>> > > > > > >>> > As raised in the java-pr we will be
> adding
> > > > > >> coverage
> > > > > >> >> for
> > > > > >> >> >>> > > extreme
> > > > > >> >> >>> > > > > > values
> > > > > >> >> >>> > > > > > >>> > (those requiring
> > > > > >> >> >>> > > > > > >>> > 63-64 bits after FOR is applied) before
> we
> > > > > close a
> > > > > >> >> vote.
> > > > > >> >> >>> > We'll
> > > > > >> >> >>> > > > aim
> > > > > >> >> >>> > > > > > to
> > > > > >> >> >>> > > > > > >>> get
> > > > > >> >> >>> > > > > > >>> > that
> > > > > >> >> >>> > > > > > >>> > done in parallel.
> > > > > >> >> >>> > > > > > >>> >
> > > > > >> >> >>> > > > > > >>> > Plan: unless there are objections, I
> intend
> > > to
> > > > > >> start a
> > > > > >> >> >>> vote
> > > > > >> >> >>> > at
> > > > > >> >> >>> > > > the
> > > > > >> >> >>> > > > > > end
> > > > > >> >> >>> > > > > > >>> of
> > > > > >> >> >>> > > > > > >>> > this week/early
> > > > > >> >> >>> > > > > > >>> > next week.
> > > > > >> >> >>> > > > > > >>> >
> > > > > >> >> >>> > > > > > >>> > Thanks
> > > > > >> >> >>> > > > > > >>> > Prateek
> > > > > >> >> >>> > > > > > >>> >
> > > > > >> >> >>> > > > > > >>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK
> > GAUR <
> > > > > >> >> >>> > > prtkgaur@gmail.com>
> > > > > >> >> >>> > > > > > >>> wrote:
> > > > > >> >> >>> > > > > > >>> >
> > > > > >> >> >>> > > > > > >>> > > Hi Team,
> > > > > >> >> >>> > > > > > >>> > >
> > > > > >> >> >>> > > > > > >>> > > Just wanted to provide some updates on
> > ALP.
> > > > > >> >> >>> > > > > > >>> > > Micah and I have done 3-4 rounds of
> > review
> > > > for
> > > > > >> the
> > > > > >> >> c++
> > > > > >> >> >>> PR :
> > > > > >> >> >>> > > > > > >>> > >
> > https://github.com/apache/arrow/pull/48345
> > > > > >> >> >>> > > > > > >>> > >
> > > > > >> >> >>> > > > > > >>> > > For the Java implementation Vinoo has
> > been
> > > > > >> working
> > > > > >> >> on
> > > > > >> >> >>> > > following
> > > > > >> >> >>> > > > > PR
> > > > > >> >> >>> > > > > > :
> > > > > >> >> >>> > > > > > >>> > >
> > > > > https://github.com/apache/parquet-java/pull/3397
> > > > > >> >> >>> > > > > > >>> > >
> > > > > >> >> >>> > > > > > >>> > > Best
> > > > > >> >> >>> > > > > > >>> > > Prateek
> > > > > >> >> >>> > > > > > >>> > >
> > > > > >> >> >>> > > > > > >>> > > On Tue, May 5, 2026 at 1:50 PM Micah
> > > > Kornfield
> > > > > <
> > > > > >> >> >>> > > > > > >>> emkornfield@gmail.com>
> > > > > >> >> >>> > > > > > >>> > > wrote:
> > > > > >> >> >>> > > > > > >>> > >
> > > > > >> >> >>> > > > > > >>> > >> Hi Antoine,
> > > > > >> >> >>> > > > > > >>> > >>
> > > > > >> >> >>> > > > > > >>> > >> > Apologies if the question was
> already
> > > > asked,
> > > > > >> but
> > > > > >> >> >>> should
> > > > > >> >> >>> > we
> > > > > >> >> >>> > > > > care
> > > > > >> >> >>> > > > > > >>> about
> > > > > >> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be
> > more
> > > > > >> >> efficient
> > > > > >> >> >>> than
> > > > > >> >> >>> > > > > FLOAT16
> > > > > >> >> >>> > > > > > +
> > > > > >> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> > > > > >> >> >>> > > > > > >>> > >>
> > > > > >> >> >>> > > > > > >>> > >> It was. We thought we could defer it
> > for
> > > > the
> > > > > >> >> >>> following
> > > > > >> >> >>> > > > reasons:
> > > > > >> >> >>> > > > > > >>> > >> 1. It's not clear there are a lot of
> > easy
> > > > > >> >> reference
> > > > > >> >> >>> > > datasets
> > > > > >> >> >>> > > > to
> > > > > >> >> >>> > > > > > >>> test
> > > > > >> >> >>> > > > > > >>> > its
> > > > > >> >> >>> > > > > > >>> > >> effectiveness. It does look like
> there
> > > > might
> > > > > be
> > > > > >> >> one
> > > > > >> >> >>> or
> > > > > >> >> >>> > two
> > > > > >> >> >>> > > on
> > > > > >> >> >>> > > > > > >>> > huggingface
> > > > > >> >> >>> > > > > > >>> > >> (e.g.
> > > > > >> >> https://huggingface.co/datasets/kikitora/curdie
> > > > > >> >> >>> ).
> > > > > >> >> >>> > > > > > >>> > >> 2. It seemed likely that float 16 was
> > > more
> > > > > >> likely
> > > > > >> >> >>> used
> > > > > >> >> >>> > for
> > > > > >> >> >>> > > > > values
> > > > > >> >> >>> > > > > > >>> that
> > > > > >> >> >>> > > > > > >>> > >> were less likely to reduce to decimal
> > > > values.
> > > > > >> >> >>> > > > > > >>> > >> 3. It could be added as an extension
> > > later
> > > > if
> > > > > >> >> needed.
> > > > > >> >> >>> > > > > > >>> > >>
> > > > > >> >> >>> > > > > > >>> > >> Cheers,
> > > > > >> >> >>> > > > > > >>> > >> Micah
> > > > > >> >> >>> > > > > > >>> > >>
> > > > > >> >> >>> > > > > > >>> > >>
> > > > > >> >> >>> > > > > > >>> > >> On Tue, May 5, 2026 at 1:40 PM Antoine
> > > > Pitrou
> > > > > <
> > > > > >> >> >>> > > > > antoine@python.org
> > > > > >> >> >>> > > > > > >
> > > > > >> >> >>> > > > > > >>> > wrote:
> > > > > >> >> >>> > > > > > >>> > >>
> > > > > >> >> >>> > > > > > >>> > >> >
> > > > > >> >> >>> > > > > > >>> > >> > Hello,
> > > > > >> >> >>> > > > > > >>> > >> >
> > > > > >> >> >>> > > > > > >>> > >> > Apologies if the question was
> already
> > > > asked,
> > > > > >> but
> > > > > >> >> >>> should
> > > > > >> >> >>> > we
> > > > > >> >> >>> > > > > care
> > > > > >> >> >>> > > > > > >>> about
> > > > > >> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be
> > more
> > > > > >> >> efficient
> > > > > >> >> >>> than
> > > > > >> >> >>> > > > > FLOAT16
> > > > > >> >> >>> > > > > > +
> > > > > >> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> > > > > >> >> >>> > > > > > >>> > >> >
> > > > > >> >> >>> > > > > > >>> > >> > Regards
> > > > > >> >> >>> > > > > > >>> > >> >
> > > > > >> >> >>> > > > > > >>> > >> > Antoine.
> > > > > >> >> >>> > > > > > >>> > >> >
> > > > > >> >> >>> > > > > > >>> > >> >
> > > > > >> >> >>> > > > > > >>> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR
> a
> > > > écrit
> > > > > :
> > > > > >> >> >>> > > > > > >>> > >> > > Thanks Andrew and Micah for review
> > > > > feedback
> > > > > >> on
> > > > > >> >> >>> the two
> > > > > >> >> >>> > > > PR's
> > > > > >> >> >>> > > > > > >>> > >> > > 1) (c++ arrow repo)
> > > > > >> >> >>> > > > > > >>> > >>
> > > > > >> https://github.com/apache/arrow/pull/48345/changes
> > > > > >> >> >>> > > > > > >>> > >> > > 2) (parquet-format repo)
> > > > > >> >> >>> > > > > > >>> > >> >
> > > > > >> >> https://github.com/apache/parquet-format/pull/557
> > > > > >> >> >>> > > > > > >>> > >> > >
> > > > > >> >> >>> > > > > > >>> > >> > > I have addressed all (unless I
> > missed
> > > > > >> >> something)
> > > > > >> >> >>> > > comments
> > > > > >> >> >>> > > > on
> > > > > >> >> >>> > > > > > >>> the two
> > > > > >> >> >>> > > > > > >>> > >> > PR's.
> > > > > >> >> >>> > > > > > >>> > >> > >
> > > > > >> >> >>> > > > > > >>> > >> > > Best
> > > > > >> >> >>> > > > > > >>> > >> > > Prateek
> > > > > >> >> >>> > > > > > >>> > >> > >
> > > > > >> >> >>> > > > > > >>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM
> > > PRATEEK
> > > > > >> GAUR <
> > > > > >> >> >>> > > > > > >>> prtkgaur@gmail.com>
> > > > > >> >> >>> > > > > > >>> > >> wrote:
> > > > > >> >> >>> > > > > > >>> > >> > >
> > > > > >> >> >>> > > > > > >>> > >> > >> Thanks Andrew and Micah.
> > > > > >> >> >>> > > > > > >>> > >> > >>
> > > > > >> >> >>> > > > > > >>> > >> > >> `fair amount of feedback on at
> > least
> > > > the
> > > > > >> >> >>> > > implementations`
> > > > > >> >> >>> > > > > > >>> > >> > >> For the c++ I have already
> started
> > > > > >> addressing
> > > > > >> >> the
> > > > > >> >> >>> > > > > feedback, I
> > > > > >> >> >>> > > > > > >>> > should
> > > > > >> >> >>> > > > > > >>> > >> be
> > > > > >> >> >>> > > > > > >>> > >> > >> done with that Monday/Tuesday.
> > > > > >> >> >>> > > > > > >>> > >> > >> I think Vinoo too has been making
> > > good
> > > > > >> >> progress
> > > > > >> >> >>> on
> > > > > >> >> >>> > the
> > > > > >> >> >>> > > > Java
> > > > > >> >> >>> > > > > > >>> > >> > implementation.
> > > > > >> >> >>> > > > > > >>> > >> > >>
> > > > > >> >> >>> > > > > > >>> > >> > >> Best
> > > > > >> >> >>> > > > > > >>> > >> > >> Prateek
> > > > > >> >> >>> > > > > > >>> > >> > >>
> > > > > >> >> >>> > > > > > >>> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM
> > > Andrew
> > > > > >> Lamb <
> > > > > >> >> >>> > > > > > >>> > andrewlamb11@gmail.com
> > > > > >> >> >>> > > > > > >>> > >> >
> > > > > >> >> >>> > > > > > >>> > >> > >> wrote:
> > > > > >> >> >>> > > > > > >>> > >> > >>
> > > > > >> >> >>> > > > > > >>> > >> > >>> Got it. Thank you for the
> > > > clarification
> > > > > >> -- I
> > > > > >> >> >>> will
> > > > > >> >> >>> > try
> > > > > >> >> >>> > > > and
> > > > > >> >> >>> > > > > > look
> > > > > >> >> >>> > > > > > >>> > into
> > > > > >> >> >>> > > > > > >>> > >> the
> > > > > >> >> >>> > > > > > >>> > >> > >>> spec and the Rust
> > implementation[1]
> > > in
> > > > > >> this
> > > > > >> >> next
> > > > > >> >> >>> > week
> > > > > >> >> >>> > > > > > >>> > >> > >>>
> > > > > >> >> >>> > > > > > >>> > >> > >>> [1]:
> > > > > >> >> >>> https://github.com/apache/arrow-rs/pull/9372
> > > > > >> >> >>> > > > > > >>> > >> > >>>
> > > > > >> >> >>> > > > > > >>> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM
> > > Micah
> > > > > >> >> >>> Kornfield <
> > > > > >> >> >>> > > > > > >>> > >> > emkornfield@gmail.com>
> > > > > >> >> >>> > > > > > >>> > >> > >>> wrote:
> > > > > >> >> >>> > > > > > >>> > >> > >>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>> Hi Andrew,
> > > > > >> >> >>> > > > > > >>> > >> > >>>> I think there is a fair amount
> of
> > > > > >> feedback
> > > > > >> >> on
> > > > > >> >> >>> at
> > > > > >> >> >>> > > least
> > > > > >> >> >>> > > > > the
> > > > > >> >> >>> > > > > > >>> > >> > >>>> implementations, typically I
> > think
> > > > > we've
> > > > > >> >> waited
> > > > > >> >> >>> > till
> > > > > >> >> >>> > > > they
> > > > > >> >> >>> > > > > > are
> > > > > >> >> >>> > > > > > >>> > >> close to
> > > > > >> >> >>> > > > > > >>> > >> > >>>> mergeable before a final vote.
> > > > > >> Otherwise I
> > > > > >> >> >>> agree
> > > > > >> >> >>> > we
> > > > > >> >> >>> > > > are
> > > > > >> >> >>> > > > > > very
> > > > > >> >> >>> > > > > > >>> > >> close.
> > > > > >> >> >>> > > > > > >>> > >> > >>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>> -Micah
> > > > > >> >> >>> > > > > > >>> > >> > >>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>> On Saturday, April 25, 2026,
> > Andrew
> > > > > Lamb
> > > > > >> <
> > > > > >> >> >>> > > > > > >>> andrewlamb11@gmail.com
> > > > > >> >> >>> > > > > > >>> > >
> > > > > >> >> >>> > > > > > >>> > >> > wrote:
> > > > > >> >> >>> > > > > > >>> > >> > >>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>> Thanks Prateek,
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>> I think from this content it
> > looks
> > > > to
> > > > > me
> > > > > >> >> like
> > > > > >> >> >>> we
> > > > > >> >> >>> > are
> > > > > >> >> >>> > > > > ready
> > > > > >> >> >>> > > > > > >>> to
> > > > > >> >> >>> > > > > > >>> > >> start a
> > > > > >> >> >>> > > > > > >>> > >> > >>>>> vote to explicitly accept ALP
> > into
> > > > > >> Parquet
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>> Does anyone know of a reason
> we
> > > > should
> > > > > >> >> >>> postpone it
> > > > > >> >> >>> > > for
> > > > > >> >> >>> > > > > > >>> longer?
> > > > > >> >> >>> > > > > > >>> > >> > >>>>> Perhaps someone needs some
> more
> > > time
> > > > > to
> > > > > >> >> >>> review?
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>> Andrew
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>> On Wed, Apr 22, 2026 at
> 1:00 PM
> > > > > PRATEEK
> > > > > >> >> GAUR <
> > > > > >> >> >>> > > > > > >>> > prtkgaur@gmail.com>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>> wrote:
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Hi team,
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Hope everyone is doing well.
> I
> > > got
> > > > a
> > > > > >> >> chance
> > > > > >> >> >>> to
> > > > > >> >> >>> > work
> > > > > >> >> >>> > > > > > >>> through all
> > > > > >> >> >>> > > > > > >>> > >> the
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> remaining feedback and update
> > the
> > > > > spec
> > > > > >> >> doc.
> > > > > >> >> >>> Here
> > > > > >> >> >>> > > are
> > > > > >> >> >>> > > > > the
> > > > > >> >> >>> > > > > > >>> new
> > > > > >> >> >>> > > > > > >>> > >> > artifacts
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 1) Spec document :
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> >
> > > > > >> >> >>> > > > > > >>> > >>
> > > > > >> >> >>> > > > > > >>> >
> > > > > >> >> >>> > > > > > >>>
> > > > > >> >> >>> > > > > >
> > > > > >> >> >>> > > > >
> > > > > >> >> >>> > > >
> > > > > >> >> >>> > >
> > > > > >> >> >>> >
> > > > > >> >> >>>
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 2) Spec document in parquet
> > > format
> > > > > >> repo :
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > https://github.com/apache/parquet-format/pull/557
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 3) Alp implementation in
> arrow
> > > c++
> > > > > >> repo :
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > https://github.com/apache/arrow/pull/48345/changes
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 4) Alp implementation in
> > > > parquet-java
> > > > > >> >> repo :
> > > > > >> >> >>> Work
> > > > > >> >> >>> > > for
> > > > > >> >> >>> > > > > > >>> Vinoo and
> > > > > >> >> >>> > > > > > >>> > >> > Julien
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > https://github.com/apache/parquet-java/pull/3397
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> 5) PR with test and
> > benchmarking
> > > > > >> >> artifacts in
> > > > > >> >> >>> > > > > > >>> parquet-testing
> > > > > >> >> >>> > > > > > >>> > >> repo :
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > https://github.com/apache/parquet-testing/pull/100
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> And
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> - Go : Arnav just
> submitted
> > > an
> > > > in
> > > > > >> >> >>> progress
> > > > > >> >> >>> > > > > > >>> implementation
> > > > > >> >> >>> > > > > > >>> > in
> > > > > >> >> >>> > > > > > >>> > >> Go.
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> https://github.com/apache/arrow-go/pull/704
> > > > > >> >> >>> > (I
> > > > > >> >> >>> > > > > > haven't
> > > > > >> >> >>> > > > > > >>> > >> started
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> looking at it yet)
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> - Rust : I remember
> Andrew
> > > > > >> mentioned
> > > > > >> >> that
> > > > > >> >> >>> > this
> > > > > >> >> >>> > > > work
> > > > > >> >> >>> > > > > > is
> > > > > >> >> >>> > > > > > >>> also
> > > > > >> >> >>> > > > > > >>> > >> in
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> progress (So 4
> languages!)
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> *Arrow C++ implementation *
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> The PR is out and was also
> used
> > > by
> > > > > >> >> Antoine to
> > > > > >> >> >>> > > report
> > > > > >> >> >>> > > > > the
> > > > > >> >> >>> > > > > > >>> > numbers
> > > > > >> >> >>> > > > > > >>> > >> as
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> reported here. Micah and
> > > Konstantin
> > > > > >> have
> > > > > >> >> >>> given 1
> > > > > >> >> >>> > > > round
> > > > > >> >> >>> > > > > of
> > > > > >> >> >>> > > > > > >>> > >> feedback
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> and I'm addressing them
> today.
> > > > Please
> > > > > >> note
> > > > > >> >> >>> that
> > > > > >> >> >>> > the
> > > > > >> >> >>> > > > > > default
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> optimization flag for
> compiling
> > > is
> > > > O2
> > > > > >> and
> > > > > >> >> not
> > > > > >> >> >>> > Q3. I
> > > > > >> >> >>> > > > got
> > > > > >> >> >>> > > > > > >>> around
> > > > > >> >> >>> > > > > > >>> > >> 70%
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> performance improvement in
> the
> > > > > decoding
> > > > > >> >> speed
> > > > > >> >> >>> > when
> > > > > >> >> >>> > > > > using
> > > > > >> >> >>> > > > > > >>> the O3
> > > > > >> >> >>> > > > > > >>> > >> > flag.
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> *Parqet-MR Java
> implementation
> > > > > (working
> > > > > >> >> with
> > > > > >> >> >>> > Vinoo
> > > > > >> >> >>> > > > and
> > > > > >> >> >>> > > > > > >>> Julien)
> > > > > >> >> >>> > > > > > >>> > >> and
> > > > > >> >> >>> > > > > > >>> > >> > **Cross
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Language testing*
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Let me know if you have
> any
> > > > > >> questions
> > > > > >> >> or
> > > > > >> >> >>> > > > feedback.
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Now pasting some performance
> > > > numbers
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Table 1: C++ ALP Double
> > > Decode —
> > > > > >> >> Spotify
> > > > > >> >> >>> > Columns
> > > > > >> >> >>> > > > > > >>> (Graviton
> > > > > >> >> >>> > > > > > >>> > 3,
> > > > > >> >> >>> > > > > > >>> > >> ARM
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> Neoverse V1)
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>>
> > > > > >> >> >>>
> > ┌──────────────────┬──────────────┬──────────────┬─────────┐
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ Column │ -O2
> > > (MB/s)
> > > > > │
> > > > > >> >> -O3
> > > > > >> >> >>> > > (MB/s) │
> > > > > >> >> >>> > > > > > >>> Speedup │
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>>
> > > > > >> >> >>>
> > ├──────────────────┼──────────────┼──────────────┼─────────┤
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ valence │
> > 3,155
> > > > > │
> > > > > >> >> >>> 5,523
> > > > > >> >> >>> > > │
> > > > > >> >> >>> > > > > > >>> 1.75x │
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ danceability │
> > 3,233
> > > > > │
> > > > > >> >> >>> 5,685
> > > > > >> >> >>> > > │
> > > > > >> >> >>> > > > > > >>> 1.76x │
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ energy │
> > 3,197
> > > > > │
> > > > > >> >> >>> 5,652
> > > > > >> >> >>> > > │
> > > > > >> >> >>> > > > > > >>> 1.77x │
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ loudness │
> > 3,186
> > > > > │
> > > > > >> >> >>> 5,473
> > > > > >> >> >>> > > │
> > > > > >> >> >>> > > > > > >>> 1.72x │
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>>
> > > > > >> >> >>>
> > └──────────────────┴──────────────┴──────────────┴─────────┘
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> On Wed, Feb 25, 2026 at
> 9:49 AM
> > > > > PRATEEK
> > > > > >> >> GAUR
> > > > > >> >> >>> <
> > > > > >> >> >>> > > > > > >>> > prtkgaur@gmail.com
> > > > > >> >> >>> > > > > > >>> > >> >
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>> wrote:
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> @Micah Kornfield <
> > > > > >> emkornfield@gmail.com>
> > > > > >> >> :
> > > > > >> >> >>> Got
> > > > > >> >> >>> > > it.
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> @Andrew Lamb <
> > > > > andrewlamb11@gmail.com>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> Do you think it would be
> good
> > > to
> > > > > >> start
> > > > > >> >> >>> moving
> > > > > >> >> >>> > the
> > > > > >> >> >>> > > > > spec
> > > > > >> >> >>> > > > > > >>> > >> development
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> into
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> markdown format, in
> > preparation
> > > > for
> > > > > >> >> >>> finalizing
> > > > > >> >> >>> > > it?
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> Yes I'll update the numbers
> > for
> > > > some
> > > > > >> of
> > > > > >> >> the
> > > > > >> >> >>> > > > examples I
> > > > > >> >> >>> > > > > > >>> have in
> > > > > >> >> >>> > > > > > >>> > >> the
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> spec based
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> on the updated header size.
> > Then
> > > > we
> > > > > >> >> should
> > > > > >> >> >>> be
> > > > > >> >> >>> > good
> > > > > >> >> >>> > > > to
> > > > > >> >> >>> > > > > go
> > > > > >> >> >>> > > > > > >>> for
> > > > > >> >> >>> > > > > > >>> > the
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> markdown format.
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>> Thanks everyone!
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> Andrew
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at
> > 7:28 PM
> > > > > >> PRATEEK
> > > > > >> >> >>> GAUR <
> > > > > >> >> >>> > > > > > >>> > >> prtkgaur@gmail.com>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> wrote:
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Hi team,
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 1) Andrew
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for working
> on
> > > test
> > > > > >> files.
> > > > > >> >> >>> My PR
> > > > > >> >> >>> > > did
> > > > > >> >> >>> > > > > add
> > > > > >> >> >>> > > > > > >>> all
> > > > > >> >> >>> > > > > > >>> > the
> > > > > >> >> >>> > > > > > >>> > >> > test
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> files I
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> used to benchmark on
> > > > datasets.
> > > > > >> >> Maybe
> > > > > >> >> >>> we
> > > > > >> >> >>> > can
> > > > > >> >> >>> > > > club
> > > > > >> >> >>> > > > > > it
> > > > > >> >> >>> > > > > > >>> > >> together.
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> WIll also
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> aid
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> cross language testing
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Kosta Tarasov
> working
> > > on
> > > > > Rust
> > > > > >> >> >>> > > > implementation.
> > > > > >> >> >>> > > > > > >>> This is
> > > > > >> >> >>> > > > > > >>> > >> > great.
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> Thanks
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 2) Antoine
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks a lot for
> > > reporting
> > > > > the
> > > > > >> >> >>> numbers
> > > > > >> >> >>> > on
> > > > > >> >> >>> > > > AMD.
> > > > > >> >> >>> > > > > > >>> Looks
> > > > > >> >> >>> > > > > > >>> > >> like
> > > > > >> >> >>> > > > > > >>> > >> > you
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> are
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> getting 8X the
> decoding
> > > > > >> >> performance of
> > > > > >> >> >>> > BSS.
> > > > > >> >> >>> > > > This
> > > > > >> >> >>> > > > > > is
> > > > > >> >> >>> > > > > > >>> > >> > amazing!!.
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for
> > acknowledging
> > > > the
> > > > > >> >> >>> sampling
> > > > > >> >> >>> > > > design.
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - I agree with you on
> > > > > >> Fastlanes. In
> > > > > >> >> >>> some
> > > > > >> >> >>> > > crude
> > > > > >> >> >>> > > > > > >>> > >> experiments I
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> didn't get
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> a good perf benefit
> from
> > > it
> > > > on
> > > > > >> >> >>> Graviton3
> > > > > >> >> >>> > > (but
> > > > > >> >> >>> > > > > > maybe
> > > > > >> >> >>> > > > > > >>> > there
> > > > > >> >> >>> > > > > > >>> > >> was
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> something
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> wrong with my
> > > > implementation).
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Locking the 16bit
> > > > exception
> > > > > >> >> >>> encoding for
> > > > > >> >> >>> > > the
> > > > > >> >> >>> > > > > > spec
> > > > > >> >> >>> > > > > > >>> in
> > > > > >> >> >>> > > > > > >>> > >> this
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> case.
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Awesome I think we
> > have
> > > > > solved
> > > > > >> >> for
> > > > > >> >> >>> all
> > > > > >> >> >>> > > open
> > > > > >> >> >>> > > > > > >>> questions
> > > > > >> >> >>> > > > > > >>> > >> minus
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> the
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> version byte :). (will
> > get
> > > > > back
> > > > > >> on
> > > > > >> >> >>> this
> > > > > >> >> >>> > > soon)
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 3) Micah
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - FastLanes : The
> > current
> > > > spec
> > > > > >> does
> > > > > >> >> >>> allow
> > > > > >> >> >>> > > for
> > > > > >> >> >>> > > > > > using
> > > > > >> >> >>> > > > > > >>> > >> FastLane
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> with the
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> configurable enum
> value
> > > for
> > > > > >> >> layout. We
> > > > > >> >> >>> > > should
> > > > > >> >> >>> > > > be
> > > > > >> >> >>> > > > > > >>> able to
> > > > > >> >> >>> > > > > > >>> > >> > inject
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> any
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> layout
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> in the current design.
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Working on resolving all
> > > > remaining
> > > > > >> open
> > > > > >> >> >>> > comments
> > > > > >> >> >>> > > > on
> > > > > >> >> >>> > > > > > the
> > > > > >> >> >>> > > > > > >>> spec
> > > > > >> >> >>> > > > > > >>> > >> this
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> week.
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Best
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Prateek
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at
> > > 3:37 AM
> > > > > >> Steve
> > > > > >> >> >>> > Loughran <
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> stevel@cloudera.com>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> wrote:
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at
> > 18:12,
> > > > > Micah
> > > > > >> >> >>> Kornfield
> > > > > >> >> >>> > <
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> emkornfield@gmail.com>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> wrote:
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> It looks like the actual
> > > issue
> > > > > >> >> >>> described for
> > > > > >> >> >>> > > ORC
> > > > > >> >> >>> > > > > in
> > > > > >> >> >>> > > > > > >>> the
> > > > > >> >> >>> > > > > > >>> > >> paper
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> is that
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> it
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> has multiple
> sub-encodings
> > > in
> > > > a
> > > > > >> >> batch.
> > > > > >> >> >>> This
> > > > > >> >> >>> > > is
> > > > > >> >> >>> > > > > > >>> different
> > > > > >> >> >>> > > > > > >>> > >> then
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> the
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> design
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> proposed here where
> there
> > is
> > > > > still
> > > > > >> >> fixed
> > > > > >> >> >>> > > > encoding
> > > > > >> >> >>> > > > > > per
> > > > > >> >> >>> > > > > > >>> page
> > > > > >> >> >>> > > > > > >>> > >> in
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> parquet.
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> Given reasonably sized
> > > pages I
> > > > > >> don't
> > > > > >> >> >>> think
> > > > > >> >> >>> > > > branch
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> misprediction should
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> be a
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> big issue for new
> > encodings.
> > > > I
> > > > > >> agree
> > > > > >> >> >>> that
> > > > > >> >> >>> > we
> > > > > >> >> >>> > > > > should
> > > > > >> >> >>> > > > > > >>> be
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> conservative in
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> general for adding new
> > > > > encodings.
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> +1
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > > >> >> >>> > > > > > >>> > >> > >
> > > > > >> >> >>> > > > > > >>> > >> >
> > > > > >> >> >>> > > > > > >>> > >> >
> > > > > >> >> >>> > > > > > >>> > >> >
> > > > > >> >> >>> > > > > > >>> > >>
> > > > > >> >> >>> > > > > > >>> > >
> > > > > >> >> >>> > > > > > >>> >
> > > > > >> >> >>> > > > > > >>>
> > > > > >> >> >>> > > > > > >>
> > > > > >> >> >>> > > > > >
> > > > > >> >> >>> > > > >
> > > > > >> >> >>> > > >
> > > > > >> >> >>> > >
> > > > > >> >> >>> >
> > > > > >> >> >>>
> > > > > >> >> >>
> > > > > >> >>
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Vinoo Ganesh <vi...@gmail.com>.
Quick update re: the Java impl. Thanks for the feedback, on
https://github.com/apache/parquet-java/pull/3397, Russel! I pulled
https://github.com/apache/parquet-java/pull/3791 into its own PR (thanks
for the quick approval too, Gang). Assuming no objections, I think we can
merge #3791, and then go ahead of #3397.
Thanks,
Vinoo Ganesh | vinoo.ganesh@gmail.com
<Vi...@gmail.com>
On Tue, Sep 8, 2026 at 10:54 AM PRATEEK GAUR <pr...@gmail.com> wrote:
> [Rust] Awesome thanks Andrew, Kostas and Devin.
> [Java] Thanks Gang for the feedback on Java PR.
> [C++] Thanks Gang for another round of feedback on the C++ one.
> [Go] Arnav and I will be working on the golang impl this week.
>
> This is great progress all around.
>
> On Sat, Sep 5, 2026 at 1:26 PM Vinoo Ganesh <vi...@gmail.com>
> wrote:
>
> > Thanks all - I'm addressing Gang's comments on
> > https://github.com/apache/parquet-java/pull/3397 now, so hopefully well
> > also be in a good state for merging the java implementation soon.
> >
> > Thanks,
> > Vinoo Ganesh | vinoo.ganesh@gmail.com
> >
> > <Vi...@gmail.com>
> >
> >
> > On Sat, Sep 5, 2026 at 8:56 AM Andrew Lamb <an...@gmail.com>
> wrote:
> >
> > > We just merged the Rust implementation[1] from Kosta and Devan. With
> the
> > > imminent release of Parquet Format 2.14.0 I think we'll be well on our
> > way.
> > >
> > > Thanks again for pushing this along,
> > > Andrew
> > >
> > > [1]: https://github.com/apache/arrow-rs/pull/9372
> > >
> > > On Fri, Sep 4, 2026 at 5:42 PM PRATEEK GAUR <pr...@gmail.com>
> wrote:
> > >
> > > > Hi team,
> > > >
> > > > Thanks Micah for multiple rounds of review and thanks Gang for
> taking a
> > > > look at the PR and giving feedback. I've addressed all comments in
> the
> > > PR.
> > > > Looking forward to the next round of feedback (or approval). Would
> like
> > > to
> > > > get it in soon.
> > > >
> > > > https://github.com/apache/arrow/pull/48345/changes
> > > >
> > > > With this done and after talking to Arnav, I plan to spend some time
> on
> > > the
> > > > ALP implementation in the arrow-go repository helping spread the
> > support
> > > to
> > > > more languages.
> > > >
> > > > Best
> > > > Prateek
> > > >
> > > > On Sun, Aug 2, 2026 at 10:42 PM PRATEEK GAUR <pr...@gmail.com>
> > wrote:
> > > >
> > > > > Thanks Divjot for bringing it up.
> > > > >
> > > > > I was under the same impression as Russell and Andrew that we'd
> have
> > it
> > > > > flag gated.
> > > > > With the new versioning proposal 'preview' might be a good tag for
> > it.
> > > > >
> > > > > Best
> > > > > Prateek
> > > > >
> > > > > On Sat, Aug 1, 2026 at 12:39 AM Andrew Lamb <
> andrewlamb11@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> Div: Here is a proposal to mark ALP in Preview: [1] (using a
> comment
> > > in
> > > > >> the
> > > > >> spec section).
> > > > >>
> > > > >> Russel: That is my understanding as well.
> > > > >>
> > > > >> Hopefully we are able to formalize the versioning process shortly
> so
> > > we
> > > > >> can
> > > > >> formalize the notion of "preview"
> > > > >>
> > > > >> Andrew
> > > > >>
> > > > >>
> > > > >> [1]: https://github.com/apache/parquet-format/pull/604
> > > > >>
> > > > >> On Fri, Jul 31, 2026 at 11:15 AM Russell Spitzer <
> > > > >> russell.spitzer@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >> > I thought we had agreed it would be merged, and features like
> this
> > > > would
> > > > >> > be feature flag gated. So a V-Old writer could always opt in to
> > > using
> > > > >> this
> > > > >> > encoding but that must be a conscious effort if you are not
> using
> > > > VNext
> > > > >> of
> > > > >> > Parquet. If you are on V-Next the writer can choose to use ALP
> > > > whenever
> > > > >> it
> > > > >> > likes without user intervention.
> > > > >> >
> > > > >> > On Fri, Jul 31, 2026 at 2:20 AM Divjot Arora via dev <
> > > > >> > dev@parquet.apache.org> wrote:
> > > > >> >
> > > > >> >> Yes, that’s my understanding as well. However, as you noted,
> the
> > > > >> >> versioning
> > > > >> >> discussion is still in early stages and hasn’t prescribed
> > concrete
> > > > >> >> mechanisms yet. If ALP goes into parquet-format 2.14.0, there’s
> > no
> > > > >> >> indication that it’s in preview or that writers shouldn’t use
> it
> > by
> > > > >> >> default
> > > > >> >> yet. Given that we’ve held off on merging other incompatible
> > > changes
> > > > >> (e.g
> > > > >> >> removal of path_in_schema) until the versioning discussion
> > closes,
> > > my
> > > > >> >> preference would be to leave ALP un-merged for the time being
> as
> > > > well.
> > > > >> >>
> > > > >> >> — Div
> > > > >> >>
> > > > >> >> On Fri, Jul 31, 2026 at 08:59 Andrew Lamb <
> > andrewlamb11@gmail.com>
> > > > >> wrote:
> > > > >> >>
> > > > >> >> > After re-reading Julien's Versioning Proposal, my
> understanding
> > > is
> > > > >> that
> > > > >> >> if
> > > > >> >> > we adopt that system, we will somehow mark ALP in "Preview"
> in
> > > > >> >> > parquet-format for release Version 3.
> > > > >> >> >
> > > > >> >> > The versioning document doesn't seem to proscribe the
> > particular
> > > > >> method
> > > > >> >> > for annotating "preview" features -- I think clearly marking
> > them
> > > > as
> > > > >> >> such
> > > > >> >> > in parquet.thrift as well as reflecting that on the
> > > > >> >> > https://parquet.apache.org/docs/file-format/versions/ page
> > > somehow
> > > > >> >> would
> > > > >> >> > be adequate.
> > > > >> >> >
> > > > >> >> > Andrew
> > > > >> >> >
> > > > >> >> > On Thu, Jul 30, 2026 at 11:50 AM Andrew Lamb <
> > > > andrewlamb11@gmail.com
> > > > >> >
> > > > >> >> > wrote:
> > > > >> >> >
> > > > >> >> >> In my mind ALP is in the same position as other encodings
> that
> > > > were
> > > > >> >> added
> > > > >> >> >> recently[1] (such as BYTE_STREAM_SPLIT for non floating
> point
> > > > types)
> > > > >> >> >>
> > > > >> >> >> That is to say it will be available in the latest open
> source
> > > > >> writers
> > > > >> >> and
> > > > >> >> >> readers, but realistically not widely used in the ecosystem
> > > where
> > > > >> >> >> compatibility is a concern (a large part)
> > > > >> >> >>
> > > > >> >> >> I view Julian's work[2][3] to gather consensus on how to
> name
> > > > >> >> >> combinations of features (e.g. "versions") as the mechanism
> by
> > > > >> which we
> > > > >> >> >> will accelerate the adoption of this change (along with
> other
> > > > >> forward
> > > > >> >> >> incompatible changes) across the ecosystem.
> > > > >> >> >>
> > > > >> >> >> > Will we exclude this change when cutting parquet-format
> > > v2.14.0?
> > > > >> >> >>
> > > > >> >> >> No I don't think so -- I would expect it to go out in the
> next
> > > > >> >> >> parquet-format version as do other changes in the spec.
> > > > >> >> >>
> > > > >> >> >> Andrew
> > > > >> >> >>
> > > > >> >> >> [1]: https://parquet.apache.org/docs/file-format/versions/
> > > > >> >> >> [2]:
> > > > >> https://lists.apache.org/thread/xmnj8h0h8ozmrgox5tydhhs7yz7q2hss
> > > > >> >> >> [3]:
> > > > >> https://lists.apache.org/thread/qlf8lg90gqq41lllyy9mk2f0skqvftqj
> > > > >> >> >>
> > > > >> >> >>
> > > > >> >> >> On Thu, Jul 30, 2026 at 3:51 AM Divjot Arora via dev <
> > > > >> >> >> dev@parquet.apache.org> wrote:
> > > > >> >> >>
> > > > >> >> >>> Hi Andrew,
> > > > >> >> >>>
> > > > >> >> >>> How does this interact with the ongoing versioning
> > discussion?
> > > > >> Given
> > > > >> >> that
> > > > >> >> >>> new encodings are not forward compatible, I would assume
> ALP
> > is
> > > > >> >> included
> > > > >> >> >>> in
> > > > >> >> >>> Parquet V<next> (V3?), but it's not clear to me how that
> > works
> > > if
> > > > >> it's
> > > > >> >> >>> merged to the parquet-format master branch now. Will we
> > exclude
> > > > >> this
> > > > >> >> >>> change
> > > > >> >> >>> when cutting parquet-format v2.14.0?
> > > > >> >> >>>
> > > > >> >> >>> -- Div
> > > > >> >> >>>
> > > > >> >> >>> On Thu, Jul 30, 2026 at 8:51 AM Andrew Lamb <
> > > > >> andrewlamb11@gmail.com>
> > > > >> >> >>> wrote:
> > > > >> >> >>>
> > > > >> >> >>> > Hello,
> > > > >> >> >>> >
> > > > >> >> >>> > As an update, thanks to the epic work from many
> > contributors,
> > > > >> >> >>> especially
> > > > >> >> >>> > Prateek, I just merged the parquet-format change for
> > ALP[1].
> > > We
> > > > >> >> plan to
> > > > >> >> >>> > make additional editorial The remaining items I know of
> are
> > > > >> merging
> > > > >> >> the
> > > > >> >> >>> > examples from parquet-testing[2], and then of course the
> > > > various
> > > > >> >> >>> > implementations
> > > > >> >> >>> >
> > > > >> >> >>> > Thanks again everyone. This is a great step forward
> > > > >> >> >>> > Andrew
> > > > >> >> >>> >
> > > > >> >> >>> > [1]: https://github.com/apache/parquet-format/pull/557
> > > > >> >> >>> > [2]: https://github.com/apache/parquet-testing/pull/100
> > > > >> >> >>> >
> > > > >> >> >>> > On Tue, Jul 21, 2026 at 4:39 PM Julien Le Dem <
> > > > julien@apache.org
> > > > >> >
> > > > >> >> >>> wrote:
> > > > >> >> >>> >
> > > > >> >> >>> > > It was also in my spam folder... :(
> > > > >> >> >>> > > I've seen a bunch of apache list emails go to my spam
> > > > recently.
> > > > >> >> >>> > >
> > > > >> >> >>> > > On Tue, Jul 21, 2026 at 10:58 AM Russell Spitzer <
> > > > >> >> >>> > > russell.spitzer@gmail.com>
> > > > >> >> >>> > > wrote:
> > > > >> >> >>> > >
> > > > >> >> >>> > > > Yes, my hope was that my vote would kick it out of
> spam
> > > but
> > > > >> we
> > > > >> >> >>> > definitely
> > > > >> >> >>> > > > had two other votes that also went to spam for me.
> > > > >> >> >>> > > >
> > > > >> >> >>> > > > For anyone looking the thread is here
> > > > >> >> >>> > > >
> > > > >> >>
> https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt
> > > > >> >> >>> > > >
> > > > >> >> >>> > > > On Tue, Jul 21, 2026 at 10:57 AM Micah Kornfield <
> > > > >> >> >>> > emkornfield@gmail.com>
> > > > >> >> >>> > > > wrote:
> > > > >> >> >>> > > >
> > > > >> >> >>> > > > > I've seen other's votes not go to spam, so
> hopefully
> > > > people
> > > > >> >> will
> > > > >> >> >>> see
> > > > >> >> >>> > it
> > > > >> >> >>> > > > in
> > > > >> >> >>> > > > > there inboxes (or see this thread and look in there
> > > > spam).
> > > > >> >> >>> > > > >
> > > > >> >> >>> > > > > On Tue, Jul 21, 2026 at 10:52 AM PRATEEK GAUR <
> > > > >> >> >>> prtkgaur@gmail.com>
> > > > >> >> >>> > > > wrote:
> > > > >> >> >>> > > > >
> > > > >> >> >>> > > > > > Okay,
> > > > >> >> >>> > > > > >
> > > > >> >> >>> > > > > > I sent out the voting email yesterday and so far
> > I've
> > > > >> heard
> > > > >> >> >>> from 3
> > > > >> >> >>> > > > people
> > > > >> >> >>> > > > > > that it ended up in their "spam folder". Dev VPN
> to
> > > be
> > > > >> >> blamed
> > > > >> >> >>> for
> > > > >> >> >>> > > it?.
> > > > >> >> >>> > > > > >
> > > > >> >> >>> > > > > > What's the protocol here?
> > > > >> >> >>> > > > > > I let people hunt for the email in their spam
> > folder
> > > or
> > > > >> send
> > > > >> >> >>> out
> > > > >> >> >>> > > > another
> > > > >> >> >>> > > > > > one?
> > > > >> >> >>> > > > > >
> > > > >> >> >>> > > > > > Best
> > > > >> >> >>> > > > > > Prateek
> > > > >> >> >>> > > > > >
> > > > >> >> >>> > > > > > On Sun, Jul 19, 2026 at 10:47 PM PRATEEK GAUR <
> > > > >> >> >>> prtkgaur@gmail.com>
> > > > >> >> >>> > > > > wrote:
> > > > >> >> >>> > > > > >
> > > > >> >> >>> > > > > > > Hi team,
> > > > >> >> >>> > > > > > >
> > > > >> >> >>> > > > > > > Vinoo has added the tests which were requested
> in
> > > the
> > > > >> last
> > > > >> >> >>> round
> > > > >> >> >>> > of
> > > > >> >> >>> > > > > > review.
> > > > >> >> >>> > > > > > > In his own words.
> > > > >> >> >>> > > > > > >
> > > > >> >> >>> > > > > > > "
> > > > >> >> >>> > > > > > > The extreme value tests Micah wanted are done
> and
> > > > >> pushed.
> > > > >> >> I
> > > > >> >> >>> added
> > > > >> >> >>> > > > > > coverage
> > > > >> >> >>> > > > > > > for values that need the full FOR bit width
> after
> > > the
> > > > >> >> frame
> > > > >> >> >>> of
> > > > >> >> >>> > > > > reference
> > > > >> >> >>> > > > > > is
> > > > >> >> >>> > > > > > > applied: 64-bit (the case where the signed max
> > > minus
> > > > >> min
> > > > >> >> >>> > > subtraction
> > > > >> >> >>> > > > > > > overflows), 63-bit (the non-overflow case), and
> > > > 32-bit
> > > > >> for
> > > > >> >> >>> > floats.
> > > > >> >> >>> > > > All
> > > > >> >> >>> > > > > > > lossless with zero exceptions. I also verified
> > them
> > > > >> >> through
> > > > >> >> >>> Arrow
> > > > >> >> >>> > > C++
> > > > >> >> >>> > > > > ALP
> > > > >> >> >>> > > > > > > reader (the local build), so the Java-written
> > > extreme
> > > > >> >> columns
> > > > >> >> >>> > > decode
> > > > >> >> >>> > > > > > > correctly in C++, bit-exact against the
> expected
> > > > >> values."
> > > > >> >> >>> > > > > > >
> > > > >> >> >>> > > > > > > State of the PR (#3397):
> > > > >> >> >>> > > > > > >
> > > > >> >> >>> > > > > > > - Merges cleanly with master (I pulled
> master
> > in
> > > > and
> > > > >> >> >>> resolved
> > > > >> >> >>> > > the
> > > > >> >> >>> > > > > > > conflicts).
> > > > >> >> >>> > > > > > > - 41 of 44 review threads resolved. The 3
> > > > remaining
> > > > >> are
> > > > >> >> >>> > > > intentional:
> > > > >> >> >>> > > > > > > one deferred perf nit, one that stays open
> > until
> > > > >> >> >>> > parquet-format
> > > > >> >> >>> > > > > ships
> > > > >> >> >>> > > > > > ALP,
> > > > >> >> >>> > > > > > >
> > > > >> >> >>> > > > > > > "
> > > > >> >> >>> > > > > > >
> > > > >> >> >>> > > > > > > With this both the
> > > > >> >> >>> > > > > > > 1) C++ PR :
> > > > >> >> >>> https://github.com/apache/arrow/pull/48345/changes
> > > > >> >> >>> > > > > > > 2) Java PR :
> > > > >> >> >>> https://github.com/apache/parquet-java/pull/3397
> > > > >> >> >>> > > > > > >
> > > > >> >> >>> > > > > > > Are cross language tested and range of test
> > > > scenarios.
> > > > >> >> >>> > > > > > >
> > > > >> >> >>> > > > > > > Best
> > > > >> >> >>> > > > > > > Prateek
> > > > >> >> >>> > > > > > >
> > > > >> >> >>> > > > > > >
> > > > >> >> >>> > > > > > > On Mon, Jul 13, 2026 at 8:38 AM PRATEEK GAUR <
> > > > >> >> >>> prtkgaur@gmail.com
> > > > >> >> >>> > >
> > > > >> >> >>> > > > > wrote:
> > > > >> >> >>> > > > > > >
> > > > >> >> >>> > > > > > >> Thanks Matt,
> > > > >> >> >>> > > > > > >>
> > > > >> >> >>> > > > > > >> I'll be on it this week.
> > > > >> >> >>> > > > > > >>
> > > > >> >> >>> > > > > > >> Best
> > > > >> >> >>> > > > > > >> Prateek
> > > > >> >> >>> > > > > > >>
> > > > >> >> >>> > > > > > >> On Mon, Jul 13, 2026 at 8:36 AM Matt Topol <
> > > > >> >> >>> > > zotthewizard@gmail.com>
> > > > >> >> >>> > > > > > >> wrote:
> > > > >> >> >>> > > > > > >>
> > > > >> >> >>> > > > > > >>> There is also the Go implementation submitted
> > by
> > > > >> Arnav (
> > > > >> >> >>> > > > > > >>> https://github.com/apache/arrow-go/pull/704)
> > > which
> > > > >> is
> > > > >> >> >>> waiting
> > > > >> >> >>> > > for
> > > > >> >> >>> > > > > > >>> updates.
> > > > >> >> >>> > > > > > >>>
> > > > >> >> >>> > > > > > >>> Just wanted to make sure it didn't get lost
> > here.
> > > > >> >> >>> > > > > > >>>
> > > > >> >> >>> > > > > > >>> --Matt
> > > > >> >> >>> > > > > > >>>
> > > > >> >> >>> > > > > > >>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK
> GAUR <
> > > > >> >> >>> > > prtkgaur@gmail.com>
> > > > >> >> >>> > > > > > >>> wrote:
> > > > >> >> >>> > > > > > >>>
> > > > >> >> >>> > > > > > >>> > Hi all,
> > > > >> >> >>> > > > > > >>> >
> > > > >> >> >>> > > > > > >>> > I'd like to share a status update on the
> ALP
> > > > >> encoding
> > > > >> >> >>> effort
> > > > >> >> >>> > > and
> > > > >> >> >>> > > > > get
> > > > >> >> >>> > > > > > a
> > > > >> >> >>> > > > > > >>> feel
> > > > >> >> >>> > > > > > >>> > from the community before starting a formal
> > > vote.
> > > > >> >> >>> > > > > > >>> >
> > > > >> >> >>> > > > > > >>> > We now have the specification plus
> > > > implementations
> > > > >> in
> > > > >> >> >>> both
> > > > >> >> >>> > > > > languages:
> > > > >> >> >>> > > > > > >>> >
> > > > >> >> >>> > > > > > >>> > - Spec:
> > apache/parquet-format#557 —
> > > > >> GH-533
> > > > >> >> >>> Add
> > > > >> >> >>> > ALP
> > > > >> >> >>> > > > > > encoding
> > > > >> >> >>> > > > > > >>> > specification
> > > > >> >> >>> > > > > > >>> > - C++ (Arrow): apache/arrow#48345 —
> > > GH-48701
> > > > >> >> >>> > > [C++][Parquet]
> > > > >> >> >>> > > > > Add
> > > > >> >> >>> > > > > > >>> ALPpd
> > > > >> >> >>> > > > > > >>> > encoding
> > > > >> >> >>> > > > > > >>> > - Java:
> apache/parquet-java#3397
> > —
> > > > >> Parquet
> > > > >> >> >>> Java
> > > > >> >> >>> > ALP
> > > > >> >> >>> > > > > > >>> > Implementation (by Vinoo)
> > > > >> >> >>> > > > > > >>> >
> > > > >> >> >>> > > > > > >>> > Current state:
> > > > >> >> >>> > > > > > >>> >
> > > > >> >> >>> > > > > > >>> > - The C++ implementation has been
> through
> > > 3-4
> > > > >> >> rounds
> > > > >> >> >>> of
> > > > >> >> >>> > > > review.
> > > > >> >> >>> > > > > > >>> > - The Java implementation has been
> > tracking
> > > > >> well
> > > > >> >> and
> > > > >> >> >>> has
> > > > >> >> >>> > > had
> > > > >> >> >>> > > > > > >>> initial
> > > > >> >> >>> > > > > > >>> > reviews from contributors, with no
> > major
> > > > >> >> remaining
> > > > >> >> >>> > > issues.
> > > > >> >> >>> > > > > > >>> > - Cross-language compatibility tests
> are
> > > > >> passing:
> > > > >> >> the
> > > > >> >> >>> > Arrow
> > > > >> >> >>> > > > C++
> > > > >> >> >>> > > > > > >>> decoder
> > > > >> >> >>> > > > > > >>> > reads Java-written data bit-exactly
> > > across
> > > > >> >> ~1.56M
> > > > >> >> >>> > values
> > > > >> >> >>> > > > and
> > > > >> >> >>> > > > > 18
> > > > >> >> >>> > > > > > >>> > fixtures, covering V1 and V2 pages,
> > > > multiple
> > > > >> >> vector
> > > > >> >> >>> > > sizes,
> > > > >> >> >>> > > > > and
> > > > >> >> >>> > > > > > >>> > several
> > > > >> >> >>> > > > > > >>> > real datasets — zero mismatches.
> > > > >> >> >>> > > > > > >>> >
> > > > >> >> >>> > > > > > >>> > As raised in the java-pr we will be adding
> > > > >> coverage
> > > > >> >> for
> > > > >> >> >>> > > extreme
> > > > >> >> >>> > > > > > values
> > > > >> >> >>> > > > > > >>> > (those requiring
> > > > >> >> >>> > > > > > >>> > 63-64 bits after FOR is applied) before we
> > > > close a
> > > > >> >> vote.
> > > > >> >> >>> > We'll
> > > > >> >> >>> > > > aim
> > > > >> >> >>> > > > > > to
> > > > >> >> >>> > > > > > >>> get
> > > > >> >> >>> > > > > > >>> > that
> > > > >> >> >>> > > > > > >>> > done in parallel.
> > > > >> >> >>> > > > > > >>> >
> > > > >> >> >>> > > > > > >>> > Plan: unless there are objections, I intend
> > to
> > > > >> start a
> > > > >> >> >>> vote
> > > > >> >> >>> > at
> > > > >> >> >>> > > > the
> > > > >> >> >>> > > > > > end
> > > > >> >> >>> > > > > > >>> of
> > > > >> >> >>> > > > > > >>> > this week/early
> > > > >> >> >>> > > > > > >>> > next week.
> > > > >> >> >>> > > > > > >>> >
> > > > >> >> >>> > > > > > >>> > Thanks
> > > > >> >> >>> > > > > > >>> > Prateek
> > > > >> >> >>> > > > > > >>> >
> > > > >> >> >>> > > > > > >>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK
> GAUR <
> > > > >> >> >>> > > prtkgaur@gmail.com>
> > > > >> >> >>> > > > > > >>> wrote:
> > > > >> >> >>> > > > > > >>> >
> > > > >> >> >>> > > > > > >>> > > Hi Team,
> > > > >> >> >>> > > > > > >>> > >
> > > > >> >> >>> > > > > > >>> > > Just wanted to provide some updates on
> ALP.
> > > > >> >> >>> > > > > > >>> > > Micah and I have done 3-4 rounds of
> review
> > > for
> > > > >> the
> > > > >> >> c++
> > > > >> >> >>> PR :
> > > > >> >> >>> > > > > > >>> > >
> https://github.com/apache/arrow/pull/48345
> > > > >> >> >>> > > > > > >>> > >
> > > > >> >> >>> > > > > > >>> > > For the Java implementation Vinoo has
> been
> > > > >> working
> > > > >> >> on
> > > > >> >> >>> > > following
> > > > >> >> >>> > > > > PR
> > > > >> >> >>> > > > > > :
> > > > >> >> >>> > > > > > >>> > >
> > > > https://github.com/apache/parquet-java/pull/3397
> > > > >> >> >>> > > > > > >>> > >
> > > > >> >> >>> > > > > > >>> > > Best
> > > > >> >> >>> > > > > > >>> > > Prateek
> > > > >> >> >>> > > > > > >>> > >
> > > > >> >> >>> > > > > > >>> > > On Tue, May 5, 2026 at 1:50 PM Micah
> > > Kornfield
> > > > <
> > > > >> >> >>> > > > > > >>> emkornfield@gmail.com>
> > > > >> >> >>> > > > > > >>> > > wrote:
> > > > >> >> >>> > > > > > >>> > >
> > > > >> >> >>> > > > > > >>> > >> Hi Antoine,
> > > > >> >> >>> > > > > > >>> > >>
> > > > >> >> >>> > > > > > >>> > >> > Apologies if the question was already
> > > asked,
> > > > >> but
> > > > >> >> >>> should
> > > > >> >> >>> > we
> > > > >> >> >>> > > > > care
> > > > >> >> >>> > > > > > >>> about
> > > > >> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be
> more
> > > > >> >> efficient
> > > > >> >> >>> than
> > > > >> >> >>> > > > > FLOAT16
> > > > >> >> >>> > > > > > +
> > > > >> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> > > > >> >> >>> > > > > > >>> > >>
> > > > >> >> >>> > > > > > >>> > >> It was. We thought we could defer it
> for
> > > the
> > > > >> >> >>> following
> > > > >> >> >>> > > > reasons:
> > > > >> >> >>> > > > > > >>> > >> 1. It's not clear there are a lot of
> easy
> > > > >> >> reference
> > > > >> >> >>> > > datasets
> > > > >> >> >>> > > > to
> > > > >> >> >>> > > > > > >>> test
> > > > >> >> >>> > > > > > >>> > its
> > > > >> >> >>> > > > > > >>> > >> effectiveness. It does look like there
> > > might
> > > > be
> > > > >> >> one
> > > > >> >> >>> or
> > > > >> >> >>> > two
> > > > >> >> >>> > > on
> > > > >> >> >>> > > > > > >>> > huggingface
> > > > >> >> >>> > > > > > >>> > >> (e.g.
> > > > >> >> https://huggingface.co/datasets/kikitora/curdie
> > > > >> >> >>> ).
> > > > >> >> >>> > > > > > >>> > >> 2. It seemed likely that float 16 was
> > more
> > > > >> likely
> > > > >> >> >>> used
> > > > >> >> >>> > for
> > > > >> >> >>> > > > > values
> > > > >> >> >>> > > > > > >>> that
> > > > >> >> >>> > > > > > >>> > >> were less likely to reduce to decimal
> > > values.
> > > > >> >> >>> > > > > > >>> > >> 3. It could be added as an extension
> > later
> > > if
> > > > >> >> needed.
> > > > >> >> >>> > > > > > >>> > >>
> > > > >> >> >>> > > > > > >>> > >> Cheers,
> > > > >> >> >>> > > > > > >>> > >> Micah
> > > > >> >> >>> > > > > > >>> > >>
> > > > >> >> >>> > > > > > >>> > >>
> > > > >> >> >>> > > > > > >>> > >> On Tue, May 5, 2026 at 1:40 PM Antoine
> > > Pitrou
> > > > <
> > > > >> >> >>> > > > > antoine@python.org
> > > > >> >> >>> > > > > > >
> > > > >> >> >>> > > > > > >>> > wrote:
> > > > >> >> >>> > > > > > >>> > >>
> > > > >> >> >>> > > > > > >>> > >> >
> > > > >> >> >>> > > > > > >>> > >> > Hello,
> > > > >> >> >>> > > > > > >>> > >> >
> > > > >> >> >>> > > > > > >>> > >> > Apologies if the question was already
> > > asked,
> > > > >> but
> > > > >> >> >>> should
> > > > >> >> >>> > we
> > > > >> >> >>> > > > > care
> > > > >> >> >>> > > > > > >>> about
> > > > >> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be
> more
> > > > >> >> efficient
> > > > >> >> >>> than
> > > > >> >> >>> > > > > FLOAT16
> > > > >> >> >>> > > > > > +
> > > > >> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> > > > >> >> >>> > > > > > >>> > >> >
> > > > >> >> >>> > > > > > >>> > >> > Regards
> > > > >> >> >>> > > > > > >>> > >> >
> > > > >> >> >>> > > > > > >>> > >> > Antoine.
> > > > >> >> >>> > > > > > >>> > >> >
> > > > >> >> >>> > > > > > >>> > >> >
> > > > >> >> >>> > > > > > >>> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a
> > > écrit
> > > > :
> > > > >> >> >>> > > > > > >>> > >> > > Thanks Andrew and Micah for review
> > > > feedback
> > > > >> on
> > > > >> >> >>> the two
> > > > >> >> >>> > > > PR's
> > > > >> >> >>> > > > > > >>> > >> > > 1) (c++ arrow repo)
> > > > >> >> >>> > > > > > >>> > >>
> > > > >> https://github.com/apache/arrow/pull/48345/changes
> > > > >> >> >>> > > > > > >>> > >> > > 2) (parquet-format repo)
> > > > >> >> >>> > > > > > >>> > >> >
> > > > >> >> https://github.com/apache/parquet-format/pull/557
> > > > >> >> >>> > > > > > >>> > >> > >
> > > > >> >> >>> > > > > > >>> > >> > > I have addressed all (unless I
> missed
> > > > >> >> something)
> > > > >> >> >>> > > comments
> > > > >> >> >>> > > > on
> > > > >> >> >>> > > > > > >>> the two
> > > > >> >> >>> > > > > > >>> > >> > PR's.
> > > > >> >> >>> > > > > > >>> > >> > >
> > > > >> >> >>> > > > > > >>> > >> > > Best
> > > > >> >> >>> > > > > > >>> > >> > > Prateek
> > > > >> >> >>> > > > > > >>> > >> > >
> > > > >> >> >>> > > > > > >>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM
> > PRATEEK
> > > > >> GAUR <
> > > > >> >> >>> > > > > > >>> prtkgaur@gmail.com>
> > > > >> >> >>> > > > > > >>> > >> wrote:
> > > > >> >> >>> > > > > > >>> > >> > >
> > > > >> >> >>> > > > > > >>> > >> > >> Thanks Andrew and Micah.
> > > > >> >> >>> > > > > > >>> > >> > >>
> > > > >> >> >>> > > > > > >>> > >> > >> `fair amount of feedback on at
> least
> > > the
> > > > >> >> >>> > > implementations`
> > > > >> >> >>> > > > > > >>> > >> > >> For the c++ I have already started
> > > > >> addressing
> > > > >> >> the
> > > > >> >> >>> > > > > feedback, I
> > > > >> >> >>> > > > > > >>> > should
> > > > >> >> >>> > > > > > >>> > >> be
> > > > >> >> >>> > > > > > >>> > >> > >> done with that Monday/Tuesday.
> > > > >> >> >>> > > > > > >>> > >> > >> I think Vinoo too has been making
> > good
> > > > >> >> progress
> > > > >> >> >>> on
> > > > >> >> >>> > the
> > > > >> >> >>> > > > Java
> > > > >> >> >>> > > > > > >>> > >> > implementation.
> > > > >> >> >>> > > > > > >>> > >> > >>
> > > > >> >> >>> > > > > > >>> > >> > >> Best
> > > > >> >> >>> > > > > > >>> > >> > >> Prateek
> > > > >> >> >>> > > > > > >>> > >> > >>
> > > > >> >> >>> > > > > > >>> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM
> > Andrew
> > > > >> Lamb <
> > > > >> >> >>> > > > > > >>> > andrewlamb11@gmail.com
> > > > >> >> >>> > > > > > >>> > >> >
> > > > >> >> >>> > > > > > >>> > >> > >> wrote:
> > > > >> >> >>> > > > > > >>> > >> > >>
> > > > >> >> >>> > > > > > >>> > >> > >>> Got it. Thank you for the
> > > clarification
> > > > >> -- I
> > > > >> >> >>> will
> > > > >> >> >>> > try
> > > > >> >> >>> > > > and
> > > > >> >> >>> > > > > > look
> > > > >> >> >>> > > > > > >>> > into
> > > > >> >> >>> > > > > > >>> > >> the
> > > > >> >> >>> > > > > > >>> > >> > >>> spec and the Rust
> implementation[1]
> > in
> > > > >> this
> > > > >> >> next
> > > > >> >> >>> > week
> > > > >> >> >>> > > > > > >>> > >> > >>>
> > > > >> >> >>> > > > > > >>> > >> > >>> [1]:
> > > > >> >> >>> https://github.com/apache/arrow-rs/pull/9372
> > > > >> >> >>> > > > > > >>> > >> > >>>
> > > > >> >> >>> > > > > > >>> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM
> > Micah
> > > > >> >> >>> Kornfield <
> > > > >> >> >>> > > > > > >>> > >> > emkornfield@gmail.com>
> > > > >> >> >>> > > > > > >>> > >> > >>> wrote:
> > > > >> >> >>> > > > > > >>> > >> > >>>
> > > > >> >> >>> > > > > > >>> > >> > >>>> Hi Andrew,
> > > > >> >> >>> > > > > > >>> > >> > >>>> I think there is a fair amount of
> > > > >> feedback
> > > > >> >> on
> > > > >> >> >>> at
> > > > >> >> >>> > > least
> > > > >> >> >>> > > > > the
> > > > >> >> >>> > > > > > >>> > >> > >>>> implementations, typically I
> think
> > > > we've
> > > > >> >> waited
> > > > >> >> >>> > till
> > > > >> >> >>> > > > they
> > > > >> >> >>> > > > > > are
> > > > >> >> >>> > > > > > >>> > >> close to
> > > > >> >> >>> > > > > > >>> > >> > >>>> mergeable before a final vote.
> > > > >> Otherwise I
> > > > >> >> >>> agree
> > > > >> >> >>> > we
> > > > >> >> >>> > > > are
> > > > >> >> >>> > > > > > very
> > > > >> >> >>> > > > > > >>> > >> close.
> > > > >> >> >>> > > > > > >>> > >> > >>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>> -Micah
> > > > >> >> >>> > > > > > >>> > >> > >>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>> On Saturday, April 25, 2026,
> Andrew
> > > > Lamb
> > > > >> <
> > > > >> >> >>> > > > > > >>> andrewlamb11@gmail.com
> > > > >> >> >>> > > > > > >>> > >
> > > > >> >> >>> > > > > > >>> > >> > wrote:
> > > > >> >> >>> > > > > > >>> > >> > >>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>> Thanks Prateek,
> > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>> I think from this content it
> looks
> > > to
> > > > me
> > > > >> >> like
> > > > >> >> >>> we
> > > > >> >> >>> > are
> > > > >> >> >>> > > > > ready
> > > > >> >> >>> > > > > > >>> to
> > > > >> >> >>> > > > > > >>> > >> start a
> > > > >> >> >>> > > > > > >>> > >> > >>>>> vote to explicitly accept ALP
> into
> > > > >> Parquet
> > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>> Does anyone know of a reason we
> > > should
> > > > >> >> >>> postpone it
> > > > >> >> >>> > > for
> > > > >> >> >>> > > > > > >>> longer?
> > > > >> >> >>> > > > > > >>> > >> > >>>>> Perhaps someone needs some more
> > time
> > > > to
> > > > >> >> >>> review?
> > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>> Andrew
> > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM
> > > > PRATEEK
> > > > >> >> GAUR <
> > > > >> >> >>> > > > > > >>> > prtkgaur@gmail.com>
> > > > >> >> >>> > > > > > >>> > >> > >>>>> wrote:
> > > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> Hi team,
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> Hope everyone is doing well. I
> > got
> > > a
> > > > >> >> chance
> > > > >> >> >>> to
> > > > >> >> >>> > work
> > > > >> >> >>> > > > > > >>> through all
> > > > >> >> >>> > > > > > >>> > >> the
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> remaining feedback and update
> the
> > > > spec
> > > > >> >> doc.
> > > > >> >> >>> Here
> > > > >> >> >>> > > are
> > > > >> >> >>> > > > > the
> > > > >> >> >>> > > > > > >>> new
> > > > >> >> >>> > > > > > >>> > >> > artifacts
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> 1) Spec document :
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> >
> > > > >> >> >>> > > > > > >>> > >>
> > > > >> >> >>> > > > > > >>> >
> > > > >> >> >>> > > > > > >>>
> > > > >> >> >>> > > > > >
> > > > >> >> >>> > > > >
> > > > >> >> >>> > > >
> > > > >> >> >>> > >
> > > > >> >> >>> >
> > > > >> >> >>>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> 2) Spec document in parquet
> > format
> > > > >> repo :
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > https://github.com/apache/parquet-format/pull/557
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> 3) Alp implementation in arrow
> > c++
> > > > >> repo :
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > https://github.com/apache/arrow/pull/48345/changes
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> 4) Alp implementation in
> > > parquet-java
> > > > >> >> repo :
> > > > >> >> >>> Work
> > > > >> >> >>> > > for
> > > > >> >> >>> > > > > > >>> Vinoo and
> > > > >> >> >>> > > > > > >>> > >> > Julien
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > https://github.com/apache/parquet-java/pull/3397
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> 5) PR with test and
> benchmarking
> > > > >> >> artifacts in
> > > > >> >> >>> > > > > > >>> parquet-testing
> > > > >> >> >>> > > > > > >>> > >> repo :
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > https://github.com/apache/parquet-testing/pull/100
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> And
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> - Go : Arnav just submitted
> > an
> > > in
> > > > >> >> >>> progress
> > > > >> >> >>> > > > > > >>> implementation
> > > > >> >> >>> > > > > > >>> > in
> > > > >> >> >>> > > > > > >>> > >> Go.
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> https://github.com/apache/arrow-go/pull/704
> > > > >> >> >>> > (I
> > > > >> >> >>> > > > > > haven't
> > > > >> >> >>> > > > > > >>> > >> started
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> looking at it yet)
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> - Rust : I remember Andrew
> > > > >> mentioned
> > > > >> >> that
> > > > >> >> >>> > this
> > > > >> >> >>> > > > work
> > > > >> >> >>> > > > > > is
> > > > >> >> >>> > > > > > >>> also
> > > > >> >> >>> > > > > > >>> > >> in
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> progress (So 4 languages!)
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> *Arrow C++ implementation *
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> The PR is out and was also used
> > by
> > > > >> >> Antoine to
> > > > >> >> >>> > > report
> > > > >> >> >>> > > > > the
> > > > >> >> >>> > > > > > >>> > numbers
> > > > >> >> >>> > > > > > >>> > >> as
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> reported here. Micah and
> > Konstantin
> > > > >> have
> > > > >> >> >>> given 1
> > > > >> >> >>> > > > round
> > > > >> >> >>> > > > > of
> > > > >> >> >>> > > > > > >>> > >> feedback
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> and I'm addressing them today.
> > > Please
> > > > >> note
> > > > >> >> >>> that
> > > > >> >> >>> > the
> > > > >> >> >>> > > > > > default
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> optimization flag for compiling
> > is
> > > O2
> > > > >> and
> > > > >> >> not
> > > > >> >> >>> > Q3. I
> > > > >> >> >>> > > > got
> > > > >> >> >>> > > > > > >>> around
> > > > >> >> >>> > > > > > >>> > >> 70%
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> performance improvement in the
> > > > decoding
> > > > >> >> speed
> > > > >> >> >>> > when
> > > > >> >> >>> > > > > using
> > > > >> >> >>> > > > > > >>> the O3
> > > > >> >> >>> > > > > > >>> > >> > flag.
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> *Parqet-MR Java implementation
> > > > (working
> > > > >> >> with
> > > > >> >> >>> > Vinoo
> > > > >> >> >>> > > > and
> > > > >> >> >>> > > > > > >>> Julien)
> > > > >> >> >>> > > > > > >>> > >> and
> > > > >> >> >>> > > > > > >>> > >> > **Cross
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> Language testing*
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> Let me know if you have any
> > > > >> questions
> > > > >> >> or
> > > > >> >> >>> > > > feedback.
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> Now pasting some performance
> > > numbers
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> Table 1: C++ ALP Double
> > Decode —
> > > > >> >> Spotify
> > > > >> >> >>> > Columns
> > > > >> >> >>> > > > > > >>> (Graviton
> > > > >> >> >>> > > > > > >>> > 3,
> > > > >> >> >>> > > > > > >>> > >> ARM
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> Neoverse V1)
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>>
> > > > >> >> >>>
> ┌──────────────────┬──────────────┬──────────────┬─────────┐
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ Column │ -O2
> > (MB/s)
> > > > │
> > > > >> >> -O3
> > > > >> >> >>> > > (MB/s) │
> > > > >> >> >>> > > > > > >>> Speedup │
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>>
> > > > >> >> >>>
> ├──────────────────┼──────────────┼──────────────┼─────────┤
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ valence │
> 3,155
> > > > │
> > > > >> >> >>> 5,523
> > > > >> >> >>> > > │
> > > > >> >> >>> > > > > > >>> 1.75x │
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ danceability │
> 3,233
> > > > │
> > > > >> >> >>> 5,685
> > > > >> >> >>> > > │
> > > > >> >> >>> > > > > > >>> 1.76x │
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ energy │
> 3,197
> > > > │
> > > > >> >> >>> 5,652
> > > > >> >> >>> > > │
> > > > >> >> >>> > > > > > >>> 1.77x │
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> │ loudness │
> 3,186
> > > > │
> > > > >> >> >>> 5,473
> > > > >> >> >>> > > │
> > > > >> >> >>> > > > > > >>> 1.72x │
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>>
> > > > >> >> >>>
> └──────────────────┴──────────────┴──────────────┴─────────┘
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM
> > > > PRATEEK
> > > > >> >> GAUR
> > > > >> >> >>> <
> > > > >> >> >>> > > > > > >>> > prtkgaur@gmail.com
> > > > >> >> >>> > > > > > >>> > >> >
> > > > >> >> >>> > > > > > >>> > >> > >>>>>> wrote:
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>> @Micah Kornfield <
> > > > >> emkornfield@gmail.com>
> > > > >> >> :
> > > > >> >> >>> Got
> > > > >> >> >>> > > it.
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>> @Andrew Lamb <
> > > > andrewlamb11@gmail.com>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> Do you think it would be good
> > to
> > > > >> start
> > > > >> >> >>> moving
> > > > >> >> >>> > the
> > > > >> >> >>> > > > > spec
> > > > >> >> >>> > > > > > >>> > >> development
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> into
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> markdown format, in
> preparation
> > > for
> > > > >> >> >>> finalizing
> > > > >> >> >>> > > it?
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>> Yes I'll update the numbers
> for
> > > some
> > > > >> of
> > > > >> >> the
> > > > >> >> >>> > > > examples I
> > > > >> >> >>> > > > > > >>> have in
> > > > >> >> >>> > > > > > >>> > >> the
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>> spec based
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>> on the updated header size.
> Then
> > > we
> > > > >> >> should
> > > > >> >> >>> be
> > > > >> >> >>> > good
> > > > >> >> >>> > > > to
> > > > >> >> >>> > > > > go
> > > > >> >> >>> > > > > > >>> for
> > > > >> >> >>> > > > > > >>> > the
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>> markdown format.
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>> Thanks everyone!
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> Andrew
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at
> 7:28 PM
> > > > >> PRATEEK
> > > > >> >> >>> GAUR <
> > > > >> >> >>> > > > > > >>> > >> prtkgaur@gmail.com>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> wrote:
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Hi team,
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 1) Andrew
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for working on
> > test
> > > > >> files.
> > > > >> >> >>> My PR
> > > > >> >> >>> > > did
> > > > >> >> >>> > > > > add
> > > > >> >> >>> > > > > > >>> all
> > > > >> >> >>> > > > > > >>> > the
> > > > >> >> >>> > > > > > >>> > >> > test
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> files I
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> used to benchmark on
> > > datasets.
> > > > >> >> Maybe
> > > > >> >> >>> we
> > > > >> >> >>> > can
> > > > >> >> >>> > > > club
> > > > >> >> >>> > > > > > it
> > > > >> >> >>> > > > > > >>> > >> together.
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> WIll also
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> aid
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> cross language testing
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Kosta Tarasov working
> > on
> > > > Rust
> > > > >> >> >>> > > > implementation.
> > > > >> >> >>> > > > > > >>> This is
> > > > >> >> >>> > > > > > >>> > >> > great.
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> Thanks
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 2) Antoine
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks a lot for
> > reporting
> > > > the
> > > > >> >> >>> numbers
> > > > >> >> >>> > on
> > > > >> >> >>> > > > AMD.
> > > > >> >> >>> > > > > > >>> Looks
> > > > >> >> >>> > > > > > >>> > >> like
> > > > >> >> >>> > > > > > >>> > >> > you
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> are
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> getting 8X the decoding
> > > > >> >> performance of
> > > > >> >> >>> > BSS.
> > > > >> >> >>> > > > This
> > > > >> >> >>> > > > > > is
> > > > >> >> >>> > > > > > >>> > >> > amazing!!.
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for
> acknowledging
> > > the
> > > > >> >> >>> sampling
> > > > >> >> >>> > > > design.
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - I agree with you on
> > > > >> Fastlanes. In
> > > > >> >> >>> some
> > > > >> >> >>> > > crude
> > > > >> >> >>> > > > > > >>> > >> experiments I
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> didn't get
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> a good perf benefit from
> > it
> > > on
> > > > >> >> >>> Graviton3
> > > > >> >> >>> > > (but
> > > > >> >> >>> > > > > > maybe
> > > > >> >> >>> > > > > > >>> > there
> > > > >> >> >>> > > > > > >>> > >> was
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> something
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> wrong with my
> > > implementation).
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Locking the 16bit
> > > exception
> > > > >> >> >>> encoding for
> > > > >> >> >>> > > the
> > > > >> >> >>> > > > > > spec
> > > > >> >> >>> > > > > > >>> in
> > > > >> >> >>> > > > > > >>> > >> this
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> case.
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Awesome I think we
> have
> > > > solved
> > > > >> >> for
> > > > >> >> >>> all
> > > > >> >> >>> > > open
> > > > >> >> >>> > > > > > >>> questions
> > > > >> >> >>> > > > > > >>> > >> minus
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> the
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> version byte :). (will
> get
> > > > back
> > > > >> on
> > > > >> >> >>> this
> > > > >> >> >>> > > soon)
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 3) Micah
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - FastLanes : The
> current
> > > spec
> > > > >> does
> > > > >> >> >>> allow
> > > > >> >> >>> > > for
> > > > >> >> >>> > > > > > using
> > > > >> >> >>> > > > > > >>> > >> FastLane
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> with the
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> configurable enum value
> > for
> > > > >> >> layout. We
> > > > >> >> >>> > > should
> > > > >> >> >>> > > > be
> > > > >> >> >>> > > > > > >>> able to
> > > > >> >> >>> > > > > > >>> > >> > inject
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> any
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> layout
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> in the current design.
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Working on resolving all
> > > remaining
> > > > >> open
> > > > >> >> >>> > comments
> > > > >> >> >>> > > > on
> > > > >> >> >>> > > > > > the
> > > > >> >> >>> > > > > > >>> spec
> > > > >> >> >>> > > > > > >>> > >> this
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> week.
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Best
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Prateek
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at
> > 3:37 AM
> > > > >> Steve
> > > > >> >> >>> > Loughran <
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> stevel@cloudera.com>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> wrote:
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at
> 18:12,
> > > > Micah
> > > > >> >> >>> Kornfield
> > > > >> >> >>> > <
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> emkornfield@gmail.com>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> wrote:
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> It looks like the actual
> > issue
> > > > >> >> >>> described for
> > > > >> >> >>> > > ORC
> > > > >> >> >>> > > > > in
> > > > >> >> >>> > > > > > >>> the
> > > > >> >> >>> > > > > > >>> > >> paper
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> is that
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> it
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> has multiple sub-encodings
> > in
> > > a
> > > > >> >> batch.
> > > > >> >> >>> This
> > > > >> >> >>> > > is
> > > > >> >> >>> > > > > > >>> different
> > > > >> >> >>> > > > > > >>> > >> then
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> the
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> design
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> proposed here where there
> is
> > > > still
> > > > >> >> fixed
> > > > >> >> >>> > > > encoding
> > > > >> >> >>> > > > > > per
> > > > >> >> >>> > > > > > >>> page
> > > > >> >> >>> > > > > > >>> > >> in
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> parquet.
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> Given reasonably sized
> > pages I
> > > > >> don't
> > > > >> >> >>> think
> > > > >> >> >>> > > > branch
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> misprediction should
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> be a
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> big issue for new
> encodings.
> > > I
> > > > >> agree
> > > > >> >> >>> that
> > > > >> >> >>> > we
> > > > >> >> >>> > > > > should
> > > > >> >> >>> > > > > > >>> be
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>> conservative in
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> general for adding new
> > > > encodings.
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> +1
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > > >> >> >>> > > > > > >>> > >> > >
> > > > >> >> >>> > > > > > >>> > >> >
> > > > >> >> >>> > > > > > >>> > >> >
> > > > >> >> >>> > > > > > >>> > >> >
> > > > >> >> >>> > > > > > >>> > >>
> > > > >> >> >>> > > > > > >>> > >
> > > > >> >> >>> > > > > > >>> >
> > > > >> >> >>> > > > > > >>>
> > > > >> >> >>> > > > > > >>
> > > > >> >> >>> > > > > >
> > > > >> >> >>> > > > >
> > > > >> >> >>> > > >
> > > > >> >> >>> > >
> > > > >> >> >>> >
> > > > >> >> >>>
> > > > >> >> >>
> > > > >> >>
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by PRATEEK GAUR <pr...@gmail.com>.
[Rust] Awesome thanks Andrew, Kostas and Devin.
[Java] Thanks Gang for the feedback on Java PR.
[C++] Thanks Gang for another round of feedback on the C++ one.
[Go] Arnav and I will be working on the golang impl this week.
This is great progress all around.
On Sat, Sep 5, 2026 at 1:26 PM Vinoo Ganesh <vi...@gmail.com> wrote:
> Thanks all - I'm addressing Gang's comments on
> https://github.com/apache/parquet-java/pull/3397 now, so hopefully well
> also be in a good state for merging the java implementation soon.
>
> Thanks,
> Vinoo Ganesh | vinoo.ganesh@gmail.com
>
> <Vi...@gmail.com>
>
>
> On Sat, Sep 5, 2026 at 8:56 AM Andrew Lamb <an...@gmail.com> wrote:
>
> > We just merged the Rust implementation[1] from Kosta and Devan. With the
> > imminent release of Parquet Format 2.14.0 I think we'll be well on our
> way.
> >
> > Thanks again for pushing this along,
> > Andrew
> >
> > [1]: https://github.com/apache/arrow-rs/pull/9372
> >
> > On Fri, Sep 4, 2026 at 5:42 PM PRATEEK GAUR <pr...@gmail.com> wrote:
> >
> > > Hi team,
> > >
> > > Thanks Micah for multiple rounds of review and thanks Gang for taking a
> > > look at the PR and giving feedback. I've addressed all comments in the
> > PR.
> > > Looking forward to the next round of feedback (or approval). Would like
> > to
> > > get it in soon.
> > >
> > > https://github.com/apache/arrow/pull/48345/changes
> > >
> > > With this done and after talking to Arnav, I plan to spend some time on
> > the
> > > ALP implementation in the arrow-go repository helping spread the
> support
> > to
> > > more languages.
> > >
> > > Best
> > > Prateek
> > >
> > > On Sun, Aug 2, 2026 at 10:42 PM PRATEEK GAUR <pr...@gmail.com>
> wrote:
> > >
> > > > Thanks Divjot for bringing it up.
> > > >
> > > > I was under the same impression as Russell and Andrew that we'd have
> it
> > > > flag gated.
> > > > With the new versioning proposal 'preview' might be a good tag for
> it.
> > > >
> > > > Best
> > > > Prateek
> > > >
> > > > On Sat, Aug 1, 2026 at 12:39 AM Andrew Lamb <an...@gmail.com>
> > > > wrote:
> > > >
> > > >> Div: Here is a proposal to mark ALP in Preview: [1] (using a comment
> > in
> > > >> the
> > > >> spec section).
> > > >>
> > > >> Russel: That is my understanding as well.
> > > >>
> > > >> Hopefully we are able to formalize the versioning process shortly so
> > we
> > > >> can
> > > >> formalize the notion of "preview"
> > > >>
> > > >> Andrew
> > > >>
> > > >>
> > > >> [1]: https://github.com/apache/parquet-format/pull/604
> > > >>
> > > >> On Fri, Jul 31, 2026 at 11:15 AM Russell Spitzer <
> > > >> russell.spitzer@gmail.com>
> > > >> wrote:
> > > >>
> > > >> > I thought we had agreed it would be merged, and features like this
> > > would
> > > >> > be feature flag gated. So a V-Old writer could always opt in to
> > using
> > > >> this
> > > >> > encoding but that must be a conscious effort if you are not using
> > > VNext
> > > >> of
> > > >> > Parquet. If you are on V-Next the writer can choose to use ALP
> > > whenever
> > > >> it
> > > >> > likes without user intervention.
> > > >> >
> > > >> > On Fri, Jul 31, 2026 at 2:20 AM Divjot Arora via dev <
> > > >> > dev@parquet.apache.org> wrote:
> > > >> >
> > > >> >> Yes, that’s my understanding as well. However, as you noted, the
> > > >> >> versioning
> > > >> >> discussion is still in early stages and hasn’t prescribed
> concrete
> > > >> >> mechanisms yet. If ALP goes into parquet-format 2.14.0, there’s
> no
> > > >> >> indication that it’s in preview or that writers shouldn’t use it
> by
> > > >> >> default
> > > >> >> yet. Given that we’ve held off on merging other incompatible
> > changes
> > > >> (e.g
> > > >> >> removal of path_in_schema) until the versioning discussion
> closes,
> > my
> > > >> >> preference would be to leave ALP un-merged for the time being as
> > > well.
> > > >> >>
> > > >> >> — Div
> > > >> >>
> > > >> >> On Fri, Jul 31, 2026 at 08:59 Andrew Lamb <
> andrewlamb11@gmail.com>
> > > >> wrote:
> > > >> >>
> > > >> >> > After re-reading Julien's Versioning Proposal, my understanding
> > is
> > > >> that
> > > >> >> if
> > > >> >> > we adopt that system, we will somehow mark ALP in "Preview" in
> > > >> >> > parquet-format for release Version 3.
> > > >> >> >
> > > >> >> > The versioning document doesn't seem to proscribe the
> particular
> > > >> method
> > > >> >> > for annotating "preview" features -- I think clearly marking
> them
> > > as
> > > >> >> such
> > > >> >> > in parquet.thrift as well as reflecting that on the
> > > >> >> > https://parquet.apache.org/docs/file-format/versions/ page
> > somehow
> > > >> >> would
> > > >> >> > be adequate.
> > > >> >> >
> > > >> >> > Andrew
> > > >> >> >
> > > >> >> > On Thu, Jul 30, 2026 at 11:50 AM Andrew Lamb <
> > > andrewlamb11@gmail.com
> > > >> >
> > > >> >> > wrote:
> > > >> >> >
> > > >> >> >> In my mind ALP is in the same position as other encodings that
> > > were
> > > >> >> added
> > > >> >> >> recently[1] (such as BYTE_STREAM_SPLIT for non floating point
> > > types)
> > > >> >> >>
> > > >> >> >> That is to say it will be available in the latest open source
> > > >> writers
> > > >> >> and
> > > >> >> >> readers, but realistically not widely used in the ecosystem
> > where
> > > >> >> >> compatibility is a concern (a large part)
> > > >> >> >>
> > > >> >> >> I view Julian's work[2][3] to gather consensus on how to name
> > > >> >> >> combinations of features (e.g. "versions") as the mechanism by
> > > >> which we
> > > >> >> >> will accelerate the adoption of this change (along with other
> > > >> forward
> > > >> >> >> incompatible changes) across the ecosystem.
> > > >> >> >>
> > > >> >> >> > Will we exclude this change when cutting parquet-format
> > v2.14.0?
> > > >> >> >>
> > > >> >> >> No I don't think so -- I would expect it to go out in the next
> > > >> >> >> parquet-format version as do other changes in the spec.
> > > >> >> >>
> > > >> >> >> Andrew
> > > >> >> >>
> > > >> >> >> [1]: https://parquet.apache.org/docs/file-format/versions/
> > > >> >> >> [2]:
> > > >> https://lists.apache.org/thread/xmnj8h0h8ozmrgox5tydhhs7yz7q2hss
> > > >> >> >> [3]:
> > > >> https://lists.apache.org/thread/qlf8lg90gqq41lllyy9mk2f0skqvftqj
> > > >> >> >>
> > > >> >> >>
> > > >> >> >> On Thu, Jul 30, 2026 at 3:51 AM Divjot Arora via dev <
> > > >> >> >> dev@parquet.apache.org> wrote:
> > > >> >> >>
> > > >> >> >>> Hi Andrew,
> > > >> >> >>>
> > > >> >> >>> How does this interact with the ongoing versioning
> discussion?
> > > >> Given
> > > >> >> that
> > > >> >> >>> new encodings are not forward compatible, I would assume ALP
> is
> > > >> >> included
> > > >> >> >>> in
> > > >> >> >>> Parquet V<next> (V3?), but it's not clear to me how that
> works
> > if
> > > >> it's
> > > >> >> >>> merged to the parquet-format master branch now. Will we
> exclude
> > > >> this
> > > >> >> >>> change
> > > >> >> >>> when cutting parquet-format v2.14.0?
> > > >> >> >>>
> > > >> >> >>> -- Div
> > > >> >> >>>
> > > >> >> >>> On Thu, Jul 30, 2026 at 8:51 AM Andrew Lamb <
> > > >> andrewlamb11@gmail.com>
> > > >> >> >>> wrote:
> > > >> >> >>>
> > > >> >> >>> > Hello,
> > > >> >> >>> >
> > > >> >> >>> > As an update, thanks to the epic work from many
> contributors,
> > > >> >> >>> especially
> > > >> >> >>> > Prateek, I just merged the parquet-format change for
> ALP[1].
> > We
> > > >> >> plan to
> > > >> >> >>> > make additional editorial The remaining items I know of are
> > > >> merging
> > > >> >> the
> > > >> >> >>> > examples from parquet-testing[2], and then of course the
> > > various
> > > >> >> >>> > implementations
> > > >> >> >>> >
> > > >> >> >>> > Thanks again everyone. This is a great step forward
> > > >> >> >>> > Andrew
> > > >> >> >>> >
> > > >> >> >>> > [1]: https://github.com/apache/parquet-format/pull/557
> > > >> >> >>> > [2]: https://github.com/apache/parquet-testing/pull/100
> > > >> >> >>> >
> > > >> >> >>> > On Tue, Jul 21, 2026 at 4:39 PM Julien Le Dem <
> > > julien@apache.org
> > > >> >
> > > >> >> >>> wrote:
> > > >> >> >>> >
> > > >> >> >>> > > It was also in my spam folder... :(
> > > >> >> >>> > > I've seen a bunch of apache list emails go to my spam
> > > recently.
> > > >> >> >>> > >
> > > >> >> >>> > > On Tue, Jul 21, 2026 at 10:58 AM Russell Spitzer <
> > > >> >> >>> > > russell.spitzer@gmail.com>
> > > >> >> >>> > > wrote:
> > > >> >> >>> > >
> > > >> >> >>> > > > Yes, my hope was that my vote would kick it out of spam
> > but
> > > >> we
> > > >> >> >>> > definitely
> > > >> >> >>> > > > had two other votes that also went to spam for me.
> > > >> >> >>> > > >
> > > >> >> >>> > > > For anyone looking the thread is here
> > > >> >> >>> > > >
> > > >> >> https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt
> > > >> >> >>> > > >
> > > >> >> >>> > > > On Tue, Jul 21, 2026 at 10:57 AM Micah Kornfield <
> > > >> >> >>> > emkornfield@gmail.com>
> > > >> >> >>> > > > wrote:
> > > >> >> >>> > > >
> > > >> >> >>> > > > > I've seen other's votes not go to spam, so hopefully
> > > people
> > > >> >> will
> > > >> >> >>> see
> > > >> >> >>> > it
> > > >> >> >>> > > > in
> > > >> >> >>> > > > > there inboxes (or see this thread and look in there
> > > spam).
> > > >> >> >>> > > > >
> > > >> >> >>> > > > > On Tue, Jul 21, 2026 at 10:52 AM PRATEEK GAUR <
> > > >> >> >>> prtkgaur@gmail.com>
> > > >> >> >>> > > > wrote:
> > > >> >> >>> > > > >
> > > >> >> >>> > > > > > Okay,
> > > >> >> >>> > > > > >
> > > >> >> >>> > > > > > I sent out the voting email yesterday and so far
> I've
> > > >> heard
> > > >> >> >>> from 3
> > > >> >> >>> > > > people
> > > >> >> >>> > > > > > that it ended up in their "spam folder". Dev VPN to
> > be
> > > >> >> blamed
> > > >> >> >>> for
> > > >> >> >>> > > it?.
> > > >> >> >>> > > > > >
> > > >> >> >>> > > > > > What's the protocol here?
> > > >> >> >>> > > > > > I let people hunt for the email in their spam
> folder
> > or
> > > >> send
> > > >> >> >>> out
> > > >> >> >>> > > > another
> > > >> >> >>> > > > > > one?
> > > >> >> >>> > > > > >
> > > >> >> >>> > > > > > Best
> > > >> >> >>> > > > > > Prateek
> > > >> >> >>> > > > > >
> > > >> >> >>> > > > > > On Sun, Jul 19, 2026 at 10:47 PM PRATEEK GAUR <
> > > >> >> >>> prtkgaur@gmail.com>
> > > >> >> >>> > > > > wrote:
> > > >> >> >>> > > > > >
> > > >> >> >>> > > > > > > Hi team,
> > > >> >> >>> > > > > > >
> > > >> >> >>> > > > > > > Vinoo has added the tests which were requested in
> > the
> > > >> last
> > > >> >> >>> round
> > > >> >> >>> > of
> > > >> >> >>> > > > > > review.
> > > >> >> >>> > > > > > > In his own words.
> > > >> >> >>> > > > > > >
> > > >> >> >>> > > > > > > "
> > > >> >> >>> > > > > > > The extreme value tests Micah wanted are done and
> > > >> pushed.
> > > >> >> I
> > > >> >> >>> added
> > > >> >> >>> > > > > > coverage
> > > >> >> >>> > > > > > > for values that need the full FOR bit width after
> > the
> > > >> >> frame
> > > >> >> >>> of
> > > >> >> >>> > > > > reference
> > > >> >> >>> > > > > > is
> > > >> >> >>> > > > > > > applied: 64-bit (the case where the signed max
> > minus
> > > >> min
> > > >> >> >>> > > subtraction
> > > >> >> >>> > > > > > > overflows), 63-bit (the non-overflow case), and
> > > 32-bit
> > > >> for
> > > >> >> >>> > floats.
> > > >> >> >>> > > > All
> > > >> >> >>> > > > > > > lossless with zero exceptions. I also verified
> them
> > > >> >> through
> > > >> >> >>> Arrow
> > > >> >> >>> > > C++
> > > >> >> >>> > > > > ALP
> > > >> >> >>> > > > > > > reader (the local build), so the Java-written
> > extreme
> > > >> >> columns
> > > >> >> >>> > > decode
> > > >> >> >>> > > > > > > correctly in C++, bit-exact against the expected
> > > >> values."
> > > >> >> >>> > > > > > >
> > > >> >> >>> > > > > > > State of the PR (#3397):
> > > >> >> >>> > > > > > >
> > > >> >> >>> > > > > > > - Merges cleanly with master (I pulled master
> in
> > > and
> > > >> >> >>> resolved
> > > >> >> >>> > > the
> > > >> >> >>> > > > > > > conflicts).
> > > >> >> >>> > > > > > > - 41 of 44 review threads resolved. The 3
> > > remaining
> > > >> are
> > > >> >> >>> > > > intentional:
> > > >> >> >>> > > > > > > one deferred perf nit, one that stays open
> until
> > > >> >> >>> > parquet-format
> > > >> >> >>> > > > > ships
> > > >> >> >>> > > > > > ALP,
> > > >> >> >>> > > > > > >
> > > >> >> >>> > > > > > > "
> > > >> >> >>> > > > > > >
> > > >> >> >>> > > > > > > With this both the
> > > >> >> >>> > > > > > > 1) C++ PR :
> > > >> >> >>> https://github.com/apache/arrow/pull/48345/changes
> > > >> >> >>> > > > > > > 2) Java PR :
> > > >> >> >>> https://github.com/apache/parquet-java/pull/3397
> > > >> >> >>> > > > > > >
> > > >> >> >>> > > > > > > Are cross language tested and range of test
> > > scenarios.
> > > >> >> >>> > > > > > >
> > > >> >> >>> > > > > > > Best
> > > >> >> >>> > > > > > > Prateek
> > > >> >> >>> > > > > > >
> > > >> >> >>> > > > > > >
> > > >> >> >>> > > > > > > On Mon, Jul 13, 2026 at 8:38 AM PRATEEK GAUR <
> > > >> >> >>> prtkgaur@gmail.com
> > > >> >> >>> > >
> > > >> >> >>> > > > > wrote:
> > > >> >> >>> > > > > > >
> > > >> >> >>> > > > > > >> Thanks Matt,
> > > >> >> >>> > > > > > >>
> > > >> >> >>> > > > > > >> I'll be on it this week.
> > > >> >> >>> > > > > > >>
> > > >> >> >>> > > > > > >> Best
> > > >> >> >>> > > > > > >> Prateek
> > > >> >> >>> > > > > > >>
> > > >> >> >>> > > > > > >> On Mon, Jul 13, 2026 at 8:36 AM Matt Topol <
> > > >> >> >>> > > zotthewizard@gmail.com>
> > > >> >> >>> > > > > > >> wrote:
> > > >> >> >>> > > > > > >>
> > > >> >> >>> > > > > > >>> There is also the Go implementation submitted
> by
> > > >> Arnav (
> > > >> >> >>> > > > > > >>> https://github.com/apache/arrow-go/pull/704)
> > which
> > > >> is
> > > >> >> >>> waiting
> > > >> >> >>> > > for
> > > >> >> >>> > > > > > >>> updates.
> > > >> >> >>> > > > > > >>>
> > > >> >> >>> > > > > > >>> Just wanted to make sure it didn't get lost
> here.
> > > >> >> >>> > > > > > >>>
> > > >> >> >>> > > > > > >>> --Matt
> > > >> >> >>> > > > > > >>>
> > > >> >> >>> > > > > > >>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK GAUR <
> > > >> >> >>> > > prtkgaur@gmail.com>
> > > >> >> >>> > > > > > >>> wrote:
> > > >> >> >>> > > > > > >>>
> > > >> >> >>> > > > > > >>> > Hi all,
> > > >> >> >>> > > > > > >>> >
> > > >> >> >>> > > > > > >>> > I'd like to share a status update on the ALP
> > > >> encoding
> > > >> >> >>> effort
> > > >> >> >>> > > and
> > > >> >> >>> > > > > get
> > > >> >> >>> > > > > > a
> > > >> >> >>> > > > > > >>> feel
> > > >> >> >>> > > > > > >>> > from the community before starting a formal
> > vote.
> > > >> >> >>> > > > > > >>> >
> > > >> >> >>> > > > > > >>> > We now have the specification plus
> > > implementations
> > > >> in
> > > >> >> >>> both
> > > >> >> >>> > > > > languages:
> > > >> >> >>> > > > > > >>> >
> > > >> >> >>> > > > > > >>> > - Spec:
> apache/parquet-format#557 —
> > > >> GH-533
> > > >> >> >>> Add
> > > >> >> >>> > ALP
> > > >> >> >>> > > > > > encoding
> > > >> >> >>> > > > > > >>> > specification
> > > >> >> >>> > > > > > >>> > - C++ (Arrow): apache/arrow#48345 —
> > GH-48701
> > > >> >> >>> > > [C++][Parquet]
> > > >> >> >>> > > > > Add
> > > >> >> >>> > > > > > >>> ALPpd
> > > >> >> >>> > > > > > >>> > encoding
> > > >> >> >>> > > > > > >>> > - Java: apache/parquet-java#3397
> —
> > > >> Parquet
> > > >> >> >>> Java
> > > >> >> >>> > ALP
> > > >> >> >>> > > > > > >>> > Implementation (by Vinoo)
> > > >> >> >>> > > > > > >>> >
> > > >> >> >>> > > > > > >>> > Current state:
> > > >> >> >>> > > > > > >>> >
> > > >> >> >>> > > > > > >>> > - The C++ implementation has been through
> > 3-4
> > > >> >> rounds
> > > >> >> >>> of
> > > >> >> >>> > > > review.
> > > >> >> >>> > > > > > >>> > - The Java implementation has been
> tracking
> > > >> well
> > > >> >> and
> > > >> >> >>> has
> > > >> >> >>> > > had
> > > >> >> >>> > > > > > >>> initial
> > > >> >> >>> > > > > > >>> > reviews from contributors, with no
> major
> > > >> >> remaining
> > > >> >> >>> > > issues.
> > > >> >> >>> > > > > > >>> > - Cross-language compatibility tests are
> > > >> passing:
> > > >> >> the
> > > >> >> >>> > Arrow
> > > >> >> >>> > > > C++
> > > >> >> >>> > > > > > >>> decoder
> > > >> >> >>> > > > > > >>> > reads Java-written data bit-exactly
> > across
> > > >> >> ~1.56M
> > > >> >> >>> > values
> > > >> >> >>> > > > and
> > > >> >> >>> > > > > 18
> > > >> >> >>> > > > > > >>> > fixtures, covering V1 and V2 pages,
> > > multiple
> > > >> >> vector
> > > >> >> >>> > > sizes,
> > > >> >> >>> > > > > and
> > > >> >> >>> > > > > > >>> > several
> > > >> >> >>> > > > > > >>> > real datasets — zero mismatches.
> > > >> >> >>> > > > > > >>> >
> > > >> >> >>> > > > > > >>> > As raised in the java-pr we will be adding
> > > >> coverage
> > > >> >> for
> > > >> >> >>> > > extreme
> > > >> >> >>> > > > > > values
> > > >> >> >>> > > > > > >>> > (those requiring
> > > >> >> >>> > > > > > >>> > 63-64 bits after FOR is applied) before we
> > > close a
> > > >> >> vote.
> > > >> >> >>> > We'll
> > > >> >> >>> > > > aim
> > > >> >> >>> > > > > > to
> > > >> >> >>> > > > > > >>> get
> > > >> >> >>> > > > > > >>> > that
> > > >> >> >>> > > > > > >>> > done in parallel.
> > > >> >> >>> > > > > > >>> >
> > > >> >> >>> > > > > > >>> > Plan: unless there are objections, I intend
> to
> > > >> start a
> > > >> >> >>> vote
> > > >> >> >>> > at
> > > >> >> >>> > > > the
> > > >> >> >>> > > > > > end
> > > >> >> >>> > > > > > >>> of
> > > >> >> >>> > > > > > >>> > this week/early
> > > >> >> >>> > > > > > >>> > next week.
> > > >> >> >>> > > > > > >>> >
> > > >> >> >>> > > > > > >>> > Thanks
> > > >> >> >>> > > > > > >>> > Prateek
> > > >> >> >>> > > > > > >>> >
> > > >> >> >>> > > > > > >>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <
> > > >> >> >>> > > prtkgaur@gmail.com>
> > > >> >> >>> > > > > > >>> wrote:
> > > >> >> >>> > > > > > >>> >
> > > >> >> >>> > > > > > >>> > > Hi Team,
> > > >> >> >>> > > > > > >>> > >
> > > >> >> >>> > > > > > >>> > > Just wanted to provide some updates on ALP.
> > > >> >> >>> > > > > > >>> > > Micah and I have done 3-4 rounds of review
> > for
> > > >> the
> > > >> >> c++
> > > >> >> >>> PR :
> > > >> >> >>> > > > > > >>> > > https://github.com/apache/arrow/pull/48345
> > > >> >> >>> > > > > > >>> > >
> > > >> >> >>> > > > > > >>> > > For the Java implementation Vinoo has been
> > > >> working
> > > >> >> on
> > > >> >> >>> > > following
> > > >> >> >>> > > > > PR
> > > >> >> >>> > > > > > :
> > > >> >> >>> > > > > > >>> > >
> > > https://github.com/apache/parquet-java/pull/3397
> > > >> >> >>> > > > > > >>> > >
> > > >> >> >>> > > > > > >>> > > Best
> > > >> >> >>> > > > > > >>> > > Prateek
> > > >> >> >>> > > > > > >>> > >
> > > >> >> >>> > > > > > >>> > > On Tue, May 5, 2026 at 1:50 PM Micah
> > Kornfield
> > > <
> > > >> >> >>> > > > > > >>> emkornfield@gmail.com>
> > > >> >> >>> > > > > > >>> > > wrote:
> > > >> >> >>> > > > > > >>> > >
> > > >> >> >>> > > > > > >>> > >> Hi Antoine,
> > > >> >> >>> > > > > > >>> > >>
> > > >> >> >>> > > > > > >>> > >> > Apologies if the question was already
> > asked,
> > > >> but
> > > >> >> >>> should
> > > >> >> >>> > we
> > > >> >> >>> > > > > care
> > > >> >> >>> > > > > > >>> about
> > > >> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more
> > > >> >> efficient
> > > >> >> >>> than
> > > >> >> >>> > > > > FLOAT16
> > > >> >> >>> > > > > > +
> > > >> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> > > >> >> >>> > > > > > >>> > >>
> > > >> >> >>> > > > > > >>> > >> It was. We thought we could defer it for
> > the
> > > >> >> >>> following
> > > >> >> >>> > > > reasons:
> > > >> >> >>> > > > > > >>> > >> 1. It's not clear there are a lot of easy
> > > >> >> reference
> > > >> >> >>> > > datasets
> > > >> >> >>> > > > to
> > > >> >> >>> > > > > > >>> test
> > > >> >> >>> > > > > > >>> > its
> > > >> >> >>> > > > > > >>> > >> effectiveness. It does look like there
> > might
> > > be
> > > >> >> one
> > > >> >> >>> or
> > > >> >> >>> > two
> > > >> >> >>> > > on
> > > >> >> >>> > > > > > >>> > huggingface
> > > >> >> >>> > > > > > >>> > >> (e.g.
> > > >> >> https://huggingface.co/datasets/kikitora/curdie
> > > >> >> >>> ).
> > > >> >> >>> > > > > > >>> > >> 2. It seemed likely that float 16 was
> more
> > > >> likely
> > > >> >> >>> used
> > > >> >> >>> > for
> > > >> >> >>> > > > > values
> > > >> >> >>> > > > > > >>> that
> > > >> >> >>> > > > > > >>> > >> were less likely to reduce to decimal
> > values.
> > > >> >> >>> > > > > > >>> > >> 3. It could be added as an extension
> later
> > if
> > > >> >> needed.
> > > >> >> >>> > > > > > >>> > >>
> > > >> >> >>> > > > > > >>> > >> Cheers,
> > > >> >> >>> > > > > > >>> > >> Micah
> > > >> >> >>> > > > > > >>> > >>
> > > >> >> >>> > > > > > >>> > >>
> > > >> >> >>> > > > > > >>> > >> On Tue, May 5, 2026 at 1:40 PM Antoine
> > Pitrou
> > > <
> > > >> >> >>> > > > > antoine@python.org
> > > >> >> >>> > > > > > >
> > > >> >> >>> > > > > > >>> > wrote:
> > > >> >> >>> > > > > > >>> > >>
> > > >> >> >>> > > > > > >>> > >> >
> > > >> >> >>> > > > > > >>> > >> > Hello,
> > > >> >> >>> > > > > > >>> > >> >
> > > >> >> >>> > > > > > >>> > >> > Apologies if the question was already
> > asked,
> > > >> but
> > > >> >> >>> should
> > > >> >> >>> > we
> > > >> >> >>> > > > > care
> > > >> >> >>> > > > > > >>> about
> > > >> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more
> > > >> >> efficient
> > > >> >> >>> than
> > > >> >> >>> > > > > FLOAT16
> > > >> >> >>> > > > > > +
> > > >> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> > > >> >> >>> > > > > > >>> > >> >
> > > >> >> >>> > > > > > >>> > >> > Regards
> > > >> >> >>> > > > > > >>> > >> >
> > > >> >> >>> > > > > > >>> > >> > Antoine.
> > > >> >> >>> > > > > > >>> > >> >
> > > >> >> >>> > > > > > >>> > >> >
> > > >> >> >>> > > > > > >>> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a
> > écrit
> > > :
> > > >> >> >>> > > > > > >>> > >> > > Thanks Andrew and Micah for review
> > > feedback
> > > >> on
> > > >> >> >>> the two
> > > >> >> >>> > > > PR's
> > > >> >> >>> > > > > > >>> > >> > > 1) (c++ arrow repo)
> > > >> >> >>> > > > > > >>> > >>
> > > >> https://github.com/apache/arrow/pull/48345/changes
> > > >> >> >>> > > > > > >>> > >> > > 2) (parquet-format repo)
> > > >> >> >>> > > > > > >>> > >> >
> > > >> >> https://github.com/apache/parquet-format/pull/557
> > > >> >> >>> > > > > > >>> > >> > >
> > > >> >> >>> > > > > > >>> > >> > > I have addressed all (unless I missed
> > > >> >> something)
> > > >> >> >>> > > comments
> > > >> >> >>> > > > on
> > > >> >> >>> > > > > > >>> the two
> > > >> >> >>> > > > > > >>> > >> > PR's.
> > > >> >> >>> > > > > > >>> > >> > >
> > > >> >> >>> > > > > > >>> > >> > > Best
> > > >> >> >>> > > > > > >>> > >> > > Prateek
> > > >> >> >>> > > > > > >>> > >> > >
> > > >> >> >>> > > > > > >>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM
> PRATEEK
> > > >> GAUR <
> > > >> >> >>> > > > > > >>> prtkgaur@gmail.com>
> > > >> >> >>> > > > > > >>> > >> wrote:
> > > >> >> >>> > > > > > >>> > >> > >
> > > >> >> >>> > > > > > >>> > >> > >> Thanks Andrew and Micah.
> > > >> >> >>> > > > > > >>> > >> > >>
> > > >> >> >>> > > > > > >>> > >> > >> `fair amount of feedback on at least
> > the
> > > >> >> >>> > > implementations`
> > > >> >> >>> > > > > > >>> > >> > >> For the c++ I have already started
> > > >> addressing
> > > >> >> the
> > > >> >> >>> > > > > feedback, I
> > > >> >> >>> > > > > > >>> > should
> > > >> >> >>> > > > > > >>> > >> be
> > > >> >> >>> > > > > > >>> > >> > >> done with that Monday/Tuesday.
> > > >> >> >>> > > > > > >>> > >> > >> I think Vinoo too has been making
> good
> > > >> >> progress
> > > >> >> >>> on
> > > >> >> >>> > the
> > > >> >> >>> > > > Java
> > > >> >> >>> > > > > > >>> > >> > implementation.
> > > >> >> >>> > > > > > >>> > >> > >>
> > > >> >> >>> > > > > > >>> > >> > >> Best
> > > >> >> >>> > > > > > >>> > >> > >> Prateek
> > > >> >> >>> > > > > > >>> > >> > >>
> > > >> >> >>> > > > > > >>> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM
> Andrew
> > > >> Lamb <
> > > >> >> >>> > > > > > >>> > andrewlamb11@gmail.com
> > > >> >> >>> > > > > > >>> > >> >
> > > >> >> >>> > > > > > >>> > >> > >> wrote:
> > > >> >> >>> > > > > > >>> > >> > >>
> > > >> >> >>> > > > > > >>> > >> > >>> Got it. Thank you for the
> > clarification
> > > >> -- I
> > > >> >> >>> will
> > > >> >> >>> > try
> > > >> >> >>> > > > and
> > > >> >> >>> > > > > > look
> > > >> >> >>> > > > > > >>> > into
> > > >> >> >>> > > > > > >>> > >> the
> > > >> >> >>> > > > > > >>> > >> > >>> spec and the Rust implementation[1]
> in
> > > >> this
> > > >> >> next
> > > >> >> >>> > week
> > > >> >> >>> > > > > > >>> > >> > >>>
> > > >> >> >>> > > > > > >>> > >> > >>> [1]:
> > > >> >> >>> https://github.com/apache/arrow-rs/pull/9372
> > > >> >> >>> > > > > > >>> > >> > >>>
> > > >> >> >>> > > > > > >>> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM
> Micah
> > > >> >> >>> Kornfield <
> > > >> >> >>> > > > > > >>> > >> > emkornfield@gmail.com>
> > > >> >> >>> > > > > > >>> > >> > >>> wrote:
> > > >> >> >>> > > > > > >>> > >> > >>>
> > > >> >> >>> > > > > > >>> > >> > >>>> Hi Andrew,
> > > >> >> >>> > > > > > >>> > >> > >>>> I think there is a fair amount of
> > > >> feedback
> > > >> >> on
> > > >> >> >>> at
> > > >> >> >>> > > least
> > > >> >> >>> > > > > the
> > > >> >> >>> > > > > > >>> > >> > >>>> implementations, typically I think
> > > we've
> > > >> >> waited
> > > >> >> >>> > till
> > > >> >> >>> > > > they
> > > >> >> >>> > > > > > are
> > > >> >> >>> > > > > > >>> > >> close to
> > > >> >> >>> > > > > > >>> > >> > >>>> mergeable before a final vote.
> > > >> Otherwise I
> > > >> >> >>> agree
> > > >> >> >>> > we
> > > >> >> >>> > > > are
> > > >> >> >>> > > > > > very
> > > >> >> >>> > > > > > >>> > >> close.
> > > >> >> >>> > > > > > >>> > >> > >>>>
> > > >> >> >>> > > > > > >>> > >> > >>>> -Micah
> > > >> >> >>> > > > > > >>> > >> > >>>>
> > > >> >> >>> > > > > > >>> > >> > >>>> On Saturday, April 25, 2026, Andrew
> > > Lamb
> > > >> <
> > > >> >> >>> > > > > > >>> andrewlamb11@gmail.com
> > > >> >> >>> > > > > > >>> > >
> > > >> >> >>> > > > > > >>> > >> > wrote:
> > > >> >> >>> > > > > > >>> > >> > >>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>> Thanks Prateek,
> > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>> I think from this content it looks
> > to
> > > me
> > > >> >> like
> > > >> >> >>> we
> > > >> >> >>> > are
> > > >> >> >>> > > > > ready
> > > >> >> >>> > > > > > >>> to
> > > >> >> >>> > > > > > >>> > >> start a
> > > >> >> >>> > > > > > >>> > >> > >>>>> vote to explicitly accept ALP into
> > > >> Parquet
> > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>> Does anyone know of a reason we
> > should
> > > >> >> >>> postpone it
> > > >> >> >>> > > for
> > > >> >> >>> > > > > > >>> longer?
> > > >> >> >>> > > > > > >>> > >> > >>>>> Perhaps someone needs some more
> time
> > > to
> > > >> >> >>> review?
> > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>> Andrew
> > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM
> > > PRATEEK
> > > >> >> GAUR <
> > > >> >> >>> > > > > > >>> > prtkgaur@gmail.com>
> > > >> >> >>> > > > > > >>> > >> > >>>>> wrote:
> > > >> >> >>> > > > > > >>> > >> > >>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>> Hi team,
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>> Hope everyone is doing well. I
> got
> > a
> > > >> >> chance
> > > >> >> >>> to
> > > >> >> >>> > work
> > > >> >> >>> > > > > > >>> through all
> > > >> >> >>> > > > > > >>> > >> the
> > > >> >> >>> > > > > > >>> > >> > >>>>>> remaining feedback and update the
> > > spec
> > > >> >> doc.
> > > >> >> >>> Here
> > > >> >> >>> > > are
> > > >> >> >>> > > > > the
> > > >> >> >>> > > > > > >>> new
> > > >> >> >>> > > > > > >>> > >> > artifacts
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>> 1) Spec document :
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> >
> > > >> >> >>> > > > > > >>> > >>
> > > >> >> >>> > > > > > >>> >
> > > >> >> >>> > > > > > >>>
> > > >> >> >>> > > > > >
> > > >> >> >>> > > > >
> > > >> >> >>> > > >
> > > >> >> >>> > >
> > > >> >> >>> >
> > > >> >> >>>
> > > >> >>
> > > >>
> > >
> >
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>> 2) Spec document in parquet
> format
> > > >> repo :
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > https://github.com/apache/parquet-format/pull/557
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>> 3) Alp implementation in arrow
> c++
> > > >> repo :
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > https://github.com/apache/arrow/pull/48345/changes
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>> 4) Alp implementation in
> > parquet-java
> > > >> >> repo :
> > > >> >> >>> Work
> > > >> >> >>> > > for
> > > >> >> >>> > > > > > >>> Vinoo and
> > > >> >> >>> > > > > > >>> > >> > Julien
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > https://github.com/apache/parquet-java/pull/3397
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>> 5) PR with test and benchmarking
> > > >> >> artifacts in
> > > >> >> >>> > > > > > >>> parquet-testing
> > > >> >> >>> > > > > > >>> > >> repo :
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > https://github.com/apache/parquet-testing/pull/100
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>> And
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>> - Go : Arnav just submitted
> an
> > in
> > > >> >> >>> progress
> > > >> >> >>> > > > > > >>> implementation
> > > >> >> >>> > > > > > >>> > in
> > > >> >> >>> > > > > > >>> > >> Go.
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> https://github.com/apache/arrow-go/pull/704
> > > >> >> >>> > (I
> > > >> >> >>> > > > > > haven't
> > > >> >> >>> > > > > > >>> > >> started
> > > >> >> >>> > > > > > >>> > >> > >>>>>> looking at it yet)
> > > >> >> >>> > > > > > >>> > >> > >>>>>> - Rust : I remember Andrew
> > > >> mentioned
> > > >> >> that
> > > >> >> >>> > this
> > > >> >> >>> > > > work
> > > >> >> >>> > > > > > is
> > > >> >> >>> > > > > > >>> also
> > > >> >> >>> > > > > > >>> > >> in
> > > >> >> >>> > > > > > >>> > >> > >>>>>> progress (So 4 languages!)
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>> *Arrow C++ implementation *
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>> The PR is out and was also used
> by
> > > >> >> Antoine to
> > > >> >> >>> > > report
> > > >> >> >>> > > > > the
> > > >> >> >>> > > > > > >>> > numbers
> > > >> >> >>> > > > > > >>> > >> as
> > > >> >> >>> > > > > > >>> > >> > >>>>>> reported here. Micah and
> Konstantin
> > > >> have
> > > >> >> >>> given 1
> > > >> >> >>> > > > round
> > > >> >> >>> > > > > of
> > > >> >> >>> > > > > > >>> > >> feedback
> > > >> >> >>> > > > > > >>> > >> > >>>>>> and I'm addressing them today.
> > Please
> > > >> note
> > > >> >> >>> that
> > > >> >> >>> > the
> > > >> >> >>> > > > > > default
> > > >> >> >>> > > > > > >>> > >> > >>>>>> optimization flag for compiling
> is
> > O2
> > > >> and
> > > >> >> not
> > > >> >> >>> > Q3. I
> > > >> >> >>> > > > got
> > > >> >> >>> > > > > > >>> around
> > > >> >> >>> > > > > > >>> > >> 70%
> > > >> >> >>> > > > > > >>> > >> > >>>>>> performance improvement in the
> > > decoding
> > > >> >> speed
> > > >> >> >>> > when
> > > >> >> >>> > > > > using
> > > >> >> >>> > > > > > >>> the O3
> > > >> >> >>> > > > > > >>> > >> > flag.
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>> *Parqet-MR Java implementation
> > > (working
> > > >> >> with
> > > >> >> >>> > Vinoo
> > > >> >> >>> > > > and
> > > >> >> >>> > > > > > >>> Julien)
> > > >> >> >>> > > > > > >>> > >> and
> > > >> >> >>> > > > > > >>> > >> > **Cross
> > > >> >> >>> > > > > > >>> > >> > >>>>>> Language testing*
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>> Let me know if you have any
> > > >> questions
> > > >> >> or
> > > >> >> >>> > > > feedback.
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>> Now pasting some performance
> > numbers
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>> Table 1: C++ ALP Double
> Decode —
> > > >> >> Spotify
> > > >> >> >>> > Columns
> > > >> >> >>> > > > > > >>> (Graviton
> > > >> >> >>> > > > > > >>> > 3,
> > > >> >> >>> > > > > > >>> > >> ARM
> > > >> >> >>> > > > > > >>> > >> > >>>>>> Neoverse V1)
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>>
> > > >> >> >>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>> │ Column │ -O2
> (MB/s)
> > > │
> > > >> >> -O3
> > > >> >> >>> > > (MB/s) │
> > > >> >> >>> > > > > > >>> Speedup │
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>>
> > > >> >> >>> ├──────────────────┼──────────────┼──────────────┼─────────┤
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>> │ valence │ 3,155
> > > │
> > > >> >> >>> 5,523
> > > >> >> >>> > > │
> > > >> >> >>> > > > > > >>> 1.75x │
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>> │ danceability │ 3,233
> > > │
> > > >> >> >>> 5,685
> > > >> >> >>> > > │
> > > >> >> >>> > > > > > >>> 1.76x │
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>> │ energy │ 3,197
> > > │
> > > >> >> >>> 5,652
> > > >> >> >>> > > │
> > > >> >> >>> > > > > > >>> 1.77x │
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>> │ loudness │ 3,186
> > > │
> > > >> >> >>> 5,473
> > > >> >> >>> > > │
> > > >> >> >>> > > > > > >>> 1.72x │
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>>
> > > >> >> >>> └──────────────────┴──────────────┴──────────────┴─────────┘
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM
> > > PRATEEK
> > > >> >> GAUR
> > > >> >> >>> <
> > > >> >> >>> > > > > > >>> > prtkgaur@gmail.com
> > > >> >> >>> > > > > > >>> > >> >
> > > >> >> >>> > > > > > >>> > >> > >>>>>> wrote:
> > > >> >> >>> > > > > > >>> > >> > >>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>> @Micah Kornfield <
> > > >> emkornfield@gmail.com>
> > > >> >> :
> > > >> >> >>> Got
> > > >> >> >>> > > it.
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>> @Andrew Lamb <
> > > andrewlamb11@gmail.com>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> Do you think it would be good
> to
> > > >> start
> > > >> >> >>> moving
> > > >> >> >>> > the
> > > >> >> >>> > > > > spec
> > > >> >> >>> > > > > > >>> > >> development
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> into
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> markdown format, in preparation
> > for
> > > >> >> >>> finalizing
> > > >> >> >>> > > it?
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>> Yes I'll update the numbers for
> > some
> > > >> of
> > > >> >> the
> > > >> >> >>> > > > examples I
> > > >> >> >>> > > > > > >>> have in
> > > >> >> >>> > > > > > >>> > >> the
> > > >> >> >>> > > > > > >>> > >> > >>>>>>> spec based
> > > >> >> >>> > > > > > >>> > >> > >>>>>>> on the updated header size. Then
> > we
> > > >> >> should
> > > >> >> >>> be
> > > >> >> >>> > good
> > > >> >> >>> > > > to
> > > >> >> >>> > > > > go
> > > >> >> >>> > > > > > >>> for
> > > >> >> >>> > > > > > >>> > the
> > > >> >> >>> > > > > > >>> > >> > >>>>>>> markdown format.
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>> Thanks everyone!
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> Andrew
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM
> > > >> PRATEEK
> > > >> >> >>> GAUR <
> > > >> >> >>> > > > > > >>> > >> prtkgaur@gmail.com>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> wrote:
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Hi team,
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 1) Andrew
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for working on
> test
> > > >> files.
> > > >> >> >>> My PR
> > > >> >> >>> > > did
> > > >> >> >>> > > > > add
> > > >> >> >>> > > > > > >>> all
> > > >> >> >>> > > > > > >>> > the
> > > >> >> >>> > > > > > >>> > >> > test
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> files I
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> used to benchmark on
> > datasets.
> > > >> >> Maybe
> > > >> >> >>> we
> > > >> >> >>> > can
> > > >> >> >>> > > > club
> > > >> >> >>> > > > > > it
> > > >> >> >>> > > > > > >>> > >> together.
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> WIll also
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> aid
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> cross language testing
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Kosta Tarasov working
> on
> > > Rust
> > > >> >> >>> > > > implementation.
> > > >> >> >>> > > > > > >>> This is
> > > >> >> >>> > > > > > >>> > >> > great.
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> Thanks
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 2) Antoine
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks a lot for
> reporting
> > > the
> > > >> >> >>> numbers
> > > >> >> >>> > on
> > > >> >> >>> > > > AMD.
> > > >> >> >>> > > > > > >>> Looks
> > > >> >> >>> > > > > > >>> > >> like
> > > >> >> >>> > > > > > >>> > >> > you
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> are
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> getting 8X the decoding
> > > >> >> performance of
> > > >> >> >>> > BSS.
> > > >> >> >>> > > > This
> > > >> >> >>> > > > > > is
> > > >> >> >>> > > > > > >>> > >> > amazing!!.
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for acknowledging
> > the
> > > >> >> >>> sampling
> > > >> >> >>> > > > design.
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - I agree with you on
> > > >> Fastlanes. In
> > > >> >> >>> some
> > > >> >> >>> > > crude
> > > >> >> >>> > > > > > >>> > >> experiments I
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> didn't get
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> a good perf benefit from
> it
> > on
> > > >> >> >>> Graviton3
> > > >> >> >>> > > (but
> > > >> >> >>> > > > > > maybe
> > > >> >> >>> > > > > > >>> > there
> > > >> >> >>> > > > > > >>> > >> was
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> something
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> wrong with my
> > implementation).
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Locking the 16bit
> > exception
> > > >> >> >>> encoding for
> > > >> >> >>> > > the
> > > >> >> >>> > > > > > spec
> > > >> >> >>> > > > > > >>> in
> > > >> >> >>> > > > > > >>> > >> this
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> case.
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Awesome I think we have
> > > solved
> > > >> >> for
> > > >> >> >>> all
> > > >> >> >>> > > open
> > > >> >> >>> > > > > > >>> questions
> > > >> >> >>> > > > > > >>> > >> minus
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> the
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> version byte :). (will get
> > > back
> > > >> on
> > > >> >> >>> this
> > > >> >> >>> > > soon)
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 3) Micah
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - FastLanes : The current
> > spec
> > > >> does
> > > >> >> >>> allow
> > > >> >> >>> > > for
> > > >> >> >>> > > > > > using
> > > >> >> >>> > > > > > >>> > >> FastLane
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> with the
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> configurable enum value
> for
> > > >> >> layout. We
> > > >> >> >>> > > should
> > > >> >> >>> > > > be
> > > >> >> >>> > > > > > >>> able to
> > > >> >> >>> > > > > > >>> > >> > inject
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> any
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> layout
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> in the current design.
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Working on resolving all
> > remaining
> > > >> open
> > > >> >> >>> > comments
> > > >> >> >>> > > > on
> > > >> >> >>> > > > > > the
> > > >> >> >>> > > > > > >>> spec
> > > >> >> >>> > > > > > >>> > >> this
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> week.
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Best
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Prateek
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at
> 3:37 AM
> > > >> Steve
> > > >> >> >>> > Loughran <
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> stevel@cloudera.com>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> wrote:
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12,
> > > Micah
> > > >> >> >>> Kornfield
> > > >> >> >>> > <
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> emkornfield@gmail.com>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> wrote:
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> It looks like the actual
> issue
> > > >> >> >>> described for
> > > >> >> >>> > > ORC
> > > >> >> >>> > > > > in
> > > >> >> >>> > > > > > >>> the
> > > >> >> >>> > > > > > >>> > >> paper
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> is that
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> it
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> has multiple sub-encodings
> in
> > a
> > > >> >> batch.
> > > >> >> >>> This
> > > >> >> >>> > > is
> > > >> >> >>> > > > > > >>> different
> > > >> >> >>> > > > > > >>> > >> then
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> the
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>> design
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> proposed here where there is
> > > still
> > > >> >> fixed
> > > >> >> >>> > > > encoding
> > > >> >> >>> > > > > > per
> > > >> >> >>> > > > > > >>> page
> > > >> >> >>> > > > > > >>> > >> in
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> parquet.
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> Given reasonably sized
> pages I
> > > >> don't
> > > >> >> >>> think
> > > >> >> >>> > > > branch
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> misprediction should
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> be a
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> big issue for new encodings.
> > I
> > > >> agree
> > > >> >> >>> that
> > > >> >> >>> > we
> > > >> >> >>> > > > > should
> > > >> >> >>> > > > > > >>> be
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>> conservative in
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> general for adding new
> > > encodings.
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> +1
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > > >> >> >>> > > > > > >>> > >> > >
> > > >> >> >>> > > > > > >>> > >> >
> > > >> >> >>> > > > > > >>> > >> >
> > > >> >> >>> > > > > > >>> > >> >
> > > >> >> >>> > > > > > >>> > >>
> > > >> >> >>> > > > > > >>> > >
> > > >> >> >>> > > > > > >>> >
> > > >> >> >>> > > > > > >>>
> > > >> >> >>> > > > > > >>
> > > >> >> >>> > > > > >
> > > >> >> >>> > > > >
> > > >> >> >>> > > >
> > > >> >> >>> > >
> > > >> >> >>> >
> > > >> >> >>>
> > > >> >> >>
> > > >> >>
> > > >> >
> > > >>
> > > >
> > >
> >
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Vinoo Ganesh <vi...@gmail.com>.
Thanks all - I'm addressing Gang's comments on
https://github.com/apache/parquet-java/pull/3397 now, so hopefully well
also be in a good state for merging the java implementation soon.
Thanks,
Vinoo Ganesh | vinoo.ganesh@gmail.com
<Vi...@gmail.com>
On Sat, Sep 5, 2026 at 8:56 AM Andrew Lamb <an...@gmail.com> wrote:
> We just merged the Rust implementation[1] from Kosta and Devan. With the
> imminent release of Parquet Format 2.14.0 I think we'll be well on our way.
>
> Thanks again for pushing this along,
> Andrew
>
> [1]: https://github.com/apache/arrow-rs/pull/9372
>
> On Fri, Sep 4, 2026 at 5:42 PM PRATEEK GAUR <pr...@gmail.com> wrote:
>
> > Hi team,
> >
> > Thanks Micah for multiple rounds of review and thanks Gang for taking a
> > look at the PR and giving feedback. I've addressed all comments in the
> PR.
> > Looking forward to the next round of feedback (or approval). Would like
> to
> > get it in soon.
> >
> > https://github.com/apache/arrow/pull/48345/changes
> >
> > With this done and after talking to Arnav, I plan to spend some time on
> the
> > ALP implementation in the arrow-go repository helping spread the support
> to
> > more languages.
> >
> > Best
> > Prateek
> >
> > On Sun, Aug 2, 2026 at 10:42 PM PRATEEK GAUR <pr...@gmail.com> wrote:
> >
> > > Thanks Divjot for bringing it up.
> > >
> > > I was under the same impression as Russell and Andrew that we'd have it
> > > flag gated.
> > > With the new versioning proposal 'preview' might be a good tag for it.
> > >
> > > Best
> > > Prateek
> > >
> > > On Sat, Aug 1, 2026 at 12:39 AM Andrew Lamb <an...@gmail.com>
> > > wrote:
> > >
> > >> Div: Here is a proposal to mark ALP in Preview: [1] (using a comment
> in
> > >> the
> > >> spec section).
> > >>
> > >> Russel: That is my understanding as well.
> > >>
> > >> Hopefully we are able to formalize the versioning process shortly so
> we
> > >> can
> > >> formalize the notion of "preview"
> > >>
> > >> Andrew
> > >>
> > >>
> > >> [1]: https://github.com/apache/parquet-format/pull/604
> > >>
> > >> On Fri, Jul 31, 2026 at 11:15 AM Russell Spitzer <
> > >> russell.spitzer@gmail.com>
> > >> wrote:
> > >>
> > >> > I thought we had agreed it would be merged, and features like this
> > would
> > >> > be feature flag gated. So a V-Old writer could always opt in to
> using
> > >> this
> > >> > encoding but that must be a conscious effort if you are not using
> > VNext
> > >> of
> > >> > Parquet. If you are on V-Next the writer can choose to use ALP
> > whenever
> > >> it
> > >> > likes without user intervention.
> > >> >
> > >> > On Fri, Jul 31, 2026 at 2:20 AM Divjot Arora via dev <
> > >> > dev@parquet.apache.org> wrote:
> > >> >
> > >> >> Yes, that’s my understanding as well. However, as you noted, the
> > >> >> versioning
> > >> >> discussion is still in early stages and hasn’t prescribed concrete
> > >> >> mechanisms yet. If ALP goes into parquet-format 2.14.0, there’s no
> > >> >> indication that it’s in preview or that writers shouldn’t use it by
> > >> >> default
> > >> >> yet. Given that we’ve held off on merging other incompatible
> changes
> > >> (e.g
> > >> >> removal of path_in_schema) until the versioning discussion closes,
> my
> > >> >> preference would be to leave ALP un-merged for the time being as
> > well.
> > >> >>
> > >> >> — Div
> > >> >>
> > >> >> On Fri, Jul 31, 2026 at 08:59 Andrew Lamb <an...@gmail.com>
> > >> wrote:
> > >> >>
> > >> >> > After re-reading Julien's Versioning Proposal, my understanding
> is
> > >> that
> > >> >> if
> > >> >> > we adopt that system, we will somehow mark ALP in "Preview" in
> > >> >> > parquet-format for release Version 3.
> > >> >> >
> > >> >> > The versioning document doesn't seem to proscribe the particular
> > >> method
> > >> >> > for annotating "preview" features -- I think clearly marking them
> > as
> > >> >> such
> > >> >> > in parquet.thrift as well as reflecting that on the
> > >> >> > https://parquet.apache.org/docs/file-format/versions/ page
> somehow
> > >> >> would
> > >> >> > be adequate.
> > >> >> >
> > >> >> > Andrew
> > >> >> >
> > >> >> > On Thu, Jul 30, 2026 at 11:50 AM Andrew Lamb <
> > andrewlamb11@gmail.com
> > >> >
> > >> >> > wrote:
> > >> >> >
> > >> >> >> In my mind ALP is in the same position as other encodings that
> > were
> > >> >> added
> > >> >> >> recently[1] (such as BYTE_STREAM_SPLIT for non floating point
> > types)
> > >> >> >>
> > >> >> >> That is to say it will be available in the latest open source
> > >> writers
> > >> >> and
> > >> >> >> readers, but realistically not widely used in the ecosystem
> where
> > >> >> >> compatibility is a concern (a large part)
> > >> >> >>
> > >> >> >> I view Julian's work[2][3] to gather consensus on how to name
> > >> >> >> combinations of features (e.g. "versions") as the mechanism by
> > >> which we
> > >> >> >> will accelerate the adoption of this change (along with other
> > >> forward
> > >> >> >> incompatible changes) across the ecosystem.
> > >> >> >>
> > >> >> >> > Will we exclude this change when cutting parquet-format
> v2.14.0?
> > >> >> >>
> > >> >> >> No I don't think so -- I would expect it to go out in the next
> > >> >> >> parquet-format version as do other changes in the spec.
> > >> >> >>
> > >> >> >> Andrew
> > >> >> >>
> > >> >> >> [1]: https://parquet.apache.org/docs/file-format/versions/
> > >> >> >> [2]:
> > >> https://lists.apache.org/thread/xmnj8h0h8ozmrgox5tydhhs7yz7q2hss
> > >> >> >> [3]:
> > >> https://lists.apache.org/thread/qlf8lg90gqq41lllyy9mk2f0skqvftqj
> > >> >> >>
> > >> >> >>
> > >> >> >> On Thu, Jul 30, 2026 at 3:51 AM Divjot Arora via dev <
> > >> >> >> dev@parquet.apache.org> wrote:
> > >> >> >>
> > >> >> >>> Hi Andrew,
> > >> >> >>>
> > >> >> >>> How does this interact with the ongoing versioning discussion?
> > >> Given
> > >> >> that
> > >> >> >>> new encodings are not forward compatible, I would assume ALP is
> > >> >> included
> > >> >> >>> in
> > >> >> >>> Parquet V<next> (V3?), but it's not clear to me how that works
> if
> > >> it's
> > >> >> >>> merged to the parquet-format master branch now. Will we exclude
> > >> this
> > >> >> >>> change
> > >> >> >>> when cutting parquet-format v2.14.0?
> > >> >> >>>
> > >> >> >>> -- Div
> > >> >> >>>
> > >> >> >>> On Thu, Jul 30, 2026 at 8:51 AM Andrew Lamb <
> > >> andrewlamb11@gmail.com>
> > >> >> >>> wrote:
> > >> >> >>>
> > >> >> >>> > Hello,
> > >> >> >>> >
> > >> >> >>> > As an update, thanks to the epic work from many contributors,
> > >> >> >>> especially
> > >> >> >>> > Prateek, I just merged the parquet-format change for ALP[1].
> We
> > >> >> plan to
> > >> >> >>> > make additional editorial The remaining items I know of are
> > >> merging
> > >> >> the
> > >> >> >>> > examples from parquet-testing[2], and then of course the
> > various
> > >> >> >>> > implementations
> > >> >> >>> >
> > >> >> >>> > Thanks again everyone. This is a great step forward
> > >> >> >>> > Andrew
> > >> >> >>> >
> > >> >> >>> > [1]: https://github.com/apache/parquet-format/pull/557
> > >> >> >>> > [2]: https://github.com/apache/parquet-testing/pull/100
> > >> >> >>> >
> > >> >> >>> > On Tue, Jul 21, 2026 at 4:39 PM Julien Le Dem <
> > julien@apache.org
> > >> >
> > >> >> >>> wrote:
> > >> >> >>> >
> > >> >> >>> > > It was also in my spam folder... :(
> > >> >> >>> > > I've seen a bunch of apache list emails go to my spam
> > recently.
> > >> >> >>> > >
> > >> >> >>> > > On Tue, Jul 21, 2026 at 10:58 AM Russell Spitzer <
> > >> >> >>> > > russell.spitzer@gmail.com>
> > >> >> >>> > > wrote:
> > >> >> >>> > >
> > >> >> >>> > > > Yes, my hope was that my vote would kick it out of spam
> but
> > >> we
> > >> >> >>> > definitely
> > >> >> >>> > > > had two other votes that also went to spam for me.
> > >> >> >>> > > >
> > >> >> >>> > > > For anyone looking the thread is here
> > >> >> >>> > > >
> > >> >> https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt
> > >> >> >>> > > >
> > >> >> >>> > > > On Tue, Jul 21, 2026 at 10:57 AM Micah Kornfield <
> > >> >> >>> > emkornfield@gmail.com>
> > >> >> >>> > > > wrote:
> > >> >> >>> > > >
> > >> >> >>> > > > > I've seen other's votes not go to spam, so hopefully
> > people
> > >> >> will
> > >> >> >>> see
> > >> >> >>> > it
> > >> >> >>> > > > in
> > >> >> >>> > > > > there inboxes (or see this thread and look in there
> > spam).
> > >> >> >>> > > > >
> > >> >> >>> > > > > On Tue, Jul 21, 2026 at 10:52 AM PRATEEK GAUR <
> > >> >> >>> prtkgaur@gmail.com>
> > >> >> >>> > > > wrote:
> > >> >> >>> > > > >
> > >> >> >>> > > > > > Okay,
> > >> >> >>> > > > > >
> > >> >> >>> > > > > > I sent out the voting email yesterday and so far I've
> > >> heard
> > >> >> >>> from 3
> > >> >> >>> > > > people
> > >> >> >>> > > > > > that it ended up in their "spam folder". Dev VPN to
> be
> > >> >> blamed
> > >> >> >>> for
> > >> >> >>> > > it?.
> > >> >> >>> > > > > >
> > >> >> >>> > > > > > What's the protocol here?
> > >> >> >>> > > > > > I let people hunt for the email in their spam folder
> or
> > >> send
> > >> >> >>> out
> > >> >> >>> > > > another
> > >> >> >>> > > > > > one?
> > >> >> >>> > > > > >
> > >> >> >>> > > > > > Best
> > >> >> >>> > > > > > Prateek
> > >> >> >>> > > > > >
> > >> >> >>> > > > > > On Sun, Jul 19, 2026 at 10:47 PM PRATEEK GAUR <
> > >> >> >>> prtkgaur@gmail.com>
> > >> >> >>> > > > > wrote:
> > >> >> >>> > > > > >
> > >> >> >>> > > > > > > Hi team,
> > >> >> >>> > > > > > >
> > >> >> >>> > > > > > > Vinoo has added the tests which were requested in
> the
> > >> last
> > >> >> >>> round
> > >> >> >>> > of
> > >> >> >>> > > > > > review.
> > >> >> >>> > > > > > > In his own words.
> > >> >> >>> > > > > > >
> > >> >> >>> > > > > > > "
> > >> >> >>> > > > > > > The extreme value tests Micah wanted are done and
> > >> pushed.
> > >> >> I
> > >> >> >>> added
> > >> >> >>> > > > > > coverage
> > >> >> >>> > > > > > > for values that need the full FOR bit width after
> the
> > >> >> frame
> > >> >> >>> of
> > >> >> >>> > > > > reference
> > >> >> >>> > > > > > is
> > >> >> >>> > > > > > > applied: 64-bit (the case where the signed max
> minus
> > >> min
> > >> >> >>> > > subtraction
> > >> >> >>> > > > > > > overflows), 63-bit (the non-overflow case), and
> > 32-bit
> > >> for
> > >> >> >>> > floats.
> > >> >> >>> > > > All
> > >> >> >>> > > > > > > lossless with zero exceptions. I also verified them
> > >> >> through
> > >> >> >>> Arrow
> > >> >> >>> > > C++
> > >> >> >>> > > > > ALP
> > >> >> >>> > > > > > > reader (the local build), so the Java-written
> extreme
> > >> >> columns
> > >> >> >>> > > decode
> > >> >> >>> > > > > > > correctly in C++, bit-exact against the expected
> > >> values."
> > >> >> >>> > > > > > >
> > >> >> >>> > > > > > > State of the PR (#3397):
> > >> >> >>> > > > > > >
> > >> >> >>> > > > > > > - Merges cleanly with master (I pulled master in
> > and
> > >> >> >>> resolved
> > >> >> >>> > > the
> > >> >> >>> > > > > > > conflicts).
> > >> >> >>> > > > > > > - 41 of 44 review threads resolved. The 3
> > remaining
> > >> are
> > >> >> >>> > > > intentional:
> > >> >> >>> > > > > > > one deferred perf nit, one that stays open until
> > >> >> >>> > parquet-format
> > >> >> >>> > > > > ships
> > >> >> >>> > > > > > ALP,
> > >> >> >>> > > > > > >
> > >> >> >>> > > > > > > "
> > >> >> >>> > > > > > >
> > >> >> >>> > > > > > > With this both the
> > >> >> >>> > > > > > > 1) C++ PR :
> > >> >> >>> https://github.com/apache/arrow/pull/48345/changes
> > >> >> >>> > > > > > > 2) Java PR :
> > >> >> >>> https://github.com/apache/parquet-java/pull/3397
> > >> >> >>> > > > > > >
> > >> >> >>> > > > > > > Are cross language tested and range of test
> > scenarios.
> > >> >> >>> > > > > > >
> > >> >> >>> > > > > > > Best
> > >> >> >>> > > > > > > Prateek
> > >> >> >>> > > > > > >
> > >> >> >>> > > > > > >
> > >> >> >>> > > > > > > On Mon, Jul 13, 2026 at 8:38 AM PRATEEK GAUR <
> > >> >> >>> prtkgaur@gmail.com
> > >> >> >>> > >
> > >> >> >>> > > > > wrote:
> > >> >> >>> > > > > > >
> > >> >> >>> > > > > > >> Thanks Matt,
> > >> >> >>> > > > > > >>
> > >> >> >>> > > > > > >> I'll be on it this week.
> > >> >> >>> > > > > > >>
> > >> >> >>> > > > > > >> Best
> > >> >> >>> > > > > > >> Prateek
> > >> >> >>> > > > > > >>
> > >> >> >>> > > > > > >> On Mon, Jul 13, 2026 at 8:36 AM Matt Topol <
> > >> >> >>> > > zotthewizard@gmail.com>
> > >> >> >>> > > > > > >> wrote:
> > >> >> >>> > > > > > >>
> > >> >> >>> > > > > > >>> There is also the Go implementation submitted by
> > >> Arnav (
> > >> >> >>> > > > > > >>> https://github.com/apache/arrow-go/pull/704)
> which
> > >> is
> > >> >> >>> waiting
> > >> >> >>> > > for
> > >> >> >>> > > > > > >>> updates.
> > >> >> >>> > > > > > >>>
> > >> >> >>> > > > > > >>> Just wanted to make sure it didn't get lost here.
> > >> >> >>> > > > > > >>>
> > >> >> >>> > > > > > >>> --Matt
> > >> >> >>> > > > > > >>>
> > >> >> >>> > > > > > >>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK GAUR <
> > >> >> >>> > > prtkgaur@gmail.com>
> > >> >> >>> > > > > > >>> wrote:
> > >> >> >>> > > > > > >>>
> > >> >> >>> > > > > > >>> > Hi all,
> > >> >> >>> > > > > > >>> >
> > >> >> >>> > > > > > >>> > I'd like to share a status update on the ALP
> > >> encoding
> > >> >> >>> effort
> > >> >> >>> > > and
> > >> >> >>> > > > > get
> > >> >> >>> > > > > > a
> > >> >> >>> > > > > > >>> feel
> > >> >> >>> > > > > > >>> > from the community before starting a formal
> vote.
> > >> >> >>> > > > > > >>> >
> > >> >> >>> > > > > > >>> > We now have the specification plus
> > implementations
> > >> in
> > >> >> >>> both
> > >> >> >>> > > > > languages:
> > >> >> >>> > > > > > >>> >
> > >> >> >>> > > > > > >>> > - Spec: apache/parquet-format#557 —
> > >> GH-533
> > >> >> >>> Add
> > >> >> >>> > ALP
> > >> >> >>> > > > > > encoding
> > >> >> >>> > > > > > >>> > specification
> > >> >> >>> > > > > > >>> > - C++ (Arrow): apache/arrow#48345 —
> GH-48701
> > >> >> >>> > > [C++][Parquet]
> > >> >> >>> > > > > Add
> > >> >> >>> > > > > > >>> ALPpd
> > >> >> >>> > > > > > >>> > encoding
> > >> >> >>> > > > > > >>> > - Java: apache/parquet-java#3397 —
> > >> Parquet
> > >> >> >>> Java
> > >> >> >>> > ALP
> > >> >> >>> > > > > > >>> > Implementation (by Vinoo)
> > >> >> >>> > > > > > >>> >
> > >> >> >>> > > > > > >>> > Current state:
> > >> >> >>> > > > > > >>> >
> > >> >> >>> > > > > > >>> > - The C++ implementation has been through
> 3-4
> > >> >> rounds
> > >> >> >>> of
> > >> >> >>> > > > review.
> > >> >> >>> > > > > > >>> > - The Java implementation has been tracking
> > >> well
> > >> >> and
> > >> >> >>> has
> > >> >> >>> > > had
> > >> >> >>> > > > > > >>> initial
> > >> >> >>> > > > > > >>> > reviews from contributors, with no major
> > >> >> remaining
> > >> >> >>> > > issues.
> > >> >> >>> > > > > > >>> > - Cross-language compatibility tests are
> > >> passing:
> > >> >> the
> > >> >> >>> > Arrow
> > >> >> >>> > > > C++
> > >> >> >>> > > > > > >>> decoder
> > >> >> >>> > > > > > >>> > reads Java-written data bit-exactly
> across
> > >> >> ~1.56M
> > >> >> >>> > values
> > >> >> >>> > > > and
> > >> >> >>> > > > > 18
> > >> >> >>> > > > > > >>> > fixtures, covering V1 and V2 pages,
> > multiple
> > >> >> vector
> > >> >> >>> > > sizes,
> > >> >> >>> > > > > and
> > >> >> >>> > > > > > >>> > several
> > >> >> >>> > > > > > >>> > real datasets — zero mismatches.
> > >> >> >>> > > > > > >>> >
> > >> >> >>> > > > > > >>> > As raised in the java-pr we will be adding
> > >> coverage
> > >> >> for
> > >> >> >>> > > extreme
> > >> >> >>> > > > > > values
> > >> >> >>> > > > > > >>> > (those requiring
> > >> >> >>> > > > > > >>> > 63-64 bits after FOR is applied) before we
> > close a
> > >> >> vote.
> > >> >> >>> > We'll
> > >> >> >>> > > > aim
> > >> >> >>> > > > > > to
> > >> >> >>> > > > > > >>> get
> > >> >> >>> > > > > > >>> > that
> > >> >> >>> > > > > > >>> > done in parallel.
> > >> >> >>> > > > > > >>> >
> > >> >> >>> > > > > > >>> > Plan: unless there are objections, I intend to
> > >> start a
> > >> >> >>> vote
> > >> >> >>> > at
> > >> >> >>> > > > the
> > >> >> >>> > > > > > end
> > >> >> >>> > > > > > >>> of
> > >> >> >>> > > > > > >>> > this week/early
> > >> >> >>> > > > > > >>> > next week.
> > >> >> >>> > > > > > >>> >
> > >> >> >>> > > > > > >>> > Thanks
> > >> >> >>> > > > > > >>> > Prateek
> > >> >> >>> > > > > > >>> >
> > >> >> >>> > > > > > >>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <
> > >> >> >>> > > prtkgaur@gmail.com>
> > >> >> >>> > > > > > >>> wrote:
> > >> >> >>> > > > > > >>> >
> > >> >> >>> > > > > > >>> > > Hi Team,
> > >> >> >>> > > > > > >>> > >
> > >> >> >>> > > > > > >>> > > Just wanted to provide some updates on ALP.
> > >> >> >>> > > > > > >>> > > Micah and I have done 3-4 rounds of review
> for
> > >> the
> > >> >> c++
> > >> >> >>> PR :
> > >> >> >>> > > > > > >>> > > https://github.com/apache/arrow/pull/48345
> > >> >> >>> > > > > > >>> > >
> > >> >> >>> > > > > > >>> > > For the Java implementation Vinoo has been
> > >> working
> > >> >> on
> > >> >> >>> > > following
> > >> >> >>> > > > > PR
> > >> >> >>> > > > > > :
> > >> >> >>> > > > > > >>> > >
> > https://github.com/apache/parquet-java/pull/3397
> > >> >> >>> > > > > > >>> > >
> > >> >> >>> > > > > > >>> > > Best
> > >> >> >>> > > > > > >>> > > Prateek
> > >> >> >>> > > > > > >>> > >
> > >> >> >>> > > > > > >>> > > On Tue, May 5, 2026 at 1:50 PM Micah
> Kornfield
> > <
> > >> >> >>> > > > > > >>> emkornfield@gmail.com>
> > >> >> >>> > > > > > >>> > > wrote:
> > >> >> >>> > > > > > >>> > >
> > >> >> >>> > > > > > >>> > >> Hi Antoine,
> > >> >> >>> > > > > > >>> > >>
> > >> >> >>> > > > > > >>> > >> > Apologies if the question was already
> asked,
> > >> but
> > >> >> >>> should
> > >> >> >>> > we
> > >> >> >>> > > > > care
> > >> >> >>> > > > > > >>> about
> > >> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more
> > >> >> efficient
> > >> >> >>> than
> > >> >> >>> > > > > FLOAT16
> > >> >> >>> > > > > > +
> > >> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> > >> >> >>> > > > > > >>> > >>
> > >> >> >>> > > > > > >>> > >> It was. We thought we could defer it for
> the
> > >> >> >>> following
> > >> >> >>> > > > reasons:
> > >> >> >>> > > > > > >>> > >> 1. It's not clear there are a lot of easy
> > >> >> reference
> > >> >> >>> > > datasets
> > >> >> >>> > > > to
> > >> >> >>> > > > > > >>> test
> > >> >> >>> > > > > > >>> > its
> > >> >> >>> > > > > > >>> > >> effectiveness. It does look like there
> might
> > be
> > >> >> one
> > >> >> >>> or
> > >> >> >>> > two
> > >> >> >>> > > on
> > >> >> >>> > > > > > >>> > huggingface
> > >> >> >>> > > > > > >>> > >> (e.g.
> > >> >> https://huggingface.co/datasets/kikitora/curdie
> > >> >> >>> ).
> > >> >> >>> > > > > > >>> > >> 2. It seemed likely that float 16 was more
> > >> likely
> > >> >> >>> used
> > >> >> >>> > for
> > >> >> >>> > > > > values
> > >> >> >>> > > > > > >>> that
> > >> >> >>> > > > > > >>> > >> were less likely to reduce to decimal
> values.
> > >> >> >>> > > > > > >>> > >> 3. It could be added as an extension later
> if
> > >> >> needed.
> > >> >> >>> > > > > > >>> > >>
> > >> >> >>> > > > > > >>> > >> Cheers,
> > >> >> >>> > > > > > >>> > >> Micah
> > >> >> >>> > > > > > >>> > >>
> > >> >> >>> > > > > > >>> > >>
> > >> >> >>> > > > > > >>> > >> On Tue, May 5, 2026 at 1:40 PM Antoine
> Pitrou
> > <
> > >> >> >>> > > > > antoine@python.org
> > >> >> >>> > > > > > >
> > >> >> >>> > > > > > >>> > wrote:
> > >> >> >>> > > > > > >>> > >>
> > >> >> >>> > > > > > >>> > >> >
> > >> >> >>> > > > > > >>> > >> > Hello,
> > >> >> >>> > > > > > >>> > >> >
> > >> >> >>> > > > > > >>> > >> > Apologies if the question was already
> asked,
> > >> but
> > >> >> >>> should
> > >> >> >>> > we
> > >> >> >>> > > > > care
> > >> >> >>> > > > > > >>> about
> > >> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more
> > >> >> efficient
> > >> >> >>> than
> > >> >> >>> > > > > FLOAT16
> > >> >> >>> > > > > > +
> > >> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> > >> >> >>> > > > > > >>> > >> >
> > >> >> >>> > > > > > >>> > >> > Regards
> > >> >> >>> > > > > > >>> > >> >
> > >> >> >>> > > > > > >>> > >> > Antoine.
> > >> >> >>> > > > > > >>> > >> >
> > >> >> >>> > > > > > >>> > >> >
> > >> >> >>> > > > > > >>> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a
> écrit
> > :
> > >> >> >>> > > > > > >>> > >> > > Thanks Andrew and Micah for review
> > feedback
> > >> on
> > >> >> >>> the two
> > >> >> >>> > > > PR's
> > >> >> >>> > > > > > >>> > >> > > 1) (c++ arrow repo)
> > >> >> >>> > > > > > >>> > >>
> > >> https://github.com/apache/arrow/pull/48345/changes
> > >> >> >>> > > > > > >>> > >> > > 2) (parquet-format repo)
> > >> >> >>> > > > > > >>> > >> >
> > >> >> https://github.com/apache/parquet-format/pull/557
> > >> >> >>> > > > > > >>> > >> > >
> > >> >> >>> > > > > > >>> > >> > > I have addressed all (unless I missed
> > >> >> something)
> > >> >> >>> > > comments
> > >> >> >>> > > > on
> > >> >> >>> > > > > > >>> the two
> > >> >> >>> > > > > > >>> > >> > PR's.
> > >> >> >>> > > > > > >>> > >> > >
> > >> >> >>> > > > > > >>> > >> > > Best
> > >> >> >>> > > > > > >>> > >> > > Prateek
> > >> >> >>> > > > > > >>> > >> > >
> > >> >> >>> > > > > > >>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK
> > >> GAUR <
> > >> >> >>> > > > > > >>> prtkgaur@gmail.com>
> > >> >> >>> > > > > > >>> > >> wrote:
> > >> >> >>> > > > > > >>> > >> > >
> > >> >> >>> > > > > > >>> > >> > >> Thanks Andrew and Micah.
> > >> >> >>> > > > > > >>> > >> > >>
> > >> >> >>> > > > > > >>> > >> > >> `fair amount of feedback on at least
> the
> > >> >> >>> > > implementations`
> > >> >> >>> > > > > > >>> > >> > >> For the c++ I have already started
> > >> addressing
> > >> >> the
> > >> >> >>> > > > > feedback, I
> > >> >> >>> > > > > > >>> > should
> > >> >> >>> > > > > > >>> > >> be
> > >> >> >>> > > > > > >>> > >> > >> done with that Monday/Tuesday.
> > >> >> >>> > > > > > >>> > >> > >> I think Vinoo too has been making good
> > >> >> progress
> > >> >> >>> on
> > >> >> >>> > the
> > >> >> >>> > > > Java
> > >> >> >>> > > > > > >>> > >> > implementation.
> > >> >> >>> > > > > > >>> > >> > >>
> > >> >> >>> > > > > > >>> > >> > >> Best
> > >> >> >>> > > > > > >>> > >> > >> Prateek
> > >> >> >>> > > > > > >>> > >> > >>
> > >> >> >>> > > > > > >>> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew
> > >> Lamb <
> > >> >> >>> > > > > > >>> > andrewlamb11@gmail.com
> > >> >> >>> > > > > > >>> > >> >
> > >> >> >>> > > > > > >>> > >> > >> wrote:
> > >> >> >>> > > > > > >>> > >> > >>
> > >> >> >>> > > > > > >>> > >> > >>> Got it. Thank you for the
> clarification
> > >> -- I
> > >> >> >>> will
> > >> >> >>> > try
> > >> >> >>> > > > and
> > >> >> >>> > > > > > look
> > >> >> >>> > > > > > >>> > into
> > >> >> >>> > > > > > >>> > >> the
> > >> >> >>> > > > > > >>> > >> > >>> spec and the Rust implementation[1] in
> > >> this
> > >> >> next
> > >> >> >>> > week
> > >> >> >>> > > > > > >>> > >> > >>>
> > >> >> >>> > > > > > >>> > >> > >>> [1]:
> > >> >> >>> https://github.com/apache/arrow-rs/pull/9372
> > >> >> >>> > > > > > >>> > >> > >>>
> > >> >> >>> > > > > > >>> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah
> > >> >> >>> Kornfield <
> > >> >> >>> > > > > > >>> > >> > emkornfield@gmail.com>
> > >> >> >>> > > > > > >>> > >> > >>> wrote:
> > >> >> >>> > > > > > >>> > >> > >>>
> > >> >> >>> > > > > > >>> > >> > >>>> Hi Andrew,
> > >> >> >>> > > > > > >>> > >> > >>>> I think there is a fair amount of
> > >> feedback
> > >> >> on
> > >> >> >>> at
> > >> >> >>> > > least
> > >> >> >>> > > > > the
> > >> >> >>> > > > > > >>> > >> > >>>> implementations, typically I think
> > we've
> > >> >> waited
> > >> >> >>> > till
> > >> >> >>> > > > they
> > >> >> >>> > > > > > are
> > >> >> >>> > > > > > >>> > >> close to
> > >> >> >>> > > > > > >>> > >> > >>>> mergeable before a final vote.
> > >> Otherwise I
> > >> >> >>> agree
> > >> >> >>> > we
> > >> >> >>> > > > are
> > >> >> >>> > > > > > very
> > >> >> >>> > > > > > >>> > >> close.
> > >> >> >>> > > > > > >>> > >> > >>>>
> > >> >> >>> > > > > > >>> > >> > >>>> -Micah
> > >> >> >>> > > > > > >>> > >> > >>>>
> > >> >> >>> > > > > > >>> > >> > >>>> On Saturday, April 25, 2026, Andrew
> > Lamb
> > >> <
> > >> >> >>> > > > > > >>> andrewlamb11@gmail.com
> > >> >> >>> > > > > > >>> > >
> > >> >> >>> > > > > > >>> > >> > wrote:
> > >> >> >>> > > > > > >>> > >> > >>>>
> > >> >> >>> > > > > > >>> > >> > >>>>> Thanks Prateek,
> > >> >> >>> > > > > > >>> > >> > >>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>> I think from this content it looks
> to
> > me
> > >> >> like
> > >> >> >>> we
> > >> >> >>> > are
> > >> >> >>> > > > > ready
> > >> >> >>> > > > > > >>> to
> > >> >> >>> > > > > > >>> > >> start a
> > >> >> >>> > > > > > >>> > >> > >>>>> vote to explicitly accept ALP into
> > >> Parquet
> > >> >> >>> > > > > > >>> > >> > >>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>> Does anyone know of a reason we
> should
> > >> >> >>> postpone it
> > >> >> >>> > > for
> > >> >> >>> > > > > > >>> longer?
> > >> >> >>> > > > > > >>> > >> > >>>>> Perhaps someone needs some more time
> > to
> > >> >> >>> review?
> > >> >> >>> > > > > > >>> > >> > >>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>> Andrew
> > >> >> >>> > > > > > >>> > >> > >>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM
> > PRATEEK
> > >> >> GAUR <
> > >> >> >>> > > > > > >>> > prtkgaur@gmail.com>
> > >> >> >>> > > > > > >>> > >> > >>>>> wrote:
> > >> >> >>> > > > > > >>> > >> > >>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>> Hi team,
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>> Hope everyone is doing well. I got
> a
> > >> >> chance
> > >> >> >>> to
> > >> >> >>> > work
> > >> >> >>> > > > > > >>> through all
> > >> >> >>> > > > > > >>> > >> the
> > >> >> >>> > > > > > >>> > >> > >>>>>> remaining feedback and update the
> > spec
> > >> >> doc.
> > >> >> >>> Here
> > >> >> >>> > > are
> > >> >> >>> > > > > the
> > >> >> >>> > > > > > >>> new
> > >> >> >>> > > > > > >>> > >> > artifacts
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>> 1) Spec document :
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> >
> > >> >> >>> > > > > > >>> > >>
> > >> >> >>> > > > > > >>> >
> > >> >> >>> > > > > > >>>
> > >> >> >>> > > > > >
> > >> >> >>> > > > >
> > >> >> >>> > > >
> > >> >> >>> > >
> > >> >> >>> >
> > >> >> >>>
> > >> >>
> > >>
> >
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>> 2) Spec document in parquet format
> > >> repo :
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > https://github.com/apache/parquet-format/pull/557
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>> 3) Alp implementation in arrow c++
> > >> repo :
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > https://github.com/apache/arrow/pull/48345/changes
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>> 4) Alp implementation in
> parquet-java
> > >> >> repo :
> > >> >> >>> Work
> > >> >> >>> > > for
> > >> >> >>> > > > > > >>> Vinoo and
> > >> >> >>> > > > > > >>> > >> > Julien
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > https://github.com/apache/parquet-java/pull/3397
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>> 5) PR with test and benchmarking
> > >> >> artifacts in
> > >> >> >>> > > > > > >>> parquet-testing
> > >> >> >>> > > > > > >>> > >> repo :
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > https://github.com/apache/parquet-testing/pull/100
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>> And
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>> - Go : Arnav just submitted an
> in
> > >> >> >>> progress
> > >> >> >>> > > > > > >>> implementation
> > >> >> >>> > > > > > >>> > in
> > >> >> >>> > > > > > >>> > >> Go.
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> https://github.com/apache/arrow-go/pull/704
> > >> >> >>> > (I
> > >> >> >>> > > > > > haven't
> > >> >> >>> > > > > > >>> > >> started
> > >> >> >>> > > > > > >>> > >> > >>>>>> looking at it yet)
> > >> >> >>> > > > > > >>> > >> > >>>>>> - Rust : I remember Andrew
> > >> mentioned
> > >> >> that
> > >> >> >>> > this
> > >> >> >>> > > > work
> > >> >> >>> > > > > > is
> > >> >> >>> > > > > > >>> also
> > >> >> >>> > > > > > >>> > >> in
> > >> >> >>> > > > > > >>> > >> > >>>>>> progress (So 4 languages!)
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>> *Arrow C++ implementation *
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>> The PR is out and was also used by
> > >> >> Antoine to
> > >> >> >>> > > report
> > >> >> >>> > > > > the
> > >> >> >>> > > > > > >>> > numbers
> > >> >> >>> > > > > > >>> > >> as
> > >> >> >>> > > > > > >>> > >> > >>>>>> reported here. Micah and Konstantin
> > >> have
> > >> >> >>> given 1
> > >> >> >>> > > > round
> > >> >> >>> > > > > of
> > >> >> >>> > > > > > >>> > >> feedback
> > >> >> >>> > > > > > >>> > >> > >>>>>> and I'm addressing them today.
> Please
> > >> note
> > >> >> >>> that
> > >> >> >>> > the
> > >> >> >>> > > > > > default
> > >> >> >>> > > > > > >>> > >> > >>>>>> optimization flag for compiling is
> O2
> > >> and
> > >> >> not
> > >> >> >>> > Q3. I
> > >> >> >>> > > > got
> > >> >> >>> > > > > > >>> around
> > >> >> >>> > > > > > >>> > >> 70%
> > >> >> >>> > > > > > >>> > >> > >>>>>> performance improvement in the
> > decoding
> > >> >> speed
> > >> >> >>> > when
> > >> >> >>> > > > > using
> > >> >> >>> > > > > > >>> the O3
> > >> >> >>> > > > > > >>> > >> > flag.
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>> *Parqet-MR Java implementation
> > (working
> > >> >> with
> > >> >> >>> > Vinoo
> > >> >> >>> > > > and
> > >> >> >>> > > > > > >>> Julien)
> > >> >> >>> > > > > > >>> > >> and
> > >> >> >>> > > > > > >>> > >> > **Cross
> > >> >> >>> > > > > > >>> > >> > >>>>>> Language testing*
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>> Let me know if you have any
> > >> questions
> > >> >> or
> > >> >> >>> > > > feedback.
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>> Now pasting some performance
> numbers
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>> Table 1: C++ ALP Double Decode —
> > >> >> Spotify
> > >> >> >>> > Columns
> > >> >> >>> > > > > > >>> (Graviton
> > >> >> >>> > > > > > >>> > 3,
> > >> >> >>> > > > > > >>> > >> ARM
> > >> >> >>> > > > > > >>> > >> > >>>>>> Neoverse V1)
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>>
> > >> >> >>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>> │ Column │ -O2 (MB/s)
> > │
> > >> >> -O3
> > >> >> >>> > > (MB/s) │
> > >> >> >>> > > > > > >>> Speedup │
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>>
> > >> >> >>> ├──────────────────┼──────────────┼──────────────┼─────────┤
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>> │ valence │ 3,155
> > │
> > >> >> >>> 5,523
> > >> >> >>> > > │
> > >> >> >>> > > > > > >>> 1.75x │
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>> │ danceability │ 3,233
> > │
> > >> >> >>> 5,685
> > >> >> >>> > > │
> > >> >> >>> > > > > > >>> 1.76x │
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>> │ energy │ 3,197
> > │
> > >> >> >>> 5,652
> > >> >> >>> > > │
> > >> >> >>> > > > > > >>> 1.77x │
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>> │ loudness │ 3,186
> > │
> > >> >> >>> 5,473
> > >> >> >>> > > │
> > >> >> >>> > > > > > >>> 1.72x │
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>>
> > >> >> >>> └──────────────────┴──────────────┴──────────────┴─────────┘
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM
> > PRATEEK
> > >> >> GAUR
> > >> >> >>> <
> > >> >> >>> > > > > > >>> > prtkgaur@gmail.com
> > >> >> >>> > > > > > >>> > >> >
> > >> >> >>> > > > > > >>> > >> > >>>>>> wrote:
> > >> >> >>> > > > > > >>> > >> > >>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>> @Micah Kornfield <
> > >> emkornfield@gmail.com>
> > >> >> :
> > >> >> >>> Got
> > >> >> >>> > > it.
> > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>> @Andrew Lamb <
> > andrewlamb11@gmail.com>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> Do you think it would be good to
> > >> start
> > >> >> >>> moving
> > >> >> >>> > the
> > >> >> >>> > > > > spec
> > >> >> >>> > > > > > >>> > >> development
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> into
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> markdown format, in preparation
> for
> > >> >> >>> finalizing
> > >> >> >>> > > it?
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>> Yes I'll update the numbers for
> some
> > >> of
> > >> >> the
> > >> >> >>> > > > examples I
> > >> >> >>> > > > > > >>> have in
> > >> >> >>> > > > > > >>> > >> the
> > >> >> >>> > > > > > >>> > >> > >>>>>>> spec based
> > >> >> >>> > > > > > >>> > >> > >>>>>>> on the updated header size. Then
> we
> > >> >> should
> > >> >> >>> be
> > >> >> >>> > good
> > >> >> >>> > > > to
> > >> >> >>> > > > > go
> > >> >> >>> > > > > > >>> for
> > >> >> >>> > > > > > >>> > the
> > >> >> >>> > > > > > >>> > >> > >>>>>>> markdown format.
> > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>> Thanks everyone!
> > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> Andrew
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM
> > >> PRATEEK
> > >> >> >>> GAUR <
> > >> >> >>> > > > > > >>> > >> prtkgaur@gmail.com>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> wrote:
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Hi team,
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 1) Andrew
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for working on test
> > >> files.
> > >> >> >>> My PR
> > >> >> >>> > > did
> > >> >> >>> > > > > add
> > >> >> >>> > > > > > >>> all
> > >> >> >>> > > > > > >>> > the
> > >> >> >>> > > > > > >>> > >> > test
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> files I
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> used to benchmark on
> datasets.
> > >> >> Maybe
> > >> >> >>> we
> > >> >> >>> > can
> > >> >> >>> > > > club
> > >> >> >>> > > > > > it
> > >> >> >>> > > > > > >>> > >> together.
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> WIll also
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> aid
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> cross language testing
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Kosta Tarasov working on
> > Rust
> > >> >> >>> > > > implementation.
> > >> >> >>> > > > > > >>> This is
> > >> >> >>> > > > > > >>> > >> > great.
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> Thanks
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 2) Antoine
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks a lot for reporting
> > the
> > >> >> >>> numbers
> > >> >> >>> > on
> > >> >> >>> > > > AMD.
> > >> >> >>> > > > > > >>> Looks
> > >> >> >>> > > > > > >>> > >> like
> > >> >> >>> > > > > > >>> > >> > you
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> are
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> getting 8X the decoding
> > >> >> performance of
> > >> >> >>> > BSS.
> > >> >> >>> > > > This
> > >> >> >>> > > > > > is
> > >> >> >>> > > > > > >>> > >> > amazing!!.
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for acknowledging
> the
> > >> >> >>> sampling
> > >> >> >>> > > > design.
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - I agree with you on
> > >> Fastlanes. In
> > >> >> >>> some
> > >> >> >>> > > crude
> > >> >> >>> > > > > > >>> > >> experiments I
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> didn't get
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> a good perf benefit from it
> on
> > >> >> >>> Graviton3
> > >> >> >>> > > (but
> > >> >> >>> > > > > > maybe
> > >> >> >>> > > > > > >>> > there
> > >> >> >>> > > > > > >>> > >> was
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> something
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> wrong with my
> implementation).
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Locking the 16bit
> exception
> > >> >> >>> encoding for
> > >> >> >>> > > the
> > >> >> >>> > > > > > spec
> > >> >> >>> > > > > > >>> in
> > >> >> >>> > > > > > >>> > >> this
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> case.
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Awesome I think we have
> > solved
> > >> >> for
> > >> >> >>> all
> > >> >> >>> > > open
> > >> >> >>> > > > > > >>> questions
> > >> >> >>> > > > > > >>> > >> minus
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> the
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> version byte :). (will get
> > back
> > >> on
> > >> >> >>> this
> > >> >> >>> > > soon)
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> 3) Micah
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> - FastLanes : The current
> spec
> > >> does
> > >> >> >>> allow
> > >> >> >>> > > for
> > >> >> >>> > > > > > using
> > >> >> >>> > > > > > >>> > >> FastLane
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> with the
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> configurable enum value for
> > >> >> layout. We
> > >> >> >>> > > should
> > >> >> >>> > > > be
> > >> >> >>> > > > > > >>> able to
> > >> >> >>> > > > > > >>> > >> > inject
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> any
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> layout
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> in the current design.
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Working on resolving all
> remaining
> > >> open
> > >> >> >>> > comments
> > >> >> >>> > > > on
> > >> >> >>> > > > > > the
> > >> >> >>> > > > > > >>> spec
> > >> >> >>> > > > > > >>> > >> this
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> week.
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Best
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> Prateek
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM
> > >> Steve
> > >> >> >>> > Loughran <
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> stevel@cloudera.com>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> wrote:
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12,
> > Micah
> > >> >> >>> Kornfield
> > >> >> >>> > <
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> emkornfield@gmail.com>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> wrote:
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> It looks like the actual issue
> > >> >> >>> described for
> > >> >> >>> > > ORC
> > >> >> >>> > > > > in
> > >> >> >>> > > > > > >>> the
> > >> >> >>> > > > > > >>> > >> paper
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> is that
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> it
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> has multiple sub-encodings in
> a
> > >> >> batch.
> > >> >> >>> This
> > >> >> >>> > > is
> > >> >> >>> > > > > > >>> different
> > >> >> >>> > > > > > >>> > >> then
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> the
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>> design
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> proposed here where there is
> > still
> > >> >> fixed
> > >> >> >>> > > > encoding
> > >> >> >>> > > > > > per
> > >> >> >>> > > > > > >>> page
> > >> >> >>> > > > > > >>> > >> in
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> parquet.
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> Given reasonably sized pages I
> > >> don't
> > >> >> >>> think
> > >> >> >>> > > > branch
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> misprediction should
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> be a
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> big issue for new encodings.
> I
> > >> agree
> > >> >> >>> that
> > >> >> >>> > we
> > >> >> >>> > > > > should
> > >> >> >>> > > > > > >>> be
> > >> >> >>> > > > > > >>> > >> > >>>>>>>> conservative in
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> general for adding new
> > encodings.
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>> +1
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>>
> > >> >> >>> > > > > > >>> > >> > >>>>>>>
> > >> >> >>> > > > > > >>> > >> > >
> > >> >> >>> > > > > > >>> > >> >
> > >> >> >>> > > > > > >>> > >> >
> > >> >> >>> > > > > > >>> > >> >
> > >> >> >>> > > > > > >>> > >>
> > >> >> >>> > > > > > >>> > >
> > >> >> >>> > > > > > >>> >
> > >> >> >>> > > > > > >>>
> > >> >> >>> > > > > > >>
> > >> >> >>> > > > > >
> > >> >> >>> > > > >
> > >> >> >>> > > >
> > >> >> >>> > >
> > >> >> >>> >
> > >> >> >>>
> > >> >> >>
> > >> >>
> > >> >
> > >>
> > >
> >
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Andrew Lamb <an...@gmail.com>.
We just merged the Rust implementation[1] from Kosta and Devan. With the
imminent release of Parquet Format 2.14.0 I think we'll be well on our way.
Thanks again for pushing this along,
Andrew
[1]: https://github.com/apache/arrow-rs/pull/9372
On Fri, Sep 4, 2026 at 5:42 PM PRATEEK GAUR <pr...@gmail.com> wrote:
> Hi team,
>
> Thanks Micah for multiple rounds of review and thanks Gang for taking a
> look at the PR and giving feedback. I've addressed all comments in the PR.
> Looking forward to the next round of feedback (or approval). Would like to
> get it in soon.
>
> https://github.com/apache/arrow/pull/48345/changes
>
> With this done and after talking to Arnav, I plan to spend some time on the
> ALP implementation in the arrow-go repository helping spread the support to
> more languages.
>
> Best
> Prateek
>
> On Sun, Aug 2, 2026 at 10:42 PM PRATEEK GAUR <pr...@gmail.com> wrote:
>
> > Thanks Divjot for bringing it up.
> >
> > I was under the same impression as Russell and Andrew that we'd have it
> > flag gated.
> > With the new versioning proposal 'preview' might be a good tag for it.
> >
> > Best
> > Prateek
> >
> > On Sat, Aug 1, 2026 at 12:39 AM Andrew Lamb <an...@gmail.com>
> > wrote:
> >
> >> Div: Here is a proposal to mark ALP in Preview: [1] (using a comment in
> >> the
> >> spec section).
> >>
> >> Russel: That is my understanding as well.
> >>
> >> Hopefully we are able to formalize the versioning process shortly so we
> >> can
> >> formalize the notion of "preview"
> >>
> >> Andrew
> >>
> >>
> >> [1]: https://github.com/apache/parquet-format/pull/604
> >>
> >> On Fri, Jul 31, 2026 at 11:15 AM Russell Spitzer <
> >> russell.spitzer@gmail.com>
> >> wrote:
> >>
> >> > I thought we had agreed it would be merged, and features like this
> would
> >> > be feature flag gated. So a V-Old writer could always opt in to using
> >> this
> >> > encoding but that must be a conscious effort if you are not using
> VNext
> >> of
> >> > Parquet. If you are on V-Next the writer can choose to use ALP
> whenever
> >> it
> >> > likes without user intervention.
> >> >
> >> > On Fri, Jul 31, 2026 at 2:20 AM Divjot Arora via dev <
> >> > dev@parquet.apache.org> wrote:
> >> >
> >> >> Yes, that’s my understanding as well. However, as you noted, the
> >> >> versioning
> >> >> discussion is still in early stages and hasn’t prescribed concrete
> >> >> mechanisms yet. If ALP goes into parquet-format 2.14.0, there’s no
> >> >> indication that it’s in preview or that writers shouldn’t use it by
> >> >> default
> >> >> yet. Given that we’ve held off on merging other incompatible changes
> >> (e.g
> >> >> removal of path_in_schema) until the versioning discussion closes, my
> >> >> preference would be to leave ALP un-merged for the time being as
> well.
> >> >>
> >> >> — Div
> >> >>
> >> >> On Fri, Jul 31, 2026 at 08:59 Andrew Lamb <an...@gmail.com>
> >> wrote:
> >> >>
> >> >> > After re-reading Julien's Versioning Proposal, my understanding is
> >> that
> >> >> if
> >> >> > we adopt that system, we will somehow mark ALP in "Preview" in
> >> >> > parquet-format for release Version 3.
> >> >> >
> >> >> > The versioning document doesn't seem to proscribe the particular
> >> method
> >> >> > for annotating "preview" features -- I think clearly marking them
> as
> >> >> such
> >> >> > in parquet.thrift as well as reflecting that on the
> >> >> > https://parquet.apache.org/docs/file-format/versions/ page somehow
> >> >> would
> >> >> > be adequate.
> >> >> >
> >> >> > Andrew
> >> >> >
> >> >> > On Thu, Jul 30, 2026 at 11:50 AM Andrew Lamb <
> andrewlamb11@gmail.com
> >> >
> >> >> > wrote:
> >> >> >
> >> >> >> In my mind ALP is in the same position as other encodings that
> were
> >> >> added
> >> >> >> recently[1] (such as BYTE_STREAM_SPLIT for non floating point
> types)
> >> >> >>
> >> >> >> That is to say it will be available in the latest open source
> >> writers
> >> >> and
> >> >> >> readers, but realistically not widely used in the ecosystem where
> >> >> >> compatibility is a concern (a large part)
> >> >> >>
> >> >> >> I view Julian's work[2][3] to gather consensus on how to name
> >> >> >> combinations of features (e.g. "versions") as the mechanism by
> >> which we
> >> >> >> will accelerate the adoption of this change (along with other
> >> forward
> >> >> >> incompatible changes) across the ecosystem.
> >> >> >>
> >> >> >> > Will we exclude this change when cutting parquet-format v2.14.0?
> >> >> >>
> >> >> >> No I don't think so -- I would expect it to go out in the next
> >> >> >> parquet-format version as do other changes in the spec.
> >> >> >>
> >> >> >> Andrew
> >> >> >>
> >> >> >> [1]: https://parquet.apache.org/docs/file-format/versions/
> >> >> >> [2]:
> >> https://lists.apache.org/thread/xmnj8h0h8ozmrgox5tydhhs7yz7q2hss
> >> >> >> [3]:
> >> https://lists.apache.org/thread/qlf8lg90gqq41lllyy9mk2f0skqvftqj
> >> >> >>
> >> >> >>
> >> >> >> On Thu, Jul 30, 2026 at 3:51 AM Divjot Arora via dev <
> >> >> >> dev@parquet.apache.org> wrote:
> >> >> >>
> >> >> >>> Hi Andrew,
> >> >> >>>
> >> >> >>> How does this interact with the ongoing versioning discussion?
> >> Given
> >> >> that
> >> >> >>> new encodings are not forward compatible, I would assume ALP is
> >> >> included
> >> >> >>> in
> >> >> >>> Parquet V<next> (V3?), but it's not clear to me how that works if
> >> it's
> >> >> >>> merged to the parquet-format master branch now. Will we exclude
> >> this
> >> >> >>> change
> >> >> >>> when cutting parquet-format v2.14.0?
> >> >> >>>
> >> >> >>> -- Div
> >> >> >>>
> >> >> >>> On Thu, Jul 30, 2026 at 8:51 AM Andrew Lamb <
> >> andrewlamb11@gmail.com>
> >> >> >>> wrote:
> >> >> >>>
> >> >> >>> > Hello,
> >> >> >>> >
> >> >> >>> > As an update, thanks to the epic work from many contributors,
> >> >> >>> especially
> >> >> >>> > Prateek, I just merged the parquet-format change for ALP[1]. We
> >> >> plan to
> >> >> >>> > make additional editorial The remaining items I know of are
> >> merging
> >> >> the
> >> >> >>> > examples from parquet-testing[2], and then of course the
> various
> >> >> >>> > implementations
> >> >> >>> >
> >> >> >>> > Thanks again everyone. This is a great step forward
> >> >> >>> > Andrew
> >> >> >>> >
> >> >> >>> > [1]: https://github.com/apache/parquet-format/pull/557
> >> >> >>> > [2]: https://github.com/apache/parquet-testing/pull/100
> >> >> >>> >
> >> >> >>> > On Tue, Jul 21, 2026 at 4:39 PM Julien Le Dem <
> julien@apache.org
> >> >
> >> >> >>> wrote:
> >> >> >>> >
> >> >> >>> > > It was also in my spam folder... :(
> >> >> >>> > > I've seen a bunch of apache list emails go to my spam
> recently.
> >> >> >>> > >
> >> >> >>> > > On Tue, Jul 21, 2026 at 10:58 AM Russell Spitzer <
> >> >> >>> > > russell.spitzer@gmail.com>
> >> >> >>> > > wrote:
> >> >> >>> > >
> >> >> >>> > > > Yes, my hope was that my vote would kick it out of spam but
> >> we
> >> >> >>> > definitely
> >> >> >>> > > > had two other votes that also went to spam for me.
> >> >> >>> > > >
> >> >> >>> > > > For anyone looking the thread is here
> >> >> >>> > > >
> >> >> https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt
> >> >> >>> > > >
> >> >> >>> > > > On Tue, Jul 21, 2026 at 10:57 AM Micah Kornfield <
> >> >> >>> > emkornfield@gmail.com>
> >> >> >>> > > > wrote:
> >> >> >>> > > >
> >> >> >>> > > > > I've seen other's votes not go to spam, so hopefully
> people
> >> >> will
> >> >> >>> see
> >> >> >>> > it
> >> >> >>> > > > in
> >> >> >>> > > > > there inboxes (or see this thread and look in there
> spam).
> >> >> >>> > > > >
> >> >> >>> > > > > On Tue, Jul 21, 2026 at 10:52 AM PRATEEK GAUR <
> >> >> >>> prtkgaur@gmail.com>
> >> >> >>> > > > wrote:
> >> >> >>> > > > >
> >> >> >>> > > > > > Okay,
> >> >> >>> > > > > >
> >> >> >>> > > > > > I sent out the voting email yesterday and so far I've
> >> heard
> >> >> >>> from 3
> >> >> >>> > > > people
> >> >> >>> > > > > > that it ended up in their "spam folder". Dev VPN to be
> >> >> blamed
> >> >> >>> for
> >> >> >>> > > it?.
> >> >> >>> > > > > >
> >> >> >>> > > > > > What's the protocol here?
> >> >> >>> > > > > > I let people hunt for the email in their spam folder or
> >> send
> >> >> >>> out
> >> >> >>> > > > another
> >> >> >>> > > > > > one?
> >> >> >>> > > > > >
> >> >> >>> > > > > > Best
> >> >> >>> > > > > > Prateek
> >> >> >>> > > > > >
> >> >> >>> > > > > > On Sun, Jul 19, 2026 at 10:47 PM PRATEEK GAUR <
> >> >> >>> prtkgaur@gmail.com>
> >> >> >>> > > > > wrote:
> >> >> >>> > > > > >
> >> >> >>> > > > > > > Hi team,
> >> >> >>> > > > > > >
> >> >> >>> > > > > > > Vinoo has added the tests which were requested in the
> >> last
> >> >> >>> round
> >> >> >>> > of
> >> >> >>> > > > > > review.
> >> >> >>> > > > > > > In his own words.
> >> >> >>> > > > > > >
> >> >> >>> > > > > > > "
> >> >> >>> > > > > > > The extreme value tests Micah wanted are done and
> >> pushed.
> >> >> I
> >> >> >>> added
> >> >> >>> > > > > > coverage
> >> >> >>> > > > > > > for values that need the full FOR bit width after the
> >> >> frame
> >> >> >>> of
> >> >> >>> > > > > reference
> >> >> >>> > > > > > is
> >> >> >>> > > > > > > applied: 64-bit (the case where the signed max minus
> >> min
> >> >> >>> > > subtraction
> >> >> >>> > > > > > > overflows), 63-bit (the non-overflow case), and
> 32-bit
> >> for
> >> >> >>> > floats.
> >> >> >>> > > > All
> >> >> >>> > > > > > > lossless with zero exceptions. I also verified them
> >> >> through
> >> >> >>> Arrow
> >> >> >>> > > C++
> >> >> >>> > > > > ALP
> >> >> >>> > > > > > > reader (the local build), so the Java-written extreme
> >> >> columns
> >> >> >>> > > decode
> >> >> >>> > > > > > > correctly in C++, bit-exact against the expected
> >> values."
> >> >> >>> > > > > > >
> >> >> >>> > > > > > > State of the PR (#3397):
> >> >> >>> > > > > > >
> >> >> >>> > > > > > > - Merges cleanly with master (I pulled master in
> and
> >> >> >>> resolved
> >> >> >>> > > the
> >> >> >>> > > > > > > conflicts).
> >> >> >>> > > > > > > - 41 of 44 review threads resolved. The 3
> remaining
> >> are
> >> >> >>> > > > intentional:
> >> >> >>> > > > > > > one deferred perf nit, one that stays open until
> >> >> >>> > parquet-format
> >> >> >>> > > > > ships
> >> >> >>> > > > > > ALP,
> >> >> >>> > > > > > >
> >> >> >>> > > > > > > "
> >> >> >>> > > > > > >
> >> >> >>> > > > > > > With this both the
> >> >> >>> > > > > > > 1) C++ PR :
> >> >> >>> https://github.com/apache/arrow/pull/48345/changes
> >> >> >>> > > > > > > 2) Java PR :
> >> >> >>> https://github.com/apache/parquet-java/pull/3397
> >> >> >>> > > > > > >
> >> >> >>> > > > > > > Are cross language tested and range of test
> scenarios.
> >> >> >>> > > > > > >
> >> >> >>> > > > > > > Best
> >> >> >>> > > > > > > Prateek
> >> >> >>> > > > > > >
> >> >> >>> > > > > > >
> >> >> >>> > > > > > > On Mon, Jul 13, 2026 at 8:38 AM PRATEEK GAUR <
> >> >> >>> prtkgaur@gmail.com
> >> >> >>> > >
> >> >> >>> > > > > wrote:
> >> >> >>> > > > > > >
> >> >> >>> > > > > > >> Thanks Matt,
> >> >> >>> > > > > > >>
> >> >> >>> > > > > > >> I'll be on it this week.
> >> >> >>> > > > > > >>
> >> >> >>> > > > > > >> Best
> >> >> >>> > > > > > >> Prateek
> >> >> >>> > > > > > >>
> >> >> >>> > > > > > >> On Mon, Jul 13, 2026 at 8:36 AM Matt Topol <
> >> >> >>> > > zotthewizard@gmail.com>
> >> >> >>> > > > > > >> wrote:
> >> >> >>> > > > > > >>
> >> >> >>> > > > > > >>> There is also the Go implementation submitted by
> >> Arnav (
> >> >> >>> > > > > > >>> https://github.com/apache/arrow-go/pull/704) which
> >> is
> >> >> >>> waiting
> >> >> >>> > > for
> >> >> >>> > > > > > >>> updates.
> >> >> >>> > > > > > >>>
> >> >> >>> > > > > > >>> Just wanted to make sure it didn't get lost here.
> >> >> >>> > > > > > >>>
> >> >> >>> > > > > > >>> --Matt
> >> >> >>> > > > > > >>>
> >> >> >>> > > > > > >>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK GAUR <
> >> >> >>> > > prtkgaur@gmail.com>
> >> >> >>> > > > > > >>> wrote:
> >> >> >>> > > > > > >>>
> >> >> >>> > > > > > >>> > Hi all,
> >> >> >>> > > > > > >>> >
> >> >> >>> > > > > > >>> > I'd like to share a status update on the ALP
> >> encoding
> >> >> >>> effort
> >> >> >>> > > and
> >> >> >>> > > > > get
> >> >> >>> > > > > > a
> >> >> >>> > > > > > >>> feel
> >> >> >>> > > > > > >>> > from the community before starting a formal vote.
> >> >> >>> > > > > > >>> >
> >> >> >>> > > > > > >>> > We now have the specification plus
> implementations
> >> in
> >> >> >>> both
> >> >> >>> > > > > languages:
> >> >> >>> > > > > > >>> >
> >> >> >>> > > > > > >>> > - Spec: apache/parquet-format#557 —
> >> GH-533
> >> >> >>> Add
> >> >> >>> > ALP
> >> >> >>> > > > > > encoding
> >> >> >>> > > > > > >>> > specification
> >> >> >>> > > > > > >>> > - C++ (Arrow): apache/arrow#48345 — GH-48701
> >> >> >>> > > [C++][Parquet]
> >> >> >>> > > > > Add
> >> >> >>> > > > > > >>> ALPpd
> >> >> >>> > > > > > >>> > encoding
> >> >> >>> > > > > > >>> > - Java: apache/parquet-java#3397 —
> >> Parquet
> >> >> >>> Java
> >> >> >>> > ALP
> >> >> >>> > > > > > >>> > Implementation (by Vinoo)
> >> >> >>> > > > > > >>> >
> >> >> >>> > > > > > >>> > Current state:
> >> >> >>> > > > > > >>> >
> >> >> >>> > > > > > >>> > - The C++ implementation has been through 3-4
> >> >> rounds
> >> >> >>> of
> >> >> >>> > > > review.
> >> >> >>> > > > > > >>> > - The Java implementation has been tracking
> >> well
> >> >> and
> >> >> >>> has
> >> >> >>> > > had
> >> >> >>> > > > > > >>> initial
> >> >> >>> > > > > > >>> > reviews from contributors, with no major
> >> >> remaining
> >> >> >>> > > issues.
> >> >> >>> > > > > > >>> > - Cross-language compatibility tests are
> >> passing:
> >> >> the
> >> >> >>> > Arrow
> >> >> >>> > > > C++
> >> >> >>> > > > > > >>> decoder
> >> >> >>> > > > > > >>> > reads Java-written data bit-exactly across
> >> >> ~1.56M
> >> >> >>> > values
> >> >> >>> > > > and
> >> >> >>> > > > > 18
> >> >> >>> > > > > > >>> > fixtures, covering V1 and V2 pages,
> multiple
> >> >> vector
> >> >> >>> > > sizes,
> >> >> >>> > > > > and
> >> >> >>> > > > > > >>> > several
> >> >> >>> > > > > > >>> > real datasets — zero mismatches.
> >> >> >>> > > > > > >>> >
> >> >> >>> > > > > > >>> > As raised in the java-pr we will be adding
> >> coverage
> >> >> for
> >> >> >>> > > extreme
> >> >> >>> > > > > > values
> >> >> >>> > > > > > >>> > (those requiring
> >> >> >>> > > > > > >>> > 63-64 bits after FOR is applied) before we
> close a
> >> >> vote.
> >> >> >>> > We'll
> >> >> >>> > > > aim
> >> >> >>> > > > > > to
> >> >> >>> > > > > > >>> get
> >> >> >>> > > > > > >>> > that
> >> >> >>> > > > > > >>> > done in parallel.
> >> >> >>> > > > > > >>> >
> >> >> >>> > > > > > >>> > Plan: unless there are objections, I intend to
> >> start a
> >> >> >>> vote
> >> >> >>> > at
> >> >> >>> > > > the
> >> >> >>> > > > > > end
> >> >> >>> > > > > > >>> of
> >> >> >>> > > > > > >>> > this week/early
> >> >> >>> > > > > > >>> > next week.
> >> >> >>> > > > > > >>> >
> >> >> >>> > > > > > >>> > Thanks
> >> >> >>> > > > > > >>> > Prateek
> >> >> >>> > > > > > >>> >
> >> >> >>> > > > > > >>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <
> >> >> >>> > > prtkgaur@gmail.com>
> >> >> >>> > > > > > >>> wrote:
> >> >> >>> > > > > > >>> >
> >> >> >>> > > > > > >>> > > Hi Team,
> >> >> >>> > > > > > >>> > >
> >> >> >>> > > > > > >>> > > Just wanted to provide some updates on ALP.
> >> >> >>> > > > > > >>> > > Micah and I have done 3-4 rounds of review for
> >> the
> >> >> c++
> >> >> >>> PR :
> >> >> >>> > > > > > >>> > > https://github.com/apache/arrow/pull/48345
> >> >> >>> > > > > > >>> > >
> >> >> >>> > > > > > >>> > > For the Java implementation Vinoo has been
> >> working
> >> >> on
> >> >> >>> > > following
> >> >> >>> > > > > PR
> >> >> >>> > > > > > :
> >> >> >>> > > > > > >>> > >
> https://github.com/apache/parquet-java/pull/3397
> >> >> >>> > > > > > >>> > >
> >> >> >>> > > > > > >>> > > Best
> >> >> >>> > > > > > >>> > > Prateek
> >> >> >>> > > > > > >>> > >
> >> >> >>> > > > > > >>> > > On Tue, May 5, 2026 at 1:50 PM Micah Kornfield
> <
> >> >> >>> > > > > > >>> emkornfield@gmail.com>
> >> >> >>> > > > > > >>> > > wrote:
> >> >> >>> > > > > > >>> > >
> >> >> >>> > > > > > >>> > >> Hi Antoine,
> >> >> >>> > > > > > >>> > >>
> >> >> >>> > > > > > >>> > >> > Apologies if the question was already asked,
> >> but
> >> >> >>> should
> >> >> >>> > we
> >> >> >>> > > > > care
> >> >> >>> > > > > > >>> about
> >> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more
> >> >> efficient
> >> >> >>> than
> >> >> >>> > > > > FLOAT16
> >> >> >>> > > > > > +
> >> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> >> >> >>> > > > > > >>> > >>
> >> >> >>> > > > > > >>> > >> It was. We thought we could defer it for the
> >> >> >>> following
> >> >> >>> > > > reasons:
> >> >> >>> > > > > > >>> > >> 1. It's not clear there are a lot of easy
> >> >> reference
> >> >> >>> > > datasets
> >> >> >>> > > > to
> >> >> >>> > > > > > >>> test
> >> >> >>> > > > > > >>> > its
> >> >> >>> > > > > > >>> > >> effectiveness. It does look like there might
> be
> >> >> one
> >> >> >>> or
> >> >> >>> > two
> >> >> >>> > > on
> >> >> >>> > > > > > >>> > huggingface
> >> >> >>> > > > > > >>> > >> (e.g.
> >> >> https://huggingface.co/datasets/kikitora/curdie
> >> >> >>> ).
> >> >> >>> > > > > > >>> > >> 2. It seemed likely that float 16 was more
> >> likely
> >> >> >>> used
> >> >> >>> > for
> >> >> >>> > > > > values
> >> >> >>> > > > > > >>> that
> >> >> >>> > > > > > >>> > >> were less likely to reduce to decimal values.
> >> >> >>> > > > > > >>> > >> 3. It could be added as an extension later if
> >> >> needed.
> >> >> >>> > > > > > >>> > >>
> >> >> >>> > > > > > >>> > >> Cheers,
> >> >> >>> > > > > > >>> > >> Micah
> >> >> >>> > > > > > >>> > >>
> >> >> >>> > > > > > >>> > >>
> >> >> >>> > > > > > >>> > >> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou
> <
> >> >> >>> > > > > antoine@python.org
> >> >> >>> > > > > > >
> >> >> >>> > > > > > >>> > wrote:
> >> >> >>> > > > > > >>> > >>
> >> >> >>> > > > > > >>> > >> >
> >> >> >>> > > > > > >>> > >> > Hello,
> >> >> >>> > > > > > >>> > >> >
> >> >> >>> > > > > > >>> > >> > Apologies if the question was already asked,
> >> but
> >> >> >>> should
> >> >> >>> > we
> >> >> >>> > > > > care
> >> >> >>> > > > > > >>> about
> >> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more
> >> >> efficient
> >> >> >>> than
> >> >> >>> > > > > FLOAT16
> >> >> >>> > > > > > +
> >> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> >> >> >>> > > > > > >>> > >> >
> >> >> >>> > > > > > >>> > >> > Regards
> >> >> >>> > > > > > >>> > >> >
> >> >> >>> > > > > > >>> > >> > Antoine.
> >> >> >>> > > > > > >>> > >> >
> >> >> >>> > > > > > >>> > >> >
> >> >> >>> > > > > > >>> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit
> :
> >> >> >>> > > > > > >>> > >> > > Thanks Andrew and Micah for review
> feedback
> >> on
> >> >> >>> the two
> >> >> >>> > > > PR's
> >> >> >>> > > > > > >>> > >> > > 1) (c++ arrow repo)
> >> >> >>> > > > > > >>> > >>
> >> https://github.com/apache/arrow/pull/48345/changes
> >> >> >>> > > > > > >>> > >> > > 2) (parquet-format repo)
> >> >> >>> > > > > > >>> > >> >
> >> >> https://github.com/apache/parquet-format/pull/557
> >> >> >>> > > > > > >>> > >> > >
> >> >> >>> > > > > > >>> > >> > > I have addressed all (unless I missed
> >> >> something)
> >> >> >>> > > comments
> >> >> >>> > > > on
> >> >> >>> > > > > > >>> the two
> >> >> >>> > > > > > >>> > >> > PR's.
> >> >> >>> > > > > > >>> > >> > >
> >> >> >>> > > > > > >>> > >> > > Best
> >> >> >>> > > > > > >>> > >> > > Prateek
> >> >> >>> > > > > > >>> > >> > >
> >> >> >>> > > > > > >>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK
> >> GAUR <
> >> >> >>> > > > > > >>> prtkgaur@gmail.com>
> >> >> >>> > > > > > >>> > >> wrote:
> >> >> >>> > > > > > >>> > >> > >
> >> >> >>> > > > > > >>> > >> > >> Thanks Andrew and Micah.
> >> >> >>> > > > > > >>> > >> > >>
> >> >> >>> > > > > > >>> > >> > >> `fair amount of feedback on at least the
> >> >> >>> > > implementations`
> >> >> >>> > > > > > >>> > >> > >> For the c++ I have already started
> >> addressing
> >> >> the
> >> >> >>> > > > > feedback, I
> >> >> >>> > > > > > >>> > should
> >> >> >>> > > > > > >>> > >> be
> >> >> >>> > > > > > >>> > >> > >> done with that Monday/Tuesday.
> >> >> >>> > > > > > >>> > >> > >> I think Vinoo too has been making good
> >> >> progress
> >> >> >>> on
> >> >> >>> > the
> >> >> >>> > > > Java
> >> >> >>> > > > > > >>> > >> > implementation.
> >> >> >>> > > > > > >>> > >> > >>
> >> >> >>> > > > > > >>> > >> > >> Best
> >> >> >>> > > > > > >>> > >> > >> Prateek
> >> >> >>> > > > > > >>> > >> > >>
> >> >> >>> > > > > > >>> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew
> >> Lamb <
> >> >> >>> > > > > > >>> > andrewlamb11@gmail.com
> >> >> >>> > > > > > >>> > >> >
> >> >> >>> > > > > > >>> > >> > >> wrote:
> >> >> >>> > > > > > >>> > >> > >>
> >> >> >>> > > > > > >>> > >> > >>> Got it. Thank you for the clarification
> >> -- I
> >> >> >>> will
> >> >> >>> > try
> >> >> >>> > > > and
> >> >> >>> > > > > > look
> >> >> >>> > > > > > >>> > into
> >> >> >>> > > > > > >>> > >> the
> >> >> >>> > > > > > >>> > >> > >>> spec and the Rust implementation[1] in
> >> this
> >> >> next
> >> >> >>> > week
> >> >> >>> > > > > > >>> > >> > >>>
> >> >> >>> > > > > > >>> > >> > >>> [1]:
> >> >> >>> https://github.com/apache/arrow-rs/pull/9372
> >> >> >>> > > > > > >>> > >> > >>>
> >> >> >>> > > > > > >>> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah
> >> >> >>> Kornfield <
> >> >> >>> > > > > > >>> > >> > emkornfield@gmail.com>
> >> >> >>> > > > > > >>> > >> > >>> wrote:
> >> >> >>> > > > > > >>> > >> > >>>
> >> >> >>> > > > > > >>> > >> > >>>> Hi Andrew,
> >> >> >>> > > > > > >>> > >> > >>>> I think there is a fair amount of
> >> feedback
> >> >> on
> >> >> >>> at
> >> >> >>> > > least
> >> >> >>> > > > > the
> >> >> >>> > > > > > >>> > >> > >>>> implementations, typically I think
> we've
> >> >> waited
> >> >> >>> > till
> >> >> >>> > > > they
> >> >> >>> > > > > > are
> >> >> >>> > > > > > >>> > >> close to
> >> >> >>> > > > > > >>> > >> > >>>> mergeable before a final vote.
> >> Otherwise I
> >> >> >>> agree
> >> >> >>> > we
> >> >> >>> > > > are
> >> >> >>> > > > > > very
> >> >> >>> > > > > > >>> > >> close.
> >> >> >>> > > > > > >>> > >> > >>>>
> >> >> >>> > > > > > >>> > >> > >>>> -Micah
> >> >> >>> > > > > > >>> > >> > >>>>
> >> >> >>> > > > > > >>> > >> > >>>> On Saturday, April 25, 2026, Andrew
> Lamb
> >> <
> >> >> >>> > > > > > >>> andrewlamb11@gmail.com
> >> >> >>> > > > > > >>> > >
> >> >> >>> > > > > > >>> > >> > wrote:
> >> >> >>> > > > > > >>> > >> > >>>>
> >> >> >>> > > > > > >>> > >> > >>>>> Thanks Prateek,
> >> >> >>> > > > > > >>> > >> > >>>>>
> >> >> >>> > > > > > >>> > >> > >>>>> I think from this content it looks to
> me
> >> >> like
> >> >> >>> we
> >> >> >>> > are
> >> >> >>> > > > > ready
> >> >> >>> > > > > > >>> to
> >> >> >>> > > > > > >>> > >> start a
> >> >> >>> > > > > > >>> > >> > >>>>> vote to explicitly accept ALP into
> >> Parquet
> >> >> >>> > > > > > >>> > >> > >>>>>
> >> >> >>> > > > > > >>> > >> > >>>>> Does anyone know of a reason we should
> >> >> >>> postpone it
> >> >> >>> > > for
> >> >> >>> > > > > > >>> longer?
> >> >> >>> > > > > > >>> > >> > >>>>> Perhaps someone needs some more time
> to
> >> >> >>> review?
> >> >> >>> > > > > > >>> > >> > >>>>>
> >> >> >>> > > > > > >>> > >> > >>>>> Andrew
> >> >> >>> > > > > > >>> > >> > >>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>
> >> >> >>> > > > > > >>> > >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM
> PRATEEK
> >> >> GAUR <
> >> >> >>> > > > > > >>> > prtkgaur@gmail.com>
> >> >> >>> > > > > > >>> > >> > >>>>> wrote:
> >> >> >>> > > > > > >>> > >> > >>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>> Hi team,
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>> Hope everyone is doing well. I got a
> >> >> chance
> >> >> >>> to
> >> >> >>> > work
> >> >> >>> > > > > > >>> through all
> >> >> >>> > > > > > >>> > >> the
> >> >> >>> > > > > > >>> > >> > >>>>>> remaining feedback and update the
> spec
> >> >> doc.
> >> >> >>> Here
> >> >> >>> > > are
> >> >> >>> > > > > the
> >> >> >>> > > > > > >>> new
> >> >> >>> > > > > > >>> > >> > artifacts
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>> 1) Spec document :
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> >
> >> >> >>> > > > > > >>> > >>
> >> >> >>> > > > > > >>> >
> >> >> >>> > > > > > >>>
> >> >> >>> > > > > >
> >> >> >>> > > > >
> >> >> >>> > > >
> >> >> >>> > >
> >> >> >>> >
> >> >> >>>
> >> >>
> >>
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>> 2) Spec document in parquet format
> >> repo :
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > https://github.com/apache/parquet-format/pull/557
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>> 3) Alp implementation in arrow c++
> >> repo :
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > https://github.com/apache/arrow/pull/48345/changes
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>> 4) Alp implementation in parquet-java
> >> >> repo :
> >> >> >>> Work
> >> >> >>> > > for
> >> >> >>> > > > > > >>> Vinoo and
> >> >> >>> > > > > > >>> > >> > Julien
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > https://github.com/apache/parquet-java/pull/3397
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>> 5) PR with test and benchmarking
> >> >> artifacts in
> >> >> >>> > > > > > >>> parquet-testing
> >> >> >>> > > > > > >>> > >> repo :
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > https://github.com/apache/parquet-testing/pull/100
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>> And
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>> - Go : Arnav just submitted an in
> >> >> >>> progress
> >> >> >>> > > > > > >>> implementation
> >> >> >>> > > > > > >>> > in
> >> >> >>> > > > > > >>> > >> Go.
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> https://github.com/apache/arrow-go/pull/704
> >> >> >>> > (I
> >> >> >>> > > > > > haven't
> >> >> >>> > > > > > >>> > >> started
> >> >> >>> > > > > > >>> > >> > >>>>>> looking at it yet)
> >> >> >>> > > > > > >>> > >> > >>>>>> - Rust : I remember Andrew
> >> mentioned
> >> >> that
> >> >> >>> > this
> >> >> >>> > > > work
> >> >> >>> > > > > > is
> >> >> >>> > > > > > >>> also
> >> >> >>> > > > > > >>> > >> in
> >> >> >>> > > > > > >>> > >> > >>>>>> progress (So 4 languages!)
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>> *Arrow C++ implementation *
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>> The PR is out and was also used by
> >> >> Antoine to
> >> >> >>> > > report
> >> >> >>> > > > > the
> >> >> >>> > > > > > >>> > numbers
> >> >> >>> > > > > > >>> > >> as
> >> >> >>> > > > > > >>> > >> > >>>>>> reported here. Micah and Konstantin
> >> have
> >> >> >>> given 1
> >> >> >>> > > > round
> >> >> >>> > > > > of
> >> >> >>> > > > > > >>> > >> feedback
> >> >> >>> > > > > > >>> > >> > >>>>>> and I'm addressing them today. Please
> >> note
> >> >> >>> that
> >> >> >>> > the
> >> >> >>> > > > > > default
> >> >> >>> > > > > > >>> > >> > >>>>>> optimization flag for compiling is O2
> >> and
> >> >> not
> >> >> >>> > Q3. I
> >> >> >>> > > > got
> >> >> >>> > > > > > >>> around
> >> >> >>> > > > > > >>> > >> 70%
> >> >> >>> > > > > > >>> > >> > >>>>>> performance improvement in the
> decoding
> >> >> speed
> >> >> >>> > when
> >> >> >>> > > > > using
> >> >> >>> > > > > > >>> the O3
> >> >> >>> > > > > > >>> > >> > flag.
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>> *Parqet-MR Java implementation
> (working
> >> >> with
> >> >> >>> > Vinoo
> >> >> >>> > > > and
> >> >> >>> > > > > > >>> Julien)
> >> >> >>> > > > > > >>> > >> and
> >> >> >>> > > > > > >>> > >> > **Cross
> >> >> >>> > > > > > >>> > >> > >>>>>> Language testing*
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>> Let me know if you have any
> >> questions
> >> >> or
> >> >> >>> > > > feedback.
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>> Now pasting some performance numbers
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>> Table 1: C++ ALP Double Decode —
> >> >> Spotify
> >> >> >>> > Columns
> >> >> >>> > > > > > >>> (Graviton
> >> >> >>> > > > > > >>> > 3,
> >> >> >>> > > > > > >>> > >> ARM
> >> >> >>> > > > > > >>> > >> > >>>>>> Neoverse V1)
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>>
> >> >> >>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>> │ Column │ -O2 (MB/s)
> │
> >> >> -O3
> >> >> >>> > > (MB/s) │
> >> >> >>> > > > > > >>> Speedup │
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>>
> >> >> >>> ├──────────────────┼──────────────┼──────────────┼─────────┤
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>> │ valence │ 3,155
> │
> >> >> >>> 5,523
> >> >> >>> > > │
> >> >> >>> > > > > > >>> 1.75x │
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>> │ danceability │ 3,233
> │
> >> >> >>> 5,685
> >> >> >>> > > │
> >> >> >>> > > > > > >>> 1.76x │
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>> │ energy │ 3,197
> │
> >> >> >>> 5,652
> >> >> >>> > > │
> >> >> >>> > > > > > >>> 1.77x │
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>> │ loudness │ 3,186
> │
> >> >> >>> 5,473
> >> >> >>> > > │
> >> >> >>> > > > > > >>> 1.72x │
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>>
> >> >> >>> └──────────────────┴──────────────┴──────────────┴─────────┘
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM
> PRATEEK
> >> >> GAUR
> >> >> >>> <
> >> >> >>> > > > > > >>> > prtkgaur@gmail.com
> >> >> >>> > > > > > >>> > >> >
> >> >> >>> > > > > > >>> > >> > >>>>>> wrote:
> >> >> >>> > > > > > >>> > >> > >>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>> @Micah Kornfield <
> >> emkornfield@gmail.com>
> >> >> :
> >> >> >>> Got
> >> >> >>> > > it.
> >> >> >>> > > > > > >>> > >> > >>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>> @Andrew Lamb <
> andrewlamb11@gmail.com>
> >> >> >>> > > > > > >>> > >> > >>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>> Do you think it would be good to
> >> start
> >> >> >>> moving
> >> >> >>> > the
> >> >> >>> > > > > spec
> >> >> >>> > > > > > >>> > >> development
> >> >> >>> > > > > > >>> > >> > >>>>>>>> into
> >> >> >>> > > > > > >>> > >> > >>>>>>>> markdown format, in preparation for
> >> >> >>> finalizing
> >> >> >>> > > it?
> >> >> >>> > > > > > >>> > >> > >>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>> Yes I'll update the numbers for some
> >> of
> >> >> the
> >> >> >>> > > > examples I
> >> >> >>> > > > > > >>> have in
> >> >> >>> > > > > > >>> > >> the
> >> >> >>> > > > > > >>> > >> > >>>>>>> spec based
> >> >> >>> > > > > > >>> > >> > >>>>>>> on the updated header size. Then we
> >> >> should
> >> >> >>> be
> >> >> >>> > good
> >> >> >>> > > > to
> >> >> >>> > > > > go
> >> >> >>> > > > > > >>> for
> >> >> >>> > > > > > >>> > the
> >> >> >>> > > > > > >>> > >> > >>>>>>> markdown format.
> >> >> >>> > > > > > >>> > >> > >>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>> Thanks everyone!
> >> >> >>> > > > > > >>> > >> > >>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>> Andrew
> >> >> >>> > > > > > >>> > >> > >>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM
> >> PRATEEK
> >> >> >>> GAUR <
> >> >> >>> > > > > > >>> > >> prtkgaur@gmail.com>
> >> >> >>> > > > > > >>> > >> > >>>>>>>> wrote:
> >> >> >>> > > > > > >>> > >> > >>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> Hi team,
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> 1) Andrew
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for working on test
> >> files.
> >> >> >>> My PR
> >> >> >>> > > did
> >> >> >>> > > > > add
> >> >> >>> > > > > > >>> all
> >> >> >>> > > > > > >>> > the
> >> >> >>> > > > > > >>> > >> > test
> >> >> >>> > > > > > >>> > >> > >>>>>>>> files I
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> used to benchmark on datasets.
> >> >> Maybe
> >> >> >>> we
> >> >> >>> > can
> >> >> >>> > > > club
> >> >> >>> > > > > > it
> >> >> >>> > > > > > >>> > >> together.
> >> >> >>> > > > > > >>> > >> > >>>>>>>> WIll also
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> aid
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> cross language testing
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Kosta Tarasov working on
> Rust
> >> >> >>> > > > implementation.
> >> >> >>> > > > > > >>> This is
> >> >> >>> > > > > > >>> > >> > great.
> >> >> >>> > > > > > >>> > >> > >>>>>>>> Thanks
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> 2) Antoine
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks a lot for reporting
> the
> >> >> >>> numbers
> >> >> >>> > on
> >> >> >>> > > > AMD.
> >> >> >>> > > > > > >>> Looks
> >> >> >>> > > > > > >>> > >> like
> >> >> >>> > > > > > >>> > >> > you
> >> >> >>> > > > > > >>> > >> > >>>>>>>> are
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> getting 8X the decoding
> >> >> performance of
> >> >> >>> > BSS.
> >> >> >>> > > > This
> >> >> >>> > > > > > is
> >> >> >>> > > > > > >>> > >> > amazing!!.
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for acknowledging the
> >> >> >>> sampling
> >> >> >>> > > > design.
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> - I agree with you on
> >> Fastlanes. In
> >> >> >>> some
> >> >> >>> > > crude
> >> >> >>> > > > > > >>> > >> experiments I
> >> >> >>> > > > > > >>> > >> > >>>>>>>> didn't get
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> a good perf benefit from it on
> >> >> >>> Graviton3
> >> >> >>> > > (but
> >> >> >>> > > > > > maybe
> >> >> >>> > > > > > >>> > there
> >> >> >>> > > > > > >>> > >> was
> >> >> >>> > > > > > >>> > >> > >>>>>>>> something
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> wrong with my implementation).
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Locking the 16bit exception
> >> >> >>> encoding for
> >> >> >>> > > the
> >> >> >>> > > > > > spec
> >> >> >>> > > > > > >>> in
> >> >> >>> > > > > > >>> > >> this
> >> >> >>> > > > > > >>> > >> > >>>>>>>> case.
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> - Awesome I think we have
> solved
> >> >> for
> >> >> >>> all
> >> >> >>> > > open
> >> >> >>> > > > > > >>> questions
> >> >> >>> > > > > > >>> > >> minus
> >> >> >>> > > > > > >>> > >> > >>>>>>>> the
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> version byte :). (will get
> back
> >> on
> >> >> >>> this
> >> >> >>> > > soon)
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> 3) Micah
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> - FastLanes : The current spec
> >> does
> >> >> >>> allow
> >> >> >>> > > for
> >> >> >>> > > > > > using
> >> >> >>> > > > > > >>> > >> FastLane
> >> >> >>> > > > > > >>> > >> > >>>>>>>> with the
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> configurable enum value for
> >> >> layout. We
> >> >> >>> > > should
> >> >> >>> > > > be
> >> >> >>> > > > > > >>> able to
> >> >> >>> > > > > > >>> > >> > inject
> >> >> >>> > > > > > >>> > >> > >>>>>>>> any
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> layout
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> in the current design.
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> Working on resolving all remaining
> >> open
> >> >> >>> > comments
> >> >> >>> > > > on
> >> >> >>> > > > > > the
> >> >> >>> > > > > > >>> spec
> >> >> >>> > > > > > >>> > >> this
> >> >> >>> > > > > > >>> > >> > >>>>>>>> week.
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> Best
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> Prateek
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM
> >> Steve
> >> >> >>> > Loughran <
> >> >> >>> > > > > > >>> > >> > >>>>>>>> stevel@cloudera.com>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> wrote:
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12,
> Micah
> >> >> >>> Kornfield
> >> >> >>> > <
> >> >> >>> > > > > > >>> > >> > >>>>>>>> emkornfield@gmail.com>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>> wrote:
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> It looks like the actual issue
> >> >> >>> described for
> >> >> >>> > > ORC
> >> >> >>> > > > > in
> >> >> >>> > > > > > >>> the
> >> >> >>> > > > > > >>> > >> paper
> >> >> >>> > > > > > >>> > >> > >>>>>>>> is that
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> it
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> has multiple sub-encodings in a
> >> >> batch.
> >> >> >>> This
> >> >> >>> > > is
> >> >> >>> > > > > > >>> different
> >> >> >>> > > > > > >>> > >> then
> >> >> >>> > > > > > >>> > >> > >>>>>>>> the
> >> >> >>> > > > > > >>> > >> > >>>>>>>>> design
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> proposed here where there is
> still
> >> >> fixed
> >> >> >>> > > > encoding
> >> >> >>> > > > > > per
> >> >> >>> > > > > > >>> page
> >> >> >>> > > > > > >>> > >> in
> >> >> >>> > > > > > >>> > >> > >>>>>>>> parquet.
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> Given reasonably sized pages I
> >> don't
> >> >> >>> think
> >> >> >>> > > > branch
> >> >> >>> > > > > > >>> > >> > >>>>>>>> misprediction should
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>> be a
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> big issue for new encodings. I
> >> agree
> >> >> >>> that
> >> >> >>> > we
> >> >> >>> > > > > should
> >> >> >>> > > > > > >>> be
> >> >> >>> > > > > > >>> > >> > >>>>>>>> conservative in
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>>> general for adding new
> encodings.
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>> +1
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>>
> >> >> >>> > > > > > >>> > >> > >>>>>>>
> >> >> >>> > > > > > >>> > >> > >
> >> >> >>> > > > > > >>> > >> >
> >> >> >>> > > > > > >>> > >> >
> >> >> >>> > > > > > >>> > >> >
> >> >> >>> > > > > > >>> > >>
> >> >> >>> > > > > > >>> > >
> >> >> >>> > > > > > >>> >
> >> >> >>> > > > > > >>>
> >> >> >>> > > > > > >>
> >> >> >>> > > > > >
> >> >> >>> > > > >
> >> >> >>> > > >
> >> >> >>> > >
> >> >> >>> >
> >> >> >>>
> >> >> >>
> >> >>
> >> >
> >>
> >
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by PRATEEK GAUR <pr...@gmail.com>.
Hi team,
Thanks Micah for multiple rounds of review and thanks Gang for taking a
look at the PR and giving feedback. I've addressed all comments in the PR.
Looking forward to the next round of feedback (or approval). Would like to
get it in soon.
https://github.com/apache/arrow/pull/48345/changes
With this done and after talking to Arnav, I plan to spend some time on the
ALP implementation in the arrow-go repository helping spread the support to
more languages.
Best
Prateek
On Sun, Aug 2, 2026 at 10:42 PM PRATEEK GAUR <pr...@gmail.com> wrote:
> Thanks Divjot for bringing it up.
>
> I was under the same impression as Russell and Andrew that we'd have it
> flag gated.
> With the new versioning proposal 'preview' might be a good tag for it.
>
> Best
> Prateek
>
> On Sat, Aug 1, 2026 at 12:39 AM Andrew Lamb <an...@gmail.com>
> wrote:
>
>> Div: Here is a proposal to mark ALP in Preview: [1] (using a comment in
>> the
>> spec section).
>>
>> Russel: That is my understanding as well.
>>
>> Hopefully we are able to formalize the versioning process shortly so we
>> can
>> formalize the notion of "preview"
>>
>> Andrew
>>
>>
>> [1]: https://github.com/apache/parquet-format/pull/604
>>
>> On Fri, Jul 31, 2026 at 11:15 AM Russell Spitzer <
>> russell.spitzer@gmail.com>
>> wrote:
>>
>> > I thought we had agreed it would be merged, and features like this would
>> > be feature flag gated. So a V-Old writer could always opt in to using
>> this
>> > encoding but that must be a conscious effort if you are not using VNext
>> of
>> > Parquet. If you are on V-Next the writer can choose to use ALP whenever
>> it
>> > likes without user intervention.
>> >
>> > On Fri, Jul 31, 2026 at 2:20 AM Divjot Arora via dev <
>> > dev@parquet.apache.org> wrote:
>> >
>> >> Yes, that’s my understanding as well. However, as you noted, the
>> >> versioning
>> >> discussion is still in early stages and hasn’t prescribed concrete
>> >> mechanisms yet. If ALP goes into parquet-format 2.14.0, there’s no
>> >> indication that it’s in preview or that writers shouldn’t use it by
>> >> default
>> >> yet. Given that we’ve held off on merging other incompatible changes
>> (e.g
>> >> removal of path_in_schema) until the versioning discussion closes, my
>> >> preference would be to leave ALP un-merged for the time being as well.
>> >>
>> >> — Div
>> >>
>> >> On Fri, Jul 31, 2026 at 08:59 Andrew Lamb <an...@gmail.com>
>> wrote:
>> >>
>> >> > After re-reading Julien's Versioning Proposal, my understanding is
>> that
>> >> if
>> >> > we adopt that system, we will somehow mark ALP in "Preview" in
>> >> > parquet-format for release Version 3.
>> >> >
>> >> > The versioning document doesn't seem to proscribe the particular
>> method
>> >> > for annotating "preview" features -- I think clearly marking them as
>> >> such
>> >> > in parquet.thrift as well as reflecting that on the
>> >> > https://parquet.apache.org/docs/file-format/versions/ page somehow
>> >> would
>> >> > be adequate.
>> >> >
>> >> > Andrew
>> >> >
>> >> > On Thu, Jul 30, 2026 at 11:50 AM Andrew Lamb <andrewlamb11@gmail.com
>> >
>> >> > wrote:
>> >> >
>> >> >> In my mind ALP is in the same position as other encodings that were
>> >> added
>> >> >> recently[1] (such as BYTE_STREAM_SPLIT for non floating point types)
>> >> >>
>> >> >> That is to say it will be available in the latest open source
>> writers
>> >> and
>> >> >> readers, but realistically not widely used in the ecosystem where
>> >> >> compatibility is a concern (a large part)
>> >> >>
>> >> >> I view Julian's work[2][3] to gather consensus on how to name
>> >> >> combinations of features (e.g. "versions") as the mechanism by
>> which we
>> >> >> will accelerate the adoption of this change (along with other
>> forward
>> >> >> incompatible changes) across the ecosystem.
>> >> >>
>> >> >> > Will we exclude this change when cutting parquet-format v2.14.0?
>> >> >>
>> >> >> No I don't think so -- I would expect it to go out in the next
>> >> >> parquet-format version as do other changes in the spec.
>> >> >>
>> >> >> Andrew
>> >> >>
>> >> >> [1]: https://parquet.apache.org/docs/file-format/versions/
>> >> >> [2]:
>> https://lists.apache.org/thread/xmnj8h0h8ozmrgox5tydhhs7yz7q2hss
>> >> >> [3]:
>> https://lists.apache.org/thread/qlf8lg90gqq41lllyy9mk2f0skqvftqj
>> >> >>
>> >> >>
>> >> >> On Thu, Jul 30, 2026 at 3:51 AM Divjot Arora via dev <
>> >> >> dev@parquet.apache.org> wrote:
>> >> >>
>> >> >>> Hi Andrew,
>> >> >>>
>> >> >>> How does this interact with the ongoing versioning discussion?
>> Given
>> >> that
>> >> >>> new encodings are not forward compatible, I would assume ALP is
>> >> included
>> >> >>> in
>> >> >>> Parquet V<next> (V3?), but it's not clear to me how that works if
>> it's
>> >> >>> merged to the parquet-format master branch now. Will we exclude
>> this
>> >> >>> change
>> >> >>> when cutting parquet-format v2.14.0?
>> >> >>>
>> >> >>> -- Div
>> >> >>>
>> >> >>> On Thu, Jul 30, 2026 at 8:51 AM Andrew Lamb <
>> andrewlamb11@gmail.com>
>> >> >>> wrote:
>> >> >>>
>> >> >>> > Hello,
>> >> >>> >
>> >> >>> > As an update, thanks to the epic work from many contributors,
>> >> >>> especially
>> >> >>> > Prateek, I just merged the parquet-format change for ALP[1]. We
>> >> plan to
>> >> >>> > make additional editorial The remaining items I know of are
>> merging
>> >> the
>> >> >>> > examples from parquet-testing[2], and then of course the various
>> >> >>> > implementations
>> >> >>> >
>> >> >>> > Thanks again everyone. This is a great step forward
>> >> >>> > Andrew
>> >> >>> >
>> >> >>> > [1]: https://github.com/apache/parquet-format/pull/557
>> >> >>> > [2]: https://github.com/apache/parquet-testing/pull/100
>> >> >>> >
>> >> >>> > On Tue, Jul 21, 2026 at 4:39 PM Julien Le Dem <julien@apache.org
>> >
>> >> >>> wrote:
>> >> >>> >
>> >> >>> > > It was also in my spam folder... :(
>> >> >>> > > I've seen a bunch of apache list emails go to my spam recently.
>> >> >>> > >
>> >> >>> > > On Tue, Jul 21, 2026 at 10:58 AM Russell Spitzer <
>> >> >>> > > russell.spitzer@gmail.com>
>> >> >>> > > wrote:
>> >> >>> > >
>> >> >>> > > > Yes, my hope was that my vote would kick it out of spam but
>> we
>> >> >>> > definitely
>> >> >>> > > > had two other votes that also went to spam for me.
>> >> >>> > > >
>> >> >>> > > > For anyone looking the thread is here
>> >> >>> > > >
>> >> https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt
>> >> >>> > > >
>> >> >>> > > > On Tue, Jul 21, 2026 at 10:57 AM Micah Kornfield <
>> >> >>> > emkornfield@gmail.com>
>> >> >>> > > > wrote:
>> >> >>> > > >
>> >> >>> > > > > I've seen other's votes not go to spam, so hopefully people
>> >> will
>> >> >>> see
>> >> >>> > it
>> >> >>> > > > in
>> >> >>> > > > > there inboxes (or see this thread and look in there spam).
>> >> >>> > > > >
>> >> >>> > > > > On Tue, Jul 21, 2026 at 10:52 AM PRATEEK GAUR <
>> >> >>> prtkgaur@gmail.com>
>> >> >>> > > > wrote:
>> >> >>> > > > >
>> >> >>> > > > > > Okay,
>> >> >>> > > > > >
>> >> >>> > > > > > I sent out the voting email yesterday and so far I've
>> heard
>> >> >>> from 3
>> >> >>> > > > people
>> >> >>> > > > > > that it ended up in their "spam folder". Dev VPN to be
>> >> blamed
>> >> >>> for
>> >> >>> > > it?.
>> >> >>> > > > > >
>> >> >>> > > > > > What's the protocol here?
>> >> >>> > > > > > I let people hunt for the email in their spam folder or
>> send
>> >> >>> out
>> >> >>> > > > another
>> >> >>> > > > > > one?
>> >> >>> > > > > >
>> >> >>> > > > > > Best
>> >> >>> > > > > > Prateek
>> >> >>> > > > > >
>> >> >>> > > > > > On Sun, Jul 19, 2026 at 10:47 PM PRATEEK GAUR <
>> >> >>> prtkgaur@gmail.com>
>> >> >>> > > > > wrote:
>> >> >>> > > > > >
>> >> >>> > > > > > > Hi team,
>> >> >>> > > > > > >
>> >> >>> > > > > > > Vinoo has added the tests which were requested in the
>> last
>> >> >>> round
>> >> >>> > of
>> >> >>> > > > > > review.
>> >> >>> > > > > > > In his own words.
>> >> >>> > > > > > >
>> >> >>> > > > > > > "
>> >> >>> > > > > > > The extreme value tests Micah wanted are done and
>> pushed.
>> >> I
>> >> >>> added
>> >> >>> > > > > > coverage
>> >> >>> > > > > > > for values that need the full FOR bit width after the
>> >> frame
>> >> >>> of
>> >> >>> > > > > reference
>> >> >>> > > > > > is
>> >> >>> > > > > > > applied: 64-bit (the case where the signed max minus
>> min
>> >> >>> > > subtraction
>> >> >>> > > > > > > overflows), 63-bit (the non-overflow case), and 32-bit
>> for
>> >> >>> > floats.
>> >> >>> > > > All
>> >> >>> > > > > > > lossless with zero exceptions. I also verified them
>> >> through
>> >> >>> Arrow
>> >> >>> > > C++
>> >> >>> > > > > ALP
>> >> >>> > > > > > > reader (the local build), so the Java-written extreme
>> >> columns
>> >> >>> > > decode
>> >> >>> > > > > > > correctly in C++, bit-exact against the expected
>> values."
>> >> >>> > > > > > >
>> >> >>> > > > > > > State of the PR (#3397):
>> >> >>> > > > > > >
>> >> >>> > > > > > > - Merges cleanly with master (I pulled master in and
>> >> >>> resolved
>> >> >>> > > the
>> >> >>> > > > > > > conflicts).
>> >> >>> > > > > > > - 41 of 44 review threads resolved. The 3 remaining
>> are
>> >> >>> > > > intentional:
>> >> >>> > > > > > > one deferred perf nit, one that stays open until
>> >> >>> > parquet-format
>> >> >>> > > > > ships
>> >> >>> > > > > > ALP,
>> >> >>> > > > > > >
>> >> >>> > > > > > > "
>> >> >>> > > > > > >
>> >> >>> > > > > > > With this both the
>> >> >>> > > > > > > 1) C++ PR :
>> >> >>> https://github.com/apache/arrow/pull/48345/changes
>> >> >>> > > > > > > 2) Java PR :
>> >> >>> https://github.com/apache/parquet-java/pull/3397
>> >> >>> > > > > > >
>> >> >>> > > > > > > Are cross language tested and range of test scenarios.
>> >> >>> > > > > > >
>> >> >>> > > > > > > Best
>> >> >>> > > > > > > Prateek
>> >> >>> > > > > > >
>> >> >>> > > > > > >
>> >> >>> > > > > > > On Mon, Jul 13, 2026 at 8:38 AM PRATEEK GAUR <
>> >> >>> prtkgaur@gmail.com
>> >> >>> > >
>> >> >>> > > > > wrote:
>> >> >>> > > > > > >
>> >> >>> > > > > > >> Thanks Matt,
>> >> >>> > > > > > >>
>> >> >>> > > > > > >> I'll be on it this week.
>> >> >>> > > > > > >>
>> >> >>> > > > > > >> Best
>> >> >>> > > > > > >> Prateek
>> >> >>> > > > > > >>
>> >> >>> > > > > > >> On Mon, Jul 13, 2026 at 8:36 AM Matt Topol <
>> >> >>> > > zotthewizard@gmail.com>
>> >> >>> > > > > > >> wrote:
>> >> >>> > > > > > >>
>> >> >>> > > > > > >>> There is also the Go implementation submitted by
>> Arnav (
>> >> >>> > > > > > >>> https://github.com/apache/arrow-go/pull/704) which
>> is
>> >> >>> waiting
>> >> >>> > > for
>> >> >>> > > > > > >>> updates.
>> >> >>> > > > > > >>>
>> >> >>> > > > > > >>> Just wanted to make sure it didn't get lost here.
>> >> >>> > > > > > >>>
>> >> >>> > > > > > >>> --Matt
>> >> >>> > > > > > >>>
>> >> >>> > > > > > >>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK GAUR <
>> >> >>> > > prtkgaur@gmail.com>
>> >> >>> > > > > > >>> wrote:
>> >> >>> > > > > > >>>
>> >> >>> > > > > > >>> > Hi all,
>> >> >>> > > > > > >>> >
>> >> >>> > > > > > >>> > I'd like to share a status update on the ALP
>> encoding
>> >> >>> effort
>> >> >>> > > and
>> >> >>> > > > > get
>> >> >>> > > > > > a
>> >> >>> > > > > > >>> feel
>> >> >>> > > > > > >>> > from the community before starting a formal vote.
>> >> >>> > > > > > >>> >
>> >> >>> > > > > > >>> > We now have the specification plus implementations
>> in
>> >> >>> both
>> >> >>> > > > > languages:
>> >> >>> > > > > > >>> >
>> >> >>> > > > > > >>> > - Spec: apache/parquet-format#557 —
>> GH-533
>> >> >>> Add
>> >> >>> > ALP
>> >> >>> > > > > > encoding
>> >> >>> > > > > > >>> > specification
>> >> >>> > > > > > >>> > - C++ (Arrow): apache/arrow#48345 — GH-48701
>> >> >>> > > [C++][Parquet]
>> >> >>> > > > > Add
>> >> >>> > > > > > >>> ALPpd
>> >> >>> > > > > > >>> > encoding
>> >> >>> > > > > > >>> > - Java: apache/parquet-java#3397 —
>> Parquet
>> >> >>> Java
>> >> >>> > ALP
>> >> >>> > > > > > >>> > Implementation (by Vinoo)
>> >> >>> > > > > > >>> >
>> >> >>> > > > > > >>> > Current state:
>> >> >>> > > > > > >>> >
>> >> >>> > > > > > >>> > - The C++ implementation has been through 3-4
>> >> rounds
>> >> >>> of
>> >> >>> > > > review.
>> >> >>> > > > > > >>> > - The Java implementation has been tracking
>> well
>> >> and
>> >> >>> has
>> >> >>> > > had
>> >> >>> > > > > > >>> initial
>> >> >>> > > > > > >>> > reviews from contributors, with no major
>> >> remaining
>> >> >>> > > issues.
>> >> >>> > > > > > >>> > - Cross-language compatibility tests are
>> passing:
>> >> the
>> >> >>> > Arrow
>> >> >>> > > > C++
>> >> >>> > > > > > >>> decoder
>> >> >>> > > > > > >>> > reads Java-written data bit-exactly across
>> >> ~1.56M
>> >> >>> > values
>> >> >>> > > > and
>> >> >>> > > > > 18
>> >> >>> > > > > > >>> > fixtures, covering V1 and V2 pages, multiple
>> >> vector
>> >> >>> > > sizes,
>> >> >>> > > > > and
>> >> >>> > > > > > >>> > several
>> >> >>> > > > > > >>> > real datasets — zero mismatches.
>> >> >>> > > > > > >>> >
>> >> >>> > > > > > >>> > As raised in the java-pr we will be adding
>> coverage
>> >> for
>> >> >>> > > extreme
>> >> >>> > > > > > values
>> >> >>> > > > > > >>> > (those requiring
>> >> >>> > > > > > >>> > 63-64 bits after FOR is applied) before we close a
>> >> vote.
>> >> >>> > We'll
>> >> >>> > > > aim
>> >> >>> > > > > > to
>> >> >>> > > > > > >>> get
>> >> >>> > > > > > >>> > that
>> >> >>> > > > > > >>> > done in parallel.
>> >> >>> > > > > > >>> >
>> >> >>> > > > > > >>> > Plan: unless there are objections, I intend to
>> start a
>> >> >>> vote
>> >> >>> > at
>> >> >>> > > > the
>> >> >>> > > > > > end
>> >> >>> > > > > > >>> of
>> >> >>> > > > > > >>> > this week/early
>> >> >>> > > > > > >>> > next week.
>> >> >>> > > > > > >>> >
>> >> >>> > > > > > >>> > Thanks
>> >> >>> > > > > > >>> > Prateek
>> >> >>> > > > > > >>> >
>> >> >>> > > > > > >>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <
>> >> >>> > > prtkgaur@gmail.com>
>> >> >>> > > > > > >>> wrote:
>> >> >>> > > > > > >>> >
>> >> >>> > > > > > >>> > > Hi Team,
>> >> >>> > > > > > >>> > >
>> >> >>> > > > > > >>> > > Just wanted to provide some updates on ALP.
>> >> >>> > > > > > >>> > > Micah and I have done 3-4 rounds of review for
>> the
>> >> c++
>> >> >>> PR :
>> >> >>> > > > > > >>> > > https://github.com/apache/arrow/pull/48345
>> >> >>> > > > > > >>> > >
>> >> >>> > > > > > >>> > > For the Java implementation Vinoo has been
>> working
>> >> on
>> >> >>> > > following
>> >> >>> > > > > PR
>> >> >>> > > > > > :
>> >> >>> > > > > > >>> > > https://github.com/apache/parquet-java/pull/3397
>> >> >>> > > > > > >>> > >
>> >> >>> > > > > > >>> > > Best
>> >> >>> > > > > > >>> > > Prateek
>> >> >>> > > > > > >>> > >
>> >> >>> > > > > > >>> > > On Tue, May 5, 2026 at 1:50 PM Micah Kornfield <
>> >> >>> > > > > > >>> emkornfield@gmail.com>
>> >> >>> > > > > > >>> > > wrote:
>> >> >>> > > > > > >>> > >
>> >> >>> > > > > > >>> > >> Hi Antoine,
>> >> >>> > > > > > >>> > >>
>> >> >>> > > > > > >>> > >> > Apologies if the question was already asked,
>> but
>> >> >>> should
>> >> >>> > we
>> >> >>> > > > > care
>> >> >>> > > > > > >>> about
>> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more
>> >> efficient
>> >> >>> than
>> >> >>> > > > > FLOAT16
>> >> >>> > > > > > +
>> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
>> >> >>> > > > > > >>> > >>
>> >> >>> > > > > > >>> > >> It was. We thought we could defer it for the
>> >> >>> following
>> >> >>> > > > reasons:
>> >> >>> > > > > > >>> > >> 1. It's not clear there are a lot of easy
>> >> reference
>> >> >>> > > datasets
>> >> >>> > > > to
>> >> >>> > > > > > >>> test
>> >> >>> > > > > > >>> > its
>> >> >>> > > > > > >>> > >> effectiveness. It does look like there might be
>> >> one
>> >> >>> or
>> >> >>> > two
>> >> >>> > > on
>> >> >>> > > > > > >>> > huggingface
>> >> >>> > > > > > >>> > >> (e.g.
>> >> https://huggingface.co/datasets/kikitora/curdie
>> >> >>> ).
>> >> >>> > > > > > >>> > >> 2. It seemed likely that float 16 was more
>> likely
>> >> >>> used
>> >> >>> > for
>> >> >>> > > > > values
>> >> >>> > > > > > >>> that
>> >> >>> > > > > > >>> > >> were less likely to reduce to decimal values.
>> >> >>> > > > > > >>> > >> 3. It could be added as an extension later if
>> >> needed.
>> >> >>> > > > > > >>> > >>
>> >> >>> > > > > > >>> > >> Cheers,
>> >> >>> > > > > > >>> > >> Micah
>> >> >>> > > > > > >>> > >>
>> >> >>> > > > > > >>> > >>
>> >> >>> > > > > > >>> > >> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <
>> >> >>> > > > > antoine@python.org
>> >> >>> > > > > > >
>> >> >>> > > > > > >>> > wrote:
>> >> >>> > > > > > >>> > >>
>> >> >>> > > > > > >>> > >> >
>> >> >>> > > > > > >>> > >> > Hello,
>> >> >>> > > > > > >>> > >> >
>> >> >>> > > > > > >>> > >> > Apologies if the question was already asked,
>> but
>> >> >>> should
>> >> >>> > we
>> >> >>> > > > > care
>> >> >>> > > > > > >>> about
>> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more
>> >> efficient
>> >> >>> than
>> >> >>> > > > > FLOAT16
>> >> >>> > > > > > +
>> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
>> >> >>> > > > > > >>> > >> >
>> >> >>> > > > > > >>> > >> > Regards
>> >> >>> > > > > > >>> > >> >
>> >> >>> > > > > > >>> > >> > Antoine.
>> >> >>> > > > > > >>> > >> >
>> >> >>> > > > > > >>> > >> >
>> >> >>> > > > > > >>> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
>> >> >>> > > > > > >>> > >> > > Thanks Andrew and Micah for review feedback
>> on
>> >> >>> the two
>> >> >>> > > > PR's
>> >> >>> > > > > > >>> > >> > > 1) (c++ arrow repo)
>> >> >>> > > > > > >>> > >>
>> https://github.com/apache/arrow/pull/48345/changes
>> >> >>> > > > > > >>> > >> > > 2) (parquet-format repo)
>> >> >>> > > > > > >>> > >> >
>> >> https://github.com/apache/parquet-format/pull/557
>> >> >>> > > > > > >>> > >> > >
>> >> >>> > > > > > >>> > >> > > I have addressed all (unless I missed
>> >> something)
>> >> >>> > > comments
>> >> >>> > > > on
>> >> >>> > > > > > >>> the two
>> >> >>> > > > > > >>> > >> > PR's.
>> >> >>> > > > > > >>> > >> > >
>> >> >>> > > > > > >>> > >> > > Best
>> >> >>> > > > > > >>> > >> > > Prateek
>> >> >>> > > > > > >>> > >> > >
>> >> >>> > > > > > >>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK
>> GAUR <
>> >> >>> > > > > > >>> prtkgaur@gmail.com>
>> >> >>> > > > > > >>> > >> wrote:
>> >> >>> > > > > > >>> > >> > >
>> >> >>> > > > > > >>> > >> > >> Thanks Andrew and Micah.
>> >> >>> > > > > > >>> > >> > >>
>> >> >>> > > > > > >>> > >> > >> `fair amount of feedback on at least the
>> >> >>> > > implementations`
>> >> >>> > > > > > >>> > >> > >> For the c++ I have already started
>> addressing
>> >> the
>> >> >>> > > > > feedback, I
>> >> >>> > > > > > >>> > should
>> >> >>> > > > > > >>> > >> be
>> >> >>> > > > > > >>> > >> > >> done with that Monday/Tuesday.
>> >> >>> > > > > > >>> > >> > >> I think Vinoo too has been making good
>> >> progress
>> >> >>> on
>> >> >>> > the
>> >> >>> > > > Java
>> >> >>> > > > > > >>> > >> > implementation.
>> >> >>> > > > > > >>> > >> > >>
>> >> >>> > > > > > >>> > >> > >> Best
>> >> >>> > > > > > >>> > >> > >> Prateek
>> >> >>> > > > > > >>> > >> > >>
>> >> >>> > > > > > >>> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew
>> Lamb <
>> >> >>> > > > > > >>> > andrewlamb11@gmail.com
>> >> >>> > > > > > >>> > >> >
>> >> >>> > > > > > >>> > >> > >> wrote:
>> >> >>> > > > > > >>> > >> > >>
>> >> >>> > > > > > >>> > >> > >>> Got it. Thank you for the clarification
>> -- I
>> >> >>> will
>> >> >>> > try
>> >> >>> > > > and
>> >> >>> > > > > > look
>> >> >>> > > > > > >>> > into
>> >> >>> > > > > > >>> > >> the
>> >> >>> > > > > > >>> > >> > >>> spec and the Rust implementation[1] in
>> this
>> >> next
>> >> >>> > week
>> >> >>> > > > > > >>> > >> > >>>
>> >> >>> > > > > > >>> > >> > >>> [1]:
>> >> >>> https://github.com/apache/arrow-rs/pull/9372
>> >> >>> > > > > > >>> > >> > >>>
>> >> >>> > > > > > >>> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah
>> >> >>> Kornfield <
>> >> >>> > > > > > >>> > >> > emkornfield@gmail.com>
>> >> >>> > > > > > >>> > >> > >>> wrote:
>> >> >>> > > > > > >>> > >> > >>>
>> >> >>> > > > > > >>> > >> > >>>> Hi Andrew,
>> >> >>> > > > > > >>> > >> > >>>> I think there is a fair amount of
>> feedback
>> >> on
>> >> >>> at
>> >> >>> > > least
>> >> >>> > > > > the
>> >> >>> > > > > > >>> > >> > >>>> implementations, typically I think we've
>> >> waited
>> >> >>> > till
>> >> >>> > > > they
>> >> >>> > > > > > are
>> >> >>> > > > > > >>> > >> close to
>> >> >>> > > > > > >>> > >> > >>>> mergeable before a final vote.
>> Otherwise I
>> >> >>> agree
>> >> >>> > we
>> >> >>> > > > are
>> >> >>> > > > > > very
>> >> >>> > > > > > >>> > >> close.
>> >> >>> > > > > > >>> > >> > >>>>
>> >> >>> > > > > > >>> > >> > >>>> -Micah
>> >> >>> > > > > > >>> > >> > >>>>
>> >> >>> > > > > > >>> > >> > >>>> On Saturday, April 25, 2026, Andrew Lamb
>> <
>> >> >>> > > > > > >>> andrewlamb11@gmail.com
>> >> >>> > > > > > >>> > >
>> >> >>> > > > > > >>> > >> > wrote:
>> >> >>> > > > > > >>> > >> > >>>>
>> >> >>> > > > > > >>> > >> > >>>>> Thanks Prateek,
>> >> >>> > > > > > >>> > >> > >>>>>
>> >> >>> > > > > > >>> > >> > >>>>> I think from this content it looks to me
>> >> like
>> >> >>> we
>> >> >>> > are
>> >> >>> > > > > ready
>> >> >>> > > > > > >>> to
>> >> >>> > > > > > >>> > >> start a
>> >> >>> > > > > > >>> > >> > >>>>> vote to explicitly accept ALP into
>> Parquet
>> >> >>> > > > > > >>> > >> > >>>>>
>> >> >>> > > > > > >>> > >> > >>>>> Does anyone know of a reason we should
>> >> >>> postpone it
>> >> >>> > > for
>> >> >>> > > > > > >>> longer?
>> >> >>> > > > > > >>> > >> > >>>>> Perhaps someone needs some more time to
>> >> >>> review?
>> >> >>> > > > > > >>> > >> > >>>>>
>> >> >>> > > > > > >>> > >> > >>>>> Andrew
>> >> >>> > > > > > >>> > >> > >>>>>
>> >> >>> > > > > > >>> > >> > >>>>>
>> >> >>> > > > > > >>> > >> > >>>>>
>> >> >>> > > > > > >>> > >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK
>> >> GAUR <
>> >> >>> > > > > > >>> > prtkgaur@gmail.com>
>> >> >>> > > > > > >>> > >> > >>>>> wrote:
>> >> >>> > > > > > >>> > >> > >>>>>
>> >> >>> > > > > > >>> > >> > >>>>>> Hi team,
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>> Hope everyone is doing well. I got a
>> >> chance
>> >> >>> to
>> >> >>> > work
>> >> >>> > > > > > >>> through all
>> >> >>> > > > > > >>> > >> the
>> >> >>> > > > > > >>> > >> > >>>>>> remaining feedback and update the spec
>> >> doc.
>> >> >>> Here
>> >> >>> > > are
>> >> >>> > > > > the
>> >> >>> > > > > > >>> new
>> >> >>> > > > > > >>> > >> > artifacts
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>> 1) Spec document :
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> >
>> >> >>> > > > > > >>> > >>
>> >> >>> > > > > > >>> >
>> >> >>> > > > > > >>>
>> >> >>> > > > > >
>> >> >>> > > > >
>> >> >>> > > >
>> >> >>> > >
>> >> >>> >
>> >> >>>
>> >>
>> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>> 2) Spec document in parquet format
>> repo :
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > https://github.com/apache/parquet-format/pull/557
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>> 3) Alp implementation in arrow c++
>> repo :
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > https://github.com/apache/arrow/pull/48345/changes
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>> 4) Alp implementation in parquet-java
>> >> repo :
>> >> >>> Work
>> >> >>> > > for
>> >> >>> > > > > > >>> Vinoo and
>> >> >>> > > > > > >>> > >> > Julien
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > https://github.com/apache/parquet-java/pull/3397
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>> 5) PR with test and benchmarking
>> >> artifacts in
>> >> >>> > > > > > >>> parquet-testing
>> >> >>> > > > > > >>> > >> repo :
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > https://github.com/apache/parquet-testing/pull/100
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>> And
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>> - Go : Arnav just submitted an in
>> >> >>> progress
>> >> >>> > > > > > >>> implementation
>> >> >>> > > > > > >>> > in
>> >> >>> > > > > > >>> > >> Go.
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> https://github.com/apache/arrow-go/pull/704
>> >> >>> > (I
>> >> >>> > > > > > haven't
>> >> >>> > > > > > >>> > >> started
>> >> >>> > > > > > >>> > >> > >>>>>> looking at it yet)
>> >> >>> > > > > > >>> > >> > >>>>>> - Rust : I remember Andrew
>> mentioned
>> >> that
>> >> >>> > this
>> >> >>> > > > work
>> >> >>> > > > > > is
>> >> >>> > > > > > >>> also
>> >> >>> > > > > > >>> > >> in
>> >> >>> > > > > > >>> > >> > >>>>>> progress (So 4 languages!)
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>> *Arrow C++ implementation *
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>> The PR is out and was also used by
>> >> Antoine to
>> >> >>> > > report
>> >> >>> > > > > the
>> >> >>> > > > > > >>> > numbers
>> >> >>> > > > > > >>> > >> as
>> >> >>> > > > > > >>> > >> > >>>>>> reported here. Micah and Konstantin
>> have
>> >> >>> given 1
>> >> >>> > > > round
>> >> >>> > > > > of
>> >> >>> > > > > > >>> > >> feedback
>> >> >>> > > > > > >>> > >> > >>>>>> and I'm addressing them today. Please
>> note
>> >> >>> that
>> >> >>> > the
>> >> >>> > > > > > default
>> >> >>> > > > > > >>> > >> > >>>>>> optimization flag for compiling is O2
>> and
>> >> not
>> >> >>> > Q3. I
>> >> >>> > > > got
>> >> >>> > > > > > >>> around
>> >> >>> > > > > > >>> > >> 70%
>> >> >>> > > > > > >>> > >> > >>>>>> performance improvement in the decoding
>> >> speed
>> >> >>> > when
>> >> >>> > > > > using
>> >> >>> > > > > > >>> the O3
>> >> >>> > > > > > >>> > >> > flag.
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>> *Parqet-MR Java implementation (working
>> >> with
>> >> >>> > Vinoo
>> >> >>> > > > and
>> >> >>> > > > > > >>> Julien)
>> >> >>> > > > > > >>> > >> and
>> >> >>> > > > > > >>> > >> > **Cross
>> >> >>> > > > > > >>> > >> > >>>>>> Language testing*
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>> Let me know if you have any
>> questions
>> >> or
>> >> >>> > > > feedback.
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>> Now pasting some performance numbers
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>> Table 1: C++ ALP Double Decode —
>> >> Spotify
>> >> >>> > Columns
>> >> >>> > > > > > >>> (Graviton
>> >> >>> > > > > > >>> > 3,
>> >> >>> > > > > > >>> > >> ARM
>> >> >>> > > > > > >>> > >> > >>>>>> Neoverse V1)
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>>
>> >> >>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>> │ Column │ -O2 (MB/s) │
>> >> -O3
>> >> >>> > > (MB/s) │
>> >> >>> > > > > > >>> Speedup │
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>>
>> >> >>> ├──────────────────┼──────────────┼──────────────┼─────────┤
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>> │ valence │ 3,155 │
>> >> >>> 5,523
>> >> >>> > > │
>> >> >>> > > > > > >>> 1.75x │
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>> │ danceability │ 3,233 │
>> >> >>> 5,685
>> >> >>> > > │
>> >> >>> > > > > > >>> 1.76x │
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>> │ energy │ 3,197 │
>> >> >>> 5,652
>> >> >>> > > │
>> >> >>> > > > > > >>> 1.77x │
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>> │ loudness │ 3,186 │
>> >> >>> 5,473
>> >> >>> > > │
>> >> >>> > > > > > >>> 1.72x │
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>>
>> >> >>> └──────────────────┴──────────────┴──────────────┴─────────┘
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK
>> >> GAUR
>> >> >>> <
>> >> >>> > > > > > >>> > prtkgaur@gmail.com
>> >> >>> > > > > > >>> > >> >
>> >> >>> > > > > > >>> > >> > >>>>>> wrote:
>> >> >>> > > > > > >>> > >> > >>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>> @Micah Kornfield <
>> emkornfield@gmail.com>
>> >> :
>> >> >>> Got
>> >> >>> > > it.
>> >> >>> > > > > > >>> > >> > >>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>> @Andrew Lamb <an...@gmail.com>
>> >> >>> > > > > > >>> > >> > >>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>> Do you think it would be good to
>> start
>> >> >>> moving
>> >> >>> > the
>> >> >>> > > > > spec
>> >> >>> > > > > > >>> > >> development
>> >> >>> > > > > > >>> > >> > >>>>>>>> into
>> >> >>> > > > > > >>> > >> > >>>>>>>> markdown format, in preparation for
>> >> >>> finalizing
>> >> >>> > > it?
>> >> >>> > > > > > >>> > >> > >>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>> Yes I'll update the numbers for some
>> of
>> >> the
>> >> >>> > > > examples I
>> >> >>> > > > > > >>> have in
>> >> >>> > > > > > >>> > >> the
>> >> >>> > > > > > >>> > >> > >>>>>>> spec based
>> >> >>> > > > > > >>> > >> > >>>>>>> on the updated header size. Then we
>> >> should
>> >> >>> be
>> >> >>> > good
>> >> >>> > > > to
>> >> >>> > > > > go
>> >> >>> > > > > > >>> for
>> >> >>> > > > > > >>> > the
>> >> >>> > > > > > >>> > >> > >>>>>>> markdown format.
>> >> >>> > > > > > >>> > >> > >>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>> Thanks everyone!
>> >> >>> > > > > > >>> > >> > >>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>> Andrew
>> >> >>> > > > > > >>> > >> > >>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM
>> PRATEEK
>> >> >>> GAUR <
>> >> >>> > > > > > >>> > >> prtkgaur@gmail.com>
>> >> >>> > > > > > >>> > >> > >>>>>>>> wrote:
>> >> >>> > > > > > >>> > >> > >>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>> Hi team,
>> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>> 1) Andrew
>> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for working on test
>> files.
>> >> >>> My PR
>> >> >>> > > did
>> >> >>> > > > > add
>> >> >>> > > > > > >>> all
>> >> >>> > > > > > >>> > the
>> >> >>> > > > > > >>> > >> > test
>> >> >>> > > > > > >>> > >> > >>>>>>>> files I
>> >> >>> > > > > > >>> > >> > >>>>>>>>> used to benchmark on datasets.
>> >> Maybe
>> >> >>> we
>> >> >>> > can
>> >> >>> > > > club
>> >> >>> > > > > > it
>> >> >>> > > > > > >>> > >> together.
>> >> >>> > > > > > >>> > >> > >>>>>>>> WIll also
>> >> >>> > > > > > >>> > >> > >>>>>>>>> aid
>> >> >>> > > > > > >>> > >> > >>>>>>>>> cross language testing
>> >> >>> > > > > > >>> > >> > >>>>>>>>> - Kosta Tarasov working on Rust
>> >> >>> > > > implementation.
>> >> >>> > > > > > >>> This is
>> >> >>> > > > > > >>> > >> > great.
>> >> >>> > > > > > >>> > >> > >>>>>>>> Thanks
>> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>> 2) Antoine
>> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks a lot for reporting the
>> >> >>> numbers
>> >> >>> > on
>> >> >>> > > > AMD.
>> >> >>> > > > > > >>> Looks
>> >> >>> > > > > > >>> > >> like
>> >> >>> > > > > > >>> > >> > you
>> >> >>> > > > > > >>> > >> > >>>>>>>> are
>> >> >>> > > > > > >>> > >> > >>>>>>>>> getting 8X the decoding
>> >> performance of
>> >> >>> > BSS.
>> >> >>> > > > This
>> >> >>> > > > > > is
>> >> >>> > > > > > >>> > >> > amazing!!.
>> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for acknowledging the
>> >> >>> sampling
>> >> >>> > > > design.
>> >> >>> > > > > > >>> > >> > >>>>>>>>> - I agree with you on
>> Fastlanes. In
>> >> >>> some
>> >> >>> > > crude
>> >> >>> > > > > > >>> > >> experiments I
>> >> >>> > > > > > >>> > >> > >>>>>>>> didn't get
>> >> >>> > > > > > >>> > >> > >>>>>>>>> a good perf benefit from it on
>> >> >>> Graviton3
>> >> >>> > > (but
>> >> >>> > > > > > maybe
>> >> >>> > > > > > >>> > there
>> >> >>> > > > > > >>> > >> was
>> >> >>> > > > > > >>> > >> > >>>>>>>> something
>> >> >>> > > > > > >>> > >> > >>>>>>>>> wrong with my implementation).
>> >> >>> > > > > > >>> > >> > >>>>>>>>> - Locking the 16bit exception
>> >> >>> encoding for
>> >> >>> > > the
>> >> >>> > > > > > spec
>> >> >>> > > > > > >>> in
>> >> >>> > > > > > >>> > >> this
>> >> >>> > > > > > >>> > >> > >>>>>>>> case.
>> >> >>> > > > > > >>> > >> > >>>>>>>>> - Awesome I think we have solved
>> >> for
>> >> >>> all
>> >> >>> > > open
>> >> >>> > > > > > >>> questions
>> >> >>> > > > > > >>> > >> minus
>> >> >>> > > > > > >>> > >> > >>>>>>>> the
>> >> >>> > > > > > >>> > >> > >>>>>>>>> version byte :). (will get back
>> on
>> >> >>> this
>> >> >>> > > soon)
>> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>> 3) Micah
>> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>> - FastLanes : The current spec
>> does
>> >> >>> allow
>> >> >>> > > for
>> >> >>> > > > > > using
>> >> >>> > > > > > >>> > >> FastLane
>> >> >>> > > > > > >>> > >> > >>>>>>>> with the
>> >> >>> > > > > > >>> > >> > >>>>>>>>> configurable enum value for
>> >> layout. We
>> >> >>> > > should
>> >> >>> > > > be
>> >> >>> > > > > > >>> able to
>> >> >>> > > > > > >>> > >> > inject
>> >> >>> > > > > > >>> > >> > >>>>>>>> any
>> >> >>> > > > > > >>> > >> > >>>>>>>>> layout
>> >> >>> > > > > > >>> > >> > >>>>>>>>> in the current design.
>> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>> Working on resolving all remaining
>> open
>> >> >>> > comments
>> >> >>> > > > on
>> >> >>> > > > > > the
>> >> >>> > > > > > >>> spec
>> >> >>> > > > > > >>> > >> this
>> >> >>> > > > > > >>> > >> > >>>>>>>> week.
>> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>> Best
>> >> >>> > > > > > >>> > >> > >>>>>>>>> Prateek
>> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM
>> Steve
>> >> >>> > Loughran <
>> >> >>> > > > > > >>> > >> > >>>>>>>> stevel@cloudera.com>
>> >> >>> > > > > > >>> > >> > >>>>>>>>> wrote:
>> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah
>> >> >>> Kornfield
>> >> >>> > <
>> >> >>> > > > > > >>> > >> > >>>>>>>> emkornfield@gmail.com>
>> >> >>> > > > > > >>> > >> > >>>>>>>>>> wrote:
>> >> >>> > > > > > >>> > >> > >>>>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>>>> It looks like the actual issue
>> >> >>> described for
>> >> >>> > > ORC
>> >> >>> > > > > in
>> >> >>> > > > > > >>> the
>> >> >>> > > > > > >>> > >> paper
>> >> >>> > > > > > >>> > >> > >>>>>>>> is that
>> >> >>> > > > > > >>> > >> > >>>>>>>>> it
>> >> >>> > > > > > >>> > >> > >>>>>>>>>>> has multiple sub-encodings in a
>> >> batch.
>> >> >>> This
>> >> >>> > > is
>> >> >>> > > > > > >>> different
>> >> >>> > > > > > >>> > >> then
>> >> >>> > > > > > >>> > >> > >>>>>>>> the
>> >> >>> > > > > > >>> > >> > >>>>>>>>> design
>> >> >>> > > > > > >>> > >> > >>>>>>>>>>> proposed here where there is still
>> >> fixed
>> >> >>> > > > encoding
>> >> >>> > > > > > per
>> >> >>> > > > > > >>> page
>> >> >>> > > > > > >>> > >> in
>> >> >>> > > > > > >>> > >> > >>>>>>>> parquet.
>> >> >>> > > > > > >>> > >> > >>>>>>>>>>> Given reasonably sized pages I
>> don't
>> >> >>> think
>> >> >>> > > > branch
>> >> >>> > > > > > >>> > >> > >>>>>>>> misprediction should
>> >> >>> > > > > > >>> > >> > >>>>>>>>>> be a
>> >> >>> > > > > > >>> > >> > >>>>>>>>>>> big issue for new encodings. I
>> agree
>> >> >>> that
>> >> >>> > we
>> >> >>> > > > > should
>> >> >>> > > > > > >>> be
>> >> >>> > > > > > >>> > >> > >>>>>>>> conservative in
>> >> >>> > > > > > >>> > >> > >>>>>>>>>>> general for adding new encodings.
>> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>>> +1
>> >> >>> > > > > > >>> > >> > >>>>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>>
>> >> >>> > > > > > >>> > >> > >>>>>>>
>> >> >>> > > > > > >>> > >> > >
>> >> >>> > > > > > >>> > >> >
>> >> >>> > > > > > >>> > >> >
>> >> >>> > > > > > >>> > >> >
>> >> >>> > > > > > >>> > >>
>> >> >>> > > > > > >>> > >
>> >> >>> > > > > > >>> >
>> >> >>> > > > > > >>>
>> >> >>> > > > > > >>
>> >> >>> > > > > >
>> >> >>> > > > >
>> >> >>> > > >
>> >> >>> > >
>> >> >>> >
>> >> >>>
>> >> >>
>> >>
>> >
>>
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by PRATEEK GAUR <pr...@gmail.com>.
Thanks Divjot for bringing it up.
I was under the same impression as Russell and Andrew that we'd have it
flag gated.
With the new versioning proposal 'preview' might be a good tag for it.
Best
Prateek
On Sat, Aug 1, 2026 at 12:39 AM Andrew Lamb <an...@gmail.com> wrote:
> Div: Here is a proposal to mark ALP in Preview: [1] (using a comment in the
> spec section).
>
> Russel: That is my understanding as well.
>
> Hopefully we are able to formalize the versioning process shortly so we can
> formalize the notion of "preview"
>
> Andrew
>
>
> [1]: https://github.com/apache/parquet-format/pull/604
>
> On Fri, Jul 31, 2026 at 11:15 AM Russell Spitzer <
> russell.spitzer@gmail.com>
> wrote:
>
> > I thought we had agreed it would be merged, and features like this would
> > be feature flag gated. So a V-Old writer could always opt in to using
> this
> > encoding but that must be a conscious effort if you are not using VNext
> of
> > Parquet. If you are on V-Next the writer can choose to use ALP whenever
> it
> > likes without user intervention.
> >
> > On Fri, Jul 31, 2026 at 2:20 AM Divjot Arora via dev <
> > dev@parquet.apache.org> wrote:
> >
> >> Yes, that’s my understanding as well. However, as you noted, the
> >> versioning
> >> discussion is still in early stages and hasn’t prescribed concrete
> >> mechanisms yet. If ALP goes into parquet-format 2.14.0, there’s no
> >> indication that it’s in preview or that writers shouldn’t use it by
> >> default
> >> yet. Given that we’ve held off on merging other incompatible changes
> (e.g
> >> removal of path_in_schema) until the versioning discussion closes, my
> >> preference would be to leave ALP un-merged for the time being as well.
> >>
> >> — Div
> >>
> >> On Fri, Jul 31, 2026 at 08:59 Andrew Lamb <an...@gmail.com>
> wrote:
> >>
> >> > After re-reading Julien's Versioning Proposal, my understanding is
> that
> >> if
> >> > we adopt that system, we will somehow mark ALP in "Preview" in
> >> > parquet-format for release Version 3.
> >> >
> >> > The versioning document doesn't seem to proscribe the particular
> method
> >> > for annotating "preview" features -- I think clearly marking them as
> >> such
> >> > in parquet.thrift as well as reflecting that on the
> >> > https://parquet.apache.org/docs/file-format/versions/ page somehow
> >> would
> >> > be adequate.
> >> >
> >> > Andrew
> >> >
> >> > On Thu, Jul 30, 2026 at 11:50 AM Andrew Lamb <an...@gmail.com>
> >> > wrote:
> >> >
> >> >> In my mind ALP is in the same position as other encodings that were
> >> added
> >> >> recently[1] (such as BYTE_STREAM_SPLIT for non floating point types)
> >> >>
> >> >> That is to say it will be available in the latest open source writers
> >> and
> >> >> readers, but realistically not widely used in the ecosystem where
> >> >> compatibility is a concern (a large part)
> >> >>
> >> >> I view Julian's work[2][3] to gather consensus on how to name
> >> >> combinations of features (e.g. "versions") as the mechanism by which
> we
> >> >> will accelerate the adoption of this change (along with other forward
> >> >> incompatible changes) across the ecosystem.
> >> >>
> >> >> > Will we exclude this change when cutting parquet-format v2.14.0?
> >> >>
> >> >> No I don't think so -- I would expect it to go out in the next
> >> >> parquet-format version as do other changes in the spec.
> >> >>
> >> >> Andrew
> >> >>
> >> >> [1]: https://parquet.apache.org/docs/file-format/versions/
> >> >> [2]:
> https://lists.apache.org/thread/xmnj8h0h8ozmrgox5tydhhs7yz7q2hss
> >> >> [3]:
> https://lists.apache.org/thread/qlf8lg90gqq41lllyy9mk2f0skqvftqj
> >> >>
> >> >>
> >> >> On Thu, Jul 30, 2026 at 3:51 AM Divjot Arora via dev <
> >> >> dev@parquet.apache.org> wrote:
> >> >>
> >> >>> Hi Andrew,
> >> >>>
> >> >>> How does this interact with the ongoing versioning discussion? Given
> >> that
> >> >>> new encodings are not forward compatible, I would assume ALP is
> >> included
> >> >>> in
> >> >>> Parquet V<next> (V3?), but it's not clear to me how that works if
> it's
> >> >>> merged to the parquet-format master branch now. Will we exclude this
> >> >>> change
> >> >>> when cutting parquet-format v2.14.0?
> >> >>>
> >> >>> -- Div
> >> >>>
> >> >>> On Thu, Jul 30, 2026 at 8:51 AM Andrew Lamb <andrewlamb11@gmail.com
> >
> >> >>> wrote:
> >> >>>
> >> >>> > Hello,
> >> >>> >
> >> >>> > As an update, thanks to the epic work from many contributors,
> >> >>> especially
> >> >>> > Prateek, I just merged the parquet-format change for ALP[1]. We
> >> plan to
> >> >>> > make additional editorial The remaining items I know of are
> merging
> >> the
> >> >>> > examples from parquet-testing[2], and then of course the various
> >> >>> > implementations
> >> >>> >
> >> >>> > Thanks again everyone. This is a great step forward
> >> >>> > Andrew
> >> >>> >
> >> >>> > [1]: https://github.com/apache/parquet-format/pull/557
> >> >>> > [2]: https://github.com/apache/parquet-testing/pull/100
> >> >>> >
> >> >>> > On Tue, Jul 21, 2026 at 4:39 PM Julien Le Dem <ju...@apache.org>
> >> >>> wrote:
> >> >>> >
> >> >>> > > It was also in my spam folder... :(
> >> >>> > > I've seen a bunch of apache list emails go to my spam recently.
> >> >>> > >
> >> >>> > > On Tue, Jul 21, 2026 at 10:58 AM Russell Spitzer <
> >> >>> > > russell.spitzer@gmail.com>
> >> >>> > > wrote:
> >> >>> > >
> >> >>> > > > Yes, my hope was that my vote would kick it out of spam but we
> >> >>> > definitely
> >> >>> > > > had two other votes that also went to spam for me.
> >> >>> > > >
> >> >>> > > > For anyone looking the thread is here
> >> >>> > > >
> >> https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt
> >> >>> > > >
> >> >>> > > > On Tue, Jul 21, 2026 at 10:57 AM Micah Kornfield <
> >> >>> > emkornfield@gmail.com>
> >> >>> > > > wrote:
> >> >>> > > >
> >> >>> > > > > I've seen other's votes not go to spam, so hopefully people
> >> will
> >> >>> see
> >> >>> > it
> >> >>> > > > in
> >> >>> > > > > there inboxes (or see this thread and look in there spam).
> >> >>> > > > >
> >> >>> > > > > On Tue, Jul 21, 2026 at 10:52 AM PRATEEK GAUR <
> >> >>> prtkgaur@gmail.com>
> >> >>> > > > wrote:
> >> >>> > > > >
> >> >>> > > > > > Okay,
> >> >>> > > > > >
> >> >>> > > > > > I sent out the voting email yesterday and so far I've
> heard
> >> >>> from 3
> >> >>> > > > people
> >> >>> > > > > > that it ended up in their "spam folder". Dev VPN to be
> >> blamed
> >> >>> for
> >> >>> > > it?.
> >> >>> > > > > >
> >> >>> > > > > > What's the protocol here?
> >> >>> > > > > > I let people hunt for the email in their spam folder or
> send
> >> >>> out
> >> >>> > > > another
> >> >>> > > > > > one?
> >> >>> > > > > >
> >> >>> > > > > > Best
> >> >>> > > > > > Prateek
> >> >>> > > > > >
> >> >>> > > > > > On Sun, Jul 19, 2026 at 10:47 PM PRATEEK GAUR <
> >> >>> prtkgaur@gmail.com>
> >> >>> > > > > wrote:
> >> >>> > > > > >
> >> >>> > > > > > > Hi team,
> >> >>> > > > > > >
> >> >>> > > > > > > Vinoo has added the tests which were requested in the
> last
> >> >>> round
> >> >>> > of
> >> >>> > > > > > review.
> >> >>> > > > > > > In his own words.
> >> >>> > > > > > >
> >> >>> > > > > > > "
> >> >>> > > > > > > The extreme value tests Micah wanted are done and
> pushed.
> >> I
> >> >>> added
> >> >>> > > > > > coverage
> >> >>> > > > > > > for values that need the full FOR bit width after the
> >> frame
> >> >>> of
> >> >>> > > > > reference
> >> >>> > > > > > is
> >> >>> > > > > > > applied: 64-bit (the case where the signed max minus min
> >> >>> > > subtraction
> >> >>> > > > > > > overflows), 63-bit (the non-overflow case), and 32-bit
> for
> >> >>> > floats.
> >> >>> > > > All
> >> >>> > > > > > > lossless with zero exceptions. I also verified them
> >> through
> >> >>> Arrow
> >> >>> > > C++
> >> >>> > > > > ALP
> >> >>> > > > > > > reader (the local build), so the Java-written extreme
> >> columns
> >> >>> > > decode
> >> >>> > > > > > > correctly in C++, bit-exact against the expected
> values."
> >> >>> > > > > > >
> >> >>> > > > > > > State of the PR (#3397):
> >> >>> > > > > > >
> >> >>> > > > > > > - Merges cleanly with master (I pulled master in and
> >> >>> resolved
> >> >>> > > the
> >> >>> > > > > > > conflicts).
> >> >>> > > > > > > - 41 of 44 review threads resolved. The 3 remaining
> are
> >> >>> > > > intentional:
> >> >>> > > > > > > one deferred perf nit, one that stays open until
> >> >>> > parquet-format
> >> >>> > > > > ships
> >> >>> > > > > > ALP,
> >> >>> > > > > > >
> >> >>> > > > > > > "
> >> >>> > > > > > >
> >> >>> > > > > > > With this both the
> >> >>> > > > > > > 1) C++ PR :
> >> >>> https://github.com/apache/arrow/pull/48345/changes
> >> >>> > > > > > > 2) Java PR :
> >> >>> https://github.com/apache/parquet-java/pull/3397
> >> >>> > > > > > >
> >> >>> > > > > > > Are cross language tested and range of test scenarios.
> >> >>> > > > > > >
> >> >>> > > > > > > Best
> >> >>> > > > > > > Prateek
> >> >>> > > > > > >
> >> >>> > > > > > >
> >> >>> > > > > > > On Mon, Jul 13, 2026 at 8:38 AM PRATEEK GAUR <
> >> >>> prtkgaur@gmail.com
> >> >>> > >
> >> >>> > > > > wrote:
> >> >>> > > > > > >
> >> >>> > > > > > >> Thanks Matt,
> >> >>> > > > > > >>
> >> >>> > > > > > >> I'll be on it this week.
> >> >>> > > > > > >>
> >> >>> > > > > > >> Best
> >> >>> > > > > > >> Prateek
> >> >>> > > > > > >>
> >> >>> > > > > > >> On Mon, Jul 13, 2026 at 8:36 AM Matt Topol <
> >> >>> > > zotthewizard@gmail.com>
> >> >>> > > > > > >> wrote:
> >> >>> > > > > > >>
> >> >>> > > > > > >>> There is also the Go implementation submitted by
> Arnav (
> >> >>> > > > > > >>> https://github.com/apache/arrow-go/pull/704) which is
> >> >>> waiting
> >> >>> > > for
> >> >>> > > > > > >>> updates.
> >> >>> > > > > > >>>
> >> >>> > > > > > >>> Just wanted to make sure it didn't get lost here.
> >> >>> > > > > > >>>
> >> >>> > > > > > >>> --Matt
> >> >>> > > > > > >>>
> >> >>> > > > > > >>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK GAUR <
> >> >>> > > prtkgaur@gmail.com>
> >> >>> > > > > > >>> wrote:
> >> >>> > > > > > >>>
> >> >>> > > > > > >>> > Hi all,
> >> >>> > > > > > >>> >
> >> >>> > > > > > >>> > I'd like to share a status update on the ALP
> encoding
> >> >>> effort
> >> >>> > > and
> >> >>> > > > > get
> >> >>> > > > > > a
> >> >>> > > > > > >>> feel
> >> >>> > > > > > >>> > from the community before starting a formal vote.
> >> >>> > > > > > >>> >
> >> >>> > > > > > >>> > We now have the specification plus implementations
> in
> >> >>> both
> >> >>> > > > > languages:
> >> >>> > > > > > >>> >
> >> >>> > > > > > >>> > - Spec: apache/parquet-format#557 —
> GH-533
> >> >>> Add
> >> >>> > ALP
> >> >>> > > > > > encoding
> >> >>> > > > > > >>> > specification
> >> >>> > > > > > >>> > - C++ (Arrow): apache/arrow#48345 — GH-48701
> >> >>> > > [C++][Parquet]
> >> >>> > > > > Add
> >> >>> > > > > > >>> ALPpd
> >> >>> > > > > > >>> > encoding
> >> >>> > > > > > >>> > - Java: apache/parquet-java#3397 —
> Parquet
> >> >>> Java
> >> >>> > ALP
> >> >>> > > > > > >>> > Implementation (by Vinoo)
> >> >>> > > > > > >>> >
> >> >>> > > > > > >>> > Current state:
> >> >>> > > > > > >>> >
> >> >>> > > > > > >>> > - The C++ implementation has been through 3-4
> >> rounds
> >> >>> of
> >> >>> > > > review.
> >> >>> > > > > > >>> > - The Java implementation has been tracking well
> >> and
> >> >>> has
> >> >>> > > had
> >> >>> > > > > > >>> initial
> >> >>> > > > > > >>> > reviews from contributors, with no major
> >> remaining
> >> >>> > > issues.
> >> >>> > > > > > >>> > - Cross-language compatibility tests are
> passing:
> >> the
> >> >>> > Arrow
> >> >>> > > > C++
> >> >>> > > > > > >>> decoder
> >> >>> > > > > > >>> > reads Java-written data bit-exactly across
> >> ~1.56M
> >> >>> > values
> >> >>> > > > and
> >> >>> > > > > 18
> >> >>> > > > > > >>> > fixtures, covering V1 and V2 pages, multiple
> >> vector
> >> >>> > > sizes,
> >> >>> > > > > and
> >> >>> > > > > > >>> > several
> >> >>> > > > > > >>> > real datasets — zero mismatches.
> >> >>> > > > > > >>> >
> >> >>> > > > > > >>> > As raised in the java-pr we will be adding coverage
> >> for
> >> >>> > > extreme
> >> >>> > > > > > values
> >> >>> > > > > > >>> > (those requiring
> >> >>> > > > > > >>> > 63-64 bits after FOR is applied) before we close a
> >> vote.
> >> >>> > We'll
> >> >>> > > > aim
> >> >>> > > > > > to
> >> >>> > > > > > >>> get
> >> >>> > > > > > >>> > that
> >> >>> > > > > > >>> > done in parallel.
> >> >>> > > > > > >>> >
> >> >>> > > > > > >>> > Plan: unless there are objections, I intend to
> start a
> >> >>> vote
> >> >>> > at
> >> >>> > > > the
> >> >>> > > > > > end
> >> >>> > > > > > >>> of
> >> >>> > > > > > >>> > this week/early
> >> >>> > > > > > >>> > next week.
> >> >>> > > > > > >>> >
> >> >>> > > > > > >>> > Thanks
> >> >>> > > > > > >>> > Prateek
> >> >>> > > > > > >>> >
> >> >>> > > > > > >>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <
> >> >>> > > prtkgaur@gmail.com>
> >> >>> > > > > > >>> wrote:
> >> >>> > > > > > >>> >
> >> >>> > > > > > >>> > > Hi Team,
> >> >>> > > > > > >>> > >
> >> >>> > > > > > >>> > > Just wanted to provide some updates on ALP.
> >> >>> > > > > > >>> > > Micah and I have done 3-4 rounds of review for the
> >> c++
> >> >>> PR :
> >> >>> > > > > > >>> > > https://github.com/apache/arrow/pull/48345
> >> >>> > > > > > >>> > >
> >> >>> > > > > > >>> > > For the Java implementation Vinoo has been working
> >> on
> >> >>> > > following
> >> >>> > > > > PR
> >> >>> > > > > > :
> >> >>> > > > > > >>> > > https://github.com/apache/parquet-java/pull/3397
> >> >>> > > > > > >>> > >
> >> >>> > > > > > >>> > > Best
> >> >>> > > > > > >>> > > Prateek
> >> >>> > > > > > >>> > >
> >> >>> > > > > > >>> > > On Tue, May 5, 2026 at 1:50 PM Micah Kornfield <
> >> >>> > > > > > >>> emkornfield@gmail.com>
> >> >>> > > > > > >>> > > wrote:
> >> >>> > > > > > >>> > >
> >> >>> > > > > > >>> > >> Hi Antoine,
> >> >>> > > > > > >>> > >>
> >> >>> > > > > > >>> > >> > Apologies if the question was already asked,
> but
> >> >>> should
> >> >>> > we
> >> >>> > > > > care
> >> >>> > > > > > >>> about
> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more
> >> efficient
> >> >>> than
> >> >>> > > > > FLOAT16
> >> >>> > > > > > +
> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> >> >>> > > > > > >>> > >>
> >> >>> > > > > > >>> > >> It was. We thought we could defer it for the
> >> >>> following
> >> >>> > > > reasons:
> >> >>> > > > > > >>> > >> 1. It's not clear there are a lot of easy
> >> reference
> >> >>> > > datasets
> >> >>> > > > to
> >> >>> > > > > > >>> test
> >> >>> > > > > > >>> > its
> >> >>> > > > > > >>> > >> effectiveness. It does look like there might be
> >> one
> >> >>> or
> >> >>> > two
> >> >>> > > on
> >> >>> > > > > > >>> > huggingface
> >> >>> > > > > > >>> > >> (e.g.
> >> https://huggingface.co/datasets/kikitora/curdie
> >> >>> ).
> >> >>> > > > > > >>> > >> 2. It seemed likely that float 16 was more
> likely
> >> >>> used
> >> >>> > for
> >> >>> > > > > values
> >> >>> > > > > > >>> that
> >> >>> > > > > > >>> > >> were less likely to reduce to decimal values.
> >> >>> > > > > > >>> > >> 3. It could be added as an extension later if
> >> needed.
> >> >>> > > > > > >>> > >>
> >> >>> > > > > > >>> > >> Cheers,
> >> >>> > > > > > >>> > >> Micah
> >> >>> > > > > > >>> > >>
> >> >>> > > > > > >>> > >>
> >> >>> > > > > > >>> > >> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <
> >> >>> > > > > antoine@python.org
> >> >>> > > > > > >
> >> >>> > > > > > >>> > wrote:
> >> >>> > > > > > >>> > >>
> >> >>> > > > > > >>> > >> >
> >> >>> > > > > > >>> > >> > Hello,
> >> >>> > > > > > >>> > >> >
> >> >>> > > > > > >>> > >> > Apologies if the question was already asked,
> but
> >> >>> should
> >> >>> > we
> >> >>> > > > > care
> >> >>> > > > > > >>> about
> >> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more
> >> efficient
> >> >>> than
> >> >>> > > > > FLOAT16
> >> >>> > > > > > +
> >> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> >> >>> > > > > > >>> > >> >
> >> >>> > > > > > >>> > >> > Regards
> >> >>> > > > > > >>> > >> >
> >> >>> > > > > > >>> > >> > Antoine.
> >> >>> > > > > > >>> > >> >
> >> >>> > > > > > >>> > >> >
> >> >>> > > > > > >>> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
> >> >>> > > > > > >>> > >> > > Thanks Andrew and Micah for review feedback
> on
> >> >>> the two
> >> >>> > > > PR's
> >> >>> > > > > > >>> > >> > > 1) (c++ arrow repo)
> >> >>> > > > > > >>> > >>
> https://github.com/apache/arrow/pull/48345/changes
> >> >>> > > > > > >>> > >> > > 2) (parquet-format repo)
> >> >>> > > > > > >>> > >> >
> >> https://github.com/apache/parquet-format/pull/557
> >> >>> > > > > > >>> > >> > >
> >> >>> > > > > > >>> > >> > > I have addressed all (unless I missed
> >> something)
> >> >>> > > comments
> >> >>> > > > on
> >> >>> > > > > > >>> the two
> >> >>> > > > > > >>> > >> > PR's.
> >> >>> > > > > > >>> > >> > >
> >> >>> > > > > > >>> > >> > > Best
> >> >>> > > > > > >>> > >> > > Prateek
> >> >>> > > > > > >>> > >> > >
> >> >>> > > > > > >>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR
> <
> >> >>> > > > > > >>> prtkgaur@gmail.com>
> >> >>> > > > > > >>> > >> wrote:
> >> >>> > > > > > >>> > >> > >
> >> >>> > > > > > >>> > >> > >> Thanks Andrew and Micah.
> >> >>> > > > > > >>> > >> > >>
> >> >>> > > > > > >>> > >> > >> `fair amount of feedback on at least the
> >> >>> > > implementations`
> >> >>> > > > > > >>> > >> > >> For the c++ I have already started
> addressing
> >> the
> >> >>> > > > > feedback, I
> >> >>> > > > > > >>> > should
> >> >>> > > > > > >>> > >> be
> >> >>> > > > > > >>> > >> > >> done with that Monday/Tuesday.
> >> >>> > > > > > >>> > >> > >> I think Vinoo too has been making good
> >> progress
> >> >>> on
> >> >>> > the
> >> >>> > > > Java
> >> >>> > > > > > >>> > >> > implementation.
> >> >>> > > > > > >>> > >> > >>
> >> >>> > > > > > >>> > >> > >> Best
> >> >>> > > > > > >>> > >> > >> Prateek
> >> >>> > > > > > >>> > >> > >>
> >> >>> > > > > > >>> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew
> Lamb <
> >> >>> > > > > > >>> > andrewlamb11@gmail.com
> >> >>> > > > > > >>> > >> >
> >> >>> > > > > > >>> > >> > >> wrote:
> >> >>> > > > > > >>> > >> > >>
> >> >>> > > > > > >>> > >> > >>> Got it. Thank you for the clarification --
> I
> >> >>> will
> >> >>> > try
> >> >>> > > > and
> >> >>> > > > > > look
> >> >>> > > > > > >>> > into
> >> >>> > > > > > >>> > >> the
> >> >>> > > > > > >>> > >> > >>> spec and the Rust implementation[1] in this
> >> next
> >> >>> > week
> >> >>> > > > > > >>> > >> > >>>
> >> >>> > > > > > >>> > >> > >>> [1]:
> >> >>> https://github.com/apache/arrow-rs/pull/9372
> >> >>> > > > > > >>> > >> > >>>
> >> >>> > > > > > >>> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah
> >> >>> Kornfield <
> >> >>> > > > > > >>> > >> > emkornfield@gmail.com>
> >> >>> > > > > > >>> > >> > >>> wrote:
> >> >>> > > > > > >>> > >> > >>>
> >> >>> > > > > > >>> > >> > >>>> Hi Andrew,
> >> >>> > > > > > >>> > >> > >>>> I think there is a fair amount of feedback
> >> on
> >> >>> at
> >> >>> > > least
> >> >>> > > > > the
> >> >>> > > > > > >>> > >> > >>>> implementations, typically I think we've
> >> waited
> >> >>> > till
> >> >>> > > > they
> >> >>> > > > > > are
> >> >>> > > > > > >>> > >> close to
> >> >>> > > > > > >>> > >> > >>>> mergeable before a final vote. Otherwise
> I
> >> >>> agree
> >> >>> > we
> >> >>> > > > are
> >> >>> > > > > > very
> >> >>> > > > > > >>> > >> close.
> >> >>> > > > > > >>> > >> > >>>>
> >> >>> > > > > > >>> > >> > >>>> -Micah
> >> >>> > > > > > >>> > >> > >>>>
> >> >>> > > > > > >>> > >> > >>>> On Saturday, April 25, 2026, Andrew Lamb <
> >> >>> > > > > > >>> andrewlamb11@gmail.com
> >> >>> > > > > > >>> > >
> >> >>> > > > > > >>> > >> > wrote:
> >> >>> > > > > > >>> > >> > >>>>
> >> >>> > > > > > >>> > >> > >>>>> Thanks Prateek,
> >> >>> > > > > > >>> > >> > >>>>>
> >> >>> > > > > > >>> > >> > >>>>> I think from this content it looks to me
> >> like
> >> >>> we
> >> >>> > are
> >> >>> > > > > ready
> >> >>> > > > > > >>> to
> >> >>> > > > > > >>> > >> start a
> >> >>> > > > > > >>> > >> > >>>>> vote to explicitly accept ALP into
> Parquet
> >> >>> > > > > > >>> > >> > >>>>>
> >> >>> > > > > > >>> > >> > >>>>> Does anyone know of a reason we should
> >> >>> postpone it
> >> >>> > > for
> >> >>> > > > > > >>> longer?
> >> >>> > > > > > >>> > >> > >>>>> Perhaps someone needs some more time to
> >> >>> review?
> >> >>> > > > > > >>> > >> > >>>>>
> >> >>> > > > > > >>> > >> > >>>>> Andrew
> >> >>> > > > > > >>> > >> > >>>>>
> >> >>> > > > > > >>> > >> > >>>>>
> >> >>> > > > > > >>> > >> > >>>>>
> >> >>> > > > > > >>> > >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK
> >> GAUR <
> >> >>> > > > > > >>> > prtkgaur@gmail.com>
> >> >>> > > > > > >>> > >> > >>>>> wrote:
> >> >>> > > > > > >>> > >> > >>>>>
> >> >>> > > > > > >>> > >> > >>>>>> Hi team,
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>> Hope everyone is doing well. I got a
> >> chance
> >> >>> to
> >> >>> > work
> >> >>> > > > > > >>> through all
> >> >>> > > > > > >>> > >> the
> >> >>> > > > > > >>> > >> > >>>>>> remaining feedback and update the spec
> >> doc.
> >> >>> Here
> >> >>> > > are
> >> >>> > > > > the
> >> >>> > > > > > >>> new
> >> >>> > > > > > >>> > >> > artifacts
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>> 1) Spec document :
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> >
> >> >>> > > > > > >>> > >>
> >> >>> > > > > > >>> >
> >> >>> > > > > > >>>
> >> >>> > > > > >
> >> >>> > > > >
> >> >>> > > >
> >> >>> > >
> >> >>> >
> >> >>>
> >>
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>> 2) Spec document in parquet format repo
> :
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > https://github.com/apache/parquet-format/pull/557
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>> 3) Alp implementation in arrow c++ repo
> :
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > https://github.com/apache/arrow/pull/48345/changes
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>> 4) Alp implementation in parquet-java
> >> repo :
> >> >>> Work
> >> >>> > > for
> >> >>> > > > > > >>> Vinoo and
> >> >>> > > > > > >>> > >> > Julien
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > https://github.com/apache/parquet-java/pull/3397
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>> 5) PR with test and benchmarking
> >> artifacts in
> >> >>> > > > > > >>> parquet-testing
> >> >>> > > > > > >>> > >> repo :
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > https://github.com/apache/parquet-testing/pull/100
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>> And
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>> - Go : Arnav just submitted an in
> >> >>> progress
> >> >>> > > > > > >>> implementation
> >> >>> > > > > > >>> > in
> >> >>> > > > > > >>> > >> Go.
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> https://github.com/apache/arrow-go/pull/704
> >> >>> > (I
> >> >>> > > > > > haven't
> >> >>> > > > > > >>> > >> started
> >> >>> > > > > > >>> > >> > >>>>>> looking at it yet)
> >> >>> > > > > > >>> > >> > >>>>>> - Rust : I remember Andrew mentioned
> >> that
> >> >>> > this
> >> >>> > > > work
> >> >>> > > > > > is
> >> >>> > > > > > >>> also
> >> >>> > > > > > >>> > >> in
> >> >>> > > > > > >>> > >> > >>>>>> progress (So 4 languages!)
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>> *Arrow C++ implementation *
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>> The PR is out and was also used by
> >> Antoine to
> >> >>> > > report
> >> >>> > > > > the
> >> >>> > > > > > >>> > numbers
> >> >>> > > > > > >>> > >> as
> >> >>> > > > > > >>> > >> > >>>>>> reported here. Micah and Konstantin have
> >> >>> given 1
> >> >>> > > > round
> >> >>> > > > > of
> >> >>> > > > > > >>> > >> feedback
> >> >>> > > > > > >>> > >> > >>>>>> and I'm addressing them today. Please
> note
> >> >>> that
> >> >>> > the
> >> >>> > > > > > default
> >> >>> > > > > > >>> > >> > >>>>>> optimization flag for compiling is O2
> and
> >> not
> >> >>> > Q3. I
> >> >>> > > > got
> >> >>> > > > > > >>> around
> >> >>> > > > > > >>> > >> 70%
> >> >>> > > > > > >>> > >> > >>>>>> performance improvement in the decoding
> >> speed
> >> >>> > when
> >> >>> > > > > using
> >> >>> > > > > > >>> the O3
> >> >>> > > > > > >>> > >> > flag.
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>> *Parqet-MR Java implementation (working
> >> with
> >> >>> > Vinoo
> >> >>> > > > and
> >> >>> > > > > > >>> Julien)
> >> >>> > > > > > >>> > >> and
> >> >>> > > > > > >>> > >> > **Cross
> >> >>> > > > > > >>> > >> > >>>>>> Language testing*
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>> Let me know if you have any
> questions
> >> or
> >> >>> > > > feedback.
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>> Now pasting some performance numbers
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>> Table 1: C++ ALP Double Decode —
> >> Spotify
> >> >>> > Columns
> >> >>> > > > > > >>> (Graviton
> >> >>> > > > > > >>> > 3,
> >> >>> > > > > > >>> > >> ARM
> >> >>> > > > > > >>> > >> > >>>>>> Neoverse V1)
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>>
> >> >>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>> │ Column │ -O2 (MB/s) │
> >> -O3
> >> >>> > > (MB/s) │
> >> >>> > > > > > >>> Speedup │
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>>
> >> >>> ├──────────────────┼──────────────┼──────────────┼─────────┤
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>> │ valence │ 3,155 │
> >> >>> 5,523
> >> >>> > > │
> >> >>> > > > > > >>> 1.75x │
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>> │ danceability │ 3,233 │
> >> >>> 5,685
> >> >>> > > │
> >> >>> > > > > > >>> 1.76x │
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>> │ energy │ 3,197 │
> >> >>> 5,652
> >> >>> > > │
> >> >>> > > > > > >>> 1.77x │
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>> │ loudness │ 3,186 │
> >> >>> 5,473
> >> >>> > > │
> >> >>> > > > > > >>> 1.72x │
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>>
> >> >>> └──────────────────┴──────────────┴──────────────┴─────────┘
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK
> >> GAUR
> >> >>> <
> >> >>> > > > > > >>> > prtkgaur@gmail.com
> >> >>> > > > > > >>> > >> >
> >> >>> > > > > > >>> > >> > >>>>>> wrote:
> >> >>> > > > > > >>> > >> > >>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>> @Micah Kornfield <
> emkornfield@gmail.com>
> >> :
> >> >>> Got
> >> >>> > > it.
> >> >>> > > > > > >>> > >> > >>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>> @Andrew Lamb <an...@gmail.com>
> >> >>> > > > > > >>> > >> > >>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>> Do you think it would be good to start
> >> >>> moving
> >> >>> > the
> >> >>> > > > > spec
> >> >>> > > > > > >>> > >> development
> >> >>> > > > > > >>> > >> > >>>>>>>> into
> >> >>> > > > > > >>> > >> > >>>>>>>> markdown format, in preparation for
> >> >>> finalizing
> >> >>> > > it?
> >> >>> > > > > > >>> > >> > >>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>> Yes I'll update the numbers for some of
> >> the
> >> >>> > > > examples I
> >> >>> > > > > > >>> have in
> >> >>> > > > > > >>> > >> the
> >> >>> > > > > > >>> > >> > >>>>>>> spec based
> >> >>> > > > > > >>> > >> > >>>>>>> on the updated header size. Then we
> >> should
> >> >>> be
> >> >>> > good
> >> >>> > > > to
> >> >>> > > > > go
> >> >>> > > > > > >>> for
> >> >>> > > > > > >>> > the
> >> >>> > > > > > >>> > >> > >>>>>>> markdown format.
> >> >>> > > > > > >>> > >> > >>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>> Thanks everyone!
> >> >>> > > > > > >>> > >> > >>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>> Andrew
> >> >>> > > > > > >>> > >> > >>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM
> PRATEEK
> >> >>> GAUR <
> >> >>> > > > > > >>> > >> prtkgaur@gmail.com>
> >> >>> > > > > > >>> > >> > >>>>>>>> wrote:
> >> >>> > > > > > >>> > >> > >>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>> Hi team,
> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>> 1) Andrew
> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for working on test
> files.
> >> >>> My PR
> >> >>> > > did
> >> >>> > > > > add
> >> >>> > > > > > >>> all
> >> >>> > > > > > >>> > the
> >> >>> > > > > > >>> > >> > test
> >> >>> > > > > > >>> > >> > >>>>>>>> files I
> >> >>> > > > > > >>> > >> > >>>>>>>>> used to benchmark on datasets.
> >> Maybe
> >> >>> we
> >> >>> > can
> >> >>> > > > club
> >> >>> > > > > > it
> >> >>> > > > > > >>> > >> together.
> >> >>> > > > > > >>> > >> > >>>>>>>> WIll also
> >> >>> > > > > > >>> > >> > >>>>>>>>> aid
> >> >>> > > > > > >>> > >> > >>>>>>>>> cross language testing
> >> >>> > > > > > >>> > >> > >>>>>>>>> - Kosta Tarasov working on Rust
> >> >>> > > > implementation.
> >> >>> > > > > > >>> This is
> >> >>> > > > > > >>> > >> > great.
> >> >>> > > > > > >>> > >> > >>>>>>>> Thanks
> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>> 2) Antoine
> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks a lot for reporting the
> >> >>> numbers
> >> >>> > on
> >> >>> > > > AMD.
> >> >>> > > > > > >>> Looks
> >> >>> > > > > > >>> > >> like
> >> >>> > > > > > >>> > >> > you
> >> >>> > > > > > >>> > >> > >>>>>>>> are
> >> >>> > > > > > >>> > >> > >>>>>>>>> getting 8X the decoding
> >> performance of
> >> >>> > BSS.
> >> >>> > > > This
> >> >>> > > > > > is
> >> >>> > > > > > >>> > >> > amazing!!.
> >> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for acknowledging the
> >> >>> sampling
> >> >>> > > > design.
> >> >>> > > > > > >>> > >> > >>>>>>>>> - I agree with you on Fastlanes.
> In
> >> >>> some
> >> >>> > > crude
> >> >>> > > > > > >>> > >> experiments I
> >> >>> > > > > > >>> > >> > >>>>>>>> didn't get
> >> >>> > > > > > >>> > >> > >>>>>>>>> a good perf benefit from it on
> >> >>> Graviton3
> >> >>> > > (but
> >> >>> > > > > > maybe
> >> >>> > > > > > >>> > there
> >> >>> > > > > > >>> > >> was
> >> >>> > > > > > >>> > >> > >>>>>>>> something
> >> >>> > > > > > >>> > >> > >>>>>>>>> wrong with my implementation).
> >> >>> > > > > > >>> > >> > >>>>>>>>> - Locking the 16bit exception
> >> >>> encoding for
> >> >>> > > the
> >> >>> > > > > > spec
> >> >>> > > > > > >>> in
> >> >>> > > > > > >>> > >> this
> >> >>> > > > > > >>> > >> > >>>>>>>> case.
> >> >>> > > > > > >>> > >> > >>>>>>>>> - Awesome I think we have solved
> >> for
> >> >>> all
> >> >>> > > open
> >> >>> > > > > > >>> questions
> >> >>> > > > > > >>> > >> minus
> >> >>> > > > > > >>> > >> > >>>>>>>> the
> >> >>> > > > > > >>> > >> > >>>>>>>>> version byte :). (will get back
> on
> >> >>> this
> >> >>> > > soon)
> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>> 3) Micah
> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>> - FastLanes : The current spec
> does
> >> >>> allow
> >> >>> > > for
> >> >>> > > > > > using
> >> >>> > > > > > >>> > >> FastLane
> >> >>> > > > > > >>> > >> > >>>>>>>> with the
> >> >>> > > > > > >>> > >> > >>>>>>>>> configurable enum value for
> >> layout. We
> >> >>> > > should
> >> >>> > > > be
> >> >>> > > > > > >>> able to
> >> >>> > > > > > >>> > >> > inject
> >> >>> > > > > > >>> > >> > >>>>>>>> any
> >> >>> > > > > > >>> > >> > >>>>>>>>> layout
> >> >>> > > > > > >>> > >> > >>>>>>>>> in the current design.
> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>> Working on resolving all remaining
> open
> >> >>> > comments
> >> >>> > > > on
> >> >>> > > > > > the
> >> >>> > > > > > >>> spec
> >> >>> > > > > > >>> > >> this
> >> >>> > > > > > >>> > >> > >>>>>>>> week.
> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>> Best
> >> >>> > > > > > >>> > >> > >>>>>>>>> Prateek
> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve
> >> >>> > Loughran <
> >> >>> > > > > > >>> > >> > >>>>>>>> stevel@cloudera.com>
> >> >>> > > > > > >>> > >> > >>>>>>>>> wrote:
> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah
> >> >>> Kornfield
> >> >>> > <
> >> >>> > > > > > >>> > >> > >>>>>>>> emkornfield@gmail.com>
> >> >>> > > > > > >>> > >> > >>>>>>>>>> wrote:
> >> >>> > > > > > >>> > >> > >>>>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>>>> It looks like the actual issue
> >> >>> described for
> >> >>> > > ORC
> >> >>> > > > > in
> >> >>> > > > > > >>> the
> >> >>> > > > > > >>> > >> paper
> >> >>> > > > > > >>> > >> > >>>>>>>> is that
> >> >>> > > > > > >>> > >> > >>>>>>>>> it
> >> >>> > > > > > >>> > >> > >>>>>>>>>>> has multiple sub-encodings in a
> >> batch.
> >> >>> This
> >> >>> > > is
> >> >>> > > > > > >>> different
> >> >>> > > > > > >>> > >> then
> >> >>> > > > > > >>> > >> > >>>>>>>> the
> >> >>> > > > > > >>> > >> > >>>>>>>>> design
> >> >>> > > > > > >>> > >> > >>>>>>>>>>> proposed here where there is still
> >> fixed
> >> >>> > > > encoding
> >> >>> > > > > > per
> >> >>> > > > > > >>> page
> >> >>> > > > > > >>> > >> in
> >> >>> > > > > > >>> > >> > >>>>>>>> parquet.
> >> >>> > > > > > >>> > >> > >>>>>>>>>>> Given reasonably sized pages I
> don't
> >> >>> think
> >> >>> > > > branch
> >> >>> > > > > > >>> > >> > >>>>>>>> misprediction should
> >> >>> > > > > > >>> > >> > >>>>>>>>>> be a
> >> >>> > > > > > >>> > >> > >>>>>>>>>>> big issue for new encodings. I
> agree
> >> >>> that
> >> >>> > we
> >> >>> > > > > should
> >> >>> > > > > > >>> be
> >> >>> > > > > > >>> > >> > >>>>>>>> conservative in
> >> >>> > > > > > >>> > >> > >>>>>>>>>>> general for adding new encodings.
> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>>> +1
> >> >>> > > > > > >>> > >> > >>>>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>>
> >> >>> > > > > > >>> > >> > >>>>>>>
> >> >>> > > > > > >>> > >> > >
> >> >>> > > > > > >>> > >> >
> >> >>> > > > > > >>> > >> >
> >> >>> > > > > > >>> > >> >
> >> >>> > > > > > >>> > >>
> >> >>> > > > > > >>> > >
> >> >>> > > > > > >>> >
> >> >>> > > > > > >>>
> >> >>> > > > > > >>
> >> >>> > > > > >
> >> >>> > > > >
> >> >>> > > >
> >> >>> > >
> >> >>> >
> >> >>>
> >> >>
> >>
> >
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Andrew Lamb <an...@gmail.com>.
Div: Here is a proposal to mark ALP in Preview: [1] (using a comment in the
spec section).
Russel: That is my understanding as well.
Hopefully we are able to formalize the versioning process shortly so we can
formalize the notion of "preview"
Andrew
[1]: https://github.com/apache/parquet-format/pull/604
On Fri, Jul 31, 2026 at 11:15 AM Russell Spitzer <ru...@gmail.com>
wrote:
> I thought we had agreed it would be merged, and features like this would
> be feature flag gated. So a V-Old writer could always opt in to using this
> encoding but that must be a conscious effort if you are not using VNext of
> Parquet. If you are on V-Next the writer can choose to use ALP whenever it
> likes without user intervention.
>
> On Fri, Jul 31, 2026 at 2:20 AM Divjot Arora via dev <
> dev@parquet.apache.org> wrote:
>
>> Yes, that’s my understanding as well. However, as you noted, the
>> versioning
>> discussion is still in early stages and hasn’t prescribed concrete
>> mechanisms yet. If ALP goes into parquet-format 2.14.0, there’s no
>> indication that it’s in preview or that writers shouldn’t use it by
>> default
>> yet. Given that we’ve held off on merging other incompatible changes (e.g
>> removal of path_in_schema) until the versioning discussion closes, my
>> preference would be to leave ALP un-merged for the time being as well.
>>
>> — Div
>>
>> On Fri, Jul 31, 2026 at 08:59 Andrew Lamb <an...@gmail.com> wrote:
>>
>> > After re-reading Julien's Versioning Proposal, my understanding is that
>> if
>> > we adopt that system, we will somehow mark ALP in "Preview" in
>> > parquet-format for release Version 3.
>> >
>> > The versioning document doesn't seem to proscribe the particular method
>> > for annotating "preview" features -- I think clearly marking them as
>> such
>> > in parquet.thrift as well as reflecting that on the
>> > https://parquet.apache.org/docs/file-format/versions/ page somehow
>> would
>> > be adequate.
>> >
>> > Andrew
>> >
>> > On Thu, Jul 30, 2026 at 11:50 AM Andrew Lamb <an...@gmail.com>
>> > wrote:
>> >
>> >> In my mind ALP is in the same position as other encodings that were
>> added
>> >> recently[1] (such as BYTE_STREAM_SPLIT for non floating point types)
>> >>
>> >> That is to say it will be available in the latest open source writers
>> and
>> >> readers, but realistically not widely used in the ecosystem where
>> >> compatibility is a concern (a large part)
>> >>
>> >> I view Julian's work[2][3] to gather consensus on how to name
>> >> combinations of features (e.g. "versions") as the mechanism by which we
>> >> will accelerate the adoption of this change (along with other forward
>> >> incompatible changes) across the ecosystem.
>> >>
>> >> > Will we exclude this change when cutting parquet-format v2.14.0?
>> >>
>> >> No I don't think so -- I would expect it to go out in the next
>> >> parquet-format version as do other changes in the spec.
>> >>
>> >> Andrew
>> >>
>> >> [1]: https://parquet.apache.org/docs/file-format/versions/
>> >> [2]: https://lists.apache.org/thread/xmnj8h0h8ozmrgox5tydhhs7yz7q2hss
>> >> [3]: https://lists.apache.org/thread/qlf8lg90gqq41lllyy9mk2f0skqvftqj
>> >>
>> >>
>> >> On Thu, Jul 30, 2026 at 3:51 AM Divjot Arora via dev <
>> >> dev@parquet.apache.org> wrote:
>> >>
>> >>> Hi Andrew,
>> >>>
>> >>> How does this interact with the ongoing versioning discussion? Given
>> that
>> >>> new encodings are not forward compatible, I would assume ALP is
>> included
>> >>> in
>> >>> Parquet V<next> (V3?), but it's not clear to me how that works if it's
>> >>> merged to the parquet-format master branch now. Will we exclude this
>> >>> change
>> >>> when cutting parquet-format v2.14.0?
>> >>>
>> >>> -- Div
>> >>>
>> >>> On Thu, Jul 30, 2026 at 8:51 AM Andrew Lamb <an...@gmail.com>
>> >>> wrote:
>> >>>
>> >>> > Hello,
>> >>> >
>> >>> > As an update, thanks to the epic work from many contributors,
>> >>> especially
>> >>> > Prateek, I just merged the parquet-format change for ALP[1]. We
>> plan to
>> >>> > make additional editorial The remaining items I know of are merging
>> the
>> >>> > examples from parquet-testing[2], and then of course the various
>> >>> > implementations
>> >>> >
>> >>> > Thanks again everyone. This is a great step forward
>> >>> > Andrew
>> >>> >
>> >>> > [1]: https://github.com/apache/parquet-format/pull/557
>> >>> > [2]: https://github.com/apache/parquet-testing/pull/100
>> >>> >
>> >>> > On Tue, Jul 21, 2026 at 4:39 PM Julien Le Dem <ju...@apache.org>
>> >>> wrote:
>> >>> >
>> >>> > > It was also in my spam folder... :(
>> >>> > > I've seen a bunch of apache list emails go to my spam recently.
>> >>> > >
>> >>> > > On Tue, Jul 21, 2026 at 10:58 AM Russell Spitzer <
>> >>> > > russell.spitzer@gmail.com>
>> >>> > > wrote:
>> >>> > >
>> >>> > > > Yes, my hope was that my vote would kick it out of spam but we
>> >>> > definitely
>> >>> > > > had two other votes that also went to spam for me.
>> >>> > > >
>> >>> > > > For anyone looking the thread is here
>> >>> > > >
>> https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt
>> >>> > > >
>> >>> > > > On Tue, Jul 21, 2026 at 10:57 AM Micah Kornfield <
>> >>> > emkornfield@gmail.com>
>> >>> > > > wrote:
>> >>> > > >
>> >>> > > > > I've seen other's votes not go to spam, so hopefully people
>> will
>> >>> see
>> >>> > it
>> >>> > > > in
>> >>> > > > > there inboxes (or see this thread and look in there spam).
>> >>> > > > >
>> >>> > > > > On Tue, Jul 21, 2026 at 10:52 AM PRATEEK GAUR <
>> >>> prtkgaur@gmail.com>
>> >>> > > > wrote:
>> >>> > > > >
>> >>> > > > > > Okay,
>> >>> > > > > >
>> >>> > > > > > I sent out the voting email yesterday and so far I've heard
>> >>> from 3
>> >>> > > > people
>> >>> > > > > > that it ended up in their "spam folder". Dev VPN to be
>> blamed
>> >>> for
>> >>> > > it?.
>> >>> > > > > >
>> >>> > > > > > What's the protocol here?
>> >>> > > > > > I let people hunt for the email in their spam folder or send
>> >>> out
>> >>> > > > another
>> >>> > > > > > one?
>> >>> > > > > >
>> >>> > > > > > Best
>> >>> > > > > > Prateek
>> >>> > > > > >
>> >>> > > > > > On Sun, Jul 19, 2026 at 10:47 PM PRATEEK GAUR <
>> >>> prtkgaur@gmail.com>
>> >>> > > > > wrote:
>> >>> > > > > >
>> >>> > > > > > > Hi team,
>> >>> > > > > > >
>> >>> > > > > > > Vinoo has added the tests which were requested in the last
>> >>> round
>> >>> > of
>> >>> > > > > > review.
>> >>> > > > > > > In his own words.
>> >>> > > > > > >
>> >>> > > > > > > "
>> >>> > > > > > > The extreme value tests Micah wanted are done and pushed.
>> I
>> >>> added
>> >>> > > > > > coverage
>> >>> > > > > > > for values that need the full FOR bit width after the
>> frame
>> >>> of
>> >>> > > > > reference
>> >>> > > > > > is
>> >>> > > > > > > applied: 64-bit (the case where the signed max minus min
>> >>> > > subtraction
>> >>> > > > > > > overflows), 63-bit (the non-overflow case), and 32-bit for
>> >>> > floats.
>> >>> > > > All
>> >>> > > > > > > lossless with zero exceptions. I also verified them
>> through
>> >>> Arrow
>> >>> > > C++
>> >>> > > > > ALP
>> >>> > > > > > > reader (the local build), so the Java-written extreme
>> columns
>> >>> > > decode
>> >>> > > > > > > correctly in C++, bit-exact against the expected values."
>> >>> > > > > > >
>> >>> > > > > > > State of the PR (#3397):
>> >>> > > > > > >
>> >>> > > > > > > - Merges cleanly with master (I pulled master in and
>> >>> resolved
>> >>> > > the
>> >>> > > > > > > conflicts).
>> >>> > > > > > > - 41 of 44 review threads resolved. The 3 remaining are
>> >>> > > > intentional:
>> >>> > > > > > > one deferred perf nit, one that stays open until
>> >>> > parquet-format
>> >>> > > > > ships
>> >>> > > > > > ALP,
>> >>> > > > > > >
>> >>> > > > > > > "
>> >>> > > > > > >
>> >>> > > > > > > With this both the
>> >>> > > > > > > 1) C++ PR :
>> >>> https://github.com/apache/arrow/pull/48345/changes
>> >>> > > > > > > 2) Java PR :
>> >>> https://github.com/apache/parquet-java/pull/3397
>> >>> > > > > > >
>> >>> > > > > > > Are cross language tested and range of test scenarios.
>> >>> > > > > > >
>> >>> > > > > > > Best
>> >>> > > > > > > Prateek
>> >>> > > > > > >
>> >>> > > > > > >
>> >>> > > > > > > On Mon, Jul 13, 2026 at 8:38 AM PRATEEK GAUR <
>> >>> prtkgaur@gmail.com
>> >>> > >
>> >>> > > > > wrote:
>> >>> > > > > > >
>> >>> > > > > > >> Thanks Matt,
>> >>> > > > > > >>
>> >>> > > > > > >> I'll be on it this week.
>> >>> > > > > > >>
>> >>> > > > > > >> Best
>> >>> > > > > > >> Prateek
>> >>> > > > > > >>
>> >>> > > > > > >> On Mon, Jul 13, 2026 at 8:36 AM Matt Topol <
>> >>> > > zotthewizard@gmail.com>
>> >>> > > > > > >> wrote:
>> >>> > > > > > >>
>> >>> > > > > > >>> There is also the Go implementation submitted by Arnav (
>> >>> > > > > > >>> https://github.com/apache/arrow-go/pull/704) which is
>> >>> waiting
>> >>> > > for
>> >>> > > > > > >>> updates.
>> >>> > > > > > >>>
>> >>> > > > > > >>> Just wanted to make sure it didn't get lost here.
>> >>> > > > > > >>>
>> >>> > > > > > >>> --Matt
>> >>> > > > > > >>>
>> >>> > > > > > >>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK GAUR <
>> >>> > > prtkgaur@gmail.com>
>> >>> > > > > > >>> wrote:
>> >>> > > > > > >>>
>> >>> > > > > > >>> > Hi all,
>> >>> > > > > > >>> >
>> >>> > > > > > >>> > I'd like to share a status update on the ALP encoding
>> >>> effort
>> >>> > > and
>> >>> > > > > get
>> >>> > > > > > a
>> >>> > > > > > >>> feel
>> >>> > > > > > >>> > from the community before starting a formal vote.
>> >>> > > > > > >>> >
>> >>> > > > > > >>> > We now have the specification plus implementations in
>> >>> both
>> >>> > > > > languages:
>> >>> > > > > > >>> >
>> >>> > > > > > >>> > - Spec: apache/parquet-format#557 — GH-533
>> >>> Add
>> >>> > ALP
>> >>> > > > > > encoding
>> >>> > > > > > >>> > specification
>> >>> > > > > > >>> > - C++ (Arrow): apache/arrow#48345 — GH-48701
>> >>> > > [C++][Parquet]
>> >>> > > > > Add
>> >>> > > > > > >>> ALPpd
>> >>> > > > > > >>> > encoding
>> >>> > > > > > >>> > - Java: apache/parquet-java#3397 — Parquet
>> >>> Java
>> >>> > ALP
>> >>> > > > > > >>> > Implementation (by Vinoo)
>> >>> > > > > > >>> >
>> >>> > > > > > >>> > Current state:
>> >>> > > > > > >>> >
>> >>> > > > > > >>> > - The C++ implementation has been through 3-4
>> rounds
>> >>> of
>> >>> > > > review.
>> >>> > > > > > >>> > - The Java implementation has been tracking well
>> and
>> >>> has
>> >>> > > had
>> >>> > > > > > >>> initial
>> >>> > > > > > >>> > reviews from contributors, with no major
>> remaining
>> >>> > > issues.
>> >>> > > > > > >>> > - Cross-language compatibility tests are passing:
>> the
>> >>> > Arrow
>> >>> > > > C++
>> >>> > > > > > >>> decoder
>> >>> > > > > > >>> > reads Java-written data bit-exactly across
>> ~1.56M
>> >>> > values
>> >>> > > > and
>> >>> > > > > 18
>> >>> > > > > > >>> > fixtures, covering V1 and V2 pages, multiple
>> vector
>> >>> > > sizes,
>> >>> > > > > and
>> >>> > > > > > >>> > several
>> >>> > > > > > >>> > real datasets — zero mismatches.
>> >>> > > > > > >>> >
>> >>> > > > > > >>> > As raised in the java-pr we will be adding coverage
>> for
>> >>> > > extreme
>> >>> > > > > > values
>> >>> > > > > > >>> > (those requiring
>> >>> > > > > > >>> > 63-64 bits after FOR is applied) before we close a
>> vote.
>> >>> > We'll
>> >>> > > > aim
>> >>> > > > > > to
>> >>> > > > > > >>> get
>> >>> > > > > > >>> > that
>> >>> > > > > > >>> > done in parallel.
>> >>> > > > > > >>> >
>> >>> > > > > > >>> > Plan: unless there are objections, I intend to start a
>> >>> vote
>> >>> > at
>> >>> > > > the
>> >>> > > > > > end
>> >>> > > > > > >>> of
>> >>> > > > > > >>> > this week/early
>> >>> > > > > > >>> > next week.
>> >>> > > > > > >>> >
>> >>> > > > > > >>> > Thanks
>> >>> > > > > > >>> > Prateek
>> >>> > > > > > >>> >
>> >>> > > > > > >>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <
>> >>> > > prtkgaur@gmail.com>
>> >>> > > > > > >>> wrote:
>> >>> > > > > > >>> >
>> >>> > > > > > >>> > > Hi Team,
>> >>> > > > > > >>> > >
>> >>> > > > > > >>> > > Just wanted to provide some updates on ALP.
>> >>> > > > > > >>> > > Micah and I have done 3-4 rounds of review for the
>> c++
>> >>> PR :
>> >>> > > > > > >>> > > https://github.com/apache/arrow/pull/48345
>> >>> > > > > > >>> > >
>> >>> > > > > > >>> > > For the Java implementation Vinoo has been working
>> on
>> >>> > > following
>> >>> > > > > PR
>> >>> > > > > > :
>> >>> > > > > > >>> > > https://github.com/apache/parquet-java/pull/3397
>> >>> > > > > > >>> > >
>> >>> > > > > > >>> > > Best
>> >>> > > > > > >>> > > Prateek
>> >>> > > > > > >>> > >
>> >>> > > > > > >>> > > On Tue, May 5, 2026 at 1:50 PM Micah Kornfield <
>> >>> > > > > > >>> emkornfield@gmail.com>
>> >>> > > > > > >>> > > wrote:
>> >>> > > > > > >>> > >
>> >>> > > > > > >>> > >> Hi Antoine,
>> >>> > > > > > >>> > >>
>> >>> > > > > > >>> > >> > Apologies if the question was already asked, but
>> >>> should
>> >>> > we
>> >>> > > > > care
>> >>> > > > > > >>> about
>> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more
>> efficient
>> >>> than
>> >>> > > > > FLOAT16
>> >>> > > > > > +
>> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
>> >>> > > > > > >>> > >>
>> >>> > > > > > >>> > >> It was. We thought we could defer it for the
>> >>> following
>> >>> > > > reasons:
>> >>> > > > > > >>> > >> 1. It's not clear there are a lot of easy
>> reference
>> >>> > > datasets
>> >>> > > > to
>> >>> > > > > > >>> test
>> >>> > > > > > >>> > its
>> >>> > > > > > >>> > >> effectiveness. It does look like there might be
>> one
>> >>> or
>> >>> > two
>> >>> > > on
>> >>> > > > > > >>> > huggingface
>> >>> > > > > > >>> > >> (e.g.
>> https://huggingface.co/datasets/kikitora/curdie
>> >>> ).
>> >>> > > > > > >>> > >> 2. It seemed likely that float 16 was more likely
>> >>> used
>> >>> > for
>> >>> > > > > values
>> >>> > > > > > >>> that
>> >>> > > > > > >>> > >> were less likely to reduce to decimal values.
>> >>> > > > > > >>> > >> 3. It could be added as an extension later if
>> needed.
>> >>> > > > > > >>> > >>
>> >>> > > > > > >>> > >> Cheers,
>> >>> > > > > > >>> > >> Micah
>> >>> > > > > > >>> > >>
>> >>> > > > > > >>> > >>
>> >>> > > > > > >>> > >> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <
>> >>> > > > > antoine@python.org
>> >>> > > > > > >
>> >>> > > > > > >>> > wrote:
>> >>> > > > > > >>> > >>
>> >>> > > > > > >>> > >> >
>> >>> > > > > > >>> > >> > Hello,
>> >>> > > > > > >>> > >> >
>> >>> > > > > > >>> > >> > Apologies if the question was already asked, but
>> >>> should
>> >>> > we
>> >>> > > > > care
>> >>> > > > > > >>> about
>> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more
>> efficient
>> >>> than
>> >>> > > > > FLOAT16
>> >>> > > > > > +
>> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
>> >>> > > > > > >>> > >> >
>> >>> > > > > > >>> > >> > Regards
>> >>> > > > > > >>> > >> >
>> >>> > > > > > >>> > >> > Antoine.
>> >>> > > > > > >>> > >> >
>> >>> > > > > > >>> > >> >
>> >>> > > > > > >>> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
>> >>> > > > > > >>> > >> > > Thanks Andrew and Micah for review feedback on
>> >>> the two
>> >>> > > > PR's
>> >>> > > > > > >>> > >> > > 1) (c++ arrow repo)
>> >>> > > > > > >>> > >> https://github.com/apache/arrow/pull/48345/changes
>> >>> > > > > > >>> > >> > > 2) (parquet-format repo)
>> >>> > > > > > >>> > >> >
>> https://github.com/apache/parquet-format/pull/557
>> >>> > > > > > >>> > >> > >
>> >>> > > > > > >>> > >> > > I have addressed all (unless I missed
>> something)
>> >>> > > comments
>> >>> > > > on
>> >>> > > > > > >>> the two
>> >>> > > > > > >>> > >> > PR's.
>> >>> > > > > > >>> > >> > >
>> >>> > > > > > >>> > >> > > Best
>> >>> > > > > > >>> > >> > > Prateek
>> >>> > > > > > >>> > >> > >
>> >>> > > > > > >>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <
>> >>> > > > > > >>> prtkgaur@gmail.com>
>> >>> > > > > > >>> > >> wrote:
>> >>> > > > > > >>> > >> > >
>> >>> > > > > > >>> > >> > >> Thanks Andrew and Micah.
>> >>> > > > > > >>> > >> > >>
>> >>> > > > > > >>> > >> > >> `fair amount of feedback on at least the
>> >>> > > implementations`
>> >>> > > > > > >>> > >> > >> For the c++ I have already started addressing
>> the
>> >>> > > > > feedback, I
>> >>> > > > > > >>> > should
>> >>> > > > > > >>> > >> be
>> >>> > > > > > >>> > >> > >> done with that Monday/Tuesday.
>> >>> > > > > > >>> > >> > >> I think Vinoo too has been making good
>> progress
>> >>> on
>> >>> > the
>> >>> > > > Java
>> >>> > > > > > >>> > >> > implementation.
>> >>> > > > > > >>> > >> > >>
>> >>> > > > > > >>> > >> > >> Best
>> >>> > > > > > >>> > >> > >> Prateek
>> >>> > > > > > >>> > >> > >>
>> >>> > > > > > >>> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <
>> >>> > > > > > >>> > andrewlamb11@gmail.com
>> >>> > > > > > >>> > >> >
>> >>> > > > > > >>> > >> > >> wrote:
>> >>> > > > > > >>> > >> > >>
>> >>> > > > > > >>> > >> > >>> Got it. Thank you for the clarification -- I
>> >>> will
>> >>> > try
>> >>> > > > and
>> >>> > > > > > look
>> >>> > > > > > >>> > into
>> >>> > > > > > >>> > >> the
>> >>> > > > > > >>> > >> > >>> spec and the Rust implementation[1] in this
>> next
>> >>> > week
>> >>> > > > > > >>> > >> > >>>
>> >>> > > > > > >>> > >> > >>> [1]:
>> >>> https://github.com/apache/arrow-rs/pull/9372
>> >>> > > > > > >>> > >> > >>>
>> >>> > > > > > >>> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah
>> >>> Kornfield <
>> >>> > > > > > >>> > >> > emkornfield@gmail.com>
>> >>> > > > > > >>> > >> > >>> wrote:
>> >>> > > > > > >>> > >> > >>>
>> >>> > > > > > >>> > >> > >>>> Hi Andrew,
>> >>> > > > > > >>> > >> > >>>> I think there is a fair amount of feedback
>> on
>> >>> at
>> >>> > > least
>> >>> > > > > the
>> >>> > > > > > >>> > >> > >>>> implementations, typically I think we've
>> waited
>> >>> > till
>> >>> > > > they
>> >>> > > > > > are
>> >>> > > > > > >>> > >> close to
>> >>> > > > > > >>> > >> > >>>> mergeable before a final vote. Otherwise I
>> >>> agree
>> >>> > we
>> >>> > > > are
>> >>> > > > > > very
>> >>> > > > > > >>> > >> close.
>> >>> > > > > > >>> > >> > >>>>
>> >>> > > > > > >>> > >> > >>>> -Micah
>> >>> > > > > > >>> > >> > >>>>
>> >>> > > > > > >>> > >> > >>>> On Saturday, April 25, 2026, Andrew Lamb <
>> >>> > > > > > >>> andrewlamb11@gmail.com
>> >>> > > > > > >>> > >
>> >>> > > > > > >>> > >> > wrote:
>> >>> > > > > > >>> > >> > >>>>
>> >>> > > > > > >>> > >> > >>>>> Thanks Prateek,
>> >>> > > > > > >>> > >> > >>>>>
>> >>> > > > > > >>> > >> > >>>>> I think from this content it looks to me
>> like
>> >>> we
>> >>> > are
>> >>> > > > > ready
>> >>> > > > > > >>> to
>> >>> > > > > > >>> > >> start a
>> >>> > > > > > >>> > >> > >>>>> vote to explicitly accept ALP into Parquet
>> >>> > > > > > >>> > >> > >>>>>
>> >>> > > > > > >>> > >> > >>>>> Does anyone know of a reason we should
>> >>> postpone it
>> >>> > > for
>> >>> > > > > > >>> longer?
>> >>> > > > > > >>> > >> > >>>>> Perhaps someone needs some more time to
>> >>> review?
>> >>> > > > > > >>> > >> > >>>>>
>> >>> > > > > > >>> > >> > >>>>> Andrew
>> >>> > > > > > >>> > >> > >>>>>
>> >>> > > > > > >>> > >> > >>>>>
>> >>> > > > > > >>> > >> > >>>>>
>> >>> > > > > > >>> > >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK
>> GAUR <
>> >>> > > > > > >>> > prtkgaur@gmail.com>
>> >>> > > > > > >>> > >> > >>>>> wrote:
>> >>> > > > > > >>> > >> > >>>>>
>> >>> > > > > > >>> > >> > >>>>>> Hi team,
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>> Hope everyone is doing well. I got a
>> chance
>> >>> to
>> >>> > work
>> >>> > > > > > >>> through all
>> >>> > > > > > >>> > >> the
>> >>> > > > > > >>> > >> > >>>>>> remaining feedback and update the spec
>> doc.
>> >>> Here
>> >>> > > are
>> >>> > > > > the
>> >>> > > > > > >>> new
>> >>> > > > > > >>> > >> > artifacts
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>> 1) Spec document :
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> >
>> >>> > > > > > >>> > >>
>> >>> > > > > > >>> >
>> >>> > > > > > >>>
>> >>> > > > > >
>> >>> > > > >
>> >>> > > >
>> >>> > >
>> >>> >
>> >>>
>> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>> 2) Spec document in parquet format repo :
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > https://github.com/apache/parquet-format/pull/557
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>> 3) Alp implementation in arrow c++ repo :
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > https://github.com/apache/arrow/pull/48345/changes
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>> 4) Alp implementation in parquet-java
>> repo :
>> >>> Work
>> >>> > > for
>> >>> > > > > > >>> Vinoo and
>> >>> > > > > > >>> > >> > Julien
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > https://github.com/apache/parquet-java/pull/3397
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>> 5) PR with test and benchmarking
>> artifacts in
>> >>> > > > > > >>> parquet-testing
>> >>> > > > > > >>> > >> repo :
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > https://github.com/apache/parquet-testing/pull/100
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>> And
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>> - Go : Arnav just submitted an in
>> >>> progress
>> >>> > > > > > >>> implementation
>> >>> > > > > > >>> > in
>> >>> > > > > > >>> > >> Go.
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> https://github.com/apache/arrow-go/pull/704
>> >>> > (I
>> >>> > > > > > haven't
>> >>> > > > > > >>> > >> started
>> >>> > > > > > >>> > >> > >>>>>> looking at it yet)
>> >>> > > > > > >>> > >> > >>>>>> - Rust : I remember Andrew mentioned
>> that
>> >>> > this
>> >>> > > > work
>> >>> > > > > > is
>> >>> > > > > > >>> also
>> >>> > > > > > >>> > >> in
>> >>> > > > > > >>> > >> > >>>>>> progress (So 4 languages!)
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>> *Arrow C++ implementation *
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>> The PR is out and was also used by
>> Antoine to
>> >>> > > report
>> >>> > > > > the
>> >>> > > > > > >>> > numbers
>> >>> > > > > > >>> > >> as
>> >>> > > > > > >>> > >> > >>>>>> reported here. Micah and Konstantin have
>> >>> given 1
>> >>> > > > round
>> >>> > > > > of
>> >>> > > > > > >>> > >> feedback
>> >>> > > > > > >>> > >> > >>>>>> and I'm addressing them today. Please note
>> >>> that
>> >>> > the
>> >>> > > > > > default
>> >>> > > > > > >>> > >> > >>>>>> optimization flag for compiling is O2 and
>> not
>> >>> > Q3. I
>> >>> > > > got
>> >>> > > > > > >>> around
>> >>> > > > > > >>> > >> 70%
>> >>> > > > > > >>> > >> > >>>>>> performance improvement in the decoding
>> speed
>> >>> > when
>> >>> > > > > using
>> >>> > > > > > >>> the O3
>> >>> > > > > > >>> > >> > flag.
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>> *Parqet-MR Java implementation (working
>> with
>> >>> > Vinoo
>> >>> > > > and
>> >>> > > > > > >>> Julien)
>> >>> > > > > > >>> > >> and
>> >>> > > > > > >>> > >> > **Cross
>> >>> > > > > > >>> > >> > >>>>>> Language testing*
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>> Let me know if you have any questions
>> or
>> >>> > > > feedback.
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>> Now pasting some performance numbers
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>> Table 1: C++ ALP Double Decode —
>> Spotify
>> >>> > Columns
>> >>> > > > > > >>> (Graviton
>> >>> > > > > > >>> > 3,
>> >>> > > > > > >>> > >> ARM
>> >>> > > > > > >>> > >> > >>>>>> Neoverse V1)
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>>
>> >>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>> │ Column │ -O2 (MB/s) │
>> -O3
>> >>> > > (MB/s) │
>> >>> > > > > > >>> Speedup │
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>>
>> >>> ├──────────────────┼──────────────┼──────────────┼─────────┤
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>> │ valence │ 3,155 │
>> >>> 5,523
>> >>> > > │
>> >>> > > > > > >>> 1.75x │
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>> │ danceability │ 3,233 │
>> >>> 5,685
>> >>> > > │
>> >>> > > > > > >>> 1.76x │
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>> │ energy │ 3,197 │
>> >>> 5,652
>> >>> > > │
>> >>> > > > > > >>> 1.77x │
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>> │ loudness │ 3,186 │
>> >>> 5,473
>> >>> > > │
>> >>> > > > > > >>> 1.72x │
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>>
>> >>> └──────────────────┴──────────────┴──────────────┴─────────┘
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK
>> GAUR
>> >>> <
>> >>> > > > > > >>> > prtkgaur@gmail.com
>> >>> > > > > > >>> > >> >
>> >>> > > > > > >>> > >> > >>>>>> wrote:
>> >>> > > > > > >>> > >> > >>>>>>
>> >>> > > > > > >>> > >> > >>>>>>> @Micah Kornfield <em...@gmail.com>
>> :
>> >>> Got
>> >>> > > it.
>> >>> > > > > > >>> > >> > >>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>> @Andrew Lamb <an...@gmail.com>
>> >>> > > > > > >>> > >> > >>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>> Do you think it would be good to start
>> >>> moving
>> >>> > the
>> >>> > > > > spec
>> >>> > > > > > >>> > >> development
>> >>> > > > > > >>> > >> > >>>>>>>> into
>> >>> > > > > > >>> > >> > >>>>>>>> markdown format, in preparation for
>> >>> finalizing
>> >>> > > it?
>> >>> > > > > > >>> > >> > >>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>> Yes I'll update the numbers for some of
>> the
>> >>> > > > examples I
>> >>> > > > > > >>> have in
>> >>> > > > > > >>> > >> the
>> >>> > > > > > >>> > >> > >>>>>>> spec based
>> >>> > > > > > >>> > >> > >>>>>>> on the updated header size. Then we
>> should
>> >>> be
>> >>> > good
>> >>> > > > to
>> >>> > > > > go
>> >>> > > > > > >>> for
>> >>> > > > > > >>> > the
>> >>> > > > > > >>> > >> > >>>>>>> markdown format.
>> >>> > > > > > >>> > >> > >>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>> Thanks everyone!
>> >>> > > > > > >>> > >> > >>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>> Andrew
>> >>> > > > > > >>> > >> > >>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK
>> >>> GAUR <
>> >>> > > > > > >>> > >> prtkgaur@gmail.com>
>> >>> > > > > > >>> > >> > >>>>>>>> wrote:
>> >>> > > > > > >>> > >> > >>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>> Hi team,
>> >>> > > > > > >>> > >> > >>>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>> 1) Andrew
>> >>> > > > > > >>> > >> > >>>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for working on test files.
>> >>> My PR
>> >>> > > did
>> >>> > > > > add
>> >>> > > > > > >>> all
>> >>> > > > > > >>> > the
>> >>> > > > > > >>> > >> > test
>> >>> > > > > > >>> > >> > >>>>>>>> files I
>> >>> > > > > > >>> > >> > >>>>>>>>> used to benchmark on datasets.
>> Maybe
>> >>> we
>> >>> > can
>> >>> > > > club
>> >>> > > > > > it
>> >>> > > > > > >>> > >> together.
>> >>> > > > > > >>> > >> > >>>>>>>> WIll also
>> >>> > > > > > >>> > >> > >>>>>>>>> aid
>> >>> > > > > > >>> > >> > >>>>>>>>> cross language testing
>> >>> > > > > > >>> > >> > >>>>>>>>> - Kosta Tarasov working on Rust
>> >>> > > > implementation.
>> >>> > > > > > >>> This is
>> >>> > > > > > >>> > >> > great.
>> >>> > > > > > >>> > >> > >>>>>>>> Thanks
>> >>> > > > > > >>> > >> > >>>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>> 2) Antoine
>> >>> > > > > > >>> > >> > >>>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks a lot for reporting the
>> >>> numbers
>> >>> > on
>> >>> > > > AMD.
>> >>> > > > > > >>> Looks
>> >>> > > > > > >>> > >> like
>> >>> > > > > > >>> > >> > you
>> >>> > > > > > >>> > >> > >>>>>>>> are
>> >>> > > > > > >>> > >> > >>>>>>>>> getting 8X the decoding
>> performance of
>> >>> > BSS.
>> >>> > > > This
>> >>> > > > > > is
>> >>> > > > > > >>> > >> > amazing!!.
>> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for acknowledging the
>> >>> sampling
>> >>> > > > design.
>> >>> > > > > > >>> > >> > >>>>>>>>> - I agree with you on Fastlanes. In
>> >>> some
>> >>> > > crude
>> >>> > > > > > >>> > >> experiments I
>> >>> > > > > > >>> > >> > >>>>>>>> didn't get
>> >>> > > > > > >>> > >> > >>>>>>>>> a good perf benefit from it on
>> >>> Graviton3
>> >>> > > (but
>> >>> > > > > > maybe
>> >>> > > > > > >>> > there
>> >>> > > > > > >>> > >> was
>> >>> > > > > > >>> > >> > >>>>>>>> something
>> >>> > > > > > >>> > >> > >>>>>>>>> wrong with my implementation).
>> >>> > > > > > >>> > >> > >>>>>>>>> - Locking the 16bit exception
>> >>> encoding for
>> >>> > > the
>> >>> > > > > > spec
>> >>> > > > > > >>> in
>> >>> > > > > > >>> > >> this
>> >>> > > > > > >>> > >> > >>>>>>>> case.
>> >>> > > > > > >>> > >> > >>>>>>>>> - Awesome I think we have solved
>> for
>> >>> all
>> >>> > > open
>> >>> > > > > > >>> questions
>> >>> > > > > > >>> > >> minus
>> >>> > > > > > >>> > >> > >>>>>>>> the
>> >>> > > > > > >>> > >> > >>>>>>>>> version byte :). (will get back on
>> >>> this
>> >>> > > soon)
>> >>> > > > > > >>> > >> > >>>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>> 3) Micah
>> >>> > > > > > >>> > >> > >>>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>> - FastLanes : The current spec does
>> >>> allow
>> >>> > > for
>> >>> > > > > > using
>> >>> > > > > > >>> > >> FastLane
>> >>> > > > > > >>> > >> > >>>>>>>> with the
>> >>> > > > > > >>> > >> > >>>>>>>>> configurable enum value for
>> layout. We
>> >>> > > should
>> >>> > > > be
>> >>> > > > > > >>> able to
>> >>> > > > > > >>> > >> > inject
>> >>> > > > > > >>> > >> > >>>>>>>> any
>> >>> > > > > > >>> > >> > >>>>>>>>> layout
>> >>> > > > > > >>> > >> > >>>>>>>>> in the current design.
>> >>> > > > > > >>> > >> > >>>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>> Working on resolving all remaining open
>> >>> > comments
>> >>> > > > on
>> >>> > > > > > the
>> >>> > > > > > >>> spec
>> >>> > > > > > >>> > >> this
>> >>> > > > > > >>> > >> > >>>>>>>> week.
>> >>> > > > > > >>> > >> > >>>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>> Best
>> >>> > > > > > >>> > >> > >>>>>>>>> Prateek
>> >>> > > > > > >>> > >> > >>>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve
>> >>> > Loughran <
>> >>> > > > > > >>> > >> > >>>>>>>> stevel@cloudera.com>
>> >>> > > > > > >>> > >> > >>>>>>>>> wrote:
>> >>> > > > > > >>> > >> > >>>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah
>> >>> Kornfield
>> >>> > <
>> >>> > > > > > >>> > >> > >>>>>>>> emkornfield@gmail.com>
>> >>> > > > > > >>> > >> > >>>>>>>>>> wrote:
>> >>> > > > > > >>> > >> > >>>>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>>>> It looks like the actual issue
>> >>> described for
>> >>> > > ORC
>> >>> > > > > in
>> >>> > > > > > >>> the
>> >>> > > > > > >>> > >> paper
>> >>> > > > > > >>> > >> > >>>>>>>> is that
>> >>> > > > > > >>> > >> > >>>>>>>>> it
>> >>> > > > > > >>> > >> > >>>>>>>>>>> has multiple sub-encodings in a
>> batch.
>> >>> This
>> >>> > > is
>> >>> > > > > > >>> different
>> >>> > > > > > >>> > >> then
>> >>> > > > > > >>> > >> > >>>>>>>> the
>> >>> > > > > > >>> > >> > >>>>>>>>> design
>> >>> > > > > > >>> > >> > >>>>>>>>>>> proposed here where there is still
>> fixed
>> >>> > > > encoding
>> >>> > > > > > per
>> >>> > > > > > >>> page
>> >>> > > > > > >>> > >> in
>> >>> > > > > > >>> > >> > >>>>>>>> parquet.
>> >>> > > > > > >>> > >> > >>>>>>>>>>> Given reasonably sized pages I don't
>> >>> think
>> >>> > > > branch
>> >>> > > > > > >>> > >> > >>>>>>>> misprediction should
>> >>> > > > > > >>> > >> > >>>>>>>>>> be a
>> >>> > > > > > >>> > >> > >>>>>>>>>>> big issue for new encodings. I agree
>> >>> that
>> >>> > we
>> >>> > > > > should
>> >>> > > > > > >>> be
>> >>> > > > > > >>> > >> > >>>>>>>> conservative in
>> >>> > > > > > >>> > >> > >>>>>>>>>>> general for adding new encodings.
>> >>> > > > > > >>> > >> > >>>>>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>>> +1
>> >>> > > > > > >>> > >> > >>>>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>>
>> >>> > > > > > >>> > >> > >>>>>>>
>> >>> > > > > > >>> > >> > >
>> >>> > > > > > >>> > >> >
>> >>> > > > > > >>> > >> >
>> >>> > > > > > >>> > >> >
>> >>> > > > > > >>> > >>
>> >>> > > > > > >>> > >
>> >>> > > > > > >>> >
>> >>> > > > > > >>>
>> >>> > > > > > >>
>> >>> > > > > >
>> >>> > > > >
>> >>> > > >
>> >>> > >
>> >>> >
>> >>>
>> >>
>>
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Russell Spitzer <ru...@gmail.com>.
I thought we had agreed it would be merged, and features like this would be
feature flag gated. So a V-Old writer could always opt in to using this
encoding but that must be a conscious effort if you are not using VNext of
Parquet. If you are on V-Next the writer can choose to use ALP whenever it
likes without user intervention.
On Fri, Jul 31, 2026 at 2:20 AM Divjot Arora via dev <de...@parquet.apache.org>
wrote:
> Yes, that’s my understanding as well. However, as you noted, the versioning
> discussion is still in early stages and hasn’t prescribed concrete
> mechanisms yet. If ALP goes into parquet-format 2.14.0, there’s no
> indication that it’s in preview or that writers shouldn’t use it by default
> yet. Given that we’ve held off on merging other incompatible changes (e.g
> removal of path_in_schema) until the versioning discussion closes, my
> preference would be to leave ALP un-merged for the time being as well.
>
> — Div
>
> On Fri, Jul 31, 2026 at 08:59 Andrew Lamb <an...@gmail.com> wrote:
>
> > After re-reading Julien's Versioning Proposal, my understanding is that
> if
> > we adopt that system, we will somehow mark ALP in "Preview" in
> > parquet-format for release Version 3.
> >
> > The versioning document doesn't seem to proscribe the particular method
> > for annotating "preview" features -- I think clearly marking them as such
> > in parquet.thrift as well as reflecting that on the
> > https://parquet.apache.org/docs/file-format/versions/ page somehow would
> > be adequate.
> >
> > Andrew
> >
> > On Thu, Jul 30, 2026 at 11:50 AM Andrew Lamb <an...@gmail.com>
> > wrote:
> >
> >> In my mind ALP is in the same position as other encodings that were
> added
> >> recently[1] (such as BYTE_STREAM_SPLIT for non floating point types)
> >>
> >> That is to say it will be available in the latest open source writers
> and
> >> readers, but realistically not widely used in the ecosystem where
> >> compatibility is a concern (a large part)
> >>
> >> I view Julian's work[2][3] to gather consensus on how to name
> >> combinations of features (e.g. "versions") as the mechanism by which we
> >> will accelerate the adoption of this change (along with other forward
> >> incompatible changes) across the ecosystem.
> >>
> >> > Will we exclude this change when cutting parquet-format v2.14.0?
> >>
> >> No I don't think so -- I would expect it to go out in the next
> >> parquet-format version as do other changes in the spec.
> >>
> >> Andrew
> >>
> >> [1]: https://parquet.apache.org/docs/file-format/versions/
> >> [2]: https://lists.apache.org/thread/xmnj8h0h8ozmrgox5tydhhs7yz7q2hss
> >> [3]: https://lists.apache.org/thread/qlf8lg90gqq41lllyy9mk2f0skqvftqj
> >>
> >>
> >> On Thu, Jul 30, 2026 at 3:51 AM Divjot Arora via dev <
> >> dev@parquet.apache.org> wrote:
> >>
> >>> Hi Andrew,
> >>>
> >>> How does this interact with the ongoing versioning discussion? Given
> that
> >>> new encodings are not forward compatible, I would assume ALP is
> included
> >>> in
> >>> Parquet V<next> (V3?), but it's not clear to me how that works if it's
> >>> merged to the parquet-format master branch now. Will we exclude this
> >>> change
> >>> when cutting parquet-format v2.14.0?
> >>>
> >>> -- Div
> >>>
> >>> On Thu, Jul 30, 2026 at 8:51 AM Andrew Lamb <an...@gmail.com>
> >>> wrote:
> >>>
> >>> > Hello,
> >>> >
> >>> > As an update, thanks to the epic work from many contributors,
> >>> especially
> >>> > Prateek, I just merged the parquet-format change for ALP[1]. We plan
> to
> >>> > make additional editorial The remaining items I know of are merging
> the
> >>> > examples from parquet-testing[2], and then of course the various
> >>> > implementations
> >>> >
> >>> > Thanks again everyone. This is a great step forward
> >>> > Andrew
> >>> >
> >>> > [1]: https://github.com/apache/parquet-format/pull/557
> >>> > [2]: https://github.com/apache/parquet-testing/pull/100
> >>> >
> >>> > On Tue, Jul 21, 2026 at 4:39 PM Julien Le Dem <ju...@apache.org>
> >>> wrote:
> >>> >
> >>> > > It was also in my spam folder... :(
> >>> > > I've seen a bunch of apache list emails go to my spam recently.
> >>> > >
> >>> > > On Tue, Jul 21, 2026 at 10:58 AM Russell Spitzer <
> >>> > > russell.spitzer@gmail.com>
> >>> > > wrote:
> >>> > >
> >>> > > > Yes, my hope was that my vote would kick it out of spam but we
> >>> > definitely
> >>> > > > had two other votes that also went to spam for me.
> >>> > > >
> >>> > > > For anyone looking the thread is here
> >>> > > > https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt
> >>> > > >
> >>> > > > On Tue, Jul 21, 2026 at 10:57 AM Micah Kornfield <
> >>> > emkornfield@gmail.com>
> >>> > > > wrote:
> >>> > > >
> >>> > > > > I've seen other's votes not go to spam, so hopefully people
> will
> >>> see
> >>> > it
> >>> > > > in
> >>> > > > > there inboxes (or see this thread and look in there spam).
> >>> > > > >
> >>> > > > > On Tue, Jul 21, 2026 at 10:52 AM PRATEEK GAUR <
> >>> prtkgaur@gmail.com>
> >>> > > > wrote:
> >>> > > > >
> >>> > > > > > Okay,
> >>> > > > > >
> >>> > > > > > I sent out the voting email yesterday and so far I've heard
> >>> from 3
> >>> > > > people
> >>> > > > > > that it ended up in their "spam folder". Dev VPN to be blamed
> >>> for
> >>> > > it?.
> >>> > > > > >
> >>> > > > > > What's the protocol here?
> >>> > > > > > I let people hunt for the email in their spam folder or send
> >>> out
> >>> > > > another
> >>> > > > > > one?
> >>> > > > > >
> >>> > > > > > Best
> >>> > > > > > Prateek
> >>> > > > > >
> >>> > > > > > On Sun, Jul 19, 2026 at 10:47 PM PRATEEK GAUR <
> >>> prtkgaur@gmail.com>
> >>> > > > > wrote:
> >>> > > > > >
> >>> > > > > > > Hi team,
> >>> > > > > > >
> >>> > > > > > > Vinoo has added the tests which were requested in the last
> >>> round
> >>> > of
> >>> > > > > > review.
> >>> > > > > > > In his own words.
> >>> > > > > > >
> >>> > > > > > > "
> >>> > > > > > > The extreme value tests Micah wanted are done and pushed. I
> >>> added
> >>> > > > > > coverage
> >>> > > > > > > for values that need the full FOR bit width after the frame
> >>> of
> >>> > > > > reference
> >>> > > > > > is
> >>> > > > > > > applied: 64-bit (the case where the signed max minus min
> >>> > > subtraction
> >>> > > > > > > overflows), 63-bit (the non-overflow case), and 32-bit for
> >>> > floats.
> >>> > > > All
> >>> > > > > > > lossless with zero exceptions. I also verified them through
> >>> Arrow
> >>> > > C++
> >>> > > > > ALP
> >>> > > > > > > reader (the local build), so the Java-written extreme
> columns
> >>> > > decode
> >>> > > > > > > correctly in C++, bit-exact against the expected values."
> >>> > > > > > >
> >>> > > > > > > State of the PR (#3397):
> >>> > > > > > >
> >>> > > > > > > - Merges cleanly with master (I pulled master in and
> >>> resolved
> >>> > > the
> >>> > > > > > > conflicts).
> >>> > > > > > > - 41 of 44 review threads resolved. The 3 remaining are
> >>> > > > intentional:
> >>> > > > > > > one deferred perf nit, one that stays open until
> >>> > parquet-format
> >>> > > > > ships
> >>> > > > > > ALP,
> >>> > > > > > >
> >>> > > > > > > "
> >>> > > > > > >
> >>> > > > > > > With this both the
> >>> > > > > > > 1) C++ PR :
> >>> https://github.com/apache/arrow/pull/48345/changes
> >>> > > > > > > 2) Java PR :
> >>> https://github.com/apache/parquet-java/pull/3397
> >>> > > > > > >
> >>> > > > > > > Are cross language tested and range of test scenarios.
> >>> > > > > > >
> >>> > > > > > > Best
> >>> > > > > > > Prateek
> >>> > > > > > >
> >>> > > > > > >
> >>> > > > > > > On Mon, Jul 13, 2026 at 8:38 AM PRATEEK GAUR <
> >>> prtkgaur@gmail.com
> >>> > >
> >>> > > > > wrote:
> >>> > > > > > >
> >>> > > > > > >> Thanks Matt,
> >>> > > > > > >>
> >>> > > > > > >> I'll be on it this week.
> >>> > > > > > >>
> >>> > > > > > >> Best
> >>> > > > > > >> Prateek
> >>> > > > > > >>
> >>> > > > > > >> On Mon, Jul 13, 2026 at 8:36 AM Matt Topol <
> >>> > > zotthewizard@gmail.com>
> >>> > > > > > >> wrote:
> >>> > > > > > >>
> >>> > > > > > >>> There is also the Go implementation submitted by Arnav (
> >>> > > > > > >>> https://github.com/apache/arrow-go/pull/704) which is
> >>> waiting
> >>> > > for
> >>> > > > > > >>> updates.
> >>> > > > > > >>>
> >>> > > > > > >>> Just wanted to make sure it didn't get lost here.
> >>> > > > > > >>>
> >>> > > > > > >>> --Matt
> >>> > > > > > >>>
> >>> > > > > > >>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK GAUR <
> >>> > > prtkgaur@gmail.com>
> >>> > > > > > >>> wrote:
> >>> > > > > > >>>
> >>> > > > > > >>> > Hi all,
> >>> > > > > > >>> >
> >>> > > > > > >>> > I'd like to share a status update on the ALP encoding
> >>> effort
> >>> > > and
> >>> > > > > get
> >>> > > > > > a
> >>> > > > > > >>> feel
> >>> > > > > > >>> > from the community before starting a formal vote.
> >>> > > > > > >>> >
> >>> > > > > > >>> > We now have the specification plus implementations in
> >>> both
> >>> > > > > languages:
> >>> > > > > > >>> >
> >>> > > > > > >>> > - Spec: apache/parquet-format#557 — GH-533
> >>> Add
> >>> > ALP
> >>> > > > > > encoding
> >>> > > > > > >>> > specification
> >>> > > > > > >>> > - C++ (Arrow): apache/arrow#48345 — GH-48701
> >>> > > [C++][Parquet]
> >>> > > > > Add
> >>> > > > > > >>> ALPpd
> >>> > > > > > >>> > encoding
> >>> > > > > > >>> > - Java: apache/parquet-java#3397 — Parquet
> >>> Java
> >>> > ALP
> >>> > > > > > >>> > Implementation (by Vinoo)
> >>> > > > > > >>> >
> >>> > > > > > >>> > Current state:
> >>> > > > > > >>> >
> >>> > > > > > >>> > - The C++ implementation has been through 3-4
> rounds
> >>> of
> >>> > > > review.
> >>> > > > > > >>> > - The Java implementation has been tracking well
> and
> >>> has
> >>> > > had
> >>> > > > > > >>> initial
> >>> > > > > > >>> > reviews from contributors, with no major
> remaining
> >>> > > issues.
> >>> > > > > > >>> > - Cross-language compatibility tests are passing:
> the
> >>> > Arrow
> >>> > > > C++
> >>> > > > > > >>> decoder
> >>> > > > > > >>> > reads Java-written data bit-exactly across ~1.56M
> >>> > values
> >>> > > > and
> >>> > > > > 18
> >>> > > > > > >>> > fixtures, covering V1 and V2 pages, multiple
> vector
> >>> > > sizes,
> >>> > > > > and
> >>> > > > > > >>> > several
> >>> > > > > > >>> > real datasets — zero mismatches.
> >>> > > > > > >>> >
> >>> > > > > > >>> > As raised in the java-pr we will be adding coverage
> for
> >>> > > extreme
> >>> > > > > > values
> >>> > > > > > >>> > (those requiring
> >>> > > > > > >>> > 63-64 bits after FOR is applied) before we close a
> vote.
> >>> > We'll
> >>> > > > aim
> >>> > > > > > to
> >>> > > > > > >>> get
> >>> > > > > > >>> > that
> >>> > > > > > >>> > done in parallel.
> >>> > > > > > >>> >
> >>> > > > > > >>> > Plan: unless there are objections, I intend to start a
> >>> vote
> >>> > at
> >>> > > > the
> >>> > > > > > end
> >>> > > > > > >>> of
> >>> > > > > > >>> > this week/early
> >>> > > > > > >>> > next week.
> >>> > > > > > >>> >
> >>> > > > > > >>> > Thanks
> >>> > > > > > >>> > Prateek
> >>> > > > > > >>> >
> >>> > > > > > >>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <
> >>> > > prtkgaur@gmail.com>
> >>> > > > > > >>> wrote:
> >>> > > > > > >>> >
> >>> > > > > > >>> > > Hi Team,
> >>> > > > > > >>> > >
> >>> > > > > > >>> > > Just wanted to provide some updates on ALP.
> >>> > > > > > >>> > > Micah and I have done 3-4 rounds of review for the
> c++
> >>> PR :
> >>> > > > > > >>> > > https://github.com/apache/arrow/pull/48345
> >>> > > > > > >>> > >
> >>> > > > > > >>> > > For the Java implementation Vinoo has been working on
> >>> > > following
> >>> > > > > PR
> >>> > > > > > :
> >>> > > > > > >>> > > https://github.com/apache/parquet-java/pull/3397
> >>> > > > > > >>> > >
> >>> > > > > > >>> > > Best
> >>> > > > > > >>> > > Prateek
> >>> > > > > > >>> > >
> >>> > > > > > >>> > > On Tue, May 5, 2026 at 1:50 PM Micah Kornfield <
> >>> > > > > > >>> emkornfield@gmail.com>
> >>> > > > > > >>> > > wrote:
> >>> > > > > > >>> > >
> >>> > > > > > >>> > >> Hi Antoine,
> >>> > > > > > >>> > >>
> >>> > > > > > >>> > >> > Apologies if the question was already asked, but
> >>> should
> >>> > we
> >>> > > > > care
> >>> > > > > > >>> about
> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient
> >>> than
> >>> > > > > FLOAT16
> >>> > > > > > +
> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> >>> > > > > > >>> > >>
> >>> > > > > > >>> > >> It was. We thought we could defer it for the
> >>> following
> >>> > > > reasons:
> >>> > > > > > >>> > >> 1. It's not clear there are a lot of easy reference
> >>> > > datasets
> >>> > > > to
> >>> > > > > > >>> test
> >>> > > > > > >>> > its
> >>> > > > > > >>> > >> effectiveness. It does look like there might be one
> >>> or
> >>> > two
> >>> > > on
> >>> > > > > > >>> > huggingface
> >>> > > > > > >>> > >> (e.g.
> https://huggingface.co/datasets/kikitora/curdie
> >>> ).
> >>> > > > > > >>> > >> 2. It seemed likely that float 16 was more likely
> >>> used
> >>> > for
> >>> > > > > values
> >>> > > > > > >>> that
> >>> > > > > > >>> > >> were less likely to reduce to decimal values.
> >>> > > > > > >>> > >> 3. It could be added as an extension later if
> needed.
> >>> > > > > > >>> > >>
> >>> > > > > > >>> > >> Cheers,
> >>> > > > > > >>> > >> Micah
> >>> > > > > > >>> > >>
> >>> > > > > > >>> > >>
> >>> > > > > > >>> > >> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <
> >>> > > > > antoine@python.org
> >>> > > > > > >
> >>> > > > > > >>> > wrote:
> >>> > > > > > >>> > >>
> >>> > > > > > >>> > >> >
> >>> > > > > > >>> > >> > Hello,
> >>> > > > > > >>> > >> >
> >>> > > > > > >>> > >> > Apologies if the question was already asked, but
> >>> should
> >>> > we
> >>> > > > > care
> >>> > > > > > >>> about
> >>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient
> >>> than
> >>> > > > > FLOAT16
> >>> > > > > > +
> >>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> >>> > > > > > >>> > >> >
> >>> > > > > > >>> > >> > Regards
> >>> > > > > > >>> > >> >
> >>> > > > > > >>> > >> > Antoine.
> >>> > > > > > >>> > >> >
> >>> > > > > > >>> > >> >
> >>> > > > > > >>> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
> >>> > > > > > >>> > >> > > Thanks Andrew and Micah for review feedback on
> >>> the two
> >>> > > > PR's
> >>> > > > > > >>> > >> > > 1) (c++ arrow repo)
> >>> > > > > > >>> > >> https://github.com/apache/arrow/pull/48345/changes
> >>> > > > > > >>> > >> > > 2) (parquet-format repo)
> >>> > > > > > >>> > >> > https://github.com/apache/parquet-format/pull/557
> >>> > > > > > >>> > >> > >
> >>> > > > > > >>> > >> > > I have addressed all (unless I missed something)
> >>> > > comments
> >>> > > > on
> >>> > > > > > >>> the two
> >>> > > > > > >>> > >> > PR's.
> >>> > > > > > >>> > >> > >
> >>> > > > > > >>> > >> > > Best
> >>> > > > > > >>> > >> > > Prateek
> >>> > > > > > >>> > >> > >
> >>> > > > > > >>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <
> >>> > > > > > >>> prtkgaur@gmail.com>
> >>> > > > > > >>> > >> wrote:
> >>> > > > > > >>> > >> > >
> >>> > > > > > >>> > >> > >> Thanks Andrew and Micah.
> >>> > > > > > >>> > >> > >>
> >>> > > > > > >>> > >> > >> `fair amount of feedback on at least the
> >>> > > implementations`
> >>> > > > > > >>> > >> > >> For the c++ I have already started addressing
> the
> >>> > > > > feedback, I
> >>> > > > > > >>> > should
> >>> > > > > > >>> > >> be
> >>> > > > > > >>> > >> > >> done with that Monday/Tuesday.
> >>> > > > > > >>> > >> > >> I think Vinoo too has been making good progress
> >>> on
> >>> > the
> >>> > > > Java
> >>> > > > > > >>> > >> > implementation.
> >>> > > > > > >>> > >> > >>
> >>> > > > > > >>> > >> > >> Best
> >>> > > > > > >>> > >> > >> Prateek
> >>> > > > > > >>> > >> > >>
> >>> > > > > > >>> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <
> >>> > > > > > >>> > andrewlamb11@gmail.com
> >>> > > > > > >>> > >> >
> >>> > > > > > >>> > >> > >> wrote:
> >>> > > > > > >>> > >> > >>
> >>> > > > > > >>> > >> > >>> Got it. Thank you for the clarification -- I
> >>> will
> >>> > try
> >>> > > > and
> >>> > > > > > look
> >>> > > > > > >>> > into
> >>> > > > > > >>> > >> the
> >>> > > > > > >>> > >> > >>> spec and the Rust implementation[1] in this
> next
> >>> > week
> >>> > > > > > >>> > >> > >>>
> >>> > > > > > >>> > >> > >>> [1]:
> >>> https://github.com/apache/arrow-rs/pull/9372
> >>> > > > > > >>> > >> > >>>
> >>> > > > > > >>> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah
> >>> Kornfield <
> >>> > > > > > >>> > >> > emkornfield@gmail.com>
> >>> > > > > > >>> > >> > >>> wrote:
> >>> > > > > > >>> > >> > >>>
> >>> > > > > > >>> > >> > >>>> Hi Andrew,
> >>> > > > > > >>> > >> > >>>> I think there is a fair amount of feedback on
> >>> at
> >>> > > least
> >>> > > > > the
> >>> > > > > > >>> > >> > >>>> implementations, typically I think we've
> waited
> >>> > till
> >>> > > > they
> >>> > > > > > are
> >>> > > > > > >>> > >> close to
> >>> > > > > > >>> > >> > >>>> mergeable before a final vote. Otherwise I
> >>> agree
> >>> > we
> >>> > > > are
> >>> > > > > > very
> >>> > > > > > >>> > >> close.
> >>> > > > > > >>> > >> > >>>>
> >>> > > > > > >>> > >> > >>>> -Micah
> >>> > > > > > >>> > >> > >>>>
> >>> > > > > > >>> > >> > >>>> On Saturday, April 25, 2026, Andrew Lamb <
> >>> > > > > > >>> andrewlamb11@gmail.com
> >>> > > > > > >>> > >
> >>> > > > > > >>> > >> > wrote:
> >>> > > > > > >>> > >> > >>>>
> >>> > > > > > >>> > >> > >>>>> Thanks Prateek,
> >>> > > > > > >>> > >> > >>>>>
> >>> > > > > > >>> > >> > >>>>> I think from this content it looks to me
> like
> >>> we
> >>> > are
> >>> > > > > ready
> >>> > > > > > >>> to
> >>> > > > > > >>> > >> start a
> >>> > > > > > >>> > >> > >>>>> vote to explicitly accept ALP into Parquet
> >>> > > > > > >>> > >> > >>>>>
> >>> > > > > > >>> > >> > >>>>> Does anyone know of a reason we should
> >>> postpone it
> >>> > > for
> >>> > > > > > >>> longer?
> >>> > > > > > >>> > >> > >>>>> Perhaps someone needs some more time to
> >>> review?
> >>> > > > > > >>> > >> > >>>>>
> >>> > > > > > >>> > >> > >>>>> Andrew
> >>> > > > > > >>> > >> > >>>>>
> >>> > > > > > >>> > >> > >>>>>
> >>> > > > > > >>> > >> > >>>>>
> >>> > > > > > >>> > >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK
> GAUR <
> >>> > > > > > >>> > prtkgaur@gmail.com>
> >>> > > > > > >>> > >> > >>>>> wrote:
> >>> > > > > > >>> > >> > >>>>>
> >>> > > > > > >>> > >> > >>>>>> Hi team,
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>> Hope everyone is doing well. I got a chance
> >>> to
> >>> > work
> >>> > > > > > >>> through all
> >>> > > > > > >>> > >> the
> >>> > > > > > >>> > >> > >>>>>> remaining feedback and update the spec doc.
> >>> Here
> >>> > > are
> >>> > > > > the
> >>> > > > > > >>> new
> >>> > > > > > >>> > >> > artifacts
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>> 1) Spec document :
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> >
> >>> > > > > > >>> > >>
> >>> > > > > > >>> >
> >>> > > > > > >>>
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>> 2) Spec document in parquet format repo :
> >>> > > > > > >>> > >> > >>>>>>
> >>> > https://github.com/apache/parquet-format/pull/557
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>> 3) Alp implementation in arrow c++ repo :
> >>> > > > > > >>> > >> > >>>>>>
> >>> > https://github.com/apache/arrow/pull/48345/changes
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>> 4) Alp implementation in parquet-java repo
> :
> >>> Work
> >>> > > for
> >>> > > > > > >>> Vinoo and
> >>> > > > > > >>> > >> > Julien
> >>> > > > > > >>> > >> > >>>>>>
> >>> > https://github.com/apache/parquet-java/pull/3397
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>> 5) PR with test and benchmarking artifacts
> in
> >>> > > > > > >>> parquet-testing
> >>> > > > > > >>> > >> repo :
> >>> > > > > > >>> > >> > >>>>>>
> >>> > https://github.com/apache/parquet-testing/pull/100
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>> And
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>> - Go : Arnav just submitted an in
> >>> progress
> >>> > > > > > >>> implementation
> >>> > > > > > >>> > in
> >>> > > > > > >>> > >> Go.
> >>> > > > > > >>> > >> > >>>>>>
> >>> https://github.com/apache/arrow-go/pull/704
> >>> > (I
> >>> > > > > > haven't
> >>> > > > > > >>> > >> started
> >>> > > > > > >>> > >> > >>>>>> looking at it yet)
> >>> > > > > > >>> > >> > >>>>>> - Rust : I remember Andrew mentioned
> that
> >>> > this
> >>> > > > work
> >>> > > > > > is
> >>> > > > > > >>> also
> >>> > > > > > >>> > >> in
> >>> > > > > > >>> > >> > >>>>>> progress (So 4 languages!)
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>> *Arrow C++ implementation *
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>> The PR is out and was also used by Antoine
> to
> >>> > > report
> >>> > > > > the
> >>> > > > > > >>> > numbers
> >>> > > > > > >>> > >> as
> >>> > > > > > >>> > >> > >>>>>> reported here. Micah and Konstantin have
> >>> given 1
> >>> > > > round
> >>> > > > > of
> >>> > > > > > >>> > >> feedback
> >>> > > > > > >>> > >> > >>>>>> and I'm addressing them today. Please note
> >>> that
> >>> > the
> >>> > > > > > default
> >>> > > > > > >>> > >> > >>>>>> optimization flag for compiling is O2 and
> not
> >>> > Q3. I
> >>> > > > got
> >>> > > > > > >>> around
> >>> > > > > > >>> > >> 70%
> >>> > > > > > >>> > >> > >>>>>> performance improvement in the decoding
> speed
> >>> > when
> >>> > > > > using
> >>> > > > > > >>> the O3
> >>> > > > > > >>> > >> > flag.
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>> *Parqet-MR Java implementation (working
> with
> >>> > Vinoo
> >>> > > > and
> >>> > > > > > >>> Julien)
> >>> > > > > > >>> > >> and
> >>> > > > > > >>> > >> > **Cross
> >>> > > > > > >>> > >> > >>>>>> Language testing*
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>> Let me know if you have any questions
> or
> >>> > > > feedback.
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>> Now pasting some performance numbers
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>> Table 1: C++ ALP Double Decode — Spotify
> >>> > Columns
> >>> > > > > > >>> (Graviton
> >>> > > > > > >>> > 3,
> >>> > > > > > >>> > >> ARM
> >>> > > > > > >>> > >> > >>>>>> Neoverse V1)
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>>
> >>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>> │ Column │ -O2 (MB/s) │ -O3
> >>> > > (MB/s) │
> >>> > > > > > >>> Speedup │
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>>
> >>> ├──────────────────┼──────────────┼──────────────┼─────────┤
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>> │ valence │ 3,155 │
> >>> 5,523
> >>> > > │
> >>> > > > > > >>> 1.75x │
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>> │ danceability │ 3,233 │
> >>> 5,685
> >>> > > │
> >>> > > > > > >>> 1.76x │
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>> │ energy │ 3,197 │
> >>> 5,652
> >>> > > │
> >>> > > > > > >>> 1.77x │
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>> │ loudness │ 3,186 │
> >>> 5,473
> >>> > > │
> >>> > > > > > >>> 1.72x │
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>>
> >>> └──────────────────┴──────────────┴──────────────┴─────────┘
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK
> GAUR
> >>> <
> >>> > > > > > >>> > prtkgaur@gmail.com
> >>> > > > > > >>> > >> >
> >>> > > > > > >>> > >> > >>>>>> wrote:
> >>> > > > > > >>> > >> > >>>>>>
> >>> > > > > > >>> > >> > >>>>>>> @Micah Kornfield <em...@gmail.com>
> :
> >>> Got
> >>> > > it.
> >>> > > > > > >>> > >> > >>>>>>>
> >>> > > > > > >>> > >> > >>>>>>> @Andrew Lamb <an...@gmail.com>
> >>> > > > > > >>> > >> > >>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>> Do you think it would be good to start
> >>> moving
> >>> > the
> >>> > > > > spec
> >>> > > > > > >>> > >> development
> >>> > > > > > >>> > >> > >>>>>>>> into
> >>> > > > > > >>> > >> > >>>>>>>> markdown format, in preparation for
> >>> finalizing
> >>> > > it?
> >>> > > > > > >>> > >> > >>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>
> >>> > > > > > >>> > >> > >>>>>>> Yes I'll update the numbers for some of
> the
> >>> > > > examples I
> >>> > > > > > >>> have in
> >>> > > > > > >>> > >> the
> >>> > > > > > >>> > >> > >>>>>>> spec based
> >>> > > > > > >>> > >> > >>>>>>> on the updated header size. Then we should
> >>> be
> >>> > good
> >>> > > > to
> >>> > > > > go
> >>> > > > > > >>> for
> >>> > > > > > >>> > the
> >>> > > > > > >>> > >> > >>>>>>> markdown format.
> >>> > > > > > >>> > >> > >>>>>>>
> >>> > > > > > >>> > >> > >>>>>>> Thanks everyone!
> >>> > > > > > >>> > >> > >>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>> Andrew
> >>> > > > > > >>> > >> > >>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK
> >>> GAUR <
> >>> > > > > > >>> > >> prtkgaur@gmail.com>
> >>> > > > > > >>> > >> > >>>>>>>> wrote:
> >>> > > > > > >>> > >> > >>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>> Hi team,
> >>> > > > > > >>> > >> > >>>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>> 1) Andrew
> >>> > > > > > >>> > >> > >>>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for working on test files.
> >>> My PR
> >>> > > did
> >>> > > > > add
> >>> > > > > > >>> all
> >>> > > > > > >>> > the
> >>> > > > > > >>> > >> > test
> >>> > > > > > >>> > >> > >>>>>>>> files I
> >>> > > > > > >>> > >> > >>>>>>>>> used to benchmark on datasets. Maybe
> >>> we
> >>> > can
> >>> > > > club
> >>> > > > > > it
> >>> > > > > > >>> > >> together.
> >>> > > > > > >>> > >> > >>>>>>>> WIll also
> >>> > > > > > >>> > >> > >>>>>>>>> aid
> >>> > > > > > >>> > >> > >>>>>>>>> cross language testing
> >>> > > > > > >>> > >> > >>>>>>>>> - Kosta Tarasov working on Rust
> >>> > > > implementation.
> >>> > > > > > >>> This is
> >>> > > > > > >>> > >> > great.
> >>> > > > > > >>> > >> > >>>>>>>> Thanks
> >>> > > > > > >>> > >> > >>>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>> 2) Antoine
> >>> > > > > > >>> > >> > >>>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks a lot for reporting the
> >>> numbers
> >>> > on
> >>> > > > AMD.
> >>> > > > > > >>> Looks
> >>> > > > > > >>> > >> like
> >>> > > > > > >>> > >> > you
> >>> > > > > > >>> > >> > >>>>>>>> are
> >>> > > > > > >>> > >> > >>>>>>>>> getting 8X the decoding performance
> of
> >>> > BSS.
> >>> > > > This
> >>> > > > > > is
> >>> > > > > > >>> > >> > amazing!!.
> >>> > > > > > >>> > >> > >>>>>>>>> - Thanks for acknowledging the
> >>> sampling
> >>> > > > design.
> >>> > > > > > >>> > >> > >>>>>>>>> - I agree with you on Fastlanes. In
> >>> some
> >>> > > crude
> >>> > > > > > >>> > >> experiments I
> >>> > > > > > >>> > >> > >>>>>>>> didn't get
> >>> > > > > > >>> > >> > >>>>>>>>> a good perf benefit from it on
> >>> Graviton3
> >>> > > (but
> >>> > > > > > maybe
> >>> > > > > > >>> > there
> >>> > > > > > >>> > >> was
> >>> > > > > > >>> > >> > >>>>>>>> something
> >>> > > > > > >>> > >> > >>>>>>>>> wrong with my implementation).
> >>> > > > > > >>> > >> > >>>>>>>>> - Locking the 16bit exception
> >>> encoding for
> >>> > > the
> >>> > > > > > spec
> >>> > > > > > >>> in
> >>> > > > > > >>> > >> this
> >>> > > > > > >>> > >> > >>>>>>>> case.
> >>> > > > > > >>> > >> > >>>>>>>>> - Awesome I think we have solved for
> >>> all
> >>> > > open
> >>> > > > > > >>> questions
> >>> > > > > > >>> > >> minus
> >>> > > > > > >>> > >> > >>>>>>>> the
> >>> > > > > > >>> > >> > >>>>>>>>> version byte :). (will get back on
> >>> this
> >>> > > soon)
> >>> > > > > > >>> > >> > >>>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>> 3) Micah
> >>> > > > > > >>> > >> > >>>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>> - FastLanes : The current spec does
> >>> allow
> >>> > > for
> >>> > > > > > using
> >>> > > > > > >>> > >> FastLane
> >>> > > > > > >>> > >> > >>>>>>>> with the
> >>> > > > > > >>> > >> > >>>>>>>>> configurable enum value for layout.
> We
> >>> > > should
> >>> > > > be
> >>> > > > > > >>> able to
> >>> > > > > > >>> > >> > inject
> >>> > > > > > >>> > >> > >>>>>>>> any
> >>> > > > > > >>> > >> > >>>>>>>>> layout
> >>> > > > > > >>> > >> > >>>>>>>>> in the current design.
> >>> > > > > > >>> > >> > >>>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>> Working on resolving all remaining open
> >>> > comments
> >>> > > > on
> >>> > > > > > the
> >>> > > > > > >>> spec
> >>> > > > > > >>> > >> this
> >>> > > > > > >>> > >> > >>>>>>>> week.
> >>> > > > > > >>> > >> > >>>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>> Best
> >>> > > > > > >>> > >> > >>>>>>>>> Prateek
> >>> > > > > > >>> > >> > >>>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve
> >>> > Loughran <
> >>> > > > > > >>> > >> > >>>>>>>> stevel@cloudera.com>
> >>> > > > > > >>> > >> > >>>>>>>>> wrote:
> >>> > > > > > >>> > >> > >>>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah
> >>> Kornfield
> >>> > <
> >>> > > > > > >>> > >> > >>>>>>>> emkornfield@gmail.com>
> >>> > > > > > >>> > >> > >>>>>>>>>> wrote:
> >>> > > > > > >>> > >> > >>>>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>>>> It looks like the actual issue
> >>> described for
> >>> > > ORC
> >>> > > > > in
> >>> > > > > > >>> the
> >>> > > > > > >>> > >> paper
> >>> > > > > > >>> > >> > >>>>>>>> is that
> >>> > > > > > >>> > >> > >>>>>>>>> it
> >>> > > > > > >>> > >> > >>>>>>>>>>> has multiple sub-encodings in a batch.
> >>> This
> >>> > > is
> >>> > > > > > >>> different
> >>> > > > > > >>> > >> then
> >>> > > > > > >>> > >> > >>>>>>>> the
> >>> > > > > > >>> > >> > >>>>>>>>> design
> >>> > > > > > >>> > >> > >>>>>>>>>>> proposed here where there is still
> fixed
> >>> > > > encoding
> >>> > > > > > per
> >>> > > > > > >>> page
> >>> > > > > > >>> > >> in
> >>> > > > > > >>> > >> > >>>>>>>> parquet.
> >>> > > > > > >>> > >> > >>>>>>>>>>> Given reasonably sized pages I don't
> >>> think
> >>> > > > branch
> >>> > > > > > >>> > >> > >>>>>>>> misprediction should
> >>> > > > > > >>> > >> > >>>>>>>>>> be a
> >>> > > > > > >>> > >> > >>>>>>>>>>> big issue for new encodings. I agree
> >>> that
> >>> > we
> >>> > > > > should
> >>> > > > > > >>> be
> >>> > > > > > >>> > >> > >>>>>>>> conservative in
> >>> > > > > > >>> > >> > >>>>>>>>>>> general for adding new encodings.
> >>> > > > > > >>> > >> > >>>>>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>>> +1
> >>> > > > > > >>> > >> > >>>>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>>
> >>> > > > > > >>> > >> > >>>>>>>
> >>> > > > > > >>> > >> > >
> >>> > > > > > >>> > >> >
> >>> > > > > > >>> > >> >
> >>> > > > > > >>> > >> >
> >>> > > > > > >>> > >>
> >>> > > > > > >>> > >
> >>> > > > > > >>> >
> >>> > > > > > >>>
> >>> > > > > > >>
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> >>
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Divjot Arora via dev <de...@parquet.apache.org>.
Yes, that’s my understanding as well. However, as you noted, the versioning
discussion is still in early stages and hasn’t prescribed concrete
mechanisms yet. If ALP goes into parquet-format 2.14.0, there’s no
indication that it’s in preview or that writers shouldn’t use it by default
yet. Given that we’ve held off on merging other incompatible changes (e.g
removal of path_in_schema) until the versioning discussion closes, my
preference would be to leave ALP un-merged for the time being as well.
— Div
On Fri, Jul 31, 2026 at 08:59 Andrew Lamb <an...@gmail.com> wrote:
> After re-reading Julien's Versioning Proposal, my understanding is that if
> we adopt that system, we will somehow mark ALP in "Preview" in
> parquet-format for release Version 3.
>
> The versioning document doesn't seem to proscribe the particular method
> for annotating "preview" features -- I think clearly marking them as such
> in parquet.thrift as well as reflecting that on the
> https://parquet.apache.org/docs/file-format/versions/ page somehow would
> be adequate.
>
> Andrew
>
> On Thu, Jul 30, 2026 at 11:50 AM Andrew Lamb <an...@gmail.com>
> wrote:
>
>> In my mind ALP is in the same position as other encodings that were added
>> recently[1] (such as BYTE_STREAM_SPLIT for non floating point types)
>>
>> That is to say it will be available in the latest open source writers and
>> readers, but realistically not widely used in the ecosystem where
>> compatibility is a concern (a large part)
>>
>> I view Julian's work[2][3] to gather consensus on how to name
>> combinations of features (e.g. "versions") as the mechanism by which we
>> will accelerate the adoption of this change (along with other forward
>> incompatible changes) across the ecosystem.
>>
>> > Will we exclude this change when cutting parquet-format v2.14.0?
>>
>> No I don't think so -- I would expect it to go out in the next
>> parquet-format version as do other changes in the spec.
>>
>> Andrew
>>
>> [1]: https://parquet.apache.org/docs/file-format/versions/
>> [2]: https://lists.apache.org/thread/xmnj8h0h8ozmrgox5tydhhs7yz7q2hss
>> [3]: https://lists.apache.org/thread/qlf8lg90gqq41lllyy9mk2f0skqvftqj
>>
>>
>> On Thu, Jul 30, 2026 at 3:51 AM Divjot Arora via dev <
>> dev@parquet.apache.org> wrote:
>>
>>> Hi Andrew,
>>>
>>> How does this interact with the ongoing versioning discussion? Given that
>>> new encodings are not forward compatible, I would assume ALP is included
>>> in
>>> Parquet V<next> (V3?), but it's not clear to me how that works if it's
>>> merged to the parquet-format master branch now. Will we exclude this
>>> change
>>> when cutting parquet-format v2.14.0?
>>>
>>> -- Div
>>>
>>> On Thu, Jul 30, 2026 at 8:51 AM Andrew Lamb <an...@gmail.com>
>>> wrote:
>>>
>>> > Hello,
>>> >
>>> > As an update, thanks to the epic work from many contributors,
>>> especially
>>> > Prateek, I just merged the parquet-format change for ALP[1]. We plan to
>>> > make additional editorial The remaining items I know of are merging the
>>> > examples from parquet-testing[2], and then of course the various
>>> > implementations
>>> >
>>> > Thanks again everyone. This is a great step forward
>>> > Andrew
>>> >
>>> > [1]: https://github.com/apache/parquet-format/pull/557
>>> > [2]: https://github.com/apache/parquet-testing/pull/100
>>> >
>>> > On Tue, Jul 21, 2026 at 4:39 PM Julien Le Dem <ju...@apache.org>
>>> wrote:
>>> >
>>> > > It was also in my spam folder... :(
>>> > > I've seen a bunch of apache list emails go to my spam recently.
>>> > >
>>> > > On Tue, Jul 21, 2026 at 10:58 AM Russell Spitzer <
>>> > > russell.spitzer@gmail.com>
>>> > > wrote:
>>> > >
>>> > > > Yes, my hope was that my vote would kick it out of spam but we
>>> > definitely
>>> > > > had two other votes that also went to spam for me.
>>> > > >
>>> > > > For anyone looking the thread is here
>>> > > > https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt
>>> > > >
>>> > > > On Tue, Jul 21, 2026 at 10:57 AM Micah Kornfield <
>>> > emkornfield@gmail.com>
>>> > > > wrote:
>>> > > >
>>> > > > > I've seen other's votes not go to spam, so hopefully people will
>>> see
>>> > it
>>> > > > in
>>> > > > > there inboxes (or see this thread and look in there spam).
>>> > > > >
>>> > > > > On Tue, Jul 21, 2026 at 10:52 AM PRATEEK GAUR <
>>> prtkgaur@gmail.com>
>>> > > > wrote:
>>> > > > >
>>> > > > > > Okay,
>>> > > > > >
>>> > > > > > I sent out the voting email yesterday and so far I've heard
>>> from 3
>>> > > > people
>>> > > > > > that it ended up in their "spam folder". Dev VPN to be blamed
>>> for
>>> > > it?.
>>> > > > > >
>>> > > > > > What's the protocol here?
>>> > > > > > I let people hunt for the email in their spam folder or send
>>> out
>>> > > > another
>>> > > > > > one?
>>> > > > > >
>>> > > > > > Best
>>> > > > > > Prateek
>>> > > > > >
>>> > > > > > On Sun, Jul 19, 2026 at 10:47 PM PRATEEK GAUR <
>>> prtkgaur@gmail.com>
>>> > > > > wrote:
>>> > > > > >
>>> > > > > > > Hi team,
>>> > > > > > >
>>> > > > > > > Vinoo has added the tests which were requested in the last
>>> round
>>> > of
>>> > > > > > review.
>>> > > > > > > In his own words.
>>> > > > > > >
>>> > > > > > > "
>>> > > > > > > The extreme value tests Micah wanted are done and pushed. I
>>> added
>>> > > > > > coverage
>>> > > > > > > for values that need the full FOR bit width after the frame
>>> of
>>> > > > > reference
>>> > > > > > is
>>> > > > > > > applied: 64-bit (the case where the signed max minus min
>>> > > subtraction
>>> > > > > > > overflows), 63-bit (the non-overflow case), and 32-bit for
>>> > floats.
>>> > > > All
>>> > > > > > > lossless with zero exceptions. I also verified them through
>>> Arrow
>>> > > C++
>>> > > > > ALP
>>> > > > > > > reader (the local build), so the Java-written extreme columns
>>> > > decode
>>> > > > > > > correctly in C++, bit-exact against the expected values."
>>> > > > > > >
>>> > > > > > > State of the PR (#3397):
>>> > > > > > >
>>> > > > > > > - Merges cleanly with master (I pulled master in and
>>> resolved
>>> > > the
>>> > > > > > > conflicts).
>>> > > > > > > - 41 of 44 review threads resolved. The 3 remaining are
>>> > > > intentional:
>>> > > > > > > one deferred perf nit, one that stays open until
>>> > parquet-format
>>> > > > > ships
>>> > > > > > ALP,
>>> > > > > > >
>>> > > > > > > "
>>> > > > > > >
>>> > > > > > > With this both the
>>> > > > > > > 1) C++ PR :
>>> https://github.com/apache/arrow/pull/48345/changes
>>> > > > > > > 2) Java PR :
>>> https://github.com/apache/parquet-java/pull/3397
>>> > > > > > >
>>> > > > > > > Are cross language tested and range of test scenarios.
>>> > > > > > >
>>> > > > > > > Best
>>> > > > > > > Prateek
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > On Mon, Jul 13, 2026 at 8:38 AM PRATEEK GAUR <
>>> prtkgaur@gmail.com
>>> > >
>>> > > > > wrote:
>>> > > > > > >
>>> > > > > > >> Thanks Matt,
>>> > > > > > >>
>>> > > > > > >> I'll be on it this week.
>>> > > > > > >>
>>> > > > > > >> Best
>>> > > > > > >> Prateek
>>> > > > > > >>
>>> > > > > > >> On Mon, Jul 13, 2026 at 8:36 AM Matt Topol <
>>> > > zotthewizard@gmail.com>
>>> > > > > > >> wrote:
>>> > > > > > >>
>>> > > > > > >>> There is also the Go implementation submitted by Arnav (
>>> > > > > > >>> https://github.com/apache/arrow-go/pull/704) which is
>>> waiting
>>> > > for
>>> > > > > > >>> updates.
>>> > > > > > >>>
>>> > > > > > >>> Just wanted to make sure it didn't get lost here.
>>> > > > > > >>>
>>> > > > > > >>> --Matt
>>> > > > > > >>>
>>> > > > > > >>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK GAUR <
>>> > > prtkgaur@gmail.com>
>>> > > > > > >>> wrote:
>>> > > > > > >>>
>>> > > > > > >>> > Hi all,
>>> > > > > > >>> >
>>> > > > > > >>> > I'd like to share a status update on the ALP encoding
>>> effort
>>> > > and
>>> > > > > get
>>> > > > > > a
>>> > > > > > >>> feel
>>> > > > > > >>> > from the community before starting a formal vote.
>>> > > > > > >>> >
>>> > > > > > >>> > We now have the specification plus implementations in
>>> both
>>> > > > > languages:
>>> > > > > > >>> >
>>> > > > > > >>> > - Spec: apache/parquet-format#557 — GH-533
>>> Add
>>> > ALP
>>> > > > > > encoding
>>> > > > > > >>> > specification
>>> > > > > > >>> > - C++ (Arrow): apache/arrow#48345 — GH-48701
>>> > > [C++][Parquet]
>>> > > > > Add
>>> > > > > > >>> ALPpd
>>> > > > > > >>> > encoding
>>> > > > > > >>> > - Java: apache/parquet-java#3397 — Parquet
>>> Java
>>> > ALP
>>> > > > > > >>> > Implementation (by Vinoo)
>>> > > > > > >>> >
>>> > > > > > >>> > Current state:
>>> > > > > > >>> >
>>> > > > > > >>> > - The C++ implementation has been through 3-4 rounds
>>> of
>>> > > > review.
>>> > > > > > >>> > - The Java implementation has been tracking well and
>>> has
>>> > > had
>>> > > > > > >>> initial
>>> > > > > > >>> > reviews from contributors, with no major remaining
>>> > > issues.
>>> > > > > > >>> > - Cross-language compatibility tests are passing: the
>>> > Arrow
>>> > > > C++
>>> > > > > > >>> decoder
>>> > > > > > >>> > reads Java-written data bit-exactly across ~1.56M
>>> > values
>>> > > > and
>>> > > > > 18
>>> > > > > > >>> > fixtures, covering V1 and V2 pages, multiple vector
>>> > > sizes,
>>> > > > > and
>>> > > > > > >>> > several
>>> > > > > > >>> > real datasets — zero mismatches.
>>> > > > > > >>> >
>>> > > > > > >>> > As raised in the java-pr we will be adding coverage for
>>> > > extreme
>>> > > > > > values
>>> > > > > > >>> > (those requiring
>>> > > > > > >>> > 63-64 bits after FOR is applied) before we close a vote.
>>> > We'll
>>> > > > aim
>>> > > > > > to
>>> > > > > > >>> get
>>> > > > > > >>> > that
>>> > > > > > >>> > done in parallel.
>>> > > > > > >>> >
>>> > > > > > >>> > Plan: unless there are objections, I intend to start a
>>> vote
>>> > at
>>> > > > the
>>> > > > > > end
>>> > > > > > >>> of
>>> > > > > > >>> > this week/early
>>> > > > > > >>> > next week.
>>> > > > > > >>> >
>>> > > > > > >>> > Thanks
>>> > > > > > >>> > Prateek
>>> > > > > > >>> >
>>> > > > > > >>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <
>>> > > prtkgaur@gmail.com>
>>> > > > > > >>> wrote:
>>> > > > > > >>> >
>>> > > > > > >>> > > Hi Team,
>>> > > > > > >>> > >
>>> > > > > > >>> > > Just wanted to provide some updates on ALP.
>>> > > > > > >>> > > Micah and I have done 3-4 rounds of review for the c++
>>> PR :
>>> > > > > > >>> > > https://github.com/apache/arrow/pull/48345
>>> > > > > > >>> > >
>>> > > > > > >>> > > For the Java implementation Vinoo has been working on
>>> > > following
>>> > > > > PR
>>> > > > > > :
>>> > > > > > >>> > > https://github.com/apache/parquet-java/pull/3397
>>> > > > > > >>> > >
>>> > > > > > >>> > > Best
>>> > > > > > >>> > > Prateek
>>> > > > > > >>> > >
>>> > > > > > >>> > > On Tue, May 5, 2026 at 1:50 PM Micah Kornfield <
>>> > > > > > >>> emkornfield@gmail.com>
>>> > > > > > >>> > > wrote:
>>> > > > > > >>> > >
>>> > > > > > >>> > >> Hi Antoine,
>>> > > > > > >>> > >>
>>> > > > > > >>> > >> > Apologies if the question was already asked, but
>>> should
>>> > we
>>> > > > > care
>>> > > > > > >>> about
>>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient
>>> than
>>> > > > > FLOAT16
>>> > > > > > +
>>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
>>> > > > > > >>> > >>
>>> > > > > > >>> > >> It was. We thought we could defer it for the
>>> following
>>> > > > reasons:
>>> > > > > > >>> > >> 1. It's not clear there are a lot of easy reference
>>> > > datasets
>>> > > > to
>>> > > > > > >>> test
>>> > > > > > >>> > its
>>> > > > > > >>> > >> effectiveness. It does look like there might be one
>>> or
>>> > two
>>> > > on
>>> > > > > > >>> > huggingface
>>> > > > > > >>> > >> (e.g. https://huggingface.co/datasets/kikitora/curdie
>>> ).
>>> > > > > > >>> > >> 2. It seemed likely that float 16 was more likely
>>> used
>>> > for
>>> > > > > values
>>> > > > > > >>> that
>>> > > > > > >>> > >> were less likely to reduce to decimal values.
>>> > > > > > >>> > >> 3. It could be added as an extension later if needed.
>>> > > > > > >>> > >>
>>> > > > > > >>> > >> Cheers,
>>> > > > > > >>> > >> Micah
>>> > > > > > >>> > >>
>>> > > > > > >>> > >>
>>> > > > > > >>> > >> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <
>>> > > > > antoine@python.org
>>> > > > > > >
>>> > > > > > >>> > wrote:
>>> > > > > > >>> > >>
>>> > > > > > >>> > >> >
>>> > > > > > >>> > >> > Hello,
>>> > > > > > >>> > >> >
>>> > > > > > >>> > >> > Apologies if the question was already asked, but
>>> should
>>> > we
>>> > > > > care
>>> > > > > > >>> about
>>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient
>>> than
>>> > > > > FLOAT16
>>> > > > > > +
>>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
>>> > > > > > >>> > >> >
>>> > > > > > >>> > >> > Regards
>>> > > > > > >>> > >> >
>>> > > > > > >>> > >> > Antoine.
>>> > > > > > >>> > >> >
>>> > > > > > >>> > >> >
>>> > > > > > >>> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
>>> > > > > > >>> > >> > > Thanks Andrew and Micah for review feedback on
>>> the two
>>> > > > PR's
>>> > > > > > >>> > >> > > 1) (c++ arrow repo)
>>> > > > > > >>> > >> https://github.com/apache/arrow/pull/48345/changes
>>> > > > > > >>> > >> > > 2) (parquet-format repo)
>>> > > > > > >>> > >> > https://github.com/apache/parquet-format/pull/557
>>> > > > > > >>> > >> > >
>>> > > > > > >>> > >> > > I have addressed all (unless I missed something)
>>> > > comments
>>> > > > on
>>> > > > > > >>> the two
>>> > > > > > >>> > >> > PR's.
>>> > > > > > >>> > >> > >
>>> > > > > > >>> > >> > > Best
>>> > > > > > >>> > >> > > Prateek
>>> > > > > > >>> > >> > >
>>> > > > > > >>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <
>>> > > > > > >>> prtkgaur@gmail.com>
>>> > > > > > >>> > >> wrote:
>>> > > > > > >>> > >> > >
>>> > > > > > >>> > >> > >> Thanks Andrew and Micah.
>>> > > > > > >>> > >> > >>
>>> > > > > > >>> > >> > >> `fair amount of feedback on at least the
>>> > > implementations`
>>> > > > > > >>> > >> > >> For the c++ I have already started addressing the
>>> > > > > feedback, I
>>> > > > > > >>> > should
>>> > > > > > >>> > >> be
>>> > > > > > >>> > >> > >> done with that Monday/Tuesday.
>>> > > > > > >>> > >> > >> I think Vinoo too has been making good progress
>>> on
>>> > the
>>> > > > Java
>>> > > > > > >>> > >> > implementation.
>>> > > > > > >>> > >> > >>
>>> > > > > > >>> > >> > >> Best
>>> > > > > > >>> > >> > >> Prateek
>>> > > > > > >>> > >> > >>
>>> > > > > > >>> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <
>>> > > > > > >>> > andrewlamb11@gmail.com
>>> > > > > > >>> > >> >
>>> > > > > > >>> > >> > >> wrote:
>>> > > > > > >>> > >> > >>
>>> > > > > > >>> > >> > >>> Got it. Thank you for the clarification -- I
>>> will
>>> > try
>>> > > > and
>>> > > > > > look
>>> > > > > > >>> > into
>>> > > > > > >>> > >> the
>>> > > > > > >>> > >> > >>> spec and the Rust implementation[1] in this next
>>> > week
>>> > > > > > >>> > >> > >>>
>>> > > > > > >>> > >> > >>> [1]:
>>> https://github.com/apache/arrow-rs/pull/9372
>>> > > > > > >>> > >> > >>>
>>> > > > > > >>> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah
>>> Kornfield <
>>> > > > > > >>> > >> > emkornfield@gmail.com>
>>> > > > > > >>> > >> > >>> wrote:
>>> > > > > > >>> > >> > >>>
>>> > > > > > >>> > >> > >>>> Hi Andrew,
>>> > > > > > >>> > >> > >>>> I think there is a fair amount of feedback on
>>> at
>>> > > least
>>> > > > > the
>>> > > > > > >>> > >> > >>>> implementations, typically I think we've waited
>>> > till
>>> > > > they
>>> > > > > > are
>>> > > > > > >>> > >> close to
>>> > > > > > >>> > >> > >>>> mergeable before a final vote. Otherwise I
>>> agree
>>> > we
>>> > > > are
>>> > > > > > very
>>> > > > > > >>> > >> close.
>>> > > > > > >>> > >> > >>>>
>>> > > > > > >>> > >> > >>>> -Micah
>>> > > > > > >>> > >> > >>>>
>>> > > > > > >>> > >> > >>>> On Saturday, April 25, 2026, Andrew Lamb <
>>> > > > > > >>> andrewlamb11@gmail.com
>>> > > > > > >>> > >
>>> > > > > > >>> > >> > wrote:
>>> > > > > > >>> > >> > >>>>
>>> > > > > > >>> > >> > >>>>> Thanks Prateek,
>>> > > > > > >>> > >> > >>>>>
>>> > > > > > >>> > >> > >>>>> I think from this content it looks to me like
>>> we
>>> > are
>>> > > > > ready
>>> > > > > > >>> to
>>> > > > > > >>> > >> start a
>>> > > > > > >>> > >> > >>>>> vote to explicitly accept ALP into Parquet
>>> > > > > > >>> > >> > >>>>>
>>> > > > > > >>> > >> > >>>>> Does anyone know of a reason we should
>>> postpone it
>>> > > for
>>> > > > > > >>> longer?
>>> > > > > > >>> > >> > >>>>> Perhaps someone needs some more time to
>>> review?
>>> > > > > > >>> > >> > >>>>>
>>> > > > > > >>> > >> > >>>>> Andrew
>>> > > > > > >>> > >> > >>>>>
>>> > > > > > >>> > >> > >>>>>
>>> > > > > > >>> > >> > >>>>>
>>> > > > > > >>> > >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <
>>> > > > > > >>> > prtkgaur@gmail.com>
>>> > > > > > >>> > >> > >>>>> wrote:
>>> > > > > > >>> > >> > >>>>>
>>> > > > > > >>> > >> > >>>>>> Hi team,
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>> Hope everyone is doing well. I got a chance
>>> to
>>> > work
>>> > > > > > >>> through all
>>> > > > > > >>> > >> the
>>> > > > > > >>> > >> > >>>>>> remaining feedback and update the spec doc.
>>> Here
>>> > > are
>>> > > > > the
>>> > > > > > >>> new
>>> > > > > > >>> > >> > artifacts
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>> 1) Spec document :
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> >
>>> > > > > > >>> > >>
>>> > > > > > >>> >
>>> > > > > > >>>
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>> 2) Spec document in parquet format repo :
>>> > > > > > >>> > >> > >>>>>>
>>> > https://github.com/apache/parquet-format/pull/557
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>> 3) Alp implementation in arrow c++ repo :
>>> > > > > > >>> > >> > >>>>>>
>>> > https://github.com/apache/arrow/pull/48345/changes
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>> 4) Alp implementation in parquet-java repo :
>>> Work
>>> > > for
>>> > > > > > >>> Vinoo and
>>> > > > > > >>> > >> > Julien
>>> > > > > > >>> > >> > >>>>>>
>>> > https://github.com/apache/parquet-java/pull/3397
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>> 5) PR with test and benchmarking artifacts in
>>> > > > > > >>> parquet-testing
>>> > > > > > >>> > >> repo :
>>> > > > > > >>> > >> > >>>>>>
>>> > https://github.com/apache/parquet-testing/pull/100
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>> And
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>> - Go : Arnav just submitted an in
>>> progress
>>> > > > > > >>> implementation
>>> > > > > > >>> > in
>>> > > > > > >>> > >> Go.
>>> > > > > > >>> > >> > >>>>>>
>>> https://github.com/apache/arrow-go/pull/704
>>> > (I
>>> > > > > > haven't
>>> > > > > > >>> > >> started
>>> > > > > > >>> > >> > >>>>>> looking at it yet)
>>> > > > > > >>> > >> > >>>>>> - Rust : I remember Andrew mentioned that
>>> > this
>>> > > > work
>>> > > > > > is
>>> > > > > > >>> also
>>> > > > > > >>> > >> in
>>> > > > > > >>> > >> > >>>>>> progress (So 4 languages!)
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>> *Arrow C++ implementation *
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>> The PR is out and was also used by Antoine to
>>> > > report
>>> > > > > the
>>> > > > > > >>> > numbers
>>> > > > > > >>> > >> as
>>> > > > > > >>> > >> > >>>>>> reported here. Micah and Konstantin have
>>> given 1
>>> > > > round
>>> > > > > of
>>> > > > > > >>> > >> feedback
>>> > > > > > >>> > >> > >>>>>> and I'm addressing them today. Please note
>>> that
>>> > the
>>> > > > > > default
>>> > > > > > >>> > >> > >>>>>> optimization flag for compiling is O2 and not
>>> > Q3. I
>>> > > > got
>>> > > > > > >>> around
>>> > > > > > >>> > >> 70%
>>> > > > > > >>> > >> > >>>>>> performance improvement in the decoding speed
>>> > when
>>> > > > > using
>>> > > > > > >>> the O3
>>> > > > > > >>> > >> > flag.
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>> *Parqet-MR Java implementation (working with
>>> > Vinoo
>>> > > > and
>>> > > > > > >>> Julien)
>>> > > > > > >>> > >> and
>>> > > > > > >>> > >> > **Cross
>>> > > > > > >>> > >> > >>>>>> Language testing*
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>> Let me know if you have any questions or
>>> > > > feedback.
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>> Now pasting some performance numbers
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>> Table 1: C++ ALP Double Decode — Spotify
>>> > Columns
>>> > > > > > >>> (Graviton
>>> > > > > > >>> > 3,
>>> > > > > > >>> > >> ARM
>>> > > > > > >>> > >> > >>>>>> Neoverse V1)
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>>
>>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>> │ Column │ -O2 (MB/s) │ -O3
>>> > > (MB/s) │
>>> > > > > > >>> Speedup │
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>>
>>> ├──────────────────┼──────────────┼──────────────┼─────────┤
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>> │ valence │ 3,155 │
>>> 5,523
>>> > > │
>>> > > > > > >>> 1.75x │
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>> │ danceability │ 3,233 │
>>> 5,685
>>> > > │
>>> > > > > > >>> 1.76x │
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>> │ energy │ 3,197 │
>>> 5,652
>>> > > │
>>> > > > > > >>> 1.77x │
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>> │ loudness │ 3,186 │
>>> 5,473
>>> > > │
>>> > > > > > >>> 1.72x │
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>>
>>> └──────────────────┴──────────────┴──────────────┴─────────┘
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR
>>> <
>>> > > > > > >>> > prtkgaur@gmail.com
>>> > > > > > >>> > >> >
>>> > > > > > >>> > >> > >>>>>> wrote:
>>> > > > > > >>> > >> > >>>>>>
>>> > > > > > >>> > >> > >>>>>>> @Micah Kornfield <em...@gmail.com> :
>>> Got
>>> > > it.
>>> > > > > > >>> > >> > >>>>>>>
>>> > > > > > >>> > >> > >>>>>>> @Andrew Lamb <an...@gmail.com>
>>> > > > > > >>> > >> > >>>>>>>
>>> > > > > > >>> > >> > >>>>>>>
>>> > > > > > >>> > >> > >>>>>>>> Do you think it would be good to start
>>> moving
>>> > the
>>> > > > > spec
>>> > > > > > >>> > >> development
>>> > > > > > >>> > >> > >>>>>>>> into
>>> > > > > > >>> > >> > >>>>>>>> markdown format, in preparation for
>>> finalizing
>>> > > it?
>>> > > > > > >>> > >> > >>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>
>>> > > > > > >>> > >> > >>>>>>> Yes I'll update the numbers for some of the
>>> > > > examples I
>>> > > > > > >>> have in
>>> > > > > > >>> > >> the
>>> > > > > > >>> > >> > >>>>>>> spec based
>>> > > > > > >>> > >> > >>>>>>> on the updated header size. Then we should
>>> be
>>> > good
>>> > > > to
>>> > > > > go
>>> > > > > > >>> for
>>> > > > > > >>> > the
>>> > > > > > >>> > >> > >>>>>>> markdown format.
>>> > > > > > >>> > >> > >>>>>>>
>>> > > > > > >>> > >> > >>>>>>> Thanks everyone!
>>> > > > > > >>> > >> > >>>>>>>
>>> > > > > > >>> > >> > >>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>> Andrew
>>> > > > > > >>> > >> > >>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK
>>> GAUR <
>>> > > > > > >>> > >> prtkgaur@gmail.com>
>>> > > > > > >>> > >> > >>>>>>>> wrote:
>>> > > > > > >>> > >> > >>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>> Hi team,
>>> > > > > > >>> > >> > >>>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>> 1) Andrew
>>> > > > > > >>> > >> > >>>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>> - Thanks for working on test files.
>>> My PR
>>> > > did
>>> > > > > add
>>> > > > > > >>> all
>>> > > > > > >>> > the
>>> > > > > > >>> > >> > test
>>> > > > > > >>> > >> > >>>>>>>> files I
>>> > > > > > >>> > >> > >>>>>>>>> used to benchmark on datasets. Maybe
>>> we
>>> > can
>>> > > > club
>>> > > > > > it
>>> > > > > > >>> > >> together.
>>> > > > > > >>> > >> > >>>>>>>> WIll also
>>> > > > > > >>> > >> > >>>>>>>>> aid
>>> > > > > > >>> > >> > >>>>>>>>> cross language testing
>>> > > > > > >>> > >> > >>>>>>>>> - Kosta Tarasov working on Rust
>>> > > > implementation.
>>> > > > > > >>> This is
>>> > > > > > >>> > >> > great.
>>> > > > > > >>> > >> > >>>>>>>> Thanks
>>> > > > > > >>> > >> > >>>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>> 2) Antoine
>>> > > > > > >>> > >> > >>>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>> - Thanks a lot for reporting the
>>> numbers
>>> > on
>>> > > > AMD.
>>> > > > > > >>> Looks
>>> > > > > > >>> > >> like
>>> > > > > > >>> > >> > you
>>> > > > > > >>> > >> > >>>>>>>> are
>>> > > > > > >>> > >> > >>>>>>>>> getting 8X the decoding performance of
>>> > BSS.
>>> > > > This
>>> > > > > > is
>>> > > > > > >>> > >> > amazing!!.
>>> > > > > > >>> > >> > >>>>>>>>> - Thanks for acknowledging the
>>> sampling
>>> > > > design.
>>> > > > > > >>> > >> > >>>>>>>>> - I agree with you on Fastlanes. In
>>> some
>>> > > crude
>>> > > > > > >>> > >> experiments I
>>> > > > > > >>> > >> > >>>>>>>> didn't get
>>> > > > > > >>> > >> > >>>>>>>>> a good perf benefit from it on
>>> Graviton3
>>> > > (but
>>> > > > > > maybe
>>> > > > > > >>> > there
>>> > > > > > >>> > >> was
>>> > > > > > >>> > >> > >>>>>>>> something
>>> > > > > > >>> > >> > >>>>>>>>> wrong with my implementation).
>>> > > > > > >>> > >> > >>>>>>>>> - Locking the 16bit exception
>>> encoding for
>>> > > the
>>> > > > > > spec
>>> > > > > > >>> in
>>> > > > > > >>> > >> this
>>> > > > > > >>> > >> > >>>>>>>> case.
>>> > > > > > >>> > >> > >>>>>>>>> - Awesome I think we have solved for
>>> all
>>> > > open
>>> > > > > > >>> questions
>>> > > > > > >>> > >> minus
>>> > > > > > >>> > >> > >>>>>>>> the
>>> > > > > > >>> > >> > >>>>>>>>> version byte :). (will get back on
>>> this
>>> > > soon)
>>> > > > > > >>> > >> > >>>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>> 3) Micah
>>> > > > > > >>> > >> > >>>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>> - FastLanes : The current spec does
>>> allow
>>> > > for
>>> > > > > > using
>>> > > > > > >>> > >> FastLane
>>> > > > > > >>> > >> > >>>>>>>> with the
>>> > > > > > >>> > >> > >>>>>>>>> configurable enum value for layout. We
>>> > > should
>>> > > > be
>>> > > > > > >>> able to
>>> > > > > > >>> > >> > inject
>>> > > > > > >>> > >> > >>>>>>>> any
>>> > > > > > >>> > >> > >>>>>>>>> layout
>>> > > > > > >>> > >> > >>>>>>>>> in the current design.
>>> > > > > > >>> > >> > >>>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>> Working on resolving all remaining open
>>> > comments
>>> > > > on
>>> > > > > > the
>>> > > > > > >>> spec
>>> > > > > > >>> > >> this
>>> > > > > > >>> > >> > >>>>>>>> week.
>>> > > > > > >>> > >> > >>>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>> Best
>>> > > > > > >>> > >> > >>>>>>>>> Prateek
>>> > > > > > >>> > >> > >>>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve
>>> > Loughran <
>>> > > > > > >>> > >> > >>>>>>>> stevel@cloudera.com>
>>> > > > > > >>> > >> > >>>>>>>>> wrote:
>>> > > > > > >>> > >> > >>>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah
>>> Kornfield
>>> > <
>>> > > > > > >>> > >> > >>>>>>>> emkornfield@gmail.com>
>>> > > > > > >>> > >> > >>>>>>>>>> wrote:
>>> > > > > > >>> > >> > >>>>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>>>> It looks like the actual issue
>>> described for
>>> > > ORC
>>> > > > > in
>>> > > > > > >>> the
>>> > > > > > >>> > >> paper
>>> > > > > > >>> > >> > >>>>>>>> is that
>>> > > > > > >>> > >> > >>>>>>>>> it
>>> > > > > > >>> > >> > >>>>>>>>>>> has multiple sub-encodings in a batch.
>>> This
>>> > > is
>>> > > > > > >>> different
>>> > > > > > >>> > >> then
>>> > > > > > >>> > >> > >>>>>>>> the
>>> > > > > > >>> > >> > >>>>>>>>> design
>>> > > > > > >>> > >> > >>>>>>>>>>> proposed here where there is still fixed
>>> > > > encoding
>>> > > > > > per
>>> > > > > > >>> page
>>> > > > > > >>> > >> in
>>> > > > > > >>> > >> > >>>>>>>> parquet.
>>> > > > > > >>> > >> > >>>>>>>>>>> Given reasonably sized pages I don't
>>> think
>>> > > > branch
>>> > > > > > >>> > >> > >>>>>>>> misprediction should
>>> > > > > > >>> > >> > >>>>>>>>>> be a
>>> > > > > > >>> > >> > >>>>>>>>>>> big issue for new encodings. I agree
>>> that
>>> > we
>>> > > > > should
>>> > > > > > >>> be
>>> > > > > > >>> > >> > >>>>>>>> conservative in
>>> > > > > > >>> > >> > >>>>>>>>>>> general for adding new encodings.
>>> > > > > > >>> > >> > >>>>>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>>> +1
>>> > > > > > >>> > >> > >>>>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>>
>>> > > > > > >>> > >> > >>>>>>>
>>> > > > > > >>> > >> > >
>>> > > > > > >>> > >> >
>>> > > > > > >>> > >> >
>>> > > > > > >>> > >> >
>>> > > > > > >>> > >>
>>> > > > > > >>> > >
>>> > > > > > >>> >
>>> > > > > > >>>
>>> > > > > > >>
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>>
>>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Andrew Lamb <an...@gmail.com>.
After re-reading Julien's Versioning Proposal, my understanding is that if
we adopt that system, we will somehow mark ALP in "Preview" in
parquet-format for release Version 3.
The versioning document doesn't seem to proscribe the particular method for
annotating "preview" features -- I think clearly marking them as such in
parquet.thrift as well as reflecting that on the
https://parquet.apache.org/docs/file-format/versions/ page somehow would be
adequate.
Andrew
On Thu, Jul 30, 2026 at 11:50 AM Andrew Lamb <an...@gmail.com> wrote:
> In my mind ALP is in the same position as other encodings that were added
> recently[1] (such as BYTE_STREAM_SPLIT for non floating point types)
>
> That is to say it will be available in the latest open source writers and
> readers, but realistically not widely used in the ecosystem where
> compatibility is a concern (a large part)
>
> I view Julian's work[2][3] to gather consensus on how to name combinations
> of features (e.g. "versions") as the mechanism by which we will accelerate
> the adoption of this change (along with other forward incompatible changes)
> across the ecosystem.
>
> > Will we exclude this change when cutting parquet-format v2.14.0?
>
> No I don't think so -- I would expect it to go out in the next
> parquet-format version as do other changes in the spec.
>
> Andrew
>
> [1]: https://parquet.apache.org/docs/file-format/versions/
> [2]: https://lists.apache.org/thread/xmnj8h0h8ozmrgox5tydhhs7yz7q2hss
> [3]: https://lists.apache.org/thread/qlf8lg90gqq41lllyy9mk2f0skqvftqj
>
>
> On Thu, Jul 30, 2026 at 3:51 AM Divjot Arora via dev <
> dev@parquet.apache.org> wrote:
>
>> Hi Andrew,
>>
>> How does this interact with the ongoing versioning discussion? Given that
>> new encodings are not forward compatible, I would assume ALP is included
>> in
>> Parquet V<next> (V3?), but it's not clear to me how that works if it's
>> merged to the parquet-format master branch now. Will we exclude this
>> change
>> when cutting parquet-format v2.14.0?
>>
>> -- Div
>>
>> On Thu, Jul 30, 2026 at 8:51 AM Andrew Lamb <an...@gmail.com>
>> wrote:
>>
>> > Hello,
>> >
>> > As an update, thanks to the epic work from many contributors, especially
>> > Prateek, I just merged the parquet-format change for ALP[1]. We plan to
>> > make additional editorial The remaining items I know of are merging the
>> > examples from parquet-testing[2], and then of course the various
>> > implementations
>> >
>> > Thanks again everyone. This is a great step forward
>> > Andrew
>> >
>> > [1]: https://github.com/apache/parquet-format/pull/557
>> > [2]: https://github.com/apache/parquet-testing/pull/100
>> >
>> > On Tue, Jul 21, 2026 at 4:39 PM Julien Le Dem <ju...@apache.org>
>> wrote:
>> >
>> > > It was also in my spam folder... :(
>> > > I've seen a bunch of apache list emails go to my spam recently.
>> > >
>> > > On Tue, Jul 21, 2026 at 10:58 AM Russell Spitzer <
>> > > russell.spitzer@gmail.com>
>> > > wrote:
>> > >
>> > > > Yes, my hope was that my vote would kick it out of spam but we
>> > definitely
>> > > > had two other votes that also went to spam for me.
>> > > >
>> > > > For anyone looking the thread is here
>> > > > https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt
>> > > >
>> > > > On Tue, Jul 21, 2026 at 10:57 AM Micah Kornfield <
>> > emkornfield@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > I've seen other's votes not go to spam, so hopefully people will
>> see
>> > it
>> > > > in
>> > > > > there inboxes (or see this thread and look in there spam).
>> > > > >
>> > > > > On Tue, Jul 21, 2026 at 10:52 AM PRATEEK GAUR <prtkgaur@gmail.com
>> >
>> > > > wrote:
>> > > > >
>> > > > > > Okay,
>> > > > > >
>> > > > > > I sent out the voting email yesterday and so far I've heard
>> from 3
>> > > > people
>> > > > > > that it ended up in their "spam folder". Dev VPN to be blamed
>> for
>> > > it?.
>> > > > > >
>> > > > > > What's the protocol here?
>> > > > > > I let people hunt for the email in their spam folder or send out
>> > > > another
>> > > > > > one?
>> > > > > >
>> > > > > > Best
>> > > > > > Prateek
>> > > > > >
>> > > > > > On Sun, Jul 19, 2026 at 10:47 PM PRATEEK GAUR <
>> prtkgaur@gmail.com>
>> > > > > wrote:
>> > > > > >
>> > > > > > > Hi team,
>> > > > > > >
>> > > > > > > Vinoo has added the tests which were requested in the last
>> round
>> > of
>> > > > > > review.
>> > > > > > > In his own words.
>> > > > > > >
>> > > > > > > "
>> > > > > > > The extreme value tests Micah wanted are done and pushed. I
>> added
>> > > > > > coverage
>> > > > > > > for values that need the full FOR bit width after the frame of
>> > > > > reference
>> > > > > > is
>> > > > > > > applied: 64-bit (the case where the signed max minus min
>> > > subtraction
>> > > > > > > overflows), 63-bit (the non-overflow case), and 32-bit for
>> > floats.
>> > > > All
>> > > > > > > lossless with zero exceptions. I also verified them through
>> Arrow
>> > > C++
>> > > > > ALP
>> > > > > > > reader (the local build), so the Java-written extreme columns
>> > > decode
>> > > > > > > correctly in C++, bit-exact against the expected values."
>> > > > > > >
>> > > > > > > State of the PR (#3397):
>> > > > > > >
>> > > > > > > - Merges cleanly with master (I pulled master in and
>> resolved
>> > > the
>> > > > > > > conflicts).
>> > > > > > > - 41 of 44 review threads resolved. The 3 remaining are
>> > > > intentional:
>> > > > > > > one deferred perf nit, one that stays open until
>> > parquet-format
>> > > > > ships
>> > > > > > ALP,
>> > > > > > >
>> > > > > > > "
>> > > > > > >
>> > > > > > > With this both the
>> > > > > > > 1) C++ PR :
>> https://github.com/apache/arrow/pull/48345/changes
>> > > > > > > 2) Java PR : https://github.com/apache/parquet-java/pull/3397
>> > > > > > >
>> > > > > > > Are cross language tested and range of test scenarios.
>> > > > > > >
>> > > > > > > Best
>> > > > > > > Prateek
>> > > > > > >
>> > > > > > >
>> > > > > > > On Mon, Jul 13, 2026 at 8:38 AM PRATEEK GAUR <
>> prtkgaur@gmail.com
>> > >
>> > > > > wrote:
>> > > > > > >
>> > > > > > >> Thanks Matt,
>> > > > > > >>
>> > > > > > >> I'll be on it this week.
>> > > > > > >>
>> > > > > > >> Best
>> > > > > > >> Prateek
>> > > > > > >>
>> > > > > > >> On Mon, Jul 13, 2026 at 8:36 AM Matt Topol <
>> > > zotthewizard@gmail.com>
>> > > > > > >> wrote:
>> > > > > > >>
>> > > > > > >>> There is also the Go implementation submitted by Arnav (
>> > > > > > >>> https://github.com/apache/arrow-go/pull/704) which is
>> waiting
>> > > for
>> > > > > > >>> updates.
>> > > > > > >>>
>> > > > > > >>> Just wanted to make sure it didn't get lost here.
>> > > > > > >>>
>> > > > > > >>> --Matt
>> > > > > > >>>
>> > > > > > >>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK GAUR <
>> > > prtkgaur@gmail.com>
>> > > > > > >>> wrote:
>> > > > > > >>>
>> > > > > > >>> > Hi all,
>> > > > > > >>> >
>> > > > > > >>> > I'd like to share a status update on the ALP encoding
>> effort
>> > > and
>> > > > > get
>> > > > > > a
>> > > > > > >>> feel
>> > > > > > >>> > from the community before starting a formal vote.
>> > > > > > >>> >
>> > > > > > >>> > We now have the specification plus implementations in both
>> > > > > languages:
>> > > > > > >>> >
>> > > > > > >>> > - Spec: apache/parquet-format#557 — GH-533 Add
>> > ALP
>> > > > > > encoding
>> > > > > > >>> > specification
>> > > > > > >>> > - C++ (Arrow): apache/arrow#48345 — GH-48701
>> > > [C++][Parquet]
>> > > > > Add
>> > > > > > >>> ALPpd
>> > > > > > >>> > encoding
>> > > > > > >>> > - Java: apache/parquet-java#3397 — Parquet
>> Java
>> > ALP
>> > > > > > >>> > Implementation (by Vinoo)
>> > > > > > >>> >
>> > > > > > >>> > Current state:
>> > > > > > >>> >
>> > > > > > >>> > - The C++ implementation has been through 3-4 rounds
>> of
>> > > > review.
>> > > > > > >>> > - The Java implementation has been tracking well and
>> has
>> > > had
>> > > > > > >>> initial
>> > > > > > >>> > reviews from contributors, with no major remaining
>> > > issues.
>> > > > > > >>> > - Cross-language compatibility tests are passing: the
>> > Arrow
>> > > > C++
>> > > > > > >>> decoder
>> > > > > > >>> > reads Java-written data bit-exactly across ~1.56M
>> > values
>> > > > and
>> > > > > 18
>> > > > > > >>> > fixtures, covering V1 and V2 pages, multiple vector
>> > > sizes,
>> > > > > and
>> > > > > > >>> > several
>> > > > > > >>> > real datasets — zero mismatches.
>> > > > > > >>> >
>> > > > > > >>> > As raised in the java-pr we will be adding coverage for
>> > > extreme
>> > > > > > values
>> > > > > > >>> > (those requiring
>> > > > > > >>> > 63-64 bits after FOR is applied) before we close a vote.
>> > We'll
>> > > > aim
>> > > > > > to
>> > > > > > >>> get
>> > > > > > >>> > that
>> > > > > > >>> > done in parallel.
>> > > > > > >>> >
>> > > > > > >>> > Plan: unless there are objections, I intend to start a
>> vote
>> > at
>> > > > the
>> > > > > > end
>> > > > > > >>> of
>> > > > > > >>> > this week/early
>> > > > > > >>> > next week.
>> > > > > > >>> >
>> > > > > > >>> > Thanks
>> > > > > > >>> > Prateek
>> > > > > > >>> >
>> > > > > > >>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <
>> > > prtkgaur@gmail.com>
>> > > > > > >>> wrote:
>> > > > > > >>> >
>> > > > > > >>> > > Hi Team,
>> > > > > > >>> > >
>> > > > > > >>> > > Just wanted to provide some updates on ALP.
>> > > > > > >>> > > Micah and I have done 3-4 rounds of review for the c++
>> PR :
>> > > > > > >>> > > https://github.com/apache/arrow/pull/48345
>> > > > > > >>> > >
>> > > > > > >>> > > For the Java implementation Vinoo has been working on
>> > > following
>> > > > > PR
>> > > > > > :
>> > > > > > >>> > > https://github.com/apache/parquet-java/pull/3397
>> > > > > > >>> > >
>> > > > > > >>> > > Best
>> > > > > > >>> > > Prateek
>> > > > > > >>> > >
>> > > > > > >>> > > On Tue, May 5, 2026 at 1:50 PM Micah Kornfield <
>> > > > > > >>> emkornfield@gmail.com>
>> > > > > > >>> > > wrote:
>> > > > > > >>> > >
>> > > > > > >>> > >> Hi Antoine,
>> > > > > > >>> > >>
>> > > > > > >>> > >> > Apologies if the question was already asked, but
>> should
>> > we
>> > > > > care
>> > > > > > >>> about
>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient
>> than
>> > > > > FLOAT16
>> > > > > > +
>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
>> > > > > > >>> > >>
>> > > > > > >>> > >> It was. We thought we could defer it for the following
>> > > > reasons:
>> > > > > > >>> > >> 1. It's not clear there are a lot of easy reference
>> > > datasets
>> > > > to
>> > > > > > >>> test
>> > > > > > >>> > its
>> > > > > > >>> > >> effectiveness. It does look like there might be one or
>> > two
>> > > on
>> > > > > > >>> > huggingface
>> > > > > > >>> > >> (e.g. https://huggingface.co/datasets/kikitora/curdie
>> ).
>> > > > > > >>> > >> 2. It seemed likely that float 16 was more likely used
>> > for
>> > > > > values
>> > > > > > >>> that
>> > > > > > >>> > >> were less likely to reduce to decimal values.
>> > > > > > >>> > >> 3. It could be added as an extension later if needed.
>> > > > > > >>> > >>
>> > > > > > >>> > >> Cheers,
>> > > > > > >>> > >> Micah
>> > > > > > >>> > >>
>> > > > > > >>> > >>
>> > > > > > >>> > >> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <
>> > > > > antoine@python.org
>> > > > > > >
>> > > > > > >>> > wrote:
>> > > > > > >>> > >>
>> > > > > > >>> > >> >
>> > > > > > >>> > >> > Hello,
>> > > > > > >>> > >> >
>> > > > > > >>> > >> > Apologies if the question was already asked, but
>> should
>> > we
>> > > > > care
>> > > > > > >>> about
>> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient
>> than
>> > > > > FLOAT16
>> > > > > > +
>> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
>> > > > > > >>> > >> >
>> > > > > > >>> > >> > Regards
>> > > > > > >>> > >> >
>> > > > > > >>> > >> > Antoine.
>> > > > > > >>> > >> >
>> > > > > > >>> > >> >
>> > > > > > >>> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
>> > > > > > >>> > >> > > Thanks Andrew and Micah for review feedback on the
>> two
>> > > > PR's
>> > > > > > >>> > >> > > 1) (c++ arrow repo)
>> > > > > > >>> > >> https://github.com/apache/arrow/pull/48345/changes
>> > > > > > >>> > >> > > 2) (parquet-format repo)
>> > > > > > >>> > >> > https://github.com/apache/parquet-format/pull/557
>> > > > > > >>> > >> > >
>> > > > > > >>> > >> > > I have addressed all (unless I missed something)
>> > > comments
>> > > > on
>> > > > > > >>> the two
>> > > > > > >>> > >> > PR's.
>> > > > > > >>> > >> > >
>> > > > > > >>> > >> > > Best
>> > > > > > >>> > >> > > Prateek
>> > > > > > >>> > >> > >
>> > > > > > >>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <
>> > > > > > >>> prtkgaur@gmail.com>
>> > > > > > >>> > >> wrote:
>> > > > > > >>> > >> > >
>> > > > > > >>> > >> > >> Thanks Andrew and Micah.
>> > > > > > >>> > >> > >>
>> > > > > > >>> > >> > >> `fair amount of feedback on at least the
>> > > implementations`
>> > > > > > >>> > >> > >> For the c++ I have already started addressing the
>> > > > > feedback, I
>> > > > > > >>> > should
>> > > > > > >>> > >> be
>> > > > > > >>> > >> > >> done with that Monday/Tuesday.
>> > > > > > >>> > >> > >> I think Vinoo too has been making good progress on
>> > the
>> > > > Java
>> > > > > > >>> > >> > implementation.
>> > > > > > >>> > >> > >>
>> > > > > > >>> > >> > >> Best
>> > > > > > >>> > >> > >> Prateek
>> > > > > > >>> > >> > >>
>> > > > > > >>> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <
>> > > > > > >>> > andrewlamb11@gmail.com
>> > > > > > >>> > >> >
>> > > > > > >>> > >> > >> wrote:
>> > > > > > >>> > >> > >>
>> > > > > > >>> > >> > >>> Got it. Thank you for the clarification -- I will
>> > try
>> > > > and
>> > > > > > look
>> > > > > > >>> > into
>> > > > > > >>> > >> the
>> > > > > > >>> > >> > >>> spec and the Rust implementation[1] in this next
>> > week
>> > > > > > >>> > >> > >>>
>> > > > > > >>> > >> > >>> [1]:
>> https://github.com/apache/arrow-rs/pull/9372
>> > > > > > >>> > >> > >>>
>> > > > > > >>> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield
>> <
>> > > > > > >>> > >> > emkornfield@gmail.com>
>> > > > > > >>> > >> > >>> wrote:
>> > > > > > >>> > >> > >>>
>> > > > > > >>> > >> > >>>> Hi Andrew,
>> > > > > > >>> > >> > >>>> I think there is a fair amount of feedback on at
>> > > least
>> > > > > the
>> > > > > > >>> > >> > >>>> implementations, typically I think we've waited
>> > till
>> > > > they
>> > > > > > are
>> > > > > > >>> > >> close to
>> > > > > > >>> > >> > >>>> mergeable before a final vote. Otherwise I
>> agree
>> > we
>> > > > are
>> > > > > > very
>> > > > > > >>> > >> close.
>> > > > > > >>> > >> > >>>>
>> > > > > > >>> > >> > >>>> -Micah
>> > > > > > >>> > >> > >>>>
>> > > > > > >>> > >> > >>>> On Saturday, April 25, 2026, Andrew Lamb <
>> > > > > > >>> andrewlamb11@gmail.com
>> > > > > > >>> > >
>> > > > > > >>> > >> > wrote:
>> > > > > > >>> > >> > >>>>
>> > > > > > >>> > >> > >>>>> Thanks Prateek,
>> > > > > > >>> > >> > >>>>>
>> > > > > > >>> > >> > >>>>> I think from this content it looks to me like
>> we
>> > are
>> > > > > ready
>> > > > > > >>> to
>> > > > > > >>> > >> start a
>> > > > > > >>> > >> > >>>>> vote to explicitly accept ALP into Parquet
>> > > > > > >>> > >> > >>>>>
>> > > > > > >>> > >> > >>>>> Does anyone know of a reason we should
>> postpone it
>> > > for
>> > > > > > >>> longer?
>> > > > > > >>> > >> > >>>>> Perhaps someone needs some more time to review?
>> > > > > > >>> > >> > >>>>>
>> > > > > > >>> > >> > >>>>> Andrew
>> > > > > > >>> > >> > >>>>>
>> > > > > > >>> > >> > >>>>>
>> > > > > > >>> > >> > >>>>>
>> > > > > > >>> > >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <
>> > > > > > >>> > prtkgaur@gmail.com>
>> > > > > > >>> > >> > >>>>> wrote:
>> > > > > > >>> > >> > >>>>>
>> > > > > > >>> > >> > >>>>>> Hi team,
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>> Hope everyone is doing well. I got a chance to
>> > work
>> > > > > > >>> through all
>> > > > > > >>> > >> the
>> > > > > > >>> > >> > >>>>>> remaining feedback and update the spec doc.
>> Here
>> > > are
>> > > > > the
>> > > > > > >>> new
>> > > > > > >>> > >> > artifacts
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>> 1) Spec document :
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> >
>> > > > > > >>> > >>
>> > > > > > >>> >
>> > > > > > >>>
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>> 2) Spec document in parquet format repo :
>> > > > > > >>> > >> > >>>>>>
>> > https://github.com/apache/parquet-format/pull/557
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>> 3) Alp implementation in arrow c++ repo :
>> > > > > > >>> > >> > >>>>>>
>> > https://github.com/apache/arrow/pull/48345/changes
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>> 4) Alp implementation in parquet-java repo :
>> Work
>> > > for
>> > > > > > >>> Vinoo and
>> > > > > > >>> > >> > Julien
>> > > > > > >>> > >> > >>>>>>
>> > https://github.com/apache/parquet-java/pull/3397
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>> 5) PR with test and benchmarking artifacts in
>> > > > > > >>> parquet-testing
>> > > > > > >>> > >> repo :
>> > > > > > >>> > >> > >>>>>>
>> > https://github.com/apache/parquet-testing/pull/100
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>> And
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>> - Go : Arnav just submitted an in progress
>> > > > > > >>> implementation
>> > > > > > >>> > in
>> > > > > > >>> > >> Go.
>> > > > > > >>> > >> > >>>>>>
>> https://github.com/apache/arrow-go/pull/704
>> > (I
>> > > > > > haven't
>> > > > > > >>> > >> started
>> > > > > > >>> > >> > >>>>>> looking at it yet)
>> > > > > > >>> > >> > >>>>>> - Rust : I remember Andrew mentioned that
>> > this
>> > > > work
>> > > > > > is
>> > > > > > >>> also
>> > > > > > >>> > >> in
>> > > > > > >>> > >> > >>>>>> progress (So 4 languages!)
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>> *Arrow C++ implementation *
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>> The PR is out and was also used by Antoine to
>> > > report
>> > > > > the
>> > > > > > >>> > numbers
>> > > > > > >>> > >> as
>> > > > > > >>> > >> > >>>>>> reported here. Micah and Konstantin have
>> given 1
>> > > > round
>> > > > > of
>> > > > > > >>> > >> feedback
>> > > > > > >>> > >> > >>>>>> and I'm addressing them today. Please note
>> that
>> > the
>> > > > > > default
>> > > > > > >>> > >> > >>>>>> optimization flag for compiling is O2 and not
>> > Q3. I
>> > > > got
>> > > > > > >>> around
>> > > > > > >>> > >> 70%
>> > > > > > >>> > >> > >>>>>> performance improvement in the decoding speed
>> > when
>> > > > > using
>> > > > > > >>> the O3
>> > > > > > >>> > >> > flag.
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>> *Parqet-MR Java implementation (working with
>> > Vinoo
>> > > > and
>> > > > > > >>> Julien)
>> > > > > > >>> > >> and
>> > > > > > >>> > >> > **Cross
>> > > > > > >>> > >> > >>>>>> Language testing*
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>> Let me know if you have any questions or
>> > > > feedback.
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>> Now pasting some performance numbers
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>> Table 1: C++ ALP Double Decode — Spotify
>> > Columns
>> > > > > > >>> (Graviton
>> > > > > > >>> > 3,
>> > > > > > >>> > >> ARM
>> > > > > > >>> > >> > >>>>>> Neoverse V1)
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>> │ Column │ -O2 (MB/s) │ -O3
>> > > (MB/s) │
>> > > > > > >>> Speedup │
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> ├──────────────────┼──────────────┼──────────────┼─────────┤
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>> │ valence │ 3,155 │
>> 5,523
>> > > │
>> > > > > > >>> 1.75x │
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>> │ danceability │ 3,233 │
>> 5,685
>> > > │
>> > > > > > >>> 1.76x │
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>> │ energy │ 3,197 │
>> 5,652
>> > > │
>> > > > > > >>> 1.77x │
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>> │ loudness │ 3,186 │
>> 5,473
>> > > │
>> > > > > > >>> 1.72x │
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> └──────────────────┴──────────────┴──────────────┴─────────┘
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <
>> > > > > > >>> > prtkgaur@gmail.com
>> > > > > > >>> > >> >
>> > > > > > >>> > >> > >>>>>> wrote:
>> > > > > > >>> > >> > >>>>>>
>> > > > > > >>> > >> > >>>>>>> @Micah Kornfield <em...@gmail.com> :
>> Got
>> > > it.
>> > > > > > >>> > >> > >>>>>>>
>> > > > > > >>> > >> > >>>>>>> @Andrew Lamb <an...@gmail.com>
>> > > > > > >>> > >> > >>>>>>>
>> > > > > > >>> > >> > >>>>>>>
>> > > > > > >>> > >> > >>>>>>>> Do you think it would be good to start
>> moving
>> > the
>> > > > > spec
>> > > > > > >>> > >> development
>> > > > > > >>> > >> > >>>>>>>> into
>> > > > > > >>> > >> > >>>>>>>> markdown format, in preparation for
>> finalizing
>> > > it?
>> > > > > > >>> > >> > >>>>>>>>
>> > > > > > >>> > >> > >>>>>>>
>> > > > > > >>> > >> > >>>>>>> Yes I'll update the numbers for some of the
>> > > > examples I
>> > > > > > >>> have in
>> > > > > > >>> > >> the
>> > > > > > >>> > >> > >>>>>>> spec based
>> > > > > > >>> > >> > >>>>>>> on the updated header size. Then we should be
>> > good
>> > > > to
>> > > > > go
>> > > > > > >>> for
>> > > > > > >>> > the
>> > > > > > >>> > >> > >>>>>>> markdown format.
>> > > > > > >>> > >> > >>>>>>>
>> > > > > > >>> > >> > >>>>>>> Thanks everyone!
>> > > > > > >>> > >> > >>>>>>>
>> > > > > > >>> > >> > >>>>>>>
>> > > > > > >>> > >> > >>>>>>>>
>> > > > > > >>> > >> > >>>>>>>> Andrew
>> > > > > > >>> > >> > >>>>>>>>
>> > > > > > >>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK
>> GAUR <
>> > > > > > >>> > >> prtkgaur@gmail.com>
>> > > > > > >>> > >> > >>>>>>>> wrote:
>> > > > > > >>> > >> > >>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>> Hi team,
>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>> 1) Andrew
>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>> - Thanks for working on test files. My
>> PR
>> > > did
>> > > > > add
>> > > > > > >>> all
>> > > > > > >>> > the
>> > > > > > >>> > >> > test
>> > > > > > >>> > >> > >>>>>>>> files I
>> > > > > > >>> > >> > >>>>>>>>> used to benchmark on datasets. Maybe we
>> > can
>> > > > club
>> > > > > > it
>> > > > > > >>> > >> together.
>> > > > > > >>> > >> > >>>>>>>> WIll also
>> > > > > > >>> > >> > >>>>>>>>> aid
>> > > > > > >>> > >> > >>>>>>>>> cross language testing
>> > > > > > >>> > >> > >>>>>>>>> - Kosta Tarasov working on Rust
>> > > > implementation.
>> > > > > > >>> This is
>> > > > > > >>> > >> > great.
>> > > > > > >>> > >> > >>>>>>>> Thanks
>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>> 2) Antoine
>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>> - Thanks a lot for reporting the
>> numbers
>> > on
>> > > > AMD.
>> > > > > > >>> Looks
>> > > > > > >>> > >> like
>> > > > > > >>> > >> > you
>> > > > > > >>> > >> > >>>>>>>> are
>> > > > > > >>> > >> > >>>>>>>>> getting 8X the decoding performance of
>> > BSS.
>> > > > This
>> > > > > > is
>> > > > > > >>> > >> > amazing!!.
>> > > > > > >>> > >> > >>>>>>>>> - Thanks for acknowledging the sampling
>> > > > design.
>> > > > > > >>> > >> > >>>>>>>>> - I agree with you on Fastlanes. In
>> some
>> > > crude
>> > > > > > >>> > >> experiments I
>> > > > > > >>> > >> > >>>>>>>> didn't get
>> > > > > > >>> > >> > >>>>>>>>> a good perf benefit from it on
>> Graviton3
>> > > (but
>> > > > > > maybe
>> > > > > > >>> > there
>> > > > > > >>> > >> was
>> > > > > > >>> > >> > >>>>>>>> something
>> > > > > > >>> > >> > >>>>>>>>> wrong with my implementation).
>> > > > > > >>> > >> > >>>>>>>>> - Locking the 16bit exception encoding
>> for
>> > > the
>> > > > > > spec
>> > > > > > >>> in
>> > > > > > >>> > >> this
>> > > > > > >>> > >> > >>>>>>>> case.
>> > > > > > >>> > >> > >>>>>>>>> - Awesome I think we have solved for
>> all
>> > > open
>> > > > > > >>> questions
>> > > > > > >>> > >> minus
>> > > > > > >>> > >> > >>>>>>>> the
>> > > > > > >>> > >> > >>>>>>>>> version byte :). (will get back on this
>> > > soon)
>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>> 3) Micah
>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>> - FastLanes : The current spec does
>> allow
>> > > for
>> > > > > > using
>> > > > > > >>> > >> FastLane
>> > > > > > >>> > >> > >>>>>>>> with the
>> > > > > > >>> > >> > >>>>>>>>> configurable enum value for layout. We
>> > > should
>> > > > be
>> > > > > > >>> able to
>> > > > > > >>> > >> > inject
>> > > > > > >>> > >> > >>>>>>>> any
>> > > > > > >>> > >> > >>>>>>>>> layout
>> > > > > > >>> > >> > >>>>>>>>> in the current design.
>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>> Working on resolving all remaining open
>> > comments
>> > > > on
>> > > > > > the
>> > > > > > >>> spec
>> > > > > > >>> > >> this
>> > > > > > >>> > >> > >>>>>>>> week.
>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>> Best
>> > > > > > >>> > >> > >>>>>>>>> Prateek
>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve
>> > Loughran <
>> > > > > > >>> > >> > >>>>>>>> stevel@cloudera.com>
>> > > > > > >>> > >> > >>>>>>>>> wrote:
>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah
>> Kornfield
>> > <
>> > > > > > >>> > >> > >>>>>>>> emkornfield@gmail.com>
>> > > > > > >>> > >> > >>>>>>>>>> wrote:
>> > > > > > >>> > >> > >>>>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>>>> It looks like the actual issue described
>> for
>> > > ORC
>> > > > > in
>> > > > > > >>> the
>> > > > > > >>> > >> paper
>> > > > > > >>> > >> > >>>>>>>> is that
>> > > > > > >>> > >> > >>>>>>>>> it
>> > > > > > >>> > >> > >>>>>>>>>>> has multiple sub-encodings in a batch.
>> This
>> > > is
>> > > > > > >>> different
>> > > > > > >>> > >> then
>> > > > > > >>> > >> > >>>>>>>> the
>> > > > > > >>> > >> > >>>>>>>>> design
>> > > > > > >>> > >> > >>>>>>>>>>> proposed here where there is still fixed
>> > > > encoding
>> > > > > > per
>> > > > > > >>> page
>> > > > > > >>> > >> in
>> > > > > > >>> > >> > >>>>>>>> parquet.
>> > > > > > >>> > >> > >>>>>>>>>>> Given reasonably sized pages I don't
>> think
>> > > > branch
>> > > > > > >>> > >> > >>>>>>>> misprediction should
>> > > > > > >>> > >> > >>>>>>>>>> be a
>> > > > > > >>> > >> > >>>>>>>>>>> big issue for new encodings. I agree
>> that
>> > we
>> > > > > should
>> > > > > > >>> be
>> > > > > > >>> > >> > >>>>>>>> conservative in
>> > > > > > >>> > >> > >>>>>>>>>>> general for adding new encodings.
>> > > > > > >>> > >> > >>>>>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>>> +1
>> > > > > > >>> > >> > >>>>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>>
>> > > > > > >>> > >> > >>>>>>>>
>> > > > > > >>> > >> > >>>>>>>
>> > > > > > >>> > >> > >
>> > > > > > >>> > >> >
>> > > > > > >>> > >> >
>> > > > > > >>> > >> >
>> > > > > > >>> > >>
>> > > > > > >>> > >
>> > > > > > >>> >
>> > > > > > >>>
>> > > > > > >>
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Andrew Lamb <an...@gmail.com>.
In my mind ALP is in the same position as other encodings that were added
recently[1] (such as BYTE_STREAM_SPLIT for non floating point types)
That is to say it will be available in the latest open source writers and
readers, but realistically not widely used in the ecosystem where
compatibility is a concern (a large part)
I view Julian's work[2][3] to gather consensus on how to name combinations
of features (e.g. "versions") as the mechanism by which we will accelerate
the adoption of this change (along with other forward incompatible changes)
across the ecosystem.
> Will we exclude this change when cutting parquet-format v2.14.0?
No I don't think so -- I would expect it to go out in the next
parquet-format version as do other changes in the spec.
Andrew
[1]: https://parquet.apache.org/docs/file-format/versions/
[2]: https://lists.apache.org/thread/xmnj8h0h8ozmrgox5tydhhs7yz7q2hss
[3]: https://lists.apache.org/thread/qlf8lg90gqq41lllyy9mk2f0skqvftqj
On Thu, Jul 30, 2026 at 3:51 AM Divjot Arora via dev <de...@parquet.apache.org>
wrote:
> Hi Andrew,
>
> How does this interact with the ongoing versioning discussion? Given that
> new encodings are not forward compatible, I would assume ALP is included in
> Parquet V<next> (V3?), but it's not clear to me how that works if it's
> merged to the parquet-format master branch now. Will we exclude this change
> when cutting parquet-format v2.14.0?
>
> -- Div
>
> On Thu, Jul 30, 2026 at 8:51 AM Andrew Lamb <an...@gmail.com>
> wrote:
>
> > Hello,
> >
> > As an update, thanks to the epic work from many contributors, especially
> > Prateek, I just merged the parquet-format change for ALP[1]. We plan to
> > make additional editorial The remaining items I know of are merging the
> > examples from parquet-testing[2], and then of course the various
> > implementations
> >
> > Thanks again everyone. This is a great step forward
> > Andrew
> >
> > [1]: https://github.com/apache/parquet-format/pull/557
> > [2]: https://github.com/apache/parquet-testing/pull/100
> >
> > On Tue, Jul 21, 2026 at 4:39 PM Julien Le Dem <ju...@apache.org> wrote:
> >
> > > It was also in my spam folder... :(
> > > I've seen a bunch of apache list emails go to my spam recently.
> > >
> > > On Tue, Jul 21, 2026 at 10:58 AM Russell Spitzer <
> > > russell.spitzer@gmail.com>
> > > wrote:
> > >
> > > > Yes, my hope was that my vote would kick it out of spam but we
> > definitely
> > > > had two other votes that also went to spam for me.
> > > >
> > > > For anyone looking the thread is here
> > > > https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt
> > > >
> > > > On Tue, Jul 21, 2026 at 10:57 AM Micah Kornfield <
> > emkornfield@gmail.com>
> > > > wrote:
> > > >
> > > > > I've seen other's votes not go to spam, so hopefully people will
> see
> > it
> > > > in
> > > > > there inboxes (or see this thread and look in there spam).
> > > > >
> > > > > On Tue, Jul 21, 2026 at 10:52 AM PRATEEK GAUR <pr...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > Okay,
> > > > > >
> > > > > > I sent out the voting email yesterday and so far I've heard from
> 3
> > > > people
> > > > > > that it ended up in their "spam folder". Dev VPN to be blamed for
> > > it?.
> > > > > >
> > > > > > What's the protocol here?
> > > > > > I let people hunt for the email in their spam folder or send out
> > > > another
> > > > > > one?
> > > > > >
> > > > > > Best
> > > > > > Prateek
> > > > > >
> > > > > > On Sun, Jul 19, 2026 at 10:47 PM PRATEEK GAUR <
> prtkgaur@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Hi team,
> > > > > > >
> > > > > > > Vinoo has added the tests which were requested in the last
> round
> > of
> > > > > > review.
> > > > > > > In his own words.
> > > > > > >
> > > > > > > "
> > > > > > > The extreme value tests Micah wanted are done and pushed. I
> added
> > > > > > coverage
> > > > > > > for values that need the full FOR bit width after the frame of
> > > > > reference
> > > > > > is
> > > > > > > applied: 64-bit (the case where the signed max minus min
> > > subtraction
> > > > > > > overflows), 63-bit (the non-overflow case), and 32-bit for
> > floats.
> > > > All
> > > > > > > lossless with zero exceptions. I also verified them through
> Arrow
> > > C++
> > > > > ALP
> > > > > > > reader (the local build), so the Java-written extreme columns
> > > decode
> > > > > > > correctly in C++, bit-exact against the expected values."
> > > > > > >
> > > > > > > State of the PR (#3397):
> > > > > > >
> > > > > > > - Merges cleanly with master (I pulled master in and
> resolved
> > > the
> > > > > > > conflicts).
> > > > > > > - 41 of 44 review threads resolved. The 3 remaining are
> > > > intentional:
> > > > > > > one deferred perf nit, one that stays open until
> > parquet-format
> > > > > ships
> > > > > > ALP,
> > > > > > >
> > > > > > > "
> > > > > > >
> > > > > > > With this both the
> > > > > > > 1) C++ PR : https://github.com/apache/arrow/pull/48345/changes
> > > > > > > 2) Java PR : https://github.com/apache/parquet-java/pull/3397
> > > > > > >
> > > > > > > Are cross language tested and range of test scenarios.
> > > > > > >
> > > > > > > Best
> > > > > > > Prateek
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Jul 13, 2026 at 8:38 AM PRATEEK GAUR <
> prtkgaur@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > >> Thanks Matt,
> > > > > > >>
> > > > > > >> I'll be on it this week.
> > > > > > >>
> > > > > > >> Best
> > > > > > >> Prateek
> > > > > > >>
> > > > > > >> On Mon, Jul 13, 2026 at 8:36 AM Matt Topol <
> > > zotthewizard@gmail.com>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >>> There is also the Go implementation submitted by Arnav (
> > > > > > >>> https://github.com/apache/arrow-go/pull/704) which is
> waiting
> > > for
> > > > > > >>> updates.
> > > > > > >>>
> > > > > > >>> Just wanted to make sure it didn't get lost here.
> > > > > > >>>
> > > > > > >>> --Matt
> > > > > > >>>
> > > > > > >>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK GAUR <
> > > prtkgaur@gmail.com>
> > > > > > >>> wrote:
> > > > > > >>>
> > > > > > >>> > Hi all,
> > > > > > >>> >
> > > > > > >>> > I'd like to share a status update on the ALP encoding
> effort
> > > and
> > > > > get
> > > > > > a
> > > > > > >>> feel
> > > > > > >>> > from the community before starting a formal vote.
> > > > > > >>> >
> > > > > > >>> > We now have the specification plus implementations in both
> > > > > languages:
> > > > > > >>> >
> > > > > > >>> > - Spec: apache/parquet-format#557 — GH-533 Add
> > ALP
> > > > > > encoding
> > > > > > >>> > specification
> > > > > > >>> > - C++ (Arrow): apache/arrow#48345 — GH-48701
> > > [C++][Parquet]
> > > > > Add
> > > > > > >>> ALPpd
> > > > > > >>> > encoding
> > > > > > >>> > - Java: apache/parquet-java#3397 — Parquet Java
> > ALP
> > > > > > >>> > Implementation (by Vinoo)
> > > > > > >>> >
> > > > > > >>> > Current state:
> > > > > > >>> >
> > > > > > >>> > - The C++ implementation has been through 3-4 rounds of
> > > > review.
> > > > > > >>> > - The Java implementation has been tracking well and
> has
> > > had
> > > > > > >>> initial
> > > > > > >>> > reviews from contributors, with no major remaining
> > > issues.
> > > > > > >>> > - Cross-language compatibility tests are passing: the
> > Arrow
> > > > C++
> > > > > > >>> decoder
> > > > > > >>> > reads Java-written data bit-exactly across ~1.56M
> > values
> > > > and
> > > > > 18
> > > > > > >>> > fixtures, covering V1 and V2 pages, multiple vector
> > > sizes,
> > > > > and
> > > > > > >>> > several
> > > > > > >>> > real datasets — zero mismatches.
> > > > > > >>> >
> > > > > > >>> > As raised in the java-pr we will be adding coverage for
> > > extreme
> > > > > > values
> > > > > > >>> > (those requiring
> > > > > > >>> > 63-64 bits after FOR is applied) before we close a vote.
> > We'll
> > > > aim
> > > > > > to
> > > > > > >>> get
> > > > > > >>> > that
> > > > > > >>> > done in parallel.
> > > > > > >>> >
> > > > > > >>> > Plan: unless there are objections, I intend to start a vote
> > at
> > > > the
> > > > > > end
> > > > > > >>> of
> > > > > > >>> > this week/early
> > > > > > >>> > next week.
> > > > > > >>> >
> > > > > > >>> > Thanks
> > > > > > >>> > Prateek
> > > > > > >>> >
> > > > > > >>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <
> > > prtkgaur@gmail.com>
> > > > > > >>> wrote:
> > > > > > >>> >
> > > > > > >>> > > Hi Team,
> > > > > > >>> > >
> > > > > > >>> > > Just wanted to provide some updates on ALP.
> > > > > > >>> > > Micah and I have done 3-4 rounds of review for the c++
> PR :
> > > > > > >>> > > https://github.com/apache/arrow/pull/48345
> > > > > > >>> > >
> > > > > > >>> > > For the Java implementation Vinoo has been working on
> > > following
> > > > > PR
> > > > > > :
> > > > > > >>> > > https://github.com/apache/parquet-java/pull/3397
> > > > > > >>> > >
> > > > > > >>> > > Best
> > > > > > >>> > > Prateek
> > > > > > >>> > >
> > > > > > >>> > > On Tue, May 5, 2026 at 1:50 PM Micah Kornfield <
> > > > > > >>> emkornfield@gmail.com>
> > > > > > >>> > > wrote:
> > > > > > >>> > >
> > > > > > >>> > >> Hi Antoine,
> > > > > > >>> > >>
> > > > > > >>> > >> > Apologies if the question was already asked, but
> should
> > we
> > > > > care
> > > > > > >>> about
> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient
> than
> > > > > FLOAT16
> > > > > > +
> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> > > > > > >>> > >>
> > > > > > >>> > >> It was. We thought we could defer it for the following
> > > > reasons:
> > > > > > >>> > >> 1. It's not clear there are a lot of easy reference
> > > datasets
> > > > to
> > > > > > >>> test
> > > > > > >>> > its
> > > > > > >>> > >> effectiveness. It does look like there might be one or
> > two
> > > on
> > > > > > >>> > huggingface
> > > > > > >>> > >> (e.g. https://huggingface.co/datasets/kikitora/curdie).
> > > > > > >>> > >> 2. It seemed likely that float 16 was more likely used
> > for
> > > > > values
> > > > > > >>> that
> > > > > > >>> > >> were less likely to reduce to decimal values.
> > > > > > >>> > >> 3. It could be added as an extension later if needed.
> > > > > > >>> > >>
> > > > > > >>> > >> Cheers,
> > > > > > >>> > >> Micah
> > > > > > >>> > >>
> > > > > > >>> > >>
> > > > > > >>> > >> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <
> > > > > antoine@python.org
> > > > > > >
> > > > > > >>> > wrote:
> > > > > > >>> > >>
> > > > > > >>> > >> >
> > > > > > >>> > >> > Hello,
> > > > > > >>> > >> >
> > > > > > >>> > >> > Apologies if the question was already asked, but
> should
> > we
> > > > > care
> > > > > > >>> about
> > > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient
> than
> > > > > FLOAT16
> > > > > > +
> > > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> > > > > > >>> > >> >
> > > > > > >>> > >> > Regards
> > > > > > >>> > >> >
> > > > > > >>> > >> > Antoine.
> > > > > > >>> > >> >
> > > > > > >>> > >> >
> > > > > > >>> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
> > > > > > >>> > >> > > Thanks Andrew and Micah for review feedback on the
> two
> > > > PR's
> > > > > > >>> > >> > > 1) (c++ arrow repo)
> > > > > > >>> > >> https://github.com/apache/arrow/pull/48345/changes
> > > > > > >>> > >> > > 2) (parquet-format repo)
> > > > > > >>> > >> > https://github.com/apache/parquet-format/pull/557
> > > > > > >>> > >> > >
> > > > > > >>> > >> > > I have addressed all (unless I missed something)
> > > comments
> > > > on
> > > > > > >>> the two
> > > > > > >>> > >> > PR's.
> > > > > > >>> > >> > >
> > > > > > >>> > >> > > Best
> > > > > > >>> > >> > > Prateek
> > > > > > >>> > >> > >
> > > > > > >>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <
> > > > > > >>> prtkgaur@gmail.com>
> > > > > > >>> > >> wrote:
> > > > > > >>> > >> > >
> > > > > > >>> > >> > >> Thanks Andrew and Micah.
> > > > > > >>> > >> > >>
> > > > > > >>> > >> > >> `fair amount of feedback on at least the
> > > implementations`
> > > > > > >>> > >> > >> For the c++ I have already started addressing the
> > > > > feedback, I
> > > > > > >>> > should
> > > > > > >>> > >> be
> > > > > > >>> > >> > >> done with that Monday/Tuesday.
> > > > > > >>> > >> > >> I think Vinoo too has been making good progress on
> > the
> > > > Java
> > > > > > >>> > >> > implementation.
> > > > > > >>> > >> > >>
> > > > > > >>> > >> > >> Best
> > > > > > >>> > >> > >> Prateek
> > > > > > >>> > >> > >>
> > > > > > >>> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <
> > > > > > >>> > andrewlamb11@gmail.com
> > > > > > >>> > >> >
> > > > > > >>> > >> > >> wrote:
> > > > > > >>> > >> > >>
> > > > > > >>> > >> > >>> Got it. Thank you for the clarification -- I will
> > try
> > > > and
> > > > > > look
> > > > > > >>> > into
> > > > > > >>> > >> the
> > > > > > >>> > >> > >>> spec and the Rust implementation[1] in this next
> > week
> > > > > > >>> > >> > >>>
> > > > > > >>> > >> > >>> [1]: https://github.com/apache/arrow-rs/pull/9372
> > > > > > >>> > >> > >>>
> > > > > > >>> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <
> > > > > > >>> > >> > emkornfield@gmail.com>
> > > > > > >>> > >> > >>> wrote:
> > > > > > >>> > >> > >>>
> > > > > > >>> > >> > >>>> Hi Andrew,
> > > > > > >>> > >> > >>>> I think there is a fair amount of feedback on at
> > > least
> > > > > the
> > > > > > >>> > >> > >>>> implementations, typically I think we've waited
> > till
> > > > they
> > > > > > are
> > > > > > >>> > >> close to
> > > > > > >>> > >> > >>>> mergeable before a final vote. Otherwise I agree
> > we
> > > > are
> > > > > > very
> > > > > > >>> > >> close.
> > > > > > >>> > >> > >>>>
> > > > > > >>> > >> > >>>> -Micah
> > > > > > >>> > >> > >>>>
> > > > > > >>> > >> > >>>> On Saturday, April 25, 2026, Andrew Lamb <
> > > > > > >>> andrewlamb11@gmail.com
> > > > > > >>> > >
> > > > > > >>> > >> > wrote:
> > > > > > >>> > >> > >>>>
> > > > > > >>> > >> > >>>>> Thanks Prateek,
> > > > > > >>> > >> > >>>>>
> > > > > > >>> > >> > >>>>> I think from this content it looks to me like we
> > are
> > > > > ready
> > > > > > >>> to
> > > > > > >>> > >> start a
> > > > > > >>> > >> > >>>>> vote to explicitly accept ALP into Parquet
> > > > > > >>> > >> > >>>>>
> > > > > > >>> > >> > >>>>> Does anyone know of a reason we should postpone
> it
> > > for
> > > > > > >>> longer?
> > > > > > >>> > >> > >>>>> Perhaps someone needs some more time to review?
> > > > > > >>> > >> > >>>>>
> > > > > > >>> > >> > >>>>> Andrew
> > > > > > >>> > >> > >>>>>
> > > > > > >>> > >> > >>>>>
> > > > > > >>> > >> > >>>>>
> > > > > > >>> > >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <
> > > > > > >>> > prtkgaur@gmail.com>
> > > > > > >>> > >> > >>>>> wrote:
> > > > > > >>> > >> > >>>>>
> > > > > > >>> > >> > >>>>>> Hi team,
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>> Hope everyone is doing well. I got a chance to
> > work
> > > > > > >>> through all
> > > > > > >>> > >> the
> > > > > > >>> > >> > >>>>>> remaining feedback and update the spec doc.
> Here
> > > are
> > > > > the
> > > > > > >>> new
> > > > > > >>> > >> > artifacts
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>> 1) Spec document :
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> >
> > > > > > >>> > >>
> > > > > > >>> >
> > > > > > >>>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>> 2) Spec document in parquet format repo :
> > > > > > >>> > >> > >>>>>>
> > https://github.com/apache/parquet-format/pull/557
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>> 3) Alp implementation in arrow c++ repo :
> > > > > > >>> > >> > >>>>>>
> > https://github.com/apache/arrow/pull/48345/changes
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>> 4) Alp implementation in parquet-java repo :
> Work
> > > for
> > > > > > >>> Vinoo and
> > > > > > >>> > >> > Julien
> > > > > > >>> > >> > >>>>>>
> > https://github.com/apache/parquet-java/pull/3397
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>> 5) PR with test and benchmarking artifacts in
> > > > > > >>> parquet-testing
> > > > > > >>> > >> repo :
> > > > > > >>> > >> > >>>>>>
> > https://github.com/apache/parquet-testing/pull/100
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>> And
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>> - Go : Arnav just submitted an in progress
> > > > > > >>> implementation
> > > > > > >>> > in
> > > > > > >>> > >> Go.
> > > > > > >>> > >> > >>>>>>
> https://github.com/apache/arrow-go/pull/704
> > (I
> > > > > > haven't
> > > > > > >>> > >> started
> > > > > > >>> > >> > >>>>>> looking at it yet)
> > > > > > >>> > >> > >>>>>> - Rust : I remember Andrew mentioned that
> > this
> > > > work
> > > > > > is
> > > > > > >>> also
> > > > > > >>> > >> in
> > > > > > >>> > >> > >>>>>> progress (So 4 languages!)
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>> *Arrow C++ implementation *
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>> The PR is out and was also used by Antoine to
> > > report
> > > > > the
> > > > > > >>> > numbers
> > > > > > >>> > >> as
> > > > > > >>> > >> > >>>>>> reported here. Micah and Konstantin have given
> 1
> > > > round
> > > > > of
> > > > > > >>> > >> feedback
> > > > > > >>> > >> > >>>>>> and I'm addressing them today. Please note that
> > the
> > > > > > default
> > > > > > >>> > >> > >>>>>> optimization flag for compiling is O2 and not
> > Q3. I
> > > > got
> > > > > > >>> around
> > > > > > >>> > >> 70%
> > > > > > >>> > >> > >>>>>> performance improvement in the decoding speed
> > when
> > > > > using
> > > > > > >>> the O3
> > > > > > >>> > >> > flag.
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>> *Parqet-MR Java implementation (working with
> > Vinoo
> > > > and
> > > > > > >>> Julien)
> > > > > > >>> > >> and
> > > > > > >>> > >> > **Cross
> > > > > > >>> > >> > >>>>>> Language testing*
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>> Let me know if you have any questions or
> > > > feedback.
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>> Now pasting some performance numbers
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>> Table 1: C++ ALP Double Decode — Spotify
> > Columns
> > > > > > >>> (Graviton
> > > > > > >>> > 3,
> > > > > > >>> > >> ARM
> > > > > > >>> > >> > >>>>>> Neoverse V1)
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>> │ Column │ -O2 (MB/s) │ -O3
> > > (MB/s) │
> > > > > > >>> Speedup │
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> ├──────────────────┼──────────────┼──────────────┼─────────┤
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>> │ valence │ 3,155 │
> 5,523
> > > │
> > > > > > >>> 1.75x │
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>> │ danceability │ 3,233 │
> 5,685
> > > │
> > > > > > >>> 1.76x │
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>> │ energy │ 3,197 │
> 5,652
> > > │
> > > > > > >>> 1.77x │
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>> │ loudness │ 3,186 │
> 5,473
> > > │
> > > > > > >>> 1.72x │
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> └──────────────────┴──────────────┴──────────────┴─────────┘
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <
> > > > > > >>> > prtkgaur@gmail.com
> > > > > > >>> > >> >
> > > > > > >>> > >> > >>>>>> wrote:
> > > > > > >>> > >> > >>>>>>
> > > > > > >>> > >> > >>>>>>> @Micah Kornfield <em...@gmail.com> :
> Got
> > > it.
> > > > > > >>> > >> > >>>>>>>
> > > > > > >>> > >> > >>>>>>> @Andrew Lamb <an...@gmail.com>
> > > > > > >>> > >> > >>>>>>>
> > > > > > >>> > >> > >>>>>>>
> > > > > > >>> > >> > >>>>>>>> Do you think it would be good to start moving
> > the
> > > > > spec
> > > > > > >>> > >> development
> > > > > > >>> > >> > >>>>>>>> into
> > > > > > >>> > >> > >>>>>>>> markdown format, in preparation for
> finalizing
> > > it?
> > > > > > >>> > >> > >>>>>>>>
> > > > > > >>> > >> > >>>>>>>
> > > > > > >>> > >> > >>>>>>> Yes I'll update the numbers for some of the
> > > > examples I
> > > > > > >>> have in
> > > > > > >>> > >> the
> > > > > > >>> > >> > >>>>>>> spec based
> > > > > > >>> > >> > >>>>>>> on the updated header size. Then we should be
> > good
> > > > to
> > > > > go
> > > > > > >>> for
> > > > > > >>> > the
> > > > > > >>> > >> > >>>>>>> markdown format.
> > > > > > >>> > >> > >>>>>>>
> > > > > > >>> > >> > >>>>>>> Thanks everyone!
> > > > > > >>> > >> > >>>>>>>
> > > > > > >>> > >> > >>>>>>>
> > > > > > >>> > >> > >>>>>>>>
> > > > > > >>> > >> > >>>>>>>> Andrew
> > > > > > >>> > >> > >>>>>>>>
> > > > > > >>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR
> <
> > > > > > >>> > >> prtkgaur@gmail.com>
> > > > > > >>> > >> > >>>>>>>> wrote:
> > > > > > >>> > >> > >>>>>>>>
> > > > > > >>> > >> > >>>>>>>>> Hi team,
> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >>> > >> > >>>>>>>>> 1) Andrew
> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >>> > >> > >>>>>>>>> - Thanks for working on test files. My
> PR
> > > did
> > > > > add
> > > > > > >>> all
> > > > > > >>> > the
> > > > > > >>> > >> > test
> > > > > > >>> > >> > >>>>>>>> files I
> > > > > > >>> > >> > >>>>>>>>> used to benchmark on datasets. Maybe we
> > can
> > > > club
> > > > > > it
> > > > > > >>> > >> together.
> > > > > > >>> > >> > >>>>>>>> WIll also
> > > > > > >>> > >> > >>>>>>>>> aid
> > > > > > >>> > >> > >>>>>>>>> cross language testing
> > > > > > >>> > >> > >>>>>>>>> - Kosta Tarasov working on Rust
> > > > implementation.
> > > > > > >>> This is
> > > > > > >>> > >> > great.
> > > > > > >>> > >> > >>>>>>>> Thanks
> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >>> > >> > >>>>>>>>> 2) Antoine
> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >>> > >> > >>>>>>>>> - Thanks a lot for reporting the numbers
> > on
> > > > AMD.
> > > > > > >>> Looks
> > > > > > >>> > >> like
> > > > > > >>> > >> > you
> > > > > > >>> > >> > >>>>>>>> are
> > > > > > >>> > >> > >>>>>>>>> getting 8X the decoding performance of
> > BSS.
> > > > This
> > > > > > is
> > > > > > >>> > >> > amazing!!.
> > > > > > >>> > >> > >>>>>>>>> - Thanks for acknowledging the sampling
> > > > design.
> > > > > > >>> > >> > >>>>>>>>> - I agree with you on Fastlanes. In some
> > > crude
> > > > > > >>> > >> experiments I
> > > > > > >>> > >> > >>>>>>>> didn't get
> > > > > > >>> > >> > >>>>>>>>> a good perf benefit from it on Graviton3
> > > (but
> > > > > > maybe
> > > > > > >>> > there
> > > > > > >>> > >> was
> > > > > > >>> > >> > >>>>>>>> something
> > > > > > >>> > >> > >>>>>>>>> wrong with my implementation).
> > > > > > >>> > >> > >>>>>>>>> - Locking the 16bit exception encoding
> for
> > > the
> > > > > > spec
> > > > > > >>> in
> > > > > > >>> > >> this
> > > > > > >>> > >> > >>>>>>>> case.
> > > > > > >>> > >> > >>>>>>>>> - Awesome I think we have solved for all
> > > open
> > > > > > >>> questions
> > > > > > >>> > >> minus
> > > > > > >>> > >> > >>>>>>>> the
> > > > > > >>> > >> > >>>>>>>>> version byte :). (will get back on this
> > > soon)
> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >>> > >> > >>>>>>>>> 3) Micah
> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >>> > >> > >>>>>>>>> - FastLanes : The current spec does
> allow
> > > for
> > > > > > using
> > > > > > >>> > >> FastLane
> > > > > > >>> > >> > >>>>>>>> with the
> > > > > > >>> > >> > >>>>>>>>> configurable enum value for layout. We
> > > should
> > > > be
> > > > > > >>> able to
> > > > > > >>> > >> > inject
> > > > > > >>> > >> > >>>>>>>> any
> > > > > > >>> > >> > >>>>>>>>> layout
> > > > > > >>> > >> > >>>>>>>>> in the current design.
> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >>> > >> > >>>>>>>>> Working on resolving all remaining open
> > comments
> > > > on
> > > > > > the
> > > > > > >>> spec
> > > > > > >>> > >> this
> > > > > > >>> > >> > >>>>>>>> week.
> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >>> > >> > >>>>>>>>> Best
> > > > > > >>> > >> > >>>>>>>>> Prateek
> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve
> > Loughran <
> > > > > > >>> > >> > >>>>>>>> stevel@cloudera.com>
> > > > > > >>> > >> > >>>>>>>>> wrote:
> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah
> Kornfield
> > <
> > > > > > >>> > >> > >>>>>>>> emkornfield@gmail.com>
> > > > > > >>> > >> > >>>>>>>>>> wrote:
> > > > > > >>> > >> > >>>>>>>>>>
> > > > > > >>> > >> > >>>>>>>>>>>
> > > > > > >>> > >> > >>>>>>>>>>>
> > > > > > >>> > >> > >>>>>>>>>>> It looks like the actual issue described
> for
> > > ORC
> > > > > in
> > > > > > >>> the
> > > > > > >>> > >> paper
> > > > > > >>> > >> > >>>>>>>> is that
> > > > > > >>> > >> > >>>>>>>>> it
> > > > > > >>> > >> > >>>>>>>>>>> has multiple sub-encodings in a batch.
> This
> > > is
> > > > > > >>> different
> > > > > > >>> > >> then
> > > > > > >>> > >> > >>>>>>>> the
> > > > > > >>> > >> > >>>>>>>>> design
> > > > > > >>> > >> > >>>>>>>>>>> proposed here where there is still fixed
> > > > encoding
> > > > > > per
> > > > > > >>> page
> > > > > > >>> > >> in
> > > > > > >>> > >> > >>>>>>>> parquet.
> > > > > > >>> > >> > >>>>>>>>>>> Given reasonably sized pages I don't think
> > > > branch
> > > > > > >>> > >> > >>>>>>>> misprediction should
> > > > > > >>> > >> > >>>>>>>>>> be a
> > > > > > >>> > >> > >>>>>>>>>>> big issue for new encodings. I agree that
> > we
> > > > > should
> > > > > > >>> be
> > > > > > >>> > >> > >>>>>>>> conservative in
> > > > > > >>> > >> > >>>>>>>>>>> general for adding new encodings.
> > > > > > >>> > >> > >>>>>>>>>>>
> > > > > > >>> > >> > >>>>>>>>>>>
> > > > > > >>> > >> > >>>>>>>>>> +1
> > > > > > >>> > >> > >>>>>>>>>>
> > > > > > >>> > >> > >>>>>>>>>
> > > > > > >>> > >> > >>>>>>>>
> > > > > > >>> > >> > >>>>>>>
> > > > > > >>> > >> > >
> > > > > > >>> > >> >
> > > > > > >>> > >> >
> > > > > > >>> > >> >
> > > > > > >>> > >>
> > > > > > >>> > >
> > > > > > >>> >
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Divjot Arora via dev <de...@parquet.apache.org>.
Hi Andrew,
How does this interact with the ongoing versioning discussion? Given that
new encodings are not forward compatible, I would assume ALP is included in
Parquet V<next> (V3?), but it's not clear to me how that works if it's
merged to the parquet-format master branch now. Will we exclude this change
when cutting parquet-format v2.14.0?
-- Div
On Thu, Jul 30, 2026 at 8:51 AM Andrew Lamb <an...@gmail.com> wrote:
> Hello,
>
> As an update, thanks to the epic work from many contributors, especially
> Prateek, I just merged the parquet-format change for ALP[1]. We plan to
> make additional editorial The remaining items I know of are merging the
> examples from parquet-testing[2], and then of course the various
> implementations
>
> Thanks again everyone. This is a great step forward
> Andrew
>
> [1]: https://github.com/apache/parquet-format/pull/557
> [2]: https://github.com/apache/parquet-testing/pull/100
>
> On Tue, Jul 21, 2026 at 4:39 PM Julien Le Dem <ju...@apache.org> wrote:
>
> > It was also in my spam folder... :(
> > I've seen a bunch of apache list emails go to my spam recently.
> >
> > On Tue, Jul 21, 2026 at 10:58 AM Russell Spitzer <
> > russell.spitzer@gmail.com>
> > wrote:
> >
> > > Yes, my hope was that my vote would kick it out of spam but we
> definitely
> > > had two other votes that also went to spam for me.
> > >
> > > For anyone looking the thread is here
> > > https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt
> > >
> > > On Tue, Jul 21, 2026 at 10:57 AM Micah Kornfield <
> emkornfield@gmail.com>
> > > wrote:
> > >
> > > > I've seen other's votes not go to spam, so hopefully people will see
> it
> > > in
> > > > there inboxes (or see this thread and look in there spam).
> > > >
> > > > On Tue, Jul 21, 2026 at 10:52 AM PRATEEK GAUR <pr...@gmail.com>
> > > wrote:
> > > >
> > > > > Okay,
> > > > >
> > > > > I sent out the voting email yesterday and so far I've heard from 3
> > > people
> > > > > that it ended up in their "spam folder". Dev VPN to be blamed for
> > it?.
> > > > >
> > > > > What's the protocol here?
> > > > > I let people hunt for the email in their spam folder or send out
> > > another
> > > > > one?
> > > > >
> > > > > Best
> > > > > Prateek
> > > > >
> > > > > On Sun, Jul 19, 2026 at 10:47 PM PRATEEK GAUR <pr...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > Hi team,
> > > > > >
> > > > > > Vinoo has added the tests which were requested in the last round
> of
> > > > > review.
> > > > > > In his own words.
> > > > > >
> > > > > > "
> > > > > > The extreme value tests Micah wanted are done and pushed. I added
> > > > > coverage
> > > > > > for values that need the full FOR bit width after the frame of
> > > > reference
> > > > > is
> > > > > > applied: 64-bit (the case where the signed max minus min
> > subtraction
> > > > > > overflows), 63-bit (the non-overflow case), and 32-bit for
> floats.
> > > All
> > > > > > lossless with zero exceptions. I also verified them through Arrow
> > C++
> > > > ALP
> > > > > > reader (the local build), so the Java-written extreme columns
> > decode
> > > > > > correctly in C++, bit-exact against the expected values."
> > > > > >
> > > > > > State of the PR (#3397):
> > > > > >
> > > > > > - Merges cleanly with master (I pulled master in and resolved
> > the
> > > > > > conflicts).
> > > > > > - 41 of 44 review threads resolved. The 3 remaining are
> > > intentional:
> > > > > > one deferred perf nit, one that stays open until
> parquet-format
> > > > ships
> > > > > ALP,
> > > > > >
> > > > > > "
> > > > > >
> > > > > > With this both the
> > > > > > 1) C++ PR : https://github.com/apache/arrow/pull/48345/changes
> > > > > > 2) Java PR : https://github.com/apache/parquet-java/pull/3397
> > > > > >
> > > > > > Are cross language tested and range of test scenarios.
> > > > > >
> > > > > > Best
> > > > > > Prateek
> > > > > >
> > > > > >
> > > > > > On Mon, Jul 13, 2026 at 8:38 AM PRATEEK GAUR <prtkgaur@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > >> Thanks Matt,
> > > > > >>
> > > > > >> I'll be on it this week.
> > > > > >>
> > > > > >> Best
> > > > > >> Prateek
> > > > > >>
> > > > > >> On Mon, Jul 13, 2026 at 8:36 AM Matt Topol <
> > zotthewizard@gmail.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >>> There is also the Go implementation submitted by Arnav (
> > > > > >>> https://github.com/apache/arrow-go/pull/704) which is waiting
> > for
> > > > > >>> updates.
> > > > > >>>
> > > > > >>> Just wanted to make sure it didn't get lost here.
> > > > > >>>
> > > > > >>> --Matt
> > > > > >>>
> > > > > >>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK GAUR <
> > prtkgaur@gmail.com>
> > > > > >>> wrote:
> > > > > >>>
> > > > > >>> > Hi all,
> > > > > >>> >
> > > > > >>> > I'd like to share a status update on the ALP encoding effort
> > and
> > > > get
> > > > > a
> > > > > >>> feel
> > > > > >>> > from the community before starting a formal vote.
> > > > > >>> >
> > > > > >>> > We now have the specification plus implementations in both
> > > > languages:
> > > > > >>> >
> > > > > >>> > - Spec: apache/parquet-format#557 — GH-533 Add
> ALP
> > > > > encoding
> > > > > >>> > specification
> > > > > >>> > - C++ (Arrow): apache/arrow#48345 — GH-48701
> > [C++][Parquet]
> > > > Add
> > > > > >>> ALPpd
> > > > > >>> > encoding
> > > > > >>> > - Java: apache/parquet-java#3397 — Parquet Java
> ALP
> > > > > >>> > Implementation (by Vinoo)
> > > > > >>> >
> > > > > >>> > Current state:
> > > > > >>> >
> > > > > >>> > - The C++ implementation has been through 3-4 rounds of
> > > review.
> > > > > >>> > - The Java implementation has been tracking well and has
> > had
> > > > > >>> initial
> > > > > >>> > reviews from contributors, with no major remaining
> > issues.
> > > > > >>> > - Cross-language compatibility tests are passing: the
> Arrow
> > > C++
> > > > > >>> decoder
> > > > > >>> > reads Java-written data bit-exactly across ~1.56M
> values
> > > and
> > > > 18
> > > > > >>> > fixtures, covering V1 and V2 pages, multiple vector
> > sizes,
> > > > and
> > > > > >>> > several
> > > > > >>> > real datasets — zero mismatches.
> > > > > >>> >
> > > > > >>> > As raised in the java-pr we will be adding coverage for
> > extreme
> > > > > values
> > > > > >>> > (those requiring
> > > > > >>> > 63-64 bits after FOR is applied) before we close a vote.
> We'll
> > > aim
> > > > > to
> > > > > >>> get
> > > > > >>> > that
> > > > > >>> > done in parallel.
> > > > > >>> >
> > > > > >>> > Plan: unless there are objections, I intend to start a vote
> at
> > > the
> > > > > end
> > > > > >>> of
> > > > > >>> > this week/early
> > > > > >>> > next week.
> > > > > >>> >
> > > > > >>> > Thanks
> > > > > >>> > Prateek
> > > > > >>> >
> > > > > >>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <
> > prtkgaur@gmail.com>
> > > > > >>> wrote:
> > > > > >>> >
> > > > > >>> > > Hi Team,
> > > > > >>> > >
> > > > > >>> > > Just wanted to provide some updates on ALP.
> > > > > >>> > > Micah and I have done 3-4 rounds of review for the c++ PR :
> > > > > >>> > > https://github.com/apache/arrow/pull/48345
> > > > > >>> > >
> > > > > >>> > > For the Java implementation Vinoo has been working on
> > following
> > > > PR
> > > > > :
> > > > > >>> > > https://github.com/apache/parquet-java/pull/3397
> > > > > >>> > >
> > > > > >>> > > Best
> > > > > >>> > > Prateek
> > > > > >>> > >
> > > > > >>> > > On Tue, May 5, 2026 at 1:50 PM Micah Kornfield <
> > > > > >>> emkornfield@gmail.com>
> > > > > >>> > > wrote:
> > > > > >>> > >
> > > > > >>> > >> Hi Antoine,
> > > > > >>> > >>
> > > > > >>> > >> > Apologies if the question was already asked, but should
> we
> > > > care
> > > > > >>> about
> > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than
> > > > FLOAT16
> > > > > +
> > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> > > > > >>> > >>
> > > > > >>> > >> It was. We thought we could defer it for the following
> > > reasons:
> > > > > >>> > >> 1. It's not clear there are a lot of easy reference
> > datasets
> > > to
> > > > > >>> test
> > > > > >>> > its
> > > > > >>> > >> effectiveness. It does look like there might be one or
> two
> > on
> > > > > >>> > huggingface
> > > > > >>> > >> (e.g. https://huggingface.co/datasets/kikitora/curdie).
> > > > > >>> > >> 2. It seemed likely that float 16 was more likely used
> for
> > > > values
> > > > > >>> that
> > > > > >>> > >> were less likely to reduce to decimal values.
> > > > > >>> > >> 3. It could be added as an extension later if needed.
> > > > > >>> > >>
> > > > > >>> > >> Cheers,
> > > > > >>> > >> Micah
> > > > > >>> > >>
> > > > > >>> > >>
> > > > > >>> > >> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <
> > > > antoine@python.org
> > > > > >
> > > > > >>> > wrote:
> > > > > >>> > >>
> > > > > >>> > >> >
> > > > > >>> > >> > Hello,
> > > > > >>> > >> >
> > > > > >>> > >> > Apologies if the question was already asked, but should
> we
> > > > care
> > > > > >>> about
> > > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than
> > > > FLOAT16
> > > > > +
> > > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> > > > > >>> > >> >
> > > > > >>> > >> > Regards
> > > > > >>> > >> >
> > > > > >>> > >> > Antoine.
> > > > > >>> > >> >
> > > > > >>> > >> >
> > > > > >>> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
> > > > > >>> > >> > > Thanks Andrew and Micah for review feedback on the two
> > > PR's
> > > > > >>> > >> > > 1) (c++ arrow repo)
> > > > > >>> > >> https://github.com/apache/arrow/pull/48345/changes
> > > > > >>> > >> > > 2) (parquet-format repo)
> > > > > >>> > >> > https://github.com/apache/parquet-format/pull/557
> > > > > >>> > >> > >
> > > > > >>> > >> > > I have addressed all (unless I missed something)
> > comments
> > > on
> > > > > >>> the two
> > > > > >>> > >> > PR's.
> > > > > >>> > >> > >
> > > > > >>> > >> > > Best
> > > > > >>> > >> > > Prateek
> > > > > >>> > >> > >
> > > > > >>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <
> > > > > >>> prtkgaur@gmail.com>
> > > > > >>> > >> wrote:
> > > > > >>> > >> > >
> > > > > >>> > >> > >> Thanks Andrew and Micah.
> > > > > >>> > >> > >>
> > > > > >>> > >> > >> `fair amount of feedback on at least the
> > implementations`
> > > > > >>> > >> > >> For the c++ I have already started addressing the
> > > > feedback, I
> > > > > >>> > should
> > > > > >>> > >> be
> > > > > >>> > >> > >> done with that Monday/Tuesday.
> > > > > >>> > >> > >> I think Vinoo too has been making good progress on
> the
> > > Java
> > > > > >>> > >> > implementation.
> > > > > >>> > >> > >>
> > > > > >>> > >> > >> Best
> > > > > >>> > >> > >> Prateek
> > > > > >>> > >> > >>
> > > > > >>> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <
> > > > > >>> > andrewlamb11@gmail.com
> > > > > >>> > >> >
> > > > > >>> > >> > >> wrote:
> > > > > >>> > >> > >>
> > > > > >>> > >> > >>> Got it. Thank you for the clarification -- I will
> try
> > > and
> > > > > look
> > > > > >>> > into
> > > > > >>> > >> the
> > > > > >>> > >> > >>> spec and the Rust implementation[1] in this next
> week
> > > > > >>> > >> > >>>
> > > > > >>> > >> > >>> [1]: https://github.com/apache/arrow-rs/pull/9372
> > > > > >>> > >> > >>>
> > > > > >>> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <
> > > > > >>> > >> > emkornfield@gmail.com>
> > > > > >>> > >> > >>> wrote:
> > > > > >>> > >> > >>>
> > > > > >>> > >> > >>>> Hi Andrew,
> > > > > >>> > >> > >>>> I think there is a fair amount of feedback on at
> > least
> > > > the
> > > > > >>> > >> > >>>> implementations, typically I think we've waited
> till
> > > they
> > > > > are
> > > > > >>> > >> close to
> > > > > >>> > >> > >>>> mergeable before a final vote. Otherwise I agree
> we
> > > are
> > > > > very
> > > > > >>> > >> close.
> > > > > >>> > >> > >>>>
> > > > > >>> > >> > >>>> -Micah
> > > > > >>> > >> > >>>>
> > > > > >>> > >> > >>>> On Saturday, April 25, 2026, Andrew Lamb <
> > > > > >>> andrewlamb11@gmail.com
> > > > > >>> > >
> > > > > >>> > >> > wrote:
> > > > > >>> > >> > >>>>
> > > > > >>> > >> > >>>>> Thanks Prateek,
> > > > > >>> > >> > >>>>>
> > > > > >>> > >> > >>>>> I think from this content it looks to me like we
> are
> > > > ready
> > > > > >>> to
> > > > > >>> > >> start a
> > > > > >>> > >> > >>>>> vote to explicitly accept ALP into Parquet
> > > > > >>> > >> > >>>>>
> > > > > >>> > >> > >>>>> Does anyone know of a reason we should postpone it
> > for
> > > > > >>> longer?
> > > > > >>> > >> > >>>>> Perhaps someone needs some more time to review?
> > > > > >>> > >> > >>>>>
> > > > > >>> > >> > >>>>> Andrew
> > > > > >>> > >> > >>>>>
> > > > > >>> > >> > >>>>>
> > > > > >>> > >> > >>>>>
> > > > > >>> > >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <
> > > > > >>> > prtkgaur@gmail.com>
> > > > > >>> > >> > >>>>> wrote:
> > > > > >>> > >> > >>>>>
> > > > > >>> > >> > >>>>>> Hi team,
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>> Hope everyone is doing well. I got a chance to
> work
> > > > > >>> through all
> > > > > >>> > >> the
> > > > > >>> > >> > >>>>>> remaining feedback and update the spec doc. Here
> > are
> > > > the
> > > > > >>> new
> > > > > >>> > >> > artifacts
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>> 1) Spec document :
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> >
> > > > > >>> > >>
> > > > > >>> >
> > > > > >>>
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>> 2) Spec document in parquet format repo :
> > > > > >>> > >> > >>>>>>
> https://github.com/apache/parquet-format/pull/557
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>> 3) Alp implementation in arrow c++ repo :
> > > > > >>> > >> > >>>>>>
> https://github.com/apache/arrow/pull/48345/changes
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>> 4) Alp implementation in parquet-java repo : Work
> > for
> > > > > >>> Vinoo and
> > > > > >>> > >> > Julien
> > > > > >>> > >> > >>>>>>
> https://github.com/apache/parquet-java/pull/3397
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>> 5) PR with test and benchmarking artifacts in
> > > > > >>> parquet-testing
> > > > > >>> > >> repo :
> > > > > >>> > >> > >>>>>>
> https://github.com/apache/parquet-testing/pull/100
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>> And
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>> - Go : Arnav just submitted an in progress
> > > > > >>> implementation
> > > > > >>> > in
> > > > > >>> > >> Go.
> > > > > >>> > >> > >>>>>> https://github.com/apache/arrow-go/pull/704
> (I
> > > > > haven't
> > > > > >>> > >> started
> > > > > >>> > >> > >>>>>> looking at it yet)
> > > > > >>> > >> > >>>>>> - Rust : I remember Andrew mentioned that
> this
> > > work
> > > > > is
> > > > > >>> also
> > > > > >>> > >> in
> > > > > >>> > >> > >>>>>> progress (So 4 languages!)
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>> *Arrow C++ implementation *
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>> The PR is out and was also used by Antoine to
> > report
> > > > the
> > > > > >>> > numbers
> > > > > >>> > >> as
> > > > > >>> > >> > >>>>>> reported here. Micah and Konstantin have given 1
> > > round
> > > > of
> > > > > >>> > >> feedback
> > > > > >>> > >> > >>>>>> and I'm addressing them today. Please note that
> the
> > > > > default
> > > > > >>> > >> > >>>>>> optimization flag for compiling is O2 and not
> Q3. I
> > > got
> > > > > >>> around
> > > > > >>> > >> 70%
> > > > > >>> > >> > >>>>>> performance improvement in the decoding speed
> when
> > > > using
> > > > > >>> the O3
> > > > > >>> > >> > flag.
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>> *Parqet-MR Java implementation (working with
> Vinoo
> > > and
> > > > > >>> Julien)
> > > > > >>> > >> and
> > > > > >>> > >> > **Cross
> > > > > >>> > >> > >>>>>> Language testing*
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>> Let me know if you have any questions or
> > > feedback.
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>> Now pasting some performance numbers
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>> Table 1: C++ ALP Double Decode — Spotify
> Columns
> > > > > >>> (Graviton
> > > > > >>> > 3,
> > > > > >>> > >> ARM
> > > > > >>> > >> > >>>>>> Neoverse V1)
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>>
> > > > > >>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>> │ Column │ -O2 (MB/s) │ -O3
> > (MB/s) │
> > > > > >>> Speedup │
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>>
> > > > > >>> ├──────────────────┼──────────────┼──────────────┼─────────┤
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>> │ valence │ 3,155 │ 5,523
> > │
> > > > > >>> 1.75x │
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>> │ danceability │ 3,233 │ 5,685
> > │
> > > > > >>> 1.76x │
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>> │ energy │ 3,197 │ 5,652
> > │
> > > > > >>> 1.77x │
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>> │ loudness │ 3,186 │ 5,473
> > │
> > > > > >>> 1.72x │
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>>
> > > > > >>> └──────────────────┴──────────────┴──────────────┴─────────┘
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <
> > > > > >>> > prtkgaur@gmail.com
> > > > > >>> > >> >
> > > > > >>> > >> > >>>>>> wrote:
> > > > > >>> > >> > >>>>>>
> > > > > >>> > >> > >>>>>>> @Micah Kornfield <em...@gmail.com> : Got
> > it.
> > > > > >>> > >> > >>>>>>>
> > > > > >>> > >> > >>>>>>> @Andrew Lamb <an...@gmail.com>
> > > > > >>> > >> > >>>>>>>
> > > > > >>> > >> > >>>>>>>
> > > > > >>> > >> > >>>>>>>> Do you think it would be good to start moving
> the
> > > > spec
> > > > > >>> > >> development
> > > > > >>> > >> > >>>>>>>> into
> > > > > >>> > >> > >>>>>>>> markdown format, in preparation for finalizing
> > it?
> > > > > >>> > >> > >>>>>>>>
> > > > > >>> > >> > >>>>>>>
> > > > > >>> > >> > >>>>>>> Yes I'll update the numbers for some of the
> > > examples I
> > > > > >>> have in
> > > > > >>> > >> the
> > > > > >>> > >> > >>>>>>> spec based
> > > > > >>> > >> > >>>>>>> on the updated header size. Then we should be
> good
> > > to
> > > > go
> > > > > >>> for
> > > > > >>> > the
> > > > > >>> > >> > >>>>>>> markdown format.
> > > > > >>> > >> > >>>>>>>
> > > > > >>> > >> > >>>>>>> Thanks everyone!
> > > > > >>> > >> > >>>>>>>
> > > > > >>> > >> > >>>>>>>
> > > > > >>> > >> > >>>>>>>>
> > > > > >>> > >> > >>>>>>>> Andrew
> > > > > >>> > >> > >>>>>>>>
> > > > > >>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <
> > > > > >>> > >> prtkgaur@gmail.com>
> > > > > >>> > >> > >>>>>>>> wrote:
> > > > > >>> > >> > >>>>>>>>
> > > > > >>> > >> > >>>>>>>>> Hi team,
> > > > > >>> > >> > >>>>>>>>>
> > > > > >>> > >> > >>>>>>>>> 1) Andrew
> > > > > >>> > >> > >>>>>>>>>
> > > > > >>> > >> > >>>>>>>>> - Thanks for working on test files. My PR
> > did
> > > > add
> > > > > >>> all
> > > > > >>> > the
> > > > > >>> > >> > test
> > > > > >>> > >> > >>>>>>>> files I
> > > > > >>> > >> > >>>>>>>>> used to benchmark on datasets. Maybe we
> can
> > > club
> > > > > it
> > > > > >>> > >> together.
> > > > > >>> > >> > >>>>>>>> WIll also
> > > > > >>> > >> > >>>>>>>>> aid
> > > > > >>> > >> > >>>>>>>>> cross language testing
> > > > > >>> > >> > >>>>>>>>> - Kosta Tarasov working on Rust
> > > implementation.
> > > > > >>> This is
> > > > > >>> > >> > great.
> > > > > >>> > >> > >>>>>>>> Thanks
> > > > > >>> > >> > >>>>>>>>>
> > > > > >>> > >> > >>>>>>>>>
> > > > > >>> > >> > >>>>>>>>> 2) Antoine
> > > > > >>> > >> > >>>>>>>>>
> > > > > >>> > >> > >>>>>>>>> - Thanks a lot for reporting the numbers
> on
> > > AMD.
> > > > > >>> Looks
> > > > > >>> > >> like
> > > > > >>> > >> > you
> > > > > >>> > >> > >>>>>>>> are
> > > > > >>> > >> > >>>>>>>>> getting 8X the decoding performance of
> BSS.
> > > This
> > > > > is
> > > > > >>> > >> > amazing!!.
> > > > > >>> > >> > >>>>>>>>> - Thanks for acknowledging the sampling
> > > design.
> > > > > >>> > >> > >>>>>>>>> - I agree with you on Fastlanes. In some
> > crude
> > > > > >>> > >> experiments I
> > > > > >>> > >> > >>>>>>>> didn't get
> > > > > >>> > >> > >>>>>>>>> a good perf benefit from it on Graviton3
> > (but
> > > > > maybe
> > > > > >>> > there
> > > > > >>> > >> was
> > > > > >>> > >> > >>>>>>>> something
> > > > > >>> > >> > >>>>>>>>> wrong with my implementation).
> > > > > >>> > >> > >>>>>>>>> - Locking the 16bit exception encoding for
> > the
> > > > > spec
> > > > > >>> in
> > > > > >>> > >> this
> > > > > >>> > >> > >>>>>>>> case.
> > > > > >>> > >> > >>>>>>>>> - Awesome I think we have solved for all
> > open
> > > > > >>> questions
> > > > > >>> > >> minus
> > > > > >>> > >> > >>>>>>>> the
> > > > > >>> > >> > >>>>>>>>> version byte :). (will get back on this
> > soon)
> > > > > >>> > >> > >>>>>>>>>
> > > > > >>> > >> > >>>>>>>>>
> > > > > >>> > >> > >>>>>>>>> 3) Micah
> > > > > >>> > >> > >>>>>>>>>
> > > > > >>> > >> > >>>>>>>>> - FastLanes : The current spec does allow
> > for
> > > > > using
> > > > > >>> > >> FastLane
> > > > > >>> > >> > >>>>>>>> with the
> > > > > >>> > >> > >>>>>>>>> configurable enum value for layout. We
> > should
> > > be
> > > > > >>> able to
> > > > > >>> > >> > inject
> > > > > >>> > >> > >>>>>>>> any
> > > > > >>> > >> > >>>>>>>>> layout
> > > > > >>> > >> > >>>>>>>>> in the current design.
> > > > > >>> > >> > >>>>>>>>>
> > > > > >>> > >> > >>>>>>>>>
> > > > > >>> > >> > >>>>>>>>> Working on resolving all remaining open
> comments
> > > on
> > > > > the
> > > > > >>> spec
> > > > > >>> > >> this
> > > > > >>> > >> > >>>>>>>> week.
> > > > > >>> > >> > >>>>>>>>>
> > > > > >>> > >> > >>>>>>>>> Best
> > > > > >>> > >> > >>>>>>>>> Prateek
> > > > > >>> > >> > >>>>>>>>>
> > > > > >>> > >> > >>>>>>>>>
> > > > > >>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve
> Loughran <
> > > > > >>> > >> > >>>>>>>> stevel@cloudera.com>
> > > > > >>> > >> > >>>>>>>>> wrote:
> > > > > >>> > >> > >>>>>>>>>
> > > > > >>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah Kornfield
> <
> > > > > >>> > >> > >>>>>>>> emkornfield@gmail.com>
> > > > > >>> > >> > >>>>>>>>>> wrote:
> > > > > >>> > >> > >>>>>>>>>>
> > > > > >>> > >> > >>>>>>>>>>>
> > > > > >>> > >> > >>>>>>>>>>>
> > > > > >>> > >> > >>>>>>>>>>> It looks like the actual issue described for
> > ORC
> > > > in
> > > > > >>> the
> > > > > >>> > >> paper
> > > > > >>> > >> > >>>>>>>> is that
> > > > > >>> > >> > >>>>>>>>> it
> > > > > >>> > >> > >>>>>>>>>>> has multiple sub-encodings in a batch. This
> > is
> > > > > >>> different
> > > > > >>> > >> then
> > > > > >>> > >> > >>>>>>>> the
> > > > > >>> > >> > >>>>>>>>> design
> > > > > >>> > >> > >>>>>>>>>>> proposed here where there is still fixed
> > > encoding
> > > > > per
> > > > > >>> page
> > > > > >>> > >> in
> > > > > >>> > >> > >>>>>>>> parquet.
> > > > > >>> > >> > >>>>>>>>>>> Given reasonably sized pages I don't think
> > > branch
> > > > > >>> > >> > >>>>>>>> misprediction should
> > > > > >>> > >> > >>>>>>>>>> be a
> > > > > >>> > >> > >>>>>>>>>>> big issue for new encodings. I agree that
> we
> > > > should
> > > > > >>> be
> > > > > >>> > >> > >>>>>>>> conservative in
> > > > > >>> > >> > >>>>>>>>>>> general for adding new encodings.
> > > > > >>> > >> > >>>>>>>>>>>
> > > > > >>> > >> > >>>>>>>>>>>
> > > > > >>> > >> > >>>>>>>>>> +1
> > > > > >>> > >> > >>>>>>>>>>
> > > > > >>> > >> > >>>>>>>>>
> > > > > >>> > >> > >>>>>>>>
> > > > > >>> > >> > >>>>>>>
> > > > > >>> > >> > >
> > > > > >>> > >> >
> > > > > >>> > >> >
> > > > > >>> > >> >
> > > > > >>> > >>
> > > > > >>> > >
> > > > > >>> >
> > > > > >>>
> > > > > >>
> > > > >
> > > >
> > >
> >
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Andrew Lamb <an...@gmail.com>.
Hello,
As an update, thanks to the epic work from many contributors, especially
Prateek, I just merged the parquet-format change for ALP[1]. We plan to
make additional editorial The remaining items I know of are merging the
examples from parquet-testing[2], and then of course the various
implementations
Thanks again everyone. This is a great step forward
Andrew
[1]: https://github.com/apache/parquet-format/pull/557
[2]: https://github.com/apache/parquet-testing/pull/100
On Tue, Jul 21, 2026 at 4:39 PM Julien Le Dem <ju...@apache.org> wrote:
> It was also in my spam folder... :(
> I've seen a bunch of apache list emails go to my spam recently.
>
> On Tue, Jul 21, 2026 at 10:58 AM Russell Spitzer <
> russell.spitzer@gmail.com>
> wrote:
>
> > Yes, my hope was that my vote would kick it out of spam but we definitely
> > had two other votes that also went to spam for me.
> >
> > For anyone looking the thread is here
> > https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt
> >
> > On Tue, Jul 21, 2026 at 10:57 AM Micah Kornfield <em...@gmail.com>
> > wrote:
> >
> > > I've seen other's votes not go to spam, so hopefully people will see it
> > in
> > > there inboxes (or see this thread and look in there spam).
> > >
> > > On Tue, Jul 21, 2026 at 10:52 AM PRATEEK GAUR <pr...@gmail.com>
> > wrote:
> > >
> > > > Okay,
> > > >
> > > > I sent out the voting email yesterday and so far I've heard from 3
> > people
> > > > that it ended up in their "spam folder". Dev VPN to be blamed for
> it?.
> > > >
> > > > What's the protocol here?
> > > > I let people hunt for the email in their spam folder or send out
> > another
> > > > one?
> > > >
> > > > Best
> > > > Prateek
> > > >
> > > > On Sun, Jul 19, 2026 at 10:47 PM PRATEEK GAUR <pr...@gmail.com>
> > > wrote:
> > > >
> > > > > Hi team,
> > > > >
> > > > > Vinoo has added the tests which were requested in the last round of
> > > > review.
> > > > > In his own words.
> > > > >
> > > > > "
> > > > > The extreme value tests Micah wanted are done and pushed. I added
> > > > coverage
> > > > > for values that need the full FOR bit width after the frame of
> > > reference
> > > > is
> > > > > applied: 64-bit (the case where the signed max minus min
> subtraction
> > > > > overflows), 63-bit (the non-overflow case), and 32-bit for floats.
> > All
> > > > > lossless with zero exceptions. I also verified them through Arrow
> C++
> > > ALP
> > > > > reader (the local build), so the Java-written extreme columns
> decode
> > > > > correctly in C++, bit-exact against the expected values."
> > > > >
> > > > > State of the PR (#3397):
> > > > >
> > > > > - Merges cleanly with master (I pulled master in and resolved
> the
> > > > > conflicts).
> > > > > - 41 of 44 review threads resolved. The 3 remaining are
> > intentional:
> > > > > one deferred perf nit, one that stays open until parquet-format
> > > ships
> > > > ALP,
> > > > >
> > > > > "
> > > > >
> > > > > With this both the
> > > > > 1) C++ PR : https://github.com/apache/arrow/pull/48345/changes
> > > > > 2) Java PR : https://github.com/apache/parquet-java/pull/3397
> > > > >
> > > > > Are cross language tested and range of test scenarios.
> > > > >
> > > > > Best
> > > > > Prateek
> > > > >
> > > > >
> > > > > On Mon, Jul 13, 2026 at 8:38 AM PRATEEK GAUR <pr...@gmail.com>
> > > wrote:
> > > > >
> > > > >> Thanks Matt,
> > > > >>
> > > > >> I'll be on it this week.
> > > > >>
> > > > >> Best
> > > > >> Prateek
> > > > >>
> > > > >> On Mon, Jul 13, 2026 at 8:36 AM Matt Topol <
> zotthewizard@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >>> There is also the Go implementation submitted by Arnav (
> > > > >>> https://github.com/apache/arrow-go/pull/704) which is waiting
> for
> > > > >>> updates.
> > > > >>>
> > > > >>> Just wanted to make sure it didn't get lost here.
> > > > >>>
> > > > >>> --Matt
> > > > >>>
> > > > >>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK GAUR <
> prtkgaur@gmail.com>
> > > > >>> wrote:
> > > > >>>
> > > > >>> > Hi all,
> > > > >>> >
> > > > >>> > I'd like to share a status update on the ALP encoding effort
> and
> > > get
> > > > a
> > > > >>> feel
> > > > >>> > from the community before starting a formal vote.
> > > > >>> >
> > > > >>> > We now have the specification plus implementations in both
> > > languages:
> > > > >>> >
> > > > >>> > - Spec: apache/parquet-format#557 — GH-533 Add ALP
> > > > encoding
> > > > >>> > specification
> > > > >>> > - C++ (Arrow): apache/arrow#48345 — GH-48701
> [C++][Parquet]
> > > Add
> > > > >>> ALPpd
> > > > >>> > encoding
> > > > >>> > - Java: apache/parquet-java#3397 — Parquet Java ALP
> > > > >>> > Implementation (by Vinoo)
> > > > >>> >
> > > > >>> > Current state:
> > > > >>> >
> > > > >>> > - The C++ implementation has been through 3-4 rounds of
> > review.
> > > > >>> > - The Java implementation has been tracking well and has
> had
> > > > >>> initial
> > > > >>> > reviews from contributors, with no major remaining
> issues.
> > > > >>> > - Cross-language compatibility tests are passing: the Arrow
> > C++
> > > > >>> decoder
> > > > >>> > reads Java-written data bit-exactly across ~1.56M values
> > and
> > > 18
> > > > >>> > fixtures, covering V1 and V2 pages, multiple vector
> sizes,
> > > and
> > > > >>> > several
> > > > >>> > real datasets — zero mismatches.
> > > > >>> >
> > > > >>> > As raised in the java-pr we will be adding coverage for
> extreme
> > > > values
> > > > >>> > (those requiring
> > > > >>> > 63-64 bits after FOR is applied) before we close a vote. We'll
> > aim
> > > > to
> > > > >>> get
> > > > >>> > that
> > > > >>> > done in parallel.
> > > > >>> >
> > > > >>> > Plan: unless there are objections, I intend to start a vote at
> > the
> > > > end
> > > > >>> of
> > > > >>> > this week/early
> > > > >>> > next week.
> > > > >>> >
> > > > >>> > Thanks
> > > > >>> > Prateek
> > > > >>> >
> > > > >>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <
> prtkgaur@gmail.com>
> > > > >>> wrote:
> > > > >>> >
> > > > >>> > > Hi Team,
> > > > >>> > >
> > > > >>> > > Just wanted to provide some updates on ALP.
> > > > >>> > > Micah and I have done 3-4 rounds of review for the c++ PR :
> > > > >>> > > https://github.com/apache/arrow/pull/48345
> > > > >>> > >
> > > > >>> > > For the Java implementation Vinoo has been working on
> following
> > > PR
> > > > :
> > > > >>> > > https://github.com/apache/parquet-java/pull/3397
> > > > >>> > >
> > > > >>> > > Best
> > > > >>> > > Prateek
> > > > >>> > >
> > > > >>> > > On Tue, May 5, 2026 at 1:50 PM Micah Kornfield <
> > > > >>> emkornfield@gmail.com>
> > > > >>> > > wrote:
> > > > >>> > >
> > > > >>> > >> Hi Antoine,
> > > > >>> > >>
> > > > >>> > >> > Apologies if the question was already asked, but should we
> > > care
> > > > >>> about
> > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than
> > > FLOAT16
> > > > +
> > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> > > > >>> > >>
> > > > >>> > >> It was. We thought we could defer it for the following
> > reasons:
> > > > >>> > >> 1. It's not clear there are a lot of easy reference
> datasets
> > to
> > > > >>> test
> > > > >>> > its
> > > > >>> > >> effectiveness. It does look like there might be one or two
> on
> > > > >>> > huggingface
> > > > >>> > >> (e.g. https://huggingface.co/datasets/kikitora/curdie).
> > > > >>> > >> 2. It seemed likely that float 16 was more likely used for
> > > values
> > > > >>> that
> > > > >>> > >> were less likely to reduce to decimal values.
> > > > >>> > >> 3. It could be added as an extension later if needed.
> > > > >>> > >>
> > > > >>> > >> Cheers,
> > > > >>> > >> Micah
> > > > >>> > >>
> > > > >>> > >>
> > > > >>> > >> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <
> > > antoine@python.org
> > > > >
> > > > >>> > wrote:
> > > > >>> > >>
> > > > >>> > >> >
> > > > >>> > >> > Hello,
> > > > >>> > >> >
> > > > >>> > >> > Apologies if the question was already asked, but should we
> > > care
> > > > >>> about
> > > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than
> > > FLOAT16
> > > > +
> > > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> > > > >>> > >> >
> > > > >>> > >> > Regards
> > > > >>> > >> >
> > > > >>> > >> > Antoine.
> > > > >>> > >> >
> > > > >>> > >> >
> > > > >>> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
> > > > >>> > >> > > Thanks Andrew and Micah for review feedback on the two
> > PR's
> > > > >>> > >> > > 1) (c++ arrow repo)
> > > > >>> > >> https://github.com/apache/arrow/pull/48345/changes
> > > > >>> > >> > > 2) (parquet-format repo)
> > > > >>> > >> > https://github.com/apache/parquet-format/pull/557
> > > > >>> > >> > >
> > > > >>> > >> > > I have addressed all (unless I missed something)
> comments
> > on
> > > > >>> the two
> > > > >>> > >> > PR's.
> > > > >>> > >> > >
> > > > >>> > >> > > Best
> > > > >>> > >> > > Prateek
> > > > >>> > >> > >
> > > > >>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <
> > > > >>> prtkgaur@gmail.com>
> > > > >>> > >> wrote:
> > > > >>> > >> > >
> > > > >>> > >> > >> Thanks Andrew and Micah.
> > > > >>> > >> > >>
> > > > >>> > >> > >> `fair amount of feedback on at least the
> implementations`
> > > > >>> > >> > >> For the c++ I have already started addressing the
> > > feedback, I
> > > > >>> > should
> > > > >>> > >> be
> > > > >>> > >> > >> done with that Monday/Tuesday.
> > > > >>> > >> > >> I think Vinoo too has been making good progress on the
> > Java
> > > > >>> > >> > implementation.
> > > > >>> > >> > >>
> > > > >>> > >> > >> Best
> > > > >>> > >> > >> Prateek
> > > > >>> > >> > >>
> > > > >>> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <
> > > > >>> > andrewlamb11@gmail.com
> > > > >>> > >> >
> > > > >>> > >> > >> wrote:
> > > > >>> > >> > >>
> > > > >>> > >> > >>> Got it. Thank you for the clarification -- I will try
> > and
> > > > look
> > > > >>> > into
> > > > >>> > >> the
> > > > >>> > >> > >>> spec and the Rust implementation[1] in this next week
> > > > >>> > >> > >>>
> > > > >>> > >> > >>> [1]: https://github.com/apache/arrow-rs/pull/9372
> > > > >>> > >> > >>>
> > > > >>> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <
> > > > >>> > >> > emkornfield@gmail.com>
> > > > >>> > >> > >>> wrote:
> > > > >>> > >> > >>>
> > > > >>> > >> > >>>> Hi Andrew,
> > > > >>> > >> > >>>> I think there is a fair amount of feedback on at
> least
> > > the
> > > > >>> > >> > >>>> implementations, typically I think we've waited till
> > they
> > > > are
> > > > >>> > >> close to
> > > > >>> > >> > >>>> mergeable before a final vote. Otherwise I agree we
> > are
> > > > very
> > > > >>> > >> close.
> > > > >>> > >> > >>>>
> > > > >>> > >> > >>>> -Micah
> > > > >>> > >> > >>>>
> > > > >>> > >> > >>>> On Saturday, April 25, 2026, Andrew Lamb <
> > > > >>> andrewlamb11@gmail.com
> > > > >>> > >
> > > > >>> > >> > wrote:
> > > > >>> > >> > >>>>
> > > > >>> > >> > >>>>> Thanks Prateek,
> > > > >>> > >> > >>>>>
> > > > >>> > >> > >>>>> I think from this content it looks to me like we are
> > > ready
> > > > >>> to
> > > > >>> > >> start a
> > > > >>> > >> > >>>>> vote to explicitly accept ALP into Parquet
> > > > >>> > >> > >>>>>
> > > > >>> > >> > >>>>> Does anyone know of a reason we should postpone it
> for
> > > > >>> longer?
> > > > >>> > >> > >>>>> Perhaps someone needs some more time to review?
> > > > >>> > >> > >>>>>
> > > > >>> > >> > >>>>> Andrew
> > > > >>> > >> > >>>>>
> > > > >>> > >> > >>>>>
> > > > >>> > >> > >>>>>
> > > > >>> > >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <
> > > > >>> > prtkgaur@gmail.com>
> > > > >>> > >> > >>>>> wrote:
> > > > >>> > >> > >>>>>
> > > > >>> > >> > >>>>>> Hi team,
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>> Hope everyone is doing well. I got a chance to work
> > > > >>> through all
> > > > >>> > >> the
> > > > >>> > >> > >>>>>> remaining feedback and update the spec doc. Here
> are
> > > the
> > > > >>> new
> > > > >>> > >> > artifacts
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>> 1) Spec document :
> > > > >>> > >> > >>>>>>
> > > > >>> > >> >
> > > > >>> > >>
> > > > >>> >
> > > > >>>
> > > >
> > >
> >
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>> 2) Spec document in parquet format repo :
> > > > >>> > >> > >>>>>> https://github.com/apache/parquet-format/pull/557
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>> 3) Alp implementation in arrow c++ repo :
> > > > >>> > >> > >>>>>> https://github.com/apache/arrow/pull/48345/changes
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>> 4) Alp implementation in parquet-java repo : Work
> for
> > > > >>> Vinoo and
> > > > >>> > >> > Julien
> > > > >>> > >> > >>>>>> https://github.com/apache/parquet-java/pull/3397
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>> 5) PR with test and benchmarking artifacts in
> > > > >>> parquet-testing
> > > > >>> > >> repo :
> > > > >>> > >> > >>>>>> https://github.com/apache/parquet-testing/pull/100
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>> And
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>> - Go : Arnav just submitted an in progress
> > > > >>> implementation
> > > > >>> > in
> > > > >>> > >> Go.
> > > > >>> > >> > >>>>>> https://github.com/apache/arrow-go/pull/704 (I
> > > > haven't
> > > > >>> > >> started
> > > > >>> > >> > >>>>>> looking at it yet)
> > > > >>> > >> > >>>>>> - Rust : I remember Andrew mentioned that this
> > work
> > > > is
> > > > >>> also
> > > > >>> > >> in
> > > > >>> > >> > >>>>>> progress (So 4 languages!)
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>> *Arrow C++ implementation *
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>> The PR is out and was also used by Antoine to
> report
> > > the
> > > > >>> > numbers
> > > > >>> > >> as
> > > > >>> > >> > >>>>>> reported here. Micah and Konstantin have given 1
> > round
> > > of
> > > > >>> > >> feedback
> > > > >>> > >> > >>>>>> and I'm addressing them today. Please note that the
> > > > default
> > > > >>> > >> > >>>>>> optimization flag for compiling is O2 and not Q3. I
> > got
> > > > >>> around
> > > > >>> > >> 70%
> > > > >>> > >> > >>>>>> performance improvement in the decoding speed when
> > > using
> > > > >>> the O3
> > > > >>> > >> > flag.
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>> *Parqet-MR Java implementation (working with Vinoo
> > and
> > > > >>> Julien)
> > > > >>> > >> and
> > > > >>> > >> > **Cross
> > > > >>> > >> > >>>>>> Language testing*
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>> Let me know if you have any questions or
> > feedback.
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>> Now pasting some performance numbers
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>> Table 1: C++ ALP Double Decode — Spotify Columns
> > > > >>> (Graviton
> > > > >>> > 3,
> > > > >>> > >> ARM
> > > > >>> > >> > >>>>>> Neoverse V1)
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>>
> > > > >>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>> │ Column │ -O2 (MB/s) │ -O3
> (MB/s) │
> > > > >>> Speedup │
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>>
> > > > >>> ├──────────────────┼──────────────┼──────────────┼─────────┤
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>> │ valence │ 3,155 │ 5,523
> │
> > > > >>> 1.75x │
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>> │ danceability │ 3,233 │ 5,685
> │
> > > > >>> 1.76x │
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>> │ energy │ 3,197 │ 5,652
> │
> > > > >>> 1.77x │
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>> │ loudness │ 3,186 │ 5,473
> │
> > > > >>> 1.72x │
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>>
> > > > >>> └──────────────────┴──────────────┴──────────────┴─────────┘
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <
> > > > >>> > prtkgaur@gmail.com
> > > > >>> > >> >
> > > > >>> > >> > >>>>>> wrote:
> > > > >>> > >> > >>>>>>
> > > > >>> > >> > >>>>>>> @Micah Kornfield <em...@gmail.com> : Got
> it.
> > > > >>> > >> > >>>>>>>
> > > > >>> > >> > >>>>>>> @Andrew Lamb <an...@gmail.com>
> > > > >>> > >> > >>>>>>>
> > > > >>> > >> > >>>>>>>
> > > > >>> > >> > >>>>>>>> Do you think it would be good to start moving the
> > > spec
> > > > >>> > >> development
> > > > >>> > >> > >>>>>>>> into
> > > > >>> > >> > >>>>>>>> markdown format, in preparation for finalizing
> it?
> > > > >>> > >> > >>>>>>>>
> > > > >>> > >> > >>>>>>>
> > > > >>> > >> > >>>>>>> Yes I'll update the numbers for some of the
> > examples I
> > > > >>> have in
> > > > >>> > >> the
> > > > >>> > >> > >>>>>>> spec based
> > > > >>> > >> > >>>>>>> on the updated header size. Then we should be good
> > to
> > > go
> > > > >>> for
> > > > >>> > the
> > > > >>> > >> > >>>>>>> markdown format.
> > > > >>> > >> > >>>>>>>
> > > > >>> > >> > >>>>>>> Thanks everyone!
> > > > >>> > >> > >>>>>>>
> > > > >>> > >> > >>>>>>>
> > > > >>> > >> > >>>>>>>>
> > > > >>> > >> > >>>>>>>> Andrew
> > > > >>> > >> > >>>>>>>>
> > > > >>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <
> > > > >>> > >> prtkgaur@gmail.com>
> > > > >>> > >> > >>>>>>>> wrote:
> > > > >>> > >> > >>>>>>>>
> > > > >>> > >> > >>>>>>>>> Hi team,
> > > > >>> > >> > >>>>>>>>>
> > > > >>> > >> > >>>>>>>>> 1) Andrew
> > > > >>> > >> > >>>>>>>>>
> > > > >>> > >> > >>>>>>>>> - Thanks for working on test files. My PR
> did
> > > add
> > > > >>> all
> > > > >>> > the
> > > > >>> > >> > test
> > > > >>> > >> > >>>>>>>> files I
> > > > >>> > >> > >>>>>>>>> used to benchmark on datasets. Maybe we can
> > club
> > > > it
> > > > >>> > >> together.
> > > > >>> > >> > >>>>>>>> WIll also
> > > > >>> > >> > >>>>>>>>> aid
> > > > >>> > >> > >>>>>>>>> cross language testing
> > > > >>> > >> > >>>>>>>>> - Kosta Tarasov working on Rust
> > implementation.
> > > > >>> This is
> > > > >>> > >> > great.
> > > > >>> > >> > >>>>>>>> Thanks
> > > > >>> > >> > >>>>>>>>>
> > > > >>> > >> > >>>>>>>>>
> > > > >>> > >> > >>>>>>>>> 2) Antoine
> > > > >>> > >> > >>>>>>>>>
> > > > >>> > >> > >>>>>>>>> - Thanks a lot for reporting the numbers on
> > AMD.
> > > > >>> Looks
> > > > >>> > >> like
> > > > >>> > >> > you
> > > > >>> > >> > >>>>>>>> are
> > > > >>> > >> > >>>>>>>>> getting 8X the decoding performance of BSS.
> > This
> > > > is
> > > > >>> > >> > amazing!!.
> > > > >>> > >> > >>>>>>>>> - Thanks for acknowledging the sampling
> > design.
> > > > >>> > >> > >>>>>>>>> - I agree with you on Fastlanes. In some
> crude
> > > > >>> > >> experiments I
> > > > >>> > >> > >>>>>>>> didn't get
> > > > >>> > >> > >>>>>>>>> a good perf benefit from it on Graviton3
> (but
> > > > maybe
> > > > >>> > there
> > > > >>> > >> was
> > > > >>> > >> > >>>>>>>> something
> > > > >>> > >> > >>>>>>>>> wrong with my implementation).
> > > > >>> > >> > >>>>>>>>> - Locking the 16bit exception encoding for
> the
> > > > spec
> > > > >>> in
> > > > >>> > >> this
> > > > >>> > >> > >>>>>>>> case.
> > > > >>> > >> > >>>>>>>>> - Awesome I think we have solved for all
> open
> > > > >>> questions
> > > > >>> > >> minus
> > > > >>> > >> > >>>>>>>> the
> > > > >>> > >> > >>>>>>>>> version byte :). (will get back on this
> soon)
> > > > >>> > >> > >>>>>>>>>
> > > > >>> > >> > >>>>>>>>>
> > > > >>> > >> > >>>>>>>>> 3) Micah
> > > > >>> > >> > >>>>>>>>>
> > > > >>> > >> > >>>>>>>>> - FastLanes : The current spec does allow
> for
> > > > using
> > > > >>> > >> FastLane
> > > > >>> > >> > >>>>>>>> with the
> > > > >>> > >> > >>>>>>>>> configurable enum value for layout. We
> should
> > be
> > > > >>> able to
> > > > >>> > >> > inject
> > > > >>> > >> > >>>>>>>> any
> > > > >>> > >> > >>>>>>>>> layout
> > > > >>> > >> > >>>>>>>>> in the current design.
> > > > >>> > >> > >>>>>>>>>
> > > > >>> > >> > >>>>>>>>>
> > > > >>> > >> > >>>>>>>>> Working on resolving all remaining open comments
> > on
> > > > the
> > > > >>> spec
> > > > >>> > >> this
> > > > >>> > >> > >>>>>>>> week.
> > > > >>> > >> > >>>>>>>>>
> > > > >>> > >> > >>>>>>>>> Best
> > > > >>> > >> > >>>>>>>>> Prateek
> > > > >>> > >> > >>>>>>>>>
> > > > >>> > >> > >>>>>>>>>
> > > > >>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <
> > > > >>> > >> > >>>>>>>> stevel@cloudera.com>
> > > > >>> > >> > >>>>>>>>> wrote:
> > > > >>> > >> > >>>>>>>>>
> > > > >>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
> > > > >>> > >> > >>>>>>>> emkornfield@gmail.com>
> > > > >>> > >> > >>>>>>>>>> wrote:
> > > > >>> > >> > >>>>>>>>>>
> > > > >>> > >> > >>>>>>>>>>>
> > > > >>> > >> > >>>>>>>>>>>
> > > > >>> > >> > >>>>>>>>>>> It looks like the actual issue described for
> ORC
> > > in
> > > > >>> the
> > > > >>> > >> paper
> > > > >>> > >> > >>>>>>>> is that
> > > > >>> > >> > >>>>>>>>> it
> > > > >>> > >> > >>>>>>>>>>> has multiple sub-encodings in a batch. This
> is
> > > > >>> different
> > > > >>> > >> then
> > > > >>> > >> > >>>>>>>> the
> > > > >>> > >> > >>>>>>>>> design
> > > > >>> > >> > >>>>>>>>>>> proposed here where there is still fixed
> > encoding
> > > > per
> > > > >>> page
> > > > >>> > >> in
> > > > >>> > >> > >>>>>>>> parquet.
> > > > >>> > >> > >>>>>>>>>>> Given reasonably sized pages I don't think
> > branch
> > > > >>> > >> > >>>>>>>> misprediction should
> > > > >>> > >> > >>>>>>>>>> be a
> > > > >>> > >> > >>>>>>>>>>> big issue for new encodings. I agree that we
> > > should
> > > > >>> be
> > > > >>> > >> > >>>>>>>> conservative in
> > > > >>> > >> > >>>>>>>>>>> general for adding new encodings.
> > > > >>> > >> > >>>>>>>>>>>
> > > > >>> > >> > >>>>>>>>>>>
> > > > >>> > >> > >>>>>>>>>> +1
> > > > >>> > >> > >>>>>>>>>>
> > > > >>> > >> > >>>>>>>>>
> > > > >>> > >> > >>>>>>>>
> > > > >>> > >> > >>>>>>>
> > > > >>> > >> > >
> > > > >>> > >> >
> > > > >>> > >> >
> > > > >>> > >> >
> > > > >>> > >>
> > > > >>> > >
> > > > >>> >
> > > > >>>
> > > > >>
> > > >
> > >
> >
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Julien Le Dem <ju...@apache.org>.
It was also in my spam folder... :(
I've seen a bunch of apache list emails go to my spam recently.
On Tue, Jul 21, 2026 at 10:58 AM Russell Spitzer <ru...@gmail.com>
wrote:
> Yes, my hope was that my vote would kick it out of spam but we definitely
> had two other votes that also went to spam for me.
>
> For anyone looking the thread is here
> https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt
>
> On Tue, Jul 21, 2026 at 10:57 AM Micah Kornfield <em...@gmail.com>
> wrote:
>
> > I've seen other's votes not go to spam, so hopefully people will see it
> in
> > there inboxes (or see this thread and look in there spam).
> >
> > On Tue, Jul 21, 2026 at 10:52 AM PRATEEK GAUR <pr...@gmail.com>
> wrote:
> >
> > > Okay,
> > >
> > > I sent out the voting email yesterday and so far I've heard from 3
> people
> > > that it ended up in their "spam folder". Dev VPN to be blamed for it?.
> > >
> > > What's the protocol here?
> > > I let people hunt for the email in their spam folder or send out
> another
> > > one?
> > >
> > > Best
> > > Prateek
> > >
> > > On Sun, Jul 19, 2026 at 10:47 PM PRATEEK GAUR <pr...@gmail.com>
> > wrote:
> > >
> > > > Hi team,
> > > >
> > > > Vinoo has added the tests which were requested in the last round of
> > > review.
> > > > In his own words.
> > > >
> > > > "
> > > > The extreme value tests Micah wanted are done and pushed. I added
> > > coverage
> > > > for values that need the full FOR bit width after the frame of
> > reference
> > > is
> > > > applied: 64-bit (the case where the signed max minus min subtraction
> > > > overflows), 63-bit (the non-overflow case), and 32-bit for floats.
> All
> > > > lossless with zero exceptions. I also verified them through Arrow C++
> > ALP
> > > > reader (the local build), so the Java-written extreme columns decode
> > > > correctly in C++, bit-exact against the expected values."
> > > >
> > > > State of the PR (#3397):
> > > >
> > > > - Merges cleanly with master (I pulled master in and resolved the
> > > > conflicts).
> > > > - 41 of 44 review threads resolved. The 3 remaining are
> intentional:
> > > > one deferred perf nit, one that stays open until parquet-format
> > ships
> > > ALP,
> > > >
> > > > "
> > > >
> > > > With this both the
> > > > 1) C++ PR : https://github.com/apache/arrow/pull/48345/changes
> > > > 2) Java PR : https://github.com/apache/parquet-java/pull/3397
> > > >
> > > > Are cross language tested and range of test scenarios.
> > > >
> > > > Best
> > > > Prateek
> > > >
> > > >
> > > > On Mon, Jul 13, 2026 at 8:38 AM PRATEEK GAUR <pr...@gmail.com>
> > wrote:
> > > >
> > > >> Thanks Matt,
> > > >>
> > > >> I'll be on it this week.
> > > >>
> > > >> Best
> > > >> Prateek
> > > >>
> > > >> On Mon, Jul 13, 2026 at 8:36 AM Matt Topol <zo...@gmail.com>
> > > >> wrote:
> > > >>
> > > >>> There is also the Go implementation submitted by Arnav (
> > > >>> https://github.com/apache/arrow-go/pull/704) which is waiting for
> > > >>> updates.
> > > >>>
> > > >>> Just wanted to make sure it didn't get lost here.
> > > >>>
> > > >>> --Matt
> > > >>>
> > > >>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK GAUR <pr...@gmail.com>
> > > >>> wrote:
> > > >>>
> > > >>> > Hi all,
> > > >>> >
> > > >>> > I'd like to share a status update on the ALP encoding effort and
> > get
> > > a
> > > >>> feel
> > > >>> > from the community before starting a formal vote.
> > > >>> >
> > > >>> > We now have the specification plus implementations in both
> > languages:
> > > >>> >
> > > >>> > - Spec: apache/parquet-format#557 — GH-533 Add ALP
> > > encoding
> > > >>> > specification
> > > >>> > - C++ (Arrow): apache/arrow#48345 — GH-48701 [C++][Parquet]
> > Add
> > > >>> ALPpd
> > > >>> > encoding
> > > >>> > - Java: apache/parquet-java#3397 — Parquet Java ALP
> > > >>> > Implementation (by Vinoo)
> > > >>> >
> > > >>> > Current state:
> > > >>> >
> > > >>> > - The C++ implementation has been through 3-4 rounds of
> review.
> > > >>> > - The Java implementation has been tracking well and has had
> > > >>> initial
> > > >>> > reviews from contributors, with no major remaining issues.
> > > >>> > - Cross-language compatibility tests are passing: the Arrow
> C++
> > > >>> decoder
> > > >>> > reads Java-written data bit-exactly across ~1.56M values
> and
> > 18
> > > >>> > fixtures, covering V1 and V2 pages, multiple vector sizes,
> > and
> > > >>> > several
> > > >>> > real datasets — zero mismatches.
> > > >>> >
> > > >>> > As raised in the java-pr we will be adding coverage for extreme
> > > values
> > > >>> > (those requiring
> > > >>> > 63-64 bits after FOR is applied) before we close a vote. We'll
> aim
> > > to
> > > >>> get
> > > >>> > that
> > > >>> > done in parallel.
> > > >>> >
> > > >>> > Plan: unless there are objections, I intend to start a vote at
> the
> > > end
> > > >>> of
> > > >>> > this week/early
> > > >>> > next week.
> > > >>> >
> > > >>> > Thanks
> > > >>> > Prateek
> > > >>> >
> > > >>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <pr...@gmail.com>
> > > >>> wrote:
> > > >>> >
> > > >>> > > Hi Team,
> > > >>> > >
> > > >>> > > Just wanted to provide some updates on ALP.
> > > >>> > > Micah and I have done 3-4 rounds of review for the c++ PR :
> > > >>> > > https://github.com/apache/arrow/pull/48345
> > > >>> > >
> > > >>> > > For the Java implementation Vinoo has been working on following
> > PR
> > > :
> > > >>> > > https://github.com/apache/parquet-java/pull/3397
> > > >>> > >
> > > >>> > > Best
> > > >>> > > Prateek
> > > >>> > >
> > > >>> > > On Tue, May 5, 2026 at 1:50 PM Micah Kornfield <
> > > >>> emkornfield@gmail.com>
> > > >>> > > wrote:
> > > >>> > >
> > > >>> > >> Hi Antoine,
> > > >>> > >>
> > > >>> > >> > Apologies if the question was already asked, but should we
> > care
> > > >>> about
> > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than
> > FLOAT16
> > > +
> > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> > > >>> > >>
> > > >>> > >> It was. We thought we could defer it for the following
> reasons:
> > > >>> > >> 1. It's not clear there are a lot of easy reference datasets
> to
> > > >>> test
> > > >>> > its
> > > >>> > >> effectiveness. It does look like there might be one or two on
> > > >>> > huggingface
> > > >>> > >> (e.g. https://huggingface.co/datasets/kikitora/curdie).
> > > >>> > >> 2. It seemed likely that float 16 was more likely used for
> > values
> > > >>> that
> > > >>> > >> were less likely to reduce to decimal values.
> > > >>> > >> 3. It could be added as an extension later if needed.
> > > >>> > >>
> > > >>> > >> Cheers,
> > > >>> > >> Micah
> > > >>> > >>
> > > >>> > >>
> > > >>> > >> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <
> > antoine@python.org
> > > >
> > > >>> > wrote:
> > > >>> > >>
> > > >>> > >> >
> > > >>> > >> > Hello,
> > > >>> > >> >
> > > >>> > >> > Apologies if the question was already asked, but should we
> > care
> > > >>> about
> > > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than
> > FLOAT16
> > > +
> > > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> > > >>> > >> >
> > > >>> > >> > Regards
> > > >>> > >> >
> > > >>> > >> > Antoine.
> > > >>> > >> >
> > > >>> > >> >
> > > >>> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
> > > >>> > >> > > Thanks Andrew and Micah for review feedback on the two
> PR's
> > > >>> > >> > > 1) (c++ arrow repo)
> > > >>> > >> https://github.com/apache/arrow/pull/48345/changes
> > > >>> > >> > > 2) (parquet-format repo)
> > > >>> > >> > https://github.com/apache/parquet-format/pull/557
> > > >>> > >> > >
> > > >>> > >> > > I have addressed all (unless I missed something) comments
> on
> > > >>> the two
> > > >>> > >> > PR's.
> > > >>> > >> > >
> > > >>> > >> > > Best
> > > >>> > >> > > Prateek
> > > >>> > >> > >
> > > >>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <
> > > >>> prtkgaur@gmail.com>
> > > >>> > >> wrote:
> > > >>> > >> > >
> > > >>> > >> > >> Thanks Andrew and Micah.
> > > >>> > >> > >>
> > > >>> > >> > >> `fair amount of feedback on at least the implementations`
> > > >>> > >> > >> For the c++ I have already started addressing the
> > feedback, I
> > > >>> > should
> > > >>> > >> be
> > > >>> > >> > >> done with that Monday/Tuesday.
> > > >>> > >> > >> I think Vinoo too has been making good progress on the
> Java
> > > >>> > >> > implementation.
> > > >>> > >> > >>
> > > >>> > >> > >> Best
> > > >>> > >> > >> Prateek
> > > >>> > >> > >>
> > > >>> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <
> > > >>> > andrewlamb11@gmail.com
> > > >>> > >> >
> > > >>> > >> > >> wrote:
> > > >>> > >> > >>
> > > >>> > >> > >>> Got it. Thank you for the clarification -- I will try
> and
> > > look
> > > >>> > into
> > > >>> > >> the
> > > >>> > >> > >>> spec and the Rust implementation[1] in this next week
> > > >>> > >> > >>>
> > > >>> > >> > >>> [1]: https://github.com/apache/arrow-rs/pull/9372
> > > >>> > >> > >>>
> > > >>> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <
> > > >>> > >> > emkornfield@gmail.com>
> > > >>> > >> > >>> wrote:
> > > >>> > >> > >>>
> > > >>> > >> > >>>> Hi Andrew,
> > > >>> > >> > >>>> I think there is a fair amount of feedback on at least
> > the
> > > >>> > >> > >>>> implementations, typically I think we've waited till
> they
> > > are
> > > >>> > >> close to
> > > >>> > >> > >>>> mergeable before a final vote. Otherwise I agree we
> are
> > > very
> > > >>> > >> close.
> > > >>> > >> > >>>>
> > > >>> > >> > >>>> -Micah
> > > >>> > >> > >>>>
> > > >>> > >> > >>>> On Saturday, April 25, 2026, Andrew Lamb <
> > > >>> andrewlamb11@gmail.com
> > > >>> > >
> > > >>> > >> > wrote:
> > > >>> > >> > >>>>
> > > >>> > >> > >>>>> Thanks Prateek,
> > > >>> > >> > >>>>>
> > > >>> > >> > >>>>> I think from this content it looks to me like we are
> > ready
> > > >>> to
> > > >>> > >> start a
> > > >>> > >> > >>>>> vote to explicitly accept ALP into Parquet
> > > >>> > >> > >>>>>
> > > >>> > >> > >>>>> Does anyone know of a reason we should postpone it for
> > > >>> longer?
> > > >>> > >> > >>>>> Perhaps someone needs some more time to review?
> > > >>> > >> > >>>>>
> > > >>> > >> > >>>>> Andrew
> > > >>> > >> > >>>>>
> > > >>> > >> > >>>>>
> > > >>> > >> > >>>>>
> > > >>> > >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <
> > > >>> > prtkgaur@gmail.com>
> > > >>> > >> > >>>>> wrote:
> > > >>> > >> > >>>>>
> > > >>> > >> > >>>>>> Hi team,
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>> Hope everyone is doing well. I got a chance to work
> > > >>> through all
> > > >>> > >> the
> > > >>> > >> > >>>>>> remaining feedback and update the spec doc. Here are
> > the
> > > >>> new
> > > >>> > >> > artifacts
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>> 1) Spec document :
> > > >>> > >> > >>>>>>
> > > >>> > >> >
> > > >>> > >>
> > > >>> >
> > > >>>
> > >
> >
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>> 2) Spec document in parquet format repo :
> > > >>> > >> > >>>>>> https://github.com/apache/parquet-format/pull/557
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>> 3) Alp implementation in arrow c++ repo :
> > > >>> > >> > >>>>>> https://github.com/apache/arrow/pull/48345/changes
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>> 4) Alp implementation in parquet-java repo : Work for
> > > >>> Vinoo and
> > > >>> > >> > Julien
> > > >>> > >> > >>>>>> https://github.com/apache/parquet-java/pull/3397
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>> 5) PR with test and benchmarking artifacts in
> > > >>> parquet-testing
> > > >>> > >> repo :
> > > >>> > >> > >>>>>> https://github.com/apache/parquet-testing/pull/100
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>> And
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>> - Go : Arnav just submitted an in progress
> > > >>> implementation
> > > >>> > in
> > > >>> > >> Go.
> > > >>> > >> > >>>>>> https://github.com/apache/arrow-go/pull/704 (I
> > > haven't
> > > >>> > >> started
> > > >>> > >> > >>>>>> looking at it yet)
> > > >>> > >> > >>>>>> - Rust : I remember Andrew mentioned that this
> work
> > > is
> > > >>> also
> > > >>> > >> in
> > > >>> > >> > >>>>>> progress (So 4 languages!)
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>> *Arrow C++ implementation *
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>> The PR is out and was also used by Antoine to report
> > the
> > > >>> > numbers
> > > >>> > >> as
> > > >>> > >> > >>>>>> reported here. Micah and Konstantin have given 1
> round
> > of
> > > >>> > >> feedback
> > > >>> > >> > >>>>>> and I'm addressing them today. Please note that the
> > > default
> > > >>> > >> > >>>>>> optimization flag for compiling is O2 and not Q3. I
> got
> > > >>> around
> > > >>> > >> 70%
> > > >>> > >> > >>>>>> performance improvement in the decoding speed when
> > using
> > > >>> the O3
> > > >>> > >> > flag.
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>> *Parqet-MR Java implementation (working with Vinoo
> and
> > > >>> Julien)
> > > >>> > >> and
> > > >>> > >> > **Cross
> > > >>> > >> > >>>>>> Language testing*
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>> Let me know if you have any questions or
> feedback.
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>> Now pasting some performance numbers
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>> Table 1: C++ ALP Double Decode — Spotify Columns
> > > >>> (Graviton
> > > >>> > 3,
> > > >>> > >> ARM
> > > >>> > >> > >>>>>> Neoverse V1)
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>>
> > > >>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │
> > > >>> Speedup │
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>>
> > > >>> ├──────────────────┼──────────────┼──────────────┼─────────┤
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>> │ valence │ 3,155 │ 5,523 │
> > > >>> 1.75x │
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>> │ danceability │ 3,233 │ 5,685 │
> > > >>> 1.76x │
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>> │ energy │ 3,197 │ 5,652 │
> > > >>> 1.77x │
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>> │ loudness │ 3,186 │ 5,473 │
> > > >>> 1.72x │
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>>
> > > >>> └──────────────────┴──────────────┴──────────────┴─────────┘
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <
> > > >>> > prtkgaur@gmail.com
> > > >>> > >> >
> > > >>> > >> > >>>>>> wrote:
> > > >>> > >> > >>>>>>
> > > >>> > >> > >>>>>>> @Micah Kornfield <em...@gmail.com> : Got it.
> > > >>> > >> > >>>>>>>
> > > >>> > >> > >>>>>>> @Andrew Lamb <an...@gmail.com>
> > > >>> > >> > >>>>>>>
> > > >>> > >> > >>>>>>>
> > > >>> > >> > >>>>>>>> Do you think it would be good to start moving the
> > spec
> > > >>> > >> development
> > > >>> > >> > >>>>>>>> into
> > > >>> > >> > >>>>>>>> markdown format, in preparation for finalizing it?
> > > >>> > >> > >>>>>>>>
> > > >>> > >> > >>>>>>>
> > > >>> > >> > >>>>>>> Yes I'll update the numbers for some of the
> examples I
> > > >>> have in
> > > >>> > >> the
> > > >>> > >> > >>>>>>> spec based
> > > >>> > >> > >>>>>>> on the updated header size. Then we should be good
> to
> > go
> > > >>> for
> > > >>> > the
> > > >>> > >> > >>>>>>> markdown format.
> > > >>> > >> > >>>>>>>
> > > >>> > >> > >>>>>>> Thanks everyone!
> > > >>> > >> > >>>>>>>
> > > >>> > >> > >>>>>>>
> > > >>> > >> > >>>>>>>>
> > > >>> > >> > >>>>>>>> Andrew
> > > >>> > >> > >>>>>>>>
> > > >>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <
> > > >>> > >> prtkgaur@gmail.com>
> > > >>> > >> > >>>>>>>> wrote:
> > > >>> > >> > >>>>>>>>
> > > >>> > >> > >>>>>>>>> Hi team,
> > > >>> > >> > >>>>>>>>>
> > > >>> > >> > >>>>>>>>> 1) Andrew
> > > >>> > >> > >>>>>>>>>
> > > >>> > >> > >>>>>>>>> - Thanks for working on test files. My PR did
> > add
> > > >>> all
> > > >>> > the
> > > >>> > >> > test
> > > >>> > >> > >>>>>>>> files I
> > > >>> > >> > >>>>>>>>> used to benchmark on datasets. Maybe we can
> club
> > > it
> > > >>> > >> together.
> > > >>> > >> > >>>>>>>> WIll also
> > > >>> > >> > >>>>>>>>> aid
> > > >>> > >> > >>>>>>>>> cross language testing
> > > >>> > >> > >>>>>>>>> - Kosta Tarasov working on Rust
> implementation.
> > > >>> This is
> > > >>> > >> > great.
> > > >>> > >> > >>>>>>>> Thanks
> > > >>> > >> > >>>>>>>>>
> > > >>> > >> > >>>>>>>>>
> > > >>> > >> > >>>>>>>>> 2) Antoine
> > > >>> > >> > >>>>>>>>>
> > > >>> > >> > >>>>>>>>> - Thanks a lot for reporting the numbers on
> AMD.
> > > >>> Looks
> > > >>> > >> like
> > > >>> > >> > you
> > > >>> > >> > >>>>>>>> are
> > > >>> > >> > >>>>>>>>> getting 8X the decoding performance of BSS.
> This
> > > is
> > > >>> > >> > amazing!!.
> > > >>> > >> > >>>>>>>>> - Thanks for acknowledging the sampling
> design.
> > > >>> > >> > >>>>>>>>> - I agree with you on Fastlanes. In some crude
> > > >>> > >> experiments I
> > > >>> > >> > >>>>>>>> didn't get
> > > >>> > >> > >>>>>>>>> a good perf benefit from it on Graviton3 (but
> > > maybe
> > > >>> > there
> > > >>> > >> was
> > > >>> > >> > >>>>>>>> something
> > > >>> > >> > >>>>>>>>> wrong with my implementation).
> > > >>> > >> > >>>>>>>>> - Locking the 16bit exception encoding for the
> > > spec
> > > >>> in
> > > >>> > >> this
> > > >>> > >> > >>>>>>>> case.
> > > >>> > >> > >>>>>>>>> - Awesome I think we have solved for all open
> > > >>> questions
> > > >>> > >> minus
> > > >>> > >> > >>>>>>>> the
> > > >>> > >> > >>>>>>>>> version byte :). (will get back on this soon)
> > > >>> > >> > >>>>>>>>>
> > > >>> > >> > >>>>>>>>>
> > > >>> > >> > >>>>>>>>> 3) Micah
> > > >>> > >> > >>>>>>>>>
> > > >>> > >> > >>>>>>>>> - FastLanes : The current spec does allow for
> > > using
> > > >>> > >> FastLane
> > > >>> > >> > >>>>>>>> with the
> > > >>> > >> > >>>>>>>>> configurable enum value for layout. We should
> be
> > > >>> able to
> > > >>> > >> > inject
> > > >>> > >> > >>>>>>>> any
> > > >>> > >> > >>>>>>>>> layout
> > > >>> > >> > >>>>>>>>> in the current design.
> > > >>> > >> > >>>>>>>>>
> > > >>> > >> > >>>>>>>>>
> > > >>> > >> > >>>>>>>>> Working on resolving all remaining open comments
> on
> > > the
> > > >>> spec
> > > >>> > >> this
> > > >>> > >> > >>>>>>>> week.
> > > >>> > >> > >>>>>>>>>
> > > >>> > >> > >>>>>>>>> Best
> > > >>> > >> > >>>>>>>>> Prateek
> > > >>> > >> > >>>>>>>>>
> > > >>> > >> > >>>>>>>>>
> > > >>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <
> > > >>> > >> > >>>>>>>> stevel@cloudera.com>
> > > >>> > >> > >>>>>>>>> wrote:
> > > >>> > >> > >>>>>>>>>
> > > >>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
> > > >>> > >> > >>>>>>>> emkornfield@gmail.com>
> > > >>> > >> > >>>>>>>>>> wrote:
> > > >>> > >> > >>>>>>>>>>
> > > >>> > >> > >>>>>>>>>>>
> > > >>> > >> > >>>>>>>>>>>
> > > >>> > >> > >>>>>>>>>>> It looks like the actual issue described for ORC
> > in
> > > >>> the
> > > >>> > >> paper
> > > >>> > >> > >>>>>>>> is that
> > > >>> > >> > >>>>>>>>> it
> > > >>> > >> > >>>>>>>>>>> has multiple sub-encodings in a batch. This is
> > > >>> different
> > > >>> > >> then
> > > >>> > >> > >>>>>>>> the
> > > >>> > >> > >>>>>>>>> design
> > > >>> > >> > >>>>>>>>>>> proposed here where there is still fixed
> encoding
> > > per
> > > >>> page
> > > >>> > >> in
> > > >>> > >> > >>>>>>>> parquet.
> > > >>> > >> > >>>>>>>>>>> Given reasonably sized pages I don't think
> branch
> > > >>> > >> > >>>>>>>> misprediction should
> > > >>> > >> > >>>>>>>>>> be a
> > > >>> > >> > >>>>>>>>>>> big issue for new encodings. I agree that we
> > should
> > > >>> be
> > > >>> > >> > >>>>>>>> conservative in
> > > >>> > >> > >>>>>>>>>>> general for adding new encodings.
> > > >>> > >> > >>>>>>>>>>>
> > > >>> > >> > >>>>>>>>>>>
> > > >>> > >> > >>>>>>>>>> +1
> > > >>> > >> > >>>>>>>>>>
> > > >>> > >> > >>>>>>>>>
> > > >>> > >> > >>>>>>>>
> > > >>> > >> > >>>>>>>
> > > >>> > >> > >
> > > >>> > >> >
> > > >>> > >> >
> > > >>> > >> >
> > > >>> > >>
> > > >>> > >
> > > >>> >
> > > >>>
> > > >>
> > >
> >
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Russell Spitzer <ru...@gmail.com>.
Yes, my hope was that my vote would kick it out of spam but we definitely
had two other votes that also went to spam for me.
For anyone looking the thread is here
https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt
On Tue, Jul 21, 2026 at 10:57 AM Micah Kornfield <em...@gmail.com>
wrote:
> I've seen other's votes not go to spam, so hopefully people will see it in
> there inboxes (or see this thread and look in there spam).
>
> On Tue, Jul 21, 2026 at 10:52 AM PRATEEK GAUR <pr...@gmail.com> wrote:
>
> > Okay,
> >
> > I sent out the voting email yesterday and so far I've heard from 3 people
> > that it ended up in their "spam folder". Dev VPN to be blamed for it?.
> >
> > What's the protocol here?
> > I let people hunt for the email in their spam folder or send out another
> > one?
> >
> > Best
> > Prateek
> >
> > On Sun, Jul 19, 2026 at 10:47 PM PRATEEK GAUR <pr...@gmail.com>
> wrote:
> >
> > > Hi team,
> > >
> > > Vinoo has added the tests which were requested in the last round of
> > review.
> > > In his own words.
> > >
> > > "
> > > The extreme value tests Micah wanted are done and pushed. I added
> > coverage
> > > for values that need the full FOR bit width after the frame of
> reference
> > is
> > > applied: 64-bit (the case where the signed max minus min subtraction
> > > overflows), 63-bit (the non-overflow case), and 32-bit for floats. All
> > > lossless with zero exceptions. I also verified them through Arrow C++
> ALP
> > > reader (the local build), so the Java-written extreme columns decode
> > > correctly in C++, bit-exact against the expected values."
> > >
> > > State of the PR (#3397):
> > >
> > > - Merges cleanly with master (I pulled master in and resolved the
> > > conflicts).
> > > - 41 of 44 review threads resolved. The 3 remaining are intentional:
> > > one deferred perf nit, one that stays open until parquet-format
> ships
> > ALP,
> > >
> > > "
> > >
> > > With this both the
> > > 1) C++ PR : https://github.com/apache/arrow/pull/48345/changes
> > > 2) Java PR : https://github.com/apache/parquet-java/pull/3397
> > >
> > > Are cross language tested and range of test scenarios.
> > >
> > > Best
> > > Prateek
> > >
> > >
> > > On Mon, Jul 13, 2026 at 8:38 AM PRATEEK GAUR <pr...@gmail.com>
> wrote:
> > >
> > >> Thanks Matt,
> > >>
> > >> I'll be on it this week.
> > >>
> > >> Best
> > >> Prateek
> > >>
> > >> On Mon, Jul 13, 2026 at 8:36 AM Matt Topol <zo...@gmail.com>
> > >> wrote:
> > >>
> > >>> There is also the Go implementation submitted by Arnav (
> > >>> https://github.com/apache/arrow-go/pull/704) which is waiting for
> > >>> updates.
> > >>>
> > >>> Just wanted to make sure it didn't get lost here.
> > >>>
> > >>> --Matt
> > >>>
> > >>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK GAUR <pr...@gmail.com>
> > >>> wrote:
> > >>>
> > >>> > Hi all,
> > >>> >
> > >>> > I'd like to share a status update on the ALP encoding effort and
> get
> > a
> > >>> feel
> > >>> > from the community before starting a formal vote.
> > >>> >
> > >>> > We now have the specification plus implementations in both
> languages:
> > >>> >
> > >>> > - Spec: apache/parquet-format#557 — GH-533 Add ALP
> > encoding
> > >>> > specification
> > >>> > - C++ (Arrow): apache/arrow#48345 — GH-48701 [C++][Parquet]
> Add
> > >>> ALPpd
> > >>> > encoding
> > >>> > - Java: apache/parquet-java#3397 — Parquet Java ALP
> > >>> > Implementation (by Vinoo)
> > >>> >
> > >>> > Current state:
> > >>> >
> > >>> > - The C++ implementation has been through 3-4 rounds of review.
> > >>> > - The Java implementation has been tracking well and has had
> > >>> initial
> > >>> > reviews from contributors, with no major remaining issues.
> > >>> > - Cross-language compatibility tests are passing: the Arrow C++
> > >>> decoder
> > >>> > reads Java-written data bit-exactly across ~1.56M values and
> 18
> > >>> > fixtures, covering V1 and V2 pages, multiple vector sizes,
> and
> > >>> > several
> > >>> > real datasets — zero mismatches.
> > >>> >
> > >>> > As raised in the java-pr we will be adding coverage for extreme
> > values
> > >>> > (those requiring
> > >>> > 63-64 bits after FOR is applied) before we close a vote. We'll aim
> > to
> > >>> get
> > >>> > that
> > >>> > done in parallel.
> > >>> >
> > >>> > Plan: unless there are objections, I intend to start a vote at the
> > end
> > >>> of
> > >>> > this week/early
> > >>> > next week.
> > >>> >
> > >>> > Thanks
> > >>> > Prateek
> > >>> >
> > >>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <pr...@gmail.com>
> > >>> wrote:
> > >>> >
> > >>> > > Hi Team,
> > >>> > >
> > >>> > > Just wanted to provide some updates on ALP.
> > >>> > > Micah and I have done 3-4 rounds of review for the c++ PR :
> > >>> > > https://github.com/apache/arrow/pull/48345
> > >>> > >
> > >>> > > For the Java implementation Vinoo has been working on following
> PR
> > :
> > >>> > > https://github.com/apache/parquet-java/pull/3397
> > >>> > >
> > >>> > > Best
> > >>> > > Prateek
> > >>> > >
> > >>> > > On Tue, May 5, 2026 at 1:50 PM Micah Kornfield <
> > >>> emkornfield@gmail.com>
> > >>> > > wrote:
> > >>> > >
> > >>> > >> Hi Antoine,
> > >>> > >>
> > >>> > >> > Apologies if the question was already asked, but should we
> care
> > >>> about
> > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than
> FLOAT16
> > +
> > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> > >>> > >>
> > >>> > >> It was. We thought we could defer it for the following reasons:
> > >>> > >> 1. It's not clear there are a lot of easy reference datasets to
> > >>> test
> > >>> > its
> > >>> > >> effectiveness. It does look like there might be one or two on
> > >>> > huggingface
> > >>> > >> (e.g. https://huggingface.co/datasets/kikitora/curdie).
> > >>> > >> 2. It seemed likely that float 16 was more likely used for
> values
> > >>> that
> > >>> > >> were less likely to reduce to decimal values.
> > >>> > >> 3. It could be added as an extension later if needed.
> > >>> > >>
> > >>> > >> Cheers,
> > >>> > >> Micah
> > >>> > >>
> > >>> > >>
> > >>> > >> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <
> antoine@python.org
> > >
> > >>> > wrote:
> > >>> > >>
> > >>> > >> >
> > >>> > >> > Hello,
> > >>> > >> >
> > >>> > >> > Apologies if the question was already asked, but should we
> care
> > >>> about
> > >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than
> FLOAT16
> > +
> > >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> > >>> > >> >
> > >>> > >> > Regards
> > >>> > >> >
> > >>> > >> > Antoine.
> > >>> > >> >
> > >>> > >> >
> > >>> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
> > >>> > >> > > Thanks Andrew and Micah for review feedback on the two PR's
> > >>> > >> > > 1) (c++ arrow repo)
> > >>> > >> https://github.com/apache/arrow/pull/48345/changes
> > >>> > >> > > 2) (parquet-format repo)
> > >>> > >> > https://github.com/apache/parquet-format/pull/557
> > >>> > >> > >
> > >>> > >> > > I have addressed all (unless I missed something) comments on
> > >>> the two
> > >>> > >> > PR's.
> > >>> > >> > >
> > >>> > >> > > Best
> > >>> > >> > > Prateek
> > >>> > >> > >
> > >>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <
> > >>> prtkgaur@gmail.com>
> > >>> > >> wrote:
> > >>> > >> > >
> > >>> > >> > >> Thanks Andrew and Micah.
> > >>> > >> > >>
> > >>> > >> > >> `fair amount of feedback on at least the implementations`
> > >>> > >> > >> For the c++ I have already started addressing the
> feedback, I
> > >>> > should
> > >>> > >> be
> > >>> > >> > >> done with that Monday/Tuesday.
> > >>> > >> > >> I think Vinoo too has been making good progress on the Java
> > >>> > >> > implementation.
> > >>> > >> > >>
> > >>> > >> > >> Best
> > >>> > >> > >> Prateek
> > >>> > >> > >>
> > >>> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <
> > >>> > andrewlamb11@gmail.com
> > >>> > >> >
> > >>> > >> > >> wrote:
> > >>> > >> > >>
> > >>> > >> > >>> Got it. Thank you for the clarification -- I will try and
> > look
> > >>> > into
> > >>> > >> the
> > >>> > >> > >>> spec and the Rust implementation[1] in this next week
> > >>> > >> > >>>
> > >>> > >> > >>> [1]: https://github.com/apache/arrow-rs/pull/9372
> > >>> > >> > >>>
> > >>> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <
> > >>> > >> > emkornfield@gmail.com>
> > >>> > >> > >>> wrote:
> > >>> > >> > >>>
> > >>> > >> > >>>> Hi Andrew,
> > >>> > >> > >>>> I think there is a fair amount of feedback on at least
> the
> > >>> > >> > >>>> implementations, typically I think we've waited till they
> > are
> > >>> > >> close to
> > >>> > >> > >>>> mergeable before a final vote. Otherwise I agree we are
> > very
> > >>> > >> close.
> > >>> > >> > >>>>
> > >>> > >> > >>>> -Micah
> > >>> > >> > >>>>
> > >>> > >> > >>>> On Saturday, April 25, 2026, Andrew Lamb <
> > >>> andrewlamb11@gmail.com
> > >>> > >
> > >>> > >> > wrote:
> > >>> > >> > >>>>
> > >>> > >> > >>>>> Thanks Prateek,
> > >>> > >> > >>>>>
> > >>> > >> > >>>>> I think from this content it looks to me like we are
> ready
> > >>> to
> > >>> > >> start a
> > >>> > >> > >>>>> vote to explicitly accept ALP into Parquet
> > >>> > >> > >>>>>
> > >>> > >> > >>>>> Does anyone know of a reason we should postpone it for
> > >>> longer?
> > >>> > >> > >>>>> Perhaps someone needs some more time to review?
> > >>> > >> > >>>>>
> > >>> > >> > >>>>> Andrew
> > >>> > >> > >>>>>
> > >>> > >> > >>>>>
> > >>> > >> > >>>>>
> > >>> > >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <
> > >>> > prtkgaur@gmail.com>
> > >>> > >> > >>>>> wrote:
> > >>> > >> > >>>>>
> > >>> > >> > >>>>>> Hi team,
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>> Hope everyone is doing well. I got a chance to work
> > >>> through all
> > >>> > >> the
> > >>> > >> > >>>>>> remaining feedback and update the spec doc. Here are
> the
> > >>> new
> > >>> > >> > artifacts
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>> 1) Spec document :
> > >>> > >> > >>>>>>
> > >>> > >> >
> > >>> > >>
> > >>> >
> > >>>
> >
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>> 2) Spec document in parquet format repo :
> > >>> > >> > >>>>>> https://github.com/apache/parquet-format/pull/557
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>> 3) Alp implementation in arrow c++ repo :
> > >>> > >> > >>>>>> https://github.com/apache/arrow/pull/48345/changes
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>> 4) Alp implementation in parquet-java repo : Work for
> > >>> Vinoo and
> > >>> > >> > Julien
> > >>> > >> > >>>>>> https://github.com/apache/parquet-java/pull/3397
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>> 5) PR with test and benchmarking artifacts in
> > >>> parquet-testing
> > >>> > >> repo :
> > >>> > >> > >>>>>> https://github.com/apache/parquet-testing/pull/100
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>> And
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>> - Go : Arnav just submitted an in progress
> > >>> implementation
> > >>> > in
> > >>> > >> Go.
> > >>> > >> > >>>>>> https://github.com/apache/arrow-go/pull/704 (I
> > haven't
> > >>> > >> started
> > >>> > >> > >>>>>> looking at it yet)
> > >>> > >> > >>>>>> - Rust : I remember Andrew mentioned that this work
> > is
> > >>> also
> > >>> > >> in
> > >>> > >> > >>>>>> progress (So 4 languages!)
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>> *Arrow C++ implementation *
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>> The PR is out and was also used by Antoine to report
> the
> > >>> > numbers
> > >>> > >> as
> > >>> > >> > >>>>>> reported here. Micah and Konstantin have given 1 round
> of
> > >>> > >> feedback
> > >>> > >> > >>>>>> and I'm addressing them today. Please note that the
> > default
> > >>> > >> > >>>>>> optimization flag for compiling is O2 and not Q3. I got
> > >>> around
> > >>> > >> 70%
> > >>> > >> > >>>>>> performance improvement in the decoding speed when
> using
> > >>> the O3
> > >>> > >> > flag.
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>> *Parqet-MR Java implementation (working with Vinoo and
> > >>> Julien)
> > >>> > >> and
> > >>> > >> > **Cross
> > >>> > >> > >>>>>> Language testing*
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>> Let me know if you have any questions or feedback.
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>> Now pasting some performance numbers
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>> Table 1: C++ ALP Double Decode — Spotify Columns
> > >>> (Graviton
> > >>> > 3,
> > >>> > >> ARM
> > >>> > >> > >>>>>> Neoverse V1)
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>>
> > >>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │
> > >>> Speedup │
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>>
> > >>> ├──────────────────┼──────────────┼──────────────┼─────────┤
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>> │ valence │ 3,155 │ 5,523 │
> > >>> 1.75x │
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>> │ danceability │ 3,233 │ 5,685 │
> > >>> 1.76x │
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>> │ energy │ 3,197 │ 5,652 │
> > >>> 1.77x │
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>> │ loudness │ 3,186 │ 5,473 │
> > >>> 1.72x │
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>>
> > >>> └──────────────────┴──────────────┴──────────────┴─────────┘
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <
> > >>> > prtkgaur@gmail.com
> > >>> > >> >
> > >>> > >> > >>>>>> wrote:
> > >>> > >> > >>>>>>
> > >>> > >> > >>>>>>> @Micah Kornfield <em...@gmail.com> : Got it.
> > >>> > >> > >>>>>>>
> > >>> > >> > >>>>>>> @Andrew Lamb <an...@gmail.com>
> > >>> > >> > >>>>>>>
> > >>> > >> > >>>>>>>
> > >>> > >> > >>>>>>>> Do you think it would be good to start moving the
> spec
> > >>> > >> development
> > >>> > >> > >>>>>>>> into
> > >>> > >> > >>>>>>>> markdown format, in preparation for finalizing it?
> > >>> > >> > >>>>>>>>
> > >>> > >> > >>>>>>>
> > >>> > >> > >>>>>>> Yes I'll update the numbers for some of the examples I
> > >>> have in
> > >>> > >> the
> > >>> > >> > >>>>>>> spec based
> > >>> > >> > >>>>>>> on the updated header size. Then we should be good to
> go
> > >>> for
> > >>> > the
> > >>> > >> > >>>>>>> markdown format.
> > >>> > >> > >>>>>>>
> > >>> > >> > >>>>>>> Thanks everyone!
> > >>> > >> > >>>>>>>
> > >>> > >> > >>>>>>>
> > >>> > >> > >>>>>>>>
> > >>> > >> > >>>>>>>> Andrew
> > >>> > >> > >>>>>>>>
> > >>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <
> > >>> > >> prtkgaur@gmail.com>
> > >>> > >> > >>>>>>>> wrote:
> > >>> > >> > >>>>>>>>
> > >>> > >> > >>>>>>>>> Hi team,
> > >>> > >> > >>>>>>>>>
> > >>> > >> > >>>>>>>>> 1) Andrew
> > >>> > >> > >>>>>>>>>
> > >>> > >> > >>>>>>>>> - Thanks for working on test files. My PR did
> add
> > >>> all
> > >>> > the
> > >>> > >> > test
> > >>> > >> > >>>>>>>> files I
> > >>> > >> > >>>>>>>>> used to benchmark on datasets. Maybe we can club
> > it
> > >>> > >> together.
> > >>> > >> > >>>>>>>> WIll also
> > >>> > >> > >>>>>>>>> aid
> > >>> > >> > >>>>>>>>> cross language testing
> > >>> > >> > >>>>>>>>> - Kosta Tarasov working on Rust implementation.
> > >>> This is
> > >>> > >> > great.
> > >>> > >> > >>>>>>>> Thanks
> > >>> > >> > >>>>>>>>>
> > >>> > >> > >>>>>>>>>
> > >>> > >> > >>>>>>>>> 2) Antoine
> > >>> > >> > >>>>>>>>>
> > >>> > >> > >>>>>>>>> - Thanks a lot for reporting the numbers on AMD.
> > >>> Looks
> > >>> > >> like
> > >>> > >> > you
> > >>> > >> > >>>>>>>> are
> > >>> > >> > >>>>>>>>> getting 8X the decoding performance of BSS. This
> > is
> > >>> > >> > amazing!!.
> > >>> > >> > >>>>>>>>> - Thanks for acknowledging the sampling design.
> > >>> > >> > >>>>>>>>> - I agree with you on Fastlanes. In some crude
> > >>> > >> experiments I
> > >>> > >> > >>>>>>>> didn't get
> > >>> > >> > >>>>>>>>> a good perf benefit from it on Graviton3 (but
> > maybe
> > >>> > there
> > >>> > >> was
> > >>> > >> > >>>>>>>> something
> > >>> > >> > >>>>>>>>> wrong with my implementation).
> > >>> > >> > >>>>>>>>> - Locking the 16bit exception encoding for the
> > spec
> > >>> in
> > >>> > >> this
> > >>> > >> > >>>>>>>> case.
> > >>> > >> > >>>>>>>>> - Awesome I think we have solved for all open
> > >>> questions
> > >>> > >> minus
> > >>> > >> > >>>>>>>> the
> > >>> > >> > >>>>>>>>> version byte :). (will get back on this soon)
> > >>> > >> > >>>>>>>>>
> > >>> > >> > >>>>>>>>>
> > >>> > >> > >>>>>>>>> 3) Micah
> > >>> > >> > >>>>>>>>>
> > >>> > >> > >>>>>>>>> - FastLanes : The current spec does allow for
> > using
> > >>> > >> FastLane
> > >>> > >> > >>>>>>>> with the
> > >>> > >> > >>>>>>>>> configurable enum value for layout. We should be
> > >>> able to
> > >>> > >> > inject
> > >>> > >> > >>>>>>>> any
> > >>> > >> > >>>>>>>>> layout
> > >>> > >> > >>>>>>>>> in the current design.
> > >>> > >> > >>>>>>>>>
> > >>> > >> > >>>>>>>>>
> > >>> > >> > >>>>>>>>> Working on resolving all remaining open comments on
> > the
> > >>> spec
> > >>> > >> this
> > >>> > >> > >>>>>>>> week.
> > >>> > >> > >>>>>>>>>
> > >>> > >> > >>>>>>>>> Best
> > >>> > >> > >>>>>>>>> Prateek
> > >>> > >> > >>>>>>>>>
> > >>> > >> > >>>>>>>>>
> > >>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <
> > >>> > >> > >>>>>>>> stevel@cloudera.com>
> > >>> > >> > >>>>>>>>> wrote:
> > >>> > >> > >>>>>>>>>
> > >>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
> > >>> > >> > >>>>>>>> emkornfield@gmail.com>
> > >>> > >> > >>>>>>>>>> wrote:
> > >>> > >> > >>>>>>>>>>
> > >>> > >> > >>>>>>>>>>>
> > >>> > >> > >>>>>>>>>>>
> > >>> > >> > >>>>>>>>>>> It looks like the actual issue described for ORC
> in
> > >>> the
> > >>> > >> paper
> > >>> > >> > >>>>>>>> is that
> > >>> > >> > >>>>>>>>> it
> > >>> > >> > >>>>>>>>>>> has multiple sub-encodings in a batch. This is
> > >>> different
> > >>> > >> then
> > >>> > >> > >>>>>>>> the
> > >>> > >> > >>>>>>>>> design
> > >>> > >> > >>>>>>>>>>> proposed here where there is still fixed encoding
> > per
> > >>> page
> > >>> > >> in
> > >>> > >> > >>>>>>>> parquet.
> > >>> > >> > >>>>>>>>>>> Given reasonably sized pages I don't think branch
> > >>> > >> > >>>>>>>> misprediction should
> > >>> > >> > >>>>>>>>>> be a
> > >>> > >> > >>>>>>>>>>> big issue for new encodings. I agree that we
> should
> > >>> be
> > >>> > >> > >>>>>>>> conservative in
> > >>> > >> > >>>>>>>>>>> general for adding new encodings.
> > >>> > >> > >>>>>>>>>>>
> > >>> > >> > >>>>>>>>>>>
> > >>> > >> > >>>>>>>>>> +1
> > >>> > >> > >>>>>>>>>>
> > >>> > >> > >>>>>>>>>
> > >>> > >> > >>>>>>>>
> > >>> > >> > >>>>>>>
> > >>> > >> > >
> > >>> > >> >
> > >>> > >> >
> > >>> > >> >
> > >>> > >>
> > >>> > >
> > >>> >
> > >>>
> > >>
> >
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Micah Kornfield <em...@gmail.com>.
I've seen other's votes not go to spam, so hopefully people will see it in
there inboxes (or see this thread and look in there spam).
On Tue, Jul 21, 2026 at 10:52 AM PRATEEK GAUR <pr...@gmail.com> wrote:
> Okay,
>
> I sent out the voting email yesterday and so far I've heard from 3 people
> that it ended up in their "spam folder". Dev VPN to be blamed for it?.
>
> What's the protocol here?
> I let people hunt for the email in their spam folder or send out another
> one?
>
> Best
> Prateek
>
> On Sun, Jul 19, 2026 at 10:47 PM PRATEEK GAUR <pr...@gmail.com> wrote:
>
> > Hi team,
> >
> > Vinoo has added the tests which were requested in the last round of
> review.
> > In his own words.
> >
> > "
> > The extreme value tests Micah wanted are done and pushed. I added
> coverage
> > for values that need the full FOR bit width after the frame of reference
> is
> > applied: 64-bit (the case where the signed max minus min subtraction
> > overflows), 63-bit (the non-overflow case), and 32-bit for floats. All
> > lossless with zero exceptions. I also verified them through Arrow C++ ALP
> > reader (the local build), so the Java-written extreme columns decode
> > correctly in C++, bit-exact against the expected values."
> >
> > State of the PR (#3397):
> >
> > - Merges cleanly with master (I pulled master in and resolved the
> > conflicts).
> > - 41 of 44 review threads resolved. The 3 remaining are intentional:
> > one deferred perf nit, one that stays open until parquet-format ships
> ALP,
> >
> > "
> >
> > With this both the
> > 1) C++ PR : https://github.com/apache/arrow/pull/48345/changes
> > 2) Java PR : https://github.com/apache/parquet-java/pull/3397
> >
> > Are cross language tested and range of test scenarios.
> >
> > Best
> > Prateek
> >
> >
> > On Mon, Jul 13, 2026 at 8:38 AM PRATEEK GAUR <pr...@gmail.com> wrote:
> >
> >> Thanks Matt,
> >>
> >> I'll be on it this week.
> >>
> >> Best
> >> Prateek
> >>
> >> On Mon, Jul 13, 2026 at 8:36 AM Matt Topol <zo...@gmail.com>
> >> wrote:
> >>
> >>> There is also the Go implementation submitted by Arnav (
> >>> https://github.com/apache/arrow-go/pull/704) which is waiting for
> >>> updates.
> >>>
> >>> Just wanted to make sure it didn't get lost here.
> >>>
> >>> --Matt
> >>>
> >>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK GAUR <pr...@gmail.com>
> >>> wrote:
> >>>
> >>> > Hi all,
> >>> >
> >>> > I'd like to share a status update on the ALP encoding effort and get
> a
> >>> feel
> >>> > from the community before starting a formal vote.
> >>> >
> >>> > We now have the specification plus implementations in both languages:
> >>> >
> >>> > - Spec: apache/parquet-format#557 — GH-533 Add ALP
> encoding
> >>> > specification
> >>> > - C++ (Arrow): apache/arrow#48345 — GH-48701 [C++][Parquet] Add
> >>> ALPpd
> >>> > encoding
> >>> > - Java: apache/parquet-java#3397 — Parquet Java ALP
> >>> > Implementation (by Vinoo)
> >>> >
> >>> > Current state:
> >>> >
> >>> > - The C++ implementation has been through 3-4 rounds of review.
> >>> > - The Java implementation has been tracking well and has had
> >>> initial
> >>> > reviews from contributors, with no major remaining issues.
> >>> > - Cross-language compatibility tests are passing: the Arrow C++
> >>> decoder
> >>> > reads Java-written data bit-exactly across ~1.56M values and 18
> >>> > fixtures, covering V1 and V2 pages, multiple vector sizes, and
> >>> > several
> >>> > real datasets — zero mismatches.
> >>> >
> >>> > As raised in the java-pr we will be adding coverage for extreme
> values
> >>> > (those requiring
> >>> > 63-64 bits after FOR is applied) before we close a vote. We'll aim
> to
> >>> get
> >>> > that
> >>> > done in parallel.
> >>> >
> >>> > Plan: unless there are objections, I intend to start a vote at the
> end
> >>> of
> >>> > this week/early
> >>> > next week.
> >>> >
> >>> > Thanks
> >>> > Prateek
> >>> >
> >>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <pr...@gmail.com>
> >>> wrote:
> >>> >
> >>> > > Hi Team,
> >>> > >
> >>> > > Just wanted to provide some updates on ALP.
> >>> > > Micah and I have done 3-4 rounds of review for the c++ PR :
> >>> > > https://github.com/apache/arrow/pull/48345
> >>> > >
> >>> > > For the Java implementation Vinoo has been working on following PR
> :
> >>> > > https://github.com/apache/parquet-java/pull/3397
> >>> > >
> >>> > > Best
> >>> > > Prateek
> >>> > >
> >>> > > On Tue, May 5, 2026 at 1:50 PM Micah Kornfield <
> >>> emkornfield@gmail.com>
> >>> > > wrote:
> >>> > >
> >>> > >> Hi Antoine,
> >>> > >>
> >>> > >> > Apologies if the question was already asked, but should we care
> >>> about
> >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16
> +
> >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> >>> > >>
> >>> > >> It was. We thought we could defer it for the following reasons:
> >>> > >> 1. It's not clear there are a lot of easy reference datasets to
> >>> test
> >>> > its
> >>> > >> effectiveness. It does look like there might be one or two on
> >>> > huggingface
> >>> > >> (e.g. https://huggingface.co/datasets/kikitora/curdie).
> >>> > >> 2. It seemed likely that float 16 was more likely used for values
> >>> that
> >>> > >> were less likely to reduce to decimal values.
> >>> > >> 3. It could be added as an extension later if needed.
> >>> > >>
> >>> > >> Cheers,
> >>> > >> Micah
> >>> > >>
> >>> > >>
> >>> > >> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <antoine@python.org
> >
> >>> > wrote:
> >>> > >>
> >>> > >> >
> >>> > >> > Hello,
> >>> > >> >
> >>> > >> > Apologies if the question was already asked, but should we care
> >>> about
> >>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16
> +
> >>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> >>> > >> >
> >>> > >> > Regards
> >>> > >> >
> >>> > >> > Antoine.
> >>> > >> >
> >>> > >> >
> >>> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
> >>> > >> > > Thanks Andrew and Micah for review feedback on the two PR's
> >>> > >> > > 1) (c++ arrow repo)
> >>> > >> https://github.com/apache/arrow/pull/48345/changes
> >>> > >> > > 2) (parquet-format repo)
> >>> > >> > https://github.com/apache/parquet-format/pull/557
> >>> > >> > >
> >>> > >> > > I have addressed all (unless I missed something) comments on
> >>> the two
> >>> > >> > PR's.
> >>> > >> > >
> >>> > >> > > Best
> >>> > >> > > Prateek
> >>> > >> > >
> >>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <
> >>> prtkgaur@gmail.com>
> >>> > >> wrote:
> >>> > >> > >
> >>> > >> > >> Thanks Andrew and Micah.
> >>> > >> > >>
> >>> > >> > >> `fair amount of feedback on at least the implementations`
> >>> > >> > >> For the c++ I have already started addressing the feedback, I
> >>> > should
> >>> > >> be
> >>> > >> > >> done with that Monday/Tuesday.
> >>> > >> > >> I think Vinoo too has been making good progress on the Java
> >>> > >> > implementation.
> >>> > >> > >>
> >>> > >> > >> Best
> >>> > >> > >> Prateek
> >>> > >> > >>
> >>> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <
> >>> > andrewlamb11@gmail.com
> >>> > >> >
> >>> > >> > >> wrote:
> >>> > >> > >>
> >>> > >> > >>> Got it. Thank you for the clarification -- I will try and
> look
> >>> > into
> >>> > >> the
> >>> > >> > >>> spec and the Rust implementation[1] in this next week
> >>> > >> > >>>
> >>> > >> > >>> [1]: https://github.com/apache/arrow-rs/pull/9372
> >>> > >> > >>>
> >>> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <
> >>> > >> > emkornfield@gmail.com>
> >>> > >> > >>> wrote:
> >>> > >> > >>>
> >>> > >> > >>>> Hi Andrew,
> >>> > >> > >>>> I think there is a fair amount of feedback on at least the
> >>> > >> > >>>> implementations, typically I think we've waited till they
> are
> >>> > >> close to
> >>> > >> > >>>> mergeable before a final vote. Otherwise I agree we are
> very
> >>> > >> close.
> >>> > >> > >>>>
> >>> > >> > >>>> -Micah
> >>> > >> > >>>>
> >>> > >> > >>>> On Saturday, April 25, 2026, Andrew Lamb <
> >>> andrewlamb11@gmail.com
> >>> > >
> >>> > >> > wrote:
> >>> > >> > >>>>
> >>> > >> > >>>>> Thanks Prateek,
> >>> > >> > >>>>>
> >>> > >> > >>>>> I think from this content it looks to me like we are ready
> >>> to
> >>> > >> start a
> >>> > >> > >>>>> vote to explicitly accept ALP into Parquet
> >>> > >> > >>>>>
> >>> > >> > >>>>> Does anyone know of a reason we should postpone it for
> >>> longer?
> >>> > >> > >>>>> Perhaps someone needs some more time to review?
> >>> > >> > >>>>>
> >>> > >> > >>>>> Andrew
> >>> > >> > >>>>>
> >>> > >> > >>>>>
> >>> > >> > >>>>>
> >>> > >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <
> >>> > prtkgaur@gmail.com>
> >>> > >> > >>>>> wrote:
> >>> > >> > >>>>>
> >>> > >> > >>>>>> Hi team,
> >>> > >> > >>>>>>
> >>> > >> > >>>>>>
> >>> > >> > >>>>>>
> >>> > >> > >>>>>> Hope everyone is doing well. I got a chance to work
> >>> through all
> >>> > >> the
> >>> > >> > >>>>>> remaining feedback and update the spec doc. Here are the
> >>> new
> >>> > >> > artifacts
> >>> > >> > >>>>>>
> >>> > >> > >>>>>> 1) Spec document :
> >>> > >> > >>>>>>
> >>> > >> >
> >>> > >>
> >>> >
> >>>
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
> >>> > >> > >>>>>>
> >>> > >> > >>>>>> 2) Spec document in parquet format repo :
> >>> > >> > >>>>>> https://github.com/apache/parquet-format/pull/557
> >>> > >> > >>>>>>
> >>> > >> > >>>>>> 3) Alp implementation in arrow c++ repo :
> >>> > >> > >>>>>> https://github.com/apache/arrow/pull/48345/changes
> >>> > >> > >>>>>>
> >>> > >> > >>>>>> 4) Alp implementation in parquet-java repo : Work for
> >>> Vinoo and
> >>> > >> > Julien
> >>> > >> > >>>>>> https://github.com/apache/parquet-java/pull/3397
> >>> > >> > >>>>>>
> >>> > >> > >>>>>> 5) PR with test and benchmarking artifacts in
> >>> parquet-testing
> >>> > >> repo :
> >>> > >> > >>>>>> https://github.com/apache/parquet-testing/pull/100
> >>> > >> > >>>>>>
> >>> > >> > >>>>>>
> >>> > >> > >>>>>> And
> >>> > >> > >>>>>>
> >>> > >> > >>>>>>
> >>> > >> > >>>>>> - Go : Arnav just submitted an in progress
> >>> implementation
> >>> > in
> >>> > >> Go.
> >>> > >> > >>>>>> https://github.com/apache/arrow-go/pull/704 (I
> haven't
> >>> > >> started
> >>> > >> > >>>>>> looking at it yet)
> >>> > >> > >>>>>> - Rust : I remember Andrew mentioned that this work
> is
> >>> also
> >>> > >> in
> >>> > >> > >>>>>> progress (So 4 languages!)
> >>> > >> > >>>>>>
> >>> > >> > >>>>>>
> >>> > >> > >>>>>> *Arrow C++ implementation *
> >>> > >> > >>>>>>
> >>> > >> > >>>>>>
> >>> > >> > >>>>>>
> >>> > >> > >>>>>> The PR is out and was also used by Antoine to report the
> >>> > numbers
> >>> > >> as
> >>> > >> > >>>>>> reported here. Micah and Konstantin have given 1 round of
> >>> > >> feedback
> >>> > >> > >>>>>> and I'm addressing them today. Please note that the
> default
> >>> > >> > >>>>>> optimization flag for compiling is O2 and not Q3. I got
> >>> around
> >>> > >> 70%
> >>> > >> > >>>>>> performance improvement in the decoding speed when using
> >>> the O3
> >>> > >> > flag.
> >>> > >> > >>>>>>
> >>> > >> > >>>>>>
> >>> > >> > >>>>>>
> >>> > >> > >>>>>> *Parqet-MR Java implementation (working with Vinoo and
> >>> Julien)
> >>> > >> and
> >>> > >> > **Cross
> >>> > >> > >>>>>> Language testing*
> >>> > >> > >>>>>>
> >>> > >> > >>>>>>
> >>> > >> > >>>>>> Let me know if you have any questions or feedback.
> >>> > >> > >>>>>>
> >>> > >> > >>>>>>
> >>> > >> > >>>>>>
> >>> > >> > >>>>>> Now pasting some performance numbers
> >>> > >> > >>>>>>
> >>> > >> > >>>>>>
> >>> > >> > >>>>>> Table 1: C++ ALP Double Decode — Spotify Columns
> >>> (Graviton
> >>> > 3,
> >>> > >> ARM
> >>> > >> > >>>>>> Neoverse V1)
> >>> > >> > >>>>>>
> >>> > >> > >>>>>>
> >>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
> >>> > >> > >>>>>>
> >>> > >> > >>>>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │
> >>> Speedup │
> >>> > >> > >>>>>>
> >>> > >> > >>>>>>
> >>> ├──────────────────┼──────────────┼──────────────┼─────────┤
> >>> > >> > >>>>>>
> >>> > >> > >>>>>> │ valence │ 3,155 │ 5,523 │
> >>> 1.75x │
> >>> > >> > >>>>>>
> >>> > >> > >>>>>> │ danceability │ 3,233 │ 5,685 │
> >>> 1.76x │
> >>> > >> > >>>>>>
> >>> > >> > >>>>>> │ energy │ 3,197 │ 5,652 │
> >>> 1.77x │
> >>> > >> > >>>>>>
> >>> > >> > >>>>>> │ loudness │ 3,186 │ 5,473 │
> >>> 1.72x │
> >>> > >> > >>>>>>
> >>> > >> > >>>>>>
> >>> └──────────────────┴──────────────┴──────────────┴─────────┘
> >>> > >> > >>>>>>
> >>> > >> > >>>>>>
> >>> > >> > >>>>>>
> >>> > >> > >>>>>>
> >>> > >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <
> >>> > prtkgaur@gmail.com
> >>> > >> >
> >>> > >> > >>>>>> wrote:
> >>> > >> > >>>>>>
> >>> > >> > >>>>>>> @Micah Kornfield <em...@gmail.com> : Got it.
> >>> > >> > >>>>>>>
> >>> > >> > >>>>>>> @Andrew Lamb <an...@gmail.com>
> >>> > >> > >>>>>>>
> >>> > >> > >>>>>>>
> >>> > >> > >>>>>>>> Do you think it would be good to start moving the spec
> >>> > >> development
> >>> > >> > >>>>>>>> into
> >>> > >> > >>>>>>>> markdown format, in preparation for finalizing it?
> >>> > >> > >>>>>>>>
> >>> > >> > >>>>>>>
> >>> > >> > >>>>>>> Yes I'll update the numbers for some of the examples I
> >>> have in
> >>> > >> the
> >>> > >> > >>>>>>> spec based
> >>> > >> > >>>>>>> on the updated header size. Then we should be good to go
> >>> for
> >>> > the
> >>> > >> > >>>>>>> markdown format.
> >>> > >> > >>>>>>>
> >>> > >> > >>>>>>> Thanks everyone!
> >>> > >> > >>>>>>>
> >>> > >> > >>>>>>>
> >>> > >> > >>>>>>>>
> >>> > >> > >>>>>>>> Andrew
> >>> > >> > >>>>>>>>
> >>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <
> >>> > >> prtkgaur@gmail.com>
> >>> > >> > >>>>>>>> wrote:
> >>> > >> > >>>>>>>>
> >>> > >> > >>>>>>>>> Hi team,
> >>> > >> > >>>>>>>>>
> >>> > >> > >>>>>>>>> 1) Andrew
> >>> > >> > >>>>>>>>>
> >>> > >> > >>>>>>>>> - Thanks for working on test files. My PR did add
> >>> all
> >>> > the
> >>> > >> > test
> >>> > >> > >>>>>>>> files I
> >>> > >> > >>>>>>>>> used to benchmark on datasets. Maybe we can club
> it
> >>> > >> together.
> >>> > >> > >>>>>>>> WIll also
> >>> > >> > >>>>>>>>> aid
> >>> > >> > >>>>>>>>> cross language testing
> >>> > >> > >>>>>>>>> - Kosta Tarasov working on Rust implementation.
> >>> This is
> >>> > >> > great.
> >>> > >> > >>>>>>>> Thanks
> >>> > >> > >>>>>>>>>
> >>> > >> > >>>>>>>>>
> >>> > >> > >>>>>>>>> 2) Antoine
> >>> > >> > >>>>>>>>>
> >>> > >> > >>>>>>>>> - Thanks a lot for reporting the numbers on AMD.
> >>> Looks
> >>> > >> like
> >>> > >> > you
> >>> > >> > >>>>>>>> are
> >>> > >> > >>>>>>>>> getting 8X the decoding performance of BSS. This
> is
> >>> > >> > amazing!!.
> >>> > >> > >>>>>>>>> - Thanks for acknowledging the sampling design.
> >>> > >> > >>>>>>>>> - I agree with you on Fastlanes. In some crude
> >>> > >> experiments I
> >>> > >> > >>>>>>>> didn't get
> >>> > >> > >>>>>>>>> a good perf benefit from it on Graviton3 (but
> maybe
> >>> > there
> >>> > >> was
> >>> > >> > >>>>>>>> something
> >>> > >> > >>>>>>>>> wrong with my implementation).
> >>> > >> > >>>>>>>>> - Locking the 16bit exception encoding for the
> spec
> >>> in
> >>> > >> this
> >>> > >> > >>>>>>>> case.
> >>> > >> > >>>>>>>>> - Awesome I think we have solved for all open
> >>> questions
> >>> > >> minus
> >>> > >> > >>>>>>>> the
> >>> > >> > >>>>>>>>> version byte :). (will get back on this soon)
> >>> > >> > >>>>>>>>>
> >>> > >> > >>>>>>>>>
> >>> > >> > >>>>>>>>> 3) Micah
> >>> > >> > >>>>>>>>>
> >>> > >> > >>>>>>>>> - FastLanes : The current spec does allow for
> using
> >>> > >> FastLane
> >>> > >> > >>>>>>>> with the
> >>> > >> > >>>>>>>>> configurable enum value for layout. We should be
> >>> able to
> >>> > >> > inject
> >>> > >> > >>>>>>>> any
> >>> > >> > >>>>>>>>> layout
> >>> > >> > >>>>>>>>> in the current design.
> >>> > >> > >>>>>>>>>
> >>> > >> > >>>>>>>>>
> >>> > >> > >>>>>>>>> Working on resolving all remaining open comments on
> the
> >>> spec
> >>> > >> this
> >>> > >> > >>>>>>>> week.
> >>> > >> > >>>>>>>>>
> >>> > >> > >>>>>>>>> Best
> >>> > >> > >>>>>>>>> Prateek
> >>> > >> > >>>>>>>>>
> >>> > >> > >>>>>>>>>
> >>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <
> >>> > >> > >>>>>>>> stevel@cloudera.com>
> >>> > >> > >>>>>>>>> wrote:
> >>> > >> > >>>>>>>>>
> >>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
> >>> > >> > >>>>>>>> emkornfield@gmail.com>
> >>> > >> > >>>>>>>>>> wrote:
> >>> > >> > >>>>>>>>>>
> >>> > >> > >>>>>>>>>>>
> >>> > >> > >>>>>>>>>>>
> >>> > >> > >>>>>>>>>>> It looks like the actual issue described for ORC in
> >>> the
> >>> > >> paper
> >>> > >> > >>>>>>>> is that
> >>> > >> > >>>>>>>>> it
> >>> > >> > >>>>>>>>>>> has multiple sub-encodings in a batch. This is
> >>> different
> >>> > >> then
> >>> > >> > >>>>>>>> the
> >>> > >> > >>>>>>>>> design
> >>> > >> > >>>>>>>>>>> proposed here where there is still fixed encoding
> per
> >>> page
> >>> > >> in
> >>> > >> > >>>>>>>> parquet.
> >>> > >> > >>>>>>>>>>> Given reasonably sized pages I don't think branch
> >>> > >> > >>>>>>>> misprediction should
> >>> > >> > >>>>>>>>>> be a
> >>> > >> > >>>>>>>>>>> big issue for new encodings. I agree that we should
> >>> be
> >>> > >> > >>>>>>>> conservative in
> >>> > >> > >>>>>>>>>>> general for adding new encodings.
> >>> > >> > >>>>>>>>>>>
> >>> > >> > >>>>>>>>>>>
> >>> > >> > >>>>>>>>>> +1
> >>> > >> > >>>>>>>>>>
> >>> > >> > >>>>>>>>>
> >>> > >> > >>>>>>>>
> >>> > >> > >>>>>>>
> >>> > >> > >
> >>> > >> >
> >>> > >> >
> >>> > >> >
> >>> > >>
> >>> > >
> >>> >
> >>>
> >>
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by PRATEEK GAUR <pr...@gmail.com>.
Okay,
I sent out the voting email yesterday and so far I've heard from 3 people
that it ended up in their "spam folder". Dev VPN to be blamed for it?.
What's the protocol here?
I let people hunt for the email in their spam folder or send out another
one?
Best
Prateek
On Sun, Jul 19, 2026 at 10:47 PM PRATEEK GAUR <pr...@gmail.com> wrote:
> Hi team,
>
> Vinoo has added the tests which were requested in the last round of review.
> In his own words.
>
> "
> The extreme value tests Micah wanted are done and pushed. I added coverage
> for values that need the full FOR bit width after the frame of reference is
> applied: 64-bit (the case where the signed max minus min subtraction
> overflows), 63-bit (the non-overflow case), and 32-bit for floats. All
> lossless with zero exceptions. I also verified them through Arrow C++ ALP
> reader (the local build), so the Java-written extreme columns decode
> correctly in C++, bit-exact against the expected values."
>
> State of the PR (#3397):
>
> - Merges cleanly with master (I pulled master in and resolved the
> conflicts).
> - 41 of 44 review threads resolved. The 3 remaining are intentional:
> one deferred perf nit, one that stays open until parquet-format ships ALP,
>
> "
>
> With this both the
> 1) C++ PR : https://github.com/apache/arrow/pull/48345/changes
> 2) Java PR : https://github.com/apache/parquet-java/pull/3397
>
> Are cross language tested and range of test scenarios.
>
> Best
> Prateek
>
>
> On Mon, Jul 13, 2026 at 8:38 AM PRATEEK GAUR <pr...@gmail.com> wrote:
>
>> Thanks Matt,
>>
>> I'll be on it this week.
>>
>> Best
>> Prateek
>>
>> On Mon, Jul 13, 2026 at 8:36 AM Matt Topol <zo...@gmail.com>
>> wrote:
>>
>>> There is also the Go implementation submitted by Arnav (
>>> https://github.com/apache/arrow-go/pull/704) which is waiting for
>>> updates.
>>>
>>> Just wanted to make sure it didn't get lost here.
>>>
>>> --Matt
>>>
>>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK GAUR <pr...@gmail.com>
>>> wrote:
>>>
>>> > Hi all,
>>> >
>>> > I'd like to share a status update on the ALP encoding effort and get a
>>> feel
>>> > from the community before starting a formal vote.
>>> >
>>> > We now have the specification plus implementations in both languages:
>>> >
>>> > - Spec: apache/parquet-format#557 — GH-533 Add ALP encoding
>>> > specification
>>> > - C++ (Arrow): apache/arrow#48345 — GH-48701 [C++][Parquet] Add
>>> ALPpd
>>> > encoding
>>> > - Java: apache/parquet-java#3397 — Parquet Java ALP
>>> > Implementation (by Vinoo)
>>> >
>>> > Current state:
>>> >
>>> > - The C++ implementation has been through 3-4 rounds of review.
>>> > - The Java implementation has been tracking well and has had
>>> initial
>>> > reviews from contributors, with no major remaining issues.
>>> > - Cross-language compatibility tests are passing: the Arrow C++
>>> decoder
>>> > reads Java-written data bit-exactly across ~1.56M values and 18
>>> > fixtures, covering V1 and V2 pages, multiple vector sizes, and
>>> > several
>>> > real datasets — zero mismatches.
>>> >
>>> > As raised in the java-pr we will be adding coverage for extreme values
>>> > (those requiring
>>> > 63-64 bits after FOR is applied) before we close a vote. We'll aim to
>>> get
>>> > that
>>> > done in parallel.
>>> >
>>> > Plan: unless there are objections, I intend to start a vote at the end
>>> of
>>> > this week/early
>>> > next week.
>>> >
>>> > Thanks
>>> > Prateek
>>> >
>>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <pr...@gmail.com>
>>> wrote:
>>> >
>>> > > Hi Team,
>>> > >
>>> > > Just wanted to provide some updates on ALP.
>>> > > Micah and I have done 3-4 rounds of review for the c++ PR :
>>> > > https://github.com/apache/arrow/pull/48345
>>> > >
>>> > > For the Java implementation Vinoo has been working on following PR :
>>> > > https://github.com/apache/parquet-java/pull/3397
>>> > >
>>> > > Best
>>> > > Prateek
>>> > >
>>> > > On Tue, May 5, 2026 at 1:50 PM Micah Kornfield <
>>> emkornfield@gmail.com>
>>> > > wrote:
>>> > >
>>> > >> Hi Antoine,
>>> > >>
>>> > >> > Apologies if the question was already asked, but should we care
>>> about
>>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
>>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
>>> > >>
>>> > >> It was. We thought we could defer it for the following reasons:
>>> > >> 1. It's not clear there are a lot of easy reference datasets to
>>> test
>>> > its
>>> > >> effectiveness. It does look like there might be one or two on
>>> > huggingface
>>> > >> (e.g. https://huggingface.co/datasets/kikitora/curdie).
>>> > >> 2. It seemed likely that float 16 was more likely used for values
>>> that
>>> > >> were less likely to reduce to decimal values.
>>> > >> 3. It could be added as an extension later if needed.
>>> > >>
>>> > >> Cheers,
>>> > >> Micah
>>> > >>
>>> > >>
>>> > >> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <an...@python.org>
>>> > wrote:
>>> > >>
>>> > >> >
>>> > >> > Hello,
>>> > >> >
>>> > >> > Apologies if the question was already asked, but should we care
>>> about
>>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
>>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
>>> > >> >
>>> > >> > Regards
>>> > >> >
>>> > >> > Antoine.
>>> > >> >
>>> > >> >
>>> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
>>> > >> > > Thanks Andrew and Micah for review feedback on the two PR's
>>> > >> > > 1) (c++ arrow repo)
>>> > >> https://github.com/apache/arrow/pull/48345/changes
>>> > >> > > 2) (parquet-format repo)
>>> > >> > https://github.com/apache/parquet-format/pull/557
>>> > >> > >
>>> > >> > > I have addressed all (unless I missed something) comments on
>>> the two
>>> > >> > PR's.
>>> > >> > >
>>> > >> > > Best
>>> > >> > > Prateek
>>> > >> > >
>>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <
>>> prtkgaur@gmail.com>
>>> > >> wrote:
>>> > >> > >
>>> > >> > >> Thanks Andrew and Micah.
>>> > >> > >>
>>> > >> > >> `fair amount of feedback on at least the implementations`
>>> > >> > >> For the c++ I have already started addressing the feedback, I
>>> > should
>>> > >> be
>>> > >> > >> done with that Monday/Tuesday.
>>> > >> > >> I think Vinoo too has been making good progress on the Java
>>> > >> > implementation.
>>> > >> > >>
>>> > >> > >> Best
>>> > >> > >> Prateek
>>> > >> > >>
>>> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <
>>> > andrewlamb11@gmail.com
>>> > >> >
>>> > >> > >> wrote:
>>> > >> > >>
>>> > >> > >>> Got it. Thank you for the clarification -- I will try and look
>>> > into
>>> > >> the
>>> > >> > >>> spec and the Rust implementation[1] in this next week
>>> > >> > >>>
>>> > >> > >>> [1]: https://github.com/apache/arrow-rs/pull/9372
>>> > >> > >>>
>>> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <
>>> > >> > emkornfield@gmail.com>
>>> > >> > >>> wrote:
>>> > >> > >>>
>>> > >> > >>>> Hi Andrew,
>>> > >> > >>>> I think there is a fair amount of feedback on at least the
>>> > >> > >>>> implementations, typically I think we've waited till they are
>>> > >> close to
>>> > >> > >>>> mergeable before a final vote. Otherwise I agree we are very
>>> > >> close.
>>> > >> > >>>>
>>> > >> > >>>> -Micah
>>> > >> > >>>>
>>> > >> > >>>> On Saturday, April 25, 2026, Andrew Lamb <
>>> andrewlamb11@gmail.com
>>> > >
>>> > >> > wrote:
>>> > >> > >>>>
>>> > >> > >>>>> Thanks Prateek,
>>> > >> > >>>>>
>>> > >> > >>>>> I think from this content it looks to me like we are ready
>>> to
>>> > >> start a
>>> > >> > >>>>> vote to explicitly accept ALP into Parquet
>>> > >> > >>>>>
>>> > >> > >>>>> Does anyone know of a reason we should postpone it for
>>> longer?
>>> > >> > >>>>> Perhaps someone needs some more time to review?
>>> > >> > >>>>>
>>> > >> > >>>>> Andrew
>>> > >> > >>>>>
>>> > >> > >>>>>
>>> > >> > >>>>>
>>> > >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <
>>> > prtkgaur@gmail.com>
>>> > >> > >>>>> wrote:
>>> > >> > >>>>>
>>> > >> > >>>>>> Hi team,
>>> > >> > >>>>>>
>>> > >> > >>>>>>
>>> > >> > >>>>>>
>>> > >> > >>>>>> Hope everyone is doing well. I got a chance to work
>>> through all
>>> > >> the
>>> > >> > >>>>>> remaining feedback and update the spec doc. Here are the
>>> new
>>> > >> > artifacts
>>> > >> > >>>>>>
>>> > >> > >>>>>> 1) Spec document :
>>> > >> > >>>>>>
>>> > >> >
>>> > >>
>>> >
>>> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
>>> > >> > >>>>>>
>>> > >> > >>>>>> 2) Spec document in parquet format repo :
>>> > >> > >>>>>> https://github.com/apache/parquet-format/pull/557
>>> > >> > >>>>>>
>>> > >> > >>>>>> 3) Alp implementation in arrow c++ repo :
>>> > >> > >>>>>> https://github.com/apache/arrow/pull/48345/changes
>>> > >> > >>>>>>
>>> > >> > >>>>>> 4) Alp implementation in parquet-java repo : Work for
>>> Vinoo and
>>> > >> > Julien
>>> > >> > >>>>>> https://github.com/apache/parquet-java/pull/3397
>>> > >> > >>>>>>
>>> > >> > >>>>>> 5) PR with test and benchmarking artifacts in
>>> parquet-testing
>>> > >> repo :
>>> > >> > >>>>>> https://github.com/apache/parquet-testing/pull/100
>>> > >> > >>>>>>
>>> > >> > >>>>>>
>>> > >> > >>>>>> And
>>> > >> > >>>>>>
>>> > >> > >>>>>>
>>> > >> > >>>>>> - Go : Arnav just submitted an in progress
>>> implementation
>>> > in
>>> > >> Go.
>>> > >> > >>>>>> https://github.com/apache/arrow-go/pull/704 (I haven't
>>> > >> started
>>> > >> > >>>>>> looking at it yet)
>>> > >> > >>>>>> - Rust : I remember Andrew mentioned that this work is
>>> also
>>> > >> in
>>> > >> > >>>>>> progress (So 4 languages!)
>>> > >> > >>>>>>
>>> > >> > >>>>>>
>>> > >> > >>>>>> *Arrow C++ implementation *
>>> > >> > >>>>>>
>>> > >> > >>>>>>
>>> > >> > >>>>>>
>>> > >> > >>>>>> The PR is out and was also used by Antoine to report the
>>> > numbers
>>> > >> as
>>> > >> > >>>>>> reported here. Micah and Konstantin have given 1 round of
>>> > >> feedback
>>> > >> > >>>>>> and I'm addressing them today. Please note that the default
>>> > >> > >>>>>> optimization flag for compiling is O2 and not Q3. I got
>>> around
>>> > >> 70%
>>> > >> > >>>>>> performance improvement in the decoding speed when using
>>> the O3
>>> > >> > flag.
>>> > >> > >>>>>>
>>> > >> > >>>>>>
>>> > >> > >>>>>>
>>> > >> > >>>>>> *Parqet-MR Java implementation (working with Vinoo and
>>> Julien)
>>> > >> and
>>> > >> > **Cross
>>> > >> > >>>>>> Language testing*
>>> > >> > >>>>>>
>>> > >> > >>>>>>
>>> > >> > >>>>>> Let me know if you have any questions or feedback.
>>> > >> > >>>>>>
>>> > >> > >>>>>>
>>> > >> > >>>>>>
>>> > >> > >>>>>> Now pasting some performance numbers
>>> > >> > >>>>>>
>>> > >> > >>>>>>
>>> > >> > >>>>>> Table 1: C++ ALP Double Decode — Spotify Columns
>>> (Graviton
>>> > 3,
>>> > >> ARM
>>> > >> > >>>>>> Neoverse V1)
>>> > >> > >>>>>>
>>> > >> > >>>>>>
>>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
>>> > >> > >>>>>>
>>> > >> > >>>>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │
>>> Speedup │
>>> > >> > >>>>>>
>>> > >> > >>>>>>
>>> ├──────────────────┼──────────────┼──────────────┼─────────┤
>>> > >> > >>>>>>
>>> > >> > >>>>>> │ valence │ 3,155 │ 5,523 │
>>> 1.75x │
>>> > >> > >>>>>>
>>> > >> > >>>>>> │ danceability │ 3,233 │ 5,685 │
>>> 1.76x │
>>> > >> > >>>>>>
>>> > >> > >>>>>> │ energy │ 3,197 │ 5,652 │
>>> 1.77x │
>>> > >> > >>>>>>
>>> > >> > >>>>>> │ loudness │ 3,186 │ 5,473 │
>>> 1.72x │
>>> > >> > >>>>>>
>>> > >> > >>>>>>
>>> └──────────────────┴──────────────┴──────────────┴─────────┘
>>> > >> > >>>>>>
>>> > >> > >>>>>>
>>> > >> > >>>>>>
>>> > >> > >>>>>>
>>> > >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <
>>> > prtkgaur@gmail.com
>>> > >> >
>>> > >> > >>>>>> wrote:
>>> > >> > >>>>>>
>>> > >> > >>>>>>> @Micah Kornfield <em...@gmail.com> : Got it.
>>> > >> > >>>>>>>
>>> > >> > >>>>>>> @Andrew Lamb <an...@gmail.com>
>>> > >> > >>>>>>>
>>> > >> > >>>>>>>
>>> > >> > >>>>>>>> Do you think it would be good to start moving the spec
>>> > >> development
>>> > >> > >>>>>>>> into
>>> > >> > >>>>>>>> markdown format, in preparation for finalizing it?
>>> > >> > >>>>>>>>
>>> > >> > >>>>>>>
>>> > >> > >>>>>>> Yes I'll update the numbers for some of the examples I
>>> have in
>>> > >> the
>>> > >> > >>>>>>> spec based
>>> > >> > >>>>>>> on the updated header size. Then we should be good to go
>>> for
>>> > the
>>> > >> > >>>>>>> markdown format.
>>> > >> > >>>>>>>
>>> > >> > >>>>>>> Thanks everyone!
>>> > >> > >>>>>>>
>>> > >> > >>>>>>>
>>> > >> > >>>>>>>>
>>> > >> > >>>>>>>> Andrew
>>> > >> > >>>>>>>>
>>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <
>>> > >> prtkgaur@gmail.com>
>>> > >> > >>>>>>>> wrote:
>>> > >> > >>>>>>>>
>>> > >> > >>>>>>>>> Hi team,
>>> > >> > >>>>>>>>>
>>> > >> > >>>>>>>>> 1) Andrew
>>> > >> > >>>>>>>>>
>>> > >> > >>>>>>>>> - Thanks for working on test files. My PR did add
>>> all
>>> > the
>>> > >> > test
>>> > >> > >>>>>>>> files I
>>> > >> > >>>>>>>>> used to benchmark on datasets. Maybe we can club it
>>> > >> together.
>>> > >> > >>>>>>>> WIll also
>>> > >> > >>>>>>>>> aid
>>> > >> > >>>>>>>>> cross language testing
>>> > >> > >>>>>>>>> - Kosta Tarasov working on Rust implementation.
>>> This is
>>> > >> > great.
>>> > >> > >>>>>>>> Thanks
>>> > >> > >>>>>>>>>
>>> > >> > >>>>>>>>>
>>> > >> > >>>>>>>>> 2) Antoine
>>> > >> > >>>>>>>>>
>>> > >> > >>>>>>>>> - Thanks a lot for reporting the numbers on AMD.
>>> Looks
>>> > >> like
>>> > >> > you
>>> > >> > >>>>>>>> are
>>> > >> > >>>>>>>>> getting 8X the decoding performance of BSS. This is
>>> > >> > amazing!!.
>>> > >> > >>>>>>>>> - Thanks for acknowledging the sampling design.
>>> > >> > >>>>>>>>> - I agree with you on Fastlanes. In some crude
>>> > >> experiments I
>>> > >> > >>>>>>>> didn't get
>>> > >> > >>>>>>>>> a good perf benefit from it on Graviton3 (but maybe
>>> > there
>>> > >> was
>>> > >> > >>>>>>>> something
>>> > >> > >>>>>>>>> wrong with my implementation).
>>> > >> > >>>>>>>>> - Locking the 16bit exception encoding for the spec
>>> in
>>> > >> this
>>> > >> > >>>>>>>> case.
>>> > >> > >>>>>>>>> - Awesome I think we have solved for all open
>>> questions
>>> > >> minus
>>> > >> > >>>>>>>> the
>>> > >> > >>>>>>>>> version byte :). (will get back on this soon)
>>> > >> > >>>>>>>>>
>>> > >> > >>>>>>>>>
>>> > >> > >>>>>>>>> 3) Micah
>>> > >> > >>>>>>>>>
>>> > >> > >>>>>>>>> - FastLanes : The current spec does allow for using
>>> > >> FastLane
>>> > >> > >>>>>>>> with the
>>> > >> > >>>>>>>>> configurable enum value for layout. We should be
>>> able to
>>> > >> > inject
>>> > >> > >>>>>>>> any
>>> > >> > >>>>>>>>> layout
>>> > >> > >>>>>>>>> in the current design.
>>> > >> > >>>>>>>>>
>>> > >> > >>>>>>>>>
>>> > >> > >>>>>>>>> Working on resolving all remaining open comments on the
>>> spec
>>> > >> this
>>> > >> > >>>>>>>> week.
>>> > >> > >>>>>>>>>
>>> > >> > >>>>>>>>> Best
>>> > >> > >>>>>>>>> Prateek
>>> > >> > >>>>>>>>>
>>> > >> > >>>>>>>>>
>>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <
>>> > >> > >>>>>>>> stevel@cloudera.com>
>>> > >> > >>>>>>>>> wrote:
>>> > >> > >>>>>>>>>
>>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
>>> > >> > >>>>>>>> emkornfield@gmail.com>
>>> > >> > >>>>>>>>>> wrote:
>>> > >> > >>>>>>>>>>
>>> > >> > >>>>>>>>>>>
>>> > >> > >>>>>>>>>>>
>>> > >> > >>>>>>>>>>> It looks like the actual issue described for ORC in
>>> the
>>> > >> paper
>>> > >> > >>>>>>>> is that
>>> > >> > >>>>>>>>> it
>>> > >> > >>>>>>>>>>> has multiple sub-encodings in a batch. This is
>>> different
>>> > >> then
>>> > >> > >>>>>>>> the
>>> > >> > >>>>>>>>> design
>>> > >> > >>>>>>>>>>> proposed here where there is still fixed encoding per
>>> page
>>> > >> in
>>> > >> > >>>>>>>> parquet.
>>> > >> > >>>>>>>>>>> Given reasonably sized pages I don't think branch
>>> > >> > >>>>>>>> misprediction should
>>> > >> > >>>>>>>>>> be a
>>> > >> > >>>>>>>>>>> big issue for new encodings. I agree that we should
>>> be
>>> > >> > >>>>>>>> conservative in
>>> > >> > >>>>>>>>>>> general for adding new encodings.
>>> > >> > >>>>>>>>>>>
>>> > >> > >>>>>>>>>>>
>>> > >> > >>>>>>>>>> +1
>>> > >> > >>>>>>>>>>
>>> > >> > >>>>>>>>>
>>> > >> > >>>>>>>>
>>> > >> > >>>>>>>
>>> > >> > >
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >>
>>> > >
>>> >
>>>
>>
Re: [Parquet] ALP Encoding for Floating point data
Posted by PRATEEK GAUR <pr...@gmail.com>.
Hi team,
Vinoo has added the tests which were requested in the last round of review.
In his own words.
"
The extreme value tests Micah wanted are done and pushed. I added coverage
for values that need the full FOR bit width after the frame of reference is
applied: 64-bit (the case where the signed max minus min subtraction
overflows), 63-bit (the non-overflow case), and 32-bit for floats. All
lossless with zero exceptions. I also verified them through Arrow C++ ALP
reader (the local build), so the Java-written extreme columns decode
correctly in C++, bit-exact against the expected values."
State of the PR (#3397):
- Merges cleanly with master (I pulled master in and resolved the
conflicts).
- 41 of 44 review threads resolved. The 3 remaining are intentional: one
deferred perf nit, one that stays open until parquet-format ships ALP,
"
With this both the
1) C++ PR : https://github.com/apache/arrow/pull/48345/changes
2) Java PR : https://github.com/apache/parquet-java/pull/3397
Are cross language tested and range of test scenarios.
Best
Prateek
On Mon, Jul 13, 2026 at 8:38 AM PRATEEK GAUR <pr...@gmail.com> wrote:
> Thanks Matt,
>
> I'll be on it this week.
>
> Best
> Prateek
>
> On Mon, Jul 13, 2026 at 8:36 AM Matt Topol <zo...@gmail.com> wrote:
>
>> There is also the Go implementation submitted by Arnav (
>> https://github.com/apache/arrow-go/pull/704) which is waiting for
>> updates.
>>
>> Just wanted to make sure it didn't get lost here.
>>
>> --Matt
>>
>> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK GAUR <pr...@gmail.com> wrote:
>>
>> > Hi all,
>> >
>> > I'd like to share a status update on the ALP encoding effort and get a
>> feel
>> > from the community before starting a formal vote.
>> >
>> > We now have the specification plus implementations in both languages:
>> >
>> > - Spec: apache/parquet-format#557 — GH-533 Add ALP encoding
>> > specification
>> > - C++ (Arrow): apache/arrow#48345 — GH-48701 [C++][Parquet] Add
>> ALPpd
>> > encoding
>> > - Java: apache/parquet-java#3397 — Parquet Java ALP
>> > Implementation (by Vinoo)
>> >
>> > Current state:
>> >
>> > - The C++ implementation has been through 3-4 rounds of review.
>> > - The Java implementation has been tracking well and has had initial
>> > reviews from contributors, with no major remaining issues.
>> > - Cross-language compatibility tests are passing: the Arrow C++
>> decoder
>> > reads Java-written data bit-exactly across ~1.56M values and 18
>> > fixtures, covering V1 and V2 pages, multiple vector sizes, and
>> > several
>> > real datasets — zero mismatches.
>> >
>> > As raised in the java-pr we will be adding coverage for extreme values
>> > (those requiring
>> > 63-64 bits after FOR is applied) before we close a vote. We'll aim to
>> get
>> > that
>> > done in parallel.
>> >
>> > Plan: unless there are objections, I intend to start a vote at the end
>> of
>> > this week/early
>> > next week.
>> >
>> > Thanks
>> > Prateek
>> >
>> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <pr...@gmail.com> wrote:
>> >
>> > > Hi Team,
>> > >
>> > > Just wanted to provide some updates on ALP.
>> > > Micah and I have done 3-4 rounds of review for the c++ PR :
>> > > https://github.com/apache/arrow/pull/48345
>> > >
>> > > For the Java implementation Vinoo has been working on following PR :
>> > > https://github.com/apache/parquet-java/pull/3397
>> > >
>> > > Best
>> > > Prateek
>> > >
>> > > On Tue, May 5, 2026 at 1:50 PM Micah Kornfield <emkornfield@gmail.com
>> >
>> > > wrote:
>> > >
>> > >> Hi Antoine,
>> > >>
>> > >> > Apologies if the question was already asked, but should we care
>> about
>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
>> > >>
>> > >> It was. We thought we could defer it for the following reasons:
>> > >> 1. It's not clear there are a lot of easy reference datasets to test
>> > its
>> > >> effectiveness. It does look like there might be one or two on
>> > huggingface
>> > >> (e.g. https://huggingface.co/datasets/kikitora/curdie).
>> > >> 2. It seemed likely that float 16 was more likely used for values
>> that
>> > >> were less likely to reduce to decimal values.
>> > >> 3. It could be added as an extension later if needed.
>> > >>
>> > >> Cheers,
>> > >> Micah
>> > >>
>> > >>
>> > >> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <an...@python.org>
>> > wrote:
>> > >>
>> > >> >
>> > >> > Hello,
>> > >> >
>> > >> > Apologies if the question was already asked, but should we care
>> about
>> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
>> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
>> > >> >
>> > >> > Regards
>> > >> >
>> > >> > Antoine.
>> > >> >
>> > >> >
>> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
>> > >> > > Thanks Andrew and Micah for review feedback on the two PR's
>> > >> > > 1) (c++ arrow repo)
>> > >> https://github.com/apache/arrow/pull/48345/changes
>> > >> > > 2) (parquet-format repo)
>> > >> > https://github.com/apache/parquet-format/pull/557
>> > >> > >
>> > >> > > I have addressed all (unless I missed something) comments on the
>> two
>> > >> > PR's.
>> > >> > >
>> > >> > > Best
>> > >> > > Prateek
>> > >> > >
>> > >> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <prtkgaur@gmail.com
>> >
>> > >> wrote:
>> > >> > >
>> > >> > >> Thanks Andrew and Micah.
>> > >> > >>
>> > >> > >> `fair amount of feedback on at least the implementations`
>> > >> > >> For the c++ I have already started addressing the feedback, I
>> > should
>> > >> be
>> > >> > >> done with that Monday/Tuesday.
>> > >> > >> I think Vinoo too has been making good progress on the Java
>> > >> > implementation.
>> > >> > >>
>> > >> > >> Best
>> > >> > >> Prateek
>> > >> > >>
>> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <
>> > andrewlamb11@gmail.com
>> > >> >
>> > >> > >> wrote:
>> > >> > >>
>> > >> > >>> Got it. Thank you for the clarification -- I will try and look
>> > into
>> > >> the
>> > >> > >>> spec and the Rust implementation[1] in this next week
>> > >> > >>>
>> > >> > >>> [1]: https://github.com/apache/arrow-rs/pull/9372
>> > >> > >>>
>> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <
>> > >> > emkornfield@gmail.com>
>> > >> > >>> wrote:
>> > >> > >>>
>> > >> > >>>> Hi Andrew,
>> > >> > >>>> I think there is a fair amount of feedback on at least the
>> > >> > >>>> implementations, typically I think we've waited till they are
>> > >> close to
>> > >> > >>>> mergeable before a final vote. Otherwise I agree we are very
>> > >> close.
>> > >> > >>>>
>> > >> > >>>> -Micah
>> > >> > >>>>
>> > >> > >>>> On Saturday, April 25, 2026, Andrew Lamb <
>> andrewlamb11@gmail.com
>> > >
>> > >> > wrote:
>> > >> > >>>>
>> > >> > >>>>> Thanks Prateek,
>> > >> > >>>>>
>> > >> > >>>>> I think from this content it looks to me like we are ready to
>> > >> start a
>> > >> > >>>>> vote to explicitly accept ALP into Parquet
>> > >> > >>>>>
>> > >> > >>>>> Does anyone know of a reason we should postpone it for
>> longer?
>> > >> > >>>>> Perhaps someone needs some more time to review?
>> > >> > >>>>>
>> > >> > >>>>> Andrew
>> > >> > >>>>>
>> > >> > >>>>>
>> > >> > >>>>>
>> > >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <
>> > prtkgaur@gmail.com>
>> > >> > >>>>> wrote:
>> > >> > >>>>>
>> > >> > >>>>>> Hi team,
>> > >> > >>>>>>
>> > >> > >>>>>>
>> > >> > >>>>>>
>> > >> > >>>>>> Hope everyone is doing well. I got a chance to work through
>> all
>> > >> the
>> > >> > >>>>>> remaining feedback and update the spec doc. Here are the new
>> > >> > artifacts
>> > >> > >>>>>>
>> > >> > >>>>>> 1) Spec document :
>> > >> > >>>>>>
>> > >> >
>> > >>
>> >
>> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
>> > >> > >>>>>>
>> > >> > >>>>>> 2) Spec document in parquet format repo :
>> > >> > >>>>>> https://github.com/apache/parquet-format/pull/557
>> > >> > >>>>>>
>> > >> > >>>>>> 3) Alp implementation in arrow c++ repo :
>> > >> > >>>>>> https://github.com/apache/arrow/pull/48345/changes
>> > >> > >>>>>>
>> > >> > >>>>>> 4) Alp implementation in parquet-java repo : Work for Vinoo
>> and
>> > >> > Julien
>> > >> > >>>>>> https://github.com/apache/parquet-java/pull/3397
>> > >> > >>>>>>
>> > >> > >>>>>> 5) PR with test and benchmarking artifacts in
>> parquet-testing
>> > >> repo :
>> > >> > >>>>>> https://github.com/apache/parquet-testing/pull/100
>> > >> > >>>>>>
>> > >> > >>>>>>
>> > >> > >>>>>> And
>> > >> > >>>>>>
>> > >> > >>>>>>
>> > >> > >>>>>> - Go : Arnav just submitted an in progress
>> implementation
>> > in
>> > >> Go.
>> > >> > >>>>>> https://github.com/apache/arrow-go/pull/704 (I haven't
>> > >> started
>> > >> > >>>>>> looking at it yet)
>> > >> > >>>>>> - Rust : I remember Andrew mentioned that this work is
>> also
>> > >> in
>> > >> > >>>>>> progress (So 4 languages!)
>> > >> > >>>>>>
>> > >> > >>>>>>
>> > >> > >>>>>> *Arrow C++ implementation *
>> > >> > >>>>>>
>> > >> > >>>>>>
>> > >> > >>>>>>
>> > >> > >>>>>> The PR is out and was also used by Antoine to report the
>> > numbers
>> > >> as
>> > >> > >>>>>> reported here. Micah and Konstantin have given 1 round of
>> > >> feedback
>> > >> > >>>>>> and I'm addressing them today. Please note that the default
>> > >> > >>>>>> optimization flag for compiling is O2 and not Q3. I got
>> around
>> > >> 70%
>> > >> > >>>>>> performance improvement in the decoding speed when using
>> the O3
>> > >> > flag.
>> > >> > >>>>>>
>> > >> > >>>>>>
>> > >> > >>>>>>
>> > >> > >>>>>> *Parqet-MR Java implementation (working with Vinoo and
>> Julien)
>> > >> and
>> > >> > **Cross
>> > >> > >>>>>> Language testing*
>> > >> > >>>>>>
>> > >> > >>>>>>
>> > >> > >>>>>> Let me know if you have any questions or feedback.
>> > >> > >>>>>>
>> > >> > >>>>>>
>> > >> > >>>>>>
>> > >> > >>>>>> Now pasting some performance numbers
>> > >> > >>>>>>
>> > >> > >>>>>>
>> > >> > >>>>>> Table 1: C++ ALP Double Decode — Spotify Columns
>> (Graviton
>> > 3,
>> > >> ARM
>> > >> > >>>>>> Neoverse V1)
>> > >> > >>>>>>
>> > >> > >>>>>>
>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
>> > >> > >>>>>>
>> > >> > >>>>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │
>> Speedup │
>> > >> > >>>>>>
>> > >> > >>>>>>
>> ├──────────────────┼──────────────┼──────────────┼─────────┤
>> > >> > >>>>>>
>> > >> > >>>>>> │ valence │ 3,155 │ 5,523 │
>> 1.75x │
>> > >> > >>>>>>
>> > >> > >>>>>> │ danceability │ 3,233 │ 5,685 │
>> 1.76x │
>> > >> > >>>>>>
>> > >> > >>>>>> │ energy │ 3,197 │ 5,652 │
>> 1.77x │
>> > >> > >>>>>>
>> > >> > >>>>>> │ loudness │ 3,186 │ 5,473 │
>> 1.72x │
>> > >> > >>>>>>
>> > >> > >>>>>>
>> └──────────────────┴──────────────┴──────────────┴─────────┘
>> > >> > >>>>>>
>> > >> > >>>>>>
>> > >> > >>>>>>
>> > >> > >>>>>>
>> > >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <
>> > prtkgaur@gmail.com
>> > >> >
>> > >> > >>>>>> wrote:
>> > >> > >>>>>>
>> > >> > >>>>>>> @Micah Kornfield <em...@gmail.com> : Got it.
>> > >> > >>>>>>>
>> > >> > >>>>>>> @Andrew Lamb <an...@gmail.com>
>> > >> > >>>>>>>
>> > >> > >>>>>>>
>> > >> > >>>>>>>> Do you think it would be good to start moving the spec
>> > >> development
>> > >> > >>>>>>>> into
>> > >> > >>>>>>>> markdown format, in preparation for finalizing it?
>> > >> > >>>>>>>>
>> > >> > >>>>>>>
>> > >> > >>>>>>> Yes I'll update the numbers for some of the examples I
>> have in
>> > >> the
>> > >> > >>>>>>> spec based
>> > >> > >>>>>>> on the updated header size. Then we should be good to go
>> for
>> > the
>> > >> > >>>>>>> markdown format.
>> > >> > >>>>>>>
>> > >> > >>>>>>> Thanks everyone!
>> > >> > >>>>>>>
>> > >> > >>>>>>>
>> > >> > >>>>>>>>
>> > >> > >>>>>>>> Andrew
>> > >> > >>>>>>>>
>> > >> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <
>> > >> prtkgaur@gmail.com>
>> > >> > >>>>>>>> wrote:
>> > >> > >>>>>>>>
>> > >> > >>>>>>>>> Hi team,
>> > >> > >>>>>>>>>
>> > >> > >>>>>>>>> 1) Andrew
>> > >> > >>>>>>>>>
>> > >> > >>>>>>>>> - Thanks for working on test files. My PR did add all
>> > the
>> > >> > test
>> > >> > >>>>>>>> files I
>> > >> > >>>>>>>>> used to benchmark on datasets. Maybe we can club it
>> > >> together.
>> > >> > >>>>>>>> WIll also
>> > >> > >>>>>>>>> aid
>> > >> > >>>>>>>>> cross language testing
>> > >> > >>>>>>>>> - Kosta Tarasov working on Rust implementation.
>> This is
>> > >> > great.
>> > >> > >>>>>>>> Thanks
>> > >> > >>>>>>>>>
>> > >> > >>>>>>>>>
>> > >> > >>>>>>>>> 2) Antoine
>> > >> > >>>>>>>>>
>> > >> > >>>>>>>>> - Thanks a lot for reporting the numbers on AMD.
>> Looks
>> > >> like
>> > >> > you
>> > >> > >>>>>>>> are
>> > >> > >>>>>>>>> getting 8X the decoding performance of BSS. This is
>> > >> > amazing!!.
>> > >> > >>>>>>>>> - Thanks for acknowledging the sampling design.
>> > >> > >>>>>>>>> - I agree with you on Fastlanes. In some crude
>> > >> experiments I
>> > >> > >>>>>>>> didn't get
>> > >> > >>>>>>>>> a good perf benefit from it on Graviton3 (but maybe
>> > there
>> > >> was
>> > >> > >>>>>>>> something
>> > >> > >>>>>>>>> wrong with my implementation).
>> > >> > >>>>>>>>> - Locking the 16bit exception encoding for the spec
>> in
>> > >> this
>> > >> > >>>>>>>> case.
>> > >> > >>>>>>>>> - Awesome I think we have solved for all open
>> questions
>> > >> minus
>> > >> > >>>>>>>> the
>> > >> > >>>>>>>>> version byte :). (will get back on this soon)
>> > >> > >>>>>>>>>
>> > >> > >>>>>>>>>
>> > >> > >>>>>>>>> 3) Micah
>> > >> > >>>>>>>>>
>> > >> > >>>>>>>>> - FastLanes : The current spec does allow for using
>> > >> FastLane
>> > >> > >>>>>>>> with the
>> > >> > >>>>>>>>> configurable enum value for layout. We should be
>> able to
>> > >> > inject
>> > >> > >>>>>>>> any
>> > >> > >>>>>>>>> layout
>> > >> > >>>>>>>>> in the current design.
>> > >> > >>>>>>>>>
>> > >> > >>>>>>>>>
>> > >> > >>>>>>>>> Working on resolving all remaining open comments on the
>> spec
>> > >> this
>> > >> > >>>>>>>> week.
>> > >> > >>>>>>>>>
>> > >> > >>>>>>>>> Best
>> > >> > >>>>>>>>> Prateek
>> > >> > >>>>>>>>>
>> > >> > >>>>>>>>>
>> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <
>> > >> > >>>>>>>> stevel@cloudera.com>
>> > >> > >>>>>>>>> wrote:
>> > >> > >>>>>>>>>
>> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
>> > >> > >>>>>>>> emkornfield@gmail.com>
>> > >> > >>>>>>>>>> wrote:
>> > >> > >>>>>>>>>>
>> > >> > >>>>>>>>>>>
>> > >> > >>>>>>>>>>>
>> > >> > >>>>>>>>>>> It looks like the actual issue described for ORC in the
>> > >> paper
>> > >> > >>>>>>>> is that
>> > >> > >>>>>>>>> it
>> > >> > >>>>>>>>>>> has multiple sub-encodings in a batch. This is
>> different
>> > >> then
>> > >> > >>>>>>>> the
>> > >> > >>>>>>>>> design
>> > >> > >>>>>>>>>>> proposed here where there is still fixed encoding per
>> page
>> > >> in
>> > >> > >>>>>>>> parquet.
>> > >> > >>>>>>>>>>> Given reasonably sized pages I don't think branch
>> > >> > >>>>>>>> misprediction should
>> > >> > >>>>>>>>>> be a
>> > >> > >>>>>>>>>>> big issue for new encodings. I agree that we should be
>> > >> > >>>>>>>> conservative in
>> > >> > >>>>>>>>>>> general for adding new encodings.
>> > >> > >>>>>>>>>>>
>> > >> > >>>>>>>>>>>
>> > >> > >>>>>>>>>> +1
>> > >> > >>>>>>>>>>
>> > >> > >>>>>>>>>
>> > >> > >>>>>>>>
>> > >> > >>>>>>>
>> > >> > >
>> > >> >
>> > >> >
>> > >> >
>> > >>
>> > >
>> >
>>
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by PRATEEK GAUR <pr...@gmail.com>.
Thanks Matt,
I'll be on it this week.
Best
Prateek
On Mon, Jul 13, 2026 at 8:36 AM Matt Topol <zo...@gmail.com> wrote:
> There is also the Go implementation submitted by Arnav (
> https://github.com/apache/arrow-go/pull/704) which is waiting for updates.
>
> Just wanted to make sure it didn't get lost here.
>
> --Matt
>
> On Mon, Jul 13, 2026 at 11:30 AM PRATEEK GAUR <pr...@gmail.com> wrote:
>
> > Hi all,
> >
> > I'd like to share a status update on the ALP encoding effort and get a
> feel
> > from the community before starting a formal vote.
> >
> > We now have the specification plus implementations in both languages:
> >
> > - Spec: apache/parquet-format#557 — GH-533 Add ALP encoding
> > specification
> > - C++ (Arrow): apache/arrow#48345 — GH-48701 [C++][Parquet] Add
> ALPpd
> > encoding
> > - Java: apache/parquet-java#3397 — Parquet Java ALP
> > Implementation (by Vinoo)
> >
> > Current state:
> >
> > - The C++ implementation has been through 3-4 rounds of review.
> > - The Java implementation has been tracking well and has had initial
> > reviews from contributors, with no major remaining issues.
> > - Cross-language compatibility tests are passing: the Arrow C++
> decoder
> > reads Java-written data bit-exactly across ~1.56M values and 18
> > fixtures, covering V1 and V2 pages, multiple vector sizes, and
> > several
> > real datasets — zero mismatches.
> >
> > As raised in the java-pr we will be adding coverage for extreme values
> > (those requiring
> > 63-64 bits after FOR is applied) before we close a vote. We'll aim to
> get
> > that
> > done in parallel.
> >
> > Plan: unless there are objections, I intend to start a vote at the end of
> > this week/early
> > next week.
> >
> > Thanks
> > Prateek
> >
> > On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <pr...@gmail.com> wrote:
> >
> > > Hi Team,
> > >
> > > Just wanted to provide some updates on ALP.
> > > Micah and I have done 3-4 rounds of review for the c++ PR :
> > > https://github.com/apache/arrow/pull/48345
> > >
> > > For the Java implementation Vinoo has been working on following PR :
> > > https://github.com/apache/parquet-java/pull/3397
> > >
> > > Best
> > > Prateek
> > >
> > > On Tue, May 5, 2026 at 1:50 PM Micah Kornfield <em...@gmail.com>
> > > wrote:
> > >
> > >> Hi Antoine,
> > >>
> > >> > Apologies if the question was already asked, but should we care
> about
> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> > >>
> > >> It was. We thought we could defer it for the following reasons:
> > >> 1. It's not clear there are a lot of easy reference datasets to test
> > its
> > >> effectiveness. It does look like there might be one or two on
> > huggingface
> > >> (e.g. https://huggingface.co/datasets/kikitora/curdie).
> > >> 2. It seemed likely that float 16 was more likely used for values
> that
> > >> were less likely to reduce to decimal values.
> > >> 3. It could be added as an extension later if needed.
> > >>
> > >> Cheers,
> > >> Micah
> > >>
> > >>
> > >> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <an...@python.org>
> > wrote:
> > >>
> > >> >
> > >> > Hello,
> > >> >
> > >> > Apologies if the question was already asked, but should we care
> about
> > >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
> > >> > BYTE_STREAM_SPLIT + LZ4 for example?
> > >> >
> > >> > Regards
> > >> >
> > >> > Antoine.
> > >> >
> > >> >
> > >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
> > >> > > Thanks Andrew and Micah for review feedback on the two PR's
> > >> > > 1) (c++ arrow repo)
> > >> https://github.com/apache/arrow/pull/48345/changes
> > >> > > 2) (parquet-format repo)
> > >> > https://github.com/apache/parquet-format/pull/557
> > >> > >
> > >> > > I have addressed all (unless I missed something) comments on the
> two
> > >> > PR's.
> > >> > >
> > >> > > Best
> > >> > > Prateek
> > >> > >
> > >> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <pr...@gmail.com>
> > >> wrote:
> > >> > >
> > >> > >> Thanks Andrew and Micah.
> > >> > >>
> > >> > >> `fair amount of feedback on at least the implementations`
> > >> > >> For the c++ I have already started addressing the feedback, I
> > should
> > >> be
> > >> > >> done with that Monday/Tuesday.
> > >> > >> I think Vinoo too has been making good progress on the Java
> > >> > implementation.
> > >> > >>
> > >> > >> Best
> > >> > >> Prateek
> > >> > >>
> > >> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <
> > andrewlamb11@gmail.com
> > >> >
> > >> > >> wrote:
> > >> > >>
> > >> > >>> Got it. Thank you for the clarification -- I will try and look
> > into
> > >> the
> > >> > >>> spec and the Rust implementation[1] in this next week
> > >> > >>>
> > >> > >>> [1]: https://github.com/apache/arrow-rs/pull/9372
> > >> > >>>
> > >> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <
> > >> > emkornfield@gmail.com>
> > >> > >>> wrote:
> > >> > >>>
> > >> > >>>> Hi Andrew,
> > >> > >>>> I think there is a fair amount of feedback on at least the
> > >> > >>>> implementations, typically I think we've waited till they are
> > >> close to
> > >> > >>>> mergeable before a final vote. Otherwise I agree we are very
> > >> close.
> > >> > >>>>
> > >> > >>>> -Micah
> > >> > >>>>
> > >> > >>>> On Saturday, April 25, 2026, Andrew Lamb <
> andrewlamb11@gmail.com
> > >
> > >> > wrote:
> > >> > >>>>
> > >> > >>>>> Thanks Prateek,
> > >> > >>>>>
> > >> > >>>>> I think from this content it looks to me like we are ready to
> > >> start a
> > >> > >>>>> vote to explicitly accept ALP into Parquet
> > >> > >>>>>
> > >> > >>>>> Does anyone know of a reason we should postpone it for longer?
> > >> > >>>>> Perhaps someone needs some more time to review?
> > >> > >>>>>
> > >> > >>>>> Andrew
> > >> > >>>>>
> > >> > >>>>>
> > >> > >>>>>
> > >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <
> > prtkgaur@gmail.com>
> > >> > >>>>> wrote:
> > >> > >>>>>
> > >> > >>>>>> Hi team,
> > >> > >>>>>>
> > >> > >>>>>>
> > >> > >>>>>>
> > >> > >>>>>> Hope everyone is doing well. I got a chance to work through
> all
> > >> the
> > >> > >>>>>> remaining feedback and update the spec doc. Here are the new
> > >> > artifacts
> > >> > >>>>>>
> > >> > >>>>>> 1) Spec document :
> > >> > >>>>>>
> > >> >
> > >>
> >
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
> > >> > >>>>>>
> > >> > >>>>>> 2) Spec document in parquet format repo :
> > >> > >>>>>> https://github.com/apache/parquet-format/pull/557
> > >> > >>>>>>
> > >> > >>>>>> 3) Alp implementation in arrow c++ repo :
> > >> > >>>>>> https://github.com/apache/arrow/pull/48345/changes
> > >> > >>>>>>
> > >> > >>>>>> 4) Alp implementation in parquet-java repo : Work for Vinoo
> and
> > >> > Julien
> > >> > >>>>>> https://github.com/apache/parquet-java/pull/3397
> > >> > >>>>>>
> > >> > >>>>>> 5) PR with test and benchmarking artifacts in parquet-testing
> > >> repo :
> > >> > >>>>>> https://github.com/apache/parquet-testing/pull/100
> > >> > >>>>>>
> > >> > >>>>>>
> > >> > >>>>>> And
> > >> > >>>>>>
> > >> > >>>>>>
> > >> > >>>>>> - Go : Arnav just submitted an in progress implementation
> > in
> > >> Go.
> > >> > >>>>>> https://github.com/apache/arrow-go/pull/704 (I haven't
> > >> started
> > >> > >>>>>> looking at it yet)
> > >> > >>>>>> - Rust : I remember Andrew mentioned that this work is
> also
> > >> in
> > >> > >>>>>> progress (So 4 languages!)
> > >> > >>>>>>
> > >> > >>>>>>
> > >> > >>>>>> *Arrow C++ implementation *
> > >> > >>>>>>
> > >> > >>>>>>
> > >> > >>>>>>
> > >> > >>>>>> The PR is out and was also used by Antoine to report the
> > numbers
> > >> as
> > >> > >>>>>> reported here. Micah and Konstantin have given 1 round of
> > >> feedback
> > >> > >>>>>> and I'm addressing them today. Please note that the default
> > >> > >>>>>> optimization flag for compiling is O2 and not Q3. I got
> around
> > >> 70%
> > >> > >>>>>> performance improvement in the decoding speed when using the
> O3
> > >> > flag.
> > >> > >>>>>>
> > >> > >>>>>>
> > >> > >>>>>>
> > >> > >>>>>> *Parqet-MR Java implementation (working with Vinoo and
> Julien)
> > >> and
> > >> > **Cross
> > >> > >>>>>> Language testing*
> > >> > >>>>>>
> > >> > >>>>>>
> > >> > >>>>>> Let me know if you have any questions or feedback.
> > >> > >>>>>>
> > >> > >>>>>>
> > >> > >>>>>>
> > >> > >>>>>> Now pasting some performance numbers
> > >> > >>>>>>
> > >> > >>>>>>
> > >> > >>>>>> Table 1: C++ ALP Double Decode — Spotify Columns (Graviton
> > 3,
> > >> ARM
> > >> > >>>>>> Neoverse V1)
> > >> > >>>>>>
> > >> > >>>>>>
> ┌──────────────────┬──────────────┬──────────────┬─────────┐
> > >> > >>>>>>
> > >> > >>>>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │
> Speedup │
> > >> > >>>>>>
> > >> > >>>>>>
> ├──────────────────┼──────────────┼──────────────┼─────────┤
> > >> > >>>>>>
> > >> > >>>>>> │ valence │ 3,155 │ 5,523 │
> 1.75x │
> > >> > >>>>>>
> > >> > >>>>>> │ danceability │ 3,233 │ 5,685 │
> 1.76x │
> > >> > >>>>>>
> > >> > >>>>>> │ energy │ 3,197 │ 5,652 │
> 1.77x │
> > >> > >>>>>>
> > >> > >>>>>> │ loudness │ 3,186 │ 5,473 │
> 1.72x │
> > >> > >>>>>>
> > >> > >>>>>>
> └──────────────────┴──────────────┴──────────────┴─────────┘
> > >> > >>>>>>
> > >> > >>>>>>
> > >> > >>>>>>
> > >> > >>>>>>
> > >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <
> > prtkgaur@gmail.com
> > >> >
> > >> > >>>>>> wrote:
> > >> > >>>>>>
> > >> > >>>>>>> @Micah Kornfield <em...@gmail.com> : Got it.
> > >> > >>>>>>>
> > >> > >>>>>>> @Andrew Lamb <an...@gmail.com>
> > >> > >>>>>>>
> > >> > >>>>>>>
> > >> > >>>>>>>> Do you think it would be good to start moving the spec
> > >> development
> > >> > >>>>>>>> into
> > >> > >>>>>>>> markdown format, in preparation for finalizing it?
> > >> > >>>>>>>>
> > >> > >>>>>>>
> > >> > >>>>>>> Yes I'll update the numbers for some of the examples I have
> in
> > >> the
> > >> > >>>>>>> spec based
> > >> > >>>>>>> on the updated header size. Then we should be good to go for
> > the
> > >> > >>>>>>> markdown format.
> > >> > >>>>>>>
> > >> > >>>>>>> Thanks everyone!
> > >> > >>>>>>>
> > >> > >>>>>>>
> > >> > >>>>>>>>
> > >> > >>>>>>>> Andrew
> > >> > >>>>>>>>
> > >> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <
> > >> prtkgaur@gmail.com>
> > >> > >>>>>>>> wrote:
> > >> > >>>>>>>>
> > >> > >>>>>>>>> Hi team,
> > >> > >>>>>>>>>
> > >> > >>>>>>>>> 1) Andrew
> > >> > >>>>>>>>>
> > >> > >>>>>>>>> - Thanks for working on test files. My PR did add all
> > the
> > >> > test
> > >> > >>>>>>>> files I
> > >> > >>>>>>>>> used to benchmark on datasets. Maybe we can club it
> > >> together.
> > >> > >>>>>>>> WIll also
> > >> > >>>>>>>>> aid
> > >> > >>>>>>>>> cross language testing
> > >> > >>>>>>>>> - Kosta Tarasov working on Rust implementation. This
> is
> > >> > great.
> > >> > >>>>>>>> Thanks
> > >> > >>>>>>>>>
> > >> > >>>>>>>>>
> > >> > >>>>>>>>> 2) Antoine
> > >> > >>>>>>>>>
> > >> > >>>>>>>>> - Thanks a lot for reporting the numbers on AMD. Looks
> > >> like
> > >> > you
> > >> > >>>>>>>> are
> > >> > >>>>>>>>> getting 8X the decoding performance of BSS. This is
> > >> > amazing!!.
> > >> > >>>>>>>>> - Thanks for acknowledging the sampling design.
> > >> > >>>>>>>>> - I agree with you on Fastlanes. In some crude
> > >> experiments I
> > >> > >>>>>>>> didn't get
> > >> > >>>>>>>>> a good perf benefit from it on Graviton3 (but maybe
> > there
> > >> was
> > >> > >>>>>>>> something
> > >> > >>>>>>>>> wrong with my implementation).
> > >> > >>>>>>>>> - Locking the 16bit exception encoding for the spec in
> > >> this
> > >> > >>>>>>>> case.
> > >> > >>>>>>>>> - Awesome I think we have solved for all open
> questions
> > >> minus
> > >> > >>>>>>>> the
> > >> > >>>>>>>>> version byte :). (will get back on this soon)
> > >> > >>>>>>>>>
> > >> > >>>>>>>>>
> > >> > >>>>>>>>> 3) Micah
> > >> > >>>>>>>>>
> > >> > >>>>>>>>> - FastLanes : The current spec does allow for using
> > >> FastLane
> > >> > >>>>>>>> with the
> > >> > >>>>>>>>> configurable enum value for layout. We should be able
> to
> > >> > inject
> > >> > >>>>>>>> any
> > >> > >>>>>>>>> layout
> > >> > >>>>>>>>> in the current design.
> > >> > >>>>>>>>>
> > >> > >>>>>>>>>
> > >> > >>>>>>>>> Working on resolving all remaining open comments on the
> spec
> > >> this
> > >> > >>>>>>>> week.
> > >> > >>>>>>>>>
> > >> > >>>>>>>>> Best
> > >> > >>>>>>>>> Prateek
> > >> > >>>>>>>>>
> > >> > >>>>>>>>>
> > >> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <
> > >> > >>>>>>>> stevel@cloudera.com>
> > >> > >>>>>>>>> wrote:
> > >> > >>>>>>>>>
> > >> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
> > >> > >>>>>>>> emkornfield@gmail.com>
> > >> > >>>>>>>>>> wrote:
> > >> > >>>>>>>>>>
> > >> > >>>>>>>>>>>
> > >> > >>>>>>>>>>>
> > >> > >>>>>>>>>>> It looks like the actual issue described for ORC in the
> > >> paper
> > >> > >>>>>>>> is that
> > >> > >>>>>>>>> it
> > >> > >>>>>>>>>>> has multiple sub-encodings in a batch. This is
> different
> > >> then
> > >> > >>>>>>>> the
> > >> > >>>>>>>>> design
> > >> > >>>>>>>>>>> proposed here where there is still fixed encoding per
> page
> > >> in
> > >> > >>>>>>>> parquet.
> > >> > >>>>>>>>>>> Given reasonably sized pages I don't think branch
> > >> > >>>>>>>> misprediction should
> > >> > >>>>>>>>>> be a
> > >> > >>>>>>>>>>> big issue for new encodings. I agree that we should be
> > >> > >>>>>>>> conservative in
> > >> > >>>>>>>>>>> general for adding new encodings.
> > >> > >>>>>>>>>>>
> > >> > >>>>>>>>>>>
> > >> > >>>>>>>>>> +1
> > >> > >>>>>>>>>>
> > >> > >>>>>>>>>
> > >> > >>>>>>>>
> > >> > >>>>>>>
> > >> > >
> > >> >
> > >> >
> > >> >
> > >>
> > >
> >
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Matt Topol <zo...@gmail.com>.
There is also the Go implementation submitted by Arnav (
https://github.com/apache/arrow-go/pull/704) which is waiting for updates.
Just wanted to make sure it didn't get lost here.
--Matt
On Mon, Jul 13, 2026 at 11:30 AM PRATEEK GAUR <pr...@gmail.com> wrote:
> Hi all,
>
> I'd like to share a status update on the ALP encoding effort and get a feel
> from the community before starting a formal vote.
>
> We now have the specification plus implementations in both languages:
>
> - Spec: apache/parquet-format#557 — GH-533 Add ALP encoding
> specification
> - C++ (Arrow): apache/arrow#48345 — GH-48701 [C++][Parquet] Add ALPpd
> encoding
> - Java: apache/parquet-java#3397 — Parquet Java ALP
> Implementation (by Vinoo)
>
> Current state:
>
> - The C++ implementation has been through 3-4 rounds of review.
> - The Java implementation has been tracking well and has had initial
> reviews from contributors, with no major remaining issues.
> - Cross-language compatibility tests are passing: the Arrow C++ decoder
> reads Java-written data bit-exactly across ~1.56M values and 18
> fixtures, covering V1 and V2 pages, multiple vector sizes, and
> several
> real datasets — zero mismatches.
>
> As raised in the java-pr we will be adding coverage for extreme values
> (those requiring
> 63-64 bits after FOR is applied) before we close a vote. We'll aim to get
> that
> done in parallel.
>
> Plan: unless there are objections, I intend to start a vote at the end of
> this week/early
> next week.
>
> Thanks
> Prateek
>
> On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <pr...@gmail.com> wrote:
>
> > Hi Team,
> >
> > Just wanted to provide some updates on ALP.
> > Micah and I have done 3-4 rounds of review for the c++ PR :
> > https://github.com/apache/arrow/pull/48345
> >
> > For the Java implementation Vinoo has been working on following PR :
> > https://github.com/apache/parquet-java/pull/3397
> >
> > Best
> > Prateek
> >
> > On Tue, May 5, 2026 at 1:50 PM Micah Kornfield <em...@gmail.com>
> > wrote:
> >
> >> Hi Antoine,
> >>
> >> > Apologies if the question was already asked, but should we care about
> >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
> >> > BYTE_STREAM_SPLIT + LZ4 for example?
> >>
> >> It was. We thought we could defer it for the following reasons:
> >> 1. It's not clear there are a lot of easy reference datasets to test
> its
> >> effectiveness. It does look like there might be one or two on
> huggingface
> >> (e.g. https://huggingface.co/datasets/kikitora/curdie).
> >> 2. It seemed likely that float 16 was more likely used for values that
> >> were less likely to reduce to decimal values.
> >> 3. It could be added as an extension later if needed.
> >>
> >> Cheers,
> >> Micah
> >>
> >>
> >> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <an...@python.org>
> wrote:
> >>
> >> >
> >> > Hello,
> >> >
> >> > Apologies if the question was already asked, but should we care about
> >> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
> >> > BYTE_STREAM_SPLIT + LZ4 for example?
> >> >
> >> > Regards
> >> >
> >> > Antoine.
> >> >
> >> >
> >> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
> >> > > Thanks Andrew and Micah for review feedback on the two PR's
> >> > > 1) (c++ arrow repo)
> >> https://github.com/apache/arrow/pull/48345/changes
> >> > > 2) (parquet-format repo)
> >> > https://github.com/apache/parquet-format/pull/557
> >> > >
> >> > > I have addressed all (unless I missed something) comments on the two
> >> > PR's.
> >> > >
> >> > > Best
> >> > > Prateek
> >> > >
> >> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <pr...@gmail.com>
> >> wrote:
> >> > >
> >> > >> Thanks Andrew and Micah.
> >> > >>
> >> > >> `fair amount of feedback on at least the implementations`
> >> > >> For the c++ I have already started addressing the feedback, I
> should
> >> be
> >> > >> done with that Monday/Tuesday.
> >> > >> I think Vinoo too has been making good progress on the Java
> >> > implementation.
> >> > >>
> >> > >> Best
> >> > >> Prateek
> >> > >>
> >> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <
> andrewlamb11@gmail.com
> >> >
> >> > >> wrote:
> >> > >>
> >> > >>> Got it. Thank you for the clarification -- I will try and look
> into
> >> the
> >> > >>> spec and the Rust implementation[1] in this next week
> >> > >>>
> >> > >>> [1]: https://github.com/apache/arrow-rs/pull/9372
> >> > >>>
> >> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <
> >> > emkornfield@gmail.com>
> >> > >>> wrote:
> >> > >>>
> >> > >>>> Hi Andrew,
> >> > >>>> I think there is a fair amount of feedback on at least the
> >> > >>>> implementations, typically I think we've waited till they are
> >> close to
> >> > >>>> mergeable before a final vote. Otherwise I agree we are very
> >> close.
> >> > >>>>
> >> > >>>> -Micah
> >> > >>>>
> >> > >>>> On Saturday, April 25, 2026, Andrew Lamb <andrewlamb11@gmail.com
> >
> >> > wrote:
> >> > >>>>
> >> > >>>>> Thanks Prateek,
> >> > >>>>>
> >> > >>>>> I think from this content it looks to me like we are ready to
> >> start a
> >> > >>>>> vote to explicitly accept ALP into Parquet
> >> > >>>>>
> >> > >>>>> Does anyone know of a reason we should postpone it for longer?
> >> > >>>>> Perhaps someone needs some more time to review?
> >> > >>>>>
> >> > >>>>> Andrew
> >> > >>>>>
> >> > >>>>>
> >> > >>>>>
> >> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <
> prtkgaur@gmail.com>
> >> > >>>>> wrote:
> >> > >>>>>
> >> > >>>>>> Hi team,
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>> Hope everyone is doing well. I got a chance to work through all
> >> the
> >> > >>>>>> remaining feedback and update the spec doc. Here are the new
> >> > artifacts
> >> > >>>>>>
> >> > >>>>>> 1) Spec document :
> >> > >>>>>>
> >> >
> >>
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
> >> > >>>>>>
> >> > >>>>>> 2) Spec document in parquet format repo :
> >> > >>>>>> https://github.com/apache/parquet-format/pull/557
> >> > >>>>>>
> >> > >>>>>> 3) Alp implementation in arrow c++ repo :
> >> > >>>>>> https://github.com/apache/arrow/pull/48345/changes
> >> > >>>>>>
> >> > >>>>>> 4) Alp implementation in parquet-java repo : Work for Vinoo and
> >> > Julien
> >> > >>>>>> https://github.com/apache/parquet-java/pull/3397
> >> > >>>>>>
> >> > >>>>>> 5) PR with test and benchmarking artifacts in parquet-testing
> >> repo :
> >> > >>>>>> https://github.com/apache/parquet-testing/pull/100
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>> And
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>> - Go : Arnav just submitted an in progress implementation
> in
> >> Go.
> >> > >>>>>> https://github.com/apache/arrow-go/pull/704 (I haven't
> >> started
> >> > >>>>>> looking at it yet)
> >> > >>>>>> - Rust : I remember Andrew mentioned that this work is also
> >> in
> >> > >>>>>> progress (So 4 languages!)
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>> *Arrow C++ implementation *
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>> The PR is out and was also used by Antoine to report the
> numbers
> >> as
> >> > >>>>>> reported here. Micah and Konstantin have given 1 round of
> >> feedback
> >> > >>>>>> and I'm addressing them today. Please note that the default
> >> > >>>>>> optimization flag for compiling is O2 and not Q3. I got around
> >> 70%
> >> > >>>>>> performance improvement in the decoding speed when using the O3
> >> > flag.
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>> *Parqet-MR Java implementation (working with Vinoo and Julien)
> >> and
> >> > **Cross
> >> > >>>>>> Language testing*
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>> Let me know if you have any questions or feedback.
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>> Now pasting some performance numbers
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>> Table 1: C++ ALP Double Decode — Spotify Columns (Graviton
> 3,
> >> ARM
> >> > >>>>>> Neoverse V1)
> >> > >>>>>>
> >> > >>>>>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
> >> > >>>>>>
> >> > >>>>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │ Speedup │
> >> > >>>>>>
> >> > >>>>>> ├──────────────────┼──────────────┼──────────────┼─────────┤
> >> > >>>>>>
> >> > >>>>>> │ valence │ 3,155 │ 5,523 │ 1.75x │
> >> > >>>>>>
> >> > >>>>>> │ danceability │ 3,233 │ 5,685 │ 1.76x │
> >> > >>>>>>
> >> > >>>>>> │ energy │ 3,197 │ 5,652 │ 1.77x │
> >> > >>>>>>
> >> > >>>>>> │ loudness │ 3,186 │ 5,473 │ 1.72x │
> >> > >>>>>>
> >> > >>>>>> └──────────────────┴──────────────┴──────────────┴─────────┘
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <
> prtkgaur@gmail.com
> >> >
> >> > >>>>>> wrote:
> >> > >>>>>>
> >> > >>>>>>> @Micah Kornfield <em...@gmail.com> : Got it.
> >> > >>>>>>>
> >> > >>>>>>> @Andrew Lamb <an...@gmail.com>
> >> > >>>>>>>
> >> > >>>>>>>
> >> > >>>>>>>> Do you think it would be good to start moving the spec
> >> development
> >> > >>>>>>>> into
> >> > >>>>>>>> markdown format, in preparation for finalizing it?
> >> > >>>>>>>>
> >> > >>>>>>>
> >> > >>>>>>> Yes I'll update the numbers for some of the examples I have in
> >> the
> >> > >>>>>>> spec based
> >> > >>>>>>> on the updated header size. Then we should be good to go for
> the
> >> > >>>>>>> markdown format.
> >> > >>>>>>>
> >> > >>>>>>> Thanks everyone!
> >> > >>>>>>>
> >> > >>>>>>>
> >> > >>>>>>>>
> >> > >>>>>>>> Andrew
> >> > >>>>>>>>
> >> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <
> >> prtkgaur@gmail.com>
> >> > >>>>>>>> wrote:
> >> > >>>>>>>>
> >> > >>>>>>>>> Hi team,
> >> > >>>>>>>>>
> >> > >>>>>>>>> 1) Andrew
> >> > >>>>>>>>>
> >> > >>>>>>>>> - Thanks for working on test files. My PR did add all
> the
> >> > test
> >> > >>>>>>>> files I
> >> > >>>>>>>>> used to benchmark on datasets. Maybe we can club it
> >> together.
> >> > >>>>>>>> WIll also
> >> > >>>>>>>>> aid
> >> > >>>>>>>>> cross language testing
> >> > >>>>>>>>> - Kosta Tarasov working on Rust implementation. This is
> >> > great.
> >> > >>>>>>>> Thanks
> >> > >>>>>>>>>
> >> > >>>>>>>>>
> >> > >>>>>>>>> 2) Antoine
> >> > >>>>>>>>>
> >> > >>>>>>>>> - Thanks a lot for reporting the numbers on AMD. Looks
> >> like
> >> > you
> >> > >>>>>>>> are
> >> > >>>>>>>>> getting 8X the decoding performance of BSS. This is
> >> > amazing!!.
> >> > >>>>>>>>> - Thanks for acknowledging the sampling design.
> >> > >>>>>>>>> - I agree with you on Fastlanes. In some crude
> >> experiments I
> >> > >>>>>>>> didn't get
> >> > >>>>>>>>> a good perf benefit from it on Graviton3 (but maybe
> there
> >> was
> >> > >>>>>>>> something
> >> > >>>>>>>>> wrong with my implementation).
> >> > >>>>>>>>> - Locking the 16bit exception encoding for the spec in
> >> this
> >> > >>>>>>>> case.
> >> > >>>>>>>>> - Awesome I think we have solved for all open questions
> >> minus
> >> > >>>>>>>> the
> >> > >>>>>>>>> version byte :). (will get back on this soon)
> >> > >>>>>>>>>
> >> > >>>>>>>>>
> >> > >>>>>>>>> 3) Micah
> >> > >>>>>>>>>
> >> > >>>>>>>>> - FastLanes : The current spec does allow for using
> >> FastLane
> >> > >>>>>>>> with the
> >> > >>>>>>>>> configurable enum value for layout. We should be able to
> >> > inject
> >> > >>>>>>>> any
> >> > >>>>>>>>> layout
> >> > >>>>>>>>> in the current design.
> >> > >>>>>>>>>
> >> > >>>>>>>>>
> >> > >>>>>>>>> Working on resolving all remaining open comments on the spec
> >> this
> >> > >>>>>>>> week.
> >> > >>>>>>>>>
> >> > >>>>>>>>> Best
> >> > >>>>>>>>> Prateek
> >> > >>>>>>>>>
> >> > >>>>>>>>>
> >> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <
> >> > >>>>>>>> stevel@cloudera.com>
> >> > >>>>>>>>> wrote:
> >> > >>>>>>>>>
> >> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
> >> > >>>>>>>> emkornfield@gmail.com>
> >> > >>>>>>>>>> wrote:
> >> > >>>>>>>>>>
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>> It looks like the actual issue described for ORC in the
> >> paper
> >> > >>>>>>>> is that
> >> > >>>>>>>>> it
> >> > >>>>>>>>>>> has multiple sub-encodings in a batch. This is different
> >> then
> >> > >>>>>>>> the
> >> > >>>>>>>>> design
> >> > >>>>>>>>>>> proposed here where there is still fixed encoding per page
> >> in
> >> > >>>>>>>> parquet.
> >> > >>>>>>>>>>> Given reasonably sized pages I don't think branch
> >> > >>>>>>>> misprediction should
> >> > >>>>>>>>>> be a
> >> > >>>>>>>>>>> big issue for new encodings. I agree that we should be
> >> > >>>>>>>> conservative in
> >> > >>>>>>>>>>> general for adding new encodings.
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>>>
> >> > >>>>>>>>>> +1
> >> > >>>>>>>>>>
> >> > >>>>>>>>>
> >> > >>>>>>>>
> >> > >>>>>>>
> >> > >
> >> >
> >> >
> >> >
> >>
> >
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by PRATEEK GAUR <pr...@gmail.com>.
Hi all,
I'd like to share a status update on the ALP encoding effort and get a feel
from the community before starting a formal vote.
We now have the specification plus implementations in both languages:
- Spec: apache/parquet-format#557 — GH-533 Add ALP encoding
specification
- C++ (Arrow): apache/arrow#48345 — GH-48701 [C++][Parquet] Add ALPpd
encoding
- Java: apache/parquet-java#3397 — Parquet Java ALP
Implementation (by Vinoo)
Current state:
- The C++ implementation has been through 3-4 rounds of review.
- The Java implementation has been tracking well and has had initial
reviews from contributors, with no major remaining issues.
- Cross-language compatibility tests are passing: the Arrow C++ decoder
reads Java-written data bit-exactly across ~1.56M values and 18
fixtures, covering V1 and V2 pages, multiple vector sizes, and several
real datasets — zero mismatches.
As raised in the java-pr we will be adding coverage for extreme values
(those requiring
63-64 bits after FOR is applied) before we close a vote. We'll aim to get
that
done in parallel.
Plan: unless there are objections, I intend to start a vote at the end of
this week/early
next week.
Thanks
Prateek
On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <pr...@gmail.com> wrote:
> Hi Team,
>
> Just wanted to provide some updates on ALP.
> Micah and I have done 3-4 rounds of review for the c++ PR :
> https://github.com/apache/arrow/pull/48345
>
> For the Java implementation Vinoo has been working on following PR :
> https://github.com/apache/parquet-java/pull/3397
>
> Best
> Prateek
>
> On Tue, May 5, 2026 at 1:50 PM Micah Kornfield <em...@gmail.com>
> wrote:
>
>> Hi Antoine,
>>
>> > Apologies if the question was already asked, but should we care about
>> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
>> > BYTE_STREAM_SPLIT + LZ4 for example?
>>
>> It was. We thought we could defer it for the following reasons:
>> 1. It's not clear there are a lot of easy reference datasets to test its
>> effectiveness. It does look like there might be one or two on huggingface
>> (e.g. https://huggingface.co/datasets/kikitora/curdie).
>> 2. It seemed likely that float 16 was more likely used for values that
>> were less likely to reduce to decimal values.
>> 3. It could be added as an extension later if needed.
>>
>> Cheers,
>> Micah
>>
>>
>> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <an...@python.org> wrote:
>>
>> >
>> > Hello,
>> >
>> > Apologies if the question was already asked, but should we care about
>> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
>> > BYTE_STREAM_SPLIT + LZ4 for example?
>> >
>> > Regards
>> >
>> > Antoine.
>> >
>> >
>> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
>> > > Thanks Andrew and Micah for review feedback on the two PR's
>> > > 1) (c++ arrow repo)
>> https://github.com/apache/arrow/pull/48345/changes
>> > > 2) (parquet-format repo)
>> > https://github.com/apache/parquet-format/pull/557
>> > >
>> > > I have addressed all (unless I missed something) comments on the two
>> > PR's.
>> > >
>> > > Best
>> > > Prateek
>> > >
>> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <pr...@gmail.com>
>> wrote:
>> > >
>> > >> Thanks Andrew and Micah.
>> > >>
>> > >> `fair amount of feedback on at least the implementations`
>> > >> For the c++ I have already started addressing the feedback, I should
>> be
>> > >> done with that Monday/Tuesday.
>> > >> I think Vinoo too has been making good progress on the Java
>> > implementation.
>> > >>
>> > >> Best
>> > >> Prateek
>> > >>
>> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <andrewlamb11@gmail.com
>> >
>> > >> wrote:
>> > >>
>> > >>> Got it. Thank you for the clarification -- I will try and look into
>> the
>> > >>> spec and the Rust implementation[1] in this next week
>> > >>>
>> > >>> [1]: https://github.com/apache/arrow-rs/pull/9372
>> > >>>
>> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <
>> > emkornfield@gmail.com>
>> > >>> wrote:
>> > >>>
>> > >>>> Hi Andrew,
>> > >>>> I think there is a fair amount of feedback on at least the
>> > >>>> implementations, typically I think we've waited till they are
>> close to
>> > >>>> mergeable before a final vote. Otherwise I agree we are very
>> close.
>> > >>>>
>> > >>>> -Micah
>> > >>>>
>> > >>>> On Saturday, April 25, 2026, Andrew Lamb <an...@gmail.com>
>> > wrote:
>> > >>>>
>> > >>>>> Thanks Prateek,
>> > >>>>>
>> > >>>>> I think from this content it looks to me like we are ready to
>> start a
>> > >>>>> vote to explicitly accept ALP into Parquet
>> > >>>>>
>> > >>>>> Does anyone know of a reason we should postpone it for longer?
>> > >>>>> Perhaps someone needs some more time to review?
>> > >>>>>
>> > >>>>> Andrew
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <pr...@gmail.com>
>> > >>>>> wrote:
>> > >>>>>
>> > >>>>>> Hi team,
>> > >>>>>>
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> Hope everyone is doing well. I got a chance to work through all
>> the
>> > >>>>>> remaining feedback and update the spec doc. Here are the new
>> > artifacts
>> > >>>>>>
>> > >>>>>> 1) Spec document :
>> > >>>>>>
>> >
>> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
>> > >>>>>>
>> > >>>>>> 2) Spec document in parquet format repo :
>> > >>>>>> https://github.com/apache/parquet-format/pull/557
>> > >>>>>>
>> > >>>>>> 3) Alp implementation in arrow c++ repo :
>> > >>>>>> https://github.com/apache/arrow/pull/48345/changes
>> > >>>>>>
>> > >>>>>> 4) Alp implementation in parquet-java repo : Work for Vinoo and
>> > Julien
>> > >>>>>> https://github.com/apache/parquet-java/pull/3397
>> > >>>>>>
>> > >>>>>> 5) PR with test and benchmarking artifacts in parquet-testing
>> repo :
>> > >>>>>> https://github.com/apache/parquet-testing/pull/100
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> And
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> - Go : Arnav just submitted an in progress implementation in
>> Go.
>> > >>>>>> https://github.com/apache/arrow-go/pull/704 (I haven't
>> started
>> > >>>>>> looking at it yet)
>> > >>>>>> - Rust : I remember Andrew mentioned that this work is also
>> in
>> > >>>>>> progress (So 4 languages!)
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> *Arrow C++ implementation *
>> > >>>>>>
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> The PR is out and was also used by Antoine to report the numbers
>> as
>> > >>>>>> reported here. Micah and Konstantin have given 1 round of
>> feedback
>> > >>>>>> and I'm addressing them today. Please note that the default
>> > >>>>>> optimization flag for compiling is O2 and not Q3. I got around
>> 70%
>> > >>>>>> performance improvement in the decoding speed when using the O3
>> > flag.
>> > >>>>>>
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> *Parqet-MR Java implementation (working with Vinoo and Julien)
>> and
>> > **Cross
>> > >>>>>> Language testing*
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> Let me know if you have any questions or feedback.
>> > >>>>>>
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> Now pasting some performance numbers
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> Table 1: C++ ALP Double Decode — Spotify Columns (Graviton 3,
>> ARM
>> > >>>>>> Neoverse V1)
>> > >>>>>>
>> > >>>>>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
>> > >>>>>>
>> > >>>>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │ Speedup │
>> > >>>>>>
>> > >>>>>> ├──────────────────┼──────────────┼──────────────┼─────────┤
>> > >>>>>>
>> > >>>>>> │ valence │ 3,155 │ 5,523 │ 1.75x │
>> > >>>>>>
>> > >>>>>> │ danceability │ 3,233 │ 5,685 │ 1.76x │
>> > >>>>>>
>> > >>>>>> │ energy │ 3,197 │ 5,652 │ 1.77x │
>> > >>>>>>
>> > >>>>>> │ loudness │ 3,186 │ 5,473 │ 1.72x │
>> > >>>>>>
>> > >>>>>> └──────────────────┴──────────────┴──────────────┴─────────┘
>> > >>>>>>
>> > >>>>>>
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <prtkgaur@gmail.com
>> >
>> > >>>>>> wrote:
>> > >>>>>>
>> > >>>>>>> @Micah Kornfield <em...@gmail.com> : Got it.
>> > >>>>>>>
>> > >>>>>>> @Andrew Lamb <an...@gmail.com>
>> > >>>>>>>
>> > >>>>>>>
>> > >>>>>>>> Do you think it would be good to start moving the spec
>> development
>> > >>>>>>>> into
>> > >>>>>>>> markdown format, in preparation for finalizing it?
>> > >>>>>>>>
>> > >>>>>>>
>> > >>>>>>> Yes I'll update the numbers for some of the examples I have in
>> the
>> > >>>>>>> spec based
>> > >>>>>>> on the updated header size. Then we should be good to go for the
>> > >>>>>>> markdown format.
>> > >>>>>>>
>> > >>>>>>> Thanks everyone!
>> > >>>>>>>
>> > >>>>>>>
>> > >>>>>>>>
>> > >>>>>>>> Andrew
>> > >>>>>>>>
>> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <
>> prtkgaur@gmail.com>
>> > >>>>>>>> wrote:
>> > >>>>>>>>
>> > >>>>>>>>> Hi team,
>> > >>>>>>>>>
>> > >>>>>>>>> 1) Andrew
>> > >>>>>>>>>
>> > >>>>>>>>> - Thanks for working on test files. My PR did add all the
>> > test
>> > >>>>>>>> files I
>> > >>>>>>>>> used to benchmark on datasets. Maybe we can club it
>> together.
>> > >>>>>>>> WIll also
>> > >>>>>>>>> aid
>> > >>>>>>>>> cross language testing
>> > >>>>>>>>> - Kosta Tarasov working on Rust implementation. This is
>> > great.
>> > >>>>>>>> Thanks
>> > >>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>> 2) Antoine
>> > >>>>>>>>>
>> > >>>>>>>>> - Thanks a lot for reporting the numbers on AMD. Looks
>> like
>> > you
>> > >>>>>>>> are
>> > >>>>>>>>> getting 8X the decoding performance of BSS. This is
>> > amazing!!.
>> > >>>>>>>>> - Thanks for acknowledging the sampling design.
>> > >>>>>>>>> - I agree with you on Fastlanes. In some crude
>> experiments I
>> > >>>>>>>> didn't get
>> > >>>>>>>>> a good perf benefit from it on Graviton3 (but maybe there
>> was
>> > >>>>>>>> something
>> > >>>>>>>>> wrong with my implementation).
>> > >>>>>>>>> - Locking the 16bit exception encoding for the spec in
>> this
>> > >>>>>>>> case.
>> > >>>>>>>>> - Awesome I think we have solved for all open questions
>> minus
>> > >>>>>>>> the
>> > >>>>>>>>> version byte :). (will get back on this soon)
>> > >>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>> 3) Micah
>> > >>>>>>>>>
>> > >>>>>>>>> - FastLanes : The current spec does allow for using
>> FastLane
>> > >>>>>>>> with the
>> > >>>>>>>>> configurable enum value for layout. We should be able to
>> > inject
>> > >>>>>>>> any
>> > >>>>>>>>> layout
>> > >>>>>>>>> in the current design.
>> > >>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>> Working on resolving all remaining open comments on the spec
>> this
>> > >>>>>>>> week.
>> > >>>>>>>>>
>> > >>>>>>>>> Best
>> > >>>>>>>>> Prateek
>> > >>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <
>> > >>>>>>>> stevel@cloudera.com>
>> > >>>>>>>>> wrote:
>> > >>>>>>>>>
>> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
>> > >>>>>>>> emkornfield@gmail.com>
>> > >>>>>>>>>> wrote:
>> > >>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> It looks like the actual issue described for ORC in the
>> paper
>> > >>>>>>>> is that
>> > >>>>>>>>> it
>> > >>>>>>>>>>> has multiple sub-encodings in a batch. This is different
>> then
>> > >>>>>>>> the
>> > >>>>>>>>> design
>> > >>>>>>>>>>> proposed here where there is still fixed encoding per page
>> in
>> > >>>>>>>> parquet.
>> > >>>>>>>>>>> Given reasonably sized pages I don't think branch
>> > >>>>>>>> misprediction should
>> > >>>>>>>>>> be a
>> > >>>>>>>>>>> big issue for new encodings. I agree that we should be
>> > >>>>>>>> conservative in
>> > >>>>>>>>>>> general for adding new encodings.
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>> +1
>> > >>>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>
>> > >
>> >
>> >
>> >
>>
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by PRATEEK GAUR <pr...@gmail.com>.
Hi all,
I'd like to share a status update on the ALP encoding effort and get a feel
from the community before starting a formal vote.
We now have the specification plus implementations in both languages:
- Spec: apache/parquet-format#557 — GH-533 Add ALP encoding
specification
- C++ (Arrow): apache/arrow#48345 — GH-48701 [C++][Parquet] Add ALPpd
encoding
- Java: apache/parquet-java#3397 — Parquet Java ALP
Implementation (by Vinoo)
Current state:
- The C++ implementation has been through 3-4 rounds of review.
- The Java implementation has been tracking well and has had initial
reviews from contributors, with no major remaining issues.
- Cross-language compatibility tests are passing: the Arrow C++ decoder
reads Java-written data bit-exactly across ~1.56M values and 18
fixtures, covering V1 and V2 pages, multiple vector sizes, and several
real datasets — zero mismatches.
As raised in the java-pr we will be adding coverage for extreme values
(those requiring
63-64 bits after FOR is applied) before we close a vote. We'll aim to get
that
done in parallel.
Plan: unless there are objections, I intend to start a vote at the end of
this week/early
next week.
Thanks
Prateek
On Wed, Jul 1, 2026 at 8:50 AM PRATEEK GAUR <pr...@gmail.com> wrote:
> Hi Team,
>
> Just wanted to provide some updates on ALP.
> Micah and I have done 3-4 rounds of review for the c++ PR :
> https://github.com/apache/arrow/pull/48345
>
> For the Java implementation Vinoo has been working on following PR :
> https://github.com/apache/parquet-java/pull/3397
>
> Best
> Prateek
>
> On Tue, May 5, 2026 at 1:50 PM Micah Kornfield <em...@gmail.com>
> wrote:
>
>> Hi Antoine,
>>
>> > Apologies if the question was already asked, but should we care about
>> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
>> > BYTE_STREAM_SPLIT + LZ4 for example?
>>
>> It was. We thought we could defer it for the following reasons:
>> 1. It's not clear there are a lot of easy reference datasets to test its
>> effectiveness. It does look like there might be one or two on huggingface
>> (e.g. https://huggingface.co/datasets/kikitora/curdie).
>> 2. It seemed likely that float 16 was more likely used for values that
>> were less likely to reduce to decimal values.
>> 3. It could be added as an extension later if needed.
>>
>> Cheers,
>> Micah
>>
>>
>> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <an...@python.org> wrote:
>>
>> >
>> > Hello,
>> >
>> > Apologies if the question was already asked, but should we care about
>> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
>> > BYTE_STREAM_SPLIT + LZ4 for example?
>> >
>> > Regards
>> >
>> > Antoine.
>> >
>> >
>> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
>> > > Thanks Andrew and Micah for review feedback on the two PR's
>> > > 1) (c++ arrow repo)
>> https://github.com/apache/arrow/pull/48345/changes
>> > > 2) (parquet-format repo)
>> > https://github.com/apache/parquet-format/pull/557
>> > >
>> > > I have addressed all (unless I missed something) comments on the two
>> > PR's.
>> > >
>> > > Best
>> > > Prateek
>> > >
>> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <pr...@gmail.com>
>> wrote:
>> > >
>> > >> Thanks Andrew and Micah.
>> > >>
>> > >> `fair amount of feedback on at least the implementations`
>> > >> For the c++ I have already started addressing the feedback, I should
>> be
>> > >> done with that Monday/Tuesday.
>> > >> I think Vinoo too has been making good progress on the Java
>> > implementation.
>> > >>
>> > >> Best
>> > >> Prateek
>> > >>
>> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <andrewlamb11@gmail.com
>> >
>> > >> wrote:
>> > >>
>> > >>> Got it. Thank you for the clarification -- I will try and look into
>> the
>> > >>> spec and the Rust implementation[1] in this next week
>> > >>>
>> > >>> [1]: https://github.com/apache/arrow-rs/pull/9372
>> > >>>
>> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <
>> > emkornfield@gmail.com>
>> > >>> wrote:
>> > >>>
>> > >>>> Hi Andrew,
>> > >>>> I think there is a fair amount of feedback on at least the
>> > >>>> implementations, typically I think we've waited till they are
>> close to
>> > >>>> mergeable before a final vote. Otherwise I agree we are very
>> close.
>> > >>>>
>> > >>>> -Micah
>> > >>>>
>> > >>>> On Saturday, April 25, 2026, Andrew Lamb <an...@gmail.com>
>> > wrote:
>> > >>>>
>> > >>>>> Thanks Prateek,
>> > >>>>>
>> > >>>>> I think from this content it looks to me like we are ready to
>> start a
>> > >>>>> vote to explicitly accept ALP into Parquet
>> > >>>>>
>> > >>>>> Does anyone know of a reason we should postpone it for longer?
>> > >>>>> Perhaps someone needs some more time to review?
>> > >>>>>
>> > >>>>> Andrew
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <pr...@gmail.com>
>> > >>>>> wrote:
>> > >>>>>
>> > >>>>>> Hi team,
>> > >>>>>>
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> Hope everyone is doing well. I got a chance to work through all
>> the
>> > >>>>>> remaining feedback and update the spec doc. Here are the new
>> > artifacts
>> > >>>>>>
>> > >>>>>> 1) Spec document :
>> > >>>>>>
>> >
>> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
>> > >>>>>>
>> > >>>>>> 2) Spec document in parquet format repo :
>> > >>>>>> https://github.com/apache/parquet-format/pull/557
>> > >>>>>>
>> > >>>>>> 3) Alp implementation in arrow c++ repo :
>> > >>>>>> https://github.com/apache/arrow/pull/48345/changes
>> > >>>>>>
>> > >>>>>> 4) Alp implementation in parquet-java repo : Work for Vinoo and
>> > Julien
>> > >>>>>> https://github.com/apache/parquet-java/pull/3397
>> > >>>>>>
>> > >>>>>> 5) PR with test and benchmarking artifacts in parquet-testing
>> repo :
>> > >>>>>> https://github.com/apache/parquet-testing/pull/100
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> And
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> - Go : Arnav just submitted an in progress implementation in
>> Go.
>> > >>>>>> https://github.com/apache/arrow-go/pull/704 (I haven't
>> started
>> > >>>>>> looking at it yet)
>> > >>>>>> - Rust : I remember Andrew mentioned that this work is also
>> in
>> > >>>>>> progress (So 4 languages!)
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> *Arrow C++ implementation *
>> > >>>>>>
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> The PR is out and was also used by Antoine to report the numbers
>> as
>> > >>>>>> reported here. Micah and Konstantin have given 1 round of
>> feedback
>> > >>>>>> and I'm addressing them today. Please note that the default
>> > >>>>>> optimization flag for compiling is O2 and not Q3. I got around
>> 70%
>> > >>>>>> performance improvement in the decoding speed when using the O3
>> > flag.
>> > >>>>>>
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> *Parqet-MR Java implementation (working with Vinoo and Julien)
>> and
>> > **Cross
>> > >>>>>> Language testing*
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> Let me know if you have any questions or feedback.
>> > >>>>>>
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> Now pasting some performance numbers
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> Table 1: C++ ALP Double Decode — Spotify Columns (Graviton 3,
>> ARM
>> > >>>>>> Neoverse V1)
>> > >>>>>>
>> > >>>>>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
>> > >>>>>>
>> > >>>>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │ Speedup │
>> > >>>>>>
>> > >>>>>> ├──────────────────┼──────────────┼──────────────┼─────────┤
>> > >>>>>>
>> > >>>>>> │ valence │ 3,155 │ 5,523 │ 1.75x │
>> > >>>>>>
>> > >>>>>> │ danceability │ 3,233 │ 5,685 │ 1.76x │
>> > >>>>>>
>> > >>>>>> │ energy │ 3,197 │ 5,652 │ 1.77x │
>> > >>>>>>
>> > >>>>>> │ loudness │ 3,186 │ 5,473 │ 1.72x │
>> > >>>>>>
>> > >>>>>> └──────────────────┴──────────────┴──────────────┴─────────┘
>> > >>>>>>
>> > >>>>>>
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <prtkgaur@gmail.com
>> >
>> > >>>>>> wrote:
>> > >>>>>>
>> > >>>>>>> @Micah Kornfield <em...@gmail.com> : Got it.
>> > >>>>>>>
>> > >>>>>>> @Andrew Lamb <an...@gmail.com>
>> > >>>>>>>
>> > >>>>>>>
>> > >>>>>>>> Do you think it would be good to start moving the spec
>> development
>> > >>>>>>>> into
>> > >>>>>>>> markdown format, in preparation for finalizing it?
>> > >>>>>>>>
>> > >>>>>>>
>> > >>>>>>> Yes I'll update the numbers for some of the examples I have in
>> the
>> > >>>>>>> spec based
>> > >>>>>>> on the updated header size. Then we should be good to go for the
>> > >>>>>>> markdown format.
>> > >>>>>>>
>> > >>>>>>> Thanks everyone!
>> > >>>>>>>
>> > >>>>>>>
>> > >>>>>>>>
>> > >>>>>>>> Andrew
>> > >>>>>>>>
>> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <
>> prtkgaur@gmail.com>
>> > >>>>>>>> wrote:
>> > >>>>>>>>
>> > >>>>>>>>> Hi team,
>> > >>>>>>>>>
>> > >>>>>>>>> 1) Andrew
>> > >>>>>>>>>
>> > >>>>>>>>> - Thanks for working on test files. My PR did add all the
>> > test
>> > >>>>>>>> files I
>> > >>>>>>>>> used to benchmark on datasets. Maybe we can club it
>> together.
>> > >>>>>>>> WIll also
>> > >>>>>>>>> aid
>> > >>>>>>>>> cross language testing
>> > >>>>>>>>> - Kosta Tarasov working on Rust implementation. This is
>> > great.
>> > >>>>>>>> Thanks
>> > >>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>> 2) Antoine
>> > >>>>>>>>>
>> > >>>>>>>>> - Thanks a lot for reporting the numbers on AMD. Looks
>> like
>> > you
>> > >>>>>>>> are
>> > >>>>>>>>> getting 8X the decoding performance of BSS. This is
>> > amazing!!.
>> > >>>>>>>>> - Thanks for acknowledging the sampling design.
>> > >>>>>>>>> - I agree with you on Fastlanes. In some crude
>> experiments I
>> > >>>>>>>> didn't get
>> > >>>>>>>>> a good perf benefit from it on Graviton3 (but maybe there
>> was
>> > >>>>>>>> something
>> > >>>>>>>>> wrong with my implementation).
>> > >>>>>>>>> - Locking the 16bit exception encoding for the spec in
>> this
>> > >>>>>>>> case.
>> > >>>>>>>>> - Awesome I think we have solved for all open questions
>> minus
>> > >>>>>>>> the
>> > >>>>>>>>> version byte :). (will get back on this soon)
>> > >>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>> 3) Micah
>> > >>>>>>>>>
>> > >>>>>>>>> - FastLanes : The current spec does allow for using
>> FastLane
>> > >>>>>>>> with the
>> > >>>>>>>>> configurable enum value for layout. We should be able to
>> > inject
>> > >>>>>>>> any
>> > >>>>>>>>> layout
>> > >>>>>>>>> in the current design.
>> > >>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>> Working on resolving all remaining open comments on the spec
>> this
>> > >>>>>>>> week.
>> > >>>>>>>>>
>> > >>>>>>>>> Best
>> > >>>>>>>>> Prateek
>> > >>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <
>> > >>>>>>>> stevel@cloudera.com>
>> > >>>>>>>>> wrote:
>> > >>>>>>>>>
>> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
>> > >>>>>>>> emkornfield@gmail.com>
>> > >>>>>>>>>> wrote:
>> > >>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>> It looks like the actual issue described for ORC in the
>> paper
>> > >>>>>>>> is that
>> > >>>>>>>>> it
>> > >>>>>>>>>>> has multiple sub-encodings in a batch. This is different
>> then
>> > >>>>>>>> the
>> > >>>>>>>>> design
>> > >>>>>>>>>>> proposed here where there is still fixed encoding per page
>> in
>> > >>>>>>>> parquet.
>> > >>>>>>>>>>> Given reasonably sized pages I don't think branch
>> > >>>>>>>> misprediction should
>> > >>>>>>>>>> be a
>> > >>>>>>>>>>> big issue for new encodings. I agree that we should be
>> > >>>>>>>> conservative in
>> > >>>>>>>>>>> general for adding new encodings.
>> > >>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>> +1
>> > >>>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>
>> > >
>> >
>> >
>> >
>>
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by PRATEEK GAUR <pr...@gmail.com>.
Hi Team,
Just wanted to provide some updates on ALP.
Micah and I have done 3-4 rounds of review for the c++ PR :
https://github.com/apache/arrow/pull/48345
For the Java implementation Vinoo has been working on following PR :
https://github.com/apache/parquet-java/pull/3397
Best
Prateek
On Tue, May 5, 2026 at 1:50 PM Micah Kornfield <em...@gmail.com>
wrote:
> Hi Antoine,
>
> > Apologies if the question was already asked, but should we care about
> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
> > BYTE_STREAM_SPLIT + LZ4 for example?
>
> It was. We thought we could defer it for the following reasons:
> 1. It's not clear there are a lot of easy reference datasets to test its
> effectiveness. It does look like there might be one or two on huggingface
> (e.g. https://huggingface.co/datasets/kikitora/curdie).
> 2. It seemed likely that float 16 was more likely used for values that
> were less likely to reduce to decimal values.
> 3. It could be added as an extension later if needed.
>
> Cheers,
> Micah
>
>
> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <an...@python.org> wrote:
>
> >
> > Hello,
> >
> > Apologies if the question was already asked, but should we care about
> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
> > BYTE_STREAM_SPLIT + LZ4 for example?
> >
> > Regards
> >
> > Antoine.
> >
> >
> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
> > > Thanks Andrew and Micah for review feedback on the two PR's
> > > 1) (c++ arrow repo) https://github.com/apache/arrow/pull/48345/changes
> > > 2) (parquet-format repo)
> > https://github.com/apache/parquet-format/pull/557
> > >
> > > I have addressed all (unless I missed something) comments on the two
> > PR's.
> > >
> > > Best
> > > Prateek
> > >
> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <pr...@gmail.com>
> wrote:
> > >
> > >> Thanks Andrew and Micah.
> > >>
> > >> `fair amount of feedback on at least the implementations`
> > >> For the c++ I have already started addressing the feedback, I should
> be
> > >> done with that Monday/Tuesday.
> > >> I think Vinoo too has been making good progress on the Java
> > implementation.
> > >>
> > >> Best
> > >> Prateek
> > >>
> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <an...@gmail.com>
> > >> wrote:
> > >>
> > >>> Got it. Thank you for the clarification -- I will try and look into
> the
> > >>> spec and the Rust implementation[1] in this next week
> > >>>
> > >>> [1]: https://github.com/apache/arrow-rs/pull/9372
> > >>>
> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <
> > emkornfield@gmail.com>
> > >>> wrote:
> > >>>
> > >>>> Hi Andrew,
> > >>>> I think there is a fair amount of feedback on at least the
> > >>>> implementations, typically I think we've waited till they are close
> to
> > >>>> mergeable before a final vote. Otherwise I agree we are very close.
> > >>>>
> > >>>> -Micah
> > >>>>
> > >>>> On Saturday, April 25, 2026, Andrew Lamb <an...@gmail.com>
> > wrote:
> > >>>>
> > >>>>> Thanks Prateek,
> > >>>>>
> > >>>>> I think from this content it looks to me like we are ready to
> start a
> > >>>>> vote to explicitly accept ALP into Parquet
> > >>>>>
> > >>>>> Does anyone know of a reason we should postpone it for longer?
> > >>>>> Perhaps someone needs some more time to review?
> > >>>>>
> > >>>>> Andrew
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <pr...@gmail.com>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Hi team,
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> Hope everyone is doing well. I got a chance to work through all
> the
> > >>>>>> remaining feedback and update the spec doc. Here are the new
> > artifacts
> > >>>>>>
> > >>>>>> 1) Spec document :
> > >>>>>>
> >
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
> > >>>>>>
> > >>>>>> 2) Spec document in parquet format repo :
> > >>>>>> https://github.com/apache/parquet-format/pull/557
> > >>>>>>
> > >>>>>> 3) Alp implementation in arrow c++ repo :
> > >>>>>> https://github.com/apache/arrow/pull/48345/changes
> > >>>>>>
> > >>>>>> 4) Alp implementation in parquet-java repo : Work for Vinoo and
> > Julien
> > >>>>>> https://github.com/apache/parquet-java/pull/3397
> > >>>>>>
> > >>>>>> 5) PR with test and benchmarking artifacts in parquet-testing
> repo :
> > >>>>>> https://github.com/apache/parquet-testing/pull/100
> > >>>>>>
> > >>>>>>
> > >>>>>> And
> > >>>>>>
> > >>>>>>
> > >>>>>> - Go : Arnav just submitted an in progress implementation in
> Go.
> > >>>>>> https://github.com/apache/arrow-go/pull/704 (I haven't
> started
> > >>>>>> looking at it yet)
> > >>>>>> - Rust : I remember Andrew mentioned that this work is also in
> > >>>>>> progress (So 4 languages!)
> > >>>>>>
> > >>>>>>
> > >>>>>> *Arrow C++ implementation *
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> The PR is out and was also used by Antoine to report the numbers
> as
> > >>>>>> reported here. Micah and Konstantin have given 1 round of feedback
> > >>>>>> and I'm addressing them today. Please note that the default
> > >>>>>> optimization flag for compiling is O2 and not Q3. I got around 70%
> > >>>>>> performance improvement in the decoding speed when using the O3
> > flag.
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> *Parqet-MR Java implementation (working with Vinoo and Julien) and
> > **Cross
> > >>>>>> Language testing*
> > >>>>>>
> > >>>>>>
> > >>>>>> Let me know if you have any questions or feedback.
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> Now pasting some performance numbers
> > >>>>>>
> > >>>>>>
> > >>>>>> Table 1: C++ ALP Double Decode — Spotify Columns (Graviton 3,
> ARM
> > >>>>>> Neoverse V1)
> > >>>>>>
> > >>>>>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
> > >>>>>>
> > >>>>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │ Speedup │
> > >>>>>>
> > >>>>>> ├──────────────────┼──────────────┼──────────────┼─────────┤
> > >>>>>>
> > >>>>>> │ valence │ 3,155 │ 5,523 │ 1.75x │
> > >>>>>>
> > >>>>>> │ danceability │ 3,233 │ 5,685 │ 1.76x │
> > >>>>>>
> > >>>>>> │ energy │ 3,197 │ 5,652 │ 1.77x │
> > >>>>>>
> > >>>>>> │ loudness │ 3,186 │ 5,473 │ 1.72x │
> > >>>>>>
> > >>>>>> └──────────────────┴──────────────┴──────────────┴─────────┘
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <pr...@gmail.com>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> @Micah Kornfield <em...@gmail.com> : Got it.
> > >>>>>>>
> > >>>>>>> @Andrew Lamb <an...@gmail.com>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>> Do you think it would be good to start moving the spec
> development
> > >>>>>>>> into
> > >>>>>>>> markdown format, in preparation for finalizing it?
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>> Yes I'll update the numbers for some of the examples I have in
> the
> > >>>>>>> spec based
> > >>>>>>> on the updated header size. Then we should be good to go for the
> > >>>>>>> markdown format.
> > >>>>>>>
> > >>>>>>> Thanks everyone!
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>>
> > >>>>>>>> Andrew
> > >>>>>>>>
> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <
> prtkgaur@gmail.com>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Hi team,
> > >>>>>>>>>
> > >>>>>>>>> 1) Andrew
> > >>>>>>>>>
> > >>>>>>>>> - Thanks for working on test files. My PR did add all the
> > test
> > >>>>>>>> files I
> > >>>>>>>>> used to benchmark on datasets. Maybe we can club it
> together.
> > >>>>>>>> WIll also
> > >>>>>>>>> aid
> > >>>>>>>>> cross language testing
> > >>>>>>>>> - Kosta Tarasov working on Rust implementation. This is
> > great.
> > >>>>>>>> Thanks
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> 2) Antoine
> > >>>>>>>>>
> > >>>>>>>>> - Thanks a lot for reporting the numbers on AMD. Looks like
> > you
> > >>>>>>>> are
> > >>>>>>>>> getting 8X the decoding performance of BSS. This is
> > amazing!!.
> > >>>>>>>>> - Thanks for acknowledging the sampling design.
> > >>>>>>>>> - I agree with you on Fastlanes. In some crude experiments
> I
> > >>>>>>>> didn't get
> > >>>>>>>>> a good perf benefit from it on Graviton3 (but maybe there
> was
> > >>>>>>>> something
> > >>>>>>>>> wrong with my implementation).
> > >>>>>>>>> - Locking the 16bit exception encoding for the spec in this
> > >>>>>>>> case.
> > >>>>>>>>> - Awesome I think we have solved for all open questions
> minus
> > >>>>>>>> the
> > >>>>>>>>> version byte :). (will get back on this soon)
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> 3) Micah
> > >>>>>>>>>
> > >>>>>>>>> - FastLanes : The current spec does allow for using
> FastLane
> > >>>>>>>> with the
> > >>>>>>>>> configurable enum value for layout. We should be able to
> > inject
> > >>>>>>>> any
> > >>>>>>>>> layout
> > >>>>>>>>> in the current design.
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> Working on resolving all remaining open comments on the spec
> this
> > >>>>>>>> week.
> > >>>>>>>>>
> > >>>>>>>>> Best
> > >>>>>>>>> Prateek
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <
> > >>>>>>>> stevel@cloudera.com>
> > >>>>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
> > >>>>>>>> emkornfield@gmail.com>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> It looks like the actual issue described for ORC in the paper
> > >>>>>>>> is that
> > >>>>>>>>> it
> > >>>>>>>>>>> has multiple sub-encodings in a batch. This is different
> then
> > >>>>>>>> the
> > >>>>>>>>> design
> > >>>>>>>>>>> proposed here where there is still fixed encoding per page in
> > >>>>>>>> parquet.
> > >>>>>>>>>>> Given reasonably sized pages I don't think branch
> > >>>>>>>> misprediction should
> > >>>>>>>>>> be a
> > >>>>>>>>>>> big issue for new encodings. I agree that we should be
> > >>>>>>>> conservative in
> > >>>>>>>>>>> general for adding new encodings.
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>> +1
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >
> >
> >
> >
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by PRATEEK GAUR <pr...@gmail.com>.
Hi Team,
Just wanted to provide some updates on ALP.
Micah and I have done 3-4 rounds of review for the c++ PR :
https://github.com/apache/arrow/pull/48345
For the Java implementation Vinoo has been working on following PR :
https://github.com/apache/parquet-java/pull/3397
Best
Prateek
On Tue, May 5, 2026 at 1:50 PM Micah Kornfield <em...@gmail.com>
wrote:
> Hi Antoine,
>
> > Apologies if the question was already asked, but should we care about
> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
> > BYTE_STREAM_SPLIT + LZ4 for example?
>
> It was. We thought we could defer it for the following reasons:
> 1. It's not clear there are a lot of easy reference datasets to test its
> effectiveness. It does look like there might be one or two on huggingface
> (e.g. https://huggingface.co/datasets/kikitora/curdie).
> 2. It seemed likely that float 16 was more likely used for values that
> were less likely to reduce to decimal values.
> 3. It could be added as an extension later if needed.
>
> Cheers,
> Micah
>
>
> On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <an...@python.org> wrote:
>
> >
> > Hello,
> >
> > Apologies if the question was already asked, but should we care about
> > FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
> > BYTE_STREAM_SPLIT + LZ4 for example?
> >
> > Regards
> >
> > Antoine.
> >
> >
> > Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
> > > Thanks Andrew and Micah for review feedback on the two PR's
> > > 1) (c++ arrow repo) https://github.com/apache/arrow/pull/48345/changes
> > > 2) (parquet-format repo)
> > https://github.com/apache/parquet-format/pull/557
> > >
> > > I have addressed all (unless I missed something) comments on the two
> > PR's.
> > >
> > > Best
> > > Prateek
> > >
> > > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <pr...@gmail.com>
> wrote:
> > >
> > >> Thanks Andrew and Micah.
> > >>
> > >> `fair amount of feedback on at least the implementations`
> > >> For the c++ I have already started addressing the feedback, I should
> be
> > >> done with that Monday/Tuesday.
> > >> I think Vinoo too has been making good progress on the Java
> > implementation.
> > >>
> > >> Best
> > >> Prateek
> > >>
> > >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <an...@gmail.com>
> > >> wrote:
> > >>
> > >>> Got it. Thank you for the clarification -- I will try and look into
> the
> > >>> spec and the Rust implementation[1] in this next week
> > >>>
> > >>> [1]: https://github.com/apache/arrow-rs/pull/9372
> > >>>
> > >>> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <
> > emkornfield@gmail.com>
> > >>> wrote:
> > >>>
> > >>>> Hi Andrew,
> > >>>> I think there is a fair amount of feedback on at least the
> > >>>> implementations, typically I think we've waited till they are close
> to
> > >>>> mergeable before a final vote. Otherwise I agree we are very close.
> > >>>>
> > >>>> -Micah
> > >>>>
> > >>>> On Saturday, April 25, 2026, Andrew Lamb <an...@gmail.com>
> > wrote:
> > >>>>
> > >>>>> Thanks Prateek,
> > >>>>>
> > >>>>> I think from this content it looks to me like we are ready to
> start a
> > >>>>> vote to explicitly accept ALP into Parquet
> > >>>>>
> > >>>>> Does anyone know of a reason we should postpone it for longer?
> > >>>>> Perhaps someone needs some more time to review?
> > >>>>>
> > >>>>> Andrew
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <pr...@gmail.com>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Hi team,
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> Hope everyone is doing well. I got a chance to work through all
> the
> > >>>>>> remaining feedback and update the spec doc. Here are the new
> > artifacts
> > >>>>>>
> > >>>>>> 1) Spec document :
> > >>>>>>
> >
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
> > >>>>>>
> > >>>>>> 2) Spec document in parquet format repo :
> > >>>>>> https://github.com/apache/parquet-format/pull/557
> > >>>>>>
> > >>>>>> 3) Alp implementation in arrow c++ repo :
> > >>>>>> https://github.com/apache/arrow/pull/48345/changes
> > >>>>>>
> > >>>>>> 4) Alp implementation in parquet-java repo : Work for Vinoo and
> > Julien
> > >>>>>> https://github.com/apache/parquet-java/pull/3397
> > >>>>>>
> > >>>>>> 5) PR with test and benchmarking artifacts in parquet-testing
> repo :
> > >>>>>> https://github.com/apache/parquet-testing/pull/100
> > >>>>>>
> > >>>>>>
> > >>>>>> And
> > >>>>>>
> > >>>>>>
> > >>>>>> - Go : Arnav just submitted an in progress implementation in
> Go.
> > >>>>>> https://github.com/apache/arrow-go/pull/704 (I haven't
> started
> > >>>>>> looking at it yet)
> > >>>>>> - Rust : I remember Andrew mentioned that this work is also in
> > >>>>>> progress (So 4 languages!)
> > >>>>>>
> > >>>>>>
> > >>>>>> *Arrow C++ implementation *
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> The PR is out and was also used by Antoine to report the numbers
> as
> > >>>>>> reported here. Micah and Konstantin have given 1 round of feedback
> > >>>>>> and I'm addressing them today. Please note that the default
> > >>>>>> optimization flag for compiling is O2 and not Q3. I got around 70%
> > >>>>>> performance improvement in the decoding speed when using the O3
> > flag.
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> *Parqet-MR Java implementation (working with Vinoo and Julien) and
> > **Cross
> > >>>>>> Language testing*
> > >>>>>>
> > >>>>>>
> > >>>>>> Let me know if you have any questions or feedback.
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> Now pasting some performance numbers
> > >>>>>>
> > >>>>>>
> > >>>>>> Table 1: C++ ALP Double Decode — Spotify Columns (Graviton 3,
> ARM
> > >>>>>> Neoverse V1)
> > >>>>>>
> > >>>>>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
> > >>>>>>
> > >>>>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │ Speedup │
> > >>>>>>
> > >>>>>> ├──────────────────┼──────────────┼──────────────┼─────────┤
> > >>>>>>
> > >>>>>> │ valence │ 3,155 │ 5,523 │ 1.75x │
> > >>>>>>
> > >>>>>> │ danceability │ 3,233 │ 5,685 │ 1.76x │
> > >>>>>>
> > >>>>>> │ energy │ 3,197 │ 5,652 │ 1.77x │
> > >>>>>>
> > >>>>>> │ loudness │ 3,186 │ 5,473 │ 1.72x │
> > >>>>>>
> > >>>>>> └──────────────────┴──────────────┴──────────────┴─────────┘
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <pr...@gmail.com>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> @Micah Kornfield <em...@gmail.com> : Got it.
> > >>>>>>>
> > >>>>>>> @Andrew Lamb <an...@gmail.com>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>> Do you think it would be good to start moving the spec
> development
> > >>>>>>>> into
> > >>>>>>>> markdown format, in preparation for finalizing it?
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>> Yes I'll update the numbers for some of the examples I have in
> the
> > >>>>>>> spec based
> > >>>>>>> on the updated header size. Then we should be good to go for the
> > >>>>>>> markdown format.
> > >>>>>>>
> > >>>>>>> Thanks everyone!
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>>
> > >>>>>>>> Andrew
> > >>>>>>>>
> > >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <
> prtkgaur@gmail.com>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Hi team,
> > >>>>>>>>>
> > >>>>>>>>> 1) Andrew
> > >>>>>>>>>
> > >>>>>>>>> - Thanks for working on test files. My PR did add all the
> > test
> > >>>>>>>> files I
> > >>>>>>>>> used to benchmark on datasets. Maybe we can club it
> together.
> > >>>>>>>> WIll also
> > >>>>>>>>> aid
> > >>>>>>>>> cross language testing
> > >>>>>>>>> - Kosta Tarasov working on Rust implementation. This is
> > great.
> > >>>>>>>> Thanks
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> 2) Antoine
> > >>>>>>>>>
> > >>>>>>>>> - Thanks a lot for reporting the numbers on AMD. Looks like
> > you
> > >>>>>>>> are
> > >>>>>>>>> getting 8X the decoding performance of BSS. This is
> > amazing!!.
> > >>>>>>>>> - Thanks for acknowledging the sampling design.
> > >>>>>>>>> - I agree with you on Fastlanes. In some crude experiments
> I
> > >>>>>>>> didn't get
> > >>>>>>>>> a good perf benefit from it on Graviton3 (but maybe there
> was
> > >>>>>>>> something
> > >>>>>>>>> wrong with my implementation).
> > >>>>>>>>> - Locking the 16bit exception encoding for the spec in this
> > >>>>>>>> case.
> > >>>>>>>>> - Awesome I think we have solved for all open questions
> minus
> > >>>>>>>> the
> > >>>>>>>>> version byte :). (will get back on this soon)
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> 3) Micah
> > >>>>>>>>>
> > >>>>>>>>> - FastLanes : The current spec does allow for using
> FastLane
> > >>>>>>>> with the
> > >>>>>>>>> configurable enum value for layout. We should be able to
> > inject
> > >>>>>>>> any
> > >>>>>>>>> layout
> > >>>>>>>>> in the current design.
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> Working on resolving all remaining open comments on the spec
> this
> > >>>>>>>> week.
> > >>>>>>>>>
> > >>>>>>>>> Best
> > >>>>>>>>> Prateek
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <
> > >>>>>>>> stevel@cloudera.com>
> > >>>>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
> > >>>>>>>> emkornfield@gmail.com>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> It looks like the actual issue described for ORC in the paper
> > >>>>>>>> is that
> > >>>>>>>>> it
> > >>>>>>>>>>> has multiple sub-encodings in a batch. This is different
> then
> > >>>>>>>> the
> > >>>>>>>>> design
> > >>>>>>>>>>> proposed here where there is still fixed encoding per page in
> > >>>>>>>> parquet.
> > >>>>>>>>>>> Given reasonably sized pages I don't think branch
> > >>>>>>>> misprediction should
> > >>>>>>>>>> be a
> > >>>>>>>>>>> big issue for new encodings. I agree that we should be
> > >>>>>>>> conservative in
> > >>>>>>>>>>> general for adding new encodings.
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>> +1
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >
> >
> >
> >
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Micah Kornfield <em...@gmail.com>.
Hi Antoine,
> Apologies if the question was already asked, but should we care about
> FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
> BYTE_STREAM_SPLIT + LZ4 for example?
It was. We thought we could defer it for the following reasons:
1. It's not clear there are a lot of easy reference datasets to test its
effectiveness. It does look like there might be one or two on huggingface
(e.g. https://huggingface.co/datasets/kikitora/curdie).
2. It seemed likely that float 16 was more likely used for values that
were less likely to reduce to decimal values.
3. It could be added as an extension later if needed.
Cheers,
Micah
On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <an...@python.org> wrote:
>
> Hello,
>
> Apologies if the question was already asked, but should we care about
> FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
> BYTE_STREAM_SPLIT + LZ4 for example?
>
> Regards
>
> Antoine.
>
>
> Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
> > Thanks Andrew and Micah for review feedback on the two PR's
> > 1) (c++ arrow repo) https://github.com/apache/arrow/pull/48345/changes
> > 2) (parquet-format repo)
> https://github.com/apache/parquet-format/pull/557
> >
> > I have addressed all (unless I missed something) comments on the two
> PR's.
> >
> > Best
> > Prateek
> >
> > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <pr...@gmail.com> wrote:
> >
> >> Thanks Andrew and Micah.
> >>
> >> `fair amount of feedback on at least the implementations`
> >> For the c++ I have already started addressing the feedback, I should be
> >> done with that Monday/Tuesday.
> >> I think Vinoo too has been making good progress on the Java
> implementation.
> >>
> >> Best
> >> Prateek
> >>
> >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <an...@gmail.com>
> >> wrote:
> >>
> >>> Got it. Thank you for the clarification -- I will try and look into the
> >>> spec and the Rust implementation[1] in this next week
> >>>
> >>> [1]: https://github.com/apache/arrow-rs/pull/9372
> >>>
> >>> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <
> emkornfield@gmail.com>
> >>> wrote:
> >>>
> >>>> Hi Andrew,
> >>>> I think there is a fair amount of feedback on at least the
> >>>> implementations, typically I think we've waited till they are close to
> >>>> mergeable before a final vote. Otherwise I agree we are very close.
> >>>>
> >>>> -Micah
> >>>>
> >>>> On Saturday, April 25, 2026, Andrew Lamb <an...@gmail.com>
> wrote:
> >>>>
> >>>>> Thanks Prateek,
> >>>>>
> >>>>> I think from this content it looks to me like we are ready to start a
> >>>>> vote to explicitly accept ALP into Parquet
> >>>>>
> >>>>> Does anyone know of a reason we should postpone it for longer?
> >>>>> Perhaps someone needs some more time to review?
> >>>>>
> >>>>> Andrew
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <pr...@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>>> Hi team,
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Hope everyone is doing well. I got a chance to work through all the
> >>>>>> remaining feedback and update the spec doc. Here are the new
> artifacts
> >>>>>>
> >>>>>> 1) Spec document :
> >>>>>>
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
> >>>>>>
> >>>>>> 2) Spec document in parquet format repo :
> >>>>>> https://github.com/apache/parquet-format/pull/557
> >>>>>>
> >>>>>> 3) Alp implementation in arrow c++ repo :
> >>>>>> https://github.com/apache/arrow/pull/48345/changes
> >>>>>>
> >>>>>> 4) Alp implementation in parquet-java repo : Work for Vinoo and
> Julien
> >>>>>> https://github.com/apache/parquet-java/pull/3397
> >>>>>>
> >>>>>> 5) PR with test and benchmarking artifacts in parquet-testing repo :
> >>>>>> https://github.com/apache/parquet-testing/pull/100
> >>>>>>
> >>>>>>
> >>>>>> And
> >>>>>>
> >>>>>>
> >>>>>> - Go : Arnav just submitted an in progress implementation in Go.
> >>>>>> https://github.com/apache/arrow-go/pull/704 (I haven't started
> >>>>>> looking at it yet)
> >>>>>> - Rust : I remember Andrew mentioned that this work is also in
> >>>>>> progress (So 4 languages!)
> >>>>>>
> >>>>>>
> >>>>>> *Arrow C++ implementation *
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> The PR is out and was also used by Antoine to report the numbers as
> >>>>>> reported here. Micah and Konstantin have given 1 round of feedback
> >>>>>> and I'm addressing them today. Please note that the default
> >>>>>> optimization flag for compiling is O2 and not Q3. I got around 70%
> >>>>>> performance improvement in the decoding speed when using the O3
> flag.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> *Parqet-MR Java implementation (working with Vinoo and Julien) and
> **Cross
> >>>>>> Language testing*
> >>>>>>
> >>>>>>
> >>>>>> Let me know if you have any questions or feedback.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Now pasting some performance numbers
> >>>>>>
> >>>>>>
> >>>>>> Table 1: C++ ALP Double Decode — Spotify Columns (Graviton 3, ARM
> >>>>>> Neoverse V1)
> >>>>>>
> >>>>>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
> >>>>>>
> >>>>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │ Speedup │
> >>>>>>
> >>>>>> ├──────────────────┼──────────────┼──────────────┼─────────┤
> >>>>>>
> >>>>>> │ valence │ 3,155 │ 5,523 │ 1.75x │
> >>>>>>
> >>>>>> │ danceability │ 3,233 │ 5,685 │ 1.76x │
> >>>>>>
> >>>>>> │ energy │ 3,197 │ 5,652 │ 1.77x │
> >>>>>>
> >>>>>> │ loudness │ 3,186 │ 5,473 │ 1.72x │
> >>>>>>
> >>>>>> └──────────────────┴──────────────┴──────────────┴─────────┘
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <pr...@gmail.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> @Micah Kornfield <em...@gmail.com> : Got it.
> >>>>>>>
> >>>>>>> @Andrew Lamb <an...@gmail.com>
> >>>>>>>
> >>>>>>>
> >>>>>>>> Do you think it would be good to start moving the spec development
> >>>>>>>> into
> >>>>>>>> markdown format, in preparation for finalizing it?
> >>>>>>>>
> >>>>>>>
> >>>>>>> Yes I'll update the numbers for some of the examples I have in the
> >>>>>>> spec based
> >>>>>>> on the updated header size. Then we should be good to go for the
> >>>>>>> markdown format.
> >>>>>>>
> >>>>>>> Thanks everyone!
> >>>>>>>
> >>>>>>>
> >>>>>>>>
> >>>>>>>> Andrew
> >>>>>>>>
> >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <pr...@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Hi team,
> >>>>>>>>>
> >>>>>>>>> 1) Andrew
> >>>>>>>>>
> >>>>>>>>> - Thanks for working on test files. My PR did add all the
> test
> >>>>>>>> files I
> >>>>>>>>> used to benchmark on datasets. Maybe we can club it together.
> >>>>>>>> WIll also
> >>>>>>>>> aid
> >>>>>>>>> cross language testing
> >>>>>>>>> - Kosta Tarasov working on Rust implementation. This is
> great.
> >>>>>>>> Thanks
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> 2) Antoine
> >>>>>>>>>
> >>>>>>>>> - Thanks a lot for reporting the numbers on AMD. Looks like
> you
> >>>>>>>> are
> >>>>>>>>> getting 8X the decoding performance of BSS. This is
> amazing!!.
> >>>>>>>>> - Thanks for acknowledging the sampling design.
> >>>>>>>>> - I agree with you on Fastlanes. In some crude experiments I
> >>>>>>>> didn't get
> >>>>>>>>> a good perf benefit from it on Graviton3 (but maybe there was
> >>>>>>>> something
> >>>>>>>>> wrong with my implementation).
> >>>>>>>>> - Locking the 16bit exception encoding for the spec in this
> >>>>>>>> case.
> >>>>>>>>> - Awesome I think we have solved for all open questions minus
> >>>>>>>> the
> >>>>>>>>> version byte :). (will get back on this soon)
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> 3) Micah
> >>>>>>>>>
> >>>>>>>>> - FastLanes : The current spec does allow for using FastLane
> >>>>>>>> with the
> >>>>>>>>> configurable enum value for layout. We should be able to
> inject
> >>>>>>>> any
> >>>>>>>>> layout
> >>>>>>>>> in the current design.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Working on resolving all remaining open comments on the spec this
> >>>>>>>> week.
> >>>>>>>>>
> >>>>>>>>> Best
> >>>>>>>>> Prateek
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <
> >>>>>>>> stevel@cloudera.com>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
> >>>>>>>> emkornfield@gmail.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> It looks like the actual issue described for ORC in the paper
> >>>>>>>> is that
> >>>>>>>>> it
> >>>>>>>>>>> has multiple sub-encodings in a batch. This is different then
> >>>>>>>> the
> >>>>>>>>> design
> >>>>>>>>>>> proposed here where there is still fixed encoding per page in
> >>>>>>>> parquet.
> >>>>>>>>>>> Given reasonably sized pages I don't think branch
> >>>>>>>> misprediction should
> >>>>>>>>>> be a
> >>>>>>>>>>> big issue for new encodings. I agree that we should be
> >>>>>>>> conservative in
> >>>>>>>>>>> general for adding new encodings.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>> +1
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >
>
>
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Micah Kornfield <em...@gmail.com>.
Hi Antoine,
> Apologies if the question was already asked, but should we care about
> FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
> BYTE_STREAM_SPLIT + LZ4 for example?
It was. We thought we could defer it for the following reasons:
1. It's not clear there are a lot of easy reference datasets to test its
effectiveness. It does look like there might be one or two on huggingface
(e.g. https://huggingface.co/datasets/kikitora/curdie).
2. It seemed likely that float 16 was more likely used for values that
were less likely to reduce to decimal values.
3. It could be added as an extension later if needed.
Cheers,
Micah
On Tue, May 5, 2026 at 1:40 PM Antoine Pitrou <an...@python.org> wrote:
>
> Hello,
>
> Apologies if the question was already asked, but should we care about
> FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
> BYTE_STREAM_SPLIT + LZ4 for example?
>
> Regards
>
> Antoine.
>
>
> Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
> > Thanks Andrew and Micah for review feedback on the two PR's
> > 1) (c++ arrow repo) https://github.com/apache/arrow/pull/48345/changes
> > 2) (parquet-format repo)
> https://github.com/apache/parquet-format/pull/557
> >
> > I have addressed all (unless I missed something) comments on the two
> PR's.
> >
> > Best
> > Prateek
> >
> > On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <pr...@gmail.com> wrote:
> >
> >> Thanks Andrew and Micah.
> >>
> >> `fair amount of feedback on at least the implementations`
> >> For the c++ I have already started addressing the feedback, I should be
> >> done with that Monday/Tuesday.
> >> I think Vinoo too has been making good progress on the Java
> implementation.
> >>
> >> Best
> >> Prateek
> >>
> >> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <an...@gmail.com>
> >> wrote:
> >>
> >>> Got it. Thank you for the clarification -- I will try and look into the
> >>> spec and the Rust implementation[1] in this next week
> >>>
> >>> [1]: https://github.com/apache/arrow-rs/pull/9372
> >>>
> >>> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <
> emkornfield@gmail.com>
> >>> wrote:
> >>>
> >>>> Hi Andrew,
> >>>> I think there is a fair amount of feedback on at least the
> >>>> implementations, typically I think we've waited till they are close to
> >>>> mergeable before a final vote. Otherwise I agree we are very close.
> >>>>
> >>>> -Micah
> >>>>
> >>>> On Saturday, April 25, 2026, Andrew Lamb <an...@gmail.com>
> wrote:
> >>>>
> >>>>> Thanks Prateek,
> >>>>>
> >>>>> I think from this content it looks to me like we are ready to start a
> >>>>> vote to explicitly accept ALP into Parquet
> >>>>>
> >>>>> Does anyone know of a reason we should postpone it for longer?
> >>>>> Perhaps someone needs some more time to review?
> >>>>>
> >>>>> Andrew
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <pr...@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>>> Hi team,
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Hope everyone is doing well. I got a chance to work through all the
> >>>>>> remaining feedback and update the spec doc. Here are the new
> artifacts
> >>>>>>
> >>>>>> 1) Spec document :
> >>>>>>
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
> >>>>>>
> >>>>>> 2) Spec document in parquet format repo :
> >>>>>> https://github.com/apache/parquet-format/pull/557
> >>>>>>
> >>>>>> 3) Alp implementation in arrow c++ repo :
> >>>>>> https://github.com/apache/arrow/pull/48345/changes
> >>>>>>
> >>>>>> 4) Alp implementation in parquet-java repo : Work for Vinoo and
> Julien
> >>>>>> https://github.com/apache/parquet-java/pull/3397
> >>>>>>
> >>>>>> 5) PR with test and benchmarking artifacts in parquet-testing repo :
> >>>>>> https://github.com/apache/parquet-testing/pull/100
> >>>>>>
> >>>>>>
> >>>>>> And
> >>>>>>
> >>>>>>
> >>>>>> - Go : Arnav just submitted an in progress implementation in Go.
> >>>>>> https://github.com/apache/arrow-go/pull/704 (I haven't started
> >>>>>> looking at it yet)
> >>>>>> - Rust : I remember Andrew mentioned that this work is also in
> >>>>>> progress (So 4 languages!)
> >>>>>>
> >>>>>>
> >>>>>> *Arrow C++ implementation *
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> The PR is out and was also used by Antoine to report the numbers as
> >>>>>> reported here. Micah and Konstantin have given 1 round of feedback
> >>>>>> and I'm addressing them today. Please note that the default
> >>>>>> optimization flag for compiling is O2 and not Q3. I got around 70%
> >>>>>> performance improvement in the decoding speed when using the O3
> flag.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> *Parqet-MR Java implementation (working with Vinoo and Julien) and
> **Cross
> >>>>>> Language testing*
> >>>>>>
> >>>>>>
> >>>>>> Let me know if you have any questions or feedback.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Now pasting some performance numbers
> >>>>>>
> >>>>>>
> >>>>>> Table 1: C++ ALP Double Decode — Spotify Columns (Graviton 3, ARM
> >>>>>> Neoverse V1)
> >>>>>>
> >>>>>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
> >>>>>>
> >>>>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │ Speedup │
> >>>>>>
> >>>>>> ├──────────────────┼──────────────┼──────────────┼─────────┤
> >>>>>>
> >>>>>> │ valence │ 3,155 │ 5,523 │ 1.75x │
> >>>>>>
> >>>>>> │ danceability │ 3,233 │ 5,685 │ 1.76x │
> >>>>>>
> >>>>>> │ energy │ 3,197 │ 5,652 │ 1.77x │
> >>>>>>
> >>>>>> │ loudness │ 3,186 │ 5,473 │ 1.72x │
> >>>>>>
> >>>>>> └──────────────────┴──────────────┴──────────────┴─────────┘
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <pr...@gmail.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> @Micah Kornfield <em...@gmail.com> : Got it.
> >>>>>>>
> >>>>>>> @Andrew Lamb <an...@gmail.com>
> >>>>>>>
> >>>>>>>
> >>>>>>>> Do you think it would be good to start moving the spec development
> >>>>>>>> into
> >>>>>>>> markdown format, in preparation for finalizing it?
> >>>>>>>>
> >>>>>>>
> >>>>>>> Yes I'll update the numbers for some of the examples I have in the
> >>>>>>> spec based
> >>>>>>> on the updated header size. Then we should be good to go for the
> >>>>>>> markdown format.
> >>>>>>>
> >>>>>>> Thanks everyone!
> >>>>>>>
> >>>>>>>
> >>>>>>>>
> >>>>>>>> Andrew
> >>>>>>>>
> >>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <pr...@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Hi team,
> >>>>>>>>>
> >>>>>>>>> 1) Andrew
> >>>>>>>>>
> >>>>>>>>> - Thanks for working on test files. My PR did add all the
> test
> >>>>>>>> files I
> >>>>>>>>> used to benchmark on datasets. Maybe we can club it together.
> >>>>>>>> WIll also
> >>>>>>>>> aid
> >>>>>>>>> cross language testing
> >>>>>>>>> - Kosta Tarasov working on Rust implementation. This is
> great.
> >>>>>>>> Thanks
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> 2) Antoine
> >>>>>>>>>
> >>>>>>>>> - Thanks a lot for reporting the numbers on AMD. Looks like
> you
> >>>>>>>> are
> >>>>>>>>> getting 8X the decoding performance of BSS. This is
> amazing!!.
> >>>>>>>>> - Thanks for acknowledging the sampling design.
> >>>>>>>>> - I agree with you on Fastlanes. In some crude experiments I
> >>>>>>>> didn't get
> >>>>>>>>> a good perf benefit from it on Graviton3 (but maybe there was
> >>>>>>>> something
> >>>>>>>>> wrong with my implementation).
> >>>>>>>>> - Locking the 16bit exception encoding for the spec in this
> >>>>>>>> case.
> >>>>>>>>> - Awesome I think we have solved for all open questions minus
> >>>>>>>> the
> >>>>>>>>> version byte :). (will get back on this soon)
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> 3) Micah
> >>>>>>>>>
> >>>>>>>>> - FastLanes : The current spec does allow for using FastLane
> >>>>>>>> with the
> >>>>>>>>> configurable enum value for layout. We should be able to
> inject
> >>>>>>>> any
> >>>>>>>>> layout
> >>>>>>>>> in the current design.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Working on resolving all remaining open comments on the spec this
> >>>>>>>> week.
> >>>>>>>>>
> >>>>>>>>> Best
> >>>>>>>>> Prateek
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <
> >>>>>>>> stevel@cloudera.com>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
> >>>>>>>> emkornfield@gmail.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> It looks like the actual issue described for ORC in the paper
> >>>>>>>> is that
> >>>>>>>>> it
> >>>>>>>>>>> has multiple sub-encodings in a batch. This is different then
> >>>>>>>> the
> >>>>>>>>> design
> >>>>>>>>>>> proposed here where there is still fixed encoding per page in
> >>>>>>>> parquet.
> >>>>>>>>>>> Given reasonably sized pages I don't think branch
> >>>>>>>> misprediction should
> >>>>>>>>>> be a
> >>>>>>>>>>> big issue for new encodings. I agree that we should be
> >>>>>>>> conservative in
> >>>>>>>>>>> general for adding new encodings.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>> +1
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >
>
>
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Antoine Pitrou <an...@python.org>.
Hello,
Apologies if the question was already asked, but should we care about
FLOAT16 for ALP? Can FLOAT + ALP be more efficient than FLOAT16 +
BYTE_STREAM_SPLIT + LZ4 for example?
Regards
Antoine.
Le 30/04/2026 à 01:10, PRATEEK GAUR a écrit :
> Thanks Andrew and Micah for review feedback on the two PR's
> 1) (c++ arrow repo) https://github.com/apache/arrow/pull/48345/changes
> 2) (parquet-format repo) https://github.com/apache/parquet-format/pull/557
>
> I have addressed all (unless I missed something) comments on the two PR's.
>
> Best
> Prateek
>
> On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <pr...@gmail.com> wrote:
>
>> Thanks Andrew and Micah.
>>
>> `fair amount of feedback on at least the implementations`
>> For the c++ I have already started addressing the feedback, I should be
>> done with that Monday/Tuesday.
>> I think Vinoo too has been making good progress on the Java implementation.
>>
>> Best
>> Prateek
>>
>> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <an...@gmail.com>
>> wrote:
>>
>>> Got it. Thank you for the clarification -- I will try and look into the
>>> spec and the Rust implementation[1] in this next week
>>>
>>> [1]: https://github.com/apache/arrow-rs/pull/9372
>>>
>>> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <em...@gmail.com>
>>> wrote:
>>>
>>>> Hi Andrew,
>>>> I think there is a fair amount of feedback on at least the
>>>> implementations, typically I think we've waited till they are close to
>>>> mergeable before a final vote. Otherwise I agree we are very close.
>>>>
>>>> -Micah
>>>>
>>>> On Saturday, April 25, 2026, Andrew Lamb <an...@gmail.com> wrote:
>>>>
>>>>> Thanks Prateek,
>>>>>
>>>>> I think from this content it looks to me like we are ready to start a
>>>>> vote to explicitly accept ALP into Parquet
>>>>>
>>>>> Does anyone know of a reason we should postpone it for longer?
>>>>> Perhaps someone needs some more time to review?
>>>>>
>>>>> Andrew
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <pr...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi team,
>>>>>>
>>>>>>
>>>>>>
>>>>>> Hope everyone is doing well. I got a chance to work through all the
>>>>>> remaining feedback and update the spec doc. Here are the new artifacts
>>>>>>
>>>>>> 1) Spec document :
>>>>>> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
>>>>>>
>>>>>> 2) Spec document in parquet format repo :
>>>>>> https://github.com/apache/parquet-format/pull/557
>>>>>>
>>>>>> 3) Alp implementation in arrow c++ repo :
>>>>>> https://github.com/apache/arrow/pull/48345/changes
>>>>>>
>>>>>> 4) Alp implementation in parquet-java repo : Work for Vinoo and Julien
>>>>>> https://github.com/apache/parquet-java/pull/3397
>>>>>>
>>>>>> 5) PR with test and benchmarking artifacts in parquet-testing repo :
>>>>>> https://github.com/apache/parquet-testing/pull/100
>>>>>>
>>>>>>
>>>>>> And
>>>>>>
>>>>>>
>>>>>> - Go : Arnav just submitted an in progress implementation in Go.
>>>>>> https://github.com/apache/arrow-go/pull/704 (I haven't started
>>>>>> looking at it yet)
>>>>>> - Rust : I remember Andrew mentioned that this work is also in
>>>>>> progress (So 4 languages!)
>>>>>>
>>>>>>
>>>>>> *Arrow C++ implementation *
>>>>>>
>>>>>>
>>>>>>
>>>>>> The PR is out and was also used by Antoine to report the numbers as
>>>>>> reported here. Micah and Konstantin have given 1 round of feedback
>>>>>> and I'm addressing them today. Please note that the default
>>>>>> optimization flag for compiling is O2 and not Q3. I got around 70%
>>>>>> performance improvement in the decoding speed when using the O3 flag.
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Parqet-MR Java implementation (working with Vinoo and Julien) and **Cross
>>>>>> Language testing*
>>>>>>
>>>>>>
>>>>>> Let me know if you have any questions or feedback.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Now pasting some performance numbers
>>>>>>
>>>>>>
>>>>>> Table 1: C++ ALP Double Decode — Spotify Columns (Graviton 3, ARM
>>>>>> Neoverse V1)
>>>>>>
>>>>>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
>>>>>>
>>>>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │ Speedup │
>>>>>>
>>>>>> ├──────────────────┼──────────────┼──────────────┼─────────┤
>>>>>>
>>>>>> │ valence │ 3,155 │ 5,523 │ 1.75x │
>>>>>>
>>>>>> │ danceability │ 3,233 │ 5,685 │ 1.76x │
>>>>>>
>>>>>> │ energy │ 3,197 │ 5,652 │ 1.77x │
>>>>>>
>>>>>> │ loudness │ 3,186 │ 5,473 │ 1.72x │
>>>>>>
>>>>>> └──────────────────┴──────────────┴──────────────┴─────────┘
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <pr...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> @Micah Kornfield <em...@gmail.com> : Got it.
>>>>>>>
>>>>>>> @Andrew Lamb <an...@gmail.com>
>>>>>>>
>>>>>>>
>>>>>>>> Do you think it would be good to start moving the spec development
>>>>>>>> into
>>>>>>>> markdown format, in preparation for finalizing it?
>>>>>>>>
>>>>>>>
>>>>>>> Yes I'll update the numbers for some of the examples I have in the
>>>>>>> spec based
>>>>>>> on the updated header size. Then we should be good to go for the
>>>>>>> markdown format.
>>>>>>>
>>>>>>> Thanks everyone!
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Andrew
>>>>>>>>
>>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <pr...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi team,
>>>>>>>>>
>>>>>>>>> 1) Andrew
>>>>>>>>>
>>>>>>>>> - Thanks for working on test files. My PR did add all the test
>>>>>>>> files I
>>>>>>>>> used to benchmark on datasets. Maybe we can club it together.
>>>>>>>> WIll also
>>>>>>>>> aid
>>>>>>>>> cross language testing
>>>>>>>>> - Kosta Tarasov working on Rust implementation. This is great.
>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2) Antoine
>>>>>>>>>
>>>>>>>>> - Thanks a lot for reporting the numbers on AMD. Looks like you
>>>>>>>> are
>>>>>>>>> getting 8X the decoding performance of BSS. This is amazing!!.
>>>>>>>>> - Thanks for acknowledging the sampling design.
>>>>>>>>> - I agree with you on Fastlanes. In some crude experiments I
>>>>>>>> didn't get
>>>>>>>>> a good perf benefit from it on Graviton3 (but maybe there was
>>>>>>>> something
>>>>>>>>> wrong with my implementation).
>>>>>>>>> - Locking the 16bit exception encoding for the spec in this
>>>>>>>> case.
>>>>>>>>> - Awesome I think we have solved for all open questions minus
>>>>>>>> the
>>>>>>>>> version byte :). (will get back on this soon)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 3) Micah
>>>>>>>>>
>>>>>>>>> - FastLanes : The current spec does allow for using FastLane
>>>>>>>> with the
>>>>>>>>> configurable enum value for layout. We should be able to inject
>>>>>>>> any
>>>>>>>>> layout
>>>>>>>>> in the current design.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Working on resolving all remaining open comments on the spec this
>>>>>>>> week.
>>>>>>>>>
>>>>>>>>> Best
>>>>>>>>> Prateek
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <
>>>>>>>> stevel@cloudera.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
>>>>>>>> emkornfield@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> It looks like the actual issue described for ORC in the paper
>>>>>>>> is that
>>>>>>>>> it
>>>>>>>>>>> has multiple sub-encodings in a batch. This is different then
>>>>>>>> the
>>>>>>>>> design
>>>>>>>>>>> proposed here where there is still fixed encoding per page in
>>>>>>>> parquet.
>>>>>>>>>>> Given reasonably sized pages I don't think branch
>>>>>>>> misprediction should
>>>>>>>>>> be a
>>>>>>>>>>> big issue for new encodings. I agree that we should be
>>>>>>>> conservative in
>>>>>>>>>>> general for adding new encodings.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> +1
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by PRATEEK GAUR <pr...@gmail.com>.
Thanks Andrew and Micah for review feedback on the two PR's
1) (c++ arrow repo) https://github.com/apache/arrow/pull/48345/changes
2) (parquet-format repo) https://github.com/apache/parquet-format/pull/557
I have addressed all (unless I missed something) comments on the two PR's.
Best
Prateek
On Sat, Apr 25, 2026 at 1:08 PM PRATEEK GAUR <pr...@gmail.com> wrote:
> Thanks Andrew and Micah.
>
> `fair amount of feedback on at least the implementations`
> For the c++ I have already started addressing the feedback, I should be
> done with that Monday/Tuesday.
> I think Vinoo too has been making good progress on the Java implementation.
>
> Best
> Prateek
>
> On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <an...@gmail.com>
> wrote:
>
>> Got it. Thank you for the clarification -- I will try and look into the
>> spec and the Rust implementation[1] in this next week
>>
>> [1]: https://github.com/apache/arrow-rs/pull/9372
>>
>> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <em...@gmail.com>
>> wrote:
>>
>>> Hi Andrew,
>>> I think there is a fair amount of feedback on at least the
>>> implementations, typically I think we've waited till they are close to
>>> mergeable before a final vote. Otherwise I agree we are very close.
>>>
>>> -Micah
>>>
>>> On Saturday, April 25, 2026, Andrew Lamb <an...@gmail.com> wrote:
>>>
>>>> Thanks Prateek,
>>>>
>>>> I think from this content it looks to me like we are ready to start a
>>>> vote to explicitly accept ALP into Parquet
>>>>
>>>> Does anyone know of a reason we should postpone it for longer?
>>>> Perhaps someone needs some more time to review?
>>>>
>>>> Andrew
>>>>
>>>>
>>>>
>>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <pr...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi team,
>>>>>
>>>>>
>>>>>
>>>>> Hope everyone is doing well. I got a chance to work through all the
>>>>> remaining feedback and update the spec doc. Here are the new artifacts
>>>>>
>>>>> 1) Spec document :
>>>>> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
>>>>>
>>>>> 2) Spec document in parquet format repo :
>>>>> https://github.com/apache/parquet-format/pull/557
>>>>>
>>>>> 3) Alp implementation in arrow c++ repo :
>>>>> https://github.com/apache/arrow/pull/48345/changes
>>>>>
>>>>> 4) Alp implementation in parquet-java repo : Work for Vinoo and Julien
>>>>> https://github.com/apache/parquet-java/pull/3397
>>>>>
>>>>> 5) PR with test and benchmarking artifacts in parquet-testing repo :
>>>>> https://github.com/apache/parquet-testing/pull/100
>>>>>
>>>>>
>>>>> And
>>>>>
>>>>>
>>>>> - Go : Arnav just submitted an in progress implementation in Go.
>>>>> https://github.com/apache/arrow-go/pull/704 (I haven't started
>>>>> looking at it yet)
>>>>> - Rust : I remember Andrew mentioned that this work is also in
>>>>> progress (So 4 languages!)
>>>>>
>>>>>
>>>>> *Arrow C++ implementation *
>>>>>
>>>>>
>>>>>
>>>>> The PR is out and was also used by Antoine to report the numbers as
>>>>> reported here. Micah and Konstantin have given 1 round of feedback
>>>>> and I'm addressing them today. Please note that the default
>>>>> optimization flag for compiling is O2 and not Q3. I got around 70%
>>>>> performance improvement in the decoding speed when using the O3 flag.
>>>>>
>>>>>
>>>>>
>>>>> *Parqet-MR Java implementation (working with Vinoo and Julien) and **Cross
>>>>> Language testing*
>>>>>
>>>>>
>>>>> Let me know if you have any questions or feedback.
>>>>>
>>>>>
>>>>>
>>>>> Now pasting some performance numbers
>>>>>
>>>>>
>>>>> Table 1: C++ ALP Double Decode — Spotify Columns (Graviton 3, ARM
>>>>> Neoverse V1)
>>>>>
>>>>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
>>>>>
>>>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │ Speedup │
>>>>>
>>>>> ├──────────────────┼──────────────┼──────────────┼─────────┤
>>>>>
>>>>> │ valence │ 3,155 │ 5,523 │ 1.75x │
>>>>>
>>>>> │ danceability │ 3,233 │ 5,685 │ 1.76x │
>>>>>
>>>>> │ energy │ 3,197 │ 5,652 │ 1.77x │
>>>>>
>>>>> │ loudness │ 3,186 │ 5,473 │ 1.72x │
>>>>>
>>>>> └──────────────────┴──────────────┴──────────────┴─────────┘
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <pr...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> @Micah Kornfield <em...@gmail.com> : Got it.
>>>>>>
>>>>>> @Andrew Lamb <an...@gmail.com>
>>>>>>
>>>>>>
>>>>>>> Do you think it would be good to start moving the spec development
>>>>>>> into
>>>>>>> markdown format, in preparation for finalizing it?
>>>>>>>
>>>>>>
>>>>>> Yes I'll update the numbers for some of the examples I have in the
>>>>>> spec based
>>>>>> on the updated header size. Then we should be good to go for the
>>>>>> markdown format.
>>>>>>
>>>>>> Thanks everyone!
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Andrew
>>>>>>>
>>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <pr...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> > Hi team,
>>>>>>> >
>>>>>>> > 1) Andrew
>>>>>>> >
>>>>>>> > - Thanks for working on test files. My PR did add all the test
>>>>>>> files I
>>>>>>> > used to benchmark on datasets. Maybe we can club it together.
>>>>>>> WIll also
>>>>>>> > aid
>>>>>>> > cross language testing
>>>>>>> > - Kosta Tarasov working on Rust implementation. This is great.
>>>>>>> Thanks
>>>>>>> >
>>>>>>> >
>>>>>>> > 2) Antoine
>>>>>>> >
>>>>>>> > - Thanks a lot for reporting the numbers on AMD. Looks like you
>>>>>>> are
>>>>>>> > getting 8X the decoding performance of BSS. This is amazing!!.
>>>>>>> > - Thanks for acknowledging the sampling design.
>>>>>>> > - I agree with you on Fastlanes. In some crude experiments I
>>>>>>> didn't get
>>>>>>> > a good perf benefit from it on Graviton3 (but maybe there was
>>>>>>> something
>>>>>>> > wrong with my implementation).
>>>>>>> > - Locking the 16bit exception encoding for the spec in this
>>>>>>> case.
>>>>>>> > - Awesome I think we have solved for all open questions minus
>>>>>>> the
>>>>>>> > version byte :). (will get back on this soon)
>>>>>>> >
>>>>>>> >
>>>>>>> > 3) Micah
>>>>>>> >
>>>>>>> > - FastLanes : The current spec does allow for using FastLane
>>>>>>> with the
>>>>>>> > configurable enum value for layout. We should be able to inject
>>>>>>> any
>>>>>>> > layout
>>>>>>> > in the current design.
>>>>>>> >
>>>>>>> >
>>>>>>> > Working on resolving all remaining open comments on the spec this
>>>>>>> week.
>>>>>>> >
>>>>>>> > Best
>>>>>>> > Prateek
>>>>>>> >
>>>>>>> >
>>>>>>> > On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <
>>>>>>> stevel@cloudera.com>
>>>>>>> > wrote:
>>>>>>> >
>>>>>>> > > On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
>>>>>>> emkornfield@gmail.com>
>>>>>>> > > wrote:
>>>>>>> > >
>>>>>>> > > >
>>>>>>> > > >
>>>>>>> > > > It looks like the actual issue described for ORC in the paper
>>>>>>> is that
>>>>>>> > it
>>>>>>> > > > has multiple sub-encodings in a batch. This is different then
>>>>>>> the
>>>>>>> > design
>>>>>>> > > > proposed here where there is still fixed encoding per page in
>>>>>>> parquet.
>>>>>>> > > > Given reasonably sized pages I don't think branch
>>>>>>> misprediction should
>>>>>>> > > be a
>>>>>>> > > > big issue for new encodings. I agree that we should be
>>>>>>> conservative in
>>>>>>> > > > general for adding new encodings.
>>>>>>> > > >
>>>>>>> > > >
>>>>>>> > > +1
>>>>>>> > >
>>>>>>> >
>>>>>>>
>>>>>>
Re: [Parquet] ALP Encoding for Floating point data
Posted by PRATEEK GAUR <pr...@gmail.com>.
Thanks Andrew and Micah.
`fair amount of feedback on at least the implementations`
For the c++ I have already started addressing the feedback, I should be
done with that Monday/Tuesday.
I think Vinoo too has been making good progress on the Java implementation.
Best
Prateek
On Sat, Apr 25, 2026 at 12:55 PM Andrew Lamb <an...@gmail.com> wrote:
> Got it. Thank you for the clarification -- I will try and look into the
> spec and the Rust implementation[1] in this next week
>
> [1]: https://github.com/apache/arrow-rs/pull/9372
>
> On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <em...@gmail.com>
> wrote:
>
>> Hi Andrew,
>> I think there is a fair amount of feedback on at least the
>> implementations, typically I think we've waited till they are close to
>> mergeable before a final vote. Otherwise I agree we are very close.
>>
>> -Micah
>>
>> On Saturday, April 25, 2026, Andrew Lamb <an...@gmail.com> wrote:
>>
>>> Thanks Prateek,
>>>
>>> I think from this content it looks to me like we are ready to start a
>>> vote to explicitly accept ALP into Parquet
>>>
>>> Does anyone know of a reason we should postpone it for longer?
>>> Perhaps someone needs some more time to review?
>>>
>>> Andrew
>>>
>>>
>>>
>>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <pr...@gmail.com> wrote:
>>>
>>>> Hi team,
>>>>
>>>>
>>>>
>>>> Hope everyone is doing well. I got a chance to work through all the
>>>> remaining feedback and update the spec doc. Here are the new artifacts
>>>>
>>>> 1) Spec document :
>>>> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
>>>>
>>>> 2) Spec document in parquet format repo :
>>>> https://github.com/apache/parquet-format/pull/557
>>>>
>>>> 3) Alp implementation in arrow c++ repo :
>>>> https://github.com/apache/arrow/pull/48345/changes
>>>>
>>>> 4) Alp implementation in parquet-java repo : Work for Vinoo and Julien
>>>> https://github.com/apache/parquet-java/pull/3397
>>>>
>>>> 5) PR with test and benchmarking artifacts in parquet-testing repo :
>>>> https://github.com/apache/parquet-testing/pull/100
>>>>
>>>>
>>>> And
>>>>
>>>>
>>>> - Go : Arnav just submitted an in progress implementation in Go.
>>>> https://github.com/apache/arrow-go/pull/704 (I haven't started
>>>> looking at it yet)
>>>> - Rust : I remember Andrew mentioned that this work is also in
>>>> progress (So 4 languages!)
>>>>
>>>>
>>>> *Arrow C++ implementation *
>>>>
>>>>
>>>>
>>>> The PR is out and was also used by Antoine to report the numbers as
>>>> reported here. Micah and Konstantin have given 1 round of feedback and
>>>> I'm addressing them today. Please note that the default optimization
>>>> flag for compiling is O2 and not Q3. I got around 70% performance
>>>> improvement in the decoding speed when using the O3 flag.
>>>>
>>>>
>>>>
>>>> *Parqet-MR Java implementation (working with Vinoo and Julien) and **Cross
>>>> Language testing*
>>>>
>>>>
>>>> Let me know if you have any questions or feedback.
>>>>
>>>>
>>>>
>>>> Now pasting some performance numbers
>>>>
>>>>
>>>> Table 1: C++ ALP Double Decode — Spotify Columns (Graviton 3, ARM
>>>> Neoverse V1)
>>>>
>>>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
>>>>
>>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │ Speedup │
>>>>
>>>> ├──────────────────┼──────────────┼──────────────┼─────────┤
>>>>
>>>> │ valence │ 3,155 │ 5,523 │ 1.75x │
>>>>
>>>> │ danceability │ 3,233 │ 5,685 │ 1.76x │
>>>>
>>>> │ energy │ 3,197 │ 5,652 │ 1.77x │
>>>>
>>>> │ loudness │ 3,186 │ 5,473 │ 1.72x │
>>>>
>>>> └──────────────────┴──────────────┴──────────────┴─────────┘
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <pr...@gmail.com>
>>>> wrote:
>>>>
>>>>> @Micah Kornfield <em...@gmail.com> : Got it.
>>>>>
>>>>> @Andrew Lamb <an...@gmail.com>
>>>>>
>>>>>
>>>>>> Do you think it would be good to start moving the spec development
>>>>>> into
>>>>>> markdown format, in preparation for finalizing it?
>>>>>>
>>>>>
>>>>> Yes I'll update the numbers for some of the examples I have in the
>>>>> spec based
>>>>> on the updated header size. Then we should be good to go for the
>>>>> markdown format.
>>>>>
>>>>> Thanks everyone!
>>>>>
>>>>>
>>>>>>
>>>>>> Andrew
>>>>>>
>>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <pr...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> > Hi team,
>>>>>> >
>>>>>> > 1) Andrew
>>>>>> >
>>>>>> > - Thanks for working on test files. My PR did add all the test
>>>>>> files I
>>>>>> > used to benchmark on datasets. Maybe we can club it together.
>>>>>> WIll also
>>>>>> > aid
>>>>>> > cross language testing
>>>>>> > - Kosta Tarasov working on Rust implementation. This is great.
>>>>>> Thanks
>>>>>> >
>>>>>> >
>>>>>> > 2) Antoine
>>>>>> >
>>>>>> > - Thanks a lot for reporting the numbers on AMD. Looks like you
>>>>>> are
>>>>>> > getting 8X the decoding performance of BSS. This is amazing!!.
>>>>>> > - Thanks for acknowledging the sampling design.
>>>>>> > - I agree with you on Fastlanes. In some crude experiments I
>>>>>> didn't get
>>>>>> > a good perf benefit from it on Graviton3 (but maybe there was
>>>>>> something
>>>>>> > wrong with my implementation).
>>>>>> > - Locking the 16bit exception encoding for the spec in this case.
>>>>>> > - Awesome I think we have solved for all open questions minus the
>>>>>> > version byte :). (will get back on this soon)
>>>>>> >
>>>>>> >
>>>>>> > 3) Micah
>>>>>> >
>>>>>> > - FastLanes : The current spec does allow for using FastLane
>>>>>> with the
>>>>>> > configurable enum value for layout. We should be able to inject
>>>>>> any
>>>>>> > layout
>>>>>> > in the current design.
>>>>>> >
>>>>>> >
>>>>>> > Working on resolving all remaining open comments on the spec this
>>>>>> week.
>>>>>> >
>>>>>> > Best
>>>>>> > Prateek
>>>>>> >
>>>>>> >
>>>>>> > On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <stevel@cloudera.com
>>>>>> >
>>>>>> > wrote:
>>>>>> >
>>>>>> > > On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
>>>>>> emkornfield@gmail.com>
>>>>>> > > wrote:
>>>>>> > >
>>>>>> > > >
>>>>>> > > >
>>>>>> > > > It looks like the actual issue described for ORC in the paper
>>>>>> is that
>>>>>> > it
>>>>>> > > > has multiple sub-encodings in a batch. This is different then
>>>>>> the
>>>>>> > design
>>>>>> > > > proposed here where there is still fixed encoding per page in
>>>>>> parquet.
>>>>>> > > > Given reasonably sized pages I don't think branch misprediction
>>>>>> should
>>>>>> > > be a
>>>>>> > > > big issue for new encodings. I agree that we should be
>>>>>> conservative in
>>>>>> > > > general for adding new encodings.
>>>>>> > > >
>>>>>> > > >
>>>>>> > > +1
>>>>>> > >
>>>>>> >
>>>>>>
>>>>>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Andrew Lamb <an...@gmail.com>.
Got it. Thank you for the clarification -- I will try and look into the
spec and the Rust implementation[1] in this next week
[1]: https://github.com/apache/arrow-rs/pull/9372
On Sat, Apr 25, 2026 at 12:01 PM Micah Kornfield <em...@gmail.com>
wrote:
> Hi Andrew,
> I think there is a fair amount of feedback on at least the
> implementations, typically I think we've waited till they are close to
> mergeable before a final vote. Otherwise I agree we are very close.
>
> -Micah
>
> On Saturday, April 25, 2026, Andrew Lamb <an...@gmail.com> wrote:
>
>> Thanks Prateek,
>>
>> I think from this content it looks to me like we are ready to start a
>> vote to explicitly accept ALP into Parquet
>>
>> Does anyone know of a reason we should postpone it for longer?
>> Perhaps someone needs some more time to review?
>>
>> Andrew
>>
>>
>>
>> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <pr...@gmail.com> wrote:
>>
>>> Hi team,
>>>
>>>
>>>
>>> Hope everyone is doing well. I got a chance to work through all the
>>> remaining feedback and update the spec doc. Here are the new artifacts
>>>
>>> 1) Spec document :
>>> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
>>>
>>> 2) Spec document in parquet format repo :
>>> https://github.com/apache/parquet-format/pull/557
>>>
>>> 3) Alp implementation in arrow c++ repo :
>>> https://github.com/apache/arrow/pull/48345/changes
>>>
>>> 4) Alp implementation in parquet-java repo : Work for Vinoo and Julien
>>> https://github.com/apache/parquet-java/pull/3397
>>>
>>> 5) PR with test and benchmarking artifacts in parquet-testing repo :
>>> https://github.com/apache/parquet-testing/pull/100
>>>
>>>
>>> And
>>>
>>>
>>> - Go : Arnav just submitted an in progress implementation in Go.
>>> https://github.com/apache/arrow-go/pull/704 (I haven't started
>>> looking at it yet)
>>> - Rust : I remember Andrew mentioned that this work is also in
>>> progress (So 4 languages!)
>>>
>>>
>>> *Arrow C++ implementation *
>>>
>>>
>>>
>>> The PR is out and was also used by Antoine to report the numbers as
>>> reported here. Micah and Konstantin have given 1 round of feedback and
>>> I'm addressing them today. Please note that the default optimization
>>> flag for compiling is O2 and not Q3. I got around 70% performance
>>> improvement in the decoding speed when using the O3 flag.
>>>
>>>
>>>
>>> *Parqet-MR Java implementation (working with Vinoo and Julien) and **Cross
>>> Language testing*
>>>
>>>
>>> Let me know if you have any questions or feedback.
>>>
>>>
>>>
>>> Now pasting some performance numbers
>>>
>>>
>>> Table 1: C++ ALP Double Decode — Spotify Columns (Graviton 3, ARM
>>> Neoverse V1)
>>>
>>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
>>>
>>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │ Speedup │
>>>
>>> ├──────────────────┼──────────────┼──────────────┼─────────┤
>>>
>>> │ valence │ 3,155 │ 5,523 │ 1.75x │
>>>
>>> │ danceability │ 3,233 │ 5,685 │ 1.76x │
>>>
>>> │ energy │ 3,197 │ 5,652 │ 1.77x │
>>>
>>> │ loudness │ 3,186 │ 5,473 │ 1.72x │
>>>
>>> └──────────────────┴──────────────┴──────────────┴─────────┘
>>>
>>>
>>>
>>>
>>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <pr...@gmail.com> wrote:
>>>
>>>> @Micah Kornfield <em...@gmail.com> : Got it.
>>>>
>>>> @Andrew Lamb <an...@gmail.com>
>>>>
>>>>
>>>>> Do you think it would be good to start moving the spec development into
>>>>> markdown format, in preparation for finalizing it?
>>>>>
>>>>
>>>> Yes I'll update the numbers for some of the examples I have in the spec
>>>> based
>>>> on the updated header size. Then we should be good to go for the
>>>> markdown format.
>>>>
>>>> Thanks everyone!
>>>>
>>>>
>>>>>
>>>>> Andrew
>>>>>
>>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <pr...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> > Hi team,
>>>>> >
>>>>> > 1) Andrew
>>>>> >
>>>>> > - Thanks for working on test files. My PR did add all the test
>>>>> files I
>>>>> > used to benchmark on datasets. Maybe we can club it together.
>>>>> WIll also
>>>>> > aid
>>>>> > cross language testing
>>>>> > - Kosta Tarasov working on Rust implementation. This is great.
>>>>> Thanks
>>>>> >
>>>>> >
>>>>> > 2) Antoine
>>>>> >
>>>>> > - Thanks a lot for reporting the numbers on AMD. Looks like you
>>>>> are
>>>>> > getting 8X the decoding performance of BSS. This is amazing!!.
>>>>> > - Thanks for acknowledging the sampling design.
>>>>> > - I agree with you on Fastlanes. In some crude experiments I
>>>>> didn't get
>>>>> > a good perf benefit from it on Graviton3 (but maybe there was
>>>>> something
>>>>> > wrong with my implementation).
>>>>> > - Locking the 16bit exception encoding for the spec in this case.
>>>>> > - Awesome I think we have solved for all open questions minus the
>>>>> > version byte :). (will get back on this soon)
>>>>> >
>>>>> >
>>>>> > 3) Micah
>>>>> >
>>>>> > - FastLanes : The current spec does allow for using FastLane with
>>>>> the
>>>>> > configurable enum value for layout. We should be able to inject
>>>>> any
>>>>> > layout
>>>>> > in the current design.
>>>>> >
>>>>> >
>>>>> > Working on resolving all remaining open comments on the spec this
>>>>> week.
>>>>> >
>>>>> > Best
>>>>> > Prateek
>>>>> >
>>>>> >
>>>>> > On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <st...@cloudera.com>
>>>>> > wrote:
>>>>> >
>>>>> > > On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <
>>>>> emkornfield@gmail.com>
>>>>> > > wrote:
>>>>> > >
>>>>> > > >
>>>>> > > >
>>>>> > > > It looks like the actual issue described for ORC in the paper is
>>>>> that
>>>>> > it
>>>>> > > > has multiple sub-encodings in a batch. This is different then
>>>>> the
>>>>> > design
>>>>> > > > proposed here where there is still fixed encoding per page in
>>>>> parquet.
>>>>> > > > Given reasonably sized pages I don't think branch misprediction
>>>>> should
>>>>> > > be a
>>>>> > > > big issue for new encodings. I agree that we should be
>>>>> conservative in
>>>>> > > > general for adding new encodings.
>>>>> > > >
>>>>> > > >
>>>>> > > +1
>>>>> > >
>>>>> >
>>>>>
>>>>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Micah Kornfield <em...@gmail.com>.
Hi Andrew,
I think there is a fair amount of feedback on at least the implementations,
typically I think we've waited till they are close to mergeable before a
final vote. Otherwise I agree we are very close.
-Micah
On Saturday, April 25, 2026, Andrew Lamb <an...@gmail.com> wrote:
> Thanks Prateek,
>
> I think from this content it looks to me like we are ready to start a vote
> to explicitly accept ALP into Parquet
>
> Does anyone know of a reason we should postpone it for longer?
> Perhaps someone needs some more time to review?
>
> Andrew
>
>
>
> On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <pr...@gmail.com> wrote:
>
>> Hi team,
>>
>>
>>
>> Hope everyone is doing well. I got a chance to work through all the
>> remaining feedback and update the spec doc. Here are the new artifacts
>>
>> 1) Spec document : https://docs.google.com/document/d/
>> 1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
>>
>> 2) Spec document in parquet format repo : https://github.com/apache/
>> parquet-format/pull/557
>>
>> 3) Alp implementation in arrow c++ repo : https://github.com/apache/
>> arrow/pull/48345/changes
>>
>> 4) Alp implementation in parquet-java repo : Work for Vinoo and Julien
>> https://github.com/apache/parquet-java/pull/3397
>>
>> 5) PR with test and benchmarking artifacts in parquet-testing repo :
>> https://github.com/apache/parquet-testing/pull/100
>>
>>
>> And
>>
>>
>> - Go : Arnav just submitted an in progress implementation in Go.
>> https://github.com/apache/arrow-go/pull/704
>> <https://github.com/apache/arrow-go/pull/704> (I haven't started
>> looking at it yet)
>> - Rust : I remember Andrew mentioned that this work is also in
>> progress (So 4 languages!)
>>
>>
>> *Arrow C++ implementation *
>>
>>
>>
>> The PR is out and was also used by Antoine to report the numbers as
>> reported here. Micah and Konstantin have given 1 round of feedback and
>> I'm addressing them today. Please note that the default optimization
>> flag for compiling is O2 and not Q3. I got around 70% performance
>> improvement in the decoding speed when using the O3 flag.
>>
>>
>>
>> *Parqet-MR Java implementation (working with Vinoo and Julien) and **Cross
>> Language testing*
>>
>>
>> Let me know if you have any questions or feedback.
>>
>>
>>
>> Now pasting some performance numbers
>>
>>
>> Table 1: C++ ALP Double Decode — Spotify Columns (Graviton 3, ARM
>> Neoverse V1)
>>
>> ┌──────────────────┬──────────────┬──────────────┬─────────┐
>>
>> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │ Speedup │
>>
>> ├──────────────────┼──────────────┼──────────────┼─────────┤
>>
>> │ valence │ 3,155 │ 5,523 │ 1.75x │
>>
>> │ danceability │ 3,233 │ 5,685 │ 1.76x │
>>
>> │ energy │ 3,197 │ 5,652 │ 1.77x │
>>
>> │ loudness │ 3,186 │ 5,473 │ 1.72x │
>>
>> └──────────────────┴──────────────┴──────────────┴─────────┘
>>
>>
>>
>>
>> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <pr...@gmail.com> wrote:
>>
>>> @Micah Kornfield <em...@gmail.com> : Got it.
>>>
>>> @Andrew Lamb <an...@gmail.com>
>>>
>>>
>>>> Do you think it would be good to start moving the spec development into
>>>> markdown format, in preparation for finalizing it?
>>>>
>>>
>>> Yes I'll update the numbers for some of the examples I have in the spec
>>> based
>>> on the updated header size. Then we should be good to go for the
>>> markdown format.
>>>
>>> Thanks everyone!
>>>
>>>
>>>>
>>>> Andrew
>>>>
>>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <pr...@gmail.com>
>>>> wrote:
>>>>
>>>> > Hi team,
>>>> >
>>>> > 1) Andrew
>>>> >
>>>> > - Thanks for working on test files. My PR did add all the test
>>>> files I
>>>> > used to benchmark on datasets. Maybe we can club it together. WIll
>>>> also
>>>> > aid
>>>> > cross language testing
>>>> > - Kosta Tarasov working on Rust implementation. This is great.
>>>> Thanks
>>>> >
>>>> >
>>>> > 2) Antoine
>>>> >
>>>> > - Thanks a lot for reporting the numbers on AMD. Looks like you are
>>>> > getting 8X the decoding performance of BSS. This is amazing!!.
>>>> > - Thanks for acknowledging the sampling design.
>>>> > - I agree with you on Fastlanes. In some crude experiments I
>>>> didn't get
>>>> > a good perf benefit from it on Graviton3 (but maybe there was
>>>> something
>>>> > wrong with my implementation).
>>>> > - Locking the 16bit exception encoding for the spec in this case.
>>>> > - Awesome I think we have solved for all open questions minus the
>>>> > version byte :). (will get back on this soon)
>>>> >
>>>> >
>>>> > 3) Micah
>>>> >
>>>> > - FastLanes : The current spec does allow for using FastLane with
>>>> the
>>>> > configurable enum value for layout. We should be able to inject any
>>>> > layout
>>>> > in the current design.
>>>> >
>>>> >
>>>> > Working on resolving all remaining open comments on the spec this
>>>> week.
>>>> >
>>>> > Best
>>>> > Prateek
>>>> >
>>>> >
>>>> > On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <st...@cloudera.com>
>>>> > wrote:
>>>> >
>>>> > > On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <emkornfield@gmail.com
>>>> >
>>>> > > wrote:
>>>> > >
>>>> > > >
>>>> > > >
>>>> > > > It looks like the actual issue described for ORC in the paper is
>>>> that
>>>> > it
>>>> > > > has multiple sub-encodings in a batch. This is different then the
>>>> > design
>>>> > > > proposed here where there is still fixed encoding per page in
>>>> parquet.
>>>> > > > Given reasonably sized pages I don't think branch misprediction
>>>> should
>>>> > > be a
>>>> > > > big issue for new encodings. I agree that we should be
>>>> conservative in
>>>> > > > general for adding new encodings.
>>>> > > >
>>>> > > >
>>>> > > +1
>>>> > >
>>>> >
>>>>
>>>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Andrew Lamb <an...@gmail.com>.
Thanks Prateek,
I think from this content it looks to me like we are ready to start a vote
to explicitly accept ALP into Parquet
Does anyone know of a reason we should postpone it for longer?
Perhaps someone needs some more time to review?
Andrew
On Wed, Apr 22, 2026 at 1:00 PM PRATEEK GAUR <pr...@gmail.com> wrote:
> Hi team,
>
>
>
> Hope everyone is doing well. I got a chance to work through all the
> remaining feedback and update the spec doc. Here are the new artifacts
>
> 1) Spec document :
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
>
> 2) Spec document in parquet format repo :
> https://github.com/apache/parquet-format/pull/557
>
> 3) Alp implementation in arrow c++ repo :
> https://github.com/apache/arrow/pull/48345/changes
>
> 4) Alp implementation in parquet-java repo : Work for Vinoo and Julien
> https://github.com/apache/parquet-java/pull/3397
>
> 5) PR with test and benchmarking artifacts in parquet-testing repo :
> https://github.com/apache/parquet-testing/pull/100
>
>
> And
>
>
> - Go : Arnav just submitted an in progress implementation in Go.
> https://github.com/apache/arrow-go/pull/704 (I haven't started looking
> at it yet)
> - Rust : I remember Andrew mentioned that this work is also in
> progress (So 4 languages!)
>
>
> *Arrow C++ implementation *
>
>
>
> The PR is out and was also used by Antoine to report the numbers as
> reported here. Micah and Konstantin have given 1 round of feedback and
> I'm addressing them today. Please note that the default optimization flag
> for compiling is O2 and not Q3. I got around 70% performance improvement in
> the decoding speed when using the O3 flag.
>
>
>
> *Parqet-MR Java implementation (working with Vinoo and Julien) and **Cross
> Language testing*
>
>
> Let me know if you have any questions or feedback.
>
>
>
> Now pasting some performance numbers
>
>
> Table 1: C++ ALP Double Decode — Spotify Columns (Graviton 3, ARM
> Neoverse V1)
>
> ┌──────────────────┬──────────────┬──────────────┬─────────┐
>
> │ Column │ -O2 (MB/s) │ -O3 (MB/s) │ Speedup │
>
> ├──────────────────┼──────────────┼──────────────┼─────────┤
>
> │ valence │ 3,155 │ 5,523 │ 1.75x │
>
> │ danceability │ 3,233 │ 5,685 │ 1.76x │
>
> │ energy │ 3,197 │ 5,652 │ 1.77x │
>
> │ loudness │ 3,186 │ 5,473 │ 1.72x │
>
> └──────────────────┴──────────────┴──────────────┴─────────┘
>
>
>
>
> On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <pr...@gmail.com> wrote:
>
>> @Micah Kornfield <em...@gmail.com> : Got it.
>>
>> @Andrew Lamb <an...@gmail.com>
>>
>>
>>> Do you think it would be good to start moving the spec development into
>>> markdown format, in preparation for finalizing it?
>>>
>>
>> Yes I'll update the numbers for some of the examples I have in the spec
>> based
>> on the updated header size. Then we should be good to go for the markdown
>> format.
>>
>> Thanks everyone!
>>
>>
>>>
>>> Andrew
>>>
>>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <pr...@gmail.com> wrote:
>>>
>>> > Hi team,
>>> >
>>> > 1) Andrew
>>> >
>>> > - Thanks for working on test files. My PR did add all the test
>>> files I
>>> > used to benchmark on datasets. Maybe we can club it together. WIll
>>> also
>>> > aid
>>> > cross language testing
>>> > - Kosta Tarasov working on Rust implementation. This is great.
>>> Thanks
>>> >
>>> >
>>> > 2) Antoine
>>> >
>>> > - Thanks a lot for reporting the numbers on AMD. Looks like you are
>>> > getting 8X the decoding performance of BSS. This is amazing!!.
>>> > - Thanks for acknowledging the sampling design.
>>> > - I agree with you on Fastlanes. In some crude experiments I didn't
>>> get
>>> > a good perf benefit from it on Graviton3 (but maybe there was
>>> something
>>> > wrong with my implementation).
>>> > - Locking the 16bit exception encoding for the spec in this case.
>>> > - Awesome I think we have solved for all open questions minus the
>>> > version byte :). (will get back on this soon)
>>> >
>>> >
>>> > 3) Micah
>>> >
>>> > - FastLanes : The current spec does allow for using FastLane with
>>> the
>>> > configurable enum value for layout. We should be able to inject any
>>> > layout
>>> > in the current design.
>>> >
>>> >
>>> > Working on resolving all remaining open comments on the spec this week.
>>> >
>>> > Best
>>> > Prateek
>>> >
>>> >
>>> > On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <st...@cloudera.com>
>>> > wrote:
>>> >
>>> > > On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <em...@gmail.com>
>>> > > wrote:
>>> > >
>>> > > >
>>> > > >
>>> > > > It looks like the actual issue described for ORC in the paper is
>>> that
>>> > it
>>> > > > has multiple sub-encodings in a batch. This is different then the
>>> > design
>>> > > > proposed here where there is still fixed encoding per page in
>>> parquet.
>>> > > > Given reasonably sized pages I don't think branch misprediction
>>> should
>>> > > be a
>>> > > > big issue for new encodings. I agree that we should be
>>> conservative in
>>> > > > general for adding new encodings.
>>> > > >
>>> > > >
>>> > > +1
>>> > >
>>> >
>>>
>>
Re: [Parquet] ALP Encoding for Floating point data
Posted by PRATEEK GAUR <pr...@gmail.com>.
Hi team,
Hope everyone is doing well. I got a chance to work through all the
remaining feedback and update the spec doc. Here are the new artifacts
1) Spec document :
https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit
2) Spec document in parquet format repo :
https://github.com/apache/parquet-format/pull/557
3) Alp implementation in arrow c++ repo :
https://github.com/apache/arrow/pull/48345/changes
4) Alp implementation in parquet-java repo : Work for Vinoo and Julien
https://github.com/apache/parquet-java/pull/3397
5) PR with test and benchmarking artifacts in parquet-testing repo :
https://github.com/apache/parquet-testing/pull/100
And
- Go : Arnav just submitted an in progress implementation in Go.
https://github.com/apache/arrow-go/pull/704 (I haven't started looking
at it yet)
- Rust : I remember Andrew mentioned that this work is also in progress
(So 4 languages!)
*Arrow C++ implementation *
The PR is out and was also used by Antoine to report the numbers as
reported here. Micah and Konstantin have given 1 round of feedback and I'm
addressing them today. Please note that the default optimization flag for
compiling is O2 and not Q3. I got around 70% performance improvement in the
decoding speed when using the O3 flag.
*Parqet-MR Java implementation (working with Vinoo and Julien) and **Cross
Language testing*
Let me know if you have any questions or feedback.
Now pasting some performance numbers
Table 1: C++ ALP Double Decode — Spotify Columns (Graviton 3, ARM
Neoverse V1)
┌──────────────────┬──────────────┬──────────────┬─────────┐
│ Column │ -O2 (MB/s) │ -O3 (MB/s) │ Speedup │
├──────────────────┼──────────────┼──────────────┼─────────┤
│ valence │ 3,155 │ 5,523 │ 1.75x │
│ danceability │ 3,233 │ 5,685 │ 1.76x │
│ energy │ 3,197 │ 5,652 │ 1.77x │
│ loudness │ 3,186 │ 5,473 │ 1.72x │
└──────────────────┴──────────────┴──────────────┴─────────┘
On Wed, Feb 25, 2026 at 9:49 AM PRATEEK GAUR <pr...@gmail.com> wrote:
> @Micah Kornfield <em...@gmail.com> : Got it.
>
> @Andrew Lamb <an...@gmail.com>
>
>
>> Do you think it would be good to start moving the spec development into
>> markdown format, in preparation for finalizing it?
>>
>
> Yes I'll update the numbers for some of the examples I have in the spec
> based
> on the updated header size. Then we should be good to go for the markdown
> format.
>
> Thanks everyone!
>
>
>>
>> Andrew
>>
>> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <pr...@gmail.com> wrote:
>>
>> > Hi team,
>> >
>> > 1) Andrew
>> >
>> > - Thanks for working on test files. My PR did add all the test files
>> I
>> > used to benchmark on datasets. Maybe we can club it together. WIll
>> also
>> > aid
>> > cross language testing
>> > - Kosta Tarasov working on Rust implementation. This is great.
>> Thanks
>> >
>> >
>> > 2) Antoine
>> >
>> > - Thanks a lot for reporting the numbers on AMD. Looks like you are
>> > getting 8X the decoding performance of BSS. This is amazing!!.
>> > - Thanks for acknowledging the sampling design.
>> > - I agree with you on Fastlanes. In some crude experiments I didn't
>> get
>> > a good perf benefit from it on Graviton3 (but maybe there was
>> something
>> > wrong with my implementation).
>> > - Locking the 16bit exception encoding for the spec in this case.
>> > - Awesome I think we have solved for all open questions minus the
>> > version byte :). (will get back on this soon)
>> >
>> >
>> > 3) Micah
>> >
>> > - FastLanes : The current spec does allow for using FastLane with the
>> > configurable enum value for layout. We should be able to inject any
>> > layout
>> > in the current design.
>> >
>> >
>> > Working on resolving all remaining open comments on the spec this week.
>> >
>> > Best
>> > Prateek
>> >
>> >
>> > On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <st...@cloudera.com>
>> > wrote:
>> >
>> > > On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <em...@gmail.com>
>> > > wrote:
>> > >
>> > > >
>> > > >
>> > > > It looks like the actual issue described for ORC in the paper is
>> that
>> > it
>> > > > has multiple sub-encodings in a batch. This is different then the
>> > design
>> > > > proposed here where there is still fixed encoding per page in
>> parquet.
>> > > > Given reasonably sized pages I don't think branch misprediction
>> should
>> > > be a
>> > > > big issue for new encodings. I agree that we should be
>> conservative in
>> > > > general for adding new encodings.
>> > > >
>> > > >
>> > > +1
>> > >
>> >
>>
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Arnav Balyan <ar...@gmail.com>.
Hi Prateek,
Thank you so much for the work on ALP, the spec is well written. I added
support for ALP in Parquet Go, (initial PR
<https://github.com/apache/arrow-go/pull/704>), I'll stay on a lookout for
any comments.
Looking forward to the successful integration of ALP in Parquet.
Warm Regards,
Arnav
On Wed, Feb 25, 2026 at 11:27 PM PRATEEK GAUR <pr...@gmail.com> wrote:
> @Micah Kornfield <em...@gmail.com> : Got it.
>
> @Andrew Lamb <an...@gmail.com>
>
>
> > Do you think it would be good to start moving the spec development into
> > markdown format, in preparation for finalizing it?
> >
>
> Yes I'll update the numbers for some of the examples I have in the spec
> based
> on the updated header size. Then we should be good to go for the markdown
> format.
>
> Thanks everyone!
>
>
> >
> > Andrew
> >
> > On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <pr...@gmail.com> wrote:
> >
> > > Hi team,
> > >
> > > 1) Andrew
> > >
> > > - Thanks for working on test files. My PR did add all the test
> files I
> > > used to benchmark on datasets. Maybe we can club it together. WIll
> > also
> > > aid
> > > cross language testing
> > > - Kosta Tarasov working on Rust implementation. This is great.
> Thanks
> > >
> > >
> > > 2) Antoine
> > >
> > > - Thanks a lot for reporting the numbers on AMD. Looks like you are
> > > getting 8X the decoding performance of BSS. This is amazing!!.
> > > - Thanks for acknowledging the sampling design.
> > > - I agree with you on Fastlanes. In some crude experiments I didn't
> > get
> > > a good perf benefit from it on Graviton3 (but maybe there was
> > something
> > > wrong with my implementation).
> > > - Locking the 16bit exception encoding for the spec in this case.
> > > - Awesome I think we have solved for all open questions minus the
> > > version byte :). (will get back on this soon)
> > >
> > >
> > > 3) Micah
> > >
> > > - FastLanes : The current spec does allow for using FastLane with
> the
> > > configurable enum value for layout. We should be able to inject any
> > > layout
> > > in the current design.
> > >
> > >
> > > Working on resolving all remaining open comments on the spec this week.
> > >
> > > Best
> > > Prateek
> > >
> > >
> > > On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <st...@cloudera.com>
> > > wrote:
> > >
> > > > On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <em...@gmail.com>
> > > > wrote:
> > > >
> > > > >
> > > > >
> > > > > It looks like the actual issue described for ORC in the paper is
> that
> > > it
> > > > > has multiple sub-encodings in a batch. This is different then the
> > > design
> > > > > proposed here where there is still fixed encoding per page in
> > parquet.
> > > > > Given reasonably sized pages I don't think branch misprediction
> > should
> > > > be a
> > > > > big issue for new encodings. I agree that we should be
> conservative
> > in
> > > > > general for adding new encodings.
> > > > >
> > > > >
> > > > +1
> > > >
> > >
> >
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by PRATEEK GAUR <pr...@gmail.com>.
@Micah Kornfield <em...@gmail.com> : Got it.
@Andrew Lamb <an...@gmail.com>
> Do you think it would be good to start moving the spec development into
> markdown format, in preparation for finalizing it?
>
Yes I'll update the numbers for some of the examples I have in the spec
based
on the updated header size. Then we should be good to go for the markdown
format.
Thanks everyone!
>
> Andrew
>
> On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <pr...@gmail.com> wrote:
>
> > Hi team,
> >
> > 1) Andrew
> >
> > - Thanks for working on test files. My PR did add all the test files I
> > used to benchmark on datasets. Maybe we can club it together. WIll
> also
> > aid
> > cross language testing
> > - Kosta Tarasov working on Rust implementation. This is great. Thanks
> >
> >
> > 2) Antoine
> >
> > - Thanks a lot for reporting the numbers on AMD. Looks like you are
> > getting 8X the decoding performance of BSS. This is amazing!!.
> > - Thanks for acknowledging the sampling design.
> > - I agree with you on Fastlanes. In some crude experiments I didn't
> get
> > a good perf benefit from it on Graviton3 (but maybe there was
> something
> > wrong with my implementation).
> > - Locking the 16bit exception encoding for the spec in this case.
> > - Awesome I think we have solved for all open questions minus the
> > version byte :). (will get back on this soon)
> >
> >
> > 3) Micah
> >
> > - FastLanes : The current spec does allow for using FastLane with the
> > configurable enum value for layout. We should be able to inject any
> > layout
> > in the current design.
> >
> >
> > Working on resolving all remaining open comments on the spec this week.
> >
> > Best
> > Prateek
> >
> >
> > On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <st...@cloudera.com>
> > wrote:
> >
> > > On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <em...@gmail.com>
> > > wrote:
> > >
> > > >
> > > >
> > > > It looks like the actual issue described for ORC in the paper is that
> > it
> > > > has multiple sub-encodings in a batch. This is different then the
> > design
> > > > proposed here where there is still fixed encoding per page in
> parquet.
> > > > Given reasonably sized pages I don't think branch misprediction
> should
> > > be a
> > > > big issue for new encodings. I agree that we should be conservative
> in
> > > > general for adding new encodings.
> > > >
> > > >
> > > +1
> > >
> >
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Andrew Lamb <an...@gmail.com>.
Thanks Prateek,
Do you think it would be good to start moving the spec development into
markdown format, in preparation for finalizing it?
Andrew
On Tue, Feb 17, 2026 at 7:28 PM PRATEEK GAUR <pr...@gmail.com> wrote:
> Hi team,
>
> 1) Andrew
>
> - Thanks for working on test files. My PR did add all the test files I
> used to benchmark on datasets. Maybe we can club it together. WIll also
> aid
> cross language testing
> - Kosta Tarasov working on Rust implementation. This is great. Thanks
>
>
> 2) Antoine
>
> - Thanks a lot for reporting the numbers on AMD. Looks like you are
> getting 8X the decoding performance of BSS. This is amazing!!.
> - Thanks for acknowledging the sampling design.
> - I agree with you on Fastlanes. In some crude experiments I didn't get
> a good perf benefit from it on Graviton3 (but maybe there was something
> wrong with my implementation).
> - Locking the 16bit exception encoding for the spec in this case.
> - Awesome I think we have solved for all open questions minus the
> version byte :). (will get back on this soon)
>
>
> 3) Micah
>
> - FastLanes : The current spec does allow for using FastLane with the
> configurable enum value for layout. We should be able to inject any
> layout
> in the current design.
>
>
> Working on resolving all remaining open comments on the spec this week.
>
> Best
> Prateek
>
>
> On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <st...@cloudera.com>
> wrote:
>
> > On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <em...@gmail.com>
> > wrote:
> >
> > >
> > >
> > > It looks like the actual issue described for ORC in the paper is that
> it
> > > has multiple sub-encodings in a batch. This is different then the
> design
> > > proposed here where there is still fixed encoding per page in parquet.
> > > Given reasonably sized pages I don't think branch misprediction should
> > be a
> > > big issue for new encodings. I agree that we should be conservative in
> > > general for adding new encodings.
> > >
> > >
> > +1
> >
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by PRATEEK GAUR <pr...@gmail.com>.
Hi team,
1) Andrew
- Thanks for working on test files. My PR did add all the test files I
used to benchmark on datasets. Maybe we can club it together. WIll also aid
cross language testing
- Kosta Tarasov working on Rust implementation. This is great. Thanks
2) Antoine
- Thanks a lot for reporting the numbers on AMD. Looks like you are
getting 8X the decoding performance of BSS. This is amazing!!.
- Thanks for acknowledging the sampling design.
- I agree with you on Fastlanes. In some crude experiments I didn't get
a good perf benefit from it on Graviton3 (but maybe there was something
wrong with my implementation).
- Locking the 16bit exception encoding for the spec in this case.
- Awesome I think we have solved for all open questions minus the
version byte :). (will get back on this soon)
3) Micah
- FastLanes : The current spec does allow for using FastLane with the
configurable enum value for layout. We should be able to inject any layout
in the current design.
Working on resolving all remaining open comments on the spec this week.
Best
Prateek
On Tue, Feb 10, 2026 at 3:37 AM Steve Loughran <st...@cloudera.com> wrote:
> On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <em...@gmail.com>
> wrote:
>
> >
> >
> > It looks like the actual issue described for ORC in the paper is that it
> > has multiple sub-encodings in a batch. This is different then the design
> > proposed here where there is still fixed encoding per page in parquet.
> > Given reasonably sized pages I don't think branch misprediction should
> be a
> > big issue for new encodings. I agree that we should be conservative in
> > general for adding new encodings.
> >
> >
> +1
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Steve Loughran <st...@cloudera.com>.
On Sun, 8 Feb 2026 at 18:12, Micah Kornfield <em...@gmail.com> wrote:
>
>
> It looks like the actual issue described for ORC in the paper is that it
> has multiple sub-encodings in a batch. This is different then the design
> proposed here where there is still fixed encoding per page in parquet.
> Given reasonably sized pages I don't think branch misprediction should be a
> big issue for new encodings. I agree that we should be conservative in
> general for adding new encodings.
>
>
+1
Re: [Parquet] ALP Encoding for Floating point data
Posted by Micah Kornfield <em...@gmail.com>.
>
> > Re-stating the points so that scrolling is not needed.
> > 1. Change of integer encoding (see debate in this thread on FOR vs
> > Delta). We also want to get fast lanes in at some point.
I'm not sure why we "want to get last lanes in some point". I don't
> think we want to obsess about performance here, given that current
> performance is already very good, and real-world bottlenecks will
> probably be elsewhere.
I think we can maybe debate this further, if/when someone proposes fast
lanes. My main point here is that there are valid use-cases for making
integer encoding configurable (or at least there is uncertainty here).
This highlights that *unless you can use cpu vector opcodes, adding more
> options can hurt branch prediction and so make overall performance worse*.
> It's a good argument for simplicity in compression and encoding choices.
It looks like the actual issue described for ORC in the paper is that it
has multiple sub-encodings in a batch. This is different then the design
proposed here where there is still fixed encoding per page in parquet.
Given reasonably sized pages I don't think branch misprediction should be a
big issue for new encodings. I agree that we should be conservative in
general for adding new encodings.
Regards,
Micah
On Thu, Feb 5, 2026 at 6:28 AM Antoine Pitrou <an...@python.org> wrote:
>
> hi Prateek,
>
> Le 03/02/2026 à 23:39, PRATEEK GAUR a écrit :
> > Hi Antoine and Micah,
> >
> > Apologies for getting back on this a little late.
> >
> > *Running Perf tests*
> > @Antoine Pitrou <an...@python.org> were you able to figure out the
> steps
> > to run the tests?
>
> Yes, I finally did that, results below on an AMD Zen 2 CPU:
> https://gist.github.com/pitrou/1f4aefb7034657ce018231d87993f437
>
> > *Sampling Frequency*
> > We want to pick the right parameters to encode the values with. That is
> > what the Spec requires.
> > From the implementation perspective you raise a good point that did
> cross my
> > mind that 'practically we don't want to sample for every page', for
> > performance
> > reasons. My thinking is each engine is free to decide this.
> > 1) Do it at page level if data is changing often
> > 2) Provide fixed presets via config
> > 3) Do it once per encoder (per column, as Micah pointed out)
> > 4) Provide a fancy config.
>
> Ok, that would sound fine to me.
>
> > Re-stating the points so that scrolling is not needed.
> > 1. Change of integer encoding (see debate in this thread on FOR vs
> > Delta). We also want to get fast lanes in at some point.
>
> I'm not sure why we "want to get last lanes in some point". I don't
> think we want to obsess about performance here, given that current
> performance is already very good, and real-world bottlenecks will
> probably be elsewhere.
>
> > For eg at this point I do see that both bitpacking of exceptions, as
> > pointed by
> > Antoine, or plain ub2 encoding should work equally well .
>
> Well, since 16 bits are actually enough for the current vector size, I'd
> say we can keep things simple.
>
> Regards
>
> Antoine.
>
>
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Micah Kornfield <em...@gmail.com>.
>
> > Re-stating the points so that scrolling is not needed.
> > 1. Change of integer encoding (see debate in this thread on FOR vs
> > Delta). We also want to get fast lanes in at some point.
I'm not sure why we "want to get last lanes in some point". I don't
> think we want to obsess about performance here, given that current
> performance is already very good, and real-world bottlenecks will
> probably be elsewhere.
I think we can maybe debate this further, if/when someone proposes fast
lanes. My main point here is that there are valid use-cases for making
integer encoding configurable (or at least there is uncertainty here).
This highlights that *unless you can use cpu vector opcodes, adding more
> options can hurt branch prediction and so make overall performance worse*.
> It's a good argument for simplicity in compression and encoding choices.
It looks like the actual issue described for ORC in the paper is that it
has multiple sub-encodings in a batch. This is different then the design
proposed here where there is still fixed encoding per page in parquet.
Given reasonably sized pages I don't think branch misprediction should be a
big issue for new encodings. I agree that we should be conservative in
general for adding new encodings.
Regards,
Micah
On Thu, Feb 5, 2026 at 6:28 AM Antoine Pitrou <an...@python.org> wrote:
>
> hi Prateek,
>
> Le 03/02/2026 à 23:39, PRATEEK GAUR a écrit :
> > Hi Antoine and Micah,
> >
> > Apologies for getting back on this a little late.
> >
> > *Running Perf tests*
> > @Antoine Pitrou <an...@python.org> were you able to figure out the
> steps
> > to run the tests?
>
> Yes, I finally did that, results below on an AMD Zen 2 CPU:
> https://gist.github.com/pitrou/1f4aefb7034657ce018231d87993f437
>
> > *Sampling Frequency*
> > We want to pick the right parameters to encode the values with. That is
> > what the Spec requires.
> > From the implementation perspective you raise a good point that did
> cross my
> > mind that 'practically we don't want to sample for every page', for
> > performance
> > reasons. My thinking is each engine is free to decide this.
> > 1) Do it at page level if data is changing often
> > 2) Provide fixed presets via config
> > 3) Do it once per encoder (per column, as Micah pointed out)
> > 4) Provide a fancy config.
>
> Ok, that would sound fine to me.
>
> > Re-stating the points so that scrolling is not needed.
> > 1. Change of integer encoding (see debate in this thread on FOR vs
> > Delta). We also want to get fast lanes in at some point.
>
> I'm not sure why we "want to get last lanes in some point". I don't
> think we want to obsess about performance here, given that current
> performance is already very good, and real-world bottlenecks will
> probably be elsewhere.
>
> > For eg at this point I do see that both bitpacking of exceptions, as
> > pointed by
> > Antoine, or plain ub2 encoding should work equally well .
>
> Well, since 16 bits are actually enough for the current vector size, I'd
> say we can keep things simple.
>
> Regards
>
> Antoine.
>
>
>
Re: [Parquet] ALP Encoding for Floating point data
Posted by Antoine Pitrou <an...@python.org>.
hi Prateek,
Le 03/02/2026 à 23:39, PRATEEK GAUR a écrit :
> Hi Antoine and Micah,
>
> Apologies for getting back on this a little late.
>
> *Running Perf tests*
> @Antoine Pitrou <an...@python.org> were you able to figure out the steps
> to run the tests?
Yes, I finally did that, results below on an AMD Zen 2 CPU:
https://gist.github.com/pitrou/1f4aefb7034657ce018231d87993f437
> *Sampling Frequency*
> We want to pick the right parameters to encode the values with. That is
> what the Spec requires.
> From the implementation perspective you raise a good point that did cross my
> mind that 'practically we don't want to sample for every page', for
> performance
> reasons. My thinking is each engine is free to decide this.
> 1) Do it at page level if data is changing often
> 2) Provide fixed presets via config
> 3) Do it once per encoder (per column, as Micah pointed out)
> 4) Provide a fancy config.
Ok, that would sound fine to me.
> Re-stating the points so that scrolling is not needed.
> 1. Change of integer encoding (see debate in this thread on FOR vs
> Delta). We also want to get fast lanes in at some point.
I'm not sure why we "want to get last lanes in some point". I don't
think we want to obsess about performance here, given that current
performance is already very good, and real-world bottlenecks will
probably be elsewhere.
> For eg at this point I do see that both bitpacking of exceptions, as
> pointed by
> Antoine, or plain ub2 encoding should work equally well .
Well, since 16 bits are actually enough for the current vector size, I'd
say we can keep things simple.
Regards
Antoine.
Re: [Parquet] ALP Encoding for Floating point data
Posted by Micah Kornfield <em...@gmail.com>.
Thanks Prateek,
> *Alp Version in Header*
> Micah's point
> *`* I'd suggest modelling fundamentally different algorithms with the
> top level encoding enum, and have versioning/control bits where we believe
> we will likely want to iterate`
> Yes this is exactly what is happening here. An enum to add AlpRd(and more)
> and version control to iterate anything fundamental (like a layout change
> of the
> metadata).
I think there might still be some misalignment on which enum we are talking
about. I was referring to an enum in parquet.thrift for encoding
<https://github.com/apache/parquet-format/blame/master/src/main/thrift/parquet.thrift#L630>
[1].
Specifically for the current proposal for ALP would ALP_PSEUDODECIMAL. For
AlpRd we would and ALP_RD (or something similar). If for some reason we
need ALP_PSEUDODECIMAL, we always have ALP_PSEUDO_DECIMAL_2. Given the
other extension points, I hope we wouldn't need ALP_PSEUDO_DECIMAL_2 (or at
least we get a better name) for a long time.
I think the main technical trade-off here is how the thrift parsers across
bindings handle unknown enum values (e.g. if these encodings show in the
footer in stats, would that make the whole file unreadable, and do we care).
Cheers,
Micah
[1]
https://github.com/apache/parquet-format/blame/master/src/main/thrift/parquet.thrift#L630
On Tue, Feb 3, 2026 at 2:40 PM PRATEEK GAUR <pr...@gmail.com> wrote:
> Hi Antoine and Micah,
>
> Apologies for getting back on this a little late.
>
> *Running Perf tests*
> @Antoine Pitrou <an...@python.org> were you able to figure out the steps
> to run the tests?
>
> *Sampling Frequency*
> We want to pick the right parameters to encode the values with. That is
> what the Spec requires.
> From the implementation perspective you raise a good point that did cross
> my
> mind that 'practically we don't want to sample for every page', for
> performance
> reasons. My thinking is each engine is free to decide this.
> 1) Do it at page level if data is changing often
> 2) Provide fixed presets via config
> 3) Do it once per encoder (per column, as Micah pointed out)
> 4) Provide a fancy config.
> I agree with Micah here '*I think maybe should clarify that the*
> *encoding algorithm in a specification is a recommendation'*
>
>
> *Number of values to pick for sampling*
> 'why does this have to be a constant'
> You are right, it doesn't need to be a constant, hence the spec doesn't say
> so.
> Everything that is segregated out in the AlpContants(c++ impl) file can be
> changed by
> configuration.
> (Did I get your question right @Antoine Pitrou <an...@python.org> ?)
>
> *Alp Version in Header*
> Micah's point
> *`* I'd suggest modelling fundamentally different algorithms with the
> top level encoding enum, and have versioning/control bits where we believe
> we will likely want to iterate`
> Yes this is exactly what is happening here. An enum to add AlpRd(and more)
> and version control to iterate anything fundamental (like a layout change
> of the
> metadata).
>
> Re-stating the points so that scrolling is not needed.
> 1. Change of integer encoding (see debate in this thread on FOR vs
> Delta). We also want to get fast lanes in at some point. I think an
> enum inside the page for versioning makes sense, as it allows for easier
> composability.
> 2. Change in structure to exceptions (e.g. G-ALP). G-ALP comes with some
> trade-offs, so it is not clear if it is something everyone would want to
> enable.
> 3. Offset indexes to vectors
> 4. Different floating point encoding algorithms (e.g. AlpRd + AlpBSS)
>
> For eg at this point I do see that both bitpacking of exceptions, as
> pointed by
> Antoine, or plain ub2 encoding should work equally well . I was running
> some benchmarks
> here and I was getting read speeds of around 20 GB/s for 10bit packed
> values which is
> quite good enough (graviton 3 processor).
> But for simplicity (and the fact that we won't get really large vectors, my
> inclination
> is towards ub2 values) but I want to keep the path open to possibly have
> bipacking
> as an option as the workloads evolve to a level we haven't thought about
> yet. We
> can always add a new encoding but I don't see a path to having 20+ top
> level
> encodings. Again I don't have a very strong bias towards keeping it or
> removing it
> but my thinking right now is let's have the flexibility and make it easier
> for people
> to evolve this encoding behind a version byte.
>
>
> Best
> Prateek
> PS : I probably have addressed all open threads raised by Antoine and
> Micah.
> (but I may have missed something)
>
>
>
>
>
> On Thu, Jan 29, 2026 at 10:52 PM Micah Kornfield <em...@gmail.com>
> wrote:
>
> > Hi Antoine and Prateek,
> >
> > > > In Parquet C++, encoding happens at page level, and I would guess
> other
> > > > implementations do something similar. Sampling cannot reasonably be
> > done
> > > > at a higher level, that would require invasive architectural changes.
> >
> >
> > At least in C++ I believe we cache the encoder at a column level [1] (I
> > believe the same is true for java). I think this implies one could sample
> > for the first page more or less, and then resample on some regular
> cadence
> > (or if compression degrades too much)? In general, the exact approach
> used
> > in implementations can vary here, so I think maybe should clarify that
> the
> > encoding algorithm in a specification is a recommendation, and we
> > concentrate the discussion on implementation to the individual language
> > binding PRs.
> >
> > 1) Addition of AlpRd (if that takes time to get in for read doubles).
> (This
> > > is easily addable with provided AlgorithmEnum)
> > > 2) Addition of AlpRd + Modified BSS (suggested by Azim) (This is easily
> > > addable with provided AlgorithmEnum)
> > > 3) Addition of different encoding (This is easily addable with provided
> > > AlgorithmEnum).
> >
> >
> > In short, I'd suggest modelling fundamentally different algorithms with
> the
> > top level encoding enum, and have versioning/control bits where we
> believe
> > we will likely want to iterate.
> >
> > Long version, based on my understanding of current open design points the
> > following extensions have been discussed:
> >
> > 1. Change of integer encoding (see debate in this thread on FOR vs
> > Delta). We also want to get fastlanes in at some point. I think an
> > enum inside the page for versioning makes sense, as it allows for easier
> > composability.
> > 2. Change in structure to exceptions (e.g. G-ALP). G-ALP comes with
> some
> > trade-offs, so it is not clear if it is something everyone would want to
> > enable.
> > 3. Offset indexes to vectors
> > 4. Different floating point encoding algorithms (e.g. AlpRd + AlpBSS)
> >
> > 1 is almost definitely an extension point and I think it pays to version
> > this within the page (if we decide to allow delta at some point, and then
> > do fast lanes we start having a lot of ALP-pseudecimal enums, that might
> > not add a lot of value). This would get worse if we multiply this by any
> > future exception layouts. 3. Feels like we can just make a decision now
> on
> > whether to add them, and if not added now can probably be added if we get
> > to cascaded encodings, or if really needed as a separate enum (but I'm
> open
> > to control bit here).
> >
> > For 4, these feel like they should be a top level enum for versioning,
> they
> > are fundamentally different algorithms with different code to decode
> them.
> > The trade-off here is that the current writers need to have some more
> > invasive changes to have better fallback or choice of initial encoder
> (but
> > this needs to happen anyways). We can easily have multiple page
> encodings
> > with a row-group from a reader's perspective
> >
> > For any other future changes that don't fall into these buckets a new
> > top-level enum is always an escape hatch (and we can learn from our
> > mistakes). Another take is it is just one byte per page, which would
> take a
> > while to add up even at Parquet's scale, but on the balance I'd lean
> > towards YAGNI.
> >
> > Cheers,
> > Micah
> >
> > [1]
> >
> >
> https://github.com/apache/arrow/blob/main/cpp/src/parquet/column_writer.cc#L1290
> >
> >
> > On Tue, Jan 27, 2026 at 9:47 AM PRATEEK GAUR <pr...@gmail.com> wrote:
> >
> > > Hi Antoine,
> > >
> > > Thanks. Replying inline to the discussion threads.
> > >
> > >
> > > > > https://github.com/apache/parquet-testing/pull/100/files.
> > > > > You might have to checkout that branch to be able to run the
> > > benchmarks.
> > > >
> > >
> > > Yeah just cherry-picking the branch should work. Let me me know if it
> > ends
> > > up
> > > being some kind of permission issue.
> > >
> > >
> > > >
> > > > Hmm, I'll take a look at that later and come back.
> > > >
> > > > > * the encoding of integers uses a custom framing with
> > > frame-of-reference
> > > > >> encoding inside it, but Parquet implementations already implement
> > > > >> DELTA_BINARY_PACKED which should have similar characteristics, so
> > why
> > > > >> not reuse that?
> > > >
> > > > Thanks. At worse it seems it would need one more bit per integer than
> > > > the proposed FOR scheme (it might need less if delta encoding allows
> > > > reducing entropy among the encoded integers). I'm not sure how that
> > > > makes it "fail".
> > > >
> > >
> > >
> > > My bad. I wanted to add another point.
> > > 1) So I looked at at a few other examples and in most cases FOR ended
> up
> > > using
> > > let's bits per value. My thinking is that for 1M (1B values) , it
> > will
> > > add up.
> > > 2) Another major point was that the bitunpacker + FOR was much much
> > faster
> > > than
> > > DeltaBitPack decoder. BitUnpacker+FOR was easily SIMDable but
> > > DeltaBitUnpack
> > > not so much. I vaguely remember the difference being around 2x. I
> can
> > > try and
> > > compute the numbers again. But today the entire bottleneck in the
> > > decoder shows
> > > up in bit-unpacking.
> > >
> > >
> > > > >> * there are a lot of fields in the headers that look a bit
> > superfluous
> > > > >> (though of course those bits are relatively cheap); for example,
> why
> > > > >> have a format "version" while we could define a new encoding for
> > > > >> incompatible evolutions?
> > > > >
> > > > > We discussed this point in the Spec
> > > > > <
> > > >
> > >
> >
> https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit>
> > > > > document a lot and have gravitated
> > > > > towards a versioning scheme for easier evolution.
> > > >
> > >
> > > > * Meta-answer:
> > > >
> > > > I don't see any such discussion in the comments and the document
> > doesn't
> > > > state any rationale for it.
> > > >
> > > > As a reference, Python PEPs include a discussion of rejected
> > > > alternatives so that people don't ask the same questions over and
> over
> > > > (see
> https://peps.python.org/pep-0810/#alternate-implementation-ideas
> > > > for an example).
> > > >
> > > > * Actual answer:
> > > >
> > > > My problem with this is that it will complicate understanding and
> > > > communicating the feature set supported by each Parquet
> implementation.
> > > > "ALP" will not be a single spec but an evolving one with its own
> > version
> > > > numbers. I'm not sure why that is better than adding a "ALP2" if we
> > even
> > > > want to evolve the spec.
> > > >
> > > > It will also complicate APIs that currently accept encoding numbers,
> > > > such as
> > > >
> > > >
> > >
> >
> https://github.com/apache/arrow/blob/5272a68c134deea82040f2f29bb6257ad7b52be0/cpp/src/parquet/properties.h#L221
> > > >
> > > > We need a clear explanation of what makes ALP so special that it
> > *needs*
> > > > its own versioning scheme. Otherwise we should remove it IMHO.
> > > >
> > >
> > > Based on the feedback I think it is more of a clarification/API
> > > complication discussion.
> > > I was thinking along these lines.
> > >
> > > Things that I thought will had to change but then we added enough
> > > flexibility in metadata structs to allow for these.
> > > 1) Addition of AlpRd (if that takes time to get in for read doubles).
> > (This
> > > is easily addable with provided AlgorithmEnum)
> > > 2) Addition of AlpRd + Modified BSS (suggested by Azim) (This is easily
> > > addable with provided AlgorithmEnum)
> > > 3) Addition of different encoding (This is easily addable with provided
> > > AlgorithmEnum).
> > >
> > > Things that I think might need a version field.
> > > 1) Right now FOR and ALP structs which are interleaved store certain
> set
> > of
> > > fields, given there have been lots of
> > > in coming suggestions my thinking was that having a version will
> > allow
> > > us to easily change them with minor
> > > changes to the code in all writers/readers (maybe that is a very
> bit
> > > task).
> > >
> > >
> > > >
> > > > >> * the "Total Encoded Element count" duplicates information already
> > in
> > > > >> the page header, with risks of inconsistent values (including
> > security
> > > > >> risks that require specific care in implementations)
> > > > >
> > > > > 'num_elements' : let me re-read and get back on this.
> > > >
> > > > Ok, Micah's answer cleared that up.
> > > >
> > >
> > > Cool.
> > >
> > >
> > > >
> > > > >> * what happens if the number of exceptions is above 65535? their
> > > indices
> > > > >> are coded as 16-bit uints. How about using the same encoding as
> for
> > > > >> bit-packed integers (e.g. DELTA_BINARY_PACKED), which will also
> > remove
> > > > >> the 65535 limitation.
> > > > >
> > > > > So, I don't see a need for a vector larger than 65535. With that
> > large
> > > > > vectors the
> > > > > overhead of metadata is small and you might as well break it into
> > > > multiple
> > > > > vectors. I'm gonna give it some more thought and get back.
> > > >
> > > > Ah, yes, I think you're right. Let's forget this :-)
> > > >
> > >
> > >
> > > :).
> > >
> > >
> > > >
> > > > > Sampling process should be statistically significant. It should
> pick
> > > > enough
> > > > > values
> > > > > and not have bias towards just the values towards the start. ALP
> > > > algorithm
> > > > > ensures
> > > > > that and tries to balance between not spending enough cycles to get
> > > right
> > > > > parameters
> > > > > and picking incorrect parameters.
> > > > >
> > > > > For a very large row group we can change the constant and have it
> > > > > selects over a larger data set.
> > > > > Or one can do it at page level too. Happy to discuss more on this.
> > > >
> > > > In Parquet C++, encoding happens at page level, and I would guess
> other
> > > > implementations do something similar. Sampling cannot reasonably be
> > done
> > > > at a higher level, that would require invasive architectural changes.
> > > >
> > > > But this raises another question: why does this have to be a
> constant?
> > > > When encoding a page, you know the population size (i.e. the actual
> > > > number of values that will be encoded). You don't need to estimate it
> > > > with a constant.
> > > >
> > >
> > > Some engines might decide to write large pages (large number of values
> > in a
> > > page) ~2B.
> > > One might not want to select all the values towards sampling process to
> > > keep sampling
> > > a slightly light weight operation. Also as @Andrew Lamb
> > > <an...@gmail.com> pointed out at times engines
> > > already know what the best values of hyper parameters are might want to
> > > provide it as
> > > input via an epic to send in pre-populated presets. This can happen
> > across
> > > multiple pages
> > > or multiple row groups after having calculated things for the first
> batch
> > > of data. This might
> > > further make sampling more light weight. I would gravitate towards
> > > flexibility here :).
> > >
> > >
> > > Thanks for this discussion
> > > Prateek
> > >
> >
>