You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by Martin Loncaric <ml...@janestreet.com.INVALID> on 2025/03/18 14:13:51 UTC
Pitch for Pcodec in Parquet (again)
Last year I sent an initial pitch for adding Pcodec (Pco for short) into
Parquet <https://lists.apache.org/thread/ht95wm8trfx2z4pq91t7170t2qjqg4yw>.
To re-introduce Pco: it's a codec for numerical sequences that almost
always gets higher compression ratio than anything else (very often in the
20-100% improvement range) and has performance on par with or slightly
faster than Parquet's existing encodings paired with ZSTD (link to paper
<https://arxiv.org/abs/2502.06112>). It goes from (array of numbers ->
bytes), as opposed to general-purpose codecs which are (bytes -> bytes).
Since there are probably exabytes of numerical Parquet data out there, I
think the value of this compression ratio improvement is colossal.
Last year, there were some valid concerns. We've made major progress to
address most of them:
1. Newness. Pco had just been released at the time. Now it is used to store
petabytes of data in Zarr (a popular tensor format), CnosDB (a time series
database), and someone is working on adding it into Postgres.
2. Limited FFI support, especially JVM. At the time, we did not have JVM
support, but now we do via JNI (io.github.pcodec/pco-jni). This is
currently compiled for the most common Linux, Mac, and Windows
architectures, and we can easily add more if needed. This is very similar
to how Zstd is used. I was told there's a rule that support in 3 languages
is required, and now we have Rust, Python, and JVM (and adding C wouldn't
be too hard).
3. Single maintainer. I now have another maintainer (@skielex) who is very
engaged, and a variety of other people have made code contributions in the
last year.
There was one other concern that can't be directly solved, but perhaps we
can avoid entirely:
4. Complexity. Parquet's existing encodings are very simple, and it sounds
like people would like to keep it that way. Pcodec's complexity is
somewhere in between that of an encoding and a compression (e.g. it has 11k
LoC, compared to Zstd's 70k). We could frame Pco as a Parquet compression
instead of encoding if that alleviates concerns; the big limitation is that
it would only work on PLAIN encoding, since it needs the original numbers.
There are more details, so we may need another thread about this choice if
we decide to move forward.
Do people have more questions? What would the next step be?
Other links:
* format spec <https://github.com/pcodec/pcodec/blob/main/docs/format.md>
* old results with Pco hacked into Parquet
<https://graphallthethings.com/posts/the-parquet-we-could-have>
Re: Pitch for Pcodec in Parquet (again)
Posted by Andrew Lamb <an...@gmail.com>.
Thank you for bringing this up Martin,
The challenge is, as I think we can see from this thread, is finding a
balance between the cost of adding something new to the spec and the number
of users (and thus contributors to help make it happen). If a feature isn't
widely adopted in the ecosystem adding it to the spec just increases
complexity meaninglessly.
While I have no comments on Pco specifically one way or the other, I
believe there *are* several research papers / proposals working their way
through the publishing process that describe 'extensible' encodings (for
example, that allow an optional machine readable implementation of a
decoder in the file, via WASM). In my opinion this is the most promising
way to extend the parquet format with new, potentially very domain
specific, encodings.
I hinted at this at one of the syncs in October[1] and I hope we'll have
more to share later this year.
Andrew
[1]: https://lists.apache.org/thread/jvdnx7ppkob0yfoty5chqc9bh4f9nxsj
On Tue, Mar 18, 2025 at 12:16 PM Antoine Pitrou <an...@python.org> wrote:
>
> Hello,
>
> While I'm rather lukewarm towards an additional, novel encoding with
> a high implementation complexity, I think your arguments are unfair,
> Alkis.
>
> On Tue, 18 Mar 2025 15:56:08 +0100 Alkis Evlogimenos wrote:
> >
> > From our internal numbers (Databricks) very little data in parquet is
> > numbers. In terms of bytes flowing through the readers (uncompressed) we
> > see the following distribution:
>
> Please remember that Databricks is only one user of Parquet. Just
> because most of your data is BINARY doesn't mean this applies to
> Parquet data around the world.
>
> (for example, Parquet files for machine learning would obviously
> contain many numeric columns)
>
> > In addition to the above distribution we also know the average
> compression
> > ratio for integers with general compressors which is about 1.5x.
>
> Not only we don't know what the actual *average* would be on the entire
> corpus of Parquet files around the world, but an average over an
> unknown statistical distribution has very little information value.
>
> For example, if the average were to be 1.5x, but with an upper decile at
> 20x, then that upper decile would be worth optimizing for (a decile of
> Parquet files is certainly a huge amount of data).
>
> > My stance is that adding future encodings should be gated with a large
> > enough experiment on *real* data showing both efficacy and wide
> > applicability.
>
> Agreed, but which "real" data? :-)
>
> Regards
>
> Antoine.
>
>
>
Re: Pitch for Pcodec in Parquet (again)
Posted by Micah Kornfield <em...@gmail.com>.
Yes, that is my hope, nothing shareable yet, I hope to have something this
week, apologies again for the delay.
On Wed, Apr 2, 2025 at 11:46 AM Martin Loncaric
<ml...@janestreet.com.invalid> wrote:
> Thanks Micah - I'll be happy to add comments/suggestions if you have a
> draft ready. I assume it'll focus on the main 3 things we discussed in the
> meeting: language support, developer support, and benchmark results.
>
> On Wed, Apr 2, 2025 at 2:48 AM Micah Kornfield <em...@gmail.com>
> wrote:
>
> > Apologies I been delayed in drafting this, should have something by end
> of
> > this week to share
> >
> > On Thursday, March 20, 2025, Micah Kornfield <em...@gmail.com>
> > wrote:
> >
> > > Based on the in person sync, I took an action item to try to write a
> > draft
> > > doc so we can come to a clear consensus on how to decide on new
> > > encodings/compression. I hope to have something to share next week but
> > it
> > > will likely need further input from the community.
> > >
> > > Thanks,
> > > Micah
> > >
> > > On Thu, Mar 20, 2025 at 3:33 AM Antoine Pitrou <an...@python.org>
> > wrote:
> > >
> > >> On Tue, 18 Mar 2025 19:08:04 +0100
> > >> Alkis Evlogimenos
> > >> <al...@databricks.com.INVALID>
> > >> wrote:
> > >> > At the end it boils down to which dataset you think is more
> > >> representative
> > >> > of the world data.
> > >>
> > >> This sentence does not even have a precise meaning. Data is plural,
> > >> there is no "representative" dataset.
> > >>
> > >> If someone tells you that the average animal on Earth is 2 millimeters
> > >> long, is that "representative" of the characteristics of mammals?
> > >>
> > >> In the end, the question is whether a new encoding brings enough
> > >> benefits in *some* cases to justify including it in Parquet. You may
> > >> care primarily about Databricks customers, but some people don't. This
> > >> is not a Databricks project.
> > >>
> > >> Regards
> > >>
> > >> Antoine.
> > >>
> > >>
> > >>
> >
>
Re: Pitch for Pcodec in Parquet (again)
Posted by Micah Kornfield <em...@gmail.com>.
Yes, that is my hope, nothing shareable yet, I hope to have something this
week, apologies again for the delay.
On Wed, Apr 2, 2025 at 11:46 AM Martin Loncaric
<ml...@janestreet.com.invalid> wrote:
> Thanks Micah - I'll be happy to add comments/suggestions if you have a
> draft ready. I assume it'll focus on the main 3 things we discussed in the
> meeting: language support, developer support, and benchmark results.
>
> On Wed, Apr 2, 2025 at 2:48 AM Micah Kornfield <em...@gmail.com>
> wrote:
>
> > Apologies I been delayed in drafting this, should have something by end
> of
> > this week to share
> >
> > On Thursday, March 20, 2025, Micah Kornfield <em...@gmail.com>
> > wrote:
> >
> > > Based on the in person sync, I took an action item to try to write a
> > draft
> > > doc so we can come to a clear consensus on how to decide on new
> > > encodings/compression. I hope to have something to share next week but
> > it
> > > will likely need further input from the community.
> > >
> > > Thanks,
> > > Micah
> > >
> > > On Thu, Mar 20, 2025 at 3:33 AM Antoine Pitrou <an...@python.org>
> > wrote:
> > >
> > >> On Tue, 18 Mar 2025 19:08:04 +0100
> > >> Alkis Evlogimenos
> > >> <al...@databricks.com.INVALID>
> > >> wrote:
> > >> > At the end it boils down to which dataset you think is more
> > >> representative
> > >> > of the world data.
> > >>
> > >> This sentence does not even have a precise meaning. Data is plural,
> > >> there is no "representative" dataset.
> > >>
> > >> If someone tells you that the average animal on Earth is 2 millimeters
> > >> long, is that "representative" of the characteristics of mammals?
> > >>
> > >> In the end, the question is whether a new encoding brings enough
> > >> benefits in *some* cases to justify including it in Parquet. You may
> > >> care primarily about Databricks customers, but some people don't. This
> > >> is not a Databricks project.
> > >>
> > >> Regards
> > >>
> > >> Antoine.
> > >>
> > >>
> > >>
> >
>
Re: Pitch for Pcodec in Parquet (again)
Posted by Martin Loncaric <ml...@janestreet.com.INVALID>.
Thanks Micah - I'll be happy to add comments/suggestions if you have a
draft ready. I assume it'll focus on the main 3 things we discussed in the
meeting: language support, developer support, and benchmark results.
On Wed, Apr 2, 2025 at 2:48 AM Micah Kornfield <em...@gmail.com>
wrote:
> Apologies I been delayed in drafting this, should have something by end of
> this week to share
>
> On Thursday, March 20, 2025, Micah Kornfield <em...@gmail.com>
> wrote:
>
> > Based on the in person sync, I took an action item to try to write a
> draft
> > doc so we can come to a clear consensus on how to decide on new
> > encodings/compression. I hope to have something to share next week but
> it
> > will likely need further input from the community.
> >
> > Thanks,
> > Micah
> >
> > On Thu, Mar 20, 2025 at 3:33 AM Antoine Pitrou <an...@python.org>
> wrote:
> >
> >> On Tue, 18 Mar 2025 19:08:04 +0100
> >> Alkis Evlogimenos
> >> <al...@databricks.com.INVALID>
> >> wrote:
> >> > At the end it boils down to which dataset you think is more
> >> representative
> >> > of the world data.
> >>
> >> This sentence does not even have a precise meaning. Data is plural,
> >> there is no "representative" dataset.
> >>
> >> If someone tells you that the average animal on Earth is 2 millimeters
> >> long, is that "representative" of the characteristics of mammals?
> >>
> >> In the end, the question is whether a new encoding brings enough
> >> benefits in *some* cases to justify including it in Parquet. You may
> >> care primarily about Databricks customers, but some people don't. This
> >> is not a Databricks project.
> >>
> >> Regards
> >>
> >> Antoine.
> >>
> >>
> >>
>
Re: Pitch for Pcodec in Parquet (again)
Posted by Micah Kornfield <em...@gmail.com>.
Apologies I been delayed in drafting this, should have something by end of
this week to share
On Thursday, March 20, 2025, Micah Kornfield <em...@gmail.com> wrote:
> Based on the in person sync, I took an action item to try to write a draft
> doc so we can come to a clear consensus on how to decide on new
> encodings/compression. I hope to have something to share next week but it
> will likely need further input from the community.
>
> Thanks,
> Micah
>
> On Thu, Mar 20, 2025 at 3:33 AM Antoine Pitrou <an...@python.org> wrote:
>
>> On Tue, 18 Mar 2025 19:08:04 +0100
>> Alkis Evlogimenos
>> <al...@databricks.com.INVALID>
>> wrote:
>> > At the end it boils down to which dataset you think is more
>> representative
>> > of the world data.
>>
>> This sentence does not even have a precise meaning. Data is plural,
>> there is no "representative" dataset.
>>
>> If someone tells you that the average animal on Earth is 2 millimeters
>> long, is that "representative" of the characteristics of mammals?
>>
>> In the end, the question is whether a new encoding brings enough
>> benefits in *some* cases to justify including it in Parquet. You may
>> care primarily about Databricks customers, but some people don't. This
>> is not a Databricks project.
>>
>> Regards
>>
>> Antoine.
>>
>>
>>
Re: Pitch for Pcodec in Parquet (again)
Posted by Micah Kornfield <em...@gmail.com>.
Apologies I been delayed in drafting this, should have something by end of
this week to share
On Thursday, March 20, 2025, Micah Kornfield <em...@gmail.com> wrote:
> Based on the in person sync, I took an action item to try to write a draft
> doc so we can come to a clear consensus on how to decide on new
> encodings/compression. I hope to have something to share next week but it
> will likely need further input from the community.
>
> Thanks,
> Micah
>
> On Thu, Mar 20, 2025 at 3:33 AM Antoine Pitrou <an...@python.org> wrote:
>
>> On Tue, 18 Mar 2025 19:08:04 +0100
>> Alkis Evlogimenos
>> <al...@databricks.com.INVALID>
>> wrote:
>> > At the end it boils down to which dataset you think is more
>> representative
>> > of the world data.
>>
>> This sentence does not even have a precise meaning. Data is plural,
>> there is no "representative" dataset.
>>
>> If someone tells you that the average animal on Earth is 2 millimeters
>> long, is that "representative" of the characteristics of mammals?
>>
>> In the end, the question is whether a new encoding brings enough
>> benefits in *some* cases to justify including it in Parquet. You may
>> care primarily about Databricks customers, but some people don't. This
>> is not a Databricks project.
>>
>> Regards
>>
>> Antoine.
>>
>>
>>
Re: Pitch for Pcodec in Parquet (again)
Posted by Micah Kornfield <em...@gmail.com>.
Based on the in person sync, I took an action item to try to write a draft
doc so we can come to a clear consensus on how to decide on new
encodings/compression. I hope to have something to share next week but it
will likely need further input from the community.
Thanks,
Micah
On Thu, Mar 20, 2025 at 3:33 AM Antoine Pitrou <an...@python.org> wrote:
> On Tue, 18 Mar 2025 19:08:04 +0100
> Alkis Evlogimenos
> <al...@databricks.com.INVALID>
> wrote:
> > At the end it boils down to which dataset you think is more
> representative
> > of the world data.
>
> This sentence does not even have a precise meaning. Data is plural,
> there is no "representative" dataset.
>
> If someone tells you that the average animal on Earth is 2 millimeters
> long, is that "representative" of the characteristics of mammals?
>
> In the end, the question is whether a new encoding brings enough
> benefits in *some* cases to justify including it in Parquet. You may
> care primarily about Databricks customers, but some people don't. This
> is not a Databricks project.
>
> Regards
>
> Antoine.
>
>
>
Re: Pitch for Pcodec in Parquet (again)
Posted by Micah Kornfield <em...@gmail.com>.
Based on the in person sync, I took an action item to try to write a draft
doc so we can come to a clear consensus on how to decide on new
encodings/compression. I hope to have something to share next week but it
will likely need further input from the community.
Thanks,
Micah
On Thu, Mar 20, 2025 at 3:33 AM Antoine Pitrou <an...@python.org> wrote:
> On Tue, 18 Mar 2025 19:08:04 +0100
> Alkis Evlogimenos
> <al...@databricks.com.INVALID>
> wrote:
> > At the end it boils down to which dataset you think is more
> representative
> > of the world data.
>
> This sentence does not even have a precise meaning. Data is plural,
> there is no "representative" dataset.
>
> If someone tells you that the average animal on Earth is 2 millimeters
> long, is that "representative" of the characteristics of mammals?
>
> In the end, the question is whether a new encoding brings enough
> benefits in *some* cases to justify including it in Parquet. You may
> care primarily about Databricks customers, but some people don't. This
> is not a Databricks project.
>
> Regards
>
> Antoine.
>
>
>
Re: Pitch for Pcodec in Parquet (again)
Posted by Antoine Pitrou <an...@python.org>.
On Tue, 18 Mar 2025 19:08:04 +0100
Alkis Evlogimenos
<al...@databricks.com.INVALID>
wrote:
> At the end it boils down to which dataset you think is more representative
> of the world data.
This sentence does not even have a precise meaning. Data is plural,
there is no "representative" dataset.
If someone tells you that the average animal on Earth is 2 millimeters
long, is that "representative" of the characteristics of mammals?
In the end, the question is whether a new encoding brings enough
benefits in *some* cases to justify including it in Parquet. You may
care primarily about Databricks customers, but some people don't. This
is not a Databricks project.
Regards
Antoine.
Re: Pitch for Pcodec in Parquet (again)
Posted by Alkis Evlogimenos <al...@databricks.com.INVALID>.
One of the things we worked on last year was to add binary protocol
extensions to parquet. This is a way to embed arbitrary metadata to any
thrift serialized blob in parquet in a way that old readers maintain
backwards compatibility:
https://github.com/apache/parquet-format/blob/master/BinaryProtocolExtensions.md
Using this scheme, one can implement Pco, encode numeric columns twice, one
with Pco and one with standard encodings (if backwards compatibility is
needed). Readers that know about Pco will read the Pco data, old readers
will read the compatibility data. If compatibility is not needed one can
even drop the second encoding. Once this is in place, run it experimentally
in some production environment and report back on the findings.
This is what we are doing with the new footer: we embed another version of
the footer in the footer itself and do experiments on real world data
before we propose to add this to the format.
I feel it would be more productive to discuss this in the sync because I
see that you are making wrong conclusions about the data I presented:
> a. A very likely explanation is that Pco simply compresses data better
than what Databricks customers are doing. This is a good thing in Pco's
favor.
This doesn't follow. The data you chose to benchmark Pco with, compress 5x
with *zstd*. The data I observe through all of Databricks compress 1.5x
with *zstd*. These numbers have nothing to do with how well Pco compresses
anything. They clearly show that the benchmark data you picked are not
representative of Databricks data.
> Even if somehow none of our 6 publicly available, columnar datasets are
representative of Databricks' data compressibility, it can still be
representative of Databricks' data patterns.
Maybe for a subset of the 7%. But that's speculation.
> Even if Databricks data exhibits different patterns not found in any of
our benchmarks, Pco can still handle them better.
This is also speculation. How do we know Pco handles all numeric data
better than zstd? As far as we know, Pco ran only the 6 publicly available
datasets and does a 20% better than zstd on data that zstd can compress by
5x. We do not know how well Pco does on data that has high entropy and
compresses 1.5x.
> Since your data is proprietary, the onus is on you to demonstrate any
properties it has empirically.
The bar to add new encodings is/should be very high. Implementing them is
very costly (many languages, many engines) and removing them is virtually
impossible. Since you are proposing for everyone to incur this cost (and it
is a very high cost - 1/6th of LoC of zstd - none of the other encodings
are anywhere close to this complexity) it is only logical that the onus
falls on you. It would be quite unfortunate if the mode operandi for
parquet format evolution is for engine owners to implement every proposed
encoding to prove/disprove their worth in practice.
On Tue, Mar 18, 2025 at 9:50 PM Martin Loncaric
<ml...@janestreet.com.invalid> wrote:
> *Alkis:*
>
> One of your arguments is that (only 7% of decompressed parquet data in
> Databricks is numerical => better numerical compression isn't worthwhile).
> Problems with that argument:
> * Even 7% of Parquet data is a tremendous amount.
> * Compressed, a much higher proportion of data is numerical. As you said
> yourself, numerical data in Databricks only averages a compression ratio of
> only 1.5. Strings are usually much more compressible.
> * As you mentioned, that's only incoming data, and stored data could be
> different.
> * Databricks isn't representative of all use cases.
>
> Another is that (Pco gets high compression ratio on benchmark datasets, and
> Databricks averages low compression ratio on its data ==(a)==> Pco's
> benchmarks aren't representative of Databricks data ==(b)==> Pco isn't good
> on Databricks data). Problems with this argument:
> a. A very likely explanation is that Pco simply compresses data better than
> what Databricks customers are doing. This is a good thing in Pco's favor.
> b.
> * Even if somehow none of our 6 publicly available, columnar datasets are
> representative of Databricks' data compressibility, it can still be
> representative of Databricks' data patterns.
> * Even if Databricks data exhibits different patterns not found in any of
> our benchmarks, Pco can still handle them better.
> * Since your data is proprietary, the onus is on you to demonstrate any
> properties it has empirically.
>
> *ZOOMING OUT*:
>
> I think we can all agree that better compression for numerical data is a
> good thing, and that Parquet should adopt newer techniques if they improve
> substantially on the status quo. I welcome any benchmarks if people would
> like to evaluate this for their own data; the pcodec CLI makes this very
> easy.
>
> Complexity seems to be the main sticking point. I would argue that a 20%
> higher compression ratio is a huge enough win to warrant a slight increase
> in complexity (Pco is <1/6 the size of zstd in LoC). To my knowledge,
> storage (not to mention network load) is the most important cost factor in
> Parquet today. I'm happy to discuss this compression/complexity tradeoff
> more in this thread or at the Parquet meet tomorrow.
>
> On Tue, Mar 18, 2025 at 2:08 PM Alkis Evlogimenos
> <al...@databricks.com.invalid> wrote:
>
> > Just to clarify, the distribution numbers I shared earlier represent data
> > after decompression, not data stored without compression. I now
> understand
> > that "uncompressed" may have been misinterpreted. In addition I want to
> > point out that these are data that go through the readers, so this is
> > skewed towards reads and not towards what is at rest. Lastly, the data
> > comes from tracking across all Databricks customers, encompassing a wide
> > variety of datasets, including financial, ML, and more. Therefore, I
> > believe this mix is fairly representative of a "world average" skewed
> > towards read performance.
> >
> > Regarding the average compression ratio of ~1.5 for integers. It is true
> > that it does not tell us a lot but I disagree that we cannot make
> > conclusions without the full distribution. We know the compression
> > Databricks observes across INT64+INT32 data is 1.5x (it is similar for
> > float as well). We know pco benchmarks are against data that compresses
> > 4-5x with general compressors. This means the pco benchmark data is not
> > representative of the data we observe in the Databricks set. This means
> the
> > performance of pco on the selected benchmark data is not a good predictor
> > of what it will do in the Databricks data. And if we believe Databricks
> > data is closer to representative of the world data, then it is not a good
> > predictor of what pco will do for the world.
> >
> > At the end it boils down to which dataset you think is more
> representative
> > of the world data. Put qualitatively, does the world (numeric) data
> > look more like taxi+air quality+reddit or more like the data of a mix of
> > the thousands of customers around the world that happen to use Databricks
> > for their data and AI needs?
> >
> >
> > On Tue, Mar 18, 2025 at 5:16 PM Antoine Pitrou <an...@python.org>
> wrote:
> >
> > >
> > > Hello,
> > >
> > > While I'm rather lukewarm towards an additional, novel encoding with
> > > a high implementation complexity, I think your arguments are unfair,
> > > Alkis.
> > >
> > > On Tue, 18 Mar 2025 15:56:08 +0100 Alkis Evlogimenos wrote:
> > > >
> > > > From our internal numbers (Databricks) very little data in parquet is
> > > > numbers. In terms of bytes flowing through the readers (uncompressed)
> > we
> > > > see the following distribution:
> > >
> > > Please remember that Databricks is only one user of Parquet. Just
> > > because most of your data is BINARY doesn't mean this applies to
> > > Parquet data around the world.
> > >
> > > (for example, Parquet files for machine learning would obviously
> > > contain many numeric columns)
> > >
> > > > In addition to the above distribution we also know the average
> > > compression
> > > > ratio for integers with general compressors which is about 1.5x.
> > >
> > > Not only we don't know what the actual *average* would be on the entire
> > > corpus of Parquet files around the world, but an average over an
> > > unknown statistical distribution has very little information value.
> > >
> > > For example, if the average were to be 1.5x, but with an upper decile
> at
> > > 20x, then that upper decile would be worth optimizing for (a decile of
> > > Parquet files is certainly a huge amount of data).
> > >
> > > > My stance is that adding future encodings should be gated with a
> large
> > > > enough experiment on *real* data showing both efficacy and wide
> > > > applicability.
> > >
> > > Agreed, but which "real" data? :-)
> > >
> > > Regards
> > >
> > > Antoine.
> > >
> > >
> > >
> >
>
Re: Pitch for Pcodec in Parquet (again)
Posted by Martin Loncaric <ml...@janestreet.com.INVALID>.
*Alkis:*
One of your arguments is that (only 7% of decompressed parquet data in
Databricks is numerical => better numerical compression isn't worthwhile).
Problems with that argument:
* Even 7% of Parquet data is a tremendous amount.
* Compressed, a much higher proportion of data is numerical. As you said
yourself, numerical data in Databricks only averages a compression ratio of
only 1.5. Strings are usually much more compressible.
* As you mentioned, that's only incoming data, and stored data could be
different.
* Databricks isn't representative of all use cases.
Another is that (Pco gets high compression ratio on benchmark datasets, and
Databricks averages low compression ratio on its data ==(a)==> Pco's
benchmarks aren't representative of Databricks data ==(b)==> Pco isn't good
on Databricks data). Problems with this argument:
a. A very likely explanation is that Pco simply compresses data better than
what Databricks customers are doing. This is a good thing in Pco's favor.
b.
* Even if somehow none of our 6 publicly available, columnar datasets are
representative of Databricks' data compressibility, it can still be
representative of Databricks' data patterns.
* Even if Databricks data exhibits different patterns not found in any of
our benchmarks, Pco can still handle them better.
* Since your data is proprietary, the onus is on you to demonstrate any
properties it has empirically.
*ZOOMING OUT*:
I think we can all agree that better compression for numerical data is a
good thing, and that Parquet should adopt newer techniques if they improve
substantially on the status quo. I welcome any benchmarks if people would
like to evaluate this for their own data; the pcodec CLI makes this very
easy.
Complexity seems to be the main sticking point. I would argue that a 20%
higher compression ratio is a huge enough win to warrant a slight increase
in complexity (Pco is <1/6 the size of zstd in LoC). To my knowledge,
storage (not to mention network load) is the most important cost factor in
Parquet today. I'm happy to discuss this compression/complexity tradeoff
more in this thread or at the Parquet meet tomorrow.
On Tue, Mar 18, 2025 at 2:08 PM Alkis Evlogimenos
<al...@databricks.com.invalid> wrote:
> Just to clarify, the distribution numbers I shared earlier represent data
> after decompression, not data stored without compression. I now understand
> that "uncompressed" may have been misinterpreted. In addition I want to
> point out that these are data that go through the readers, so this is
> skewed towards reads and not towards what is at rest. Lastly, the data
> comes from tracking across all Databricks customers, encompassing a wide
> variety of datasets, including financial, ML, and more. Therefore, I
> believe this mix is fairly representative of a "world average" skewed
> towards read performance.
>
> Regarding the average compression ratio of ~1.5 for integers. It is true
> that it does not tell us a lot but I disagree that we cannot make
> conclusions without the full distribution. We know the compression
> Databricks observes across INT64+INT32 data is 1.5x (it is similar for
> float as well). We know pco benchmarks are against data that compresses
> 4-5x with general compressors. This means the pco benchmark data is not
> representative of the data we observe in the Databricks set. This means the
> performance of pco on the selected benchmark data is not a good predictor
> of what it will do in the Databricks data. And if we believe Databricks
> data is closer to representative of the world data, then it is not a good
> predictor of what pco will do for the world.
>
> At the end it boils down to which dataset you think is more representative
> of the world data. Put qualitatively, does the world (numeric) data
> look more like taxi+air quality+reddit or more like the data of a mix of
> the thousands of customers around the world that happen to use Databricks
> for their data and AI needs?
>
>
> On Tue, Mar 18, 2025 at 5:16 PM Antoine Pitrou <an...@python.org> wrote:
>
> >
> > Hello,
> >
> > While I'm rather lukewarm towards an additional, novel encoding with
> > a high implementation complexity, I think your arguments are unfair,
> > Alkis.
> >
> > On Tue, 18 Mar 2025 15:56:08 +0100 Alkis Evlogimenos wrote:
> > >
> > > From our internal numbers (Databricks) very little data in parquet is
> > > numbers. In terms of bytes flowing through the readers (uncompressed)
> we
> > > see the following distribution:
> >
> > Please remember that Databricks is only one user of Parquet. Just
> > because most of your data is BINARY doesn't mean this applies to
> > Parquet data around the world.
> >
> > (for example, Parquet files for machine learning would obviously
> > contain many numeric columns)
> >
> > > In addition to the above distribution we also know the average
> > compression
> > > ratio for integers with general compressors which is about 1.5x.
> >
> > Not only we don't know what the actual *average* would be on the entire
> > corpus of Parquet files around the world, but an average over an
> > unknown statistical distribution has very little information value.
> >
> > For example, if the average were to be 1.5x, but with an upper decile at
> > 20x, then that upper decile would be worth optimizing for (a decile of
> > Parquet files is certainly a huge amount of data).
> >
> > > My stance is that adding future encodings should be gated with a large
> > > enough experiment on *real* data showing both efficacy and wide
> > > applicability.
> >
> > Agreed, but which "real" data? :-)
> >
> > Regards
> >
> > Antoine.
> >
> >
> >
>
Re: Pitch for Pcodec in Parquet (again)
Posted by Alkis Evlogimenos <al...@databricks.com.INVALID>.
Just to clarify, the distribution numbers I shared earlier represent data
after decompression, not data stored without compression. I now understand
that "uncompressed" may have been misinterpreted. In addition I want to
point out that these are data that go through the readers, so this is
skewed towards reads and not towards what is at rest. Lastly, the data
comes from tracking across all Databricks customers, encompassing a wide
variety of datasets, including financial, ML, and more. Therefore, I
believe this mix is fairly representative of a "world average" skewed
towards read performance.
Regarding the average compression ratio of ~1.5 for integers. It is true
that it does not tell us a lot but I disagree that we cannot make
conclusions without the full distribution. We know the compression
Databricks observes across INT64+INT32 data is 1.5x (it is similar for
float as well). We know pco benchmarks are against data that compresses
4-5x with general compressors. This means the pco benchmark data is not
representative of the data we observe in the Databricks set. This means the
performance of pco on the selected benchmark data is not a good predictor
of what it will do in the Databricks data. And if we believe Databricks
data is closer to representative of the world data, then it is not a good
predictor of what pco will do for the world.
At the end it boils down to which dataset you think is more representative
of the world data. Put qualitatively, does the world (numeric) data
look more like taxi+air quality+reddit or more like the data of a mix of
the thousands of customers around the world that happen to use Databricks
for their data and AI needs?
On Tue, Mar 18, 2025 at 5:16 PM Antoine Pitrou <an...@python.org> wrote:
>
> Hello,
>
> While I'm rather lukewarm towards an additional, novel encoding with
> a high implementation complexity, I think your arguments are unfair,
> Alkis.
>
> On Tue, 18 Mar 2025 15:56:08 +0100 Alkis Evlogimenos wrote:
> >
> > From our internal numbers (Databricks) very little data in parquet is
> > numbers. In terms of bytes flowing through the readers (uncompressed) we
> > see the following distribution:
>
> Please remember that Databricks is only one user of Parquet. Just
> because most of your data is BINARY doesn't mean this applies to
> Parquet data around the world.
>
> (for example, Parquet files for machine learning would obviously
> contain many numeric columns)
>
> > In addition to the above distribution we also know the average
> compression
> > ratio for integers with general compressors which is about 1.5x.
>
> Not only we don't know what the actual *average* would be on the entire
> corpus of Parquet files around the world, but an average over an
> unknown statistical distribution has very little information value.
>
> For example, if the average were to be 1.5x, but with an upper decile at
> 20x, then that upper decile would be worth optimizing for (a decile of
> Parquet files is certainly a huge amount of data).
>
> > My stance is that adding future encodings should be gated with a large
> > enough experiment on *real* data showing both efficacy and wide
> > applicability.
>
> Agreed, but which "real" data? :-)
>
> Regards
>
> Antoine.
>
>
>
Re: Pitch for Pcodec in Parquet (again)
Posted by Andrew Lamb <an...@gmail.com>.
Thank you for bringing this up Martin,
The challenge is, as I think we can see from this thread, is finding a
balance between the cost of adding something new to the spec and the number
of users (and thus contributors to help make it happen). If a feature isn't
widely adopted in the ecosystem adding it to the spec just increases
complexity meaninglessly.
While I have no comments on Pco specifically one way or the other, I
believe there *are* several research papers / proposals working their way
through the publishing process that describe 'extensible' encodings (for
example, that allow an optional machine readable implementation of a
decoder in the file, via WASM). In my opinion this is the most promising
way to extend the parquet format with new, potentially very domain
specific, encodings.
I hinted at this at one of the syncs in October[1] and I hope we'll have
more to share later this year.
Andrew
[1]: https://lists.apache.org/thread/jvdnx7ppkob0yfoty5chqc9bh4f9nxsj
On Tue, Mar 18, 2025 at 12:16 PM Antoine Pitrou <an...@python.org> wrote:
>
> Hello,
>
> While I'm rather lukewarm towards an additional, novel encoding with
> a high implementation complexity, I think your arguments are unfair,
> Alkis.
>
> On Tue, 18 Mar 2025 15:56:08 +0100 Alkis Evlogimenos wrote:
> >
> > From our internal numbers (Databricks) very little data in parquet is
> > numbers. In terms of bytes flowing through the readers (uncompressed) we
> > see the following distribution:
>
> Please remember that Databricks is only one user of Parquet. Just
> because most of your data is BINARY doesn't mean this applies to
> Parquet data around the world.
>
> (for example, Parquet files for machine learning would obviously
> contain many numeric columns)
>
> > In addition to the above distribution we also know the average
> compression
> > ratio for integers with general compressors which is about 1.5x.
>
> Not only we don't know what the actual *average* would be on the entire
> corpus of Parquet files around the world, but an average over an
> unknown statistical distribution has very little information value.
>
> For example, if the average were to be 1.5x, but with an upper decile at
> 20x, then that upper decile would be worth optimizing for (a decile of
> Parquet files is certainly a huge amount of data).
>
> > My stance is that adding future encodings should be gated with a large
> > enough experiment on *real* data showing both efficacy and wide
> > applicability.
>
> Agreed, but which "real" data? :-)
>
> Regards
>
> Antoine.
>
>
>
Re: Pitch for Pcodec in Parquet (again)
Posted by Antoine Pitrou <an...@python.org>.
Hello,
While I'm rather lukewarm towards an additional, novel encoding with
a high implementation complexity, I think your arguments are unfair,
Alkis.
On Tue, 18 Mar 2025 15:56:08 +0100 Alkis Evlogimenos wrote:
>
> From our internal numbers (Databricks) very little data in parquet is
> numbers. In terms of bytes flowing through the readers (uncompressed) we
> see the following distribution:
Please remember that Databricks is only one user of Parquet. Just
because most of your data is BINARY doesn't mean this applies to
Parquet data around the world.
(for example, Parquet files for machine learning would obviously
contain many numeric columns)
> In addition to the above distribution we also know the average compression
> ratio for integers with general compressors which is about 1.5x.
Not only we don't know what the actual *average* would be on the entire
corpus of Parquet files around the world, but an average over an
unknown statistical distribution has very little information value.
For example, if the average were to be 1.5x, but with an upper decile at
20x, then that upper decile would be worth optimizing for (a decile of
Parquet files is certainly a huge amount of data).
> My stance is that adding future encodings should be gated with a large
> enough experiment on *real* data showing both efficacy and wide
> applicability.
Agreed, but which "real" data? :-)
Regards
Antoine.
Re: Pitch for Pcodec in Parquet (again)
Posted by Martin Loncaric <ml...@janestreet.com.INVALID>.
*Curt:*
> Out of curiosity, how does this compare with ALP in terms of size and
> speed?
I hadn't seen ALP before - looks like it's extremely new. From skimming
their repo, I think you can expect Pco's compression ratio to be much
higher, but with much slower speed. ALP is probably better for in-memory
data transfers, whereas Pco would be better for data read over the network
or stored on disk. ALP seems to only apply to floats.
*Alkis:*
> From our internal numbers (Databricks) very little data in parquet is
> numbers...
Strings typically compress much better than numbers (dictionary benefits
them a lot more), so this distribution changes a lot when looking at
compressed data. Also, Databricks caters to a particular sort of Parquet
user. For instance, in the financial industry, almost all Parquet data
(even uncompressed) is numerical.
is the additional complexity of pco justified to optimize ~3% of data out
> there...
Even with the uncompressed Databricks accounting above, that's ~7% of the
data (all int or float types, and FIXED_LEN_BYTE_ARRAY is often float16).
it looks like most integer data is not amenable to compression by general
> compression schemes...
If Pco improves compression ratio from 1.5 -> 1.8, then it's even more
valuable than improving compression ratio from 5->6.
This raises the concern that the data used to benchmark pco is rather niche
We compare against 6 datasets in the paper. If you're skeptical, then I
encourage you to try it on some data (cargo install pco_cli; pcodec bench
-i /path/to/parquet -c pco,parquet:compression=zstd1 --iters 1)
On Tue, Mar 18, 2025 at 10:56 AM Alkis Evlogimenos
<al...@databricks.com.invalid> wrote:
> As Curt said, how does it compare with ALP is the obvious question - both
> in terms of compression/decompression speeds and compression ratio?
>
> From our internal numbers (Databricks) very little data in parquet is
> numbers. In terms of bytes flowing through the readers (uncompressed) we
> see the following distribution:
> BINARY 92.26%
> INT96 1.95%
> INT64 1.94%
> INT32 1.29%
> DOUBLE 1.24%
> FIXED_LEN_BYTE_ARRAY 1.24%
> FLOAT 0.07%
> BOOLEAN 0.02%
>
> Given the above, is the additional complexity of pco justified to optimize
> ~3% of data out there (assume what we observe is a representative sample of
> the worlds data)?
>
> In addition to the above distribution we also know the average compression
> ratio for integers with general compressors which is about 1.5x. Granted
> most parquet files are snappy compressed with zstd gaining traction but it
> looks like most integer data is not amenable to compression by general
> compression schemes. This does not align with the data used in the
> experiments for pco posted here
> https://graphallthethings.com/posts/the-parquet-we-could-have. This raises
> the concern that the data used to benchmark pco is rather niche, which
> means that pco might even apply to much less than the 3% of total parquet
> data of the distribution above. This would make the case of pulling pco in
> the spec even harder.
>
> My stance is that adding future encodings should be gated with a large
> enough experiment on *real* data showing both efficacy and wide
> applicability.
>
> On Tue, Mar 18, 2025 at 3:22 PM Curt Hagenlocher <cu...@hagenlocher.org>
> wrote:
>
> > Out of curiosity, how does this compare with ALP in terms of size and
> > speed?
> >
> > On Tue, Mar 18, 2025 at 7:14 AM Martin Loncaric
> > <ml...@janestreet.com.invalid> wrote:
> >
> > > Last year I sent an initial pitch for adding Pcodec (Pco for short)
> into
> > > Parquet <
> > https://lists.apache.org/thread/ht95wm8trfx2z4pq91t7170t2qjqg4yw
> > > >.
> > > To re-introduce Pco: it's a codec for numerical sequences that almost
> > > always gets higher compression ratio than anything else (very often in
> > the
> > > 20-100% improvement range) and has performance on par with or slightly
> > > faster than Parquet's existing encodings paired with ZSTD (link to
> paper
> > > <https://arxiv.org/abs/2502.06112>). It goes from (array of numbers ->
> > > bytes), as opposed to general-purpose codecs which are (bytes ->
> bytes).
> > > Since there are probably exabytes of numerical Parquet data out there,
> I
> > > think the value of this compression ratio improvement is colossal.
> > >
> > > Last year, there were some valid concerns. We've made major progress to
> > > address most of them:
> > >
> > > 1. Newness. Pco had just been released at the time. Now it is used to
> > store
> > > petabytes of data in Zarr (a popular tensor format), CnosDB (a time
> > series
> > > database), and someone is working on adding it into Postgres.
> > > 2. Limited FFI support, especially JVM. At the time, we did not have
> JVM
> > > support, but now we do via JNI (io.github.pcodec/pco-jni). This is
> > > currently compiled for the most common Linux, Mac, and Windows
> > > architectures, and we can easily add more if needed. This is very
> similar
> > > to how Zstd is used. I was told there's a rule that support in 3
> > languages
> > > is required, and now we have Rust, Python, and JVM (and adding C
> wouldn't
> > > be too hard).
> > > 3. Single maintainer. I now have another maintainer (@skielex) who is
> > very
> > > engaged, and a variety of other people have made code contributions in
> > the
> > > last year.
> > >
> > > There was one other concern that can't be directly solved, but perhaps
> we
> > > can avoid entirely:
> > >
> > > 4. Complexity. Parquet's existing encodings are very simple, and it
> > sounds
> > > like people would like to keep it that way. Pcodec's complexity is
> > > somewhere in between that of an encoding and a compression (e.g. it has
> > 11k
> > > LoC, compared to Zstd's 70k). We could frame Pco as a Parquet
> compression
> > > instead of encoding if that alleviates concerns; the big limitation is
> > that
> > > it would only work on PLAIN encoding, since it needs the original
> > numbers.
> > > There are more details, so we may need another thread about this choice
> > if
> > > we decide to move forward.
> > >
> > > Do people have more questions? What would the next step be?
> > >
> > > Other links:
> > > * format spec <
> https://github.com/pcodec/pcodec/blob/main/docs/format.md
> > >
> > > * old results with Pco hacked into Parquet
> > > <https://graphallthethings.com/posts/the-parquet-we-could-have>
> > >
> >
>
Re: Pitch for Pcodec in Parquet (again)
Posted by Alkis Evlogimenos <al...@databricks.com.INVALID>.
As Curt said, how does it compare with ALP is the obvious question - both
in terms of compression/decompression speeds and compression ratio?
From our internal numbers (Databricks) very little data in parquet is
numbers. In terms of bytes flowing through the readers (uncompressed) we
see the following distribution:
BINARY 92.26%
INT96 1.95%
INT64 1.94%
INT32 1.29%
DOUBLE 1.24%
FIXED_LEN_BYTE_ARRAY 1.24%
FLOAT 0.07%
BOOLEAN 0.02%
Given the above, is the additional complexity of pco justified to optimize
~3% of data out there (assume what we observe is a representative sample of
the worlds data)?
In addition to the above distribution we also know the average compression
ratio for integers with general compressors which is about 1.5x. Granted
most parquet files are snappy compressed with zstd gaining traction but it
looks like most integer data is not amenable to compression by general
compression schemes. This does not align with the data used in the
experiments for pco posted here
https://graphallthethings.com/posts/the-parquet-we-could-have. This raises
the concern that the data used to benchmark pco is rather niche, which
means that pco might even apply to much less than the 3% of total parquet
data of the distribution above. This would make the case of pulling pco in
the spec even harder.
My stance is that adding future encodings should be gated with a large
enough experiment on *real* data showing both efficacy and wide
applicability.
On Tue, Mar 18, 2025 at 3:22 PM Curt Hagenlocher <cu...@hagenlocher.org>
wrote:
> Out of curiosity, how does this compare with ALP in terms of size and
> speed?
>
> On Tue, Mar 18, 2025 at 7:14 AM Martin Loncaric
> <ml...@janestreet.com.invalid> wrote:
>
> > Last year I sent an initial pitch for adding Pcodec (Pco for short) into
> > Parquet <
> https://lists.apache.org/thread/ht95wm8trfx2z4pq91t7170t2qjqg4yw
> > >.
> > To re-introduce Pco: it's a codec for numerical sequences that almost
> > always gets higher compression ratio than anything else (very often in
> the
> > 20-100% improvement range) and has performance on par with or slightly
> > faster than Parquet's existing encodings paired with ZSTD (link to paper
> > <https://arxiv.org/abs/2502.06112>). It goes from (array of numbers ->
> > bytes), as opposed to general-purpose codecs which are (bytes -> bytes).
> > Since there are probably exabytes of numerical Parquet data out there, I
> > think the value of this compression ratio improvement is colossal.
> >
> > Last year, there were some valid concerns. We've made major progress to
> > address most of them:
> >
> > 1. Newness. Pco had just been released at the time. Now it is used to
> store
> > petabytes of data in Zarr (a popular tensor format), CnosDB (a time
> series
> > database), and someone is working on adding it into Postgres.
> > 2. Limited FFI support, especially JVM. At the time, we did not have JVM
> > support, but now we do via JNI (io.github.pcodec/pco-jni). This is
> > currently compiled for the most common Linux, Mac, and Windows
> > architectures, and we can easily add more if needed. This is very similar
> > to how Zstd is used. I was told there's a rule that support in 3
> languages
> > is required, and now we have Rust, Python, and JVM (and adding C wouldn't
> > be too hard).
> > 3. Single maintainer. I now have another maintainer (@skielex) who is
> very
> > engaged, and a variety of other people have made code contributions in
> the
> > last year.
> >
> > There was one other concern that can't be directly solved, but perhaps we
> > can avoid entirely:
> >
> > 4. Complexity. Parquet's existing encodings are very simple, and it
> sounds
> > like people would like to keep it that way. Pcodec's complexity is
> > somewhere in between that of an encoding and a compression (e.g. it has
> 11k
> > LoC, compared to Zstd's 70k). We could frame Pco as a Parquet compression
> > instead of encoding if that alleviates concerns; the big limitation is
> that
> > it would only work on PLAIN encoding, since it needs the original
> numbers.
> > There are more details, so we may need another thread about this choice
> if
> > we decide to move forward.
> >
> > Do people have more questions? What would the next step be?
> >
> > Other links:
> > * format spec <https://github.com/pcodec/pcodec/blob/main/docs/format.md
> >
> > * old results with Pco hacked into Parquet
> > <https://graphallthethings.com/posts/the-parquet-we-could-have>
> >
>
Re: Pitch for Pcodec in Parquet (again)
Posted by Curt Hagenlocher <cu...@hagenlocher.org>.
Out of curiosity, how does this compare with ALP in terms of size and speed?
On Tue, Mar 18, 2025 at 7:14 AM Martin Loncaric
<ml...@janestreet.com.invalid> wrote:
> Last year I sent an initial pitch for adding Pcodec (Pco for short) into
> Parquet <https://lists.apache.org/thread/ht95wm8trfx2z4pq91t7170t2qjqg4yw
> >.
> To re-introduce Pco: it's a codec for numerical sequences that almost
> always gets higher compression ratio than anything else (very often in the
> 20-100% improvement range) and has performance on par with or slightly
> faster than Parquet's existing encodings paired with ZSTD (link to paper
> <https://arxiv.org/abs/2502.06112>). It goes from (array of numbers ->
> bytes), as opposed to general-purpose codecs which are (bytes -> bytes).
> Since there are probably exabytes of numerical Parquet data out there, I
> think the value of this compression ratio improvement is colossal.
>
> Last year, there were some valid concerns. We've made major progress to
> address most of them:
>
> 1. Newness. Pco had just been released at the time. Now it is used to store
> petabytes of data in Zarr (a popular tensor format), CnosDB (a time series
> database), and someone is working on adding it into Postgres.
> 2. Limited FFI support, especially JVM. At the time, we did not have JVM
> support, but now we do via JNI (io.github.pcodec/pco-jni). This is
> currently compiled for the most common Linux, Mac, and Windows
> architectures, and we can easily add more if needed. This is very similar
> to how Zstd is used. I was told there's a rule that support in 3 languages
> is required, and now we have Rust, Python, and JVM (and adding C wouldn't
> be too hard).
> 3. Single maintainer. I now have another maintainer (@skielex) who is very
> engaged, and a variety of other people have made code contributions in the
> last year.
>
> There was one other concern that can't be directly solved, but perhaps we
> can avoid entirely:
>
> 4. Complexity. Parquet's existing encodings are very simple, and it sounds
> like people would like to keep it that way. Pcodec's complexity is
> somewhere in between that of an encoding and a compression (e.g. it has 11k
> LoC, compared to Zstd's 70k). We could frame Pco as a Parquet compression
> instead of encoding if that alleviates concerns; the big limitation is that
> it would only work on PLAIN encoding, since it needs the original numbers.
> There are more details, so we may need another thread about this choice if
> we decide to move forward.
>
> Do people have more questions? What would the next step be?
>
> Other links:
> * format spec <https://github.com/pcodec/pcodec/blob/main/docs/format.md>
> * old results with Pco hacked into Parquet
> <https://graphallthethings.com/posts/the-parquet-we-could-have>
>