You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by Jiayi Wang <wj...@gmail.com> on 2026/09/21 15:26:15 UTC
Introducing the Parquet Modular Footer
Hi everyone,
We’ve been discussing the Modular Footer for quite a while, including
through a dedicated footer working group meeting every two weeks, and we
finally have a proposal ready for broader review.
The proposal covers the motivation, format design, compatibility
considerations, and benchmark results. Huge thanks to everyone who has
contributed to the project and the discussions so far.
The proposal doc, parquet-format PR, and tracking issue are all available
here:
Proposal doc:
https://docs.google.com/document/d/1xN4Nqh-fh-eH-cUz9e6Rqs_gPXRBKUh4s29EzX_NU8M/edit?usp=sharing
PR: https://github.com/apache/parquet-format/pull/623/changes
Parquet-format issue: https://github.com/apache/parquet-format/issues/622
The proposal introduces several significant conceptual changes, so I expect
there will be a lot of discussion and feedback. For the first round of
review, I’d suggest focusing primarily on the high-level direction and
design choices, and leaving more detailed questions (for example, whether a
particular field can be deprecated) for later discussion. I think this will
help us converge on the major design questions first and keep the review
process moving efficiently.
Thanks!
Best,
Jiayi
Re: Introducing the Parquet Modular Footer
Posted by Jiayi Wang <wj...@gmail.com>.
Hi everyone,
As mentioned previously, we have revisited the proposal [1] to make it
more flexible and also address existing feedback. I’d like to share the new
version of the proposal [2], which is included as a new tab in the same
document. A summary of what changed and why:
Flexible Data Layouts
Per discussions on evolving Parquet data layout (e.g. the “non-contiguous
pages” proposal), our goal is to have the new footer handle both existing
and potential future layouts seamlessly. The updated proposal doesn’t
assume that row groups are physical boundaries. It preserves the wording of
“column chunks”, but these are simply defined to mean a contiguous set of
pages for one column. Different columns may have different numbers of
chunks in a file. The “row group” wording is preserved as well, but row
group metadata is made optional. Lastly, stats are re-defined based on row
ranges rather than row groups; different columns may publish stats at
different granularities.
Page-based Representation
A big feedback point on the initial proposal was that the EncodedArray
concept was opaque/confusing. It was added to efficiently model arrays in
Thrift and provide random access, at the expense of readability. The
updated proposal uses Parquet pages to represent modules, which yields a
self-describing layout that evolves naturally as Parquet gains new
encodings. It also allows for optional per-page compression.
Overall, the new proposal yields a footer that is comparable in size and
performance to the original proposal, but is simpler and more flexible for
handling future Parquet layout changes. We appreciate any feedback on the
proposal.
Best,
Jiayi
[1]
https://docs.google.com/document/d/1xN4Nqh-fh-eH-cUz9e6Rqs_gPXRBKUh4s29EzX_NU8M/edit?tab=t.o42wlmqbbdma#heading=h.kfs863ye4rzw
[2]
https://docs.google.com/document/d/1xN4Nqh-fh-eH-cUz9e6Rqs_gPXRBKUh4s29EzX_NU8M/edit?tab=t.cbl5qtgc5xhz
Jiayi Wang <wj...@gmail.com> 于2026年10月19日周一 20:49写道：
> Hi everyone,
>
>
> I'd like to share an update on the progress for the modular footer
> proposal [1]. We've received some valuable feedback and one recurring
> suggestion was to make the footer more future-proof by reusing Parquet
> encodings.
>
>
> We previously ruled out this approach because existing Parquet encodings
> don't support random access. However, with PFOR under development and
> potentially more random-access-friendly encodings coming in the future, we
> think it may be worth revisiting this idea.
>
>
> Another frequently raised point was compression, especially for cases
> where the footer becomes large enough that its size is a major bottleneck
> for cold reads. If we reuse Parquet pages to store footer metadata, we
> could also get compression support for free, potentially reducing footer
> size even further.
>
>
> At the same time, we want to make the modular footer flexible enough to
> accommodate future changes to the Parquet data layout. For example, the
> previously proposed non-contiguous pages layout [2] would require the
> optional offset index to be present, and we'd like to make its
> representation more efficient. Our goal is to design a footer that can
> support potential layout changes like this while still representing the
> current layout efficiently.
>
>
> We're still exploring these directions and will share updates as we learn
> more. In the meantime, please feel free to share any thoughts, concerns, or
> alternative ideas!
>
>
> Thanks again to everyone who reviewed the proposal and provided feedback!
>
>
> Best,
> Jiayi
>
>
> [1]
> https://docs.google.com/document/d/1xN4Nqh-fh-eH-cUz9e6Rqs_gPXRBKUh4s29EzX_NU8M/edit?tab=t.o42wlmqbbdma
>
> [2]
> https://docs.google.com/document/d/1nntcYM98PFSkHT70RexSBPtCnWqg1uRJ5_7m--ZgbsA/edit?tab=t.0#heading=h.k4r8orckhbx0
>
> Jiayi Wang <wj...@gmail.com> 于2026年9月21日周一 17:26写道：
>
>> Hi everyone,
>>
>> We’ve been discussing the Modular Footer for quite a while, including
>> through a dedicated footer working group meeting every two weeks, and we
>> finally have a proposal ready for broader review.
>>
>> The proposal covers the motivation, format design, compatibility
>> considerations, and benchmark results. Huge thanks to everyone who has
>> contributed to the project and the discussions so far.
>>
>> The proposal doc, parquet-format PR, and tracking issue are all available
>> here:
>>
>> Proposal doc:
>> https://docs.google.com/document/d/1xN4Nqh-fh-eH-cUz9e6Rqs_gPXRBKUh4s29EzX_NU8M/edit?usp=sharing
>>
>> PR: https://github.com/apache/parquet-format/pull/623/changes
>>
>> Parquet-format issue: https://github.com/apache/parquet-format/issues/622
>>
>> The proposal introduces several significant conceptual changes, so I
>> expect there will be a lot of discussion and feedback. For the first round
>> of review, I’d suggest focusing primarily on the high-level direction and
>> design choices, and leaving more detailed questions (for example, whether a
>> particular field can be deprecated) for later discussion. I think this will
>> help us converge on the major design questions first and keep the review
>> process moving efficiently.
>>
>> Thanks!
>>
>> Best,
>> Jiayi
>>
>
Jiayi Wang <wj...@gmail.com> 于2026年10月19日周一 20:49写道：
> Hi everyone,
>
>
> I'd like to share an update on the progress for the modular footer
> proposal [1]. We've received some valuable feedback and one recurring
> suggestion was to make the footer more future-proof by reusing Parquet
> encodings.
>
>
> We previously ruled out this approach because existing Parquet encodings
> don't support random access. However, with PFOR under development and
> potentially more random-access-friendly encodings coming in the future, we
> think it may be worth revisiting this idea.
>
>
> Another frequently raised point was compression, especially for cases
> where the footer becomes large enough that its size is a major bottleneck
> for cold reads. If we reuse Parquet pages to store footer metadata, we
> could also get compression support for free, potentially reducing footer
> size even further.
>
>
> At the same time, we want to make the modular footer flexible enough to
> accommodate future changes to the Parquet data layout. For example, the
> previously proposed non-contiguous pages layout [2] would require the
> optional offset index to be present, and we'd like to make its
> representation more efficient. Our goal is to design a footer that can
> support potential layout changes like this while still representing the
> current layout efficiently.
>
>
> We're still exploring these directions and will share updates as we learn
> more. In the meantime, please feel free to share any thoughts, concerns, or
> alternative ideas!
>
>
> Thanks again to everyone who reviewed the proposal and provided feedback!
>
>
> Best,
> Jiayi
>
>
> [1]
> https://docs.google.com/document/d/1xN4Nqh-fh-eH-cUz9e6Rqs_gPXRBKUh4s29EzX_NU8M/edit?tab=t.o42wlmqbbdma
>
> [2]
> https://docs.google.com/document/d/1nntcYM98PFSkHT70RexSBPtCnWqg1uRJ5_7m--ZgbsA/edit?tab=t.0#heading=h.k4r8orckhbx0
>
> Jiayi Wang <wj...@gmail.com> 于2026年9月21日周一 17:26写道：
>
>> Hi everyone,
>>
>> We’ve been discussing the Modular Footer for quite a while, including
>> through a dedicated footer working group meeting every two weeks, and we
>> finally have a proposal ready for broader review.
>>
>> The proposal covers the motivation, format design, compatibility
>> considerations, and benchmark results. Huge thanks to everyone who has
>> contributed to the project and the discussions so far.
>>
>> The proposal doc, parquet-format PR, and tracking issue are all available
>> here:
>>
>> Proposal doc:
>> https://docs.google.com/document/d/1xN4Nqh-fh-eH-cUz9e6Rqs_gPXRBKUh4s29EzX_NU8M/edit?usp=sharing
>>
>> PR: https://github.com/apache/parquet-format/pull/623/changes
>>
>> Parquet-format issue: https://github.com/apache/parquet-format/issues/622
>>
>> The proposal introduces several significant conceptual changes, so I
>> expect there will be a lot of discussion and feedback. For the first round
>> of review, I’d suggest focusing primarily on the high-level direction and
>> design choices, and leaving more detailed questions (for example, whether a
>> particular field can be deprecated) for later discussion. I think this will
>> help us converge on the major design questions first and keep the review
>> process moving efficiently.
>>
>> Thanks!
>>
>> Best,
>> Jiayi
>>
>
Re: Introducing the Parquet Modular Footer
Posted by Jiayi Wang <wj...@gmail.com>.
Hi everyone,
I'd like to share an update on the progress for the modular footer proposal
[1]. We've received some valuable feedback and one recurring suggestion was
to make the footer more future-proof by reusing Parquet encodings.
We previously ruled out this approach because existing Parquet encodings
don't support random access. However, with PFOR under development and
potentially more random-access-friendly encodings coming in the future, we
think it may be worth revisiting this idea.
Another frequently raised point was compression, especially for cases where
the footer becomes large enough that its size is a major bottleneck for
cold reads. If we reuse Parquet pages to store footer metadata, we could
also get compression support for free, potentially reducing footer size
even further.
At the same time, we want to make the modular footer flexible enough to
accommodate future changes to the Parquet data layout. For example, the
previously proposed non-contiguous pages layout [2] would require the
optional offset index to be present, and we'd like to make its
representation more efficient. Our goal is to design a footer that can
support potential layout changes like this while still representing the
current layout efficiently.
We're still exploring these directions and will share updates as we learn
more. In the meantime, please feel free to share any thoughts, concerns, or
alternative ideas!
Thanks again to everyone who reviewed the proposal and provided feedback!
Best,
Jiayi
[1]
https://docs.google.com/document/d/1xN4Nqh-fh-eH-cUz9e6Rqs_gPXRBKUh4s29EzX_NU8M/edit?tab=t.o42wlmqbbdma
[2]
https://docs.google.com/document/d/1nntcYM98PFSkHT70RexSBPtCnWqg1uRJ5_7m--ZgbsA/edit?tab=t.0#heading=h.k4r8orckhbx0
Jiayi Wang <wj...@gmail.com> 于2026年9月21日周一 17:26写道：
> Hi everyone,
>
> We’ve been discussing the Modular Footer for quite a while, including
> through a dedicated footer working group meeting every two weeks, and we
> finally have a proposal ready for broader review.
>
> The proposal covers the motivation, format design, compatibility
> considerations, and benchmark results. Huge thanks to everyone who has
> contributed to the project and the discussions so far.
>
> The proposal doc, parquet-format PR, and tracking issue are all available
> here:
>
> Proposal doc:
> https://docs.google.com/document/d/1xN4Nqh-fh-eH-cUz9e6Rqs_gPXRBKUh4s29EzX_NU8M/edit?usp=sharing
>
> PR: https://github.com/apache/parquet-format/pull/623/changes
>
> Parquet-format issue: https://github.com/apache/parquet-format/issues/622
>
> The proposal introduces several significant conceptual changes, so I
> expect there will be a lot of discussion and feedback. For the first round
> of review, I’d suggest focusing primarily on the high-level direction and
> design choices, and leaving more detailed questions (for example, whether a
> particular field can be deprecated) for later discussion. I think this will
> help us converge on the major design questions first and keep the review
> process moving efficiently.
>
> Thanks!
>
> Best,
> Jiayi
>