You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by Andrew Bell <an...@gmail.com> on 2025/10/14 20:50:23 UTC
Metadata Descriptions
Hi,
Is there a document that explains the metadata (
https://parquet.apache.org/docs/file-format/metadata) in English? I can
read code, but I'd rather not :) There seems to be some hand-wavy language
that defines certain bits, but I haven't found anything that defines each
field in the metadata or anything that really defines the format itself
other than the metadata picture and this:
https://parquet.apache.org/docs/file-format
Thanks,
--
Andrew Bell
andrew.bell.ia@gmail.com
Re: Metadata Descriptions
Posted by Andrew Bell <an...@gmail.com>.
Arnav,
On Tue, Oct 14, 2025 at 8:52 PM Arnav Balyan <ar...@gmail.com> wrote:
> Hey Andrew (Bell),
> I'm working on a cookbook and this would be a great addition. Let me know
> if you'd like to contribute a section on metadata descriptions.
> Will reach out once an initial draft is ready.
>
I would be happy to write something as I document for my own purposes, but
I'm not much for graphics or design. I can forward you some text and if you
want to enhance it (if appropriate), that would be great.
--
Andrew Bell
andrew.bell.ia@gmail.com
Re: Metadata Descriptions
Posted by Arnav Balyan <ar...@gmail.com>.
Hey Andrew (Bell),
I'm working on a cookbook and this would be a great addition. Let me know
if you'd like to contribute a section on metadata descriptions.
Will reach out once an initial draft is ready.
Thanks
Arnav
On Wed, Oct 15, 2025 at 2:55 AM Andrew Lamb <an...@gmail.com> wrote:
> For additional spec reading pleasure, the format of the parquet.thrift file
> is Thrift Interface Definition Language[1].
>
> Parquet metadata is stored using the binary Thrift Compact Protocol[2].
>
> [1]: https://github.com/apache/thrift/blob/master/doc/specs/idl.md
> [2]:
>
> https://github.com/apache/thrift/blob/master/doc/specs/thrift-compact-protocol.md
>
> On Tue, Oct 14, 2025 at 5:04 PM Sylvain Lesage <severo@rednegra.net.invalid
> >
> wrote:
>
> > Maybe the comments in the specification (
> >
> https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift
> )
> > are sufficiently clear?
> >
> >
> > On Tuesday, 14 October 2025 at 10:50 PM, Andrew Bell <
> > andrew.bell.ia@gmail.com> wrote:
> >
> > >
> > >
> > > Hi,
> > >
> > > Is there a document that explains the metadata (
> > > https://parquet.apache.org/docs/file-format/metadata) in English? I
> can
> > > read code, but I'd rather not :) There seems to be some hand-wavy
> > language
> > > that defines certain bits, but I haven't found anything that defines
> each
> > > field in the metadata or anything that really defines the format itself
> > > other than the metadata picture and this:
> > > https://parquet.apache.org/docs/file-format
> > >
> > > Thanks,
> > >
> > > --
> > > Andrew Bell
> > > andrew.bell.ia@gmail.com
> >
>
Re: Metadata Descriptions
Posted by Andrew Lamb <an...@gmail.com>.
For additional spec reading pleasure, the format of the parquet.thrift file
is Thrift Interface Definition Language[1].
Parquet metadata is stored using the binary Thrift Compact Protocol[2].
[1]: https://github.com/apache/thrift/blob/master/doc/specs/idl.md
[2]:
https://github.com/apache/thrift/blob/master/doc/specs/thrift-compact-protocol.md
On Tue, Oct 14, 2025 at 5:04 PM Sylvain Lesage <se...@rednegra.net.invalid>
wrote:
> Maybe the comments in the specification (
> https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift)
> are sufficiently clear?
>
>
> On Tuesday, 14 October 2025 at 10:50 PM, Andrew Bell <
> andrew.bell.ia@gmail.com> wrote:
>
> >
> >
> > Hi,
> >
> > Is there a document that explains the metadata (
> > https://parquet.apache.org/docs/file-format/metadata) in English? I can
> > read code, but I'd rather not :) There seems to be some hand-wavy
> language
> > that defines certain bits, but I haven't found anything that defines each
> > field in the metadata or anything that really defines the format itself
> > other than the metadata picture and this:
> > https://parquet.apache.org/docs/file-format
> >
> > Thanks,
> >
> > --
> > Andrew Bell
> > andrew.bell.ia@gmail.com
>
Re: Metadata Descriptions
Posted by Sylvain Lesage <se...@rednegra.net.INVALID>.
Maybe the comments in the specification (https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift) are sufficiently clear?
On Tuesday, 14 October 2025 at 10:50 PM, Andrew Bell <an...@gmail.com> wrote:
>
>
> Hi,
>
> Is there a document that explains the metadata (
> https://parquet.apache.org/docs/file-format/metadata) in English? I can
> read code, but I'd rather not :) There seems to be some hand-wavy language
> that defines certain bits, but I haven't found anything that defines each
> field in the metadata or anything that really defines the format itself
> other than the metadata picture and this:
> https://parquet.apache.org/docs/file-format
>
> Thanks,
>
> --
> Andrew Bell
> andrew.bell.ia@gmail.com