[DISCUSS] Finalizing the versioning proposal

[Julien Le Dem <ju...@apache.org>]

Following up on the discussion during the Parquet sync on Wednesday, I have
added a new tab to the Versioning document
<https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab=t.nywy9gjvpwqf>
to discuss 4 remaining decision points for the implementation. For each I
have listed options and pros and cons.
1. Mechanism to decide what Parquet features writers should use
2. Mechanism to write Parquet files that use a preview feature
3. Mechanism to record in the file what features it uses
4. Describing reader behavior when reading a file that is not fully
supported

The stated guiding principle is: We want to minimize prescribing how
readers and writers should work and focus on giving them the information to
implement the behavior they deem appropriate.

Please comment in the doc to help finalize consensus.
(thank you Andrew for early feedback and helping me think this through!)

Re: [DISCUSS] Finalizing the versioning proposal

[Ryan Blue <bl...@apache.org>]

> It seems you're referring to things that were discussed in private (or at
least not on this list)

Sorry for the confusion here. This was discussed during our bi-weekly
community sync discussion, and I took the action to summarize the
conversation for the dev list so we can continue here. I didn't mean to say
that anything was final, just to record what I thought the general
consensus was at the time.

> I would strongly recommend that this discussion be made email-first. Perhaps
the Parquet community needs more explicit guidelines around discussing spec
additions/changes, or perhaps even a better process altogether?

Regular face-to-face discussions have been an important part of the
community since before I started contributing. I think they are really
helpful. Maybe you have some ideas for how to make the transition between
those discussions and the mailing list more effective?

I would just ask that we start a separate thread for the feedback and
discussion about how to make decisions. Although the discussion didn't
entirely take place on the list, it's at least here now and it would help
to keep this thread focused.

Thanks,

Ryan

On Wed, Sep 2, 2026 at 3:42 AM Antoine Pitrou <an...@python.org> wrote:

>
> Hello Ryan,
>
> This is becoming very difficult to follow. It seems you're referring to
> things that were discussed in private (or at least not on this list). It
> also seems to be straying away from the original versioning discussion
> to something else (magic bytes? why?).
>
> I would strongly recommend that this discussion be made email-first.
> Perhaps the Parquet community needs more explicit guidelines around
> discussing spec additions/changes, or perhaps even a better process
> altogether?
>
> Regards
>
> Antoine.
>
>
> Le 01/09/2026 à 02:08, Ryan Blue a écrit :
> > I want to follow up on the open points here.
> >
> > First, on the open question about magic bytes, I've confirmed with the
> > folks proposing the modular footer that we think we can handle encryption
> > within the footer. The reason for using the magic bytes for this before
> > (PARE) was that we needed to encrypt the entire footer to not leak
> > unencrypted data values. But with the modular design, we can leave the
> > first module of the footer unencrypted because it just contains offsets
> > that aren't sensitive and store a flag to indicate that the rest of the
> > modules are encrypted.
> >
> > I think it still makes sense to use the magic bytes to encode the
> > Parquet format version, so that the entire footer content is covered by
> the
> > version. That avoids having two versions, one for the format and one for
> > the first footer module. I propose incrementing the last byte of the
> magic
> > bytes, starting with the next version we release. That is, if we release
> v3
> > next, we would use PAR3. v4 would be PAR4, etc. When we get to Parquet
> v10,
> > we could continue this pattern with 0x3A or skip to `PARA`... but this
> > decision would be a long time from now. In the meantime, we'd have one
> > version that is clearly visible from the magic bytes.
> >
> > Second, for the question about whether to list preview features, I think
> > that we _should_ list the preview feature names in the footer. Extra
> > information is rarely harmful. I think the strongest argument against
> > listing the preview features is that implementations can use this
> > information to fail without a best-effort read. I think that a strict or
> > pedantic mode is a bad idea, but I prefer that it is an implementation
> > decision. There is going to be plenty of pressure not to fail needlessly,
> > so we don't need to encourage it at a format level.
> >
> > Any other perspectives on these open issues?
> >
> > Ryan
> >
> > On Fri, Aug 28, 2026 at 2:32 PM Ryan Blue <bl...@apache.org> wrote:
> >
> >> Sorry, I initially sent this through the wrong account that gets dropped
> >> by ASF lists. Here's my summary from our conversation on Wednesday...
> >>
> >> We discussed the open questions that Julien summarized above in the sync
> >> and had agreement on a couple points, but there are also some open areas
> >> for further discussion.
> >>
> >> Broad agreement:
> >> - *For Q1 and Q2, we choose option B*. We want consistency across
> >> implementations, but cannot dictate behaviors like specific API methods
> or
> >> reader modes.
> >> - We will define consistent names or labels for preview features to help
> >> consistency
> >> - *For Q3, we eliminated option B*: We do not want to indicate preview
> >> features without saying what those preview features are
> >> - Q3 covers two separate questions: magic bytes and how to track preview
> >> features (see open questions)
> >> - *For Q4, we choose option C*: it is an implementation decision to have
> >> a strict mode
> >>
> >> Open questions:
> >> - Do we want to add the format version to the magic bytes, OR change
> magic
> >> bytes only when the footer encoding changes?
> >> - Do we want to list the preview features that are used in a file, OR do
> >> we omit them because we don't want to encourage unnecessary failures?
> >>
> >> Here's a more detailed summary:
> >>
> >> For 1 and 2, there was general consensus that we want to have
> consistency
> >> across implementations, but it doesn't make sense to mandate writer API
> >> options in a file format's spec that covers on-disk representation. We
> >> agreed that we want to define labels for preview features to help ensure
> >> consistency (and these may be stored in the format). For example, we
> could
> >> agree and document that "alp-encoding" means the new ALP encoding for
> float
> >> and double values, so that people can refer to the feature consistently,
> >> even if the API to enable features is idiomatic and slightly different
> >> across languages. We also clarified the requirement that preview
> features
> >> must cause readers that do not implement support to fail; readers must
> not
> >> produce incorrect results.
> >>
> >> Micah also suggested that we introduce minor versions (which must be
> >> forward-compatible within a version) and allow pinning to those minor
> >> versions. This would prevent using newer features even though they do
> not
> >> affect correctness. There wasn't an example when this would be useful
> and
> >> we concluded that this can be done later if we discover a use case.
> >>
> >> For 3, we discussed that there are two decision points. 3.1 is how the
> >> magic bytes relate to the format version, and 3.2 is whether we want the
> >> preview feature names included in files when they are used. (Option 2
> was
> >> discarded; it was for, essentially, a boolean flag to indicate preview
> >> features were used but did not specify which ones.)
> >>
> >> In the discussion about magic bytes (3.1), we outlined two options: we
> can
> >> encode the version in the magic bytes and update it each time the format
> >> version changes, or we can store the version in the footer and use a
> >> separate version in the magic bytes that is incremented when the footer
> >> representation changes ("PARA", "PARB", etc.). If we store the format
> >> version in the footer, we would need to stop using "PAR1" and "PARE"
> >> because we would want to be able to encode more than versions 3-9 and
> have
> >> to handle encryption.
> >>
> >> The discussion about preview features relates to question 4, which is
> >> whether we want to require readers to expose a "pedantic" mode (think:
> >> strict). Several people were advocating that readers should always
> attempt
> >> to read (best effort), and therefore we don't need to make changes to
> >> enable a strict mode like keeping a list of preview feature names. If
> >> implementations choose to have a pedantic mode, they can easily detect
> >> changes like unsupported encodings in columns that are not projected and
> >> fail with a good error message. We agreed that pedantic mode is a
> decision
> >> for the read implementation (outside the file format spec's scope), but
> did
> >> not conclude that we didn't want the list of preview features.
> >>
> >>
> >>
> >> On Fri, Aug 14, 2026 at 11:25 AM Julien Le Dem <ju...@apache.org>
> wrote:
> >>
> >>> Following up on the discussion during the Parquet sync on Wednesday, I
> >>> have
> >>> added a new tab to the Versioning document
> >>> <
> >>>
> https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab=t.nywy9gjvpwqf
> >>>>
> >>> to discuss 4 remaining decision points for the implementation. For
> each I
> >>> have listed options and pros and cons.
> >>> 1. Mechanism to decide what Parquet features writers should use
> >>> 2. Mechanism to write Parquet files that use a preview feature
> >>> 3. Mechanism to record in the file what features it uses
> >>> 4. Describing reader behavior when reading a file that is not fully
> >>> supported
> >>>
> >>> The stated guiding principle is: We want to minimize prescribing how
> >>> readers and writers should work and focus on giving them the
> information
> >>> to
> >>> implement the behavior they deem appropriate.
> >>>
> >>> Please comment in the doc to help finalize consensus.
> >>> (thank you Andrew for early feedback and helping me think this
> through!)
> >>>
> >>
> >
>
>
>

Re: [DISCUSS] Finalizing the versioning proposal

[Russell Spitzer <ru...@gmail.com>]

Antoine,

The Versioning doc Ryan is referring to is this one:

https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab=t.nywy9gjvpwqf

It was sent to this list on July 9 (
https://www.mail-archive.com/dev@parquet.apache.org/msg27611.html), and
what we're discussing now is the latest tab.

Magic bytes are item 3 in that tab, "Mechanism to record in the file what
features it uses," Option D. So this isn't a new topic and it isn't one
from a private channel.

I agree that decisions need to happen on this list which is why I
appreciate Ryan's email. He summarized the public weekly sync and posted it
here so that people who weren't in the sync can weigh in, explicitly
calling out the open questions as still open. Nothing has been decided and
folks should feel free to re-open questions on other things on the
documents if they like although I would urge folks to try to find consensus
when possible.

I'd rather keep this thread focused on the design and try to stay away from
meta-discussions about process. If there is anything we need to discuss
from the document we should probably bring it up here and go into more
depth.

On Wed, Sep 2, 2026 at 10:58 AM Russell Spitzer <ru...@gmail.com>
wrote:

> Micah,
>
> I do not think 1 and 2 are contentious; they follow from the vote.
>
> 1. A file that uses a preview feature must be readable by any other reader
> that implements that preview, and by readers of the major version that
> later adopts it.
>
> 2. A reader that does not support a preview feature must not return
> incorrect data. If existing readers would silently misread it, it is not a
> preview feature; it belongs in the next major version.
>
> I would draw 3 and 4 a bit differently.
>
> 3. If a reader cannot parse the Thrift footer, it does not support that
> version of Parquet. For everything else (preview features), a detailed
> error is fine.
>
> 4. This mixes version and preview. A V2 reader should fail fast on a file
> labeled V3; we cannot guarantee a correct read even if it undrestands some
> of the structure. A V2 reader should be able to safely try to read a V2
> file that uses preview features it does not implement. It should only fail
> if it reaches a feature it does not understand and cannot skip (Example -
> Encoding, Logical Type, ...)
>
> Russ
>
>
>
>
> On Wed, Sep 2, 2026 at 5:42 AM Antoine Pitrou <an...@python.org> wrote:
>
>>
>> Hello Ryan,
>>
>> This is becoming very difficult to follow. It seems you're referring to
>> things that were discussed in private (or at least not on this list). It
>> also seems to be straying away from the original versioning discussion
>> to something else (magic bytes? why?).
>>
>> I would strongly recommend that this discussion be made email-first.
>> Perhaps the Parquet community needs more explicit guidelines around
>> discussing spec additions/changes, or perhaps even a better process
>> altogether?
>>
>> Regards
>>
>> Antoine.
>>
>>
>> Le 01/09/2026 à 02:08, Ryan Blue a écrit :
>> > I want to follow up on the open points here.
>> >
>> > First, on the open question about magic bytes, I've confirmed with the
>> > folks proposing the modular footer that we think we can handle
>> encryption
>> > within the footer. The reason for using the magic bytes for this before
>> > (PARE) was that we needed to encrypt the entire footer to not leak
>> > unencrypted data values. But with the modular design, we can leave the
>> > first module of the footer unencrypted because it just contains offsets
>> > that aren't sensitive and store a flag to indicate that the rest of the
>> > modules are encrypted.
>> >
>> > I think it still makes sense to use the magic bytes to encode the
>> > Parquet format version, so that the entire footer content is covered by
>> the
>> > version. That avoids having two versions, one for the format and one for
>> > the first footer module. I propose incrementing the last byte of the
>> magic
>> > bytes, starting with the next version we release. That is, if we
>> release v3
>> > next, we would use PAR3. v4 would be PAR4, etc. When we get to Parquet
>> v10,
>> > we could continue this pattern with 0x3A or skip to `PARA`... but this
>> > decision would be a long time from now. In the meantime, we'd have one
>> > version that is clearly visible from the magic bytes.
>> >
>> > Second, for the question about whether to list preview features, I think
>> > that we _should_ list the preview feature names in the footer. Extra
>> > information is rarely harmful. I think the strongest argument against
>> > listing the preview features is that implementations can use this
>> > information to fail without a best-effort read. I think that a strict or
>> > pedantic mode is a bad idea, but I prefer that it is an implementation
>> > decision. There is going to be plenty of pressure not to fail
>> needlessly,
>> > so we don't need to encourage it at a format level.
>> >
>> > Any other perspectives on these open issues?
>> >
>> > Ryan
>> >
>> > On Fri, Aug 28, 2026 at 2:32 PM Ryan Blue <bl...@apache.org> wrote:
>> >
>> >> Sorry, I initially sent this through the wrong account that gets
>> dropped
>> >> by ASF lists. Here's my summary from our conversation on Wednesday...
>> >>
>> >> We discussed the open questions that Julien summarized above in the
>> sync
>> >> and had agreement on a couple points, but there are also some open
>> areas
>> >> for further discussion.
>> >>
>> >> Broad agreement:
>> >> - *For Q1 and Q2, we choose option B*. We want consistency across
>> >> implementations, but cannot dictate behaviors like specific API
>> methods or
>> >> reader modes.
>> >> - We will define consistent names or labels for preview features to
>> help
>> >> consistency
>> >> - *For Q3, we eliminated option B*: We do not want to indicate preview
>> >> features without saying what those preview features are
>> >> - Q3 covers two separate questions: magic bytes and how to track
>> preview
>> >> features (see open questions)
>> >> - *For Q4, we choose option C*: it is an implementation decision to
>> have
>> >> a strict mode
>> >>
>> >> Open questions:
>> >> - Do we want to add the format version to the magic bytes, OR change
>> magic
>> >> bytes only when the footer encoding changes?
>> >> - Do we want to list the preview features that are used in a file, OR
>> do
>> >> we omit them because we don't want to encourage unnecessary failures?
>> >>
>> >> Here's a more detailed summary:
>> >>
>> >> For 1 and 2, there was general consensus that we want to have
>> consistency
>> >> across implementations, but it doesn't make sense to mandate writer API
>> >> options in a file format's spec that covers on-disk representation. We
>> >> agreed that we want to define labels for preview features to help
>> ensure
>> >> consistency (and these may be stored in the format). For example, we
>> could
>> >> agree and document that "alp-encoding" means the new ALP encoding for
>> float
>> >> and double values, so that people can refer to the feature
>> consistently,
>> >> even if the API to enable features is idiomatic and slightly different
>> >> across languages. We also clarified the requirement that preview
>> features
>> >> must cause readers that do not implement support to fail; readers must
>> not
>> >> produce incorrect results.
>> >>
>> >> Micah also suggested that we introduce minor versions (which must be
>> >> forward-compatible within a version) and allow pinning to those minor
>> >> versions. This would prevent using newer features even though they do
>> not
>> >> affect correctness. There wasn't an example when this would be useful
>> and
>> >> we concluded that this can be done later if we discover a use case.
>> >>
>> >> For 3, we discussed that there are two decision points. 3.1 is how the
>> >> magic bytes relate to the format version, and 3.2 is whether we want
>> the
>> >> preview feature names included in files when they are used. (Option 2
>> was
>> >> discarded; it was for, essentially, a boolean flag to indicate preview
>> >> features were used but did not specify which ones.)
>> >>
>> >> In the discussion about magic bytes (3.1), we outlined two options: we
>> can
>> >> encode the version in the magic bytes and update it each time the
>> format
>> >> version changes, or we can store the version in the footer and use a
>> >> separate version in the magic bytes that is incremented when the footer
>> >> representation changes ("PARA", "PARB", etc.). If we store the format
>> >> version in the footer, we would need to stop using "PAR1" and "PARE"
>> >> because we would want to be able to encode more than versions 3-9 and
>> have
>> >> to handle encryption.
>> >>
>> >> The discussion about preview features relates to question 4, which is
>> >> whether we want to require readers to expose a "pedantic" mode (think:
>> >> strict). Several people were advocating that readers should always
>> attempt
>> >> to read (best effort), and therefore we don't need to make changes to
>> >> enable a strict mode like keeping a list of preview feature names. If
>> >> implementations choose to have a pedantic mode, they can easily detect
>> >> changes like unsupported encodings in columns that are not projected
>> and
>> >> fail with a good error message. We agreed that pedantic mode is a
>> decision
>> >> for the read implementation (outside the file format spec's scope),
>> but did
>> >> not conclude that we didn't want the list of preview features.
>> >>
>> >>
>> >>
>> >> On Fri, Aug 14, 2026 at 11:25 AM Julien Le Dem <ju...@apache.org>
>> wrote:
>> >>
>> >>> Following up on the discussion during the Parquet sync on Wednesday, I
>> >>> have
>> >>> added a new tab to the Versioning document
>> >>> <
>> >>>
>> https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab=t.nywy9gjvpwqf
>> >>>>
>> >>> to discuss 4 remaining decision points for the implementation. For
>> each I
>> >>> have listed options and pros and cons.
>> >>> 1. Mechanism to decide what Parquet features writers should use
>> >>> 2. Mechanism to write Parquet files that use a preview feature
>> >>> 3. Mechanism to record in the file what features it uses
>> >>> 4. Describing reader behavior when reading a file that is not fully
>> >>> supported
>> >>>
>> >>> The stated guiding principle is: We want to minimize prescribing how
>> >>> readers and writers should work and focus on giving them the
>> information
>> >>> to
>> >>> implement the behavior they deem appropriate.
>> >>>
>> >>> Please comment in the doc to help finalize consensus.
>> >>> (thank you Andrew for early feedback and helping me think this
>> through!)
>> >>>
>> >>
>> >
>>
>>
>>

Re: [DISCUSS] Finalizing the versioning proposal

[Russell Spitzer <ru...@gmail.com>]

Antoine,

The Versioning doc Ryan is referring to is this one:

https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab=t.nywy9gjvpwqf

It was sent to this list on July 9 (
https://www.mail-archive.com/dev@parquet.apache.org/msg27611.html), and
what we're discussing now is the latest tab.

Magic bytes are item 3 in that tab, "Mechanism to record in the file what
features it uses," Option D. So this isn't a new topic and it isn't one
from a private channel.

I agree that decisions need to happen on this list which is why I
appreciate Ryan's email. He summarized the public weekly sync and posted it
here so that people who weren't in the sync can weigh in, explicitly
calling out the open questions as still open. Nothing has been decided and
folks should feel free to re-open questions on other things on the
documents if they like although I would urge folks to try to find consensus
when possible.

I'd rather keep this thread focused on the design and try to stay away from
meta-discussions about process. If there is anything we need to discuss
from the document we should probably bring it up here and go into more
depth.

On Wed, Sep 2, 2026 at 10:58 AM Russell Spitzer <ru...@gmail.com>
wrote:

> Micah,
>
> I do not think 1 and 2 are contentious; they follow from the vote.
>
> 1. A file that uses a preview feature must be readable by any other reader
> that implements that preview, and by readers of the major version that
> later adopts it.
>
> 2. A reader that does not support a preview feature must not return
> incorrect data. If existing readers would silently misread it, it is not a
> preview feature; it belongs in the next major version.
>
> I would draw 3 and 4 a bit differently.
>
> 3. If a reader cannot parse the Thrift footer, it does not support that
> version of Parquet. For everything else (preview features), a detailed
> error is fine.
>
> 4. This mixes version and preview. A V2 reader should fail fast on a file
> labeled V3; we cannot guarantee a correct read even if it undrestands some
> of the structure. A V2 reader should be able to safely try to read a V2
> file that uses preview features it does not implement. It should only fail
> if it reaches a feature it does not understand and cannot skip (Example -
> Encoding, Logical Type, ...)
>
> Russ
>
>
>
>
> On Wed, Sep 2, 2026 at 5:42 AM Antoine Pitrou <an...@python.org> wrote:
>
>>
>> Hello Ryan,
>>
>> This is becoming very difficult to follow. It seems you're referring to
>> things that were discussed in private (or at least not on this list). It
>> also seems to be straying away from the original versioning discussion
>> to something else (magic bytes? why?).
>>
>> I would strongly recommend that this discussion be made email-first.
>> Perhaps the Parquet community needs more explicit guidelines around
>> discussing spec additions/changes, or perhaps even a better process
>> altogether?
>>
>> Regards
>>
>> Antoine.
>>
>>
>> Le 01/09/2026 à 02:08, Ryan Blue a écrit :
>> > I want to follow up on the open points here.
>> >
>> > First, on the open question about magic bytes, I've confirmed with the
>> > folks proposing the modular footer that we think we can handle
>> encryption
>> > within the footer. The reason for using the magic bytes for this before
>> > (PARE) was that we needed to encrypt the entire footer to not leak
>> > unencrypted data values. But with the modular design, we can leave the
>> > first module of the footer unencrypted because it just contains offsets
>> > that aren't sensitive and store a flag to indicate that the rest of the
>> > modules are encrypted.
>> >
>> > I think it still makes sense to use the magic bytes to encode the
>> > Parquet format version, so that the entire footer content is covered by
>> the
>> > version. That avoids having two versions, one for the format and one for
>> > the first footer module. I propose incrementing the last byte of the
>> magic
>> > bytes, starting with the next version we release. That is, if we
>> release v3
>> > next, we would use PAR3. v4 would be PAR4, etc. When we get to Parquet
>> v10,
>> > we could continue this pattern with 0x3A or skip to `PARA`... but this
>> > decision would be a long time from now. In the meantime, we'd have one
>> > version that is clearly visible from the magic bytes.
>> >
>> > Second, for the question about whether to list preview features, I think
>> > that we _should_ list the preview feature names in the footer. Extra
>> > information is rarely harmful. I think the strongest argument against
>> > listing the preview features is that implementations can use this
>> > information to fail without a best-effort read. I think that a strict or
>> > pedantic mode is a bad idea, but I prefer that it is an implementation
>> > decision. There is going to be plenty of pressure not to fail
>> needlessly,
>> > so we don't need to encourage it at a format level.
>> >
>> > Any other perspectives on these open issues?
>> >
>> > Ryan
>> >
>> > On Fri, Aug 28, 2026 at 2:32 PM Ryan Blue <bl...@apache.org> wrote:
>> >
>> >> Sorry, I initially sent this through the wrong account that gets
>> dropped
>> >> by ASF lists. Here's my summary from our conversation on Wednesday...
>> >>
>> >> We discussed the open questions that Julien summarized above in the
>> sync
>> >> and had agreement on a couple points, but there are also some open
>> areas
>> >> for further discussion.
>> >>
>> >> Broad agreement:
>> >> - *For Q1 and Q2, we choose option B*. We want consistency across
>> >> implementations, but cannot dictate behaviors like specific API
>> methods or
>> >> reader modes.
>> >> - We will define consistent names or labels for preview features to
>> help
>> >> consistency
>> >> - *For Q3, we eliminated option B*: We do not want to indicate preview
>> >> features without saying what those preview features are
>> >> - Q3 covers two separate questions: magic bytes and how to track
>> preview
>> >> features (see open questions)
>> >> - *For Q4, we choose option C*: it is an implementation decision to
>> have
>> >> a strict mode
>> >>
>> >> Open questions:
>> >> - Do we want to add the format version to the magic bytes, OR change
>> magic
>> >> bytes only when the footer encoding changes?
>> >> - Do we want to list the preview features that are used in a file, OR
>> do
>> >> we omit them because we don't want to encourage unnecessary failures?
>> >>
>> >> Here's a more detailed summary:
>> >>
>> >> For 1 and 2, there was general consensus that we want to have
>> consistency
>> >> across implementations, but it doesn't make sense to mandate writer API
>> >> options in a file format's spec that covers on-disk representation. We
>> >> agreed that we want to define labels for preview features to help
>> ensure
>> >> consistency (and these may be stored in the format). For example, we
>> could
>> >> agree and document that "alp-encoding" means the new ALP encoding for
>> float
>> >> and double values, so that people can refer to the feature
>> consistently,
>> >> even if the API to enable features is idiomatic and slightly different
>> >> across languages. We also clarified the requirement that preview
>> features
>> >> must cause readers that do not implement support to fail; readers must
>> not
>> >> produce incorrect results.
>> >>
>> >> Micah also suggested that we introduce minor versions (which must be
>> >> forward-compatible within a version) and allow pinning to those minor
>> >> versions. This would prevent using newer features even though they do
>> not
>> >> affect correctness. There wasn't an example when this would be useful
>> and
>> >> we concluded that this can be done later if we discover a use case.
>> >>
>> >> For 3, we discussed that there are two decision points. 3.1 is how the
>> >> magic bytes relate to the format version, and 3.2 is whether we want
>> the
>> >> preview feature names included in files when they are used. (Option 2
>> was
>> >> discarded; it was for, essentially, a boolean flag to indicate preview
>> >> features were used but did not specify which ones.)
>> >>
>> >> In the discussion about magic bytes (3.1), we outlined two options: we
>> can
>> >> encode the version in the magic bytes and update it each time the
>> format
>> >> version changes, or we can store the version in the footer and use a
>> >> separate version in the magic bytes that is incremented when the footer
>> >> representation changes ("PARA", "PARB", etc.). If we store the format
>> >> version in the footer, we would need to stop using "PAR1" and "PARE"
>> >> because we would want to be able to encode more than versions 3-9 and
>> have
>> >> to handle encryption.
>> >>
>> >> The discussion about preview features relates to question 4, which is
>> >> whether we want to require readers to expose a "pedantic" mode (think:
>> >> strict). Several people were advocating that readers should always
>> attempt
>> >> to read (best effort), and therefore we don't need to make changes to
>> >> enable a strict mode like keeping a list of preview feature names. If
>> >> implementations choose to have a pedantic mode, they can easily detect
>> >> changes like unsupported encodings in columns that are not projected
>> and
>> >> fail with a good error message. We agreed that pedantic mode is a
>> decision
>> >> for the read implementation (outside the file format spec's scope),
>> but did
>> >> not conclude that we didn't want the list of preview features.
>> >>
>> >>
>> >>
>> >> On Fri, Aug 14, 2026 at 11:25 AM Julien Le Dem <ju...@apache.org>
>> wrote:
>> >>
>> >>> Following up on the discussion during the Parquet sync on Wednesday, I
>> >>> have
>> >>> added a new tab to the Versioning document
>> >>> <
>> >>>
>> https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab=t.nywy9gjvpwqf
>> >>>>
>> >>> to discuss 4 remaining decision points for the implementation. For
>> each I
>> >>> have listed options and pros and cons.
>> >>> 1. Mechanism to decide what Parquet features writers should use
>> >>> 2. Mechanism to write Parquet files that use a preview feature
>> >>> 3. Mechanism to record in the file what features it uses
>> >>> 4. Describing reader behavior when reading a file that is not fully
>> >>> supported
>> >>>
>> >>> The stated guiding principle is: We want to minimize prescribing how
>> >>> readers and writers should work and focus on giving them the
>> information
>> >>> to
>> >>> implement the behavior they deem appropriate.
>> >>>
>> >>> Please comment in the doc to help finalize consensus.
>> >>> (thank you Andrew for early feedback and helping me think this
>> through!)
>> >>>
>> >>
>> >
>>
>>
>>

Re: [DISCUSS] Finalizing the versioning proposal

[Russell Spitzer <ru...@gmail.com>]

Micah,

I do not think 1 and 2 are contentious; they follow from the vote.

1. A file that uses a preview feature must be readable by any other reader
that implements that preview, and by readers of the major version that
later adopts it.

2. A reader that does not support a preview feature must not return
incorrect data. If existing readers would silently misread it, it is not a
preview feature; it belongs in the next major version.

I would draw 3 and 4 a bit differently.

3. If a reader cannot parse the Thrift footer, it does not support that
version of Parquet. For everything else (preview features), a detailed
error is fine.

4. This mixes version and preview. A V2 reader should fail fast on a file
labeled V3; we cannot guarantee a correct read even if it undrestands some
of the structure. A V2 reader should be able to safely try to read a V2
file that uses preview features it does not implement. It should only fail
if it reaches a feature it does not understand and cannot skip (Example -
Encoding, Logical Type, ...)

Russ




On Wed, Sep 2, 2026 at 5:42 AM Antoine Pitrou <an...@python.org> wrote:

>
> Hello Ryan,
>
> This is becoming very difficult to follow. It seems you're referring to
> things that were discussed in private (or at least not on this list). It
> also seems to be straying away from the original versioning discussion
> to something else (magic bytes? why?).
>
> I would strongly recommend that this discussion be made email-first.
> Perhaps the Parquet community needs more explicit guidelines around
> discussing spec additions/changes, or perhaps even a better process
> altogether?
>
> Regards
>
> Antoine.
>
>
> Le 01/09/2026 à 02:08, Ryan Blue a écrit :
> > I want to follow up on the open points here.
> >
> > First, on the open question about magic bytes, I've confirmed with the
> > folks proposing the modular footer that we think we can handle encryption
> > within the footer. The reason for using the magic bytes for this before
> > (PARE) was that we needed to encrypt the entire footer to not leak
> > unencrypted data values. But with the modular design, we can leave the
> > first module of the footer unencrypted because it just contains offsets
> > that aren't sensitive and store a flag to indicate that the rest of the
> > modules are encrypted.
> >
> > I think it still makes sense to use the magic bytes to encode the
> > Parquet format version, so that the entire footer content is covered by
> the
> > version. That avoids having two versions, one for the format and one for
> > the first footer module. I propose incrementing the last byte of the
> magic
> > bytes, starting with the next version we release. That is, if we release
> v3
> > next, we would use PAR3. v4 would be PAR4, etc. When we get to Parquet
> v10,
> > we could continue this pattern with 0x3A or skip to `PARA`... but this
> > decision would be a long time from now. In the meantime, we'd have one
> > version that is clearly visible from the magic bytes.
> >
> > Second, for the question about whether to list preview features, I think
> > that we _should_ list the preview feature names in the footer. Extra
> > information is rarely harmful. I think the strongest argument against
> > listing the preview features is that implementations can use this
> > information to fail without a best-effort read. I think that a strict or
> > pedantic mode is a bad idea, but I prefer that it is an implementation
> > decision. There is going to be plenty of pressure not to fail needlessly,
> > so we don't need to encourage it at a format level.
> >
> > Any other perspectives on these open issues?
> >
> > Ryan
> >
> > On Fri, Aug 28, 2026 at 2:32 PM Ryan Blue <bl...@apache.org> wrote:
> >
> >> Sorry, I initially sent this through the wrong account that gets dropped
> >> by ASF lists. Here's my summary from our conversation on Wednesday...
> >>
> >> We discussed the open questions that Julien summarized above in the sync
> >> and had agreement on a couple points, but there are also some open areas
> >> for further discussion.
> >>
> >> Broad agreement:
> >> - *For Q1 and Q2, we choose option B*. We want consistency across
> >> implementations, but cannot dictate behaviors like specific API methods
> or
> >> reader modes.
> >> - We will define consistent names or labels for preview features to help
> >> consistency
> >> - *For Q3, we eliminated option B*: We do not want to indicate preview
> >> features without saying what those preview features are
> >> - Q3 covers two separate questions: magic bytes and how to track preview
> >> features (see open questions)
> >> - *For Q4, we choose option C*: it is an implementation decision to have
> >> a strict mode
> >>
> >> Open questions:
> >> - Do we want to add the format version to the magic bytes, OR change
> magic
> >> bytes only when the footer encoding changes?
> >> - Do we want to list the preview features that are used in a file, OR do
> >> we omit them because we don't want to encourage unnecessary failures?
> >>
> >> Here's a more detailed summary:
> >>
> >> For 1 and 2, there was general consensus that we want to have
> consistency
> >> across implementations, but it doesn't make sense to mandate writer API
> >> options in a file format's spec that covers on-disk representation. We
> >> agreed that we want to define labels for preview features to help ensure
> >> consistency (and these may be stored in the format). For example, we
> could
> >> agree and document that "alp-encoding" means the new ALP encoding for
> float
> >> and double values, so that people can refer to the feature consistently,
> >> even if the API to enable features is idiomatic and slightly different
> >> across languages. We also clarified the requirement that preview
> features
> >> must cause readers that do not implement support to fail; readers must
> not
> >> produce incorrect results.
> >>
> >> Micah also suggested that we introduce minor versions (which must be
> >> forward-compatible within a version) and allow pinning to those minor
> >> versions. This would prevent using newer features even though they do
> not
> >> affect correctness. There wasn't an example when this would be useful
> and
> >> we concluded that this can be done later if we discover a use case.
> >>
> >> For 3, we discussed that there are two decision points. 3.1 is how the
> >> magic bytes relate to the format version, and 3.2 is whether we want the
> >> preview feature names included in files when they are used. (Option 2
> was
> >> discarded; it was for, essentially, a boolean flag to indicate preview
> >> features were used but did not specify which ones.)
> >>
> >> In the discussion about magic bytes (3.1), we outlined two options: we
> can
> >> encode the version in the magic bytes and update it each time the format
> >> version changes, or we can store the version in the footer and use a
> >> separate version in the magic bytes that is incremented when the footer
> >> representation changes ("PARA", "PARB", etc.). If we store the format
> >> version in the footer, we would need to stop using "PAR1" and "PARE"
> >> because we would want to be able to encode more than versions 3-9 and
> have
> >> to handle encryption.
> >>
> >> The discussion about preview features relates to question 4, which is
> >> whether we want to require readers to expose a "pedantic" mode (think:
> >> strict). Several people were advocating that readers should always
> attempt
> >> to read (best effort), and therefore we don't need to make changes to
> >> enable a strict mode like keeping a list of preview feature names. If
> >> implementations choose to have a pedantic mode, they can easily detect
> >> changes like unsupported encodings in columns that are not projected and
> >> fail with a good error message. We agreed that pedantic mode is a
> decision
> >> for the read implementation (outside the file format spec's scope), but
> did
> >> not conclude that we didn't want the list of preview features.
> >>
> >>
> >>
> >> On Fri, Aug 14, 2026 at 11:25 AM Julien Le Dem <ju...@apache.org>
> wrote:
> >>
> >>> Following up on the discussion during the Parquet sync on Wednesday, I
> >>> have
> >>> added a new tab to the Versioning document
> >>> <
> >>>
> https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab=t.nywy9gjvpwqf
> >>>>
> >>> to discuss 4 remaining decision points for the implementation. For
> each I
> >>> have listed options and pros and cons.
> >>> 1. Mechanism to decide what Parquet features writers should use
> >>> 2. Mechanism to write Parquet files that use a preview feature
> >>> 3. Mechanism to record in the file what features it uses
> >>> 4. Describing reader behavior when reading a file that is not fully
> >>> supported
> >>>
> >>> The stated guiding principle is: We want to minimize prescribing how
> >>> readers and writers should work and focus on giving them the
> information
> >>> to
> >>> implement the behavior they deem appropriate.
> >>>
> >>> Please comment in the doc to help finalize consensus.
> >>> (thank you Andrew for early feedback and helping me think this
> through!)
> >>>
> >>
> >
>
>
>

Re: [DISCUSS] Finalizing the versioning proposal

[Russell Spitzer <ru...@gmail.com>]

Micah,

I do not think 1 and 2 are contentious; they follow from the vote.

1. A file that uses a preview feature must be readable by any other reader
that implements that preview, and by readers of the major version that
later adopts it.

2. A reader that does not support a preview feature must not return
incorrect data. If existing readers would silently misread it, it is not a
preview feature; it belongs in the next major version.

I would draw 3 and 4 a bit differently.

3. If a reader cannot parse the Thrift footer, it does not support that
version of Parquet. For everything else (preview features), a detailed
error is fine.

4. This mixes version and preview. A V2 reader should fail fast on a file
labeled V3; we cannot guarantee a correct read even if it undrestands some
of the structure. A V2 reader should be able to safely try to read a V2
file that uses preview features it does not implement. It should only fail
if it reaches a feature it does not understand and cannot skip (Example -
Encoding, Logical Type, ...)

Russ




On Wed, Sep 2, 2026 at 5:42 AM Antoine Pitrou <an...@python.org> wrote:

>
> Hello Ryan,
>
> This is becoming very difficult to follow. It seems you're referring to
> things that were discussed in private (or at least not on this list). It
> also seems to be straying away from the original versioning discussion
> to something else (magic bytes? why?).
>
> I would strongly recommend that this discussion be made email-first.
> Perhaps the Parquet community needs more explicit guidelines around
> discussing spec additions/changes, or perhaps even a better process
> altogether?
>
> Regards
>
> Antoine.
>
>
> Le 01/09/2026 à 02:08, Ryan Blue a écrit :
> > I want to follow up on the open points here.
> >
> > First, on the open question about magic bytes, I've confirmed with the
> > folks proposing the modular footer that we think we can handle encryption
> > within the footer. The reason for using the magic bytes for this before
> > (PARE) was that we needed to encrypt the entire footer to not leak
> > unencrypted data values. But with the modular design, we can leave the
> > first module of the footer unencrypted because it just contains offsets
> > that aren't sensitive and store a flag to indicate that the rest of the
> > modules are encrypted.
> >
> > I think it still makes sense to use the magic bytes to encode the
> > Parquet format version, so that the entire footer content is covered by
> the
> > version. That avoids having two versions, one for the format and one for
> > the first footer module. I propose incrementing the last byte of the
> magic
> > bytes, starting with the next version we release. That is, if we release
> v3
> > next, we would use PAR3. v4 would be PAR4, etc. When we get to Parquet
> v10,
> > we could continue this pattern with 0x3A or skip to `PARA`... but this
> > decision would be a long time from now. In the meantime, we'd have one
> > version that is clearly visible from the magic bytes.
> >
> > Second, for the question about whether to list preview features, I think
> > that we _should_ list the preview feature names in the footer. Extra
> > information is rarely harmful. I think the strongest argument against
> > listing the preview features is that implementations can use this
> > information to fail without a best-effort read. I think that a strict or
> > pedantic mode is a bad idea, but I prefer that it is an implementation
> > decision. There is going to be plenty of pressure not to fail needlessly,
> > so we don't need to encourage it at a format level.
> >
> > Any other perspectives on these open issues?
> >
> > Ryan
> >
> > On Fri, Aug 28, 2026 at 2:32 PM Ryan Blue <bl...@apache.org> wrote:
> >
> >> Sorry, I initially sent this through the wrong account that gets dropped
> >> by ASF lists. Here's my summary from our conversation on Wednesday...
> >>
> >> We discussed the open questions that Julien summarized above in the sync
> >> and had agreement on a couple points, but there are also some open areas
> >> for further discussion.
> >>
> >> Broad agreement:
> >> - *For Q1 and Q2, we choose option B*. We want consistency across
> >> implementations, but cannot dictate behaviors like specific API methods
> or
> >> reader modes.
> >> - We will define consistent names or labels for preview features to help
> >> consistency
> >> - *For Q3, we eliminated option B*: We do not want to indicate preview
> >> features without saying what those preview features are
> >> - Q3 covers two separate questions: magic bytes and how to track preview
> >> features (see open questions)
> >> - *For Q4, we choose option C*: it is an implementation decision to have
> >> a strict mode
> >>
> >> Open questions:
> >> - Do we want to add the format version to the magic bytes, OR change
> magic
> >> bytes only when the footer encoding changes?
> >> - Do we want to list the preview features that are used in a file, OR do
> >> we omit them because we don't want to encourage unnecessary failures?
> >>
> >> Here's a more detailed summary:
> >>
> >> For 1 and 2, there was general consensus that we want to have
> consistency
> >> across implementations, but it doesn't make sense to mandate writer API
> >> options in a file format's spec that covers on-disk representation. We
> >> agreed that we want to define labels for preview features to help ensure
> >> consistency (and these may be stored in the format). For example, we
> could
> >> agree and document that "alp-encoding" means the new ALP encoding for
> float
> >> and double values, so that people can refer to the feature consistently,
> >> even if the API to enable features is idiomatic and slightly different
> >> across languages. We also clarified the requirement that preview
> features
> >> must cause readers that do not implement support to fail; readers must
> not
> >> produce incorrect results.
> >>
> >> Micah also suggested that we introduce minor versions (which must be
> >> forward-compatible within a version) and allow pinning to those minor
> >> versions. This would prevent using newer features even though they do
> not
> >> affect correctness. There wasn't an example when this would be useful
> and
> >> we concluded that this can be done later if we discover a use case.
> >>
> >> For 3, we discussed that there are two decision points. 3.1 is how the
> >> magic bytes relate to the format version, and 3.2 is whether we want the
> >> preview feature names included in files when they are used. (Option 2
> was
> >> discarded; it was for, essentially, a boolean flag to indicate preview
> >> features were used but did not specify which ones.)
> >>
> >> In the discussion about magic bytes (3.1), we outlined two options: we
> can
> >> encode the version in the magic bytes and update it each time the format
> >> version changes, or we can store the version in the footer and use a
> >> separate version in the magic bytes that is incremented when the footer
> >> representation changes ("PARA", "PARB", etc.). If we store the format
> >> version in the footer, we would need to stop using "PAR1" and "PARE"
> >> because we would want to be able to encode more than versions 3-9 and
> have
> >> to handle encryption.
> >>
> >> The discussion about preview features relates to question 4, which is
> >> whether we want to require readers to expose a "pedantic" mode (think:
> >> strict). Several people were advocating that readers should always
> attempt
> >> to read (best effort), and therefore we don't need to make changes to
> >> enable a strict mode like keeping a list of preview feature names. If
> >> implementations choose to have a pedantic mode, they can easily detect
> >> changes like unsupported encodings in columns that are not projected and
> >> fail with a good error message. We agreed that pedantic mode is a
> decision
> >> for the read implementation (outside the file format spec's scope), but
> did
> >> not conclude that we didn't want the list of preview features.
> >>
> >>
> >>
> >> On Fri, Aug 14, 2026 at 11:25 AM Julien Le Dem <ju...@apache.org>
> wrote:
> >>
> >>> Following up on the discussion during the Parquet sync on Wednesday, I
> >>> have
> >>> added a new tab to the Versioning document
> >>> <
> >>>
> https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab=t.nywy9gjvpwqf
> >>>>
> >>> to discuss 4 remaining decision points for the implementation. For
> each I
> >>> have listed options and pros and cons.
> >>> 1. Mechanism to decide what Parquet features writers should use
> >>> 2. Mechanism to write Parquet files that use a preview feature
> >>> 3. Mechanism to record in the file what features it uses
> >>> 4. Describing reader behavior when reading a file that is not fully
> >>> supported
> >>>
> >>> The stated guiding principle is: We want to minimize prescribing how
> >>> readers and writers should work and focus on giving them the
> information
> >>> to
> >>> implement the behavior they deem appropriate.
> >>>
> >>> Please comment in the doc to help finalize consensus.
> >>> (thank you Andrew for early feedback and helping me think this
> through!)
> >>>
> >>
> >
>
>
>

Re: [DISCUSS] Finalizing the versioning proposal

[Antoine Pitrou <an...@python.org>]

Hello Ryan,

This is becoming very difficult to follow. It seems you're referring to 
things that were discussed in private (or at least not on this list). It 
also seems to be straying away from the original versioning discussion 
to something else (magic bytes? why?).

I would strongly recommend that this discussion be made email-first.
Perhaps the Parquet community needs more explicit guidelines around 
discussing spec additions/changes, or perhaps even a better process 
altogether?

Regards

Antoine.


Le 01/09/2026 à 02:08, Ryan Blue a écrit :
> I want to follow up on the open points here.
> 
> First, on the open question about magic bytes, I've confirmed with the
> folks proposing the modular footer that we think we can handle encryption
> within the footer. The reason for using the magic bytes for this before
> (PARE) was that we needed to encrypt the entire footer to not leak
> unencrypted data values. But with the modular design, we can leave the
> first module of the footer unencrypted because it just contains offsets
> that aren't sensitive and store a flag to indicate that the rest of the
> modules are encrypted.
> 
> I think it still makes sense to use the magic bytes to encode the
> Parquet format version, so that the entire footer content is covered by the
> version. That avoids having two versions, one for the format and one for
> the first footer module. I propose incrementing the last byte of the magic
> bytes, starting with the next version we release. That is, if we release v3
> next, we would use PAR3. v4 would be PAR4, etc. When we get to Parquet v10,
> we could continue this pattern with 0x3A or skip to `PARA`... but this
> decision would be a long time from now. In the meantime, we'd have one
> version that is clearly visible from the magic bytes.
> 
> Second, for the question about whether to list preview features, I think
> that we _should_ list the preview feature names in the footer. Extra
> information is rarely harmful. I think the strongest argument against
> listing the preview features is that implementations can use this
> information to fail without a best-effort read. I think that a strict or
> pedantic mode is a bad idea, but I prefer that it is an implementation
> decision. There is going to be plenty of pressure not to fail needlessly,
> so we don't need to encourage it at a format level.
> 
> Any other perspectives on these open issues?
> 
> Ryan
> 
> On Fri, Aug 28, 2026 at 2:32 PM Ryan Blue <bl...@apache.org> wrote:
> 
>> Sorry, I initially sent this through the wrong account that gets dropped
>> by ASF lists. Here's my summary from our conversation on Wednesday...
>>
>> We discussed the open questions that Julien summarized above in the sync
>> and had agreement on a couple points, but there are also some open areas
>> for further discussion.
>>
>> Broad agreement:
>> - *For Q1 and Q2, we choose option B*. We want consistency across
>> implementations, but cannot dictate behaviors like specific API methods or
>> reader modes.
>> - We will define consistent names or labels for preview features to help
>> consistency
>> - *For Q3, we eliminated option B*: We do not want to indicate preview
>> features without saying what those preview features are
>> - Q3 covers two separate questions: magic bytes and how to track preview
>> features (see open questions)
>> - *For Q4, we choose option C*: it is an implementation decision to have
>> a strict mode
>>
>> Open questions:
>> - Do we want to add the format version to the magic bytes, OR change magic
>> bytes only when the footer encoding changes?
>> - Do we want to list the preview features that are used in a file, OR do
>> we omit them because we don't want to encourage unnecessary failures?
>>
>> Here's a more detailed summary:
>>
>> For 1 and 2, there was general consensus that we want to have consistency
>> across implementations, but it doesn't make sense to mandate writer API
>> options in a file format's spec that covers on-disk representation. We
>> agreed that we want to define labels for preview features to help ensure
>> consistency (and these may be stored in the format). For example, we could
>> agree and document that "alp-encoding" means the new ALP encoding for float
>> and double values, so that people can refer to the feature consistently,
>> even if the API to enable features is idiomatic and slightly different
>> across languages. We also clarified the requirement that preview features
>> must cause readers that do not implement support to fail; readers must not
>> produce incorrect results.
>>
>> Micah also suggested that we introduce minor versions (which must be
>> forward-compatible within a version) and allow pinning to those minor
>> versions. This would prevent using newer features even though they do not
>> affect correctness. There wasn't an example when this would be useful and
>> we concluded that this can be done later if we discover a use case.
>>
>> For 3, we discussed that there are two decision points. 3.1 is how the
>> magic bytes relate to the format version, and 3.2 is whether we want the
>> preview feature names included in files when they are used. (Option 2 was
>> discarded; it was for, essentially, a boolean flag to indicate preview
>> features were used but did not specify which ones.)
>>
>> In the discussion about magic bytes (3.1), we outlined two options: we can
>> encode the version in the magic bytes and update it each time the format
>> version changes, or we can store the version in the footer and use a
>> separate version in the magic bytes that is incremented when the footer
>> representation changes ("PARA", "PARB", etc.). If we store the format
>> version in the footer, we would need to stop using "PAR1" and "PARE"
>> because we would want to be able to encode more than versions 3-9 and have
>> to handle encryption.
>>
>> The discussion about preview features relates to question 4, which is
>> whether we want to require readers to expose a "pedantic" mode (think:
>> strict). Several people were advocating that readers should always attempt
>> to read (best effort), and therefore we don't need to make changes to
>> enable a strict mode like keeping a list of preview feature names. If
>> implementations choose to have a pedantic mode, they can easily detect
>> changes like unsupported encodings in columns that are not projected and
>> fail with a good error message. We agreed that pedantic mode is a decision
>> for the read implementation (outside the file format spec's scope), but did
>> not conclude that we didn't want the list of preview features.
>>
>>
>>
>> On Fri, Aug 14, 2026 at 11:25 AM Julien Le Dem <ju...@apache.org> wrote:
>>
>>> Following up on the discussion during the Parquet sync on Wednesday, I
>>> have
>>> added a new tab to the Versioning document
>>> <
>>> https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab=t.nywy9gjvpwqf
>>>>
>>> to discuss 4 remaining decision points for the implementation. For each I
>>> have listed options and pros and cons.
>>> 1. Mechanism to decide what Parquet features writers should use
>>> 2. Mechanism to write Parquet files that use a preview feature
>>> 3. Mechanism to record in the file what features it uses
>>> 4. Describing reader behavior when reading a file that is not fully
>>> supported
>>>
>>> The stated guiding principle is: We want to minimize prescribing how
>>> readers and writers should work and focus on giving them the information
>>> to
>>> implement the behavior they deem appropriate.
>>>
>>> Please comment in the doc to help finalize consensus.
>>> (thank you Andrew for early feedback and helping me think this through!)
>>>
>>
> 

Re: [DISCUSS] Finalizing the versioning proposal

[Micah Kornfield <em...@gmail.com>]

I think before we get to the exact logistics of what goes in the footer, we
should agree on requirements.  I put these in Julien's doc and there were
some indications that others agreed with themL

1.  A file written with a preview feature MUST be readable by another
reader that supports the preview features used.  Corollary 1, a file
written with a preview feature must be readable by a reader that supports
reading the major version the preview feature was fully adopted.

2.  A reader MUST never return incorrect data if it doesn't understand a
preview feature (writers must always communicate preview features in a way
that prevents readers from giving back incorrect data to their clients).

3.  A reader SHOULD be able to give a detailed error message about why it
couldn't read the file, regardless of if it can parse the thrift metadata
structure.

4. Old readers should be able to read partial data from new files even if
they don't support a particular encoding or version the encoding was added
in.


Ryan/Russell/Divjot are any of these contentious to you? I ask because I
think these require a little bit more detail then simply adding the version
to the magic bytes.


Thanks,

Micah


On Tue, Sep 1, 2026 at 8:23 AM Russell Spitzer <ru...@gmail.com>
wrote:

> 1. Ryan’s suggestion makes sense to me. If encryption can live in the
> modular footer, using the magic bytes for the format version (PAR3, PAR4,
> …) looks like the safest version signal.
>
> 2. I don't really like adding something for implementations without actual
> community demand. Unless someone plans to use the feature list, I'm not
> sure we should include it. Technically, I get nervous about having a set of
> metadata that exists in parallel to the changes it describes. That's just
> one more thing that could be written incorrectly and cause
> spec-noncompliance (but with no visible issues). So I'm leaning toward a -0
> on the feature list. If anyone here actually plans on using this list for
> something, I'd be on board +1, but right now it just feels like we are
> adding more work for ourselves with only a theoretical use-case that no one
> here wants to actually build.
>
> On Mon, Aug 31, 2026 at 7:35 PM Divjot Arora via dev <
> dev@parquet.apache.org>
> wrote:
>
> > Thanks for summarizing Ryan, this is a good representation of what we
> > discussed last week. Some thoughts on the open questions:
> >
> > 1. Magic bytes: I dislike using the magic bytes to signal encryption and
> > agree that we can (and should) get rid of that with the
> > modular footer. I'd be OK with using the magic bytes to encode version if
> > we also remove the version from the footer Thrift definition.
> > In general, I prefer one way to encode the version rather than 2 and
> using
> > the magic bytes means we can signal both data and
> > metadata changes (e.g. new footer).
> >
> > 2. Preview features list: if we accept that we some implementations may
> > want a "strict" mode and that the preview features used should
> > be fully discoverable from metadata so that such implementations can fail
> > fast, I think a list of preview features in the footer is required.
> > This is because there are some format changes that are not discoverable
> in
> > required metadata (e.g. a new data page header type would
> > be encoded in metadata only if the optional PageEncodingStats are
> present).
> > I generally agree that strict mode is a bad idea and the format
> > should not mandate it, but it's an implementation/library choice to have
> > it.
> >
> > One caveat on (2): the preview features list is option 3C in the
> versioning
> > proposal [1], which says:
> >
> > > Set a field in the footer to the Parquet version used to write it
> > (footer.version: “3”).
> > > If a feature flag has been used in addition to the version, then the
> > version is “3+”
> > > A new field in the footer contains the list of the feature flags that
> > have been turned on (flags: [“alp”])
> >
> > If we include the features list, I'd be opposed to setting the version to
> > "3+" (regardless of where the version is encoded). This goes back
> > to having a single way to signal something: if preview features are used,
> > version "3" plus a non-empty features list should be sufficient
> > to communicate that; I don't see a reason for both having the list and
> > using a different version number or identifier.
> >
> > -- Div
> >
> > [1]
> >
> >
> https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab=t.nywy9gjvpwqf#heading=h.ro8jfbj80f91
> >
> >
> > On Mon, Aug 31, 2026 at 8:09 PM Ryan Blue <bl...@apache.org> wrote:
> >
> > > I want to follow up on the open points here.
> > >
> > > First, on the open question about magic bytes, I've confirmed with the
> > > folks proposing the modular footer that we think we can handle
> encryption
> > > within the footer. The reason for using the magic bytes for this before
> > > (PARE) was that we needed to encrypt the entire footer to not leak
> > > unencrypted data values. But with the modular design, we can leave the
> > > first module of the footer unencrypted because it just contains offsets
> > > that aren't sensitive and store a flag to indicate that the rest of the
> > > modules are encrypted.
> > >
> > > I think it still makes sense to use the magic bytes to encode the
> > > Parquet format version, so that the entire footer content is covered by
> > the
> > > version. That avoids having two versions, one for the format and one
> for
> > > the first footer module. I propose incrementing the last byte of the
> > magic
> > > bytes, starting with the next version we release. That is, if we
> release
> > v3
> > > next, we would use PAR3. v4 would be PAR4, etc. When we get to Parquet
> > v10,
> > > we could continue this pattern with 0x3A or skip to `PARA`... but this
> > > decision would be a long time from now. In the meantime, we'd have one
> > > version that is clearly visible from the magic bytes.
> > >
> > > Second, for the question about whether to list preview features, I
> think
> > > that we _should_ list the preview feature names in the footer. Extra
> > > information is rarely harmful. I think the strongest argument against
> > > listing the preview features is that implementations can use this
> > > information to fail without a best-effort read. I think that a strict
> or
> > > pedantic mode is a bad idea, but I prefer that it is an implementation
> > > decision. There is going to be plenty of pressure not to fail
> needlessly,
> > > so we don't need to encourage it at a format level.
> > >
> > > Any other perspectives on these open issues?
> > >
> > > Ryan
> > >
> > > On Fri, Aug 28, 2026 at 2:32 PM Ryan Blue <bl...@apache.org> wrote:
> > >
> > > > Sorry, I initially sent this through the wrong account that gets
> > dropped
> > > > by ASF lists. Here's my summary from our conversation on Wednesday...
> > > >
> > > > We discussed the open questions that Julien summarized above in the
> > sync
> > > > and had agreement on a couple points, but there are also some open
> > areas
> > > > for further discussion.
> > > >
> > > > Broad agreement:
> > > > - *For Q1 and Q2, we choose option B*. We want consistency across
> > > > implementations, but cannot dictate behaviors like specific API
> methods
> > > or
> > > > reader modes.
> > > > - We will define consistent names or labels for preview features to
> > help
> > > > consistency
> > > > - *For Q3, we eliminated option B*: We do not want to indicate
> preview
> > > > features without saying what those preview features are
> > > > - Q3 covers two separate questions: magic bytes and how to track
> > preview
> > > > features (see open questions)
> > > > - *For Q4, we choose option C*: it is an implementation decision to
> > have
> > > > a strict mode
> > > >
> > > > Open questions:
> > > > - Do we want to add the format version to the magic bytes, OR change
> > > magic
> > > > bytes only when the footer encoding changes?
> > > > - Do we want to list the preview features that are used in a file, OR
> > do
> > > > we omit them because we don't want to encourage unnecessary failures?
> > > >
> > > > Here's a more detailed summary:
> > > >
> > > > For 1 and 2, there was general consensus that we want to have
> > consistency
> > > > across implementations, but it doesn't make sense to mandate writer
> API
> > > > options in a file format's spec that covers on-disk representation.
> We
> > > > agreed that we want to define labels for preview features to help
> > ensure
> > > > consistency (and these may be stored in the format). For example, we
> > > could
> > > > agree and document that "alp-encoding" means the new ALP encoding for
> > > float
> > > > and double values, so that people can refer to the feature
> > consistently,
> > > > even if the API to enable features is idiomatic and slightly
> different
> > > > across languages. We also clarified the requirement that preview
> > features
> > > > must cause readers that do not implement support to fail; readers
> must
> > > not
> > > > produce incorrect results.
> > > >
> > > > Micah also suggested that we introduce minor versions (which must be
> > > > forward-compatible within a version) and allow pinning to those minor
> > > > versions. This would prevent using newer features even though they do
> > not
> > > > affect correctness. There wasn't an example when this would be useful
> > and
> > > > we concluded that this can be done later if we discover a use case.
> > > >
> > > > For 3, we discussed that there are two decision points. 3.1 is how
> the
> > > > magic bytes relate to the format version, and 3.2 is whether we want
> > the
> > > > preview feature names included in files when they are used. (Option 2
> > was
> > > > discarded; it was for, essentially, a boolean flag to indicate
> preview
> > > > features were used but did not specify which ones.)
> > > >
> > > > In the discussion about magic bytes (3.1), we outlined two options:
> we
> > > can
> > > > encode the version in the magic bytes and update it each time the
> > format
> > > > version changes, or we can store the version in the footer and use a
> > > > separate version in the magic bytes that is incremented when the
> footer
> > > > representation changes ("PARA", "PARB", etc.). If we store the format
> > > > version in the footer, we would need to stop using "PAR1" and "PARE"
> > > > because we would want to be able to encode more than versions 3-9 and
> > > have
> > > > to handle encryption.
> > > >
> > > > The discussion about preview features relates to question 4, which is
> > > > whether we want to require readers to expose a "pedantic" mode
> (think:
> > > > strict). Several people were advocating that readers should always
> > > attempt
> > > > to read (best effort), and therefore we don't need to make changes to
> > > > enable a strict mode like keeping a list of preview feature names. If
> > > > implementations choose to have a pedantic mode, they can easily
> detect
> > > > changes like unsupported encodings in columns that are not projected
> > and
> > > > fail with a good error message. We agreed that pedantic mode is a
> > > decision
> > > > for the read implementation (outside the file format spec's scope),
> but
> > > did
> > > > not conclude that we didn't want the list of preview features.
> > > >
> > > >
> > > >
> > > > On Fri, Aug 14, 2026 at 11:25 AM Julien Le Dem <ju...@apache.org>
> > > wrote:
> > > >
> > > >> Following up on the discussion during the Parquet sync on
> Wednesday, I
> > > >> have
> > > >> added a new tab to the Versioning document
> > > >> <
> > > >>
> > >
> >
> https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab=t.nywy9gjvpwqf
> > > >> >
> > > >> to discuss 4 remaining decision points for the implementation. For
> > each
> > > I
> > > >> have listed options and pros and cons.
> > > >> 1. Mechanism to decide what Parquet features writers should use
> > > >> 2. Mechanism to write Parquet files that use a preview feature
> > > >> 3. Mechanism to record in the file what features it uses
> > > >> 4. Describing reader behavior when reading a file that is not fully
> > > >> supported
> > > >>
> > > >> The stated guiding principle is: We want to minimize prescribing how
> > > >> readers and writers should work and focus on giving them the
> > information
> > > >> to
> > > >> implement the behavior they deem appropriate.
> > > >>
> > > >> Please comment in the doc to help finalize consensus.
> > > >> (thank you Andrew for early feedback and helping me think this
> > through!)
> > > >>
> > > >
> > >
> >
>

Re: [DISCUSS] Finalizing the versioning proposal

[Russell Spitzer <ru...@gmail.com>]

1. Ryan’s suggestion makes sense to me. If encryption can live in the
modular footer, using the magic bytes for the format version (PAR3, PAR4,
…) looks like the safest version signal.

2. I don't really like adding something for implementations without actual
community demand. Unless someone plans to use the feature list, I'm not
sure we should include it. Technically, I get nervous about having a set of
metadata that exists in parallel to the changes it describes. That's just
one more thing that could be written incorrectly and cause
spec-noncompliance (but with no visible issues). So I'm leaning toward a -0
on the feature list. If anyone here actually plans on using this list for
something, I'd be on board +1, but right now it just feels like we are
adding more work for ourselves with only a theoretical use-case that no one
here wants to actually build.

On Mon, Aug 31, 2026 at 7:35 PM Divjot Arora via dev <de...@parquet.apache.org>
wrote:

> Thanks for summarizing Ryan, this is a good representation of what we
> discussed last week. Some thoughts on the open questions:
>
> 1. Magic bytes: I dislike using the magic bytes to signal encryption and
> agree that we can (and should) get rid of that with the
> modular footer. I'd be OK with using the magic bytes to encode version if
> we also remove the version from the footer Thrift definition.
> In general, I prefer one way to encode the version rather than 2 and using
> the magic bytes means we can signal both data and
> metadata changes (e.g. new footer).
>
> 2. Preview features list: if we accept that we some implementations may
> want a "strict" mode and that the preview features used should
> be fully discoverable from metadata so that such implementations can fail
> fast, I think a list of preview features in the footer is required.
> This is because there are some format changes that are not discoverable in
> required metadata (e.g. a new data page header type would
> be encoded in metadata only if the optional PageEncodingStats are present).
> I generally agree that strict mode is a bad idea and the format
> should not mandate it, but it's an implementation/library choice to have
> it.
>
> One caveat on (2): the preview features list is option 3C in the versioning
> proposal [1], which says:
>
> > Set a field in the footer to the Parquet version used to write it
> (footer.version: “3”).
> > If a feature flag has been used in addition to the version, then the
> version is “3+”
> > A new field in the footer contains the list of the feature flags that
> have been turned on (flags: [“alp”])
>
> If we include the features list, I'd be opposed to setting the version to
> "3+" (regardless of where the version is encoded). This goes back
> to having a single way to signal something: if preview features are used,
> version "3" plus a non-empty features list should be sufficient
> to communicate that; I don't see a reason for both having the list and
> using a different version number or identifier.
>
> -- Div
>
> [1]
>
> https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab=t.nywy9gjvpwqf#heading=h.ro8jfbj80f91
>
>
> On Mon, Aug 31, 2026 at 8:09 PM Ryan Blue <bl...@apache.org> wrote:
>
> > I want to follow up on the open points here.
> >
> > First, on the open question about magic bytes, I've confirmed with the
> > folks proposing the modular footer that we think we can handle encryption
> > within the footer. The reason for using the magic bytes for this before
> > (PARE) was that we needed to encrypt the entire footer to not leak
> > unencrypted data values. But with the modular design, we can leave the
> > first module of the footer unencrypted because it just contains offsets
> > that aren't sensitive and store a flag to indicate that the rest of the
> > modules are encrypted.
> >
> > I think it still makes sense to use the magic bytes to encode the
> > Parquet format version, so that the entire footer content is covered by
> the
> > version. That avoids having two versions, one for the format and one for
> > the first footer module. I propose incrementing the last byte of the
> magic
> > bytes, starting with the next version we release. That is, if we release
> v3
> > next, we would use PAR3. v4 would be PAR4, etc. When we get to Parquet
> v10,
> > we could continue this pattern with 0x3A or skip to `PARA`... but this
> > decision would be a long time from now. In the meantime, we'd have one
> > version that is clearly visible from the magic bytes.
> >
> > Second, for the question about whether to list preview features, I think
> > that we _should_ list the preview feature names in the footer. Extra
> > information is rarely harmful. I think the strongest argument against
> > listing the preview features is that implementations can use this
> > information to fail without a best-effort read. I think that a strict or
> > pedantic mode is a bad idea, but I prefer that it is an implementation
> > decision. There is going to be plenty of pressure not to fail needlessly,
> > so we don't need to encourage it at a format level.
> >
> > Any other perspectives on these open issues?
> >
> > Ryan
> >
> > On Fri, Aug 28, 2026 at 2:32 PM Ryan Blue <bl...@apache.org> wrote:
> >
> > > Sorry, I initially sent this through the wrong account that gets
> dropped
> > > by ASF lists. Here's my summary from our conversation on Wednesday...
> > >
> > > We discussed the open questions that Julien summarized above in the
> sync
> > > and had agreement on a couple points, but there are also some open
> areas
> > > for further discussion.
> > >
> > > Broad agreement:
> > > - *For Q1 and Q2, we choose option B*. We want consistency across
> > > implementations, but cannot dictate behaviors like specific API methods
> > or
> > > reader modes.
> > > - We will define consistent names or labels for preview features to
> help
> > > consistency
> > > - *For Q3, we eliminated option B*: We do not want to indicate preview
> > > features without saying what those preview features are
> > > - Q3 covers two separate questions: magic bytes and how to track
> preview
> > > features (see open questions)
> > > - *For Q4, we choose option C*: it is an implementation decision to
> have
> > > a strict mode
> > >
> > > Open questions:
> > > - Do we want to add the format version to the magic bytes, OR change
> > magic
> > > bytes only when the footer encoding changes?
> > > - Do we want to list the preview features that are used in a file, OR
> do
> > > we omit them because we don't want to encourage unnecessary failures?
> > >
> > > Here's a more detailed summary:
> > >
> > > For 1 and 2, there was general consensus that we want to have
> consistency
> > > across implementations, but it doesn't make sense to mandate writer API
> > > options in a file format's spec that covers on-disk representation. We
> > > agreed that we want to define labels for preview features to help
> ensure
> > > consistency (and these may be stored in the format). For example, we
> > could
> > > agree and document that "alp-encoding" means the new ALP encoding for
> > float
> > > and double values, so that people can refer to the feature
> consistently,
> > > even if the API to enable features is idiomatic and slightly different
> > > across languages. We also clarified the requirement that preview
> features
> > > must cause readers that do not implement support to fail; readers must
> > not
> > > produce incorrect results.
> > >
> > > Micah also suggested that we introduce minor versions (which must be
> > > forward-compatible within a version) and allow pinning to those minor
> > > versions. This would prevent using newer features even though they do
> not
> > > affect correctness. There wasn't an example when this would be useful
> and
> > > we concluded that this can be done later if we discover a use case.
> > >
> > > For 3, we discussed that there are two decision points. 3.1 is how the
> > > magic bytes relate to the format version, and 3.2 is whether we want
> the
> > > preview feature names included in files when they are used. (Option 2
> was
> > > discarded; it was for, essentially, a boolean flag to indicate preview
> > > features were used but did not specify which ones.)
> > >
> > > In the discussion about magic bytes (3.1), we outlined two options: we
> > can
> > > encode the version in the magic bytes and update it each time the
> format
> > > version changes, or we can store the version in the footer and use a
> > > separate version in the magic bytes that is incremented when the footer
> > > representation changes ("PARA", "PARB", etc.). If we store the format
> > > version in the footer, we would need to stop using "PAR1" and "PARE"
> > > because we would want to be able to encode more than versions 3-9 and
> > have
> > > to handle encryption.
> > >
> > > The discussion about preview features relates to question 4, which is
> > > whether we want to require readers to expose a "pedantic" mode (think:
> > > strict). Several people were advocating that readers should always
> > attempt
> > > to read (best effort), and therefore we don't need to make changes to
> > > enable a strict mode like keeping a list of preview feature names. If
> > > implementations choose to have a pedantic mode, they can easily detect
> > > changes like unsupported encodings in columns that are not projected
> and
> > > fail with a good error message. We agreed that pedantic mode is a
> > decision
> > > for the read implementation (outside the file format spec's scope), but
> > did
> > > not conclude that we didn't want the list of preview features.
> > >
> > >
> > >
> > > On Fri, Aug 14, 2026 at 11:25 AM Julien Le Dem <ju...@apache.org>
> > wrote:
> > >
> > >> Following up on the discussion during the Parquet sync on Wednesday, I
> > >> have
> > >> added a new tab to the Versioning document
> > >> <
> > >>
> >
> https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab=t.nywy9gjvpwqf
> > >> >
> > >> to discuss 4 remaining decision points for the implementation. For
> each
> > I
> > >> have listed options and pros and cons.
> > >> 1. Mechanism to decide what Parquet features writers should use
> > >> 2. Mechanism to write Parquet files that use a preview feature
> > >> 3. Mechanism to record in the file what features it uses
> > >> 4. Describing reader behavior when reading a file that is not fully
> > >> supported
> > >>
> > >> The stated guiding principle is: We want to minimize prescribing how
> > >> readers and writers should work and focus on giving them the
> information
> > >> to
> > >> implement the behavior they deem appropriate.
> > >>
> > >> Please comment in the doc to help finalize consensus.
> > >> (thank you Andrew for early feedback and helping me think this
> through!)
> > >>
> > >
> >
>

Re: [DISCUSS] Finalizing the versioning proposal

[Divjot Arora via dev <de...@parquet.apache.org>]

Thanks for summarizing Ryan, this is a good representation of what we
discussed last week. Some thoughts on the open questions:

1. Magic bytes: I dislike using the magic bytes to signal encryption and
agree that we can (and should) get rid of that with the
modular footer. I'd be OK with using the magic bytes to encode version if
we also remove the version from the footer Thrift definition.
In general, I prefer one way to encode the version rather than 2 and using
the magic bytes means we can signal both data and
metadata changes (e.g. new footer).

2. Preview features list: if we accept that we some implementations may
want a "strict" mode and that the preview features used should
be fully discoverable from metadata so that such implementations can fail
fast, I think a list of preview features in the footer is required.
This is because there are some format changes that are not discoverable in
required metadata (e.g. a new data page header type would
be encoded in metadata only if the optional PageEncodingStats are present).
I generally agree that strict mode is a bad idea and the format
should not mandate it, but it's an implementation/library choice to have it.

One caveat on (2): the preview features list is option 3C in the versioning
proposal [1], which says:

> Set a field in the footer to the Parquet version used to write it
(footer.version: “3”).
> If a feature flag has been used in addition to the version, then the
version is “3+”
> A new field in the footer contains the list of the feature flags that
have been turned on (flags: [“alp”])

If we include the features list, I'd be opposed to setting the version to
"3+" (regardless of where the version is encoded). This goes back
to having a single way to signal something: if preview features are used,
version "3" plus a non-empty features list should be sufficient
to communicate that; I don't see a reason for both having the list and
using a different version number or identifier.

-- Div

[1]
https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab=t.nywy9gjvpwqf#heading=h.ro8jfbj80f91


On Mon, Aug 31, 2026 at 8:09 PM Ryan Blue <bl...@apache.org> wrote:

> I want to follow up on the open points here.
>
> First, on the open question about magic bytes, I've confirmed with the
> folks proposing the modular footer that we think we can handle encryption
> within the footer. The reason for using the magic bytes for this before
> (PARE) was that we needed to encrypt the entire footer to not leak
> unencrypted data values. But with the modular design, we can leave the
> first module of the footer unencrypted because it just contains offsets
> that aren't sensitive and store a flag to indicate that the rest of the
> modules are encrypted.
>
> I think it still makes sense to use the magic bytes to encode the
> Parquet format version, so that the entire footer content is covered by the
> version. That avoids having two versions, one for the format and one for
> the first footer module. I propose incrementing the last byte of the magic
> bytes, starting with the next version we release. That is, if we release v3
> next, we would use PAR3. v4 would be PAR4, etc. When we get to Parquet v10,
> we could continue this pattern with 0x3A or skip to `PARA`... but this
> decision would be a long time from now. In the meantime, we'd have one
> version that is clearly visible from the magic bytes.
>
> Second, for the question about whether to list preview features, I think
> that we _should_ list the preview feature names in the footer. Extra
> information is rarely harmful. I think the strongest argument against
> listing the preview features is that implementations can use this
> information to fail without a best-effort read. I think that a strict or
> pedantic mode is a bad idea, but I prefer that it is an implementation
> decision. There is going to be plenty of pressure not to fail needlessly,
> so we don't need to encourage it at a format level.
>
> Any other perspectives on these open issues?
>
> Ryan
>
> On Fri, Aug 28, 2026 at 2:32 PM Ryan Blue <bl...@apache.org> wrote:
>
> > Sorry, I initially sent this through the wrong account that gets dropped
> > by ASF lists. Here's my summary from our conversation on Wednesday...
> >
> > We discussed the open questions that Julien summarized above in the sync
> > and had agreement on a couple points, but there are also some open areas
> > for further discussion.
> >
> > Broad agreement:
> > - *For Q1 and Q2, we choose option B*. We want consistency across
> > implementations, but cannot dictate behaviors like specific API methods
> or
> > reader modes.
> > - We will define consistent names or labels for preview features to help
> > consistency
> > - *For Q3, we eliminated option B*: We do not want to indicate preview
> > features without saying what those preview features are
> > - Q3 covers two separate questions: magic bytes and how to track preview
> > features (see open questions)
> > - *For Q4, we choose option C*: it is an implementation decision to have
> > a strict mode
> >
> > Open questions:
> > - Do we want to add the format version to the magic bytes, OR change
> magic
> > bytes only when the footer encoding changes?
> > - Do we want to list the preview features that are used in a file, OR do
> > we omit them because we don't want to encourage unnecessary failures?
> >
> > Here's a more detailed summary:
> >
> > For 1 and 2, there was general consensus that we want to have consistency
> > across implementations, but it doesn't make sense to mandate writer API
> > options in a file format's spec that covers on-disk representation. We
> > agreed that we want to define labels for preview features to help ensure
> > consistency (and these may be stored in the format). For example, we
> could
> > agree and document that "alp-encoding" means the new ALP encoding for
> float
> > and double values, so that people can refer to the feature consistently,
> > even if the API to enable features is idiomatic and slightly different
> > across languages. We also clarified the requirement that preview features
> > must cause readers that do not implement support to fail; readers must
> not
> > produce incorrect results.
> >
> > Micah also suggested that we introduce minor versions (which must be
> > forward-compatible within a version) and allow pinning to those minor
> > versions. This would prevent using newer features even though they do not
> > affect correctness. There wasn't an example when this would be useful and
> > we concluded that this can be done later if we discover a use case.
> >
> > For 3, we discussed that there are two decision points. 3.1 is how the
> > magic bytes relate to the format version, and 3.2 is whether we want the
> > preview feature names included in files when they are used. (Option 2 was
> > discarded; it was for, essentially, a boolean flag to indicate preview
> > features were used but did not specify which ones.)
> >
> > In the discussion about magic bytes (3.1), we outlined two options: we
> can
> > encode the version in the magic bytes and update it each time the format
> > version changes, or we can store the version in the footer and use a
> > separate version in the magic bytes that is incremented when the footer
> > representation changes ("PARA", "PARB", etc.). If we store the format
> > version in the footer, we would need to stop using "PAR1" and "PARE"
> > because we would want to be able to encode more than versions 3-9 and
> have
> > to handle encryption.
> >
> > The discussion about preview features relates to question 4, which is
> > whether we want to require readers to expose a "pedantic" mode (think:
> > strict). Several people were advocating that readers should always
> attempt
> > to read (best effort), and therefore we don't need to make changes to
> > enable a strict mode like keeping a list of preview feature names. If
> > implementations choose to have a pedantic mode, they can easily detect
> > changes like unsupported encodings in columns that are not projected and
> > fail with a good error message. We agreed that pedantic mode is a
> decision
> > for the read implementation (outside the file format spec's scope), but
> did
> > not conclude that we didn't want the list of preview features.
> >
> >
> >
> > On Fri, Aug 14, 2026 at 11:25 AM Julien Le Dem <ju...@apache.org>
> wrote:
> >
> >> Following up on the discussion during the Parquet sync on Wednesday, I
> >> have
> >> added a new tab to the Versioning document
> >> <
> >>
> https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab=t.nywy9gjvpwqf
> >> >
> >> to discuss 4 remaining decision points for the implementation. For each
> I
> >> have listed options and pros and cons.
> >> 1. Mechanism to decide what Parquet features writers should use
> >> 2. Mechanism to write Parquet files that use a preview feature
> >> 3. Mechanism to record in the file what features it uses
> >> 4. Describing reader behavior when reading a file that is not fully
> >> supported
> >>
> >> The stated guiding principle is: We want to minimize prescribing how
> >> readers and writers should work and focus on giving them the information
> >> to
> >> implement the behavior they deem appropriate.
> >>
> >> Please comment in the doc to help finalize consensus.
> >> (thank you Andrew for early feedback and helping me think this through!)
> >>
> >
>

Re: [DISCUSS] Finalizing the versioning proposal

[Ryan Blue <bl...@apache.org>]

I want to follow up on the open points here.

First, on the open question about magic bytes, I've confirmed with the
folks proposing the modular footer that we think we can handle encryption
within the footer. The reason for using the magic bytes for this before
(PARE) was that we needed to encrypt the entire footer to not leak
unencrypted data values. But with the modular design, we can leave the
first module of the footer unencrypted because it just contains offsets
that aren't sensitive and store a flag to indicate that the rest of the
modules are encrypted.

I think it still makes sense to use the magic bytes to encode the
Parquet format version, so that the entire footer content is covered by the
version. That avoids having two versions, one for the format and one for
the first footer module. I propose incrementing the last byte of the magic
bytes, starting with the next version we release. That is, if we release v3
next, we would use PAR3. v4 would be PAR4, etc. When we get to Parquet v10,
we could continue this pattern with 0x3A or skip to `PARA`... but this
decision would be a long time from now. In the meantime, we'd have one
version that is clearly visible from the magic bytes.

Second, for the question about whether to list preview features, I think
that we _should_ list the preview feature names in the footer. Extra
information is rarely harmful. I think the strongest argument against
listing the preview features is that implementations can use this
information to fail without a best-effort read. I think that a strict or
pedantic mode is a bad idea, but I prefer that it is an implementation
decision. There is going to be plenty of pressure not to fail needlessly,
so we don't need to encourage it at a format level.

Any other perspectives on these open issues?

Ryan

On Fri, Aug 28, 2026 at 2:32 PM Ryan Blue <bl...@apache.org> wrote:

> Sorry, I initially sent this through the wrong account that gets dropped
> by ASF lists. Here's my summary from our conversation on Wednesday...
>
> We discussed the open questions that Julien summarized above in the sync
> and had agreement on a couple points, but there are also some open areas
> for further discussion.
>
> Broad agreement:
> - *For Q1 and Q2, we choose option B*. We want consistency across
> implementations, but cannot dictate behaviors like specific API methods or
> reader modes.
> - We will define consistent names or labels for preview features to help
> consistency
> - *For Q3, we eliminated option B*: We do not want to indicate preview
> features without saying what those preview features are
> - Q3 covers two separate questions: magic bytes and how to track preview
> features (see open questions)
> - *For Q4, we choose option C*: it is an implementation decision to have
> a strict mode
>
> Open questions:
> - Do we want to add the format version to the magic bytes, OR change magic
> bytes only when the footer encoding changes?
> - Do we want to list the preview features that are used in a file, OR do
> we omit them because we don't want to encourage unnecessary failures?
>
> Here's a more detailed summary:
>
> For 1 and 2, there was general consensus that we want to have consistency
> across implementations, but it doesn't make sense to mandate writer API
> options in a file format's spec that covers on-disk representation. We
> agreed that we want to define labels for preview features to help ensure
> consistency (and these may be stored in the format). For example, we could
> agree and document that "alp-encoding" means the new ALP encoding for float
> and double values, so that people can refer to the feature consistently,
> even if the API to enable features is idiomatic and slightly different
> across languages. We also clarified the requirement that preview features
> must cause readers that do not implement support to fail; readers must not
> produce incorrect results.
>
> Micah also suggested that we introduce minor versions (which must be
> forward-compatible within a version) and allow pinning to those minor
> versions. This would prevent using newer features even though they do not
> affect correctness. There wasn't an example when this would be useful and
> we concluded that this can be done later if we discover a use case.
>
> For 3, we discussed that there are two decision points. 3.1 is how the
> magic bytes relate to the format version, and 3.2 is whether we want the
> preview feature names included in files when they are used. (Option 2 was
> discarded; it was for, essentially, a boolean flag to indicate preview
> features were used but did not specify which ones.)
>
> In the discussion about magic bytes (3.1), we outlined two options: we can
> encode the version in the magic bytes and update it each time the format
> version changes, or we can store the version in the footer and use a
> separate version in the magic bytes that is incremented when the footer
> representation changes ("PARA", "PARB", etc.). If we store the format
> version in the footer, we would need to stop using "PAR1" and "PARE"
> because we would want to be able to encode more than versions 3-9 and have
> to handle encryption.
>
> The discussion about preview features relates to question 4, which is
> whether we want to require readers to expose a "pedantic" mode (think:
> strict). Several people were advocating that readers should always attempt
> to read (best effort), and therefore we don't need to make changes to
> enable a strict mode like keeping a list of preview feature names. If
> implementations choose to have a pedantic mode, they can easily detect
> changes like unsupported encodings in columns that are not projected and
> fail with a good error message. We agreed that pedantic mode is a decision
> for the read implementation (outside the file format spec's scope), but did
> not conclude that we didn't want the list of preview features.
>
>
>
> On Fri, Aug 14, 2026 at 11:25 AM Julien Le Dem <ju...@apache.org> wrote:
>
>> Following up on the discussion during the Parquet sync on Wednesday, I
>> have
>> added a new tab to the Versioning document
>> <
>> https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab=t.nywy9gjvpwqf
>> >
>> to discuss 4 remaining decision points for the implementation. For each I
>> have listed options and pros and cons.
>> 1. Mechanism to decide what Parquet features writers should use
>> 2. Mechanism to write Parquet files that use a preview feature
>> 3. Mechanism to record in the file what features it uses
>> 4. Describing reader behavior when reading a file that is not fully
>> supported
>>
>> The stated guiding principle is: We want to minimize prescribing how
>> readers and writers should work and focus on giving them the information
>> to
>> implement the behavior they deem appropriate.
>>
>> Please comment in the doc to help finalize consensus.
>> (thank you Andrew for early feedback and helping me think this through!)
>>
>

Re: [DISCUSS] Finalizing the versioning proposal

[Ryan Blue <bl...@apache.org>]

Sorry, I initially sent this through the wrong account that gets dropped by
ASF lists. Here's my summary from our conversation on Wednesday...

We discussed the open questions that Julien summarized above in the sync
and had agreement on a couple points, but there are also some open areas
for further discussion.

Broad agreement:
- *For Q1 and Q2, we choose option B*. We want consistency across
implementations, but cannot dictate behaviors like specific API methods or
reader modes.
- We will define consistent names or labels for preview features to help
consistency
- *For Q3, we eliminated option B*: We do not want to indicate preview
features without saying what those preview features are
- Q3 covers two separate questions: magic bytes and how to track preview
features (see open questions)
- *For Q4, we choose option C*: it is an implementation decision to have a
strict mode

Open questions:
- Do we want to add the format version to the magic bytes, OR change magic
bytes only when the footer encoding changes?
- Do we want to list the preview features that are used in a file, OR do we
omit them because we don't want to encourage unnecessary failures?

Here's a more detailed summary:

For 1 and 2, there was general consensus that we want to have consistency
across implementations, but it doesn't make sense to mandate writer API
options in a file format's spec that covers on-disk representation. We
agreed that we want to define labels for preview features to help ensure
consistency (and these may be stored in the format). For example, we could
agree and document that "alp-encoding" means the new ALP encoding for float
and double values, so that people can refer to the feature consistently,
even if the API to enable features is idiomatic and slightly different
across languages. We also clarified the requirement that preview features
must cause readers that do not implement support to fail; readers must not
produce incorrect results.

Micah also suggested that we introduce minor versions (which must be
forward-compatible within a version) and allow pinning to those minor
versions. This would prevent using newer features even though they do not
affect correctness. There wasn't an example when this would be useful and
we concluded that this can be done later if we discover a use case.

For 3, we discussed that there are two decision points. 3.1 is how the
magic bytes relate to the format version, and 3.2 is whether we want the
preview feature names included in files when they are used. (Option 2 was
discarded; it was for, essentially, a boolean flag to indicate preview
features were used but did not specify which ones.)

In the discussion about magic bytes (3.1), we outlined two options: we can
encode the version in the magic bytes and update it each time the format
version changes, or we can store the version in the footer and use a
separate version in the magic bytes that is incremented when the footer
representation changes ("PARA", "PARB", etc.). If we store the format
version in the footer, we would need to stop using "PAR1" and "PARE"
because we would want to be able to encode more than versions 3-9 and have
to handle encryption.

The discussion about preview features relates to question 4, which is
whether we want to require readers to expose a "pedantic" mode (think:
strict). Several people were advocating that readers should always attempt
to read (best effort), and therefore we don't need to make changes to
enable a strict mode like keeping a list of preview feature names. If
implementations choose to have a pedantic mode, they can easily detect
changes like unsupported encodings in columns that are not projected and
fail with a good error message. We agreed that pedantic mode is a decision
for the read implementation (outside the file format spec's scope), but did
not conclude that we didn't want the list of preview features.



On Fri, Aug 14, 2026 at 11:25 AM Julien Le Dem <ju...@apache.org> wrote:

> Following up on the discussion during the Parquet sync on Wednesday, I have
> added a new tab to the Versioning document
> <
> https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab=t.nywy9gjvpwqf
> >
> to discuss 4 remaining decision points for the implementation. For each I
> have listed options and pros and cons.
> 1. Mechanism to decide what Parquet features writers should use
> 2. Mechanism to write Parquet files that use a preview feature
> 3. Mechanism to record in the file what features it uses
> 4. Describing reader behavior when reading a file that is not fully
> supported
>
> The stated guiding principle is: We want to minimize prescribing how
> readers and writers should work and focus on giving them the information to
> implement the behavior they deem appropriate.
>
> Please comment in the doc to help finalize consensus.
> (thank you Andrew for early feedback and helping me think this through!)
>

Re: [DISCUSS] Finalizing the versioning proposal

[Micah Kornfield <em...@gmail.com>]

Hi Ryan and Russell,
Thank you for the responses.  I'm replying where I think we might have the
most misalignment:


> +1 - Preview features are a good idea, but thinking through how they work
> is secondary to fixing metadata to enable versions.


We specifically defined preview features in the first vote. Not
considering them as part of determining how readers and writers coordinate
on a technical level leaves things incomplete.

A reader that does not support a preview feature must not return
> incorrect data. If existing readers would silently misread it, it is not a
> preview feature; it belongs in the next major version.


The vote specifically said preview features can optionally be enabled with
feature flags. So this case should be considered.

4. This mixes version and preview. A V2 reader should fail fast on a file
> labeled V3; we cannot guarantee a correct read even if it undrestands some
> of the structure. A V2 reader should be able to safely try to read a V2
> file that uses preview features it does not implement. It should only fail
> if it reaches a feature it does not understand and cannot skip (Example -
> Encoding, Logical Type, ...)


I think this statement mixes two concepts on versioning:
1. How libraries communicate to humans what they support. This I believe we
already voted on and I agree versions are a good short-hand to group
features. This allows users to easily determine sets of features supported
by a particular reader or writer.
2. How readers and writers of parquet coordinate on a physical level. This
is what we are discussing now.

I don't think we should be lumping these two together. When we look at what
we get by failing early on an unrecognized version number there are roughly
three categories of changes that are conflated into one:

1. New features required for reading that have a clear semantic distinction
via enum (e.g. encodings).
2. A change in semantics to existing fields (or adding new fields that are
necessary for semantic correctness).  To my knowledge this has never
actually been done.  The only open proposal that I think might fall into
this category is out-of-line pages, but I think even this we could come up
with a design that does not necessarily require this.
3. A structural change to the footer metadata that makes it unparseable.
IIUC, the only change like this has been encryption. This was communicated
with a new magic number. Open proposals that fall into this bucket are the
modular footer  work and possible removing path_in_schema.

Checking the version and failing early treats all these as equal and
results in a complete failure even if:
1. The reader being used actually supports all the new features in the
specified version but was never updated to accept that specific version
number.
2. Most or all of the file is still readable by the reader.  When we look
at breaking changes more often they affect a specific column type:
      a.  We've added 5 encodings and a new page type.  This type of
change  is likely to be the most common change going forward, and is easily
detectable on read.
      b.  We've done one structural change to the footer.

Based on this assessment, I have a hard time supporting the requirement
that readers MUST or SHOULD fail on an unrecognized version number.

The one case that really seems to require a hard failure for which we
haven't designed a solution is the "semantic change/additional load bearing
field".  I think we can decide on a concrete design once we agree on
requirements.

I am in favor of recording the version number that clients of Parquet
requested while writing, and possibly feature flags, so that libraries can
provide clear error messages to help operators and/or users solve their
problems quickly. e.g. it would be very nice when a reader fails for it to
generate error messages like:
"Failed to read file.  Likely cause: the reader currently supports version
2 but the parquet file was written with version 3 features"
"Failed to read file.  Unknown encoding found.  The file was written with
the following preview features that are not supported by the reader 'FSST',
'ALP'"


Thanks,
Micah

On Wed, Sep 2, 2026 at 1:39 PM Ryan Blue <bl...@apache.org> wrote:

> > the core problem we need to solve here is how Parquet writers communicate
> to Parquet readers which features must be supported in order to guarantee
> they can completely read a given file.
>
> +1 - Preview features are a good idea, but thinking through how they work
> is secondary to fixing metadata to enable versions.
>
> I have specific replies to Micah below, but after having written responses,
> I think his 4 points are mostly about preview features (except 3, but
> that's phrased as a "SHOULD"). Unless I'm mistaken, these aren't really
> blockers and I think we should try to move forward with the magic and
> version field discussion as Andrew suggests.
>
> > I do not think 1 and 2 are contentious; they follow from the vote.
>
> I agree with Russell here. These points are good limits on preview
> features. We can never produce incorrect results, so unsupported preview
> features must cause reasonable failures (2). And we shouldn't release
> features as a preview if they are unfinished and could change in a way that
> causes incorrect results or failures in readers with full support (1).
>
> > A reader SHOULD be able to give a detailed error message about why it
> couldn't read the file, regardless of if it can parse the thrift metadata
> structure.
>
> The most important clarification to this is that not supporting the format
> version is the correct error message when you can't decode the footer. The
> second part, "regardless of if it can parse the thrift metadata" is very
> confusing to me. When a reader encounters a newer format version, it can't
> just assume that the metadata is thrift and attempt to read it.
>
> > Old readers should be able to read partial data from new files even if
> they
> don't support a particular encoding or version the encoding was added in.
>
> I _think_ this is referring to preview features, because readers cannot
> read newer format versions and forward compatibility is already required
> within a format version: new features within a format version cannot cause
> older readers to fail or read incorrect data. However, we want more
> flexibility with preview features and in that context, this is a good idea.
>
> We want to minimize the disruption caused by preview features. I'm not sure
> we want to say more than that or how we might phrase this to be narrow and
> clear. The encoding example is good, but just an example. This phrasing is
> okay, but is still just a recommendation since it uses "should be able to".
> I'm not sure it's worth making this guideline "official" because it is
> going to come down to judgement.
>

Re: [DISCUSS] Finalizing the versioning proposal

[Ryan Blue <bl...@apache.org>]

> the core problem we need to solve here is how Parquet writers communicate
to Parquet readers which features must be supported in order to guarantee
they can completely read a given file.

+1 - Preview features are a good idea, but thinking through how they work
is secondary to fixing metadata to enable versions.

I have specific replies to Micah below, but after having written responses,
I think his 4 points are mostly about preview features (except 3, but
that's phrased as a "SHOULD"). Unless I'm mistaken, these aren't really
blockers and I think we should try to move forward with the magic and
version field discussion as Andrew suggests.

> I do not think 1 and 2 are contentious; they follow from the vote.

I agree with Russell here. These points are good limits on preview
features. We can never produce incorrect results, so unsupported preview
features must cause reasonable failures (2). And we shouldn't release
features as a preview if they are unfinished and could change in a way that
causes incorrect results or failures in readers with full support (1).

> A reader SHOULD be able to give a detailed error message about why it
couldn't read the file, regardless of if it can parse the thrift metadata
structure.

The most important clarification to this is that not supporting the format
version is the correct error message when you can't decode the footer. The
second part, "regardless of if it can parse the thrift metadata" is very
confusing to me. When a reader encounters a newer format version, it can't
just assume that the metadata is thrift and attempt to read it.

> Old readers should be able to read partial data from new files even if they
don't support a particular encoding or version the encoding was added in.

I _think_ this is referring to preview features, because readers cannot
read newer format versions and forward compatibility is already required
within a format version: new features within a format version cannot cause
older readers to fail or read incorrect data. However, we want more
flexibility with preview features and in that context, this is a good idea.

We want to minimize the disruption caused by preview features. I'm not sure
we want to say more than that or how we might phrase this to be narrow and
clear. The encoding example is good, but just an example. This phrasing is
okay, but is still just a recommendation since it uses "should be able to".
I'm not sure it's worth making this guideline "official" because it is
going to come down to judgement.

Re: [DISCUSS] Finalizing the versioning proposal

[Antoine Pitrou <an...@python.org>]

Le 02/09/2026 à 19:46, Andrew Lamb a écrit :
>> """
>> Readers need a simple mechanism to know whether they support all
>> features they may encounter when reading a specific file.
>> """
>>
>> I would like to dispute this. No argument is given as to why readers
>> need something else than what is already embodied in the metadata.
> 
> Antoine,
> 
> In my mind, the core usecase is actually knowing how to configure the
> writer (aka how can a writer have confidence to use a certain parquet
> feature and make sure the readers they care about can read the file)

It seems this contradicts your previous message in response to the same 
message where you said:

"""
In my opinion, the core problem we need to solve here is how Parquet
writers communicate to Parquet readers which features must be supported 
in order to guarantee they can completely read a given file.
"""

I agree with the need to configure the writer. It doesn't require the 
writer configuration to be baked in the Parquet metadata.

Regards

Antoine.

Re: [DISCUSS] Finalizing the versioning proposal

[Andrew Lamb <an...@gmail.com>]

> """
> Readers need a simple mechanism to know whether they support all
> features they may encounter when reading a specific file.
> """
>
> I would like to dispute this. No argument is given as to why readers
> need something else than what is already embodied in the metadata.

Antoine,

In my mind, the core usecase is actually knowing how to configure the
writer (aka how can a writer have confidence to use a certain parquet
feature and make sure the readers they care about can read the file)

Andrew

On Wed, Sep 2, 2026 at 12:42 PM Antoine Pitrou <an...@python.org> wrote:

>
> """
> Readers need a simple mechanism to know whether they support all
> features they may encounter when reading a specific file.
> """
>
> I would like to dispute this. No argument is given as to why readers
> need something else than what is already embodied in the metadata.
>
> It's also not obvious why that information is granular enough (some
> features are optional, the user may not want to read all columns anyway,
> etc.).
>
> At least in Arrow, I don't remember a single user complaining that they
> don't have a simpler access to that information.
>
> Regards
>
> Antoine.
>
>
>
> Le 14/08/2026 à 20:25, Julien Le Dem a écrit :
> > Following up on the discussion during the Parquet sync on Wednesday, I
> have
> > added a new tab to the Versioning document
> > <
> https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab=t.nywy9gjvpwqf
> >
> > to discuss 4 remaining decision points for the implementation. For each I
> > have listed options and pros and cons.
> > 1. Mechanism to decide what Parquet features writers should use
> > 2. Mechanism to write Parquet files that use a preview feature
> > 3. Mechanism to record in the file what features it uses
> > 4. Describing reader behavior when reading a file that is not fully
> > supported
> >
> > The stated guiding principle is: We want to minimize prescribing how
> > readers and writers should work and focus on giving them the information
> to
> > implement the behavior they deem appropriate.
> >
> > Please comment in the doc to help finalize consensus.
> > (thank you Andrew for early feedback and helping me think this through!)
>
>
>

Re: [DISCUSS] Finalizing the versioning proposal

[Andrew Lamb <an...@gmail.com>]

> """
> Readers need a simple mechanism to know whether they support all
> features they may encounter when reading a specific file.
> """
>
> I would like to dispute this. No argument is given as to why readers
> need something else than what is already embodied in the metadata.

Antoine,

In my mind, the core usecase is actually knowing how to configure the
writer (aka how can a writer have confidence to use a certain parquet
feature and make sure the readers they care about can read the file)

Andrew

On Wed, Sep 2, 2026 at 12:42 PM Antoine Pitrou <an...@python.org> wrote:

>
> """
> Readers need a simple mechanism to know whether they support all
> features they may encounter when reading a specific file.
> """
>
> I would like to dispute this. No argument is given as to why readers
> need something else than what is already embodied in the metadata.
>
> It's also not obvious why that information is granular enough (some
> features are optional, the user may not want to read all columns anyway,
> etc.).
>
> At least in Arrow, I don't remember a single user complaining that they
> don't have a simpler access to that information.
>
> Regards
>
> Antoine.
>
>
>
> Le 14/08/2026 à 20:25, Julien Le Dem a écrit :
> > Following up on the discussion during the Parquet sync on Wednesday, I
> have
> > added a new tab to the Versioning document
> > <
> https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab=t.nywy9gjvpwqf
> >
> > to discuss 4 remaining decision points for the implementation. For each I
> > have listed options and pros and cons.
> > 1. Mechanism to decide what Parquet features writers should use
> > 2. Mechanism to write Parquet files that use a preview feature
> > 3. Mechanism to record in the file what features it uses
> > 4. Describing reader behavior when reading a file that is not fully
> > supported
> >
> > The stated guiding principle is: We want to minimize prescribing how
> > readers and writers should work and focus on giving them the information
> to
> > implement the behavior they deem appropriate.
> >
> > Please comment in the doc to help finalize consensus.
> > (thank you Andrew for early feedback and helping me think this through!)
>
>
>

Re: [DISCUSS] Finalizing the versioning proposal

[Will Edwards via dev <de...@parquet.apache.org>]

Following with some scenarios I just thought up to illustrate:

Imagine a reader which knows version N encounters a file that contains a
new feature.

If the feature is not breaking the footer and if the version in the footer
is still N then the reader will ignore benign new features but fail if it
needs to access the feature etc.  Like today.

If the feature is a breaking change to the footer then we’d better have
bumped the version number in the footer and we need the reader to fail.

It the feature is not breaking the footer but we bumped the version number
anyway then the reader has to fail- there is no correct way to do graceful
degradation.

This final case is surprising but it’s useful to illustrate it by making up
a scenario where the reader proceeding to parse a footer with a version
number it doesn’t understand would corrupt data:

The new feature in the file is a new kind of schema.

The reader proceeds but doesn’t recognise the new schema so wrongly
believes the columns being accessed are not present.

Most engines using the reader are used to encountering parquet files with
missing columns and interprets them as NULLs.

Ergo not versioning the footer means we can’t do breaking changes to the
footer itself, and if we commingle features and footer version in the same
footer version then readers can’t distinguish and must fail.

So my conclusion is that footer version we have in the footer thrift today
should be the footer version, only bumped if we have breaking changes to
the footer itself, and the current mechanism of thrift field ids continues
to version the features used in the rest of the file.

Which is what we have today?

Re: [DISCUSS] Finalizing the versioning proposal

[Will Edwards via dev <de...@parquet.apache.org>]

>
> It worries me if the version number in the footer describes the features
used in the file generally, rather than only versioning the footer protocol.

Versioning the footer format itself allows us to introduce a breaking
change.  If we do anything in the footer that means that unaware readers
would mis-parse the footer then we’d bump the version number.  (Making a
required field optional is a breaking change.  And proposals for breaking
footers would always start their new footers with a tiny thrift object with
the version number. Almost everything else can always be introduced in a
non-breaking way.)

If the footer isn’t itself versioned then there is no way to introduce
breaking changes that impact the footer itself.

Whereas if the version in the footer is saying that there is some features
of that future version somewhere in the file, that doesn't help a reader
who doesn’t know what that version is to understand it it is safe to
proceed at all!  This is forcing the fail fast upgrade treadmill that I
fear will upset users and make parquet unsuitable for long term storage :(

>
>
> p.s. I really like Will's formulation of the tension between long term
> stable storage and platform vendor upgrade cycles.


Thanks :blush:



>

Re: [DISCUSS] Finalizing the versioning proposal

[Will Edwards via dev <de...@parquet.apache.org>]

>
> It worries me if the version number in the footer describes the features
used in the file generally, rather than only versioning the footer protocol.

Versioning the footer format itself allows us to introduce a breaking
change.  If we do anything in the footer that means that unaware readers
would mis-parse the footer then we’d bump the version number.  (Making a
required field optional is a breaking change.  And proposals for breaking
footers would always start their new footers with a tiny thrift object with
the version number. Almost everything else can always be introduced in a
non-breaking way.)

If the footer isn’t itself versioned then there is no way to introduce
breaking changes that impact the footer itself.

Whereas if the version in the footer is saying that there is some features
of that future version somewhere in the file, that doesn't help a reader
who doesn’t know what that version is to understand it it is safe to
proceed at all!  This is forcing the fail fast upgrade treadmill that I
fear will upset users and make parquet unsuitable for long term storage :(

>
>
> p.s. I really like Will's formulation of the tension between long term
> stable storage and platform vendor upgrade cycles.


Thanks :blush:



>

Re: [DISCUSS] Finalizing the versioning proposal

[Andrew Lamb <an...@gmail.com>]

> Personally, I don't think readers should up front refuse reading a file
> that has a version number higher than their official compatibility
> level. Other people (Micah, Will, at least) see to be of the same opinion.

In my mind, writing a version number into a file does not obligate any
particular behavior in the reader; It does not have to refuse reading any
files.

A version number summarizes, in a single number, what features may be
encountered in that file. The reader is perfectly free to try and read
files for which it may not support all features.

The only obligation to readers would be that if they say "supports Apache
Parquet V3" they do indeed support all features of V3 (or whatever version)

Andrew

p.s. I really like Will's formulation of the tension between long term
stable storage and platform vendor upgrade cycles.


On Fri, Sep 4, 2026 at 5:20 PM Russell Spitzer <ru...@gmail.com>
wrote:

> Let me try to clarify, as I still think we are talking about different
> things.
>
> The version marker on a file is the oldest format version whose structure
> that file actually uses. A file labeled V3 contains a structure a V2 reader
> cannot correctly parse, or a semantic change it will get wrong. There is no
> partial-success path for a V2 reader on that file. (I believe this is the
> point Micah was trying to probe.) If a V2 reader could read it at all,
> labeling it V3 would be incorrect; the writer should have labeled it V2.
> The file becomes V3 when it contains a breaking change that older readers
> cannot handle, causes incorrect results, or that they would mishandle
> without a clear error. That is why a new encoding does not, by itself, bump
> the file version. A V2 reader can see an encoding or logical type it does
> not understand and fail on that column.
>
> Separately, “this engine supports V3” is a claim about the reader, not a
> stamp on every file that uses a V3-owned feature. A V3 reader is expected
> to handle the V3 bundle (including encodings that may also appear as
> preview in V2 files) by default. The encoding is owned by V3; the file is
> only V3 if it also contains the breaking change.
>
> If anyone has had different interpretations please let me know, but this
> has been my understanding
>
>
> On Fri, Sep 4, 2026 at 3:32 PM Antoine Pitrou <an...@python.org> wrote:
>
> >
> > Hey Ryan,
> >
> > Let's get the process issue straight. We voted A, not A + B. Perhaps you
> > hoped that A implies B, but that is your personal opinion on the topic.
> >
> > (as a reminder, you were the one who insisted on a first vote on a very
> > minimal proposition, before discussing the specifics)
> >
> > Having the community accept B implies another discussion and a vote. And
> > that's the discussion we're having currently, actually, so I don't even
> > know why you seem to be objecting.
> >
> > Regards
> >
> > Antoine.
> >
> >
> >
> > Le 04/09/2026 à 22:02, Ryan Blue a écrit :
> > >  From the doc, here's the point that we've been discussing:
> > >
> > >> Readers need a simple mechanism to know whether they support all
> > features
> > > they may encounter when reading a specific file.
> > >
> > > This is a follow-up to decide between options:
> > > - Set footer version to 3 (for example)
> > > - Set footer version to 3+ when preview features are present
> > > - Set footer version to 3/3+ and record preview feature names
> > > - Use magic bytes for version - PAR3
> > >
> > >  From this, it appears to me that most of us (at least those of us
> > > discussing it) have been working under the assumption that our vote
> meant
> > > using the version by writing it into files. Personally, I think this is
> > > also a natural conclusion. A reader needs to know what version a file
> was
> > > written with in order to decide whether it has guarantees (which is
> > another
> > > way to say what's in the versioning doc).
> > >
> > > I don't think that this is a dishonest or bad faith argument.
> > >
> > > My suggestion to ask the community is really to clarify what we all
> think
> > > we voted on. If there was genuine confusion about this then let's talk
> > > through it. But, again, I don't think that this was unclear to most
> > people.
> > >
> > > Do you agree that it's reasonable to ask what people thought they were
> > > agreeing to?
> > >
> > > Ryan
> > >
> > > On Fri, Sep 4, 2026 at 12:26 PM Antoine Pitrou <an...@python.org>
> > wrote:
> > >
> > >>
> > >> Le 04/09/2026 à 20:35, Ryan Blue a écrit :
> > >>> It sounds like you're suggesting that we don't version data files. I
> > >> don't
> > >>> think that makes sense with versioning and I'm pretty sure most
> people
> > >> that
> > >>> voted to use versions would agree with me.
> > >>
> > >> The *only* implication of the vote was to move forward with the usage
> of
> > >> community-agreed versions (rather than, say, mechanistic
> calendar-based
> > >> presets) to signal levels of compatibility between readers and
> writers.
> > >>
> > >> If you wanted the vote to mean something more, then you should have
> had
> > >> the vote explicitly mention it. Anything else is dishonest.
> > >>
> > >>> Should I start a thread dedicated to asking the
> > >>> community whether most people thought that "using versions" would
> mean
> > >>> writing the version used for a file in the file metadata?
> > >>
> > >> No, you should wait for this discussion to happen in the hope of
> > >> reaching a consensus.
> > >>
> > >> Personally, I don't think readers should up front refuse reading a
> file
> > >> that has a version number higher than their official compatibility
> > >> level. Other people (Micah, Will, at least) see to be of the same
> > opinion.
> > >>
> > >> Regards
> > >>
> > >> Antoine.
> > >>
> > >>
> > >>
> > >
> >
> >
> >
>

Re: [DISCUSS] Finalizing the versioning proposal

[Micah Kornfield <em...@gmail.com>]

Hi Russell,

The version marker on a file is the oldest format version whose structure
> that file actually uses. A file labeled V3 contains a structure a V2 reader
> cannot correctly parse, or a semantic change it will get wrong. There is no
> partial-success path for a V2 reader on that file. (I believe this is the
> point Micah was trying to probe.) If a V2 reader could read it at all,
> labeling it V3 would be incorrect; the writer should have labeled it V2.
> The file becomes V3 when it contains a breaking change that older readers
> cannot handle, causes incorrect results, or that they would mishandle
> without a clear error. That is why a new encoding does not, by itself, bump
> the file version. A V2 reader can see an encoding or logical type it does
> not understand and fail on that column


This seems to directly contradict the wording in Julien's doc:

"Readers need a simple mechanism to know whether they support **all**
features they may encounter when reading a specific file."

I think a new encoding is a feature?

I also think your interpretation has different tradeoffs on requirements.
It makes it harder to generate a meaningful error message for the user (and
therefore let them know what library version they need to upgrade to) if we
are tagging a file with a version different then what is configured as the
writer.

The thing that is clear from all the replies on this thread, is that a lot
of people have different opinions on:
1.  What version gets written to the file (and where it gets written).
2.  What requirements readers have for reading/interpreting the version.


I'd recommend  before debating solutions we update the document with:
1.  Requirements for what we are trying to achieve when recording feature
information in the file (I already have a proposed suggestion for the ones
I think are important but of course I'm sure there will be different
opinions and can try to expand the ones that aren't clear based on Ryan's
feedback).
2.  How the proposed  implementation options in the document map to those
requirements, along with worked concrete examples, for each of the changes
that have been discussed on the thread.

We can then see which options actually make sense or if there are other
alternatives designs that might meet the requirements.

Thanks,
Micah







On Fri, Sep 4, 2026 at 2:20 PM Russell Spitzer <ru...@gmail.com>
wrote:

> Let me try to clarify, as I still think we are talking about different
> things.
>
> The version marker on a file is the oldest format version whose structure
> that file actually uses. A file labeled V3 contains a structure a V2 reader
> cannot correctly parse, or a semantic change it will get wrong. There is no
> partial-success path for a V2 reader on that file. (I believe this is the
> point Micah was trying to probe.) If a V2 reader could read it at all,
> labeling it V3 would be incorrect; the writer should have labeled it V2.
> The file becomes V3 when it contains a breaking change that older readers
> cannot handle, causes incorrect results, or that they would mishandle
> without a clear error. That is why a new encoding does not, by itself, bump
> the file version. A V2 reader can see an encoding or logical type it does
> not understand and fail on that column.
>
> Separately, “this engine supports V3” is a claim about the reader, not a
> stamp on every file that uses a V3-owned feature. A V3 reader is expected
> to handle the V3 bundle (including encodings that may also appear as
> preview in V2 files) by default. The encoding is owned by V3; the file is
> only V3 if it also contains the breaking change.
>
> If anyone has had different interpretations please let me know, but this
> has been my understanding
>
>
> On Fri, Sep 4, 2026 at 3:32 PM Antoine Pitrou <an...@python.org> wrote:
>
> >
> > Hey Ryan,
> >
> > Let's get the process issue straight. We voted A, not A + B. Perhaps you
> > hoped that A implies B, but that is your personal opinion on the topic.
> >
> > (as a reminder, you were the one who insisted on a first vote on a very
> > minimal proposition, before discussing the specifics)
> >
> > Having the community accept B implies another discussion and a vote. And
> > that's the discussion we're having currently, actually, so I don't even
> > know why you seem to be objecting.
> >
> > Regards
> >
> > Antoine.
> >
> >
> >
> > Le 04/09/2026 à 22:02, Ryan Blue a écrit :
> > >  From the doc, here's the point that we've been discussing:
> > >
> > >> Readers need a simple mechanism to know whether they support all
> > features
> > > they may encounter when reading a specific file.
> > >
> > > This is a follow-up to decide between options:
> > > - Set footer version to 3 (for example)
> > > - Set footer version to 3+ when preview features are present
> > > - Set footer version to 3/3+ and record preview feature names
> > > - Use magic bytes for version - PAR3
> > >
> > >  From this, it appears to me that most of us (at least those of us
> > > discussing it) have been working under the assumption that our vote
> meant
> > > using the version by writing it into files. Personally, I think this is
> > > also a natural conclusion. A reader needs to know what version a file
> was
> > > written with in order to decide whether it has guarantees (which is
> > another
> > > way to say what's in the versioning doc).
> > >
> > > I don't think that this is a dishonest or bad faith argument.
> > >
> > > My suggestion to ask the community is really to clarify what we all
> think
> > > we voted on. If there was genuine confusion about this then let's talk
> > > through it. But, again, I don't think that this was unclear to most
> > people.
> > >
> > > Do you agree that it's reasonable to ask what people thought they were
> > > agreeing to?
> > >
> > > Ryan
> > >
> > > On Fri, Sep 4, 2026 at 12:26 PM Antoine Pitrou <an...@python.org>
> > wrote:
> > >
> > >>
> > >> Le 04/09/2026 à 20:35, Ryan Blue a écrit :
> > >>> It sounds like you're suggesting that we don't version data files. I
> > >> don't
> > >>> think that makes sense with versioning and I'm pretty sure most
> people
> > >> that
> > >>> voted to use versions would agree with me.
> > >>
> > >> The *only* implication of the vote was to move forward with the usage
> of
> > >> community-agreed versions (rather than, say, mechanistic
> calendar-based
> > >> presets) to signal levels of compatibility between readers and
> writers.
> > >>
> > >> If you wanted the vote to mean something more, then you should have
> had
> > >> the vote explicitly mention it. Anything else is dishonest.
> > >>
> > >>> Should I start a thread dedicated to asking the
> > >>> community whether most people thought that "using versions" would
> mean
> > >>> writing the version used for a file in the file metadata?
> > >>
> > >> No, you should wait for this discussion to happen in the hope of
> > >> reaching a consensus.
> > >>
> > >> Personally, I don't think readers should up front refuse reading a
> file
> > >> that has a version number higher than their official compatibility
> > >> level. Other people (Micah, Will, at least) see to be of the same
> > opinion.
> > >>
> > >> Regards
> > >>
> > >> Antoine.
> > >>
> > >>
> > >>
> > >
> >
> >
> >
>

Re: [DISCUSS] Finalizing the versioning proposal

[Micah Kornfield <em...@gmail.com>]

Hi Russell,

The version marker on a file is the oldest format version whose structure
> that file actually uses. A file labeled V3 contains a structure a V2 reader
> cannot correctly parse, or a semantic change it will get wrong. There is no
> partial-success path for a V2 reader on that file. (I believe this is the
> point Micah was trying to probe.) If a V2 reader could read it at all,
> labeling it V3 would be incorrect; the writer should have labeled it V2.
> The file becomes V3 when it contains a breaking change that older readers
> cannot handle, causes incorrect results, or that they would mishandle
> without a clear error. That is why a new encoding does not, by itself, bump
> the file version. A V2 reader can see an encoding or logical type it does
> not understand and fail on that column


This seems to directly contradict the wording in Julien's doc:

"Readers need a simple mechanism to know whether they support **all**
features they may encounter when reading a specific file."

I think a new encoding is a feature?

I also think your interpretation has different tradeoffs on requirements.
It makes it harder to generate a meaningful error message for the user (and
therefore let them know what library version they need to upgrade to) if we
are tagging a file with a version different then what is configured as the
writer.

The thing that is clear from all the replies on this thread, is that a lot
of people have different opinions on:
1.  What version gets written to the file (and where it gets written).
2.  What requirements readers have for reading/interpreting the version.


I'd recommend  before debating solutions we update the document with:
1.  Requirements for what we are trying to achieve when recording feature
information in the file (I already have a proposed suggestion for the ones
I think are important but of course I'm sure there will be different
opinions and can try to expand the ones that aren't clear based on Ryan's
feedback).
2.  How the proposed  implementation options in the document map to those
requirements, along with worked concrete examples, for each of the changes
that have been discussed on the thread.

We can then see which options actually make sense or if there are other
alternatives designs that might meet the requirements.

Thanks,
Micah







On Fri, Sep 4, 2026 at 2:20 PM Russell Spitzer <ru...@gmail.com>
wrote:

> Let me try to clarify, as I still think we are talking about different
> things.
>
> The version marker on a file is the oldest format version whose structure
> that file actually uses. A file labeled V3 contains a structure a V2 reader
> cannot correctly parse, or a semantic change it will get wrong. There is no
> partial-success path for a V2 reader on that file. (I believe this is the
> point Micah was trying to probe.) If a V2 reader could read it at all,
> labeling it V3 would be incorrect; the writer should have labeled it V2.
> The file becomes V3 when it contains a breaking change that older readers
> cannot handle, causes incorrect results, or that they would mishandle
> without a clear error. That is why a new encoding does not, by itself, bump
> the file version. A V2 reader can see an encoding or logical type it does
> not understand and fail on that column.
>
> Separately, “this engine supports V3” is a claim about the reader, not a
> stamp on every file that uses a V3-owned feature. A V3 reader is expected
> to handle the V3 bundle (including encodings that may also appear as
> preview in V2 files) by default. The encoding is owned by V3; the file is
> only V3 if it also contains the breaking change.
>
> If anyone has had different interpretations please let me know, but this
> has been my understanding
>
>
> On Fri, Sep 4, 2026 at 3:32 PM Antoine Pitrou <an...@python.org> wrote:
>
> >
> > Hey Ryan,
> >
> > Let's get the process issue straight. We voted A, not A + B. Perhaps you
> > hoped that A implies B, but that is your personal opinion on the topic.
> >
> > (as a reminder, you were the one who insisted on a first vote on a very
> > minimal proposition, before discussing the specifics)
> >
> > Having the community accept B implies another discussion and a vote. And
> > that's the discussion we're having currently, actually, so I don't even
> > know why you seem to be objecting.
> >
> > Regards
> >
> > Antoine.
> >
> >
> >
> > Le 04/09/2026 à 22:02, Ryan Blue a écrit :
> > >  From the doc, here's the point that we've been discussing:
> > >
> > >> Readers need a simple mechanism to know whether they support all
> > features
> > > they may encounter when reading a specific file.
> > >
> > > This is a follow-up to decide between options:
> > > - Set footer version to 3 (for example)
> > > - Set footer version to 3+ when preview features are present
> > > - Set footer version to 3/3+ and record preview feature names
> > > - Use magic bytes for version - PAR3
> > >
> > >  From this, it appears to me that most of us (at least those of us
> > > discussing it) have been working under the assumption that our vote
> meant
> > > using the version by writing it into files. Personally, I think this is
> > > also a natural conclusion. A reader needs to know what version a file
> was
> > > written with in order to decide whether it has guarantees (which is
> > another
> > > way to say what's in the versioning doc).
> > >
> > > I don't think that this is a dishonest or bad faith argument.
> > >
> > > My suggestion to ask the community is really to clarify what we all
> think
> > > we voted on. If there was genuine confusion about this then let's talk
> > > through it. But, again, I don't think that this was unclear to most
> > people.
> > >
> > > Do you agree that it's reasonable to ask what people thought they were
> > > agreeing to?
> > >
> > > Ryan
> > >
> > > On Fri, Sep 4, 2026 at 12:26 PM Antoine Pitrou <an...@python.org>
> > wrote:
> > >
> > >>
> > >> Le 04/09/2026 à 20:35, Ryan Blue a écrit :
> > >>> It sounds like you're suggesting that we don't version data files. I
> > >> don't
> > >>> think that makes sense with versioning and I'm pretty sure most
> people
> > >> that
> > >>> voted to use versions would agree with me.
> > >>
> > >> The *only* implication of the vote was to move forward with the usage
> of
> > >> community-agreed versions (rather than, say, mechanistic
> calendar-based
> > >> presets) to signal levels of compatibility between readers and
> writers.
> > >>
> > >> If you wanted the vote to mean something more, then you should have
> had
> > >> the vote explicitly mention it. Anything else is dishonest.
> > >>
> > >>> Should I start a thread dedicated to asking the
> > >>> community whether most people thought that "using versions" would
> mean
> > >>> writing the version used for a file in the file metadata?
> > >>
> > >> No, you should wait for this discussion to happen in the hope of
> > >> reaching a consensus.
> > >>
> > >> Personally, I don't think readers should up front refuse reading a
> file
> > >> that has a version number higher than their official compatibility
> > >> level. Other people (Micah, Will, at least) see to be of the same
> > opinion.
> > >>
> > >> Regards
> > >>
> > >> Antoine.
> > >>
> > >>
> > >>
> > >
> >
> >
> >
>

Re: [DISCUSS] Finalizing the versioning proposal

[Andrew Lamb <an...@gmail.com>]

> Personally, I don't think readers should up front refuse reading a file
> that has a version number higher than their official compatibility
> level. Other people (Micah, Will, at least) see to be of the same opinion.

In my mind, writing a version number into a file does not obligate any
particular behavior in the reader; It does not have to refuse reading any
files.

A version number summarizes, in a single number, what features may be
encountered in that file. The reader is perfectly free to try and read
files for which it may not support all features.

The only obligation to readers would be that if they say "supports Apache
Parquet V3" they do indeed support all features of V3 (or whatever version)

Andrew

p.s. I really like Will's formulation of the tension between long term
stable storage and platform vendor upgrade cycles.


On Fri, Sep 4, 2026 at 5:20 PM Russell Spitzer <ru...@gmail.com>
wrote:

> Let me try to clarify, as I still think we are talking about different
> things.
>
> The version marker on a file is the oldest format version whose structure
> that file actually uses. A file labeled V3 contains a structure a V2 reader
> cannot correctly parse, or a semantic change it will get wrong. There is no
> partial-success path for a V2 reader on that file. (I believe this is the
> point Micah was trying to probe.) If a V2 reader could read it at all,
> labeling it V3 would be incorrect; the writer should have labeled it V2.
> The file becomes V3 when it contains a breaking change that older readers
> cannot handle, causes incorrect results, or that they would mishandle
> without a clear error. That is why a new encoding does not, by itself, bump
> the file version. A V2 reader can see an encoding or logical type it does
> not understand and fail on that column.
>
> Separately, “this engine supports V3” is a claim about the reader, not a
> stamp on every file that uses a V3-owned feature. A V3 reader is expected
> to handle the V3 bundle (including encodings that may also appear as
> preview in V2 files) by default. The encoding is owned by V3; the file is
> only V3 if it also contains the breaking change.
>
> If anyone has had different interpretations please let me know, but this
> has been my understanding
>
>
> On Fri, Sep 4, 2026 at 3:32 PM Antoine Pitrou <an...@python.org> wrote:
>
> >
> > Hey Ryan,
> >
> > Let's get the process issue straight. We voted A, not A + B. Perhaps you
> > hoped that A implies B, but that is your personal opinion on the topic.
> >
> > (as a reminder, you were the one who insisted on a first vote on a very
> > minimal proposition, before discussing the specifics)
> >
> > Having the community accept B implies another discussion and a vote. And
> > that's the discussion we're having currently, actually, so I don't even
> > know why you seem to be objecting.
> >
> > Regards
> >
> > Antoine.
> >
> >
> >
> > Le 04/09/2026 à 22:02, Ryan Blue a écrit :
> > >  From the doc, here's the point that we've been discussing:
> > >
> > >> Readers need a simple mechanism to know whether they support all
> > features
> > > they may encounter when reading a specific file.
> > >
> > > This is a follow-up to decide between options:
> > > - Set footer version to 3 (for example)
> > > - Set footer version to 3+ when preview features are present
> > > - Set footer version to 3/3+ and record preview feature names
> > > - Use magic bytes for version - PAR3
> > >
> > >  From this, it appears to me that most of us (at least those of us
> > > discussing it) have been working under the assumption that our vote
> meant
> > > using the version by writing it into files. Personally, I think this is
> > > also a natural conclusion. A reader needs to know what version a file
> was
> > > written with in order to decide whether it has guarantees (which is
> > another
> > > way to say what's in the versioning doc).
> > >
> > > I don't think that this is a dishonest or bad faith argument.
> > >
> > > My suggestion to ask the community is really to clarify what we all
> think
> > > we voted on. If there was genuine confusion about this then let's talk
> > > through it. But, again, I don't think that this was unclear to most
> > people.
> > >
> > > Do you agree that it's reasonable to ask what people thought they were
> > > agreeing to?
> > >
> > > Ryan
> > >
> > > On Fri, Sep 4, 2026 at 12:26 PM Antoine Pitrou <an...@python.org>
> > wrote:
> > >
> > >>
> > >> Le 04/09/2026 à 20:35, Ryan Blue a écrit :
> > >>> It sounds like you're suggesting that we don't version data files. I
> > >> don't
> > >>> think that makes sense with versioning and I'm pretty sure most
> people
> > >> that
> > >>> voted to use versions would agree with me.
> > >>
> > >> The *only* implication of the vote was to move forward with the usage
> of
> > >> community-agreed versions (rather than, say, mechanistic
> calendar-based
> > >> presets) to signal levels of compatibility between readers and
> writers.
> > >>
> > >> If you wanted the vote to mean something more, then you should have
> had
> > >> the vote explicitly mention it. Anything else is dishonest.
> > >>
> > >>> Should I start a thread dedicated to asking the
> > >>> community whether most people thought that "using versions" would
> mean
> > >>> writing the version used for a file in the file metadata?
> > >>
> > >> No, you should wait for this discussion to happen in the hope of
> > >> reaching a consensus.
> > >>
> > >> Personally, I don't think readers should up front refuse reading a
> file
> > >> that has a version number higher than their official compatibility
> > >> level. Other people (Micah, Will, at least) see to be of the same
> > opinion.
> > >>
> > >> Regards
> > >>
> > >> Antoine.
> > >>
> > >>
> > >>
> > >
> >
> >
> >
>

Re: [DISCUSS] Finalizing the versioning proposal

[Russell Spitzer <ru...@gmail.com>]

Let me try to clarify, as I still think we are talking about different
things.

The version marker on a file is the oldest format version whose structure
that file actually uses. A file labeled V3 contains a structure a V2 reader
cannot correctly parse, or a semantic change it will get wrong. There is no
partial-success path for a V2 reader on that file. (I believe this is the
point Micah was trying to probe.) If a V2 reader could read it at all,
labeling it V3 would be incorrect; the writer should have labeled it V2.
The file becomes V3 when it contains a breaking change that older readers
cannot handle, causes incorrect results, or that they would mishandle
without a clear error. That is why a new encoding does not, by itself, bump
the file version. A V2 reader can see an encoding or logical type it does
not understand and fail on that column.

Separately, “this engine supports V3” is a claim about the reader, not a
stamp on every file that uses a V3-owned feature. A V3 reader is expected
to handle the V3 bundle (including encodings that may also appear as
preview in V2 files) by default. The encoding is owned by V3; the file is
only V3 if it also contains the breaking change.

If anyone has had different interpretations please let me know, but this
has been my understanding


On Fri, Sep 4, 2026 at 3:32 PM Antoine Pitrou <an...@python.org> wrote:

>
> Hey Ryan,
>
> Let's get the process issue straight. We voted A, not A + B. Perhaps you
> hoped that A implies B, but that is your personal opinion on the topic.
>
> (as a reminder, you were the one who insisted on a first vote on a very
> minimal proposition, before discussing the specifics)
>
> Having the community accept B implies another discussion and a vote. And
> that's the discussion we're having currently, actually, so I don't even
> know why you seem to be objecting.
>
> Regards
>
> Antoine.
>
>
>
> Le 04/09/2026 à 22:02, Ryan Blue a écrit :
> >  From the doc, here's the point that we've been discussing:
> >
> >> Readers need a simple mechanism to know whether they support all
> features
> > they may encounter when reading a specific file.
> >
> > This is a follow-up to decide between options:
> > - Set footer version to 3 (for example)
> > - Set footer version to 3+ when preview features are present
> > - Set footer version to 3/3+ and record preview feature names
> > - Use magic bytes for version - PAR3
> >
> >  From this, it appears to me that most of us (at least those of us
> > discussing it) have been working under the assumption that our vote meant
> > using the version by writing it into files. Personally, I think this is
> > also a natural conclusion. A reader needs to know what version a file was
> > written with in order to decide whether it has guarantees (which is
> another
> > way to say what's in the versioning doc).
> >
> > I don't think that this is a dishonest or bad faith argument.
> >
> > My suggestion to ask the community is really to clarify what we all think
> > we voted on. If there was genuine confusion about this then let's talk
> > through it. But, again, I don't think that this was unclear to most
> people.
> >
> > Do you agree that it's reasonable to ask what people thought they were
> > agreeing to?
> >
> > Ryan
> >
> > On Fri, Sep 4, 2026 at 12:26 PM Antoine Pitrou <an...@python.org>
> wrote:
> >
> >>
> >> Le 04/09/2026 à 20:35, Ryan Blue a écrit :
> >>> It sounds like you're suggesting that we don't version data files. I
> >> don't
> >>> think that makes sense with versioning and I'm pretty sure most people
> >> that
> >>> voted to use versions would agree with me.
> >>
> >> The *only* implication of the vote was to move forward with the usage of
> >> community-agreed versions (rather than, say, mechanistic calendar-based
> >> presets) to signal levels of compatibility between readers and writers.
> >>
> >> If you wanted the vote to mean something more, then you should have had
> >> the vote explicitly mention it. Anything else is dishonest.
> >>
> >>> Should I start a thread dedicated to asking the
> >>> community whether most people thought that "using versions" would mean
> >>> writing the version used for a file in the file metadata?
> >>
> >> No, you should wait for this discussion to happen in the hope of
> >> reaching a consensus.
> >>
> >> Personally, I don't think readers should up front refuse reading a file
> >> that has a version number higher than their official compatibility
> >> level. Other people (Micah, Will, at least) see to be of the same
> opinion.
> >>
> >> Regards
> >>
> >> Antoine.
> >>
> >>
> >>
> >
>
>
>

Re: [DISCUSS] Finalizing the versioning proposal

[Russell Spitzer <ru...@gmail.com>]

Let me try to clarify, as I still think we are talking about different
things.

The version marker on a file is the oldest format version whose structure
that file actually uses. A file labeled V3 contains a structure a V2 reader
cannot correctly parse, or a semantic change it will get wrong. There is no
partial-success path for a V2 reader on that file. (I believe this is the
point Micah was trying to probe.) If a V2 reader could read it at all,
labeling it V3 would be incorrect; the writer should have labeled it V2.
The file becomes V3 when it contains a breaking change that older readers
cannot handle, causes incorrect results, or that they would mishandle
without a clear error. That is why a new encoding does not, by itself, bump
the file version. A V2 reader can see an encoding or logical type it does
not understand and fail on that column.

Separately, “this engine supports V3” is a claim about the reader, not a
stamp on every file that uses a V3-owned feature. A V3 reader is expected
to handle the V3 bundle (including encodings that may also appear as
preview in V2 files) by default. The encoding is owned by V3; the file is
only V3 if it also contains the breaking change.

If anyone has had different interpretations please let me know, but this
has been my understanding


On Fri, Sep 4, 2026 at 3:32 PM Antoine Pitrou <an...@python.org> wrote:

>
> Hey Ryan,
>
> Let's get the process issue straight. We voted A, not A + B. Perhaps you
> hoped that A implies B, but that is your personal opinion on the topic.
>
> (as a reminder, you were the one who insisted on a first vote on a very
> minimal proposition, before discussing the specifics)
>
> Having the community accept B implies another discussion and a vote. And
> that's the discussion we're having currently, actually, so I don't even
> know why you seem to be objecting.
>
> Regards
>
> Antoine.
>
>
>
> Le 04/09/2026 à 22:02, Ryan Blue a écrit :
> >  From the doc, here's the point that we've been discussing:
> >
> >> Readers need a simple mechanism to know whether they support all
> features
> > they may encounter when reading a specific file.
> >
> > This is a follow-up to decide between options:
> > - Set footer version to 3 (for example)
> > - Set footer version to 3+ when preview features are present
> > - Set footer version to 3/3+ and record preview feature names
> > - Use magic bytes for version - PAR3
> >
> >  From this, it appears to me that most of us (at least those of us
> > discussing it) have been working under the assumption that our vote meant
> > using the version by writing it into files. Personally, I think this is
> > also a natural conclusion. A reader needs to know what version a file was
> > written with in order to decide whether it has guarantees (which is
> another
> > way to say what's in the versioning doc).
> >
> > I don't think that this is a dishonest or bad faith argument.
> >
> > My suggestion to ask the community is really to clarify what we all think
> > we voted on. If there was genuine confusion about this then let's talk
> > through it. But, again, I don't think that this was unclear to most
> people.
> >
> > Do you agree that it's reasonable to ask what people thought they were
> > agreeing to?
> >
> > Ryan
> >
> > On Fri, Sep 4, 2026 at 12:26 PM Antoine Pitrou <an...@python.org>
> wrote:
> >
> >>
> >> Le 04/09/2026 à 20:35, Ryan Blue a écrit :
> >>> It sounds like you're suggesting that we don't version data files. I
> >> don't
> >>> think that makes sense with versioning and I'm pretty sure most people
> >> that
> >>> voted to use versions would agree with me.
> >>
> >> The *only* implication of the vote was to move forward with the usage of
> >> community-agreed versions (rather than, say, mechanistic calendar-based
> >> presets) to signal levels of compatibility between readers and writers.
> >>
> >> If you wanted the vote to mean something more, then you should have had
> >> the vote explicitly mention it. Anything else is dishonest.
> >>
> >>> Should I start a thread dedicated to asking the
> >>> community whether most people thought that "using versions" would mean
> >>> writing the version used for a file in the file metadata?
> >>
> >> No, you should wait for this discussion to happen in the hope of
> >> reaching a consensus.
> >>
> >> Personally, I don't think readers should up front refuse reading a file
> >> that has a version number higher than their official compatibility
> >> level. Other people (Micah, Will, at least) see to be of the same
> opinion.
> >>
> >> Regards
> >>
> >> Antoine.
> >>
> >>
> >>
> >
>
>
>

Re: [DISCUSS] Finalizing the versioning proposal

[Antoine Pitrou <an...@python.org>]

Hey Ryan,

Let's get the process issue straight. We voted A, not A + B. Perhaps you 
hoped that A implies B, but that is your personal opinion on the topic.

(as a reminder, you were the one who insisted on a first vote on a very 
minimal proposition, before discussing the specifics)

Having the community accept B implies another discussion and a vote. And 
that's the discussion we're having currently, actually, so I don't even 
know why you seem to be objecting.

Regards

Antoine.



Le 04/09/2026 à 22:02, Ryan Blue a écrit :
>  From the doc, here's the point that we've been discussing:
> 
>> Readers need a simple mechanism to know whether they support all features
> they may encounter when reading a specific file.
> 
> This is a follow-up to decide between options:
> - Set footer version to 3 (for example)
> - Set footer version to 3+ when preview features are present
> - Set footer version to 3/3+ and record preview feature names
> - Use magic bytes for version - PAR3
> 
>  From this, it appears to me that most of us (at least those of us
> discussing it) have been working under the assumption that our vote meant
> using the version by writing it into files. Personally, I think this is
> also a natural conclusion. A reader needs to know what version a file was
> written with in order to decide whether it has guarantees (which is another
> way to say what's in the versioning doc).
> 
> I don't think that this is a dishonest or bad faith argument.
> 
> My suggestion to ask the community is really to clarify what we all think
> we voted on. If there was genuine confusion about this then let's talk
> through it. But, again, I don't think that this was unclear to most people.
> 
> Do you agree that it's reasonable to ask what people thought they were
> agreeing to?
> 
> Ryan
> 
> On Fri, Sep 4, 2026 at 12:26 PM Antoine Pitrou <an...@python.org> wrote:
> 
>>
>> Le 04/09/2026 à 20:35, Ryan Blue a écrit :
>>> It sounds like you're suggesting that we don't version data files. I
>> don't
>>> think that makes sense with versioning and I'm pretty sure most people
>> that
>>> voted to use versions would agree with me.
>>
>> The *only* implication of the vote was to move forward with the usage of
>> community-agreed versions (rather than, say, mechanistic calendar-based
>> presets) to signal levels of compatibility between readers and writers.
>>
>> If you wanted the vote to mean something more, then you should have had
>> the vote explicitly mention it. Anything else is dishonest.
>>
>>> Should I start a thread dedicated to asking the
>>> community whether most people thought that "using versions" would mean
>>> writing the version used for a file in the file metadata?
>>
>> No, you should wait for this discussion to happen in the hope of
>> reaching a consensus.
>>
>> Personally, I don't think readers should up front refuse reading a file
>> that has a version number higher than their official compatibility
>> level. Other people (Micah, Will, at least) see to be of the same opinion.
>>
>> Regards
>>
>> Antoine.
>>
>>
>>
> 

Re: [DISCUSS] Finalizing the versioning proposal

[Ryan Blue <bl...@apache.org>]

From the doc, here's the point that we've been discussing:

> Readers need a simple mechanism to know whether they support all features
they may encounter when reading a specific file.

This is a follow-up to decide between options:
- Set footer version to 3 (for example)
- Set footer version to 3+ when preview features are present
- Set footer version to 3/3+ and record preview feature names
- Use magic bytes for version - PAR3

From this, it appears to me that most of us (at least those of us
discussing it) have been working under the assumption that our vote meant
using the version by writing it into files. Personally, I think this is
also a natural conclusion. A reader needs to know what version a file was
written with in order to decide whether it has guarantees (which is another
way to say what's in the versioning doc).

I don't think that this is a dishonest or bad faith argument.

My suggestion to ask the community is really to clarify what we all think
we voted on. If there was genuine confusion about this then let's talk
through it. But, again, I don't think that this was unclear to most people.

Do you agree that it's reasonable to ask what people thought they were
agreeing to?

Ryan

On Fri, Sep 4, 2026 at 12:26 PM Antoine Pitrou <an...@python.org> wrote:

>
> Le 04/09/2026 à 20:35, Ryan Blue a écrit :
> > It sounds like you're suggesting that we don't version data files. I
> don't
> > think that makes sense with versioning and I'm pretty sure most people
> that
> > voted to use versions would agree with me.
>
> The *only* implication of the vote was to move forward with the usage of
> community-agreed versions (rather than, say, mechanistic calendar-based
> presets) to signal levels of compatibility between readers and writers.
>
> If you wanted the vote to mean something more, then you should have had
> the vote explicitly mention it. Anything else is dishonest.
>
> > Should I start a thread dedicated to asking the
> > community whether most people thought that "using versions" would mean
> > writing the version used for a file in the file metadata?
>
> No, you should wait for this discussion to happen in the hope of
> reaching a consensus.
>
> Personally, I don't think readers should up front refuse reading a file
> that has a version number higher than their official compatibility
> level. Other people (Micah, Will, at least) see to be of the same opinion.
>
> Regards
>
> Antoine.
>
>
>

Re: [DISCUSS] Finalizing the versioning proposal

[Antoine Pitrou <an...@python.org>]

Le 04/09/2026 à 20:35, Ryan Blue a écrit :
> It sounds like you're suggesting that we don't version data files. I don't
> think that makes sense with versioning and I'm pretty sure most people that
> voted to use versions would agree with me.

The *only* implication of the vote was to move forward with the usage of 
community-agreed versions (rather than, say, mechanistic calendar-based 
presets) to signal levels of compatibility between readers and writers.

If you wanted the vote to mean something more, then you should have had 
the vote explicitly mention it. Anything else is dishonest.

> Should I start a thread dedicated to asking the
> community whether most people thought that "using versions" would mean
> writing the version used for a file in the file metadata?

No, you should wait for this discussion to happen in the hope of 
reaching a consensus.

Personally, I don't think readers should up front refuse reading a file 
that has a version number higher than their official compatibility 
level. Other people (Micah, Will, at least) see to be of the same opinion.

Regards

Antoine.

Re: [DISCUSS] Finalizing the versioning proposal

[Ryan Blue <bl...@apache.org>]

> Which "line of reasoning"?

You said, "why is it easier if the version number is written in the file,
than simply by trying to read the file?"

We've decided to use versions to coordinate between readers and writers,
with the guarantee that forward-incompatible changes will not go into an
existing closed/adopted version. That doesn't work without communicating
the version used for a data file because the version defines the
compatibility.

To me, asking whether we really want to use a version (vs just trying to
read) is revisiting the question of whether we want to use a version. And
we do want versions. We decided that with a vote.

> What is "this"?

We have decided by a community vote to use a version number to bundle
changes together. Sorry for the confusion, I thought it was clear from the
context.

> We have decided to adopt a versioning scheme. We have not decided to
> write a version number in the footer and that is what I am discussing
here.

I think that most of us think that using a versioning scheme requires
writing those versions into files. Otherwise you have no idea what
guarantees from that versioning scheme are available to a reader.

It sounds like you're suggesting that we don't version data files. I don't
think that makes sense with versioning and I'm pretty sure most people that
voted to use versions would agree with me. That's why I'm asking you to let
us move forward. No one else I've spoken with is surprised by needing to
communicate a file's version. And this has come up in several of the
discussion threads so I'm struggling to see how it has not been clear.

We need to move forward. Should I start a thread dedicated to asking the
community whether most people thought that "using versions" would mean
writing the version used for a file in the file metadata?

Ryan

On Fri, Sep 4, 2026 at 12:41 AM Antoine Pitrou <an...@python.org> wrote:

> Le 03/09/2026 à 23:28, Ryan Blue a écrit :
> >
> > My interpretation is also that this line of reasoning is advocating for
> not
> > using versioning.
>
> Which "line of reasoning"?
>
> > Antoine, I think this has been clearly decided [...]
>
> What is "this"?
>
> We have decided to adopt a versioning scheme. We have not decided to
> write a version number in the footer and that is what I am discussing here.
>
> Regards
>
> Antoine.
>
>
>

Re: [DISCUSS] Finalizing the versioning proposal

[Antoine Pitrou <an...@python.org>]

Le 03/09/2026 à 23:28, Ryan Blue a écrit :
> 
> My interpretation is also that this line of reasoning is advocating for not
> using versioning.

Which "line of reasoning"?

> Antoine, I think this has been clearly decided [...]

What is "this"?

We have decided to adopt a versioning scheme. We have not decided to 
write a version number in the footer and that is what I am discussing here.

Regards

Antoine.

Re: [DISCUSS] Finalizing the versioning proposal

[Ryan Blue <bl...@apache.org>]

> Is it fair to say you are advocating for not adopting more formal
> versioning, and instead perhaps put our efforts more into communication
and
> education?

My interpretation is also that this line of reasoning is advocating for not
using versioning. I'm concerned that this would set us back at a time when
it is critical for the future of the project to solve this problem.

The community voted to use versioning
<https://lists.apache.org/thread/rd271soqncrskd11kcr174gchd1vzpkc>, after
broad agreement that we need to bundle features together to help coordinate
between readers and writers and make it easier to reason about
compatibility (rather than checking a rather large matrix of changes). I
think it is a mistake to re-open the issue that we just decided by vote. The
purpose of that vote was to decide how we look to address incompatible
changes going forward and we should approach this within the context of
that decision, not reconsider it.

Antoine, I think this has been clearly decided and I would like to ask you
to, please, let the community move forward.

On Thu, Sep 3, 2026 at 3:31 AM Andrew Lamb <an...@gmail.com> wrote:

> > why is it easier if the version number is written in the file, than
> simply by trying to read the file?
>
> In my opinion, it is easier because you can reason/communicate about a
> single thing (version number) and compare that quickly to documented
> compatibility levels in projects/systems rather than having to empirically
> test your file with all your potential readers.
>
> I do agree that such a version could technically be achieved without
> changing what is written in the file (e.g. to make some tool to check what
> features are used), but the check would need to be more complicated than
> simply looking at a version
>
> > The current approach seems to work: don’t write features your readers
> don’t support; readers fail only when an unsupported feature is on the
> critical path and ignore what they don’t need.
>
> I agree the current approach works in the sense that there is wide
> interoperability of parquet readers, but that is achieved by largely using
> the lowest common denominator feature set.  I don't think the current
> approach works particularly well to let people adopt new features
>
> Is it fair to say you are advocating for not adopting more formal
> versioning, and instead perhaps put our efforts more into communication and
> education?
>
> Andrew
>
> On Thu, Sep 3, 2026 at 4:52 AM Will Edwards via dev <
> dev@parquet.apache.org>
> wrote:
>
> > This is a very good point, Antoine!
> >
> > Stepping back, Parquet’s openness comes from ubiquitous interoperability
> > through time and vendor-space, not just from an open specification.
> >
> > The current approach seems to work: don’t write features your readers
> don’t
> > support; readers fail only when an unsupported feature is on the critical
> > path and ignore what they don’t need.
> >
> > I fear versioning might not improve interoperability but would fragment
> > Parquet into incompatible dialects.
> >
> > And the end result - that a user encounters a file with data they cannot
> > read - stays the same whichever way?
> >
> > regards,
> > Will
> >
> > On Thu, 3 Sept 2026 at 08:58, Antoine Pitrou <an...@python.org> wrote:
> >
> > > Le 03/09/2026 à 01:08, Andrew Lamb a écrit :
> > > > I am not sure it is proof, but here is my logic:
> > > >
> > > > Given any particular Parquet file, there is no simple, practical way
> to
> > > > understand what versions of what products/projects can read it.
> > > >
> > > > I understand we have the compatibility chart[1] (which I helped to
> > > create),
> > > > but that doesn't have entries for end user systems (e.g. Spark
> version
> > > 3.5)
> > > > and I believe there is a widely held belief that writers must stick
> to
> > > very
> > > > conservative feature lists (see blogs below)
> > > >
> > > > In my mind, having a single clear version in the file (this file is
> > > written
> > > > with version X Parquet features), makes it clear, unambiguous, and
> easy
> > > to
> > > > verify what systems can read it (e.g. DB X can read parquet files
> with
> > > > version Y).
> > >
> > > My underlying question is: why is it easier if the version number is
> > > written in the file, than simply by trying to read the file?
> > >
> > > (you have to read the file anyway to get the version number that's
> > > written in it :-))
> > >
> > > Regards
> > >
> > > Antoine.
> > >
> > >
> > >
> >
>

Re: [DISCUSS] Writing the version number in the footer

[Andrew Lamb <an...@gmail.com>]

> So, in the end, writing this version number in the footer would mostly be
informational.

Yes, I agree with this framing. Thank you for this clear wording

> I guess that can make it useful anyway, but less than seems to be
believed by the proponents of this feature :-)

Indeed -- I agree there are different opinions on how useful such a version
would be -- and I fully admit I am being optimistic (and perhaps naive)
about its influence to get implementations to align :)

Thank you for the discussion,
Andrew

On Thu, Sep 3, 2026 at 6:48 AM Antoine Pitrou <an...@python.org> wrote:

>
> Le 03/09/2026 à 12:31, Andrew Lamb a écrit :
> >> why is it easier if the version number is written in the file, than
> > simply by trying to read the file?
> >
> > In my opinion, it is easier because you can reason/communicate about a
> > single thing (version number) and compare that quickly to documented
> > compatibility levels in projects/systems rather than having to
> empirically
> > test your file with all your potential readers.
> >
> > I do agree that such a version could technically be achieved without
> > changing what is written in the file (e.g. to make some tool to check
> what
> > features are used), but the check would need to be more complicated than
> > simply looking at a version
>
> I think there are two limitations here:
>
> 1) readers don't necessarily support "only V3" or "all of V3", so a
> simple version flag in the footer is not an exact answer to the question
> "will the reader be able to read the file"?
>
> 2) users can ask for a subset of a file's contents (especially columns),
> such that, for example, a ALP-encoded float column in a Parquet "V3"
> file need not stop a user who only wants to read int and string columns
> from that file with a pre-"V3" Parquet reader.
>
> So, in the end, writing this version number in the footer would mostly
> be informational. I guess that can make it useful anyway, but less than
> seems to be believed by the proponents of this feature :-)
>
> Regards
>
> Antoine.
>
>
>

Re: [DISCUSS] Writing the version number in the footer

[Andrew Lamb <an...@gmail.com>]

> So, in the end, writing this version number in the footer would mostly be
informational.

Yes, I agree with this framing. Thank you for this clear wording

> I guess that can make it useful anyway, but less than seems to be
believed by the proponents of this feature :-)

Indeed -- I agree there are different opinions on how useful such a version
would be -- and I fully admit I am being optimistic (and perhaps naive)
about its influence to get implementations to align :)

Thank you for the discussion,
Andrew

On Thu, Sep 3, 2026 at 6:48 AM Antoine Pitrou <an...@python.org> wrote:

>
> Le 03/09/2026 à 12:31, Andrew Lamb a écrit :
> >> why is it easier if the version number is written in the file, than
> > simply by trying to read the file?
> >
> > In my opinion, it is easier because you can reason/communicate about a
> > single thing (version number) and compare that quickly to documented
> > compatibility levels in projects/systems rather than having to
> empirically
> > test your file with all your potential readers.
> >
> > I do agree that such a version could technically be achieved without
> > changing what is written in the file (e.g. to make some tool to check
> what
> > features are used), but the check would need to be more complicated than
> > simply looking at a version
>
> I think there are two limitations here:
>
> 1) readers don't necessarily support "only V3" or "all of V3", so a
> simple version flag in the footer is not an exact answer to the question
> "will the reader be able to read the file"?
>
> 2) users can ask for a subset of a file's contents (especially columns),
> such that, for example, a ALP-encoded float column in a Parquet "V3"
> file need not stop a user who only wants to read int and string columns
> from that file with a pre-"V3" Parquet reader.
>
> So, in the end, writing this version number in the footer would mostly
> be informational. I guess that can make it useful anyway, but less than
> seems to be believed by the proponents of this feature :-)
>
> Regards
>
> Antoine.
>
>
>

Re: [DISCUSS] Writing the version number in the footer

[Ryan Blue <bl...@apache.org>]

> 1) readers don't necessarily support "only V3" or "all of V3", so a
> simple version flag in the footer is not an exact answer to the question
> "will the reader be able to read the file"?

It sounds like there is some confusion over how versions work, so I'll try
to clarify. Much of this follows from the restrictions that a
forward-compatibility breaking change must be released in a new version and
cannot be added to one that is already closed and adopted. That means that
each format version creates a compatibility level that can be used to
coordinate between readers and writers.

Readers have a guarantee -- from the restriction on forward-breaking
changes -- that if they support a closed / adopted format version, then
they can correctly read all files produced for that format version. If a
reader's documentation states that it supports a version, then producing
that version with any writer means the reader can consume it.

As a result, a reader that supports "only v3" is guaranteed to be able to
correctly read files produced by a writer that supports "all of v3". That's
why what matters is the format version for which a file was written.

> 2) users can ask for a subset of a file's contents (especially columns),
> such that, for example, a ALP-encoded float column in a Parquet "V3"
> file need not stop a user who only wants to read int and string columns
> from that file with a pre-"V3" Parquet reader.

Yes, a file written for a version can use structures that are compatible
with older versions. If a file is completely compatible with an older
version, the writer can label it that way. That's what I would recommend
because a file written to a newer format version can have substantial
changes.

For instance, the "modular footer" work to replace the current footer
structure will break older readers. Older readers can attempt to read newer
format versions, but they have no guarantees that they will be able to read
them correctly. Given that there is no correctness guarantee for older
readers and newer format versions, I think that the correct reader behavior
is to fail when it sees a newer format version.

I should also note that we've also created a way to work around this. New
encodings are well-contained and can be used as preview features, where we
deliberately allow finished features to be used in an older format version.
That way you can use ALP in a v2 file and older readers can correctly read
the non-ALP subset of columns. Or if a reader supports v2 + ALP, it can
read all the columns.

I think that the version system that we've agreed on will work well and has
accounted for these situations in a reasonable and understandable way.

On Thu, Sep 3, 2026 at 3:48 AM Antoine Pitrou <an...@python.org> wrote:

>
> Le 03/09/2026 à 12:31, Andrew Lamb a écrit :
> >> why is it easier if the version number is written in the file, than
> > simply by trying to read the file?
> >
> > In my opinion, it is easier because you can reason/communicate about a
> > single thing (version number) and compare that quickly to documented
> > compatibility levels in projects/systems rather than having to
> empirically
> > test your file with all your potential readers.
> >
> > I do agree that such a version could technically be achieved without
> > changing what is written in the file (e.g. to make some tool to check
> what
> > features are used), but the check would need to be more complicated than
> > simply looking at a version
>
> I think there are two limitations here:
>
> 1) readers don't necessarily support "only V3" or "all of V3", so a
> simple version flag in the footer is not an exact answer to the question
> "will the reader be able to read the file"?
>
> 2) users can ask for a subset of a file's contents (especially columns),
> such that, for example, a ALP-encoded float column in a Parquet "V3"
> file need not stop a user who only wants to read int and string columns
> from that file with a pre-"V3" Parquet reader.
>
> So, in the end, writing this version number in the footer would mostly
> be informational. I guess that can make it useful anyway, but less than
> seems to be believed by the proponents of this feature :-)
>
> Regards
>
> Antoine.
>
>
>

[DISCUSS] Writing the version number in the footer

[Antoine Pitrou <an...@python.org>]

Le 03/09/2026 à 12:31, Andrew Lamb a écrit :
>> why is it easier if the version number is written in the file, than
> simply by trying to read the file?
> 
> In my opinion, it is easier because you can reason/communicate about a
> single thing (version number) and compare that quickly to documented
> compatibility levels in projects/systems rather than having to empirically
> test your file with all your potential readers.
> 
> I do agree that such a version could technically be achieved without
> changing what is written in the file (e.g. to make some tool to check what
> features are used), but the check would need to be more complicated than
> simply looking at a version

I think there are two limitations here:

1) readers don't necessarily support "only V3" or "all of V3", so a 
simple version flag in the footer is not an exact answer to the question 
"will the reader be able to read the file"?

2) users can ask for a subset of a file's contents (especially columns), 
such that, for example, a ALP-encoded float column in a Parquet "V3" 
file need not stop a user who only wants to read int and string columns 
from that file with a pre-"V3" Parquet reader.

So, in the end, writing this version number in the footer would mostly 
be informational. I guess that can make it useful anyway, but less than 
seems to be believed by the proponents of this feature :-)

Regards

Antoine.

Re: [DISCUSS] Finalizing the versioning proposal

[Andrew Lamb <an...@gmail.com>]

> why is it easier if the version number is written in the file, than
simply by trying to read the file?

In my opinion, it is easier because you can reason/communicate about a
single thing (version number) and compare that quickly to documented
compatibility levels in projects/systems rather than having to empirically
test your file with all your potential readers.

I do agree that such a version could technically be achieved without
changing what is written in the file (e.g. to make some tool to check what
features are used), but the check would need to be more complicated than
simply looking at a version

> The current approach seems to work: don’t write features your readers
don’t support; readers fail only when an unsupported feature is on the
critical path and ignore what they don’t need.

I agree the current approach works in the sense that there is wide
interoperability of parquet readers, but that is achieved by largely using
the lowest common denominator feature set.  I don't think the current
approach works particularly well to let people adopt new features

Is it fair to say you are advocating for not adopting more formal
versioning, and instead perhaps put our efforts more into communication and
education?

Andrew

On Thu, Sep 3, 2026 at 4:52 AM Will Edwards via dev <de...@parquet.apache.org>
wrote:

> This is a very good point, Antoine!
>
> Stepping back, Parquet’s openness comes from ubiquitous interoperability
> through time and vendor-space, not just from an open specification.
>
> The current approach seems to work: don’t write features your readers don’t
> support; readers fail only when an unsupported feature is on the critical
> path and ignore what they don’t need.
>
> I fear versioning might not improve interoperability but would fragment
> Parquet into incompatible dialects.
>
> And the end result - that a user encounters a file with data they cannot
> read - stays the same whichever way?
>
> regards,
> Will
>
> On Thu, 3 Sept 2026 at 08:58, Antoine Pitrou <an...@python.org> wrote:
>
> > Le 03/09/2026 à 01:08, Andrew Lamb a écrit :
> > > I am not sure it is proof, but here is my logic:
> > >
> > > Given any particular Parquet file, there is no simple, practical way to
> > > understand what versions of what products/projects can read it.
> > >
> > > I understand we have the compatibility chart[1] (which I helped to
> > create),
> > > but that doesn't have entries for end user systems (e.g. Spark version
> > 3.5)
> > > and I believe there is a widely held belief that writers must stick to
> > very
> > > conservative feature lists (see blogs below)
> > >
> > > In my mind, having a single clear version in the file (this file is
> > written
> > > with version X Parquet features), makes it clear, unambiguous, and easy
> > to
> > > verify what systems can read it (e.g. DB X can read parquet files with
> > > version Y).
> >
> > My underlying question is: why is it easier if the version number is
> > written in the file, than simply by trying to read the file?
> >
> > (you have to read the file anyway to get the version number that's
> > written in it :-))
> >
> > Regards
> >
> > Antoine.
> >
> >
> >
>

Re: [DISCUSS] Finalizing the versioning proposal

[Will Edwards via dev <de...@parquet.apache.org>]

This is a very good point, Antoine!

Stepping back, Parquet’s openness comes from ubiquitous interoperability
through time and vendor-space, not just from an open specification.

The current approach seems to work: don’t write features your readers don’t
support; readers fail only when an unsupported feature is on the critical
path and ignore what they don’t need.

I fear versioning might not improve interoperability but would fragment
Parquet into incompatible dialects.

And the end result - that a user encounters a file with data they cannot
read - stays the same whichever way?

regards,
Will

On Thu, 3 Sept 2026 at 08:58, Antoine Pitrou <an...@python.org> wrote:

> Le 03/09/2026 à 01:08, Andrew Lamb a écrit :
> > I am not sure it is proof, but here is my logic:
> >
> > Given any particular Parquet file, there is no simple, practical way to
> > understand what versions of what products/projects can read it.
> >
> > I understand we have the compatibility chart[1] (which I helped to
> create),
> > but that doesn't have entries for end user systems (e.g. Spark version
> 3.5)
> > and I believe there is a widely held belief that writers must stick to
> very
> > conservative feature lists (see blogs below)
> >
> > In my mind, having a single clear version in the file (this file is
> written
> > with version X Parquet features), makes it clear, unambiguous, and easy
> to
> > verify what systems can read it (e.g. DB X can read parquet files with
> > version Y).
>
> My underlying question is: why is it easier if the version number is
> written in the file, than simply by trying to read the file?
>
> (you have to read the file anyway to get the version number that's
> written in it :-))
>
> Regards
>
> Antoine.
>
>
>

Re: [DISCUSS] Finalizing the versioning proposal

[Antoine Pitrou <an...@python.org>]

Le 03/09/2026 à 01:08, Andrew Lamb a écrit :
> I am not sure it is proof, but here is my logic:
> 
> Given any particular Parquet file, there is no simple, practical way to
> understand what versions of what products/projects can read it.
> 
> I understand we have the compatibility chart[1] (which I helped to create),
> but that doesn't have entries for end user systems (e.g. Spark version 3.5)
> and I believe there is a widely held belief that writers must stick to very
> conservative feature lists (see blogs below)
> 
> In my mind, having a single clear version in the file (this file is written
> with version X Parquet features), makes it clear, unambiguous, and easy to
> verify what systems can read it (e.g. DB X can read parquet files with
> version Y).

My underlying question is: why is it easier if the version number is 
written in the file, than simply by trying to read the file?

(you have to read the file anyway to get the version number that's 
written in it :-))

Regards

Antoine.

Re: [DISCUSS] Finalizing the versioning proposal

[Andrew Lamb <an...@gmail.com>]

I am not sure it is proof, but here is my logic:

Given any particular Parquet file, there is no simple, practical way to
understand what versions of what products/projects can read it.

I understand we have the compatibility chart[1] (which I helped to create),
but that doesn't have entries for end user systems (e.g. Spark version 3.5)
and I believe there is a widely held belief that writers must stick to very
conservative feature lists (see blogs below)

In my mind, having a single clear version in the file (this file is written
with version X Parquet features), makes it clear, unambiguous, and easy to
verify what systems can read it (e.g. DB X can read parquet files with
version Y).

I can see the argument that version number standardization will not
necessarily force vendors to update their implementations or actually
implement the full feature set for a version, but I think it would help
immensely to standardize the terminology. I n perhaps missing what the
alternative is to a clear version identifier in the file (especially given
we already have a version marker in the metadata)

Andrew



Here are some blogs / papers expressing the sentiment that writers can't
use newer features in parquet because there is no way to understand what
set of features potential consumers may support:


https://duckdb.org/2025/01/22/parquet-encodings
Query Engines: Gatekeepers of the Parquet File Format
"...despite these new encodings being available in 1.2.0, DuckDB will not
write them by default. If DuckDB did this, many of our users would have a
frustrating experience because some mainstream query engines still do not
support reading these encodings."

Lance: Efficient Random Access in Columnar Storage through Adaptive
Structural Encodings
https://arxiv.org/abs/2504.15247
"The format is not very clearly versioned, as different implementations
support different sets of optional features" and "For Parquet, especially,
there is a wide gap between what the format can achieve and what users
typically use"

Andrew

[1]: https://parquet.apache.org/docs/file-format/implementationstatus/





On Wed, Sep 2, 2026 at 5:15 PM Antoine Pitrou <an...@python.org> wrote:

>
> Le 02/09/2026 à 19:43, Andrew Lamb a écrit :
> > In my opinion, the core problem we need to solve here is how Parquet
> > writers communicate to Parquet readers which features must be supported
> in
> > order to guarantee they can completely read a given file.
>
> I don't think that's a problem we need to solve.
>
> I would like to have proof that it's a problem before changing my mind.
>
> > The only open question I think is how to encode this
> > version in the file:
> > 1. The magic bytes in the file PAR1, PARE[1], etc.
> > 2. The version field in the metadata[2].
>
> Magic bytes are a very crude mechanism. Also, introducing new magic
> bytes means any third-party file type detection mechanism that relies on
> reading those magic bytes must be updated for the new magic bytes.
>
> We should typically only change the magic bytes if we are introducing an
> incompatible format (for example because we are changing the footer
> format to something else, such as Flatbuffers, or introducing a layer of
> wrapping, such as for encrypted files).
>
> Regards
>
> Antoine.
>
>
>

Re: [DISCUSS] Finalizing the versioning proposal

[Andrew Lamb <an...@gmail.com>]

I am not sure it is proof, but here is my logic:

Given any particular Parquet file, there is no simple, practical way to
understand what versions of what products/projects can read it.

I understand we have the compatibility chart[1] (which I helped to create),
but that doesn't have entries for end user systems (e.g. Spark version 3.5)
and I believe there is a widely held belief that writers must stick to very
conservative feature lists (see blogs below)

In my mind, having a single clear version in the file (this file is written
with version X Parquet features), makes it clear, unambiguous, and easy to
verify what systems can read it (e.g. DB X can read parquet files with
version Y).

I can see the argument that version number standardization will not
necessarily force vendors to update their implementations or actually
implement the full feature set for a version, but I think it would help
immensely to standardize the terminology. I n perhaps missing what the
alternative is to a clear version identifier in the file (especially given
we already have a version marker in the metadata)

Andrew



Here are some blogs / papers expressing the sentiment that writers can't
use newer features in parquet because there is no way to understand what
set of features potential consumers may support:


https://duckdb.org/2025/01/22/parquet-encodings
Query Engines: Gatekeepers of the Parquet File Format
"...despite these new encodings being available in 1.2.0, DuckDB will not
write them by default. If DuckDB did this, many of our users would have a
frustrating experience because some mainstream query engines still do not
support reading these encodings."

Lance: Efficient Random Access in Columnar Storage through Adaptive
Structural Encodings
https://arxiv.org/abs/2504.15247
"The format is not very clearly versioned, as different implementations
support different sets of optional features" and "For Parquet, especially,
there is a wide gap between what the format can achieve and what users
typically use"

Andrew

[1]: https://parquet.apache.org/docs/file-format/implementationstatus/





On Wed, Sep 2, 2026 at 5:15 PM Antoine Pitrou <an...@python.org> wrote:

>
> Le 02/09/2026 à 19:43, Andrew Lamb a écrit :
> > In my opinion, the core problem we need to solve here is how Parquet
> > writers communicate to Parquet readers which features must be supported
> in
> > order to guarantee they can completely read a given file.
>
> I don't think that's a problem we need to solve.
>
> I would like to have proof that it's a problem before changing my mind.
>
> > The only open question I think is how to encode this
> > version in the file:
> > 1. The magic bytes in the file PAR1, PARE[1], etc.
> > 2. The version field in the metadata[2].
>
> Magic bytes are a very crude mechanism. Also, introducing new magic
> bytes means any third-party file type detection mechanism that relies on
> reading those magic bytes must be updated for the new magic bytes.
>
> We should typically only change the magic bytes if we are introducing an
> incompatible format (for example because we are changing the footer
> format to something else, such as Flatbuffers, or introducing a layer of
> wrapping, such as for encrypted files).
>
> Regards
>
> Antoine.
>
>
>

Re: [DISCUSS] Finalizing the versioning proposal

[Antoine Pitrou <an...@python.org>]

Le 02/09/2026 à 19:43, Andrew Lamb a écrit :
> In my opinion, the core problem we need to solve here is how Parquet
> writers communicate to Parquet readers which features must be supported in
> order to guarantee they can completely read a given file.

I don't think that's a problem we need to solve.

I would like to have proof that it's a problem before changing my mind.

> The only open question I think is how to encode this
> version in the file:
> 1. The magic bytes in the file PAR1, PARE[1], etc.
> 2. The version field in the metadata[2].

Magic bytes are a very crude mechanism. Also, introducing new magic 
bytes means any third-party file type detection mechanism that relies on 
reading those magic bytes must be updated for the new magic bytes.

We should typically only change the magic bytes if we are introducing an 
incompatible format (for example because we are changing the footer 
format to something else, such as Flatbuffers, or introducing a layer of 
wrapping, such as for encrypted files).

Regards

Antoine.

Re: [DISCUSS] Finalizing the versioning proposal

[Andrew Lamb <an...@gmail.com>]

In my opinion, the core problem we need to solve here is how Parquet
writers communicate to Parquet readers which features must be supported in
order to guarantee they can completely read a given file.

I thought that we all basically agree that a version is the way to
communicate this. The only open question I think is how to encode this
version in the file:
1. The magic bytes in the file PAR1, PARE[1], etc.
2. The version field in the metadata[2].

I personally think either would work, and choosing either one would be an
improvement over the status quo.

Also, while it is important to work through other cases (e.g. communicating
preview feature inclusion) I think we can treat that as a follow on project
/ vote as it will likely involve a separate mechanism.

Andrew

[1]: https://parquet.apache.org/docs/file-format/
[2]:
https://github.com/apache/parquet-format/blob/a645b5a2cfa1453b49eef2aa0eec5bb4498a69cd/src/main/thrift/parquet.thrift#L1408-L1416

On Wed, Sep 2, 2026 at 12:42 PM Antoine Pitrou <an...@python.org> wrote:

>
> """
> Readers need a simple mechanism to know whether they support all
> features they may encounter when reading a specific file.
> """
>
> I would like to dispute this. No argument is given as to why readers
> need something else than what is already embodied in the metadata.
>
> It's also not obvious why that information is granular enough (some
> features are optional, the user may not want to read all columns anyway,
> etc.).
>
> At least in Arrow, I don't remember a single user complaining that they
> don't have a simpler access to that information.
>
> Regards
>
> Antoine.
>
>
>
> Le 14/08/2026 à 20:25, Julien Le Dem a écrit :
> > Following up on the discussion during the Parquet sync on Wednesday, I
> have
> > added a new tab to the Versioning document
> > <
> https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab=t.nywy9gjvpwqf
> >
> > to discuss 4 remaining decision points for the implementation. For each I
> > have listed options and pros and cons.
> > 1. Mechanism to decide what Parquet features writers should use
> > 2. Mechanism to write Parquet files that use a preview feature
> > 3. Mechanism to record in the file what features it uses
> > 4. Describing reader behavior when reading a file that is not fully
> > supported
> >
> > The stated guiding principle is: We want to minimize prescribing how
> > readers and writers should work and focus on giving them the information
> to
> > implement the behavior they deem appropriate.
> >
> > Please comment in the doc to help finalize consensus.
> > (thank you Andrew for early feedback and helping me think this through!)
>
>
>

Re: [DISCUSS] Finalizing the versioning proposal

[Andrew Lamb <an...@gmail.com>]

In my opinion, the core problem we need to solve here is how Parquet
writers communicate to Parquet readers which features must be supported in
order to guarantee they can completely read a given file.

I thought that we all basically agree that a version is the way to
communicate this. The only open question I think is how to encode this
version in the file:
1. The magic bytes in the file PAR1, PARE[1], etc.
2. The version field in the metadata[2].

I personally think either would work, and choosing either one would be an
improvement over the status quo.

Also, while it is important to work through other cases (e.g. communicating
preview feature inclusion) I think we can treat that as a follow on project
/ vote as it will likely involve a separate mechanism.

Andrew

[1]: https://parquet.apache.org/docs/file-format/
[2]:
https://github.com/apache/parquet-format/blob/a645b5a2cfa1453b49eef2aa0eec5bb4498a69cd/src/main/thrift/parquet.thrift#L1408-L1416

On Wed, Sep 2, 2026 at 12:42 PM Antoine Pitrou <an...@python.org> wrote:

>
> """
> Readers need a simple mechanism to know whether they support all
> features they may encounter when reading a specific file.
> """
>
> I would like to dispute this. No argument is given as to why readers
> need something else than what is already embodied in the metadata.
>
> It's also not obvious why that information is granular enough (some
> features are optional, the user may not want to read all columns anyway,
> etc.).
>
> At least in Arrow, I don't remember a single user complaining that they
> don't have a simpler access to that information.
>
> Regards
>
> Antoine.
>
>
>
> Le 14/08/2026 à 20:25, Julien Le Dem a écrit :
> > Following up on the discussion during the Parquet sync on Wednesday, I
> have
> > added a new tab to the Versioning document
> > <
> https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab=t.nywy9gjvpwqf
> >
> > to discuss 4 remaining decision points for the implementation. For each I
> > have listed options and pros and cons.
> > 1. Mechanism to decide what Parquet features writers should use
> > 2. Mechanism to write Parquet files that use a preview feature
> > 3. Mechanism to record in the file what features it uses
> > 4. Describing reader behavior when reading a file that is not fully
> > supported
> >
> > The stated guiding principle is: We want to minimize prescribing how
> > readers and writers should work and focus on giving them the information
> to
> > implement the behavior they deem appropriate.
> >
> > Please comment in the doc to help finalize consensus.
> > (thank you Andrew for early feedback and helping me think this through!)
>
>
>

Re: [DISCUSS] Finalizing the versioning proposal

[Antoine Pitrou <an...@python.org>]

"""
Readers need a simple mechanism to know whether they support all 
features they may encounter when reading a specific file.
"""

I would like to dispute this. No argument is given as to why readers 
need something else than what is already embodied in the metadata.

It's also not obvious why that information is granular enough (some 
features are optional, the user may not want to read all columns anyway, 
etc.).

At least in Arrow, I don't remember a single user complaining that they 
don't have a simpler access to that information.

Regards

Antoine.



Le 14/08/2026 à 20:25, Julien Le Dem a écrit :
> Following up on the discussion during the Parquet sync on Wednesday, I have
> added a new tab to the Versioning document
> <https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab=t.nywy9gjvpwqf>
> to discuss 4 remaining decision points for the implementation. For each I
> have listed options and pros and cons.
> 1. Mechanism to decide what Parquet features writers should use
> 2. Mechanism to write Parquet files that use a preview feature
> 3. Mechanism to record in the file what features it uses
> 4. Describing reader behavior when reading a file that is not fully
> supported
> 
> The stated guiding principle is: We want to minimize prescribing how
> readers and writers should work and focus on giving them the information to
> implement the behavior they deem appropriate.
> 
> Please comment in the doc to help finalize consensus.
> (thank you Andrew for early feedback and helping me think this through!)

Re: [DISCUSS] Finalizing the versioning proposal

[Daniel Weeks <dw...@apache.org>]

Hey everyone,

This topic was discussed at today's community sync and I offered to write
up a summary of the discussion points.

Disclaimer: there was discussion, not decisions made and want to continue
the discussion on the list.

Links:
- AI summary and Transcript
<https://docs.google.com/document/d/1CbnDTWdLRJ7UWFtPO-6W99_mYJTpu7kI/edit#heading=h.a7xlpmgg0yqx>
- Recording
<https://drive.google.com/file/d/1FZRU7hEldfOZju92bFjJVY7DG_bYqUCH/view?usp=sharing&t=2182.616>
(timestamped to relevent discusion)

Summary and Key Points:

   1. Clarified the vote on grouping features by version and points of
   contention
      1. Many expected that if we group features by version, that intent
      would be captured in the file (in some way)
      2. Capturing the version used in the file is likely not the
      underlying point of contention
      3. The expectations around reader expectations and guarantees are
      more likely the root of the differing views
   2. Views from this thread (and in person syncs) largely fall into three
   categories:
      1. Readers should always attempt to read, regardless of how the data
      was produced (rely on thrift for breaking changes)
      2. Implementations should decide whether to read based on metadata
      (including version)
      3. Reads should strictly gate on versioning and not attempt to read
      newer versions they do not understand
   3. Thrift Versioning vs. Format Versioning Discussion
      1. Many of the examples we discuss
      (encodings/datatypes/forward-compatible changes) are already handled via
      thrift
      2. Thrift can't represent all types of evolutionary changes (semantic
      meaning or structural changes)
      3. Versioned changes are beyond what the project has historically
      addressed (or would require massive and complicated thrift duplication)
   4. Ryan will follow up with specific examples

This is just my interpretation of the discussion.  If you want the full
context, please reference the transcript/recording above.

-Dan




On Wed, Sep 9, 2026 at 2:58 AM Antoine Pitrou <an...@python.org> wrote:

> Le 09/09/2026 à 02:10, Ryan Blue a écrit :
> >
> > And finally I think there's a group of us (Dan, Alkis, and I) who would
> > prefer requiring readers to fail if they encounter a newer format version
> > rather than trying to read it. The main argument for this is to make
> > correctness guarantees. I don't think it is enough to rely on Thrift
> > evolution rules, and I think that attempting to will inevitably lead to
> > correctness bugs when our changes have unforeseen consequences in
> readers.
>
> If such correctness bugs were likely, then surely they would have
> happened already, given all the features we have added in the past?
>
> Do we have any example of correctness bugs introduced in older readers
> by the existence of a new feature?
>
>  > I think that relying on Thrift evolution rules also limits the ways
> that we
>  > can evolve the format and make it significantly harder to implement and
>  > maintain.
>
> I agree that changes that break *existing* semantics need some signaling
> in the Parquet metadata. The problem is that we are conflating such
> changes, with other changes that only introduce *new* semantics.
>
> For example, the ALP encoding is introducing new semantics that won't be
> understood by existing readers. This deserves specific guidance to the
> user for when to produce files with ALP enabled (hence the idea of
> compatibility levels, e.g. "please write a file with V3-level
> features"). But the ALP encoding doesn't threaten interoperability for
> whatever features a pre-ALP reader supports.
>
> Most features we add to the Parquet format are similar to the ALP
> encoding. They can be ignored by older readers.
>
> Regards
>
> Antoine.
>
>
>

Re: [DISCUSS] Finalizing the versioning proposal

[Antoine Pitrou <an...@python.org>]

Le 09/09/2026 à 02:10, Ryan Blue a écrit :
> 
> And finally I think there's a group of us (Dan, Alkis, and I) who would
> prefer requiring readers to fail if they encounter a newer format version
> rather than trying to read it. The main argument for this is to make
> correctness guarantees. I don't think it is enough to rely on Thrift
> evolution rules, and I think that attempting to will inevitably lead to
> correctness bugs when our changes have unforeseen consequences in readers.

If such correctness bugs were likely, then surely they would have 
happened already, given all the features we have added in the past?

Do we have any example of correctness bugs introduced in older readers 
by the existence of a new feature?

 > I think that relying on Thrift evolution rules also limits the ways 
that we
 > can evolve the format and make it significantly harder to implement and
 > maintain.

I agree that changes that break *existing* semantics need some signaling 
in the Parquet metadata. The problem is that we are conflating such 
changes, with other changes that only introduce *new* semantics.

For example, the ALP encoding is introducing new semantics that won't be 
understood by existing readers. This deserves specific guidance to the 
user for when to produce files with ALP enabled (hence the idea of 
compatibility levels, e.g. "please write a file with V3-level 
features"). But the ALP encoding doesn't threaten interoperability for 
whatever features a pre-ALP reader supports.

Most features we add to the Parquet format are similar to the ALP 
encoding. They can be ignored by older readers.

Regards

Antoine.

Re: [DISCUSS] Finalizing the versioning proposal

[Ryan Blue <bl...@apache.org>]

I've been reviewing this thread after the long weekend in the US and I
agree with Russell's sentiment that it seems like we're not talking about
the same issues. A major contributing factor is that we seem to be having a
proxy debate: we are arguing about how or whether to write the format
version number when the real issue is whether or not a reader should fail
when it sees a file with an unsupported format version number.

Let's try to separate those two things: first the format version number and
second behavior based on it. (This is setting aside the magic bytes, which
we should assume are a separate thing for now.)

I think we can agree that readers can ignore a version number in a file.
Readers can still try their best to read the file, even if it has a
different number encoded. I think we're focused on this because the
opposite is not true: if there is no version number then you can't check
compatibility based on it. That's why I think encoding the version has been
a proxy issue.

If we agree that readers can ignore a format version, then we can discuss
whether it is a good idea for a reader to fail on an unknown version or to
attempt to read it. That seems to be the true unresolved issue.

It looks like Antoine, Will, and Micah prefer attempting to read a file.
Huge thanks to Will for his clear description of why he thinks this is a
good idea, which I think comes down to this: if we can use Thrift's
evolution rules to maintain compatibility (outside of occasional footer
redesigns) then we generally should. And since we have made it this far, do
we really need anything else?

There's also a group that thinks we should annotate files with their target
format version, but leave the decision of whether to read future versions
up to implementations. I think this includes Russell and Andrew.

And finally I think there's a group of us (Dan, Alkis, and I) who would
prefer requiring readers to fail if they encounter a newer format version
rather than trying to read it. The main argument for this is to make
correctness guarantees. I don't think it is enough to rely on Thrift
evolution rules, and I think that attempting to will inevitably lead to
correctness bugs when our changes have unforeseen consequences in readers.
I think that relying on Thrift evolution rules also limits the ways that we
can evolve the format and make it significantly harder to implement and
maintain.

I'd like to see who else agrees with this framing. I think if we agree that
the problem is whether to require failure, require a best-effort read, or
leave it to implementations, then we can have a more productive
conversation about the trade-offs.

I should also note that I think we all agree that we want to avoid
forward-compatibility breaking changes and that we want to make it possible
to introduce new encodings safely within older format versions (or other
similar things that can be done through "preview" features).

Since this is long enough, I'll see if there's agreement for reframing the
debate before moving on to why I think we should require readers to fail on
a newer format version.

Ryan

On Tue, Sep 8, 2026 at 1:24 PM Antoine Pitrou <an...@python.org> wrote:

>
> Le 08/09/2026 à 22:12, Daniel Weeks a écrit :
> > I'm a little late to the conversation here, but the discussion so far
> > leaves me thinking that we're getting a bit lost in the nuances.
> >
> > The intent behind grouping features and capturing the version in the file
> > is to signify the expected level of compatibility.
>
> I don't think that's the case that it's "the" intent.
>
> *My* intent is to make it easier at the writer level to decide which
> features to enable. Saying that I want to write a "V3"-compatible file,
> with adequate documentation on the Parquet website about reader
> compatibility, is much easier and safer than toggling individual feature
> bits in the dark.
>
> For this concern, it's not useful for the "version number" to be written
> in the file. The version number is purely a way for the user of the
> Parquet writer to communicate their intents.
>
> Regards
>
> Antoine.
>
>
>

Re: [DISCUSS] Finalizing the versioning proposal

[Antoine Pitrou <an...@python.org>]

Le 08/09/2026 à 22:12, Daniel Weeks a écrit :
> I'm a little late to the conversation here, but the discussion so far
> leaves me thinking that we're getting a bit lost in the nuances.
> 
> The intent behind grouping features and capturing the version in the file
> is to signify the expected level of compatibility.

I don't think that's the case that it's "the" intent.

*My* intent is to make it easier at the writer level to decide which 
features to enable. Saying that I want to write a "V3"-compatible file, 
with adequate documentation on the Parquet website about reader 
compatibility, is much easier and safer than toggling individual feature 
bits in the dark.

For this concern, it's not useful for the "version number" to be written 
in the file. The version number is purely a way for the user of the 
Parquet writer to communicate their intents.

Regards

Antoine.

Re: [DISCUSS] Finalizing the versioning proposal

[Daniel Weeks <dw...@apache.org>]

I'm a little late to the conversation here, but the discussion so far
leaves me thinking that we're getting a bit lost in the nuances.

The intent behind grouping features and capturing the version in the file
is to signify the expected level of compatibility.  While we can envision
numerous scenarios where a parquet file was written with a newer version
and a reader could possibly consume parts of the data, I don't see a
scenario where we can generally consider that "safe".

The intent of capturing the version and gating older readers is that we
can't predetermine how the representation will change or guarantee safety.
The writer should be responsible for producing and capturing the correct
compatibility level.  Readers should never assume they can read data for a
version they don't support (just because you can still read it, doesn't
mean it's semantically correct).

I don't think we can rely on how we've handled changes historically within
the project because they've all been designed with the intent to be forward
compatible or allowed to break via structural changes like new enum
values.  However, we now need to make significant changes to keep the
format relevant and address limitations that have accumulated over time.

We seem to keep struggling to find a path forward for incompatible changes
while simultaneously trying to make them compatible.

I think it's more important that the project moves forward than trying to
preserve a narrow amount of compatibility through a transition.  Effort is
better spent updating projects to support new features than trying to
bridge versions.

-Dan


On Mon, Sep 7, 2026 at 8:27 AM Fokko Driesprong <fo...@driesprong.frl>
wrote:

> Thanks, everyone, for participating here. My perspective below closely
> matches what Russell summarized earlier.
>
> > the reader can choose to try and read any file on a best effort basis,
> and it may error when reading a particular file if it encounters something
> required (e.g. a new encoding) that it doesn't know about
>
> I agree with that statement. For example, if you have a specific reader
> that doesn't support geo stuff at all, it could simply fail when it tries
> to read any of those types. The other way around: if you have a very
> specific geo application, you might not implement all the ones out there
> (think about the deprecated ones).
>
> Coming back to the vote, my interpretation is that I'm reluctant to
> introduce yet another versioning mechanism after 13 years of Parquet. For
> most non-breaking changes, a new enum value will just signal to the reader
> that something is new to him (encoding, type, etc.). The only thing is when
> we introduce breaking changes. As a community, we need to weigh whether the
> benefits outweigh the cost of a breaking change. Mostly thinking of the new
> footer format here, since we need to fail fast.
>
> Cheers,
> Fokko
>
>
>
> Sent with Proton Mail secure email.
>
> On Monday, September 7th, 2026 at 17:01, Micah Kornfield <
> emkornfield@gmail.com> wrote:
>
> > Andrew, I think there are examples where this would break in undesirable
> > ways.
> >
> >  As I said in another part of this thread I think there are different
> > opinions on:
> > - what gets written where
> > - how readers are supposed to consume or not consume the value(s).
> >
> > I think we beed a clear set of requirements.  Once we have those we need
> > more specifics on how writers and readers are expected to behave.  Then
> we
> > can have worked examples on how the proposals achieve the requirements. I
> > think having these would allow for a much more straightforward
> discussion.
> >
> > Julien, can you own (or delegate) updating the doc  to include these
> > details.
> >
> > Thanks,
> > Micah
> >
> > On Monday, September 7, 2026, Andrew Lamb <an...@gmail.com>
> wrote:
> >
> > > > It the feature is not breaking the footer but we bumped the version
> > > number
> > > > anyway then the reader has to fail- there is no correct way to do
> > > graceful
> > > > degradation.
> > >
> > > I disagree with this statement -- the reader can choose to try and
> read any
> > > file on a best effort basis, and it may error when reading a particular
> > > file if it encounters something required (e.g. a new encoding) that it
> > > doesn't know about
> > >
> > > In my mind, the ONLY obligation placed on readers with version numbers
> is
> > > that if they advertise "Support Version X of Parquet" then they are
> > > obligated to support all features included in Version X.
> > >
> > > Andrew
> > >
> > >
> > > On Mon, Sep 7, 2026 at 5:27 AM Will Edwards via dev <
> > > dev@parquet.apache.org>
> > > wrote:
> > >
> > > > With Monday morning clearheadedness, my position is:
> > > >
> > > > 1. Parquet is already versioned through Thrift fields and enums.  We
> have
> > > > not missed versioning anything.
> > > >
> > > > 2. We do not need a separate feature list. The Thrift structures
> already
> > > > describe the features that are actually present and where they are
> used.
> > > >
> > > > 3. Unsupported features should fail only when required by the read.
> > > Unused
> > > > columns and optional metadata should remain skippable.
> > > >
> > > > 4. FileMetaData.version should version only the footer
> representation.
> > > > Readers must fail if they cannot parse that footer version.  (And any
> > > > future non-thrift footer could have a tiny thrift object preamble
> with
> > > just
> > > > the footer version, etc.))
> > > >
> > > > 5. The file envelope describes the file framing:
> > > >       - Leading PAR1 identifies the file, including to Unix file.
> Change
> > > > this at our peril!
> > > >       - Trailing PAR1 says the preceding four bytes contain the
> footer
> > > > length and that the footer is Thrift FileMetaData.
> > > >       - Readers must fail if they don't understand the envelope.
> > > >
> > > >  6. We should not overload the footer or envelope version with a
> coarse
> > > > feature bundle that makes otherwise readable files fail.
> > > >
> > > > Regards
> > > > Will
> > > >
> > > >
> > > > On Fri, 4 Sept 2026 at 08:49, Will Edwards <we...@spotify.com>
> wrote:
> > > >
> > > > > Julien,
> > > > >
> > > > > Apologies for being late to this thread, but this touches on some
> of
> > > the
> > > > > deepest architectural principles of the Parquet project and
> warrants
> > > > > careful, deliberate reflection.
> > > > >
> > > > > At its core, we’re highlighting the fundamental tension between
> vendor
> > > > > priorities and end-user needs.
> > > > >
> > > > > For a platform vendor, upgrade cycles carry low friction, and
> > > "fail-fast"
> > > > > version gates simplify internal engine support matrices.
> > > > >
> > > > > But for end users, Parquet's primary value lies in its role as an
> > > > > immutable, long-term archival format—the boring, cross-vendor
> lingua
> > > > franca
> > > > > of the data ecosystem. Users depend on writing a file today and
> knowing
> > > > > that an unmaintained C++ service, an older BI connector, or a
> legacy
> > > > > microservice can still read it five years from now without forcing
> the
> > > > > organization onto a continuous infrastructure upgrade treadmill.
> > > > >
> > > > > Looking through that user-first lens, here is how I think users
> would
> > > > want
> > > > > us to address the four decision points:
> > > > >
> > > > > 1. Mechanism to decide what features writers use: mainstream
> writers
> > > must
> > > > > default to baseline compatibility and users must have control over
> > > opting
> > > > > in to newer features.
> > > > >
> > > > > 2. Mechanism to write preview features: same as normal features.
> > > Preview
> > > > > functionality doesn't need separate versioning machinery; it is
> simply
> > > an
> > > > > optional feature governed by the Thrift semantics and reader
> fallback
> > > > rules
> > > > > outlined below.
> > > > >
> > > > > 3. Mechanism to record feature usage: rely on how the specification
> > > > > already works fundamentally: Thrift ignores fields and tags the
> reader
> > > > does
> > > > > not recognize. Recording metadata using standard Thrift structures
> > > > ensures
> > > > > that older readers parse the file footer seamlessly, ignoring
> unknown
> > > > > fields and still succeeding whenever they have enough
> understanding of
> > > > the
> > > > > data.
> > > > >
> > > > > 4. Reader behavior on unsupported files: Readers must adopt
> > > query-scoped,
> > > > > graceful degradation under Postel's Law. We can point to the
> current
> > > > > ongoing work on logical types as the ideal model for how we
> tighten the
> > > > > spec here. By clearly defining fallback behaviors and explicit
> reader
> > > > rules
> > > > > in the specification, a reader will fail only if an unsupported
> feature
> > > > > directly blocks the reading of a column explicitly requested by the
> > > user,
> > > > > while safely completing the read if the unsupported feature lies in
> > > > > unselected columns or auxiliary structures.
> > > > >
> > > > > Absorbing "Postel pain"—the heavy engineering discipline required
> to
> > > > > maintain backward compatibility and graceful degradation—is
> precisely
> > > > what
> > > > > made Parquet the ubiquitous, trusted standard it is today. If we
> > > > surrender
> > > > > to forced upgrade cycles and hard version gates, we risk
> alienating the
> > > > > vast majority of users and opening up space for a new, truly stable
> > > > format
> > > > > to displace Parquet as the default storage layer.
> > > > >
> > > > > Crucially, embracing Postel’s Law does not preclude technical
> velocity
> > > or
> > > > > innovation; leaning on native Thrift semantics and tightening
> reader
> > > > > fallbacks gives us a safe, structured path to introduce
> > > high-performance
> > > > > features without breaking the ecosystem.
> > > > >
> > > > > On Fri, 14 Aug 2026 at 20:26, Julien Le Dem <ju...@apache.org>
> wrote:
> > > > >
> > > > >> Following up on the discussion during the Parquet sync on
> Wednesday, I
> > > > >> have
> > > > >> added a new tab to the Versioning document
> > > > >> <
> > > > >>
> > > > https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6g
> > > mIFaSEWzkxUeJw/edit?tab=t.nywy9gjvpwqf
> > > > >> >
> > > > >> to discuss 4 remaining decision points for the implementation. For
> > > each
> > > > I
> > > > >> have listed options and pros and cons.
> > > > >> 1. Mechanism to decide what Parquet features writers should use
> > > > >> 2. Mechanism to write Parquet files that use a preview feature
> > > > >> 3. Mechanism to record in the file what features it uses
> > > > >> 4. Describing reader behavior when reading a file that is not
> fully
> > > > >> supported
> > > > >>
> > > > >> The stated guiding principle is: We want to minimize prescribing
> how
> > > > >> readers and writers should work and focus on giving them the
> > > information
> > > > >> to
> > > > >> implement the behavior they deem appropriate.
> > > > >>
> > > > >> Please comment in the doc to help finalize consensus.
> > > > >> (thank you Andrew for early feedback and helping me think this
> > > through!)
> > > > >>
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Finalizing the versioning proposal

[Fokko Driesprong <fo...@driesprong.frl>]

Thanks, everyone, for participating here. My perspective below closely matches what Russell summarized earlier.

> the reader can choose to try and read any file on a best effort basis, and it may error when reading a particular file if it encounters something required (e.g. a new encoding) that it doesn't know about

I agree with that statement. For example, if you have a specific reader that doesn't support geo stuff at all, it could simply fail when it tries to read any of those types. The other way around: if you have a very specific geo application, you might not implement all the ones out there (think about the deprecated ones).

Coming back to the vote, my interpretation is that I'm reluctant to introduce yet another versioning mechanism after 13 years of Parquet. For most non-breaking changes, a new enum value will just signal to the reader that something is new to him (encoding, type, etc.). The only thing is when we introduce breaking changes. As a community, we need to weigh whether the benefits outweigh the cost of a breaking change. Mostly thinking of the new footer format here, since we need to fail fast.

Cheers,
Fokko



Sent with Proton Mail secure email.

On Monday, September 7th, 2026 at 17:01, Micah Kornfield <em...@gmail.com> wrote:

> Andrew, I think there are examples where this would break in undesirable
> ways.
> 
>  As I said in another part of this thread I think there are different
> opinions on:
> - what gets written where
> - how readers are supposed to consume or not consume the value(s).
> 
> I think we beed a clear set of requirements.  Once we have those we need
> more specifics on how writers and readers are expected to behave.  Then we
> can have worked examples on how the proposals achieve the requirements. I
> think having these would allow for a much more straightforward discussion.
> 
> Julien, can you own (or delegate) updating the doc  to include these
> details.
> 
> Thanks,
> Micah
> 
> On Monday, September 7, 2026, Andrew Lamb <an...@gmail.com> wrote:
> 
> > > It the feature is not breaking the footer but we bumped the version
> > number
> > > anyway then the reader has to fail- there is no correct way to do
> > graceful
> > > degradation.
> >
> > I disagree with this statement -- the reader can choose to try and read any
> > file on a best effort basis, and it may error when reading a particular
> > file if it encounters something required (e.g. a new encoding) that it
> > doesn't know about
> >
> > In my mind, the ONLY obligation placed on readers with version numbers is
> > that if they advertise "Support Version X of Parquet" then they are
> > obligated to support all features included in Version X.
> >
> > Andrew
> >
> >
> > On Mon, Sep 7, 2026 at 5:27 AM Will Edwards via dev <
> > dev@parquet.apache.org>
> > wrote:
> >
> > > With Monday morning clearheadedness, my position is:
> > >
> > > 1. Parquet is already versioned through Thrift fields and enums.  We have
> > > not missed versioning anything.
> > >
> > > 2. We do not need a separate feature list. The Thrift structures already
> > > describe the features that are actually present and where they are used.
> > >
> > > 3. Unsupported features should fail only when required by the read.
> > Unused
> > > columns and optional metadata should remain skippable.
> > >
> > > 4. FileMetaData.version should version only the footer representation.
> > > Readers must fail if they cannot parse that footer version.  (And any
> > > future non-thrift footer could have a tiny thrift object preamble with
> > just
> > > the footer version, etc.))
> > >
> > > 5. The file envelope describes the file framing:
> > >       - Leading PAR1 identifies the file, including to Unix file.  Change
> > > this at our peril!
> > >       - Trailing PAR1 says the preceding four bytes contain the footer
> > > length and that the footer is Thrift FileMetaData.
> > >       - Readers must fail if they don't understand the envelope.
> > >
> > >  6. We should not overload the footer or envelope version with a coarse
> > > feature bundle that makes otherwise readable files fail.
> > >
> > > Regards
> > > Will
> > >
> > >
> > > On Fri, 4 Sept 2026 at 08:49, Will Edwards <we...@spotify.com> wrote:
> > >
> > > > Julien,
> > > >
> > > > Apologies for being late to this thread, but this touches on some of
> > the
> > > > deepest architectural principles of the Parquet project and warrants
> > > > careful, deliberate reflection.
> > > >
> > > > At its core, we’re highlighting the fundamental tension between vendor
> > > > priorities and end-user needs.
> > > >
> > > > For a platform vendor, upgrade cycles carry low friction, and
> > "fail-fast"
> > > > version gates simplify internal engine support matrices.
> > > >
> > > > But for end users, Parquet's primary value lies in its role as an
> > > > immutable, long-term archival format—the boring, cross-vendor lingua
> > > franca
> > > > of the data ecosystem. Users depend on writing a file today and knowing
> > > > that an unmaintained C++ service, an older BI connector, or a legacy
> > > > microservice can still read it five years from now without forcing the
> > > > organization onto a continuous infrastructure upgrade treadmill.
> > > >
> > > > Looking through that user-first lens, here is how I think users would
> > > want
> > > > us to address the four decision points:
> > > >
> > > > 1. Mechanism to decide what features writers use: mainstream writers
> > must
> > > > default to baseline compatibility and users must have control over
> > opting
> > > > in to newer features.
> > > >
> > > > 2. Mechanism to write preview features: same as normal features.
> > Preview
> > > > functionality doesn't need separate versioning machinery; it is simply
> > an
> > > > optional feature governed by the Thrift semantics and reader fallback
> > > rules
> > > > outlined below.
> > > >
> > > > 3. Mechanism to record feature usage: rely on how the specification
> > > > already works fundamentally: Thrift ignores fields and tags the reader
> > > does
> > > > not recognize. Recording metadata using standard Thrift structures
> > > ensures
> > > > that older readers parse the file footer seamlessly, ignoring unknown
> > > > fields and still succeeding whenever they have enough understanding of
> > > the
> > > > data.
> > > >
> > > > 4. Reader behavior on unsupported files: Readers must adopt
> > query-scoped,
> > > > graceful degradation under Postel's Law. We can point to the current
> > > > ongoing work on logical types as the ideal model for how we tighten the
> > > > spec here. By clearly defining fallback behaviors and explicit reader
> > > rules
> > > > in the specification, a reader will fail only if an unsupported feature
> > > > directly blocks the reading of a column explicitly requested by the
> > user,
> > > > while safely completing the read if the unsupported feature lies in
> > > > unselected columns or auxiliary structures.
> > > >
> > > > Absorbing "Postel pain"—the heavy engineering discipline required to
> > > > maintain backward compatibility and graceful degradation—is precisely
> > > what
> > > > made Parquet the ubiquitous, trusted standard it is today. If we
> > > surrender
> > > > to forced upgrade cycles and hard version gates, we risk alienating the
> > > > vast majority of users and opening up space for a new, truly stable
> > > format
> > > > to displace Parquet as the default storage layer.
> > > >
> > > > Crucially, embracing Postel’s Law does not preclude technical velocity
> > or
> > > > innovation; leaning on native Thrift semantics and tightening reader
> > > > fallbacks gives us a safe, structured path to introduce
> > high-performance
> > > > features without breaking the ecosystem.
> > > >
> > > > On Fri, 14 Aug 2026 at 20:26, Julien Le Dem <ju...@apache.org> wrote:
> > > >
> > > >> Following up on the discussion during the Parquet sync on Wednesday, I
> > > >> have
> > > >> added a new tab to the Versioning document
> > > >> <
> > > >>
> > > https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6g
> > mIFaSEWzkxUeJw/edit?tab=t.nywy9gjvpwqf
> > > >> >
> > > >> to discuss 4 remaining decision points for the implementation. For
> > each
> > > I
> > > >> have listed options and pros and cons.
> > > >> 1. Mechanism to decide what Parquet features writers should use
> > > >> 2. Mechanism to write Parquet files that use a preview feature
> > > >> 3. Mechanism to record in the file what features it uses
> > > >> 4. Describing reader behavior when reading a file that is not fully
> > > >> supported
> > > >>
> > > >> The stated guiding principle is: We want to minimize prescribing how
> > > >> readers and writers should work and focus on giving them the
> > information
> > > >> to
> > > >> implement the behavior they deem appropriate.
> > > >>
> > > >> Please comment in the doc to help finalize consensus.
> > > >> (thank you Andrew for early feedback and helping me think this
> > through!)
> > > >>
> > > >
> > >
> >
>

Re: [DISCUSS] Finalizing the versioning proposal

[Micah Kornfield <em...@gmail.com>]

Andrew, I think there are examples where this would break in undesirable
ways.

 As I said in another part of this thread I think there are different
opinions on:
- what gets written where
- how readers are supposed to consume or not consume the value(s).

I think we beed a clear set of requirements.  Once we have those we need
more specifics on how writers and readers are expected to behave.  Then we
can have worked examples on how the proposals achieve the requirements. I
think having these would allow for a much more straightforward discussion.

Julien, can you own (or delegate) updating the doc  to include these
details.

Thanks,
Micah

On Monday, September 7, 2026, Andrew Lamb <an...@gmail.com> wrote:

> > It the feature is not breaking the footer but we bumped the version
> number
> > anyway then the reader has to fail- there is no correct way to do
> graceful
> > degradation.
>
> I disagree with this statement -- the reader can choose to try and read any
> file on a best effort basis, and it may error when reading a particular
> file if it encounters something required (e.g. a new encoding) that it
> doesn't know about
>
> In my mind, the ONLY obligation placed on readers with version numbers is
> that if they advertise "Support Version X of Parquet" then they are
> obligated to support all features included in Version X.
>
> Andrew
>
>
> On Mon, Sep 7, 2026 at 5:27 AM Will Edwards via dev <
> dev@parquet.apache.org>
> wrote:
>
> > With Monday morning clearheadedness, my position is:
> >
> > 1. Parquet is already versioned through Thrift fields and enums.  We have
> > not missed versioning anything.
> >
> > 2. We do not need a separate feature list. The Thrift structures already
> > describe the features that are actually present and where they are used.
> >
> > 3. Unsupported features should fail only when required by the read.
> Unused
> > columns and optional metadata should remain skippable.
> >
> > 4. FileMetaData.version should version only the footer representation.
> > Readers must fail if they cannot parse that footer version.  (And any
> > future non-thrift footer could have a tiny thrift object preamble with
> just
> > the footer version, etc.))
> >
> > 5. The file envelope describes the file framing:
> >       - Leading PAR1 identifies the file, including to Unix file.  Change
> > this at our peril!
> >       - Trailing PAR1 says the preceding four bytes contain the footer
> > length and that the footer is Thrift FileMetaData.
> >       - Readers must fail if they don't understand the envelope.
> >
> >  6. We should not overload the footer or envelope version with a coarse
> > feature bundle that makes otherwise readable files fail.
> >
> > Regards
> > Will
> >
> >
> > On Fri, 4 Sept 2026 at 08:49, Will Edwards <we...@spotify.com> wrote:
> >
> > > Julien,
> > >
> > > Apologies for being late to this thread, but this touches on some of
> the
> > > deepest architectural principles of the Parquet project and warrants
> > > careful, deliberate reflection.
> > >
> > > At its core, we’re highlighting the fundamental tension between vendor
> > > priorities and end-user needs.
> > >
> > > For a platform vendor, upgrade cycles carry low friction, and
> "fail-fast"
> > > version gates simplify internal engine support matrices.
> > >
> > > But for end users, Parquet's primary value lies in its role as an
> > > immutable, long-term archival format—the boring, cross-vendor lingua
> > franca
> > > of the data ecosystem. Users depend on writing a file today and knowing
> > > that an unmaintained C++ service, an older BI connector, or a legacy
> > > microservice can still read it five years from now without forcing the
> > > organization onto a continuous infrastructure upgrade treadmill.
> > >
> > > Looking through that user-first lens, here is how I think users would
> > want
> > > us to address the four decision points:
> > >
> > > 1. Mechanism to decide what features writers use: mainstream writers
> must
> > > default to baseline compatibility and users must have control over
> opting
> > > in to newer features.
> > >
> > > 2. Mechanism to write preview features: same as normal features.
> Preview
> > > functionality doesn't need separate versioning machinery; it is simply
> an
> > > optional feature governed by the Thrift semantics and reader fallback
> > rules
> > > outlined below.
> > >
> > > 3. Mechanism to record feature usage: rely on how the specification
> > > already works fundamentally: Thrift ignores fields and tags the reader
> > does
> > > not recognize. Recording metadata using standard Thrift structures
> > ensures
> > > that older readers parse the file footer seamlessly, ignoring unknown
> > > fields and still succeeding whenever they have enough understanding of
> > the
> > > data.
> > >
> > > 4. Reader behavior on unsupported files: Readers must adopt
> query-scoped,
> > > graceful degradation under Postel's Law. We can point to the current
> > > ongoing work on logical types as the ideal model for how we tighten the
> > > spec here. By clearly defining fallback behaviors and explicit reader
> > rules
> > > in the specification, a reader will fail only if an unsupported feature
> > > directly blocks the reading of a column explicitly requested by the
> user,
> > > while safely completing the read if the unsupported feature lies in
> > > unselected columns or auxiliary structures.
> > >
> > > Absorbing "Postel pain"—the heavy engineering discipline required to
> > > maintain backward compatibility and graceful degradation—is precisely
> > what
> > > made Parquet the ubiquitous, trusted standard it is today. If we
> > surrender
> > > to forced upgrade cycles and hard version gates, we risk alienating the
> > > vast majority of users and opening up space for a new, truly stable
> > format
> > > to displace Parquet as the default storage layer.
> > >
> > > Crucially, embracing Postel’s Law does not preclude technical velocity
> or
> > > innovation; leaning on native Thrift semantics and tightening reader
> > > fallbacks gives us a safe, structured path to introduce
> high-performance
> > > features without breaking the ecosystem.
> > >
> > > On Fri, 14 Aug 2026 at 20:26, Julien Le Dem <ju...@apache.org> wrote:
> > >
> > >> Following up on the discussion during the Parquet sync on Wednesday, I
> > >> have
> > >> added a new tab to the Versioning document
> > >> <
> > >>
> > https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6g
> mIFaSEWzkxUeJw/edit?tab=t.nywy9gjvpwqf
> > >> >
> > >> to discuss 4 remaining decision points for the implementation. For
> each
> > I
> > >> have listed options and pros and cons.
> > >> 1. Mechanism to decide what Parquet features writers should use
> > >> 2. Mechanism to write Parquet files that use a preview feature
> > >> 3. Mechanism to record in the file what features it uses
> > >> 4. Describing reader behavior when reading a file that is not fully
> > >> supported
> > >>
> > >> The stated guiding principle is: We want to minimize prescribing how
> > >> readers and writers should work and focus on giving them the
> information
> > >> to
> > >> implement the behavior they deem appropriate.
> > >>
> > >> Please comment in the doc to help finalize consensus.
> > >> (thank you Andrew for early feedback and helping me think this
> through!)
> > >>
> > >
> >
>

Re: [DISCUSS] Finalizing the versioning proposal

[Andrew Lamb <an...@gmail.com>]

> It the feature is not breaking the footer but we bumped the version number
> anyway then the reader has to fail- there is no correct way to do graceful
> degradation.

I disagree with this statement -- the reader can choose to try and read any
file on a best effort basis, and it may error when reading a particular
file if it encounters something required (e.g. a new encoding) that it
doesn't know about

In my mind, the ONLY obligation placed on readers with version numbers is
that if they advertise "Support Version X of Parquet" then they are
obligated to support all features included in Version X.

Andrew


On Mon, Sep 7, 2026 at 5:27 AM Will Edwards via dev <de...@parquet.apache.org>
wrote:

> With Monday morning clearheadedness, my position is:
>
> 1. Parquet is already versioned through Thrift fields and enums.  We have
> not missed versioning anything.
>
> 2. We do not need a separate feature list. The Thrift structures already
> describe the features that are actually present and where they are used.
>
> 3. Unsupported features should fail only when required by the read. Unused
> columns and optional metadata should remain skippable.
>
> 4. FileMetaData.version should version only the footer representation.
> Readers must fail if they cannot parse that footer version.  (And any
> future non-thrift footer could have a tiny thrift object preamble with just
> the footer version, etc.))
>
> 5. The file envelope describes the file framing:
>       - Leading PAR1 identifies the file, including to Unix file.  Change
> this at our peril!
>       - Trailing PAR1 says the preceding four bytes contain the footer
> length and that the footer is Thrift FileMetaData.
>       - Readers must fail if they don't understand the envelope.
>
>  6. We should not overload the footer or envelope version with a coarse
> feature bundle that makes otherwise readable files fail.
>
> Regards
> Will
>
>
> On Fri, 4 Sept 2026 at 08:49, Will Edwards <we...@spotify.com> wrote:
>
> > Julien,
> >
> > Apologies for being late to this thread, but this touches on some of the
> > deepest architectural principles of the Parquet project and warrants
> > careful, deliberate reflection.
> >
> > At its core, we’re highlighting the fundamental tension between vendor
> > priorities and end-user needs.
> >
> > For a platform vendor, upgrade cycles carry low friction, and "fail-fast"
> > version gates simplify internal engine support matrices.
> >
> > But for end users, Parquet's primary value lies in its role as an
> > immutable, long-term archival format—the boring, cross-vendor lingua
> franca
> > of the data ecosystem. Users depend on writing a file today and knowing
> > that an unmaintained C++ service, an older BI connector, or a legacy
> > microservice can still read it five years from now without forcing the
> > organization onto a continuous infrastructure upgrade treadmill.
> >
> > Looking through that user-first lens, here is how I think users would
> want
> > us to address the four decision points:
> >
> > 1. Mechanism to decide what features writers use: mainstream writers must
> > default to baseline compatibility and users must have control over opting
> > in to newer features.
> >
> > 2. Mechanism to write preview features: same as normal features.  Preview
> > functionality doesn't need separate versioning machinery; it is simply an
> > optional feature governed by the Thrift semantics and reader fallback
> rules
> > outlined below.
> >
> > 3. Mechanism to record feature usage: rely on how the specification
> > already works fundamentally: Thrift ignores fields and tags the reader
> does
> > not recognize. Recording metadata using standard Thrift structures
> ensures
> > that older readers parse the file footer seamlessly, ignoring unknown
> > fields and still succeeding whenever they have enough understanding of
> the
> > data.
> >
> > 4. Reader behavior on unsupported files: Readers must adopt query-scoped,
> > graceful degradation under Postel's Law. We can point to the current
> > ongoing work on logical types as the ideal model for how we tighten the
> > spec here. By clearly defining fallback behaviors and explicit reader
> rules
> > in the specification, a reader will fail only if an unsupported feature
> > directly blocks the reading of a column explicitly requested by the user,
> > while safely completing the read if the unsupported feature lies in
> > unselected columns or auxiliary structures.
> >
> > Absorbing "Postel pain"—the heavy engineering discipline required to
> > maintain backward compatibility and graceful degradation—is precisely
> what
> > made Parquet the ubiquitous, trusted standard it is today. If we
> surrender
> > to forced upgrade cycles and hard version gates, we risk alienating the
> > vast majority of users and opening up space for a new, truly stable
> format
> > to displace Parquet as the default storage layer.
> >
> > Crucially, embracing Postel’s Law does not preclude technical velocity or
> > innovation; leaning on native Thrift semantics and tightening reader
> > fallbacks gives us a safe, structured path to introduce high-performance
> > features without breaking the ecosystem.
> >
> > On Fri, 14 Aug 2026 at 20:26, Julien Le Dem <ju...@apache.org> wrote:
> >
> >> Following up on the discussion during the Parquet sync on Wednesday, I
> >> have
> >> added a new tab to the Versioning document
> >> <
> >>
> https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab=t.nywy9gjvpwqf
> >> >
> >> to discuss 4 remaining decision points for the implementation. For each
> I
> >> have listed options and pros and cons.
> >> 1. Mechanism to decide what Parquet features writers should use
> >> 2. Mechanism to write Parquet files that use a preview feature
> >> 3. Mechanism to record in the file what features it uses
> >> 4. Describing reader behavior when reading a file that is not fully
> >> supported
> >>
> >> The stated guiding principle is: We want to minimize prescribing how
> >> readers and writers should work and focus on giving them the information
> >> to
> >> implement the behavior they deem appropriate.
> >>
> >> Please comment in the doc to help finalize consensus.
> >> (thank you Andrew for early feedback and helping me think this through!)
> >>
> >
>

Re: [DISCUSS] Finalizing the versioning proposal

[Will Edwards via dev <de...@parquet.apache.org>]

With Monday morning clearheadedness, my position is:

1. Parquet is already versioned through Thrift fields and enums.  We have
not missed versioning anything.

2. We do not need a separate feature list. The Thrift structures already
describe the features that are actually present and where they are used.

3. Unsupported features should fail only when required by the read. Unused
columns and optional metadata should remain skippable.

4. FileMetaData.version should version only the footer representation.
Readers must fail if they cannot parse that footer version.  (And any
future non-thrift footer could have a tiny thrift object preamble with just
the footer version, etc.))

5. The file envelope describes the file framing:
      - Leading PAR1 identifies the file, including to Unix file.  Change
this at our peril!
      - Trailing PAR1 says the preceding four bytes contain the footer
length and that the footer is Thrift FileMetaData.
      - Readers must fail if they don't understand the envelope.

 6. We should not overload the footer or envelope version with a coarse
feature bundle that makes otherwise readable files fail.

Regards
Will


On Fri, 4 Sept 2026 at 08:49, Will Edwards <we...@spotify.com> wrote:

> Julien,
>
> Apologies for being late to this thread, but this touches on some of the
> deepest architectural principles of the Parquet project and warrants
> careful, deliberate reflection.
>
> At its core, we’re highlighting the fundamental tension between vendor
> priorities and end-user needs.
>
> For a platform vendor, upgrade cycles carry low friction, and "fail-fast"
> version gates simplify internal engine support matrices.
>
> But for end users, Parquet's primary value lies in its role as an
> immutable, long-term archival format—the boring, cross-vendor lingua franca
> of the data ecosystem. Users depend on writing a file today and knowing
> that an unmaintained C++ service, an older BI connector, or a legacy
> microservice can still read it five years from now without forcing the
> organization onto a continuous infrastructure upgrade treadmill.
>
> Looking through that user-first lens, here is how I think users would want
> us to address the four decision points:
>
> 1. Mechanism to decide what features writers use: mainstream writers must
> default to baseline compatibility and users must have control over opting
> in to newer features.
>
> 2. Mechanism to write preview features: same as normal features.  Preview
> functionality doesn't need separate versioning machinery; it is simply an
> optional feature governed by the Thrift semantics and reader fallback rules
> outlined below.
>
> 3. Mechanism to record feature usage: rely on how the specification
> already works fundamentally: Thrift ignores fields and tags the reader does
> not recognize. Recording metadata using standard Thrift structures ensures
> that older readers parse the file footer seamlessly, ignoring unknown
> fields and still succeeding whenever they have enough understanding of the
> data.
>
> 4. Reader behavior on unsupported files: Readers must adopt query-scoped,
> graceful degradation under Postel's Law. We can point to the current
> ongoing work on logical types as the ideal model for how we tighten the
> spec here. By clearly defining fallback behaviors and explicit reader rules
> in the specification, a reader will fail only if an unsupported feature
> directly blocks the reading of a column explicitly requested by the user,
> while safely completing the read if the unsupported feature lies in
> unselected columns or auxiliary structures.
>
> Absorbing "Postel pain"—the heavy engineering discipline required to
> maintain backward compatibility and graceful degradation—is precisely what
> made Parquet the ubiquitous, trusted standard it is today. If we surrender
> to forced upgrade cycles and hard version gates, we risk alienating the
> vast majority of users and opening up space for a new, truly stable format
> to displace Parquet as the default storage layer.
>
> Crucially, embracing Postel’s Law does not preclude technical velocity or
> innovation; leaning on native Thrift semantics and tightening reader
> fallbacks gives us a safe, structured path to introduce high-performance
> features without breaking the ecosystem.
>
> On Fri, 14 Aug 2026 at 20:26, Julien Le Dem <ju...@apache.org> wrote:
>
>> Following up on the discussion during the Parquet sync on Wednesday, I
>> have
>> added a new tab to the Versioning document
>> <
>> https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab=t.nywy9gjvpwqf
>> >
>> to discuss 4 remaining decision points for the implementation. For each I
>> have listed options and pros and cons.
>> 1. Mechanism to decide what Parquet features writers should use
>> 2. Mechanism to write Parquet files that use a preview feature
>> 3. Mechanism to record in the file what features it uses
>> 4. Describing reader behavior when reading a file that is not fully
>> supported
>>
>> The stated guiding principle is: We want to minimize prescribing how
>> readers and writers should work and focus on giving them the information
>> to
>> implement the behavior they deem appropriate.
>>
>> Please comment in the doc to help finalize consensus.
>> (thank you Andrew for early feedback and helping me think this through!)
>>
>

Re: [DISCUSS] Finalizing the versioning proposal

[Will Edwards via dev <de...@parquet.apache.org>]

Julien,

Apologies for being late to this thread, but this touches on some of the
deepest architectural principles of the Parquet project and warrants
careful, deliberate reflection.

At its core, we’re highlighting the fundamental tension between vendor
priorities and end-user needs.

For a platform vendor, upgrade cycles carry low friction, and "fail-fast"
version gates simplify internal engine support matrices.

But for end users, Parquet's primary value lies in its role as an
immutable, long-term archival format—the boring, cross-vendor lingua franca
of the data ecosystem. Users depend on writing a file today and knowing
that an unmaintained C++ service, an older BI connector, or a legacy
microservice can still read it five years from now without forcing the
organization onto a continuous infrastructure upgrade treadmill.

Looking through that user-first lens, here is how I think users would want
us to address the four decision points:

1. Mechanism to decide what features writers use: mainstream writers must
default to baseline compatibility and users must have control over opting
in to newer features.

2. Mechanism to write preview features: same as normal features.  Preview
functionality doesn't need separate versioning machinery; it is simply an
optional feature governed by the Thrift semantics and reader fallback rules
outlined below.

3. Mechanism to record feature usage: rely on how the specification already
works fundamentally: Thrift ignores fields and tags the reader does not
recognize. Recording metadata using standard Thrift structures ensures that
older readers parse the file footer seamlessly, ignoring unknown fields and
still succeeding whenever they have enough understanding of the data.

4. Reader behavior on unsupported files: Readers must adopt query-scoped,
graceful degradation under Postel's Law. We can point to the current
ongoing work on logical types as the ideal model for how we tighten the
spec here. By clearly defining fallback behaviors and explicit reader rules
in the specification, a reader will fail only if an unsupported feature
directly blocks the reading of a column explicitly requested by the user,
while safely completing the read if the unsupported feature lies in
unselected columns or auxiliary structures.

Absorbing "Postel pain"—the heavy engineering discipline required to
maintain backward compatibility and graceful degradation—is precisely what
made Parquet the ubiquitous, trusted standard it is today. If we surrender
to forced upgrade cycles and hard version gates, we risk alienating the
vast majority of users and opening up space for a new, truly stable format
to displace Parquet as the default storage layer.

Crucially, embracing Postel’s Law does not preclude technical velocity or
innovation; leaning on native Thrift semantics and tightening reader
fallbacks gives us a safe, structured path to introduce high-performance
features without breaking the ecosystem.

On Fri, 14 Aug 2026 at 20:26, Julien Le Dem <ju...@apache.org> wrote:

> Following up on the discussion during the Parquet sync on Wednesday, I have
> added a new tab to the Versioning document
> <
> https://docs.google.com/document/d/19bOTy1WcLs9NP88DrysG4wG50pM6gmIFaSEWzkxUeJw/edit?tab=t.nywy9gjvpwqf
> >
> to discuss 4 remaining decision points for the implementation. For each I
> have listed options and pros and cons.
> 1. Mechanism to decide what Parquet features writers should use
> 2. Mechanism to write Parquet files that use a preview feature
> 3. Mechanism to record in the file what features it uses
> 4. Describing reader behavior when reading a file that is not fully
> supported
>
> The stated guiding principle is: We want to minimize prescribing how
> readers and writers should work and focus on giving them the information to
> implement the behavior they deem appropriate.
>
> Please comment in the doc to help finalize consensus.
> (thank you Andrew for early feedback and helping me think this through!)
>