[DISCUSS] How should readers behave with unsupported format versions?

[Ryan Blue <bl...@apache.org>]

Hi everyone,

This is a new thread to discuss an issue raised in the "Finalizing the
versioning proposal" thread. I suggested starting a new thread to focus on
this question in the community sync and there was agreement so that we can
get more attention to the issue.

The question that we identified is: How should a reader behave when it
encounters a file written with an unsupported format version?

There are 3 main options:
1. A reader should fail because it does not support the version
2. A reader should attempt to read the file
3. This choice is left up to implementations

I'll cover each option in more detail below, but first I want to clarify
that we are not talking about "preview" features like encodings or
forward-compatible changes like new logical types. Preview features will
break readers that do not support them and only affect specific columns
using the feature. For preview features, the expectation is that readers
will attempt to read the file and will fail if they need to project a
column that cannot be read.

The choice of how to handle an unsupported format version primarily affects
changes that add, remove, or modify the semantics of metadata fields. For
example:
- Changing `path_in_schema` from required to optional
- Adding `offset` and `size` fields to a page header to relocate page data
- Fixing stats written with the wrong sort order

If we choose option 1, then a v3 reader that encounters a v4 file would be
required to fail with an error that the file is too new to be read. With
the current versioning scheme, the forward-compatibility within a version
means that a v3 reader can read any v3 file and will produce the correct
data. But this guarantee does not extend to any higher version. Without a
guarantee that results are correct, the v3 reader must reject a v4 file.

If we choose option 2, then the v3 reader will attempt to read the v4 file.
In this case, Parquet will need to provide an additional guarantee that
reading future formats will either fail or produce correct results. If I
understand correctly, this option would use Thrift's semantics to cause
failures. For example, path_in_schema would be required in Thrift and
readers are expected to fail in decoding.

Option 3 would mean that readers may choose to attempt to read, but do not
have additional guarantees.

Hopefully this makes the choices clear and we can have a discussion on this
thread. Let's try to keep this thread on the topic of these options, either
to clarify them or to debate the effects. We can discuss process or other
topics on new threads.

Ryan

Re: [DISCUSS] How should readers behave with unsupported format versions?

[Ryan Blue <bl...@apache.org>]

> If the spec requires that readers fail fast on unknown versions ("Option
1"),
> * Pro: New changes to the spec don't have to consider existing readers,
and
> are thus in theory are easier/faster to make (e.g. Ryan's example of
> relocatable page headers)
> * Con: requires readers to fail on all files with newer versions, even
> those files that the reader could have read correctly

I think this con overstates the impact of failing on an unsupported
version. I don't expect readers to be able to read new format versions.

A new version means that we've included something that is forward
incompatible. That could be either a change expected to break older readers
or a correctness change not expected to break older readers -- and instead
cause them to read incorrect data. In either of those cases, the right
thing to do is to fail.

For a change that affects correctness but doesn't break older readers, we
clearly want to have readers fail. Part of the debate is whether we want
this category to exist, but if it does exist either because we choose to
change the meaning of a field or struct (pages with offset/size) or if it
exists because an assumption fails (if path_in_schema doesn't cause a
failure) then we also want a guaranteed exception.

For a change expected to break older readers, we're better off with a
guaranteed failure. Removing `path_in_schema` falls in this category. We
_want_ it to fail in metadata deserialization. It's actually safer to fail
on the version so we have higher confidence.

The case where we would not want a guaranteed failure is when we want a
failure only when a specific column is projected. But I put those changes
in the "preview" category (even if we rename it eventually). When we have a
change with a high confidence of failure and that failure happens only when
the affected column is projected, I think it would be a preview feature.
Then we can use the feature with metadata compatible with older readers and
turn it on by default when bundled with other incompatible changes.


On Thu, Sep 10, 2026 at 2:12 AM Andrew Lamb <an...@gmail.com> wrote:

> Here is my attempt to summarize the tradeoffs (Ryan's explanation during
> the call was very helpful for me).
>
> The core tradeoff is in requirements for future changes to the Parquet spec
> itself vs how many files particular readers can read.
>
> If the spec requires that readers fail fast on unknown versions ("Option
> 1"),
> * Pro: New changes to the spec don't have to consider existing readers, and
> are thus in theory are easier/faster to make (e.g. Ryan's example of
> relocatable page headers)
> * Con: requires readers to fail on all files with newer versions, even
> those files that the reader could have read correctly
>
> If the spec allows readers to attempt to read unknown versions ("Option 2")
> * Pro: Readers will be able to read more files, though it will be harder to
> reason up front if a reader can read any file (may have to test it)
> * Con: There is a requirement on any future changes to the spec to ensure
> old readers don't interpret new features ("additional guarantee that
> reading future formats will either fail or produce correct results"). This
> requirement is hard to define precisely given the wide and unknown variety
> of readers
>
> From what I can tell, the current state of parquet is implicitly Option 2
> (as readers can and do try to read any file and skip or error when
> encountering unsupported features), and don't check version numbers.
>
> My personal opinion is that allowing readers to read unknown versions
> (option 2 / option 3) is the most practical: Changing the current implicit
> behavior would be quite confusing, and making the spec harder to change for
> wider read interoperability is the right tradeoff in my mind.
>
> Andrew
>
>
> p.s.
>
> > I think path_in_schema is a good example of why option 2 will inevitably
> produce
> correctness bugs
>
> It seems to me that making path_in_schema optional will simply cause old
> readers to fail if they need it, so is not a good example of why option 2
> would necessarily correctness issues compared to option 1. The example
> about java hash sets could be avoided with adequate testing, for example,
> and I don't see how gating that code behind reading a new version number
> would make that bug any more/less likely.
>
> > My second argument for why we should not attempt to read all future
> versions
> of Parquet is that it ends up limiting how we can evolve the format.
>
> This makes a lot of sense to me -- evolving the format would have
> additional requirements (but the same requirements already implicitly
> exist); Adding relocatable pages, for example could be achieved by adding a
> new DataPageHeaderV3 rather than modifying the existing structure. That is
> more complicated to be sure, but not impossible.
>
>
>
>
> On Thu, Sep 10, 2026 at 4:39 AM Antoine Pitrou <an...@python.org> wrote:
>
> > Le 10/09/2026 à 00:38, Ryan Blue a écrit :
> > >
> > > There are 3 main options:
> > > 1. A reader should fail because it does not support the version
> > > 2. A reader should attempt to read the file
> > > 3. This choice is left up to implementations
> > >
> > > I'll cover each option in more detail below, but first I want to
> clarify
> > > that we are not talking about "preview" features like encodings or
> > > forward-compatible changes like new logical types. Preview features
> will
> > > break readers that do not support them and only affect specific columns
> > > using the feature. For preview features, the expectation is that
> readers
> > > will attempt to read the file and will fail if they need to project a
> > > column that cannot be read.
> >
> > "Preview features" is a very weird terminology. It sounds like
> > "unfinished" or "experimental".
> >
> > > The choice of how to handle an unsupported format version primarily
> > affects
> > > changes that add, remove, or modify the semantics of metadata fields.
> For
> > > example:
> > > - Changing `path_in_schema` from required to optional
> >
> > This depends whether the Thrift parser checks that required fields are
> > actually present in the serialized payload? Do we know what their
> > current behavior is?
> >
> > i.e., does a Thrift parser generated with a required `path_in_schema`
> > specification accept a serialized payload without that field?
> >
> > It also depends what the Parquet reader actually *does* with the
> > `path_in_schema`? AFAICT, the Parquet C++ reader isn't doing anything
> > specific with it.
> >
> > > Option 3 would mean that readers may choose to attempt to read, but do
> > not
> > > have additional guarantees.
> >
> > Option 3 can also mean "the reader is exposing an option to let the user
> > choose the behavior (reject up front or attempt to read anyway)".
> >
> > (this would be more costly to implement, so I'm not sure any
> > implementation would actually do that; but it's at least conceptually
> > possible)
> >
> > Regards
> >
> > Antoine.
> >
> >
> >
>

Re: [DISCUSS] How should readers behave with unsupported format versions?

[ku...@gmail.com]

Hi Everyone,

I’ve been reading along and listening during the community syncs. I’m having difficulty understanding the scenarios options 0,2,&3 are optimizing for.

For option 1 when I envision myself as a user just reading data or building on top of parquet as my storage layer or working on a proposal to extend Parquet. Failing if the Reader doesn’t fully support X version of Parquet. Provides a very reasonable and actionable error of either searching out a Parquet Reader that can read the latest version or it provides a very well encapsulated set of changes I need to add to my current Reader.

My take away from reading this email thread and listening at the community syncs is that the other options are trying to prevent changes in existing Readers. Where that doesn’t seem to me to be strong or the correct trade off is that we are adding new encodings and other forward incompatible changes then they’ll need to make changes anyway.

Another point I’ve read in this thread is the concern that it’s impossible to enforce this and that implementations can just skip reading the version number. I would push back on that concern being a blocker. If an implementation chooses to not be specification compliant then that implementation is choosing to live with the consequences of that.

I’m still relatively new to the Parquet and Arrow spaces. Andrew and Will did I misunderstand and/or mischaracterize your motivations for voicing favor to Option 2 and 0 respectively? 

Can you please help me understand who your option is optimizing for. Is it active Parquet Reader implementations? Is it other systems? Is it optimizing for specification development? Please let me know.

Best Regards,
Kurtis Wright

> On Sep 14, 2026, at 13:49, Ryan Blue <bl...@apache.org> wrote:
> 
> 
>> 
>> I support "option 0," which I'll place at the top of the list as:
>> 'Changes in Parquet files are versioned in the current way.'
> 
> I think that the "current way" is that we don't make breaking changes. I
> voted against changing `path_in_schema` to optional because I think it
> breaks our guarantees. I think it's important for us to be able to agree on
> how to make breaking changes.
> 
> 
> 
>> On Sun, Sep 13, 2026 at 11:50 PM Will Edwards via dev <
>> dev@parquet.apache.org> wrote:
>> 
>> Hi Ryan, thx for clarifying :thumb-up:
>> 
>> So, I support "option 0," which I'll place at the top of the list as:
>> 'Changes in Parquet files are versioned in the current way.'
>> 
>> Using path_in_schema as the example big braking change I could imagine
>> getting traction:
>> 
>> When we make path-in-schema optional we might bump the version number of
>> the parquet-format package as a hint for implementers to read the release
>> notes, but we won't put a version gate on the files themselves.  We expect
>> it to be opted-in only by those most affected by footer bloat and only on
>> those files most affected by that bloat.  With the passage of time more
>> users might be reaching for it and the reader base will have matured and we
>> might contemplate some writers making it the default or a threshold-driven
>> default.  But that'll take time.
>> 
>> As you mentioned, Ryan, readers may not correctly follow Thrift versioning
>> and could be buggy or implementers might not read release notes.  This is
>> true but I reason that those upgrading parquet-format without reading
>> release notes would also be those who don't know to gate on a new version
>> field, etc., in the other options on the list. Also, I know of one
>> mainstream widely-used Parquet reader that doesn't check the trailing file
>> magic for PAR1 either. So, it is what it is and the same problems will be
>> faced by any version-gate-in-file approach.
>> 
>> When this was voted on previously I had thought the versioning discussion
>> would focus on making it easier to reason about what readers and writers
>> support in feature matrices and those kinds of decisions that users opting
>> into features want to know about, perhaps by bumping the version of
>> parquet-format.  I wasn't thinking it'd be a version gate on the produced
>> Parquet files at rest themselves :D
>> 
>> Best,
>> Will
>> 
>>> On Thu, 10 Sept 2026 at 21:25, Ryan Blue <bl...@apache.org> wrote:
>>> 
>>> I'm also replying to Andrew here, but I think it helps to keep the
>> replies
>>> separate and focused on one topic.
>>> 
>>>> From what I can tell, the current state of parquet is implicitly
>> Option 2
>>>> (as readers can and do try to read any file and skip or error when
>>>> encountering unsupported features), and don't check version numbers.
>>> 
>>>> [With option 2] evolving the format would have
>>>> additional requirements (but the same requirements already implicitly
>>>> exist)
>>> 
>>> I don't agree that option 2 is the current state. I think the current
>> state
>>> is that we don't make these breaking changes, and that's the problem I
>> want
>>> to fix.
>>> 
>>> We've released new encodings and compression that guarantee failure
>> because
>>> they use a new enum symbol, but will only break readers when those
>> columns
>>> are read. However, we have structured all of the other changes to be
>>> forward-compatible. For instance, when we found the sort bug for string
>>> columns, we introduced a second set of fields for lower and upper bounds.
>>> We also use structs to mimic enums when we want them to be forward
>>> compatible (like logical type annotations).
>>> 
>>> Without a way to make and coordinate breaking changes, I think we must
>>> adopt a guarantee like the one in Option 2. If we do that, I think we've
>>> made it even harder to evolve the format and I don't see why we would
>>> decide to make a breaking change to fix something like path_in_schema.
>>> 
>>> There's also another way to look at this: if we are confident that
>>> path_in_schema will break all older readers, why not use that
>> compatibility
>>> break to get other cleanup features in? Doing that is one of the
>> advantages
>>> of bundling.
>>> 
>>> On Thu, Sep 10, 2026 at 2:12 AM Andrew Lamb <an...@gmail.com>
>>> wrote:
>>> 
>>>> Here is my attempt to summarize the tradeoffs (Ryan's explanation
>> during
>>>> the call was very helpful for me).
>>>> 
>>>> The core tradeoff is in requirements for future changes to the Parquet
>>> spec
>>>> itself vs how many files particular readers can read.
>>>> 
>>>> If the spec requires that readers fail fast on unknown versions
>> ("Option
>>>> 1"),
>>>> * Pro: New changes to the spec don't have to consider existing readers,
>>> and
>>>> are thus in theory are easier/faster to make (e.g. Ryan's example of
>>>> relocatable page headers)
>>>> * Con: requires readers to fail on all files with newer versions, even
>>>> those files that the reader could have read correctly
>>>> 
>>>> If the spec allows readers to attempt to read unknown versions ("Option
>>> 2")
>>>> * Pro: Readers will be able to read more files, though it will be
>> harder
>>> to
>>>> reason up front if a reader can read any file (may have to test it)
>>>> * Con: There is a requirement on any future changes to the spec to
>> ensure
>>>> old readers don't interpret new features ("additional guarantee that
>>>> reading future formats will either fail or produce correct results").
>>> This
>>>> requirement is hard to define precisely given the wide and unknown
>>> variety
>>>> of readers
>>>> 
>>>> From what I can tell, the current state of parquet is implicitly
>> Option 2
>>>> (as readers can and do try to read any file and skip or error when
>>>> encountering unsupported features), and don't check version numbers.
>>>> 
>>>> My personal opinion is that allowing readers to read unknown versions
>>>> (option 2 / option 3) is the most practical: Changing the current
>>> implicit
>>>> behavior would be quite confusing, and making the spec harder to change
>>> for
>>>> wider read interoperability is the right tradeoff in my mind.
>>>> 
>>>> Andrew
>>>> 
>>>> 
>>>> p.s.
>>>> 
>>>>> I think path_in_schema is a good example of why option 2 will
>>> inevitably
>>>> produce
>>>> correctness bugs
>>>> 
>>>> It seems to me that making path_in_schema optional will simply cause
>> old
>>>> readers to fail if they need it, so is not a good example of why
>> option 2
>>>> would necessarily correctness issues compared to option 1. The example
>>>> about java hash sets could be avoided with adequate testing, for
>> example,
>>>> and I don't see how gating that code behind reading a new version
>> number
>>>> would make that bug any more/less likely.
>>>> 
>>>>> My second argument for why we should not attempt to read all future
>>>> versions
>>>> of Parquet is that it ends up limiting how we can evolve the format.
>>>> 
>>>> This makes a lot of sense to me -- evolving the format would have
>>>> additional requirements (but the same requirements already implicitly
>>>> exist); Adding relocatable pages, for example could be achieved by
>>> adding a
>>>> new DataPageHeaderV3 rather than modifying the existing structure. That
>>> is
>>>> more complicated to be sure, but not impossible.
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On Thu, Sep 10, 2026 at 4:39 AM Antoine Pitrou <an...@python.org>
>>> wrote:
>>>> 
>>>>> Le 10/09/2026 à 00:38, Ryan Blue a écrit :
>>>>>> 
>>>>>> There are 3 main options:
>>>>>> 1. A reader should fail because it does not support the version
>>>>>> 2. A reader should attempt to read the file
>>>>>> 3. This choice is left up to implementations
>>>>>> 
>>>>>> I'll cover each option in more detail below, but first I want to
>>>> clarify
>>>>>> that we are not talking about "preview" features like encodings or
>>>>>> forward-compatible changes like new logical types. Preview features
>>>> will
>>>>>> break readers that do not support them and only affect specific
>>> columns
>>>>>> using the feature. For preview features, the expectation is that
>>>> readers
>>>>>> will attempt to read the file and will fail if they need to
>> project a
>>>>>> column that cannot be read.
>>>>> 
>>>>> "Preview features" is a very weird terminology. It sounds like
>>>>> "unfinished" or "experimental".
>>>>> 
>>>>>> The choice of how to handle an unsupported format version primarily
>>>>> affects
>>>>>> changes that add, remove, or modify the semantics of metadata
>> fields.
>>>> For
>>>>>> example:
>>>>>> - Changing `path_in_schema` from required to optional
>>>>> 
>>>>> This depends whether the Thrift parser checks that required fields
>> are
>>>>> actually present in the serialized payload? Do we know what their
>>>>> current behavior is?
>>>>> 
>>>>> i.e., does a Thrift parser generated with a required `path_in_schema`
>>>>> specification accept a serialized payload without that field?
>>>>> 
>>>>> It also depends what the Parquet reader actually *does* with the
>>>>> `path_in_schema`? AFAICT, the Parquet C++ reader isn't doing anything
>>>>> specific with it.
>>>>> 
>>>>>> Option 3 would mean that readers may choose to attempt to read, but
>>> do
>>>>> not
>>>>>> have additional guarantees.
>>>>> 
>>>>> Option 3 can also mean "the reader is exposing an option to let the
>>> user
>>>>> choose the behavior (reject up front or attempt to read anyway)".
>>>>> 
>>>>> (this would be more costly to implement, so I'm not sure any
>>>>> implementation would actually do that; but it's at least conceptually
>>>>> possible)
>>>>> 
>>>>> Regards
>>>>> 
>>>>> Antoine.
>>>>> 
>>>>> 
>>>>> 
>>>> 
>>> 
>> 

Re: [DISCUSS] How should readers behave with unsupported format versions?

[Andrew Bell <an...@gmail.com>]

On Mon, Sep 14, 2026 at 4:50 PM Ryan Blue <bl...@apache.org> wrote:

> > I support "option 0," which I'll place at the top of the list as:
> > 'Changes in Parquet files are versioned in the current way.'
>
> I think that the "current way" is that we don't make breaking changes. I
> voted against changing `path_in_schema` to optional because I think it
> breaks our guarantees. I think it's important for us to be able to agree on
> how to make breaking changes.
>

Well, you don't make breaking changes :) But enough people seem not to care
that the horse is out of the barn and "what a reader should do" seems
beside the point. How can you enforce such a thing? A reader can do
whatever it wants and it may do so at the convenience of the author or
users.  If you want to make suggested behavior, that seems fine and perhaps
just stick it as a comment in the thrift file for places where there is an
exception on an individual basis -- I don't think you need to formulate
blanket policy.

The thrift idl already has oddities like ColumnChunk::meta_data that is
defined as "optional", but comes with a comment that says "As such, writers
MUST populate this field".

What are you going to do?

-- 
Andrew Bell
andrew.bell.ia@gmail.com

Re: [DISCUSS] How should readers behave with unsupported format versions?

[Ryan Blue <bl...@apache.org>]

> I support "option 0," which I'll place at the top of the list as:
> 'Changes in Parquet files are versioned in the current way.'

I think that the "current way" is that we don't make breaking changes. I
voted against changing `path_in_schema` to optional because I think it
breaks our guarantees. I think it's important for us to be able to agree on
how to make breaking changes.



On Sun, Sep 13, 2026 at 11:50 PM Will Edwards via dev <
dev@parquet.apache.org> wrote:

> Hi Ryan, thx for clarifying :thumb-up:
>
> So, I support "option 0," which I'll place at the top of the list as:
> 'Changes in Parquet files are versioned in the current way.'
>
> Using path_in_schema as the example big braking change I could imagine
> getting traction:
>
> When we make path-in-schema optional we might bump the version number of
> the parquet-format package as a hint for implementers to read the release
> notes, but we won't put a version gate on the files themselves.  We expect
> it to be opted-in only by those most affected by footer bloat and only on
> those files most affected by that bloat.  With the passage of time more
> users might be reaching for it and the reader base will have matured and we
> might contemplate some writers making it the default or a threshold-driven
> default.  But that'll take time.
>
> As you mentioned, Ryan, readers may not correctly follow Thrift versioning
> and could be buggy or implementers might not read release notes.  This is
> true but I reason that those upgrading parquet-format without reading
> release notes would also be those who don't know to gate on a new version
> field, etc., in the other options on the list. Also, I know of one
> mainstream widely-used Parquet reader that doesn't check the trailing file
> magic for PAR1 either. So, it is what it is and the same problems will be
> faced by any version-gate-in-file approach.
>
> When this was voted on previously I had thought the versioning discussion
> would focus on making it easier to reason about what readers and writers
> support in feature matrices and those kinds of decisions that users opting
> into features want to know about, perhaps by bumping the version of
> parquet-format.  I wasn't thinking it'd be a version gate on the produced
> Parquet files at rest themselves :D
>
> Best,
> Will
>
> On Thu, 10 Sept 2026 at 21:25, Ryan Blue <bl...@apache.org> wrote:
>
> > I'm also replying to Andrew here, but I think it helps to keep the
> replies
> > separate and focused on one topic.
> >
> > > From what I can tell, the current state of parquet is implicitly
> Option 2
> > > (as readers can and do try to read any file and skip or error when
> > > encountering unsupported features), and don't check version numbers.
> >
> > > [With option 2] evolving the format would have
> > > additional requirements (but the same requirements already implicitly
> > > exist)
> >
> > I don't agree that option 2 is the current state. I think the current
> state
> > is that we don't make these breaking changes, and that's the problem I
> want
> > to fix.
> >
> > We've released new encodings and compression that guarantee failure
> because
> > they use a new enum symbol, but will only break readers when those
> columns
> > are read. However, we have structured all of the other changes to be
> > forward-compatible. For instance, when we found the sort bug for string
> > columns, we introduced a second set of fields for lower and upper bounds.
> > We also use structs to mimic enums when we want them to be forward
> > compatible (like logical type annotations).
> >
> > Without a way to make and coordinate breaking changes, I think we must
> > adopt a guarantee like the one in Option 2. If we do that, I think we've
> > made it even harder to evolve the format and I don't see why we would
> > decide to make a breaking change to fix something like path_in_schema.
> >
> > There's also another way to look at this: if we are confident that
> > path_in_schema will break all older readers, why not use that
> compatibility
> > break to get other cleanup features in? Doing that is one of the
> advantages
> > of bundling.
> >
> > On Thu, Sep 10, 2026 at 2:12 AM Andrew Lamb <an...@gmail.com>
> > wrote:
> >
> > > Here is my attempt to summarize the tradeoffs (Ryan's explanation
> during
> > > the call was very helpful for me).
> > >
> > > The core tradeoff is in requirements for future changes to the Parquet
> > spec
> > > itself vs how many files particular readers can read.
> > >
> > > If the spec requires that readers fail fast on unknown versions
> ("Option
> > > 1"),
> > > * Pro: New changes to the spec don't have to consider existing readers,
> > and
> > > are thus in theory are easier/faster to make (e.g. Ryan's example of
> > > relocatable page headers)
> > > * Con: requires readers to fail on all files with newer versions, even
> > > those files that the reader could have read correctly
> > >
> > > If the spec allows readers to attempt to read unknown versions ("Option
> > 2")
> > > * Pro: Readers will be able to read more files, though it will be
> harder
> > to
> > > reason up front if a reader can read any file (may have to test it)
> > > * Con: There is a requirement on any future changes to the spec to
> ensure
> > > old readers don't interpret new features ("additional guarantee that
> > > reading future formats will either fail or produce correct results").
> > This
> > > requirement is hard to define precisely given the wide and unknown
> > variety
> > > of readers
> > >
> > > From what I can tell, the current state of parquet is implicitly
> Option 2
> > > (as readers can and do try to read any file and skip or error when
> > > encountering unsupported features), and don't check version numbers.
> > >
> > > My personal opinion is that allowing readers to read unknown versions
> > > (option 2 / option 3) is the most practical: Changing the current
> > implicit
> > > behavior would be quite confusing, and making the spec harder to change
> > for
> > > wider read interoperability is the right tradeoff in my mind.
> > >
> > > Andrew
> > >
> > >
> > > p.s.
> > >
> > > > I think path_in_schema is a good example of why option 2 will
> > inevitably
> > > produce
> > > correctness bugs
> > >
> > > It seems to me that making path_in_schema optional will simply cause
> old
> > > readers to fail if they need it, so is not a good example of why
> option 2
> > > would necessarily correctness issues compared to option 1. The example
> > > about java hash sets could be avoided with adequate testing, for
> example,
> > > and I don't see how gating that code behind reading a new version
> number
> > > would make that bug any more/less likely.
> > >
> > > > My second argument for why we should not attempt to read all future
> > > versions
> > > of Parquet is that it ends up limiting how we can evolve the format.
> > >
> > > This makes a lot of sense to me -- evolving the format would have
> > > additional requirements (but the same requirements already implicitly
> > > exist); Adding relocatable pages, for example could be achieved by
> > adding a
> > > new DataPageHeaderV3 rather than modifying the existing structure. That
> > is
> > > more complicated to be sure, but not impossible.
> > >
> > >
> > >
> > >
> > > On Thu, Sep 10, 2026 at 4:39 AM Antoine Pitrou <an...@python.org>
> > wrote:
> > >
> > > > Le 10/09/2026 à 00:38, Ryan Blue a écrit :
> > > > >
> > > > > There are 3 main options:
> > > > > 1. A reader should fail because it does not support the version
> > > > > 2. A reader should attempt to read the file
> > > > > 3. This choice is left up to implementations
> > > > >
> > > > > I'll cover each option in more detail below, but first I want to
> > > clarify
> > > > > that we are not talking about "preview" features like encodings or
> > > > > forward-compatible changes like new logical types. Preview features
> > > will
> > > > > break readers that do not support them and only affect specific
> > columns
> > > > > using the feature. For preview features, the expectation is that
> > > readers
> > > > > will attempt to read the file and will fail if they need to
> project a
> > > > > column that cannot be read.
> > > >
> > > > "Preview features" is a very weird terminology. It sounds like
> > > > "unfinished" or "experimental".
> > > >
> > > > > The choice of how to handle an unsupported format version primarily
> > > > affects
> > > > > changes that add, remove, or modify the semantics of metadata
> fields.
> > > For
> > > > > example:
> > > > > - Changing `path_in_schema` from required to optional
> > > >
> > > > This depends whether the Thrift parser checks that required fields
> are
> > > > actually present in the serialized payload? Do we know what their
> > > > current behavior is?
> > > >
> > > > i.e., does a Thrift parser generated with a required `path_in_schema`
> > > > specification accept a serialized payload without that field?
> > > >
> > > > It also depends what the Parquet reader actually *does* with the
> > > > `path_in_schema`? AFAICT, the Parquet C++ reader isn't doing anything
> > > > specific with it.
> > > >
> > > > > Option 3 would mean that readers may choose to attempt to read, but
> > do
> > > > not
> > > > > have additional guarantees.
> > > >
> > > > Option 3 can also mean "the reader is exposing an option to let the
> > user
> > > > choose the behavior (reject up front or attempt to read anyway)".
> > > >
> > > > (this would be more costly to implement, so I'm not sure any
> > > > implementation would actually do that; but it's at least conceptually
> > > > possible)
> > > >
> > > > Regards
> > > >
> > > > Antoine.
> > > >
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] How should readers behave with unsupported format versions?

[Will Edwards via dev <de...@parquet.apache.org>]

Hi Ryan, thx for clarifying :thumb-up:

So, I support "option 0," which I'll place at the top of the list as:
'Changes in Parquet files are versioned in the current way.'

Using path_in_schema as the example big braking change I could imagine
getting traction:

When we make path-in-schema optional we might bump the version number of
the parquet-format package as a hint for implementers to read the release
notes, but we won't put a version gate on the files themselves.  We expect
it to be opted-in only by those most affected by footer bloat and only on
those files most affected by that bloat.  With the passage of time more
users might be reaching for it and the reader base will have matured and we
might contemplate some writers making it the default or a threshold-driven
default.  But that'll take time.

As you mentioned, Ryan, readers may not correctly follow Thrift versioning
and could be buggy or implementers might not read release notes.  This is
true but I reason that those upgrading parquet-format without reading
release notes would also be those who don't know to gate on a new version
field, etc., in the other options on the list. Also, I know of one
mainstream widely-used Parquet reader that doesn't check the trailing file
magic for PAR1 either. So, it is what it is and the same problems will be
faced by any version-gate-in-file approach.

When this was voted on previously I had thought the versioning discussion
would focus on making it easier to reason about what readers and writers
support in feature matrices and those kinds of decisions that users opting
into features want to know about, perhaps by bumping the version of
parquet-format.  I wasn't thinking it'd be a version gate on the produced
Parquet files at rest themselves :D

Best,
Will

On Thu, 10 Sept 2026 at 21:25, Ryan Blue <bl...@apache.org> wrote:

> I'm also replying to Andrew here, but I think it helps to keep the replies
> separate and focused on one topic.
>
> > From what I can tell, the current state of parquet is implicitly Option 2
> > (as readers can and do try to read any file and skip or error when
> > encountering unsupported features), and don't check version numbers.
>
> > [With option 2] evolving the format would have
> > additional requirements (but the same requirements already implicitly
> > exist)
>
> I don't agree that option 2 is the current state. I think the current state
> is that we don't make these breaking changes, and that's the problem I want
> to fix.
>
> We've released new encodings and compression that guarantee failure because
> they use a new enum symbol, but will only break readers when those columns
> are read. However, we have structured all of the other changes to be
> forward-compatible. For instance, when we found the sort bug for string
> columns, we introduced a second set of fields for lower and upper bounds.
> We also use structs to mimic enums when we want them to be forward
> compatible (like logical type annotations).
>
> Without a way to make and coordinate breaking changes, I think we must
> adopt a guarantee like the one in Option 2. If we do that, I think we've
> made it even harder to evolve the format and I don't see why we would
> decide to make a breaking change to fix something like path_in_schema.
>
> There's also another way to look at this: if we are confident that
> path_in_schema will break all older readers, why not use that compatibility
> break to get other cleanup features in? Doing that is one of the advantages
> of bundling.
>
> On Thu, Sep 10, 2026 at 2:12 AM Andrew Lamb <an...@gmail.com>
> wrote:
>
> > Here is my attempt to summarize the tradeoffs (Ryan's explanation during
> > the call was very helpful for me).
> >
> > The core tradeoff is in requirements for future changes to the Parquet
> spec
> > itself vs how many files particular readers can read.
> >
> > If the spec requires that readers fail fast on unknown versions ("Option
> > 1"),
> > * Pro: New changes to the spec don't have to consider existing readers,
> and
> > are thus in theory are easier/faster to make (e.g. Ryan's example of
> > relocatable page headers)
> > * Con: requires readers to fail on all files with newer versions, even
> > those files that the reader could have read correctly
> >
> > If the spec allows readers to attempt to read unknown versions ("Option
> 2")
> > * Pro: Readers will be able to read more files, though it will be harder
> to
> > reason up front if a reader can read any file (may have to test it)
> > * Con: There is a requirement on any future changes to the spec to ensure
> > old readers don't interpret new features ("additional guarantee that
> > reading future formats will either fail or produce correct results").
> This
> > requirement is hard to define precisely given the wide and unknown
> variety
> > of readers
> >
> > From what I can tell, the current state of parquet is implicitly Option 2
> > (as readers can and do try to read any file and skip or error when
> > encountering unsupported features), and don't check version numbers.
> >
> > My personal opinion is that allowing readers to read unknown versions
> > (option 2 / option 3) is the most practical: Changing the current
> implicit
> > behavior would be quite confusing, and making the spec harder to change
> for
> > wider read interoperability is the right tradeoff in my mind.
> >
> > Andrew
> >
> >
> > p.s.
> >
> > > I think path_in_schema is a good example of why option 2 will
> inevitably
> > produce
> > correctness bugs
> >
> > It seems to me that making path_in_schema optional will simply cause old
> > readers to fail if they need it, so is not a good example of why option 2
> > would necessarily correctness issues compared to option 1. The example
> > about java hash sets could be avoided with adequate testing, for example,
> > and I don't see how gating that code behind reading a new version number
> > would make that bug any more/less likely.
> >
> > > My second argument for why we should not attempt to read all future
> > versions
> > of Parquet is that it ends up limiting how we can evolve the format.
> >
> > This makes a lot of sense to me -- evolving the format would have
> > additional requirements (but the same requirements already implicitly
> > exist); Adding relocatable pages, for example could be achieved by
> adding a
> > new DataPageHeaderV3 rather than modifying the existing structure. That
> is
> > more complicated to be sure, but not impossible.
> >
> >
> >
> >
> > On Thu, Sep 10, 2026 at 4:39 AM Antoine Pitrou <an...@python.org>
> wrote:
> >
> > > Le 10/09/2026 à 00:38, Ryan Blue a écrit :
> > > >
> > > > There are 3 main options:
> > > > 1. A reader should fail because it does not support the version
> > > > 2. A reader should attempt to read the file
> > > > 3. This choice is left up to implementations
> > > >
> > > > I'll cover each option in more detail below, but first I want to
> > clarify
> > > > that we are not talking about "preview" features like encodings or
> > > > forward-compatible changes like new logical types. Preview features
> > will
> > > > break readers that do not support them and only affect specific
> columns
> > > > using the feature. For preview features, the expectation is that
> > readers
> > > > will attempt to read the file and will fail if they need to project a
> > > > column that cannot be read.
> > >
> > > "Preview features" is a very weird terminology. It sounds like
> > > "unfinished" or "experimental".
> > >
> > > > The choice of how to handle an unsupported format version primarily
> > > affects
> > > > changes that add, remove, or modify the semantics of metadata fields.
> > For
> > > > example:
> > > > - Changing `path_in_schema` from required to optional
> > >
> > > This depends whether the Thrift parser checks that required fields are
> > > actually present in the serialized payload? Do we know what their
> > > current behavior is?
> > >
> > > i.e., does a Thrift parser generated with a required `path_in_schema`
> > > specification accept a serialized payload without that field?
> > >
> > > It also depends what the Parquet reader actually *does* with the
> > > `path_in_schema`? AFAICT, the Parquet C++ reader isn't doing anything
> > > specific with it.
> > >
> > > > Option 3 would mean that readers may choose to attempt to read, but
> do
> > > not
> > > > have additional guarantees.
> > >
> > > Option 3 can also mean "the reader is exposing an option to let the
> user
> > > choose the behavior (reject up front or attempt to read anyway)".
> > >
> > > (this would be more costly to implement, so I'm not sure any
> > > implementation would actually do that; but it's at least conceptually
> > > possible)
> > >
> > > Regards
> > >
> > > Antoine.
> > >
> > >
> > >
> >
>

Re: [DISCUSS] How should readers behave with unsupported format versions?

[Ryan Blue <bl...@apache.org>]

I'm also replying to Andrew here, but I think it helps to keep the replies
separate and focused on one topic.

> From what I can tell, the current state of parquet is implicitly Option 2
> (as readers can and do try to read any file and skip or error when
> encountering unsupported features), and don't check version numbers.

> [With option 2] evolving the format would have
> additional requirements (but the same requirements already implicitly
> exist)

I don't agree that option 2 is the current state. I think the current state
is that we don't make these breaking changes, and that's the problem I want
to fix.

We've released new encodings and compression that guarantee failure because
they use a new enum symbol, but will only break readers when those columns
are read. However, we have structured all of the other changes to be
forward-compatible. For instance, when we found the sort bug for string
columns, we introduced a second set of fields for lower and upper bounds.
We also use structs to mimic enums when we want them to be forward
compatible (like logical type annotations).

Without a way to make and coordinate breaking changes, I think we must
adopt a guarantee like the one in Option 2. If we do that, I think we've
made it even harder to evolve the format and I don't see why we would
decide to make a breaking change to fix something like path_in_schema.

There's also another way to look at this: if we are confident that
path_in_schema will break all older readers, why not use that compatibility
break to get other cleanup features in? Doing that is one of the advantages
of bundling.

On Thu, Sep 10, 2026 at 2:12 AM Andrew Lamb <an...@gmail.com> wrote:

> Here is my attempt to summarize the tradeoffs (Ryan's explanation during
> the call was very helpful for me).
>
> The core tradeoff is in requirements for future changes to the Parquet spec
> itself vs how many files particular readers can read.
>
> If the spec requires that readers fail fast on unknown versions ("Option
> 1"),
> * Pro: New changes to the spec don't have to consider existing readers, and
> are thus in theory are easier/faster to make (e.g. Ryan's example of
> relocatable page headers)
> * Con: requires readers to fail on all files with newer versions, even
> those files that the reader could have read correctly
>
> If the spec allows readers to attempt to read unknown versions ("Option 2")
> * Pro: Readers will be able to read more files, though it will be harder to
> reason up front if a reader can read any file (may have to test it)
> * Con: There is a requirement on any future changes to the spec to ensure
> old readers don't interpret new features ("additional guarantee that
> reading future formats will either fail or produce correct results"). This
> requirement is hard to define precisely given the wide and unknown variety
> of readers
>
> From what I can tell, the current state of parquet is implicitly Option 2
> (as readers can and do try to read any file and skip or error when
> encountering unsupported features), and don't check version numbers.
>
> My personal opinion is that allowing readers to read unknown versions
> (option 2 / option 3) is the most practical: Changing the current implicit
> behavior would be quite confusing, and making the spec harder to change for
> wider read interoperability is the right tradeoff in my mind.
>
> Andrew
>
>
> p.s.
>
> > I think path_in_schema is a good example of why option 2 will inevitably
> produce
> correctness bugs
>
> It seems to me that making path_in_schema optional will simply cause old
> readers to fail if they need it, so is not a good example of why option 2
> would necessarily correctness issues compared to option 1. The example
> about java hash sets could be avoided with adequate testing, for example,
> and I don't see how gating that code behind reading a new version number
> would make that bug any more/less likely.
>
> > My second argument for why we should not attempt to read all future
> versions
> of Parquet is that it ends up limiting how we can evolve the format.
>
> This makes a lot of sense to me -- evolving the format would have
> additional requirements (but the same requirements already implicitly
> exist); Adding relocatable pages, for example could be achieved by adding a
> new DataPageHeaderV3 rather than modifying the existing structure. That is
> more complicated to be sure, but not impossible.
>
>
>
>
> On Thu, Sep 10, 2026 at 4:39 AM Antoine Pitrou <an...@python.org> wrote:
>
> > Le 10/09/2026 à 00:38, Ryan Blue a écrit :
> > >
> > > There are 3 main options:
> > > 1. A reader should fail because it does not support the version
> > > 2. A reader should attempt to read the file
> > > 3. This choice is left up to implementations
> > >
> > > I'll cover each option in more detail below, but first I want to
> clarify
> > > that we are not talking about "preview" features like encodings or
> > > forward-compatible changes like new logical types. Preview features
> will
> > > break readers that do not support them and only affect specific columns
> > > using the feature. For preview features, the expectation is that
> readers
> > > will attempt to read the file and will fail if they need to project a
> > > column that cannot be read.
> >
> > "Preview features" is a very weird terminology. It sounds like
> > "unfinished" or "experimental".
> >
> > > The choice of how to handle an unsupported format version primarily
> > affects
> > > changes that add, remove, or modify the semantics of metadata fields.
> For
> > > example:
> > > - Changing `path_in_schema` from required to optional
> >
> > This depends whether the Thrift parser checks that required fields are
> > actually present in the serialized payload? Do we know what their
> > current behavior is?
> >
> > i.e., does a Thrift parser generated with a required `path_in_schema`
> > specification accept a serialized payload without that field?
> >
> > It also depends what the Parquet reader actually *does* with the
> > `path_in_schema`? AFAICT, the Parquet C++ reader isn't doing anything
> > specific with it.
> >
> > > Option 3 would mean that readers may choose to attempt to read, but do
> > not
> > > have additional guarantees.
> >
> > Option 3 can also mean "the reader is exposing an option to let the user
> > choose the behavior (reject up front or attempt to read anyway)".
> >
> > (this would be more costly to implement, so I'm not sure any
> > implementation would actually do that; but it's at least conceptually
> > possible)
> >
> > Regards
> >
> > Antoine.
> >
> >
> >
>

Re: [DISCUSS] How should readers behave with unsupported format versions?

[Andrew Lamb <an...@gmail.com>]

Here is my attempt to summarize the tradeoffs (Ryan's explanation during
the call was very helpful for me).

The core tradeoff is in requirements for future changes to the Parquet spec
itself vs how many files particular readers can read.

If the spec requires that readers fail fast on unknown versions ("Option
1"),
* Pro: New changes to the spec don't have to consider existing readers, and
are thus in theory are easier/faster to make (e.g. Ryan's example of
relocatable page headers)
* Con: requires readers to fail on all files with newer versions, even
those files that the reader could have read correctly

If the spec allows readers to attempt to read unknown versions ("Option 2")
* Pro: Readers will be able to read more files, though it will be harder to
reason up front if a reader can read any file (may have to test it)
* Con: There is a requirement on any future changes to the spec to ensure
old readers don't interpret new features ("additional guarantee that
reading future formats will either fail or produce correct results"). This
requirement is hard to define precisely given the wide and unknown variety
of readers

From what I can tell, the current state of parquet is implicitly Option 2
(as readers can and do try to read any file and skip or error when
encountering unsupported features), and don't check version numbers.

My personal opinion is that allowing readers to read unknown versions
(option 2 / option 3) is the most practical: Changing the current implicit
behavior would be quite confusing, and making the spec harder to change for
wider read interoperability is the right tradeoff in my mind.

Andrew


p.s.

> I think path_in_schema is a good example of why option 2 will inevitably produce
correctness bugs

It seems to me that making path_in_schema optional will simply cause old
readers to fail if they need it, so is not a good example of why option 2
would necessarily correctness issues compared to option 1. The example
about java hash sets could be avoided with adequate testing, for example,
and I don't see how gating that code behind reading a new version number
would make that bug any more/less likely.

> My second argument for why we should not attempt to read all future versions
of Parquet is that it ends up limiting how we can evolve the format.

This makes a lot of sense to me -- evolving the format would have
additional requirements (but the same requirements already implicitly
exist); Adding relocatable pages, for example could be achieved by adding a
new DataPageHeaderV3 rather than modifying the existing structure. That is
more complicated to be sure, but not impossible.




On Thu, Sep 10, 2026 at 4:39 AM Antoine Pitrou <an...@python.org> wrote:

> Le 10/09/2026 à 00:38, Ryan Blue a écrit :
> >
> > There are 3 main options:
> > 1. A reader should fail because it does not support the version
> > 2. A reader should attempt to read the file
> > 3. This choice is left up to implementations
> >
> > I'll cover each option in more detail below, but first I want to clarify
> > that we are not talking about "preview" features like encodings or
> > forward-compatible changes like new logical types. Preview features will
> > break readers that do not support them and only affect specific columns
> > using the feature. For preview features, the expectation is that readers
> > will attempt to read the file and will fail if they need to project a
> > column that cannot be read.
>
> "Preview features" is a very weird terminology. It sounds like
> "unfinished" or "experimental".
>
> > The choice of how to handle an unsupported format version primarily
> affects
> > changes that add, remove, or modify the semantics of metadata fields. For
> > example:
> > - Changing `path_in_schema` from required to optional
>
> This depends whether the Thrift parser checks that required fields are
> actually present in the serialized payload? Do we know what their
> current behavior is?
>
> i.e., does a Thrift parser generated with a required `path_in_schema`
> specification accept a serialized payload without that field?
>
> It also depends what the Parquet reader actually *does* with the
> `path_in_schema`? AFAICT, the Parquet C++ reader isn't doing anything
> specific with it.
>
> > Option 3 would mean that readers may choose to attempt to read, but do
> not
> > have additional guarantees.
>
> Option 3 can also mean "the reader is exposing an option to let the user
> choose the behavior (reject up front or attempt to read anyway)".
>
> (this would be more costly to implement, so I'm not sure any
> implementation would actually do that; but it's at least conceptually
> possible)
>
> Regards
>
> Antoine.
>
>
>

Re: [DISCUSS] How should readers behave with unsupported format versions?

[Andrew Lamb <an...@gmail.com>]

Here is my attempt to summarize the tradeoffs (Ryan's explanation during
the call was very helpful for me).

The core tradeoff is in requirements for future changes to the Parquet spec
itself vs how many files particular readers can read.

If the spec requires that readers fail fast on unknown versions ("Option
1"),
* Pro: New changes to the spec don't have to consider existing readers, and
are thus in theory are easier/faster to make (e.g. Ryan's example of
relocatable page headers)
* Con: requires readers to fail on all files with newer versions, even
those files that the reader could have read correctly

If the spec allows readers to attempt to read unknown versions ("Option 2")
* Pro: Readers will be able to read more files, though it will be harder to
reason up front if a reader can read any file (may have to test it)
* Con: There is a requirement on any future changes to the spec to ensure
old readers don't interpret new features ("additional guarantee that
reading future formats will either fail or produce correct results"). This
requirement is hard to define precisely given the wide and unknown variety
of readers

From what I can tell, the current state of parquet is implicitly Option 2
(as readers can and do try to read any file and skip or error when
encountering unsupported features), and don't check version numbers.

My personal opinion is that allowing readers to read unknown versions
(option 2 / option 3) is the most practical: Changing the current implicit
behavior would be quite confusing, and making the spec harder to change for
wider read interoperability is the right tradeoff in my mind.

Andrew


p.s.

> I think path_in_schema is a good example of why option 2 will inevitably produce
correctness bugs

It seems to me that making path_in_schema optional will simply cause old
readers to fail if they need it, so is not a good example of why option 2
would necessarily correctness issues compared to option 1. The example
about java hash sets could be avoided with adequate testing, for example,
and I don't see how gating that code behind reading a new version number
would make that bug any more/less likely.

> My second argument for why we should not attempt to read all future versions
of Parquet is that it ends up limiting how we can evolve the format.

This makes a lot of sense to me -- evolving the format would have
additional requirements (but the same requirements already implicitly
exist); Adding relocatable pages, for example could be achieved by adding a
new DataPageHeaderV3 rather than modifying the existing structure. That is
more complicated to be sure, but not impossible.




On Thu, Sep 10, 2026 at 4:39 AM Antoine Pitrou <an...@python.org> wrote:

> Le 10/09/2026 à 00:38, Ryan Blue a écrit :
> >
> > There are 3 main options:
> > 1. A reader should fail because it does not support the version
> > 2. A reader should attempt to read the file
> > 3. This choice is left up to implementations
> >
> > I'll cover each option in more detail below, but first I want to clarify
> > that we are not talking about "preview" features like encodings or
> > forward-compatible changes like new logical types. Preview features will
> > break readers that do not support them and only affect specific columns
> > using the feature. For preview features, the expectation is that readers
> > will attempt to read the file and will fail if they need to project a
> > column that cannot be read.
>
> "Preview features" is a very weird terminology. It sounds like
> "unfinished" or "experimental".
>
> > The choice of how to handle an unsupported format version primarily
> affects
> > changes that add, remove, or modify the semantics of metadata fields. For
> > example:
> > - Changing `path_in_schema` from required to optional
>
> This depends whether the Thrift parser checks that required fields are
> actually present in the serialized payload? Do we know what their
> current behavior is?
>
> i.e., does a Thrift parser generated with a required `path_in_schema`
> specification accept a serialized payload without that field?
>
> It also depends what the Parquet reader actually *does* with the
> `path_in_schema`? AFAICT, the Parquet C++ reader isn't doing anything
> specific with it.
>
> > Option 3 would mean that readers may choose to attempt to read, but do
> not
> > have additional guarantees.
>
> Option 3 can also mean "the reader is exposing an option to let the user
> choose the behavior (reject up front or attempt to read anyway)".
>
> (this would be more costly to implement, so I'm not sure any
> implementation would actually do that; but it's at least conceptually
> possible)
>
> Regards
>
> Antoine.
>
>
>

Re: [DISCUSS] How should readers behave with unsupported format versions?

[Antoine Pitrou <an...@python.org>]

Le 10/09/2026 à 00:38, Ryan Blue a écrit :
> 
> There are 3 main options:
> 1. A reader should fail because it does not support the version
> 2. A reader should attempt to read the file
> 3. This choice is left up to implementations
> 
> I'll cover each option in more detail below, but first I want to clarify
> that we are not talking about "preview" features like encodings or
> forward-compatible changes like new logical types. Preview features will
> break readers that do not support them and only affect specific columns
> using the feature. For preview features, the expectation is that readers
> will attempt to read the file and will fail if they need to project a
> column that cannot be read.

"Preview features" is a very weird terminology. It sounds like 
"unfinished" or "experimental".

> The choice of how to handle an unsupported format version primarily affects
> changes that add, remove, or modify the semantics of metadata fields. For
> example:
> - Changing `path_in_schema` from required to optional

This depends whether the Thrift parser checks that required fields are 
actually present in the serialized payload? Do we know what their 
current behavior is?

i.e., does a Thrift parser generated with a required `path_in_schema` 
specification accept a serialized payload without that field?

It also depends what the Parquet reader actually *does* with the 
`path_in_schema`? AFAICT, the Parquet C++ reader isn't doing anything 
specific with it.

> Option 3 would mean that readers may choose to attempt to read, but do not
> have additional guarantees.

Option 3 can also mean "the reader is exposing an option to let the user 
choose the behavior (reject up front or attempt to read anyway)".

(this would be more costly to implement, so I'm not sure any 
implementation would actually do that; but it's at least conceptually 
possible)

Regards

Antoine.

Re: [DISCUSS] How should readers behave with unsupported format versions?

[Will Edwards via dev <de...@parquet.apache.org>]

Howdy folks,

> The question that we identified is: How should a reader behave when it
encounters a file written with an unsupported format version?

Isn’t option 2 essentially how Parquet uses Thrift versioning today? A
recorded format version should not, by itself, be grounds for rejecting a
file.

Preserving existing interpretations, making compatible additions ignorable,
and explicitly distinguishing features that require new understanding is a
widely used format-evolution strategy. It is particularly well suited to
Parquet’s role as a ubiquitous interchange and archival format, where
readers and writers evolve (or don't!) independently.

The examples to motivate a need for a version gate need to distinguish
implementation defects, incompatible changes to existing structures, and
limitations of the evolution model.  None of them seem to be limitations in
the current evolution model:  Missing required fields already have defined
failure behavior. New page semantics can receive a new page type. Incorrect
statistics require correcting the writer and handling affected files.

I'm not seeing a need to change the interpretation of existing page types
or to bump versions to deal with bugs in specific writers etc.  Preserving
interoperability seems a good reason to accept the thrift versioning
constraints.  We can carry on using the current thrift versioning system.

Best,
Will


On Thu, 10 Sept 2026 at 00:38, Ryan Blue <bl...@apache.org> wrote:

> Hi everyone,
>
> This is a new thread to discuss an issue raised in the "Finalizing the
> versioning proposal" thread. I suggested starting a new thread to focus on
> this question in the community sync and there was agreement so that we can
> get more attention to the issue.
>
> The question that we identified is: How should a reader behave when it
> encounters a file written with an unsupported format version?
>
> There are 3 main options:
> 1. A reader should fail because it does not support the version
> 2. A reader should attempt to read the file
> 3. This choice is left up to implementations
>
> I'll cover each option in more detail below, but first I want to clarify
> that we are not talking about "preview" features like encodings or
> forward-compatible changes like new logical types. Preview features will
> break readers that do not support them and only affect specific columns
> using the feature. For preview features, the expectation is that readers
> will attempt to read the file and will fail if they need to project a
> column that cannot be read.
>
> The choice of how to handle an unsupported format version primarily affects
> changes that add, remove, or modify the semantics of metadata fields. For
> example:
> - Changing `path_in_schema` from required to optional
> - Adding `offset` and `size` fields to a page header to relocate page data
> - Fixing stats written with the wrong sort order
>
> If we choose option 1, then a v3 reader that encounters a v4 file would be
> required to fail with an error that the file is too new to be read. With
> the current versioning scheme, the forward-compatibility within a version
> means that a v3 reader can read any v3 file and will produce the correct
> data. But this guarantee does not extend to any higher version. Without a
> guarantee that results are correct, the v3 reader must reject a v4 file.
>
> If we choose option 2, then the v3 reader will attempt to read the v4 file.
> In this case, Parquet will need to provide an additional guarantee that
> reading future formats will either fail or produce correct results. If I
> understand correctly, this option would use Thrift's semantics to cause
> failures. For example, path_in_schema would be required in Thrift and
> readers are expected to fail in decoding.
>
> Option 3 would mean that readers may choose to attempt to read, but do not
> have additional guarantees.
>
> Hopefully this makes the choices clear and we can have a discussion on this
> thread. Let's try to keep this thread on the topic of these options, either
> to clarify them or to debate the effects. We can discuss process or other
> topics on new threads.
>
> Ryan
>

Re: [DISCUSS] How should readers behave with unsupported format versions?

[Ryan Blue <bl...@apache.org>]

I'll start off discussion on this thread in a separate email so my opinion
is separate from the context for this discussion where I was staying
neutral.

I strongly support option 1, that readers must fail when reading a file
written with an unsupported format version.

The trade-off between these options comes down to the additional guarantee
needed for option 2, that readers will either read correctly or fail. The
scope of this guarantee is impractical, for two reasons. First, the scope
includes all implementations and all future versions. This huge surface
area will inevitably lead to bad assumptions and correctness bugs. Second,
this guarantee would severely limit Parquet's ability to evolve as a
format. Every metadata change would be more expensive to design and
validate.

In the Parquet sync, Will asked why Thrift's evolution rules aren't enough.
I think path_in_schema is a good example of why option 2 will inevitably
produce correctness bugs. For anyone not familiar, path_in_schema is a
string list in each ColumnMetaData that contains the column path, like
["request", "url"]. Many implementations don't need this path and we would
like to make the footer lighter by omitting it.

The rationale is that when Thrift encounters a required field that is
missing, it should throw an exception. This assumes the error behavior is
universal -- and we don't know that. I know of at least two custom Thrift
implementations that were built for faster deserialization. I don't know
how those implementations deserialize required fields. Maybe they assume
path_in_schema is always present because of a writer guarantee that has
been true up until now? Couldn't you assume that if it's required, then the
writer will always produce it?

We also cannot necessarily rely on the input to Thrift marking the field
required. Some projects automatically pull the thrift definition from
parquet-format (including Parquet Java until recently). If an
implementation updated its Thrift definition, it could easily pull in a new
version in which path_in_schema is optional and the read-time validation we
expect from Thrift would no longer happen. From there, it could easily lead
to a correctness bug if the implementation used utilities like Java's
HashMap and Apache Commons StringUtils, which are both null-tolerant:
StringUtils.join(null, ".") produces null, and HashMap works fine with null
keys. Unexpected null paths could lead to missing columns when accessed by
name.

I'll note that Will objected to the path_in_schema example in the sync
because implementations "should not" sync the Thrift definition like this.
But I don't think that's true. Parquet Java would sync the Thrift file
<https://github.com/apache/parquet-java/pull/3611> as I'm describing. The
writer requirement to continue writing path_in_schema kept implementations
safe, not the Thrift definition. The safe option is to not make a breaking
change to omit the field, but I think we all agree that we need to be able
to make this type of change.

My second argument for why we should not attempt to read all future
versions of Parquet is that it ends up limiting how we can evolve the
format. I think one of the most important outcomes of this versioning
discussion is the ability to move more quickly and update the format.
Adding a guarantee that all older readers will continue to read correctly
or will fail means that we can never change the meaning of a field or add
fields that change how a structure is interpreted. A quick example is
path_in_schema again, where I think the safe option is to continue writing
the field forever.

Another example is a discussion related to Dan's proposal for
non-contiguous pages (not a single chunk per column per row group). One of
the alternatives we have brought up internally is still using the same page
layout, but being able to point to 1page data elsewhere in the file by
adding `offset` and `size` fields to the page header.

If we were to add `offset` and `size` to a page header, we would change the
meaning of the header. Readers that don't know about these two new fields
would ignore them (hm, probably?). These older readers would not know the
content was relocated and could start decoding the bits directly after the
header. That's not good, so if we are making a guarantee that these readers
should never produce bad data, then we should instead add a new symbol to
the PageType, DATA_PAGE_V3, and copy all the fields from the previous page
header. But we should only use DATA_PAGE_V3 for pages that require the
`offset` and `size`, or else existing readers can't read all of the
non-relocated columns. Now we would have 3 page types, along with the code
to handle each one <https://github.com/apache/parquet-java/pull/3611>.

Each time we want to add columns that affect how a metadata struct is
interpreted or slightly change the meaning of a column, we have to add a
copy instead in order to guarantee that the old one is available,
unchanged, for older readers. This guarantee would mean we can't move on
from past mistakes (like PARQUET-251 or other stats bugs) and would inflate
the size of metadata.

I think a better idea is the alternative, which is to break reader
compatibility at each new format version. When a file is targeted for a new
format version, we can remove path_in_schema and we can add new fields and
reuse existing structures. I think the project will move faster if we do
this and will be more reliable because we won't have to worry about the
behavior of unknown implementations. We state required behavior for a new
version and then we can rely on it.

I know there's also a concern about how to move between format releases.
This is a legitimate concern. I think the solution is to choose what is
forward-compatible, what goes into preview features and can be used early,
and what goes into a breaking format change. If at all possible we should
make changes forward-compatible so that we can add them to the current
format version. New features like encodings should be released as preview
features so they can be enabled early. People shouldn't have to weigh a
breaking format version against new features if possible. If we do those,
then hopefully we can make changes in a new format version small and easy
to support, like the changes for handling a missing path_in_schema field.

Ryan

On Wed, Sep 9, 2026 at 3:38 PM Ryan Blue <bl...@apache.org> wrote:

> Hi everyone,
>
> This is a new thread to discuss an issue raised in the "Finalizing the
> versioning proposal" thread. I suggested starting a new thread to focus on
> this question in the community sync and there was agreement so that we can
> get more attention to the issue.
>
> The question that we identified is: How should a reader behave when it
> encounters a file written with an unsupported format version?
>
> There are 3 main options:
> 1. A reader should fail because it does not support the version
> 2. A reader should attempt to read the file
> 3. This choice is left up to implementations
>
> I'll cover each option in more detail below, but first I want to clarify
> that we are not talking about "preview" features like encodings or
> forward-compatible changes like new logical types. Preview features will
> break readers that do not support them and only affect specific columns
> using the feature. For preview features, the expectation is that readers
> will attempt to read the file and will fail if they need to project a
> column that cannot be read.
>
> The choice of how to handle an unsupported format version primarily
> affects changes that add, remove, or modify the semantics of metadata
> fields. For example:
> - Changing `path_in_schema` from required to optional
> - Adding `offset` and `size` fields to a page header to relocate page data
> - Fixing stats written with the wrong sort order
>
> If we choose option 1, then a v3 reader that encounters a v4 file would be
> required to fail with an error that the file is too new to be read. With
> the current versioning scheme, the forward-compatibility within a version
> means that a v3 reader can read any v3 file and will produce the correct
> data. But this guarantee does not extend to any higher version. Without a
> guarantee that results are correct, the v3 reader must reject a v4 file.
>
> If we choose option 2, then the v3 reader will attempt to read the v4
> file. In this case, Parquet will need to provide an additional guarantee
> that reading future formats will either fail or produce correct results. If
> I understand correctly, this option would use Thrift's semantics to cause
> failures. For example, path_in_schema would be required in Thrift and
> readers are expected to fail in decoding.
>
> Option 3 would mean that readers may choose to attempt to read, but do not
> have additional guarantees.
>
> Hopefully this makes the choices clear and we can have a discussion on
> this thread. Let's try to keep this thread on the topic of these options,
> either to clarify them or to debate the effects. We can discuss process or
> other topics on new threads.
>
> Ryan
>

Re: [DISCUSS] How should readers behave with unsupported format versions?

[Steve Loughran <st...@apache.org>]

I don't see how any option except #1, "fail on an unsupported version" is
realistic. As "pretending you support a version you don't understand" is
simply a pretence.

The fact that a parquet lib is required to fail fast on an unknown version
is a strong argument against moving to a new version unless you have a
really, really good reason. the entire work for having flatbuf alongside
the thrift buffers is to deliver performance improvements while retaining
compatibility. I would argue for trying to go that way as long as possible

At the same time as that "as long as possible" clause, I could imagine a
parquet release "2.0" which can generate and understand classic parquet +
with flatbuff & thrift, or parquet without the thrift and maybe with some
other optimisations. By default it' might generate parquet 1.x files, but
it would be aware of the later parquet version so *would* be forward
compatible.

steve

(originally sent from my gmail account; should have been blocked, apologies
if it does actually get through and so arrives twice)

On Wed, 16 Sept 2026 at 20:56, Fokko Driesprong <fo...@apache.org> wrote:

> Sorry, I had to catch up here. I'm in favor of 1.
>
> Proposed proper breaking changes, like the new footer, will always break
> existing readers. I don't think we can handle this gracefully with option
> ≥2.
>
> For breaking changes, we should fail because we cannot control how the
> client will behave. Also, we don't control all implementations out there,
> so we can't make assumptions about whether they use a field. The only thing
> we control is the specification.
>
> The path-in-schema sounds like a harmless change, but I think it opens the
> door to a slippery slope of further breaking changes that will cause
> existing readers to break more and more, and folks will lose trust in the
> project. Like Xiening is saying, #1 is the clean option. Instead of cutting
> corners, we could make the path-in-schema optimization part of the new
> footer. Bundle all of this in a new version.
>
> Kind regards,
> Fokko
>
> On 2026/09/15 18:26:22 Xiening Dai wrote:
> > Short answer - I vote for option 1.
> >
> > I think what you are talking about here for option 1 and 2 are regarding
> the Apache Parquet library, or Parquet official spec definition. We cannot
> prevent anyone from implementing their own Parquet reader (and in fact I
> know a few engines do have their own implementation). So #3 is already a
> given for anything outside of Apache Parquet.
> >
> > Now when comes to option 1 and 2. I believe #1 is clean. A full stop
> provides well defined behavior. #2 is going to complicate the reader
> implementation. It's hard to test,  hard to define the behavior, and hard
> for user to understand and predict the results. On the surface, the reader
> can potential consume more files, but the extra complexity and ambiguities
> makes it less appealing.
> >
> > In addition to this, I've seen a proposal to change Parquet magic number
> to achieve option #1. I don't support such approach. There are many
> tools/scripts rely on that particular magic number to detect file type,
> such as s3 object crawler, etc., and they don't even need to be a reader.
> And they will be broken if we change magic number. I think using the file
> version should be good enough. We should just release a reader patch to
> fail fast when version is higher than what's supported. And this patch
> should be released before we upgrade Parquet version and introduce any
> breaking change.
> >
> > On 2026/09/09 22:38:19 Ryan Blue wrote:
> > > Hi everyone,
> > >
> > > This is a new thread to discuss an issue raised in the "Finalizing the
> > > versioning proposal" thread. I suggested starting a new thread to
> focus on
> > > this question in the community sync and there was agreement so that we
> can
> > > get more attention to the issue.
> > >
> > > The question that we identified is: How should a reader behave when it
> > > encounters a file written with an unsupported format version?
> > >
> > > There are 3 main options:
> > > 1. A reader should fail because it does not support the version
> > > 2. A reader should attempt to read the file
> > > 3. This choice is left up to implementations
> > >
> > > I'll cover each option in more detail below, but first I want to
> clarify
> > > that we are not talking about "preview" features like encodings or
> > > forward-compatible changes like new logical types. Preview features
> will
> > > break readers that do not support them and only affect specific columns
> > > using the feature. For preview features, the expectation is that
> readers
> > > will attempt to read the file and will fail if they need to project a
> > > column that cannot be read.
> > >
> > > The choice of how to handle an unsupported format version primarily
> affects
> > > changes that add, remove, or modify the semantics of metadata fields.
> For
> > > example:
> > > - Changing `path_in_schema` from required to optional
> > > - Adding `offset` and `size` fields to a page header to relocate page
> data
> > > - Fixing stats written with the wrong sort order
> > >
> > > If we choose option 1, then a v3 reader that encounters a v4 file
> would be
> > > required to fail with an error that the file is too new to be read.
> With
> > > the current versioning scheme, the forward-compatibility within a
> version
> > > means that a v3 reader can read any v3 file and will produce the
> correct
> > > data. But this guarantee does not extend to any higher version.
> Without a
> > > guarantee that results are correct, the v3 reader must reject a v4
> file.
> > >
> > > If we choose option 2, then the v3 reader will attempt to read the v4
> file.
> > > In this case, Parquet will need to provide an additional guarantee that
> > > reading future formats will either fail or produce correct results. If
> I
> > > understand correctly, this option would use Thrift's semantics to cause
> > > failures. For example, path_in_schema would be required in Thrift and
> > > readers are expected to fail in decoding.
> > >
> > > Option 3 would mean that readers may choose to attempt to read, but do
> not
> > > have additional guarantees.
> > >
> > > Hopefully this makes the choices clear and we can have a discussion on
> this
> > > thread. Let's try to keep this thread on the topic of these options,
> either
> > > to clarify them or to debate the effects. We can discuss process or
> other
> > > topics on new threads.
> > >
> > > Ryan
> > >
> >
>

Re: [DISCUSS] How should readers behave with unsupported format versions?

[Steve Loughran <st...@gmail.com>]

I don't see how any option except #1, "fail on an unsupported version" is
realistic. As "pretending you support a version you don't understand" is
simply a pretence.

The fact that a parquet lib is required to fail fast on an unknown version
is a strong argument against moving to a new version unless you have a
really, really good reason. the entire work for having flatbuf alongside
the thrift buffers is to deliver performance improvements while retaining
compatibility. I would argue for trying to go that way as long as possible

At the same time as that "as long as possible" clause, I could imagine a
parquet release "2.0" which can generate and understand classic parquet +
with flatbuff & thrift, or parquet without the thrift and maybe with some
other optimisations. By default it' might generate parquet 1.x files, but
it would be aware of the later parquet version so *would* be forward
compatible.

On Wed, 16 Sept 2026 at 20:56, Fokko Driesprong <fo...@apache.org> wrote:

> Sorry, I had to catch up here. I'm in favor of 1.
>
> Proposed proper breaking changes, like the new footer, will always break
> existing readers. I don't think we can handle this gracefully with option
> ≥2.
>
> For breaking changes, we should fail because we cannot control how the
> client will behave. Also, we don't control all implementations out there,
> so we can't make assumptions about whether they use a field. The only thing
> we control is the specification.
>
> The path-in-schema sounds like a harmless change, but I think it opens the
> door to a slippery slope of further breaking changes that will cause
> existing readers to break more and more, and folks will lose trust in the
> project. Like Xiening is saying, #1 is the clean option. Instead of cutting
> corners, we could make the path-in-schema optimization part of the new
> footer. Bundle all of this in a new version.
>
> Kind regards,
> Fokko
>
> On 2026/09/15 18:26:22 Xiening Dai wrote:
> > Short answer - I vote for option 1.
> >
> > I think what you are talking about here for option 1 and 2 are regarding
> the Apache Parquet library, or Parquet official spec definition. We cannot
> prevent anyone from implementing their own Parquet reader (and in fact I
> know a few engines do have their own implementation). So #3 is already a
> given for anything outside of Apache Parquet.
> >
> > Now when comes to option 1 and 2. I believe #1 is clean. A full stop
> provides well defined behavior. #2 is going to complicate the reader
> implementation. It's hard to test,  hard to define the behavior, and hard
> for user to understand and predict the results. On the surface, the reader
> can potential consume more files, but the extra complexity and ambiguities
> makes it less appealing.
> >
> > In addition to this, I've seen a proposal to change Parquet magic number
> to achieve option #1. I don't support such approach. There are many
> tools/scripts rely on that particular magic number to detect file type,
> such as s3 object crawler, etc., and they don't even need to be a reader.
> And they will be broken if we change magic number. I think using the file
> version should be good enough. We should just release a reader patch to
> fail fast when version is higher than what's supported. And this patch
> should be released before we upgrade Parquet version and introduce any
> breaking change.
> >
> > On 2026/09/09 22:38:19 Ryan Blue wrote:
> > > Hi everyone,
> > >
> > > This is a new thread to discuss an issue raised in the "Finalizing the
> > > versioning proposal" thread. I suggested starting a new thread to
> focus on
> > > this question in the community sync and there was agreement so that we
> can
> > > get more attention to the issue.
> > >
> > > The question that we identified is: How should a reader behave when it
> > > encounters a file written with an unsupported format version?
> > >
> > > There are 3 main options:
> > > 1. A reader should fail because it does not support the version
> > > 2. A reader should attempt to read the file
> > > 3. This choice is left up to implementations
> > >
> > > I'll cover each option in more detail below, but first I want to
> clarify
> > > that we are not talking about "preview" features like encodings or
> > > forward-compatible changes like new logical types. Preview features
> will
> > > break readers that do not support them and only affect specific columns
> > > using the feature. For preview features, the expectation is that
> readers
> > > will attempt to read the file and will fail if they need to project a
> > > column that cannot be read.
> > >
> > > The choice of how to handle an unsupported format version primarily
> affects
> > > changes that add, remove, or modify the semantics of metadata fields.
> For
> > > example:
> > > - Changing `path_in_schema` from required to optional
> > > - Adding `offset` and `size` fields to a page header to relocate page
> data
> > > - Fixing stats written with the wrong sort order
> > >
> > > If we choose option 1, then a v3 reader that encounters a v4 file
> would be
> > > required to fail with an error that the file is too new to be read.
> With
> > > the current versioning scheme, the forward-compatibility within a
> version
> > > means that a v3 reader can read any v3 file and will produce the
> correct
> > > data. But this guarantee does not extend to any higher version.
> Without a
> > > guarantee that results are correct, the v3 reader must reject a v4
> file.
> > >
> > > If we choose option 2, then the v3 reader will attempt to read the v4
> file.
> > > In this case, Parquet will need to provide an additional guarantee that
> > > reading future formats will either fail or produce correct results. If
> I
> > > understand correctly, this option would use Thrift's semantics to cause
> > > failures. For example, path_in_schema would be required in Thrift and
> > > readers are expected to fail in decoding.
> > >
> > > Option 3 would mean that readers may choose to attempt to read, but do
> not
> > > have additional guarantees.
> > >
> > > Hopefully this makes the choices clear and we can have a discussion on
> this
> > > thread. Let's try to keep this thread on the topic of these options,
> either
> > > to clarify them or to debate the effects. We can discuss process or
> other
> > > topics on new threads.
> > >
> > > Ryan
> > >
> >
>

Re: [DISCUSS] How should readers behave with unsupported format versions?

[Fokko Driesprong <fo...@apache.org>]

Sorry, I had to catch up here. I'm in favor of 1.

Proposed proper breaking changes, like the new footer, will always break existing readers. I don't think we can handle this gracefully with option ≥2.

For breaking changes, we should fail because we cannot control how the client will behave. Also, we don't control all implementations out there, so we can't make assumptions about whether they use a field. The only thing we control is the specification.

The path-in-schema sounds like a harmless change, but I think it opens the door to a slippery slope of further breaking changes that will cause existing readers to break more and more, and folks will lose trust in the project. Like Xiening is saying, #1 is the clean option. Instead of cutting corners, we could make the path-in-schema optimization part of the new footer. Bundle all of this in a new version.

Kind regards,
Fokko

On 2026/09/15 18:26:22 Xiening Dai wrote:
> Short answer - I vote for option 1.
> 
> I think what you are talking about here for option 1 and 2 are regarding the Apache Parquet library, or Parquet official spec definition. We cannot prevent anyone from implementing their own Parquet reader (and in fact I know a few engines do have their own implementation). So #3 is already a given for anything outside of Apache Parquet.
> 
> Now when comes to option 1 and 2. I believe #1 is clean. A full stop provides well defined behavior. #2 is going to complicate the reader implementation. It's hard to test,  hard to define the behavior, and hard for user to understand and predict the results. On the surface, the reader can potential consume more files, but the extra complexity and ambiguities makes it less appealing. 
> 
> In addition to this, I've seen a proposal to change Parquet magic number to achieve option #1. I don't support such approach. There are many tools/scripts rely on that particular magic number to detect file type, such as s3 object crawler, etc., and they don't even need to be a reader. And they will be broken if we change magic number. I think using the file version should be good enough. We should just release a reader patch to fail fast when version is higher than what's supported. And this patch should be released before we upgrade Parquet version and introduce any breaking change.
> 
> On 2026/09/09 22:38:19 Ryan Blue wrote:
> > Hi everyone,
> > 
> > This is a new thread to discuss an issue raised in the "Finalizing the
> > versioning proposal" thread. I suggested starting a new thread to focus on
> > this question in the community sync and there was agreement so that we can
> > get more attention to the issue.
> > 
> > The question that we identified is: How should a reader behave when it
> > encounters a file written with an unsupported format version?
> > 
> > There are 3 main options:
> > 1. A reader should fail because it does not support the version
> > 2. A reader should attempt to read the file
> > 3. This choice is left up to implementations
> > 
> > I'll cover each option in more detail below, but first I want to clarify
> > that we are not talking about "preview" features like encodings or
> > forward-compatible changes like new logical types. Preview features will
> > break readers that do not support them and only affect specific columns
> > using the feature. For preview features, the expectation is that readers
> > will attempt to read the file and will fail if they need to project a
> > column that cannot be read.
> > 
> > The choice of how to handle an unsupported format version primarily affects
> > changes that add, remove, or modify the semantics of metadata fields. For
> > example:
> > - Changing `path_in_schema` from required to optional
> > - Adding `offset` and `size` fields to a page header to relocate page data
> > - Fixing stats written with the wrong sort order
> > 
> > If we choose option 1, then a v3 reader that encounters a v4 file would be
> > required to fail with an error that the file is too new to be read. With
> > the current versioning scheme, the forward-compatibility within a version
> > means that a v3 reader can read any v3 file and will produce the correct
> > data. But this guarantee does not extend to any higher version. Without a
> > guarantee that results are correct, the v3 reader must reject a v4 file.
> > 
> > If we choose option 2, then the v3 reader will attempt to read the v4 file.
> > In this case, Parquet will need to provide an additional guarantee that
> > reading future formats will either fail or produce correct results. If I
> > understand correctly, this option would use Thrift's semantics to cause
> > failures. For example, path_in_schema would be required in Thrift and
> > readers are expected to fail in decoding.
> > 
> > Option 3 would mean that readers may choose to attempt to read, but do not
> > have additional guarantees.
> > 
> > Hopefully this makes the choices clear and we can have a discussion on this
> > thread. Let's try to keep this thread on the topic of these options, either
> > to clarify them or to debate the effects. We can discuss process or other
> > topics on new threads.
> > 
> > Ryan
> > 
> 

Re: [DISCUSS] How should readers behave with unsupported format versions?

[Xiening Dai <xn...@apache.org>]

Short answer - I vote for option 1.

I think what you are talking about here for option 1 and 2 are regarding the Apache Parquet library, or Parquet official spec definition. We cannot prevent anyone from implementing their own Parquet reader (and in fact I know a few engines do have their own implementation). So #3 is already a given for anything outside of Apache Parquet.

Now when comes to option 1 and 2. I believe #1 is clean. A full stop provides well defined behavior. #2 is going to complicate the reader implementation. It's hard to test,  hard to define the behavior, and hard for user to understand and predict the results. On the surface, the reader can potential consume more files, but the extra complexity and ambiguities makes it less appealing. 

In addition to this, I've seen a proposal to change Parquet magic number to achieve option #1. I don't support such approach. There are many tools/scripts rely on that particular magic number to detect file type, such as s3 object crawler, etc., and they don't even need to be a reader. And they will be broken if we change magic number. I think using the file version should be good enough. We should just release a reader patch to fail fast when version is higher than what's supported. And this patch should be released before we upgrade Parquet version and introduce any breaking change.

On 2026/09/09 22:38:19 Ryan Blue wrote:
> Hi everyone,
> 
> This is a new thread to discuss an issue raised in the "Finalizing the
> versioning proposal" thread. I suggested starting a new thread to focus on
> this question in the community sync and there was agreement so that we can
> get more attention to the issue.
> 
> The question that we identified is: How should a reader behave when it
> encounters a file written with an unsupported format version?
> 
> There are 3 main options:
> 1. A reader should fail because it does not support the version
> 2. A reader should attempt to read the file
> 3. This choice is left up to implementations
> 
> I'll cover each option in more detail below, but first I want to clarify
> that we are not talking about "preview" features like encodings or
> forward-compatible changes like new logical types. Preview features will
> break readers that do not support them and only affect specific columns
> using the feature. For preview features, the expectation is that readers
> will attempt to read the file and will fail if they need to project a
> column that cannot be read.
> 
> The choice of how to handle an unsupported format version primarily affects
> changes that add, remove, or modify the semantics of metadata fields. For
> example:
> - Changing `path_in_schema` from required to optional
> - Adding `offset` and `size` fields to a page header to relocate page data
> - Fixing stats written with the wrong sort order
> 
> If we choose option 1, then a v3 reader that encounters a v4 file would be
> required to fail with an error that the file is too new to be read. With
> the current versioning scheme, the forward-compatibility within a version
> means that a v3 reader can read any v3 file and will produce the correct
> data. But this guarantee does not extend to any higher version. Without a
> guarantee that results are correct, the v3 reader must reject a v4 file.
> 
> If we choose option 2, then the v3 reader will attempt to read the v4 file.
> In this case, Parquet will need to provide an additional guarantee that
> reading future formats will either fail or produce correct results. If I
> understand correctly, this option would use Thrift's semantics to cause
> failures. For example, path_in_schema would be required in Thrift and
> readers are expected to fail in decoding.
> 
> Option 3 would mean that readers may choose to attempt to read, but do not
> have additional guarantees.
> 
> Hopefully this makes the choices clear and we can have a discussion on this
> thread. Let's try to keep this thread on the topic of these options, either
> to clarify them or to debate the effects. We can discuss process or other
> topics on new threads.
> 
> Ryan
>