You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by Ed Seidl <et...@LIVE.COM> on 2026/06/01 14:07:18 UTC
[VOTE] GH-563: Make ColumnMetaData.path_in_schema optional
I would like to propose a vote on adopting the format change described in
GH-563: Make ColumnMetaData.path_in_schema optional.
This was proposed on the mailing list [1] and in GitHub issue #563 [2].
The proposed format specification changes are available in the PR:
https://github.com/apache/parquet-format/issues/563
To verify this design's compatibility and correctness, three PoC implementations
were developed:
1. Java: https://github.com/apache/parquet-java/pull/3470
2. C++: https://github.com/apache/arrow/pull/49707
3. Rust: https://github.com/apache/arrow-rs/pull/9678
All three PoCs have been verified against a test file produced by the Rust PoC:
https://github.com/apache/parquet-testing/pull/108
The vote will be open for at least 72 hours.
[ ] +1 Approve the proposed format change
[ ] +0 No opinion
[ ] -1 Do not approve (please provide specific reasons)
Thanks,
Ed Seidl
[1] https://lists.apache.org/thread/900503q07v95vyh6fk3qfn7ynb4w6yn2
[2] https://github.com/apache/parquet-format/issues/563
Re: [VOTE] GH-563: Make ColumnMetaData.path_in_schema optional
Posted by Ryan Blue <bl...@apache.org>.
Thanks, Ed.
The discussion about versioning is making progress so I think we may be
close to having a way to introduce these forward-incompatible changes
safely. To me, that means requiring in the thrift spec that the field is
present for v1/v2 files and only allowing it to be null/omitted from the
next forward-incompatible change boundary.
On Thu, Jun 4, 2026 at 8:34 AM Ed Seidl <et...@apache.org> wrote:
> I'm going to close this vote as failed due to Ryan's objection. I'm leaving
> the issue [1] and PR [2] open pending a resolution of the versioning
> discussion [3].
>
> Thanks all for participating.
> Ed
>
> [1] https://github.com/apache/parquet-format/issues/563
> [2] https://github.com/apache/parquet-format/pull/564
> [3] https://lists.apache.org/thread/st8l40z5n4cx5c22rcog50ws4pkzdc3s
>
> On 2026/06/03 15:33:17 Ryan Blue wrote:
> > Resending this because it didn't go through from my other account.
> >
> > -1
> >
> > I think there's a big difference between other features that we want to
> > turn on using write-side flags and this one. This isn't introducing a
> > beneficial feature like encryption or a new encoding. It is just omitting
> > metadata as a minor cleanup. I don't think it makes sense to allow
> > switching this on when we know that it breaks older readers.
> >
> > This exposes the big gap in how we currently think about compatibility.
> > While it makes sense to introduce a few beneficial features with
> > write-side flags, introducing flags for lots of minor cases like this
> will
> > cause fragmentation and needless incompatibilities through the Parquet
> > ecosystem. I think we need to come up with a way to release these
> > incompatible changes (and update defaults for the feature flags) as a
> group
> > so that downstream users can reason about compatibility.
> >
> > I'd change my vote to +1 if we had a plan for this. For instance, if we
> > were to say that this is a part of the next group of forward-incompatible
> > changes and had a way to block it being used until then, I would be +1.
> >
> > On Tue, Jun 2, 2026 at 8:48 PM Micah Kornfield <em...@gmail.com>
> > wrote:
> >
> > > +1
> > >
> > > But practically we don't really intend to wait until 2028 and we're
> just
> > > > going to start doing this immediately.
> > >
> > >
> > > It looks like all the reference implementations are keeping this as
> default
> > > to true. I understand some people will use the knob the turn it off
> but I
> > > expect that people that do so will understand the blast radius of the
> > > change.
> > >
> > > -Micah
> > >
> > > On Tue, Jun 2, 2026 at 7:01 PM Gang Wu <us...@gmail.com> wrote:
> > >
> > > > +1
> > > >
> > > > While I understand the concerns about playing loose with the spec
> and the
> > > > practical timeline for ecosystem adoption, I believe Parquet must be
> bold
> > > > in fixing its own issues, especially when facing internal challenges
> and
> > > > competition from other formats and scenarios.
> > > >
> > > > This specific change is benign and targets limited use cases.
> > > Furthermore,
> > > > since the PMC will rigorously review every spec change case by case,
> the
> > > > overall impact is controllable.
> > > >
> > > > Best,
> > > > Gang
> > > >
> > > > On Tue, Jun 2, 2026 at 8:49 PM Andrew Lamb <an...@gmail.com>
> > > wrote:
> > > >
> > > > > +1
> > > > >
> > > > > While I share the concern that it is non trivial to understand the
> > > > rollout
> > > > > status of various features across the Parquet Ecosystem, I don't
> think
> > > > this
> > > > > particular change is any better/worse than existing features such
> as
> > > > > modular encryption, so gating it on getting such a consensus seems
> > > unfair
> > > > > to me
> > > > >
> > > > > I personally think the best thing we can do to help the ecosystem
> adopt
> > > > > features is continuing to refine our existing matrix of file format
> > > > > implementation status [1]
> > > > >
> > > > > Andrew
> > > > >
> > > > > [1]:
> > > > >
> > > > >
> > > >
> > >
> https://parquet.apache.org/docs/file-format/implementationstatus/#read-support-by-year
> > > > >
> > > > >
> > > > > On Mon, Jun 1, 2026 at 6:03 PM Daniel Weeks <dw...@apache.org>
> wrote:
> > > > >
> > > > > > -0 (though on the fence here).
> > > > > >
> > > > > > I'm a little concerned that we're starting to play very loose
> with
> > > spec
> > > > > > with a change like this. Some of the justification seems to be
> that
> > > > it's
> > > > > > already supported in some projects and some readers don't break
> by
> > > > making
> > > > > > this change.
> > > > > >
> > > > > > I see we've added the comment:
> > > > > >
> > > > > > > Writers are encouraged to make the writing of
> > > > > > > * this field optional, but for maximal compatibility should
> > > > default
> > > > > to
> > > > > > > * writing the field until at least September 2028.
> > > > > >
> > > > > >
> > > > > > But practically we don't really intend to wait until 2028 and
> we're
> > > > just
> > > > > > going to start doing this immediately.
> > > > > >
> > > > > > -Dan
> > > > > >
> > > > > > On Mon, Jun 1, 2026 at 7:12 AM Ed Seidl <et...@apache.org>
> wrote:
> > > > > >
> > > > > > > Correction: the link to the PR is
> > > > > > > https://github.com/apache/parquet-format/pull/564
> > > > > > >
> > > > > > > Ed
> > > > > > >
> > > > > > > On 2026/06/01 14:07:18 Ed Seidl wrote:
> > > > > > > > I would like to propose a vote on adopting the format change
> > > > > described
> > > > > > in
> > > > > > > > GH-563: Make ColumnMetaData.path_in_schema optional.
> > > > > > > >
> > > > > > > > This was proposed on the mailing list [1] and in GitHub issue
> > > #563
> > > > > [2].
> > > > > > > >
> > > > > > > > The proposed format specification changes are available in
> the
> > > PR:
> > > > > > > > https://github.com/apache/parquet-format/issues/563
> > > > > > > >
> > > > > > > > To verify this design's compatibility and correctness, three
> PoC
> > > > > > > implementations
> > > > > > > > were developed:
> > > > > > > > 1. Java: https://github.com/apache/parquet-java/pull/3470
> > > > > > > > 2. C++: https://github.com/apache/arrow/pull/49707
> > > > > > > > 3. Rust: https://github.com/apache/arrow-rs/pull/9678
> > > > > > > >
> > > > > > > > All three PoCs have been verified against a test file
> produced by
> > > > the
> > > > > > > Rust PoC:
> > > > > > > > https://github.com/apache/parquet-testing/pull/108
> > > > > > > >
> > > > > > > > The vote will be open for at least 72 hours.
> > > > > > > >
> > > > > > > > [ ] +1 Approve the proposed format change
> > > > > > > > [ ] +0 No opinion
> > > > > > > > [ ] -1 Do not approve (please provide specific reasons)
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Ed Seidl
> > > > > > > >
> > > > > > > > [1]
> > > > https://lists.apache.org/thread/900503q07v95vyh6fk3qfn7ynb4w6yn2
> > > > > > > > [2] https://github.com/apache/parquet-format/issues/563
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [VOTE] GH-563: Make ColumnMetaData.path_in_schema optional
Posted by Ed Seidl <et...@apache.org>.
I'm going to close this vote as failed due to Ryan's objection. I'm leaving
the issue [1] and PR [2] open pending a resolution of the versioning discussion [3].
Thanks all for participating.
Ed
[1] https://github.com/apache/parquet-format/issues/563
[2] https://github.com/apache/parquet-format/pull/564
[3] https://lists.apache.org/thread/st8l40z5n4cx5c22rcog50ws4pkzdc3s
On 2026/06/03 15:33:17 Ryan Blue wrote:
> Resending this because it didn't go through from my other account.
>
> -1
>
> I think there's a big difference between other features that we want to
> turn on using write-side flags and this one. This isn't introducing a
> beneficial feature like encryption or a new encoding. It is just omitting
> metadata as a minor cleanup. I don't think it makes sense to allow
> switching this on when we know that it breaks older readers.
>
> This exposes the big gap in how we currently think about compatibility.
> While it makes sense to introduce a few beneficial features with
> write-side flags, introducing flags for lots of minor cases like this will
> cause fragmentation and needless incompatibilities through the Parquet
> ecosystem. I think we need to come up with a way to release these
> incompatible changes (and update defaults for the feature flags) as a group
> so that downstream users can reason about compatibility.
>
> I'd change my vote to +1 if we had a plan for this. For instance, if we
> were to say that this is a part of the next group of forward-incompatible
> changes and had a way to block it being used until then, I would be +1.
>
> On Tue, Jun 2, 2026 at 8:48 PM Micah Kornfield <em...@gmail.com>
> wrote:
>
> > +1
> >
> > But practically we don't really intend to wait until 2028 and we're just
> > > going to start doing this immediately.
> >
> >
> > It looks like all the reference implementations are keeping this as default
> > to true. I understand some people will use the knob the turn it off but I
> > expect that people that do so will understand the blast radius of the
> > change.
> >
> > -Micah
> >
> > On Tue, Jun 2, 2026 at 7:01 PM Gang Wu <us...@gmail.com> wrote:
> >
> > > +1
> > >
> > > While I understand the concerns about playing loose with the spec and the
> > > practical timeline for ecosystem adoption, I believe Parquet must be bold
> > > in fixing its own issues, especially when facing internal challenges and
> > > competition from other formats and scenarios.
> > >
> > > This specific change is benign and targets limited use cases.
> > Furthermore,
> > > since the PMC will rigorously review every spec change case by case, the
> > > overall impact is controllable.
> > >
> > > Best,
> > > Gang
> > >
> > > On Tue, Jun 2, 2026 at 8:49 PM Andrew Lamb <an...@gmail.com>
> > wrote:
> > >
> > > > +1
> > > >
> > > > While I share the concern that it is non trivial to understand the
> > > rollout
> > > > status of various features across the Parquet Ecosystem, I don't think
> > > this
> > > > particular change is any better/worse than existing features such as
> > > > modular encryption, so gating it on getting such a consensus seems
> > unfair
> > > > to me
> > > >
> > > > I personally think the best thing we can do to help the ecosystem adopt
> > > > features is continuing to refine our existing matrix of file format
> > > > implementation status [1]
> > > >
> > > > Andrew
> > > >
> > > > [1]:
> > > >
> > > >
> > >
> > https://parquet.apache.org/docs/file-format/implementationstatus/#read-support-by-year
> > > >
> > > >
> > > > On Mon, Jun 1, 2026 at 6:03 PM Daniel Weeks <dw...@apache.org> wrote:
> > > >
> > > > > -0 (though on the fence here).
> > > > >
> > > > > I'm a little concerned that we're starting to play very loose with
> > spec
> > > > > with a change like this. Some of the justification seems to be that
> > > it's
> > > > > already supported in some projects and some readers don't break by
> > > making
> > > > > this change.
> > > > >
> > > > > I see we've added the comment:
> > > > >
> > > > > > Writers are encouraged to make the writing of
> > > > > > * this field optional, but for maximal compatibility should
> > > default
> > > > to
> > > > > > * writing the field until at least September 2028.
> > > > >
> > > > >
> > > > > But practically we don't really intend to wait until 2028 and we're
> > > just
> > > > > going to start doing this immediately.
> > > > >
> > > > > -Dan
> > > > >
> > > > > On Mon, Jun 1, 2026 at 7:12 AM Ed Seidl <et...@apache.org> wrote:
> > > > >
> > > > > > Correction: the link to the PR is
> > > > > > https://github.com/apache/parquet-format/pull/564
> > > > > >
> > > > > > Ed
> > > > > >
> > > > > > On 2026/06/01 14:07:18 Ed Seidl wrote:
> > > > > > > I would like to propose a vote on adopting the format change
> > > > described
> > > > > in
> > > > > > > GH-563: Make ColumnMetaData.path_in_schema optional.
> > > > > > >
> > > > > > > This was proposed on the mailing list [1] and in GitHub issue
> > #563
> > > > [2].
> > > > > > >
> > > > > > > The proposed format specification changes are available in the
> > PR:
> > > > > > > https://github.com/apache/parquet-format/issues/563
> > > > > > >
> > > > > > > To verify this design's compatibility and correctness, three PoC
> > > > > > implementations
> > > > > > > were developed:
> > > > > > > 1. Java: https://github.com/apache/parquet-java/pull/3470
> > > > > > > 2. C++: https://github.com/apache/arrow/pull/49707
> > > > > > > 3. Rust: https://github.com/apache/arrow-rs/pull/9678
> > > > > > >
> > > > > > > All three PoCs have been verified against a test file produced by
> > > the
> > > > > > Rust PoC:
> > > > > > > https://github.com/apache/parquet-testing/pull/108
> > > > > > >
> > > > > > > The vote will be open for at least 72 hours.
> > > > > > >
> > > > > > > [ ] +1 Approve the proposed format change
> > > > > > > [ ] +0 No opinion
> > > > > > > [ ] -1 Do not approve (please provide specific reasons)
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Ed Seidl
> > > > > > >
> > > > > > > [1]
> > > https://lists.apache.org/thread/900503q07v95vyh6fk3qfn7ynb4w6yn2
> > > > > > > [2] https://github.com/apache/parquet-format/issues/563
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [VOTE] GH-563: Make ColumnMetaData.path_in_schema optional
Posted by Ryan Blue <bl...@apache.org>.
Resending this because it didn't go through from my other account.
-1
I think there's a big difference between other features that we want to
turn on using write-side flags and this one. This isn't introducing a
beneficial feature like encryption or a new encoding. It is just omitting
metadata as a minor cleanup. I don't think it makes sense to allow
switching this on when we know that it breaks older readers.
This exposes the big gap in how we currently think about compatibility.
While it makes sense to introduce a few beneficial features with
write-side flags, introducing flags for lots of minor cases like this will
cause fragmentation and needless incompatibilities through the Parquet
ecosystem. I think we need to come up with a way to release these
incompatible changes (and update defaults for the feature flags) as a group
so that downstream users can reason about compatibility.
I'd change my vote to +1 if we had a plan for this. For instance, if we
were to say that this is a part of the next group of forward-incompatible
changes and had a way to block it being used until then, I would be +1.
On Tue, Jun 2, 2026 at 8:48 PM Micah Kornfield <em...@gmail.com>
wrote:
> +1
>
> But practically we don't really intend to wait until 2028 and we're just
> > going to start doing this immediately.
>
>
> It looks like all the reference implementations are keeping this as default
> to true. I understand some people will use the knob the turn it off but I
> expect that people that do so will understand the blast radius of the
> change.
>
> -Micah
>
> On Tue, Jun 2, 2026 at 7:01 PM Gang Wu <us...@gmail.com> wrote:
>
> > +1
> >
> > While I understand the concerns about playing loose with the spec and the
> > practical timeline for ecosystem adoption, I believe Parquet must be bold
> > in fixing its own issues, especially when facing internal challenges and
> > competition from other formats and scenarios.
> >
> > This specific change is benign and targets limited use cases.
> Furthermore,
> > since the PMC will rigorously review every spec change case by case, the
> > overall impact is controllable.
> >
> > Best,
> > Gang
> >
> > On Tue, Jun 2, 2026 at 8:49 PM Andrew Lamb <an...@gmail.com>
> wrote:
> >
> > > +1
> > >
> > > While I share the concern that it is non trivial to understand the
> > rollout
> > > status of various features across the Parquet Ecosystem, I don't think
> > this
> > > particular change is any better/worse than existing features such as
> > > modular encryption, so gating it on getting such a consensus seems
> unfair
> > > to me
> > >
> > > I personally think the best thing we can do to help the ecosystem adopt
> > > features is continuing to refine our existing matrix of file format
> > > implementation status [1]
> > >
> > > Andrew
> > >
> > > [1]:
> > >
> > >
> >
> https://parquet.apache.org/docs/file-format/implementationstatus/#read-support-by-year
> > >
> > >
> > > On Mon, Jun 1, 2026 at 6:03 PM Daniel Weeks <dw...@apache.org> wrote:
> > >
> > > > -0 (though on the fence here).
> > > >
> > > > I'm a little concerned that we're starting to play very loose with
> spec
> > > > with a change like this. Some of the justification seems to be that
> > it's
> > > > already supported in some projects and some readers don't break by
> > making
> > > > this change.
> > > >
> > > > I see we've added the comment:
> > > >
> > > > > Writers are encouraged to make the writing of
> > > > > * this field optional, but for maximal compatibility should
> > default
> > > to
> > > > > * writing the field until at least September 2028.
> > > >
> > > >
> > > > But practically we don't really intend to wait until 2028 and we're
> > just
> > > > going to start doing this immediately.
> > > >
> > > > -Dan
> > > >
> > > > On Mon, Jun 1, 2026 at 7:12 AM Ed Seidl <et...@apache.org> wrote:
> > > >
> > > > > Correction: the link to the PR is
> > > > > https://github.com/apache/parquet-format/pull/564
> > > > >
> > > > > Ed
> > > > >
> > > > > On 2026/06/01 14:07:18 Ed Seidl wrote:
> > > > > > I would like to propose a vote on adopting the format change
> > > described
> > > > in
> > > > > > GH-563: Make ColumnMetaData.path_in_schema optional.
> > > > > >
> > > > > > This was proposed on the mailing list [1] and in GitHub issue
> #563
> > > [2].
> > > > > >
> > > > > > The proposed format specification changes are available in the
> PR:
> > > > > > https://github.com/apache/parquet-format/issues/563
> > > > > >
> > > > > > To verify this design's compatibility and correctness, three PoC
> > > > > implementations
> > > > > > were developed:
> > > > > > 1. Java: https://github.com/apache/parquet-java/pull/3470
> > > > > > 2. C++: https://github.com/apache/arrow/pull/49707
> > > > > > 3. Rust: https://github.com/apache/arrow-rs/pull/9678
> > > > > >
> > > > > > All three PoCs have been verified against a test file produced by
> > the
> > > > > Rust PoC:
> > > > > > https://github.com/apache/parquet-testing/pull/108
> > > > > >
> > > > > > The vote will be open for at least 72 hours.
> > > > > >
> > > > > > [ ] +1 Approve the proposed format change
> > > > > > [ ] +0 No opinion
> > > > > > [ ] -1 Do not approve (please provide specific reasons)
> > > > > >
> > > > > > Thanks,
> > > > > > Ed Seidl
> > > > > >
> > > > > > [1]
> > https://lists.apache.org/thread/900503q07v95vyh6fk3qfn7ynb4w6yn2
> > > > > > [2] https://github.com/apache/parquet-format/issues/563
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [VOTE] GH-563: Make ColumnMetaData.path_in_schema optional
Posted by Micah Kornfield <em...@gmail.com>.
+1
But practically we don't really intend to wait until 2028 and we're just
> going to start doing this immediately.
It looks like all the reference implementations are keeping this as default
to true. I understand some people will use the knob the turn it off but I
expect that people that do so will understand the blast radius of the
change.
-Micah
On Tue, Jun 2, 2026 at 7:01 PM Gang Wu <us...@gmail.com> wrote:
> +1
>
> While I understand the concerns about playing loose with the spec and the
> practical timeline for ecosystem adoption, I believe Parquet must be bold
> in fixing its own issues, especially when facing internal challenges and
> competition from other formats and scenarios.
>
> This specific change is benign and targets limited use cases. Furthermore,
> since the PMC will rigorously review every spec change case by case, the
> overall impact is controllable.
>
> Best,
> Gang
>
> On Tue, Jun 2, 2026 at 8:49 PM Andrew Lamb <an...@gmail.com> wrote:
>
> > +1
> >
> > While I share the concern that it is non trivial to understand the
> rollout
> > status of various features across the Parquet Ecosystem, I don't think
> this
> > particular change is any better/worse than existing features such as
> > modular encryption, so gating it on getting such a consensus seems unfair
> > to me
> >
> > I personally think the best thing we can do to help the ecosystem adopt
> > features is continuing to refine our existing matrix of file format
> > implementation status [1]
> >
> > Andrew
> >
> > [1]:
> >
> >
> https://parquet.apache.org/docs/file-format/implementationstatus/#read-support-by-year
> >
> >
> > On Mon, Jun 1, 2026 at 6:03 PM Daniel Weeks <dw...@apache.org> wrote:
> >
> > > -0 (though on the fence here).
> > >
> > > I'm a little concerned that we're starting to play very loose with spec
> > > with a change like this. Some of the justification seems to be that
> it's
> > > already supported in some projects and some readers don't break by
> making
> > > this change.
> > >
> > > I see we've added the comment:
> > >
> > > > Writers are encouraged to make the writing of
> > > > * this field optional, but for maximal compatibility should
> default
> > to
> > > > * writing the field until at least September 2028.
> > >
> > >
> > > But practically we don't really intend to wait until 2028 and we're
> just
> > > going to start doing this immediately.
> > >
> > > -Dan
> > >
> > > On Mon, Jun 1, 2026 at 7:12 AM Ed Seidl <et...@apache.org> wrote:
> > >
> > > > Correction: the link to the PR is
> > > > https://github.com/apache/parquet-format/pull/564
> > > >
> > > > Ed
> > > >
> > > > On 2026/06/01 14:07:18 Ed Seidl wrote:
> > > > > I would like to propose a vote on adopting the format change
> > described
> > > in
> > > > > GH-563: Make ColumnMetaData.path_in_schema optional.
> > > > >
> > > > > This was proposed on the mailing list [1] and in GitHub issue #563
> > [2].
> > > > >
> > > > > The proposed format specification changes are available in the PR:
> > > > > https://github.com/apache/parquet-format/issues/563
> > > > >
> > > > > To verify this design's compatibility and correctness, three PoC
> > > > implementations
> > > > > were developed:
> > > > > 1. Java: https://github.com/apache/parquet-java/pull/3470
> > > > > 2. C++: https://github.com/apache/arrow/pull/49707
> > > > > 3. Rust: https://github.com/apache/arrow-rs/pull/9678
> > > > >
> > > > > All three PoCs have been verified against a test file produced by
> the
> > > > Rust PoC:
> > > > > https://github.com/apache/parquet-testing/pull/108
> > > > >
> > > > > The vote will be open for at least 72 hours.
> > > > >
> > > > > [ ] +1 Approve the proposed format change
> > > > > [ ] +0 No opinion
> > > > > [ ] -1 Do not approve (please provide specific reasons)
> > > > >
> > > > > Thanks,
> > > > > Ed Seidl
> > > > >
> > > > > [1]
> https://lists.apache.org/thread/900503q07v95vyh6fk3qfn7ynb4w6yn2
> > > > > [2] https://github.com/apache/parquet-format/issues/563
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: [VOTE] GH-563: Make ColumnMetaData.path_in_schema optional
Posted by Gang Wu <us...@gmail.com>.
+1
While I understand the concerns about playing loose with the spec and the
practical timeline for ecosystem adoption, I believe Parquet must be bold
in fixing its own issues, especially when facing internal challenges and
competition from other formats and scenarios.
This specific change is benign and targets limited use cases. Furthermore,
since the PMC will rigorously review every spec change case by case, the
overall impact is controllable.
Best,
Gang
On Tue, Jun 2, 2026 at 8:49 PM Andrew Lamb <an...@gmail.com> wrote:
> +1
>
> While I share the concern that it is non trivial to understand the rollout
> status of various features across the Parquet Ecosystem, I don't think this
> particular change is any better/worse than existing features such as
> modular encryption, so gating it on getting such a consensus seems unfair
> to me
>
> I personally think the best thing we can do to help the ecosystem adopt
> features is continuing to refine our existing matrix of file format
> implementation status [1]
>
> Andrew
>
> [1]:
>
> https://parquet.apache.org/docs/file-format/implementationstatus/#read-support-by-year
>
>
> On Mon, Jun 1, 2026 at 6:03 PM Daniel Weeks <dw...@apache.org> wrote:
>
> > -0 (though on the fence here).
> >
> > I'm a little concerned that we're starting to play very loose with spec
> > with a change like this. Some of the justification seems to be that it's
> > already supported in some projects and some readers don't break by making
> > this change.
> >
> > I see we've added the comment:
> >
> > > Writers are encouraged to make the writing of
> > > * this field optional, but for maximal compatibility should default
> to
> > > * writing the field until at least September 2028.
> >
> >
> > But practically we don't really intend to wait until 2028 and we're just
> > going to start doing this immediately.
> >
> > -Dan
> >
> > On Mon, Jun 1, 2026 at 7:12 AM Ed Seidl <et...@apache.org> wrote:
> >
> > > Correction: the link to the PR is
> > > https://github.com/apache/parquet-format/pull/564
> > >
> > > Ed
> > >
> > > On 2026/06/01 14:07:18 Ed Seidl wrote:
> > > > I would like to propose a vote on adopting the format change
> described
> > in
> > > > GH-563: Make ColumnMetaData.path_in_schema optional.
> > > >
> > > > This was proposed on the mailing list [1] and in GitHub issue #563
> [2].
> > > >
> > > > The proposed format specification changes are available in the PR:
> > > > https://github.com/apache/parquet-format/issues/563
> > > >
> > > > To verify this design's compatibility and correctness, three PoC
> > > implementations
> > > > were developed:
> > > > 1. Java: https://github.com/apache/parquet-java/pull/3470
> > > > 2. C++: https://github.com/apache/arrow/pull/49707
> > > > 3. Rust: https://github.com/apache/arrow-rs/pull/9678
> > > >
> > > > All three PoCs have been verified against a test file produced by the
> > > Rust PoC:
> > > > https://github.com/apache/parquet-testing/pull/108
> > > >
> > > > The vote will be open for at least 72 hours.
> > > >
> > > > [ ] +1 Approve the proposed format change
> > > > [ ] +0 No opinion
> > > > [ ] -1 Do not approve (please provide specific reasons)
> > > >
> > > > Thanks,
> > > > Ed Seidl
> > > >
> > > > [1] https://lists.apache.org/thread/900503q07v95vyh6fk3qfn7ynb4w6yn2
> > > > [2] https://github.com/apache/parquet-format/issues/563
> > > >
> > > >
> > >
> >
>
Re: [VOTE] GH-563: Make ColumnMetaData.path_in_schema optional
Posted by Andrew Lamb <an...@gmail.com>.
+1
While I share the concern that it is non trivial to understand the rollout
status of various features across the Parquet Ecosystem, I don't think this
particular change is any better/worse than existing features such as
modular encryption, so gating it on getting such a consensus seems unfair
to me
I personally think the best thing we can do to help the ecosystem adopt
features is continuing to refine our existing matrix of file format
implementation status [1]
Andrew
[1]:
https://parquet.apache.org/docs/file-format/implementationstatus/#read-support-by-year
On Mon, Jun 1, 2026 at 6:03 PM Daniel Weeks <dw...@apache.org> wrote:
> -0 (though on the fence here).
>
> I'm a little concerned that we're starting to play very loose with spec
> with a change like this. Some of the justification seems to be that it's
> already supported in some projects and some readers don't break by making
> this change.
>
> I see we've added the comment:
>
> > Writers are encouraged to make the writing of
> > * this field optional, but for maximal compatibility should default to
> > * writing the field until at least September 2028.
>
>
> But practically we don't really intend to wait until 2028 and we're just
> going to start doing this immediately.
>
> -Dan
>
> On Mon, Jun 1, 2026 at 7:12 AM Ed Seidl <et...@apache.org> wrote:
>
> > Correction: the link to the PR is
> > https://github.com/apache/parquet-format/pull/564
> >
> > Ed
> >
> > On 2026/06/01 14:07:18 Ed Seidl wrote:
> > > I would like to propose a vote on adopting the format change described
> in
> > > GH-563: Make ColumnMetaData.path_in_schema optional.
> > >
> > > This was proposed on the mailing list [1] and in GitHub issue #563 [2].
> > >
> > > The proposed format specification changes are available in the PR:
> > > https://github.com/apache/parquet-format/issues/563
> > >
> > > To verify this design's compatibility and correctness, three PoC
> > implementations
> > > were developed:
> > > 1. Java: https://github.com/apache/parquet-java/pull/3470
> > > 2. C++: https://github.com/apache/arrow/pull/49707
> > > 3. Rust: https://github.com/apache/arrow-rs/pull/9678
> > >
> > > All three PoCs have been verified against a test file produced by the
> > Rust PoC:
> > > https://github.com/apache/parquet-testing/pull/108
> > >
> > > The vote will be open for at least 72 hours.
> > >
> > > [ ] +1 Approve the proposed format change
> > > [ ] +0 No opinion
> > > [ ] -1 Do not approve (please provide specific reasons)
> > >
> > > Thanks,
> > > Ed Seidl
> > >
> > > [1] https://lists.apache.org/thread/900503q07v95vyh6fk3qfn7ynb4w6yn2
> > > [2] https://github.com/apache/parquet-format/issues/563
> > >
> > >
> >
>
Re: [VOTE] GH-563: Make ColumnMetaData.path_in_schema optional
Posted by Daniel Weeks <dw...@apache.org>.
-0 (though on the fence here).
I'm a little concerned that we're starting to play very loose with spec
with a change like this. Some of the justification seems to be that it's
already supported in some projects and some readers don't break by making
this change.
I see we've added the comment:
> Writers are encouraged to make the writing of
> * this field optional, but for maximal compatibility should default to
> * writing the field until at least September 2028.
But practically we don't really intend to wait until 2028 and we're just
going to start doing this immediately.
-Dan
On Mon, Jun 1, 2026 at 7:12 AM Ed Seidl <et...@apache.org> wrote:
> Correction: the link to the PR is
> https://github.com/apache/parquet-format/pull/564
>
> Ed
>
> On 2026/06/01 14:07:18 Ed Seidl wrote:
> > I would like to propose a vote on adopting the format change described in
> > GH-563: Make ColumnMetaData.path_in_schema optional.
> >
> > This was proposed on the mailing list [1] and in GitHub issue #563 [2].
> >
> > The proposed format specification changes are available in the PR:
> > https://github.com/apache/parquet-format/issues/563
> >
> > To verify this design's compatibility and correctness, three PoC
> implementations
> > were developed:
> > 1. Java: https://github.com/apache/parquet-java/pull/3470
> > 2. C++: https://github.com/apache/arrow/pull/49707
> > 3. Rust: https://github.com/apache/arrow-rs/pull/9678
> >
> > All three PoCs have been verified against a test file produced by the
> Rust PoC:
> > https://github.com/apache/parquet-testing/pull/108
> >
> > The vote will be open for at least 72 hours.
> >
> > [ ] +1 Approve the proposed format change
> > [ ] +0 No opinion
> > [ ] -1 Do not approve (please provide specific reasons)
> >
> > Thanks,
> > Ed Seidl
> >
> > [1] https://lists.apache.org/thread/900503q07v95vyh6fk3qfn7ynb4w6yn2
> > [2] https://github.com/apache/parquet-format/issues/563
> >
> >
>
Re: [VOTE] GH-563: Make ColumnMetaData.path_in_schema optional
Posted by Ed Seidl <et...@apache.org>.
Correction: the link to the PR is https://github.com/apache/parquet-format/pull/564
Ed
On 2026/06/01 14:07:18 Ed Seidl wrote:
> I would like to propose a vote on adopting the format change described in
> GH-563: Make ColumnMetaData.path_in_schema optional.
>
> This was proposed on the mailing list [1] and in GitHub issue #563 [2].
>
> The proposed format specification changes are available in the PR:
> https://github.com/apache/parquet-format/issues/563
>
> To verify this design's compatibility and correctness, three PoC implementations
> were developed:
> 1. Java: https://github.com/apache/parquet-java/pull/3470
> 2. C++: https://github.com/apache/arrow/pull/49707
> 3. Rust: https://github.com/apache/arrow-rs/pull/9678
>
> All three PoCs have been verified against a test file produced by the Rust PoC:
> https://github.com/apache/parquet-testing/pull/108
>
> The vote will be open for at least 72 hours.
>
> [ ] +1 Approve the proposed format change
> [ ] +0 No opinion
> [ ] -1 Do not approve (please provide specific reasons)
>
> Thanks,
> Ed Seidl
>
> [1] https://lists.apache.org/thread/900503q07v95vyh6fk3qfn7ynb4w6yn2
> [2] https://github.com/apache/parquet-format/issues/563
>
>