You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by Alkis Evlogimenos <al...@databricks.com.INVALID> on 2024/07/19 05:22:13 UTC
Re: [DISCUSS] Parquet extensions
After Jul 17th's Parquet Sync feedback I have updated the extensions
proposal to remove the "reservation" mechanism. The updates are already
reflected in the document
<https://docs.google.com/document/d/1KkoR0DjzYnLQXO-d0oRBv2k157IZU0_injqd4eV4WiI/edit>
and
the PR <https://github.com/apache/parquet-format/pull/254>.
On Fri, Jun 28, 2024 at 10:02 AM Alkis Evlogimenos <
alkis.evlogimenos@databricks.com> wrote:
> > I think we can at least have wording to encourage people doing
> extensions to post them publicly and as part of the "reservation" mechanism
> post a link the repo that they are being developed in, if anyone is curious.
>
> Good point. I will try to come up with something in the PR - unless you
> beat me to it :)
>
> On Fri, Jun 28, 2024 at 7:15 AM Micah Kornfield <em...@gmail.com>
> wrote:
>
>> >
>> > 1. experimentation/prototyping is more often than not faster to iterate
>> if
>> > it is closed. Allowing this model of development was a primary goal of
>> the
>> > design.
>>
>>
>> I agree there are advantages here. I think a large amount of speed comes
>> from not having to gain consensus in the community.
>>
>> At the end of the day, I don't think there is any mechanism here to ensure
>> everybody works in public, but I think we can at least have wording to
>> encourage people doing extensions to post them publicly and as part of the
>> "reservation" mechanism post a link the repo that they are being developed
>> in, if anyone is curious. I think this would be particularly useful if
>> there really is an intent for a number of organizations to experiment with
>> new footer designs (but possibly also in others).
>>
>> Thanks,
>> Micah
>>
>>
>>
>>
>> On Wed, Jun 26, 2024 at 9:33 AM Alkis Evlogimenos
>> <al...@databricks.com.invalid> wrote:
>>
>> > Thank you for taking a look Micah.
>> >
>> > On the topic of openness there are various aspects that we have
>> considered.
>> > 1. experimentation/prototyping is more often than not faster to iterate
>> if
>> > it is closed. Allowing this model of development was a primary goal of
>> the
>> > design.
>> > 2. when the design is final, keeping the design closed should have some
>> > drawbacks. Duplicating content to support old readers puts some natural
>> > incentive to make extensions official because at that point one can drop
>> > the fat from the files and move on. Another aspect of the design is the
>> > choice of a single extension field-id which makes the extension space
>> tiny.
>> > This in turn means that it is difficult to interop with others without
>> > breaking their extensions. Ergo the easiest path to any interop is to
>> open
>> > the extension.
>> >
>> > The above, while not enforcing work to happen in the open, strike some
>> > balance in between.
>> >
>> > I am open to suggestions on how to further incentivize opening
>> extensions.
>> >
>> > On Wed, Jun 26, 2024 at 6:04 PM Micah Kornfield <em...@gmail.com>
>> > wrote:
>> >
>> > > Hi Alkis,
>> > > I'm generally in favor of this, my main concern/question is trying to
>> > > encourage work to be in the open. I don't think in the long run it is
>> > good
>> > > for users to always have proprietary extensions inside of Parquet.
>> > >
>> > > IMO, I think the next steps would be to add implementations to write
>> out
>> > > the footer extension points.
>> > >
>> > > Thanks,
>> > > Micah
>> > >
>> > > On Mon, Jun 24, 2024 at 1:24 PM Alkis Evlogimenos
>> > > <al...@databricks.com.invalid> wrote:
>> > >
>> > > > The snafus are fixed. The original should work now.
>> > > >
>> > > > On Sun, 23 Jun 2024, 17:58 Alkis Evlogimenos, <
>> > > > alkis.evlogimenos@databricks.com> wrote:
>> > > >
>> > > > > Due to some sharing snafus with automation, please request access
>> to
>> > > > > comment. If you are just reading I've published this here:
>> > > > >
>> > > >
>> > >
>> >
>> https://docs.google.com/document/d/e/2PACX-1vThXkhHNozn_p1ZZWF-nCzOtoP1lKmkaV4Legq2FaRiIgwyY2XC9AmKpBtpeF8jbBB4wfjmQ6UTg03k/pub
>> > > > >
>> > > > > On Fri, Jun 21, 2024 at 10:29 AM Alkis Evlogimenos <
>> > > > > alkis.evlogimenos@databricks.com> wrote:
>> > > > >
>> > > > >> Hey folks.
>> > > > >>
>> > > > >> I want to move the extension PR
>> > > > >> <https://github.com/apache/parquet-format/pull/254> forward.
>> > > > >> Unfortunately the discussion was spread across the PR, other
>> threads
>> > > and
>> > > > >> documents making it slow to progress. To avoid further
>> > fragmentation I
>> > > > have
>> > > > >> put together a document
>> > > > >> <
>> > > >
>> > >
>> >
>> https://docs.google.com/document/d/1KkoR0DjzYnLQXO-d0oRBv2k157IZU0_injqd4eV4WiI/edit
>> > > > >
>> > > > >> discussing the extensions mechanism in isolation. I believe the
>> > > document
>> > > > >> addresses all the concerns/comments from the PR and mailing list
>> > > > >> discussions brought forward so far.
>> > > > >>
>> > > > >> I propose we continue the discussion in the document and once
>> > > everything
>> > > > >> is addressed, we finalize the PR.
>> > > > >>
>> > > > >> Thank you,
>> > > > >>
>> > > > >
>> > > >
>> > >
>> >
>>
>
Re: [DISCUSS] Parquet extensions
Posted by Julien Le Dem <ju...@apache.org>.
Here is the invite link: https://calendar.app.google/UMMmbUV1JMh7ffGt6
today 9:30am PT (in 30 mins)
On Wed, Jul 31, 2024 at 8:04 AM Alkis Evlogimenos
<al...@databricks.com.invalid> wrote:
> Thank you folks, sounds exciting!
>
> I don't see an invite for the sync. Is it happening today?
>
> On Wed, Jul 31, 2024 at 3:12 AM Julien Le Dem <ju...@apache.org> wrote:
>
> > It sounds like everybody is happy with the proposal.
> > Tomorrow is the Parquet sync, we can finalize then.
> >
> > On Wed, Jul 24, 2024 at 9:20 AM Julien Le Dem <ju...@apache.org> wrote:
> >
> > > Hi Alkis,
> > > I saw you addressed and resolved the comments in the doc. Thank you.
> > > This looks good to me.
> > > I would recommend others that have been active in this conversation to
> > > take a final look.
> > > Best
> > > Julien
> > >
> > > On Tue, Jul 23, 2024 at 3:06 PM Julien Le Dem <ju...@apache.org>
> wrote:
> > >
> > >> I am also OK with the proposed solution in the document.
> > >> However I think the doc itself needs one last wording change.
> > >> I have left more details in comments but here is the gist:
> > >> This effort is driven by a group of people in the community and not
> one
> > >> vendor in particular even if said people do sometimes work for
> vendors.
> > >> To reflect this, instead of saying the UUID identifies a Vendor, we
> > >> should describe it as an extension ID.
> > >> Then I'd remove all instances of the word "Vendor" and instead
> > >> refer to "Extensions" identified by this UUID.
> > >> This might not change anything to the implementation but it is
> important
> > >> to reflecting how the community works in the document.
> > >>
> > >> Specifically:
> > >>
> > >> "Vendor introduces a Flatbuffers variant of FileMetaData." => "This
> > >> extension introduces a Flatbuffers variant of FileMetaData..."
> > >>
> > >> "The UUID is picked by the Vendor once and used throughout the
> > >> experiments." => "The UUID is picked for this specific extension and
> > used
> > >> throughout the experiments."
> > >>
> > >> "At some point Vendor decides that this is amazing and should be
> shared
> > >> with the world at large to advance Parquet. " => "At some point, the
> > >> community decides this extension is ready and proposed for inclusion."
> > >>
> > >>
> > >> On Mon, Jul 22, 2024 at 10:11 PM Micah Kornfield <
> emkornfield@gmail.com
> > >
> > >> wrote:
> > >>
> > >>> Hi Alkis,
> > >>> Thanks for the revision. I'm OK with this as is, we can maybe wait a
> > few
> > >>> more days to see if anybody else has comments and then discuss
> > >>> implementation of the extension mechanism?
> > >>>
> > >>> Cheers,
> > >>> Micah
> > >>>
> > >>> On Thu, Jul 18, 2024 at 10:22 PM Alkis Evlogimenos
> > >>> <al...@databricks.com.invalid> wrote:
> > >>>
> > >>> > After Jul 17th's Parquet Sync feedback I have updated the
> extensions
> > >>> > proposal to remove the "reservation" mechanism. The updates are
> > already
> > >>> > reflected in the document
> > >>> > <
> > >>> >
> > >>>
> >
> https://docs.google.com/document/d/1KkoR0DjzYnLQXO-d0oRBv2k157IZU0_injqd4eV4WiI/edit
> > >>> > >
> > >>> > and
> > >>> > the PR <https://github.com/apache/parquet-format/pull/254>.
> > >>> >
> > >>> > On Fri, Jun 28, 2024 at 10:02 AM Alkis Evlogimenos <
> > >>> > alkis.evlogimenos@databricks.com> wrote:
> > >>> >
> > >>> > > > I think we can at least have wording to encourage people doing
> > >>> > > extensions to post them publicly and as part of the "reservation"
> > >>> > mechanism
> > >>> > > post a link the repo that they are being developed in, if anyone
> is
> > >>> > curious.
> > >>> > >
> > >>> > > Good point. I will try to come up with something in the PR -
> unless
> > >>> you
> > >>> > > beat me to it :)
> > >>> > >
> > >>> > > On Fri, Jun 28, 2024 at 7:15 AM Micah Kornfield <
> > >>> emkornfield@gmail.com>
> > >>> > > wrote:
> > >>> > >
> > >>> > >> >
> > >>> > >> > 1. experimentation/prototyping is more often than not faster
> to
> > >>> > iterate
> > >>> > >> if
> > >>> > >> > it is closed. Allowing this model of development was a primary
> > >>> goal of
> > >>> > >> the
> > >>> > >> > design.
> > >>> > >>
> > >>> > >>
> > >>> > >> I agree there are advantages here. I think a large amount of
> > speed
> > >>> > comes
> > >>> > >> from not having to gain consensus in the community.
> > >>> > >>
> > >>> > >> At the end of the day, I don't think there is any mechanism here
> > to
> > >>> > ensure
> > >>> > >> everybody works in public, but I think we can at least have
> > wording
> > >>> to
> > >>> > >> encourage people doing extensions to post them publicly and as
> > part
> > >>> of
> > >>> > the
> > >>> > >> "reservation" mechanism post a link the repo that they are being
> > >>> > developed
> > >>> > >> in, if anyone is curious. I think this would be particularly
> > >>> useful if
> > >>> > >> there really is an intent for a number of organizations to
> > >>> experiment
> > >>> > with
> > >>> > >> new footer designs (but possibly also in others).
> > >>> > >>
> > >>> > >> Thanks,
> > >>> > >> Micah
> > >>> > >>
> > >>> > >>
> > >>> > >>
> > >>> > >>
> > >>> > >> On Wed, Jun 26, 2024 at 9:33 AM Alkis Evlogimenos
> > >>> > >> <al...@databricks.com.invalid> wrote:
> > >>> > >>
> > >>> > >> > Thank you for taking a look Micah.
> > >>> > >> >
> > >>> > >> > On the topic of openness there are various aspects that we
> have
> > >>> > >> considered.
> > >>> > >> > 1. experimentation/prototyping is more often than not faster
> to
> > >>> > iterate
> > >>> > >> if
> > >>> > >> > it is closed. Allowing this model of development was a primary
> > >>> goal of
> > >>> > >> the
> > >>> > >> > design.
> > >>> > >> > 2. when the design is final, keeping the design closed should
> > have
> > >>> > some
> > >>> > >> > drawbacks. Duplicating content to support old readers puts
> some
> > >>> > natural
> > >>> > >> > incentive to make extensions official because at that point
> one
> > >>> can
> > >>> > drop
> > >>> > >> > the fat from the files and move on. Another aspect of the
> design
> > >>> is
> > >>> > the
> > >>> > >> > choice of a single extension field-id which makes the
> extension
> > >>> space
> > >>> > >> tiny.
> > >>> > >> > This in turn means that it is difficult to interop with others
> > >>> without
> > >>> > >> > breaking their extensions. Ergo the easiest path to any
> interop
> > >>> is to
> > >>> > >> open
> > >>> > >> > the extension.
> > >>> > >> >
> > >>> > >> > The above, while not enforcing work to happen in the open,
> > strike
> > >>> some
> > >>> > >> > balance in between.
> > >>> > >> >
> > >>> > >> > I am open to suggestions on how to further incentivize opening
> > >>> > >> extensions.
> > >>> > >> >
> > >>> > >> > On Wed, Jun 26, 2024 at 6:04 PM Micah Kornfield <
> > >>> > emkornfield@gmail.com>
> > >>> > >> > wrote:
> > >>> > >> >
> > >>> > >> > > Hi Alkis,
> > >>> > >> > > I'm generally in favor of this, my main concern/question is
> > >>> trying
> > >>> > to
> > >>> > >> > > encourage work to be in the open. I don't think in the long
> > >>> run it
> > >>> > is
> > >>> > >> > good
> > >>> > >> > > for users to always have proprietary extensions inside of
> > >>> Parquet.
> > >>> > >> > >
> > >>> > >> > > IMO, I think the next steps would be to add implementations
> to
> > >>> write
> > >>> > >> out
> > >>> > >> > > the footer extension points.
> > >>> > >> > >
> > >>> > >> > > Thanks,
> > >>> > >> > > Micah
> > >>> > >> > >
> > >>> > >> > > On Mon, Jun 24, 2024 at 1:24 PM Alkis Evlogimenos
> > >>> > >> > > <al...@databricks.com.invalid> wrote:
> > >>> > >> > >
> > >>> > >> > > > The snafus are fixed. The original should work now.
> > >>> > >> > > >
> > >>> > >> > > > On Sun, 23 Jun 2024, 17:58 Alkis Evlogimenos, <
> > >>> > >> > > > alkis.evlogimenos@databricks.com> wrote:
> > >>> > >> > > >
> > >>> > >> > > > > Due to some sharing snafus with automation, please
> request
> > >>> > access
> > >>> > >> to
> > >>> > >> > > > > comment. If you are just reading I've published this
> here:
> > >>> > >> > > > >
> > >>> > >> > > >
> > >>> > >> > >
> > >>> > >> >
> > >>> > >>
> > >>> >
> > >>>
> >
> https://docs.google.com/document/d/e/2PACX-1vThXkhHNozn_p1ZZWF-nCzOtoP1lKmkaV4Legq2FaRiIgwyY2XC9AmKpBtpeF8jbBB4wfjmQ6UTg03k/pub
> > >>> > >> > > > >
> > >>> > >> > > > > On Fri, Jun 21, 2024 at 10:29 AM Alkis Evlogimenos <
> > >>> > >> > > > > alkis.evlogimenos@databricks.com> wrote:
> > >>> > >> > > > >
> > >>> > >> > > > >> Hey folks.
> > >>> > >> > > > >>
> > >>> > >> > > > >> I want to move the extension PR
> > >>> > >> > > > >> <https://github.com/apache/parquet-format/pull/254>
> > >>> forward.
> > >>> > >> > > > >> Unfortunately the discussion was spread across the PR,
> > >>> other
> > >>> > >> threads
> > >>> > >> > > and
> > >>> > >> > > > >> documents making it slow to progress. To avoid further
> > >>> > >> > fragmentation I
> > >>> > >> > > > have
> > >>> > >> > > > >> put together a document
> > >>> > >> > > > >> <
> > >>> > >> > > >
> > >>> > >> > >
> > >>> > >> >
> > >>> > >>
> > >>> >
> > >>>
> >
> https://docs.google.com/document/d/1KkoR0DjzYnLQXO-d0oRBv2k157IZU0_injqd4eV4WiI/edit
> > >>> > >> > > > >
> > >>> > >> > > > >> discussing the extensions mechanism in isolation. I
> > >>> believe the
> > >>> > >> > > document
> > >>> > >> > > > >> addresses all the concerns/comments from the PR and
> > mailing
> > >>> > list
> > >>> > >> > > > >> discussions brought forward so far.
> > >>> > >> > > > >>
> > >>> > >> > > > >> I propose we continue the discussion in the document
> and
> > >>> once
> > >>> > >> > > everything
> > >>> > >> > > > >> is addressed, we finalize the PR.
> > >>> > >> > > > >>
> > >>> > >> > > > >> Thank you,
> > >>> > >> > > > >>
> > >>> > >> > > > >
> > >>> > >> > > >
> > >>> > >> > >
> > >>> > >> >
> > >>> > >>
> > >>> > >
> > >>> >
> > >>>
> > >>
> >
>
Re: [DISCUSS] Parquet extensions
Posted by Alkis Evlogimenos <al...@databricks.com.INVALID>.
Thank you folks, sounds exciting!
I don't see an invite for the sync. Is it happening today?
On Wed, Jul 31, 2024 at 3:12 AM Julien Le Dem <ju...@apache.org> wrote:
> It sounds like everybody is happy with the proposal.
> Tomorrow is the Parquet sync, we can finalize then.
>
> On Wed, Jul 24, 2024 at 9:20 AM Julien Le Dem <ju...@apache.org> wrote:
>
> > Hi Alkis,
> > I saw you addressed and resolved the comments in the doc. Thank you.
> > This looks good to me.
> > I would recommend others that have been active in this conversation to
> > take a final look.
> > Best
> > Julien
> >
> > On Tue, Jul 23, 2024 at 3:06 PM Julien Le Dem <ju...@apache.org> wrote:
> >
> >> I am also OK with the proposed solution in the document.
> >> However I think the doc itself needs one last wording change.
> >> I have left more details in comments but here is the gist:
> >> This effort is driven by a group of people in the community and not one
> >> vendor in particular even if said people do sometimes work for vendors.
> >> To reflect this, instead of saying the UUID identifies a Vendor, we
> >> should describe it as an extension ID.
> >> Then I'd remove all instances of the word "Vendor" and instead
> >> refer to "Extensions" identified by this UUID.
> >> This might not change anything to the implementation but it is important
> >> to reflecting how the community works in the document.
> >>
> >> Specifically:
> >>
> >> "Vendor introduces a Flatbuffers variant of FileMetaData." => "This
> >> extension introduces a Flatbuffers variant of FileMetaData..."
> >>
> >> "The UUID is picked by the Vendor once and used throughout the
> >> experiments." => "The UUID is picked for this specific extension and
> used
> >> throughout the experiments."
> >>
> >> "At some point Vendor decides that this is amazing and should be shared
> >> with the world at large to advance Parquet. " => "At some point, the
> >> community decides this extension is ready and proposed for inclusion."
> >>
> >>
> >> On Mon, Jul 22, 2024 at 10:11 PM Micah Kornfield <emkornfield@gmail.com
> >
> >> wrote:
> >>
> >>> Hi Alkis,
> >>> Thanks for the revision. I'm OK with this as is, we can maybe wait a
> few
> >>> more days to see if anybody else has comments and then discuss
> >>> implementation of the extension mechanism?
> >>>
> >>> Cheers,
> >>> Micah
> >>>
> >>> On Thu, Jul 18, 2024 at 10:22 PM Alkis Evlogimenos
> >>> <al...@databricks.com.invalid> wrote:
> >>>
> >>> > After Jul 17th's Parquet Sync feedback I have updated the extensions
> >>> > proposal to remove the "reservation" mechanism. The updates are
> already
> >>> > reflected in the document
> >>> > <
> >>> >
> >>>
> https://docs.google.com/document/d/1KkoR0DjzYnLQXO-d0oRBv2k157IZU0_injqd4eV4WiI/edit
> >>> > >
> >>> > and
> >>> > the PR <https://github.com/apache/parquet-format/pull/254>.
> >>> >
> >>> > On Fri, Jun 28, 2024 at 10:02 AM Alkis Evlogimenos <
> >>> > alkis.evlogimenos@databricks.com> wrote:
> >>> >
> >>> > > > I think we can at least have wording to encourage people doing
> >>> > > extensions to post them publicly and as part of the "reservation"
> >>> > mechanism
> >>> > > post a link the repo that they are being developed in, if anyone is
> >>> > curious.
> >>> > >
> >>> > > Good point. I will try to come up with something in the PR - unless
> >>> you
> >>> > > beat me to it :)
> >>> > >
> >>> > > On Fri, Jun 28, 2024 at 7:15 AM Micah Kornfield <
> >>> emkornfield@gmail.com>
> >>> > > wrote:
> >>> > >
> >>> > >> >
> >>> > >> > 1. experimentation/prototyping is more often than not faster to
> >>> > iterate
> >>> > >> if
> >>> > >> > it is closed. Allowing this model of development was a primary
> >>> goal of
> >>> > >> the
> >>> > >> > design.
> >>> > >>
> >>> > >>
> >>> > >> I agree there are advantages here. I think a large amount of
> speed
> >>> > comes
> >>> > >> from not having to gain consensus in the community.
> >>> > >>
> >>> > >> At the end of the day, I don't think there is any mechanism here
> to
> >>> > ensure
> >>> > >> everybody works in public, but I think we can at least have
> wording
> >>> to
> >>> > >> encourage people doing extensions to post them publicly and as
> part
> >>> of
> >>> > the
> >>> > >> "reservation" mechanism post a link the repo that they are being
> >>> > developed
> >>> > >> in, if anyone is curious. I think this would be particularly
> >>> useful if
> >>> > >> there really is an intent for a number of organizations to
> >>> experiment
> >>> > with
> >>> > >> new footer designs (but possibly also in others).
> >>> > >>
> >>> > >> Thanks,
> >>> > >> Micah
> >>> > >>
> >>> > >>
> >>> > >>
> >>> > >>
> >>> > >> On Wed, Jun 26, 2024 at 9:33 AM Alkis Evlogimenos
> >>> > >> <al...@databricks.com.invalid> wrote:
> >>> > >>
> >>> > >> > Thank you for taking a look Micah.
> >>> > >> >
> >>> > >> > On the topic of openness there are various aspects that we have
> >>> > >> considered.
> >>> > >> > 1. experimentation/prototyping is more often than not faster to
> >>> > iterate
> >>> > >> if
> >>> > >> > it is closed. Allowing this model of development was a primary
> >>> goal of
> >>> > >> the
> >>> > >> > design.
> >>> > >> > 2. when the design is final, keeping the design closed should
> have
> >>> > some
> >>> > >> > drawbacks. Duplicating content to support old readers puts some
> >>> > natural
> >>> > >> > incentive to make extensions official because at that point one
> >>> can
> >>> > drop
> >>> > >> > the fat from the files and move on. Another aspect of the design
> >>> is
> >>> > the
> >>> > >> > choice of a single extension field-id which makes the extension
> >>> space
> >>> > >> tiny.
> >>> > >> > This in turn means that it is difficult to interop with others
> >>> without
> >>> > >> > breaking their extensions. Ergo the easiest path to any interop
> >>> is to
> >>> > >> open
> >>> > >> > the extension.
> >>> > >> >
> >>> > >> > The above, while not enforcing work to happen in the open,
> strike
> >>> some
> >>> > >> > balance in between.
> >>> > >> >
> >>> > >> > I am open to suggestions on how to further incentivize opening
> >>> > >> extensions.
> >>> > >> >
> >>> > >> > On Wed, Jun 26, 2024 at 6:04 PM Micah Kornfield <
> >>> > emkornfield@gmail.com>
> >>> > >> > wrote:
> >>> > >> >
> >>> > >> > > Hi Alkis,
> >>> > >> > > I'm generally in favor of this, my main concern/question is
> >>> trying
> >>> > to
> >>> > >> > > encourage work to be in the open. I don't think in the long
> >>> run it
> >>> > is
> >>> > >> > good
> >>> > >> > > for users to always have proprietary extensions inside of
> >>> Parquet.
> >>> > >> > >
> >>> > >> > > IMO, I think the next steps would be to add implementations to
> >>> write
> >>> > >> out
> >>> > >> > > the footer extension points.
> >>> > >> > >
> >>> > >> > > Thanks,
> >>> > >> > > Micah
> >>> > >> > >
> >>> > >> > > On Mon, Jun 24, 2024 at 1:24 PM Alkis Evlogimenos
> >>> > >> > > <al...@databricks.com.invalid> wrote:
> >>> > >> > >
> >>> > >> > > > The snafus are fixed. The original should work now.
> >>> > >> > > >
> >>> > >> > > > On Sun, 23 Jun 2024, 17:58 Alkis Evlogimenos, <
> >>> > >> > > > alkis.evlogimenos@databricks.com> wrote:
> >>> > >> > > >
> >>> > >> > > > > Due to some sharing snafus with automation, please request
> >>> > access
> >>> > >> to
> >>> > >> > > > > comment. If you are just reading I've published this here:
> >>> > >> > > > >
> >>> > >> > > >
> >>> > >> > >
> >>> > >> >
> >>> > >>
> >>> >
> >>>
> https://docs.google.com/document/d/e/2PACX-1vThXkhHNozn_p1ZZWF-nCzOtoP1lKmkaV4Legq2FaRiIgwyY2XC9AmKpBtpeF8jbBB4wfjmQ6UTg03k/pub
> >>> > >> > > > >
> >>> > >> > > > > On Fri, Jun 21, 2024 at 10:29 AM Alkis Evlogimenos <
> >>> > >> > > > > alkis.evlogimenos@databricks.com> wrote:
> >>> > >> > > > >
> >>> > >> > > > >> Hey folks.
> >>> > >> > > > >>
> >>> > >> > > > >> I want to move the extension PR
> >>> > >> > > > >> <https://github.com/apache/parquet-format/pull/254>
> >>> forward.
> >>> > >> > > > >> Unfortunately the discussion was spread across the PR,
> >>> other
> >>> > >> threads
> >>> > >> > > and
> >>> > >> > > > >> documents making it slow to progress. To avoid further
> >>> > >> > fragmentation I
> >>> > >> > > > have
> >>> > >> > > > >> put together a document
> >>> > >> > > > >> <
> >>> > >> > > >
> >>> > >> > >
> >>> > >> >
> >>> > >>
> >>> >
> >>>
> https://docs.google.com/document/d/1KkoR0DjzYnLQXO-d0oRBv2k157IZU0_injqd4eV4WiI/edit
> >>> > >> > > > >
> >>> > >> > > > >> discussing the extensions mechanism in isolation. I
> >>> believe the
> >>> > >> > > document
> >>> > >> > > > >> addresses all the concerns/comments from the PR and
> mailing
> >>> > list
> >>> > >> > > > >> discussions brought forward so far.
> >>> > >> > > > >>
> >>> > >> > > > >> I propose we continue the discussion in the document and
> >>> once
> >>> > >> > > everything
> >>> > >> > > > >> is addressed, we finalize the PR.
> >>> > >> > > > >>
> >>> > >> > > > >> Thank you,
> >>> > >> > > > >>
> >>> > >> > > > >
> >>> > >> > > >
> >>> > >> > >
> >>> > >> >
> >>> > >>
> >>> > >
> >>> >
> >>>
> >>
>
Re: [DISCUSS] Parquet extensions
Posted by Julien Le Dem <ju...@apache.org>.
It sounds like everybody is happy with the proposal.
Tomorrow is the Parquet sync, we can finalize then.
On Wed, Jul 24, 2024 at 9:20 AM Julien Le Dem <ju...@apache.org> wrote:
> Hi Alkis,
> I saw you addressed and resolved the comments in the doc. Thank you.
> This looks good to me.
> I would recommend others that have been active in this conversation to
> take a final look.
> Best
> Julien
>
> On Tue, Jul 23, 2024 at 3:06 PM Julien Le Dem <ju...@apache.org> wrote:
>
>> I am also OK with the proposed solution in the document.
>> However I think the doc itself needs one last wording change.
>> I have left more details in comments but here is the gist:
>> This effort is driven by a group of people in the community and not one
>> vendor in particular even if said people do sometimes work for vendors.
>> To reflect this, instead of saying the UUID identifies a Vendor, we
>> should describe it as an extension ID.
>> Then I'd remove all instances of the word "Vendor" and instead
>> refer to "Extensions" identified by this UUID.
>> This might not change anything to the implementation but it is important
>> to reflecting how the community works in the document.
>>
>> Specifically:
>>
>> "Vendor introduces a Flatbuffers variant of FileMetaData." => "This
>> extension introduces a Flatbuffers variant of FileMetaData..."
>>
>> "The UUID is picked by the Vendor once and used throughout the
>> experiments." => "The UUID is picked for this specific extension and used
>> throughout the experiments."
>>
>> "At some point Vendor decides that this is amazing and should be shared
>> with the world at large to advance Parquet. " => "At some point, the
>> community decides this extension is ready and proposed for inclusion."
>>
>>
>> On Mon, Jul 22, 2024 at 10:11 PM Micah Kornfield <em...@gmail.com>
>> wrote:
>>
>>> Hi Alkis,
>>> Thanks for the revision. I'm OK with this as is, we can maybe wait a few
>>> more days to see if anybody else has comments and then discuss
>>> implementation of the extension mechanism?
>>>
>>> Cheers,
>>> Micah
>>>
>>> On Thu, Jul 18, 2024 at 10:22 PM Alkis Evlogimenos
>>> <al...@databricks.com.invalid> wrote:
>>>
>>> > After Jul 17th's Parquet Sync feedback I have updated the extensions
>>> > proposal to remove the "reservation" mechanism. The updates are already
>>> > reflected in the document
>>> > <
>>> >
>>> https://docs.google.com/document/d/1KkoR0DjzYnLQXO-d0oRBv2k157IZU0_injqd4eV4WiI/edit
>>> > >
>>> > and
>>> > the PR <https://github.com/apache/parquet-format/pull/254>.
>>> >
>>> > On Fri, Jun 28, 2024 at 10:02 AM Alkis Evlogimenos <
>>> > alkis.evlogimenos@databricks.com> wrote:
>>> >
>>> > > > I think we can at least have wording to encourage people doing
>>> > > extensions to post them publicly and as part of the "reservation"
>>> > mechanism
>>> > > post a link the repo that they are being developed in, if anyone is
>>> > curious.
>>> > >
>>> > > Good point. I will try to come up with something in the PR - unless
>>> you
>>> > > beat me to it :)
>>> > >
>>> > > On Fri, Jun 28, 2024 at 7:15 AM Micah Kornfield <
>>> emkornfield@gmail.com>
>>> > > wrote:
>>> > >
>>> > >> >
>>> > >> > 1. experimentation/prototyping is more often than not faster to
>>> > iterate
>>> > >> if
>>> > >> > it is closed. Allowing this model of development was a primary
>>> goal of
>>> > >> the
>>> > >> > design.
>>> > >>
>>> > >>
>>> > >> I agree there are advantages here. I think a large amount of speed
>>> > comes
>>> > >> from not having to gain consensus in the community.
>>> > >>
>>> > >> At the end of the day, I don't think there is any mechanism here to
>>> > ensure
>>> > >> everybody works in public, but I think we can at least have wording
>>> to
>>> > >> encourage people doing extensions to post them publicly and as part
>>> of
>>> > the
>>> > >> "reservation" mechanism post a link the repo that they are being
>>> > developed
>>> > >> in, if anyone is curious. I think this would be particularly
>>> useful if
>>> > >> there really is an intent for a number of organizations to
>>> experiment
>>> > with
>>> > >> new footer designs (but possibly also in others).
>>> > >>
>>> > >> Thanks,
>>> > >> Micah
>>> > >>
>>> > >>
>>> > >>
>>> > >>
>>> > >> On Wed, Jun 26, 2024 at 9:33 AM Alkis Evlogimenos
>>> > >> <al...@databricks.com.invalid> wrote:
>>> > >>
>>> > >> > Thank you for taking a look Micah.
>>> > >> >
>>> > >> > On the topic of openness there are various aspects that we have
>>> > >> considered.
>>> > >> > 1. experimentation/prototyping is more often than not faster to
>>> > iterate
>>> > >> if
>>> > >> > it is closed. Allowing this model of development was a primary
>>> goal of
>>> > >> the
>>> > >> > design.
>>> > >> > 2. when the design is final, keeping the design closed should have
>>> > some
>>> > >> > drawbacks. Duplicating content to support old readers puts some
>>> > natural
>>> > >> > incentive to make extensions official because at that point one
>>> can
>>> > drop
>>> > >> > the fat from the files and move on. Another aspect of the design
>>> is
>>> > the
>>> > >> > choice of a single extension field-id which makes the extension
>>> space
>>> > >> tiny.
>>> > >> > This in turn means that it is difficult to interop with others
>>> without
>>> > >> > breaking their extensions. Ergo the easiest path to any interop
>>> is to
>>> > >> open
>>> > >> > the extension.
>>> > >> >
>>> > >> > The above, while not enforcing work to happen in the open, strike
>>> some
>>> > >> > balance in between.
>>> > >> >
>>> > >> > I am open to suggestions on how to further incentivize opening
>>> > >> extensions.
>>> > >> >
>>> > >> > On Wed, Jun 26, 2024 at 6:04 PM Micah Kornfield <
>>> > emkornfield@gmail.com>
>>> > >> > wrote:
>>> > >> >
>>> > >> > > Hi Alkis,
>>> > >> > > I'm generally in favor of this, my main concern/question is
>>> trying
>>> > to
>>> > >> > > encourage work to be in the open. I don't think in the long
>>> run it
>>> > is
>>> > >> > good
>>> > >> > > for users to always have proprietary extensions inside of
>>> Parquet.
>>> > >> > >
>>> > >> > > IMO, I think the next steps would be to add implementations to
>>> write
>>> > >> out
>>> > >> > > the footer extension points.
>>> > >> > >
>>> > >> > > Thanks,
>>> > >> > > Micah
>>> > >> > >
>>> > >> > > On Mon, Jun 24, 2024 at 1:24 PM Alkis Evlogimenos
>>> > >> > > <al...@databricks.com.invalid> wrote:
>>> > >> > >
>>> > >> > > > The snafus are fixed. The original should work now.
>>> > >> > > >
>>> > >> > > > On Sun, 23 Jun 2024, 17:58 Alkis Evlogimenos, <
>>> > >> > > > alkis.evlogimenos@databricks.com> wrote:
>>> > >> > > >
>>> > >> > > > > Due to some sharing snafus with automation, please request
>>> > access
>>> > >> to
>>> > >> > > > > comment. If you are just reading I've published this here:
>>> > >> > > > >
>>> > >> > > >
>>> > >> > >
>>> > >> >
>>> > >>
>>> >
>>> https://docs.google.com/document/d/e/2PACX-1vThXkhHNozn_p1ZZWF-nCzOtoP1lKmkaV4Legq2FaRiIgwyY2XC9AmKpBtpeF8jbBB4wfjmQ6UTg03k/pub
>>> > >> > > > >
>>> > >> > > > > On Fri, Jun 21, 2024 at 10:29 AM Alkis Evlogimenos <
>>> > >> > > > > alkis.evlogimenos@databricks.com> wrote:
>>> > >> > > > >
>>> > >> > > > >> Hey folks.
>>> > >> > > > >>
>>> > >> > > > >> I want to move the extension PR
>>> > >> > > > >> <https://github.com/apache/parquet-format/pull/254>
>>> forward.
>>> > >> > > > >> Unfortunately the discussion was spread across the PR,
>>> other
>>> > >> threads
>>> > >> > > and
>>> > >> > > > >> documents making it slow to progress. To avoid further
>>> > >> > fragmentation I
>>> > >> > > > have
>>> > >> > > > >> put together a document
>>> > >> > > > >> <
>>> > >> > > >
>>> > >> > >
>>> > >> >
>>> > >>
>>> >
>>> https://docs.google.com/document/d/1KkoR0DjzYnLQXO-d0oRBv2k157IZU0_injqd4eV4WiI/edit
>>> > >> > > > >
>>> > >> > > > >> discussing the extensions mechanism in isolation. I
>>> believe the
>>> > >> > > document
>>> > >> > > > >> addresses all the concerns/comments from the PR and mailing
>>> > list
>>> > >> > > > >> discussions brought forward so far.
>>> > >> > > > >>
>>> > >> > > > >> I propose we continue the discussion in the document and
>>> once
>>> > >> > > everything
>>> > >> > > > >> is addressed, we finalize the PR.
>>> > >> > > > >>
>>> > >> > > > >> Thank you,
>>> > >> > > > >>
>>> > >> > > > >
>>> > >> > > >
>>> > >> > >
>>> > >> >
>>> > >>
>>> > >
>>> >
>>>
>>
Re: [DISCUSS] Parquet extensions
Posted by Julien Le Dem <ju...@apache.org>.
Hi Alkis,
I saw you addressed and resolved the comments in the doc. Thank you.
This looks good to me.
I would recommend others that have been active in this conversation to take
a final look.
Best
Julien
On Tue, Jul 23, 2024 at 3:06 PM Julien Le Dem <ju...@apache.org> wrote:
> I am also OK with the proposed solution in the document.
> However I think the doc itself needs one last wording change.
> I have left more details in comments but here is the gist:
> This effort is driven by a group of people in the community and not one
> vendor in particular even if said people do sometimes work for vendors.
> To reflect this, instead of saying the UUID identifies a Vendor, we should
> describe it as an extension ID.
> Then I'd remove all instances of the word "Vendor" and instead
> refer to "Extensions" identified by this UUID.
> This might not change anything to the implementation but it is important
> to reflecting how the community works in the document.
>
> Specifically:
>
> "Vendor introduces a Flatbuffers variant of FileMetaData." => "This
> extension introduces a Flatbuffers variant of FileMetaData..."
>
> "The UUID is picked by the Vendor once and used throughout the
> experiments." => "The UUID is picked for this specific extension and used
> throughout the experiments."
>
> "At some point Vendor decides that this is amazing and should be shared
> with the world at large to advance Parquet. " => "At some point, the
> community decides this extension is ready and proposed for inclusion."
>
>
> On Mon, Jul 22, 2024 at 10:11 PM Micah Kornfield <em...@gmail.com>
> wrote:
>
>> Hi Alkis,
>> Thanks for the revision. I'm OK with this as is, we can maybe wait a few
>> more days to see if anybody else has comments and then discuss
>> implementation of the extension mechanism?
>>
>> Cheers,
>> Micah
>>
>> On Thu, Jul 18, 2024 at 10:22 PM Alkis Evlogimenos
>> <al...@databricks.com.invalid> wrote:
>>
>> > After Jul 17th's Parquet Sync feedback I have updated the extensions
>> > proposal to remove the "reservation" mechanism. The updates are already
>> > reflected in the document
>> > <
>> >
>> https://docs.google.com/document/d/1KkoR0DjzYnLQXO-d0oRBv2k157IZU0_injqd4eV4WiI/edit
>> > >
>> > and
>> > the PR <https://github.com/apache/parquet-format/pull/254>.
>> >
>> > On Fri, Jun 28, 2024 at 10:02 AM Alkis Evlogimenos <
>> > alkis.evlogimenos@databricks.com> wrote:
>> >
>> > > > I think we can at least have wording to encourage people doing
>> > > extensions to post them publicly and as part of the "reservation"
>> > mechanism
>> > > post a link the repo that they are being developed in, if anyone is
>> > curious.
>> > >
>> > > Good point. I will try to come up with something in the PR - unless
>> you
>> > > beat me to it :)
>> > >
>> > > On Fri, Jun 28, 2024 at 7:15 AM Micah Kornfield <
>> emkornfield@gmail.com>
>> > > wrote:
>> > >
>> > >> >
>> > >> > 1. experimentation/prototyping is more often than not faster to
>> > iterate
>> > >> if
>> > >> > it is closed. Allowing this model of development was a primary
>> goal of
>> > >> the
>> > >> > design.
>> > >>
>> > >>
>> > >> I agree there are advantages here. I think a large amount of speed
>> > comes
>> > >> from not having to gain consensus in the community.
>> > >>
>> > >> At the end of the day, I don't think there is any mechanism here to
>> > ensure
>> > >> everybody works in public, but I think we can at least have wording
>> to
>> > >> encourage people doing extensions to post them publicly and as part
>> of
>> > the
>> > >> "reservation" mechanism post a link the repo that they are being
>> > developed
>> > >> in, if anyone is curious. I think this would be particularly useful
>> if
>> > >> there really is an intent for a number of organizations to experiment
>> > with
>> > >> new footer designs (but possibly also in others).
>> > >>
>> > >> Thanks,
>> > >> Micah
>> > >>
>> > >>
>> > >>
>> > >>
>> > >> On Wed, Jun 26, 2024 at 9:33 AM Alkis Evlogimenos
>> > >> <al...@databricks.com.invalid> wrote:
>> > >>
>> > >> > Thank you for taking a look Micah.
>> > >> >
>> > >> > On the topic of openness there are various aspects that we have
>> > >> considered.
>> > >> > 1. experimentation/prototyping is more often than not faster to
>> > iterate
>> > >> if
>> > >> > it is closed. Allowing this model of development was a primary
>> goal of
>> > >> the
>> > >> > design.
>> > >> > 2. when the design is final, keeping the design closed should have
>> > some
>> > >> > drawbacks. Duplicating content to support old readers puts some
>> > natural
>> > >> > incentive to make extensions official because at that point one can
>> > drop
>> > >> > the fat from the files and move on. Another aspect of the design is
>> > the
>> > >> > choice of a single extension field-id which makes the extension
>> space
>> > >> tiny.
>> > >> > This in turn means that it is difficult to interop with others
>> without
>> > >> > breaking their extensions. Ergo the easiest path to any interop is
>> to
>> > >> open
>> > >> > the extension.
>> > >> >
>> > >> > The above, while not enforcing work to happen in the open, strike
>> some
>> > >> > balance in between.
>> > >> >
>> > >> > I am open to suggestions on how to further incentivize opening
>> > >> extensions.
>> > >> >
>> > >> > On Wed, Jun 26, 2024 at 6:04 PM Micah Kornfield <
>> > emkornfield@gmail.com>
>> > >> > wrote:
>> > >> >
>> > >> > > Hi Alkis,
>> > >> > > I'm generally in favor of this, my main concern/question is
>> trying
>> > to
>> > >> > > encourage work to be in the open. I don't think in the long run
>> it
>> > is
>> > >> > good
>> > >> > > for users to always have proprietary extensions inside of
>> Parquet.
>> > >> > >
>> > >> > > IMO, I think the next steps would be to add implementations to
>> write
>> > >> out
>> > >> > > the footer extension points.
>> > >> > >
>> > >> > > Thanks,
>> > >> > > Micah
>> > >> > >
>> > >> > > On Mon, Jun 24, 2024 at 1:24 PM Alkis Evlogimenos
>> > >> > > <al...@databricks.com.invalid> wrote:
>> > >> > >
>> > >> > > > The snafus are fixed. The original should work now.
>> > >> > > >
>> > >> > > > On Sun, 23 Jun 2024, 17:58 Alkis Evlogimenos, <
>> > >> > > > alkis.evlogimenos@databricks.com> wrote:
>> > >> > > >
>> > >> > > > > Due to some sharing snafus with automation, please request
>> > access
>> > >> to
>> > >> > > > > comment. If you are just reading I've published this here:
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> >
>> https://docs.google.com/document/d/e/2PACX-1vThXkhHNozn_p1ZZWF-nCzOtoP1lKmkaV4Legq2FaRiIgwyY2XC9AmKpBtpeF8jbBB4wfjmQ6UTg03k/pub
>> > >> > > > >
>> > >> > > > > On Fri, Jun 21, 2024 at 10:29 AM Alkis Evlogimenos <
>> > >> > > > > alkis.evlogimenos@databricks.com> wrote:
>> > >> > > > >
>> > >> > > > >> Hey folks.
>> > >> > > > >>
>> > >> > > > >> I want to move the extension PR
>> > >> > > > >> <https://github.com/apache/parquet-format/pull/254>
>> forward.
>> > >> > > > >> Unfortunately the discussion was spread across the PR, other
>> > >> threads
>> > >> > > and
>> > >> > > > >> documents making it slow to progress. To avoid further
>> > >> > fragmentation I
>> > >> > > > have
>> > >> > > > >> put together a document
>> > >> > > > >> <
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> >
>> https://docs.google.com/document/d/1KkoR0DjzYnLQXO-d0oRBv2k157IZU0_injqd4eV4WiI/edit
>> > >> > > > >
>> > >> > > > >> discussing the extensions mechanism in isolation. I believe
>> the
>> > >> > > document
>> > >> > > > >> addresses all the concerns/comments from the PR and mailing
>> > list
>> > >> > > > >> discussions brought forward so far.
>> > >> > > > >>
>> > >> > > > >> I propose we continue the discussion in the document and
>> once
>> > >> > > everything
>> > >> > > > >> is addressed, we finalize the PR.
>> > >> > > > >>
>> > >> > > > >> Thank you,
>> > >> > > > >>
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> > >
>> >
>>
>
Re: [DISCUSS] Parquet extensions
Posted by Julien Le Dem <ju...@apache.org>.
I am also OK with the proposed solution in the document.
However I think the doc itself needs one last wording change.
I have left more details in comments but here is the gist:
This effort is driven by a group of people in the community and not one
vendor in particular even if said people do sometimes work for vendors.
To reflect this, instead of saying the UUID identifies a Vendor, we should
describe it as an extension ID.
Then I'd remove all instances of the word "Vendor" and instead
refer to "Extensions" identified by this UUID.
This might not change anything to the implementation but it is important to
reflecting how the community works in the document.
Specifically:
"Vendor introduces a Flatbuffers variant of FileMetaData." => "This
extension introduces a Flatbuffers variant of FileMetaData..."
"The UUID is picked by the Vendor once and used throughout the
experiments." => "The UUID is picked for this specific extension and used
throughout the experiments."
"At some point Vendor decides that this is amazing and should be shared
with the world at large to advance Parquet. " => "At some point, the
community decides this extension is ready and proposed for inclusion."
On Mon, Jul 22, 2024 at 10:11 PM Micah Kornfield <em...@gmail.com>
wrote:
> Hi Alkis,
> Thanks for the revision. I'm OK with this as is, we can maybe wait a few
> more days to see if anybody else has comments and then discuss
> implementation of the extension mechanism?
>
> Cheers,
> Micah
>
> On Thu, Jul 18, 2024 at 10:22 PM Alkis Evlogimenos
> <al...@databricks.com.invalid> wrote:
>
> > After Jul 17th's Parquet Sync feedback I have updated the extensions
> > proposal to remove the "reservation" mechanism. The updates are already
> > reflected in the document
> > <
> >
> https://docs.google.com/document/d/1KkoR0DjzYnLQXO-d0oRBv2k157IZU0_injqd4eV4WiI/edit
> > >
> > and
> > the PR <https://github.com/apache/parquet-format/pull/254>.
> >
> > On Fri, Jun 28, 2024 at 10:02 AM Alkis Evlogimenos <
> > alkis.evlogimenos@databricks.com> wrote:
> >
> > > > I think we can at least have wording to encourage people doing
> > > extensions to post them publicly and as part of the "reservation"
> > mechanism
> > > post a link the repo that they are being developed in, if anyone is
> > curious.
> > >
> > > Good point. I will try to come up with something in the PR - unless you
> > > beat me to it :)
> > >
> > > On Fri, Jun 28, 2024 at 7:15 AM Micah Kornfield <emkornfield@gmail.com
> >
> > > wrote:
> > >
> > >> >
> > >> > 1. experimentation/prototyping is more often than not faster to
> > iterate
> > >> if
> > >> > it is closed. Allowing this model of development was a primary goal
> of
> > >> the
> > >> > design.
> > >>
> > >>
> > >> I agree there are advantages here. I think a large amount of speed
> > comes
> > >> from not having to gain consensus in the community.
> > >>
> > >> At the end of the day, I don't think there is any mechanism here to
> > ensure
> > >> everybody works in public, but I think we can at least have wording to
> > >> encourage people doing extensions to post them publicly and as part of
> > the
> > >> "reservation" mechanism post a link the repo that they are being
> > developed
> > >> in, if anyone is curious. I think this would be particularly useful
> if
> > >> there really is an intent for a number of organizations to experiment
> > with
> > >> new footer designs (but possibly also in others).
> > >>
> > >> Thanks,
> > >> Micah
> > >>
> > >>
> > >>
> > >>
> > >> On Wed, Jun 26, 2024 at 9:33 AM Alkis Evlogimenos
> > >> <al...@databricks.com.invalid> wrote:
> > >>
> > >> > Thank you for taking a look Micah.
> > >> >
> > >> > On the topic of openness there are various aspects that we have
> > >> considered.
> > >> > 1. experimentation/prototyping is more often than not faster to
> > iterate
> > >> if
> > >> > it is closed. Allowing this model of development was a primary goal
> of
> > >> the
> > >> > design.
> > >> > 2. when the design is final, keeping the design closed should have
> > some
> > >> > drawbacks. Duplicating content to support old readers puts some
> > natural
> > >> > incentive to make extensions official because at that point one can
> > drop
> > >> > the fat from the files and move on. Another aspect of the design is
> > the
> > >> > choice of a single extension field-id which makes the extension
> space
> > >> tiny.
> > >> > This in turn means that it is difficult to interop with others
> without
> > >> > breaking their extensions. Ergo the easiest path to any interop is
> to
> > >> open
> > >> > the extension.
> > >> >
> > >> > The above, while not enforcing work to happen in the open, strike
> some
> > >> > balance in between.
> > >> >
> > >> > I am open to suggestions on how to further incentivize opening
> > >> extensions.
> > >> >
> > >> > On Wed, Jun 26, 2024 at 6:04 PM Micah Kornfield <
> > emkornfield@gmail.com>
> > >> > wrote:
> > >> >
> > >> > > Hi Alkis,
> > >> > > I'm generally in favor of this, my main concern/question is trying
> > to
> > >> > > encourage work to be in the open. I don't think in the long run
> it
> > is
> > >> > good
> > >> > > for users to always have proprietary extensions inside of Parquet.
> > >> > >
> > >> > > IMO, I think the next steps would be to add implementations to
> write
> > >> out
> > >> > > the footer extension points.
> > >> > >
> > >> > > Thanks,
> > >> > > Micah
> > >> > >
> > >> > > On Mon, Jun 24, 2024 at 1:24 PM Alkis Evlogimenos
> > >> > > <al...@databricks.com.invalid> wrote:
> > >> > >
> > >> > > > The snafus are fixed. The original should work now.
> > >> > > >
> > >> > > > On Sun, 23 Jun 2024, 17:58 Alkis Evlogimenos, <
> > >> > > > alkis.evlogimenos@databricks.com> wrote:
> > >> > > >
> > >> > > > > Due to some sharing snafus with automation, please request
> > access
> > >> to
> > >> > > > > comment. If you are just reading I've published this here:
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://docs.google.com/document/d/e/2PACX-1vThXkhHNozn_p1ZZWF-nCzOtoP1lKmkaV4Legq2FaRiIgwyY2XC9AmKpBtpeF8jbBB4wfjmQ6UTg03k/pub
> > >> > > > >
> > >> > > > > On Fri, Jun 21, 2024 at 10:29 AM Alkis Evlogimenos <
> > >> > > > > alkis.evlogimenos@databricks.com> wrote:
> > >> > > > >
> > >> > > > >> Hey folks.
> > >> > > > >>
> > >> > > > >> I want to move the extension PR
> > >> > > > >> <https://github.com/apache/parquet-format/pull/254> forward.
> > >> > > > >> Unfortunately the discussion was spread across the PR, other
> > >> threads
> > >> > > and
> > >> > > > >> documents making it slow to progress. To avoid further
> > >> > fragmentation I
> > >> > > > have
> > >> > > > >> put together a document
> > >> > > > >> <
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://docs.google.com/document/d/1KkoR0DjzYnLQXO-d0oRBv2k157IZU0_injqd4eV4WiI/edit
> > >> > > > >
> > >> > > > >> discussing the extensions mechanism in isolation. I believe
> the
> > >> > > document
> > >> > > > >> addresses all the concerns/comments from the PR and mailing
> > list
> > >> > > > >> discussions brought forward so far.
> > >> > > > >>
> > >> > > > >> I propose we continue the discussion in the document and once
> > >> > > everything
> > >> > > > >> is addressed, we finalize the PR.
> > >> > > > >>
> > >> > > > >> Thank you,
> > >> > > > >>
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>
Re: [DISCUSS] Parquet extensions
Posted by Micah Kornfield <em...@gmail.com>.
Hi Alkis,
Thanks for the revision. I'm OK with this as is, we can maybe wait a few
more days to see if anybody else has comments and then discuss
implementation of the extension mechanism?
Cheers,
Micah
On Thu, Jul 18, 2024 at 10:22 PM Alkis Evlogimenos
<al...@databricks.com.invalid> wrote:
> After Jul 17th's Parquet Sync feedback I have updated the extensions
> proposal to remove the "reservation" mechanism. The updates are already
> reflected in the document
> <
> https://docs.google.com/document/d/1KkoR0DjzYnLQXO-d0oRBv2k157IZU0_injqd4eV4WiI/edit
> >
> and
> the PR <https://github.com/apache/parquet-format/pull/254>.
>
> On Fri, Jun 28, 2024 at 10:02 AM Alkis Evlogimenos <
> alkis.evlogimenos@databricks.com> wrote:
>
> > > I think we can at least have wording to encourage people doing
> > extensions to post them publicly and as part of the "reservation"
> mechanism
> > post a link the repo that they are being developed in, if anyone is
> curious.
> >
> > Good point. I will try to come up with something in the PR - unless you
> > beat me to it :)
> >
> > On Fri, Jun 28, 2024 at 7:15 AM Micah Kornfield <em...@gmail.com>
> > wrote:
> >
> >> >
> >> > 1. experimentation/prototyping is more often than not faster to
> iterate
> >> if
> >> > it is closed. Allowing this model of development was a primary goal of
> >> the
> >> > design.
> >>
> >>
> >> I agree there are advantages here. I think a large amount of speed
> comes
> >> from not having to gain consensus in the community.
> >>
> >> At the end of the day, I don't think there is any mechanism here to
> ensure
> >> everybody works in public, but I think we can at least have wording to
> >> encourage people doing extensions to post them publicly and as part of
> the
> >> "reservation" mechanism post a link the repo that they are being
> developed
> >> in, if anyone is curious. I think this would be particularly useful if
> >> there really is an intent for a number of organizations to experiment
> with
> >> new footer designs (but possibly also in others).
> >>
> >> Thanks,
> >> Micah
> >>
> >>
> >>
> >>
> >> On Wed, Jun 26, 2024 at 9:33 AM Alkis Evlogimenos
> >> <al...@databricks.com.invalid> wrote:
> >>
> >> > Thank you for taking a look Micah.
> >> >
> >> > On the topic of openness there are various aspects that we have
> >> considered.
> >> > 1. experimentation/prototyping is more often than not faster to
> iterate
> >> if
> >> > it is closed. Allowing this model of development was a primary goal of
> >> the
> >> > design.
> >> > 2. when the design is final, keeping the design closed should have
> some
> >> > drawbacks. Duplicating content to support old readers puts some
> natural
> >> > incentive to make extensions official because at that point one can
> drop
> >> > the fat from the files and move on. Another aspect of the design is
> the
> >> > choice of a single extension field-id which makes the extension space
> >> tiny.
> >> > This in turn means that it is difficult to interop with others without
> >> > breaking their extensions. Ergo the easiest path to any interop is to
> >> open
> >> > the extension.
> >> >
> >> > The above, while not enforcing work to happen in the open, strike some
> >> > balance in between.
> >> >
> >> > I am open to suggestions on how to further incentivize opening
> >> extensions.
> >> >
> >> > On Wed, Jun 26, 2024 at 6:04 PM Micah Kornfield <
> emkornfield@gmail.com>
> >> > wrote:
> >> >
> >> > > Hi Alkis,
> >> > > I'm generally in favor of this, my main concern/question is trying
> to
> >> > > encourage work to be in the open. I don't think in the long run it
> is
> >> > good
> >> > > for users to always have proprietary extensions inside of Parquet.
> >> > >
> >> > > IMO, I think the next steps would be to add implementations to write
> >> out
> >> > > the footer extension points.
> >> > >
> >> > > Thanks,
> >> > > Micah
> >> > >
> >> > > On Mon, Jun 24, 2024 at 1:24 PM Alkis Evlogimenos
> >> > > <al...@databricks.com.invalid> wrote:
> >> > >
> >> > > > The snafus are fixed. The original should work now.
> >> > > >
> >> > > > On Sun, 23 Jun 2024, 17:58 Alkis Evlogimenos, <
> >> > > > alkis.evlogimenos@databricks.com> wrote:
> >> > > >
> >> > > > > Due to some sharing snafus with automation, please request
> access
> >> to
> >> > > > > comment. If you are just reading I've published this here:
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://docs.google.com/document/d/e/2PACX-1vThXkhHNozn_p1ZZWF-nCzOtoP1lKmkaV4Legq2FaRiIgwyY2XC9AmKpBtpeF8jbBB4wfjmQ6UTg03k/pub
> >> > > > >
> >> > > > > On Fri, Jun 21, 2024 at 10:29 AM Alkis Evlogimenos <
> >> > > > > alkis.evlogimenos@databricks.com> wrote:
> >> > > > >
> >> > > > >> Hey folks.
> >> > > > >>
> >> > > > >> I want to move the extension PR
> >> > > > >> <https://github.com/apache/parquet-format/pull/254> forward.
> >> > > > >> Unfortunately the discussion was spread across the PR, other
> >> threads
> >> > > and
> >> > > > >> documents making it slow to progress. To avoid further
> >> > fragmentation I
> >> > > > have
> >> > > > >> put together a document
> >> > > > >> <
> >> > > >
> >> > >
> >> >
> >>
> https://docs.google.com/document/d/1KkoR0DjzYnLQXO-d0oRBv2k157IZU0_injqd4eV4WiI/edit
> >> > > > >
> >> > > > >> discussing the extensions mechanism in isolation. I believe the
> >> > > document
> >> > > > >> addresses all the concerns/comments from the PR and mailing
> list
> >> > > > >> discussions brought forward so far.
> >> > > > >>
> >> > > > >> I propose we continue the discussion in the document and once
> >> > > everything
> >> > > > >> is addressed, we finalize the PR.
> >> > > > >>
> >> > > > >> Thank you,
> >> > > > >>
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>