You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by Andrew Lamb <an...@gmail.com> on 2024/07/04 09:31:38 UTC
Re: [DISCUSS] Merge initial Implementation Status PR and incrementally improve it
The initial feature matrix PR[1], has all outstanding comments addressed
and is ready to merge from my opinion.
Could one of the committers consider merging it so we can start filling the
matrix out?
Thanks,
Andrew
[1]: https://github.com/apache/parquet-site/pull/34
On Wed, Jun 26, 2024 at 11:22 AM Andrew Lamb <an...@gmail.com> wrote:
> I filed a ticket [1] to discuss integration testing / feature
> compatibility testing
>
> [1]: https://github.com/apache/parquet-format/issues/441
>
> On Wed, Jun 26, 2024 at 10:03 AM Andrew Lamb <an...@gmail.com>
> wrote:
>
>> I will file a ticket for discussion shortly
>>
>> On Wed, Jun 26, 2024, 06:55 Julien Le Dem <ju...@apache.org> wrote:
>>
>>> I agree with Antoine for the separate thread/ticket for integration
>>> testing
>>>
>>> For the « implementation status » PR, I agree we can merge and iterate. I
>>> added a couple of comments that can easily be addressed afterwards.
>>> Julien
>>>
>>> On Wed, Jun 26, 2024 at 10:43 Antoine Pitrou <an...@python.org> wrote:
>>>
>>> >
>>> > IMHO, we should either start a dedicated discussion thread for
>>> > integration testing, or open a GH issue and discuss it there.
>>> >
>>> > Regards
>>> >
>>> > Antoine.
>>> >
>>> >
>>> >
>>> > On Wed, 26 Jun 2024 09:21:33 +0200
>>> > Alkis Evlogimenos
>>> > <al...@databricks.com.INVALID>
>>> > wrote:
>>> > > It would be nice if the integration suite specifies how a driver can
>>> be
>>> > > executed. Then each implementation can provide a driver and the suite
>>> > will
>>> > > use that for validation.
>>> > >
>>> > > By specifying both reads and writes for the driver we get a lot more
>>> > power.
>>> > > Given an input we can roundtrip all combination of readers/writers
>>> and
>>> > > verify they can roundtrip.
>>> > >
>>> > > On Tue, Jun 25, 2024 at 6:42 PM Andrew Lamb <
>>> > andrewlamb11-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>> > >
>>> > > > FWIW I started hacking up a prototype[1] of what a parquet-testing
>>> > > > integration suite might look like if anyone is interested
>>> > > >
>>> > > >
>>> > > >
>>> > > > [1]: https://github.com/apache/arrow-rs/pull/5956
>>> > > >
>>> > > > On Tue, Jun 18, 2024 at 10:39 AM Alkis Evlogimenos
>>> > > > <al...@databricks.com.invalid> wrote:
>>> > > >
>>> > > > > +1.
>>> > > > >
>>> > > > > I would suggest you address the comments first? I went through
>>> the
>>> > open
>>> > > > > ones and most of them make sense to me (and left few additional
>>> > > > comments).
>>> > > > >
>>> > > > > On Tue, Jun 18, 2024 at 12:42 PM Andrew Lamb <
>>> andrewlamb11@gmail.com
>>> > >
>>> > > > > wrote:
>>> > > > >
>>> > > > > > Thank you
>>> > > > > >
>>> > > > > > On Mon, Jun 17, 2024 at 11:40 PM Micah Kornfield <
>>> > > > emkornfield@gmail.com>
>>> > > > > > wrote:
>>> > > > > >
>>> > > > > > > Hi Andrew,
>>> > > > > > > I agree with this sentiment, I asked on the PR if there would
>>> > be
>>> > > > > another
>>> > > > > > > pass and then I can merge it.
>>> > > > > > >
>>> > > > > > > Cheers,
>>> > > > > > > Micah
>>> > > > > > >
>>> > > > > > > On Fri, Jun 14, 2024 at 3:20 AM Andrew Lamb <
>>> > andrewlamb11@gmail.com>
>>> > > > > > > wrote:
>>> > > > > > >
>>> > > > > > > > Hello Parquet Devs,
>>> > > > > > > >
>>> > > > > > > > I propose we merge the first (admittedly bare bones)
>>> > > > "Implementation
>>> > > > > > > > Status" page PR [1] to the website soon. I think this page
>>> is
>>> > vital
>>> > > > > to
>>> > > > > > > the
>>> > > > > > > > Parquet community (and to any attempt to extend the
>>> format) so
>>> > the
>>> > > > > > sooner
>>> > > > > > > > the better.
>>> > > > > > > >
>>> > > > > > > > The reason to merge the PR now is to have a base from which
>>> > to
>>> > > > build.
>>> > > > > > > That
>>> > > > > > > > PR is already over a year old and has so many comments it
>>> is
>>> > hard
>>> > > > to
>>> > > > > > > follow
>>> > > > > > > > or know what the path forward is. If we insist on sorting
>>> all
>>> > the
>>> > > > > > details
>>> > > > > > > > out before we merge it I fear it will never merge.
>>> > > > > > > >
>>> > > > > > > > Once we have a page, I think the next steps are to add a
>>> > preamble
>>> > > > > > > > explaining what it is for and to start trying to fill out
>>> the
>>> > chart
>>> > > > > for
>>> > > > > > > an
>>> > > > > > > > implementation (I am happy to try for parquet-rs). I
>>> suspect
>>> > during
>>> > > > > > that
>>> > > > > > > > process we will have to adjust some of the charts more.
>>> > > > > > > >
>>> > > > > > > > Thank you for your consideration (and thank you for all the
>>> > > > comments
>>> > > > > so
>>> > > > > > > > far)
>>> > > > > > > >
>>> > > > > > > > Andrew
>>> > > > > > > >
>>> > > > > > > > [1]: https://github.com/apache/parquet-site/pull/34
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> >
>>> >
>>> >
>>>
>>
Re: [DISCUSS] Merge initial Implementation Status PR and incrementally improve it
Posted by Andrew Lamb <an...@gmail.com>.
A brief update here:
The current implementation status page[1] is looking quite nice. I think
the amount of green on that page illustrates visually that despite rumors
to the contrary the state of Parquet compatibility is actually good.
Thanks to everyone who has contributed status for their implementations
[2][3][4] and more!
Andrew
[1]: https://parquet.apache.org/docs/file-format/implementationstatus/
[2]: https://github.com/apache/parquet-site/pull/104
[3]: https://github.com/apache/parquet-site/pull/101
[4]: https://github.com/apache/parquet-site/pull/100
On Fri, Feb 7, 2025 at 8:57 AM Andrew Lamb <an...@gmail.com> wrote:
> Thank you!. Once that gets in I will find time (or really try and find
> someone else to find time) to fill it out for parquet-rs
>
> On Thu, Feb 6, 2025 at 12:04 AM Gang Wu <us...@gmail.com> wrote:
>
>> Just a heads-up that I've created a PR [1] to update the matrix
>> for parquet-cpp and parquet-java. Let me know if you have any
>> suggestion. Thanks!
>>
>> [1] https://github.com/apache/parquet-site/pull/100
>>
>>
>> On Tue, Feb 4, 2025 at 10:11 PM Andrew Lamb <an...@gmail.com>
>> wrote:
>>
>> > I wanted to highlight that there is a PR[1] to fill out this matrix for
>> > cuda . Super exciting to see this starting
>> >
>> > [1] https://github.com/apache/parquet-site/pull/99#
>> >
>> > On Thu, Jul 4, 2024 at 2:57 PM Andrew Lamb <an...@gmail.com>
>> wrote:
>> >
>> > > Thank you! A small step perhaps but a major step forward
>> > >
>> > > On Thu, Jul 4, 2024 at 1:03 PM Micah Kornfield <emkornfield@gmail.com
>> >
>> > > wrote:
>> > >
>> > >> I just merged it.
>> > >>
>> > >> Cheers,
>> > >> Micah
>> > >>
>> > >> On Thu, Jul 4, 2024 at 2:32 AM Andrew Lamb <an...@gmail.com>
>> > >> wrote:
>> > >>
>> > >> > The initial feature matrix PR[1], has all outstanding comments
>> > addressed
>> > >> > and is ready to merge from my opinion.
>> > >> >
>> > >> > Could one of the committers consider merging it so we can start
>> > filling
>> > >> the
>> > >> > matrix out?
>> > >> >
>> > >> > Thanks,
>> > >> > Andrew
>> > >> >
>> > >> > [1]: https://github.com/apache/parquet-site/pull/34
>> > >> >
>> > >> >
>> > >> > On Wed, Jun 26, 2024 at 11:22 AM Andrew Lamb <
>> andrewlamb11@gmail.com>
>> > >> > wrote:
>> > >> >
>> > >> > > I filed a ticket [1] to discuss integration testing / feature
>> > >> > > compatibility testing
>> > >> > >
>> > >> > > [1]: https://github.com/apache/parquet-format/issues/441
>> > >> > >
>> > >> > > On Wed, Jun 26, 2024 at 10:03 AM Andrew Lamb <
>> > andrewlamb11@gmail.com>
>> > >> > > wrote:
>> > >> > >
>> > >> > >> I will file a ticket for discussion shortly
>> > >> > >>
>> > >> > >> On Wed, Jun 26, 2024, 06:55 Julien Le Dem <ju...@apache.org>
>> > wrote:
>> > >> > >>
>> > >> > >>> I agree with Antoine for the separate thread/ticket for
>> > integration
>> > >> > >>> testing
>> > >> > >>>
>> > >> > >>> For the « implementation status » PR, I agree we can merge and
>> > >> > iterate. I
>> > >> > >>> added a couple of comments that can easily be addressed
>> > afterwards.
>> > >> > >>> Julien
>> > >> > >>>
>> > >> > >>> On Wed, Jun 26, 2024 at 10:43 Antoine Pitrou <
>> antoine@python.org>
>> > >> > wrote:
>> > >> > >>>
>> > >> > >>> >
>> > >> > >>> > IMHO, we should either start a dedicated discussion thread
>> for
>> > >> > >>> > integration testing, or open a GH issue and discuss it there.
>> > >> > >>> >
>> > >> > >>> > Regards
>> > >> > >>> >
>> > >> > >>> > Antoine.
>> > >> > >>> >
>> > >> > >>> >
>> > >> > >>> >
>> > >> > >>> > On Wed, 26 Jun 2024 09:21:33 +0200
>> > >> > >>> > Alkis Evlogimenos
>> > >> > >>> > <al...@databricks.com.INVALID>
>> > >> > >>> > wrote:
>> > >> > >>> > > It would be nice if the integration suite specifies how a
>> > driver
>> > >> > can
>> > >> > >>> be
>> > >> > >>> > > executed. Then each implementation can provide a driver and
>> > the
>> > >> > suite
>> > >> > >>> > will
>> > >> > >>> > > use that for validation.
>> > >> > >>> > >
>> > >> > >>> > > By specifying both reads and writes for the driver we get a
>> > lot
>> > >> > more
>> > >> > >>> > power.
>> > >> > >>> > > Given an input we can roundtrip all combination of
>> > >> readers/writers
>> > >> > >>> and
>> > >> > >>> > > verify they can roundtrip.
>> > >> > >>> > >
>> > >> > >>> > > On Tue, Jun 25, 2024 at 6:42 PM Andrew Lamb <
>> > >> > >>> > andrewlamb11-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> > >> > >>> > >
>> > >> > >>> > > > FWIW I started hacking up a prototype[1] of what a
>> > >> > parquet-testing
>> > >> > >>> > > > integration suite might look like if anyone is interested
>> > >> > >>> > > >
>> > >> > >>> > > >
>> > >> > >>> > > >
>> > >> > >>> > > > [1]: https://github.com/apache/arrow-rs/pull/5956
>> > >> > >>> > > >
>> > >> > >>> > > > On Tue, Jun 18, 2024 at 10:39 AM Alkis Evlogimenos
>> > >> > >>> > > > <al...@databricks.com.invalid> wrote:
>> > >> > >>> > > >
>> > >> > >>> > > > > +1.
>> > >> > >>> > > > >
>> > >> > >>> > > > > I would suggest you address the comments first? I went
>> > >> through
>> > >> > >>> the
>> > >> > >>> > open
>> > >> > >>> > > > > ones and most of them make sense to me (and left few
>> > >> additional
>> > >> > >>> > > > comments).
>> > >> > >>> > > > >
>> > >> > >>> > > > > On Tue, Jun 18, 2024 at 12:42 PM Andrew Lamb <
>> > >> > >>> andrewlamb11@gmail.com
>> > >> > >>> > >
>> > >> > >>> > > > > wrote:
>> > >> > >>> > > > >
>> > >> > >>> > > > > > Thank you
>> > >> > >>> > > > > >
>> > >> > >>> > > > > > On Mon, Jun 17, 2024 at 11:40 PM Micah Kornfield <
>> > >> > >>> > > > emkornfield@gmail.com>
>> > >> > >>> > > > > > wrote:
>> > >> > >>> > > > > >
>> > >> > >>> > > > > > > Hi Andrew,
>> > >> > >>> > > > > > > I agree with this sentiment, I asked on the PR if
>> > there
>> > >> > would
>> > >> > >>> > be
>> > >> > >>> > > > > another
>> > >> > >>> > > > > > > pass and then I can merge it.
>> > >> > >>> > > > > > >
>> > >> > >>> > > > > > > Cheers,
>> > >> > >>> > > > > > > Micah
>> > >> > >>> > > > > > >
>> > >> > >>> > > > > > > On Fri, Jun 14, 2024 at 3:20 AM Andrew Lamb <
>> > >> > >>> > andrewlamb11@gmail.com>
>> > >> > >>> > > > > > > wrote:
>> > >> > >>> > > > > > >
>> > >> > >>> > > > > > > > Hello Parquet Devs,
>> > >> > >>> > > > > > > >
>> > >> > >>> > > > > > > > I propose we merge the first (admittedly bare
>> bones)
>> > >> > >>> > > > "Implementation
>> > >> > >>> > > > > > > > Status" page PR [1] to the website soon. I think
>> > this
>> > >> > page
>> > >> > >>> is
>> > >> > >>> > vital
>> > >> > >>> > > > > to
>> > >> > >>> > > > > > > the
>> > >> > >>> > > > > > > > Parquet community (and to any attempt to extend
>> the
>> > >> > >>> format) so
>> > >> > >>> > the
>> > >> > >>> > > > > > sooner
>> > >> > >>> > > > > > > > the better.
>> > >> > >>> > > > > > > >
>> > >> > >>> > > > > > > > The reason to merge the PR now is to have a base
>> > from
>> > >> > which
>> > >> > >>> > to
>> > >> > >>> > > > build.
>> > >> > >>> > > > > > > That
>> > >> > >>> > > > > > > > PR is already over a year old and has so many
>> > >> comments it
>> > >> > >>> is
>> > >> > >>> > hard
>> > >> > >>> > > > to
>> > >> > >>> > > > > > > follow
>> > >> > >>> > > > > > > > or know what the path forward is. If we insist on
>> > >> sorting
>> > >> > >>> all
>> > >> > >>> > the
>> > >> > >>> > > > > > details
>> > >> > >>> > > > > > > > out before we merge it I fear it will never
>> merge.
>> > >> > >>> > > > > > > >
>> > >> > >>> > > > > > > > Once we have a page, I think the next steps are
>> to
>> > >> add a
>> > >> > >>> > preamble
>> > >> > >>> > > > > > > > explaining what it is for and to start trying to
>> > fill
>> > >> out
>> > >> > >>> the
>> > >> > >>> > chart
>> > >> > >>> > > > > for
>> > >> > >>> > > > > > > an
>> > >> > >>> > > > > > > > implementation (I am happy to try for
>> parquet-rs). I
>> > >> > >>> suspect
>> > >> > >>> > during
>> > >> > >>> > > > > > that
>> > >> > >>> > > > > > > > process we will have to adjust some of the charts
>> > >> more.
>> > >> > >>> > > > > > > >
>> > >> > >>> > > > > > > > Thank you for your consideration (and thank you
>> for
>> > >> all
>> > >> > the
>> > >> > >>> > > > comments
>> > >> > >>> > > > > so
>> > >> > >>> > > > > > > > far)
>> > >> > >>> > > > > > > >
>> > >> > >>> > > > > > > > Andrew
>> > >> > >>> > > > > > > >
>> > >> > >>> > > > > > > > [1]:
>> https://github.com/apache/parquet-site/pull/34
>> > >> > >>> > > > > > > >
>> > >> > >>> > > > > > >
>> > >> > >>> > > > > >
>> > >> > >>> > > > >
>> > >> > >>> > > >
>> > >> > >>> > >
>> > >> > >>> >
>> > >> > >>> >
>> > >> > >>> >
>> > >> > >>> >
>> > >> > >>>
>> > >> > >>
>> > >> >
>> > >>
>> > >
>> >
>>
>
Re: [DISCUSS] Merge initial Implementation Status PR and incrementally improve it
Posted by Andrew Lamb <an...@gmail.com>.
A brief update here:
The current implementation status page[1] is looking quite nice. I think
the amount of green on that page illustrates visually that despite rumors
to the contrary the state of Parquet compatibility is actually good.
Thanks to everyone who has contributed status for their implementations
[2][3][4] and more!
Andrew
[1]: https://parquet.apache.org/docs/file-format/implementationstatus/
[2]: https://github.com/apache/parquet-site/pull/104
[3]: https://github.com/apache/parquet-site/pull/101
[4]: https://github.com/apache/parquet-site/pull/100
On Fri, Feb 7, 2025 at 8:57 AM Andrew Lamb <an...@gmail.com> wrote:
> Thank you!. Once that gets in I will find time (or really try and find
> someone else to find time) to fill it out for parquet-rs
>
> On Thu, Feb 6, 2025 at 12:04 AM Gang Wu <us...@gmail.com> wrote:
>
>> Just a heads-up that I've created a PR [1] to update the matrix
>> for parquet-cpp and parquet-java. Let me know if you have any
>> suggestion. Thanks!
>>
>> [1] https://github.com/apache/parquet-site/pull/100
>>
>>
>> On Tue, Feb 4, 2025 at 10:11 PM Andrew Lamb <an...@gmail.com>
>> wrote:
>>
>> > I wanted to highlight that there is a PR[1] to fill out this matrix for
>> > cuda . Super exciting to see this starting
>> >
>> > [1] https://github.com/apache/parquet-site/pull/99#
>> >
>> > On Thu, Jul 4, 2024 at 2:57 PM Andrew Lamb <an...@gmail.com>
>> wrote:
>> >
>> > > Thank you! A small step perhaps but a major step forward
>> > >
>> > > On Thu, Jul 4, 2024 at 1:03 PM Micah Kornfield <emkornfield@gmail.com
>> >
>> > > wrote:
>> > >
>> > >> I just merged it.
>> > >>
>> > >> Cheers,
>> > >> Micah
>> > >>
>> > >> On Thu, Jul 4, 2024 at 2:32 AM Andrew Lamb <an...@gmail.com>
>> > >> wrote:
>> > >>
>> > >> > The initial feature matrix PR[1], has all outstanding comments
>> > addressed
>> > >> > and is ready to merge from my opinion.
>> > >> >
>> > >> > Could one of the committers consider merging it so we can start
>> > filling
>> > >> the
>> > >> > matrix out?
>> > >> >
>> > >> > Thanks,
>> > >> > Andrew
>> > >> >
>> > >> > [1]: https://github.com/apache/parquet-site/pull/34
>> > >> >
>> > >> >
>> > >> > On Wed, Jun 26, 2024 at 11:22 AM Andrew Lamb <
>> andrewlamb11@gmail.com>
>> > >> > wrote:
>> > >> >
>> > >> > > I filed a ticket [1] to discuss integration testing / feature
>> > >> > > compatibility testing
>> > >> > >
>> > >> > > [1]: https://github.com/apache/parquet-format/issues/441
>> > >> > >
>> > >> > > On Wed, Jun 26, 2024 at 10:03 AM Andrew Lamb <
>> > andrewlamb11@gmail.com>
>> > >> > > wrote:
>> > >> > >
>> > >> > >> I will file a ticket for discussion shortly
>> > >> > >>
>> > >> > >> On Wed, Jun 26, 2024, 06:55 Julien Le Dem <ju...@apache.org>
>> > wrote:
>> > >> > >>
>> > >> > >>> I agree with Antoine for the separate thread/ticket for
>> > integration
>> > >> > >>> testing
>> > >> > >>>
>> > >> > >>> For the « implementation status » PR, I agree we can merge and
>> > >> > iterate. I
>> > >> > >>> added a couple of comments that can easily be addressed
>> > afterwards.
>> > >> > >>> Julien
>> > >> > >>>
>> > >> > >>> On Wed, Jun 26, 2024 at 10:43 Antoine Pitrou <
>> antoine@python.org>
>> > >> > wrote:
>> > >> > >>>
>> > >> > >>> >
>> > >> > >>> > IMHO, we should either start a dedicated discussion thread
>> for
>> > >> > >>> > integration testing, or open a GH issue and discuss it there.
>> > >> > >>> >
>> > >> > >>> > Regards
>> > >> > >>> >
>> > >> > >>> > Antoine.
>> > >> > >>> >
>> > >> > >>> >
>> > >> > >>> >
>> > >> > >>> > On Wed, 26 Jun 2024 09:21:33 +0200
>> > >> > >>> > Alkis Evlogimenos
>> > >> > >>> > <al...@databricks.com.INVALID>
>> > >> > >>> > wrote:
>> > >> > >>> > > It would be nice if the integration suite specifies how a
>> > driver
>> > >> > can
>> > >> > >>> be
>> > >> > >>> > > executed. Then each implementation can provide a driver and
>> > the
>> > >> > suite
>> > >> > >>> > will
>> > >> > >>> > > use that for validation.
>> > >> > >>> > >
>> > >> > >>> > > By specifying both reads and writes for the driver we get a
>> > lot
>> > >> > more
>> > >> > >>> > power.
>> > >> > >>> > > Given an input we can roundtrip all combination of
>> > >> readers/writers
>> > >> > >>> and
>> > >> > >>> > > verify they can roundtrip.
>> > >> > >>> > >
>> > >> > >>> > > On Tue, Jun 25, 2024 at 6:42 PM Andrew Lamb <
>> > >> > >>> > andrewlamb11-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> > >> > >>> > >
>> > >> > >>> > > > FWIW I started hacking up a prototype[1] of what a
>> > >> > parquet-testing
>> > >> > >>> > > > integration suite might look like if anyone is interested
>> > >> > >>> > > >
>> > >> > >>> > > >
>> > >> > >>> > > >
>> > >> > >>> > > > [1]: https://github.com/apache/arrow-rs/pull/5956
>> > >> > >>> > > >
>> > >> > >>> > > > On Tue, Jun 18, 2024 at 10:39 AM Alkis Evlogimenos
>> > >> > >>> > > > <al...@databricks.com.invalid> wrote:
>> > >> > >>> > > >
>> > >> > >>> > > > > +1.
>> > >> > >>> > > > >
>> > >> > >>> > > > > I would suggest you address the comments first? I went
>> > >> through
>> > >> > >>> the
>> > >> > >>> > open
>> > >> > >>> > > > > ones and most of them make sense to me (and left few
>> > >> additional
>> > >> > >>> > > > comments).
>> > >> > >>> > > > >
>> > >> > >>> > > > > On Tue, Jun 18, 2024 at 12:42 PM Andrew Lamb <
>> > >> > >>> andrewlamb11@gmail.com
>> > >> > >>> > >
>> > >> > >>> > > > > wrote:
>> > >> > >>> > > > >
>> > >> > >>> > > > > > Thank you
>> > >> > >>> > > > > >
>> > >> > >>> > > > > > On Mon, Jun 17, 2024 at 11:40 PM Micah Kornfield <
>> > >> > >>> > > > emkornfield@gmail.com>
>> > >> > >>> > > > > > wrote:
>> > >> > >>> > > > > >
>> > >> > >>> > > > > > > Hi Andrew,
>> > >> > >>> > > > > > > I agree with this sentiment, I asked on the PR if
>> > there
>> > >> > would
>> > >> > >>> > be
>> > >> > >>> > > > > another
>> > >> > >>> > > > > > > pass and then I can merge it.
>> > >> > >>> > > > > > >
>> > >> > >>> > > > > > > Cheers,
>> > >> > >>> > > > > > > Micah
>> > >> > >>> > > > > > >
>> > >> > >>> > > > > > > On Fri, Jun 14, 2024 at 3:20 AM Andrew Lamb <
>> > >> > >>> > andrewlamb11@gmail.com>
>> > >> > >>> > > > > > > wrote:
>> > >> > >>> > > > > > >
>> > >> > >>> > > > > > > > Hello Parquet Devs,
>> > >> > >>> > > > > > > >
>> > >> > >>> > > > > > > > I propose we merge the first (admittedly bare
>> bones)
>> > >> > >>> > > > "Implementation
>> > >> > >>> > > > > > > > Status" page PR [1] to the website soon. I think
>> > this
>> > >> > page
>> > >> > >>> is
>> > >> > >>> > vital
>> > >> > >>> > > > > to
>> > >> > >>> > > > > > > the
>> > >> > >>> > > > > > > > Parquet community (and to any attempt to extend
>> the
>> > >> > >>> format) so
>> > >> > >>> > the
>> > >> > >>> > > > > > sooner
>> > >> > >>> > > > > > > > the better.
>> > >> > >>> > > > > > > >
>> > >> > >>> > > > > > > > The reason to merge the PR now is to have a base
>> > from
>> > >> > which
>> > >> > >>> > to
>> > >> > >>> > > > build.
>> > >> > >>> > > > > > > That
>> > >> > >>> > > > > > > > PR is already over a year old and has so many
>> > >> comments it
>> > >> > >>> is
>> > >> > >>> > hard
>> > >> > >>> > > > to
>> > >> > >>> > > > > > > follow
>> > >> > >>> > > > > > > > or know what the path forward is. If we insist on
>> > >> sorting
>> > >> > >>> all
>> > >> > >>> > the
>> > >> > >>> > > > > > details
>> > >> > >>> > > > > > > > out before we merge it I fear it will never
>> merge.
>> > >> > >>> > > > > > > >
>> > >> > >>> > > > > > > > Once we have a page, I think the next steps are
>> to
>> > >> add a
>> > >> > >>> > preamble
>> > >> > >>> > > > > > > > explaining what it is for and to start trying to
>> > fill
>> > >> out
>> > >> > >>> the
>> > >> > >>> > chart
>> > >> > >>> > > > > for
>> > >> > >>> > > > > > > an
>> > >> > >>> > > > > > > > implementation (I am happy to try for
>> parquet-rs). I
>> > >> > >>> suspect
>> > >> > >>> > during
>> > >> > >>> > > > > > that
>> > >> > >>> > > > > > > > process we will have to adjust some of the charts
>> > >> more.
>> > >> > >>> > > > > > > >
>> > >> > >>> > > > > > > > Thank you for your consideration (and thank you
>> for
>> > >> all
>> > >> > the
>> > >> > >>> > > > comments
>> > >> > >>> > > > > so
>> > >> > >>> > > > > > > > far)
>> > >> > >>> > > > > > > >
>> > >> > >>> > > > > > > > Andrew
>> > >> > >>> > > > > > > >
>> > >> > >>> > > > > > > > [1]:
>> https://github.com/apache/parquet-site/pull/34
>> > >> > >>> > > > > > > >
>> > >> > >>> > > > > > >
>> > >> > >>> > > > > >
>> > >> > >>> > > > >
>> > >> > >>> > > >
>> > >> > >>> > >
>> > >> > >>> >
>> > >> > >>> >
>> > >> > >>> >
>> > >> > >>> >
>> > >> > >>>
>> > >> > >>
>> > >> >
>> > >>
>> > >
>> >
>>
>
Re: [DISCUSS] Merge initial Implementation Status PR and incrementally improve it
Posted by Andrew Lamb <an...@gmail.com>.
Thank you!. Once that gets in I will find time (or really try and find
someone else to find time) to fill it out for parquet-rs
On Thu, Feb 6, 2025 at 12:04 AM Gang Wu <us...@gmail.com> wrote:
> Just a heads-up that I've created a PR [1] to update the matrix
> for parquet-cpp and parquet-java. Let me know if you have any
> suggestion. Thanks!
>
> [1] https://github.com/apache/parquet-site/pull/100
>
>
> On Tue, Feb 4, 2025 at 10:11 PM Andrew Lamb <an...@gmail.com>
> wrote:
>
> > I wanted to highlight that there is a PR[1] to fill out this matrix for
> > cuda . Super exciting to see this starting
> >
> > [1] https://github.com/apache/parquet-site/pull/99#
> >
> > On Thu, Jul 4, 2024 at 2:57 PM Andrew Lamb <an...@gmail.com>
> wrote:
> >
> > > Thank you! A small step perhaps but a major step forward
> > >
> > > On Thu, Jul 4, 2024 at 1:03 PM Micah Kornfield <em...@gmail.com>
> > > wrote:
> > >
> > >> I just merged it.
> > >>
> > >> Cheers,
> > >> Micah
> > >>
> > >> On Thu, Jul 4, 2024 at 2:32 AM Andrew Lamb <an...@gmail.com>
> > >> wrote:
> > >>
> > >> > The initial feature matrix PR[1], has all outstanding comments
> > addressed
> > >> > and is ready to merge from my opinion.
> > >> >
> > >> > Could one of the committers consider merging it so we can start
> > filling
> > >> the
> > >> > matrix out?
> > >> >
> > >> > Thanks,
> > >> > Andrew
> > >> >
> > >> > [1]: https://github.com/apache/parquet-site/pull/34
> > >> >
> > >> >
> > >> > On Wed, Jun 26, 2024 at 11:22 AM Andrew Lamb <
> andrewlamb11@gmail.com>
> > >> > wrote:
> > >> >
> > >> > > I filed a ticket [1] to discuss integration testing / feature
> > >> > > compatibility testing
> > >> > >
> > >> > > [1]: https://github.com/apache/parquet-format/issues/441
> > >> > >
> > >> > > On Wed, Jun 26, 2024 at 10:03 AM Andrew Lamb <
> > andrewlamb11@gmail.com>
> > >> > > wrote:
> > >> > >
> > >> > >> I will file a ticket for discussion shortly
> > >> > >>
> > >> > >> On Wed, Jun 26, 2024, 06:55 Julien Le Dem <ju...@apache.org>
> > wrote:
> > >> > >>
> > >> > >>> I agree with Antoine for the separate thread/ticket for
> > integration
> > >> > >>> testing
> > >> > >>>
> > >> > >>> For the « implementation status » PR, I agree we can merge and
> > >> > iterate. I
> > >> > >>> added a couple of comments that can easily be addressed
> > afterwards.
> > >> > >>> Julien
> > >> > >>>
> > >> > >>> On Wed, Jun 26, 2024 at 10:43 Antoine Pitrou <
> antoine@python.org>
> > >> > wrote:
> > >> > >>>
> > >> > >>> >
> > >> > >>> > IMHO, we should either start a dedicated discussion thread for
> > >> > >>> > integration testing, or open a GH issue and discuss it there.
> > >> > >>> >
> > >> > >>> > Regards
> > >> > >>> >
> > >> > >>> > Antoine.
> > >> > >>> >
> > >> > >>> >
> > >> > >>> >
> > >> > >>> > On Wed, 26 Jun 2024 09:21:33 +0200
> > >> > >>> > Alkis Evlogimenos
> > >> > >>> > <al...@databricks.com.INVALID>
> > >> > >>> > wrote:
> > >> > >>> > > It would be nice if the integration suite specifies how a
> > driver
> > >> > can
> > >> > >>> be
> > >> > >>> > > executed. Then each implementation can provide a driver and
> > the
> > >> > suite
> > >> > >>> > will
> > >> > >>> > > use that for validation.
> > >> > >>> > >
> > >> > >>> > > By specifying both reads and writes for the driver we get a
> > lot
> > >> > more
> > >> > >>> > power.
> > >> > >>> > > Given an input we can roundtrip all combination of
> > >> readers/writers
> > >> > >>> and
> > >> > >>> > > verify they can roundtrip.
> > >> > >>> > >
> > >> > >>> > > On Tue, Jun 25, 2024 at 6:42 PM Andrew Lamb <
> > >> > >>> > andrewlamb11-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > >> > >>> > >
> > >> > >>> > > > FWIW I started hacking up a prototype[1] of what a
> > >> > parquet-testing
> > >> > >>> > > > integration suite might look like if anyone is interested
> > >> > >>> > > >
> > >> > >>> > > >
> > >> > >>> > > >
> > >> > >>> > > > [1]: https://github.com/apache/arrow-rs/pull/5956
> > >> > >>> > > >
> > >> > >>> > > > On Tue, Jun 18, 2024 at 10:39 AM Alkis Evlogimenos
> > >> > >>> > > > <al...@databricks.com.invalid> wrote:
> > >> > >>> > > >
> > >> > >>> > > > > +1.
> > >> > >>> > > > >
> > >> > >>> > > > > I would suggest you address the comments first? I went
> > >> through
> > >> > >>> the
> > >> > >>> > open
> > >> > >>> > > > > ones and most of them make sense to me (and left few
> > >> additional
> > >> > >>> > > > comments).
> > >> > >>> > > > >
> > >> > >>> > > > > On Tue, Jun 18, 2024 at 12:42 PM Andrew Lamb <
> > >> > >>> andrewlamb11@gmail.com
> > >> > >>> > >
> > >> > >>> > > > > wrote:
> > >> > >>> > > > >
> > >> > >>> > > > > > Thank you
> > >> > >>> > > > > >
> > >> > >>> > > > > > On Mon, Jun 17, 2024 at 11:40 PM Micah Kornfield <
> > >> > >>> > > > emkornfield@gmail.com>
> > >> > >>> > > > > > wrote:
> > >> > >>> > > > > >
> > >> > >>> > > > > > > Hi Andrew,
> > >> > >>> > > > > > > I agree with this sentiment, I asked on the PR if
> > there
> > >> > would
> > >> > >>> > be
> > >> > >>> > > > > another
> > >> > >>> > > > > > > pass and then I can merge it.
> > >> > >>> > > > > > >
> > >> > >>> > > > > > > Cheers,
> > >> > >>> > > > > > > Micah
> > >> > >>> > > > > > >
> > >> > >>> > > > > > > On Fri, Jun 14, 2024 at 3:20 AM Andrew Lamb <
> > >> > >>> > andrewlamb11@gmail.com>
> > >> > >>> > > > > > > wrote:
> > >> > >>> > > > > > >
> > >> > >>> > > > > > > > Hello Parquet Devs,
> > >> > >>> > > > > > > >
> > >> > >>> > > > > > > > I propose we merge the first (admittedly bare
> bones)
> > >> > >>> > > > "Implementation
> > >> > >>> > > > > > > > Status" page PR [1] to the website soon. I think
> > this
> > >> > page
> > >> > >>> is
> > >> > >>> > vital
> > >> > >>> > > > > to
> > >> > >>> > > > > > > the
> > >> > >>> > > > > > > > Parquet community (and to any attempt to extend
> the
> > >> > >>> format) so
> > >> > >>> > the
> > >> > >>> > > > > > sooner
> > >> > >>> > > > > > > > the better.
> > >> > >>> > > > > > > >
> > >> > >>> > > > > > > > The reason to merge the PR now is to have a base
> > from
> > >> > which
> > >> > >>> > to
> > >> > >>> > > > build.
> > >> > >>> > > > > > > That
> > >> > >>> > > > > > > > PR is already over a year old and has so many
> > >> comments it
> > >> > >>> is
> > >> > >>> > hard
> > >> > >>> > > > to
> > >> > >>> > > > > > > follow
> > >> > >>> > > > > > > > or know what the path forward is. If we insist on
> > >> sorting
> > >> > >>> all
> > >> > >>> > the
> > >> > >>> > > > > > details
> > >> > >>> > > > > > > > out before we merge it I fear it will never merge.
> > >> > >>> > > > > > > >
> > >> > >>> > > > > > > > Once we have a page, I think the next steps are to
> > >> add a
> > >> > >>> > preamble
> > >> > >>> > > > > > > > explaining what it is for and to start trying to
> > fill
> > >> out
> > >> > >>> the
> > >> > >>> > chart
> > >> > >>> > > > > for
> > >> > >>> > > > > > > an
> > >> > >>> > > > > > > > implementation (I am happy to try for
> parquet-rs). I
> > >> > >>> suspect
> > >> > >>> > during
> > >> > >>> > > > > > that
> > >> > >>> > > > > > > > process we will have to adjust some of the charts
> > >> more.
> > >> > >>> > > > > > > >
> > >> > >>> > > > > > > > Thank you for your consideration (and thank you
> for
> > >> all
> > >> > the
> > >> > >>> > > > comments
> > >> > >>> > > > > so
> > >> > >>> > > > > > > > far)
> > >> > >>> > > > > > > >
> > >> > >>> > > > > > > > Andrew
> > >> > >>> > > > > > > >
> > >> > >>> > > > > > > > [1]:
> https://github.com/apache/parquet-site/pull/34
> > >> > >>> > > > > > > >
> > >> > >>> > > > > > >
> > >> > >>> > > > > >
> > >> > >>> > > > >
> > >> > >>> > > >
> > >> > >>> > >
> > >> > >>> >
> > >> > >>> >
> > >> > >>> >
> > >> > >>> >
> > >> > >>>
> > >> > >>
> > >> >
> > >>
> > >
> >
>
Re: [DISCUSS] Merge initial Implementation Status PR and incrementally improve it
Posted by Andrew Lamb <an...@gmail.com>.
Thank you!. Once that gets in I will find time (or really try and find
someone else to find time) to fill it out for parquet-rs
On Thu, Feb 6, 2025 at 12:04 AM Gang Wu <us...@gmail.com> wrote:
> Just a heads-up that I've created a PR [1] to update the matrix
> for parquet-cpp and parquet-java. Let me know if you have any
> suggestion. Thanks!
>
> [1] https://github.com/apache/parquet-site/pull/100
>
>
> On Tue, Feb 4, 2025 at 10:11 PM Andrew Lamb <an...@gmail.com>
> wrote:
>
> > I wanted to highlight that there is a PR[1] to fill out this matrix for
> > cuda . Super exciting to see this starting
> >
> > [1] https://github.com/apache/parquet-site/pull/99#
> >
> > On Thu, Jul 4, 2024 at 2:57 PM Andrew Lamb <an...@gmail.com>
> wrote:
> >
> > > Thank you! A small step perhaps but a major step forward
> > >
> > > On Thu, Jul 4, 2024 at 1:03 PM Micah Kornfield <em...@gmail.com>
> > > wrote:
> > >
> > >> I just merged it.
> > >>
> > >> Cheers,
> > >> Micah
> > >>
> > >> On Thu, Jul 4, 2024 at 2:32 AM Andrew Lamb <an...@gmail.com>
> > >> wrote:
> > >>
> > >> > The initial feature matrix PR[1], has all outstanding comments
> > addressed
> > >> > and is ready to merge from my opinion.
> > >> >
> > >> > Could one of the committers consider merging it so we can start
> > filling
> > >> the
> > >> > matrix out?
> > >> >
> > >> > Thanks,
> > >> > Andrew
> > >> >
> > >> > [1]: https://github.com/apache/parquet-site/pull/34
> > >> >
> > >> >
> > >> > On Wed, Jun 26, 2024 at 11:22 AM Andrew Lamb <
> andrewlamb11@gmail.com>
> > >> > wrote:
> > >> >
> > >> > > I filed a ticket [1] to discuss integration testing / feature
> > >> > > compatibility testing
> > >> > >
> > >> > > [1]: https://github.com/apache/parquet-format/issues/441
> > >> > >
> > >> > > On Wed, Jun 26, 2024 at 10:03 AM Andrew Lamb <
> > andrewlamb11@gmail.com>
> > >> > > wrote:
> > >> > >
> > >> > >> I will file a ticket for discussion shortly
> > >> > >>
> > >> > >> On Wed, Jun 26, 2024, 06:55 Julien Le Dem <ju...@apache.org>
> > wrote:
> > >> > >>
> > >> > >>> I agree with Antoine for the separate thread/ticket for
> > integration
> > >> > >>> testing
> > >> > >>>
> > >> > >>> For the « implementation status » PR, I agree we can merge and
> > >> > iterate. I
> > >> > >>> added a couple of comments that can easily be addressed
> > afterwards.
> > >> > >>> Julien
> > >> > >>>
> > >> > >>> On Wed, Jun 26, 2024 at 10:43 Antoine Pitrou <
> antoine@python.org>
> > >> > wrote:
> > >> > >>>
> > >> > >>> >
> > >> > >>> > IMHO, we should either start a dedicated discussion thread for
> > >> > >>> > integration testing, or open a GH issue and discuss it there.
> > >> > >>> >
> > >> > >>> > Regards
> > >> > >>> >
> > >> > >>> > Antoine.
> > >> > >>> >
> > >> > >>> >
> > >> > >>> >
> > >> > >>> > On Wed, 26 Jun 2024 09:21:33 +0200
> > >> > >>> > Alkis Evlogimenos
> > >> > >>> > <al...@databricks.com.INVALID>
> > >> > >>> > wrote:
> > >> > >>> > > It would be nice if the integration suite specifies how a
> > driver
> > >> > can
> > >> > >>> be
> > >> > >>> > > executed. Then each implementation can provide a driver and
> > the
> > >> > suite
> > >> > >>> > will
> > >> > >>> > > use that for validation.
> > >> > >>> > >
> > >> > >>> > > By specifying both reads and writes for the driver we get a
> > lot
> > >> > more
> > >> > >>> > power.
> > >> > >>> > > Given an input we can roundtrip all combination of
> > >> readers/writers
> > >> > >>> and
> > >> > >>> > > verify they can roundtrip.
> > >> > >>> > >
> > >> > >>> > > On Tue, Jun 25, 2024 at 6:42 PM Andrew Lamb <
> > >> > >>> > andrewlamb11-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > >> > >>> > >
> > >> > >>> > > > FWIW I started hacking up a prototype[1] of what a
> > >> > parquet-testing
> > >> > >>> > > > integration suite might look like if anyone is interested
> > >> > >>> > > >
> > >> > >>> > > >
> > >> > >>> > > >
> > >> > >>> > > > [1]: https://github.com/apache/arrow-rs/pull/5956
> > >> > >>> > > >
> > >> > >>> > > > On Tue, Jun 18, 2024 at 10:39 AM Alkis Evlogimenos
> > >> > >>> > > > <al...@databricks.com.invalid> wrote:
> > >> > >>> > > >
> > >> > >>> > > > > +1.
> > >> > >>> > > > >
> > >> > >>> > > > > I would suggest you address the comments first? I went
> > >> through
> > >> > >>> the
> > >> > >>> > open
> > >> > >>> > > > > ones and most of them make sense to me (and left few
> > >> additional
> > >> > >>> > > > comments).
> > >> > >>> > > > >
> > >> > >>> > > > > On Tue, Jun 18, 2024 at 12:42 PM Andrew Lamb <
> > >> > >>> andrewlamb11@gmail.com
> > >> > >>> > >
> > >> > >>> > > > > wrote:
> > >> > >>> > > > >
> > >> > >>> > > > > > Thank you
> > >> > >>> > > > > >
> > >> > >>> > > > > > On Mon, Jun 17, 2024 at 11:40 PM Micah Kornfield <
> > >> > >>> > > > emkornfield@gmail.com>
> > >> > >>> > > > > > wrote:
> > >> > >>> > > > > >
> > >> > >>> > > > > > > Hi Andrew,
> > >> > >>> > > > > > > I agree with this sentiment, I asked on the PR if
> > there
> > >> > would
> > >> > >>> > be
> > >> > >>> > > > > another
> > >> > >>> > > > > > > pass and then I can merge it.
> > >> > >>> > > > > > >
> > >> > >>> > > > > > > Cheers,
> > >> > >>> > > > > > > Micah
> > >> > >>> > > > > > >
> > >> > >>> > > > > > > On Fri, Jun 14, 2024 at 3:20 AM Andrew Lamb <
> > >> > >>> > andrewlamb11@gmail.com>
> > >> > >>> > > > > > > wrote:
> > >> > >>> > > > > > >
> > >> > >>> > > > > > > > Hello Parquet Devs,
> > >> > >>> > > > > > > >
> > >> > >>> > > > > > > > I propose we merge the first (admittedly bare
> bones)
> > >> > >>> > > > "Implementation
> > >> > >>> > > > > > > > Status" page PR [1] to the website soon. I think
> > this
> > >> > page
> > >> > >>> is
> > >> > >>> > vital
> > >> > >>> > > > > to
> > >> > >>> > > > > > > the
> > >> > >>> > > > > > > > Parquet community (and to any attempt to extend
> the
> > >> > >>> format) so
> > >> > >>> > the
> > >> > >>> > > > > > sooner
> > >> > >>> > > > > > > > the better.
> > >> > >>> > > > > > > >
> > >> > >>> > > > > > > > The reason to merge the PR now is to have a base
> > from
> > >> > which
> > >> > >>> > to
> > >> > >>> > > > build.
> > >> > >>> > > > > > > That
> > >> > >>> > > > > > > > PR is already over a year old and has so many
> > >> comments it
> > >> > >>> is
> > >> > >>> > hard
> > >> > >>> > > > to
> > >> > >>> > > > > > > follow
> > >> > >>> > > > > > > > or know what the path forward is. If we insist on
> > >> sorting
> > >> > >>> all
> > >> > >>> > the
> > >> > >>> > > > > > details
> > >> > >>> > > > > > > > out before we merge it I fear it will never merge.
> > >> > >>> > > > > > > >
> > >> > >>> > > > > > > > Once we have a page, I think the next steps are to
> > >> add a
> > >> > >>> > preamble
> > >> > >>> > > > > > > > explaining what it is for and to start trying to
> > fill
> > >> out
> > >> > >>> the
> > >> > >>> > chart
> > >> > >>> > > > > for
> > >> > >>> > > > > > > an
> > >> > >>> > > > > > > > implementation (I am happy to try for
> parquet-rs). I
> > >> > >>> suspect
> > >> > >>> > during
> > >> > >>> > > > > > that
> > >> > >>> > > > > > > > process we will have to adjust some of the charts
> > >> more.
> > >> > >>> > > > > > > >
> > >> > >>> > > > > > > > Thank you for your consideration (and thank you
> for
> > >> all
> > >> > the
> > >> > >>> > > > comments
> > >> > >>> > > > > so
> > >> > >>> > > > > > > > far)
> > >> > >>> > > > > > > >
> > >> > >>> > > > > > > > Andrew
> > >> > >>> > > > > > > >
> > >> > >>> > > > > > > > [1]:
> https://github.com/apache/parquet-site/pull/34
> > >> > >>> > > > > > > >
> > >> > >>> > > > > > >
> > >> > >>> > > > > >
> > >> > >>> > > > >
> > >> > >>> > > >
> > >> > >>> > >
> > >> > >>> >
> > >> > >>> >
> > >> > >>> >
> > >> > >>> >
> > >> > >>>
> > >> > >>
> > >> >
> > >>
> > >
> >
>
Re: [DISCUSS] Merge initial Implementation Status PR and incrementally improve it
Posted by Gang Wu <us...@gmail.com>.
Just a heads-up that I've created a PR [1] to update the matrix
for parquet-cpp and parquet-java. Let me know if you have any
suggestion. Thanks!
[1] https://github.com/apache/parquet-site/pull/100
On Tue, Feb 4, 2025 at 10:11 PM Andrew Lamb <an...@gmail.com> wrote:
> I wanted to highlight that there is a PR[1] to fill out this matrix for
> cuda . Super exciting to see this starting
>
> [1] https://github.com/apache/parquet-site/pull/99#
>
> On Thu, Jul 4, 2024 at 2:57 PM Andrew Lamb <an...@gmail.com> wrote:
>
> > Thank you! A small step perhaps but a major step forward
> >
> > On Thu, Jul 4, 2024 at 1:03 PM Micah Kornfield <em...@gmail.com>
> > wrote:
> >
> >> I just merged it.
> >>
> >> Cheers,
> >> Micah
> >>
> >> On Thu, Jul 4, 2024 at 2:32 AM Andrew Lamb <an...@gmail.com>
> >> wrote:
> >>
> >> > The initial feature matrix PR[1], has all outstanding comments
> addressed
> >> > and is ready to merge from my opinion.
> >> >
> >> > Could one of the committers consider merging it so we can start
> filling
> >> the
> >> > matrix out?
> >> >
> >> > Thanks,
> >> > Andrew
> >> >
> >> > [1]: https://github.com/apache/parquet-site/pull/34
> >> >
> >> >
> >> > On Wed, Jun 26, 2024 at 11:22 AM Andrew Lamb <an...@gmail.com>
> >> > wrote:
> >> >
> >> > > I filed a ticket [1] to discuss integration testing / feature
> >> > > compatibility testing
> >> > >
> >> > > [1]: https://github.com/apache/parquet-format/issues/441
> >> > >
> >> > > On Wed, Jun 26, 2024 at 10:03 AM Andrew Lamb <
> andrewlamb11@gmail.com>
> >> > > wrote:
> >> > >
> >> > >> I will file a ticket for discussion shortly
> >> > >>
> >> > >> On Wed, Jun 26, 2024, 06:55 Julien Le Dem <ju...@apache.org>
> wrote:
> >> > >>
> >> > >>> I agree with Antoine for the separate thread/ticket for
> integration
> >> > >>> testing
> >> > >>>
> >> > >>> For the « implementation status » PR, I agree we can merge and
> >> > iterate. I
> >> > >>> added a couple of comments that can easily be addressed
> afterwards.
> >> > >>> Julien
> >> > >>>
> >> > >>> On Wed, Jun 26, 2024 at 10:43 Antoine Pitrou <an...@python.org>
> >> > wrote:
> >> > >>>
> >> > >>> >
> >> > >>> > IMHO, we should either start a dedicated discussion thread for
> >> > >>> > integration testing, or open a GH issue and discuss it there.
> >> > >>> >
> >> > >>> > Regards
> >> > >>> >
> >> > >>> > Antoine.
> >> > >>> >
> >> > >>> >
> >> > >>> >
> >> > >>> > On Wed, 26 Jun 2024 09:21:33 +0200
> >> > >>> > Alkis Evlogimenos
> >> > >>> > <al...@databricks.com.INVALID>
> >> > >>> > wrote:
> >> > >>> > > It would be nice if the integration suite specifies how a
> driver
> >> > can
> >> > >>> be
> >> > >>> > > executed. Then each implementation can provide a driver and
> the
> >> > suite
> >> > >>> > will
> >> > >>> > > use that for validation.
> >> > >>> > >
> >> > >>> > > By specifying both reads and writes for the driver we get a
> lot
> >> > more
> >> > >>> > power.
> >> > >>> > > Given an input we can roundtrip all combination of
> >> readers/writers
> >> > >>> and
> >> > >>> > > verify they can roundtrip.
> >> > >>> > >
> >> > >>> > > On Tue, Jun 25, 2024 at 6:42 PM Andrew Lamb <
> >> > >>> > andrewlamb11-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> >> > >>> > >
> >> > >>> > > > FWIW I started hacking up a prototype[1] of what a
> >> > parquet-testing
> >> > >>> > > > integration suite might look like if anyone is interested
> >> > >>> > > >
> >> > >>> > > >
> >> > >>> > > >
> >> > >>> > > > [1]: https://github.com/apache/arrow-rs/pull/5956
> >> > >>> > > >
> >> > >>> > > > On Tue, Jun 18, 2024 at 10:39 AM Alkis Evlogimenos
> >> > >>> > > > <al...@databricks.com.invalid> wrote:
> >> > >>> > > >
> >> > >>> > > > > +1.
> >> > >>> > > > >
> >> > >>> > > > > I would suggest you address the comments first? I went
> >> through
> >> > >>> the
> >> > >>> > open
> >> > >>> > > > > ones and most of them make sense to me (and left few
> >> additional
> >> > >>> > > > comments).
> >> > >>> > > > >
> >> > >>> > > > > On Tue, Jun 18, 2024 at 12:42 PM Andrew Lamb <
> >> > >>> andrewlamb11@gmail.com
> >> > >>> > >
> >> > >>> > > > > wrote:
> >> > >>> > > > >
> >> > >>> > > > > > Thank you
> >> > >>> > > > > >
> >> > >>> > > > > > On Mon, Jun 17, 2024 at 11:40 PM Micah Kornfield <
> >> > >>> > > > emkornfield@gmail.com>
> >> > >>> > > > > > wrote:
> >> > >>> > > > > >
> >> > >>> > > > > > > Hi Andrew,
> >> > >>> > > > > > > I agree with this sentiment, I asked on the PR if
> there
> >> > would
> >> > >>> > be
> >> > >>> > > > > another
> >> > >>> > > > > > > pass and then I can merge it.
> >> > >>> > > > > > >
> >> > >>> > > > > > > Cheers,
> >> > >>> > > > > > > Micah
> >> > >>> > > > > > >
> >> > >>> > > > > > > On Fri, Jun 14, 2024 at 3:20 AM Andrew Lamb <
> >> > >>> > andrewlamb11@gmail.com>
> >> > >>> > > > > > > wrote:
> >> > >>> > > > > > >
> >> > >>> > > > > > > > Hello Parquet Devs,
> >> > >>> > > > > > > >
> >> > >>> > > > > > > > I propose we merge the first (admittedly bare bones)
> >> > >>> > > > "Implementation
> >> > >>> > > > > > > > Status" page PR [1] to the website soon. I think
> this
> >> > page
> >> > >>> is
> >> > >>> > vital
> >> > >>> > > > > to
> >> > >>> > > > > > > the
> >> > >>> > > > > > > > Parquet community (and to any attempt to extend the
> >> > >>> format) so
> >> > >>> > the
> >> > >>> > > > > > sooner
> >> > >>> > > > > > > > the better.
> >> > >>> > > > > > > >
> >> > >>> > > > > > > > The reason to merge the PR now is to have a base
> from
> >> > which
> >> > >>> > to
> >> > >>> > > > build.
> >> > >>> > > > > > > That
> >> > >>> > > > > > > > PR is already over a year old and has so many
> >> comments it
> >> > >>> is
> >> > >>> > hard
> >> > >>> > > > to
> >> > >>> > > > > > > follow
> >> > >>> > > > > > > > or know what the path forward is. If we insist on
> >> sorting
> >> > >>> all
> >> > >>> > the
> >> > >>> > > > > > details
> >> > >>> > > > > > > > out before we merge it I fear it will never merge.
> >> > >>> > > > > > > >
> >> > >>> > > > > > > > Once we have a page, I think the next steps are to
> >> add a
> >> > >>> > preamble
> >> > >>> > > > > > > > explaining what it is for and to start trying to
> fill
> >> out
> >> > >>> the
> >> > >>> > chart
> >> > >>> > > > > for
> >> > >>> > > > > > > an
> >> > >>> > > > > > > > implementation (I am happy to try for parquet-rs). I
> >> > >>> suspect
> >> > >>> > during
> >> > >>> > > > > > that
> >> > >>> > > > > > > > process we will have to adjust some of the charts
> >> more.
> >> > >>> > > > > > > >
> >> > >>> > > > > > > > Thank you for your consideration (and thank you for
> >> all
> >> > the
> >> > >>> > > > comments
> >> > >>> > > > > so
> >> > >>> > > > > > > > far)
> >> > >>> > > > > > > >
> >> > >>> > > > > > > > Andrew
> >> > >>> > > > > > > >
> >> > >>> > > > > > > > [1]: https://github.com/apache/parquet-site/pull/34
> >> > >>> > > > > > > >
> >> > >>> > > > > > >
> >> > >>> > > > > >
> >> > >>> > > > >
> >> > >>> > > >
> >> > >>> > >
> >> > >>> >
> >> > >>> >
> >> > >>> >
> >> > >>> >
> >> > >>>
> >> > >>
> >> >
> >>
> >
>
Re: [DISCUSS] Merge initial Implementation Status PR and incrementally improve it
Posted by Gang Wu <us...@gmail.com>.
Just a heads-up that I've created a PR [1] to update the matrix
for parquet-cpp and parquet-java. Let me know if you have any
suggestion. Thanks!
[1] https://github.com/apache/parquet-site/pull/100
On Tue, Feb 4, 2025 at 10:11 PM Andrew Lamb <an...@gmail.com> wrote:
> I wanted to highlight that there is a PR[1] to fill out this matrix for
> cuda . Super exciting to see this starting
>
> [1] https://github.com/apache/parquet-site/pull/99#
>
> On Thu, Jul 4, 2024 at 2:57 PM Andrew Lamb <an...@gmail.com> wrote:
>
> > Thank you! A small step perhaps but a major step forward
> >
> > On Thu, Jul 4, 2024 at 1:03 PM Micah Kornfield <em...@gmail.com>
> > wrote:
> >
> >> I just merged it.
> >>
> >> Cheers,
> >> Micah
> >>
> >> On Thu, Jul 4, 2024 at 2:32 AM Andrew Lamb <an...@gmail.com>
> >> wrote:
> >>
> >> > The initial feature matrix PR[1], has all outstanding comments
> addressed
> >> > and is ready to merge from my opinion.
> >> >
> >> > Could one of the committers consider merging it so we can start
> filling
> >> the
> >> > matrix out?
> >> >
> >> > Thanks,
> >> > Andrew
> >> >
> >> > [1]: https://github.com/apache/parquet-site/pull/34
> >> >
> >> >
> >> > On Wed, Jun 26, 2024 at 11:22 AM Andrew Lamb <an...@gmail.com>
> >> > wrote:
> >> >
> >> > > I filed a ticket [1] to discuss integration testing / feature
> >> > > compatibility testing
> >> > >
> >> > > [1]: https://github.com/apache/parquet-format/issues/441
> >> > >
> >> > > On Wed, Jun 26, 2024 at 10:03 AM Andrew Lamb <
> andrewlamb11@gmail.com>
> >> > > wrote:
> >> > >
> >> > >> I will file a ticket for discussion shortly
> >> > >>
> >> > >> On Wed, Jun 26, 2024, 06:55 Julien Le Dem <ju...@apache.org>
> wrote:
> >> > >>
> >> > >>> I agree with Antoine for the separate thread/ticket for
> integration
> >> > >>> testing
> >> > >>>
> >> > >>> For the « implementation status » PR, I agree we can merge and
> >> > iterate. I
> >> > >>> added a couple of comments that can easily be addressed
> afterwards.
> >> > >>> Julien
> >> > >>>
> >> > >>> On Wed, Jun 26, 2024 at 10:43 Antoine Pitrou <an...@python.org>
> >> > wrote:
> >> > >>>
> >> > >>> >
> >> > >>> > IMHO, we should either start a dedicated discussion thread for
> >> > >>> > integration testing, or open a GH issue and discuss it there.
> >> > >>> >
> >> > >>> > Regards
> >> > >>> >
> >> > >>> > Antoine.
> >> > >>> >
> >> > >>> >
> >> > >>> >
> >> > >>> > On Wed, 26 Jun 2024 09:21:33 +0200
> >> > >>> > Alkis Evlogimenos
> >> > >>> > <al...@databricks.com.INVALID>
> >> > >>> > wrote:
> >> > >>> > > It would be nice if the integration suite specifies how a
> driver
> >> > can
> >> > >>> be
> >> > >>> > > executed. Then each implementation can provide a driver and
> the
> >> > suite
> >> > >>> > will
> >> > >>> > > use that for validation.
> >> > >>> > >
> >> > >>> > > By specifying both reads and writes for the driver we get a
> lot
> >> > more
> >> > >>> > power.
> >> > >>> > > Given an input we can roundtrip all combination of
> >> readers/writers
> >> > >>> and
> >> > >>> > > verify they can roundtrip.
> >> > >>> > >
> >> > >>> > > On Tue, Jun 25, 2024 at 6:42 PM Andrew Lamb <
> >> > >>> > andrewlamb11-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> >> > >>> > >
> >> > >>> > > > FWIW I started hacking up a prototype[1] of what a
> >> > parquet-testing
> >> > >>> > > > integration suite might look like if anyone is interested
> >> > >>> > > >
> >> > >>> > > >
> >> > >>> > > >
> >> > >>> > > > [1]: https://github.com/apache/arrow-rs/pull/5956
> >> > >>> > > >
> >> > >>> > > > On Tue, Jun 18, 2024 at 10:39 AM Alkis Evlogimenos
> >> > >>> > > > <al...@databricks.com.invalid> wrote:
> >> > >>> > > >
> >> > >>> > > > > +1.
> >> > >>> > > > >
> >> > >>> > > > > I would suggest you address the comments first? I went
> >> through
> >> > >>> the
> >> > >>> > open
> >> > >>> > > > > ones and most of them make sense to me (and left few
> >> additional
> >> > >>> > > > comments).
> >> > >>> > > > >
> >> > >>> > > > > On Tue, Jun 18, 2024 at 12:42 PM Andrew Lamb <
> >> > >>> andrewlamb11@gmail.com
> >> > >>> > >
> >> > >>> > > > > wrote:
> >> > >>> > > > >
> >> > >>> > > > > > Thank you
> >> > >>> > > > > >
> >> > >>> > > > > > On Mon, Jun 17, 2024 at 11:40 PM Micah Kornfield <
> >> > >>> > > > emkornfield@gmail.com>
> >> > >>> > > > > > wrote:
> >> > >>> > > > > >
> >> > >>> > > > > > > Hi Andrew,
> >> > >>> > > > > > > I agree with this sentiment, I asked on the PR if
> there
> >> > would
> >> > >>> > be
> >> > >>> > > > > another
> >> > >>> > > > > > > pass and then I can merge it.
> >> > >>> > > > > > >
> >> > >>> > > > > > > Cheers,
> >> > >>> > > > > > > Micah
> >> > >>> > > > > > >
> >> > >>> > > > > > > On Fri, Jun 14, 2024 at 3:20 AM Andrew Lamb <
> >> > >>> > andrewlamb11@gmail.com>
> >> > >>> > > > > > > wrote:
> >> > >>> > > > > > >
> >> > >>> > > > > > > > Hello Parquet Devs,
> >> > >>> > > > > > > >
> >> > >>> > > > > > > > I propose we merge the first (admittedly bare bones)
> >> > >>> > > > "Implementation
> >> > >>> > > > > > > > Status" page PR [1] to the website soon. I think
> this
> >> > page
> >> > >>> is
> >> > >>> > vital
> >> > >>> > > > > to
> >> > >>> > > > > > > the
> >> > >>> > > > > > > > Parquet community (and to any attempt to extend the
> >> > >>> format) so
> >> > >>> > the
> >> > >>> > > > > > sooner
> >> > >>> > > > > > > > the better.
> >> > >>> > > > > > > >
> >> > >>> > > > > > > > The reason to merge the PR now is to have a base
> from
> >> > which
> >> > >>> > to
> >> > >>> > > > build.
> >> > >>> > > > > > > That
> >> > >>> > > > > > > > PR is already over a year old and has so many
> >> comments it
> >> > >>> is
> >> > >>> > hard
> >> > >>> > > > to
> >> > >>> > > > > > > follow
> >> > >>> > > > > > > > or know what the path forward is. If we insist on
> >> sorting
> >> > >>> all
> >> > >>> > the
> >> > >>> > > > > > details
> >> > >>> > > > > > > > out before we merge it I fear it will never merge.
> >> > >>> > > > > > > >
> >> > >>> > > > > > > > Once we have a page, I think the next steps are to
> >> add a
> >> > >>> > preamble
> >> > >>> > > > > > > > explaining what it is for and to start trying to
> fill
> >> out
> >> > >>> the
> >> > >>> > chart
> >> > >>> > > > > for
> >> > >>> > > > > > > an
> >> > >>> > > > > > > > implementation (I am happy to try for parquet-rs). I
> >> > >>> suspect
> >> > >>> > during
> >> > >>> > > > > > that
> >> > >>> > > > > > > > process we will have to adjust some of the charts
> >> more.
> >> > >>> > > > > > > >
> >> > >>> > > > > > > > Thank you for your consideration (and thank you for
> >> all
> >> > the
> >> > >>> > > > comments
> >> > >>> > > > > so
> >> > >>> > > > > > > > far)
> >> > >>> > > > > > > >
> >> > >>> > > > > > > > Andrew
> >> > >>> > > > > > > >
> >> > >>> > > > > > > > [1]: https://github.com/apache/parquet-site/pull/34
> >> > >>> > > > > > > >
> >> > >>> > > > > > >
> >> > >>> > > > > >
> >> > >>> > > > >
> >> > >>> > > >
> >> > >>> > >
> >> > >>> >
> >> > >>> >
> >> > >>> >
> >> > >>> >
> >> > >>>
> >> > >>
> >> >
> >>
> >
>
Re: [DISCUSS] Merge initial Implementation Status PR and incrementally improve it
Posted by Andrew Lamb <an...@gmail.com>.
I wanted to highlight that there is a PR[1] to fill out this matrix for
cuda . Super exciting to see this starting
[1] https://github.com/apache/parquet-site/pull/99#
On Thu, Jul 4, 2024 at 2:57 PM Andrew Lamb <an...@gmail.com> wrote:
> Thank you! A small step perhaps but a major step forward
>
> On Thu, Jul 4, 2024 at 1:03 PM Micah Kornfield <em...@gmail.com>
> wrote:
>
>> I just merged it.
>>
>> Cheers,
>> Micah
>>
>> On Thu, Jul 4, 2024 at 2:32 AM Andrew Lamb <an...@gmail.com>
>> wrote:
>>
>> > The initial feature matrix PR[1], has all outstanding comments addressed
>> > and is ready to merge from my opinion.
>> >
>> > Could one of the committers consider merging it so we can start filling
>> the
>> > matrix out?
>> >
>> > Thanks,
>> > Andrew
>> >
>> > [1]: https://github.com/apache/parquet-site/pull/34
>> >
>> >
>> > On Wed, Jun 26, 2024 at 11:22 AM Andrew Lamb <an...@gmail.com>
>> > wrote:
>> >
>> > > I filed a ticket [1] to discuss integration testing / feature
>> > > compatibility testing
>> > >
>> > > [1]: https://github.com/apache/parquet-format/issues/441
>> > >
>> > > On Wed, Jun 26, 2024 at 10:03 AM Andrew Lamb <an...@gmail.com>
>> > > wrote:
>> > >
>> > >> I will file a ticket for discussion shortly
>> > >>
>> > >> On Wed, Jun 26, 2024, 06:55 Julien Le Dem <ju...@apache.org> wrote:
>> > >>
>> > >>> I agree with Antoine for the separate thread/ticket for integration
>> > >>> testing
>> > >>>
>> > >>> For the « implementation status » PR, I agree we can merge and
>> > iterate. I
>> > >>> added a couple of comments that can easily be addressed afterwards.
>> > >>> Julien
>> > >>>
>> > >>> On Wed, Jun 26, 2024 at 10:43 Antoine Pitrou <an...@python.org>
>> > wrote:
>> > >>>
>> > >>> >
>> > >>> > IMHO, we should either start a dedicated discussion thread for
>> > >>> > integration testing, or open a GH issue and discuss it there.
>> > >>> >
>> > >>> > Regards
>> > >>> >
>> > >>> > Antoine.
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> > On Wed, 26 Jun 2024 09:21:33 +0200
>> > >>> > Alkis Evlogimenos
>> > >>> > <al...@databricks.com.INVALID>
>> > >>> > wrote:
>> > >>> > > It would be nice if the integration suite specifies how a driver
>> > can
>> > >>> be
>> > >>> > > executed. Then each implementation can provide a driver and the
>> > suite
>> > >>> > will
>> > >>> > > use that for validation.
>> > >>> > >
>> > >>> > > By specifying both reads and writes for the driver we get a lot
>> > more
>> > >>> > power.
>> > >>> > > Given an input we can roundtrip all combination of
>> readers/writers
>> > >>> and
>> > >>> > > verify they can roundtrip.
>> > >>> > >
>> > >>> > > On Tue, Jun 25, 2024 at 6:42 PM Andrew Lamb <
>> > >>> > andrewlamb11-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> > >>> > >
>> > >>> > > > FWIW I started hacking up a prototype[1] of what a
>> > parquet-testing
>> > >>> > > > integration suite might look like if anyone is interested
>> > >>> > > >
>> > >>> > > >
>> > >>> > > >
>> > >>> > > > [1]: https://github.com/apache/arrow-rs/pull/5956
>> > >>> > > >
>> > >>> > > > On Tue, Jun 18, 2024 at 10:39 AM Alkis Evlogimenos
>> > >>> > > > <al...@databricks.com.invalid> wrote:
>> > >>> > > >
>> > >>> > > > > +1.
>> > >>> > > > >
>> > >>> > > > > I would suggest you address the comments first? I went
>> through
>> > >>> the
>> > >>> > open
>> > >>> > > > > ones and most of them make sense to me (and left few
>> additional
>> > >>> > > > comments).
>> > >>> > > > >
>> > >>> > > > > On Tue, Jun 18, 2024 at 12:42 PM Andrew Lamb <
>> > >>> andrewlamb11@gmail.com
>> > >>> > >
>> > >>> > > > > wrote:
>> > >>> > > > >
>> > >>> > > > > > Thank you
>> > >>> > > > > >
>> > >>> > > > > > On Mon, Jun 17, 2024 at 11:40 PM Micah Kornfield <
>> > >>> > > > emkornfield@gmail.com>
>> > >>> > > > > > wrote:
>> > >>> > > > > >
>> > >>> > > > > > > Hi Andrew,
>> > >>> > > > > > > I agree with this sentiment, I asked on the PR if there
>> > would
>> > >>> > be
>> > >>> > > > > another
>> > >>> > > > > > > pass and then I can merge it.
>> > >>> > > > > > >
>> > >>> > > > > > > Cheers,
>> > >>> > > > > > > Micah
>> > >>> > > > > > >
>> > >>> > > > > > > On Fri, Jun 14, 2024 at 3:20 AM Andrew Lamb <
>> > >>> > andrewlamb11@gmail.com>
>> > >>> > > > > > > wrote:
>> > >>> > > > > > >
>> > >>> > > > > > > > Hello Parquet Devs,
>> > >>> > > > > > > >
>> > >>> > > > > > > > I propose we merge the first (admittedly bare bones)
>> > >>> > > > "Implementation
>> > >>> > > > > > > > Status" page PR [1] to the website soon. I think this
>> > page
>> > >>> is
>> > >>> > vital
>> > >>> > > > > to
>> > >>> > > > > > > the
>> > >>> > > > > > > > Parquet community (and to any attempt to extend the
>> > >>> format) so
>> > >>> > the
>> > >>> > > > > > sooner
>> > >>> > > > > > > > the better.
>> > >>> > > > > > > >
>> > >>> > > > > > > > The reason to merge the PR now is to have a base from
>> > which
>> > >>> > to
>> > >>> > > > build.
>> > >>> > > > > > > That
>> > >>> > > > > > > > PR is already over a year old and has so many
>> comments it
>> > >>> is
>> > >>> > hard
>> > >>> > > > to
>> > >>> > > > > > > follow
>> > >>> > > > > > > > or know what the path forward is. If we insist on
>> sorting
>> > >>> all
>> > >>> > the
>> > >>> > > > > > details
>> > >>> > > > > > > > out before we merge it I fear it will never merge.
>> > >>> > > > > > > >
>> > >>> > > > > > > > Once we have a page, I think the next steps are to
>> add a
>> > >>> > preamble
>> > >>> > > > > > > > explaining what it is for and to start trying to fill
>> out
>> > >>> the
>> > >>> > chart
>> > >>> > > > > for
>> > >>> > > > > > > an
>> > >>> > > > > > > > implementation (I am happy to try for parquet-rs). I
>> > >>> suspect
>> > >>> > during
>> > >>> > > > > > that
>> > >>> > > > > > > > process we will have to adjust some of the charts
>> more.
>> > >>> > > > > > > >
>> > >>> > > > > > > > Thank you for your consideration (and thank you for
>> all
>> > the
>> > >>> > > > comments
>> > >>> > > > > so
>> > >>> > > > > > > > far)
>> > >>> > > > > > > >
>> > >>> > > > > > > > Andrew
>> > >>> > > > > > > >
>> > >>> > > > > > > > [1]: https://github.com/apache/parquet-site/pull/34
>> > >>> > > > > > > >
>> > >>> > > > > > >
>> > >>> > > > > >
>> > >>> > > > >
>> > >>> > > >
>> > >>> > >
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>>
>> > >>
>> >
>>
>
Re: [DISCUSS] Merge initial Implementation Status PR and incrementally improve it
Posted by Andrew Lamb <an...@gmail.com>.
I wanted to highlight that there is a PR[1] to fill out this matrix for
cuda . Super exciting to see this starting
[1] https://github.com/apache/parquet-site/pull/99#
On Thu, Jul 4, 2024 at 2:57 PM Andrew Lamb <an...@gmail.com> wrote:
> Thank you! A small step perhaps but a major step forward
>
> On Thu, Jul 4, 2024 at 1:03 PM Micah Kornfield <em...@gmail.com>
> wrote:
>
>> I just merged it.
>>
>> Cheers,
>> Micah
>>
>> On Thu, Jul 4, 2024 at 2:32 AM Andrew Lamb <an...@gmail.com>
>> wrote:
>>
>> > The initial feature matrix PR[1], has all outstanding comments addressed
>> > and is ready to merge from my opinion.
>> >
>> > Could one of the committers consider merging it so we can start filling
>> the
>> > matrix out?
>> >
>> > Thanks,
>> > Andrew
>> >
>> > [1]: https://github.com/apache/parquet-site/pull/34
>> >
>> >
>> > On Wed, Jun 26, 2024 at 11:22 AM Andrew Lamb <an...@gmail.com>
>> > wrote:
>> >
>> > > I filed a ticket [1] to discuss integration testing / feature
>> > > compatibility testing
>> > >
>> > > [1]: https://github.com/apache/parquet-format/issues/441
>> > >
>> > > On Wed, Jun 26, 2024 at 10:03 AM Andrew Lamb <an...@gmail.com>
>> > > wrote:
>> > >
>> > >> I will file a ticket for discussion shortly
>> > >>
>> > >> On Wed, Jun 26, 2024, 06:55 Julien Le Dem <ju...@apache.org> wrote:
>> > >>
>> > >>> I agree with Antoine for the separate thread/ticket for integration
>> > >>> testing
>> > >>>
>> > >>> For the « implementation status » PR, I agree we can merge and
>> > iterate. I
>> > >>> added a couple of comments that can easily be addressed afterwards.
>> > >>> Julien
>> > >>>
>> > >>> On Wed, Jun 26, 2024 at 10:43 Antoine Pitrou <an...@python.org>
>> > wrote:
>> > >>>
>> > >>> >
>> > >>> > IMHO, we should either start a dedicated discussion thread for
>> > >>> > integration testing, or open a GH issue and discuss it there.
>> > >>> >
>> > >>> > Regards
>> > >>> >
>> > >>> > Antoine.
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> > On Wed, 26 Jun 2024 09:21:33 +0200
>> > >>> > Alkis Evlogimenos
>> > >>> > <al...@databricks.com.INVALID>
>> > >>> > wrote:
>> > >>> > > It would be nice if the integration suite specifies how a driver
>> > can
>> > >>> be
>> > >>> > > executed. Then each implementation can provide a driver and the
>> > suite
>> > >>> > will
>> > >>> > > use that for validation.
>> > >>> > >
>> > >>> > > By specifying both reads and writes for the driver we get a lot
>> > more
>> > >>> > power.
>> > >>> > > Given an input we can roundtrip all combination of
>> readers/writers
>> > >>> and
>> > >>> > > verify they can roundtrip.
>> > >>> > >
>> > >>> > > On Tue, Jun 25, 2024 at 6:42 PM Andrew Lamb <
>> > >>> > andrewlamb11-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> > >>> > >
>> > >>> > > > FWIW I started hacking up a prototype[1] of what a
>> > parquet-testing
>> > >>> > > > integration suite might look like if anyone is interested
>> > >>> > > >
>> > >>> > > >
>> > >>> > > >
>> > >>> > > > [1]: https://github.com/apache/arrow-rs/pull/5956
>> > >>> > > >
>> > >>> > > > On Tue, Jun 18, 2024 at 10:39 AM Alkis Evlogimenos
>> > >>> > > > <al...@databricks.com.invalid> wrote:
>> > >>> > > >
>> > >>> > > > > +1.
>> > >>> > > > >
>> > >>> > > > > I would suggest you address the comments first? I went
>> through
>> > >>> the
>> > >>> > open
>> > >>> > > > > ones and most of them make sense to me (and left few
>> additional
>> > >>> > > > comments).
>> > >>> > > > >
>> > >>> > > > > On Tue, Jun 18, 2024 at 12:42 PM Andrew Lamb <
>> > >>> andrewlamb11@gmail.com
>> > >>> > >
>> > >>> > > > > wrote:
>> > >>> > > > >
>> > >>> > > > > > Thank you
>> > >>> > > > > >
>> > >>> > > > > > On Mon, Jun 17, 2024 at 11:40 PM Micah Kornfield <
>> > >>> > > > emkornfield@gmail.com>
>> > >>> > > > > > wrote:
>> > >>> > > > > >
>> > >>> > > > > > > Hi Andrew,
>> > >>> > > > > > > I agree with this sentiment, I asked on the PR if there
>> > would
>> > >>> > be
>> > >>> > > > > another
>> > >>> > > > > > > pass and then I can merge it.
>> > >>> > > > > > >
>> > >>> > > > > > > Cheers,
>> > >>> > > > > > > Micah
>> > >>> > > > > > >
>> > >>> > > > > > > On Fri, Jun 14, 2024 at 3:20 AM Andrew Lamb <
>> > >>> > andrewlamb11@gmail.com>
>> > >>> > > > > > > wrote:
>> > >>> > > > > > >
>> > >>> > > > > > > > Hello Parquet Devs,
>> > >>> > > > > > > >
>> > >>> > > > > > > > I propose we merge the first (admittedly bare bones)
>> > >>> > > > "Implementation
>> > >>> > > > > > > > Status" page PR [1] to the website soon. I think this
>> > page
>> > >>> is
>> > >>> > vital
>> > >>> > > > > to
>> > >>> > > > > > > the
>> > >>> > > > > > > > Parquet community (and to any attempt to extend the
>> > >>> format) so
>> > >>> > the
>> > >>> > > > > > sooner
>> > >>> > > > > > > > the better.
>> > >>> > > > > > > >
>> > >>> > > > > > > > The reason to merge the PR now is to have a base from
>> > which
>> > >>> > to
>> > >>> > > > build.
>> > >>> > > > > > > That
>> > >>> > > > > > > > PR is already over a year old and has so many
>> comments it
>> > >>> is
>> > >>> > hard
>> > >>> > > > to
>> > >>> > > > > > > follow
>> > >>> > > > > > > > or know what the path forward is. If we insist on
>> sorting
>> > >>> all
>> > >>> > the
>> > >>> > > > > > details
>> > >>> > > > > > > > out before we merge it I fear it will never merge.
>> > >>> > > > > > > >
>> > >>> > > > > > > > Once we have a page, I think the next steps are to
>> add a
>> > >>> > preamble
>> > >>> > > > > > > > explaining what it is for and to start trying to fill
>> out
>> > >>> the
>> > >>> > chart
>> > >>> > > > > for
>> > >>> > > > > > > an
>> > >>> > > > > > > > implementation (I am happy to try for parquet-rs). I
>> > >>> suspect
>> > >>> > during
>> > >>> > > > > > that
>> > >>> > > > > > > > process we will have to adjust some of the charts
>> more.
>> > >>> > > > > > > >
>> > >>> > > > > > > > Thank you for your consideration (and thank you for
>> all
>> > the
>> > >>> > > > comments
>> > >>> > > > > so
>> > >>> > > > > > > > far)
>> > >>> > > > > > > >
>> > >>> > > > > > > > Andrew
>> > >>> > > > > > > >
>> > >>> > > > > > > > [1]: https://github.com/apache/parquet-site/pull/34
>> > >>> > > > > > > >
>> > >>> > > > > > >
>> > >>> > > > > >
>> > >>> > > > >
>> > >>> > > >
>> > >>> > >
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>>
>> > >>
>> >
>>
>
Re: [DISCUSS] Merge initial Implementation Status PR and incrementally improve it
Posted by Andrew Lamb <an...@gmail.com>.
Thank you! A small step perhaps but a major step forward
On Thu, Jul 4, 2024 at 1:03 PM Micah Kornfield <em...@gmail.com>
wrote:
> I just merged it.
>
> Cheers,
> Micah
>
> On Thu, Jul 4, 2024 at 2:32 AM Andrew Lamb <an...@gmail.com> wrote:
>
> > The initial feature matrix PR[1], has all outstanding comments addressed
> > and is ready to merge from my opinion.
> >
> > Could one of the committers consider merging it so we can start filling
> the
> > matrix out?
> >
> > Thanks,
> > Andrew
> >
> > [1]: https://github.com/apache/parquet-site/pull/34
> >
> >
> > On Wed, Jun 26, 2024 at 11:22 AM Andrew Lamb <an...@gmail.com>
> > wrote:
> >
> > > I filed a ticket [1] to discuss integration testing / feature
> > > compatibility testing
> > >
> > > [1]: https://github.com/apache/parquet-format/issues/441
> > >
> > > On Wed, Jun 26, 2024 at 10:03 AM Andrew Lamb <an...@gmail.com>
> > > wrote:
> > >
> > >> I will file a ticket for discussion shortly
> > >>
> > >> On Wed, Jun 26, 2024, 06:55 Julien Le Dem <ju...@apache.org> wrote:
> > >>
> > >>> I agree with Antoine for the separate thread/ticket for integration
> > >>> testing
> > >>>
> > >>> For the « implementation status » PR, I agree we can merge and
> > iterate. I
> > >>> added a couple of comments that can easily be addressed afterwards.
> > >>> Julien
> > >>>
> > >>> On Wed, Jun 26, 2024 at 10:43 Antoine Pitrou <an...@python.org>
> > wrote:
> > >>>
> > >>> >
> > >>> > IMHO, we should either start a dedicated discussion thread for
> > >>> > integration testing, or open a GH issue and discuss it there.
> > >>> >
> > >>> > Regards
> > >>> >
> > >>> > Antoine.
> > >>> >
> > >>> >
> > >>> >
> > >>> > On Wed, 26 Jun 2024 09:21:33 +0200
> > >>> > Alkis Evlogimenos
> > >>> > <al...@databricks.com.INVALID>
> > >>> > wrote:
> > >>> > > It would be nice if the integration suite specifies how a driver
> > can
> > >>> be
> > >>> > > executed. Then each implementation can provide a driver and the
> > suite
> > >>> > will
> > >>> > > use that for validation.
> > >>> > >
> > >>> > > By specifying both reads and writes for the driver we get a lot
> > more
> > >>> > power.
> > >>> > > Given an input we can roundtrip all combination of
> readers/writers
> > >>> and
> > >>> > > verify they can roundtrip.
> > >>> > >
> > >>> > > On Tue, Jun 25, 2024 at 6:42 PM Andrew Lamb <
> > >>> > andrewlamb11-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > >>> > >
> > >>> > > > FWIW I started hacking up a prototype[1] of what a
> > parquet-testing
> > >>> > > > integration suite might look like if anyone is interested
> > >>> > > >
> > >>> > > >
> > >>> > > >
> > >>> > > > [1]: https://github.com/apache/arrow-rs/pull/5956
> > >>> > > >
> > >>> > > > On Tue, Jun 18, 2024 at 10:39 AM Alkis Evlogimenos
> > >>> > > > <al...@databricks.com.invalid> wrote:
> > >>> > > >
> > >>> > > > > +1.
> > >>> > > > >
> > >>> > > > > I would suggest you address the comments first? I went
> through
> > >>> the
> > >>> > open
> > >>> > > > > ones and most of them make sense to me (and left few
> additional
> > >>> > > > comments).
> > >>> > > > >
> > >>> > > > > On Tue, Jun 18, 2024 at 12:42 PM Andrew Lamb <
> > >>> andrewlamb11@gmail.com
> > >>> > >
> > >>> > > > > wrote:
> > >>> > > > >
> > >>> > > > > > Thank you
> > >>> > > > > >
> > >>> > > > > > On Mon, Jun 17, 2024 at 11:40 PM Micah Kornfield <
> > >>> > > > emkornfield@gmail.com>
> > >>> > > > > > wrote:
> > >>> > > > > >
> > >>> > > > > > > Hi Andrew,
> > >>> > > > > > > I agree with this sentiment, I asked on the PR if there
> > would
> > >>> > be
> > >>> > > > > another
> > >>> > > > > > > pass and then I can merge it.
> > >>> > > > > > >
> > >>> > > > > > > Cheers,
> > >>> > > > > > > Micah
> > >>> > > > > > >
> > >>> > > > > > > On Fri, Jun 14, 2024 at 3:20 AM Andrew Lamb <
> > >>> > andrewlamb11@gmail.com>
> > >>> > > > > > > wrote:
> > >>> > > > > > >
> > >>> > > > > > > > Hello Parquet Devs,
> > >>> > > > > > > >
> > >>> > > > > > > > I propose we merge the first (admittedly bare bones)
> > >>> > > > "Implementation
> > >>> > > > > > > > Status" page PR [1] to the website soon. I think this
> > page
> > >>> is
> > >>> > vital
> > >>> > > > > to
> > >>> > > > > > > the
> > >>> > > > > > > > Parquet community (and to any attempt to extend the
> > >>> format) so
> > >>> > the
> > >>> > > > > > sooner
> > >>> > > > > > > > the better.
> > >>> > > > > > > >
> > >>> > > > > > > > The reason to merge the PR now is to have a base from
> > which
> > >>> > to
> > >>> > > > build.
> > >>> > > > > > > That
> > >>> > > > > > > > PR is already over a year old and has so many comments
> it
> > >>> is
> > >>> > hard
> > >>> > > > to
> > >>> > > > > > > follow
> > >>> > > > > > > > or know what the path forward is. If we insist on
> sorting
> > >>> all
> > >>> > the
> > >>> > > > > > details
> > >>> > > > > > > > out before we merge it I fear it will never merge.
> > >>> > > > > > > >
> > >>> > > > > > > > Once we have a page, I think the next steps are to add
> a
> > >>> > preamble
> > >>> > > > > > > > explaining what it is for and to start trying to fill
> out
> > >>> the
> > >>> > chart
> > >>> > > > > for
> > >>> > > > > > > an
> > >>> > > > > > > > implementation (I am happy to try for parquet-rs). I
> > >>> suspect
> > >>> > during
> > >>> > > > > > that
> > >>> > > > > > > > process we will have to adjust some of the charts more.
> > >>> > > > > > > >
> > >>> > > > > > > > Thank you for your consideration (and thank you for all
> > the
> > >>> > > > comments
> > >>> > > > > so
> > >>> > > > > > > > far)
> > >>> > > > > > > >
> > >>> > > > > > > > Andrew
> > >>> > > > > > > >
> > >>> > > > > > > > [1]: https://github.com/apache/parquet-site/pull/34
> > >>> > > > > > > >
> > >>> > > > > > >
> > >>> > > > > >
> > >>> > > > >
> > >>> > > >
> > >>> > >
> > >>> >
> > >>> >
> > >>> >
> > >>> >
> > >>>
> > >>
> >
>
Re: [DISCUSS] Merge initial Implementation Status PR and incrementally improve it
Posted by Andrew Lamb <an...@gmail.com>.
Thank you! A small step perhaps but a major step forward
On Thu, Jul 4, 2024 at 1:03 PM Micah Kornfield <em...@gmail.com>
wrote:
> I just merged it.
>
> Cheers,
> Micah
>
> On Thu, Jul 4, 2024 at 2:32 AM Andrew Lamb <an...@gmail.com> wrote:
>
> > The initial feature matrix PR[1], has all outstanding comments addressed
> > and is ready to merge from my opinion.
> >
> > Could one of the committers consider merging it so we can start filling
> the
> > matrix out?
> >
> > Thanks,
> > Andrew
> >
> > [1]: https://github.com/apache/parquet-site/pull/34
> >
> >
> > On Wed, Jun 26, 2024 at 11:22 AM Andrew Lamb <an...@gmail.com>
> > wrote:
> >
> > > I filed a ticket [1] to discuss integration testing / feature
> > > compatibility testing
> > >
> > > [1]: https://github.com/apache/parquet-format/issues/441
> > >
> > > On Wed, Jun 26, 2024 at 10:03 AM Andrew Lamb <an...@gmail.com>
> > > wrote:
> > >
> > >> I will file a ticket for discussion shortly
> > >>
> > >> On Wed, Jun 26, 2024, 06:55 Julien Le Dem <ju...@apache.org> wrote:
> > >>
> > >>> I agree with Antoine for the separate thread/ticket for integration
> > >>> testing
> > >>>
> > >>> For the « implementation status » PR, I agree we can merge and
> > iterate. I
> > >>> added a couple of comments that can easily be addressed afterwards.
> > >>> Julien
> > >>>
> > >>> On Wed, Jun 26, 2024 at 10:43 Antoine Pitrou <an...@python.org>
> > wrote:
> > >>>
> > >>> >
> > >>> > IMHO, we should either start a dedicated discussion thread for
> > >>> > integration testing, or open a GH issue and discuss it there.
> > >>> >
> > >>> > Regards
> > >>> >
> > >>> > Antoine.
> > >>> >
> > >>> >
> > >>> >
> > >>> > On Wed, 26 Jun 2024 09:21:33 +0200
> > >>> > Alkis Evlogimenos
> > >>> > <al...@databricks.com.INVALID>
> > >>> > wrote:
> > >>> > > It would be nice if the integration suite specifies how a driver
> > can
> > >>> be
> > >>> > > executed. Then each implementation can provide a driver and the
> > suite
> > >>> > will
> > >>> > > use that for validation.
> > >>> > >
> > >>> > > By specifying both reads and writes for the driver we get a lot
> > more
> > >>> > power.
> > >>> > > Given an input we can roundtrip all combination of
> readers/writers
> > >>> and
> > >>> > > verify they can roundtrip.
> > >>> > >
> > >>> > > On Tue, Jun 25, 2024 at 6:42 PM Andrew Lamb <
> > >>> > andrewlamb11-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > >>> > >
> > >>> > > > FWIW I started hacking up a prototype[1] of what a
> > parquet-testing
> > >>> > > > integration suite might look like if anyone is interested
> > >>> > > >
> > >>> > > >
> > >>> > > >
> > >>> > > > [1]: https://github.com/apache/arrow-rs/pull/5956
> > >>> > > >
> > >>> > > > On Tue, Jun 18, 2024 at 10:39 AM Alkis Evlogimenos
> > >>> > > > <al...@databricks.com.invalid> wrote:
> > >>> > > >
> > >>> > > > > +1.
> > >>> > > > >
> > >>> > > > > I would suggest you address the comments first? I went
> through
> > >>> the
> > >>> > open
> > >>> > > > > ones and most of them make sense to me (and left few
> additional
> > >>> > > > comments).
> > >>> > > > >
> > >>> > > > > On Tue, Jun 18, 2024 at 12:42 PM Andrew Lamb <
> > >>> andrewlamb11@gmail.com
> > >>> > >
> > >>> > > > > wrote:
> > >>> > > > >
> > >>> > > > > > Thank you
> > >>> > > > > >
> > >>> > > > > > On Mon, Jun 17, 2024 at 11:40 PM Micah Kornfield <
> > >>> > > > emkornfield@gmail.com>
> > >>> > > > > > wrote:
> > >>> > > > > >
> > >>> > > > > > > Hi Andrew,
> > >>> > > > > > > I agree with this sentiment, I asked on the PR if there
> > would
> > >>> > be
> > >>> > > > > another
> > >>> > > > > > > pass and then I can merge it.
> > >>> > > > > > >
> > >>> > > > > > > Cheers,
> > >>> > > > > > > Micah
> > >>> > > > > > >
> > >>> > > > > > > On Fri, Jun 14, 2024 at 3:20 AM Andrew Lamb <
> > >>> > andrewlamb11@gmail.com>
> > >>> > > > > > > wrote:
> > >>> > > > > > >
> > >>> > > > > > > > Hello Parquet Devs,
> > >>> > > > > > > >
> > >>> > > > > > > > I propose we merge the first (admittedly bare bones)
> > >>> > > > "Implementation
> > >>> > > > > > > > Status" page PR [1] to the website soon. I think this
> > page
> > >>> is
> > >>> > vital
> > >>> > > > > to
> > >>> > > > > > > the
> > >>> > > > > > > > Parquet community (and to any attempt to extend the
> > >>> format) so
> > >>> > the
> > >>> > > > > > sooner
> > >>> > > > > > > > the better.
> > >>> > > > > > > >
> > >>> > > > > > > > The reason to merge the PR now is to have a base from
> > which
> > >>> > to
> > >>> > > > build.
> > >>> > > > > > > That
> > >>> > > > > > > > PR is already over a year old and has so many comments
> it
> > >>> is
> > >>> > hard
> > >>> > > > to
> > >>> > > > > > > follow
> > >>> > > > > > > > or know what the path forward is. If we insist on
> sorting
> > >>> all
> > >>> > the
> > >>> > > > > > details
> > >>> > > > > > > > out before we merge it I fear it will never merge.
> > >>> > > > > > > >
> > >>> > > > > > > > Once we have a page, I think the next steps are to add
> a
> > >>> > preamble
> > >>> > > > > > > > explaining what it is for and to start trying to fill
> out
> > >>> the
> > >>> > chart
> > >>> > > > > for
> > >>> > > > > > > an
> > >>> > > > > > > > implementation (I am happy to try for parquet-rs). I
> > >>> suspect
> > >>> > during
> > >>> > > > > > that
> > >>> > > > > > > > process we will have to adjust some of the charts more.
> > >>> > > > > > > >
> > >>> > > > > > > > Thank you for your consideration (and thank you for all
> > the
> > >>> > > > comments
> > >>> > > > > so
> > >>> > > > > > > > far)
> > >>> > > > > > > >
> > >>> > > > > > > > Andrew
> > >>> > > > > > > >
> > >>> > > > > > > > [1]: https://github.com/apache/parquet-site/pull/34
> > >>> > > > > > > >
> > >>> > > > > > >
> > >>> > > > > >
> > >>> > > > >
> > >>> > > >
> > >>> > >
> > >>> >
> > >>> >
> > >>> >
> > >>> >
> > >>>
> > >>
> >
>
Re: [DISCUSS] Merge initial Implementation Status PR and incrementally improve it
Posted by Micah Kornfield <em...@gmail.com>.
I just merged it.
Cheers,
Micah
On Thu, Jul 4, 2024 at 2:32 AM Andrew Lamb <an...@gmail.com> wrote:
> The initial feature matrix PR[1], has all outstanding comments addressed
> and is ready to merge from my opinion.
>
> Could one of the committers consider merging it so we can start filling the
> matrix out?
>
> Thanks,
> Andrew
>
> [1]: https://github.com/apache/parquet-site/pull/34
>
>
> On Wed, Jun 26, 2024 at 11:22 AM Andrew Lamb <an...@gmail.com>
> wrote:
>
> > I filed a ticket [1] to discuss integration testing / feature
> > compatibility testing
> >
> > [1]: https://github.com/apache/parquet-format/issues/441
> >
> > On Wed, Jun 26, 2024 at 10:03 AM Andrew Lamb <an...@gmail.com>
> > wrote:
> >
> >> I will file a ticket for discussion shortly
> >>
> >> On Wed, Jun 26, 2024, 06:55 Julien Le Dem <ju...@apache.org> wrote:
> >>
> >>> I agree with Antoine for the separate thread/ticket for integration
> >>> testing
> >>>
> >>> For the « implementation status » PR, I agree we can merge and
> iterate. I
> >>> added a couple of comments that can easily be addressed afterwards.
> >>> Julien
> >>>
> >>> On Wed, Jun 26, 2024 at 10:43 Antoine Pitrou <an...@python.org>
> wrote:
> >>>
> >>> >
> >>> > IMHO, we should either start a dedicated discussion thread for
> >>> > integration testing, or open a GH issue and discuss it there.
> >>> >
> >>> > Regards
> >>> >
> >>> > Antoine.
> >>> >
> >>> >
> >>> >
> >>> > On Wed, 26 Jun 2024 09:21:33 +0200
> >>> > Alkis Evlogimenos
> >>> > <al...@databricks.com.INVALID>
> >>> > wrote:
> >>> > > It would be nice if the integration suite specifies how a driver
> can
> >>> be
> >>> > > executed. Then each implementation can provide a driver and the
> suite
> >>> > will
> >>> > > use that for validation.
> >>> > >
> >>> > > By specifying both reads and writes for the driver we get a lot
> more
> >>> > power.
> >>> > > Given an input we can roundtrip all combination of readers/writers
> >>> and
> >>> > > verify they can roundtrip.
> >>> > >
> >>> > > On Tue, Jun 25, 2024 at 6:42 PM Andrew Lamb <
> >>> > andrewlamb11-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> >>> > >
> >>> > > > FWIW I started hacking up a prototype[1] of what a
> parquet-testing
> >>> > > > integration suite might look like if anyone is interested
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > > > [1]: https://github.com/apache/arrow-rs/pull/5956
> >>> > > >
> >>> > > > On Tue, Jun 18, 2024 at 10:39 AM Alkis Evlogimenos
> >>> > > > <al...@databricks.com.invalid> wrote:
> >>> > > >
> >>> > > > > +1.
> >>> > > > >
> >>> > > > > I would suggest you address the comments first? I went through
> >>> the
> >>> > open
> >>> > > > > ones and most of them make sense to me (and left few additional
> >>> > > > comments).
> >>> > > > >
> >>> > > > > On Tue, Jun 18, 2024 at 12:42 PM Andrew Lamb <
> >>> andrewlamb11@gmail.com
> >>> > >
> >>> > > > > wrote:
> >>> > > > >
> >>> > > > > > Thank you
> >>> > > > > >
> >>> > > > > > On Mon, Jun 17, 2024 at 11:40 PM Micah Kornfield <
> >>> > > > emkornfield@gmail.com>
> >>> > > > > > wrote:
> >>> > > > > >
> >>> > > > > > > Hi Andrew,
> >>> > > > > > > I agree with this sentiment, I asked on the PR if there
> would
> >>> > be
> >>> > > > > another
> >>> > > > > > > pass and then I can merge it.
> >>> > > > > > >
> >>> > > > > > > Cheers,
> >>> > > > > > > Micah
> >>> > > > > > >
> >>> > > > > > > On Fri, Jun 14, 2024 at 3:20 AM Andrew Lamb <
> >>> > andrewlamb11@gmail.com>
> >>> > > > > > > wrote:
> >>> > > > > > >
> >>> > > > > > > > Hello Parquet Devs,
> >>> > > > > > > >
> >>> > > > > > > > I propose we merge the first (admittedly bare bones)
> >>> > > > "Implementation
> >>> > > > > > > > Status" page PR [1] to the website soon. I think this
> page
> >>> is
> >>> > vital
> >>> > > > > to
> >>> > > > > > > the
> >>> > > > > > > > Parquet community (and to any attempt to extend the
> >>> format) so
> >>> > the
> >>> > > > > > sooner
> >>> > > > > > > > the better.
> >>> > > > > > > >
> >>> > > > > > > > The reason to merge the PR now is to have a base from
> which
> >>> > to
> >>> > > > build.
> >>> > > > > > > That
> >>> > > > > > > > PR is already over a year old and has so many comments it
> >>> is
> >>> > hard
> >>> > > > to
> >>> > > > > > > follow
> >>> > > > > > > > or know what the path forward is. If we insist on sorting
> >>> all
> >>> > the
> >>> > > > > > details
> >>> > > > > > > > out before we merge it I fear it will never merge.
> >>> > > > > > > >
> >>> > > > > > > > Once we have a page, I think the next steps are to add a
> >>> > preamble
> >>> > > > > > > > explaining what it is for and to start trying to fill out
> >>> the
> >>> > chart
> >>> > > > > for
> >>> > > > > > > an
> >>> > > > > > > > implementation (I am happy to try for parquet-rs). I
> >>> suspect
> >>> > during
> >>> > > > > > that
> >>> > > > > > > > process we will have to adjust some of the charts more.
> >>> > > > > > > >
> >>> > > > > > > > Thank you for your consideration (and thank you for all
> the
> >>> > > > comments
> >>> > > > > so
> >>> > > > > > > > far)
> >>> > > > > > > >
> >>> > > > > > > > Andrew
> >>> > > > > > > >
> >>> > > > > > > > [1]: https://github.com/apache/parquet-site/pull/34
> >>> > > > > > > >
> >>> > > > > > >
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>> >
> >>> >
> >>> >
> >>>
> >>
>
Re: [DISCUSS] Merge initial Implementation Status PR and incrementally improve it
Posted by Micah Kornfield <em...@gmail.com>.
I just merged it.
Cheers,
Micah
On Thu, Jul 4, 2024 at 2:32 AM Andrew Lamb <an...@gmail.com> wrote:
> The initial feature matrix PR[1], has all outstanding comments addressed
> and is ready to merge from my opinion.
>
> Could one of the committers consider merging it so we can start filling the
> matrix out?
>
> Thanks,
> Andrew
>
> [1]: https://github.com/apache/parquet-site/pull/34
>
>
> On Wed, Jun 26, 2024 at 11:22 AM Andrew Lamb <an...@gmail.com>
> wrote:
>
> > I filed a ticket [1] to discuss integration testing / feature
> > compatibility testing
> >
> > [1]: https://github.com/apache/parquet-format/issues/441
> >
> > On Wed, Jun 26, 2024 at 10:03 AM Andrew Lamb <an...@gmail.com>
> > wrote:
> >
> >> I will file a ticket for discussion shortly
> >>
> >> On Wed, Jun 26, 2024, 06:55 Julien Le Dem <ju...@apache.org> wrote:
> >>
> >>> I agree with Antoine for the separate thread/ticket for integration
> >>> testing
> >>>
> >>> For the « implementation status » PR, I agree we can merge and
> iterate. I
> >>> added a couple of comments that can easily be addressed afterwards.
> >>> Julien
> >>>
> >>> On Wed, Jun 26, 2024 at 10:43 Antoine Pitrou <an...@python.org>
> wrote:
> >>>
> >>> >
> >>> > IMHO, we should either start a dedicated discussion thread for
> >>> > integration testing, or open a GH issue and discuss it there.
> >>> >
> >>> > Regards
> >>> >
> >>> > Antoine.
> >>> >
> >>> >
> >>> >
> >>> > On Wed, 26 Jun 2024 09:21:33 +0200
> >>> > Alkis Evlogimenos
> >>> > <al...@databricks.com.INVALID>
> >>> > wrote:
> >>> > > It would be nice if the integration suite specifies how a driver
> can
> >>> be
> >>> > > executed. Then each implementation can provide a driver and the
> suite
> >>> > will
> >>> > > use that for validation.
> >>> > >
> >>> > > By specifying both reads and writes for the driver we get a lot
> more
> >>> > power.
> >>> > > Given an input we can roundtrip all combination of readers/writers
> >>> and
> >>> > > verify they can roundtrip.
> >>> > >
> >>> > > On Tue, Jun 25, 2024 at 6:42 PM Andrew Lamb <
> >>> > andrewlamb11-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> >>> > >
> >>> > > > FWIW I started hacking up a prototype[1] of what a
> parquet-testing
> >>> > > > integration suite might look like if anyone is interested
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > > > [1]: https://github.com/apache/arrow-rs/pull/5956
> >>> > > >
> >>> > > > On Tue, Jun 18, 2024 at 10:39 AM Alkis Evlogimenos
> >>> > > > <al...@databricks.com.invalid> wrote:
> >>> > > >
> >>> > > > > +1.
> >>> > > > >
> >>> > > > > I would suggest you address the comments first? I went through
> >>> the
> >>> > open
> >>> > > > > ones and most of them make sense to me (and left few additional
> >>> > > > comments).
> >>> > > > >
> >>> > > > > On Tue, Jun 18, 2024 at 12:42 PM Andrew Lamb <
> >>> andrewlamb11@gmail.com
> >>> > >
> >>> > > > > wrote:
> >>> > > > >
> >>> > > > > > Thank you
> >>> > > > > >
> >>> > > > > > On Mon, Jun 17, 2024 at 11:40 PM Micah Kornfield <
> >>> > > > emkornfield@gmail.com>
> >>> > > > > > wrote:
> >>> > > > > >
> >>> > > > > > > Hi Andrew,
> >>> > > > > > > I agree with this sentiment, I asked on the PR if there
> would
> >>> > be
> >>> > > > > another
> >>> > > > > > > pass and then I can merge it.
> >>> > > > > > >
> >>> > > > > > > Cheers,
> >>> > > > > > > Micah
> >>> > > > > > >
> >>> > > > > > > On Fri, Jun 14, 2024 at 3:20 AM Andrew Lamb <
> >>> > andrewlamb11@gmail.com>
> >>> > > > > > > wrote:
> >>> > > > > > >
> >>> > > > > > > > Hello Parquet Devs,
> >>> > > > > > > >
> >>> > > > > > > > I propose we merge the first (admittedly bare bones)
> >>> > > > "Implementation
> >>> > > > > > > > Status" page PR [1] to the website soon. I think this
> page
> >>> is
> >>> > vital
> >>> > > > > to
> >>> > > > > > > the
> >>> > > > > > > > Parquet community (and to any attempt to extend the
> >>> format) so
> >>> > the
> >>> > > > > > sooner
> >>> > > > > > > > the better.
> >>> > > > > > > >
> >>> > > > > > > > The reason to merge the PR now is to have a base from
> which
> >>> > to
> >>> > > > build.
> >>> > > > > > > That
> >>> > > > > > > > PR is already over a year old and has so many comments it
> >>> is
> >>> > hard
> >>> > > > to
> >>> > > > > > > follow
> >>> > > > > > > > or know what the path forward is. If we insist on sorting
> >>> all
> >>> > the
> >>> > > > > > details
> >>> > > > > > > > out before we merge it I fear it will never merge.
> >>> > > > > > > >
> >>> > > > > > > > Once we have a page, I think the next steps are to add a
> >>> > preamble
> >>> > > > > > > > explaining what it is for and to start trying to fill out
> >>> the
> >>> > chart
> >>> > > > > for
> >>> > > > > > > an
> >>> > > > > > > > implementation (I am happy to try for parquet-rs). I
> >>> suspect
> >>> > during
> >>> > > > > > that
> >>> > > > > > > > process we will have to adjust some of the charts more.
> >>> > > > > > > >
> >>> > > > > > > > Thank you for your consideration (and thank you for all
> the
> >>> > > > comments
> >>> > > > > so
> >>> > > > > > > > far)
> >>> > > > > > > >
> >>> > > > > > > > Andrew
> >>> > > > > > > >
> >>> > > > > > > > [1]: https://github.com/apache/parquet-site/pull/34
> >>> > > > > > > >
> >>> > > > > > >
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>> >
> >>> >
> >>> >
> >>>
> >>
>