You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Gang Wu <us...@gmail.com> on 2024/05/11 02:08:28 UTC
Re: Fwd: [C++] Parquet and Arrow overlap
I can initiate the vote. But before the vote, I think we need to revisit
the states of all unresolved tickets and close some as needed.
BTW, do we really need to make a full copy of them to have a mirror
in the Arrow GitHub issues?
I'd like to seek a consensus here before sending the vote.
Best,
Gang
On Sat, May 11, 2024 at 8:46 AM Jacob Wujciak <as...@apache.org> wrote:
> Hello Everyone!
>
> It seems there is general agreement on this topic, it would be great if a
> committer/PMC could start a (lazy consensus) procedural vote.
>
> I will inquire how to handle the parquet-cpp component in jira (ideally
> disabling it, not removing).
> There are currently only ~70 open tickets for parquet-cpp, with the change
> in repo it is probably easier to just move open tickets but I'll leave that
> to Rok who managed the transition of Arrows 20k+ tickets too :D
>
> Thanks,
> Jacob
>
> Arrow committer
>
> On 2024/04/25 05:31:18 Gang Wu wrote:
> > I know we have some non-Java committers and PMCs. But after the
> parquet-cpp
> > donation, it seems that no one worked on Parquet from arrow (cpp, rust,
> go,
> > etc.)
> > and other projects are promoted as a Parquet committer. It would be
> > inconvenient
> > for non-Java Parquet developers to work with apache/parquet-format and
> > apache/parquet-testing repositories. Furthermore, votes from these
> > developers
> > are not binding for a format change in the ML.
> >
> > Best,
> > Gang
> >
> > On Wed, Apr 24, 2024 at 8:42 PM Uwe L. Korn <uw...@xhochy.com> wrote:
> >
> > > > Should we consider
> > > > Parquet developers from other projects than parquet-mr as Parquet
> > > commuters?
> > >
> > > We are doing this (speaking as a Parquet PMC who didn't work on
> > > parquet-mr, but parquet-cpp).
> > >
> > > Best
> > > Uwe
> > >
> > > On Wed, Apr 24, 2024, at 2:38 PM, Gang Wu wrote:
> > > > +1 for moving parquet-cpp issues from Apache Jira to Arrow's GitHub
> > > issue.
> > > >
> > > > Besides, I want to echo Will's question in the thread. Should we
> consider
> > > > Parquet developers from other projects than parquet-mr as Parquet
> > > commiters?
> > > > Currently apache/parquet-format and apache/parquet-testing
> repositories
> > > are
> > > > solely governed by Apache Parquet PMC. It would be better for the
> entire
> > > > Parquet community if developers with sufficient contributions to open
> > > source
> > > > Parquet projects (including but not limited to parquet-cpp, arrow-rs,
> > > cudf,
> > > > etc.)
> > > > can be considered as Parquet committer and PMC.
> > > >
> > > > Best,
> > > > Gang
> > > >
> > > > On Wed, Apr 24, 2024 at 7:04 PM Uwe L. Korn <uw...@xhochy.com>
> wrote:
> > > >
> > > >> I would be very supportive of this move. The Parquet C++
> development has
> > > >> been under the umbrella of the Arrow repository for more than
> five(six?)
> > > >> years now. Thus, the issues should also be aligned with the Arrow
> > > project.
> > > >>
> > > >> Uwe
> > > >>
> > > >> On Tue, Apr 23, 2024, at 8:27 PM, Rok Mihevc wrote:
> > > >> > Bumping this thread again to see if there is will to call for a
> vote
> > > and
> > > >> > move parquet-cpp issues from Apache Jira to Arrow's GitHub issue
> as
> > > was
> > > >> > done for Arrow.
> > > >> > I'm willing to do the move as I already did it for Arrow.
> > > >> >
> > > >> > Rok
> > > >> >
> > > >> > On Sat, Apr 15, 2023 at 4:53 AM Micah Kornfield <
> > > emkornfield@apache.org>
> > > >> > wrote:
> > > >> >
> > > >> >> Bumping this thread again to see in any Parquet PMC members can
> chime
> > > >> >> in/maybe start a formal vote to move governance of Parquet-CPP
> under
> > > the
> > > >> >> umbrella.
> > > >> >>
> > > >> >> -Micah
> > > >> >>
> > > >> >> On 2023/02/02 10:34:25 Antoine Pitrou wrote:
> > > >> >> >
> > > >> >> >
> > > >> >> > Hi Will,
> > > >> >> >
> > > >> >> > Le 01/02/2023 à 20:27, Will Jones a écrit :
> > > >> >> > >
> > > >> >> > > First, it's not obvious where issues are supposed to be
> open: In
> > > >> >> Parquet
> > > >> >> > > Jira or Arrow GitHub issues. Looking back at some of the
> original
> > > >> >> > > discussion, it looks like the intention was
> > > >> >> > >
> > > >> >> > > * use PARQUET-XXX for issues relating to Parquet core
> > > >> >> > >> * use ARROW-XXX for issues relation to Arrow's consumption
> of
> > > >> Parquet
> > > >> >> > >> core (e.g. changes that are in parquet/arrow right now)
> > > >> >> > >>
> > > >> >> > > The README for the old parquet-cpp repo [3] states instead in
> > > it's
> > > >> >> > > migration note:
> > > >> >> > >
> > > >> >> > > JIRA issues should continue to be opened in the PARQUET
> JIRA
> > > >> project.
> > > >> >> > >
> > > >> >> > > Either way, it doesn't seem like this process is obvious to
> > > people.
> > > >> >> Perhaps
> > > >> >> > > we could clarify this and add notices to Arrow's GitHub
> issues
> > > >> >> template?
> > > >> >> >
> > > >> >> > I agree we should clarify this. I have no personal preference,
> but
> > > I
> > > >> >> will note
> > > >> >> > that Github issues decrease friction as having a GH account is
> > > already
> > > >> >> necessary
> > > >> >> > for submitting PRs.
> > > >> >> >
> > > >> >> > > Second, committer status is a little unclear. I am a
> committer on
> > > >> >> Arrow,
> > > >> >> > > but not on Parquet right now. Does that mean I should only
> merge
> > > >> >> Parquet
> > > >> >> > > C++ PRs for code changes in parquet/arrow? Or that I
> shouldn't
> > > merge
> > > >> >> > > Parquet changes at all?
> > > >> >> >
> > > >> >> > Since Parquet C++ is part of Arrow C++, you are allowed to
> merge
> > > >> Parquet
> > > >> >> C++
> > > >> >> > changes. As always you should ensure you have sufficient
> > > understanding
> > > >> >> of the
> > > >> >> > contribution, and that it follows established practices:
> > > >> >> > https://arrow.apache.org/docs/dev/developers/reviewing.html
> > > >> >> >
> > > >> >> > > Also, are the contributions to Arrow C++ Parquet being
> actively
> > > >> >> reviewed
> > > >> >> > > for potential new committers?
> > > >> >> >
> > > >> >> > I would certainly do.
> > > >> >> >
> > > >> >> > Regards
> > > >> >> >
> > > >> >> > Antoine.
> > > >> >> >
> > > >> >> >
> > > >> >>
> > > >>
> > >
> >
>
Re: Fwd: [C++] Parquet and Arrow overlap
Posted by Gang Wu <us...@gmail.com>.
I have just finished a round of checking parquet-cpp open issues
and resolved some of them which I believe are completed. I will
start a vote next week.
Once the migration is done, what should we do with the Parquet
tickets? Now all the Arrow tickets are immutable. However, Parquet
tickets should still be used for parquet-mr, parquet-format, etc. I'm
not sure if it is enough to delete the parquet-cpp component from
the Parquet JIRA project to prevent new tickets against parquet-cpp.
> Another option would be a new parquet monorepo.
Considering the dependency below, I think a new parquet monorepo
is hard to achieve:
arrow core -> parquet-cpp -> arrow dataset -> pyarrow
Best,
Gang
On Sun, May 12, 2024 at 8:13 AM Rok Mihevc <ro...@gmail.com> wrote:
> > BTW, do we really need to make a full copy of them to have a mirror in
> the Arrow GitHub issues?
>
> I'm assuming we want to move all parquet-cpp issues (open and closed) to
> Arrow's GitHub issue tracker. The amount of effort required is about the
> same if we move all or just open tickets. When migrating Arrow's issues we
> decided to move all so as to have them searchable in one place [1].
>
> We could also use the opportunity to move issues related to other
> implementations to relevant GitHub issue tracker if there is interest. I
> think Arrow's issue tracker has much less friction than the ASF Jira.
>
> > As activity picks up, if the desire is to build more community around
> > Parquet, perhaps the Parquet PMC wants to encourage moving code back to
> > repositories managed by parquet (and out of arrow, for example). I
> realize
> > this would be a technical burden, but it might clarify communities and
> > committers.
>
> Another option would be a new parquet monorepo.
>
> [1] https://github.com/apache/arrow/issues/14546
>
> Rok
>
> On Sat, May 11, 2024 at 11:51 AM Andrew Lamb <an...@gmail.com>
> wrote:
>
> > It is great to see some additional enthusiasm and momentum around the
> > Apache Parquet implementation (congratulations on the release of
> parquet-mr
> > 1.14[1]!).
> >
> > As activity picks up, if the desire is to build more community around
> > Parquet, perhaps the Parquet PMC wants to encourage moving code back to
> > repositories managed by parquet (and out of arrow, for example). I
> realize
> > this would be a technical burden, but it might clarify communities and
> > committers.
> >
> > Andrew
> >
> > [1]: https://lists.apache.org/thread/2gggm938z0x9fx3wtwctfm5htsxlf3z4
> >
> >
> >
> > On Fri, May 10, 2024 at 11:45 PM Matt Topol <zo...@gmail.com>
> > wrote:
> >
> > > I just wanted to also poke the question of non-Java developers who have
> > > worked on the other parquet implementations potentially being
> recognized
> > as
> > > committers or otherwise on the Parquet project (speaking as the primary
> > > developer of the Go parquet implementation which also lives in the
> Arrow
> > > repository). It would be great to see some active contributors to
> > > parquet-cpp, parquet-go, or otherwise not just being considered but
> > > actively becoming committers.
> > >
> > > That's just my two cents from a community perspective.
> > >
> > > --Matt
> > >
> > > On Fri, May 10, 2024, 10:35 PM Jacob Wujciak <as...@apache.org>
> > > wrote:
> > >
> > > > Thank you, that sounds great! On first glance some seem to be rather
> > old
> > > > and probably don't apply anymore.
> > > >
> > > > > BTW, do we really need to make a full copy of them to have a mirror
> > in
> > > > the Arrow GitHub issues?
> > > >
> > > > I am not sure I understand what you mean? A full copy of the
> > > > open/closed/all issues? I'd say only the (remaining) open issues
> would
> > be
> > > > fine.
> > > > For reference this is what an imported issue looks like:
> > > > https://github.com/apache/arrow/issues/30543
> > > >
> > > > Am Sa., 11. Mai 2024 um 04:09 Uhr schrieb Gang Wu <ustcwg@gmail.com
> >:
> > > >
> > > > > I can initiate the vote. But before the vote, I think we need to
> > > revisit
> > > > > the states of all unresolved tickets and close some as needed.
> > > > >
> > > > > BTW, do we really need to make a full copy of them to have a mirror
> > > > > in the Arrow GitHub issues?
> > > > >
> > > > > I'd like to seek a consensus here before sending the vote.
> > > > >
> > > > > Best,
> > > > > Gang
> > > > >
> > > > > On Sat, May 11, 2024 at 8:46 AM Jacob Wujciak <
> assignuser@apache.org
> > >
> > > > > wrote:
> > > > >
> > > > > > Hello Everyone!
> > > > > >
> > > > > > It seems there is general agreement on this topic, it would be
> > great
> > > > if a
> > > > > > committer/PMC could start a (lazy consensus) procedural vote.
> > > > > >
> > > > > > I will inquire how to handle the parquet-cpp component in jira
> > > (ideally
> > > > > > disabling it, not removing).
> > > > > > There are currently only ~70 open tickets for parquet-cpp, with
> the
> > > > > change
> > > > > > in repo it is probably easier to just move open tickets but I'll
> > > leave
> > > > > that
> > > > > > to Rok who managed the transition of Arrows 20k+ tickets too :D
> > > > > >
> > > > > > Thanks,
> > > > > > Jacob
> > > > > >
> > > > > > Arrow committer
> > > > > >
> > > > > > On 2024/04/25 05:31:18 Gang Wu wrote:
> > > > > > > I know we have some non-Java committers and PMCs. But after the
> > > > > > parquet-cpp
> > > > > > > donation, it seems that no one worked on Parquet from arrow
> (cpp,
> > > > rust,
> > > > > > go,
> > > > > > > etc.)
> > > > > > > and other projects are promoted as a Parquet committer. It
> would
> > be
> > > > > > > inconvenient
> > > > > > > for non-Java Parquet developers to work with
> > apache/parquet-format
> > > > and
> > > > > > > apache/parquet-testing repositories. Furthermore, votes from
> > these
> > > > > > > developers
> > > > > > > are not binding for a format change in the ML.
> > > > > > >
> > > > > > > Best,
> > > > > > > Gang
> > > > > > >
> > > > > > > On Wed, Apr 24, 2024 at 8:42 PM Uwe L. Korn <uw...@xhochy.com>
> > > > wrote:
> > > > > > >
> > > > > > > > > Should we consider
> > > > > > > > > Parquet developers from other projects than parquet-mr as
> > > Parquet
> > > > > > > > commuters?
> > > > > > > >
> > > > > > > > We are doing this (speaking as a Parquet PMC who didn't work
> on
> > > > > > > > parquet-mr, but parquet-cpp).
> > > > > > > >
> > > > > > > > Best
> > > > > > > > Uwe
> > > > > > > >
> > > > > > > > On Wed, Apr 24, 2024, at 2:38 PM, Gang Wu wrote:
> > > > > > > > > +1 for moving parquet-cpp issues from Apache Jira to
> Arrow's
> > > > GitHub
> > > > > > > > issue.
> > > > > > > > >
> > > > > > > > > Besides, I want to echo Will's question in the thread.
> Should
> > > we
> > > > > > consider
> > > > > > > > > Parquet developers from other projects than parquet-mr as
> > > Parquet
> > > > > > > > commiters?
> > > > > > > > > Currently apache/parquet-format and apache/parquet-testing
> > > > > > repositories
> > > > > > > > are
> > > > > > > > > solely governed by Apache Parquet PMC. It would be better
> for
> > > the
> > > > > > entire
> > > > > > > > > Parquet community if developers with sufficient
> contributions
> > > to
> > > > > open
> > > > > > > > source
> > > > > > > > > Parquet projects (including but not limited to parquet-cpp,
> > > > > arrow-rs,
> > > > > > > > cudf,
> > > > > > > > > etc.)
> > > > > > > > > can be considered as Parquet committer and PMC.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Gang
> > > > > > > > >
> > > > > > > > > On Wed, Apr 24, 2024 at 7:04 PM Uwe L. Korn <
> > uwelk@xhochy.com>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> I would be very supportive of this move. The Parquet C++
> > > > > > development has
> > > > > > > > >> been under the umbrella of the Arrow repository for more
> > than
> > > > > > five(six?)
> > > > > > > > >> years now. Thus, the issues should also be aligned with
> the
> > > > Arrow
> > > > > > > > project.
> > > > > > > > >>
> > > > > > > > >> Uwe
> > > > > > > > >>
> > > > > > > > >> On Tue, Apr 23, 2024, at 8:27 PM, Rok Mihevc wrote:
> > > > > > > > >> > Bumping this thread again to see if there is will to
> call
> > > for
> > > > a
> > > > > > vote
> > > > > > > > and
> > > > > > > > >> > move parquet-cpp issues from Apache Jira to Arrow's
> GitHub
> > > > issue
> > > > > > as
> > > > > > > > was
> > > > > > > > >> > done for Arrow.
> > > > > > > > >> > I'm willing to do the move as I already did it for
> Arrow.
> > > > > > > > >> >
> > > > > > > > >> > Rok
> > > > > > > > >> >
> > > > > > > > >> > On Sat, Apr 15, 2023 at 4:53 AM Micah Kornfield <
> > > > > > > > emkornfield@apache.org>
> > > > > > > > >> > wrote:
> > > > > > > > >> >
> > > > > > > > >> >> Bumping this thread again to see in any Parquet PMC
> > members
> > > > can
> > > > > > chime
> > > > > > > > >> >> in/maybe start a formal vote to move governance of
> > > > Parquet-CPP
> > > > > > under
> > > > > > > > the
> > > > > > > > >> >> umbrella.
> > > > > > > > >> >>
> > > > > > > > >> >> -Micah
> > > > > > > > >> >>
> > > > > > > > >> >> On 2023/02/02 10:34:25 Antoine Pitrou wrote:
> > > > > > > > >> >> >
> > > > > > > > >> >> >
> > > > > > > > >> >> > Hi Will,
> > > > > > > > >> >> >
> > > > > > > > >> >> > Le 01/02/2023 à 20:27, Will Jones a écrit :
> > > > > > > > >> >> > >
> > > > > > > > >> >> > > First, it's not obvious where issues are supposed
> to
> > be
> > > > > > open: In
> > > > > > > > >> >> Parquet
> > > > > > > > >> >> > > Jira or Arrow GitHub issues. Looking back at some
> of
> > > the
> > > > > > original
> > > > > > > > >> >> > > discussion, it looks like the intention was
> > > > > > > > >> >> > >
> > > > > > > > >> >> > > * use PARQUET-XXX for issues relating to Parquet
> core
> > > > > > > > >> >> > >> * use ARROW-XXX for issues relation to Arrow's
> > > > consumption
> > > > > > of
> > > > > > > > >> Parquet
> > > > > > > > >> >> > >> core (e.g. changes that are in parquet/arrow right
> > > now)
> > > > > > > > >> >> > >>
> > > > > > > > >> >> > > The README for the old parquet-cpp repo [3] states
> > > > instead
> > > > > in
> > > > > > > > it's
> > > > > > > > >> >> > > migration note:
> > > > > > > > >> >> > >
> > > > > > > > >> >> > > JIRA issues should continue to be opened in the
> > > PARQUET
> > > > > > JIRA
> > > > > > > > >> project.
> > > > > > > > >> >> > >
> > > > > > > > >> >> > > Either way, it doesn't seem like this process is
> > > obvious
> > > > to
> > > > > > > > people.
> > > > > > > > >> >> Perhaps
> > > > > > > > >> >> > > we could clarify this and add notices to Arrow's
> > GitHub
> > > > > > issues
> > > > > > > > >> >> template?
> > > > > > > > >> >> >
> > > > > > > > >> >> > I agree we should clarify this. I have no personal
> > > > > preference,
> > > > > > but
> > > > > > > > I
> > > > > > > > >> >> will note
> > > > > > > > >> >> > that Github issues decrease friction as having a GH
> > > account
> > > > > is
> > > > > > > > already
> > > > > > > > >> >> necessary
> > > > > > > > >> >> > for submitting PRs.
> > > > > > > > >> >> >
> > > > > > > > >> >> > > Second, committer status is a little unclear. I am
> a
> > > > > > committer on
> > > > > > > > >> >> Arrow,
> > > > > > > > >> >> > > but not on Parquet right now. Does that mean I
> should
> > > > only
> > > > > > merge
> > > > > > > > >> >> Parquet
> > > > > > > > >> >> > > C++ PRs for code changes in parquet/arrow? Or that
> I
> > > > > > shouldn't
> > > > > > > > merge
> > > > > > > > >> >> > > Parquet changes at all?
> > > > > > > > >> >> >
> > > > > > > > >> >> > Since Parquet C++ is part of Arrow C++, you are
> allowed
> > > to
> > > > > > merge
> > > > > > > > >> Parquet
> > > > > > > > >> >> C++
> > > > > > > > >> >> > changes. As always you should ensure you have
> > sufficient
> > > > > > > > understanding
> > > > > > > > >> >> of the
> > > > > > > > >> >> > contribution, and that it follows established
> > practices:
> > > > > > > > >> >> >
> > > > https://arrow.apache.org/docs/dev/developers/reviewing.html
> > > > > > > > >> >> >
> > > > > > > > >> >> > > Also, are the contributions to Arrow C++ Parquet
> > being
> > > > > > actively
> > > > > > > > >> >> reviewed
> > > > > > > > >> >> > > for potential new committers?
> > > > > > > > >> >> >
> > > > > > > > >> >> > I would certainly do.
> > > > > > > > >> >> >
> > > > > > > > >> >> > Regards
> > > > > > > > >> >> >
> > > > > > > > >> >> > Antoine.
> > > > > > > > >> >> >
> > > > > > > > >> >> >
> > > > > > > > >> >>
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Fwd: [C++] Parquet and Arrow overlap
Posted by Gang Wu <us...@gmail.com>.
I have just finished a round of checking parquet-cpp open issues
and resolved some of them which I believe are completed. I will
start a vote next week.
Once the migration is done, what should we do with the Parquet
tickets? Now all the Arrow tickets are immutable. However, Parquet
tickets should still be used for parquet-mr, parquet-format, etc. I'm
not sure if it is enough to delete the parquet-cpp component from
the Parquet JIRA project to prevent new tickets against parquet-cpp.
> Another option would be a new parquet monorepo.
Considering the dependency below, I think a new parquet monorepo
is hard to achieve:
arrow core -> parquet-cpp -> arrow dataset -> pyarrow
Best,
Gang
On Sun, May 12, 2024 at 8:13 AM Rok Mihevc <ro...@gmail.com> wrote:
> > BTW, do we really need to make a full copy of them to have a mirror in
> the Arrow GitHub issues?
>
> I'm assuming we want to move all parquet-cpp issues (open and closed) to
> Arrow's GitHub issue tracker. The amount of effort required is about the
> same if we move all or just open tickets. When migrating Arrow's issues we
> decided to move all so as to have them searchable in one place [1].
>
> We could also use the opportunity to move issues related to other
> implementations to relevant GitHub issue tracker if there is interest. I
> think Arrow's issue tracker has much less friction than the ASF Jira.
>
> > As activity picks up, if the desire is to build more community around
> > Parquet, perhaps the Parquet PMC wants to encourage moving code back to
> > repositories managed by parquet (and out of arrow, for example). I
> realize
> > this would be a technical burden, but it might clarify communities and
> > committers.
>
> Another option would be a new parquet monorepo.
>
> [1] https://github.com/apache/arrow/issues/14546
>
> Rok
>
> On Sat, May 11, 2024 at 11:51 AM Andrew Lamb <an...@gmail.com>
> wrote:
>
> > It is great to see some additional enthusiasm and momentum around the
> > Apache Parquet implementation (congratulations on the release of
> parquet-mr
> > 1.14[1]!).
> >
> > As activity picks up, if the desire is to build more community around
> > Parquet, perhaps the Parquet PMC wants to encourage moving code back to
> > repositories managed by parquet (and out of arrow, for example). I
> realize
> > this would be a technical burden, but it might clarify communities and
> > committers.
> >
> > Andrew
> >
> > [1]: https://lists.apache.org/thread/2gggm938z0x9fx3wtwctfm5htsxlf3z4
> >
> >
> >
> > On Fri, May 10, 2024 at 11:45 PM Matt Topol <zo...@gmail.com>
> > wrote:
> >
> > > I just wanted to also poke the question of non-Java developers who have
> > > worked on the other parquet implementations potentially being
> recognized
> > as
> > > committers or otherwise on the Parquet project (speaking as the primary
> > > developer of the Go parquet implementation which also lives in the
> Arrow
> > > repository). It would be great to see some active contributors to
> > > parquet-cpp, parquet-go, or otherwise not just being considered but
> > > actively becoming committers.
> > >
> > > That's just my two cents from a community perspective.
> > >
> > > --Matt
> > >
> > > On Fri, May 10, 2024, 10:35 PM Jacob Wujciak <as...@apache.org>
> > > wrote:
> > >
> > > > Thank you, that sounds great! On first glance some seem to be rather
> > old
> > > > and probably don't apply anymore.
> > > >
> > > > > BTW, do we really need to make a full copy of them to have a mirror
> > in
> > > > the Arrow GitHub issues?
> > > >
> > > > I am not sure I understand what you mean? A full copy of the
> > > > open/closed/all issues? I'd say only the (remaining) open issues
> would
> > be
> > > > fine.
> > > > For reference this is what an imported issue looks like:
> > > > https://github.com/apache/arrow/issues/30543
> > > >
> > > > Am Sa., 11. Mai 2024 um 04:09 Uhr schrieb Gang Wu <ustcwg@gmail.com
> >:
> > > >
> > > > > I can initiate the vote. But before the vote, I think we need to
> > > revisit
> > > > > the states of all unresolved tickets and close some as needed.
> > > > >
> > > > > BTW, do we really need to make a full copy of them to have a mirror
> > > > > in the Arrow GitHub issues?
> > > > >
> > > > > I'd like to seek a consensus here before sending the vote.
> > > > >
> > > > > Best,
> > > > > Gang
> > > > >
> > > > > On Sat, May 11, 2024 at 8:46 AM Jacob Wujciak <
> assignuser@apache.org
> > >
> > > > > wrote:
> > > > >
> > > > > > Hello Everyone!
> > > > > >
> > > > > > It seems there is general agreement on this topic, it would be
> > great
> > > > if a
> > > > > > committer/PMC could start a (lazy consensus) procedural vote.
> > > > > >
> > > > > > I will inquire how to handle the parquet-cpp component in jira
> > > (ideally
> > > > > > disabling it, not removing).
> > > > > > There are currently only ~70 open tickets for parquet-cpp, with
> the
> > > > > change
> > > > > > in repo it is probably easier to just move open tickets but I'll
> > > leave
> > > > > that
> > > > > > to Rok who managed the transition of Arrows 20k+ tickets too :D
> > > > > >
> > > > > > Thanks,
> > > > > > Jacob
> > > > > >
> > > > > > Arrow committer
> > > > > >
> > > > > > On 2024/04/25 05:31:18 Gang Wu wrote:
> > > > > > > I know we have some non-Java committers and PMCs. But after the
> > > > > > parquet-cpp
> > > > > > > donation, it seems that no one worked on Parquet from arrow
> (cpp,
> > > > rust,
> > > > > > go,
> > > > > > > etc.)
> > > > > > > and other projects are promoted as a Parquet committer. It
> would
> > be
> > > > > > > inconvenient
> > > > > > > for non-Java Parquet developers to work with
> > apache/parquet-format
> > > > and
> > > > > > > apache/parquet-testing repositories. Furthermore, votes from
> > these
> > > > > > > developers
> > > > > > > are not binding for a format change in the ML.
> > > > > > >
> > > > > > > Best,
> > > > > > > Gang
> > > > > > >
> > > > > > > On Wed, Apr 24, 2024 at 8:42 PM Uwe L. Korn <uw...@xhochy.com>
> > > > wrote:
> > > > > > >
> > > > > > > > > Should we consider
> > > > > > > > > Parquet developers from other projects than parquet-mr as
> > > Parquet
> > > > > > > > commuters?
> > > > > > > >
> > > > > > > > We are doing this (speaking as a Parquet PMC who didn't work
> on
> > > > > > > > parquet-mr, but parquet-cpp).
> > > > > > > >
> > > > > > > > Best
> > > > > > > > Uwe
> > > > > > > >
> > > > > > > > On Wed, Apr 24, 2024, at 2:38 PM, Gang Wu wrote:
> > > > > > > > > +1 for moving parquet-cpp issues from Apache Jira to
> Arrow's
> > > > GitHub
> > > > > > > > issue.
> > > > > > > > >
> > > > > > > > > Besides, I want to echo Will's question in the thread.
> Should
> > > we
> > > > > > consider
> > > > > > > > > Parquet developers from other projects than parquet-mr as
> > > Parquet
> > > > > > > > commiters?
> > > > > > > > > Currently apache/parquet-format and apache/parquet-testing
> > > > > > repositories
> > > > > > > > are
> > > > > > > > > solely governed by Apache Parquet PMC. It would be better
> for
> > > the
> > > > > > entire
> > > > > > > > > Parquet community if developers with sufficient
> contributions
> > > to
> > > > > open
> > > > > > > > source
> > > > > > > > > Parquet projects (including but not limited to parquet-cpp,
> > > > > arrow-rs,
> > > > > > > > cudf,
> > > > > > > > > etc.)
> > > > > > > > > can be considered as Parquet committer and PMC.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Gang
> > > > > > > > >
> > > > > > > > > On Wed, Apr 24, 2024 at 7:04 PM Uwe L. Korn <
> > uwelk@xhochy.com>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> I would be very supportive of this move. The Parquet C++
> > > > > > development has
> > > > > > > > >> been under the umbrella of the Arrow repository for more
> > than
> > > > > > five(six?)
> > > > > > > > >> years now. Thus, the issues should also be aligned with
> the
> > > > Arrow
> > > > > > > > project.
> > > > > > > > >>
> > > > > > > > >> Uwe
> > > > > > > > >>
> > > > > > > > >> On Tue, Apr 23, 2024, at 8:27 PM, Rok Mihevc wrote:
> > > > > > > > >> > Bumping this thread again to see if there is will to
> call
> > > for
> > > > a
> > > > > > vote
> > > > > > > > and
> > > > > > > > >> > move parquet-cpp issues from Apache Jira to Arrow's
> GitHub
> > > > issue
> > > > > > as
> > > > > > > > was
> > > > > > > > >> > done for Arrow.
> > > > > > > > >> > I'm willing to do the move as I already did it for
> Arrow.
> > > > > > > > >> >
> > > > > > > > >> > Rok
> > > > > > > > >> >
> > > > > > > > >> > On Sat, Apr 15, 2023 at 4:53 AM Micah Kornfield <
> > > > > > > > emkornfield@apache.org>
> > > > > > > > >> > wrote:
> > > > > > > > >> >
> > > > > > > > >> >> Bumping this thread again to see in any Parquet PMC
> > members
> > > > can
> > > > > > chime
> > > > > > > > >> >> in/maybe start a formal vote to move governance of
> > > > Parquet-CPP
> > > > > > under
> > > > > > > > the
> > > > > > > > >> >> umbrella.
> > > > > > > > >> >>
> > > > > > > > >> >> -Micah
> > > > > > > > >> >>
> > > > > > > > >> >> On 2023/02/02 10:34:25 Antoine Pitrou wrote:
> > > > > > > > >> >> >
> > > > > > > > >> >> >
> > > > > > > > >> >> > Hi Will,
> > > > > > > > >> >> >
> > > > > > > > >> >> > Le 01/02/2023 à 20:27, Will Jones a écrit :
> > > > > > > > >> >> > >
> > > > > > > > >> >> > > First, it's not obvious where issues are supposed
> to
> > be
> > > > > > open: In
> > > > > > > > >> >> Parquet
> > > > > > > > >> >> > > Jira or Arrow GitHub issues. Looking back at some
> of
> > > the
> > > > > > original
> > > > > > > > >> >> > > discussion, it looks like the intention was
> > > > > > > > >> >> > >
> > > > > > > > >> >> > > * use PARQUET-XXX for issues relating to Parquet
> core
> > > > > > > > >> >> > >> * use ARROW-XXX for issues relation to Arrow's
> > > > consumption
> > > > > > of
> > > > > > > > >> Parquet
> > > > > > > > >> >> > >> core (e.g. changes that are in parquet/arrow right
> > > now)
> > > > > > > > >> >> > >>
> > > > > > > > >> >> > > The README for the old parquet-cpp repo [3] states
> > > > instead
> > > > > in
> > > > > > > > it's
> > > > > > > > >> >> > > migration note:
> > > > > > > > >> >> > >
> > > > > > > > >> >> > > JIRA issues should continue to be opened in the
> > > PARQUET
> > > > > > JIRA
> > > > > > > > >> project.
> > > > > > > > >> >> > >
> > > > > > > > >> >> > > Either way, it doesn't seem like this process is
> > > obvious
> > > > to
> > > > > > > > people.
> > > > > > > > >> >> Perhaps
> > > > > > > > >> >> > > we could clarify this and add notices to Arrow's
> > GitHub
> > > > > > issues
> > > > > > > > >> >> template?
> > > > > > > > >> >> >
> > > > > > > > >> >> > I agree we should clarify this. I have no personal
> > > > > preference,
> > > > > > but
> > > > > > > > I
> > > > > > > > >> >> will note
> > > > > > > > >> >> > that Github issues decrease friction as having a GH
> > > account
> > > > > is
> > > > > > > > already
> > > > > > > > >> >> necessary
> > > > > > > > >> >> > for submitting PRs.
> > > > > > > > >> >> >
> > > > > > > > >> >> > > Second, committer status is a little unclear. I am
> a
> > > > > > committer on
> > > > > > > > >> >> Arrow,
> > > > > > > > >> >> > > but not on Parquet right now. Does that mean I
> should
> > > > only
> > > > > > merge
> > > > > > > > >> >> Parquet
> > > > > > > > >> >> > > C++ PRs for code changes in parquet/arrow? Or that
> I
> > > > > > shouldn't
> > > > > > > > merge
> > > > > > > > >> >> > > Parquet changes at all?
> > > > > > > > >> >> >
> > > > > > > > >> >> > Since Parquet C++ is part of Arrow C++, you are
> allowed
> > > to
> > > > > > merge
> > > > > > > > >> Parquet
> > > > > > > > >> >> C++
> > > > > > > > >> >> > changes. As always you should ensure you have
> > sufficient
> > > > > > > > understanding
> > > > > > > > >> >> of the
> > > > > > > > >> >> > contribution, and that it follows established
> > practices:
> > > > > > > > >> >> >
> > > > https://arrow.apache.org/docs/dev/developers/reviewing.html
> > > > > > > > >> >> >
> > > > > > > > >> >> > > Also, are the contributions to Arrow C++ Parquet
> > being
> > > > > > actively
> > > > > > > > >> >> reviewed
> > > > > > > > >> >> > > for potential new committers?
> > > > > > > > >> >> >
> > > > > > > > >> >> > I would certainly do.
> > > > > > > > >> >> >
> > > > > > > > >> >> > Regards
> > > > > > > > >> >> >
> > > > > > > > >> >> > Antoine.
> > > > > > > > >> >> >
> > > > > > > > >> >> >
> > > > > > > > >> >>
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Fwd: [C++] Parquet and Arrow overlap
Posted by Rok Mihevc <ro...@gmail.com>.
> BTW, do we really need to make a full copy of them to have a mirror in
the Arrow GitHub issues?
I'm assuming we want to move all parquet-cpp issues (open and closed) to
Arrow's GitHub issue tracker. The amount of effort required is about the
same if we move all or just open tickets. When migrating Arrow's issues we
decided to move all so as to have them searchable in one place [1].
We could also use the opportunity to move issues related to other
implementations to relevant GitHub issue tracker if there is interest. I
think Arrow's issue tracker has much less friction than the ASF Jira.
> As activity picks up, if the desire is to build more community around
> Parquet, perhaps the Parquet PMC wants to encourage moving code back to
> repositories managed by parquet (and out of arrow, for example). I realize
> this would be a technical burden, but it might clarify communities and
> committers.
Another option would be a new parquet monorepo.
[1] https://github.com/apache/arrow/issues/14546
Rok
On Sat, May 11, 2024 at 11:51 AM Andrew Lamb <an...@gmail.com> wrote:
> It is great to see some additional enthusiasm and momentum around the
> Apache Parquet implementation (congratulations on the release of parquet-mr
> 1.14[1]!).
>
> As activity picks up, if the desire is to build more community around
> Parquet, perhaps the Parquet PMC wants to encourage moving code back to
> repositories managed by parquet (and out of arrow, for example). I realize
> this would be a technical burden, but it might clarify communities and
> committers.
>
> Andrew
>
> [1]: https://lists.apache.org/thread/2gggm938z0x9fx3wtwctfm5htsxlf3z4
>
>
>
> On Fri, May 10, 2024 at 11:45 PM Matt Topol <zo...@gmail.com>
> wrote:
>
> > I just wanted to also poke the question of non-Java developers who have
> > worked on the other parquet implementations potentially being recognized
> as
> > committers or otherwise on the Parquet project (speaking as the primary
> > developer of the Go parquet implementation which also lives in the Arrow
> > repository). It would be great to see some active contributors to
> > parquet-cpp, parquet-go, or otherwise not just being considered but
> > actively becoming committers.
> >
> > That's just my two cents from a community perspective.
> >
> > --Matt
> >
> > On Fri, May 10, 2024, 10:35 PM Jacob Wujciak <as...@apache.org>
> > wrote:
> >
> > > Thank you, that sounds great! On first glance some seem to be rather
> old
> > > and probably don't apply anymore.
> > >
> > > > BTW, do we really need to make a full copy of them to have a mirror
> in
> > > the Arrow GitHub issues?
> > >
> > > I am not sure I understand what you mean? A full copy of the
> > > open/closed/all issues? I'd say only the (remaining) open issues would
> be
> > > fine.
> > > For reference this is what an imported issue looks like:
> > > https://github.com/apache/arrow/issues/30543
> > >
> > > Am Sa., 11. Mai 2024 um 04:09 Uhr schrieb Gang Wu <us...@gmail.com>:
> > >
> > > > I can initiate the vote. But before the vote, I think we need to
> > revisit
> > > > the states of all unresolved tickets and close some as needed.
> > > >
> > > > BTW, do we really need to make a full copy of them to have a mirror
> > > > in the Arrow GitHub issues?
> > > >
> > > > I'd like to seek a consensus here before sending the vote.
> > > >
> > > > Best,
> > > > Gang
> > > >
> > > > On Sat, May 11, 2024 at 8:46 AM Jacob Wujciak <assignuser@apache.org
> >
> > > > wrote:
> > > >
> > > > > Hello Everyone!
> > > > >
> > > > > It seems there is general agreement on this topic, it would be
> great
> > > if a
> > > > > committer/PMC could start a (lazy consensus) procedural vote.
> > > > >
> > > > > I will inquire how to handle the parquet-cpp component in jira
> > (ideally
> > > > > disabling it, not removing).
> > > > > There are currently only ~70 open tickets for parquet-cpp, with the
> > > > change
> > > > > in repo it is probably easier to just move open tickets but I'll
> > leave
> > > > that
> > > > > to Rok who managed the transition of Arrows 20k+ tickets too :D
> > > > >
> > > > > Thanks,
> > > > > Jacob
> > > > >
> > > > > Arrow committer
> > > > >
> > > > > On 2024/04/25 05:31:18 Gang Wu wrote:
> > > > > > I know we have some non-Java committers and PMCs. But after the
> > > > > parquet-cpp
> > > > > > donation, it seems that no one worked on Parquet from arrow (cpp,
> > > rust,
> > > > > go,
> > > > > > etc.)
> > > > > > and other projects are promoted as a Parquet committer. It would
> be
> > > > > > inconvenient
> > > > > > for non-Java Parquet developers to work with
> apache/parquet-format
> > > and
> > > > > > apache/parquet-testing repositories. Furthermore, votes from
> these
> > > > > > developers
> > > > > > are not binding for a format change in the ML.
> > > > > >
> > > > > > Best,
> > > > > > Gang
> > > > > >
> > > > > > On Wed, Apr 24, 2024 at 8:42 PM Uwe L. Korn <uw...@xhochy.com>
> > > wrote:
> > > > > >
> > > > > > > > Should we consider
> > > > > > > > Parquet developers from other projects than parquet-mr as
> > Parquet
> > > > > > > commuters?
> > > > > > >
> > > > > > > We are doing this (speaking as a Parquet PMC who didn't work on
> > > > > > > parquet-mr, but parquet-cpp).
> > > > > > >
> > > > > > > Best
> > > > > > > Uwe
> > > > > > >
> > > > > > > On Wed, Apr 24, 2024, at 2:38 PM, Gang Wu wrote:
> > > > > > > > +1 for moving parquet-cpp issues from Apache Jira to Arrow's
> > > GitHub
> > > > > > > issue.
> > > > > > > >
> > > > > > > > Besides, I want to echo Will's question in the thread. Should
> > we
> > > > > consider
> > > > > > > > Parquet developers from other projects than parquet-mr as
> > Parquet
> > > > > > > commiters?
> > > > > > > > Currently apache/parquet-format and apache/parquet-testing
> > > > > repositories
> > > > > > > are
> > > > > > > > solely governed by Apache Parquet PMC. It would be better for
> > the
> > > > > entire
> > > > > > > > Parquet community if developers with sufficient contributions
> > to
> > > > open
> > > > > > > source
> > > > > > > > Parquet projects (including but not limited to parquet-cpp,
> > > > arrow-rs,
> > > > > > > cudf,
> > > > > > > > etc.)
> > > > > > > > can be considered as Parquet committer and PMC.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Gang
> > > > > > > >
> > > > > > > > On Wed, Apr 24, 2024 at 7:04 PM Uwe L. Korn <
> uwelk@xhochy.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > >> I would be very supportive of this move. The Parquet C++
> > > > > development has
> > > > > > > >> been under the umbrella of the Arrow repository for more
> than
> > > > > five(six?)
> > > > > > > >> years now. Thus, the issues should also be aligned with the
> > > Arrow
> > > > > > > project.
> > > > > > > >>
> > > > > > > >> Uwe
> > > > > > > >>
> > > > > > > >> On Tue, Apr 23, 2024, at 8:27 PM, Rok Mihevc wrote:
> > > > > > > >> > Bumping this thread again to see if there is will to call
> > for
> > > a
> > > > > vote
> > > > > > > and
> > > > > > > >> > move parquet-cpp issues from Apache Jira to Arrow's GitHub
> > > issue
> > > > > as
> > > > > > > was
> > > > > > > >> > done for Arrow.
> > > > > > > >> > I'm willing to do the move as I already did it for Arrow.
> > > > > > > >> >
> > > > > > > >> > Rok
> > > > > > > >> >
> > > > > > > >> > On Sat, Apr 15, 2023 at 4:53 AM Micah Kornfield <
> > > > > > > emkornfield@apache.org>
> > > > > > > >> > wrote:
> > > > > > > >> >
> > > > > > > >> >> Bumping this thread again to see in any Parquet PMC
> members
> > > can
> > > > > chime
> > > > > > > >> >> in/maybe start a formal vote to move governance of
> > > Parquet-CPP
> > > > > under
> > > > > > > the
> > > > > > > >> >> umbrella.
> > > > > > > >> >>
> > > > > > > >> >> -Micah
> > > > > > > >> >>
> > > > > > > >> >> On 2023/02/02 10:34:25 Antoine Pitrou wrote:
> > > > > > > >> >> >
> > > > > > > >> >> >
> > > > > > > >> >> > Hi Will,
> > > > > > > >> >> >
> > > > > > > >> >> > Le 01/02/2023 à 20:27, Will Jones a écrit :
> > > > > > > >> >> > >
> > > > > > > >> >> > > First, it's not obvious where issues are supposed to
> be
> > > > > open: In
> > > > > > > >> >> Parquet
> > > > > > > >> >> > > Jira or Arrow GitHub issues. Looking back at some of
> > the
> > > > > original
> > > > > > > >> >> > > discussion, it looks like the intention was
> > > > > > > >> >> > >
> > > > > > > >> >> > > * use PARQUET-XXX for issues relating to Parquet core
> > > > > > > >> >> > >> * use ARROW-XXX for issues relation to Arrow's
> > > consumption
> > > > > of
> > > > > > > >> Parquet
> > > > > > > >> >> > >> core (e.g. changes that are in parquet/arrow right
> > now)
> > > > > > > >> >> > >>
> > > > > > > >> >> > > The README for the old parquet-cpp repo [3] states
> > > instead
> > > > in
> > > > > > > it's
> > > > > > > >> >> > > migration note:
> > > > > > > >> >> > >
> > > > > > > >> >> > > JIRA issues should continue to be opened in the
> > PARQUET
> > > > > JIRA
> > > > > > > >> project.
> > > > > > > >> >> > >
> > > > > > > >> >> > > Either way, it doesn't seem like this process is
> > obvious
> > > to
> > > > > > > people.
> > > > > > > >> >> Perhaps
> > > > > > > >> >> > > we could clarify this and add notices to Arrow's
> GitHub
> > > > > issues
> > > > > > > >> >> template?
> > > > > > > >> >> >
> > > > > > > >> >> > I agree we should clarify this. I have no personal
> > > > preference,
> > > > > but
> > > > > > > I
> > > > > > > >> >> will note
> > > > > > > >> >> > that Github issues decrease friction as having a GH
> > account
> > > > is
> > > > > > > already
> > > > > > > >> >> necessary
> > > > > > > >> >> > for submitting PRs.
> > > > > > > >> >> >
> > > > > > > >> >> > > Second, committer status is a little unclear. I am a
> > > > > committer on
> > > > > > > >> >> Arrow,
> > > > > > > >> >> > > but not on Parquet right now. Does that mean I should
> > > only
> > > > > merge
> > > > > > > >> >> Parquet
> > > > > > > >> >> > > C++ PRs for code changes in parquet/arrow? Or that I
> > > > > shouldn't
> > > > > > > merge
> > > > > > > >> >> > > Parquet changes at all?
> > > > > > > >> >> >
> > > > > > > >> >> > Since Parquet C++ is part of Arrow C++, you are allowed
> > to
> > > > > merge
> > > > > > > >> Parquet
> > > > > > > >> >> C++
> > > > > > > >> >> > changes. As always you should ensure you have
> sufficient
> > > > > > > understanding
> > > > > > > >> >> of the
> > > > > > > >> >> > contribution, and that it follows established
> practices:
> > > > > > > >> >> >
> > > https://arrow.apache.org/docs/dev/developers/reviewing.html
> > > > > > > >> >> >
> > > > > > > >> >> > > Also, are the contributions to Arrow C++ Parquet
> being
> > > > > actively
> > > > > > > >> >> reviewed
> > > > > > > >> >> > > for potential new committers?
> > > > > > > >> >> >
> > > > > > > >> >> > I would certainly do.
> > > > > > > >> >> >
> > > > > > > >> >> > Regards
> > > > > > > >> >> >
> > > > > > > >> >> > Antoine.
> > > > > > > >> >> >
> > > > > > > >> >> >
> > > > > > > >> >>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Fwd: [C++] Parquet and Arrow overlap
Posted by Rok Mihevc <ro...@gmail.com>.
> BTW, do we really need to make a full copy of them to have a mirror in
the Arrow GitHub issues?
I'm assuming we want to move all parquet-cpp issues (open and closed) to
Arrow's GitHub issue tracker. The amount of effort required is about the
same if we move all or just open tickets. When migrating Arrow's issues we
decided to move all so as to have them searchable in one place [1].
We could also use the opportunity to move issues related to other
implementations to relevant GitHub issue tracker if there is interest. I
think Arrow's issue tracker has much less friction than the ASF Jira.
> As activity picks up, if the desire is to build more community around
> Parquet, perhaps the Parquet PMC wants to encourage moving code back to
> repositories managed by parquet (and out of arrow, for example). I realize
> this would be a technical burden, but it might clarify communities and
> committers.
Another option would be a new parquet monorepo.
[1] https://github.com/apache/arrow/issues/14546
Rok
On Sat, May 11, 2024 at 11:51 AM Andrew Lamb <an...@gmail.com> wrote:
> It is great to see some additional enthusiasm and momentum around the
> Apache Parquet implementation (congratulations on the release of parquet-mr
> 1.14[1]!).
>
> As activity picks up, if the desire is to build more community around
> Parquet, perhaps the Parquet PMC wants to encourage moving code back to
> repositories managed by parquet (and out of arrow, for example). I realize
> this would be a technical burden, but it might clarify communities and
> committers.
>
> Andrew
>
> [1]: https://lists.apache.org/thread/2gggm938z0x9fx3wtwctfm5htsxlf3z4
>
>
>
> On Fri, May 10, 2024 at 11:45 PM Matt Topol <zo...@gmail.com>
> wrote:
>
> > I just wanted to also poke the question of non-Java developers who have
> > worked on the other parquet implementations potentially being recognized
> as
> > committers or otherwise on the Parquet project (speaking as the primary
> > developer of the Go parquet implementation which also lives in the Arrow
> > repository). It would be great to see some active contributors to
> > parquet-cpp, parquet-go, or otherwise not just being considered but
> > actively becoming committers.
> >
> > That's just my two cents from a community perspective.
> >
> > --Matt
> >
> > On Fri, May 10, 2024, 10:35 PM Jacob Wujciak <as...@apache.org>
> > wrote:
> >
> > > Thank you, that sounds great! On first glance some seem to be rather
> old
> > > and probably don't apply anymore.
> > >
> > > > BTW, do we really need to make a full copy of them to have a mirror
> in
> > > the Arrow GitHub issues?
> > >
> > > I am not sure I understand what you mean? A full copy of the
> > > open/closed/all issues? I'd say only the (remaining) open issues would
> be
> > > fine.
> > > For reference this is what an imported issue looks like:
> > > https://github.com/apache/arrow/issues/30543
> > >
> > > Am Sa., 11. Mai 2024 um 04:09 Uhr schrieb Gang Wu <us...@gmail.com>:
> > >
> > > > I can initiate the vote. But before the vote, I think we need to
> > revisit
> > > > the states of all unresolved tickets and close some as needed.
> > > >
> > > > BTW, do we really need to make a full copy of them to have a mirror
> > > > in the Arrow GitHub issues?
> > > >
> > > > I'd like to seek a consensus here before sending the vote.
> > > >
> > > > Best,
> > > > Gang
> > > >
> > > > On Sat, May 11, 2024 at 8:46 AM Jacob Wujciak <assignuser@apache.org
> >
> > > > wrote:
> > > >
> > > > > Hello Everyone!
> > > > >
> > > > > It seems there is general agreement on this topic, it would be
> great
> > > if a
> > > > > committer/PMC could start a (lazy consensus) procedural vote.
> > > > >
> > > > > I will inquire how to handle the parquet-cpp component in jira
> > (ideally
> > > > > disabling it, not removing).
> > > > > There are currently only ~70 open tickets for parquet-cpp, with the
> > > > change
> > > > > in repo it is probably easier to just move open tickets but I'll
> > leave
> > > > that
> > > > > to Rok who managed the transition of Arrows 20k+ tickets too :D
> > > > >
> > > > > Thanks,
> > > > > Jacob
> > > > >
> > > > > Arrow committer
> > > > >
> > > > > On 2024/04/25 05:31:18 Gang Wu wrote:
> > > > > > I know we have some non-Java committers and PMCs. But after the
> > > > > parquet-cpp
> > > > > > donation, it seems that no one worked on Parquet from arrow (cpp,
> > > rust,
> > > > > go,
> > > > > > etc.)
> > > > > > and other projects are promoted as a Parquet committer. It would
> be
> > > > > > inconvenient
> > > > > > for non-Java Parquet developers to work with
> apache/parquet-format
> > > and
> > > > > > apache/parquet-testing repositories. Furthermore, votes from
> these
> > > > > > developers
> > > > > > are not binding for a format change in the ML.
> > > > > >
> > > > > > Best,
> > > > > > Gang
> > > > > >
> > > > > > On Wed, Apr 24, 2024 at 8:42 PM Uwe L. Korn <uw...@xhochy.com>
> > > wrote:
> > > > > >
> > > > > > > > Should we consider
> > > > > > > > Parquet developers from other projects than parquet-mr as
> > Parquet
> > > > > > > commuters?
> > > > > > >
> > > > > > > We are doing this (speaking as a Parquet PMC who didn't work on
> > > > > > > parquet-mr, but parquet-cpp).
> > > > > > >
> > > > > > > Best
> > > > > > > Uwe
> > > > > > >
> > > > > > > On Wed, Apr 24, 2024, at 2:38 PM, Gang Wu wrote:
> > > > > > > > +1 for moving parquet-cpp issues from Apache Jira to Arrow's
> > > GitHub
> > > > > > > issue.
> > > > > > > >
> > > > > > > > Besides, I want to echo Will's question in the thread. Should
> > we
> > > > > consider
> > > > > > > > Parquet developers from other projects than parquet-mr as
> > Parquet
> > > > > > > commiters?
> > > > > > > > Currently apache/parquet-format and apache/parquet-testing
> > > > > repositories
> > > > > > > are
> > > > > > > > solely governed by Apache Parquet PMC. It would be better for
> > the
> > > > > entire
> > > > > > > > Parquet community if developers with sufficient contributions
> > to
> > > > open
> > > > > > > source
> > > > > > > > Parquet projects (including but not limited to parquet-cpp,
> > > > arrow-rs,
> > > > > > > cudf,
> > > > > > > > etc.)
> > > > > > > > can be considered as Parquet committer and PMC.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Gang
> > > > > > > >
> > > > > > > > On Wed, Apr 24, 2024 at 7:04 PM Uwe L. Korn <
> uwelk@xhochy.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > >> I would be very supportive of this move. The Parquet C++
> > > > > development has
> > > > > > > >> been under the umbrella of the Arrow repository for more
> than
> > > > > five(six?)
> > > > > > > >> years now. Thus, the issues should also be aligned with the
> > > Arrow
> > > > > > > project.
> > > > > > > >>
> > > > > > > >> Uwe
> > > > > > > >>
> > > > > > > >> On Tue, Apr 23, 2024, at 8:27 PM, Rok Mihevc wrote:
> > > > > > > >> > Bumping this thread again to see if there is will to call
> > for
> > > a
> > > > > vote
> > > > > > > and
> > > > > > > >> > move parquet-cpp issues from Apache Jira to Arrow's GitHub
> > > issue
> > > > > as
> > > > > > > was
> > > > > > > >> > done for Arrow.
> > > > > > > >> > I'm willing to do the move as I already did it for Arrow.
> > > > > > > >> >
> > > > > > > >> > Rok
> > > > > > > >> >
> > > > > > > >> > On Sat, Apr 15, 2023 at 4:53 AM Micah Kornfield <
> > > > > > > emkornfield@apache.org>
> > > > > > > >> > wrote:
> > > > > > > >> >
> > > > > > > >> >> Bumping this thread again to see in any Parquet PMC
> members
> > > can
> > > > > chime
> > > > > > > >> >> in/maybe start a formal vote to move governance of
> > > Parquet-CPP
> > > > > under
> > > > > > > the
> > > > > > > >> >> umbrella.
> > > > > > > >> >>
> > > > > > > >> >> -Micah
> > > > > > > >> >>
> > > > > > > >> >> On 2023/02/02 10:34:25 Antoine Pitrou wrote:
> > > > > > > >> >> >
> > > > > > > >> >> >
> > > > > > > >> >> > Hi Will,
> > > > > > > >> >> >
> > > > > > > >> >> > Le 01/02/2023 à 20:27, Will Jones a écrit :
> > > > > > > >> >> > >
> > > > > > > >> >> > > First, it's not obvious where issues are supposed to
> be
> > > > > open: In
> > > > > > > >> >> Parquet
> > > > > > > >> >> > > Jira or Arrow GitHub issues. Looking back at some of
> > the
> > > > > original
> > > > > > > >> >> > > discussion, it looks like the intention was
> > > > > > > >> >> > >
> > > > > > > >> >> > > * use PARQUET-XXX for issues relating to Parquet core
> > > > > > > >> >> > >> * use ARROW-XXX for issues relation to Arrow's
> > > consumption
> > > > > of
> > > > > > > >> Parquet
> > > > > > > >> >> > >> core (e.g. changes that are in parquet/arrow right
> > now)
> > > > > > > >> >> > >>
> > > > > > > >> >> > > The README for the old parquet-cpp repo [3] states
> > > instead
> > > > in
> > > > > > > it's
> > > > > > > >> >> > > migration note:
> > > > > > > >> >> > >
> > > > > > > >> >> > > JIRA issues should continue to be opened in the
> > PARQUET
> > > > > JIRA
> > > > > > > >> project.
> > > > > > > >> >> > >
> > > > > > > >> >> > > Either way, it doesn't seem like this process is
> > obvious
> > > to
> > > > > > > people.
> > > > > > > >> >> Perhaps
> > > > > > > >> >> > > we could clarify this and add notices to Arrow's
> GitHub
> > > > > issues
> > > > > > > >> >> template?
> > > > > > > >> >> >
> > > > > > > >> >> > I agree we should clarify this. I have no personal
> > > > preference,
> > > > > but
> > > > > > > I
> > > > > > > >> >> will note
> > > > > > > >> >> > that Github issues decrease friction as having a GH
> > account
> > > > is
> > > > > > > already
> > > > > > > >> >> necessary
> > > > > > > >> >> > for submitting PRs.
> > > > > > > >> >> >
> > > > > > > >> >> > > Second, committer status is a little unclear. I am a
> > > > > committer on
> > > > > > > >> >> Arrow,
> > > > > > > >> >> > > but not on Parquet right now. Does that mean I should
> > > only
> > > > > merge
> > > > > > > >> >> Parquet
> > > > > > > >> >> > > C++ PRs for code changes in parquet/arrow? Or that I
> > > > > shouldn't
> > > > > > > merge
> > > > > > > >> >> > > Parquet changes at all?
> > > > > > > >> >> >
> > > > > > > >> >> > Since Parquet C++ is part of Arrow C++, you are allowed
> > to
> > > > > merge
> > > > > > > >> Parquet
> > > > > > > >> >> C++
> > > > > > > >> >> > changes. As always you should ensure you have
> sufficient
> > > > > > > understanding
> > > > > > > >> >> of the
> > > > > > > >> >> > contribution, and that it follows established
> practices:
> > > > > > > >> >> >
> > > https://arrow.apache.org/docs/dev/developers/reviewing.html
> > > > > > > >> >> >
> > > > > > > >> >> > > Also, are the contributions to Arrow C++ Parquet
> being
> > > > > actively
> > > > > > > >> >> reviewed
> > > > > > > >> >> > > for potential new committers?
> > > > > > > >> >> >
> > > > > > > >> >> > I would certainly do.
> > > > > > > >> >> >
> > > > > > > >> >> > Regards
> > > > > > > >> >> >
> > > > > > > >> >> > Antoine.
> > > > > > > >> >> >
> > > > > > > >> >> >
> > > > > > > >> >>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [C++] Parquet and Arrow overlap
Posted by Julien Le Dem <ju...@apache.org>.
If we deem that it would be too hard to move it back for the moment, we
need at a minimum to clarify and reduce the confusion.
If practice doesn't match what the PMC voted on, we need to improve the
practice.
Do we have suggestions on improving that?
perhaps OWNERSFILE in the parquet folder in the arrow repo? (just an idea)
On Fri, May 17, 2024 at 2:49 AM Uwe L. Korn <uw...@xhochy.com> wrote:
>
>
> On Fri, May 17, 2024, at 10:36 AM, Antoine Pitrou wrote:
> > Hi Julien,
> >
> > On Thu, 16 May 2024 18:23:33 -0700
> > Julien Le Dem <ju...@apache.org> wrote:
> >>
> >> As discussed, that code was moved in the arrow repo for convenience:
> >> https://lists.apache.org/thread/gkvbm6yyly1r4cg3f6xtnqkjz6ogn6o2
> >>
> >> To take an excerpt of that original decision:
> >>
> >> 4) The Parquet and Arrow C++ communities will collaborate to provide
> >> development workflows to enable contributors working exclusively on the
> >> Parquet core functionality to be able to work unencumbered with
> unnecessary
> >> build or test dependencies from the rest of the Arrow codebase. Note
> that
> >> parquet-cpp already builds a significant portion of Apache Arrow en
> route
> >> to creating its libraries 5) The Parquet community can create scripts to
> >> "cut" Parquet C++ releases by packaging up the appropriate components
> and
> >> ensuring that they can be built and installed independently as now
> >
> > Unfortunately, these two points haven't happened at all. On the
> > contrary, the Arrow C++ dependency has infused much deeper in Parquet
> > C++ (I was not there at the beginning of Parquet C++, but I get the
> > impression there was originally an effort to have a Arrow-independent
> > Parquet C++ core; that "core" doesn't exist anymore).
>
> As an example, we had in the beginning separate I/O primitives in Arrow
> and Parquet. But during the further development, we realised that we were
> implementing exactly the same code paths only in different namespaces.
>
> There are some core "utilities" hidden in Arrow that are required to build
> any modern C++based data processing library. If you would separate that
> into its own repository would enable parquet-cpp to be separated more
> easily. But given that the development around this is still very active in
> Arrow, it would bring a massive slowdown to the overall project.
>
> Best
> Uwe
>
Re: [C++] Parquet and Arrow overlap
Posted by "Uwe L. Korn" <uw...@xhochy.com>.
On Fri, May 17, 2024, at 10:36 AM, Antoine Pitrou wrote:
> Hi Julien,
>
> On Thu, 16 May 2024 18:23:33 -0700
> Julien Le Dem <ju...@apache.org> wrote:
>>
>> As discussed, that code was moved in the arrow repo for convenience:
>> https://lists.apache.org/thread/gkvbm6yyly1r4cg3f6xtnqkjz6ogn6o2
>>
>> To take an excerpt of that original decision:
>>
>> 4) The Parquet and Arrow C++ communities will collaborate to provide
>> development workflows to enable contributors working exclusively on the
>> Parquet core functionality to be able to work unencumbered with unnecessary
>> build or test dependencies from the rest of the Arrow codebase. Note that
>> parquet-cpp already builds a significant portion of Apache Arrow en route
>> to creating its libraries 5) The Parquet community can create scripts to
>> "cut" Parquet C++ releases by packaging up the appropriate components and
>> ensuring that they can be built and installed independently as now
>
> Unfortunately, these two points haven't happened at all. On the
> contrary, the Arrow C++ dependency has infused much deeper in Parquet
> C++ (I was not there at the beginning of Parquet C++, but I get the
> impression there was originally an effort to have a Arrow-independent
> Parquet C++ core; that "core" doesn't exist anymore).
As an example, we had in the beginning separate I/O primitives in Arrow and Parquet. But during the further development, we realised that we were implementing exactly the same code paths only in different namespaces.
There are some core "utilities" hidden in Arrow that are required to build any modern C++based data processing library. If you would separate that into its own repository would enable parquet-cpp to be separated more easily. But given that the development around this is still very active in Arrow, it would bring a massive slowdown to the overall project.
Best
Uwe
Re: [C++] Parquet and Arrow overlap
Posted by Antoine Pitrou <an...@python.org>.
Hi Julien,
On Thu, 16 May 2024 18:23:33 -0700
Julien Le Dem <ju...@apache.org> wrote:
>
> As discussed, that code was moved in the arrow repo for convenience:
> https://lists.apache.org/thread/gkvbm6yyly1r4cg3f6xtnqkjz6ogn6o2
>
> To take an excerpt of that original decision:
>
> 4) The Parquet and Arrow C++ communities will collaborate to provide
> development workflows to enable contributors working exclusively on the
> Parquet core functionality to be able to work unencumbered with unnecessary
> build or test dependencies from the rest of the Arrow codebase. Note that
> parquet-cpp already builds a significant portion of Apache Arrow en route
> to creating its libraries 5) The Parquet community can create scripts to
> "cut" Parquet C++ releases by packaging up the appropriate components and
> ensuring that they can be built and installed independently as now
Unfortunately, these two points haven't happened at all. On the
contrary, the Arrow C++ dependency has infused much deeper in Parquet
C++ (I was not there at the beginning of Parquet C++, but I get the
impression there was originally an effort to have a Arrow-independent
Parquet C++ core; that "core" doesn't exist anymore).
Note that this doesn't mean that Parquet C++ forces you to read Parquet
files as Arrow-formatted data (*). It's just that Parquet C++ uses a
large number of assorted utilities that live in the Arrow C++ codebase.
(*) though I would argue that it's better to do so, as it's probably
more efficient, especially for BYTE_ARRAY data
> The alternative is to live up to the part where we agreed that the two
> communities collaborate on making it easy for the Parquet community to
> govern its code base in the arrow repo.
> Would you agree?
Yep. I don't think there has been any problem in that regard, TBH. It's
just that the situation is difficult to understand for people.
Regards
Antoine.
Re: [C++] Parquet and Arrow overlap
Posted by Julien Le Dem <ju...@apache.org>.
> Hmm... I'm not sure I understand your point here. The Parquet spec and
> the Java implementation are already living in distinct repos and have
> distinct versioning schemes. The main thing that they share in common is
> the JIRA instance (while the C++ Parquet implementation mostly relies on
> Arrow's GH issue tracker), but is that really important?
It is not a problem that they are in separate repos. The problem is the
friction created because it makes access control difficult and creates
confusion on governance.
This thread "[DISCUSS] Parquet C++ under which PMC?" is a clear example of
it: https://lists.apache.org/thread/128wv5cwv51scm8vdfn1g9gskw717qyt
All I'm suggesting is that if the inconvenience this creates around unclear
governance discussions is greater than the convenience of being in the same
repo, we should revisit that decision.
It would be less inconvenient today to have parquet-cpp in its own repo
than it was at the time.
As discussed, that code was moved in the arrow repo for convenience:
https://lists.apache.org/thread/gkvbm6yyly1r4cg3f6xtnqkjz6ogn6o2
To take an excerpt of that original decision:
4) The Parquet and Arrow C++ communities will collaborate to provide
development workflows to enable contributors working exclusively on the
Parquet core functionality to be able to work unencumbered with unnecessary
build or test dependencies from the rest of the Arrow codebase. Note that
parquet-cpp already builds a significant portion of Apache Arrow en route
to creating its libraries 5) The Parquet community can create scripts to
"cut" Parquet C++ releases by packaging up the appropriate components and
ensuring that they can be built and installed independently as now
development workflows to enable contributors working exclusively on the
Parquet core functionality to be able to work unencumbered with unnecessary
build or test dependencies from the rest of the Arrow codebase. Note that
parquet-cpp already builds a significant portion of Apache Arrow en route
to creating its libraries
The alternative is to live up to the part where we agreed that the two
communities collaborate on making it easy for the Parquet community to
govern its code base in the arrow repo.
Would you agree?
On Thu, May 16, 2024 at 1:00 AM Micah Kornfield <em...@gmail.com>
wrote:
> From my perspective I agree, that I don't think there is benefit of moving
> parquet C++ out of arrow given what it would actually cost to make clean
> boundaries. I also don't think it will hurt iteration speed.
>
> I think the main challenge could be in compatibility testing, but Arrow has
> solved this between implementations that live in different repositories so
> I think the same solutions could apply for Parquet.
>
> On Thu, May 16, 2024 at 12:57 AM Antoine Pitrou <an...@python.org>
> wrote:
>
> > On Tue, 14 May 2024 10:22:37 -0700
> > Julien Le Dem <ju...@apache.org> wrote:
> > > 1. I think we should make it easy for people contributing to the C++
> > > codebase. (which is why I voted for the move at the time)
> > > 2. If merging repos removes the need to deal with the circular
> dependency
> > > between repos issue for the C++ code bases, it does it at the expense
> of
> > > making it easy to evolve the parquet spec and the java and c++
> > > implementations together.
> >
> > Hmm... I'm not sure I understand your point here. The Parquet spec and
> > the Java implementation are already living in distinct repos and have
> > distinct versioning schemes. The main thing that they share in common is
> > the JIRA instance (while the C++ Parquet implementation mostly relies on
> > Arrow's GH issue tracker), but is that really important?
> >
> > > parquet-cpp depends only on arrow-core that does not have to depend on
> > > parquet-cpp.
> >
> > That is true.
> >
> > > Other components like
> > > arrow-dataset and pyarrow can depend on parquet-cpp just like they
> depend
> > > on orc externally.
> >
> > Ideally yes. In practice there are two problems:
> > 1) it creates a circular dependency between *repositories*.
> > 2) the C++ Arrow Datasets component is not built independently, it is an
> > optional component when building Arrow C++. So we would also have a
> > chicken-and-egg problem when building Arrow C++ and Parquet C++.
> >
> > > I realize that would be work to make it happen, but the current
> location
> > of
> > > the parquet-cpp codebase is a big trade-off of prioritizing quick
> > iteration
> > > on the C++ implementations over iteration on the format.
> >
> > Having recently worked on a format addition and its respective
> > implementations (in Java and C++), I haven't found the current setup
> > more difficult to work with for Parquet C++ than it was for Parquet
> > Java. Admittedly I'm biased, being a heavy contributor to Arrow C++,
> > but I'm curious why the current situation would be detrimental to
> > iteration on the format.
> >
> > Regards
> >
> > Antoine.
> >
> >
> >
>
Re: [C++] Parquet and Arrow overlap
Posted by Micah Kornfield <em...@gmail.com>.
From my perspective I agree, that I don't think there is benefit of moving
parquet C++ out of arrow given what it would actually cost to make clean
boundaries. I also don't think it will hurt iteration speed.
I think the main challenge could be in compatibility testing, but Arrow has
solved this between implementations that live in different repositories so
I think the same solutions could apply for Parquet.
On Thu, May 16, 2024 at 12:57 AM Antoine Pitrou <an...@python.org> wrote:
> On Tue, 14 May 2024 10:22:37 -0700
> Julien Le Dem <ju...@apache.org> wrote:
> > 1. I think we should make it easy for people contributing to the C++
> > codebase. (which is why I voted for the move at the time)
> > 2. If merging repos removes the need to deal with the circular dependency
> > between repos issue for the C++ code bases, it does it at the expense of
> > making it easy to evolve the parquet spec and the java and c++
> > implementations together.
>
> Hmm... I'm not sure I understand your point here. The Parquet spec and
> the Java implementation are already living in distinct repos and have
> distinct versioning schemes. The main thing that they share in common is
> the JIRA instance (while the C++ Parquet implementation mostly relies on
> Arrow's GH issue tracker), but is that really important?
>
> > parquet-cpp depends only on arrow-core that does not have to depend on
> > parquet-cpp.
>
> That is true.
>
> > Other components like
> > arrow-dataset and pyarrow can depend on parquet-cpp just like they depend
> > on orc externally.
>
> Ideally yes. In practice there are two problems:
> 1) it creates a circular dependency between *repositories*.
> 2) the C++ Arrow Datasets component is not built independently, it is an
> optional component when building Arrow C++. So we would also have a
> chicken-and-egg problem when building Arrow C++ and Parquet C++.
>
> > I realize that would be work to make it happen, but the current location
> of
> > the parquet-cpp codebase is a big trade-off of prioritizing quick
> iteration
> > on the C++ implementations over iteration on the format.
>
> Having recently worked on a format addition and its respective
> implementations (in Java and C++), I haven't found the current setup
> more difficult to work with for Parquet C++ than it was for Parquet
> Java. Admittedly I'm biased, being a heavy contributor to Arrow C++,
> but I'm curious why the current situation would be detrimental to
> iteration on the format.
>
> Regards
>
> Antoine.
>
>
>
Re: [C++] Parquet and Arrow overlap
Posted by Antoine Pitrou <an...@python.org>.
On Fri, 17 May 2024 07:48:18 +0200
Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> Hi
>
> Technically speaking moving back to parquet would be challenging short
> term.
>
> In terms of governance, why not having some parquet maintainer/PMC member
> invited to arrow ? It would simplify the review and governance.
The Arrow and Parquet PMCs already have several members in common
(though some of them might be relatively inactive), so that's not a
problem.
Regards
Antoine.
Re: [C++] Parquet and Arrow overlap
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi
Technically speaking moving back to parquet would be challenging short
term.
In terms of governance, why not having some parquet maintainer/PMC member
invited to arrow ? It would simplify the review and governance.
From projects perspective, it’s the life of the projects to reevaluate the
direction. So if we think that it makes more sense for parquet-cpp to be
back in parquet tlp with a dependency to arrow, it makes sense to do it.
That would simplify the governance imho.
Happy to help on that.
Regards
JB
Le mar. 14 mai 2024 à 19:22, Julien Le Dem <ju...@apache.org> a écrit :
> 1. I think we should make it easy for people contributing to the C++
> codebase. (which is why I voted for the move at the time)
> 2. If merging repos removes the need to deal with the circular dependency
> between repos issue for the C++ code bases, it does it at the expense of
> making it easy to evolve the parquet spec and the java and c++
> implementations together.
> This setup was optimized for quick iterations on the APIs on the C++ side.
> Now that those APIs are more stable, it is less needed IMO.
>
> parquet-cpp depends only on arrow-core that does not have to depend on
> parquet-cpp. It really just needs the vectors. Other components like
> arrow-dataset and pyarrow can depend on parquet-cpp just like they depend
> on orc externally.
>
> I realize that would be work to make it happen, but the current location of
> the parquet-cpp codebase is a big trade-off of prioritizing quick iteration
> on the C++ implementations over iteration on the format. As interest grows
> in evolving the format, I think it warrants a re-evaluation.
>
>
>
> On Tue, May 14, 2024 at 9:20 AM Antoine Pitrou <an...@python.org> wrote:
>
> >
> > Moving Parquet C++ out of Arrow C++ would basically recreate the
> > problems that motivated the integration of Parquet C++ into Arrow C++
> > :-)
> >
> > Regards
> >
> > Antoine.
> >
> >
> > On Tue, 14 May 2024 13:52:15 +0800
> > Gang Wu <us...@gmail.com> wrote:
> > > IMO, moving parquet-cpp out of arrow is challenging as the dependency
> > > chain looks like: arrow core <- parquet-cpp <- arrow dataset <- pyarrow
> > >
> > > Best,
> > > Gang
> > >
> > > On Tue, May 14, 2024 at 12:38 PM Julien Le Dem <
> > julien-1oDqGaOF3Lkdnm+yROfE0A@public.gmane.org> wrote:
> > >
> > > > It is great to see more momentum building.
> > > > I have myself a little bit more time to contribute to Parquet.
> > > >
> > > > Personally I think moving it back would make sense.
> > > > *However* I have personally contributed a lot more to the Java than
> > the C++
> > > > code base.
> > > > That move was done initially because people contributing to the Arrow
> > and
> > > > Parquet C++ code bases were the same ones and circular dependencies
> > were
> > > > getting in the way (does Parquet depend on Arrow or the other way
> > around?
> > > > At the time it was both ways.). So to make this happen, we need
> enough
> > > > Parquet C++ contributors that would be happy with the move and
> clarify
> > > > which way the dependency goes. My take is that Parquet depends on
> > Arrow but
> > > > I'd be curious to see what others think.
> > > > Julien
> > > >
> > > > On Sat, May 11, 2024 at 2:51 AM Andrew Lamb <an...@gmail.com>
> > > > wrote:
> > > >
> > > > > It is great to see some additional enthusiasm and momentum around
> the
> > > > > Apache Parquet implementation (congratulations on the release of
> > > > parquet-mr
> > > > > 1.14[1]!).
> > > > >
> > > > > As activity picks up, if the desire is to build more community
> around
> > > > > Parquet, perhaps the Parquet PMC wants to encourage moving code
> back
> > to
> > > > > repositories managed by parquet (and out of arrow, for example). I
> > > > realize
> > > > > this would be a technical burden, but it might clarify communities
> > and
> > > > > committers.
> > > > >
> > > > > Andrew
> > > > >
> > > > > [1]:
> > https://lists.apache.org/thread/2gggm938z0x9fx3wtwctfm5htsxlf3z4
> > > > >
> > > > >
> > > > >
> > > > > On Fri, May 10, 2024 at 11:45 PM Matt Topol <
> zotthewizard@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > I just wanted to also poke the question of non-Java developers
> who
> > have
> > > > > > worked on the other parquet implementations potentially being
> > > > recognized
> > > > > as
> > > > > > committers or otherwise on the Parquet project (speaking as the
> > primary
> > > > > > developer of the Go parquet implementation which also lives in
> > the
> > > > Arrow
> > > > > > repository). It would be great to see some active contributors to
> > > > > > parquet-cpp, parquet-go, or otherwise not just being considered
> but
> > > > > > actively becoming committers.
> > > > > >
> > > > > > That's just my two cents from a community perspective.
> > > > > >
> > > > > > --Matt
> > > > > >
> > > > > > On Fri, May 10, 2024, 10:35 PM Jacob Wujciak <
> > assignuser@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Thank you, that sounds great! On first glance some seem to be
> > rather
> > > > > old
> > > > > > > and probably don't apply anymore.
> > > > > > >
> > > > > > > > BTW, do we really need to make a full copy of them to have a
> > mirror
> > > > > in
> > > > > > > the Arrow GitHub issues?
> > > > > > >
> > > > > > > I am not sure I understand what you mean? A full copy of the
> > > > > > > open/closed/all issues? I'd say only the (remaining) open
> > issues
> > > > would
> > > > > be
> > > > > > > fine.
> > > > > > > For reference this is what an imported issue looks like:
> > > > > > > https://github.com/apache/arrow/issues/30543
> > > > > > >
> > > > > > > Am Sa., 11. Mai 2024 um 04:09 Uhr schrieb Gang Wu <
> > ustcwg-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
> > > > >:
> > > > > > >
> > > > > > > > I can initiate the vote. But before the vote, I think we need
> > to
> > > > > > revisit
> > > > > > > > the states of all unresolved tickets and close some as
> needed.
> > > > > > > >
> > > > > > > > BTW, do we really need to make a full copy of them to have a
> > mirror
> > > > > > > > in the Arrow GitHub issues?
> > > > > > > >
> > > > > > > > I'd like to seek a consensus here before sending the vote.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Gang
> > > > > > > >
> > > > > > > > On Sat, May 11, 2024 at 8:46 AM Jacob Wujciak <
> > > > assignuser@apache.org
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hello Everyone!
> > > > > > > > >
> > > > > > > > > It seems there is general agreement on this topic, it would
> > be
> > > > > great
> > > > > > > if a
> > > > > > > > > committer/PMC could start a (lazy consensus) procedural
> vote.
> > > > > > > > >
> > > > > > > > > I will inquire how to handle the parquet-cpp component in
> > jira
> > > > > > (ideally
> > > > > > > > > disabling it, not removing).
> > > > > > > > > There are currently only ~70 open tickets for parquet-cpp,
> > with
> > > > the
> > > > > > > > change
> > > > > > > > > in repo it is probably easier to just move open tickets but
> > I'll
> > > > > > leave
> > > > > > > > that
> > > > > > > > > to Rok who managed the transition of Arrows 20k+ tickets
> too
> > :D
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Jacob
> > > > > > > > >
> > > > > > > > > Arrow committer
> > > > > > > > >
> > > > > > > > > On 2024/04/25 05:31:18 Gang Wu wrote:
> > > > > > > > > > I know we have some non-Java committers and PMCs. But
> > after the
> > > > > > > > > parquet-cpp
> > > > > > > > > > donation, it seems that no one worked on Parquet from
> > arrow
> > > > (cpp,
> > > > > > > rust,
> > > > > > > > > go,
> > > > > > > > > > etc.)
> > > > > > > > > > and other projects are promoted as a Parquet committer.
> > It
> > > > would
> > > > > be
> > > > > > > > > > inconvenient
> > > > > > > > > > for non-Java Parquet developers to work with
> > > > > apache/parquet-format
> > > > > > > and
> > > > > > > > > > apache/parquet-testing repositories. Furthermore, votes
> > from
> > > > > these
> > > > > > > > > > developers
> > > > > > > > > > are not binding for a format change in the ML.
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Gang
> > > > > > > > > >
> > > > > > > > > > On Wed, Apr 24, 2024 at 8:42 PM Uwe L. Korn <
> > uwelk@xhochy.com>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > > Should we consider
> > > > > > > > > > > > Parquet developers from other projects than
> parquet-mr
> > as
> > > > > > Parquet
> > > > > > > > > > > commuters?
> > > > > > > > > > >
> > > > > > > > > > > We are doing this (speaking as a Parquet PMC who didn't
> > work
> > > > on
> > > > > > > > > > > parquet-mr, but parquet-cpp).
> > > > > > > > > > >
> > > > > > > > > > > Best
> > > > > > > > > > > Uwe
> > > > > > > > > > >
> > > > > > > > > > > On Wed, Apr 24, 2024, at 2:38 PM, Gang Wu wrote:
> > > > > > > > > > > > +1 for moving parquet-cpp issues from Apache Jira to
> > > > Arrow's
> > > > > > > GitHub
> > > > > > > > > > > issue.
> > > > > > > > > > > >
> > > > > > > > > > > > Besides, I want to echo Will's question in the
> > thread.
> > > > Should
> > > > > > we
> > > > > > > > > consider
> > > > > > > > > > > > Parquet developers from other projects than
> parquet-mr
> > as
> > > > > > Parquet
> > > > > > > > > > > commiters?
> > > > > > > > > > > > Currently apache/parquet-format and
> > apache/parquet-testing
> > > > > > > > > repositories
> > > > > > > > > > > are
> > > > > > > > > > > > solely governed by Apache Parquet PMC. It would be
> > better
> > > > for
> > > > > > the
> > > > > > > > > entire
> > > > > > > > > > > > Parquet community if developers with sufficient
> > > > contributions
> > > > > > to
> > > > > > > > open
> > > > > > > > > > > source
> > > > > > > > > > > > Parquet projects (including but not limited to
> > parquet-cpp,
> > > > > > > > arrow-rs,
> > > > > > > > > > > cudf,
> > > > > > > > > > > > etc.)
> > > > > > > > > > > > can be considered as Parquet committer and PMC.
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Gang
> > > > > > > > > > > >
> > > > > > > > > > > > On Wed, Apr 24, 2024 at 7:04 PM Uwe L. Korn <
> > > > > uwelk@xhochy.com>
> > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > >> I would be very supportive of this move. The Parquet
> > C++
> > > > > > > > > development has
> > > > > > > > > > > >> been under the umbrella of the Arrow repository for
> > more
> > > > > than
> > > > > > > > > five(six?)
> > > > > > > > > > > >> years now. Thus, the issues should also be aligned
> > with
> > > > the
> > > > > > > Arrow
> > > > > > > > > > > project.
> > > > > > > > > > > >>
> > > > > > > > > > > >> Uwe
> > > > > > > > > > > >>
> > > > > > > > > > > >> On Tue, Apr 23, 2024, at 8:27 PM, Rok Mihevc wrote:
> > > > > > > > > > > >> > Bumping this thread again to see if there is will
> > to
> > > > call
> > > > > > for
> > > > > > > a
> > > > > > > > > vote
> > > > > > > > > > > and
> > > > > > > > > > > >> > move parquet-cpp issues from Apache Jira to
> > Arrow's
> > > > GitHub
> > > > > > > issue
> > > > > > > > > as
> > > > > > > > > > > was
> > > > > > > > > > > >> > done for Arrow.
> > > > > > > > > > > >> > I'm willing to do the move as I already did it for
> > > > Arrow.
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > Rok
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > On Sat, Apr 15, 2023 at 4:53 AM Micah Kornfield <
> > > > > > > > > > > emkornfield@apache.org>
> > > > > > > > > > > >> > wrote:
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >> Bumping this thread again to see in any Parquet
> > PMC
> > > > > members
> > > > > > > can
> > > > > > > > > chime
> > > > > > > > > > > >> >> in/maybe start a formal vote to move governance
> > of
> > > > > > > Parquet-CPP
> > > > > > > > > under
> > > > > > > > > > > the
> > > > > > > > > > > >> >> umbrella.
> > > > > > > > > > > >> >>
> > > > > > > > > > > >> >> -Micah
> > > > > > > > > > > >> >>
> > > > > > > > > > > >> >> On 2023/02/02 10:34:25 Antoine Pitrou wrote:
> > > > > > > > > > > >> >> >
> > > > > > > > > > > >> >> >
> > > > > > > > > > > >> >> > Hi Will,
> > > > > > > > > > > >> >> >
> > > > > > > > > > > >> >> > Le 01/02/2023 à 20:27, Will Jones a écrit :
> > > > > > > > > > > >> >> > >
> > > > > > > > > > > >> >> > > First, it's not obvious where issues are
> > supposed
> > > > to
> > > > > be
> > > > > > > > > open: In
> > > > > > > > > > > >> >> Parquet
> > > > > > > > > > > >> >> > > Jira or Arrow GitHub issues. Looking back at
> > some
> > > > of
> > > > > > the
> > > > > > > > > original
> > > > > > > > > > > >> >> > > discussion, it looks like the intention was
> > > > > > > > > > > >> >> > >
> > > > > > > > > > > >> >> > > * use PARQUET-XXX for issues relating to
> > Parquet
> > > > core
> > > > > > > > > > > >> >> > >> * use ARROW-XXX for issues relation to
> > Arrow's
> > > > > > > consumption
> > > > > > > > > of
> > > > > > > > > > > >> Parquet
> > > > > > > > > > > >> >> > >> core (e.g. changes that are in parquet/arrow
> > right
> > > > > > now)
> > > > > > > > > > > >> >> > >>
> > > > > > > > > > > >> >> > > The README for the old parquet-cpp repo [3]
> > states
> > > > > > > instead
> > > > > > > > in
> > > > > > > > > > > it's
> > > > > > > > > > > >> >> > > migration note:
> > > > > > > > > > > >> >> > >
> > > > > > > > > > > >> >> > > JIRA issues should continue to be opened in
> > the
> > > > > > PARQUET
> > > > > > > > > JIRA
> > > > > > > > > > > >> project.
> > > > > > > > > > > >> >> > >
> > > > > > > > > > > >> >> > > Either way, it doesn't seem like this process
> > is
> > > > > > obvious
> > > > > > > to
> > > > > > > > > > > people.
> > > > > > > > > > > >> >> Perhaps
> > > > > > > > > > > >> >> > > we could clarify this and add notices to
> > Arrow's
> > > > > GitHub
> > > > > > > > > issues
> > > > > > > > > > > >> >> template?
> > > > > > > > > > > >> >> >
> > > > > > > > > > > >> >> > I agree we should clarify this. I have no
> > personal
> > > > > > > > preference,
> > > > > > > > > but
> > > > > > > > > > > I
> > > > > > > > > > > >> >> will note
> > > > > > > > > > > >> >> > that Github issues decrease friction as having
> a
> > GH
> > > > > > account
> > > > > > > > is
> > > > > > > > > > > already
> > > > > > > > > > > >> >> necessary
> > > > > > > > > > > >> >> > for submitting PRs.
> > > > > > > > > > > >> >> >
> > > > > > > > > > > >> >> > > Second, committer status is a little unclear.
> > I am
> > > > a
> > > > > > > > > committer on
> > > > > > > > > > > >> >> Arrow,
> > > > > > > > > > > >> >> > > but not on Parquet right now. Does that mean
> > I
> > > > should
> > > > > > > only
> > > > > > > > > merge
> > > > > > > > > > > >> >> Parquet
> > > > > > > > > > > >> >> > > C++ PRs for code changes in parquet/arrow? Or
> > that
> > > > I
> > > > > > > > > shouldn't
> > > > > > > > > > > merge
> > > > > > > > > > > >> >> > > Parquet changes at all?
> > > > > > > > > > > >> >> >
> > > > > > > > > > > >> >> > Since Parquet C++ is part of Arrow C++, you are
> > > > allowed
> > > > > > to
> > > > > > > > > merge
> > > > > > > > > > > >> Parquet
> > > > > > > > > > > >> >> C++
> > > > > > > > > > > >> >> > changes. As always you should ensure you have
> > > > > sufficient
> > > > > > > > > > > understanding
> > > > > > > > > > > >> >> of the
> > > > > > > > > > > >> >> > contribution, and that it follows established
> > > > > practices:
> > > > > > > > > > > >> >> >
> > > > > > > https://arrow.apache.org/docs/dev/developers/reviewing.html
> > > > > > > > > > > >> >> >
> > > > > > > > > > > >> >> > > Also, are the contributions to Arrow C++
> > Parquet
> > > > > being
> > > > > > > > > actively
> > > > > > > > > > > >> >> reviewed
> > > > > > > > > > > >> >> > > for potential new committers?
> > > > > > > > > > > >> >> >
> > > > > > > > > > > >> >> > I would certainly do.
> > > > > > > > > > > >> >> >
> > > > > > > > > > > >> >> > Regards
> > > > > > > > > > > >> >> >
> > > > > > > > > > > >> >> > Antoine.
> > > > > > > > > > > >> >> >
> > > > > > > > > > > >> >> >
> > > > > > > > > > > >> >>
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> >
>
Re: [C++] Parquet and Arrow overlap
Posted by Micah Kornfield <em...@gmail.com>.
From my perspective I agree, that I don't think there is benefit of moving
parquet C++ out of arrow given what it would actually cost to make clean
boundaries. I also don't think it will hurt iteration speed.
I think the main challenge could be in compatibility testing, but Arrow has
solved this between implementations that live in different repositories so
I think the same solutions could apply for Parquet.
On Thu, May 16, 2024 at 12:57 AM Antoine Pitrou <an...@python.org> wrote:
> On Tue, 14 May 2024 10:22:37 -0700
> Julien Le Dem <ju...@apache.org> wrote:
> > 1. I think we should make it easy for people contributing to the C++
> > codebase. (which is why I voted for the move at the time)
> > 2. If merging repos removes the need to deal with the circular dependency
> > between repos issue for the C++ code bases, it does it at the expense of
> > making it easy to evolve the parquet spec and the java and c++
> > implementations together.
>
> Hmm... I'm not sure I understand your point here. The Parquet spec and
> the Java implementation are already living in distinct repos and have
> distinct versioning schemes. The main thing that they share in common is
> the JIRA instance (while the C++ Parquet implementation mostly relies on
> Arrow's GH issue tracker), but is that really important?
>
> > parquet-cpp depends only on arrow-core that does not have to depend on
> > parquet-cpp.
>
> That is true.
>
> > Other components like
> > arrow-dataset and pyarrow can depend on parquet-cpp just like they depend
> > on orc externally.
>
> Ideally yes. In practice there are two problems:
> 1) it creates a circular dependency between *repositories*.
> 2) the C++ Arrow Datasets component is not built independently, it is an
> optional component when building Arrow C++. So we would also have a
> chicken-and-egg problem when building Arrow C++ and Parquet C++.
>
> > I realize that would be work to make it happen, but the current location
> of
> > the parquet-cpp codebase is a big trade-off of prioritizing quick
> iteration
> > on the C++ implementations over iteration on the format.
>
> Having recently worked on a format addition and its respective
> implementations (in Java and C++), I haven't found the current setup
> more difficult to work with for Parquet C++ than it was for Parquet
> Java. Admittedly I'm biased, being a heavy contributor to Arrow C++,
> but I'm curious why the current situation would be detrimental to
> iteration on the format.
>
> Regards
>
> Antoine.
>
>
>
Re: [C++] Parquet and Arrow overlap
Posted by Antoine Pitrou <an...@python.org>.
On Tue, 14 May 2024 10:22:37 -0700
Julien Le Dem <ju...@apache.org> wrote:
> 1. I think we should make it easy for people contributing to the C++
> codebase. (which is why I voted for the move at the time)
> 2. If merging repos removes the need to deal with the circular dependency
> between repos issue for the C++ code bases, it does it at the expense of
> making it easy to evolve the parquet spec and the java and c++
> implementations together.
Hmm... I'm not sure I understand your point here. The Parquet spec and
the Java implementation are already living in distinct repos and have
distinct versioning schemes. The main thing that they share in common is
the JIRA instance (while the C++ Parquet implementation mostly relies on
Arrow's GH issue tracker), but is that really important?
> parquet-cpp depends only on arrow-core that does not have to depend on
> parquet-cpp.
That is true.
> Other components like
> arrow-dataset and pyarrow can depend on parquet-cpp just like they depend
> on orc externally.
Ideally yes. In practice there are two problems:
1) it creates a circular dependency between *repositories*.
2) the C++ Arrow Datasets component is not built independently, it is an
optional component when building Arrow C++. So we would also have a
chicken-and-egg problem when building Arrow C++ and Parquet C++.
> I realize that would be work to make it happen, but the current location of
> the parquet-cpp codebase is a big trade-off of prioritizing quick iteration
> on the C++ implementations over iteration on the format.
Having recently worked on a format addition and its respective
implementations (in Java and C++), I haven't found the current setup
more difficult to work with for Parquet C++ than it was for Parquet
Java. Admittedly I'm biased, being a heavy contributor to Arrow C++,
but I'm curious why the current situation would be detrimental to
iteration on the format.
Regards
Antoine.
Re: [C++] Parquet and Arrow overlap
Posted by Julien Le Dem <ju...@apache.org>.
1. I think we should make it easy for people contributing to the C++
codebase. (which is why I voted for the move at the time)
2. If merging repos removes the need to deal with the circular dependency
between repos issue for the C++ code bases, it does it at the expense of
making it easy to evolve the parquet spec and the java and c++
implementations together.
This setup was optimized for quick iterations on the APIs on the C++ side.
Now that those APIs are more stable, it is less needed IMO.
parquet-cpp depends only on arrow-core that does not have to depend on
parquet-cpp. It really just needs the vectors. Other components like
arrow-dataset and pyarrow can depend on parquet-cpp just like they depend
on orc externally.
I realize that would be work to make it happen, but the current location of
the parquet-cpp codebase is a big trade-off of prioritizing quick iteration
on the C++ implementations over iteration on the format. As interest grows
in evolving the format, I think it warrants a re-evaluation.
On Tue, May 14, 2024 at 9:20 AM Antoine Pitrou <an...@python.org> wrote:
>
> Moving Parquet C++ out of Arrow C++ would basically recreate the
> problems that motivated the integration of Parquet C++ into Arrow C++
> :-)
>
> Regards
>
> Antoine.
>
>
> On Tue, 14 May 2024 13:52:15 +0800
> Gang Wu <us...@gmail.com> wrote:
> > IMO, moving parquet-cpp out of arrow is challenging as the dependency
> > chain looks like: arrow core <- parquet-cpp <- arrow dataset <- pyarrow
> >
> > Best,
> > Gang
> >
> > On Tue, May 14, 2024 at 12:38 PM Julien Le Dem <
> julien-1oDqGaOF3Lkdnm+yROfE0A@public.gmane.org> wrote:
> >
> > > It is great to see more momentum building.
> > > I have myself a little bit more time to contribute to Parquet.
> > >
> > > Personally I think moving it back would make sense.
> > > *However* I have personally contributed a lot more to the Java than
> the C++
> > > code base.
> > > That move was done initially because people contributing to the Arrow
> and
> > > Parquet C++ code bases were the same ones and circular dependencies
> were
> > > getting in the way (does Parquet depend on Arrow or the other way
> around?
> > > At the time it was both ways.). So to make this happen, we need enough
> > > Parquet C++ contributors that would be happy with the move and clarify
> > > which way the dependency goes. My take is that Parquet depends on
> Arrow but
> > > I'd be curious to see what others think.
> > > Julien
> > >
> > > On Sat, May 11, 2024 at 2:51 AM Andrew Lamb <an...@gmail.com>
> > > wrote:
> > >
> > > > It is great to see some additional enthusiasm and momentum around the
> > > > Apache Parquet implementation (congratulations on the release of
> > > parquet-mr
> > > > 1.14[1]!).
> > > >
> > > > As activity picks up, if the desire is to build more community around
> > > > Parquet, perhaps the Parquet PMC wants to encourage moving code back
> to
> > > > repositories managed by parquet (and out of arrow, for example). I
> > > realize
> > > > this would be a technical burden, but it might clarify communities
> and
> > > > committers.
> > > >
> > > > Andrew
> > > >
> > > > [1]:
> https://lists.apache.org/thread/2gggm938z0x9fx3wtwctfm5htsxlf3z4
> > > >
> > > >
> > > >
> > > > On Fri, May 10, 2024 at 11:45 PM Matt Topol <zo...@gmail.com>
> > > > wrote:
> > > >
> > > > > I just wanted to also poke the question of non-Java developers who
> have
> > > > > worked on the other parquet implementations potentially being
> > > recognized
> > > > as
> > > > > committers or otherwise on the Parquet project (speaking as the
> primary
> > > > > developer of the Go parquet implementation which also lives in
> the
> > > Arrow
> > > > > repository). It would be great to see some active contributors to
> > > > > parquet-cpp, parquet-go, or otherwise not just being considered but
> > > > > actively becoming committers.
> > > > >
> > > > > That's just my two cents from a community perspective.
> > > > >
> > > > > --Matt
> > > > >
> > > > > On Fri, May 10, 2024, 10:35 PM Jacob Wujciak <
> assignuser@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Thank you, that sounds great! On first glance some seem to be
> rather
> > > > old
> > > > > > and probably don't apply anymore.
> > > > > >
> > > > > > > BTW, do we really need to make a full copy of them to have a
> mirror
> > > > in
> > > > > > the Arrow GitHub issues?
> > > > > >
> > > > > > I am not sure I understand what you mean? A full copy of the
> > > > > > open/closed/all issues? I'd say only the (remaining) open
> issues
> > > would
> > > > be
> > > > > > fine.
> > > > > > For reference this is what an imported issue looks like:
> > > > > > https://github.com/apache/arrow/issues/30543
> > > > > >
> > > > > > Am Sa., 11. Mai 2024 um 04:09 Uhr schrieb Gang Wu <
> ustcwg-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
> > > >:
> > > > > >
> > > > > > > I can initiate the vote. But before the vote, I think we need
> to
> > > > > revisit
> > > > > > > the states of all unresolved tickets and close some as needed.
> > > > > > >
> > > > > > > BTW, do we really need to make a full copy of them to have a
> mirror
> > > > > > > in the Arrow GitHub issues?
> > > > > > >
> > > > > > > I'd like to seek a consensus here before sending the vote.
> > > > > > >
> > > > > > > Best,
> > > > > > > Gang
> > > > > > >
> > > > > > > On Sat, May 11, 2024 at 8:46 AM Jacob Wujciak <
> > > assignuser@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello Everyone!
> > > > > > > >
> > > > > > > > It seems there is general agreement on this topic, it would
> be
> > > > great
> > > > > > if a
> > > > > > > > committer/PMC could start a (lazy consensus) procedural vote.
> > > > > > > >
> > > > > > > > I will inquire how to handle the parquet-cpp component in
> jira
> > > > > (ideally
> > > > > > > > disabling it, not removing).
> > > > > > > > There are currently only ~70 open tickets for parquet-cpp,
> with
> > > the
> > > > > > > change
> > > > > > > > in repo it is probably easier to just move open tickets but
> I'll
> > > > > leave
> > > > > > > that
> > > > > > > > to Rok who managed the transition of Arrows 20k+ tickets too
> :D
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Jacob
> > > > > > > >
> > > > > > > > Arrow committer
> > > > > > > >
> > > > > > > > On 2024/04/25 05:31:18 Gang Wu wrote:
> > > > > > > > > I know we have some non-Java committers and PMCs. But
> after the
> > > > > > > > parquet-cpp
> > > > > > > > > donation, it seems that no one worked on Parquet from
> arrow
> > > (cpp,
> > > > > > rust,
> > > > > > > > go,
> > > > > > > > > etc.)
> > > > > > > > > and other projects are promoted as a Parquet committer.
> It
> > > would
> > > > be
> > > > > > > > > inconvenient
> > > > > > > > > for non-Java Parquet developers to work with
> > > > apache/parquet-format
> > > > > > and
> > > > > > > > > apache/parquet-testing repositories. Furthermore, votes
> from
> > > > these
> > > > > > > > > developers
> > > > > > > > > are not binding for a format change in the ML.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Gang
> > > > > > > > >
> > > > > > > > > On Wed, Apr 24, 2024 at 8:42 PM Uwe L. Korn <
> uwelk@xhochy.com>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > > Should we consider
> > > > > > > > > > > Parquet developers from other projects than parquet-mr
> as
> > > > > Parquet
> > > > > > > > > > commuters?
> > > > > > > > > >
> > > > > > > > > > We are doing this (speaking as a Parquet PMC who didn't
> work
> > > on
> > > > > > > > > > parquet-mr, but parquet-cpp).
> > > > > > > > > >
> > > > > > > > > > Best
> > > > > > > > > > Uwe
> > > > > > > > > >
> > > > > > > > > > On Wed, Apr 24, 2024, at 2:38 PM, Gang Wu wrote:
> > > > > > > > > > > +1 for moving parquet-cpp issues from Apache Jira to
> > > Arrow's
> > > > > > GitHub
> > > > > > > > > > issue.
> > > > > > > > > > >
> > > > > > > > > > > Besides, I want to echo Will's question in the
> thread.
> > > Should
> > > > > we
> > > > > > > > consider
> > > > > > > > > > > Parquet developers from other projects than parquet-mr
> as
> > > > > Parquet
> > > > > > > > > > commiters?
> > > > > > > > > > > Currently apache/parquet-format and
> apache/parquet-testing
> > > > > > > > repositories
> > > > > > > > > > are
> > > > > > > > > > > solely governed by Apache Parquet PMC. It would be
> better
> > > for
> > > > > the
> > > > > > > > entire
> > > > > > > > > > > Parquet community if developers with sufficient
> > > contributions
> > > > > to
> > > > > > > open
> > > > > > > > > > source
> > > > > > > > > > > Parquet projects (including but not limited to
> parquet-cpp,
> > > > > > > arrow-rs,
> > > > > > > > > > cudf,
> > > > > > > > > > > etc.)
> > > > > > > > > > > can be considered as Parquet committer and PMC.
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Gang
> > > > > > > > > > >
> > > > > > > > > > > On Wed, Apr 24, 2024 at 7:04 PM Uwe L. Korn <
> > > > uwelk@xhochy.com>
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >> I would be very supportive of this move. The Parquet
> C++
> > > > > > > > development has
> > > > > > > > > > >> been under the umbrella of the Arrow repository for
> more
> > > > than
> > > > > > > > five(six?)
> > > > > > > > > > >> years now. Thus, the issues should also be aligned
> with
> > > the
> > > > > > Arrow
> > > > > > > > > > project.
> > > > > > > > > > >>
> > > > > > > > > > >> Uwe
> > > > > > > > > > >>
> > > > > > > > > > >> On Tue, Apr 23, 2024, at 8:27 PM, Rok Mihevc wrote:
> > > > > > > > > > >> > Bumping this thread again to see if there is will
> to
> > > call
> > > > > for
> > > > > > a
> > > > > > > > vote
> > > > > > > > > > and
> > > > > > > > > > >> > move parquet-cpp issues from Apache Jira to
> Arrow's
> > > GitHub
> > > > > > issue
> > > > > > > > as
> > > > > > > > > > was
> > > > > > > > > > >> > done for Arrow.
> > > > > > > > > > >> > I'm willing to do the move as I already did it for
> > > Arrow.
> > > > > > > > > > >> >
> > > > > > > > > > >> > Rok
> > > > > > > > > > >> >
> > > > > > > > > > >> > On Sat, Apr 15, 2023 at 4:53 AM Micah Kornfield <
> > > > > > > > > > emkornfield@apache.org>
> > > > > > > > > > >> > wrote:
> > > > > > > > > > >> >
> > > > > > > > > > >> >> Bumping this thread again to see in any Parquet
> PMC
> > > > members
> > > > > > can
> > > > > > > > chime
> > > > > > > > > > >> >> in/maybe start a formal vote to move governance
> of
> > > > > > Parquet-CPP
> > > > > > > > under
> > > > > > > > > > the
> > > > > > > > > > >> >> umbrella.
> > > > > > > > > > >> >>
> > > > > > > > > > >> >> -Micah
> > > > > > > > > > >> >>
> > > > > > > > > > >> >> On 2023/02/02 10:34:25 Antoine Pitrou wrote:
> > > > > > > > > > >> >> >
> > > > > > > > > > >> >> >
> > > > > > > > > > >> >> > Hi Will,
> > > > > > > > > > >> >> >
> > > > > > > > > > >> >> > Le 01/02/2023 à 20:27, Will Jones a écrit :
> > > > > > > > > > >> >> > >
> > > > > > > > > > >> >> > > First, it's not obvious where issues are
> supposed
> > > to
> > > > be
> > > > > > > > open: In
> > > > > > > > > > >> >> Parquet
> > > > > > > > > > >> >> > > Jira or Arrow GitHub issues. Looking back at
> some
> > > of
> > > > > the
> > > > > > > > original
> > > > > > > > > > >> >> > > discussion, it looks like the intention was
> > > > > > > > > > >> >> > >
> > > > > > > > > > >> >> > > * use PARQUET-XXX for issues relating to
> Parquet
> > > core
> > > > > > > > > > >> >> > >> * use ARROW-XXX for issues relation to
> Arrow's
> > > > > > consumption
> > > > > > > > of
> > > > > > > > > > >> Parquet
> > > > > > > > > > >> >> > >> core (e.g. changes that are in parquet/arrow
> right
> > > > > now)
> > > > > > > > > > >> >> > >>
> > > > > > > > > > >> >> > > The README for the old parquet-cpp repo [3]
> states
> > > > > > instead
> > > > > > > in
> > > > > > > > > > it's
> > > > > > > > > > >> >> > > migration note:
> > > > > > > > > > >> >> > >
> > > > > > > > > > >> >> > > JIRA issues should continue to be opened in
> the
> > > > > PARQUET
> > > > > > > > JIRA
> > > > > > > > > > >> project.
> > > > > > > > > > >> >> > >
> > > > > > > > > > >> >> > > Either way, it doesn't seem like this process
> is
> > > > > obvious
> > > > > > to
> > > > > > > > > > people.
> > > > > > > > > > >> >> Perhaps
> > > > > > > > > > >> >> > > we could clarify this and add notices to
> Arrow's
> > > > GitHub
> > > > > > > > issues
> > > > > > > > > > >> >> template?
> > > > > > > > > > >> >> >
> > > > > > > > > > >> >> > I agree we should clarify this. I have no
> personal
> > > > > > > preference,
> > > > > > > > but
> > > > > > > > > > I
> > > > > > > > > > >> >> will note
> > > > > > > > > > >> >> > that Github issues decrease friction as having a
> GH
> > > > > account
> > > > > > > is
> > > > > > > > > > already
> > > > > > > > > > >> >> necessary
> > > > > > > > > > >> >> > for submitting PRs.
> > > > > > > > > > >> >> >
> > > > > > > > > > >> >> > > Second, committer status is a little unclear.
> I am
> > > a
> > > > > > > > committer on
> > > > > > > > > > >> >> Arrow,
> > > > > > > > > > >> >> > > but not on Parquet right now. Does that mean
> I
> > > should
> > > > > > only
> > > > > > > > merge
> > > > > > > > > > >> >> Parquet
> > > > > > > > > > >> >> > > C++ PRs for code changes in parquet/arrow? Or
> that
> > > I
> > > > > > > > shouldn't
> > > > > > > > > > merge
> > > > > > > > > > >> >> > > Parquet changes at all?
> > > > > > > > > > >> >> >
> > > > > > > > > > >> >> > Since Parquet C++ is part of Arrow C++, you are
> > > allowed
> > > > > to
> > > > > > > > merge
> > > > > > > > > > >> Parquet
> > > > > > > > > > >> >> C++
> > > > > > > > > > >> >> > changes. As always you should ensure you have
> > > > sufficient
> > > > > > > > > > understanding
> > > > > > > > > > >> >> of the
> > > > > > > > > > >> >> > contribution, and that it follows established
> > > > practices:
> > > > > > > > > > >> >> >
> > > > > > https://arrow.apache.org/docs/dev/developers/reviewing.html
> > > > > > > > > > >> >> >
> > > > > > > > > > >> >> > > Also, are the contributions to Arrow C++
> Parquet
> > > > being
> > > > > > > > actively
> > > > > > > > > > >> >> reviewed
> > > > > > > > > > >> >> > > for potential new committers?
> > > > > > > > > > >> >> >
> > > > > > > > > > >> >> > I would certainly do.
> > > > > > > > > > >> >> >
> > > > > > > > > > >> >> > Regards
> > > > > > > > > > >> >> >
> > > > > > > > > > >> >> > Antoine.
> > > > > > > > > > >> >> >
> > > > > > > > > > >> >> >
> > > > > > > > > > >> >>
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
>
>
Re: [C++] Parquet and Arrow overlap
Posted by Antoine Pitrou <an...@python.org>.
Moving Parquet C++ out of Arrow C++ would basically recreate the
problems that motivated the integration of Parquet C++ into Arrow C++
:-)
Regards
Antoine.
On Tue, 14 May 2024 13:52:15 +0800
Gang Wu <us...@gmail.com> wrote:
> IMO, moving parquet-cpp out of arrow is challenging as the dependency
> chain looks like: arrow core <- parquet-cpp <- arrow dataset <- pyarrow
>
> Best,
> Gang
>
> On Tue, May 14, 2024 at 12:38 PM Julien Le Dem <ju...@public.gmane.org> wrote:
>
> > It is great to see more momentum building.
> > I have myself a little bit more time to contribute to Parquet.
> >
> > Personally I think moving it back would make sense.
> > *However* I have personally contributed a lot more to the Java than the C++
> > code base.
> > That move was done initially because people contributing to the Arrow and
> > Parquet C++ code bases were the same ones and circular dependencies were
> > getting in the way (does Parquet depend on Arrow or the other way around?
> > At the time it was both ways.). So to make this happen, we need enough
> > Parquet C++ contributors that would be happy with the move and clarify
> > which way the dependency goes. My take is that Parquet depends on Arrow but
> > I'd be curious to see what others think.
> > Julien
> >
> > On Sat, May 11, 2024 at 2:51 AM Andrew Lamb <an...@gmail.com>
> > wrote:
> >
> > > It is great to see some additional enthusiasm and momentum around the
> > > Apache Parquet implementation (congratulations on the release of
> > parquet-mr
> > > 1.14[1]!).
> > >
> > > As activity picks up, if the desire is to build more community around
> > > Parquet, perhaps the Parquet PMC wants to encourage moving code back to
> > > repositories managed by parquet (and out of arrow, for example). I
> > realize
> > > this would be a technical burden, but it might clarify communities and
> > > committers.
> > >
> > > Andrew
> > >
> > > [1]: https://lists.apache.org/thread/2gggm938z0x9fx3wtwctfm5htsxlf3z4
> > >
> > >
> > >
> > > On Fri, May 10, 2024 at 11:45 PM Matt Topol <zo...@gmail.com>
> > > wrote:
> > >
> > > > I just wanted to also poke the question of non-Java developers who have
> > > > worked on the other parquet implementations potentially being
> > recognized
> > > as
> > > > committers or otherwise on the Parquet project (speaking as the primary
> > > > developer of the Go parquet implementation which also lives in the
> > Arrow
> > > > repository). It would be great to see some active contributors to
> > > > parquet-cpp, parquet-go, or otherwise not just being considered but
> > > > actively becoming committers.
> > > >
> > > > That's just my two cents from a community perspective.
> > > >
> > > > --Matt
> > > >
> > > > On Fri, May 10, 2024, 10:35 PM Jacob Wujciak <as...@apache.org>
> > > > wrote:
> > > >
> > > > > Thank you, that sounds great! On first glance some seem to be rather
> > > old
> > > > > and probably don't apply anymore.
> > > > >
> > > > > > BTW, do we really need to make a full copy of them to have a mirror
> > > in
> > > > > the Arrow GitHub issues?
> > > > >
> > > > > I am not sure I understand what you mean? A full copy of the
> > > > > open/closed/all issues? I'd say only the (remaining) open issues
> > would
> > > be
> > > > > fine.
> > > > > For reference this is what an imported issue looks like:
> > > > > https://github.com/apache/arrow/issues/30543
> > > > >
> > > > > Am Sa., 11. Mai 2024 um 04:09 Uhr schrieb Gang Wu <ustcwg-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
> > >:
> > > > >
> > > > > > I can initiate the vote. But before the vote, I think we need to
> > > > revisit
> > > > > > the states of all unresolved tickets and close some as needed.
> > > > > >
> > > > > > BTW, do we really need to make a full copy of them to have a mirror
> > > > > > in the Arrow GitHub issues?
> > > > > >
> > > > > > I'd like to seek a consensus here before sending the vote.
> > > > > >
> > > > > > Best,
> > > > > > Gang
> > > > > >
> > > > > > On Sat, May 11, 2024 at 8:46 AM Jacob Wujciak <
> > assignuser@apache.org
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hello Everyone!
> > > > > > >
> > > > > > > It seems there is general agreement on this topic, it would be
> > > great
> > > > > if a
> > > > > > > committer/PMC could start a (lazy consensus) procedural vote.
> > > > > > >
> > > > > > > I will inquire how to handle the parquet-cpp component in jira
> > > > (ideally
> > > > > > > disabling it, not removing).
> > > > > > > There are currently only ~70 open tickets for parquet-cpp, with
> > the
> > > > > > change
> > > > > > > in repo it is probably easier to just move open tickets but I'll
> > > > leave
> > > > > > that
> > > > > > > to Rok who managed the transition of Arrows 20k+ tickets too :D
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Jacob
> > > > > > >
> > > > > > > Arrow committer
> > > > > > >
> > > > > > > On 2024/04/25 05:31:18 Gang Wu wrote:
> > > > > > > > I know we have some non-Java committers and PMCs. But after the
> > > > > > > parquet-cpp
> > > > > > > > donation, it seems that no one worked on Parquet from arrow
> > (cpp,
> > > > > rust,
> > > > > > > go,
> > > > > > > > etc.)
> > > > > > > > and other projects are promoted as a Parquet committer. It
> > would
> > > be
> > > > > > > > inconvenient
> > > > > > > > for non-Java Parquet developers to work with
> > > apache/parquet-format
> > > > > and
> > > > > > > > apache/parquet-testing repositories. Furthermore, votes from
> > > these
> > > > > > > > developers
> > > > > > > > are not binding for a format change in the ML.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Gang
> > > > > > > >
> > > > > > > > On Wed, Apr 24, 2024 at 8:42 PM Uwe L. Korn <uw...@xhochy.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > > Should we consider
> > > > > > > > > > Parquet developers from other projects than parquet-mr as
> > > > Parquet
> > > > > > > > > commuters?
> > > > > > > > >
> > > > > > > > > We are doing this (speaking as a Parquet PMC who didn't work
> > on
> > > > > > > > > parquet-mr, but parquet-cpp).
> > > > > > > > >
> > > > > > > > > Best
> > > > > > > > > Uwe
> > > > > > > > >
> > > > > > > > > On Wed, Apr 24, 2024, at 2:38 PM, Gang Wu wrote:
> > > > > > > > > > +1 for moving parquet-cpp issues from Apache Jira to
> > Arrow's
> > > > > GitHub
> > > > > > > > > issue.
> > > > > > > > > >
> > > > > > > > > > Besides, I want to echo Will's question in the thread.
> > Should
> > > > we
> > > > > > > consider
> > > > > > > > > > Parquet developers from other projects than parquet-mr as
> > > > Parquet
> > > > > > > > > commiters?
> > > > > > > > > > Currently apache/parquet-format and apache/parquet-testing
> > > > > > > repositories
> > > > > > > > > are
> > > > > > > > > > solely governed by Apache Parquet PMC. It would be better
> > for
> > > > the
> > > > > > > entire
> > > > > > > > > > Parquet community if developers with sufficient
> > contributions
> > > > to
> > > > > > open
> > > > > > > > > source
> > > > > > > > > > Parquet projects (including but not limited to parquet-cpp,
> > > > > > arrow-rs,
> > > > > > > > > cudf,
> > > > > > > > > > etc.)
> > > > > > > > > > can be considered as Parquet committer and PMC.
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Gang
> > > > > > > > > >
> > > > > > > > > > On Wed, Apr 24, 2024 at 7:04 PM Uwe L. Korn <
> > > uwelk@xhochy.com>
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> I would be very supportive of this move. The Parquet C++
> > > > > > > development has
> > > > > > > > > >> been under the umbrella of the Arrow repository for more
> > > than
> > > > > > > five(six?)
> > > > > > > > > >> years now. Thus, the issues should also be aligned with
> > the
> > > > > Arrow
> > > > > > > > > project.
> > > > > > > > > >>
> > > > > > > > > >> Uwe
> > > > > > > > > >>
> > > > > > > > > >> On Tue, Apr 23, 2024, at 8:27 PM, Rok Mihevc wrote:
> > > > > > > > > >> > Bumping this thread again to see if there is will to
> > call
> > > > for
> > > > > a
> > > > > > > vote
> > > > > > > > > and
> > > > > > > > > >> > move parquet-cpp issues from Apache Jira to Arrow's
> > GitHub
> > > > > issue
> > > > > > > as
> > > > > > > > > was
> > > > > > > > > >> > done for Arrow.
> > > > > > > > > >> > I'm willing to do the move as I already did it for
> > Arrow.
> > > > > > > > > >> >
> > > > > > > > > >> > Rok
> > > > > > > > > >> >
> > > > > > > > > >> > On Sat, Apr 15, 2023 at 4:53 AM Micah Kornfield <
> > > > > > > > > emkornfield@apache.org>
> > > > > > > > > >> > wrote:
> > > > > > > > > >> >
> > > > > > > > > >> >> Bumping this thread again to see in any Parquet PMC
> > > members
> > > > > can
> > > > > > > chime
> > > > > > > > > >> >> in/maybe start a formal vote to move governance of
> > > > > Parquet-CPP
> > > > > > > under
> > > > > > > > > the
> > > > > > > > > >> >> umbrella.
> > > > > > > > > >> >>
> > > > > > > > > >> >> -Micah
> > > > > > > > > >> >>
> > > > > > > > > >> >> On 2023/02/02 10:34:25 Antoine Pitrou wrote:
> > > > > > > > > >> >> >
> > > > > > > > > >> >> >
> > > > > > > > > >> >> > Hi Will,
> > > > > > > > > >> >> >
> > > > > > > > > >> >> > Le 01/02/2023 à 20:27, Will Jones a écrit :
> > > > > > > > > >> >> > >
> > > > > > > > > >> >> > > First, it's not obvious where issues are supposed
> > to
> > > be
> > > > > > > open: In
> > > > > > > > > >> >> Parquet
> > > > > > > > > >> >> > > Jira or Arrow GitHub issues. Looking back at some
> > of
> > > > the
> > > > > > > original
> > > > > > > > > >> >> > > discussion, it looks like the intention was
> > > > > > > > > >> >> > >
> > > > > > > > > >> >> > > * use PARQUET-XXX for issues relating to Parquet
> > core
> > > > > > > > > >> >> > >> * use ARROW-XXX for issues relation to Arrow's
> > > > > consumption
> > > > > > > of
> > > > > > > > > >> Parquet
> > > > > > > > > >> >> > >> core (e.g. changes that are in parquet/arrow right
> > > > now)
> > > > > > > > > >> >> > >>
> > > > > > > > > >> >> > > The README for the old parquet-cpp repo [3] states
> > > > > instead
> > > > > > in
> > > > > > > > > it's
> > > > > > > > > >> >> > > migration note:
> > > > > > > > > >> >> > >
> > > > > > > > > >> >> > > JIRA issues should continue to be opened in the
> > > > PARQUET
> > > > > > > JIRA
> > > > > > > > > >> project.
> > > > > > > > > >> >> > >
> > > > > > > > > >> >> > > Either way, it doesn't seem like this process is
> > > > obvious
> > > > > to
> > > > > > > > > people.
> > > > > > > > > >> >> Perhaps
> > > > > > > > > >> >> > > we could clarify this and add notices to Arrow's
> > > GitHub
> > > > > > > issues
> > > > > > > > > >> >> template?
> > > > > > > > > >> >> >
> > > > > > > > > >> >> > I agree we should clarify this. I have no personal
> > > > > > preference,
> > > > > > > but
> > > > > > > > > I
> > > > > > > > > >> >> will note
> > > > > > > > > >> >> > that Github issues decrease friction as having a GH
> > > > account
> > > > > > is
> > > > > > > > > already
> > > > > > > > > >> >> necessary
> > > > > > > > > >> >> > for submitting PRs.
> > > > > > > > > >> >> >
> > > > > > > > > >> >> > > Second, committer status is a little unclear. I am
> > a
> > > > > > > committer on
> > > > > > > > > >> >> Arrow,
> > > > > > > > > >> >> > > but not on Parquet right now. Does that mean I
> > should
> > > > > only
> > > > > > > merge
> > > > > > > > > >> >> Parquet
> > > > > > > > > >> >> > > C++ PRs for code changes in parquet/arrow? Or that
> > I
> > > > > > > shouldn't
> > > > > > > > > merge
> > > > > > > > > >> >> > > Parquet changes at all?
> > > > > > > > > >> >> >
> > > > > > > > > >> >> > Since Parquet C++ is part of Arrow C++, you are
> > allowed
> > > > to
> > > > > > > merge
> > > > > > > > > >> Parquet
> > > > > > > > > >> >> C++
> > > > > > > > > >> >> > changes. As always you should ensure you have
> > > sufficient
> > > > > > > > > understanding
> > > > > > > > > >> >> of the
> > > > > > > > > >> >> > contribution, and that it follows established
> > > practices:
> > > > > > > > > >> >> >
> > > > > https://arrow.apache.org/docs/dev/developers/reviewing.html
> > > > > > > > > >> >> >
> > > > > > > > > >> >> > > Also, are the contributions to Arrow C++ Parquet
> > > being
> > > > > > > actively
> > > > > > > > > >> >> reviewed
> > > > > > > > > >> >> > > for potential new committers?
> > > > > > > > > >> >> >
> > > > > > > > > >> >> > I would certainly do.
> > > > > > > > > >> >> >
> > > > > > > > > >> >> > Regards
> > > > > > > > > >> >> >
> > > > > > > > > >> >> > Antoine.
> > > > > > > > > >> >> >
> > > > > > > > > >> >> >
> > > > > > > > > >> >>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Fwd: [C++] Parquet and Arrow overlap
Posted by Gang Wu <us...@gmail.com>.
IMO, moving parquet-cpp out of arrow is challenging as the dependency
chain looks like: arrow core <- parquet-cpp <- arrow dataset <- pyarrow
Best,
Gang
On Tue, May 14, 2024 at 12:38 PM Julien Le Dem <ju...@apache.org> wrote:
> It is great to see more momentum building.
> I have myself a little bit more time to contribute to Parquet.
>
> Personally I think moving it back would make sense.
> *However* I have personally contributed a lot more to the Java than the C++
> code base.
> That move was done initially because people contributing to the Arrow and
> Parquet C++ code bases were the same ones and circular dependencies were
> getting in the way (does Parquet depend on Arrow or the other way around?
> At the time it was both ways.). So to make this happen, we need enough
> Parquet C++ contributors that would be happy with the move and clarify
> which way the dependency goes. My take is that Parquet depends on Arrow but
> I'd be curious to see what others think.
> Julien
>
> On Sat, May 11, 2024 at 2:51 AM Andrew Lamb <an...@gmail.com>
> wrote:
>
> > It is great to see some additional enthusiasm and momentum around the
> > Apache Parquet implementation (congratulations on the release of
> parquet-mr
> > 1.14[1]!).
> >
> > As activity picks up, if the desire is to build more community around
> > Parquet, perhaps the Parquet PMC wants to encourage moving code back to
> > repositories managed by parquet (and out of arrow, for example). I
> realize
> > this would be a technical burden, but it might clarify communities and
> > committers.
> >
> > Andrew
> >
> > [1]: https://lists.apache.org/thread/2gggm938z0x9fx3wtwctfm5htsxlf3z4
> >
> >
> >
> > On Fri, May 10, 2024 at 11:45 PM Matt Topol <zo...@gmail.com>
> > wrote:
> >
> > > I just wanted to also poke the question of non-Java developers who have
> > > worked on the other parquet implementations potentially being
> recognized
> > as
> > > committers or otherwise on the Parquet project (speaking as the primary
> > > developer of the Go parquet implementation which also lives in the
> Arrow
> > > repository). It would be great to see some active contributors to
> > > parquet-cpp, parquet-go, or otherwise not just being considered but
> > > actively becoming committers.
> > >
> > > That's just my two cents from a community perspective.
> > >
> > > --Matt
> > >
> > > On Fri, May 10, 2024, 10:35 PM Jacob Wujciak <as...@apache.org>
> > > wrote:
> > >
> > > > Thank you, that sounds great! On first glance some seem to be rather
> > old
> > > > and probably don't apply anymore.
> > > >
> > > > > BTW, do we really need to make a full copy of them to have a mirror
> > in
> > > > the Arrow GitHub issues?
> > > >
> > > > I am not sure I understand what you mean? A full copy of the
> > > > open/closed/all issues? I'd say only the (remaining) open issues
> would
> > be
> > > > fine.
> > > > For reference this is what an imported issue looks like:
> > > > https://github.com/apache/arrow/issues/30543
> > > >
> > > > Am Sa., 11. Mai 2024 um 04:09 Uhr schrieb Gang Wu <ustcwg@gmail.com
> >:
> > > >
> > > > > I can initiate the vote. But before the vote, I think we need to
> > > revisit
> > > > > the states of all unresolved tickets and close some as needed.
> > > > >
> > > > > BTW, do we really need to make a full copy of them to have a mirror
> > > > > in the Arrow GitHub issues?
> > > > >
> > > > > I'd like to seek a consensus here before sending the vote.
> > > > >
> > > > > Best,
> > > > > Gang
> > > > >
> > > > > On Sat, May 11, 2024 at 8:46 AM Jacob Wujciak <
> assignuser@apache.org
> > >
> > > > > wrote:
> > > > >
> > > > > > Hello Everyone!
> > > > > >
> > > > > > It seems there is general agreement on this topic, it would be
> > great
> > > > if a
> > > > > > committer/PMC could start a (lazy consensus) procedural vote.
> > > > > >
> > > > > > I will inquire how to handle the parquet-cpp component in jira
> > > (ideally
> > > > > > disabling it, not removing).
> > > > > > There are currently only ~70 open tickets for parquet-cpp, with
> the
> > > > > change
> > > > > > in repo it is probably easier to just move open tickets but I'll
> > > leave
> > > > > that
> > > > > > to Rok who managed the transition of Arrows 20k+ tickets too :D
> > > > > >
> > > > > > Thanks,
> > > > > > Jacob
> > > > > >
> > > > > > Arrow committer
> > > > > >
> > > > > > On 2024/04/25 05:31:18 Gang Wu wrote:
> > > > > > > I know we have some non-Java committers and PMCs. But after the
> > > > > > parquet-cpp
> > > > > > > donation, it seems that no one worked on Parquet from arrow
> (cpp,
> > > > rust,
> > > > > > go,
> > > > > > > etc.)
> > > > > > > and other projects are promoted as a Parquet committer. It
> would
> > be
> > > > > > > inconvenient
> > > > > > > for non-Java Parquet developers to work with
> > apache/parquet-format
> > > > and
> > > > > > > apache/parquet-testing repositories. Furthermore, votes from
> > these
> > > > > > > developers
> > > > > > > are not binding for a format change in the ML.
> > > > > > >
> > > > > > > Best,
> > > > > > > Gang
> > > > > > >
> > > > > > > On Wed, Apr 24, 2024 at 8:42 PM Uwe L. Korn <uw...@xhochy.com>
> > > > wrote:
> > > > > > >
> > > > > > > > > Should we consider
> > > > > > > > > Parquet developers from other projects than parquet-mr as
> > > Parquet
> > > > > > > > commuters?
> > > > > > > >
> > > > > > > > We are doing this (speaking as a Parquet PMC who didn't work
> on
> > > > > > > > parquet-mr, but parquet-cpp).
> > > > > > > >
> > > > > > > > Best
> > > > > > > > Uwe
> > > > > > > >
> > > > > > > > On Wed, Apr 24, 2024, at 2:38 PM, Gang Wu wrote:
> > > > > > > > > +1 for moving parquet-cpp issues from Apache Jira to
> Arrow's
> > > > GitHub
> > > > > > > > issue.
> > > > > > > > >
> > > > > > > > > Besides, I want to echo Will's question in the thread.
> Should
> > > we
> > > > > > consider
> > > > > > > > > Parquet developers from other projects than parquet-mr as
> > > Parquet
> > > > > > > > commiters?
> > > > > > > > > Currently apache/parquet-format and apache/parquet-testing
> > > > > > repositories
> > > > > > > > are
> > > > > > > > > solely governed by Apache Parquet PMC. It would be better
> for
> > > the
> > > > > > entire
> > > > > > > > > Parquet community if developers with sufficient
> contributions
> > > to
> > > > > open
> > > > > > > > source
> > > > > > > > > Parquet projects (including but not limited to parquet-cpp,
> > > > > arrow-rs,
> > > > > > > > cudf,
> > > > > > > > > etc.)
> > > > > > > > > can be considered as Parquet committer and PMC.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Gang
> > > > > > > > >
> > > > > > > > > On Wed, Apr 24, 2024 at 7:04 PM Uwe L. Korn <
> > uwelk@xhochy.com>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> I would be very supportive of this move. The Parquet C++
> > > > > > development has
> > > > > > > > >> been under the umbrella of the Arrow repository for more
> > than
> > > > > > five(six?)
> > > > > > > > >> years now. Thus, the issues should also be aligned with
> the
> > > > Arrow
> > > > > > > > project.
> > > > > > > > >>
> > > > > > > > >> Uwe
> > > > > > > > >>
> > > > > > > > >> On Tue, Apr 23, 2024, at 8:27 PM, Rok Mihevc wrote:
> > > > > > > > >> > Bumping this thread again to see if there is will to
> call
> > > for
> > > > a
> > > > > > vote
> > > > > > > > and
> > > > > > > > >> > move parquet-cpp issues from Apache Jira to Arrow's
> GitHub
> > > > issue
> > > > > > as
> > > > > > > > was
> > > > > > > > >> > done for Arrow.
> > > > > > > > >> > I'm willing to do the move as I already did it for
> Arrow.
> > > > > > > > >> >
> > > > > > > > >> > Rok
> > > > > > > > >> >
> > > > > > > > >> > On Sat, Apr 15, 2023 at 4:53 AM Micah Kornfield <
> > > > > > > > emkornfield@apache.org>
> > > > > > > > >> > wrote:
> > > > > > > > >> >
> > > > > > > > >> >> Bumping this thread again to see in any Parquet PMC
> > members
> > > > can
> > > > > > chime
> > > > > > > > >> >> in/maybe start a formal vote to move governance of
> > > > Parquet-CPP
> > > > > > under
> > > > > > > > the
> > > > > > > > >> >> umbrella.
> > > > > > > > >> >>
> > > > > > > > >> >> -Micah
> > > > > > > > >> >>
> > > > > > > > >> >> On 2023/02/02 10:34:25 Antoine Pitrou wrote:
> > > > > > > > >> >> >
> > > > > > > > >> >> >
> > > > > > > > >> >> > Hi Will,
> > > > > > > > >> >> >
> > > > > > > > >> >> > Le 01/02/2023 à 20:27, Will Jones a écrit :
> > > > > > > > >> >> > >
> > > > > > > > >> >> > > First, it's not obvious where issues are supposed
> to
> > be
> > > > > > open: In
> > > > > > > > >> >> Parquet
> > > > > > > > >> >> > > Jira or Arrow GitHub issues. Looking back at some
> of
> > > the
> > > > > > original
> > > > > > > > >> >> > > discussion, it looks like the intention was
> > > > > > > > >> >> > >
> > > > > > > > >> >> > > * use PARQUET-XXX for issues relating to Parquet
> core
> > > > > > > > >> >> > >> * use ARROW-XXX for issues relation to Arrow's
> > > > consumption
> > > > > > of
> > > > > > > > >> Parquet
> > > > > > > > >> >> > >> core (e.g. changes that are in parquet/arrow right
> > > now)
> > > > > > > > >> >> > >>
> > > > > > > > >> >> > > The README for the old parquet-cpp repo [3] states
> > > > instead
> > > > > in
> > > > > > > > it's
> > > > > > > > >> >> > > migration note:
> > > > > > > > >> >> > >
> > > > > > > > >> >> > > JIRA issues should continue to be opened in the
> > > PARQUET
> > > > > > JIRA
> > > > > > > > >> project.
> > > > > > > > >> >> > >
> > > > > > > > >> >> > > Either way, it doesn't seem like this process is
> > > obvious
> > > > to
> > > > > > > > people.
> > > > > > > > >> >> Perhaps
> > > > > > > > >> >> > > we could clarify this and add notices to Arrow's
> > GitHub
> > > > > > issues
> > > > > > > > >> >> template?
> > > > > > > > >> >> >
> > > > > > > > >> >> > I agree we should clarify this. I have no personal
> > > > > preference,
> > > > > > but
> > > > > > > > I
> > > > > > > > >> >> will note
> > > > > > > > >> >> > that Github issues decrease friction as having a GH
> > > account
> > > > > is
> > > > > > > > already
> > > > > > > > >> >> necessary
> > > > > > > > >> >> > for submitting PRs.
> > > > > > > > >> >> >
> > > > > > > > >> >> > > Second, committer status is a little unclear. I am
> a
> > > > > > committer on
> > > > > > > > >> >> Arrow,
> > > > > > > > >> >> > > but not on Parquet right now. Does that mean I
> should
> > > > only
> > > > > > merge
> > > > > > > > >> >> Parquet
> > > > > > > > >> >> > > C++ PRs for code changes in parquet/arrow? Or that
> I
> > > > > > shouldn't
> > > > > > > > merge
> > > > > > > > >> >> > > Parquet changes at all?
> > > > > > > > >> >> >
> > > > > > > > >> >> > Since Parquet C++ is part of Arrow C++, you are
> allowed
> > > to
> > > > > > merge
> > > > > > > > >> Parquet
> > > > > > > > >> >> C++
> > > > > > > > >> >> > changes. As always you should ensure you have
> > sufficient
> > > > > > > > understanding
> > > > > > > > >> >> of the
> > > > > > > > >> >> > contribution, and that it follows established
> > practices:
> > > > > > > > >> >> >
> > > > https://arrow.apache.org/docs/dev/developers/reviewing.html
> > > > > > > > >> >> >
> > > > > > > > >> >> > > Also, are the contributions to Arrow C++ Parquet
> > being
> > > > > > actively
> > > > > > > > >> >> reviewed
> > > > > > > > >> >> > > for potential new committers?
> > > > > > > > >> >> >
> > > > > > > > >> >> > I would certainly do.
> > > > > > > > >> >> >
> > > > > > > > >> >> > Regards
> > > > > > > > >> >> >
> > > > > > > > >> >> > Antoine.
> > > > > > > > >> >> >
> > > > > > > > >> >> >
> > > > > > > > >> >>
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Fwd: [C++] Parquet and Arrow overlap
Posted by Julien Le Dem <ju...@apache.org>.
It is great to see more momentum building.
I have myself a little bit more time to contribute to Parquet.
Personally I think moving it back would make sense.
*However* I have personally contributed a lot more to the Java than the C++
code base.
That move was done initially because people contributing to the Arrow and
Parquet C++ code bases were the same ones and circular dependencies were
getting in the way (does Parquet depend on Arrow or the other way around?
At the time it was both ways.). So to make this happen, we need enough
Parquet C++ contributors that would be happy with the move and clarify
which way the dependency goes. My take is that Parquet depends on Arrow but
I'd be curious to see what others think.
Julien
On Sat, May 11, 2024 at 2:51 AM Andrew Lamb <an...@gmail.com> wrote:
> It is great to see some additional enthusiasm and momentum around the
> Apache Parquet implementation (congratulations on the release of parquet-mr
> 1.14[1]!).
>
> As activity picks up, if the desire is to build more community around
> Parquet, perhaps the Parquet PMC wants to encourage moving code back to
> repositories managed by parquet (and out of arrow, for example). I realize
> this would be a technical burden, but it might clarify communities and
> committers.
>
> Andrew
>
> [1]: https://lists.apache.org/thread/2gggm938z0x9fx3wtwctfm5htsxlf3z4
>
>
>
> On Fri, May 10, 2024 at 11:45 PM Matt Topol <zo...@gmail.com>
> wrote:
>
> > I just wanted to also poke the question of non-Java developers who have
> > worked on the other parquet implementations potentially being recognized
> as
> > committers or otherwise on the Parquet project (speaking as the primary
> > developer of the Go parquet implementation which also lives in the Arrow
> > repository). It would be great to see some active contributors to
> > parquet-cpp, parquet-go, or otherwise not just being considered but
> > actively becoming committers.
> >
> > That's just my two cents from a community perspective.
> >
> > --Matt
> >
> > On Fri, May 10, 2024, 10:35 PM Jacob Wujciak <as...@apache.org>
> > wrote:
> >
> > > Thank you, that sounds great! On first glance some seem to be rather
> old
> > > and probably don't apply anymore.
> > >
> > > > BTW, do we really need to make a full copy of them to have a mirror
> in
> > > the Arrow GitHub issues?
> > >
> > > I am not sure I understand what you mean? A full copy of the
> > > open/closed/all issues? I'd say only the (remaining) open issues would
> be
> > > fine.
> > > For reference this is what an imported issue looks like:
> > > https://github.com/apache/arrow/issues/30543
> > >
> > > Am Sa., 11. Mai 2024 um 04:09 Uhr schrieb Gang Wu <us...@gmail.com>:
> > >
> > > > I can initiate the vote. But before the vote, I think we need to
> > revisit
> > > > the states of all unresolved tickets and close some as needed.
> > > >
> > > > BTW, do we really need to make a full copy of them to have a mirror
> > > > in the Arrow GitHub issues?
> > > >
> > > > I'd like to seek a consensus here before sending the vote.
> > > >
> > > > Best,
> > > > Gang
> > > >
> > > > On Sat, May 11, 2024 at 8:46 AM Jacob Wujciak <assignuser@apache.org
> >
> > > > wrote:
> > > >
> > > > > Hello Everyone!
> > > > >
> > > > > It seems there is general agreement on this topic, it would be
> great
> > > if a
> > > > > committer/PMC could start a (lazy consensus) procedural vote.
> > > > >
> > > > > I will inquire how to handle the parquet-cpp component in jira
> > (ideally
> > > > > disabling it, not removing).
> > > > > There are currently only ~70 open tickets for parquet-cpp, with the
> > > > change
> > > > > in repo it is probably easier to just move open tickets but I'll
> > leave
> > > > that
> > > > > to Rok who managed the transition of Arrows 20k+ tickets too :D
> > > > >
> > > > > Thanks,
> > > > > Jacob
> > > > >
> > > > > Arrow committer
> > > > >
> > > > > On 2024/04/25 05:31:18 Gang Wu wrote:
> > > > > > I know we have some non-Java committers and PMCs. But after the
> > > > > parquet-cpp
> > > > > > donation, it seems that no one worked on Parquet from arrow (cpp,
> > > rust,
> > > > > go,
> > > > > > etc.)
> > > > > > and other projects are promoted as a Parquet committer. It would
> be
> > > > > > inconvenient
> > > > > > for non-Java Parquet developers to work with
> apache/parquet-format
> > > and
> > > > > > apache/parquet-testing repositories. Furthermore, votes from
> these
> > > > > > developers
> > > > > > are not binding for a format change in the ML.
> > > > > >
> > > > > > Best,
> > > > > > Gang
> > > > > >
> > > > > > On Wed, Apr 24, 2024 at 8:42 PM Uwe L. Korn <uw...@xhochy.com>
> > > wrote:
> > > > > >
> > > > > > > > Should we consider
> > > > > > > > Parquet developers from other projects than parquet-mr as
> > Parquet
> > > > > > > commuters?
> > > > > > >
> > > > > > > We are doing this (speaking as a Parquet PMC who didn't work on
> > > > > > > parquet-mr, but parquet-cpp).
> > > > > > >
> > > > > > > Best
> > > > > > > Uwe
> > > > > > >
> > > > > > > On Wed, Apr 24, 2024, at 2:38 PM, Gang Wu wrote:
> > > > > > > > +1 for moving parquet-cpp issues from Apache Jira to Arrow's
> > > GitHub
> > > > > > > issue.
> > > > > > > >
> > > > > > > > Besides, I want to echo Will's question in the thread. Should
> > we
> > > > > consider
> > > > > > > > Parquet developers from other projects than parquet-mr as
> > Parquet
> > > > > > > commiters?
> > > > > > > > Currently apache/parquet-format and apache/parquet-testing
> > > > > repositories
> > > > > > > are
> > > > > > > > solely governed by Apache Parquet PMC. It would be better for
> > the
> > > > > entire
> > > > > > > > Parquet community if developers with sufficient contributions
> > to
> > > > open
> > > > > > > source
> > > > > > > > Parquet projects (including but not limited to parquet-cpp,
> > > > arrow-rs,
> > > > > > > cudf,
> > > > > > > > etc.)
> > > > > > > > can be considered as Parquet committer and PMC.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Gang
> > > > > > > >
> > > > > > > > On Wed, Apr 24, 2024 at 7:04 PM Uwe L. Korn <
> uwelk@xhochy.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > >> I would be very supportive of this move. The Parquet C++
> > > > > development has
> > > > > > > >> been under the umbrella of the Arrow repository for more
> than
> > > > > five(six?)
> > > > > > > >> years now. Thus, the issues should also be aligned with the
> > > Arrow
> > > > > > > project.
> > > > > > > >>
> > > > > > > >> Uwe
> > > > > > > >>
> > > > > > > >> On Tue, Apr 23, 2024, at 8:27 PM, Rok Mihevc wrote:
> > > > > > > >> > Bumping this thread again to see if there is will to call
> > for
> > > a
> > > > > vote
> > > > > > > and
> > > > > > > >> > move parquet-cpp issues from Apache Jira to Arrow's GitHub
> > > issue
> > > > > as
> > > > > > > was
> > > > > > > >> > done for Arrow.
> > > > > > > >> > I'm willing to do the move as I already did it for Arrow.
> > > > > > > >> >
> > > > > > > >> > Rok
> > > > > > > >> >
> > > > > > > >> > On Sat, Apr 15, 2023 at 4:53 AM Micah Kornfield <
> > > > > > > emkornfield@apache.org>
> > > > > > > >> > wrote:
> > > > > > > >> >
> > > > > > > >> >> Bumping this thread again to see in any Parquet PMC
> members
> > > can
> > > > > chime
> > > > > > > >> >> in/maybe start a formal vote to move governance of
> > > Parquet-CPP
> > > > > under
> > > > > > > the
> > > > > > > >> >> umbrella.
> > > > > > > >> >>
> > > > > > > >> >> -Micah
> > > > > > > >> >>
> > > > > > > >> >> On 2023/02/02 10:34:25 Antoine Pitrou wrote:
> > > > > > > >> >> >
> > > > > > > >> >> >
> > > > > > > >> >> > Hi Will,
> > > > > > > >> >> >
> > > > > > > >> >> > Le 01/02/2023 à 20:27, Will Jones a écrit :
> > > > > > > >> >> > >
> > > > > > > >> >> > > First, it's not obvious where issues are supposed to
> be
> > > > > open: In
> > > > > > > >> >> Parquet
> > > > > > > >> >> > > Jira or Arrow GitHub issues. Looking back at some of
> > the
> > > > > original
> > > > > > > >> >> > > discussion, it looks like the intention was
> > > > > > > >> >> > >
> > > > > > > >> >> > > * use PARQUET-XXX for issues relating to Parquet core
> > > > > > > >> >> > >> * use ARROW-XXX for issues relation to Arrow's
> > > consumption
> > > > > of
> > > > > > > >> Parquet
> > > > > > > >> >> > >> core (e.g. changes that are in parquet/arrow right
> > now)
> > > > > > > >> >> > >>
> > > > > > > >> >> > > The README for the old parquet-cpp repo [3] states
> > > instead
> > > > in
> > > > > > > it's
> > > > > > > >> >> > > migration note:
> > > > > > > >> >> > >
> > > > > > > >> >> > > JIRA issues should continue to be opened in the
> > PARQUET
> > > > > JIRA
> > > > > > > >> project.
> > > > > > > >> >> > >
> > > > > > > >> >> > > Either way, it doesn't seem like this process is
> > obvious
> > > to
> > > > > > > people.
> > > > > > > >> >> Perhaps
> > > > > > > >> >> > > we could clarify this and add notices to Arrow's
> GitHub
> > > > > issues
> > > > > > > >> >> template?
> > > > > > > >> >> >
> > > > > > > >> >> > I agree we should clarify this. I have no personal
> > > > preference,
> > > > > but
> > > > > > > I
> > > > > > > >> >> will note
> > > > > > > >> >> > that Github issues decrease friction as having a GH
> > account
> > > > is
> > > > > > > already
> > > > > > > >> >> necessary
> > > > > > > >> >> > for submitting PRs.
> > > > > > > >> >> >
> > > > > > > >> >> > > Second, committer status is a little unclear. I am a
> > > > > committer on
> > > > > > > >> >> Arrow,
> > > > > > > >> >> > > but not on Parquet right now. Does that mean I should
> > > only
> > > > > merge
> > > > > > > >> >> Parquet
> > > > > > > >> >> > > C++ PRs for code changes in parquet/arrow? Or that I
> > > > > shouldn't
> > > > > > > merge
> > > > > > > >> >> > > Parquet changes at all?
> > > > > > > >> >> >
> > > > > > > >> >> > Since Parquet C++ is part of Arrow C++, you are allowed
> > to
> > > > > merge
> > > > > > > >> Parquet
> > > > > > > >> >> C++
> > > > > > > >> >> > changes. As always you should ensure you have
> sufficient
> > > > > > > understanding
> > > > > > > >> >> of the
> > > > > > > >> >> > contribution, and that it follows established
> practices:
> > > > > > > >> >> >
> > > https://arrow.apache.org/docs/dev/developers/reviewing.html
> > > > > > > >> >> >
> > > > > > > >> >> > > Also, are the contributions to Arrow C++ Parquet
> being
> > > > > actively
> > > > > > > >> >> reviewed
> > > > > > > >> >> > > for potential new committers?
> > > > > > > >> >> >
> > > > > > > >> >> > I would certainly do.
> > > > > > > >> >> >
> > > > > > > >> >> > Regards
> > > > > > > >> >> >
> > > > > > > >> >> > Antoine.
> > > > > > > >> >> >
> > > > > > > >> >> >
> > > > > > > >> >>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Fwd: [C++] Parquet and Arrow overlap
Posted by Andrew Lamb <an...@gmail.com>.
It is great to see some additional enthusiasm and momentum around the
Apache Parquet implementation (congratulations on the release of parquet-mr
1.14[1]!).
As activity picks up, if the desire is to build more community around
Parquet, perhaps the Parquet PMC wants to encourage moving code back to
repositories managed by parquet (and out of arrow, for example). I realize
this would be a technical burden, but it might clarify communities and
committers.
Andrew
[1]: https://lists.apache.org/thread/2gggm938z0x9fx3wtwctfm5htsxlf3z4
On Fri, May 10, 2024 at 11:45 PM Matt Topol <zo...@gmail.com> wrote:
> I just wanted to also poke the question of non-Java developers who have
> worked on the other parquet implementations potentially being recognized as
> committers or otherwise on the Parquet project (speaking as the primary
> developer of the Go parquet implementation which also lives in the Arrow
> repository). It would be great to see some active contributors to
> parquet-cpp, parquet-go, or otherwise not just being considered but
> actively becoming committers.
>
> That's just my two cents from a community perspective.
>
> --Matt
>
> On Fri, May 10, 2024, 10:35 PM Jacob Wujciak <as...@apache.org>
> wrote:
>
> > Thank you, that sounds great! On first glance some seem to be rather old
> > and probably don't apply anymore.
> >
> > > BTW, do we really need to make a full copy of them to have a mirror in
> > the Arrow GitHub issues?
> >
> > I am not sure I understand what you mean? A full copy of the
> > open/closed/all issues? I'd say only the (remaining) open issues would be
> > fine.
> > For reference this is what an imported issue looks like:
> > https://github.com/apache/arrow/issues/30543
> >
> > Am Sa., 11. Mai 2024 um 04:09 Uhr schrieb Gang Wu <us...@gmail.com>:
> >
> > > I can initiate the vote. But before the vote, I think we need to
> revisit
> > > the states of all unresolved tickets and close some as needed.
> > >
> > > BTW, do we really need to make a full copy of them to have a mirror
> > > in the Arrow GitHub issues?
> > >
> > > I'd like to seek a consensus here before sending the vote.
> > >
> > > Best,
> > > Gang
> > >
> > > On Sat, May 11, 2024 at 8:46 AM Jacob Wujciak <as...@apache.org>
> > > wrote:
> > >
> > > > Hello Everyone!
> > > >
> > > > It seems there is general agreement on this topic, it would be great
> > if a
> > > > committer/PMC could start a (lazy consensus) procedural vote.
> > > >
> > > > I will inquire how to handle the parquet-cpp component in jira
> (ideally
> > > > disabling it, not removing).
> > > > There are currently only ~70 open tickets for parquet-cpp, with the
> > > change
> > > > in repo it is probably easier to just move open tickets but I'll
> leave
> > > that
> > > > to Rok who managed the transition of Arrows 20k+ tickets too :D
> > > >
> > > > Thanks,
> > > > Jacob
> > > >
> > > > Arrow committer
> > > >
> > > > On 2024/04/25 05:31:18 Gang Wu wrote:
> > > > > I know we have some non-Java committers and PMCs. But after the
> > > > parquet-cpp
> > > > > donation, it seems that no one worked on Parquet from arrow (cpp,
> > rust,
> > > > go,
> > > > > etc.)
> > > > > and other projects are promoted as a Parquet committer. It would be
> > > > > inconvenient
> > > > > for non-Java Parquet developers to work with apache/parquet-format
> > and
> > > > > apache/parquet-testing repositories. Furthermore, votes from these
> > > > > developers
> > > > > are not binding for a format change in the ML.
> > > > >
> > > > > Best,
> > > > > Gang
> > > > >
> > > > > On Wed, Apr 24, 2024 at 8:42 PM Uwe L. Korn <uw...@xhochy.com>
> > wrote:
> > > > >
> > > > > > > Should we consider
> > > > > > > Parquet developers from other projects than parquet-mr as
> Parquet
> > > > > > commuters?
> > > > > >
> > > > > > We are doing this (speaking as a Parquet PMC who didn't work on
> > > > > > parquet-mr, but parquet-cpp).
> > > > > >
> > > > > > Best
> > > > > > Uwe
> > > > > >
> > > > > > On Wed, Apr 24, 2024, at 2:38 PM, Gang Wu wrote:
> > > > > > > +1 for moving parquet-cpp issues from Apache Jira to Arrow's
> > GitHub
> > > > > > issue.
> > > > > > >
> > > > > > > Besides, I want to echo Will's question in the thread. Should
> we
> > > > consider
> > > > > > > Parquet developers from other projects than parquet-mr as
> Parquet
> > > > > > commiters?
> > > > > > > Currently apache/parquet-format and apache/parquet-testing
> > > > repositories
> > > > > > are
> > > > > > > solely governed by Apache Parquet PMC. It would be better for
> the
> > > > entire
> > > > > > > Parquet community if developers with sufficient contributions
> to
> > > open
> > > > > > source
> > > > > > > Parquet projects (including but not limited to parquet-cpp,
> > > arrow-rs,
> > > > > > cudf,
> > > > > > > etc.)
> > > > > > > can be considered as Parquet committer and PMC.
> > > > > > >
> > > > > > > Best,
> > > > > > > Gang
> > > > > > >
> > > > > > > On Wed, Apr 24, 2024 at 7:04 PM Uwe L. Korn <uw...@xhochy.com>
> > > > wrote:
> > > > > > >
> > > > > > >> I would be very supportive of this move. The Parquet C++
> > > > development has
> > > > > > >> been under the umbrella of the Arrow repository for more than
> > > > five(six?)
> > > > > > >> years now. Thus, the issues should also be aligned with the
> > Arrow
> > > > > > project.
> > > > > > >>
> > > > > > >> Uwe
> > > > > > >>
> > > > > > >> On Tue, Apr 23, 2024, at 8:27 PM, Rok Mihevc wrote:
> > > > > > >> > Bumping this thread again to see if there is will to call
> for
> > a
> > > > vote
> > > > > > and
> > > > > > >> > move parquet-cpp issues from Apache Jira to Arrow's GitHub
> > issue
> > > > as
> > > > > > was
> > > > > > >> > done for Arrow.
> > > > > > >> > I'm willing to do the move as I already did it for Arrow.
> > > > > > >> >
> > > > > > >> > Rok
> > > > > > >> >
> > > > > > >> > On Sat, Apr 15, 2023 at 4:53 AM Micah Kornfield <
> > > > > > emkornfield@apache.org>
> > > > > > >> > wrote:
> > > > > > >> >
> > > > > > >> >> Bumping this thread again to see in any Parquet PMC members
> > can
> > > > chime
> > > > > > >> >> in/maybe start a formal vote to move governance of
> > Parquet-CPP
> > > > under
> > > > > > the
> > > > > > >> >> umbrella.
> > > > > > >> >>
> > > > > > >> >> -Micah
> > > > > > >> >>
> > > > > > >> >> On 2023/02/02 10:34:25 Antoine Pitrou wrote:
> > > > > > >> >> >
> > > > > > >> >> >
> > > > > > >> >> > Hi Will,
> > > > > > >> >> >
> > > > > > >> >> > Le 01/02/2023 à 20:27, Will Jones a écrit :
> > > > > > >> >> > >
> > > > > > >> >> > > First, it's not obvious where issues are supposed to be
> > > > open: In
> > > > > > >> >> Parquet
> > > > > > >> >> > > Jira or Arrow GitHub issues. Looking back at some of
> the
> > > > original
> > > > > > >> >> > > discussion, it looks like the intention was
> > > > > > >> >> > >
> > > > > > >> >> > > * use PARQUET-XXX for issues relating to Parquet core
> > > > > > >> >> > >> * use ARROW-XXX for issues relation to Arrow's
> > consumption
> > > > of
> > > > > > >> Parquet
> > > > > > >> >> > >> core (e.g. changes that are in parquet/arrow right
> now)
> > > > > > >> >> > >>
> > > > > > >> >> > > The README for the old parquet-cpp repo [3] states
> > instead
> > > in
> > > > > > it's
> > > > > > >> >> > > migration note:
> > > > > > >> >> > >
> > > > > > >> >> > > JIRA issues should continue to be opened in the
> PARQUET
> > > > JIRA
> > > > > > >> project.
> > > > > > >> >> > >
> > > > > > >> >> > > Either way, it doesn't seem like this process is
> obvious
> > to
> > > > > > people.
> > > > > > >> >> Perhaps
> > > > > > >> >> > > we could clarify this and add notices to Arrow's GitHub
> > > > issues
> > > > > > >> >> template?
> > > > > > >> >> >
> > > > > > >> >> > I agree we should clarify this. I have no personal
> > > preference,
> > > > but
> > > > > > I
> > > > > > >> >> will note
> > > > > > >> >> > that Github issues decrease friction as having a GH
> account
> > > is
> > > > > > already
> > > > > > >> >> necessary
> > > > > > >> >> > for submitting PRs.
> > > > > > >> >> >
> > > > > > >> >> > > Second, committer status is a little unclear. I am a
> > > > committer on
> > > > > > >> >> Arrow,
> > > > > > >> >> > > but not on Parquet right now. Does that mean I should
> > only
> > > > merge
> > > > > > >> >> Parquet
> > > > > > >> >> > > C++ PRs for code changes in parquet/arrow? Or that I
> > > > shouldn't
> > > > > > merge
> > > > > > >> >> > > Parquet changes at all?
> > > > > > >> >> >
> > > > > > >> >> > Since Parquet C++ is part of Arrow C++, you are allowed
> to
> > > > merge
> > > > > > >> Parquet
> > > > > > >> >> C++
> > > > > > >> >> > changes. As always you should ensure you have sufficient
> > > > > > understanding
> > > > > > >> >> of the
> > > > > > >> >> > contribution, and that it follows established practices:
> > > > > > >> >> >
> > https://arrow.apache.org/docs/dev/developers/reviewing.html
> > > > > > >> >> >
> > > > > > >> >> > > Also, are the contributions to Arrow C++ Parquet being
> > > > actively
> > > > > > >> >> reviewed
> > > > > > >> >> > > for potential new committers?
> > > > > > >> >> >
> > > > > > >> >> > I would certainly do.
> > > > > > >> >> >
> > > > > > >> >> > Regards
> > > > > > >> >> >
> > > > > > >> >> > Antoine.
> > > > > > >> >> >
> > > > > > >> >> >
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Fwd: [C++] Parquet and Arrow overlap
Posted by Andrew Lamb <an...@gmail.com>.
It is great to see some additional enthusiasm and momentum around the
Apache Parquet implementation (congratulations on the release of parquet-mr
1.14[1]!).
As activity picks up, if the desire is to build more community around
Parquet, perhaps the Parquet PMC wants to encourage moving code back to
repositories managed by parquet (and out of arrow, for example). I realize
this would be a technical burden, but it might clarify communities and
committers.
Andrew
[1]: https://lists.apache.org/thread/2gggm938z0x9fx3wtwctfm5htsxlf3z4
On Fri, May 10, 2024 at 11:45 PM Matt Topol <zo...@gmail.com> wrote:
> I just wanted to also poke the question of non-Java developers who have
> worked on the other parquet implementations potentially being recognized as
> committers or otherwise on the Parquet project (speaking as the primary
> developer of the Go parquet implementation which also lives in the Arrow
> repository). It would be great to see some active contributors to
> parquet-cpp, parquet-go, or otherwise not just being considered but
> actively becoming committers.
>
> That's just my two cents from a community perspective.
>
> --Matt
>
> On Fri, May 10, 2024, 10:35 PM Jacob Wujciak <as...@apache.org>
> wrote:
>
> > Thank you, that sounds great! On first glance some seem to be rather old
> > and probably don't apply anymore.
> >
> > > BTW, do we really need to make a full copy of them to have a mirror in
> > the Arrow GitHub issues?
> >
> > I am not sure I understand what you mean? A full copy of the
> > open/closed/all issues? I'd say only the (remaining) open issues would be
> > fine.
> > For reference this is what an imported issue looks like:
> > https://github.com/apache/arrow/issues/30543
> >
> > Am Sa., 11. Mai 2024 um 04:09 Uhr schrieb Gang Wu <us...@gmail.com>:
> >
> > > I can initiate the vote. But before the vote, I think we need to
> revisit
> > > the states of all unresolved tickets and close some as needed.
> > >
> > > BTW, do we really need to make a full copy of them to have a mirror
> > > in the Arrow GitHub issues?
> > >
> > > I'd like to seek a consensus here before sending the vote.
> > >
> > > Best,
> > > Gang
> > >
> > > On Sat, May 11, 2024 at 8:46 AM Jacob Wujciak <as...@apache.org>
> > > wrote:
> > >
> > > > Hello Everyone!
> > > >
> > > > It seems there is general agreement on this topic, it would be great
> > if a
> > > > committer/PMC could start a (lazy consensus) procedural vote.
> > > >
> > > > I will inquire how to handle the parquet-cpp component in jira
> (ideally
> > > > disabling it, not removing).
> > > > There are currently only ~70 open tickets for parquet-cpp, with the
> > > change
> > > > in repo it is probably easier to just move open tickets but I'll
> leave
> > > that
> > > > to Rok who managed the transition of Arrows 20k+ tickets too :D
> > > >
> > > > Thanks,
> > > > Jacob
> > > >
> > > > Arrow committer
> > > >
> > > > On 2024/04/25 05:31:18 Gang Wu wrote:
> > > > > I know we have some non-Java committers and PMCs. But after the
> > > > parquet-cpp
> > > > > donation, it seems that no one worked on Parquet from arrow (cpp,
> > rust,
> > > > go,
> > > > > etc.)
> > > > > and other projects are promoted as a Parquet committer. It would be
> > > > > inconvenient
> > > > > for non-Java Parquet developers to work with apache/parquet-format
> > and
> > > > > apache/parquet-testing repositories. Furthermore, votes from these
> > > > > developers
> > > > > are not binding for a format change in the ML.
> > > > >
> > > > > Best,
> > > > > Gang
> > > > >
> > > > > On Wed, Apr 24, 2024 at 8:42 PM Uwe L. Korn <uw...@xhochy.com>
> > wrote:
> > > > >
> > > > > > > Should we consider
> > > > > > > Parquet developers from other projects than parquet-mr as
> Parquet
> > > > > > commuters?
> > > > > >
> > > > > > We are doing this (speaking as a Parquet PMC who didn't work on
> > > > > > parquet-mr, but parquet-cpp).
> > > > > >
> > > > > > Best
> > > > > > Uwe
> > > > > >
> > > > > > On Wed, Apr 24, 2024, at 2:38 PM, Gang Wu wrote:
> > > > > > > +1 for moving parquet-cpp issues from Apache Jira to Arrow's
> > GitHub
> > > > > > issue.
> > > > > > >
> > > > > > > Besides, I want to echo Will's question in the thread. Should
> we
> > > > consider
> > > > > > > Parquet developers from other projects than parquet-mr as
> Parquet
> > > > > > commiters?
> > > > > > > Currently apache/parquet-format and apache/parquet-testing
> > > > repositories
> > > > > > are
> > > > > > > solely governed by Apache Parquet PMC. It would be better for
> the
> > > > entire
> > > > > > > Parquet community if developers with sufficient contributions
> to
> > > open
> > > > > > source
> > > > > > > Parquet projects (including but not limited to parquet-cpp,
> > > arrow-rs,
> > > > > > cudf,
> > > > > > > etc.)
> > > > > > > can be considered as Parquet committer and PMC.
> > > > > > >
> > > > > > > Best,
> > > > > > > Gang
> > > > > > >
> > > > > > > On Wed, Apr 24, 2024 at 7:04 PM Uwe L. Korn <uw...@xhochy.com>
> > > > wrote:
> > > > > > >
> > > > > > >> I would be very supportive of this move. The Parquet C++
> > > > development has
> > > > > > >> been under the umbrella of the Arrow repository for more than
> > > > five(six?)
> > > > > > >> years now. Thus, the issues should also be aligned with the
> > Arrow
> > > > > > project.
> > > > > > >>
> > > > > > >> Uwe
> > > > > > >>
> > > > > > >> On Tue, Apr 23, 2024, at 8:27 PM, Rok Mihevc wrote:
> > > > > > >> > Bumping this thread again to see if there is will to call
> for
> > a
> > > > vote
> > > > > > and
> > > > > > >> > move parquet-cpp issues from Apache Jira to Arrow's GitHub
> > issue
> > > > as
> > > > > > was
> > > > > > >> > done for Arrow.
> > > > > > >> > I'm willing to do the move as I already did it for Arrow.
> > > > > > >> >
> > > > > > >> > Rok
> > > > > > >> >
> > > > > > >> > On Sat, Apr 15, 2023 at 4:53 AM Micah Kornfield <
> > > > > > emkornfield@apache.org>
> > > > > > >> > wrote:
> > > > > > >> >
> > > > > > >> >> Bumping this thread again to see in any Parquet PMC members
> > can
> > > > chime
> > > > > > >> >> in/maybe start a formal vote to move governance of
> > Parquet-CPP
> > > > under
> > > > > > the
> > > > > > >> >> umbrella.
> > > > > > >> >>
> > > > > > >> >> -Micah
> > > > > > >> >>
> > > > > > >> >> On 2023/02/02 10:34:25 Antoine Pitrou wrote:
> > > > > > >> >> >
> > > > > > >> >> >
> > > > > > >> >> > Hi Will,
> > > > > > >> >> >
> > > > > > >> >> > Le 01/02/2023 à 20:27, Will Jones a écrit :
> > > > > > >> >> > >
> > > > > > >> >> > > First, it's not obvious where issues are supposed to be
> > > > open: In
> > > > > > >> >> Parquet
> > > > > > >> >> > > Jira or Arrow GitHub issues. Looking back at some of
> the
> > > > original
> > > > > > >> >> > > discussion, it looks like the intention was
> > > > > > >> >> > >
> > > > > > >> >> > > * use PARQUET-XXX for issues relating to Parquet core
> > > > > > >> >> > >> * use ARROW-XXX for issues relation to Arrow's
> > consumption
> > > > of
> > > > > > >> Parquet
> > > > > > >> >> > >> core (e.g. changes that are in parquet/arrow right
> now)
> > > > > > >> >> > >>
> > > > > > >> >> > > The README for the old parquet-cpp repo [3] states
> > instead
> > > in
> > > > > > it's
> > > > > > >> >> > > migration note:
> > > > > > >> >> > >
> > > > > > >> >> > > JIRA issues should continue to be opened in the
> PARQUET
> > > > JIRA
> > > > > > >> project.
> > > > > > >> >> > >
> > > > > > >> >> > > Either way, it doesn't seem like this process is
> obvious
> > to
> > > > > > people.
> > > > > > >> >> Perhaps
> > > > > > >> >> > > we could clarify this and add notices to Arrow's GitHub
> > > > issues
> > > > > > >> >> template?
> > > > > > >> >> >
> > > > > > >> >> > I agree we should clarify this. I have no personal
> > > preference,
> > > > but
> > > > > > I
> > > > > > >> >> will note
> > > > > > >> >> > that Github issues decrease friction as having a GH
> account
> > > is
> > > > > > already
> > > > > > >> >> necessary
> > > > > > >> >> > for submitting PRs.
> > > > > > >> >> >
> > > > > > >> >> > > Second, committer status is a little unclear. I am a
> > > > committer on
> > > > > > >> >> Arrow,
> > > > > > >> >> > > but not on Parquet right now. Does that mean I should
> > only
> > > > merge
> > > > > > >> >> Parquet
> > > > > > >> >> > > C++ PRs for code changes in parquet/arrow? Or that I
> > > > shouldn't
> > > > > > merge
> > > > > > >> >> > > Parquet changes at all?
> > > > > > >> >> >
> > > > > > >> >> > Since Parquet C++ is part of Arrow C++, you are allowed
> to
> > > > merge
> > > > > > >> Parquet
> > > > > > >> >> C++
> > > > > > >> >> > changes. As always you should ensure you have sufficient
> > > > > > understanding
> > > > > > >> >> of the
> > > > > > >> >> > contribution, and that it follows established practices:
> > > > > > >> >> >
> > https://arrow.apache.org/docs/dev/developers/reviewing.html
> > > > > > >> >> >
> > > > > > >> >> > > Also, are the contributions to Arrow C++ Parquet being
> > > > actively
> > > > > > >> >> reviewed
> > > > > > >> >> > > for potential new committers?
> > > > > > >> >> >
> > > > > > >> >> > I would certainly do.
> > > > > > >> >> >
> > > > > > >> >> > Regards
> > > > > > >> >> >
> > > > > > >> >> > Antoine.
> > > > > > >> >> >
> > > > > > >> >> >
> > > > > > >> >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Fwd: [C++] Parquet and Arrow overlap
Posted by Matt Topol <zo...@gmail.com>.
I just wanted to also poke the question of non-Java developers who have
worked on the other parquet implementations potentially being recognized as
committers or otherwise on the Parquet project (speaking as the primary
developer of the Go parquet implementation which also lives in the Arrow
repository). It would be great to see some active contributors to
parquet-cpp, parquet-go, or otherwise not just being considered but
actively becoming committers.
That's just my two cents from a community perspective.
--Matt
On Fri, May 10, 2024, 10:35 PM Jacob Wujciak <as...@apache.org> wrote:
> Thank you, that sounds great! On first glance some seem to be rather old
> and probably don't apply anymore.
>
> > BTW, do we really need to make a full copy of them to have a mirror in
> the Arrow GitHub issues?
>
> I am not sure I understand what you mean? A full copy of the
> open/closed/all issues? I'd say only the (remaining) open issues would be
> fine.
> For reference this is what an imported issue looks like:
> https://github.com/apache/arrow/issues/30543
>
> Am Sa., 11. Mai 2024 um 04:09 Uhr schrieb Gang Wu <us...@gmail.com>:
>
> > I can initiate the vote. But before the vote, I think we need to revisit
> > the states of all unresolved tickets and close some as needed.
> >
> > BTW, do we really need to make a full copy of them to have a mirror
> > in the Arrow GitHub issues?
> >
> > I'd like to seek a consensus here before sending the vote.
> >
> > Best,
> > Gang
> >
> > On Sat, May 11, 2024 at 8:46 AM Jacob Wujciak <as...@apache.org>
> > wrote:
> >
> > > Hello Everyone!
> > >
> > > It seems there is general agreement on this topic, it would be great
> if a
> > > committer/PMC could start a (lazy consensus) procedural vote.
> > >
> > > I will inquire how to handle the parquet-cpp component in jira (ideally
> > > disabling it, not removing).
> > > There are currently only ~70 open tickets for parquet-cpp, with the
> > change
> > > in repo it is probably easier to just move open tickets but I'll leave
> > that
> > > to Rok who managed the transition of Arrows 20k+ tickets too :D
> > >
> > > Thanks,
> > > Jacob
> > >
> > > Arrow committer
> > >
> > > On 2024/04/25 05:31:18 Gang Wu wrote:
> > > > I know we have some non-Java committers and PMCs. But after the
> > > parquet-cpp
> > > > donation, it seems that no one worked on Parquet from arrow (cpp,
> rust,
> > > go,
> > > > etc.)
> > > > and other projects are promoted as a Parquet committer. It would be
> > > > inconvenient
> > > > for non-Java Parquet developers to work with apache/parquet-format
> and
> > > > apache/parquet-testing repositories. Furthermore, votes from these
> > > > developers
> > > > are not binding for a format change in the ML.
> > > >
> > > > Best,
> > > > Gang
> > > >
> > > > On Wed, Apr 24, 2024 at 8:42 PM Uwe L. Korn <uw...@xhochy.com>
> wrote:
> > > >
> > > > > > Should we consider
> > > > > > Parquet developers from other projects than parquet-mr as Parquet
> > > > > commuters?
> > > > >
> > > > > We are doing this (speaking as a Parquet PMC who didn't work on
> > > > > parquet-mr, but parquet-cpp).
> > > > >
> > > > > Best
> > > > > Uwe
> > > > >
> > > > > On Wed, Apr 24, 2024, at 2:38 PM, Gang Wu wrote:
> > > > > > +1 for moving parquet-cpp issues from Apache Jira to Arrow's
> GitHub
> > > > > issue.
> > > > > >
> > > > > > Besides, I want to echo Will's question in the thread. Should we
> > > consider
> > > > > > Parquet developers from other projects than parquet-mr as Parquet
> > > > > commiters?
> > > > > > Currently apache/parquet-format and apache/parquet-testing
> > > repositories
> > > > > are
> > > > > > solely governed by Apache Parquet PMC. It would be better for the
> > > entire
> > > > > > Parquet community if developers with sufficient contributions to
> > open
> > > > > source
> > > > > > Parquet projects (including but not limited to parquet-cpp,
> > arrow-rs,
> > > > > cudf,
> > > > > > etc.)
> > > > > > can be considered as Parquet committer and PMC.
> > > > > >
> > > > > > Best,
> > > > > > Gang
> > > > > >
> > > > > > On Wed, Apr 24, 2024 at 7:04 PM Uwe L. Korn <uw...@xhochy.com>
> > > wrote:
> > > > > >
> > > > > >> I would be very supportive of this move. The Parquet C++
> > > development has
> > > > > >> been under the umbrella of the Arrow repository for more than
> > > five(six?)
> > > > > >> years now. Thus, the issues should also be aligned with the
> Arrow
> > > > > project.
> > > > > >>
> > > > > >> Uwe
> > > > > >>
> > > > > >> On Tue, Apr 23, 2024, at 8:27 PM, Rok Mihevc wrote:
> > > > > >> > Bumping this thread again to see if there is will to call for
> a
> > > vote
> > > > > and
> > > > > >> > move parquet-cpp issues from Apache Jira to Arrow's GitHub
> issue
> > > as
> > > > > was
> > > > > >> > done for Arrow.
> > > > > >> > I'm willing to do the move as I already did it for Arrow.
> > > > > >> >
> > > > > >> > Rok
> > > > > >> >
> > > > > >> > On Sat, Apr 15, 2023 at 4:53 AM Micah Kornfield <
> > > > > emkornfield@apache.org>
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> >> Bumping this thread again to see in any Parquet PMC members
> can
> > > chime
> > > > > >> >> in/maybe start a formal vote to move governance of
> Parquet-CPP
> > > under
> > > > > the
> > > > > >> >> umbrella.
> > > > > >> >>
> > > > > >> >> -Micah
> > > > > >> >>
> > > > > >> >> On 2023/02/02 10:34:25 Antoine Pitrou wrote:
> > > > > >> >> >
> > > > > >> >> >
> > > > > >> >> > Hi Will,
> > > > > >> >> >
> > > > > >> >> > Le 01/02/2023 à 20:27, Will Jones a écrit :
> > > > > >> >> > >
> > > > > >> >> > > First, it's not obvious where issues are supposed to be
> > > open: In
> > > > > >> >> Parquet
> > > > > >> >> > > Jira or Arrow GitHub issues. Looking back at some of the
> > > original
> > > > > >> >> > > discussion, it looks like the intention was
> > > > > >> >> > >
> > > > > >> >> > > * use PARQUET-XXX for issues relating to Parquet core
> > > > > >> >> > >> * use ARROW-XXX for issues relation to Arrow's
> consumption
> > > of
> > > > > >> Parquet
> > > > > >> >> > >> core (e.g. changes that are in parquet/arrow right now)
> > > > > >> >> > >>
> > > > > >> >> > > The README for the old parquet-cpp repo [3] states
> instead
> > in
> > > > > it's
> > > > > >> >> > > migration note:
> > > > > >> >> > >
> > > > > >> >> > > JIRA issues should continue to be opened in the PARQUET
> > > JIRA
> > > > > >> project.
> > > > > >> >> > >
> > > > > >> >> > > Either way, it doesn't seem like this process is obvious
> to
> > > > > people.
> > > > > >> >> Perhaps
> > > > > >> >> > > we could clarify this and add notices to Arrow's GitHub
> > > issues
> > > > > >> >> template?
> > > > > >> >> >
> > > > > >> >> > I agree we should clarify this. I have no personal
> > preference,
> > > but
> > > > > I
> > > > > >> >> will note
> > > > > >> >> > that Github issues decrease friction as having a GH account
> > is
> > > > > already
> > > > > >> >> necessary
> > > > > >> >> > for submitting PRs.
> > > > > >> >> >
> > > > > >> >> > > Second, committer status is a little unclear. I am a
> > > committer on
> > > > > >> >> Arrow,
> > > > > >> >> > > but not on Parquet right now. Does that mean I should
> only
> > > merge
> > > > > >> >> Parquet
> > > > > >> >> > > C++ PRs for code changes in parquet/arrow? Or that I
> > > shouldn't
> > > > > merge
> > > > > >> >> > > Parquet changes at all?
> > > > > >> >> >
> > > > > >> >> > Since Parquet C++ is part of Arrow C++, you are allowed to
> > > merge
> > > > > >> Parquet
> > > > > >> >> C++
> > > > > >> >> > changes. As always you should ensure you have sufficient
> > > > > understanding
> > > > > >> >> of the
> > > > > >> >> > contribution, and that it follows established practices:
> > > > > >> >> >
> https://arrow.apache.org/docs/dev/developers/reviewing.html
> > > > > >> >> >
> > > > > >> >> > > Also, are the contributions to Arrow C++ Parquet being
> > > actively
> > > > > >> >> reviewed
> > > > > >> >> > > for potential new committers?
> > > > > >> >> >
> > > > > >> >> > I would certainly do.
> > > > > >> >> >
> > > > > >> >> > Regards
> > > > > >> >> >
> > > > > >> >> > Antoine.
> > > > > >> >> >
> > > > > >> >> >
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
>
Re: Fwd: [C++] Parquet and Arrow overlap
Posted by Matt Topol <zo...@gmail.com>.
I just wanted to also poke the question of non-Java developers who have
worked on the other parquet implementations potentially being recognized as
committers or otherwise on the Parquet project (speaking as the primary
developer of the Go parquet implementation which also lives in the Arrow
repository). It would be great to see some active contributors to
parquet-cpp, parquet-go, or otherwise not just being considered but
actively becoming committers.
That's just my two cents from a community perspective.
--Matt
On Fri, May 10, 2024, 10:35 PM Jacob Wujciak <as...@apache.org> wrote:
> Thank you, that sounds great! On first glance some seem to be rather old
> and probably don't apply anymore.
>
> > BTW, do we really need to make a full copy of them to have a mirror in
> the Arrow GitHub issues?
>
> I am not sure I understand what you mean? A full copy of the
> open/closed/all issues? I'd say only the (remaining) open issues would be
> fine.
> For reference this is what an imported issue looks like:
> https://github.com/apache/arrow/issues/30543
>
> Am Sa., 11. Mai 2024 um 04:09 Uhr schrieb Gang Wu <us...@gmail.com>:
>
> > I can initiate the vote. But before the vote, I think we need to revisit
> > the states of all unresolved tickets and close some as needed.
> >
> > BTW, do we really need to make a full copy of them to have a mirror
> > in the Arrow GitHub issues?
> >
> > I'd like to seek a consensus here before sending the vote.
> >
> > Best,
> > Gang
> >
> > On Sat, May 11, 2024 at 8:46 AM Jacob Wujciak <as...@apache.org>
> > wrote:
> >
> > > Hello Everyone!
> > >
> > > It seems there is general agreement on this topic, it would be great
> if a
> > > committer/PMC could start a (lazy consensus) procedural vote.
> > >
> > > I will inquire how to handle the parquet-cpp component in jira (ideally
> > > disabling it, not removing).
> > > There are currently only ~70 open tickets for parquet-cpp, with the
> > change
> > > in repo it is probably easier to just move open tickets but I'll leave
> > that
> > > to Rok who managed the transition of Arrows 20k+ tickets too :D
> > >
> > > Thanks,
> > > Jacob
> > >
> > > Arrow committer
> > >
> > > On 2024/04/25 05:31:18 Gang Wu wrote:
> > > > I know we have some non-Java committers and PMCs. But after the
> > > parquet-cpp
> > > > donation, it seems that no one worked on Parquet from arrow (cpp,
> rust,
> > > go,
> > > > etc.)
> > > > and other projects are promoted as a Parquet committer. It would be
> > > > inconvenient
> > > > for non-Java Parquet developers to work with apache/parquet-format
> and
> > > > apache/parquet-testing repositories. Furthermore, votes from these
> > > > developers
> > > > are not binding for a format change in the ML.
> > > >
> > > > Best,
> > > > Gang
> > > >
> > > > On Wed, Apr 24, 2024 at 8:42 PM Uwe L. Korn <uw...@xhochy.com>
> wrote:
> > > >
> > > > > > Should we consider
> > > > > > Parquet developers from other projects than parquet-mr as Parquet
> > > > > commuters?
> > > > >
> > > > > We are doing this (speaking as a Parquet PMC who didn't work on
> > > > > parquet-mr, but parquet-cpp).
> > > > >
> > > > > Best
> > > > > Uwe
> > > > >
> > > > > On Wed, Apr 24, 2024, at 2:38 PM, Gang Wu wrote:
> > > > > > +1 for moving parquet-cpp issues from Apache Jira to Arrow's
> GitHub
> > > > > issue.
> > > > > >
> > > > > > Besides, I want to echo Will's question in the thread. Should we
> > > consider
> > > > > > Parquet developers from other projects than parquet-mr as Parquet
> > > > > commiters?
> > > > > > Currently apache/parquet-format and apache/parquet-testing
> > > repositories
> > > > > are
> > > > > > solely governed by Apache Parquet PMC. It would be better for the
> > > entire
> > > > > > Parquet community if developers with sufficient contributions to
> > open
> > > > > source
> > > > > > Parquet projects (including but not limited to parquet-cpp,
> > arrow-rs,
> > > > > cudf,
> > > > > > etc.)
> > > > > > can be considered as Parquet committer and PMC.
> > > > > >
> > > > > > Best,
> > > > > > Gang
> > > > > >
> > > > > > On Wed, Apr 24, 2024 at 7:04 PM Uwe L. Korn <uw...@xhochy.com>
> > > wrote:
> > > > > >
> > > > > >> I would be very supportive of this move. The Parquet C++
> > > development has
> > > > > >> been under the umbrella of the Arrow repository for more than
> > > five(six?)
> > > > > >> years now. Thus, the issues should also be aligned with the
> Arrow
> > > > > project.
> > > > > >>
> > > > > >> Uwe
> > > > > >>
> > > > > >> On Tue, Apr 23, 2024, at 8:27 PM, Rok Mihevc wrote:
> > > > > >> > Bumping this thread again to see if there is will to call for
> a
> > > vote
> > > > > and
> > > > > >> > move parquet-cpp issues from Apache Jira to Arrow's GitHub
> issue
> > > as
> > > > > was
> > > > > >> > done for Arrow.
> > > > > >> > I'm willing to do the move as I already did it for Arrow.
> > > > > >> >
> > > > > >> > Rok
> > > > > >> >
> > > > > >> > On Sat, Apr 15, 2023 at 4:53 AM Micah Kornfield <
> > > > > emkornfield@apache.org>
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> >> Bumping this thread again to see in any Parquet PMC members
> can
> > > chime
> > > > > >> >> in/maybe start a formal vote to move governance of
> Parquet-CPP
> > > under
> > > > > the
> > > > > >> >> umbrella.
> > > > > >> >>
> > > > > >> >> -Micah
> > > > > >> >>
> > > > > >> >> On 2023/02/02 10:34:25 Antoine Pitrou wrote:
> > > > > >> >> >
> > > > > >> >> >
> > > > > >> >> > Hi Will,
> > > > > >> >> >
> > > > > >> >> > Le 01/02/2023 à 20:27, Will Jones a écrit :
> > > > > >> >> > >
> > > > > >> >> > > First, it's not obvious where issues are supposed to be
> > > open: In
> > > > > >> >> Parquet
> > > > > >> >> > > Jira or Arrow GitHub issues. Looking back at some of the
> > > original
> > > > > >> >> > > discussion, it looks like the intention was
> > > > > >> >> > >
> > > > > >> >> > > * use PARQUET-XXX for issues relating to Parquet core
> > > > > >> >> > >> * use ARROW-XXX for issues relation to Arrow's
> consumption
> > > of
> > > > > >> Parquet
> > > > > >> >> > >> core (e.g. changes that are in parquet/arrow right now)
> > > > > >> >> > >>
> > > > > >> >> > > The README for the old parquet-cpp repo [3] states
> instead
> > in
> > > > > it's
> > > > > >> >> > > migration note:
> > > > > >> >> > >
> > > > > >> >> > > JIRA issues should continue to be opened in the PARQUET
> > > JIRA
> > > > > >> project.
> > > > > >> >> > >
> > > > > >> >> > > Either way, it doesn't seem like this process is obvious
> to
> > > > > people.
> > > > > >> >> Perhaps
> > > > > >> >> > > we could clarify this and add notices to Arrow's GitHub
> > > issues
> > > > > >> >> template?
> > > > > >> >> >
> > > > > >> >> > I agree we should clarify this. I have no personal
> > preference,
> > > but
> > > > > I
> > > > > >> >> will note
> > > > > >> >> > that Github issues decrease friction as having a GH account
> > is
> > > > > already
> > > > > >> >> necessary
> > > > > >> >> > for submitting PRs.
> > > > > >> >> >
> > > > > >> >> > > Second, committer status is a little unclear. I am a
> > > committer on
> > > > > >> >> Arrow,
> > > > > >> >> > > but not on Parquet right now. Does that mean I should
> only
> > > merge
> > > > > >> >> Parquet
> > > > > >> >> > > C++ PRs for code changes in parquet/arrow? Or that I
> > > shouldn't
> > > > > merge
> > > > > >> >> > > Parquet changes at all?
> > > > > >> >> >
> > > > > >> >> > Since Parquet C++ is part of Arrow C++, you are allowed to
> > > merge
> > > > > >> Parquet
> > > > > >> >> C++
> > > > > >> >> > changes. As always you should ensure you have sufficient
> > > > > understanding
> > > > > >> >> of the
> > > > > >> >> > contribution, and that it follows established practices:
> > > > > >> >> >
> https://arrow.apache.org/docs/dev/developers/reviewing.html
> > > > > >> >> >
> > > > > >> >> > > Also, are the contributions to Arrow C++ Parquet being
> > > actively
> > > > > >> >> reviewed
> > > > > >> >> > > for potential new committers?
> > > > > >> >> >
> > > > > >> >> > I would certainly do.
> > > > > >> >> >
> > > > > >> >> > Regards
> > > > > >> >> >
> > > > > >> >> > Antoine.
> > > > > >> >> >
> > > > > >> >> >
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
>
Re: Fwd: [C++] Parquet and Arrow overlap
Posted by Jacob Wujciak <as...@apache.org>.
Thank you, that sounds great! On first glance some seem to be rather old
and probably don't apply anymore.
> BTW, do we really need to make a full copy of them to have a mirror in
the Arrow GitHub issues?
I am not sure I understand what you mean? A full copy of the
open/closed/all issues? I'd say only the (remaining) open issues would be
fine.
For reference this is what an imported issue looks like:
https://github.com/apache/arrow/issues/30543
Am Sa., 11. Mai 2024 um 04:09 Uhr schrieb Gang Wu <us...@gmail.com>:
> I can initiate the vote. But before the vote, I think we need to revisit
> the states of all unresolved tickets and close some as needed.
>
> BTW, do we really need to make a full copy of them to have a mirror
> in the Arrow GitHub issues?
>
> I'd like to seek a consensus here before sending the vote.
>
> Best,
> Gang
>
> On Sat, May 11, 2024 at 8:46 AM Jacob Wujciak <as...@apache.org>
> wrote:
>
> > Hello Everyone!
> >
> > It seems there is general agreement on this topic, it would be great if a
> > committer/PMC could start a (lazy consensus) procedural vote.
> >
> > I will inquire how to handle the parquet-cpp component in jira (ideally
> > disabling it, not removing).
> > There are currently only ~70 open tickets for parquet-cpp, with the
> change
> > in repo it is probably easier to just move open tickets but I'll leave
> that
> > to Rok who managed the transition of Arrows 20k+ tickets too :D
> >
> > Thanks,
> > Jacob
> >
> > Arrow committer
> >
> > On 2024/04/25 05:31:18 Gang Wu wrote:
> > > I know we have some non-Java committers and PMCs. But after the
> > parquet-cpp
> > > donation, it seems that no one worked on Parquet from arrow (cpp, rust,
> > go,
> > > etc.)
> > > and other projects are promoted as a Parquet committer. It would be
> > > inconvenient
> > > for non-Java Parquet developers to work with apache/parquet-format and
> > > apache/parquet-testing repositories. Furthermore, votes from these
> > > developers
> > > are not binding for a format change in the ML.
> > >
> > > Best,
> > > Gang
> > >
> > > On Wed, Apr 24, 2024 at 8:42 PM Uwe L. Korn <uw...@xhochy.com> wrote:
> > >
> > > > > Should we consider
> > > > > Parquet developers from other projects than parquet-mr as Parquet
> > > > commuters?
> > > >
> > > > We are doing this (speaking as a Parquet PMC who didn't work on
> > > > parquet-mr, but parquet-cpp).
> > > >
> > > > Best
> > > > Uwe
> > > >
> > > > On Wed, Apr 24, 2024, at 2:38 PM, Gang Wu wrote:
> > > > > +1 for moving parquet-cpp issues from Apache Jira to Arrow's GitHub
> > > > issue.
> > > > >
> > > > > Besides, I want to echo Will's question in the thread. Should we
> > consider
> > > > > Parquet developers from other projects than parquet-mr as Parquet
> > > > commiters?
> > > > > Currently apache/parquet-format and apache/parquet-testing
> > repositories
> > > > are
> > > > > solely governed by Apache Parquet PMC. It would be better for the
> > entire
> > > > > Parquet community if developers with sufficient contributions to
> open
> > > > source
> > > > > Parquet projects (including but not limited to parquet-cpp,
> arrow-rs,
> > > > cudf,
> > > > > etc.)
> > > > > can be considered as Parquet committer and PMC.
> > > > >
> > > > > Best,
> > > > > Gang
> > > > >
> > > > > On Wed, Apr 24, 2024 at 7:04 PM Uwe L. Korn <uw...@xhochy.com>
> > wrote:
> > > > >
> > > > >> I would be very supportive of this move. The Parquet C++
> > development has
> > > > >> been under the umbrella of the Arrow repository for more than
> > five(six?)
> > > > >> years now. Thus, the issues should also be aligned with the Arrow
> > > > project.
> > > > >>
> > > > >> Uwe
> > > > >>
> > > > >> On Tue, Apr 23, 2024, at 8:27 PM, Rok Mihevc wrote:
> > > > >> > Bumping this thread again to see if there is will to call for a
> > vote
> > > > and
> > > > >> > move parquet-cpp issues from Apache Jira to Arrow's GitHub issue
> > as
> > > > was
> > > > >> > done for Arrow.
> > > > >> > I'm willing to do the move as I already did it for Arrow.
> > > > >> >
> > > > >> > Rok
> > > > >> >
> > > > >> > On Sat, Apr 15, 2023 at 4:53 AM Micah Kornfield <
> > > > emkornfield@apache.org>
> > > > >> > wrote:
> > > > >> >
> > > > >> >> Bumping this thread again to see in any Parquet PMC members can
> > chime
> > > > >> >> in/maybe start a formal vote to move governance of Parquet-CPP
> > under
> > > > the
> > > > >> >> umbrella.
> > > > >> >>
> > > > >> >> -Micah
> > > > >> >>
> > > > >> >> On 2023/02/02 10:34:25 Antoine Pitrou wrote:
> > > > >> >> >
> > > > >> >> >
> > > > >> >> > Hi Will,
> > > > >> >> >
> > > > >> >> > Le 01/02/2023 à 20:27, Will Jones a écrit :
> > > > >> >> > >
> > > > >> >> > > First, it's not obvious where issues are supposed to be
> > open: In
> > > > >> >> Parquet
> > > > >> >> > > Jira or Arrow GitHub issues. Looking back at some of the
> > original
> > > > >> >> > > discussion, it looks like the intention was
> > > > >> >> > >
> > > > >> >> > > * use PARQUET-XXX for issues relating to Parquet core
> > > > >> >> > >> * use ARROW-XXX for issues relation to Arrow's consumption
> > of
> > > > >> Parquet
> > > > >> >> > >> core (e.g. changes that are in parquet/arrow right now)
> > > > >> >> > >>
> > > > >> >> > > The README for the old parquet-cpp repo [3] states instead
> in
> > > > it's
> > > > >> >> > > migration note:
> > > > >> >> > >
> > > > >> >> > > JIRA issues should continue to be opened in the PARQUET
> > JIRA
> > > > >> project.
> > > > >> >> > >
> > > > >> >> > > Either way, it doesn't seem like this process is obvious to
> > > > people.
> > > > >> >> Perhaps
> > > > >> >> > > we could clarify this and add notices to Arrow's GitHub
> > issues
> > > > >> >> template?
> > > > >> >> >
> > > > >> >> > I agree we should clarify this. I have no personal
> preference,
> > but
> > > > I
> > > > >> >> will note
> > > > >> >> > that Github issues decrease friction as having a GH account
> is
> > > > already
> > > > >> >> necessary
> > > > >> >> > for submitting PRs.
> > > > >> >> >
> > > > >> >> > > Second, committer status is a little unclear. I am a
> > committer on
> > > > >> >> Arrow,
> > > > >> >> > > but not on Parquet right now. Does that mean I should only
> > merge
> > > > >> >> Parquet
> > > > >> >> > > C++ PRs for code changes in parquet/arrow? Or that I
> > shouldn't
> > > > merge
> > > > >> >> > > Parquet changes at all?
> > > > >> >> >
> > > > >> >> > Since Parquet C++ is part of Arrow C++, you are allowed to
> > merge
> > > > >> Parquet
> > > > >> >> C++
> > > > >> >> > changes. As always you should ensure you have sufficient
> > > > understanding
> > > > >> >> of the
> > > > >> >> > contribution, and that it follows established practices:
> > > > >> >> > https://arrow.apache.org/docs/dev/developers/reviewing.html
> > > > >> >> >
> > > > >> >> > > Also, are the contributions to Arrow C++ Parquet being
> > actively
> > > > >> >> reviewed
> > > > >> >> > > for potential new committers?
> > > > >> >> >
> > > > >> >> > I would certainly do.
> > > > >> >> >
> > > > >> >> > Regards
> > > > >> >> >
> > > > >> >> > Antoine.
> > > > >> >> >
> > > > >> >> >
> > > > >> >>
> > > > >>
> > > >
> > >
> >
>
Re: Fwd: [C++] Parquet and Arrow overlap
Posted by Jacob Wujciak <as...@apache.org>.
Thank you, that sounds great! On first glance some seem to be rather old
and probably don't apply anymore.
> BTW, do we really need to make a full copy of them to have a mirror in
the Arrow GitHub issues?
I am not sure I understand what you mean? A full copy of the
open/closed/all issues? I'd say only the (remaining) open issues would be
fine.
For reference this is what an imported issue looks like:
https://github.com/apache/arrow/issues/30543
Am Sa., 11. Mai 2024 um 04:09 Uhr schrieb Gang Wu <us...@gmail.com>:
> I can initiate the vote. But before the vote, I think we need to revisit
> the states of all unresolved tickets and close some as needed.
>
> BTW, do we really need to make a full copy of them to have a mirror
> in the Arrow GitHub issues?
>
> I'd like to seek a consensus here before sending the vote.
>
> Best,
> Gang
>
> On Sat, May 11, 2024 at 8:46 AM Jacob Wujciak <as...@apache.org>
> wrote:
>
> > Hello Everyone!
> >
> > It seems there is general agreement on this topic, it would be great if a
> > committer/PMC could start a (lazy consensus) procedural vote.
> >
> > I will inquire how to handle the parquet-cpp component in jira (ideally
> > disabling it, not removing).
> > There are currently only ~70 open tickets for parquet-cpp, with the
> change
> > in repo it is probably easier to just move open tickets but I'll leave
> that
> > to Rok who managed the transition of Arrows 20k+ tickets too :D
> >
> > Thanks,
> > Jacob
> >
> > Arrow committer
> >
> > On 2024/04/25 05:31:18 Gang Wu wrote:
> > > I know we have some non-Java committers and PMCs. But after the
> > parquet-cpp
> > > donation, it seems that no one worked on Parquet from arrow (cpp, rust,
> > go,
> > > etc.)
> > > and other projects are promoted as a Parquet committer. It would be
> > > inconvenient
> > > for non-Java Parquet developers to work with apache/parquet-format and
> > > apache/parquet-testing repositories. Furthermore, votes from these
> > > developers
> > > are not binding for a format change in the ML.
> > >
> > > Best,
> > > Gang
> > >
> > > On Wed, Apr 24, 2024 at 8:42 PM Uwe L. Korn <uw...@xhochy.com> wrote:
> > >
> > > > > Should we consider
> > > > > Parquet developers from other projects than parquet-mr as Parquet
> > > > commuters?
> > > >
> > > > We are doing this (speaking as a Parquet PMC who didn't work on
> > > > parquet-mr, but parquet-cpp).
> > > >
> > > > Best
> > > > Uwe
> > > >
> > > > On Wed, Apr 24, 2024, at 2:38 PM, Gang Wu wrote:
> > > > > +1 for moving parquet-cpp issues from Apache Jira to Arrow's GitHub
> > > > issue.
> > > > >
> > > > > Besides, I want to echo Will's question in the thread. Should we
> > consider
> > > > > Parquet developers from other projects than parquet-mr as Parquet
> > > > commiters?
> > > > > Currently apache/parquet-format and apache/parquet-testing
> > repositories
> > > > are
> > > > > solely governed by Apache Parquet PMC. It would be better for the
> > entire
> > > > > Parquet community if developers with sufficient contributions to
> open
> > > > source
> > > > > Parquet projects (including but not limited to parquet-cpp,
> arrow-rs,
> > > > cudf,
> > > > > etc.)
> > > > > can be considered as Parquet committer and PMC.
> > > > >
> > > > > Best,
> > > > > Gang
> > > > >
> > > > > On Wed, Apr 24, 2024 at 7:04 PM Uwe L. Korn <uw...@xhochy.com>
> > wrote:
> > > > >
> > > > >> I would be very supportive of this move. The Parquet C++
> > development has
> > > > >> been under the umbrella of the Arrow repository for more than
> > five(six?)
> > > > >> years now. Thus, the issues should also be aligned with the Arrow
> > > > project.
> > > > >>
> > > > >> Uwe
> > > > >>
> > > > >> On Tue, Apr 23, 2024, at 8:27 PM, Rok Mihevc wrote:
> > > > >> > Bumping this thread again to see if there is will to call for a
> > vote
> > > > and
> > > > >> > move parquet-cpp issues from Apache Jira to Arrow's GitHub issue
> > as
> > > > was
> > > > >> > done for Arrow.
> > > > >> > I'm willing to do the move as I already did it for Arrow.
> > > > >> >
> > > > >> > Rok
> > > > >> >
> > > > >> > On Sat, Apr 15, 2023 at 4:53 AM Micah Kornfield <
> > > > emkornfield@apache.org>
> > > > >> > wrote:
> > > > >> >
> > > > >> >> Bumping this thread again to see in any Parquet PMC members can
> > chime
> > > > >> >> in/maybe start a formal vote to move governance of Parquet-CPP
> > under
> > > > the
> > > > >> >> umbrella.
> > > > >> >>
> > > > >> >> -Micah
> > > > >> >>
> > > > >> >> On 2023/02/02 10:34:25 Antoine Pitrou wrote:
> > > > >> >> >
> > > > >> >> >
> > > > >> >> > Hi Will,
> > > > >> >> >
> > > > >> >> > Le 01/02/2023 à 20:27, Will Jones a écrit :
> > > > >> >> > >
> > > > >> >> > > First, it's not obvious where issues are supposed to be
> > open: In
> > > > >> >> Parquet
> > > > >> >> > > Jira or Arrow GitHub issues. Looking back at some of the
> > original
> > > > >> >> > > discussion, it looks like the intention was
> > > > >> >> > >
> > > > >> >> > > * use PARQUET-XXX for issues relating to Parquet core
> > > > >> >> > >> * use ARROW-XXX for issues relation to Arrow's consumption
> > of
> > > > >> Parquet
> > > > >> >> > >> core (e.g. changes that are in parquet/arrow right now)
> > > > >> >> > >>
> > > > >> >> > > The README for the old parquet-cpp repo [3] states instead
> in
> > > > it's
> > > > >> >> > > migration note:
> > > > >> >> > >
> > > > >> >> > > JIRA issues should continue to be opened in the PARQUET
> > JIRA
> > > > >> project.
> > > > >> >> > >
> > > > >> >> > > Either way, it doesn't seem like this process is obvious to
> > > > people.
> > > > >> >> Perhaps
> > > > >> >> > > we could clarify this and add notices to Arrow's GitHub
> > issues
> > > > >> >> template?
> > > > >> >> >
> > > > >> >> > I agree we should clarify this. I have no personal
> preference,
> > but
> > > > I
> > > > >> >> will note
> > > > >> >> > that Github issues decrease friction as having a GH account
> is
> > > > already
> > > > >> >> necessary
> > > > >> >> > for submitting PRs.
> > > > >> >> >
> > > > >> >> > > Second, committer status is a little unclear. I am a
> > committer on
> > > > >> >> Arrow,
> > > > >> >> > > but not on Parquet right now. Does that mean I should only
> > merge
> > > > >> >> Parquet
> > > > >> >> > > C++ PRs for code changes in parquet/arrow? Or that I
> > shouldn't
> > > > merge
> > > > >> >> > > Parquet changes at all?
> > > > >> >> >
> > > > >> >> > Since Parquet C++ is part of Arrow C++, you are allowed to
> > merge
> > > > >> Parquet
> > > > >> >> C++
> > > > >> >> > changes. As always you should ensure you have sufficient
> > > > understanding
> > > > >> >> of the
> > > > >> >> > contribution, and that it follows established practices:
> > > > >> >> > https://arrow.apache.org/docs/dev/developers/reviewing.html
> > > > >> >> >
> > > > >> >> > > Also, are the contributions to Arrow C++ Parquet being
> > actively
> > > > >> >> reviewed
> > > > >> >> > > for potential new committers?
> > > > >> >> >
> > > > >> >> > I would certainly do.
> > > > >> >> >
> > > > >> >> > Regards
> > > > >> >> >
> > > > >> >> > Antoine.
> > > > >> >> >
> > > > >> >> >
> > > > >> >>
> > > > >>
> > > >
> > >
> >
>