You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Andrew Lamb <al...@influxdata.com> on 2024/03/01 11:33:08 UTC
[VOTE] Move Arrow DataFusion Subproject to new Top Level Apache Project
Hello,
As we have discussed[1][2] I would like to vote on the proposal to
create a new Apache Top Level Project for DataFusion. The text of the
proposed resolution and background document is copy/pasted below
If the community is in favor of this, we plan to submit the resolution
to the ASF board for approval with the next Arrow report (for the
April 2024 board meeting).
The vote will be open for at least 7 days.
[ ] +1 Accept this Proposal
[ ] +0
[ ] -1 Do not accept this proposal because...
Andrew
[1] https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
[2] https://github.com/apache/arrow-datafusion/discussions/6475
---------- Proposed Resolution ---------
Resolution to Create the Apache DataFusion Project from the Apache
Arrow DataFusion Sub Project
=============================================================
X. Establish the Apache DataFusion Project
WHEREAS, the Board of Directors deems it to be in the best
interests of the Foundation and consistent with the
Foundation's purpose to establish a Project Management
Committee charged with the creation and maintenance of
open-source software related to an extensible query engine
for distribution at no charge to the public.
NOW, THEREFORE, BE IT RESOLVED, that a Project Management
Committee (PMC), to be known as the "Apache DataFusion Project",
be and hereby is established pursuant to Bylaws of the
Foundation; and be it further
RESOLVED, that the Apache DataFusion Project be and hereby is
responsible for the creation and maintenance of software
related to an extensible query engine; and be it further
RESOLVED, that the office of "Vice President, Apache DataFusion" be
and hereby is created, the person holding such office to
serve at the direction of the Board of Directors as the chair
of the Apache DataFusion Project, and to have primary responsibility
for management of the projects within the scope of
responsibility of the Apache DataFusion Project; and be it further
RESOLVED, that the persons listed immediately below be and
hereby are appointed to serve as the initial members of the
Apache DataFusion Project:
* Andy Grove (agrove@apache.org)
* Andrew Lamb (alamb@apache.org)
* Daniël Heres (dheres@apache.org)
* Jie Wen (jakevin@apache.org)
* Kun Liu (liukun@apache.org)
* Liang-Chi Hsieh (viirya@apache.org)
* Qingping Hou: (houqp@apache.org)
* Wes McKinney(wesm@apache.org)
* Will Jones (wjones127@apache.org)
RESOLVED, that the Apache DataFusion Project be and hereby
is tasked with the migration and rationalization of the Apache
Arrow DataFusion sub-project; and be it further
RESOLVED, that all responsibilities pertaining to the Apache
Arrow DataFusion sub-project encumbered upon the
Apache Arrow Project are hereafter discharged.
NOW, THEREFORE, BE IT FURTHER RESOLVED, that Andrew Lamb
be appointed to the office of Vice President, Apache DataFusion, to
serve in accordance with and subject to the direction of the
Board of Directors and the Bylaws of the Foundation until
death, resignation, retirement, removal or disqualification,
or until a successor is appointed.
=============================================================
-------
Summary:
We propose creating a new top level project, Apache DataFusion, from
an existing sub project of Apache Arrow to facilitate additional
community and project growth.
Abstract
Apache Arrow DataFusion[1] is a very fast, extensible query engine
for building high-quality data-centric systems in Rust, using the
Apache Arrow in-memory format. DataFusion offers SQL and Dataframe
APIs, excellent performance, built-in support for CSV, Parquet, JSON,
and Avro, extensive customization, and a great community.
[1] https://arrow.apache.org/datafusion/
Proposal
We propose creating a new top level ASF project, Apache DataFusion,
governed initially by a subset of the Apache Arrow project’s PMC and
committers. The project’s code is in five existing git repositories,
currently governed by Apache Arrow which would transfer to the new top
level project.
Background
When DataFusion was initially donated to the Arrow project, it did not
have a strong enough community to stand on its own. It has since grown
significantly, and benefited immensely from being part of Arrow and
nurturing of the Apache Way, and now has a community strong enough to
stand on its own and that would benefit from focused governance
attention.
The community has discussed this idea publicly for more than 6 months
https://github.com/apache/arrow-datafusion/discussions/6475 and
briefly on the Arrow PMC mailing list
https://lists.apache.org/thread/thv2jdm6640l6gm88hy8jhk5prjww0cs. As
of the time of this writing both had exclusively positive reactions.
Several current members of the Arrow PMC are both active contributors
to DataFusion and understand and believe deeply in the Apache Way, and
play active governance roles in the Arrow project as PMC members and
PMC chairs, guiding the community, and releasing software versions.
With this existing governance experience and structure, the new top
level project will be able to function well immediately and
independently.
Overview of DataFusion
Current Status
Meritocracy
DataFusion has been developed as part of Apache Arrow and thus has
been operating as a meritocracy. Many of the developers of DataFusion
are Arrow PMC members or committers. The DataFusion project plans to
continue adding new PMC and committers as the project matures and
grows.
Community
The DataFusion development team seeks to foster the development and
user communities. We hope that becoming a separate project will help
both Arrow and DataFusion communities by being more focused. Focused
governance will make it easier to grow the community of committers and
PMC members and make the organization more clear to others.
Alignment
The ASF is a natural host for DataFusion given that it is already the
home of Arrow, Parquet, and other related distributed system, storage
and query execution systems.
Project Leadership
Proposed Initial PMC
We propose the following people as the initial DataFusion PMC members.
This is a subset of the existing Arrow PMC members who contribute to
DataFusion https://people.apache.org/phonebook.html?unix=arrow
Andy Grove (agrove): Arrow PMC Chair
Andrew Lamb (alamb): Arrow PMC, past Arrow PMC Chair
Daniël Heres (dheres) Arrow PMC
Jie Wen (jakevin): Arrow PMC, Doris Committer
Kun Liu (liukun): Arrow PMC, IoTDB PMC, TSFile PMC
Liang-Chi Hsieh (viirya): Arrow PMC, Spark PMC
Qingping Hou: (houqp): Arrow PMC
Wes McKinney(wesm): Arrow PMC, ASF Member
Will Jones (wjones127): Arrow PMC
We’d like to propose Andrew Lamb as the initial Chair, (and thus ASF
VP) for the DataFusion project.
Affiliations
Andy Grove (agrove): NVidia
Andrew Lamb (alamb): InfluxData
Daniël Heres (dheres): Coralogix
Jie Wen (jakevin): SelectDB
Kun Liu (liukun): Ebay
Liang-Chi Hsieh (viirya): Apple
Qingping Hou: (houqp): Scribd
Wes McKinney(wesm): Posit
Will Jones (wjones127): LanceDB
Proposed Initial Committers
In addition to the PMC, we propose the following people as the initial
DataFusion committers. This is a subset of the existing Arrow
committers who contribute to DataFusion
https://people.apache.org/phonebook.html?unix=arrow
akurmustafa Mustafa Akur (Synnada)
avantgardner Brent Gardner (Coralogix)
comphead Oleks V. (Unaffiliated)
jayzhan Jay Zhan (Unaffiliated)
jeffreyvo Jeffry Vo (Unaffiliated)
jiayuliu Liu Jiayu (Airbnb)
mete Metehan Yildirim (Synnada)
mingmwang Wang Mingming (Ebay)
mneumann Marco Neumann (InfluxData)
nju_yaho Zhong Yanghong (Ebay)
ozankabak Mehmet Ozan Kabak (Synnada)
paddyhoran Paddy Horan (Assured Allies)
rdettai Rémi Dettai (Cloudfuse)
sunchao Chao Sun (Apple)
thinkharderdev Daniel Harris (Coralogix)
tustvold Raphael Taylor-Davies (InfluxData)
wayne Ruihang Xia (Greptime)
xudong963 Xudong Wang (ByteDance)
yjshen Yijie Shen (Space and Time)
yangjiang Yang Jiang (ebay)
Risk Assessments
Naming / Trademarks
As a sub-project of Arrow, the DataFusion name has been used for over
4 years without any known issues. A podling name search did not turn
up any concerns and was approved:
https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
Legal / IP Clearance
All DataFusion code has either been donated to the Arrow project with
appropriate IP clearance or has been developed directly under ASF
processes and procedures. Thus creating a new top level project poses
no new Legal or IP risks.
Code Extraction
The relevant code is already in 5 separate repositories:
https://github.com/apache/arrow-datafusion/
https://github.com/apache/arrow-datafusion-python
https://github.com/apache/arrow-ballista
https://github.com/apache/arrow-ballista-python
https://github.com/apache/arrow-datafusion-comet
We foresee no issues with code extraction and propose these
repositories be renamed to reflect top level projects
Note: https://github.com/apache/arrow-rs, the Rust implementation of
Arrow, would remain part of the Arrow project.
Orphaned Products
DataFusion is known to be used in many open source and commercial
projects https://arrow.apache.org/datafusion/user-guide/introduction.html#known-users,
has had multiple commits daily for several years, and its adoption and
number of contributors appears to be growing. We do not foresee the
project being orphaned in the next several years.
Inexperience with Open Source
The proposed PMC has extensive experience with Apache Arrow and other
Apache projects, and includes PMC members, PMC chairs and an ASF
Member. The DataFusion PMC and more experienced committers will
continue to coach new community members who may be less familiar with
the Apache Way.
Homogeneous Developers
The 9 proposed PMC members are from 9 different employers and the
proposed committers are similarly distributed across affiliations. No
specific entity employs more than 3 total proposed developers.
Reliance on Salaried Developers
A substantial amount of work on DataFusion has been by salaried
developers, but it also has a long tradition of attracting
contributions from students and hobbyists and we plan no changes in
contribution structure.
Relationships with Other Apache Products
DataFusion will obviously have a strong relationship with the Arrow
project given the overlap in people. We don’t foresee close
collaboration with other projects at this time.
Cryptography
DataFusion does not directly support encryption and there are no
near-term plans to add support for encryption. Users who need this
functionality can use the extension APIs.
Required Resources
Mailing Lists
- private@datafusion.apache.org for private PMC discussions (with
moderated subscriptions)
- dev@datafusion.apache.org
- commits@datafusion.apache.org
- user@datafusion.apache.org
Version Control
We propose to continue to use git for source control and github for
hosting and testing resources.
We also need to rename the github repositories to reflect the new top
level names:
https://github.com/apache/arrow-datafusion/ → apache/datafusion
https://github.com/apache/arrow-datafusion-python → apache/datafusion-python
https://github.com/apache/arrow-ballista → apache/datafusion-ballista
https://github.com/apache/arrow-ballista-python →
apache/datafusion-ballista-python
https://github.com/apache/arrow-datafusion-comet → apache/datafusion-comet
Issue Tracking
DataFusion would continue to use github for its issue tracking and
communications
Other Resources
The existing repositories already make use of existing Apache
infrastructure, and we expect no change in the initial resource usage.
As the project continues to grow, we expect continued infrastructure
demand growth.
FAQ: Has a sub project been promoted to a top level project before?
Yes, and it appears to happen commonly. The Arrow project itself was
created as a top level project from work that started in Apache Drill,
and there are many sub projects of Hadoop that spun out as their own
top level projects such as Mahout, Avro and HBase:
https://news.apache.org/foundation/entry/the_apache_software_foundation_announces4
Related material:
Name search request / research for DataFusion:
https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
Discussion about this proposal on the arrow mailing list:
https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
Discussion about which repositories on the arrow mailing list:
https://lists.apache.org/thread/ob3n0d9ky0bgrryl3xn39w9k566bq00q
Discussion about initial PMC on the arrow mailing list:
https://lists.apache.org/thread/pymrzcdw4qdptvby85f69rg3pcckl15b
Discussion in github about creating a new DataFusion top level
project: https://github.com/apache/arrow-datafusion/discussions/6475
Discussion about graduating on incubator list:
https://lists.apache.org/thread/r4n73pmms1lv0jbohyx1o1z13d615t99
Original Proposal for the Arrow project:
https://lists.apache.org/thread/x2qzdwglm8pkqp9gv03bbgw17khl7pq3
Re: [VOTE] Move Arrow DataFusion Subproject to new Top Level Apache Project
Posted by Raúl Cumplido <ra...@apache.org>.
+1 (binding)
El sáb, 2 mar 2024, 7:28, Jean-Baptiste Onofré <jb...@nanthrax.net> escribió:
> +1 (non binding)
>
> Regards
> JB
>
> On Fri, Mar 1, 2024 at 12:33 PM Andrew Lamb <al...@influxdata.com> wrote:
> >
> > Hello,
> >
> > As we have discussed[1][2] I would like to vote on the proposal to
> > create a new Apache Top Level Project for DataFusion. The text of the
> > proposed resolution and background document is copy/pasted below
> >
> > If the community is in favor of this, we plan to submit the resolution
> > to the ASF board for approval with the next Arrow report (for the
> > April 2024 board meeting).
> >
> > The vote will be open for at least 7 days.
> >
> > [ ] +1 Accept this Proposal
> > [ ] +0
> > [ ] -1 Do not accept this proposal because...
> >
> > Andrew
> >
> > [1] https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> > [2] https://github.com/apache/arrow-datafusion/discussions/6475
> >
> > ---------- Proposed Resolution ---------
> >
> > Resolution to Create the Apache DataFusion Project from the Apache
> > Arrow DataFusion Sub Project
> >
> > =============================================================
> >
> > X. Establish the Apache DataFusion Project
> >
> > WHEREAS, the Board of Directors deems it to be in the best
> > interests of the Foundation and consistent with the
> > Foundation's purpose to establish a Project Management
> > Committee charged with the creation and maintenance of
> > open-source software related to an extensible query engine
> > for distribution at no charge to the public.
> >
> > NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> > Committee (PMC), to be known as the "Apache DataFusion Project",
> > be and hereby is established pursuant to Bylaws of the
> > Foundation; and be it further
> >
> > RESOLVED, that the Apache DataFusion Project be and hereby is
> > responsible for the creation and maintenance of software
> > related to an extensible query engine; and be it further
> >
> > RESOLVED, that the office of "Vice President, Apache DataFusion" be
> > and hereby is created, the person holding such office to
> > serve at the direction of the Board of Directors as the chair
> > of the Apache DataFusion Project, and to have primary responsibility
> > for management of the projects within the scope of
> > responsibility of the Apache DataFusion Project; and be it further
> >
> > RESOLVED, that the persons listed immediately below be and
> > hereby are appointed to serve as the initial members of the
> > Apache DataFusion Project:
> >
> > * Andy Grove (agrove@apache.org)
> > * Andrew Lamb (alamb@apache.org)
> > * Daniël Heres (dheres@apache.org)
> > * Jie Wen (jakevin@apache.org)
> > * Kun Liu (liukun@apache.org)
> > * Liang-Chi Hsieh (viirya@apache.org)
> > * Qingping Hou: (houqp@apache.org)
> > * Wes McKinney(wesm@apache.org)
> > * Will Jones (wjones127@apache.org)
> >
> > RESOLVED, that the Apache DataFusion Project be and hereby
> > is tasked with the migration and rationalization of the Apache
> > Arrow DataFusion sub-project; and be it further
> >
> > RESOLVED, that all responsibilities pertaining to the Apache
> > Arrow DataFusion sub-project encumbered upon the
> > Apache Arrow Project are hereafter discharged.
> >
> > NOW, THEREFORE, BE IT FURTHER RESOLVED, that Andrew Lamb
> > be appointed to the office of Vice President, Apache DataFusion, to
> > serve in accordance with and subject to the direction of the
> > Board of Directors and the Bylaws of the Foundation until
> > death, resignation, retirement, removal or disqualification,
> > or until a successor is appointed.
> > =============================================================
> >
> >
> > -------
> >
> >
> > Summary:
> >
> > We propose creating a new top level project, Apache DataFusion, from
> > an existing sub project of Apache Arrow to facilitate additional
> > community and project growth.
> >
> > Abstract
> >
> > Apache Arrow DataFusion[1] is a very fast, extensible query engine
> > for building high-quality data-centric systems in Rust, using the
> > Apache Arrow in-memory format. DataFusion offers SQL and Dataframe
> > APIs, excellent performance, built-in support for CSV, Parquet, JSON,
> > and Avro, extensive customization, and a great community.
> >
> > [1] https://arrow.apache.org/datafusion/
> >
> >
> > Proposal
> >
> > We propose creating a new top level ASF project, Apache DataFusion,
> > governed initially by a subset of the Apache Arrow project’s PMC and
> > committers. The project’s code is in five existing git repositories,
> > currently governed by Apache Arrow which would transfer to the new top
> > level project.
> >
> > Background
> >
> > When DataFusion was initially donated to the Arrow project, it did not
> > have a strong enough community to stand on its own. It has since grown
> > significantly, and benefited immensely from being part of Arrow and
> > nurturing of the Apache Way, and now has a community strong enough to
> > stand on its own and that would benefit from focused governance
> > attention.
> >
> > The community has discussed this idea publicly for more than 6 months
> > https://github.com/apache/arrow-datafusion/discussions/6475 and
> > briefly on the Arrow PMC mailing list
> > https://lists.apache.org/thread/thv2jdm6640l6gm88hy8jhk5prjww0cs. As
> > of the time of this writing both had exclusively positive reactions.
> >
> > Several current members of the Arrow PMC are both active contributors
> > to DataFusion and understand and believe deeply in the Apache Way, and
> > play active governance roles in the Arrow project as PMC members and
> > PMC chairs, guiding the community, and releasing software versions.
> > With this existing governance experience and structure, the new top
> > level project will be able to function well immediately and
> > independently.
> >
> > Overview of DataFusion
> >
> > Current Status
> >
> > Meritocracy
> >
> > DataFusion has been developed as part of Apache Arrow and thus has
> > been operating as a meritocracy. Many of the developers of DataFusion
> > are Arrow PMC members or committers. The DataFusion project plans to
> > continue adding new PMC and committers as the project matures and
> > grows.
> >
> > Community
> >
> > The DataFusion development team seeks to foster the development and
> > user communities. We hope that becoming a separate project will help
> > both Arrow and DataFusion communities by being more focused. Focused
> > governance will make it easier to grow the community of committers and
> > PMC members and make the organization more clear to others.
> >
> > Alignment
> >
> > The ASF is a natural host for DataFusion given that it is already the
> > home of Arrow, Parquet, and other related distributed system, storage
> > and query execution systems.
> >
> > Project Leadership
> >
> > Proposed Initial PMC
> >
> > We propose the following people as the initial DataFusion PMC members.
> > This is a subset of the existing Arrow PMC members who contribute to
> > DataFusion https://people.apache.org/phonebook.html?unix=arrow
> >
> > Andy Grove (agrove): Arrow PMC Chair
> > Andrew Lamb (alamb): Arrow PMC, past Arrow PMC Chair
> > Daniël Heres (dheres) Arrow PMC
> > Jie Wen (jakevin): Arrow PMC, Doris Committer
> > Kun Liu (liukun): Arrow PMC, IoTDB PMC, TSFile PMC
> > Liang-Chi Hsieh (viirya): Arrow PMC, Spark PMC
> > Qingping Hou: (houqp): Arrow PMC
> > Wes McKinney(wesm): Arrow PMC, ASF Member
> > Will Jones (wjones127): Arrow PMC
> >
> > We’d like to propose Andrew Lamb as the initial Chair, (and thus ASF
> > VP) for the DataFusion project.
> >
> > Affiliations
> >
> > Andy Grove (agrove): NVidia
> > Andrew Lamb (alamb): InfluxData
> > Daniël Heres (dheres): Coralogix
> > Jie Wen (jakevin): SelectDB
> > Kun Liu (liukun): Ebay
> > Liang-Chi Hsieh (viirya): Apple
> > Qingping Hou: (houqp): Scribd
> > Wes McKinney(wesm): Posit
> > Will Jones (wjones127): LanceDB
> >
> > Proposed Initial Committers
> >
> > In addition to the PMC, we propose the following people as the initial
> > DataFusion committers. This is a subset of the existing Arrow
> > committers who contribute to DataFusion
> > https://people.apache.org/phonebook.html?unix=arrow
> >
> > akurmustafa Mustafa Akur (Synnada)
> > avantgardner Brent Gardner (Coralogix)
> > comphead Oleks V. (Unaffiliated)
> > jayzhan Jay Zhan (Unaffiliated)
> > jeffreyvo Jeffry Vo (Unaffiliated)
> > jiayuliu Liu Jiayu (Airbnb)
> > mete Metehan Yildirim (Synnada)
> > mingmwang Wang Mingming (Ebay)
> > mneumann Marco Neumann (InfluxData)
> > nju_yaho Zhong Yanghong (Ebay)
> > ozankabak Mehmet Ozan Kabak (Synnada)
> > paddyhoran Paddy Horan (Assured Allies)
> > rdettai Rémi Dettai (Cloudfuse)
> > sunchao Chao Sun (Apple)
> > thinkharderdev Daniel Harris (Coralogix)
> > tustvold Raphael Taylor-Davies (InfluxData)
> > wayne Ruihang Xia (Greptime)
> > xudong963 Xudong Wang (ByteDance)
> > yjshen Yijie Shen (Space and Time)
> > yangjiang Yang Jiang (ebay)
> >
> >
> > Risk Assessments
> >
> > Naming / Trademarks
> >
> > As a sub-project of Arrow, the DataFusion name has been used for over
> > 4 years without any known issues. A podling name search did not turn
> > up any concerns and was approved:
> > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> >
> > Legal / IP Clearance
> >
> > All DataFusion code has either been donated to the Arrow project with
> > appropriate IP clearance or has been developed directly under ASF
> > processes and procedures. Thus creating a new top level project poses
> > no new Legal or IP risks.
> >
> > Code Extraction
> >
> > The relevant code is already in 5 separate repositories:
> > https://github.com/apache/arrow-datafusion/
> > https://github.com/apache/arrow-datafusion-python
> > https://github.com/apache/arrow-ballista
> > https://github.com/apache/arrow-ballista-python
> > https://github.com/apache/arrow-datafusion-comet
> >
> > We foresee no issues with code extraction and propose these
> > repositories be renamed to reflect top level projects
> >
> > Note: https://github.com/apache/arrow-rs, the Rust implementation of
> > Arrow, would remain part of the Arrow project.
> >
> > Orphaned Products
> >
> > DataFusion is known to be used in many open source and commercial
> > projects
> https://arrow.apache.org/datafusion/user-guide/introduction.html#known-users
> ,
> > has had multiple commits daily for several years, and its adoption and
> > number of contributors appears to be growing. We do not foresee the
> > project being orphaned in the next several years.
> >
> > Inexperience with Open Source
> >
> > The proposed PMC has extensive experience with Apache Arrow and other
> > Apache projects, and includes PMC members, PMC chairs and an ASF
> > Member. The DataFusion PMC and more experienced committers will
> > continue to coach new community members who may be less familiar with
> > the Apache Way.
> >
> > Homogeneous Developers
> >
> > The 9 proposed PMC members are from 9 different employers and the
> > proposed committers are similarly distributed across affiliations. No
> > specific entity employs more than 3 total proposed developers.
> >
> > Reliance on Salaried Developers
> >
> > A substantial amount of work on DataFusion has been by salaried
> > developers, but it also has a long tradition of attracting
> > contributions from students and hobbyists and we plan no changes in
> > contribution structure.
> >
> > Relationships with Other Apache Products
> >
> > DataFusion will obviously have a strong relationship with the Arrow
> > project given the overlap in people. We don’t foresee close
> > collaboration with other projects at this time.
> >
> > Cryptography
> >
> > DataFusion does not directly support encryption and there are no
> > near-term plans to add support for encryption. Users who need this
> > functionality can use the extension APIs.
> >
> > Required Resources
> >
> > Mailing Lists
> >
> > - private@datafusion.apache.org for private PMC discussions (with
> > moderated subscriptions)
> > - dev@datafusion.apache.org
> > - commits@datafusion.apache.org
> > - user@datafusion.apache.org
> >
> > Version Control
> >
> > We propose to continue to use git for source control and github for
> > hosting and testing resources.
> >
> > We also need to rename the github repositories to reflect the new top
> > level names:
> >
> > https://github.com/apache/arrow-datafusion/ → apache/datafusion
> > https://github.com/apache/arrow-datafusion-python →
> apache/datafusion-python
> > https://github.com/apache/arrow-ballista → apache/datafusion-ballista
> > https://github.com/apache/arrow-ballista-python →
> > apache/datafusion-ballista-python
> > https://github.com/apache/arrow-datafusion-comet →
> apache/datafusion-comet
> >
> >
> >
> > Issue Tracking
> >
> > DataFusion would continue to use github for its issue tracking and
> > communications
> >
> > Other Resources
> >
> > The existing repositories already make use of existing Apache
> > infrastructure, and we expect no change in the initial resource usage.
> > As the project continues to grow, we expect continued infrastructure
> > demand growth.
> >
> >
> > FAQ: Has a sub project been promoted to a top level project before?
> >
> > Yes, and it appears to happen commonly. The Arrow project itself was
> > created as a top level project from work that started in Apache Drill,
> > and there are many sub projects of Hadoop that spun out as their own
> > top level projects such as Mahout, Avro and HBase:
> >
> https://news.apache.org/foundation/entry/the_apache_software_foundation_announces4
> >
> >
> >
> > Related material:
> > Name search request / research for DataFusion:
> > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> > Discussion about this proposal on the arrow mailing list:
> > https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> > Discussion about which repositories on the arrow mailing list:
> > https://lists.apache.org/thread/ob3n0d9ky0bgrryl3xn39w9k566bq00q
> > Discussion about initial PMC on the arrow mailing list:
> > https://lists.apache.org/thread/pymrzcdw4qdptvby85f69rg3pcckl15b
> > Discussion in github about creating a new DataFusion top level
> > project: https://github.com/apache/arrow-datafusion/discussions/6475
> > Discussion about graduating on incubator list:
> > https://lists.apache.org/thread/r4n73pmms1lv0jbohyx1o1z13d615t99
> > Original Proposal for the Arrow project:
> > https://lists.apache.org/thread/x2qzdwglm8pkqp9gv03bbgw17khl7pq3
>
Re: [VOTE] Move Arrow DataFusion Subproject to new Top Level Apache Project
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
+1 (non binding)
Regards
JB
On Fri, Mar 1, 2024 at 12:33 PM Andrew Lamb <al...@influxdata.com> wrote:
>
> Hello,
>
> As we have discussed[1][2] I would like to vote on the proposal to
> create a new Apache Top Level Project for DataFusion. The text of the
> proposed resolution and background document is copy/pasted below
>
> If the community is in favor of this, we plan to submit the resolution
> to the ASF board for approval with the next Arrow report (for the
> April 2024 board meeting).
>
> The vote will be open for at least 7 days.
>
> [ ] +1 Accept this Proposal
> [ ] +0
> [ ] -1 Do not accept this proposal because...
>
> Andrew
>
> [1] https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> [2] https://github.com/apache/arrow-datafusion/discussions/6475
>
> ---------- Proposed Resolution ---------
>
> Resolution to Create the Apache DataFusion Project from the Apache
> Arrow DataFusion Sub Project
>
> =============================================================
>
> X. Establish the Apache DataFusion Project
>
> WHEREAS, the Board of Directors deems it to be in the best
> interests of the Foundation and consistent with the
> Foundation's purpose to establish a Project Management
> Committee charged with the creation and maintenance of
> open-source software related to an extensible query engine
> for distribution at no charge to the public.
>
> NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> Committee (PMC), to be known as the "Apache DataFusion Project",
> be and hereby is established pursuant to Bylaws of the
> Foundation; and be it further
>
> RESOLVED, that the Apache DataFusion Project be and hereby is
> responsible for the creation and maintenance of software
> related to an extensible query engine; and be it further
>
> RESOLVED, that the office of "Vice President, Apache DataFusion" be
> and hereby is created, the person holding such office to
> serve at the direction of the Board of Directors as the chair
> of the Apache DataFusion Project, and to have primary responsibility
> for management of the projects within the scope of
> responsibility of the Apache DataFusion Project; and be it further
>
> RESOLVED, that the persons listed immediately below be and
> hereby are appointed to serve as the initial members of the
> Apache DataFusion Project:
>
> * Andy Grove (agrove@apache.org)
> * Andrew Lamb (alamb@apache.org)
> * Daniël Heres (dheres@apache.org)
> * Jie Wen (jakevin@apache.org)
> * Kun Liu (liukun@apache.org)
> * Liang-Chi Hsieh (viirya@apache.org)
> * Qingping Hou: (houqp@apache.org)
> * Wes McKinney(wesm@apache.org)
> * Will Jones (wjones127@apache.org)
>
> RESOLVED, that the Apache DataFusion Project be and hereby
> is tasked with the migration and rationalization of the Apache
> Arrow DataFusion sub-project; and be it further
>
> RESOLVED, that all responsibilities pertaining to the Apache
> Arrow DataFusion sub-project encumbered upon the
> Apache Arrow Project are hereafter discharged.
>
> NOW, THEREFORE, BE IT FURTHER RESOLVED, that Andrew Lamb
> be appointed to the office of Vice President, Apache DataFusion, to
> serve in accordance with and subject to the direction of the
> Board of Directors and the Bylaws of the Foundation until
> death, resignation, retirement, removal or disqualification,
> or until a successor is appointed.
> =============================================================
>
>
> -------
>
>
> Summary:
>
> We propose creating a new top level project, Apache DataFusion, from
> an existing sub project of Apache Arrow to facilitate additional
> community and project growth.
>
> Abstract
>
> Apache Arrow DataFusion[1] is a very fast, extensible query engine
> for building high-quality data-centric systems in Rust, using the
> Apache Arrow in-memory format. DataFusion offers SQL and Dataframe
> APIs, excellent performance, built-in support for CSV, Parquet, JSON,
> and Avro, extensive customization, and a great community.
>
> [1] https://arrow.apache.org/datafusion/
>
>
> Proposal
>
> We propose creating a new top level ASF project, Apache DataFusion,
> governed initially by a subset of the Apache Arrow project’s PMC and
> committers. The project’s code is in five existing git repositories,
> currently governed by Apache Arrow which would transfer to the new top
> level project.
>
> Background
>
> When DataFusion was initially donated to the Arrow project, it did not
> have a strong enough community to stand on its own. It has since grown
> significantly, and benefited immensely from being part of Arrow and
> nurturing of the Apache Way, and now has a community strong enough to
> stand on its own and that would benefit from focused governance
> attention.
>
> The community has discussed this idea publicly for more than 6 months
> https://github.com/apache/arrow-datafusion/discussions/6475 and
> briefly on the Arrow PMC mailing list
> https://lists.apache.org/thread/thv2jdm6640l6gm88hy8jhk5prjww0cs. As
> of the time of this writing both had exclusively positive reactions.
>
> Several current members of the Arrow PMC are both active contributors
> to DataFusion and understand and believe deeply in the Apache Way, and
> play active governance roles in the Arrow project as PMC members and
> PMC chairs, guiding the community, and releasing software versions.
> With this existing governance experience and structure, the new top
> level project will be able to function well immediately and
> independently.
>
> Overview of DataFusion
>
> Current Status
>
> Meritocracy
>
> DataFusion has been developed as part of Apache Arrow and thus has
> been operating as a meritocracy. Many of the developers of DataFusion
> are Arrow PMC members or committers. The DataFusion project plans to
> continue adding new PMC and committers as the project matures and
> grows.
>
> Community
>
> The DataFusion development team seeks to foster the development and
> user communities. We hope that becoming a separate project will help
> both Arrow and DataFusion communities by being more focused. Focused
> governance will make it easier to grow the community of committers and
> PMC members and make the organization more clear to others.
>
> Alignment
>
> The ASF is a natural host for DataFusion given that it is already the
> home of Arrow, Parquet, and other related distributed system, storage
> and query execution systems.
>
> Project Leadership
>
> Proposed Initial PMC
>
> We propose the following people as the initial DataFusion PMC members.
> This is a subset of the existing Arrow PMC members who contribute to
> DataFusion https://people.apache.org/phonebook.html?unix=arrow
>
> Andy Grove (agrove): Arrow PMC Chair
> Andrew Lamb (alamb): Arrow PMC, past Arrow PMC Chair
> Daniël Heres (dheres) Arrow PMC
> Jie Wen (jakevin): Arrow PMC, Doris Committer
> Kun Liu (liukun): Arrow PMC, IoTDB PMC, TSFile PMC
> Liang-Chi Hsieh (viirya): Arrow PMC, Spark PMC
> Qingping Hou: (houqp): Arrow PMC
> Wes McKinney(wesm): Arrow PMC, ASF Member
> Will Jones (wjones127): Arrow PMC
>
> We’d like to propose Andrew Lamb as the initial Chair, (and thus ASF
> VP) for the DataFusion project.
>
> Affiliations
>
> Andy Grove (agrove): NVidia
> Andrew Lamb (alamb): InfluxData
> Daniël Heres (dheres): Coralogix
> Jie Wen (jakevin): SelectDB
> Kun Liu (liukun): Ebay
> Liang-Chi Hsieh (viirya): Apple
> Qingping Hou: (houqp): Scribd
> Wes McKinney(wesm): Posit
> Will Jones (wjones127): LanceDB
>
> Proposed Initial Committers
>
> In addition to the PMC, we propose the following people as the initial
> DataFusion committers. This is a subset of the existing Arrow
> committers who contribute to DataFusion
> https://people.apache.org/phonebook.html?unix=arrow
>
> akurmustafa Mustafa Akur (Synnada)
> avantgardner Brent Gardner (Coralogix)
> comphead Oleks V. (Unaffiliated)
> jayzhan Jay Zhan (Unaffiliated)
> jeffreyvo Jeffry Vo (Unaffiliated)
> jiayuliu Liu Jiayu (Airbnb)
> mete Metehan Yildirim (Synnada)
> mingmwang Wang Mingming (Ebay)
> mneumann Marco Neumann (InfluxData)
> nju_yaho Zhong Yanghong (Ebay)
> ozankabak Mehmet Ozan Kabak (Synnada)
> paddyhoran Paddy Horan (Assured Allies)
> rdettai Rémi Dettai (Cloudfuse)
> sunchao Chao Sun (Apple)
> thinkharderdev Daniel Harris (Coralogix)
> tustvold Raphael Taylor-Davies (InfluxData)
> wayne Ruihang Xia (Greptime)
> xudong963 Xudong Wang (ByteDance)
> yjshen Yijie Shen (Space and Time)
> yangjiang Yang Jiang (ebay)
>
>
> Risk Assessments
>
> Naming / Trademarks
>
> As a sub-project of Arrow, the DataFusion name has been used for over
> 4 years without any known issues. A podling name search did not turn
> up any concerns and was approved:
> https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
>
> Legal / IP Clearance
>
> All DataFusion code has either been donated to the Arrow project with
> appropriate IP clearance or has been developed directly under ASF
> processes and procedures. Thus creating a new top level project poses
> no new Legal or IP risks.
>
> Code Extraction
>
> The relevant code is already in 5 separate repositories:
> https://github.com/apache/arrow-datafusion/
> https://github.com/apache/arrow-datafusion-python
> https://github.com/apache/arrow-ballista
> https://github.com/apache/arrow-ballista-python
> https://github.com/apache/arrow-datafusion-comet
>
> We foresee no issues with code extraction and propose these
> repositories be renamed to reflect top level projects
>
> Note: https://github.com/apache/arrow-rs, the Rust implementation of
> Arrow, would remain part of the Arrow project.
>
> Orphaned Products
>
> DataFusion is known to be used in many open source and commercial
> projects https://arrow.apache.org/datafusion/user-guide/introduction.html#known-users,
> has had multiple commits daily for several years, and its adoption and
> number of contributors appears to be growing. We do not foresee the
> project being orphaned in the next several years.
>
> Inexperience with Open Source
>
> The proposed PMC has extensive experience with Apache Arrow and other
> Apache projects, and includes PMC members, PMC chairs and an ASF
> Member. The DataFusion PMC and more experienced committers will
> continue to coach new community members who may be less familiar with
> the Apache Way.
>
> Homogeneous Developers
>
> The 9 proposed PMC members are from 9 different employers and the
> proposed committers are similarly distributed across affiliations. No
> specific entity employs more than 3 total proposed developers.
>
> Reliance on Salaried Developers
>
> A substantial amount of work on DataFusion has been by salaried
> developers, but it also has a long tradition of attracting
> contributions from students and hobbyists and we plan no changes in
> contribution structure.
>
> Relationships with Other Apache Products
>
> DataFusion will obviously have a strong relationship with the Arrow
> project given the overlap in people. We don’t foresee close
> collaboration with other projects at this time.
>
> Cryptography
>
> DataFusion does not directly support encryption and there are no
> near-term plans to add support for encryption. Users who need this
> functionality can use the extension APIs.
>
> Required Resources
>
> Mailing Lists
>
> - private@datafusion.apache.org for private PMC discussions (with
> moderated subscriptions)
> - dev@datafusion.apache.org
> - commits@datafusion.apache.org
> - user@datafusion.apache.org
>
> Version Control
>
> We propose to continue to use git for source control and github for
> hosting and testing resources.
>
> We also need to rename the github repositories to reflect the new top
> level names:
>
> https://github.com/apache/arrow-datafusion/ → apache/datafusion
> https://github.com/apache/arrow-datafusion-python → apache/datafusion-python
> https://github.com/apache/arrow-ballista → apache/datafusion-ballista
> https://github.com/apache/arrow-ballista-python →
> apache/datafusion-ballista-python
> https://github.com/apache/arrow-datafusion-comet → apache/datafusion-comet
>
>
>
> Issue Tracking
>
> DataFusion would continue to use github for its issue tracking and
> communications
>
> Other Resources
>
> The existing repositories already make use of existing Apache
> infrastructure, and we expect no change in the initial resource usage.
> As the project continues to grow, we expect continued infrastructure
> demand growth.
>
>
> FAQ: Has a sub project been promoted to a top level project before?
>
> Yes, and it appears to happen commonly. The Arrow project itself was
> created as a top level project from work that started in Apache Drill,
> and there are many sub projects of Hadoop that spun out as their own
> top level projects such as Mahout, Avro and HBase:
> https://news.apache.org/foundation/entry/the_apache_software_foundation_announces4
>
>
>
> Related material:
> Name search request / research for DataFusion:
> https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> Discussion about this proposal on the arrow mailing list:
> https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> Discussion about which repositories on the arrow mailing list:
> https://lists.apache.org/thread/ob3n0d9ky0bgrryl3xn39w9k566bq00q
> Discussion about initial PMC on the arrow mailing list:
> https://lists.apache.org/thread/pymrzcdw4qdptvby85f69rg3pcckl15b
> Discussion in github about creating a new DataFusion top level
> project: https://github.com/apache/arrow-datafusion/discussions/6475
> Discussion about graduating on incubator list:
> https://lists.apache.org/thread/r4n73pmms1lv0jbohyx1o1z13d615t99
> Original Proposal for the Arrow project:
> https://lists.apache.org/thread/x2qzdwglm8pkqp9gv03bbgw17khl7pq3
Re: [VOTE] Move Arrow DataFusion Subproject to new Top Level Apache Project
Posted by Matt Topol <zo...@gmail.com>.
+1 (binding)
On Fri, Mar 1, 2024, 12:58 PM QP Hou <qp...@neuralink.com> wrote:
> +1 (binding)
>
> exciting milestone :)
>
> On Fri, Mar 1, 2024 at 9:49 AM David Li <li...@apache.org> wrote:
> >
> > +1
> >
> > On Fri, Mar 1, 2024, at 12:06, Jorge Cardoso Leitão wrote:
> > > +1 - great work!!!
> > >
> > > On Fri, Mar 1, 2024 at 5:49 PM Micah Kornfield <em...@gmail.com>
> > > wrote:
> > >
> > >> +1 (binding)
> > >>
> > >> On Friday, March 1, 2024, Uwe L. Korn <uw...@xhochy.com> wrote:
> > >>
> > >> > +1 (binding)
> > >> >
> > >> > On Fri, Mar 1, 2024, at 2:37 PM, Andy Grove wrote:
> > >> > > +1 (binding)
> > >> > >
> > >> > > On Fri, Mar 1, 2024 at 6:20 AM Weston Pace <weston.pace@gmail.com
> >
> > >> > wrote:
> > >> > >
> > >> > >> +1 (binding)
> > >> > >>
> > >> > >> On Fri, Mar 1, 2024 at 3:33 AM Andrew Lamb <alamb@influxdata.com
> >
> > >> > wrote:
> > >> > >>
> > >> > >> > Hello,
> > >> > >> >
> > >> > >> > As we have discussed[1][2] I would like to vote on the
> proposal to
> > >> > >> > create a new Apache Top Level Project for DataFusion. The text
> of
> > >> the
> > >> > >> > proposed resolution and background document is copy/pasted
> below
> > >> > >> >
> > >> > >> > If the community is in favor of this, we plan to submit the
> > >> resolution
> > >> > >> > to the ASF board for approval with the next Arrow report (for
> the
> > >> > >> > April 2024 board meeting).
> > >> > >> >
> > >> > >> > The vote will be open for at least 7 days.
> > >> > >> >
> > >> > >> > [ ] +1 Accept this Proposal
> > >> > >> > [ ] +0
> > >> > >> > [ ] -1 Do not accept this proposal because...
> > >> > >> >
> > >> > >> > Andrew
> > >> > >> >
> > >> > >> > [1]
> > >> https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> > >> > >> > [2]
> https://github.com/apache/arrow-datafusion/discussions/6475
> > >> > >> >
> > >> > >> > ---------- Proposed Resolution ---------
> > >> > >> >
> > >> > >> > Resolution to Create the Apache DataFusion Project from the
> Apache
> > >> > >> > Arrow DataFusion Sub Project
> > >> > >> >
> > >> > >> > =============================================================
> > >> > >> >
> > >> > >> > X. Establish the Apache DataFusion Project
> > >> > >> >
> > >> > >> > WHEREAS, the Board of Directors deems it to be in the best
> > >> > >> > interests of the Foundation and consistent with the
> > >> > >> > Foundation's purpose to establish a Project Management
> > >> > >> > Committee charged with the creation and maintenance of
> > >> > >> > open-source software related to an extensible query engine
> > >> > >> > for distribution at no charge to the public.
> > >> > >> >
> > >> > >> > NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> > >> > >> > Committee (PMC), to be known as the "Apache DataFusion
> Project",
> > >> > >> > be and hereby is established pursuant to Bylaws of the
> > >> > >> > Foundation; and be it further
> > >> > >> >
> > >> > >> > RESOLVED, that the Apache DataFusion Project be and hereby is
> > >> > >> > responsible for the creation and maintenance of software
> > >> > >> > related to an extensible query engine; and be it further
> > >> > >> >
> > >> > >> > RESOLVED, that the office of "Vice President, Apache
> DataFusion" be
> > >> > >> > and hereby is created, the person holding such office to
> > >> > >> > serve at the direction of the Board of Directors as the chair
> > >> > >> > of the Apache DataFusion Project, and to have primary
> responsibility
> > >> > >> > for management of the projects within the scope of
> > >> > >> > responsibility of the Apache DataFusion Project; and be it
> further
> > >> > >> >
> > >> > >> > RESOLVED, that the persons listed immediately below be and
> > >> > >> > hereby are appointed to serve as the initial members of the
> > >> > >> > Apache DataFusion Project:
> > >> > >> >
> > >> > >> > * Andy Grove (agrove@apache.org)
> > >> > >> > * Andrew Lamb (alamb@apache.org)
> > >> > >> > * Daniël Heres (dheres@apache.org)
> > >> > >> > * Jie Wen (jakevin@apache.org)
> > >> > >> > * Kun Liu (liukun@apache.org)
> > >> > >> > * Liang-Chi Hsieh (viirya@apache.org)
> > >> > >> > * Qingping Hou: (houqp@apache.org)
> > >> > >> > * Wes McKinney(wesm@apache.org)
> > >> > >> > * Will Jones (wjones127@apache.org)
> > >> > >> >
> > >> > >> > RESOLVED, that the Apache DataFusion Project be and hereby
> > >> > >> > is tasked with the migration and rationalization of the Apache
> > >> > >> > Arrow DataFusion sub-project; and be it further
> > >> > >> >
> > >> > >> > RESOLVED, that all responsibilities pertaining to the Apache
> > >> > >> > Arrow DataFusion sub-project encumbered upon the
> > >> > >> > Apache Arrow Project are hereafter discharged.
> > >> > >> >
> > >> > >> > NOW, THEREFORE, BE IT FURTHER RESOLVED, that Andrew Lamb
> > >> > >> > be appointed to the office of Vice President, Apache
> DataFusion, to
> > >> > >> > serve in accordance with and subject to the direction of the
> > >> > >> > Board of Directors and the Bylaws of the Foundation until
> > >> > >> > death, resignation, retirement, removal or disqualification,
> > >> > >> > or until a successor is appointed.
> > >> > >> > =============================================================
> > >> > >> >
> > >> > >> >
> > >> > >> > -------
> > >> > >> >
> > >> > >> >
> > >> > >> > Summary:
> > >> > >> >
> > >> > >> > We propose creating a new top level project, Apache
> DataFusion, from
> > >> > >> > an existing sub project of Apache Arrow to facilitate
> additional
> > >> > >> > community and project growth.
> > >> > >> >
> > >> > >> > Abstract
> > >> > >> >
> > >> > >> > Apache Arrow DataFusion[1] is a very fast, extensible query
> engine
> > >> > >> > for building high-quality data-centric systems in Rust, using
> the
> > >> > >> > Apache Arrow in-memory format. DataFusion offers SQL and
> Dataframe
> > >> > >> > APIs, excellent performance, built-in support for CSV, Parquet,
> > >> JSON,
> > >> > >> > and Avro, extensive customization, and a great community.
> > >> > >> >
> > >> > >> > [1] https://arrow.apache.org/datafusion/
> > >> > >> >
> > >> > >> >
> > >> > >> > Proposal
> > >> > >> >
> > >> > >> > We propose creating a new top level ASF project, Apache
> DataFusion,
> > >> > >> > governed initially by a subset of the Apache Arrow project’s
> PMC and
> > >> > >> > committers. The project’s code is in five existing git
> repositories,
> > >> > >> > currently governed by Apache Arrow which would transfer to the
> new
> > >> top
> > >> > >> > level project.
> > >> > >> >
> > >> > >> > Background
> > >> > >> >
> > >> > >> > When DataFusion was initially donated to the Arrow project, it
> did
> > >> not
> > >> > >> > have a strong enough community to stand on its own. It has
> since
> > >> grown
> > >> > >> > significantly, and benefited immensely from being part of
> Arrow and
> > >> > >> > nurturing of the Apache Way, and now has a community strong
> enough
> > >> to
> > >> > >> > stand on its own and that would benefit from focused governance
> > >> > >> > attention.
> > >> > >> >
> > >> > >> > The community has discussed this idea publicly for more than 6
> > >> months
> > >> > >> > https://github.com/apache/arrow-datafusion/discussions/6475
> and
> > >> > >> > briefly on the Arrow PMC mailing list
> > >> > >> >
> https://lists.apache.org/thread/thv2jdm6640l6gm88hy8jhk5prjww0cs.
> > >> As
> > >> > >> > of the time of this writing both had exclusively positive
> reactions.
> > >> > >> >
> > >> > >> > Several current members of the Arrow PMC are both active
> > >> contributors
> > >> > >> > to DataFusion and understand and believe deeply in the Apache
> Way,
> > >> and
> > >> > >> > play active governance roles in the Arrow project as PMC
> members and
> > >> > >> > PMC chairs, guiding the community, and releasing software
> versions.
> > >> > >> > With this existing governance experience and structure, the
> new top
> > >> > >> > level project will be able to function well immediately and
> > >> > >> > independently.
> > >> > >> >
> > >> > >> > Overview of DataFusion
> > >> > >> >
> > >> > >> > Current Status
> > >> > >> >
> > >> > >> > Meritocracy
> > >> > >> >
> > >> > >> > DataFusion has been developed as part of Apache Arrow and thus
> has
> > >> > >> > been operating as a meritocracy. Many of the developers of
> > >> DataFusion
> > >> > >> > are Arrow PMC members or committers. The DataFusion project
> plans to
> > >> > >> > continue adding new PMC and committers as the project matures
> and
> > >> > >> > grows.
> > >> > >> >
> > >> > >> > Community
> > >> > >> >
> > >> > >> > The DataFusion development team seeks to foster the
> development and
> > >> > >> > user communities. We hope that becoming a separate project
> will help
> > >> > >> > both Arrow and DataFusion communities by being more focused.
> > >> Focused
> > >> > >> > governance will make it easier to grow the community of
> committers
> > >> and
> > >> > >> > PMC members and make the organization more clear to others.
> > >> > >> >
> > >> > >> > Alignment
> > >> > >> >
> > >> > >> > The ASF is a natural host for DataFusion given that it is
> already
> > >> the
> > >> > >> > home of Arrow, Parquet, and other related distributed system,
> > >> storage
> > >> > >> > and query execution systems.
> > >> > >> >
> > >> > >> > Project Leadership
> > >> > >> >
> > >> > >> > Proposed Initial PMC
> > >> > >> >
> > >> > >> > We propose the following people as the initial DataFusion PMC
> > >> members.
> > >> > >> > This is a subset of the existing Arrow PMC members who
> contribute to
> > >> > >> > DataFusion https://people.apache.org/phonebook.html?unix=arrow
> > >> > >> >
> > >> > >> > Andy Grove (agrove): Arrow PMC Chair
> > >> > >> > Andrew Lamb (alamb): Arrow PMC, past Arrow PMC Chair
> > >> > >> > Daniël Heres (dheres) Arrow PMC
> > >> > >> > Jie Wen (jakevin): Arrow PMC, Doris Committer
> > >> > >> > Kun Liu (liukun): Arrow PMC, IoTDB PMC, TSFile PMC
> > >> > >> > Liang-Chi Hsieh (viirya): Arrow PMC, Spark PMC
> > >> > >> > Qingping Hou: (houqp): Arrow PMC
> > >> > >> > Wes McKinney(wesm): Arrow PMC, ASF Member
> > >> > >> > Will Jones (wjones127): Arrow PMC
> > >> > >> >
> > >> > >> > We’d like to propose Andrew Lamb as the initial Chair, (and
> thus ASF
> > >> > >> > VP) for the DataFusion project.
> > >> > >> >
> > >> > >> > Affiliations
> > >> > >> >
> > >> > >> > Andy Grove (agrove): NVidia
> > >> > >> > Andrew Lamb (alamb): InfluxData
> > >> > >> > Daniël Heres (dheres): Coralogix
> > >> > >> > Jie Wen (jakevin): SelectDB
> > >> > >> > Kun Liu (liukun): Ebay
> > >> > >> > Liang-Chi Hsieh (viirya): Apple
> > >> > >> > Qingping Hou: (houqp): Scribd
> > >> > >> > Wes McKinney(wesm): Posit
> > >> > >> > Will Jones (wjones127): LanceDB
> > >> > >> >
> > >> > >> > Proposed Initial Committers
> > >> > >> >
> > >> > >> > In addition to the PMC, we propose the following people as the
> > >> initial
> > >> > >> > DataFusion committers. This is a subset of the existing Arrow
> > >> > >> > committers who contribute to DataFusion
> > >> > >> > https://people.apache.org/phonebook.html?unix=arrow
> > >> > >> >
> > >> > >> > akurmustafa Mustafa Akur (Synnada)
> > >> > >> > avantgardner Brent Gardner (Coralogix)
> > >> > >> > comphead Oleks V. (Unaffiliated)
> > >> > >> > jayzhan Jay Zhan (Unaffiliated)
> > >> > >> > jeffreyvo Jeffry Vo (Unaffiliated)
> > >> > >> > jiayuliu Liu Jiayu (Airbnb)
> > >> > >> > mete Metehan Yildirim (Synnada)
> > >> > >> > mingmwang Wang Mingming (Ebay)
> > >> > >> > mneumann Marco Neumann (InfluxData)
> > >> > >> > nju_yaho Zhong Yanghong (Ebay)
> > >> > >> > ozankabak Mehmet Ozan Kabak (Synnada)
> > >> > >> > paddyhoran Paddy Horan (Assured Allies)
> > >> > >> > rdettai Rémi Dettai (Cloudfuse)
> > >> > >> > sunchao Chao Sun (Apple)
> > >> > >> > thinkharderdev Daniel Harris (Coralogix)
> > >> > >> > tustvold Raphael Taylor-Davies (InfluxData)
> > >> > >> > wayne Ruihang Xia (Greptime)
> > >> > >> > xudong963 Xudong Wang (ByteDance)
> > >> > >> > yjshen Yijie Shen (Space and Time)
> > >> > >> > yangjiang Yang Jiang (ebay)
> > >> > >> >
> > >> > >> >
> > >> > >> > Risk Assessments
> > >> > >> >
> > >> > >> > Naming / Trademarks
> > >> > >> >
> > >> > >> > As a sub-project of Arrow, the DataFusion name has been used
> for
> > >> over
> > >> > >> > 4 years without any known issues. A podling name search did
> not turn
> > >> > >> > up any concerns and was approved:
> > >> > >> > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> > >> > >> >
> > >> > >> > Legal / IP Clearance
> > >> > >> >
> > >> > >> > All DataFusion code has either been donated to the Arrow
> project
> > >> with
> > >> > >> > appropriate IP clearance or has been developed directly under
> ASF
> > >> > >> > processes and procedures. Thus creating a new top level project
> > >> poses
> > >> > >> > no new Legal or IP risks.
> > >> > >> >
> > >> > >> > Code Extraction
> > >> > >> >
> > >> > >> > The relevant code is already in 5 separate repositories:
> > >> > >> > https://github.com/apache/arrow-datafusion/
> > >> > >> > https://github.com/apache/arrow-datafusion-python
> > >> > >> > https://github.com/apache/arrow-ballista
> > >> > >> > https://github.com/apache/arrow-ballista-python
> > >> > >> > https://github.com/apache/arrow-datafusion-comet
> > >> > >> >
> > >> > >> > We foresee no issues with code extraction and propose these
> > >> > >> > repositories be renamed to reflect top level projects
> > >> > >> >
> > >> > >> > Note: https://github.com/apache/arrow-rs, the Rust
> implementation
> > >> of
> > >> > >> > Arrow, would remain part of the Arrow project.
> > >> > >> >
> > >> > >> > Orphaned Products
> > >> > >> >
> > >> > >> > DataFusion is known to be used in many open source and
> commercial
> > >> > >> > projects
> > >> > >> >
> > >> > >> https://arrow.apache.org/datafusion/user-guide/
> > >> > introduction.html#known-users
> > >> > >> > ,
> > >> > >> > has had multiple commits daily for several years, and its
> adoption
> > >> and
> > >> > >> > number of contributors appears to be growing. We do not
> foresee the
> > >> > >> > project being orphaned in the next several years.
> > >> > >> >
> > >> > >> > Inexperience with Open Source
> > >> > >> >
> > >> > >> > The proposed PMC has extensive experience with Apache Arrow and
> > >> other
> > >> > >> > Apache projects, and includes PMC members, PMC chairs and an
> ASF
> > >> > >> > Member. The DataFusion PMC and more experienced committers will
> > >> > >> > continue to coach new community members who may be less
> familiar
> > >> with
> > >> > >> > the Apache Way.
> > >> > >> >
> > >> > >> > Homogeneous Developers
> > >> > >> >
> > >> > >> > The 9 proposed PMC members are from 9 different employers and
> the
> > >> > >> > proposed committers are similarly distributed across
> affiliations.
> > >> No
> > >> > >> > specific entity employs more than 3 total proposed developers.
> > >> > >> >
> > >> > >> > Reliance on Salaried Developers
> > >> > >> >
> > >> > >> > A substantial amount of work on DataFusion has been by salaried
> > >> > >> > developers, but it also has a long tradition of attracting
> > >> > >> > contributions from students and hobbyists and we plan no
> changes in
> > >> > >> > contribution structure.
> > >> > >> >
> > >> > >> > Relationships with Other Apache Products
> > >> > >> >
> > >> > >> > DataFusion will obviously have a strong relationship with the
> Arrow
> > >> > >> > project given the overlap in people. We don’t foresee close
> > >> > >> > collaboration with other projects at this time.
> > >> > >> >
> > >> > >> > Cryptography
> > >> > >> >
> > >> > >> > DataFusion does not directly support encryption and there are
> no
> > >> > >> > near-term plans to add support for encryption. Users who need
> this
> > >> > >> > functionality can use the extension APIs.
> > >> > >> >
> > >> > >> > Required Resources
> > >> > >> >
> > >> > >> > Mailing Lists
> > >> > >> >
> > >> > >> > - private@datafusion.apache.org for private PMC discussions
> (with
> > >> > >> > moderated subscriptions)
> > >> > >> > - dev@datafusion.apache.org
> > >> > >> > - commits@datafusion.apache.org
> > >> > >> > - user@datafusion.apache.org
> > >> > >> >
> > >> > >> > Version Control
> > >> > >> >
> > >> > >> > We propose to continue to use git for source control and
> github for
> > >> > >> > hosting and testing resources.
> > >> > >> >
> > >> > >> > We also need to rename the github repositories to reflect the
> new
> > >> top
> > >> > >> > level names:
> > >> > >> >
> > >> > >> > https://github.com/apache/arrow-datafusion/ →
> apache/datafusion
> > >> > >> > https://github.com/apache/arrow-datafusion-python →
> > >> > >> > apache/datafusion-python
> > >> > >> > https://github.com/apache/arrow-ballista →
> > >> apache/datafusion-ballista
> > >> > >> > https://github.com/apache/arrow-ballista-python →
> > >> > >> > apache/datafusion-ballista-python
> > >> > >> > https://github.com/apache/arrow-datafusion-comet →
> > >> > >> apache/datafusion-comet
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> > Issue Tracking
> > >> > >> >
> > >> > >> > DataFusion would continue to use github for its issue tracking
> and
> > >> > >> > communications
> > >> > >> >
> > >> > >> > Other Resources
> > >> > >> >
> > >> > >> > The existing repositories already make use of existing Apache
> > >> > >> > infrastructure, and we expect no change in the initial resource
> > >> usage.
> > >> > >> > As the project continues to grow, we expect continued
> infrastructure
> > >> > >> > demand growth.
> > >> > >> >
> > >> > >> >
> > >> > >> > FAQ: Has a sub project been promoted to a top level project
> before?
> > >> > >> >
> > >> > >> > Yes, and it appears to happen commonly. The Arrow project
> itself was
> > >> > >> > created as a top level project from work that started in Apache
> > >> Drill,
> > >> > >> > and there are many sub projects of Hadoop that spun out as
> their own
> > >> > >> > top level projects such as Mahout, Avro and HBase:
> > >> > >> >
> > >> > >> >
> > >> > >> https://news.apache.org/foundation/entry/the_apache_
> > >> > software_foundation_announces4
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> > Related material:
> > >> > >> > Name search request / research for DataFusion:
> > >> > >> > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> > >> > >> > Discussion about this proposal on the arrow mailing list:
> > >> > >> >
> https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> > >> > >> > Discussion about which repositories on the arrow mailing list:
> > >> > >> >
> https://lists.apache.org/thread/ob3n0d9ky0bgrryl3xn39w9k566bq00q
> > >> > >> > Discussion about initial PMC on the arrow mailing list:
> > >> > >> >
> https://lists.apache.org/thread/pymrzcdw4qdptvby85f69rg3pcckl15b
> > >> > >> > Discussion in github about creating a new DataFusion top level
> > >> > >> > project:
> > >> https://github.com/apache/arrow-datafusion/discussions/6475
> > >> > >> > Discussion about graduating on incubator list:
> > >> > >> >
> https://lists.apache.org/thread/r4n73pmms1lv0jbohyx1o1z13d615t99
> > >> > >> > Original Proposal for the Arrow project:
> > >> > >> >
> https://lists.apache.org/thread/x2qzdwglm8pkqp9gv03bbgw17khl7pq3
> > >> > >> >
> > >> > >>
> > >> >
> > >>
>
Re: [RESULT] [VOTE] Move Arrow DataFusion Subproject to new Top Level Apache Project
Posted by Andrew Lamb <al...@influxdata.com>.
As a follow up to this thread, and several others on this mailing list, I
am pleased to announce that the proposal to create the DataFusion Top Level
Project passed unanimously at the April 2023 ASF board meeting.
Thank you to everyone in the Arrow community who has helped nurture this
subproject over the years. It has truly been an amazing experience to see
how it is grown and now "graduated" to its own top level project.
We are tracking subtasks for the transition on [1]. Please feel free to
suggest additional items or comments there
Andrew
[1]: https://github.com/apache/arrow-datafusion/issues/9691
On Sat, Mar 9, 2024 at 3:23 AM Andrew Lamb <al...@influxdata.com> wrote:
> With 30 +1 votes (18 binding by my count), and 0 -1 votes, the proposal is
> approved
>
> Thank you everyone who voted and participated in helping form the proposal
> (and the community over the last few years). I think there are exciting
> times ahead for DataFusion.
>
> We will submit the proposed motion to the ASF board with our next
> quarterly report, for the April 2024 meeting, as previously discussed.
>
> Thanks again
> Andrew
>
>
>
> On Tue, Mar 5, 2024 at 2:00 PM Benjamin Kietzman <be...@gmail.com>
> wrote:
>
>> +1 (binding)
>>
>> On Tue, Mar 5, 2024, 13:03 Peter Toth <pe...@gmail.com> wrote:
>>
>> > +1 (non-binding)
>> >
>> > Parth Chandra <pa...@apache.org> ezt írta (időpont: 2024. márc. 5., K,
>> > 18:15):
>> >
>> > > +1 (non-binding)
>> > >
>> > > On Sun, Mar 3, 2024 at 8:04 PM Mehmet Ozan Kabak <oz...@synnada.ai>
>> > wrote:
>> > >
>> > > > +1 (non-binding)
>> > > > --
>> > > > *Mehmet Ozan Kabak*
>> > > > Co-founder & CEO @ Synnada, Inc.
>> > > >
>> > > >
>> > > > On Sun, Mar 3, 2024 at 7:33 PM Jacob Wujciak-Jens
>> > > > <ja...@voltrondata.com.invalid> wrote:
>> > > >
>> > > > > +1 (non-binding)
>> > > > >
>> > > > > On Mon, Mar 4, 2024 at 3:39 AM Yang Jiang <ya...@apache.org>
>> > > wrote:
>> > > > >
>> > > > > > +1 (non-binding)
>> > > > > >
>> > > > > > On 2024/03/01 18:08:26 Daniël Heres wrote:
>> > > > > > > +1 (binding)
>> > > > > > >
>> > > > > > > On Fri, Mar 1, 2024, 19:05 Chao Sun <su...@apache.org>
>> wrote:
>> > > > > > >
>> > > > > > > > +1 (non-binding)
>> > > > > > > >
>> > > > > > > > On Fri, Mar 1, 2024 at 9:59 AM QP Hou <qp...@neuralink.com>
>> > wrote:
>> > > > > > > >
>> > > > > > > > > +1 (binding)
>> > > > > > > > >
>> > > > > > > > > exciting milestone :)
>> > > > > > > > >
>> > > > > > > > > On Fri, Mar 1, 2024 at 9:49 AM David Li <
>> lidavidm@apache.org
>> > >
>> > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > +1
>> > > > > > > > > >
>> > > > > > > > > > On Fri, Mar 1, 2024, at 12:06, Jorge Cardoso Leitão
>> wrote:
>> > > > > > > > > > > +1 - great work!!!
>> > > > > > > > > > >
>> > > > > > > > > > > On Fri, Mar 1, 2024 at 5:49 PM Micah Kornfield <
>> > > > > > > > emkornfield@gmail.com>
>> > > > > > > > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > >> +1 (binding)
>> > > > > > > > > > >>
>> > > > > > > > > > >> On Friday, March 1, 2024, Uwe L. Korn <
>> uwelk@xhochy.com
>> > >
>> > > > > wrote:
>> > > > > > > > > > >>
>> > > > > > > > > > >> > +1 (binding)
>> > > > > > > > > > >> >
>> > > > > > > > > > >> > On Fri, Mar 1, 2024, at 2:37 PM, Andy Grove wrote:
>> > > > > > > > > > >> > > +1 (binding)
>> > > > > > > > > > >> > >
>> > > > > > > > > > >> > > On Fri, Mar 1, 2024 at 6:20 AM Weston Pace <
>> > > > > > > > weston.pace@gmail.com
>> > > > > > > > > >
>> > > > > > > > > > >> > wrote:
>> > > > > > > > > > >> > >
>> > > > > > > > > > >> > >> +1 (binding)
>> > > > > > > > > > >> > >>
>> > > > > > > > > > >> > >> On Fri, Mar 1, 2024 at 3:33 AM Andrew Lamb <
>> > > > > > > > alamb@influxdata.com
>> > > > > > > > > >
>> > > > > > > > > > >> > wrote:
>> > > > > > > > > > >> > >>
>> > > > > > > > > > >> > >> > Hello,
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > As we have discussed[1][2] I would like to
>> vote
>> > on
>> > > > the
>> > > > > > > > > proposal to
>> > > > > > > > > > >> > >> > create a new Apache Top Level Project for
>> > > DataFusion.
>> > > > > The
>> > > > > > > > text
>> > > > > > > > > of
>> > > > > > > > > > >> the
>> > > > > > > > > > >> > >> > proposed resolution and background document is
>> > > > > > copy/pasted
>> > > > > > > > > below
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > If the community is in favor of this, we plan
>> to
>> > > > submit
>> > > > > > the
>> > > > > > > > > > >> resolution
>> > > > > > > > > > >> > >> > to the ASF board for approval with the next
>> Arrow
>> > > > > report
>> > > > > > (for
>> > > > > > > > > the
>> > > > > > > > > > >> > >> > April 2024 board meeting).
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > The vote will be open for at least 7 days.
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > [ ] +1 Accept this Proposal
>> > > > > > > > > > >> > >> > [ ] +0
>> > > > > > > > > > >> > >> > [ ] -1 Do not accept this proposal because...
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > Andrew
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > [1]
>> > > > > > > > > > >>
>> > > > > >
>> https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
>> > > > > > > > > > >> > >> > [2]
>> > > > > > > > >
>> https://github.com/apache/arrow-datafusion/discussions/6475
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > ---------- Proposed Resolution ---------
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > Resolution to Create the Apache DataFusion
>> > Project
>> > > > from
>> > > > > > the
>> > > > > > > > > Apache
>> > > > > > > > > > >> > >> > Arrow DataFusion Sub Project
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> >
>> > > > > > =============================================================
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > X. Establish the Apache DataFusion Project
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > WHEREAS, the Board of Directors deems it to
>> be in
>> > > the
>> > > > > > best
>> > > > > > > > > > >> > >> > interests of the Foundation and consistent
>> with
>> > the
>> > > > > > > > > > >> > >> > Foundation's purpose to establish a Project
>> > > > Management
>> > > > > > > > > > >> > >> > Committee charged with the creation and
>> > maintenance
>> > > > of
>> > > > > > > > > > >> > >> > open-source software related to an extensible
>> > query
>> > > > > > engine
>> > > > > > > > > > >> > >> > for distribution at no charge to the public.
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > NOW, THEREFORE, BE IT RESOLVED, that a Project
>> > > > > Management
>> > > > > > > > > > >> > >> > Committee (PMC), to be known as the "Apache
>> > > > DataFusion
>> > > > > > > > > Project",
>> > > > > > > > > > >> > >> > be and hereby is established pursuant to
>> Bylaws
>> > of
>> > > > the
>> > > > > > > > > > >> > >> > Foundation; and be it further
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > RESOLVED, that the Apache DataFusion Project
>> be
>> > and
>> > > > > > hereby is
>> > > > > > > > > > >> > >> > responsible for the creation and maintenance
>> of
>> > > > > software
>> > > > > > > > > > >> > >> > related to an extensible query engine; and be
>> it
>> > > > > further
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > RESOLVED, that the office of "Vice President,
>> > > Apache
>> > > > > > > > > DataFusion" be
>> > > > > > > > > > >> > >> > and hereby is created, the person holding such
>> > > office
>> > > > > to
>> > > > > > > > > > >> > >> > serve at the direction of the Board of
>> Directors
>> > as
>> > > > the
>> > > > > > chair
>> > > > > > > > > > >> > >> > of the Apache DataFusion Project, and to have
>> > > primary
>> > > > > > > > > responsibility
>> > > > > > > > > > >> > >> > for management of the projects within the
>> scope
>> > of
>> > > > > > > > > > >> > >> > responsibility of the Apache DataFusion
>> Project;
>> > > and
>> > > > be
>> > > > > > it
>> > > > > > > > > further
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > RESOLVED, that the persons listed immediately
>> > below
>> > > > be
>> > > > > > and
>> > > > > > > > > > >> > >> > hereby are appointed to serve as the initial
>> > > members
>> > > > of
>> > > > > > the
>> > > > > > > > > > >> > >> > Apache DataFusion Project:
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > * Andy Grove (agrove@apache.org)
>> > > > > > > > > > >> > >> > * Andrew Lamb (alamb@apache.org)
>> > > > > > > > > > >> > >> > * Daniël Heres (dheres@apache.org)
>> > > > > > > > > > >> > >> > * Jie Wen (jakevin@apache.org)
>> > > > > > > > > > >> > >> > * Kun Liu (liukun@apache.org)
>> > > > > > > > > > >> > >> > * Liang-Chi Hsieh (viirya@apache.org)
>> > > > > > > > > > >> > >> > * Qingping Hou: (houqp@apache.org)
>> > > > > > > > > > >> > >> > * Wes McKinney(wesm@apache.org)
>> > > > > > > > > > >> > >> > * Will Jones (wjones127@apache.org)
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > RESOLVED, that the Apache DataFusion Project
>> be
>> > and
>> > > > > > hereby
>> > > > > > > > > > >> > >> > is tasked with the migration and
>> rationalization
>> > of
>> > > > the
>> > > > > > > > Apache
>> > > > > > > > > > >> > >> > Arrow DataFusion sub-project; and be it
>> further
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > RESOLVED, that all responsibilities
>> pertaining to
>> > > the
>> > > > > > Apache
>> > > > > > > > > > >> > >> > Arrow DataFusion sub-project encumbered upon
>> the
>> > > > > > > > > > >> > >> > Apache Arrow Project are hereafter discharged.
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > NOW, THEREFORE, BE IT FURTHER RESOLVED, that
>> > Andrew
>> > > > > Lamb
>> > > > > > > > > > >> > >> > be appointed to the office of Vice President,
>> > > Apache
>> > > > > > > > > DataFusion, to
>> > > > > > > > > > >> > >> > serve in accordance with and subject to the
>> > > direction
>> > > > > of
>> > > > > > the
>> > > > > > > > > > >> > >> > Board of Directors and the Bylaws of the
>> > Foundation
>> > > > > until
>> > > > > > > > > > >> > >> > death, resignation, retirement, removal or
>> > > > > > disqualification,
>> > > > > > > > > > >> > >> > or until a successor is appointed.
>> > > > > > > > > > >> > >> >
>> > > > > > =============================================================
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > -------
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > Summary:
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > We propose creating a new top level project,
>> > Apache
>> > > > > > > > > DataFusion, from
>> > > > > > > > > > >> > >> > an existing sub project of Apache Arrow to
>> > > facilitate
>> > > > > > > > > additional
>> > > > > > > > > > >> > >> > community and project growth.
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > Abstract
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > Apache Arrow DataFusion[1] is a very fast,
>> > > > extensible
>> > > > > > query
>> > > > > > > > > engine
>> > > > > > > > > > >> > >> > for building high-quality data-centric
>> systems in
>> > > > Rust,
>> > > > > > using
>> > > > > > > > > the
>> > > > > > > > > > >> > >> > Apache Arrow in-memory format. DataFusion
>> offers
>> > > SQL
>> > > > > and
>> > > > > > > > > Dataframe
>> > > > > > > > > > >> > >> > APIs, excellent performance, built-in support
>> for
>> > > > CSV,
>> > > > > > > > Parquet,
>> > > > > > > > > > >> JSON,
>> > > > > > > > > > >> > >> > and Avro, extensive customization, and a great
>> > > > > community.
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > [1] https://arrow.apache.org/datafusion/
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > Proposal
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > We propose creating a new top level ASF
>> project,
>> > > > Apache
>> > > > > > > > > DataFusion,
>> > > > > > > > > > >> > >> > governed initially by a subset of the Apache
>> > Arrow
>> > > > > > project’s
>> > > > > > > > > PMC and
>> > > > > > > > > > >> > >> > committers. The project’s code is in five
>> > existing
>> > > > git
>> > > > > > > > > repositories,
>> > > > > > > > > > >> > >> > currently governed by Apache Arrow which would
>> > > > transfer
>> > > > > > to
>> > > > > > > > the
>> > > > > > > > > new
>> > > > > > > > > > >> top
>> > > > > > > > > > >> > >> > level project.
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > Background
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > When DataFusion was initially donated to the
>> > Arrow
>> > > > > > project,
>> > > > > > > > it
>> > > > > > > > > did
>> > > > > > > > > > >> not
>> > > > > > > > > > >> > >> > have a strong enough community to stand on its
>> > own.
>> > > > It
>> > > > > > has
>> > > > > > > > > since
>> > > > > > > > > > >> grown
>> > > > > > > > > > >> > >> > significantly, and benefited immensely from
>> being
>> > > > part
>> > > > > of
>> > > > > > > > > Arrow and
>> > > > > > > > > > >> > >> > nurturing of the Apache Way, and now has a
>> > > community
>> > > > > > strong
>> > > > > > > > > enough
>> > > > > > > > > > >> to
>> > > > > > > > > > >> > >> > stand on its own and that would benefit from
>> > > focused
>> > > > > > > > governance
>> > > > > > > > > > >> > >> > attention.
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > The community has discussed this idea publicly
>> > for
>> > > > more
>> > > > > > than
>> > > > > > > > 6
>> > > > > > > > > > >> months
>> > > > > > > > > > >> > >> >
>> > > > > > https://github.com/apache/arrow-datafusion/discussions/6475
>> > > > > > > > > and
>> > > > > > > > > > >> > >> > briefly on the Arrow PMC mailing list
>> > > > > > > > > > >> > >> >
>> > > > > > > > >
>> > > https://lists.apache.org/thread/thv2jdm6640l6gm88hy8jhk5prjww0cs
>> > > > .
>> > > > > > > > > > >> As
>> > > > > > > > > > >> > >> > of the time of this writing both had
>> exclusively
>> > > > > positive
>> > > > > > > > > reactions.
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > Several current members of the Arrow PMC are
>> both
>> > > > > active
>> > > > > > > > > > >> contributors
>> > > > > > > > > > >> > >> > to DataFusion and understand and believe
>> deeply
>> > in
>> > > > the
>> > > > > > Apache
>> > > > > > > > > Way,
>> > > > > > > > > > >> and
>> > > > > > > > > > >> > >> > play active governance roles in the Arrow
>> project
>> > > as
>> > > > > PMC
>> > > > > > > > > members and
>> > > > > > > > > > >> > >> > PMC chairs, guiding the community, and
>> releasing
>> > > > > software
>> > > > > > > > > versions.
>> > > > > > > > > > >> > >> > With this existing governance experience and
>> > > > structure,
>> > > > > > the
>> > > > > > > > > new top
>> > > > > > > > > > >> > >> > level project will be able to function well
>> > > > immediately
>> > > > > > and
>> > > > > > > > > > >> > >> > independently.
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > Overview of DataFusion
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > Current Status
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > Meritocracy
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > DataFusion has been developed as part of
>> Apache
>> > > Arrow
>> > > > > and
>> > > > > > > > thus
>> > > > > > > > > has
>> > > > > > > > > > >> > >> > been operating as a meritocracy. Many of the
>> > > > developers
>> > > > > > of
>> > > > > > > > > > >> DataFusion
>> > > > > > > > > > >> > >> > are Arrow PMC members or committers. The
>> > DataFusion
>> > > > > > project
>> > > > > > > > > plans to
>> > > > > > > > > > >> > >> > continue adding new PMC and committers as the
>> > > project
>> > > > > > matures
>> > > > > > > > > and
>> > > > > > > > > > >> > >> > grows.
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > Community
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > The DataFusion development team seeks to
>> foster
>> > the
>> > > > > > > > > development and
>> > > > > > > > > > >> > >> > user communities. We hope that becoming a
>> > separate
>> > > > > > project
>> > > > > > > > > will help
>> > > > > > > > > > >> > >> > both Arrow and DataFusion communities by being
>> > more
>> > > > > > focused.
>> > > > > > > > > > >> Focused
>> > > > > > > > > > >> > >> > governance will make it easier to grow the
>> > > community
>> > > > of
>> > > > > > > > > committers
>> > > > > > > > > > >> and
>> > > > > > > > > > >> > >> > PMC members and make the organization more
>> clear
>> > to
>> > > > > > others.
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > Alignment
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > The ASF is a natural host for DataFusion given
>> > that
>> > > > it
>> > > > > is
>> > > > > > > > > already
>> > > > > > > > > > >> the
>> > > > > > > > > > >> > >> > home of Arrow, Parquet, and other related
>> > > distributed
>> > > > > > system,
>> > > > > > > > > > >> storage
>> > > > > > > > > > >> > >> > and query execution systems.
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > Project Leadership
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > Proposed Initial PMC
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > We propose the following people as the initial
>> > > > > > DataFusion PMC
>> > > > > > > > > > >> members.
>> > > > > > > > > > >> > >> > This is a subset of the existing Arrow PMC
>> > members
>> > > > who
>> > > > > > > > > contribute to
>> > > > > > > > > > >> > >> > DataFusion
>> > > > > > > > https://people.apache.org/phonebook.html?unix=arrow
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > Andy Grove (agrove): Arrow PMC Chair
>> > > > > > > > > > >> > >> > Andrew Lamb (alamb): Arrow PMC, past Arrow PMC
>> > > Chair
>> > > > > > > > > > >> > >> > Daniël Heres (dheres) Arrow PMC
>> > > > > > > > > > >> > >> > Jie Wen (jakevin): Arrow PMC, Doris Committer
>> > > > > > > > > > >> > >> > Kun Liu (liukun): Arrow PMC, IoTDB PMC, TSFile
>> > PMC
>> > > > > > > > > > >> > >> > Liang-Chi Hsieh (viirya): Arrow PMC, Spark PMC
>> > > > > > > > > > >> > >> > Qingping Hou: (houqp): Arrow PMC
>> > > > > > > > > > >> > >> > Wes McKinney(wesm): Arrow PMC, ASF Member
>> > > > > > > > > > >> > >> > Will Jones (wjones127): Arrow PMC
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > We’d like to propose Andrew Lamb as the
>> initial
>> > > > Chair,
>> > > > > > (and
>> > > > > > > > > thus ASF
>> > > > > > > > > > >> > >> > VP) for the DataFusion project.
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > Affiliations
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > Andy Grove (agrove): NVidia
>> > > > > > > > > > >> > >> > Andrew Lamb (alamb): InfluxData
>> > > > > > > > > > >> > >> > Daniël Heres (dheres): Coralogix
>> > > > > > > > > > >> > >> > Jie Wen (jakevin): SelectDB
>> > > > > > > > > > >> > >> > Kun Liu (liukun): Ebay
>> > > > > > > > > > >> > >> > Liang-Chi Hsieh (viirya): Apple
>> > > > > > > > > > >> > >> > Qingping Hou: (houqp): Scribd
>> > > > > > > > > > >> > >> > Wes McKinney(wesm): Posit
>> > > > > > > > > > >> > >> > Will Jones (wjones127): LanceDB
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > Proposed Initial Committers
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > In addition to the PMC, we propose the
>> following
>> > > > people
>> > > > > > as
>> > > > > > > > the
>> > > > > > > > > > >> initial
>> > > > > > > > > > >> > >> > DataFusion committers. This is a subset of the
>> > > > existing
>> > > > > > Arrow
>> > > > > > > > > > >> > >> > committers who contribute to DataFusion
>> > > > > > > > > > >> > >> >
>> > > https://people.apache.org/phonebook.html?unix=arrow
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > akurmustafa Mustafa Akur (Synnada)
>> > > > > > > > > > >> > >> > avantgardner Brent Gardner (Coralogix)
>> > > > > > > > > > >> > >> > comphead Oleks V. (Unaffiliated)
>> > > > > > > > > > >> > >> > jayzhan Jay Zhan (Unaffiliated)
>> > > > > > > > > > >> > >> > jeffreyvo Jeffry Vo (Unaffiliated)
>> > > > > > > > > > >> > >> > jiayuliu Liu Jiayu (Airbnb)
>> > > > > > > > > > >> > >> > mete Metehan Yildirim (Synnada)
>> > > > > > > > > > >> > >> > mingmwang Wang Mingming (Ebay)
>> > > > > > > > > > >> > >> > mneumann Marco Neumann (InfluxData)
>> > > > > > > > > > >> > >> > nju_yaho Zhong Yanghong (Ebay)
>> > > > > > > > > > >> > >> > ozankabak Mehmet Ozan Kabak (Synnada)
>> > > > > > > > > > >> > >> > paddyhoran Paddy Horan (Assured Allies)
>> > > > > > > > > > >> > >> > rdettai Rémi Dettai (Cloudfuse)
>> > > > > > > > > > >> > >> > sunchao Chao Sun (Apple)
>> > > > > > > > > > >> > >> > thinkharderdev Daniel Harris (Coralogix)
>> > > > > > > > > > >> > >> > tustvold Raphael Taylor-Davies (InfluxData)
>> > > > > > > > > > >> > >> > wayne Ruihang Xia (Greptime)
>> > > > > > > > > > >> > >> > xudong963 Xudong Wang (ByteDance)
>> > > > > > > > > > >> > >> > yjshen Yijie Shen (Space and Time)
>> > > > > > > > > > >> > >> > yangjiang Yang Jiang (ebay)
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > Risk Assessments
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > Naming / Trademarks
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > As a sub-project of Arrow, the DataFusion name
>> > has
>> > > > been
>> > > > > > used
>> > > > > > > > > for
>> > > > > > > > > > >> over
>> > > > > > > > > > >> > >> > 4 years without any known issues. A podling
>> name
>> > > > search
>> > > > > > did
>> > > > > > > > > not turn
>> > > > > > > > > > >> > >> > up any concerns and was approved:
>> > > > > > > > > > >> > >> >
>> > > > > > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > Legal / IP Clearance
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > All DataFusion code has either been donated to
>> > the
>> > > > > Arrow
>> > > > > > > > > project
>> > > > > > > > > > >> with
>> > > > > > > > > > >> > >> > appropriate IP clearance or has been
>> developed
>> > > > > directly
>> > > > > > > > under
>> > > > > > > > > ASF
>> > > > > > > > > > >> > >> > processes and procedures. Thus creating a new
>> top
>> > > > level
>> > > > > > > > project
>> > > > > > > > > > >> poses
>> > > > > > > > > > >> > >> > no new Legal or IP risks.
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > Code Extraction
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > The relevant code is already in 5 separate
>> > > > > repositories:
>> > > > > > > > > > >> > >> > https://github.com/apache/arrow-datafusion/
>> > > > > > > > > > >> > >> >
>> > https://github.com/apache/arrow-datafusion-python
>> > > > > > > > > > >> > >> > https://github.com/apache/arrow-ballista
>> > > > > > > > > > >> > >> >
>> https://github.com/apache/arrow-ballista-python
>> > > > > > > > > > >> > >> >
>> https://github.com/apache/arrow-datafusion-comet
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > We foresee no issues with code extraction and
>> > > propose
>> > > > > > these
>> > > > > > > > > > >> > >> > repositories be renamed to reflect top level
>> > > > projects
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > Note: https://github.com/apache/arrow-rs,
>> the
>> > > Rust
>> > > > > > > > > implementation
>> > > > > > > > > > >> of
>> > > > > > > > > > >> > >> > Arrow, would remain part of the Arrow project.
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > Orphaned Products
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > DataFusion is known to be used in many open
>> > source
>> > > > and
>> > > > > > > > > commercial
>> > > > > > > > > > >> > >> > projects
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> https://arrow.apache.org/datafusion/user-guide/
>> > > > > > > > > > >> > introduction.html#known-users
>> > > > > > > > > > >> > >> > ,
>> > > > > > > > > > >> > >> > has had multiple commits daily for several
>> years,
>> > > and
>> > > > > its
>> > > > > > > > > adoption
>> > > > > > > > > > >> and
>> > > > > > > > > > >> > >> > number of contributors appears to be growing.
>> We
>> > do
>> > > > not
>> > > > > > > > > foresee the
>> > > > > > > > > > >> > >> > project being orphaned in the next several
>> years.
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > Inexperience with Open Source
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > The proposed PMC has extensive experience with
>> > > Apache
>> > > > > > Arrow
>> > > > > > > > and
>> > > > > > > > > > >> other
>> > > > > > > > > > >> > >> > Apache projects, and includes PMC members, PMC
>> > > chairs
>> > > > > > and an
>> > > > > > > > > ASF
>> > > > > > > > > > >> > >> > Member. The DataFusion PMC and more
>> experienced
>> > > > > > committers
>> > > > > > > > will
>> > > > > > > > > > >> > >> > continue to coach new community members who
>> may
>> > be
>> > > > less
>> > > > > > > > > familiar
>> > > > > > > > > > >> with
>> > > > > > > > > > >> > >> > the Apache Way.
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > Homogeneous Developers
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > The 9 proposed PMC members are from 9
>> different
>> > > > > > employers and
>> > > > > > > > > the
>> > > > > > > > > > >> > >> > proposed committers are similarly distributed
>> > > across
>> > > > > > > > > affiliations.
>> > > > > > > > > > >> No
>> > > > > > > > > > >> > >> > specific entity employs more than 3 total
>> > proposed
>> > > > > > > > developers.
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > Reliance on Salaried Developers
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > A substantial amount of work on DataFusion has
>> > been
>> > > > by
>> > > > > > > > salaried
>> > > > > > > > > > >> > >> > developers, but it also has a long tradition
>> of
>> > > > > > attracting
>> > > > > > > > > > >> > >> > contributions from students and hobbyists and
>> we
>> > > plan
>> > > > > no
>> > > > > > > > > changes in
>> > > > > > > > > > >> > >> > contribution structure.
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > Relationships with Other Apache Products
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > DataFusion will obviously have a strong
>> > > relationship
>> > > > > > with the
>> > > > > > > > > Arrow
>> > > > > > > > > > >> > >> > project given the overlap in people. We don’t
>> > > foresee
>> > > > > > close
>> > > > > > > > > > >> > >> > collaboration with other projects at this
>> time.
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > Cryptography
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > DataFusion does not directly support
>> encryption
>> > and
>> > > > > > there are
>> > > > > > > > > no
>> > > > > > > > > > >> > >> > near-term plans to add support for encryption.
>> > > Users
>> > > > > who
>> > > > > > need
>> > > > > > > > > this
>> > > > > > > > > > >> > >> > functionality can use the extension APIs.
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > Required Resources
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > Mailing Lists
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > - private@datafusion.apache.org for private
>> PMC
>> > > > > > discussions
>> > > > > > > > > (with
>> > > > > > > > > > >> > >> > moderated subscriptions)
>> > > > > > > > > > >> > >> > - dev@datafusion.apache.org
>> > > > > > > > > > >> > >> > - commits@datafusion.apache.org
>> > > > > > > > > > >> > >> > - user@datafusion.apache.org
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > Version Control
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > We propose to continue to use git for source
>> > > control
>> > > > > and
>> > > > > > > > > github for
>> > > > > > > > > > >> > >> > hosting and testing resources.
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > We also need to rename the github
>> repositories to
>> > > > > > reflect the
>> > > > > > > > > new
>> > > > > > > > > > >> top
>> > > > > > > > > > >> > >> > level names:
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > https://github.com/apache/arrow-datafusion/ →
>> > > > > > > > > apache/datafusion
>> > > > > > > > > > >> > >> >
>> > https://github.com/apache/arrow-datafusion-python
>> > > →
>> > > > > > > > > > >> > >> > apache/datafusion-python
>> > > > > > > > > > >> > >> > https://github.com/apache/arrow-ballista →
>> > > > > > > > > > >> apache/datafusion-ballista
>> > > > > > > > > > >> > >> >
>> https://github.com/apache/arrow-ballista-python
>> > →
>> > > > > > > > > > >> > >> > apache/datafusion-ballista-python
>> > > > > > > > > > >> > >> >
>> https://github.com/apache/arrow-datafusion-comet
>> > →
>> > > > > > > > > > >> > >> apache/datafusion-comet
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > Issue Tracking
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > DataFusion would continue to use github for
>> its
>> > > issue
>> > > > > > > > tracking
>> > > > > > > > > and
>> > > > > > > > > > >> > >> > communications
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > Other Resources
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > The existing repositories already make use of
>> > > > existing
>> > > > > > Apache
>> > > > > > > > > > >> > >> > infrastructure, and we expect no change in the
>> > > > initial
>> > > > > > > > resource
>> > > > > > > > > > >> usage.
>> > > > > > > > > > >> > >> > As the project continues to grow, we expect
>> > > continued
>> > > > > > > > > infrastructure
>> > > > > > > > > > >> > >> > demand growth.
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > FAQ: Has a sub project been promoted to a top
>> > level
>> > > > > > project
>> > > > > > > > > before?
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > Yes, and it appears to happen commonly. The
>> Arrow
>> > > > > project
>> > > > > > > > > itself was
>> > > > > > > > > > >> > >> > created as a top level project from work that
>> > > started
>> > > > > in
>> > > > > > > > Apache
>> > > > > > > > > > >> Drill,
>> > > > > > > > > > >> > >> > and there are many sub projects of Hadoop that
>> > spun
>> > > > out
>> > > > > > as
>> > > > > > > > > their own
>> > > > > > > > > > >> > >> > top level projects such as Mahout, Avro and
>> > HBase:
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >>
>> > https://news.apache.org/foundation/entry/the_apache_
>> > > > > > > > > > >> > software_foundation_announces4
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >> > Related material:
>> > > > > > > > > > >> > >> > Name search request / research for DataFusion:
>> > > > > > > > > > >> > >> >
>> > > > > > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
>> > > > > > > > > > >> > >> > Discussion about this proposal on the arrow
>> > mailing
>> > > > > list:
>> > > > > > > > > > >> > >> >
>> > > > > > > > >
>> > > https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
>> > > > > > > > > > >> > >> > Discussion about which repositories on the
>> arrow
>> > > > > mailing
>> > > > > > > > list:
>> > > > > > > > > > >> > >> >
>> > > > > > > > >
>> > > https://lists.apache.org/thread/ob3n0d9ky0bgrryl3xn39w9k566bq00q
>> > > > > > > > > > >> > >> > Discussion about initial PMC on the arrow
>> mailing
>> > > > list:
>> > > > > > > > > > >> > >> >
>> > > > > > > > >
>> > > https://lists.apache.org/thread/pymrzcdw4qdptvby85f69rg3pcckl15b
>> > > > > > > > > > >> > >> > Discussion in github about creating a new
>> > > DataFusion
>> > > > > top
>> > > > > > > > level
>> > > > > > > > > > >> > >> > project:
>> > > > > > > > > > >>
>> > > https://github.com/apache/arrow-datafusion/discussions/6475
>> > > > > > > > > > >> > >> > Discussion about graduating on incubator list:
>> > > > > > > > > > >> > >> >
>> > > > > > > > >
>> > > https://lists.apache.org/thread/r4n73pmms1lv0jbohyx1o1z13d615t99
>> > > > > > > > > > >> > >> > Original Proposal for the Arrow project:
>> > > > > > > > > > >> > >> >
>> > > > > > > > >
>> > > https://lists.apache.org/thread/x2qzdwglm8pkqp9gv03bbgw17khl7pq3
>> > > > > > > > > > >> > >> >
>> > > > > > > > > > >> > >>
>> > > > > > > > > > >> >
>> > > > > > > > > > >>
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
[RESULT] [VOTE] Move Arrow DataFusion Subproject to new Top Level Apache Project
Posted by Andrew Lamb <al...@influxdata.com>.
With 30 +1 votes (18 binding by my count), and 0 -1 votes, the proposal is
approved
Thank you everyone who voted and participated in helping form the proposal
(and the community over the last few years). I think there are exciting
times ahead for DataFusion.
We will submit the proposed motion to the ASF board with our next quarterly
report, for the April 2024 meeting, as previously discussed.
Thanks again
Andrew
On Tue, Mar 5, 2024 at 2:00 PM Benjamin Kietzman <be...@gmail.com>
wrote:
> +1 (binding)
>
> On Tue, Mar 5, 2024, 13:03 Peter Toth <pe...@gmail.com> wrote:
>
> > +1 (non-binding)
> >
> > Parth Chandra <pa...@apache.org> ezt írta (időpont: 2024. márc. 5., K,
> > 18:15):
> >
> > > +1 (non-binding)
> > >
> > > On Sun, Mar 3, 2024 at 8:04 PM Mehmet Ozan Kabak <oz...@synnada.ai>
> > wrote:
> > >
> > > > +1 (non-binding)
> > > > --
> > > > *Mehmet Ozan Kabak*
> > > > Co-founder & CEO @ Synnada, Inc.
> > > >
> > > >
> > > > On Sun, Mar 3, 2024 at 7:33 PM Jacob Wujciak-Jens
> > > > <ja...@voltrondata.com.invalid> wrote:
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > On Mon, Mar 4, 2024 at 3:39 AM Yang Jiang <ya...@apache.org>
> > > wrote:
> > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > On 2024/03/01 18:08:26 Daniël Heres wrote:
> > > > > > > +1 (binding)
> > > > > > >
> > > > > > > On Fri, Mar 1, 2024, 19:05 Chao Sun <su...@apache.org>
> wrote:
> > > > > > >
> > > > > > > > +1 (non-binding)
> > > > > > > >
> > > > > > > > On Fri, Mar 1, 2024 at 9:59 AM QP Hou <qp...@neuralink.com>
> > wrote:
> > > > > > > >
> > > > > > > > > +1 (binding)
> > > > > > > > >
> > > > > > > > > exciting milestone :)
> > > > > > > > >
> > > > > > > > > On Fri, Mar 1, 2024 at 9:49 AM David Li <
> lidavidm@apache.org
> > >
> > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > +1
> > > > > > > > > >
> > > > > > > > > > On Fri, Mar 1, 2024, at 12:06, Jorge Cardoso Leitão
> wrote:
> > > > > > > > > > > +1 - great work!!!
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Mar 1, 2024 at 5:49 PM Micah Kornfield <
> > > > > > > > emkornfield@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >> +1 (binding)
> > > > > > > > > > >>
> > > > > > > > > > >> On Friday, March 1, 2024, Uwe L. Korn <
> uwelk@xhochy.com
> > >
> > > > > wrote:
> > > > > > > > > > >>
> > > > > > > > > > >> > +1 (binding)
> > > > > > > > > > >> >
> > > > > > > > > > >> > On Fri, Mar 1, 2024, at 2:37 PM, Andy Grove wrote:
> > > > > > > > > > >> > > +1 (binding)
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > On Fri, Mar 1, 2024 at 6:20 AM Weston Pace <
> > > > > > > > weston.pace@gmail.com
> > > > > > > > > >
> > > > > > > > > > >> > wrote:
> > > > > > > > > > >> > >
> > > > > > > > > > >> > >> +1 (binding)
> > > > > > > > > > >> > >>
> > > > > > > > > > >> > >> On Fri, Mar 1, 2024 at 3:33 AM Andrew Lamb <
> > > > > > > > alamb@influxdata.com
> > > > > > > > > >
> > > > > > > > > > >> > wrote:
> > > > > > > > > > >> > >>
> > > > > > > > > > >> > >> > Hello,
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > As we have discussed[1][2] I would like to vote
> > on
> > > > the
> > > > > > > > > proposal to
> > > > > > > > > > >> > >> > create a new Apache Top Level Project for
> > > DataFusion.
> > > > > The
> > > > > > > > text
> > > > > > > > > of
> > > > > > > > > > >> the
> > > > > > > > > > >> > >> > proposed resolution and background document is
> > > > > > copy/pasted
> > > > > > > > > below
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > If the community is in favor of this, we plan
> to
> > > > submit
> > > > > > the
> > > > > > > > > > >> resolution
> > > > > > > > > > >> > >> > to the ASF board for approval with the next
> Arrow
> > > > > report
> > > > > > (for
> > > > > > > > > the
> > > > > > > > > > >> > >> > April 2024 board meeting).
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > The vote will be open for at least 7 days.
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > [ ] +1 Accept this Proposal
> > > > > > > > > > >> > >> > [ ] +0
> > > > > > > > > > >> > >> > [ ] -1 Do not accept this proposal because...
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > Andrew
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > [1]
> > > > > > > > > > >>
> > > > > > https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> > > > > > > > > > >> > >> > [2]
> > > > > > > > >
> https://github.com/apache/arrow-datafusion/discussions/6475
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > ---------- Proposed Resolution ---------
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > Resolution to Create the Apache DataFusion
> > Project
> > > > from
> > > > > > the
> > > > > > > > > Apache
> > > > > > > > > > >> > >> > Arrow DataFusion Sub Project
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> >
> > > > > > =============================================================
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > X. Establish the Apache DataFusion Project
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > WHEREAS, the Board of Directors deems it to be
> in
> > > the
> > > > > > best
> > > > > > > > > > >> > >> > interests of the Foundation and consistent with
> > the
> > > > > > > > > > >> > >> > Foundation's purpose to establish a Project
> > > > Management
> > > > > > > > > > >> > >> > Committee charged with the creation and
> > maintenance
> > > > of
> > > > > > > > > > >> > >> > open-source software related to an extensible
> > query
> > > > > > engine
> > > > > > > > > > >> > >> > for distribution at no charge to the public.
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > NOW, THEREFORE, BE IT RESOLVED, that a Project
> > > > > Management
> > > > > > > > > > >> > >> > Committee (PMC), to be known as the "Apache
> > > > DataFusion
> > > > > > > > > Project",
> > > > > > > > > > >> > >> > be and hereby is established pursuant to Bylaws
> > of
> > > > the
> > > > > > > > > > >> > >> > Foundation; and be it further
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > RESOLVED, that the Apache DataFusion Project be
> > and
> > > > > > hereby is
> > > > > > > > > > >> > >> > responsible for the creation and maintenance of
> > > > > software
> > > > > > > > > > >> > >> > related to an extensible query engine; and be
> it
> > > > > further
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > RESOLVED, that the office of "Vice President,
> > > Apache
> > > > > > > > > DataFusion" be
> > > > > > > > > > >> > >> > and hereby is created, the person holding such
> > > office
> > > > > to
> > > > > > > > > > >> > >> > serve at the direction of the Board of
> Directors
> > as
> > > > the
> > > > > > chair
> > > > > > > > > > >> > >> > of the Apache DataFusion Project, and to have
> > > primary
> > > > > > > > > responsibility
> > > > > > > > > > >> > >> > for management of the projects within the scope
> > of
> > > > > > > > > > >> > >> > responsibility of the Apache DataFusion
> Project;
> > > and
> > > > be
> > > > > > it
> > > > > > > > > further
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > RESOLVED, that the persons listed immediately
> > below
> > > > be
> > > > > > and
> > > > > > > > > > >> > >> > hereby are appointed to serve as the initial
> > > members
> > > > of
> > > > > > the
> > > > > > > > > > >> > >> > Apache DataFusion Project:
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > * Andy Grove (agrove@apache.org)
> > > > > > > > > > >> > >> > * Andrew Lamb (alamb@apache.org)
> > > > > > > > > > >> > >> > * Daniël Heres (dheres@apache.org)
> > > > > > > > > > >> > >> > * Jie Wen (jakevin@apache.org)
> > > > > > > > > > >> > >> > * Kun Liu (liukun@apache.org)
> > > > > > > > > > >> > >> > * Liang-Chi Hsieh (viirya@apache.org)
> > > > > > > > > > >> > >> > * Qingping Hou: (houqp@apache.org)
> > > > > > > > > > >> > >> > * Wes McKinney(wesm@apache.org)
> > > > > > > > > > >> > >> > * Will Jones (wjones127@apache.org)
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > RESOLVED, that the Apache DataFusion Project be
> > and
> > > > > > hereby
> > > > > > > > > > >> > >> > is tasked with the migration and
> rationalization
> > of
> > > > the
> > > > > > > > Apache
> > > > > > > > > > >> > >> > Arrow DataFusion sub-project; and be it further
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > RESOLVED, that all responsibilities pertaining
> to
> > > the
> > > > > > Apache
> > > > > > > > > > >> > >> > Arrow DataFusion sub-project encumbered upon
> the
> > > > > > > > > > >> > >> > Apache Arrow Project are hereafter discharged.
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > NOW, THEREFORE, BE IT FURTHER RESOLVED, that
> > Andrew
> > > > > Lamb
> > > > > > > > > > >> > >> > be appointed to the office of Vice President,
> > > Apache
> > > > > > > > > DataFusion, to
> > > > > > > > > > >> > >> > serve in accordance with and subject to the
> > > direction
> > > > > of
> > > > > > the
> > > > > > > > > > >> > >> > Board of Directors and the Bylaws of the
> > Foundation
> > > > > until
> > > > > > > > > > >> > >> > death, resignation, retirement, removal or
> > > > > > disqualification,
> > > > > > > > > > >> > >> > or until a successor is appointed.
> > > > > > > > > > >> > >> >
> > > > > > =============================================================
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > -------
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > Summary:
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > We propose creating a new top level project,
> > Apache
> > > > > > > > > DataFusion, from
> > > > > > > > > > >> > >> > an existing sub project of Apache Arrow to
> > > facilitate
> > > > > > > > > additional
> > > > > > > > > > >> > >> > community and project growth.
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > Abstract
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > Apache Arrow DataFusion[1] is a very fast,
> > > > extensible
> > > > > > query
> > > > > > > > > engine
> > > > > > > > > > >> > >> > for building high-quality data-centric systems
> in
> > > > Rust,
> > > > > > using
> > > > > > > > > the
> > > > > > > > > > >> > >> > Apache Arrow in-memory format. DataFusion
> offers
> > > SQL
> > > > > and
> > > > > > > > > Dataframe
> > > > > > > > > > >> > >> > APIs, excellent performance, built-in support
> for
> > > > CSV,
> > > > > > > > Parquet,
> > > > > > > > > > >> JSON,
> > > > > > > > > > >> > >> > and Avro, extensive customization, and a great
> > > > > community.
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > [1] https://arrow.apache.org/datafusion/
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > Proposal
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > We propose creating a new top level ASF
> project,
> > > > Apache
> > > > > > > > > DataFusion,
> > > > > > > > > > >> > >> > governed initially by a subset of the Apache
> > Arrow
> > > > > > project’s
> > > > > > > > > PMC and
> > > > > > > > > > >> > >> > committers. The project’s code is in five
> > existing
> > > > git
> > > > > > > > > repositories,
> > > > > > > > > > >> > >> > currently governed by Apache Arrow which would
> > > > transfer
> > > > > > to
> > > > > > > > the
> > > > > > > > > new
> > > > > > > > > > >> top
> > > > > > > > > > >> > >> > level project.
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > Background
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > When DataFusion was initially donated to the
> > Arrow
> > > > > > project,
> > > > > > > > it
> > > > > > > > > did
> > > > > > > > > > >> not
> > > > > > > > > > >> > >> > have a strong enough community to stand on its
> > own.
> > > > It
> > > > > > has
> > > > > > > > > since
> > > > > > > > > > >> grown
> > > > > > > > > > >> > >> > significantly, and benefited immensely from
> being
> > > > part
> > > > > of
> > > > > > > > > Arrow and
> > > > > > > > > > >> > >> > nurturing of the Apache Way, and now has a
> > > community
> > > > > > strong
> > > > > > > > > enough
> > > > > > > > > > >> to
> > > > > > > > > > >> > >> > stand on its own and that would benefit from
> > > focused
> > > > > > > > governance
> > > > > > > > > > >> > >> > attention.
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > The community has discussed this idea publicly
> > for
> > > > more
> > > > > > than
> > > > > > > > 6
> > > > > > > > > > >> months
> > > > > > > > > > >> > >> >
> > > > > > https://github.com/apache/arrow-datafusion/discussions/6475
> > > > > > > > > and
> > > > > > > > > > >> > >> > briefly on the Arrow PMC mailing list
> > > > > > > > > > >> > >> >
> > > > > > > > >
> > > https://lists.apache.org/thread/thv2jdm6640l6gm88hy8jhk5prjww0cs
> > > > .
> > > > > > > > > > >> As
> > > > > > > > > > >> > >> > of the time of this writing both had
> exclusively
> > > > > positive
> > > > > > > > > reactions.
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > Several current members of the Arrow PMC are
> both
> > > > > active
> > > > > > > > > > >> contributors
> > > > > > > > > > >> > >> > to DataFusion and understand and believe deeply
> > in
> > > > the
> > > > > > Apache
> > > > > > > > > Way,
> > > > > > > > > > >> and
> > > > > > > > > > >> > >> > play active governance roles in the Arrow
> project
> > > as
> > > > > PMC
> > > > > > > > > members and
> > > > > > > > > > >> > >> > PMC chairs, guiding the community, and
> releasing
> > > > > software
> > > > > > > > > versions.
> > > > > > > > > > >> > >> > With this existing governance experience and
> > > > structure,
> > > > > > the
> > > > > > > > > new top
> > > > > > > > > > >> > >> > level project will be able to function well
> > > > immediately
> > > > > > and
> > > > > > > > > > >> > >> > independently.
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > Overview of DataFusion
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > Current Status
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > Meritocracy
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > DataFusion has been developed as part of Apache
> > > Arrow
> > > > > and
> > > > > > > > thus
> > > > > > > > > has
> > > > > > > > > > >> > >> > been operating as a meritocracy. Many of the
> > > > developers
> > > > > > of
> > > > > > > > > > >> DataFusion
> > > > > > > > > > >> > >> > are Arrow PMC members or committers. The
> > DataFusion
> > > > > > project
> > > > > > > > > plans to
> > > > > > > > > > >> > >> > continue adding new PMC and committers as the
> > > project
> > > > > > matures
> > > > > > > > > and
> > > > > > > > > > >> > >> > grows.
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > Community
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > The DataFusion development team seeks to foster
> > the
> > > > > > > > > development and
> > > > > > > > > > >> > >> > user communities. We hope that becoming a
> > separate
> > > > > > project
> > > > > > > > > will help
> > > > > > > > > > >> > >> > both Arrow and DataFusion communities by being
> > more
> > > > > > focused.
> > > > > > > > > > >> Focused
> > > > > > > > > > >> > >> > governance will make it easier to grow the
> > > community
> > > > of
> > > > > > > > > committers
> > > > > > > > > > >> and
> > > > > > > > > > >> > >> > PMC members and make the organization more
> clear
> > to
> > > > > > others.
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > Alignment
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > The ASF is a natural host for DataFusion given
> > that
> > > > it
> > > > > is
> > > > > > > > > already
> > > > > > > > > > >> the
> > > > > > > > > > >> > >> > home of Arrow, Parquet, and other related
> > > distributed
> > > > > > system,
> > > > > > > > > > >> storage
> > > > > > > > > > >> > >> > and query execution systems.
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > Project Leadership
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > Proposed Initial PMC
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > We propose the following people as the initial
> > > > > > DataFusion PMC
> > > > > > > > > > >> members.
> > > > > > > > > > >> > >> > This is a subset of the existing Arrow PMC
> > members
> > > > who
> > > > > > > > > contribute to
> > > > > > > > > > >> > >> > DataFusion
> > > > > > > > https://people.apache.org/phonebook.html?unix=arrow
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > Andy Grove (agrove): Arrow PMC Chair
> > > > > > > > > > >> > >> > Andrew Lamb (alamb): Arrow PMC, past Arrow PMC
> > > Chair
> > > > > > > > > > >> > >> > Daniël Heres (dheres) Arrow PMC
> > > > > > > > > > >> > >> > Jie Wen (jakevin): Arrow PMC, Doris Committer
> > > > > > > > > > >> > >> > Kun Liu (liukun): Arrow PMC, IoTDB PMC, TSFile
> > PMC
> > > > > > > > > > >> > >> > Liang-Chi Hsieh (viirya): Arrow PMC, Spark PMC
> > > > > > > > > > >> > >> > Qingping Hou: (houqp): Arrow PMC
> > > > > > > > > > >> > >> > Wes McKinney(wesm): Arrow PMC, ASF Member
> > > > > > > > > > >> > >> > Will Jones (wjones127): Arrow PMC
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > We’d like to propose Andrew Lamb as the initial
> > > > Chair,
> > > > > > (and
> > > > > > > > > thus ASF
> > > > > > > > > > >> > >> > VP) for the DataFusion project.
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > Affiliations
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > Andy Grove (agrove): NVidia
> > > > > > > > > > >> > >> > Andrew Lamb (alamb): InfluxData
> > > > > > > > > > >> > >> > Daniël Heres (dheres): Coralogix
> > > > > > > > > > >> > >> > Jie Wen (jakevin): SelectDB
> > > > > > > > > > >> > >> > Kun Liu (liukun): Ebay
> > > > > > > > > > >> > >> > Liang-Chi Hsieh (viirya): Apple
> > > > > > > > > > >> > >> > Qingping Hou: (houqp): Scribd
> > > > > > > > > > >> > >> > Wes McKinney(wesm): Posit
> > > > > > > > > > >> > >> > Will Jones (wjones127): LanceDB
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > Proposed Initial Committers
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > In addition to the PMC, we propose the
> following
> > > > people
> > > > > > as
> > > > > > > > the
> > > > > > > > > > >> initial
> > > > > > > > > > >> > >> > DataFusion committers. This is a subset of the
> > > > existing
> > > > > > Arrow
> > > > > > > > > > >> > >> > committers who contribute to DataFusion
> > > > > > > > > > >> > >> >
> > > https://people.apache.org/phonebook.html?unix=arrow
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > akurmustafa Mustafa Akur (Synnada)
> > > > > > > > > > >> > >> > avantgardner Brent Gardner (Coralogix)
> > > > > > > > > > >> > >> > comphead Oleks V. (Unaffiliated)
> > > > > > > > > > >> > >> > jayzhan Jay Zhan (Unaffiliated)
> > > > > > > > > > >> > >> > jeffreyvo Jeffry Vo (Unaffiliated)
> > > > > > > > > > >> > >> > jiayuliu Liu Jiayu (Airbnb)
> > > > > > > > > > >> > >> > mete Metehan Yildirim (Synnada)
> > > > > > > > > > >> > >> > mingmwang Wang Mingming (Ebay)
> > > > > > > > > > >> > >> > mneumann Marco Neumann (InfluxData)
> > > > > > > > > > >> > >> > nju_yaho Zhong Yanghong (Ebay)
> > > > > > > > > > >> > >> > ozankabak Mehmet Ozan Kabak (Synnada)
> > > > > > > > > > >> > >> > paddyhoran Paddy Horan (Assured Allies)
> > > > > > > > > > >> > >> > rdettai Rémi Dettai (Cloudfuse)
> > > > > > > > > > >> > >> > sunchao Chao Sun (Apple)
> > > > > > > > > > >> > >> > thinkharderdev Daniel Harris (Coralogix)
> > > > > > > > > > >> > >> > tustvold Raphael Taylor-Davies (InfluxData)
> > > > > > > > > > >> > >> > wayne Ruihang Xia (Greptime)
> > > > > > > > > > >> > >> > xudong963 Xudong Wang (ByteDance)
> > > > > > > > > > >> > >> > yjshen Yijie Shen (Space and Time)
> > > > > > > > > > >> > >> > yangjiang Yang Jiang (ebay)
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > Risk Assessments
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > Naming / Trademarks
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > As a sub-project of Arrow, the DataFusion name
> > has
> > > > been
> > > > > > used
> > > > > > > > > for
> > > > > > > > > > >> over
> > > > > > > > > > >> > >> > 4 years without any known issues. A podling
> name
> > > > search
> > > > > > did
> > > > > > > > > not turn
> > > > > > > > > > >> > >> > up any concerns and was approved:
> > > > > > > > > > >> > >> >
> > > > > > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > Legal / IP Clearance
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > All DataFusion code has either been donated to
> > the
> > > > > Arrow
> > > > > > > > > project
> > > > > > > > > > >> with
> > > > > > > > > > >> > >> > appropriate IP clearance or has been developed
> > > > > directly
> > > > > > > > under
> > > > > > > > > ASF
> > > > > > > > > > >> > >> > processes and procedures. Thus creating a new
> top
> > > > level
> > > > > > > > project
> > > > > > > > > > >> poses
> > > > > > > > > > >> > >> > no new Legal or IP risks.
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > Code Extraction
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > The relevant code is already in 5 separate
> > > > > repositories:
> > > > > > > > > > >> > >> > https://github.com/apache/arrow-datafusion/
> > > > > > > > > > >> > >> >
> > https://github.com/apache/arrow-datafusion-python
> > > > > > > > > > >> > >> > https://github.com/apache/arrow-ballista
> > > > > > > > > > >> > >> >
> https://github.com/apache/arrow-ballista-python
> > > > > > > > > > >> > >> >
> https://github.com/apache/arrow-datafusion-comet
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > We foresee no issues with code extraction and
> > > propose
> > > > > > these
> > > > > > > > > > >> > >> > repositories be renamed to reflect top level
> > > > projects
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > Note: https://github.com/apache/arrow-rs, the
> > > Rust
> > > > > > > > > implementation
> > > > > > > > > > >> of
> > > > > > > > > > >> > >> > Arrow, would remain part of the Arrow project.
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > Orphaned Products
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > DataFusion is known to be used in many open
> > source
> > > > and
> > > > > > > > > commercial
> > > > > > > > > > >> > >> > projects
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> https://arrow.apache.org/datafusion/user-guide/
> > > > > > > > > > >> > introduction.html#known-users
> > > > > > > > > > >> > >> > ,
> > > > > > > > > > >> > >> > has had multiple commits daily for several
> years,
> > > and
> > > > > its
> > > > > > > > > adoption
> > > > > > > > > > >> and
> > > > > > > > > > >> > >> > number of contributors appears to be growing.
> We
> > do
> > > > not
> > > > > > > > > foresee the
> > > > > > > > > > >> > >> > project being orphaned in the next several
> years.
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > Inexperience with Open Source
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > The proposed PMC has extensive experience with
> > > Apache
> > > > > > Arrow
> > > > > > > > and
> > > > > > > > > > >> other
> > > > > > > > > > >> > >> > Apache projects, and includes PMC members, PMC
> > > chairs
> > > > > > and an
> > > > > > > > > ASF
> > > > > > > > > > >> > >> > Member. The DataFusion PMC and more experienced
> > > > > > committers
> > > > > > > > will
> > > > > > > > > > >> > >> > continue to coach new community members who may
> > be
> > > > less
> > > > > > > > > familiar
> > > > > > > > > > >> with
> > > > > > > > > > >> > >> > the Apache Way.
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > Homogeneous Developers
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > The 9 proposed PMC members are from 9 different
> > > > > > employers and
> > > > > > > > > the
> > > > > > > > > > >> > >> > proposed committers are similarly distributed
> > > across
> > > > > > > > > affiliations.
> > > > > > > > > > >> No
> > > > > > > > > > >> > >> > specific entity employs more than 3 total
> > proposed
> > > > > > > > developers.
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > Reliance on Salaried Developers
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > A substantial amount of work on DataFusion has
> > been
> > > > by
> > > > > > > > salaried
> > > > > > > > > > >> > >> > developers, but it also has a long tradition of
> > > > > > attracting
> > > > > > > > > > >> > >> > contributions from students and hobbyists and
> we
> > > plan
> > > > > no
> > > > > > > > > changes in
> > > > > > > > > > >> > >> > contribution structure.
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > Relationships with Other Apache Products
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > DataFusion will obviously have a strong
> > > relationship
> > > > > > with the
> > > > > > > > > Arrow
> > > > > > > > > > >> > >> > project given the overlap in people. We don’t
> > > foresee
> > > > > > close
> > > > > > > > > > >> > >> > collaboration with other projects at this time.
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > Cryptography
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > DataFusion does not directly support encryption
> > and
> > > > > > there are
> > > > > > > > > no
> > > > > > > > > > >> > >> > near-term plans to add support for encryption.
> > > Users
> > > > > who
> > > > > > need
> > > > > > > > > this
> > > > > > > > > > >> > >> > functionality can use the extension APIs.
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > Required Resources
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > Mailing Lists
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > - private@datafusion.apache.org for private
> PMC
> > > > > > discussions
> > > > > > > > > (with
> > > > > > > > > > >> > >> > moderated subscriptions)
> > > > > > > > > > >> > >> > - dev@datafusion.apache.org
> > > > > > > > > > >> > >> > - commits@datafusion.apache.org
> > > > > > > > > > >> > >> > - user@datafusion.apache.org
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > Version Control
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > We propose to continue to use git for source
> > > control
> > > > > and
> > > > > > > > > github for
> > > > > > > > > > >> > >> > hosting and testing resources.
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > We also need to rename the github repositories
> to
> > > > > > reflect the
> > > > > > > > > new
> > > > > > > > > > >> top
> > > > > > > > > > >> > >> > level names:
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > https://github.com/apache/arrow-datafusion/ →
> > > > > > > > > apache/datafusion
> > > > > > > > > > >> > >> >
> > https://github.com/apache/arrow-datafusion-python
> > > →
> > > > > > > > > > >> > >> > apache/datafusion-python
> > > > > > > > > > >> > >> > https://github.com/apache/arrow-ballista →
> > > > > > > > > > >> apache/datafusion-ballista
> > > > > > > > > > >> > >> >
> https://github.com/apache/arrow-ballista-python
> > →
> > > > > > > > > > >> > >> > apache/datafusion-ballista-python
> > > > > > > > > > >> > >> >
> https://github.com/apache/arrow-datafusion-comet
> > →
> > > > > > > > > > >> > >> apache/datafusion-comet
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > Issue Tracking
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > DataFusion would continue to use github for its
> > > issue
> > > > > > > > tracking
> > > > > > > > > and
> > > > > > > > > > >> > >> > communications
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > Other Resources
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > The existing repositories already make use of
> > > > existing
> > > > > > Apache
> > > > > > > > > > >> > >> > infrastructure, and we expect no change in the
> > > > initial
> > > > > > > > resource
> > > > > > > > > > >> usage.
> > > > > > > > > > >> > >> > As the project continues to grow, we expect
> > > continued
> > > > > > > > > infrastructure
> > > > > > > > > > >> > >> > demand growth.
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > FAQ: Has a sub project been promoted to a top
> > level
> > > > > > project
> > > > > > > > > before?
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > Yes, and it appears to happen commonly. The
> Arrow
> > > > > project
> > > > > > > > > itself was
> > > > > > > > > > >> > >> > created as a top level project from work that
> > > started
> > > > > in
> > > > > > > > Apache
> > > > > > > > > > >> Drill,
> > > > > > > > > > >> > >> > and there are many sub projects of Hadoop that
> > spun
> > > > out
> > > > > > as
> > > > > > > > > their own
> > > > > > > > > > >> > >> > top level projects such as Mahout, Avro and
> > HBase:
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >>
> > https://news.apache.org/foundation/entry/the_apache_
> > > > > > > > > > >> > software_foundation_announces4
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >> > Related material:
> > > > > > > > > > >> > >> > Name search request / research for DataFusion:
> > > > > > > > > > >> > >> >
> > > > > > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> > > > > > > > > > >> > >> > Discussion about this proposal on the arrow
> > mailing
> > > > > list:
> > > > > > > > > > >> > >> >
> > > > > > > > >
> > > https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> > > > > > > > > > >> > >> > Discussion about which repositories on the
> arrow
> > > > > mailing
> > > > > > > > list:
> > > > > > > > > > >> > >> >
> > > > > > > > >
> > > https://lists.apache.org/thread/ob3n0d9ky0bgrryl3xn39w9k566bq00q
> > > > > > > > > > >> > >> > Discussion about initial PMC on the arrow
> mailing
> > > > list:
> > > > > > > > > > >> > >> >
> > > > > > > > >
> > > https://lists.apache.org/thread/pymrzcdw4qdptvby85f69rg3pcckl15b
> > > > > > > > > > >> > >> > Discussion in github about creating a new
> > > DataFusion
> > > > > top
> > > > > > > > level
> > > > > > > > > > >> > >> > project:
> > > > > > > > > > >>
> > > https://github.com/apache/arrow-datafusion/discussions/6475
> > > > > > > > > > >> > >> > Discussion about graduating on incubator list:
> > > > > > > > > > >> > >> >
> > > > > > > > >
> > > https://lists.apache.org/thread/r4n73pmms1lv0jbohyx1o1z13d615t99
> > > > > > > > > > >> > >> > Original Proposal for the Arrow project:
> > > > > > > > > > >> > >> >
> > > > > > > > >
> > > https://lists.apache.org/thread/x2qzdwglm8pkqp9gv03bbgw17khl7pq3
> > > > > > > > > > >> > >> >
> > > > > > > > > > >> > >>
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [VOTE] Move Arrow DataFusion Subproject to new Top Level Apache Project
Posted by Benjamin Kietzman <be...@gmail.com>.
+1 (binding)
On Tue, Mar 5, 2024, 13:03 Peter Toth <pe...@gmail.com> wrote:
> +1 (non-binding)
>
> Parth Chandra <pa...@apache.org> ezt írta (időpont: 2024. márc. 5., K,
> 18:15):
>
> > +1 (non-binding)
> >
> > On Sun, Mar 3, 2024 at 8:04 PM Mehmet Ozan Kabak <oz...@synnada.ai>
> wrote:
> >
> > > +1 (non-binding)
> > > --
> > > *Mehmet Ozan Kabak*
> > > Co-founder & CEO @ Synnada, Inc.
> > >
> > >
> > > On Sun, Mar 3, 2024 at 7:33 PM Jacob Wujciak-Jens
> > > <ja...@voltrondata.com.invalid> wrote:
> > >
> > > > +1 (non-binding)
> > > >
> > > > On Mon, Mar 4, 2024 at 3:39 AM Yang Jiang <ya...@apache.org>
> > wrote:
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > On 2024/03/01 18:08:26 Daniël Heres wrote:
> > > > > > +1 (binding)
> > > > > >
> > > > > > On Fri, Mar 1, 2024, 19:05 Chao Sun <su...@apache.org> wrote:
> > > > > >
> > > > > > > +1 (non-binding)
> > > > > > >
> > > > > > > On Fri, Mar 1, 2024 at 9:59 AM QP Hou <qp...@neuralink.com>
> wrote:
> > > > > > >
> > > > > > > > +1 (binding)
> > > > > > > >
> > > > > > > > exciting milestone :)
> > > > > > > >
> > > > > > > > On Fri, Mar 1, 2024 at 9:49 AM David Li <lidavidm@apache.org
> >
> > > > wrote:
> > > > > > > > >
> > > > > > > > > +1
> > > > > > > > >
> > > > > > > > > On Fri, Mar 1, 2024, at 12:06, Jorge Cardoso Leitão wrote:
> > > > > > > > > > +1 - great work!!!
> > > > > > > > > >
> > > > > > > > > > On Fri, Mar 1, 2024 at 5:49 PM Micah Kornfield <
> > > > > > > emkornfield@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> +1 (binding)
> > > > > > > > > >>
> > > > > > > > > >> On Friday, March 1, 2024, Uwe L. Korn <uwelk@xhochy.com
> >
> > > > wrote:
> > > > > > > > > >>
> > > > > > > > > >> > +1 (binding)
> > > > > > > > > >> >
> > > > > > > > > >> > On Fri, Mar 1, 2024, at 2:37 PM, Andy Grove wrote:
> > > > > > > > > >> > > +1 (binding)
> > > > > > > > > >> > >
> > > > > > > > > >> > > On Fri, Mar 1, 2024 at 6:20 AM Weston Pace <
> > > > > > > weston.pace@gmail.com
> > > > > > > > >
> > > > > > > > > >> > wrote:
> > > > > > > > > >> > >
> > > > > > > > > >> > >> +1 (binding)
> > > > > > > > > >> > >>
> > > > > > > > > >> > >> On Fri, Mar 1, 2024 at 3:33 AM Andrew Lamb <
> > > > > > > alamb@influxdata.com
> > > > > > > > >
> > > > > > > > > >> > wrote:
> > > > > > > > > >> > >>
> > > > > > > > > >> > >> > Hello,
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > As we have discussed[1][2] I would like to vote
> on
> > > the
> > > > > > > > proposal to
> > > > > > > > > >> > >> > create a new Apache Top Level Project for
> > DataFusion.
> > > > The
> > > > > > > text
> > > > > > > > of
> > > > > > > > > >> the
> > > > > > > > > >> > >> > proposed resolution and background document is
> > > > > copy/pasted
> > > > > > > > below
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > If the community is in favor of this, we plan to
> > > submit
> > > > > the
> > > > > > > > > >> resolution
> > > > > > > > > >> > >> > to the ASF board for approval with the next Arrow
> > > > report
> > > > > (for
> > > > > > > > the
> > > > > > > > > >> > >> > April 2024 board meeting).
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > The vote will be open for at least 7 days.
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > [ ] +1 Accept this Proposal
> > > > > > > > > >> > >> > [ ] +0
> > > > > > > > > >> > >> > [ ] -1 Do not accept this proposal because...
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > Andrew
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > [1]
> > > > > > > > > >>
> > > > > https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> > > > > > > > > >> > >> > [2]
> > > > > > > > https://github.com/apache/arrow-datafusion/discussions/6475
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > ---------- Proposed Resolution ---------
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > Resolution to Create the Apache DataFusion
> Project
> > > from
> > > > > the
> > > > > > > > Apache
> > > > > > > > > >> > >> > Arrow DataFusion Sub Project
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> >
> > > > > =============================================================
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > X. Establish the Apache DataFusion Project
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > WHEREAS, the Board of Directors deems it to be in
> > the
> > > > > best
> > > > > > > > > >> > >> > interests of the Foundation and consistent with
> the
> > > > > > > > > >> > >> > Foundation's purpose to establish a Project
> > > Management
> > > > > > > > > >> > >> > Committee charged with the creation and
> maintenance
> > > of
> > > > > > > > > >> > >> > open-source software related to an extensible
> query
> > > > > engine
> > > > > > > > > >> > >> > for distribution at no charge to the public.
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > NOW, THEREFORE, BE IT RESOLVED, that a Project
> > > > Management
> > > > > > > > > >> > >> > Committee (PMC), to be known as the "Apache
> > > DataFusion
> > > > > > > > Project",
> > > > > > > > > >> > >> > be and hereby is established pursuant to Bylaws
> of
> > > the
> > > > > > > > > >> > >> > Foundation; and be it further
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > RESOLVED, that the Apache DataFusion Project be
> and
> > > > > hereby is
> > > > > > > > > >> > >> > responsible for the creation and maintenance of
> > > > software
> > > > > > > > > >> > >> > related to an extensible query engine; and be it
> > > > further
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > RESOLVED, that the office of "Vice President,
> > Apache
> > > > > > > > DataFusion" be
> > > > > > > > > >> > >> > and hereby is created, the person holding such
> > office
> > > > to
> > > > > > > > > >> > >> > serve at the direction of the Board of Directors
> as
> > > the
> > > > > chair
> > > > > > > > > >> > >> > of the Apache DataFusion Project, and to have
> > primary
> > > > > > > > responsibility
> > > > > > > > > >> > >> > for management of the projects within the scope
> of
> > > > > > > > > >> > >> > responsibility of the Apache DataFusion Project;
> > and
> > > be
> > > > > it
> > > > > > > > further
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > RESOLVED, that the persons listed immediately
> below
> > > be
> > > > > and
> > > > > > > > > >> > >> > hereby are appointed to serve as the initial
> > members
> > > of
> > > > > the
> > > > > > > > > >> > >> > Apache DataFusion Project:
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > * Andy Grove (agrove@apache.org)
> > > > > > > > > >> > >> > * Andrew Lamb (alamb@apache.org)
> > > > > > > > > >> > >> > * Daniël Heres (dheres@apache.org)
> > > > > > > > > >> > >> > * Jie Wen (jakevin@apache.org)
> > > > > > > > > >> > >> > * Kun Liu (liukun@apache.org)
> > > > > > > > > >> > >> > * Liang-Chi Hsieh (viirya@apache.org)
> > > > > > > > > >> > >> > * Qingping Hou: (houqp@apache.org)
> > > > > > > > > >> > >> > * Wes McKinney(wesm@apache.org)
> > > > > > > > > >> > >> > * Will Jones (wjones127@apache.org)
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > RESOLVED, that the Apache DataFusion Project be
> and
> > > > > hereby
> > > > > > > > > >> > >> > is tasked with the migration and rationalization
> of
> > > the
> > > > > > > Apache
> > > > > > > > > >> > >> > Arrow DataFusion sub-project; and be it further
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > RESOLVED, that all responsibilities pertaining to
> > the
> > > > > Apache
> > > > > > > > > >> > >> > Arrow DataFusion sub-project encumbered upon the
> > > > > > > > > >> > >> > Apache Arrow Project are hereafter discharged.
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > NOW, THEREFORE, BE IT FURTHER RESOLVED, that
> Andrew
> > > > Lamb
> > > > > > > > > >> > >> > be appointed to the office of Vice President,
> > Apache
> > > > > > > > DataFusion, to
> > > > > > > > > >> > >> > serve in accordance with and subject to the
> > direction
> > > > of
> > > > > the
> > > > > > > > > >> > >> > Board of Directors and the Bylaws of the
> Foundation
> > > > until
> > > > > > > > > >> > >> > death, resignation, retirement, removal or
> > > > > disqualification,
> > > > > > > > > >> > >> > or until a successor is appointed.
> > > > > > > > > >> > >> >
> > > > > =============================================================
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > -------
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > Summary:
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > We propose creating a new top level project,
> Apache
> > > > > > > > DataFusion, from
> > > > > > > > > >> > >> > an existing sub project of Apache Arrow to
> > facilitate
> > > > > > > > additional
> > > > > > > > > >> > >> > community and project growth.
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > Abstract
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > Apache Arrow DataFusion[1] is a very fast,
> > > extensible
> > > > > query
> > > > > > > > engine
> > > > > > > > > >> > >> > for building high-quality data-centric systems in
> > > Rust,
> > > > > using
> > > > > > > > the
> > > > > > > > > >> > >> > Apache Arrow in-memory format. DataFusion offers
> > SQL
> > > > and
> > > > > > > > Dataframe
> > > > > > > > > >> > >> > APIs, excellent performance, built-in support for
> > > CSV,
> > > > > > > Parquet,
> > > > > > > > > >> JSON,
> > > > > > > > > >> > >> > and Avro, extensive customization, and a great
> > > > community.
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > [1] https://arrow.apache.org/datafusion/
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > Proposal
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > We propose creating a new top level ASF project,
> > > Apache
> > > > > > > > DataFusion,
> > > > > > > > > >> > >> > governed initially by a subset of the Apache
> Arrow
> > > > > project’s
> > > > > > > > PMC and
> > > > > > > > > >> > >> > committers. The project’s code is in five
> existing
> > > git
> > > > > > > > repositories,
> > > > > > > > > >> > >> > currently governed by Apache Arrow which would
> > > transfer
> > > > > to
> > > > > > > the
> > > > > > > > new
> > > > > > > > > >> top
> > > > > > > > > >> > >> > level project.
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > Background
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > When DataFusion was initially donated to the
> Arrow
> > > > > project,
> > > > > > > it
> > > > > > > > did
> > > > > > > > > >> not
> > > > > > > > > >> > >> > have a strong enough community to stand on its
> own.
> > > It
> > > > > has
> > > > > > > > since
> > > > > > > > > >> grown
> > > > > > > > > >> > >> > significantly, and benefited immensely from being
> > > part
> > > > of
> > > > > > > > Arrow and
> > > > > > > > > >> > >> > nurturing of the Apache Way, and now has a
> > community
> > > > > strong
> > > > > > > > enough
> > > > > > > > > >> to
> > > > > > > > > >> > >> > stand on its own and that would benefit from
> > focused
> > > > > > > governance
> > > > > > > > > >> > >> > attention.
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > The community has discussed this idea publicly
> for
> > > more
> > > > > than
> > > > > > > 6
> > > > > > > > > >> months
> > > > > > > > > >> > >> >
> > > > > https://github.com/apache/arrow-datafusion/discussions/6475
> > > > > > > > and
> > > > > > > > > >> > >> > briefly on the Arrow PMC mailing list
> > > > > > > > > >> > >> >
> > > > > > > >
> > https://lists.apache.org/thread/thv2jdm6640l6gm88hy8jhk5prjww0cs
> > > .
> > > > > > > > > >> As
> > > > > > > > > >> > >> > of the time of this writing both had exclusively
> > > > positive
> > > > > > > > reactions.
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > Several current members of the Arrow PMC are both
> > > > active
> > > > > > > > > >> contributors
> > > > > > > > > >> > >> > to DataFusion and understand and believe deeply
> in
> > > the
> > > > > Apache
> > > > > > > > Way,
> > > > > > > > > >> and
> > > > > > > > > >> > >> > play active governance roles in the Arrow project
> > as
> > > > PMC
> > > > > > > > members and
> > > > > > > > > >> > >> > PMC chairs, guiding the community, and releasing
> > > > software
> > > > > > > > versions.
> > > > > > > > > >> > >> > With this existing governance experience and
> > > structure,
> > > > > the
> > > > > > > > new top
> > > > > > > > > >> > >> > level project will be able to function well
> > > immediately
> > > > > and
> > > > > > > > > >> > >> > independently.
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > Overview of DataFusion
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > Current Status
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > Meritocracy
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > DataFusion has been developed as part of Apache
> > Arrow
> > > > and
> > > > > > > thus
> > > > > > > > has
> > > > > > > > > >> > >> > been operating as a meritocracy. Many of the
> > > developers
> > > > > of
> > > > > > > > > >> DataFusion
> > > > > > > > > >> > >> > are Arrow PMC members or committers. The
> DataFusion
> > > > > project
> > > > > > > > plans to
> > > > > > > > > >> > >> > continue adding new PMC and committers as the
> > project
> > > > > matures
> > > > > > > > and
> > > > > > > > > >> > >> > grows.
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > Community
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > The DataFusion development team seeks to foster
> the
> > > > > > > > development and
> > > > > > > > > >> > >> > user communities. We hope that becoming a
> separate
> > > > > project
> > > > > > > > will help
> > > > > > > > > >> > >> > both Arrow and DataFusion communities by being
> more
> > > > > focused.
> > > > > > > > > >> Focused
> > > > > > > > > >> > >> > governance will make it easier to grow the
> > community
> > > of
> > > > > > > > committers
> > > > > > > > > >> and
> > > > > > > > > >> > >> > PMC members and make the organization more clear
> to
> > > > > others.
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > Alignment
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > The ASF is a natural host for DataFusion given
> that
> > > it
> > > > is
> > > > > > > > already
> > > > > > > > > >> the
> > > > > > > > > >> > >> > home of Arrow, Parquet, and other related
> > distributed
> > > > > system,
> > > > > > > > > >> storage
> > > > > > > > > >> > >> > and query execution systems.
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > Project Leadership
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > Proposed Initial PMC
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > We propose the following people as the initial
> > > > > DataFusion PMC
> > > > > > > > > >> members.
> > > > > > > > > >> > >> > This is a subset of the existing Arrow PMC
> members
> > > who
> > > > > > > > contribute to
> > > > > > > > > >> > >> > DataFusion
> > > > > > > https://people.apache.org/phonebook.html?unix=arrow
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > Andy Grove (agrove): Arrow PMC Chair
> > > > > > > > > >> > >> > Andrew Lamb (alamb): Arrow PMC, past Arrow PMC
> > Chair
> > > > > > > > > >> > >> > Daniël Heres (dheres) Arrow PMC
> > > > > > > > > >> > >> > Jie Wen (jakevin): Arrow PMC, Doris Committer
> > > > > > > > > >> > >> > Kun Liu (liukun): Arrow PMC, IoTDB PMC, TSFile
> PMC
> > > > > > > > > >> > >> > Liang-Chi Hsieh (viirya): Arrow PMC, Spark PMC
> > > > > > > > > >> > >> > Qingping Hou: (houqp): Arrow PMC
> > > > > > > > > >> > >> > Wes McKinney(wesm): Arrow PMC, ASF Member
> > > > > > > > > >> > >> > Will Jones (wjones127): Arrow PMC
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > We’d like to propose Andrew Lamb as the initial
> > > Chair,
> > > > > (and
> > > > > > > > thus ASF
> > > > > > > > > >> > >> > VP) for the DataFusion project.
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > Affiliations
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > Andy Grove (agrove): NVidia
> > > > > > > > > >> > >> > Andrew Lamb (alamb): InfluxData
> > > > > > > > > >> > >> > Daniël Heres (dheres): Coralogix
> > > > > > > > > >> > >> > Jie Wen (jakevin): SelectDB
> > > > > > > > > >> > >> > Kun Liu (liukun): Ebay
> > > > > > > > > >> > >> > Liang-Chi Hsieh (viirya): Apple
> > > > > > > > > >> > >> > Qingping Hou: (houqp): Scribd
> > > > > > > > > >> > >> > Wes McKinney(wesm): Posit
> > > > > > > > > >> > >> > Will Jones (wjones127): LanceDB
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > Proposed Initial Committers
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > In addition to the PMC, we propose the following
> > > people
> > > > > as
> > > > > > > the
> > > > > > > > > >> initial
> > > > > > > > > >> > >> > DataFusion committers. This is a subset of the
> > > existing
> > > > > Arrow
> > > > > > > > > >> > >> > committers who contribute to DataFusion
> > > > > > > > > >> > >> >
> > https://people.apache.org/phonebook.html?unix=arrow
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > akurmustafa Mustafa Akur (Synnada)
> > > > > > > > > >> > >> > avantgardner Brent Gardner (Coralogix)
> > > > > > > > > >> > >> > comphead Oleks V. (Unaffiliated)
> > > > > > > > > >> > >> > jayzhan Jay Zhan (Unaffiliated)
> > > > > > > > > >> > >> > jeffreyvo Jeffry Vo (Unaffiliated)
> > > > > > > > > >> > >> > jiayuliu Liu Jiayu (Airbnb)
> > > > > > > > > >> > >> > mete Metehan Yildirim (Synnada)
> > > > > > > > > >> > >> > mingmwang Wang Mingming (Ebay)
> > > > > > > > > >> > >> > mneumann Marco Neumann (InfluxData)
> > > > > > > > > >> > >> > nju_yaho Zhong Yanghong (Ebay)
> > > > > > > > > >> > >> > ozankabak Mehmet Ozan Kabak (Synnada)
> > > > > > > > > >> > >> > paddyhoran Paddy Horan (Assured Allies)
> > > > > > > > > >> > >> > rdettai Rémi Dettai (Cloudfuse)
> > > > > > > > > >> > >> > sunchao Chao Sun (Apple)
> > > > > > > > > >> > >> > thinkharderdev Daniel Harris (Coralogix)
> > > > > > > > > >> > >> > tustvold Raphael Taylor-Davies (InfluxData)
> > > > > > > > > >> > >> > wayne Ruihang Xia (Greptime)
> > > > > > > > > >> > >> > xudong963 Xudong Wang (ByteDance)
> > > > > > > > > >> > >> > yjshen Yijie Shen (Space and Time)
> > > > > > > > > >> > >> > yangjiang Yang Jiang (ebay)
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > Risk Assessments
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > Naming / Trademarks
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > As a sub-project of Arrow, the DataFusion name
> has
> > > been
> > > > > used
> > > > > > > > for
> > > > > > > > > >> over
> > > > > > > > > >> > >> > 4 years without any known issues. A podling name
> > > search
> > > > > did
> > > > > > > > not turn
> > > > > > > > > >> > >> > up any concerns and was approved:
> > > > > > > > > >> > >> >
> > > > > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > Legal / IP Clearance
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > All DataFusion code has either been donated to
> the
> > > > Arrow
> > > > > > > > project
> > > > > > > > > >> with
> > > > > > > > > >> > >> > appropriate IP clearance or has been developed
> > > > directly
> > > > > > > under
> > > > > > > > ASF
> > > > > > > > > >> > >> > processes and procedures. Thus creating a new top
> > > level
> > > > > > > project
> > > > > > > > > >> poses
> > > > > > > > > >> > >> > no new Legal or IP risks.
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > Code Extraction
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > The relevant code is already in 5 separate
> > > > repositories:
> > > > > > > > > >> > >> > https://github.com/apache/arrow-datafusion/
> > > > > > > > > >> > >> >
> https://github.com/apache/arrow-datafusion-python
> > > > > > > > > >> > >> > https://github.com/apache/arrow-ballista
> > > > > > > > > >> > >> > https://github.com/apache/arrow-ballista-python
> > > > > > > > > >> > >> > https://github.com/apache/arrow-datafusion-comet
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > We foresee no issues with code extraction and
> > propose
> > > > > these
> > > > > > > > > >> > >> > repositories be renamed to reflect top level
> > > projects
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > Note: https://github.com/apache/arrow-rs, the
> > Rust
> > > > > > > > implementation
> > > > > > > > > >> of
> > > > > > > > > >> > >> > Arrow, would remain part of the Arrow project.
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > Orphaned Products
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > DataFusion is known to be used in many open
> source
> > > and
> > > > > > > > commercial
> > > > > > > > > >> > >> > projects
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> https://arrow.apache.org/datafusion/user-guide/
> > > > > > > > > >> > introduction.html#known-users
> > > > > > > > > >> > >> > ,
> > > > > > > > > >> > >> > has had multiple commits daily for several years,
> > and
> > > > its
> > > > > > > > adoption
> > > > > > > > > >> and
> > > > > > > > > >> > >> > number of contributors appears to be growing. We
> do
> > > not
> > > > > > > > foresee the
> > > > > > > > > >> > >> > project being orphaned in the next several years.
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > Inexperience with Open Source
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > The proposed PMC has extensive experience with
> > Apache
> > > > > Arrow
> > > > > > > and
> > > > > > > > > >> other
> > > > > > > > > >> > >> > Apache projects, and includes PMC members, PMC
> > chairs
> > > > > and an
> > > > > > > > ASF
> > > > > > > > > >> > >> > Member. The DataFusion PMC and more experienced
> > > > > committers
> > > > > > > will
> > > > > > > > > >> > >> > continue to coach new community members who may
> be
> > > less
> > > > > > > > familiar
> > > > > > > > > >> with
> > > > > > > > > >> > >> > the Apache Way.
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > Homogeneous Developers
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > The 9 proposed PMC members are from 9 different
> > > > > employers and
> > > > > > > > the
> > > > > > > > > >> > >> > proposed committers are similarly distributed
> > across
> > > > > > > > affiliations.
> > > > > > > > > >> No
> > > > > > > > > >> > >> > specific entity employs more than 3 total
> proposed
> > > > > > > developers.
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > Reliance on Salaried Developers
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > A substantial amount of work on DataFusion has
> been
> > > by
> > > > > > > salaried
> > > > > > > > > >> > >> > developers, but it also has a long tradition of
> > > > > attracting
> > > > > > > > > >> > >> > contributions from students and hobbyists and we
> > plan
> > > > no
> > > > > > > > changes in
> > > > > > > > > >> > >> > contribution structure.
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > Relationships with Other Apache Products
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > DataFusion will obviously have a strong
> > relationship
> > > > > with the
> > > > > > > > Arrow
> > > > > > > > > >> > >> > project given the overlap in people. We don’t
> > foresee
> > > > > close
> > > > > > > > > >> > >> > collaboration with other projects at this time.
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > Cryptography
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > DataFusion does not directly support encryption
> and
> > > > > there are
> > > > > > > > no
> > > > > > > > > >> > >> > near-term plans to add support for encryption.
> > Users
> > > > who
> > > > > need
> > > > > > > > this
> > > > > > > > > >> > >> > functionality can use the extension APIs.
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > Required Resources
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > Mailing Lists
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > - private@datafusion.apache.org for private PMC
> > > > > discussions
> > > > > > > > (with
> > > > > > > > > >> > >> > moderated subscriptions)
> > > > > > > > > >> > >> > - dev@datafusion.apache.org
> > > > > > > > > >> > >> > - commits@datafusion.apache.org
> > > > > > > > > >> > >> > - user@datafusion.apache.org
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > Version Control
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > We propose to continue to use git for source
> > control
> > > > and
> > > > > > > > github for
> > > > > > > > > >> > >> > hosting and testing resources.
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > We also need to rename the github repositories to
> > > > > reflect the
> > > > > > > > new
> > > > > > > > > >> top
> > > > > > > > > >> > >> > level names:
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > https://github.com/apache/arrow-datafusion/ →
> > > > > > > > apache/datafusion
> > > > > > > > > >> > >> >
> https://github.com/apache/arrow-datafusion-python
> > →
> > > > > > > > > >> > >> > apache/datafusion-python
> > > > > > > > > >> > >> > https://github.com/apache/arrow-ballista →
> > > > > > > > > >> apache/datafusion-ballista
> > > > > > > > > >> > >> > https://github.com/apache/arrow-ballista-python
> →
> > > > > > > > > >> > >> > apache/datafusion-ballista-python
> > > > > > > > > >> > >> > https://github.com/apache/arrow-datafusion-comet
> →
> > > > > > > > > >> > >> apache/datafusion-comet
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > Issue Tracking
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > DataFusion would continue to use github for its
> > issue
> > > > > > > tracking
> > > > > > > > and
> > > > > > > > > >> > >> > communications
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > Other Resources
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > The existing repositories already make use of
> > > existing
> > > > > Apache
> > > > > > > > > >> > >> > infrastructure, and we expect no change in the
> > > initial
> > > > > > > resource
> > > > > > > > > >> usage.
> > > > > > > > > >> > >> > As the project continues to grow, we expect
> > continued
> > > > > > > > infrastructure
> > > > > > > > > >> > >> > demand growth.
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > FAQ: Has a sub project been promoted to a top
> level
> > > > > project
> > > > > > > > before?
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > Yes, and it appears to happen commonly. The Arrow
> > > > project
> > > > > > > > itself was
> > > > > > > > > >> > >> > created as a top level project from work that
> > started
> > > > in
> > > > > > > Apache
> > > > > > > > > >> Drill,
> > > > > > > > > >> > >> > and there are many sub projects of Hadoop that
> spun
> > > out
> > > > > as
> > > > > > > > their own
> > > > > > > > > >> > >> > top level projects such as Mahout, Avro and
> HBase:
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >>
> https://news.apache.org/foundation/entry/the_apache_
> > > > > > > > > >> > software_foundation_announces4
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >> > Related material:
> > > > > > > > > >> > >> > Name search request / research for DataFusion:
> > > > > > > > > >> > >> >
> > > > > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> > > > > > > > > >> > >> > Discussion about this proposal on the arrow
> mailing
> > > > list:
> > > > > > > > > >> > >> >
> > > > > > > >
> > https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> > > > > > > > > >> > >> > Discussion about which repositories on the arrow
> > > > mailing
> > > > > > > list:
> > > > > > > > > >> > >> >
> > > > > > > >
> > https://lists.apache.org/thread/ob3n0d9ky0bgrryl3xn39w9k566bq00q
> > > > > > > > > >> > >> > Discussion about initial PMC on the arrow mailing
> > > list:
> > > > > > > > > >> > >> >
> > > > > > > >
> > https://lists.apache.org/thread/pymrzcdw4qdptvby85f69rg3pcckl15b
> > > > > > > > > >> > >> > Discussion in github about creating a new
> > DataFusion
> > > > top
> > > > > > > level
> > > > > > > > > >> > >> > project:
> > > > > > > > > >>
> > https://github.com/apache/arrow-datafusion/discussions/6475
> > > > > > > > > >> > >> > Discussion about graduating on incubator list:
> > > > > > > > > >> > >> >
> > > > > > > >
> > https://lists.apache.org/thread/r4n73pmms1lv0jbohyx1o1z13d615t99
> > > > > > > > > >> > >> > Original Proposal for the Arrow project:
> > > > > > > > > >> > >> >
> > > > > > > >
> > https://lists.apache.org/thread/x2qzdwglm8pkqp9gv03bbgw17khl7pq3
> > > > > > > > > >> > >> >
> > > > > > > > > >> > >>
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [VOTE] Move Arrow DataFusion Subproject to new Top Level Apache Project
Posted by Peter Toth <pe...@gmail.com>.
+1 (non-binding)
Parth Chandra <pa...@apache.org> ezt írta (időpont: 2024. márc. 5., K,
18:15):
> +1 (non-binding)
>
> On Sun, Mar 3, 2024 at 8:04 PM Mehmet Ozan Kabak <oz...@synnada.ai> wrote:
>
> > +1 (non-binding)
> > --
> > *Mehmet Ozan Kabak*
> > Co-founder & CEO @ Synnada, Inc.
> >
> >
> > On Sun, Mar 3, 2024 at 7:33 PM Jacob Wujciak-Jens
> > <ja...@voltrondata.com.invalid> wrote:
> >
> > > +1 (non-binding)
> > >
> > > On Mon, Mar 4, 2024 at 3:39 AM Yang Jiang <ya...@apache.org>
> wrote:
> > >
> > > > +1 (non-binding)
> > > >
> > > > On 2024/03/01 18:08:26 Daniël Heres wrote:
> > > > > +1 (binding)
> > > > >
> > > > > On Fri, Mar 1, 2024, 19:05 Chao Sun <su...@apache.org> wrote:
> > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > On Fri, Mar 1, 2024 at 9:59 AM QP Hou <qp...@neuralink.com> wrote:
> > > > > >
> > > > > > > +1 (binding)
> > > > > > >
> > > > > > > exciting milestone :)
> > > > > > >
> > > > > > > On Fri, Mar 1, 2024 at 9:49 AM David Li <li...@apache.org>
> > > wrote:
> > > > > > > >
> > > > > > > > +1
> > > > > > > >
> > > > > > > > On Fri, Mar 1, 2024, at 12:06, Jorge Cardoso Leitão wrote:
> > > > > > > > > +1 - great work!!!
> > > > > > > > >
> > > > > > > > > On Fri, Mar 1, 2024 at 5:49 PM Micah Kornfield <
> > > > > > emkornfield@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> +1 (binding)
> > > > > > > > >>
> > > > > > > > >> On Friday, March 1, 2024, Uwe L. Korn <uw...@xhochy.com>
> > > wrote:
> > > > > > > > >>
> > > > > > > > >> > +1 (binding)
> > > > > > > > >> >
> > > > > > > > >> > On Fri, Mar 1, 2024, at 2:37 PM, Andy Grove wrote:
> > > > > > > > >> > > +1 (binding)
> > > > > > > > >> > >
> > > > > > > > >> > > On Fri, Mar 1, 2024 at 6:20 AM Weston Pace <
> > > > > > weston.pace@gmail.com
> > > > > > > >
> > > > > > > > >> > wrote:
> > > > > > > > >> > >
> > > > > > > > >> > >> +1 (binding)
> > > > > > > > >> > >>
> > > > > > > > >> > >> On Fri, Mar 1, 2024 at 3:33 AM Andrew Lamb <
> > > > > > alamb@influxdata.com
> > > > > > > >
> > > > > > > > >> > wrote:
> > > > > > > > >> > >>
> > > > > > > > >> > >> > Hello,
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > As we have discussed[1][2] I would like to vote on
> > the
> > > > > > > proposal to
> > > > > > > > >> > >> > create a new Apache Top Level Project for
> DataFusion.
> > > The
> > > > > > text
> > > > > > > of
> > > > > > > > >> the
> > > > > > > > >> > >> > proposed resolution and background document is
> > > > copy/pasted
> > > > > > > below
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > If the community is in favor of this, we plan to
> > submit
> > > > the
> > > > > > > > >> resolution
> > > > > > > > >> > >> > to the ASF board for approval with the next Arrow
> > > report
> > > > (for
> > > > > > > the
> > > > > > > > >> > >> > April 2024 board meeting).
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > The vote will be open for at least 7 days.
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > [ ] +1 Accept this Proposal
> > > > > > > > >> > >> > [ ] +0
> > > > > > > > >> > >> > [ ] -1 Do not accept this proposal because...
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > Andrew
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > [1]
> > > > > > > > >>
> > > > https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> > > > > > > > >> > >> > [2]
> > > > > > > https://github.com/apache/arrow-datafusion/discussions/6475
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > ---------- Proposed Resolution ---------
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > Resolution to Create the Apache DataFusion Project
> > from
> > > > the
> > > > > > > Apache
> > > > > > > > >> > >> > Arrow DataFusion Sub Project
> > > > > > > > >> > >> >
> > > > > > > > >> > >> >
> > > > =============================================================
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > X. Establish the Apache DataFusion Project
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > WHEREAS, the Board of Directors deems it to be in
> the
> > > > best
> > > > > > > > >> > >> > interests of the Foundation and consistent with the
> > > > > > > > >> > >> > Foundation's purpose to establish a Project
> > Management
> > > > > > > > >> > >> > Committee charged with the creation and maintenance
> > of
> > > > > > > > >> > >> > open-source software related to an extensible query
> > > > engine
> > > > > > > > >> > >> > for distribution at no charge to the public.
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > NOW, THEREFORE, BE IT RESOLVED, that a Project
> > > Management
> > > > > > > > >> > >> > Committee (PMC), to be known as the "Apache
> > DataFusion
> > > > > > > Project",
> > > > > > > > >> > >> > be and hereby is established pursuant to Bylaws of
> > the
> > > > > > > > >> > >> > Foundation; and be it further
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > RESOLVED, that the Apache DataFusion Project be and
> > > > hereby is
> > > > > > > > >> > >> > responsible for the creation and maintenance of
> > > software
> > > > > > > > >> > >> > related to an extensible query engine; and be it
> > > further
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > RESOLVED, that the office of "Vice President,
> Apache
> > > > > > > DataFusion" be
> > > > > > > > >> > >> > and hereby is created, the person holding such
> office
> > > to
> > > > > > > > >> > >> > serve at the direction of the Board of Directors as
> > the
> > > > chair
> > > > > > > > >> > >> > of the Apache DataFusion Project, and to have
> primary
> > > > > > > responsibility
> > > > > > > > >> > >> > for management of the projects within the scope of
> > > > > > > > >> > >> > responsibility of the Apache DataFusion Project;
> and
> > be
> > > > it
> > > > > > > further
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > RESOLVED, that the persons listed immediately below
> > be
> > > > and
> > > > > > > > >> > >> > hereby are appointed to serve as the initial
> members
> > of
> > > > the
> > > > > > > > >> > >> > Apache DataFusion Project:
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > * Andy Grove (agrove@apache.org)
> > > > > > > > >> > >> > * Andrew Lamb (alamb@apache.org)
> > > > > > > > >> > >> > * Daniël Heres (dheres@apache.org)
> > > > > > > > >> > >> > * Jie Wen (jakevin@apache.org)
> > > > > > > > >> > >> > * Kun Liu (liukun@apache.org)
> > > > > > > > >> > >> > * Liang-Chi Hsieh (viirya@apache.org)
> > > > > > > > >> > >> > * Qingping Hou: (houqp@apache.org)
> > > > > > > > >> > >> > * Wes McKinney(wesm@apache.org)
> > > > > > > > >> > >> > * Will Jones (wjones127@apache.org)
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > RESOLVED, that the Apache DataFusion Project be and
> > > > hereby
> > > > > > > > >> > >> > is tasked with the migration and rationalization of
> > the
> > > > > > Apache
> > > > > > > > >> > >> > Arrow DataFusion sub-project; and be it further
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > RESOLVED, that all responsibilities pertaining to
> the
> > > > Apache
> > > > > > > > >> > >> > Arrow DataFusion sub-project encumbered upon the
> > > > > > > > >> > >> > Apache Arrow Project are hereafter discharged.
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > NOW, THEREFORE, BE IT FURTHER RESOLVED, that Andrew
> > > Lamb
> > > > > > > > >> > >> > be appointed to the office of Vice President,
> Apache
> > > > > > > DataFusion, to
> > > > > > > > >> > >> > serve in accordance with and subject to the
> direction
> > > of
> > > > the
> > > > > > > > >> > >> > Board of Directors and the Bylaws of the Foundation
> > > until
> > > > > > > > >> > >> > death, resignation, retirement, removal or
> > > > disqualification,
> > > > > > > > >> > >> > or until a successor is appointed.
> > > > > > > > >> > >> >
> > > > =============================================================
> > > > > > > > >> > >> >
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > -------
> > > > > > > > >> > >> >
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > Summary:
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > We propose creating a new top level project, Apache
> > > > > > > DataFusion, from
> > > > > > > > >> > >> > an existing sub project of Apache Arrow to
> facilitate
> > > > > > > additional
> > > > > > > > >> > >> > community and project growth.
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > Abstract
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > Apache Arrow DataFusion[1] is a very fast,
> > extensible
> > > > query
> > > > > > > engine
> > > > > > > > >> > >> > for building high-quality data-centric systems in
> > Rust,
> > > > using
> > > > > > > the
> > > > > > > > >> > >> > Apache Arrow in-memory format. DataFusion offers
> SQL
> > > and
> > > > > > > Dataframe
> > > > > > > > >> > >> > APIs, excellent performance, built-in support for
> > CSV,
> > > > > > Parquet,
> > > > > > > > >> JSON,
> > > > > > > > >> > >> > and Avro, extensive customization, and a great
> > > community.
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > [1] https://arrow.apache.org/datafusion/
> > > > > > > > >> > >> >
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > Proposal
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > We propose creating a new top level ASF project,
> > Apache
> > > > > > > DataFusion,
> > > > > > > > >> > >> > governed initially by a subset of the Apache Arrow
> > > > project’s
> > > > > > > PMC and
> > > > > > > > >> > >> > committers. The project’s code is in five existing
> > git
> > > > > > > repositories,
> > > > > > > > >> > >> > currently governed by Apache Arrow which would
> > transfer
> > > > to
> > > > > > the
> > > > > > > new
> > > > > > > > >> top
> > > > > > > > >> > >> > level project.
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > Background
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > When DataFusion was initially donated to the Arrow
> > > > project,
> > > > > > it
> > > > > > > did
> > > > > > > > >> not
> > > > > > > > >> > >> > have a strong enough community to stand on its own.
> > It
> > > > has
> > > > > > > since
> > > > > > > > >> grown
> > > > > > > > >> > >> > significantly, and benefited immensely from being
> > part
> > > of
> > > > > > > Arrow and
> > > > > > > > >> > >> > nurturing of the Apache Way, and now has a
> community
> > > > strong
> > > > > > > enough
> > > > > > > > >> to
> > > > > > > > >> > >> > stand on its own and that would benefit from
> focused
> > > > > > governance
> > > > > > > > >> > >> > attention.
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > The community has discussed this idea publicly for
> > more
> > > > than
> > > > > > 6
> > > > > > > > >> months
> > > > > > > > >> > >> >
> > > > https://github.com/apache/arrow-datafusion/discussions/6475
> > > > > > > and
> > > > > > > > >> > >> > briefly on the Arrow PMC mailing list
> > > > > > > > >> > >> >
> > > > > > >
> https://lists.apache.org/thread/thv2jdm6640l6gm88hy8jhk5prjww0cs
> > .
> > > > > > > > >> As
> > > > > > > > >> > >> > of the time of this writing both had exclusively
> > > positive
> > > > > > > reactions.
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > Several current members of the Arrow PMC are both
> > > active
> > > > > > > > >> contributors
> > > > > > > > >> > >> > to DataFusion and understand and believe deeply in
> > the
> > > > Apache
> > > > > > > Way,
> > > > > > > > >> and
> > > > > > > > >> > >> > play active governance roles in the Arrow project
> as
> > > PMC
> > > > > > > members and
> > > > > > > > >> > >> > PMC chairs, guiding the community, and releasing
> > > software
> > > > > > > versions.
> > > > > > > > >> > >> > With this existing governance experience and
> > structure,
> > > > the
> > > > > > > new top
> > > > > > > > >> > >> > level project will be able to function well
> > immediately
> > > > and
> > > > > > > > >> > >> > independently.
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > Overview of DataFusion
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > Current Status
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > Meritocracy
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > DataFusion has been developed as part of Apache
> Arrow
> > > and
> > > > > > thus
> > > > > > > has
> > > > > > > > >> > >> > been operating as a meritocracy. Many of the
> > developers
> > > > of
> > > > > > > > >> DataFusion
> > > > > > > > >> > >> > are Arrow PMC members or committers. The DataFusion
> > > > project
> > > > > > > plans to
> > > > > > > > >> > >> > continue adding new PMC and committers as the
> project
> > > > matures
> > > > > > > and
> > > > > > > > >> > >> > grows.
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > Community
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > The DataFusion development team seeks to foster the
> > > > > > > development and
> > > > > > > > >> > >> > user communities. We hope that becoming a separate
> > > > project
> > > > > > > will help
> > > > > > > > >> > >> > both Arrow and DataFusion communities by being more
> > > > focused.
> > > > > > > > >> Focused
> > > > > > > > >> > >> > governance will make it easier to grow the
> community
> > of
> > > > > > > committers
> > > > > > > > >> and
> > > > > > > > >> > >> > PMC members and make the organization more clear to
> > > > others.
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > Alignment
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > The ASF is a natural host for DataFusion given that
> > it
> > > is
> > > > > > > already
> > > > > > > > >> the
> > > > > > > > >> > >> > home of Arrow, Parquet, and other related
> distributed
> > > > system,
> > > > > > > > >> storage
> > > > > > > > >> > >> > and query execution systems.
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > Project Leadership
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > Proposed Initial PMC
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > We propose the following people as the initial
> > > > DataFusion PMC
> > > > > > > > >> members.
> > > > > > > > >> > >> > This is a subset of the existing Arrow PMC members
> > who
> > > > > > > contribute to
> > > > > > > > >> > >> > DataFusion
> > > > > > https://people.apache.org/phonebook.html?unix=arrow
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > Andy Grove (agrove): Arrow PMC Chair
> > > > > > > > >> > >> > Andrew Lamb (alamb): Arrow PMC, past Arrow PMC
> Chair
> > > > > > > > >> > >> > Daniël Heres (dheres) Arrow PMC
> > > > > > > > >> > >> > Jie Wen (jakevin): Arrow PMC, Doris Committer
> > > > > > > > >> > >> > Kun Liu (liukun): Arrow PMC, IoTDB PMC, TSFile PMC
> > > > > > > > >> > >> > Liang-Chi Hsieh (viirya): Arrow PMC, Spark PMC
> > > > > > > > >> > >> > Qingping Hou: (houqp): Arrow PMC
> > > > > > > > >> > >> > Wes McKinney(wesm): Arrow PMC, ASF Member
> > > > > > > > >> > >> > Will Jones (wjones127): Arrow PMC
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > We’d like to propose Andrew Lamb as the initial
> > Chair,
> > > > (and
> > > > > > > thus ASF
> > > > > > > > >> > >> > VP) for the DataFusion project.
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > Affiliations
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > Andy Grove (agrove): NVidia
> > > > > > > > >> > >> > Andrew Lamb (alamb): InfluxData
> > > > > > > > >> > >> > Daniël Heres (dheres): Coralogix
> > > > > > > > >> > >> > Jie Wen (jakevin): SelectDB
> > > > > > > > >> > >> > Kun Liu (liukun): Ebay
> > > > > > > > >> > >> > Liang-Chi Hsieh (viirya): Apple
> > > > > > > > >> > >> > Qingping Hou: (houqp): Scribd
> > > > > > > > >> > >> > Wes McKinney(wesm): Posit
> > > > > > > > >> > >> > Will Jones (wjones127): LanceDB
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > Proposed Initial Committers
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > In addition to the PMC, we propose the following
> > people
> > > > as
> > > > > > the
> > > > > > > > >> initial
> > > > > > > > >> > >> > DataFusion committers. This is a subset of the
> > existing
> > > > Arrow
> > > > > > > > >> > >> > committers who contribute to DataFusion
> > > > > > > > >> > >> >
> https://people.apache.org/phonebook.html?unix=arrow
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > akurmustafa Mustafa Akur (Synnada)
> > > > > > > > >> > >> > avantgardner Brent Gardner (Coralogix)
> > > > > > > > >> > >> > comphead Oleks V. (Unaffiliated)
> > > > > > > > >> > >> > jayzhan Jay Zhan (Unaffiliated)
> > > > > > > > >> > >> > jeffreyvo Jeffry Vo (Unaffiliated)
> > > > > > > > >> > >> > jiayuliu Liu Jiayu (Airbnb)
> > > > > > > > >> > >> > mete Metehan Yildirim (Synnada)
> > > > > > > > >> > >> > mingmwang Wang Mingming (Ebay)
> > > > > > > > >> > >> > mneumann Marco Neumann (InfluxData)
> > > > > > > > >> > >> > nju_yaho Zhong Yanghong (Ebay)
> > > > > > > > >> > >> > ozankabak Mehmet Ozan Kabak (Synnada)
> > > > > > > > >> > >> > paddyhoran Paddy Horan (Assured Allies)
> > > > > > > > >> > >> > rdettai Rémi Dettai (Cloudfuse)
> > > > > > > > >> > >> > sunchao Chao Sun (Apple)
> > > > > > > > >> > >> > thinkharderdev Daniel Harris (Coralogix)
> > > > > > > > >> > >> > tustvold Raphael Taylor-Davies (InfluxData)
> > > > > > > > >> > >> > wayne Ruihang Xia (Greptime)
> > > > > > > > >> > >> > xudong963 Xudong Wang (ByteDance)
> > > > > > > > >> > >> > yjshen Yijie Shen (Space and Time)
> > > > > > > > >> > >> > yangjiang Yang Jiang (ebay)
> > > > > > > > >> > >> >
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > Risk Assessments
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > Naming / Trademarks
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > As a sub-project of Arrow, the DataFusion name has
> > been
> > > > used
> > > > > > > for
> > > > > > > > >> over
> > > > > > > > >> > >> > 4 years without any known issues. A podling name
> > search
> > > > did
> > > > > > > not turn
> > > > > > > > >> > >> > up any concerns and was approved:
> > > > > > > > >> > >> >
> > > > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > Legal / IP Clearance
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > All DataFusion code has either been donated to the
> > > Arrow
> > > > > > > project
> > > > > > > > >> with
> > > > > > > > >> > >> > appropriate IP clearance or has been developed
> > > directly
> > > > > > under
> > > > > > > ASF
> > > > > > > > >> > >> > processes and procedures. Thus creating a new top
> > level
> > > > > > project
> > > > > > > > >> poses
> > > > > > > > >> > >> > no new Legal or IP risks.
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > Code Extraction
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > The relevant code is already in 5 separate
> > > repositories:
> > > > > > > > >> > >> > https://github.com/apache/arrow-datafusion/
> > > > > > > > >> > >> > https://github.com/apache/arrow-datafusion-python
> > > > > > > > >> > >> > https://github.com/apache/arrow-ballista
> > > > > > > > >> > >> > https://github.com/apache/arrow-ballista-python
> > > > > > > > >> > >> > https://github.com/apache/arrow-datafusion-comet
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > We foresee no issues with code extraction and
> propose
> > > > these
> > > > > > > > >> > >> > repositories be renamed to reflect top level
> > projects
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > Note: https://github.com/apache/arrow-rs, the
> Rust
> > > > > > > implementation
> > > > > > > > >> of
> > > > > > > > >> > >> > Arrow, would remain part of the Arrow project.
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > Orphaned Products
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > DataFusion is known to be used in many open source
> > and
> > > > > > > commercial
> > > > > > > > >> > >> > projects
> > > > > > > > >> > >> >
> > > > > > > > >> > >> https://arrow.apache.org/datafusion/user-guide/
> > > > > > > > >> > introduction.html#known-users
> > > > > > > > >> > >> > ,
> > > > > > > > >> > >> > has had multiple commits daily for several years,
> and
> > > its
> > > > > > > adoption
> > > > > > > > >> and
> > > > > > > > >> > >> > number of contributors appears to be growing. We do
> > not
> > > > > > > foresee the
> > > > > > > > >> > >> > project being orphaned in the next several years.
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > Inexperience with Open Source
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > The proposed PMC has extensive experience with
> Apache
> > > > Arrow
> > > > > > and
> > > > > > > > >> other
> > > > > > > > >> > >> > Apache projects, and includes PMC members, PMC
> chairs
> > > > and an
> > > > > > > ASF
> > > > > > > > >> > >> > Member. The DataFusion PMC and more experienced
> > > > committers
> > > > > > will
> > > > > > > > >> > >> > continue to coach new community members who may be
> > less
> > > > > > > familiar
> > > > > > > > >> with
> > > > > > > > >> > >> > the Apache Way.
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > Homogeneous Developers
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > The 9 proposed PMC members are from 9 different
> > > > employers and
> > > > > > > the
> > > > > > > > >> > >> > proposed committers are similarly distributed
> across
> > > > > > > affiliations.
> > > > > > > > >> No
> > > > > > > > >> > >> > specific entity employs more than 3 total proposed
> > > > > > developers.
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > Reliance on Salaried Developers
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > A substantial amount of work on DataFusion has been
> > by
> > > > > > salaried
> > > > > > > > >> > >> > developers, but it also has a long tradition of
> > > > attracting
> > > > > > > > >> > >> > contributions from students and hobbyists and we
> plan
> > > no
> > > > > > > changes in
> > > > > > > > >> > >> > contribution structure.
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > Relationships with Other Apache Products
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > DataFusion will obviously have a strong
> relationship
> > > > with the
> > > > > > > Arrow
> > > > > > > > >> > >> > project given the overlap in people. We don’t
> foresee
> > > > close
> > > > > > > > >> > >> > collaboration with other projects at this time.
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > Cryptography
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > DataFusion does not directly support encryption and
> > > > there are
> > > > > > > no
> > > > > > > > >> > >> > near-term plans to add support for encryption.
> Users
> > > who
> > > > need
> > > > > > > this
> > > > > > > > >> > >> > functionality can use the extension APIs.
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > Required Resources
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > Mailing Lists
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > - private@datafusion.apache.org for private PMC
> > > > discussions
> > > > > > > (with
> > > > > > > > >> > >> > moderated subscriptions)
> > > > > > > > >> > >> > - dev@datafusion.apache.org
> > > > > > > > >> > >> > - commits@datafusion.apache.org
> > > > > > > > >> > >> > - user@datafusion.apache.org
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > Version Control
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > We propose to continue to use git for source
> control
> > > and
> > > > > > > github for
> > > > > > > > >> > >> > hosting and testing resources.
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > We also need to rename the github repositories to
> > > > reflect the
> > > > > > > new
> > > > > > > > >> top
> > > > > > > > >> > >> > level names:
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > https://github.com/apache/arrow-datafusion/ →
> > > > > > > apache/datafusion
> > > > > > > > >> > >> > https://github.com/apache/arrow-datafusion-python
> →
> > > > > > > > >> > >> > apache/datafusion-python
> > > > > > > > >> > >> > https://github.com/apache/arrow-ballista →
> > > > > > > > >> apache/datafusion-ballista
> > > > > > > > >> > >> > https://github.com/apache/arrow-ballista-python →
> > > > > > > > >> > >> > apache/datafusion-ballista-python
> > > > > > > > >> > >> > https://github.com/apache/arrow-datafusion-comet →
> > > > > > > > >> > >> apache/datafusion-comet
> > > > > > > > >> > >> >
> > > > > > > > >> > >> >
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > Issue Tracking
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > DataFusion would continue to use github for its
> issue
> > > > > > tracking
> > > > > > > and
> > > > > > > > >> > >> > communications
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > Other Resources
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > The existing repositories already make use of
> > existing
> > > > Apache
> > > > > > > > >> > >> > infrastructure, and we expect no change in the
> > initial
> > > > > > resource
> > > > > > > > >> usage.
> > > > > > > > >> > >> > As the project continues to grow, we expect
> continued
> > > > > > > infrastructure
> > > > > > > > >> > >> > demand growth.
> > > > > > > > >> > >> >
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > FAQ: Has a sub project been promoted to a top level
> > > > project
> > > > > > > before?
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > Yes, and it appears to happen commonly. The Arrow
> > > project
> > > > > > > itself was
> > > > > > > > >> > >> > created as a top level project from work that
> started
> > > in
> > > > > > Apache
> > > > > > > > >> Drill,
> > > > > > > > >> > >> > and there are many sub projects of Hadoop that spun
> > out
> > > > as
> > > > > > > their own
> > > > > > > > >> > >> > top level projects such as Mahout, Avro and HBase:
> > > > > > > > >> > >> >
> > > > > > > > >> > >> >
> > > > > > > > >> > >> https://news.apache.org/foundation/entry/the_apache_
> > > > > > > > >> > software_foundation_announces4
> > > > > > > > >> > >> >
> > > > > > > > >> > >> >
> > > > > > > > >> > >> >
> > > > > > > > >> > >> > Related material:
> > > > > > > > >> > >> > Name search request / research for DataFusion:
> > > > > > > > >> > >> >
> > > > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> > > > > > > > >> > >> > Discussion about this proposal on the arrow mailing
> > > list:
> > > > > > > > >> > >> >
> > > > > > >
> https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> > > > > > > > >> > >> > Discussion about which repositories on the arrow
> > > mailing
> > > > > > list:
> > > > > > > > >> > >> >
> > > > > > >
> https://lists.apache.org/thread/ob3n0d9ky0bgrryl3xn39w9k566bq00q
> > > > > > > > >> > >> > Discussion about initial PMC on the arrow mailing
> > list:
> > > > > > > > >> > >> >
> > > > > > >
> https://lists.apache.org/thread/pymrzcdw4qdptvby85f69rg3pcckl15b
> > > > > > > > >> > >> > Discussion in github about creating a new
> DataFusion
> > > top
> > > > > > level
> > > > > > > > >> > >> > project:
> > > > > > > > >>
> https://github.com/apache/arrow-datafusion/discussions/6475
> > > > > > > > >> > >> > Discussion about graduating on incubator list:
> > > > > > > > >> > >> >
> > > > > > >
> https://lists.apache.org/thread/r4n73pmms1lv0jbohyx1o1z13d615t99
> > > > > > > > >> > >> > Original Proposal for the Arrow project:
> > > > > > > > >> > >> >
> > > > > > >
> https://lists.apache.org/thread/x2qzdwglm8pkqp9gv03bbgw17khl7pq3
> > > > > > > > >> > >> >
> > > > > > > > >> > >>
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [VOTE] Move Arrow DataFusion Subproject to new Top Level Apache Project
Posted by Parth Chandra <pa...@apache.org>.
+1 (non-binding)
On Sun, Mar 3, 2024 at 8:04 PM Mehmet Ozan Kabak <oz...@synnada.ai> wrote:
> +1 (non-binding)
> --
> *Mehmet Ozan Kabak*
> Co-founder & CEO @ Synnada, Inc.
>
>
> On Sun, Mar 3, 2024 at 7:33 PM Jacob Wujciak-Jens
> <ja...@voltrondata.com.invalid> wrote:
>
> > +1 (non-binding)
> >
> > On Mon, Mar 4, 2024 at 3:39 AM Yang Jiang <ya...@apache.org> wrote:
> >
> > > +1 (non-binding)
> > >
> > > On 2024/03/01 18:08:26 Daniël Heres wrote:
> > > > +1 (binding)
> > > >
> > > > On Fri, Mar 1, 2024, 19:05 Chao Sun <su...@apache.org> wrote:
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > On Fri, Mar 1, 2024 at 9:59 AM QP Hou <qp...@neuralink.com> wrote:
> > > > >
> > > > > > +1 (binding)
> > > > > >
> > > > > > exciting milestone :)
> > > > > >
> > > > > > On Fri, Mar 1, 2024 at 9:49 AM David Li <li...@apache.org>
> > wrote:
> > > > > > >
> > > > > > > +1
> > > > > > >
> > > > > > > On Fri, Mar 1, 2024, at 12:06, Jorge Cardoso Leitão wrote:
> > > > > > > > +1 - great work!!!
> > > > > > > >
> > > > > > > > On Fri, Mar 1, 2024 at 5:49 PM Micah Kornfield <
> > > > > emkornfield@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> +1 (binding)
> > > > > > > >>
> > > > > > > >> On Friday, March 1, 2024, Uwe L. Korn <uw...@xhochy.com>
> > wrote:
> > > > > > > >>
> > > > > > > >> > +1 (binding)
> > > > > > > >> >
> > > > > > > >> > On Fri, Mar 1, 2024, at 2:37 PM, Andy Grove wrote:
> > > > > > > >> > > +1 (binding)
> > > > > > > >> > >
> > > > > > > >> > > On Fri, Mar 1, 2024 at 6:20 AM Weston Pace <
> > > > > weston.pace@gmail.com
> > > > > > >
> > > > > > > >> > wrote:
> > > > > > > >> > >
> > > > > > > >> > >> +1 (binding)
> > > > > > > >> > >>
> > > > > > > >> > >> On Fri, Mar 1, 2024 at 3:33 AM Andrew Lamb <
> > > > > alamb@influxdata.com
> > > > > > >
> > > > > > > >> > wrote:
> > > > > > > >> > >>
> > > > > > > >> > >> > Hello,
> > > > > > > >> > >> >
> > > > > > > >> > >> > As we have discussed[1][2] I would like to vote on
> the
> > > > > > proposal to
> > > > > > > >> > >> > create a new Apache Top Level Project for DataFusion.
> > The
> > > > > text
> > > > > > of
> > > > > > > >> the
> > > > > > > >> > >> > proposed resolution and background document is
> > > copy/pasted
> > > > > > below
> > > > > > > >> > >> >
> > > > > > > >> > >> > If the community is in favor of this, we plan to
> submit
> > > the
> > > > > > > >> resolution
> > > > > > > >> > >> > to the ASF board for approval with the next Arrow
> > report
> > > (for
> > > > > > the
> > > > > > > >> > >> > April 2024 board meeting).
> > > > > > > >> > >> >
> > > > > > > >> > >> > The vote will be open for at least 7 days.
> > > > > > > >> > >> >
> > > > > > > >> > >> > [ ] +1 Accept this Proposal
> > > > > > > >> > >> > [ ] +0
> > > > > > > >> > >> > [ ] -1 Do not accept this proposal because...
> > > > > > > >> > >> >
> > > > > > > >> > >> > Andrew
> > > > > > > >> > >> >
> > > > > > > >> > >> > [1]
> > > > > > > >>
> > > https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> > > > > > > >> > >> > [2]
> > > > > > https://github.com/apache/arrow-datafusion/discussions/6475
> > > > > > > >> > >> >
> > > > > > > >> > >> > ---------- Proposed Resolution ---------
> > > > > > > >> > >> >
> > > > > > > >> > >> > Resolution to Create the Apache DataFusion Project
> from
> > > the
> > > > > > Apache
> > > > > > > >> > >> > Arrow DataFusion Sub Project
> > > > > > > >> > >> >
> > > > > > > >> > >> >
> > > =============================================================
> > > > > > > >> > >> >
> > > > > > > >> > >> > X. Establish the Apache DataFusion Project
> > > > > > > >> > >> >
> > > > > > > >> > >> > WHEREAS, the Board of Directors deems it to be in the
> > > best
> > > > > > > >> > >> > interests of the Foundation and consistent with the
> > > > > > > >> > >> > Foundation's purpose to establish a Project
> Management
> > > > > > > >> > >> > Committee charged with the creation and maintenance
> of
> > > > > > > >> > >> > open-source software related to an extensible query
> > > engine
> > > > > > > >> > >> > for distribution at no charge to the public.
> > > > > > > >> > >> >
> > > > > > > >> > >> > NOW, THEREFORE, BE IT RESOLVED, that a Project
> > Management
> > > > > > > >> > >> > Committee (PMC), to be known as the "Apache
> DataFusion
> > > > > > Project",
> > > > > > > >> > >> > be and hereby is established pursuant to Bylaws of
> the
> > > > > > > >> > >> > Foundation; and be it further
> > > > > > > >> > >> >
> > > > > > > >> > >> > RESOLVED, that the Apache DataFusion Project be and
> > > hereby is
> > > > > > > >> > >> > responsible for the creation and maintenance of
> > software
> > > > > > > >> > >> > related to an extensible query engine; and be it
> > further
> > > > > > > >> > >> >
> > > > > > > >> > >> > RESOLVED, that the office of "Vice President, Apache
> > > > > > DataFusion" be
> > > > > > > >> > >> > and hereby is created, the person holding such office
> > to
> > > > > > > >> > >> > serve at the direction of the Board of Directors as
> the
> > > chair
> > > > > > > >> > >> > of the Apache DataFusion Project, and to have primary
> > > > > > responsibility
> > > > > > > >> > >> > for management of the projects within the scope of
> > > > > > > >> > >> > responsibility of the Apache DataFusion Project; and
> be
> > > it
> > > > > > further
> > > > > > > >> > >> >
> > > > > > > >> > >> > RESOLVED, that the persons listed immediately below
> be
> > > and
> > > > > > > >> > >> > hereby are appointed to serve as the initial members
> of
> > > the
> > > > > > > >> > >> > Apache DataFusion Project:
> > > > > > > >> > >> >
> > > > > > > >> > >> > * Andy Grove (agrove@apache.org)
> > > > > > > >> > >> > * Andrew Lamb (alamb@apache.org)
> > > > > > > >> > >> > * Daniël Heres (dheres@apache.org)
> > > > > > > >> > >> > * Jie Wen (jakevin@apache.org)
> > > > > > > >> > >> > * Kun Liu (liukun@apache.org)
> > > > > > > >> > >> > * Liang-Chi Hsieh (viirya@apache.org)
> > > > > > > >> > >> > * Qingping Hou: (houqp@apache.org)
> > > > > > > >> > >> > * Wes McKinney(wesm@apache.org)
> > > > > > > >> > >> > * Will Jones (wjones127@apache.org)
> > > > > > > >> > >> >
> > > > > > > >> > >> > RESOLVED, that the Apache DataFusion Project be and
> > > hereby
> > > > > > > >> > >> > is tasked with the migration and rationalization of
> the
> > > > > Apache
> > > > > > > >> > >> > Arrow DataFusion sub-project; and be it further
> > > > > > > >> > >> >
> > > > > > > >> > >> > RESOLVED, that all responsibilities pertaining to the
> > > Apache
> > > > > > > >> > >> > Arrow DataFusion sub-project encumbered upon the
> > > > > > > >> > >> > Apache Arrow Project are hereafter discharged.
> > > > > > > >> > >> >
> > > > > > > >> > >> > NOW, THEREFORE, BE IT FURTHER RESOLVED, that Andrew
> > Lamb
> > > > > > > >> > >> > be appointed to the office of Vice President, Apache
> > > > > > DataFusion, to
> > > > > > > >> > >> > serve in accordance with and subject to the direction
> > of
> > > the
> > > > > > > >> > >> > Board of Directors and the Bylaws of the Foundation
> > until
> > > > > > > >> > >> > death, resignation, retirement, removal or
> > > disqualification,
> > > > > > > >> > >> > or until a successor is appointed.
> > > > > > > >> > >> >
> > > =============================================================
> > > > > > > >> > >> >
> > > > > > > >> > >> >
> > > > > > > >> > >> > -------
> > > > > > > >> > >> >
> > > > > > > >> > >> >
> > > > > > > >> > >> > Summary:
> > > > > > > >> > >> >
> > > > > > > >> > >> > We propose creating a new top level project, Apache
> > > > > > DataFusion, from
> > > > > > > >> > >> > an existing sub project of Apache Arrow to facilitate
> > > > > > additional
> > > > > > > >> > >> > community and project growth.
> > > > > > > >> > >> >
> > > > > > > >> > >> > Abstract
> > > > > > > >> > >> >
> > > > > > > >> > >> > Apache Arrow DataFusion[1] is a very fast,
> extensible
> > > query
> > > > > > engine
> > > > > > > >> > >> > for building high-quality data-centric systems in
> Rust,
> > > using
> > > > > > the
> > > > > > > >> > >> > Apache Arrow in-memory format. DataFusion offers SQL
> > and
> > > > > > Dataframe
> > > > > > > >> > >> > APIs, excellent performance, built-in support for
> CSV,
> > > > > Parquet,
> > > > > > > >> JSON,
> > > > > > > >> > >> > and Avro, extensive customization, and a great
> > community.
> > > > > > > >> > >> >
> > > > > > > >> > >> > [1] https://arrow.apache.org/datafusion/
> > > > > > > >> > >> >
> > > > > > > >> > >> >
> > > > > > > >> > >> > Proposal
> > > > > > > >> > >> >
> > > > > > > >> > >> > We propose creating a new top level ASF project,
> Apache
> > > > > > DataFusion,
> > > > > > > >> > >> > governed initially by a subset of the Apache Arrow
> > > project’s
> > > > > > PMC and
> > > > > > > >> > >> > committers. The project’s code is in five existing
> git
> > > > > > repositories,
> > > > > > > >> > >> > currently governed by Apache Arrow which would
> transfer
> > > to
> > > > > the
> > > > > > new
> > > > > > > >> top
> > > > > > > >> > >> > level project.
> > > > > > > >> > >> >
> > > > > > > >> > >> > Background
> > > > > > > >> > >> >
> > > > > > > >> > >> > When DataFusion was initially donated to the Arrow
> > > project,
> > > > > it
> > > > > > did
> > > > > > > >> not
> > > > > > > >> > >> > have a strong enough community to stand on its own.
> It
> > > has
> > > > > > since
> > > > > > > >> grown
> > > > > > > >> > >> > significantly, and benefited immensely from being
> part
> > of
> > > > > > Arrow and
> > > > > > > >> > >> > nurturing of the Apache Way, and now has a community
> > > strong
> > > > > > enough
> > > > > > > >> to
> > > > > > > >> > >> > stand on its own and that would benefit from focused
> > > > > governance
> > > > > > > >> > >> > attention.
> > > > > > > >> > >> >
> > > > > > > >> > >> > The community has discussed this idea publicly for
> more
> > > than
> > > > > 6
> > > > > > > >> months
> > > > > > > >> > >> >
> > > https://github.com/apache/arrow-datafusion/discussions/6475
> > > > > > and
> > > > > > > >> > >> > briefly on the Arrow PMC mailing list
> > > > > > > >> > >> >
> > > > > > https://lists.apache.org/thread/thv2jdm6640l6gm88hy8jhk5prjww0cs
> .
> > > > > > > >> As
> > > > > > > >> > >> > of the time of this writing both had exclusively
> > positive
> > > > > > reactions.
> > > > > > > >> > >> >
> > > > > > > >> > >> > Several current members of the Arrow PMC are both
> > active
> > > > > > > >> contributors
> > > > > > > >> > >> > to DataFusion and understand and believe deeply in
> the
> > > Apache
> > > > > > Way,
> > > > > > > >> and
> > > > > > > >> > >> > play active governance roles in the Arrow project as
> > PMC
> > > > > > members and
> > > > > > > >> > >> > PMC chairs, guiding the community, and releasing
> > software
> > > > > > versions.
> > > > > > > >> > >> > With this existing governance experience and
> structure,
> > > the
> > > > > > new top
> > > > > > > >> > >> > level project will be able to function well
> immediately
> > > and
> > > > > > > >> > >> > independently.
> > > > > > > >> > >> >
> > > > > > > >> > >> > Overview of DataFusion
> > > > > > > >> > >> >
> > > > > > > >> > >> > Current Status
> > > > > > > >> > >> >
> > > > > > > >> > >> > Meritocracy
> > > > > > > >> > >> >
> > > > > > > >> > >> > DataFusion has been developed as part of Apache Arrow
> > and
> > > > > thus
> > > > > > has
> > > > > > > >> > >> > been operating as a meritocracy. Many of the
> developers
> > > of
> > > > > > > >> DataFusion
> > > > > > > >> > >> > are Arrow PMC members or committers. The DataFusion
> > > project
> > > > > > plans to
> > > > > > > >> > >> > continue adding new PMC and committers as the project
> > > matures
> > > > > > and
> > > > > > > >> > >> > grows.
> > > > > > > >> > >> >
> > > > > > > >> > >> > Community
> > > > > > > >> > >> >
> > > > > > > >> > >> > The DataFusion development team seeks to foster the
> > > > > > development and
> > > > > > > >> > >> > user communities. We hope that becoming a separate
> > > project
> > > > > > will help
> > > > > > > >> > >> > both Arrow and DataFusion communities by being more
> > > focused.
> > > > > > > >> Focused
> > > > > > > >> > >> > governance will make it easier to grow the community
> of
> > > > > > committers
> > > > > > > >> and
> > > > > > > >> > >> > PMC members and make the organization more clear to
> > > others.
> > > > > > > >> > >> >
> > > > > > > >> > >> > Alignment
> > > > > > > >> > >> >
> > > > > > > >> > >> > The ASF is a natural host for DataFusion given that
> it
> > is
> > > > > > already
> > > > > > > >> the
> > > > > > > >> > >> > home of Arrow, Parquet, and other related distributed
> > > system,
> > > > > > > >> storage
> > > > > > > >> > >> > and query execution systems.
> > > > > > > >> > >> >
> > > > > > > >> > >> > Project Leadership
> > > > > > > >> > >> >
> > > > > > > >> > >> > Proposed Initial PMC
> > > > > > > >> > >> >
> > > > > > > >> > >> > We propose the following people as the initial
> > > DataFusion PMC
> > > > > > > >> members.
> > > > > > > >> > >> > This is a subset of the existing Arrow PMC members
> who
> > > > > > contribute to
> > > > > > > >> > >> > DataFusion
> > > > > https://people.apache.org/phonebook.html?unix=arrow
> > > > > > > >> > >> >
> > > > > > > >> > >> > Andy Grove (agrove): Arrow PMC Chair
> > > > > > > >> > >> > Andrew Lamb (alamb): Arrow PMC, past Arrow PMC Chair
> > > > > > > >> > >> > Daniël Heres (dheres) Arrow PMC
> > > > > > > >> > >> > Jie Wen (jakevin): Arrow PMC, Doris Committer
> > > > > > > >> > >> > Kun Liu (liukun): Arrow PMC, IoTDB PMC, TSFile PMC
> > > > > > > >> > >> > Liang-Chi Hsieh (viirya): Arrow PMC, Spark PMC
> > > > > > > >> > >> > Qingping Hou: (houqp): Arrow PMC
> > > > > > > >> > >> > Wes McKinney(wesm): Arrow PMC, ASF Member
> > > > > > > >> > >> > Will Jones (wjones127): Arrow PMC
> > > > > > > >> > >> >
> > > > > > > >> > >> > We’d like to propose Andrew Lamb as the initial
> Chair,
> > > (and
> > > > > > thus ASF
> > > > > > > >> > >> > VP) for the DataFusion project.
> > > > > > > >> > >> >
> > > > > > > >> > >> > Affiliations
> > > > > > > >> > >> >
> > > > > > > >> > >> > Andy Grove (agrove): NVidia
> > > > > > > >> > >> > Andrew Lamb (alamb): InfluxData
> > > > > > > >> > >> > Daniël Heres (dheres): Coralogix
> > > > > > > >> > >> > Jie Wen (jakevin): SelectDB
> > > > > > > >> > >> > Kun Liu (liukun): Ebay
> > > > > > > >> > >> > Liang-Chi Hsieh (viirya): Apple
> > > > > > > >> > >> > Qingping Hou: (houqp): Scribd
> > > > > > > >> > >> > Wes McKinney(wesm): Posit
> > > > > > > >> > >> > Will Jones (wjones127): LanceDB
> > > > > > > >> > >> >
> > > > > > > >> > >> > Proposed Initial Committers
> > > > > > > >> > >> >
> > > > > > > >> > >> > In addition to the PMC, we propose the following
> people
> > > as
> > > > > the
> > > > > > > >> initial
> > > > > > > >> > >> > DataFusion committers. This is a subset of the
> existing
> > > Arrow
> > > > > > > >> > >> > committers who contribute to DataFusion
> > > > > > > >> > >> > https://people.apache.org/phonebook.html?unix=arrow
> > > > > > > >> > >> >
> > > > > > > >> > >> > akurmustafa Mustafa Akur (Synnada)
> > > > > > > >> > >> > avantgardner Brent Gardner (Coralogix)
> > > > > > > >> > >> > comphead Oleks V. (Unaffiliated)
> > > > > > > >> > >> > jayzhan Jay Zhan (Unaffiliated)
> > > > > > > >> > >> > jeffreyvo Jeffry Vo (Unaffiliated)
> > > > > > > >> > >> > jiayuliu Liu Jiayu (Airbnb)
> > > > > > > >> > >> > mete Metehan Yildirim (Synnada)
> > > > > > > >> > >> > mingmwang Wang Mingming (Ebay)
> > > > > > > >> > >> > mneumann Marco Neumann (InfluxData)
> > > > > > > >> > >> > nju_yaho Zhong Yanghong (Ebay)
> > > > > > > >> > >> > ozankabak Mehmet Ozan Kabak (Synnada)
> > > > > > > >> > >> > paddyhoran Paddy Horan (Assured Allies)
> > > > > > > >> > >> > rdettai Rémi Dettai (Cloudfuse)
> > > > > > > >> > >> > sunchao Chao Sun (Apple)
> > > > > > > >> > >> > thinkharderdev Daniel Harris (Coralogix)
> > > > > > > >> > >> > tustvold Raphael Taylor-Davies (InfluxData)
> > > > > > > >> > >> > wayne Ruihang Xia (Greptime)
> > > > > > > >> > >> > xudong963 Xudong Wang (ByteDance)
> > > > > > > >> > >> > yjshen Yijie Shen (Space and Time)
> > > > > > > >> > >> > yangjiang Yang Jiang (ebay)
> > > > > > > >> > >> >
> > > > > > > >> > >> >
> > > > > > > >> > >> > Risk Assessments
> > > > > > > >> > >> >
> > > > > > > >> > >> > Naming / Trademarks
> > > > > > > >> > >> >
> > > > > > > >> > >> > As a sub-project of Arrow, the DataFusion name has
> been
> > > used
> > > > > > for
> > > > > > > >> over
> > > > > > > >> > >> > 4 years without any known issues. A podling name
> search
> > > did
> > > > > > not turn
> > > > > > > >> > >> > up any concerns and was approved:
> > > > > > > >> > >> >
> > > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> > > > > > > >> > >> >
> > > > > > > >> > >> > Legal / IP Clearance
> > > > > > > >> > >> >
> > > > > > > >> > >> > All DataFusion code has either been donated to the
> > Arrow
> > > > > > project
> > > > > > > >> with
> > > > > > > >> > >> > appropriate IP clearance or has been developed
> > directly
> > > > > under
> > > > > > ASF
> > > > > > > >> > >> > processes and procedures. Thus creating a new top
> level
> > > > > project
> > > > > > > >> poses
> > > > > > > >> > >> > no new Legal or IP risks.
> > > > > > > >> > >> >
> > > > > > > >> > >> > Code Extraction
> > > > > > > >> > >> >
> > > > > > > >> > >> > The relevant code is already in 5 separate
> > repositories:
> > > > > > > >> > >> > https://github.com/apache/arrow-datafusion/
> > > > > > > >> > >> > https://github.com/apache/arrow-datafusion-python
> > > > > > > >> > >> > https://github.com/apache/arrow-ballista
> > > > > > > >> > >> > https://github.com/apache/arrow-ballista-python
> > > > > > > >> > >> > https://github.com/apache/arrow-datafusion-comet
> > > > > > > >> > >> >
> > > > > > > >> > >> > We foresee no issues with code extraction and propose
> > > these
> > > > > > > >> > >> > repositories be renamed to reflect top level
> projects
> > > > > > > >> > >> >
> > > > > > > >> > >> > Note: https://github.com/apache/arrow-rs, the Rust
> > > > > > implementation
> > > > > > > >> of
> > > > > > > >> > >> > Arrow, would remain part of the Arrow project.
> > > > > > > >> > >> >
> > > > > > > >> > >> > Orphaned Products
> > > > > > > >> > >> >
> > > > > > > >> > >> > DataFusion is known to be used in many open source
> and
> > > > > > commercial
> > > > > > > >> > >> > projects
> > > > > > > >> > >> >
> > > > > > > >> > >> https://arrow.apache.org/datafusion/user-guide/
> > > > > > > >> > introduction.html#known-users
> > > > > > > >> > >> > ,
> > > > > > > >> > >> > has had multiple commits daily for several years, and
> > its
> > > > > > adoption
> > > > > > > >> and
> > > > > > > >> > >> > number of contributors appears to be growing. We do
> not
> > > > > > foresee the
> > > > > > > >> > >> > project being orphaned in the next several years.
> > > > > > > >> > >> >
> > > > > > > >> > >> > Inexperience with Open Source
> > > > > > > >> > >> >
> > > > > > > >> > >> > The proposed PMC has extensive experience with Apache
> > > Arrow
> > > > > and
> > > > > > > >> other
> > > > > > > >> > >> > Apache projects, and includes PMC members, PMC chairs
> > > and an
> > > > > > ASF
> > > > > > > >> > >> > Member. The DataFusion PMC and more experienced
> > > committers
> > > > > will
> > > > > > > >> > >> > continue to coach new community members who may be
> less
> > > > > > familiar
> > > > > > > >> with
> > > > > > > >> > >> > the Apache Way.
> > > > > > > >> > >> >
> > > > > > > >> > >> > Homogeneous Developers
> > > > > > > >> > >> >
> > > > > > > >> > >> > The 9 proposed PMC members are from 9 different
> > > employers and
> > > > > > the
> > > > > > > >> > >> > proposed committers are similarly distributed across
> > > > > > affiliations.
> > > > > > > >> No
> > > > > > > >> > >> > specific entity employs more than 3 total proposed
> > > > > developers.
> > > > > > > >> > >> >
> > > > > > > >> > >> > Reliance on Salaried Developers
> > > > > > > >> > >> >
> > > > > > > >> > >> > A substantial amount of work on DataFusion has been
> by
> > > > > salaried
> > > > > > > >> > >> > developers, but it also has a long tradition of
> > > attracting
> > > > > > > >> > >> > contributions from students and hobbyists and we plan
> > no
> > > > > > changes in
> > > > > > > >> > >> > contribution structure.
> > > > > > > >> > >> >
> > > > > > > >> > >> > Relationships with Other Apache Products
> > > > > > > >> > >> >
> > > > > > > >> > >> > DataFusion will obviously have a strong relationship
> > > with the
> > > > > > Arrow
> > > > > > > >> > >> > project given the overlap in people. We don’t foresee
> > > close
> > > > > > > >> > >> > collaboration with other projects at this time.
> > > > > > > >> > >> >
> > > > > > > >> > >> > Cryptography
> > > > > > > >> > >> >
> > > > > > > >> > >> > DataFusion does not directly support encryption and
> > > there are
> > > > > > no
> > > > > > > >> > >> > near-term plans to add support for encryption. Users
> > who
> > > need
> > > > > > this
> > > > > > > >> > >> > functionality can use the extension APIs.
> > > > > > > >> > >> >
> > > > > > > >> > >> > Required Resources
> > > > > > > >> > >> >
> > > > > > > >> > >> > Mailing Lists
> > > > > > > >> > >> >
> > > > > > > >> > >> > - private@datafusion.apache.org for private PMC
> > > discussions
> > > > > > (with
> > > > > > > >> > >> > moderated subscriptions)
> > > > > > > >> > >> > - dev@datafusion.apache.org
> > > > > > > >> > >> > - commits@datafusion.apache.org
> > > > > > > >> > >> > - user@datafusion.apache.org
> > > > > > > >> > >> >
> > > > > > > >> > >> > Version Control
> > > > > > > >> > >> >
> > > > > > > >> > >> > We propose to continue to use git for source control
> > and
> > > > > > github for
> > > > > > > >> > >> > hosting and testing resources.
> > > > > > > >> > >> >
> > > > > > > >> > >> > We also need to rename the github repositories to
> > > reflect the
> > > > > > new
> > > > > > > >> top
> > > > > > > >> > >> > level names:
> > > > > > > >> > >> >
> > > > > > > >> > >> > https://github.com/apache/arrow-datafusion/ →
> > > > > > apache/datafusion
> > > > > > > >> > >> > https://github.com/apache/arrow-datafusion-python →
> > > > > > > >> > >> > apache/datafusion-python
> > > > > > > >> > >> > https://github.com/apache/arrow-ballista →
> > > > > > > >> apache/datafusion-ballista
> > > > > > > >> > >> > https://github.com/apache/arrow-ballista-python →
> > > > > > > >> > >> > apache/datafusion-ballista-python
> > > > > > > >> > >> > https://github.com/apache/arrow-datafusion-comet →
> > > > > > > >> > >> apache/datafusion-comet
> > > > > > > >> > >> >
> > > > > > > >> > >> >
> > > > > > > >> > >> >
> > > > > > > >> > >> > Issue Tracking
> > > > > > > >> > >> >
> > > > > > > >> > >> > DataFusion would continue to use github for its issue
> > > > > tracking
> > > > > > and
> > > > > > > >> > >> > communications
> > > > > > > >> > >> >
> > > > > > > >> > >> > Other Resources
> > > > > > > >> > >> >
> > > > > > > >> > >> > The existing repositories already make use of
> existing
> > > Apache
> > > > > > > >> > >> > infrastructure, and we expect no change in the
> initial
> > > > > resource
> > > > > > > >> usage.
> > > > > > > >> > >> > As the project continues to grow, we expect continued
> > > > > > infrastructure
> > > > > > > >> > >> > demand growth.
> > > > > > > >> > >> >
> > > > > > > >> > >> >
> > > > > > > >> > >> > FAQ: Has a sub project been promoted to a top level
> > > project
> > > > > > before?
> > > > > > > >> > >> >
> > > > > > > >> > >> > Yes, and it appears to happen commonly. The Arrow
> > project
> > > > > > itself was
> > > > > > > >> > >> > created as a top level project from work that started
> > in
> > > > > Apache
> > > > > > > >> Drill,
> > > > > > > >> > >> > and there are many sub projects of Hadoop that spun
> out
> > > as
> > > > > > their own
> > > > > > > >> > >> > top level projects such as Mahout, Avro and HBase:
> > > > > > > >> > >> >
> > > > > > > >> > >> >
> > > > > > > >> > >> https://news.apache.org/foundation/entry/the_apache_
> > > > > > > >> > software_foundation_announces4
> > > > > > > >> > >> >
> > > > > > > >> > >> >
> > > > > > > >> > >> >
> > > > > > > >> > >> > Related material:
> > > > > > > >> > >> > Name search request / research for DataFusion:
> > > > > > > >> > >> >
> > > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> > > > > > > >> > >> > Discussion about this proposal on the arrow mailing
> > list:
> > > > > > > >> > >> >
> > > > > > https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> > > > > > > >> > >> > Discussion about which repositories on the arrow
> > mailing
> > > > > list:
> > > > > > > >> > >> >
> > > > > > https://lists.apache.org/thread/ob3n0d9ky0bgrryl3xn39w9k566bq00q
> > > > > > > >> > >> > Discussion about initial PMC on the arrow mailing
> list:
> > > > > > > >> > >> >
> > > > > > https://lists.apache.org/thread/pymrzcdw4qdptvby85f69rg3pcckl15b
> > > > > > > >> > >> > Discussion in github about creating a new DataFusion
> > top
> > > > > level
> > > > > > > >> > >> > project:
> > > > > > > >> https://github.com/apache/arrow-datafusion/discussions/6475
> > > > > > > >> > >> > Discussion about graduating on incubator list:
> > > > > > > >> > >> >
> > > > > > https://lists.apache.org/thread/r4n73pmms1lv0jbohyx1o1z13d615t99
> > > > > > > >> > >> > Original Proposal for the Arrow project:
> > > > > > > >> > >> >
> > > > > > https://lists.apache.org/thread/x2qzdwglm8pkqp9gv03bbgw17khl7pq3
> > > > > > > >> > >> >
> > > > > > > >> > >>
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [VOTE] Move Arrow DataFusion Subproject to new Top Level Apache Project
Posted by Mehmet Ozan Kabak <oz...@synnada.ai>.
+1 (non-binding)
--
*Mehmet Ozan Kabak*
Co-founder & CEO @ Synnada, Inc.
On Sun, Mar 3, 2024 at 7:33 PM Jacob Wujciak-Jens
<ja...@voltrondata.com.invalid> wrote:
> +1 (non-binding)
>
> On Mon, Mar 4, 2024 at 3:39 AM Yang Jiang <ya...@apache.org> wrote:
>
> > +1 (non-binding)
> >
> > On 2024/03/01 18:08:26 Daniël Heres wrote:
> > > +1 (binding)
> > >
> > > On Fri, Mar 1, 2024, 19:05 Chao Sun <su...@apache.org> wrote:
> > >
> > > > +1 (non-binding)
> > > >
> > > > On Fri, Mar 1, 2024 at 9:59 AM QP Hou <qp...@neuralink.com> wrote:
> > > >
> > > > > +1 (binding)
> > > > >
> > > > > exciting milestone :)
> > > > >
> > > > > On Fri, Mar 1, 2024 at 9:49 AM David Li <li...@apache.org>
> wrote:
> > > > > >
> > > > > > +1
> > > > > >
> > > > > > On Fri, Mar 1, 2024, at 12:06, Jorge Cardoso Leitão wrote:
> > > > > > > +1 - great work!!!
> > > > > > >
> > > > > > > On Fri, Mar 1, 2024 at 5:49 PM Micah Kornfield <
> > > > emkornfield@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> +1 (binding)
> > > > > > >>
> > > > > > >> On Friday, March 1, 2024, Uwe L. Korn <uw...@xhochy.com>
> wrote:
> > > > > > >>
> > > > > > >> > +1 (binding)
> > > > > > >> >
> > > > > > >> > On Fri, Mar 1, 2024, at 2:37 PM, Andy Grove wrote:
> > > > > > >> > > +1 (binding)
> > > > > > >> > >
> > > > > > >> > > On Fri, Mar 1, 2024 at 6:20 AM Weston Pace <
> > > > weston.pace@gmail.com
> > > > > >
> > > > > > >> > wrote:
> > > > > > >> > >
> > > > > > >> > >> +1 (binding)
> > > > > > >> > >>
> > > > > > >> > >> On Fri, Mar 1, 2024 at 3:33 AM Andrew Lamb <
> > > > alamb@influxdata.com
> > > > > >
> > > > > > >> > wrote:
> > > > > > >> > >>
> > > > > > >> > >> > Hello,
> > > > > > >> > >> >
> > > > > > >> > >> > As we have discussed[1][2] I would like to vote on the
> > > > > proposal to
> > > > > > >> > >> > create a new Apache Top Level Project for DataFusion.
> The
> > > > text
> > > > > of
> > > > > > >> the
> > > > > > >> > >> > proposed resolution and background document is
> > copy/pasted
> > > > > below
> > > > > > >> > >> >
> > > > > > >> > >> > If the community is in favor of this, we plan to submit
> > the
> > > > > > >> resolution
> > > > > > >> > >> > to the ASF board for approval with the next Arrow
> report
> > (for
> > > > > the
> > > > > > >> > >> > April 2024 board meeting).
> > > > > > >> > >> >
> > > > > > >> > >> > The vote will be open for at least 7 days.
> > > > > > >> > >> >
> > > > > > >> > >> > [ ] +1 Accept this Proposal
> > > > > > >> > >> > [ ] +0
> > > > > > >> > >> > [ ] -1 Do not accept this proposal because...
> > > > > > >> > >> >
> > > > > > >> > >> > Andrew
> > > > > > >> > >> >
> > > > > > >> > >> > [1]
> > > > > > >>
> > https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> > > > > > >> > >> > [2]
> > > > > https://github.com/apache/arrow-datafusion/discussions/6475
> > > > > > >> > >> >
> > > > > > >> > >> > ---------- Proposed Resolution ---------
> > > > > > >> > >> >
> > > > > > >> > >> > Resolution to Create the Apache DataFusion Project from
> > the
> > > > > Apache
> > > > > > >> > >> > Arrow DataFusion Sub Project
> > > > > > >> > >> >
> > > > > > >> > >> >
> > =============================================================
> > > > > > >> > >> >
> > > > > > >> > >> > X. Establish the Apache DataFusion Project
> > > > > > >> > >> >
> > > > > > >> > >> > WHEREAS, the Board of Directors deems it to be in the
> > best
> > > > > > >> > >> > interests of the Foundation and consistent with the
> > > > > > >> > >> > Foundation's purpose to establish a Project Management
> > > > > > >> > >> > Committee charged with the creation and maintenance of
> > > > > > >> > >> > open-source software related to an extensible query
> > engine
> > > > > > >> > >> > for distribution at no charge to the public.
> > > > > > >> > >> >
> > > > > > >> > >> > NOW, THEREFORE, BE IT RESOLVED, that a Project
> Management
> > > > > > >> > >> > Committee (PMC), to be known as the "Apache DataFusion
> > > > > Project",
> > > > > > >> > >> > be and hereby is established pursuant to Bylaws of the
> > > > > > >> > >> > Foundation; and be it further
> > > > > > >> > >> >
> > > > > > >> > >> > RESOLVED, that the Apache DataFusion Project be and
> > hereby is
> > > > > > >> > >> > responsible for the creation and maintenance of
> software
> > > > > > >> > >> > related to an extensible query engine; and be it
> further
> > > > > > >> > >> >
> > > > > > >> > >> > RESOLVED, that the office of "Vice President, Apache
> > > > > DataFusion" be
> > > > > > >> > >> > and hereby is created, the person holding such office
> to
> > > > > > >> > >> > serve at the direction of the Board of Directors as the
> > chair
> > > > > > >> > >> > of the Apache DataFusion Project, and to have primary
> > > > > responsibility
> > > > > > >> > >> > for management of the projects within the scope of
> > > > > > >> > >> > responsibility of the Apache DataFusion Project; and be
> > it
> > > > > further
> > > > > > >> > >> >
> > > > > > >> > >> > RESOLVED, that the persons listed immediately below be
> > and
> > > > > > >> > >> > hereby are appointed to serve as the initial members of
> > the
> > > > > > >> > >> > Apache DataFusion Project:
> > > > > > >> > >> >
> > > > > > >> > >> > * Andy Grove (agrove@apache.org)
> > > > > > >> > >> > * Andrew Lamb (alamb@apache.org)
> > > > > > >> > >> > * Daniël Heres (dheres@apache.org)
> > > > > > >> > >> > * Jie Wen (jakevin@apache.org)
> > > > > > >> > >> > * Kun Liu (liukun@apache.org)
> > > > > > >> > >> > * Liang-Chi Hsieh (viirya@apache.org)
> > > > > > >> > >> > * Qingping Hou: (houqp@apache.org)
> > > > > > >> > >> > * Wes McKinney(wesm@apache.org)
> > > > > > >> > >> > * Will Jones (wjones127@apache.org)
> > > > > > >> > >> >
> > > > > > >> > >> > RESOLVED, that the Apache DataFusion Project be and
> > hereby
> > > > > > >> > >> > is tasked with the migration and rationalization of the
> > > > Apache
> > > > > > >> > >> > Arrow DataFusion sub-project; and be it further
> > > > > > >> > >> >
> > > > > > >> > >> > RESOLVED, that all responsibilities pertaining to the
> > Apache
> > > > > > >> > >> > Arrow DataFusion sub-project encumbered upon the
> > > > > > >> > >> > Apache Arrow Project are hereafter discharged.
> > > > > > >> > >> >
> > > > > > >> > >> > NOW, THEREFORE, BE IT FURTHER RESOLVED, that Andrew
> Lamb
> > > > > > >> > >> > be appointed to the office of Vice President, Apache
> > > > > DataFusion, to
> > > > > > >> > >> > serve in accordance with and subject to the direction
> of
> > the
> > > > > > >> > >> > Board of Directors and the Bylaws of the Foundation
> until
> > > > > > >> > >> > death, resignation, retirement, removal or
> > disqualification,
> > > > > > >> > >> > or until a successor is appointed.
> > > > > > >> > >> >
> > =============================================================
> > > > > > >> > >> >
> > > > > > >> > >> >
> > > > > > >> > >> > -------
> > > > > > >> > >> >
> > > > > > >> > >> >
> > > > > > >> > >> > Summary:
> > > > > > >> > >> >
> > > > > > >> > >> > We propose creating a new top level project, Apache
> > > > > DataFusion, from
> > > > > > >> > >> > an existing sub project of Apache Arrow to facilitate
> > > > > additional
> > > > > > >> > >> > community and project growth.
> > > > > > >> > >> >
> > > > > > >> > >> > Abstract
> > > > > > >> > >> >
> > > > > > >> > >> > Apache Arrow DataFusion[1] is a very fast, extensible
> > query
> > > > > engine
> > > > > > >> > >> > for building high-quality data-centric systems in Rust,
> > using
> > > > > the
> > > > > > >> > >> > Apache Arrow in-memory format. DataFusion offers SQL
> and
> > > > > Dataframe
> > > > > > >> > >> > APIs, excellent performance, built-in support for CSV,
> > > > Parquet,
> > > > > > >> JSON,
> > > > > > >> > >> > and Avro, extensive customization, and a great
> community.
> > > > > > >> > >> >
> > > > > > >> > >> > [1] https://arrow.apache.org/datafusion/
> > > > > > >> > >> >
> > > > > > >> > >> >
> > > > > > >> > >> > Proposal
> > > > > > >> > >> >
> > > > > > >> > >> > We propose creating a new top level ASF project, Apache
> > > > > DataFusion,
> > > > > > >> > >> > governed initially by a subset of the Apache Arrow
> > project’s
> > > > > PMC and
> > > > > > >> > >> > committers. The project’s code is in five existing git
> > > > > repositories,
> > > > > > >> > >> > currently governed by Apache Arrow which would transfer
> > to
> > > > the
> > > > > new
> > > > > > >> top
> > > > > > >> > >> > level project.
> > > > > > >> > >> >
> > > > > > >> > >> > Background
> > > > > > >> > >> >
> > > > > > >> > >> > When DataFusion was initially donated to the Arrow
> > project,
> > > > it
> > > > > did
> > > > > > >> not
> > > > > > >> > >> > have a strong enough community to stand on its own. It
> > has
> > > > > since
> > > > > > >> grown
> > > > > > >> > >> > significantly, and benefited immensely from being part
> of
> > > > > Arrow and
> > > > > > >> > >> > nurturing of the Apache Way, and now has a community
> > strong
> > > > > enough
> > > > > > >> to
> > > > > > >> > >> > stand on its own and that would benefit from focused
> > > > governance
> > > > > > >> > >> > attention.
> > > > > > >> > >> >
> > > > > > >> > >> > The community has discussed this idea publicly for more
> > than
> > > > 6
> > > > > > >> months
> > > > > > >> > >> >
> > https://github.com/apache/arrow-datafusion/discussions/6475
> > > > > and
> > > > > > >> > >> > briefly on the Arrow PMC mailing list
> > > > > > >> > >> >
> > > > > https://lists.apache.org/thread/thv2jdm6640l6gm88hy8jhk5prjww0cs.
> > > > > > >> As
> > > > > > >> > >> > of the time of this writing both had exclusively
> positive
> > > > > reactions.
> > > > > > >> > >> >
> > > > > > >> > >> > Several current members of the Arrow PMC are both
> active
> > > > > > >> contributors
> > > > > > >> > >> > to DataFusion and understand and believe deeply in the
> > Apache
> > > > > Way,
> > > > > > >> and
> > > > > > >> > >> > play active governance roles in the Arrow project as
> PMC
> > > > > members and
> > > > > > >> > >> > PMC chairs, guiding the community, and releasing
> software
> > > > > versions.
> > > > > > >> > >> > With this existing governance experience and structure,
> > the
> > > > > new top
> > > > > > >> > >> > level project will be able to function well immediately
> > and
> > > > > > >> > >> > independently.
> > > > > > >> > >> >
> > > > > > >> > >> > Overview of DataFusion
> > > > > > >> > >> >
> > > > > > >> > >> > Current Status
> > > > > > >> > >> >
> > > > > > >> > >> > Meritocracy
> > > > > > >> > >> >
> > > > > > >> > >> > DataFusion has been developed as part of Apache Arrow
> and
> > > > thus
> > > > > has
> > > > > > >> > >> > been operating as a meritocracy. Many of the developers
> > of
> > > > > > >> DataFusion
> > > > > > >> > >> > are Arrow PMC members or committers. The DataFusion
> > project
> > > > > plans to
> > > > > > >> > >> > continue adding new PMC and committers as the project
> > matures
> > > > > and
> > > > > > >> > >> > grows.
> > > > > > >> > >> >
> > > > > > >> > >> > Community
> > > > > > >> > >> >
> > > > > > >> > >> > The DataFusion development team seeks to foster the
> > > > > development and
> > > > > > >> > >> > user communities. We hope that becoming a separate
> > project
> > > > > will help
> > > > > > >> > >> > both Arrow and DataFusion communities by being more
> > focused.
> > > > > > >> Focused
> > > > > > >> > >> > governance will make it easier to grow the community of
> > > > > committers
> > > > > > >> and
> > > > > > >> > >> > PMC members and make the organization more clear to
> > others.
> > > > > > >> > >> >
> > > > > > >> > >> > Alignment
> > > > > > >> > >> >
> > > > > > >> > >> > The ASF is a natural host for DataFusion given that it
> is
> > > > > already
> > > > > > >> the
> > > > > > >> > >> > home of Arrow, Parquet, and other related distributed
> > system,
> > > > > > >> storage
> > > > > > >> > >> > and query execution systems.
> > > > > > >> > >> >
> > > > > > >> > >> > Project Leadership
> > > > > > >> > >> >
> > > > > > >> > >> > Proposed Initial PMC
> > > > > > >> > >> >
> > > > > > >> > >> > We propose the following people as the initial
> > DataFusion PMC
> > > > > > >> members.
> > > > > > >> > >> > This is a subset of the existing Arrow PMC members who
> > > > > contribute to
> > > > > > >> > >> > DataFusion
> > > > https://people.apache.org/phonebook.html?unix=arrow
> > > > > > >> > >> >
> > > > > > >> > >> > Andy Grove (agrove): Arrow PMC Chair
> > > > > > >> > >> > Andrew Lamb (alamb): Arrow PMC, past Arrow PMC Chair
> > > > > > >> > >> > Daniël Heres (dheres) Arrow PMC
> > > > > > >> > >> > Jie Wen (jakevin): Arrow PMC, Doris Committer
> > > > > > >> > >> > Kun Liu (liukun): Arrow PMC, IoTDB PMC, TSFile PMC
> > > > > > >> > >> > Liang-Chi Hsieh (viirya): Arrow PMC, Spark PMC
> > > > > > >> > >> > Qingping Hou: (houqp): Arrow PMC
> > > > > > >> > >> > Wes McKinney(wesm): Arrow PMC, ASF Member
> > > > > > >> > >> > Will Jones (wjones127): Arrow PMC
> > > > > > >> > >> >
> > > > > > >> > >> > We’d like to propose Andrew Lamb as the initial Chair,
> > (and
> > > > > thus ASF
> > > > > > >> > >> > VP) for the DataFusion project.
> > > > > > >> > >> >
> > > > > > >> > >> > Affiliations
> > > > > > >> > >> >
> > > > > > >> > >> > Andy Grove (agrove): NVidia
> > > > > > >> > >> > Andrew Lamb (alamb): InfluxData
> > > > > > >> > >> > Daniël Heres (dheres): Coralogix
> > > > > > >> > >> > Jie Wen (jakevin): SelectDB
> > > > > > >> > >> > Kun Liu (liukun): Ebay
> > > > > > >> > >> > Liang-Chi Hsieh (viirya): Apple
> > > > > > >> > >> > Qingping Hou: (houqp): Scribd
> > > > > > >> > >> > Wes McKinney(wesm): Posit
> > > > > > >> > >> > Will Jones (wjones127): LanceDB
> > > > > > >> > >> >
> > > > > > >> > >> > Proposed Initial Committers
> > > > > > >> > >> >
> > > > > > >> > >> > In addition to the PMC, we propose the following people
> > as
> > > > the
> > > > > > >> initial
> > > > > > >> > >> > DataFusion committers. This is a subset of the existing
> > Arrow
> > > > > > >> > >> > committers who contribute to DataFusion
> > > > > > >> > >> > https://people.apache.org/phonebook.html?unix=arrow
> > > > > > >> > >> >
> > > > > > >> > >> > akurmustafa Mustafa Akur (Synnada)
> > > > > > >> > >> > avantgardner Brent Gardner (Coralogix)
> > > > > > >> > >> > comphead Oleks V. (Unaffiliated)
> > > > > > >> > >> > jayzhan Jay Zhan (Unaffiliated)
> > > > > > >> > >> > jeffreyvo Jeffry Vo (Unaffiliated)
> > > > > > >> > >> > jiayuliu Liu Jiayu (Airbnb)
> > > > > > >> > >> > mete Metehan Yildirim (Synnada)
> > > > > > >> > >> > mingmwang Wang Mingming (Ebay)
> > > > > > >> > >> > mneumann Marco Neumann (InfluxData)
> > > > > > >> > >> > nju_yaho Zhong Yanghong (Ebay)
> > > > > > >> > >> > ozankabak Mehmet Ozan Kabak (Synnada)
> > > > > > >> > >> > paddyhoran Paddy Horan (Assured Allies)
> > > > > > >> > >> > rdettai Rémi Dettai (Cloudfuse)
> > > > > > >> > >> > sunchao Chao Sun (Apple)
> > > > > > >> > >> > thinkharderdev Daniel Harris (Coralogix)
> > > > > > >> > >> > tustvold Raphael Taylor-Davies (InfluxData)
> > > > > > >> > >> > wayne Ruihang Xia (Greptime)
> > > > > > >> > >> > xudong963 Xudong Wang (ByteDance)
> > > > > > >> > >> > yjshen Yijie Shen (Space and Time)
> > > > > > >> > >> > yangjiang Yang Jiang (ebay)
> > > > > > >> > >> >
> > > > > > >> > >> >
> > > > > > >> > >> > Risk Assessments
> > > > > > >> > >> >
> > > > > > >> > >> > Naming / Trademarks
> > > > > > >> > >> >
> > > > > > >> > >> > As a sub-project of Arrow, the DataFusion name has been
> > used
> > > > > for
> > > > > > >> over
> > > > > > >> > >> > 4 years without any known issues. A podling name search
> > did
> > > > > not turn
> > > > > > >> > >> > up any concerns and was approved:
> > > > > > >> > >> >
> > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> > > > > > >> > >> >
> > > > > > >> > >> > Legal / IP Clearance
> > > > > > >> > >> >
> > > > > > >> > >> > All DataFusion code has either been donated to the
> Arrow
> > > > > project
> > > > > > >> with
> > > > > > >> > >> > appropriate IP clearance or has been developed
> directly
> > > > under
> > > > > ASF
> > > > > > >> > >> > processes and procedures. Thus creating a new top level
> > > > project
> > > > > > >> poses
> > > > > > >> > >> > no new Legal or IP risks.
> > > > > > >> > >> >
> > > > > > >> > >> > Code Extraction
> > > > > > >> > >> >
> > > > > > >> > >> > The relevant code is already in 5 separate
> repositories:
> > > > > > >> > >> > https://github.com/apache/arrow-datafusion/
> > > > > > >> > >> > https://github.com/apache/arrow-datafusion-python
> > > > > > >> > >> > https://github.com/apache/arrow-ballista
> > > > > > >> > >> > https://github.com/apache/arrow-ballista-python
> > > > > > >> > >> > https://github.com/apache/arrow-datafusion-comet
> > > > > > >> > >> >
> > > > > > >> > >> > We foresee no issues with code extraction and propose
> > these
> > > > > > >> > >> > repositories be renamed to reflect top level projects
> > > > > > >> > >> >
> > > > > > >> > >> > Note: https://github.com/apache/arrow-rs, the Rust
> > > > > implementation
> > > > > > >> of
> > > > > > >> > >> > Arrow, would remain part of the Arrow project.
> > > > > > >> > >> >
> > > > > > >> > >> > Orphaned Products
> > > > > > >> > >> >
> > > > > > >> > >> > DataFusion is known to be used in many open source and
> > > > > commercial
> > > > > > >> > >> > projects
> > > > > > >> > >> >
> > > > > > >> > >> https://arrow.apache.org/datafusion/user-guide/
> > > > > > >> > introduction.html#known-users
> > > > > > >> > >> > ,
> > > > > > >> > >> > has had multiple commits daily for several years, and
> its
> > > > > adoption
> > > > > > >> and
> > > > > > >> > >> > number of contributors appears to be growing. We do not
> > > > > foresee the
> > > > > > >> > >> > project being orphaned in the next several years.
> > > > > > >> > >> >
> > > > > > >> > >> > Inexperience with Open Source
> > > > > > >> > >> >
> > > > > > >> > >> > The proposed PMC has extensive experience with Apache
> > Arrow
> > > > and
> > > > > > >> other
> > > > > > >> > >> > Apache projects, and includes PMC members, PMC chairs
> > and an
> > > > > ASF
> > > > > > >> > >> > Member. The DataFusion PMC and more experienced
> > committers
> > > > will
> > > > > > >> > >> > continue to coach new community members who may be less
> > > > > familiar
> > > > > > >> with
> > > > > > >> > >> > the Apache Way.
> > > > > > >> > >> >
> > > > > > >> > >> > Homogeneous Developers
> > > > > > >> > >> >
> > > > > > >> > >> > The 9 proposed PMC members are from 9 different
> > employers and
> > > > > the
> > > > > > >> > >> > proposed committers are similarly distributed across
> > > > > affiliations.
> > > > > > >> No
> > > > > > >> > >> > specific entity employs more than 3 total proposed
> > > > developers.
> > > > > > >> > >> >
> > > > > > >> > >> > Reliance on Salaried Developers
> > > > > > >> > >> >
> > > > > > >> > >> > A substantial amount of work on DataFusion has been by
> > > > salaried
> > > > > > >> > >> > developers, but it also has a long tradition of
> > attracting
> > > > > > >> > >> > contributions from students and hobbyists and we plan
> no
> > > > > changes in
> > > > > > >> > >> > contribution structure.
> > > > > > >> > >> >
> > > > > > >> > >> > Relationships with Other Apache Products
> > > > > > >> > >> >
> > > > > > >> > >> > DataFusion will obviously have a strong relationship
> > with the
> > > > > Arrow
> > > > > > >> > >> > project given the overlap in people. We don’t foresee
> > close
> > > > > > >> > >> > collaboration with other projects at this time.
> > > > > > >> > >> >
> > > > > > >> > >> > Cryptography
> > > > > > >> > >> >
> > > > > > >> > >> > DataFusion does not directly support encryption and
> > there are
> > > > > no
> > > > > > >> > >> > near-term plans to add support for encryption. Users
> who
> > need
> > > > > this
> > > > > > >> > >> > functionality can use the extension APIs.
> > > > > > >> > >> >
> > > > > > >> > >> > Required Resources
> > > > > > >> > >> >
> > > > > > >> > >> > Mailing Lists
> > > > > > >> > >> >
> > > > > > >> > >> > - private@datafusion.apache.org for private PMC
> > discussions
> > > > > (with
> > > > > > >> > >> > moderated subscriptions)
> > > > > > >> > >> > - dev@datafusion.apache.org
> > > > > > >> > >> > - commits@datafusion.apache.org
> > > > > > >> > >> > - user@datafusion.apache.org
> > > > > > >> > >> >
> > > > > > >> > >> > Version Control
> > > > > > >> > >> >
> > > > > > >> > >> > We propose to continue to use git for source control
> and
> > > > > github for
> > > > > > >> > >> > hosting and testing resources.
> > > > > > >> > >> >
> > > > > > >> > >> > We also need to rename the github repositories to
> > reflect the
> > > > > new
> > > > > > >> top
> > > > > > >> > >> > level names:
> > > > > > >> > >> >
> > > > > > >> > >> > https://github.com/apache/arrow-datafusion/ →
> > > > > apache/datafusion
> > > > > > >> > >> > https://github.com/apache/arrow-datafusion-python →
> > > > > > >> > >> > apache/datafusion-python
> > > > > > >> > >> > https://github.com/apache/arrow-ballista →
> > > > > > >> apache/datafusion-ballista
> > > > > > >> > >> > https://github.com/apache/arrow-ballista-python →
> > > > > > >> > >> > apache/datafusion-ballista-python
> > > > > > >> > >> > https://github.com/apache/arrow-datafusion-comet →
> > > > > > >> > >> apache/datafusion-comet
> > > > > > >> > >> >
> > > > > > >> > >> >
> > > > > > >> > >> >
> > > > > > >> > >> > Issue Tracking
> > > > > > >> > >> >
> > > > > > >> > >> > DataFusion would continue to use github for its issue
> > > > tracking
> > > > > and
> > > > > > >> > >> > communications
> > > > > > >> > >> >
> > > > > > >> > >> > Other Resources
> > > > > > >> > >> >
> > > > > > >> > >> > The existing repositories already make use of existing
> > Apache
> > > > > > >> > >> > infrastructure, and we expect no change in the initial
> > > > resource
> > > > > > >> usage.
> > > > > > >> > >> > As the project continues to grow, we expect continued
> > > > > infrastructure
> > > > > > >> > >> > demand growth.
> > > > > > >> > >> >
> > > > > > >> > >> >
> > > > > > >> > >> > FAQ: Has a sub project been promoted to a top level
> > project
> > > > > before?
> > > > > > >> > >> >
> > > > > > >> > >> > Yes, and it appears to happen commonly. The Arrow
> project
> > > > > itself was
> > > > > > >> > >> > created as a top level project from work that started
> in
> > > > Apache
> > > > > > >> Drill,
> > > > > > >> > >> > and there are many sub projects of Hadoop that spun out
> > as
> > > > > their own
> > > > > > >> > >> > top level projects such as Mahout, Avro and HBase:
> > > > > > >> > >> >
> > > > > > >> > >> >
> > > > > > >> > >> https://news.apache.org/foundation/entry/the_apache_
> > > > > > >> > software_foundation_announces4
> > > > > > >> > >> >
> > > > > > >> > >> >
> > > > > > >> > >> >
> > > > > > >> > >> > Related material:
> > > > > > >> > >> > Name search request / research for DataFusion:
> > > > > > >> > >> >
> > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> > > > > > >> > >> > Discussion about this proposal on the arrow mailing
> list:
> > > > > > >> > >> >
> > > > > https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> > > > > > >> > >> > Discussion about which repositories on the arrow
> mailing
> > > > list:
> > > > > > >> > >> >
> > > > > https://lists.apache.org/thread/ob3n0d9ky0bgrryl3xn39w9k566bq00q
> > > > > > >> > >> > Discussion about initial PMC on the arrow mailing list:
> > > > > > >> > >> >
> > > > > https://lists.apache.org/thread/pymrzcdw4qdptvby85f69rg3pcckl15b
> > > > > > >> > >> > Discussion in github about creating a new DataFusion
> top
> > > > level
> > > > > > >> > >> > project:
> > > > > > >> https://github.com/apache/arrow-datafusion/discussions/6475
> > > > > > >> > >> > Discussion about graduating on incubator list:
> > > > > > >> > >> >
> > > > > https://lists.apache.org/thread/r4n73pmms1lv0jbohyx1o1z13d615t99
> > > > > > >> > >> > Original Proposal for the Arrow project:
> > > > > > >> > >> >
> > > > > https://lists.apache.org/thread/x2qzdwglm8pkqp9gv03bbgw17khl7pq3
> > > > > > >> > >> >
> > > > > > >> > >>
> > > > > > >> >
> > > > > > >>
> > > > >
> > > >
> > >
> >
>
Re: [VOTE] Move Arrow DataFusion Subproject to new Top Level Apache Project
Posted by Jacob Wujciak-Jens <ja...@voltrondata.com.INVALID>.
+1 (non-binding)
On Mon, Mar 4, 2024 at 3:39 AM Yang Jiang <ya...@apache.org> wrote:
> +1 (non-binding)
>
> On 2024/03/01 18:08:26 Daniël Heres wrote:
> > +1 (binding)
> >
> > On Fri, Mar 1, 2024, 19:05 Chao Sun <su...@apache.org> wrote:
> >
> > > +1 (non-binding)
> > >
> > > On Fri, Mar 1, 2024 at 9:59 AM QP Hou <qp...@neuralink.com> wrote:
> > >
> > > > +1 (binding)
> > > >
> > > > exciting milestone :)
> > > >
> > > > On Fri, Mar 1, 2024 at 9:49 AM David Li <li...@apache.org> wrote:
> > > > >
> > > > > +1
> > > > >
> > > > > On Fri, Mar 1, 2024, at 12:06, Jorge Cardoso Leitão wrote:
> > > > > > +1 - great work!!!
> > > > > >
> > > > > > On Fri, Mar 1, 2024 at 5:49 PM Micah Kornfield <
> > > emkornfield@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > >> +1 (binding)
> > > > > >>
> > > > > >> On Friday, March 1, 2024, Uwe L. Korn <uw...@xhochy.com> wrote:
> > > > > >>
> > > > > >> > +1 (binding)
> > > > > >> >
> > > > > >> > On Fri, Mar 1, 2024, at 2:37 PM, Andy Grove wrote:
> > > > > >> > > +1 (binding)
> > > > > >> > >
> > > > > >> > > On Fri, Mar 1, 2024 at 6:20 AM Weston Pace <
> > > weston.pace@gmail.com
> > > > >
> > > > > >> > wrote:
> > > > > >> > >
> > > > > >> > >> +1 (binding)
> > > > > >> > >>
> > > > > >> > >> On Fri, Mar 1, 2024 at 3:33 AM Andrew Lamb <
> > > alamb@influxdata.com
> > > > >
> > > > > >> > wrote:
> > > > > >> > >>
> > > > > >> > >> > Hello,
> > > > > >> > >> >
> > > > > >> > >> > As we have discussed[1][2] I would like to vote on the
> > > > proposal to
> > > > > >> > >> > create a new Apache Top Level Project for DataFusion. The
> > > text
> > > > of
> > > > > >> the
> > > > > >> > >> > proposed resolution and background document is
> copy/pasted
> > > > below
> > > > > >> > >> >
> > > > > >> > >> > If the community is in favor of this, we plan to submit
> the
> > > > > >> resolution
> > > > > >> > >> > to the ASF board for approval with the next Arrow report
> (for
> > > > the
> > > > > >> > >> > April 2024 board meeting).
> > > > > >> > >> >
> > > > > >> > >> > The vote will be open for at least 7 days.
> > > > > >> > >> >
> > > > > >> > >> > [ ] +1 Accept this Proposal
> > > > > >> > >> > [ ] +0
> > > > > >> > >> > [ ] -1 Do not accept this proposal because...
> > > > > >> > >> >
> > > > > >> > >> > Andrew
> > > > > >> > >> >
> > > > > >> > >> > [1]
> > > > > >>
> https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> > > > > >> > >> > [2]
> > > > https://github.com/apache/arrow-datafusion/discussions/6475
> > > > > >> > >> >
> > > > > >> > >> > ---------- Proposed Resolution ---------
> > > > > >> > >> >
> > > > > >> > >> > Resolution to Create the Apache DataFusion Project from
> the
> > > > Apache
> > > > > >> > >> > Arrow DataFusion Sub Project
> > > > > >> > >> >
> > > > > >> > >> >
> =============================================================
> > > > > >> > >> >
> > > > > >> > >> > X. Establish the Apache DataFusion Project
> > > > > >> > >> >
> > > > > >> > >> > WHEREAS, the Board of Directors deems it to be in the
> best
> > > > > >> > >> > interests of the Foundation and consistent with the
> > > > > >> > >> > Foundation's purpose to establish a Project Management
> > > > > >> > >> > Committee charged with the creation and maintenance of
> > > > > >> > >> > open-source software related to an extensible query
> engine
> > > > > >> > >> > for distribution at no charge to the public.
> > > > > >> > >> >
> > > > > >> > >> > NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> > > > > >> > >> > Committee (PMC), to be known as the "Apache DataFusion
> > > > Project",
> > > > > >> > >> > be and hereby is established pursuant to Bylaws of the
> > > > > >> > >> > Foundation; and be it further
> > > > > >> > >> >
> > > > > >> > >> > RESOLVED, that the Apache DataFusion Project be and
> hereby is
> > > > > >> > >> > responsible for the creation and maintenance of software
> > > > > >> > >> > related to an extensible query engine; and be it further
> > > > > >> > >> >
> > > > > >> > >> > RESOLVED, that the office of "Vice President, Apache
> > > > DataFusion" be
> > > > > >> > >> > and hereby is created, the person holding such office to
> > > > > >> > >> > serve at the direction of the Board of Directors as the
> chair
> > > > > >> > >> > of the Apache DataFusion Project, and to have primary
> > > > responsibility
> > > > > >> > >> > for management of the projects within the scope of
> > > > > >> > >> > responsibility of the Apache DataFusion Project; and be
> it
> > > > further
> > > > > >> > >> >
> > > > > >> > >> > RESOLVED, that the persons listed immediately below be
> and
> > > > > >> > >> > hereby are appointed to serve as the initial members of
> the
> > > > > >> > >> > Apache DataFusion Project:
> > > > > >> > >> >
> > > > > >> > >> > * Andy Grove (agrove@apache.org)
> > > > > >> > >> > * Andrew Lamb (alamb@apache.org)
> > > > > >> > >> > * Daniël Heres (dheres@apache.org)
> > > > > >> > >> > * Jie Wen (jakevin@apache.org)
> > > > > >> > >> > * Kun Liu (liukun@apache.org)
> > > > > >> > >> > * Liang-Chi Hsieh (viirya@apache.org)
> > > > > >> > >> > * Qingping Hou: (houqp@apache.org)
> > > > > >> > >> > * Wes McKinney(wesm@apache.org)
> > > > > >> > >> > * Will Jones (wjones127@apache.org)
> > > > > >> > >> >
> > > > > >> > >> > RESOLVED, that the Apache DataFusion Project be and
> hereby
> > > > > >> > >> > is tasked with the migration and rationalization of the
> > > Apache
> > > > > >> > >> > Arrow DataFusion sub-project; and be it further
> > > > > >> > >> >
> > > > > >> > >> > RESOLVED, that all responsibilities pertaining to the
> Apache
> > > > > >> > >> > Arrow DataFusion sub-project encumbered upon the
> > > > > >> > >> > Apache Arrow Project are hereafter discharged.
> > > > > >> > >> >
> > > > > >> > >> > NOW, THEREFORE, BE IT FURTHER RESOLVED, that Andrew Lamb
> > > > > >> > >> > be appointed to the office of Vice President, Apache
> > > > DataFusion, to
> > > > > >> > >> > serve in accordance with and subject to the direction of
> the
> > > > > >> > >> > Board of Directors and the Bylaws of the Foundation until
> > > > > >> > >> > death, resignation, retirement, removal or
> disqualification,
> > > > > >> > >> > or until a successor is appointed.
> > > > > >> > >> >
> =============================================================
> > > > > >> > >> >
> > > > > >> > >> >
> > > > > >> > >> > -------
> > > > > >> > >> >
> > > > > >> > >> >
> > > > > >> > >> > Summary:
> > > > > >> > >> >
> > > > > >> > >> > We propose creating a new top level project, Apache
> > > > DataFusion, from
> > > > > >> > >> > an existing sub project of Apache Arrow to facilitate
> > > > additional
> > > > > >> > >> > community and project growth.
> > > > > >> > >> >
> > > > > >> > >> > Abstract
> > > > > >> > >> >
> > > > > >> > >> > Apache Arrow DataFusion[1] is a very fast, extensible
> query
> > > > engine
> > > > > >> > >> > for building high-quality data-centric systems in Rust,
> using
> > > > the
> > > > > >> > >> > Apache Arrow in-memory format. DataFusion offers SQL and
> > > > Dataframe
> > > > > >> > >> > APIs, excellent performance, built-in support for CSV,
> > > Parquet,
> > > > > >> JSON,
> > > > > >> > >> > and Avro, extensive customization, and a great community.
> > > > > >> > >> >
> > > > > >> > >> > [1] https://arrow.apache.org/datafusion/
> > > > > >> > >> >
> > > > > >> > >> >
> > > > > >> > >> > Proposal
> > > > > >> > >> >
> > > > > >> > >> > We propose creating a new top level ASF project, Apache
> > > > DataFusion,
> > > > > >> > >> > governed initially by a subset of the Apache Arrow
> project’s
> > > > PMC and
> > > > > >> > >> > committers. The project’s code is in five existing git
> > > > repositories,
> > > > > >> > >> > currently governed by Apache Arrow which would transfer
> to
> > > the
> > > > new
> > > > > >> top
> > > > > >> > >> > level project.
> > > > > >> > >> >
> > > > > >> > >> > Background
> > > > > >> > >> >
> > > > > >> > >> > When DataFusion was initially donated to the Arrow
> project,
> > > it
> > > > did
> > > > > >> not
> > > > > >> > >> > have a strong enough community to stand on its own. It
> has
> > > > since
> > > > > >> grown
> > > > > >> > >> > significantly, and benefited immensely from being part of
> > > > Arrow and
> > > > > >> > >> > nurturing of the Apache Way, and now has a community
> strong
> > > > enough
> > > > > >> to
> > > > > >> > >> > stand on its own and that would benefit from focused
> > > governance
> > > > > >> > >> > attention.
> > > > > >> > >> >
> > > > > >> > >> > The community has discussed this idea publicly for more
> than
> > > 6
> > > > > >> months
> > > > > >> > >> >
> https://github.com/apache/arrow-datafusion/discussions/6475
> > > > and
> > > > > >> > >> > briefly on the Arrow PMC mailing list
> > > > > >> > >> >
> > > > https://lists.apache.org/thread/thv2jdm6640l6gm88hy8jhk5prjww0cs.
> > > > > >> As
> > > > > >> > >> > of the time of this writing both had exclusively positive
> > > > reactions.
> > > > > >> > >> >
> > > > > >> > >> > Several current members of the Arrow PMC are both active
> > > > > >> contributors
> > > > > >> > >> > to DataFusion and understand and believe deeply in the
> Apache
> > > > Way,
> > > > > >> and
> > > > > >> > >> > play active governance roles in the Arrow project as PMC
> > > > members and
> > > > > >> > >> > PMC chairs, guiding the community, and releasing software
> > > > versions.
> > > > > >> > >> > With this existing governance experience and structure,
> the
> > > > new top
> > > > > >> > >> > level project will be able to function well immediately
> and
> > > > > >> > >> > independently.
> > > > > >> > >> >
> > > > > >> > >> > Overview of DataFusion
> > > > > >> > >> >
> > > > > >> > >> > Current Status
> > > > > >> > >> >
> > > > > >> > >> > Meritocracy
> > > > > >> > >> >
> > > > > >> > >> > DataFusion has been developed as part of Apache Arrow and
> > > thus
> > > > has
> > > > > >> > >> > been operating as a meritocracy. Many of the developers
> of
> > > > > >> DataFusion
> > > > > >> > >> > are Arrow PMC members or committers. The DataFusion
> project
> > > > plans to
> > > > > >> > >> > continue adding new PMC and committers as the project
> matures
> > > > and
> > > > > >> > >> > grows.
> > > > > >> > >> >
> > > > > >> > >> > Community
> > > > > >> > >> >
> > > > > >> > >> > The DataFusion development team seeks to foster the
> > > > development and
> > > > > >> > >> > user communities. We hope that becoming a separate
> project
> > > > will help
> > > > > >> > >> > both Arrow and DataFusion communities by being more
> focused.
> > > > > >> Focused
> > > > > >> > >> > governance will make it easier to grow the community of
> > > > committers
> > > > > >> and
> > > > > >> > >> > PMC members and make the organization more clear to
> others.
> > > > > >> > >> >
> > > > > >> > >> > Alignment
> > > > > >> > >> >
> > > > > >> > >> > The ASF is a natural host for DataFusion given that it is
> > > > already
> > > > > >> the
> > > > > >> > >> > home of Arrow, Parquet, and other related distributed
> system,
> > > > > >> storage
> > > > > >> > >> > and query execution systems.
> > > > > >> > >> >
> > > > > >> > >> > Project Leadership
> > > > > >> > >> >
> > > > > >> > >> > Proposed Initial PMC
> > > > > >> > >> >
> > > > > >> > >> > We propose the following people as the initial
> DataFusion PMC
> > > > > >> members.
> > > > > >> > >> > This is a subset of the existing Arrow PMC members who
> > > > contribute to
> > > > > >> > >> > DataFusion
> > > https://people.apache.org/phonebook.html?unix=arrow
> > > > > >> > >> >
> > > > > >> > >> > Andy Grove (agrove): Arrow PMC Chair
> > > > > >> > >> > Andrew Lamb (alamb): Arrow PMC, past Arrow PMC Chair
> > > > > >> > >> > Daniël Heres (dheres) Arrow PMC
> > > > > >> > >> > Jie Wen (jakevin): Arrow PMC, Doris Committer
> > > > > >> > >> > Kun Liu (liukun): Arrow PMC, IoTDB PMC, TSFile PMC
> > > > > >> > >> > Liang-Chi Hsieh (viirya): Arrow PMC, Spark PMC
> > > > > >> > >> > Qingping Hou: (houqp): Arrow PMC
> > > > > >> > >> > Wes McKinney(wesm): Arrow PMC, ASF Member
> > > > > >> > >> > Will Jones (wjones127): Arrow PMC
> > > > > >> > >> >
> > > > > >> > >> > We’d like to propose Andrew Lamb as the initial Chair,
> (and
> > > > thus ASF
> > > > > >> > >> > VP) for the DataFusion project.
> > > > > >> > >> >
> > > > > >> > >> > Affiliations
> > > > > >> > >> >
> > > > > >> > >> > Andy Grove (agrove): NVidia
> > > > > >> > >> > Andrew Lamb (alamb): InfluxData
> > > > > >> > >> > Daniël Heres (dheres): Coralogix
> > > > > >> > >> > Jie Wen (jakevin): SelectDB
> > > > > >> > >> > Kun Liu (liukun): Ebay
> > > > > >> > >> > Liang-Chi Hsieh (viirya): Apple
> > > > > >> > >> > Qingping Hou: (houqp): Scribd
> > > > > >> > >> > Wes McKinney(wesm): Posit
> > > > > >> > >> > Will Jones (wjones127): LanceDB
> > > > > >> > >> >
> > > > > >> > >> > Proposed Initial Committers
> > > > > >> > >> >
> > > > > >> > >> > In addition to the PMC, we propose the following people
> as
> > > the
> > > > > >> initial
> > > > > >> > >> > DataFusion committers. This is a subset of the existing
> Arrow
> > > > > >> > >> > committers who contribute to DataFusion
> > > > > >> > >> > https://people.apache.org/phonebook.html?unix=arrow
> > > > > >> > >> >
> > > > > >> > >> > akurmustafa Mustafa Akur (Synnada)
> > > > > >> > >> > avantgardner Brent Gardner (Coralogix)
> > > > > >> > >> > comphead Oleks V. (Unaffiliated)
> > > > > >> > >> > jayzhan Jay Zhan (Unaffiliated)
> > > > > >> > >> > jeffreyvo Jeffry Vo (Unaffiliated)
> > > > > >> > >> > jiayuliu Liu Jiayu (Airbnb)
> > > > > >> > >> > mete Metehan Yildirim (Synnada)
> > > > > >> > >> > mingmwang Wang Mingming (Ebay)
> > > > > >> > >> > mneumann Marco Neumann (InfluxData)
> > > > > >> > >> > nju_yaho Zhong Yanghong (Ebay)
> > > > > >> > >> > ozankabak Mehmet Ozan Kabak (Synnada)
> > > > > >> > >> > paddyhoran Paddy Horan (Assured Allies)
> > > > > >> > >> > rdettai Rémi Dettai (Cloudfuse)
> > > > > >> > >> > sunchao Chao Sun (Apple)
> > > > > >> > >> > thinkharderdev Daniel Harris (Coralogix)
> > > > > >> > >> > tustvold Raphael Taylor-Davies (InfluxData)
> > > > > >> > >> > wayne Ruihang Xia (Greptime)
> > > > > >> > >> > xudong963 Xudong Wang (ByteDance)
> > > > > >> > >> > yjshen Yijie Shen (Space and Time)
> > > > > >> > >> > yangjiang Yang Jiang (ebay)
> > > > > >> > >> >
> > > > > >> > >> >
> > > > > >> > >> > Risk Assessments
> > > > > >> > >> >
> > > > > >> > >> > Naming / Trademarks
> > > > > >> > >> >
> > > > > >> > >> > As a sub-project of Arrow, the DataFusion name has been
> used
> > > > for
> > > > > >> over
> > > > > >> > >> > 4 years without any known issues. A podling name search
> did
> > > > not turn
> > > > > >> > >> > up any concerns and was approved:
> > > > > >> > >> >
> https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> > > > > >> > >> >
> > > > > >> > >> > Legal / IP Clearance
> > > > > >> > >> >
> > > > > >> > >> > All DataFusion code has either been donated to the Arrow
> > > > project
> > > > > >> with
> > > > > >> > >> > appropriate IP clearance or has been developed directly
> > > under
> > > > ASF
> > > > > >> > >> > processes and procedures. Thus creating a new top level
> > > project
> > > > > >> poses
> > > > > >> > >> > no new Legal or IP risks.
> > > > > >> > >> >
> > > > > >> > >> > Code Extraction
> > > > > >> > >> >
> > > > > >> > >> > The relevant code is already in 5 separate repositories:
> > > > > >> > >> > https://github.com/apache/arrow-datafusion/
> > > > > >> > >> > https://github.com/apache/arrow-datafusion-python
> > > > > >> > >> > https://github.com/apache/arrow-ballista
> > > > > >> > >> > https://github.com/apache/arrow-ballista-python
> > > > > >> > >> > https://github.com/apache/arrow-datafusion-comet
> > > > > >> > >> >
> > > > > >> > >> > We foresee no issues with code extraction and propose
> these
> > > > > >> > >> > repositories be renamed to reflect top level projects
> > > > > >> > >> >
> > > > > >> > >> > Note: https://github.com/apache/arrow-rs, the Rust
> > > > implementation
> > > > > >> of
> > > > > >> > >> > Arrow, would remain part of the Arrow project.
> > > > > >> > >> >
> > > > > >> > >> > Orphaned Products
> > > > > >> > >> >
> > > > > >> > >> > DataFusion is known to be used in many open source and
> > > > commercial
> > > > > >> > >> > projects
> > > > > >> > >> >
> > > > > >> > >> https://arrow.apache.org/datafusion/user-guide/
> > > > > >> > introduction.html#known-users
> > > > > >> > >> > ,
> > > > > >> > >> > has had multiple commits daily for several years, and its
> > > > adoption
> > > > > >> and
> > > > > >> > >> > number of contributors appears to be growing. We do not
> > > > foresee the
> > > > > >> > >> > project being orphaned in the next several years.
> > > > > >> > >> >
> > > > > >> > >> > Inexperience with Open Source
> > > > > >> > >> >
> > > > > >> > >> > The proposed PMC has extensive experience with Apache
> Arrow
> > > and
> > > > > >> other
> > > > > >> > >> > Apache projects, and includes PMC members, PMC chairs
> and an
> > > > ASF
> > > > > >> > >> > Member. The DataFusion PMC and more experienced
> committers
> > > will
> > > > > >> > >> > continue to coach new community members who may be less
> > > > familiar
> > > > > >> with
> > > > > >> > >> > the Apache Way.
> > > > > >> > >> >
> > > > > >> > >> > Homogeneous Developers
> > > > > >> > >> >
> > > > > >> > >> > The 9 proposed PMC members are from 9 different
> employers and
> > > > the
> > > > > >> > >> > proposed committers are similarly distributed across
> > > > affiliations.
> > > > > >> No
> > > > > >> > >> > specific entity employs more than 3 total proposed
> > > developers.
> > > > > >> > >> >
> > > > > >> > >> > Reliance on Salaried Developers
> > > > > >> > >> >
> > > > > >> > >> > A substantial amount of work on DataFusion has been by
> > > salaried
> > > > > >> > >> > developers, but it also has a long tradition of
> attracting
> > > > > >> > >> > contributions from students and hobbyists and we plan no
> > > > changes in
> > > > > >> > >> > contribution structure.
> > > > > >> > >> >
> > > > > >> > >> > Relationships with Other Apache Products
> > > > > >> > >> >
> > > > > >> > >> > DataFusion will obviously have a strong relationship
> with the
> > > > Arrow
> > > > > >> > >> > project given the overlap in people. We don’t foresee
> close
> > > > > >> > >> > collaboration with other projects at this time.
> > > > > >> > >> >
> > > > > >> > >> > Cryptography
> > > > > >> > >> >
> > > > > >> > >> > DataFusion does not directly support encryption and
> there are
> > > > no
> > > > > >> > >> > near-term plans to add support for encryption. Users who
> need
> > > > this
> > > > > >> > >> > functionality can use the extension APIs.
> > > > > >> > >> >
> > > > > >> > >> > Required Resources
> > > > > >> > >> >
> > > > > >> > >> > Mailing Lists
> > > > > >> > >> >
> > > > > >> > >> > - private@datafusion.apache.org for private PMC
> discussions
> > > > (with
> > > > > >> > >> > moderated subscriptions)
> > > > > >> > >> > - dev@datafusion.apache.org
> > > > > >> > >> > - commits@datafusion.apache.org
> > > > > >> > >> > - user@datafusion.apache.org
> > > > > >> > >> >
> > > > > >> > >> > Version Control
> > > > > >> > >> >
> > > > > >> > >> > We propose to continue to use git for source control and
> > > > github for
> > > > > >> > >> > hosting and testing resources.
> > > > > >> > >> >
> > > > > >> > >> > We also need to rename the github repositories to
> reflect the
> > > > new
> > > > > >> top
> > > > > >> > >> > level names:
> > > > > >> > >> >
> > > > > >> > >> > https://github.com/apache/arrow-datafusion/ →
> > > > apache/datafusion
> > > > > >> > >> > https://github.com/apache/arrow-datafusion-python →
> > > > > >> > >> > apache/datafusion-python
> > > > > >> > >> > https://github.com/apache/arrow-ballista →
> > > > > >> apache/datafusion-ballista
> > > > > >> > >> > https://github.com/apache/arrow-ballista-python →
> > > > > >> > >> > apache/datafusion-ballista-python
> > > > > >> > >> > https://github.com/apache/arrow-datafusion-comet →
> > > > > >> > >> apache/datafusion-comet
> > > > > >> > >> >
> > > > > >> > >> >
> > > > > >> > >> >
> > > > > >> > >> > Issue Tracking
> > > > > >> > >> >
> > > > > >> > >> > DataFusion would continue to use github for its issue
> > > tracking
> > > > and
> > > > > >> > >> > communications
> > > > > >> > >> >
> > > > > >> > >> > Other Resources
> > > > > >> > >> >
> > > > > >> > >> > The existing repositories already make use of existing
> Apache
> > > > > >> > >> > infrastructure, and we expect no change in the initial
> > > resource
> > > > > >> usage.
> > > > > >> > >> > As the project continues to grow, we expect continued
> > > > infrastructure
> > > > > >> > >> > demand growth.
> > > > > >> > >> >
> > > > > >> > >> >
> > > > > >> > >> > FAQ: Has a sub project been promoted to a top level
> project
> > > > before?
> > > > > >> > >> >
> > > > > >> > >> > Yes, and it appears to happen commonly. The Arrow project
> > > > itself was
> > > > > >> > >> > created as a top level project from work that started in
> > > Apache
> > > > > >> Drill,
> > > > > >> > >> > and there are many sub projects of Hadoop that spun out
> as
> > > > their own
> > > > > >> > >> > top level projects such as Mahout, Avro and HBase:
> > > > > >> > >> >
> > > > > >> > >> >
> > > > > >> > >> https://news.apache.org/foundation/entry/the_apache_
> > > > > >> > software_foundation_announces4
> > > > > >> > >> >
> > > > > >> > >> >
> > > > > >> > >> >
> > > > > >> > >> > Related material:
> > > > > >> > >> > Name search request / research for DataFusion:
> > > > > >> > >> >
> https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> > > > > >> > >> > Discussion about this proposal on the arrow mailing list:
> > > > > >> > >> >
> > > > https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> > > > > >> > >> > Discussion about which repositories on the arrow mailing
> > > list:
> > > > > >> > >> >
> > > > https://lists.apache.org/thread/ob3n0d9ky0bgrryl3xn39w9k566bq00q
> > > > > >> > >> > Discussion about initial PMC on the arrow mailing list:
> > > > > >> > >> >
> > > > https://lists.apache.org/thread/pymrzcdw4qdptvby85f69rg3pcckl15b
> > > > > >> > >> > Discussion in github about creating a new DataFusion top
> > > level
> > > > > >> > >> > project:
> > > > > >> https://github.com/apache/arrow-datafusion/discussions/6475
> > > > > >> > >> > Discussion about graduating on incubator list:
> > > > > >> > >> >
> > > > https://lists.apache.org/thread/r4n73pmms1lv0jbohyx1o1z13d615t99
> > > > > >> > >> > Original Proposal for the Arrow project:
> > > > > >> > >> >
> > > > https://lists.apache.org/thread/x2qzdwglm8pkqp9gv03bbgw17khl7pq3
> > > > > >> > >> >
> > > > > >> > >>
> > > > > >> >
> > > > > >>
> > > >
> > >
> >
>
Re: [VOTE] Move Arrow DataFusion Subproject to new Top Level Apache Project
Posted by Yang Jiang <ya...@apache.org>.
+1 (non-binding)
On 2024/03/01 18:08:26 Daniël Heres wrote:
> +1 (binding)
>
> On Fri, Mar 1, 2024, 19:05 Chao Sun <su...@apache.org> wrote:
>
> > +1 (non-binding)
> >
> > On Fri, Mar 1, 2024 at 9:59 AM QP Hou <qp...@neuralink.com> wrote:
> >
> > > +1 (binding)
> > >
> > > exciting milestone :)
> > >
> > > On Fri, Mar 1, 2024 at 9:49 AM David Li <li...@apache.org> wrote:
> > > >
> > > > +1
> > > >
> > > > On Fri, Mar 1, 2024, at 12:06, Jorge Cardoso Leitão wrote:
> > > > > +1 - great work!!!
> > > > >
> > > > > On Fri, Mar 1, 2024 at 5:49 PM Micah Kornfield <
> > emkornfield@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> +1 (binding)
> > > > >>
> > > > >> On Friday, March 1, 2024, Uwe L. Korn <uw...@xhochy.com> wrote:
> > > > >>
> > > > >> > +1 (binding)
> > > > >> >
> > > > >> > On Fri, Mar 1, 2024, at 2:37 PM, Andy Grove wrote:
> > > > >> > > +1 (binding)
> > > > >> > >
> > > > >> > > On Fri, Mar 1, 2024 at 6:20 AM Weston Pace <
> > weston.pace@gmail.com
> > > >
> > > > >> > wrote:
> > > > >> > >
> > > > >> > >> +1 (binding)
> > > > >> > >>
> > > > >> > >> On Fri, Mar 1, 2024 at 3:33 AM Andrew Lamb <
> > alamb@influxdata.com
> > > >
> > > > >> > wrote:
> > > > >> > >>
> > > > >> > >> > Hello,
> > > > >> > >> >
> > > > >> > >> > As we have discussed[1][2] I would like to vote on the
> > > proposal to
> > > > >> > >> > create a new Apache Top Level Project for DataFusion. The
> > text
> > > of
> > > > >> the
> > > > >> > >> > proposed resolution and background document is copy/pasted
> > > below
> > > > >> > >> >
> > > > >> > >> > If the community is in favor of this, we plan to submit the
> > > > >> resolution
> > > > >> > >> > to the ASF board for approval with the next Arrow report (for
> > > the
> > > > >> > >> > April 2024 board meeting).
> > > > >> > >> >
> > > > >> > >> > The vote will be open for at least 7 days.
> > > > >> > >> >
> > > > >> > >> > [ ] +1 Accept this Proposal
> > > > >> > >> > [ ] +0
> > > > >> > >> > [ ] -1 Do not accept this proposal because...
> > > > >> > >> >
> > > > >> > >> > Andrew
> > > > >> > >> >
> > > > >> > >> > [1]
> > > > >> https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> > > > >> > >> > [2]
> > > https://github.com/apache/arrow-datafusion/discussions/6475
> > > > >> > >> >
> > > > >> > >> > ---------- Proposed Resolution ---------
> > > > >> > >> >
> > > > >> > >> > Resolution to Create the Apache DataFusion Project from the
> > > Apache
> > > > >> > >> > Arrow DataFusion Sub Project
> > > > >> > >> >
> > > > >> > >> > =============================================================
> > > > >> > >> >
> > > > >> > >> > X. Establish the Apache DataFusion Project
> > > > >> > >> >
> > > > >> > >> > WHEREAS, the Board of Directors deems it to be in the best
> > > > >> > >> > interests of the Foundation and consistent with the
> > > > >> > >> > Foundation's purpose to establish a Project Management
> > > > >> > >> > Committee charged with the creation and maintenance of
> > > > >> > >> > open-source software related to an extensible query engine
> > > > >> > >> > for distribution at no charge to the public.
> > > > >> > >> >
> > > > >> > >> > NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> > > > >> > >> > Committee (PMC), to be known as the "Apache DataFusion
> > > Project",
> > > > >> > >> > be and hereby is established pursuant to Bylaws of the
> > > > >> > >> > Foundation; and be it further
> > > > >> > >> >
> > > > >> > >> > RESOLVED, that the Apache DataFusion Project be and hereby is
> > > > >> > >> > responsible for the creation and maintenance of software
> > > > >> > >> > related to an extensible query engine; and be it further
> > > > >> > >> >
> > > > >> > >> > RESOLVED, that the office of "Vice President, Apache
> > > DataFusion" be
> > > > >> > >> > and hereby is created, the person holding such office to
> > > > >> > >> > serve at the direction of the Board of Directors as the chair
> > > > >> > >> > of the Apache DataFusion Project, and to have primary
> > > responsibility
> > > > >> > >> > for management of the projects within the scope of
> > > > >> > >> > responsibility of the Apache DataFusion Project; and be it
> > > further
> > > > >> > >> >
> > > > >> > >> > RESOLVED, that the persons listed immediately below be and
> > > > >> > >> > hereby are appointed to serve as the initial members of the
> > > > >> > >> > Apache DataFusion Project:
> > > > >> > >> >
> > > > >> > >> > * Andy Grove (agrove@apache.org)
> > > > >> > >> > * Andrew Lamb (alamb@apache.org)
> > > > >> > >> > * Daniël Heres (dheres@apache.org)
> > > > >> > >> > * Jie Wen (jakevin@apache.org)
> > > > >> > >> > * Kun Liu (liukun@apache.org)
> > > > >> > >> > * Liang-Chi Hsieh (viirya@apache.org)
> > > > >> > >> > * Qingping Hou: (houqp@apache.org)
> > > > >> > >> > * Wes McKinney(wesm@apache.org)
> > > > >> > >> > * Will Jones (wjones127@apache.org)
> > > > >> > >> >
> > > > >> > >> > RESOLVED, that the Apache DataFusion Project be and hereby
> > > > >> > >> > is tasked with the migration and rationalization of the
> > Apache
> > > > >> > >> > Arrow DataFusion sub-project; and be it further
> > > > >> > >> >
> > > > >> > >> > RESOLVED, that all responsibilities pertaining to the Apache
> > > > >> > >> > Arrow DataFusion sub-project encumbered upon the
> > > > >> > >> > Apache Arrow Project are hereafter discharged.
> > > > >> > >> >
> > > > >> > >> > NOW, THEREFORE, BE IT FURTHER RESOLVED, that Andrew Lamb
> > > > >> > >> > be appointed to the office of Vice President, Apache
> > > DataFusion, to
> > > > >> > >> > serve in accordance with and subject to the direction of the
> > > > >> > >> > Board of Directors and the Bylaws of the Foundation until
> > > > >> > >> > death, resignation, retirement, removal or disqualification,
> > > > >> > >> > or until a successor is appointed.
> > > > >> > >> > =============================================================
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> > -------
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> > Summary:
> > > > >> > >> >
> > > > >> > >> > We propose creating a new top level project, Apache
> > > DataFusion, from
> > > > >> > >> > an existing sub project of Apache Arrow to facilitate
> > > additional
> > > > >> > >> > community and project growth.
> > > > >> > >> >
> > > > >> > >> > Abstract
> > > > >> > >> >
> > > > >> > >> > Apache Arrow DataFusion[1] is a very fast, extensible query
> > > engine
> > > > >> > >> > for building high-quality data-centric systems in Rust, using
> > > the
> > > > >> > >> > Apache Arrow in-memory format. DataFusion offers SQL and
> > > Dataframe
> > > > >> > >> > APIs, excellent performance, built-in support for CSV,
> > Parquet,
> > > > >> JSON,
> > > > >> > >> > and Avro, extensive customization, and a great community.
> > > > >> > >> >
> > > > >> > >> > [1] https://arrow.apache.org/datafusion/
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> > Proposal
> > > > >> > >> >
> > > > >> > >> > We propose creating a new top level ASF project, Apache
> > > DataFusion,
> > > > >> > >> > governed initially by a subset of the Apache Arrow project’s
> > > PMC and
> > > > >> > >> > committers. The project’s code is in five existing git
> > > repositories,
> > > > >> > >> > currently governed by Apache Arrow which would transfer to
> > the
> > > new
> > > > >> top
> > > > >> > >> > level project.
> > > > >> > >> >
> > > > >> > >> > Background
> > > > >> > >> >
> > > > >> > >> > When DataFusion was initially donated to the Arrow project,
> > it
> > > did
> > > > >> not
> > > > >> > >> > have a strong enough community to stand on its own. It has
> > > since
> > > > >> grown
> > > > >> > >> > significantly, and benefited immensely from being part of
> > > Arrow and
> > > > >> > >> > nurturing of the Apache Way, and now has a community strong
> > > enough
> > > > >> to
> > > > >> > >> > stand on its own and that would benefit from focused
> > governance
> > > > >> > >> > attention.
> > > > >> > >> >
> > > > >> > >> > The community has discussed this idea publicly for more than
> > 6
> > > > >> months
> > > > >> > >> > https://github.com/apache/arrow-datafusion/discussions/6475
> > > and
> > > > >> > >> > briefly on the Arrow PMC mailing list
> > > > >> > >> >
> > > https://lists.apache.org/thread/thv2jdm6640l6gm88hy8jhk5prjww0cs.
> > > > >> As
> > > > >> > >> > of the time of this writing both had exclusively positive
> > > reactions.
> > > > >> > >> >
> > > > >> > >> > Several current members of the Arrow PMC are both active
> > > > >> contributors
> > > > >> > >> > to DataFusion and understand and believe deeply in the Apache
> > > Way,
> > > > >> and
> > > > >> > >> > play active governance roles in the Arrow project as PMC
> > > members and
> > > > >> > >> > PMC chairs, guiding the community, and releasing software
> > > versions.
> > > > >> > >> > With this existing governance experience and structure, the
> > > new top
> > > > >> > >> > level project will be able to function well immediately and
> > > > >> > >> > independently.
> > > > >> > >> >
> > > > >> > >> > Overview of DataFusion
> > > > >> > >> >
> > > > >> > >> > Current Status
> > > > >> > >> >
> > > > >> > >> > Meritocracy
> > > > >> > >> >
> > > > >> > >> > DataFusion has been developed as part of Apache Arrow and
> > thus
> > > has
> > > > >> > >> > been operating as a meritocracy. Many of the developers of
> > > > >> DataFusion
> > > > >> > >> > are Arrow PMC members or committers. The DataFusion project
> > > plans to
> > > > >> > >> > continue adding new PMC and committers as the project matures
> > > and
> > > > >> > >> > grows.
> > > > >> > >> >
> > > > >> > >> > Community
> > > > >> > >> >
> > > > >> > >> > The DataFusion development team seeks to foster the
> > > development and
> > > > >> > >> > user communities. We hope that becoming a separate project
> > > will help
> > > > >> > >> > both Arrow and DataFusion communities by being more focused.
> > > > >> Focused
> > > > >> > >> > governance will make it easier to grow the community of
> > > committers
> > > > >> and
> > > > >> > >> > PMC members and make the organization more clear to others.
> > > > >> > >> >
> > > > >> > >> > Alignment
> > > > >> > >> >
> > > > >> > >> > The ASF is a natural host for DataFusion given that it is
> > > already
> > > > >> the
> > > > >> > >> > home of Arrow, Parquet, and other related distributed system,
> > > > >> storage
> > > > >> > >> > and query execution systems.
> > > > >> > >> >
> > > > >> > >> > Project Leadership
> > > > >> > >> >
> > > > >> > >> > Proposed Initial PMC
> > > > >> > >> >
> > > > >> > >> > We propose the following people as the initial DataFusion PMC
> > > > >> members.
> > > > >> > >> > This is a subset of the existing Arrow PMC members who
> > > contribute to
> > > > >> > >> > DataFusion
> > https://people.apache.org/phonebook.html?unix=arrow
> > > > >> > >> >
> > > > >> > >> > Andy Grove (agrove): Arrow PMC Chair
> > > > >> > >> > Andrew Lamb (alamb): Arrow PMC, past Arrow PMC Chair
> > > > >> > >> > Daniël Heres (dheres) Arrow PMC
> > > > >> > >> > Jie Wen (jakevin): Arrow PMC, Doris Committer
> > > > >> > >> > Kun Liu (liukun): Arrow PMC, IoTDB PMC, TSFile PMC
> > > > >> > >> > Liang-Chi Hsieh (viirya): Arrow PMC, Spark PMC
> > > > >> > >> > Qingping Hou: (houqp): Arrow PMC
> > > > >> > >> > Wes McKinney(wesm): Arrow PMC, ASF Member
> > > > >> > >> > Will Jones (wjones127): Arrow PMC
> > > > >> > >> >
> > > > >> > >> > We’d like to propose Andrew Lamb as the initial Chair, (and
> > > thus ASF
> > > > >> > >> > VP) for the DataFusion project.
> > > > >> > >> >
> > > > >> > >> > Affiliations
> > > > >> > >> >
> > > > >> > >> > Andy Grove (agrove): NVidia
> > > > >> > >> > Andrew Lamb (alamb): InfluxData
> > > > >> > >> > Daniël Heres (dheres): Coralogix
> > > > >> > >> > Jie Wen (jakevin): SelectDB
> > > > >> > >> > Kun Liu (liukun): Ebay
> > > > >> > >> > Liang-Chi Hsieh (viirya): Apple
> > > > >> > >> > Qingping Hou: (houqp): Scribd
> > > > >> > >> > Wes McKinney(wesm): Posit
> > > > >> > >> > Will Jones (wjones127): LanceDB
> > > > >> > >> >
> > > > >> > >> > Proposed Initial Committers
> > > > >> > >> >
> > > > >> > >> > In addition to the PMC, we propose the following people as
> > the
> > > > >> initial
> > > > >> > >> > DataFusion committers. This is a subset of the existing Arrow
> > > > >> > >> > committers who contribute to DataFusion
> > > > >> > >> > https://people.apache.org/phonebook.html?unix=arrow
> > > > >> > >> >
> > > > >> > >> > akurmustafa Mustafa Akur (Synnada)
> > > > >> > >> > avantgardner Brent Gardner (Coralogix)
> > > > >> > >> > comphead Oleks V. (Unaffiliated)
> > > > >> > >> > jayzhan Jay Zhan (Unaffiliated)
> > > > >> > >> > jeffreyvo Jeffry Vo (Unaffiliated)
> > > > >> > >> > jiayuliu Liu Jiayu (Airbnb)
> > > > >> > >> > mete Metehan Yildirim (Synnada)
> > > > >> > >> > mingmwang Wang Mingming (Ebay)
> > > > >> > >> > mneumann Marco Neumann (InfluxData)
> > > > >> > >> > nju_yaho Zhong Yanghong (Ebay)
> > > > >> > >> > ozankabak Mehmet Ozan Kabak (Synnada)
> > > > >> > >> > paddyhoran Paddy Horan (Assured Allies)
> > > > >> > >> > rdettai Rémi Dettai (Cloudfuse)
> > > > >> > >> > sunchao Chao Sun (Apple)
> > > > >> > >> > thinkharderdev Daniel Harris (Coralogix)
> > > > >> > >> > tustvold Raphael Taylor-Davies (InfluxData)
> > > > >> > >> > wayne Ruihang Xia (Greptime)
> > > > >> > >> > xudong963 Xudong Wang (ByteDance)
> > > > >> > >> > yjshen Yijie Shen (Space and Time)
> > > > >> > >> > yangjiang Yang Jiang (ebay)
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> > Risk Assessments
> > > > >> > >> >
> > > > >> > >> > Naming / Trademarks
> > > > >> > >> >
> > > > >> > >> > As a sub-project of Arrow, the DataFusion name has been used
> > > for
> > > > >> over
> > > > >> > >> > 4 years without any known issues. A podling name search did
> > > not turn
> > > > >> > >> > up any concerns and was approved:
> > > > >> > >> > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> > > > >> > >> >
> > > > >> > >> > Legal / IP Clearance
> > > > >> > >> >
> > > > >> > >> > All DataFusion code has either been donated to the Arrow
> > > project
> > > > >> with
> > > > >> > >> > appropriate IP clearance or has been developed directly
> > under
> > > ASF
> > > > >> > >> > processes and procedures. Thus creating a new top level
> > project
> > > > >> poses
> > > > >> > >> > no new Legal or IP risks.
> > > > >> > >> >
> > > > >> > >> > Code Extraction
> > > > >> > >> >
> > > > >> > >> > The relevant code is already in 5 separate repositories:
> > > > >> > >> > https://github.com/apache/arrow-datafusion/
> > > > >> > >> > https://github.com/apache/arrow-datafusion-python
> > > > >> > >> > https://github.com/apache/arrow-ballista
> > > > >> > >> > https://github.com/apache/arrow-ballista-python
> > > > >> > >> > https://github.com/apache/arrow-datafusion-comet
> > > > >> > >> >
> > > > >> > >> > We foresee no issues with code extraction and propose these
> > > > >> > >> > repositories be renamed to reflect top level projects
> > > > >> > >> >
> > > > >> > >> > Note: https://github.com/apache/arrow-rs, the Rust
> > > implementation
> > > > >> of
> > > > >> > >> > Arrow, would remain part of the Arrow project.
> > > > >> > >> >
> > > > >> > >> > Orphaned Products
> > > > >> > >> >
> > > > >> > >> > DataFusion is known to be used in many open source and
> > > commercial
> > > > >> > >> > projects
> > > > >> > >> >
> > > > >> > >> https://arrow.apache.org/datafusion/user-guide/
> > > > >> > introduction.html#known-users
> > > > >> > >> > ,
> > > > >> > >> > has had multiple commits daily for several years, and its
> > > adoption
> > > > >> and
> > > > >> > >> > number of contributors appears to be growing. We do not
> > > foresee the
> > > > >> > >> > project being orphaned in the next several years.
> > > > >> > >> >
> > > > >> > >> > Inexperience with Open Source
> > > > >> > >> >
> > > > >> > >> > The proposed PMC has extensive experience with Apache Arrow
> > and
> > > > >> other
> > > > >> > >> > Apache projects, and includes PMC members, PMC chairs and an
> > > ASF
> > > > >> > >> > Member. The DataFusion PMC and more experienced committers
> > will
> > > > >> > >> > continue to coach new community members who may be less
> > > familiar
> > > > >> with
> > > > >> > >> > the Apache Way.
> > > > >> > >> >
> > > > >> > >> > Homogeneous Developers
> > > > >> > >> >
> > > > >> > >> > The 9 proposed PMC members are from 9 different employers and
> > > the
> > > > >> > >> > proposed committers are similarly distributed across
> > > affiliations.
> > > > >> No
> > > > >> > >> > specific entity employs more than 3 total proposed
> > developers.
> > > > >> > >> >
> > > > >> > >> > Reliance on Salaried Developers
> > > > >> > >> >
> > > > >> > >> > A substantial amount of work on DataFusion has been by
> > salaried
> > > > >> > >> > developers, but it also has a long tradition of attracting
> > > > >> > >> > contributions from students and hobbyists and we plan no
> > > changes in
> > > > >> > >> > contribution structure.
> > > > >> > >> >
> > > > >> > >> > Relationships with Other Apache Products
> > > > >> > >> >
> > > > >> > >> > DataFusion will obviously have a strong relationship with the
> > > Arrow
> > > > >> > >> > project given the overlap in people. We don’t foresee close
> > > > >> > >> > collaboration with other projects at this time.
> > > > >> > >> >
> > > > >> > >> > Cryptography
> > > > >> > >> >
> > > > >> > >> > DataFusion does not directly support encryption and there are
> > > no
> > > > >> > >> > near-term plans to add support for encryption. Users who need
> > > this
> > > > >> > >> > functionality can use the extension APIs.
> > > > >> > >> >
> > > > >> > >> > Required Resources
> > > > >> > >> >
> > > > >> > >> > Mailing Lists
> > > > >> > >> >
> > > > >> > >> > - private@datafusion.apache.org for private PMC discussions
> > > (with
> > > > >> > >> > moderated subscriptions)
> > > > >> > >> > - dev@datafusion.apache.org
> > > > >> > >> > - commits@datafusion.apache.org
> > > > >> > >> > - user@datafusion.apache.org
> > > > >> > >> >
> > > > >> > >> > Version Control
> > > > >> > >> >
> > > > >> > >> > We propose to continue to use git for source control and
> > > github for
> > > > >> > >> > hosting and testing resources.
> > > > >> > >> >
> > > > >> > >> > We also need to rename the github repositories to reflect the
> > > new
> > > > >> top
> > > > >> > >> > level names:
> > > > >> > >> >
> > > > >> > >> > https://github.com/apache/arrow-datafusion/ →
> > > apache/datafusion
> > > > >> > >> > https://github.com/apache/arrow-datafusion-python →
> > > > >> > >> > apache/datafusion-python
> > > > >> > >> > https://github.com/apache/arrow-ballista →
> > > > >> apache/datafusion-ballista
> > > > >> > >> > https://github.com/apache/arrow-ballista-python →
> > > > >> > >> > apache/datafusion-ballista-python
> > > > >> > >> > https://github.com/apache/arrow-datafusion-comet →
> > > > >> > >> apache/datafusion-comet
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> > Issue Tracking
> > > > >> > >> >
> > > > >> > >> > DataFusion would continue to use github for its issue
> > tracking
> > > and
> > > > >> > >> > communications
> > > > >> > >> >
> > > > >> > >> > Other Resources
> > > > >> > >> >
> > > > >> > >> > The existing repositories already make use of existing Apache
> > > > >> > >> > infrastructure, and we expect no change in the initial
> > resource
> > > > >> usage.
> > > > >> > >> > As the project continues to grow, we expect continued
> > > infrastructure
> > > > >> > >> > demand growth.
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> > FAQ: Has a sub project been promoted to a top level project
> > > before?
> > > > >> > >> >
> > > > >> > >> > Yes, and it appears to happen commonly. The Arrow project
> > > itself was
> > > > >> > >> > created as a top level project from work that started in
> > Apache
> > > > >> Drill,
> > > > >> > >> > and there are many sub projects of Hadoop that spun out as
> > > their own
> > > > >> > >> > top level projects such as Mahout, Avro and HBase:
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> https://news.apache.org/foundation/entry/the_apache_
> > > > >> > software_foundation_announces4
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> > Related material:
> > > > >> > >> > Name search request / research for DataFusion:
> > > > >> > >> > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> > > > >> > >> > Discussion about this proposal on the arrow mailing list:
> > > > >> > >> >
> > > https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> > > > >> > >> > Discussion about which repositories on the arrow mailing
> > list:
> > > > >> > >> >
> > > https://lists.apache.org/thread/ob3n0d9ky0bgrryl3xn39w9k566bq00q
> > > > >> > >> > Discussion about initial PMC on the arrow mailing list:
> > > > >> > >> >
> > > https://lists.apache.org/thread/pymrzcdw4qdptvby85f69rg3pcckl15b
> > > > >> > >> > Discussion in github about creating a new DataFusion top
> > level
> > > > >> > >> > project:
> > > > >> https://github.com/apache/arrow-datafusion/discussions/6475
> > > > >> > >> > Discussion about graduating on incubator list:
> > > > >> > >> >
> > > https://lists.apache.org/thread/r4n73pmms1lv0jbohyx1o1z13d615t99
> > > > >> > >> > Original Proposal for the Arrow project:
> > > > >> > >> >
> > > https://lists.apache.org/thread/x2qzdwglm8pkqp9gv03bbgw17khl7pq3
> > > > >> > >> >
> > > > >> > >>
> > > > >> >
> > > > >>
> > >
> >
>
Re: [VOTE] Move Arrow DataFusion Subproject to new Top Level Apache Project
Posted by Daniël Heres <da...@gmail.com>.
+1 (binding)
On Fri, Mar 1, 2024, 19:05 Chao Sun <su...@apache.org> wrote:
> +1 (non-binding)
>
> On Fri, Mar 1, 2024 at 9:59 AM QP Hou <qp...@neuralink.com> wrote:
>
> > +1 (binding)
> >
> > exciting milestone :)
> >
> > On Fri, Mar 1, 2024 at 9:49 AM David Li <li...@apache.org> wrote:
> > >
> > > +1
> > >
> > > On Fri, Mar 1, 2024, at 12:06, Jorge Cardoso Leitão wrote:
> > > > +1 - great work!!!
> > > >
> > > > On Fri, Mar 1, 2024 at 5:49 PM Micah Kornfield <
> emkornfield@gmail.com>
> > > > wrote:
> > > >
> > > >> +1 (binding)
> > > >>
> > > >> On Friday, March 1, 2024, Uwe L. Korn <uw...@xhochy.com> wrote:
> > > >>
> > > >> > +1 (binding)
> > > >> >
> > > >> > On Fri, Mar 1, 2024, at 2:37 PM, Andy Grove wrote:
> > > >> > > +1 (binding)
> > > >> > >
> > > >> > > On Fri, Mar 1, 2024 at 6:20 AM Weston Pace <
> weston.pace@gmail.com
> > >
> > > >> > wrote:
> > > >> > >
> > > >> > >> +1 (binding)
> > > >> > >>
> > > >> > >> On Fri, Mar 1, 2024 at 3:33 AM Andrew Lamb <
> alamb@influxdata.com
> > >
> > > >> > wrote:
> > > >> > >>
> > > >> > >> > Hello,
> > > >> > >> >
> > > >> > >> > As we have discussed[1][2] I would like to vote on the
> > proposal to
> > > >> > >> > create a new Apache Top Level Project for DataFusion. The
> text
> > of
> > > >> the
> > > >> > >> > proposed resolution and background document is copy/pasted
> > below
> > > >> > >> >
> > > >> > >> > If the community is in favor of this, we plan to submit the
> > > >> resolution
> > > >> > >> > to the ASF board for approval with the next Arrow report (for
> > the
> > > >> > >> > April 2024 board meeting).
> > > >> > >> >
> > > >> > >> > The vote will be open for at least 7 days.
> > > >> > >> >
> > > >> > >> > [ ] +1 Accept this Proposal
> > > >> > >> > [ ] +0
> > > >> > >> > [ ] -1 Do not accept this proposal because...
> > > >> > >> >
> > > >> > >> > Andrew
> > > >> > >> >
> > > >> > >> > [1]
> > > >> https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> > > >> > >> > [2]
> > https://github.com/apache/arrow-datafusion/discussions/6475
> > > >> > >> >
> > > >> > >> > ---------- Proposed Resolution ---------
> > > >> > >> >
> > > >> > >> > Resolution to Create the Apache DataFusion Project from the
> > Apache
> > > >> > >> > Arrow DataFusion Sub Project
> > > >> > >> >
> > > >> > >> > =============================================================
> > > >> > >> >
> > > >> > >> > X. Establish the Apache DataFusion Project
> > > >> > >> >
> > > >> > >> > WHEREAS, the Board of Directors deems it to be in the best
> > > >> > >> > interests of the Foundation and consistent with the
> > > >> > >> > Foundation's purpose to establish a Project Management
> > > >> > >> > Committee charged with the creation and maintenance of
> > > >> > >> > open-source software related to an extensible query engine
> > > >> > >> > for distribution at no charge to the public.
> > > >> > >> >
> > > >> > >> > NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> > > >> > >> > Committee (PMC), to be known as the "Apache DataFusion
> > Project",
> > > >> > >> > be and hereby is established pursuant to Bylaws of the
> > > >> > >> > Foundation; and be it further
> > > >> > >> >
> > > >> > >> > RESOLVED, that the Apache DataFusion Project be and hereby is
> > > >> > >> > responsible for the creation and maintenance of software
> > > >> > >> > related to an extensible query engine; and be it further
> > > >> > >> >
> > > >> > >> > RESOLVED, that the office of "Vice President, Apache
> > DataFusion" be
> > > >> > >> > and hereby is created, the person holding such office to
> > > >> > >> > serve at the direction of the Board of Directors as the chair
> > > >> > >> > of the Apache DataFusion Project, and to have primary
> > responsibility
> > > >> > >> > for management of the projects within the scope of
> > > >> > >> > responsibility of the Apache DataFusion Project; and be it
> > further
> > > >> > >> >
> > > >> > >> > RESOLVED, that the persons listed immediately below be and
> > > >> > >> > hereby are appointed to serve as the initial members of the
> > > >> > >> > Apache DataFusion Project:
> > > >> > >> >
> > > >> > >> > * Andy Grove (agrove@apache.org)
> > > >> > >> > * Andrew Lamb (alamb@apache.org)
> > > >> > >> > * Daniël Heres (dheres@apache.org)
> > > >> > >> > * Jie Wen (jakevin@apache.org)
> > > >> > >> > * Kun Liu (liukun@apache.org)
> > > >> > >> > * Liang-Chi Hsieh (viirya@apache.org)
> > > >> > >> > * Qingping Hou: (houqp@apache.org)
> > > >> > >> > * Wes McKinney(wesm@apache.org)
> > > >> > >> > * Will Jones (wjones127@apache.org)
> > > >> > >> >
> > > >> > >> > RESOLVED, that the Apache DataFusion Project be and hereby
> > > >> > >> > is tasked with the migration and rationalization of the
> Apache
> > > >> > >> > Arrow DataFusion sub-project; and be it further
> > > >> > >> >
> > > >> > >> > RESOLVED, that all responsibilities pertaining to the Apache
> > > >> > >> > Arrow DataFusion sub-project encumbered upon the
> > > >> > >> > Apache Arrow Project are hereafter discharged.
> > > >> > >> >
> > > >> > >> > NOW, THEREFORE, BE IT FURTHER RESOLVED, that Andrew Lamb
> > > >> > >> > be appointed to the office of Vice President, Apache
> > DataFusion, to
> > > >> > >> > serve in accordance with and subject to the direction of the
> > > >> > >> > Board of Directors and the Bylaws of the Foundation until
> > > >> > >> > death, resignation, retirement, removal or disqualification,
> > > >> > >> > or until a successor is appointed.
> > > >> > >> > =============================================================
> > > >> > >> >
> > > >> > >> >
> > > >> > >> > -------
> > > >> > >> >
> > > >> > >> >
> > > >> > >> > Summary:
> > > >> > >> >
> > > >> > >> > We propose creating a new top level project, Apache
> > DataFusion, from
> > > >> > >> > an existing sub project of Apache Arrow to facilitate
> > additional
> > > >> > >> > community and project growth.
> > > >> > >> >
> > > >> > >> > Abstract
> > > >> > >> >
> > > >> > >> > Apache Arrow DataFusion[1] is a very fast, extensible query
> > engine
> > > >> > >> > for building high-quality data-centric systems in Rust, using
> > the
> > > >> > >> > Apache Arrow in-memory format. DataFusion offers SQL and
> > Dataframe
> > > >> > >> > APIs, excellent performance, built-in support for CSV,
> Parquet,
> > > >> JSON,
> > > >> > >> > and Avro, extensive customization, and a great community.
> > > >> > >> >
> > > >> > >> > [1] https://arrow.apache.org/datafusion/
> > > >> > >> >
> > > >> > >> >
> > > >> > >> > Proposal
> > > >> > >> >
> > > >> > >> > We propose creating a new top level ASF project, Apache
> > DataFusion,
> > > >> > >> > governed initially by a subset of the Apache Arrow project’s
> > PMC and
> > > >> > >> > committers. The project’s code is in five existing git
> > repositories,
> > > >> > >> > currently governed by Apache Arrow which would transfer to
> the
> > new
> > > >> top
> > > >> > >> > level project.
> > > >> > >> >
> > > >> > >> > Background
> > > >> > >> >
> > > >> > >> > When DataFusion was initially donated to the Arrow project,
> it
> > did
> > > >> not
> > > >> > >> > have a strong enough community to stand on its own. It has
> > since
> > > >> grown
> > > >> > >> > significantly, and benefited immensely from being part of
> > Arrow and
> > > >> > >> > nurturing of the Apache Way, and now has a community strong
> > enough
> > > >> to
> > > >> > >> > stand on its own and that would benefit from focused
> governance
> > > >> > >> > attention.
> > > >> > >> >
> > > >> > >> > The community has discussed this idea publicly for more than
> 6
> > > >> months
> > > >> > >> > https://github.com/apache/arrow-datafusion/discussions/6475
> > and
> > > >> > >> > briefly on the Arrow PMC mailing list
> > > >> > >> >
> > https://lists.apache.org/thread/thv2jdm6640l6gm88hy8jhk5prjww0cs.
> > > >> As
> > > >> > >> > of the time of this writing both had exclusively positive
> > reactions.
> > > >> > >> >
> > > >> > >> > Several current members of the Arrow PMC are both active
> > > >> contributors
> > > >> > >> > to DataFusion and understand and believe deeply in the Apache
> > Way,
> > > >> and
> > > >> > >> > play active governance roles in the Arrow project as PMC
> > members and
> > > >> > >> > PMC chairs, guiding the community, and releasing software
> > versions.
> > > >> > >> > With this existing governance experience and structure, the
> > new top
> > > >> > >> > level project will be able to function well immediately and
> > > >> > >> > independently.
> > > >> > >> >
> > > >> > >> > Overview of DataFusion
> > > >> > >> >
> > > >> > >> > Current Status
> > > >> > >> >
> > > >> > >> > Meritocracy
> > > >> > >> >
> > > >> > >> > DataFusion has been developed as part of Apache Arrow and
> thus
> > has
> > > >> > >> > been operating as a meritocracy. Many of the developers of
> > > >> DataFusion
> > > >> > >> > are Arrow PMC members or committers. The DataFusion project
> > plans to
> > > >> > >> > continue adding new PMC and committers as the project matures
> > and
> > > >> > >> > grows.
> > > >> > >> >
> > > >> > >> > Community
> > > >> > >> >
> > > >> > >> > The DataFusion development team seeks to foster the
> > development and
> > > >> > >> > user communities. We hope that becoming a separate project
> > will help
> > > >> > >> > both Arrow and DataFusion communities by being more focused.
> > > >> Focused
> > > >> > >> > governance will make it easier to grow the community of
> > committers
> > > >> and
> > > >> > >> > PMC members and make the organization more clear to others.
> > > >> > >> >
> > > >> > >> > Alignment
> > > >> > >> >
> > > >> > >> > The ASF is a natural host for DataFusion given that it is
> > already
> > > >> the
> > > >> > >> > home of Arrow, Parquet, and other related distributed system,
> > > >> storage
> > > >> > >> > and query execution systems.
> > > >> > >> >
> > > >> > >> > Project Leadership
> > > >> > >> >
> > > >> > >> > Proposed Initial PMC
> > > >> > >> >
> > > >> > >> > We propose the following people as the initial DataFusion PMC
> > > >> members.
> > > >> > >> > This is a subset of the existing Arrow PMC members who
> > contribute to
> > > >> > >> > DataFusion
> https://people.apache.org/phonebook.html?unix=arrow
> > > >> > >> >
> > > >> > >> > Andy Grove (agrove): Arrow PMC Chair
> > > >> > >> > Andrew Lamb (alamb): Arrow PMC, past Arrow PMC Chair
> > > >> > >> > Daniël Heres (dheres) Arrow PMC
> > > >> > >> > Jie Wen (jakevin): Arrow PMC, Doris Committer
> > > >> > >> > Kun Liu (liukun): Arrow PMC, IoTDB PMC, TSFile PMC
> > > >> > >> > Liang-Chi Hsieh (viirya): Arrow PMC, Spark PMC
> > > >> > >> > Qingping Hou: (houqp): Arrow PMC
> > > >> > >> > Wes McKinney(wesm): Arrow PMC, ASF Member
> > > >> > >> > Will Jones (wjones127): Arrow PMC
> > > >> > >> >
> > > >> > >> > We’d like to propose Andrew Lamb as the initial Chair, (and
> > thus ASF
> > > >> > >> > VP) for the DataFusion project.
> > > >> > >> >
> > > >> > >> > Affiliations
> > > >> > >> >
> > > >> > >> > Andy Grove (agrove): NVidia
> > > >> > >> > Andrew Lamb (alamb): InfluxData
> > > >> > >> > Daniël Heres (dheres): Coralogix
> > > >> > >> > Jie Wen (jakevin): SelectDB
> > > >> > >> > Kun Liu (liukun): Ebay
> > > >> > >> > Liang-Chi Hsieh (viirya): Apple
> > > >> > >> > Qingping Hou: (houqp): Scribd
> > > >> > >> > Wes McKinney(wesm): Posit
> > > >> > >> > Will Jones (wjones127): LanceDB
> > > >> > >> >
> > > >> > >> > Proposed Initial Committers
> > > >> > >> >
> > > >> > >> > In addition to the PMC, we propose the following people as
> the
> > > >> initial
> > > >> > >> > DataFusion committers. This is a subset of the existing Arrow
> > > >> > >> > committers who contribute to DataFusion
> > > >> > >> > https://people.apache.org/phonebook.html?unix=arrow
> > > >> > >> >
> > > >> > >> > akurmustafa Mustafa Akur (Synnada)
> > > >> > >> > avantgardner Brent Gardner (Coralogix)
> > > >> > >> > comphead Oleks V. (Unaffiliated)
> > > >> > >> > jayzhan Jay Zhan (Unaffiliated)
> > > >> > >> > jeffreyvo Jeffry Vo (Unaffiliated)
> > > >> > >> > jiayuliu Liu Jiayu (Airbnb)
> > > >> > >> > mete Metehan Yildirim (Synnada)
> > > >> > >> > mingmwang Wang Mingming (Ebay)
> > > >> > >> > mneumann Marco Neumann (InfluxData)
> > > >> > >> > nju_yaho Zhong Yanghong (Ebay)
> > > >> > >> > ozankabak Mehmet Ozan Kabak (Synnada)
> > > >> > >> > paddyhoran Paddy Horan (Assured Allies)
> > > >> > >> > rdettai Rémi Dettai (Cloudfuse)
> > > >> > >> > sunchao Chao Sun (Apple)
> > > >> > >> > thinkharderdev Daniel Harris (Coralogix)
> > > >> > >> > tustvold Raphael Taylor-Davies (InfluxData)
> > > >> > >> > wayne Ruihang Xia (Greptime)
> > > >> > >> > xudong963 Xudong Wang (ByteDance)
> > > >> > >> > yjshen Yijie Shen (Space and Time)
> > > >> > >> > yangjiang Yang Jiang (ebay)
> > > >> > >> >
> > > >> > >> >
> > > >> > >> > Risk Assessments
> > > >> > >> >
> > > >> > >> > Naming / Trademarks
> > > >> > >> >
> > > >> > >> > As a sub-project of Arrow, the DataFusion name has been used
> > for
> > > >> over
> > > >> > >> > 4 years without any known issues. A podling name search did
> > not turn
> > > >> > >> > up any concerns and was approved:
> > > >> > >> > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> > > >> > >> >
> > > >> > >> > Legal / IP Clearance
> > > >> > >> >
> > > >> > >> > All DataFusion code has either been donated to the Arrow
> > project
> > > >> with
> > > >> > >> > appropriate IP clearance or has been developed directly
> under
> > ASF
> > > >> > >> > processes and procedures. Thus creating a new top level
> project
> > > >> poses
> > > >> > >> > no new Legal or IP risks.
> > > >> > >> >
> > > >> > >> > Code Extraction
> > > >> > >> >
> > > >> > >> > The relevant code is already in 5 separate repositories:
> > > >> > >> > https://github.com/apache/arrow-datafusion/
> > > >> > >> > https://github.com/apache/arrow-datafusion-python
> > > >> > >> > https://github.com/apache/arrow-ballista
> > > >> > >> > https://github.com/apache/arrow-ballista-python
> > > >> > >> > https://github.com/apache/arrow-datafusion-comet
> > > >> > >> >
> > > >> > >> > We foresee no issues with code extraction and propose these
> > > >> > >> > repositories be renamed to reflect top level projects
> > > >> > >> >
> > > >> > >> > Note: https://github.com/apache/arrow-rs, the Rust
> > implementation
> > > >> of
> > > >> > >> > Arrow, would remain part of the Arrow project.
> > > >> > >> >
> > > >> > >> > Orphaned Products
> > > >> > >> >
> > > >> > >> > DataFusion is known to be used in many open source and
> > commercial
> > > >> > >> > projects
> > > >> > >> >
> > > >> > >> https://arrow.apache.org/datafusion/user-guide/
> > > >> > introduction.html#known-users
> > > >> > >> > ,
> > > >> > >> > has had multiple commits daily for several years, and its
> > adoption
> > > >> and
> > > >> > >> > number of contributors appears to be growing. We do not
> > foresee the
> > > >> > >> > project being orphaned in the next several years.
> > > >> > >> >
> > > >> > >> > Inexperience with Open Source
> > > >> > >> >
> > > >> > >> > The proposed PMC has extensive experience with Apache Arrow
> and
> > > >> other
> > > >> > >> > Apache projects, and includes PMC members, PMC chairs and an
> > ASF
> > > >> > >> > Member. The DataFusion PMC and more experienced committers
> will
> > > >> > >> > continue to coach new community members who may be less
> > familiar
> > > >> with
> > > >> > >> > the Apache Way.
> > > >> > >> >
> > > >> > >> > Homogeneous Developers
> > > >> > >> >
> > > >> > >> > The 9 proposed PMC members are from 9 different employers and
> > the
> > > >> > >> > proposed committers are similarly distributed across
> > affiliations.
> > > >> No
> > > >> > >> > specific entity employs more than 3 total proposed
> developers.
> > > >> > >> >
> > > >> > >> > Reliance on Salaried Developers
> > > >> > >> >
> > > >> > >> > A substantial amount of work on DataFusion has been by
> salaried
> > > >> > >> > developers, but it also has a long tradition of attracting
> > > >> > >> > contributions from students and hobbyists and we plan no
> > changes in
> > > >> > >> > contribution structure.
> > > >> > >> >
> > > >> > >> > Relationships with Other Apache Products
> > > >> > >> >
> > > >> > >> > DataFusion will obviously have a strong relationship with the
> > Arrow
> > > >> > >> > project given the overlap in people. We don’t foresee close
> > > >> > >> > collaboration with other projects at this time.
> > > >> > >> >
> > > >> > >> > Cryptography
> > > >> > >> >
> > > >> > >> > DataFusion does not directly support encryption and there are
> > no
> > > >> > >> > near-term plans to add support for encryption. Users who need
> > this
> > > >> > >> > functionality can use the extension APIs.
> > > >> > >> >
> > > >> > >> > Required Resources
> > > >> > >> >
> > > >> > >> > Mailing Lists
> > > >> > >> >
> > > >> > >> > - private@datafusion.apache.org for private PMC discussions
> > (with
> > > >> > >> > moderated subscriptions)
> > > >> > >> > - dev@datafusion.apache.org
> > > >> > >> > - commits@datafusion.apache.org
> > > >> > >> > - user@datafusion.apache.org
> > > >> > >> >
> > > >> > >> > Version Control
> > > >> > >> >
> > > >> > >> > We propose to continue to use git for source control and
> > github for
> > > >> > >> > hosting and testing resources.
> > > >> > >> >
> > > >> > >> > We also need to rename the github repositories to reflect the
> > new
> > > >> top
> > > >> > >> > level names:
> > > >> > >> >
> > > >> > >> > https://github.com/apache/arrow-datafusion/ →
> > apache/datafusion
> > > >> > >> > https://github.com/apache/arrow-datafusion-python →
> > > >> > >> > apache/datafusion-python
> > > >> > >> > https://github.com/apache/arrow-ballista →
> > > >> apache/datafusion-ballista
> > > >> > >> > https://github.com/apache/arrow-ballista-python →
> > > >> > >> > apache/datafusion-ballista-python
> > > >> > >> > https://github.com/apache/arrow-datafusion-comet →
> > > >> > >> apache/datafusion-comet
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> > Issue Tracking
> > > >> > >> >
> > > >> > >> > DataFusion would continue to use github for its issue
> tracking
> > and
> > > >> > >> > communications
> > > >> > >> >
> > > >> > >> > Other Resources
> > > >> > >> >
> > > >> > >> > The existing repositories already make use of existing Apache
> > > >> > >> > infrastructure, and we expect no change in the initial
> resource
> > > >> usage.
> > > >> > >> > As the project continues to grow, we expect continued
> > infrastructure
> > > >> > >> > demand growth.
> > > >> > >> >
> > > >> > >> >
> > > >> > >> > FAQ: Has a sub project been promoted to a top level project
> > before?
> > > >> > >> >
> > > >> > >> > Yes, and it appears to happen commonly. The Arrow project
> > itself was
> > > >> > >> > created as a top level project from work that started in
> Apache
> > > >> Drill,
> > > >> > >> > and there are many sub projects of Hadoop that spun out as
> > their own
> > > >> > >> > top level projects such as Mahout, Avro and HBase:
> > > >> > >> >
> > > >> > >> >
> > > >> > >> https://news.apache.org/foundation/entry/the_apache_
> > > >> > software_foundation_announces4
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> > Related material:
> > > >> > >> > Name search request / research for DataFusion:
> > > >> > >> > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> > > >> > >> > Discussion about this proposal on the arrow mailing list:
> > > >> > >> >
> > https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> > > >> > >> > Discussion about which repositories on the arrow mailing
> list:
> > > >> > >> >
> > https://lists.apache.org/thread/ob3n0d9ky0bgrryl3xn39w9k566bq00q
> > > >> > >> > Discussion about initial PMC on the arrow mailing list:
> > > >> > >> >
> > https://lists.apache.org/thread/pymrzcdw4qdptvby85f69rg3pcckl15b
> > > >> > >> > Discussion in github about creating a new DataFusion top
> level
> > > >> > >> > project:
> > > >> https://github.com/apache/arrow-datafusion/discussions/6475
> > > >> > >> > Discussion about graduating on incubator list:
> > > >> > >> >
> > https://lists.apache.org/thread/r4n73pmms1lv0jbohyx1o1z13d615t99
> > > >> > >> > Original Proposal for the Arrow project:
> > > >> > >> >
> > https://lists.apache.org/thread/x2qzdwglm8pkqp9gv03bbgw17khl7pq3
> > > >> > >> >
> > > >> > >>
> > > >> >
> > > >>
> >
>
Re: [VOTE] Move Arrow DataFusion Subproject to new Top Level Apache Project
Posted by Chao Sun <su...@apache.org>.
+1 (non-binding)
On Fri, Mar 1, 2024 at 9:59 AM QP Hou <qp...@neuralink.com> wrote:
> +1 (binding)
>
> exciting milestone :)
>
> On Fri, Mar 1, 2024 at 9:49 AM David Li <li...@apache.org> wrote:
> >
> > +1
> >
> > On Fri, Mar 1, 2024, at 12:06, Jorge Cardoso Leitão wrote:
> > > +1 - great work!!!
> > >
> > > On Fri, Mar 1, 2024 at 5:49 PM Micah Kornfield <em...@gmail.com>
> > > wrote:
> > >
> > >> +1 (binding)
> > >>
> > >> On Friday, March 1, 2024, Uwe L. Korn <uw...@xhochy.com> wrote:
> > >>
> > >> > +1 (binding)
> > >> >
> > >> > On Fri, Mar 1, 2024, at 2:37 PM, Andy Grove wrote:
> > >> > > +1 (binding)
> > >> > >
> > >> > > On Fri, Mar 1, 2024 at 6:20 AM Weston Pace <weston.pace@gmail.com
> >
> > >> > wrote:
> > >> > >
> > >> > >> +1 (binding)
> > >> > >>
> > >> > >> On Fri, Mar 1, 2024 at 3:33 AM Andrew Lamb <alamb@influxdata.com
> >
> > >> > wrote:
> > >> > >>
> > >> > >> > Hello,
> > >> > >> >
> > >> > >> > As we have discussed[1][2] I would like to vote on the
> proposal to
> > >> > >> > create a new Apache Top Level Project for DataFusion. The text
> of
> > >> the
> > >> > >> > proposed resolution and background document is copy/pasted
> below
> > >> > >> >
> > >> > >> > If the community is in favor of this, we plan to submit the
> > >> resolution
> > >> > >> > to the ASF board for approval with the next Arrow report (for
> the
> > >> > >> > April 2024 board meeting).
> > >> > >> >
> > >> > >> > The vote will be open for at least 7 days.
> > >> > >> >
> > >> > >> > [ ] +1 Accept this Proposal
> > >> > >> > [ ] +0
> > >> > >> > [ ] -1 Do not accept this proposal because...
> > >> > >> >
> > >> > >> > Andrew
> > >> > >> >
> > >> > >> > [1]
> > >> https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> > >> > >> > [2]
> https://github.com/apache/arrow-datafusion/discussions/6475
> > >> > >> >
> > >> > >> > ---------- Proposed Resolution ---------
> > >> > >> >
> > >> > >> > Resolution to Create the Apache DataFusion Project from the
> Apache
> > >> > >> > Arrow DataFusion Sub Project
> > >> > >> >
> > >> > >> > =============================================================
> > >> > >> >
> > >> > >> > X. Establish the Apache DataFusion Project
> > >> > >> >
> > >> > >> > WHEREAS, the Board of Directors deems it to be in the best
> > >> > >> > interests of the Foundation and consistent with the
> > >> > >> > Foundation's purpose to establish a Project Management
> > >> > >> > Committee charged with the creation and maintenance of
> > >> > >> > open-source software related to an extensible query engine
> > >> > >> > for distribution at no charge to the public.
> > >> > >> >
> > >> > >> > NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> > >> > >> > Committee (PMC), to be known as the "Apache DataFusion
> Project",
> > >> > >> > be and hereby is established pursuant to Bylaws of the
> > >> > >> > Foundation; and be it further
> > >> > >> >
> > >> > >> > RESOLVED, that the Apache DataFusion Project be and hereby is
> > >> > >> > responsible for the creation and maintenance of software
> > >> > >> > related to an extensible query engine; and be it further
> > >> > >> >
> > >> > >> > RESOLVED, that the office of "Vice President, Apache
> DataFusion" be
> > >> > >> > and hereby is created, the person holding such office to
> > >> > >> > serve at the direction of the Board of Directors as the chair
> > >> > >> > of the Apache DataFusion Project, and to have primary
> responsibility
> > >> > >> > for management of the projects within the scope of
> > >> > >> > responsibility of the Apache DataFusion Project; and be it
> further
> > >> > >> >
> > >> > >> > RESOLVED, that the persons listed immediately below be and
> > >> > >> > hereby are appointed to serve as the initial members of the
> > >> > >> > Apache DataFusion Project:
> > >> > >> >
> > >> > >> > * Andy Grove (agrove@apache.org)
> > >> > >> > * Andrew Lamb (alamb@apache.org)
> > >> > >> > * Daniël Heres (dheres@apache.org)
> > >> > >> > * Jie Wen (jakevin@apache.org)
> > >> > >> > * Kun Liu (liukun@apache.org)
> > >> > >> > * Liang-Chi Hsieh (viirya@apache.org)
> > >> > >> > * Qingping Hou: (houqp@apache.org)
> > >> > >> > * Wes McKinney(wesm@apache.org)
> > >> > >> > * Will Jones (wjones127@apache.org)
> > >> > >> >
> > >> > >> > RESOLVED, that the Apache DataFusion Project be and hereby
> > >> > >> > is tasked with the migration and rationalization of the Apache
> > >> > >> > Arrow DataFusion sub-project; and be it further
> > >> > >> >
> > >> > >> > RESOLVED, that all responsibilities pertaining to the Apache
> > >> > >> > Arrow DataFusion sub-project encumbered upon the
> > >> > >> > Apache Arrow Project are hereafter discharged.
> > >> > >> >
> > >> > >> > NOW, THEREFORE, BE IT FURTHER RESOLVED, that Andrew Lamb
> > >> > >> > be appointed to the office of Vice President, Apache
> DataFusion, to
> > >> > >> > serve in accordance with and subject to the direction of the
> > >> > >> > Board of Directors and the Bylaws of the Foundation until
> > >> > >> > death, resignation, retirement, removal or disqualification,
> > >> > >> > or until a successor is appointed.
> > >> > >> > =============================================================
> > >> > >> >
> > >> > >> >
> > >> > >> > -------
> > >> > >> >
> > >> > >> >
> > >> > >> > Summary:
> > >> > >> >
> > >> > >> > We propose creating a new top level project, Apache
> DataFusion, from
> > >> > >> > an existing sub project of Apache Arrow to facilitate
> additional
> > >> > >> > community and project growth.
> > >> > >> >
> > >> > >> > Abstract
> > >> > >> >
> > >> > >> > Apache Arrow DataFusion[1] is a very fast, extensible query
> engine
> > >> > >> > for building high-quality data-centric systems in Rust, using
> the
> > >> > >> > Apache Arrow in-memory format. DataFusion offers SQL and
> Dataframe
> > >> > >> > APIs, excellent performance, built-in support for CSV, Parquet,
> > >> JSON,
> > >> > >> > and Avro, extensive customization, and a great community.
> > >> > >> >
> > >> > >> > [1] https://arrow.apache.org/datafusion/
> > >> > >> >
> > >> > >> >
> > >> > >> > Proposal
> > >> > >> >
> > >> > >> > We propose creating a new top level ASF project, Apache
> DataFusion,
> > >> > >> > governed initially by a subset of the Apache Arrow project’s
> PMC and
> > >> > >> > committers. The project’s code is in five existing git
> repositories,
> > >> > >> > currently governed by Apache Arrow which would transfer to the
> new
> > >> top
> > >> > >> > level project.
> > >> > >> >
> > >> > >> > Background
> > >> > >> >
> > >> > >> > When DataFusion was initially donated to the Arrow project, it
> did
> > >> not
> > >> > >> > have a strong enough community to stand on its own. It has
> since
> > >> grown
> > >> > >> > significantly, and benefited immensely from being part of
> Arrow and
> > >> > >> > nurturing of the Apache Way, and now has a community strong
> enough
> > >> to
> > >> > >> > stand on its own and that would benefit from focused governance
> > >> > >> > attention.
> > >> > >> >
> > >> > >> > The community has discussed this idea publicly for more than 6
> > >> months
> > >> > >> > https://github.com/apache/arrow-datafusion/discussions/6475
> and
> > >> > >> > briefly on the Arrow PMC mailing list
> > >> > >> >
> https://lists.apache.org/thread/thv2jdm6640l6gm88hy8jhk5prjww0cs.
> > >> As
> > >> > >> > of the time of this writing both had exclusively positive
> reactions.
> > >> > >> >
> > >> > >> > Several current members of the Arrow PMC are both active
> > >> contributors
> > >> > >> > to DataFusion and understand and believe deeply in the Apache
> Way,
> > >> and
> > >> > >> > play active governance roles in the Arrow project as PMC
> members and
> > >> > >> > PMC chairs, guiding the community, and releasing software
> versions.
> > >> > >> > With this existing governance experience and structure, the
> new top
> > >> > >> > level project will be able to function well immediately and
> > >> > >> > independently.
> > >> > >> >
> > >> > >> > Overview of DataFusion
> > >> > >> >
> > >> > >> > Current Status
> > >> > >> >
> > >> > >> > Meritocracy
> > >> > >> >
> > >> > >> > DataFusion has been developed as part of Apache Arrow and thus
> has
> > >> > >> > been operating as a meritocracy. Many of the developers of
> > >> DataFusion
> > >> > >> > are Arrow PMC members or committers. The DataFusion project
> plans to
> > >> > >> > continue adding new PMC and committers as the project matures
> and
> > >> > >> > grows.
> > >> > >> >
> > >> > >> > Community
> > >> > >> >
> > >> > >> > The DataFusion development team seeks to foster the
> development and
> > >> > >> > user communities. We hope that becoming a separate project
> will help
> > >> > >> > both Arrow and DataFusion communities by being more focused.
> > >> Focused
> > >> > >> > governance will make it easier to grow the community of
> committers
> > >> and
> > >> > >> > PMC members and make the organization more clear to others.
> > >> > >> >
> > >> > >> > Alignment
> > >> > >> >
> > >> > >> > The ASF is a natural host for DataFusion given that it is
> already
> > >> the
> > >> > >> > home of Arrow, Parquet, and other related distributed system,
> > >> storage
> > >> > >> > and query execution systems.
> > >> > >> >
> > >> > >> > Project Leadership
> > >> > >> >
> > >> > >> > Proposed Initial PMC
> > >> > >> >
> > >> > >> > We propose the following people as the initial DataFusion PMC
> > >> members.
> > >> > >> > This is a subset of the existing Arrow PMC members who
> contribute to
> > >> > >> > DataFusion https://people.apache.org/phonebook.html?unix=arrow
> > >> > >> >
> > >> > >> > Andy Grove (agrove): Arrow PMC Chair
> > >> > >> > Andrew Lamb (alamb): Arrow PMC, past Arrow PMC Chair
> > >> > >> > Daniël Heres (dheres) Arrow PMC
> > >> > >> > Jie Wen (jakevin): Arrow PMC, Doris Committer
> > >> > >> > Kun Liu (liukun): Arrow PMC, IoTDB PMC, TSFile PMC
> > >> > >> > Liang-Chi Hsieh (viirya): Arrow PMC, Spark PMC
> > >> > >> > Qingping Hou: (houqp): Arrow PMC
> > >> > >> > Wes McKinney(wesm): Arrow PMC, ASF Member
> > >> > >> > Will Jones (wjones127): Arrow PMC
> > >> > >> >
> > >> > >> > We’d like to propose Andrew Lamb as the initial Chair, (and
> thus ASF
> > >> > >> > VP) for the DataFusion project.
> > >> > >> >
> > >> > >> > Affiliations
> > >> > >> >
> > >> > >> > Andy Grove (agrove): NVidia
> > >> > >> > Andrew Lamb (alamb): InfluxData
> > >> > >> > Daniël Heres (dheres): Coralogix
> > >> > >> > Jie Wen (jakevin): SelectDB
> > >> > >> > Kun Liu (liukun): Ebay
> > >> > >> > Liang-Chi Hsieh (viirya): Apple
> > >> > >> > Qingping Hou: (houqp): Scribd
> > >> > >> > Wes McKinney(wesm): Posit
> > >> > >> > Will Jones (wjones127): LanceDB
> > >> > >> >
> > >> > >> > Proposed Initial Committers
> > >> > >> >
> > >> > >> > In addition to the PMC, we propose the following people as the
> > >> initial
> > >> > >> > DataFusion committers. This is a subset of the existing Arrow
> > >> > >> > committers who contribute to DataFusion
> > >> > >> > https://people.apache.org/phonebook.html?unix=arrow
> > >> > >> >
> > >> > >> > akurmustafa Mustafa Akur (Synnada)
> > >> > >> > avantgardner Brent Gardner (Coralogix)
> > >> > >> > comphead Oleks V. (Unaffiliated)
> > >> > >> > jayzhan Jay Zhan (Unaffiliated)
> > >> > >> > jeffreyvo Jeffry Vo (Unaffiliated)
> > >> > >> > jiayuliu Liu Jiayu (Airbnb)
> > >> > >> > mete Metehan Yildirim (Synnada)
> > >> > >> > mingmwang Wang Mingming (Ebay)
> > >> > >> > mneumann Marco Neumann (InfluxData)
> > >> > >> > nju_yaho Zhong Yanghong (Ebay)
> > >> > >> > ozankabak Mehmet Ozan Kabak (Synnada)
> > >> > >> > paddyhoran Paddy Horan (Assured Allies)
> > >> > >> > rdettai Rémi Dettai (Cloudfuse)
> > >> > >> > sunchao Chao Sun (Apple)
> > >> > >> > thinkharderdev Daniel Harris (Coralogix)
> > >> > >> > tustvold Raphael Taylor-Davies (InfluxData)
> > >> > >> > wayne Ruihang Xia (Greptime)
> > >> > >> > xudong963 Xudong Wang (ByteDance)
> > >> > >> > yjshen Yijie Shen (Space and Time)
> > >> > >> > yangjiang Yang Jiang (ebay)
> > >> > >> >
> > >> > >> >
> > >> > >> > Risk Assessments
> > >> > >> >
> > >> > >> > Naming / Trademarks
> > >> > >> >
> > >> > >> > As a sub-project of Arrow, the DataFusion name has been used
> for
> > >> over
> > >> > >> > 4 years without any known issues. A podling name search did
> not turn
> > >> > >> > up any concerns and was approved:
> > >> > >> > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> > >> > >> >
> > >> > >> > Legal / IP Clearance
> > >> > >> >
> > >> > >> > All DataFusion code has either been donated to the Arrow
> project
> > >> with
> > >> > >> > appropriate IP clearance or has been developed directly under
> ASF
> > >> > >> > processes and procedures. Thus creating a new top level project
> > >> poses
> > >> > >> > no new Legal or IP risks.
> > >> > >> >
> > >> > >> > Code Extraction
> > >> > >> >
> > >> > >> > The relevant code is already in 5 separate repositories:
> > >> > >> > https://github.com/apache/arrow-datafusion/
> > >> > >> > https://github.com/apache/arrow-datafusion-python
> > >> > >> > https://github.com/apache/arrow-ballista
> > >> > >> > https://github.com/apache/arrow-ballista-python
> > >> > >> > https://github.com/apache/arrow-datafusion-comet
> > >> > >> >
> > >> > >> > We foresee no issues with code extraction and propose these
> > >> > >> > repositories be renamed to reflect top level projects
> > >> > >> >
> > >> > >> > Note: https://github.com/apache/arrow-rs, the Rust
> implementation
> > >> of
> > >> > >> > Arrow, would remain part of the Arrow project.
> > >> > >> >
> > >> > >> > Orphaned Products
> > >> > >> >
> > >> > >> > DataFusion is known to be used in many open source and
> commercial
> > >> > >> > projects
> > >> > >> >
> > >> > >> https://arrow.apache.org/datafusion/user-guide/
> > >> > introduction.html#known-users
> > >> > >> > ,
> > >> > >> > has had multiple commits daily for several years, and its
> adoption
> > >> and
> > >> > >> > number of contributors appears to be growing. We do not
> foresee the
> > >> > >> > project being orphaned in the next several years.
> > >> > >> >
> > >> > >> > Inexperience with Open Source
> > >> > >> >
> > >> > >> > The proposed PMC has extensive experience with Apache Arrow and
> > >> other
> > >> > >> > Apache projects, and includes PMC members, PMC chairs and an
> ASF
> > >> > >> > Member. The DataFusion PMC and more experienced committers will
> > >> > >> > continue to coach new community members who may be less
> familiar
> > >> with
> > >> > >> > the Apache Way.
> > >> > >> >
> > >> > >> > Homogeneous Developers
> > >> > >> >
> > >> > >> > The 9 proposed PMC members are from 9 different employers and
> the
> > >> > >> > proposed committers are similarly distributed across
> affiliations.
> > >> No
> > >> > >> > specific entity employs more than 3 total proposed developers.
> > >> > >> >
> > >> > >> > Reliance on Salaried Developers
> > >> > >> >
> > >> > >> > A substantial amount of work on DataFusion has been by salaried
> > >> > >> > developers, but it also has a long tradition of attracting
> > >> > >> > contributions from students and hobbyists and we plan no
> changes in
> > >> > >> > contribution structure.
> > >> > >> >
> > >> > >> > Relationships with Other Apache Products
> > >> > >> >
> > >> > >> > DataFusion will obviously have a strong relationship with the
> Arrow
> > >> > >> > project given the overlap in people. We don’t foresee close
> > >> > >> > collaboration with other projects at this time.
> > >> > >> >
> > >> > >> > Cryptography
> > >> > >> >
> > >> > >> > DataFusion does not directly support encryption and there are
> no
> > >> > >> > near-term plans to add support for encryption. Users who need
> this
> > >> > >> > functionality can use the extension APIs.
> > >> > >> >
> > >> > >> > Required Resources
> > >> > >> >
> > >> > >> > Mailing Lists
> > >> > >> >
> > >> > >> > - private@datafusion.apache.org for private PMC discussions
> (with
> > >> > >> > moderated subscriptions)
> > >> > >> > - dev@datafusion.apache.org
> > >> > >> > - commits@datafusion.apache.org
> > >> > >> > - user@datafusion.apache.org
> > >> > >> >
> > >> > >> > Version Control
> > >> > >> >
> > >> > >> > We propose to continue to use git for source control and
> github for
> > >> > >> > hosting and testing resources.
> > >> > >> >
> > >> > >> > We also need to rename the github repositories to reflect the
> new
> > >> top
> > >> > >> > level names:
> > >> > >> >
> > >> > >> > https://github.com/apache/arrow-datafusion/ →
> apache/datafusion
> > >> > >> > https://github.com/apache/arrow-datafusion-python →
> > >> > >> > apache/datafusion-python
> > >> > >> > https://github.com/apache/arrow-ballista →
> > >> apache/datafusion-ballista
> > >> > >> > https://github.com/apache/arrow-ballista-python →
> > >> > >> > apache/datafusion-ballista-python
> > >> > >> > https://github.com/apache/arrow-datafusion-comet →
> > >> > >> apache/datafusion-comet
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> > Issue Tracking
> > >> > >> >
> > >> > >> > DataFusion would continue to use github for its issue tracking
> and
> > >> > >> > communications
> > >> > >> >
> > >> > >> > Other Resources
> > >> > >> >
> > >> > >> > The existing repositories already make use of existing Apache
> > >> > >> > infrastructure, and we expect no change in the initial resource
> > >> usage.
> > >> > >> > As the project continues to grow, we expect continued
> infrastructure
> > >> > >> > demand growth.
> > >> > >> >
> > >> > >> >
> > >> > >> > FAQ: Has a sub project been promoted to a top level project
> before?
> > >> > >> >
> > >> > >> > Yes, and it appears to happen commonly. The Arrow project
> itself was
> > >> > >> > created as a top level project from work that started in Apache
> > >> Drill,
> > >> > >> > and there are many sub projects of Hadoop that spun out as
> their own
> > >> > >> > top level projects such as Mahout, Avro and HBase:
> > >> > >> >
> > >> > >> >
> > >> > >> https://news.apache.org/foundation/entry/the_apache_
> > >> > software_foundation_announces4
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> > Related material:
> > >> > >> > Name search request / research for DataFusion:
> > >> > >> > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> > >> > >> > Discussion about this proposal on the arrow mailing list:
> > >> > >> >
> https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> > >> > >> > Discussion about which repositories on the arrow mailing list:
> > >> > >> >
> https://lists.apache.org/thread/ob3n0d9ky0bgrryl3xn39w9k566bq00q
> > >> > >> > Discussion about initial PMC on the arrow mailing list:
> > >> > >> >
> https://lists.apache.org/thread/pymrzcdw4qdptvby85f69rg3pcckl15b
> > >> > >> > Discussion in github about creating a new DataFusion top level
> > >> > >> > project:
> > >> https://github.com/apache/arrow-datafusion/discussions/6475
> > >> > >> > Discussion about graduating on incubator list:
> > >> > >> >
> https://lists.apache.org/thread/r4n73pmms1lv0jbohyx1o1z13d615t99
> > >> > >> > Original Proposal for the Arrow project:
> > >> > >> >
> https://lists.apache.org/thread/x2qzdwglm8pkqp9gv03bbgw17khl7pq3
> > >> > >> >
> > >> > >>
> > >> >
> > >>
>
Re: [VOTE] Move Arrow DataFusion Subproject to new Top Level Apache Project
Posted by QP Hou <qp...@neuralink.com>.
+1 (binding)
exciting milestone :)
On Fri, Mar 1, 2024 at 9:49 AM David Li <li...@apache.org> wrote:
>
> +1
>
> On Fri, Mar 1, 2024, at 12:06, Jorge Cardoso Leitão wrote:
> > +1 - great work!!!
> >
> > On Fri, Mar 1, 2024 at 5:49 PM Micah Kornfield <em...@gmail.com>
> > wrote:
> >
> >> +1 (binding)
> >>
> >> On Friday, March 1, 2024, Uwe L. Korn <uw...@xhochy.com> wrote:
> >>
> >> > +1 (binding)
> >> >
> >> > On Fri, Mar 1, 2024, at 2:37 PM, Andy Grove wrote:
> >> > > +1 (binding)
> >> > >
> >> > > On Fri, Mar 1, 2024 at 6:20 AM Weston Pace <we...@gmail.com>
> >> > wrote:
> >> > >
> >> > >> +1 (binding)
> >> > >>
> >> > >> On Fri, Mar 1, 2024 at 3:33 AM Andrew Lamb <al...@influxdata.com>
> >> > wrote:
> >> > >>
> >> > >> > Hello,
> >> > >> >
> >> > >> > As we have discussed[1][2] I would like to vote on the proposal to
> >> > >> > create a new Apache Top Level Project for DataFusion. The text of
> >> the
> >> > >> > proposed resolution and background document is copy/pasted below
> >> > >> >
> >> > >> > If the community is in favor of this, we plan to submit the
> >> resolution
> >> > >> > to the ASF board for approval with the next Arrow report (for the
> >> > >> > April 2024 board meeting).
> >> > >> >
> >> > >> > The vote will be open for at least 7 days.
> >> > >> >
> >> > >> > [ ] +1 Accept this Proposal
> >> > >> > [ ] +0
> >> > >> > [ ] -1 Do not accept this proposal because...
> >> > >> >
> >> > >> > Andrew
> >> > >> >
> >> > >> > [1]
> >> https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> >> > >> > [2] https://github.com/apache/arrow-datafusion/discussions/6475
> >> > >> >
> >> > >> > ---------- Proposed Resolution ---------
> >> > >> >
> >> > >> > Resolution to Create the Apache DataFusion Project from the Apache
> >> > >> > Arrow DataFusion Sub Project
> >> > >> >
> >> > >> > =============================================================
> >> > >> >
> >> > >> > X. Establish the Apache DataFusion Project
> >> > >> >
> >> > >> > WHEREAS, the Board of Directors deems it to be in the best
> >> > >> > interests of the Foundation and consistent with the
> >> > >> > Foundation's purpose to establish a Project Management
> >> > >> > Committee charged with the creation and maintenance of
> >> > >> > open-source software related to an extensible query engine
> >> > >> > for distribution at no charge to the public.
> >> > >> >
> >> > >> > NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> >> > >> > Committee (PMC), to be known as the "Apache DataFusion Project",
> >> > >> > be and hereby is established pursuant to Bylaws of the
> >> > >> > Foundation; and be it further
> >> > >> >
> >> > >> > RESOLVED, that the Apache DataFusion Project be and hereby is
> >> > >> > responsible for the creation and maintenance of software
> >> > >> > related to an extensible query engine; and be it further
> >> > >> >
> >> > >> > RESOLVED, that the office of "Vice President, Apache DataFusion" be
> >> > >> > and hereby is created, the person holding such office to
> >> > >> > serve at the direction of the Board of Directors as the chair
> >> > >> > of the Apache DataFusion Project, and to have primary responsibility
> >> > >> > for management of the projects within the scope of
> >> > >> > responsibility of the Apache DataFusion Project; and be it further
> >> > >> >
> >> > >> > RESOLVED, that the persons listed immediately below be and
> >> > >> > hereby are appointed to serve as the initial members of the
> >> > >> > Apache DataFusion Project:
> >> > >> >
> >> > >> > * Andy Grove (agrove@apache.org)
> >> > >> > * Andrew Lamb (alamb@apache.org)
> >> > >> > * Daniël Heres (dheres@apache.org)
> >> > >> > * Jie Wen (jakevin@apache.org)
> >> > >> > * Kun Liu (liukun@apache.org)
> >> > >> > * Liang-Chi Hsieh (viirya@apache.org)
> >> > >> > * Qingping Hou: (houqp@apache.org)
> >> > >> > * Wes McKinney(wesm@apache.org)
> >> > >> > * Will Jones (wjones127@apache.org)
> >> > >> >
> >> > >> > RESOLVED, that the Apache DataFusion Project be and hereby
> >> > >> > is tasked with the migration and rationalization of the Apache
> >> > >> > Arrow DataFusion sub-project; and be it further
> >> > >> >
> >> > >> > RESOLVED, that all responsibilities pertaining to the Apache
> >> > >> > Arrow DataFusion sub-project encumbered upon the
> >> > >> > Apache Arrow Project are hereafter discharged.
> >> > >> >
> >> > >> > NOW, THEREFORE, BE IT FURTHER RESOLVED, that Andrew Lamb
> >> > >> > be appointed to the office of Vice President, Apache DataFusion, to
> >> > >> > serve in accordance with and subject to the direction of the
> >> > >> > Board of Directors and the Bylaws of the Foundation until
> >> > >> > death, resignation, retirement, removal or disqualification,
> >> > >> > or until a successor is appointed.
> >> > >> > =============================================================
> >> > >> >
> >> > >> >
> >> > >> > -------
> >> > >> >
> >> > >> >
> >> > >> > Summary:
> >> > >> >
> >> > >> > We propose creating a new top level project, Apache DataFusion, from
> >> > >> > an existing sub project of Apache Arrow to facilitate additional
> >> > >> > community and project growth.
> >> > >> >
> >> > >> > Abstract
> >> > >> >
> >> > >> > Apache Arrow DataFusion[1] is a very fast, extensible query engine
> >> > >> > for building high-quality data-centric systems in Rust, using the
> >> > >> > Apache Arrow in-memory format. DataFusion offers SQL and Dataframe
> >> > >> > APIs, excellent performance, built-in support for CSV, Parquet,
> >> JSON,
> >> > >> > and Avro, extensive customization, and a great community.
> >> > >> >
> >> > >> > [1] https://arrow.apache.org/datafusion/
> >> > >> >
> >> > >> >
> >> > >> > Proposal
> >> > >> >
> >> > >> > We propose creating a new top level ASF project, Apache DataFusion,
> >> > >> > governed initially by a subset of the Apache Arrow project’s PMC and
> >> > >> > committers. The project’s code is in five existing git repositories,
> >> > >> > currently governed by Apache Arrow which would transfer to the new
> >> top
> >> > >> > level project.
> >> > >> >
> >> > >> > Background
> >> > >> >
> >> > >> > When DataFusion was initially donated to the Arrow project, it did
> >> not
> >> > >> > have a strong enough community to stand on its own. It has since
> >> grown
> >> > >> > significantly, and benefited immensely from being part of Arrow and
> >> > >> > nurturing of the Apache Way, and now has a community strong enough
> >> to
> >> > >> > stand on its own and that would benefit from focused governance
> >> > >> > attention.
> >> > >> >
> >> > >> > The community has discussed this idea publicly for more than 6
> >> months
> >> > >> > https://github.com/apache/arrow-datafusion/discussions/6475 and
> >> > >> > briefly on the Arrow PMC mailing list
> >> > >> > https://lists.apache.org/thread/thv2jdm6640l6gm88hy8jhk5prjww0cs.
> >> As
> >> > >> > of the time of this writing both had exclusively positive reactions.
> >> > >> >
> >> > >> > Several current members of the Arrow PMC are both active
> >> contributors
> >> > >> > to DataFusion and understand and believe deeply in the Apache Way,
> >> and
> >> > >> > play active governance roles in the Arrow project as PMC members and
> >> > >> > PMC chairs, guiding the community, and releasing software versions.
> >> > >> > With this existing governance experience and structure, the new top
> >> > >> > level project will be able to function well immediately and
> >> > >> > independently.
> >> > >> >
> >> > >> > Overview of DataFusion
> >> > >> >
> >> > >> > Current Status
> >> > >> >
> >> > >> > Meritocracy
> >> > >> >
> >> > >> > DataFusion has been developed as part of Apache Arrow and thus has
> >> > >> > been operating as a meritocracy. Many of the developers of
> >> DataFusion
> >> > >> > are Arrow PMC members or committers. The DataFusion project plans to
> >> > >> > continue adding new PMC and committers as the project matures and
> >> > >> > grows.
> >> > >> >
> >> > >> > Community
> >> > >> >
> >> > >> > The DataFusion development team seeks to foster the development and
> >> > >> > user communities. We hope that becoming a separate project will help
> >> > >> > both Arrow and DataFusion communities by being more focused.
> >> Focused
> >> > >> > governance will make it easier to grow the community of committers
> >> and
> >> > >> > PMC members and make the organization more clear to others.
> >> > >> >
> >> > >> > Alignment
> >> > >> >
> >> > >> > The ASF is a natural host for DataFusion given that it is already
> >> the
> >> > >> > home of Arrow, Parquet, and other related distributed system,
> >> storage
> >> > >> > and query execution systems.
> >> > >> >
> >> > >> > Project Leadership
> >> > >> >
> >> > >> > Proposed Initial PMC
> >> > >> >
> >> > >> > We propose the following people as the initial DataFusion PMC
> >> members.
> >> > >> > This is a subset of the existing Arrow PMC members who contribute to
> >> > >> > DataFusion https://people.apache.org/phonebook.html?unix=arrow
> >> > >> >
> >> > >> > Andy Grove (agrove): Arrow PMC Chair
> >> > >> > Andrew Lamb (alamb): Arrow PMC, past Arrow PMC Chair
> >> > >> > Daniël Heres (dheres) Arrow PMC
> >> > >> > Jie Wen (jakevin): Arrow PMC, Doris Committer
> >> > >> > Kun Liu (liukun): Arrow PMC, IoTDB PMC, TSFile PMC
> >> > >> > Liang-Chi Hsieh (viirya): Arrow PMC, Spark PMC
> >> > >> > Qingping Hou: (houqp): Arrow PMC
> >> > >> > Wes McKinney(wesm): Arrow PMC, ASF Member
> >> > >> > Will Jones (wjones127): Arrow PMC
> >> > >> >
> >> > >> > We’d like to propose Andrew Lamb as the initial Chair, (and thus ASF
> >> > >> > VP) for the DataFusion project.
> >> > >> >
> >> > >> > Affiliations
> >> > >> >
> >> > >> > Andy Grove (agrove): NVidia
> >> > >> > Andrew Lamb (alamb): InfluxData
> >> > >> > Daniël Heres (dheres): Coralogix
> >> > >> > Jie Wen (jakevin): SelectDB
> >> > >> > Kun Liu (liukun): Ebay
> >> > >> > Liang-Chi Hsieh (viirya): Apple
> >> > >> > Qingping Hou: (houqp): Scribd
> >> > >> > Wes McKinney(wesm): Posit
> >> > >> > Will Jones (wjones127): LanceDB
> >> > >> >
> >> > >> > Proposed Initial Committers
> >> > >> >
> >> > >> > In addition to the PMC, we propose the following people as the
> >> initial
> >> > >> > DataFusion committers. This is a subset of the existing Arrow
> >> > >> > committers who contribute to DataFusion
> >> > >> > https://people.apache.org/phonebook.html?unix=arrow
> >> > >> >
> >> > >> > akurmustafa Mustafa Akur (Synnada)
> >> > >> > avantgardner Brent Gardner (Coralogix)
> >> > >> > comphead Oleks V. (Unaffiliated)
> >> > >> > jayzhan Jay Zhan (Unaffiliated)
> >> > >> > jeffreyvo Jeffry Vo (Unaffiliated)
> >> > >> > jiayuliu Liu Jiayu (Airbnb)
> >> > >> > mete Metehan Yildirim (Synnada)
> >> > >> > mingmwang Wang Mingming (Ebay)
> >> > >> > mneumann Marco Neumann (InfluxData)
> >> > >> > nju_yaho Zhong Yanghong (Ebay)
> >> > >> > ozankabak Mehmet Ozan Kabak (Synnada)
> >> > >> > paddyhoran Paddy Horan (Assured Allies)
> >> > >> > rdettai Rémi Dettai (Cloudfuse)
> >> > >> > sunchao Chao Sun (Apple)
> >> > >> > thinkharderdev Daniel Harris (Coralogix)
> >> > >> > tustvold Raphael Taylor-Davies (InfluxData)
> >> > >> > wayne Ruihang Xia (Greptime)
> >> > >> > xudong963 Xudong Wang (ByteDance)
> >> > >> > yjshen Yijie Shen (Space and Time)
> >> > >> > yangjiang Yang Jiang (ebay)
> >> > >> >
> >> > >> >
> >> > >> > Risk Assessments
> >> > >> >
> >> > >> > Naming / Trademarks
> >> > >> >
> >> > >> > As a sub-project of Arrow, the DataFusion name has been used for
> >> over
> >> > >> > 4 years without any known issues. A podling name search did not turn
> >> > >> > up any concerns and was approved:
> >> > >> > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> >> > >> >
> >> > >> > Legal / IP Clearance
> >> > >> >
> >> > >> > All DataFusion code has either been donated to the Arrow project
> >> with
> >> > >> > appropriate IP clearance or has been developed directly under ASF
> >> > >> > processes and procedures. Thus creating a new top level project
> >> poses
> >> > >> > no new Legal or IP risks.
> >> > >> >
> >> > >> > Code Extraction
> >> > >> >
> >> > >> > The relevant code is already in 5 separate repositories:
> >> > >> > https://github.com/apache/arrow-datafusion/
> >> > >> > https://github.com/apache/arrow-datafusion-python
> >> > >> > https://github.com/apache/arrow-ballista
> >> > >> > https://github.com/apache/arrow-ballista-python
> >> > >> > https://github.com/apache/arrow-datafusion-comet
> >> > >> >
> >> > >> > We foresee no issues with code extraction and propose these
> >> > >> > repositories be renamed to reflect top level projects
> >> > >> >
> >> > >> > Note: https://github.com/apache/arrow-rs, the Rust implementation
> >> of
> >> > >> > Arrow, would remain part of the Arrow project.
> >> > >> >
> >> > >> > Orphaned Products
> >> > >> >
> >> > >> > DataFusion is known to be used in many open source and commercial
> >> > >> > projects
> >> > >> >
> >> > >> https://arrow.apache.org/datafusion/user-guide/
> >> > introduction.html#known-users
> >> > >> > ,
> >> > >> > has had multiple commits daily for several years, and its adoption
> >> and
> >> > >> > number of contributors appears to be growing. We do not foresee the
> >> > >> > project being orphaned in the next several years.
> >> > >> >
> >> > >> > Inexperience with Open Source
> >> > >> >
> >> > >> > The proposed PMC has extensive experience with Apache Arrow and
> >> other
> >> > >> > Apache projects, and includes PMC members, PMC chairs and an ASF
> >> > >> > Member. The DataFusion PMC and more experienced committers will
> >> > >> > continue to coach new community members who may be less familiar
> >> with
> >> > >> > the Apache Way.
> >> > >> >
> >> > >> > Homogeneous Developers
> >> > >> >
> >> > >> > The 9 proposed PMC members are from 9 different employers and the
> >> > >> > proposed committers are similarly distributed across affiliations.
> >> No
> >> > >> > specific entity employs more than 3 total proposed developers.
> >> > >> >
> >> > >> > Reliance on Salaried Developers
> >> > >> >
> >> > >> > A substantial amount of work on DataFusion has been by salaried
> >> > >> > developers, but it also has a long tradition of attracting
> >> > >> > contributions from students and hobbyists and we plan no changes in
> >> > >> > contribution structure.
> >> > >> >
> >> > >> > Relationships with Other Apache Products
> >> > >> >
> >> > >> > DataFusion will obviously have a strong relationship with the Arrow
> >> > >> > project given the overlap in people. We don’t foresee close
> >> > >> > collaboration with other projects at this time.
> >> > >> >
> >> > >> > Cryptography
> >> > >> >
> >> > >> > DataFusion does not directly support encryption and there are no
> >> > >> > near-term plans to add support for encryption. Users who need this
> >> > >> > functionality can use the extension APIs.
> >> > >> >
> >> > >> > Required Resources
> >> > >> >
> >> > >> > Mailing Lists
> >> > >> >
> >> > >> > - private@datafusion.apache.org for private PMC discussions (with
> >> > >> > moderated subscriptions)
> >> > >> > - dev@datafusion.apache.org
> >> > >> > - commits@datafusion.apache.org
> >> > >> > - user@datafusion.apache.org
> >> > >> >
> >> > >> > Version Control
> >> > >> >
> >> > >> > We propose to continue to use git for source control and github for
> >> > >> > hosting and testing resources.
> >> > >> >
> >> > >> > We also need to rename the github repositories to reflect the new
> >> top
> >> > >> > level names:
> >> > >> >
> >> > >> > https://github.com/apache/arrow-datafusion/ → apache/datafusion
> >> > >> > https://github.com/apache/arrow-datafusion-python →
> >> > >> > apache/datafusion-python
> >> > >> > https://github.com/apache/arrow-ballista →
> >> apache/datafusion-ballista
> >> > >> > https://github.com/apache/arrow-ballista-python →
> >> > >> > apache/datafusion-ballista-python
> >> > >> > https://github.com/apache/arrow-datafusion-comet →
> >> > >> apache/datafusion-comet
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> > Issue Tracking
> >> > >> >
> >> > >> > DataFusion would continue to use github for its issue tracking and
> >> > >> > communications
> >> > >> >
> >> > >> > Other Resources
> >> > >> >
> >> > >> > The existing repositories already make use of existing Apache
> >> > >> > infrastructure, and we expect no change in the initial resource
> >> usage.
> >> > >> > As the project continues to grow, we expect continued infrastructure
> >> > >> > demand growth.
> >> > >> >
> >> > >> >
> >> > >> > FAQ: Has a sub project been promoted to a top level project before?
> >> > >> >
> >> > >> > Yes, and it appears to happen commonly. The Arrow project itself was
> >> > >> > created as a top level project from work that started in Apache
> >> Drill,
> >> > >> > and there are many sub projects of Hadoop that spun out as their own
> >> > >> > top level projects such as Mahout, Avro and HBase:
> >> > >> >
> >> > >> >
> >> > >> https://news.apache.org/foundation/entry/the_apache_
> >> > software_foundation_announces4
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> > Related material:
> >> > >> > Name search request / research for DataFusion:
> >> > >> > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> >> > >> > Discussion about this proposal on the arrow mailing list:
> >> > >> > https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> >> > >> > Discussion about which repositories on the arrow mailing list:
> >> > >> > https://lists.apache.org/thread/ob3n0d9ky0bgrryl3xn39w9k566bq00q
> >> > >> > Discussion about initial PMC on the arrow mailing list:
> >> > >> > https://lists.apache.org/thread/pymrzcdw4qdptvby85f69rg3pcckl15b
> >> > >> > Discussion in github about creating a new DataFusion top level
> >> > >> > project:
> >> https://github.com/apache/arrow-datafusion/discussions/6475
> >> > >> > Discussion about graduating on incubator list:
> >> > >> > https://lists.apache.org/thread/r4n73pmms1lv0jbohyx1o1z13d615t99
> >> > >> > Original Proposal for the Arrow project:
> >> > >> > https://lists.apache.org/thread/x2qzdwglm8pkqp9gv03bbgw17khl7pq3
> >> > >> >
> >> > >>
> >> >
> >>
Re: [VOTE] Move Arrow DataFusion Subproject to new Top Level Apache Project
Posted by David Li <li...@apache.org>.
+1
On Fri, Mar 1, 2024, at 12:06, Jorge Cardoso Leitão wrote:
> +1 - great work!!!
>
> On Fri, Mar 1, 2024 at 5:49 PM Micah Kornfield <em...@gmail.com>
> wrote:
>
>> +1 (binding)
>>
>> On Friday, March 1, 2024, Uwe L. Korn <uw...@xhochy.com> wrote:
>>
>> > +1 (binding)
>> >
>> > On Fri, Mar 1, 2024, at 2:37 PM, Andy Grove wrote:
>> > > +1 (binding)
>> > >
>> > > On Fri, Mar 1, 2024 at 6:20 AM Weston Pace <we...@gmail.com>
>> > wrote:
>> > >
>> > >> +1 (binding)
>> > >>
>> > >> On Fri, Mar 1, 2024 at 3:33 AM Andrew Lamb <al...@influxdata.com>
>> > wrote:
>> > >>
>> > >> > Hello,
>> > >> >
>> > >> > As we have discussed[1][2] I would like to vote on the proposal to
>> > >> > create a new Apache Top Level Project for DataFusion. The text of
>> the
>> > >> > proposed resolution and background document is copy/pasted below
>> > >> >
>> > >> > If the community is in favor of this, we plan to submit the
>> resolution
>> > >> > to the ASF board for approval with the next Arrow report (for the
>> > >> > April 2024 board meeting).
>> > >> >
>> > >> > The vote will be open for at least 7 days.
>> > >> >
>> > >> > [ ] +1 Accept this Proposal
>> > >> > [ ] +0
>> > >> > [ ] -1 Do not accept this proposal because...
>> > >> >
>> > >> > Andrew
>> > >> >
>> > >> > [1]
>> https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
>> > >> > [2] https://github.com/apache/arrow-datafusion/discussions/6475
>> > >> >
>> > >> > ---------- Proposed Resolution ---------
>> > >> >
>> > >> > Resolution to Create the Apache DataFusion Project from the Apache
>> > >> > Arrow DataFusion Sub Project
>> > >> >
>> > >> > =============================================================
>> > >> >
>> > >> > X. Establish the Apache DataFusion Project
>> > >> >
>> > >> > WHEREAS, the Board of Directors deems it to be in the best
>> > >> > interests of the Foundation and consistent with the
>> > >> > Foundation's purpose to establish a Project Management
>> > >> > Committee charged with the creation and maintenance of
>> > >> > open-source software related to an extensible query engine
>> > >> > for distribution at no charge to the public.
>> > >> >
>> > >> > NOW, THEREFORE, BE IT RESOLVED, that a Project Management
>> > >> > Committee (PMC), to be known as the "Apache DataFusion Project",
>> > >> > be and hereby is established pursuant to Bylaws of the
>> > >> > Foundation; and be it further
>> > >> >
>> > >> > RESOLVED, that the Apache DataFusion Project be and hereby is
>> > >> > responsible for the creation and maintenance of software
>> > >> > related to an extensible query engine; and be it further
>> > >> >
>> > >> > RESOLVED, that the office of "Vice President, Apache DataFusion" be
>> > >> > and hereby is created, the person holding such office to
>> > >> > serve at the direction of the Board of Directors as the chair
>> > >> > of the Apache DataFusion Project, and to have primary responsibility
>> > >> > for management of the projects within the scope of
>> > >> > responsibility of the Apache DataFusion Project; and be it further
>> > >> >
>> > >> > RESOLVED, that the persons listed immediately below be and
>> > >> > hereby are appointed to serve as the initial members of the
>> > >> > Apache DataFusion Project:
>> > >> >
>> > >> > * Andy Grove (agrove@apache.org)
>> > >> > * Andrew Lamb (alamb@apache.org)
>> > >> > * Daniël Heres (dheres@apache.org)
>> > >> > * Jie Wen (jakevin@apache.org)
>> > >> > * Kun Liu (liukun@apache.org)
>> > >> > * Liang-Chi Hsieh (viirya@apache.org)
>> > >> > * Qingping Hou: (houqp@apache.org)
>> > >> > * Wes McKinney(wesm@apache.org)
>> > >> > * Will Jones (wjones127@apache.org)
>> > >> >
>> > >> > RESOLVED, that the Apache DataFusion Project be and hereby
>> > >> > is tasked with the migration and rationalization of the Apache
>> > >> > Arrow DataFusion sub-project; and be it further
>> > >> >
>> > >> > RESOLVED, that all responsibilities pertaining to the Apache
>> > >> > Arrow DataFusion sub-project encumbered upon the
>> > >> > Apache Arrow Project are hereafter discharged.
>> > >> >
>> > >> > NOW, THEREFORE, BE IT FURTHER RESOLVED, that Andrew Lamb
>> > >> > be appointed to the office of Vice President, Apache DataFusion, to
>> > >> > serve in accordance with and subject to the direction of the
>> > >> > Board of Directors and the Bylaws of the Foundation until
>> > >> > death, resignation, retirement, removal or disqualification,
>> > >> > or until a successor is appointed.
>> > >> > =============================================================
>> > >> >
>> > >> >
>> > >> > -------
>> > >> >
>> > >> >
>> > >> > Summary:
>> > >> >
>> > >> > We propose creating a new top level project, Apache DataFusion, from
>> > >> > an existing sub project of Apache Arrow to facilitate additional
>> > >> > community and project growth.
>> > >> >
>> > >> > Abstract
>> > >> >
>> > >> > Apache Arrow DataFusion[1] is a very fast, extensible query engine
>> > >> > for building high-quality data-centric systems in Rust, using the
>> > >> > Apache Arrow in-memory format. DataFusion offers SQL and Dataframe
>> > >> > APIs, excellent performance, built-in support for CSV, Parquet,
>> JSON,
>> > >> > and Avro, extensive customization, and a great community.
>> > >> >
>> > >> > [1] https://arrow.apache.org/datafusion/
>> > >> >
>> > >> >
>> > >> > Proposal
>> > >> >
>> > >> > We propose creating a new top level ASF project, Apache DataFusion,
>> > >> > governed initially by a subset of the Apache Arrow project’s PMC and
>> > >> > committers. The project’s code is in five existing git repositories,
>> > >> > currently governed by Apache Arrow which would transfer to the new
>> top
>> > >> > level project.
>> > >> >
>> > >> > Background
>> > >> >
>> > >> > When DataFusion was initially donated to the Arrow project, it did
>> not
>> > >> > have a strong enough community to stand on its own. It has since
>> grown
>> > >> > significantly, and benefited immensely from being part of Arrow and
>> > >> > nurturing of the Apache Way, and now has a community strong enough
>> to
>> > >> > stand on its own and that would benefit from focused governance
>> > >> > attention.
>> > >> >
>> > >> > The community has discussed this idea publicly for more than 6
>> months
>> > >> > https://github.com/apache/arrow-datafusion/discussions/6475 and
>> > >> > briefly on the Arrow PMC mailing list
>> > >> > https://lists.apache.org/thread/thv2jdm6640l6gm88hy8jhk5prjww0cs.
>> As
>> > >> > of the time of this writing both had exclusively positive reactions.
>> > >> >
>> > >> > Several current members of the Arrow PMC are both active
>> contributors
>> > >> > to DataFusion and understand and believe deeply in the Apache Way,
>> and
>> > >> > play active governance roles in the Arrow project as PMC members and
>> > >> > PMC chairs, guiding the community, and releasing software versions.
>> > >> > With this existing governance experience and structure, the new top
>> > >> > level project will be able to function well immediately and
>> > >> > independently.
>> > >> >
>> > >> > Overview of DataFusion
>> > >> >
>> > >> > Current Status
>> > >> >
>> > >> > Meritocracy
>> > >> >
>> > >> > DataFusion has been developed as part of Apache Arrow and thus has
>> > >> > been operating as a meritocracy. Many of the developers of
>> DataFusion
>> > >> > are Arrow PMC members or committers. The DataFusion project plans to
>> > >> > continue adding new PMC and committers as the project matures and
>> > >> > grows.
>> > >> >
>> > >> > Community
>> > >> >
>> > >> > The DataFusion development team seeks to foster the development and
>> > >> > user communities. We hope that becoming a separate project will help
>> > >> > both Arrow and DataFusion communities by being more focused.
>> Focused
>> > >> > governance will make it easier to grow the community of committers
>> and
>> > >> > PMC members and make the organization more clear to others.
>> > >> >
>> > >> > Alignment
>> > >> >
>> > >> > The ASF is a natural host for DataFusion given that it is already
>> the
>> > >> > home of Arrow, Parquet, and other related distributed system,
>> storage
>> > >> > and query execution systems.
>> > >> >
>> > >> > Project Leadership
>> > >> >
>> > >> > Proposed Initial PMC
>> > >> >
>> > >> > We propose the following people as the initial DataFusion PMC
>> members.
>> > >> > This is a subset of the existing Arrow PMC members who contribute to
>> > >> > DataFusion https://people.apache.org/phonebook.html?unix=arrow
>> > >> >
>> > >> > Andy Grove (agrove): Arrow PMC Chair
>> > >> > Andrew Lamb (alamb): Arrow PMC, past Arrow PMC Chair
>> > >> > Daniël Heres (dheres) Arrow PMC
>> > >> > Jie Wen (jakevin): Arrow PMC, Doris Committer
>> > >> > Kun Liu (liukun): Arrow PMC, IoTDB PMC, TSFile PMC
>> > >> > Liang-Chi Hsieh (viirya): Arrow PMC, Spark PMC
>> > >> > Qingping Hou: (houqp): Arrow PMC
>> > >> > Wes McKinney(wesm): Arrow PMC, ASF Member
>> > >> > Will Jones (wjones127): Arrow PMC
>> > >> >
>> > >> > We’d like to propose Andrew Lamb as the initial Chair, (and thus ASF
>> > >> > VP) for the DataFusion project.
>> > >> >
>> > >> > Affiliations
>> > >> >
>> > >> > Andy Grove (agrove): NVidia
>> > >> > Andrew Lamb (alamb): InfluxData
>> > >> > Daniël Heres (dheres): Coralogix
>> > >> > Jie Wen (jakevin): SelectDB
>> > >> > Kun Liu (liukun): Ebay
>> > >> > Liang-Chi Hsieh (viirya): Apple
>> > >> > Qingping Hou: (houqp): Scribd
>> > >> > Wes McKinney(wesm): Posit
>> > >> > Will Jones (wjones127): LanceDB
>> > >> >
>> > >> > Proposed Initial Committers
>> > >> >
>> > >> > In addition to the PMC, we propose the following people as the
>> initial
>> > >> > DataFusion committers. This is a subset of the existing Arrow
>> > >> > committers who contribute to DataFusion
>> > >> > https://people.apache.org/phonebook.html?unix=arrow
>> > >> >
>> > >> > akurmustafa Mustafa Akur (Synnada)
>> > >> > avantgardner Brent Gardner (Coralogix)
>> > >> > comphead Oleks V. (Unaffiliated)
>> > >> > jayzhan Jay Zhan (Unaffiliated)
>> > >> > jeffreyvo Jeffry Vo (Unaffiliated)
>> > >> > jiayuliu Liu Jiayu (Airbnb)
>> > >> > mete Metehan Yildirim (Synnada)
>> > >> > mingmwang Wang Mingming (Ebay)
>> > >> > mneumann Marco Neumann (InfluxData)
>> > >> > nju_yaho Zhong Yanghong (Ebay)
>> > >> > ozankabak Mehmet Ozan Kabak (Synnada)
>> > >> > paddyhoran Paddy Horan (Assured Allies)
>> > >> > rdettai Rémi Dettai (Cloudfuse)
>> > >> > sunchao Chao Sun (Apple)
>> > >> > thinkharderdev Daniel Harris (Coralogix)
>> > >> > tustvold Raphael Taylor-Davies (InfluxData)
>> > >> > wayne Ruihang Xia (Greptime)
>> > >> > xudong963 Xudong Wang (ByteDance)
>> > >> > yjshen Yijie Shen (Space and Time)
>> > >> > yangjiang Yang Jiang (ebay)
>> > >> >
>> > >> >
>> > >> > Risk Assessments
>> > >> >
>> > >> > Naming / Trademarks
>> > >> >
>> > >> > As a sub-project of Arrow, the DataFusion name has been used for
>> over
>> > >> > 4 years without any known issues. A podling name search did not turn
>> > >> > up any concerns and was approved:
>> > >> > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
>> > >> >
>> > >> > Legal / IP Clearance
>> > >> >
>> > >> > All DataFusion code has either been donated to the Arrow project
>> with
>> > >> > appropriate IP clearance or has been developed directly under ASF
>> > >> > processes and procedures. Thus creating a new top level project
>> poses
>> > >> > no new Legal or IP risks.
>> > >> >
>> > >> > Code Extraction
>> > >> >
>> > >> > The relevant code is already in 5 separate repositories:
>> > >> > https://github.com/apache/arrow-datafusion/
>> > >> > https://github.com/apache/arrow-datafusion-python
>> > >> > https://github.com/apache/arrow-ballista
>> > >> > https://github.com/apache/arrow-ballista-python
>> > >> > https://github.com/apache/arrow-datafusion-comet
>> > >> >
>> > >> > We foresee no issues with code extraction and propose these
>> > >> > repositories be renamed to reflect top level projects
>> > >> >
>> > >> > Note: https://github.com/apache/arrow-rs, the Rust implementation
>> of
>> > >> > Arrow, would remain part of the Arrow project.
>> > >> >
>> > >> > Orphaned Products
>> > >> >
>> > >> > DataFusion is known to be used in many open source and commercial
>> > >> > projects
>> > >> >
>> > >> https://arrow.apache.org/datafusion/user-guide/
>> > introduction.html#known-users
>> > >> > ,
>> > >> > has had multiple commits daily for several years, and its adoption
>> and
>> > >> > number of contributors appears to be growing. We do not foresee the
>> > >> > project being orphaned in the next several years.
>> > >> >
>> > >> > Inexperience with Open Source
>> > >> >
>> > >> > The proposed PMC has extensive experience with Apache Arrow and
>> other
>> > >> > Apache projects, and includes PMC members, PMC chairs and an ASF
>> > >> > Member. The DataFusion PMC and more experienced committers will
>> > >> > continue to coach new community members who may be less familiar
>> with
>> > >> > the Apache Way.
>> > >> >
>> > >> > Homogeneous Developers
>> > >> >
>> > >> > The 9 proposed PMC members are from 9 different employers and the
>> > >> > proposed committers are similarly distributed across affiliations.
>> No
>> > >> > specific entity employs more than 3 total proposed developers.
>> > >> >
>> > >> > Reliance on Salaried Developers
>> > >> >
>> > >> > A substantial amount of work on DataFusion has been by salaried
>> > >> > developers, but it also has a long tradition of attracting
>> > >> > contributions from students and hobbyists and we plan no changes in
>> > >> > contribution structure.
>> > >> >
>> > >> > Relationships with Other Apache Products
>> > >> >
>> > >> > DataFusion will obviously have a strong relationship with the Arrow
>> > >> > project given the overlap in people. We don’t foresee close
>> > >> > collaboration with other projects at this time.
>> > >> >
>> > >> > Cryptography
>> > >> >
>> > >> > DataFusion does not directly support encryption and there are no
>> > >> > near-term plans to add support for encryption. Users who need this
>> > >> > functionality can use the extension APIs.
>> > >> >
>> > >> > Required Resources
>> > >> >
>> > >> > Mailing Lists
>> > >> >
>> > >> > - private@datafusion.apache.org for private PMC discussions (with
>> > >> > moderated subscriptions)
>> > >> > - dev@datafusion.apache.org
>> > >> > - commits@datafusion.apache.org
>> > >> > - user@datafusion.apache.org
>> > >> >
>> > >> > Version Control
>> > >> >
>> > >> > We propose to continue to use git for source control and github for
>> > >> > hosting and testing resources.
>> > >> >
>> > >> > We also need to rename the github repositories to reflect the new
>> top
>> > >> > level names:
>> > >> >
>> > >> > https://github.com/apache/arrow-datafusion/ → apache/datafusion
>> > >> > https://github.com/apache/arrow-datafusion-python →
>> > >> > apache/datafusion-python
>> > >> > https://github.com/apache/arrow-ballista →
>> apache/datafusion-ballista
>> > >> > https://github.com/apache/arrow-ballista-python →
>> > >> > apache/datafusion-ballista-python
>> > >> > https://github.com/apache/arrow-datafusion-comet →
>> > >> apache/datafusion-comet
>> > >> >
>> > >> >
>> > >> >
>> > >> > Issue Tracking
>> > >> >
>> > >> > DataFusion would continue to use github for its issue tracking and
>> > >> > communications
>> > >> >
>> > >> > Other Resources
>> > >> >
>> > >> > The existing repositories already make use of existing Apache
>> > >> > infrastructure, and we expect no change in the initial resource
>> usage.
>> > >> > As the project continues to grow, we expect continued infrastructure
>> > >> > demand growth.
>> > >> >
>> > >> >
>> > >> > FAQ: Has a sub project been promoted to a top level project before?
>> > >> >
>> > >> > Yes, and it appears to happen commonly. The Arrow project itself was
>> > >> > created as a top level project from work that started in Apache
>> Drill,
>> > >> > and there are many sub projects of Hadoop that spun out as their own
>> > >> > top level projects such as Mahout, Avro and HBase:
>> > >> >
>> > >> >
>> > >> https://news.apache.org/foundation/entry/the_apache_
>> > software_foundation_announces4
>> > >> >
>> > >> >
>> > >> >
>> > >> > Related material:
>> > >> > Name search request / research for DataFusion:
>> > >> > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
>> > >> > Discussion about this proposal on the arrow mailing list:
>> > >> > https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
>> > >> > Discussion about which repositories on the arrow mailing list:
>> > >> > https://lists.apache.org/thread/ob3n0d9ky0bgrryl3xn39w9k566bq00q
>> > >> > Discussion about initial PMC on the arrow mailing list:
>> > >> > https://lists.apache.org/thread/pymrzcdw4qdptvby85f69rg3pcckl15b
>> > >> > Discussion in github about creating a new DataFusion top level
>> > >> > project:
>> https://github.com/apache/arrow-datafusion/discussions/6475
>> > >> > Discussion about graduating on incubator list:
>> > >> > https://lists.apache.org/thread/r4n73pmms1lv0jbohyx1o1z13d615t99
>> > >> > Original Proposal for the Arrow project:
>> > >> > https://lists.apache.org/thread/x2qzdwglm8pkqp9gv03bbgw17khl7pq3
>> > >> >
>> > >>
>> >
>>
Re: [VOTE] Move Arrow DataFusion Subproject to new Top Level Apache Project
Posted by Jorge Cardoso Leitão <jo...@gmail.com>.
+1 - great work!!!
On Fri, Mar 1, 2024 at 5:49 PM Micah Kornfield <em...@gmail.com>
wrote:
> +1 (binding)
>
> On Friday, March 1, 2024, Uwe L. Korn <uw...@xhochy.com> wrote:
>
> > +1 (binding)
> >
> > On Fri, Mar 1, 2024, at 2:37 PM, Andy Grove wrote:
> > > +1 (binding)
> > >
> > > On Fri, Mar 1, 2024 at 6:20 AM Weston Pace <we...@gmail.com>
> > wrote:
> > >
> > >> +1 (binding)
> > >>
> > >> On Fri, Mar 1, 2024 at 3:33 AM Andrew Lamb <al...@influxdata.com>
> > wrote:
> > >>
> > >> > Hello,
> > >> >
> > >> > As we have discussed[1][2] I would like to vote on the proposal to
> > >> > create a new Apache Top Level Project for DataFusion. The text of
> the
> > >> > proposed resolution and background document is copy/pasted below
> > >> >
> > >> > If the community is in favor of this, we plan to submit the
> resolution
> > >> > to the ASF board for approval with the next Arrow report (for the
> > >> > April 2024 board meeting).
> > >> >
> > >> > The vote will be open for at least 7 days.
> > >> >
> > >> > [ ] +1 Accept this Proposal
> > >> > [ ] +0
> > >> > [ ] -1 Do not accept this proposal because...
> > >> >
> > >> > Andrew
> > >> >
> > >> > [1]
> https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> > >> > [2] https://github.com/apache/arrow-datafusion/discussions/6475
> > >> >
> > >> > ---------- Proposed Resolution ---------
> > >> >
> > >> > Resolution to Create the Apache DataFusion Project from the Apache
> > >> > Arrow DataFusion Sub Project
> > >> >
> > >> > =============================================================
> > >> >
> > >> > X. Establish the Apache DataFusion Project
> > >> >
> > >> > WHEREAS, the Board of Directors deems it to be in the best
> > >> > interests of the Foundation and consistent with the
> > >> > Foundation's purpose to establish a Project Management
> > >> > Committee charged with the creation and maintenance of
> > >> > open-source software related to an extensible query engine
> > >> > for distribution at no charge to the public.
> > >> >
> > >> > NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> > >> > Committee (PMC), to be known as the "Apache DataFusion Project",
> > >> > be and hereby is established pursuant to Bylaws of the
> > >> > Foundation; and be it further
> > >> >
> > >> > RESOLVED, that the Apache DataFusion Project be and hereby is
> > >> > responsible for the creation and maintenance of software
> > >> > related to an extensible query engine; and be it further
> > >> >
> > >> > RESOLVED, that the office of "Vice President, Apache DataFusion" be
> > >> > and hereby is created, the person holding such office to
> > >> > serve at the direction of the Board of Directors as the chair
> > >> > of the Apache DataFusion Project, and to have primary responsibility
> > >> > for management of the projects within the scope of
> > >> > responsibility of the Apache DataFusion Project; and be it further
> > >> >
> > >> > RESOLVED, that the persons listed immediately below be and
> > >> > hereby are appointed to serve as the initial members of the
> > >> > Apache DataFusion Project:
> > >> >
> > >> > * Andy Grove (agrove@apache.org)
> > >> > * Andrew Lamb (alamb@apache.org)
> > >> > * Daniël Heres (dheres@apache.org)
> > >> > * Jie Wen (jakevin@apache.org)
> > >> > * Kun Liu (liukun@apache.org)
> > >> > * Liang-Chi Hsieh (viirya@apache.org)
> > >> > * Qingping Hou: (houqp@apache.org)
> > >> > * Wes McKinney(wesm@apache.org)
> > >> > * Will Jones (wjones127@apache.org)
> > >> >
> > >> > RESOLVED, that the Apache DataFusion Project be and hereby
> > >> > is tasked with the migration and rationalization of the Apache
> > >> > Arrow DataFusion sub-project; and be it further
> > >> >
> > >> > RESOLVED, that all responsibilities pertaining to the Apache
> > >> > Arrow DataFusion sub-project encumbered upon the
> > >> > Apache Arrow Project are hereafter discharged.
> > >> >
> > >> > NOW, THEREFORE, BE IT FURTHER RESOLVED, that Andrew Lamb
> > >> > be appointed to the office of Vice President, Apache DataFusion, to
> > >> > serve in accordance with and subject to the direction of the
> > >> > Board of Directors and the Bylaws of the Foundation until
> > >> > death, resignation, retirement, removal or disqualification,
> > >> > or until a successor is appointed.
> > >> > =============================================================
> > >> >
> > >> >
> > >> > -------
> > >> >
> > >> >
> > >> > Summary:
> > >> >
> > >> > We propose creating a new top level project, Apache DataFusion, from
> > >> > an existing sub project of Apache Arrow to facilitate additional
> > >> > community and project growth.
> > >> >
> > >> > Abstract
> > >> >
> > >> > Apache Arrow DataFusion[1] is a very fast, extensible query engine
> > >> > for building high-quality data-centric systems in Rust, using the
> > >> > Apache Arrow in-memory format. DataFusion offers SQL and Dataframe
> > >> > APIs, excellent performance, built-in support for CSV, Parquet,
> JSON,
> > >> > and Avro, extensive customization, and a great community.
> > >> >
> > >> > [1] https://arrow.apache.org/datafusion/
> > >> >
> > >> >
> > >> > Proposal
> > >> >
> > >> > We propose creating a new top level ASF project, Apache DataFusion,
> > >> > governed initially by a subset of the Apache Arrow project’s PMC and
> > >> > committers. The project’s code is in five existing git repositories,
> > >> > currently governed by Apache Arrow which would transfer to the new
> top
> > >> > level project.
> > >> >
> > >> > Background
> > >> >
> > >> > When DataFusion was initially donated to the Arrow project, it did
> not
> > >> > have a strong enough community to stand on its own. It has since
> grown
> > >> > significantly, and benefited immensely from being part of Arrow and
> > >> > nurturing of the Apache Way, and now has a community strong enough
> to
> > >> > stand on its own and that would benefit from focused governance
> > >> > attention.
> > >> >
> > >> > The community has discussed this idea publicly for more than 6
> months
> > >> > https://github.com/apache/arrow-datafusion/discussions/6475 and
> > >> > briefly on the Arrow PMC mailing list
> > >> > https://lists.apache.org/thread/thv2jdm6640l6gm88hy8jhk5prjww0cs.
> As
> > >> > of the time of this writing both had exclusively positive reactions.
> > >> >
> > >> > Several current members of the Arrow PMC are both active
> contributors
> > >> > to DataFusion and understand and believe deeply in the Apache Way,
> and
> > >> > play active governance roles in the Arrow project as PMC members and
> > >> > PMC chairs, guiding the community, and releasing software versions.
> > >> > With this existing governance experience and structure, the new top
> > >> > level project will be able to function well immediately and
> > >> > independently.
> > >> >
> > >> > Overview of DataFusion
> > >> >
> > >> > Current Status
> > >> >
> > >> > Meritocracy
> > >> >
> > >> > DataFusion has been developed as part of Apache Arrow and thus has
> > >> > been operating as a meritocracy. Many of the developers of
> DataFusion
> > >> > are Arrow PMC members or committers. The DataFusion project plans to
> > >> > continue adding new PMC and committers as the project matures and
> > >> > grows.
> > >> >
> > >> > Community
> > >> >
> > >> > The DataFusion development team seeks to foster the development and
> > >> > user communities. We hope that becoming a separate project will help
> > >> > both Arrow and DataFusion communities by being more focused.
> Focused
> > >> > governance will make it easier to grow the community of committers
> and
> > >> > PMC members and make the organization more clear to others.
> > >> >
> > >> > Alignment
> > >> >
> > >> > The ASF is a natural host for DataFusion given that it is already
> the
> > >> > home of Arrow, Parquet, and other related distributed system,
> storage
> > >> > and query execution systems.
> > >> >
> > >> > Project Leadership
> > >> >
> > >> > Proposed Initial PMC
> > >> >
> > >> > We propose the following people as the initial DataFusion PMC
> members.
> > >> > This is a subset of the existing Arrow PMC members who contribute to
> > >> > DataFusion https://people.apache.org/phonebook.html?unix=arrow
> > >> >
> > >> > Andy Grove (agrove): Arrow PMC Chair
> > >> > Andrew Lamb (alamb): Arrow PMC, past Arrow PMC Chair
> > >> > Daniël Heres (dheres) Arrow PMC
> > >> > Jie Wen (jakevin): Arrow PMC, Doris Committer
> > >> > Kun Liu (liukun): Arrow PMC, IoTDB PMC, TSFile PMC
> > >> > Liang-Chi Hsieh (viirya): Arrow PMC, Spark PMC
> > >> > Qingping Hou: (houqp): Arrow PMC
> > >> > Wes McKinney(wesm): Arrow PMC, ASF Member
> > >> > Will Jones (wjones127): Arrow PMC
> > >> >
> > >> > We’d like to propose Andrew Lamb as the initial Chair, (and thus ASF
> > >> > VP) for the DataFusion project.
> > >> >
> > >> > Affiliations
> > >> >
> > >> > Andy Grove (agrove): NVidia
> > >> > Andrew Lamb (alamb): InfluxData
> > >> > Daniël Heres (dheres): Coralogix
> > >> > Jie Wen (jakevin): SelectDB
> > >> > Kun Liu (liukun): Ebay
> > >> > Liang-Chi Hsieh (viirya): Apple
> > >> > Qingping Hou: (houqp): Scribd
> > >> > Wes McKinney(wesm): Posit
> > >> > Will Jones (wjones127): LanceDB
> > >> >
> > >> > Proposed Initial Committers
> > >> >
> > >> > In addition to the PMC, we propose the following people as the
> initial
> > >> > DataFusion committers. This is a subset of the existing Arrow
> > >> > committers who contribute to DataFusion
> > >> > https://people.apache.org/phonebook.html?unix=arrow
> > >> >
> > >> > akurmustafa Mustafa Akur (Synnada)
> > >> > avantgardner Brent Gardner (Coralogix)
> > >> > comphead Oleks V. (Unaffiliated)
> > >> > jayzhan Jay Zhan (Unaffiliated)
> > >> > jeffreyvo Jeffry Vo (Unaffiliated)
> > >> > jiayuliu Liu Jiayu (Airbnb)
> > >> > mete Metehan Yildirim (Synnada)
> > >> > mingmwang Wang Mingming (Ebay)
> > >> > mneumann Marco Neumann (InfluxData)
> > >> > nju_yaho Zhong Yanghong (Ebay)
> > >> > ozankabak Mehmet Ozan Kabak (Synnada)
> > >> > paddyhoran Paddy Horan (Assured Allies)
> > >> > rdettai Rémi Dettai (Cloudfuse)
> > >> > sunchao Chao Sun (Apple)
> > >> > thinkharderdev Daniel Harris (Coralogix)
> > >> > tustvold Raphael Taylor-Davies (InfluxData)
> > >> > wayne Ruihang Xia (Greptime)
> > >> > xudong963 Xudong Wang (ByteDance)
> > >> > yjshen Yijie Shen (Space and Time)
> > >> > yangjiang Yang Jiang (ebay)
> > >> >
> > >> >
> > >> > Risk Assessments
> > >> >
> > >> > Naming / Trademarks
> > >> >
> > >> > As a sub-project of Arrow, the DataFusion name has been used for
> over
> > >> > 4 years without any known issues. A podling name search did not turn
> > >> > up any concerns and was approved:
> > >> > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> > >> >
> > >> > Legal / IP Clearance
> > >> >
> > >> > All DataFusion code has either been donated to the Arrow project
> with
> > >> > appropriate IP clearance or has been developed directly under ASF
> > >> > processes and procedures. Thus creating a new top level project
> poses
> > >> > no new Legal or IP risks.
> > >> >
> > >> > Code Extraction
> > >> >
> > >> > The relevant code is already in 5 separate repositories:
> > >> > https://github.com/apache/arrow-datafusion/
> > >> > https://github.com/apache/arrow-datafusion-python
> > >> > https://github.com/apache/arrow-ballista
> > >> > https://github.com/apache/arrow-ballista-python
> > >> > https://github.com/apache/arrow-datafusion-comet
> > >> >
> > >> > We foresee no issues with code extraction and propose these
> > >> > repositories be renamed to reflect top level projects
> > >> >
> > >> > Note: https://github.com/apache/arrow-rs, the Rust implementation
> of
> > >> > Arrow, would remain part of the Arrow project.
> > >> >
> > >> > Orphaned Products
> > >> >
> > >> > DataFusion is known to be used in many open source and commercial
> > >> > projects
> > >> >
> > >> https://arrow.apache.org/datafusion/user-guide/
> > introduction.html#known-users
> > >> > ,
> > >> > has had multiple commits daily for several years, and its adoption
> and
> > >> > number of contributors appears to be growing. We do not foresee the
> > >> > project being orphaned in the next several years.
> > >> >
> > >> > Inexperience with Open Source
> > >> >
> > >> > The proposed PMC has extensive experience with Apache Arrow and
> other
> > >> > Apache projects, and includes PMC members, PMC chairs and an ASF
> > >> > Member. The DataFusion PMC and more experienced committers will
> > >> > continue to coach new community members who may be less familiar
> with
> > >> > the Apache Way.
> > >> >
> > >> > Homogeneous Developers
> > >> >
> > >> > The 9 proposed PMC members are from 9 different employers and the
> > >> > proposed committers are similarly distributed across affiliations.
> No
> > >> > specific entity employs more than 3 total proposed developers.
> > >> >
> > >> > Reliance on Salaried Developers
> > >> >
> > >> > A substantial amount of work on DataFusion has been by salaried
> > >> > developers, but it also has a long tradition of attracting
> > >> > contributions from students and hobbyists and we plan no changes in
> > >> > contribution structure.
> > >> >
> > >> > Relationships with Other Apache Products
> > >> >
> > >> > DataFusion will obviously have a strong relationship with the Arrow
> > >> > project given the overlap in people. We don’t foresee close
> > >> > collaboration with other projects at this time.
> > >> >
> > >> > Cryptography
> > >> >
> > >> > DataFusion does not directly support encryption and there are no
> > >> > near-term plans to add support for encryption. Users who need this
> > >> > functionality can use the extension APIs.
> > >> >
> > >> > Required Resources
> > >> >
> > >> > Mailing Lists
> > >> >
> > >> > - private@datafusion.apache.org for private PMC discussions (with
> > >> > moderated subscriptions)
> > >> > - dev@datafusion.apache.org
> > >> > - commits@datafusion.apache.org
> > >> > - user@datafusion.apache.org
> > >> >
> > >> > Version Control
> > >> >
> > >> > We propose to continue to use git for source control and github for
> > >> > hosting and testing resources.
> > >> >
> > >> > We also need to rename the github repositories to reflect the new
> top
> > >> > level names:
> > >> >
> > >> > https://github.com/apache/arrow-datafusion/ → apache/datafusion
> > >> > https://github.com/apache/arrow-datafusion-python →
> > >> > apache/datafusion-python
> > >> > https://github.com/apache/arrow-ballista →
> apache/datafusion-ballista
> > >> > https://github.com/apache/arrow-ballista-python →
> > >> > apache/datafusion-ballista-python
> > >> > https://github.com/apache/arrow-datafusion-comet →
> > >> apache/datafusion-comet
> > >> >
> > >> >
> > >> >
> > >> > Issue Tracking
> > >> >
> > >> > DataFusion would continue to use github for its issue tracking and
> > >> > communications
> > >> >
> > >> > Other Resources
> > >> >
> > >> > The existing repositories already make use of existing Apache
> > >> > infrastructure, and we expect no change in the initial resource
> usage.
> > >> > As the project continues to grow, we expect continued infrastructure
> > >> > demand growth.
> > >> >
> > >> >
> > >> > FAQ: Has a sub project been promoted to a top level project before?
> > >> >
> > >> > Yes, and it appears to happen commonly. The Arrow project itself was
> > >> > created as a top level project from work that started in Apache
> Drill,
> > >> > and there are many sub projects of Hadoop that spun out as their own
> > >> > top level projects such as Mahout, Avro and HBase:
> > >> >
> > >> >
> > >> https://news.apache.org/foundation/entry/the_apache_
> > software_foundation_announces4
> > >> >
> > >> >
> > >> >
> > >> > Related material:
> > >> > Name search request / research for DataFusion:
> > >> > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> > >> > Discussion about this proposal on the arrow mailing list:
> > >> > https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> > >> > Discussion about which repositories on the arrow mailing list:
> > >> > https://lists.apache.org/thread/ob3n0d9ky0bgrryl3xn39w9k566bq00q
> > >> > Discussion about initial PMC on the arrow mailing list:
> > >> > https://lists.apache.org/thread/pymrzcdw4qdptvby85f69rg3pcckl15b
> > >> > Discussion in github about creating a new DataFusion top level
> > >> > project:
> https://github.com/apache/arrow-datafusion/discussions/6475
> > >> > Discussion about graduating on incubator list:
> > >> > https://lists.apache.org/thread/r4n73pmms1lv0jbohyx1o1z13d615t99
> > >> > Original Proposal for the Arrow project:
> > >> > https://lists.apache.org/thread/x2qzdwglm8pkqp9gv03bbgw17khl7pq3
> > >> >
> > >>
> >
>
Re: [VOTE] Move Arrow DataFusion Subproject to new Top Level Apache Project
Posted by Wes McKinney <we...@gmail.com>.
+1 (binding)
On Fri, Mar 1, 2024 at 09:49 Micah Kornfield <em...@gmail.com> wrote:
> +1 (binding)
>
> On Friday, March 1, 2024, Uwe L. Korn <uw...@xhochy.com> wrote:
>
> > +1 (binding)
> >
> > On Fri, Mar 1, 2024, at 2:37 PM, Andy Grove wrote:
> > > +1 (binding)
> > >
> > > On Fri, Mar 1, 2024 at 6:20 AM Weston Pace <we...@gmail.com>
> > wrote:
> > >
> > >> +1 (binding)
> > >>
> > >> On Fri, Mar 1, 2024 at 3:33 AM Andrew Lamb <al...@influxdata.com>
> > wrote:
> > >>
> > >> > Hello,
> > >> >
> > >> > As we have discussed[1][2] I would like to vote on the proposal to
> > >> > create a new Apache Top Level Project for DataFusion. The text of
> the
> > >> > proposed resolution and background document is copy/pasted below
> > >> >
> > >> > If the community is in favor of this, we plan to submit the
> resolution
> > >> > to the ASF board for approval with the next Arrow report (for the
> > >> > April 2024 board meeting).
> > >> >
> > >> > The vote will be open for at least 7 days.
> > >> >
> > >> > [ ] +1 Accept this Proposal
> > >> > [ ] +0
> > >> > [ ] -1 Do not accept this proposal because...
> > >> >
> > >> > Andrew
> > >> >
> > >> > [1]
> https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> > >> > [2] https://github.com/apache/arrow-datafusion/discussions/6475
> > >> >
> > >> > ---------- Proposed Resolution ---------
> > >> >
> > >> > Resolution to Create the Apache DataFusion Project from the Apache
> > >> > Arrow DataFusion Sub Project
> > >> >
> > >> > =============================================================
> > >> >
> > >> > X. Establish the Apache DataFusion Project
> > >> >
> > >> > WHEREAS, the Board of Directors deems it to be in the best
> > >> > interests of the Foundation and consistent with the
> > >> > Foundation's purpose to establish a Project Management
> > >> > Committee charged with the creation and maintenance of
> > >> > open-source software related to an extensible query engine
> > >> > for distribution at no charge to the public.
> > >> >
> > >> > NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> > >> > Committee (PMC), to be known as the "Apache DataFusion Project",
> > >> > be and hereby is established pursuant to Bylaws of the
> > >> > Foundation; and be it further
> > >> >
> > >> > RESOLVED, that the Apache DataFusion Project be and hereby is
> > >> > responsible for the creation and maintenance of software
> > >> > related to an extensible query engine; and be it further
> > >> >
> > >> > RESOLVED, that the office of "Vice President, Apache DataFusion" be
> > >> > and hereby is created, the person holding such office to
> > >> > serve at the direction of the Board of Directors as the chair
> > >> > of the Apache DataFusion Project, and to have primary responsibility
> > >> > for management of the projects within the scope of
> > >> > responsibility of the Apache DataFusion Project; and be it further
> > >> >
> > >> > RESOLVED, that the persons listed immediately below be and
> > >> > hereby are appointed to serve as the initial members of the
> > >> > Apache DataFusion Project:
> > >> >
> > >> > * Andy Grove (agrove@apache.org)
> > >> > * Andrew Lamb (alamb@apache.org)
> > >> > * Daniël Heres (dheres@apache.org)
> > >> > * Jie Wen (jakevin@apache.org)
> > >> > * Kun Liu (liukun@apache.org)
> > >> > * Liang-Chi Hsieh (viirya@apache.org)
> > >> > * Qingping Hou: (houqp@apache.org)
> > >> > * Wes McKinney(wesm@apache.org)
> > >> > * Will Jones (wjones127@apache.org)
> > >> >
> > >> > RESOLVED, that the Apache DataFusion Project be and hereby
> > >> > is tasked with the migration and rationalization of the Apache
> > >> > Arrow DataFusion sub-project; and be it further
> > >> >
> > >> > RESOLVED, that all responsibilities pertaining to the Apache
> > >> > Arrow DataFusion sub-project encumbered upon the
> > >> > Apache Arrow Project are hereafter discharged.
> > >> >
> > >> > NOW, THEREFORE, BE IT FURTHER RESOLVED, that Andrew Lamb
> > >> > be appointed to the office of Vice President, Apache DataFusion, to
> > >> > serve in accordance with and subject to the direction of the
> > >> > Board of Directors and the Bylaws of the Foundation until
> > >> > death, resignation, retirement, removal or disqualification,
> > >> > or until a successor is appointed.
> > >> > =============================================================
> > >> >
> > >> >
> > >> > -------
> > >> >
> > >> >
> > >> > Summary:
> > >> >
> > >> > We propose creating a new top level project, Apache DataFusion, from
> > >> > an existing sub project of Apache Arrow to facilitate additional
> > >> > community and project growth.
> > >> >
> > >> > Abstract
> > >> >
> > >> > Apache Arrow DataFusion[1] is a very fast, extensible query engine
> > >> > for building high-quality data-centric systems in Rust, using the
> > >> > Apache Arrow in-memory format. DataFusion offers SQL and Dataframe
> > >> > APIs, excellent performance, built-in support for CSV, Parquet,
> JSON,
> > >> > and Avro, extensive customization, and a great community.
> > >> >
> > >> > [1] https://arrow.apache.org/datafusion/
> > >> >
> > >> >
> > >> > Proposal
> > >> >
> > >> > We propose creating a new top level ASF project, Apache DataFusion,
> > >> > governed initially by a subset of the Apache Arrow project’s PMC and
> > >> > committers. The project’s code is in five existing git repositories,
> > >> > currently governed by Apache Arrow which would transfer to the new
> top
> > >> > level project.
> > >> >
> > >> > Background
> > >> >
> > >> > When DataFusion was initially donated to the Arrow project, it did
> not
> > >> > have a strong enough community to stand on its own. It has since
> grown
> > >> > significantly, and benefited immensely from being part of Arrow and
> > >> > nurturing of the Apache Way, and now has a community strong enough
> to
> > >> > stand on its own and that would benefit from focused governance
> > >> > attention.
> > >> >
> > >> > The community has discussed this idea publicly for more than 6
> months
> > >> > https://github.com/apache/arrow-datafusion/discussions/6475 and
> > >> > briefly on the Arrow PMC mailing list
> > >> > https://lists.apache.org/thread/thv2jdm6640l6gm88hy8jhk5prjww0cs.
> As
> > >> > of the time of this writing both had exclusively positive reactions.
> > >> >
> > >> > Several current members of the Arrow PMC are both active
> contributors
> > >> > to DataFusion and understand and believe deeply in the Apache Way,
> and
> > >> > play active governance roles in the Arrow project as PMC members and
> > >> > PMC chairs, guiding the community, and releasing software versions.
> > >> > With this existing governance experience and structure, the new top
> > >> > level project will be able to function well immediately and
> > >> > independently.
> > >> >
> > >> > Overview of DataFusion
> > >> >
> > >> > Current Status
> > >> >
> > >> > Meritocracy
> > >> >
> > >> > DataFusion has been developed as part of Apache Arrow and thus has
> > >> > been operating as a meritocracy. Many of the developers of
> DataFusion
> > >> > are Arrow PMC members or committers. The DataFusion project plans to
> > >> > continue adding new PMC and committers as the project matures and
> > >> > grows.
> > >> >
> > >> > Community
> > >> >
> > >> > The DataFusion development team seeks to foster the development and
> > >> > user communities. We hope that becoming a separate project will help
> > >> > both Arrow and DataFusion communities by being more focused.
> Focused
> > >> > governance will make it easier to grow the community of committers
> and
> > >> > PMC members and make the organization more clear to others.
> > >> >
> > >> > Alignment
> > >> >
> > >> > The ASF is a natural host for DataFusion given that it is already
> the
> > >> > home of Arrow, Parquet, and other related distributed system,
> storage
> > >> > and query execution systems.
> > >> >
> > >> > Project Leadership
> > >> >
> > >> > Proposed Initial PMC
> > >> >
> > >> > We propose the following people as the initial DataFusion PMC
> members.
> > >> > This is a subset of the existing Arrow PMC members who contribute to
> > >> > DataFusion https://people.apache.org/phonebook.html?unix=arrow
> > >> >
> > >> > Andy Grove (agrove): Arrow PMC Chair
> > >> > Andrew Lamb (alamb): Arrow PMC, past Arrow PMC Chair
> > >> > Daniël Heres (dheres) Arrow PMC
> > >> > Jie Wen (jakevin): Arrow PMC, Doris Committer
> > >> > Kun Liu (liukun): Arrow PMC, IoTDB PMC, TSFile PMC
> > >> > Liang-Chi Hsieh (viirya): Arrow PMC, Spark PMC
> > >> > Qingping Hou: (houqp): Arrow PMC
> > >> > Wes McKinney(wesm): Arrow PMC, ASF Member
> > >> > Will Jones (wjones127): Arrow PMC
> > >> >
> > >> > We’d like to propose Andrew Lamb as the initial Chair, (and thus ASF
> > >> > VP) for the DataFusion project.
> > >> >
> > >> > Affiliations
> > >> >
> > >> > Andy Grove (agrove): NVidia
> > >> > Andrew Lamb (alamb): InfluxData
> > >> > Daniël Heres (dheres): Coralogix
> > >> > Jie Wen (jakevin): SelectDB
> > >> > Kun Liu (liukun): Ebay
> > >> > Liang-Chi Hsieh (viirya): Apple
> > >> > Qingping Hou: (houqp): Scribd
> > >> > Wes McKinney(wesm): Posit
> > >> > Will Jones (wjones127): LanceDB
> > >> >
> > >> > Proposed Initial Committers
> > >> >
> > >> > In addition to the PMC, we propose the following people as the
> initial
> > >> > DataFusion committers. This is a subset of the existing Arrow
> > >> > committers who contribute to DataFusion
> > >> > https://people.apache.org/phonebook.html?unix=arrow
> > >> >
> > >> > akurmustafa Mustafa Akur (Synnada)
> > >> > avantgardner Brent Gardner (Coralogix)
> > >> > comphead Oleks V. (Unaffiliated)
> > >> > jayzhan Jay Zhan (Unaffiliated)
> > >> > jeffreyvo Jeffry Vo (Unaffiliated)
> > >> > jiayuliu Liu Jiayu (Airbnb)
> > >> > mete Metehan Yildirim (Synnada)
> > >> > mingmwang Wang Mingming (Ebay)
> > >> > mneumann Marco Neumann (InfluxData)
> > >> > nju_yaho Zhong Yanghong (Ebay)
> > >> > ozankabak Mehmet Ozan Kabak (Synnada)
> > >> > paddyhoran Paddy Horan (Assured Allies)
> > >> > rdettai Rémi Dettai (Cloudfuse)
> > >> > sunchao Chao Sun (Apple)
> > >> > thinkharderdev Daniel Harris (Coralogix)
> > >> > tustvold Raphael Taylor-Davies (InfluxData)
> > >> > wayne Ruihang Xia (Greptime)
> > >> > xudong963 Xudong Wang (ByteDance)
> > >> > yjshen Yijie Shen (Space and Time)
> > >> > yangjiang Yang Jiang (ebay)
> > >> >
> > >> >
> > >> > Risk Assessments
> > >> >
> > >> > Naming / Trademarks
> > >> >
> > >> > As a sub-project of Arrow, the DataFusion name has been used for
> over
> > >> > 4 years without any known issues. A podling name search did not turn
> > >> > up any concerns and was approved:
> > >> > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> > >> >
> > >> > Legal / IP Clearance
> > >> >
> > >> > All DataFusion code has either been donated to the Arrow project
> with
> > >> > appropriate IP clearance or has been developed directly under ASF
> > >> > processes and procedures. Thus creating a new top level project
> poses
> > >> > no new Legal or IP risks.
> > >> >
> > >> > Code Extraction
> > >> >
> > >> > The relevant code is already in 5 separate repositories:
> > >> > https://github.com/apache/arrow-datafusion/
> > >> > https://github.com/apache/arrow-datafusion-python
> > >> > https://github.com/apache/arrow-ballista
> > >> > https://github.com/apache/arrow-ballista-python
> > >> > https://github.com/apache/arrow-datafusion-comet
> > >> >
> > >> > We foresee no issues with code extraction and propose these
> > >> > repositories be renamed to reflect top level projects
> > >> >
> > >> > Note: https://github.com/apache/arrow-rs, the Rust implementation
> of
> > >> > Arrow, would remain part of the Arrow project.
> > >> >
> > >> > Orphaned Products
> > >> >
> > >> > DataFusion is known to be used in many open source and commercial
> > >> > projects
> > >> >
> > >> https://arrow.apache.org/datafusion/user-guide/
> > introduction.html#known-users
> > >> > ,
> > >> > has had multiple commits daily for several years, and its adoption
> and
> > >> > number of contributors appears to be growing. We do not foresee the
> > >> > project being orphaned in the next several years.
> > >> >
> > >> > Inexperience with Open Source
> > >> >
> > >> > The proposed PMC has extensive experience with Apache Arrow and
> other
> > >> > Apache projects, and includes PMC members, PMC chairs and an ASF
> > >> > Member. The DataFusion PMC and more experienced committers will
> > >> > continue to coach new community members who may be less familiar
> with
> > >> > the Apache Way.
> > >> >
> > >> > Homogeneous Developers
> > >> >
> > >> > The 9 proposed PMC members are from 9 different employers and the
> > >> > proposed committers are similarly distributed across affiliations.
> No
> > >> > specific entity employs more than 3 total proposed developers.
> > >> >
> > >> > Reliance on Salaried Developers
> > >> >
> > >> > A substantial amount of work on DataFusion has been by salaried
> > >> > developers, but it also has a long tradition of attracting
> > >> > contributions from students and hobbyists and we plan no changes in
> > >> > contribution structure.
> > >> >
> > >> > Relationships with Other Apache Products
> > >> >
> > >> > DataFusion will obviously have a strong relationship with the Arrow
> > >> > project given the overlap in people. We don’t foresee close
> > >> > collaboration with other projects at this time.
> > >> >
> > >> > Cryptography
> > >> >
> > >> > DataFusion does not directly support encryption and there are no
> > >> > near-term plans to add support for encryption. Users who need this
> > >> > functionality can use the extension APIs.
> > >> >
> > >> > Required Resources
> > >> >
> > >> > Mailing Lists
> > >> >
> > >> > - private@datafusion.apache.org for private PMC discussions (with
> > >> > moderated subscriptions)
> > >> > - dev@datafusion.apache.org
> > >> > - commits@datafusion.apache.org
> > >> > - user@datafusion.apache.org
> > >> >
> > >> > Version Control
> > >> >
> > >> > We propose to continue to use git for source control and github for
> > >> > hosting and testing resources.
> > >> >
> > >> > We also need to rename the github repositories to reflect the new
> top
> > >> > level names:
> > >> >
> > >> > https://github.com/apache/arrow-datafusion/ → apache/datafusion
> > >> > https://github.com/apache/arrow-datafusion-python →
> > >> > apache/datafusion-python
> > >> > https://github.com/apache/arrow-ballista →
> apache/datafusion-ballista
> > >> > https://github.com/apache/arrow-ballista-python →
> > >> > apache/datafusion-ballista-python
> > >> > https://github.com/apache/arrow-datafusion-comet →
> > >> apache/datafusion-comet
> > >> >
> > >> >
> > >> >
> > >> > Issue Tracking
> > >> >
> > >> > DataFusion would continue to use github for its issue tracking and
> > >> > communications
> > >> >
> > >> > Other Resources
> > >> >
> > >> > The existing repositories already make use of existing Apache
> > >> > infrastructure, and we expect no change in the initial resource
> usage.
> > >> > As the project continues to grow, we expect continued infrastructure
> > >> > demand growth.
> > >> >
> > >> >
> > >> > FAQ: Has a sub project been promoted to a top level project before?
> > >> >
> > >> > Yes, and it appears to happen commonly. The Arrow project itself was
> > >> > created as a top level project from work that started in Apache
> Drill,
> > >> > and there are many sub projects of Hadoop that spun out as their own
> > >> > top level projects such as Mahout, Avro and HBase:
> > >> >
> > >> >
> > >> https://news.apache.org/foundation/entry/the_apache_
> > software_foundation_announces4
> > >> >
> > >> >
> > >> >
> > >> > Related material:
> > >> > Name search request / research for DataFusion:
> > >> > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> > >> > Discussion about this proposal on the arrow mailing list:
> > >> > https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> > >> > Discussion about which repositories on the arrow mailing list:
> > >> > https://lists.apache.org/thread/ob3n0d9ky0bgrryl3xn39w9k566bq00q
> > >> > Discussion about initial PMC on the arrow mailing list:
> > >> > https://lists.apache.org/thread/pymrzcdw4qdptvby85f69rg3pcckl15b
> > >> > Discussion in github about creating a new DataFusion top level
> > >> > project:
> https://github.com/apache/arrow-datafusion/discussions/6475
> > >> > Discussion about graduating on incubator list:
> > >> > https://lists.apache.org/thread/r4n73pmms1lv0jbohyx1o1z13d615t99
> > >> > Original Proposal for the Arrow project:
> > >> > https://lists.apache.org/thread/x2qzdwglm8pkqp9gv03bbgw17khl7pq3
> > >> >
> > >>
> >
>
Re: [VOTE] Move Arrow DataFusion Subproject to new Top Level Apache Project
Posted by Micah Kornfield <em...@gmail.com>.
+1 (binding)
On Friday, March 1, 2024, Uwe L. Korn <uw...@xhochy.com> wrote:
> +1 (binding)
>
> On Fri, Mar 1, 2024, at 2:37 PM, Andy Grove wrote:
> > +1 (binding)
> >
> > On Fri, Mar 1, 2024 at 6:20 AM Weston Pace <we...@gmail.com>
> wrote:
> >
> >> +1 (binding)
> >>
> >> On Fri, Mar 1, 2024 at 3:33 AM Andrew Lamb <al...@influxdata.com>
> wrote:
> >>
> >> > Hello,
> >> >
> >> > As we have discussed[1][2] I would like to vote on the proposal to
> >> > create a new Apache Top Level Project for DataFusion. The text of the
> >> > proposed resolution and background document is copy/pasted below
> >> >
> >> > If the community is in favor of this, we plan to submit the resolution
> >> > to the ASF board for approval with the next Arrow report (for the
> >> > April 2024 board meeting).
> >> >
> >> > The vote will be open for at least 7 days.
> >> >
> >> > [ ] +1 Accept this Proposal
> >> > [ ] +0
> >> > [ ] -1 Do not accept this proposal because...
> >> >
> >> > Andrew
> >> >
> >> > [1] https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> >> > [2] https://github.com/apache/arrow-datafusion/discussions/6475
> >> >
> >> > ---------- Proposed Resolution ---------
> >> >
> >> > Resolution to Create the Apache DataFusion Project from the Apache
> >> > Arrow DataFusion Sub Project
> >> >
> >> > =============================================================
> >> >
> >> > X. Establish the Apache DataFusion Project
> >> >
> >> > WHEREAS, the Board of Directors deems it to be in the best
> >> > interests of the Foundation and consistent with the
> >> > Foundation's purpose to establish a Project Management
> >> > Committee charged with the creation and maintenance of
> >> > open-source software related to an extensible query engine
> >> > for distribution at no charge to the public.
> >> >
> >> > NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> >> > Committee (PMC), to be known as the "Apache DataFusion Project",
> >> > be and hereby is established pursuant to Bylaws of the
> >> > Foundation; and be it further
> >> >
> >> > RESOLVED, that the Apache DataFusion Project be and hereby is
> >> > responsible for the creation and maintenance of software
> >> > related to an extensible query engine; and be it further
> >> >
> >> > RESOLVED, that the office of "Vice President, Apache DataFusion" be
> >> > and hereby is created, the person holding such office to
> >> > serve at the direction of the Board of Directors as the chair
> >> > of the Apache DataFusion Project, and to have primary responsibility
> >> > for management of the projects within the scope of
> >> > responsibility of the Apache DataFusion Project; and be it further
> >> >
> >> > RESOLVED, that the persons listed immediately below be and
> >> > hereby are appointed to serve as the initial members of the
> >> > Apache DataFusion Project:
> >> >
> >> > * Andy Grove (agrove@apache.org)
> >> > * Andrew Lamb (alamb@apache.org)
> >> > * Daniël Heres (dheres@apache.org)
> >> > * Jie Wen (jakevin@apache.org)
> >> > * Kun Liu (liukun@apache.org)
> >> > * Liang-Chi Hsieh (viirya@apache.org)
> >> > * Qingping Hou: (houqp@apache.org)
> >> > * Wes McKinney(wesm@apache.org)
> >> > * Will Jones (wjones127@apache.org)
> >> >
> >> > RESOLVED, that the Apache DataFusion Project be and hereby
> >> > is tasked with the migration and rationalization of the Apache
> >> > Arrow DataFusion sub-project; and be it further
> >> >
> >> > RESOLVED, that all responsibilities pertaining to the Apache
> >> > Arrow DataFusion sub-project encumbered upon the
> >> > Apache Arrow Project are hereafter discharged.
> >> >
> >> > NOW, THEREFORE, BE IT FURTHER RESOLVED, that Andrew Lamb
> >> > be appointed to the office of Vice President, Apache DataFusion, to
> >> > serve in accordance with and subject to the direction of the
> >> > Board of Directors and the Bylaws of the Foundation until
> >> > death, resignation, retirement, removal or disqualification,
> >> > or until a successor is appointed.
> >> > =============================================================
> >> >
> >> >
> >> > -------
> >> >
> >> >
> >> > Summary:
> >> >
> >> > We propose creating a new top level project, Apache DataFusion, from
> >> > an existing sub project of Apache Arrow to facilitate additional
> >> > community and project growth.
> >> >
> >> > Abstract
> >> >
> >> > Apache Arrow DataFusion[1] is a very fast, extensible query engine
> >> > for building high-quality data-centric systems in Rust, using the
> >> > Apache Arrow in-memory format. DataFusion offers SQL and Dataframe
> >> > APIs, excellent performance, built-in support for CSV, Parquet, JSON,
> >> > and Avro, extensive customization, and a great community.
> >> >
> >> > [1] https://arrow.apache.org/datafusion/
> >> >
> >> >
> >> > Proposal
> >> >
> >> > We propose creating a new top level ASF project, Apache DataFusion,
> >> > governed initially by a subset of the Apache Arrow project’s PMC and
> >> > committers. The project’s code is in five existing git repositories,
> >> > currently governed by Apache Arrow which would transfer to the new top
> >> > level project.
> >> >
> >> > Background
> >> >
> >> > When DataFusion was initially donated to the Arrow project, it did not
> >> > have a strong enough community to stand on its own. It has since grown
> >> > significantly, and benefited immensely from being part of Arrow and
> >> > nurturing of the Apache Way, and now has a community strong enough to
> >> > stand on its own and that would benefit from focused governance
> >> > attention.
> >> >
> >> > The community has discussed this idea publicly for more than 6 months
> >> > https://github.com/apache/arrow-datafusion/discussions/6475 and
> >> > briefly on the Arrow PMC mailing list
> >> > https://lists.apache.org/thread/thv2jdm6640l6gm88hy8jhk5prjww0cs. As
> >> > of the time of this writing both had exclusively positive reactions.
> >> >
> >> > Several current members of the Arrow PMC are both active contributors
> >> > to DataFusion and understand and believe deeply in the Apache Way, and
> >> > play active governance roles in the Arrow project as PMC members and
> >> > PMC chairs, guiding the community, and releasing software versions.
> >> > With this existing governance experience and structure, the new top
> >> > level project will be able to function well immediately and
> >> > independently.
> >> >
> >> > Overview of DataFusion
> >> >
> >> > Current Status
> >> >
> >> > Meritocracy
> >> >
> >> > DataFusion has been developed as part of Apache Arrow and thus has
> >> > been operating as a meritocracy. Many of the developers of DataFusion
> >> > are Arrow PMC members or committers. The DataFusion project plans to
> >> > continue adding new PMC and committers as the project matures and
> >> > grows.
> >> >
> >> > Community
> >> >
> >> > The DataFusion development team seeks to foster the development and
> >> > user communities. We hope that becoming a separate project will help
> >> > both Arrow and DataFusion communities by being more focused. Focused
> >> > governance will make it easier to grow the community of committers and
> >> > PMC members and make the organization more clear to others.
> >> >
> >> > Alignment
> >> >
> >> > The ASF is a natural host for DataFusion given that it is already the
> >> > home of Arrow, Parquet, and other related distributed system, storage
> >> > and query execution systems.
> >> >
> >> > Project Leadership
> >> >
> >> > Proposed Initial PMC
> >> >
> >> > We propose the following people as the initial DataFusion PMC members.
> >> > This is a subset of the existing Arrow PMC members who contribute to
> >> > DataFusion https://people.apache.org/phonebook.html?unix=arrow
> >> >
> >> > Andy Grove (agrove): Arrow PMC Chair
> >> > Andrew Lamb (alamb): Arrow PMC, past Arrow PMC Chair
> >> > Daniël Heres (dheres) Arrow PMC
> >> > Jie Wen (jakevin): Arrow PMC, Doris Committer
> >> > Kun Liu (liukun): Arrow PMC, IoTDB PMC, TSFile PMC
> >> > Liang-Chi Hsieh (viirya): Arrow PMC, Spark PMC
> >> > Qingping Hou: (houqp): Arrow PMC
> >> > Wes McKinney(wesm): Arrow PMC, ASF Member
> >> > Will Jones (wjones127): Arrow PMC
> >> >
> >> > We’d like to propose Andrew Lamb as the initial Chair, (and thus ASF
> >> > VP) for the DataFusion project.
> >> >
> >> > Affiliations
> >> >
> >> > Andy Grove (agrove): NVidia
> >> > Andrew Lamb (alamb): InfluxData
> >> > Daniël Heres (dheres): Coralogix
> >> > Jie Wen (jakevin): SelectDB
> >> > Kun Liu (liukun): Ebay
> >> > Liang-Chi Hsieh (viirya): Apple
> >> > Qingping Hou: (houqp): Scribd
> >> > Wes McKinney(wesm): Posit
> >> > Will Jones (wjones127): LanceDB
> >> >
> >> > Proposed Initial Committers
> >> >
> >> > In addition to the PMC, we propose the following people as the initial
> >> > DataFusion committers. This is a subset of the existing Arrow
> >> > committers who contribute to DataFusion
> >> > https://people.apache.org/phonebook.html?unix=arrow
> >> >
> >> > akurmustafa Mustafa Akur (Synnada)
> >> > avantgardner Brent Gardner (Coralogix)
> >> > comphead Oleks V. (Unaffiliated)
> >> > jayzhan Jay Zhan (Unaffiliated)
> >> > jeffreyvo Jeffry Vo (Unaffiliated)
> >> > jiayuliu Liu Jiayu (Airbnb)
> >> > mete Metehan Yildirim (Synnada)
> >> > mingmwang Wang Mingming (Ebay)
> >> > mneumann Marco Neumann (InfluxData)
> >> > nju_yaho Zhong Yanghong (Ebay)
> >> > ozankabak Mehmet Ozan Kabak (Synnada)
> >> > paddyhoran Paddy Horan (Assured Allies)
> >> > rdettai Rémi Dettai (Cloudfuse)
> >> > sunchao Chao Sun (Apple)
> >> > thinkharderdev Daniel Harris (Coralogix)
> >> > tustvold Raphael Taylor-Davies (InfluxData)
> >> > wayne Ruihang Xia (Greptime)
> >> > xudong963 Xudong Wang (ByteDance)
> >> > yjshen Yijie Shen (Space and Time)
> >> > yangjiang Yang Jiang (ebay)
> >> >
> >> >
> >> > Risk Assessments
> >> >
> >> > Naming / Trademarks
> >> >
> >> > As a sub-project of Arrow, the DataFusion name has been used for over
> >> > 4 years without any known issues. A podling name search did not turn
> >> > up any concerns and was approved:
> >> > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> >> >
> >> > Legal / IP Clearance
> >> >
> >> > All DataFusion code has either been donated to the Arrow project with
> >> > appropriate IP clearance or has been developed directly under ASF
> >> > processes and procedures. Thus creating a new top level project poses
> >> > no new Legal or IP risks.
> >> >
> >> > Code Extraction
> >> >
> >> > The relevant code is already in 5 separate repositories:
> >> > https://github.com/apache/arrow-datafusion/
> >> > https://github.com/apache/arrow-datafusion-python
> >> > https://github.com/apache/arrow-ballista
> >> > https://github.com/apache/arrow-ballista-python
> >> > https://github.com/apache/arrow-datafusion-comet
> >> >
> >> > We foresee no issues with code extraction and propose these
> >> > repositories be renamed to reflect top level projects
> >> >
> >> > Note: https://github.com/apache/arrow-rs, the Rust implementation of
> >> > Arrow, would remain part of the Arrow project.
> >> >
> >> > Orphaned Products
> >> >
> >> > DataFusion is known to be used in many open source and commercial
> >> > projects
> >> >
> >> https://arrow.apache.org/datafusion/user-guide/
> introduction.html#known-users
> >> > ,
> >> > has had multiple commits daily for several years, and its adoption and
> >> > number of contributors appears to be growing. We do not foresee the
> >> > project being orphaned in the next several years.
> >> >
> >> > Inexperience with Open Source
> >> >
> >> > The proposed PMC has extensive experience with Apache Arrow and other
> >> > Apache projects, and includes PMC members, PMC chairs and an ASF
> >> > Member. The DataFusion PMC and more experienced committers will
> >> > continue to coach new community members who may be less familiar with
> >> > the Apache Way.
> >> >
> >> > Homogeneous Developers
> >> >
> >> > The 9 proposed PMC members are from 9 different employers and the
> >> > proposed committers are similarly distributed across affiliations. No
> >> > specific entity employs more than 3 total proposed developers.
> >> >
> >> > Reliance on Salaried Developers
> >> >
> >> > A substantial amount of work on DataFusion has been by salaried
> >> > developers, but it also has a long tradition of attracting
> >> > contributions from students and hobbyists and we plan no changes in
> >> > contribution structure.
> >> >
> >> > Relationships with Other Apache Products
> >> >
> >> > DataFusion will obviously have a strong relationship with the Arrow
> >> > project given the overlap in people. We don’t foresee close
> >> > collaboration with other projects at this time.
> >> >
> >> > Cryptography
> >> >
> >> > DataFusion does not directly support encryption and there are no
> >> > near-term plans to add support for encryption. Users who need this
> >> > functionality can use the extension APIs.
> >> >
> >> > Required Resources
> >> >
> >> > Mailing Lists
> >> >
> >> > - private@datafusion.apache.org for private PMC discussions (with
> >> > moderated subscriptions)
> >> > - dev@datafusion.apache.org
> >> > - commits@datafusion.apache.org
> >> > - user@datafusion.apache.org
> >> >
> >> > Version Control
> >> >
> >> > We propose to continue to use git for source control and github for
> >> > hosting and testing resources.
> >> >
> >> > We also need to rename the github repositories to reflect the new top
> >> > level names:
> >> >
> >> > https://github.com/apache/arrow-datafusion/ → apache/datafusion
> >> > https://github.com/apache/arrow-datafusion-python →
> >> > apache/datafusion-python
> >> > https://github.com/apache/arrow-ballista → apache/datafusion-ballista
> >> > https://github.com/apache/arrow-ballista-python →
> >> > apache/datafusion-ballista-python
> >> > https://github.com/apache/arrow-datafusion-comet →
> >> apache/datafusion-comet
> >> >
> >> >
> >> >
> >> > Issue Tracking
> >> >
> >> > DataFusion would continue to use github for its issue tracking and
> >> > communications
> >> >
> >> > Other Resources
> >> >
> >> > The existing repositories already make use of existing Apache
> >> > infrastructure, and we expect no change in the initial resource usage.
> >> > As the project continues to grow, we expect continued infrastructure
> >> > demand growth.
> >> >
> >> >
> >> > FAQ: Has a sub project been promoted to a top level project before?
> >> >
> >> > Yes, and it appears to happen commonly. The Arrow project itself was
> >> > created as a top level project from work that started in Apache Drill,
> >> > and there are many sub projects of Hadoop that spun out as their own
> >> > top level projects such as Mahout, Avro and HBase:
> >> >
> >> >
> >> https://news.apache.org/foundation/entry/the_apache_
> software_foundation_announces4
> >> >
> >> >
> >> >
> >> > Related material:
> >> > Name search request / research for DataFusion:
> >> > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> >> > Discussion about this proposal on the arrow mailing list:
> >> > https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> >> > Discussion about which repositories on the arrow mailing list:
> >> > https://lists.apache.org/thread/ob3n0d9ky0bgrryl3xn39w9k566bq00q
> >> > Discussion about initial PMC on the arrow mailing list:
> >> > https://lists.apache.org/thread/pymrzcdw4qdptvby85f69rg3pcckl15b
> >> > Discussion in github about creating a new DataFusion top level
> >> > project: https://github.com/apache/arrow-datafusion/discussions/6475
> >> > Discussion about graduating on incubator list:
> >> > https://lists.apache.org/thread/r4n73pmms1lv0jbohyx1o1z13d615t99
> >> > Original Proposal for the Arrow project:
> >> > https://lists.apache.org/thread/x2qzdwglm8pkqp9gv03bbgw17khl7pq3
> >> >
> >>
>
Re: [VOTE] Move Arrow DataFusion Subproject to new Top Level Apache Project
Posted by "Uwe L. Korn" <uw...@xhochy.com>.
+1 (binding)
On Fri, Mar 1, 2024, at 2:37 PM, Andy Grove wrote:
> +1 (binding)
>
> On Fri, Mar 1, 2024 at 6:20 AM Weston Pace <we...@gmail.com> wrote:
>
>> +1 (binding)
>>
>> On Fri, Mar 1, 2024 at 3:33 AM Andrew Lamb <al...@influxdata.com> wrote:
>>
>> > Hello,
>> >
>> > As we have discussed[1][2] I would like to vote on the proposal to
>> > create a new Apache Top Level Project for DataFusion. The text of the
>> > proposed resolution and background document is copy/pasted below
>> >
>> > If the community is in favor of this, we plan to submit the resolution
>> > to the ASF board for approval with the next Arrow report (for the
>> > April 2024 board meeting).
>> >
>> > The vote will be open for at least 7 days.
>> >
>> > [ ] +1 Accept this Proposal
>> > [ ] +0
>> > [ ] -1 Do not accept this proposal because...
>> >
>> > Andrew
>> >
>> > [1] https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
>> > [2] https://github.com/apache/arrow-datafusion/discussions/6475
>> >
>> > ---------- Proposed Resolution ---------
>> >
>> > Resolution to Create the Apache DataFusion Project from the Apache
>> > Arrow DataFusion Sub Project
>> >
>> > =============================================================
>> >
>> > X. Establish the Apache DataFusion Project
>> >
>> > WHEREAS, the Board of Directors deems it to be in the best
>> > interests of the Foundation and consistent with the
>> > Foundation's purpose to establish a Project Management
>> > Committee charged with the creation and maintenance of
>> > open-source software related to an extensible query engine
>> > for distribution at no charge to the public.
>> >
>> > NOW, THEREFORE, BE IT RESOLVED, that a Project Management
>> > Committee (PMC), to be known as the "Apache DataFusion Project",
>> > be and hereby is established pursuant to Bylaws of the
>> > Foundation; and be it further
>> >
>> > RESOLVED, that the Apache DataFusion Project be and hereby is
>> > responsible for the creation and maintenance of software
>> > related to an extensible query engine; and be it further
>> >
>> > RESOLVED, that the office of "Vice President, Apache DataFusion" be
>> > and hereby is created, the person holding such office to
>> > serve at the direction of the Board of Directors as the chair
>> > of the Apache DataFusion Project, and to have primary responsibility
>> > for management of the projects within the scope of
>> > responsibility of the Apache DataFusion Project; and be it further
>> >
>> > RESOLVED, that the persons listed immediately below be and
>> > hereby are appointed to serve as the initial members of the
>> > Apache DataFusion Project:
>> >
>> > * Andy Grove (agrove@apache.org)
>> > * Andrew Lamb (alamb@apache.org)
>> > * Daniël Heres (dheres@apache.org)
>> > * Jie Wen (jakevin@apache.org)
>> > * Kun Liu (liukun@apache.org)
>> > * Liang-Chi Hsieh (viirya@apache.org)
>> > * Qingping Hou: (houqp@apache.org)
>> > * Wes McKinney(wesm@apache.org)
>> > * Will Jones (wjones127@apache.org)
>> >
>> > RESOLVED, that the Apache DataFusion Project be and hereby
>> > is tasked with the migration and rationalization of the Apache
>> > Arrow DataFusion sub-project; and be it further
>> >
>> > RESOLVED, that all responsibilities pertaining to the Apache
>> > Arrow DataFusion sub-project encumbered upon the
>> > Apache Arrow Project are hereafter discharged.
>> >
>> > NOW, THEREFORE, BE IT FURTHER RESOLVED, that Andrew Lamb
>> > be appointed to the office of Vice President, Apache DataFusion, to
>> > serve in accordance with and subject to the direction of the
>> > Board of Directors and the Bylaws of the Foundation until
>> > death, resignation, retirement, removal or disqualification,
>> > or until a successor is appointed.
>> > =============================================================
>> >
>> >
>> > -------
>> >
>> >
>> > Summary:
>> >
>> > We propose creating a new top level project, Apache DataFusion, from
>> > an existing sub project of Apache Arrow to facilitate additional
>> > community and project growth.
>> >
>> > Abstract
>> >
>> > Apache Arrow DataFusion[1] is a very fast, extensible query engine
>> > for building high-quality data-centric systems in Rust, using the
>> > Apache Arrow in-memory format. DataFusion offers SQL and Dataframe
>> > APIs, excellent performance, built-in support for CSV, Parquet, JSON,
>> > and Avro, extensive customization, and a great community.
>> >
>> > [1] https://arrow.apache.org/datafusion/
>> >
>> >
>> > Proposal
>> >
>> > We propose creating a new top level ASF project, Apache DataFusion,
>> > governed initially by a subset of the Apache Arrow project’s PMC and
>> > committers. The project’s code is in five existing git repositories,
>> > currently governed by Apache Arrow which would transfer to the new top
>> > level project.
>> >
>> > Background
>> >
>> > When DataFusion was initially donated to the Arrow project, it did not
>> > have a strong enough community to stand on its own. It has since grown
>> > significantly, and benefited immensely from being part of Arrow and
>> > nurturing of the Apache Way, and now has a community strong enough to
>> > stand on its own and that would benefit from focused governance
>> > attention.
>> >
>> > The community has discussed this idea publicly for more than 6 months
>> > https://github.com/apache/arrow-datafusion/discussions/6475 and
>> > briefly on the Arrow PMC mailing list
>> > https://lists.apache.org/thread/thv2jdm6640l6gm88hy8jhk5prjww0cs. As
>> > of the time of this writing both had exclusively positive reactions.
>> >
>> > Several current members of the Arrow PMC are both active contributors
>> > to DataFusion and understand and believe deeply in the Apache Way, and
>> > play active governance roles in the Arrow project as PMC members and
>> > PMC chairs, guiding the community, and releasing software versions.
>> > With this existing governance experience and structure, the new top
>> > level project will be able to function well immediately and
>> > independently.
>> >
>> > Overview of DataFusion
>> >
>> > Current Status
>> >
>> > Meritocracy
>> >
>> > DataFusion has been developed as part of Apache Arrow and thus has
>> > been operating as a meritocracy. Many of the developers of DataFusion
>> > are Arrow PMC members or committers. The DataFusion project plans to
>> > continue adding new PMC and committers as the project matures and
>> > grows.
>> >
>> > Community
>> >
>> > The DataFusion development team seeks to foster the development and
>> > user communities. We hope that becoming a separate project will help
>> > both Arrow and DataFusion communities by being more focused. Focused
>> > governance will make it easier to grow the community of committers and
>> > PMC members and make the organization more clear to others.
>> >
>> > Alignment
>> >
>> > The ASF is a natural host for DataFusion given that it is already the
>> > home of Arrow, Parquet, and other related distributed system, storage
>> > and query execution systems.
>> >
>> > Project Leadership
>> >
>> > Proposed Initial PMC
>> >
>> > We propose the following people as the initial DataFusion PMC members.
>> > This is a subset of the existing Arrow PMC members who contribute to
>> > DataFusion https://people.apache.org/phonebook.html?unix=arrow
>> >
>> > Andy Grove (agrove): Arrow PMC Chair
>> > Andrew Lamb (alamb): Arrow PMC, past Arrow PMC Chair
>> > Daniël Heres (dheres) Arrow PMC
>> > Jie Wen (jakevin): Arrow PMC, Doris Committer
>> > Kun Liu (liukun): Arrow PMC, IoTDB PMC, TSFile PMC
>> > Liang-Chi Hsieh (viirya): Arrow PMC, Spark PMC
>> > Qingping Hou: (houqp): Arrow PMC
>> > Wes McKinney(wesm): Arrow PMC, ASF Member
>> > Will Jones (wjones127): Arrow PMC
>> >
>> > We’d like to propose Andrew Lamb as the initial Chair, (and thus ASF
>> > VP) for the DataFusion project.
>> >
>> > Affiliations
>> >
>> > Andy Grove (agrove): NVidia
>> > Andrew Lamb (alamb): InfluxData
>> > Daniël Heres (dheres): Coralogix
>> > Jie Wen (jakevin): SelectDB
>> > Kun Liu (liukun): Ebay
>> > Liang-Chi Hsieh (viirya): Apple
>> > Qingping Hou: (houqp): Scribd
>> > Wes McKinney(wesm): Posit
>> > Will Jones (wjones127): LanceDB
>> >
>> > Proposed Initial Committers
>> >
>> > In addition to the PMC, we propose the following people as the initial
>> > DataFusion committers. This is a subset of the existing Arrow
>> > committers who contribute to DataFusion
>> > https://people.apache.org/phonebook.html?unix=arrow
>> >
>> > akurmustafa Mustafa Akur (Synnada)
>> > avantgardner Brent Gardner (Coralogix)
>> > comphead Oleks V. (Unaffiliated)
>> > jayzhan Jay Zhan (Unaffiliated)
>> > jeffreyvo Jeffry Vo (Unaffiliated)
>> > jiayuliu Liu Jiayu (Airbnb)
>> > mete Metehan Yildirim (Synnada)
>> > mingmwang Wang Mingming (Ebay)
>> > mneumann Marco Neumann (InfluxData)
>> > nju_yaho Zhong Yanghong (Ebay)
>> > ozankabak Mehmet Ozan Kabak (Synnada)
>> > paddyhoran Paddy Horan (Assured Allies)
>> > rdettai Rémi Dettai (Cloudfuse)
>> > sunchao Chao Sun (Apple)
>> > thinkharderdev Daniel Harris (Coralogix)
>> > tustvold Raphael Taylor-Davies (InfluxData)
>> > wayne Ruihang Xia (Greptime)
>> > xudong963 Xudong Wang (ByteDance)
>> > yjshen Yijie Shen (Space and Time)
>> > yangjiang Yang Jiang (ebay)
>> >
>> >
>> > Risk Assessments
>> >
>> > Naming / Trademarks
>> >
>> > As a sub-project of Arrow, the DataFusion name has been used for over
>> > 4 years without any known issues. A podling name search did not turn
>> > up any concerns and was approved:
>> > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
>> >
>> > Legal / IP Clearance
>> >
>> > All DataFusion code has either been donated to the Arrow project with
>> > appropriate IP clearance or has been developed directly under ASF
>> > processes and procedures. Thus creating a new top level project poses
>> > no new Legal or IP risks.
>> >
>> > Code Extraction
>> >
>> > The relevant code is already in 5 separate repositories:
>> > https://github.com/apache/arrow-datafusion/
>> > https://github.com/apache/arrow-datafusion-python
>> > https://github.com/apache/arrow-ballista
>> > https://github.com/apache/arrow-ballista-python
>> > https://github.com/apache/arrow-datafusion-comet
>> >
>> > We foresee no issues with code extraction and propose these
>> > repositories be renamed to reflect top level projects
>> >
>> > Note: https://github.com/apache/arrow-rs, the Rust implementation of
>> > Arrow, would remain part of the Arrow project.
>> >
>> > Orphaned Products
>> >
>> > DataFusion is known to be used in many open source and commercial
>> > projects
>> >
>> https://arrow.apache.org/datafusion/user-guide/introduction.html#known-users
>> > ,
>> > has had multiple commits daily for several years, and its adoption and
>> > number of contributors appears to be growing. We do not foresee the
>> > project being orphaned in the next several years.
>> >
>> > Inexperience with Open Source
>> >
>> > The proposed PMC has extensive experience with Apache Arrow and other
>> > Apache projects, and includes PMC members, PMC chairs and an ASF
>> > Member. The DataFusion PMC and more experienced committers will
>> > continue to coach new community members who may be less familiar with
>> > the Apache Way.
>> >
>> > Homogeneous Developers
>> >
>> > The 9 proposed PMC members are from 9 different employers and the
>> > proposed committers are similarly distributed across affiliations. No
>> > specific entity employs more than 3 total proposed developers.
>> >
>> > Reliance on Salaried Developers
>> >
>> > A substantial amount of work on DataFusion has been by salaried
>> > developers, but it also has a long tradition of attracting
>> > contributions from students and hobbyists and we plan no changes in
>> > contribution structure.
>> >
>> > Relationships with Other Apache Products
>> >
>> > DataFusion will obviously have a strong relationship with the Arrow
>> > project given the overlap in people. We don’t foresee close
>> > collaboration with other projects at this time.
>> >
>> > Cryptography
>> >
>> > DataFusion does not directly support encryption and there are no
>> > near-term plans to add support for encryption. Users who need this
>> > functionality can use the extension APIs.
>> >
>> > Required Resources
>> >
>> > Mailing Lists
>> >
>> > - private@datafusion.apache.org for private PMC discussions (with
>> > moderated subscriptions)
>> > - dev@datafusion.apache.org
>> > - commits@datafusion.apache.org
>> > - user@datafusion.apache.org
>> >
>> > Version Control
>> >
>> > We propose to continue to use git for source control and github for
>> > hosting and testing resources.
>> >
>> > We also need to rename the github repositories to reflect the new top
>> > level names:
>> >
>> > https://github.com/apache/arrow-datafusion/ → apache/datafusion
>> > https://github.com/apache/arrow-datafusion-python →
>> > apache/datafusion-python
>> > https://github.com/apache/arrow-ballista → apache/datafusion-ballista
>> > https://github.com/apache/arrow-ballista-python →
>> > apache/datafusion-ballista-python
>> > https://github.com/apache/arrow-datafusion-comet →
>> apache/datafusion-comet
>> >
>> >
>> >
>> > Issue Tracking
>> >
>> > DataFusion would continue to use github for its issue tracking and
>> > communications
>> >
>> > Other Resources
>> >
>> > The existing repositories already make use of existing Apache
>> > infrastructure, and we expect no change in the initial resource usage.
>> > As the project continues to grow, we expect continued infrastructure
>> > demand growth.
>> >
>> >
>> > FAQ: Has a sub project been promoted to a top level project before?
>> >
>> > Yes, and it appears to happen commonly. The Arrow project itself was
>> > created as a top level project from work that started in Apache Drill,
>> > and there are many sub projects of Hadoop that spun out as their own
>> > top level projects such as Mahout, Avro and HBase:
>> >
>> >
>> https://news.apache.org/foundation/entry/the_apache_software_foundation_announces4
>> >
>> >
>> >
>> > Related material:
>> > Name search request / research for DataFusion:
>> > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
>> > Discussion about this proposal on the arrow mailing list:
>> > https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
>> > Discussion about which repositories on the arrow mailing list:
>> > https://lists.apache.org/thread/ob3n0d9ky0bgrryl3xn39w9k566bq00q
>> > Discussion about initial PMC on the arrow mailing list:
>> > https://lists.apache.org/thread/pymrzcdw4qdptvby85f69rg3pcckl15b
>> > Discussion in github about creating a new DataFusion top level
>> > project: https://github.com/apache/arrow-datafusion/discussions/6475
>> > Discussion about graduating on incubator list:
>> > https://lists.apache.org/thread/r4n73pmms1lv0jbohyx1o1z13d615t99
>> > Original Proposal for the Arrow project:
>> > https://lists.apache.org/thread/x2qzdwglm8pkqp9gv03bbgw17khl7pq3
>> >
>>
Re: [VOTE] Move Arrow DataFusion Subproject to new Top Level Apache Project
Posted by Andy Grove <an...@gmail.com>.
+1 (binding)
On Fri, Mar 1, 2024 at 6:20 AM Weston Pace <we...@gmail.com> wrote:
> +1 (binding)
>
> On Fri, Mar 1, 2024 at 3:33 AM Andrew Lamb <al...@influxdata.com> wrote:
>
> > Hello,
> >
> > As we have discussed[1][2] I would like to vote on the proposal to
> > create a new Apache Top Level Project for DataFusion. The text of the
> > proposed resolution and background document is copy/pasted below
> >
> > If the community is in favor of this, we plan to submit the resolution
> > to the ASF board for approval with the next Arrow report (for the
> > April 2024 board meeting).
> >
> > The vote will be open for at least 7 days.
> >
> > [ ] +1 Accept this Proposal
> > [ ] +0
> > [ ] -1 Do not accept this proposal because...
> >
> > Andrew
> >
> > [1] https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> > [2] https://github.com/apache/arrow-datafusion/discussions/6475
> >
> > ---------- Proposed Resolution ---------
> >
> > Resolution to Create the Apache DataFusion Project from the Apache
> > Arrow DataFusion Sub Project
> >
> > =============================================================
> >
> > X. Establish the Apache DataFusion Project
> >
> > WHEREAS, the Board of Directors deems it to be in the best
> > interests of the Foundation and consistent with the
> > Foundation's purpose to establish a Project Management
> > Committee charged with the creation and maintenance of
> > open-source software related to an extensible query engine
> > for distribution at no charge to the public.
> >
> > NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> > Committee (PMC), to be known as the "Apache DataFusion Project",
> > be and hereby is established pursuant to Bylaws of the
> > Foundation; and be it further
> >
> > RESOLVED, that the Apache DataFusion Project be and hereby is
> > responsible for the creation and maintenance of software
> > related to an extensible query engine; and be it further
> >
> > RESOLVED, that the office of "Vice President, Apache DataFusion" be
> > and hereby is created, the person holding such office to
> > serve at the direction of the Board of Directors as the chair
> > of the Apache DataFusion Project, and to have primary responsibility
> > for management of the projects within the scope of
> > responsibility of the Apache DataFusion Project; and be it further
> >
> > RESOLVED, that the persons listed immediately below be and
> > hereby are appointed to serve as the initial members of the
> > Apache DataFusion Project:
> >
> > * Andy Grove (agrove@apache.org)
> > * Andrew Lamb (alamb@apache.org)
> > * Daniël Heres (dheres@apache.org)
> > * Jie Wen (jakevin@apache.org)
> > * Kun Liu (liukun@apache.org)
> > * Liang-Chi Hsieh (viirya@apache.org)
> > * Qingping Hou: (houqp@apache.org)
> > * Wes McKinney(wesm@apache.org)
> > * Will Jones (wjones127@apache.org)
> >
> > RESOLVED, that the Apache DataFusion Project be and hereby
> > is tasked with the migration and rationalization of the Apache
> > Arrow DataFusion sub-project; and be it further
> >
> > RESOLVED, that all responsibilities pertaining to the Apache
> > Arrow DataFusion sub-project encumbered upon the
> > Apache Arrow Project are hereafter discharged.
> >
> > NOW, THEREFORE, BE IT FURTHER RESOLVED, that Andrew Lamb
> > be appointed to the office of Vice President, Apache DataFusion, to
> > serve in accordance with and subject to the direction of the
> > Board of Directors and the Bylaws of the Foundation until
> > death, resignation, retirement, removal or disqualification,
> > or until a successor is appointed.
> > =============================================================
> >
> >
> > -------
> >
> >
> > Summary:
> >
> > We propose creating a new top level project, Apache DataFusion, from
> > an existing sub project of Apache Arrow to facilitate additional
> > community and project growth.
> >
> > Abstract
> >
> > Apache Arrow DataFusion[1] is a very fast, extensible query engine
> > for building high-quality data-centric systems in Rust, using the
> > Apache Arrow in-memory format. DataFusion offers SQL and Dataframe
> > APIs, excellent performance, built-in support for CSV, Parquet, JSON,
> > and Avro, extensive customization, and a great community.
> >
> > [1] https://arrow.apache.org/datafusion/
> >
> >
> > Proposal
> >
> > We propose creating a new top level ASF project, Apache DataFusion,
> > governed initially by a subset of the Apache Arrow project’s PMC and
> > committers. The project’s code is in five existing git repositories,
> > currently governed by Apache Arrow which would transfer to the new top
> > level project.
> >
> > Background
> >
> > When DataFusion was initially donated to the Arrow project, it did not
> > have a strong enough community to stand on its own. It has since grown
> > significantly, and benefited immensely from being part of Arrow and
> > nurturing of the Apache Way, and now has a community strong enough to
> > stand on its own and that would benefit from focused governance
> > attention.
> >
> > The community has discussed this idea publicly for more than 6 months
> > https://github.com/apache/arrow-datafusion/discussions/6475 and
> > briefly on the Arrow PMC mailing list
> > https://lists.apache.org/thread/thv2jdm6640l6gm88hy8jhk5prjww0cs. As
> > of the time of this writing both had exclusively positive reactions.
> >
> > Several current members of the Arrow PMC are both active contributors
> > to DataFusion and understand and believe deeply in the Apache Way, and
> > play active governance roles in the Arrow project as PMC members and
> > PMC chairs, guiding the community, and releasing software versions.
> > With this existing governance experience and structure, the new top
> > level project will be able to function well immediately and
> > independently.
> >
> > Overview of DataFusion
> >
> > Current Status
> >
> > Meritocracy
> >
> > DataFusion has been developed as part of Apache Arrow and thus has
> > been operating as a meritocracy. Many of the developers of DataFusion
> > are Arrow PMC members or committers. The DataFusion project plans to
> > continue adding new PMC and committers as the project matures and
> > grows.
> >
> > Community
> >
> > The DataFusion development team seeks to foster the development and
> > user communities. We hope that becoming a separate project will help
> > both Arrow and DataFusion communities by being more focused. Focused
> > governance will make it easier to grow the community of committers and
> > PMC members and make the organization more clear to others.
> >
> > Alignment
> >
> > The ASF is a natural host for DataFusion given that it is already the
> > home of Arrow, Parquet, and other related distributed system, storage
> > and query execution systems.
> >
> > Project Leadership
> >
> > Proposed Initial PMC
> >
> > We propose the following people as the initial DataFusion PMC members.
> > This is a subset of the existing Arrow PMC members who contribute to
> > DataFusion https://people.apache.org/phonebook.html?unix=arrow
> >
> > Andy Grove (agrove): Arrow PMC Chair
> > Andrew Lamb (alamb): Arrow PMC, past Arrow PMC Chair
> > Daniël Heres (dheres) Arrow PMC
> > Jie Wen (jakevin): Arrow PMC, Doris Committer
> > Kun Liu (liukun): Arrow PMC, IoTDB PMC, TSFile PMC
> > Liang-Chi Hsieh (viirya): Arrow PMC, Spark PMC
> > Qingping Hou: (houqp): Arrow PMC
> > Wes McKinney(wesm): Arrow PMC, ASF Member
> > Will Jones (wjones127): Arrow PMC
> >
> > We’d like to propose Andrew Lamb as the initial Chair, (and thus ASF
> > VP) for the DataFusion project.
> >
> > Affiliations
> >
> > Andy Grove (agrove): NVidia
> > Andrew Lamb (alamb): InfluxData
> > Daniël Heres (dheres): Coralogix
> > Jie Wen (jakevin): SelectDB
> > Kun Liu (liukun): Ebay
> > Liang-Chi Hsieh (viirya): Apple
> > Qingping Hou: (houqp): Scribd
> > Wes McKinney(wesm): Posit
> > Will Jones (wjones127): LanceDB
> >
> > Proposed Initial Committers
> >
> > In addition to the PMC, we propose the following people as the initial
> > DataFusion committers. This is a subset of the existing Arrow
> > committers who contribute to DataFusion
> > https://people.apache.org/phonebook.html?unix=arrow
> >
> > akurmustafa Mustafa Akur (Synnada)
> > avantgardner Brent Gardner (Coralogix)
> > comphead Oleks V. (Unaffiliated)
> > jayzhan Jay Zhan (Unaffiliated)
> > jeffreyvo Jeffry Vo (Unaffiliated)
> > jiayuliu Liu Jiayu (Airbnb)
> > mete Metehan Yildirim (Synnada)
> > mingmwang Wang Mingming (Ebay)
> > mneumann Marco Neumann (InfluxData)
> > nju_yaho Zhong Yanghong (Ebay)
> > ozankabak Mehmet Ozan Kabak (Synnada)
> > paddyhoran Paddy Horan (Assured Allies)
> > rdettai Rémi Dettai (Cloudfuse)
> > sunchao Chao Sun (Apple)
> > thinkharderdev Daniel Harris (Coralogix)
> > tustvold Raphael Taylor-Davies (InfluxData)
> > wayne Ruihang Xia (Greptime)
> > xudong963 Xudong Wang (ByteDance)
> > yjshen Yijie Shen (Space and Time)
> > yangjiang Yang Jiang (ebay)
> >
> >
> > Risk Assessments
> >
> > Naming / Trademarks
> >
> > As a sub-project of Arrow, the DataFusion name has been used for over
> > 4 years without any known issues. A podling name search did not turn
> > up any concerns and was approved:
> > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> >
> > Legal / IP Clearance
> >
> > All DataFusion code has either been donated to the Arrow project with
> > appropriate IP clearance or has been developed directly under ASF
> > processes and procedures. Thus creating a new top level project poses
> > no new Legal or IP risks.
> >
> > Code Extraction
> >
> > The relevant code is already in 5 separate repositories:
> > https://github.com/apache/arrow-datafusion/
> > https://github.com/apache/arrow-datafusion-python
> > https://github.com/apache/arrow-ballista
> > https://github.com/apache/arrow-ballista-python
> > https://github.com/apache/arrow-datafusion-comet
> >
> > We foresee no issues with code extraction and propose these
> > repositories be renamed to reflect top level projects
> >
> > Note: https://github.com/apache/arrow-rs, the Rust implementation of
> > Arrow, would remain part of the Arrow project.
> >
> > Orphaned Products
> >
> > DataFusion is known to be used in many open source and commercial
> > projects
> >
> https://arrow.apache.org/datafusion/user-guide/introduction.html#known-users
> > ,
> > has had multiple commits daily for several years, and its adoption and
> > number of contributors appears to be growing. We do not foresee the
> > project being orphaned in the next several years.
> >
> > Inexperience with Open Source
> >
> > The proposed PMC has extensive experience with Apache Arrow and other
> > Apache projects, and includes PMC members, PMC chairs and an ASF
> > Member. The DataFusion PMC and more experienced committers will
> > continue to coach new community members who may be less familiar with
> > the Apache Way.
> >
> > Homogeneous Developers
> >
> > The 9 proposed PMC members are from 9 different employers and the
> > proposed committers are similarly distributed across affiliations. No
> > specific entity employs more than 3 total proposed developers.
> >
> > Reliance on Salaried Developers
> >
> > A substantial amount of work on DataFusion has been by salaried
> > developers, but it also has a long tradition of attracting
> > contributions from students and hobbyists and we plan no changes in
> > contribution structure.
> >
> > Relationships with Other Apache Products
> >
> > DataFusion will obviously have a strong relationship with the Arrow
> > project given the overlap in people. We don’t foresee close
> > collaboration with other projects at this time.
> >
> > Cryptography
> >
> > DataFusion does not directly support encryption and there are no
> > near-term plans to add support for encryption. Users who need this
> > functionality can use the extension APIs.
> >
> > Required Resources
> >
> > Mailing Lists
> >
> > - private@datafusion.apache.org for private PMC discussions (with
> > moderated subscriptions)
> > - dev@datafusion.apache.org
> > - commits@datafusion.apache.org
> > - user@datafusion.apache.org
> >
> > Version Control
> >
> > We propose to continue to use git for source control and github for
> > hosting and testing resources.
> >
> > We also need to rename the github repositories to reflect the new top
> > level names:
> >
> > https://github.com/apache/arrow-datafusion/ → apache/datafusion
> > https://github.com/apache/arrow-datafusion-python →
> > apache/datafusion-python
> > https://github.com/apache/arrow-ballista → apache/datafusion-ballista
> > https://github.com/apache/arrow-ballista-python →
> > apache/datafusion-ballista-python
> > https://github.com/apache/arrow-datafusion-comet →
> apache/datafusion-comet
> >
> >
> >
> > Issue Tracking
> >
> > DataFusion would continue to use github for its issue tracking and
> > communications
> >
> > Other Resources
> >
> > The existing repositories already make use of existing Apache
> > infrastructure, and we expect no change in the initial resource usage.
> > As the project continues to grow, we expect continued infrastructure
> > demand growth.
> >
> >
> > FAQ: Has a sub project been promoted to a top level project before?
> >
> > Yes, and it appears to happen commonly. The Arrow project itself was
> > created as a top level project from work that started in Apache Drill,
> > and there are many sub projects of Hadoop that spun out as their own
> > top level projects such as Mahout, Avro and HBase:
> >
> >
> https://news.apache.org/foundation/entry/the_apache_software_foundation_announces4
> >
> >
> >
> > Related material:
> > Name search request / research for DataFusion:
> > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> > Discussion about this proposal on the arrow mailing list:
> > https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> > Discussion about which repositories on the arrow mailing list:
> > https://lists.apache.org/thread/ob3n0d9ky0bgrryl3xn39w9k566bq00q
> > Discussion about initial PMC on the arrow mailing list:
> > https://lists.apache.org/thread/pymrzcdw4qdptvby85f69rg3pcckl15b
> > Discussion in github about creating a new DataFusion top level
> > project: https://github.com/apache/arrow-datafusion/discussions/6475
> > Discussion about graduating on incubator list:
> > https://lists.apache.org/thread/r4n73pmms1lv0jbohyx1o1z13d615t99
> > Original Proposal for the Arrow project:
> > https://lists.apache.org/thread/x2qzdwglm8pkqp9gv03bbgw17khl7pq3
> >
>
Re: [VOTE] Move Arrow DataFusion Subproject to new Top Level Apache Project
Posted by Weston Pace <we...@gmail.com>.
+1 (binding)
On Fri, Mar 1, 2024 at 3:33 AM Andrew Lamb <al...@influxdata.com> wrote:
> Hello,
>
> As we have discussed[1][2] I would like to vote on the proposal to
> create a new Apache Top Level Project for DataFusion. The text of the
> proposed resolution and background document is copy/pasted below
>
> If the community is in favor of this, we plan to submit the resolution
> to the ASF board for approval with the next Arrow report (for the
> April 2024 board meeting).
>
> The vote will be open for at least 7 days.
>
> [ ] +1 Accept this Proposal
> [ ] +0
> [ ] -1 Do not accept this proposal because...
>
> Andrew
>
> [1] https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> [2] https://github.com/apache/arrow-datafusion/discussions/6475
>
> ---------- Proposed Resolution ---------
>
> Resolution to Create the Apache DataFusion Project from the Apache
> Arrow DataFusion Sub Project
>
> =============================================================
>
> X. Establish the Apache DataFusion Project
>
> WHEREAS, the Board of Directors deems it to be in the best
> interests of the Foundation and consistent with the
> Foundation's purpose to establish a Project Management
> Committee charged with the creation and maintenance of
> open-source software related to an extensible query engine
> for distribution at no charge to the public.
>
> NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> Committee (PMC), to be known as the "Apache DataFusion Project",
> be and hereby is established pursuant to Bylaws of the
> Foundation; and be it further
>
> RESOLVED, that the Apache DataFusion Project be and hereby is
> responsible for the creation and maintenance of software
> related to an extensible query engine; and be it further
>
> RESOLVED, that the office of "Vice President, Apache DataFusion" be
> and hereby is created, the person holding such office to
> serve at the direction of the Board of Directors as the chair
> of the Apache DataFusion Project, and to have primary responsibility
> for management of the projects within the scope of
> responsibility of the Apache DataFusion Project; and be it further
>
> RESOLVED, that the persons listed immediately below be and
> hereby are appointed to serve as the initial members of the
> Apache DataFusion Project:
>
> * Andy Grove (agrove@apache.org)
> * Andrew Lamb (alamb@apache.org)
> * Daniël Heres (dheres@apache.org)
> * Jie Wen (jakevin@apache.org)
> * Kun Liu (liukun@apache.org)
> * Liang-Chi Hsieh (viirya@apache.org)
> * Qingping Hou: (houqp@apache.org)
> * Wes McKinney(wesm@apache.org)
> * Will Jones (wjones127@apache.org)
>
> RESOLVED, that the Apache DataFusion Project be and hereby
> is tasked with the migration and rationalization of the Apache
> Arrow DataFusion sub-project; and be it further
>
> RESOLVED, that all responsibilities pertaining to the Apache
> Arrow DataFusion sub-project encumbered upon the
> Apache Arrow Project are hereafter discharged.
>
> NOW, THEREFORE, BE IT FURTHER RESOLVED, that Andrew Lamb
> be appointed to the office of Vice President, Apache DataFusion, to
> serve in accordance with and subject to the direction of the
> Board of Directors and the Bylaws of the Foundation until
> death, resignation, retirement, removal or disqualification,
> or until a successor is appointed.
> =============================================================
>
>
> -------
>
>
> Summary:
>
> We propose creating a new top level project, Apache DataFusion, from
> an existing sub project of Apache Arrow to facilitate additional
> community and project growth.
>
> Abstract
>
> Apache Arrow DataFusion[1] is a very fast, extensible query engine
> for building high-quality data-centric systems in Rust, using the
> Apache Arrow in-memory format. DataFusion offers SQL and Dataframe
> APIs, excellent performance, built-in support for CSV, Parquet, JSON,
> and Avro, extensive customization, and a great community.
>
> [1] https://arrow.apache.org/datafusion/
>
>
> Proposal
>
> We propose creating a new top level ASF project, Apache DataFusion,
> governed initially by a subset of the Apache Arrow project’s PMC and
> committers. The project’s code is in five existing git repositories,
> currently governed by Apache Arrow which would transfer to the new top
> level project.
>
> Background
>
> When DataFusion was initially donated to the Arrow project, it did not
> have a strong enough community to stand on its own. It has since grown
> significantly, and benefited immensely from being part of Arrow and
> nurturing of the Apache Way, and now has a community strong enough to
> stand on its own and that would benefit from focused governance
> attention.
>
> The community has discussed this idea publicly for more than 6 months
> https://github.com/apache/arrow-datafusion/discussions/6475 and
> briefly on the Arrow PMC mailing list
> https://lists.apache.org/thread/thv2jdm6640l6gm88hy8jhk5prjww0cs. As
> of the time of this writing both had exclusively positive reactions.
>
> Several current members of the Arrow PMC are both active contributors
> to DataFusion and understand and believe deeply in the Apache Way, and
> play active governance roles in the Arrow project as PMC members and
> PMC chairs, guiding the community, and releasing software versions.
> With this existing governance experience and structure, the new top
> level project will be able to function well immediately and
> independently.
>
> Overview of DataFusion
>
> Current Status
>
> Meritocracy
>
> DataFusion has been developed as part of Apache Arrow and thus has
> been operating as a meritocracy. Many of the developers of DataFusion
> are Arrow PMC members or committers. The DataFusion project plans to
> continue adding new PMC and committers as the project matures and
> grows.
>
> Community
>
> The DataFusion development team seeks to foster the development and
> user communities. We hope that becoming a separate project will help
> both Arrow and DataFusion communities by being more focused. Focused
> governance will make it easier to grow the community of committers and
> PMC members and make the organization more clear to others.
>
> Alignment
>
> The ASF is a natural host for DataFusion given that it is already the
> home of Arrow, Parquet, and other related distributed system, storage
> and query execution systems.
>
> Project Leadership
>
> Proposed Initial PMC
>
> We propose the following people as the initial DataFusion PMC members.
> This is a subset of the existing Arrow PMC members who contribute to
> DataFusion https://people.apache.org/phonebook.html?unix=arrow
>
> Andy Grove (agrove): Arrow PMC Chair
> Andrew Lamb (alamb): Arrow PMC, past Arrow PMC Chair
> Daniël Heres (dheres) Arrow PMC
> Jie Wen (jakevin): Arrow PMC, Doris Committer
> Kun Liu (liukun): Arrow PMC, IoTDB PMC, TSFile PMC
> Liang-Chi Hsieh (viirya): Arrow PMC, Spark PMC
> Qingping Hou: (houqp): Arrow PMC
> Wes McKinney(wesm): Arrow PMC, ASF Member
> Will Jones (wjones127): Arrow PMC
>
> We’d like to propose Andrew Lamb as the initial Chair, (and thus ASF
> VP) for the DataFusion project.
>
> Affiliations
>
> Andy Grove (agrove): NVidia
> Andrew Lamb (alamb): InfluxData
> Daniël Heres (dheres): Coralogix
> Jie Wen (jakevin): SelectDB
> Kun Liu (liukun): Ebay
> Liang-Chi Hsieh (viirya): Apple
> Qingping Hou: (houqp): Scribd
> Wes McKinney(wesm): Posit
> Will Jones (wjones127): LanceDB
>
> Proposed Initial Committers
>
> In addition to the PMC, we propose the following people as the initial
> DataFusion committers. This is a subset of the existing Arrow
> committers who contribute to DataFusion
> https://people.apache.org/phonebook.html?unix=arrow
>
> akurmustafa Mustafa Akur (Synnada)
> avantgardner Brent Gardner (Coralogix)
> comphead Oleks V. (Unaffiliated)
> jayzhan Jay Zhan (Unaffiliated)
> jeffreyvo Jeffry Vo (Unaffiliated)
> jiayuliu Liu Jiayu (Airbnb)
> mete Metehan Yildirim (Synnada)
> mingmwang Wang Mingming (Ebay)
> mneumann Marco Neumann (InfluxData)
> nju_yaho Zhong Yanghong (Ebay)
> ozankabak Mehmet Ozan Kabak (Synnada)
> paddyhoran Paddy Horan (Assured Allies)
> rdettai Rémi Dettai (Cloudfuse)
> sunchao Chao Sun (Apple)
> thinkharderdev Daniel Harris (Coralogix)
> tustvold Raphael Taylor-Davies (InfluxData)
> wayne Ruihang Xia (Greptime)
> xudong963 Xudong Wang (ByteDance)
> yjshen Yijie Shen (Space and Time)
> yangjiang Yang Jiang (ebay)
>
>
> Risk Assessments
>
> Naming / Trademarks
>
> As a sub-project of Arrow, the DataFusion name has been used for over
> 4 years without any known issues. A podling name search did not turn
> up any concerns and was approved:
> https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
>
> Legal / IP Clearance
>
> All DataFusion code has either been donated to the Arrow project with
> appropriate IP clearance or has been developed directly under ASF
> processes and procedures. Thus creating a new top level project poses
> no new Legal or IP risks.
>
> Code Extraction
>
> The relevant code is already in 5 separate repositories:
> https://github.com/apache/arrow-datafusion/
> https://github.com/apache/arrow-datafusion-python
> https://github.com/apache/arrow-ballista
> https://github.com/apache/arrow-ballista-python
> https://github.com/apache/arrow-datafusion-comet
>
> We foresee no issues with code extraction and propose these
> repositories be renamed to reflect top level projects
>
> Note: https://github.com/apache/arrow-rs, the Rust implementation of
> Arrow, would remain part of the Arrow project.
>
> Orphaned Products
>
> DataFusion is known to be used in many open source and commercial
> projects
> https://arrow.apache.org/datafusion/user-guide/introduction.html#known-users
> ,
> has had multiple commits daily for several years, and its adoption and
> number of contributors appears to be growing. We do not foresee the
> project being orphaned in the next several years.
>
> Inexperience with Open Source
>
> The proposed PMC has extensive experience with Apache Arrow and other
> Apache projects, and includes PMC members, PMC chairs and an ASF
> Member. The DataFusion PMC and more experienced committers will
> continue to coach new community members who may be less familiar with
> the Apache Way.
>
> Homogeneous Developers
>
> The 9 proposed PMC members are from 9 different employers and the
> proposed committers are similarly distributed across affiliations. No
> specific entity employs more than 3 total proposed developers.
>
> Reliance on Salaried Developers
>
> A substantial amount of work on DataFusion has been by salaried
> developers, but it also has a long tradition of attracting
> contributions from students and hobbyists and we plan no changes in
> contribution structure.
>
> Relationships with Other Apache Products
>
> DataFusion will obviously have a strong relationship with the Arrow
> project given the overlap in people. We don’t foresee close
> collaboration with other projects at this time.
>
> Cryptography
>
> DataFusion does not directly support encryption and there are no
> near-term plans to add support for encryption. Users who need this
> functionality can use the extension APIs.
>
> Required Resources
>
> Mailing Lists
>
> - private@datafusion.apache.org for private PMC discussions (with
> moderated subscriptions)
> - dev@datafusion.apache.org
> - commits@datafusion.apache.org
> - user@datafusion.apache.org
>
> Version Control
>
> We propose to continue to use git for source control and github for
> hosting and testing resources.
>
> We also need to rename the github repositories to reflect the new top
> level names:
>
> https://github.com/apache/arrow-datafusion/ → apache/datafusion
> https://github.com/apache/arrow-datafusion-python →
> apache/datafusion-python
> https://github.com/apache/arrow-ballista → apache/datafusion-ballista
> https://github.com/apache/arrow-ballista-python →
> apache/datafusion-ballista-python
> https://github.com/apache/arrow-datafusion-comet → apache/datafusion-comet
>
>
>
> Issue Tracking
>
> DataFusion would continue to use github for its issue tracking and
> communications
>
> Other Resources
>
> The existing repositories already make use of existing Apache
> infrastructure, and we expect no change in the initial resource usage.
> As the project continues to grow, we expect continued infrastructure
> demand growth.
>
>
> FAQ: Has a sub project been promoted to a top level project before?
>
> Yes, and it appears to happen commonly. The Arrow project itself was
> created as a top level project from work that started in Apache Drill,
> and there are many sub projects of Hadoop that spun out as their own
> top level projects such as Mahout, Avro and HBase:
>
> https://news.apache.org/foundation/entry/the_apache_software_foundation_announces4
>
>
>
> Related material:
> Name search request / research for DataFusion:
> https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> Discussion about this proposal on the arrow mailing list:
> https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> Discussion about which repositories on the arrow mailing list:
> https://lists.apache.org/thread/ob3n0d9ky0bgrryl3xn39w9k566bq00q
> Discussion about initial PMC on the arrow mailing list:
> https://lists.apache.org/thread/pymrzcdw4qdptvby85f69rg3pcckl15b
> Discussion in github about creating a new DataFusion top level
> project: https://github.com/apache/arrow-datafusion/discussions/6475
> Discussion about graduating on incubator list:
> https://lists.apache.org/thread/r4n73pmms1lv0jbohyx1o1z13d615t99
> Original Proposal for the Arrow project:
> https://lists.apache.org/thread/x2qzdwglm8pkqp9gv03bbgw17khl7pq3
>
Re: [VOTE] Move Arrow DataFusion Subproject to new Top Level Apache Project
Posted by Denny Lee <de...@gmail.com>.
+1 (non-binding)
On Fri, Mar 1, 2024 at 18:58 kazuyuki tanimura <kt...@apple.com.invalid>
wrote:
> +1 (non-binding)
>
> Kazu
>
> > On Mar 1, 2024, at 5:44 PM, L. C. Hsieh <vi...@gmail.com> wrote:
> >
> > +1 (binding)
> >
> > On Fri, Mar 1, 2024 at 1:25 PM Joris Van den Bossche
> > <jo...@gmail.com> wrote:
> >>
> >> +1 (binding)
> >>
> >> On Fri, 1 Mar 2024 at 22:18, Sutou Kouhei <ko...@clear-code.com> wrote:
> >>>
> >>> +1
> >>>
> >>> In <CAFhtnRy2J9GCU6e2K56-KPVc=GAwEMuiPEYHmnwcd+hTkFA6AQ@mail.gmail.com
> >
> >>> "[VOTE] Move Arrow DataFusion Subproject to new Top Level Apache
> Project" on Fri, 1 Mar 2024 06:33:08 -0500,
> >>> Andrew Lamb <al...@influxdata.com> wrote:
> >>>
> >>>> Hello,
> >>>>
> >>>> As we have discussed[1][2] I would like to vote on the proposal to
> >>>> create a new Apache Top Level Project for DataFusion. The text of the
> >>>> proposed resolution and background document is copy/pasted below
> >>>>
> >>>> If the community is in favor of this, we plan to submit the resolution
> >>>> to the ASF board for approval with the next Arrow report (for the
> >>>> April 2024 board meeting).
> >>>>
> >>>> The vote will be open for at least 7 days.
> >>>>
> >>>> [ ] +1 Accept this Proposal
> >>>> [ ] +0
> >>>> [ ] -1 Do not accept this proposal because...
> >>>>
> >>>> Andrew
> >>>>
> >>>> [1] https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> >>>> [2] https://github.com/apache/arrow-datafusion/discussions/6475
> >>>>
> >>>> ---------- Proposed Resolution ---------
> >>>>
> >>>> Resolution to Create the Apache DataFusion Project from the Apache
> >>>> Arrow DataFusion Sub Project
> >>>>
> >>>> =============================================================
> >>>>
> >>>> X. Establish the Apache DataFusion Project
> >>>>
> >>>> WHEREAS, the Board of Directors deems it to be in the best
> >>>> interests of the Foundation and consistent with the
> >>>> Foundation's purpose to establish a Project Management
> >>>> Committee charged with the creation and maintenance of
> >>>> open-source software related to an extensible query engine
> >>>> for distribution at no charge to the public.
> >>>>
> >>>> NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> >>>> Committee (PMC), to be known as the "Apache DataFusion Project",
> >>>> be and hereby is established pursuant to Bylaws of the
> >>>> Foundation; and be it further
> >>>>
> >>>> RESOLVED, that the Apache DataFusion Project be and hereby is
> >>>> responsible for the creation and maintenance of software
> >>>> related to an extensible query engine; and be it further
> >>>>
> >>>> RESOLVED, that the office of "Vice President, Apache DataFusion" be
> >>>> and hereby is created, the person holding such office to
> >>>> serve at the direction of the Board of Directors as the chair
> >>>> of the Apache DataFusion Project, and to have primary responsibility
> >>>> for management of the projects within the scope of
> >>>> responsibility of the Apache DataFusion Project; and be it further
> >>>>
> >>>> RESOLVED, that the persons listed immediately below be and
> >>>> hereby are appointed to serve as the initial members of the
> >>>> Apache DataFusion Project:
> >>>>
> >>>> * Andy Grove (agrove@apache.org)
> >>>> * Andrew Lamb (alamb@apache.org)
> >>>> * Daniël Heres (dheres@apache.org)
> >>>> * Jie Wen (jakevin@apache.org)
> >>>> * Kun Liu (liukun@apache.org)
> >>>> * Liang-Chi Hsieh (viirya@apache.org)
> >>>> * Qingping Hou: (houqp@apache.org)
> >>>> * Wes McKinney(wesm@apache.org)
> >>>> * Will Jones (wjones127@apache.org)
> >>>>
> >>>> RESOLVED, that the Apache DataFusion Project be and hereby
> >>>> is tasked with the migration and rationalization of the Apache
> >>>> Arrow DataFusion sub-project; and be it further
> >>>>
> >>>> RESOLVED, that all responsibilities pertaining to the Apache
> >>>> Arrow DataFusion sub-project encumbered upon the
> >>>> Apache Arrow Project are hereafter discharged.
> >>>>
> >>>> NOW, THEREFORE, BE IT FURTHER RESOLVED, that Andrew Lamb
> >>>> be appointed to the office of Vice President, Apache DataFusion, to
> >>>> serve in accordance with and subject to the direction of the
> >>>> Board of Directors and the Bylaws of the Foundation until
> >>>> death, resignation, retirement, removal or disqualification,
> >>>> or until a successor is appointed.
> >>>> =============================================================
> >>>>
> >>>>
> >>>> -------
> >>>>
> >>>>
> >>>> Summary:
> >>>>
> >>>> We propose creating a new top level project, Apache DataFusion, from
> >>>> an existing sub project of Apache Arrow to facilitate additional
> >>>> community and project growth.
> >>>>
> >>>> Abstract
> >>>>
> >>>> Apache Arrow DataFusion[1] is a very fast, extensible query engine
> >>>> for building high-quality data-centric systems in Rust, using the
> >>>> Apache Arrow in-memory format. DataFusion offers SQL and Dataframe
> >>>> APIs, excellent performance, built-in support for CSV, Parquet, JSON,
> >>>> and Avro, extensive customization, and a great community.
> >>>>
> >>>> [1] https://arrow.apache.org/datafusion/
> >>>>
> >>>>
> >>>> Proposal
> >>>>
> >>>> We propose creating a new top level ASF project, Apache DataFusion,
> >>>> governed initially by a subset of the Apache Arrow project’s PMC and
> >>>> committers. The project’s code is in five existing git repositories,
> >>>> currently governed by Apache Arrow which would transfer to the new top
> >>>> level project.
> >>>>
> >>>> Background
> >>>>
> >>>> When DataFusion was initially donated to the Arrow project, it did not
> >>>> have a strong enough community to stand on its own. It has since grown
> >>>> significantly, and benefited immensely from being part of Arrow and
> >>>> nurturing of the Apache Way, and now has a community strong enough to
> >>>> stand on its own and that would benefit from focused governance
> >>>> attention.
> >>>>
> >>>> The community has discussed this idea publicly for more than 6 months
> >>>> https://github.com/apache/arrow-datafusion/discussions/6475 and
> >>>> briefly on the Arrow PMC mailing list
> >>>> https://lists.apache.org/thread/thv2jdm6640l6gm88hy8jhk5prjww0cs. As
> >>>> of the time of this writing both had exclusively positive reactions.
> >>>>
> >>>> Several current members of the Arrow PMC are both active contributors
> >>>> to DataFusion and understand and believe deeply in the Apache Way, and
> >>>> play active governance roles in the Arrow project as PMC members and
> >>>> PMC chairs, guiding the community, and releasing software versions.
> >>>> With this existing governance experience and structure, the new top
> >>>> level project will be able to function well immediately and
> >>>> independently.
> >>>>
> >>>> Overview of DataFusion
> >>>>
> >>>> Current Status
> >>>>
> >>>> Meritocracy
> >>>>
> >>>> DataFusion has been developed as part of Apache Arrow and thus has
> >>>> been operating as a meritocracy. Many of the developers of DataFusion
> >>>> are Arrow PMC members or committers. The DataFusion project plans to
> >>>> continue adding new PMC and committers as the project matures and
> >>>> grows.
> >>>>
> >>>> Community
> >>>>
> >>>> The DataFusion development team seeks to foster the development and
> >>>> user communities. We hope that becoming a separate project will help
> >>>> both Arrow and DataFusion communities by being more focused. Focused
> >>>> governance will make it easier to grow the community of committers and
> >>>> PMC members and make the organization more clear to others.
> >>>>
> >>>> Alignment
> >>>>
> >>>> The ASF is a natural host for DataFusion given that it is already the
> >>>> home of Arrow, Parquet, and other related distributed system, storage
> >>>> and query execution systems.
> >>>>
> >>>> Project Leadership
> >>>>
> >>>> Proposed Initial PMC
> >>>>
> >>>> We propose the following people as the initial DataFusion PMC members.
> >>>> This is a subset of the existing Arrow PMC members who contribute to
> >>>> DataFusion https://people.apache.org/phonebook.html?unix=arrow
> >>>>
> >>>> Andy Grove (agrove): Arrow PMC Chair
> >>>> Andrew Lamb (alamb): Arrow PMC, past Arrow PMC Chair
> >>>> Daniël Heres (dheres) Arrow PMC
> >>>> Jie Wen (jakevin): Arrow PMC, Doris Committer
> >>>> Kun Liu (liukun): Arrow PMC, IoTDB PMC, TSFile PMC
> >>>> Liang-Chi Hsieh (viirya): Arrow PMC, Spark PMC
> >>>> Qingping Hou: (houqp): Arrow PMC
> >>>> Wes McKinney(wesm): Arrow PMC, ASF Member
> >>>> Will Jones (wjones127): Arrow PMC
> >>>>
> >>>> We’d like to propose Andrew Lamb as the initial Chair, (and thus ASF
> >>>> VP) for the DataFusion project.
> >>>>
> >>>> Affiliations
> >>>>
> >>>> Andy Grove (agrove): NVidia
> >>>> Andrew Lamb (alamb): InfluxData
> >>>> Daniël Heres (dheres): Coralogix
> >>>> Jie Wen (jakevin): SelectDB
> >>>> Kun Liu (liukun): Ebay
> >>>> Liang-Chi Hsieh (viirya): Apple
> >>>> Qingping Hou: (houqp): Scribd
> >>>> Wes McKinney(wesm): Posit
> >>>> Will Jones (wjones127): LanceDB
> >>>>
> >>>> Proposed Initial Committers
> >>>>
> >>>> In addition to the PMC, we propose the following people as the initial
> >>>> DataFusion committers. This is a subset of the existing Arrow
> >>>> committers who contribute to DataFusion
> >>>> https://people.apache.org/phonebook.html?unix=arrow
> >>>>
> >>>> akurmustafa Mustafa Akur (Synnada)
> >>>> avantgardner Brent Gardner (Coralogix)
> >>>> comphead Oleks V. (Unaffiliated)
> >>>> jayzhan Jay Zhan (Unaffiliated)
> >>>> jeffreyvo Jeffry Vo (Unaffiliated)
> >>>> jiayuliu Liu Jiayu (Airbnb)
> >>>> mete Metehan Yildirim (Synnada)
> >>>> mingmwang Wang Mingming (Ebay)
> >>>> mneumann Marco Neumann (InfluxData)
> >>>> nju_yaho Zhong Yanghong (Ebay)
> >>>> ozankabak Mehmet Ozan Kabak (Synnada)
> >>>> paddyhoran Paddy Horan (Assured Allies)
> >>>> rdettai Rémi Dettai (Cloudfuse)
> >>>> sunchao Chao Sun (Apple)
> >>>> thinkharderdev Daniel Harris (Coralogix)
> >>>> tustvold Raphael Taylor-Davies (InfluxData)
> >>>> wayne Ruihang Xia (Greptime)
> >>>> xudong963 Xudong Wang (ByteDance)
> >>>> yjshen Yijie Shen (Space and Time)
> >>>> yangjiang Yang Jiang (ebay)
> >>>>
> >>>>
> >>>> Risk Assessments
> >>>>
> >>>> Naming / Trademarks
> >>>>
> >>>> As a sub-project of Arrow, the DataFusion name has been used for over
> >>>> 4 years without any known issues. A podling name search did not turn
> >>>> up any concerns and was approved:
> >>>> https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> >>>>
> >>>> Legal / IP Clearance
> >>>>
> >>>> All DataFusion code has either been donated to the Arrow project with
> >>>> appropriate IP clearance or has been developed directly under ASF
> >>>> processes and procedures. Thus creating a new top level project poses
> >>>> no new Legal or IP risks.
> >>>>
> >>>> Code Extraction
> >>>>
> >>>> The relevant code is already in 5 separate repositories:
> >>>> https://github.com/apache/arrow-datafusion/
> >>>> https://github.com/apache/arrow-datafusion-python
> >>>> https://github.com/apache/arrow-ballista
> >>>> https://github.com/apache/arrow-ballista-python
> >>>> https://github.com/apache/arrow-datafusion-comet
> >>>>
> >>>> We foresee no issues with code extraction and propose these
> >>>> repositories be renamed to reflect top level projects
> >>>>
> >>>> Note: https://github.com/apache/arrow-rs, the Rust implementation of
> >>>> Arrow, would remain part of the Arrow project.
> >>>>
> >>>> Orphaned Products
> >>>>
> >>>> DataFusion is known to be used in many open source and commercial
> >>>> projects
> https://arrow.apache.org/datafusion/user-guide/introduction.html#known-users
> ,
> >>>> has had multiple commits daily for several years, and its adoption and
> >>>> number of contributors appears to be growing. We do not foresee the
> >>>> project being orphaned in the next several years.
> >>>>
> >>>> Inexperience with Open Source
> >>>>
> >>>> The proposed PMC has extensive experience with Apache Arrow and other
> >>>> Apache projects, and includes PMC members, PMC chairs and an ASF
> >>>> Member. The DataFusion PMC and more experienced committers will
> >>>> continue to coach new community members who may be less familiar with
> >>>> the Apache Way.
> >>>>
> >>>> Homogeneous Developers
> >>>>
> >>>> The 9 proposed PMC members are from 9 different employers and the
> >>>> proposed committers are similarly distributed across affiliations. No
> >>>> specific entity employs more than 3 total proposed developers.
> >>>>
> >>>> Reliance on Salaried Developers
> >>>>
> >>>> A substantial amount of work on DataFusion has been by salaried
> >>>> developers, but it also has a long tradition of attracting
> >>>> contributions from students and hobbyists and we plan no changes in
> >>>> contribution structure.
> >>>>
> >>>> Relationships with Other Apache Products
> >>>>
> >>>> DataFusion will obviously have a strong relationship with the Arrow
> >>>> project given the overlap in people. We don’t foresee close
> >>>> collaboration with other projects at this time.
> >>>>
> >>>> Cryptography
> >>>>
> >>>> DataFusion does not directly support encryption and there are no
> >>>> near-term plans to add support for encryption. Users who need this
> >>>> functionality can use the extension APIs.
> >>>>
> >>>> Required Resources
> >>>>
> >>>> Mailing Lists
> >>>>
> >>>> - private@datafusion.apache.org for private PMC discussions (with
> >>>> moderated subscriptions)
> >>>> - dev@datafusion.apache.org
> >>>> - commits@datafusion.apache.org
> >>>> - user@datafusion.apache.org
> >>>>
> >>>> Version Control
> >>>>
> >>>> We propose to continue to use git for source control and github for
> >>>> hosting and testing resources.
> >>>>
> >>>> We also need to rename the github repositories to reflect the new top
> >>>> level names:
> >>>>
> >>>> https://github.com/apache/arrow-datafusion/ → apache/datafusion
> >>>> https://github.com/apache/arrow-datafusion-python →
> apache/datafusion-python
> >>>> https://github.com/apache/arrow-ballista → apache/datafusion-ballista
> >>>> https://github.com/apache/arrow-ballista-python →
> >>>> apache/datafusion-ballista-python
> >>>> https://github.com/apache/arrow-datafusion-comet →
> apache/datafusion-comet
> >>>>
> >>>>
> >>>>
> >>>> Issue Tracking
> >>>>
> >>>> DataFusion would continue to use github for its issue tracking and
> >>>> communications
> >>>>
> >>>> Other Resources
> >>>>
> >>>> The existing repositories already make use of existing Apache
> >>>> infrastructure, and we expect no change in the initial resource usage.
> >>>> As the project continues to grow, we expect continued infrastructure
> >>>> demand growth.
> >>>>
> >>>>
> >>>> FAQ: Has a sub project been promoted to a top level project before?
> >>>>
> >>>> Yes, and it appears to happen commonly. The Arrow project itself was
> >>>> created as a top level project from work that started in Apache Drill,
> >>>> and there are many sub projects of Hadoop that spun out as their own
> >>>> top level projects such as Mahout, Avro and HBase:
> >>>>
> https://news.apache.org/foundation/entry/the_apache_software_foundation_announces4
> >>>>
> >>>>
> >>>>
> >>>> Related material:
> >>>> Name search request / research for DataFusion:
> >>>> https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> >>>> Discussion about this proposal on the arrow mailing list:
> >>>> https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> >>>> Discussion about which repositories on the arrow mailing list:
> >>>> https://lists.apache.org/thread/ob3n0d9ky0bgrryl3xn39w9k566bq00q
> >>>> Discussion about initial PMC on the arrow mailing list:
> >>>> https://lists.apache.org/thread/pymrzcdw4qdptvby85f69rg3pcckl15b
> >>>> Discussion in github about creating a new DataFusion top level
> >>>> project: https://github.com/apache/arrow-datafusion/discussions/6475
> >>>> Discussion about graduating on incubator list:
> >>>> https://lists.apache.org/thread/r4n73pmms1lv0jbohyx1o1z13d615t99
> >>>> Original Proposal for the Arrow project:
> >>>> https://lists.apache.org/thread/x2qzdwglm8pkqp9gv03bbgw17khl7pq3
>
>
Re: [VOTE] Move Arrow DataFusion Subproject to new Top Level Apache Project
Posted by kazuyuki tanimura <kt...@apple.com.INVALID>.
+1 (non-binding)
Kazu
> On Mar 1, 2024, at 5:44 PM, L. C. Hsieh <vi...@gmail.com> wrote:
>
> +1 (binding)
>
> On Fri, Mar 1, 2024 at 1:25 PM Joris Van den Bossche
> <jo...@gmail.com> wrote:
>>
>> +1 (binding)
>>
>> On Fri, 1 Mar 2024 at 22:18, Sutou Kouhei <ko...@clear-code.com> wrote:
>>>
>>> +1
>>>
>>> In <CA...@mail.gmail.com>
>>> "[VOTE] Move Arrow DataFusion Subproject to new Top Level Apache Project" on Fri, 1 Mar 2024 06:33:08 -0500,
>>> Andrew Lamb <al...@influxdata.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> As we have discussed[1][2] I would like to vote on the proposal to
>>>> create a new Apache Top Level Project for DataFusion. The text of the
>>>> proposed resolution and background document is copy/pasted below
>>>>
>>>> If the community is in favor of this, we plan to submit the resolution
>>>> to the ASF board for approval with the next Arrow report (for the
>>>> April 2024 board meeting).
>>>>
>>>> The vote will be open for at least 7 days.
>>>>
>>>> [ ] +1 Accept this Proposal
>>>> [ ] +0
>>>> [ ] -1 Do not accept this proposal because...
>>>>
>>>> Andrew
>>>>
>>>> [1] https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
>>>> [2] https://github.com/apache/arrow-datafusion/discussions/6475
>>>>
>>>> ---------- Proposed Resolution ---------
>>>>
>>>> Resolution to Create the Apache DataFusion Project from the Apache
>>>> Arrow DataFusion Sub Project
>>>>
>>>> =============================================================
>>>>
>>>> X. Establish the Apache DataFusion Project
>>>>
>>>> WHEREAS, the Board of Directors deems it to be in the best
>>>> interests of the Foundation and consistent with the
>>>> Foundation's purpose to establish a Project Management
>>>> Committee charged with the creation and maintenance of
>>>> open-source software related to an extensible query engine
>>>> for distribution at no charge to the public.
>>>>
>>>> NOW, THEREFORE, BE IT RESOLVED, that a Project Management
>>>> Committee (PMC), to be known as the "Apache DataFusion Project",
>>>> be and hereby is established pursuant to Bylaws of the
>>>> Foundation; and be it further
>>>>
>>>> RESOLVED, that the Apache DataFusion Project be and hereby is
>>>> responsible for the creation and maintenance of software
>>>> related to an extensible query engine; and be it further
>>>>
>>>> RESOLVED, that the office of "Vice President, Apache DataFusion" be
>>>> and hereby is created, the person holding such office to
>>>> serve at the direction of the Board of Directors as the chair
>>>> of the Apache DataFusion Project, and to have primary responsibility
>>>> for management of the projects within the scope of
>>>> responsibility of the Apache DataFusion Project; and be it further
>>>>
>>>> RESOLVED, that the persons listed immediately below be and
>>>> hereby are appointed to serve as the initial members of the
>>>> Apache DataFusion Project:
>>>>
>>>> * Andy Grove (agrove@apache.org)
>>>> * Andrew Lamb (alamb@apache.org)
>>>> * Daniël Heres (dheres@apache.org)
>>>> * Jie Wen (jakevin@apache.org)
>>>> * Kun Liu (liukun@apache.org)
>>>> * Liang-Chi Hsieh (viirya@apache.org)
>>>> * Qingping Hou: (houqp@apache.org)
>>>> * Wes McKinney(wesm@apache.org)
>>>> * Will Jones (wjones127@apache.org)
>>>>
>>>> RESOLVED, that the Apache DataFusion Project be and hereby
>>>> is tasked with the migration and rationalization of the Apache
>>>> Arrow DataFusion sub-project; and be it further
>>>>
>>>> RESOLVED, that all responsibilities pertaining to the Apache
>>>> Arrow DataFusion sub-project encumbered upon the
>>>> Apache Arrow Project are hereafter discharged.
>>>>
>>>> NOW, THEREFORE, BE IT FURTHER RESOLVED, that Andrew Lamb
>>>> be appointed to the office of Vice President, Apache DataFusion, to
>>>> serve in accordance with and subject to the direction of the
>>>> Board of Directors and the Bylaws of the Foundation until
>>>> death, resignation, retirement, removal or disqualification,
>>>> or until a successor is appointed.
>>>> =============================================================
>>>>
>>>>
>>>> -------
>>>>
>>>>
>>>> Summary:
>>>>
>>>> We propose creating a new top level project, Apache DataFusion, from
>>>> an existing sub project of Apache Arrow to facilitate additional
>>>> community and project growth.
>>>>
>>>> Abstract
>>>>
>>>> Apache Arrow DataFusion[1] is a very fast, extensible query engine
>>>> for building high-quality data-centric systems in Rust, using the
>>>> Apache Arrow in-memory format. DataFusion offers SQL and Dataframe
>>>> APIs, excellent performance, built-in support for CSV, Parquet, JSON,
>>>> and Avro, extensive customization, and a great community.
>>>>
>>>> [1] https://arrow.apache.org/datafusion/
>>>>
>>>>
>>>> Proposal
>>>>
>>>> We propose creating a new top level ASF project, Apache DataFusion,
>>>> governed initially by a subset of the Apache Arrow project’s PMC and
>>>> committers. The project’s code is in five existing git repositories,
>>>> currently governed by Apache Arrow which would transfer to the new top
>>>> level project.
>>>>
>>>> Background
>>>>
>>>> When DataFusion was initially donated to the Arrow project, it did not
>>>> have a strong enough community to stand on its own. It has since grown
>>>> significantly, and benefited immensely from being part of Arrow and
>>>> nurturing of the Apache Way, and now has a community strong enough to
>>>> stand on its own and that would benefit from focused governance
>>>> attention.
>>>>
>>>> The community has discussed this idea publicly for more than 6 months
>>>> https://github.com/apache/arrow-datafusion/discussions/6475 and
>>>> briefly on the Arrow PMC mailing list
>>>> https://lists.apache.org/thread/thv2jdm6640l6gm88hy8jhk5prjww0cs. As
>>>> of the time of this writing both had exclusively positive reactions.
>>>>
>>>> Several current members of the Arrow PMC are both active contributors
>>>> to DataFusion and understand and believe deeply in the Apache Way, and
>>>> play active governance roles in the Arrow project as PMC members and
>>>> PMC chairs, guiding the community, and releasing software versions.
>>>> With this existing governance experience and structure, the new top
>>>> level project will be able to function well immediately and
>>>> independently.
>>>>
>>>> Overview of DataFusion
>>>>
>>>> Current Status
>>>>
>>>> Meritocracy
>>>>
>>>> DataFusion has been developed as part of Apache Arrow and thus has
>>>> been operating as a meritocracy. Many of the developers of DataFusion
>>>> are Arrow PMC members or committers. The DataFusion project plans to
>>>> continue adding new PMC and committers as the project matures and
>>>> grows.
>>>>
>>>> Community
>>>>
>>>> The DataFusion development team seeks to foster the development and
>>>> user communities. We hope that becoming a separate project will help
>>>> both Arrow and DataFusion communities by being more focused. Focused
>>>> governance will make it easier to grow the community of committers and
>>>> PMC members and make the organization more clear to others.
>>>>
>>>> Alignment
>>>>
>>>> The ASF is a natural host for DataFusion given that it is already the
>>>> home of Arrow, Parquet, and other related distributed system, storage
>>>> and query execution systems.
>>>>
>>>> Project Leadership
>>>>
>>>> Proposed Initial PMC
>>>>
>>>> We propose the following people as the initial DataFusion PMC members.
>>>> This is a subset of the existing Arrow PMC members who contribute to
>>>> DataFusion https://people.apache.org/phonebook.html?unix=arrow
>>>>
>>>> Andy Grove (agrove): Arrow PMC Chair
>>>> Andrew Lamb (alamb): Arrow PMC, past Arrow PMC Chair
>>>> Daniël Heres (dheres) Arrow PMC
>>>> Jie Wen (jakevin): Arrow PMC, Doris Committer
>>>> Kun Liu (liukun): Arrow PMC, IoTDB PMC, TSFile PMC
>>>> Liang-Chi Hsieh (viirya): Arrow PMC, Spark PMC
>>>> Qingping Hou: (houqp): Arrow PMC
>>>> Wes McKinney(wesm): Arrow PMC, ASF Member
>>>> Will Jones (wjones127): Arrow PMC
>>>>
>>>> We’d like to propose Andrew Lamb as the initial Chair, (and thus ASF
>>>> VP) for the DataFusion project.
>>>>
>>>> Affiliations
>>>>
>>>> Andy Grove (agrove): NVidia
>>>> Andrew Lamb (alamb): InfluxData
>>>> Daniël Heres (dheres): Coralogix
>>>> Jie Wen (jakevin): SelectDB
>>>> Kun Liu (liukun): Ebay
>>>> Liang-Chi Hsieh (viirya): Apple
>>>> Qingping Hou: (houqp): Scribd
>>>> Wes McKinney(wesm): Posit
>>>> Will Jones (wjones127): LanceDB
>>>>
>>>> Proposed Initial Committers
>>>>
>>>> In addition to the PMC, we propose the following people as the initial
>>>> DataFusion committers. This is a subset of the existing Arrow
>>>> committers who contribute to DataFusion
>>>> https://people.apache.org/phonebook.html?unix=arrow
>>>>
>>>> akurmustafa Mustafa Akur (Synnada)
>>>> avantgardner Brent Gardner (Coralogix)
>>>> comphead Oleks V. (Unaffiliated)
>>>> jayzhan Jay Zhan (Unaffiliated)
>>>> jeffreyvo Jeffry Vo (Unaffiliated)
>>>> jiayuliu Liu Jiayu (Airbnb)
>>>> mete Metehan Yildirim (Synnada)
>>>> mingmwang Wang Mingming (Ebay)
>>>> mneumann Marco Neumann (InfluxData)
>>>> nju_yaho Zhong Yanghong (Ebay)
>>>> ozankabak Mehmet Ozan Kabak (Synnada)
>>>> paddyhoran Paddy Horan (Assured Allies)
>>>> rdettai Rémi Dettai (Cloudfuse)
>>>> sunchao Chao Sun (Apple)
>>>> thinkharderdev Daniel Harris (Coralogix)
>>>> tustvold Raphael Taylor-Davies (InfluxData)
>>>> wayne Ruihang Xia (Greptime)
>>>> xudong963 Xudong Wang (ByteDance)
>>>> yjshen Yijie Shen (Space and Time)
>>>> yangjiang Yang Jiang (ebay)
>>>>
>>>>
>>>> Risk Assessments
>>>>
>>>> Naming / Trademarks
>>>>
>>>> As a sub-project of Arrow, the DataFusion name has been used for over
>>>> 4 years without any known issues. A podling name search did not turn
>>>> up any concerns and was approved:
>>>> https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
>>>>
>>>> Legal / IP Clearance
>>>>
>>>> All DataFusion code has either been donated to the Arrow project with
>>>> appropriate IP clearance or has been developed directly under ASF
>>>> processes and procedures. Thus creating a new top level project poses
>>>> no new Legal or IP risks.
>>>>
>>>> Code Extraction
>>>>
>>>> The relevant code is already in 5 separate repositories:
>>>> https://github.com/apache/arrow-datafusion/
>>>> https://github.com/apache/arrow-datafusion-python
>>>> https://github.com/apache/arrow-ballista
>>>> https://github.com/apache/arrow-ballista-python
>>>> https://github.com/apache/arrow-datafusion-comet
>>>>
>>>> We foresee no issues with code extraction and propose these
>>>> repositories be renamed to reflect top level projects
>>>>
>>>> Note: https://github.com/apache/arrow-rs, the Rust implementation of
>>>> Arrow, would remain part of the Arrow project.
>>>>
>>>> Orphaned Products
>>>>
>>>> DataFusion is known to be used in many open source and commercial
>>>> projects https://arrow.apache.org/datafusion/user-guide/introduction.html#known-users,
>>>> has had multiple commits daily for several years, and its adoption and
>>>> number of contributors appears to be growing. We do not foresee the
>>>> project being orphaned in the next several years.
>>>>
>>>> Inexperience with Open Source
>>>>
>>>> The proposed PMC has extensive experience with Apache Arrow and other
>>>> Apache projects, and includes PMC members, PMC chairs and an ASF
>>>> Member. The DataFusion PMC and more experienced committers will
>>>> continue to coach new community members who may be less familiar with
>>>> the Apache Way.
>>>>
>>>> Homogeneous Developers
>>>>
>>>> The 9 proposed PMC members are from 9 different employers and the
>>>> proposed committers are similarly distributed across affiliations. No
>>>> specific entity employs more than 3 total proposed developers.
>>>>
>>>> Reliance on Salaried Developers
>>>>
>>>> A substantial amount of work on DataFusion has been by salaried
>>>> developers, but it also has a long tradition of attracting
>>>> contributions from students and hobbyists and we plan no changes in
>>>> contribution structure.
>>>>
>>>> Relationships with Other Apache Products
>>>>
>>>> DataFusion will obviously have a strong relationship with the Arrow
>>>> project given the overlap in people. We don’t foresee close
>>>> collaboration with other projects at this time.
>>>>
>>>> Cryptography
>>>>
>>>> DataFusion does not directly support encryption and there are no
>>>> near-term plans to add support for encryption. Users who need this
>>>> functionality can use the extension APIs.
>>>>
>>>> Required Resources
>>>>
>>>> Mailing Lists
>>>>
>>>> - private@datafusion.apache.org for private PMC discussions (with
>>>> moderated subscriptions)
>>>> - dev@datafusion.apache.org
>>>> - commits@datafusion.apache.org
>>>> - user@datafusion.apache.org
>>>>
>>>> Version Control
>>>>
>>>> We propose to continue to use git for source control and github for
>>>> hosting and testing resources.
>>>>
>>>> We also need to rename the github repositories to reflect the new top
>>>> level names:
>>>>
>>>> https://github.com/apache/arrow-datafusion/ → apache/datafusion
>>>> https://github.com/apache/arrow-datafusion-python → apache/datafusion-python
>>>> https://github.com/apache/arrow-ballista → apache/datafusion-ballista
>>>> https://github.com/apache/arrow-ballista-python →
>>>> apache/datafusion-ballista-python
>>>> https://github.com/apache/arrow-datafusion-comet → apache/datafusion-comet
>>>>
>>>>
>>>>
>>>> Issue Tracking
>>>>
>>>> DataFusion would continue to use github for its issue tracking and
>>>> communications
>>>>
>>>> Other Resources
>>>>
>>>> The existing repositories already make use of existing Apache
>>>> infrastructure, and we expect no change in the initial resource usage.
>>>> As the project continues to grow, we expect continued infrastructure
>>>> demand growth.
>>>>
>>>>
>>>> FAQ: Has a sub project been promoted to a top level project before?
>>>>
>>>> Yes, and it appears to happen commonly. The Arrow project itself was
>>>> created as a top level project from work that started in Apache Drill,
>>>> and there are many sub projects of Hadoop that spun out as their own
>>>> top level projects such as Mahout, Avro and HBase:
>>>> https://news.apache.org/foundation/entry/the_apache_software_foundation_announces4
>>>>
>>>>
>>>>
>>>> Related material:
>>>> Name search request / research for DataFusion:
>>>> https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
>>>> Discussion about this proposal on the arrow mailing list:
>>>> https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
>>>> Discussion about which repositories on the arrow mailing list:
>>>> https://lists.apache.org/thread/ob3n0d9ky0bgrryl3xn39w9k566bq00q
>>>> Discussion about initial PMC on the arrow mailing list:
>>>> https://lists.apache.org/thread/pymrzcdw4qdptvby85f69rg3pcckl15b
>>>> Discussion in github about creating a new DataFusion top level
>>>> project: https://github.com/apache/arrow-datafusion/discussions/6475
>>>> Discussion about graduating on incubator list:
>>>> https://lists.apache.org/thread/r4n73pmms1lv0jbohyx1o1z13d615t99
>>>> Original Proposal for the Arrow project:
>>>> https://lists.apache.org/thread/x2qzdwglm8pkqp9gv03bbgw17khl7pq3
Re: [VOTE] Move Arrow DataFusion Subproject to new Top Level Apache Project
Posted by "L. C. Hsieh" <vi...@gmail.com>.
+1 (binding)
On Fri, Mar 1, 2024 at 1:25 PM Joris Van den Bossche
<jo...@gmail.com> wrote:
>
> +1 (binding)
>
> On Fri, 1 Mar 2024 at 22:18, Sutou Kouhei <ko...@clear-code.com> wrote:
> >
> > +1
> >
> > In <CA...@mail.gmail.com>
> > "[VOTE] Move Arrow DataFusion Subproject to new Top Level Apache Project" on Fri, 1 Mar 2024 06:33:08 -0500,
> > Andrew Lamb <al...@influxdata.com> wrote:
> >
> > > Hello,
> > >
> > > As we have discussed[1][2] I would like to vote on the proposal to
> > > create a new Apache Top Level Project for DataFusion. The text of the
> > > proposed resolution and background document is copy/pasted below
> > >
> > > If the community is in favor of this, we plan to submit the resolution
> > > to the ASF board for approval with the next Arrow report (for the
> > > April 2024 board meeting).
> > >
> > > The vote will be open for at least 7 days.
> > >
> > > [ ] +1 Accept this Proposal
> > > [ ] +0
> > > [ ] -1 Do not accept this proposal because...
> > >
> > > Andrew
> > >
> > > [1] https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> > > [2] https://github.com/apache/arrow-datafusion/discussions/6475
> > >
> > > ---------- Proposed Resolution ---------
> > >
> > > Resolution to Create the Apache DataFusion Project from the Apache
> > > Arrow DataFusion Sub Project
> > >
> > > =============================================================
> > >
> > > X. Establish the Apache DataFusion Project
> > >
> > > WHEREAS, the Board of Directors deems it to be in the best
> > > interests of the Foundation and consistent with the
> > > Foundation's purpose to establish a Project Management
> > > Committee charged with the creation and maintenance of
> > > open-source software related to an extensible query engine
> > > for distribution at no charge to the public.
> > >
> > > NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> > > Committee (PMC), to be known as the "Apache DataFusion Project",
> > > be and hereby is established pursuant to Bylaws of the
> > > Foundation; and be it further
> > >
> > > RESOLVED, that the Apache DataFusion Project be and hereby is
> > > responsible for the creation and maintenance of software
> > > related to an extensible query engine; and be it further
> > >
> > > RESOLVED, that the office of "Vice President, Apache DataFusion" be
> > > and hereby is created, the person holding such office to
> > > serve at the direction of the Board of Directors as the chair
> > > of the Apache DataFusion Project, and to have primary responsibility
> > > for management of the projects within the scope of
> > > responsibility of the Apache DataFusion Project; and be it further
> > >
> > > RESOLVED, that the persons listed immediately below be and
> > > hereby are appointed to serve as the initial members of the
> > > Apache DataFusion Project:
> > >
> > > * Andy Grove (agrove@apache.org)
> > > * Andrew Lamb (alamb@apache.org)
> > > * Daniël Heres (dheres@apache.org)
> > > * Jie Wen (jakevin@apache.org)
> > > * Kun Liu (liukun@apache.org)
> > > * Liang-Chi Hsieh (viirya@apache.org)
> > > * Qingping Hou: (houqp@apache.org)
> > > * Wes McKinney(wesm@apache.org)
> > > * Will Jones (wjones127@apache.org)
> > >
> > > RESOLVED, that the Apache DataFusion Project be and hereby
> > > is tasked with the migration and rationalization of the Apache
> > > Arrow DataFusion sub-project; and be it further
> > >
> > > RESOLVED, that all responsibilities pertaining to the Apache
> > > Arrow DataFusion sub-project encumbered upon the
> > > Apache Arrow Project are hereafter discharged.
> > >
> > > NOW, THEREFORE, BE IT FURTHER RESOLVED, that Andrew Lamb
> > > be appointed to the office of Vice President, Apache DataFusion, to
> > > serve in accordance with and subject to the direction of the
> > > Board of Directors and the Bylaws of the Foundation until
> > > death, resignation, retirement, removal or disqualification,
> > > or until a successor is appointed.
> > > =============================================================
> > >
> > >
> > > -------
> > >
> > >
> > > Summary:
> > >
> > > We propose creating a new top level project, Apache DataFusion, from
> > > an existing sub project of Apache Arrow to facilitate additional
> > > community and project growth.
> > >
> > > Abstract
> > >
> > > Apache Arrow DataFusion[1] is a very fast, extensible query engine
> > > for building high-quality data-centric systems in Rust, using the
> > > Apache Arrow in-memory format. DataFusion offers SQL and Dataframe
> > > APIs, excellent performance, built-in support for CSV, Parquet, JSON,
> > > and Avro, extensive customization, and a great community.
> > >
> > > [1] https://arrow.apache.org/datafusion/
> > >
> > >
> > > Proposal
> > >
> > > We propose creating a new top level ASF project, Apache DataFusion,
> > > governed initially by a subset of the Apache Arrow project’s PMC and
> > > committers. The project’s code is in five existing git repositories,
> > > currently governed by Apache Arrow which would transfer to the new top
> > > level project.
> > >
> > > Background
> > >
> > > When DataFusion was initially donated to the Arrow project, it did not
> > > have a strong enough community to stand on its own. It has since grown
> > > significantly, and benefited immensely from being part of Arrow and
> > > nurturing of the Apache Way, and now has a community strong enough to
> > > stand on its own and that would benefit from focused governance
> > > attention.
> > >
> > > The community has discussed this idea publicly for more than 6 months
> > > https://github.com/apache/arrow-datafusion/discussions/6475 and
> > > briefly on the Arrow PMC mailing list
> > > https://lists.apache.org/thread/thv2jdm6640l6gm88hy8jhk5prjww0cs. As
> > > of the time of this writing both had exclusively positive reactions.
> > >
> > > Several current members of the Arrow PMC are both active contributors
> > > to DataFusion and understand and believe deeply in the Apache Way, and
> > > play active governance roles in the Arrow project as PMC members and
> > > PMC chairs, guiding the community, and releasing software versions.
> > > With this existing governance experience and structure, the new top
> > > level project will be able to function well immediately and
> > > independently.
> > >
> > > Overview of DataFusion
> > >
> > > Current Status
> > >
> > > Meritocracy
> > >
> > > DataFusion has been developed as part of Apache Arrow and thus has
> > > been operating as a meritocracy. Many of the developers of DataFusion
> > > are Arrow PMC members or committers. The DataFusion project plans to
> > > continue adding new PMC and committers as the project matures and
> > > grows.
> > >
> > > Community
> > >
> > > The DataFusion development team seeks to foster the development and
> > > user communities. We hope that becoming a separate project will help
> > > both Arrow and DataFusion communities by being more focused. Focused
> > > governance will make it easier to grow the community of committers and
> > > PMC members and make the organization more clear to others.
> > >
> > > Alignment
> > >
> > > The ASF is a natural host for DataFusion given that it is already the
> > > home of Arrow, Parquet, and other related distributed system, storage
> > > and query execution systems.
> > >
> > > Project Leadership
> > >
> > > Proposed Initial PMC
> > >
> > > We propose the following people as the initial DataFusion PMC members.
> > > This is a subset of the existing Arrow PMC members who contribute to
> > > DataFusion https://people.apache.org/phonebook.html?unix=arrow
> > >
> > > Andy Grove (agrove): Arrow PMC Chair
> > > Andrew Lamb (alamb): Arrow PMC, past Arrow PMC Chair
> > > Daniël Heres (dheres) Arrow PMC
> > > Jie Wen (jakevin): Arrow PMC, Doris Committer
> > > Kun Liu (liukun): Arrow PMC, IoTDB PMC, TSFile PMC
> > > Liang-Chi Hsieh (viirya): Arrow PMC, Spark PMC
> > > Qingping Hou: (houqp): Arrow PMC
> > > Wes McKinney(wesm): Arrow PMC, ASF Member
> > > Will Jones (wjones127): Arrow PMC
> > >
> > > We’d like to propose Andrew Lamb as the initial Chair, (and thus ASF
> > > VP) for the DataFusion project.
> > >
> > > Affiliations
> > >
> > > Andy Grove (agrove): NVidia
> > > Andrew Lamb (alamb): InfluxData
> > > Daniël Heres (dheres): Coralogix
> > > Jie Wen (jakevin): SelectDB
> > > Kun Liu (liukun): Ebay
> > > Liang-Chi Hsieh (viirya): Apple
> > > Qingping Hou: (houqp): Scribd
> > > Wes McKinney(wesm): Posit
> > > Will Jones (wjones127): LanceDB
> > >
> > > Proposed Initial Committers
> > >
> > > In addition to the PMC, we propose the following people as the initial
> > > DataFusion committers. This is a subset of the existing Arrow
> > > committers who contribute to DataFusion
> > > https://people.apache.org/phonebook.html?unix=arrow
> > >
> > > akurmustafa Mustafa Akur (Synnada)
> > > avantgardner Brent Gardner (Coralogix)
> > > comphead Oleks V. (Unaffiliated)
> > > jayzhan Jay Zhan (Unaffiliated)
> > > jeffreyvo Jeffry Vo (Unaffiliated)
> > > jiayuliu Liu Jiayu (Airbnb)
> > > mete Metehan Yildirim (Synnada)
> > > mingmwang Wang Mingming (Ebay)
> > > mneumann Marco Neumann (InfluxData)
> > > nju_yaho Zhong Yanghong (Ebay)
> > > ozankabak Mehmet Ozan Kabak (Synnada)
> > > paddyhoran Paddy Horan (Assured Allies)
> > > rdettai Rémi Dettai (Cloudfuse)
> > > sunchao Chao Sun (Apple)
> > > thinkharderdev Daniel Harris (Coralogix)
> > > tustvold Raphael Taylor-Davies (InfluxData)
> > > wayne Ruihang Xia (Greptime)
> > > xudong963 Xudong Wang (ByteDance)
> > > yjshen Yijie Shen (Space and Time)
> > > yangjiang Yang Jiang (ebay)
> > >
> > >
> > > Risk Assessments
> > >
> > > Naming / Trademarks
> > >
> > > As a sub-project of Arrow, the DataFusion name has been used for over
> > > 4 years without any known issues. A podling name search did not turn
> > > up any concerns and was approved:
> > > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> > >
> > > Legal / IP Clearance
> > >
> > > All DataFusion code has either been donated to the Arrow project with
> > > appropriate IP clearance or has been developed directly under ASF
> > > processes and procedures. Thus creating a new top level project poses
> > > no new Legal or IP risks.
> > >
> > > Code Extraction
> > >
> > > The relevant code is already in 5 separate repositories:
> > > https://github.com/apache/arrow-datafusion/
> > > https://github.com/apache/arrow-datafusion-python
> > > https://github.com/apache/arrow-ballista
> > > https://github.com/apache/arrow-ballista-python
> > > https://github.com/apache/arrow-datafusion-comet
> > >
> > > We foresee no issues with code extraction and propose these
> > > repositories be renamed to reflect top level projects
> > >
> > > Note: https://github.com/apache/arrow-rs, the Rust implementation of
> > > Arrow, would remain part of the Arrow project.
> > >
> > > Orphaned Products
> > >
> > > DataFusion is known to be used in many open source and commercial
> > > projects https://arrow.apache.org/datafusion/user-guide/introduction.html#known-users,
> > > has had multiple commits daily for several years, and its adoption and
> > > number of contributors appears to be growing. We do not foresee the
> > > project being orphaned in the next several years.
> > >
> > > Inexperience with Open Source
> > >
> > > The proposed PMC has extensive experience with Apache Arrow and other
> > > Apache projects, and includes PMC members, PMC chairs and an ASF
> > > Member. The DataFusion PMC and more experienced committers will
> > > continue to coach new community members who may be less familiar with
> > > the Apache Way.
> > >
> > > Homogeneous Developers
> > >
> > > The 9 proposed PMC members are from 9 different employers and the
> > > proposed committers are similarly distributed across affiliations. No
> > > specific entity employs more than 3 total proposed developers.
> > >
> > > Reliance on Salaried Developers
> > >
> > > A substantial amount of work on DataFusion has been by salaried
> > > developers, but it also has a long tradition of attracting
> > > contributions from students and hobbyists and we plan no changes in
> > > contribution structure.
> > >
> > > Relationships with Other Apache Products
> > >
> > > DataFusion will obviously have a strong relationship with the Arrow
> > > project given the overlap in people. We don’t foresee close
> > > collaboration with other projects at this time.
> > >
> > > Cryptography
> > >
> > > DataFusion does not directly support encryption and there are no
> > > near-term plans to add support for encryption. Users who need this
> > > functionality can use the extension APIs.
> > >
> > > Required Resources
> > >
> > > Mailing Lists
> > >
> > > - private@datafusion.apache.org for private PMC discussions (with
> > > moderated subscriptions)
> > > - dev@datafusion.apache.org
> > > - commits@datafusion.apache.org
> > > - user@datafusion.apache.org
> > >
> > > Version Control
> > >
> > > We propose to continue to use git for source control and github for
> > > hosting and testing resources.
> > >
> > > We also need to rename the github repositories to reflect the new top
> > > level names:
> > >
> > > https://github.com/apache/arrow-datafusion/ → apache/datafusion
> > > https://github.com/apache/arrow-datafusion-python → apache/datafusion-python
> > > https://github.com/apache/arrow-ballista → apache/datafusion-ballista
> > > https://github.com/apache/arrow-ballista-python →
> > > apache/datafusion-ballista-python
> > > https://github.com/apache/arrow-datafusion-comet → apache/datafusion-comet
> > >
> > >
> > >
> > > Issue Tracking
> > >
> > > DataFusion would continue to use github for its issue tracking and
> > > communications
> > >
> > > Other Resources
> > >
> > > The existing repositories already make use of existing Apache
> > > infrastructure, and we expect no change in the initial resource usage.
> > > As the project continues to grow, we expect continued infrastructure
> > > demand growth.
> > >
> > >
> > > FAQ: Has a sub project been promoted to a top level project before?
> > >
> > > Yes, and it appears to happen commonly. The Arrow project itself was
> > > created as a top level project from work that started in Apache Drill,
> > > and there are many sub projects of Hadoop that spun out as their own
> > > top level projects such as Mahout, Avro and HBase:
> > > https://news.apache.org/foundation/entry/the_apache_software_foundation_announces4
> > >
> > >
> > >
> > > Related material:
> > > Name search request / research for DataFusion:
> > > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> > > Discussion about this proposal on the arrow mailing list:
> > > https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> > > Discussion about which repositories on the arrow mailing list:
> > > https://lists.apache.org/thread/ob3n0d9ky0bgrryl3xn39w9k566bq00q
> > > Discussion about initial PMC on the arrow mailing list:
> > > https://lists.apache.org/thread/pymrzcdw4qdptvby85f69rg3pcckl15b
> > > Discussion in github about creating a new DataFusion top level
> > > project: https://github.com/apache/arrow-datafusion/discussions/6475
> > > Discussion about graduating on incubator list:
> > > https://lists.apache.org/thread/r4n73pmms1lv0jbohyx1o1z13d615t99
> > > Original Proposal for the Arrow project:
> > > https://lists.apache.org/thread/x2qzdwglm8pkqp9gv03bbgw17khl7pq3
Re: [VOTE] Move Arrow DataFusion Subproject to new Top Level Apache Project
Posted by Joris Van den Bossche <jo...@gmail.com>.
+1 (binding)
On Fri, 1 Mar 2024 at 22:18, Sutou Kouhei <ko...@clear-code.com> wrote:
>
> +1
>
> In <CA...@mail.gmail.com>
> "[VOTE] Move Arrow DataFusion Subproject to new Top Level Apache Project" on Fri, 1 Mar 2024 06:33:08 -0500,
> Andrew Lamb <al...@influxdata.com> wrote:
>
> > Hello,
> >
> > As we have discussed[1][2] I would like to vote on the proposal to
> > create a new Apache Top Level Project for DataFusion. The text of the
> > proposed resolution and background document is copy/pasted below
> >
> > If the community is in favor of this, we plan to submit the resolution
> > to the ASF board for approval with the next Arrow report (for the
> > April 2024 board meeting).
> >
> > The vote will be open for at least 7 days.
> >
> > [ ] +1 Accept this Proposal
> > [ ] +0
> > [ ] -1 Do not accept this proposal because...
> >
> > Andrew
> >
> > [1] https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> > [2] https://github.com/apache/arrow-datafusion/discussions/6475
> >
> > ---------- Proposed Resolution ---------
> >
> > Resolution to Create the Apache DataFusion Project from the Apache
> > Arrow DataFusion Sub Project
> >
> > =============================================================
> >
> > X. Establish the Apache DataFusion Project
> >
> > WHEREAS, the Board of Directors deems it to be in the best
> > interests of the Foundation and consistent with the
> > Foundation's purpose to establish a Project Management
> > Committee charged with the creation and maintenance of
> > open-source software related to an extensible query engine
> > for distribution at no charge to the public.
> >
> > NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> > Committee (PMC), to be known as the "Apache DataFusion Project",
> > be and hereby is established pursuant to Bylaws of the
> > Foundation; and be it further
> >
> > RESOLVED, that the Apache DataFusion Project be and hereby is
> > responsible for the creation and maintenance of software
> > related to an extensible query engine; and be it further
> >
> > RESOLVED, that the office of "Vice President, Apache DataFusion" be
> > and hereby is created, the person holding such office to
> > serve at the direction of the Board of Directors as the chair
> > of the Apache DataFusion Project, and to have primary responsibility
> > for management of the projects within the scope of
> > responsibility of the Apache DataFusion Project; and be it further
> >
> > RESOLVED, that the persons listed immediately below be and
> > hereby are appointed to serve as the initial members of the
> > Apache DataFusion Project:
> >
> > * Andy Grove (agrove@apache.org)
> > * Andrew Lamb (alamb@apache.org)
> > * Daniël Heres (dheres@apache.org)
> > * Jie Wen (jakevin@apache.org)
> > * Kun Liu (liukun@apache.org)
> > * Liang-Chi Hsieh (viirya@apache.org)
> > * Qingping Hou: (houqp@apache.org)
> > * Wes McKinney(wesm@apache.org)
> > * Will Jones (wjones127@apache.org)
> >
> > RESOLVED, that the Apache DataFusion Project be and hereby
> > is tasked with the migration and rationalization of the Apache
> > Arrow DataFusion sub-project; and be it further
> >
> > RESOLVED, that all responsibilities pertaining to the Apache
> > Arrow DataFusion sub-project encumbered upon the
> > Apache Arrow Project are hereafter discharged.
> >
> > NOW, THEREFORE, BE IT FURTHER RESOLVED, that Andrew Lamb
> > be appointed to the office of Vice President, Apache DataFusion, to
> > serve in accordance with and subject to the direction of the
> > Board of Directors and the Bylaws of the Foundation until
> > death, resignation, retirement, removal or disqualification,
> > or until a successor is appointed.
> > =============================================================
> >
> >
> > -------
> >
> >
> > Summary:
> >
> > We propose creating a new top level project, Apache DataFusion, from
> > an existing sub project of Apache Arrow to facilitate additional
> > community and project growth.
> >
> > Abstract
> >
> > Apache Arrow DataFusion[1] is a very fast, extensible query engine
> > for building high-quality data-centric systems in Rust, using the
> > Apache Arrow in-memory format. DataFusion offers SQL and Dataframe
> > APIs, excellent performance, built-in support for CSV, Parquet, JSON,
> > and Avro, extensive customization, and a great community.
> >
> > [1] https://arrow.apache.org/datafusion/
> >
> >
> > Proposal
> >
> > We propose creating a new top level ASF project, Apache DataFusion,
> > governed initially by a subset of the Apache Arrow project’s PMC and
> > committers. The project’s code is in five existing git repositories,
> > currently governed by Apache Arrow which would transfer to the new top
> > level project.
> >
> > Background
> >
> > When DataFusion was initially donated to the Arrow project, it did not
> > have a strong enough community to stand on its own. It has since grown
> > significantly, and benefited immensely from being part of Arrow and
> > nurturing of the Apache Way, and now has a community strong enough to
> > stand on its own and that would benefit from focused governance
> > attention.
> >
> > The community has discussed this idea publicly for more than 6 months
> > https://github.com/apache/arrow-datafusion/discussions/6475 and
> > briefly on the Arrow PMC mailing list
> > https://lists.apache.org/thread/thv2jdm6640l6gm88hy8jhk5prjww0cs. As
> > of the time of this writing both had exclusively positive reactions.
> >
> > Several current members of the Arrow PMC are both active contributors
> > to DataFusion and understand and believe deeply in the Apache Way, and
> > play active governance roles in the Arrow project as PMC members and
> > PMC chairs, guiding the community, and releasing software versions.
> > With this existing governance experience and structure, the new top
> > level project will be able to function well immediately and
> > independently.
> >
> > Overview of DataFusion
> >
> > Current Status
> >
> > Meritocracy
> >
> > DataFusion has been developed as part of Apache Arrow and thus has
> > been operating as a meritocracy. Many of the developers of DataFusion
> > are Arrow PMC members or committers. The DataFusion project plans to
> > continue adding new PMC and committers as the project matures and
> > grows.
> >
> > Community
> >
> > The DataFusion development team seeks to foster the development and
> > user communities. We hope that becoming a separate project will help
> > both Arrow and DataFusion communities by being more focused. Focused
> > governance will make it easier to grow the community of committers and
> > PMC members and make the organization more clear to others.
> >
> > Alignment
> >
> > The ASF is a natural host for DataFusion given that it is already the
> > home of Arrow, Parquet, and other related distributed system, storage
> > and query execution systems.
> >
> > Project Leadership
> >
> > Proposed Initial PMC
> >
> > We propose the following people as the initial DataFusion PMC members.
> > This is a subset of the existing Arrow PMC members who contribute to
> > DataFusion https://people.apache.org/phonebook.html?unix=arrow
> >
> > Andy Grove (agrove): Arrow PMC Chair
> > Andrew Lamb (alamb): Arrow PMC, past Arrow PMC Chair
> > Daniël Heres (dheres) Arrow PMC
> > Jie Wen (jakevin): Arrow PMC, Doris Committer
> > Kun Liu (liukun): Arrow PMC, IoTDB PMC, TSFile PMC
> > Liang-Chi Hsieh (viirya): Arrow PMC, Spark PMC
> > Qingping Hou: (houqp): Arrow PMC
> > Wes McKinney(wesm): Arrow PMC, ASF Member
> > Will Jones (wjones127): Arrow PMC
> >
> > We’d like to propose Andrew Lamb as the initial Chair, (and thus ASF
> > VP) for the DataFusion project.
> >
> > Affiliations
> >
> > Andy Grove (agrove): NVidia
> > Andrew Lamb (alamb): InfluxData
> > Daniël Heres (dheres): Coralogix
> > Jie Wen (jakevin): SelectDB
> > Kun Liu (liukun): Ebay
> > Liang-Chi Hsieh (viirya): Apple
> > Qingping Hou: (houqp): Scribd
> > Wes McKinney(wesm): Posit
> > Will Jones (wjones127): LanceDB
> >
> > Proposed Initial Committers
> >
> > In addition to the PMC, we propose the following people as the initial
> > DataFusion committers. This is a subset of the existing Arrow
> > committers who contribute to DataFusion
> > https://people.apache.org/phonebook.html?unix=arrow
> >
> > akurmustafa Mustafa Akur (Synnada)
> > avantgardner Brent Gardner (Coralogix)
> > comphead Oleks V. (Unaffiliated)
> > jayzhan Jay Zhan (Unaffiliated)
> > jeffreyvo Jeffry Vo (Unaffiliated)
> > jiayuliu Liu Jiayu (Airbnb)
> > mete Metehan Yildirim (Synnada)
> > mingmwang Wang Mingming (Ebay)
> > mneumann Marco Neumann (InfluxData)
> > nju_yaho Zhong Yanghong (Ebay)
> > ozankabak Mehmet Ozan Kabak (Synnada)
> > paddyhoran Paddy Horan (Assured Allies)
> > rdettai Rémi Dettai (Cloudfuse)
> > sunchao Chao Sun (Apple)
> > thinkharderdev Daniel Harris (Coralogix)
> > tustvold Raphael Taylor-Davies (InfluxData)
> > wayne Ruihang Xia (Greptime)
> > xudong963 Xudong Wang (ByteDance)
> > yjshen Yijie Shen (Space and Time)
> > yangjiang Yang Jiang (ebay)
> >
> >
> > Risk Assessments
> >
> > Naming / Trademarks
> >
> > As a sub-project of Arrow, the DataFusion name has been used for over
> > 4 years without any known issues. A podling name search did not turn
> > up any concerns and was approved:
> > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> >
> > Legal / IP Clearance
> >
> > All DataFusion code has either been donated to the Arrow project with
> > appropriate IP clearance or has been developed directly under ASF
> > processes and procedures. Thus creating a new top level project poses
> > no new Legal or IP risks.
> >
> > Code Extraction
> >
> > The relevant code is already in 5 separate repositories:
> > https://github.com/apache/arrow-datafusion/
> > https://github.com/apache/arrow-datafusion-python
> > https://github.com/apache/arrow-ballista
> > https://github.com/apache/arrow-ballista-python
> > https://github.com/apache/arrow-datafusion-comet
> >
> > We foresee no issues with code extraction and propose these
> > repositories be renamed to reflect top level projects
> >
> > Note: https://github.com/apache/arrow-rs, the Rust implementation of
> > Arrow, would remain part of the Arrow project.
> >
> > Orphaned Products
> >
> > DataFusion is known to be used in many open source and commercial
> > projects https://arrow.apache.org/datafusion/user-guide/introduction.html#known-users,
> > has had multiple commits daily for several years, and its adoption and
> > number of contributors appears to be growing. We do not foresee the
> > project being orphaned in the next several years.
> >
> > Inexperience with Open Source
> >
> > The proposed PMC has extensive experience with Apache Arrow and other
> > Apache projects, and includes PMC members, PMC chairs and an ASF
> > Member. The DataFusion PMC and more experienced committers will
> > continue to coach new community members who may be less familiar with
> > the Apache Way.
> >
> > Homogeneous Developers
> >
> > The 9 proposed PMC members are from 9 different employers and the
> > proposed committers are similarly distributed across affiliations. No
> > specific entity employs more than 3 total proposed developers.
> >
> > Reliance on Salaried Developers
> >
> > A substantial amount of work on DataFusion has been by salaried
> > developers, but it also has a long tradition of attracting
> > contributions from students and hobbyists and we plan no changes in
> > contribution structure.
> >
> > Relationships with Other Apache Products
> >
> > DataFusion will obviously have a strong relationship with the Arrow
> > project given the overlap in people. We don’t foresee close
> > collaboration with other projects at this time.
> >
> > Cryptography
> >
> > DataFusion does not directly support encryption and there are no
> > near-term plans to add support for encryption. Users who need this
> > functionality can use the extension APIs.
> >
> > Required Resources
> >
> > Mailing Lists
> >
> > - private@datafusion.apache.org for private PMC discussions (with
> > moderated subscriptions)
> > - dev@datafusion.apache.org
> > - commits@datafusion.apache.org
> > - user@datafusion.apache.org
> >
> > Version Control
> >
> > We propose to continue to use git for source control and github for
> > hosting and testing resources.
> >
> > We also need to rename the github repositories to reflect the new top
> > level names:
> >
> > https://github.com/apache/arrow-datafusion/ → apache/datafusion
> > https://github.com/apache/arrow-datafusion-python → apache/datafusion-python
> > https://github.com/apache/arrow-ballista → apache/datafusion-ballista
> > https://github.com/apache/arrow-ballista-python →
> > apache/datafusion-ballista-python
> > https://github.com/apache/arrow-datafusion-comet → apache/datafusion-comet
> >
> >
> >
> > Issue Tracking
> >
> > DataFusion would continue to use github for its issue tracking and
> > communications
> >
> > Other Resources
> >
> > The existing repositories already make use of existing Apache
> > infrastructure, and we expect no change in the initial resource usage.
> > As the project continues to grow, we expect continued infrastructure
> > demand growth.
> >
> >
> > FAQ: Has a sub project been promoted to a top level project before?
> >
> > Yes, and it appears to happen commonly. The Arrow project itself was
> > created as a top level project from work that started in Apache Drill,
> > and there are many sub projects of Hadoop that spun out as their own
> > top level projects such as Mahout, Avro and HBase:
> > https://news.apache.org/foundation/entry/the_apache_software_foundation_announces4
> >
> >
> >
> > Related material:
> > Name search request / research for DataFusion:
> > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> > Discussion about this proposal on the arrow mailing list:
> > https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> > Discussion about which repositories on the arrow mailing list:
> > https://lists.apache.org/thread/ob3n0d9ky0bgrryl3xn39w9k566bq00q
> > Discussion about initial PMC on the arrow mailing list:
> > https://lists.apache.org/thread/pymrzcdw4qdptvby85f69rg3pcckl15b
> > Discussion in github about creating a new DataFusion top level
> > project: https://github.com/apache/arrow-datafusion/discussions/6475
> > Discussion about graduating on incubator list:
> > https://lists.apache.org/thread/r4n73pmms1lv0jbohyx1o1z13d615t99
> > Original Proposal for the Arrow project:
> > https://lists.apache.org/thread/x2qzdwglm8pkqp9gv03bbgw17khl7pq3
Re: [VOTE] Move Arrow DataFusion Subproject to new Top Level Apache Project
Posted by Sutou Kouhei <ko...@clear-code.com>.
+1
In <CA...@mail.gmail.com>
"[VOTE] Move Arrow DataFusion Subproject to new Top Level Apache Project" on Fri, 1 Mar 2024 06:33:08 -0500,
Andrew Lamb <al...@influxdata.com> wrote:
> Hello,
>
> As we have discussed[1][2] I would like to vote on the proposal to
> create a new Apache Top Level Project for DataFusion. The text of the
> proposed resolution and background document is copy/pasted below
>
> If the community is in favor of this, we plan to submit the resolution
> to the ASF board for approval with the next Arrow report (for the
> April 2024 board meeting).
>
> The vote will be open for at least 7 days.
>
> [ ] +1 Accept this Proposal
> [ ] +0
> [ ] -1 Do not accept this proposal because...
>
> Andrew
>
> [1] https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> [2] https://github.com/apache/arrow-datafusion/discussions/6475
>
> ---------- Proposed Resolution ---------
>
> Resolution to Create the Apache DataFusion Project from the Apache
> Arrow DataFusion Sub Project
>
> =============================================================
>
> X. Establish the Apache DataFusion Project
>
> WHEREAS, the Board of Directors deems it to be in the best
> interests of the Foundation and consistent with the
> Foundation's purpose to establish a Project Management
> Committee charged with the creation and maintenance of
> open-source software related to an extensible query engine
> for distribution at no charge to the public.
>
> NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> Committee (PMC), to be known as the "Apache DataFusion Project",
> be and hereby is established pursuant to Bylaws of the
> Foundation; and be it further
>
> RESOLVED, that the Apache DataFusion Project be and hereby is
> responsible for the creation and maintenance of software
> related to an extensible query engine; and be it further
>
> RESOLVED, that the office of "Vice President, Apache DataFusion" be
> and hereby is created, the person holding such office to
> serve at the direction of the Board of Directors as the chair
> of the Apache DataFusion Project, and to have primary responsibility
> for management of the projects within the scope of
> responsibility of the Apache DataFusion Project; and be it further
>
> RESOLVED, that the persons listed immediately below be and
> hereby are appointed to serve as the initial members of the
> Apache DataFusion Project:
>
> * Andy Grove (agrove@apache.org)
> * Andrew Lamb (alamb@apache.org)
> * Daniël Heres (dheres@apache.org)
> * Jie Wen (jakevin@apache.org)
> * Kun Liu (liukun@apache.org)
> * Liang-Chi Hsieh (viirya@apache.org)
> * Qingping Hou: (houqp@apache.org)
> * Wes McKinney(wesm@apache.org)
> * Will Jones (wjones127@apache.org)
>
> RESOLVED, that the Apache DataFusion Project be and hereby
> is tasked with the migration and rationalization of the Apache
> Arrow DataFusion sub-project; and be it further
>
> RESOLVED, that all responsibilities pertaining to the Apache
> Arrow DataFusion sub-project encumbered upon the
> Apache Arrow Project are hereafter discharged.
>
> NOW, THEREFORE, BE IT FURTHER RESOLVED, that Andrew Lamb
> be appointed to the office of Vice President, Apache DataFusion, to
> serve in accordance with and subject to the direction of the
> Board of Directors and the Bylaws of the Foundation until
> death, resignation, retirement, removal or disqualification,
> or until a successor is appointed.
> =============================================================
>
>
> -------
>
>
> Summary:
>
> We propose creating a new top level project, Apache DataFusion, from
> an existing sub project of Apache Arrow to facilitate additional
> community and project growth.
>
> Abstract
>
> Apache Arrow DataFusion[1] is a very fast, extensible query engine
> for building high-quality data-centric systems in Rust, using the
> Apache Arrow in-memory format. DataFusion offers SQL and Dataframe
> APIs, excellent performance, built-in support for CSV, Parquet, JSON,
> and Avro, extensive customization, and a great community.
>
> [1] https://arrow.apache.org/datafusion/
>
>
> Proposal
>
> We propose creating a new top level ASF project, Apache DataFusion,
> governed initially by a subset of the Apache Arrow project’s PMC and
> committers. The project’s code is in five existing git repositories,
> currently governed by Apache Arrow which would transfer to the new top
> level project.
>
> Background
>
> When DataFusion was initially donated to the Arrow project, it did not
> have a strong enough community to stand on its own. It has since grown
> significantly, and benefited immensely from being part of Arrow and
> nurturing of the Apache Way, and now has a community strong enough to
> stand on its own and that would benefit from focused governance
> attention.
>
> The community has discussed this idea publicly for more than 6 months
> https://github.com/apache/arrow-datafusion/discussions/6475 and
> briefly on the Arrow PMC mailing list
> https://lists.apache.org/thread/thv2jdm6640l6gm88hy8jhk5prjww0cs. As
> of the time of this writing both had exclusively positive reactions.
>
> Several current members of the Arrow PMC are both active contributors
> to DataFusion and understand and believe deeply in the Apache Way, and
> play active governance roles in the Arrow project as PMC members and
> PMC chairs, guiding the community, and releasing software versions.
> With this existing governance experience and structure, the new top
> level project will be able to function well immediately and
> independently.
>
> Overview of DataFusion
>
> Current Status
>
> Meritocracy
>
> DataFusion has been developed as part of Apache Arrow and thus has
> been operating as a meritocracy. Many of the developers of DataFusion
> are Arrow PMC members or committers. The DataFusion project plans to
> continue adding new PMC and committers as the project matures and
> grows.
>
> Community
>
> The DataFusion development team seeks to foster the development and
> user communities. We hope that becoming a separate project will help
> both Arrow and DataFusion communities by being more focused. Focused
> governance will make it easier to grow the community of committers and
> PMC members and make the organization more clear to others.
>
> Alignment
>
> The ASF is a natural host for DataFusion given that it is already the
> home of Arrow, Parquet, and other related distributed system, storage
> and query execution systems.
>
> Project Leadership
>
> Proposed Initial PMC
>
> We propose the following people as the initial DataFusion PMC members.
> This is a subset of the existing Arrow PMC members who contribute to
> DataFusion https://people.apache.org/phonebook.html?unix=arrow
>
> Andy Grove (agrove): Arrow PMC Chair
> Andrew Lamb (alamb): Arrow PMC, past Arrow PMC Chair
> Daniël Heres (dheres) Arrow PMC
> Jie Wen (jakevin): Arrow PMC, Doris Committer
> Kun Liu (liukun): Arrow PMC, IoTDB PMC, TSFile PMC
> Liang-Chi Hsieh (viirya): Arrow PMC, Spark PMC
> Qingping Hou: (houqp): Arrow PMC
> Wes McKinney(wesm): Arrow PMC, ASF Member
> Will Jones (wjones127): Arrow PMC
>
> We’d like to propose Andrew Lamb as the initial Chair, (and thus ASF
> VP) for the DataFusion project.
>
> Affiliations
>
> Andy Grove (agrove): NVidia
> Andrew Lamb (alamb): InfluxData
> Daniël Heres (dheres): Coralogix
> Jie Wen (jakevin): SelectDB
> Kun Liu (liukun): Ebay
> Liang-Chi Hsieh (viirya): Apple
> Qingping Hou: (houqp): Scribd
> Wes McKinney(wesm): Posit
> Will Jones (wjones127): LanceDB
>
> Proposed Initial Committers
>
> In addition to the PMC, we propose the following people as the initial
> DataFusion committers. This is a subset of the existing Arrow
> committers who contribute to DataFusion
> https://people.apache.org/phonebook.html?unix=arrow
>
> akurmustafa Mustafa Akur (Synnada)
> avantgardner Brent Gardner (Coralogix)
> comphead Oleks V. (Unaffiliated)
> jayzhan Jay Zhan (Unaffiliated)
> jeffreyvo Jeffry Vo (Unaffiliated)
> jiayuliu Liu Jiayu (Airbnb)
> mete Metehan Yildirim (Synnada)
> mingmwang Wang Mingming (Ebay)
> mneumann Marco Neumann (InfluxData)
> nju_yaho Zhong Yanghong (Ebay)
> ozankabak Mehmet Ozan Kabak (Synnada)
> paddyhoran Paddy Horan (Assured Allies)
> rdettai Rémi Dettai (Cloudfuse)
> sunchao Chao Sun (Apple)
> thinkharderdev Daniel Harris (Coralogix)
> tustvold Raphael Taylor-Davies (InfluxData)
> wayne Ruihang Xia (Greptime)
> xudong963 Xudong Wang (ByteDance)
> yjshen Yijie Shen (Space and Time)
> yangjiang Yang Jiang (ebay)
>
>
> Risk Assessments
>
> Naming / Trademarks
>
> As a sub-project of Arrow, the DataFusion name has been used for over
> 4 years without any known issues. A podling name search did not turn
> up any concerns and was approved:
> https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
>
> Legal / IP Clearance
>
> All DataFusion code has either been donated to the Arrow project with
> appropriate IP clearance or has been developed directly under ASF
> processes and procedures. Thus creating a new top level project poses
> no new Legal or IP risks.
>
> Code Extraction
>
> The relevant code is already in 5 separate repositories:
> https://github.com/apache/arrow-datafusion/
> https://github.com/apache/arrow-datafusion-python
> https://github.com/apache/arrow-ballista
> https://github.com/apache/arrow-ballista-python
> https://github.com/apache/arrow-datafusion-comet
>
> We foresee no issues with code extraction and propose these
> repositories be renamed to reflect top level projects
>
> Note: https://github.com/apache/arrow-rs, the Rust implementation of
> Arrow, would remain part of the Arrow project.
>
> Orphaned Products
>
> DataFusion is known to be used in many open source and commercial
> projects https://arrow.apache.org/datafusion/user-guide/introduction.html#known-users,
> has had multiple commits daily for several years, and its adoption and
> number of contributors appears to be growing. We do not foresee the
> project being orphaned in the next several years.
>
> Inexperience with Open Source
>
> The proposed PMC has extensive experience with Apache Arrow and other
> Apache projects, and includes PMC members, PMC chairs and an ASF
> Member. The DataFusion PMC and more experienced committers will
> continue to coach new community members who may be less familiar with
> the Apache Way.
>
> Homogeneous Developers
>
> The 9 proposed PMC members are from 9 different employers and the
> proposed committers are similarly distributed across affiliations. No
> specific entity employs more than 3 total proposed developers.
>
> Reliance on Salaried Developers
>
> A substantial amount of work on DataFusion has been by salaried
> developers, but it also has a long tradition of attracting
> contributions from students and hobbyists and we plan no changes in
> contribution structure.
>
> Relationships with Other Apache Products
>
> DataFusion will obviously have a strong relationship with the Arrow
> project given the overlap in people. We don’t foresee close
> collaboration with other projects at this time.
>
> Cryptography
>
> DataFusion does not directly support encryption and there are no
> near-term plans to add support for encryption. Users who need this
> functionality can use the extension APIs.
>
> Required Resources
>
> Mailing Lists
>
> - private@datafusion.apache.org for private PMC discussions (with
> moderated subscriptions)
> - dev@datafusion.apache.org
> - commits@datafusion.apache.org
> - user@datafusion.apache.org
>
> Version Control
>
> We propose to continue to use git for source control and github for
> hosting and testing resources.
>
> We also need to rename the github repositories to reflect the new top
> level names:
>
> https://github.com/apache/arrow-datafusion/ → apache/datafusion
> https://github.com/apache/arrow-datafusion-python → apache/datafusion-python
> https://github.com/apache/arrow-ballista → apache/datafusion-ballista
> https://github.com/apache/arrow-ballista-python →
> apache/datafusion-ballista-python
> https://github.com/apache/arrow-datafusion-comet → apache/datafusion-comet
>
>
>
> Issue Tracking
>
> DataFusion would continue to use github for its issue tracking and
> communications
>
> Other Resources
>
> The existing repositories already make use of existing Apache
> infrastructure, and we expect no change in the initial resource usage.
> As the project continues to grow, we expect continued infrastructure
> demand growth.
>
>
> FAQ: Has a sub project been promoted to a top level project before?
>
> Yes, and it appears to happen commonly. The Arrow project itself was
> created as a top level project from work that started in Apache Drill,
> and there are many sub projects of Hadoop that spun out as their own
> top level projects such as Mahout, Avro and HBase:
> https://news.apache.org/foundation/entry/the_apache_software_foundation_announces4
>
>
>
> Related material:
> Name search request / research for DataFusion:
> https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> Discussion about this proposal on the arrow mailing list:
> https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> Discussion about which repositories on the arrow mailing list:
> https://lists.apache.org/thread/ob3n0d9ky0bgrryl3xn39w9k566bq00q
> Discussion about initial PMC on the arrow mailing list:
> https://lists.apache.org/thread/pymrzcdw4qdptvby85f69rg3pcckl15b
> Discussion in github about creating a new DataFusion top level
> project: https://github.com/apache/arrow-datafusion/discussions/6475
> Discussion about graduating on incubator list:
> https://lists.apache.org/thread/r4n73pmms1lv0jbohyx1o1z13d615t99
> Original Proposal for the Arrow project:
> https://lists.apache.org/thread/x2qzdwglm8pkqp9gv03bbgw17khl7pq3
Re: [VOTE] Move Arrow DataFusion Subproject to new Top Level Apache Project
Posted by Will Jones <wi...@gmail.com>.
+1 (binding)
On Sun, Mar 3, 2024 at 09:43 Wayne Xia <wa...@gmail.com> wrote:
> +1 (non-binding)
>
> Regards,
> Wayne
>
> Julian Hyde <jh...@gmail.com> 于 2024年3月4日周一 上午1:38写道：
>
> > +1 (binding)
> >
> > > On Mar 2, 2024, at 2:28 PM, Dewey Dunnington
> > <de...@voltrondata.com.invalid> wrote:
> > >
> > > +1 (binding)
> > >
> > >> On Sat, Mar 2, 2024 at 8:08 AM vin jake <ja...@gmail.com> wrote:
> > >>
> > >> +1 (binding)
> > >>
> > >>> On Fri, Mar 1, 2024 at 7:33 PM Andrew Lamb <al...@influxdata.com>
> > wrote:
> > >>>
> > >>> Hello,
> > >>>
> > >>> As we have discussed[1][2] I would like to vote on the proposal to
> > >>> create a new Apache Top Level Project for DataFusion. The text of the
> > >>> proposed resolution and background document is copy/pasted below
> > >>>
> > >>> If the community is in favor of this, we plan to submit the
> resolution
> > >>> to the ASF board for approval with the next Arrow report (for the
> > >>> April 2024 board meeting).
> > >>>
> > >>> The vote will be open for at least 7 days.
> > >>>
> > >>> [ ] +1 Accept this Proposal
> > >>> [ ] +0
> > >>> [ ] -1 Do not accept this proposal because...
> > >>>
> > >>> Andrew
> > >>>
> > >>> [1] https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> > >>> [2] https://github.com/apache/arrow-datafusion/discussions/6475
> > >>>
> > >>> ---------- Proposed Resolution ---------
> > >>>
> > >>> Resolution to Create the Apache DataFusion Project from the Apache
> > >>> Arrow DataFusion Sub Project
> > >>>
> > >>> =============================================================
> > >>>
> > >>> X. Establish the Apache DataFusion Project
> > >>>
> > >>> WHEREAS, the Board of Directors deems it to be in the best
> > >>> interests of the Foundation and consistent with the
> > >>> Foundation's purpose to establish a Project Management
> > >>> Committee charged with the creation and maintenance of
> > >>> open-source software related to an extensible query engine
> > >>> for distribution at no charge to the public.
> > >>>
> > >>> NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> > >>> Committee (PMC), to be known as the "Apache DataFusion Project",
> > >>> be and hereby is established pursuant to Bylaws of the
> > >>> Foundation; and be it further
> > >>>
> > >>> RESOLVED, that the Apache DataFusion Project be and hereby is
> > >>> responsible for the creation and maintenance of software
> > >>> related to an extensible query engine; and be it further
> > >>>
> > >>> RESOLVED, that the office of "Vice President, Apache DataFusion" be
> > >>> and hereby is created, the person holding such office to
> > >>> serve at the direction of the Board of Directors as the chair
> > >>> of the Apache DataFusion Project, and to have primary responsibility
> > >>> for management of the projects within the scope of
> > >>> responsibility of the Apache DataFusion Project; and be it further
> > >>>
> > >>> RESOLVED, that the persons listed immediately below be and
> > >>> hereby are appointed to serve as the initial members of the
> > >>> Apache DataFusion Project:
> > >>>
> > >>> * Andy Grove (agrove@apache.org)
> > >>> * Andrew Lamb (alamb@apache.org)
> > >>> * Daniël Heres (dheres@apache.org)
> > >>> * Jie Wen (jakevin@apache.org)
> > >>> * Kun Liu (liukun@apache.org)
> > >>> * Liang-Chi Hsieh (viirya@apache.org)
> > >>> * Qingping Hou: (houqp@apache.org)
> > >>> * Wes McKinney(wesm@apache.org)
> > >>> * Will Jones (wjones127@apache.org)
> > >>>
> > >>> RESOLVED, that the Apache DataFusion Project be and hereby
> > >>> is tasked with the migration and rationalization of the Apache
> > >>> Arrow DataFusion sub-project; and be it further
> > >>>
> > >>> RESOLVED, that all responsibilities pertaining to the Apache
> > >>> Arrow DataFusion sub-project encumbered upon the
> > >>> Apache Arrow Project are hereafter discharged.
> > >>>
> > >>> NOW, THEREFORE, BE IT FURTHER RESOLVED, that Andrew Lamb
> > >>> be appointed to the office of Vice President, Apache DataFusion, to
> > >>> serve in accordance with and subject to the direction of the
> > >>> Board of Directors and the Bylaws of the Foundation until
> > >>> death, resignation, retirement, removal or disqualification,
> > >>> or until a successor is appointed.
> > >>> =============================================================
> > >>>
> > >>>
> > >>> -------
> > >>>
> > >>>
> > >>> Summary:
> > >>>
> > >>> We propose creating a new top level project, Apache DataFusion, from
> > >>> an existing sub project of Apache Arrow to facilitate additional
> > >>> community and project growth.
> > >>>
> > >>> Abstract
> > >>>
> > >>> Apache Arrow DataFusion[1] is a very fast, extensible query engine
> > >>> for building high-quality data-centric systems in Rust, using the
> > >>> Apache Arrow in-memory format. DataFusion offers SQL and Dataframe
> > >>> APIs, excellent performance, built-in support for CSV, Parquet, JSON,
> > >>> and Avro, extensive customization, and a great community.
> > >>>
> > >>> [1] https://arrow.apache.org/datafusion/
> > >>>
> > >>>
> > >>> Proposal
> > >>>
> > >>> We propose creating a new top level ASF project, Apache DataFusion,
> > >>> governed initially by a subset of the Apache Arrow project’s PMC and
> > >>> committers. The project’s code is in five existing git repositories,
> > >>> currently governed by Apache Arrow which would transfer to the new
> top
> > >>> level project.
> > >>>
> > >>> Background
> > >>>
> > >>> When DataFusion was initially donated to the Arrow project, it did
> not
> > >>> have a strong enough community to stand on its own. It has since
> grown
> > >>> significantly, and benefited immensely from being part of Arrow and
> > >>> nurturing of the Apache Way, and now has a community strong enough to
> > >>> stand on its own and that would benefit from focused governance
> > >>> attention.
> > >>>
> > >>> The community has discussed this idea publicly for more than 6 months
> > >>> https://github.com/apache/arrow-datafusion/discussions/6475 and
> > >>> briefly on the Arrow PMC mailing list
> > >>> https://lists.apache.org/thread/thv2jdm6640l6gm88hy8jhk5prjww0cs. As
> > >>> of the time of this writing both had exclusively positive reactions.
> > >>>
> > >>> Several current members of the Arrow PMC are both active contributors
> > >>> to DataFusion and understand and believe deeply in the Apache Way,
> and
> > >>> play active governance roles in the Arrow project as PMC members and
> > >>> PMC chairs, guiding the community, and releasing software versions.
> > >>> With this existing governance experience and structure, the new top
> > >>> level project will be able to function well immediately and
> > >>> independently.
> > >>>
> > >>> Overview of DataFusion
> > >>>
> > >>> Current Status
> > >>>
> > >>> Meritocracy
> > >>>
> > >>> DataFusion has been developed as part of Apache Arrow and thus has
> > >>> been operating as a meritocracy. Many of the developers of DataFusion
> > >>> are Arrow PMC members or committers. The DataFusion project plans to
> > >>> continue adding new PMC and committers as the project matures and
> > >>> grows.
> > >>>
> > >>> Community
> > >>>
> > >>> The DataFusion development team seeks to foster the development and
> > >>> user communities. We hope that becoming a separate project will help
> > >>> both Arrow and DataFusion communities by being more focused. Focused
> > >>> governance will make it easier to grow the community of committers
> and
> > >>> PMC members and make the organization more clear to others.
> > >>>
> > >>> Alignment
> > >>>
> > >>> The ASF is a natural host for DataFusion given that it is already the
> > >>> home of Arrow, Parquet, and other related distributed system, storage
> > >>> and query execution systems.
> > >>>
> > >>> Project Leadership
> > >>>
> > >>> Proposed Initial PMC
> > >>>
> > >>> We propose the following people as the initial DataFusion PMC
> members.
> > >>> This is a subset of the existing Arrow PMC members who contribute to
> > >>> DataFusion https://people.apache.org/phonebook.html?unix=arrow
> > >>>
> > >>> Andy Grove (agrove): Arrow PMC Chair
> > >>> Andrew Lamb (alamb): Arrow PMC, past Arrow PMC Chair
> > >>> Daniël Heres (dheres) Arrow PMC
> > >>> Jie Wen (jakevin): Arrow PMC, Doris Committer
> > >>> Kun Liu (liukun): Arrow PMC, IoTDB PMC, TSFile PMC
> > >>> Liang-Chi Hsieh (viirya): Arrow PMC, Spark PMC
> > >>> Qingping Hou: (houqp): Arrow PMC
> > >>> Wes McKinney(wesm): Arrow PMC, ASF Member
> > >>> Will Jones (wjones127): Arrow PMC
> > >>>
> > >>> We’d like to propose Andrew Lamb as the initial Chair, (and thus ASF
> > >>> VP) for the DataFusion project.
> > >>>
> > >>> Affiliations
> > >>>
> > >>> Andy Grove (agrove): NVidia
> > >>> Andrew Lamb (alamb): InfluxData
> > >>> Daniël Heres (dheres): Coralogix
> > >>> Jie Wen (jakevin): SelectDB
> > >>> Kun Liu (liukun): Ebay
> > >>> Liang-Chi Hsieh (viirya): Apple
> > >>> Qingping Hou: (houqp): Scribd
> > >>> Wes McKinney(wesm): Posit
> > >>> Will Jones (wjones127): LanceDB
> > >>>
> > >>> Proposed Initial Committers
> > >>>
> > >>> In addition to the PMC, we propose the following people as the
> initial
> > >>> DataFusion committers. This is a subset of the existing Arrow
> > >>> committers who contribute to DataFusion
> > >>> https://people.apache.org/phonebook.html?unix=arrow
> > >>>
> > >>> akurmustafa Mustafa Akur (Synnada)
> > >>> avantgardner Brent Gardner (Coralogix)
> > >>> comphead Oleks V. (Unaffiliated)
> > >>> jayzhan Jay Zhan (Unaffiliated)
> > >>> jeffreyvo Jeffry Vo (Unaffiliated)
> > >>> jiayuliu Liu Jiayu (Airbnb)
> > >>> mete Metehan Yildirim (Synnada)
> > >>> mingmwang Wang Mingming (Ebay)
> > >>> mneumann Marco Neumann (InfluxData)
> > >>> nju_yaho Zhong Yanghong (Ebay)
> > >>> ozankabak Mehmet Ozan Kabak (Synnada)
> > >>> paddyhoran Paddy Horan (Assured Allies)
> > >>> rdettai Rémi Dettai (Cloudfuse)
> > >>> sunchao Chao Sun (Apple)
> > >>> thinkharderdev Daniel Harris (Coralogix)
> > >>> tustvold Raphael Taylor-Davies (InfluxData)
> > >>> wayne Ruihang Xia (Greptime)
> > >>> xudong963 Xudong Wang (ByteDance)
> > >>> yjshen Yijie Shen (Space and Time)
> > >>> yangjiang Yang Jiang (ebay)
> > >>>
> > >>>
> > >>> Risk Assessments
> > >>>
> > >>> Naming / Trademarks
> > >>>
> > >>> As a sub-project of Arrow, the DataFusion name has been used for over
> > >>> 4 years without any known issues. A podling name search did not turn
> > >>> up any concerns and was approved:
> > >>> https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> > >>>
> > >>> Legal / IP Clearance
> > >>>
> > >>> All DataFusion code has either been donated to the Arrow project with
> > >>> appropriate IP clearance or has been developed directly under ASF
> > >>> processes and procedures. Thus creating a new top level project poses
> > >>> no new Legal or IP risks.
> > >>>
> > >>> Code Extraction
> > >>>
> > >>> The relevant code is already in 5 separate repositories:
> > >>> https://github.com/apache/arrow-datafusion/
> > >>> https://github.com/apache/arrow-datafusion-python
> > >>> https://github.com/apache/arrow-ballista
> > >>> https://github.com/apache/arrow-ballista-python
> > >>> https://github.com/apache/arrow-datafusion-comet
> > >>>
> > >>> We foresee no issues with code extraction and propose these
> > >>> repositories be renamed to reflect top level projects
> > >>>
> > >>> Note: https://github.com/apache/arrow-rs, the Rust implementation
> of
> > >>> Arrow, would remain part of the Arrow project.
> > >>>
> > >>> Orphaned Products
> > >>>
> > >>> DataFusion is known to be used in many open source and commercial
> > >>> projects
> > >>>
> >
> https://arrow.apache.org/datafusion/user-guide/introduction.html#known-users
> > >>> ,
> > >>> has had multiple commits daily for several years, and its adoption
> and
> > >>> number of contributors appears to be growing. We do not foresee the
> > >>> project being orphaned in the next several years.
> > >>>
> > >>> Inexperience with Open Source
> > >>>
> > >>> The proposed PMC has extensive experience with Apache Arrow and other
> > >>> Apache projects, and includes PMC members, PMC chairs and an ASF
> > >>> Member. The DataFusion PMC and more experienced committers will
> > >>> continue to coach new community members who may be less familiar with
> > >>> the Apache Way.
> > >>>
> > >>> Homogeneous Developers
> > >>>
> > >>> The 9 proposed PMC members are from 9 different employers and the
> > >>> proposed committers are similarly distributed across affiliations. No
> > >>> specific entity employs more than 3 total proposed developers.
> > >>>
> > >>> Reliance on Salaried Developers
> > >>>
> > >>> A substantial amount of work on DataFusion has been by salaried
> > >>> developers, but it also has a long tradition of attracting
> > >>> contributions from students and hobbyists and we plan no changes in
> > >>> contribution structure.
> > >>>
> > >>> Relationships with Other Apache Products
> > >>>
> > >>> DataFusion will obviously have a strong relationship with the Arrow
> > >>> project given the overlap in people. We don’t foresee close
> > >>> collaboration with other projects at this time.
> > >>>
> > >>> Cryptography
> > >>>
> > >>> DataFusion does not directly support encryption and there are no
> > >>> near-term plans to add support for encryption. Users who need this
> > >>> functionality can use the extension APIs.
> > >>>
> > >>> Required Resources
> > >>>
> > >>> Mailing Lists
> > >>>
> > >>> - private@datafusion.apache.org for private PMC discussions (with
> > >>> moderated subscriptions)
> > >>> - dev@datafusion.apache.org
> > >>> - commits@datafusion.apache.org
> > >>> - user@datafusion.apache.org
> > >>>
> > >>> Version Control
> > >>>
> > >>> We propose to continue to use git for source control and github for
> > >>> hosting and testing resources.
> > >>>
> > >>> We also need to rename the github repositories to reflect the new top
> > >>> level names:
> > >>>
> > >>> https://github.com/apache/arrow-datafusion/ → apache/datafusion
> > >>> https://github.com/apache/arrow-datafusion-python →
> > >>> apache/datafusion-python
> > >>> https://github.com/apache/arrow-ballista →
> apache/datafusion-ballista
> > >>> https://github.com/apache/arrow-ballista-python →
> > >>> apache/datafusion-ballista-python
> > >>> https://github.com/apache/arrow-datafusion-comet →
> > apache/datafusion-comet
> > >>>
> > >>>
> > >>>
> > >>> Issue Tracking
> > >>>
> > >>> DataFusion would continue to use github for its issue tracking and
> > >>> communications
> > >>>
> > >>> Other Resources
> > >>>
> > >>> The existing repositories already make use of existing Apache
> > >>> infrastructure, and we expect no change in the initial resource
> usage.
> > >>> As the project continues to grow, we expect continued infrastructure
> > >>> demand growth.
> > >>>
> > >>>
> > >>> FAQ: Has a sub project been promoted to a top level project before?
> > >>>
> > >>> Yes, and it appears to happen commonly. The Arrow project itself was
> > >>> created as a top level project from work that started in Apache
> Drill,
> > >>> and there are many sub projects of Hadoop that spun out as their own
> > >>> top level projects such as Mahout, Avro and HBase:
> > >>>
> > >>>
> >
> https://news.apache.org/foundation/entry/the_apache_software_foundation_announces4
> > >>>
> > >>>
> > >>>
> > >>> Related material:
> > >>> Name search request / research for DataFusion:
> > >>> https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> > >>> Discussion about this proposal on the arrow mailing list:
> > >>> https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> > >>> Discussion about which repositories on the arrow mailing list:
> > >>> https://lists.apache.org/thread/ob3n0d9ky0bgrryl3xn39w9k566bq00q
> > >>> Discussion about initial PMC on the arrow mailing list:
> > >>> https://lists.apache.org/thread/pymrzcdw4qdptvby85f69rg3pcckl15b
> > >>> Discussion in github about creating a new DataFusion top level
> > >>> project: https://github.com/apache/arrow-datafusion/discussions/6475
> > >>> Discussion about graduating on incubator list:
> > >>> https://lists.apache.org/thread/r4n73pmms1lv0jbohyx1o1z13d615t99
> > >>> Original Proposal for the Arrow project:
> > >>> https://lists.apache.org/thread/x2qzdwglm8pkqp9gv03bbgw17khl7pq3
> > >>>
> >
>
Re: [VOTE] Move Arrow DataFusion Subproject to new Top Level Apache Project
Posted by Wayne Xia <wa...@gmail.com>.
+1 (non-binding)
Regards,
Wayne
Julian Hyde <jh...@gmail.com> 于 2024年3月4日周一 上午1:38写道：
> +1 (binding)
>
> > On Mar 2, 2024, at 2:28 PM, Dewey Dunnington
> <de...@voltrondata.com.invalid> wrote:
> >
> > +1 (binding)
> >
> >> On Sat, Mar 2, 2024 at 8:08 AM vin jake <ja...@gmail.com> wrote:
> >>
> >> +1 (binding)
> >>
> >>> On Fri, Mar 1, 2024 at 7:33 PM Andrew Lamb <al...@influxdata.com>
> wrote:
> >>>
> >>> Hello,
> >>>
> >>> As we have discussed[1][2] I would like to vote on the proposal to
> >>> create a new Apache Top Level Project for DataFusion. The text of the
> >>> proposed resolution and background document is copy/pasted below
> >>>
> >>> If the community is in favor of this, we plan to submit the resolution
> >>> to the ASF board for approval with the next Arrow report (for the
> >>> April 2024 board meeting).
> >>>
> >>> The vote will be open for at least 7 days.
> >>>
> >>> [ ] +1 Accept this Proposal
> >>> [ ] +0
> >>> [ ] -1 Do not accept this proposal because...
> >>>
> >>> Andrew
> >>>
> >>> [1] https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> >>> [2] https://github.com/apache/arrow-datafusion/discussions/6475
> >>>
> >>> ---------- Proposed Resolution ---------
> >>>
> >>> Resolution to Create the Apache DataFusion Project from the Apache
> >>> Arrow DataFusion Sub Project
> >>>
> >>> =============================================================
> >>>
> >>> X. Establish the Apache DataFusion Project
> >>>
> >>> WHEREAS, the Board of Directors deems it to be in the best
> >>> interests of the Foundation and consistent with the
> >>> Foundation's purpose to establish a Project Management
> >>> Committee charged with the creation and maintenance of
> >>> open-source software related to an extensible query engine
> >>> for distribution at no charge to the public.
> >>>
> >>> NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> >>> Committee (PMC), to be known as the "Apache DataFusion Project",
> >>> be and hereby is established pursuant to Bylaws of the
> >>> Foundation; and be it further
> >>>
> >>> RESOLVED, that the Apache DataFusion Project be and hereby is
> >>> responsible for the creation and maintenance of software
> >>> related to an extensible query engine; and be it further
> >>>
> >>> RESOLVED, that the office of "Vice President, Apache DataFusion" be
> >>> and hereby is created, the person holding such office to
> >>> serve at the direction of the Board of Directors as the chair
> >>> of the Apache DataFusion Project, and to have primary responsibility
> >>> for management of the projects within the scope of
> >>> responsibility of the Apache DataFusion Project; and be it further
> >>>
> >>> RESOLVED, that the persons listed immediately below be and
> >>> hereby are appointed to serve as the initial members of the
> >>> Apache DataFusion Project:
> >>>
> >>> * Andy Grove (agrove@apache.org)
> >>> * Andrew Lamb (alamb@apache.org)
> >>> * Daniël Heres (dheres@apache.org)
> >>> * Jie Wen (jakevin@apache.org)
> >>> * Kun Liu (liukun@apache.org)
> >>> * Liang-Chi Hsieh (viirya@apache.org)
> >>> * Qingping Hou: (houqp@apache.org)
> >>> * Wes McKinney(wesm@apache.org)
> >>> * Will Jones (wjones127@apache.org)
> >>>
> >>> RESOLVED, that the Apache DataFusion Project be and hereby
> >>> is tasked with the migration and rationalization of the Apache
> >>> Arrow DataFusion sub-project; and be it further
> >>>
> >>> RESOLVED, that all responsibilities pertaining to the Apache
> >>> Arrow DataFusion sub-project encumbered upon the
> >>> Apache Arrow Project are hereafter discharged.
> >>>
> >>> NOW, THEREFORE, BE IT FURTHER RESOLVED, that Andrew Lamb
> >>> be appointed to the office of Vice President, Apache DataFusion, to
> >>> serve in accordance with and subject to the direction of the
> >>> Board of Directors and the Bylaws of the Foundation until
> >>> death, resignation, retirement, removal or disqualification,
> >>> or until a successor is appointed.
> >>> =============================================================
> >>>
> >>>
> >>> -------
> >>>
> >>>
> >>> Summary:
> >>>
> >>> We propose creating a new top level project, Apache DataFusion, from
> >>> an existing sub project of Apache Arrow to facilitate additional
> >>> community and project growth.
> >>>
> >>> Abstract
> >>>
> >>> Apache Arrow DataFusion[1] is a very fast, extensible query engine
> >>> for building high-quality data-centric systems in Rust, using the
> >>> Apache Arrow in-memory format. DataFusion offers SQL and Dataframe
> >>> APIs, excellent performance, built-in support for CSV, Parquet, JSON,
> >>> and Avro, extensive customization, and a great community.
> >>>
> >>> [1] https://arrow.apache.org/datafusion/
> >>>
> >>>
> >>> Proposal
> >>>
> >>> We propose creating a new top level ASF project, Apache DataFusion,
> >>> governed initially by a subset of the Apache Arrow project’s PMC and
> >>> committers. The project’s code is in five existing git repositories,
> >>> currently governed by Apache Arrow which would transfer to the new top
> >>> level project.
> >>>
> >>> Background
> >>>
> >>> When DataFusion was initially donated to the Arrow project, it did not
> >>> have a strong enough community to stand on its own. It has since grown
> >>> significantly, and benefited immensely from being part of Arrow and
> >>> nurturing of the Apache Way, and now has a community strong enough to
> >>> stand on its own and that would benefit from focused governance
> >>> attention.
> >>>
> >>> The community has discussed this idea publicly for more than 6 months
> >>> https://github.com/apache/arrow-datafusion/discussions/6475 and
> >>> briefly on the Arrow PMC mailing list
> >>> https://lists.apache.org/thread/thv2jdm6640l6gm88hy8jhk5prjww0cs. As
> >>> of the time of this writing both had exclusively positive reactions.
> >>>
> >>> Several current members of the Arrow PMC are both active contributors
> >>> to DataFusion and understand and believe deeply in the Apache Way, and
> >>> play active governance roles in the Arrow project as PMC members and
> >>> PMC chairs, guiding the community, and releasing software versions.
> >>> With this existing governance experience and structure, the new top
> >>> level project will be able to function well immediately and
> >>> independently.
> >>>
> >>> Overview of DataFusion
> >>>
> >>> Current Status
> >>>
> >>> Meritocracy
> >>>
> >>> DataFusion has been developed as part of Apache Arrow and thus has
> >>> been operating as a meritocracy. Many of the developers of DataFusion
> >>> are Arrow PMC members or committers. The DataFusion project plans to
> >>> continue adding new PMC and committers as the project matures and
> >>> grows.
> >>>
> >>> Community
> >>>
> >>> The DataFusion development team seeks to foster the development and
> >>> user communities. We hope that becoming a separate project will help
> >>> both Arrow and DataFusion communities by being more focused. Focused
> >>> governance will make it easier to grow the community of committers and
> >>> PMC members and make the organization more clear to others.
> >>>
> >>> Alignment
> >>>
> >>> The ASF is a natural host for DataFusion given that it is already the
> >>> home of Arrow, Parquet, and other related distributed system, storage
> >>> and query execution systems.
> >>>
> >>> Project Leadership
> >>>
> >>> Proposed Initial PMC
> >>>
> >>> We propose the following people as the initial DataFusion PMC members.
> >>> This is a subset of the existing Arrow PMC members who contribute to
> >>> DataFusion https://people.apache.org/phonebook.html?unix=arrow
> >>>
> >>> Andy Grove (agrove): Arrow PMC Chair
> >>> Andrew Lamb (alamb): Arrow PMC, past Arrow PMC Chair
> >>> Daniël Heres (dheres) Arrow PMC
> >>> Jie Wen (jakevin): Arrow PMC, Doris Committer
> >>> Kun Liu (liukun): Arrow PMC, IoTDB PMC, TSFile PMC
> >>> Liang-Chi Hsieh (viirya): Arrow PMC, Spark PMC
> >>> Qingping Hou: (houqp): Arrow PMC
> >>> Wes McKinney(wesm): Arrow PMC, ASF Member
> >>> Will Jones (wjones127): Arrow PMC
> >>>
> >>> We’d like to propose Andrew Lamb as the initial Chair, (and thus ASF
> >>> VP) for the DataFusion project.
> >>>
> >>> Affiliations
> >>>
> >>> Andy Grove (agrove): NVidia
> >>> Andrew Lamb (alamb): InfluxData
> >>> Daniël Heres (dheres): Coralogix
> >>> Jie Wen (jakevin): SelectDB
> >>> Kun Liu (liukun): Ebay
> >>> Liang-Chi Hsieh (viirya): Apple
> >>> Qingping Hou: (houqp): Scribd
> >>> Wes McKinney(wesm): Posit
> >>> Will Jones (wjones127): LanceDB
> >>>
> >>> Proposed Initial Committers
> >>>
> >>> In addition to the PMC, we propose the following people as the initial
> >>> DataFusion committers. This is a subset of the existing Arrow
> >>> committers who contribute to DataFusion
> >>> https://people.apache.org/phonebook.html?unix=arrow
> >>>
> >>> akurmustafa Mustafa Akur (Synnada)
> >>> avantgardner Brent Gardner (Coralogix)
> >>> comphead Oleks V. (Unaffiliated)
> >>> jayzhan Jay Zhan (Unaffiliated)
> >>> jeffreyvo Jeffry Vo (Unaffiliated)
> >>> jiayuliu Liu Jiayu (Airbnb)
> >>> mete Metehan Yildirim (Synnada)
> >>> mingmwang Wang Mingming (Ebay)
> >>> mneumann Marco Neumann (InfluxData)
> >>> nju_yaho Zhong Yanghong (Ebay)
> >>> ozankabak Mehmet Ozan Kabak (Synnada)
> >>> paddyhoran Paddy Horan (Assured Allies)
> >>> rdettai Rémi Dettai (Cloudfuse)
> >>> sunchao Chao Sun (Apple)
> >>> thinkharderdev Daniel Harris (Coralogix)
> >>> tustvold Raphael Taylor-Davies (InfluxData)
> >>> wayne Ruihang Xia (Greptime)
> >>> xudong963 Xudong Wang (ByteDance)
> >>> yjshen Yijie Shen (Space and Time)
> >>> yangjiang Yang Jiang (ebay)
> >>>
> >>>
> >>> Risk Assessments
> >>>
> >>> Naming / Trademarks
> >>>
> >>> As a sub-project of Arrow, the DataFusion name has been used for over
> >>> 4 years without any known issues. A podling name search did not turn
> >>> up any concerns and was approved:
> >>> https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> >>>
> >>> Legal / IP Clearance
> >>>
> >>> All DataFusion code has either been donated to the Arrow project with
> >>> appropriate IP clearance or has been developed directly under ASF
> >>> processes and procedures. Thus creating a new top level project poses
> >>> no new Legal or IP risks.
> >>>
> >>> Code Extraction
> >>>
> >>> The relevant code is already in 5 separate repositories:
> >>> https://github.com/apache/arrow-datafusion/
> >>> https://github.com/apache/arrow-datafusion-python
> >>> https://github.com/apache/arrow-ballista
> >>> https://github.com/apache/arrow-ballista-python
> >>> https://github.com/apache/arrow-datafusion-comet
> >>>
> >>> We foresee no issues with code extraction and propose these
> >>> repositories be renamed to reflect top level projects
> >>>
> >>> Note: https://github.com/apache/arrow-rs, the Rust implementation of
> >>> Arrow, would remain part of the Arrow project.
> >>>
> >>> Orphaned Products
> >>>
> >>> DataFusion is known to be used in many open source and commercial
> >>> projects
> >>>
> https://arrow.apache.org/datafusion/user-guide/introduction.html#known-users
> >>> ,
> >>> has had multiple commits daily for several years, and its adoption and
> >>> number of contributors appears to be growing. We do not foresee the
> >>> project being orphaned in the next several years.
> >>>
> >>> Inexperience with Open Source
> >>>
> >>> The proposed PMC has extensive experience with Apache Arrow and other
> >>> Apache projects, and includes PMC members, PMC chairs and an ASF
> >>> Member. The DataFusion PMC and more experienced committers will
> >>> continue to coach new community members who may be less familiar with
> >>> the Apache Way.
> >>>
> >>> Homogeneous Developers
> >>>
> >>> The 9 proposed PMC members are from 9 different employers and the
> >>> proposed committers are similarly distributed across affiliations. No
> >>> specific entity employs more than 3 total proposed developers.
> >>>
> >>> Reliance on Salaried Developers
> >>>
> >>> A substantial amount of work on DataFusion has been by salaried
> >>> developers, but it also has a long tradition of attracting
> >>> contributions from students and hobbyists and we plan no changes in
> >>> contribution structure.
> >>>
> >>> Relationships with Other Apache Products
> >>>
> >>> DataFusion will obviously have a strong relationship with the Arrow
> >>> project given the overlap in people. We don’t foresee close
> >>> collaboration with other projects at this time.
> >>>
> >>> Cryptography
> >>>
> >>> DataFusion does not directly support encryption and there are no
> >>> near-term plans to add support for encryption. Users who need this
> >>> functionality can use the extension APIs.
> >>>
> >>> Required Resources
> >>>
> >>> Mailing Lists
> >>>
> >>> - private@datafusion.apache.org for private PMC discussions (with
> >>> moderated subscriptions)
> >>> - dev@datafusion.apache.org
> >>> - commits@datafusion.apache.org
> >>> - user@datafusion.apache.org
> >>>
> >>> Version Control
> >>>
> >>> We propose to continue to use git for source control and github for
> >>> hosting and testing resources.
> >>>
> >>> We also need to rename the github repositories to reflect the new top
> >>> level names:
> >>>
> >>> https://github.com/apache/arrow-datafusion/ → apache/datafusion
> >>> https://github.com/apache/arrow-datafusion-python →
> >>> apache/datafusion-python
> >>> https://github.com/apache/arrow-ballista → apache/datafusion-ballista
> >>> https://github.com/apache/arrow-ballista-python →
> >>> apache/datafusion-ballista-python
> >>> https://github.com/apache/arrow-datafusion-comet →
> apache/datafusion-comet
> >>>
> >>>
> >>>
> >>> Issue Tracking
> >>>
> >>> DataFusion would continue to use github for its issue tracking and
> >>> communications
> >>>
> >>> Other Resources
> >>>
> >>> The existing repositories already make use of existing Apache
> >>> infrastructure, and we expect no change in the initial resource usage.
> >>> As the project continues to grow, we expect continued infrastructure
> >>> demand growth.
> >>>
> >>>
> >>> FAQ: Has a sub project been promoted to a top level project before?
> >>>
> >>> Yes, and it appears to happen commonly. The Arrow project itself was
> >>> created as a top level project from work that started in Apache Drill,
> >>> and there are many sub projects of Hadoop that spun out as their own
> >>> top level projects such as Mahout, Avro and HBase:
> >>>
> >>>
> https://news.apache.org/foundation/entry/the_apache_software_foundation_announces4
> >>>
> >>>
> >>>
> >>> Related material:
> >>> Name search request / research for DataFusion:
> >>> https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> >>> Discussion about this proposal on the arrow mailing list:
> >>> https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> >>> Discussion about which repositories on the arrow mailing list:
> >>> https://lists.apache.org/thread/ob3n0d9ky0bgrryl3xn39w9k566bq00q
> >>> Discussion about initial PMC on the arrow mailing list:
> >>> https://lists.apache.org/thread/pymrzcdw4qdptvby85f69rg3pcckl15b
> >>> Discussion in github about creating a new DataFusion top level
> >>> project: https://github.com/apache/arrow-datafusion/discussions/6475
> >>> Discussion about graduating on incubator list:
> >>> https://lists.apache.org/thread/r4n73pmms1lv0jbohyx1o1z13d615t99
> >>> Original Proposal for the Arrow project:
> >>> https://lists.apache.org/thread/x2qzdwglm8pkqp9gv03bbgw17khl7pq3
> >>>
>
Re: [VOTE] Move Arrow DataFusion Subproject to new Top Level Apache Project
Posted by Julian Hyde <jh...@gmail.com>.
+1 (binding)
> On Mar 2, 2024, at 2:28 PM, Dewey Dunnington <de...@voltrondata.com.invalid> wrote:
>
> +1 (binding)
>
>> On Sat, Mar 2, 2024 at 8:08 AM vin jake <ja...@gmail.com> wrote:
>>
>> +1 (binding)
>>
>>> On Fri, Mar 1, 2024 at 7:33 PM Andrew Lamb <al...@influxdata.com> wrote:
>>>
>>> Hello,
>>>
>>> As we have discussed[1][2] I would like to vote on the proposal to
>>> create a new Apache Top Level Project for DataFusion. The text of the
>>> proposed resolution and background document is copy/pasted below
>>>
>>> If the community is in favor of this, we plan to submit the resolution
>>> to the ASF board for approval with the next Arrow report (for the
>>> April 2024 board meeting).
>>>
>>> The vote will be open for at least 7 days.
>>>
>>> [ ] +1 Accept this Proposal
>>> [ ] +0
>>> [ ] -1 Do not accept this proposal because...
>>>
>>> Andrew
>>>
>>> [1] https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
>>> [2] https://github.com/apache/arrow-datafusion/discussions/6475
>>>
>>> ---------- Proposed Resolution ---------
>>>
>>> Resolution to Create the Apache DataFusion Project from the Apache
>>> Arrow DataFusion Sub Project
>>>
>>> =============================================================
>>>
>>> X. Establish the Apache DataFusion Project
>>>
>>> WHEREAS, the Board of Directors deems it to be in the best
>>> interests of the Foundation and consistent with the
>>> Foundation's purpose to establish a Project Management
>>> Committee charged with the creation and maintenance of
>>> open-source software related to an extensible query engine
>>> for distribution at no charge to the public.
>>>
>>> NOW, THEREFORE, BE IT RESOLVED, that a Project Management
>>> Committee (PMC), to be known as the "Apache DataFusion Project",
>>> be and hereby is established pursuant to Bylaws of the
>>> Foundation; and be it further
>>>
>>> RESOLVED, that the Apache DataFusion Project be and hereby is
>>> responsible for the creation and maintenance of software
>>> related to an extensible query engine; and be it further
>>>
>>> RESOLVED, that the office of "Vice President, Apache DataFusion" be
>>> and hereby is created, the person holding such office to
>>> serve at the direction of the Board of Directors as the chair
>>> of the Apache DataFusion Project, and to have primary responsibility
>>> for management of the projects within the scope of
>>> responsibility of the Apache DataFusion Project; and be it further
>>>
>>> RESOLVED, that the persons listed immediately below be and
>>> hereby are appointed to serve as the initial members of the
>>> Apache DataFusion Project:
>>>
>>> * Andy Grove (agrove@apache.org)
>>> * Andrew Lamb (alamb@apache.org)
>>> * Daniël Heres (dheres@apache.org)
>>> * Jie Wen (jakevin@apache.org)
>>> * Kun Liu (liukun@apache.org)
>>> * Liang-Chi Hsieh (viirya@apache.org)
>>> * Qingping Hou: (houqp@apache.org)
>>> * Wes McKinney(wesm@apache.org)
>>> * Will Jones (wjones127@apache.org)
>>>
>>> RESOLVED, that the Apache DataFusion Project be and hereby
>>> is tasked with the migration and rationalization of the Apache
>>> Arrow DataFusion sub-project; and be it further
>>>
>>> RESOLVED, that all responsibilities pertaining to the Apache
>>> Arrow DataFusion sub-project encumbered upon the
>>> Apache Arrow Project are hereafter discharged.
>>>
>>> NOW, THEREFORE, BE IT FURTHER RESOLVED, that Andrew Lamb
>>> be appointed to the office of Vice President, Apache DataFusion, to
>>> serve in accordance with and subject to the direction of the
>>> Board of Directors and the Bylaws of the Foundation until
>>> death, resignation, retirement, removal or disqualification,
>>> or until a successor is appointed.
>>> =============================================================
>>>
>>>
>>> -------
>>>
>>>
>>> Summary:
>>>
>>> We propose creating a new top level project, Apache DataFusion, from
>>> an existing sub project of Apache Arrow to facilitate additional
>>> community and project growth.
>>>
>>> Abstract
>>>
>>> Apache Arrow DataFusion[1] is a very fast, extensible query engine
>>> for building high-quality data-centric systems in Rust, using the
>>> Apache Arrow in-memory format. DataFusion offers SQL and Dataframe
>>> APIs, excellent performance, built-in support for CSV, Parquet, JSON,
>>> and Avro, extensive customization, and a great community.
>>>
>>> [1] https://arrow.apache.org/datafusion/
>>>
>>>
>>> Proposal
>>>
>>> We propose creating a new top level ASF project, Apache DataFusion,
>>> governed initially by a subset of the Apache Arrow project’s PMC and
>>> committers. The project’s code is in five existing git repositories,
>>> currently governed by Apache Arrow which would transfer to the new top
>>> level project.
>>>
>>> Background
>>>
>>> When DataFusion was initially donated to the Arrow project, it did not
>>> have a strong enough community to stand on its own. It has since grown
>>> significantly, and benefited immensely from being part of Arrow and
>>> nurturing of the Apache Way, and now has a community strong enough to
>>> stand on its own and that would benefit from focused governance
>>> attention.
>>>
>>> The community has discussed this idea publicly for more than 6 months
>>> https://github.com/apache/arrow-datafusion/discussions/6475 and
>>> briefly on the Arrow PMC mailing list
>>> https://lists.apache.org/thread/thv2jdm6640l6gm88hy8jhk5prjww0cs. As
>>> of the time of this writing both had exclusively positive reactions.
>>>
>>> Several current members of the Arrow PMC are both active contributors
>>> to DataFusion and understand and believe deeply in the Apache Way, and
>>> play active governance roles in the Arrow project as PMC members and
>>> PMC chairs, guiding the community, and releasing software versions.
>>> With this existing governance experience and structure, the new top
>>> level project will be able to function well immediately and
>>> independently.
>>>
>>> Overview of DataFusion
>>>
>>> Current Status
>>>
>>> Meritocracy
>>>
>>> DataFusion has been developed as part of Apache Arrow and thus has
>>> been operating as a meritocracy. Many of the developers of DataFusion
>>> are Arrow PMC members or committers. The DataFusion project plans to
>>> continue adding new PMC and committers as the project matures and
>>> grows.
>>>
>>> Community
>>>
>>> The DataFusion development team seeks to foster the development and
>>> user communities. We hope that becoming a separate project will help
>>> both Arrow and DataFusion communities by being more focused. Focused
>>> governance will make it easier to grow the community of committers and
>>> PMC members and make the organization more clear to others.
>>>
>>> Alignment
>>>
>>> The ASF is a natural host for DataFusion given that it is already the
>>> home of Arrow, Parquet, and other related distributed system, storage
>>> and query execution systems.
>>>
>>> Project Leadership
>>>
>>> Proposed Initial PMC
>>>
>>> We propose the following people as the initial DataFusion PMC members.
>>> This is a subset of the existing Arrow PMC members who contribute to
>>> DataFusion https://people.apache.org/phonebook.html?unix=arrow
>>>
>>> Andy Grove (agrove): Arrow PMC Chair
>>> Andrew Lamb (alamb): Arrow PMC, past Arrow PMC Chair
>>> Daniël Heres (dheres) Arrow PMC
>>> Jie Wen (jakevin): Arrow PMC, Doris Committer
>>> Kun Liu (liukun): Arrow PMC, IoTDB PMC, TSFile PMC
>>> Liang-Chi Hsieh (viirya): Arrow PMC, Spark PMC
>>> Qingping Hou: (houqp): Arrow PMC
>>> Wes McKinney(wesm): Arrow PMC, ASF Member
>>> Will Jones (wjones127): Arrow PMC
>>>
>>> We’d like to propose Andrew Lamb as the initial Chair, (and thus ASF
>>> VP) for the DataFusion project.
>>>
>>> Affiliations
>>>
>>> Andy Grove (agrove): NVidia
>>> Andrew Lamb (alamb): InfluxData
>>> Daniël Heres (dheres): Coralogix
>>> Jie Wen (jakevin): SelectDB
>>> Kun Liu (liukun): Ebay
>>> Liang-Chi Hsieh (viirya): Apple
>>> Qingping Hou: (houqp): Scribd
>>> Wes McKinney(wesm): Posit
>>> Will Jones (wjones127): LanceDB
>>>
>>> Proposed Initial Committers
>>>
>>> In addition to the PMC, we propose the following people as the initial
>>> DataFusion committers. This is a subset of the existing Arrow
>>> committers who contribute to DataFusion
>>> https://people.apache.org/phonebook.html?unix=arrow
>>>
>>> akurmustafa Mustafa Akur (Synnada)
>>> avantgardner Brent Gardner (Coralogix)
>>> comphead Oleks V. (Unaffiliated)
>>> jayzhan Jay Zhan (Unaffiliated)
>>> jeffreyvo Jeffry Vo (Unaffiliated)
>>> jiayuliu Liu Jiayu (Airbnb)
>>> mete Metehan Yildirim (Synnada)
>>> mingmwang Wang Mingming (Ebay)
>>> mneumann Marco Neumann (InfluxData)
>>> nju_yaho Zhong Yanghong (Ebay)
>>> ozankabak Mehmet Ozan Kabak (Synnada)
>>> paddyhoran Paddy Horan (Assured Allies)
>>> rdettai Rémi Dettai (Cloudfuse)
>>> sunchao Chao Sun (Apple)
>>> thinkharderdev Daniel Harris (Coralogix)
>>> tustvold Raphael Taylor-Davies (InfluxData)
>>> wayne Ruihang Xia (Greptime)
>>> xudong963 Xudong Wang (ByteDance)
>>> yjshen Yijie Shen (Space and Time)
>>> yangjiang Yang Jiang (ebay)
>>>
>>>
>>> Risk Assessments
>>>
>>> Naming / Trademarks
>>>
>>> As a sub-project of Arrow, the DataFusion name has been used for over
>>> 4 years without any known issues. A podling name search did not turn
>>> up any concerns and was approved:
>>> https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
>>>
>>> Legal / IP Clearance
>>>
>>> All DataFusion code has either been donated to the Arrow project with
>>> appropriate IP clearance or has been developed directly under ASF
>>> processes and procedures. Thus creating a new top level project poses
>>> no new Legal or IP risks.
>>>
>>> Code Extraction
>>>
>>> The relevant code is already in 5 separate repositories:
>>> https://github.com/apache/arrow-datafusion/
>>> https://github.com/apache/arrow-datafusion-python
>>> https://github.com/apache/arrow-ballista
>>> https://github.com/apache/arrow-ballista-python
>>> https://github.com/apache/arrow-datafusion-comet
>>>
>>> We foresee no issues with code extraction and propose these
>>> repositories be renamed to reflect top level projects
>>>
>>> Note: https://github.com/apache/arrow-rs, the Rust implementation of
>>> Arrow, would remain part of the Arrow project.
>>>
>>> Orphaned Products
>>>
>>> DataFusion is known to be used in many open source and commercial
>>> projects
>>> https://arrow.apache.org/datafusion/user-guide/introduction.html#known-users
>>> ,
>>> has had multiple commits daily for several years, and its adoption and
>>> number of contributors appears to be growing. We do not foresee the
>>> project being orphaned in the next several years.
>>>
>>> Inexperience with Open Source
>>>
>>> The proposed PMC has extensive experience with Apache Arrow and other
>>> Apache projects, and includes PMC members, PMC chairs and an ASF
>>> Member. The DataFusion PMC and more experienced committers will
>>> continue to coach new community members who may be less familiar with
>>> the Apache Way.
>>>
>>> Homogeneous Developers
>>>
>>> The 9 proposed PMC members are from 9 different employers and the
>>> proposed committers are similarly distributed across affiliations. No
>>> specific entity employs more than 3 total proposed developers.
>>>
>>> Reliance on Salaried Developers
>>>
>>> A substantial amount of work on DataFusion has been by salaried
>>> developers, but it also has a long tradition of attracting
>>> contributions from students and hobbyists and we plan no changes in
>>> contribution structure.
>>>
>>> Relationships with Other Apache Products
>>>
>>> DataFusion will obviously have a strong relationship with the Arrow
>>> project given the overlap in people. We don’t foresee close
>>> collaboration with other projects at this time.
>>>
>>> Cryptography
>>>
>>> DataFusion does not directly support encryption and there are no
>>> near-term plans to add support for encryption. Users who need this
>>> functionality can use the extension APIs.
>>>
>>> Required Resources
>>>
>>> Mailing Lists
>>>
>>> - private@datafusion.apache.org for private PMC discussions (with
>>> moderated subscriptions)
>>> - dev@datafusion.apache.org
>>> - commits@datafusion.apache.org
>>> - user@datafusion.apache.org
>>>
>>> Version Control
>>>
>>> We propose to continue to use git for source control and github for
>>> hosting and testing resources.
>>>
>>> We also need to rename the github repositories to reflect the new top
>>> level names:
>>>
>>> https://github.com/apache/arrow-datafusion/ → apache/datafusion
>>> https://github.com/apache/arrow-datafusion-python →
>>> apache/datafusion-python
>>> https://github.com/apache/arrow-ballista → apache/datafusion-ballista
>>> https://github.com/apache/arrow-ballista-python →
>>> apache/datafusion-ballista-python
>>> https://github.com/apache/arrow-datafusion-comet → apache/datafusion-comet
>>>
>>>
>>>
>>> Issue Tracking
>>>
>>> DataFusion would continue to use github for its issue tracking and
>>> communications
>>>
>>> Other Resources
>>>
>>> The existing repositories already make use of existing Apache
>>> infrastructure, and we expect no change in the initial resource usage.
>>> As the project continues to grow, we expect continued infrastructure
>>> demand growth.
>>>
>>>
>>> FAQ: Has a sub project been promoted to a top level project before?
>>>
>>> Yes, and it appears to happen commonly. The Arrow project itself was
>>> created as a top level project from work that started in Apache Drill,
>>> and there are many sub projects of Hadoop that spun out as their own
>>> top level projects such as Mahout, Avro and HBase:
>>>
>>> https://news.apache.org/foundation/entry/the_apache_software_foundation_announces4
>>>
>>>
>>>
>>> Related material:
>>> Name search request / research for DataFusion:
>>> https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
>>> Discussion about this proposal on the arrow mailing list:
>>> https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
>>> Discussion about which repositories on the arrow mailing list:
>>> https://lists.apache.org/thread/ob3n0d9ky0bgrryl3xn39w9k566bq00q
>>> Discussion about initial PMC on the arrow mailing list:
>>> https://lists.apache.org/thread/pymrzcdw4qdptvby85f69rg3pcckl15b
>>> Discussion in github about creating a new DataFusion top level
>>> project: https://github.com/apache/arrow-datafusion/discussions/6475
>>> Discussion about graduating on incubator list:
>>> https://lists.apache.org/thread/r4n73pmms1lv0jbohyx1o1z13d615t99
>>> Original Proposal for the Arrow project:
>>> https://lists.apache.org/thread/x2qzdwglm8pkqp9gv03bbgw17khl7pq3
>>>
Re: [VOTE] Move Arrow DataFusion Subproject to new Top Level Apache Project
Posted by Dewey Dunnington <de...@voltrondata.com.INVALID>.
+1 (binding)
On Sat, Mar 2, 2024 at 8:08 AM vin jake <ja...@gmail.com> wrote:
>
> +1 (binding)
>
> On Fri, Mar 1, 2024 at 7:33 PM Andrew Lamb <al...@influxdata.com> wrote:
>
> > Hello,
> >
> > As we have discussed[1][2] I would like to vote on the proposal to
> > create a new Apache Top Level Project for DataFusion. The text of the
> > proposed resolution and background document is copy/pasted below
> >
> > If the community is in favor of this, we plan to submit the resolution
> > to the ASF board for approval with the next Arrow report (for the
> > April 2024 board meeting).
> >
> > The vote will be open for at least 7 days.
> >
> > [ ] +1 Accept this Proposal
> > [ ] +0
> > [ ] -1 Do not accept this proposal because...
> >
> > Andrew
> >
> > [1] https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> > [2] https://github.com/apache/arrow-datafusion/discussions/6475
> >
> > ---------- Proposed Resolution ---------
> >
> > Resolution to Create the Apache DataFusion Project from the Apache
> > Arrow DataFusion Sub Project
> >
> > =============================================================
> >
> > X. Establish the Apache DataFusion Project
> >
> > WHEREAS, the Board of Directors deems it to be in the best
> > interests of the Foundation and consistent with the
> > Foundation's purpose to establish a Project Management
> > Committee charged with the creation and maintenance of
> > open-source software related to an extensible query engine
> > for distribution at no charge to the public.
> >
> > NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> > Committee (PMC), to be known as the "Apache DataFusion Project",
> > be and hereby is established pursuant to Bylaws of the
> > Foundation; and be it further
> >
> > RESOLVED, that the Apache DataFusion Project be and hereby is
> > responsible for the creation and maintenance of software
> > related to an extensible query engine; and be it further
> >
> > RESOLVED, that the office of "Vice President, Apache DataFusion" be
> > and hereby is created, the person holding such office to
> > serve at the direction of the Board of Directors as the chair
> > of the Apache DataFusion Project, and to have primary responsibility
> > for management of the projects within the scope of
> > responsibility of the Apache DataFusion Project; and be it further
> >
> > RESOLVED, that the persons listed immediately below be and
> > hereby are appointed to serve as the initial members of the
> > Apache DataFusion Project:
> >
> > * Andy Grove (agrove@apache.org)
> > * Andrew Lamb (alamb@apache.org)
> > * Daniël Heres (dheres@apache.org)
> > * Jie Wen (jakevin@apache.org)
> > * Kun Liu (liukun@apache.org)
> > * Liang-Chi Hsieh (viirya@apache.org)
> > * Qingping Hou: (houqp@apache.org)
> > * Wes McKinney(wesm@apache.org)
> > * Will Jones (wjones127@apache.org)
> >
> > RESOLVED, that the Apache DataFusion Project be and hereby
> > is tasked with the migration and rationalization of the Apache
> > Arrow DataFusion sub-project; and be it further
> >
> > RESOLVED, that all responsibilities pertaining to the Apache
> > Arrow DataFusion sub-project encumbered upon the
> > Apache Arrow Project are hereafter discharged.
> >
> > NOW, THEREFORE, BE IT FURTHER RESOLVED, that Andrew Lamb
> > be appointed to the office of Vice President, Apache DataFusion, to
> > serve in accordance with and subject to the direction of the
> > Board of Directors and the Bylaws of the Foundation until
> > death, resignation, retirement, removal or disqualification,
> > or until a successor is appointed.
> > =============================================================
> >
> >
> > -------
> >
> >
> > Summary:
> >
> > We propose creating a new top level project, Apache DataFusion, from
> > an existing sub project of Apache Arrow to facilitate additional
> > community and project growth.
> >
> > Abstract
> >
> > Apache Arrow DataFusion[1] is a very fast, extensible query engine
> > for building high-quality data-centric systems in Rust, using the
> > Apache Arrow in-memory format. DataFusion offers SQL and Dataframe
> > APIs, excellent performance, built-in support for CSV, Parquet, JSON,
> > and Avro, extensive customization, and a great community.
> >
> > [1] https://arrow.apache.org/datafusion/
> >
> >
> > Proposal
> >
> > We propose creating a new top level ASF project, Apache DataFusion,
> > governed initially by a subset of the Apache Arrow project’s PMC and
> > committers. The project’s code is in five existing git repositories,
> > currently governed by Apache Arrow which would transfer to the new top
> > level project.
> >
> > Background
> >
> > When DataFusion was initially donated to the Arrow project, it did not
> > have a strong enough community to stand on its own. It has since grown
> > significantly, and benefited immensely from being part of Arrow and
> > nurturing of the Apache Way, and now has a community strong enough to
> > stand on its own and that would benefit from focused governance
> > attention.
> >
> > The community has discussed this idea publicly for more than 6 months
> > https://github.com/apache/arrow-datafusion/discussions/6475 and
> > briefly on the Arrow PMC mailing list
> > https://lists.apache.org/thread/thv2jdm6640l6gm88hy8jhk5prjww0cs. As
> > of the time of this writing both had exclusively positive reactions.
> >
> > Several current members of the Arrow PMC are both active contributors
> > to DataFusion and understand and believe deeply in the Apache Way, and
> > play active governance roles in the Arrow project as PMC members and
> > PMC chairs, guiding the community, and releasing software versions.
> > With this existing governance experience and structure, the new top
> > level project will be able to function well immediately and
> > independently.
> >
> > Overview of DataFusion
> >
> > Current Status
> >
> > Meritocracy
> >
> > DataFusion has been developed as part of Apache Arrow and thus has
> > been operating as a meritocracy. Many of the developers of DataFusion
> > are Arrow PMC members or committers. The DataFusion project plans to
> > continue adding new PMC and committers as the project matures and
> > grows.
> >
> > Community
> >
> > The DataFusion development team seeks to foster the development and
> > user communities. We hope that becoming a separate project will help
> > both Arrow and DataFusion communities by being more focused. Focused
> > governance will make it easier to grow the community of committers and
> > PMC members and make the organization more clear to others.
> >
> > Alignment
> >
> > The ASF is a natural host for DataFusion given that it is already the
> > home of Arrow, Parquet, and other related distributed system, storage
> > and query execution systems.
> >
> > Project Leadership
> >
> > Proposed Initial PMC
> >
> > We propose the following people as the initial DataFusion PMC members.
> > This is a subset of the existing Arrow PMC members who contribute to
> > DataFusion https://people.apache.org/phonebook.html?unix=arrow
> >
> > Andy Grove (agrove): Arrow PMC Chair
> > Andrew Lamb (alamb): Arrow PMC, past Arrow PMC Chair
> > Daniël Heres (dheres) Arrow PMC
> > Jie Wen (jakevin): Arrow PMC, Doris Committer
> > Kun Liu (liukun): Arrow PMC, IoTDB PMC, TSFile PMC
> > Liang-Chi Hsieh (viirya): Arrow PMC, Spark PMC
> > Qingping Hou: (houqp): Arrow PMC
> > Wes McKinney(wesm): Arrow PMC, ASF Member
> > Will Jones (wjones127): Arrow PMC
> >
> > We’d like to propose Andrew Lamb as the initial Chair, (and thus ASF
> > VP) for the DataFusion project.
> >
> > Affiliations
> >
> > Andy Grove (agrove): NVidia
> > Andrew Lamb (alamb): InfluxData
> > Daniël Heres (dheres): Coralogix
> > Jie Wen (jakevin): SelectDB
> > Kun Liu (liukun): Ebay
> > Liang-Chi Hsieh (viirya): Apple
> > Qingping Hou: (houqp): Scribd
> > Wes McKinney(wesm): Posit
> > Will Jones (wjones127): LanceDB
> >
> > Proposed Initial Committers
> >
> > In addition to the PMC, we propose the following people as the initial
> > DataFusion committers. This is a subset of the existing Arrow
> > committers who contribute to DataFusion
> > https://people.apache.org/phonebook.html?unix=arrow
> >
> > akurmustafa Mustafa Akur (Synnada)
> > avantgardner Brent Gardner (Coralogix)
> > comphead Oleks V. (Unaffiliated)
> > jayzhan Jay Zhan (Unaffiliated)
> > jeffreyvo Jeffry Vo (Unaffiliated)
> > jiayuliu Liu Jiayu (Airbnb)
> > mete Metehan Yildirim (Synnada)
> > mingmwang Wang Mingming (Ebay)
> > mneumann Marco Neumann (InfluxData)
> > nju_yaho Zhong Yanghong (Ebay)
> > ozankabak Mehmet Ozan Kabak (Synnada)
> > paddyhoran Paddy Horan (Assured Allies)
> > rdettai Rémi Dettai (Cloudfuse)
> > sunchao Chao Sun (Apple)
> > thinkharderdev Daniel Harris (Coralogix)
> > tustvold Raphael Taylor-Davies (InfluxData)
> > wayne Ruihang Xia (Greptime)
> > xudong963 Xudong Wang (ByteDance)
> > yjshen Yijie Shen (Space and Time)
> > yangjiang Yang Jiang (ebay)
> >
> >
> > Risk Assessments
> >
> > Naming / Trademarks
> >
> > As a sub-project of Arrow, the DataFusion name has been used for over
> > 4 years without any known issues. A podling name search did not turn
> > up any concerns and was approved:
> > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> >
> > Legal / IP Clearance
> >
> > All DataFusion code has either been donated to the Arrow project with
> > appropriate IP clearance or has been developed directly under ASF
> > processes and procedures. Thus creating a new top level project poses
> > no new Legal or IP risks.
> >
> > Code Extraction
> >
> > The relevant code is already in 5 separate repositories:
> > https://github.com/apache/arrow-datafusion/
> > https://github.com/apache/arrow-datafusion-python
> > https://github.com/apache/arrow-ballista
> > https://github.com/apache/arrow-ballista-python
> > https://github.com/apache/arrow-datafusion-comet
> >
> > We foresee no issues with code extraction and propose these
> > repositories be renamed to reflect top level projects
> >
> > Note: https://github.com/apache/arrow-rs, the Rust implementation of
> > Arrow, would remain part of the Arrow project.
> >
> > Orphaned Products
> >
> > DataFusion is known to be used in many open source and commercial
> > projects
> > https://arrow.apache.org/datafusion/user-guide/introduction.html#known-users
> > ,
> > has had multiple commits daily for several years, and its adoption and
> > number of contributors appears to be growing. We do not foresee the
> > project being orphaned in the next several years.
> >
> > Inexperience with Open Source
> >
> > The proposed PMC has extensive experience with Apache Arrow and other
> > Apache projects, and includes PMC members, PMC chairs and an ASF
> > Member. The DataFusion PMC and more experienced committers will
> > continue to coach new community members who may be less familiar with
> > the Apache Way.
> >
> > Homogeneous Developers
> >
> > The 9 proposed PMC members are from 9 different employers and the
> > proposed committers are similarly distributed across affiliations. No
> > specific entity employs more than 3 total proposed developers.
> >
> > Reliance on Salaried Developers
> >
> > A substantial amount of work on DataFusion has been by salaried
> > developers, but it also has a long tradition of attracting
> > contributions from students and hobbyists and we plan no changes in
> > contribution structure.
> >
> > Relationships with Other Apache Products
> >
> > DataFusion will obviously have a strong relationship with the Arrow
> > project given the overlap in people. We don’t foresee close
> > collaboration with other projects at this time.
> >
> > Cryptography
> >
> > DataFusion does not directly support encryption and there are no
> > near-term plans to add support for encryption. Users who need this
> > functionality can use the extension APIs.
> >
> > Required Resources
> >
> > Mailing Lists
> >
> > - private@datafusion.apache.org for private PMC discussions (with
> > moderated subscriptions)
> > - dev@datafusion.apache.org
> > - commits@datafusion.apache.org
> > - user@datafusion.apache.org
> >
> > Version Control
> >
> > We propose to continue to use git for source control and github for
> > hosting and testing resources.
> >
> > We also need to rename the github repositories to reflect the new top
> > level names:
> >
> > https://github.com/apache/arrow-datafusion/ → apache/datafusion
> > https://github.com/apache/arrow-datafusion-python →
> > apache/datafusion-python
> > https://github.com/apache/arrow-ballista → apache/datafusion-ballista
> > https://github.com/apache/arrow-ballista-python →
> > apache/datafusion-ballista-python
> > https://github.com/apache/arrow-datafusion-comet → apache/datafusion-comet
> >
> >
> >
> > Issue Tracking
> >
> > DataFusion would continue to use github for its issue tracking and
> > communications
> >
> > Other Resources
> >
> > The existing repositories already make use of existing Apache
> > infrastructure, and we expect no change in the initial resource usage.
> > As the project continues to grow, we expect continued infrastructure
> > demand growth.
> >
> >
> > FAQ: Has a sub project been promoted to a top level project before?
> >
> > Yes, and it appears to happen commonly. The Arrow project itself was
> > created as a top level project from work that started in Apache Drill,
> > and there are many sub projects of Hadoop that spun out as their own
> > top level projects such as Mahout, Avro and HBase:
> >
> > https://news.apache.org/foundation/entry/the_apache_software_foundation_announces4
> >
> >
> >
> > Related material:
> > Name search request / research for DataFusion:
> > https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> > Discussion about this proposal on the arrow mailing list:
> > https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> > Discussion about which repositories on the arrow mailing list:
> > https://lists.apache.org/thread/ob3n0d9ky0bgrryl3xn39w9k566bq00q
> > Discussion about initial PMC on the arrow mailing list:
> > https://lists.apache.org/thread/pymrzcdw4qdptvby85f69rg3pcckl15b
> > Discussion in github about creating a new DataFusion top level
> > project: https://github.com/apache/arrow-datafusion/discussions/6475
> > Discussion about graduating on incubator list:
> > https://lists.apache.org/thread/r4n73pmms1lv0jbohyx1o1z13d615t99
> > Original Proposal for the Arrow project:
> > https://lists.apache.org/thread/x2qzdwglm8pkqp9gv03bbgw17khl7pq3
> >
Re: [VOTE] Move Arrow DataFusion Subproject to new Top Level Apache Project
Posted by vin jake <ja...@gmail.com>.
+1 (binding)
On Fri, Mar 1, 2024 at 7:33 PM Andrew Lamb <al...@influxdata.com> wrote:
> Hello,
>
> As we have discussed[1][2] I would like to vote on the proposal to
> create a new Apache Top Level Project for DataFusion. The text of the
> proposed resolution and background document is copy/pasted below
>
> If the community is in favor of this, we plan to submit the resolution
> to the ASF board for approval with the next Arrow report (for the
> April 2024 board meeting).
>
> The vote will be open for at least 7 days.
>
> [ ] +1 Accept this Proposal
> [ ] +0
> [ ] -1 Do not accept this proposal because...
>
> Andrew
>
> [1] https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> [2] https://github.com/apache/arrow-datafusion/discussions/6475
>
> ---------- Proposed Resolution ---------
>
> Resolution to Create the Apache DataFusion Project from the Apache
> Arrow DataFusion Sub Project
>
> =============================================================
>
> X. Establish the Apache DataFusion Project
>
> WHEREAS, the Board of Directors deems it to be in the best
> interests of the Foundation and consistent with the
> Foundation's purpose to establish a Project Management
> Committee charged with the creation and maintenance of
> open-source software related to an extensible query engine
> for distribution at no charge to the public.
>
> NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> Committee (PMC), to be known as the "Apache DataFusion Project",
> be and hereby is established pursuant to Bylaws of the
> Foundation; and be it further
>
> RESOLVED, that the Apache DataFusion Project be and hereby is
> responsible for the creation and maintenance of software
> related to an extensible query engine; and be it further
>
> RESOLVED, that the office of "Vice President, Apache DataFusion" be
> and hereby is created, the person holding such office to
> serve at the direction of the Board of Directors as the chair
> of the Apache DataFusion Project, and to have primary responsibility
> for management of the projects within the scope of
> responsibility of the Apache DataFusion Project; and be it further
>
> RESOLVED, that the persons listed immediately below be and
> hereby are appointed to serve as the initial members of the
> Apache DataFusion Project:
>
> * Andy Grove (agrove@apache.org)
> * Andrew Lamb (alamb@apache.org)
> * Daniël Heres (dheres@apache.org)
> * Jie Wen (jakevin@apache.org)
> * Kun Liu (liukun@apache.org)
> * Liang-Chi Hsieh (viirya@apache.org)
> * Qingping Hou: (houqp@apache.org)
> * Wes McKinney(wesm@apache.org)
> * Will Jones (wjones127@apache.org)
>
> RESOLVED, that the Apache DataFusion Project be and hereby
> is tasked with the migration and rationalization of the Apache
> Arrow DataFusion sub-project; and be it further
>
> RESOLVED, that all responsibilities pertaining to the Apache
> Arrow DataFusion sub-project encumbered upon the
> Apache Arrow Project are hereafter discharged.
>
> NOW, THEREFORE, BE IT FURTHER RESOLVED, that Andrew Lamb
> be appointed to the office of Vice President, Apache DataFusion, to
> serve in accordance with and subject to the direction of the
> Board of Directors and the Bylaws of the Foundation until
> death, resignation, retirement, removal or disqualification,
> or until a successor is appointed.
> =============================================================
>
>
> -------
>
>
> Summary:
>
> We propose creating a new top level project, Apache DataFusion, from
> an existing sub project of Apache Arrow to facilitate additional
> community and project growth.
>
> Abstract
>
> Apache Arrow DataFusion[1] is a very fast, extensible query engine
> for building high-quality data-centric systems in Rust, using the
> Apache Arrow in-memory format. DataFusion offers SQL and Dataframe
> APIs, excellent performance, built-in support for CSV, Parquet, JSON,
> and Avro, extensive customization, and a great community.
>
> [1] https://arrow.apache.org/datafusion/
>
>
> Proposal
>
> We propose creating a new top level ASF project, Apache DataFusion,
> governed initially by a subset of the Apache Arrow project’s PMC and
> committers. The project’s code is in five existing git repositories,
> currently governed by Apache Arrow which would transfer to the new top
> level project.
>
> Background
>
> When DataFusion was initially donated to the Arrow project, it did not
> have a strong enough community to stand on its own. It has since grown
> significantly, and benefited immensely from being part of Arrow and
> nurturing of the Apache Way, and now has a community strong enough to
> stand on its own and that would benefit from focused governance
> attention.
>
> The community has discussed this idea publicly for more than 6 months
> https://github.com/apache/arrow-datafusion/discussions/6475 and
> briefly on the Arrow PMC mailing list
> https://lists.apache.org/thread/thv2jdm6640l6gm88hy8jhk5prjww0cs. As
> of the time of this writing both had exclusively positive reactions.
>
> Several current members of the Arrow PMC are both active contributors
> to DataFusion and understand and believe deeply in the Apache Way, and
> play active governance roles in the Arrow project as PMC members and
> PMC chairs, guiding the community, and releasing software versions.
> With this existing governance experience and structure, the new top
> level project will be able to function well immediately and
> independently.
>
> Overview of DataFusion
>
> Current Status
>
> Meritocracy
>
> DataFusion has been developed as part of Apache Arrow and thus has
> been operating as a meritocracy. Many of the developers of DataFusion
> are Arrow PMC members or committers. The DataFusion project plans to
> continue adding new PMC and committers as the project matures and
> grows.
>
> Community
>
> The DataFusion development team seeks to foster the development and
> user communities. We hope that becoming a separate project will help
> both Arrow and DataFusion communities by being more focused. Focused
> governance will make it easier to grow the community of committers and
> PMC members and make the organization more clear to others.
>
> Alignment
>
> The ASF is a natural host for DataFusion given that it is already the
> home of Arrow, Parquet, and other related distributed system, storage
> and query execution systems.
>
> Project Leadership
>
> Proposed Initial PMC
>
> We propose the following people as the initial DataFusion PMC members.
> This is a subset of the existing Arrow PMC members who contribute to
> DataFusion https://people.apache.org/phonebook.html?unix=arrow
>
> Andy Grove (agrove): Arrow PMC Chair
> Andrew Lamb (alamb): Arrow PMC, past Arrow PMC Chair
> Daniël Heres (dheres) Arrow PMC
> Jie Wen (jakevin): Arrow PMC, Doris Committer
> Kun Liu (liukun): Arrow PMC, IoTDB PMC, TSFile PMC
> Liang-Chi Hsieh (viirya): Arrow PMC, Spark PMC
> Qingping Hou: (houqp): Arrow PMC
> Wes McKinney(wesm): Arrow PMC, ASF Member
> Will Jones (wjones127): Arrow PMC
>
> We’d like to propose Andrew Lamb as the initial Chair, (and thus ASF
> VP) for the DataFusion project.
>
> Affiliations
>
> Andy Grove (agrove): NVidia
> Andrew Lamb (alamb): InfluxData
> Daniël Heres (dheres): Coralogix
> Jie Wen (jakevin): SelectDB
> Kun Liu (liukun): Ebay
> Liang-Chi Hsieh (viirya): Apple
> Qingping Hou: (houqp): Scribd
> Wes McKinney(wesm): Posit
> Will Jones (wjones127): LanceDB
>
> Proposed Initial Committers
>
> In addition to the PMC, we propose the following people as the initial
> DataFusion committers. This is a subset of the existing Arrow
> committers who contribute to DataFusion
> https://people.apache.org/phonebook.html?unix=arrow
>
> akurmustafa Mustafa Akur (Synnada)
> avantgardner Brent Gardner (Coralogix)
> comphead Oleks V. (Unaffiliated)
> jayzhan Jay Zhan (Unaffiliated)
> jeffreyvo Jeffry Vo (Unaffiliated)
> jiayuliu Liu Jiayu (Airbnb)
> mete Metehan Yildirim (Synnada)
> mingmwang Wang Mingming (Ebay)
> mneumann Marco Neumann (InfluxData)
> nju_yaho Zhong Yanghong (Ebay)
> ozankabak Mehmet Ozan Kabak (Synnada)
> paddyhoran Paddy Horan (Assured Allies)
> rdettai Rémi Dettai (Cloudfuse)
> sunchao Chao Sun (Apple)
> thinkharderdev Daniel Harris (Coralogix)
> tustvold Raphael Taylor-Davies (InfluxData)
> wayne Ruihang Xia (Greptime)
> xudong963 Xudong Wang (ByteDance)
> yjshen Yijie Shen (Space and Time)
> yangjiang Yang Jiang (ebay)
>
>
> Risk Assessments
>
> Naming / Trademarks
>
> As a sub-project of Arrow, the DataFusion name has been used for over
> 4 years without any known issues. A podling name search did not turn
> up any concerns and was approved:
> https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
>
> Legal / IP Clearance
>
> All DataFusion code has either been donated to the Arrow project with
> appropriate IP clearance or has been developed directly under ASF
> processes and procedures. Thus creating a new top level project poses
> no new Legal or IP risks.
>
> Code Extraction
>
> The relevant code is already in 5 separate repositories:
> https://github.com/apache/arrow-datafusion/
> https://github.com/apache/arrow-datafusion-python
> https://github.com/apache/arrow-ballista
> https://github.com/apache/arrow-ballista-python
> https://github.com/apache/arrow-datafusion-comet
>
> We foresee no issues with code extraction and propose these
> repositories be renamed to reflect top level projects
>
> Note: https://github.com/apache/arrow-rs, the Rust implementation of
> Arrow, would remain part of the Arrow project.
>
> Orphaned Products
>
> DataFusion is known to be used in many open source and commercial
> projects
> https://arrow.apache.org/datafusion/user-guide/introduction.html#known-users
> ,
> has had multiple commits daily for several years, and its adoption and
> number of contributors appears to be growing. We do not foresee the
> project being orphaned in the next several years.
>
> Inexperience with Open Source
>
> The proposed PMC has extensive experience with Apache Arrow and other
> Apache projects, and includes PMC members, PMC chairs and an ASF
> Member. The DataFusion PMC and more experienced committers will
> continue to coach new community members who may be less familiar with
> the Apache Way.
>
> Homogeneous Developers
>
> The 9 proposed PMC members are from 9 different employers and the
> proposed committers are similarly distributed across affiliations. No
> specific entity employs more than 3 total proposed developers.
>
> Reliance on Salaried Developers
>
> A substantial amount of work on DataFusion has been by salaried
> developers, but it also has a long tradition of attracting
> contributions from students and hobbyists and we plan no changes in
> contribution structure.
>
> Relationships with Other Apache Products
>
> DataFusion will obviously have a strong relationship with the Arrow
> project given the overlap in people. We don’t foresee close
> collaboration with other projects at this time.
>
> Cryptography
>
> DataFusion does not directly support encryption and there are no
> near-term plans to add support for encryption. Users who need this
> functionality can use the extension APIs.
>
> Required Resources
>
> Mailing Lists
>
> - private@datafusion.apache.org for private PMC discussions (with
> moderated subscriptions)
> - dev@datafusion.apache.org
> - commits@datafusion.apache.org
> - user@datafusion.apache.org
>
> Version Control
>
> We propose to continue to use git for source control and github for
> hosting and testing resources.
>
> We also need to rename the github repositories to reflect the new top
> level names:
>
> https://github.com/apache/arrow-datafusion/ → apache/datafusion
> https://github.com/apache/arrow-datafusion-python →
> apache/datafusion-python
> https://github.com/apache/arrow-ballista → apache/datafusion-ballista
> https://github.com/apache/arrow-ballista-python →
> apache/datafusion-ballista-python
> https://github.com/apache/arrow-datafusion-comet → apache/datafusion-comet
>
>
>
> Issue Tracking
>
> DataFusion would continue to use github for its issue tracking and
> communications
>
> Other Resources
>
> The existing repositories already make use of existing Apache
> infrastructure, and we expect no change in the initial resource usage.
> As the project continues to grow, we expect continued infrastructure
> demand growth.
>
>
> FAQ: Has a sub project been promoted to a top level project before?
>
> Yes, and it appears to happen commonly. The Arrow project itself was
> created as a top level project from work that started in Apache Drill,
> and there are many sub projects of Hadoop that spun out as their own
> top level projects such as Mahout, Avro and HBase:
>
> https://news.apache.org/foundation/entry/the_apache_software_foundation_announces4
>
>
>
> Related material:
> Name search request / research for DataFusion:
> https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219
> Discussion about this proposal on the arrow mailing list:
> https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341
> Discussion about which repositories on the arrow mailing list:
> https://lists.apache.org/thread/ob3n0d9ky0bgrryl3xn39w9k566bq00q
> Discussion about initial PMC on the arrow mailing list:
> https://lists.apache.org/thread/pymrzcdw4qdptvby85f69rg3pcckl15b
> Discussion in github about creating a new DataFusion top level
> project: https://github.com/apache/arrow-datafusion/discussions/6475
> Discussion about graduating on incubator list:
> https://lists.apache.org/thread/r4n73pmms1lv0jbohyx1o1z13d615t99
> Original Proposal for the Arrow project:
> https://lists.apache.org/thread/x2qzdwglm8pkqp9gv03bbgw17khl7pq3
>