You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@incubator.apache.org by "Adunuthula, Seshu" <sa...@ebay.com> on 2016/03/25 14:20:02 UTC
AirPal to Apache? Re: [VOTE] Accept Airflow into the Incubator
Any plans from AirBnB to bring AirPal to Apache?
On 3/24/16, 8:00 PM, "Siddharth Anand" <sa...@agari.com.INVALID> wrote:
>Following the discussion earlier:
> https://s.apache.org/AirflowDiscussion
>
>I would like to call a VOTE for accepting Airflow as a new incubator
>project.
>
>The proposal is available at:
>https://wiki.apache.org/incubator/AirflowProposal
>
>The proposal is also included at the bottom of this email.
>
>Vote is open until at least Tues, 29 March 2016, 23:59:00 PDT
>[ ] +1 accept Airflow into the Apache Incubator
>[ ] ±0
>[ ] -1 because...
>
>+1 (non-binding)
>
>Thanks,
>-s (Sid)
>
>
>== Abstract ==
>
>Airflow is a workflow automation and scheduling system that can be
>used to author and manage data pipelines.
>
>== Proposal ==
>
>Airflow provides a system for authoring and managing workflows a.k.a.
>data pipelines a.k.a. DAGs (Directed Acyclic Graphs). The developer
>authors DAGs in Python using an Airflow-provided framework. He/She
>then executes the DAG using Airflow¹s scheduler or registers the DAG
>for event-based execution. A web-based UI provides the developer with
>a range of options for managing and viewing his/her data pipelines.
>Background
>
>Airflow was developed at Airbnb to enable easier authorship and
>management of DAGs than were possible with existing solutions such as
>Oozie and Azkaban. For starters, both Oozie and Azkaban rely on one or
>more XML or property files to be bundled together to define a
>workflow. This separation of code and config can present a challenge
>to understanding the DAG - in Azkaban, a DAG¹s structure is reflected
>by its file system tree and one can find himself/herself traversing
>the file system when inspecting or changing the structure of the DAG.
>Airflow workflows, on the other hand, are simply and elegantly defined
>in Python code, often a single file. Airflow merges the powerful
>Web-based management aspects of projects like Azkaban and Oozie with
>the simplicity and elegance of defining workflows in Python. Airflow,
>less than a year old in terms of its Open Source launch, is currently
>used in production environments in more than 30 companies and boasts
>an active contributor list of more than 100 developers, the vast
>majority of which (>95%) are outside of Airbnb.
>
>We would like to share it with the ASF and begin developing a
>community of developers and users within Apache.
>
>== Rationale ==
>
>Many organizations (>30) already benefit from running Airflow to
>manage data pipelines. Our 100+ contributors continue to provide
>integrations with 3rd party systems through the implementation of new
>hooks and operators, both of which are used in defining the tasks that
>compose workflows.
>
>== Current Status ==
>
>=== Meritocracy ===
>
>Our intent with this incubator proposal is to start building a diverse
>developer community around Airflow following the Apache meritocracy
>model. Since Airflow was open-sourced in mid-2015, we have had fast
>adoption and contributions by multiple organizations the world over.
>We plan to continue to support new contributors and we will work to
>actively promote those who contribute significantly to the project to
>committers.
>
>=== Community ===
>
>Airflow is currently being used in over 30 companies. We hope to
>extend our contributor base significantly and invite all those who are
>interested in building large-scale distributed systems to participate.
>
>=== Core Developers ===
>
>Airflow is currently being developed by four engineers: Maxime
>Beauchemin, Siddharth Anand, Bolke de Bruin, and Chris Riccomini.
>Chris is a member of the Apache Samza PMC and a contributor to various
>Apache projects, including Apache Kafka and Apache YARN. Maxime,
>Siddharth, and Bolke have contributed to Airflow.
>
>=== Alignment ===
>The ASF is the natural choice to host the Airflow project as its goal
>of encouraging community-driven open-source projects fits with our
>vision for Airflow.
>
>== Known Risks ==
>
>=== Orphaned Products ===
>
>The core developers plan to work part time on the project. There is
>very little risk of Airflow being abandoned as all of our companies
>rely on it.
>
>=== Inexperience with Open Source ===
>
>All of the core developers have experience with open source
>development. Chris is a member of the Apache Samza PMC and a
>contributor to various Apache projects, including Apache Kafka and
>Apache YARN. Bolke is contributor on multiple open source projects and
>a few Apache projects as well, including Apache Hive, Apache Hadoop,
>and Apache Ranger.
>
>=== Homogeneous Developers ===
>
>The current core developers are all from different companies. Our
>community of 100 contributors hail from over 30 different companies
>from across the world.
>
>=== Reliance on Salaried Developers ===
>
>Currently, the only developer paid to work on this project is Maxime.
>
>=== Relationships with Other Apache Products ===
>
>Airflow is deeply integrated with Apache products. It currently
>provides hooks and operators to enable workflows to leverage Apache
>Pig, Apache Hive, Apache Spark, Apache Sqoop, Apache Hadoop, etcŠ We
>plan to add support for other Apache projects in the future.
>
>=== An Excessive Fascination with the Apache Brand ===
>
>While we respect the reputation of the Apache brand and have no doubts
>that it will attract contributors and users, our interest is primarily
>to give Airflow a solid home as an open source project following an
>established development model. We have also given reasons in the
>Rationale and Alignment sections.
>
>== Documentation ==
>http://wiki.apache.org/incubator/AirflowProposal
>
>== Initial Source ==
>https://github.com/airbnb/airflow
>
>== Source and Intellectual Property Submission Plan ==
>
>As soon as Airflow is approved to join Apache Incubator, Airbnb will
>execute a Software Grant Agreement and the source code will be
>transitioned onto ASF infrastructure. The code is already licensed
>under the Apache Software License, version 2.0. We know of no legal
>encumberments that would inhibit the transfer of source code to the
>ASF.
>
>== External Dependencies ==
>
>The dependencies all have Apache compatible licenses.
>
> *
>[[https://bitbucket.org/zzzeek/alembic/src/9538c3e1a71c946a53f8762e68e94cf
>bcb9f932f/LICENSE?fileviewer=file-view-default|alembic
>(MIT)]]
> * [[https://github.com/boto/boto/blob/develop/LICENSE|boto (MIT)]]
> * [[https://github.com/celery/celery/blob/master/LICENSE|celery (BSD)]]
> * [[https://github.com/mher/chartkick.py/blob/master/LICENSE|chartkick
>(MIT)]]
> *
>[[https://github.com/pyca/cryptography/blob/master/LICENSE.APACHE|cryptogr
>aphy
>(Apache 2.0/BSD)]]
> *
>[[https://bitbucket.org/ned/coveragepy/src/b74c40b2c107db17f0775be5ec6c44f
>5e1cf5cbf/LICENSE.txt?fileviewer=file-view-default|coverage
>(Apache 2.0)]]
> *
>[[https://github.com/coagulant/coveralls-python/blob/master/LICENCE|covera
>lls
>(MIT)]]
> * [[https://pypi.python.org/pypi/croniter|croniter (MIT)]]
> * [[https://github.com/uqfoundation/dill/blob/master/LICENSE|dill (BSD)]]
> * [[https://github.com/docker/docker-py/blob/master/LICENSE|docker-py
>(Apache 2.0)]]
> *
>[[https://bitbucket.org/fabian/filechunkio/src/84289d7599a207f575cb28db719
>dd9d44e880208/LICENCE?fileviewer=file-view-default|filechunkio
>(MIT)]]
> *
>[[https://bitbucket.org/tarek/flake8/src/a209fb69350c572c9b2d7b4b09c7657be
>153be5e/LICENSE?fileviewer=file-view-default|flake8
>(MIT)]]
> * [[https://github.com/mitsuhiko/flask/blob/master/LICENSE|flask (BSD)]]
> *
>[[https://github.com/flask-admin/flask-admin/blob/master/LICENSE|flask-adm
>in
>(BSD)]]
> *
>[[https://github.com/thadeusb/flask-cache/blob/master/LICENSE|flask-cache
>(BSD)]]
> *
>[[https://github.com/maxcountryman/flask-login/blob/master/LICENSE|flask-l
>ogin
>(MIT)]]
> * [[https://github.com/mher/flower/blob/master/LICENSE|flower (BSD)]]
> *
>[[https://github.com/PythonCharmers/python-future/blob/master/LICENSE.txt|
>future
>(MIT)]]
> * [[https://github.com/benoitc/gunicorn/blob/master/LICENSE|gunicorn
>(MIT)]]
> *
>[[https://github.com/youngwookim/hive-thrift-py/blob/master/setup.py|hive-
>thrift-py
>(Apache 2.0)]]
> * [[https://github.com/ipython/ipython/blob/master/COPYING.rst|ipython
>(BSD)]]
> * [[https://github.com/mitsuhiko/jinja2/blob/master/LICENSE|jinja2
>(BSD)]]
> *
>[[https://github.com/waylan/Python-Markdown/blob/master/LICENSE.md|markdow
>n
>(BSD)]]
> * [[https://github.com/pydata/pandas/blob/master/LICENSE|pandas (BSD)]]
> * [[https://pypi.python.org/pypi/Pygments|pygments (BSD)]]
> * pyhive
> * pydruid
> * PyOpenSSL
> * PySmbClient
> * python-dateutil
> * redis
> * requests
> * setproctitle
> * statsd
> * sphinx
> * sphinx-argparse
> * sphinx_rtd_theme
> * Sphinx-PyPI-upload
> * sqlalchemy (MIT)
> * thrift
> * jaydebeapi
> * mysqlclient
> * unicodecsv
> * slackclient
> * ldap3
> * Flask-WTF
> * lxml
> * [[https://github.com/bgamble/pykerberos/blob/master/LICENSE|pykerberos
>(Apache 2.0)]]
> * [[https://github.com/pyca/bcrypt/blob/master/LICENSE|bcrypt (Apache
>2.0)]]
> *
>[[https://github.com/maxcountryman/flask-bcrypt/blob/master/LICENSE|flask-
>bcrypt
>(BSD)]]
> * [[https://github.com/testing-cabal/mock/blob/master/LICENSE.txt|mock
>(BSD)]]
> * [[https://github.com/mtth/hdfs/blob/master/LICENSE|hdfs (MIT)]]
>
>== Cryptography ==
>
>None
>
>== Required Resources ==
>
>=== Mailing Lists ===
>
> * private@airflow.incubator.apache.org (moderated)
> * dev@airflow.incubator.apache.org
> * commits@airflow.incubator.apache.org
>
>=== Subversion Directory ===
>
>Git is the preferred source control system: git://git.apache.org/Airflow
>
>=== Issue Tracking ===
>
>JIRA Airflow (Airflow)
>
>=== Other Resources ===
>
>The existing code already has unit tests, so we would like a Travis
>instance to run them whenever a new patch is submitted. This can be
>added after project creation.
>
>== Initial Committers ==
>
> * Maxime Beauchemin
> * Siddharth Anand
> * Chris Riccomini
> * Bolke de Bruin
> * Arthur Wiedmer
> * Dan Davydov
> * Jeremiah Lowin
> * Patrick Leo Tardif
>
>== Affiliations ==
>
> * Maxime Beauchemin (Airbnb)
> * Siddharth Anand (Agari)
> * Chris Riccomini (WePay)
> * Bolke de Bruin (ING)
> * Arthur Wiedmer (Airbnb)
> * Dan Davydov (Airbnb)
> * Jeremiah Lowin (Kokino)
> * Patrick Leo Tardif (Airbnb)
>
>== Sponsors ==
>
>=== Champion ===
>
>Chris Riccomini (WePay, Apache Samza PMC)
>
>=== Nominated Mentors ===
>
> * Chris Nauroth (HortonWorks, Apache Hadoop Committer/PMC Member,
>Apache ZooKeeper Committer, Apache Software Foundation Member)
> * Hitesh Shah (HortonWorks, Apache Hadoop Committer/PMC Member,
>Apache Ambari Committer/PMC Member, Apache Tez Committer/PMC Member,
>Apache Software Foundation Member)
> * Jakob Homan (OfferUp, Apache Hadoop Committer/PMC Member, Apache
>Kafka Committer/PMC Member, Apache Samza Committer/PMC Member, Apache
>Giraph Committer/PMC Member, Apache Software Foundation Member)
>
>=== Sponsoring Entity ===
>
>We are requesting the Incubator to sponsor this project.
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: AirPal to Apache? Re: [VOTE] Accept Airflow into the Incubator
Posted by "Adunuthula, Seshu" <sa...@ebay.com>.
I am hoping there are a few AirBnB guys on this DL, though its dependency
on Presto could prohibit it becoming an Apache project…
On 3/25/16, 7:38 AM, "John D. Ament" <jo...@apache.org> wrote:
>You would need to ask AirBnB.
>
>John
>
>On Fri, Mar 25, 2016 at 9:20 AM Adunuthula, Seshu <sa...@ebay.com>
>wrote:
>
>> Any plans from AirBnB to bring AirPal to Apache?
>>
>>
>>
>> On 3/24/16, 8:00 PM, "Siddharth Anand" <sa...@agari.com.INVALID> wrote:
>>
>> >Following the discussion earlier:
>> > https://s.apache.org/AirflowDiscussion
>> >
>> >I would like to call a VOTE for accepting Airflow as a new incubator
>> >project.
>> >
>> >The proposal is available at:
>> >https://wiki.apache.org/incubator/AirflowProposal
>> >
>> >The proposal is also included at the bottom of this email.
>> >
>> >Vote is open until at least Tues, 29 March 2016, 23:59:00 PDT
>> >[ ] +1 accept Airflow into the Apache Incubator
>> >[ ] ±0
>> >[ ] -1 because...
>> >
>> >+1 (non-binding)
>> >
>> >Thanks,
>> >-s (Sid)
>> >
>> >
>> >== Abstract ==
>> >
>> >Airflow is a workflow automation and scheduling system that can be
>> >used to author and manage data pipelines.
>> >
>> >== Proposal ==
>> >
>> >Airflow provides a system for authoring and managing workflows a.k.a.
>> >data pipelines a.k.a. DAGs (Directed Acyclic Graphs). The developer
>> >authors DAGs in Python using an Airflow-provided framework. He/She
>> >then executes the DAG using Airflow¹s scheduler or registers the DAG
>> >for event-based execution. A web-based UI provides the developer with
>> >a range of options for managing and viewing his/her data pipelines.
>> >Background
>> >
>> >Airflow was developed at Airbnb to enable easier authorship and
>> >management of DAGs than were possible with existing solutions such as
>> >Oozie and Azkaban. For starters, both Oozie and Azkaban rely on one or
>> >more XML or property files to be bundled together to define a
>> >workflow. This separation of code and config can present a challenge
>> >to understanding the DAG - in Azkaban, a DAG¹s structure is reflected
>> >by its file system tree and one can find himself/herself traversing
>> >the file system when inspecting or changing the structure of the DAG.
>> >Airflow workflows, on the other hand, are simply and elegantly defined
>> >in Python code, often a single file. Airflow merges the powerful
>> >Web-based management aspects of projects like Azkaban and Oozie with
>> >the simplicity and elegance of defining workflows in Python. Airflow,
>> >less than a year old in terms of its Open Source launch, is currently
>> >used in production environments in more than 30 companies and boasts
>> >an active contributor list of more than 100 developers, the vast
>> >majority of which (>95%) are outside of Airbnb.
>> >
>> >We would like to share it with the ASF and begin developing a
>> >community of developers and users within Apache.
>> >
>> >== Rationale ==
>> >
>> >Many organizations (>30) already benefit from running Airflow to
>> >manage data pipelines. Our 100+ contributors continue to provide
>> >integrations with 3rd party systems through the implementation of new
>> >hooks and operators, both of which are used in defining the tasks that
>> >compose workflows.
>> >
>> >== Current Status ==
>> >
>> >=== Meritocracy ===
>> >
>> >Our intent with this incubator proposal is to start building a diverse
>> >developer community around Airflow following the Apache meritocracy
>> >model. Since Airflow was open-sourced in mid-2015, we have had fast
>> >adoption and contributions by multiple organizations the world over.
>> >We plan to continue to support new contributors and we will work to
>> >actively promote those who contribute significantly to the project to
>> >committers.
>> >
>> >=== Community ===
>> >
>> >Airflow is currently being used in over 30 companies. We hope to
>> >extend our contributor base significantly and invite all those who are
>> >interested in building large-scale distributed systems to participate.
>> >
>> >=== Core Developers ===
>> >
>> >Airflow is currently being developed by four engineers: Maxime
>> >Beauchemin, Siddharth Anand, Bolke de Bruin, and Chris Riccomini.
>> >Chris is a member of the Apache Samza PMC and a contributor to various
>> >Apache projects, including Apache Kafka and Apache YARN. Maxime,
>> >Siddharth, and Bolke have contributed to Airflow.
>> >
>> >=== Alignment ===
>> >The ASF is the natural choice to host the Airflow project as its goal
>> >of encouraging community-driven open-source projects fits with our
>> >vision for Airflow.
>> >
>> >== Known Risks ==
>> >
>> >=== Orphaned Products ===
>> >
>> >The core developers plan to work part time on the project. There is
>> >very little risk of Airflow being abandoned as all of our companies
>> >rely on it.
>> >
>> >=== Inexperience with Open Source ===
>> >
>> >All of the core developers have experience with open source
>> >development. Chris is a member of the Apache Samza PMC and a
>> >contributor to various Apache projects, including Apache Kafka and
>> >Apache YARN. Bolke is contributor on multiple open source projects and
>> >a few Apache projects as well, including Apache Hive, Apache Hadoop,
>> >and Apache Ranger.
>> >
>> >=== Homogeneous Developers ===
>> >
>> >The current core developers are all from different companies. Our
>> >community of 100 contributors hail from over 30 different companies
>> >from across the world.
>> >
>> >=== Reliance on Salaried Developers ===
>> >
>> >Currently, the only developer paid to work on this project is Maxime.
>> >
>> >=== Relationships with Other Apache Products ===
>> >
>> >Airflow is deeply integrated with Apache products. It currently
>> >provides hooks and operators to enable workflows to leverage Apache
>> >Pig, Apache Hive, Apache Spark, Apache Sqoop, Apache Hadoop, etcŠ We
>> >plan to add support for other Apache projects in the future.
>> >
>> >=== An Excessive Fascination with the Apache Brand ===
>> >
>> >While we respect the reputation of the Apache brand and have no doubts
>> >that it will attract contributors and users, our interest is primarily
>> >to give Airflow a solid home as an open source project following an
>> >established development model. We have also given reasons in the
>> >Rationale and Alignment sections.
>> >
>> >== Documentation ==
>> >http://wiki.apache.org/incubator/AirflowProposal
>> >
>> >== Initial Source ==
>> >https://github.com/airbnb/airflow
>> >
>> >== Source and Intellectual Property Submission Plan ==
>> >
>> >As soon as Airflow is approved to join Apache Incubator, Airbnb will
>> >execute a Software Grant Agreement and the source code will be
>> >transitioned onto ASF infrastructure. The code is already licensed
>> >under the Apache Software License, version 2.0. We know of no legal
>> >encumberments that would inhibit the transfer of source code to the
>> >ASF.
>> >
>> >== External Dependencies ==
>> >
>> >The dependencies all have Apache compatible licenses.
>> >
>> > *
>> >[[
>> https://bitbucket.org/zzzeek/alembic/src/9538c3e1a71c946a53f8762e68e94cf
>> >bcb9f932f/LICENSE?fileviewer=file-view-default|alembic
>> >(MIT)]]
>> > * [[https://github.com/boto/boto/blob/develop/LICENSE|boto (MIT)]]
>> > * [[https://github.com/celery/celery/blob/master/LICENSE|celery
>>(BSD)]]
>> > * [[https://github.com/mher/chartkick.py/blob/master/LICENSE|chartkick
>> >(MIT)]]
>> > *
>> >[[
>> https://github.com/pyca/cryptography/blob/master/LICENSE.APACHE|cryptogr
>> >aphy
>> >(Apache 2.0/BSD)]]
>> > *
>> >[[
>> https://bitbucket.org/ned/coveragepy/src/b74c40b2c107db17f0775be5ec6c44f
>> >5e1cf5cbf/LICENSE.txt?fileviewer=file-view-default|coverage
>> >(Apache 2.0)]]
>> > *
>> >[[
>> https://github.com/coagulant/coveralls-python/blob/master/LICENCE|covera
>> >lls
>> >(MIT)]]
>> > * [[https://pypi.python.org/pypi/croniter|croniter (MIT)]]
>> > * [[https://github.com/uqfoundation/dill/blob/master/LICENSE|dill
>> (BSD)]]
>> > * [[https://github.com/docker/docker-py/blob/master/LICENSE|docker-py
>> >(Apache 2.0)]]
>> > *
>> >[[
>> https://bitbucket.org/fabian/filechunkio/src/84289d7599a207f575cb28db719
>> >dd9d44e880208/LICENCE?fileviewer=file-view-default|filechunkio
>> >(MIT)]]
>> > *
>> >[[
>> https://bitbucket.org/tarek/flake8/src/a209fb69350c572c9b2d7b4b09c7657be
>> >153be5e/LICENSE?fileviewer=file-view-default|flake8
>> >(MIT)]]
>> > * [[https://github.com/mitsuhiko/flask/blob/master/LICENSE|flask
>>(BSD)]]
>> > *
>> >[[
>> https://github.com/flask-admin/flask-admin/blob/master/LICENSE|flask-adm
>> >in
>> >(BSD)]]
>> > *
>>
>>>[[https://github.com/thadeusb/flask-cache/blob/master/LICENSE|flask-cach
>>>e
>> >(BSD)]]
>> > *
>> >[[
>> https://github.com/maxcountryman/flask-login/blob/master/LICENSE|flask-l
>> >ogin
>> >(MIT)]]
>> > * [[https://github.com/mher/flower/blob/master/LICENSE|flower (BSD)]]
>> > *
>> >[[
>> https://github.com/PythonCharmers/python-future/blob/master/LICENSE.txt|
>> >future
>> >(MIT)]]
>> > * [[https://github.com/benoitc/gunicorn/blob/master/LICENSE|gunicorn
>> >(MIT)]]
>> > *
>> >[[
>> https://github.com/youngwookim/hive-thrift-py/blob/master/setup.py|hive-
>> >thrift-py
>> >(Apache 2.0)]]
>> > * [[https://github.com/ipython/ipython/blob/master/COPYING.rst|ipython
>> >(BSD)]]
>> > * [[https://github.com/mitsuhiko/jinja2/blob/master/LICENSE|jinja2
>> >(BSD)]]
>> > *
>> >[[
>> https://github.com/waylan/Python-Markdown/blob/master/LICENSE.md|markdow
>> >n
>> >(BSD)]]
>> > * [[https://github.com/pydata/pandas/blob/master/LICENSE|pandas
>>(BSD)]]
>> > * [[https://pypi.python.org/pypi/Pygments|pygments (BSD)]]
>> > * pyhive
>> > * pydruid
>> > * PyOpenSSL
>> > * PySmbClient
>> > * python-dateutil
>> > * redis
>> > * requests
>> > * setproctitle
>> > * statsd
>> > * sphinx
>> > * sphinx-argparse
>> > * sphinx_rtd_theme
>> > * Sphinx-PyPI-upload
>> > * sqlalchemy (MIT)
>> > * thrift
>> > * jaydebeapi
>> > * mysqlclient
>> > * unicodecsv
>> > * slackclient
>> > * ldap3
>> > * Flask-WTF
>> > * lxml
>> > *
>>[[https://github.com/bgamble/pykerberos/blob/master/LICENSE|pykerberos
>> >(Apache 2.0)]]
>> > * [[https://github.com/pyca/bcrypt/blob/master/LICENSE|bcrypt (Apache
>> >2.0)]]
>> > *
>> >[[
>> https://github.com/maxcountryman/flask-bcrypt/blob/master/LICENSE|flask-
>> >bcrypt
>> >(BSD)]]
>> > * [[https://github.com/testing-cabal/mock/blob/master/LICENSE.txt|mock
>> >(BSD)]]
>> > * [[https://github.com/mtth/hdfs/blob/master/LICENSE|hdfs (MIT)]]
>> >
>> >== Cryptography ==
>> >
>> >None
>> >
>> >== Required Resources ==
>> >
>> >=== Mailing Lists ===
>> >
>> > * private@airflow.incubator.apache.org (moderated)
>> > * dev@airflow.incubator.apache.org
>> > * commits@airflow.incubator.apache.org
>> >
>> >=== Subversion Directory ===
>> >
>> >Git is the preferred source control system:
>>git://git.apache.org/Airflow
>> >
>> >=== Issue Tracking ===
>> >
>> >JIRA Airflow (Airflow)
>> >
>> >=== Other Resources ===
>> >
>> >The existing code already has unit tests, so we would like a Travis
>> >instance to run them whenever a new patch is submitted. This can be
>> >added after project creation.
>> >
>> >== Initial Committers ==
>> >
>> > * Maxime Beauchemin
>> > * Siddharth Anand
>> > * Chris Riccomini
>> > * Bolke de Bruin
>> > * Arthur Wiedmer
>> > * Dan Davydov
>> > * Jeremiah Lowin
>> > * Patrick Leo Tardif
>> >
>> >== Affiliations ==
>> >
>> > * Maxime Beauchemin (Airbnb)
>> > * Siddharth Anand (Agari)
>> > * Chris Riccomini (WePay)
>> > * Bolke de Bruin (ING)
>> > * Arthur Wiedmer (Airbnb)
>> > * Dan Davydov (Airbnb)
>> > * Jeremiah Lowin (Kokino)
>> > * Patrick Leo Tardif (Airbnb)
>> >
>> >== Sponsors ==
>> >
>> >=== Champion ===
>> >
>> >Chris Riccomini (WePay, Apache Samza PMC)
>> >
>> >=== Nominated Mentors ===
>> >
>> > * Chris Nauroth (HortonWorks, Apache Hadoop Committer/PMC Member,
>> >Apache ZooKeeper Committer, Apache Software Foundation Member)
>> > * Hitesh Shah (HortonWorks, Apache Hadoop Committer/PMC Member,
>> >Apache Ambari Committer/PMC Member, Apache Tez Committer/PMC Member,
>> >Apache Software Foundation Member)
>> > * Jakob Homan (OfferUp, Apache Hadoop Committer/PMC Member, Apache
>> >Kafka Committer/PMC Member, Apache Samza Committer/PMC Member, Apache
>> >Giraph Committer/PMC Member, Apache Software Foundation Member)
>> >
>> >=== Sponsoring Entity ===
>> >
>> >We are requesting the Incubator to sponsor this project.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: general-help@incubator.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: AirPal to Apache? Re: [VOTE] Accept Airflow into the Incubator
Posted by "John D. Ament" <jo...@apache.org>.
You would need to ask AirBnB.
John
On Fri, Mar 25, 2016 at 9:20 AM Adunuthula, Seshu <sa...@ebay.com>
wrote:
> Any plans from AirBnB to bring AirPal to Apache?
>
>
>
> On 3/24/16, 8:00 PM, "Siddharth Anand" <sa...@agari.com.INVALID> wrote:
>
> >Following the discussion earlier:
> > https://s.apache.org/AirflowDiscussion
> >
> >I would like to call a VOTE for accepting Airflow as a new incubator
> >project.
> >
> >The proposal is available at:
> >https://wiki.apache.org/incubator/AirflowProposal
> >
> >The proposal is also included at the bottom of this email.
> >
> >Vote is open until at least Tues, 29 March 2016, 23:59:00 PDT
> >[ ] +1 accept Airflow into the Apache Incubator
> >[ ] ±0
> >[ ] -1 because...
> >
> >+1 (non-binding)
> >
> >Thanks,
> >-s (Sid)
> >
> >
> >== Abstract ==
> >
> >Airflow is a workflow automation and scheduling system that can be
> >used to author and manage data pipelines.
> >
> >== Proposal ==
> >
> >Airflow provides a system for authoring and managing workflows a.k.a.
> >data pipelines a.k.a. DAGs (Directed Acyclic Graphs). The developer
> >authors DAGs in Python using an Airflow-provided framework. He/She
> >then executes the DAG using Airflow¹s scheduler or registers the DAG
> >for event-based execution. A web-based UI provides the developer with
> >a range of options for managing and viewing his/her data pipelines.
> >Background
> >
> >Airflow was developed at Airbnb to enable easier authorship and
> >management of DAGs than were possible with existing solutions such as
> >Oozie and Azkaban. For starters, both Oozie and Azkaban rely on one or
> >more XML or property files to be bundled together to define a
> >workflow. This separation of code and config can present a challenge
> >to understanding the DAG - in Azkaban, a DAG¹s structure is reflected
> >by its file system tree and one can find himself/herself traversing
> >the file system when inspecting or changing the structure of the DAG.
> >Airflow workflows, on the other hand, are simply and elegantly defined
> >in Python code, often a single file. Airflow merges the powerful
> >Web-based management aspects of projects like Azkaban and Oozie with
> >the simplicity and elegance of defining workflows in Python. Airflow,
> >less than a year old in terms of its Open Source launch, is currently
> >used in production environments in more than 30 companies and boasts
> >an active contributor list of more than 100 developers, the vast
> >majority of which (>95%) are outside of Airbnb.
> >
> >We would like to share it with the ASF and begin developing a
> >community of developers and users within Apache.
> >
> >== Rationale ==
> >
> >Many organizations (>30) already benefit from running Airflow to
> >manage data pipelines. Our 100+ contributors continue to provide
> >integrations with 3rd party systems through the implementation of new
> >hooks and operators, both of which are used in defining the tasks that
> >compose workflows.
> >
> >== Current Status ==
> >
> >=== Meritocracy ===
> >
> >Our intent with this incubator proposal is to start building a diverse
> >developer community around Airflow following the Apache meritocracy
> >model. Since Airflow was open-sourced in mid-2015, we have had fast
> >adoption and contributions by multiple organizations the world over.
> >We plan to continue to support new contributors and we will work to
> >actively promote those who contribute significantly to the project to
> >committers.
> >
> >=== Community ===
> >
> >Airflow is currently being used in over 30 companies. We hope to
> >extend our contributor base significantly and invite all those who are
> >interested in building large-scale distributed systems to participate.
> >
> >=== Core Developers ===
> >
> >Airflow is currently being developed by four engineers: Maxime
> >Beauchemin, Siddharth Anand, Bolke de Bruin, and Chris Riccomini.
> >Chris is a member of the Apache Samza PMC and a contributor to various
> >Apache projects, including Apache Kafka and Apache YARN. Maxime,
> >Siddharth, and Bolke have contributed to Airflow.
> >
> >=== Alignment ===
> >The ASF is the natural choice to host the Airflow project as its goal
> >of encouraging community-driven open-source projects fits with our
> >vision for Airflow.
> >
> >== Known Risks ==
> >
> >=== Orphaned Products ===
> >
> >The core developers plan to work part time on the project. There is
> >very little risk of Airflow being abandoned as all of our companies
> >rely on it.
> >
> >=== Inexperience with Open Source ===
> >
> >All of the core developers have experience with open source
> >development. Chris is a member of the Apache Samza PMC and a
> >contributor to various Apache projects, including Apache Kafka and
> >Apache YARN. Bolke is contributor on multiple open source projects and
> >a few Apache projects as well, including Apache Hive, Apache Hadoop,
> >and Apache Ranger.
> >
> >=== Homogeneous Developers ===
> >
> >The current core developers are all from different companies. Our
> >community of 100 contributors hail from over 30 different companies
> >from across the world.
> >
> >=== Reliance on Salaried Developers ===
> >
> >Currently, the only developer paid to work on this project is Maxime.
> >
> >=== Relationships with Other Apache Products ===
> >
> >Airflow is deeply integrated with Apache products. It currently
> >provides hooks and operators to enable workflows to leverage Apache
> >Pig, Apache Hive, Apache Spark, Apache Sqoop, Apache Hadoop, etcŠ We
> >plan to add support for other Apache projects in the future.
> >
> >=== An Excessive Fascination with the Apache Brand ===
> >
> >While we respect the reputation of the Apache brand and have no doubts
> >that it will attract contributors and users, our interest is primarily
> >to give Airflow a solid home as an open source project following an
> >established development model. We have also given reasons in the
> >Rationale and Alignment sections.
> >
> >== Documentation ==
> >http://wiki.apache.org/incubator/AirflowProposal
> >
> >== Initial Source ==
> >https://github.com/airbnb/airflow
> >
> >== Source and Intellectual Property Submission Plan ==
> >
> >As soon as Airflow is approved to join Apache Incubator, Airbnb will
> >execute a Software Grant Agreement and the source code will be
> >transitioned onto ASF infrastructure. The code is already licensed
> >under the Apache Software License, version 2.0. We know of no legal
> >encumberments that would inhibit the transfer of source code to the
> >ASF.
> >
> >== External Dependencies ==
> >
> >The dependencies all have Apache compatible licenses.
> >
> > *
> >[[
> https://bitbucket.org/zzzeek/alembic/src/9538c3e1a71c946a53f8762e68e94cf
> >bcb9f932f/LICENSE?fileviewer=file-view-default|alembic
> >(MIT)]]
> > * [[https://github.com/boto/boto/blob/develop/LICENSE|boto (MIT)]]
> > * [[https://github.com/celery/celery/blob/master/LICENSE|celery (BSD)]]
> > * [[https://github.com/mher/chartkick.py/blob/master/LICENSE|chartkick
> >(MIT)]]
> > *
> >[[
> https://github.com/pyca/cryptography/blob/master/LICENSE.APACHE|cryptogr
> >aphy
> >(Apache 2.0/BSD)]]
> > *
> >[[
> https://bitbucket.org/ned/coveragepy/src/b74c40b2c107db17f0775be5ec6c44f
> >5e1cf5cbf/LICENSE.txt?fileviewer=file-view-default|coverage
> >(Apache 2.0)]]
> > *
> >[[
> https://github.com/coagulant/coveralls-python/blob/master/LICENCE|covera
> >lls
> >(MIT)]]
> > * [[https://pypi.python.org/pypi/croniter|croniter (MIT)]]
> > * [[https://github.com/uqfoundation/dill/blob/master/LICENSE|dill
> (BSD)]]
> > * [[https://github.com/docker/docker-py/blob/master/LICENSE|docker-py
> >(Apache 2.0)]]
> > *
> >[[
> https://bitbucket.org/fabian/filechunkio/src/84289d7599a207f575cb28db719
> >dd9d44e880208/LICENCE?fileviewer=file-view-default|filechunkio
> >(MIT)]]
> > *
> >[[
> https://bitbucket.org/tarek/flake8/src/a209fb69350c572c9b2d7b4b09c7657be
> >153be5e/LICENSE?fileviewer=file-view-default|flake8
> >(MIT)]]
> > * [[https://github.com/mitsuhiko/flask/blob/master/LICENSE|flask (BSD)]]
> > *
> >[[
> https://github.com/flask-admin/flask-admin/blob/master/LICENSE|flask-adm
> >in
> >(BSD)]]
> > *
> >[[https://github.com/thadeusb/flask-cache/blob/master/LICENSE|flask-cache
> >(BSD)]]
> > *
> >[[
> https://github.com/maxcountryman/flask-login/blob/master/LICENSE|flask-l
> >ogin
> >(MIT)]]
> > * [[https://github.com/mher/flower/blob/master/LICENSE|flower (BSD)]]
> > *
> >[[
> https://github.com/PythonCharmers/python-future/blob/master/LICENSE.txt|
> >future
> >(MIT)]]
> > * [[https://github.com/benoitc/gunicorn/blob/master/LICENSE|gunicorn
> >(MIT)]]
> > *
> >[[
> https://github.com/youngwookim/hive-thrift-py/blob/master/setup.py|hive-
> >thrift-py
> >(Apache 2.0)]]
> > * [[https://github.com/ipython/ipython/blob/master/COPYING.rst|ipython
> >(BSD)]]
> > * [[https://github.com/mitsuhiko/jinja2/blob/master/LICENSE|jinja2
> >(BSD)]]
> > *
> >[[
> https://github.com/waylan/Python-Markdown/blob/master/LICENSE.md|markdow
> >n
> >(BSD)]]
> > * [[https://github.com/pydata/pandas/blob/master/LICENSE|pandas (BSD)]]
> > * [[https://pypi.python.org/pypi/Pygments|pygments (BSD)]]
> > * pyhive
> > * pydruid
> > * PyOpenSSL
> > * PySmbClient
> > * python-dateutil
> > * redis
> > * requests
> > * setproctitle
> > * statsd
> > * sphinx
> > * sphinx-argparse
> > * sphinx_rtd_theme
> > * Sphinx-PyPI-upload
> > * sqlalchemy (MIT)
> > * thrift
> > * jaydebeapi
> > * mysqlclient
> > * unicodecsv
> > * slackclient
> > * ldap3
> > * Flask-WTF
> > * lxml
> > * [[https://github.com/bgamble/pykerberos/blob/master/LICENSE|pykerberos
> >(Apache 2.0)]]
> > * [[https://github.com/pyca/bcrypt/blob/master/LICENSE|bcrypt (Apache
> >2.0)]]
> > *
> >[[
> https://github.com/maxcountryman/flask-bcrypt/blob/master/LICENSE|flask-
> >bcrypt
> >(BSD)]]
> > * [[https://github.com/testing-cabal/mock/blob/master/LICENSE.txt|mock
> >(BSD)]]
> > * [[https://github.com/mtth/hdfs/blob/master/LICENSE|hdfs (MIT)]]
> >
> >== Cryptography ==
> >
> >None
> >
> >== Required Resources ==
> >
> >=== Mailing Lists ===
> >
> > * private@airflow.incubator.apache.org (moderated)
> > * dev@airflow.incubator.apache.org
> > * commits@airflow.incubator.apache.org
> >
> >=== Subversion Directory ===
> >
> >Git is the preferred source control system: git://git.apache.org/Airflow
> >
> >=== Issue Tracking ===
> >
> >JIRA Airflow (Airflow)
> >
> >=== Other Resources ===
> >
> >The existing code already has unit tests, so we would like a Travis
> >instance to run them whenever a new patch is submitted. This can be
> >added after project creation.
> >
> >== Initial Committers ==
> >
> > * Maxime Beauchemin
> > * Siddharth Anand
> > * Chris Riccomini
> > * Bolke de Bruin
> > * Arthur Wiedmer
> > * Dan Davydov
> > * Jeremiah Lowin
> > * Patrick Leo Tardif
> >
> >== Affiliations ==
> >
> > * Maxime Beauchemin (Airbnb)
> > * Siddharth Anand (Agari)
> > * Chris Riccomini (WePay)
> > * Bolke de Bruin (ING)
> > * Arthur Wiedmer (Airbnb)
> > * Dan Davydov (Airbnb)
> > * Jeremiah Lowin (Kokino)
> > * Patrick Leo Tardif (Airbnb)
> >
> >== Sponsors ==
> >
> >=== Champion ===
> >
> >Chris Riccomini (WePay, Apache Samza PMC)
> >
> >=== Nominated Mentors ===
> >
> > * Chris Nauroth (HortonWorks, Apache Hadoop Committer/PMC Member,
> >Apache ZooKeeper Committer, Apache Software Foundation Member)
> > * Hitesh Shah (HortonWorks, Apache Hadoop Committer/PMC Member,
> >Apache Ambari Committer/PMC Member, Apache Tez Committer/PMC Member,
> >Apache Software Foundation Member)
> > * Jakob Homan (OfferUp, Apache Hadoop Committer/PMC Member, Apache
> >Kafka Committer/PMC Member, Apache Samza Committer/PMC Member, Apache
> >Giraph Committer/PMC Member, Apache Software Foundation Member)
> >
> >=== Sponsoring Entity ===
> >
> >We are requesting the Incubator to sponsor this project.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>