[PROPOSAL] : Airflow

[Siddharth Anand <sa...@agari.com.INVALID>]

https://wiki.apache.org/incubator/AirflowProposal

Thoughts and comments are welcome!
-s (Sid)

Re: [PROPOSAL] : Airflow

[Siddharth Anand <sa...@agari.com.INVALID>]

Hi Stian,
Thanks for the feedback. We are committed to moving towards Apache
community-friendly licensing as documented on
http://www.apache.org/legal/resolved.html#category-x

I've added this to our Roadmap:
https://github.com/airbnb/airflow/wiki/Roadmap and I expect to tackle it
during the incubation phase.

Regarding our Google group - yes. We will move to the
dev@airflow.incubator.apache.org mailing list.  We currently have *258
members*, with about 10 new members being added daily! If you have any
thoughts on how to make that transition more amenable to our community,
they are welcome. For one, the google group offers more functionality than
the old-style mailing lists. I don't mind moving. I would like to keep the
old group's messages searchable and provide an announcement feature to
point people to the Apache mailing list so they know where to go next if
they land on the Google group. Some threads will unfortunately be
decapitated, but we can provide a few months' time for our users to
transition to the Apache mailing lists, wrapping up topic threads or
copying them over.  We will also need to stop accepting new members and
posts in the existing group, but I think that is pretty simple to do.

Thanks for the heads up on CWL and Taverna. We will definitely look into
both. On our side, our user base is growing faster than we can keep up.
Hence, we are investing in growing the committer base and documentation,
and in leveraging processes in order to better meet the growing needs of
our community. We hope to reduce turn-around time on PR reviews/merges,
time between releases, bringing new developers on board, and converting
contributors to committers.


-s

On Mon, Mar 21, 2016 at 4:23 PM, Stian Soiland-Reyes <st...@apache.org>
wrote:

> Airflow sounds like an interesting workflow project!
>
>
> You will need to review the licenses of dependencies - some of them
> are not compatible with ASF policy [1], e.g.
>
> https://pypi.python.org/pypi/psycopg2
> is LGPL, which is not permitted.
>
> (I didn't check all of them)
>
> You may be able to work around this particular one if you are using
> Python's DB API 2.0 and don't have an explicit dependency on psycopg2
> (e.g. so a downstream user can reasonably use Airflow without
> psycopg2)
>
>
> (Note: you can solve this as part of incubation process - but say if
> your project strongly used 15 GPL and LGPL dependencies, then your
> project might want to reconsider if it would be worth the effort)
>
>
>
> I recognize the challenge of engaging extension developers to also
> care about the 'core'. Moving to an "Apache Way" open development will
> probably help for this, as all dev discussions are in the open. Is
> your plan to move the existing Airflow Google Groups to what will be
> the dev@airflow.incubator.apache.org mailing list?
>
>
> Have you been talking to the SciDap folks doing Common Workflow
> Language [3] support with Airflow? That could be one angle to look at
> increasing the user base. We're also working on Docker and CWL support
> in Apache Taverna [4] - so I'm interested in seeing what we could have
> in common!
>
>
> [1] http://www.apache.org/legal/resolved.html
> [2] https://github.com/SciDAP/scidap
> [3] http://www.commonwl.org/
> [4] http://taverna.incubator.apache.org/
>
> On 17 March 2016 at 00:28, Siddharth Anand <sa...@agari.com.invalid>
> wrote:
> > https://wiki.apache.org/incubator/AirflowProposal
> >
> > Thoughts and comments are welcome!
> > -s (Sid)
>
>
>
> --
> Stian Soiland-Reyes
> Apache Taverna (incubating), Apache Commons RDF (incubating)
> http://orcid.org/0000-0001-9842-9718
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>

Re: [PROPOSAL] : Airflow

[Stian Soiland-Reyes <st...@apache.org>]

Airflow sounds like an interesting workflow project!


You will need to review the licenses of dependencies - some of them
are not compatible with ASF policy [1], e.g.

https://pypi.python.org/pypi/psycopg2
is LGPL, which is not permitted.

(I didn't check all of them)

You may be able to work around this particular one if you are using
Python's DB API 2.0 and don't have an explicit dependency on psycopg2
(e.g. so a downstream user can reasonably use Airflow without
psycopg2)


(Note: you can solve this as part of incubation process - but say if
your project strongly used 15 GPL and LGPL dependencies, then your
project might want to reconsider if it would be worth the effort)



I recognize the challenge of engaging extension developers to also
care about the 'core'. Moving to an "Apache Way" open development will
probably help for this, as all dev discussions are in the open. Is
your plan to move the existing Airflow Google Groups to what will be
the dev@airflow.incubator.apache.org mailing list?


Have you been talking to the SciDap folks doing Common Workflow
Language [3] support with Airflow? That could be one angle to look at
increasing the user base. We're also working on Docker and CWL support
in Apache Taverna [4] - so I'm interested in seeing what we could have
in common!


[1] http://www.apache.org/legal/resolved.html
[2] https://github.com/SciDAP/scidap
[3] http://www.commonwl.org/
[4] http://taverna.incubator.apache.org/

On 17 March 2016 at 00:28, Siddharth Anand <sa...@agari.com.invalid> wrote:
> https://wiki.apache.org/incubator/AirflowProposal
>
> Thoughts and comments are welcome!
> -s (Sid)



-- 
Stian Soiland-Reyes
Apache Taverna (incubating), Apache Commons RDF (incubating)
http://orcid.org/0000-0001-9842-9718

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [PROPOSAL] : Airflow

[Chris Riccomini <cr...@apache.org>]

@Jake, may I join the IPMC? :)

On Thu, Mar 17, 2016 at 9:06 AM, Jake Farrell <jf...@apache.org> wrote:

> Hey Chris
> As an officer you can be a champion if your project you chair (Samza) is
> sponsoring the incoming podling with the intent of merging the two
> communities - "Where the Champion is not a Member of the Foundation (i.e.
> is an Officer only), the Champion shall be a member of the PMC of the
> Sponsor.", did not think this was the case based on the proposal.
>
> Makes sense about the committers and developing the community during
> incubation, was just curious because there where some people with 20+
> commits not in the initial committers list. Was not looking for a change,
> more that there had been some thought behind this exclusion and how the
> project intended to start developing that community
>
> -Jake
>
> On Thu, Mar 17, 2016 at 11:58 AM, Chris Riccomini <cr...@apache.org>
> wrote:
>
> > > The proposal has the incubator listed as the sponsor and a quick check
> > shows Chris Riccomini is not on the IPMC or a member currently, details
> on
> > roles are available at [1].
> >
> > I was under the impression that I could sponsor because I am an officer:
> >
> > "A candidate project shall be sponsored by an Officer
> > <http://www.apache.org/foundation/index.html> or Member
> > <http://www.apache.org/foundation/members.html> of the Foundation"
> >
> > The officer page lists me as a VP of Apache Samza. Perhaps I'm
> misreading?
> >
> > > Mailing lists are in the old format
> >
> > Fixed.
> >
> > > Github shows 109 contributors, but the proposal lists only 6 initial
> > committers, where any of the other existing contributors considered for
> the
> > proposal?
> >
> > In terms of the high ratio of contributors to committers (109:6), the
> > reason is as follows. Most contributions are to the fringes of the system
> > and are in the form of 3rd party contributed hooks/operators (for 3rd
> party
> > integrations). For example, if you want to use Airflow to speak to
> > Cassandra, you may want to contribute a Cassandra Hook and CRUD
> operators.
> > Airflow allows for this extensibility and allows for our users to adapt
> > Airflow to their needs. However, the committer list is limited to folks
> who
> > will touch the internals of the system (e.g. scheduler, metadata
> > management, etc...). Overtime, we would like other contributors to get
> > familiar with the internals to the point where they can also become
> > maintainers, but they don't currently have the expertise.
> >
> >
> > On Thu, Mar 17, 2016 at 8:16 AM, Scott Deboy <sc...@gmail.com>
> > wrote:
> >
> > > Ll
> > > On Mar 17, 2016 6:51 AM, "Jake Farrell" <jf...@apache.org> wrote:
> > >
> > > > Hi Siddharth
> > > > Thanks for drafting a proposal and looking to bring Airflow to the
> > Apache
> > > > Incubator. Overall the proposal looks good, just a couple comments.
> The
> > > > proposal has the incubator listed as the sponsor and a quick check
> > shows
> > > > Chris Riccomini is not on the IPMC or a member currently, details on
> > > roles
> > > > are available at [1]. Not a blocker as you have members listed in
> your
> > > > group of mentors and can ask that they step up as the champion.
> > > >
> > > > Mailing lists are in the old format, they should be
> > > > - private@airflow.incubator.apache.org (moderated)
> > > > - dev@airflow.incubator.apache.org
> > > > - commits@airflow.incubator.apache.org
> > > >
> > > > Github shows 109 contributors, but the proposal lists only 6 initial
> > > > committers, where any of the other existing contributors considered
> for
> > > the
> > > > proposal?
> > > >
> > > > When you submit the proposal please make sure to include the entire
> > > > proposal in the email. If you have any questions please let us know
> > > >
> > > > -Jake
> > > >
> > > >
> > > > [1]:
> > > >
> > > >
> > >
> >
> http://incubator.apache.org/incubation/Roles_and_Responsibilities.html#Champion
> > > >
> > > > On Wed, Mar 16, 2016 at 8:28 PM, Siddharth Anand
> > > <sanand@agari.com.invalid
> > > > >
> > > > wrote:
> > > >
> > > > > https://wiki.apache.org/incubator/AirflowProposal
> > > > >
> > > > > Thoughts and comments are welcome!
> > > > > -s (Sid)
> > > > >
> > > >
> > >
> >
>

Re: [PROPOSAL] : Airflow

[Jake Farrell <jf...@apache.org>]

Hey Chris
As an officer you can be a champion if your project you chair (Samza) is
sponsoring the incoming podling with the intent of merging the two
communities - "Where the Champion is not a Member of the Foundation (i.e.
is an Officer only), the Champion shall be a member of the PMC of the
Sponsor.", did not think this was the case based on the proposal.

Makes sense about the committers and developing the community during
incubation, was just curious because there where some people with 20+
commits not in the initial committers list. Was not looking for a change,
more that there had been some thought behind this exclusion and how the
project intended to start developing that community

-Jake

On Thu, Mar 17, 2016 at 11:58 AM, Chris Riccomini <cr...@apache.org>
wrote:

> > The proposal has the incubator listed as the sponsor and a quick check
> shows Chris Riccomini is not on the IPMC or a member currently, details on
> roles are available at [1].
>
> I was under the impression that I could sponsor because I am an officer:
>
> "A candidate project shall be sponsored by an Officer
> <http://www.apache.org/foundation/index.html> or Member
> <http://www.apache.org/foundation/members.html> of the Foundation"
>
> The officer page lists me as a VP of Apache Samza. Perhaps I'm misreading?
>
> > Mailing lists are in the old format
>
> Fixed.
>
> > Github shows 109 contributors, but the proposal lists only 6 initial
> committers, where any of the other existing contributors considered for the
> proposal?
>
> In terms of the high ratio of contributors to committers (109:6), the
> reason is as follows. Most contributions are to the fringes of the system
> and are in the form of 3rd party contributed hooks/operators (for 3rd party
> integrations). For example, if you want to use Airflow to speak to
> Cassandra, you may want to contribute a Cassandra Hook and CRUD operators.
> Airflow allows for this extensibility and allows for our users to adapt
> Airflow to their needs. However, the committer list is limited to folks who
> will touch the internals of the system (e.g. scheduler, metadata
> management, etc...). Overtime, we would like other contributors to get
> familiar with the internals to the point where they can also become
> maintainers, but they don't currently have the expertise.
>
>
> On Thu, Mar 17, 2016 at 8:16 AM, Scott Deboy <sc...@gmail.com>
> wrote:
>
> > Ll
> > On Mar 17, 2016 6:51 AM, "Jake Farrell" <jf...@apache.org> wrote:
> >
> > > Hi Siddharth
> > > Thanks for drafting a proposal and looking to bring Airflow to the
> Apache
> > > Incubator. Overall the proposal looks good, just a couple comments. The
> > > proposal has the incubator listed as the sponsor and a quick check
> shows
> > > Chris Riccomini is not on the IPMC or a member currently, details on
> > roles
> > > are available at [1]. Not a blocker as you have members listed in your
> > > group of mentors and can ask that they step up as the champion.
> > >
> > > Mailing lists are in the old format, they should be
> > > - private@airflow.incubator.apache.org (moderated)
> > > - dev@airflow.incubator.apache.org
> > > - commits@airflow.incubator.apache.org
> > >
> > > Github shows 109 contributors, but the proposal lists only 6 initial
> > > committers, where any of the other existing contributors considered for
> > the
> > > proposal?
> > >
> > > When you submit the proposal please make sure to include the entire
> > > proposal in the email. If you have any questions please let us know
> > >
> > > -Jake
> > >
> > >
> > > [1]:
> > >
> > >
> >
> http://incubator.apache.org/incubation/Roles_and_Responsibilities.html#Champion
> > >
> > > On Wed, Mar 16, 2016 at 8:28 PM, Siddharth Anand
> > <sanand@agari.com.invalid
> > > >
> > > wrote:
> > >
> > > > https://wiki.apache.org/incubator/AirflowProposal
> > > >
> > > > Thoughts and comments are welcome!
> > > > -s (Sid)
> > > >
> > >
> >
>

Re: [PROPOSAL] : Airflow

[Chris Riccomini <cr...@apache.org>]

> The proposal has the incubator listed as the sponsor and a quick check
shows Chris Riccomini is not on the IPMC or a member currently, details on
roles are available at [1].

I was under the impression that I could sponsor because I am an officer:

"A candidate project shall be sponsored by an Officer
<http://www.apache.org/foundation/index.html> or Member
<http://www.apache.org/foundation/members.html> of the Foundation"

The officer page lists me as a VP of Apache Samza. Perhaps I'm misreading?

> Mailing lists are in the old format

Fixed.

> Github shows 109 contributors, but the proposal lists only 6 initial
committers, where any of the other existing contributors considered for the
proposal?

In terms of the high ratio of contributors to committers (109:6), the
reason is as follows. Most contributions are to the fringes of the system
and are in the form of 3rd party contributed hooks/operators (for 3rd party
integrations). For example, if you want to use Airflow to speak to
Cassandra, you may want to contribute a Cassandra Hook and CRUD operators.
Airflow allows for this extensibility and allows for our users to adapt
Airflow to their needs. However, the committer list is limited to folks who
will touch the internals of the system (e.g. scheduler, metadata
management, etc...). Overtime, we would like other contributors to get
familiar with the internals to the point where they can also become
maintainers, but they don't currently have the expertise.


On Thu, Mar 17, 2016 at 8:16 AM, Scott Deboy <sc...@gmail.com> wrote:

> Ll
> On Mar 17, 2016 6:51 AM, "Jake Farrell" <jf...@apache.org> wrote:
>
> > Hi Siddharth
> > Thanks for drafting a proposal and looking to bring Airflow to the Apache
> > Incubator. Overall the proposal looks good, just a couple comments. The
> > proposal has the incubator listed as the sponsor and a quick check shows
> > Chris Riccomini is not on the IPMC or a member currently, details on
> roles
> > are available at [1]. Not a blocker as you have members listed in your
> > group of mentors and can ask that they step up as the champion.
> >
> > Mailing lists are in the old format, they should be
> > - private@airflow.incubator.apache.org (moderated)
> > - dev@airflow.incubator.apache.org
> > - commits@airflow.incubator.apache.org
> >
> > Github shows 109 contributors, but the proposal lists only 6 initial
> > committers, where any of the other existing contributors considered for
> the
> > proposal?
> >
> > When you submit the proposal please make sure to include the entire
> > proposal in the email. If you have any questions please let us know
> >
> > -Jake
> >
> >
> > [1]:
> >
> >
> http://incubator.apache.org/incubation/Roles_and_Responsibilities.html#Champion
> >
> > On Wed, Mar 16, 2016 at 8:28 PM, Siddharth Anand
> <sanand@agari.com.invalid
> > >
> > wrote:
> >
> > > https://wiki.apache.org/incubator/AirflowProposal
> > >
> > > Thoughts and comments are welcome!
> > > -s (Sid)
> > >
> >
>

Re: [PROPOSAL] : Airflow

[Scott Deboy <sc...@gmail.com>]

Ll
On Mar 17, 2016 6:51 AM, "Jake Farrell" <jf...@apache.org> wrote:

> Hi Siddharth
> Thanks for drafting a proposal and looking to bring Airflow to the Apache
> Incubator. Overall the proposal looks good, just a couple comments. The
> proposal has the incubator listed as the sponsor and a quick check shows
> Chris Riccomini is not on the IPMC or a member currently, details on roles
> are available at [1]. Not a blocker as you have members listed in your
> group of mentors and can ask that they step up as the champion.
>
> Mailing lists are in the old format, they should be
> - private@airflow.incubator.apache.org (moderated)
> - dev@airflow.incubator.apache.org
> - commits@airflow.incubator.apache.org
>
> Github shows 109 contributors, but the proposal lists only 6 initial
> committers, where any of the other existing contributors considered for the
> proposal?
>
> When you submit the proposal please make sure to include the entire
> proposal in the email. If you have any questions please let us know
>
> -Jake
>
>
> [1]:
>
> http://incubator.apache.org/incubation/Roles_and_Responsibilities.html#Champion
>
> On Wed, Mar 16, 2016 at 8:28 PM, Siddharth Anand <sanand@agari.com.invalid
> >
> wrote:
>
> > https://wiki.apache.org/incubator/AirflowProposal
> >
> > Thoughts and comments are welcome!
> > -s (Sid)
> >
>

Re: [PROPOSAL] : Airflow

[Jake Farrell <jf...@apache.org>]

Hi Siddharth
Thanks for drafting a proposal and looking to bring Airflow to the Apache
Incubator. Overall the proposal looks good, just a couple comments. The
proposal has the incubator listed as the sponsor and a quick check shows
Chris Riccomini is not on the IPMC or a member currently, details on roles
are available at [1]. Not a blocker as you have members listed in your
group of mentors and can ask that they step up as the champion.

Mailing lists are in the old format, they should be
- private@airflow.incubator.apache.org (moderated)
- dev@airflow.incubator.apache.org
- commits@airflow.incubator.apache.org

Github shows 109 contributors, but the proposal lists only 6 initial
committers, where any of the other existing contributors considered for the
proposal?

When you submit the proposal please make sure to include the entire
proposal in the email. If you have any questions please let us know

-Jake


[1]:
http://incubator.apache.org/incubation/Roles_and_Responsibilities.html#Champion

On Wed, Mar 16, 2016 at 8:28 PM, Siddharth Anand <sa...@agari.com.invalid>
wrote:

> https://wiki.apache.org/incubator/AirflowProposal
>
> Thoughts and comments are welcome!
> -s (Sid)
>