You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@incubator.apache.org by Bill Graham <bi...@gmail.com> on 2017/06/16 20:41:21 UTC

[VOTE] Heron to enter Apache Incubator

Hi,

Based on the discussion on the incubator mailing list[1] I would like to
call a vote to add Heron to the Apache Incubator.

The full proposal is available below, and is also available on the Apache
Incubator wiki at:
    https://wiki.apache.org/incubator/HeronProposal

Please vote:
  [ ] +1, bring Heron into Incubator
  [ ] -1, do not bring Heron into Incubator, because...

The vote will open for 7 days until Friday June 23 at 14:00 PT.

Thank you

1 -
https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E



= Heron Proposal =

= Abstract =
Heron is a real-time, distributed, fault-tolerant stream processing engine
initially developed by Twitter.

= Proposal =

Heron is a real-time stream processing engine built for high performance,
ease of manageability, performance predictability and developer
productivity[1]. We wish to develop a community around Heron to increase
contributions and see Heron thrive in an open forum.

= Background =

Heron provides the ability for developers to compose directed acyclic
graphs (DAGs) of real-time query execution logic (i.e. a topology) and
submit the topology to execute on a pluggable job scheduling system (e.g.,
Apache Aurora, YARN, Marathon, etc). Users can employ either the native
Heron API or the Apache Storm API to develop the topology. Heron supports
the Storm API for ease of migration, but beyond that Heron’s architecture
differs considerably from Storm’s.

Users submit a topology to the scheduler using the Heron client, which uses
the Heron binary libraries to deploy all daemons required to run and manage
the topology. The topology therefore has no reliance on centrally managed
Heron services, only on a generic job scheduling system, which lends itself
well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN (among
others).

The scheduler runs each topology as a job consisting of multiple
containers. One of the containers runs the topology master, responsible for
managing the topology. The remaining containers each runs a stream manager
responsible for data routing, a metrics manager that collects and reports
various metrics and a number of processes called Heron instances which run
the user-defined logic on the stream of tuples. Parallelism is achieved via
process-based isolation of Heron instances, which provides predictable
performance while simplifying debugging. The containers are allocated and
managed by the scheduler framework based on resource availability of nodes
in the cluster. The metadata for the topology, such as the physical plan
and execution details, are stored in the pluggable Heron State Manager
(e.g. Apache ZooKeeper).

= Rationale =

Heron is a general-purpose, modular and extensible platform that can be
leveraged to support common, real-time analytics use cases. There is an
increasing demand for open-source, scalable real-time analytics systems. We
believe that Heron can be leveraged by other organizations to build
streaming applications that can benefit from its robustness, high
performance, adaptability to cloud environments and ease of use. Moreover,
we hope that open-sourcing Heron will help to further evolve the technology
as the project attracts contributors with diverse backgrounds and areas of
expertise.

We believe the Apache foundation is a great fit as the long-term home for
Heron, as it provides an established process for community-driven
development and decision making by consensus. This is exactly the model we
want for future Heron development.

= Initial Goals =

 * Move the existing codebase, website, documentation, and mailing lists to
Apache-hosted infrastructure.
 * Integrate with the Apache development process.
 * Ensure all dependencies are compliant with Apache License version 2.0.
 * Incrementally develop and release per Apache guidelines.

= Current Status =

Heron is a stable project used in production at Twitter since 2014 and open
sourced under the ASL v2 license in 2016. The Heron source code is
currently hosted at github.com (https://github.com/twitter/heron), which
will seed the Apache git repository.

= Meritocracy =

By submitting this incubator proposal, we’re expressing our intent to build
a diverse developer community around Heron that will conduct itself
according to The Apache Way and use a meritocratic means of building it's
committer base. Several companies and universities have already expressed
interest in and contributed to Heron. Our goal is to grow the Heron
community by encouraging open communication, contribution and participation
of all types, and ensuring that contributors are recognized appropriately.

= Community =

Heron is currently being used by Twitter, Google, Machine Zone and
ndustrial.io and has received significant contributions by Microsoft and
Streamlio. By bringing Heron into the Apache ecosystem, we believe we can
attract even more developers who are interested in creating real-time
systems to build the project's contributor base.

== Core Developers ==

Current core developers are engineers from Twitter, Google, Microsoft and
Streamlio.

== Alignment ==

Heron utilizes a number of Apache technologies. Heron leverages Apache
ZooKeeper for coordination and has scheduler implementations to integrate
with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache REEF)
as well as spout implementations to integrate with Apache Kafka and metrics
implementations to integrate with Scribe. Heron also implements the Apache
Storm user-level API, which allows topologies written against Storm to run
in Heron. We believe that having Heron at Apache will help further the
growth of the streaming compute community, as well as encourage cooperation
and developer cross pollination with other Apache projects.

= Known Risks =

== Orphaned Products ==

The risk of the Heron project being abandoned is minimal. It is used in
production at Twitter and Google and other companies are evaluating or
adopting it for production use.

== Inexperience with Open Source ==

All of the core contributors to the project have considerable experience
with open source software development. Bill Graham[2], Ashvin Agrawal[3]
and Supun Kamburugamuve[4], committers on the project, are PMCs on other
Apache projects and Bill and Ashvin have gone through the Apache incubator
process. Twitter has already donated numerous projects to the ASF (e.g.,
Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be mentored
by experienced ASF members that can help with any roadblocks.

== Homogenous Developers ==

Initial committers come from 5 separate organizations. Our intention is
increase the diversity of contributing developers and their affiliations.
To date github contributions have come from approximately 50 contributors
from outside the Twitter team.

== Reliance on Salaried Developers ==

It is expected that Heron development will occur on both salaried time and
on volunteer time. The majority of initial committers are paid by their
employers to contribute to this project. We are committed to recruiting
additional committers from other organizations as well as non-salaried
committers to join project.

== Relationships with Other Apache Products ==

As mentioned in the Alignment section, Heron implements the Apache Storm
API and integrates with multiple Apache schedulers (Apache Mesos, Apache
Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache
Thrift.

== An Excessive Fascination with the Apache Brand ==

Heron's popularity is growing in the streaming compute space and we are
long time supporters of the Apache brand. This proposal is not for the
purpose of generating publicity through. Rather, the primary benefits to
joining Apache are those of community building and open decision making
outlined in the Rationale section.

== Documentation ==

This proposal exists online as
http://wiki.apache.org/incubator/HeronProposal. Extensive documentation can
be found on github at https://twitter.github.io/heron and the source code
is well documented.

== Source and Intellectual Property Submission Plan ==

The Heron codebase is currently hosted on Github:
https://github.com/twitter/heron. During incubation, the codebase will be
migrated to Apache infrastructure. The source code is already ASF 2.0
licensed.

== External Dependencies ==

All external libraries have ASF 2.0 compatible licenses except for pylint.
The pylint library is GPL licensed, but is only used for pre-build Python
style checks and is neither bundled with, nor relied upon by, the Heron
source or binary release artifacts.

== Cryptography ==

Heron does not use any cryptography libraries.

= Required Resources =

== Mailing lists ==

 * private@heron.incubator.apache.org (with moderated subscriptions)
 * dev@heron.incubator.apache.org
 * commits@heron.incubator.apache.org
 * user@heron.incubator.apache.org

== Subversion Directory ==

Git is the preferred source control system: git://git.apache.org/heron

== Issue Tracking ==

JIRA: Heron (HERON)

== Initial Committers ==

 * Andrew Jorgensen (andrew at andrewjorgensen dot com)
 * Ashvin Agrawal (ashvin at apache dot org)*
 * Avrilia Floratou (avrilia dot floratou at gmail dot com)
 * Bill Graham (billgraham at apache dot org)*
 * Brian Hatfield (bmhatfield at gmail dot com)
 * Chris Kellogg (cckellogg at gmail dot com)
 * Huijun Wu (huijun dot wu dot 2010 at gmail dot com)
 * Karthik Ramasamy (karthik at gmail dot com)
 * Maosong Fu (maosongfu at gmail dot com)
 * Neng Lu(freeneng at gmail dot com)
 * Runhang Li (obj dot runhang at gmail dot com)
 * Sanjeev Kulkarni (sanjeevrk at gmail dot com)
 * Supun Kamburugamuve (supun at apache dot org)*
 * Thomas Sun (tom dot ssf at gmail dot com)
 * Yaliang Wang (yaliang dot w dot wang at ieee dot org)

== Affiliations ==

 * Andrew Jorgensen (Google)
 * Ashvin Agrawal (Microsoft)
 * Avrilia Floratou (Microsoft)
 * Bill Graham (Twitter)
 * Brian Hatfield (Google)
 * Chris Kellogg (Twitter)
 * Huijun Wu (Twitter)
 * Karthik Ramasamy (Streamlio)
 * Maosong Fu (Twitter)
 * Neng Lu (Twitter)
 * Runhang Li (Twitter)
 * Sanjeev Kulkarni (Streamlio)
 * Supun Kamburugamuve (Indiana University)
 * Thomas Sun (Twitter)
 * Yaliang Wang (Twitter)

= Sponsors =

== Champion ==

 * Julien Le Dem (julien at apache dot org)

== Nominated Mentors ==

 * Jake Farrell (jfarrell at apache dot org)
 * Jacques Nadeau (jacques at apache dot org)
 * Julien Le Dem (julien at apache dot org)
 * P. Taylor Goetz (ptgoetz at apache dot org)

== Sponsoring Entity ==

The Apache Incubator

== Footnotes ==

 * 1 - Papers detailing Heron are available at
http://dl.acm.org/citation.cfm?id=2742788 and
http://sites.computer.org/debull/A15dec/p15.pdf.
 * 2 - http://home.apache.org/phonebook.html?uid=billgraham
 * 3 - http://home.apache.org/phonebook.html?uid=ashvin
 * 4 - http://home.apache.org/phonebook.html?uid=supun

Re: [VOTE] Heron to enter Apache Incubator

Posted by Byung-Gon Chun <bg...@gmail.com>.
+1 (non-binding)


On Sat, Jun 17, 2017 at 5:41 AM, Bill Graham <bi...@gmail.com> wrote:

> Hi,
>
> Based on the discussion on the incubator mailing list[1] I would like to
> call a vote to add Heron to the Apache Incubator.
>
> The full proposal is available below, and is also available on the Apache
> Incubator wiki at:
>     https://wiki.apache.org/incubator/HeronProposal
>
> Please vote:
>   [ ] +1, bring Heron into Incubator
>   [ ] -1, do not bring Heron into Incubator, because...
>
> The vote will open for 7 days until Friday June 23 at 14:00 PT.
>
> Thank you
>
> 1 -
> https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7
> 786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E
>
>
>
> = Heron Proposal =
>
> = Abstract =
> Heron is a real-time, distributed, fault-tolerant stream processing engine
> initially developed by Twitter.
>
> = Proposal =
>
> Heron is a real-time stream processing engine built for high performance,
> ease of manageability, performance predictability and developer
> productivity[1]. We wish to develop a community around Heron to increase
> contributions and see Heron thrive in an open forum.
>
> = Background =
>
> Heron provides the ability for developers to compose directed acyclic
> graphs (DAGs) of real-time query execution logic (i.e. a topology) and
> submit the topology to execute on a pluggable job scheduling system (e.g.,
> Apache Aurora, YARN, Marathon, etc). Users can employ either the native
> Heron API or the Apache Storm API to develop the topology. Heron supports
> the Storm API for ease of migration, but beyond that Heron’s architecture
> differs considerably from Storm’s.
>
> Users submit a topology to the scheduler using the Heron client, which uses
> the Heron binary libraries to deploy all daemons required to run and manage
> the topology. The topology therefore has no reliance on centrally managed
> Heron services, only on a generic job scheduling system, which lends itself
> well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN (among
> others).
>
> The scheduler runs each topology as a job consisting of multiple
> containers. One of the containers runs the topology master, responsible for
> managing the topology. The remaining containers each runs a stream manager
> responsible for data routing, a metrics manager that collects and reports
> various metrics and a number of processes called Heron instances which run
> the user-defined logic on the stream of tuples. Parallelism is achieved via
> process-based isolation of Heron instances, which provides predictable
> performance while simplifying debugging. The containers are allocated and
> managed by the scheduler framework based on resource availability of nodes
> in the cluster. The metadata for the topology, such as the physical plan
> and execution details, are stored in the pluggable Heron State Manager
> (e.g. Apache ZooKeeper).
>
> = Rationale =
>
> Heron is a general-purpose, modular and extensible platform that can be
> leveraged to support common, real-time analytics use cases. There is an
> increasing demand for open-source, scalable real-time analytics systems. We
> believe that Heron can be leveraged by other organizations to build
> streaming applications that can benefit from its robustness, high
> performance, adaptability to cloud environments and ease of use. Moreover,
> we hope that open-sourcing Heron will help to further evolve the technology
> as the project attracts contributors with diverse backgrounds and areas of
> expertise.
>
> We believe the Apache foundation is a great fit as the long-term home for
> Heron, as it provides an established process for community-driven
> development and decision making by consensus. This is exactly the model we
> want for future Heron development.
>
> = Initial Goals =
>
>  * Move the existing codebase, website, documentation, and mailing lists to
> Apache-hosted infrastructure.
>  * Integrate with the Apache development process.
>  * Ensure all dependencies are compliant with Apache License version 2.0.
>  * Incrementally develop and release per Apache guidelines.
>
> = Current Status =
>
> Heron is a stable project used in production at Twitter since 2014 and open
> sourced under the ASL v2 license in 2016. The Heron source code is
> currently hosted at github.com (https://github.com/twitter/heron), which
> will seed the Apache git repository.
>
> = Meritocracy =
>
> By submitting this incubator proposal, we’re expressing our intent to build
> a diverse developer community around Heron that will conduct itself
> according to The Apache Way and use a meritocratic means of building it's
> committer base. Several companies and universities have already expressed
> interest in and contributed to Heron. Our goal is to grow the Heron
> community by encouraging open communication, contribution and participation
> of all types, and ensuring that contributors are recognized appropriately.
>
> = Community =
>
> Heron is currently being used by Twitter, Google, Machine Zone and
> ndustrial.io and has received significant contributions by Microsoft and
> Streamlio. By bringing Heron into the Apache ecosystem, we believe we can
> attract even more developers who are interested in creating real-time
> systems to build the project's contributor base.
>
> == Core Developers ==
>
> Current core developers are engineers from Twitter, Google, Microsoft and
> Streamlio.
>
> == Alignment ==
>
> Heron utilizes a number of Apache technologies. Heron leverages Apache
> ZooKeeper for coordination and has scheduler implementations to integrate
> with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache REEF)
> as well as spout implementations to integrate with Apache Kafka and metrics
> implementations to integrate with Scribe. Heron also implements the Apache
> Storm user-level API, which allows topologies written against Storm to run
> in Heron. We believe that having Heron at Apache will help further the
> growth of the streaming compute community, as well as encourage cooperation
> and developer cross pollination with other Apache projects.
>
> = Known Risks =
>
> == Orphaned Products ==
>
> The risk of the Heron project being abandoned is minimal. It is used in
> production at Twitter and Google and other companies are evaluating or
> adopting it for production use.
>
> == Inexperience with Open Source ==
>
> All of the core contributors to the project have considerable experience
> with open source software development. Bill Graham[2], Ashvin Agrawal[3]
> and Supun Kamburugamuve[4], committers on the project, are PMCs on other
> Apache projects and Bill and Ashvin have gone through the Apache incubator
> process. Twitter has already donated numerous projects to the ASF (e.g.,
> Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be mentored
> by experienced ASF members that can help with any roadblocks.
>
> == Homogenous Developers ==
>
> Initial committers come from 5 separate organizations. Our intention is
> increase the diversity of contributing developers and their affiliations.
> To date github contributions have come from approximately 50 contributors
> from outside the Twitter team.
>
> == Reliance on Salaried Developers ==
>
> It is expected that Heron development will occur on both salaried time and
> on volunteer time. The majority of initial committers are paid by their
> employers to contribute to this project. We are committed to recruiting
> additional committers from other organizations as well as non-salaried
> committers to join project.
>
> == Relationships with Other Apache Products ==
>
> As mentioned in the Alignment section, Heron implements the Apache Storm
> API and integrates with multiple Apache schedulers (Apache Mesos, Apache
> Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache
> Thrift.
>
> == An Excessive Fascination with the Apache Brand ==
>
> Heron's popularity is growing in the streaming compute space and we are
> long time supporters of the Apache brand. This proposal is not for the
> purpose of generating publicity through. Rather, the primary benefits to
> joining Apache are those of community building and open decision making
> outlined in the Rationale section.
>
> == Documentation ==
>
> This proposal exists online as
> http://wiki.apache.org/incubator/HeronProposal. Extensive documentation
> can
> be found on github at https://twitter.github.io/heron and the source code
> is well documented.
>
> == Source and Intellectual Property Submission Plan ==
>
> The Heron codebase is currently hosted on Github:
> https://github.com/twitter/heron. During incubation, the codebase will be
> migrated to Apache infrastructure. The source code is already ASF 2.0
> licensed.
>
> == External Dependencies ==
>
> All external libraries have ASF 2.0 compatible licenses except for pylint.
> The pylint library is GPL licensed, but is only used for pre-build Python
> style checks and is neither bundled with, nor relied upon by, the Heron
> source or binary release artifacts.
>
> == Cryptography ==
>
> Heron does not use any cryptography libraries.
>
> = Required Resources =
>
> == Mailing lists ==
>
>  * private@heron.incubator.apache.org (with moderated subscriptions)
>  * dev@heron.incubator.apache.org
>  * commits@heron.incubator.apache.org
>  * user@heron.incubator.apache.org
>
> == Subversion Directory ==
>
> Git is the preferred source control system: git://git.apache.org/heron
>
> == Issue Tracking ==
>
> JIRA: Heron (HERON)
>
> == Initial Committers ==
>
>  * Andrew Jorgensen (andrew at andrewjorgensen dot com)
>  * Ashvin Agrawal (ashvin at apache dot org)*
>  * Avrilia Floratou (avrilia dot floratou at gmail dot com)
>  * Bill Graham (billgraham at apache dot org)*
>  * Brian Hatfield (bmhatfield at gmail dot com)
>  * Chris Kellogg (cckellogg at gmail dot com)
>  * Huijun Wu (huijun dot wu dot 2010 at gmail dot com)
>  * Karthik Ramasamy (karthik at gmail dot com)
>  * Maosong Fu (maosongfu at gmail dot com)
>  * Neng Lu(freeneng at gmail dot com)
>  * Runhang Li (obj dot runhang at gmail dot com)
>  * Sanjeev Kulkarni (sanjeevrk at gmail dot com)
>  * Supun Kamburugamuve (supun at apache dot org)*
>  * Thomas Sun (tom dot ssf at gmail dot com)
>  * Yaliang Wang (yaliang dot w dot wang at ieee dot org)
>
> == Affiliations ==
>
>  * Andrew Jorgensen (Google)
>  * Ashvin Agrawal (Microsoft)
>  * Avrilia Floratou (Microsoft)
>  * Bill Graham (Twitter)
>  * Brian Hatfield (Google)
>  * Chris Kellogg (Twitter)
>  * Huijun Wu (Twitter)
>  * Karthik Ramasamy (Streamlio)
>  * Maosong Fu (Twitter)
>  * Neng Lu (Twitter)
>  * Runhang Li (Twitter)
>  * Sanjeev Kulkarni (Streamlio)
>  * Supun Kamburugamuve (Indiana University)
>  * Thomas Sun (Twitter)
>  * Yaliang Wang (Twitter)
>
> = Sponsors =
>
> == Champion ==
>
>  * Julien Le Dem (julien at apache dot org)
>
> == Nominated Mentors ==
>
>  * Jake Farrell (jfarrell at apache dot org)
>  * Jacques Nadeau (jacques at apache dot org)
>  * Julien Le Dem (julien at apache dot org)
>  * P. Taylor Goetz (ptgoetz at apache dot org)
>
> == Sponsoring Entity ==
>
> The Apache Incubator
>
> == Footnotes ==
>
>  * 1 - Papers detailing Heron are available at
> http://dl.acm.org/citation.cfm?id=2742788 and
> http://sites.computer.org/debull/A15dec/p15.pdf.
>  * 2 - http://home.apache.org/phonebook.html?uid=billgraham
>  * 3 - http://home.apache.org/phonebook.html?uid=ashvin
>  * 4 - http://home.apache.org/phonebook.html?uid=supun
>



-- 
Byung-Gon Chun

Re: [VOTE] Heron to enter Apache Incubator

Posted by Jake Farrell <jf...@apache.org>.
+1 binding

-Jake

On Fri, Jun 16, 2017 at 4:41 PM, Bill Graham <bi...@gmail.com> wrote:

> Hi,
>
> Based on the discussion on the incubator mailing list[1] I would like to
> call a vote to add Heron to the Apache Incubator.
>
> The full proposal is available below, and is also available on the Apache
> Incubator wiki at:
>     https://wiki.apache.org/incubator/HeronProposal
>
> Please vote:
>   [ ] +1, bring Heron into Incubator
>   [ ] -1, do not bring Heron into Incubator, because...
>
> The vote will open for 7 days until Friday June 23 at 14:00 PT.
>
> Thank you
>
> 1 -
> https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7
> 786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E
>
>
>
> = Heron Proposal =
>
> = Abstract =
> Heron is a real-time, distributed, fault-tolerant stream processing engine
> initially developed by Twitter.
>
> = Proposal =
>
> Heron is a real-time stream processing engine built for high performance,
> ease of manageability, performance predictability and developer
> productivity[1]. We wish to develop a community around Heron to increase
> contributions and see Heron thrive in an open forum.
>
> = Background =
>
> Heron provides the ability for developers to compose directed acyclic
> graphs (DAGs) of real-time query execution logic (i.e. a topology) and
> submit the topology to execute on a pluggable job scheduling system (e.g.,
> Apache Aurora, YARN, Marathon, etc). Users can employ either the native
> Heron API or the Apache Storm API to develop the topology. Heron supports
> the Storm API for ease of migration, but beyond that Heron’s architecture
> differs considerably from Storm’s.
>
> Users submit a topology to the scheduler using the Heron client, which uses
> the Heron binary libraries to deploy all daemons required to run and manage
> the topology. The topology therefore has no reliance on centrally managed
> Heron services, only on a generic job scheduling system, which lends itself
> well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN (among
> others).
>
> The scheduler runs each topology as a job consisting of multiple
> containers. One of the containers runs the topology master, responsible for
> managing the topology. The remaining containers each runs a stream manager
> responsible for data routing, a metrics manager that collects and reports
> various metrics and a number of processes called Heron instances which run
> the user-defined logic on the stream of tuples. Parallelism is achieved via
> process-based isolation of Heron instances, which provides predictable
> performance while simplifying debugging. The containers are allocated and
> managed by the scheduler framework based on resource availability of nodes
> in the cluster. The metadata for the topology, such as the physical plan
> and execution details, are stored in the pluggable Heron State Manager
> (e.g. Apache ZooKeeper).
>
> = Rationale =
>
> Heron is a general-purpose, modular and extensible platform that can be
> leveraged to support common, real-time analytics use cases. There is an
> increasing demand for open-source, scalable real-time analytics systems. We
> believe that Heron can be leveraged by other organizations to build
> streaming applications that can benefit from its robustness, high
> performance, adaptability to cloud environments and ease of use. Moreover,
> we hope that open-sourcing Heron will help to further evolve the technology
> as the project attracts contributors with diverse backgrounds and areas of
> expertise.
>
> We believe the Apache foundation is a great fit as the long-term home for
> Heron, as it provides an established process for community-driven
> development and decision making by consensus. This is exactly the model we
> want for future Heron development.
>
> = Initial Goals =
>
>  * Move the existing codebase, website, documentation, and mailing lists to
> Apache-hosted infrastructure.
>  * Integrate with the Apache development process.
>  * Ensure all dependencies are compliant with Apache License version 2.0.
>  * Incrementally develop and release per Apache guidelines.
>
> = Current Status =
>
> Heron is a stable project used in production at Twitter since 2014 and open
> sourced under the ASL v2 license in 2016. The Heron source code is
> currently hosted at github.com (https://github.com/twitter/heron), which
> will seed the Apache git repository.
>
> = Meritocracy =
>
> By submitting this incubator proposal, we’re expressing our intent to build
> a diverse developer community around Heron that will conduct itself
> according to The Apache Way and use a meritocratic means of building it's
> committer base. Several companies and universities have already expressed
> interest in and contributed to Heron. Our goal is to grow the Heron
> community by encouraging open communication, contribution and participation
> of all types, and ensuring that contributors are recognized appropriately.
>
> = Community =
>
> Heron is currently being used by Twitter, Google, Machine Zone and
> ndustrial.io and has received significant contributions by Microsoft and
> Streamlio. By bringing Heron into the Apache ecosystem, we believe we can
> attract even more developers who are interested in creating real-time
> systems to build the project's contributor base.
>
> == Core Developers ==
>
> Current core developers are engineers from Twitter, Google, Microsoft and
> Streamlio.
>
> == Alignment ==
>
> Heron utilizes a number of Apache technologies. Heron leverages Apache
> ZooKeeper for coordination and has scheduler implementations to integrate
> with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache REEF)
> as well as spout implementations to integrate with Apache Kafka and metrics
> implementations to integrate with Scribe. Heron also implements the Apache
> Storm user-level API, which allows topologies written against Storm to run
> in Heron. We believe that having Heron at Apache will help further the
> growth of the streaming compute community, as well as encourage cooperation
> and developer cross pollination with other Apache projects.
>
> = Known Risks =
>
> == Orphaned Products ==
>
> The risk of the Heron project being abandoned is minimal. It is used in
> production at Twitter and Google and other companies are evaluating or
> adopting it for production use.
>
> == Inexperience with Open Source ==
>
> All of the core contributors to the project have considerable experience
> with open source software development. Bill Graham[2], Ashvin Agrawal[3]
> and Supun Kamburugamuve[4], committers on the project, are PMCs on other
> Apache projects and Bill and Ashvin have gone through the Apache incubator
> process. Twitter has already donated numerous projects to the ASF (e.g.,
> Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be mentored
> by experienced ASF members that can help with any roadblocks.
>
> == Homogenous Developers ==
>
> Initial committers come from 5 separate organizations. Our intention is
> increase the diversity of contributing developers and their affiliations.
> To date github contributions have come from approximately 50 contributors
> from outside the Twitter team.
>
> == Reliance on Salaried Developers ==
>
> It is expected that Heron development will occur on both salaried time and
> on volunteer time. The majority of initial committers are paid by their
> employers to contribute to this project. We are committed to recruiting
> additional committers from other organizations as well as non-salaried
> committers to join project.
>
> == Relationships with Other Apache Products ==
>
> As mentioned in the Alignment section, Heron implements the Apache Storm
> API and integrates with multiple Apache schedulers (Apache Mesos, Apache
> Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache
> Thrift.
>
> == An Excessive Fascination with the Apache Brand ==
>
> Heron's popularity is growing in the streaming compute space and we are
> long time supporters of the Apache brand. This proposal is not for the
> purpose of generating publicity through. Rather, the primary benefits to
> joining Apache are those of community building and open decision making
> outlined in the Rationale section.
>
> == Documentation ==
>
> This proposal exists online as
> http://wiki.apache.org/incubator/HeronProposal. Extensive documentation
> can
> be found on github at https://twitter.github.io/heron and the source code
> is well documented.
>
> == Source and Intellectual Property Submission Plan ==
>
> The Heron codebase is currently hosted on Github:
> https://github.com/twitter/heron. During incubation, the codebase will be
> migrated to Apache infrastructure. The source code is already ASF 2.0
> licensed.
>
> == External Dependencies ==
>
> All external libraries have ASF 2.0 compatible licenses except for pylint.
> The pylint library is GPL licensed, but is only used for pre-build Python
> style checks and is neither bundled with, nor relied upon by, the Heron
> source or binary release artifacts.
>
> == Cryptography ==
>
> Heron does not use any cryptography libraries.
>
> = Required Resources =
>
> == Mailing lists ==
>
>  * private@heron.incubator.apache.org (with moderated subscriptions)
>  * dev@heron.incubator.apache.org
>  * commits@heron.incubator.apache.org
>  * user@heron.incubator.apache.org
>
> == Subversion Directory ==
>
> Git is the preferred source control system: git://git.apache.org/heron
>
> == Issue Tracking ==
>
> JIRA: Heron (HERON)
>
> == Initial Committers ==
>
>  * Andrew Jorgensen (andrew at andrewjorgensen dot com)
>  * Ashvin Agrawal (ashvin at apache dot org)*
>  * Avrilia Floratou (avrilia dot floratou at gmail dot com)
>  * Bill Graham (billgraham at apache dot org)*
>  * Brian Hatfield (bmhatfield at gmail dot com)
>  * Chris Kellogg (cckellogg at gmail dot com)
>  * Huijun Wu (huijun dot wu dot 2010 at gmail dot com)
>  * Karthik Ramasamy (karthik at gmail dot com)
>  * Maosong Fu (maosongfu at gmail dot com)
>  * Neng Lu(freeneng at gmail dot com)
>  * Runhang Li (obj dot runhang at gmail dot com)
>  * Sanjeev Kulkarni (sanjeevrk at gmail dot com)
>  * Supun Kamburugamuve (supun at apache dot org)*
>  * Thomas Sun (tom dot ssf at gmail dot com)
>  * Yaliang Wang (yaliang dot w dot wang at ieee dot org)
>
> == Affiliations ==
>
>  * Andrew Jorgensen (Google)
>  * Ashvin Agrawal (Microsoft)
>  * Avrilia Floratou (Microsoft)
>  * Bill Graham (Twitter)
>  * Brian Hatfield (Google)
>  * Chris Kellogg (Twitter)
>  * Huijun Wu (Twitter)
>  * Karthik Ramasamy (Streamlio)
>  * Maosong Fu (Twitter)
>  * Neng Lu (Twitter)
>  * Runhang Li (Twitter)
>  * Sanjeev Kulkarni (Streamlio)
>  * Supun Kamburugamuve (Indiana University)
>  * Thomas Sun (Twitter)
>  * Yaliang Wang (Twitter)
>
> = Sponsors =
>
> == Champion ==
>
>  * Julien Le Dem (julien at apache dot org)
>
> == Nominated Mentors ==
>
>  * Jake Farrell (jfarrell at apache dot org)
>  * Jacques Nadeau (jacques at apache dot org)
>  * Julien Le Dem (julien at apache dot org)
>  * P. Taylor Goetz (ptgoetz at apache dot org)
>
> == Sponsoring Entity ==
>
> The Apache Incubator
>
> == Footnotes ==
>
>  * 1 - Papers detailing Heron are available at
> http://dl.acm.org/citation.cfm?id=2742788 and
> http://sites.computer.org/debull/A15dec/p15.pdf.
>  * 2 - http://home.apache.org/phonebook.html?uid=billgraham
>  * 3 - http://home.apache.org/phonebook.html?uid=ashvin
>  * 4 - http://home.apache.org/phonebook.html?uid=supun
>

Re: [VOTE] Heron to enter Apache Incubator

Posted by Karthik Ramasamy <kr...@gmail.com>.
+1 (non-binding)

On 2017-06-16 13:41 (-0700), Bill Graham <bi...@gmail.com> wrote: 
> Hi,
> 
> Based on the discussion on the incubator mailing list[1] I would like to
> call a vote to add Heron to the Apache Incubator.
> 
> The full proposal is available below, and is also available on the Apache
> Incubator wiki at:
>     https://wiki.apache.org/incubator/HeronProposal
> 
> Please vote:
>   [ ] +1, bring Heron into Incubator
>   [ ] -1, do not bring Heron into Incubator, because...
> 
> The vote will open for 7 days until Friday June 23 at 14:00 PT.
> 
> Thank you
> 
> 1 -
> https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E
> 
> 
> 
> = Heron Proposal =
> 
> = Abstract =
> Heron is a real-time, distributed, fault-tolerant stream processing engine
> initially developed by Twitter.
> 
> = Proposal =
> 
> Heron is a real-time stream processing engine built for high performance,
> ease of manageability, performance predictability and developer
> productivity[1]. We wish to develop a community around Heron to increase
> contributions and see Heron thrive in an open forum.
> 
> = Background =
> 
> Heron provides the ability for developers to compose directed acyclic
> graphs (DAGs) of real-time query execution logic (i.e. a topology) and
> submit the topology to execute on a pluggable job scheduling system (e.g.,
> Apache Aurora, YARN, Marathon, etc). Users can employ either the native
> Heron API or the Apache Storm API to develop the topology. Heron supports
> the Storm API for ease of migration, but beyond that Heron’s architecture
> differs considerably from Storm’s.
> 
> Users submit a topology to the scheduler using the Heron client, which uses
> the Heron binary libraries to deploy all daemons required to run and manage
> the topology. The topology therefore has no reliance on centrally managed
> Heron services, only on a generic job scheduling system, which lends itself
> well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN (among
> others).
> 
> The scheduler runs each topology as a job consisting of multiple
> containers. One of the containers runs the topology master, responsible for
> managing the topology. The remaining containers each runs a stream manager
> responsible for data routing, a metrics manager that collects and reports
> various metrics and a number of processes called Heron instances which run
> the user-defined logic on the stream of tuples. Parallelism is achieved via
> process-based isolation of Heron instances, which provides predictable
> performance while simplifying debugging. The containers are allocated and
> managed by the scheduler framework based on resource availability of nodes
> in the cluster. The metadata for the topology, such as the physical plan
> and execution details, are stored in the pluggable Heron State Manager
> (e.g. Apache ZooKeeper).
> 
> = Rationale =
> 
> Heron is a general-purpose, modular and extensible platform that can be
> leveraged to support common, real-time analytics use cases. There is an
> increasing demand for open-source, scalable real-time analytics systems. We
> believe that Heron can be leveraged by other organizations to build
> streaming applications that can benefit from its robustness, high
> performance, adaptability to cloud environments and ease of use. Moreover,
> we hope that open-sourcing Heron will help to further evolve the technology
> as the project attracts contributors with diverse backgrounds and areas of
> expertise.
> 
> We believe the Apache foundation is a great fit as the long-term home for
> Heron, as it provides an established process for community-driven
> development and decision making by consensus. This is exactly the model we
> want for future Heron development.
> 
> = Initial Goals =
> 
>  * Move the existing codebase, website, documentation, and mailing lists to
> Apache-hosted infrastructure.
>  * Integrate with the Apache development process.
>  * Ensure all dependencies are compliant with Apache License version 2.0.
>  * Incrementally develop and release per Apache guidelines.
> 
> = Current Status =
> 
> Heron is a stable project used in production at Twitter since 2014 and open
> sourced under the ASL v2 license in 2016. The Heron source code is
> currently hosted at github.com (https://github.com/twitter/heron), which
> will seed the Apache git repository.
> 
> = Meritocracy =
> 
> By submitting this incubator proposal, we’re expressing our intent to build
> a diverse developer community around Heron that will conduct itself
> according to The Apache Way and use a meritocratic means of building it's
> committer base. Several companies and universities have already expressed
> interest in and contributed to Heron. Our goal is to grow the Heron
> community by encouraging open communication, contribution and participation
> of all types, and ensuring that contributors are recognized appropriately.
> 
> = Community =
> 
> Heron is currently being used by Twitter, Google, Machine Zone and
> ndustrial.io and has received significant contributions by Microsoft and
> Streamlio. By bringing Heron into the Apache ecosystem, we believe we can
> attract even more developers who are interested in creating real-time
> systems to build the project's contributor base.
> 
> == Core Developers ==
> 
> Current core developers are engineers from Twitter, Google, Microsoft and
> Streamlio.
> 
> == Alignment ==
> 
> Heron utilizes a number of Apache technologies. Heron leverages Apache
> ZooKeeper for coordination and has scheduler implementations to integrate
> with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache REEF)
> as well as spout implementations to integrate with Apache Kafka and metrics
> implementations to integrate with Scribe. Heron also implements the Apache
> Storm user-level API, which allows topologies written against Storm to run
> in Heron. We believe that having Heron at Apache will help further the
> growth of the streaming compute community, as well as encourage cooperation
> and developer cross pollination with other Apache projects.
> 
> = Known Risks =
> 
> == Orphaned Products ==
> 
> The risk of the Heron project being abandoned is minimal. It is used in
> production at Twitter and Google and other companies are evaluating or
> adopting it for production use.
> 
> == Inexperience with Open Source ==
> 
> All of the core contributors to the project have considerable experience
> with open source software development. Bill Graham[2], Ashvin Agrawal[3]
> and Supun Kamburugamuve[4], committers on the project, are PMCs on other
> Apache projects and Bill and Ashvin have gone through the Apache incubator
> process. Twitter has already donated numerous projects to the ASF (e.g.,
> Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be mentored
> by experienced ASF members that can help with any roadblocks.
> 
> == Homogenous Developers ==
> 
> Initial committers come from 5 separate organizations. Our intention is
> increase the diversity of contributing developers and their affiliations.
> To date github contributions have come from approximately 50 contributors
> from outside the Twitter team.
> 
> == Reliance on Salaried Developers ==
> 
> It is expected that Heron development will occur on both salaried time and
> on volunteer time. The majority of initial committers are paid by their
> employers to contribute to this project. We are committed to recruiting
> additional committers from other organizations as well as non-salaried
> committers to join project.
> 
> == Relationships with Other Apache Products ==
> 
> As mentioned in the Alignment section, Heron implements the Apache Storm
> API and integrates with multiple Apache schedulers (Apache Mesos, Apache
> Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache
> Thrift.
> 
> == An Excessive Fascination with the Apache Brand ==
> 
> Heron's popularity is growing in the streaming compute space and we are
> long time supporters of the Apache brand. This proposal is not for the
> purpose of generating publicity through. Rather, the primary benefits to
> joining Apache are those of community building and open decision making
> outlined in the Rationale section.
> 
> == Documentation ==
> 
> This proposal exists online as
> http://wiki.apache.org/incubator/HeronProposal. Extensive documentation can
> be found on github at https://twitter.github.io/heron and the source code
> is well documented.
> 
> == Source and Intellectual Property Submission Plan ==
> 
> The Heron codebase is currently hosted on Github:
> https://github.com/twitter/heron. During incubation, the codebase will be
> migrated to Apache infrastructure. The source code is already ASF 2.0
> licensed.
> 
> == External Dependencies ==
> 
> All external libraries have ASF 2.0 compatible licenses except for pylint.
> The pylint library is GPL licensed, but is only used for pre-build Python
> style checks and is neither bundled with, nor relied upon by, the Heron
> source or binary release artifacts.
> 
> == Cryptography ==
> 
> Heron does not use any cryptography libraries.
> 
> = Required Resources =
> 
> == Mailing lists ==
> 
>  * private@heron.incubator.apache.org (with moderated subscriptions)
>  * dev@heron.incubator.apache.org
>  * commits@heron.incubator.apache.org
>  * user@heron.incubator.apache.org
> 
> == Subversion Directory ==
> 
> Git is the preferred source control system: git://git.apache.org/heron
> 
> == Issue Tracking ==
> 
> JIRA: Heron (HERON)
> 
> == Initial Committers ==
> 
>  * Andrew Jorgensen (andrew at andrewjorgensen dot com)
>  * Ashvin Agrawal (ashvin at apache dot org)*
>  * Avrilia Floratou (avrilia dot floratou at gmail dot com)
>  * Bill Graham (billgraham at apache dot org)*
>  * Brian Hatfield (bmhatfield at gmail dot com)
>  * Chris Kellogg (cckellogg at gmail dot com)
>  * Huijun Wu (huijun dot wu dot 2010 at gmail dot com)
>  * Karthik Ramasamy (karthik at gmail dot com)
>  * Maosong Fu (maosongfu at gmail dot com)
>  * Neng Lu(freeneng at gmail dot com)
>  * Runhang Li (obj dot runhang at gmail dot com)
>  * Sanjeev Kulkarni (sanjeevrk at gmail dot com)
>  * Supun Kamburugamuve (supun at apache dot org)*
>  * Thomas Sun (tom dot ssf at gmail dot com)
>  * Yaliang Wang (yaliang dot w dot wang at ieee dot org)
> 
> == Affiliations ==
> 
>  * Andrew Jorgensen (Google)
>  * Ashvin Agrawal (Microsoft)
>  * Avrilia Floratou (Microsoft)
>  * Bill Graham (Twitter)
>  * Brian Hatfield (Google)
>  * Chris Kellogg (Twitter)
>  * Huijun Wu (Twitter)
>  * Karthik Ramasamy (Streamlio)
>  * Maosong Fu (Twitter)
>  * Neng Lu (Twitter)
>  * Runhang Li (Twitter)
>  * Sanjeev Kulkarni (Streamlio)
>  * Supun Kamburugamuve (Indiana University)
>  * Thomas Sun (Twitter)
>  * Yaliang Wang (Twitter)
> 
> = Sponsors =
> 
> == Champion ==
> 
>  * Julien Le Dem (julien at apache dot org)
> 
> == Nominated Mentors ==
> 
>  * Jake Farrell (jfarrell at apache dot org)
>  * Jacques Nadeau (jacques at apache dot org)
>  * Julien Le Dem (julien at apache dot org)
>  * P. Taylor Goetz (ptgoetz at apache dot org)
> 
> == Sponsoring Entity ==
> 
> The Apache Incubator
> 
> == Footnotes ==
> 
>  * 1 - Papers detailing Heron are available at
> http://dl.acm.org/citation.cfm?id=2742788 and
> http://sites.computer.org/debull/A15dec/p15.pdf.
>  * 2 - http://home.apache.org/phonebook.html?uid=billgraham
>  * 3 - http://home.apache.org/phonebook.html?uid=ashvin
>  * 4 - http://home.apache.org/phonebook.html?uid=supun
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [VOTE] Heron to enter Apache Incubator

Posted by Julian Hyde <jh...@apache.org>.
+1 (binding)

> On Jun 16, 2017, at 8:43 PM, Nabarun Nag <nn...@apache.org> wrote:
> 
> +1
> 
> On Fri, Jun 16, 2017 at 8:28 PM Jacques Nadeau <ja...@apache.org> wrote:
> 
>> +1
>> 
>> On Jun 16, 2017 7:26 PM, "Ashish" <pa...@gmail.com> wrote:
>> 
>> +1 (non-binding)
>> 
>> On Fri, Jun 16, 2017 at 1:41 PM, Bill Graham <bi...@gmail.com> wrote:
>>> Hi,
>>> 
>>> Based on the discussion on the incubator mailing list[1] I would like to
>>> call a vote to add Heron to the Apache Incubator.
>>> 
>>> The full proposal is available below, and is also available on the Apache
>>> Incubator wiki at:
>>>    https://wiki.apache.org/incubator/HeronProposal
>>> 
>>> Please vote:
>>>  [ ] +1, bring Heron into Incubator
>>>  [ ] -1, do not bring Heron into Incubator, because...
>>> 
>>> The vote will open for 7 days until Friday June 23 at 14:00 PT.
>>> 
>>> Thank you
>>> 
>>> 1 -
>>> https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7
>> 786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E
>>> 
>>> 
>>> 
>>> = Heron Proposal =
>>> 
>>> = Abstract =
>>> Heron is a real-time, distributed, fault-tolerant stream processing
>> engine
>>> initially developed by Twitter.
>>> 
>>> = Proposal =
>>> 
>>> Heron is a real-time stream processing engine built for high performance,
>>> ease of manageability, performance predictability and developer
>>> productivity[1]. We wish to develop a community around Heron to increase
>>> contributions and see Heron thrive in an open forum.
>>> 
>>> = Background =
>>> 
>>> Heron provides the ability for developers to compose directed acyclic
>>> graphs (DAGs) of real-time query execution logic (i.e. a topology) and
>>> submit the topology to execute on a pluggable job scheduling system
>> (e.g.,
>>> Apache Aurora, YARN, Marathon, etc). Users can employ either the native
>>> Heron API or the Apache Storm API to develop the topology. Heron supports
>>> the Storm API for ease of migration, but beyond that Heron’s architecture
>>> differs considerably from Storm’s.
>>> 
>>> Users submit a topology to the scheduler using the Heron client, which
>> uses
>>> the Heron binary libraries to deploy all daemons required to run and
>> manage
>>> the topology. The topology therefore has no reliance on centrally managed
>>> Heron services, only on a generic job scheduling system, which lends
>> itself
>>> well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN (among
>>> others).
>>> 
>>> The scheduler runs each topology as a job consisting of multiple
>>> containers. One of the containers runs the topology master, responsible
>> for
>>> managing the topology. The remaining containers each runs a stream
>> manager
>>> responsible for data routing, a metrics manager that collects and reports
>>> various metrics and a number of processes called Heron instances which
>> run
>>> the user-defined logic on the stream of tuples. Parallelism is achieved
>> via
>>> process-based isolation of Heron instances, which provides predictable
>>> performance while simplifying debugging. The containers are allocated and
>>> managed by the scheduler framework based on resource availability of
>> nodes
>>> in the cluster. The metadata for the topology, such as the physical plan
>>> and execution details, are stored in the pluggable Heron State Manager
>>> (e.g. Apache ZooKeeper).
>>> 
>>> = Rationale =
>>> 
>>> Heron is a general-purpose, modular and extensible platform that can be
>>> leveraged to support common, real-time analytics use cases. There is an
>>> increasing demand for open-source, scalable real-time analytics systems.
>> We
>>> believe that Heron can be leveraged by other organizations to build
>>> streaming applications that can benefit from its robustness, high
>>> performance, adaptability to cloud environments and ease of use.
>> Moreover,
>>> we hope that open-sourcing Heron will help to further evolve the
>> technology
>>> as the project attracts contributors with diverse backgrounds and areas
>> of
>>> expertise.
>>> 
>>> We believe the Apache foundation is a great fit as the long-term home for
>>> Heron, as it provides an established process for community-driven
>>> development and decision making by consensus. This is exactly the model
>> we
>>> want for future Heron development.
>>> 
>>> = Initial Goals =
>>> 
>>> * Move the existing codebase, website, documentation, and mailing lists
>> to
>>> Apache-hosted infrastructure.
>>> * Integrate with the Apache development process.
>>> * Ensure all dependencies are compliant with Apache License version 2.0.
>>> * Incrementally develop and release per Apache guidelines.
>>> 
>>> = Current Status =
>>> 
>>> Heron is a stable project used in production at Twitter since 2014 and
>> open
>>> sourced under the ASL v2 license in 2016. The Heron source code is
>>> currently hosted at github.com (https://github.com/twitter/heron), which
>>> will seed the Apache git repository.
>>> 
>>> = Meritocracy =
>>> 
>>> By submitting this incubator proposal, we’re expressing our intent to
>> build
>>> a diverse developer community around Heron that will conduct itself
>>> according to The Apache Way and use a meritocratic means of building it's
>>> committer base. Several companies and universities have already expressed
>>> interest in and contributed to Heron. Our goal is to grow the Heron
>>> community by encouraging open communication, contribution and
>> participation
>>> of all types, and ensuring that contributors are recognized
>> appropriately.
>>> 
>>> = Community =
>>> 
>>> Heron is currently being used by Twitter, Google, Machine Zone and
>>> ndustrial.io and has received significant contributions by Microsoft and
>>> Streamlio. By bringing Heron into the Apache ecosystem, we believe we can
>>> attract even more developers who are interested in creating real-time
>>> systems to build the project's contributor base.
>>> 
>>> == Core Developers ==
>>> 
>>> Current core developers are engineers from Twitter, Google, Microsoft and
>>> Streamlio.
>>> 
>>> == Alignment ==
>>> 
>>> Heron utilizes a number of Apache technologies. Heron leverages Apache
>>> ZooKeeper for coordination and has scheduler implementations to integrate
>>> with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache
>> REEF)
>>> as well as spout implementations to integrate with Apache Kafka and
>> metrics
>>> implementations to integrate with Scribe. Heron also implements the
>> Apache
>>> Storm user-level API, which allows topologies written against Storm to
>> run
>>> in Heron. We believe that having Heron at Apache will help further the
>>> growth of the streaming compute community, as well as encourage
>> cooperation
>>> and developer cross pollination with other Apache projects.
>>> 
>>> = Known Risks =
>>> 
>>> == Orphaned Products ==
>>> 
>>> The risk of the Heron project being abandoned is minimal. It is used in
>>> production at Twitter and Google and other companies are evaluating or
>>> adopting it for production use.
>>> 
>>> == Inexperience with Open Source ==
>>> 
>>> All of the core contributors to the project have considerable experience
>>> with open source software development. Bill Graham[2], Ashvin Agrawal[3]
>>> and Supun Kamburugamuve[4], committers on the project, are PMCs on other
>>> Apache projects and Bill and Ashvin have gone through the Apache
>> incubator
>>> process. Twitter has already donated numerous projects to the ASF (e.g.,
>>> Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be mentored
>>> by experienced ASF members that can help with any roadblocks.
>>> 
>>> == Homogenous Developers ==
>>> 
>>> Initial committers come from 5 separate organizations. Our intention is
>>> increase the diversity of contributing developers and their affiliations.
>>> To date github contributions have come from approximately 50 contributors
>>> from outside the Twitter team.
>>> 
>>> == Reliance on Salaried Developers ==
>>> 
>>> It is expected that Heron development will occur on both salaried time
>> and
>>> on volunteer time. The majority of initial committers are paid by their
>>> employers to contribute to this project. We are committed to recruiting
>>> additional committers from other organizations as well as non-salaried
>>> committers to join project.
>>> 
>>> == Relationships with Other Apache Products ==
>>> 
>>> As mentioned in the Alignment section, Heron implements the Apache Storm
>>> API and integrates with multiple Apache schedulers (Apache Mesos, Apache
>>> Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache
>>> Thrift.
>>> 
>>> == An Excessive Fascination with the Apache Brand ==
>>> 
>>> Heron's popularity is growing in the streaming compute space and we are
>>> long time supporters of the Apache brand. This proposal is not for the
>>> purpose of generating publicity through. Rather, the primary benefits to
>>> joining Apache are those of community building and open decision making
>>> outlined in the Rationale section.
>>> 
>>> == Documentation ==
>>> 
>>> This proposal exists online as
>>> http://wiki.apache.org/incubator/HeronProposal. Extensive documentation
>> can
>>> be found on github at https://twitter.github.io/heron and the source
>> code
>>> is well documented.
>>> 
>>> == Source and Intellectual Property Submission Plan ==
>>> 
>>> The Heron codebase is currently hosted on Github:
>>> https://github.com/twitter/heron. During incubation, the codebase will
>> be
>>> migrated to Apache infrastructure. The source code is already ASF 2.0
>>> licensed.
>>> 
>>> == External Dependencies ==
>>> 
>>> All external libraries have ASF 2.0 compatible licenses except for
>> pylint.
>>> The pylint library is GPL licensed, but is only used for pre-build Python
>>> style checks and is neither bundled with, nor relied upon by, the Heron
>>> source or binary release artifacts.
>>> 
>>> == Cryptography ==
>>> 
>>> Heron does not use any cryptography libraries.
>>> 
>>> = Required Resources =
>>> 
>>> == Mailing lists ==
>>> 
>>> * private@heron.incubator.apache.org (with moderated subscriptions)
>>> * dev@heron.incubator.apache.org
>>> * commits@heron.incubator.apache.org
>>> * user@heron.incubator.apache.org
>>> 
>>> == Subversion Directory ==
>>> 
>>> Git is the preferred source control system: git://git.apache.org/heron
>>> 
>>> == Issue Tracking ==
>>> 
>>> JIRA: Heron (HERON)
>>> 
>>> == Initial Committers ==
>>> 
>>> * Andrew Jorgensen (andrew at andrewjorgensen dot com)
>>> * Ashvin Agrawal (ashvin at apache dot org)*
>>> * Avrilia Floratou (avrilia dot floratou at gmail dot com)
>>> * Bill Graham (billgraham at apache dot org)*
>>> * Brian Hatfield (bmhatfield at gmail dot com)
>>> * Chris Kellogg (cckellogg at gmail dot com)
>>> * Huijun Wu (huijun dot wu dot 2010 at gmail dot com)
>>> * Karthik Ramasamy (karthik at gmail dot com)
>>> * Maosong Fu (maosongfu at gmail dot com)
>>> * Neng Lu(freeneng at gmail dot com)
>>> * Runhang Li (obj dot runhang at gmail dot com)
>>> * Sanjeev Kulkarni (sanjeevrk at gmail dot com)
>>> * Supun Kamburugamuve (supun at apache dot org)*
>>> * Thomas Sun (tom dot ssf at gmail dot com)
>>> * Yaliang Wang (yaliang dot w dot wang at ieee dot org)
>>> 
>>> == Affiliations ==
>>> 
>>> * Andrew Jorgensen (Google)
>>> * Ashvin Agrawal (Microsoft)
>>> * Avrilia Floratou (Microsoft)
>>> * Bill Graham (Twitter)
>>> * Brian Hatfield (Google)
>>> * Chris Kellogg (Twitter)
>>> * Huijun Wu (Twitter)
>>> * Karthik Ramasamy (Streamlio)
>>> * Maosong Fu (Twitter)
>>> * Neng Lu (Twitter)
>>> * Runhang Li (Twitter)
>>> * Sanjeev Kulkarni (Streamlio)
>>> * Supun Kamburugamuve (Indiana University)
>>> * Thomas Sun (Twitter)
>>> * Yaliang Wang (Twitter)
>>> 
>>> = Sponsors =
>>> 
>>> == Champion ==
>>> 
>>> * Julien Le Dem (julien at apache dot org)
>>> 
>>> == Nominated Mentors ==
>>> 
>>> * Jake Farrell (jfarrell at apache dot org)
>>> * Jacques Nadeau (jacques at apache dot org)
>>> * Julien Le Dem (julien at apache dot org)
>>> * P. Taylor Goetz (ptgoetz at apache dot org)
>>> 
>>> == Sponsoring Entity ==
>>> 
>>> The Apache Incubator
>>> 
>>> == Footnotes ==
>>> 
>>> * 1 - Papers detailing Heron are available at
>>> http://dl.acm.org/citation.cfm?id=2742788 and
>>> http://sites.computer.org/debull/A15dec/p15.pdf.
>>> * 2 - http://home.apache.org/phonebook.html?uid=billgraham
>>> * 3 - http://home.apache.org/phonebook.html?uid=ashvin
>>> * 4 - http://home.apache.org/phonebook.html?uid=supun
>> 
>> 
>> 
>> --
>> thanks
>> ashish
>> 
>> Blog: http://www.ashishpaliwal.com/blog
>> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: general-help@incubator.apache.org
>> 


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [VOTE] Heron to enter Apache Incubator

Posted by Nabarun Nag <nn...@apache.org>.
+1

On Fri, Jun 16, 2017 at 8:28 PM Jacques Nadeau <ja...@apache.org> wrote:

> +1
>
> On Jun 16, 2017 7:26 PM, "Ashish" <pa...@gmail.com> wrote:
>
> +1 (non-binding)
>
> On Fri, Jun 16, 2017 at 1:41 PM, Bill Graham <bi...@gmail.com> wrote:
> > Hi,
> >
> > Based on the discussion on the incubator mailing list[1] I would like to
> > call a vote to add Heron to the Apache Incubator.
> >
> > The full proposal is available below, and is also available on the Apache
> > Incubator wiki at:
> >     https://wiki.apache.org/incubator/HeronProposal
> >
> > Please vote:
> >   [ ] +1, bring Heron into Incubator
> >   [ ] -1, do not bring Heron into Incubator, because...
> >
> > The vote will open for 7 days until Friday June 23 at 14:00 PT.
> >
> > Thank you
> >
> > 1 -
> > https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7
> 786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E
> >
> >
> >
> > = Heron Proposal =
> >
> > = Abstract =
> > Heron is a real-time, distributed, fault-tolerant stream processing
> engine
> > initially developed by Twitter.
> >
> > = Proposal =
> >
> > Heron is a real-time stream processing engine built for high performance,
> > ease of manageability, performance predictability and developer
> > productivity[1]. We wish to develop a community around Heron to increase
> > contributions and see Heron thrive in an open forum.
> >
> > = Background =
> >
> > Heron provides the ability for developers to compose directed acyclic
> > graphs (DAGs) of real-time query execution logic (i.e. a topology) and
> > submit the topology to execute on a pluggable job scheduling system
> (e.g.,
> > Apache Aurora, YARN, Marathon, etc). Users can employ either the native
> > Heron API or the Apache Storm API to develop the topology. Heron supports
> > the Storm API for ease of migration, but beyond that Heron’s architecture
> > differs considerably from Storm’s.
> >
> > Users submit a topology to the scheduler using the Heron client, which
> uses
> > the Heron binary libraries to deploy all daemons required to run and
> manage
> > the topology. The topology therefore has no reliance on centrally managed
> > Heron services, only on a generic job scheduling system, which lends
> itself
> > well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN (among
> > others).
> >
> > The scheduler runs each topology as a job consisting of multiple
> > containers. One of the containers runs the topology master, responsible
> for
> > managing the topology. The remaining containers each runs a stream
> manager
> > responsible for data routing, a metrics manager that collects and reports
> > various metrics and a number of processes called Heron instances which
> run
> > the user-defined logic on the stream of tuples. Parallelism is achieved
> via
> > process-based isolation of Heron instances, which provides predictable
> > performance while simplifying debugging. The containers are allocated and
> > managed by the scheduler framework based on resource availability of
> nodes
> > in the cluster. The metadata for the topology, such as the physical plan
> > and execution details, are stored in the pluggable Heron State Manager
> > (e.g. Apache ZooKeeper).
> >
> > = Rationale =
> >
> > Heron is a general-purpose, modular and extensible platform that can be
> > leveraged to support common, real-time analytics use cases. There is an
> > increasing demand for open-source, scalable real-time analytics systems.
> We
> > believe that Heron can be leveraged by other organizations to build
> > streaming applications that can benefit from its robustness, high
> > performance, adaptability to cloud environments and ease of use.
> Moreover,
> > we hope that open-sourcing Heron will help to further evolve the
> technology
> > as the project attracts contributors with diverse backgrounds and areas
> of
> > expertise.
> >
> > We believe the Apache foundation is a great fit as the long-term home for
> > Heron, as it provides an established process for community-driven
> > development and decision making by consensus. This is exactly the model
> we
> > want for future Heron development.
> >
> > = Initial Goals =
> >
> >  * Move the existing codebase, website, documentation, and mailing lists
> to
> > Apache-hosted infrastructure.
> >  * Integrate with the Apache development process.
> >  * Ensure all dependencies are compliant with Apache License version 2.0.
> >  * Incrementally develop and release per Apache guidelines.
> >
> > = Current Status =
> >
> > Heron is a stable project used in production at Twitter since 2014 and
> open
> > sourced under the ASL v2 license in 2016. The Heron source code is
> > currently hosted at github.com (https://github.com/twitter/heron), which
> > will seed the Apache git repository.
> >
> > = Meritocracy =
> >
> > By submitting this incubator proposal, we’re expressing our intent to
> build
> > a diverse developer community around Heron that will conduct itself
> > according to The Apache Way and use a meritocratic means of building it's
> > committer base. Several companies and universities have already expressed
> > interest in and contributed to Heron. Our goal is to grow the Heron
> > community by encouraging open communication, contribution and
> participation
> > of all types, and ensuring that contributors are recognized
> appropriately.
> >
> > = Community =
> >
> > Heron is currently being used by Twitter, Google, Machine Zone and
> > ndustrial.io and has received significant contributions by Microsoft and
> > Streamlio. By bringing Heron into the Apache ecosystem, we believe we can
> > attract even more developers who are interested in creating real-time
> > systems to build the project's contributor base.
> >
> > == Core Developers ==
> >
> > Current core developers are engineers from Twitter, Google, Microsoft and
> > Streamlio.
> >
> > == Alignment ==
> >
> > Heron utilizes a number of Apache technologies. Heron leverages Apache
> > ZooKeeper for coordination and has scheduler implementations to integrate
> > with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache
> REEF)
> > as well as spout implementations to integrate with Apache Kafka and
> metrics
> > implementations to integrate with Scribe. Heron also implements the
> Apache
> > Storm user-level API, which allows topologies written against Storm to
> run
> > in Heron. We believe that having Heron at Apache will help further the
> > growth of the streaming compute community, as well as encourage
> cooperation
> > and developer cross pollination with other Apache projects.
> >
> > = Known Risks =
> >
> > == Orphaned Products ==
> >
> > The risk of the Heron project being abandoned is minimal. It is used in
> > production at Twitter and Google and other companies are evaluating or
> > adopting it for production use.
> >
> > == Inexperience with Open Source ==
> >
> > All of the core contributors to the project have considerable experience
> > with open source software development. Bill Graham[2], Ashvin Agrawal[3]
> > and Supun Kamburugamuve[4], committers on the project, are PMCs on other
> > Apache projects and Bill and Ashvin have gone through the Apache
> incubator
> > process. Twitter has already donated numerous projects to the ASF (e.g.,
> > Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be mentored
> > by experienced ASF members that can help with any roadblocks.
> >
> > == Homogenous Developers ==
> >
> > Initial committers come from 5 separate organizations. Our intention is
> > increase the diversity of contributing developers and their affiliations.
> > To date github contributions have come from approximately 50 contributors
> > from outside the Twitter team.
> >
> > == Reliance on Salaried Developers ==
> >
> > It is expected that Heron development will occur on both salaried time
> and
> > on volunteer time. The majority of initial committers are paid by their
> > employers to contribute to this project. We are committed to recruiting
> > additional committers from other organizations as well as non-salaried
> > committers to join project.
> >
> > == Relationships with Other Apache Products ==
> >
> > As mentioned in the Alignment section, Heron implements the Apache Storm
> > API and integrates with multiple Apache schedulers (Apache Mesos, Apache
> > Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache
> > Thrift.
> >
> > == An Excessive Fascination with the Apache Brand ==
> >
> > Heron's popularity is growing in the streaming compute space and we are
> > long time supporters of the Apache brand. This proposal is not for the
> > purpose of generating publicity through. Rather, the primary benefits to
> > joining Apache are those of community building and open decision making
> > outlined in the Rationale section.
> >
> > == Documentation ==
> >
> > This proposal exists online as
> > http://wiki.apache.org/incubator/HeronProposal. Extensive documentation
> can
> > be found on github at https://twitter.github.io/heron and the source
> code
> > is well documented.
> >
> > == Source and Intellectual Property Submission Plan ==
> >
> > The Heron codebase is currently hosted on Github:
> > https://github.com/twitter/heron. During incubation, the codebase will
> be
> > migrated to Apache infrastructure. The source code is already ASF 2.0
> > licensed.
> >
> > == External Dependencies ==
> >
> > All external libraries have ASF 2.0 compatible licenses except for
> pylint.
> > The pylint library is GPL licensed, but is only used for pre-build Python
> > style checks and is neither bundled with, nor relied upon by, the Heron
> > source or binary release artifacts.
> >
> > == Cryptography ==
> >
> > Heron does not use any cryptography libraries.
> >
> > = Required Resources =
> >
> > == Mailing lists ==
> >
> >  * private@heron.incubator.apache.org (with moderated subscriptions)
> >  * dev@heron.incubator.apache.org
> >  * commits@heron.incubator.apache.org
> >  * user@heron.incubator.apache.org
> >
> > == Subversion Directory ==
> >
> > Git is the preferred source control system: git://git.apache.org/heron
> >
> > == Issue Tracking ==
> >
> > JIRA: Heron (HERON)
> >
> > == Initial Committers ==
> >
> >  * Andrew Jorgensen (andrew at andrewjorgensen dot com)
> >  * Ashvin Agrawal (ashvin at apache dot org)*
> >  * Avrilia Floratou (avrilia dot floratou at gmail dot com)
> >  * Bill Graham (billgraham at apache dot org)*
> >  * Brian Hatfield (bmhatfield at gmail dot com)
> >  * Chris Kellogg (cckellogg at gmail dot com)
> >  * Huijun Wu (huijun dot wu dot 2010 at gmail dot com)
> >  * Karthik Ramasamy (karthik at gmail dot com)
> >  * Maosong Fu (maosongfu at gmail dot com)
> >  * Neng Lu(freeneng at gmail dot com)
> >  * Runhang Li (obj dot runhang at gmail dot com)
> >  * Sanjeev Kulkarni (sanjeevrk at gmail dot com)
> >  * Supun Kamburugamuve (supun at apache dot org)*
> >  * Thomas Sun (tom dot ssf at gmail dot com)
> >  * Yaliang Wang (yaliang dot w dot wang at ieee dot org)
> >
> > == Affiliations ==
> >
> >  * Andrew Jorgensen (Google)
> >  * Ashvin Agrawal (Microsoft)
> >  * Avrilia Floratou (Microsoft)
> >  * Bill Graham (Twitter)
> >  * Brian Hatfield (Google)
> >  * Chris Kellogg (Twitter)
> >  * Huijun Wu (Twitter)
> >  * Karthik Ramasamy (Streamlio)
> >  * Maosong Fu (Twitter)
> >  * Neng Lu (Twitter)
> >  * Runhang Li (Twitter)
> >  * Sanjeev Kulkarni (Streamlio)
> >  * Supun Kamburugamuve (Indiana University)
> >  * Thomas Sun (Twitter)
> >  * Yaliang Wang (Twitter)
> >
> > = Sponsors =
> >
> > == Champion ==
> >
> >  * Julien Le Dem (julien at apache dot org)
> >
> > == Nominated Mentors ==
> >
> >  * Jake Farrell (jfarrell at apache dot org)
> >  * Jacques Nadeau (jacques at apache dot org)
> >  * Julien Le Dem (julien at apache dot org)
> >  * P. Taylor Goetz (ptgoetz at apache dot org)
> >
> > == Sponsoring Entity ==
> >
> > The Apache Incubator
> >
> > == Footnotes ==
> >
> >  * 1 - Papers detailing Heron are available at
> > http://dl.acm.org/citation.cfm?id=2742788 and
> > http://sites.computer.org/debull/A15dec/p15.pdf.
> >  * 2 - http://home.apache.org/phonebook.html?uid=billgraham
> >  * 3 - http://home.apache.org/phonebook.html?uid=ashvin
> >  * 4 - http://home.apache.org/phonebook.html?uid=supun
>
>
>
> --
> thanks
> ashish
>
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>

Re: [VOTE] Heron to enter Apache Incubator

Posted by Jacques Nadeau <ja...@apache.org>.
+1

On Jun 16, 2017 7:26 PM, "Ashish" <pa...@gmail.com> wrote:

+1 (non-binding)

On Fri, Jun 16, 2017 at 1:41 PM, Bill Graham <bi...@gmail.com> wrote:
> Hi,
>
> Based on the discussion on the incubator mailing list[1] I would like to
> call a vote to add Heron to the Apache Incubator.
>
> The full proposal is available below, and is also available on the Apache
> Incubator wiki at:
>     https://wiki.apache.org/incubator/HeronProposal
>
> Please vote:
>   [ ] +1, bring Heron into Incubator
>   [ ] -1, do not bring Heron into Incubator, because...
>
> The vote will open for 7 days until Friday June 23 at 14:00 PT.
>
> Thank you
>
> 1 -
> https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7
786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E
>
>
>
> = Heron Proposal =
>
> = Abstract =
> Heron is a real-time, distributed, fault-tolerant stream processing engine
> initially developed by Twitter.
>
> = Proposal =
>
> Heron is a real-time stream processing engine built for high performance,
> ease of manageability, performance predictability and developer
> productivity[1]. We wish to develop a community around Heron to increase
> contributions and see Heron thrive in an open forum.
>
> = Background =
>
> Heron provides the ability for developers to compose directed acyclic
> graphs (DAGs) of real-time query execution logic (i.e. a topology) and
> submit the topology to execute on a pluggable job scheduling system (e.g.,
> Apache Aurora, YARN, Marathon, etc). Users can employ either the native
> Heron API or the Apache Storm API to develop the topology. Heron supports
> the Storm API for ease of migration, but beyond that Heron’s architecture
> differs considerably from Storm’s.
>
> Users submit a topology to the scheduler using the Heron client, which
uses
> the Heron binary libraries to deploy all daemons required to run and
manage
> the topology. The topology therefore has no reliance on centrally managed
> Heron services, only on a generic job scheduling system, which lends
itself
> well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN (among
> others).
>
> The scheduler runs each topology as a job consisting of multiple
> containers. One of the containers runs the topology master, responsible
for
> managing the topology. The remaining containers each runs a stream manager
> responsible for data routing, a metrics manager that collects and reports
> various metrics and a number of processes called Heron instances which run
> the user-defined logic on the stream of tuples. Parallelism is achieved
via
> process-based isolation of Heron instances, which provides predictable
> performance while simplifying debugging. The containers are allocated and
> managed by the scheduler framework based on resource availability of nodes
> in the cluster. The metadata for the topology, such as the physical plan
> and execution details, are stored in the pluggable Heron State Manager
> (e.g. Apache ZooKeeper).
>
> = Rationale =
>
> Heron is a general-purpose, modular and extensible platform that can be
> leveraged to support common, real-time analytics use cases. There is an
> increasing demand for open-source, scalable real-time analytics systems.
We
> believe that Heron can be leveraged by other organizations to build
> streaming applications that can benefit from its robustness, high
> performance, adaptability to cloud environments and ease of use. Moreover,
> we hope that open-sourcing Heron will help to further evolve the
technology
> as the project attracts contributors with diverse backgrounds and areas of
> expertise.
>
> We believe the Apache foundation is a great fit as the long-term home for
> Heron, as it provides an established process for community-driven
> development and decision making by consensus. This is exactly the model we
> want for future Heron development.
>
> = Initial Goals =
>
>  * Move the existing codebase, website, documentation, and mailing lists
to
> Apache-hosted infrastructure.
>  * Integrate with the Apache development process.
>  * Ensure all dependencies are compliant with Apache License version 2.0.
>  * Incrementally develop and release per Apache guidelines.
>
> = Current Status =
>
> Heron is a stable project used in production at Twitter since 2014 and
open
> sourced under the ASL v2 license in 2016. The Heron source code is
> currently hosted at github.com (https://github.com/twitter/heron), which
> will seed the Apache git repository.
>
> = Meritocracy =
>
> By submitting this incubator proposal, we’re expressing our intent to
build
> a diverse developer community around Heron that will conduct itself
> according to The Apache Way and use a meritocratic means of building it's
> committer base. Several companies and universities have already expressed
> interest in and contributed to Heron. Our goal is to grow the Heron
> community by encouraging open communication, contribution and
participation
> of all types, and ensuring that contributors are recognized appropriately.
>
> = Community =
>
> Heron is currently being used by Twitter, Google, Machine Zone and
> ndustrial.io and has received significant contributions by Microsoft and
> Streamlio. By bringing Heron into the Apache ecosystem, we believe we can
> attract even more developers who are interested in creating real-time
> systems to build the project's contributor base.
>
> == Core Developers ==
>
> Current core developers are engineers from Twitter, Google, Microsoft and
> Streamlio.
>
> == Alignment ==
>
> Heron utilizes a number of Apache technologies. Heron leverages Apache
> ZooKeeper for coordination and has scheduler implementations to integrate
> with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache
REEF)
> as well as spout implementations to integrate with Apache Kafka and
metrics
> implementations to integrate with Scribe. Heron also implements the Apache
> Storm user-level API, which allows topologies written against Storm to run
> in Heron. We believe that having Heron at Apache will help further the
> growth of the streaming compute community, as well as encourage
cooperation
> and developer cross pollination with other Apache projects.
>
> = Known Risks =
>
> == Orphaned Products ==
>
> The risk of the Heron project being abandoned is minimal. It is used in
> production at Twitter and Google and other companies are evaluating or
> adopting it for production use.
>
> == Inexperience with Open Source ==
>
> All of the core contributors to the project have considerable experience
> with open source software development. Bill Graham[2], Ashvin Agrawal[3]
> and Supun Kamburugamuve[4], committers on the project, are PMCs on other
> Apache projects and Bill and Ashvin have gone through the Apache incubator
> process. Twitter has already donated numerous projects to the ASF (e.g.,
> Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be mentored
> by experienced ASF members that can help with any roadblocks.
>
> == Homogenous Developers ==
>
> Initial committers come from 5 separate organizations. Our intention is
> increase the diversity of contributing developers and their affiliations.
> To date github contributions have come from approximately 50 contributors
> from outside the Twitter team.
>
> == Reliance on Salaried Developers ==
>
> It is expected that Heron development will occur on both salaried time and
> on volunteer time. The majority of initial committers are paid by their
> employers to contribute to this project. We are committed to recruiting
> additional committers from other organizations as well as non-salaried
> committers to join project.
>
> == Relationships with Other Apache Products ==
>
> As mentioned in the Alignment section, Heron implements the Apache Storm
> API and integrates with multiple Apache schedulers (Apache Mesos, Apache
> Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache
> Thrift.
>
> == An Excessive Fascination with the Apache Brand ==
>
> Heron's popularity is growing in the streaming compute space and we are
> long time supporters of the Apache brand. This proposal is not for the
> purpose of generating publicity through. Rather, the primary benefits to
> joining Apache are those of community building and open decision making
> outlined in the Rationale section.
>
> == Documentation ==
>
> This proposal exists online as
> http://wiki.apache.org/incubator/HeronProposal. Extensive documentation
can
> be found on github at https://twitter.github.io/heron and the source code
> is well documented.
>
> == Source and Intellectual Property Submission Plan ==
>
> The Heron codebase is currently hosted on Github:
> https://github.com/twitter/heron. During incubation, the codebase will be
> migrated to Apache infrastructure. The source code is already ASF 2.0
> licensed.
>
> == External Dependencies ==
>
> All external libraries have ASF 2.0 compatible licenses except for pylint.
> The pylint library is GPL licensed, but is only used for pre-build Python
> style checks and is neither bundled with, nor relied upon by, the Heron
> source or binary release artifacts.
>
> == Cryptography ==
>
> Heron does not use any cryptography libraries.
>
> = Required Resources =
>
> == Mailing lists ==
>
>  * private@heron.incubator.apache.org (with moderated subscriptions)
>  * dev@heron.incubator.apache.org
>  * commits@heron.incubator.apache.org
>  * user@heron.incubator.apache.org
>
> == Subversion Directory ==
>
> Git is the preferred source control system: git://git.apache.org/heron
>
> == Issue Tracking ==
>
> JIRA: Heron (HERON)
>
> == Initial Committers ==
>
>  * Andrew Jorgensen (andrew at andrewjorgensen dot com)
>  * Ashvin Agrawal (ashvin at apache dot org)*
>  * Avrilia Floratou (avrilia dot floratou at gmail dot com)
>  * Bill Graham (billgraham at apache dot org)*
>  * Brian Hatfield (bmhatfield at gmail dot com)
>  * Chris Kellogg (cckellogg at gmail dot com)
>  * Huijun Wu (huijun dot wu dot 2010 at gmail dot com)
>  * Karthik Ramasamy (karthik at gmail dot com)
>  * Maosong Fu (maosongfu at gmail dot com)
>  * Neng Lu(freeneng at gmail dot com)
>  * Runhang Li (obj dot runhang at gmail dot com)
>  * Sanjeev Kulkarni (sanjeevrk at gmail dot com)
>  * Supun Kamburugamuve (supun at apache dot org)*
>  * Thomas Sun (tom dot ssf at gmail dot com)
>  * Yaliang Wang (yaliang dot w dot wang at ieee dot org)
>
> == Affiliations ==
>
>  * Andrew Jorgensen (Google)
>  * Ashvin Agrawal (Microsoft)
>  * Avrilia Floratou (Microsoft)
>  * Bill Graham (Twitter)
>  * Brian Hatfield (Google)
>  * Chris Kellogg (Twitter)
>  * Huijun Wu (Twitter)
>  * Karthik Ramasamy (Streamlio)
>  * Maosong Fu (Twitter)
>  * Neng Lu (Twitter)
>  * Runhang Li (Twitter)
>  * Sanjeev Kulkarni (Streamlio)
>  * Supun Kamburugamuve (Indiana University)
>  * Thomas Sun (Twitter)
>  * Yaliang Wang (Twitter)
>
> = Sponsors =
>
> == Champion ==
>
>  * Julien Le Dem (julien at apache dot org)
>
> == Nominated Mentors ==
>
>  * Jake Farrell (jfarrell at apache dot org)
>  * Jacques Nadeau (jacques at apache dot org)
>  * Julien Le Dem (julien at apache dot org)
>  * P. Taylor Goetz (ptgoetz at apache dot org)
>
> == Sponsoring Entity ==
>
> The Apache Incubator
>
> == Footnotes ==
>
>  * 1 - Papers detailing Heron are available at
> http://dl.acm.org/citation.cfm?id=2742788 and
> http://sites.computer.org/debull/A15dec/p15.pdf.
>  * 2 - http://home.apache.org/phonebook.html?uid=billgraham
>  * 3 - http://home.apache.org/phonebook.html?uid=ashvin
>  * 4 - http://home.apache.org/phonebook.html?uid=supun



--
thanks
ashish

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [VOTE] Heron to enter Apache Incubator

Posted by Ashish <pa...@gmail.com>.
+1 (non-binding)

On Fri, Jun 16, 2017 at 1:41 PM, Bill Graham <bi...@gmail.com> wrote:
> Hi,
>
> Based on the discussion on the incubator mailing list[1] I would like to
> call a vote to add Heron to the Apache Incubator.
>
> The full proposal is available below, and is also available on the Apache
> Incubator wiki at:
>     https://wiki.apache.org/incubator/HeronProposal
>
> Please vote:
>   [ ] +1, bring Heron into Incubator
>   [ ] -1, do not bring Heron into Incubator, because...
>
> The vote will open for 7 days until Friday June 23 at 14:00 PT.
>
> Thank you
>
> 1 -
> https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E
>
>
>
> = Heron Proposal =
>
> = Abstract =
> Heron is a real-time, distributed, fault-tolerant stream processing engine
> initially developed by Twitter.
>
> = Proposal =
>
> Heron is a real-time stream processing engine built for high performance,
> ease of manageability, performance predictability and developer
> productivity[1]. We wish to develop a community around Heron to increase
> contributions and see Heron thrive in an open forum.
>
> = Background =
>
> Heron provides the ability for developers to compose directed acyclic
> graphs (DAGs) of real-time query execution logic (i.e. a topology) and
> submit the topology to execute on a pluggable job scheduling system (e.g.,
> Apache Aurora, YARN, Marathon, etc). Users can employ either the native
> Heron API or the Apache Storm API to develop the topology. Heron supports
> the Storm API for ease of migration, but beyond that Heron’s architecture
> differs considerably from Storm’s.
>
> Users submit a topology to the scheduler using the Heron client, which uses
> the Heron binary libraries to deploy all daemons required to run and manage
> the topology. The topology therefore has no reliance on centrally managed
> Heron services, only on a generic job scheduling system, which lends itself
> well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN (among
> others).
>
> The scheduler runs each topology as a job consisting of multiple
> containers. One of the containers runs the topology master, responsible for
> managing the topology. The remaining containers each runs a stream manager
> responsible for data routing, a metrics manager that collects and reports
> various metrics and a number of processes called Heron instances which run
> the user-defined logic on the stream of tuples. Parallelism is achieved via
> process-based isolation of Heron instances, which provides predictable
> performance while simplifying debugging. The containers are allocated and
> managed by the scheduler framework based on resource availability of nodes
> in the cluster. The metadata for the topology, such as the physical plan
> and execution details, are stored in the pluggable Heron State Manager
> (e.g. Apache ZooKeeper).
>
> = Rationale =
>
> Heron is a general-purpose, modular and extensible platform that can be
> leveraged to support common, real-time analytics use cases. There is an
> increasing demand for open-source, scalable real-time analytics systems. We
> believe that Heron can be leveraged by other organizations to build
> streaming applications that can benefit from its robustness, high
> performance, adaptability to cloud environments and ease of use. Moreover,
> we hope that open-sourcing Heron will help to further evolve the technology
> as the project attracts contributors with diverse backgrounds and areas of
> expertise.
>
> We believe the Apache foundation is a great fit as the long-term home for
> Heron, as it provides an established process for community-driven
> development and decision making by consensus. This is exactly the model we
> want for future Heron development.
>
> = Initial Goals =
>
>  * Move the existing codebase, website, documentation, and mailing lists to
> Apache-hosted infrastructure.
>  * Integrate with the Apache development process.
>  * Ensure all dependencies are compliant with Apache License version 2.0.
>  * Incrementally develop and release per Apache guidelines.
>
> = Current Status =
>
> Heron is a stable project used in production at Twitter since 2014 and open
> sourced under the ASL v2 license in 2016. The Heron source code is
> currently hosted at github.com (https://github.com/twitter/heron), which
> will seed the Apache git repository.
>
> = Meritocracy =
>
> By submitting this incubator proposal, we’re expressing our intent to build
> a diverse developer community around Heron that will conduct itself
> according to The Apache Way and use a meritocratic means of building it's
> committer base. Several companies and universities have already expressed
> interest in and contributed to Heron. Our goal is to grow the Heron
> community by encouraging open communication, contribution and participation
> of all types, and ensuring that contributors are recognized appropriately.
>
> = Community =
>
> Heron is currently being used by Twitter, Google, Machine Zone and
> ndustrial.io and has received significant contributions by Microsoft and
> Streamlio. By bringing Heron into the Apache ecosystem, we believe we can
> attract even more developers who are interested in creating real-time
> systems to build the project's contributor base.
>
> == Core Developers ==
>
> Current core developers are engineers from Twitter, Google, Microsoft and
> Streamlio.
>
> == Alignment ==
>
> Heron utilizes a number of Apache technologies. Heron leverages Apache
> ZooKeeper for coordination and has scheduler implementations to integrate
> with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache REEF)
> as well as spout implementations to integrate with Apache Kafka and metrics
> implementations to integrate with Scribe. Heron also implements the Apache
> Storm user-level API, which allows topologies written against Storm to run
> in Heron. We believe that having Heron at Apache will help further the
> growth of the streaming compute community, as well as encourage cooperation
> and developer cross pollination with other Apache projects.
>
> = Known Risks =
>
> == Orphaned Products ==
>
> The risk of the Heron project being abandoned is minimal. It is used in
> production at Twitter and Google and other companies are evaluating or
> adopting it for production use.
>
> == Inexperience with Open Source ==
>
> All of the core contributors to the project have considerable experience
> with open source software development. Bill Graham[2], Ashvin Agrawal[3]
> and Supun Kamburugamuve[4], committers on the project, are PMCs on other
> Apache projects and Bill and Ashvin have gone through the Apache incubator
> process. Twitter has already donated numerous projects to the ASF (e.g.,
> Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be mentored
> by experienced ASF members that can help with any roadblocks.
>
> == Homogenous Developers ==
>
> Initial committers come from 5 separate organizations. Our intention is
> increase the diversity of contributing developers and their affiliations.
> To date github contributions have come from approximately 50 contributors
> from outside the Twitter team.
>
> == Reliance on Salaried Developers ==
>
> It is expected that Heron development will occur on both salaried time and
> on volunteer time. The majority of initial committers are paid by their
> employers to contribute to this project. We are committed to recruiting
> additional committers from other organizations as well as non-salaried
> committers to join project.
>
> == Relationships with Other Apache Products ==
>
> As mentioned in the Alignment section, Heron implements the Apache Storm
> API and integrates with multiple Apache schedulers (Apache Mesos, Apache
> Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache
> Thrift.
>
> == An Excessive Fascination with the Apache Brand ==
>
> Heron's popularity is growing in the streaming compute space and we are
> long time supporters of the Apache brand. This proposal is not for the
> purpose of generating publicity through. Rather, the primary benefits to
> joining Apache are those of community building and open decision making
> outlined in the Rationale section.
>
> == Documentation ==
>
> This proposal exists online as
> http://wiki.apache.org/incubator/HeronProposal. Extensive documentation can
> be found on github at https://twitter.github.io/heron and the source code
> is well documented.
>
> == Source and Intellectual Property Submission Plan ==
>
> The Heron codebase is currently hosted on Github:
> https://github.com/twitter/heron. During incubation, the codebase will be
> migrated to Apache infrastructure. The source code is already ASF 2.0
> licensed.
>
> == External Dependencies ==
>
> All external libraries have ASF 2.0 compatible licenses except for pylint.
> The pylint library is GPL licensed, but is only used for pre-build Python
> style checks and is neither bundled with, nor relied upon by, the Heron
> source or binary release artifacts.
>
> == Cryptography ==
>
> Heron does not use any cryptography libraries.
>
> = Required Resources =
>
> == Mailing lists ==
>
>  * private@heron.incubator.apache.org (with moderated subscriptions)
>  * dev@heron.incubator.apache.org
>  * commits@heron.incubator.apache.org
>  * user@heron.incubator.apache.org
>
> == Subversion Directory ==
>
> Git is the preferred source control system: git://git.apache.org/heron
>
> == Issue Tracking ==
>
> JIRA: Heron (HERON)
>
> == Initial Committers ==
>
>  * Andrew Jorgensen (andrew at andrewjorgensen dot com)
>  * Ashvin Agrawal (ashvin at apache dot org)*
>  * Avrilia Floratou (avrilia dot floratou at gmail dot com)
>  * Bill Graham (billgraham at apache dot org)*
>  * Brian Hatfield (bmhatfield at gmail dot com)
>  * Chris Kellogg (cckellogg at gmail dot com)
>  * Huijun Wu (huijun dot wu dot 2010 at gmail dot com)
>  * Karthik Ramasamy (karthik at gmail dot com)
>  * Maosong Fu (maosongfu at gmail dot com)
>  * Neng Lu(freeneng at gmail dot com)
>  * Runhang Li (obj dot runhang at gmail dot com)
>  * Sanjeev Kulkarni (sanjeevrk at gmail dot com)
>  * Supun Kamburugamuve (supun at apache dot org)*
>  * Thomas Sun (tom dot ssf at gmail dot com)
>  * Yaliang Wang (yaliang dot w dot wang at ieee dot org)
>
> == Affiliations ==
>
>  * Andrew Jorgensen (Google)
>  * Ashvin Agrawal (Microsoft)
>  * Avrilia Floratou (Microsoft)
>  * Bill Graham (Twitter)
>  * Brian Hatfield (Google)
>  * Chris Kellogg (Twitter)
>  * Huijun Wu (Twitter)
>  * Karthik Ramasamy (Streamlio)
>  * Maosong Fu (Twitter)
>  * Neng Lu (Twitter)
>  * Runhang Li (Twitter)
>  * Sanjeev Kulkarni (Streamlio)
>  * Supun Kamburugamuve (Indiana University)
>  * Thomas Sun (Twitter)
>  * Yaliang Wang (Twitter)
>
> = Sponsors =
>
> == Champion ==
>
>  * Julien Le Dem (julien at apache dot org)
>
> == Nominated Mentors ==
>
>  * Jake Farrell (jfarrell at apache dot org)
>  * Jacques Nadeau (jacques at apache dot org)
>  * Julien Le Dem (julien at apache dot org)
>  * P. Taylor Goetz (ptgoetz at apache dot org)
>
> == Sponsoring Entity ==
>
> The Apache Incubator
>
> == Footnotes ==
>
>  * 1 - Papers detailing Heron are available at
> http://dl.acm.org/citation.cfm?id=2742788 and
> http://sites.computer.org/debull/A15dec/p15.pdf.
>  * 2 - http://home.apache.org/phonebook.html?uid=billgraham
>  * 3 - http://home.apache.org/phonebook.html?uid=ashvin
>  * 4 - http://home.apache.org/phonebook.html?uid=supun



-- 
thanks
ashish

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [VOTE] Heron to enter Apache Incubator

Posted by Bill Graham <bi...@gmail.com>.
+1 (non-binding)

On Mon, Jun 19, 2017 at 6:15 AM, Jake Farrell <jf...@apache.org> wrote:

> Thanks John
> Comments inline, will ensure that your points are addressed before the
> first release candidate.
>
> -Jake
>
>
> On Sun, Jun 18, 2017 at 6:35 AM, John D. Ament <jo...@apache.org>
> wrote:
>
>> +1, however a few things to note about the proposal (and follow up will be
>> required when bringing Heron in):
>>
>> - There is no ASF 2.0 license (missed when putting together the proposal)
>>
>
> Will ensure that all licensing checkboxes are addressed before the first
> release candidate goes up for a vote.
>
>
> - The IP section doesn't mention anything about a SGA being sent, is your
>> intention to not send an SGA?
>>
>
> SGA is not required to be filed prior to an incubator acceptance vote, it
> is 100% required before the codebase can be imported by infra, which the
> mentors will ensure does occur. (i've all ready asked the project to get
> this rolling)
>
>
>
>> - The NOTICE for the repo indicates there is some source code from Yahoo!.
>> - The contents of
>> https://github.com/twitter/heron/tree/master/third_party seems
>> to be mostly binary files, and you'll need to clean that up for your first
>> release.
>> - Your 3rd party section mentions everything is ASF 2.0, however this
>> includes glog and similar tools that include an odd buildchain license
>> that
>> is actually GPL, we'll need to get clearance if this is actually compliant
>> or not.  Some of the contents in third_party are missing license headers.
>>
>>
> This is similar to other projects using a local third_party cache
> directory that have come to the Apache Incubator, Cassandra, Mesos and
> Aurora are a couple that jump into mind. We will ensure that this is
> addressed and that no source release contains any of these files.
>
>
>
>> John
>>
>> On Fri, Jun 16, 2017 at 4:41 PM Bill Graham <bi...@gmail.com> wrote:
>>
>> > Hi,
>> >
>> > Based on the discussion on the incubator mailing list[1] I would like to
>> > call a vote to add Heron to the Apache Incubator.
>> >
>> > The full proposal is available below, and is also available on the
>> Apache
>> > Incubator wiki at:
>> >     https://wiki.apache.org/incubator/HeronProposal
>> >
>> > Please vote:
>> >   [ ] +1, bring Heron into Incubator
>> >   [ ] -1, do not bring Heron into Incubator, because...
>> >
>> > The vote will open for 7 days until Friday June 23 at 14:00 PT.
>> >
>> > Thank you
>> >
>> > 1 -
>> >
>> > https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2
>> c9d93b7786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E
>> >
>> >
>> >
>> > = Heron Proposal =
>> >
>> > = Abstract =
>> > Heron is a real-time, distributed, fault-tolerant stream processing
>> engine
>> > initially developed by Twitter.
>> >
>> > = Proposal =
>> >
>> > Heron is a real-time stream processing engine built for high
>> performance,
>> > ease of manageability, performance predictability and developer
>> > productivity[1]. We wish to develop a community around Heron to increase
>> > contributions and see Heron thrive in an open forum.
>> >
>> > = Background =
>> >
>> > Heron provides the ability for developers to compose directed acyclic
>> > graphs (DAGs) of real-time query execution logic (i.e. a topology) and
>> > submit the topology to execute on a pluggable job scheduling system
>> (e.g.,
>> > Apache Aurora, YARN, Marathon, etc). Users can employ either the native
>> > Heron API or the Apache Storm API to develop the topology. Heron
>> supports
>> > the Storm API for ease of migration, but beyond that Heron’s
>> architecture
>> > differs considerably from Storm’s.
>> >
>> > Users submit a topology to the scheduler using the Heron client, which
>> uses
>> > the Heron binary libraries to deploy all daemons required to run and
>> manage
>> > the topology. The topology therefore has no reliance on centrally
>> managed
>> > Heron services, only on a generic job scheduling system, which lends
>> itself
>> > well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN
>> (among
>> > others).
>> >
>> > The scheduler runs each topology as a job consisting of multiple
>> > containers. One of the containers runs the topology master, responsible
>> for
>> > managing the topology. The remaining containers each runs a stream
>> manager
>> > responsible for data routing, a metrics manager that collects and
>> reports
>> > various metrics and a number of processes called Heron instances which
>> run
>> > the user-defined logic on the stream of tuples. Parallelism is achieved
>> via
>> > process-based isolation of Heron instances, which provides predictable
>> > performance while simplifying debugging. The containers are allocated
>> and
>> > managed by the scheduler framework based on resource availability of
>> nodes
>> > in the cluster. The metadata for the topology, such as the physical plan
>> > and execution details, are stored in the pluggable Heron State Manager
>> > (e.g. Apache ZooKeeper).
>> >
>> > = Rationale =
>> >
>> > Heron is a general-purpose, modular and extensible platform that can be
>> > leveraged to support common, real-time analytics use cases. There is an
>> > increasing demand for open-source, scalable real-time analytics
>> systems. We
>> > believe that Heron can be leveraged by other organizations to build
>> > streaming applications that can benefit from its robustness, high
>> > performance, adaptability to cloud environments and ease of use.
>> Moreover,
>> > we hope that open-sourcing Heron will help to further evolve the
>> technology
>> > as the project attracts contributors with diverse backgrounds and areas
>> of
>> > expertise.
>> >
>> > We believe the Apache foundation is a great fit as the long-term home
>> for
>> > Heron, as it provides an established process for community-driven
>> > development and decision making by consensus. This is exactly the model
>> we
>> > want for future Heron development.
>> >
>> > = Initial Goals =
>> >
>> >  * Move the existing codebase, website, documentation, and mailing
>> lists to
>> > Apache-hosted infrastructure.
>> >  * Integrate with the Apache development process.
>> >  * Ensure all dependencies are compliant with Apache License version
>> 2.0.
>> >  * Incrementally develop and release per Apache guidelines.
>> >
>> > = Current Status =
>> >
>> > Heron is a stable project used in production at Twitter since 2014 and
>> open
>> > sourced under the ASL v2 license in 2016. The Heron source code is
>> > currently hosted at github.com (https://github.com/twitter/heron),
>> which
>> > will seed the Apache git repository.
>> >
>> > = Meritocracy =
>> >
>> > By submitting this incubator proposal, we’re expressing our intent to
>> build
>> > a diverse developer community around Heron that will conduct itself
>> > according to The Apache Way and use a meritocratic means of building
>> it's
>> > committer base. Several companies and universities have already
>> expressed
>> > interest in and contributed to Heron. Our goal is to grow the Heron
>> > community by encouraging open communication, contribution and
>> participation
>> > of all types, and ensuring that contributors are recognized
>> appropriately.
>> >
>> > = Community =
>> >
>> > Heron is currently being used by Twitter, Google, Machine Zone and
>> > ndustrial.io and has received significant contributions by Microsoft
>> and
>> > Streamlio. By bringing Heron into the Apache ecosystem, we believe we
>> can
>> > attract even more developers who are interested in creating real-time
>> > systems to build the project's contributor base.
>> >
>> > == Core Developers ==
>> >
>> > Current core developers are engineers from Twitter, Google, Microsoft
>> and
>> > Streamlio.
>> >
>> > == Alignment ==
>> >
>> > Heron utilizes a number of Apache technologies. Heron leverages Apache
>> > ZooKeeper for coordination and has scheduler implementations to
>> integrate
>> > with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache
>> REEF)
>> > as well as spout implementations to integrate with Apache Kafka and
>> metrics
>> > implementations to integrate with Scribe. Heron also implements the
>> Apache
>> > Storm user-level API, which allows topologies written against Storm to
>> run
>> > in Heron. We believe that having Heron at Apache will help further the
>> > growth of the streaming compute community, as well as encourage
>> cooperation
>> > and developer cross pollination with other Apache projects.
>> >
>> > = Known Risks =
>> >
>> > == Orphaned Products ==
>> >
>> > The risk of the Heron project being abandoned is minimal. It is used in
>> > production at Twitter and Google and other companies are evaluating or
>> > adopting it for production use.
>> >
>> > == Inexperience with Open Source ==
>> >
>> > All of the core contributors to the project have considerable experience
>> > with open source software development. Bill Graham[2], Ashvin Agrawal[3]
>> > and Supun Kamburugamuve[4], committers on the project, are PMCs on other
>> > Apache projects and Bill and Ashvin have gone through the Apache
>> incubator
>> > process. Twitter has already donated numerous projects to the ASF (e.g.,
>> > Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be
>> mentored
>> > by experienced ASF members that can help with any roadblocks.
>> >
>> > == Homogenous Developers ==
>> >
>> > Initial committers come from 5 separate organizations. Our intention is
>> > increase the diversity of contributing developers and their
>> affiliations.
>> > To date github contributions have come from approximately 50
>> contributors
>> > from outside the Twitter team.
>> >
>> > == Reliance on Salaried Developers ==
>> >
>> > It is expected that Heron development will occur on both salaried time
>> and
>> > on volunteer time. The majority of initial committers are paid by their
>> > employers to contribute to this project. We are committed to recruiting
>> > additional committers from other organizations as well as non-salaried
>> > committers to join project.
>> >
>> > == Relationships with Other Apache Products ==
>> >
>> > As mentioned in the Alignment section, Heron implements the Apache Storm
>> > API and integrates with multiple Apache schedulers (Apache Mesos, Apache
>> > Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache
>> > Thrift.
>> >
>> > == An Excessive Fascination with the Apache Brand ==
>> >
>> > Heron's popularity is growing in the streaming compute space and we are
>> > long time supporters of the Apache brand. This proposal is not for the
>> > purpose of generating publicity through. Rather, the primary benefits to
>> > joining Apache are those of community building and open decision making
>> > outlined in the Rationale section.
>> >
>> > == Documentation ==
>> >
>> > This proposal exists online as
>> > http://wiki.apache.org/incubator/HeronProposal. Extensive documentation
>> > can
>> > be found on github at https://twitter.github.io/heron and the source
>> code
>> > is well documented.
>> >
>> > == Source and Intellectual Property Submission Plan ==
>> >
>> > The Heron codebase is currently hosted on Github:
>> > https://github.com/twitter/heron. During incubation, the codebase will
>> be
>> > migrated to Apache infrastructure. The source code is already ASF 2.0
>> > licensed.
>> >
>> > == External Dependencies ==
>> >
>> > All external libraries have ASF 2.0 compatible licenses except for
>> pylint.
>> > The pylint library is GPL licensed, but is only used for pre-build
>> Python
>> > style checks and is neither bundled with, nor relied upon by, the Heron
>> > source or binary release artifacts.
>> >
>> > == Cryptography ==
>> >
>> > Heron does not use any cryptography libraries.
>> >
>> > = Required Resources =
>> >
>> > == Mailing lists ==
>> >
>> >  * private@heron.incubator.apache.org (with moderated subscriptions)
>> >  * dev@heron.incubator.apache.org
>> >  * commits@heron.incubator.apache.org
>> >  * user@heron.incubator.apache.org
>> >
>> > == Subversion Directory ==
>> >
>> > Git is the preferred source control system: git://git.apache.org/heron
>> >
>> > == Issue Tracking ==
>> >
>> > JIRA: Heron (HERON)
>> >
>> > == Initial Committers ==
>> >
>> >  * Andrew Jorgensen (andrew at andrewjorgensen dot com)
>> >  * Ashvin Agrawal (ashvin at apache dot org)*
>> >  * Avrilia Floratou (avrilia dot floratou at gmail dot com)
>> >  * Bill Graham (billgraham at apache dot org)*
>> >  * Brian Hatfield (bmhatfield at gmail dot com)
>> >  * Chris Kellogg (cckellogg at gmail dot com)
>> >  * Huijun Wu (huijun dot wu dot 2010 at gmail dot com)
>> >  * Karthik Ramasamy (karthik at gmail dot com)
>> >  * Maosong Fu (maosongfu at gmail dot com)
>> >  * Neng Lu(freeneng at gmail dot com)
>> >  * Runhang Li (obj dot runhang at gmail dot com)
>> >  * Sanjeev Kulkarni (sanjeevrk at gmail dot com)
>> >  * Supun Kamburugamuve (supun at apache dot org)*
>> >  * Thomas Sun (tom dot ssf at gmail dot com)
>> >  * Yaliang Wang (yaliang dot w dot wang at ieee dot org)
>> >
>> > == Affiliations ==
>> >
>> >  * Andrew Jorgensen (Google)
>> >  * Ashvin Agrawal (Microsoft)
>> >  * Avrilia Floratou (Microsoft)
>> >  * Bill Graham (Twitter)
>> >  * Brian Hatfield (Google)
>> >  * Chris Kellogg (Twitter)
>> >  * Huijun Wu (Twitter)
>> >  * Karthik Ramasamy (Streamlio)
>> >  * Maosong Fu (Twitter)
>> >  * Neng Lu (Twitter)
>> >  * Runhang Li (Twitter)
>> >  * Sanjeev Kulkarni (Streamlio)
>> >  * Supun Kamburugamuve (Indiana University)
>> >  * Thomas Sun (Twitter)
>> >  * Yaliang Wang (Twitter)
>> >
>> > = Sponsors =
>> >
>> > == Champion ==
>> >
>> >  * Julien Le Dem (julien at apache dot org)
>> >
>> > == Nominated Mentors ==
>> >
>> >  * Jake Farrell (jfarrell at apache dot org)
>> >  * Jacques Nadeau (jacques at apache dot org)
>> >  * Julien Le Dem (julien at apache dot org)
>> >  * P. Taylor Goetz (ptgoetz at apache dot org)
>> >
>> > == Sponsoring Entity ==
>> >
>> > The Apache Incubator
>> >
>> > == Footnotes ==
>> >
>> >  * 1 - Papers detailing Heron are available at
>> > http://dl.acm.org/citation.cfm?id=2742788 and
>> > http://sites.computer.org/debull/A15dec/p15.pdf.
>> >  * 2 - http://home.apache.org/phonebook.html?uid=billgraham
>> >  * 3 - http://home.apache.org/phonebook.html?uid=ashvin
>> >  * 4 - http://home.apache.org/phonebook.html?uid=supun
>> >
>>
>
>

Re: [VOTE] Heron to enter Apache Incubator

Posted by Jake Farrell <jf...@apache.org>.
Thanks John
Comments inline, will ensure that your points are addressed before the
first release candidate.

-Jake


On Sun, Jun 18, 2017 at 6:35 AM, John D. Ament <jo...@apache.org>
wrote:

> +1, however a few things to note about the proposal (and follow up will be
> required when bringing Heron in):
>
> - There is no ASF 2.0 license (missed when putting together the proposal)
>

Will ensure that all licensing checkboxes are addressed before the first
release candidate goes up for a vote.


- The IP section doesn't mention anything about a SGA being sent, is your
> intention to not send an SGA?
>

SGA is not required to be filed prior to an incubator acceptance vote, it
is 100% required before the codebase can be imported by infra, which the
mentors will ensure does occur. (i've all ready asked the project to get
this rolling)



> - The NOTICE for the repo indicates there is some source code from Yahoo!.
> - The contents of
> https://github.com/twitter/heron/tree/master/third_party seems
> to be mostly binary files, and you'll need to clean that up for your first
> release.
> - Your 3rd party section mentions everything is ASF 2.0, however this
> includes glog and similar tools that include an odd buildchain license that
> is actually GPL, we'll need to get clearance if this is actually compliant
> or not.  Some of the contents in third_party are missing license headers.
>
>
This is similar to other projects using a local third_party cache directory
that have come to the Apache Incubator, Cassandra, Mesos and Aurora are a
couple that jump into mind. We will ensure that this is addressed and that
no source release contains any of these files.



> John
>
> On Fri, Jun 16, 2017 at 4:41 PM Bill Graham <bi...@gmail.com> wrote:
>
> > Hi,
> >
> > Based on the discussion on the incubator mailing list[1] I would like to
> > call a vote to add Heron to the Apache Incubator.
> >
> > The full proposal is available below, and is also available on the Apache
> > Incubator wiki at:
> >     https://wiki.apache.org/incubator/HeronProposal
> >
> > Please vote:
> >   [ ] +1, bring Heron into Incubator
> >   [ ] -1, do not bring Heron into Incubator, because...
> >
> > The vote will open for 7 days until Friday June 23 at 14:00 PT.
> >
> > Thank you
> >
> > 1 -
> >
> > https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7
> 786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E
> >
> >
> >
> > = Heron Proposal =
> >
> > = Abstract =
> > Heron is a real-time, distributed, fault-tolerant stream processing
> engine
> > initially developed by Twitter.
> >
> > = Proposal =
> >
> > Heron is a real-time stream processing engine built for high performance,
> > ease of manageability, performance predictability and developer
> > productivity[1]. We wish to develop a community around Heron to increase
> > contributions and see Heron thrive in an open forum.
> >
> > = Background =
> >
> > Heron provides the ability for developers to compose directed acyclic
> > graphs (DAGs) of real-time query execution logic (i.e. a topology) and
> > submit the topology to execute on a pluggable job scheduling system
> (e.g.,
> > Apache Aurora, YARN, Marathon, etc). Users can employ either the native
> > Heron API or the Apache Storm API to develop the topology. Heron supports
> > the Storm API for ease of migration, but beyond that Heron’s architecture
> > differs considerably from Storm’s.
> >
> > Users submit a topology to the scheduler using the Heron client, which
> uses
> > the Heron binary libraries to deploy all daemons required to run and
> manage
> > the topology. The topology therefore has no reliance on centrally managed
> > Heron services, only on a generic job scheduling system, which lends
> itself
> > well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN (among
> > others).
> >
> > The scheduler runs each topology as a job consisting of multiple
> > containers. One of the containers runs the topology master, responsible
> for
> > managing the topology. The remaining containers each runs a stream
> manager
> > responsible for data routing, a metrics manager that collects and reports
> > various metrics and a number of processes called Heron instances which
> run
> > the user-defined logic on the stream of tuples. Parallelism is achieved
> via
> > process-based isolation of Heron instances, which provides predictable
> > performance while simplifying debugging. The containers are allocated and
> > managed by the scheduler framework based on resource availability of
> nodes
> > in the cluster. The metadata for the topology, such as the physical plan
> > and execution details, are stored in the pluggable Heron State Manager
> > (e.g. Apache ZooKeeper).
> >
> > = Rationale =
> >
> > Heron is a general-purpose, modular and extensible platform that can be
> > leveraged to support common, real-time analytics use cases. There is an
> > increasing demand for open-source, scalable real-time analytics systems.
> We
> > believe that Heron can be leveraged by other organizations to build
> > streaming applications that can benefit from its robustness, high
> > performance, adaptability to cloud environments and ease of use.
> Moreover,
> > we hope that open-sourcing Heron will help to further evolve the
> technology
> > as the project attracts contributors with diverse backgrounds and areas
> of
> > expertise.
> >
> > We believe the Apache foundation is a great fit as the long-term home for
> > Heron, as it provides an established process for community-driven
> > development and decision making by consensus. This is exactly the model
> we
> > want for future Heron development.
> >
> > = Initial Goals =
> >
> >  * Move the existing codebase, website, documentation, and mailing lists
> to
> > Apache-hosted infrastructure.
> >  * Integrate with the Apache development process.
> >  * Ensure all dependencies are compliant with Apache License version 2.0.
> >  * Incrementally develop and release per Apache guidelines.
> >
> > = Current Status =
> >
> > Heron is a stable project used in production at Twitter since 2014 and
> open
> > sourced under the ASL v2 license in 2016. The Heron source code is
> > currently hosted at github.com (https://github.com/twitter/heron), which
> > will seed the Apache git repository.
> >
> > = Meritocracy =
> >
> > By submitting this incubator proposal, we’re expressing our intent to
> build
> > a diverse developer community around Heron that will conduct itself
> > according to The Apache Way and use a meritocratic means of building it's
> > committer base. Several companies and universities have already expressed
> > interest in and contributed to Heron. Our goal is to grow the Heron
> > community by encouraging open communication, contribution and
> participation
> > of all types, and ensuring that contributors are recognized
> appropriately.
> >
> > = Community =
> >
> > Heron is currently being used by Twitter, Google, Machine Zone and
> > ndustrial.io and has received significant contributions by Microsoft and
> > Streamlio. By bringing Heron into the Apache ecosystem, we believe we can
> > attract even more developers who are interested in creating real-time
> > systems to build the project's contributor base.
> >
> > == Core Developers ==
> >
> > Current core developers are engineers from Twitter, Google, Microsoft and
> > Streamlio.
> >
> > == Alignment ==
> >
> > Heron utilizes a number of Apache technologies. Heron leverages Apache
> > ZooKeeper for coordination and has scheduler implementations to integrate
> > with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache
> REEF)
> > as well as spout implementations to integrate with Apache Kafka and
> metrics
> > implementations to integrate with Scribe. Heron also implements the
> Apache
> > Storm user-level API, which allows topologies written against Storm to
> run
> > in Heron. We believe that having Heron at Apache will help further the
> > growth of the streaming compute community, as well as encourage
> cooperation
> > and developer cross pollination with other Apache projects.
> >
> > = Known Risks =
> >
> > == Orphaned Products ==
> >
> > The risk of the Heron project being abandoned is minimal. It is used in
> > production at Twitter and Google and other companies are evaluating or
> > adopting it for production use.
> >
> > == Inexperience with Open Source ==
> >
> > All of the core contributors to the project have considerable experience
> > with open source software development. Bill Graham[2], Ashvin Agrawal[3]
> > and Supun Kamburugamuve[4], committers on the project, are PMCs on other
> > Apache projects and Bill and Ashvin have gone through the Apache
> incubator
> > process. Twitter has already donated numerous projects to the ASF (e.g.,
> > Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be mentored
> > by experienced ASF members that can help with any roadblocks.
> >
> > == Homogenous Developers ==
> >
> > Initial committers come from 5 separate organizations. Our intention is
> > increase the diversity of contributing developers and their affiliations.
> > To date github contributions have come from approximately 50 contributors
> > from outside the Twitter team.
> >
> > == Reliance on Salaried Developers ==
> >
> > It is expected that Heron development will occur on both salaried time
> and
> > on volunteer time. The majority of initial committers are paid by their
> > employers to contribute to this project. We are committed to recruiting
> > additional committers from other organizations as well as non-salaried
> > committers to join project.
> >
> > == Relationships with Other Apache Products ==
> >
> > As mentioned in the Alignment section, Heron implements the Apache Storm
> > API and integrates with multiple Apache schedulers (Apache Mesos, Apache
> > Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache
> > Thrift.
> >
> > == An Excessive Fascination with the Apache Brand ==
> >
> > Heron's popularity is growing in the streaming compute space and we are
> > long time supporters of the Apache brand. This proposal is not for the
> > purpose of generating publicity through. Rather, the primary benefits to
> > joining Apache are those of community building and open decision making
> > outlined in the Rationale section.
> >
> > == Documentation ==
> >
> > This proposal exists online as
> > http://wiki.apache.org/incubator/HeronProposal. Extensive documentation
> > can
> > be found on github at https://twitter.github.io/heron and the source
> code
> > is well documented.
> >
> > == Source and Intellectual Property Submission Plan ==
> >
> > The Heron codebase is currently hosted on Github:
> > https://github.com/twitter/heron. During incubation, the codebase will
> be
> > migrated to Apache infrastructure. The source code is already ASF 2.0
> > licensed.
> >
> > == External Dependencies ==
> >
> > All external libraries have ASF 2.0 compatible licenses except for
> pylint.
> > The pylint library is GPL licensed, but is only used for pre-build Python
> > style checks and is neither bundled with, nor relied upon by, the Heron
> > source or binary release artifacts.
> >
> > == Cryptography ==
> >
> > Heron does not use any cryptography libraries.
> >
> > = Required Resources =
> >
> > == Mailing lists ==
> >
> >  * private@heron.incubator.apache.org (with moderated subscriptions)
> >  * dev@heron.incubator.apache.org
> >  * commits@heron.incubator.apache.org
> >  * user@heron.incubator.apache.org
> >
> > == Subversion Directory ==
> >
> > Git is the preferred source control system: git://git.apache.org/heron
> >
> > == Issue Tracking ==
> >
> > JIRA: Heron (HERON)
> >
> > == Initial Committers ==
> >
> >  * Andrew Jorgensen (andrew at andrewjorgensen dot com)
> >  * Ashvin Agrawal (ashvin at apache dot org)*
> >  * Avrilia Floratou (avrilia dot floratou at gmail dot com)
> >  * Bill Graham (billgraham at apache dot org)*
> >  * Brian Hatfield (bmhatfield at gmail dot com)
> >  * Chris Kellogg (cckellogg at gmail dot com)
> >  * Huijun Wu (huijun dot wu dot 2010 at gmail dot com)
> >  * Karthik Ramasamy (karthik at gmail dot com)
> >  * Maosong Fu (maosongfu at gmail dot com)
> >  * Neng Lu(freeneng at gmail dot com)
> >  * Runhang Li (obj dot runhang at gmail dot com)
> >  * Sanjeev Kulkarni (sanjeevrk at gmail dot com)
> >  * Supun Kamburugamuve (supun at apache dot org)*
> >  * Thomas Sun (tom dot ssf at gmail dot com)
> >  * Yaliang Wang (yaliang dot w dot wang at ieee dot org)
> >
> > == Affiliations ==
> >
> >  * Andrew Jorgensen (Google)
> >  * Ashvin Agrawal (Microsoft)
> >  * Avrilia Floratou (Microsoft)
> >  * Bill Graham (Twitter)
> >  * Brian Hatfield (Google)
> >  * Chris Kellogg (Twitter)
> >  * Huijun Wu (Twitter)
> >  * Karthik Ramasamy (Streamlio)
> >  * Maosong Fu (Twitter)
> >  * Neng Lu (Twitter)
> >  * Runhang Li (Twitter)
> >  * Sanjeev Kulkarni (Streamlio)
> >  * Supun Kamburugamuve (Indiana University)
> >  * Thomas Sun (Twitter)
> >  * Yaliang Wang (Twitter)
> >
> > = Sponsors =
> >
> > == Champion ==
> >
> >  * Julien Le Dem (julien at apache dot org)
> >
> > == Nominated Mentors ==
> >
> >  * Jake Farrell (jfarrell at apache dot org)
> >  * Jacques Nadeau (jacques at apache dot org)
> >  * Julien Le Dem (julien at apache dot org)
> >  * P. Taylor Goetz (ptgoetz at apache dot org)
> >
> > == Sponsoring Entity ==
> >
> > The Apache Incubator
> >
> > == Footnotes ==
> >
> >  * 1 - Papers detailing Heron are available at
> > http://dl.acm.org/citation.cfm?id=2742788 and
> > http://sites.computer.org/debull/A15dec/p15.pdf.
> >  * 2 - http://home.apache.org/phonebook.html?uid=billgraham
> >  * 3 - http://home.apache.org/phonebook.html?uid=ashvin
> >  * 4 - http://home.apache.org/phonebook.html?uid=supun
> >
>

Re: [VOTE] Heron to enter Apache Incubator

Posted by "John D. Ament" <jo...@apache.org>.
All,

I'm requesting that the community holds on closing this vote for a few
extra days to address concerns raised by the Storm community and others.

Please don't interpret this as a -1, but do interpret it as a need for some
extra due dilligence due to code usage from Apache Storm within Heron.

John

On Sun, Jun 18, 2017 at 6:35 AM John D. Ament <jo...@apache.org> wrote:

> +1, however a few things to note about the proposal (and follow up will be
> required when bringing Heron in):
>
> - There is no ASF 2.0 license (missed when putting together the proposal)
> - The IP section doesn't mention anything about a SGA being sent, is your
> intention to not send an SGA?
> - The NOTICE for the repo indicates there is some source code from Yahoo!.
>
> - The contents of https://github.com/twitter/heron/tree/master/third_party seems
> to be mostly binary files, and you'll need to clean that up for your first
> release.
> - Your 3rd party section mentions everything is ASF 2.0, however this
> includes glog and similar tools that include an odd buildchain license that
> is actually GPL, we'll need to get clearance if this is actually compliant
> or not.  Some of the contents in third_party are missing license headers.
>
> John
>
>
> On Fri, Jun 16, 2017 at 4:41 PM Bill Graham <bi...@gmail.com> wrote:
>
>> Hi,
>>
>> Based on the discussion on the incubator mailing list[1] I would like to
>> call a vote to add Heron to the Apache Incubator.
>>
>> The full proposal is available below, and is also available on the Apache
>> Incubator wiki at:
>>     https://wiki.apache.org/incubator/HeronProposal
>>
>> Please vote:
>>   [ ] +1, bring Heron into Incubator
>>   [ ] -1, do not bring Heron into Incubator, because...
>>
>> The vote will open for 7 days until Friday June 23 at 14:00 PT.
>>
>> Thank you
>>
>> 1 -
>>
>> https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E
>>
>>
>>
>> = Heron Proposal =
>>
>> = Abstract =
>> Heron is a real-time, distributed, fault-tolerant stream processing engine
>> initially developed by Twitter.
>>
>> = Proposal =
>>
>> Heron is a real-time stream processing engine built for high performance,
>> ease of manageability, performance predictability and developer
>> productivity[1]. We wish to develop a community around Heron to increase
>> contributions and see Heron thrive in an open forum.
>>
>> = Background =
>>
>> Heron provides the ability for developers to compose directed acyclic
>> graphs (DAGs) of real-time query execution logic (i.e. a topology) and
>> submit the topology to execute on a pluggable job scheduling system (e.g.,
>> Apache Aurora, YARN, Marathon, etc). Users can employ either the native
>> Heron API or the Apache Storm API to develop the topology. Heron supports
>> the Storm API for ease of migration, but beyond that Heron’s architecture
>> differs considerably from Storm’s.
>>
>> Users submit a topology to the scheduler using the Heron client, which
>> uses
>> the Heron binary libraries to deploy all daemons required to run and
>> manage
>> the topology. The topology therefore has no reliance on centrally managed
>> Heron services, only on a generic job scheduling system, which lends
>> itself
>> well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN (among
>> others).
>>
>> The scheduler runs each topology as a job consisting of multiple
>> containers. One of the containers runs the topology master, responsible
>> for
>> managing the topology. The remaining containers each runs a stream manager
>> responsible for data routing, a metrics manager that collects and reports
>> various metrics and a number of processes called Heron instances which run
>> the user-defined logic on the stream of tuples. Parallelism is achieved
>> via
>> process-based isolation of Heron instances, which provides predictable
>> performance while simplifying debugging. The containers are allocated and
>> managed by the scheduler framework based on resource availability of nodes
>> in the cluster. The metadata for the topology, such as the physical plan
>> and execution details, are stored in the pluggable Heron State Manager
>> (e.g. Apache ZooKeeper).
>>
>> = Rationale =
>>
>> Heron is a general-purpose, modular and extensible platform that can be
>> leveraged to support common, real-time analytics use cases. There is an
>> increasing demand for open-source, scalable real-time analytics systems.
>> We
>> believe that Heron can be leveraged by other organizations to build
>> streaming applications that can benefit from its robustness, high
>> performance, adaptability to cloud environments and ease of use. Moreover,
>> we hope that open-sourcing Heron will help to further evolve the
>> technology
>> as the project attracts contributors with diverse backgrounds and areas of
>> expertise.
>>
>> We believe the Apache foundation is a great fit as the long-term home for
>> Heron, as it provides an established process for community-driven
>> development and decision making by consensus. This is exactly the model we
>> want for future Heron development.
>>
>> = Initial Goals =
>>
>>  * Move the existing codebase, website, documentation, and mailing lists
>> to
>> Apache-hosted infrastructure.
>>  * Integrate with the Apache development process.
>>  * Ensure all dependencies are compliant with Apache License version 2.0.
>>  * Incrementally develop and release per Apache guidelines.
>>
>> = Current Status =
>>
>> Heron is a stable project used in production at Twitter since 2014 and
>> open
>> sourced under the ASL v2 license in 2016. The Heron source code is
>> currently hosted at github.com (https://github.com/twitter/heron), which
>> will seed the Apache git repository.
>>
>> = Meritocracy =
>>
>> By submitting this incubator proposal, we’re expressing our intent to
>> build
>> a diverse developer community around Heron that will conduct itself
>> according to The Apache Way and use a meritocratic means of building it's
>> committer base. Several companies and universities have already expressed
>> interest in and contributed to Heron. Our goal is to grow the Heron
>> community by encouraging open communication, contribution and
>> participation
>> of all types, and ensuring that contributors are recognized appropriately.
>>
>> = Community =
>>
>> Heron is currently being used by Twitter, Google, Machine Zone and
>> ndustrial.io and has received significant contributions by Microsoft and
>> Streamlio. By bringing Heron into the Apache ecosystem, we believe we can
>> attract even more developers who are interested in creating real-time
>> systems to build the project's contributor base.
>>
>> == Core Developers ==
>>
>> Current core developers are engineers from Twitter, Google, Microsoft and
>> Streamlio.
>>
>> == Alignment ==
>>
>> Heron utilizes a number of Apache technologies. Heron leverages Apache
>> ZooKeeper for coordination and has scheduler implementations to integrate
>> with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache
>> REEF)
>> as well as spout implementations to integrate with Apache Kafka and
>> metrics
>> implementations to integrate with Scribe. Heron also implements the Apache
>> Storm user-level API, which allows topologies written against Storm to run
>> in Heron. We believe that having Heron at Apache will help further the
>> growth of the streaming compute community, as well as encourage
>> cooperation
>> and developer cross pollination with other Apache projects.
>>
>> = Known Risks =
>>
>> == Orphaned Products ==
>>
>> The risk of the Heron project being abandoned is minimal. It is used in
>> production at Twitter and Google and other companies are evaluating or
>> adopting it for production use.
>>
>> == Inexperience with Open Source ==
>>
>> All of the core contributors to the project have considerable experience
>> with open source software development. Bill Graham[2], Ashvin Agrawal[3]
>> and Supun Kamburugamuve[4], committers on the project, are PMCs on other
>> Apache projects and Bill and Ashvin have gone through the Apache incubator
>> process. Twitter has already donated numerous projects to the ASF (e.g.,
>> Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be mentored
>> by experienced ASF members that can help with any roadblocks.
>>
>> == Homogenous Developers ==
>>
>> Initial committers come from 5 separate organizations. Our intention is
>> increase the diversity of contributing developers and their affiliations.
>> To date github contributions have come from approximately 50 contributors
>> from outside the Twitter team.
>>
>> == Reliance on Salaried Developers ==
>>
>> It is expected that Heron development will occur on both salaried time and
>> on volunteer time. The majority of initial committers are paid by their
>> employers to contribute to this project. We are committed to recruiting
>> additional committers from other organizations as well as non-salaried
>> committers to join project.
>>
>> == Relationships with Other Apache Products ==
>>
>> As mentioned in the Alignment section, Heron implements the Apache Storm
>> API and integrates with multiple Apache schedulers (Apache Mesos, Apache
>> Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache
>> Thrift.
>>
>> == An Excessive Fascination with the Apache Brand ==
>>
>> Heron's popularity is growing in the streaming compute space and we are
>> long time supporters of the Apache brand. This proposal is not for the
>> purpose of generating publicity through. Rather, the primary benefits to
>> joining Apache are those of community building and open decision making
>> outlined in the Rationale section.
>>
>> == Documentation ==
>>
>> This proposal exists online as
>> http://wiki.apache.org/incubator/HeronProposal. Extensive documentation
>> can
>> be found on github at https://twitter.github.io/heron and the source code
>> is well documented.
>>
>> == Source and Intellectual Property Submission Plan ==
>>
>> The Heron codebase is currently hosted on Github:
>> https://github.com/twitter/heron. During incubation, the codebase will be
>> migrated to Apache infrastructure. The source code is already ASF 2.0
>> licensed.
>>
>> == External Dependencies ==
>>
>> All external libraries have ASF 2.0 compatible licenses except for pylint.
>> The pylint library is GPL licensed, but is only used for pre-build Python
>> style checks and is neither bundled with, nor relied upon by, the Heron
>> source or binary release artifacts.
>>
>> == Cryptography ==
>>
>> Heron does not use any cryptography libraries.
>>
>> = Required Resources =
>>
>> == Mailing lists ==
>>
>>  * private@heron.incubator.apache.org (with moderated subscriptions)
>>  * dev@heron.incubator.apache.org
>>  * commits@heron.incubator.apache.org
>>  * user@heron.incubator.apache.org
>>
>> == Subversion Directory ==
>>
>> Git is the preferred source control system: git://git.apache.org/heron
>>
>> == Issue Tracking ==
>>
>> JIRA: Heron (HERON)
>>
>> == Initial Committers ==
>>
>>  * Andrew Jorgensen (andrew at andrewjorgensen dot com)
>>  * Ashvin Agrawal (ashvin at apache dot org)*
>>  * Avrilia Floratou (avrilia dot floratou at gmail dot com)
>>  * Bill Graham (billgraham at apache dot org)*
>>  * Brian Hatfield (bmhatfield at gmail dot com)
>>  * Chris Kellogg (cckellogg at gmail dot com)
>>  * Huijun Wu (huijun dot wu dot 2010 at gmail dot com)
>>  * Karthik Ramasamy (karthik at gmail dot com)
>>  * Maosong Fu (maosongfu at gmail dot com)
>>  * Neng Lu(freeneng at gmail dot com)
>>  * Runhang Li (obj dot runhang at gmail dot com)
>>  * Sanjeev Kulkarni (sanjeevrk at gmail dot com)
>>  * Supun Kamburugamuve (supun at apache dot org)*
>>  * Thomas Sun (tom dot ssf at gmail dot com)
>>  * Yaliang Wang (yaliang dot w dot wang at ieee dot org)
>>
>> == Affiliations ==
>>
>>  * Andrew Jorgensen (Google)
>>  * Ashvin Agrawal (Microsoft)
>>  * Avrilia Floratou (Microsoft)
>>  * Bill Graham (Twitter)
>>  * Brian Hatfield (Google)
>>  * Chris Kellogg (Twitter)
>>  * Huijun Wu (Twitter)
>>  * Karthik Ramasamy (Streamlio)
>>  * Maosong Fu (Twitter)
>>  * Neng Lu (Twitter)
>>  * Runhang Li (Twitter)
>>  * Sanjeev Kulkarni (Streamlio)
>>  * Supun Kamburugamuve (Indiana University)
>>  * Thomas Sun (Twitter)
>>  * Yaliang Wang (Twitter)
>>
>> = Sponsors =
>>
>> == Champion ==
>>
>>  * Julien Le Dem (julien at apache dot org)
>>
>> == Nominated Mentors ==
>>
>>  * Jake Farrell (jfarrell at apache dot org)
>>  * Jacques Nadeau (jacques at apache dot org)
>>  * Julien Le Dem (julien at apache dot org)
>>  * P. Taylor Goetz (ptgoetz at apache dot org)
>>
>> == Sponsoring Entity ==
>>
>> The Apache Incubator
>>
>> == Footnotes ==
>>
>>  * 1 - Papers detailing Heron are available at
>> http://dl.acm.org/citation.cfm?id=2742788 and
>> http://sites.computer.org/debull/A15dec/p15.pdf.
>>  * 2 - http://home.apache.org/phonebook.html?uid=billgraham
>>  * 3 - http://home.apache.org/phonebook.html?uid=ashvin
>>  * 4 - http://home.apache.org/phonebook.html?uid=supun
>>
>

Re: [VOTE] Heron to enter Apache Incubator

Posted by "John D. Ament" <jo...@apache.org>.
+1, however a few things to note about the proposal (and follow up will be
required when bringing Heron in):

- There is no ASF 2.0 license (missed when putting together the proposal)
- The IP section doesn't mention anything about a SGA being sent, is your
intention to not send an SGA?
- The NOTICE for the repo indicates there is some source code from Yahoo!.
- The contents of
https://github.com/twitter/heron/tree/master/third_party seems
to be mostly binary files, and you'll need to clean that up for your first
release.
- Your 3rd party section mentions everything is ASF 2.0, however this
includes glog and similar tools that include an odd buildchain license that
is actually GPL, we'll need to get clearance if this is actually compliant
or not.  Some of the contents in third_party are missing license headers.

John

On Fri, Jun 16, 2017 at 4:41 PM Bill Graham <bi...@gmail.com> wrote:

> Hi,
>
> Based on the discussion on the incubator mailing list[1] I would like to
> call a vote to add Heron to the Apache Incubator.
>
> The full proposal is available below, and is also available on the Apache
> Incubator wiki at:
>     https://wiki.apache.org/incubator/HeronProposal
>
> Please vote:
>   [ ] +1, bring Heron into Incubator
>   [ ] -1, do not bring Heron into Incubator, because...
>
> The vote will open for 7 days until Friday June 23 at 14:00 PT.
>
> Thank you
>
> 1 -
>
> https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E
>
>
>
> = Heron Proposal =
>
> = Abstract =
> Heron is a real-time, distributed, fault-tolerant stream processing engine
> initially developed by Twitter.
>
> = Proposal =
>
> Heron is a real-time stream processing engine built for high performance,
> ease of manageability, performance predictability and developer
> productivity[1]. We wish to develop a community around Heron to increase
> contributions and see Heron thrive in an open forum.
>
> = Background =
>
> Heron provides the ability for developers to compose directed acyclic
> graphs (DAGs) of real-time query execution logic (i.e. a topology) and
> submit the topology to execute on a pluggable job scheduling system (e.g.,
> Apache Aurora, YARN, Marathon, etc). Users can employ either the native
> Heron API or the Apache Storm API to develop the topology. Heron supports
> the Storm API for ease of migration, but beyond that Heron’s architecture
> differs considerably from Storm’s.
>
> Users submit a topology to the scheduler using the Heron client, which uses
> the Heron binary libraries to deploy all daemons required to run and manage
> the topology. The topology therefore has no reliance on centrally managed
> Heron services, only on a generic job scheduling system, which lends itself
> well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN (among
> others).
>
> The scheduler runs each topology as a job consisting of multiple
> containers. One of the containers runs the topology master, responsible for
> managing the topology. The remaining containers each runs a stream manager
> responsible for data routing, a metrics manager that collects and reports
> various metrics and a number of processes called Heron instances which run
> the user-defined logic on the stream of tuples. Parallelism is achieved via
> process-based isolation of Heron instances, which provides predictable
> performance while simplifying debugging. The containers are allocated and
> managed by the scheduler framework based on resource availability of nodes
> in the cluster. The metadata for the topology, such as the physical plan
> and execution details, are stored in the pluggable Heron State Manager
> (e.g. Apache ZooKeeper).
>
> = Rationale =
>
> Heron is a general-purpose, modular and extensible platform that can be
> leveraged to support common, real-time analytics use cases. There is an
> increasing demand for open-source, scalable real-time analytics systems. We
> believe that Heron can be leveraged by other organizations to build
> streaming applications that can benefit from its robustness, high
> performance, adaptability to cloud environments and ease of use. Moreover,
> we hope that open-sourcing Heron will help to further evolve the technology
> as the project attracts contributors with diverse backgrounds and areas of
> expertise.
>
> We believe the Apache foundation is a great fit as the long-term home for
> Heron, as it provides an established process for community-driven
> development and decision making by consensus. This is exactly the model we
> want for future Heron development.
>
> = Initial Goals =
>
>  * Move the existing codebase, website, documentation, and mailing lists to
> Apache-hosted infrastructure.
>  * Integrate with the Apache development process.
>  * Ensure all dependencies are compliant with Apache License version 2.0.
>  * Incrementally develop and release per Apache guidelines.
>
> = Current Status =
>
> Heron is a stable project used in production at Twitter since 2014 and open
> sourced under the ASL v2 license in 2016. The Heron source code is
> currently hosted at github.com (https://github.com/twitter/heron), which
> will seed the Apache git repository.
>
> = Meritocracy =
>
> By submitting this incubator proposal, we’re expressing our intent to build
> a diverse developer community around Heron that will conduct itself
> according to The Apache Way and use a meritocratic means of building it's
> committer base. Several companies and universities have already expressed
> interest in and contributed to Heron. Our goal is to grow the Heron
> community by encouraging open communication, contribution and participation
> of all types, and ensuring that contributors are recognized appropriately.
>
> = Community =
>
> Heron is currently being used by Twitter, Google, Machine Zone and
> ndustrial.io and has received significant contributions by Microsoft and
> Streamlio. By bringing Heron into the Apache ecosystem, we believe we can
> attract even more developers who are interested in creating real-time
> systems to build the project's contributor base.
>
> == Core Developers ==
>
> Current core developers are engineers from Twitter, Google, Microsoft and
> Streamlio.
>
> == Alignment ==
>
> Heron utilizes a number of Apache technologies. Heron leverages Apache
> ZooKeeper for coordination and has scheduler implementations to integrate
> with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache REEF)
> as well as spout implementations to integrate with Apache Kafka and metrics
> implementations to integrate with Scribe. Heron also implements the Apache
> Storm user-level API, which allows topologies written against Storm to run
> in Heron. We believe that having Heron at Apache will help further the
> growth of the streaming compute community, as well as encourage cooperation
> and developer cross pollination with other Apache projects.
>
> = Known Risks =
>
> == Orphaned Products ==
>
> The risk of the Heron project being abandoned is minimal. It is used in
> production at Twitter and Google and other companies are evaluating or
> adopting it for production use.
>
> == Inexperience with Open Source ==
>
> All of the core contributors to the project have considerable experience
> with open source software development. Bill Graham[2], Ashvin Agrawal[3]
> and Supun Kamburugamuve[4], committers on the project, are PMCs on other
> Apache projects and Bill and Ashvin have gone through the Apache incubator
> process. Twitter has already donated numerous projects to the ASF (e.g.,
> Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be mentored
> by experienced ASF members that can help with any roadblocks.
>
> == Homogenous Developers ==
>
> Initial committers come from 5 separate organizations. Our intention is
> increase the diversity of contributing developers and their affiliations.
> To date github contributions have come from approximately 50 contributors
> from outside the Twitter team.
>
> == Reliance on Salaried Developers ==
>
> It is expected that Heron development will occur on both salaried time and
> on volunteer time. The majority of initial committers are paid by their
> employers to contribute to this project. We are committed to recruiting
> additional committers from other organizations as well as non-salaried
> committers to join project.
>
> == Relationships with Other Apache Products ==
>
> As mentioned in the Alignment section, Heron implements the Apache Storm
> API and integrates with multiple Apache schedulers (Apache Mesos, Apache
> Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache
> Thrift.
>
> == An Excessive Fascination with the Apache Brand ==
>
> Heron's popularity is growing in the streaming compute space and we are
> long time supporters of the Apache brand. This proposal is not for the
> purpose of generating publicity through. Rather, the primary benefits to
> joining Apache are those of community building and open decision making
> outlined in the Rationale section.
>
> == Documentation ==
>
> This proposal exists online as
> http://wiki.apache.org/incubator/HeronProposal. Extensive documentation
> can
> be found on github at https://twitter.github.io/heron and the source code
> is well documented.
>
> == Source and Intellectual Property Submission Plan ==
>
> The Heron codebase is currently hosted on Github:
> https://github.com/twitter/heron. During incubation, the codebase will be
> migrated to Apache infrastructure. The source code is already ASF 2.0
> licensed.
>
> == External Dependencies ==
>
> All external libraries have ASF 2.0 compatible licenses except for pylint.
> The pylint library is GPL licensed, but is only used for pre-build Python
> style checks and is neither bundled with, nor relied upon by, the Heron
> source or binary release artifacts.
>
> == Cryptography ==
>
> Heron does not use any cryptography libraries.
>
> = Required Resources =
>
> == Mailing lists ==
>
>  * private@heron.incubator.apache.org (with moderated subscriptions)
>  * dev@heron.incubator.apache.org
>  * commits@heron.incubator.apache.org
>  * user@heron.incubator.apache.org
>
> == Subversion Directory ==
>
> Git is the preferred source control system: git://git.apache.org/heron
>
> == Issue Tracking ==
>
> JIRA: Heron (HERON)
>
> == Initial Committers ==
>
>  * Andrew Jorgensen (andrew at andrewjorgensen dot com)
>  * Ashvin Agrawal (ashvin at apache dot org)*
>  * Avrilia Floratou (avrilia dot floratou at gmail dot com)
>  * Bill Graham (billgraham at apache dot org)*
>  * Brian Hatfield (bmhatfield at gmail dot com)
>  * Chris Kellogg (cckellogg at gmail dot com)
>  * Huijun Wu (huijun dot wu dot 2010 at gmail dot com)
>  * Karthik Ramasamy (karthik at gmail dot com)
>  * Maosong Fu (maosongfu at gmail dot com)
>  * Neng Lu(freeneng at gmail dot com)
>  * Runhang Li (obj dot runhang at gmail dot com)
>  * Sanjeev Kulkarni (sanjeevrk at gmail dot com)
>  * Supun Kamburugamuve (supun at apache dot org)*
>  * Thomas Sun (tom dot ssf at gmail dot com)
>  * Yaliang Wang (yaliang dot w dot wang at ieee dot org)
>
> == Affiliations ==
>
>  * Andrew Jorgensen (Google)
>  * Ashvin Agrawal (Microsoft)
>  * Avrilia Floratou (Microsoft)
>  * Bill Graham (Twitter)
>  * Brian Hatfield (Google)
>  * Chris Kellogg (Twitter)
>  * Huijun Wu (Twitter)
>  * Karthik Ramasamy (Streamlio)
>  * Maosong Fu (Twitter)
>  * Neng Lu (Twitter)
>  * Runhang Li (Twitter)
>  * Sanjeev Kulkarni (Streamlio)
>  * Supun Kamburugamuve (Indiana University)
>  * Thomas Sun (Twitter)
>  * Yaliang Wang (Twitter)
>
> = Sponsors =
>
> == Champion ==
>
>  * Julien Le Dem (julien at apache dot org)
>
> == Nominated Mentors ==
>
>  * Jake Farrell (jfarrell at apache dot org)
>  * Jacques Nadeau (jacques at apache dot org)
>  * Julien Le Dem (julien at apache dot org)
>  * P. Taylor Goetz (ptgoetz at apache dot org)
>
> == Sponsoring Entity ==
>
> The Apache Incubator
>
> == Footnotes ==
>
>  * 1 - Papers detailing Heron are available at
> http://dl.acm.org/citation.cfm?id=2742788 and
> http://sites.computer.org/debull/A15dec/p15.pdf.
>  * 2 - http://home.apache.org/phonebook.html?uid=billgraham
>  * 3 - http://home.apache.org/phonebook.html?uid=ashvin
>  * 4 - http://home.apache.org/phonebook.html?uid=supun
>

Re: [VOTE] Heron to enter Apache Incubator

Posted by Sijie Guo <gu...@gmail.com>.
+1 (non-binding)

- Sijie

On Fri, Jun 16, 2017 at 1:41 PM, Bill Graham <bi...@gmail.com> wrote:

> Hi,
>
> Based on the discussion on the incubator mailing list[1] I would like to
> call a vote to add Heron to the Apache Incubator.
>
> The full proposal is available below, and is also available on the Apache
> Incubator wiki at:
>     https://wiki.apache.org/incubator/HeronProposal
>
> Please vote:
>   [ ] +1, bring Heron into Incubator
>   [ ] -1, do not bring Heron into Incubator, because...
>
> The vote will open for 7 days until Friday June 23 at 14:00 PT.
>
> Thank you
>
> 1 -
> https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7
> 786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E
>
>
>
> = Heron Proposal =
>
> = Abstract =
> Heron is a real-time, distributed, fault-tolerant stream processing engine
> initially developed by Twitter.
>
> = Proposal =
>
> Heron is a real-time stream processing engine built for high performance,
> ease of manageability, performance predictability and developer
> productivity[1]. We wish to develop a community around Heron to increase
> contributions and see Heron thrive in an open forum.
>
> = Background =
>
> Heron provides the ability for developers to compose directed acyclic
> graphs (DAGs) of real-time query execution logic (i.e. a topology) and
> submit the topology to execute on a pluggable job scheduling system (e.g.,
> Apache Aurora, YARN, Marathon, etc). Users can employ either the native
> Heron API or the Apache Storm API to develop the topology. Heron supports
> the Storm API for ease of migration, but beyond that Heron’s architecture
> differs considerably from Storm’s.
>
> Users submit a topology to the scheduler using the Heron client, which uses
> the Heron binary libraries to deploy all daemons required to run and manage
> the topology. The topology therefore has no reliance on centrally managed
> Heron services, only on a generic job scheduling system, which lends itself
> well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN (among
> others).
>
> The scheduler runs each topology as a job consisting of multiple
> containers. One of the containers runs the topology master, responsible for
> managing the topology. The remaining containers each runs a stream manager
> responsible for data routing, a metrics manager that collects and reports
> various metrics and a number of processes called Heron instances which run
> the user-defined logic on the stream of tuples. Parallelism is achieved via
> process-based isolation of Heron instances, which provides predictable
> performance while simplifying debugging. The containers are allocated and
> managed by the scheduler framework based on resource availability of nodes
> in the cluster. The metadata for the topology, such as the physical plan
> and execution details, are stored in the pluggable Heron State Manager
> (e.g. Apache ZooKeeper).
>
> = Rationale =
>
> Heron is a general-purpose, modular and extensible platform that can be
> leveraged to support common, real-time analytics use cases. There is an
> increasing demand for open-source, scalable real-time analytics systems. We
> believe that Heron can be leveraged by other organizations to build
> streaming applications that can benefit from its robustness, high
> performance, adaptability to cloud environments and ease of use. Moreover,
> we hope that open-sourcing Heron will help to further evolve the technology
> as the project attracts contributors with diverse backgrounds and areas of
> expertise.
>
> We believe the Apache foundation is a great fit as the long-term home for
> Heron, as it provides an established process for community-driven
> development and decision making by consensus. This is exactly the model we
> want for future Heron development.
>
> = Initial Goals =
>
>  * Move the existing codebase, website, documentation, and mailing lists to
> Apache-hosted infrastructure.
>  * Integrate with the Apache development process.
>  * Ensure all dependencies are compliant with Apache License version 2.0.
>  * Incrementally develop and release per Apache guidelines.
>
> = Current Status =
>
> Heron is a stable project used in production at Twitter since 2014 and open
> sourced under the ASL v2 license in 2016. The Heron source code is
> currently hosted at github.com (https://github.com/twitter/heron), which
> will seed the Apache git repository.
>
> = Meritocracy =
>
> By submitting this incubator proposal, we’re expressing our intent to build
> a diverse developer community around Heron that will conduct itself
> according to The Apache Way and use a meritocratic means of building it's
> committer base. Several companies and universities have already expressed
> interest in and contributed to Heron. Our goal is to grow the Heron
> community by encouraging open communication, contribution and participation
> of all types, and ensuring that contributors are recognized appropriately.
>
> = Community =
>
> Heron is currently being used by Twitter, Google, Machine Zone and
> ndustrial.io and has received significant contributions by Microsoft and
> Streamlio. By bringing Heron into the Apache ecosystem, we believe we can
> attract even more developers who are interested in creating real-time
> systems to build the project's contributor base.
>
> == Core Developers ==
>
> Current core developers are engineers from Twitter, Google, Microsoft and
> Streamlio.
>
> == Alignment ==
>
> Heron utilizes a number of Apache technologies. Heron leverages Apache
> ZooKeeper for coordination and has scheduler implementations to integrate
> with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache REEF)
> as well as spout implementations to integrate with Apache Kafka and metrics
> implementations to integrate with Scribe. Heron also implements the Apache
> Storm user-level API, which allows topologies written against Storm to run
> in Heron. We believe that having Heron at Apache will help further the
> growth of the streaming compute community, as well as encourage cooperation
> and developer cross pollination with other Apache projects.
>
> = Known Risks =
>
> == Orphaned Products ==
>
> The risk of the Heron project being abandoned is minimal. It is used in
> production at Twitter and Google and other companies are evaluating or
> adopting it for production use.
>
> == Inexperience with Open Source ==
>
> All of the core contributors to the project have considerable experience
> with open source software development. Bill Graham[2], Ashvin Agrawal[3]
> and Supun Kamburugamuve[4], committers on the project, are PMCs on other
> Apache projects and Bill and Ashvin have gone through the Apache incubator
> process. Twitter has already donated numerous projects to the ASF (e.g.,
> Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be mentored
> by experienced ASF members that can help with any roadblocks.
>
> == Homogenous Developers ==
>
> Initial committers come from 5 separate organizations. Our intention is
> increase the diversity of contributing developers and their affiliations.
> To date github contributions have come from approximately 50 contributors
> from outside the Twitter team.
>
> == Reliance on Salaried Developers ==
>
> It is expected that Heron development will occur on both salaried time and
> on volunteer time. The majority of initial committers are paid by their
> employers to contribute to this project. We are committed to recruiting
> additional committers from other organizations as well as non-salaried
> committers to join project.
>
> == Relationships with Other Apache Products ==
>
> As mentioned in the Alignment section, Heron implements the Apache Storm
> API and integrates with multiple Apache schedulers (Apache Mesos, Apache
> Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache
> Thrift.
>
> == An Excessive Fascination with the Apache Brand ==
>
> Heron's popularity is growing in the streaming compute space and we are
> long time supporters of the Apache brand. This proposal is not for the
> purpose of generating publicity through. Rather, the primary benefits to
> joining Apache are those of community building and open decision making
> outlined in the Rationale section.
>
> == Documentation ==
>
> This proposal exists online as
> http://wiki.apache.org/incubator/HeronProposal. Extensive documentation
> can
> be found on github at https://twitter.github.io/heron and the source code
> is well documented.
>
> == Source and Intellectual Property Submission Plan ==
>
> The Heron codebase is currently hosted on Github:
> https://github.com/twitter/heron. During incubation, the codebase will be
> migrated to Apache infrastructure. The source code is already ASF 2.0
> licensed.
>
> == External Dependencies ==
>
> All external libraries have ASF 2.0 compatible licenses except for pylint.
> The pylint library is GPL licensed, but is only used for pre-build Python
> style checks and is neither bundled with, nor relied upon by, the Heron
> source or binary release artifacts.
>
> == Cryptography ==
>
> Heron does not use any cryptography libraries.
>
> = Required Resources =
>
> == Mailing lists ==
>
>  * private@heron.incubator.apache.org (with moderated subscriptions)
>  * dev@heron.incubator.apache.org
>  * commits@heron.incubator.apache.org
>  * user@heron.incubator.apache.org
>
> == Subversion Directory ==
>
> Git is the preferred source control system: git://git.apache.org/heron
>
> == Issue Tracking ==
>
> JIRA: Heron (HERON)
>
> == Initial Committers ==
>
>  * Andrew Jorgensen (andrew at andrewjorgensen dot com)
>  * Ashvin Agrawal (ashvin at apache dot org)*
>  * Avrilia Floratou (avrilia dot floratou at gmail dot com)
>  * Bill Graham (billgraham at apache dot org)*
>  * Brian Hatfield (bmhatfield at gmail dot com)
>  * Chris Kellogg (cckellogg at gmail dot com)
>  * Huijun Wu (huijun dot wu dot 2010 at gmail dot com)
>  * Karthik Ramasamy (karthik at gmail dot com)
>  * Maosong Fu (maosongfu at gmail dot com)
>  * Neng Lu(freeneng at gmail dot com)
>  * Runhang Li (obj dot runhang at gmail dot com)
>  * Sanjeev Kulkarni (sanjeevrk at gmail dot com)
>  * Supun Kamburugamuve (supun at apache dot org)*
>  * Thomas Sun (tom dot ssf at gmail dot com)
>  * Yaliang Wang (yaliang dot w dot wang at ieee dot org)
>
> == Affiliations ==
>
>  * Andrew Jorgensen (Google)
>  * Ashvin Agrawal (Microsoft)
>  * Avrilia Floratou (Microsoft)
>  * Bill Graham (Twitter)
>  * Brian Hatfield (Google)
>  * Chris Kellogg (Twitter)
>  * Huijun Wu (Twitter)
>  * Karthik Ramasamy (Streamlio)
>  * Maosong Fu (Twitter)
>  * Neng Lu (Twitter)
>  * Runhang Li (Twitter)
>  * Sanjeev Kulkarni (Streamlio)
>  * Supun Kamburugamuve (Indiana University)
>  * Thomas Sun (Twitter)
>  * Yaliang Wang (Twitter)
>
> = Sponsors =
>
> == Champion ==
>
>  * Julien Le Dem (julien at apache dot org)
>
> == Nominated Mentors ==
>
>  * Jake Farrell (jfarrell at apache dot org)
>  * Jacques Nadeau (jacques at apache dot org)
>  * Julien Le Dem (julien at apache dot org)
>  * P. Taylor Goetz (ptgoetz at apache dot org)
>
> == Sponsoring Entity ==
>
> The Apache Incubator
>
> == Footnotes ==
>
>  * 1 - Papers detailing Heron are available at
> http://dl.acm.org/citation.cfm?id=2742788 and
> http://sites.computer.org/debull/A15dec/p15.pdf.
>  * 2 - http://home.apache.org/phonebook.html?uid=billgraham
>  * 3 - http://home.apache.org/phonebook.html?uid=ashvin
>  * 4 - http://home.apache.org/phonebook.html?uid=supun
>

Re: [VOTE] Heron to enter Apache Incubator

Posted by Ashvin A <aa...@gmail.com>.
+1

On Fri, Jun 16, 2017 at 2:00 PM, Karthik Ramasamy <kr...@gmail.com>
wrote:

> +1
>
> On 2017-06-16 13:50 (-0700), Roy Lenferink <le...@gmail.com> wrote:
> > +1
> >
> > Op vr 16 jun. 2017 om 22:49 schreef Julien Le Dem <
> julien.ledem@gmail.com>
> >
> > > +1
> > >
> > > On Fri, Jun 16, 2017 at 1:46 PM, William Markito Oliveira <
> > > markito@apache.org> wrote:
> > >
> > > > +1
> > > >
> > > > On Fri, Jun 16, 2017 at 3:45 PM, Debo Dutta (dedutta) <
> dedutta@cisco.com
> > > >
> > > > wrote:
> > > >
> > > > > +1
> > > > >
> > > > > On 6/16/17, 1:41 PM, "Bill Graham" <bi...@gmail.com> wrote:
> > > > >
> > > > >     Hi,
> > > > >
> > > > >     Based on the discussion on the incubator mailing list[1] I
> would
> > > like
> > > > > to
> > > > >     call a vote to add Heron to the Apache Incubator.
> > > > >
> > > > >     The full proposal is available below, and is also available on
> the
> > > > > Apache
> > > > >     Incubator wiki at:
> > > > >         https://wiki.apache.org/incubator/HeronProposal
> > > > >
> > > > >     Please vote:
> > > > >       [ ] +1, bring Heron into Incubator
> > > > >       [ ] -1, do not bring Heron into Incubator, because...
> > > > >
> > > > >     The vote will open for 7 days until Friday June 23 at 14:00 PT.
> > > > >
> > > > >     Thank you
> > > > >
> > > > >     1 -
> > > > >
> > > https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7
> > > > > 786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E
> > > > >
> > > > >
> > > > >
> > > > >     = Heron Proposal =
> > > > >
> > > > >     = Abstract =
> > > > >     Heron is a real-time, distributed, fault-tolerant stream
> processing
> > > > > engine
> > > > >     initially developed by Twitter.
> > > > >
> > > > >     = Proposal =
> > > > >
> > > > >     Heron is a real-time stream processing engine built for high
> > > > > performance,
> > > > >     ease of manageability, performance predictability and developer
> > > > >     productivity[1]. We wish to develop a community around Heron to
> > > > > increase
> > > > >     contributions and see Heron thrive in an open forum.
> > > > >
> > > > >     = Background =
> > > > >
> > > > >     Heron provides the ability for developers to compose directed
> > > acyclic
> > > > >     graphs (DAGs) of real-time query execution logic (i.e. a
> topology)
> > > > and
> > > > >     submit the topology to execute on a pluggable job scheduling
> system
> > > > > (e.g.,
> > > > >     Apache Aurora, YARN, Marathon, etc). Users can employ either
> the
> > > > native
> > > > >     Heron API or the Apache Storm API to develop the topology.
> Heron
> > > > > supports
> > > > >     the Storm API for ease of migration, but beyond that Heron’s
> > > > > architecture
> > > > >     differs considerably from Storm’s.
> > > > >
> > > > >     Users submit a topology to the scheduler using the Heron
> client,
> > > > which
> > > > > uses
> > > > >     the Heron binary libraries to deploy all daemons required to
> run
> > > and
> > > > > manage
> > > > >     the topology. The topology therefore has no reliance on
> centrally
> > > > > managed
> > > > >     Heron services, only on a generic job scheduling system, which
> > > lends
> > > > > itself
> > > > >     well to be run on top of Apache Aurora/Mesos or Apache
> Hadoop/YARN
> > > > > (among
> > > > >     others).
> > > > >
> > > > >     The scheduler runs each topology as a job consisting of
> multiple
> > > > >     containers. One of the containers runs the topology master,
> > > > > responsible for
> > > > >     managing the topology. The remaining containers each runs a
> stream
> > > > > manager
> > > > >     responsible for data routing, a metrics manager that collects
> and
> > > > > reports
> > > > >     various metrics and a number of processes called Heron
> instances
> > > > which
> > > > > run
> > > > >     the user-defined logic on the stream of tuples. Parallelism is
> > > > > achieved via
> > > > >     process-based isolation of Heron instances, which provides
> > > > predictable
> > > > >     performance while simplifying debugging. The containers are
> > > allocated
> > > > > and
> > > > >     managed by the scheduler framework based on resource
> availability
> > > of
> > > > > nodes
> > > > >     in the cluster. The metadata for the topology, such as the
> physical
> > > > > plan
> > > > >     and execution details, are stored in the pluggable Heron State
> > > > Manager
> > > > >     (e.g. Apache ZooKeeper).
> > > > >
> > > > >     = Rationale =
> > > > >
> > > > >     Heron is a general-purpose, modular and extensible platform
> that
> > > can
> > > > be
> > > > >     leveraged to support common, real-time analytics use cases.
> There
> > > is
> > > > an
> > > > >     increasing demand for open-source, scalable real-time analytics
> > > > > systems. We
> > > > >     believe that Heron can be leveraged by other organizations to
> build
> > > > >     streaming applications that can benefit from its robustness,
> high
> > > > >     performance, adaptability to cloud environments and ease of
> use.
> > > > > Moreover,
> > > > >     we hope that open-sourcing Heron will help to further evolve
> the
> > > > > technology
> > > > >     as the project attracts contributors with diverse backgrounds
> and
> > > > > areas of
> > > > >     expertise.
> > > > >
> > > > >     We believe the Apache foundation is a great fit as the
> long-term
> > > home
> > > > > for
> > > > >     Heron, as it provides an established process for
> community-driven
> > > > >     development and decision making by consensus. This is exactly
> the
> > > > > model we
> > > > >     want for future Heron development.
> > > > >
> > > > >     = Initial Goals =
> > > > >
> > > > >      * Move the existing codebase, website, documentation, and
> mailing
> > > > > lists to
> > > > >     Apache-hosted infrastructure.
> > > > >      * Integrate with the Apache development process.
> > > > >      * Ensure all dependencies are compliant with Apache License
> > > version
> > > > > 2.0.
> > > > >      * Incrementally develop and release per Apache guidelines.
> > > > >
> > > > >     = Current Status =
> > > > >
> > > > >     Heron is a stable project used in production at Twitter since
> 2014
> > > > and
> > > > > open
> > > > >     sourced under the ASL v2 license in 2016. The Heron source
> code is
> > > > >     currently hosted at github.com (https://github.com/twitter/
> heron),
> > > > > which
> > > > >     will seed the Apache git repository.
> > > > >
> > > > >     = Meritocracy =
> > > > >
> > > > >     By submitting this incubator proposal, we’re expressing our
> intent
> > > to
> > > > > build
> > > > >     a diverse developer community around Heron that will conduct
> itself
> > > > >     according to The Apache Way and use a meritocratic means of
> > > building
> > > > > it's
> > > > >     committer base. Several companies and universities have already
> > > > > expressed
> > > > >     interest in and contributed to Heron. Our goal is to grow the
> Heron
> > > > >     community by encouraging open communication, contribution and
> > > > > participation
> > > > >     of all types, and ensuring that contributors are recognized
> > > > > appropriately.
> > > > >
> > > > >     = Community =
> > > > >
> > > > >     Heron is currently being used by Twitter, Google, Machine Zone
> and
> > > > >     ndustrial.io and has received significant contributions by
> > > Microsoft
> > > > > and
> > > > >     Streamlio. By bringing Heron into the Apache ecosystem, we
> believe
> > > we
> > > > > can
> > > > >     attract even more developers who are interested in creating
> > > real-time
> > > > >     systems to build the project's contributor base.
> > > > >
> > > > >     == Core Developers ==
> > > > >
> > > > >     Current core developers are engineers from Twitter, Google,
> > > Microsoft
> > > > > and
> > > > >     Streamlio.
> > > > >
> > > > >     == Alignment ==
> > > > >
> > > > >     Heron utilizes a number of Apache technologies. Heron leverages
> > > > Apache
> > > > >     ZooKeeper for coordination and has scheduler implementations to
> > > > > integrate
> > > > >     with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via
> > > Apache
> > > > > REEF)
> > > > >     as well as spout implementations to integrate with Apache
> Kafka and
> > > > > metrics
> > > > >     implementations to integrate with Scribe. Heron also
> implements the
> > > > > Apache
> > > > >     Storm user-level API, which allows topologies written against
> Storm
> > > > to
> > > > > run
> > > > >     in Heron. We believe that having Heron at Apache will help
> further
> > > > the
> > > > >     growth of the streaming compute community, as well as encourage
> > > > > cooperation
> > > > >     and developer cross pollination with other Apache projects.
> > > > >
> > > > >     = Known Risks =
> > > > >
> > > > >     == Orphaned Products ==
> > > > >
> > > > >     The risk of the Heron project being abandoned is minimal. It is
> > > used
> > > > in
> > > > >     production at Twitter and Google and other companies are
> evaluating
> > > > or
> > > > >     adopting it for production use.
> > > > >
> > > > >     == Inexperience with Open Source ==
> > > > >
> > > > >     All of the core contributors to the project have considerable
> > > > > experience
> > > > >     with open source software development. Bill Graham[2], Ashvin
> > > > > Agrawal[3]
> > > > >     and Supun Kamburugamuve[4], committers on the project, are
> PMCs on
> > > > > other
> > > > >     Apache projects and Bill and Ashvin have gone through the
> Apache
> > > > > incubator
> > > > >     process. Twitter has already donated numerous projects to the
> ASF
> > > > > (e.g.,
> > > > >     Apache Mesos, Apache Aurora, Apache Parquet). We also plan to
> be
> > > > > mentored
> > > > >     by experienced ASF members that can help with any roadblocks.
> > > > >
> > > > >     == Homogenous Developers ==
> > > > >
> > > > >     Initial committers come from 5 separate organizations. Our
> > > intention
> > > > is
> > > > >     increase the diversity of contributing developers and their
> > > > > affiliations.
> > > > >     To date github contributions have come from approximately 50
> > > > > contributors
> > > > >     from outside the Twitter team.
> > > > >
> > > > >     == Reliance on Salaried Developers ==
> > > > >
> > > > >     It is expected that Heron development will occur on both
> salaried
> > > > time
> > > > > and
> > > > >     on volunteer time. The majority of initial committers are paid
> by
> > > > their
> > > > >     employers to contribute to this project. We are committed to
> > > > recruiting
> > > > >     additional committers from other organizations as well as
> > > > non-salaried
> > > > >     committers to join project.
> > > > >
> > > > >     == Relationships with Other Apache Products ==
> > > > >
> > > > >     As mentioned in the Alignment section, Heron implements the
> Apache
> > > > > Storm
> > > > >     API and integrates with multiple Apache schedulers (Apache
> Mesos,
> > > > > Apache
> > > > >     Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper
> and
> > > > Apache
> > > > >     Thrift.
> > > > >
> > > > >     == An Excessive Fascination with the Apache Brand ==
> > > > >
> > > > >     Heron's popularity is growing in the streaming compute space
> and we
> > > > are
> > > > >     long time supporters of the Apache brand. This proposal is not
> for
> > > > the
> > > > >     purpose of generating publicity through. Rather, the primary
> > > benefits
> > > > > to
> > > > >     joining Apache are those of community building and open
> decision
> > > > making
> > > > >     outlined in the Rationale section.
> > > > >
> > > > >     == Documentation ==
> > > > >
> > > > >     This proposal exists online as
> > > > >     http://wiki.apache.org/incubator/HeronProposal. Extensive
> > > > > documentation can
> > > > >     be found on github at https://twitter.github.io/heron and the
> > > source
> > > > > code
> > > > >     is well documented.
> > > > >
> > > > >     == Source and Intellectual Property Submission Plan ==
> > > > >
> > > > >     The Heron codebase is currently hosted on Github:
> > > > >     https://github.com/twitter/heron. During incubation, the
> codebase
> > > > > will be
> > > > >     migrated to Apache infrastructure. The source code is already
> ASF
> > > 2.0
> > > > >     licensed.
> > > > >
> > > > >     == External Dependencies ==
> > > > >
> > > > >     All external libraries have ASF 2.0 compatible licenses except
> for
> > > > > pylint.
> > > > >     The pylint library is GPL licensed, but is only used for
> pre-build
> > > > > Python
> > > > >     style checks and is neither bundled with, nor relied upon by,
> the
> > > > Heron
> > > > >     source or binary release artifacts.
> > > > >
> > > > >     == Cryptography ==
> > > > >
> > > > >     Heron does not use any cryptography libraries.
> > > > >
> > > > >     = Required Resources =
> > > > >
> > > > >     == Mailing lists ==
> > > > >
> > > > >      * private@heron.incubator.apache.org (with moderated
> > > subscriptions)
> > > > >      * dev@heron.incubator.apache.org
> > > > >      * commits@heron.incubator.apache.org
> > > > >      * user@heron.incubator.apache.org
> > > > >
> > > > >     == Subversion Directory ==
> > > > >
> > > > >     Git is the preferred source control system: git://
> > > > git.apache.org/heron
> > > > >
> > > > >     == Issue Tracking ==
> > > > >
> > > > >     JIRA: Heron (HERON)
> > > > >
> > > > >     == Initial Committers ==
> > > > >
> > > > >      * Andrew Jorgensen (andrew at andrewjorgensen dot com)
> > > > >      * Ashvin Agrawal (ashvin at apache dot org)*
> > > > >      * Avrilia Floratou (avrilia dot floratou at gmail dot com)
> > > > >      * Bill Graham (billgraham at apache dot org)*
> > > > >      * Brian Hatfield (bmhatfield at gmail dot com)
> > > > >      * Chris Kellogg (cckellogg at gmail dot com)
> > > > >      * Huijun Wu (huijun dot wu dot 2010 at gmail dot com)
> > > > >      * Karthik Ramasamy (karthik at gmail dot com)
> > > > >      * Maosong Fu (maosongfu at gmail dot com)
> > > > >      * Neng Lu(freeneng at gmail dot com)
> > > > >      * Runhang Li (obj dot runhang at gmail dot com)
> > > > >      * Sanjeev Kulkarni (sanjeevrk at gmail dot com)
> > > > >      * Supun Kamburugamuve (supun at apache dot org)*
> > > > >      * Thomas Sun (tom dot ssf at gmail dot com)
> > > > >      * Yaliang Wang (yaliang dot w dot wang at ieee dot org)
> > > > >
> > > > >     == Affiliations ==
> > > > >
> > > > >      * Andrew Jorgensen (Google)
> > > > >      * Ashvin Agrawal (Microsoft)
> > > > >      * Avrilia Floratou (Microsoft)
> > > > >      * Bill Graham (Twitter)
> > > > >      * Brian Hatfield (Google)
> > > > >      * Chris Kellogg (Twitter)
> > > > >      * Huijun Wu (Twitter)
> > > > >      * Karthik Ramasamy (Streamlio)
> > > > >      * Maosong Fu (Twitter)
> > > > >      * Neng Lu (Twitter)
> > > > >      * Runhang Li (Twitter)
> > > > >      * Sanjeev Kulkarni (Streamlio)
> > > > >      * Supun Kamburugamuve (Indiana University)
> > > > >      * Thomas Sun (Twitter)
> > > > >      * Yaliang Wang (Twitter)
> > > > >
> > > > >     = Sponsors =
> > > > >
> > > > >     == Champion ==
> > > > >
> > > > >      * Julien Le Dem (julien at apache dot org)
> > > > >
> > > > >     == Nominated Mentors ==
> > > > >
> > > > >      * Jake Farrell (jfarrell at apache dot org)
> > > > >      * Jacques Nadeau (jacques at apache dot org)
> > > > >      * Julien Le Dem (julien at apache dot org)
> > > > >      * P. Taylor Goetz (ptgoetz at apache dot org)
> > > > >
> > > > >     == Sponsoring Entity ==
> > > > >
> > > > >     The Apache Incubator
> > > > >
> > > > >     == Footnotes ==
> > > > >
> > > > >      * 1 - Papers detailing Heron are available at
> > > > >     http://dl.acm.org/citation.cfm?id=2742788 and
> > > > >     http://sites.computer.org/debull/A15dec/p15.pdf.
> > > > >      * 2 - http://home.apache.org/phonebook.html?uid=billgraham
> > > > >      * 3 - http://home.apache.org/phonebook.html?uid=ashvin
> > > > >      * 4 - http://home.apache.org/phonebook.html?uid=supun
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>

Re: [VOTE] Heron to enter Apache Incubator

Posted by Karthik Ramasamy <kr...@gmail.com>.
+1

On 2017-06-16 13:50 (-0700), Roy Lenferink <le...@gmail.com> wrote: 
> +1
> 
> Op vr 16 jun. 2017 om 22:49 schreef Julien Le Dem <ju...@gmail.com>
> 
> > +1
> >
> > On Fri, Jun 16, 2017 at 1:46 PM, William Markito Oliveira <
> > markito@apache.org> wrote:
> >
> > > +1
> > >
> > > On Fri, Jun 16, 2017 at 3:45 PM, Debo Dutta (dedutta) <dedutta@cisco.com
> > >
> > > wrote:
> > >
> > > > +1
> > > >
> > > > On 6/16/17, 1:41 PM, "Bill Graham" <bi...@gmail.com> wrote:
> > > >
> > > >     Hi,
> > > >
> > > >     Based on the discussion on the incubator mailing list[1] I would
> > like
> > > > to
> > > >     call a vote to add Heron to the Apache Incubator.
> > > >
> > > >     The full proposal is available below, and is also available on the
> > > > Apache
> > > >     Incubator wiki at:
> > > >         https://wiki.apache.org/incubator/HeronProposal
> > > >
> > > >     Please vote:
> > > >       [ ] +1, bring Heron into Incubator
> > > >       [ ] -1, do not bring Heron into Incubator, because...
> > > >
> > > >     The vote will open for 7 days until Friday June 23 at 14:00 PT.
> > > >
> > > >     Thank you
> > > >
> > > >     1 -
> > > >
> > https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7
> > > > 786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E
> > > >
> > > >
> > > >
> > > >     = Heron Proposal =
> > > >
> > > >     = Abstract =
> > > >     Heron is a real-time, distributed, fault-tolerant stream processing
> > > > engine
> > > >     initially developed by Twitter.
> > > >
> > > >     = Proposal =
> > > >
> > > >     Heron is a real-time stream processing engine built for high
> > > > performance,
> > > >     ease of manageability, performance predictability and developer
> > > >     productivity[1]. We wish to develop a community around Heron to
> > > > increase
> > > >     contributions and see Heron thrive in an open forum.
> > > >
> > > >     = Background =
> > > >
> > > >     Heron provides the ability for developers to compose directed
> > acyclic
> > > >     graphs (DAGs) of real-time query execution logic (i.e. a topology)
> > > and
> > > >     submit the topology to execute on a pluggable job scheduling system
> > > > (e.g.,
> > > >     Apache Aurora, YARN, Marathon, etc). Users can employ either the
> > > native
> > > >     Heron API or the Apache Storm API to develop the topology. Heron
> > > > supports
> > > >     the Storm API for ease of migration, but beyond that Heron’s
> > > > architecture
> > > >     differs considerably from Storm’s.
> > > >
> > > >     Users submit a topology to the scheduler using the Heron client,
> > > which
> > > > uses
> > > >     the Heron binary libraries to deploy all daemons required to run
> > and
> > > > manage
> > > >     the topology. The topology therefore has no reliance on centrally
> > > > managed
> > > >     Heron services, only on a generic job scheduling system, which
> > lends
> > > > itself
> > > >     well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN
> > > > (among
> > > >     others).
> > > >
> > > >     The scheduler runs each topology as a job consisting of multiple
> > > >     containers. One of the containers runs the topology master,
> > > > responsible for
> > > >     managing the topology. The remaining containers each runs a stream
> > > > manager
> > > >     responsible for data routing, a metrics manager that collects and
> > > > reports
> > > >     various metrics and a number of processes called Heron instances
> > > which
> > > > run
> > > >     the user-defined logic on the stream of tuples. Parallelism is
> > > > achieved via
> > > >     process-based isolation of Heron instances, which provides
> > > predictable
> > > >     performance while simplifying debugging. The containers are
> > allocated
> > > > and
> > > >     managed by the scheduler framework based on resource availability
> > of
> > > > nodes
> > > >     in the cluster. The metadata for the topology, such as the physical
> > > > plan
> > > >     and execution details, are stored in the pluggable Heron State
> > > Manager
> > > >     (e.g. Apache ZooKeeper).
> > > >
> > > >     = Rationale =
> > > >
> > > >     Heron is a general-purpose, modular and extensible platform that
> > can
> > > be
> > > >     leveraged to support common, real-time analytics use cases. There
> > is
> > > an
> > > >     increasing demand for open-source, scalable real-time analytics
> > > > systems. We
> > > >     believe that Heron can be leveraged by other organizations to build
> > > >     streaming applications that can benefit from its robustness, high
> > > >     performance, adaptability to cloud environments and ease of use.
> > > > Moreover,
> > > >     we hope that open-sourcing Heron will help to further evolve the
> > > > technology
> > > >     as the project attracts contributors with diverse backgrounds and
> > > > areas of
> > > >     expertise.
> > > >
> > > >     We believe the Apache foundation is a great fit as the long-term
> > home
> > > > for
> > > >     Heron, as it provides an established process for community-driven
> > > >     development and decision making by consensus. This is exactly the
> > > > model we
> > > >     want for future Heron development.
> > > >
> > > >     = Initial Goals =
> > > >
> > > >      * Move the existing codebase, website, documentation, and mailing
> > > > lists to
> > > >     Apache-hosted infrastructure.
> > > >      * Integrate with the Apache development process.
> > > >      * Ensure all dependencies are compliant with Apache License
> > version
> > > > 2.0.
> > > >      * Incrementally develop and release per Apache guidelines.
> > > >
> > > >     = Current Status =
> > > >
> > > >     Heron is a stable project used in production at Twitter since 2014
> > > and
> > > > open
> > > >     sourced under the ASL v2 license in 2016. The Heron source code is
> > > >     currently hosted at github.com (https://github.com/twitter/heron),
> > > > which
> > > >     will seed the Apache git repository.
> > > >
> > > >     = Meritocracy =
> > > >
> > > >     By submitting this incubator proposal, we’re expressing our intent
> > to
> > > > build
> > > >     a diverse developer community around Heron that will conduct itself
> > > >     according to The Apache Way and use a meritocratic means of
> > building
> > > > it's
> > > >     committer base. Several companies and universities have already
> > > > expressed
> > > >     interest in and contributed to Heron. Our goal is to grow the Heron
> > > >     community by encouraging open communication, contribution and
> > > > participation
> > > >     of all types, and ensuring that contributors are recognized
> > > > appropriately.
> > > >
> > > >     = Community =
> > > >
> > > >     Heron is currently being used by Twitter, Google, Machine Zone and
> > > >     ndustrial.io and has received significant contributions by
> > Microsoft
> > > > and
> > > >     Streamlio. By bringing Heron into the Apache ecosystem, we believe
> > we
> > > > can
> > > >     attract even more developers who are interested in creating
> > real-time
> > > >     systems to build the project's contributor base.
> > > >
> > > >     == Core Developers ==
> > > >
> > > >     Current core developers are engineers from Twitter, Google,
> > Microsoft
> > > > and
> > > >     Streamlio.
> > > >
> > > >     == Alignment ==
> > > >
> > > >     Heron utilizes a number of Apache technologies. Heron leverages
> > > Apache
> > > >     ZooKeeper for coordination and has scheduler implementations to
> > > > integrate
> > > >     with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via
> > Apache
> > > > REEF)
> > > >     as well as spout implementations to integrate with Apache Kafka and
> > > > metrics
> > > >     implementations to integrate with Scribe. Heron also implements the
> > > > Apache
> > > >     Storm user-level API, which allows topologies written against Storm
> > > to
> > > > run
> > > >     in Heron. We believe that having Heron at Apache will help further
> > > the
> > > >     growth of the streaming compute community, as well as encourage
> > > > cooperation
> > > >     and developer cross pollination with other Apache projects.
> > > >
> > > >     = Known Risks =
> > > >
> > > >     == Orphaned Products ==
> > > >
> > > >     The risk of the Heron project being abandoned is minimal. It is
> > used
> > > in
> > > >     production at Twitter and Google and other companies are evaluating
> > > or
> > > >     adopting it for production use.
> > > >
> > > >     == Inexperience with Open Source ==
> > > >
> > > >     All of the core contributors to the project have considerable
> > > > experience
> > > >     with open source software development. Bill Graham[2], Ashvin
> > > > Agrawal[3]
> > > >     and Supun Kamburugamuve[4], committers on the project, are PMCs on
> > > > other
> > > >     Apache projects and Bill and Ashvin have gone through the Apache
> > > > incubator
> > > >     process. Twitter has already donated numerous projects to the ASF
> > > > (e.g.,
> > > >     Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be
> > > > mentored
> > > >     by experienced ASF members that can help with any roadblocks.
> > > >
> > > >     == Homogenous Developers ==
> > > >
> > > >     Initial committers come from 5 separate organizations. Our
> > intention
> > > is
> > > >     increase the diversity of contributing developers and their
> > > > affiliations.
> > > >     To date github contributions have come from approximately 50
> > > > contributors
> > > >     from outside the Twitter team.
> > > >
> > > >     == Reliance on Salaried Developers ==
> > > >
> > > >     It is expected that Heron development will occur on both salaried
> > > time
> > > > and
> > > >     on volunteer time. The majority of initial committers are paid by
> > > their
> > > >     employers to contribute to this project. We are committed to
> > > recruiting
> > > >     additional committers from other organizations as well as
> > > non-salaried
> > > >     committers to join project.
> > > >
> > > >     == Relationships with Other Apache Products ==
> > > >
> > > >     As mentioned in the Alignment section, Heron implements the Apache
> > > > Storm
> > > >     API and integrates with multiple Apache schedulers (Apache Mesos,
> > > > Apache
> > > >     Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and
> > > Apache
> > > >     Thrift.
> > > >
> > > >     == An Excessive Fascination with the Apache Brand ==
> > > >
> > > >     Heron's popularity is growing in the streaming compute space and we
> > > are
> > > >     long time supporters of the Apache brand. This proposal is not for
> > > the
> > > >     purpose of generating publicity through. Rather, the primary
> > benefits
> > > > to
> > > >     joining Apache are those of community building and open decision
> > > making
> > > >     outlined in the Rationale section.
> > > >
> > > >     == Documentation ==
> > > >
> > > >     This proposal exists online as
> > > >     http://wiki.apache.org/incubator/HeronProposal. Extensive
> > > > documentation can
> > > >     be found on github at https://twitter.github.io/heron and the
> > source
> > > > code
> > > >     is well documented.
> > > >
> > > >     == Source and Intellectual Property Submission Plan ==
> > > >
> > > >     The Heron codebase is currently hosted on Github:
> > > >     https://github.com/twitter/heron. During incubation, the codebase
> > > > will be
> > > >     migrated to Apache infrastructure. The source code is already ASF
> > 2.0
> > > >     licensed.
> > > >
> > > >     == External Dependencies ==
> > > >
> > > >     All external libraries have ASF 2.0 compatible licenses except for
> > > > pylint.
> > > >     The pylint library is GPL licensed, but is only used for pre-build
> > > > Python
> > > >     style checks and is neither bundled with, nor relied upon by, the
> > > Heron
> > > >     source or binary release artifacts.
> > > >
> > > >     == Cryptography ==
> > > >
> > > >     Heron does not use any cryptography libraries.
> > > >
> > > >     = Required Resources =
> > > >
> > > >     == Mailing lists ==
> > > >
> > > >      * private@heron.incubator.apache.org (with moderated
> > subscriptions)
> > > >      * dev@heron.incubator.apache.org
> > > >      * commits@heron.incubator.apache.org
> > > >      * user@heron.incubator.apache.org
> > > >
> > > >     == Subversion Directory ==
> > > >
> > > >     Git is the preferred source control system: git://
> > > git.apache.org/heron
> > > >
> > > >     == Issue Tracking ==
> > > >
> > > >     JIRA: Heron (HERON)
> > > >
> > > >     == Initial Committers ==
> > > >
> > > >      * Andrew Jorgensen (andrew at andrewjorgensen dot com)
> > > >      * Ashvin Agrawal (ashvin at apache dot org)*
> > > >      * Avrilia Floratou (avrilia dot floratou at gmail dot com)
> > > >      * Bill Graham (billgraham at apache dot org)*
> > > >      * Brian Hatfield (bmhatfield at gmail dot com)
> > > >      * Chris Kellogg (cckellogg at gmail dot com)
> > > >      * Huijun Wu (huijun dot wu dot 2010 at gmail dot com)
> > > >      * Karthik Ramasamy (karthik at gmail dot com)
> > > >      * Maosong Fu (maosongfu at gmail dot com)
> > > >      * Neng Lu(freeneng at gmail dot com)
> > > >      * Runhang Li (obj dot runhang at gmail dot com)
> > > >      * Sanjeev Kulkarni (sanjeevrk at gmail dot com)
> > > >      * Supun Kamburugamuve (supun at apache dot org)*
> > > >      * Thomas Sun (tom dot ssf at gmail dot com)
> > > >      * Yaliang Wang (yaliang dot w dot wang at ieee dot org)
> > > >
> > > >     == Affiliations ==
> > > >
> > > >      * Andrew Jorgensen (Google)
> > > >      * Ashvin Agrawal (Microsoft)
> > > >      * Avrilia Floratou (Microsoft)
> > > >      * Bill Graham (Twitter)
> > > >      * Brian Hatfield (Google)
> > > >      * Chris Kellogg (Twitter)
> > > >      * Huijun Wu (Twitter)
> > > >      * Karthik Ramasamy (Streamlio)
> > > >      * Maosong Fu (Twitter)
> > > >      * Neng Lu (Twitter)
> > > >      * Runhang Li (Twitter)
> > > >      * Sanjeev Kulkarni (Streamlio)
> > > >      * Supun Kamburugamuve (Indiana University)
> > > >      * Thomas Sun (Twitter)
> > > >      * Yaliang Wang (Twitter)
> > > >
> > > >     = Sponsors =
> > > >
> > > >     == Champion ==
> > > >
> > > >      * Julien Le Dem (julien at apache dot org)
> > > >
> > > >     == Nominated Mentors ==
> > > >
> > > >      * Jake Farrell (jfarrell at apache dot org)
> > > >      * Jacques Nadeau (jacques at apache dot org)
> > > >      * Julien Le Dem (julien at apache dot org)
> > > >      * P. Taylor Goetz (ptgoetz at apache dot org)
> > > >
> > > >     == Sponsoring Entity ==
> > > >
> > > >     The Apache Incubator
> > > >
> > > >     == Footnotes ==
> > > >
> > > >      * 1 - Papers detailing Heron are available at
> > > >     http://dl.acm.org/citation.cfm?id=2742788 and
> > > >     http://sites.computer.org/debull/A15dec/p15.pdf.
> > > >      * 2 - http://home.apache.org/phonebook.html?uid=billgraham
> > > >      * 3 - http://home.apache.org/phonebook.html?uid=ashvin
> > > >      * 4 - http://home.apache.org/phonebook.html?uid=supun
> > > >
> > > >
> > > >
> > >
> >
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [VOTE] Heron to enter Apache Incubator

Posted by Roy Lenferink <le...@gmail.com>.
+1

Op vr 16 jun. 2017 om 22:49 schreef Julien Le Dem <ju...@gmail.com>

> +1
>
> On Fri, Jun 16, 2017 at 1:46 PM, William Markito Oliveira <
> markito@apache.org> wrote:
>
> > +1
> >
> > On Fri, Jun 16, 2017 at 3:45 PM, Debo Dutta (dedutta) <dedutta@cisco.com
> >
> > wrote:
> >
> > > +1
> > >
> > > On 6/16/17, 1:41 PM, "Bill Graham" <bi...@gmail.com> wrote:
> > >
> > >     Hi,
> > >
> > >     Based on the discussion on the incubator mailing list[1] I would
> like
> > > to
> > >     call a vote to add Heron to the Apache Incubator.
> > >
> > >     The full proposal is available below, and is also available on the
> > > Apache
> > >     Incubator wiki at:
> > >         https://wiki.apache.org/incubator/HeronProposal
> > >
> > >     Please vote:
> > >       [ ] +1, bring Heron into Incubator
> > >       [ ] -1, do not bring Heron into Incubator, because...
> > >
> > >     The vote will open for 7 days until Friday June 23 at 14:00 PT.
> > >
> > >     Thank you
> > >
> > >     1 -
> > >
> https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7
> > > 786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E
> > >
> > >
> > >
> > >     = Heron Proposal =
> > >
> > >     = Abstract =
> > >     Heron is a real-time, distributed, fault-tolerant stream processing
> > > engine
> > >     initially developed by Twitter.
> > >
> > >     = Proposal =
> > >
> > >     Heron is a real-time stream processing engine built for high
> > > performance,
> > >     ease of manageability, performance predictability and developer
> > >     productivity[1]. We wish to develop a community around Heron to
> > > increase
> > >     contributions and see Heron thrive in an open forum.
> > >
> > >     = Background =
> > >
> > >     Heron provides the ability for developers to compose directed
> acyclic
> > >     graphs (DAGs) of real-time query execution logic (i.e. a topology)
> > and
> > >     submit the topology to execute on a pluggable job scheduling system
> > > (e.g.,
> > >     Apache Aurora, YARN, Marathon, etc). Users can employ either the
> > native
> > >     Heron API or the Apache Storm API to develop the topology. Heron
> > > supports
> > >     the Storm API for ease of migration, but beyond that Heron’s
> > > architecture
> > >     differs considerably from Storm’s.
> > >
> > >     Users submit a topology to the scheduler using the Heron client,
> > which
> > > uses
> > >     the Heron binary libraries to deploy all daemons required to run
> and
> > > manage
> > >     the topology. The topology therefore has no reliance on centrally
> > > managed
> > >     Heron services, only on a generic job scheduling system, which
> lends
> > > itself
> > >     well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN
> > > (among
> > >     others).
> > >
> > >     The scheduler runs each topology as a job consisting of multiple
> > >     containers. One of the containers runs the topology master,
> > > responsible for
> > >     managing the topology. The remaining containers each runs a stream
> > > manager
> > >     responsible for data routing, a metrics manager that collects and
> > > reports
> > >     various metrics and a number of processes called Heron instances
> > which
> > > run
> > >     the user-defined logic on the stream of tuples. Parallelism is
> > > achieved via
> > >     process-based isolation of Heron instances, which provides
> > predictable
> > >     performance while simplifying debugging. The containers are
> allocated
> > > and
> > >     managed by the scheduler framework based on resource availability
> of
> > > nodes
> > >     in the cluster. The metadata for the topology, such as the physical
> > > plan
> > >     and execution details, are stored in the pluggable Heron State
> > Manager
> > >     (e.g. Apache ZooKeeper).
> > >
> > >     = Rationale =
> > >
> > >     Heron is a general-purpose, modular and extensible platform that
> can
> > be
> > >     leveraged to support common, real-time analytics use cases. There
> is
> > an
> > >     increasing demand for open-source, scalable real-time analytics
> > > systems. We
> > >     believe that Heron can be leveraged by other organizations to build
> > >     streaming applications that can benefit from its robustness, high
> > >     performance, adaptability to cloud environments and ease of use.
> > > Moreover,
> > >     we hope that open-sourcing Heron will help to further evolve the
> > > technology
> > >     as the project attracts contributors with diverse backgrounds and
> > > areas of
> > >     expertise.
> > >
> > >     We believe the Apache foundation is a great fit as the long-term
> home
> > > for
> > >     Heron, as it provides an established process for community-driven
> > >     development and decision making by consensus. This is exactly the
> > > model we
> > >     want for future Heron development.
> > >
> > >     = Initial Goals =
> > >
> > >      * Move the existing codebase, website, documentation, and mailing
> > > lists to
> > >     Apache-hosted infrastructure.
> > >      * Integrate with the Apache development process.
> > >      * Ensure all dependencies are compliant with Apache License
> version
> > > 2.0.
> > >      * Incrementally develop and release per Apache guidelines.
> > >
> > >     = Current Status =
> > >
> > >     Heron is a stable project used in production at Twitter since 2014
> > and
> > > open
> > >     sourced under the ASL v2 license in 2016. The Heron source code is
> > >     currently hosted at github.com (https://github.com/twitter/heron),
> > > which
> > >     will seed the Apache git repository.
> > >
> > >     = Meritocracy =
> > >
> > >     By submitting this incubator proposal, we’re expressing our intent
> to
> > > build
> > >     a diverse developer community around Heron that will conduct itself
> > >     according to The Apache Way and use a meritocratic means of
> building
> > > it's
> > >     committer base. Several companies and universities have already
> > > expressed
> > >     interest in and contributed to Heron. Our goal is to grow the Heron
> > >     community by encouraging open communication, contribution and
> > > participation
> > >     of all types, and ensuring that contributors are recognized
> > > appropriately.
> > >
> > >     = Community =
> > >
> > >     Heron is currently being used by Twitter, Google, Machine Zone and
> > >     ndustrial.io and has received significant contributions by
> Microsoft
> > > and
> > >     Streamlio. By bringing Heron into the Apache ecosystem, we believe
> we
> > > can
> > >     attract even more developers who are interested in creating
> real-time
> > >     systems to build the project's contributor base.
> > >
> > >     == Core Developers ==
> > >
> > >     Current core developers are engineers from Twitter, Google,
> Microsoft
> > > and
> > >     Streamlio.
> > >
> > >     == Alignment ==
> > >
> > >     Heron utilizes a number of Apache technologies. Heron leverages
> > Apache
> > >     ZooKeeper for coordination and has scheduler implementations to
> > > integrate
> > >     with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via
> Apache
> > > REEF)
> > >     as well as spout implementations to integrate with Apache Kafka and
> > > metrics
> > >     implementations to integrate with Scribe. Heron also implements the
> > > Apache
> > >     Storm user-level API, which allows topologies written against Storm
> > to
> > > run
> > >     in Heron. We believe that having Heron at Apache will help further
> > the
> > >     growth of the streaming compute community, as well as encourage
> > > cooperation
> > >     and developer cross pollination with other Apache projects.
> > >
> > >     = Known Risks =
> > >
> > >     == Orphaned Products ==
> > >
> > >     The risk of the Heron project being abandoned is minimal. It is
> used
> > in
> > >     production at Twitter and Google and other companies are evaluating
> > or
> > >     adopting it for production use.
> > >
> > >     == Inexperience with Open Source ==
> > >
> > >     All of the core contributors to the project have considerable
> > > experience
> > >     with open source software development. Bill Graham[2], Ashvin
> > > Agrawal[3]
> > >     and Supun Kamburugamuve[4], committers on the project, are PMCs on
> > > other
> > >     Apache projects and Bill and Ashvin have gone through the Apache
> > > incubator
> > >     process. Twitter has already donated numerous projects to the ASF
> > > (e.g.,
> > >     Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be
> > > mentored
> > >     by experienced ASF members that can help with any roadblocks.
> > >
> > >     == Homogenous Developers ==
> > >
> > >     Initial committers come from 5 separate organizations. Our
> intention
> > is
> > >     increase the diversity of contributing developers and their
> > > affiliations.
> > >     To date github contributions have come from approximately 50
> > > contributors
> > >     from outside the Twitter team.
> > >
> > >     == Reliance on Salaried Developers ==
> > >
> > >     It is expected that Heron development will occur on both salaried
> > time
> > > and
> > >     on volunteer time. The majority of initial committers are paid by
> > their
> > >     employers to contribute to this project. We are committed to
> > recruiting
> > >     additional committers from other organizations as well as
> > non-salaried
> > >     committers to join project.
> > >
> > >     == Relationships with Other Apache Products ==
> > >
> > >     As mentioned in the Alignment section, Heron implements the Apache
> > > Storm
> > >     API and integrates with multiple Apache schedulers (Apache Mesos,
> > > Apache
> > >     Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and
> > Apache
> > >     Thrift.
> > >
> > >     == An Excessive Fascination with the Apache Brand ==
> > >
> > >     Heron's popularity is growing in the streaming compute space and we
> > are
> > >     long time supporters of the Apache brand. This proposal is not for
> > the
> > >     purpose of generating publicity through. Rather, the primary
> benefits
> > > to
> > >     joining Apache are those of community building and open decision
> > making
> > >     outlined in the Rationale section.
> > >
> > >     == Documentation ==
> > >
> > >     This proposal exists online as
> > >     http://wiki.apache.org/incubator/HeronProposal. Extensive
> > > documentation can
> > >     be found on github at https://twitter.github.io/heron and the
> source
> > > code
> > >     is well documented.
> > >
> > >     == Source and Intellectual Property Submission Plan ==
> > >
> > >     The Heron codebase is currently hosted on Github:
> > >     https://github.com/twitter/heron. During incubation, the codebase
> > > will be
> > >     migrated to Apache infrastructure. The source code is already ASF
> 2.0
> > >     licensed.
> > >
> > >     == External Dependencies ==
> > >
> > >     All external libraries have ASF 2.0 compatible licenses except for
> > > pylint.
> > >     The pylint library is GPL licensed, but is only used for pre-build
> > > Python
> > >     style checks and is neither bundled with, nor relied upon by, the
> > Heron
> > >     source or binary release artifacts.
> > >
> > >     == Cryptography ==
> > >
> > >     Heron does not use any cryptography libraries.
> > >
> > >     = Required Resources =
> > >
> > >     == Mailing lists ==
> > >
> > >      * private@heron.incubator.apache.org (with moderated
> subscriptions)
> > >      * dev@heron.incubator.apache.org
> > >      * commits@heron.incubator.apache.org
> > >      * user@heron.incubator.apache.org
> > >
> > >     == Subversion Directory ==
> > >
> > >     Git is the preferred source control system: git://
> > git.apache.org/heron
> > >
> > >     == Issue Tracking ==
> > >
> > >     JIRA: Heron (HERON)
> > >
> > >     == Initial Committers ==
> > >
> > >      * Andrew Jorgensen (andrew at andrewjorgensen dot com)
> > >      * Ashvin Agrawal (ashvin at apache dot org)*
> > >      * Avrilia Floratou (avrilia dot floratou at gmail dot com)
> > >      * Bill Graham (billgraham at apache dot org)*
> > >      * Brian Hatfield (bmhatfield at gmail dot com)
> > >      * Chris Kellogg (cckellogg at gmail dot com)
> > >      * Huijun Wu (huijun dot wu dot 2010 at gmail dot com)
> > >      * Karthik Ramasamy (karthik at gmail dot com)
> > >      * Maosong Fu (maosongfu at gmail dot com)
> > >      * Neng Lu(freeneng at gmail dot com)
> > >      * Runhang Li (obj dot runhang at gmail dot com)
> > >      * Sanjeev Kulkarni (sanjeevrk at gmail dot com)
> > >      * Supun Kamburugamuve (supun at apache dot org)*
> > >      * Thomas Sun (tom dot ssf at gmail dot com)
> > >      * Yaliang Wang (yaliang dot w dot wang at ieee dot org)
> > >
> > >     == Affiliations ==
> > >
> > >      * Andrew Jorgensen (Google)
> > >      * Ashvin Agrawal (Microsoft)
> > >      * Avrilia Floratou (Microsoft)
> > >      * Bill Graham (Twitter)
> > >      * Brian Hatfield (Google)
> > >      * Chris Kellogg (Twitter)
> > >      * Huijun Wu (Twitter)
> > >      * Karthik Ramasamy (Streamlio)
> > >      * Maosong Fu (Twitter)
> > >      * Neng Lu (Twitter)
> > >      * Runhang Li (Twitter)
> > >      * Sanjeev Kulkarni (Streamlio)
> > >      * Supun Kamburugamuve (Indiana University)
> > >      * Thomas Sun (Twitter)
> > >      * Yaliang Wang (Twitter)
> > >
> > >     = Sponsors =
> > >
> > >     == Champion ==
> > >
> > >      * Julien Le Dem (julien at apache dot org)
> > >
> > >     == Nominated Mentors ==
> > >
> > >      * Jake Farrell (jfarrell at apache dot org)
> > >      * Jacques Nadeau (jacques at apache dot org)
> > >      * Julien Le Dem (julien at apache dot org)
> > >      * P. Taylor Goetz (ptgoetz at apache dot org)
> > >
> > >     == Sponsoring Entity ==
> > >
> > >     The Apache Incubator
> > >
> > >     == Footnotes ==
> > >
> > >      * 1 - Papers detailing Heron are available at
> > >     http://dl.acm.org/citation.cfm?id=2742788 and
> > >     http://sites.computer.org/debull/A15dec/p15.pdf.
> > >      * 2 - http://home.apache.org/phonebook.html?uid=billgraham
> > >      * 3 - http://home.apache.org/phonebook.html?uid=ashvin
> > >      * 4 - http://home.apache.org/phonebook.html?uid=supun
> > >
> > >
> > >
> >
>

Re: [VOTE] Heron to enter Apache Incubator

Posted by Julien Le Dem <ju...@gmail.com>.
+1

On Fri, Jun 16, 2017 at 1:46 PM, William Markito Oliveira <
markito@apache.org> wrote:

> +1
>
> On Fri, Jun 16, 2017 at 3:45 PM, Debo Dutta (dedutta) <de...@cisco.com>
> wrote:
>
> > +1
> >
> > On 6/16/17, 1:41 PM, "Bill Graham" <bi...@gmail.com> wrote:
> >
> >     Hi,
> >
> >     Based on the discussion on the incubator mailing list[1] I would like
> > to
> >     call a vote to add Heron to the Apache Incubator.
> >
> >     The full proposal is available below, and is also available on the
> > Apache
> >     Incubator wiki at:
> >         https://wiki.apache.org/incubator/HeronProposal
> >
> >     Please vote:
> >       [ ] +1, bring Heron into Incubator
> >       [ ] -1, do not bring Heron into Incubator, because...
> >
> >     The vote will open for 7 days until Friday June 23 at 14:00 PT.
> >
> >     Thank you
> >
> >     1 -
> >     https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7
> > 786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E
> >
> >
> >
> >     = Heron Proposal =
> >
> >     = Abstract =
> >     Heron is a real-time, distributed, fault-tolerant stream processing
> > engine
> >     initially developed by Twitter.
> >
> >     = Proposal =
> >
> >     Heron is a real-time stream processing engine built for high
> > performance,
> >     ease of manageability, performance predictability and developer
> >     productivity[1]. We wish to develop a community around Heron to
> > increase
> >     contributions and see Heron thrive in an open forum.
> >
> >     = Background =
> >
> >     Heron provides the ability for developers to compose directed acyclic
> >     graphs (DAGs) of real-time query execution logic (i.e. a topology)
> and
> >     submit the topology to execute on a pluggable job scheduling system
> > (e.g.,
> >     Apache Aurora, YARN, Marathon, etc). Users can employ either the
> native
> >     Heron API or the Apache Storm API to develop the topology. Heron
> > supports
> >     the Storm API for ease of migration, but beyond that Heron’s
> > architecture
> >     differs considerably from Storm’s.
> >
> >     Users submit a topology to the scheduler using the Heron client,
> which
> > uses
> >     the Heron binary libraries to deploy all daemons required to run and
> > manage
> >     the topology. The topology therefore has no reliance on centrally
> > managed
> >     Heron services, only on a generic job scheduling system, which lends
> > itself
> >     well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN
> > (among
> >     others).
> >
> >     The scheduler runs each topology as a job consisting of multiple
> >     containers. One of the containers runs the topology master,
> > responsible for
> >     managing the topology. The remaining containers each runs a stream
> > manager
> >     responsible for data routing, a metrics manager that collects and
> > reports
> >     various metrics and a number of processes called Heron instances
> which
> > run
> >     the user-defined logic on the stream of tuples. Parallelism is
> > achieved via
> >     process-based isolation of Heron instances, which provides
> predictable
> >     performance while simplifying debugging. The containers are allocated
> > and
> >     managed by the scheduler framework based on resource availability of
> > nodes
> >     in the cluster. The metadata for the topology, such as the physical
> > plan
> >     and execution details, are stored in the pluggable Heron State
> Manager
> >     (e.g. Apache ZooKeeper).
> >
> >     = Rationale =
> >
> >     Heron is a general-purpose, modular and extensible platform that can
> be
> >     leveraged to support common, real-time analytics use cases. There is
> an
> >     increasing demand for open-source, scalable real-time analytics
> > systems. We
> >     believe that Heron can be leveraged by other organizations to build
> >     streaming applications that can benefit from its robustness, high
> >     performance, adaptability to cloud environments and ease of use.
> > Moreover,
> >     we hope that open-sourcing Heron will help to further evolve the
> > technology
> >     as the project attracts contributors with diverse backgrounds and
> > areas of
> >     expertise.
> >
> >     We believe the Apache foundation is a great fit as the long-term home
> > for
> >     Heron, as it provides an established process for community-driven
> >     development and decision making by consensus. This is exactly the
> > model we
> >     want for future Heron development.
> >
> >     = Initial Goals =
> >
> >      * Move the existing codebase, website, documentation, and mailing
> > lists to
> >     Apache-hosted infrastructure.
> >      * Integrate with the Apache development process.
> >      * Ensure all dependencies are compliant with Apache License version
> > 2.0.
> >      * Incrementally develop and release per Apache guidelines.
> >
> >     = Current Status =
> >
> >     Heron is a stable project used in production at Twitter since 2014
> and
> > open
> >     sourced under the ASL v2 license in 2016. The Heron source code is
> >     currently hosted at github.com (https://github.com/twitter/heron),
> > which
> >     will seed the Apache git repository.
> >
> >     = Meritocracy =
> >
> >     By submitting this incubator proposal, we’re expressing our intent to
> > build
> >     a diverse developer community around Heron that will conduct itself
> >     according to The Apache Way and use a meritocratic means of building
> > it's
> >     committer base. Several companies and universities have already
> > expressed
> >     interest in and contributed to Heron. Our goal is to grow the Heron
> >     community by encouraging open communication, contribution and
> > participation
> >     of all types, and ensuring that contributors are recognized
> > appropriately.
> >
> >     = Community =
> >
> >     Heron is currently being used by Twitter, Google, Machine Zone and
> >     ndustrial.io and has received significant contributions by Microsoft
> > and
> >     Streamlio. By bringing Heron into the Apache ecosystem, we believe we
> > can
> >     attract even more developers who are interested in creating real-time
> >     systems to build the project's contributor base.
> >
> >     == Core Developers ==
> >
> >     Current core developers are engineers from Twitter, Google, Microsoft
> > and
> >     Streamlio.
> >
> >     == Alignment ==
> >
> >     Heron utilizes a number of Apache technologies. Heron leverages
> Apache
> >     ZooKeeper for coordination and has scheduler implementations to
> > integrate
> >     with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache
> > REEF)
> >     as well as spout implementations to integrate with Apache Kafka and
> > metrics
> >     implementations to integrate with Scribe. Heron also implements the
> > Apache
> >     Storm user-level API, which allows topologies written against Storm
> to
> > run
> >     in Heron. We believe that having Heron at Apache will help further
> the
> >     growth of the streaming compute community, as well as encourage
> > cooperation
> >     and developer cross pollination with other Apache projects.
> >
> >     = Known Risks =
> >
> >     == Orphaned Products ==
> >
> >     The risk of the Heron project being abandoned is minimal. It is used
> in
> >     production at Twitter and Google and other companies are evaluating
> or
> >     adopting it for production use.
> >
> >     == Inexperience with Open Source ==
> >
> >     All of the core contributors to the project have considerable
> > experience
> >     with open source software development. Bill Graham[2], Ashvin
> > Agrawal[3]
> >     and Supun Kamburugamuve[4], committers on the project, are PMCs on
> > other
> >     Apache projects and Bill and Ashvin have gone through the Apache
> > incubator
> >     process. Twitter has already donated numerous projects to the ASF
> > (e.g.,
> >     Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be
> > mentored
> >     by experienced ASF members that can help with any roadblocks.
> >
> >     == Homogenous Developers ==
> >
> >     Initial committers come from 5 separate organizations. Our intention
> is
> >     increase the diversity of contributing developers and their
> > affiliations.
> >     To date github contributions have come from approximately 50
> > contributors
> >     from outside the Twitter team.
> >
> >     == Reliance on Salaried Developers ==
> >
> >     It is expected that Heron development will occur on both salaried
> time
> > and
> >     on volunteer time. The majority of initial committers are paid by
> their
> >     employers to contribute to this project. We are committed to
> recruiting
> >     additional committers from other organizations as well as
> non-salaried
> >     committers to join project.
> >
> >     == Relationships with Other Apache Products ==
> >
> >     As mentioned in the Alignment section, Heron implements the Apache
> > Storm
> >     API and integrates with multiple Apache schedulers (Apache Mesos,
> > Apache
> >     Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and
> Apache
> >     Thrift.
> >
> >     == An Excessive Fascination with the Apache Brand ==
> >
> >     Heron's popularity is growing in the streaming compute space and we
> are
> >     long time supporters of the Apache brand. This proposal is not for
> the
> >     purpose of generating publicity through. Rather, the primary benefits
> > to
> >     joining Apache are those of community building and open decision
> making
> >     outlined in the Rationale section.
> >
> >     == Documentation ==
> >
> >     This proposal exists online as
> >     http://wiki.apache.org/incubator/HeronProposal. Extensive
> > documentation can
> >     be found on github at https://twitter.github.io/heron and the source
> > code
> >     is well documented.
> >
> >     == Source and Intellectual Property Submission Plan ==
> >
> >     The Heron codebase is currently hosted on Github:
> >     https://github.com/twitter/heron. During incubation, the codebase
> > will be
> >     migrated to Apache infrastructure. The source code is already ASF 2.0
> >     licensed.
> >
> >     == External Dependencies ==
> >
> >     All external libraries have ASF 2.0 compatible licenses except for
> > pylint.
> >     The pylint library is GPL licensed, but is only used for pre-build
> > Python
> >     style checks and is neither bundled with, nor relied upon by, the
> Heron
> >     source or binary release artifacts.
> >
> >     == Cryptography ==
> >
> >     Heron does not use any cryptography libraries.
> >
> >     = Required Resources =
> >
> >     == Mailing lists ==
> >
> >      * private@heron.incubator.apache.org (with moderated subscriptions)
> >      * dev@heron.incubator.apache.org
> >      * commits@heron.incubator.apache.org
> >      * user@heron.incubator.apache.org
> >
> >     == Subversion Directory ==
> >
> >     Git is the preferred source control system: git://
> git.apache.org/heron
> >
> >     == Issue Tracking ==
> >
> >     JIRA: Heron (HERON)
> >
> >     == Initial Committers ==
> >
> >      * Andrew Jorgensen (andrew at andrewjorgensen dot com)
> >      * Ashvin Agrawal (ashvin at apache dot org)*
> >      * Avrilia Floratou (avrilia dot floratou at gmail dot com)
> >      * Bill Graham (billgraham at apache dot org)*
> >      * Brian Hatfield (bmhatfield at gmail dot com)
> >      * Chris Kellogg (cckellogg at gmail dot com)
> >      * Huijun Wu (huijun dot wu dot 2010 at gmail dot com)
> >      * Karthik Ramasamy (karthik at gmail dot com)
> >      * Maosong Fu (maosongfu at gmail dot com)
> >      * Neng Lu(freeneng at gmail dot com)
> >      * Runhang Li (obj dot runhang at gmail dot com)
> >      * Sanjeev Kulkarni (sanjeevrk at gmail dot com)
> >      * Supun Kamburugamuve (supun at apache dot org)*
> >      * Thomas Sun (tom dot ssf at gmail dot com)
> >      * Yaliang Wang (yaliang dot w dot wang at ieee dot org)
> >
> >     == Affiliations ==
> >
> >      * Andrew Jorgensen (Google)
> >      * Ashvin Agrawal (Microsoft)
> >      * Avrilia Floratou (Microsoft)
> >      * Bill Graham (Twitter)
> >      * Brian Hatfield (Google)
> >      * Chris Kellogg (Twitter)
> >      * Huijun Wu (Twitter)
> >      * Karthik Ramasamy (Streamlio)
> >      * Maosong Fu (Twitter)
> >      * Neng Lu (Twitter)
> >      * Runhang Li (Twitter)
> >      * Sanjeev Kulkarni (Streamlio)
> >      * Supun Kamburugamuve (Indiana University)
> >      * Thomas Sun (Twitter)
> >      * Yaliang Wang (Twitter)
> >
> >     = Sponsors =
> >
> >     == Champion ==
> >
> >      * Julien Le Dem (julien at apache dot org)
> >
> >     == Nominated Mentors ==
> >
> >      * Jake Farrell (jfarrell at apache dot org)
> >      * Jacques Nadeau (jacques at apache dot org)
> >      * Julien Le Dem (julien at apache dot org)
> >      * P. Taylor Goetz (ptgoetz at apache dot org)
> >
> >     == Sponsoring Entity ==
> >
> >     The Apache Incubator
> >
> >     == Footnotes ==
> >
> >      * 1 - Papers detailing Heron are available at
> >     http://dl.acm.org/citation.cfm?id=2742788 and
> >     http://sites.computer.org/debull/A15dec/p15.pdf.
> >      * 2 - http://home.apache.org/phonebook.html?uid=billgraham
> >      * 3 - http://home.apache.org/phonebook.html?uid=ashvin
> >      * 4 - http://home.apache.org/phonebook.html?uid=supun
> >
> >
> >
>

Re: [VOTE] Heron to enter Apache Incubator

Posted by William Markito Oliveira <ma...@apache.org>.
+1

On Fri, Jun 16, 2017 at 3:45 PM, Debo Dutta (dedutta) <de...@cisco.com>
wrote:

> +1
>
> On 6/16/17, 1:41 PM, "Bill Graham" <bi...@gmail.com> wrote:
>
>     Hi,
>
>     Based on the discussion on the incubator mailing list[1] I would like
> to
>     call a vote to add Heron to the Apache Incubator.
>
>     The full proposal is available below, and is also available on the
> Apache
>     Incubator wiki at:
>         https://wiki.apache.org/incubator/HeronProposal
>
>     Please vote:
>       [ ] +1, bring Heron into Incubator
>       [ ] -1, do not bring Heron into Incubator, because...
>
>     The vote will open for 7 days until Friday June 23 at 14:00 PT.
>
>     Thank you
>
>     1 -
>     https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7
> 786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E
>
>
>
>     = Heron Proposal =
>
>     = Abstract =
>     Heron is a real-time, distributed, fault-tolerant stream processing
> engine
>     initially developed by Twitter.
>
>     = Proposal =
>
>     Heron is a real-time stream processing engine built for high
> performance,
>     ease of manageability, performance predictability and developer
>     productivity[1]. We wish to develop a community around Heron to
> increase
>     contributions and see Heron thrive in an open forum.
>
>     = Background =
>
>     Heron provides the ability for developers to compose directed acyclic
>     graphs (DAGs) of real-time query execution logic (i.e. a topology) and
>     submit the topology to execute on a pluggable job scheduling system
> (e.g.,
>     Apache Aurora, YARN, Marathon, etc). Users can employ either the native
>     Heron API or the Apache Storm API to develop the topology. Heron
> supports
>     the Storm API for ease of migration, but beyond that Heron’s
> architecture
>     differs considerably from Storm’s.
>
>     Users submit a topology to the scheduler using the Heron client, which
> uses
>     the Heron binary libraries to deploy all daemons required to run and
> manage
>     the topology. The topology therefore has no reliance on centrally
> managed
>     Heron services, only on a generic job scheduling system, which lends
> itself
>     well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN
> (among
>     others).
>
>     The scheduler runs each topology as a job consisting of multiple
>     containers. One of the containers runs the topology master,
> responsible for
>     managing the topology. The remaining containers each runs a stream
> manager
>     responsible for data routing, a metrics manager that collects and
> reports
>     various metrics and a number of processes called Heron instances which
> run
>     the user-defined logic on the stream of tuples. Parallelism is
> achieved via
>     process-based isolation of Heron instances, which provides predictable
>     performance while simplifying debugging. The containers are allocated
> and
>     managed by the scheduler framework based on resource availability of
> nodes
>     in the cluster. The metadata for the topology, such as the physical
> plan
>     and execution details, are stored in the pluggable Heron State Manager
>     (e.g. Apache ZooKeeper).
>
>     = Rationale =
>
>     Heron is a general-purpose, modular and extensible platform that can be
>     leveraged to support common, real-time analytics use cases. There is an
>     increasing demand for open-source, scalable real-time analytics
> systems. We
>     believe that Heron can be leveraged by other organizations to build
>     streaming applications that can benefit from its robustness, high
>     performance, adaptability to cloud environments and ease of use.
> Moreover,
>     we hope that open-sourcing Heron will help to further evolve the
> technology
>     as the project attracts contributors with diverse backgrounds and
> areas of
>     expertise.
>
>     We believe the Apache foundation is a great fit as the long-term home
> for
>     Heron, as it provides an established process for community-driven
>     development and decision making by consensus. This is exactly the
> model we
>     want for future Heron development.
>
>     = Initial Goals =
>
>      * Move the existing codebase, website, documentation, and mailing
> lists to
>     Apache-hosted infrastructure.
>      * Integrate with the Apache development process.
>      * Ensure all dependencies are compliant with Apache License version
> 2.0.
>      * Incrementally develop and release per Apache guidelines.
>
>     = Current Status =
>
>     Heron is a stable project used in production at Twitter since 2014 and
> open
>     sourced under the ASL v2 license in 2016. The Heron source code is
>     currently hosted at github.com (https://github.com/twitter/heron),
> which
>     will seed the Apache git repository.
>
>     = Meritocracy =
>
>     By submitting this incubator proposal, we’re expressing our intent to
> build
>     a diverse developer community around Heron that will conduct itself
>     according to The Apache Way and use a meritocratic means of building
> it's
>     committer base. Several companies and universities have already
> expressed
>     interest in and contributed to Heron. Our goal is to grow the Heron
>     community by encouraging open communication, contribution and
> participation
>     of all types, and ensuring that contributors are recognized
> appropriately.
>
>     = Community =
>
>     Heron is currently being used by Twitter, Google, Machine Zone and
>     ndustrial.io and has received significant contributions by Microsoft
> and
>     Streamlio. By bringing Heron into the Apache ecosystem, we believe we
> can
>     attract even more developers who are interested in creating real-time
>     systems to build the project's contributor base.
>
>     == Core Developers ==
>
>     Current core developers are engineers from Twitter, Google, Microsoft
> and
>     Streamlio.
>
>     == Alignment ==
>
>     Heron utilizes a number of Apache technologies. Heron leverages Apache
>     ZooKeeper for coordination and has scheduler implementations to
> integrate
>     with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache
> REEF)
>     as well as spout implementations to integrate with Apache Kafka and
> metrics
>     implementations to integrate with Scribe. Heron also implements the
> Apache
>     Storm user-level API, which allows topologies written against Storm to
> run
>     in Heron. We believe that having Heron at Apache will help further the
>     growth of the streaming compute community, as well as encourage
> cooperation
>     and developer cross pollination with other Apache projects.
>
>     = Known Risks =
>
>     == Orphaned Products ==
>
>     The risk of the Heron project being abandoned is minimal. It is used in
>     production at Twitter and Google and other companies are evaluating or
>     adopting it for production use.
>
>     == Inexperience with Open Source ==
>
>     All of the core contributors to the project have considerable
> experience
>     with open source software development. Bill Graham[2], Ashvin
> Agrawal[3]
>     and Supun Kamburugamuve[4], committers on the project, are PMCs on
> other
>     Apache projects and Bill and Ashvin have gone through the Apache
> incubator
>     process. Twitter has already donated numerous projects to the ASF
> (e.g.,
>     Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be
> mentored
>     by experienced ASF members that can help with any roadblocks.
>
>     == Homogenous Developers ==
>
>     Initial committers come from 5 separate organizations. Our intention is
>     increase the diversity of contributing developers and their
> affiliations.
>     To date github contributions have come from approximately 50
> contributors
>     from outside the Twitter team.
>
>     == Reliance on Salaried Developers ==
>
>     It is expected that Heron development will occur on both salaried time
> and
>     on volunteer time. The majority of initial committers are paid by their
>     employers to contribute to this project. We are committed to recruiting
>     additional committers from other organizations as well as non-salaried
>     committers to join project.
>
>     == Relationships with Other Apache Products ==
>
>     As mentioned in the Alignment section, Heron implements the Apache
> Storm
>     API and integrates with multiple Apache schedulers (Apache Mesos,
> Apache
>     Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache
>     Thrift.
>
>     == An Excessive Fascination with the Apache Brand ==
>
>     Heron's popularity is growing in the streaming compute space and we are
>     long time supporters of the Apache brand. This proposal is not for the
>     purpose of generating publicity through. Rather, the primary benefits
> to
>     joining Apache are those of community building and open decision making
>     outlined in the Rationale section.
>
>     == Documentation ==
>
>     This proposal exists online as
>     http://wiki.apache.org/incubator/HeronProposal. Extensive
> documentation can
>     be found on github at https://twitter.github.io/heron and the source
> code
>     is well documented.
>
>     == Source and Intellectual Property Submission Plan ==
>
>     The Heron codebase is currently hosted on Github:
>     https://github.com/twitter/heron. During incubation, the codebase
> will be
>     migrated to Apache infrastructure. The source code is already ASF 2.0
>     licensed.
>
>     == External Dependencies ==
>
>     All external libraries have ASF 2.0 compatible licenses except for
> pylint.
>     The pylint library is GPL licensed, but is only used for pre-build
> Python
>     style checks and is neither bundled with, nor relied upon by, the Heron
>     source or binary release artifacts.
>
>     == Cryptography ==
>
>     Heron does not use any cryptography libraries.
>
>     = Required Resources =
>
>     == Mailing lists ==
>
>      * private@heron.incubator.apache.org (with moderated subscriptions)
>      * dev@heron.incubator.apache.org
>      * commits@heron.incubator.apache.org
>      * user@heron.incubator.apache.org
>
>     == Subversion Directory ==
>
>     Git is the preferred source control system: git://git.apache.org/heron
>
>     == Issue Tracking ==
>
>     JIRA: Heron (HERON)
>
>     == Initial Committers ==
>
>      * Andrew Jorgensen (andrew at andrewjorgensen dot com)
>      * Ashvin Agrawal (ashvin at apache dot org)*
>      * Avrilia Floratou (avrilia dot floratou at gmail dot com)
>      * Bill Graham (billgraham at apache dot org)*
>      * Brian Hatfield (bmhatfield at gmail dot com)
>      * Chris Kellogg (cckellogg at gmail dot com)
>      * Huijun Wu (huijun dot wu dot 2010 at gmail dot com)
>      * Karthik Ramasamy (karthik at gmail dot com)
>      * Maosong Fu (maosongfu at gmail dot com)
>      * Neng Lu(freeneng at gmail dot com)
>      * Runhang Li (obj dot runhang at gmail dot com)
>      * Sanjeev Kulkarni (sanjeevrk at gmail dot com)
>      * Supun Kamburugamuve (supun at apache dot org)*
>      * Thomas Sun (tom dot ssf at gmail dot com)
>      * Yaliang Wang (yaliang dot w dot wang at ieee dot org)
>
>     == Affiliations ==
>
>      * Andrew Jorgensen (Google)
>      * Ashvin Agrawal (Microsoft)
>      * Avrilia Floratou (Microsoft)
>      * Bill Graham (Twitter)
>      * Brian Hatfield (Google)
>      * Chris Kellogg (Twitter)
>      * Huijun Wu (Twitter)
>      * Karthik Ramasamy (Streamlio)
>      * Maosong Fu (Twitter)
>      * Neng Lu (Twitter)
>      * Runhang Li (Twitter)
>      * Sanjeev Kulkarni (Streamlio)
>      * Supun Kamburugamuve (Indiana University)
>      * Thomas Sun (Twitter)
>      * Yaliang Wang (Twitter)
>
>     = Sponsors =
>
>     == Champion ==
>
>      * Julien Le Dem (julien at apache dot org)
>
>     == Nominated Mentors ==
>
>      * Jake Farrell (jfarrell at apache dot org)
>      * Jacques Nadeau (jacques at apache dot org)
>      * Julien Le Dem (julien at apache dot org)
>      * P. Taylor Goetz (ptgoetz at apache dot org)
>
>     == Sponsoring Entity ==
>
>     The Apache Incubator
>
>     == Footnotes ==
>
>      * 1 - Papers detailing Heron are available at
>     http://dl.acm.org/citation.cfm?id=2742788 and
>     http://sites.computer.org/debull/A15dec/p15.pdf.
>      * 2 - http://home.apache.org/phonebook.html?uid=billgraham
>      * 3 - http://home.apache.org/phonebook.html?uid=ashvin
>      * 4 - http://home.apache.org/phonebook.html?uid=supun
>
>
>

Re: [VOTE] Heron to enter Apache Incubator

Posted by "Debo Dutta (dedutta)" <de...@cisco.com>.
+1

On 6/16/17, 1:41 PM, "Bill Graham" <bi...@gmail.com> wrote:

    Hi,
    
    Based on the discussion on the incubator mailing list[1] I would like to
    call a vote to add Heron to the Apache Incubator.
    
    The full proposal is available below, and is also available on the Apache
    Incubator wiki at:
        https://wiki.apache.org/incubator/HeronProposal
    
    Please vote:
      [ ] +1, bring Heron into Incubator
      [ ] -1, do not bring Heron into Incubator, because...
    
    The vote will open for 7 days until Friday June 23 at 14:00 PT.
    
    Thank you
    
    1 -
    https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E
    
    
    
    = Heron Proposal =
    
    = Abstract =
    Heron is a real-time, distributed, fault-tolerant stream processing engine
    initially developed by Twitter.
    
    = Proposal =
    
    Heron is a real-time stream processing engine built for high performance,
    ease of manageability, performance predictability and developer
    productivity[1]. We wish to develop a community around Heron to increase
    contributions and see Heron thrive in an open forum.
    
    = Background =
    
    Heron provides the ability for developers to compose directed acyclic
    graphs (DAGs) of real-time query execution logic (i.e. a topology) and
    submit the topology to execute on a pluggable job scheduling system (e.g.,
    Apache Aurora, YARN, Marathon, etc). Users can employ either the native
    Heron API or the Apache Storm API to develop the topology. Heron supports
    the Storm API for ease of migration, but beyond that Heron’s architecture
    differs considerably from Storm’s.
    
    Users submit a topology to the scheduler using the Heron client, which uses
    the Heron binary libraries to deploy all daemons required to run and manage
    the topology. The topology therefore has no reliance on centrally managed
    Heron services, only on a generic job scheduling system, which lends itself
    well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN (among
    others).
    
    The scheduler runs each topology as a job consisting of multiple
    containers. One of the containers runs the topology master, responsible for
    managing the topology. The remaining containers each runs a stream manager
    responsible for data routing, a metrics manager that collects and reports
    various metrics and a number of processes called Heron instances which run
    the user-defined logic on the stream of tuples. Parallelism is achieved via
    process-based isolation of Heron instances, which provides predictable
    performance while simplifying debugging. The containers are allocated and
    managed by the scheduler framework based on resource availability of nodes
    in the cluster. The metadata for the topology, such as the physical plan
    and execution details, are stored in the pluggable Heron State Manager
    (e.g. Apache ZooKeeper).
    
    = Rationale =
    
    Heron is a general-purpose, modular and extensible platform that can be
    leveraged to support common, real-time analytics use cases. There is an
    increasing demand for open-source, scalable real-time analytics systems. We
    believe that Heron can be leveraged by other organizations to build
    streaming applications that can benefit from its robustness, high
    performance, adaptability to cloud environments and ease of use. Moreover,
    we hope that open-sourcing Heron will help to further evolve the technology
    as the project attracts contributors with diverse backgrounds and areas of
    expertise.
    
    We believe the Apache foundation is a great fit as the long-term home for
    Heron, as it provides an established process for community-driven
    development and decision making by consensus. This is exactly the model we
    want for future Heron development.
    
    = Initial Goals =
    
     * Move the existing codebase, website, documentation, and mailing lists to
    Apache-hosted infrastructure.
     * Integrate with the Apache development process.
     * Ensure all dependencies are compliant with Apache License version 2.0.
     * Incrementally develop and release per Apache guidelines.
    
    = Current Status =
    
    Heron is a stable project used in production at Twitter since 2014 and open
    sourced under the ASL v2 license in 2016. The Heron source code is
    currently hosted at github.com (https://github.com/twitter/heron), which
    will seed the Apache git repository.
    
    = Meritocracy =
    
    By submitting this incubator proposal, we’re expressing our intent to build
    a diverse developer community around Heron that will conduct itself
    according to The Apache Way and use a meritocratic means of building it's
    committer base. Several companies and universities have already expressed
    interest in and contributed to Heron. Our goal is to grow the Heron
    community by encouraging open communication, contribution and participation
    of all types, and ensuring that contributors are recognized appropriately.
    
    = Community =
    
    Heron is currently being used by Twitter, Google, Machine Zone and
    ndustrial.io and has received significant contributions by Microsoft and
    Streamlio. By bringing Heron into the Apache ecosystem, we believe we can
    attract even more developers who are interested in creating real-time
    systems to build the project's contributor base.
    
    == Core Developers ==
    
    Current core developers are engineers from Twitter, Google, Microsoft and
    Streamlio.
    
    == Alignment ==
    
    Heron utilizes a number of Apache technologies. Heron leverages Apache
    ZooKeeper for coordination and has scheduler implementations to integrate
    with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache REEF)
    as well as spout implementations to integrate with Apache Kafka and metrics
    implementations to integrate with Scribe. Heron also implements the Apache
    Storm user-level API, which allows topologies written against Storm to run
    in Heron. We believe that having Heron at Apache will help further the
    growth of the streaming compute community, as well as encourage cooperation
    and developer cross pollination with other Apache projects.
    
    = Known Risks =
    
    == Orphaned Products ==
    
    The risk of the Heron project being abandoned is minimal. It is used in
    production at Twitter and Google and other companies are evaluating or
    adopting it for production use.
    
    == Inexperience with Open Source ==
    
    All of the core contributors to the project have considerable experience
    with open source software development. Bill Graham[2], Ashvin Agrawal[3]
    and Supun Kamburugamuve[4], committers on the project, are PMCs on other
    Apache projects and Bill and Ashvin have gone through the Apache incubator
    process. Twitter has already donated numerous projects to the ASF (e.g.,
    Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be mentored
    by experienced ASF members that can help with any roadblocks.
    
    == Homogenous Developers ==
    
    Initial committers come from 5 separate organizations. Our intention is
    increase the diversity of contributing developers and their affiliations.
    To date github contributions have come from approximately 50 contributors
    from outside the Twitter team.
    
    == Reliance on Salaried Developers ==
    
    It is expected that Heron development will occur on both salaried time and
    on volunteer time. The majority of initial committers are paid by their
    employers to contribute to this project. We are committed to recruiting
    additional committers from other organizations as well as non-salaried
    committers to join project.
    
    == Relationships with Other Apache Products ==
    
    As mentioned in the Alignment section, Heron implements the Apache Storm
    API and integrates with multiple Apache schedulers (Apache Mesos, Apache
    Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache
    Thrift.
    
    == An Excessive Fascination with the Apache Brand ==
    
    Heron's popularity is growing in the streaming compute space and we are
    long time supporters of the Apache brand. This proposal is not for the
    purpose of generating publicity through. Rather, the primary benefits to
    joining Apache are those of community building and open decision making
    outlined in the Rationale section.
    
    == Documentation ==
    
    This proposal exists online as
    http://wiki.apache.org/incubator/HeronProposal. Extensive documentation can
    be found on github at https://twitter.github.io/heron and the source code
    is well documented.
    
    == Source and Intellectual Property Submission Plan ==
    
    The Heron codebase is currently hosted on Github:
    https://github.com/twitter/heron. During incubation, the codebase will be
    migrated to Apache infrastructure. The source code is already ASF 2.0
    licensed.
    
    == External Dependencies ==
    
    All external libraries have ASF 2.0 compatible licenses except for pylint.
    The pylint library is GPL licensed, but is only used for pre-build Python
    style checks and is neither bundled with, nor relied upon by, the Heron
    source or binary release artifacts.
    
    == Cryptography ==
    
    Heron does not use any cryptography libraries.
    
    = Required Resources =
    
    == Mailing lists ==
    
     * private@heron.incubator.apache.org (with moderated subscriptions)
     * dev@heron.incubator.apache.org
     * commits@heron.incubator.apache.org
     * user@heron.incubator.apache.org
    
    == Subversion Directory ==
    
    Git is the preferred source control system: git://git.apache.org/heron
    
    == Issue Tracking ==
    
    JIRA: Heron (HERON)
    
    == Initial Committers ==
    
     * Andrew Jorgensen (andrew at andrewjorgensen dot com)
     * Ashvin Agrawal (ashvin at apache dot org)*
     * Avrilia Floratou (avrilia dot floratou at gmail dot com)
     * Bill Graham (billgraham at apache dot org)*
     * Brian Hatfield (bmhatfield at gmail dot com)
     * Chris Kellogg (cckellogg at gmail dot com)
     * Huijun Wu (huijun dot wu dot 2010 at gmail dot com)
     * Karthik Ramasamy (karthik at gmail dot com)
     * Maosong Fu (maosongfu at gmail dot com)
     * Neng Lu(freeneng at gmail dot com)
     * Runhang Li (obj dot runhang at gmail dot com)
     * Sanjeev Kulkarni (sanjeevrk at gmail dot com)
     * Supun Kamburugamuve (supun at apache dot org)*
     * Thomas Sun (tom dot ssf at gmail dot com)
     * Yaliang Wang (yaliang dot w dot wang at ieee dot org)
    
    == Affiliations ==
    
     * Andrew Jorgensen (Google)
     * Ashvin Agrawal (Microsoft)
     * Avrilia Floratou (Microsoft)
     * Bill Graham (Twitter)
     * Brian Hatfield (Google)
     * Chris Kellogg (Twitter)
     * Huijun Wu (Twitter)
     * Karthik Ramasamy (Streamlio)
     * Maosong Fu (Twitter)
     * Neng Lu (Twitter)
     * Runhang Li (Twitter)
     * Sanjeev Kulkarni (Streamlio)
     * Supun Kamburugamuve (Indiana University)
     * Thomas Sun (Twitter)
     * Yaliang Wang (Twitter)
    
    = Sponsors =
    
    == Champion ==
    
     * Julien Le Dem (julien at apache dot org)
    
    == Nominated Mentors ==
    
     * Jake Farrell (jfarrell at apache dot org)
     * Jacques Nadeau (jacques at apache dot org)
     * Julien Le Dem (julien at apache dot org)
     * P. Taylor Goetz (ptgoetz at apache dot org)
    
    == Sponsoring Entity ==
    
    The Apache Incubator
    
    == Footnotes ==
    
     * 1 - Papers detailing Heron are available at
    http://dl.acm.org/citation.cfm?id=2742788 and
    http://sites.computer.org/debull/A15dec/p15.pdf.
     * 2 - http://home.apache.org/phonebook.html?uid=billgraham
     * 3 - http://home.apache.org/phonebook.html?uid=ashvin
     * 4 - http://home.apache.org/phonebook.html?uid=supun

Re: [VOTE] Heron to enter Apache Incubator

Posted by Pierre Smits <pi...@gmail.com>.
+1 (non-privileged contributor)

Best regards,

Pierre Smits

ORRTIZ.COM <http://www.orrtiz.com>
OFBiz based solutions & services

OFBiz Extensions Marketplace
http://oem.ofbizci.net/oci-2/

On Sun, Jun 18, 2017 at 12:11 AM, Chris Douglas <ch...@gmail.com>
wrote:

> +1 (binding) -C
>
> On Fri, Jun 16, 2017 at 1:41 PM, Bill Graham <bi...@gmail.com> wrote:
> > Hi,
> >
> > Based on the discussion on the incubator mailing list[1] I would like to
> > call a vote to add Heron to the Apache Incubator.
> >
> > The full proposal is available below, and is also available on the Apache
> > Incubator wiki at:
> >     https://wiki.apache.org/incubator/HeronProposal
> >
> > Please vote:
> >   [ ] +1, bring Heron into Incubator
> >   [ ] -1, do not bring Heron into Incubator, because...
> >
> > The vote will open for 7 days until Friday June 23 at 14:00 PT.
> >
> > Thank you
> >
> > 1 -
> > https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7
> 786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E
> >
> >
> >
> > = Heron Proposal =
> >
> > = Abstract =
> > Heron is a real-time, distributed, fault-tolerant stream processing
> engine
> > initially developed by Twitter.
> >
> > = Proposal =
> >
> > Heron is a real-time stream processing engine built for high performance,
> > ease of manageability, performance predictability and developer
> > productivity[1]. We wish to develop a community around Heron to increase
> > contributions and see Heron thrive in an open forum.
> >
> > = Background =
> >
> > Heron provides the ability for developers to compose directed acyclic
> > graphs (DAGs) of real-time query execution logic (i.e. a topology) and
> > submit the topology to execute on a pluggable job scheduling system
> (e.g.,
> > Apache Aurora, YARN, Marathon, etc). Users can employ either the native
> > Heron API or the Apache Storm API to develop the topology. Heron supports
> > the Storm API for ease of migration, but beyond that Heron’s architecture
> > differs considerably from Storm’s.
> >
> > Users submit a topology to the scheduler using the Heron client, which
> uses
> > the Heron binary libraries to deploy all daemons required to run and
> manage
> > the topology. The topology therefore has no reliance on centrally managed
> > Heron services, only on a generic job scheduling system, which lends
> itself
> > well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN (among
> > others).
> >
> > The scheduler runs each topology as a job consisting of multiple
> > containers. One of the containers runs the topology master, responsible
> for
> > managing the topology. The remaining containers each runs a stream
> manager
> > responsible for data routing, a metrics manager that collects and reports
> > various metrics and a number of processes called Heron instances which
> run
> > the user-defined logic on the stream of tuples. Parallelism is achieved
> via
> > process-based isolation of Heron instances, which provides predictable
> > performance while simplifying debugging. The containers are allocated and
> > managed by the scheduler framework based on resource availability of
> nodes
> > in the cluster. The metadata for the topology, such as the physical plan
> > and execution details, are stored in the pluggable Heron State Manager
> > (e.g. Apache ZooKeeper).
> >
> > = Rationale =
> >
> > Heron is a general-purpose, modular and extensible platform that can be
> > leveraged to support common, real-time analytics use cases. There is an
> > increasing demand for open-source, scalable real-time analytics systems.
> We
> > believe that Heron can be leveraged by other organizations to build
> > streaming applications that can benefit from its robustness, high
> > performance, adaptability to cloud environments and ease of use.
> Moreover,
> > we hope that open-sourcing Heron will help to further evolve the
> technology
> > as the project attracts contributors with diverse backgrounds and areas
> of
> > expertise.
> >
> > We believe the Apache foundation is a great fit as the long-term home for
> > Heron, as it provides an established process for community-driven
> > development and decision making by consensus. This is exactly the model
> we
> > want for future Heron development.
> >
> > = Initial Goals =
> >
> >  * Move the existing codebase, website, documentation, and mailing lists
> to
> > Apache-hosted infrastructure.
> >  * Integrate with the Apache development process.
> >  * Ensure all dependencies are compliant with Apache License version 2.0.
> >  * Incrementally develop and release per Apache guidelines.
> >
> > = Current Status =
> >
> > Heron is a stable project used in production at Twitter since 2014 and
> open
> > sourced under the ASL v2 license in 2016. The Heron source code is
> > currently hosted at github.com (https://github.com/twitter/heron), which
> > will seed the Apache git repository.
> >
> > = Meritocracy =
> >
> > By submitting this incubator proposal, we’re expressing our intent to
> build
> > a diverse developer community around Heron that will conduct itself
> > according to The Apache Way and use a meritocratic means of building it's
> > committer base. Several companies and universities have already expressed
> > interest in and contributed to Heron. Our goal is to grow the Heron
> > community by encouraging open communication, contribution and
> participation
> > of all types, and ensuring that contributors are recognized
> appropriately.
> >
> > = Community =
> >
> > Heron is currently being used by Twitter, Google, Machine Zone and
> > ndustrial.io and has received significant contributions by Microsoft and
> > Streamlio. By bringing Heron into the Apache ecosystem, we believe we can
> > attract even more developers who are interested in creating real-time
> > systems to build the project's contributor base.
> >
> > == Core Developers ==
> >
> > Current core developers are engineers from Twitter, Google, Microsoft and
> > Streamlio.
> >
> > == Alignment ==
> >
> > Heron utilizes a number of Apache technologies. Heron leverages Apache
> > ZooKeeper for coordination and has scheduler implementations to integrate
> > with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache
> REEF)
> > as well as spout implementations to integrate with Apache Kafka and
> metrics
> > implementations to integrate with Scribe. Heron also implements the
> Apache
> > Storm user-level API, which allows topologies written against Storm to
> run
> > in Heron. We believe that having Heron at Apache will help further the
> > growth of the streaming compute community, as well as encourage
> cooperation
> > and developer cross pollination with other Apache projects.
> >
> > = Known Risks =
> >
> > == Orphaned Products ==
> >
> > The risk of the Heron project being abandoned is minimal. It is used in
> > production at Twitter and Google and other companies are evaluating or
> > adopting it for production use.
> >
> > == Inexperience with Open Source ==
> >
> > All of the core contributors to the project have considerable experience
> > with open source software development. Bill Graham[2], Ashvin Agrawal[3]
> > and Supun Kamburugamuve[4], committers on the project, are PMCs on other
> > Apache projects and Bill and Ashvin have gone through the Apache
> incubator
> > process. Twitter has already donated numerous projects to the ASF (e.g.,
> > Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be mentored
> > by experienced ASF members that can help with any roadblocks.
> >
> > == Homogenous Developers ==
> >
> > Initial committers come from 5 separate organizations. Our intention is
> > increase the diversity of contributing developers and their affiliations.
> > To date github contributions have come from approximately 50 contributors
> > from outside the Twitter team.
> >
> > == Reliance on Salaried Developers ==
> >
> > It is expected that Heron development will occur on both salaried time
> and
> > on volunteer time. The majority of initial committers are paid by their
> > employers to contribute to this project. We are committed to recruiting
> > additional committers from other organizations as well as non-salaried
> > committers to join project.
> >
> > == Relationships with Other Apache Products ==
> >
> > As mentioned in the Alignment section, Heron implements the Apache Storm
> > API and integrates with multiple Apache schedulers (Apache Mesos, Apache
> > Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache
> > Thrift.
> >
> > == An Excessive Fascination with the Apache Brand ==
> >
> > Heron's popularity is growing in the streaming compute space and we are
> > long time supporters of the Apache brand. This proposal is not for the
> > purpose of generating publicity through. Rather, the primary benefits to
> > joining Apache are those of community building and open decision making
> > outlined in the Rationale section.
> >
> > == Documentation ==
> >
> > This proposal exists online as
> > http://wiki.apache.org/incubator/HeronProposal. Extensive documentation
> can
> > be found on github at https://twitter.github.io/heron and the source
> code
> > is well documented.
> >
> > == Source and Intellectual Property Submission Plan ==
> >
> > The Heron codebase is currently hosted on Github:
> > https://github.com/twitter/heron. During incubation, the codebase will
> be
> > migrated to Apache infrastructure. The source code is already ASF 2.0
> > licensed.
> >
> > == External Dependencies ==
> >
> > All external libraries have ASF 2.0 compatible licenses except for
> pylint.
> > The pylint library is GPL licensed, but is only used for pre-build Python
> > style checks and is neither bundled with, nor relied upon by, the Heron
> > source or binary release artifacts.
> >
> > == Cryptography ==
> >
> > Heron does not use any cryptography libraries.
> >
> > = Required Resources =
> >
> > == Mailing lists ==
> >
> >  * private@heron.incubator.apache.org (with moderated subscriptions)
> >  * dev@heron.incubator.apache.org
> >  * commits@heron.incubator.apache.org
> >  * user@heron.incubator.apache.org
> >
> > == Subversion Directory ==
> >
> > Git is the preferred source control system: git://git.apache.org/heron
> >
> > == Issue Tracking ==
> >
> > JIRA: Heron (HERON)
> >
> > == Initial Committers ==
> >
> >  * Andrew Jorgensen (andrew at andrewjorgensen dot com)
> >  * Ashvin Agrawal (ashvin at apache dot org)*
> >  * Avrilia Floratou (avrilia dot floratou at gmail dot com)
> >  * Bill Graham (billgraham at apache dot org)*
> >  * Brian Hatfield (bmhatfield at gmail dot com)
> >  * Chris Kellogg (cckellogg at gmail dot com)
> >  * Huijun Wu (huijun dot wu dot 2010 at gmail dot com)
> >  * Karthik Ramasamy (karthik at gmail dot com)
> >  * Maosong Fu (maosongfu at gmail dot com)
> >  * Neng Lu(freeneng at gmail dot com)
> >  * Runhang Li (obj dot runhang at gmail dot com)
> >  * Sanjeev Kulkarni (sanjeevrk at gmail dot com)
> >  * Supun Kamburugamuve (supun at apache dot org)*
> >  * Thomas Sun (tom dot ssf at gmail dot com)
> >  * Yaliang Wang (yaliang dot w dot wang at ieee dot org)
> >
> > == Affiliations ==
> >
> >  * Andrew Jorgensen (Google)
> >  * Ashvin Agrawal (Microsoft)
> >  * Avrilia Floratou (Microsoft)
> >  * Bill Graham (Twitter)
> >  * Brian Hatfield (Google)
> >  * Chris Kellogg (Twitter)
> >  * Huijun Wu (Twitter)
> >  * Karthik Ramasamy (Streamlio)
> >  * Maosong Fu (Twitter)
> >  * Neng Lu (Twitter)
> >  * Runhang Li (Twitter)
> >  * Sanjeev Kulkarni (Streamlio)
> >  * Supun Kamburugamuve (Indiana University)
> >  * Thomas Sun (Twitter)
> >  * Yaliang Wang (Twitter)
> >
> > = Sponsors =
> >
> > == Champion ==
> >
> >  * Julien Le Dem (julien at apache dot org)
> >
> > == Nominated Mentors ==
> >
> >  * Jake Farrell (jfarrell at apache dot org)
> >  * Jacques Nadeau (jacques at apache dot org)
> >  * Julien Le Dem (julien at apache dot org)
> >  * P. Taylor Goetz (ptgoetz at apache dot org)
> >
> > == Sponsoring Entity ==
> >
> > The Apache Incubator
> >
> > == Footnotes ==
> >
> >  * 1 - Papers detailing Heron are available at
> > http://dl.acm.org/citation.cfm?id=2742788 and
> > http://sites.computer.org/debull/A15dec/p15.pdf.
> >  * 2 - http://home.apache.org/phonebook.html?uid=billgraham
> >  * 3 - http://home.apache.org/phonebook.html?uid=ashvin
> >  * 4 - http://home.apache.org/phonebook.html?uid=supun
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>

Re: [VOTE] Heron to enter Apache Incubator

Posted by Chris Douglas <ch...@gmail.com>.
+1 (binding) -C

On Fri, Jun 16, 2017 at 1:41 PM, Bill Graham <bi...@gmail.com> wrote:
> Hi,
>
> Based on the discussion on the incubator mailing list[1] I would like to
> call a vote to add Heron to the Apache Incubator.
>
> The full proposal is available below, and is also available on the Apache
> Incubator wiki at:
>     https://wiki.apache.org/incubator/HeronProposal
>
> Please vote:
>   [ ] +1, bring Heron into Incubator
>   [ ] -1, do not bring Heron into Incubator, because...
>
> The vote will open for 7 days until Friday June 23 at 14:00 PT.
>
> Thank you
>
> 1 -
> https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E
>
>
>
> = Heron Proposal =
>
> = Abstract =
> Heron is a real-time, distributed, fault-tolerant stream processing engine
> initially developed by Twitter.
>
> = Proposal =
>
> Heron is a real-time stream processing engine built for high performance,
> ease of manageability, performance predictability and developer
> productivity[1]. We wish to develop a community around Heron to increase
> contributions and see Heron thrive in an open forum.
>
> = Background =
>
> Heron provides the ability for developers to compose directed acyclic
> graphs (DAGs) of real-time query execution logic (i.e. a topology) and
> submit the topology to execute on a pluggable job scheduling system (e.g.,
> Apache Aurora, YARN, Marathon, etc). Users can employ either the native
> Heron API or the Apache Storm API to develop the topology. Heron supports
> the Storm API for ease of migration, but beyond that Heron’s architecture
> differs considerably from Storm’s.
>
> Users submit a topology to the scheduler using the Heron client, which uses
> the Heron binary libraries to deploy all daemons required to run and manage
> the topology. The topology therefore has no reliance on centrally managed
> Heron services, only on a generic job scheduling system, which lends itself
> well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN (among
> others).
>
> The scheduler runs each topology as a job consisting of multiple
> containers. One of the containers runs the topology master, responsible for
> managing the topology. The remaining containers each runs a stream manager
> responsible for data routing, a metrics manager that collects and reports
> various metrics and a number of processes called Heron instances which run
> the user-defined logic on the stream of tuples. Parallelism is achieved via
> process-based isolation of Heron instances, which provides predictable
> performance while simplifying debugging. The containers are allocated and
> managed by the scheduler framework based on resource availability of nodes
> in the cluster. The metadata for the topology, such as the physical plan
> and execution details, are stored in the pluggable Heron State Manager
> (e.g. Apache ZooKeeper).
>
> = Rationale =
>
> Heron is a general-purpose, modular and extensible platform that can be
> leveraged to support common, real-time analytics use cases. There is an
> increasing demand for open-source, scalable real-time analytics systems. We
> believe that Heron can be leveraged by other organizations to build
> streaming applications that can benefit from its robustness, high
> performance, adaptability to cloud environments and ease of use. Moreover,
> we hope that open-sourcing Heron will help to further evolve the technology
> as the project attracts contributors with diverse backgrounds and areas of
> expertise.
>
> We believe the Apache foundation is a great fit as the long-term home for
> Heron, as it provides an established process for community-driven
> development and decision making by consensus. This is exactly the model we
> want for future Heron development.
>
> = Initial Goals =
>
>  * Move the existing codebase, website, documentation, and mailing lists to
> Apache-hosted infrastructure.
>  * Integrate with the Apache development process.
>  * Ensure all dependencies are compliant with Apache License version 2.0.
>  * Incrementally develop and release per Apache guidelines.
>
> = Current Status =
>
> Heron is a stable project used in production at Twitter since 2014 and open
> sourced under the ASL v2 license in 2016. The Heron source code is
> currently hosted at github.com (https://github.com/twitter/heron), which
> will seed the Apache git repository.
>
> = Meritocracy =
>
> By submitting this incubator proposal, we’re expressing our intent to build
> a diverse developer community around Heron that will conduct itself
> according to The Apache Way and use a meritocratic means of building it's
> committer base. Several companies and universities have already expressed
> interest in and contributed to Heron. Our goal is to grow the Heron
> community by encouraging open communication, contribution and participation
> of all types, and ensuring that contributors are recognized appropriately.
>
> = Community =
>
> Heron is currently being used by Twitter, Google, Machine Zone and
> ndustrial.io and has received significant contributions by Microsoft and
> Streamlio. By bringing Heron into the Apache ecosystem, we believe we can
> attract even more developers who are interested in creating real-time
> systems to build the project's contributor base.
>
> == Core Developers ==
>
> Current core developers are engineers from Twitter, Google, Microsoft and
> Streamlio.
>
> == Alignment ==
>
> Heron utilizes a number of Apache technologies. Heron leverages Apache
> ZooKeeper for coordination and has scheduler implementations to integrate
> with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache REEF)
> as well as spout implementations to integrate with Apache Kafka and metrics
> implementations to integrate with Scribe. Heron also implements the Apache
> Storm user-level API, which allows topologies written against Storm to run
> in Heron. We believe that having Heron at Apache will help further the
> growth of the streaming compute community, as well as encourage cooperation
> and developer cross pollination with other Apache projects.
>
> = Known Risks =
>
> == Orphaned Products ==
>
> The risk of the Heron project being abandoned is minimal. It is used in
> production at Twitter and Google and other companies are evaluating or
> adopting it for production use.
>
> == Inexperience with Open Source ==
>
> All of the core contributors to the project have considerable experience
> with open source software development. Bill Graham[2], Ashvin Agrawal[3]
> and Supun Kamburugamuve[4], committers on the project, are PMCs on other
> Apache projects and Bill and Ashvin have gone through the Apache incubator
> process. Twitter has already donated numerous projects to the ASF (e.g.,
> Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be mentored
> by experienced ASF members that can help with any roadblocks.
>
> == Homogenous Developers ==
>
> Initial committers come from 5 separate organizations. Our intention is
> increase the diversity of contributing developers and their affiliations.
> To date github contributions have come from approximately 50 contributors
> from outside the Twitter team.
>
> == Reliance on Salaried Developers ==
>
> It is expected that Heron development will occur on both salaried time and
> on volunteer time. The majority of initial committers are paid by their
> employers to contribute to this project. We are committed to recruiting
> additional committers from other organizations as well as non-salaried
> committers to join project.
>
> == Relationships with Other Apache Products ==
>
> As mentioned in the Alignment section, Heron implements the Apache Storm
> API and integrates with multiple Apache schedulers (Apache Mesos, Apache
> Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache
> Thrift.
>
> == An Excessive Fascination with the Apache Brand ==
>
> Heron's popularity is growing in the streaming compute space and we are
> long time supporters of the Apache brand. This proposal is not for the
> purpose of generating publicity through. Rather, the primary benefits to
> joining Apache are those of community building and open decision making
> outlined in the Rationale section.
>
> == Documentation ==
>
> This proposal exists online as
> http://wiki.apache.org/incubator/HeronProposal. Extensive documentation can
> be found on github at https://twitter.github.io/heron and the source code
> is well documented.
>
> == Source and Intellectual Property Submission Plan ==
>
> The Heron codebase is currently hosted on Github:
> https://github.com/twitter/heron. During incubation, the codebase will be
> migrated to Apache infrastructure. The source code is already ASF 2.0
> licensed.
>
> == External Dependencies ==
>
> All external libraries have ASF 2.0 compatible licenses except for pylint.
> The pylint library is GPL licensed, but is only used for pre-build Python
> style checks and is neither bundled with, nor relied upon by, the Heron
> source or binary release artifacts.
>
> == Cryptography ==
>
> Heron does not use any cryptography libraries.
>
> = Required Resources =
>
> == Mailing lists ==
>
>  * private@heron.incubator.apache.org (with moderated subscriptions)
>  * dev@heron.incubator.apache.org
>  * commits@heron.incubator.apache.org
>  * user@heron.incubator.apache.org
>
> == Subversion Directory ==
>
> Git is the preferred source control system: git://git.apache.org/heron
>
> == Issue Tracking ==
>
> JIRA: Heron (HERON)
>
> == Initial Committers ==
>
>  * Andrew Jorgensen (andrew at andrewjorgensen dot com)
>  * Ashvin Agrawal (ashvin at apache dot org)*
>  * Avrilia Floratou (avrilia dot floratou at gmail dot com)
>  * Bill Graham (billgraham at apache dot org)*
>  * Brian Hatfield (bmhatfield at gmail dot com)
>  * Chris Kellogg (cckellogg at gmail dot com)
>  * Huijun Wu (huijun dot wu dot 2010 at gmail dot com)
>  * Karthik Ramasamy (karthik at gmail dot com)
>  * Maosong Fu (maosongfu at gmail dot com)
>  * Neng Lu(freeneng at gmail dot com)
>  * Runhang Li (obj dot runhang at gmail dot com)
>  * Sanjeev Kulkarni (sanjeevrk at gmail dot com)
>  * Supun Kamburugamuve (supun at apache dot org)*
>  * Thomas Sun (tom dot ssf at gmail dot com)
>  * Yaliang Wang (yaliang dot w dot wang at ieee dot org)
>
> == Affiliations ==
>
>  * Andrew Jorgensen (Google)
>  * Ashvin Agrawal (Microsoft)
>  * Avrilia Floratou (Microsoft)
>  * Bill Graham (Twitter)
>  * Brian Hatfield (Google)
>  * Chris Kellogg (Twitter)
>  * Huijun Wu (Twitter)
>  * Karthik Ramasamy (Streamlio)
>  * Maosong Fu (Twitter)
>  * Neng Lu (Twitter)
>  * Runhang Li (Twitter)
>  * Sanjeev Kulkarni (Streamlio)
>  * Supun Kamburugamuve (Indiana University)
>  * Thomas Sun (Twitter)
>  * Yaliang Wang (Twitter)
>
> = Sponsors =
>
> == Champion ==
>
>  * Julien Le Dem (julien at apache dot org)
>
> == Nominated Mentors ==
>
>  * Jake Farrell (jfarrell at apache dot org)
>  * Jacques Nadeau (jacques at apache dot org)
>  * Julien Le Dem (julien at apache dot org)
>  * P. Taylor Goetz (ptgoetz at apache dot org)
>
> == Sponsoring Entity ==
>
> The Apache Incubator
>
> == Footnotes ==
>
>  * 1 - Papers detailing Heron are available at
> http://dl.acm.org/citation.cfm?id=2742788 and
> http://sites.computer.org/debull/A15dec/p15.pdf.
>  * 2 - http://home.apache.org/phonebook.html?uid=billgraham
>  * 3 - http://home.apache.org/phonebook.html?uid=ashvin
>  * 4 - http://home.apache.org/phonebook.html?uid=supun

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [VOTE] Heron to enter Apache Incubator

Posted by Chris Aniszczyk <ca...@gmail.com>.
+1 (non-binding)

On Fri, Jun 16, 2017 at 3:41 PM, Bill Graham <bi...@gmail.com> wrote:

> Hi,
>
> Based on the discussion on the incubator mailing list[1] I would like to
> call a vote to add Heron to the Apache Incubator.
>
> The full proposal is available below, and is also available on the Apache
> Incubator wiki at:
>     https://wiki.apache.org/incubator/HeronProposal
>
> Please vote:
>   [ ] +1, bring Heron into Incubator
>   [ ] -1, do not bring Heron into Incubator, because...
>
> The vote will open for 7 days until Friday June 23 at 14:00 PT.
>
> Thank you
>
> 1 -
> https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7
> 786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E
>
>
>
> = Heron Proposal =
>
> = Abstract =
> Heron is a real-time, distributed, fault-tolerant stream processing engine
> initially developed by Twitter.
>
> = Proposal =
>
> Heron is a real-time stream processing engine built for high performance,
> ease of manageability, performance predictability and developer
> productivity[1]. We wish to develop a community around Heron to increase
> contributions and see Heron thrive in an open forum.
>
> = Background =
>
> Heron provides the ability for developers to compose directed acyclic
> graphs (DAGs) of real-time query execution logic (i.e. a topology) and
> submit the topology to execute on a pluggable job scheduling system (e.g.,
> Apache Aurora, YARN, Marathon, etc). Users can employ either the native
> Heron API or the Apache Storm API to develop the topology. Heron supports
> the Storm API for ease of migration, but beyond that Heron’s architecture
> differs considerably from Storm’s.
>
> Users submit a topology to the scheduler using the Heron client, which uses
> the Heron binary libraries to deploy all daemons required to run and manage
> the topology. The topology therefore has no reliance on centrally managed
> Heron services, only on a generic job scheduling system, which lends itself
> well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN (among
> others).
>
> The scheduler runs each topology as a job consisting of multiple
> containers. One of the containers runs the topology master, responsible for
> managing the topology. The remaining containers each runs a stream manager
> responsible for data routing, a metrics manager that collects and reports
> various metrics and a number of processes called Heron instances which run
> the user-defined logic on the stream of tuples. Parallelism is achieved via
> process-based isolation of Heron instances, which provides predictable
> performance while simplifying debugging. The containers are allocated and
> managed by the scheduler framework based on resource availability of nodes
> in the cluster. The metadata for the topology, such as the physical plan
> and execution details, are stored in the pluggable Heron State Manager
> (e.g. Apache ZooKeeper).
>
> = Rationale =
>
> Heron is a general-purpose, modular and extensible platform that can be
> leveraged to support common, real-time analytics use cases. There is an
> increasing demand for open-source, scalable real-time analytics systems. We
> believe that Heron can be leveraged by other organizations to build
> streaming applications that can benefit from its robustness, high
> performance, adaptability to cloud environments and ease of use. Moreover,
> we hope that open-sourcing Heron will help to further evolve the technology
> as the project attracts contributors with diverse backgrounds and areas of
> expertise.
>
> We believe the Apache foundation is a great fit as the long-term home for
> Heron, as it provides an established process for community-driven
> development and decision making by consensus. This is exactly the model we
> want for future Heron development.
>
> = Initial Goals =
>
>  * Move the existing codebase, website, documentation, and mailing lists to
> Apache-hosted infrastructure.
>  * Integrate with the Apache development process.
>  * Ensure all dependencies are compliant with Apache License version 2.0.
>  * Incrementally develop and release per Apache guidelines.
>
> = Current Status =
>
> Heron is a stable project used in production at Twitter since 2014 and open
> sourced under the ASL v2 license in 2016. The Heron source code is
> currently hosted at github.com (https://github.com/twitter/heron), which
> will seed the Apache git repository.
>
> = Meritocracy =
>
> By submitting this incubator proposal, we’re expressing our intent to build
> a diverse developer community around Heron that will conduct itself
> according to The Apache Way and use a meritocratic means of building it's
> committer base. Several companies and universities have already expressed
> interest in and contributed to Heron. Our goal is to grow the Heron
> community by encouraging open communication, contribution and participation
> of all types, and ensuring that contributors are recognized appropriately.
>
> = Community =
>
> Heron is currently being used by Twitter, Google, Machine Zone and
> ndustrial.io and has received significant contributions by Microsoft and
> Streamlio. By bringing Heron into the Apache ecosystem, we believe we can
> attract even more developers who are interested in creating real-time
> systems to build the project's contributor base.
>
> == Core Developers ==
>
> Current core developers are engineers from Twitter, Google, Microsoft and
> Streamlio.
>
> == Alignment ==
>
> Heron utilizes a number of Apache technologies. Heron leverages Apache
> ZooKeeper for coordination and has scheduler implementations to integrate
> with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache REEF)
> as well as spout implementations to integrate with Apache Kafka and metrics
> implementations to integrate with Scribe. Heron also implements the Apache
> Storm user-level API, which allows topologies written against Storm to run
> in Heron. We believe that having Heron at Apache will help further the
> growth of the streaming compute community, as well as encourage cooperation
> and developer cross pollination with other Apache projects.
>
> = Known Risks =
>
> == Orphaned Products ==
>
> The risk of the Heron project being abandoned is minimal. It is used in
> production at Twitter and Google and other companies are evaluating or
> adopting it for production use.
>
> == Inexperience with Open Source ==
>
> All of the core contributors to the project have considerable experience
> with open source software development. Bill Graham[2], Ashvin Agrawal[3]
> and Supun Kamburugamuve[4], committers on the project, are PMCs on other
> Apache projects and Bill and Ashvin have gone through the Apache incubator
> process. Twitter has already donated numerous projects to the ASF (e.g.,
> Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be mentored
> by experienced ASF members that can help with any roadblocks.
>
> == Homogenous Developers ==
>
> Initial committers come from 5 separate organizations. Our intention is
> increase the diversity of contributing developers and their affiliations.
> To date github contributions have come from approximately 50 contributors
> from outside the Twitter team.
>
> == Reliance on Salaried Developers ==
>
> It is expected that Heron development will occur on both salaried time and
> on volunteer time. The majority of initial committers are paid by their
> employers to contribute to this project. We are committed to recruiting
> additional committers from other organizations as well as non-salaried
> committers to join project.
>
> == Relationships with Other Apache Products ==
>
> As mentioned in the Alignment section, Heron implements the Apache Storm
> API and integrates with multiple Apache schedulers (Apache Mesos, Apache
> Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache
> Thrift.
>
> == An Excessive Fascination with the Apache Brand ==
>
> Heron's popularity is growing in the streaming compute space and we are
> long time supporters of the Apache brand. This proposal is not for the
> purpose of generating publicity through. Rather, the primary benefits to
> joining Apache are those of community building and open decision making
> outlined in the Rationale section.
>
> == Documentation ==
>
> This proposal exists online as
> http://wiki.apache.org/incubator/HeronProposal. Extensive documentation
> can
> be found on github at https://twitter.github.io/heron and the source code
> is well documented.
>
> == Source and Intellectual Property Submission Plan ==
>
> The Heron codebase is currently hosted on Github:
> https://github.com/twitter/heron. During incubation, the codebase will be
> migrated to Apache infrastructure. The source code is already ASF 2.0
> licensed.
>
> == External Dependencies ==
>
> All external libraries have ASF 2.0 compatible licenses except for pylint.
> The pylint library is GPL licensed, but is only used for pre-build Python
> style checks and is neither bundled with, nor relied upon by, the Heron
> source or binary release artifacts.
>
> == Cryptography ==
>
> Heron does not use any cryptography libraries.
>
> = Required Resources =
>
> == Mailing lists ==
>
>  * private@heron.incubator.apache.org (with moderated subscriptions)
>  * dev@heron.incubator.apache.org
>  * commits@heron.incubator.apache.org
>  * user@heron.incubator.apache.org
>
> == Subversion Directory ==
>
> Git is the preferred source control system: git://git.apache.org/heron
>
> == Issue Tracking ==
>
> JIRA: Heron (HERON)
>
> == Initial Committers ==
>
>  * Andrew Jorgensen (andrew at andrewjorgensen dot com)
>  * Ashvin Agrawal (ashvin at apache dot org)*
>  * Avrilia Floratou (avrilia dot floratou at gmail dot com)
>  * Bill Graham (billgraham at apache dot org)*
>  * Brian Hatfield (bmhatfield at gmail dot com)
>  * Chris Kellogg (cckellogg at gmail dot com)
>  * Huijun Wu (huijun dot wu dot 2010 at gmail dot com)
>  * Karthik Ramasamy (karthik at gmail dot com)
>  * Maosong Fu (maosongfu at gmail dot com)
>  * Neng Lu(freeneng at gmail dot com)
>  * Runhang Li (obj dot runhang at gmail dot com)
>  * Sanjeev Kulkarni (sanjeevrk at gmail dot com)
>  * Supun Kamburugamuve (supun at apache dot org)*
>  * Thomas Sun (tom dot ssf at gmail dot com)
>  * Yaliang Wang (yaliang dot w dot wang at ieee dot org)
>
> == Affiliations ==
>
>  * Andrew Jorgensen (Google)
>  * Ashvin Agrawal (Microsoft)
>  * Avrilia Floratou (Microsoft)
>  * Bill Graham (Twitter)
>  * Brian Hatfield (Google)
>  * Chris Kellogg (Twitter)
>  * Huijun Wu (Twitter)
>  * Karthik Ramasamy (Streamlio)
>  * Maosong Fu (Twitter)
>  * Neng Lu (Twitter)
>  * Runhang Li (Twitter)
>  * Sanjeev Kulkarni (Streamlio)
>  * Supun Kamburugamuve (Indiana University)
>  * Thomas Sun (Twitter)
>  * Yaliang Wang (Twitter)
>
> = Sponsors =
>
> == Champion ==
>
>  * Julien Le Dem (julien at apache dot org)
>
> == Nominated Mentors ==
>
>  * Jake Farrell (jfarrell at apache dot org)
>  * Jacques Nadeau (jacques at apache dot org)
>  * Julien Le Dem (julien at apache dot org)
>  * P. Taylor Goetz (ptgoetz at apache dot org)
>
> == Sponsoring Entity ==
>
> The Apache Incubator
>
> == Footnotes ==
>
>  * 1 - Papers detailing Heron are available at
> http://dl.acm.org/citation.cfm?id=2742788 and
> http://sites.computer.org/debull/A15dec/p15.pdf.
>  * 2 - http://home.apache.org/phonebook.html?uid=billgraham
>  * 3 - http://home.apache.org/phonebook.html?uid=ashvin
>  * 4 - http://home.apache.org/phonebook.html?uid=supun
>



-- 
Cheers,

Chris Aniszczyk
http://aniszczyk.org
+1 512 961 6719

Re: [VOTE] Heron to enter Apache Incubator

Posted by Raphael Bircher <rb...@gmail.com>.
Go for it! +1 (binding)

Am .06.2017, 22:41 Uhr, schrieb Bill Graham <bi...@gmail.com>:

> Hi,
>
> Based on the discussion on the incubator mailing list[1] I would like to
> call a vote to add Heron to the Apache Incubator.
>
> The full proposal is available below, and is also available on the Apache
> Incubator wiki at:
>     https://wiki.apache.org/incubator/HeronProposal
>
> Please vote:
>   [ ] +1, bring Heron into Incubator
>   [ ] -1, do not bring Heron into Incubator, because...
>
> The vote will open for 7 days until Friday June 23 at 14:00 PT.
>
> Thank you
>
> 1 -
> https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E
>
>
>
> = Heron Proposal =
>
> = Abstract =
> Heron is a real-time, distributed, fault-tolerant stream processing  
> engine
> initially developed by Twitter.
>
> = Proposal =
>
> Heron is a real-time stream processing engine built for high performance,
> ease of manageability, performance predictability and developer
> productivity[1]. We wish to develop a community around Heron to increase
> contributions and see Heron thrive in an open forum.
>
> = Background =
>
> Heron provides the ability for developers to compose directed acyclic
> graphs (DAGs) of real-time query execution logic (i.e. a topology) and
> submit the topology to execute on a pluggable job scheduling system  
> (e.g.,
> Apache Aurora, YARN, Marathon, etc). Users can employ either the native
> Heron API or the Apache Storm API to develop the topology. Heron supports
> the Storm API for ease of migration, but beyond that Heron’s architecture
> differs considerably from Storm’s.
>
> Users submit a topology to the scheduler using the Heron client, which  
> uses
> the Heron binary libraries to deploy all daemons required to run and  
> manage
> the topology. The topology therefore has no reliance on centrally managed
> Heron services, only on a generic job scheduling system, which lends  
> itself
> well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN (among
> others).
>
> The scheduler runs each topology as a job consisting of multiple
> containers. One of the containers runs the topology master, responsible  
> for
> managing the topology. The remaining containers each runs a stream  
> manager
> responsible for data routing, a metrics manager that collects and reports
> various metrics and a number of processes called Heron instances which  
> run
> the user-defined logic on the stream of tuples. Parallelism is achieved  
> via
> process-based isolation of Heron instances, which provides predictable
> performance while simplifying debugging. The containers are allocated and
> managed by the scheduler framework based on resource availability of  
> nodes
> in the cluster. The metadata for the topology, such as the physical plan
> and execution details, are stored in the pluggable Heron State Manager
> (e.g. Apache ZooKeeper).
>
> = Rationale =
>
> Heron is a general-purpose, modular and extensible platform that can be
> leveraged to support common, real-time analytics use cases. There is an
> increasing demand for open-source, scalable real-time analytics systems.  
> We
> believe that Heron can be leveraged by other organizations to build
> streaming applications that can benefit from its robustness, high
> performance, adaptability to cloud environments and ease of use.  
> Moreover,
> we hope that open-sourcing Heron will help to further evolve the  
> technology
> as the project attracts contributors with diverse backgrounds and areas  
> of
> expertise.
>
> We believe the Apache foundation is a great fit as the long-term home for
> Heron, as it provides an established process for community-driven
> development and decision making by consensus. This is exactly the model  
> we
> want for future Heron development.
>
> = Initial Goals =
>
>  * Move the existing codebase, website, documentation, and mailing lists  
> to
> Apache-hosted infrastructure.
>  * Integrate with the Apache development process.
>  * Ensure all dependencies are compliant with Apache License version 2.0.
>  * Incrementally develop and release per Apache guidelines.
>
> = Current Status =
>
> Heron is a stable project used in production at Twitter since 2014 and  
> open
> sourced under the ASL v2 license in 2016. The Heron source code is
> currently hosted at github.com (https://github.com/twitter/heron), which
> will seed the Apache git repository.
>
> = Meritocracy =
>
> By submitting this incubator proposal, we’re expressing our intent to  
> build
> a diverse developer community around Heron that will conduct itself
> according to The Apache Way and use a meritocratic means of building it's
> committer base. Several companies and universities have already expressed
> interest in and contributed to Heron. Our goal is to grow the Heron
> community by encouraging open communication, contribution and  
> participation
> of all types, and ensuring that contributors are recognized  
> appropriately.
>
> = Community =
>
> Heron is currently being used by Twitter, Google, Machine Zone and
> ndustrial.io and has received significant contributions by Microsoft and
> Streamlio. By bringing Heron into the Apache ecosystem, we believe we can
> attract even more developers who are interested in creating real-time
> systems to build the project's contributor base.
>
> == Core Developers ==
>
> Current core developers are engineers from Twitter, Google, Microsoft and
> Streamlio.
>
> == Alignment ==
>
> Heron utilizes a number of Apache technologies. Heron leverages Apache
> ZooKeeper for coordination and has scheduler implementations to integrate
> with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache  
> REEF)
> as well as spout implementations to integrate with Apache Kafka and  
> metrics
> implementations to integrate with Scribe. Heron also implements the  
> Apache
> Storm user-level API, which allows topologies written against Storm to  
> run
> in Heron. We believe that having Heron at Apache will help further the
> growth of the streaming compute community, as well as encourage  
> cooperation
> and developer cross pollination with other Apache projects.
>
> = Known Risks =
>
> == Orphaned Products ==
>
> The risk of the Heron project being abandoned is minimal. It is used in
> production at Twitter and Google and other companies are evaluating or
> adopting it for production use.
>
> == Inexperience with Open Source ==
>
> All of the core contributors to the project have considerable experience
> with open source software development. Bill Graham[2], Ashvin Agrawal[3]
> and Supun Kamburugamuve[4], committers on the project, are PMCs on other
> Apache projects and Bill and Ashvin have gone through the Apache  
> incubator
> process. Twitter has already donated numerous projects to the ASF (e.g.,
> Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be mentored
> by experienced ASF members that can help with any roadblocks.
>
> == Homogenous Developers ==
>
> Initial committers come from 5 separate organizations. Our intention is
> increase the diversity of contributing developers and their affiliations.
> To date github contributions have come from approximately 50 contributors
> from outside the Twitter team.
>
> == Reliance on Salaried Developers ==
>
> It is expected that Heron development will occur on both salaried time  
> and
> on volunteer time. The majority of initial committers are paid by their
> employers to contribute to this project. We are committed to recruiting
> additional committers from other organizations as well as non-salaried
> committers to join project.
>
> == Relationships with Other Apache Products ==
>
> As mentioned in the Alignment section, Heron implements the Apache Storm
> API and integrates with multiple Apache schedulers (Apache Mesos, Apache
> Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache
> Thrift.
>
> == An Excessive Fascination with the Apache Brand ==
>
> Heron's popularity is growing in the streaming compute space and we are
> long time supporters of the Apache brand. This proposal is not for the
> purpose of generating publicity through. Rather, the primary benefits to
> joining Apache are those of community building and open decision making
> outlined in the Rationale section.
>
> == Documentation ==
>
> This proposal exists online as
> http://wiki.apache.org/incubator/HeronProposal. Extensive documentation  
> can
> be found on github at https://twitter.github.io/heron and the source code
> is well documented.
>
> == Source and Intellectual Property Submission Plan ==
>
> The Heron codebase is currently hosted on Github:
> https://github.com/twitter/heron. During incubation, the codebase will be
> migrated to Apache infrastructure. The source code is already ASF 2.0
> licensed.
>
> == External Dependencies ==
>
> All external libraries have ASF 2.0 compatible licenses except for  
> pylint.
> The pylint library is GPL licensed, but is only used for pre-build Python
> style checks and is neither bundled with, nor relied upon by, the Heron
> source or binary release artifacts.
>
> == Cryptography ==
>
> Heron does not use any cryptography libraries.
>
> = Required Resources =
>
> == Mailing lists ==
>
>  * private@heron.incubator.apache.org (with moderated subscriptions)
>  * dev@heron.incubator.apache.org
>  * commits@heron.incubator.apache.org
>  * user@heron.incubator.apache.org
>
> == Subversion Directory ==
>
> Git is the preferred source control system: git://git.apache.org/heron
>
> == Issue Tracking ==
>
> JIRA: Heron (HERON)
>
> == Initial Committers ==
>
>  * Andrew Jorgensen (andrew at andrewjorgensen dot com)
>  * Ashvin Agrawal (ashvin at apache dot org)*
>  * Avrilia Floratou (avrilia dot floratou at gmail dot com)
>  * Bill Graham (billgraham at apache dot org)*
>  * Brian Hatfield (bmhatfield at gmail dot com)
>  * Chris Kellogg (cckellogg at gmail dot com)
>  * Huijun Wu (huijun dot wu dot 2010 at gmail dot com)
>  * Karthik Ramasamy (karthik at gmail dot com)
>  * Maosong Fu (maosongfu at gmail dot com)
>  * Neng Lu(freeneng at gmail dot com)
>  * Runhang Li (obj dot runhang at gmail dot com)
>  * Sanjeev Kulkarni (sanjeevrk at gmail dot com)
>  * Supun Kamburugamuve (supun at apache dot org)*
>  * Thomas Sun (tom dot ssf at gmail dot com)
>  * Yaliang Wang (yaliang dot w dot wang at ieee dot org)
>
> == Affiliations ==
>
>  * Andrew Jorgensen (Google)
>  * Ashvin Agrawal (Microsoft)
>  * Avrilia Floratou (Microsoft)
>  * Bill Graham (Twitter)
>  * Brian Hatfield (Google)
>  * Chris Kellogg (Twitter)
>  * Huijun Wu (Twitter)
>  * Karthik Ramasamy (Streamlio)
>  * Maosong Fu (Twitter)
>  * Neng Lu (Twitter)
>  * Runhang Li (Twitter)
>  * Sanjeev Kulkarni (Streamlio)
>  * Supun Kamburugamuve (Indiana University)
>  * Thomas Sun (Twitter)
>  * Yaliang Wang (Twitter)
>
> = Sponsors =
>
> == Champion ==
>
>  * Julien Le Dem (julien at apache dot org)
>
> == Nominated Mentors ==
>
>  * Jake Farrell (jfarrell at apache dot org)
>  * Jacques Nadeau (jacques at apache dot org)
>  * Julien Le Dem (julien at apache dot org)
>  * P. Taylor Goetz (ptgoetz at apache dot org)
>
> == Sponsoring Entity ==
>
> The Apache Incubator
>
> == Footnotes ==
>
>  * 1 - Papers detailing Heron are available at
> http://dl.acm.org/citation.cfm?id=2742788 and
> http://sites.computer.org/debull/A15dec/p15.pdf.
>  * 2 - http://home.apache.org/phonebook.html?uid=billgraham
>  * 3 - http://home.apache.org/phonebook.html?uid=ashvin
>  * 4 - http://home.apache.org/phonebook.html?uid=supun


-- 
My introduction https://youtu.be/Ln4vly5sxYU

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [VOTE] Heron to enter Apache Incubator

Posted by Edward Capriolo <ed...@gmail.com>.
"*"We believe that
having Heron at Apache will help further the growth of the streaming
compute community, as well as encourage cooperation and developer cross
pollination with other Apache projects."

I realize that each incubator proposal has a statement like this. What I am
saying is entering the incubator technically is moving source code, moving
mailing lists, etc. The act of moving does not encourage participation
directly. In the case of Heron, the majority of people that know of
big-data-streaming likely know that Heron exists. In the ASF there is
already Apex, KafkaStreams, Spark, Storm, Beam, probably 3 others I can not
remember.

Diversity is not a requirement, but in the end I believe it is what will
make long term success. Over time even large successful projects change
hands, there committers move onto other challenges and entities that prop
them up initially often reallocate resources.

"Your concerns are valid and should be addressed during incubation (ensuring
cross project collaboration, building more diversity, ...)"

Right. I am not suggesting you have the answers upfront, no one knows what
the future holds. However having some plan of attack is something I would
suggest, code sharing, integration with other tools, a creative plan new
committer outreach.  "Move it to ASF and make a tweet or a press release"
happens once. :)

On Fri, Jun 23, 2017 at 12:31 PM, Julien Le Dem <ju...@gmail.com>
wrote:

> Hi Edward,
>
> A better comparison is SQL. Heron provides an implementation of the Storm
> topology api just like a query engine would implement SQL.
> It is a statement to the Storm API that it became a reference for
> streaming. This is the shared component and I agree that both projects
> should collaborate around it.
>
> The proposal already has a statement of cooperation: *"We believe that
> having Heron at Apache will help further the growth of the streaming
> compute community, as well as encourage cooperation and developer cross
> pollination with other Apache projects."*
> If Heron started at Twitter it has now contributors from more companies. In
> particular Microsoft which has been presenting this work in conferences.
> Joining the incubator is also about growing the community. Diversity is a
> goal but not a requirement to enter the incubator. Many successful projects
> have started with a little diversity and grown.
> Heron is its own project, different form Storm both in the programming
> language used and the deployment approach.
>
> This is not a new situation, the Apache foundation has Thrift and Avro,
> Parquet and ORC to name a few competing projects that address similar needs
> with a different approach each with their own strengths and weaknesses.
>
> Your concerns are valid and should be addressed during incubation (ensuring
> cross project collaboration, building more diversity, ...)
>
> On Fri, Jun 23, 2017 at 6:54 AM, Edward Capriolo <ed...@gmail.com>
> wrote:
>
> > "The only overlap is that Heron supports the Storm user API for ease of
> > migration."
> >
> > It sounds possible possible that storm could be one user facing API with
> > two back ends inside one project.
> >
> > "Accumulo vs HBase" I do not think Accumulo and HBase is a valid
> comparison
> > one did not start out to emulate or be compatible with the other.
> >
> > In any case the largest issue I see is community. The proposed Heron
> > committer list is mostly a single company. Storm has already established
> a
> > community
> > with diverse committers. Also in terms of adoption, suppose you are a
> Storm
> > user do you run Heron side by side? Suppose your a vendor that packages
> > Hadoop and friends do you ship
> > both? Suppose you provide a no-sql database do you manage (test,
> document)
> > a connector for Heron and Storm? In my experience it is not trivial to
> keep
> > something working for example AbcBolt across Storm versions now that
> matrix
> > would double.
> >
> > I wish their was a stronger statement of cooperation in the proposal, for
> > example, "We with to establish a middle ground repo with shared
> components
> > etc". If nothing is shared other than a mentor or PMC your could run
> > into "software
> > X is the fasted way to run your storm bolts and spouts because of our
> > special sauce software Y does not have" and "software x is 2.2 years
> behind
> > the api of software Y they only implement and test 10% of the spouts we
> > support"
> >
> >
> > On Fri, Jun 23, 2017 at 8:08 AM, John D. Ament <jo...@apache.org>
> > wrote:
> >
> > > Bill,
> > >
> > > Would I be correct in understanding that Heron implements the same
> > protocol
> > > as Storm, but the actual implementation is different?
> > >
> > > John
> > >
> > > On Fri, Jun 23, 2017 at 1:36 AM Bill Graham <bi...@gmail.com>
> > wrote:
> > >
> > > > It's grossly inaccurate to refer to Heron as a Storm fork. There are
> > > about
> > > > 132k lines of code in the Heron codebase (plus 166k of codegen), of
> > which
> > > > about 7k are to implement the Apache Storm API bindings to the Heron
> > API.
> > > >
> > > > The Rationale section of the proposal discusses the Heron
> architecture,
> > > > which is a complete rewrite with little in common with Storm. The
> only
> > > > overlap is that Heron supports the Storm user API for ease of
> > migration.
> > > >
> > > > The value of having multiple projects to solve a common need is that
> > each
> > > > can foster innovation, collaboration and exchange of ideas in
> different
> > > > ways. This is not a new concept to Apache. You can look at the
> > incubator
> > > > discussions around Accumulo vs HBase (two implementations of the
> > BigTable
> > > > paper) for example, to see how two different approaches to a shared
> > > problem
> > > > can be a good thing.
> > > >
> > > > thanks,
> > > > Bill
> > > >
> > > > On Thu, Jun 22, 2017 at 6:45 PM, Von Gosling <vo...@apache.org>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I will give +1(Non-binding), but,
> > > > >
> > > > > I have the similar question about so many streaming framework in
> the
> > > > > apache, how to develop community for themselves.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Best Regards,
> > > > > Von Gosling
> > > > >
> > > > >
> > > > >
> > > > > 在 2017年6月23日，08:51，Edward Capriolo <ed...@gmail.com> 写道：
> > > > >
> > > > > I believe heron and storm should be merged back together. I do not
> > see
> > > > the
> > > > > value of storm and a storm fork in the asf.
> > > > >
> > > > > On Thursday, June 22, 2017, Bill Graham <bi...@gmail.com>
> > wrote:
> > > > >
> > > > > Thanks Taylor for relaying these sentiments, especially the part
> > about
> > > > the
> > > > > Heron website which is indeed poorly worded (I suspect this could
> > have
> > > > been
> > > > > the result of internal docs being open-sourced). I've opened this
> > pull
> > > > > request to update the language regarding Storm:
> > > > >
> > > > > https://github.com/twitter/heron/pull/1979
> > > > >
> > > > > On Thu, Jun 22, 2017 at 12:21 PM, P. Taylor Goetz <
> ptgoetz@gmail.com
> > > > > <javascript:;>> wrote:
> > > > >
> > > > > The Apache Storm PMC had a discussion regarding the Heron proposal.
> > In
> > > > >
> > > > > the
> > > > >
> > > > > spirit of openness I wanted to bring some of the sentiments
> expressed
> > > in
> > > > > that discussion back to this list. Please note that I am
> paraphrasing
> > > > >
> > > > > from
> > > > >
> > > > > that discussion and attempting to relay opinions of the collective
> > PMC,
> > > > >
> > > > > not
> > > > >
> > > > > necessarily that of any individual.
> > > > >
> > > > > * There is a general disappointment that the Heron community chose
> > not
> > > to
> > > > > engage with the Storm community and instead chose a separate path.
> > > > > * A majority of the PMC supports Heron’s incubation, though some
> felt
> > > it
> > > > > would result in unnecessary duplication of effort.
> > > > > * A majority of the PMC supports the two projects working closely
> > > > > together. A number of PMC members suggested the two projects merge
> in
> > > > >
> > > > > some
> > > > >
> > > > > way.
> > > > > * Many PMC members took issue some of the marketing language on the
> > > Heron
> > > > > website, particularly Heron being billed as “the direct successor
> to
> > > > >
> > > > > Apache
> > > > >
> > > > > Storm” and the prominent “Upgrade from Storm” links.  The main
> > concern
> > > > >
> > > > > here
> > > > >
> > > > > was such phrasing has somewhat of a hostile tone and undermines the
> > > > >
> > > > > desire
> > > > >
> > > > > for better collaboration, as well as confusing users.
> > > > >
> > > > > One of my goals as a proposed mentor for Heron and a Storm PMC
> member
> > > is
> > > > > to address some of these concerns and encourage collaboration. As I
> > > > > mentioned to the Storm PMC on that thread, if there are ongoing
> > > concerns
> > > > > from either the Storm PMC or the Heron PPMC about me acting as a
> > > mentor,
> > > > >
> > > > > I
> > > > >
> > > > > would be willing to step down.
> > > > >
> > > > > +1 (binding)
> > > > >
> > > > > -Taylor
> > > > >
> > > > > On Jun 16, 2017, at 4:41 PM, Bill Graham <billgraham@gmail.com
> > > > >
> > > > > <javascript:;>> wrote:
> > > > >
> > > > >
> > > > > Hi,
> > > > >
> > > > > Based on the discussion on the incubator mailing list[1] I would
> like
> > > > >
> > > > > to
> > > > >
> > > > > call a vote to add Heron to the Apache Incubator.
> > > > >
> > > > > The full proposal is available below, and is also available on the
> > > > >
> > > > > Apache
> > > > >
> > > > > Incubator wiki at:
> > > > >   https://wiki.apache.org/incubator/HeronProposal
> > > > >
> > > > > Please vote:
> > > > > [ ] +1, bring Heron into Incubator
> > > > > [ ] -1, do not bring Heron into Incubator, because...
> > > > >
> > > > > The vote will open for 7 days until Friday June 23 at 14:00 PT.
> > > > >
> > > > > Thank you
> > > > >
> > > > > 1 -
> > > > > https://lists.apache.org/thread.html/
> fb91f527ef479bb5df45bf2c9d93b7
> > > > >
> > > > > 786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > = Heron Proposal =
> > > > >
> > > > > = Abstract =
> > > > > Heron is a real-time, distributed, fault-tolerant stream processing
> > > > >
> > > > > engine
> > > > >
> > > > > initially developed by Twitter.
> > > > >
> > > > > = Proposal =
> > > > >
> > > > > Heron is a real-time stream processing engine built for high
> > > > >
> > > > > performance,
> > > > >
> > > > > ease of manageability, performance predictability and developer
> > > > > productivity[1]. We wish to develop a community around Heron to
> > > > >
> > > > > increase
> > > > >
> > > > > contributions and see Heron thrive in an open forum.
> > > > >
> > > > > = Background =
> > > > >
> > > > > Heron provides the ability for developers to compose directed
> acyclic
> > > > > graphs (DAGs) of real-time query execution logic (i.e. a topology)
> > and
> > > > > submit the topology to execute on a pluggable job scheduling system
> > > > >
> > > > > (e.g.,
> > > > >
> > > > > Apache Aurora, YARN, Marathon, etc). Users can employ either the
> > native
> > > > > Heron API or the Apache Storm API to develop the topology. Heron
> > > > >
> > > > > supports
> > > > >
> > > > > the Storm API for ease of migration, but beyond that Heron’s
> > > > >
> > > > > architecture
> > > > >
> > > > > differs considerably from Storm’s.
> > > > >
> > > > > Users submit a topology to the scheduler using the Heron client,
> > which
> > > > >
> > > > > uses
> > > > >
> > > > > the Heron binary libraries to deploy all daemons required to run
> and
> > > > >
> > > > > manage
> > > > >
> > > > > the topology. The topology therefore has no reliance on centrally
> > > > >
> > > > > managed
> > > > >
> > > > > Heron services, only on a generic job scheduling system, which
> lends
> > > > >
> > > > > itself
> > > > >
> > > > > well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN
> > > > >
> > > > > (among
> > > > >
> > > > > others).
> > > > >
> > > > > The scheduler runs each topology as a job consisting of multiple
> > > > > containers. One of the containers runs the topology master,
> > responsible
> > > > >
> > > > > for
> > > > >
> > > > > managing the topology. The remaining containers each runs a stream
> > > > >
> > > > > manager
> > > > >
> > > > > responsible for data routing, a metrics manager that collects and
> > > > >
> > > > > reports
> > > > >
> > > > > various metrics and a number of processes called Heron instances
> > which
> > > > >
> > > > > run
> > > > >
> > > > > the user-defined logic on the stream of tuples. Parallelism is
> > achieved
> > > > >
> > > > > via
> > > > >
> > > > > process-based isolation of Heron instances, which provides
> > predictable
> > > > > performance while simplifying debugging. The containers are
> allocated
> > > > >
> > > > > and
> > > > >
> > > > > managed by the scheduler framework based on resource availability
> of
> > > > >
> > > > > nodes
> > > > >
> > > > > in the cluster. The metadata for the topology, such as the physical
> > > > >
> > > > > plan
> > > > >
> > > > > and execution details, are stored in the pluggable Heron State
> > Manager
> > > > > (e.g. Apache ZooKeeper).
> > > > >
> > > > > = Rationale =
> > > > >
> > > > > Heron is a general-purpose, modular and extensible platform that
> can
> > be
> > > > > leveraged to support common, real-time analytics use cases. There
> is
> > an
> > > > > increasing demand for open-source, scalable real-time analytics
> > > > >
> > > > > systems.
> > > > >
> > > > > We
> > > > >
> > > > > believe that Heron can be leveraged by other organizations to build
> > > > > streaming applications that can benefit from its robustness, high
> > > > > performance, adaptability to cloud environments and ease of use.
> > > > >
> > > > > Moreover,
> > > > >
> > > > > we hope that open-sourcing Heron will help to further evolve the
> > > > >
> > > > > technology
> > > > >
> > > > > as the project attracts contributors with diverse backgrounds and
> > areas
> > > > >
> > > > > of
> > > > >
> > > > > expertise.
> > > > >
> > > > > We believe the Apache foundation is a great fit as the long-term
> home
> > > > >
> > > > > for
> > > > >
> > > > > Heron, as it provides an established process for community-driven
> > > > > development and decision making by consensus. This is exactly the
> > model
> > > > >
> > > > > we
> > > > >
> > > > > want for future Heron development.
> > > > >
> > > > > = Initial Goals =
> > > > >
> > > > > * Move the existing codebase, website, documentation, and mailing
> > lists
> > > > >
> > > > > to
> > > > >
> > > > > Apache-hosted infrastructure.
> > > > > * Integrate with the Apache development process.
> > > > > * Ensure all dependencies are compliant with Apache License version
> > > > >
> > > > > 2.0.
> > > > >
> > > > > * Incrementally develop and release per Apache guidelines.
> > > > >
> > > > > = Current Status =
> > > > >
> > > > > Heron is a stable project used in production at Twitter since 2014
> > and
> > > > >
> > > > > open
> > > > >
> > > > > sourced under the ASL v2 license in 2016. The Heron source code is
> > > > > currently hosted at github.com (https://github.com/twitter/heron),
> > > > >
> > > > > which
> > > > >
> > > > > will seed the Apache git repository.
> > > > >
> > > > > = Meritocracy =
> > > > >
> > > > > By submitting this incubator proposal, we’re expressing our intent
> to
> > > > >
> > > > > build
> > > > >
> > > > > a diverse developer community around Heron that will conduct itself
> > > > > according to The Apache Way and use a meritocratic means of
> building
> > > > >
> > > > > it's
> > > > >
> > > > > committer base. Several companies and universities have already
> > > > >
> > > > > expressed
> > > > >
> > > > > interest in and contributed to Heron. Our goal is to grow the Heron
> > > > > community by encouraging open communication, contribution and
> > > > >
> > > > > participation
> > > > >
> > > > > of all types, and ensuring that contributors are recognized
> > > > >
> > > > > appropriately.
> > > > >
> > > > >
> > > > > = Community =
> > > > >
> > > > > Heron is currently being used by Twitter, Google, Machine Zone and
> > > > > ndustrial.io and has received significant contributions by
> Microsoft
> > > > >
> > > > > and
> > > > >
> > > > > Streamlio. By bringing Heron into the Apache ecosystem, we believe
> we
> > > > >
> > > > > can
> > > > >
> > > > > attract even more developers who are interested in creating
> real-time
> > > > > systems to build the project's contributor base.
> > > > >
> > > > > == Core Developers ==
> > > > >
> > > > > Current core developers are engineers from Twitter, Google,
> Microsoft
> > > > >
> > > > > and
> > > > >
> > > > > Streamlio.
> > > > >
> > > > > == Alignment ==
> > > > >
> > > > > Heron utilizes a number of Apache technologies. Heron leverages
> > Apache
> > > > > ZooKeeper for coordination and has scheduler implementations to
> > > > >
> > > > > integrate
> > > > >
> > > > > with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via
> Apache
> > > > >
> > > > > REEF)
> > > > >
> > > > > as well as spout implementations to integrate with Apache Kafka and
> > > > >
> > > > > metrics
> > > > >
> > > > > implementations to integrate with Scribe. Heron also implements the
> > > > >
> > > > > Apache
> > > > >
> > > > > Storm user-level API, which allows topologies written against Storm
> > to
> > > > >
> > > > > run
> > > > >
> > > > > in Heron. We believe that having Heron at Apache will help further
> > the
> > > > > growth of the streaming compute community, as well as encourage
> > > > >
> > > > > cooperation
> > > > >
> > > > > and developer cross pollination with other Apache projects.
> > > > >
> > > > > = Known Risks =
> > > > >
> > > > > == Orphaned Products ==
> > > > >
> > > > > The risk of the Heron project being abandoned is minimal. It is
> used
> > in
> > > > > production at Twitter and Google and other companies are evaluating
> > or
> > > > > adopting it for production use.
> > > > >
> > > > > == Inexperience with Open Source ==
> > > > >
> > > > > All of the core contributors to the project have considerable
> > > > >
> > > > > experience
> > > > >
> > > > > with open source software development. Bill Graham[2], Ashvin
> > > > >
> > > > > Agrawal[3]
> > > > >
> > > > > and Supun Kamburugamuve[4], committers on the project, are PMCs on
> > > > >
> > > > > other
> > > > >
> > > > > Apache projects and Bill and Ashvin have gone through the Apache
> > > > >
> > > > > incubator
> > > > >
> > > > > process. Twitter has already donated numerous projects to the ASF
> > > > >
> > > > > (e.g.,
> > > > >
> > > > > Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be
> > > > >
> > > > > mentored
> > > > >
> > > > > by experienced ASF members that can help with any roadblocks.
> > > > >
> > > > > == Homogenous Developers ==
> > > > >
> > > > > Initial committers come from 5 separate organizations. Our
> intention
> > is
> > > > > increase the diversity of contributing developers and their
> > > > >
> > > > > affiliations.
> > > > >
> > > > > To date github contributions have come from approximately 50
> > > > >
> > > > > contributors
> > > > >
> > > > > from outside the Twitter team.
> > > > >
> > > > > == Reliance on Salaried Developers ==
> > > > >
> > > > > It is expected that Heron development will occur on both salaried
> > time
> > > > >
> > > > > and
> > > > >
> > > > > on volunteer time. The majority of initial committers are paid by
> > their
> > > > > employers to contribute to this project. We are committed to
> > recruiting
> > > > > additional committers from other organizations as well as
> > non-salaried
> > > > > committers to join project.
> > > > >
> > > > > == Relationships with Other Apache Products ==
> > > > >
> > > > > As mentioned in the Alignment section, Heron implements the Apache
> > > > >
> > > > > Storm
> > > > >
> > > > > API and integrates with multiple Apache schedulers (Apache Mesos,
> > > > >
> > > > > Apache
> > > > >
> > > > > Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and
> > Apache
> > > > > Thrift.
> > > > >
> > > > > == An Excessive Fascination with the Apache Brand ==
> > > > >
> > > > > Heron's popularity is growing in the streaming compute space and we
> > are
> > > > > long time supporters of the Apache brand. This proposal is not for
> > the
> > > > > purpose of generating publicity through. Rather, the primary
> benefits
> > > > >
> > > > > to
> > > > >
> > > > > joining Apache are those of community building and open decision
> > making
> > > > > outlined in the Rationale section.
> > > > >
> > > > > == Documentation ==
> > > > >
> > > > > This proposal exists online as
> > > > > http://wiki.apache.org/incubator/HeronProposal. Extensive
> > > > >
> > > > > documentation
> > > > >
> > > > > can
> > > > >
> > > > > be found on github at https://twitter.github.io/heron and the
> source
> > > > >
> > > > > code
> > > > >
> > > > > is well documented.
> > > > >
> > > > > == Source and Intellectual Property Submission Plan ==
> > > > >
> > > > > The Heron codebase is currently hosted on Github:
> > > > > https://github.com/twitter/heron. During incubation, the codebase
> > will
> > > > >
> > > > > be
> > > > >
> > > > > migrated to Apache infrastructure. The source code is already ASF
> 2.0
> > > > > licensed.
> > > > >
> > > > > == External Dependencies ==
> > > > >
> > > > > All external libraries have ASF 2.0 compatible licenses except for
> > > > >
> > > > > pylint.
> > > > >
> > > > > The pylint library is GPL licensed, but is only used for pre-build
> > > > >
> > > > > Python
> > > > >
> > > > > style checks and is neither bundled with, nor relied upon by, the
> > Heron
> > > > > source or binary release artifacts.
> > > > >
> > > > > == Cryptography ==
> > > > >
> > > > > Heron does not use any cryptography libraries.
> > > > >
> > > > > = Required Resources =
> > > > >
> > > > > == Mailing lists ==
> > > > >
> > > > > * private@heron.incubator.apache.org <javascript:;> (with
> moderated
> > > > >
> > > > > subscriptions)
> > > > >
> > > > > * dev@heron.incubator.apache.org <javascript:;>
> > > > > * commits@heron.incubator.apache.org <javascript:;>
> > > > > * user@heron.incubator.apache.org <javascript:;>
> > > > >
> > > > >
> > > > > == Subversion Directory ==
> > > > >
> > > > > Git is the preferred source control system: git://
> > git.apache.org/heron
> > > > >
> > > > > == Issue Tracking ==
> > > > >
> > > > > JIRA: Heron (HERON)
> > > > >
> > > > > == Initial Committers ==
> > > > >
> > > > > * Andrew Jorgensen (andrew at andrewjorgensen dot com)
> > > > > * Ashvin Agrawal (ashvin at apache dot org)*
> > > > > * Avrilia Floratou (avrilia dot floratou at gmail dot com)
> > > > > * Bill Graham (billgraham at apache dot org)*
> > > > > * Brian Hatfield (bmhatfield at gmail dot com)
> > > > > * Chris Kellogg (cckellogg at gmail dot com)
> > > > > * Huijun Wu (huijun dot wu dot 2010 at gmail dot com)
> > > > > * Karthik Ramasamy (karthik at gmail dot com)
> > > > > * Maosong Fu (maosongfu at gmail dot com)
> > > > > * Neng Lu(freeneng at gmail dot com)
> > > > > * Runhang Li (obj dot runhang at gmail dot com)
> > > > > * Sanjeev Kulkarni (sanjeevrk at gmail dot com)
> > > > > * Supun Kamburugamuve (supun at apache dot org)*
> > > > > * Thomas Sun (tom dot ssf at gmail dot com)
> > > > > * Yaliang Wang (yaliang dot w dot wang at ieee dot org)
> > > > >
> > > > > == Affiliations ==
> > > > >
> > > > > * Andrew Jorgensen (Google)
> > > > > * Ashvin Agrawal (Microsoft)
> > > > > * Avrilia Floratou (Microsoft)
> > > > > * Bill Graham (Twitter)
> > > > > * Brian Hatfield (Google)
> > > > > * Chris Kellogg (Twitter)
> > > > > * Huijun Wu (Twitter)
> > > > > * Karthik Ramasamy (Streamlio)
> > > > > * Maosong Fu (Twitter)
> > > > > * Neng Lu (Twitter)
> > > > > * Runhang Li (Twitter)
> > > > > * Sanjeev Kulkarni (Streamlio)
> > > > > * Supun Kamburugamuve (Indiana University)
> > > > > * Thomas Sun (Twitter)
> > > > > * Yaliang Wang (Twitter)
> > > > >
> > > > > = Sponsors =
> > > > >
> > > > > == Champion ==
> > > > >
> > > > > * Julien Le Dem (julien at apache dot org)
> > > > >
> > > > > == Nominated Mentors ==
> > > > >
> > > > > * Jake Farrell (jfarrell at apache dot org)
> > > > > * Jacques Nadeau (jacques at apache dot org)
> > > > > * Julien Le Dem (julien at apache dot org)
> > > > > * P. Taylor Goetz (ptgoetz at apache dot org)
> > > > >
> > > > > == Sponsoring Entity ==
> > > > >
> > > > > The Apache Incubator
> > > > >
> > > > > == Footnotes ==
> > > > >
> > > > > * 1 - Papers detailing Heron are available at
> > > > > http://dl.acm.org/citation.cfm?id=2742788 and
> > > > > http://sites.computer.org/debull/A15dec/p15.pdf.
> > > > > * 2 - http://home.apache.org/phonebook.html?uid=billgraham
> > > > > * 3 - http://home.apache.org/phonebook.html?uid=ashvin
> > > > > * 4 - http://home.apache.org/phonebook.html?uid=supun
> > > > >
> > > > >
> > > > >
> > > > > ------------------------------------------------------------
> > ---------
> > > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > >
> > > > > <javascript:;>
> > > > >
> > > > > For additional commands, e-mail: general-help@incubator.apache.org
> > > > >
> > > > > <javascript:;>
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Sorry this was sent from mobile. Will do less grammar and spell
> check
> > > > than
> > > > > usual.
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] Heron to enter Apache Incubator

Posted by Bill Graham <bi...@gmail.com>.
Thanks John. We'll keep the originally posted vote close time then.

To answer your previous question about protocol, basically yes at the user
spec/API level used to author topologies, but not at the internal APIs and
communications protocol, those are different. It's roughly analogous to two
different implementations of the JMS or Servlet specs, where both implement
the same spec, but have their own architecture, internal protocols and
additional features.


On Fri, Jun 23, 2017 at 9:59 AM, John D. Ament <jo...@apache.org>
wrote:

> Based on the additional comments, I'm OK with this continuing graduation.
> I would like the proposed podling to undertake a specific task to ensure
> its clear what is different between Storm and Heron, to avoid any
> unexpected competition or user confusion.
>
> John
>
> On Fri, Jun 23, 2017 at 12:32 PM Julien Le Dem <ju...@gmail.com>
> wrote:
>
> > Hi Edward,
> >
> > A better comparison is SQL. Heron provides an implementation of the Storm
> > topology api just like a query engine would implement SQL.
> > It is a statement to the Storm API that it became a reference for
> > streaming. This is the shared component and I agree that both projects
> > should collaborate around it.
> >
> > The proposal already has a statement of cooperation: *"We believe that
> > having Heron at Apache will help further the growth of the streaming
> > compute community, as well as encourage cooperation and developer cross
> > pollination with other Apache projects."*
> > If Heron started at Twitter it has now contributors from more companies.
> In
> > particular Microsoft which has been presenting this work in conferences.
> > Joining the incubator is also about growing the community. Diversity is a
> > goal but not a requirement to enter the incubator. Many successful
> projects
> > have started with a little diversity and grown.
> > Heron is its own project, different form Storm both in the programming
> > language used and the deployment approach.
> >
> > This is not a new situation, the Apache foundation has Thrift and Avro,
> > Parquet and ORC to name a few competing projects that address similar
> needs
> > with a different approach each with their own strengths and weaknesses.
> >
> > Your concerns are valid and should be addressed during incubation
> (ensuring
> > cross project collaboration, building more diversity, ...)
> >
> > On Fri, Jun 23, 2017 at 6:54 AM, Edward Capriolo <ed...@gmail.com>
> > wrote:
> >
> > > "The only overlap is that Heron supports the Storm user API for ease of
> > > migration."
> > >
> > > It sounds possible possible that storm could be one user facing API
> with
> > > two back ends inside one project.
> > >
> > > "Accumulo vs HBase" I do not think Accumulo and HBase is a valid
> > comparison
> > > one did not start out to emulate or be compatible with the other.
> > >
> > > In any case the largest issue I see is community. The proposed Heron
> > > committer list is mostly a single company. Storm has already
> established
> > a
> > > community
> > > with diverse committers. Also in terms of adoption, suppose you are a
> > Storm
> > > user do you run Heron side by side? Suppose your a vendor that packages
> > > Hadoop and friends do you ship
> > > both? Suppose you provide a no-sql database do you manage (test,
> > document)
> > > a connector for Heron and Storm? In my experience it is not trivial to
> > keep
> > > something working for example AbcBolt across Storm versions now that
> > matrix
> > > would double.
> > >
> > > I wish their was a stronger statement of cooperation in the proposal,
> for
> > > example, "We with to establish a middle ground repo with shared
> > components
> > > etc". If nothing is shared other than a mentor or PMC your could run
> > > into "software
> > > X is the fasted way to run your storm bolts and spouts because of our
> > > special sauce software Y does not have" and "software x is 2.2 years
> > behind
> > > the api of software Y they only implement and test 10% of the spouts we
> > > support"
> > >
> > >
> > > On Fri, Jun 23, 2017 at 8:08 AM, John D. Ament <jo...@apache.org>
> > > wrote:
> > >
> > > > Bill,
> > > >
> > > > Would I be correct in understanding that Heron implements the same
> > > protocol
> > > > as Storm, but the actual implementation is different?
> > > >
> > > > John
> > > >
> > > > On Fri, Jun 23, 2017 at 1:36 AM Bill Graham <bi...@gmail.com>
> > > wrote:
> > > >
> > > > > It's grossly inaccurate to refer to Heron as a Storm fork. There
> are
> > > > about
> > > > > 132k lines of code in the Heron codebase (plus 166k of codegen), of
> > > which
> > > > > about 7k are to implement the Apache Storm API bindings to the
> Heron
> > > API.
> > > > >
> > > > > The Rationale section of the proposal discusses the Heron
> > architecture,
> > > > > which is a complete rewrite with little in common with Storm. The
> > only
> > > > > overlap is that Heron supports the Storm user API for ease of
> > > migration.
> > > > >
> > > > > The value of having multiple projects to solve a common need is
> that
> > > each
> > > > > can foster innovation, collaboration and exchange of ideas in
> > different
> > > > > ways. This is not a new concept to Apache. You can look at the
> > > incubator
> > > > > discussions around Accumulo vs HBase (two implementations of the
> > > BigTable
> > > > > paper) for example, to see how two different approaches to a shared
> > > > problem
> > > > > can be a good thing.
> > > > >
> > > > > thanks,
> > > > > Bill
> > > > >
> > > > > On Thu, Jun 22, 2017 at 6:45 PM, Von Gosling <
> vongosling@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I will give +1(Non-binding), but,
> > > > > >
> > > > > > I have the similar question about so many streaming framework in
> > the
> > > > > > apache, how to develop community for themselves.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Best Regards,
> > > > > > Von Gosling
> > > > > >
> > > > > >
> > > > > >
> > > > > > 在 2017年6月23日，08:51，Edward Capriolo <ed...@gmail.com> 写道：
> > > > > >
> > > > > > I believe heron and storm should be merged back together. I do
> not
> > > see
> > > > > the
> > > > > > value of storm and a storm fork in the asf.
> > > > > >
> > > > > > On Thursday, June 22, 2017, Bill Graham <bi...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > Thanks Taylor for relaying these sentiments, especially the part
> > > about
> > > > > the
> > > > > > Heron website which is indeed poorly worded (I suspect this could
> > > have
> > > > > been
> > > > > > the result of internal docs being open-sourced). I've opened this
> > > pull
> > > > > > request to update the language regarding Storm:
> > > > > >
> > > > > > https://github.com/twitter/heron/pull/1979
> > > > > >
> > > > > > On Thu, Jun 22, 2017 at 12:21 PM, P. Taylor Goetz <
> > ptgoetz@gmail.com
> > > > > > <javascript:;>> wrote:
> > > > > >
> > > > > > The Apache Storm PMC had a discussion regarding the Heron
> proposal.
> > > In
> > > > > >
> > > > > > the
> > > > > >
> > > > > > spirit of openness I wanted to bring some of the sentiments
> > expressed
> > > > in
> > > > > > that discussion back to this list. Please note that I am
> > paraphrasing
> > > > > >
> > > > > > from
> > > > > >
> > > > > > that discussion and attempting to relay opinions of the
> collective
> > > PMC,
> > > > > >
> > > > > > not
> > > > > >
> > > > > > necessarily that of any individual.
> > > > > >
> > > > > > * There is a general disappointment that the Heron community
> chose
> > > not
> > > > to
> > > > > > engage with the Storm community and instead chose a separate
> path.
> > > > > > * A majority of the PMC supports Heron’s incubation, though some
> > felt
> > > > it
> > > > > > would result in unnecessary duplication of effort.
> > > > > > * A majority of the PMC supports the two projects working closely
> > > > > > together. A number of PMC members suggested the two projects
> merge
> > in
> > > > > >
> > > > > > some
> > > > > >
> > > > > > way.
> > > > > > * Many PMC members took issue some of the marketing language on
> the
> > > > Heron
> > > > > > website, particularly Heron being billed as “the direct successor
> > to
> > > > > >
> > > > > > Apache
> > > > > >
> > > > > > Storm” and the prominent “Upgrade from Storm” links.  The main
> > > concern
> > > > > >
> > > > > > here
> > > > > >
> > > > > > was such phrasing has somewhat of a hostile tone and undermines
> the
> > > > > >
> > > > > > desire
> > > > > >
> > > > > > for better collaboration, as well as confusing users.
> > > > > >
> > > > > > One of my goals as a proposed mentor for Heron and a Storm PMC
> > member
> > > > is
> > > > > > to address some of these concerns and encourage collaboration.
> As I
> > > > > > mentioned to the Storm PMC on that thread, if there are ongoing
> > > > concerns
> > > > > > from either the Storm PMC or the Heron PPMC about me acting as a
> > > > mentor,
> > > > > >
> > > > > > I
> > > > > >
> > > > > > would be willing to step down.
> > > > > >
> > > > > > +1 (binding)
> > > > > >
> > > > > > -Taylor
> > > > > >
> > > > > > On Jun 16, 2017, at 4:41 PM, Bill Graham <billgraham@gmail.com
> > > > > >
> > > > > > <javascript:;>> wrote:
> > > > > >
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Based on the discussion on the incubator mailing list[1] I would
> > like
> > > > > >
> > > > > > to
> > > > > >
> > > > > > call a vote to add Heron to the Apache Incubator.
> > > > > >
> > > > > > The full proposal is available below, and is also available on
> the
> > > > > >
> > > > > > Apache
> > > > > >
> > > > > > Incubator wiki at:
> > > > > >   https://wiki.apache.org/incubator/HeronProposal
> > > > > >
> > > > > > Please vote:
> > > > > > [ ] +1, bring Heron into Incubator
> > > > > > [ ] -1, do not bring Heron into Incubator, because...
> > > > > >
> > > > > > The vote will open for 7 days until Friday June 23 at 14:00 PT.
> > > > > >
> > > > > > Thank you
> > > > > >
> > > > > > 1 -
> > > > > >
> > https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7
> > > > > >
> > > > > > 786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > = Heron Proposal =
> > > > > >
> > > > > > = Abstract =
> > > > > > Heron is a real-time, distributed, fault-tolerant stream
> processing
> > > > > >
> > > > > > engine
> > > > > >
> > > > > > initially developed by Twitter.
> > > > > >
> > > > > > = Proposal =
> > > > > >
> > > > > > Heron is a real-time stream processing engine built for high
> > > > > >
> > > > > > performance,
> > > > > >
> > > > > > ease of manageability, performance predictability and developer
> > > > > > productivity[1]. We wish to develop a community around Heron to
> > > > > >
> > > > > > increase
> > > > > >
> > > > > > contributions and see Heron thrive in an open forum.
> > > > > >
> > > > > > = Background =
> > > > > >
> > > > > > Heron provides the ability for developers to compose directed
> > acyclic
> > > > > > graphs (DAGs) of real-time query execution logic (i.e. a
> topology)
> > > and
> > > > > > submit the topology to execute on a pluggable job scheduling
> system
> > > > > >
> > > > > > (e.g.,
> > > > > >
> > > > > > Apache Aurora, YARN, Marathon, etc). Users can employ either the
> > > native
> > > > > > Heron API or the Apache Storm API to develop the topology. Heron
> > > > > >
> > > > > > supports
> > > > > >
> > > > > > the Storm API for ease of migration, but beyond that Heron’s
> > > > > >
> > > > > > architecture
> > > > > >
> > > > > > differs considerably from Storm’s.
> > > > > >
> > > > > > Users submit a topology to the scheduler using the Heron client,
> > > which
> > > > > >
> > > > > > uses
> > > > > >
> > > > > > the Heron binary libraries to deploy all daemons required to run
> > and
> > > > > >
> > > > > > manage
> > > > > >
> > > > > > the topology. The topology therefore has no reliance on centrally
> > > > > >
> > > > > > managed
> > > > > >
> > > > > > Heron services, only on a generic job scheduling system, which
> > lends
> > > > > >
> > > > > > itself
> > > > > >
> > > > > > well to be run on top of Apache Aurora/Mesos or Apache
> Hadoop/YARN
> > > > > >
> > > > > > (among
> > > > > >
> > > > > > others).
> > > > > >
> > > > > > The scheduler runs each topology as a job consisting of multiple
> > > > > > containers. One of the containers runs the topology master,
> > > responsible
> > > > > >
> > > > > > for
> > > > > >
> > > > > > managing the topology. The remaining containers each runs a
> stream
> > > > > >
> > > > > > manager
> > > > > >
> > > > > > responsible for data routing, a metrics manager that collects and
> > > > > >
> > > > > > reports
> > > > > >
> > > > > > various metrics and a number of processes called Heron instances
> > > which
> > > > > >
> > > > > > run
> > > > > >
> > > > > > the user-defined logic on the stream of tuples. Parallelism is
> > > achieved
> > > > > >
> > > > > > via
> > > > > >
> > > > > > process-based isolation of Heron instances, which provides
> > > predictable
> > > > > > performance while simplifying debugging. The containers are
> > allocated
> > > > > >
> > > > > > and
> > > > > >
> > > > > > managed by the scheduler framework based on resource availability
> > of
> > > > > >
> > > > > > nodes
> > > > > >
> > > > > > in the cluster. The metadata for the topology, such as the
> physical
> > > > > >
> > > > > > plan
> > > > > >
> > > > > > and execution details, are stored in the pluggable Heron State
> > > Manager
> > > > > > (e.g. Apache ZooKeeper).
> > > > > >
> > > > > > = Rationale =
> > > > > >
> > > > > > Heron is a general-purpose, modular and extensible platform that
> > can
> > > be
> > > > > > leveraged to support common, real-time analytics use cases. There
> > is
> > > an
> > > > > > increasing demand for open-source, scalable real-time analytics
> > > > > >
> > > > > > systems.
> > > > > >
> > > > > > We
> > > > > >
> > > > > > believe that Heron can be leveraged by other organizations to
> build
> > > > > > streaming applications that can benefit from its robustness, high
> > > > > > performance, adaptability to cloud environments and ease of use.
> > > > > >
> > > > > > Moreover,
> > > > > >
> > > > > > we hope that open-sourcing Heron will help to further evolve the
> > > > > >
> > > > > > technology
> > > > > >
> > > > > > as the project attracts contributors with diverse backgrounds and
> > > areas
> > > > > >
> > > > > > of
> > > > > >
> > > > > > expertise.
> > > > > >
> > > > > > We believe the Apache foundation is a great fit as the long-term
> > home
> > > > > >
> > > > > > for
> > > > > >
> > > > > > Heron, as it provides an established process for community-driven
> > > > > > development and decision making by consensus. This is exactly the
> > > model
> > > > > >
> > > > > > we
> > > > > >
> > > > > > want for future Heron development.
> > > > > >
> > > > > > = Initial Goals =
> > > > > >
> > > > > > * Move the existing codebase, website, documentation, and mailing
> > > lists
> > > > > >
> > > > > > to
> > > > > >
> > > > > > Apache-hosted infrastructure.
> > > > > > * Integrate with the Apache development process.
> > > > > > * Ensure all dependencies are compliant with Apache License
> version
> > > > > >
> > > > > > 2.0.
> > > > > >
> > > > > > * Incrementally develop and release per Apache guidelines.
> > > > > >
> > > > > > = Current Status =
> > > > > >
> > > > > > Heron is a stable project used in production at Twitter since
> 2014
> > > and
> > > > > >
> > > > > > open
> > > > > >
> > > > > > sourced under the ASL v2 license in 2016. The Heron source code
> is
> > > > > > currently hosted at github.com (https://github.com/twitter/heron
> ),
> > > > > >
> > > > > > which
> > > > > >
> > > > > > will seed the Apache git repository.
> > > > > >
> > > > > > = Meritocracy =
> > > > > >
> > > > > > By submitting this incubator proposal, we’re expressing our
> intent
> > to
> > > > > >
> > > > > > build
> > > > > >
> > > > > > a diverse developer community around Heron that will conduct
> itself
> > > > > > according to The Apache Way and use a meritocratic means of
> > building
> > > > > >
> > > > > > it's
> > > > > >
> > > > > > committer base. Several companies and universities have already
> > > > > >
> > > > > > expressed
> > > > > >
> > > > > > interest in and contributed to Heron. Our goal is to grow the
> Heron
> > > > > > community by encouraging open communication, contribution and
> > > > > >
> > > > > > participation
> > > > > >
> > > > > > of all types, and ensuring that contributors are recognized
> > > > > >
> > > > > > appropriately.
> > > > > >
> > > > > >
> > > > > > = Community =
> > > > > >
> > > > > > Heron is currently being used by Twitter, Google, Machine Zone
> and
> > > > > > ndustrial.io and has received significant contributions by
> > Microsoft
> > > > > >
> > > > > > and
> > > > > >
> > > > > > Streamlio. By bringing Heron into the Apache ecosystem, we
> believe
> > we
> > > > > >
> > > > > > can
> > > > > >
> > > > > > attract even more developers who are interested in creating
> > real-time
> > > > > > systems to build the project's contributor base.
> > > > > >
> > > > > > == Core Developers ==
> > > > > >
> > > > > > Current core developers are engineers from Twitter, Google,
> > Microsoft
> > > > > >
> > > > > > and
> > > > > >
> > > > > > Streamlio.
> > > > > >
> > > > > > == Alignment ==
> > > > > >
> > > > > > Heron utilizes a number of Apache technologies. Heron leverages
> > > Apache
> > > > > > ZooKeeper for coordination and has scheduler implementations to
> > > > > >
> > > > > > integrate
> > > > > >
> > > > > > with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via
> > Apache
> > > > > >
> > > > > > REEF)
> > > > > >
> > > > > > as well as spout implementations to integrate with Apache Kafka
> and
> > > > > >
> > > > > > metrics
> > > > > >
> > > > > > implementations to integrate with Scribe. Heron also implements
> the
> > > > > >
> > > > > > Apache
> > > > > >
> > > > > > Storm user-level API, which allows topologies written against
> Storm
> > > to
> > > > > >
> > > > > > run
> > > > > >
> > > > > > in Heron. We believe that having Heron at Apache will help
> further
> > > the
> > > > > > growth of the streaming compute community, as well as encourage
> > > > > >
> > > > > > cooperation
> > > > > >
> > > > > > and developer cross pollination with other Apache projects.
> > > > > >
> > > > > > = Known Risks =
> > > > > >
> > > > > > == Orphaned Products ==
> > > > > >
> > > > > > The risk of the Heron project being abandoned is minimal. It is
> > used
> > > in
> > > > > > production at Twitter and Google and other companies are
> evaluating
> > > or
> > > > > > adopting it for production use.
> > > > > >
> > > > > > == Inexperience with Open Source ==
> > > > > >
> > > > > > All of the core contributors to the project have considerable
> > > > > >
> > > > > > experience
> > > > > >
> > > > > > with open source software development. Bill Graham[2], Ashvin
> > > > > >
> > > > > > Agrawal[3]
> > > > > >
> > > > > > and Supun Kamburugamuve[4], committers on the project, are PMCs
> on
> > > > > >
> > > > > > other
> > > > > >
> > > > > > Apache projects and Bill and Ashvin have gone through the Apache
> > > > > >
> > > > > > incubator
> > > > > >
> > > > > > process. Twitter has already donated numerous projects to the ASF
> > > > > >
> > > > > > (e.g.,
> > > > > >
> > > > > > Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be
> > > > > >
> > > > > > mentored
> > > > > >
> > > > > > by experienced ASF members that can help with any roadblocks.
> > > > > >
> > > > > > == Homogenous Developers ==
> > > > > >
> > > > > > Initial committers come from 5 separate organizations. Our
> > intention
> > > is
> > > > > > increase the diversity of contributing developers and their
> > > > > >
> > > > > > affiliations.
> > > > > >
> > > > > > To date github contributions have come from approximately 50
> > > > > >
> > > > > > contributors
> > > > > >
> > > > > > from outside the Twitter team.
> > > > > >
> > > > > > == Reliance on Salaried Developers ==
> > > > > >
> > > > > > It is expected that Heron development will occur on both salaried
> > > time
> > > > > >
> > > > > > and
> > > > > >
> > > > > > on volunteer time. The majority of initial committers are paid by
> > > their
> > > > > > employers to contribute to this project. We are committed to
> > > recruiting
> > > > > > additional committers from other organizations as well as
> > > non-salaried
> > > > > > committers to join project.
> > > > > >
> > > > > > == Relationships with Other Apache Products ==
> > > > > >
> > > > > > As mentioned in the Alignment section, Heron implements the
> Apache
> > > > > >
> > > > > > Storm
> > > > > >
> > > > > > API and integrates with multiple Apache schedulers (Apache Mesos,
> > > > > >
> > > > > > Apache
> > > > > >
> > > > > > Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and
> > > Apache
> > > > > > Thrift.
> > > > > >
> > > > > > == An Excessive Fascination with the Apache Brand ==
> > > > > >
> > > > > > Heron's popularity is growing in the streaming compute space and
> we
> > > are
> > > > > > long time supporters of the Apache brand. This proposal is not
> for
> > > the
> > > > > > purpose of generating publicity through. Rather, the primary
> > benefits
> > > > > >
> > > > > > to
> > > > > >
> > > > > > joining Apache are those of community building and open decision
> > > making
> > > > > > outlined in the Rationale section.
> > > > > >
> > > > > > == Documentation ==
> > > > > >
> > > > > > This proposal exists online as
> > > > > > http://wiki.apache.org/incubator/HeronProposal. Extensive
> > > > > >
> > > > > > documentation
> > > > > >
> > > > > > can
> > > > > >
> > > > > > be found on github at https://twitter.github.io/heron and the
> > source
> > > > > >
> > > > > > code
> > > > > >
> > > > > > is well documented.
> > > > > >
> > > > > > == Source and Intellectual Property Submission Plan ==
> > > > > >
> > > > > > The Heron codebase is currently hosted on Github:
> > > > > > https://github.com/twitter/heron. During incubation, the
> codebase
> > > will
> > > > > >
> > > > > > be
> > > > > >
> > > > > > migrated to Apache infrastructure. The source code is already ASF
> > 2.0
> > > > > > licensed.
> > > > > >
> > > > > > == External Dependencies ==
> > > > > >
> > > > > > All external libraries have ASF 2.0 compatible licenses except
> for
> > > > > >
> > > > > > pylint.
> > > > > >
> > > > > > The pylint library is GPL licensed, but is only used for
> pre-build
> > > > > >
> > > > > > Python
> > > > > >
> > > > > > style checks and is neither bundled with, nor relied upon by, the
> > > Heron
> > > > > > source or binary release artifacts.
> > > > > >
> > > > > > == Cryptography ==
> > > > > >
> > > > > > Heron does not use any cryptography libraries.
> > > > > >
> > > > > > = Required Resources =
> > > > > >
> > > > > > == Mailing lists ==
> > > > > >
> > > > > > * private@heron.incubator.apache.org <javascript:;> (with
> > moderated
> > > > > >
> > > > > > subscriptions)
> > > > > >
> > > > > > * dev@heron.incubator.apache.org <javascript:;>
> > > > > > * commits@heron.incubator.apache.org <javascript:;>
> > > > > > * user@heron.incubator.apache.org <javascript:;>
> > > > > >
> > > > > >
> > > > > > == Subversion Directory ==
> > > > > >
> > > > > > Git is the preferred source control system: git://
> > > git.apache.org/heron
> > > > > >
> > > > > > == Issue Tracking ==
> > > > > >
> > > > > > JIRA: Heron (HERON)
> > > > > >
> > > > > > == Initial Committers ==
> > > > > >
> > > > > > * Andrew Jorgensen (andrew at andrewjorgensen dot com)
> > > > > > * Ashvin Agrawal (ashvin at apache dot org)*
> > > > > > * Avrilia Floratou (avrilia dot floratou at gmail dot com)
> > > > > > * Bill Graham (billgraham at apache dot org)*
> > > > > > * Brian Hatfield (bmhatfield at gmail dot com)
> > > > > > * Chris Kellogg (cckellogg at gmail dot com)
> > > > > > * Huijun Wu (huijun dot wu dot 2010 at gmail dot com)
> > > > > > * Karthik Ramasamy (karthik at gmail dot com)
> > > > > > * Maosong Fu (maosongfu at gmail dot com)
> > > > > > * Neng Lu(freeneng at gmail dot com)
> > > > > > * Runhang Li (obj dot runhang at gmail dot com)
> > > > > > * Sanjeev Kulkarni (sanjeevrk at gmail dot com)
> > > > > > * Supun Kamburugamuve (supun at apache dot org)*
> > > > > > * Thomas Sun (tom dot ssf at gmail dot com)
> > > > > > * Yaliang Wang (yaliang dot w dot wang at ieee dot org)
> > > > > >
> > > > > > == Affiliations ==
> > > > > >
> > > > > > * Andrew Jorgensen (Google)
> > > > > > * Ashvin Agrawal (Microsoft)
> > > > > > * Avrilia Floratou (Microsoft)
> > > > > > * Bill Graham (Twitter)
> > > > > > * Brian Hatfield (Google)
> > > > > > * Chris Kellogg (Twitter)
> > > > > > * Huijun Wu (Twitter)
> > > > > > * Karthik Ramasamy (Streamlio)
> > > > > > * Maosong Fu (Twitter)
> > > > > > * Neng Lu (Twitter)
> > > > > > * Runhang Li (Twitter)
> > > > > > * Sanjeev Kulkarni (Streamlio)
> > > > > > * Supun Kamburugamuve (Indiana University)
> > > > > > * Thomas Sun (Twitter)
> > > > > > * Yaliang Wang (Twitter)
> > > > > >
> > > > > > = Sponsors =
> > > > > >
> > > > > > == Champion ==
> > > > > >
> > > > > > * Julien Le Dem (julien at apache dot org)
> > > > > >
> > > > > > == Nominated Mentors ==
> > > > > >
> > > > > > * Jake Farrell (jfarrell at apache dot org)
> > > > > > * Jacques Nadeau (jacques at apache dot org)
> > > > > > * Julien Le Dem (julien at apache dot org)
> > > > > > * P. Taylor Goetz (ptgoetz at apache dot org)
> > > > > >
> > > > > > == Sponsoring Entity ==
> > > > > >
> > > > > > The Apache Incubator
> > > > > >
> > > > > > == Footnotes ==
> > > > > >
> > > > > > * 1 - Papers detailing Heron are available at
> > > > > > http://dl.acm.org/citation.cfm?id=2742788 and
> > > > > > http://sites.computer.org/debull/A15dec/p15.pdf.
> > > > > > * 2 - http://home.apache.org/phonebook.html?uid=billgraham
> > > > > > * 3 - http://home.apache.org/phonebook.html?uid=ashvin
> > > > > > * 4 - http://home.apache.org/phonebook.html?uid=supun
> > > > > >
> > > > > >
> > > > > >
> > > > > > ------------------------------------------------------------
> > > ---------
> > > > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > > >
> > > > > > <javascript:;>
> > > > > >
> > > > > > For additional commands, e-mail: general-help@incubator.apache.
> org
> > > > > >
> > > > > > <javascript:;>
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Sorry this was sent from mobile. Will do less grammar and spell
> > check
> > > > > than
> > > > > > usual.
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] Heron to enter Apache Incubator

Posted by "John D. Ament" <jo...@apache.org>.
Based on the additional comments, I'm OK with this continuing graduation.
I would like the proposed podling to undertake a specific task to ensure
its clear what is different between Storm and Heron, to avoid any
unexpected competition or user confusion.

John

On Fri, Jun 23, 2017 at 12:32 PM Julien Le Dem <ju...@gmail.com>
wrote:

> Hi Edward,
>
> A better comparison is SQL. Heron provides an implementation of the Storm
> topology api just like a query engine would implement SQL.
> It is a statement to the Storm API that it became a reference for
> streaming. This is the shared component and I agree that both projects
> should collaborate around it.
>
> The proposal already has a statement of cooperation: *"We believe that
> having Heron at Apache will help further the growth of the streaming
> compute community, as well as encourage cooperation and developer cross
> pollination with other Apache projects."*
> If Heron started at Twitter it has now contributors from more companies. In
> particular Microsoft which has been presenting this work in conferences.
> Joining the incubator is also about growing the community. Diversity is a
> goal but not a requirement to enter the incubator. Many successful projects
> have started with a little diversity and grown.
> Heron is its own project, different form Storm both in the programming
> language used and the deployment approach.
>
> This is not a new situation, the Apache foundation has Thrift and Avro,
> Parquet and ORC to name a few competing projects that address similar needs
> with a different approach each with their own strengths and weaknesses.
>
> Your concerns are valid and should be addressed during incubation (ensuring
> cross project collaboration, building more diversity, ...)
>
> On Fri, Jun 23, 2017 at 6:54 AM, Edward Capriolo <ed...@gmail.com>
> wrote:
>
> > "The only overlap is that Heron supports the Storm user API for ease of
> > migration."
> >
> > It sounds possible possible that storm could be one user facing API with
> > two back ends inside one project.
> >
> > "Accumulo vs HBase" I do not think Accumulo and HBase is a valid
> comparison
> > one did not start out to emulate or be compatible with the other.
> >
> > In any case the largest issue I see is community. The proposed Heron
> > committer list is mostly a single company. Storm has already established
> a
> > community
> > with diverse committers. Also in terms of adoption, suppose you are a
> Storm
> > user do you run Heron side by side? Suppose your a vendor that packages
> > Hadoop and friends do you ship
> > both? Suppose you provide a no-sql database do you manage (test,
> document)
> > a connector for Heron and Storm? In my experience it is not trivial to
> keep
> > something working for example AbcBolt across Storm versions now that
> matrix
> > would double.
> >
> > I wish their was a stronger statement of cooperation in the proposal, for
> > example, "We with to establish a middle ground repo with shared
> components
> > etc". If nothing is shared other than a mentor or PMC your could run
> > into "software
> > X is the fasted way to run your storm bolts and spouts because of our
> > special sauce software Y does not have" and "software x is 2.2 years
> behind
> > the api of software Y they only implement and test 10% of the spouts we
> > support"
> >
> >
> > On Fri, Jun 23, 2017 at 8:08 AM, John D. Ament <jo...@apache.org>
> > wrote:
> >
> > > Bill,
> > >
> > > Would I be correct in understanding that Heron implements the same
> > protocol
> > > as Storm, but the actual implementation is different?
> > >
> > > John
> > >
> > > On Fri, Jun 23, 2017 at 1:36 AM Bill Graham <bi...@gmail.com>
> > wrote:
> > >
> > > > It's grossly inaccurate to refer to Heron as a Storm fork. There are
> > > about
> > > > 132k lines of code in the Heron codebase (plus 166k of codegen), of
> > which
> > > > about 7k are to implement the Apache Storm API bindings to the Heron
> > API.
> > > >
> > > > The Rationale section of the proposal discusses the Heron
> architecture,
> > > > which is a complete rewrite with little in common with Storm. The
> only
> > > > overlap is that Heron supports the Storm user API for ease of
> > migration.
> > > >
> > > > The value of having multiple projects to solve a common need is that
> > each
> > > > can foster innovation, collaboration and exchange of ideas in
> different
> > > > ways. This is not a new concept to Apache. You can look at the
> > incubator
> > > > discussions around Accumulo vs HBase (two implementations of the
> > BigTable
> > > > paper) for example, to see how two different approaches to a shared
> > > problem
> > > > can be a good thing.
> > > >
> > > > thanks,
> > > > Bill
> > > >
> > > > On Thu, Jun 22, 2017 at 6:45 PM, Von Gosling <vo...@apache.org>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I will give +1(Non-binding), but,
> > > > >
> > > > > I have the similar question about so many streaming framework in
> the
> > > > > apache, how to develop community for themselves.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Best Regards,
> > > > > Von Gosling
> > > > >
> > > > >
> > > > >
> > > > > 在 2017年6月23日，08:51，Edward Capriolo <ed...@gmail.com> 写道：
> > > > >
> > > > > I believe heron and storm should be merged back together. I do not
> > see
> > > > the
> > > > > value of storm and a storm fork in the asf.
> > > > >
> > > > > On Thursday, June 22, 2017, Bill Graham <bi...@gmail.com>
> > wrote:
> > > > >
> > > > > Thanks Taylor for relaying these sentiments, especially the part
> > about
> > > > the
> > > > > Heron website which is indeed poorly worded (I suspect this could
> > have
> > > > been
> > > > > the result of internal docs being open-sourced). I've opened this
> > pull
> > > > > request to update the language regarding Storm:
> > > > >
> > > > > https://github.com/twitter/heron/pull/1979
> > > > >
> > > > > On Thu, Jun 22, 2017 at 12:21 PM, P. Taylor Goetz <
> ptgoetz@gmail.com
> > > > > <javascript:;>> wrote:
> > > > >
> > > > > The Apache Storm PMC had a discussion regarding the Heron proposal.
> > In
> > > > >
> > > > > the
> > > > >
> > > > > spirit of openness I wanted to bring some of the sentiments
> expressed
> > > in
> > > > > that discussion back to this list. Please note that I am
> paraphrasing
> > > > >
> > > > > from
> > > > >
> > > > > that discussion and attempting to relay opinions of the collective
> > PMC,
> > > > >
> > > > > not
> > > > >
> > > > > necessarily that of any individual.
> > > > >
> > > > > * There is a general disappointment that the Heron community chose
> > not
> > > to
> > > > > engage with the Storm community and instead chose a separate path.
> > > > > * A majority of the PMC supports Heron’s incubation, though some
> felt
> > > it
> > > > > would result in unnecessary duplication of effort.
> > > > > * A majority of the PMC supports the two projects working closely
> > > > > together. A number of PMC members suggested the two projects merge
> in
> > > > >
> > > > > some
> > > > >
> > > > > way.
> > > > > * Many PMC members took issue some of the marketing language on the
> > > Heron
> > > > > website, particularly Heron being billed as “the direct successor
> to
> > > > >
> > > > > Apache
> > > > >
> > > > > Storm” and the prominent “Upgrade from Storm” links.  The main
> > concern
> > > > >
> > > > > here
> > > > >
> > > > > was such phrasing has somewhat of a hostile tone and undermines the
> > > > >
> > > > > desire
> > > > >
> > > > > for better collaboration, as well as confusing users.
> > > > >
> > > > > One of my goals as a proposed mentor for Heron and a Storm PMC
> member
> > > is
> > > > > to address some of these concerns and encourage collaboration. As I
> > > > > mentioned to the Storm PMC on that thread, if there are ongoing
> > > concerns
> > > > > from either the Storm PMC or the Heron PPMC about me acting as a
> > > mentor,
> > > > >
> > > > > I
> > > > >
> > > > > would be willing to step down.
> > > > >
> > > > > +1 (binding)
> > > > >
> > > > > -Taylor
> > > > >
> > > > > On Jun 16, 2017, at 4:41 PM, Bill Graham <billgraham@gmail.com
> > > > >
> > > > > <javascript:;>> wrote:
> > > > >
> > > > >
> > > > > Hi,
> > > > >
> > > > > Based on the discussion on the incubator mailing list[1] I would
> like
> > > > >
> > > > > to
> > > > >
> > > > > call a vote to add Heron to the Apache Incubator.
> > > > >
> > > > > The full proposal is available below, and is also available on the
> > > > >
> > > > > Apache
> > > > >
> > > > > Incubator wiki at:
> > > > >   https://wiki.apache.org/incubator/HeronProposal
> > > > >
> > > > > Please vote:
> > > > > [ ] +1, bring Heron into Incubator
> > > > > [ ] -1, do not bring Heron into Incubator, because...
> > > > >
> > > > > The vote will open for 7 days until Friday June 23 at 14:00 PT.
> > > > >
> > > > > Thank you
> > > > >
> > > > > 1 -
> > > > >
> https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7
> > > > >
> > > > > 786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > = Heron Proposal =
> > > > >
> > > > > = Abstract =
> > > > > Heron is a real-time, distributed, fault-tolerant stream processing
> > > > >
> > > > > engine
> > > > >
> > > > > initially developed by Twitter.
> > > > >
> > > > > = Proposal =
> > > > >
> > > > > Heron is a real-time stream processing engine built for high
> > > > >
> > > > > performance,
> > > > >
> > > > > ease of manageability, performance predictability and developer
> > > > > productivity[1]. We wish to develop a community around Heron to
> > > > >
> > > > > increase
> > > > >
> > > > > contributions and see Heron thrive in an open forum.
> > > > >
> > > > > = Background =
> > > > >
> > > > > Heron provides the ability for developers to compose directed
> acyclic
> > > > > graphs (DAGs) of real-time query execution logic (i.e. a topology)
> > and
> > > > > submit the topology to execute on a pluggable job scheduling system
> > > > >
> > > > > (e.g.,
> > > > >
> > > > > Apache Aurora, YARN, Marathon, etc). Users can employ either the
> > native
> > > > > Heron API or the Apache Storm API to develop the topology. Heron
> > > > >
> > > > > supports
> > > > >
> > > > > the Storm API for ease of migration, but beyond that Heron’s
> > > > >
> > > > > architecture
> > > > >
> > > > > differs considerably from Storm’s.
> > > > >
> > > > > Users submit a topology to the scheduler using the Heron client,
> > which
> > > > >
> > > > > uses
> > > > >
> > > > > the Heron binary libraries to deploy all daemons required to run
> and
> > > > >
> > > > > manage
> > > > >
> > > > > the topology. The topology therefore has no reliance on centrally
> > > > >
> > > > > managed
> > > > >
> > > > > Heron services, only on a generic job scheduling system, which
> lends
> > > > >
> > > > > itself
> > > > >
> > > > > well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN
> > > > >
> > > > > (among
> > > > >
> > > > > others).
> > > > >
> > > > > The scheduler runs each topology as a job consisting of multiple
> > > > > containers. One of the containers runs the topology master,
> > responsible
> > > > >
> > > > > for
> > > > >
> > > > > managing the topology. The remaining containers each runs a stream
> > > > >
> > > > > manager
> > > > >
> > > > > responsible for data routing, a metrics manager that collects and
> > > > >
> > > > > reports
> > > > >
> > > > > various metrics and a number of processes called Heron instances
> > which
> > > > >
> > > > > run
> > > > >
> > > > > the user-defined logic on the stream of tuples. Parallelism is
> > achieved
> > > > >
> > > > > via
> > > > >
> > > > > process-based isolation of Heron instances, which provides
> > predictable
> > > > > performance while simplifying debugging. The containers are
> allocated
> > > > >
> > > > > and
> > > > >
> > > > > managed by the scheduler framework based on resource availability
> of
> > > > >
> > > > > nodes
> > > > >
> > > > > in the cluster. The metadata for the topology, such as the physical
> > > > >
> > > > > plan
> > > > >
> > > > > and execution details, are stored in the pluggable Heron State
> > Manager
> > > > > (e.g. Apache ZooKeeper).
> > > > >
> > > > > = Rationale =
> > > > >
> > > > > Heron is a general-purpose, modular and extensible platform that
> can
> > be
> > > > > leveraged to support common, real-time analytics use cases. There
> is
> > an
> > > > > increasing demand for open-source, scalable real-time analytics
> > > > >
> > > > > systems.
> > > > >
> > > > > We
> > > > >
> > > > > believe that Heron can be leveraged by other organizations to build
> > > > > streaming applications that can benefit from its robustness, high
> > > > > performance, adaptability to cloud environments and ease of use.
> > > > >
> > > > > Moreover,
> > > > >
> > > > > we hope that open-sourcing Heron will help to further evolve the
> > > > >
> > > > > technology
> > > > >
> > > > > as the project attracts contributors with diverse backgrounds and
> > areas
> > > > >
> > > > > of
> > > > >
> > > > > expertise.
> > > > >
> > > > > We believe the Apache foundation is a great fit as the long-term
> home
> > > > >
> > > > > for
> > > > >
> > > > > Heron, as it provides an established process for community-driven
> > > > > development and decision making by consensus. This is exactly the
> > model
> > > > >
> > > > > we
> > > > >
> > > > > want for future Heron development.
> > > > >
> > > > > = Initial Goals =
> > > > >
> > > > > * Move the existing codebase, website, documentation, and mailing
> > lists
> > > > >
> > > > > to
> > > > >
> > > > > Apache-hosted infrastructure.
> > > > > * Integrate with the Apache development process.
> > > > > * Ensure all dependencies are compliant with Apache License version
> > > > >
> > > > > 2.0.
> > > > >
> > > > > * Incrementally develop and release per Apache guidelines.
> > > > >
> > > > > = Current Status =
> > > > >
> > > > > Heron is a stable project used in production at Twitter since 2014
> > and
> > > > >
> > > > > open
> > > > >
> > > > > sourced under the ASL v2 license in 2016. The Heron source code is
> > > > > currently hosted at github.com (https://github.com/twitter/heron),
> > > > >
> > > > > which
> > > > >
> > > > > will seed the Apache git repository.
> > > > >
> > > > > = Meritocracy =
> > > > >
> > > > > By submitting this incubator proposal, we’re expressing our intent
> to
> > > > >
> > > > > build
> > > > >
> > > > > a diverse developer community around Heron that will conduct itself
> > > > > according to The Apache Way and use a meritocratic means of
> building
> > > > >
> > > > > it's
> > > > >
> > > > > committer base. Several companies and universities have already
> > > > >
> > > > > expressed
> > > > >
> > > > > interest in and contributed to Heron. Our goal is to grow the Heron
> > > > > community by encouraging open communication, contribution and
> > > > >
> > > > > participation
> > > > >
> > > > > of all types, and ensuring that contributors are recognized
> > > > >
> > > > > appropriately.
> > > > >
> > > > >
> > > > > = Community =
> > > > >
> > > > > Heron is currently being used by Twitter, Google, Machine Zone and
> > > > > ndustrial.io and has received significant contributions by
> Microsoft
> > > > >
> > > > > and
> > > > >
> > > > > Streamlio. By bringing Heron into the Apache ecosystem, we believe
> we
> > > > >
> > > > > can
> > > > >
> > > > > attract even more developers who are interested in creating
> real-time
> > > > > systems to build the project's contributor base.
> > > > >
> > > > > == Core Developers ==
> > > > >
> > > > > Current core developers are engineers from Twitter, Google,
> Microsoft
> > > > >
> > > > > and
> > > > >
> > > > > Streamlio.
> > > > >
> > > > > == Alignment ==
> > > > >
> > > > > Heron utilizes a number of Apache technologies. Heron leverages
> > Apache
> > > > > ZooKeeper for coordination and has scheduler implementations to
> > > > >
> > > > > integrate
> > > > >
> > > > > with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via
> Apache
> > > > >
> > > > > REEF)
> > > > >
> > > > > as well as spout implementations to integrate with Apache Kafka and
> > > > >
> > > > > metrics
> > > > >
> > > > > implementations to integrate with Scribe. Heron also implements the
> > > > >
> > > > > Apache
> > > > >
> > > > > Storm user-level API, which allows topologies written against Storm
> > to
> > > > >
> > > > > run
> > > > >
> > > > > in Heron. We believe that having Heron at Apache will help further
> > the
> > > > > growth of the streaming compute community, as well as encourage
> > > > >
> > > > > cooperation
> > > > >
> > > > > and developer cross pollination with other Apache projects.
> > > > >
> > > > > = Known Risks =
> > > > >
> > > > > == Orphaned Products ==
> > > > >
> > > > > The risk of the Heron project being abandoned is minimal. It is
> used
> > in
> > > > > production at Twitter and Google and other companies are evaluating
> > or
> > > > > adopting it for production use.
> > > > >
> > > > > == Inexperience with Open Source ==
> > > > >
> > > > > All of the core contributors to the project have considerable
> > > > >
> > > > > experience
> > > > >
> > > > > with open source software development. Bill Graham[2], Ashvin
> > > > >
> > > > > Agrawal[3]
> > > > >
> > > > > and Supun Kamburugamuve[4], committers on the project, are PMCs on
> > > > >
> > > > > other
> > > > >
> > > > > Apache projects and Bill and Ashvin have gone through the Apache
> > > > >
> > > > > incubator
> > > > >
> > > > > process. Twitter has already donated numerous projects to the ASF
> > > > >
> > > > > (e.g.,
> > > > >
> > > > > Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be
> > > > >
> > > > > mentored
> > > > >
> > > > > by experienced ASF members that can help with any roadblocks.
> > > > >
> > > > > == Homogenous Developers ==
> > > > >
> > > > > Initial committers come from 5 separate organizations. Our
> intention
> > is
> > > > > increase the diversity of contributing developers and their
> > > > >
> > > > > affiliations.
> > > > >
> > > > > To date github contributions have come from approximately 50
> > > > >
> > > > > contributors
> > > > >
> > > > > from outside the Twitter team.
> > > > >
> > > > > == Reliance on Salaried Developers ==
> > > > >
> > > > > It is expected that Heron development will occur on both salaried
> > time
> > > > >
> > > > > and
> > > > >
> > > > > on volunteer time. The majority of initial committers are paid by
> > their
> > > > > employers to contribute to this project. We are committed to
> > recruiting
> > > > > additional committers from other organizations as well as
> > non-salaried
> > > > > committers to join project.
> > > > >
> > > > > == Relationships with Other Apache Products ==
> > > > >
> > > > > As mentioned in the Alignment section, Heron implements the Apache
> > > > >
> > > > > Storm
> > > > >
> > > > > API and integrates with multiple Apache schedulers (Apache Mesos,
> > > > >
> > > > > Apache
> > > > >
> > > > > Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and
> > Apache
> > > > > Thrift.
> > > > >
> > > > > == An Excessive Fascination with the Apache Brand ==
> > > > >
> > > > > Heron's popularity is growing in the streaming compute space and we
> > are
> > > > > long time supporters of the Apache brand. This proposal is not for
> > the
> > > > > purpose of generating publicity through. Rather, the primary
> benefits
> > > > >
> > > > > to
> > > > >
> > > > > joining Apache are those of community building and open decision
> > making
> > > > > outlined in the Rationale section.
> > > > >
> > > > > == Documentation ==
> > > > >
> > > > > This proposal exists online as
> > > > > http://wiki.apache.org/incubator/HeronProposal. Extensive
> > > > >
> > > > > documentation
> > > > >
> > > > > can
> > > > >
> > > > > be found on github at https://twitter.github.io/heron and the
> source
> > > > >
> > > > > code
> > > > >
> > > > > is well documented.
> > > > >
> > > > > == Source and Intellectual Property Submission Plan ==
> > > > >
> > > > > The Heron codebase is currently hosted on Github:
> > > > > https://github.com/twitter/heron. During incubation, the codebase
> > will
> > > > >
> > > > > be
> > > > >
> > > > > migrated to Apache infrastructure. The source code is already ASF
> 2.0
> > > > > licensed.
> > > > >
> > > > > == External Dependencies ==
> > > > >
> > > > > All external libraries have ASF 2.0 compatible licenses except for
> > > > >
> > > > > pylint.
> > > > >
> > > > > The pylint library is GPL licensed, but is only used for pre-build
> > > > >
> > > > > Python
> > > > >
> > > > > style checks and is neither bundled with, nor relied upon by, the
> > Heron
> > > > > source or binary release artifacts.
> > > > >
> > > > > == Cryptography ==
> > > > >
> > > > > Heron does not use any cryptography libraries.
> > > > >
> > > > > = Required Resources =
> > > > >
> > > > > == Mailing lists ==
> > > > >
> > > > > * private@heron.incubator.apache.org <javascript:;> (with
> moderated
> > > > >
> > > > > subscriptions)
> > > > >
> > > > > * dev@heron.incubator.apache.org <javascript:;>
> > > > > * commits@heron.incubator.apache.org <javascript:;>
> > > > > * user@heron.incubator.apache.org <javascript:;>
> > > > >
> > > > >
> > > > > == Subversion Directory ==
> > > > >
> > > > > Git is the preferred source control system: git://
> > git.apache.org/heron
> > > > >
> > > > > == Issue Tracking ==
> > > > >
> > > > > JIRA: Heron (HERON)
> > > > >
> > > > > == Initial Committers ==
> > > > >
> > > > > * Andrew Jorgensen (andrew at andrewjorgensen dot com)
> > > > > * Ashvin Agrawal (ashvin at apache dot org)*
> > > > > * Avrilia Floratou (avrilia dot floratou at gmail dot com)
> > > > > * Bill Graham (billgraham at apache dot org)*
> > > > > * Brian Hatfield (bmhatfield at gmail dot com)
> > > > > * Chris Kellogg (cckellogg at gmail dot com)
> > > > > * Huijun Wu (huijun dot wu dot 2010 at gmail dot com)
> > > > > * Karthik Ramasamy (karthik at gmail dot com)
> > > > > * Maosong Fu (maosongfu at gmail dot com)
> > > > > * Neng Lu(freeneng at gmail dot com)
> > > > > * Runhang Li (obj dot runhang at gmail dot com)
> > > > > * Sanjeev Kulkarni (sanjeevrk at gmail dot com)
> > > > > * Supun Kamburugamuve (supun at apache dot org)*
> > > > > * Thomas Sun (tom dot ssf at gmail dot com)
> > > > > * Yaliang Wang (yaliang dot w dot wang at ieee dot org)
> > > > >
> > > > > == Affiliations ==
> > > > >
> > > > > * Andrew Jorgensen (Google)
> > > > > * Ashvin Agrawal (Microsoft)
> > > > > * Avrilia Floratou (Microsoft)
> > > > > * Bill Graham (Twitter)
> > > > > * Brian Hatfield (Google)
> > > > > * Chris Kellogg (Twitter)
> > > > > * Huijun Wu (Twitter)
> > > > > * Karthik Ramasamy (Streamlio)
> > > > > * Maosong Fu (Twitter)
> > > > > * Neng Lu (Twitter)
> > > > > * Runhang Li (Twitter)
> > > > > * Sanjeev Kulkarni (Streamlio)
> > > > > * Supun Kamburugamuve (Indiana University)
> > > > > * Thomas Sun (Twitter)
> > > > > * Yaliang Wang (Twitter)
> > > > >
> > > > > = Sponsors =
> > > > >
> > > > > == Champion ==
> > > > >
> > > > > * Julien Le Dem (julien at apache dot org)
> > > > >
> > > > > == Nominated Mentors ==
> > > > >
> > > > > * Jake Farrell (jfarrell at apache dot org)
> > > > > * Jacques Nadeau (jacques at apache dot org)
> > > > > * Julien Le Dem (julien at apache dot org)
> > > > > * P. Taylor Goetz (ptgoetz at apache dot org)
> > > > >
> > > > > == Sponsoring Entity ==
> > > > >
> > > > > The Apache Incubator
> > > > >
> > > > > == Footnotes ==
> > > > >
> > > > > * 1 - Papers detailing Heron are available at
> > > > > http://dl.acm.org/citation.cfm?id=2742788 and
> > > > > http://sites.computer.org/debull/A15dec/p15.pdf.
> > > > > * 2 - http://home.apache.org/phonebook.html?uid=billgraham
> > > > > * 3 - http://home.apache.org/phonebook.html?uid=ashvin
> > > > > * 4 - http://home.apache.org/phonebook.html?uid=supun
> > > > >
> > > > >
> > > > >
> > > > > ------------------------------------------------------------
> > ---------
> > > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > > >
> > > > > <javascript:;>
> > > > >
> > > > > For additional commands, e-mail: general-help@incubator.apache.org
> > > > >
> > > > > <javascript:;>
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Sorry this was sent from mobile. Will do less grammar and spell
> check
> > > > than
> > > > > usual.
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] Heron to enter Apache Incubator

Posted by Julien Le Dem <ju...@gmail.com>.
Hi Edward,

A better comparison is SQL. Heron provides an implementation of the Storm
topology api just like a query engine would implement SQL.
It is a statement to the Storm API that it became a reference for
streaming. This is the shared component and I agree that both projects
should collaborate around it.

The proposal already has a statement of cooperation: *"We believe that
having Heron at Apache will help further the growth of the streaming
compute community, as well as encourage cooperation and developer cross
pollination with other Apache projects."*
If Heron started at Twitter it has now contributors from more companies. In
particular Microsoft which has been presenting this work in conferences.
Joining the incubator is also about growing the community. Diversity is a
goal but not a requirement to enter the incubator. Many successful projects
have started with a little diversity and grown.
Heron is its own project, different form Storm both in the programming
language used and the deployment approach.

This is not a new situation, the Apache foundation has Thrift and Avro,
Parquet and ORC to name a few competing projects that address similar needs
with a different approach each with their own strengths and weaknesses.

Your concerns are valid and should be addressed during incubation (ensuring
cross project collaboration, building more diversity, ...)

On Fri, Jun 23, 2017 at 6:54 AM, Edward Capriolo <ed...@gmail.com>
wrote:

> "The only overlap is that Heron supports the Storm user API for ease of
> migration."
>
> It sounds possible possible that storm could be one user facing API with
> two back ends inside one project.
>
> "Accumulo vs HBase" I do not think Accumulo and HBase is a valid comparison
> one did not start out to emulate or be compatible with the other.
>
> In any case the largest issue I see is community. The proposed Heron
> committer list is mostly a single company. Storm has already established a
> community
> with diverse committers. Also in terms of adoption, suppose you are a Storm
> user do you run Heron side by side? Suppose your a vendor that packages
> Hadoop and friends do you ship
> both? Suppose you provide a no-sql database do you manage (test, document)
> a connector for Heron and Storm? In my experience it is not trivial to keep
> something working for example AbcBolt across Storm versions now that matrix
> would double.
>
> I wish their was a stronger statement of cooperation in the proposal, for
> example, "We with to establish a middle ground repo with shared components
> etc". If nothing is shared other than a mentor or PMC your could run
> into "software
> X is the fasted way to run your storm bolts and spouts because of our
> special sauce software Y does not have" and "software x is 2.2 years behind
> the api of software Y they only implement and test 10% of the spouts we
> support"
>
>
> On Fri, Jun 23, 2017 at 8:08 AM, John D. Ament <jo...@apache.org>
> wrote:
>
> > Bill,
> >
> > Would I be correct in understanding that Heron implements the same
> protocol
> > as Storm, but the actual implementation is different?
> >
> > John
> >
> > On Fri, Jun 23, 2017 at 1:36 AM Bill Graham <bi...@gmail.com>
> wrote:
> >
> > > It's grossly inaccurate to refer to Heron as a Storm fork. There are
> > about
> > > 132k lines of code in the Heron codebase (plus 166k of codegen), of
> which
> > > about 7k are to implement the Apache Storm API bindings to the Heron
> API.
> > >
> > > The Rationale section of the proposal discusses the Heron architecture,
> > > which is a complete rewrite with little in common with Storm. The only
> > > overlap is that Heron supports the Storm user API for ease of
> migration.
> > >
> > > The value of having multiple projects to solve a common need is that
> each
> > > can foster innovation, collaboration and exchange of ideas in different
> > > ways. This is not a new concept to Apache. You can look at the
> incubator
> > > discussions around Accumulo vs HBase (two implementations of the
> BigTable
> > > paper) for example, to see how two different approaches to a shared
> > problem
> > > can be a good thing.
> > >
> > > thanks,
> > > Bill
> > >
> > > On Thu, Jun 22, 2017 at 6:45 PM, Von Gosling <vo...@apache.org>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I will give +1(Non-binding), but,
> > > >
> > > > I have the similar question about so many streaming framework in the
> > > > apache, how to develop community for themselves.
> > > >
> > > >
> > > >
> > > >
> > > > Best Regards,
> > > > Von Gosling
> > > >
> > > >
> > > >
> > > > 在 2017年6月23日，08:51，Edward Capriolo <ed...@gmail.com> 写道：
> > > >
> > > > I believe heron and storm should be merged back together. I do not
> see
> > > the
> > > > value of storm and a storm fork in the asf.
> > > >
> > > > On Thursday, June 22, 2017, Bill Graham <bi...@gmail.com>
> wrote:
> > > >
> > > > Thanks Taylor for relaying these sentiments, especially the part
> about
> > > the
> > > > Heron website which is indeed poorly worded (I suspect this could
> have
> > > been
> > > > the result of internal docs being open-sourced). I've opened this
> pull
> > > > request to update the language regarding Storm:
> > > >
> > > > https://github.com/twitter/heron/pull/1979
> > > >
> > > > On Thu, Jun 22, 2017 at 12:21 PM, P. Taylor Goetz <ptgoetz@gmail.com
> > > > <javascript:;>> wrote:
> > > >
> > > > The Apache Storm PMC had a discussion regarding the Heron proposal.
> In
> > > >
> > > > the
> > > >
> > > > spirit of openness I wanted to bring some of the sentiments expressed
> > in
> > > > that discussion back to this list. Please note that I am paraphrasing
> > > >
> > > > from
> > > >
> > > > that discussion and attempting to relay opinions of the collective
> PMC,
> > > >
> > > > not
> > > >
> > > > necessarily that of any individual.
> > > >
> > > > * There is a general disappointment that the Heron community chose
> not
> > to
> > > > engage with the Storm community and instead chose a separate path.
> > > > * A majority of the PMC supports Heron’s incubation, though some felt
> > it
> > > > would result in unnecessary duplication of effort.
> > > > * A majority of the PMC supports the two projects working closely
> > > > together. A number of PMC members suggested the two projects merge in
> > > >
> > > > some
> > > >
> > > > way.
> > > > * Many PMC members took issue some of the marketing language on the
> > Heron
> > > > website, particularly Heron being billed as “the direct successor to
> > > >
> > > > Apache
> > > >
> > > > Storm” and the prominent “Upgrade from Storm” links.  The main
> concern
> > > >
> > > > here
> > > >
> > > > was such phrasing has somewhat of a hostile tone and undermines the
> > > >
> > > > desire
> > > >
> > > > for better collaboration, as well as confusing users.
> > > >
> > > > One of my goals as a proposed mentor for Heron and a Storm PMC member
> > is
> > > > to address some of these concerns and encourage collaboration. As I
> > > > mentioned to the Storm PMC on that thread, if there are ongoing
> > concerns
> > > > from either the Storm PMC or the Heron PPMC about me acting as a
> > mentor,
> > > >
> > > > I
> > > >
> > > > would be willing to step down.
> > > >
> > > > +1 (binding)
> > > >
> > > > -Taylor
> > > >
> > > > On Jun 16, 2017, at 4:41 PM, Bill Graham <billgraham@gmail.com
> > > >
> > > > <javascript:;>> wrote:
> > > >
> > > >
> > > > Hi,
> > > >
> > > > Based on the discussion on the incubator mailing list[1] I would like
> > > >
> > > > to
> > > >
> > > > call a vote to add Heron to the Apache Incubator.
> > > >
> > > > The full proposal is available below, and is also available on the
> > > >
> > > > Apache
> > > >
> > > > Incubator wiki at:
> > > >   https://wiki.apache.org/incubator/HeronProposal
> > > >
> > > > Please vote:
> > > > [ ] +1, bring Heron into Incubator
> > > > [ ] -1, do not bring Heron into Incubator, because...
> > > >
> > > > The vote will open for 7 days until Friday June 23 at 14:00 PT.
> > > >
> > > > Thank you
> > > >
> > > > 1 -
> > > > https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7
> > > >
> > > > 786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E
> > > >
> > > >
> > > >
> > > >
> > > > = Heron Proposal =
> > > >
> > > > = Abstract =
> > > > Heron is a real-time, distributed, fault-tolerant stream processing
> > > >
> > > > engine
> > > >
> > > > initially developed by Twitter.
> > > >
> > > > = Proposal =
> > > >
> > > > Heron is a real-time stream processing engine built for high
> > > >
> > > > performance,
> > > >
> > > > ease of manageability, performance predictability and developer
> > > > productivity[1]. We wish to develop a community around Heron to
> > > >
> > > > increase
> > > >
> > > > contributions and see Heron thrive in an open forum.
> > > >
> > > > = Background =
> > > >
> > > > Heron provides the ability for developers to compose directed acyclic
> > > > graphs (DAGs) of real-time query execution logic (i.e. a topology)
> and
> > > > submit the topology to execute on a pluggable job scheduling system
> > > >
> > > > (e.g.,
> > > >
> > > > Apache Aurora, YARN, Marathon, etc). Users can employ either the
> native
> > > > Heron API or the Apache Storm API to develop the topology. Heron
> > > >
> > > > supports
> > > >
> > > > the Storm API for ease of migration, but beyond that Heron’s
> > > >
> > > > architecture
> > > >
> > > > differs considerably from Storm’s.
> > > >
> > > > Users submit a topology to the scheduler using the Heron client,
> which
> > > >
> > > > uses
> > > >
> > > > the Heron binary libraries to deploy all daemons required to run and
> > > >
> > > > manage
> > > >
> > > > the topology. The topology therefore has no reliance on centrally
> > > >
> > > > managed
> > > >
> > > > Heron services, only on a generic job scheduling system, which lends
> > > >
> > > > itself
> > > >
> > > > well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN
> > > >
> > > > (among
> > > >
> > > > others).
> > > >
> > > > The scheduler runs each topology as a job consisting of multiple
> > > > containers. One of the containers runs the topology master,
> responsible
> > > >
> > > > for
> > > >
> > > > managing the topology. The remaining containers each runs a stream
> > > >
> > > > manager
> > > >
> > > > responsible for data routing, a metrics manager that collects and
> > > >
> > > > reports
> > > >
> > > > various metrics and a number of processes called Heron instances
> which
> > > >
> > > > run
> > > >
> > > > the user-defined logic on the stream of tuples. Parallelism is
> achieved
> > > >
> > > > via
> > > >
> > > > process-based isolation of Heron instances, which provides
> predictable
> > > > performance while simplifying debugging. The containers are allocated
> > > >
> > > > and
> > > >
> > > > managed by the scheduler framework based on resource availability of
> > > >
> > > > nodes
> > > >
> > > > in the cluster. The metadata for the topology, such as the physical
> > > >
> > > > plan
> > > >
> > > > and execution details, are stored in the pluggable Heron State
> Manager
> > > > (e.g. Apache ZooKeeper).
> > > >
> > > > = Rationale =
> > > >
> > > > Heron is a general-purpose, modular and extensible platform that can
> be
> > > > leveraged to support common, real-time analytics use cases. There is
> an
> > > > increasing demand for open-source, scalable real-time analytics
> > > >
> > > > systems.
> > > >
> > > > We
> > > >
> > > > believe that Heron can be leveraged by other organizations to build
> > > > streaming applications that can benefit from its robustness, high
> > > > performance, adaptability to cloud environments and ease of use.
> > > >
> > > > Moreover,
> > > >
> > > > we hope that open-sourcing Heron will help to further evolve the
> > > >
> > > > technology
> > > >
> > > > as the project attracts contributors with diverse backgrounds and
> areas
> > > >
> > > > of
> > > >
> > > > expertise.
> > > >
> > > > We believe the Apache foundation is a great fit as the long-term home
> > > >
> > > > for
> > > >
> > > > Heron, as it provides an established process for community-driven
> > > > development and decision making by consensus. This is exactly the
> model
> > > >
> > > > we
> > > >
> > > > want for future Heron development.
> > > >
> > > > = Initial Goals =
> > > >
> > > > * Move the existing codebase, website, documentation, and mailing
> lists
> > > >
> > > > to
> > > >
> > > > Apache-hosted infrastructure.
> > > > * Integrate with the Apache development process.
> > > > * Ensure all dependencies are compliant with Apache License version
> > > >
> > > > 2.0.
> > > >
> > > > * Incrementally develop and release per Apache guidelines.
> > > >
> > > > = Current Status =
> > > >
> > > > Heron is a stable project used in production at Twitter since 2014
> and
> > > >
> > > > open
> > > >
> > > > sourced under the ASL v2 license in 2016. The Heron source code is
> > > > currently hosted at github.com (https://github.com/twitter/heron),
> > > >
> > > > which
> > > >
> > > > will seed the Apache git repository.
> > > >
> > > > = Meritocracy =
> > > >
> > > > By submitting this incubator proposal, we’re expressing our intent to
> > > >
> > > > build
> > > >
> > > > a diverse developer community around Heron that will conduct itself
> > > > according to The Apache Way and use a meritocratic means of building
> > > >
> > > > it's
> > > >
> > > > committer base. Several companies and universities have already
> > > >
> > > > expressed
> > > >
> > > > interest in and contributed to Heron. Our goal is to grow the Heron
> > > > community by encouraging open communication, contribution and
> > > >
> > > > participation
> > > >
> > > > of all types, and ensuring that contributors are recognized
> > > >
> > > > appropriately.
> > > >
> > > >
> > > > = Community =
> > > >
> > > > Heron is currently being used by Twitter, Google, Machine Zone and
> > > > ndustrial.io and has received significant contributions by Microsoft
> > > >
> > > > and
> > > >
> > > > Streamlio. By bringing Heron into the Apache ecosystem, we believe we
> > > >
> > > > can
> > > >
> > > > attract even more developers who are interested in creating real-time
> > > > systems to build the project's contributor base.
> > > >
> > > > == Core Developers ==
> > > >
> > > > Current core developers are engineers from Twitter, Google, Microsoft
> > > >
> > > > and
> > > >
> > > > Streamlio.
> > > >
> > > > == Alignment ==
> > > >
> > > > Heron utilizes a number of Apache technologies. Heron leverages
> Apache
> > > > ZooKeeper for coordination and has scheduler implementations to
> > > >
> > > > integrate
> > > >
> > > > with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache
> > > >
> > > > REEF)
> > > >
> > > > as well as spout implementations to integrate with Apache Kafka and
> > > >
> > > > metrics
> > > >
> > > > implementations to integrate with Scribe. Heron also implements the
> > > >
> > > > Apache
> > > >
> > > > Storm user-level API, which allows topologies written against Storm
> to
> > > >
> > > > run
> > > >
> > > > in Heron. We believe that having Heron at Apache will help further
> the
> > > > growth of the streaming compute community, as well as encourage
> > > >
> > > > cooperation
> > > >
> > > > and developer cross pollination with other Apache projects.
> > > >
> > > > = Known Risks =
> > > >
> > > > == Orphaned Products ==
> > > >
> > > > The risk of the Heron project being abandoned is minimal. It is used
> in
> > > > production at Twitter and Google and other companies are evaluating
> or
> > > > adopting it for production use.
> > > >
> > > > == Inexperience with Open Source ==
> > > >
> > > > All of the core contributors to the project have considerable
> > > >
> > > > experience
> > > >
> > > > with open source software development. Bill Graham[2], Ashvin
> > > >
> > > > Agrawal[3]
> > > >
> > > > and Supun Kamburugamuve[4], committers on the project, are PMCs on
> > > >
> > > > other
> > > >
> > > > Apache projects and Bill and Ashvin have gone through the Apache
> > > >
> > > > incubator
> > > >
> > > > process. Twitter has already donated numerous projects to the ASF
> > > >
> > > > (e.g.,
> > > >
> > > > Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be
> > > >
> > > > mentored
> > > >
> > > > by experienced ASF members that can help with any roadblocks.
> > > >
> > > > == Homogenous Developers ==
> > > >
> > > > Initial committers come from 5 separate organizations. Our intention
> is
> > > > increase the diversity of contributing developers and their
> > > >
> > > > affiliations.
> > > >
> > > > To date github contributions have come from approximately 50
> > > >
> > > > contributors
> > > >
> > > > from outside the Twitter team.
> > > >
> > > > == Reliance on Salaried Developers ==
> > > >
> > > > It is expected that Heron development will occur on both salaried
> time
> > > >
> > > > and
> > > >
> > > > on volunteer time. The majority of initial committers are paid by
> their
> > > > employers to contribute to this project. We are committed to
> recruiting
> > > > additional committers from other organizations as well as
> non-salaried
> > > > committers to join project.
> > > >
> > > > == Relationships with Other Apache Products ==
> > > >
> > > > As mentioned in the Alignment section, Heron implements the Apache
> > > >
> > > > Storm
> > > >
> > > > API and integrates with multiple Apache schedulers (Apache Mesos,
> > > >
> > > > Apache
> > > >
> > > > Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and
> Apache
> > > > Thrift.
> > > >
> > > > == An Excessive Fascination with the Apache Brand ==
> > > >
> > > > Heron's popularity is growing in the streaming compute space and we
> are
> > > > long time supporters of the Apache brand. This proposal is not for
> the
> > > > purpose of generating publicity through. Rather, the primary benefits
> > > >
> > > > to
> > > >
> > > > joining Apache are those of community building and open decision
> making
> > > > outlined in the Rationale section.
> > > >
> > > > == Documentation ==
> > > >
> > > > This proposal exists online as
> > > > http://wiki.apache.org/incubator/HeronProposal. Extensive
> > > >
> > > > documentation
> > > >
> > > > can
> > > >
> > > > be found on github at https://twitter.github.io/heron and the source
> > > >
> > > > code
> > > >
> > > > is well documented.
> > > >
> > > > == Source and Intellectual Property Submission Plan ==
> > > >
> > > > The Heron codebase is currently hosted on Github:
> > > > https://github.com/twitter/heron. During incubation, the codebase
> will
> > > >
> > > > be
> > > >
> > > > migrated to Apache infrastructure. The source code is already ASF 2.0
> > > > licensed.
> > > >
> > > > == External Dependencies ==
> > > >
> > > > All external libraries have ASF 2.0 compatible licenses except for
> > > >
> > > > pylint.
> > > >
> > > > The pylint library is GPL licensed, but is only used for pre-build
> > > >
> > > > Python
> > > >
> > > > style checks and is neither bundled with, nor relied upon by, the
> Heron
> > > > source or binary release artifacts.
> > > >
> > > > == Cryptography ==
> > > >
> > > > Heron does not use any cryptography libraries.
> > > >
> > > > = Required Resources =
> > > >
> > > > == Mailing lists ==
> > > >
> > > > * private@heron.incubator.apache.org <javascript:;> (with moderated
> > > >
> > > > subscriptions)
> > > >
> > > > * dev@heron.incubator.apache.org <javascript:;>
> > > > * commits@heron.incubator.apache.org <javascript:;>
> > > > * user@heron.incubator.apache.org <javascript:;>
> > > >
> > > >
> > > > == Subversion Directory ==
> > > >
> > > > Git is the preferred source control system: git://
> git.apache.org/heron
> > > >
> > > > == Issue Tracking ==
> > > >
> > > > JIRA: Heron (HERON)
> > > >
> > > > == Initial Committers ==
> > > >
> > > > * Andrew Jorgensen (andrew at andrewjorgensen dot com)
> > > > * Ashvin Agrawal (ashvin at apache dot org)*
> > > > * Avrilia Floratou (avrilia dot floratou at gmail dot com)
> > > > * Bill Graham (billgraham at apache dot org)*
> > > > * Brian Hatfield (bmhatfield at gmail dot com)
> > > > * Chris Kellogg (cckellogg at gmail dot com)
> > > > * Huijun Wu (huijun dot wu dot 2010 at gmail dot com)
> > > > * Karthik Ramasamy (karthik at gmail dot com)
> > > > * Maosong Fu (maosongfu at gmail dot com)
> > > > * Neng Lu(freeneng at gmail dot com)
> > > > * Runhang Li (obj dot runhang at gmail dot com)
> > > > * Sanjeev Kulkarni (sanjeevrk at gmail dot com)
> > > > * Supun Kamburugamuve (supun at apache dot org)*
> > > > * Thomas Sun (tom dot ssf at gmail dot com)
> > > > * Yaliang Wang (yaliang dot w dot wang at ieee dot org)
> > > >
> > > > == Affiliations ==
> > > >
> > > > * Andrew Jorgensen (Google)
> > > > * Ashvin Agrawal (Microsoft)
> > > > * Avrilia Floratou (Microsoft)
> > > > * Bill Graham (Twitter)
> > > > * Brian Hatfield (Google)
> > > > * Chris Kellogg (Twitter)
> > > > * Huijun Wu (Twitter)
> > > > * Karthik Ramasamy (Streamlio)
> > > > * Maosong Fu (Twitter)
> > > > * Neng Lu (Twitter)
> > > > * Runhang Li (Twitter)
> > > > * Sanjeev Kulkarni (Streamlio)
> > > > * Supun Kamburugamuve (Indiana University)
> > > > * Thomas Sun (Twitter)
> > > > * Yaliang Wang (Twitter)
> > > >
> > > > = Sponsors =
> > > >
> > > > == Champion ==
> > > >
> > > > * Julien Le Dem (julien at apache dot org)
> > > >
> > > > == Nominated Mentors ==
> > > >
> > > > * Jake Farrell (jfarrell at apache dot org)
> > > > * Jacques Nadeau (jacques at apache dot org)
> > > > * Julien Le Dem (julien at apache dot org)
> > > > * P. Taylor Goetz (ptgoetz at apache dot org)
> > > >
> > > > == Sponsoring Entity ==
> > > >
> > > > The Apache Incubator
> > > >
> > > > == Footnotes ==
> > > >
> > > > * 1 - Papers detailing Heron are available at
> > > > http://dl.acm.org/citation.cfm?id=2742788 and
> > > > http://sites.computer.org/debull/A15dec/p15.pdf.
> > > > * 2 - http://home.apache.org/phonebook.html?uid=billgraham
> > > > * 3 - http://home.apache.org/phonebook.html?uid=ashvin
> > > > * 4 - http://home.apache.org/phonebook.html?uid=supun
> > > >
> > > >
> > > >
> > > > ------------------------------------------------------------
> ---------
> > > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > >
> > > > <javascript:;>
> > > >
> > > > For additional commands, e-mail: general-help@incubator.apache.org
> > > >
> > > > <javascript:;>
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Sorry this was sent from mobile. Will do less grammar and spell check
> > > than
> > > > usual.
> > > >
> > > >
> > > >
> > >
> >
>

Re: [VOTE] Heron to enter Apache Incubator

Posted by Edward Capriolo <ed...@gmail.com>.
"The only overlap is that Heron supports the Storm user API for ease of
migration."

It sounds possible possible that storm could be one user facing API with
two back ends inside one project.

"Accumulo vs HBase" I do not think Accumulo and HBase is a valid comparison
one did not start out to emulate or be compatible with the other.

In any case the largest issue I see is community. The proposed Heron
committer list is mostly a single company. Storm has already established a
community
with diverse committers. Also in terms of adoption, suppose you are a Storm
user do you run Heron side by side? Suppose your a vendor that packages
Hadoop and friends do you ship
both? Suppose you provide a no-sql database do you manage (test, document)
a connector for Heron and Storm? In my experience it is not trivial to keep
something working for example AbcBolt across Storm versions now that matrix
would double.

I wish their was a stronger statement of cooperation in the proposal, for
example, "We with to establish a middle ground repo with shared components
etc". If nothing is shared other than a mentor or PMC your could run
into "software
X is the fasted way to run your storm bolts and spouts because of our
special sauce software Y does not have" and "software x is 2.2 years behind
the api of software Y they only implement and test 10% of the spouts we
support"


On Fri, Jun 23, 2017 at 8:08 AM, John D. Ament <jo...@apache.org>
wrote:

> Bill,
>
> Would I be correct in understanding that Heron implements the same protocol
> as Storm, but the actual implementation is different?
>
> John
>
> On Fri, Jun 23, 2017 at 1:36 AM Bill Graham <bi...@gmail.com> wrote:
>
> > It's grossly inaccurate to refer to Heron as a Storm fork. There are
> about
> > 132k lines of code in the Heron codebase (plus 166k of codegen), of which
> > about 7k are to implement the Apache Storm API bindings to the Heron API.
> >
> > The Rationale section of the proposal discusses the Heron architecture,
> > which is a complete rewrite with little in common with Storm. The only
> > overlap is that Heron supports the Storm user API for ease of migration.
> >
> > The value of having multiple projects to solve a common need is that each
> > can foster innovation, collaboration and exchange of ideas in different
> > ways. This is not a new concept to Apache. You can look at the incubator
> > discussions around Accumulo vs HBase (two implementations of the BigTable
> > paper) for example, to see how two different approaches to a shared
> problem
> > can be a good thing.
> >
> > thanks,
> > Bill
> >
> > On Thu, Jun 22, 2017 at 6:45 PM, Von Gosling <vo...@apache.org>
> > wrote:
> >
> > > Hi,
> > >
> > > I will give +1(Non-binding), but,
> > >
> > > I have the similar question about so many streaming framework in the
> > > apache, how to develop community for themselves.
> > >
> > >
> > >
> > >
> > > Best Regards,
> > > Von Gosling
> > >
> > >
> > >
> > > 在 2017年6月23日，08:51，Edward Capriolo <ed...@gmail.com> 写道：
> > >
> > > I believe heron and storm should be merged back together. I do not see
> > the
> > > value of storm and a storm fork in the asf.
> > >
> > > On Thursday, June 22, 2017, Bill Graham <bi...@gmail.com> wrote:
> > >
> > > Thanks Taylor for relaying these sentiments, especially the part about
> > the
> > > Heron website which is indeed poorly worded (I suspect this could have
> > been
> > > the result of internal docs being open-sourced). I've opened this pull
> > > request to update the language regarding Storm:
> > >
> > > https://github.com/twitter/heron/pull/1979
> > >
> > > On Thu, Jun 22, 2017 at 12:21 PM, P. Taylor Goetz <ptgoetz@gmail.com
> > > <javascript:;>> wrote:
> > >
> > > The Apache Storm PMC had a discussion regarding the Heron proposal. In
> > >
> > > the
> > >
> > > spirit of openness I wanted to bring some of the sentiments expressed
> in
> > > that discussion back to this list. Please note that I am paraphrasing
> > >
> > > from
> > >
> > > that discussion and attempting to relay opinions of the collective PMC,
> > >
> > > not
> > >
> > > necessarily that of any individual.
> > >
> > > * There is a general disappointment that the Heron community chose not
> to
> > > engage with the Storm community and instead chose a separate path.
> > > * A majority of the PMC supports Heron’s incubation, though some felt
> it
> > > would result in unnecessary duplication of effort.
> > > * A majority of the PMC supports the two projects working closely
> > > together. A number of PMC members suggested the two projects merge in
> > >
> > > some
> > >
> > > way.
> > > * Many PMC members took issue some of the marketing language on the
> Heron
> > > website, particularly Heron being billed as “the direct successor to
> > >
> > > Apache
> > >
> > > Storm” and the prominent “Upgrade from Storm” links.  The main concern
> > >
> > > here
> > >
> > > was such phrasing has somewhat of a hostile tone and undermines the
> > >
> > > desire
> > >
> > > for better collaboration, as well as confusing users.
> > >
> > > One of my goals as a proposed mentor for Heron and a Storm PMC member
> is
> > > to address some of these concerns and encourage collaboration. As I
> > > mentioned to the Storm PMC on that thread, if there are ongoing
> concerns
> > > from either the Storm PMC or the Heron PPMC about me acting as a
> mentor,
> > >
> > > I
> > >
> > > would be willing to step down.
> > >
> > > +1 (binding)
> > >
> > > -Taylor
> > >
> > > On Jun 16, 2017, at 4:41 PM, Bill Graham <billgraham@gmail.com
> > >
> > > <javascript:;>> wrote:
> > >
> > >
> > > Hi,
> > >
> > > Based on the discussion on the incubator mailing list[1] I would like
> > >
> > > to
> > >
> > > call a vote to add Heron to the Apache Incubator.
> > >
> > > The full proposal is available below, and is also available on the
> > >
> > > Apache
> > >
> > > Incubator wiki at:
> > >   https://wiki.apache.org/incubator/HeronProposal
> > >
> > > Please vote:
> > > [ ] +1, bring Heron into Incubator
> > > [ ] -1, do not bring Heron into Incubator, because...
> > >
> > > The vote will open for 7 days until Friday June 23 at 14:00 PT.
> > >
> > > Thank you
> > >
> > > 1 -
> > > https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7
> > >
> > > 786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E
> > >
> > >
> > >
> > >
> > > = Heron Proposal =
> > >
> > > = Abstract =
> > > Heron is a real-time, distributed, fault-tolerant stream processing
> > >
> > > engine
> > >
> > > initially developed by Twitter.
> > >
> > > = Proposal =
> > >
> > > Heron is a real-time stream processing engine built for high
> > >
> > > performance,
> > >
> > > ease of manageability, performance predictability and developer
> > > productivity[1]. We wish to develop a community around Heron to
> > >
> > > increase
> > >
> > > contributions and see Heron thrive in an open forum.
> > >
> > > = Background =
> > >
> > > Heron provides the ability for developers to compose directed acyclic
> > > graphs (DAGs) of real-time query execution logic (i.e. a topology) and
> > > submit the topology to execute on a pluggable job scheduling system
> > >
> > > (e.g.,
> > >
> > > Apache Aurora, YARN, Marathon, etc). Users can employ either the native
> > > Heron API or the Apache Storm API to develop the topology. Heron
> > >
> > > supports
> > >
> > > the Storm API for ease of migration, but beyond that Heron’s
> > >
> > > architecture
> > >
> > > differs considerably from Storm’s.
> > >
> > > Users submit a topology to the scheduler using the Heron client, which
> > >
> > > uses
> > >
> > > the Heron binary libraries to deploy all daemons required to run and
> > >
> > > manage
> > >
> > > the topology. The topology therefore has no reliance on centrally
> > >
> > > managed
> > >
> > > Heron services, only on a generic job scheduling system, which lends
> > >
> > > itself
> > >
> > > well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN
> > >
> > > (among
> > >
> > > others).
> > >
> > > The scheduler runs each topology as a job consisting of multiple
> > > containers. One of the containers runs the topology master, responsible
> > >
> > > for
> > >
> > > managing the topology. The remaining containers each runs a stream
> > >
> > > manager
> > >
> > > responsible for data routing, a metrics manager that collects and
> > >
> > > reports
> > >
> > > various metrics and a number of processes called Heron instances which
> > >
> > > run
> > >
> > > the user-defined logic on the stream of tuples. Parallelism is achieved
> > >
> > > via
> > >
> > > process-based isolation of Heron instances, which provides predictable
> > > performance while simplifying debugging. The containers are allocated
> > >
> > > and
> > >
> > > managed by the scheduler framework based on resource availability of
> > >
> > > nodes
> > >
> > > in the cluster. The metadata for the topology, such as the physical
> > >
> > > plan
> > >
> > > and execution details, are stored in the pluggable Heron State Manager
> > > (e.g. Apache ZooKeeper).
> > >
> > > = Rationale =
> > >
> > > Heron is a general-purpose, modular and extensible platform that can be
> > > leveraged to support common, real-time analytics use cases. There is an
> > > increasing demand for open-source, scalable real-time analytics
> > >
> > > systems.
> > >
> > > We
> > >
> > > believe that Heron can be leveraged by other organizations to build
> > > streaming applications that can benefit from its robustness, high
> > > performance, adaptability to cloud environments and ease of use.
> > >
> > > Moreover,
> > >
> > > we hope that open-sourcing Heron will help to further evolve the
> > >
> > > technology
> > >
> > > as the project attracts contributors with diverse backgrounds and areas
> > >
> > > of
> > >
> > > expertise.
> > >
> > > We believe the Apache foundation is a great fit as the long-term home
> > >
> > > for
> > >
> > > Heron, as it provides an established process for community-driven
> > > development and decision making by consensus. This is exactly the model
> > >
> > > we
> > >
> > > want for future Heron development.
> > >
> > > = Initial Goals =
> > >
> > > * Move the existing codebase, website, documentation, and mailing lists
> > >
> > > to
> > >
> > > Apache-hosted infrastructure.
> > > * Integrate with the Apache development process.
> > > * Ensure all dependencies are compliant with Apache License version
> > >
> > > 2.0.
> > >
> > > * Incrementally develop and release per Apache guidelines.
> > >
> > > = Current Status =
> > >
> > > Heron is a stable project used in production at Twitter since 2014 and
> > >
> > > open
> > >
> > > sourced under the ASL v2 license in 2016. The Heron source code is
> > > currently hosted at github.com (https://github.com/twitter/heron),
> > >
> > > which
> > >
> > > will seed the Apache git repository.
> > >
> > > = Meritocracy =
> > >
> > > By submitting this incubator proposal, we’re expressing our intent to
> > >
> > > build
> > >
> > > a diverse developer community around Heron that will conduct itself
> > > according to The Apache Way and use a meritocratic means of building
> > >
> > > it's
> > >
> > > committer base. Several companies and universities have already
> > >
> > > expressed
> > >
> > > interest in and contributed to Heron. Our goal is to grow the Heron
> > > community by encouraging open communication, contribution and
> > >
> > > participation
> > >
> > > of all types, and ensuring that contributors are recognized
> > >
> > > appropriately.
> > >
> > >
> > > = Community =
> > >
> > > Heron is currently being used by Twitter, Google, Machine Zone and
> > > ndustrial.io and has received significant contributions by Microsoft
> > >
> > > and
> > >
> > > Streamlio. By bringing Heron into the Apache ecosystem, we believe we
> > >
> > > can
> > >
> > > attract even more developers who are interested in creating real-time
> > > systems to build the project's contributor base.
> > >
> > > == Core Developers ==
> > >
> > > Current core developers are engineers from Twitter, Google, Microsoft
> > >
> > > and
> > >
> > > Streamlio.
> > >
> > > == Alignment ==
> > >
> > > Heron utilizes a number of Apache technologies. Heron leverages Apache
> > > ZooKeeper for coordination and has scheduler implementations to
> > >
> > > integrate
> > >
> > > with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache
> > >
> > > REEF)
> > >
> > > as well as spout implementations to integrate with Apache Kafka and
> > >
> > > metrics
> > >
> > > implementations to integrate with Scribe. Heron also implements the
> > >
> > > Apache
> > >
> > > Storm user-level API, which allows topologies written against Storm to
> > >
> > > run
> > >
> > > in Heron. We believe that having Heron at Apache will help further the
> > > growth of the streaming compute community, as well as encourage
> > >
> > > cooperation
> > >
> > > and developer cross pollination with other Apache projects.
> > >
> > > = Known Risks =
> > >
> > > == Orphaned Products ==
> > >
> > > The risk of the Heron project being abandoned is minimal. It is used in
> > > production at Twitter and Google and other companies are evaluating or
> > > adopting it for production use.
> > >
> > > == Inexperience with Open Source ==
> > >
> > > All of the core contributors to the project have considerable
> > >
> > > experience
> > >
> > > with open source software development. Bill Graham[2], Ashvin
> > >
> > > Agrawal[3]
> > >
> > > and Supun Kamburugamuve[4], committers on the project, are PMCs on
> > >
> > > other
> > >
> > > Apache projects and Bill and Ashvin have gone through the Apache
> > >
> > > incubator
> > >
> > > process. Twitter has already donated numerous projects to the ASF
> > >
> > > (e.g.,
> > >
> > > Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be
> > >
> > > mentored
> > >
> > > by experienced ASF members that can help with any roadblocks.
> > >
> > > == Homogenous Developers ==
> > >
> > > Initial committers come from 5 separate organizations. Our intention is
> > > increase the diversity of contributing developers and their
> > >
> > > affiliations.
> > >
> > > To date github contributions have come from approximately 50
> > >
> > > contributors
> > >
> > > from outside the Twitter team.
> > >
> > > == Reliance on Salaried Developers ==
> > >
> > > It is expected that Heron development will occur on both salaried time
> > >
> > > and
> > >
> > > on volunteer time. The majority of initial committers are paid by their
> > > employers to contribute to this project. We are committed to recruiting
> > > additional committers from other organizations as well as non-salaried
> > > committers to join project.
> > >
> > > == Relationships with Other Apache Products ==
> > >
> > > As mentioned in the Alignment section, Heron implements the Apache
> > >
> > > Storm
> > >
> > > API and integrates with multiple Apache schedulers (Apache Mesos,
> > >
> > > Apache
> > >
> > > Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache
> > > Thrift.
> > >
> > > == An Excessive Fascination with the Apache Brand ==
> > >
> > > Heron's popularity is growing in the streaming compute space and we are
> > > long time supporters of the Apache brand. This proposal is not for the
> > > purpose of generating publicity through. Rather, the primary benefits
> > >
> > > to
> > >
> > > joining Apache are those of community building and open decision making
> > > outlined in the Rationale section.
> > >
> > > == Documentation ==
> > >
> > > This proposal exists online as
> > > http://wiki.apache.org/incubator/HeronProposal. Extensive
> > >
> > > documentation
> > >
> > > can
> > >
> > > be found on github at https://twitter.github.io/heron and the source
> > >
> > > code
> > >
> > > is well documented.
> > >
> > > == Source and Intellectual Property Submission Plan ==
> > >
> > > The Heron codebase is currently hosted on Github:
> > > https://github.com/twitter/heron. During incubation, the codebase will
> > >
> > > be
> > >
> > > migrated to Apache infrastructure. The source code is already ASF 2.0
> > > licensed.
> > >
> > > == External Dependencies ==
> > >
> > > All external libraries have ASF 2.0 compatible licenses except for
> > >
> > > pylint.
> > >
> > > The pylint library is GPL licensed, but is only used for pre-build
> > >
> > > Python
> > >
> > > style checks and is neither bundled with, nor relied upon by, the Heron
> > > source or binary release artifacts.
> > >
> > > == Cryptography ==
> > >
> > > Heron does not use any cryptography libraries.
> > >
> > > = Required Resources =
> > >
> > > == Mailing lists ==
> > >
> > > * private@heron.incubator.apache.org <javascript:;> (with moderated
> > >
> > > subscriptions)
> > >
> > > * dev@heron.incubator.apache.org <javascript:;>
> > > * commits@heron.incubator.apache.org <javascript:;>
> > > * user@heron.incubator.apache.org <javascript:;>
> > >
> > >
> > > == Subversion Directory ==
> > >
> > > Git is the preferred source control system: git://git.apache.org/heron
> > >
> > > == Issue Tracking ==
> > >
> > > JIRA: Heron (HERON)
> > >
> > > == Initial Committers ==
> > >
> > > * Andrew Jorgensen (andrew at andrewjorgensen dot com)
> > > * Ashvin Agrawal (ashvin at apache dot org)*
> > > * Avrilia Floratou (avrilia dot floratou at gmail dot com)
> > > * Bill Graham (billgraham at apache dot org)*
> > > * Brian Hatfield (bmhatfield at gmail dot com)
> > > * Chris Kellogg (cckellogg at gmail dot com)
> > > * Huijun Wu (huijun dot wu dot 2010 at gmail dot com)
> > > * Karthik Ramasamy (karthik at gmail dot com)
> > > * Maosong Fu (maosongfu at gmail dot com)
> > > * Neng Lu(freeneng at gmail dot com)
> > > * Runhang Li (obj dot runhang at gmail dot com)
> > > * Sanjeev Kulkarni (sanjeevrk at gmail dot com)
> > > * Supun Kamburugamuve (supun at apache dot org)*
> > > * Thomas Sun (tom dot ssf at gmail dot com)
> > > * Yaliang Wang (yaliang dot w dot wang at ieee dot org)
> > >
> > > == Affiliations ==
> > >
> > > * Andrew Jorgensen (Google)
> > > * Ashvin Agrawal (Microsoft)
> > > * Avrilia Floratou (Microsoft)
> > > * Bill Graham (Twitter)
> > > * Brian Hatfield (Google)
> > > * Chris Kellogg (Twitter)
> > > * Huijun Wu (Twitter)
> > > * Karthik Ramasamy (Streamlio)
> > > * Maosong Fu (Twitter)
> > > * Neng Lu (Twitter)
> > > * Runhang Li (Twitter)
> > > * Sanjeev Kulkarni (Streamlio)
> > > * Supun Kamburugamuve (Indiana University)
> > > * Thomas Sun (Twitter)
> > > * Yaliang Wang (Twitter)
> > >
> > > = Sponsors =
> > >
> > > == Champion ==
> > >
> > > * Julien Le Dem (julien at apache dot org)
> > >
> > > == Nominated Mentors ==
> > >
> > > * Jake Farrell (jfarrell at apache dot org)
> > > * Jacques Nadeau (jacques at apache dot org)
> > > * Julien Le Dem (julien at apache dot org)
> > > * P. Taylor Goetz (ptgoetz at apache dot org)
> > >
> > > == Sponsoring Entity ==
> > >
> > > The Apache Incubator
> > >
> > > == Footnotes ==
> > >
> > > * 1 - Papers detailing Heron are available at
> > > http://dl.acm.org/citation.cfm?id=2742788 and
> > > http://sites.computer.org/debull/A15dec/p15.pdf.
> > > * 2 - http://home.apache.org/phonebook.html?uid=billgraham
> > > * 3 - http://home.apache.org/phonebook.html?uid=ashvin
> > > * 4 - http://home.apache.org/phonebook.html?uid=supun
> > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > >
> > > <javascript:;>
> > >
> > > For additional commands, e-mail: general-help@incubator.apache.org
> > >
> > > <javascript:;>
> > >
> > >
> > >
> > >
> > >
> > >
> > > --
> > > Sorry this was sent from mobile. Will do less grammar and spell check
> > than
> > > usual.
> > >
> > >
> > >
> >
>

Re: [VOTE] Heron to enter Apache Incubator

Posted by "John D. Ament" <jo...@apache.org>.
Bill,

Would I be correct in understanding that Heron implements the same protocol
as Storm, but the actual implementation is different?

John

On Fri, Jun 23, 2017 at 1:36 AM Bill Graham <bi...@gmail.com> wrote:

> It's grossly inaccurate to refer to Heron as a Storm fork. There are about
> 132k lines of code in the Heron codebase (plus 166k of codegen), of which
> about 7k are to implement the Apache Storm API bindings to the Heron API.
>
> The Rationale section of the proposal discusses the Heron architecture,
> which is a complete rewrite with little in common with Storm. The only
> overlap is that Heron supports the Storm user API for ease of migration.
>
> The value of having multiple projects to solve a common need is that each
> can foster innovation, collaboration and exchange of ideas in different
> ways. This is not a new concept to Apache. You can look at the incubator
> discussions around Accumulo vs HBase (two implementations of the BigTable
> paper) for example, to see how two different approaches to a shared problem
> can be a good thing.
>
> thanks,
> Bill
>
> On Thu, Jun 22, 2017 at 6:45 PM, Von Gosling <vo...@apache.org>
> wrote:
>
> > Hi,
> >
> > I will give +1(Non-binding), but,
> >
> > I have the similar question about so many streaming framework in the
> > apache, how to develop community for themselves.
> >
> >
> >
> >
> > Best Regards,
> > Von Gosling
> >
> >
> >
> > 在 2017年6月23日，08:51，Edward Capriolo <ed...@gmail.com> 写道：
> >
> > I believe heron and storm should be merged back together. I do not see
> the
> > value of storm and a storm fork in the asf.
> >
> > On Thursday, June 22, 2017, Bill Graham <bi...@gmail.com> wrote:
> >
> > Thanks Taylor for relaying these sentiments, especially the part about
> the
> > Heron website which is indeed poorly worded (I suspect this could have
> been
> > the result of internal docs being open-sourced). I've opened this pull
> > request to update the language regarding Storm:
> >
> > https://github.com/twitter/heron/pull/1979
> >
> > On Thu, Jun 22, 2017 at 12:21 PM, P. Taylor Goetz <ptgoetz@gmail.com
> > <javascript:;>> wrote:
> >
> > The Apache Storm PMC had a discussion regarding the Heron proposal. In
> >
> > the
> >
> > spirit of openness I wanted to bring some of the sentiments expressed in
> > that discussion back to this list. Please note that I am paraphrasing
> >
> > from
> >
> > that discussion and attempting to relay opinions of the collective PMC,
> >
> > not
> >
> > necessarily that of any individual.
> >
> > * There is a general disappointment that the Heron community chose not to
> > engage with the Storm community and instead chose a separate path.
> > * A majority of the PMC supports Heron’s incubation, though some felt it
> > would result in unnecessary duplication of effort.
> > * A majority of the PMC supports the two projects working closely
> > together. A number of PMC members suggested the two projects merge in
> >
> > some
> >
> > way.
> > * Many PMC members took issue some of the marketing language on the Heron
> > website, particularly Heron being billed as “the direct successor to
> >
> > Apache
> >
> > Storm” and the prominent “Upgrade from Storm” links.  The main concern
> >
> > here
> >
> > was such phrasing has somewhat of a hostile tone and undermines the
> >
> > desire
> >
> > for better collaboration, as well as confusing users.
> >
> > One of my goals as a proposed mentor for Heron and a Storm PMC member is
> > to address some of these concerns and encourage collaboration. As I
> > mentioned to the Storm PMC on that thread, if there are ongoing concerns
> > from either the Storm PMC or the Heron PPMC about me acting as a mentor,
> >
> > I
> >
> > would be willing to step down.
> >
> > +1 (binding)
> >
> > -Taylor
> >
> > On Jun 16, 2017, at 4:41 PM, Bill Graham <billgraham@gmail.com
> >
> > <javascript:;>> wrote:
> >
> >
> > Hi,
> >
> > Based on the discussion on the incubator mailing list[1] I would like
> >
> > to
> >
> > call a vote to add Heron to the Apache Incubator.
> >
> > The full proposal is available below, and is also available on the
> >
> > Apache
> >
> > Incubator wiki at:
> >   https://wiki.apache.org/incubator/HeronProposal
> >
> > Please vote:
> > [ ] +1, bring Heron into Incubator
> > [ ] -1, do not bring Heron into Incubator, because...
> >
> > The vote will open for 7 days until Friday June 23 at 14:00 PT.
> >
> > Thank you
> >
> > 1 -
> > https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7
> >
> > 786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E
> >
> >
> >
> >
> > = Heron Proposal =
> >
> > = Abstract =
> > Heron is a real-time, distributed, fault-tolerant stream processing
> >
> > engine
> >
> > initially developed by Twitter.
> >
> > = Proposal =
> >
> > Heron is a real-time stream processing engine built for high
> >
> > performance,
> >
> > ease of manageability, performance predictability and developer
> > productivity[1]. We wish to develop a community around Heron to
> >
> > increase
> >
> > contributions and see Heron thrive in an open forum.
> >
> > = Background =
> >
> > Heron provides the ability for developers to compose directed acyclic
> > graphs (DAGs) of real-time query execution logic (i.e. a topology) and
> > submit the topology to execute on a pluggable job scheduling system
> >
> > (e.g.,
> >
> > Apache Aurora, YARN, Marathon, etc). Users can employ either the native
> > Heron API or the Apache Storm API to develop the topology. Heron
> >
> > supports
> >
> > the Storm API for ease of migration, but beyond that Heron’s
> >
> > architecture
> >
> > differs considerably from Storm’s.
> >
> > Users submit a topology to the scheduler using the Heron client, which
> >
> > uses
> >
> > the Heron binary libraries to deploy all daemons required to run and
> >
> > manage
> >
> > the topology. The topology therefore has no reliance on centrally
> >
> > managed
> >
> > Heron services, only on a generic job scheduling system, which lends
> >
> > itself
> >
> > well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN
> >
> > (among
> >
> > others).
> >
> > The scheduler runs each topology as a job consisting of multiple
> > containers. One of the containers runs the topology master, responsible
> >
> > for
> >
> > managing the topology. The remaining containers each runs a stream
> >
> > manager
> >
> > responsible for data routing, a metrics manager that collects and
> >
> > reports
> >
> > various metrics and a number of processes called Heron instances which
> >
> > run
> >
> > the user-defined logic on the stream of tuples. Parallelism is achieved
> >
> > via
> >
> > process-based isolation of Heron instances, which provides predictable
> > performance while simplifying debugging. The containers are allocated
> >
> > and
> >
> > managed by the scheduler framework based on resource availability of
> >
> > nodes
> >
> > in the cluster. The metadata for the topology, such as the physical
> >
> > plan
> >
> > and execution details, are stored in the pluggable Heron State Manager
> > (e.g. Apache ZooKeeper).
> >
> > = Rationale =
> >
> > Heron is a general-purpose, modular and extensible platform that can be
> > leveraged to support common, real-time analytics use cases. There is an
> > increasing demand for open-source, scalable real-time analytics
> >
> > systems.
> >
> > We
> >
> > believe that Heron can be leveraged by other organizations to build
> > streaming applications that can benefit from its robustness, high
> > performance, adaptability to cloud environments and ease of use.
> >
> > Moreover,
> >
> > we hope that open-sourcing Heron will help to further evolve the
> >
> > technology
> >
> > as the project attracts contributors with diverse backgrounds and areas
> >
> > of
> >
> > expertise.
> >
> > We believe the Apache foundation is a great fit as the long-term home
> >
> > for
> >
> > Heron, as it provides an established process for community-driven
> > development and decision making by consensus. This is exactly the model
> >
> > we
> >
> > want for future Heron development.
> >
> > = Initial Goals =
> >
> > * Move the existing codebase, website, documentation, and mailing lists
> >
> > to
> >
> > Apache-hosted infrastructure.
> > * Integrate with the Apache development process.
> > * Ensure all dependencies are compliant with Apache License version
> >
> > 2.0.
> >
> > * Incrementally develop and release per Apache guidelines.
> >
> > = Current Status =
> >
> > Heron is a stable project used in production at Twitter since 2014 and
> >
> > open
> >
> > sourced under the ASL v2 license in 2016. The Heron source code is
> > currently hosted at github.com (https://github.com/twitter/heron),
> >
> > which
> >
> > will seed the Apache git repository.
> >
> > = Meritocracy =
> >
> > By submitting this incubator proposal, we’re expressing our intent to
> >
> > build
> >
> > a diverse developer community around Heron that will conduct itself
> > according to The Apache Way and use a meritocratic means of building
> >
> > it's
> >
> > committer base. Several companies and universities have already
> >
> > expressed
> >
> > interest in and contributed to Heron. Our goal is to grow the Heron
> > community by encouraging open communication, contribution and
> >
> > participation
> >
> > of all types, and ensuring that contributors are recognized
> >
> > appropriately.
> >
> >
> > = Community =
> >
> > Heron is currently being used by Twitter, Google, Machine Zone and
> > ndustrial.io and has received significant contributions by Microsoft
> >
> > and
> >
> > Streamlio. By bringing Heron into the Apache ecosystem, we believe we
> >
> > can
> >
> > attract even more developers who are interested in creating real-time
> > systems to build the project's contributor base.
> >
> > == Core Developers ==
> >
> > Current core developers are engineers from Twitter, Google, Microsoft
> >
> > and
> >
> > Streamlio.
> >
> > == Alignment ==
> >
> > Heron utilizes a number of Apache technologies. Heron leverages Apache
> > ZooKeeper for coordination and has scheduler implementations to
> >
> > integrate
> >
> > with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache
> >
> > REEF)
> >
> > as well as spout implementations to integrate with Apache Kafka and
> >
> > metrics
> >
> > implementations to integrate with Scribe. Heron also implements the
> >
> > Apache
> >
> > Storm user-level API, which allows topologies written against Storm to
> >
> > run
> >
> > in Heron. We believe that having Heron at Apache will help further the
> > growth of the streaming compute community, as well as encourage
> >
> > cooperation
> >
> > and developer cross pollination with other Apache projects.
> >
> > = Known Risks =
> >
> > == Orphaned Products ==
> >
> > The risk of the Heron project being abandoned is minimal. It is used in
> > production at Twitter and Google and other companies are evaluating or
> > adopting it for production use.
> >
> > == Inexperience with Open Source ==
> >
> > All of the core contributors to the project have considerable
> >
> > experience
> >
> > with open source software development. Bill Graham[2], Ashvin
> >
> > Agrawal[3]
> >
> > and Supun Kamburugamuve[4], committers on the project, are PMCs on
> >
> > other
> >
> > Apache projects and Bill and Ashvin have gone through the Apache
> >
> > incubator
> >
> > process. Twitter has already donated numerous projects to the ASF
> >
> > (e.g.,
> >
> > Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be
> >
> > mentored
> >
> > by experienced ASF members that can help with any roadblocks.
> >
> > == Homogenous Developers ==
> >
> > Initial committers come from 5 separate organizations. Our intention is
> > increase the diversity of contributing developers and their
> >
> > affiliations.
> >
> > To date github contributions have come from approximately 50
> >
> > contributors
> >
> > from outside the Twitter team.
> >
> > == Reliance on Salaried Developers ==
> >
> > It is expected that Heron development will occur on both salaried time
> >
> > and
> >
> > on volunteer time. The majority of initial committers are paid by their
> > employers to contribute to this project. We are committed to recruiting
> > additional committers from other organizations as well as non-salaried
> > committers to join project.
> >
> > == Relationships with Other Apache Products ==
> >
> > As mentioned in the Alignment section, Heron implements the Apache
> >
> > Storm
> >
> > API and integrates with multiple Apache schedulers (Apache Mesos,
> >
> > Apache
> >
> > Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache
> > Thrift.
> >
> > == An Excessive Fascination with the Apache Brand ==
> >
> > Heron's popularity is growing in the streaming compute space and we are
> > long time supporters of the Apache brand. This proposal is not for the
> > purpose of generating publicity through. Rather, the primary benefits
> >
> > to
> >
> > joining Apache are those of community building and open decision making
> > outlined in the Rationale section.
> >
> > == Documentation ==
> >
> > This proposal exists online as
> > http://wiki.apache.org/incubator/HeronProposal. Extensive
> >
> > documentation
> >
> > can
> >
> > be found on github at https://twitter.github.io/heron and the source
> >
> > code
> >
> > is well documented.
> >
> > == Source and Intellectual Property Submission Plan ==
> >
> > The Heron codebase is currently hosted on Github:
> > https://github.com/twitter/heron. During incubation, the codebase will
> >
> > be
> >
> > migrated to Apache infrastructure. The source code is already ASF 2.0
> > licensed.
> >
> > == External Dependencies ==
> >
> > All external libraries have ASF 2.0 compatible licenses except for
> >
> > pylint.
> >
> > The pylint library is GPL licensed, but is only used for pre-build
> >
> > Python
> >
> > style checks and is neither bundled with, nor relied upon by, the Heron
> > source or binary release artifacts.
> >
> > == Cryptography ==
> >
> > Heron does not use any cryptography libraries.
> >
> > = Required Resources =
> >
> > == Mailing lists ==
> >
> > * private@heron.incubator.apache.org <javascript:;> (with moderated
> >
> > subscriptions)
> >
> > * dev@heron.incubator.apache.org <javascript:;>
> > * commits@heron.incubator.apache.org <javascript:;>
> > * user@heron.incubator.apache.org <javascript:;>
> >
> >
> > == Subversion Directory ==
> >
> > Git is the preferred source control system: git://git.apache.org/heron
> >
> > == Issue Tracking ==
> >
> > JIRA: Heron (HERON)
> >
> > == Initial Committers ==
> >
> > * Andrew Jorgensen (andrew at andrewjorgensen dot com)
> > * Ashvin Agrawal (ashvin at apache dot org)*
> > * Avrilia Floratou (avrilia dot floratou at gmail dot com)
> > * Bill Graham (billgraham at apache dot org)*
> > * Brian Hatfield (bmhatfield at gmail dot com)
> > * Chris Kellogg (cckellogg at gmail dot com)
> > * Huijun Wu (huijun dot wu dot 2010 at gmail dot com)
> > * Karthik Ramasamy (karthik at gmail dot com)
> > * Maosong Fu (maosongfu at gmail dot com)
> > * Neng Lu(freeneng at gmail dot com)
> > * Runhang Li (obj dot runhang at gmail dot com)
> > * Sanjeev Kulkarni (sanjeevrk at gmail dot com)
> > * Supun Kamburugamuve (supun at apache dot org)*
> > * Thomas Sun (tom dot ssf at gmail dot com)
> > * Yaliang Wang (yaliang dot w dot wang at ieee dot org)
> >
> > == Affiliations ==
> >
> > * Andrew Jorgensen (Google)
> > * Ashvin Agrawal (Microsoft)
> > * Avrilia Floratou (Microsoft)
> > * Bill Graham (Twitter)
> > * Brian Hatfield (Google)
> > * Chris Kellogg (Twitter)
> > * Huijun Wu (Twitter)
> > * Karthik Ramasamy (Streamlio)
> > * Maosong Fu (Twitter)
> > * Neng Lu (Twitter)
> > * Runhang Li (Twitter)
> > * Sanjeev Kulkarni (Streamlio)
> > * Supun Kamburugamuve (Indiana University)
> > * Thomas Sun (Twitter)
> > * Yaliang Wang (Twitter)
> >
> > = Sponsors =
> >
> > == Champion ==
> >
> > * Julien Le Dem (julien at apache dot org)
> >
> > == Nominated Mentors ==
> >
> > * Jake Farrell (jfarrell at apache dot org)
> > * Jacques Nadeau (jacques at apache dot org)
> > * Julien Le Dem (julien at apache dot org)
> > * P. Taylor Goetz (ptgoetz at apache dot org)
> >
> > == Sponsoring Entity ==
> >
> > The Apache Incubator
> >
> > == Footnotes ==
> >
> > * 1 - Papers detailing Heron are available at
> > http://dl.acm.org/citation.cfm?id=2742788 and
> > http://sites.computer.org/debull/A15dec/p15.pdf.
> > * 2 - http://home.apache.org/phonebook.html?uid=billgraham
> > * 3 - http://home.apache.org/phonebook.html?uid=ashvin
> > * 4 - http://home.apache.org/phonebook.html?uid=supun
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> >
> > <javascript:;>
> >
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
> > <javascript:;>
> >
> >
> >
> >
> >
> >
> > --
> > Sorry this was sent from mobile. Will do less grammar and spell check
> than
> > usual.
> >
> >
> >
>

Re: [VOTE] Heron to enter Apache Incubator

Posted by Bill Graham <bi...@gmail.com>.
It's grossly inaccurate to refer to Heron as a Storm fork. There are about
132k lines of code in the Heron codebase (plus 166k of codegen), of which
about 7k are to implement the Apache Storm API bindings to the Heron API.

The Rationale section of the proposal discusses the Heron architecture,
which is a complete rewrite with little in common with Storm. The only
overlap is that Heron supports the Storm user API for ease of migration.

The value of having multiple projects to solve a common need is that each
can foster innovation, collaboration and exchange of ideas in different
ways. This is not a new concept to Apache. You can look at the incubator
discussions around Accumulo vs HBase (two implementations of the BigTable
paper) for example, to see how two different approaches to a shared problem
can be a good thing.

thanks,
Bill

On Thu, Jun 22, 2017 at 6:45 PM, Von Gosling <vo...@apache.org> wrote:

> Hi,
>
> I will give +1(Non-binding), but,
>
> I have the similar question about so many streaming framework in the
> apache, how to develop community for themselves.
>
>
>
>
> Best Regards,
> Von Gosling
>
>
>
> 在 2017年6月23日，08:51，Edward Capriolo <ed...@gmail.com> 写道：
>
> I believe heron and storm should be merged back together. I do not see the
> value of storm and a storm fork in the asf.
>
> On Thursday, June 22, 2017, Bill Graham <bi...@gmail.com> wrote:
>
> Thanks Taylor for relaying these sentiments, especially the part about the
> Heron website which is indeed poorly worded (I suspect this could have been
> the result of internal docs being open-sourced). I've opened this pull
> request to update the language regarding Storm:
>
> https://github.com/twitter/heron/pull/1979
>
> On Thu, Jun 22, 2017 at 12:21 PM, P. Taylor Goetz <ptgoetz@gmail.com
> <javascript:;>> wrote:
>
> The Apache Storm PMC had a discussion regarding the Heron proposal. In
>
> the
>
> spirit of openness I wanted to bring some of the sentiments expressed in
> that discussion back to this list. Please note that I am paraphrasing
>
> from
>
> that discussion and attempting to relay opinions of the collective PMC,
>
> not
>
> necessarily that of any individual.
>
> * There is a general disappointment that the Heron community chose not to
> engage with the Storm community and instead chose a separate path.
> * A majority of the PMC supports Heron’s incubation, though some felt it
> would result in unnecessary duplication of effort.
> * A majority of the PMC supports the two projects working closely
> together. A number of PMC members suggested the two projects merge in
>
> some
>
> way.
> * Many PMC members took issue some of the marketing language on the Heron
> website, particularly Heron being billed as “the direct successor to
>
> Apache
>
> Storm” and the prominent “Upgrade from Storm” links.  The main concern
>
> here
>
> was such phrasing has somewhat of a hostile tone and undermines the
>
> desire
>
> for better collaboration, as well as confusing users.
>
> One of my goals as a proposed mentor for Heron and a Storm PMC member is
> to address some of these concerns and encourage collaboration. As I
> mentioned to the Storm PMC on that thread, if there are ongoing concerns
> from either the Storm PMC or the Heron PPMC about me acting as a mentor,
>
> I
>
> would be willing to step down.
>
> +1 (binding)
>
> -Taylor
>
> On Jun 16, 2017, at 4:41 PM, Bill Graham <billgraham@gmail.com
>
> <javascript:;>> wrote:
>
>
> Hi,
>
> Based on the discussion on the incubator mailing list[1] I would like
>
> to
>
> call a vote to add Heron to the Apache Incubator.
>
> The full proposal is available below, and is also available on the
>
> Apache
>
> Incubator wiki at:
>   https://wiki.apache.org/incubator/HeronProposal
>
> Please vote:
> [ ] +1, bring Heron into Incubator
> [ ] -1, do not bring Heron into Incubator, because...
>
> The vote will open for 7 days until Friday June 23 at 14:00 PT.
>
> Thank you
>
> 1 -
> https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7
>
> 786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E
>
>
>
>
> = Heron Proposal =
>
> = Abstract =
> Heron is a real-time, distributed, fault-tolerant stream processing
>
> engine
>
> initially developed by Twitter.
>
> = Proposal =
>
> Heron is a real-time stream processing engine built for high
>
> performance,
>
> ease of manageability, performance predictability and developer
> productivity[1]. We wish to develop a community around Heron to
>
> increase
>
> contributions and see Heron thrive in an open forum.
>
> = Background =
>
> Heron provides the ability for developers to compose directed acyclic
> graphs (DAGs) of real-time query execution logic (i.e. a topology) and
> submit the topology to execute on a pluggable job scheduling system
>
> (e.g.,
>
> Apache Aurora, YARN, Marathon, etc). Users can employ either the native
> Heron API or the Apache Storm API to develop the topology. Heron
>
> supports
>
> the Storm API for ease of migration, but beyond that Heron’s
>
> architecture
>
> differs considerably from Storm’s.
>
> Users submit a topology to the scheduler using the Heron client, which
>
> uses
>
> the Heron binary libraries to deploy all daemons required to run and
>
> manage
>
> the topology. The topology therefore has no reliance on centrally
>
> managed
>
> Heron services, only on a generic job scheduling system, which lends
>
> itself
>
> well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN
>
> (among
>
> others).
>
> The scheduler runs each topology as a job consisting of multiple
> containers. One of the containers runs the topology master, responsible
>
> for
>
> managing the topology. The remaining containers each runs a stream
>
> manager
>
> responsible for data routing, a metrics manager that collects and
>
> reports
>
> various metrics and a number of processes called Heron instances which
>
> run
>
> the user-defined logic on the stream of tuples. Parallelism is achieved
>
> via
>
> process-based isolation of Heron instances, which provides predictable
> performance while simplifying debugging. The containers are allocated
>
> and
>
> managed by the scheduler framework based on resource availability of
>
> nodes
>
> in the cluster. The metadata for the topology, such as the physical
>
> plan
>
> and execution details, are stored in the pluggable Heron State Manager
> (e.g. Apache ZooKeeper).
>
> = Rationale =
>
> Heron is a general-purpose, modular and extensible platform that can be
> leveraged to support common, real-time analytics use cases. There is an
> increasing demand for open-source, scalable real-time analytics
>
> systems.
>
> We
>
> believe that Heron can be leveraged by other organizations to build
> streaming applications that can benefit from its robustness, high
> performance, adaptability to cloud environments and ease of use.
>
> Moreover,
>
> we hope that open-sourcing Heron will help to further evolve the
>
> technology
>
> as the project attracts contributors with diverse backgrounds and areas
>
> of
>
> expertise.
>
> We believe the Apache foundation is a great fit as the long-term home
>
> for
>
> Heron, as it provides an established process for community-driven
> development and decision making by consensus. This is exactly the model
>
> we
>
> want for future Heron development.
>
> = Initial Goals =
>
> * Move the existing codebase, website, documentation, and mailing lists
>
> to
>
> Apache-hosted infrastructure.
> * Integrate with the Apache development process.
> * Ensure all dependencies are compliant with Apache License version
>
> 2.0.
>
> * Incrementally develop and release per Apache guidelines.
>
> = Current Status =
>
> Heron is a stable project used in production at Twitter since 2014 and
>
> open
>
> sourced under the ASL v2 license in 2016. The Heron source code is
> currently hosted at github.com (https://github.com/twitter/heron),
>
> which
>
> will seed the Apache git repository.
>
> = Meritocracy =
>
> By submitting this incubator proposal, we’re expressing our intent to
>
> build
>
> a diverse developer community around Heron that will conduct itself
> according to The Apache Way and use a meritocratic means of building
>
> it's
>
> committer base. Several companies and universities have already
>
> expressed
>
> interest in and contributed to Heron. Our goal is to grow the Heron
> community by encouraging open communication, contribution and
>
> participation
>
> of all types, and ensuring that contributors are recognized
>
> appropriately.
>
>
> = Community =
>
> Heron is currently being used by Twitter, Google, Machine Zone and
> ndustrial.io and has received significant contributions by Microsoft
>
> and
>
> Streamlio. By bringing Heron into the Apache ecosystem, we believe we
>
> can
>
> attract even more developers who are interested in creating real-time
> systems to build the project's contributor base.
>
> == Core Developers ==
>
> Current core developers are engineers from Twitter, Google, Microsoft
>
> and
>
> Streamlio.
>
> == Alignment ==
>
> Heron utilizes a number of Apache technologies. Heron leverages Apache
> ZooKeeper for coordination and has scheduler implementations to
>
> integrate
>
> with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache
>
> REEF)
>
> as well as spout implementations to integrate with Apache Kafka and
>
> metrics
>
> implementations to integrate with Scribe. Heron also implements the
>
> Apache
>
> Storm user-level API, which allows topologies written against Storm to
>
> run
>
> in Heron. We believe that having Heron at Apache will help further the
> growth of the streaming compute community, as well as encourage
>
> cooperation
>
> and developer cross pollination with other Apache projects.
>
> = Known Risks =
>
> == Orphaned Products ==
>
> The risk of the Heron project being abandoned is minimal. It is used in
> production at Twitter and Google and other companies are evaluating or
> adopting it for production use.
>
> == Inexperience with Open Source ==
>
> All of the core contributors to the project have considerable
>
> experience
>
> with open source software development. Bill Graham[2], Ashvin
>
> Agrawal[3]
>
> and Supun Kamburugamuve[4], committers on the project, are PMCs on
>
> other
>
> Apache projects and Bill and Ashvin have gone through the Apache
>
> incubator
>
> process. Twitter has already donated numerous projects to the ASF
>
> (e.g.,
>
> Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be
>
> mentored
>
> by experienced ASF members that can help with any roadblocks.
>
> == Homogenous Developers ==
>
> Initial committers come from 5 separate organizations. Our intention is
> increase the diversity of contributing developers and their
>
> affiliations.
>
> To date github contributions have come from approximately 50
>
> contributors
>
> from outside the Twitter team.
>
> == Reliance on Salaried Developers ==
>
> It is expected that Heron development will occur on both salaried time
>
> and
>
> on volunteer time. The majority of initial committers are paid by their
> employers to contribute to this project. We are committed to recruiting
> additional committers from other organizations as well as non-salaried
> committers to join project.
>
> == Relationships with Other Apache Products ==
>
> As mentioned in the Alignment section, Heron implements the Apache
>
> Storm
>
> API and integrates with multiple Apache schedulers (Apache Mesos,
>
> Apache
>
> Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache
> Thrift.
>
> == An Excessive Fascination with the Apache Brand ==
>
> Heron's popularity is growing in the streaming compute space and we are
> long time supporters of the Apache brand. This proposal is not for the
> purpose of generating publicity through. Rather, the primary benefits
>
> to
>
> joining Apache are those of community building and open decision making
> outlined in the Rationale section.
>
> == Documentation ==
>
> This proposal exists online as
> http://wiki.apache.org/incubator/HeronProposal. Extensive
>
> documentation
>
> can
>
> be found on github at https://twitter.github.io/heron and the source
>
> code
>
> is well documented.
>
> == Source and Intellectual Property Submission Plan ==
>
> The Heron codebase is currently hosted on Github:
> https://github.com/twitter/heron. During incubation, the codebase will
>
> be
>
> migrated to Apache infrastructure. The source code is already ASF 2.0
> licensed.
>
> == External Dependencies ==
>
> All external libraries have ASF 2.0 compatible licenses except for
>
> pylint.
>
> The pylint library is GPL licensed, but is only used for pre-build
>
> Python
>
> style checks and is neither bundled with, nor relied upon by, the Heron
> source or binary release artifacts.
>
> == Cryptography ==
>
> Heron does not use any cryptography libraries.
>
> = Required Resources =
>
> == Mailing lists ==
>
> * private@heron.incubator.apache.org <javascript:;> (with moderated
>
> subscriptions)
>
> * dev@heron.incubator.apache.org <javascript:;>
> * commits@heron.incubator.apache.org <javascript:;>
> * user@heron.incubator.apache.org <javascript:;>
>
>
> == Subversion Directory ==
>
> Git is the preferred source control system: git://git.apache.org/heron
>
> == Issue Tracking ==
>
> JIRA: Heron (HERON)
>
> == Initial Committers ==
>
> * Andrew Jorgensen (andrew at andrewjorgensen dot com)
> * Ashvin Agrawal (ashvin at apache dot org)*
> * Avrilia Floratou (avrilia dot floratou at gmail dot com)
> * Bill Graham (billgraham at apache dot org)*
> * Brian Hatfield (bmhatfield at gmail dot com)
> * Chris Kellogg (cckellogg at gmail dot com)
> * Huijun Wu (huijun dot wu dot 2010 at gmail dot com)
> * Karthik Ramasamy (karthik at gmail dot com)
> * Maosong Fu (maosongfu at gmail dot com)
> * Neng Lu(freeneng at gmail dot com)
> * Runhang Li (obj dot runhang at gmail dot com)
> * Sanjeev Kulkarni (sanjeevrk at gmail dot com)
> * Supun Kamburugamuve (supun at apache dot org)*
> * Thomas Sun (tom dot ssf at gmail dot com)
> * Yaliang Wang (yaliang dot w dot wang at ieee dot org)
>
> == Affiliations ==
>
> * Andrew Jorgensen (Google)
> * Ashvin Agrawal (Microsoft)
> * Avrilia Floratou (Microsoft)
> * Bill Graham (Twitter)
> * Brian Hatfield (Google)
> * Chris Kellogg (Twitter)
> * Huijun Wu (Twitter)
> * Karthik Ramasamy (Streamlio)
> * Maosong Fu (Twitter)
> * Neng Lu (Twitter)
> * Runhang Li (Twitter)
> * Sanjeev Kulkarni (Streamlio)
> * Supun Kamburugamuve (Indiana University)
> * Thomas Sun (Twitter)
> * Yaliang Wang (Twitter)
>
> = Sponsors =
>
> == Champion ==
>
> * Julien Le Dem (julien at apache dot org)
>
> == Nominated Mentors ==
>
> * Jake Farrell (jfarrell at apache dot org)
> * Jacques Nadeau (jacques at apache dot org)
> * Julien Le Dem (julien at apache dot org)
> * P. Taylor Goetz (ptgoetz at apache dot org)
>
> == Sponsoring Entity ==
>
> The Apache Incubator
>
> == Footnotes ==
>
> * 1 - Papers detailing Heron are available at
> http://dl.acm.org/citation.cfm?id=2742788 and
> http://sites.computer.org/debull/A15dec/p15.pdf.
> * 2 - http://home.apache.org/phonebook.html?uid=billgraham
> * 3 - http://home.apache.org/phonebook.html?uid=ashvin
> * 4 - http://home.apache.org/phonebook.html?uid=supun
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>
> <javascript:;>
>
> For additional commands, e-mail: general-help@incubator.apache.org
>
> <javascript:;>
>
>
>
>
>
>
> --
> Sorry this was sent from mobile. Will do less grammar and spell check than
> usual.
>
>
>

Re: [VOTE] Heron to enter Apache Incubator

Posted by Von Gosling <vo...@apache.org>.
Hi,

I will give +1(Non-binding), but,

I have the similar question about so many streaming framework in the apache, how to develop community for themselves. 




Best Regards,
Von Gosling



> 在 2017年6月23日，08:51，Edward Capriolo <ed...@gmail.com> 写道：
> 
> I believe heron and storm should be merged back together. I do not see the
> value of storm and a storm fork in the asf.
> 
> On Thursday, June 22, 2017, Bill Graham <billgraham@gmail.com <ma...@gmail.com>> wrote:
> 
>> Thanks Taylor for relaying these sentiments, especially the part about the
>> Heron website which is indeed poorly worded (I suspect this could have been
>> the result of internal docs being open-sourced). I've opened this pull
>> request to update the language regarding Storm:
>> 
>> https://github.com/twitter/heron/pull/1979
>> 
>> On Thu, Jun 22, 2017 at 12:21 PM, P. Taylor Goetz <ptgoetz@gmail.com
>> <javascript:;>> wrote:
>> 
>>> The Apache Storm PMC had a discussion regarding the Heron proposal. In
>> the
>>> spirit of openness I wanted to bring some of the sentiments expressed in
>>> that discussion back to this list. Please note that I am paraphrasing
>> from
>>> that discussion and attempting to relay opinions of the collective PMC,
>> not
>>> necessarily that of any individual.
>>> 
>>> * There is a general disappointment that the Heron community chose not to
>>> engage with the Storm community and instead chose a separate path.
>>> * A majority of the PMC supports Heron’s incubation, though some felt it
>>> would result in unnecessary duplication of effort.
>>> * A majority of the PMC supports the two projects working closely
>>> together. A number of PMC members suggested the two projects merge in
>> some
>>> way.
>>> * Many PMC members took issue some of the marketing language on the Heron
>>> website, particularly Heron being billed as “the direct successor to
>> Apache
>>> Storm” and the prominent “Upgrade from Storm” links.  The main concern
>> here
>>> was such phrasing has somewhat of a hostile tone and undermines the
>> desire
>>> for better collaboration, as well as confusing users.
>>> 
>>> One of my goals as a proposed mentor for Heron and a Storm PMC member is
>>> to address some of these concerns and encourage collaboration. As I
>>> mentioned to the Storm PMC on that thread, if there are ongoing concerns
>>> from either the Storm PMC or the Heron PPMC about me acting as a mentor,
>> I
>>> would be willing to step down.
>>> 
>>> +1 (binding)
>>> 
>>> -Taylor
>>> 
>>>> On Jun 16, 2017, at 4:41 PM, Bill Graham <billgraham@gmail.com
>> <javascript:;>> wrote:
>>>> 
>>>> Hi,
>>>> 
>>>> Based on the discussion on the incubator mailing list[1] I would like
>> to
>>>> call a vote to add Heron to the Apache Incubator.
>>>> 
>>>> The full proposal is available below, and is also available on the
>> Apache
>>>> Incubator wiki at:
>>>>   https://wiki.apache.org/incubator/HeronProposal
>>>> 
>>>> Please vote:
>>>> [ ] +1, bring Heron into Incubator
>>>> [ ] -1, do not bring Heron into Incubator, because...
>>>> 
>>>> The vote will open for 7 days until Friday June 23 at 14:00 PT.
>>>> 
>>>> Thank you
>>>> 
>>>> 1 -
>>>> https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7
>>> 786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E
>>>> 
>>>> 
>>>> 
>>>> = Heron Proposal =
>>>> 
>>>> = Abstract =
>>>> Heron is a real-time, distributed, fault-tolerant stream processing
>>> engine
>>>> initially developed by Twitter.
>>>> 
>>>> = Proposal =
>>>> 
>>>> Heron is a real-time stream processing engine built for high
>> performance,
>>>> ease of manageability, performance predictability and developer
>>>> productivity[1]. We wish to develop a community around Heron to
>> increase
>>>> contributions and see Heron thrive in an open forum.
>>>> 
>>>> = Background =
>>>> 
>>>> Heron provides the ability for developers to compose directed acyclic
>>>> graphs (DAGs) of real-time query execution logic (i.e. a topology) and
>>>> submit the topology to execute on a pluggable job scheduling system
>>> (e.g.,
>>>> Apache Aurora, YARN, Marathon, etc). Users can employ either the native
>>>> Heron API or the Apache Storm API to develop the topology. Heron
>> supports
>>>> the Storm API for ease of migration, but beyond that Heron’s
>> architecture
>>>> differs considerably from Storm’s.
>>>> 
>>>> Users submit a topology to the scheduler using the Heron client, which
>>> uses
>>>> the Heron binary libraries to deploy all daemons required to run and
>>> manage
>>>> the topology. The topology therefore has no reliance on centrally
>> managed
>>>> Heron services, only on a generic job scheduling system, which lends
>>> itself
>>>> well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN
>> (among
>>>> others).
>>>> 
>>>> The scheduler runs each topology as a job consisting of multiple
>>>> containers. One of the containers runs the topology master, responsible
>>> for
>>>> managing the topology. The remaining containers each runs a stream
>>> manager
>>>> responsible for data routing, a metrics manager that collects and
>> reports
>>>> various metrics and a number of processes called Heron instances which
>>> run
>>>> the user-defined logic on the stream of tuples. Parallelism is achieved
>>> via
>>>> process-based isolation of Heron instances, which provides predictable
>>>> performance while simplifying debugging. The containers are allocated
>> and
>>>> managed by the scheduler framework based on resource availability of
>>> nodes
>>>> in the cluster. The metadata for the topology, such as the physical
>> plan
>>>> and execution details, are stored in the pluggable Heron State Manager
>>>> (e.g. Apache ZooKeeper).
>>>> 
>>>> = Rationale =
>>>> 
>>>> Heron is a general-purpose, modular and extensible platform that can be
>>>> leveraged to support common, real-time analytics use cases. There is an
>>>> increasing demand for open-source, scalable real-time analytics
>> systems.
>>> We
>>>> believe that Heron can be leveraged by other organizations to build
>>>> streaming applications that can benefit from its robustness, high
>>>> performance, adaptability to cloud environments and ease of use.
>>> Moreover,
>>>> we hope that open-sourcing Heron will help to further evolve the
>>> technology
>>>> as the project attracts contributors with diverse backgrounds and areas
>>> of
>>>> expertise.
>>>> 
>>>> We believe the Apache foundation is a great fit as the long-term home
>> for
>>>> Heron, as it provides an established process for community-driven
>>>> development and decision making by consensus. This is exactly the model
>>> we
>>>> want for future Heron development.
>>>> 
>>>> = Initial Goals =
>>>> 
>>>> * Move the existing codebase, website, documentation, and mailing lists
>>> to
>>>> Apache-hosted infrastructure.
>>>> * Integrate with the Apache development process.
>>>> * Ensure all dependencies are compliant with Apache License version
>> 2.0.
>>>> * Incrementally develop and release per Apache guidelines.
>>>> 
>>>> = Current Status =
>>>> 
>>>> Heron is a stable project used in production at Twitter since 2014 and
>>> open
>>>> sourced under the ASL v2 license in 2016. The Heron source code is
>>>> currently hosted at github.com (https://github.com/twitter/heron),
>> which
>>>> will seed the Apache git repository.
>>>> 
>>>> = Meritocracy =
>>>> 
>>>> By submitting this incubator proposal, we’re expressing our intent to
>>> build
>>>> a diverse developer community around Heron that will conduct itself
>>>> according to The Apache Way and use a meritocratic means of building
>> it's
>>>> committer base. Several companies and universities have already
>> expressed
>>>> interest in and contributed to Heron. Our goal is to grow the Heron
>>>> community by encouraging open communication, contribution and
>>> participation
>>>> of all types, and ensuring that contributors are recognized
>>> appropriately.
>>>> 
>>>> = Community =
>>>> 
>>>> Heron is currently being used by Twitter, Google, Machine Zone and
>>>> ndustrial.io and has received significant contributions by Microsoft
>> and
>>>> Streamlio. By bringing Heron into the Apache ecosystem, we believe we
>> can
>>>> attract even more developers who are interested in creating real-time
>>>> systems to build the project's contributor base.
>>>> 
>>>> == Core Developers ==
>>>> 
>>>> Current core developers are engineers from Twitter, Google, Microsoft
>> and
>>>> Streamlio.
>>>> 
>>>> == Alignment ==
>>>> 
>>>> Heron utilizes a number of Apache technologies. Heron leverages Apache
>>>> ZooKeeper for coordination and has scheduler implementations to
>> integrate
>>>> with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache
>>> REEF)
>>>> as well as spout implementations to integrate with Apache Kafka and
>>> metrics
>>>> implementations to integrate with Scribe. Heron also implements the
>>> Apache
>>>> Storm user-level API, which allows topologies written against Storm to
>>> run
>>>> in Heron. We believe that having Heron at Apache will help further the
>>>> growth of the streaming compute community, as well as encourage
>>> cooperation
>>>> and developer cross pollination with other Apache projects.
>>>> 
>>>> = Known Risks =
>>>> 
>>>> == Orphaned Products ==
>>>> 
>>>> The risk of the Heron project being abandoned is minimal. It is used in
>>>> production at Twitter and Google and other companies are evaluating or
>>>> adopting it for production use.
>>>> 
>>>> == Inexperience with Open Source ==
>>>> 
>>>> All of the core contributors to the project have considerable
>> experience
>>>> with open source software development. Bill Graham[2], Ashvin
>> Agrawal[3]
>>>> and Supun Kamburugamuve[4], committers on the project, are PMCs on
>> other
>>>> Apache projects and Bill and Ashvin have gone through the Apache
>>> incubator
>>>> process. Twitter has already donated numerous projects to the ASF
>> (e.g.,
>>>> Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be
>> mentored
>>>> by experienced ASF members that can help with any roadblocks.
>>>> 
>>>> == Homogenous Developers ==
>>>> 
>>>> Initial committers come from 5 separate organizations. Our intention is
>>>> increase the diversity of contributing developers and their
>> affiliations.
>>>> To date github contributions have come from approximately 50
>> contributors
>>>> from outside the Twitter team.
>>>> 
>>>> == Reliance on Salaried Developers ==
>>>> 
>>>> It is expected that Heron development will occur on both salaried time
>>> and
>>>> on volunteer time. The majority of initial committers are paid by their
>>>> employers to contribute to this project. We are committed to recruiting
>>>> additional committers from other organizations as well as non-salaried
>>>> committers to join project.
>>>> 
>>>> == Relationships with Other Apache Products ==
>>>> 
>>>> As mentioned in the Alignment section, Heron implements the Apache
>> Storm
>>>> API and integrates with multiple Apache schedulers (Apache Mesos,
>> Apache
>>>> Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache
>>>> Thrift.
>>>> 
>>>> == An Excessive Fascination with the Apache Brand ==
>>>> 
>>>> Heron's popularity is growing in the streaming compute space and we are
>>>> long time supporters of the Apache brand. This proposal is not for the
>>>> purpose of generating publicity through. Rather, the primary benefits
>> to
>>>> joining Apache are those of community building and open decision making
>>>> outlined in the Rationale section.
>>>> 
>>>> == Documentation ==
>>>> 
>>>> This proposal exists online as
>>>> http://wiki.apache.org/incubator/HeronProposal. Extensive
>> documentation
>>> can
>>>> be found on github at https://twitter.github.io/heron and the source
>>> code
>>>> is well documented.
>>>> 
>>>> == Source and Intellectual Property Submission Plan ==
>>>> 
>>>> The Heron codebase is currently hosted on Github:
>>>> https://github.com/twitter/heron. During incubation, the codebase will
>>> be
>>>> migrated to Apache infrastructure. The source code is already ASF 2.0
>>>> licensed.
>>>> 
>>>> == External Dependencies ==
>>>> 
>>>> All external libraries have ASF 2.0 compatible licenses except for
>>> pylint.
>>>> The pylint library is GPL licensed, but is only used for pre-build
>> Python
>>>> style checks and is neither bundled with, nor relied upon by, the Heron
>>>> source or binary release artifacts.
>>>> 
>>>> == Cryptography ==
>>>> 
>>>> Heron does not use any cryptography libraries.
>>>> 
>>>> = Required Resources =
>>>> 
>>>> == Mailing lists ==
>>>> 
>>>> * private@heron.incubator.apache.org <javascript:;> (with moderated
>> subscriptions)
>>>> * dev@heron.incubator.apache.org <javascript:;>
>>>> * commits@heron.incubator.apache.org <javascript:;>
>>>> * user@heron.incubator.apache.org <javascript:;>
>>>> 
>>>> == Subversion Directory ==
>>>> 
>>>> Git is the preferred source control system: git://git.apache.org/heron
>>>> 
>>>> == Issue Tracking ==
>>>> 
>>>> JIRA: Heron (HERON)
>>>> 
>>>> == Initial Committers ==
>>>> 
>>>> * Andrew Jorgensen (andrew at andrewjorgensen dot com)
>>>> * Ashvin Agrawal (ashvin at apache dot org)*
>>>> * Avrilia Floratou (avrilia dot floratou at gmail dot com)
>>>> * Bill Graham (billgraham at apache dot org)*
>>>> * Brian Hatfield (bmhatfield at gmail dot com)
>>>> * Chris Kellogg (cckellogg at gmail dot com)
>>>> * Huijun Wu (huijun dot wu dot 2010 at gmail dot com)
>>>> * Karthik Ramasamy (karthik at gmail dot com)
>>>> * Maosong Fu (maosongfu at gmail dot com)
>>>> * Neng Lu(freeneng at gmail dot com)
>>>> * Runhang Li (obj dot runhang at gmail dot com)
>>>> * Sanjeev Kulkarni (sanjeevrk at gmail dot com)
>>>> * Supun Kamburugamuve (supun at apache dot org)*
>>>> * Thomas Sun (tom dot ssf at gmail dot com)
>>>> * Yaliang Wang (yaliang dot w dot wang at ieee dot org)
>>>> 
>>>> == Affiliations ==
>>>> 
>>>> * Andrew Jorgensen (Google)
>>>> * Ashvin Agrawal (Microsoft)
>>>> * Avrilia Floratou (Microsoft)
>>>> * Bill Graham (Twitter)
>>>> * Brian Hatfield (Google)
>>>> * Chris Kellogg (Twitter)
>>>> * Huijun Wu (Twitter)
>>>> * Karthik Ramasamy (Streamlio)
>>>> * Maosong Fu (Twitter)
>>>> * Neng Lu (Twitter)
>>>> * Runhang Li (Twitter)
>>>> * Sanjeev Kulkarni (Streamlio)
>>>> * Supun Kamburugamuve (Indiana University)
>>>> * Thomas Sun (Twitter)
>>>> * Yaliang Wang (Twitter)
>>>> 
>>>> = Sponsors =
>>>> 
>>>> == Champion ==
>>>> 
>>>> * Julien Le Dem (julien at apache dot org)
>>>> 
>>>> == Nominated Mentors ==
>>>> 
>>>> * Jake Farrell (jfarrell at apache dot org)
>>>> * Jacques Nadeau (jacques at apache dot org)
>>>> * Julien Le Dem (julien at apache dot org)
>>>> * P. Taylor Goetz (ptgoetz at apache dot org)
>>>> 
>>>> == Sponsoring Entity ==
>>>> 
>>>> The Apache Incubator
>>>> 
>>>> == Footnotes ==
>>>> 
>>>> * 1 - Papers detailing Heron are available at
>>>> http://dl.acm.org/citation.cfm?id=2742788 and
>>>> http://sites.computer.org/debull/A15dec/p15.pdf.
>>>> * 2 - http://home.apache.org/phonebook.html?uid=billgraham
>>>> * 3 - http://home.apache.org/phonebook.html?uid=ashvin
>>>> * 4 - http://home.apache.org/phonebook.html?uid=supun
>>> 
>>> 
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> <javascript:;>
>>> For additional commands, e-mail: general-help@incubator.apache.org <ma...@incubator.apache.org>
>> <javascript:;>
>>> 
>>> 
>> 
> 
> 
> -- 
> Sorry this was sent from mobile. Will do less grammar and spell check than
> usual.

Re: [VOTE] Heron to enter Apache Incubator

Posted by Edward Capriolo <ed...@gmail.com>.
I believe heron and storm should be merged back together. I do not see the
value of storm and a storm fork in the asf.

On Thursday, June 22, 2017, Bill Graham <bi...@gmail.com> wrote:

> Thanks Taylor for relaying these sentiments, especially the part about the
> Heron website which is indeed poorly worded (I suspect this could have been
> the result of internal docs being open-sourced). I've opened this pull
> request to update the language regarding Storm:
>
> https://github.com/twitter/heron/pull/1979
>
> On Thu, Jun 22, 2017 at 12:21 PM, P. Taylor Goetz <ptgoetz@gmail.com
> <javascript:;>> wrote:
>
> > The Apache Storm PMC had a discussion regarding the Heron proposal. In
> the
> > spirit of openness I wanted to bring some of the sentiments expressed in
> > that discussion back to this list. Please note that I am paraphrasing
> from
> > that discussion and attempting to relay opinions of the collective PMC,
> not
> > necessarily that of any individual.
> >
> > * There is a general disappointment that the Heron community chose not to
> > engage with the Storm community and instead chose a separate path.
> > * A majority of the PMC supports Heron’s incubation, though some felt it
> > would result in unnecessary duplication of effort.
> > * A majority of the PMC supports the two projects working closely
> > together. A number of PMC members suggested the two projects merge in
> some
> > way.
> > * Many PMC members took issue some of the marketing language on the Heron
> > website, particularly Heron being billed as “the direct successor to
> Apache
> > Storm” and the prominent “Upgrade from Storm” links.  The main concern
> here
> > was such phrasing has somewhat of a hostile tone and undermines the
> desire
> > for better collaboration, as well as confusing users.
> >
> > One of my goals as a proposed mentor for Heron and a Storm PMC member is
> > to address some of these concerns and encourage collaboration. As I
> > mentioned to the Storm PMC on that thread, if there are ongoing concerns
> > from either the Storm PMC or the Heron PPMC about me acting as a mentor,
> I
> > would be willing to step down.
> >
> > +1 (binding)
> >
> > -Taylor
> >
> > > On Jun 16, 2017, at 4:41 PM, Bill Graham <billgraham@gmail.com
> <javascript:;>> wrote:
> > >
> > > Hi,
> > >
> > > Based on the discussion on the incubator mailing list[1] I would like
> to
> > > call a vote to add Heron to the Apache Incubator.
> > >
> > > The full proposal is available below, and is also available on the
> Apache
> > > Incubator wiki at:
> > >    https://wiki.apache.org/incubator/HeronProposal
> > >
> > > Please vote:
> > >  [ ] +1, bring Heron into Incubator
> > >  [ ] -1, do not bring Heron into Incubator, because...
> > >
> > > The vote will open for 7 days until Friday June 23 at 14:00 PT.
> > >
> > > Thank you
> > >
> > > 1 -
> > > https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7
> > 786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E
> > >
> > >
> > >
> > > = Heron Proposal =
> > >
> > > = Abstract =
> > > Heron is a real-time, distributed, fault-tolerant stream processing
> > engine
> > > initially developed by Twitter.
> > >
> > > = Proposal =
> > >
> > > Heron is a real-time stream processing engine built for high
> performance,
> > > ease of manageability, performance predictability and developer
> > > productivity[1]. We wish to develop a community around Heron to
> increase
> > > contributions and see Heron thrive in an open forum.
> > >
> > > = Background =
> > >
> > > Heron provides the ability for developers to compose directed acyclic
> > > graphs (DAGs) of real-time query execution logic (i.e. a topology) and
> > > submit the topology to execute on a pluggable job scheduling system
> > (e.g.,
> > > Apache Aurora, YARN, Marathon, etc). Users can employ either the native
> > > Heron API or the Apache Storm API to develop the topology. Heron
> supports
> > > the Storm API for ease of migration, but beyond that Heron’s
> architecture
> > > differs considerably from Storm’s.
> > >
> > > Users submit a topology to the scheduler using the Heron client, which
> > uses
> > > the Heron binary libraries to deploy all daemons required to run and
> > manage
> > > the topology. The topology therefore has no reliance on centrally
> managed
> > > Heron services, only on a generic job scheduling system, which lends
> > itself
> > > well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN
> (among
> > > others).
> > >
> > > The scheduler runs each topology as a job consisting of multiple
> > > containers. One of the containers runs the topology master, responsible
> > for
> > > managing the topology. The remaining containers each runs a stream
> > manager
> > > responsible for data routing, a metrics manager that collects and
> reports
> > > various metrics and a number of processes called Heron instances which
> > run
> > > the user-defined logic on the stream of tuples. Parallelism is achieved
> > via
> > > process-based isolation of Heron instances, which provides predictable
> > > performance while simplifying debugging. The containers are allocated
> and
> > > managed by the scheduler framework based on resource availability of
> > nodes
> > > in the cluster. The metadata for the topology, such as the physical
> plan
> > > and execution details, are stored in the pluggable Heron State Manager
> > > (e.g. Apache ZooKeeper).
> > >
> > > = Rationale =
> > >
> > > Heron is a general-purpose, modular and extensible platform that can be
> > > leveraged to support common, real-time analytics use cases. There is an
> > > increasing demand for open-source, scalable real-time analytics
> systems.
> > We
> > > believe that Heron can be leveraged by other organizations to build
> > > streaming applications that can benefit from its robustness, high
> > > performance, adaptability to cloud environments and ease of use.
> > Moreover,
> > > we hope that open-sourcing Heron will help to further evolve the
> > technology
> > > as the project attracts contributors with diverse backgrounds and areas
> > of
> > > expertise.
> > >
> > > We believe the Apache foundation is a great fit as the long-term home
> for
> > > Heron, as it provides an established process for community-driven
> > > development and decision making by consensus. This is exactly the model
> > we
> > > want for future Heron development.
> > >
> > > = Initial Goals =
> > >
> > > * Move the existing codebase, website, documentation, and mailing lists
> > to
> > > Apache-hosted infrastructure.
> > > * Integrate with the Apache development process.
> > > * Ensure all dependencies are compliant with Apache License version
> 2.0.
> > > * Incrementally develop and release per Apache guidelines.
> > >
> > > = Current Status =
> > >
> > > Heron is a stable project used in production at Twitter since 2014 and
> > open
> > > sourced under the ASL v2 license in 2016. The Heron source code is
> > > currently hosted at github.com (https://github.com/twitter/heron),
> which
> > > will seed the Apache git repository.
> > >
> > > = Meritocracy =
> > >
> > > By submitting this incubator proposal, we’re expressing our intent to
> > build
> > > a diverse developer community around Heron that will conduct itself
> > > according to The Apache Way and use a meritocratic means of building
> it's
> > > committer base. Several companies and universities have already
> expressed
> > > interest in and contributed to Heron. Our goal is to grow the Heron
> > > community by encouraging open communication, contribution and
> > participation
> > > of all types, and ensuring that contributors are recognized
> > appropriately.
> > >
> > > = Community =
> > >
> > > Heron is currently being used by Twitter, Google, Machine Zone and
> > > ndustrial.io and has received significant contributions by Microsoft
> and
> > > Streamlio. By bringing Heron into the Apache ecosystem, we believe we
> can
> > > attract even more developers who are interested in creating real-time
> > > systems to build the project's contributor base.
> > >
> > > == Core Developers ==
> > >
> > > Current core developers are engineers from Twitter, Google, Microsoft
> and
> > > Streamlio.
> > >
> > > == Alignment ==
> > >
> > > Heron utilizes a number of Apache technologies. Heron leverages Apache
> > > ZooKeeper for coordination and has scheduler implementations to
> integrate
> > > with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache
> > REEF)
> > > as well as spout implementations to integrate with Apache Kafka and
> > metrics
> > > implementations to integrate with Scribe. Heron also implements the
> > Apache
> > > Storm user-level API, which allows topologies written against Storm to
> > run
> > > in Heron. We believe that having Heron at Apache will help further the
> > > growth of the streaming compute community, as well as encourage
> > cooperation
> > > and developer cross pollination with other Apache projects.
> > >
> > > = Known Risks =
> > >
> > > == Orphaned Products ==
> > >
> > > The risk of the Heron project being abandoned is minimal. It is used in
> > > production at Twitter and Google and other companies are evaluating or
> > > adopting it for production use.
> > >
> > > == Inexperience with Open Source ==
> > >
> > > All of the core contributors to the project have considerable
> experience
> > > with open source software development. Bill Graham[2], Ashvin
> Agrawal[3]
> > > and Supun Kamburugamuve[4], committers on the project, are PMCs on
> other
> > > Apache projects and Bill and Ashvin have gone through the Apache
> > incubator
> > > process. Twitter has already donated numerous projects to the ASF
> (e.g.,
> > > Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be
> mentored
> > > by experienced ASF members that can help with any roadblocks.
> > >
> > > == Homogenous Developers ==
> > >
> > > Initial committers come from 5 separate organizations. Our intention is
> > > increase the diversity of contributing developers and their
> affiliations.
> > > To date github contributions have come from approximately 50
> contributors
> > > from outside the Twitter team.
> > >
> > > == Reliance on Salaried Developers ==
> > >
> > > It is expected that Heron development will occur on both salaried time
> > and
> > > on volunteer time. The majority of initial committers are paid by their
> > > employers to contribute to this project. We are committed to recruiting
> > > additional committers from other organizations as well as non-salaried
> > > committers to join project.
> > >
> > > == Relationships with Other Apache Products ==
> > >
> > > As mentioned in the Alignment section, Heron implements the Apache
> Storm
> > > API and integrates with multiple Apache schedulers (Apache Mesos,
> Apache
> > > Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache
> > > Thrift.
> > >
> > > == An Excessive Fascination with the Apache Brand ==
> > >
> > > Heron's popularity is growing in the streaming compute space and we are
> > > long time supporters of the Apache brand. This proposal is not for the
> > > purpose of generating publicity through. Rather, the primary benefits
> to
> > > joining Apache are those of community building and open decision making
> > > outlined in the Rationale section.
> > >
> > > == Documentation ==
> > >
> > > This proposal exists online as
> > > http://wiki.apache.org/incubator/HeronProposal. Extensive
> documentation
> > can
> > > be found on github at https://twitter.github.io/heron and the source
> > code
> > > is well documented.
> > >
> > > == Source and Intellectual Property Submission Plan ==
> > >
> > > The Heron codebase is currently hosted on Github:
> > > https://github.com/twitter/heron. During incubation, the codebase will
> > be
> > > migrated to Apache infrastructure. The source code is already ASF 2.0
> > > licensed.
> > >
> > > == External Dependencies ==
> > >
> > > All external libraries have ASF 2.0 compatible licenses except for
> > pylint.
> > > The pylint library is GPL licensed, but is only used for pre-build
> Python
> > > style checks and is neither bundled with, nor relied upon by, the Heron
> > > source or binary release artifacts.
> > >
> > > == Cryptography ==
> > >
> > > Heron does not use any cryptography libraries.
> > >
> > > = Required Resources =
> > >
> > > == Mailing lists ==
> > >
> > > * private@heron.incubator.apache.org <javascript:;> (with moderated
> subscriptions)
> > > * dev@heron.incubator.apache.org <javascript:;>
> > > * commits@heron.incubator.apache.org <javascript:;>
> > > * user@heron.incubator.apache.org <javascript:;>
> > >
> > > == Subversion Directory ==
> > >
> > > Git is the preferred source control system: git://git.apache.org/heron
> > >
> > > == Issue Tracking ==
> > >
> > > JIRA: Heron (HERON)
> > >
> > > == Initial Committers ==
> > >
> > > * Andrew Jorgensen (andrew at andrewjorgensen dot com)
> > > * Ashvin Agrawal (ashvin at apache dot org)*
> > > * Avrilia Floratou (avrilia dot floratou at gmail dot com)
> > > * Bill Graham (billgraham at apache dot org)*
> > > * Brian Hatfield (bmhatfield at gmail dot com)
> > > * Chris Kellogg (cckellogg at gmail dot com)
> > > * Huijun Wu (huijun dot wu dot 2010 at gmail dot com)
> > > * Karthik Ramasamy (karthik at gmail dot com)
> > > * Maosong Fu (maosongfu at gmail dot com)
> > > * Neng Lu(freeneng at gmail dot com)
> > > * Runhang Li (obj dot runhang at gmail dot com)
> > > * Sanjeev Kulkarni (sanjeevrk at gmail dot com)
> > > * Supun Kamburugamuve (supun at apache dot org)*
> > > * Thomas Sun (tom dot ssf at gmail dot com)
> > > * Yaliang Wang (yaliang dot w dot wang at ieee dot org)
> > >
> > > == Affiliations ==
> > >
> > > * Andrew Jorgensen (Google)
> > > * Ashvin Agrawal (Microsoft)
> > > * Avrilia Floratou (Microsoft)
> > > * Bill Graham (Twitter)
> > > * Brian Hatfield (Google)
> > > * Chris Kellogg (Twitter)
> > > * Huijun Wu (Twitter)
> > > * Karthik Ramasamy (Streamlio)
> > > * Maosong Fu (Twitter)
> > > * Neng Lu (Twitter)
> > > * Runhang Li (Twitter)
> > > * Sanjeev Kulkarni (Streamlio)
> > > * Supun Kamburugamuve (Indiana University)
> > > * Thomas Sun (Twitter)
> > > * Yaliang Wang (Twitter)
> > >
> > > = Sponsors =
> > >
> > > == Champion ==
> > >
> > > * Julien Le Dem (julien at apache dot org)
> > >
> > > == Nominated Mentors ==
> > >
> > > * Jake Farrell (jfarrell at apache dot org)
> > > * Jacques Nadeau (jacques at apache dot org)
> > > * Julien Le Dem (julien at apache dot org)
> > > * P. Taylor Goetz (ptgoetz at apache dot org)
> > >
> > > == Sponsoring Entity ==
> > >
> > > The Apache Incubator
> > >
> > > == Footnotes ==
> > >
> > > * 1 - Papers detailing Heron are available at
> > > http://dl.acm.org/citation.cfm?id=2742788 and
> > > http://sites.computer.org/debull/A15dec/p15.pdf.
> > > * 2 - http://home.apache.org/phonebook.html?uid=billgraham
> > > * 3 - http://home.apache.org/phonebook.html?uid=ashvin
> > > * 4 - http://home.apache.org/phonebook.html?uid=supun
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> <javascript:;>
> > For additional commands, e-mail: general-help@incubator.apache.org
> <javascript:;>
> >
> >
>


-- 
Sorry this was sent from mobile. Will do less grammar and spell check than
usual.

Re: [VOTE] Heron to enter Apache Incubator

Posted by Bill Graham <bi...@gmail.com>.
Thanks Taylor for relaying these sentiments, especially the part about the
Heron website which is indeed poorly worded (I suspect this could have been
the result of internal docs being open-sourced). I've opened this pull
request to update the language regarding Storm:

https://github.com/twitter/heron/pull/1979

On Thu, Jun 22, 2017 at 12:21 PM, P. Taylor Goetz <pt...@gmail.com> wrote:

> The Apache Storm PMC had a discussion regarding the Heron proposal. In the
> spirit of openness I wanted to bring some of the sentiments expressed in
> that discussion back to this list. Please note that I am paraphrasing from
> that discussion and attempting to relay opinions of the collective PMC, not
> necessarily that of any individual.
>
> * There is a general disappointment that the Heron community chose not to
> engage with the Storm community and instead chose a separate path.
> * A majority of the PMC supports Heron’s incubation, though some felt it
> would result in unnecessary duplication of effort.
> * A majority of the PMC supports the two projects working closely
> together. A number of PMC members suggested the two projects merge in some
> way.
> * Many PMC members took issue some of the marketing language on the Heron
> website, particularly Heron being billed as “the direct successor to Apache
> Storm” and the prominent “Upgrade from Storm” links.  The main concern here
> was such phrasing has somewhat of a hostile tone and undermines the desire
> for better collaboration, as well as confusing users.
>
> One of my goals as a proposed mentor for Heron and a Storm PMC member is
> to address some of these concerns and encourage collaboration. As I
> mentioned to the Storm PMC on that thread, if there are ongoing concerns
> from either the Storm PMC or the Heron PPMC about me acting as a mentor, I
> would be willing to step down.
>
> +1 (binding)
>
> -Taylor
>
> > On Jun 16, 2017, at 4:41 PM, Bill Graham <bi...@gmail.com> wrote:
> >
> > Hi,
> >
> > Based on the discussion on the incubator mailing list[1] I would like to
> > call a vote to add Heron to the Apache Incubator.
> >
> > The full proposal is available below, and is also available on the Apache
> > Incubator wiki at:
> >    https://wiki.apache.org/incubator/HeronProposal
> >
> > Please vote:
> >  [ ] +1, bring Heron into Incubator
> >  [ ] -1, do not bring Heron into Incubator, because...
> >
> > The vote will open for 7 days until Friday June 23 at 14:00 PT.
> >
> > Thank you
> >
> > 1 -
> > https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7
> 786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E
> >
> >
> >
> > = Heron Proposal =
> >
> > = Abstract =
> > Heron is a real-time, distributed, fault-tolerant stream processing
> engine
> > initially developed by Twitter.
> >
> > = Proposal =
> >
> > Heron is a real-time stream processing engine built for high performance,
> > ease of manageability, performance predictability and developer
> > productivity[1]. We wish to develop a community around Heron to increase
> > contributions and see Heron thrive in an open forum.
> >
> > = Background =
> >
> > Heron provides the ability for developers to compose directed acyclic
> > graphs (DAGs) of real-time query execution logic (i.e. a topology) and
> > submit the topology to execute on a pluggable job scheduling system
> (e.g.,
> > Apache Aurora, YARN, Marathon, etc). Users can employ either the native
> > Heron API or the Apache Storm API to develop the topology. Heron supports
> > the Storm API for ease of migration, but beyond that Heron’s architecture
> > differs considerably from Storm’s.
> >
> > Users submit a topology to the scheduler using the Heron client, which
> uses
> > the Heron binary libraries to deploy all daemons required to run and
> manage
> > the topology. The topology therefore has no reliance on centrally managed
> > Heron services, only on a generic job scheduling system, which lends
> itself
> > well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN (among
> > others).
> >
> > The scheduler runs each topology as a job consisting of multiple
> > containers. One of the containers runs the topology master, responsible
> for
> > managing the topology. The remaining containers each runs a stream
> manager
> > responsible for data routing, a metrics manager that collects and reports
> > various metrics and a number of processes called Heron instances which
> run
> > the user-defined logic on the stream of tuples. Parallelism is achieved
> via
> > process-based isolation of Heron instances, which provides predictable
> > performance while simplifying debugging. The containers are allocated and
> > managed by the scheduler framework based on resource availability of
> nodes
> > in the cluster. The metadata for the topology, such as the physical plan
> > and execution details, are stored in the pluggable Heron State Manager
> > (e.g. Apache ZooKeeper).
> >
> > = Rationale =
> >
> > Heron is a general-purpose, modular and extensible platform that can be
> > leveraged to support common, real-time analytics use cases. There is an
> > increasing demand for open-source, scalable real-time analytics systems.
> We
> > believe that Heron can be leveraged by other organizations to build
> > streaming applications that can benefit from its robustness, high
> > performance, adaptability to cloud environments and ease of use.
> Moreover,
> > we hope that open-sourcing Heron will help to further evolve the
> technology
> > as the project attracts contributors with diverse backgrounds and areas
> of
> > expertise.
> >
> > We believe the Apache foundation is a great fit as the long-term home for
> > Heron, as it provides an established process for community-driven
> > development and decision making by consensus. This is exactly the model
> we
> > want for future Heron development.
> >
> > = Initial Goals =
> >
> > * Move the existing codebase, website, documentation, and mailing lists
> to
> > Apache-hosted infrastructure.
> > * Integrate with the Apache development process.
> > * Ensure all dependencies are compliant with Apache License version 2.0.
> > * Incrementally develop and release per Apache guidelines.
> >
> > = Current Status =
> >
> > Heron is a stable project used in production at Twitter since 2014 and
> open
> > sourced under the ASL v2 license in 2016. The Heron source code is
> > currently hosted at github.com (https://github.com/twitter/heron), which
> > will seed the Apache git repository.
> >
> > = Meritocracy =
> >
> > By submitting this incubator proposal, we’re expressing our intent to
> build
> > a diverse developer community around Heron that will conduct itself
> > according to The Apache Way and use a meritocratic means of building it's
> > committer base. Several companies and universities have already expressed
> > interest in and contributed to Heron. Our goal is to grow the Heron
> > community by encouraging open communication, contribution and
> participation
> > of all types, and ensuring that contributors are recognized
> appropriately.
> >
> > = Community =
> >
> > Heron is currently being used by Twitter, Google, Machine Zone and
> > ndustrial.io and has received significant contributions by Microsoft and
> > Streamlio. By bringing Heron into the Apache ecosystem, we believe we can
> > attract even more developers who are interested in creating real-time
> > systems to build the project's contributor base.
> >
> > == Core Developers ==
> >
> > Current core developers are engineers from Twitter, Google, Microsoft and
> > Streamlio.
> >
> > == Alignment ==
> >
> > Heron utilizes a number of Apache technologies. Heron leverages Apache
> > ZooKeeper for coordination and has scheduler implementations to integrate
> > with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache
> REEF)
> > as well as spout implementations to integrate with Apache Kafka and
> metrics
> > implementations to integrate with Scribe. Heron also implements the
> Apache
> > Storm user-level API, which allows topologies written against Storm to
> run
> > in Heron. We believe that having Heron at Apache will help further the
> > growth of the streaming compute community, as well as encourage
> cooperation
> > and developer cross pollination with other Apache projects.
> >
> > = Known Risks =
> >
> > == Orphaned Products ==
> >
> > The risk of the Heron project being abandoned is minimal. It is used in
> > production at Twitter and Google and other companies are evaluating or
> > adopting it for production use.
> >
> > == Inexperience with Open Source ==
> >
> > All of the core contributors to the project have considerable experience
> > with open source software development. Bill Graham[2], Ashvin Agrawal[3]
> > and Supun Kamburugamuve[4], committers on the project, are PMCs on other
> > Apache projects and Bill and Ashvin have gone through the Apache
> incubator
> > process. Twitter has already donated numerous projects to the ASF (e.g.,
> > Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be mentored
> > by experienced ASF members that can help with any roadblocks.
> >
> > == Homogenous Developers ==
> >
> > Initial committers come from 5 separate organizations. Our intention is
> > increase the diversity of contributing developers and their affiliations.
> > To date github contributions have come from approximately 50 contributors
> > from outside the Twitter team.
> >
> > == Reliance on Salaried Developers ==
> >
> > It is expected that Heron development will occur on both salaried time
> and
> > on volunteer time. The majority of initial committers are paid by their
> > employers to contribute to this project. We are committed to recruiting
> > additional committers from other organizations as well as non-salaried
> > committers to join project.
> >
> > == Relationships with Other Apache Products ==
> >
> > As mentioned in the Alignment section, Heron implements the Apache Storm
> > API and integrates with multiple Apache schedulers (Apache Mesos, Apache
> > Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache
> > Thrift.
> >
> > == An Excessive Fascination with the Apache Brand ==
> >
> > Heron's popularity is growing in the streaming compute space and we are
> > long time supporters of the Apache brand. This proposal is not for the
> > purpose of generating publicity through. Rather, the primary benefits to
> > joining Apache are those of community building and open decision making
> > outlined in the Rationale section.
> >
> > == Documentation ==
> >
> > This proposal exists online as
> > http://wiki.apache.org/incubator/HeronProposal. Extensive documentation
> can
> > be found on github at https://twitter.github.io/heron and the source
> code
> > is well documented.
> >
> > == Source and Intellectual Property Submission Plan ==
> >
> > The Heron codebase is currently hosted on Github:
> > https://github.com/twitter/heron. During incubation, the codebase will
> be
> > migrated to Apache infrastructure. The source code is already ASF 2.0
> > licensed.
> >
> > == External Dependencies ==
> >
> > All external libraries have ASF 2.0 compatible licenses except for
> pylint.
> > The pylint library is GPL licensed, but is only used for pre-build Python
> > style checks and is neither bundled with, nor relied upon by, the Heron
> > source or binary release artifacts.
> >
> > == Cryptography ==
> >
> > Heron does not use any cryptography libraries.
> >
> > = Required Resources =
> >
> > == Mailing lists ==
> >
> > * private@heron.incubator.apache.org (with moderated subscriptions)
> > * dev@heron.incubator.apache.org
> > * commits@heron.incubator.apache.org
> > * user@heron.incubator.apache.org
> >
> > == Subversion Directory ==
> >
> > Git is the preferred source control system: git://git.apache.org/heron
> >
> > == Issue Tracking ==
> >
> > JIRA: Heron (HERON)
> >
> > == Initial Committers ==
> >
> > * Andrew Jorgensen (andrew at andrewjorgensen dot com)
> > * Ashvin Agrawal (ashvin at apache dot org)*
> > * Avrilia Floratou (avrilia dot floratou at gmail dot com)
> > * Bill Graham (billgraham at apache dot org)*
> > * Brian Hatfield (bmhatfield at gmail dot com)
> > * Chris Kellogg (cckellogg at gmail dot com)
> > * Huijun Wu (huijun dot wu dot 2010 at gmail dot com)
> > * Karthik Ramasamy (karthik at gmail dot com)
> > * Maosong Fu (maosongfu at gmail dot com)
> > * Neng Lu(freeneng at gmail dot com)
> > * Runhang Li (obj dot runhang at gmail dot com)
> > * Sanjeev Kulkarni (sanjeevrk at gmail dot com)
> > * Supun Kamburugamuve (supun at apache dot org)*
> > * Thomas Sun (tom dot ssf at gmail dot com)
> > * Yaliang Wang (yaliang dot w dot wang at ieee dot org)
> >
> > == Affiliations ==
> >
> > * Andrew Jorgensen (Google)
> > * Ashvin Agrawal (Microsoft)
> > * Avrilia Floratou (Microsoft)
> > * Bill Graham (Twitter)
> > * Brian Hatfield (Google)
> > * Chris Kellogg (Twitter)
> > * Huijun Wu (Twitter)
> > * Karthik Ramasamy (Streamlio)
> > * Maosong Fu (Twitter)
> > * Neng Lu (Twitter)
> > * Runhang Li (Twitter)
> > * Sanjeev Kulkarni (Streamlio)
> > * Supun Kamburugamuve (Indiana University)
> > * Thomas Sun (Twitter)
> > * Yaliang Wang (Twitter)
> >
> > = Sponsors =
> >
> > == Champion ==
> >
> > * Julien Le Dem (julien at apache dot org)
> >
> > == Nominated Mentors ==
> >
> > * Jake Farrell (jfarrell at apache dot org)
> > * Jacques Nadeau (jacques at apache dot org)
> > * Julien Le Dem (julien at apache dot org)
> > * P. Taylor Goetz (ptgoetz at apache dot org)
> >
> > == Sponsoring Entity ==
> >
> > The Apache Incubator
> >
> > == Footnotes ==
> >
> > * 1 - Papers detailing Heron are available at
> > http://dl.acm.org/citation.cfm?id=2742788 and
> > http://sites.computer.org/debull/A15dec/p15.pdf.
> > * 2 - http://home.apache.org/phonebook.html?uid=billgraham
> > * 3 - http://home.apache.org/phonebook.html?uid=ashvin
> > * 4 - http://home.apache.org/phonebook.html?uid=supun
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>

Re: [VOTE] Heron to enter Apache Incubator

Posted by Bill Graham <bi...@gmail.com>.
This vote is now close and it has passed. Thanks to all who all who
participated in the proposal review and vote.

The vote tally is as follows:

24 [+1] votes (8 binding, 16 non-binding)

Binding Votes:
================

Julien Le Dem
Raphael Bircher
Jake Farrell
Jacques Nadeau
Julian Hyde
Chris Douglas
John Ament
P. Taylor Goetz

Non-binding Votes:
================

Debo Dutta
William Markito Oliveira
Roy Lenferink
Sijie Guo
Sanjeev Kulkarni
Chris Aniszczyk
Supun Kamburugamuva
Jia Zhai
Karthi Ramasamy
Ashvin A
Ashish
Nabarun Nag
Pierre Smits
Byung-Gon Chun
Bill Graham
Van Gosling


On Fri, Jun 23, 2017 at 10:25 AM, Debo Dutta (dedutta) <de...@cisco.com>
wrote:

> +1 to Ted’s comment.
>
> As a user, I would love to pick one system and reuse the storm topologies.
> Ideally pick one converged solution.
>
> +1 to the incubation since it will eventually lead to a better options
> within Apache.
>
> debo
>
> On 6/23/17, 10:08 AM, "Ted Dunning" <te...@gmail.com> wrote:
>
>     Anybody who worries about you serving as mentor needs a dose of
> reality.
>     They can't get anybody better.
>
>     On Jun 22, 2017 12:21 PM, "P. Taylor Goetz" <pt...@gmail.com> wrote:
>
>     if there are ongoing concerns from either the Storm PMC or the Heron
> PPMC
>     about me acting as a mentor, I would be willing to step down.
>
>     +1 (binding)
>
>     -Taylor
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>

Re: [VOTE] Heron to enter Apache Incubator

Posted by "Debo Dutta (dedutta)" <de...@cisco.com>.
+1 to Ted’s comment.

As a user, I would love to pick one system and reuse the storm topologies. Ideally pick one converged solution.

+1 to the incubation since it will eventually lead to a better options within Apache. 

debo

On 6/23/17, 10:08 AM, "Ted Dunning" <te...@gmail.com> wrote:

    Anybody who worries about you serving as mentor needs a dose of reality.
    They can't get anybody better.
    
    On Jun 22, 2017 12:21 PM, "P. Taylor Goetz" <pt...@gmail.com> wrote:
    
    if there are ongoing concerns from either the Storm PMC or the Heron PPMC
    about me acting as a mentor, I would be willing to step down.
    
    +1 (binding)
    
    -Taylor
    


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [VOTE] Heron to enter Apache Incubator

Posted by Ted Dunning <te...@gmail.com>.
Anybody who worries about you serving as mentor needs a dose of reality.
They can't get anybody better.

On Jun 22, 2017 12:21 PM, "P. Taylor Goetz" <pt...@gmail.com> wrote:

if there are ongoing concerns from either the Storm PMC or the Heron PPMC
about me acting as a mentor, I would be willing to step down.

+1 (binding)

-Taylor

Re: [VOTE] Heron to enter Apache Incubator

Posted by "P. Taylor Goetz" <pt...@gmail.com>.
The Apache Storm PMC had a discussion regarding the Heron proposal. In the spirit of openness I wanted to bring some of the sentiments expressed in that discussion back to this list. Please note that I am paraphrasing from that discussion and attempting to relay opinions of the collective PMC, not necessarily that of any individual.

* There is a general disappointment that the Heron community chose not to engage with the Storm community and instead chose a separate path.
* A majority of the PMC supports Heron’s incubation, though some felt it would result in unnecessary duplication of effort.
* A majority of the PMC supports the two projects working closely together. A number of PMC members suggested the two projects merge in some way.
* Many PMC members took issue some of the marketing language on the Heron website, particularly Heron being billed as “the direct successor to Apache Storm” and the prominent “Upgrade from Storm” links.  The main concern here was such phrasing has somewhat of a hostile tone and undermines the desire for better collaboration, as well as confusing users.

One of my goals as a proposed mentor for Heron and a Storm PMC member is to address some of these concerns and encourage collaboration. As I mentioned to the Storm PMC on that thread, if there are ongoing concerns from either the Storm PMC or the Heron PPMC about me acting as a mentor, I would be willing to step down.

+1 (binding)

-Taylor

> On Jun 16, 2017, at 4:41 PM, Bill Graham <bi...@gmail.com> wrote:
> 
> Hi,
> 
> Based on the discussion on the incubator mailing list[1] I would like to
> call a vote to add Heron to the Apache Incubator.
> 
> The full proposal is available below, and is also available on the Apache
> Incubator wiki at:
>    https://wiki.apache.org/incubator/HeronProposal
> 
> Please vote:
>  [ ] +1, bring Heron into Incubator
>  [ ] -1, do not bring Heron into Incubator, because...
> 
> The vote will open for 7 days until Friday June 23 at 14:00 PT.
> 
> Thank you
> 
> 1 -
> https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E
> 
> 
> 
> = Heron Proposal =
> 
> = Abstract =
> Heron is a real-time, distributed, fault-tolerant stream processing engine
> initially developed by Twitter.
> 
> = Proposal =
> 
> Heron is a real-time stream processing engine built for high performance,
> ease of manageability, performance predictability and developer
> productivity[1]. We wish to develop a community around Heron to increase
> contributions and see Heron thrive in an open forum.
> 
> = Background =
> 
> Heron provides the ability for developers to compose directed acyclic
> graphs (DAGs) of real-time query execution logic (i.e. a topology) and
> submit the topology to execute on a pluggable job scheduling system (e.g.,
> Apache Aurora, YARN, Marathon, etc). Users can employ either the native
> Heron API or the Apache Storm API to develop the topology. Heron supports
> the Storm API for ease of migration, but beyond that Heron’s architecture
> differs considerably from Storm’s.
> 
> Users submit a topology to the scheduler using the Heron client, which uses
> the Heron binary libraries to deploy all daemons required to run and manage
> the topology. The topology therefore has no reliance on centrally managed
> Heron services, only on a generic job scheduling system, which lends itself
> well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN (among
> others).
> 
> The scheduler runs each topology as a job consisting of multiple
> containers. One of the containers runs the topology master, responsible for
> managing the topology. The remaining containers each runs a stream manager
> responsible for data routing, a metrics manager that collects and reports
> various metrics and a number of processes called Heron instances which run
> the user-defined logic on the stream of tuples. Parallelism is achieved via
> process-based isolation of Heron instances, which provides predictable
> performance while simplifying debugging. The containers are allocated and
> managed by the scheduler framework based on resource availability of nodes
> in the cluster. The metadata for the topology, such as the physical plan
> and execution details, are stored in the pluggable Heron State Manager
> (e.g. Apache ZooKeeper).
> 
> = Rationale =
> 
> Heron is a general-purpose, modular and extensible platform that can be
> leveraged to support common, real-time analytics use cases. There is an
> increasing demand for open-source, scalable real-time analytics systems. We
> believe that Heron can be leveraged by other organizations to build
> streaming applications that can benefit from its robustness, high
> performance, adaptability to cloud environments and ease of use. Moreover,
> we hope that open-sourcing Heron will help to further evolve the technology
> as the project attracts contributors with diverse backgrounds and areas of
> expertise.
> 
> We believe the Apache foundation is a great fit as the long-term home for
> Heron, as it provides an established process for community-driven
> development and decision making by consensus. This is exactly the model we
> want for future Heron development.
> 
> = Initial Goals =
> 
> * Move the existing codebase, website, documentation, and mailing lists to
> Apache-hosted infrastructure.
> * Integrate with the Apache development process.
> * Ensure all dependencies are compliant with Apache License version 2.0.
> * Incrementally develop and release per Apache guidelines.
> 
> = Current Status =
> 
> Heron is a stable project used in production at Twitter since 2014 and open
> sourced under the ASL v2 license in 2016. The Heron source code is
> currently hosted at github.com (https://github.com/twitter/heron), which
> will seed the Apache git repository.
> 
> = Meritocracy =
> 
> By submitting this incubator proposal, we’re expressing our intent to build
> a diverse developer community around Heron that will conduct itself
> according to The Apache Way and use a meritocratic means of building it's
> committer base. Several companies and universities have already expressed
> interest in and contributed to Heron. Our goal is to grow the Heron
> community by encouraging open communication, contribution and participation
> of all types, and ensuring that contributors are recognized appropriately.
> 
> = Community =
> 
> Heron is currently being used by Twitter, Google, Machine Zone and
> ndustrial.io and has received significant contributions by Microsoft and
> Streamlio. By bringing Heron into the Apache ecosystem, we believe we can
> attract even more developers who are interested in creating real-time
> systems to build the project's contributor base.
> 
> == Core Developers ==
> 
> Current core developers are engineers from Twitter, Google, Microsoft and
> Streamlio.
> 
> == Alignment ==
> 
> Heron utilizes a number of Apache technologies. Heron leverages Apache
> ZooKeeper for coordination and has scheduler implementations to integrate
> with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache REEF)
> as well as spout implementations to integrate with Apache Kafka and metrics
> implementations to integrate with Scribe. Heron also implements the Apache
> Storm user-level API, which allows topologies written against Storm to run
> in Heron. We believe that having Heron at Apache will help further the
> growth of the streaming compute community, as well as encourage cooperation
> and developer cross pollination with other Apache projects.
> 
> = Known Risks =
> 
> == Orphaned Products ==
> 
> The risk of the Heron project being abandoned is minimal. It is used in
> production at Twitter and Google and other companies are evaluating or
> adopting it for production use.
> 
> == Inexperience with Open Source ==
> 
> All of the core contributors to the project have considerable experience
> with open source software development. Bill Graham[2], Ashvin Agrawal[3]
> and Supun Kamburugamuve[4], committers on the project, are PMCs on other
> Apache projects and Bill and Ashvin have gone through the Apache incubator
> process. Twitter has already donated numerous projects to the ASF (e.g.,
> Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be mentored
> by experienced ASF members that can help with any roadblocks.
> 
> == Homogenous Developers ==
> 
> Initial committers come from 5 separate organizations. Our intention is
> increase the diversity of contributing developers and their affiliations.
> To date github contributions have come from approximately 50 contributors
> from outside the Twitter team.
> 
> == Reliance on Salaried Developers ==
> 
> It is expected that Heron development will occur on both salaried time and
> on volunteer time. The majority of initial committers are paid by their
> employers to contribute to this project. We are committed to recruiting
> additional committers from other organizations as well as non-salaried
> committers to join project.
> 
> == Relationships with Other Apache Products ==
> 
> As mentioned in the Alignment section, Heron implements the Apache Storm
> API and integrates with multiple Apache schedulers (Apache Mesos, Apache
> Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache
> Thrift.
> 
> == An Excessive Fascination with the Apache Brand ==
> 
> Heron's popularity is growing in the streaming compute space and we are
> long time supporters of the Apache brand. This proposal is not for the
> purpose of generating publicity through. Rather, the primary benefits to
> joining Apache are those of community building and open decision making
> outlined in the Rationale section.
> 
> == Documentation ==
> 
> This proposal exists online as
> http://wiki.apache.org/incubator/HeronProposal. Extensive documentation can
> be found on github at https://twitter.github.io/heron and the source code
> is well documented.
> 
> == Source and Intellectual Property Submission Plan ==
> 
> The Heron codebase is currently hosted on Github:
> https://github.com/twitter/heron. During incubation, the codebase will be
> migrated to Apache infrastructure. The source code is already ASF 2.0
> licensed.
> 
> == External Dependencies ==
> 
> All external libraries have ASF 2.0 compatible licenses except for pylint.
> The pylint library is GPL licensed, but is only used for pre-build Python
> style checks and is neither bundled with, nor relied upon by, the Heron
> source or binary release artifacts.
> 
> == Cryptography ==
> 
> Heron does not use any cryptography libraries.
> 
> = Required Resources =
> 
> == Mailing lists ==
> 
> * private@heron.incubator.apache.org (with moderated subscriptions)
> * dev@heron.incubator.apache.org
> * commits@heron.incubator.apache.org
> * user@heron.incubator.apache.org
> 
> == Subversion Directory ==
> 
> Git is the preferred source control system: git://git.apache.org/heron
> 
> == Issue Tracking ==
> 
> JIRA: Heron (HERON)
> 
> == Initial Committers ==
> 
> * Andrew Jorgensen (andrew at andrewjorgensen dot com)
> * Ashvin Agrawal (ashvin at apache dot org)*
> * Avrilia Floratou (avrilia dot floratou at gmail dot com)
> * Bill Graham (billgraham at apache dot org)*
> * Brian Hatfield (bmhatfield at gmail dot com)
> * Chris Kellogg (cckellogg at gmail dot com)
> * Huijun Wu (huijun dot wu dot 2010 at gmail dot com)
> * Karthik Ramasamy (karthik at gmail dot com)
> * Maosong Fu (maosongfu at gmail dot com)
> * Neng Lu(freeneng at gmail dot com)
> * Runhang Li (obj dot runhang at gmail dot com)
> * Sanjeev Kulkarni (sanjeevrk at gmail dot com)
> * Supun Kamburugamuve (supun at apache dot org)*
> * Thomas Sun (tom dot ssf at gmail dot com)
> * Yaliang Wang (yaliang dot w dot wang at ieee dot org)
> 
> == Affiliations ==
> 
> * Andrew Jorgensen (Google)
> * Ashvin Agrawal (Microsoft)
> * Avrilia Floratou (Microsoft)
> * Bill Graham (Twitter)
> * Brian Hatfield (Google)
> * Chris Kellogg (Twitter)
> * Huijun Wu (Twitter)
> * Karthik Ramasamy (Streamlio)
> * Maosong Fu (Twitter)
> * Neng Lu (Twitter)
> * Runhang Li (Twitter)
> * Sanjeev Kulkarni (Streamlio)
> * Supun Kamburugamuve (Indiana University)
> * Thomas Sun (Twitter)
> * Yaliang Wang (Twitter)
> 
> = Sponsors =
> 
> == Champion ==
> 
> * Julien Le Dem (julien at apache dot org)
> 
> == Nominated Mentors ==
> 
> * Jake Farrell (jfarrell at apache dot org)
> * Jacques Nadeau (jacques at apache dot org)
> * Julien Le Dem (julien at apache dot org)
> * P. Taylor Goetz (ptgoetz at apache dot org)
> 
> == Sponsoring Entity ==
> 
> The Apache Incubator
> 
> == Footnotes ==
> 
> * 1 - Papers detailing Heron are available at
> http://dl.acm.org/citation.cfm?id=2742788 and
> http://sites.computer.org/debull/A15dec/p15.pdf.
> * 2 - http://home.apache.org/phonebook.html?uid=billgraham
> * 3 - http://home.apache.org/phonebook.html?uid=ashvin
> * 4 - http://home.apache.org/phonebook.html?uid=supun


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org

Re: [VOTE] Heron to enter Apache Incubator

Posted by Jia Zhai <zh...@gmail.com>.
+1 (non-binding)

On Fri, Jun 16, 2017 at 2:02 PM, Supun Kamburugamuva <su...@apache.org>
wrote:

> +1
>
> Thanks,
> Supun..
>
> On Fri, Jun 16, 2017 at 4:59 PM, Sanjeev Kulkarni <sa...@streaml.io>
> wrote:
>
> > +1
> >
> > On Fri, Jun 16, 2017 at 1:41 PM, Bill Graham <bi...@gmail.com>
> wrote:
> >
> > > Hi,
> > >
> > > Based on the discussion on the incubator mailing list[1] I would like
> to
> > > call a vote to add Heron to the Apache Incubator.
> > >
> > > The full proposal is available below, and is also available on the
> Apache
> > > Incubator wiki at:
> > >     https://wiki.apache.org/incubator/HeronProposal
> > >
> > > Please vote:
> > >   [ ] +1, bring Heron into Incubator
> > >   [ ] -1, do not bring Heron into Incubator, because...
> > >
> > > The vote will open for 7 days until Friday June 23 at 14:00 PT.
> > >
> > > Thank you
> > >
> > > 1 -
> > > https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7
> > > 786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E
> > >
> > >
> > >
> > > = Heron Proposal =
> > >
> > > = Abstract =
> > > Heron is a real-time, distributed, fault-tolerant stream processing
> > engine
> > > initially developed by Twitter.
> > >
> > > = Proposal =
> > >
> > > Heron is a real-time stream processing engine built for high
> performance,
> > > ease of manageability, performance predictability and developer
> > > productivity[1]. We wish to develop a community around Heron to
> increase
> > > contributions and see Heron thrive in an open forum.
> > >
> > > = Background =
> > >
> > > Heron provides the ability for developers to compose directed acyclic
> > > graphs (DAGs) of real-time query execution logic (i.e. a topology) and
> > > submit the topology to execute on a pluggable job scheduling system
> > (e.g.,
> > > Apache Aurora, YARN, Marathon, etc). Users can employ either the native
> > > Heron API or the Apache Storm API to develop the topology. Heron
> supports
> > > the Storm API for ease of migration, but beyond that Heron’s
> architecture
> > > differs considerably from Storm’s.
> > >
> > > Users submit a topology to the scheduler using the Heron client, which
> > uses
> > > the Heron binary libraries to deploy all daemons required to run and
> > manage
> > > the topology. The topology therefore has no reliance on centrally
> managed
> > > Heron services, only on a generic job scheduling system, which lends
> > itself
> > > well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN
> (among
> > > others).
> > >
> > > The scheduler runs each topology as a job consisting of multiple
> > > containers. One of the containers runs the topology master, responsible
> > for
> > > managing the topology. The remaining containers each runs a stream
> > manager
> > > responsible for data routing, a metrics manager that collects and
> reports
> > > various metrics and a number of processes called Heron instances which
> > run
> > > the user-defined logic on the stream of tuples. Parallelism is achieved
> > via
> > > process-based isolation of Heron instances, which provides predictable
> > > performance while simplifying debugging. The containers are allocated
> and
> > > managed by the scheduler framework based on resource availability of
> > nodes
> > > in the cluster. The metadata for the topology, such as the physical
> plan
> > > and execution details, are stored in the pluggable Heron State Manager
> > > (e.g. Apache ZooKeeper).
> > >
> > > = Rationale =
> > >
> > > Heron is a general-purpose, modular and extensible platform that can be
> > > leveraged to support common, real-time analytics use cases. There is an
> > > increasing demand for open-source, scalable real-time analytics
> systems.
> > We
> > > believe that Heron can be leveraged by other organizations to build
> > > streaming applications that can benefit from its robustness, high
> > > performance, adaptability to cloud environments and ease of use.
> > Moreover,
> > > we hope that open-sourcing Heron will help to further evolve the
> > technology
> > > as the project attracts contributors with diverse backgrounds and areas
> > of
> > > expertise.
> > >
> > > We believe the Apache foundation is a great fit as the long-term home
> for
> > > Heron, as it provides an established process for community-driven
> > > development and decision making by consensus. This is exactly the model
> > we
> > > want for future Heron development.
> > >
> > > = Initial Goals =
> > >
> > >  * Move the existing codebase, website, documentation, and mailing
> lists
> > to
> > > Apache-hosted infrastructure.
> > >  * Integrate with the Apache development process.
> > >  * Ensure all dependencies are compliant with Apache License version
> 2.0.
> > >  * Incrementally develop and release per Apache guidelines.
> > >
> > > = Current Status =
> > >
> > > Heron is a stable project used in production at Twitter since 2014 and
> > open
> > > sourced under the ASL v2 license in 2016. The Heron source code is
> > > currently hosted at github.com (https://github.com/twitter/heron),
> which
> > > will seed the Apache git repository.
> > >
> > > = Meritocracy =
> > >
> > > By submitting this incubator proposal, we’re expressing our intent to
> > build
> > > a diverse developer community around Heron that will conduct itself
> > > according to The Apache Way and use a meritocratic means of building
> it's
> > > committer base. Several companies and universities have already
> expressed
> > > interest in and contributed to Heron. Our goal is to grow the Heron
> > > community by encouraging open communication, contribution and
> > participation
> > > of all types, and ensuring that contributors are recognized
> > appropriately.
> > >
> > > = Community =
> > >
> > > Heron is currently being used by Twitter, Google, Machine Zone and
> > > ndustrial.io and has received significant contributions by Microsoft
> and
> > > Streamlio. By bringing Heron into the Apache ecosystem, we believe we
> can
> > > attract even more developers who are interested in creating real-time
> > > systems to build the project's contributor base.
> > >
> > > == Core Developers ==
> > >
> > > Current core developers are engineers from Twitter, Google, Microsoft
> and
> > > Streamlio.
> > >
> > > == Alignment ==
> > >
> > > Heron utilizes a number of Apache technologies. Heron leverages Apache
> > > ZooKeeper for coordination and has scheduler implementations to
> integrate
> > > with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache
> > REEF)
> > > as well as spout implementations to integrate with Apache Kafka and
> > metrics
> > > implementations to integrate with Scribe. Heron also implements the
> > Apache
> > > Storm user-level API, which allows topologies written against Storm to
> > run
> > > in Heron. We believe that having Heron at Apache will help further the
> > > growth of the streaming compute community, as well as encourage
> > cooperation
> > > and developer cross pollination with other Apache projects.
> > >
> > > = Known Risks =
> > >
> > > == Orphaned Products ==
> > >
> > > The risk of the Heron project being abandoned is minimal. It is used in
> > > production at Twitter and Google and other companies are evaluating or
> > > adopting it for production use.
> > >
> > > == Inexperience with Open Source ==
> > >
> > > All of the core contributors to the project have considerable
> experience
> > > with open source software development. Bill Graham[2], Ashvin
> Agrawal[3]
> > > and Supun Kamburugamuve[4], committers on the project, are PMCs on
> other
> > > Apache projects and Bill and Ashvin have gone through the Apache
> > incubator
> > > process. Twitter has already donated numerous projects to the ASF
> (e.g.,
> > > Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be
> mentored
> > > by experienced ASF members that can help with any roadblocks.
> > >
> > > == Homogenous Developers ==
> > >
> > > Initial committers come from 5 separate organizations. Our intention is
> > > increase the diversity of contributing developers and their
> affiliations.
> > > To date github contributions have come from approximately 50
> contributors
> > > from outside the Twitter team.
> > >
> > > == Reliance on Salaried Developers ==
> > >
> > > It is expected that Heron development will occur on both salaried time
> > and
> > > on volunteer time. The majority of initial committers are paid by their
> > > employers to contribute to this project. We are committed to recruiting
> > > additional committers from other organizations as well as non-salaried
> > > committers to join project.
> > >
> > > == Relationships with Other Apache Products ==
> > >
> > > As mentioned in the Alignment section, Heron implements the Apache
> Storm
> > > API and integrates with multiple Apache schedulers (Apache Mesos,
> Apache
> > > Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache
> > > Thrift.
> > >
> > > == An Excessive Fascination with the Apache Brand ==
> > >
> > > Heron's popularity is growing in the streaming compute space and we are
> > > long time supporters of the Apache brand. This proposal is not for the
> > > purpose of generating publicity through. Rather, the primary benefits
> to
> > > joining Apache are those of community building and open decision making
> > > outlined in the Rationale section.
> > >
> > > == Documentation ==
> > >
> > > This proposal exists online as
> > > http://wiki.apache.org/incubator/HeronProposal. Extensive
> documentation
> > > can
> > > be found on github at https://twitter.github.io/heron and the source
> > code
> > > is well documented.
> > >
> > > == Source and Intellectual Property Submission Plan ==
> > >
> > > The Heron codebase is currently hosted on Github:
> > > https://github.com/twitter/heron. During incubation, the codebase will
> > be
> > > migrated to Apache infrastructure. The source code is already ASF 2.0
> > > licensed.
> > >
> > > == External Dependencies ==
> > >
> > > All external libraries have ASF 2.0 compatible licenses except for
> > pylint.
> > > The pylint library is GPL licensed, but is only used for pre-build
> Python
> > > style checks and is neither bundled with, nor relied upon by, the Heron
> > > source or binary release artifacts.
> > >
> > > == Cryptography ==
> > >
> > > Heron does not use any cryptography libraries.
> > >
> > > = Required Resources =
> > >
> > > == Mailing lists ==
> > >
> > >  * private@heron.incubator.apache.org (with moderated subscriptions)
> > >  * dev@heron.incubator.apache.org
> > >  * commits@heron.incubator.apache.org
> > >  * user@heron.incubator.apache.org
> > >
> > > == Subversion Directory ==
> > >
> > > Git is the preferred source control system: git://git.apache.org/heron
> > >
> > > == Issue Tracking ==
> > >
> > > JIRA: Heron (HERON)
> > >
> > > == Initial Committers ==
> > >
> > >  * Andrew Jorgensen (andrew at andrewjorgensen dot com)
> > >  * Ashvin Agrawal (ashvin at apache dot org)*
> > >  * Avrilia Floratou (avrilia dot floratou at gmail dot com)
> > >  * Bill Graham (billgraham at apache dot org)*
> > >  * Brian Hatfield (bmhatfield at gmail dot com)
> > >  * Chris Kellogg (cckellogg at gmail dot com)
> > >  * Huijun Wu (huijun dot wu dot 2010 at gmail dot com)
> > >  * Karthik Ramasamy (karthik at gmail dot com)
> > >  * Maosong Fu (maosongfu at gmail dot com)
> > >  * Neng Lu(freeneng at gmail dot com)
> > >  * Runhang Li (obj dot runhang at gmail dot com)
> > >  * Sanjeev Kulkarni (sanjeevrk at gmail dot com)
> > >  * Supun Kamburugamuve (supun at apache dot org)*
> > >  * Thomas Sun (tom dot ssf at gmail dot com)
> > >  * Yaliang Wang (yaliang dot w dot wang at ieee dot org)
> > >
> > > == Affiliations ==
> > >
> > >  * Andrew Jorgensen (Google)
> > >  * Ashvin Agrawal (Microsoft)
> > >  * Avrilia Floratou (Microsoft)
> > >  * Bill Graham (Twitter)
> > >  * Brian Hatfield (Google)
> > >  * Chris Kellogg (Twitter)
> > >  * Huijun Wu (Twitter)
> > >  * Karthik Ramasamy (Streamlio)
> > >  * Maosong Fu (Twitter)
> > >  * Neng Lu (Twitter)
> > >  * Runhang Li (Twitter)
> > >  * Sanjeev Kulkarni (Streamlio)
> > >  * Supun Kamburugamuve (Indiana University)
> > >  * Thomas Sun (Twitter)
> > >  * Yaliang Wang (Twitter)
> > >
> > > = Sponsors =
> > >
> > > == Champion ==
> > >
> > >  * Julien Le Dem (julien at apache dot org)
> > >
> > > == Nominated Mentors ==
> > >
> > >  * Jake Farrell (jfarrell at apache dot org)
> > >  * Jacques Nadeau (jacques at apache dot org)
> > >  * Julien Le Dem (julien at apache dot org)
> > >  * P. Taylor Goetz (ptgoetz at apache dot org)
> > >
> > > == Sponsoring Entity ==
> > >
> > > The Apache Incubator
> > >
> > > == Footnotes ==
> > >
> > >  * 1 - Papers detailing Heron are available at
> > > http://dl.acm.org/citation.cfm?id=2742788 and
> > > http://sites.computer.org/debull/A15dec/p15.pdf.
> > >  * 2 - http://home.apache.org/phonebook.html?uid=billgraham
> > >  * 3 - http://home.apache.org/phonebook.html?uid=ashvin
> > >  * 4 - http://home.apache.org/phonebook.html?uid=supun
> > >
> >
>
>
>
> --
> Tech Lead, WSO2 Inc
> http://wso2.org
> supunk.blogspot.com
>

Re: [VOTE] Heron to enter Apache Incubator

Posted by Supun Kamburugamuva <su...@apache.org>.
+1

Thanks,
Supun..

On Fri, Jun 16, 2017 at 4:59 PM, Sanjeev Kulkarni <sa...@streaml.io>
wrote:

> +1
>
> On Fri, Jun 16, 2017 at 1:41 PM, Bill Graham <bi...@gmail.com> wrote:
>
> > Hi,
> >
> > Based on the discussion on the incubator mailing list[1] I would like to
> > call a vote to add Heron to the Apache Incubator.
> >
> > The full proposal is available below, and is also available on the Apache
> > Incubator wiki at:
> >     https://wiki.apache.org/incubator/HeronProposal
> >
> > Please vote:
> >   [ ] +1, bring Heron into Incubator
> >   [ ] -1, do not bring Heron into Incubator, because...
> >
> > The vote will open for 7 days until Friday June 23 at 14:00 PT.
> >
> > Thank you
> >
> > 1 -
> > https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7
> > 786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E
> >
> >
> >
> > = Heron Proposal =
> >
> > = Abstract =
> > Heron is a real-time, distributed, fault-tolerant stream processing
> engine
> > initially developed by Twitter.
> >
> > = Proposal =
> >
> > Heron is a real-time stream processing engine built for high performance,
> > ease of manageability, performance predictability and developer
> > productivity[1]. We wish to develop a community around Heron to increase
> > contributions and see Heron thrive in an open forum.
> >
> > = Background =
> >
> > Heron provides the ability for developers to compose directed acyclic
> > graphs (DAGs) of real-time query execution logic (i.e. a topology) and
> > submit the topology to execute on a pluggable job scheduling system
> (e.g.,
> > Apache Aurora, YARN, Marathon, etc). Users can employ either the native
> > Heron API or the Apache Storm API to develop the topology. Heron supports
> > the Storm API for ease of migration, but beyond that Heron’s architecture
> > differs considerably from Storm’s.
> >
> > Users submit a topology to the scheduler using the Heron client, which
> uses
> > the Heron binary libraries to deploy all daemons required to run and
> manage
> > the topology. The topology therefore has no reliance on centrally managed
> > Heron services, only on a generic job scheduling system, which lends
> itself
> > well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN (among
> > others).
> >
> > The scheduler runs each topology as a job consisting of multiple
> > containers. One of the containers runs the topology master, responsible
> for
> > managing the topology. The remaining containers each runs a stream
> manager
> > responsible for data routing, a metrics manager that collects and reports
> > various metrics and a number of processes called Heron instances which
> run
> > the user-defined logic on the stream of tuples. Parallelism is achieved
> via
> > process-based isolation of Heron instances, which provides predictable
> > performance while simplifying debugging. The containers are allocated and
> > managed by the scheduler framework based on resource availability of
> nodes
> > in the cluster. The metadata for the topology, such as the physical plan
> > and execution details, are stored in the pluggable Heron State Manager
> > (e.g. Apache ZooKeeper).
> >
> > = Rationale =
> >
> > Heron is a general-purpose, modular and extensible platform that can be
> > leveraged to support common, real-time analytics use cases. There is an
> > increasing demand for open-source, scalable real-time analytics systems.
> We
> > believe that Heron can be leveraged by other organizations to build
> > streaming applications that can benefit from its robustness, high
> > performance, adaptability to cloud environments and ease of use.
> Moreover,
> > we hope that open-sourcing Heron will help to further evolve the
> technology
> > as the project attracts contributors with diverse backgrounds and areas
> of
> > expertise.
> >
> > We believe the Apache foundation is a great fit as the long-term home for
> > Heron, as it provides an established process for community-driven
> > development and decision making by consensus. This is exactly the model
> we
> > want for future Heron development.
> >
> > = Initial Goals =
> >
> >  * Move the existing codebase, website, documentation, and mailing lists
> to
> > Apache-hosted infrastructure.
> >  * Integrate with the Apache development process.
> >  * Ensure all dependencies are compliant with Apache License version 2.0.
> >  * Incrementally develop and release per Apache guidelines.
> >
> > = Current Status =
> >
> > Heron is a stable project used in production at Twitter since 2014 and
> open
> > sourced under the ASL v2 license in 2016. The Heron source code is
> > currently hosted at github.com (https://github.com/twitter/heron), which
> > will seed the Apache git repository.
> >
> > = Meritocracy =
> >
> > By submitting this incubator proposal, we’re expressing our intent to
> build
> > a diverse developer community around Heron that will conduct itself
> > according to The Apache Way and use a meritocratic means of building it's
> > committer base. Several companies and universities have already expressed
> > interest in and contributed to Heron. Our goal is to grow the Heron
> > community by encouraging open communication, contribution and
> participation
> > of all types, and ensuring that contributors are recognized
> appropriately.
> >
> > = Community =
> >
> > Heron is currently being used by Twitter, Google, Machine Zone and
> > ndustrial.io and has received significant contributions by Microsoft and
> > Streamlio. By bringing Heron into the Apache ecosystem, we believe we can
> > attract even more developers who are interested in creating real-time
> > systems to build the project's contributor base.
> >
> > == Core Developers ==
> >
> > Current core developers are engineers from Twitter, Google, Microsoft and
> > Streamlio.
> >
> > == Alignment ==
> >
> > Heron utilizes a number of Apache technologies. Heron leverages Apache
> > ZooKeeper for coordination and has scheduler implementations to integrate
> > with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache
> REEF)
> > as well as spout implementations to integrate with Apache Kafka and
> metrics
> > implementations to integrate with Scribe. Heron also implements the
> Apache
> > Storm user-level API, which allows topologies written against Storm to
> run
> > in Heron. We believe that having Heron at Apache will help further the
> > growth of the streaming compute community, as well as encourage
> cooperation
> > and developer cross pollination with other Apache projects.
> >
> > = Known Risks =
> >
> > == Orphaned Products ==
> >
> > The risk of the Heron project being abandoned is minimal. It is used in
> > production at Twitter and Google and other companies are evaluating or
> > adopting it for production use.
> >
> > == Inexperience with Open Source ==
> >
> > All of the core contributors to the project have considerable experience
> > with open source software development. Bill Graham[2], Ashvin Agrawal[3]
> > and Supun Kamburugamuve[4], committers on the project, are PMCs on other
> > Apache projects and Bill and Ashvin have gone through the Apache
> incubator
> > process. Twitter has already donated numerous projects to the ASF (e.g.,
> > Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be mentored
> > by experienced ASF members that can help with any roadblocks.
> >
> > == Homogenous Developers ==
> >
> > Initial committers come from 5 separate organizations. Our intention is
> > increase the diversity of contributing developers and their affiliations.
> > To date github contributions have come from approximately 50 contributors
> > from outside the Twitter team.
> >
> > == Reliance on Salaried Developers ==
> >
> > It is expected that Heron development will occur on both salaried time
> and
> > on volunteer time. The majority of initial committers are paid by their
> > employers to contribute to this project. We are committed to recruiting
> > additional committers from other organizations as well as non-salaried
> > committers to join project.
> >
> > == Relationships with Other Apache Products ==
> >
> > As mentioned in the Alignment section, Heron implements the Apache Storm
> > API and integrates with multiple Apache schedulers (Apache Mesos, Apache
> > Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache
> > Thrift.
> >
> > == An Excessive Fascination with the Apache Brand ==
> >
> > Heron's popularity is growing in the streaming compute space and we are
> > long time supporters of the Apache brand. This proposal is not for the
> > purpose of generating publicity through. Rather, the primary benefits to
> > joining Apache are those of community building and open decision making
> > outlined in the Rationale section.
> >
> > == Documentation ==
> >
> > This proposal exists online as
> > http://wiki.apache.org/incubator/HeronProposal. Extensive documentation
> > can
> > be found on github at https://twitter.github.io/heron and the source
> code
> > is well documented.
> >
> > == Source and Intellectual Property Submission Plan ==
> >
> > The Heron codebase is currently hosted on Github:
> > https://github.com/twitter/heron. During incubation, the codebase will
> be
> > migrated to Apache infrastructure. The source code is already ASF 2.0
> > licensed.
> >
> > == External Dependencies ==
> >
> > All external libraries have ASF 2.0 compatible licenses except for
> pylint.
> > The pylint library is GPL licensed, but is only used for pre-build Python
> > style checks and is neither bundled with, nor relied upon by, the Heron
> > source or binary release artifacts.
> >
> > == Cryptography ==
> >
> > Heron does not use any cryptography libraries.
> >
> > = Required Resources =
> >
> > == Mailing lists ==
> >
> >  * private@heron.incubator.apache.org (with moderated subscriptions)
> >  * dev@heron.incubator.apache.org
> >  * commits@heron.incubator.apache.org
> >  * user@heron.incubator.apache.org
> >
> > == Subversion Directory ==
> >
> > Git is the preferred source control system: git://git.apache.org/heron
> >
> > == Issue Tracking ==
> >
> > JIRA: Heron (HERON)
> >
> > == Initial Committers ==
> >
> >  * Andrew Jorgensen (andrew at andrewjorgensen dot com)
> >  * Ashvin Agrawal (ashvin at apache dot org)*
> >  * Avrilia Floratou (avrilia dot floratou at gmail dot com)
> >  * Bill Graham (billgraham at apache dot org)*
> >  * Brian Hatfield (bmhatfield at gmail dot com)
> >  * Chris Kellogg (cckellogg at gmail dot com)
> >  * Huijun Wu (huijun dot wu dot 2010 at gmail dot com)
> >  * Karthik Ramasamy (karthik at gmail dot com)
> >  * Maosong Fu (maosongfu at gmail dot com)
> >  * Neng Lu(freeneng at gmail dot com)
> >  * Runhang Li (obj dot runhang at gmail dot com)
> >  * Sanjeev Kulkarni (sanjeevrk at gmail dot com)
> >  * Supun Kamburugamuve (supun at apache dot org)*
> >  * Thomas Sun (tom dot ssf at gmail dot com)
> >  * Yaliang Wang (yaliang dot w dot wang at ieee dot org)
> >
> > == Affiliations ==
> >
> >  * Andrew Jorgensen (Google)
> >  * Ashvin Agrawal (Microsoft)
> >  * Avrilia Floratou (Microsoft)
> >  * Bill Graham (Twitter)
> >  * Brian Hatfield (Google)
> >  * Chris Kellogg (Twitter)
> >  * Huijun Wu (Twitter)
> >  * Karthik Ramasamy (Streamlio)
> >  * Maosong Fu (Twitter)
> >  * Neng Lu (Twitter)
> >  * Runhang Li (Twitter)
> >  * Sanjeev Kulkarni (Streamlio)
> >  * Supun Kamburugamuve (Indiana University)
> >  * Thomas Sun (Twitter)
> >  * Yaliang Wang (Twitter)
> >
> > = Sponsors =
> >
> > == Champion ==
> >
> >  * Julien Le Dem (julien at apache dot org)
> >
> > == Nominated Mentors ==
> >
> >  * Jake Farrell (jfarrell at apache dot org)
> >  * Jacques Nadeau (jacques at apache dot org)
> >  * Julien Le Dem (julien at apache dot org)
> >  * P. Taylor Goetz (ptgoetz at apache dot org)
> >
> > == Sponsoring Entity ==
> >
> > The Apache Incubator
> >
> > == Footnotes ==
> >
> >  * 1 - Papers detailing Heron are available at
> > http://dl.acm.org/citation.cfm?id=2742788 and
> > http://sites.computer.org/debull/A15dec/p15.pdf.
> >  * 2 - http://home.apache.org/phonebook.html?uid=billgraham
> >  * 3 - http://home.apache.org/phonebook.html?uid=ashvin
> >  * 4 - http://home.apache.org/phonebook.html?uid=supun
> >
>



-- 
Tech Lead, WSO2 Inc
http://wso2.org
supunk.blogspot.com

Re: [VOTE] Heron to enter Apache Incubator

Posted by Sanjeev Kulkarni <sa...@streaml.io>.
+1

On Fri, Jun 16, 2017 at 1:41 PM, Bill Graham <bi...@gmail.com> wrote:

> Hi,
>
> Based on the discussion on the incubator mailing list[1] I would like to
> call a vote to add Heron to the Apache Incubator.
>
> The full proposal is available below, and is also available on the Apache
> Incubator wiki at:
>     https://wiki.apache.org/incubator/HeronProposal
>
> Please vote:
>   [ ] +1, bring Heron into Incubator
>   [ ] -1, do not bring Heron into Incubator, because...
>
> The vote will open for 7 days until Friday June 23 at 14:00 PT.
>
> Thank you
>
> 1 -
> https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7
> 786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E
>
>
>
> = Heron Proposal =
>
> = Abstract =
> Heron is a real-time, distributed, fault-tolerant stream processing engine
> initially developed by Twitter.
>
> = Proposal =
>
> Heron is a real-time stream processing engine built for high performance,
> ease of manageability, performance predictability and developer
> productivity[1]. We wish to develop a community around Heron to increase
> contributions and see Heron thrive in an open forum.
>
> = Background =
>
> Heron provides the ability for developers to compose directed acyclic
> graphs (DAGs) of real-time query execution logic (i.e. a topology) and
> submit the topology to execute on a pluggable job scheduling system (e.g.,
> Apache Aurora, YARN, Marathon, etc). Users can employ either the native
> Heron API or the Apache Storm API to develop the topology. Heron supports
> the Storm API for ease of migration, but beyond that Heron’s architecture
> differs considerably from Storm’s.
>
> Users submit a topology to the scheduler using the Heron client, which uses
> the Heron binary libraries to deploy all daemons required to run and manage
> the topology. The topology therefore has no reliance on centrally managed
> Heron services, only on a generic job scheduling system, which lends itself
> well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN (among
> others).
>
> The scheduler runs each topology as a job consisting of multiple
> containers. One of the containers runs the topology master, responsible for
> managing the topology. The remaining containers each runs a stream manager
> responsible for data routing, a metrics manager that collects and reports
> various metrics and a number of processes called Heron instances which run
> the user-defined logic on the stream of tuples. Parallelism is achieved via
> process-based isolation of Heron instances, which provides predictable
> performance while simplifying debugging. The containers are allocated and
> managed by the scheduler framework based on resource availability of nodes
> in the cluster. The metadata for the topology, such as the physical plan
> and execution details, are stored in the pluggable Heron State Manager
> (e.g. Apache ZooKeeper).
>
> = Rationale =
>
> Heron is a general-purpose, modular and extensible platform that can be
> leveraged to support common, real-time analytics use cases. There is an
> increasing demand for open-source, scalable real-time analytics systems. We
> believe that Heron can be leveraged by other organizations to build
> streaming applications that can benefit from its robustness, high
> performance, adaptability to cloud environments and ease of use. Moreover,
> we hope that open-sourcing Heron will help to further evolve the technology
> as the project attracts contributors with diverse backgrounds and areas of
> expertise.
>
> We believe the Apache foundation is a great fit as the long-term home for
> Heron, as it provides an established process for community-driven
> development and decision making by consensus. This is exactly the model we
> want for future Heron development.
>
> = Initial Goals =
>
>  * Move the existing codebase, website, documentation, and mailing lists to
> Apache-hosted infrastructure.
>  * Integrate with the Apache development process.
>  * Ensure all dependencies are compliant with Apache License version 2.0.
>  * Incrementally develop and release per Apache guidelines.
>
> = Current Status =
>
> Heron is a stable project used in production at Twitter since 2014 and open
> sourced under the ASL v2 license in 2016. The Heron source code is
> currently hosted at github.com (https://github.com/twitter/heron), which
> will seed the Apache git repository.
>
> = Meritocracy =
>
> By submitting this incubator proposal, we’re expressing our intent to build
> a diverse developer community around Heron that will conduct itself
> according to The Apache Way and use a meritocratic means of building it's
> committer base. Several companies and universities have already expressed
> interest in and contributed to Heron. Our goal is to grow the Heron
> community by encouraging open communication, contribution and participation
> of all types, and ensuring that contributors are recognized appropriately.
>
> = Community =
>
> Heron is currently being used by Twitter, Google, Machine Zone and
> ndustrial.io and has received significant contributions by Microsoft and
> Streamlio. By bringing Heron into the Apache ecosystem, we believe we can
> attract even more developers who are interested in creating real-time
> systems to build the project's contributor base.
>
> == Core Developers ==
>
> Current core developers are engineers from Twitter, Google, Microsoft and
> Streamlio.
>
> == Alignment ==
>
> Heron utilizes a number of Apache technologies. Heron leverages Apache
> ZooKeeper for coordination and has scheduler implementations to integrate
> with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache REEF)
> as well as spout implementations to integrate with Apache Kafka and metrics
> implementations to integrate with Scribe. Heron also implements the Apache
> Storm user-level API, which allows topologies written against Storm to run
> in Heron. We believe that having Heron at Apache will help further the
> growth of the streaming compute community, as well as encourage cooperation
> and developer cross pollination with other Apache projects.
>
> = Known Risks =
>
> == Orphaned Products ==
>
> The risk of the Heron project being abandoned is minimal. It is used in
> production at Twitter and Google and other companies are evaluating or
> adopting it for production use.
>
> == Inexperience with Open Source ==
>
> All of the core contributors to the project have considerable experience
> with open source software development. Bill Graham[2], Ashvin Agrawal[3]
> and Supun Kamburugamuve[4], committers on the project, are PMCs on other
> Apache projects and Bill and Ashvin have gone through the Apache incubator
> process. Twitter has already donated numerous projects to the ASF (e.g.,
> Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be mentored
> by experienced ASF members that can help with any roadblocks.
>
> == Homogenous Developers ==
>
> Initial committers come from 5 separate organizations. Our intention is
> increase the diversity of contributing developers and their affiliations.
> To date github contributions have come from approximately 50 contributors
> from outside the Twitter team.
>
> == Reliance on Salaried Developers ==
>
> It is expected that Heron development will occur on both salaried time and
> on volunteer time. The majority of initial committers are paid by their
> employers to contribute to this project. We are committed to recruiting
> additional committers from other organizations as well as non-salaried
> committers to join project.
>
> == Relationships with Other Apache Products ==
>
> As mentioned in the Alignment section, Heron implements the Apache Storm
> API and integrates with multiple Apache schedulers (Apache Mesos, Apache
> Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache
> Thrift.
>
> == An Excessive Fascination with the Apache Brand ==
>
> Heron's popularity is growing in the streaming compute space and we are
> long time supporters of the Apache brand. This proposal is not for the
> purpose of generating publicity through. Rather, the primary benefits to
> joining Apache are those of community building and open decision making
> outlined in the Rationale section.
>
> == Documentation ==
>
> This proposal exists online as
> http://wiki.apache.org/incubator/HeronProposal. Extensive documentation
> can
> be found on github at https://twitter.github.io/heron and the source code
> is well documented.
>
> == Source and Intellectual Property Submission Plan ==
>
> The Heron codebase is currently hosted on Github:
> https://github.com/twitter/heron. During incubation, the codebase will be
> migrated to Apache infrastructure. The source code is already ASF 2.0
> licensed.
>
> == External Dependencies ==
>
> All external libraries have ASF 2.0 compatible licenses except for pylint.
> The pylint library is GPL licensed, but is only used for pre-build Python
> style checks and is neither bundled with, nor relied upon by, the Heron
> source or binary release artifacts.
>
> == Cryptography ==
>
> Heron does not use any cryptography libraries.
>
> = Required Resources =
>
> == Mailing lists ==
>
>  * private@heron.incubator.apache.org (with moderated subscriptions)
>  * dev@heron.incubator.apache.org
>  * commits@heron.incubator.apache.org
>  * user@heron.incubator.apache.org
>
> == Subversion Directory ==
>
> Git is the preferred source control system: git://git.apache.org/heron
>
> == Issue Tracking ==
>
> JIRA: Heron (HERON)
>
> == Initial Committers ==
>
>  * Andrew Jorgensen (andrew at andrewjorgensen dot com)
>  * Ashvin Agrawal (ashvin at apache dot org)*
>  * Avrilia Floratou (avrilia dot floratou at gmail dot com)
>  * Bill Graham (billgraham at apache dot org)*
>  * Brian Hatfield (bmhatfield at gmail dot com)
>  * Chris Kellogg (cckellogg at gmail dot com)
>  * Huijun Wu (huijun dot wu dot 2010 at gmail dot com)
>  * Karthik Ramasamy (karthik at gmail dot com)
>  * Maosong Fu (maosongfu at gmail dot com)
>  * Neng Lu(freeneng at gmail dot com)
>  * Runhang Li (obj dot runhang at gmail dot com)
>  * Sanjeev Kulkarni (sanjeevrk at gmail dot com)
>  * Supun Kamburugamuve (supun at apache dot org)*
>  * Thomas Sun (tom dot ssf at gmail dot com)
>  * Yaliang Wang (yaliang dot w dot wang at ieee dot org)
>
> == Affiliations ==
>
>  * Andrew Jorgensen (Google)
>  * Ashvin Agrawal (Microsoft)
>  * Avrilia Floratou (Microsoft)
>  * Bill Graham (Twitter)
>  * Brian Hatfield (Google)
>  * Chris Kellogg (Twitter)
>  * Huijun Wu (Twitter)
>  * Karthik Ramasamy (Streamlio)
>  * Maosong Fu (Twitter)
>  * Neng Lu (Twitter)
>  * Runhang Li (Twitter)
>  * Sanjeev Kulkarni (Streamlio)
>  * Supun Kamburugamuve (Indiana University)
>  * Thomas Sun (Twitter)
>  * Yaliang Wang (Twitter)
>
> = Sponsors =
>
> == Champion ==
>
>  * Julien Le Dem (julien at apache dot org)
>
> == Nominated Mentors ==
>
>  * Jake Farrell (jfarrell at apache dot org)
>  * Jacques Nadeau (jacques at apache dot org)
>  * Julien Le Dem (julien at apache dot org)
>  * P. Taylor Goetz (ptgoetz at apache dot org)
>
> == Sponsoring Entity ==
>
> The Apache Incubator
>
> == Footnotes ==
>
>  * 1 - Papers detailing Heron are available at
> http://dl.acm.org/citation.cfm?id=2742788 and
> http://sites.computer.org/debull/A15dec/p15.pdf.
>  * 2 - http://home.apache.org/phonebook.html?uid=billgraham
>  * 3 - http://home.apache.org/phonebook.html?uid=ashvin
>  * 4 - http://home.apache.org/phonebook.html?uid=supun
>