You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@incubator.apache.org by Jean-Baptiste Onofré <jb...@nanthrax.net> on 2017/04/16 18:20:44 UTC
[VOTE] Accept Amaterasu into the Apache Incubator
Hi all,
following the discussion thread, I think we can start the vote on accepting
Amaterasu into the Apache Incubator.
The ASF voting rules are described:
http://www.apache.org/foundation/voting.html
A vote for accepting a new Apache Incubator podling is a majority vote for which
only Incubator PMC member votes are binding.
This vote will run for at least 72 hours. Please VOTE as follows
[] +1 Accept Amaterasu into the Apache Incubator
[] +0 Abstain.
[] -1 Do not accept Amaterasu into the Apache Incubator because ...
The proposal is listed below, but you can also access it on the wiki:
https://wiki.apache.org/incubator/AmaterasuProposal
Note that we are looking for one more mentor on the proposal.
One concern has been raised about the "Amaterasu" name. We want to move forward
with Amaterasu name but open to change if needed.
Thanks
Regards
--
Jean-Baptiste Onofr�
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
= Apache Amaterasu =
== Abstract ==
Apache Amaterasu is a framework providing continuous deployment for Big Data
pipelines.
It provides the following capabilities:
* '''Continuous integration''' tools to '''package pipelines and run tests'''.
* A repository to store those packaged applications: the '''applications
repository'''.
* A repository to store the pipelines, and engine configuration (for instance,
location of the Spark master, etc.): per environment - the '''configuration
repository'''.
* A '''dashboard''' to monitor the pipelines.
* A '''DSL and integration hooks''' allowing third parties to easily integrate.
== Proposal ==
Amaterasu is a simple and powerful framework to build and dispense pipelines.
It aims to help data engineers and data scientists to compose, configure, test,
package, deploy and execute data pipelines written using multiple tools,
languages and frameworks. Amaterasu provides a standard repo structure to
package big data pipelines, a YAML based Domain Specific Languages (DSL) for
data engineers, data scientists and operations engineers to manage complex
pipelines throughout their entire lifecycle (Dev, UAT, Prod, etc.).
== Background ==
Amaterasu is a relatively new project that was created to deal with some of the
issues that as Consultants, we have seen recurring at different client sites.
Mainly the need to continuously deploy complex pipelines built in multiple tools
and languages.
Amaterasu started as a pet project and is currently being evaluated by a couple
of organizations, supported by the contributors, on a personal time and
voluntary bases.
== Rational ==
As software engineers working on big data projects we have straggled for a long
time to apply the same CI/CD practices that have become the standard in the
software industry for the last few years. While some of them are possible, for
example Apache Spark is easy to unit test. However large scale pipelines are
more complex and often use data, which might be un-structured as integration
point, which requires heavy integration tests.
To automate such tests and complex deployments, we have found the need to often
handcraft scripts and use a mixture tools, so we have decided to finally build a
tool we can apply in a general way and not on a project by project basis.
Another issue Amaterasu is trying to tackle is the Integrating between the work
of software engineers, data scientists, and sometimes operations engineers. The
approach Amaterasu takes to integrate between those three schools of thought it
to provide a simple YAML based DSL that provides a simple way to integrate
different pipeline written in the native tools for each task (R, Spark in
different languages, etc.).
== Initial Goals ==
Our initial goals are to bring Amaterasu into the ASF, transition internal
engineering processes into the open, and foster a collaborative development
model according to the "Apache Way".
In addition, we intend to continue the development of Amaterasu, add new
features as well as integrate better with other frameworks, including:
* Apache Arrow
* Apache Hive
* Apache Drill
* Apache Beam
* Apache YARN
* Farther and more complete integration with Apache Spark
Other frameworks will be evaluated after those initial goals are reached.
== Current Status ==
Amaterasu is preview state but provide a large set of features. We plan to
stabilize and head to a first production ready release during the incubation
process. The current license is already Apache 2.0.
=== Meritocracy ===
We intend to radically expand the initial developer and user community by
running the project in accordance with the "Apache Way". Users and new
contributors will be treated with respect and welcomed. By participating in the
community and providing quality patches/support that move the project forward,
they will earn merit. They also will be encouraged to provide non-code
contributions (documentation, events, community management, etc.) and will gain
merit for doing so. Those with a proven support and quality track record will be
encouraged to become committers.
=== Community ===
As a relatively new project, Amaterasu has a small, but growing community.
Amaterasu is an open project, not just with it\u2019s source code but also with our
discussions which are held openly in our slack https://shintoio.slack.com which
contains channels for design, tech and future directions discussions.
If Amaterasu is accepted for incubation, the primary initial goal is to build a
large and strong community. We are confident that Amaterasu can become a key
project for big data operations, which hopefully will create a large community
of users and developers.
=== Known Risks ===
Development has been sponsored mostly by a one company. For the project to fully
transition to the Apache Way governance model, development must shift towards
the meritocracy-centric model of growing a community of contributors balanced
with the needs for extreme stability and core implementation coherency.
=== Orphaned products ===
We are fully committed on Amaterasu. A few organizations have expressed their
interest in using Amaterasu.
=== Inexperience with Open Source ===
We have been developing and using open source software for a long time.
Additionally, several ASF veterans have agreed to mentor the project and they
are listed in this proposal. The project will rely on their guidance and
collective wisdom to quickly transition the entire team of initial committers
towards practicing the Apache Way.
=== Reliance on Salaried Developers ===
Most of the current contributors are employed in the Big Data space. While they
might wander from their current employers, they are unlikely to venture far from
their core expertises and thus will continue to be engaged with the project
regardless of their current employers.
=== An Excessive Fascination with the Apache Brand ===
While we intend to leverage the Apache \u2018branding\u2019 when talking to other projects
as testament of our project\u2019s \u2018neutrality\u2019, we have no plans for making use of
Apache brand in press releases nor posting billboards advertising acceptance of
Amaterasu into Apache Incubator.
The main purpose in applying for Apache incubation is due to the fact that
Amaterasu is built with integration already in mind for many tools which are
Apache projects, and we see Amaterasu as an extension of these projects. We hope
that by being an Apache project, we can integrate better, and collaborate more
effectively with the relevant projects. As Amaterasu matures, we see mutual
benefits for all involved.
=== Initial Source ===
https://github.com/shintoio/amaterasu
=== External Dependencies ===
All external dependencies are licensed under an Apache 2.0 license or
Apache-compatible license. As we grow the Amaterasu community we will configure
our build process to require and validate all contributions and dependencies are
licensed under the Apache 2.0 license or are under an Apache-compatible license.
* Apache Spark
* Apache Hadoop
* Apache Maven (maven-core)
* Apache Commons
* Apache Log4j
* Apache Mesos
* Apache Zookeeper
* Apache Curator
* Scala
* Junit
* Py4j
Future versions are planned to integrate with:
* Apache YARN
* Apache Hive
* Apache Drill
=== Required Resources ===
==== Mailing lists ====
* private@amaterasu.incubator.apache.org (moderated subscriptions)
* commits@amaterasu.incubator.apache.org
* dev@amaterasu.incubator.apache.org
* issues@amaterasu.incubator.apache.org
==== Git Repository ====
* https://git-wip-us.apache.org/repos/asf/incubator-amaterasu.git
==== Issue Tracking ====
* JIRA Project Amaterasu
==== Initial Committers ====
* Yaniv Rodenski
* Jean-Baptiste Onofr�
* Eyal Ben Ivri
* Karel Alfonso
* Kirupagaran (Kirupa) Devarajan
* Nadav Har Tzvi
==== Affiliations ====
* Yaniv Rodenski - Shinto
* Jean-Baptiste Onofr� - Talend
* Olivier Lamy - Webtide
==== Sponsors ====
==== Champion ====
* Jean-Baptiste Onofr�
==== Mentors ====
* Jean-Baptiste Onofr�
* Olivier Lamy
==== Sponsoring Entity ====
The Apache Incubator
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [RESULT][VOTE] Accept Amaterasu into the Apache Incubator
Posted by "John D. Ament" <jo...@apache.org>.
Hi,
Just wondering, did this project ever get going? I don't see any infra
tickets for it, no entries in podlings.xml, no mailing lists etc.
John
On Tue, May 23, 2017 at 2:57 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Hi,
>
> I think we gave enough time to vote on this proposal.
>
> I close this vote with the following result:
>
> +1 (binding): Bertrand Delacretaz, Henry Saputra, Luciano Resende, Olivier
> Lamy,
> Jean-Baptiste Onofré
> +1 (non binding): Abhishek Tiwari, Yaniv Rodenski, Pierre Smits, Felix
> Cheung
>
> I will request the creation of infra and we will prepare the code donation
> (SGA,
> licensing, package names, ...).
>
> Thanks all for your vote and review
>
> Regards
> JB
>
> On 04/16/2017 08:20 PM, Jean-Baptiste Onofré wrote:
> > Hi all,
> >
> > following the discussion thread, I think we can start the vote on
> accepting
> > Amaterasu into the Apache Incubator.
> >
> > The ASF voting rules are described:
> >
> > http://www.apache.org/foundation/voting.html
> >
> > A vote for accepting a new Apache Incubator podling is a majority vote
> for which
> > only Incubator PMC member votes are binding.
> >
> > This vote will run for at least 72 hours. Please VOTE as follows
> > [] +1 Accept Amaterasu into the Apache Incubator
> > [] +0 Abstain.
> > [] -1 Do not accept Amaterasu into the Apache Incubator because ...
> >
> > The proposal is listed below, but you can also access it on the wiki:
> >
> > https://wiki.apache.org/incubator/AmaterasuProposal
> >
> > Note that we are looking for one more mentor on the proposal.
> > One concern has been raised about the "Amaterasu" name. We want to move
> forward
> > with Amaterasu name but open to change if needed.
> >
> > Thanks
> > Regards
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
[RESULT][VOTE] Accept Amaterasu into the Apache Incubator
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi,
I think we gave enough time to vote on this proposal.
I close this vote with the following result:
+1 (binding): Bertrand Delacretaz, Henry Saputra, Luciano Resende, Olivier Lamy,
Jean-Baptiste Onofré
+1 (non binding): Abhishek Tiwari, Yaniv Rodenski, Pierre Smits, Felix Cheung
I will request the creation of infra and we will prepare the code donation (SGA,
licensing, package names, ...).
Thanks all for your vote and review
Regards
JB
On 04/16/2017 08:20 PM, Jean-Baptiste Onofré wrote:
> Hi all,
>
> following the discussion thread, I think we can start the vote on accepting
> Amaterasu into the Apache Incubator.
>
> The ASF voting rules are described:
>
> http://www.apache.org/foundation/voting.html
>
> A vote for accepting a new Apache Incubator podling is a majority vote for which
> only Incubator PMC member votes are binding.
>
> This vote will run for at least 72 hours. Please VOTE as follows
> [] +1 Accept Amaterasu into the Apache Incubator
> [] +0 Abstain.
> [] -1 Do not accept Amaterasu into the Apache Incubator because ...
>
> The proposal is listed below, but you can also access it on the wiki:
>
> https://wiki.apache.org/incubator/AmaterasuProposal
>
> Note that we are looking for one more mentor on the proposal.
> One concern has been raised about the "Amaterasu" name. We want to move forward
> with Amaterasu name but open to change if needed.
>
> Thanks
> Regards
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [VOTE] Accept Amaterasu into the Apache Incubator
Posted by Abhishek Tiwari <ab...@apache.org>.
+1 (non-binding)
Looks interesting. CD and management of complex data pipelines is often a
repetitive and painful process, I will love to see Amaterasu evolve as a
strong solution.
Regards,
Abhishek
On Tue, Apr 18, 2017 at 11:01 PM, Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Any thoughts ?
>
> Regards
> JB
>
> On 04/16/2017 08:20 PM, Jean-Baptiste Onofré wrote:
>
>> Hi all,
>>
>> following the discussion thread, I think we can start the vote on
>> accepting
>> Amaterasu into the Apache Incubator.
>>
>> The ASF voting rules are described:
>>
>> http://www.apache.org/foundation/voting.html
>>
>> A vote for accepting a new Apache Incubator podling is a majority vote
>> for which
>> only Incubator PMC member votes are binding.
>>
>> This vote will run for at least 72 hours. Please VOTE as follows
>> [] +1 Accept Amaterasu into the Apache Incubator
>> [] +0 Abstain.
>> [] -1 Do not accept Amaterasu into the Apache Incubator because ...
>>
>> The proposal is listed below, but you can also access it on the wiki:
>>
>> https://wiki.apache.org/incubator/AmaterasuProposal
>>
>> Note that we are looking for one more mentor on the proposal.
>> One concern has been raised about the "Amaterasu" name. We want to move
>> forward
>> with Amaterasu name but open to change if needed.
>>
>> Thanks
>> Regards
>>
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
Re: [VOTE] Accept Amaterasu into the Apache Incubator
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Any thoughts ?
Regards
JB
On 04/16/2017 08:20 PM, Jean-Baptiste Onofr� wrote:
> Hi all,
>
> following the discussion thread, I think we can start the vote on accepting
> Amaterasu into the Apache Incubator.
>
> The ASF voting rules are described:
>
> http://www.apache.org/foundation/voting.html
>
> A vote for accepting a new Apache Incubator podling is a majority vote for which
> only Incubator PMC member votes are binding.
>
> This vote will run for at least 72 hours. Please VOTE as follows
> [] +1 Accept Amaterasu into the Apache Incubator
> [] +0 Abstain.
> [] -1 Do not accept Amaterasu into the Apache Incubator because ...
>
> The proposal is listed below, but you can also access it on the wiki:
>
> https://wiki.apache.org/incubator/AmaterasuProposal
>
> Note that we are looking for one more mentor on the proposal.
> One concern has been raised about the "Amaterasu" name. We want to move forward
> with Amaterasu name but open to change if needed.
>
> Thanks
> Regards
--
Jean-Baptiste Onofr�
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [VOTE] Accept Amaterasu into the Apache Incubator
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi guys,
I plan to close the vote today. Please, if you want to vote, do it asap.
Thanks !
Regards
JB
On 04/16/2017 08:20 PM, Jean-Baptiste Onofré wrote:
> Hi all,
>
> following the discussion thread, I think we can start the vote on accepting
> Amaterasu into the Apache Incubator.
>
> The ASF voting rules are described:
>
> http://www.apache.org/foundation/voting.html
>
> A vote for accepting a new Apache Incubator podling is a majority vote for which
> only Incubator PMC member votes are binding.
>
> This vote will run for at least 72 hours. Please VOTE as follows
> [] +1 Accept Amaterasu into the Apache Incubator
> [] +0 Abstain.
> [] -1 Do not accept Amaterasu into the Apache Incubator because ...
>
> The proposal is listed below, but you can also access it on the wiki:
>
> https://wiki.apache.org/incubator/AmaterasuProposal
>
> Note that we are looking for one more mentor on the proposal.
> One concern has been raised about the "Amaterasu" name. We want to move forward
> with Amaterasu name but open to change if needed.
>
> Thanks
> Regards
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [VOTE] Accept Amaterasu into the Apache Incubator
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
+1 (binding)
Regards
JB
On 04/16/2017 08:20 PM, Jean-Baptiste Onofré wrote:
> Hi all,
>
> following the discussion thread, I think we can start the vote on accepting
> Amaterasu into the Apache Incubator.
>
> The ASF voting rules are described:
>
> http://www.apache.org/foundation/voting.html
>
> A vote for accepting a new Apache Incubator podling is a majority vote for which
> only Incubator PMC member votes are binding.
>
> This vote will run for at least 72 hours. Please VOTE as follows
> [] +1 Accept Amaterasu into the Apache Incubator
> [] +0 Abstain.
> [] -1 Do not accept Amaterasu into the Apache Incubator because ...
>
> The proposal is listed below, but you can also access it on the wiki:
>
> https://wiki.apache.org/incubator/AmaterasuProposal
>
> Note that we are looking for one more mentor on the proposal.
> One concern has been raised about the "Amaterasu" name. We want to move forward
> with Amaterasu name but open to change if needed.
>
> Thanks
> Regards
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [VOTE] Accept Amaterasu into the Apache Incubator
Posted by Felix Cheung <fe...@apache.org>.
+1 (non binding)
On Wed, Apr 19, 2017 at 4:59 PM Olivier Lamy <ol...@apache.org> wrote:
> +1 (binding)
>
> On 17 April 2017 at 04:20, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>
> > Hi all,
> >
> > following the discussion thread, I think we can start the vote on
> > accepting Amaterasu into the Apache Incubator.
> >
> > The ASF voting rules are described:
> >
> > http://www.apache.org/foundation/voting.html
> >
> > A vote for accepting a new Apache Incubator podling is a majority vote
> for
> > which only Incubator PMC member votes are binding.
> >
> > This vote will run for at least 72 hours. Please VOTE as follows
> > [] +1 Accept Amaterasu into the Apache Incubator
> > [] +0 Abstain.
> > [] -1 Do not accept Amaterasu into the Apache Incubator because ...
> >
> > The proposal is listed below, but you can also access it on the wiki:
> >
> > https://wiki.apache.org/incubator/AmaterasuProposal
> >
> > Note that we are looking for one more mentor on the proposal.
> > One concern has been raised about the "Amaterasu" name. We want to move
> > forward with Amaterasu name but open to change if needed.
> >
> > Thanks
> > Regards
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
> > = Apache Amaterasu =
> >
> > == Abstract ==
> >
> > Apache Amaterasu is a framework providing continuous deployment for Big
> > Data pipelines.
> >
> > It provides the following capabilities:
> >
> > * '''Continuous integration''' tools to '''package pipelines and run
> > tests'''.
> > * A repository to store those packaged applications: the '''applications
> > repository'''.
> > * A repository to store the pipelines, and engine configuration (for
> > instance, location of the Spark master, etc.): per environment - the
> > '''configuration repository'''.
> > * A '''dashboard''' to monitor the pipelines.
> > * A '''DSL and integration hooks''' allowing third parties to easily
> > integrate.
> >
> > == Proposal ==
> >
> > Amaterasu is a simple and powerful framework to build and dispense
> > pipelines. It aims to help data engineers and data scientists to compose,
> > configure, test, package, deploy and execute data pipelines written using
> > multiple tools, languages and frameworks. Amaterasu provides a standard
> > repo structure to package big data pipelines, a YAML based Domain
> Specific
> > Languages (DSL) for data engineers, data scientists and operations
> > engineers to manage complex pipelines throughout their entire lifecycle
> > (Dev, UAT, Prod, etc.).
> >
> > == Background ==
> >
> > Amaterasu is a relatively new project that was created to deal with some
> > of the issues that as Consultants, we have seen recurring at different
> > client sites. Mainly the need to continuously deploy complex pipelines
> > built in multiple tools and languages.
> > Amaterasu started as a pet project and is currently being evaluated by a
> > couple of organizations, supported by the contributors, on a personal
> time
> > and voluntary bases.
> >
> > == Rational ==
> >
> > As software engineers working on big data projects we have straggled for
> a
> > long time to apply the same CI/CD practices that have become the standard
> > in the software industry for the last few years. While some of them are
> > possible, for example Apache Spark is easy to unit test. However large
> > scale pipelines are more complex and often use data, which might be
> > un-structured as integration point, which requires heavy integration
> tests.
> >
> > To automate such tests and complex deployments, we have found the need to
> > often handcraft scripts and use a mixture tools, so we have decided to
> > finally build a tool we can apply in a general way and not on a project
> by
> > project basis.
> >
> > Another issue Amaterasu is trying to tackle is the Integrating between
> the
> > work of software engineers, data scientists, and sometimes operations
> > engineers. The approach Amaterasu takes to integrate between those three
> > schools of thought it to provide a simple YAML based DSL that provides a
> > simple way to integrate different pipeline written in the native tools
> for
> > each task (R, Spark in different languages, etc.).
> >
> > == Initial Goals ==
> >
> > Our initial goals are to bring Amaterasu into the ASF, transition
> internal
> > engineering processes into the open, and foster a collaborative
> development
> > model according to the "Apache Way".
> >
> > In addition, we intend to continue the development of Amaterasu, add new
> > features as well as integrate better with other frameworks, including:
> >
> > * Apache Arrow
> > * Apache Hive
> > * Apache Drill
> > * Apache Beam
> > * Apache YARN
> > * Farther and more complete integration with Apache Spark
> >
> > Other frameworks will be evaluated after those initial goals are reached.
> >
> > == Current Status ==
> >
> > Amaterasu is preview state but provide a large set of features. We plan
> to
> > stabilize and head to a first production ready release during the
> > incubation process. The current license is already Apache 2.0.
> >
> > === Meritocracy ===
> >
> > We intend to radically expand the initial developer and user community by
> > running the project in accordance with the "Apache Way". Users and new
> > contributors will be treated with respect and welcomed. By participating
> in
> > the community and providing quality patches/support that move the project
> > forward, they will earn merit. They also will be encouraged to provide
> > non-code contributions (documentation, events, community management,
> etc.)
> > and will gain merit for doing so. Those with a proven support and quality
> > track record will be encouraged to become committers.
> >
> > === Community ===
> >
> > As a relatively new project, Amaterasu has a small, but growing
> community.
> > Amaterasu is an open project, not just with it’s source code but also
> with
> > our discussions which are held openly in our slack
> > https://shintoio.slack.com which contains channels for design, tech and
> > future directions discussions.
> >
> > If Amaterasu is accepted for incubation, the primary initial goal is to
> > build a large and strong community. We are confident that Amaterasu can
> > become a key project for big data operations, which hopefully will
> create a
> > large community of users and developers.
> >
> > === Known Risks ===
> >
> > Development has been sponsored mostly by a one company. For the project
> to
> > fully transition to the Apache Way governance model, development must
> shift
> > towards the meritocracy-centric model of growing a community of
> > contributors balanced with the needs for extreme stability and core
> > implementation coherency.
> >
> > === Orphaned products ===
> >
> > We are fully committed on Amaterasu. A few organizations have expressed
> > their interest in using Amaterasu.
> >
> > === Inexperience with Open Source ===
> >
> > We have been developing and using open source software for a long time.
> > Additionally, several ASF veterans have agreed to mentor the project and
> > they are listed in this proposal. The project will rely on their guidance
> > and collective wisdom to quickly transition the entire team of initial
> > committers towards practicing the Apache Way.
> >
> > === Reliance on Salaried Developers ===
> >
> > Most of the current contributors are employed in the Big Data space.
> While
> > they might wander from their current employers, they are unlikely to
> > venture far from their core expertises and thus will continue to be
> engaged
> > with the project regardless of their current employers.
> >
> > === An Excessive Fascination with the Apache Brand ===
> >
> > While we intend to leverage the Apache ‘branding’ when talking to other
> > projects as testament of our project’s ‘neutrality’, we have no plans for
> > making use of Apache brand in press releases nor posting billboards
> > advertising acceptance of Amaterasu into Apache Incubator.
> >
> > The main purpose in applying for Apache incubation is due to the fact
> that
> > Amaterasu is built with integration already in mind for many tools which
> > are Apache projects, and we see Amaterasu as an extension of these
> > projects. We hope that by being an Apache project, we can integrate
> better,
> > and collaborate more effectively with the relevant projects. As Amaterasu
> > matures, we see mutual benefits for all involved.
> >
> > === Initial Source ===
> >
> > https://github.com/shintoio/amaterasu
> >
> > === External Dependencies ===
> >
> > All external dependencies are licensed under an Apache 2.0 license or
> > Apache-compatible license. As we grow the Amaterasu community we will
> > configure our build process to require and validate all contributions and
> > dependencies are licensed under the Apache 2.0 license or are under an
> > Apache-compatible license.
> >
> > * Apache Spark
> > * Apache Hadoop
> > * Apache Maven (maven-core)
> > * Apache Commons
> > * Apache Log4j
> > * Apache Mesos
> > * Apache Zookeeper
> > * Apache Curator
> > * Scala
> > * Junit
> > * Py4j
> >
> > Future versions are planned to integrate with:
> >
> > * Apache YARN
> > * Apache Hive
> > * Apache Drill
> >
> > === Required Resources ===
> >
> > ==== Mailing lists ====
> >
> > * private@amaterasu.incubator.apache.org (moderated subscriptions)
> > * commits@amaterasu.incubator.apache.org
> > * dev@amaterasu.incubator.apache.org
> > * issues@amaterasu.incubator.apache.org
> >
> > ==== Git Repository ====
> >
> > * https://git-wip-us.apache.org/repos/asf/incubator-amaterasu.git
> >
> > ==== Issue Tracking ====
> >
> > * JIRA Project Amaterasu
> >
> > ==== Initial Committers ====
> >
> > * Yaniv Rodenski
> > * Jean-Baptiste Onofré
> > * Eyal Ben Ivri
> > * Karel Alfonso
> > * Kirupagaran (Kirupa) Devarajan
> > * Nadav Har Tzvi
> >
> > ==== Affiliations ====
> >
> > * Yaniv Rodenski - Shinto
> > * Jean-Baptiste Onofré - Talend
> > * Olivier Lamy - Webtide
> >
> > ==== Sponsors ====
> >
> > ==== Champion ====
> >
> > * Jean-Baptiste Onofré
> >
> > ==== Mentors ====
> >
> > * Jean-Baptiste Onofré
> > * Olivier Lamy
> >
> > ==== Sponsoring Entity ====
> >
> > The Apache Incubator
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
> >
>
>
> --
> Olivier Lamy
> http://twitter.com/olamy | http://linkedin.com/in/olamy
>
Re: [VOTE] Accept Amaterasu into the Apache Incubator
Posted by Olivier Lamy <ol...@apache.org>.
+1 (binding)
On 17 April 2017 at 04:20, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> Hi all,
>
> following the discussion thread, I think we can start the vote on
> accepting Amaterasu into the Apache Incubator.
>
> The ASF voting rules are described:
>
> http://www.apache.org/foundation/voting.html
>
> A vote for accepting a new Apache Incubator podling is a majority vote for
> which only Incubator PMC member votes are binding.
>
> This vote will run for at least 72 hours. Please VOTE as follows
> [] +1 Accept Amaterasu into the Apache Incubator
> [] +0 Abstain.
> [] -1 Do not accept Amaterasu into the Apache Incubator because ...
>
> The proposal is listed below, but you can also access it on the wiki:
>
> https://wiki.apache.org/incubator/AmaterasuProposal
>
> Note that we are looking for one more mentor on the proposal.
> One concern has been raised about the "Amaterasu" name. We want to move
> forward with Amaterasu name but open to change if needed.
>
> Thanks
> Regards
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
> = Apache Amaterasu =
>
> == Abstract ==
>
> Apache Amaterasu is a framework providing continuous deployment for Big
> Data pipelines.
>
> It provides the following capabilities:
>
> * '''Continuous integration''' tools to '''package pipelines and run
> tests'''.
> * A repository to store those packaged applications: the '''applications
> repository'''.
> * A repository to store the pipelines, and engine configuration (for
> instance, location of the Spark master, etc.): per environment - the
> '''configuration repository'''.
> * A '''dashboard''' to monitor the pipelines.
> * A '''DSL and integration hooks''' allowing third parties to easily
> integrate.
>
> == Proposal ==
>
> Amaterasu is a simple and powerful framework to build and dispense
> pipelines. It aims to help data engineers and data scientists to compose,
> configure, test, package, deploy and execute data pipelines written using
> multiple tools, languages and frameworks. Amaterasu provides a standard
> repo structure to package big data pipelines, a YAML based Domain Specific
> Languages (DSL) for data engineers, data scientists and operations
> engineers to manage complex pipelines throughout their entire lifecycle
> (Dev, UAT, Prod, etc.).
>
> == Background ==
>
> Amaterasu is a relatively new project that was created to deal with some
> of the issues that as Consultants, we have seen recurring at different
> client sites. Mainly the need to continuously deploy complex pipelines
> built in multiple tools and languages.
> Amaterasu started as a pet project and is currently being evaluated by a
> couple of organizations, supported by the contributors, on a personal time
> and voluntary bases.
>
> == Rational ==
>
> As software engineers working on big data projects we have straggled for a
> long time to apply the same CI/CD practices that have become the standard
> in the software industry for the last few years. While some of them are
> possible, for example Apache Spark is easy to unit test. However large
> scale pipelines are more complex and often use data, which might be
> un-structured as integration point, which requires heavy integration tests.
>
> To automate such tests and complex deployments, we have found the need to
> often handcraft scripts and use a mixture tools, so we have decided to
> finally build a tool we can apply in a general way and not on a project by
> project basis.
>
> Another issue Amaterasu is trying to tackle is the Integrating between the
> work of software engineers, data scientists, and sometimes operations
> engineers. The approach Amaterasu takes to integrate between those three
> schools of thought it to provide a simple YAML based DSL that provides a
> simple way to integrate different pipeline written in the native tools for
> each task (R, Spark in different languages, etc.).
>
> == Initial Goals ==
>
> Our initial goals are to bring Amaterasu into the ASF, transition internal
> engineering processes into the open, and foster a collaborative development
> model according to the "Apache Way".
>
> In addition, we intend to continue the development of Amaterasu, add new
> features as well as integrate better with other frameworks, including:
>
> * Apache Arrow
> * Apache Hive
> * Apache Drill
> * Apache Beam
> * Apache YARN
> * Farther and more complete integration with Apache Spark
>
> Other frameworks will be evaluated after those initial goals are reached.
>
> == Current Status ==
>
> Amaterasu is preview state but provide a large set of features. We plan to
> stabilize and head to a first production ready release during the
> incubation process. The current license is already Apache 2.0.
>
> === Meritocracy ===
>
> We intend to radically expand the initial developer and user community by
> running the project in accordance with the "Apache Way". Users and new
> contributors will be treated with respect and welcomed. By participating in
> the community and providing quality patches/support that move the project
> forward, they will earn merit. They also will be encouraged to provide
> non-code contributions (documentation, events, community management, etc.)
> and will gain merit for doing so. Those with a proven support and quality
> track record will be encouraged to become committers.
>
> === Community ===
>
> As a relatively new project, Amaterasu has a small, but growing community.
> Amaterasu is an open project, not just with it’s source code but also with
> our discussions which are held openly in our slack
> https://shintoio.slack.com which contains channels for design, tech and
> future directions discussions.
>
> If Amaterasu is accepted for incubation, the primary initial goal is to
> build a large and strong community. We are confident that Amaterasu can
> become a key project for big data operations, which hopefully will create a
> large community of users and developers.
>
> === Known Risks ===
>
> Development has been sponsored mostly by a one company. For the project to
> fully transition to the Apache Way governance model, development must shift
> towards the meritocracy-centric model of growing a community of
> contributors balanced with the needs for extreme stability and core
> implementation coherency.
>
> === Orphaned products ===
>
> We are fully committed on Amaterasu. A few organizations have expressed
> their interest in using Amaterasu.
>
> === Inexperience with Open Source ===
>
> We have been developing and using open source software for a long time.
> Additionally, several ASF veterans have agreed to mentor the project and
> they are listed in this proposal. The project will rely on their guidance
> and collective wisdom to quickly transition the entire team of initial
> committers towards practicing the Apache Way.
>
> === Reliance on Salaried Developers ===
>
> Most of the current contributors are employed in the Big Data space. While
> they might wander from their current employers, they are unlikely to
> venture far from their core expertises and thus will continue to be engaged
> with the project regardless of their current employers.
>
> === An Excessive Fascination with the Apache Brand ===
>
> While we intend to leverage the Apache ‘branding’ when talking to other
> projects as testament of our project’s ‘neutrality’, we have no plans for
> making use of Apache brand in press releases nor posting billboards
> advertising acceptance of Amaterasu into Apache Incubator.
>
> The main purpose in applying for Apache incubation is due to the fact that
> Amaterasu is built with integration already in mind for many tools which
> are Apache projects, and we see Amaterasu as an extension of these
> projects. We hope that by being an Apache project, we can integrate better,
> and collaborate more effectively with the relevant projects. As Amaterasu
> matures, we see mutual benefits for all involved.
>
> === Initial Source ===
>
> https://github.com/shintoio/amaterasu
>
> === External Dependencies ===
>
> All external dependencies are licensed under an Apache 2.0 license or
> Apache-compatible license. As we grow the Amaterasu community we will
> configure our build process to require and validate all contributions and
> dependencies are licensed under the Apache 2.0 license or are under an
> Apache-compatible license.
>
> * Apache Spark
> * Apache Hadoop
> * Apache Maven (maven-core)
> * Apache Commons
> * Apache Log4j
> * Apache Mesos
> * Apache Zookeeper
> * Apache Curator
> * Scala
> * Junit
> * Py4j
>
> Future versions are planned to integrate with:
>
> * Apache YARN
> * Apache Hive
> * Apache Drill
>
> === Required Resources ===
>
> ==== Mailing lists ====
>
> * private@amaterasu.incubator.apache.org (moderated subscriptions)
> * commits@amaterasu.incubator.apache.org
> * dev@amaterasu.incubator.apache.org
> * issues@amaterasu.incubator.apache.org
>
> ==== Git Repository ====
>
> * https://git-wip-us.apache.org/repos/asf/incubator-amaterasu.git
>
> ==== Issue Tracking ====
>
> * JIRA Project Amaterasu
>
> ==== Initial Committers ====
>
> * Yaniv Rodenski
> * Jean-Baptiste Onofré
> * Eyal Ben Ivri
> * Karel Alfonso
> * Kirupagaran (Kirupa) Devarajan
> * Nadav Har Tzvi
>
> ==== Affiliations ====
>
> * Yaniv Rodenski - Shinto
> * Jean-Baptiste Onofré - Talend
> * Olivier Lamy - Webtide
>
> ==== Sponsors ====
>
> ==== Champion ====
>
> * Jean-Baptiste Onofré
>
> ==== Mentors ====
>
> * Jean-Baptiste Onofré
> * Olivier Lamy
>
> ==== Sponsoring Entity ====
>
> The Apache Incubator
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
--
Olivier Lamy
http://twitter.com/olamy | http://linkedin.com/in/olamy
Re: [VOTE] Accept Amaterasu into the Apache Incubator
Posted by Pierre Smits <pi...@gmail.com>.
+1 (from the peanut gallery).
Best regards,
Pierre Smits
ORRTIZ.COM <http://www.orrtiz.com>
OFBiz based solutions & services
OFBiz Extensions Marketplace
http://oem.ofbizci.net/oci-2/
On Wed, Apr 19, 2017 at 11:40 AM, Yaniv Rodenski <ro...@gmail.com> wrote:
> +1 non-binding
>
> We are very happy with both of them :) but if anyone wants to join that
> would be great too.
>
> On Wed, 19 Apr 2017 at 7:35 pm, Bertrand Delacretaz <
> bdelacretaz@codeconsult.ch> wrote:
>
> > On Sun, Apr 16, 2017 at 8:20 PM, Jean-Baptiste Onofré <jb...@nanthrax.net>
> > wrote:
> > > ...following the discussion thread, I think we can start the vote on
> > accepting
> > > Amaterasu into the Apache Incubator...
> >
> > +1
> >
> > Having only two mentors is fine IMO if they are active.
> >
> > -Bertrand
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
> >
>
Re: [VOTE] Accept Amaterasu into the Apache Incubator
Posted by Yaniv Rodenski <ro...@gmail.com>.
+1 non-binding
We are very happy with both of them :) but if anyone wants to join that
would be great too.
On Wed, 19 Apr 2017 at 7:35 pm, Bertrand Delacretaz <
bdelacretaz@codeconsult.ch> wrote:
> On Sun, Apr 16, 2017 at 8:20 PM, Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
> > ...following the discussion thread, I think we can start the vote on
> accepting
> > Amaterasu into the Apache Incubator...
>
> +1
>
> Having only two mentors is fine IMO if they are active.
>
> -Bertrand
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
Re: [VOTE] Accept Amaterasu into the Apache Incubator
Posted by Bertrand Delacretaz <bd...@codeconsult.ch>.
On Sun, Apr 16, 2017 at 8:20 PM, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> ...following the discussion thread, I think we can start the vote on accepting
> Amaterasu into the Apache Incubator...
+1
Having only two mentors is fine IMO if they are active.
-Bertrand
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [VOTE] Accept Amaterasu into the Apache Incubator
Posted by Luciano Resende <lu...@gmail.com>.
+1 (binding)
On Sun, Apr 16, 2017 at 11:20 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Hi all,
>
> following the discussion thread, I think we can start the vote on
> accepting Amaterasu into the Apache Incubator.
>
> The ASF voting rules are described:
>
> http://www.apache.org/foundation/voting.html
>
> A vote for accepting a new Apache Incubator podling is a majority vote for
> which only Incubator PMC member votes are binding.
>
> This vote will run for at least 72 hours. Please VOTE as follows
> [] +1 Accept Amaterasu into the Apache Incubator
> [] +0 Abstain.
> [] -1 Do not accept Amaterasu into the Apache Incubator because ...
>
> The proposal is listed below, but you can also access it on the wiki:
>
> https://wiki.apache.org/incubator/AmaterasuProposal
>
> Note that we are looking for one more mentor on the proposal.
> One concern has been raised about the "Amaterasu" name. We want to move
> forward with Amaterasu name but open to change if needed.
>
> Thanks
> Regards
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
> = Apache Amaterasu =
>
> == Abstract ==
>
> Apache Amaterasu is a framework providing continuous deployment for Big
> Data pipelines.
>
> It provides the following capabilities:
>
> * '''Continuous integration''' tools to '''package pipelines and run
> tests'''.
> * A repository to store those packaged applications: the '''applications
> repository'''.
> * A repository to store the pipelines, and engine configuration (for
> instance, location of the Spark master, etc.): per environment - the
> '''configuration repository'''.
> * A '''dashboard''' to monitor the pipelines.
> * A '''DSL and integration hooks''' allowing third parties to easily
> integrate.
>
> == Proposal ==
>
> Amaterasu is a simple and powerful framework to build and dispense
> pipelines. It aims to help data engineers and data scientists to compose,
> configure, test, package, deploy and execute data pipelines written using
> multiple tools, languages and frameworks. Amaterasu provides a standard
> repo structure to package big data pipelines, a YAML based Domain Specific
> Languages (DSL) for data engineers, data scientists and operations
> engineers to manage complex pipelines throughout their entire lifecycle
> (Dev, UAT, Prod, etc.).
>
> == Background ==
>
> Amaterasu is a relatively new project that was created to deal with some
> of the issues that as Consultants, we have seen recurring at different
> client sites. Mainly the need to continuously deploy complex pipelines
> built in multiple tools and languages.
> Amaterasu started as a pet project and is currently being evaluated by a
> couple of organizations, supported by the contributors, on a personal time
> and voluntary bases.
>
> == Rational ==
>
> As software engineers working on big data projects we have straggled for a
> long time to apply the same CI/CD practices that have become the standard
> in the software industry for the last few years. While some of them are
> possible, for example Apache Spark is easy to unit test. However large
> scale pipelines are more complex and often use data, which might be
> un-structured as integration point, which requires heavy integration tests.
>
> To automate such tests and complex deployments, we have found the need to
> often handcraft scripts and use a mixture tools, so we have decided to
> finally build a tool we can apply in a general way and not on a project by
> project basis.
>
> Another issue Amaterasu is trying to tackle is the Integrating between the
> work of software engineers, data scientists, and sometimes operations
> engineers. The approach Amaterasu takes to integrate between those three
> schools of thought it to provide a simple YAML based DSL that provides a
> simple way to integrate different pipeline written in the native tools for
> each task (R, Spark in different languages, etc.).
>
> == Initial Goals ==
>
> Our initial goals are to bring Amaterasu into the ASF, transition internal
> engineering processes into the open, and foster a collaborative development
> model according to the "Apache Way".
>
> In addition, we intend to continue the development of Amaterasu, add new
> features as well as integrate better with other frameworks, including:
>
> * Apache Arrow
> * Apache Hive
> * Apache Drill
> * Apache Beam
> * Apache YARN
> * Farther and more complete integration with Apache Spark
>
> Other frameworks will be evaluated after those initial goals are reached.
>
> == Current Status ==
>
> Amaterasu is preview state but provide a large set of features. We plan to
> stabilize and head to a first production ready release during the
> incubation process. The current license is already Apache 2.0.
>
> === Meritocracy ===
>
> We intend to radically expand the initial developer and user community by
> running the project in accordance with the "Apache Way". Users and new
> contributors will be treated with respect and welcomed. By participating in
> the community and providing quality patches/support that move the project
> forward, they will earn merit. They also will be encouraged to provide
> non-code contributions (documentation, events, community management, etc.)
> and will gain merit for doing so. Those with a proven support and quality
> track record will be encouraged to become committers.
>
> === Community ===
>
> As a relatively new project, Amaterasu has a small, but growing community.
> Amaterasu is an open project, not just with it’s source code but also with
> our discussions which are held openly in our slack
> https://shintoio.slack.com which contains channels for design, tech and
> future directions discussions.
>
> If Amaterasu is accepted for incubation, the primary initial goal is to
> build a large and strong community. We are confident that Amaterasu can
> become a key project for big data operations, which hopefully will create a
> large community of users and developers.
>
> === Known Risks ===
>
> Development has been sponsored mostly by a one company. For the project to
> fully transition to the Apache Way governance model, development must shift
> towards the meritocracy-centric model of growing a community of
> contributors balanced with the needs for extreme stability and core
> implementation coherency.
>
> === Orphaned products ===
>
> We are fully committed on Amaterasu. A few organizations have expressed
> their interest in using Amaterasu.
>
> === Inexperience with Open Source ===
>
> We have been developing and using open source software for a long time.
> Additionally, several ASF veterans have agreed to mentor the project and
> they are listed in this proposal. The project will rely on their guidance
> and collective wisdom to quickly transition the entire team of initial
> committers towards practicing the Apache Way.
>
> === Reliance on Salaried Developers ===
>
> Most of the current contributors are employed in the Big Data space. While
> they might wander from their current employers, they are unlikely to
> venture far from their core expertises and thus will continue to be engaged
> with the project regardless of their current employers.
>
> === An Excessive Fascination with the Apache Brand ===
>
> While we intend to leverage the Apache ‘branding’ when talking to other
> projects as testament of our project’s ‘neutrality’, we have no plans for
> making use of Apache brand in press releases nor posting billboards
> advertising acceptance of Amaterasu into Apache Incubator.
>
> The main purpose in applying for Apache incubation is due to the fact that
> Amaterasu is built with integration already in mind for many tools which
> are Apache projects, and we see Amaterasu as an extension of these
> projects. We hope that by being an Apache project, we can integrate better,
> and collaborate more effectively with the relevant projects. As Amaterasu
> matures, we see mutual benefits for all involved.
>
> === Initial Source ===
>
> https://github.com/shintoio/amaterasu
>
> === External Dependencies ===
>
> All external dependencies are licensed under an Apache 2.0 license or
> Apache-compatible license. As we grow the Amaterasu community we will
> configure our build process to require and validate all contributions and
> dependencies are licensed under the Apache 2.0 license or are under an
> Apache-compatible license.
>
> * Apache Spark
> * Apache Hadoop
> * Apache Maven (maven-core)
> * Apache Commons
> * Apache Log4j
> * Apache Mesos
> * Apache Zookeeper
> * Apache Curator
> * Scala
> * Junit
> * Py4j
>
> Future versions are planned to integrate with:
>
> * Apache YARN
> * Apache Hive
> * Apache Drill
>
> === Required Resources ===
>
> ==== Mailing lists ====
>
> * private@amaterasu.incubator.apache.org (moderated subscriptions)
> * commits@amaterasu.incubator.apache.org
> * dev@amaterasu.incubator.apache.org
> * issues@amaterasu.incubator.apache.org
>
> ==== Git Repository ====
>
> * https://git-wip-us.apache.org/repos/asf/incubator-amaterasu.git
>
> ==== Issue Tracking ====
>
> * JIRA Project Amaterasu
>
> ==== Initial Committers ====
>
> * Yaniv Rodenski
> * Jean-Baptiste Onofré
> * Eyal Ben Ivri
> * Karel Alfonso
> * Kirupagaran (Kirupa) Devarajan
> * Nadav Har Tzvi
>
> ==== Affiliations ====
>
> * Yaniv Rodenski - Shinto
> * Jean-Baptiste Onofré - Talend
> * Olivier Lamy - Webtide
>
> ==== Sponsors ====
>
> ==== Champion ====
>
> * Jean-Baptiste Onofré
>
> ==== Mentors ====
>
> * Jean-Baptiste Onofré
> * Olivier Lamy
>
> ==== Sponsoring Entity ====
>
> The Apache Incubator
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
--
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/
Re: [VOTE] Accept Amaterasu into the Apache Incubator
Posted by Henry Saputra <he...@gmail.com>.
+1 (binding)
On Sun, Apr 16, 2017 at 11:20 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Hi all,
>
> following the discussion thread, I think we can start the vote on
> accepting Amaterasu into the Apache Incubator.
>
> The ASF voting rules are described:
>
> http://www.apache.org/foundation/voting.html
>
> A vote for accepting a new Apache Incubator podling is a majority vote for
> which only Incubator PMC member votes are binding.
>
> This vote will run for at least 72 hours. Please VOTE as follows
> [] +1 Accept Amaterasu into the Apache Incubator
> [] +0 Abstain.
> [] -1 Do not accept Amaterasu into the Apache Incubator because ...
>
> The proposal is listed below, but you can also access it on the wiki:
>
> https://wiki.apache.org/incubator/AmaterasuProposal
>
> Note that we are looking for one more mentor on the proposal.
> One concern has been raised about the "Amaterasu" name. We want to move
> forward with Amaterasu name but open to change if needed.
>
> Thanks
> Regards
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
> = Apache Amaterasu =
>
> == Abstract ==
>
> Apache Amaterasu is a framework providing continuous deployment for Big
> Data pipelines.
>
> It provides the following capabilities:
>
> * '''Continuous integration''' tools to '''package pipelines and run
> tests'''.
> * A repository to store those packaged applications: the '''applications
> repository'''.
> * A repository to store the pipelines, and engine configuration (for
> instance, location of the Spark master, etc.): per environment - the
> '''configuration repository'''.
> * A '''dashboard''' to monitor the pipelines.
> * A '''DSL and integration hooks''' allowing third parties to easily
> integrate.
>
> == Proposal ==
>
> Amaterasu is a simple and powerful framework to build and dispense
> pipelines. It aims to help data engineers and data scientists to compose,
> configure, test, package, deploy and execute data pipelines written using
> multiple tools, languages and frameworks. Amaterasu provides a standard
> repo structure to package big data pipelines, a YAML based Domain Specific
> Languages (DSL) for data engineers, data scientists and operations
> engineers to manage complex pipelines throughout their entire lifecycle
> (Dev, UAT, Prod, etc.).
>
> == Background ==
>
> Amaterasu is a relatively new project that was created to deal with some
> of the issues that as Consultants, we have seen recurring at different
> client sites. Mainly the need to continuously deploy complex pipelines
> built in multiple tools and languages.
> Amaterasu started as a pet project and is currently being evaluated by a
> couple of organizations, supported by the contributors, on a personal time
> and voluntary bases.
>
> == Rational ==
>
> As software engineers working on big data projects we have straggled for a
> long time to apply the same CI/CD practices that have become the standard
> in the software industry for the last few years. While some of them are
> possible, for example Apache Spark is easy to unit test. However large
> scale pipelines are more complex and often use data, which might be
> un-structured as integration point, which requires heavy integration tests.
>
> To automate such tests and complex deployments, we have found the need to
> often handcraft scripts and use a mixture tools, so we have decided to
> finally build a tool we can apply in a general way and not on a project by
> project basis.
>
> Another issue Amaterasu is trying to tackle is the Integrating between the
> work of software engineers, data scientists, and sometimes operations
> engineers. The approach Amaterasu takes to integrate between those three
> schools of thought it to provide a simple YAML based DSL that provides a
> simple way to integrate different pipeline written in the native tools for
> each task (R, Spark in different languages, etc.).
>
> == Initial Goals ==
>
> Our initial goals are to bring Amaterasu into the ASF, transition internal
> engineering processes into the open, and foster a collaborative development
> model according to the "Apache Way".
>
> In addition, we intend to continue the development of Amaterasu, add new
> features as well as integrate better with other frameworks, including:
>
> * Apache Arrow
> * Apache Hive
> * Apache Drill
> * Apache Beam
> * Apache YARN
> * Farther and more complete integration with Apache Spark
>
> Other frameworks will be evaluated after those initial goals are reached.
>
> == Current Status ==
>
> Amaterasu is preview state but provide a large set of features. We plan to
> stabilize and head to a first production ready release during the
> incubation process. The current license is already Apache 2.0.
>
> === Meritocracy ===
>
> We intend to radically expand the initial developer and user community by
> running the project in accordance with the "Apache Way". Users and new
> contributors will be treated with respect and welcomed. By participating in
> the community and providing quality patches/support that move the project
> forward, they will earn merit. They also will be encouraged to provide
> non-code contributions (documentation, events, community management, etc.)
> and will gain merit for doing so. Those with a proven support and quality
> track record will be encouraged to become committers.
>
> === Community ===
>
> As a relatively new project, Amaterasu has a small, but growing community.
> Amaterasu is an open project, not just with it’s source code but also with
> our discussions which are held openly in our slack
> https://shintoio.slack.com which contains channels for design, tech and
> future directions discussions.
>
> If Amaterasu is accepted for incubation, the primary initial goal is to
> build a large and strong community. We are confident that Amaterasu can
> become a key project for big data operations, which hopefully will create a
> large community of users and developers.
>
> === Known Risks ===
>
> Development has been sponsored mostly by a one company. For the project to
> fully transition to the Apache Way governance model, development must shift
> towards the meritocracy-centric model of growing a community of
> contributors balanced with the needs for extreme stability and core
> implementation coherency.
>
> === Orphaned products ===
>
> We are fully committed on Amaterasu. A few organizations have expressed
> their interest in using Amaterasu.
>
> === Inexperience with Open Source ===
>
> We have been developing and using open source software for a long time.
> Additionally, several ASF veterans have agreed to mentor the project and
> they are listed in this proposal. The project will rely on their guidance
> and collective wisdom to quickly transition the entire team of initial
> committers towards practicing the Apache Way.
>
> === Reliance on Salaried Developers ===
>
> Most of the current contributors are employed in the Big Data space. While
> they might wander from their current employers, they are unlikely to
> venture far from their core expertises and thus will continue to be engaged
> with the project regardless of their current employers.
>
> === An Excessive Fascination with the Apache Brand ===
>
> While we intend to leverage the Apache ‘branding’ when talking to other
> projects as testament of our project’s ‘neutrality’, we have no plans for
> making use of Apache brand in press releases nor posting billboards
> advertising acceptance of Amaterasu into Apache Incubator.
>
> The main purpose in applying for Apache incubation is due to the fact that
> Amaterasu is built with integration already in mind for many tools which
> are Apache projects, and we see Amaterasu as an extension of these
> projects. We hope that by being an Apache project, we can integrate better,
> and collaborate more effectively with the relevant projects. As Amaterasu
> matures, we see mutual benefits for all involved.
>
> === Initial Source ===
>
> https://github.com/shintoio/amaterasu
>
> === External Dependencies ===
>
> All external dependencies are licensed under an Apache 2.0 license or
> Apache-compatible license. As we grow the Amaterasu community we will
> configure our build process to require and validate all contributions and
> dependencies are licensed under the Apache 2.0 license or are under an
> Apache-compatible license.
>
> * Apache Spark
> * Apache Hadoop
> * Apache Maven (maven-core)
> * Apache Commons
> * Apache Log4j
> * Apache Mesos
> * Apache Zookeeper
> * Apache Curator
> * Scala
> * Junit
> * Py4j
>
> Future versions are planned to integrate with:
>
> * Apache YARN
> * Apache Hive
> * Apache Drill
>
> === Required Resources ===
>
> ==== Mailing lists ====
>
> * private@amaterasu.incubator.apache.org (moderated subscriptions)
> * commits@amaterasu.incubator.apache.org
> * dev@amaterasu.incubator.apache.org
> * issues@amaterasu.incubator.apache.org
>
> ==== Git Repository ====
>
> * https://git-wip-us.apache.org/repos/asf/incubator-amaterasu.git
>
> ==== Issue Tracking ====
>
> * JIRA Project Amaterasu
>
> ==== Initial Committers ====
>
> * Yaniv Rodenski
> * Jean-Baptiste Onofré
> * Eyal Ben Ivri
> * Karel Alfonso
> * Kirupagaran (Kirupa) Devarajan
> * Nadav Har Tzvi
>
> ==== Affiliations ====
>
> * Yaniv Rodenski - Shinto
> * Jean-Baptiste Onofré - Talend
> * Olivier Lamy - Webtide
>
> ==== Sponsors ====
>
> ==== Champion ====
>
> * Jean-Baptiste Onofré
>
> ==== Mentors ====
>
> * Jean-Baptiste Onofré
> * Olivier Lamy
>
> ==== Sponsoring Entity ====
>
> The Apache Incubator
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>